Skip to content

feat(optimization): preserve GEPA candidate graph - #577

Merged
drewstone merged 2 commits into
mainfrom
feat/gepa-population-artifact-20260811
Aug 11, 2026
Merged

feat(optimization): preserve GEPA candidate graph#577
drewstone merged 2 commits into
mainfrom
feat/gepa-population-artifact-20260811

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

What changed

  • write a bounded, digest-addressed artifact from the official GEPA 0.1.4 candidate result
  • preserve every accepted candidate, exact parent indices, aggregate and per-case selection scores, and discovery evaluation counts
  • expose a public verified reader and bind its summary into method provenance
  • keep callback observations as the complete source for rejected and refused proposals

Ownership

GEPA owns search and parentage. Agent Eval validates and stores the exact result. Runtime can join graph rows to callback proposals by the canonical candidate hash. No parent is inferred. Final cases never enter the bridge.

Proof

  • pnpm lint
  • pnpm typecheck
  • pnpm verify:package
  • full TypeScript suite: 390 files passed, 5,393 tests passed, 3 skipped
  • pinned gepa==0.1.4 text and component integration plus bridge tests: 29 passed, 1 source-composition skip
  • real text callback surfaces join artifact candidates 1:1; real component callback surface equals the artifact surface

The general Python suite requires mutually exclusive GEPA-source, GEPA-release, SkillOpt, and DSPy environments. Its release-GEPA slice is the applicable one and is green.

tangletools
tangletools previously approved these changes Aug 11, 2026

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — bd590e37

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-08-11T03:53:06Z

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Value Audit — sound-with-nits

Verdict sound-with-nits
Concerns 1 (1 weak-concern)
Heuristic 0.0s
Duplication 0.0s
Interrogation 118.4s (2 bridge agents)
Total 118.4s

💰 Value — sound-with-nits

Adds a content-addressed GEPA candidate-graph artifact (writer in the Python bridge, verified reader in TS) bound into method provenance with a winner cross-check — sound and in-grain; only nit is triplicated assertion helpers.

  • What it does: After each GEPA run, the Python bridge (clients/python/src/agent_eval_rpc/gepa_bridge.py:248-265) writes a NEW artifact candidate-population-{attempt}.json capturing GEPA's official result object verbatim: every accepted candidate, exact parent indices, aggregate + per-selection-scenario scores, discovery evaluation counts, and best index. A new TS reader src/campaign/gepa-candidate-population.ts:
  • Goals it achieves: Make the full GEPA search graph (not just the winner) tamper-evident and reproducible; give runtime a verified, joinable provenance handle; prove GEPA's declared winner is identical to the population's best; keep callback observations authoritative for rejected/refused proposals (per the ownership split in the PR body, which the code matches).
  • Assessment: Good change on its merits. Validation is symmetric (Python writer validates structure at write time in _candidate_population_artifact; TS reader re-validates digest+bytes+schema+invariants at read time), everything fails loud with RuntimeError/Error (matches the repo's 'No fallbacks. Fail loud.' doctrine), and it reuses existing primitives — contentHash from verdict-cache, deepFreezeCanonicalJson,
  • Better / existing approach: Mostly none — a separate GEPA-result artifact is the right call and is not covered by the existing callback-observation reader. Minor: the new file adds a third local copy of assertExactKeys (now in src/analyst/benchmark-command-artifact.ts:210 [exported], src/campaign/external-optimizer-observations.ts:381, and the new src/campaign/gepa-candidate-population.ts:341) and a third copy of assertPosit
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 2
  • Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound

Preserves GEPA's exact candidate graph as a digest-addressed artifact wired through the bridge, method provenance, a verified public reader, and a hash-join key consistent with callback observations.

  • Integration: Fully reachable end-to-end. The Python bridge writes candidate-population-{attemptId}.json and returns its summary (gepa_bridge.py:249-288); gepaOptimizationMethod reads it via the verified reader, asserts the winner hash matches the population's best candidate, and binds the summary into provenance.gepaCandidatePopulation (gepa-optimization-method.ts:455-471, 548-550); compareOptimizationMethods
  • Fit with existing patterns: Follows the codebase's established artifact-summary-in-provenance pattern exactly — same shape as observations and modelExecutions: bridge writes a digest-addressed JSON file, returns a {scope,path,sha256,bytes,...} summary, the TS method binds it into provenance, and a verified reader re-checks digest+bounds+schema on read. No competing capability exists; observations carry rejected/refused propo
  • Real-world viability: Strong validation on both sides: sha256+byte-count+bounds checks, parent acyclicity (parent<index, root has single null parent), aggregate-must-equal-mean-of-selection-scores, winner hash match, discovery-count typing. The strict aggregate-consistency assertion (rel_tol 1e-9) is grounded by the pinned-GEPA-0.1.4 release test and fails loud per repo doctrine if a future version changes the reductio
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 1

💰 Value Audit

🟡 assertExactKeys / assertPositiveSafeInteger now triplicated [duplication] ``

The new src/campaign/gepa-candidate-population.ts:341,353 introduces local assertExactKeys and assertPositiveSafeInteger. assertExactKeys already exists as an export at src/analyst/benchmark-command-artifact.ts:210 and as a local fn at src/campaign/external-optimizer-observations.ts:381; assertPositiveSafeInteger already exists at src/campaign/gepa-optimization-config.ts:364 and src/campaign/skillopt-optimization-config.ts:154. Consider extracting these (plus the shared read->verify-digest->veri


What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass What it asks
Heuristic Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication Do added function/class names already exist elsewhere in the repo?
Value Audit What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

value-audit · 20260811T035709Z

@drewstone
drewstone merged commit c3819f3 into main Aug 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants