feat(optimization): preserve GEPA candidate graph - #577
Conversation
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — bd590e37
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-08-11T03:53:06Z
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Concerns | 1 (1 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.0s |
| Interrogation | 118.4s (2 bridge agents) |
| Total | 118.4s |
💰 Value — sound-with-nits
Adds a content-addressed GEPA candidate-graph artifact (writer in the Python bridge, verified reader in TS) bound into method provenance with a winner cross-check — sound and in-grain; only nit is triplicated assertion helpers.
- What it does: After each GEPA run, the Python bridge (clients/python/src/agent_eval_rpc/gepa_bridge.py:248-265) writes a NEW artifact candidate-population-{attempt}.json capturing GEPA's official result object verbatim: every accepted candidate, exact parent indices, aggregate + per-selection-scenario scores, discovery evaluation counts, and best index. A new TS reader src/campaign/gepa-candidate-population.ts:
- Goals it achieves: Make the full GEPA search graph (not just the winner) tamper-evident and reproducible; give runtime a verified, joinable provenance handle; prove GEPA's declared winner is identical to the population's best; keep callback observations authoritative for rejected/refused proposals (per the ownership split in the PR body, which the code matches).
- Assessment: Good change on its merits. Validation is symmetric (Python writer validates structure at write time in _candidate_population_artifact; TS reader re-validates digest+bytes+schema+invariants at read time), everything fails loud with RuntimeError/Error (matches the repo's 'No fallbacks. Fail loud.' doctrine), and it reuses existing primitives — contentHash from verdict-cache, deepFreezeCanonicalJson,
- Better / existing approach: Mostly none — a separate GEPA-result artifact is the right call and is not covered by the existing callback-observation reader. Minor: the new file adds a third local copy of assertExactKeys (now in src/analyst/benchmark-command-artifact.ts:210 [exported], src/campaign/external-optimizer-observations.ts:381, and the new src/campaign/gepa-candidate-population.ts:341) and a third copy of assertPosit
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error
🎯 Usefulness — sound
Preserves GEPA's exact candidate graph as a digest-addressed artifact wired through the bridge, method provenance, a verified public reader, and a hash-join key consistent with callback observations.
- Integration: Fully reachable end-to-end. The Python bridge writes candidate-population-{attemptId}.json and returns its summary (gepa_bridge.py:249-288); gepaOptimizationMethod reads it via the verified reader, asserts the winner hash matches the population's best candidate, and binds the summary into provenance.gepaCandidatePopulation (gepa-optimization-method.ts:455-471, 548-550); compareOptimizationMethods
- Fit with existing patterns: Follows the codebase's established artifact-summary-in-provenance pattern exactly — same shape as observations and modelExecutions: bridge writes a digest-addressed JSON file, returns a {scope,path,sha256,bytes,...} summary, the TS method binds it into provenance, and a verified reader re-checks digest+bounds+schema on read. No competing capability exists; observations carry rejected/refused propo
- Real-world viability: Strong validation on both sides: sha256+byte-count+bounds checks, parent acyclicity (parent<index, root has single null parent), aggregate-must-equal-mean-of-selection-scores, winner hash match, discovery-count typing. The strict aggregate-consistency assertion (rel_tol 1e-9) is grounded by the pinned-GEPA-0.1.4 release test and fails loud per repo doctrine if a future version changes the reductio
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟡 assertExactKeys / assertPositiveSafeInteger now triplicated [duplication] ``
The new src/campaign/gepa-candidate-population.ts:341,353 introduces local assertExactKeys and assertPositiveSafeInteger. assertExactKeys already exists as an export at src/analyst/benchmark-command-artifact.ts:210 and as a local fn at src/campaign/external-optimizer-observations.ts:381; assertPositiveSafeInteger already exists at src/campaign/gepa-optimization-config.ts:364 and src/campaign/skillopt-optimization-config.ts:154. Consider extracting these (plus the shared read->verify-digest->veri
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
What changed
Ownership
GEPA owns search and parentage. Agent Eval validates and stores the exact result. Runtime can join graph rows to callback proposals by the canonical candidate hash. No parent is inferred. Final cases never enter the bridge.
Proof
pnpm lintpnpm typecheckpnpm verify:packagegepa==0.1.4text and component integration plus bridge tests: 29 passed, 1 source-composition skipThe general Python suite requires mutually exclusive GEPA-source, GEPA-release, SkillOpt, and DSPy environments. Its release-GEPA slice is the applicable one and is green.