Skip to content

feat(orchestrator): dispatch builds per speculation path - #503

Open
behinddwalls wants to merge 1 commit into
preetam/speculation-path-build-storefrom
preetam/speculation-build-dispatch
Open

feat(orchestrator): dispatch builds per speculation path#503
behinddwalls wants to merge 1 commit into
preetam/speculation-path-build-storefrom
preetam/speculation-build-dispatch

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

The build stage triggered one build per batch, on the batch's full dependency list. Speculation needs one build per path, on the subset of dependencies the path assumes will succeed — that subset is what lets a batch be verified before the batches ahead of it resolve.

What?

The dispatch controller now reads the head's path set and starts a build for every pending path, on that path's own base. It only starts builds — stopping them belongs to the poll loop (next commit) — and it never writes the path set, which stays the speculate run's single-writer state; what this stage knows is the build it started, recorded in per-build rows of its own.

The write order is Trigger → Build record → write-once link (entity.PathBuild) → signal; each write makes the previous one reachable, and the link is the idempotency point for redeliveries. A lost link race publishes both builds' signals and acks — the poll loop keeps the one the link names. On redeliveries only, every live linked path gets its signal re-published, closing the crash window between link and signal without forking duplicate poll chains past the queue's dedup horizon.

The build signal carries only the runner's build ID and partitions on it, so one slow build cannot block a head's other paths. Known gap: a crash between Trigger and the link orphans that build; the fix is an idempotency key on BuildRunner.Trigger, marked TODO.

Also adds submitqueue/core/publish — the registry-lookup-and-send plumbing and the message-ID deduplication rule in one place.

Test Plan

bazel test //submitqueue/orchestrator/controller/build/... //submitqueue/core/publish/... — write order, the base holding only assumed-success dependencies, redelivery republish without rebuild, lost-race double signal, halted batch starting nothing, redelivery healing for live linked paths, and a pin that no path-set write of any kind happens.

make fmt, make gazelle, make mocks

Stack

  1. docs(rfc): explain best-first speculation generation #513
  2. feat(speculation): generator contract and bestfirst impl #446
  3. feat(speculation): allocator contract and sticky impl #450
  4. feat(speculation): standard composed speculator #451
  5. feat(storage): speculation path set store #501
  6. feat(storage): path-build link store for per-path builds #502
  7. @ feat(orchestrator): dispatch builds per speculation path #503
  8. feat(orchestrator): poll builds and stop the ones nothing wants #504
  9. fix(orchestrator): mint distinct message IDs for cancel re-publishes #505
  10. feat(orchestrator): re-plan the queue from the Speculator each run #506
  11. feat(orchestrator)!: finalize batches from their speculation paths #507
  12. feat(orchestrator): compose per-queue speculators and turn speculation on #508

@behinddwalls
behinddwalls marked this pull request as ready for review August 4, 2026 01:52
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners August 4, 2026 01:52
@behinddwalls
behinddwalls force-pushed the preetam/speculation-build-dispatch branch from 9cd7353 to 7ff20d4 Compare August 4, 2026 01:58
@behinddwalls
behinddwalls force-pushed the preetam/speculation-build-dispatch branch from 7ff20d4 to 2e32d07 Compare August 4, 2026 05:11
@behinddwalls
behinddwalls force-pushed the preetam/speculation-build-dispatch branch from 2e32d07 to e37f128 Compare August 5, 2026 03:15
@behinddwalls
behinddwalls force-pushed the preetam/speculation-build-dispatch branch from e37f128 to 70285fb Compare August 5, 2026 03:26
@behinddwalls
behinddwalls force-pushed the preetam/speculation-build-dispatch branch from 70285fb to 3983b14 Compare August 5, 2026 03:38
@behinddwalls
behinddwalls force-pushed the preetam/speculation-build-dispatch branch from 3983b14 to 57dcb4f Compare August 5, 2026 03:43
@behinddwalls
behinddwalls force-pushed the preetam/speculation-build-dispatch branch from 57dcb4f to 125e288 Compare August 5, 2026 04:11
## Summary

### Why?

The build stage triggered one build per batch, on the batch's full dependency list. Speculation needs one build per *path*, on the subset of dependencies the path assumes will succeed — that subset is what lets a batch be verified before the batches ahead of it resolve.

### What?

The dispatch controller now reads the head's path set and starts a build for every pending path, on that path's own base. It **only starts builds** — stopping them belongs to the poll loop (next commit) — and it **never writes the path set**, which stays the speculate run's single-writer state; what this stage knows is the build it started, recorded in per-build rows of its own.

The write order is Trigger → Build record → write-once link (`entity.PathBuild`) → signal; each write makes the previous one reachable, and the link is the idempotency point for redeliveries. A lost link race publishes both builds' signals and acks — the poll loop keeps the one the link names. On redeliveries only, every live linked path gets its signal re-published, closing the crash window between link and signal without forking duplicate poll chains past the queue's dedup horizon.

The build signal carries only the runner's build ID and partitions on it, so one slow build cannot block a head's other paths. Known gap: a crash between `Trigger` and the link orphans that build; the fix is an idempotency key on `BuildRunner.Trigger`, marked TODO.

Also adds `submitqueue/core/publish` — the registry-lookup-and-send plumbing and the message-ID deduplication rule in one place.

## Test Plan

✅ `bazel test //submitqueue/orchestrator/controller/build/... //submitqueue/core/publish/...` — write order, the base holding only assumed-success dependencies, redelivery republish without rebuild, lost-race double signal, halted batch starting nothing, redelivery healing for live linked paths, and a pin that no path-set write of any kind happens.

✅ `make fmt`, `make gazelle`, `make mocks`
@behinddwalls
behinddwalls force-pushed the preetam/speculation-build-dispatch branch from 125e288 to f41ca0f Compare August 5, 2026 04:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant