Make the orchestrator a real worker so it stops exiting on boot - #235
Merged
Conversation
The Railway container reported EXITED on every deploy. The build was fine; the process genuinely ended. `startOrchestrator()` registered two EventEmitter listeners and returned, and `bus.on(...)` does not hold the event loop open — so it printed its banner and exited. The same EventEmitter was also why work went nowhere. The web app enqueues a run when a webhook arrives; the orchestrator listened on its own in-process bus in a different container and never heard it. The previous commit fixed the producer to write to the durable queue but left the consumer on the bus, so runs were still dropped. Both queues are now database-backed: workflow_jobs a run needs its next step decided workflow_steps a node needs executing The orchestrator polls both with the existing Worker, which keeps the process alive, lets several instances share the work, and survives a restart. Steps get a 120s visibility timeout because they make outbound HTTP calls with their own retries, and a 30s lease could lapse mid-flight and cause a duplicate execution. Also fixed while rewiring: - A terminate node reported next: null, which replay could not distinguish from "not started" — the run restarted from its first node forever. Replay now tracks termination separately. - Completing a run is idempotent; a redelivered job appended a second run_completed event and rewrote finished_at. - An unknown node type threw instead of silently succeeding, which had been letting a run complete as though an unexecutable node had worked. - The process exits non-zero with a clear message when TURSO_DATABASE_URL is missing or still points at Postgres, rather than crash-looping on the first query. - SIGTERM/SIGINT drain in-flight jobs before exiting, so a redeploy does not strand leases until they time out. Verified end to end against a local libSQL file: the process stays up, drains a queued run through step_started/step_succeeded to run_completed, leaves the queue empty, and exits cleanly on SIGTERM. Tests: 380 passing (122 vitest incl. 12 new orchestrator tests, 258 node:test). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ThreatCrush Security Scan17 finding(s) HIGH/CRITICAL: 6 | MEDIUM: 10 | LOW: 1
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
meshhookRailway service reports EXITED on every deploy. The build is fine — the process genuinely ends.Why
startOrchestrator()registered two EventEmitter listeners and returned.bus.on(...)does not hold the Node event loop open, so the process printed its banner and exited immediately.The same EventEmitter was also why work went nowhere. The web app enqueues a run when a webhook arrives; the orchestrator listened on its own in-process bus in a different container and never heard it. #234 fixed the producer to write to the durable queue but left the consumer on the bus, so runs were still silently dropped.
What changed
Both queues are now database-backed:
workflow_jobsworkflow_stepsThe orchestrator polls both with the existing
Worker. That keeps the process alive, lets several instances share the work, and means queued work survives a restart.Steps get a 120s visibility timeout rather than the default 30s — a step makes an outbound HTTP call with its own retries, and a shorter lease could lapse while the work is legitimately still in flight, causing a duplicate execution.
Bugs found while rewiring
next: null, which replay could not distinguish from "not started", so it looped back to the first node. Replay now tracks termination separately.run_completedevent and rewrotefinished_at.TURSO_DATABASE_URLis unset or still points at Postgres.Verification
End to end against a local libSQL file:
380 tests passing (122 vitest including 12 new orchestrator tests, 258 node:test). Web app still builds.
Still needed to actually run in production
The service has no application environment variables set — no
TURSO_DATABASE_URL,TURSO_AUTH_TOKEN, orSECRETS_ENCRYPTION_KEY. With this change it will exit(1) with a clear message instead of crash-looping, but it still needs a database before it can do any work.🤖 Generated with Claude Code