Layered EdgeZero deploy actions + Fastly staging lifecycle (design + impl, supersedes #303)

[aram356]

Summary

Layered, adapter-independent GitHub Actions for deploying an EdgeZero app to Fastly Compute — design + implementation, complete — superseding the Fastly-only monolith in #303. The EdgeZero CLI is the boundary: the actions compile the app's own CLI, scope credentials, and invoke it; they never reproduce provider build/deploy logic in YAML, so adding another provider later is a new thin wrapper, not an engine rewrite. Based off main.

Actions

• build-app-cli — compiles the CLI package the application provides (a crate in the app's own workspace) from the app checkout, with an isolated CARGO_TARGET_DIR + --locked; publishes a self-describing tar (app-cli-meta.json) so downstream steps need no re-pass. Credential-free by design.
• deploy-core — adapter-independent shared engine scripts sourced by the wrappers (not a standalone action). Provider credentials/flags flow only through provider-env (deploy-step-scoped), provider-env-clear, deploy-flags, and deploy-args.
• deploy-fastly — minimal wrapper; installs the pinned, checksum-verified Fastly CLI; stage: true produces a staged draft; outputs fastly-version, previous-version (the rollback target captured pre-deploy), mutation-attempted, and the installed provider-CLI version.
• healthcheck-fastly / rollback-fastly — Fastly staging lifecycle (parity with stackpop/trusted-server-actions), driven by the app CLI over the Fastly API.
• config-push-fastly — pushes the app's typed config to a Fastly config store (the production key, or the isolated _staging twin).

CLI

edgezero-adapter-fastly and the scaffolded downstream template gain the lifecycle command surface the actions drive: build, deploy (--staging, --service-id), active-version, healthcheck, rollback, and typed config push / validate / diff. --staging is one consistent verb across the lifecycle; the legacy --stage is rejected (never aliased) and cannot slip through -- passthrough into a production deploy.

Security & isolation

• Provider credentials reach only the deploy/lifecycle steps that need them; every other step blanks the shipped provider aliases and BASH_ENV/ENV, and the credential-free CLI build re-execs with the GitHub file-command channels stripped.
• target/ caching is credential-free: the cache is seeded and saved from the build-mode: always build before the token-bearing deploy, never after — so a build script cannot persist a secret into the cache. With build-mode: never it is a documented no-op.
• A repository-wide pin gate (check-action-pins.sh) parses every workflow and action structurally with a pinned, checksum-verified yq (installed under RUNNER_TEMP), rejecting any uses: on a mutable branch/floating ref while accepting released version tags or full SHAs. actionlint (all workflows, with its ShellCheck integration) and zizmor back it up.
• A mutation-attempted reconcile signal on the mutating actions, fail-closed input validation, a dirty-source guard, and committed-source-only deploys.

End-to-end smoke coverage

The Deploy actions workflow drives the real wrappers against a fake fastly / curl served through the installer's genuine download + checksum + extract path:

• static-checks — actionlint, the structural pin gate, zizmor, ShellCheck, the Bash contract suite, and the docs build.
• composite-smoke — production deploy, the credential boundary, and rollback threading.
• handoff-build / handoff-deploy — cross-job artifact handoff by literal name.
• cache-smoke — cache populate + restore-hit, plus a negative (build-mode: never) no-op case.
• recovery-smoke — an induced lost-version deploy failure, recovered via active-version + rollback-fastly threading previous-version.
• config-push-smoke — typed staging and production config push.
• lifecycle-smoke — staged deploy + healthcheck.

Docs

• docs/specs/edgezero-deploy-github-action.md — normative spec.
• docs/specs/edgezero-deploy-action-implementation-plan.md — plan (+ Add Fastly deploy action with config push #303 port map).
• docs/specs/edgezero-deploy-adoption-guide.md — adoption guide (any app repo).
• docs/guide/deploy-github-actions.md — practical how-to; docs/guide/cli-reference.md — CLI surface.

Notes

Supersedes #303 (and the earlier stacked docs PR #315); #303's unrelated changes (KV timing logs, dep bumps) are not carried here. All CI is green: Run Tests, Run Format, CodeQL, Fastly installer check, and the Deploy-actions smoke suite.

[prk-Jr]

PR Review

Summary

15.5k lines across 73 files, and the layering thesis holds up: the app CLI really is the boundary, the wrappers really are thin, and adding a second provider really would be a new wrapper rather than an engine rewrite. Read every changed file; ran all five CI gates locally (all pass). The Bash contract suite, the golden public-surface test, and the credential-free cache ordering are better than most repos ship, and the code is unusually honest about its own residual risks.

The blocking findings share one root cause: build-app-cli establishes a real untrusted-build boundary, and deploy-fastly's seed build does not inherit it. Everything the build-app-cli docstrings warn about — a build.rs writing $GITHUB_PATH, persisting state a later privileged step consumes — applies to deploy-fastly's build-mode: always step, which runs in the same job that then receives the Fastly token. Two more findings (the .curlrc channel, the mutable major-tag pins) are the same shape: something an earlier step can leave behind that a token-bearing step later trusts.

😃 Praise

• assert_safe_tarball (deploy-core/scripts/common.sh:158) identifies and fixes a genuine pipefail + SIGPIPE fail-open where the symlink check would have been false precisely when a symlink was found. Documented in place. That class of bug is normally found in production, not review.
• The fake is delivered through the real trust path. make-fake-fastly-env.sh:247-267 packages the fake fastly as a tar.gz and repoints the checked-out versions.json with a matching SHA-256, so install-fastly.sh verifies and extracts it through its genuine download → checksum → extract path instead of adopting a planted binary — and run.sh:757-788 guards against that override ever being committed. The part that can't be faked gets its own path-filtered job against the real release. That split is rare and right.
• The golden public-surface test (run.sh:1631-1755) pins the exact input/output names, required flags, and defaults per action, which is why the docs findings below are docs bugs rather than implementation drift.
• EDGEZERO__ as a one-rule credential boundary (run-app-cli.sh:144-176): a single compgen -e prefix scrub instead of a list that rots, with EDGEZERO_MANIFEST deliberately outside it — and a docstring that states plainly what the boundary is not ("NOT a process-image boundary — on Linux this shell's original environment stays readable via /proc/<ppid>/environ"). Documented residual risk beats an overclaimed guarantee, here and at cli.rs:1999-2002.
• Fail-closed version parsing throughout. parse_canonical_version_line rejecting version=15.2.0 rather than reading 15, last_version_after requiring a terminator, and resolve_active_version refusing to read a garbled or multi-active response as "no active version." Each of those is a wrong-version-into-rollback bug that isn't there.

Findings

Blocking

• 🔧 Untrusted seed build shares a job with the token — deploy-core/scripts/run-app-cli.sh:194. build mode runs the app's and every dependency's build.rs with GITHUB_ENV/GITHUB_PATH/GITHUB_OUTPUT live; build-app-cli strips exactly these via env -u … exec and run_untrusted, and this path does not. Reaches the Deploy and Capture rollback target steps' jq/tee/grep calls. Gated on build-mode: always — which is also the only mode where the advertised target/ caching works.
• 🔧 Pin gate accepts the mutable refs it claims to reject — deploy-core/tests/check-action-pins.sh:48. (\.[0-9]+)* admits bare major tags; the branch head then moved every third-party ref onto them, including actions-rust-lang/setup-rust-toolchain@v1 inside the token-bearing job. Three separate comments (this file, zizmor.yml:11-13, run.sh:1758) assert immutability that isn't there.
• 🔧 curl reads ~/.curlrc into the token-bearing config document — crates/edgezero-adapter-fastly/src/cli.rs:2200. No -q, so a build.rs in the same job can plant a proxy = directive and receive the Fastly-Key header. The env-var variant of this attack is closed by the re-exec; the filesystem variant is not.
• 🔧 BASH_ENV/ENV blanked on 4 of 10 bash steps — deploy-fastly/action.yml:109 and the same gap in all three other wrappers. The unprotected steps are the ones producing deploy-flags-file and provider-env-clear-file, and the ones installing the two binaries the token steps execute. build-app-cli gets this right on all four of its steps, so it's an evenness problem, not a missing idea. run.sh:443-479 tests the invariant against a hardcoded step whitelist, which hides the gap.
• 🔧 deploy.sh takes "last wins" where its sibling requires "exactly one" — deploy-fastly/scripts/deploy.sh:51. Not reachable through EdgeZero's own CLI (its canonical line is emitted last); reachable with a non-conforming app CLI, which is the exact producer capture-previous.sh:78-90 was hardened against.
• 🔧 Tool checksums fetched from the asset's own origin — scripts/install-yq.sh:60, scripts/install-actionlint.sh:47-52; cargo install zizmor has no pin at all. yq is the pin gate, and check-action-pins.sh:81-83 reports success on an empty ref list — a substituted yq makes the gate green while parsing nothing. versions.json:6 already demonstrates the in-repo-digest pattern.
• 🔧 Production config push has no committed-source guard — config-push-fastly/scripts/config-push.sh:65. No Resolve project, so no assert_committed_source and GITHUB_WORKSPACE as the confinement root — the boundary resolve-project.sh:137-141 explicitly rejects for the separate-repo layout. An uncommitted config can be pushed to the store the live service reads, with no source-revision to reconcile.

Non-blocking

• 🤔 Staging IP unvalidated; IPv6 misparses — cli.rs:2088. Fails closed, but the outcome is an automatic rollback of a healthy staged deploy, reported as "unhealthy." One parse::<IpAddr>() fixes both this and the recursive-descent looseness in find_staging_ip.
• 🤔 zizmor's hardcoded file list — deploy-action.yml:106. Skips codeql.yml (security-events: write) and deploy-docs.yml (pages: write + id-token: write), and no tool audits composite action.yml for template injection. A new wrapper with ${{ github.event.* }} in a run: block passes every gate here.
• 🤔 docker:// is wholly exempt from the pin gate — check-action-pins.sh:63-65. docker://ghcr.io/x/y:latest passes a gate whose docstring promises to reject floating refs. Also, only .github/workflows (maxdepth 1) and .github/actions/**/action.y*ml are scanned, so a valid local action at e.g. tools/deploy/action.yml is never parsed.
• 🤔 The checksum comparison is never negatively tested — install-fastly.sh:74. The smoke computes the expected SHA from the archive it just built (make-fake-fastly-env.sh:255), so inverting or deleting the comparison leaves every job green. A run.sh case that corrupts the archive and asserts the mismatch message would close it. (Relatedly, fastly-installer-check.yml:3-4 now contradicts make-fake-fastly-env.sh:17-23 about whether the real checksum path is exercised elsewhere.)
• 🤔 Skipped tests are counted as passes — run.sh:289, :1952, :2019 call pass "…(skipped: non-Linux runner)", and :958/:1268/:1333/:1499 do git init -q 2>/dev/null || return 0, so a git init failure silently deletes four whole suites with no diagnostic. On a macOS dev box the run is green with several suites never executed. A separate skip() counter reported apart from Passed: would make that visible.
• 🤔 test_recovery_version_parse covers no production code — run.sh:2211-2262 tests a string literal defined inside the test, and the shipped helper recovery-active-version.sh:22-25 implements a third, incompatible parse that rejects the empty version= the test asserts must succeed. Extracting one parse and having both the doc snippet and the helper use it would make the test mean something.
• 🤔 Partial download poisons the tool root — install-fastly.sh:71-74. curl --output truncates before the transfer completes, so a reset mid-download leaves a short archive; the documented idempotency then skips the refetch on retry and fails the checksum forever, reading as a supply-chain alarm rather than a network blip. Download to a scratch name and mv after verification.
• 🤔 Lifecycle log lives outside the workspace cleanup deletes — deploy-core/scripts/common.sh:218-222 puts it in RUNNER_TEMP, so the in-process EXIT trap is the only thing that removes it — and no trap survives the SIGKILL after a cancellation grace period. Mode 600 doesn't help; every step in a job is the same uid. EDGEZERO__ACTION__WORKSPACE is already exported to those steps.
• 🤔 Inline config written to a predictable path, with a trap that breaks on quotes — config-push-fastly/scripts/config-push.sh:127-134. > follows symlinks and doesn't create exclusively, and $$ is small and reusable; on a self-hosted runner RUNNER_TEMP persists across jobs. Separately, the immediate-expansion trap (trap "… '$inline_file'") fails to parse if the path contains a quote, silently losing the cleanup. Single-quote the trap and mktemp the file — cleanup_sensitive_temps already tolerates an empty path.
• 🤔 Resolved Rust toolchain is never shape-checked — resolve-project.sh:45-52 / build-app-cli.sh:60-68. It reaches rustup toolchain install, cargo +…, a third-party action input, and the cache key. It's the only free-form repo-sourced string in the PR with no regex gate; a checked-in rust-toolchain of --profile complete is parsed as an option. Worst case is a confusing failure, and the guard is one line.
• 🤔 Allowlist admits one unvalidated token per permitted flag — validate-inputs.sh:76 assumes every allowlisted flag takes a value. Not exploitable with today's single-entry allowlist (--comment), but the moment a boolean flag is added, --flag <anything> becomes an unchecked path into the provider argv, silently.
• 🤔 local argv=("$(resolve_app_cli)" …) masks the :? guard — healthcheck-fastly/scripts/healthcheck.sh:74-84. local x=$(false) exits 0, so the diagnostic prints and execution continues with argv[0]="". Fails closed at 127, but it's the errexit-masking pattern the rest of the PR is careful to avoid, and this is the only lifecycle script omitting require_cmd "$cli_bin".
• 🤔 cache: true is a silent no-op under the documented default — resolve-project.sh:98-104 maps auto → never for Fastly, and both cache steps require always. validate-inputs.sh:103-106 accepts cache: true regardless with no warning, so a user who sets only cache: true gets nothing and no signal.
• 🤔 cache_key omits build-args — resolve-project.sh:218. Two invocations at the same revision with different --features share one entry, and the first writer wins for the key's life. Cargo's fingerprinting degrades this to extra rebuilds rather than a wrong artifact, but it defeats the exact-key contract the surrounding comment establishes for workspace identity.
• 🤔 provider-env-clear: '' degrades to [] instead of failing closed — build-app-cli/action.yml:24-37 documents "the build fails closed"; an explicitly-empty input doesn't take the default, and provider_env_clear_names '[]' validates cleanly with zero names. The static layer still covers all 25 shipped aliases, so only a caller's own alias leaks.
• 🤔 Two smoke assertions are outcome-only — deploy-action.yml:619-621, :670-672 assert outcome == "failure" under whole-composite continue-on-error, so they'd pass if prepare-workspace or the installer checksum failed instead. assert-stale-rollback-refused.sh:46 already demonstrates the log-delta pattern that fixes it.
• 🤔 Fake fastly never records the credential in scope, and always exits 0. The fake curl does record PROBE-TOKEN= and gets a real assertion; the fake fastly doesn't, so the staged-deploy and config-push jobs never verify which token arrived, and curl_config_capture's non-zero-exit path (cli.rs:2223-2229) is unreachable in test. Also config-store-entry describe always returns the "absent" shape, so an update-an-existing-key push is untested.
• 🤔 Assertions on source text rather than behavior — run.sh:494-495 greps an env key name in YAML under the description "cleanup removes the workspace root"; :2122-2126 pins the spelling of a loop, so rewriting it as an array fails with correct behavior; :917-927 is a bare grep … || fail with no matching pass, so success is invisible in the count.
• 🔧 Docs: --yes is mandatory without a TTY and isn't documented — config.rs:854 errors, cli-reference.md:257 only mentions prompting, and both the adoption guide (:232) and deploy guide (:742) invite users to run config push directly. A copy-pasted CI invocation hard-fails. The action compensates silently (config-push.sh:148 appends --yes --no-diff) and no input table says so.
• 🔧 Docs: cli-reference.md has no section for healthcheck, rollback, or active-version — the exact three commands deploy-github-actions.md:38-40 requires a hand-written app CLI to expose. Their required flags and defaults appear nowhere; the env-var table (:418-424) also omits FASTLY_API_TOKEN and FASTLY_SERVICE_ID, without which every lifecycle command hard-fails.
• ⛏ app-cli-bin's default is documented wrong in four tables — deploy-github-actions.md:306, :464, :484, :509 say "artifact's name"; the real default is the app-cli-bin field of app-cli-meta.json (download-app-cli.sh:69), i.e. the built binary name. The spec has it right.
• ⛏ The separate-repo example won't run as pasted — deploy-github-actions.md:96-126 uses token: ${{ steps.app-token.outputs.token }} with no id: app-token step in the snippet, and ref: ${{ inputs.ref }} with no on: block. The adoption guide states the assumption in prose; the snippet doesn't. Worth one line noting @<ref> must be replaced too, since no example in either guide is runnable verbatim.
• ⛏ cli-reference.md:236/:298 attribute the full config push/config diff flag surfaces to the bundled edgezero binary, whose subcommands are hidden trailing-var sinks that exit 2. The typed split is explained, but only after the flags are attributed. Also :315/:327 claim exit 0 without --exit-code, while config.rs:408 returns 2 for Unsupported regardless.
• ♻️ Three spellings of one verb across the action surface — stage: vs deploy-to: vs --staging. See the inline note on deploy-fastly/action.yml:42; these are the names downstream repos pin, so the window to change them closes at merge.
• ♻️ deploy-fastly/scripts/common.sh is a third diverged copy of the helper set and is never sourced by run.sh; 5 of its 13 functions have no consumer anywhere, including the path-confinement helper is_under:81. No test would catch further drift.
• 🌱 Zero-coverage production code with real logic: write-summary.sh (runs if: always() in five jobs; its "never emits credentials" contract is unenforced and nothing reads GITHUB_STEP_SUMMARY), verify-installed-version.sh:18, install-fastly.sh:33 provider_bin_dir (its entire rationale — an app CLI legitimately named fastly — can't fire because the fixture CLI is fixture-app-cli), and resolve-project.sh:164-169's symlink-escape guard (whose twin in config-push.sh is tested).
• 📝 is_healthy_status counts 3xx as healthy (cli.rs:1851). Defensible and documented, but for a gate that triggers an automatic rollback, a staged version answering 301 to an error page passes. Worth a thought about 2xx-only or making it configurable.
• 📝 tee_stream swallows read errors (adapter.rs:191): Ok(0) | Err(_) => break means one non-UTF-8 byte in a child's output silently truncates both the captured text and the operator-visible echo. Logging the error before breaking would make that diagnosable.
• 📝 actions/checkout is pinned at both @v6 and @v7 in the same repo. $GITHUB_ACTION_PATH/... is unquoted in every run: (harmless on hosted runners; word-splits under a workspace path with spaces). build-app-cli.sh:226 re-declares local workspace_real, changing the variable's meaning mid-function.

📌 Out of Scope

• Service-scoped serialization for deploy/rollback. ensure_rollback_from_is_active narrows the clobber window and cli.rs:1999-2002 says plainly that it can't close it. The fix is a concurrency group per service in the calling workflow, not in this PR — but it belongs in the guide's reconcile section, since a rollback that lands after a newer deploy is the failure this PR can't prevent alone.
• post: cleanup. Composite actions have none, so cancellation/SIGKILL can leave the chmod-600 lifecycle log behind. Harmless on hosted runners, a cross-job leak on self-hosted — and validate-inputs.sh:37-42 doesn't exclude self-hosted. Tracking a documented "hosted runners only" constraint (or a reaper) is probably its own issue.
• The production deploy path in composite-smoke/handoff-deploy runs a fake shell deploy command (make-smoke-fixture.sh:184-186), so no real fastly compute deploy/activate is ever exercised. The staged path does bypass manifest commands and test the real adapter, so this is a known and reasonable limit — worth naming in the spec's testing section rather than fixing here.

CI Status

Run locally against 93a9b90:

• cargo fmt --all -- --check — PASS
• cargo clippy --workspace --all-targets --all-features -- -D warnings — PASS
• cargo test --workspace --all-targets — PASS (1238 passed, 0 failed)
• cargo check --workspace --all-targets --features "fastly cloudflare spin" — PASS
• cargo check -p edgezero-adapter-spin --target wasm32-wasip2 --features spin — PASS

GitHub checks are all green (23/23), including the full Deploy-actions smoke suite. No CI failures — every finding above is a design/hardening issue the gates don't cover.

[ChristianPavilonis]

Summary

Reviewed the Trusted Server/EdgeZero deployment, staging, healthcheck, rollback, security, and recovery paths. I found four actionable issues; details are inline. Current CI is green. Approving under the review rubric, while recommending that the P1 findings be addressed before merge.