Skip to content

CI: render readable integration test output - #7770

Open
CharlieTLe wants to merge 1 commit into
cortexproject:masterfrom
CharlieTLe:ci-readable-integration-logs
Open

CI: render readable integration test output#7770
CharlieTLe wants to merge 1 commit into
cortexproject:masterfrom
CharlieTLe:ci-readable-integration-logs

Conversation

@CharlieTLe

Copy link
Copy Markdown
Member

What this PR does:

Makes a failing integration shard readable in CI.

Today an integration shard's log is a single undifferentiated wall of text, and with 12 build tags × 2 arches there are 24 of them to open one at a time when a run goes red. Container logs make it worse: integration/e2e pipes every container's stdout/stderr straight to the process's stdout (LinePrefixLogger → the package-global logger), so they interleave with the -test.v framing instead of going through t.Log. There is no output processing, no artifacts, no annotations and no job summary.

This pipes the test binary through test2json and renders the event stream with a new in-repo tool, tools/gha-testlog. Per shard the log now has:

  • a collapsed ::group:: per top-level test, holding its subtests' output and its containers';
  • an ungrouped PASS|FAIL|SKIP <Test> (12.34s) line after each group, so the collapsed log reads as a scannable index of results;
  • a ===== FAILURES ===== section repeating each failing test's tail;
  • one ::error:: annotation per failure, so failures also land in the run's Annotations panel and on the PR diff;
  • a job summary with counts and a <details> per failure.

A new integration-summary job then renders one table across all shards, so a red run can be triaged without opening any shard log. Failing shards upload their raw test2json stream as an artifact (7 day retention) — containers run with --rm and Scenario.Close() deletes sharedDir, so the captured stdout is the only surviving copy of their logs.

Both helpers are built in the existing build-integration-tests job from the same Go toolchain as the test binary (the ^V framing markers are a private contract between testing and test2json) and ride along in the existing tarball, so the integration job still needs no checkout and no Go toolchain.

Notes on the approach:

  • -test.v=test2json rather than plain -test.v: that is what makes testing emit the framing markers test2json needs to attribute container output to the test that produced it.
  • No changes to integration/e2e. That package is consumed by other projects; and in test2json mode attribution comes from the framing markers, so an exported logger setter or indented logger output would buy nothing.
  • gotestsum was considered and rejected: it has no GitHub Actions output format, so the grouping and annotations would have to be written either way; its --junitfile would have no consumer under permissions: read-all; and installing it means an unpinned network fetch on both arches or a new vendored dependency tree.
  • No name: was added to the integration matrix job, to avoid renaming 24 status checks. Shard identity is carried in the annotation titles and the summary table instead.

Also quiets the docker load/docker pull progress output and drops v from four tar invocations — that noise is a large fraction of what currently pads a shard log.

No CHANGELOG.md entry: this is CI-only and not user-facing.

Which issue(s) this PR fixes:

N/A

Verification

tools/gha-testlog has table-driven unit tests over raw test2json JSONL fixtures, covering the whole state machine: grouping, subtests, package-level output, a line split across two events (test2json splits output at 1 KiB), a truncated final line, a stream that ends mid-test, ::-injection from container output, the annotation cap, the -tail ring buffer, and source-location/message extraction. These run in the existing test job via make test.

Beyond that it was exercised end to end against real integration tests, both locally and on a scratch branch of my fork, in the passing, failing-assertion and -test.timeout cases. Things worth calling out that those runs settled:

  • Group markers come out balanced in the real web log viewer (31 ##[group]/##[endgroup] on a 19-test integration_ruler shard).
  • Annotations anchor on the failing subtest and on the assertion's own line. Getting this right needed care: the first foo_test.go:NN: in a test's output is usually a t.Logf (there are 30 such calls under integration/), and testify's Error Trace: lists frames outermost-last, so a naive "first reference" or "last reference anywhere" rule blames the wrong line.
  • A failing shard's artifact uploads and the integration-summary job downloads and consumes it; a passing shard uploads nothing, which is what makes "no report" mean "this shard passed".
  • The raw stream is small — tens of KB per shard, not the tens of MB one might assume, because services run at -log.level=warn.

Checklist

  • Tests updated
  • Documentation added — .github/workflows-doc.md
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

AI usage disclosure (per GENAI_POLICY.md): the bulk of this contribution was written with Claude Code — the tools/gha-testlog package and its tests, the summary script, and the workflow changes. I have reviewed it and validated it end to end on CI runs in my fork as described above.

An integration shard's log was a single undifferentiated wall of text, and with
12 build tags x 2 arches there are 24 of them to scroll through when a run goes
red. Container logs make it worse: integration/e2e pipes every container's
stdout straight to the process's stdout, so they interleave with the -test.v
framing instead of going through t.Log.

Pipe the test binary through test2json and render the event stream with a new
in-repo tool, tools/gha-testlog. Per shard it now emits a collapsed ::group::
per top-level test (holding its subtests' and its containers' output), an
ungrouped PASS/FAIL/SKIP result line after each group so the collapsed log reads
as an index, a FAILURES section repeating each failure's tail, ::error::
annotations so failures reach the Annotations panel and the PR diff, and a job
summary. A new integration-summary job then renders one table across all shards.

Both helpers are built in build-integration-tests and ride along in the existing
tarball, so the integration job still needs no checkout and no Go toolchain.
Failing shards upload the raw test2json stream: containers run with --rm and
their shared directory is deleted on close, so that capture is the only
surviving copy of their logs.

Also quiets the docker load/pull and tar output the shard log was padded with.

Signed-off-by: Charlie Le <charlie_le@apple.com>
@dosubot dosubot Bot added ci/cd type/chore Something that needs to be done; not a bug or a feature labels Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd size/XXL type/chore Something that needs to be done; not a bug or a feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant