CI: render readable integration test output - #7770
Open
CharlieTLe wants to merge 1 commit into
Open
Conversation
An integration shard's log was a single undifferentiated wall of text, and with 12 build tags x 2 arches there are 24 of them to scroll through when a run goes red. Container logs make it worse: integration/e2e pipes every container's stdout straight to the process's stdout, so they interleave with the -test.v framing instead of going through t.Log. Pipe the test binary through test2json and render the event stream with a new in-repo tool, tools/gha-testlog. Per shard it now emits a collapsed ::group:: per top-level test (holding its subtests' and its containers' output), an ungrouped PASS/FAIL/SKIP result line after each group so the collapsed log reads as an index, a FAILURES section repeating each failure's tail, ::error:: annotations so failures reach the Annotations panel and the PR diff, and a job summary. A new integration-summary job then renders one table across all shards. Both helpers are built in build-integration-tests and ride along in the existing tarball, so the integration job still needs no checkout and no Go toolchain. Failing shards upload the raw test2json stream: containers run with --rm and their shared directory is deleted on close, so that capture is the only surviving copy of their logs. Also quiets the docker load/pull and tar output the shard log was padded with. Signed-off-by: Charlie Le <charlie_le@apple.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Makes a failing integration shard readable in CI.
Today an integration shard's log is a single undifferentiated wall of text, and with 12 build tags × 2 arches there are 24 of them to open one at a time when a run goes red. Container logs make it worse:
integration/e2epipes every container's stdout/stderr straight to the process's stdout (LinePrefixLogger→ the package-global logger), so they interleave with the-test.vframing instead of going throught.Log. There is no output processing, no artifacts, no annotations and no job summary.This pipes the test binary through
test2jsonand renders the event stream with a new in-repo tool,tools/gha-testlog. Per shard the log now has:::group::per top-level test, holding its subtests' output and its containers';PASS|FAIL|SKIP <Test> (12.34s)line after each group, so the collapsed log reads as a scannable index of results;===== FAILURES =====section repeating each failing test's tail;::error::annotation per failure, so failures also land in the run's Annotations panel and on the PR diff;<details>per failure.A new
integration-summaryjob then renders one table across all shards, so a red run can be triaged without opening any shard log. Failing shards upload their rawtest2jsonstream as an artifact (7 day retention) — containers run with--rmandScenario.Close()deletessharedDir, so the captured stdout is the only surviving copy of their logs.Both helpers are built in the existing
build-integration-testsjob from the same Go toolchain as the test binary (the^Vframing markers are a private contract betweentestingandtest2json) and ride along in the existing tarball, so theintegrationjob still needs no checkout and no Go toolchain.Notes on the approach:
-test.v=test2jsonrather than plain-test.v: that is what makestestingemit the framing markerstest2jsonneeds to attribute container output to the test that produced it.integration/e2e. That package is consumed by other projects; and intest2jsonmode attribution comes from the framing markers, so an exported logger setter or indented logger output would buy nothing.gotestsumwas considered and rejected: it has no GitHub Actions output format, so the grouping and annotations would have to be written either way; its--junitfilewould have no consumer underpermissions: read-all; and installing it means an unpinned network fetch on both arches or a new vendored dependency tree.name:was added to theintegrationmatrix job, to avoid renaming 24 status checks. Shard identity is carried in the annotation titles and the summary table instead.Also quiets the
docker load/docker pullprogress output and dropsvfrom fourtarinvocations — that noise is a large fraction of what currently pads a shard log.No
CHANGELOG.mdentry: this is CI-only and not user-facing.Which issue(s) this PR fixes:
N/A
Verification
tools/gha-testloghas table-driven unit tests over rawtest2jsonJSONL fixtures, covering the whole state machine: grouping, subtests, package-level output, a line split across two events (test2jsonsplits output at 1 KiB), a truncated final line, a stream that ends mid-test,::-injection from container output, the annotation cap, the-tailring buffer, and source-location/message extraction. These run in the existingtestjob viamake test.Beyond that it was exercised end to end against real integration tests, both locally and on a scratch branch of my fork, in the passing, failing-assertion and
-test.timeoutcases. Things worth calling out that those runs settled:##[group]/##[endgroup]on a 19-testintegration_rulershard).foo_test.go:NN:in a test's output is usually at.Logf(there are 30 such calls underintegration/), and testify'sError Trace:lists frames outermost-last, so a naive "first reference" or "last reference anywhere" rule blames the wrong line.integration-summaryjob downloads and consumes it; a passing shard uploads nothing, which is what makes "no report" mean "this shard passed".-log.level=warn.Checklist
.github/workflows-doc.mdCHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flagsAI usage disclosure (per GENAI_POLICY.md): the bulk of this contribution was written with Claude Code — the
tools/gha-testlogpackage and its tests, the summary script, and the workflow changes. I have reviewed it and validated it end to end on CI runs in my fork as described above.