From 46159e00202fd36f8188da440507eac02199e64e Mon Sep 17 00:00:00 2001 From: Charlie Le Date: Sat, 15 Aug 2026 11:29:44 -0700 Subject: [PATCH] CI: render readable integration test output An integration shard's log was a single undifferentiated wall of text, and with 12 build tags x 2 arches there are 24 of them to scroll through when a run goes red. Container logs make it worse: integration/e2e pipes every container's stdout straight to the process's stdout, so they interleave with the -test.v framing instead of going through t.Log. Pipe the test binary through test2json and render the event stream with a new in-repo tool, tools/gha-testlog. Per shard it now emits a collapsed ::group:: per top-level test (holding its subtests' and its containers' output), an ungrouped PASS/FAIL/SKIP result line after each group so the collapsed log reads as an index, a FAILURES section repeating each failure's tail, ::error:: annotations so failures reach the Annotations panel and the PR diff, and a job summary. A new integration-summary job then renders one table across all shards. Both helpers are built in build-integration-tests and ride along in the existing tarball, so the integration job still needs no checkout and no Go toolchain. Failing shards upload the raw test2json stream: containers run with --rm and their shared directory is deleted on close, so that capture is the only surviving copy of their logs. Also quiets the docker load/pull and tar output the shard log was padded with. Signed-off-by: Charlie Le --- .github/workflows-doc.md | 27 + .../scripts/summarize-integration-tests.sh | 105 ++++ .github/workflows/test-build-deploy.yml | 132 ++++- tools/gha-testlog/main.go | 503 ++++++++++++++++++ tools/gha-testlog/main_test.go | 479 +++++++++++++++++ tools/gha-testlog/summary.go | 168 ++++++ 6 files changed, 1390 insertions(+), 24 deletions(-) create mode 100755 .github/workflows/scripts/summarize-integration-tests.sh create mode 100644 tools/gha-testlog/main.go create mode 100644 tools/gha-testlog/main_test.go create mode 100644 tools/gha-testlog/summary.go diff --git a/.github/workflows-doc.md b/.github/workflows-doc.md index 2152edcd27e..58d15a829f2 100644 --- a/.github/workflows-doc.md +++ b/.github/workflows-doc.md @@ -18,6 +18,7 @@ test-build-deploy.yml specifies a workflow that runs all Cortex continuous integ | lint | Runs linting and ensures vendor directory, protos and generated documentation are consistent. | CI | | test | Runs units tests on Cassandra testing framework. | CI | | integration | Runs integration tests after upgrading golang, pulling necessary docker images and downloading necessary module dependencies. | CI | +| integration-summary | Renders one cross-shard summary of the integration matrix into the job summary, so a red run can be triaged without opening every shard's log. | CI | | Security/CodeQL | CodeQL is a semantic code analysis engine used for automating security checks. | CI | | build | Builds and saves an up-to-date Cortex image and website. | CI | | deploy_website | Deploys the latest version of Cortex website to gh-pages branch. Triggered within workflow. | CD | @@ -32,6 +33,30 @@ Internal dependencies between jobs illustrated below. Jobs run concurrently wher ### Key Details +**Integration Test Output** + +The `integration` matrix runs one shard per build tag per architecture, so a failure could +otherwise mean scrolling an undifferentiated wall of text in one of two dozen jobs. Instead the +test binary runs under `bin/test2json` and its event stream is rendered by `bin/gha-testlog` +(built from [`tools/gha-testlog`](../tools/gha-testlog) and shipped in the +`integration-tests-` artifact, so the job still needs no checkout and no Go toolchain): + +- Each top-level test becomes a collapsed `::group::`, holding its own output, its subtests' + and that of any docker container it started. Note this needs `-test.v=test2json` rather than + plain `-test.v`, so that `testing` emits the framing markers `test2json` uses to attribute + container output to the test that produced it. +- An ungrouped `PASS|FAIL|SKIP (12.34s)` line follows each group, turning the collapsed + log into a scannable index of results. +- Failures are repeated in a `===== FAILURES =====` section and emitted as `::error::` + annotations, so they also appear in the run's Annotations panel and on the pull request diff. +- Every shard appends counts and a `
` per failure to its own job summary; + `integration-summary` then renders one table across all shards. + +A failing shard uploads its raw `test2json` stream and JSON report as +`integration-logs--` (7 day retention). That stream is the authoritative record: +containers run with `--rm` and their shared directory is deleted when the scenario closes, so +the captured stdout is the only surviving copy of their logs. + **Naming Convention** Each step in a job has a clear name that encapsulates the purpose of the command. The convention we are using is each word in the name should be capitalized except articles and prepositions. This creates consistent labeling when looking at the progress of the current workflow on GitHub. @@ -62,6 +87,8 @@ As of October 2020, GitHub Actions do not persist between different jobs in the |-------------------------------|-----------|---------------------------------------------|-----------------------------| | website public | build | deploy_website | share data between jobs | | Docker Images | build | deploy, integration | share data between jobs | +| integration-tests-\ | build-integration-tests | integration | share the compiled test binary, its output renderers and its testdata | +| integration-logs-\-\ | integration (on failure) | integration-summary, humans | keep the raw test2json stream and JSON report of a failing shard | *Note:* Docker Images are zipped before uploading as a workaround. The images contain characters that are illegal in the upload-artifact action. ```yaml diff --git a/.github/workflows/scripts/summarize-integration-tests.sh b/.github/workflows/scripts/summarize-integration-tests.sh new file mode 100755 index 00000000000..a1bf7c1a214 --- /dev/null +++ b/.github/workflows/scripts/summarize-integration-tests.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# Renders a single cross-shard summary of the integration test matrix. +# +# Input is a directory of integration-report--.json files, written by +# tools/gha-testlog and uploaded by the `integration` job. Only failing shards upload a +# report, so a shard missing from the summary passed. +# +# The summary is written to stdout *and*, when set, appended to $GITHUB_STEP_SUMMARY. It goes +# to stdout as well so that this job's log explains its own red X: without it the log reads +# only "Process completed with exit code 1". +# +# INTEGRATION_RESULT must carry the `integration` matrix job's aggregate result; the script +# exits non-zero unless it is "success", so this job's status mirrors the matrix. + +set -euo pipefail + +REPORTS_DIR="${1:-.}" +RESULT="${INTEGRATION_RESULT:-unknown}" + +OUT="$(mktemp)" +flush() { + cat "$OUT" + if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then + cat "$OUT" >>"$GITHUB_STEP_SUMMARY" + fi + rm -f "$OUT" +} +# An EXIT trap so every early `exit` below still renders what it built. The trap preserves +# the script's exit status. +trap flush EXIT + +emit() { + printf '%s\n' "${1-}" >>"$OUT" +} + +REPORTS=() +if [ -d "$REPORTS_DIR" ]; then + while IFS= read -r report; do + REPORTS+=("$report") + done < <(find "$REPORTS_DIR" -maxdepth 1 -name 'integration-report-*.json' | sort) +fi + +emit "## Integration tests" +emit "" + +if [ "${#REPORTS[@]}" -eq 0 ]; then + if [ "$RESULT" = "success" ]; then + emit "All integration test shards passed." + exit 0 + fi + emit "The \`integration\` job result was \`${RESULT}\`, but no shard uploaded a report." + emit "" + emit "That means a shard died before its output could be rendered — a runner failure, a" + emit "cancelled run, or a process killed outside the test binary. Open the red" + emit "\`integration\` jobs directly." + exit 1 +fi + +emit "The \`integration\` job result was \`${RESULT}\`. A shard missing from this table either" +emit "passed, or failed before it could report — check for red \`integration\` jobs not listed here." +emit "" +emit "| Shard | Tests | Failed |" +emit "|---|---:|---:|" + +VALID=() +for report in "${REPORTS[@]}"; do + if ! jq -e . "$report" >/dev/null 2>&1; then + emit "| \`$(basename "$report")\` (unreadable) | — | — |" + continue + fi + VALID+=("$report") + # Both columns count top-level tests, so they are directly comparable; the individual + # failing subtests are listed in the
block below. + jq -r '"| \(.shard) | \(.tests.total) | \(.tests.failed) |"' "$report" >>"$OUT" +done + +emit "" + +for report in "${VALID[@]+"${VALID[@]}"}"; do + jq -r ' + if (.failures | length) == 0 then + "
⚠️ \(.shard) — no test failures recorded", + "", + "The shard failed outside the tests (setup, docker, or a killed process). See its job log.", + "", + "
", + "" + else + "
❌ \(.shard) — \(.tests.failed) of \(.tests.total) test(s) failed", + "", + (.failures[] | + ("- `\(.test)`" + + (if (.file // "") != "" then " — `\(.file):\(.line)`" else "" end) + + (if .incomplete then " _(never reported a result)_" else "" end)), + ((.message // "") | select(. != "") | " > " + gsub("\n"; "\n > "))), + "", + "
", + "" + end + ' "$report" >>"$OUT" +done + +if [ "$RESULT" != "success" ]; then + exit 1 +fi diff --git a/.github/workflows/test-build-deploy.yml b/.github/workflows/test-build-deploy.yml index 6950325c945..01dbf5ee223 100644 --- a/.github/workflows/test-build-deploy.yml +++ b/.github/workflows/test-build-deploy.yml @@ -195,7 +195,7 @@ jobs: ln -s /tmp/images ./docker-images make BUILD_IN_CONTAINER=false save-images - name: Create Docker Images Archive - run: tar -cvf images.tar /tmp/images + run: tar -cf images.tar /tmp/images - name: Upload Docker Images Artifact uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: @@ -232,6 +232,17 @@ jobs: ALL_TAGS="slicelabels,integration,${{ needs.discover-tags.outputs.tags_csv }}" mkdir -p out/bin go test -c -tags="${ALL_TAGS}" -o out/bin/integration.test ./integration/ + - name: Compile Test Output Renderers + # test2json turns the test binary's -test.v=test2json framing into a JSON event + # stream; gha-testlog renders that stream as collapsible groups, a result index, + # annotations and a job summary. Both ride along in the tarball below so the + # integration job still needs no checkout and no Go toolchain. + # + # test2json must be built by the same toolchain that built integration.test: the + # framing markers are a private contract between `testing` and test2json. + run: | + go build -o out/bin/test2json cmd/test2json + go build -o out/bin/gha-testlog ./tools/gha-testlog - name: Generate Run-Pattern Manifest # For each build tag, derive a -test.run regex listing every TestX function in source files # gated by that tag. The integration job will read these to select which tests to run. @@ -259,7 +270,7 @@ jobs: cp -r docs/configuration out/testdata/docs/ cp VERSION out/testdata/ - name: Create Integration Tests Archive - run: tar -C out -czvf integration-tests-${{ matrix.arch }}.tar.gz bin testdata + run: tar -C out -czf integration-tests-${{ matrix.arch }}.tar.gz bin testdata - name: Upload Integration Tests Artifact uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: @@ -279,21 +290,21 @@ jobs: with: name: Docker Images - name: Extract Docker Images Archive - run: tar -xvf images.tar -C / + run: tar -xf images.tar -C / - name: Load Docker Images # Load every saved docker image tar into the runner's docker daemon. Each tar in /tmp/images # was produced by `make save-images` in the build job (one file per image:tag-arch). run: | for img in /tmp/images/*; do [ -f "$img" ] || continue - docker load -i "$img" + docker load -q -i "$img" done - name: Download Integration Tests Artifact uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: name: integration-tests-${{ matrix.arch }} - name: Extract Integration Tests Archive - run: tar -xzvf integration-tests-${{ matrix.arch }}.tar.gz + run: tar -xzf integration-tests-${{ matrix.arch }}.tar.gz - name: Preload Images # We download docker images used by integration tests so that all images are available # locally and the download time doesn't account in the test execution time, which is subject @@ -317,40 +328,113 @@ jobs: done } - retry docker pull minio/minio:RELEASE.2024-05-28T17-19-04Z - retry docker pull consul:1.8.4 - retry docker pull quay.io/coreos/etcd:v3.5.29 + retry docker pull -q minio/minio:RELEASE.2024-05-28T17-19-04Z + retry docker pull -q consul:1.8.4 + retry docker pull -q quay.io/coreos/etcd:v3.5.29 if [ "$TEST_TAGS" = "integration_backward_compatibility" ]; then - retry docker pull quay.io/cortexproject/cortex:v1.16.1 - retry docker pull quay.io/cortexproject/cortex:v1.17.2 - retry docker pull quay.io/cortexproject/cortex:v1.18.1 - retry docker pull quay.io/cortexproject/cortex:v1.19.1 - retry docker pull quay.io/cortexproject/cortex:v1.20.1 - retry docker pull quay.io/cortexproject/cortex:v1.21.0 - retry docker pull quay.io/cortexproject/cortex:v1.21.1 + retry docker pull -q quay.io/cortexproject/cortex:v1.16.1 + retry docker pull -q quay.io/cortexproject/cortex:v1.17.2 + retry docker pull -q quay.io/cortexproject/cortex:v1.18.1 + retry docker pull -q quay.io/cortexproject/cortex:v1.19.1 + retry docker pull -q quay.io/cortexproject/cortex:v1.20.1 + retry docker pull -q quay.io/cortexproject/cortex:v1.21.0 + retry docker pull -q quay.io/cortexproject/cortex:v1.21.1 elif [ "$TEST_TAGS" = "integration_query_fuzz" ]; then - retry docker pull quay.io/cortexproject/cortex:v$(cat testdata/VERSION) - retry docker pull quay.io/prometheus/prometheus:v3.9.1 + retry docker pull -q quay.io/cortexproject/cortex:v$(cat testdata/VERSION) + retry docker pull -q quay.io/prometheus/prometheus:v3.9.1 elif [ "$TEST_TAGS" = "integration_configs_db" ]; then - retry docker pull postgres:9.6.16 + retry docker pull -q postgres:9.6.16 fi - retry docker pull memcached:1.6.1 - retry docker pull redis:7.0.4-alpine + retry docker pull -q memcached:1.6.1 + retry docker pull -q redis:7.0.4-alpine env: TEST_TAGS: ${{ matrix.tags }} - name: Integration Tests + # -test.timeout=2400s (40m) deliberately stays below this step's timeout, so a hung + # test is killed by Go rather than by the runner: its timeout panic then flows through + # as ordinary output inside the hanging test's group and still gets a FAIL result. timeout-minutes: 45 run: | + set -o pipefail export CORTEX_IMAGE_PREFIX="${IMAGE_PREFIX:-quay.io/cortexproject/}" export IMAGE_TAG="${{ needs.build.outputs.image_tag }}" - export CORTEX_IMAGE="${CORTEX_IMAGE_PREFIX}cortex:${IMAGE_TAG}-${{ matrix.arch }}" + export CORTEX_IMAGE="${CORTEX_IMAGE_PREFIX}cortex:${IMAGE_TAG}-${TEST_ARCH}" export CORTEX_CHECKOUT_DIR="$PWD/testdata" - PATTERN="$(cat bin/run-pattern-${{ matrix.tags }}.txt)" - echo "Running integration tests on ${{ matrix.arch }} with image: ${CORTEX_IMAGE}" + PATTERN="$(cat "bin/run-pattern-${TEST_TAGS}.txt")" + echo "Running integration tests on ${TEST_ARCH} with image: ${CORTEX_IMAGE}" echo "Selecting tests via -test.run=${PATTERN}" - ./bin/integration.test -test.timeout=2400s -test.v -test.count=1 -test.run="${PATTERN}" + + RAW="integration-test2json-${TEST_ARCH}-${TEST_TAGS}.jsonl" + + # -test.v=test2json (not plain -test.v) makes `testing` emit the framing markers + # test2json needs to attribute output — including the docker container logs the + # tests stream to stdout — to the test that produced it. bin/gha-testlog turns the + # resulting event stream into collapsible groups, a scannable result index, + # ::error:: annotations and a job summary. The raw stream is kept as the + # authoritative copy: containers are removed after each test, so this is the only + # surviving record of their logs. + # + # errexit is off around the pipeline so the raw stream is still archived, and the + # step's status still comes from the test binary, when tests fail. + set +e + ./bin/test2json -t -p integration \ + ./bin/integration.test -test.timeout=2400s -test.count=1 -test.v=test2json -test.run="${PATTERN}" \ + | tee "${RAW}" \ + | ./bin/gha-testlog \ + -shard "${TEST_ARCH} / ${TEST_TAGS}" \ + -arch "${TEST_ARCH}" \ + -tags "${TEST_TAGS}" \ + -report "integration-report-${TEST_ARCH}-${TEST_TAGS}.json" + STATUS=("${PIPESTATUS[@]}") + set -e + + gzip -f "${RAW}" + + if [ "${STATUS[2]}" -ne 0 ]; then + echo "::warning::bin/gha-testlog exited ${STATUS[2]}; the rendering above may be incomplete. The uploaded raw test2json stream is authoritative." + fi + exit "${STATUS[0]}" env: IMAGE_PREFIX: ${{ secrets.IMAGE_PREFIX }} + TEST_ARCH: ${{ matrix.arch }} + TEST_TAGS: ${{ matrix.tags }} + - name: Upload Integration Test Logs + # Failure-only: an artifact per shard on every green run is clutter, and "no report" + # then cleanly means "this shard passed" to the integration-summary job. + if: failure() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: integration-logs-${{ matrix.arch }}-${{ matrix.tags }} + path: | + integration-test2json-*.jsonl.gz + integration-report-*.json + if-no-files-found: ignore + retention-days: 7 + + integration-summary: + # Renders one table across every integration shard, so a red run can be triaged without + # opening 24 job logs one at a time. Reads the reports uploaded by failing shards; shards + # that pass upload nothing and are simply absent. + needs: [integration] + if: ${{ !cancelled() && needs.integration.result != 'skipped' }} + runs-on: ubuntu-24.04 + steps: + - name: Checkout Workflow Scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + sparse-checkout: .github/workflows/scripts + - name: Download Integration Test Reports + # No artifacts at all is the green case, which download-artifact treats as an error. + continue-on-error: true + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + pattern: integration-logs-* + merge-multiple: true + path: reports + - name: Summarize Integration Tests + run: ./.github/workflows/scripts/summarize-integration-tests.sh reports + env: + INTEGRATION_RESULT: ${{ needs.integration.result }} deploy: needs: [build, test, lint, integration] diff --git a/tools/gha-testlog/main.go b/tools/gha-testlog/main.go new file mode 100644 index 00000000000..6e48a2dd1ef --- /dev/null +++ b/tools/gha-testlog/main.go @@ -0,0 +1,503 @@ +// gha-testlog turns a `go tool test2json` event stream into readable GitHub Actions output. +// +// It reads the JSON lines emitted by test2json on stdin and writes to stdout: +// +// - one collapsible ::group:: per top-level test, holding that test's output, its +// subtests' output and the output of any docker container it started; +// - an ungrouped "PASS|FAIL|SKIP (12.34s)" line after each group, so the +// collapsed log reads as a scannable index of results; +// - a "===== FAILURES =====" section repeating the tail of every failing test; +// - one ::error:: annotation per failure, so failures show up in the run's +// Annotations panel and on the pull request diff. +// +// It also appends a markdown report to $GITHUB_STEP_SUMMARY and, with -report, writes a +// small JSON summary for cross-shard aggregation. +// +// The exit status is 0 unless rendering itself failed; the caller is expected to +// propagate the test binary's own status (see .github/workflows/test-build-deploy.yml). +package main + +import ( + "bufio" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "os" + "path" + "regexp" + "strconv" + "strings" +) + +const ( + // Number of trailing output lines repeated for each failing test. + defaultTailLines = 60 + + // GitHub only displays 10 annotations per step, so emitting more is wasted work. + defaultMaxAnnotations = 10 + + // $GITHUB_STEP_SUMMARY is rejected above 1 MiB; stay well clear of the limit. + maxStepSummaryBytes = 900 * 1024 + + // An annotation message is a tooltip, not a log: keep it short. + maxMessageLines = 4 + maxMessageChars = 800 +) + +// testFileRef matches the "foo_test.go:123:" prefix `testing` puts in front of everything a +// test logs. It is deliberately anchored to the start of the line: the absolute paths inside +// testify's "Error Trace:" block would otherwise win, and its frames run outermost-last, so +// the blame would land on the caller of an assertion helper rather than on the assertion. +var testFileRef = regexp.MustCompile(`^\s*([A-Za-z0-9_.-]+_test\.go):(\d+):`) + +// traceFrame matches a bare "path/to/file.go:123" stack frame, which is what the +// continuation lines of testify's "Error Trace:" block look like once trimmed. +var traceFrame = regexp.MustCompile(`^\S+\.go:\d+$`) + +// event is the subset of the test2json event schema we care about. Package-level events +// carry an empty Test. +type event struct { + Action string `json:"Action"` + Test string `json:"Test"` + Output string `json:"Output"` + Elapsed float64 `json:"Elapsed"` +} + +// testResult accumulates state for one test name. Entries exist for subtests too, keyed +// by their full "Parent/Sub" name; tail and subs are only populated for top-level tests. +type testResult struct { + name string + status string // PASS, FAIL or SKIP; empty until the result event arrives. + elapsed float64 + incomplete bool // No result event ever arrived (panic, timeout, killed process). + + tail []string // Ring buffer of the last N output lines, dropped once the test passes. + subs []string // Failing subtests, in the order they failed. + + file string // First "foo_test.go" seen in this test's own output. + line int // ...and the line number next to it. + message []string // First error message seen, for the annotation. + capturing bool // Still collecting message lines. +} + +type options struct { + tailLines int + maxAnnotations int + shard string + arch string + tags string + reportPath string + summaryPath string // $GITHUB_STEP_SUMMARY; nothing is written when empty. +} + +type renderer struct { + out io.Writer + opts options + + order []string // Top-level test names, in first-seen order. + tests map[string]*testResult // Keyed by full test name. + + openGroup string // Top-level test whose ::group:: is currently open. + partial strings.Builder + partialKey string // Test the buffered partial line belongs to. +} + +func main() { + var opts options + flag.IntVar(&opts.tailLines, "tail", defaultTailLines, "Number of trailing output lines to repeat for each failing test.") + flag.IntVar(&opts.maxAnnotations, "max-annotations", defaultMaxAnnotations, "Maximum number of ::error:: annotations to emit.") + flag.StringVar(&opts.shard, "shard", "", "Human readable shard name, used in annotation titles and the job summary.") + flag.StringVar(&opts.arch, "arch", "", "Architecture this shard ran on, recorded in the JSON report.") + flag.StringVar(&opts.tags, "tags", "", "Build tag this shard ran, recorded in the JSON report.") + flag.StringVar(&opts.reportPath, "report", "", "Write a JSON summary to this path, for cross-shard aggregation.") + flag.Parse() + + opts.summaryPath = os.Getenv("GITHUB_STEP_SUMMARY") + + r := newRenderer(os.Stdout, opts) + if err := r.run(os.Stdin); err != nil { + fmt.Fprintf(os.Stderr, "gha-testlog: %v\n", err) + os.Exit(1) + } +} + +func newRenderer(out io.Writer, opts options) *renderer { + return &renderer{ + out: out, + opts: opts, + tests: map[string]*testResult{}, + } +} + +// run consumes a test2json stream and renders it. It returns an error only when rendering +// itself failed; whatever was already rendered is still written out. +func (r *renderer) run(in io.Reader) error { + sc := bufio.NewScanner(in) + // test2json splits output events at 1 KiB, but the JSON envelope of a long line can + // still exceed bufio's 64 KiB default. + sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024) + + for sc.Scan() { + var ev event + if err := json.Unmarshal(sc.Bytes(), &ev); err != nil || ev.Action == "" { + // Not a test2json event — most likely a final line truncated by a killed + // process. Pass it through rather than dropping the evidence. + r.printLine(sc.Text()) + continue + } + r.handle(ev) + } + + return errors.Join(sc.Err(), r.finalize()) +} + +func (r *renderer) handle(ev event) { + switch ev.Action { + case "run": + if ev.Test != "" { + r.ensureGroup(topLevel(ev.Test)) + } + case "output": + r.output(ev.Test, ev.Output) + case "pass", "fail", "skip": + r.finish(ev.Test, strings.ToUpper(ev.Action), ev.Elapsed) + } +} + +// output renders an output event, which is not necessarily a whole line: test2json splits +// long lines across events, so partial lines are buffered until their newline arrives. +func (r *renderer) output(test, s string) { + if test != r.partialKey { + // Flush first, so the buffered tail of the previous line is still rendered + // inside the group it belongs to. + r.flushPartial() + r.partialKey = test + } + if test != "" { + r.ensureGroup(topLevel(test)) + } + + for { + line, rest, found := strings.Cut(s, "\n") + if !found { + r.partial.WriteString(s) + return + } + r.partial.WriteString(line) + complete := r.partial.String() + r.partial.Reset() + r.emitLine(test, complete) + s = rest + } +} + +func (r *renderer) finish(test, status string, elapsed float64) { + if test == "" { + // Package-level result; the per-test result lines already say everything. + return + } + + t := r.entry(test) + t.status = status + t.elapsed = elapsed + + top := topLevel(test) + if test != top { + if status == "FAIL" { + parent := r.entry(top) + parent.subs = append(parent.subs, test) + } + return + } + + r.flushPartial() + r.closeGroup() + if status != "FAIL" { + t.tail = nil // Only failures need their output repeated. + } + r.printf("%s %s (%.2fs)\n", status, top, elapsed) +} + +// finalize closes any dangling group, accounts for tests that never reported a result, +// and writes the failure section, annotations, job summary and JSON report. +func (r *renderer) finalize() error { + r.flushPartial() + r.closeGroup() + + for _, name := range r.order { + t := r.tests[name] + if t.status != "" { + continue + } + // The stream ended mid-test: a panic, Go's own -test.timeout, or a killed + // process. Report it rather than letting it vanish. + t.status = "FAIL" + t.incomplete = true + r.printf("FAIL %s (incomplete)\n", name) + } + + failed := r.failures() + r.printFailures(failed) + r.printAnnotations(failed) + + return errors.Join(r.writeStepSummary(), r.writeReport()) +} + +func (r *renderer) ensureGroup(top string) { + if r.openGroup == top { + return + } + r.closeGroup() + r.entry(top) + r.printf("::group::%s\n", top) + r.openGroup = top +} + +func (r *renderer) closeGroup() { + if r.openGroup == "" { + return + } + r.flushPartial() + r.printf("::endgroup::\n") + r.openGroup = "" +} + +func (r *renderer) flushPartial() { + if r.partial.Len() == 0 { + return + } + line := r.partial.String() + r.partial.Reset() + r.emitLine(r.partialKey, line) +} + +func (r *renderer) emitLine(test, line string) { + if test != "" { + r.entry(test).recordDiagnostics(line) + if top := topLevel(test); r.opts.tailLines > 0 { + t := r.entry(top) + t.tail = append(t.tail, line) + if len(t.tail) > r.opts.tailLines { + t.tail = t.tail[1:] + } + } + } + r.printLine(line) +} + +// printLine writes one line of test output. A line that looks like a workflow command is +// indented by one space first: a Cortex container logging "::endgroup::" would otherwise +// silently destroy the grouping of the whole log. +func (r *renderer) printLine(line string) { + if strings.HasPrefix(strings.TrimSpace(line), "::") { + line = " " + line + } + r.printf("%s\n", line) +} + +func (r *renderer) printf(format string, args ...any) { + // Nothing useful can be done if stdout is broken, and the caller's exit status is + // driven by the test binary, not by us. + _, _ = fmt.Fprintf(r.out, format, args...) +} + +func (r *renderer) entry(name string) *testResult { + if t, ok := r.tests[name]; ok { + return t + } + t := &testResult{name: name} + r.tests[name] = t + if name == topLevel(name) { + r.order = append(r.order, name) + } + return t +} + +// failures returns the failing top-level tests, in the order they ran. +func (r *renderer) failures() []*testResult { + var out []*testResult + for _, name := range r.order { + if t := r.tests[name]; t.status == "FAIL" { + out = append(out, t) + } + } + return out +} + +func (r *renderer) printFailures(failed []*testResult) { + if len(failed) == 0 { + return + } + + r.printf("\n===== FAILURES (%d) =====\n", len(failed)) + for _, t := range failed { + if t.incomplete { + r.printf("\n--- FAIL %s (incomplete: the test never reported a result)\n", t.name) + } else { + r.printf("\n--- FAIL %s (%.2fs)\n", t.name, t.elapsed) + } + for _, sub := range t.subs { + r.printf(" failed subtest: %s\n", sub) + } + if ref := r.sourceRef(t); ref != "" { + r.printf(" at %s\n", ref) + } + if len(t.tail) > 0 { + r.printf(" --- last %d line(s) of output ---\n", len(t.tail)) + for _, line := range t.tail { + r.printLine(line) + } + } + } +} + +// printAnnotations emits one ::error:: per failure, anchored on the most specific failing +// name so a failing subtest is named instead of just its parent. +func (r *renderer) printAnnotations(failed []*testResult) { + targets := r.annotationTargets(failed) + + shown := targets + if len(shown) > r.opts.maxAnnotations { + shown = shown[:r.opts.maxAnnotations] + } + + for _, t := range shown { + title := t.name + if r.opts.shard != "" { + title = r.opts.shard + " / " + t.name + } + + props := "title=" + escapeProperty(title) + if file, line := r.location(t); file != "" { + props = fmt.Sprintf("file=%s,line=%d,%s", escapeProperty(file), line, props) + } + r.printf("::error %s::%s\n", props, escapeData(r.annotationMessage(t))) + } + + if n := len(targets) - len(shown); n > 0 { + r.printf("::warning::%d further failure(s) not annotated; see the FAILURES section above.\n", n) + } +} + +func (r *renderer) annotationTargets(failed []*testResult) []*testResult { + var out []*testResult + for _, t := range failed { + if len(t.subs) == 0 { + out = append(out, t) + continue + } + for _, sub := range t.subs { + out = append(out, r.entry(sub)) + } + } + return out +} + +// diagnostics returns the tests whose output may hold a source reference or error message +// for t, most specific first: t itself, then its failing subtests (a parent reports no +// detail of its own), then its parent (a shared helper is attributed to the parent). +func (r *renderer) diagnostics(t *testResult) []*testResult { + out := []*testResult{t} + for _, sub := range t.subs { + if s, ok := r.tests[sub]; ok { + out = append(out, s) + } + } + if parent, ok := r.tests[topLevel(t.name)]; ok && parent != t { + out = append(out, parent) + } + return out +} + +func (r *renderer) location(t *testResult) (string, int) { + for _, c := range r.diagnostics(t) { + if c.file != "" { + return path.Join("integration", c.file), c.line + } + } + return "", 0 +} + +func (r *renderer) sourceRef(t *testResult) string { + file, line := r.location(t) + if file == "" { + return "" + } + return fmt.Sprintf("%s:%d", file, line) +} + +func (r *renderer) annotationMessage(t *testResult) string { + if t.incomplete { + return t.name + " never reported a result (panic, timeout or killed process)." + } + + for _, c := range r.diagnostics(t) { + if len(c.message) == 0 { + continue + } + msg := strings.Join(c.message, "\n") + if len(msg) > maxMessageChars { + // ToValidUTF8 drops the partial rune the cut may have left behind. + msg = strings.ToValidUTF8(msg[:maxMessageChars], "") + "…" + } + return msg + } + return t.name + " failed." +} + +// recordDiagnostics keeps the source reference and error message the annotation and the job +// summary point at. +func (t *testResult) recordDiagnostics(line string) { + if loc := testFileRef.FindStringSubmatchIndex(line); loc != nil { + // The *last* reference wins, not the first. Integration tests log with t.Logf as + // they go, and each of those lines carries a "foo_test.go:NN:" prefix too, so the + // first reference would usually blame a log line rather than the assertion. + t.file = line[loc[2]:loc[3]] + t.line, _ = strconv.Atoi(line[loc[4]:loc[5]]) + t.message = nil + t.capturing = true + // Whatever follows "foo_test.go:123:" on the same line starts the message. + if rest := strings.TrimSpace(line[loc[1]:]); rest != "" { + t.message = append(t.message, rest) + } + return + } + + if !t.capturing { + return + } + trimmed := strings.TrimSpace(line) + switch { + case trimmed == "" || strings.HasPrefix(trimmed, "Error Trace:") || traceFrame.MatchString(trimmed): + // Blank padding and stack frames add nothing to a one-line annotation. + return + case strings.HasPrefix(trimmed, "---") || strings.HasPrefix(trimmed, "==="), + trimmed == "Diff:", // testify's rendered diff repeats expected/actual. + len(t.message) >= maxMessageLines: + t.capturing = false + return + } + t.message = append(t.message, trimmed) +} + +func topLevel(name string) string { + top, _, _ := strings.Cut(name, "/") + return top +} + +// escapeData escapes a workflow command's message payload. +func escapeData(s string) string { + s = strings.ReplaceAll(s, "%", "%25") + s = strings.ReplaceAll(s, "\r", "%0D") + return strings.ReplaceAll(s, "\n", "%0A") +} + +// escapeProperty escapes a workflow command property value, where "," and ":" are +// structural. +func escapeProperty(s string) string { + s = escapeData(s) + s = strings.ReplaceAll(s, ":", "%3A") + return strings.ReplaceAll(s, ",", "%2C") +} diff --git a/tools/gha-testlog/main_test.go b/tools/gha-testlog/main_test.go new file mode 100644 index 00000000000..42ee8d20a81 --- /dev/null +++ b/tools/gha-testlog/main_test.go @@ -0,0 +1,479 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" +) + +// The helpers below build the raw test2json JSON lines the renderer consumes. + +func evRun(test string) string { + return fmt.Sprintf(`{"Action":"run","Package":"integration","Test":%q}`, test) +} + +func evOut(test, output string) string { + return fmt.Sprintf(`{"Action":"output","Package":"integration","Test":%q,"Output":%q}`, test, output) +} + +func evPkgOut(output string) string { + return fmt.Sprintf(`{"Action":"output","Package":"integration","Output":%q}`, output) +} + +func evResult(action, test string, elapsed float64) string { + return fmt.Sprintf(`{"Action":%q,"Package":"integration","Test":%q,"Elapsed":%v}`, action, test, elapsed) +} + +func jsonl(lines ...string) string { + return strings.Join(lines, "\n") + "\n" +} + +func render(t *testing.T, opts options, input string) string { + t.Helper() + + if opts.tailLines == 0 { + opts.tailLines = defaultTailLines + } + if opts.maxAnnotations == 0 { + opts.maxAnnotations = defaultMaxAnnotations + } + + var buf bytes.Buffer + r := newRenderer(&buf, opts) + if err := r.run(strings.NewReader(input)); err != nil { + t.Fatalf("run: %v", err) + } + return buf.String() +} + +func countLines(out, line string) int { + n := 0 + for l := range strings.SplitSeq(out, "\n") { + if l == line { + n++ + } + } + return n +} + +func TestRender(t *testing.T) { + tests := []struct { + name string + opts options + input string + want string + contains []string + notContains []string + }{ + { + name: "passing test is grouped and followed by a result line", + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", "=== RUN TestFoo\n"), + evOut("TestFoo", "hello\n"), + evOut("TestFoo", "--- PASS: TestFoo (1.00s)\n"), + evResult("pass", "TestFoo", 1), + ), + want: "::group::TestFoo\n" + + "=== RUN TestFoo\n" + + "hello\n" + + "--- PASS: TestFoo (1.00s)\n" + + "::endgroup::\n" + + "PASS TestFoo (1.00s)\n", + }, + { + name: "each top level test gets its own group", + input: jsonl( + evRun("TestA"), + evOut("TestA", "a\n"), + evResult("pass", "TestA", 1), + evRun("TestB"), + evOut("TestB", "b\n"), + evResult("skip", "TestB", 0), + ), + want: "::group::TestA\n" + + "a\n" + + "::endgroup::\n" + + "PASS TestA (1.00s)\n" + + "::group::TestB\n" + + "b\n" + + "::endgroup::\n" + + "SKIP TestB (0.00s)\n", + }, + { + name: "subtests share their parent's group", + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", "=== RUN TestFoo\n"), + evRun("TestFoo/Sub"), + evOut("TestFoo/Sub", "inside sub\n"), + evResult("pass", "TestFoo/Sub", 0.5), + evOut("TestFoo", "back in parent\n"), + evResult("pass", "TestFoo", 1), + ), + want: "::group::TestFoo\n" + + "=== RUN TestFoo\n" + + "inside sub\n" + + "back in parent\n" + + "::endgroup::\n" + + "PASS TestFoo (1.00s)\n", + }, + { + name: "package level output stays outside any group", + input: jsonl( + evPkgOut("before any test\n"), + evRun("TestFoo"), + evOut("TestFoo", "in test\n"), + evResult("pass", "TestFoo", 1), + evPkgOut("PASS\n"), + evPkgOut("ok \tintegration\t1.234s\n"), + ), + want: "before any test\n" + + "::group::TestFoo\n" + + "in test\n" + + "::endgroup::\n" + + "PASS TestFoo (1.00s)\n" + + "PASS\n" + + "ok \tintegration\t1.234s\n", + }, + { + name: "a line split across events is rendered once", + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", "first half "), + evOut("TestFoo", "second half\n"), + evResult("pass", "TestFoo", 1), + ), + want: "::group::TestFoo\n" + + "first half second half\n" + + "::endgroup::\n" + + "PASS TestFoo (1.00s)\n", + }, + { + name: "an unterminated final line is still rendered inside its group", + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", "no trailing newline"), + evResult("pass", "TestFoo", 1), + ), + want: "::group::TestFoo\n" + + "no trailing newline\n" + + "::endgroup::\n" + + "PASS TestFoo (1.00s)\n", + }, + { + name: "workflow commands in test output cannot break grouping", + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", "::endgroup::\n"), + evOut("TestFoo", " ::error::from a container\n"), + evOut("TestFoo", "::group::sneaky\n"), + evResult("pass", "TestFoo", 1), + ), + want: "::group::TestFoo\n" + + " ::endgroup::\n" + + " ::error::from a container\n" + + " ::group::sneaky\n" + + "::endgroup::\n" + + "PASS TestFoo (1.00s)\n", + }, + { + name: "a truncated trailing line is passed through verbatim", + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", "in test\n"), + evResult("pass", "TestFoo", 1), + ) + `{"Action":"output","Package":"integration","Out`, + want: "::group::TestFoo\n" + + "in test\n" + + "::endgroup::\n" + + "PASS TestFoo (1.00s)\n" + + `{"Action":"output","Package":"integration","Out` + "\n", + }, + { + name: "a stream that ends mid test reports the test as incomplete", + input: jsonl( + evRun("TestHang"), + evOut("TestHang", "started\n"), + ), + contains: []string{ + "::group::TestHang\n", + "started\n", + "::endgroup::\n", + "FAIL TestHang (incomplete)\n", + "===== FAILURES (1) =====", + "--- FAIL TestHang (incomplete: the test never reported a result)", + "::error title=TestHang::TestHang never reported a result", + }, + }, + { + name: "the assertion is blamed, not an earlier t.Logf", + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", " querier_test.go:10: Testing: up == 1\n"), + evOut("TestFoo", " querier_test.go:20: Query returned 3 series\n"), + evOut("TestFoo", " querier_test.go:31: \n"), + evOut("TestFoo", " \tError Trace:\t/go/src/cortex/integration/querier_test.go:31\n"), + evOut("TestFoo", " \tError: \tReceived unexpected error:\n"), + evOut("TestFoo", " \t \tcontext deadline exceeded\n"), + evResult("fail", "TestFoo", 1), + ), + contains: []string{ + "::error file=integration/querier_test.go,line=31,title=TestFoo::Error: \tReceived unexpected error:%0Acontext deadline exceeded\n", + }, + notContains: []string{ + "line=10", + "line=20", + }, + }, + { + name: "testify's Error Trace frames do not override the assertion's own line", + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", " ruler_test.go:495: \n"), + evOut("TestFoo", " \tError Trace:\t/go/src/cortex/integration/ruler_test.go:495\n"), + evOut("TestFoo", " \t \t\t\t\t/go/src/cortex/integration/ruler_test.go:586\n"), + evOut("TestFoo", " \tError: \tNot equal\n"), + evResult("fail", "TestFoo", 1), + ), + contains: []string{ + " at integration/ruler_test.go:495\n", + "::error file=integration/ruler_test.go,line=495,title=TestFoo::Error: \tNot equal\n", + }, + notContains: []string{ + // 586 is the caller of the assertion helper, not the assertion, and the + // trace frames themselves are noise in a tooltip. + "line=586", + "ruler_test.go:586%0A", + }, + }, + { + name: "a failure is annotated with its source location and message", + opts: options{shard: "amd64 / integration_ruler"}, + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", " ruler_test.go:123: \n"), + evOut("TestFoo", " \tError Trace:\t/go/src/ruler_test.go:123\n"), + evOut("TestFoo", " \tError: \tNot equal: 1 != 2\n"), + evOut("TestFoo", "--- FAIL: TestFoo (1.00s)\n"), + evResult("fail", "TestFoo", 1), + ), + contains: []string{ + "FAIL TestFoo (1.00s)\n", + " at integration/ruler_test.go:123\n", + // The message stops before the stack trace, which is noise in a tooltip. + "::error file=integration/ruler_test.go,line=123,title=amd64 / integration_ruler / TestFoo::Error: \tNot equal: 1 != 2\n", + }, + }, + { + name: "annotations name the failing subtest rather than its parent", + input: jsonl( + evRun("TestFoo"), + evRun("TestFoo/Sub"), + evOut("TestFoo/Sub", " foo_test.go:7: boom\n"), + evResult("fail", "TestFoo/Sub", 0.5), + evResult("fail", "TestFoo", 1), + ), + contains: []string{ + " failed subtest: TestFoo/Sub\n", + "::error file=integration/foo_test.go,line=7,title=TestFoo/Sub::boom\n", + }, + notContains: []string{ + "title=TestFoo::", + }, + }, + { + name: "annotations are capped with an overflow warning", + opts: options{maxAnnotations: 2}, + input: jsonl( + evRun("TestA"), evResult("fail", "TestA", 1), + evRun("TestB"), evResult("fail", "TestB", 1), + evRun("TestC"), evResult("fail", "TestC", 1), + ), + contains: []string{ + "===== FAILURES (3) =====", + "::error title=TestA::TestA failed.\n", + "::error title=TestB::TestB failed.\n", + "::warning::1 further failure(s) not annotated; see the FAILURES section above.\n", + }, + notContains: []string{ + "::error title=TestC::", + }, + }, + { + name: "the failure section repeats only the last -tail lines", + opts: options{tailLines: 2}, + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", "line1\n"), + evOut("TestFoo", "line2\n"), + evOut("TestFoo", "line3\n"), + evOut("TestFoo", "line4\n"), + evResult("fail", "TestFoo", 1), + ), + contains: []string{ + " --- last 2 line(s) of output ---\nline3\nline4\n", + }, + notContains: []string{ + // line1 and line2 appear once in the group but must not be repeated. + "line1\nline2\nline3\nline4\n --- last", + }, + }, + { + name: "a passing run emits no failure section and no annotations", + input: jsonl( + evRun("TestFoo"), + evResult("pass", "TestFoo", 1), + ), + notContains: []string{"FAILURES", "::error", "::warning"}, + }, + { + name: "percent signs and newlines in a message are escaped", + input: jsonl( + evRun("TestFoo"), + evOut("TestFoo", " foo_test.go:1: 100% wrong\n"), + evOut("TestFoo", " second line\n"), + evResult("fail", "TestFoo", 1), + ), + contains: []string{ + "::error file=integration/foo_test.go,line=1,title=TestFoo::100%25 wrong%0Asecond line\n", + }, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := render(t, tc.opts, tc.input) + + if tc.want != "" && got != tc.want { + t.Errorf("unexpected output\n--- got ---\n%s\n--- want ---\n%s", got, tc.want) + } + for _, want := range tc.contains { + if !strings.Contains(got, want) { + t.Errorf("output does not contain %q\n--- got ---\n%s", want, got) + } + } + for _, unwanted := range tc.notContains { + if strings.Contains(got, unwanted) { + t.Errorf("output unexpectedly contains %q\n--- got ---\n%s", unwanted, got) + } + } + if open, closed := strings.Count(got, "\n::group::")+boolToInt(strings.HasPrefix(got, "::group::")), countLines(got, "::endgroup::"); open != closed { + t.Errorf("unbalanced groups: %d ::group:: vs %d ::endgroup::\n--- got ---\n%s", open, closed, got) + } + }) + } +} + +func boolToInt(b bool) int { + if b { + return 1 + } + return 0 +} + +func TestReportAndStepSummary(t *testing.T) { + dir := t.TempDir() + summaryPath := filepath.Join(dir, "summary.md") + reportPath := filepath.Join(dir, "report.json") + + input := jsonl( + evRun("TestPasses"), + evResult("pass", "TestPasses", 1), + evRun("TestSkips"), + evResult("skip", "TestSkips", 0), + evRun("TestFails"), + evRun("TestFails/Sub"), + evOut("TestFails/Sub", " ruler_test.go:42: boom\n"), + evResult("fail", "TestFails/Sub", 0.5), + evResult("fail", "TestFails", 1), + ) + + opts := options{ + tailLines: defaultTailLines, + maxAnnotations: defaultMaxAnnotations, + shard: "amd64 / integration_ruler", + arch: "amd64", + tags: "integration_ruler", + reportPath: reportPath, + summaryPath: summaryPath, + } + render(t, opts, input) + + raw, err := os.ReadFile(reportPath) + if err != nil { + t.Fatalf("reading report: %v", err) + } + var rep report + if err := json.Unmarshal(raw, &rep); err != nil { + t.Fatalf("unmarshalling report: %v", err) + } + + if want := (counts{Total: 3, Passed: 1, Failed: 1, Skipped: 1}); rep.Tests != want { + t.Errorf("counts = %+v, want %+v", rep.Tests, want) + } + if rep.Arch != "amd64" || rep.Tags != "integration_ruler" || rep.Shard != "amd64 / integration_ruler" { + t.Errorf("unexpected shard identity: %+v", rep) + } + if len(rep.Failures) != 1 { + t.Fatalf("failures = %+v, want 1 entry", rep.Failures) + } + got := rep.Failures[0] + want := reportFailure{Test: "TestFails/Sub", TopLevel: "TestFails", File: "integration/ruler_test.go", Line: 42, Message: "boom"} + if got != want { + t.Errorf("failure = %+v, want %+v", got, want) + } + + summary, err := os.ReadFile(summaryPath) + if err != nil { + t.Fatalf("reading step summary: %v", err) + } + for _, want := range []string{ + "### Integration tests — amd64 / integration_ruler", + "3 test(s): 1 passed, 1 failed, 1 skipped", + "
❌ TestFails — integration/ruler_test.go:42", + "- `TestFails/Sub`", + } { + if !strings.Contains(string(summary), want) { + t.Errorf("step summary does not contain %q\n--- got ---\n%s", want, summary) + } + } +} + +func TestStepSummaryIsSkippedWithoutAPath(t *testing.T) { + // Nothing to write to, so nothing is written: this is how the tool behaves outside + // GitHub Actions, and how it must behave so the unit tests above cannot append to a + // real $GITHUB_STEP_SUMMARY. + render(t, options{}, jsonl(evRun("TestFoo"), evResult("pass", "TestFoo", 1))) +} + +func TestStepSummaryIsCapped(t *testing.T) { + summaryPath := filepath.Join(t.TempDir(), "summary.md") + + // Two failures, each with more output than the whole summary is allowed to hold. + var lines []string + for i := range 2 { + test := fmt.Sprintf("TestFail%d", i) + lines = append(lines, evRun(test)) + for range 40 { + lines = append(lines, evOut(test, strings.Repeat("x", 20*1024)+"\n")) + } + lines = append(lines, evResult("fail", test, 1)) + } + render(t, options{tailLines: 40, summaryPath: summaryPath}, jsonl(lines...)) + + summary, err := os.ReadFile(summaryPath) + if err != nil { + t.Fatalf("reading step summary: %v", err) + } + if len(summary) > maxStepSummaryBytes { + t.Errorf("step summary is %d bytes, want at most %d", len(summary), maxStepSummaryBytes) + } + if !strings.Contains(string(summary), "more failure(s) omitted") { + t.Errorf("capped step summary does not say what was omitted:\n%s", summary[:min(len(summary), 2000)]) + } +} diff --git a/tools/gha-testlog/summary.go b/tools/gha-testlog/summary.go new file mode 100644 index 00000000000..98e64bbbce4 --- /dev/null +++ b/tools/gha-testlog/summary.go @@ -0,0 +1,168 @@ +package main + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "strings" +) + +// counts are over top-level tests, matching the PASS/FAIL/SKIP result lines in the log. +type counts struct { + Total int `json:"total"` + Passed int `json:"passed"` + Failed int `json:"failed"` + Skipped int `json:"skipped"` +} + +type reportFailure struct { + Test string `json:"test"` + TopLevel string `json:"top_level"` + File string `json:"file,omitempty"` + Line int `json:"line,omitempty"` + Message string `json:"message,omitempty"` + Incomplete bool `json:"incomplete,omitempty"` +} + +// report is consumed by .github/workflows/scripts/summarize-integration-tests.sh to build +// a single table across all shards. +type report struct { + Shard string `json:"shard"` + Arch string `json:"arch"` + Tags string `json:"tags"` + Tests counts `json:"tests"` + Failures []reportFailure `json:"failures"` +} + +func (r *renderer) counts() counts { + var c counts + for _, name := range r.order { + c.Total++ + switch r.tests[name].status { + case "PASS": + c.Passed++ + case "FAIL": + c.Failed++ + case "SKIP": + c.Skipped++ + } + } + return c +} + +func (r *renderer) report() report { + failed := r.failures() + + rep := report{ + Shard: r.opts.shard, + Arch: r.opts.arch, + Tags: r.opts.tags, + Tests: r.counts(), + Failures: []reportFailure{}, + } + for _, t := range r.annotationTargets(failed) { + file, line := r.location(t) + rep.Failures = append(rep.Failures, reportFailure{ + Test: t.name, + TopLevel: topLevel(t.name), + File: file, + Line: line, + Message: r.annotationMessage(t), + Incomplete: t.incomplete, + }) + } + return rep +} + +func (r *renderer) writeReport() error { + if r.opts.reportPath == "" { + return nil + } + b, err := json.MarshalIndent(r.report(), "", " ") + if err != nil { + return fmt.Errorf("marshalling report: %w", err) + } + if err := os.WriteFile(r.opts.reportPath, append(b, '\n'), 0o644); err != nil { + return fmt.Errorf("writing report: %w", err) + } + return nil +} + +func (r *renderer) writeStepSummary() error { + if r.opts.summaryPath == "" { + return nil + } + + f, err := os.OpenFile(r.opts.summaryPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) + if err != nil { + return fmt.Errorf("opening step summary: %w", err) + } + _, writeErr := f.WriteString(r.buildStepSummary()) + if writeErr != nil { + writeErr = fmt.Errorf("writing step summary: %w", writeErr) + } + return errors.Join(writeErr, f.Close()) +} + +func (r *renderer) buildStepSummary() string { + c := r.counts() + + var b strings.Builder + title := "Integration tests" + if r.opts.shard != "" { + title += " — " + r.opts.shard + } + fmt.Fprintf(&b, "### %s\n\n", title) + fmt.Fprintf(&b, "%d test(s): %d passed, %d failed, %d skipped\n", c.Total, c.Passed, c.Failed, c.Skipped) + + failed := r.failures() + if len(failed) == 0 { + return b.String() + "\n" + } + + b.WriteString("\n") + for i, t := range failed { + block := r.summaryFailureBlock(t) + if b.Len()+len(block) > maxStepSummaryBytes { + fmt.Fprintf(&b, "_%d more failure(s) omitted; see the job log._\n", len(failed)-i) + break + } + b.WriteString(block) + } + return b.String() +} + +func (r *renderer) summaryFailureBlock(t *testResult) string { + var b strings.Builder + + heading := t.name + if ref := r.sourceRef(t); ref != "" { + heading += " — " + ref + } + if t.incomplete { + heading += " (never reported a result)" + } + fmt.Fprintf(&b, "
❌ %s\n\n", heading) + + if len(t.subs) > 0 { + b.WriteString("Failed subtests:\n\n") + for _, sub := range t.subs { + fmt.Fprintf(&b, "- `%s`\n", sub) + } + b.WriteString("\n") + } + + if len(t.tail) > 0 { + // Four backticks so a fenced block inside the test output cannot break out. + b.WriteString("````\n") + for _, line := range t.tail { + b.WriteString(line) + b.WriteString("\n") + } + b.WriteString("````\n") + } + + b.WriteString("\n
\n\n") + return b.String() +}