Skip to content

CI: render readable integration test output #7770

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
CharlieTLe wants to merge 1 commit into
base: master
Choose a base branch
from
Open

CI: render readable integration test output #7770

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/workflows-doc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ test-build-deploy.yml specifies a workflow that runs all Cortex continuous integ
| lint | Runs linting and ensures vendor directory, protos and generated documentation are consistent. | CI |
| test | Runs units tests on Cassandra testing framework. | CI |
| integration | Runs integration tests after upgrading golang, pulling necessary docker images and downloading necessary module dependencies. | CI |
| integration-summary | Renders one cross-shard summary of the integration matrix into the job summary, so a red run can be triaged without opening every shard's log. | CI |
| Security/CodeQL | CodeQL is a semantic code analysis engine used for automating security checks. | CI |
| build | Builds and saves an up-to-date Cortex image and website. | CI |
| deploy_website | Deploys the latest version of Cortex website to gh-pages branch. Triggered within workflow. | CD |
Expand All @@ -32,6 +33,30 @@ Internal dependencies between jobs illustrated below. Jobs run concurrently wher

### Key Details

**Integration Test Output**

The `integration` matrix runs one shard per build tag per architecture, so a failure could
otherwise mean scrolling an undifferentiated wall of text in one of two dozen jobs. Instead the
test binary runs under `bin/test2json` and its event stream is rendered by `bin/gha-testlog`
(built from [`tools/gha-testlog`](../tools/gha-testlog) and shipped in the
`integration-tests-<arch>` artifact, so the job still needs no checkout and no Go toolchain):

- Each top-level test becomes a collapsed `::group::`, holding its own output, its subtests'
and that of any docker container it started. Note this needs `-test.v=test2json` rather than
plain `-test.v`, so that `testing` emits the framing markers `test2json` uses to attribute
container output to the test that produced it.
- An ungrouped `PASS|FAIL|SKIP <Test> (12.34s)` line follows each group, turning the collapsed
log into a scannable index of results.
- Failures are repeated in a `===== FAILURES =====` section and emitted as `::error::`
annotations, so they also appear in the run's Annotations panel and on the pull request diff.
- Every shard appends counts and a `<details>` per failure to its own job summary;
`integration-summary` then renders one table across all shards.

A failing shard uploads its raw `test2json` stream and JSON report as
`integration-logs-<arch>-<tag>` (7 day retention). That stream is the authoritative record:
containers run with `--rm` and their shared directory is deleted when the scenario closes, so
the captured stdout is the only surviving copy of their logs.

**Naming Convention**

Each step in a job has a clear name that encapsulates the purpose of the command. The convention we are using is each word in the name should be capitalized except articles and prepositions. This creates consistent labeling when looking at the progress of the current workflow on GitHub.
Expand Down Expand Up @@ -62,6 +87,8 @@ As of October 2020, GitHub Actions do not persist between different jobs in the
|-------------------------------|-----------|---------------------------------------------|-----------------------------|
| website public | build | deploy_website | share data between jobs |
| Docker Images | build | deploy, integration | share data between jobs |
| integration-tests-\<arch\> | build-integration-tests | integration | share the compiled test binary, its output renderers and its testdata |
| integration-logs-\<arch\>-\<tag\> | integration (on failure) | integration-summary, humans | keep the raw test2json stream and JSON report of a failing shard |

*Note:* Docker Images are zipped before uploading as a workaround. The images contain characters that are illegal in the upload-artifact action.
```yaml
Expand Down
105 changes: 105 additions & 0 deletions .github/workflows/scripts/summarize-integration-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
# Renders a single cross-shard summary of the integration test matrix.
#
# Input is a directory of integration-report-<arch>-<tag>.json files, written by
# tools/gha-testlog and uploaded by the `integration` job. Only failing shards upload a
# report, so a shard missing from the summary passed.
#
# The summary is written to stdout *and*, when set, appended to $GITHUB_STEP_SUMMARY. It goes
# to stdout as well so that this job's log explains its own red X: without it the log reads
# only "Process completed with exit code 1".
#
# INTEGRATION_RESULT must carry the `integration` matrix job's aggregate result; the script
# exits non-zero unless it is "success", so this job's status mirrors the matrix.

set -euo pipefail

REPORTS_DIR="${1:-.}"
RESULT="${INTEGRATION_RESULT:-unknown}"

OUT="$(mktemp)"
flush() {
cat "$OUT"
if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
cat "$OUT" >>"$GITHUB_STEP_SUMMARY"
fi
rm -f "$OUT"
}
# An EXIT trap so every early `exit` below still renders what it built. The trap preserves
# the script's exit status.
trap flush EXIT

emit() {
printf '%s\n' "${1-}" >>"$OUT"
}

REPORTS=()
if [ -d "$REPORTS_DIR" ]; then
while IFS= read -r report; do
REPORTS+=("$report")
done < <(find "$REPORTS_DIR" -maxdepth 1 -name 'integration-report-*.json' | sort)
fi

emit "## Integration tests"
emit ""

if [ "${#REPORTS[@]}" -eq 0 ]; then
if [ "$RESULT" = "success" ]; then
emit "All integration test shards passed."
exit 0
fi
emit "The \`integration\` job result was \`${RESULT}\`, but no shard uploaded a report."
emit ""
emit "That means a shard died before its output could be rendered — a runner failure, a"
emit "cancelled run, or a process killed outside the test binary. Open the red"
emit "\`integration\` jobs directly."
exit 1
fi

emit "The \`integration\` job result was \`${RESULT}\`. A shard missing from this table either"
emit "passed, or failed before it could report — check for red \`integration\` jobs not listed here."
emit ""
emit "| Shard | Tests | Failed |"
emit "|---|---:|---:|"

VALID=()
for report in "${REPORTS[@]}"; do
if ! jq -e . "$report" >/dev/null 2>&1; then
emit "| \`$(basename "$report")\` (unreadable) | — | — |"
continue
fi
VALID+=("$report")
# Both columns count top-level tests, so they are directly comparable; the individual
# failing subtests are listed in the <details> block below.
jq -r '"| \(.shard) | \(.tests.total) | \(.tests.failed) |"' "$report" >>"$OUT"
done

emit ""

for report in "${VALID[@]+"${VALID[@]}"}"; do
jq -r '
if (.failures | length) == 0 then
"<details><summary>⚠️ \(.shard) — no test failures recorded</summary>",
"",
"The shard failed outside the tests (setup, docker, or a killed process). See its job log.",
"",
"</details>",
""
else
"<details><summary>❌ \(.shard) — \(.tests.failed) of \(.tests.total) test(s) failed</summary>",
"",
(.failures[] |
("- `\(.test)`"
+ (if (.file // "") != "" then " — `\(.file):\(.line)`" else "" end)
+ (if .incomplete then " _(never reported a result)_" else "" end)),
((.message // "") | select(. != "") | " > " + gsub("\n"; "\n > "))),
"",
"</details>",
""
end
' "$report" >>"$OUT"
done

if [ "$RESULT" != "success" ]; then
exit 1
fi
132 changes: 108 additions & 24 deletions .github/workflows/test-build-deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ jobs:
ln -s /tmp/images ./docker-images
make BUILD_IN_CONTAINER=false save-images
- name: Create Docker Images Archive
run: tar -cvf images.tar /tmp/images
run: tar -cf images.tar /tmp/images
- name: Upload Docker Images Artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
Expand Down Expand Up @@ -232,6 +232,17 @@ jobs:
ALL_TAGS="slicelabels,integration,${{ needs.discover-tags.outputs.tags_csv }}"
mkdir -p out/bin
go test -c -tags="${ALL_TAGS}" -o out/bin/integration.test ./integration/
- name: Compile Test Output Renderers
# test2json turns the test binary's -test.v=test2json framing into a JSON event
# stream; gha-testlog renders that stream as collapsible groups, a result index,
# annotations and a job summary. Both ride along in the tarball below so the
# integration job still needs no checkout and no Go toolchain.
#
# test2json must be built by the same toolchain that built integration.test: the
# framing markers are a private contract between `testing` and test2json.
run: |
go build -o out/bin/test2json cmd/test2json
go build -o out/bin/gha-testlog ./tools/gha-testlog
- name: Generate Run-Pattern Manifest
# For each build tag, derive a -test.run regex listing every TestX function in source files
# gated by that tag. The integration job will read these to select which tests to run.
Expand Down Expand Up @@ -259,7 +270,7 @@ jobs:
cp -r docs/configuration out/testdata/docs/
cp VERSION out/testdata/
- name: Create Integration Tests Archive
run: tar -C out -czvf integration-tests-${{ matrix.arch }}.tar.gz bin testdata
run: tar -C out -czf integration-tests-${{ matrix.arch }}.tar.gz bin testdata
- name: Upload Integration Tests Artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
Expand All @@ -279,21 +290,21 @@ jobs:
with:
name: Docker Images
- name: Extract Docker Images Archive
run: tar -xvf images.tar -C /
run: tar -xf images.tar -C /
- name: Load Docker Images
# Load every saved docker image tar into the runner's docker daemon. Each tar in /tmp/images
# was produced by `make save-images` in the build job (one file per image:tag-arch).
run: |
for img in /tmp/images/*; do
[ -f "$img" ] || continue
docker load -i "$img"
docker load -q -i "$img"
done
- name: Download Integration Tests Artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: integration-tests-${{ matrix.arch }}
- name: Extract Integration Tests Archive
run: tar -xzvf integration-tests-${{ matrix.arch }}.tar.gz
run: tar -xzf integration-tests-${{ matrix.arch }}.tar.gz
- name: Preload Images
# We download docker images used by integration tests so that all images are available
# locally and the download time doesn't account in the test execution time, which is subject
Expand All @@ -317,40 +328,113 @@ jobs:
done
}

retry docker pull minio/minio:RELEASE.2024-05-28T17-19-04Z
retry docker pull consul:1.8.4
retry docker pull quay.io/coreos/etcd:v3.5.29
retry docker pull -q minio/minio:RELEASE.2024-05-28T17-19-04Z
retry docker pull -q consul:1.8.4
retry docker pull -q quay.io/coreos/etcd:v3.5.29
if [ "$TEST_TAGS" = "integration_backward_compatibility" ]; then
retry docker pull quay.io/cortexproject/cortex:v1.16.1
retry docker pull quay.io/cortexproject/cortex:v1.17.2
retry docker pull quay.io/cortexproject/cortex:v1.18.1
retry docker pull quay.io/cortexproject/cortex:v1.19.1
retry docker pull quay.io/cortexproject/cortex:v1.20.1
retry docker pull quay.io/cortexproject/cortex:v1.21.0
retry docker pull quay.io/cortexproject/cortex:v1.21.1
retry docker pull -q quay.io/cortexproject/cortex:v1.16.1
retry docker pull -q quay.io/cortexproject/cortex:v1.17.2
retry docker pull -q quay.io/cortexproject/cortex:v1.18.1
retry docker pull -q quay.io/cortexproject/cortex:v1.19.1
retry docker pull -q quay.io/cortexproject/cortex:v1.20.1
retry docker pull -q quay.io/cortexproject/cortex:v1.21.0
retry docker pull -q quay.io/cortexproject/cortex:v1.21.1
elif [ "$TEST_TAGS" = "integration_query_fuzz" ]; then
retry docker pull quay.io/cortexproject/cortex:v$(cat testdata/VERSION)
retry docker pull quay.io/prometheus/prometheus:v3.9.1
retry docker pull -q quay.io/cortexproject/cortex:v$(cat testdata/VERSION)
retry docker pull -q quay.io/prometheus/prometheus:v3.9.1
elif [ "$TEST_TAGS" = "integration_configs_db" ]; then
retry docker pull postgres:9.6.16
retry docker pull -q postgres:9.6.16
fi
retry docker pull memcached:1.6.1
retry docker pull redis:7.0.4-alpine
retry docker pull -q memcached:1.6.1
retry docker pull -q redis:7.0.4-alpine
env:
TEST_TAGS: ${{ matrix.tags }}
- name: Integration Tests
# -test.timeout=2400s (40m) deliberately stays below this step's timeout, so a hung
# test is killed by Go rather than by the runner: its timeout panic then flows through
# as ordinary output inside the hanging test's group and still gets a FAIL result.
timeout-minutes: 45
run: |
set -o pipefail
export CORTEX_IMAGE_PREFIX="${IMAGE_PREFIX:-quay.io/cortexproject/}"
export IMAGE_TAG="${{ needs.build.outputs.image_tag }}"
export CORTEX_IMAGE="${CORTEX_IMAGE_PREFIX}cortex:${IMAGE_TAG}-${{ matrix.arch }}"
export CORTEX_IMAGE="${CORTEX_IMAGE_PREFIX}cortex:${IMAGE_TAG}-${TEST_ARCH}"
export CORTEX_CHECKOUT_DIR="$PWD/testdata"
PATTERN="$(cat bin/run-pattern-${{ matrix.tags }}.txt)"
echo "Running integration tests on ${{ matrix.arch }} with image: ${CORTEX_IMAGE}"
PATTERN="$(cat "bin/run-pattern-${TEST_TAGS}.txt")"
echo "Running integration tests on ${TEST_ARCH} with image: ${CORTEX_IMAGE}"
echo "Selecting tests via -test.run=${PATTERN}"
./bin/integration.test -test.timeout=2400s -test.v -test.count=1 -test.run="${PATTERN}"

RAW="integration-test2json-${TEST_ARCH}-${TEST_TAGS}.jsonl"

# -test.v=test2json (not plain -test.v) makes `testing` emit the framing markers
# test2json needs to attribute output — including the docker container logs the
# tests stream to stdout — to the test that produced it. bin/gha-testlog turns the
# resulting event stream into collapsible groups, a scannable result index,
# ::error:: annotations and a job summary. The raw stream is kept as the
# authoritative copy: containers are removed after each test, so this is the only
# surviving record of their logs.
#
# errexit is off around the pipeline so the raw stream is still archived, and the
# step's status still comes from the test binary, when tests fail.
set +e
./bin/test2json -t -p integration \
./bin/integration.test -test.timeout=2400s -test.count=1 -test.v=test2json -test.run="${PATTERN}" \
| tee "${RAW}" \
| ./bin/gha-testlog \
-shard "${TEST_ARCH} / ${TEST_TAGS}" \
-arch "${TEST_ARCH}" \
-tags "${TEST_TAGS}" \
-report "integration-report-${TEST_ARCH}-${TEST_TAGS}.json"
STATUS=("${PIPESTATUS[@]}")
set -e

gzip -f "${RAW}"

if [ "${STATUS[2]}" -ne 0 ]; then
echo "::warning::bin/gha-testlog exited ${STATUS[2]}; the rendering above may be incomplete. The uploaded raw test2json stream is authoritative."
fi
exit "${STATUS[0]}"
env:
IMAGE_PREFIX: ${{ secrets.IMAGE_PREFIX }}
TEST_ARCH: ${{ matrix.arch }}
TEST_TAGS: ${{ matrix.tags }}
- name: Upload Integration Test Logs
# Failure-only: an artifact per shard on every green run is clutter, and "no report"
# then cleanly means "this shard passed" to the integration-summary job.
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: integration-logs-${{ matrix.arch }}-${{ matrix.tags }}
path: |
integration-test2json-*.jsonl.gz
integration-report-*.json
if-no-files-found: ignore
retention-days: 7

integration-summary:
# Renders one table across every integration shard, so a red run can be triaged without
# opening 24 job logs one at a time. Reads the reports uploaded by failing shards; shards
# that pass upload nothing and are simply absent.
needs: [integration]
if: ${{ !cancelled() && needs.integration.result != 'skipped' }}
runs-on: ubuntu-24.04
steps:
- name: Checkout Workflow Scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
sparse-checkout: .github/workflows/scripts
- name: Download Integration Test Reports
# No artifacts at all is the green case, which download-artifact treats as an error.
continue-on-error: true
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: integration-logs-*
merge-multiple: true
path: reports
- name: Summarize Integration Tests
run: ./.github/workflows/scripts/summarize-integration-tests.sh reports
env:
INTEGRATION_RESULT: ${{ needs.integration.result }}

deploy:
needs: [build, test, lint, integration]
Expand Down
Loading