Skip to content

Stop reporting a failed comparison as a clean result - #199

Open
reuvenharrison wants to merge 1 commit into
mainfrom
fix/comparison-failure-not-reported-as-clean
Open

Stop reporting a failed comparison as a clean result#199
reuvenharrison wants to merge 1 commit into
mainfrom
fix/comparison-failure-not-reported-as-clean

Conversation

@reuvenharrison

Copy link
Copy Markdown
Contributor

When oasdiff cannot load or parse a spec (exit code >= 2), the breaking and diff actions printed the error annotation and then carried straight on to write No breaking changes / No changes.

The step still failed, so CI caught it, but everything a person actually reads described the run as clean: the step output, the job summary, and the PR comment. It is also wrong for anyone using continue-on-error.

The fix

Exit as soon as the comparison fails. This is what the changelog action already does:

if [ "$exit_code" -ge 2 ]; then
    [ -s "$_err" ] && echo "::error::$(tr '\n' ' ' < "$_err")"
    if [ "$exit_code" -eq 123 ]; then
        echo "::error::…allow-external-refs remedy…"
    fi
    rm -f "$_err"
    exit "$exit_code"
fi

breaking and diff were simply missing the early exit. Nothing downstream has anything to report about a comparison that never ran, and stopping there keeps the existing 123 remedy message as the most specific thing the user sees rather than burying it under a generic failure notice.

Tests

One regression job per action, using a spec whose $ref cannot be resolved (exit 102). Each asserts the step fails and that the output never contains the clean-result text, so a future refactor cannot reintroduce the fallthrough.

Credit

Found and reported by @lucas-monteiro-g in #198, along with the observation that the failure has to stay observable in a fixture. That PR also added comparison-status and comparison-error outputs; those are left out here since the step's own failure already signals the condition and new outputs are permanent API.

When oasdiff cannot load or parse a spec (exit code >= 2), the breaking and
diff actions printed the error annotation and then carried on to write
'No breaking changes' / 'No changes'. The step still failed, so CI caught
it, but the output, job summary and PR comment all described the run as
clean, which is misleading for anyone reading them or using
continue-on-error.

Exit as soon as the comparison fails, which is what the changelog action
already does. Nothing downstream has anything to report about a comparison
that never ran.

Adds a regression test per action using a spec with an unresolvable $ref,
asserting the step fails and never emits the clean-result text.

Reported by @lucas-monteiro-g in #198.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant