Stop reporting a failed comparison as a clean result - #199
Open
reuvenharrison wants to merge 1 commit into
Open
Conversation
When oasdiff cannot load or parse a spec (exit code >= 2), the breaking and diff actions printed the error annotation and then carried on to write 'No breaking changes' / 'No changes'. The step still failed, so CI caught it, but the output, job summary and PR comment all described the run as clean, which is misleading for anyone reading them or using continue-on-error. Exit as soon as the comparison fails, which is what the changelog action already does. Nothing downstream has anything to report about a comparison that never ran. Adds a regression test per action using a spec with an unresolvable $ref, asserting the step fails and never emits the clean-result text. Reported by @lucas-monteiro-g in #198.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When oasdiff cannot load or parse a spec (exit code >= 2), the
breakinganddiffactions printed the error annotation and then carried straight on to writeNo breaking changes/No changes.The step still failed, so CI caught it, but everything a person actually reads described the run as clean: the step output, the job summary, and the PR comment. It is also wrong for anyone using
continue-on-error.The fix
Exit as soon as the comparison fails. This is what the
changelogaction already does:breakinganddiffwere simply missing the early exit. Nothing downstream has anything to report about a comparison that never ran, and stopping there keeps the existing 123 remedy message as the most specific thing the user sees rather than burying it under a generic failure notice.Tests
One regression job per action, using a spec whose
$refcannot be resolved (exit 102). Each asserts the step fails and that the output never contains the clean-result text, so a future refactor cannot reintroduce the fallthrough.Credit
Found and reported by @lucas-monteiro-g in #198, along with the observation that the failure has to stay observable in a fixture. That PR also added
comparison-statusandcomparison-erroroutputs; those are left out here since the step's own failure already signals the condition and new outputs are permanent API.