Skip to content

docs: match the adopter claim to what was verified, and guard the compact samples - #11

Merged
shenxianpeng merged 3 commits into
mainfrom
claude/refresh-sample-output-602anc
Aug 5, 2026
Merged

docs: match the adopter claim to what was verified, and guard the compact samples#11
shenxianpeng merged 3 commits into
mainfrom
claude/refresh-sample-output-602anc

Conversation

@shenxianpeng

@shenxianpeng shenxianpeng commented Aug 5, 2026

Copy link
Copy Markdown
Member

What

The last item from the structural review, plus a hole it exposed in the anti-drift tests.

The adopter claim said more than was checked

The organizations were verified by hand — these orgs have repositories that run Commit Check. Which repositories was not pinned down.

The wording went a step further than that:

Trusted by developers worldwide

Used by developers and organizations worldwide in their production workflows.

A repository can be a demo, a template or an experiment, so "in their production workflows" claimed something that was not established. It was also the least load-bearing part of the sentence — eighteen logos carry the point without it.

What was established is the better claim anyway, because a reader can check it: the dependents graph is one click away, whereas "production" is not falsifiable.

Used by

Commit Check runs in repositories across these organizations, and in many more.

The dependents link moves into the sentence rather than dangling after the logo grid, and dropping "worldwide" from the heading stops it and the sentence repeating the same two words back to back. Confirmed nothing linked to the old anchor; all 18 logos and the dependents link are intact.

Nothing was checking the --compact samples

The two text formats spell a check name differently:

Format Prints Example
Default the kebab-case name CC003 subject-imperative check failed ==>
--compact the config key [FAIL] CC003 subject_imperative:

A sample of one therefore cannot be validated against the other, and _SAMPLE_FAILURE only matches the default format. So every --compact sample on the site was covered by nothing.

That is the same blind spot that let a pre-2.13 sample sit unnoticed in the troubleshooting page — stale in a format the guard could not see, with every test still passing. Fixing that one sample did not close the hole it came through.

Two samples were uncovered, in example.md and rules.md. One is indented inside a content tab and one is not, so the pattern allows leading whitespace — worth noting because a stricter first attempt at counting them found only one of the two.

Verified by breaking one deliberately rather than by trusting a green run:

example.md: CC003 shown as 'subject-imperative', --compact prints 'subject_imperative'

It names the file and both spellings, rather than only going red.

This also settles what happens if commit-check/commit-check#528 reconciles the two formats upstream: today that would quietly leave both samples wrong; now docs-sync fails and says which lines to rewrite.

Test plan

  • mkdocs build --strict clean
  • 9 tests pass, up from 8
  • New guard verified to fail on a deliberately stale sample and to pass once restored
  • Logo count and dependents link checked in the built output

Note

The proxy here blocks external hosts, so I could not open the deploy preview; everything was checked against a local build.

This closes the review list. The only open item left from it is upstream: commit-check/commit-check#528.


Generated by Claude Code

Summary by CodeRabbit

  • Documentation
    • Updated the “Trusted by developers worldwide” section to “Used by” with refreshed introductory text.
    • Removed the separate “And many more” line.
    • Added validation to ensure documented compact failure examples match the tool’s displayed configuration keys.

claude added 2 commits August 5, 2026 18:26
The claim was "Used by developers and organizations worldwide in their
production workflows", under a heading reading "Trusted by developers
worldwide".

The organizations were verified by hand, at organization level — these
orgs have repositories that run Commit Check. Which repositories was not
pinned down, so "in their production workflows" claimed more than that:
a repository can be a demo, a template or an experiment. It was also the
least load-bearing part of the sentence, since eighteen logos carry the
point on their own.

What was verified is the stronger claim anyway, because a reader can
check it — the dependents graph is right there, and "production" is not
falsifiable. So the section says that instead, and the dependents link
moves into the sentence rather than dangling after the logo grid.

Dropping "worldwide" from the heading also stops it and the sentence
below repeating the same two words back to back.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
The two text formats spell a check name differently — the default output
prints the kebab-case name, --compact prints the config key — so a sample
of one cannot be validated against the other. The existing guard only
matches the default format, which meant every --compact sample on the
site was checked by nothing at all.

That is the same blind spot that let a pre-2.13 sample sit unnoticed in
the troubleshooting page: the sample was stale in a format the guard
could not see, and every test still passed. Fixing that one sample did
not close the hole it came through.

Two samples were uncovered, in example.md and rules.md — one indented
inside a content tab, one not, so the pattern allows leading whitespace.
Verified by breaking one deliberately: the guard names the file and both
spellings, rather than only going red.

This also decides what happens if commit-check#528 reconciles the two
formats upstream. Today that would quietly leave both samples wrong;
now docs-sync fails and says which lines to rewrite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
@netlify

netlify Bot commented Aug 5, 2026

Copy link
Copy Markdown

Deploy Preview for commit-check ready!

Name Link
🔨 Latest commit 936d8cb
🔍 Latest deploy log https://app.netlify.com/projects/commit-check/deploys/6a738ae80974cf000888b9e5
😎 Deploy Preview https://deploy-preview-11--commit-check.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@shenxianpeng, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 55 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 56ca826b-d1f9-4116-94ea-5adda02c4481

📥 Commits

Reviewing files that changed from the base of the PR and between 2d2cd17 and 936d8cb.

📒 Files selected for processing (1)
  • tests/docs_sync_test.py
📝 Walkthrough

Walkthrough

The pull request updates the dependents section in docs/index.md. It also adds a documentation test that validates compact failure output keys against ALL_RULES.

Changes

Documentation and validation

Layer / File(s) Summary
Update dependents section
docs/index.md
Renames “Trusted by developers worldwide” to “Used by”, updates the introductory text, and removes the separate “And many more” link.
Validate compact output samples
tests/docs_sync_test.py
Detects compact failure samples in Markdown files and verifies each printed configuration key against the corresponding rule’s entry.check value.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes both main changes: narrowing the adopter claim and adding validation for compact samples.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/refresh-sample-output-602anc

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/docs_sync_test.py`:
- Around line 103-108: Update the documentation scan in the test loop over
_COMPACT_FAILURE matches to explicitly record or fail on rule IDs absent from
by_id, rather than skipping them via if entry. Emit a diagnostic identifying the
unknown rule ID, then only compare printed with entry.check when the rule
exists, preserving the existing stale-check behavior.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 0d23529b-4d8d-4e52-b340-42803490a52d

📥 Commits

Reviewing files that changed from the base of the PR and between d5cf040 and 2d2cd17.

📒 Files selected for processing (2)
  • docs/index.md
  • tests/docs_sync_test.py

Comment thread tests/docs_sync_test.py Outdated
Both sample guards looked up the rule ID and then wrote `if entry and
...`, so an ID the package does not define fell through the condition
and the sample passed. A typo, or an ID retired upstream, could sit in
the docs with every test green — which is the exact failure these guards
exist to catch, so it is the one they must not wave through.

Review raised this against the compact guard, because that is what was
in the diff. The older guard had the same line, so fixing only the new
one would have left the hole in the more established of the two and made
the pair inconsistent. Both now share one helper rather than the fix
being written twice.

Verified by breaking each case in turn: a stale name in the default
format, a stale name in --compact, and CC999 in place of a real ID. All
three now name the file and the problem; the third previously passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn

Copy link
Copy Markdown
Member Author

Good catch — fixed in 936d8cb, and it applied more widely than the comment could see.

if entry and ... meant an unrecognised rule ID fell through the condition and the sample passed. For a guard whose whole purpose is catching stale samples, silently ignoring the ID it cannot resolve is the worst case to let through.

The older guard had the same line. It was flagged only on _COMPACT_FAILURE because that is what appeared in the diff, but test_sample_output_matches_what_the_tool_prints was written the same way at line 83. Fixing just the new one would have left the hole in the more established of the two and made the pair behave differently for no reason, so both now go through one helper instead of the fix being written twice — which also removes a loop that was duplicated apart from which attribute it compares (entry.name for the default format, entry.check for --compact).

Verified by breaking each case rather than by a green run:

Broken sample Result
CC003 subject_imperative check failed ==> in rules.md rules.md: CC003 shown as 'subject_imperative', the tool prints 'subject-imperative'
[FAIL] CC003 subject-imperative: in example.md example.md: CC003 shown as 'subject-imperative', --compact prints 'subject_imperative'
[FAIL] CC999 subject_imperative: in example.md example.md: CC999 is not a rule the package defines

The third is the case you raised; it passed before this change.

One note on the tooling output attached to the comment: the ast-grep xpath-injection-python warning on _COMPACT_FAILURE.findall(page.read_text("utf-8")) is a false positive. That is re.Pattern.findall over a Markdown file in the repository — no XPath, no XML, and no input from outside the checkout. Nothing changed for it.


Generated by Claude Code

@shenxianpeng
shenxianpeng merged commit dc84741 into main Aug 5, 2026
8 checks passed
@shenxianpeng
shenxianpeng deleted the claude/refresh-sample-output-602anc branch August 5, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants