Publish an interactive architecture diagram on cortexmetrics.io - #7769
Open
CharlieTLe wants to merge 1 commit into
Open
Publish an interactive architecture diagram on cortexmetrics.io#7769CharlieTLe wants to merge 1 commit into
CharlieTLe wants to merge 1 commit into
Conversation
The only editable architecture diagram source in the repo, tools/diagram/cortex-architecture.drawio, dates from 2019 and depicts the chunks-storage architecture that was removed years ago: a Table Manager, BigTable/Cassandra/DynamoDB, and no store-gateway, compactor or query-scheduler. The images/architecture.png that superseded it and that docs/architecture.md embeds has no source at all, so nobody can update it either. Meanwhile the facts a reader wants from a diagram - which ring a component uses, what protocol a hop speaks, which endpoint it hits, whether the component is stateful - live only in the prose beside it. Add tools/diagram/cortex-architecture.html: a self-contained D3 node-link diagram of the write path, read path, blocks lifecycle and optional services, with the protocol and endpoint on every connector and the ring, endpoints, -target value and source file on every component. It has toggles for the three places the topology genuinely forks - the query-scheduler versus the query-frontend's own queue, the ruler's own querier stack versus -ruler.frontend-address, and the parquet queryable off versus on - plus guided walkthroughs of the write, read, rule-evaluation and blocks flows, a table view and a light/dark toggle. The page is served two ways. docs/architecture-diagram.md is a thin sidebar page, so it lands in the docs nav and the offline search index; /diagrams/cortex-architecture.html is the full-bleed page it links to. tools/website/web-pre.sh copies both files from tools/diagram/ into website/static/diagrams/ at build time, keeping a single source of truth, and tools/website/website.go rewrites the repo-relative link so one link works both on GitHub and on the site. D3 7.9.0 is vendored rather than fetched from a CDN so an official docs page makes no third-party request and the page renders with no network access. The bytes are the published dist/d3.min.js, matching the jsDelivr SRI hash for that version; provenance and license are recorded in tools/diagram/readme.md and VENDORED_CODE.md. docs/architecture.md now points at the interactive diagram instead of embedding images/architecture.png. The PNG itself stays in images/ because external pages hotlink cortexmetrics.io/images/architecture.png. The diagram's metadata is hand-maintained, not generated. Every src path, -target value, ring KV prefix and key, ring status page, gRPC method and HTTP route in it was checked against the tree. Signed-off-by: Charlie Le <charlie_le@apple.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Replaces the project's unmaintainable architecture picture with an interactive diagram published on cortexmetrics.io.
The problem
The only editable diagram source in the repo,
tools/diagram/cortex-architecture.drawio, dates from 2019 and depicts the chunks-storage architecture that was removed years ago — a Table Manager, BigTable/Cassandra/DynamoDB, separate index and chunk memcaches, and no store-gateway, compactor or query-scheduler. It has been wrong for long enough that nobody looks at it.What
docs/architecture.mdactually embeds isimages/architecture.png, which is newer but was not generated from that.drawioand has no source at all. So the diagram the docs show cannot be updated by anyone: adding the query-scheduler, or the OTLP ingest endpoint, or the parquet-converter means redrawing a PNG by hand.Separately, the facts a reader wants from an architecture diagram — which hash ring a component joins, what protocol a given hop speaks, which endpoint it hits, whether the component is stateful — are in the ~2,700 words of prose next to the picture, not in the picture.
What's added
tools/diagram/cortex-architecture.html, a self-contained D3 node-link diagram of the write path, read path, blocks lifecycle and optional services. Hover a connector and it names the protocol and endpoint for that hop; select a component and it gives the role, statefulness, ring KV prefix and key, ring status page, served endpoints,-targetvalue and the source file that implements it.It has toggles for the three places the topology genuinely forks, instead of drawing one deployment and calling it typical:
-ruler.frontend-addressPlus guided walkthroughs of the write, read, rule-evaluation and blocks flows, a table view of every component and flow, and a light/dark toggle.
All seven caches are drawn separately because no two share a consumer set — the results cache is the query-frontend's, the index cache the store-gateway's alone, the chunks cache the store-gateway's and querier's, the metadata cache those two plus the compactor, the two parquet caches only with the parquet queryable enabled, and the expanded postings cache is in-process in the ingester under
blocks-storage.tsdb.*.How it's published
Two URLs:
/docs/architecture-diagram/— a thin sidebar page (docs/architecture-diagram.md), so the diagram lands in the docs nav and the Lunr offline search index without being squeezed into Docsy's narrow content column. It sits at weight 3, directly under Architecture;docs/blocks-storage/_index.mdmoves from weight 3 to the previously-unused 4, so nothing else in the nav shifts./diagrams/cortex-architecture.html— the full-bleed page.tools/diagram/stays the single source of truth.tools/website/web-pre.shcopies the page and its D3 intowebsite/static/diagrams/at build time, exactly the pattern it already uses forcp images/* website/static/images. Static files bypass Hugo's pipeline, sohugo --minifydoes not touch the page (verified: the built output is byte-identical to the source).tools/website/website.gogains one rewrite next to the existingimagesone, becauseconvertLinksleaves any link without.mdalone and a repo-relative link to the HTML would 404 on the site. That keeps a single link correct in both renderings — repo-relative on GitHub,/diagrams/…on the site.Why
d3.min.jsis committedThe alternative was a pinned CDN
<script>with a subresource-integrity hash, which is what the page used before this PR. I vendored it instead so that:file://.It is the unmodified published
dist/d3.min.jsfor d3 7.9.0 — 279,706 bytes, ISC (Copyright 2010-2023 Mike Bostock). Its SHA-384 matches the jsDelivr SRI hash for that exact version,sha384-CjloA8y00+1SDAUkjs099PVfnY2KmDC2BZnws9kh8D/lX1s46w6EPhpXdqMfjK6i, so the provenance is checkable. Version, size, hash and upstream URL are recorded intools/diagram/readme.md, and there is an entry inVENDORED_CODE.md(as a separate paragraph — that file otherwise only covers./vendor/Go deps).The
<script>tag deliberately carries nointegrity/crossorigin: underfile://the response has an opaque origin, so SRI would block the script outright rather than skip the check, and on the site the fetch is same-origin, which makes SRI redundant.images/architecture.pngdocs/architecture.mdnow links to the interactive diagram instead of embedding the PNG. Two consequences I'm taking deliberately:docs/architecture.mdno longer has an inline diagram.The file stays in
images/rather than being deleted, because external blog posts hotlinkcortexmetrics.io/images/architecture.pngand breaking those costs more than the cleanup saves.Verification
file://with no console errors and no failed requests; all three toggles, the flow walkthroughs, the table view and the theme toggle exercised../tools/website/web-pre.sh+ a fullhugo --minifybuild:/diagrams/cortex-architecture.htmland/docs/architecture-diagram/both serve, the link between them resolves, the sidebar reads Architecture → Interactive Diagram → Blocks Storage,/docs/architecture/no longer shows the PNG, and the new page is inoffline-search-index.json.git statusclean after a website build (website/static/diagramsis build output and is gitignored).make check-white-noiseandmake check-docboth pass with the tree clean afterwards.The metadata in the diagram is hand-maintained against the tree rather than generated, so I checked all of it before opening this: 10/10
srcpaths exist, 10/10-targetvalues match the constants inpkg/cortex/modules.go, and all 7 ring prefix/key pairs, 7 ring status pages, gRPC method names and HTTP routes matchpkg/*/[*]_ring.go, the.protofiles andpkg/api/api.go. Thesrcpath shown in each panel is documented as the authority precisely because this metadata can drift.AI usage disclosure
Per §3 of
GENAI_POLICY.md: this contribution was developed with substantial AI assistance (Claude Code) — the diagram page, the documentation page and the build wiring were largely AI-drafted. I have reviewed all of it, and per §1 I checked every factual claim the diagram makes (ring keys and KV prefixes, endpoints,-targetvalues, source paths) against the Cortex source rather than trusting the generated metadata; the audit is summarised above.d3.min.jsis third-party ISC-licensed code, vendored unmodified and attributed inVENDORED_CODE.md.Which issue(s) this PR fixes:
N/A
Checklist
CHANGELOG.mdupdateddocs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags — n/a, no new flagsFollow-up, deliberately not in this PR
A geometry audit (overlapping boxes, labels wider than their box, connectors crossing unrelated components, dangling references, every flow step under every mode combination) and a jsdom interaction test exist for this page. I ran both against the file in this PR: the interaction suite passes 73/73, and the audit is clean apart from two checks that this PR makes obsolete by design — it asserts a pinned CDN
<script>tag with anintegrityattribute and compares that hash againstnode_modules/d3, which needs to become "the vendored bytes are d3 7.9.0" instead.Porting them needs a
tools/diagram/package.json+ lockfile and a CI job. Node is already installed in.github/workflows/web.ymlfor the website build, so there is a natural place to hang it. Keeping that out of this PR keeps the diff reviewable; happy to fold it in here instead if maintainers prefer.