diff --git a/.gitignore b/.gitignore index af0730eb7fd..638227ff530 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,7 @@ website/.hugo_build.lock website/public website/resources website/content/en/docs +website/static/diagrams e2e_integration_test* active-query-tracker dist/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 4fe16f3a8bd..740ef55c728 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -54,6 +54,7 @@ * [ENHANCEMENT] Compactor: Reduce object storage GET calls when updating the bucket index by skipping re-reading parquet converter markers for blocks that already have a valid-version parquet entry in the previous index. #7669 * [ENHANCEMENT] Upgrade Thanos and promql-engine to latest. #7740 * [ENHANCEMENT] Ruler: Adjust ruler frontend decoder to not wrap query error messages with execution prefix, this makes error responses consistent between internal and external ruler paths. #7741 +* [ENHANCEMENT] Docs: Add an interactive architecture diagram at `/diagrams/cortex-architecture.html`, linked from the new Interactive Diagram documentation page. It replaces the static `images/architecture.png` embed in the architecture documentation and shows the protocol, endpoint, hash ring and source file behind each component and hop. #7769 * [BUGFIX] Querier: Fix queryWithRetry and labelsWithRetry returning (nil, nil) on cancelled context by propagating ctx.Err(). #7370 * [BUGFIX] Metrics Helper: Fix non-deterministic bucket order in merged histograms by sorting buckets after map iteration, matching Prometheus client library behavior. #7380 * [BUGFIX] Distributor: Return HTTP 401 Unauthorized when tenant ID resolution fails in the Prometheus Remote Write 2.0 path. #7389 diff --git a/VENDORED_CODE.md b/VENDORED_CODE.md index dd5991f7791..1a18f254224 100644 --- a/VENDORED_CODE.md +++ b/VENDORED_CODE.md @@ -16,3 +16,10 @@ in the ./vendor/ directory: [One file used in tests](COPYING.LGPL-3) is under LGPL-3, that's why we ship the license text in this repository. + +Outside of ./vendor/, [tools/diagram/d3.min.js](tools/diagram/d3.min.js) is an +unmodified copy of [D3](https://d3js.org/) 7.9.0, which is under the ISC license +(Copyright 2010-2023 Mike Bostock). It is vendored so the interactive +architecture diagram renders offline and the documentation site makes no +third-party request; see [tools/diagram/readme.md](tools/diagram/readme.md) for +its provenance. diff --git a/docs/architecture-diagram.md b/docs/architecture-diagram.md new file mode 100644 index 00000000000..65e8f552ded --- /dev/null +++ b/docs/architecture-diagram.md @@ -0,0 +1,34 @@ +--- +title: "Interactive Architecture Diagram" +linkTitle: "Interactive Diagram" +weight: 3 +slug: architecture-diagram +--- + +The [interactive architecture diagram](../tools/diagram/cortex-architecture.html) +draws the same system [Architecture](./architecture.md) describes in prose, with the +details attached to the picture instead of scattered through the text. It covers +the write path, the read path, the blocks lifecycle and the optional services — +ruler, alertmanager, compactor, store-gateway, query-scheduler and the caches. +Hover a connector and it names the protocol and the endpoint that hop actually +uses; select a component and it gives you the role, whether it is stateful, which +hash ring it joins, the endpoints it serves, its `-target` value and the file in +the Cortex tree that implements it. + +Three toggles cover the places where the topology genuinely forks, rather than +drawing one deployment and calling it typical: the query-frontend's own queue +versus a separate query-scheduler, the ruler evaluating rules in its own querier +stack versus delegating to the query-frontend with `-ruler.frontend-address`, and +the parquet queryable off versus on. There are also guided walkthroughs that step +through the write, read, rule-evaluation and blocks flows one hop at a time, a +table view of every component and flow, and a light/dark theme toggle. + +The diagram's metadata is hand-maintained against the Cortex source rather than +generated from it, so the `src` path shown in each component's panel is the +authority — if a ring key, prefix or endpoint disagrees with the code, the code is +right and the diagram needs fixing. It also deliberately shows a few things the +prose does not yet cover, such as the OTLP ingest endpoint and the +parquet-converter, which is marked experimental for that reason. Its source lives +in [`tools/diagram/`](https://github.com/cortexproject/cortex/tree/master/tools/diagram). + +**[Open the interactive architecture diagram →](../tools/diagram/cortex-architecture.html)** diff --git a/docs/architecture.md b/docs/architecture.md index b532d83239a..1b4f602a087 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,9 +7,7 @@ slug: architecture Cortex consists of multiple horizontally scalable microservices. Each microservice uses the most appropriate technique for horizontal scaling; most are stateless and can handle requests for any users while some (namely the [ingesters](#ingester)) are semi-stateful and depend on consistent hashing. This document provides a basic overview of Cortex's architecture. -The following diagram does not include all the Cortex services, but does represent a typical deployment topology. - -

+ The write path, read path and blocks lifecycle, as the code actually wires them up. + Hover a connector for its protocol and endpoint; select a component for its ring, + endpoints and source file. Nothing here is generated — the topology is + hand-maintained, so treat the source references in each panel as the authority. +
+0?d[i-1]:l,v.x1=i
0)for(i=0;i=n)&&(e=n);else{let r=-1;for(let i of t)null!=(i=n(i,++r,t))&&(e=i)&&(e=i)}return e}function tt(t,n){let e,r=-1,i=-1;if(void 0===n)for(const n of t)++i,null!=n&&(e =0;)(r=i[o])&&(a&&4^r.compareDocumentPosition(a)&&a.parentNode.insertBefore(r,a),a=r);return this},sort:function(t){function n(n,e){return n&&e?t(n.__data__,e.__data__):!n-!e}t||(t=un);for(var e=this._groups,r=e.length,i=new Array(r),o=0;o >1)+h+t+M+S.slice(A);break;default:t=S+h+t+M}return u(t)}return y=void 0===y?6:/[gprs]/.test(_)?Math.max(1,Math.min(21,y)):Math.max(0,Math.min(20,y)),M.toString=function(){return t+""},M}return{format:l,formatPrefix:function(t,n){var e=l(((t=Jc(t)).type="f",t)),r=3*Math.max(-8,Math.min(8,Math.floor(Zc(n)/3))),i=Math.pow(10,-r),o=uf[8+r/3];return function(t){return e(i*t)+o}}}}function ff(n){return of=cf(n),t.format=of.format,t.formatPrefix=of.formatPrefix,of}function sf(t){return Math.max(0,-Zc(Math.abs(t)))}function lf(t,n){return Math.max(0,3*Math.max(-8,Math.min(8,Math.floor(Zc(n)/3)))-Zc(Math.abs(t)))}function hf(t,n){return t=Math.abs(t),n=Math.abs(n)-t,Math.max(0,Zc(n)-Zc(t))+1}t.format=void 0,t.formatPrefix=void 0,ff({thousands:",",grouping:[3],currency:["$",""]});var df=1e-6,pf=1e-12,gf=Math.PI,yf=gf/2,vf=gf/4,_f=2*gf,bf=180/gf,mf=gf/180,xf=Math.abs,wf=Math.atan,Mf=Math.atan2,Tf=Math.cos,Af=Math.ceil,Sf=Math.exp,Ef=Math.hypot,Nf=Math.log,kf=Math.pow,Cf=Math.sin,Pf=Math.sign||function(t){return t>0?1:t<0?-1:0},zf=Math.sqrt,$f=Math.tan;function Df(t){return t>1?0:t<-1?gf:Math.acos(t)}function Rf(t){return t>1?yf:t<-1?-yf:Math.asin(t)}function Ff(t){return(t=Cf(t/2))*t}function qf(){}function Uf(t,n){t&&Of.hasOwnProperty(t.type)&&Of[t.type](t,n)}var If={Feature:function(t,n){Uf(t.geometry,n)},FeatureCollection:function(t,n){for(var e=t.features,r=-1,i=e.length;++r=0?1:-1,i=r*e,o=Tf(n=(n*=mf)/2+vf),a=Cf(n),u=Vf*a,c=Gf*o+u*Tf(i),f=u*r*Cf(i);as.add(Mf(f,c)),Xf=t,Gf=o,Vf=a}function ds(t){return[Mf(t[1],t[0]),Rf(t[2])]}function ps(t){var n=t[0],e=t[1],r=Tf(e);return[r*Tf(n),r*Cf(n),Cf(e)]}function gs(t,n){return t[0]*n[0]+t[1]*n[1]+t[2]*n[2]}function ys(t,n){return[t[1]*n[2]-t[2]*n[1],t[2]*n[0]-t[0]*n[2],t[0]*n[1]-t[1]*n[0]]}function vs(t,n){t[0]+=n[0],t[1]+=n[1],t[2]+=n[2]}function _s(t,n){return[t[0]*n,t[1]*n,t[2]*n]}function bs(t){var n=zf(t[0]*t[0]+t[1]*t[1]+t[2]*t[2]);t[0]/=n,t[1]/=n,t[2]/=n}var ms,xs,ws,Ms,Ts,As,Ss,Es,Ns,ks,Cs,Ps,zs,$s,Ds,Rs,Fs={point:qs,lineStart:Is,lineEnd:Os,polygonStart:function(){Fs.point=Bs,Fs.lineStart=Ys,Fs.lineEnd=Ls,rs=new T,cs.polygonStart()},polygonEnd:function(){cs.polygonEnd(),Fs.point=qs,Fs.lineStart=Is,Fs.lineEnd=Os,as<0?(Wf=-(Kf=180),Zf=-(Qf=90)):rs>df?Qf=90:rs<-df&&(Zf=-90),os[0]=Wf,os[1]=Kf},sphere:function(){Wf=-(Kf=180),Zf=-(Qf=90)}};function qs(t,n){is.push(os=[Wf=t,Kf=t]),n t.r&&(t.r=t[n].r)}function c(){if(n){var r,i,o=n.length;for(e=new Array(o),r=0;r >>1;f[g]1;)i-=2;for(let t=2;t0){if(n>=this.ymax)return null;(i=(this.ymax-n)/r)0){if(t>=this.xmax)return null;(i=(this.xmax-t)/e)this.xmax?2:0)|(n9999?"+"+Ku(n,6):Ku(n,4))+"-"+Ku(t.getUTCMonth()+1,2)+"-"+Ku(t.getUTCDate(),2)+(o?"T"+Ku(e,2)+":"+Ku(r,2)+":"+Ku(i,2)+"."+Ku(o,3)+"Z":i?"T"+Ku(e,2)+":"+Ku(r,2)+":"+Ku(i,2)+"Z":r||e?"T"+Ku(e,2)+":"+Ku(r,2)+"Z":"")}function Ju(t){var n=new RegExp('["'+t+"\n\r]"),e=t.charCodeAt(0);function r(t,n){var r,i=[],o=t.length,a=0,u=0,c=o<=0,f=!1;function s(){if(c)return Hu;if(f)return f=!1,ju;var n,r,i=a;if(t.charCodeAt(i)===Xu){for(;a++=v)<<1|t>=y)&&(c=p[p.length-1],p[p.length-1]=p[p.length-1-f],p[p.length-1-f]=c)}else{var _=t-+this._x.call(null,g.data),b=n-+this._y.call(null,g.data),m=_*_+b*b;if(m=u)){(t.data!==n||t.next)&&(0===l&&(p+=(l=Uc(e))*l),0===h&&(p+=(h=Uc(e))*h),p(t=(Lc*t+jc)%Hc)/Hc}();function l(){h(),f.call("tick",n),e1?(f.on(t,e),n):f.on(t)}}},t.forceX=function(t){var n,e,r,i=qc(.1);function o(t){for(var i,o=0,a=n.length;o=.12&&i<.234&&r>=-.425&&r<-.214?u:i>=.166&&i<.234&&r>=-.214&&r<-.115?c:a).invert(t)},s.stream=function(e){return t&&n===e?t:(r=[a.stream(n=e),u.stream(e),c.stream(e)],i=r.length,t={point:function(t,n){for(var e=-1;++ejs(r[0],r[1])&&(r[1]=i[1]),js(i[0],r[1])>js(r[0],r[1])&&(r[0]=i[0])):o.push(r=i);for(a=-1/0,n=0,r=o[e=o.length-1];n<=e;r=i,++n)i=o[n],(u=js(r[1],i[0]))>a&&(a=u,Wf=i[0],Kf=r[1])}return is=os=null,Wf===1/0||Zf===1/0?[[NaN,NaN],[NaN,NaN]]:[[Wf,Zf],[Kf,Qf]]},t.geoCentroid=function(t){ms=xs=ws=Ms=Ts=As=Ss=Es=0,Ns=new T,ks=new T,Cs=new T,Lf(t,Gs);var n=+Ns,e=+ks,r=+Cs,i=Ef(n,e,r);return i`; node positions are deliberately hand-placed rather
+than force-directed, so the reading order survives a reload. Two things to keep
+in mind when changing it:
+
+- **The three path colours are not free choices.** They are documented
+ categorical palette slots, validated for colour-vision deficiency on the
+ all-pairs pairlist in both light and dark mode. A node-link diagram lets any
+ two marks sit adjacent, and only three-hue subsets clear that gate — adding a
+ fourth hued category fails it. The aqua slot is below 3:1 on the light
+ surface, which is why the table view ships as part of the page: it is the
+ relief channel, not an extra.
+- **The metadata is hand-maintained**, so the `src` path on each node is the
+ authority. If you change a ring key, prefix or endpoint, check it against the
+ code first.
+
+### Where it is published
+
+`tools/diagram/` is the single source of truth.
+[`tools/website/web-pre.sh`](../website/web-pre.sh) copies the page and its D3
+copy into `website/static/diagrams/` at build time, so the page is served at
+`/diagrams/cortex-architecture.html`. The `