fix(api): mark dead content hosts - #1017
Conversation
Refusing connections since at least 2026-08-11 (`curl: (7) Failed to connect ... port 443`) while still registered on chain as validator Id 99, so rendezvous kept ranking it — it came back as the primary host on 9 of 25 sampled requests for one artist's cover photo, and ~8% of sampled trending artists have it in their host set. Nothing is broken by this: useImageSize walks artwork.mirrors on a failed preload, and StoreAllNodes is always appended to the candidate set, so clients recover transparently. This just stops those image loads from spending a failed request on a dead host first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
raymondjacobson
left a comment
There was a problem hiding this comment.
Good for now. I think there's an underlying issue though if a down node can cause this!
Diffing the eth registry (76) against core's non-jailed validator set (70) turns up six nodes core has already concluded are gone. All six fail to serve content: content.grassfed.network connection refused (already listed) audius.zeogrid.com connection refused cn0.mainnet.audiusindex.org 521 on health + content cn3.mainnet.audiusindex.org 521 on health + content cn4.mainnet.audiusindex.org 521 on health + content audius-nodes.com health_check 200, 502 on every content CID That last one is the interesting case: it passes a health check while serving nothing, so the active-probing approach in #1016 would have waved it through. Core's jailed flag caught it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Agreed, and I think I found it — the network already knows. I've pushed a second commit widening this PR off the back of it, so worth a re-look. The signal already exists. Right now those two disagree by six nodes: Every one of the six fails to serve content:
Two things fall out of that:
So the durable fix is probably sourcing the rendezvous set from core's non-jailed view rather than the raw eth registry — no new probing machinery, no hand-maintained list, and strictly better signal than what I was proposing. One caveat before anyone acts on that: jailing tracks core consensus participation, not content serving. The two correlate perfectly in this sample (6/6), but they're not the same property — a node could plausibly serve content fine while jailed for a consensus reason, and we'd be dropping a healthy replica. Low blast radius given mirrors + This PR stays the stopgap. Happy to open the core-sourced one if you think it's the right direction. |
Replaces #1016 — @raymondjacobson was right that the health-checking approach there was over-engineered, since
StoreAllNodesplus client-side mirror fallback already cover the failure. Details in the comment on that PR.audius.zeogrid.comis refusing connections but is still registered on chain as validator Id 99, so rendezvous keeps ranking it:It came back as the primary host on 9 of 25 sampled requests for one artist's cover photo, and ~8% of sampled trending artists have it in their host set.
Nothing is user-visibly broken by this.
useImageSizewalksartwork.mirrorswhenpreloadrejects, andSelect()always appendsStoreAllNodes, so there's guaranteed to be a live mirror. This one-liner just stops those image loads from spending a failed request on a dead host before recovering — using the mechanism that already exists for exactly this.🤖 Generated with Claude Code