fix(api): retain flushed new-chain queue rows instead of deleting them

[rickyrombo]

Why

The flusher deletes each row after forwarding it (new_chain_flusher.go — DELETE FROM new_chain_queue WHERE id = $1). That is unrecoverable, and it collides with a fact about the migration: the genesis chain gets regenerated before it ships.

The CometBFT validator key is hashed into every block header (ValidatorsHash, plus the proposer address and chain ID), so the validator set cannot be swapped by editing genesis — the chain has to be rebuilt with a key the bootstrap node actually holds. Any write already forwarded and deleted would then survive only on the chain being discarded.

Queueing is already enabled in production, so this wants to land before flushing is turned on.

What

Mark rows instead of deleting them. A forwarded row gets flushed_at set and stays put, making the queue a durable log — repoint the flusher at the rebuilt chain, set NewChainFlushFromBlock to the new backfill's end height, and re-drive. Replay is idempotent: the same signed transactions are rejected as duplicates by a chain that already has them, and accepted by a rebuilt one.

The two non-forwarding paths are marked for the same reason and record why in skip_reason:

path	before	after
forwarded successfully	DELETE	flushed_at = now()
covered by genesis backfill	DELETE	flushed_at = now(), skip_reason = 'backfilled'
corrupt tx_data	DELETE	flushed_at = now(), skip_reason = 'corrupt'

Corrupt rows previously took the evidence with them; they are now inspectable.

trimBackfillRows only touches pending rows, so re-running it with a higher flush-from block after a regeneration marks the newly-covered rows without disturbing the record of what was already sent.

Ordering and performance

Unchanged in the steady state. The flusher still processes strictly in id order, one at a time, and still refuses to advance past a failed row — a failure leaves flushed_at NULL, so the next fetch returns that same row first.

Reads select only pending rows, and a partial index (ON new_chain_queue (id) WHERE flushed_at IS NULL) keeps retained rows out of the index entirely, so the hot path does not degrade as the table grows.

Tradeoff

The table no longer self-truncates. Retention becomes an explicit operator decision rather than a side effect of flushing — deliberate here, since the whole point is not to lose rows, but worth a purge step once the migration is finished and rollback is off the table.

Migration

0238_new_chain_queue_cursor.sql — additive only: two nullable columns and one partial index. Existing rows default to flushed_at IS NULL, i.e. pending, which is correct given flushing has never run.

Testing

go test ./api/ -run 'NewChainFlusher|EnqueueForNewChain' — 5 passed.

Existing tests updated for retention semantics (queueDepth split into pendingDepth / totalDepth, plus a skipReasons helper), and each now asserts that rows survive rather than vanish.

New TestNewChainFlusherRedriveAfterRegeneration covers the capability this PR exists for: drain the queue against one chain, clear flushed_at as an operator would when repointing at the rebuilt chain, and confirm the same transactions forward again. Under delete-on-success there would be nothing left to re-drive.

🤖 Generated with Claude Code