Skip to main content

Indexer chain parked

The indexer stops one chain after a failed chain reorganisation repair: recognise a parked chain, heal it with a restart, and know when it needs a rebuild.

One chain has stopped advancing while the indexer pod stays Ready, because the indexer parked it after a chain reorganisation repair it refused. Read the fork block from the log and compare it with the chain's rollup_block and finality_block: a restart heals the park when the fork is higher than both, and a rebuild is the only fix when it is not. For a chain that is only behind, or for a pod that restarts, start with Ledger indexing stalled instead.

Each indexer loop advances one chain. When a loop hits a failure that a retry cannot fix, the indexer parks that chain and keeps the other chains running. A parked chain stops advancing, so Ledger Data for it freezes at the last indexed block while the network moves on.

A chain reorganisation is a change in the chain's recent blocks. The indexer keeps undo data for every block above the rollup_block of the chain's row in idxr_chain_watermark. It can rewind to any block above rollup_block and above finality_block. It cannot rewind below them.

The park lives in memory only. A restart of the indexer clears it.

Symptoms

  • A chain stops advancing and the pod stays Ready. The probes answer for the process, not for chain state.
  • The indexer log holds the line EVM span chain parked; its siblings continue.
  • The failure the line names has the tag repair. The failure text reads Repair stopped <reason> at retained floor <block>.

Checks

  1. Search the indexer logs for EVM span chain parked; its siblings continue. The line carries the chain and the failure that parked it. A park for another failure tag is not this runbook: use Ledger indexing stalled.

  2. Read the failure text. The fork block is the position of the repair failure. The retained floor is the number after at retained floor. The reason is one of below-retained-floor, at-or-below-finality, or beyond-repair-depth.

  3. Read the stored reach for the chain from the database:

    SELECT chain_id, committed_block, rollup_block, finality_block
FROM idxr_chain_watermark
WHERE chain_id = <chain id>;

    Run it in the indexer schema of the instance.

  4. Compare the fork block with rollup_block and finality_block:

    • The fork block is above both. Undo data exists for it, so the indexer can repair it.
    • The fork block is below rollup_block, or at or below finality_block. The indexer has no undo data to rewind to.

Actions

  • If the fork block is above rollup_block and above finality_block, restart the indexer deployment. The first attempt after the restart verifies the chain before it writes to idxr_chain_watermark, finds the common ancestor above the rollup floor, rewinds to it, and continues. Reach comes from the stored rollup_block and finality_block, so a chain that was down for more than 64 blocks still repairs.
  • If the fork block is below rollup_block, or at or below finality_block, a restart parks the chain again. The data needed to rewind is gone. Rebuild through a reindex. A platform administrator starts one with POST /api/v2/admin/operator/blockchain-monitoring/reindexes, naming the chain's chainId. A reindex rebuilds the indexed data of every network at once, and only one runs at a time. It indexes again from each network's start block into a new dataset while the current one keeps serving, so reads for the parked chain stay frozen at its last indexed block until the new dataset takes over. Upgrades and migrations explains how a rebuild runs, and the Indexing detail page on the Console's Platform status screen shows the chain as Reindexing until it drains.

Verify recovery

After a restart, committed_block for the chain advances and the indexer logs no new park line for it. After a rebuild, the chain leaves “Reindexing” on the Indexing detail page.

Escalate

If the chain parks again after a restart with a fork block higher than both values, escalate to your DALP support contact as a platform defect. Include the chain id, the park line, the failure text, and the row from idxr_chain_watermark.

Was this page helpful?