Ledger indexing stalled

Console, the API or reports have stopped reflecting new on-chain activity on one or more chains. Read the chain's indexed position: a position that still advances is a backlog, a position that stopped moving means an indexer pod failed on that chain's ingest, and an absent position means the chain never started. This page assumes the indexer pod is running and passing its probes. For a pod that Kubernetes holds out of Ready, start with Platform API not ready instead. It covers the readiness legs, and its indexer leg hands back here.

Ledger Data is the platform's own record of the chains a DALP instance follows. It reads each block once, decodes the events into business records, and Console, the API and reports answer from that record rather than from the live network. When indexing stops advancing on a chain, those surfaces freeze at the last indexed block while the network moves on. The checks separate the three signatures, cheapest first.

Symptoms#

• Console screens, API responses, or reports stop reflecting new on-chain activity while the underlying network keeps producing blocks.
• The Console's Platform status screen shows a chain out of sync in its Indexing section, a growing worst sync-lag, or a degraded data-freshness panel.
• The gap between the chain's head and the indexed position grows instead of shrinking.
• The indexer pod may still report Ready, and that is expected: its probes answer for the process, not for chain state. A chain that lags, a chain that keeps retrying a failed read and a chain that has stopped all leave the pod green, and the indexer keeps running its other chains (Check 2). A green pod with a frozen position is the normal shape of a stall, so read the chain's log lines rather than the pod status.

Checks#

1. Confirm the chain is frozen, not catching up. The signal is the indexed position over time: the Console's Platform status screen shows chains in sync and the worst sync-lag in blocks, and the indexer exports the same lag for each network as the dalp_didx_watermark_lag_blocks metric. Healthy catch-up means the indexed position advances and the lag shrinks. A constant lag can mean ingestion matches block production but is not catching up.

   The platform's own monitor marks a chain's indexing degraded once it is 10 or more blocks behind or has gone 5 minutes without progress, and critical past 100 blocks or 15 minutes.

   Two shapes are not stalls. A rebuild after a platform upgrade reports as reindexing rather than degraded. That is the indexer working, so let it drain. A chain that is fully caught up on a quiet network is idle, not stalled: the stall clock only runs while the chain is behind. Chains that mint blocks only when a transaction is submitted skip the no-progress rule entirely.

2. On a chain whose position stopped moving, read the logs of every indexer pod for that chain's chain_id in the host.didx.evm-span category. A failed attempt and a stop log different lines.

   EVM span attempt failed; retrying from the committed watermark is a warning. The attempt failed, for example on an RPC error, and the chain tries again from its last committed position after a short wait. A cause that clears lets the next attempt through, and a cause still present repeats the same warning.

   EVM span chain parked; its siblings continue is an error. The chain stopped on a failure it does not retry, and the indexer keeps indexing its other chains. A stop the platform records, such as a handler failure, a reorganisation deeper than the repair depth or a replaced chain, also logs EVM span attempt stopped. The lines carry failure_tag and error_detail, which name the failure. A parked chain stays stopped until the indexer restarts, and the restart resumes it from the last committed watermark. For a park whose failure tag is repair, use Indexer chain parked.

3. For a chain with no indexed position at all from the moment the instance started, and no failed-attempt or parked line for it on any replica, check that the chain is in the instance's networks entry and that an indexer pod owns it, as the next check describes.

4. For positions that froze or flapped around a redeploy, a replica restart, or a replica-count change, look at the ownership handoff. Ownership of each network is a lease with an expiry and an epoch. A process can take the lease only after the current holder's lease has expired, and each takeover raises the epoch, so a write from an owner whose lease lapsed is rejected as a conflict rather than silently accepted. An owner that finds its lease lapsed or taken logs Chain lease renewal found the lease lapsed or taken; the lane acquires it again on its next span attempt in the same category. A healthy handoff therefore shows the departing owner's writes failing briefly while the new owner continues from the stored cursor. Owners keep no state of their own, so nothing is lost in the transfer.

   Each pod carries only the chains it owns, so a redeploy or a replica-count change moves chains between pods and a chain can legitimately have no owner on the pod you happened to read. Check every replica before treating a frozen chain as one nobody is working on.

   A process outside the deployment that uses the same database, for example an operator's ad hoc instance, competes for the same leases. Check for one, because a chain it holds is not indexed by the managed deployment.

Actions#

• If the position is behind but advancing, it is a backlog: monitor it. Divide the lag in blocks (dalp_didx_watermark_lag_blocks) by the observed rate at which that lag decreases to size the window it should close in. Revisit your RPC provider limits or the chain's scope if it is not closing within that window. Do not add indexer replicas to speed one chain up: each network's live ingest is a fenced singleton owned by one process, so extra replicas spread chains across pods and buy failover, never per-chain throughput.
• If the chain's position stopped moving, read its failed-attempt or parked line: the failure it names decides the fix. A failure the database refused, such as a missing table or column, a constraint violation or a revoked permission, is fixed in the database. The refused transaction rolled back whole, so nothing already indexed is lost and the checkpoint did not move. A failure naming the chain's configuration is fixed in the instance's networks entry and needs a rollout. A chain that parked resumes when the indexer restarts after the fix.
• If an RPC provider is unavailable or rate-limiting, the chain's attempts fail on it and it retries from the last committed watermark until the provider recovers. Check the provider's status and your configured endpoint first: it is quicker to rule out than duplicate ownership. The chain needs nothing from you while you investigate.
• If a rebuild after an upgrade stops at an old block and the indexer logs show a missing-state or pruned-history error from the provider, the provider cannot serve that block. Point the chain's RPC pool at an archive-capable endpoint that also keeps logs and block headers from the chain's start block, as What RPC provider does a rebuild need? describes. The chain picks the rebuild up again on its next try, which comes every 5 minutes.
• If the failure is a handler failure, escalate to your DALP support contact as a platform defect with the held block, the event name, and the parked line's text. Restarting the indexer before the fix only parks the chain again. Nothing already indexed is lost while the chain is stopped: once the release carrying the fix deploys, the restart resumes the chain from the held checkpoint. A release whose fix changes what earlier data should have looked like declares its own rebuild, which reports as reindexing while it drains.
• If the stop line names failure_tag=chain-generation, the database holds data from a chain that no longer exists, for example a development chain that restarted without its state. The indexer stops that chain and refuses its reads, so no screen shows the old data as current. Start a reindex: a platform administrator sends POST /api/v2/admin/operator/blockchain-monitoring/reindexes with the chain's chainId. The rebuild indexes the live chain and replaces the old data when it is complete. A reindex rebuilds every network at once, and on a long history it runs for hours.
• If a process outside the managed deployment holds chain leases, stop that process.

Verify recovery#

The chain's indexed position advances, the Platform status screen shows the chain in sync, and dalp_didx_watermark_lag_blocks falls for the chain.