Ledger Data for one contract on an EVM network has stopped advancing because the indexer quarantined that contract while continuing to index the rest of the network. Every screen, API response and report shows that contract's last indexed state without a staleness banner.
Neither a pod restart nor indexer retries clear a quarantine. Identify the cause (handler, oversized-block or discovery), fix it first, then trigger an index rebuild. A platform administrator who saw “Attention” on Indexing scope starts at Check 3, gets the cause from the operator's Check 2, and runs the rebuild once fixed.
Symptoms
- The indexer logs
EVM span quarantined a contract; the chain keeps indexing without it(error level, categoryhost.didx.evm-span) once after the failing span. - With observability alerting enabled,
Indexerfires for one hour and then auto-resolves even though the contract remains quarantined. The database row is the durable record.Contract Quarantined - In the Console under Platform admin → Indexing scope, the owning system shows “Attention” (or “Hidden from reads” if excluded). System-less contracts such as the Directory or factories appear on no row.
- Screens for the contract freeze at the block before the failure while sibling contracts keep advancing.
Checks
-
Find the serving schema in PostgreSQL:
SELECT schema_name FROM idxr_deployment_registry WHERE is_serving; -
List quarantined contracts in that schema:
SELECT address, type, system_address, quarantined_at_block, quarantine_reason, quarantine_detail FROM <schema_name>.idxr_indexed_contract WHERE chain_id = <chain id> AND sync_status = 'failed';quarantine_reason:handler,oversized-blockordiscovery(determines the action below).quarantine_detail: names the failing handler (handler), the oversized block and whether the RPC provider or the indexer's response limit refused it (oversized-block), orDiscovery passed 8 fetch rounds without converging.(discovery).quarantined_at_block: the last block indexed for the contract.type: ifDirectory, a factory or a registry, contracts it would have registered after that block are missing too.- Empty
system_address: no system owns the contract, so Indexing scope does not list it.
Multiple rows indicate several contracts were quarantined in one or more spans.
-
Confirm impact in the Console. As a platform administrator, open Platform admin → Indexing scope and locate the system on its network. “Attention” confirms the quarantine affects an active system. The contract address itself comes from Check 2 or Check 4.
-
Search recent indexer logs if still within retention:
kubectl -n dalp logs -l 'app.kubernetes.io/component in (indexer,indexer-rebuild)' --tail=500 \ | grep 'EVM span quarantined a contract'The log fields map directly to the row:
subject_address(address),failure_tag(quarantine_reason),error_detail(quarantine_detail),chain_idandblock_number(quarantined_).at_ block + 1
Actions
-
System shows “Hidden from reads”: No screen reads the contract, so you can leave it quarantined. Fix the cause and rebuild before re-including that system in reads.
-
Cause is
handler: Escalate to your DALP support contact as a platform defect withchain_id,address,quarantine_detailand the database row. Wait for the DALP release with the fix before rebuilding, or the contract will be quarantined again. If the release starts its own rebuild (Platform admin → Platform networks reads “Rebuilding indexed data”), let it run. Otherwise start a rebuild below. -
Cause is
oversized-block(RPC provider cap): Point the network's RPC pool in yournetworksconfiguration at an endpoint whoseeth_getLogslimit accepts that block, then rebuild. -
Cause is
oversized-block(indexer response limit): No RPC endpoint raises the indexer's built-in limit. Escalate to your DALP support contact as a platform defect with the same fields ashandler. -
Cause is
discovery: Escalate as a platform defect with the same fields ashandlerand rebuild after the fix lands. The 8-round bound is fixed in code, though raisingglobal.makes hitting it across a single span more likely.dalp App. config. indexer. batch. max Blocks -
Trigger a rebuild once fixed: Have a platform administrator call the reindex API (no Console button exists):
curl -X POST "https://<console-host>/api/v2/admin/operator/blockchain-monitoring/reindexes" \ -H "Authorization: Bearer <admin-token>" \ -H "Content-Type: application/json" \ -d '{"chainId": <chain id>}'One reindex runs at a time and rebuilds every configured network into a new dataset while the current dataset continues serving. Ensure every network's RPC pool can serve historical blocks (What RPC provider does a rebuild need?) and monitor progress with Index rebuild after an upgrade.
Verify recovery
After the rebuilt dataset promotes, the system clears “Attention” on Indexing scope. Run Check 1 again for the new serving schema, then run Check 2 and confirm zero rows return for the contract and no new quarantine log lines appear.
Escalate
If the contract is quarantined again after rebuilding on a fixed release or higher-cap RPC endpoint, escalate to your DALP support contact with chain_id, contract address, the quarantine log line and the full row from idxr_indexed_contract.
How contract quarantine works
The indexer reads each EVM network in block spans committed in one database transaction. When a single contract blocks a span, the indexer quarantines that contract instead of halting the whole network, commits the span without the contract's events from the failing block onward, and excludes the contract from subsequent spans.
Three failure types trigger quarantine:
handler: an event handler throws while projecting the contract's event.oversized-block: the contract's logs in a single block exceed the RPC provider's or the indexer'seth_getLogsresponse limit.discovery: a chain of contract or account registrations from one event exceeds eight fetch rounds inside a single span (one span fetch plus up to seven registration rounds). Trusted issuers and topic scheme registries also cap parent-registry discovery at three hops, since compliance reads follow at most two.
Because the quarantine is stored on the contract's row in PostgreSQL, restarting the indexer never clears it. Only a full dataset rebuild (or a chain reorganisation rewinding past the failing block) retries the contract.
If several contracts fail in the same span, the indexer quarantines each one, replays the already-fetched logs for the span, and writes one log line per contract. Escalate multi-contract quarantines from the same span as one defect.
The whole network still parks instead of quarantining when a handler fails on an event unlinked to a registered contract, when an end-of-span step fails, or when a handler's chain read hits a transient RPC error. If a span contains both a park condition and a quarantine condition, the entire span parks without quarantining any contract (Ledger indexing stalled). Rate-limit refusals retry normally without quarantining.