# Indexer contract quarantined

Source: https://docs.settlemint.com/docs/deployment/runbooks/indexer-contract-quarantined
The indexer quarantines one contract it cannot index and keeps indexing the network: find the contract and its cause, then rebuild once the cause is fixed.

Ledger Data for one contract on an EVM network has stopped advancing because the indexer
quarantined that contract while continuing to index the rest of the network. Every screen, API response and report shows
that contract's last indexed state without a staleness banner.

Neither a pod restart nor indexer retries clear a quarantine. Identify the cause (`handler`, `oversized-block` or `discovery`), fix it first, then trigger an index rebuild. A platform administrator who saw "Attention" on **Indexing scope** starts at Check 3, gets the cause from the operator's Check 2, and runs the rebuild once fixed.

## Symptoms [#symptoms]

- The indexer logs `EVM span quarantined a contract; the chain keeps indexing without it` (error level, category `host.didx.evm-span`) once after the failing span.
- With observability alerting enabled, `IndexerContractQuarantined` fires for one hour and then auto-resolves even though the contract remains quarantined. The database row is the durable record.
- In the Console under **Platform admin** → **Indexing scope**, the owning system shows "Attention" (or "Hidden from reads" if excluded). System-less contracts such as the Directory or factories appear on no row.
- Screens for the contract freeze at the block before the failure while sibling contracts keep advancing.

## Checks [#checks]

1. **Find the serving schema in PostgreSQL:**

   ```sql
   SELECT schema_name FROM idxr_deployment_registry WHERE is_serving;
   ```

2. **List quarantined contracts in that schema:**

   ```sql
   SELECT address, type, system_address, quarantined_at_block, quarantine_reason, quarantine_detail
   FROM <schema_name>.idxr_indexed_contract
   WHERE chain_id = <chain id> AND sync_status = 'failed';
   ```

   - `quarantine_reason`: `handler`, `oversized-block` or `discovery` (determines the action below).
   - `quarantine_detail`: names the failing handler (`handler`), the oversized block and whether the RPC provider or the indexer's response limit refused it (`oversized-block`), or `Discovery passed 8 fetch rounds without converging.` (`discovery`).
   - `quarantined_at_block`: the last block indexed for the contract.
   - `type`: if `Directory`, a factory or a registry, contracts it would have registered after that block are missing too.
   - Empty `system_address`: no system owns the contract, so **Indexing scope** does not list it.

   Multiple rows indicate several contracts were quarantined in one or more spans.

3. **Confirm impact in the Console.** As a platform administrator, open **Platform admin** → **Indexing scope** and locate the system on its network. "Attention" confirms the quarantine affects an active system. The contract address itself comes from Check 2 or Check 4.

4. **Search recent indexer logs if still within retention:**

   ```bash
   kubectl -n dalp logs -l 'app.kubernetes.io/component in (indexer,indexer-rebuild)' --tail=500 \
     | grep 'EVM span quarantined a contract'
   ```

   The log fields map directly to the row: `subject_address` (`address`), `failure_tag` (`quarantine_reason`), `error_detail` (`quarantine_detail`), `chain_id` and `block_number` (`quarantined_at_block + 1`).

## Actions [#actions]

- **System shows "Hidden from reads":** No screen reads the contract, so you can leave it quarantined. Fix the cause and rebuild before re-including that system in reads.
- **Cause is `handler`:** Escalate to your DALP support contact as a platform defect with `chain_id`, `address`, `quarantine_detail` and the database row. Wait for the DALP release with the fix before rebuilding, or the contract will be quarantined again. If the release starts its own rebuild (**Platform admin** → **Platform networks** reads "Rebuilding indexed data"), let it run. Otherwise start a rebuild below.
- **Cause is `oversized-block` (RPC provider cap):** Point the network's RPC pool in your [`networks` configuration](https://docs.settlemint.com/docs/deployment/networks) at an endpoint whose `eth_getLogs` limit accepts that block, then rebuild.
- **Cause is `oversized-block` (indexer response limit):** No RPC endpoint raises the indexer's built-in limit. Escalate to your DALP support contact as a platform defect with the same fields as `handler`.
- **Cause is `discovery`:** Escalate as a platform defect with the same fields as `handler` and rebuild after the fix lands. The 8-round bound is fixed in code, though raising `global.dalpApp.config.indexer.batch.maxBlocks` makes hitting it across a single span more likely.
- **Trigger a rebuild once fixed:** Have a platform administrator call the reindex API (no Console button exists):

  ```bash
  curl -X POST "https://<console-host>/api/v2/admin/operator/blockchain-monitoring/reindexes" \
    -H "Authorization: Bearer <admin-token>" \
    -H "Content-Type: application/json" \
    -d '{"chainId": <chain id>}'
  ```

  One reindex runs at a time and rebuilds every configured network into a new dataset while the current dataset continues serving. Ensure every network's RPC pool can serve historical blocks ([What RPC provider does a rebuild need?](https://docs.settlemint.com/docs/deployment/runbooks/ledger-indexing-stalled#what-rpc-provider-does-a-rebuild-need)) and monitor progress with [Index rebuild after an upgrade](https://docs.settlemint.com/docs/deployment/runbooks/index-rebuild-after-upgrade).

## Verify recovery [#verify-recovery]

After the rebuilt dataset promotes, the system clears "Attention" on **Indexing scope**. Run Check 1 again for the new serving schema, then run Check 2 and confirm zero rows return for the contract and no new quarantine log lines appear.

## Escalate [#escalate]

If the contract is quarantined again after rebuilding on a fixed release or higher-cap RPC endpoint, escalate to your DALP support contact with `chain_id`, contract `address`, the quarantine log line and the full row from `idxr_indexed_contract`.

## How contract quarantine works [#how-contract-quarantine-works]

The indexer reads each EVM network in block spans committed in one database transaction. When a single contract blocks a span, the indexer quarantines that contract instead of halting the whole network, commits the span without the contract's events from the failing block onward, and excludes the contract from subsequent spans.

Three failure types trigger quarantine:

- `handler`: an event handler throws while projecting the contract's event.
- `oversized-block`: the contract's logs in a single block exceed the RPC provider's or the indexer's `eth_getLogs` response limit.
- `discovery`: a chain of contract or account registrations from one event exceeds eight fetch rounds inside a single span (one span fetch plus up to seven registration rounds). Trusted issuers and topic scheme registries also cap parent-registry discovery at three hops, since compliance reads follow at most two.

Because the quarantine is stored on the contract's row in PostgreSQL, restarting the indexer never clears it. Only a full dataset rebuild (or a chain reorganisation rewinding past the failing block) retries the contract.

If several contracts fail in the same span, the indexer quarantines each one, replays the already-fetched logs for the span, and writes one log line per contract. Escalate multi-contract quarantines from the same span as one defect.

The whole network still parks instead of quarantining when a handler fails on an event unlinked to a registered contract, when an end-of-span step fails, or when a handler's chain read hits a transient RPC error. If a span contains both a park condition and a quarantine condition, the entire span parks without quarantining any contract ([Ledger indexing stalled](https://docs.settlemint.com/docs/deployment/runbooks/ledger-indexing-stalled)). Rate-limit refusals retry normally without quarantining.