# Index rebuild after an upgrade

Source: https://docs.settlemint.com/docs/deployment/runbooks/index-rebuild-after-upgrade
An upgrade that changes the indexer's data version rebuilds indexed chain data for hours: read its progress, check its provider and spot a stalled rebuild.

After an upgrade that changes the indexer's data version, the indexer rebuilds its data while the current data keeps serving reads. Act only when the build stops advancing, the provider throttles it, or `IndexerBuildFailed` fires.

The data version is the format the indexer records chain data in. The first time such a release starts, the indexer builds a new dataset for Ledger Data. It indexes each EVM network again from the network's configured start block, or else from the block where DALP's Directory was deployed. With a legacy Directory, it starts at the earlier of the two. The current dataset keeps serving Console, API and report reads while the new one builds, and those reads stay current while the serving lane can take the release's writes, as Check 3 explains. When the new dataset has caught up, the platform switches reads to it in one step. Its duration grows with the number of blocks from the start block to the head and with the provider's `eth_getLogs` throughput. A rebuild over a long chain history can take hours, so measure one on a non-production copy before the change window.

Rolling back to an older release after the switch is not self-service, as [Actions](#actions) explains. [Upgrades and migrations](https://docs.settlemint.com/docs/deployment/upgrades-migrations) explains where the rebuild sits in a change window.

## Symptoms [#symptoms]

- The indexer logs, category `didx.db.deployment`, hold `Dataset created; it indexes from the chain's start` with `status` `building` and the new `version`.
- The network's card on "Platform admin", "Platform networks" reads `Rebuilding indexed data · <percent>% · block <current> of <target> · started <time>. Reads use the current data until the rebuild catches up.` A progress bar sits under the line. Before the build records its first block on that network, the line reads `Rebuilding indexed data · started <time>. Reads use the current data until the rebuild catches up.`
- The Console's [Submissions tray](https://docs.settlemint.com/docs/console/essentials/submissions) shows a confirmed write as "Recorded on-chain", with the line "Recorded on-chain. The platform rebuilds its data, and your lists catch up on their own. Nothing is needed from you."
- Creating or retrying an organisation deployment, creating an invitation, or accepting one, with `Prefer: dalp-fail-on-indexer-reindexing` is answered `503` with `DALP-12331` and the message "The indexer is still catching up on this chain". The response carries `Retry-After` and `data.retryAfterSeconds`. Without that preference, these requests are not refused for the rebuild. The refusal also needs the `indexerReadinessGate` feature, which is on by default.

## How the rebuild runs [#how-the-rebuild-runs]

Each lane is a set of indexer processes that writes one dataset:

- **The build lane** indexes each EVM network into the new dataset. With the default chart values, a separate rebuild Deployment of two pods runs it: `dapi.indexerDeployment.rebuild.enabled` is `true` and `dapi.indexerDeployment.rebuild.replicas` is `2`. With the rebuild Deployment turned off, every indexer pod runs both lanes.
- **The serving lane** keeps the current dataset following the chain head, so reads stay current during the build.

The platform switches reads to the new dataset when two conditions hold on every network that is not paused. The build has reached the finalised block it last recorded for the network, and it has caught up with the serving dataset. A network whose provider had no history at the start block counts as ready, so promotion does not wait for it. When the build still trails and every trailing network committed new blocks in the last 60 seconds, the platform holds the serving dataset still for at most 30 seconds so the build can catch up, and logs `Serving dataset frozen for the handover; the build catches up before promotion`. The switch logs `Dataset promoted; reads now serve from the rebuild`. The platform drops the old dataset at its first check after one hour has passed since the switch.

## Checks [#checks]

1. Read the progress. On "Platform networks", the card's `block <current> of <target>` compares the build's position with the serving dataset's position on that network. The percentage is `<current>` divided by `<target>`. The build starts at the network's start block, so the first reading is that block's share of `<target>`, not zero. `<target>` moves as the serving dataset follows the chain, unless the serving lane is refused, as in step 3. Read both numbers twice, five minutes apart. If the gap between them shrinks, the build is catching up, and the gap divided by how fast it shrinks gives the time left.

2. Confirm that reads are still current. Open "Organisation settings", "Infrastructure", "Platform status", then the "Indexing" detail. Its snapshot log describes the serving dataset that reads use, not the build. "Status" and "Lag" show by default. "Stall" is hidden by default, so turn that column on. A low "Lag" means the serving lane follows the chain while the build runs.

3. If "Lag" and "Stall" grow on that page during the build, search the indexer logs, category `host.didx.serving-lane`, for `Serving lane refused: the serving dataset cannot take this version's writes, so it stays where it is until it can or until promotion`. The release changed the data layout in a way the old dataset cannot take. Reads then stay where they were when the earlier release stopped, until the switch.

4. If the percentage stops rising, check that the build lane runs: `kubectl get pods -l app.kubernetes.io/component=indexer-rebuild`. With the rebuild Deployment on, only these pods write the build, so the build does not advance while they are down. `IndexerBuildFailed` does not fire while the build is still behind the finalised block, so a flat `<current>` with no alert can still be a stall. Then check the alerts and logs in [When to worry](#when-to-worry).

5. Check the provider of the slow network. The build reads the whole indexed range of the chain through the indexer's RPC endpoint pool, so the provider sets its pace. A rising `dalp_rpc_admission_throttles_total` for that network means the provider answers `429` and the pool retries on the same endpoint. [RPC upstreams failing over](https://docs.settlemint.com/docs/deployment/runbooks/rpc-upstream-failover) covers the pool's health.

## Actions [#actions]

- **The rebuild advances.** Let it drain. Adding indexer replicas does not make one network faster: one process at a time holds a network's lane, so extra replicas add failover and leave throughput unchanged.
- **The provider limits the pace.** The build does not use the network's "RPC call limits" from the Console. It reads spans of at most `global.dalpApp.config.indexer.batch.maxBlocks` blocks in the chart values, which the pod's `DIDX_BATCH_MAX_BLOCKS` variable can move, and 2,000 by default. It halves the span after repeated fetch failures and doubles it again, up to that maximum, after three clean commits. Raise the maximum only as far as the provider's `eth_getLogs` range limit allows, or raise the plan at the provider.
- **The build stops at an old block with a missing-state or pruned-history error.** The provider does not keep that history. Point the network's pool at an archive-capable endpoint, as [What RPC provider does a rebuild need?](https://docs.settlemint.com/docs/deployment/runbooks/ledger-indexing-stalled#what-rpc-provider-does-a-rebuild-need) describes. The build resumes from its last committed position. Fix the endpoint before the switch, because a network whose provider has no history at the start block does not hold back promotion.
- **Do not roll back or upgrade again during a rebuild unless you accept a fresh start.** A release with a different indexer data version discards a build that is not its own and logs "Retiring dataset" with reason `build for another version`. The next release with the new data version starts its build again from the beginning.
- **Rolling back after the switch.** An older release than the one serving does not bind the data. It logs a line that starts with "Runtime is behind the serving deployment", and its readiness waits for an operator. Escalate instead of forcing it. A database restore from the backup taken before the upgrade, as [Backup and disaster recovery](https://docs.settlemint.com/docs/deployment/backup-recovery) describes, is the way back to the older release's data.

## When to worry [#when-to-worry]

| Signal | What it means |
| --- | --- |
| `IndexerBuildFailed` fires, and `dalp_indexer_build_failures_total` rose with `phase` `validate` | The build reached the finalised block on every network but still trails the serving dataset on one. That network's build has not committed for 60 seconds, or its block at the serving position has a different hash. Serving data is unchanged. Compare the network cards to find the network whose `<current>` stays flat. |
| `IndexerBuildFailed` fires, and the counter rose with `phase` `promote` or `cleanup`, with `Dataset promotion check failed; the next check retries` or `Drained dataset cleanup failed; the next check retries` in the logs | The switch or the removal of the old dataset failed. The platform retries on the next check. Escalate if it keeps firing. |
| `Indexing stalled: no new block for <age>.` on the network card | The serving dataset has not advanced for 15 minutes while the chain has. This line replaces the rebuild line on the card. It is a stall, and rebuild progress does not explain it: use [Ledger indexing stalled](https://docs.settlemint.com/docs/deployment/runbooks/ledger-indexing-stalled). |
| `EVM span chain parked; its siblings continue` in the indexer logs | One network stopped on a failure. Use [Indexer chain parked](https://docs.settlemint.com/docs/deployment/runbooks/indexer-chain-parked). |
| `IndexerSpanRetryLoop` fires, or `EVM span attempt failed; retrying from the committed watermark` repeats with a `driver_sqlstate` for one network and `operation` | One lane fails the same database statement on every attempt, so its position does not move. On the rebuild pods that holds the build back with no `IndexerBuildFailed`. The log line's `error_detail` names the `SQLSTATE` and the server message. A release that adds a column without a new data version adds it to the build when the rebuild pods start, so a missing column or table here means the release did not finish rolling out to the rebuild pods: check that they run the new release. |
| `IndexerWritingLaneRefused` fires, or `Writing lane refused: the dataset cannot take this version's writes, so it stays where it is until it can or until a new build replaces it` in category `host.didx.serving-lane` | The build was created by a release whose data layout this release cannot extend in place, so the rebuild pods do not write it, and it never promotes. The pods plan it again only on the next deployment change or pod restart. Escalate: the build needs a new data version to start over. Only a build is refused: with no build running, the indexer keeps writing the serving dataset and logs `Writing lane kept on a serving dataset whose layout differs from this version's` once instead. A database restored from a backup of a dataset this release created is never refused. |

## Verify recovery [#verify-recovery]

The network card no longer reads "Rebuilding indexed data", the indexer logs `Dataset promoted; reads now serve from the rebuild`, and `IndexerBuildFailed` is quiet.

## Escalate [#escalate]

Escalate to your DALP support contact when `IndexerBuildFailed` keeps firing, when the gap on the card stays the same for longer than your change window allows with the build lane running and no provider errors, or after a rollback past the switch. Include:

- the release you upgraded from and to.
- the network and chain ID, and two readings of the card's `block <current> of <target>` with their times.
- the `phase` values of `dalp_indexer_build_failures_total`, and the log lines in categories `didx.db.deployment`, `didx.dataset` and `host.didx.serving-lane` from the start of the build.
- the RPC provider, and the value of `DIDX_BATCH_MAX_BLOCKS` if you changed it.