Skip to main content

Runbooks

Diagnose and fix readiness, indexing, funding, RPC, signing and webhook delivery problems on a DALP instance, starting from the symptom you see.

Find what you are seeing in Match the symptom to a runbook and open that runbook. Each runbook opens from a symptom you can observe directly: kubectl output, a Console screen, an alert or a log line. Its checks name the signal to read and, where the platform defines one, the value that means healthy, and it ends with the fix.

Match the symptom to a runbook

What you observe
Open
Why this one
kubectl get pods shows a pod Running but not 1/1 Ready, or Console and API calls fail with connection errors or 503s while every pod shows Running.
Platform API not ready
The pod is out of rotation. Check the readiness of the other replicas.
Console, the API, or reports stop reflecting new on-chain activity while the pods themselves report Ready.
Ledger indexing stalled
The pod is healthy. Only the chain data behind it has stopped advancing.
A registered webhook endpoint stops receiving events, alone or while sibling endpoints on the same instance keep working, or its status in Console flips to “Auto-disabled” on its own.
Webhook deliveries failing
Start with that endpoint's delivery history and breaker state.
Every registered webhook endpoint on the instance goes quiet at the same time.
Ledger indexing stalled
The indexer writes each webhook event in the same transaction that advances its indexed state, so a stalled indexer can stop new chain events across endpoints. Also inspect the shared delivery runner.
Work that DALP filed through a Utila configuration stays in AWAITING_SIGNATURE in the Utila console, or the co-signer does not show as running.
Utila signing agent
The customer-run agent signs only after DALP answers. Read its answers first.
The Console's Submissions tray shows work as “Waiting for funds”, or the API answers TRANSACTION_QUEUE_FUNDING_REQUIRED.
Transactions waiting for funding
The paying wallet could not cover gas. Work is paused while the Submissions tray shows “Waiting for funds”, and has failed when a Prefer: wait call answers the code.
Logs show RPC upstream failed over or All RPC upstreams exhausted, dalp_chain_evm_rpc_upstreams_exhausted_total rises, or RpcHealthSelectionDisabled fires.
RPC upstreams failing over
One or more of the network's RPC endpoints fail. The log lines and the alert come from the indexer. For queued transactions, read the dalp_chain_evm_rpc_* counters.
The indexer logs EVM span chain parked; its siblings continue for one chain, and that chain stays frozen while the indexer pod is Ready and its other chains advance.
Indexer chain parked
The indexer stopped that chain on a failure a retry cannot fix. A restart clears the park, and the failure decides whether that is enough.
After an upgrade, the network card on “Platform networks” reads “Rebuilding indexed data” and the percentage rises slowly.
Index rebuild after an upgrade
The release rebuilds indexed data while the current data keeps serving. Read its pace before you intervene.
IndexerSpanRetryLoop or IndexerWritingLaneRefused fires, or the rebuild percentage on “Platform networks” stops rising after an upgrade.
Index rebuild after an upgrade
The build is not being written: a database statement fails on every attempt, or this release cannot write the build's data layout. The runbook names the log line for each.

An indexer pod answers /healthz and /readyz with 200 while its process runs, so its pod status does not show a stalled chain. A chain the indexer cannot retry is parked: the indexer logs EVM span chain parked; its siblings continue in category host.didx.evm-span, with chain_id, failure_tag and error_detail, and keeps indexing its other chains. The pod stays Ready and does not restart. Read that log line, not pod restarts.

When transactions stop going out

Check these four causes in order. Fix the first one whose signal you find, because a later cause can be an effect of an earlier one.

  1. The paying wallet is short of gas. The Submissions tray shows “Waiting for funds”, the Transaction funding page shows a notice such as <wallet> needs <amount>, or a transaction request failed with DALP-9087 on its status or DALP-14062 from the API. Open Transactions waiting for funding.
  2. The network's RPC endpoints fail. dalp_chain_evm_rpc_attempts_total with outcome failure rises for the chain, or dalp_chain_evm_rpc_upstreams_exhausted_total rises on a network with more than one endpoint. These come from the client that sends queued transactions, which uses only the endpoints in the deployment values. All RPC upstreams exhausted in the logs and an “Unhealthy” or “Excluded” badge on “Platform networks” describe the indexer's pool. Open RPC upstreams failing over.
  3. A transaction stays open. A row on the “Transactions” detail of Platform status stays open without settling, and the first two causes are ruled out. Use “Set next nonce” when a nonce gap holds it, or “Force fail” it when it can no longer complete, as Recover a stuck transaction describes. Before you force fail a transaction that has a transaction hash, check on a block explorer that no transaction from that sender with that nonce was mined or is still pending. A force-failed transaction that lands later, followed by a resent write, runs the operation twice.
  4. The transactions settle but screens do not show them. That is indexing, not sending. Open Ledger indexing stalled, or Index rebuild after an upgrade when a release started a rebuild.

How the roles depend on each other

A DALP instance runs four backend roles, and these runbooks cover three of them: one serves Console and API traffic, one indexes chains into Ledger Data, and the runner executes durable workflows, including webhook delivery. The fourth, the scheduler, runs background jobs. The indexer writes each webhook event as part of the same transaction that advances its indexed state, and the runner reads that queue on its own cadence to deliver it, so a stalled indexer starves webhook deliveries too. Each of the three answers /readyz, but only the Platform API reports boot phases and dependency verdicts there. The runner and the indexer report only whether the process is up or draining. All three write into one PostgreSQL database, so a symptom on one surface can trace back to another. The scheduler's /readyz answers 503 when the bank FIX simulator cannot listen and, in external mode, lists the bank FIX sessions it dials, as Platform API not ready describes.

Three roles run one instance, and they share one database.Three cards inside one instance outline show the backend roles. The indexer reads chains into Ledger Data and publishes a servable schema once it is caught up. The Platform API serves Console and API traffic, and its readiness waits on that schema being bound. The runner, the Workflow Engine role, owns durable execution and sends webhook deliveries. An accent arrow carries the schema bind from the indexer into the Platform API's readiness, which is why a readiness symptom can turn out to be an indexing fault. A second accent arrow runs from the indexer to the runner, carrying the webhook queue the indexer writes as part of the same transaction that advances its indexed state, which is why a stalled indexer starves webhook deliveries too. Below the instance outline, a connector leads to one PostgreSQL database that holds durable execution state, chain positions, and webhook delivery history for every role, which is why a database problem can surface as more than one symptom at once.Three roles run one instance, and they share one database.The same instance map as the desktop variant, drawn portrait. The indexer reads chains into Ledger Data and publishes a servable schema, feeding the schema bind to the Platform API's readiness, which is why a readiness symptom can turn out to be an indexing fault. The runner, the Workflow Engine role, owns durable execution and delivers webhooks. A second arrow runs from the indexer past the Platform API to the runner, carrying the webhook queue the indexer writes as part of the same transaction as its indexed state, so a stalled indexer starves deliveries too. Below the instance outline, one PostgreSQL database holds durable execution state, chain positions, and delivery history for every role, which is why a database problem can surface as more than one symptom at once.

The indexer feeds the Platform API's readiness and the runner's webhook queue, and all three roles write to one PostgreSQL database.

Open a runbook

Platform API not ready

A pod stays out of Ready: tell a slow boot from a hang and read its status and logs.

Ledger indexing stalled

Indexed data stops advancing while pods are Ready: read the indexer's progress and logs.

Webhook deliveries failing

A webhook endpoint goes quiet: read the retry schedule, circuit breaker and recovery.

Utila signing agent

Utila work stops being signed: read the agent's answers, then enrol, rotate or recover it.

Transactions waiting for funding

Work pauses because the paying wallet cannot cover gas: fund it inside the window.

RPC upstreams failing over

RPC endpoints fail over or run out: find the failing one and fix it where it is read.

Indexer chain parked

One chain stops on a failure a retry cannot fix: tell a restart from a rebuild.

Index rebuild after an upgrade

A release rebuilds indexed data for hours: read its progress and know when it is a fault.

When nothing here matches

If the pod is Ready, Ledger Data is current, and webhook deliveries are flowing, but you are still looking at a failure none of these runbooks name, capture the evidence before you touch anything. Take the pod's logs and kubectl describe pod output, the delivery history, or whatever error your own integration returned. A restart or a resend can destroy the failure reason you need. Escalate to your DALP support contact with what you captured.

Was this page helpful?