RPC upstreams failing over

Checks#

1. For queued transactions, read dalp_chain_evm_rpc_attempts_total for the chain's chain_id by rpc_upstream_id and outcome. A rising failure count on one id names the failing endpoint. Compute the SHA-256 of each endpoint URL in the deployment values to match the id. Endpoint URLs often carry a provider API key, so keep them out of shared channels and redact the key from logs and screenshots you share. A rising dalp_chain_evm_rpc_upstreams_exhausted_total means calls ran out of endpoints.

2. For the indexer's pool, tell failover from exhaustion, using the ranking rules in How the indexer's pool picks an endpoint. A rising failover counter with a flat exhaustion counter means the endpoints absorb the failures. Calls can still fail for other reasons, such as a deadline or an error the endpoint returns for the call itself. Any All RPC upstreams exhausted line means calls ran out of endpoints.

3. Read the pool's card. Open “Platform admin”, “Platform networks” and read the network's card. When the health readings load, each endpoint shows a badge, “Healthy”, “Unhealthy”, “Excluded” or “Not measured”, with its latency as <value>ms and its error rate as <value>% errors. An excluded endpoint shows why, for example Left out of routing: 120 blocks behind the chain head, health score 0.18. “Serving now” marks the highest-priority endpoint whose latest reading is healthy. A reading older than three minutes counts as “Not measured”. The card lists the pool, which can differ from the endpoints in the deployment values.

4. Read the card's notices. “Endpoint health cannot be read right now, so nothing below states whether an endpoint is reachable.” means the health readings themselves are unavailable. “The platform has not measured these endpoints yet.” means no endpoint has a reading. “Chain head unknown: no endpoint answered.” means no endpoint returned the chain head.

5. Read the exhaustion line's fields. excluded_by_health and excluded_by_tip_lag count endpoints the scores left out. excluded_by_local_liveness counts endpoints skipped after a recent failure. failure_count against upstream_count shows how many endpoints the call tried and lost.

6. Look for rate limiting. A rising dalp_rpc_admission_throttles_total means the provider answers 429 and the indexer's pool retries on the same endpoint.

7. If RpcHealthSelectionDisabled fires, read the Health scores unavailable by cause panel on the dashboard, or dalp_rpc_health_scores_unavailable_total by reason: timeout, query_failed, read_unavailable or scoreless. Until scores return, the indexer ranks its pool by “Priority”, then configured order, so a lagging endpoint is no longer left out. The log line RPC health-based upstream selection restored marks the recovery.

Actions#

• Queued transactions fail on an endpoint. Fix the endpoint at the provider, or replace it in the network's rpc entry in the deployment values. A change there takes a helm upgrade and a restart of the affected components, as Adding or changing a network describes. An edit on “Platform networks” does not change the endpoints transactions are sent through.
• The network has one endpoint. With one endpoint, a provider outage fails every call to that network until the provider returns. Add a second endpoint from an independent provider in the deployment values, so calls can fail over when one provider is down.
• One endpoint in the pool fails or lags. On “Platform networks”, select “Edit network”, then work under “RPC endpoints”. Lower the endpoint's “Priority”, or select its remove icon, labelled “Remove endpoint”, once another usable endpoint remains. Save with “Save network”. Saving stores the new pool and signals the running processes to reload the network. The previous pool is not kept as a version you can restore, so record the current endpoints before you save. An endpoint added only on this page is not probed, so its card reads “Not measured” and the indexer never excludes it on health.
• The provider rate-limits. Raise the plan at the provider, or add an endpoint from another provider.
• Every endpoint is excluded for lag on a healthy chain. Raise “Max block-head lag” under “Endpoint health scoring” only when the chain's normal lag is larger than the default. The indexer's pool then uses the new value. The card's “Excluded” badge and reason, and the transaction engine's client, keep the defaults of 50 blocks and 0.25.
• Answers report missing state or pruned history for old blocks. The endpoint does not keep the history the platform reads. Use an archive-capable endpoint, as What RPC provider does a rebuild need? describes.
• Health scores are unavailable. Restore the health readings named by the reason. Meanwhile, give the most reliable endpoint the highest “Priority”, because priority and then configured order decide.

Verify recovery#

dalp_chain_evm_rpc_upstreams_exhausted_total and dalp_rpc_upstreams_exhausted_total stop rising, the logs show no new All RPC upstreams exhausted line, and the network card shows every endpoint you kept as “Healthy”.

Escalate#

Escalate to your DALP support contact when dalp_chain_evm_rpc_upstreams_exhausted_total or All RPC upstreams exhausted keeps rising after the provider confirms its endpoints are up, when RpcHealthSelectionDisabled keeps firing and its reason does not clear, or when an endpoint the card shows as “Healthy” keeps failing calls. Include:

• the network name and chain ID.
• dalp_chain_evm_rpc_attempts_total by rpc_upstream_id and outcome, and the other dalp_chain_evm_rpc_* and dalp_rpc_* counters for that network, over the time window.
• the All RPC upstreams exhausted lines with their network and all their fields.
• each endpoint's badge, latency, error rate and exclusion reason from the card.
• the provider's own status for the same window.

How the indexer's pool picks an endpoint#

The indexer ranks its pool on health scores when the network has more than one endpoint, at least one of them HTTP, and “Score endpoint health” is on, which is the default. Otherwise it tries the endpoints in configured order, and none of the network.rpc-pool lines, the dalp_rpc_failovers_total and dalp_rpc_upstreams_exhausted_total counters or the exhaustion error appear.

• Exclusion first. Routing leaves out an endpoint that is unhealthy, that scores below “Minimum score (0 to 1)”, or that trails the chain head by more than “Max block-head lag”. The defaults are 0.25 and 50 blocks. While any endpoint is not excluded, the pool sends calls only to endpoints that are not excluded, whatever their priority. When every endpoint is left out, the pool still serves from them in ranked order rather than refusing every call.
• Then priority. Among the remaining endpoints, a higher “Priority” wins. Among endpoints with the same priority, the higher health score wins, then configuration order.
• Health score. The platform probes each endpoint in the deployment values and scores it from 0 to 1. Whether the last probe failed to return the chain head counts for 45%, latency for 20%, the lag behind the chain head for 25% and the finalisation lag for 10%. An answer that takes 5 seconds or more scores zero on latency.
• Penalty. An endpoint that fails a call is skipped for at least one second and loses score for a while. The penalty halves every “Penalty half-life (ms)”, 60,000 by default.
• Stickiness. Within “Min switch interval (ms)”, 60,000 by default, a contender with the same priority takes over only when its score beats the current endpoint by “Switch hysteresis (0 to 1)”, 0.1 by default.
• Budget. One call may spend “Maximum endpoint attempts”, 32 by default, across retries and failovers, within “Request deadline (ms)”, 120,000 by default. “RPC retries”, 3 by default, applies per endpoint. A transaction broadcast, eth_sendRawTransaction, is not retried on the same endpoint by default, but it still fails over to the next one. A network that sets its own per-method retry overrides replaces that default.