Networks

Validation at boot#

The instance rejects the networks section at startup, before it serves traffic, when any of these hold:

• The section is empty. It needs at least one network.
• No EVM network sets default: true. The boot failure is Exactly one EVM network must have 'default: true'.
• More than one EVM network sets default: true. This boot failure names every offending network: Multiple EVM networks marked as default: <names>.
• The network that sets default: true does not also set enabled: true. The boot failure is Default network '<name>' must be enabled.

A chain ID can repeat across two entries on purpose. That lets you rotate an RPC provider, or promote a secondary alias to primary, by keeping the old entry turned off beside its enabled replacement. The indexer reads a chain ID only once, through the first enabled entry for it.

Adding or changing a network#

A network reaches the catalogue through one of two write paths, and they carry different consequences.

Deployment values. The networks section of the Helm values declares the baseline. Changing it takes a helm upgrade and a restart of the affected components, so it goes through the same change control as the rest of the release. At startup the platform adds each baseline network that its stored catalogue does not hold yet. It does not rewrite a network the catalogue already holds, so runtime changes survive a release. To apply a changed baseline entry to a network the catalogue already holds, remove the override on that network in the platform network catalogue. That puts back the entry from the current baseline.

Platform network catalogue. A platform administrator adds or edits a network at runtime, with no redeployment, on the platform networks page. A save writes the network's definition: its chain alias or chain ID, name, finality and indexing settings, RPC policy and gas block. The RPC endpoints and their credentials are a separate write to the network's RPC endpoint pool. A runtime write passes these checks before the platform accepts it:

1. The new entry must decode as a valid network definition.
2. The write names the catalogue version it expects to change. A concurrent change elsewhere fails the write instead of being overwritten.
3. The catalogue as it would stand after the write must still pass the boot validation rules, such as exactly one EVM default.
4. A write that would leave a chain ID with no enabled network, while an organisation has systems deployed on that chain ID, is refused and names the stranded chain IDs.
5. A write that moves a network to another chain ID while RPC endpoints are stored for that network is refused.

A runtime network definition has no WebSocket endpoint, and runtime pool endpoints must use HTTP or HTTPS. A network that exists only in the platform network catalogue therefore has no rpc.wsUrl, and the indexer does not index it. The transaction engine's RPC client also takes its networks and endpoints from the deployment values only, so it does not find a network that is declared nowhere else. Declare a network in the deployment values, with rpc.wsUrl, before an organisation deploys to it.

For a network declared in the deployment values, the indexer follows the runtime catalogue. It starts indexing the network when you enable it and stops when you disable it, with no restart. A disabled network releases its chain, and its card stops showing indexing state.

The runtime catalogue cannot change the default network or rpc.allowPrivateUpstreams. Change the default with a Helm upgrade, then remove the overrides on the old and the new default network in the platform network catalogue, so that the stored catalogue takes the new default.

A removal depends on where the network came from. A network that also has a baseline entry goes back to that baseline and keeps its RPC endpoints. A network that exists only at runtime leaves the catalogue, and its RPC endpoints are cleared with it. Check 4 applies first in both cases.

The boot validation does not check whether an organisation has systems on a network. A Helm upgrade that disables or removes a network therefore has no stranding check. Confirm that nothing live depends on the network before you take it out in a release.

Node and endpoint requirements#

SettleMint does not run nodes for you. You run the nodes, or contract a node provider, that serve the standard EVM JSON-RPC interface, and you list their endpoints on each network.

The indexer needs the WebSocket endpoint and has no HTTP polling fallback. A network with no rpc.wsUrl is not indexed, and the indexer logs EVM indexing needs a WebSocket endpoint that supports eth_subscribe newHeads. The indexer also reads historical state. When the endpoint keeps none, the indexer logs EVM chain RPC keeps no historical state; the chain waits for an archive RPC and that network waits until an endpoint that keeps history answers.

Endpoints added at runtime to a network's RPC endpoint pool must use HTTP or HTTPS. A runtime pool write rejects a URL that embeds credentials or carries a query string or fragment, and an endpoint that the private upstream guard refuses. When the network has a chain ID stored, the write also asks each endpoint for its chain ID and refuses one that reports a different chain ID. The probes in one write share a 15-second deadline, and an endpoint that does not answer in time is still saved. Credentials go in the endpoint's own API key or basic authentication fields and are stored encrypted.

What the platform trusts#

The platform trusts the endpoint that answers. Each call goes to one endpoint, and the first successful answer is the answer the platform uses. It does not compare answers across endpoints and does not run a light client. It does check two things: the indexer's health-ranked pool refuses an answer that reports a lower block than one it has already seen from any endpoint, and a runtime pool write refuses an endpoint whose chain ID differs from the network's. List only nodes that you, or a provider you have a contract with, operate and secure.

Private upstream guard#

The platform refuses to send a request to an endpoint whose hostname resolves to an internal address. It checks the addresses that the hostname resolves to, not only the hostname, and checks again when the cached resolution expires.

• Link-local addresses, cloud metadata services and other blocked ranges are always refused.
• Loopback, RFC 1918 private, IPv6 unique local and carrier-grade NAT addresses are refused unless the network sets rpc.allowPrivateUpstreams: true.

rpc.allowPrivateUpstreams is a deployment-time value. The platform network catalogue cannot change it, and a runtime pool on a network takes the value that the network's deployment values set. Set it to true only on a network whose nodes run on your private network.

Finality#

Every EVM network decides finality by distance from the chain head. A block is final once it is the network's finality distance below the newest block the indexer has seen. The platform does not read the finalized block tag. supportsFinalizedTag, finalityConfirmations and the chain alias choose the distance:

A network with supportsFinalizedTag: false and no confirmations fails validation outside test mode. The error ends Production chains must set finalityConfirmations>=1 to preserve a pruning safety margin.

Example: a QBFT network. QBFT finalises each block when the validators produce it and has no finalized tag. Set supportsFinalizedTag: false and finalityConfirmations: 1. A block is then final one block below the head. Webhook events that wait for finality still wait for the 64-block chain reorganisation window, unless reorgDepth sets a smaller one.

How the platform waits for finality#

The indexer records a final block for each network, the head minus the finality distance, every time it commits new blocks. Normal indexing never moves that watermark back: a lower value, for example from an endpoint that lags, leaves the stored value in place. Only a rewind after a chain reorganisation, or a rebuild of the network's indexed data, lowers it. Among the parts of the platform that wait for finality:

• Webhook events. An event that waits for finality is released for delivery once its block is at or below the bottom of the chain reorganisation window. That is the indexed block minus reorgDepth, or minus the larger of the finality distance and 64 blocks when reorgDepth is unset. On a QBFT network with finalityConfirmations: 1 and no reorgDepth, an event waits until it is 64 blocks behind the indexed block.
• New-token detection. Tokens that the platform detects on its own are read at the lower of the indexed block and the indexer's final block.
• Custody deposit reports. Inbound transfers to a custody gateway are reported only up to the RPC head minus the finality distance.

The transaction queue does not wait for finality. It completes a write once the node returns the receipt and the indexer has processed that block. Data from blocks newer than the final block can still change after a chain reorganisation.

supportsFinalizedTag, finalityConfirmations and reorgDepth, which the Console labels “Reorg depth”, are editable on a network in the platform network catalogue.

Chain reorganisation handling#

A chain reorganisation replaces recent blocks with different ones. The indexer follows each block by its hash, so it detects a replaced block at the same height. It then rolls its records back to the last block that both versions share and indexes the new blocks again.

The indexer keeps undo data for each block in a window of recent blocks. reorgDepth, 1 or more, sets how many recent blocks that window keeps. With no reorgDepth, the window is the larger of the finality distance and 64 blocks. A repair never reaches at or below the stored final block, so a window deeper than the finality distance adds no repair reach. It does delay webhook events that wait for finality, because they are released at the bottom of the window. At startup the platform logs a warning when reorgDepth exceeds the expected finality distance of a built-in chain with a recommended distance, or of a network with no finalized tag.

The indexer measures its reach from the stored final block and window, not from the current head, so it still repairs a chain reorganisation that it finds after a long downtime.

Beyond the window#

The indexer does not repair a fork that starts at or below the stored final block, or below the bottom of the retained window. It parks that network instead: the network stops advancing, its Ledger Data freezes at the last indexed block, and the other networks keep running. The log line is EVM span chain parked; its siblings continue, with the failure Repair stopped <reason> at retained floor <block>., where the reason is at-or-below-finality, below-retained-floor or beyond-repair-depth. The platform does not rewind on its own. Follow Indexer chain parked: a restart heals a park whose fork block is still inside the window, and a fork below it needs a rebuild of the network's indexed data.

The counter dalp_didx_reorg_detected counts the chain reorganisations the indexer meets, whether it repairs them or not. The gauge dalp_didx_reorg_depth holds how many blocks the last repair rewound. Both carry a chain label.

Endpoint failover and retries#

Three RPC clients make most calls to a network, and they choose endpoints in different ways:

• The indexer reads blocks and logs, and runs its event handlers, through the network's RPC endpoint pool with health ranking. The endpoints in the deployment values form the baseline pool, and a platform administrator adds, removes and weights endpoints at runtime on the platform networks page. Health scoring, weights, hysteresis and the retry settings in this section apply to this client.
• The Platform API uses the same pool, and ranks the endpoints by a score that blends each endpoint's recent successes and failures, with the listed order breaking ties. An endpoint that the persisted health data shows more than 50 blocks behind the chain head, unhealthy or below the minimum score is left out while another endpoint qualifies. A call tries the endpoints in that ranked order. Each endpoint gets timeout per attempt and retries a failure retryCount times, retryDelay apart. When every endpoint fails, the client runs through the whole list again, up to retryCount more times. A revert or another deterministic answer returns at once. weight, hysteresis and the rpc.health thresholds do not apply to it. requestRpcDeadlineMs bounds the whole call while a Platform API request waits on it.
• The transaction engine signs, broadcasts and confirms transactions, and reads the chain head and the final block, through its own client. That client uses only the endpoints in the deployment values. To give the transaction engine a second endpoint, list several entries in rpc.urls in the deployment values. Endpoints added on the platform networks page do not reach it. Endpoints added at runtime, weight, hysteresis and the rpc.health thresholds do not apply to it. How the transaction engine picks an endpoint describes it.

The Platform API and the indexer's event handlers reload a changed pool when they are notified, and a catch-up pass every 60 seconds applies a change they missed.

With a single endpoint, none of the ranking and failover in this section applies: the call retries that endpoint retryCount times and then fails.

Health scoring#

One collector measures every endpoint and stores the scores. The indexer ranks the pool on them, and the transaction engine's client reads them too. The collector runs every 60 seconds by default, and monitoring.blockchainHealthCollector.intervalSeconds changes that. A score runs from 0 to 1:

A score older than staleScoreMs is stale. That setting defaults to three collector intervals, 180 seconds, and a lower value is raised to three intervals. While another endpoint has a fresh score, the pool leaves an endpoint with a stale score out of routing. When every score is stale or missing, the indexer asks every endpoint for its block number and ranks the endpoints on head lag and latency until fresh scores arrive.

Selection and hysteresis#

With health ranking, the pool sends each call to one endpoint. The highest weight wins among usable endpoints, and health score, then configuration order, decides between endpoints with the same weight. Weight sets priority and does not split traffic. The pool leaves an endpoint out of routing when it trails the chain head by more than maxBlockHeadLag blocks, scores below minScore, or the collector reports it unhealthy. When every endpoint is left out, the pool still serves from them as a degraded fallback rather than refusing all traffic.

Hysteresis stops two endpoints of similar quality from trading places. Within minSwitchIntervalMs of the last switch, an endpoint with the same weight takes over from the current one only when its score is at least hysteresis higher. After that interval, a better score switches at once. A failed call moves to the next endpoint without waiting for either rule.

A failed call lowers that endpoint's score by 0.2 in the calling process, up to 1 in total, and the penalty halves every penaltyHalfLifeMs. The same failure takes the endpoint out of rotation in that process for retryDelay or one second, whichever is longer. A revert, an invalid request or an empty answer carries no penalty.

Retries, timeouts and deadlines#

These rpc settings bound each call through the pool with health ranking. The Platform API's client uses only timeout, retryCount and retryDelay from this table, and requestRpcDeadlineMs.

In the health-ranked pool, two timeouts in a row on one endpoint move the call to the next endpoint. A deterministic answer, such as a revert or an invalid request, returns at once with no retry and no failover. An empty answer to a receipt lookup, a transaction lookup or a log query up to the head counts as a lagging node, and the call tries the next endpoint once before it returns the empty answer.

With health ranking, when every usable endpoint has failed, the call fails with the retryable error RpcUpstreamsExhaustedError, whose message reads All healthy RPC upstreams exhausted for network <name>. Restore an endpoint, correct its configuration, or retry once the transient failures recover. The metrics dalp_rpc_failovers_total and dalp_rpc_upstreams_exhausted_total count failovers and exhausted calls per network.

How the transaction engine picks an endpoint#

The transaction engine's client keeps its own success score for each endpoint in the deployment values. Every endpoint starts at 0.5, and each call moves the score 30% of the way towards 1 on success or towards 0 on failure. The client ranks endpoints by that score, then by configuration order. It tries first the endpoints that score more than 0.25 and that the collector reports as healthy, with a fresh score of at least 0.25 and no more than 50 blocks behind the chain head. Endpoints with no fresh collector score come next. When no endpoint qualifies, the client tries every endpoint. These limits are fixed and do not follow rpc.health. A collector score counts as fresh for three collector intervals, 180 seconds by default.

A call moves through the endpoints in that order and tries each one once. A transient failure, a lost connection, a timeout, an endpoint configuration error, a method the endpoint does not support, or a refusal by the endpoint moves the call to the next endpoint. A revert or any other answer returns at once. An endpoint that answers with a rate limit cools down, and when every endpoint is rate limited the client can wait for the shortest cooldown and try again. When the last endpoint fails, the call fails with that endpoint's error. The metrics dalp_chain_evm_rpc_failovers_total and dalp_chain_evm_rpc_upstreams_exhausted_total count this client's failovers and exhausted calls.

When a transaction broadcast fails over to another endpoint, the client marks the outcome as uncertain, because the first endpoint may have taken the transaction. The transaction engine then decides what happened from the chain, as Nonce management describes.

Per-network indexing#

An EVM entry's indexing block tunes how the indexer reads that chain. Its runtime settings carry schema defaults and are editable on a network that the platform network catalogue holds. chunkSize, minChunkSize and maxChunkSize bound the adaptive read range, and verifyBlockReceipts re-checks a block that a log fetch flagged as possibly incomplete. While the indexer is behind the head it reads the next range at once.

indexing.startBlock is the exception in that block. It is a deployment-time value the platform network catalogue refuses, and it names the lowest block the indexer may start scanning at on a chain it has not read yet, for a chain whose Directory deploy height is already known. With the key absent, the DALP Instance resolves that height exactly: the earliest registered contract, else the Directory row, else block 0. On a long public chain, block 0 means walking millions of blocks before the first event the deployment cares about.

Two rules bound what it can do:

• A registered contract always wins. The floor fills the case with no rows to anchor on, and never raises the scan above a contract the indexer already knows about, because a contract that registers while the chain is catching up has to be inside the next range read.
• It only decides where a first pass starts. While a chain catches up, the indexer widens its reads and jumps over regions it has proven empty. Setting this key does not make an existing catch-up faster.

When the floor is safe, and when it is not#

Set it only where the deployment block is known from the chain's own deploy record: a non-production chain, or a public testnet whose Directory you deployed and whose deployment block you can read back.

Leave it unset, or set it to a block at or below the Directory's deployment, on any network where the indexer has to find the deployment itself. That discovery reads the Directory's own deployment and registration events to find the contracts seeded there and everything they deploy afterwards, and the same history is what bounds which implementations the indexer may still trust. The indexer never revisits a range it has passed, so a floor above those events is a floor above the discovery.

History scan floor for imported custody wallets#

indexing.custodyHistoryScanFloor is a second deployment-time value the platform network catalogue refuses. The first time the platform watches a custody wallet it was not asked to create, it scans that wallet's token deposits from block 0. This key sets the lowest block that first scan may start at. It is off by default, and an empty value leaves it unset.

Wallets the platform created start about an hour before their earliest record whatever this key says, and never below it. A floor above the chain's head is ignored for that pass and logged as an error.

Deposits to an imported wallet below the floor are not detected, and changing the floor later has no effect on watches already opened.

Gas-charging networks#

On a network that charges gas, the platform submits smart-account operations through a bundler, the service that wraps and pays for them. Two bundler values behave differently there than on the QBFT chain the base chart ships, which charges no gas. Neither failure shows on a dashboard.

• bundler.priorityFeeBaseFeeMultiplier bounds the tip a smart-account operation may carry, relative to the network's base fee, with bundler.minPriorityFee as the absolute floor. Without a bound, a network on which the platform is the dominant sender can drive its own fee suggestion far over the base fee until the sponsorship ceiling rejects the operation. The multiplier defaults to 3 and the floor defaults to 0. A negligible base fee makes the relative bound negligible too. Raise minPriorityFee on those chains to keep the bound effective.
• bundler.callGasBufferNonDeploy gives ordinary operations that do not deploy a contract headroom on top of the protocol's own gas-forwarding correction. The default is 1.3125, which gives a 1.333x total margin. Set too low, the transaction is included while the work inside it runs out of gas: the operation looks mined and did nothing.

Confirm both against the target network before it carries production traffic. The symptom in each case is an operation that never completes, with no error at the point of submission.

The platform treats a chain's own fee suggestion as an input. The backend caps the tip a UserOperation may carry at max(minPriorityFee, baseFee * priorityFeeBaseFeeMultiplier) before the operation is hashed and signed, because the fee fields are inputs to that hash and cannot be corrected afterwards. The bound applies only to operations the platform signs: an externally signed operation whose tip is under the configured floor is rejected, since raising it would invalidate the signature that covers it.

Both limits have a cost. A tip ceiling that is too tight leaves operations uncompetitive during congestion, and one that is too loose lets the tip run away again. Both show as operations that never reach inclusion, so pair any change with broadcast-latency and inclusion monitoring.

Per-network gas settings#

The two bundler settings in this section bound what a smart-account operation may pay. A separate per-network gas block decides what the platform asks an operator to deposit when a wallet cannot pay at all, how long that request stays open, and whether the chain charges for gas in the first place. Gas settings is the reference for every key, its unit, its accepted range, and its default.

Each key resolves through four layers, and the first layer that names a field decides it:

1. The network's own settings, saved through the platform network catalogue.
2. The gas block on the network entry in the deployment values.
3. The chainGasDefaults section, keyed by chain ID, carrying calibrated figures for chains the platform has already measured. Eight chain IDs ship there, so a deployment that adds one of them needs no entry of its own. Chains that ship calibrated lists them.
4. The built-in defaults.

Resolution is per field. A network that sets one value keeps every other value live against the layer beneath it, so a later calibration change still reaches that network.

zeroGas is the exception to that ladder. Left unset by every layer, it is not defaulted at all: the platform probes the chain once and caches the answer. When the probe cannot reach an answer, the chain is treated as one that charges gas, because a chain wrongly treated as free puts every write back on the path where an unfunded transaction fails at the node with no request raised and no wallet named. The cost of the opposite error is one funding request an operator declines.

What the base chart values declare#

The base chart values declare one network, keyed mintnet, and nothing else. It is a custom EVM network with chain ID 53771311147 and the name DALP, set enabled: true and default: true. It describes a QBFT network, so it sets supportsFinalizedTag: false and finalityConfirmations: 1.

Its RPC settings point at in-cluster node service names: two HTTP endpoints, http://besu-rpc-0.besu-rpc-headless:8545 and http://besu-rpc-1.besu-rpc-headless:8545, and the WebSocket endpoint ws://besu-rpc-0.besu-rpc-headless:8546. The chart does not install those nodes. The entry also turns on JSON-RPC batching of up to 20 calls per request. For the indexer, it turns on a cache of historical responses from blocks at least 128 below the head, each kept for 24 hours.

Override these values before a deployment goes live on its own network:

An environment overlay can add more EVM networks. The merged configuration must still pass the boot validation rules.

Failure modes#

Frequently asked questions#