Skip to main content

Run several networks and recover stuck transactions

Move indexing between node providers from the Console, run one organisation on several networks and recover stuck transactions with the evidence shown.

Your operations team moves indexing, which reads each network's chain data into the platform, to another node provider from the Console, and recovery bumps or retries a stuck transaction, or holds it for a person while it may still land. A lagging node must not feed stale balances to the people who act on them, and a stuck payment must not be chased across logs by hand. Each network's records stay apart, one organisation can run on several networks, and Platform status shows the evidence behind every verdict.

Each network is indexed and recovered on its own, and every verdict shows its evidence

A platform administrator edits the endpoints each network is indexed from without a restart. A network that cannot be indexed stops on its own, and recovery never files a transaction again on its own while it may still land.

Indexing endpoints
A Console pool per network
Lagging node
Left out of indexing
Stalled network
Stops alone, others continue
Platform API
Four roles that scale apart

Before and after

Before 3.2: changing a node provider means a values change and a rollout. When one network's node falls behind, an operator finds out from a client who sees an old balance, and a transaction that stops moving is chased by hand across the node, the queue and the logs.

With 3.2: A platform administrator manages the node endpoints each network is indexed from in the Console, and Ledger Data, the platform's queryable copy of each network's chain data, leaves out an endpoint that falls behind. A network that stalls stops alone. Recovery checks every stuck transaction against every configured endpoint before it raises a fee, retries or holds the request for a person, and Platform status lists the requests, transactions and workflows behind each verdict.

Each network is indexed, read and recovered on its own paths, and each fails and recovers on its own.A platform administrator manages the endpoints each network is indexed from in the Console: hosted providers or the institution's own node. Ledger Data indexing leaves out an endpoint that lags more than 50 blocks behind by default, and a network that stalls stops alone while the others keep indexing. Transactions and most Platform API reads use the endpoints set in the deployment values, and a stuck transaction gets a fee bump or a retry, or is held for a person, after recovery asks every configured endpoint about it. Platform status shows the requests, transactions and workflows behind each verdict, with Force retry and Force fail on a stuck request.Each network is indexed, read and recovered on its own paths, and each fails and recovers on its own.A platform administrator manages the endpoints each network is indexed from in the Console: hosted providers or the institution's own node. Ledger Data indexing leaves out an endpoint that lags more than 50 blocks behind by default, and a network that stalls stops alone while the others keep indexing. Transactions and most Platform API reads use the endpoints set in the deployment values, and a stuck transaction gets a fee bump or a retry, or is held for a person, after recovery asks every configured endpoint about it. Platform status shows the requests, transactions and workflows behind each verdict, with Force retry and Force fail on a stuck request.

Node endpoints managed in the Console feed indexing for each network, recovery asks every configured endpoint about a stuck transaction, and Platform status shows the requests and transactions behind each verdict.

Move indexing to another node provider without a redeploy

A platform administrator edits a network's node endpoints on Platform networks in the Console, and indexing picks up the change without a restart. On an instance SettleMint operates, a platform administrator may be SettleMint staff. An endpoint can be a hosted provider that authenticates with an API key header, or your own node behind basic authentication. Credentials are stored encrypted and never shown again. Each endpoint card shows its health, latency and error rate, and why an endpoint is left out of routing.

A platform administrator manages each network's node endpoints.

A node provider on one network starts to lag. Ledger Data indexing scores it below its cutoff, which by default leaves out an endpoint that trails the chain head by more than 50 blocks, and moves to the next usable endpoint, and the card marks the lagging one as excluded with the number of blocks it trails. The administrator adds a second provider to the pool from the same sheet, and the platform refuses it if it answers with another chain id or resolves to a private address the network does not allow. An endpoint that rate-limits is cooled down and the call moves to the next one at once.

Each pool change leaves an audit record. Transactions and most Platform API reads use the endpoints in the deployment values, which a Console edit does not change, and a network the deployment values do not declare with a subscription endpoint is not indexed.

Run one organisation on several networks

An organisation can hold one system per network. An organisation owner turns on multi-network support, picks an enabled network from the platform catalogue, and the platform deploys the organisation's system there and shows its progress. A platform administrator decides with Offer network choice at onboarding whether new organisations can start on a network other than the platform default.

An Organisation owner adds a second network for a new bond programme. The platform checks that the organisation can sign there, through the owner's custody account or a managed provider bound to that network, and refuses the add otherwise. Once the system is active, the Console's Network menu switches between the two.

The organisation keeps one set of members and settings across its networks. A request that names no network acts on the organisation's default network, and a request for a network the organisation cannot use is refused and never falls back to the default. An owner cannot turn multi-network support off while an added network is in use.

Keep each network's records apart when one stalls

Ledger Data follows each block near the chain tip by its hash, so a replacement block at the same height is detected and rolled back to the common ancestor before the new blocks are replayed. A network that cannot be indexed stops on its own and retries from its last indexed block, while every other network keeps indexing.

A node on one network keeps answering but stops sending new blocks. Ledger Data counts that as a transport failure and reconnects, and after 15 minutes without a committed block the network's card reads “Indexing stalled” with how long it has waited. The other networks' cards still read live. When the endpoint recovers, indexing continues from the last block it recorded. A platform administrator can also pause one network's indexing with Pause indexing and keep its data readable.

A replacement that reaches below the chain's final block or deeper than the configured repair depth is not repaired automatically, and that network waits for an operator.

Recover a stuck transaction without filing it twice

The platform checks every request that has not moved for five minutes. A passing fault, such as an unavailable node, puts it back on the queue within a retry budget. On a network with a public mempool, a signed transaction that misses inclusion gets a higher fee without a change to its call or its nonce. Before recovery calls a transaction dropped, it asks every configured endpoint for each hash the request recorded. Recovery never files a request again on its own while it may still land: when safe retries run out, it is held as a dead letter while the platform keeps watching for a late receipt. The bundler that submits smart-account transactions also fills a nonce its own wallet skipped, so one gap does not stop its later bundles.

An Asset manager mints during a fee spike, and the mint sits in the mempool. After three blocks, recovery raises each fee field by at least 12.5 percent and broadcasts the replacement at the same nonce. The replacement confirms, and the request completes as the original mint. Had the request ended as a dead letter that can no longer land, an Organisation owner could open it on Platform status and use Force retry or “Force fail” with a written reason.

Force retry and Force fail are refused while signed bytes for the request may still reach the chain, unless an API caller states that the custodian stopped a call the chain does not show, which DALP does not check. A workflow whose dependency stays down retries with backoff a bounded number of times and can still end failed.

See the evidence behind each status verdict

Each verdict on Platform status opens a detail screen that lists the requests, chain read samples, transactions, workflows and webhook endpoints behind it. FIX sessions, email, batch jobs and scheduled jobs have their own panels, and an owner or administrator can export a redacted debug bundle for support. The cluster's dashboards and alert rules ship with the deployment and cover Ledger Data falling behind the network, a wedged transaction lane, a funding pause that never resumes, and database and upgrade faults.

The Workflows panel on an Organisation owner's Platform status turns amber when 20 of every 1,000 completed workflows stall. The owner opens its Details and sees which workflows stalled, then opens Transactions, where a failed row shows the decoded contract error behind it.

Platform status shows only to an Organisation owner or a Platform administrator, and it reads only, except the recovery actions on a transaction.

Size each kind of work for its own load

The Platform API runs as four roles: serving requests in separate v1 and v2 fleets, indexing the network, durable work and scheduled work. The request and durable-work roles autoscale, indexing runs a fixed pair by default with its own pods for rebuilds, and the scheduler runs one pod. Durable work runs inside the platform, with no separate workflow engine cluster to operate.

A large corporate action run adds durable-work pods while the request fleets keep their own capacity, and a slow index rebuild never takes request pods with it. Each role sizes its own database pool, and the chart refuses a configuration whose connection pools do not fit the database's connection limit.

Upgrading

3.2 deploys no RPC gateway. Each network takes its node endpoints directly, a node on a private address needs that network to allow it, and every network needs a subscription endpoint for indexing. 3.2 also replaces the workflow engine, so let every operation in progress reach a final state before the upgrade, and plan a window in which nobody starts new ones. Sizing moves to the four Platform API roles, and the index rebuilds after the upgrade while reads keep using the current data. Multi-network support stays off for each organisation until an owner turns it on. Upgrade a deployment to DALP 3.2 has each step, and Networks describes the network settings.

Why it matters

  • Banks: A bank that runs its own nodes can index from them behind basic authentication, and moving indexing to another provider is a Console edit with an audit record.
  • Exchanges, CSDs and market infrastructures: A stalled network stops alone. A stuck settlement transaction is recovered or held for a person, and recovery never files it again on its own while it may still land.
  • Partners: one organisation runs on several networks behind the same API, and a request can name the network it acts on.

Next steps

Was this page helpful?