Upgrade a deployment to DALP 3.2

Before you upgrade#

Finish in-flight work#

3.2 runs workflows on a new workflow engine, and the chart no longer deploys the workflow engine that 3.1 and 3.0 ran. A workflow that the earlier workflow engine is still running at the upgrade does not continue on 3.2, because no 3.2 component reads the earlier engine's state. Plan a window in which nobody starts new operations. The platform has no setting that blocks new operations, so agree the window with the people and systems that start them. Before the upgrade, let every operation in progress reach a final state: transaction requests, issuance runs, corporate action runs, invitations and identity recoveries. GET /api/v2/transaction-requests lists transaction requests with their status, GET /api/v2/token-issuance-runs/{runId} reads one issuance run, and GET /api/v2/tokens/{assetId}/corporate-actions lists an asset's corporate action runs.

Update Ripple policies#

3.2 offers Ripple custody only when global.dalpApp.config.custody.providers.ripple.enabled, the CUSTODY_PROVIDER_RIPPLE_ENABLED variable, is true, and it ships false. With it off, existing Ripple custody configurations keep signing, a new one is refused, and pods whose SIGNER_PROVIDER is ripple refuse to start. A deployment that uses Ripple sets it to true in the values for the upgrade.

An organisation that uses Ripple custody files an intent with Ripple for each operation, and DALP names the operation in the intent's dalpKind property. 3.1 named most direct contract writes after the contract function, such as contract.grantRole, and 3.2 names them after the operation, such as system.grant-role or token.grant-role. 3.2 also marks a role change with dalpRolesUndescribed=true when it cannot list every role that the change grants or revokes. A Ripple policy that matches the 3.1 names stops matching after the upgrade, so a fallback policy that was not written for the operation can approve or refuse it.

Update the policies before the upgrade. Ripple policy reference lists the changed names, and Policies for role changes describes the policy that the role mark needs.

Update Fireblocks settings#

3.2 refuses a Fireblocks Raw signing request by default, before it reaches Fireblocks: an EVM operation signs as a Typed Message or sends a Contract Call. FIREBLOCKS_ALLOW_RAW_SIGNING set to "true" turns Raw signing back on, for a workspace whose policy allows it. 3.2 also attaches a Fireblocks tag to every vault account it creates, named by FIREBLOCKS_DALP_VAULT_TAG_ID, and the policy rule that allows the DALP API user to sign from those vaults matches that tag. With the value empty, new vaults carry no tag and their requests reach the catch-all block rule. Before the upgrade, create the tag, set its id, tag the vaults DALP created earlier, and check your rules against Fireblocks policy.

Move your values to the 3.2 chart#

The 3.2 chart splits the Platform API into four roles inside one subchart and drops the subcharts that 3.1 ran beside it. A values file written for 3.1 keeps keys that the 3.2 chart no longer reads. Move each of them:

The chart fails the render on a config.app block under dapp or dapi, because the Console and the Platform API must read one public URL from global.dalpApp.config.app. It ignores the other 3.1 keys you leave in the values file, so remove each one yourself. A 3.1 dapi.resources block no longer sizes the API pods, because dapi.v1.resources and dapi.v2.resources have their own defaults and take precedence. Move your sizing to those keys and to the resources of dapi.indexerDeployment, dapi.runnerDeployment and dapi.schedulerDeployment. dapi.routesReplicaCount is ignored: set replicas under dapi.v1 and dapi.v2. dapi.ingress, dapi.httpRoute and dapi.openShiftRoute keep their 3.1 meaning, and dapi.config is merged over global.dalpApp.config for every Platform API role.

A 3.1 override under networks.default no longer reaches the shipped network, so an RPC endpoint you set there must move to networks.mintnet. 3.2 ignores platform.instance and stamps every log, metric and trace with global.dalpInstance and global.deploymentEnvironment, which both default to local. Set them to this deployment's name and its environment: development, test, staging, demo or production.

Client addresses behind a proxy. The 3.1 chart read the client address from the forwarding header that an NGINX ingress controller writes, and 3.2 ships global.dalpApp.config.trustedProxy.profile as none, which discards every forwarding header. Behind a proxy, none keys sign-in rate limits and the request addresses in audit records on the proxy's address. Set nginx, envoy or openshift to match the controller in front of the Console and the Platform API.

Network endpoints. 3.1 sent every network call through an RPC gateway that the chart deployed, and each network's rpc.url pointed at that gateway. 3.2 deploys no gateway. Each network under global.dalpApp.config.networks takes its node endpoints directly in rpc.urls, a list. Copy the node endpoints that you configured as gateway upstreams into rpc.urls. 3.2 refuses a node endpoint that resolves to a private or loopback address unless the network sets rpc.allowPrivateUpstreams to true, so set it on a network whose nodes run inside your own network. Networks describes the setting. Set rpc.wsUrl on every network as well: the 3.2 indexer follows each network through that subscription endpoint and does not fall back to polling an HTTP endpoint.

Custody execution mode. 3.1 let you choose whether DFNS and Ripple sign as the provider's own transactions or as approvals that DALP submits, and a later 3.1 patch release added a bundler setting that did the same for Fireblocks. 3.2 has no such settings: each custody provider declares how it signs on each network, and the platform uses that. Remove the settings and their environment variables, DFNS_EXECUTION_MODE, RIPPLE_EXECUTION_MODE and BUNDLER_PROVIDER_NATIVE_SUBMISSION.

Environment variables. The 3.2 configuration no longer reads these variables. Remove them from any environment file or Secret you maintain:

• DIDX_URL, DIDX_PORT, DIDX_HOSTNAME, DIDX_INTERNAL_HOSTNAME, DDWF_URL, DDWF_PORT, DDWF_HOSTNAME and DDWF_INTERNAL_HOSTNAME. The chart sets the internal addresses of the four roles itself.
• DATABASE_POOL_MIN and DATABASE_POOL_MAX. 3.2 sizes one pool for each role, with these defaults: DATABASE_POOL_MAX_API 40, DATABASE_POOL_MAX_INDEXER 24, DATABASE_POOL_MAX_RUNNER 16 and DATABASE_POOL_MAX_SCHEDULER 32. Check the sum of the pools across all pods against your database's connection limit.

Set the values that 3.2 adds#

3.1 had neither of these values, so a 3.1 values file sets neither, and the upgrade fails until both are set:

• global.dalpApp.env.DALP_GOVERNANCE_DOMAIN_SCOPE names the domain under which the deployment signs governance poll ballots. helm upgrade fails the render while it is empty or local. Set a name unique to this deployment, as Governance ballot domain describes.
• global.dalpApp.env.CUSTODY_CREDENTIAL_IDENTITY_KEY is the key under which the platform hashes custodian credential identities, at least 32 characters long. The chart ships a development placeholder, and the Platform API pods, which run in production mode, refuse to start on it. Set a value unique to this deployment and keep CUSTODY_CREDENTIAL_IDENTITY_KEY_ID at v1.

Decide on Ask AI#

3.2 adds Ask AI, a question-and-answer chat on the documentation site that the chart deploys. The documentation chart reads its model provider key from the same Secret entry that the Console assistant uses, global.dalpApp.env.ASSISTANT_OPENROUTER_API_KEY. The chart points the documentation at that entry by default, through docs.openRouter.existingSecret and docs.openRouter.existingSecretKeys.apiKey. The entry is empty by default, and Ask AI stays off while it is empty. As soon as you set the key for the Console assistant, Ask AI turns on too, for anyone who can reach the documentation site, and each answer is billed to that key. Ask AI allows each client 10 questions a minute and all clients together 300 an hour. The chart passes global.dalpApp.env.REDIS_URL to the documentation pods, so they keep these counts in Redis and share them. While Redis does not answer, Ask AI refuses questions.

Choose one of three settings in the same values change that sets the key:

• To share the key, set ASSISTANT_OPENROUTER_API_KEY and leave the docs.openRouter values as they ship.
• To keep Ask AI off, set docs.openRouter.existingSecret to "" and leave docs.openRouter.apiKey empty.
• To bill Ask AI to a separate key, set docs.openRouter.existingSecret to the name of a Secret you manage and docs.openRouter.existingSecretKeys.apiKey to the entry that holds the key.

Ask AI provider key describes each value and the combinations that fail the render.

Decide on legacy asset creation#

The seven asset families of the original catalogue, bond, equity, fund, stablecoin, deposit, real-estate and precious-metal, cannot be created unless the deployment turns legacy asset creation on. A request to create a token of one of these types, or to install its factory, is refused, and organisations and networks deployed while the setting is off get no factories for them. Tokens and factories that already exist stay on chain: the platform removes nothing.

The setting arrived in the 3.1.19 patch release, so a deployment on 3.1.19 or later keeps the value it already runs with. 3.0 and the 3.1 releases before 3.1.19 had no such setting, so a deployment on one of them whose users still issue these families loses that ability at the upgrade unless you turn the setting on. To keep issuing them, set global.dalpApp.config.features.legacyAssetCreation to "true", or the environment variable FEATURE_LEGACY_ASSET_CREATION to true, in the values for the upgrade. The setting is false by default.

Decide on the offerings surfaces#

The Console shows the Offerings sidebar item and the issuer offering pages only when global.dalpApp.config.features.tokenSale is true and the organisation has deployed the token sale add-on. The 3.0 and 3.1 charts shipped tokenSale: false, and the 3.2 chart ships tokenSale: true. A deployment whose values file does not set the key therefore has the setting on after the upgrade, and an organisation that deploys the add-on gets the Offerings surfaces. Helm keeps a value that your values file sets, so a deployment whose overlay sets tokenSale to false stays without the Offerings surfaces. The same setting switches on fund dealing, the subscription and redemption requests of a fund and the settlement of its dealing days, so with it false the Dealing tab, the Dealing requests card and the Funds section of Invest are absent too. Corporate actions do not depend on it. Check your overlay for the key. Remove it or set it to true if issuers on this deployment run offerings or fund dealing, and set it to false to keep the surfaces off.

Decide which application layers to switch on#

3.2 sells its application layers as licensed modules, and each one has a switch under global.dalpApp.config.features. Switch application layers on or off lists every switch with its default. The Secondary market (tradingVenue), Reporting and finance operations (regulatoryReporting) and Market surveillance (surveillance) are off by default. A module that is off has no screens and its routes answer 404, so for each module your licence includes, set its switch to true in the values for the upgrade.

Multi-signer wallets belong to the Self-custody module, features.custodyOps, which is on by default. A deployment that sets it to false accepts a signer set only when it has one signer and a threshold of one. It refuses any other signer set with HTTP 403 and the error DALP-12261 MULTISIG_NOT_ENABLED, whose message is “Multi-signer wallets are switched off on this deployment”. An organisation owner who switches Self-custody off for one organisation gets the same limit for that organisation, refused with custody.multisig.OFF_FOR_ORGANIZATION. Multisig explains what a multi-signer wallet is.

Check that the custody provider signs user operations#

At start-up, the API, runner, indexer and scheduler pods check that the platform's custody provider, SIGNER_PROVIDER, can sign on each enabled network. With account abstraction on, they also check that the provider can sign user operations, the transactions that smart accounts send. A provider that cannot stops the pods from starting with an error that names the chain ID and ends with Set SIGNER_PROVIDER to a provider that signs user operations. A deployment that runs without account abstraction can set FEATURE_ACCOUNT_ABSTRACTION=false instead. Confirm before the upgrade that your provider supports every enabled network, or that you can turn account abstraction off.

Prepare the custody census#

The custody census is the 3.2 migration step that records which custodian holds each existing wallet before it moves wallets to custody accounts. It archives wallets that hold no signing key. When it cannot prove the custodian of any other wallet, it stops with the message The custody census still has <n> unresolved row(s), followed by the count of each kind of row it could not resolve, and the upgrade does not complete. If the deployment still has the legacy transaction signer database, which a deployment first installed before 3.1 can have, set TXSIGNER_DATABASE_URL under dapp.migrator.env. The Job reads that database as evidence of which wallets DFNS or Fireblocks hold. The runtime pods do not read this value.

Upgrading from 3.0#

A 3.0 deployment that upgrades directly to 3.2 needs every step above, and these steps from the 3.1 releases as well. Make them part of the same values change.

Legacy asset creation. 3.0, and 3.1 before 3.1.19, created the seven original asset families without a setting. Decide on the setting before the upgrade, as Decide on legacy asset creation describes.

Custody records per organisation. 3.0 configured one custody provider for the whole deployment. Since 3.1 each organisation has its own custody provider configuration. Keep SIGNER_PROVIDER and the provider credentials set to the custodian that holds the existing keys during the upgrade. A 3.0 deployment is the most likely to still have the legacy transaction signer database, so read Prepare the custody census before you upgrade.

Data sealed under an earlier password. If the migration Job stops with the message ENCRYPTION_PREVIOUS_MASTER_PASSWORD is required, some secrets or KYC files are still sealed in an older format or under another password, and the message names how many. Give the Job the password that sealed them through dapp.migrator.env.ENCRYPTION_PREVIOUS_MASTER_PASSWORD, and it seals them again under ENCRYPTION_MASTER_PASSWORD. Only the migration Job receives this value. Encryption at rest describes the procedure.

Fireblocks Gas Station. Since 3.1.12, new Fireblocks vault accounts are enrolled in Fireblocks Gas Station by default. A deployment on 3.0, or on a 3.1 release before 3.1.12, that uses a Fireblocks workspace where Fireblocks has not set up Gas Station sets FIREBLOCKS_AUTO_FUEL to "false", because a vault account creation that Fireblocks rejects cannot be retried.

Network defaults for DFNS and Fireblocks. DFNS_DEFAULT_NETWORK, added in 3.1, names the DFNS network on which a new wallet is created when the request names no network and carries no chain ID that maps to a DFNS network, and in 3.2 it defaults to Ethereum. A DFNS custody configuration whose “Default network” field is set uses that field instead. A request that carries a mapped chain ID creates the wallet on that chain's DFNS network instead, as Which network a DFNS wallet is created on describes. FIREBLOCKS_DEFAULT_ASSET_ID, also added in 3.1, names the Fireblocks asset used when a request names no network. A wallet is bound to its network when it is created. On a test network, set both to that network, such as EthereumSepolia for DFNS or ETH_TEST5 for a Fireblocks Sepolia workspace.

Integrations. Clients built against 3.0 need the changes on Upgrade an integration to DALP 3.2 as well.

Take a recovery point#

A Helm rollback does not reverse the database migrations of the upgrade, as Rolling back explains. Once in-flight work has finished, take a database backup and keep it until you have confirmed the 3.2 deployment. Backup and recovery covers what to back up.

Run the Helm upgrade#

Run helm upgrade with the 3.2 chart version and your updated values:

helm upgrade dalp oci://pkg.distribution.settlemint.com/settlemint/dalp \
  --version <3.2 chart version> \
  --namespace dalp \
  --values my-values.yaml \
  --wait

The migration Job runs first, as Upgrades and migrations describes, and Helm then replaces the earlier workloads with the 3.2 ones. Under the default pre hook, a catch-up run of the same migration follows the rollout and can still fail the release, so pass --wait to hold the catch-up until the new workloads are ready. When the command returns, check that each Deployment has finished rolling out:

for d in dalp-api-v1 dalp-api-v2 dalp-indexer dalp-indexer-rebuild dalp-runner dalp-scheduler dalp-dapp; do
  kubectl rollout status deployment/"$d" --namespace dalp
done

Leave dalp-indexer-rebuild out of the list when dapi.indexerDeployment.rebuild.enabled is false, as the OpenShift overlay sets it.

The Platform API's workloads change name, because the one API workload and the two subcharts of 3.1 become separate Deployments in 3.2. With the default dapi.fullnameOverride of dalp-dapi, the 3.2 Deployments are named as follows:

The API Services are dalp-api, which serves /api/v2, and dalp-api-v1. The chart's own Ingress, Route and NetworkPolicy objects follow the new names. Anything outside the chart that names a 3.1 object, such as an alert rule, a dashboard, a NetworkPolicy you added, an HPA or capacity rule, or a client inside the cluster that dials the dalp-dapi Service, needs the new name.

3.2 also runs no RPC gateway, so the upgrade removes the dalp-erpc Deployment and, where you turned it on, dalp-nodecore, together with their Services and any Ingress or Route you enabled for them. Search your alert rules, dashboards and NetworkPolicies for dalp-erpc and dalp-nodecore. Move an egress rule that allowed traffic to the gateway onto the API, indexer, runner and scheduler pods, which call the node endpoints in rpc.urls directly, and remove the rules and panels that watched the gateway. A client outside the platform that used the gateway's own hostname needs a node endpoint of its own. The upgrade also starts the PostgreSQL exporter, postgres-exporter.enabled, on by default, which holds one direct database session.

The chart has no step that sequences the cutover. Helm creates the 3.2 Deployments and Services and removes the 3.1 ones in the same release, except the 3.1 workflow Deployment, Service and ConfigMap, which Helm keeps until you delete them, as Verify the upgrade describes. The API answers only once the new API pods are ready.

The upgrade does not delete the data that the earlier workflow engine stored in its persistent volumes and in object storage. It stays until you delete it. Delete it once you no longer need a way back to 3.1.

After the upgrade#

Watch the index rebuild#

3.2 changes how the platform indexes chain data, so the first 3.2 start builds a new index next to the existing one. The existing index keeps answering reads while the new one replays the history of each enabled network, and the platform promotes the new index only when it has caught up. A Helm release that reports success does not mean the rebuild has finished.

How long the rebuild takes depends on the number of enabled networks, the blocks and events to replay, and how fast your RPC nodes and database answer. The platform gives no fixed duration. Follow it in three places:

• In the Console, open Platform status, open the details of the Data freshness panel, and wait for each network to move from Reindexing to Healthy.
• The metric dalp_indexer_build_progress_ratio runs from 0 to 1 for the index being built. dalp_indexer_build_failures_total counts failed build steps, and dalp_indexer_promotion_age_seconds reports the age of the last promotion.
• A client that sends Prefer: dalp-fail-on-indexer-reindexing on POST /api/v2/organizations, POST /api/v2/organizations/bootstrap, POST /api/v2/invitations or POST /api/v2/invitation-acceptances receives HTTP 503 with code platform.http.REINDEXING (DALP-12331) while the network's index is rebuilding, instead of the request going ahead. The response carries data.retryAfterSeconds and a Retry-After header. The refusal is retryable.

Run System updates#

The Helm upgrade changes the platform, not the contracts of each organisation's system on chain, so 3.2 does not require a System updates run. An organisation's system keeps running on the contract versions it has, and each organisation decides when to run System updates on each network it uses. An organisation runs it when it needs a token feature, add-on or document anchoring that the 3.2 contracts add.

Until an organisation runs System updates on a network:

• Its installed contracts on that network stay on the versions they run.
• The token feature factories and add-ons that 3.2 adds are not on its system, among them the factories for holds, supply delegation, interest and dividend yield, distribution waterfalls and register lines, and the custody gateway add-on. Attaching a token feature whose factory the system lacks is refused with HTTP 409 TOKEN_FEATURE_FACTORY_NOT_FOUND (DALP-9465). Creating a token from a template that uses such a feature is refused before anything is signed, with HTTP 409 TOKEN_CREATE_FEATURE_FACTORY_NOT_REGISTERED (DALP-207901), and Issue asset marks the template as needing a feature factory. Every system template uses holds. Reading the custody gateway on a system without the add-on answers HTTP 404 GATEWAY_ADDON_NOT_DEPLOYED (DALP-9813). Anchoring a document is refused with HTTP 409 ANCHOR_WRITE_REFUSED (DALP-0749) until the system has the document anchor registry. A System manager can also install one factory or add-on at a time from Factories, without a full run. The custody gateway add-on is not offered there: install it with System updates or with POST /api/v2/system/addon-factories.
• The run's data steps stay outstanding: price feeds for assets that lack one, the organisation's trust for the claim topics it issues, topic scheme updates, investor eligibility topic signatures, escrow claims and the advanced accounts infrastructure.
• The organisation sees the shipped reporting obligations but has no editable copies of them, because the run is what makes those copies.

System updates compares the organisation's system with the newest release directory on the network. An Admin or a System manager of each organisation opens System updates in the Console, reviews the components marked Upgrade available or Not installed, and starts the run. System updates describes the run, which cannot be undone once its steps are written to the network.

Verify the upgrade#

Check each of these after the upgrade:

1. helm upgrade returned without an error. Under the default pre hook, that means the migration Job succeeded before Helm touched the workloads, and the catch-up run succeeded after the rollout.
2. helm history dalp --namespace dalp lists the 3.2 chart version as the deployed revision.
3. Each Deployment in the rollout loop above has finished rolling out, and kubectl get deployments --namespace dalp no longer lists dalp-dapi, dalp-didx or dalp-erpc. The 3.1 workflow Deployment dalp-ddwf-v<version>, its Service of the same name and its dalp-ddwf-v<version>-config ConfigMap carry Helm's keep policy, so the upgrade leaves them running. Delete them once the 3.2 runner is ready: kubectl delete deployment,service dalp-ddwf-v<version> --namespace dalp and kubectl delete configmap dalp-ddwf-v<version>-config --namespace dalp.
4. A /api/v1 request and a /api/v2 request both answer through your Ingress, Route or HTTPRoute. The chart sends /api/v2 to the v2 fleet and the rest of /api to the v1 fleet.
5. Platform status shows each enabled network as Healthy in the Data freshness panel, once the index rebuild has finished.

Rolling back#

helm rollback returns the Kubernetes objects to the manifests of the release you upgraded from. It does not reverse the database migrations, it does not restore the earlier workflow engine's state, and it does not undo a System updates run, which is written on chain. The 3.1 code then runs against a database that 3.2 has migrated. To return to 3.1, restore the database backup you took before the upgrade, then roll the release back. Any operation that ran on 3.2 after the backup is not in the restored database, while its on-chain results stay on the ledger, so compare the restored records with the ledger before you reopen traffic, as Point-in-time recovery describes. Rolling back after a bad upgrade explains why a rollback leaves the schema as it is.