# Upgrade to 3.2

Source: https://docs.settlemint.com/docs/deployment/upgrade-to-3-2
The operator runbook to move a self-hosted DALP 3.1 or 3.0 deployment to 3.2: backup, values to move, renamed workloads, the index rebuild and System updates.

This runbook moves a self-hosted DALP deployment from 3.1, or directly from 3.0, to 3.2 with Helm. Plan it as a maintenance window: the upgrade replaces the Platform API workloads and the workflow engine, does not carry running workflows across, and can be reversed only by restoring the database backup you take before it, as [Rolling back](#rolling-back) explains.

Changes that affect API clients, the SDK and the CLI are on [Upgrade an integration to DALP 3.2](https://docs.settlemint.com/docs/api-reference/guides/upgrade-to-3-2). [Upgrades and migrations](https://docs.settlemint.com/docs/deployment/upgrades-migrations) explains the database migration Job that every upgrade runs.

## Which steps apply to you [#which-steps-apply-to-you]

`helm list --namespace dalp` shows the version you run in its CHART column. A deployment on 3.1.19 or later follows the steps in the table that apply to it. A deployment on 3.0 also follows [Upgrading from 3.0](#upgrading-from-30), which holds the steps from the 3.1 releases that a direct upgrade skips. A deployment on an earlier 3.1 patch release needs two of those: the legacy asset creation decision before 3.1.19, and the Gas Station setting before 3.1.12. You can upgrade from 3.0 directly to 3.2 without installing 3.1 first.

| Step | Who it applies to |
| --- | --- |
| [Finish in-flight work](#finish-in-flight-work) | Every deployment |
| [Update Ripple policies](#update-ripple-policies) | Deployments where an organisation uses Ripple custody |
| [Update Fireblocks settings](#update-fireblocks-settings) | Deployments that use Fireblocks custody |
| [Move your values to the 3.2 chart](#move-your-values-to-the-32-chart) | Every deployment with its own values file |
| [Set the values that 3.2 adds](#set-the-values-that-32-adds) | Every deployment |
| [Decide on Ask AI](#decide-on-ask-ai) | Every deployment |
| [Decide on legacy asset creation](#decide-on-legacy-asset-creation) | Deployments on 3.0, or on a 3.1 release before 3.1.19, whose users issue the original asset families |
| [Decide on the offerings surfaces](#decide-on-the-offerings-surfaces) | Every deployment |
| [Decide which application layers to switch on](#decide-which-application-layers-to-switch-on) | Every deployment |
| [Check that the custody provider signs user operations](#check-that-the-custody-provider-signs-user-operations) | Deployments with account abstraction on |
| [Prepare the custody census](#prepare-the-custody-census) | Every deployment |
| [Upgrading from 3.0](#upgrading-from-30) | Deployments on 3.0, and the two early 3.1 cases in this table |
| [Take a recovery point](#take-a-recovery-point) | Every deployment |
| [Run the Helm upgrade](#run-the-helm-upgrade) | Every deployment |
| [Watch the index rebuild](#watch-the-index-rebuild) | Every deployment |
| [Run System updates](#run-system-updates) | Each organisation, on each network, when it wants the 3.2 contracts |

Make the values changes ahead of the day. On the day, start the quiet window, let in-flight operations finish, take the database backup, then run `helm upgrade`. A backup taken before the drain misses the records the drain writes.

## Before you upgrade [#before-you-upgrade]

### Finish in-flight work [#finish-in-flight-work]

3.2 runs workflows on a new workflow engine, and the chart no longer deploys the workflow engine that 3.1 and 3.0 ran. A workflow that the earlier workflow engine is still running at the upgrade does not continue on 3.2, because no 3.2 component reads the earlier engine's state. Plan a window in which nobody starts new operations. The platform has no setting that blocks new operations, so agree the window with the people and systems that start them. Before the upgrade, let every operation in progress reach a final state: transaction requests, issuance runs, corporate action runs, invitations and identity recoveries. `GET /api/v2/transaction-requests` lists transaction requests with their status, `GET /api/v2/token-issuance-runs/{runId}` reads one issuance run, and `GET /api/v2/tokens/{assetId}/corporate-actions` lists an asset's corporate action runs.

### Update Ripple policies [#update-ripple-policies]

3.2 offers Ripple custody only when `global.dalpApp.config.custody.providers.ripple.enabled`, the `CUSTODY_PROVIDER_RIPPLE_ENABLED` variable, is `true`, and it ships `false`. With it off, existing Ripple custody configurations keep signing, a new one is refused, and pods whose `SIGNER_PROVIDER` is `ripple` refuse to start. A deployment that uses Ripple sets it to `true` in the values for the upgrade.

An organisation that uses Ripple custody files an intent with Ripple for each operation, and DALP names the operation in the intent's `dalpKind` property. 3.1 named most direct contract writes after the contract function, such as `contract.grantRole`, and 3.2 names them after the operation, such as `system.grant-role` or `token.grant-role`. 3.2 also marks a role change with `dalpRolesUndescribed=true` when it cannot list every role that the change grants or revokes. A Ripple policy that matches the 3.1 names stops matching after the upgrade, so a fallback policy that was not written for the operation can approve or refuse it.

Update the policies before the upgrade. [Ripple policy reference](https://docs.settlemint.com/docs/deployment/ripple-policy#operation-names-that-changed-from-31-to-32) lists the changed names, and [Policies for role changes](https://docs.settlemint.com/docs/deployment/ripple-policy#policies-for-role-changes) describes the policy that the role mark needs.

### Update Fireblocks settings [#update-fireblocks-settings]

3.2 refuses a Fireblocks Raw signing request by default, before it reaches Fireblocks: an EVM operation signs as a Typed Message or sends a Contract Call. `FIREBLOCKS_ALLOW_RAW_SIGNING` set to `"true"` turns Raw signing back on, for a workspace whose policy allows it. 3.2 also attaches a Fireblocks tag to every vault account it creates, named by `FIREBLOCKS_DALP_VAULT_TAG_ID`, and the policy rule that allows the DALP API user to sign from those vaults matches that tag. With the value empty, new vaults carry no tag and their requests reach the catch-all block rule. Before the upgrade, create the tag, set its id, tag the vaults DALP created earlier, and check your rules against [Fireblocks policy](https://docs.settlemint.com/docs/deployment/fireblocks-policy#tag-the-dalp-vaults).

### Move your values to the 3.2 chart [#move-your-values-to-the-32-chart]

The 3.2 chart splits the Platform API into four roles inside one subchart and drops the subcharts that 3.1 ran beside it. A values file written for 3.1 keeps keys that the 3.2 chart no longer reads. Move each of them:

| 3.1 values key | 3.2 values key |
| --- | --- |
| `dapi.resources`, `dapi.routesReplicaCount` | `dapi.v1.resources` and `dapi.v2.resources`, and replicas under `dapi.v1` and `dapi.v2` |
| `didx.*` (the indexer subchart) | `dapi.indexerDeployment.*`, with `rebuild.enabled` and `rebuild.replicas` for the rebuild pods |
| `ddwf.*` (the workflow subchart) | `dapi.runnerDeployment.*` for durable work and `dapi.schedulerDeployment.*` for scheduled work |
| `docs.searchIndexing.*` | `global.searchIndexing.*` |
| `dapp.config.app.http.url` | `global.dalpApp.config.app.http.url`, a literal `https://` URL |
| `global.dalpApp.config.networks.default.*` | `global.dalpApp.config.networks.mintnet.*`. The `default: true` flag, not the key, marks the default network |
| `global.dalpApp.config.platform.instance` | `global.dalpInstance`, with `global.deploymentEnvironment` set to the environment |
| `global.dalpApp.config.app.logOrpcRequests` | `global.dalpApp.config.app.logApiRequests` |
| `erpc`, `nodecore`, `global.rpcGateway` | None. Remove them and set `rpc.urls` on each network, as **Network endpoints** below describes. |
| `global.dalpApp.config.bundler.providerNativeSubmission` | None. Remove it. |
| `global.dalpApp.config.signer.dfns.executionMode`, `global.dalpApp.config.signer.ripple.executionMode` | None. Remove them. |

The chart fails the render on a `config.app` block under `dapp` or `dapi`, because the Console and the Platform API must read one public URL from `global.dalpApp.config.app`. It ignores the other 3.1 keys you leave in the values file, so remove each one yourself. A 3.1 `dapi.resources` block no longer sizes the API pods, because `dapi.v1.resources` and `dapi.v2.resources` have their own defaults and take precedence. Move your sizing to those keys and to the `resources` of `dapi.indexerDeployment`, `dapi.runnerDeployment` and `dapi.schedulerDeployment`. `dapi.routesReplicaCount` is ignored: set replicas under `dapi.v1` and `dapi.v2`. `dapi.ingress`, `dapi.httpRoute` and `dapi.openShiftRoute` keep their 3.1 meaning, and `dapi.config` is merged over `global.dalpApp.config` for every Platform API role.

A 3.1 override under `networks.default` no longer reaches the shipped network, so an RPC endpoint you set there must move to `networks.mintnet`. 3.2 ignores `platform.instance` and stamps every log, metric and trace with `global.dalpInstance` and `global.deploymentEnvironment`, which both default to `local`. Set them to this deployment's name and its environment: `development`, `test`, `staging`, `demo` or `production`.

**Client addresses behind a proxy.** The 3.1 chart read the client address from the forwarding header that an NGINX ingress controller writes, and 3.2 ships `global.dalpApp.config.trustedProxy.profile` as `none`, which discards every forwarding header. Behind a proxy, `none` keys sign-in rate limits and the request addresses in audit records on the proxy's address. Set `nginx`, `envoy` or `openshift` to match the controller in front of the Console and the Platform API.

**Network endpoints.** 3.1 sent every network call through an RPC gateway that the chart deployed, and each network's `rpc.url` pointed at that gateway. 3.2 deploys no gateway. Each network under `global.dalpApp.config.networks` takes its node endpoints directly in `rpc.urls`, a list. Copy the node endpoints that you configured as gateway upstreams into `rpc.urls`. 3.2 refuses a node endpoint that resolves to a private or loopback address unless the network sets `rpc.allowPrivateUpstreams` to `true`, so set it on a network whose nodes run inside your own network. [Networks](https://docs.settlemint.com/docs/deployment/networks) describes the setting. Set `rpc.wsUrl` on every network as well: the 3.2 indexer follows each network through that subscription endpoint and does not fall back to polling an HTTP endpoint.

**Custody execution mode.** 3.1 let you choose whether DFNS and Ripple sign as the provider's own transactions or as approvals that DALP submits, and a later 3.1 patch release added a bundler setting that did the same for Fireblocks. 3.2 has no such settings: each custody provider declares how it signs on each network, and the platform uses that. Remove the settings and their environment variables, `DFNS_EXECUTION_MODE`, `RIPPLE_EXECUTION_MODE` and `BUNDLER_PROVIDER_NATIVE_SUBMISSION`.

**Environment variables.** The 3.2 configuration no longer reads these variables. Remove them from any environment file or Secret you maintain:

- `DIDX_URL`, `DIDX_PORT`, `DIDX_HOSTNAME`, `DIDX_INTERNAL_HOSTNAME`, `DDWF_URL`, `DDWF_PORT`, `DDWF_HOSTNAME` and `DDWF_INTERNAL_HOSTNAME`. The chart sets the internal addresses of the four roles itself.
- `DATABASE_POOL_MIN` and `DATABASE_POOL_MAX`. 3.2 sizes one pool for each role, with these defaults: `DATABASE_POOL_MAX_API` 40, `DATABASE_POOL_MAX_INDEXER` 24, `DATABASE_POOL_MAX_RUNNER` 16 and `DATABASE_POOL_MAX_SCHEDULER` 32. Check the sum of the pools across all pods against your database's connection limit.

### Set the values that 3.2 adds [#set-the-values-that-32-adds]

3.1 had neither of these values, so a 3.1 values file sets neither, and the upgrade fails until both are set:

- `global.dalpApp.env.DALP_GOVERNANCE_DOMAIN_SCOPE` names the domain under which the deployment signs governance poll ballots. `helm upgrade` fails the render while it is empty or `local`. Set a name unique to this deployment, as [Governance ballot domain](https://docs.settlemint.com/docs/deployment/install/values-secrets#governance-ballot-domain) describes.
- `global.dalpApp.env.CUSTODY_CREDENTIAL_IDENTITY_KEY` is the key under which the platform hashes custodian credential identities, at least 32 characters long. The chart ships a development placeholder, and the Platform API pods, which run in production mode, refuse to start on it. Set a value unique to this deployment and keep `CUSTODY_CREDENTIAL_IDENTITY_KEY_ID` at `v1`.

### Decide on Ask AI [#decide-on-ask-ai]

3.2 adds Ask AI, a question-and-answer chat on the documentation site that the chart deploys. The documentation chart reads its model provider key from the same Secret entry that the Console assistant uses, `global.dalpApp.env.ASSISTANT_OPENROUTER_API_KEY`. The chart points the documentation at that entry by default, through `docs.openRouter.existingSecret` and `docs.openRouter.existingSecretKeys.apiKey`. The entry is empty by default, and Ask AI stays off while it is empty. As soon as you set the key for the Console assistant, Ask AI turns on too, for anyone who can reach the documentation site, and each answer is billed to that key. Ask AI allows each client 10 questions a minute and all clients together 300 an hour. The chart passes `global.dalpApp.env.REDIS_URL` to the documentation pods, so they keep these counts in Redis and share them. While Redis does not answer, Ask AI refuses questions.

Choose one of three settings in the same values change that sets the key:

- To share the key, set `ASSISTANT_OPENROUTER_API_KEY` and leave the `docs.openRouter` values as they ship.
- To keep Ask AI off, set `docs.openRouter.existingSecret` to `""` and leave `docs.openRouter.apiKey` empty.
- To bill Ask AI to a separate key, set `docs.openRouter.existingSecret` to the name of a Secret you manage and `docs.openRouter.existingSecretKeys.apiKey` to the entry that holds the key.

[Ask AI provider key](https://docs.settlemint.com/docs/deployment/install/values-secrets#ask-ai-provider-key) describes each value and the combinations that fail the render.

### Decide on legacy asset creation [#decide-on-legacy-asset-creation]

The seven asset families of the original catalogue, `bond`, `equity`, `fund`, `stablecoin`, `deposit`, `real-estate` and `precious-metal`, cannot be created unless the deployment turns legacy asset creation on. A request to create a token of one of these types, or to install its factory, is refused, and organisations and networks deployed while the setting is off get no factories for them. Tokens and factories that already exist stay on chain: the platform removes nothing.

The setting arrived in the 3.1.19 patch release, so a deployment on 3.1.19 or later keeps the value it already runs with. 3.0 and the 3.1 releases before 3.1.19 had no such setting, so a deployment on one of them whose users still issue these families loses that ability at the upgrade unless you turn the setting on. To keep issuing them, set `global.dalpApp.config.features.legacyAssetCreation` to `"true"`, or the environment variable `FEATURE_LEGACY_ASSET_CREATION` to `true`, in the values for the upgrade. The setting is `false` by default.

### Decide on the offerings surfaces [#decide-on-the-offerings-surfaces]

The Console shows the Offerings sidebar item and the issuer offering pages only when `global.dalpApp.config.features.tokenSale` is `true` and the organisation has deployed the token sale add-on. The 3.0 and 3.1 charts shipped `tokenSale: false`, and the 3.2 chart ships `tokenSale: true`. A deployment whose values file does not set the key therefore has the setting on after the upgrade, and an organisation that deploys the add-on gets the Offerings surfaces. Helm keeps a value that your values file sets, so a deployment whose overlay sets `tokenSale` to `false` stays without the Offerings surfaces. The same setting switches on fund dealing, the subscription and redemption requests of a fund and the settlement of its dealing days, so with it `false` the Dealing tab, the Dealing requests card and the Funds section of Invest are absent too. Corporate actions do not depend on it. Check your overlay for the key. Remove it or set it to `true` if issuers on this deployment run offerings or fund dealing, and set it to `false` to keep the surfaces off.

### Decide which application layers to switch on [#decide-which-application-layers-to-switch-on]

3.2 sells its application layers as licensed modules, and each one has a switch under `global.dalpApp.config.features`. [Switch application layers on or off](https://docs.settlemint.com/docs/deployment/install/values-secrets#switch-application-layers-on-or-off) lists every switch with its default. The Secondary market (`tradingVenue`), Reporting and finance operations (`regulatoryReporting`) and Market surveillance (`surveillance`) are off by default. A module that is off has no screens and its routes answer `404`, so for each module your licence includes, set its switch to `true` in the values for the upgrade.

Multi-signer wallets belong to the Self-custody module, `features.custodyOps`, which is on by default. A deployment that sets it to `false` accepts a signer set only when it has one signer and a threshold of one. It refuses any other signer set with HTTP 403 and the error `DALP-12261` `MULTISIG_NOT_ENABLED`, whose message is "Multi-signer wallets are switched off on this deployment". An organisation owner who switches Self-custody off for one organisation gets the same limit for that organisation, refused with `custody.multisig.OFF_FOR_ORGANIZATION`. [Multisig](https://docs.settlemint.com/docs/concepts/custody-and-signing/multisig) explains what a multi-signer wallet is.

### Check that the custody provider signs user operations [#check-that-the-custody-provider-signs-user-operations]

At start-up, the API, runner, indexer and scheduler pods check that the platform's custody provider, `SIGNER_PROVIDER`, can sign on each enabled network. With account abstraction on, they also check that the provider can sign user operations, the transactions that smart accounts send. A provider that cannot stops the pods from starting with an error that names the chain ID and ends with `Set SIGNER_PROVIDER to a provider that signs user operations. A deployment that runs without account abstraction can set FEATURE_ACCOUNT_ABSTRACTION=false instead.` Confirm before the upgrade that your provider supports every enabled network, or that you can turn account abstraction off.

### Prepare the custody census [#prepare-the-custody-census]

The custody census is the 3.2 migration step that records which custodian holds each existing wallet before it moves wallets to custody accounts. It archives wallets that hold no signing key. When it cannot prove the custodian of any other wallet, it stops with the message `The custody census still has <n> unresolved row(s)`, followed by the count of each kind of row it could not resolve, and the upgrade does not complete. If the deployment still has the legacy transaction signer database, which a deployment first installed before 3.1 can have, set `TXSIGNER_DATABASE_URL` under `dapp.migrator.env`. The Job reads that database as evidence of which wallets DFNS or Fireblocks hold. The runtime pods do not read this value.

### Upgrading from 3.0 [#upgrading-from-30]

A 3.0 deployment that upgrades directly to 3.2 needs every step above, and these steps from the 3.1 releases as well. Make them part of the same values change.

**Legacy asset creation.** 3.0, and 3.1 before 3.1.19, created the seven original asset families without a setting. Decide on the setting before the upgrade, as [Decide on legacy asset creation](#decide-on-legacy-asset-creation) describes.

**Custody records per organisation.** 3.0 configured one custody provider for the whole deployment. Since 3.1 each organisation has its own custody provider configuration. Keep `SIGNER_PROVIDER` and the provider credentials set to the custodian that holds the existing keys during the upgrade. A 3.0 deployment is the most likely to still have the legacy transaction signer database, so read [Prepare the custody census](#prepare-the-custody-census) before you upgrade.

**Data sealed under an earlier password.** If the migration Job stops with the message `ENCRYPTION_PREVIOUS_MASTER_PASSWORD is required`, some secrets or KYC files are still sealed in an older format or under another password, and the message names how many. Give the Job the password that sealed them through `dapp.migrator.env.ENCRYPTION_PREVIOUS_MASTER_PASSWORD`, and it seals them again under `ENCRYPTION_MASTER_PASSWORD`. Only the migration Job receives this value. [Encryption at rest](https://docs.settlemint.com/docs/deployment/install/values-secrets#encryption-at-rest) describes the procedure.

**Fireblocks Gas Station.** Since 3.1.12, new Fireblocks vault accounts are enrolled in Fireblocks Gas Station by default. A deployment on 3.0, or on a 3.1 release before 3.1.12, that uses a Fireblocks workspace where Fireblocks has not set up Gas Station sets `FIREBLOCKS_AUTO_FUEL` to `"false"`, because a vault account creation that Fireblocks rejects cannot be retried.

**Network defaults for DFNS and Fireblocks.** `DFNS_DEFAULT_NETWORK`, added in 3.1, names the DFNS network on which a new wallet is created when the request names no network and carries no chain ID that maps to a DFNS network, and in 3.2 it defaults to `Ethereum`. A DFNS custody configuration whose "Default network" field is set uses that field instead. A request that carries a mapped chain ID creates the wallet on that chain's DFNS network instead, as [Which network a DFNS wallet is created on](https://docs.settlemint.com/docs/concepts/custody-and-signing/custody-provider#which-network-a-dfns-wallet-is-created-on) describes. `FIREBLOCKS_DEFAULT_ASSET_ID`, also added in 3.1, names the Fireblocks asset used when a request names no network. A wallet is bound to its network when it is created. On a test network, set both to that network, such as `EthereumSepolia` for DFNS or `ETH_TEST5` for a Fireblocks Sepolia workspace.

**Integrations.** Clients built against 3.0 need the changes on [Upgrade an integration to DALP 3.2](https://docs.settlemint.com/docs/api-reference/guides/upgrade-to-3-2#upgrading-from-30) as well.

### Take a recovery point [#take-a-recovery-point]

A Helm rollback does not reverse the database migrations of the upgrade, as [Rolling back](#rolling-back) explains. Once in-flight work has finished, take a database backup and keep it until you have confirmed the 3.2 deployment. [Backup and recovery](https://docs.settlemint.com/docs/deployment/backup-recovery) covers what to back up.

## Run the Helm upgrade [#run-the-helm-upgrade]

Run `helm upgrade` with the 3.2 chart version and your updated values:

```bash
helm upgrade dalp oci://pkg.distribution.settlemint.com/settlemint/dalp \
  --version <3.2 chart version> \
  --namespace dalp \
  --values my-values.yaml \
  --wait
```

The migration Job runs first, as [Upgrades and migrations](https://docs.settlemint.com/docs/deployment/upgrades-migrations) describes, and Helm then replaces the earlier workloads with the 3.2 ones. Under the default `pre` hook, a catch-up run of the same migration follows the rollout and can still fail the release, so pass `--wait` to hold the catch-up until the new workloads are ready. When the command returns, check that each Deployment has finished rolling out:

```bash
for d in dalp-api-v1 dalp-api-v2 dalp-indexer dalp-indexer-rebuild dalp-runner dalp-scheduler dalp-dapp; do
  kubectl rollout status deployment/"$d" --namespace dalp
done
```

Leave `dalp-indexer-rebuild` out of the list when `dapi.indexerDeployment.rebuild.enabled` is `false`, as the OpenShift overlay sets it.

The Platform API's workloads change name, because the one API workload and the two subcharts of 3.1 become separate Deployments in 3.2. With the default `dapi.fullnameOverride` of `dalp-dapi`, the 3.2 Deployments are named as follows:

| 3.1 workload | 3.2 Deployments |
| --- | --- |
| `dalp-dapi` | `dalp-api-v1` for `/api/v1`, `dalp-api-v2` for `/api/v2` |
| `dalp-didx` | `dalp-indexer`, plus `dalp-indexer-rebuild` for index rebuilds |
| `dalp-ddwf-v<version>` | `dalp-runner` and `dalp-scheduler` |
| `dalp-dapp` | `dalp-dapp`, unchanged |

The API Services are `dalp-api`, which serves `/api/v2`, and `dalp-api-v1`. The chart's own Ingress, Route and NetworkPolicy objects follow the new names. Anything outside the chart that names a 3.1 object, such as an alert rule, a dashboard, a NetworkPolicy you added, an HPA or capacity rule, or a client inside the cluster that dials the `dalp-dapi` Service, needs the new name.

3.2 also runs no RPC gateway, so the upgrade removes the `dalp-erpc` Deployment and, where you turned it on, `dalp-nodecore`, together with their Services and any Ingress or Route you enabled for them. Search your alert rules, dashboards and NetworkPolicies for `dalp-erpc` and `dalp-nodecore`. Move an egress rule that allowed traffic to the gateway onto the API, indexer, runner and scheduler pods, which call the node endpoints in `rpc.urls` directly, and remove the rules and panels that watched the gateway. A client outside the platform that used the gateway's own hostname needs a node endpoint of its own. The upgrade also starts the PostgreSQL exporter, `postgres-exporter.enabled`, on by default, which holds one direct database session.

The chart has no step that sequences the cutover. Helm creates the 3.2 Deployments and Services and removes the 3.1 ones in the same release, except the 3.1 workflow Deployment, Service and ConfigMap, which Helm keeps until you delete them, as [Verify the upgrade](#verify-the-upgrade) describes. The API answers only once the new API pods are ready.

The upgrade does not delete the data that the earlier workflow engine stored in its persistent volumes and in object storage. It stays until you delete it. Delete it once you no longer need a way back to 3.1.

## After the upgrade [#after-the-upgrade]

### Watch the index rebuild [#watch-the-index-rebuild]

3.2 changes how the platform indexes chain data, so the first 3.2 start builds a new index next to the existing one. The existing index keeps answering reads while the new one replays the history of each enabled network, and the platform promotes the new index only when it has caught up. A Helm release that reports success does not mean the rebuild has finished.

How long the rebuild takes depends on the number of enabled networks, the blocks and events to replay, and how fast your RPC nodes and database answer. The platform gives no fixed duration. Follow it in three places:

- In the Console, open Platform status, open the details of the Data freshness panel, and wait for each network to move from Reindexing to Healthy.
- The metric `dalp_indexer_build_progress_ratio` runs from 0 to 1 for the index being built. `dalp_indexer_build_failures_total` counts failed build steps, and `dalp_indexer_promotion_age_seconds` reports the age of the last promotion.
- A client that sends `Prefer: dalp-fail-on-indexer-reindexing` on `POST /api/v2/organizations`, `POST /api/v2/organizations/bootstrap`, `POST /api/v2/invitations` or `POST /api/v2/invitation-acceptances` receives HTTP 503 with code `platform.http.REINDEXING` (`DALP-12331`) while the network's index is rebuilding, instead of the request going ahead. The response carries `data.retryAfterSeconds` and a `Retry-After` header. The refusal is retryable.

### Run System updates [#run-system-updates]

The Helm upgrade changes the platform, not the contracts of each organisation's system on chain, so 3.2 does not require a System updates run. An organisation's system keeps running on the contract versions it has, and each organisation decides when to run System updates on each network it uses. An organisation runs it when it needs a token feature, add-on or document anchoring that the 3.2 contracts add.

Until an organisation runs System updates on a network:

- Its installed contracts on that network stay on the versions they run.
- The token feature factories and add-ons that 3.2 adds are not on its system, among them the factories for holds, supply delegation, interest and dividend yield, distribution waterfalls and register lines, and the custody gateway add-on. Attaching a token feature whose factory the system lacks is refused with HTTP 409 `TOKEN_FEATURE_FACTORY_NOT_FOUND` (`DALP-9465`). Creating a token from a template that uses such a feature is refused before anything is signed, with HTTP 409 `TOKEN_CREATE_FEATURE_FACTORY_NOT_REGISTERED` (`DALP-207901`), and Issue asset marks the template as needing a feature factory. Every system template uses holds. Reading the custody gateway on a system without the add-on answers HTTP 404 `GATEWAY_ADDON_NOT_DEPLOYED` (`DALP-9813`). Anchoring a document is refused with HTTP 409 `ANCHOR_WRITE_REFUSED` (`DALP-0749`) until the system has the document anchor registry. A System manager can also install one factory or add-on at a time from [Factories](https://docs.settlemint.com/docs/console/organisation-settings/factories), without a full run. The custody gateway add-on is not offered there: install it with System updates or with `POST /api/v2/system/addon-factories`.
- The run's data steps stay outstanding: price feeds for assets that lack one, the organisation's trust for the claim topics it issues, topic scheme updates, investor eligibility topic signatures, escrow claims and the advanced accounts infrastructure.
- The organisation sees the shipped reporting obligations but has no editable copies of them, because the run is what makes those copies.

System updates compares the organisation's system with the newest release directory on the network. An Admin or a System manager of each organisation opens System updates in the Console, reviews the components marked Upgrade available or Not installed, and starts the run. [System updates](https://docs.settlemint.com/docs/console/organisation-settings/system-updates) describes the run, which cannot be undone once its steps are written to the network.

### Verify the upgrade [#verify-the-upgrade]

Check each of these after the upgrade:

1. `helm upgrade` returned without an error. Under the default `pre` hook, that means the migration Job succeeded before Helm touched the workloads, and the catch-up run succeeded after the rollout.
2. `helm history dalp --namespace dalp` lists the 3.2 chart version as the `deployed` revision.
3. Each Deployment in the rollout loop above has finished rolling out, and `kubectl get deployments --namespace dalp` no longer lists `dalp-dapi`, `dalp-didx` or `dalp-erpc`. The 3.1 workflow Deployment `dalp-ddwf-v<version>`, its Service of the same name and its `dalp-ddwf-v<version>-config` ConfigMap carry Helm's keep policy, so the upgrade leaves them running. Delete them once the 3.2 runner is ready: `kubectl delete deployment,service dalp-ddwf-v<version> --namespace dalp` and `kubectl delete configmap dalp-ddwf-v<version>-config --namespace dalp`.
4. A `/api/v1` request and a `/api/v2` request both answer through your Ingress, Route or HTTPRoute. The chart sends `/api/v2` to the v2 fleet and the rest of `/api` to the v1 fleet.
5. Platform status shows each enabled network as Healthy in the Data freshness panel, once the index rebuild has finished.

## Rolling back [#rolling-back]

`helm rollback` returns the Kubernetes objects to the manifests of the release you upgraded from. It does not reverse the database migrations, it does not restore the earlier workflow engine's state, and it does not undo a System updates run, which is written on chain. The 3.1 code then runs against a database that 3.2 has migrated. To return to 3.1, restore the database backup you took before the upgrade, then roll the release back. Any operation that ran on 3.2 after the backup is not in the restored database, while its on-chain results stay on the ledger, so compare the restored records with the ledger before you reopen traffic, as [Point-in-time recovery](https://docs.settlemint.com/docs/deployment/backup-recovery#point-in-time-recovery) describes. [Rolling back after a bad upgrade](https://docs.settlemint.com/docs/deployment/upgrades-migrations#rolling-back-after-a-bad-upgrade) explains why a rollback leaves the schema as it is.