Skip to main content

Upgrade DALP and run its database migrations

How the database migration Job runs inside a Helm upgrade, which hook type to choose, and what a failed migration leaves behind for you to inspect.

A dalp Helm upgrade updates the platform release and runs its database migration Job. With the default pre-upgrade hook, the Job must succeed before Helm updates application Deployments. Database changes are separate from the application rollout and do not reverse with helm rollback. Check migration compatibility, recovery points, and any required index rebuild before the upgrade. To move a 3.1 or 3.0 deployment to 3.2, follow Upgrade a deployment to DALP 3.2.

What runs, and when

The dapp subchart renders a Kubernetes Job named <release>-migrations whenever dapp.migrator.enabled is true, which is the shipped default. Set dapp.migrator.enabled: false and no migration runs, so your application pods start against whatever schema the database already has. That setting is for a pipeline that runs migrations through its own tooling before it calls helm upgrade. On every other path, disabling it removes the check between a schema change and a pod that expects it.

dapp.migrator.hookType decides when the Job runs. Only the pre hook is a hard gate.

dapp.migrator.hookType
Runs
Use when
Consequence
pre (default)
pre-install,pre-upgrade hook at weight -5
Postgres is external or already running, as in production
The Job completes before Helm updates the application workloads.
post
post-install,post-upgrade hook at weight 0
The same release also creates the database the migration targets
Helm applies and waits on the workloads before the Job runs, so new pods can start before the migration finishes. A readiness check that needs the new schema can stop the wait from ever reaching the post hook.
none
An ordinary resource in the release, with no hook annotations
Your own pipeline orders migrations outside Helm
Helm's before-hook-creation cleanup and the hook ordering no longer apply, so you own migration ordering.

With pre, the chart also runs the migrator a second time after the rollout, as the <release>-migrations-catchup Job: a post-install,post-upgrade hook at weight 10. It picks up rows an old pod wrote while the rollout ran, and it is a no-op when nothing is left to migrate. Pass --wait to helm upgrade so the catch-up runs only after the new pods replace the old ones. Without it, the catch-up can run while old pods still serve. A failure there fails the release after the new pods already serve. Both Jobs carry the label app.kubernetes.io/component: migrations, so kubectl get jobs -l app.kubernetes.io/component=migrations lists them together. Set dapp.migrator.catchUp.enabled: false where a second Job per upgrade is not acceptable.

Before running its migration command, the Job's init containers wait for the Postgres endpoint to accept connections and, when you enable Conjur, fetch a Conjur access token. That wait draws from the same clock the migration itself runs on: activeDeadlineSeconds bounds the Job from the moment it starts, init containers included. The Postgres wait's own timeout defaults to 120 seconds, so a slow database can use a fifth of the default budget before the migration command runs. The migration container runs with a default backoffLimit of 3 and an activeDeadlineSeconds of 600 seconds, 10 minutes in total.

Its database connection is not the one the application containers use: a dedicated migration Secret gives it a direct, non-pooled DATABASE_URL, bypassing the connection pooler even when pooling is enabled for the application. A transaction-mode pooler would break the session-scoped advisory lock the migration takes. Everything else about how it reads configuration matches the application containers: a mounted config.yml, an envFrom secret, and, when you configure Conjur secrets for the migrator or inherit them from the runtime dapp configuration, a summon-wrapped entrypoint. dapp.migrator.env can also carry TXSIGNER_DATABASE_URL for a deployment that still has the legacy transaction signer database, which Prepare the custody census describes: the Job reads that database as evidence of which custodian holds each wallet's keys when it records each organisation's custody provider. It can carry the encryption-rotation keys too (ENCRYPTION_MASTER_PASSWORD, ENCRYPTION_PREVIOUS_MASTER_PASSWORD) for a local, non-Conjur rotation run. Values and secrets covers the two-step rotation and when the old key can be removed. It can carry PLATFORM_ADMIN_RECOVERY_EMAIL for one upgrade, to give an existing account the platform administrator role when no administrator is left or none can sign in, as Restore the platform administrator role describes.

With the default pre hook type, Helm does not change your running Deployments until the migration Job succeeds. The previous pods keep serving, but migration locks, data changes or incompatible schema changes can still slow or fail their requests, so plan the upgrade in a maintenance window. An index rebuild is not a gate. Some releases require one, so check the release notes for your target version. A rebuild runs behind a deployment that already serves, and the read models it populates are incomplete for earlier periods until it finishes draining. Treat affected historical figures as provisional during the rebuild. Open the Indexing detail screen from Platform status and wait for the affected chain to return from Reindexing to Healthy. Helm release completion does not prove that the rebuild finished.

A migration Job gates the release. An index rebuild never does.A vertical timeline runs down the figure from when helm upgrade starts to well after the release finishes. The migration Job sits directly on that timeline: on its default pre-install and pre-upgrade hook it runs before pods roll to the new version, and a solid gate marker shows the timeline cannot pass this point until the Job succeeds, so a failure stops the release right there while your prior release's serving pods stay untouched. Two independent clocks remove the finished or failed Job afterward: the Helm hook-delete policy, which by default keeps a failed Job until the next upgrade attempt, and a Kubernetes-native ttlSecondsAfterFinished defaulting to 24 hours, whichever fires first. Past the gate, pods roll and the timeline keeps moving. The index rebuild attaches differently: it starts once pods are already rolling, and its own activity is drawn straddling the point where the release reports complete, because the release proceeds regardless of whether the rebuild has finished and the rebuild keeps draining on its own clock afterward. While it drains, the affected chain's status shows as Reindexing in the Indexing detail page, and moves to Healthy once the rebuild finishes.Migration Job blocks. Index rebuild runs alongsideOne obligation gates the release, one never doesHelm upgrade startsMigration jobPre-hook default, before pods rollServing pods stay untouchedA failure blocks the release hereTwo clocks remove the JobHook policy, or a 24h TTLWhichever fires first winsGatePods roll to the new versionIndex rebuildNot a gate, runs after pods rollBehind the healthy deploymentRelease proceeds either wayRelease reports completeFigures stay provisionalShown as Reindexing in theIndexing detail page, then HealthyKeeps draining independently of the releaseA migration Job gates the release. An index rebuild never does.The same figure as the desktop variant, drawn portrait for phone widths. A vertical timeline runs from when helm upgrade starts to well after the release finishes. The migration Job sits directly on that timeline: on its default pre-install and pre-upgrade hook it runs before pods roll to the new version, and a solid gate marker shows the timeline cannot pass this point until the Job succeeds, so a failure stops the release right there while your prior release's serving pods stay untouched. Two independent clocks remove the finished or failed Job afterward: the Helm hook-delete policy, which by default keeps a failed Job until the next upgrade attempt, and a Kubernetes-native ttlSecondsAfterFinished defaulting to 24 hours, whichever fires first. Past the gate, pods roll and the timeline keeps moving. The index rebuild starts once pods are already rolling, and its own activity is drawn straddling the point where the release reports complete, because the release proceeds regardless of whether the rebuild has finished. While it drains, the affected chain's status shows as Reindexing in the Indexing detail page, and moves to Healthy once the rebuild finishes.Migration Job blocks.Index rebuild runs alongsideOne obligation gates the release,One never doesHelm upgrade startsMigration jobPre-hook default, before pods rollServing pods stay untouchedA failure blocks the release hereTwo clocks remove the JobHook policy, or a 24h TTLWhichever fires first winsGatePods roll to the new versionIndex rebuildNot a gate, runs after pods rollBehind the healthy deploymentRelease proceeds either wayRelease reports completeFigures stay provisionalShown as Reindexing in theIndexing detail page, then HealthyKeeps draining independently of the release

The pre-upgrade hook holds the application rollout while the migration runs, and an index rebuild continues alongside without gating the release.

What a failed migration leaves behind

The Job's hook delete policy defaults to before-hook-creation,hook-succeeded, without hook-failed. A hook-failed policy would make Helm delete the failed Job and its pod immediately, which removes the pod-local evidence before you inspect it. Centralised logs may still retain the error. With the default policy, Helm leaves a failed migration Job and its pod in the cluster for you to inspect (kubectl logs job/<release>-migrations, kubectl describe job/<release>-migrations) until the next upgrade attempt, at which point before-hook-creation deletes the previous Job to make room for the new one.

Helm's hook-delete policy is not the only thing that can remove that Job. The Job template also sets a Kubernetes-native ttlSecondsAfterFinished, defaulting to 86400 seconds, or 24 hours, which the cluster's own TTL controller uses to delete the Job, on success or on failure, independently of anything Helm decides. If you upgrade less than once a day, that clock is the shorter one: by default you have 24 hours after a migration Job finishes to read its logs. If your process needs longer to notice a failed upgrade and pull logs, raise dapp.migrator.ttlSecondsAfterFinished. Changing dapp.migrator.hookDeletePolicy alone does not extend that window.

Before you upgrade

  • Drain the waiting workflows, as Drain waiting workflows before you upgrade describes.
  • Read the release notes for the version you are upgrading to. A migration Job runs on every helm upgrade unless dapp.migrator.enabled is false, whether or not that release changed the schema, so check for a required index rebuild, which needs its own place in the change window. Upgrade to 3.2 lists the changes one version step asks an operator and an integrator to make. Supply chain and support policy states which release lines take patches and how a version number maps to the images it selects.
  • If the application reaches Postgres through a pooler that does not track named prepared statements (PgBouncer before 1.21, statement mode, or a session-pinning proxy), set global.dalpApp.config.database.preparedStatementCacheSize to 0 before you upgrade. Pooler requirements for prepared statements has the details.
  • Confirm the Postgres endpoint the new release uses is reachable and has enough headroom for the migration to complete inside the 10 minute default deadline. Raise dapp.migrator.timeout for a migration you expect to run longer. The migration connects direct to Postgres, so that headroom comes from your server's own max_connections, not from the pooler's fixed pools. Values and secrets covers how to compare those pools against your server's ceiling.
  • If your cluster enforces NetworkPolicy, confirm the migration Job's egress rule permits the Postgres port and any external Postgres host you use. That policy ships at an earlier hook weight than the Job, so it is in place before the Job needs it. If the rule is wrong, the pre-upgrade Job cannot reach Postgres, the hook fails, and the manifests that carry the fix never apply.
  • Confirm dapp.migrator.ttlSecondsAfterFinished gives your on-call process enough time to pull logs from a failed Job before the cluster deletes it. The default is 86400 seconds (24 hours).
  • Size the --timeout you pass to helm upgrade. The migration Job's deadline and that release-wide wait are separate clocks, and Helm charts explains how to size the wait.

Drain waiting workflows before you upgrade

The Workflow Engine stores each workflow it runs as a journal of steps and replays that journal after a restart. A workflow that is waiting survives an upgrade only if the new release still reads the step names and data in its journal. The release does not check this. A wait can be long: a funding request stays open for up to its network's pauseWindowSeconds, which can be 7 days. See Gas settings. Finish or cancel the waiting work first, then upgrade. When you move from 3.1 to 3.2, also finish the work in Finish in-flight work, because the workflow engine changes in 3.2.

  1. List the unfinished workflows. Call GET /api/v2/platform-status/workflows-list?limit=500 as a platform administrator or as the organisation's owner, once for each organisation. The Workflows detail screen on Platform status shows the same list.
  2. Read the status of each item. Treat waiting and running alike: both mean the workflow has not finished, and a workflow sleeping on a timer or an external event can show either one. The list shows unfinished items first. If truncated is true, more items exist than limit returned.
  3. Let each unfinished workflow finish, or cancel the operation it belongs to from the Console screen that started it.
  4. List again. Run helm upgrade when no item has status waiting or running.

Find out why a running item with stalled: true stopped before you cancel it. Platform status explains what a stalled workflow means.

After you upgrade

All charts in a release share one version: Helm charts covers why mixing subchart versions inside one release is not supported. After the upgrade, confirm the deployed state:

  • helm list reports the release's current chart version. Confirm it matches the version you upgraded to.
  • helm history <release> shows which revision landed, so you can compare it against the one you expected.
  • Confirm both migration Jobs succeeded: kubectl get jobs -l app.kubernetes.io/component=migrations.
  • If the release requires an index rebuild, open Data freshness on the Console's Platform status screen, then open the Indexing detail screen. Confirm that the affected chain has moved from Reindexing to Healthy. Release completion does not prove rebuild completion. See What runs, and when.

Rolling back after a bad upgrade

A Helm rollback (helm rollback <release> <revision>) returns your Deployments, Services, and other templated resources to a prior release's rendered manifests. It does not run a migration Job to reverse a schema change, and it does not restore database rows a completed migration already altered. Rolling back the release moves your running pods back to code that expects the older schema. If the failed or completed migration already changed the schema in a way the older application code cannot read, a Helm rollback alone leaves the older code running against a schema it cannot read.

The same holds for an index rebuild. It runs as an indexer process behind the deployment, not as a Helm hook, so a rollback has no lever over it. Rolling back does not pause or reverse a rebuild already in progress, and it does not restore the projections a rebuild has already rewritten.

Retry safety depends on the failed migration and the state it left behind. Treat a failed migration, and any rollback decision that follows it, as a stop-and-inspect event: read the Job's logs, confirm what state the schema is in, and involve whoever owns your database change process before retrying or rolling back. Do not assume a backoffLimit retry is always safe to repeat.

To return to the previous version after a migration changed the schema, restore the recovery point you took before the upgrade, then run helm rollback <release> <revision>. Records written after that recovery point are not in the restored database. Backup and disaster recovery lists what to restore and in what order.

Frequently asked questions

Is a helm upgrade --timeout failure the same thing as a failed migration Job?

No, though both can abort the same upgrade. --timeout bounds how long Helm waits for the whole release, migration Job and every workload's rollout together, before it gives up and reports a timeout. dapp.migrator.timeout (rendered as the Job's activeDeadlineSeconds) bounds only the migration Job itself. A migration that finishes at second 590 of its own 600-second budget can still exceed a --timeout sized for the rollout alone.

Is it safe to remove the previous encryption master password as soon as the migration Job finishes?

Yes, once that Job succeeded, for the live database. Keep the previous password until every backup you may restore has aged out, because a restore from an older backup brings back rows sealed under it. Rotation is one of the migration Job's required boot steps, not a separate job, so a row that fails to rotate fails the step, and a failed required step fails the Job. A migration Job that exited successfully has already reported a clean rotation.

The step logs an Encrypted secret rotation summary line and a matching KYC encryption rotation summary line, each with scanned, rotated, skipped and failed counts. A skipped row is already stored under the current key, so a rotated count below scanned is normal. The count that matters is zero failed.

The platform re-checks the decision after you clear the value. With ENCRYPTION_PREVIOUS_MASTER_PASSWORD empty, the step confirms no pre-v2 data remains and fails the Job with a count of the secret rows and KYC objects still under the old key. That guard catches a premature removal at the next upgrade. What it cannot undo is destroying the key material itself, because a row still encrypted under a master password nobody holds is unreadable. Encryption at rest covers the two-step sequence.

Was this page helpful?