Backup and disaster recovery

What is stateful#

Platform pods keep no durable state of their own: caches and in-flight work in a pod are lost when it stops. Everything that must survive a failure sits in one of these stores.

What gets backed up, and what does not#

When you turn on the dalp chart's backup block, the charts back up what runs inside your Kubernetes cluster through Velero, the Kubernetes-native backup tool they drive. A service outside the cluster, such as your PostgreSQL, Redis or object storage, backs up through its own mechanism, not through DALP. Prerequisites lists a tested backup and restore for PostgreSQL and object storage as a go-live item.

The schedule runs no Velero backup hook, so a Velero copy of a database volume is a crash-consistent file-system copy, not a pg_dump or an application-quiesced snapshot. If you run PostgreSQL inside the cluster, use a database-native backup and verify recovery for production data.

How the backup block works#

The dalp chart never deploys a Velero server, and it is the one chart that decides what gets backed up and where. Its backup block, off by default, creates the BackupStorageLocation and Schedule. These custom resources need a Velero server that you run, for example an installation of OpenShift API for Data Protection (OADP) or Velero itself. Enabling the block backs up nothing until that server runs. Velero reconciles only its own namespace. The chart renders these resources in the release namespace unless you set backup.veleroNamespace to the Velero server's namespace, for example openshift-adp for OADP. An existingSecret you name must live in that same namespace. A workload identity (IRSA, Azure managed identity, GCP workload identity) takes effect once you bind it to the Velero server's service account, plus whatever pod settings your provider's Velero plugin documents. The shipped storage defaults are for development: the endpoint is an in-cluster address and backup.storage.s3.insecureSkipTLSVerify is true. Set both before you enable the block, as Decisions to make before you need a restore describes. What the block does with the credentials Secret depends on what you set.

What the default schedule captures#

These are the dalp chart's own backup.* defaults.

Point-in-time recovery#

Point-in-time recovery returns a database to a chosen moment between two backups by replaying its write-ahead log. The DALP charts do not configure it. The Velero schedule copies each volume's files once a day by default, so its recovery points are the schedule's run times and not arbitrary moments.

For production, point-in-time recovery is a feature of your managed PostgreSQL service. Ask the provider for three answers and record them in the database's recovery plan:

• The retention window, meaning how far back you can restore.
• The finest restore granularity.
• Whether a restore creates a new instance, which decides how you repoint the platform to it.

A restore to an earlier moment removes the off-chain records written after it. Whatever reached the ledger after that moment stays on the ledger. With the default local secret manager, a signer key or secret created after the restore point is lost with it, and the platform can no longer sign for the on-chain assets or roles that key controls. The indexer resumes from the position stored in the restored database and reads forward from there. Compare restored records with the ledger before you reopen traffic.

Decisions to make before you need a restore#

Decide your storage target and who owns its credentials. backup.storage.provider accepts s3, aws, azure, or gcp. The shipped default endpoint is an in-cluster address, and the default s3 location ships with backup.storage.s3.insecureSkipTLSVerify set to true, so point it at a bucket you supply and set that value to false. Store production recovery points outside the failure domain they must recover from. The chart creates no inline credentials secret for GCP. Either supply backup.storage.gcp.existingSecret or set backup.storage.gcp.useWorkloadIdentity and skip the secret entirely.

Decide your retention window from your compliance or evidentiary needs. The 30-day default TTL is the only retention control. Velero keeps no minimum count of backups once older ones expire. If a regulator or an internal policy expects a longer recovery window, raise retention.days. Do not assume a floor exists underneath it.

Decide whether the default label scope covers what you need backed up. A resource that never carried kots.io/app-slug: settlemint-dalp, or that loses it later, silently falls outside the schedule. The schedule keeps completing successfully and backs up less than you expect. The pgbouncer and postgres-exporter subcharts are on by default and do not carry the label. The umbrella chart's own resources, including the shared application environment Secret that holds the encryption master password and the session secret, carry it only when you add kots.io/app-slug: settlemint-dalp to global.labels on the release. Adding that label puts the Secret, and with it the master password, in the backup bucket. Without it, a Velero-only restore needs a Helm upgrade to recreate that Secret from your secret manager. Confirm the scope with kubectl get all,pvc,configmap,secret -l kots.io/app-slug=settlemint-dalp -n <namespace>, run the same command without the label, and compare the two lists against what you expect to matter in a restore.

Decide how each datastore fails over and recovers. You supply PostgreSQL, Redis and object storage, so a restore and a failover promise only what your providers promise. See Failing over.

Confirm the backup alerts reach someone. Three Velero alert rules ship in the observability stack, and all three default to warning severity, not critical. They fire only after you switch on alerting in the observability stack and only when the Velero server's metrics are scraped. Until then, nothing alerts on a failed backup.

A stale schedule can run silently broken for more than a day before VeleroBackupStale fires, because the rule waits for 24 hours without a success and then holds for 15 minutes. If your operating process needs faster notice than that, do not assume the default severity routes to an on-call page. Confirm it does, or raise it.

Verify a backup is usable#

The Grafana dashboard Cluster · Backups & Policy (uid backups-policy) gives you the automated signal: its “Backup failures” panel and “Last successful backup” panel tell you a schedule ran and did not error, and the three Velero alerts tell you when a running schedule fails. The alerts read metrics from the Velero server. They stay silent when no Velero server runs, when its metrics are not scraped, or when a schedule has never completed a backup. Neither the dashboard nor the alerts tell you the data that came back is usable. Only a test restore does that.

Run a test restore on a fixed schedule that your recovery plan names, and at least before your first production go-live and after any change to backup.storage, backup.labelSelector, or backup.schedule. Restore into a namespace other than the one the backup came from, so you can inspect the result without touching a live workload. Confirm the resource and volume counts match what you expected. For anything that held database data, confirm the application starts cleanly against it, rather than only checking that files arrived. A restored environment Secret points at your production database, cache and ledger endpoints, and with the default labels the restore lacks that Secret and the pooler until a Helm upgrade recreates them. Before any restored pod starts, apply the isolation in Rehearse a recovery, step 1. Delete the test restore's namespace once you have confirmed it, so a stale test copy does not linger as a second, unmanaged instance.

Restoring from a backup#

1. Confirm which backup you are restoring from. A backup's phase must read Completed, not PartiallyFailed, before you trust it as your restore source. A partial failure means some resources or volumes did not make it in.

   kubectl get backups.velero.io -n <velero-namespace>

   This lists every backup in the namespace, including those the <fullname>-daily schedule created. <velero-namespace> is the release namespace unless you set backup.veleroNamespace.

2. Decide your target before you create the restore. Restoring into the same namespace where the original resources still exist behaves differently from restoring into an empty namespace or a new cluster. Existing-resource handling depends on Velero's configured restore policy. Do not assume a restore overwrites live resources. For a test restore, set spec.namespaceMapping on the Restore (step 3) to restore into an empty namespace instead of overwriting anything live.

3. Create the restore from the chosen backup by applying a Restore resource, then watch its phase rather than assuming it finished because the command returned.

   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: <restore-name>
     namespace: <velero-namespace>
   spec:
     backupName: <backup-name>
     namespaceMapping:
       <original-namespace>: <target-namespace>

   kubectl get restores.velero.io <restore-name> -n <velero-namespace> -w

4. Wait for volume restores to finish separately from the resources that reference them. The default schedule backs up volumes at the file-system level, so a restored pod can stay pending after its other resources already exist, while its volume's own restore is still copying data in the background. Watch the PodVolumeRestore resources Velero creates for this restore, not only the pod's own status:

   kubectl get podvolumerestores.velero.io -n <velero-namespace>

5. Confirm what came back before you call the restore done. A Completed restore proves Velero moved the bytes it had. It does not prove the data is transactionally consistent, since no backup hook runs against a database volume before the copy is taken. Start the application against the restored data and confirm it comes up cleanly, as Verify a backup is usable describes.

Restoring a whole instance#

A Velero restore brings back cluster resources and volumes. A full recovery after losing a cluster or a region also needs the datastores, the secrets and the ledger connection. Before step 1, confirm that the lost instance is stopped, so that two instances never act on the same ledger at once. Restore in this order. Steps 1 and 2 gate the platform, and steps 3 and 4 can run beside step 2. Keep traffic closed until step 8.

1. Recreate the secrets. Take ENCRYPTION_MASTER_PASSWORD and BETTER_AUTH_SECRET from your secret manager, and set DALP_GOVERNANCE_DOMAIN_SCOPE from your recorded values file, all to the values the lost instance used, together with CUSTODY_CREDENTIAL_IDENTITY_KEY and CUSTODY_CREDENTIAL_IDENTITY_KEY_ID (Custody credential identity key). The platform refuses to start without the master password, a different password cannot decrypt the restored rows, a changed governance domain scope invalidates every ballot signed under the old name, and a changed custodian credential identity key leaves stored credential identities incomparable until the backfill re-hashes them. Restore the credentials of a custody provider from the same source. When secretManager.provider is not local, restore that store to the same point as PostgreSQL. Values and secrets describes each value.
2. Restore PostgreSQL through your provider, either from a backup or to a chosen moment (Point-in-time recovery). Point global.datastores at the restored instance. The network catalogue and the RPC endpoint pools are in this database, so they return with it. Their credentials decrypt with the master password from step 1.
3. Restore object storage through your provider, from its versions or its replica.
4. Provide a Redis service. There is nothing to restore, because the platform treats Redis as a cache and notification transport.
5. Install or upgrade the dalp chart. With the default hook type, the migration Job runs first, and the application Deployments update only after it succeeds. See Upgrades and migrations.
6. Confirm that every network's endpoint pool is reachable from the new site, and that each network still has at least two endpoints.
7. Watch the indexer catch up, as Catch-up and re-index time describes. Confirm that the application starts cleanly against the restored data, that the Platform status screen shows no stalled workflows, and that the newest restored records agree with the ledger.
8. Switch DNS, ingress and routing to the recovered site, and reopen traffic.

Catch-up and re-index time#

The platform publishes no figure for how long indexing takes after a restore, because the time depends on your chain and your endpoints. Two cases differ.

After a database restore, the indexer reads only the blocks between the stored position and the chain head. Divide that gap in blocks by the rate at which the lag shrinks to size the catch-up. The Console's Platform status screen shows how many chains are in sync and the delay of the furthest-behind chain, its Indexing detail view shows the state per network, and the indexer exports the dalp_didx_watermark_lag_blocks and dalp_indexer_serving_lag_blocks gauges. Ledger indexing stalled explains how to tell catch-up from a stall.

A rebuild is longer. A release that changes how earlier data is recorded declares one, and an operator can request one, as Ledger indexing stalled describes for a chain generation mismatch. The platform then indexes each network again from its start block into a new dataset while the current one keeps serving. Rebuild time grows with the length of the chain history, the volume of events and your provider's rate limits. The rebuild needs an archive-capable endpoint, because it asks for state at old blocks. Reads keep using the current dataset, and a read that needs data that is not available yet can receive a retryable rebuilding response. Measure a rebuild on your own data before you rely on it in a recovery plan.

Failing over#

Failover splits into three jobs, because they answer to three different owners: compute, the stateful layer and the ledger connection.

Compute failover is a reschedule, because every Platform API pod and the Console are stateless. Recovering compute also requires reachable datastores, secrets, registry access, network endpoints, and routing in the target environment. Validate those dependencies before sending traffic there.

The stateful layer is where failover is decided, and it depends entirely on the services you supply. Failover for PostgreSQL, Redis and object storage is your provider's own replication and promotion mechanism, not something these charts orchestrate. A provider running a live standby promotes it, and your recovery point tracks that standby's replication lag, not the age of your last Velero backup.

A datastore with no live standby has nothing to fail over to. A cluster or regional loss then means restoring your most recent backup into new infrastructure, not promoting a replica. Recovery reaches the latest usable backup. Successful daily backups can leave a 24-hour gap, and missed or unusable backups make the gap longer. Choose services with a replica or standby for production.

Failover to the ledger is a third job, and the platform does it for you when a network has more than one endpoint. The platform scores each endpoint in a network's pool on its error rate, its latency, how far it lags behind the chain head and how far it lags behind finality. Each endpoint also carries a weight that a platform administrator sets in the pool. The platform then routes as follows:

• It leaves out an endpoint whose head lag passes 50 blocks or whose score falls below 0.25.
• It sends calls to the healthy endpoint with the highest weight, and prefers the better score among endpoints of equal weight.
• It retries a transient error with backoff and moves away from a dead endpoint.
• If every reachable endpoint is excluded, it keeps calling the excluded ones rather than failing.
• It keeps the serving endpoint when the best rival has the same weight, the last switch was under 60 seconds ago and the rival scores less than 0.1 higher. This stops two healthy endpoints from trading places. An endpoint that falls out of the healthy set is replaced without waiting for that interval.
• If no endpoint can serve a call, the caller receives an error.

The limits and the switching values are defaults. A platform administrator edits the pool in the Console under Networks, and the change applies without a restart. Health-based selection needs fresh scores. When the platform holds no usable score for any endpoint of a network, selection is not active for that network and the platform logs it as degraded. It then chooses endpoints in their configured order and stops excluding unhealthy ones, and after 15 minutes the RpcHealthSelectionDisabled alert fires. Treat that alert as degraded failover until scores return. A network with a single endpoint has nothing to fail over to.

What sets your RTO and RPO#

Your RPO is the longest span of recent data you accept losing, stated as a time. Your RTO is the longest outage you accept before the platform serves traffic again. DALP states neither as a number, and no figure on this page is a commitment. Each one depends on choices that sit with different owners, and the weakest store sets the objective for the platform as a whole.

Set the objectives per data class with your risk owner. Off-chain records and documents have no other copy, and usually need the tightest RPO. Indexed read models can be rebuilt from the ledger, so a looser RPO costs time and not data. Retention (backup.retention.days, 30 by default) sets how far back you can restore, and does not set the RPO. A provider's published service levels apply to that provider's service alone and do not add up to an objective for the platform as a whole.

Rehearse a recovery#

DALP ships no disaster recovery test tool, so you schedule and record the drill. Run it on the occasions that Verify a backup is usable lists, and after a change of database or object storage provider.

1. Build an isolated target. The target holds production's master password, workflow journals and, with the default secret manager, signer keys, and its restored endpoint pools point at production endpoints.
   • Use a separate namespace or cluster, and never the namespace that serves production.
   • Before the pods start, block all egress except to ledger endpoints that serve reads only. That includes your custody provider and webhook receivers. A local signer signs inside the platform, so blocking the custody provider alone does not stop it.
   • Give the target its own DALP_GOVERNANCE_DOMAIN_SCOPE, because the chart's values say never to copy the scope between installs. Ballots signed under the production scope then stop verifying in the target, which a drill does not need.
   • Copy CUSTODY_CREDENTIAL_IDENTITY_KEY, although the values give the same rule for it, so the restored custody credentials stay readable. That is one more reason to keep the target isolated and delete it in step 6.
   • After the pods start, replace each pool in the Console under Networks with endpoints that serve reads only and refuse transaction submission, before you time the indexer catch-up. The network catalogue in the database overrides the deployment values.
2. Note the moment at which you simulate the failure. Restore PostgreSQL, object storage and the secrets from the sources your plan names, following steps 1 to 7 of Restoring a whole instance, apart from the governance domain scope in step 1. Run the checks in step 5 below before any routing change, and simulate step 8 without moving production traffic.
3. Time the drill from the moment you noted to the point where the target could serve traffic. Record each stage as well: the secrets, the database restore, the object storage restore, the migration Job, the pods becoming ready and the indexer's lag reaching zero. Stages that run beside each other count once. The result is your measured RTO. If your drill skips a stage, say which one.
4. Compare the newest record in the restored instance with production at the moment you noted in step 2. The difference is your measured RPO.
5. Run the checks in Verify a backup is usable and confirm that the application starts against the restored data.
6. Record the timings and the gaps against the objectives, delete the target, and fix what the drill exposed before the next one.

What deletion removes#

Once retention.days elapses, 30 days by default, Velero deletes that recovery point and the objects it wrote from your bucket, with no undo. Deleting the bucket a BackupStorageLocation points at destroys every recovery point that location owns, not only the newest one.

Measure recovery time with your own volumes. The chart does not guarantee a restore duration.

For restores into a namespace with existing resources, check the behaviour of your installed Velero version and its resource policy first. An isolated test target avoids relying on overwrite behaviour.

Frequently asked questions#