# Hot-cold backup recovery

Source: https://docs.settlemint.com/docs/architects/self-hosting/high-availability/hot-cold
Use hot-cold disaster recovery when a self-hosted DALP deployment can accept restore-based recovery, backup-dependent RPO, and multi-hour RTO in exchange for a lower standby cost.



Related pages: [High availability overview](/docs/architects/self-hosting/high-availability), [Hot-warm active-standby](/docs/architects/self-hosting/high-availability/hot-warm), [Backup and recovery](/docs/architects/self-hosting/high-availability/backup-recovery), and [Self-hosting prerequisites](/docs/architects/self-hosting/prerequisites)

***

Hot-cold recovery keeps one DALP environment active. To recover, you rebuild the DALP service in a fresh environment from backups, infrastructure-as-code, and a tested restore runbook. Choose hot-cold only when the deployment can tolerate a backup-dependent recovery point and recovery measured in hours.

<Callout type="warn">
  Hot-cold is a restore pattern, not a live failover pattern. Do not use it for production financial workloads that need
  near-zero data loss, automatic failover, or recovery that is measured in minutes.
</Callout>

## When hot-cold fits [#when-hot-cold-fits]

Hot-cold fits environments where cost matters more than fast recovery. Confirm all four conditions before adopting this pattern.

* The environment is development, staging, sandbox, or non-critical production-adjacent.
* A multi-hour outage is acceptable during a regional incident.
* Application state can be restored from PostgreSQL backups and reconciled against the chain.
* The operator can rebuild the DALP namespace, secrets, ingress, database, object storage access, monitoring stack, and RPC configuration from versioned runbooks.

Use [cloud-native HA](/docs/architects/self-hosting/high-availability/cloud-native) for the default self-hosted production baseline. Use [hot-warm active-standby](/docs/architects/self-hosting/high-availability/hot-warm) when the recovery region must already contain a warm DALP stack and database replica.

## Architecture [#architecture]

<Mermaid
  chart="`flowchart TB
  users[Operators and API clients]
  active[Active DALP environment]
  db[(PostgreSQL data)]
  storage[(Object storage and exports)]
  backups[Backup schedules and WAL archive]
  git[Versioned Helm values and runbooks]
  cold[Cold recovery environment]
  restore[Restore and validate service]
  chain[EVM network and RPC endpoints]

  users --> active
  active --> db
  active --> storage
  db --> backups
  storage --> backups
  git --> cold
  backups --> cold
  cold --> restore
  chain --> restore
  restore --> users

`"
/>

The active environment serves traffic and writes application state. Backup jobs preserve PostgreSQL data, Kubernetes resources, object storage data, observability data, and configuration history. During an incident, the operator provisions or activates the recovery environment, restores the latest approved backup set, points DALP services at the restored dependencies, and validates chain-facing workflows before reopening service.

## Recovery metrics [#recovery-metrics]

Hot-cold targets are deployment-specific. Treat the numbers below as planning ranges that you must prove with drills.

| Metric | Planning range | What drives the result                                                                                       |
| ------ | -------------- | ------------------------------------------------------------------------------------------------------------ |
| RTO    | 8 to 72 hours  | Cluster provisioning, restore speed, image availability, secrets access, DNS or ingress changes, validation  |
| RPO    | 4 to 24 hours  | PostgreSQL backup frequency, WAL retention, object-storage replication, and the last successful backup check |
| RTT    | 12 to 96 hours | Full restore, chain reconciliation, indexer catch-up, downstream checks, and incident closure evidence       |

Do not publish an RTO or RPO commitment from this pattern alone. Your commitment belongs to the specific deployment, the infrastructure providers it relies on, and the tested operating procedure.

## Restore sequence [#restore-sequence]

1. Declare the incident and stop writes to the affected environment when it is still reachable.
2. Select the latest backup set that satisfies the deployment recovery point target.
3. Provision or activate the recovery Kubernetes or OpenShift environment.
4. Restore PostgreSQL to the selected point in time.
5. Restore Kubernetes resources, secrets, configuration, object storage access, and observability components needed by DALP.
6. Start DALP services against the restored database and dependency configuration.
7. Reconnect RPC endpoints, custody or signing dependencies, and monitoring routes.
8. Validate login, API availability, asset reads, transaction submission, event indexing, and audit exports.
9. Record the achieved RTO and RPO, then update the runbook if any manual step was missing or slower than expected.

If you cannot rebuild the environment from Git, from your approved backups, and from the approved secret-management process, the hot-cold plan is not ready.

## What must be backed up [#what-must-be-backed-up]

| Surface              | Recovery expectation                                                                                            |
| -------------------- | --------------------------------------------------------------------------------------------------------------- |
| PostgreSQL           | Restore application state to a selected point in time through managed PITR or WAL-backed backups.               |
| Kubernetes resources | Recreate namespace resources, services, ingress, config maps, secrets references, and persistent data.          |
| Object storage       | Recover files, exported artefacts, backup payloads, and storage configuration needed by the services.           |
| Configuration        | Reapply Helm values, environment configuration, ingress settings, and network-specific RPC settings.            |
| Observability        | Preserve enough telemetry (logs, metrics, traces, and alert history) to review the incident and prove recovery. |

For backup scheduling and PITR window configuration, see [Backup and recovery](/docs/architects/self-hosting/high-availability/backup-recovery). That page also covers the restore-test evidence you need before approving the deployment for production use.

## Operator checks [#operator-checks]

Before you approve hot-cold for an environment, confirm these conditions are already in place:

* The backup job writes to storage outside the failed cluster or failed region.
* At least one full restore test has succeeded in an isolated environment.
* Database restore, namespace restore, object-storage access, and service startup are written as repeatable runbook steps.
* Secrets and key material can be restored through the approved secret-management process without copying secrets into documentation.
* DNS, ingress, TLS certificates, RPC endpoints, and custody or signing provider access are included in the incident checklist.
* Monitoring alerts cover backup failure, restore-test age, pod availability, database health, storage access, and RPC availability.
* The incident owner knows when to keep the environment offline for reconciliation instead of reopening service quickly.

## Comparison with other patterns [#comparison-with-other-patterns]

| Pattern      | Best fit                                    | Recovery posture                                                          |
| ------------ | ------------------------------------------- | ------------------------------------------------------------------------- |
| Cloud-native | Standard self-hosted production baseline    | Multi-zone application placement with managed-service HA.                 |
| Hot-warm     | Regional recovery with a ready standby      | Manual promotion of a warm region and replicated data.                    |
| Hot-cold     | Lower-cost recovery for tolerant workloads  | Rebuild from backup data, versioned runbooks, and provider configuration. |
| Hot-hot      | Active-active regional service requirements | Multiple active regions with stronger consistency controls.               |

## Related pages [#related-pages]

* [High availability overview](/docs/architects/self-hosting/high-availability)
* [Cloud-native HA](/docs/architects/self-hosting/high-availability/cloud-native)
* [Hot-warm active-standby](/docs/architects/self-hosting/high-availability/hot-warm)
* [Backup and recovery](/docs/architects/self-hosting/high-availability/backup-recovery)
* [Self-hosting prerequisites](/docs/architects/self-hosting/prerequisites)