# Hot-warm active-standby

Source: https://docs.settlemint.com/docs/architects/self-hosting/high-availability/hot-warm
Geographic recovery pattern that keeps a second DALP cluster ready for promotion, with RTO of 30 to 180 minutes and a manual operator-driven failover sequence.



Related pages: [HA overview](/docs/architects/self-hosting/high-availability), [Cloud-native HA](/docs/architects/self-hosting/high-availability/cloud-native), [Hot-cold HA](/docs/architects/self-hosting/high-availability/hot-cold), [Backup and recovery](/docs/architects/self-hosting/high-availability/backup-recovery), [Self-hosting prerequisites](/docs/architects/self-hosting/prerequisites)

***

A hot-warm deployment runs one active DALP cluster and keeps a second cluster ready to promote. Use this pattern when one region serves live traffic and another region needs recovery without a full rebuild. The operating model assumes a manual failover window measured in tens of minutes.

## Architecture [#architecture]

<Mermaid
  chart="`flowchart TB
  gslb(Traffic management)
  subgraph active[&#x22;Active cluster A&#x22;]
    av(Live validators)
    arpc(Active RPC or dApp routes)
    apg(Primary PostgreSQL)
    av --> arpc
  end
  subgraph standby[&#x22;Warm standby cluster B&#x22;]
    wv(Warm validators with staged operations)
    wrpc(Warm RPC or dApp routes)
    rpg(Replica PostgreSQL)
    wv --> wrpc
  end
  gslb -->|Production traffic| active
  apg -->|Continuous replication| rpg
  subgraph failover[&#x22;Manual failover sequence&#x22;]
    f1(1. Stop writes in cluster A)
    f2(2. Promote PostgreSQL replica)
    f3(3. Start standby DALP workloads)
    f4(4. Switch DNS or traffic)
    f5(5. Validate quorum and routes)
    f1 --> f2 --> f3 --> f4 --> f5
  end
`"
/>

The active cluster handles all user and RPC traffic plus validator operations. The warm cluster stays ready for promotion with infrastructure pre-staged, images cached, secrets loaded, and routes configured. Warm workloads do not serve production writes until failover. PostgreSQL replication moves application state from the active region to the standby region. Object storage and backups follow the choices in the self-hosting prerequisites. Monitoring follows those same choices.

## Quickstart [#quickstart]

Run these checks before you call a standby cluster warm. Replace the namespace and labels with the values used in your installation.

```bash
export ACTIVE_CONTEXT=dalp-active
export STANDBY_CONTEXT=dalp-standby
export DALP_NAMESPACE=dalp

kubectl --context "$ACTIVE_CONTEXT" -n "$DALP_NAMESPACE" get pods
kubectl --context "$STANDBY_CONTEXT" -n "$DALP_NAMESPACE" get pods
kubectl --context "$STANDBY_CONTEXT" -n "$DALP_NAMESPACE" get secrets
```

A healthy pre-failover result shows the active cluster serving workloads and the standby cluster holding the resources needed for promotion.

```text
NAME                              READY   STATUS    RESTARTS   AGE
dalp-dapp-6fdbf8f6f8-abc12        1/1     Running   0          3d
dalp-dapi-7f9d7bcb7c-def34        1/1     Running   0          3d
postgresql-primary-1              1/1     Running   0          3d
```

```text
NAME                              READY   STATUS    RESTARTS   AGE
dalp-dapp-6b87c7d45b-ghi56        1/1     Running   0          3d
dalp-dapi-66c76f8b74-jkl78        1/1     Running   0          3d
postgresql-replica-1              1/1     Running   0          3d
```

Treat this quickstart as an operator readiness check, not as a failover command. The warm workloads can be ready without serving production writes. Actual failover changes database leadership, validator participation, and traffic routing. Run failover only through a controlled incident procedure.

## When hot-warm fits [#when-hot-warm-fits]

Use hot-warm when all of these conditions are true:

* You need geographic recovery for a regional outage.
* You accept an RTO of 30 to 180 minutes.
* You accept an RPO of 5 to 60 minutes, depending on replication lag and backup posture.
* You can keep trained operators available for manual promotion and validation.
* You can pre-stage validator operations and secrets in the standby region. DNS or traffic-manager changes and alert configuration must also be ready before failover.

Hot-warm is not a substitute for single-region multi-AZ protection. Use [cloud-native HA](/docs/architects/self-hosting/high-availability/cloud-native) when the failure target falls within one region.

## Recovery metrics [#recovery-metrics]

| Metric | Target            | What drives the number                                                                                            |
| ------ | ----------------- | ----------------------------------------------------------------------------------------------------------------- |
| RTO    | 30 to 180 minutes | Operator availability, replica promotion, workload start time, DNS or traffic-manager change, and validation time |
| RPO    | 5 to 60 minutes   | PostgreSQL replication lag, backup frequency, and object-storage replication posture                              |
| RTT    | 1 to 6 hours      | Failover execution, application validation, reconciliation checks, and rollback decision time                     |

The platform does not make a manual failover automatic. Your runbook and staffing model determine whether the deployment meets these targets. Drills and monitoring records provide the evidence.

## Production requirements [#production-requirements]

| Requirement            | Production expectation                                                                                                                                                                                            |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Two clusters           | Run separate Kubernetes or OpenShift clusters in separate failure domains. Keep cluster versions, namespaces, ingress, and chart configuration in sync.                                                           |
| PostgreSQL replication | Use managed cross-region replication or CloudNativePG replication, depending on whether PostgreSQL is managed or self-hosted. Monitor lag continuously.                                                           |
| Backups                | Use provider-managed backups or Velero and CloudNativePG backups. Verify restore, not only backup creation.                                                                                                       |
| Object storage         | Use managed object storage or RustFS with S3-compatible configuration. Match retention and replication settings to the recovery point objective.                                                                  |
| Secrets and keys       | Pre-stage required Kubernetes secrets and key material in the standby cluster through your approved secret-management process. Do not start duplicate signing or validator operations against production traffic. |
| Traffic management     | Keep DNS, load balancer, or global traffic-manager changes documented and rehearsed. Set TTLs that match the expected failover window.                                                                            |
| Observability          | Collect metrics, logs, traces, and alerts from both clusters. Alert on standby health, replication lag, backup failures, and expired certificates.                                                                |
| Operator runbook       | Keep a dated runbook that names the decision owner, promotion steps, validation checks, rollback conditions, and communication path.                                                                              |

## Manual failover sequence [#manual-failover-sequence]

1. Declare the incident and freeze production writes if the active region still accepts traffic.
2. Confirm the latest usable PostgreSQL replica or backup in the standby region.
3. Promote the standby PostgreSQL replica according to your managed database or CloudNativePG procedure.
4. Start the standby DALP workloads that depend on the promoted database.
5. Enable standby validator operations and confirm the former active validators cannot produce duplicate signatures.
6. Switch DNS, load balancer, or global traffic-manager routing to the standby cluster.
7. Validate Console routes, API routes, RPC access, and validator health. Confirm observability coverage and audit evidence.
8. Keep the former active region isolated until reconciliation confirms whether it can return as standby.

Each step needs a named owner and a stop condition. If PostgreSQL promotion, route checks, or validator health fails, stop the failover. Follow the backup-recovery runbook rather than continuing with a partially promoted region.

## Operational checks [#operational-checks]

| Check                      | Minimum frequency        | Evidence to keep                                                                       |
| -------------------------- | ------------------------ | -------------------------------------------------------------------------------------- |
| Replication lag            | Daily, plus alerting     | Current lag, threshold, and last healthy timestamp                                     |
| Standby workload readiness | Daily                    | Pod readiness, image versions, required secrets, and pending configuration drift       |
| Backup restore test        | Weekly for critical data | Restore timestamp, restored object count or database checkpoint, and validation result |
| Certificate and DNS review | Monthly                  | Expiry dates, DNS TTLs, and active routing target                                      |
| Failover drill             | Quarterly                | RTO, RPO, failed steps, owner, and remediation items                                   |
| Security patching          | Monthly                  | Patched cluster versions, operator versions, and workload image versions               |

A hot-warm design loses value when the standby region drifts. Treat drift as a live incident when it blocks promotion, breaks recovery records, or leaves keys, secrets, certificates, or routing targets stale.

## Compliance and audit notes [#compliance-and-audit-notes]

Auditors verify uptime and processing integrity from the failover record, not only from infrastructure status. Keep records covering the incident decision, replica or backup timestamp, data-loss assessment, operator steps, route switch, validation checks, and the reconciliation outcome.

If your deployment handles regulated workloads, map the runbook to the applicable uptime, confidentiality, and processing-integrity controls. Do not publish an RTO or RPO target externally unless drills and operational records back it.

## Next steps [#next-steps]

* Use [self-hosting prerequisites](/docs/architects/self-hosting/prerequisites) to choose managed or self-hosted PostgreSQL, object storage, backup services, and observability tooling.
* Use [backup and recovery](/docs/architects/self-hosting/high-availability/backup-recovery) to define restore validation and recovery evidence.
* Use [hot-hot HA](/docs/architects/self-hosting/high-availability/hot-hot) only when you need concurrent active regions and can operate the added consensus and traffic-routing complexity.