Hot-warm active-standby

Related pages: HA overview, Cloud-native HA, Hot-cold HA, Backup and recovery, Self-hosting prerequisites

A hot-warm deployment runs one active DALP cluster and keeps a second cluster ready to promote. Use this pattern when one region serves live traffic and another region needs recovery without a full rebuild. The operating model assumes a manual failover window measured in tens of minutes.

Architecture

The active cluster handles all user and RPC traffic plus validator operations. The warm cluster stays ready for promotion with infrastructure pre-staged, images cached, secrets loaded, and routes configured. Warm workloads do not serve production writes until failover. PostgreSQL replication moves application state from the active region to the standby region. Object storage and backups follow the choices in the self-hosting prerequisites. Monitoring follows those same choices.

Quickstart

Run these checks before you call a standby cluster warm. Replace the namespace and labels with the values used in your installation.

export ACTIVE_CONTEXT=dalp-active
export STANDBY_CONTEXT=dalp-standby
export DALP_NAMESPACE=dalp

kubectl --context "$ACTIVE_CONTEXT" -n "$DALP_NAMESPACE" get pods
kubectl --context "$STANDBY_CONTEXT" -n "$DALP_NAMESPACE" get pods
kubectl --context "$STANDBY_CONTEXT" -n "$DALP_NAMESPACE" get secrets

A healthy pre-failover result shows the active cluster serving workloads and the standby cluster holding the resources needed for promotion.

NAME                              READY   STATUS    RESTARTS   AGE
dalp-dapp-6fdbf8f6f8-abc12        1/1     Running   0          3d
dalp-dapi-7f9d7bcb7c-def34        1/1     Running   0          3d
postgresql-primary-1              1/1     Running   0          3d

NAME                              READY   STATUS    RESTARTS   AGE
dalp-dapp-6b87c7d45b-ghi56        1/1     Running   0          3d
dalp-dapi-66c76f8b74-jkl78        1/1     Running   0          3d
postgresql-replica-1              1/1     Running   0          3d

Treat this quickstart as an operator readiness check, not as a failover command. The warm workloads can be ready without serving production writes. Actual failover changes database leadership, validator participation, and traffic routing. Run failover only through a controlled incident procedure.

When hot-warm fits

Use hot-warm when all of these conditions are true:

• You need geographic recovery for a regional outage.
• You accept an RTO of 30 to 180 minutes.
• You accept an RPO of 5 to 60 minutes, depending on replication lag and backup posture.
• You can keep trained operators available for manual promotion and validation.
• You can pre-stage validator operations and secrets in the standby region. DNS or traffic-manager changes and alert configuration must also be ready before failover.

Hot-warm is not a substitute for single-region multi-AZ protection. Use cloud-native HA when the failure target falls within one region.

Recovery metrics

The platform does not make a manual failover automatic. Your runbook and staffing model determine whether the deployment meets these targets. Drills and monitoring records provide the evidence.

Production requirements

Manual failover sequence

1. Declare the incident and freeze production writes if the active region still accepts traffic.
2. Confirm the latest usable PostgreSQL replica or backup in the standby region.
3. Promote the standby PostgreSQL replica according to your managed database or CloudNativePG procedure.
4. Start the standby DALP workloads that depend on the promoted database.
5. Enable standby validator operations and confirm the former active validators cannot produce duplicate signatures.
6. Switch DNS, load balancer, or global traffic-manager routing to the standby cluster.
7. Validate Console routes, API routes, RPC access, and validator health. Confirm observability coverage and audit evidence.
8. Keep the former active region isolated until reconciliation confirms whether it can return as standby.

Each step needs a named owner and a stop condition. If PostgreSQL promotion, route checks, or validator health fails, stop the failover. Follow the backup-recovery runbook rather than continuing with a partially promoted region.

Operational checks

A hot-warm design loses value when the standby region drifts. Treat drift as a live incident when it blocks promotion, breaks recovery records, or leaves keys, secrets, certificates, or routing targets stale.

Compliance and audit notes

Auditors verify uptime and processing integrity from the failover record, not only from infrastructure status. Keep records covering the incident decision, replica or backup timestamp, data-loss assessment, operator steps, route switch, validation checks, and the reconciliation outcome.

If your deployment handles regulated workloads, map the runbook to the applicable uptime, confidentiality, and processing-integrity controls. Do not publish an RTO or RPO target externally unless drills and operational records back it.

Next steps

• Use self-hosting prerequisites to choose managed or self-hosted PostgreSQL, object storage, backup services, and observability tooling.
• Use backup and recovery to define restore validation and recovery evidence.
• Use hot-hot HA only when you need concurrent active regions and can operate the added consensus and traffic-routing complexity.