# Hot-hot active-active HA

Source: https://docs.settlemint.com/docs/architects/self-hosting/high-availability/hot-hot
Compare DALP hot-hot and hybrid multi-region deployment patterns for consortium and public EVM networks, including provider patterns, outage behaviour, recovery targets, and when to choose this model.



Hot-hot is DALP's active-active availability pattern. More than one region serves traffic at the same time, which reduces user-facing failover time but raises the operating burden for traffic routing, data consistency, indexed-state reconciliation, and incident ownership.

Before you choose this pattern, compare the simpler alternatives. Start with [cloud-native](/docs/architects/self-hosting/high-availability/cloud-native), then weigh [hot-warm](/docs/architects/self-hosting/high-availability/hot-warm) or [hot-cold](/docs/architects/self-hosting/high-availability/hot-cold). Most self-hosted deployments need nothing beyond multi-zone HA or hot-warm.

## Choose the correct hot-hot variant [#choose-the-correct-hot-hot-variant]

DALP uses the same active-active idea in two different operating models:

| Variant            | Use when                                                                                   | Recovery model                                                                                | Main operator burden                                                                                       |
| ------------------ | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Consortium network | You operate validators or validator-adjacent infrastructure across regions.                | Keep validators, RPC access, DALP services, and PostgreSQL topology healthy across clusters.  | Consensus participation, validator placement, cross-cluster database design, and regional traffic routing. |
| Public EVM network | The chain is external and DALP reads on-chain truth through RPC and Ledger Index indexing. | Shift user traffic to a healthy cluster and rebuild indexed state from the chain when needed. | RPC availability, Ledger Index sync health, database failover, and reconciliation after cluster failover.  |

## Hybrid multi-region deployments [#hybrid-multi-region-deployments]

DALP supports hybrid deployments where core platform services remain in an on-premises Kubernetes or OpenShift estate while blockchain access, RPC nodes, validator-adjacent infrastructure, or Ledger Index indexing capacity runs in a separate cloud estate. AWS, Azure, and GCP are each supported. Each region must offer the full managed-services stack: Kubernetes or OpenShift, relational database (PostgreSQL), cache, object storage, backup, plus monitoring. SettleMint confirms the exact provider regions during deployment planning, as region availability depends on the selected cloud account and regulatory boundary.

<Mermaid
  chart="`flowchart LR
  users[&#x22;Users and integrations&#x22;] --> core[&#x22;On-premises DALP estate<br/>dApp, API, workers, database&#x22;]
  core --> primary[&#x22;Cloud region A<br/>RPC nodes or DIDX&#x22;]
  core --> secondary[&#x22;Cloud region B<br/>RPC nodes or DIDX standby&#x22;]
  primary --> chain[&#x22;EVM network<br/>source of truth&#x22;]
  secondary --> chain
  primary -. regional outage .-> secondary
`"
/>

Treat the hybrid split as an operating boundary, not only as a network diagram. Each side needs clear ownership, health checks, credentials, route failover, and recovery evidence. Plan your runbook and assign incident ownership for both sides before production.

| Surface          | What can be split across estates                                                                                                                                                                                                                   | What must stay consistent                                                                                                     |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| DALP services    | Console, API, workers, ingress, and observability can run in the primary application cluster or on-premises estate.                                                                                                                                | Chart values, secrets, PostgreSQL connectivity, Redis connectivity, object storage access, and route health.                  |
| Blockchain nodes | Consortium deployments can run validators and RPC nodes in separate regions or clusters when the network design supports regional node placement. Public-network deployments use external RPC access instead of operating public-chain validators. | Chain ID, genesis or network configuration, finality assumptions, RPC authentication, provider limits, and failover runbooks. |
| Ledger Index     | The Ledger Index can run beside the DALP services or in a cloud estate with RPC access. The Ledger Index rebuilds chain-derived state from the chain and PostgreSQL checkpoints.                                                                   | Per-chain checkpoints, block lag, reorg handling, registered contract coverage, and indexed-state validation after failover.  |
| Data services    | PostgreSQL, cache, object storage, and backups can use managed cloud services or approved self-hosted services.                                                                                                                                    | HA mode, replication lag, backup retention, restore access, and tested application reconnection.                              |

## Supported cloud provider pattern [#supported-cloud-provider-pattern]

Use AWS, Azure, or GCP regions that meet the self-hosting prerequisites. DALP does not require a fixed region list. Your deployment must use pairs or recovery regions approved by the operator, the cloud account, and the data-residency requirement.

| Provider family | Cloud services used in the pattern                                                                                                                               | Region requirement                                                                                      |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| AWS             | EKS or OpenShift, RDS PostgreSQL Multi-AZ, ElastiCache Multi-AZ, S3, CloudWatch, Managed Prometheus, and Managed Grafana.                                        | Choose primary and recovery regions where these services are available and approved for the deployment. |
| Azure           | AKS or OpenShift, Azure Database for PostgreSQL Flexible Server with zone-redundant HA, Azure Cache for Redis, Blob Storage, Azure Monitor, and Managed Grafana. | Choose primary and recovery regions where these services are available and approved for the deployment. |
| GCP             | GKE or OpenShift, Cloud SQL Regional HA, Memorystore Standard tier, Cloud Storage, Cloud Monitoring, and Cloud Logging.                                          | Choose primary and recovery regions where these services are available and approved for the deployment. |

## Regional cloud outage behaviour [#regional-cloud-outage-behaviour]

During a regional outage, DALP can continue operating through the remaining active region only for the surfaces you have deployed and tested there. If the cloud-hosted node or indexer is single-region, the on-premises DALP estate can stay up, but chain reads, chain writes, and indexed-state freshness depend on restoring RPC and Ledger Index access.

| Surface                       | Failover behaviour                                                                                        | RTO expectation | RPO expectation                                                                                              |
| ----------------------------- | --------------------------------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------ |
| RPC nodes or external RPC     | Route DALP to the healthy RPC endpoint or provider region after health checks fail.                       | 1 to 10 minutes | Seconds to minutes for endpoint freshness; 0 for on-chain state because the EVM chain remains authoritative. |
| Consortium validators         | Surviving validators keep the network healthy only when the consensus design tolerates the failed region. | 1 to 10 minutes | Seconds to minutes, depending on finality and database replication lag.                                      |
| Ledger Index with healthy RPC | The indexer resumes from the last checkpoint and catches up from chain data.                              | 1 to 10 minutes | Seconds to minutes for checkpointed indexed state.                                                           |
| Ledger Index full rebuild     | Rebuild indexed state from the chain when checkpointed state or the indexed database cannot be trusted.   | 5 to 60 minutes | Not applicable to on-chain truth.                                                                            |
| On-premises DALP services     | The application estate stays available if its database, cache, routes, and secrets remain healthy.        | 1 to 10 minutes | Seconds to minutes, depending on database, cache, and route failover state.                                  |

A functioning blockchain node does not prove the application estate is ready. A running application pod does not prove that RPC, indexing, or database recovery can survive a regional incident. Production readiness requires both views: service health from Kubernetes and DALP observability, plus chain health from RPC and Ledger Index lag. Include finality signals and reorg signals in the same check.

## Consortium networks [#consortium-networks]

In a consortium network, hot-hot means several active regions participate in the operating model. Each region runs DALP services, RPC access, and a PostgreSQL instance. Add any validator infrastructure your network design requires.

<Mermaid
  chart="`flowchart TB
  gslb(Global traffic management)
  incident(Shared incident owner)
  subgraph clusterA[&#x22;Cluster A&#x22;]
    va(Validator)
    rpca(RPC nodes)
    dappa(DALP services)
    pga(PostgreSQL)
  end
  subgraph clusterB[&#x22;Cluster B&#x22;]
    vb(Validator)
    rpcb(RPC nodes)
    dappb(DALP services)
    pgb(PostgreSQL)
  end
  subgraph clusterC[&#x22;Cluster C&#x22;]
    vc(Validator)
    rpcc(RPC nodes)
    dappc(DALP services)
    pgc(PostgreSQL)
  end
  gslb --> clusterA
  gslb --> clusterB
  gslb --> clusterC
  incident --> clusterA
  incident --> clusterB
  incident --> clusterC
`"
/>

### Recovery targets [#recovery-targets]

| Metric             | Target             | Notes                                                                 |
| ------------------ | ------------------ | --------------------------------------------------------------------- |
| RTO                | 1 to 10 minutes    | Traffic management shifts users away from an unhealthy region.        |
| RPO                | Seconds to minutes | Depends on database replication lag and the final failover procedure. |
| Recovery test time | 10 to 60 minutes   | Includes health checks, traffic rerouting, and operator validation.   |

### Setup and maintenance [#setup-and-maintenance]

| Task                                  | Time estimate | Client role              |
| ------------------------------------- | ------------- | ------------------------ |
| Four-cluster provisioning             | 1 to 2 days   | Platform engineer        |
| Network connectivity, peering, or VPN | 1 to 2 days   | Network engineer         |
| CloudNativePG setup across clusters   | 1 to 2 days   | Platform engineer        |
| PostgreSQL distributed topology       | 2 to 3 days   | DBA or platform engineer |
| Failover automation and testing       | 2 to 3 days   | Platform engineer        |
| End-to-end DR drill                   | 1 to 2 days   | Platform team            |
| Initial setup                         | 3 to 5 weeks  | 2 to 3 client engineers  |

| Activity                             | Frequency | Time per cycle |
| ------------------------------------ | --------- | -------------- |
| Cross-cluster replication monitoring | Daily     | 30 minutes     |
| Backup verification across clusters  | Weekly    | 2 hours        |
| Helm chart updates across clusters   | Monthly   | 4 to 8 hours   |
| DR drill or failover test            | Quarterly | 1 to 2 days    |
| Security patching across clusters    | Monthly   | 1 to 2 days    |
| Monthly effort                       |           | 40 to 60 hours |

Plan for 1.5 to 2 FTE platform engineers plus DBA support. A 24/7 on-call rotation is required. Choose this model only when concurrent active regions and low failover time outweigh the added operating burden.

## Public EVM networks [#public-evm-networks]

For public EVM networks, DALP does not operate the chain validators. The external chain remains the source of truth. DALP keeps your user-facing services available across regions and uses RPC together with the Ledger Index to read chain data and rebuild chain-derived state.

<Mermaid
  chart="`flowchart TB
  gslb(Global traffic management)
  subgraph clusterA[&#x22;Cluster A&#x22;]
    rpca(RPC access)
    dappa(DALP services)
    indexa(DIDX indexing)
    pga(PostgreSQL indexed data)
  end
  subgraph clusterB[&#x22;Cluster B&#x22;]
    rpcb(RPC access)
    dappb(DALP services)
    indexb(DIDX indexing)
    pgb(PostgreSQL indexed data)
  end
  subgraph chain[&#x22;Public EVM network&#x22;]
    truth(On-chain source of truth)
  end
  gslb --> clusterA
  gslb --> clusterB
  clusterA --> truth
  clusterB --> truth
`"
/>

### What changes from consortium hot-hot [#what-changes-from-consortium-hot-hot]

* The operator does not manage validators for the public chain.
* Indexed data can be rebuilt by replaying chain data through Ledger Index.
* Regional failover focuses on service health, RPC reachability, and Ledger Index sync. PostgreSQL availability is part of the same check.
* Recovery evidence must include indexed-state checks, not only Kubernetes pod health.

### Recovery targets [#recovery-targets-1]

| Scenario               | RTO                | RPO            | Notes                                                                     |
| ---------------------- | ------------------ | -------------- | ------------------------------------------------------------------------- |
| Single pod failure     | Less than 1 minute | 0              | Kubernetes reschedules automatically.                                     |
| Database failover      | 1 to 5 minutes     | Seconds        | CloudNativePG or the managed database service promotes a healthy replica. |
| Cluster failover       | 1 to 10 minutes    | 1 to 5 minutes | Traffic shifts to a healthy cluster after health checks fail.             |
| Full re-index required | 5 to 60 minutes    | Not applicable | Timing depends on chain size, RPC throughput, and Ledger Index backlog.   |

### Setup and maintenance [#setup-and-maintenance-1]

| Task                                | Time estimate  | Client role             |
| ----------------------------------- | -------------- | ----------------------- |
| Two-cluster provisioning            | 1 day          | Platform engineer       |
| CloudNativePG setup across clusters | 1 day          | Platform engineer       |
| Ledger Index setup                  | 1 to 2 days    | Platform engineer       |
| Global traffic management           | 4 to 8 hours   | Platform engineer       |
| Initial setup                       | 1.5 to 2 weeks | 1 to 2 client engineers |

| Activity                          | Frequency | Time per cycle |
| --------------------------------- | --------- | -------------- |
| Replication-lag monitoring        | Daily     | 15 minutes     |
| Ledger Index sync verification    | Daily     | 15 minutes     |
| DR drill or failover test         | Quarterly | 4 to 8 hours   |
| Security patching across clusters | Monthly   | 4 to 8 hours   |
| Monthly effort                    |           | 20 to 30 hours |

Plan for 0.5 to 1 FTE platform engineer. The model is lighter than consortium hot-hot because the public chain owns consensus. Operators still need clear ownership for RPC health, indexing lag, database promotion, and route switchover.

## Operating checks before production [#operating-checks-before-production]

Before you run DALP in hot-hot mode, confirm each of the following:

* Traffic management can remove a failed region without sending users to a partially healthy DALP stack.
* PostgreSQL promotion, backup restore, and point-in-time recovery are tested for the chosen managed or CloudNativePG topology.
* Ledger Index sync, handler errors, and backfill progress are monitored for every active public-network region.
* DR drills cover application, database, chain/RPC, and user-facing route checks.
* One incident owner can decide when to drain a region, promote a database, or rebuild indexed state.

Use [observability](/docs/architects/operability/observability) for Ledger Index and runtime alerting, and [backup and recovery](/docs/architects/self-hosting/high-availability/backup-recovery) for restore-test evidence.