Durable Execution Engine recovery

The Durable Execution Engine recovery routes sit behind the DALP operator API. They are for platform operators and trusted automation that already have system operate permission. They are not tenant-facing product routes.

Always run doctor first. It reads Durable Execution Engine ingress, admin API, deployment, service, and invocation health without changing workflow state. Use a write route only after the doctor response identifies the unhealthy component. Run doctor again after the write route to confirm the result.

DALP records each operator route attempt in the operations audit log. If a request includes a URL with username or password information, the audit row redacts that user information before storing the request arguments.

In prose, the recovery path is inspect, act once, then verify. force-redeploy handles service registration. cleanup-stale-deployments handles old deployment records. recover-stuck-workflow handles one failed workflow key after DALP verifies that retry is safe.

Choose the recovery action

Start from the doctor readout and change only the component that is unhealthy. This keeps the incident narrow and gives the next operator a clean audit trail.

Prerequisites

Confirm these facts before you call a recovery route:

Set the example environment variables:

export DALP_API_URL="https://platform.example.com"
export DALP_API_KEY="sm_dalp_operator_1234567890"

Quickstart: inspect and re-register a workflow service

This quickstart uses one coherent recovery case: doctor reports the live durable workflow service URL, you re-register that URL, then you check doctor again.

1. Run doctor

curl -sS -X POST "$DALP_API_URL/api/v2/admin/operator/restate/doctor" \
  -H "X-Api-Key: $DALP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

{
  "ingress": { "status": "ok", "latencyMs": 12 },
  "admin": { "status": "ok", "latencyMs": 9, "version": "1.4.0" },
  "deployments": {
    "status": "ok",
    "items": [
      {
        "id": "dp_01j8m7k2q3r4s5t6u7v8w9x0y1",
        "serviceUrl": "http://ddwf:9080/v1",
        "createdAt": "2026-05-09T10:00:00.000Z"
      }
    ],
    "error": null
  },
  "services": {
    "status": "ok",
    "items": [{ "name": "IdentityRecoveryWorkflow", "revision": 3 }],
    "error": null
  },
  "invocations": {
    "status": "ok",
    "byStatus": { "invoked": 2, "suspended": 1 },
    "recentFailures": [
      {
        "id": "inv_01j8m7p2q3r4s5t6u7v8w9x0z2",
        "serviceName": "IdentityRecoveryWorkflow",
        "serviceKey": "invitation_01j8m7k2q3r4s5t6u7v8w9x0y1",
        "failedAt": "2026-05-09T10:07:19.000Z",
        "errorMessage": "TerminalError: upstream provider returned a terminal failure"
      }
    ],
    "error": null
  }
}

2. Re-register the service URL

Use the serviceUrl value from deployments.items. Keep the string exact, including scheme, host, port, path, and trailing slash if present.

curl -sS -X POST "$DALP_API_URL/api/v2/admin/operator/restate/force-redeploy" \
  -H "X-Api-Key: $DALP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "serviceUrl": "http://ddwf:9080/v1",
    "force": true
  }'

{
  "acknowledged": true,
  "deploymentId": "dp_01j8m8b9c0d1e2f3g4h5j6k7m8"
}

3. Verify with doctor

curl -sS -X POST "$DALP_API_URL/api/v2/admin/operator/restate/doctor" \
  -H "X-Api-Key: $DALP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

{
  "ingress": { "status": "ok", "latencyMs": 10 },
  "admin": { "status": "ok", "latencyMs": 8, "version": "1.4.0" },
  "deployments": {
    "status": "ok",
    "items": [
      {
        "id": "dp_01j8m8b9c0d1e2f3g4h5j6k7m8",
        "serviceUrl": "http://ddwf:9080/v1",
        "createdAt": "2026-05-09T10:10:00.000Z"
      }
    ],
    "error": null
  },
  "services": {
    "status": "ok",
    "items": [{ "name": "IdentityRecoveryWorkflow", "revision": 3 }],
    "error": null
  },
  "invocations": {
    "status": "ok",
    "byStatus": { "invoked": 2, "suspended": 1 },
    "recentFailures": [
      {
        "id": "inv_01j8m7p2q3r4s5t6u7v8w9x0z2",
        "serviceName": "IdentityRecoveryWorkflow",
        "serviceKey": "invitation_01j8m7k2q3r4s5t6u7v8w9x0y1",
        "failedAt": "2026-05-09T10:07:19.000Z",
        "errorMessage": "TerminalError: upstream provider returned a terminal failure"
      }
    ],
    "error": null
  }
}

Redeploy changes service registration. It does not clear invocation failures. If recentFailures still names a blocked workflow key, inspect that workflow before closing the incident.

If doctor still shows old deployment records beside the deployment you want to keep, continue with stale deployment cleanup.

Recovery order

Do not skip the first doctor call. The write routes share the admin API dependency, and a broken admin path makes the write routes fail or block.

Doctor

doctor is read-only. It probes the Durable Execution Engine ingress URL, admin API, deployment list, service list, and invocation table. A failed component marks only that component as degraded or unreachable. The route fails only when DALP cannot resolve the admin URL before probing.

POST /api/v2/admin/operator/restate/doctor

Request body

Send an empty JSON object.

{}

Response fields

Force redeploy

force-redeploy registers the durable workflow service URL with the Durable Execution Engine admin API. It returns the deployment id for the registration. It does not drain or delete old deployments.

POST /api/v2/admin/operator/restate/force-redeploy

Request body

{
  "serviceUrl": "http://ddwf:9080/v1",
  "force": true
}

Success response

{
  "acknowledged": true,
  "deploymentId": "dp_01j8m8b9c0d1e2f3g4h5j6k7m8"
}

Operational notes

• Run doctor before this route so you can compare deployment state before and after registration.
• Run cleanup-stale-deployments after redeploy when doctor still shows old deployment rows.
• Do not treat acknowledged: true as stale deployment cleanup. It means registration completed.

Cleanup stale deployments

cleanup-stale-deployments keeps the deployment matching serviceUrl. DALP resolves that deployment id server-side, then drains and deletes every other registered deployment. A failure to list deployments returns an admin-unreachable error. Per-deployment drain or delete failures are logged by the cleanup helper and do not change the success response shape.

POST /api/v2/admin/operator/restate/cleanup-stale-deployments

Request body

{
  "serviceUrl": "http://ddwf:9080/v1",
  "forceDrain": false
}

Success response

{
  "acknowledged": true
}

Operational notes

• forceDrain: false is the normal production value. DALP drains stale deployments through the admin API, but it can still kill pinned invocations when a stale deployment is unreachable and cannot drain normally.
• forceDrain: true is for a known dead stale deployment where you accept forced drain behavior even when pending invocations remain.
• Run doctor after cleanup and inspect deployments.items before you close the incident.

Recover stuck workflow

recover-stuck-workflow prepares one workflow key for retry. DALP queries prior invocations for the supplied (serviceName, serviceKey) pair, refuses unsafe recovery states, kills and purges terminal-failed invocations, then clears keyed workflow state. The next workflow submission starts from a blank state.

POST /api/v2/admin/operator/restate/recover-stuck-workflow

Request body

{
  "serviceName": "IdentityRecoveryWorkflow",
  "serviceKey": "invitation_01j8m7k2q3r4s5t6u7v8w9x0y1"
}

Success response

{
  "acknowledged": true
}

Blocked recovery response

DALP refuses recovery when the workflow has an active invocation or already succeeded. The error includes a reason and the relevant invocation ids.

{
  "code": "RESTATE_WORKFLOW_RETRY_BLOCKED",
  "message": "Workflow retry is blocked by an active invocation",
  "data": {
    "reason": "active-invocation",
    "invocationIds": ["inv_01j8m7p2q3r4s5t6u7v8w9x0z2"]
  }
}

Retry blocked reasons

Error reference

DAPI error responses include the normal request identifier for support correlation. Keep that identifier with the incident record and the operations audit row.

Audit, security, and production boundaries

Related pages

• Durable Execution Engine operator API reference
• DAPI error reference
• Authorization architecture
• Blockchain monitoring operations
• Transaction tracking operations