Skip to main content

Platform API not ready

A Platform API, indexer, or Workflow Engine pod runs but is not Ready: how to read the pod's own diagnosis, which waits are normal, and when a restart helps.

Kubernetes routes traffic only to pods that report Ready. A pod that is Running but not Ready is alive and deliberately out of rotation: its Service sends it nothing, so Console screens and API calls fail while kubectl get pods looks healthy. This runbook covers that state on the three backend roles of a DALP instance: the Platform API, the Ledger Data indexer, and the Workflow Engine. The Workflow Engine carries multi-step operations across crashes and restarts instead of running them as one call that has to finish in one pass. It also covers the readiness answer of the scheduler, the role that runs background jobs.

Read RESTARTS first, then the /readyz body of a Platform API pod. A climbing restart count is a boot failure, draining is routine, starting means the pod is still building, and degraded names the verdict holding the pod out. Do not restart a pod for a postgres verdict.

kubectl get pods does not print those role names. The chart labels each role's Deployment with app.kubernetes.io/component, so find the pod a check names by that label:

  • Platform API: kubectl get pods -l 'app.kubernetes.io/component in (api-v1,api-v2)'
  • Indexer: kubectl get pods -l 'app.kubernetes.io/component in (indexer,indexer-rebuild)'
  • Workflow Engine: kubectl get pods -l app.kubernetes.io/component=runner
  • Scheduler: kubectl get pods -l app.kubernetes.io/component=scheduler

The pod publishes its own diagnosis, but the contract depends on its role. Platform API pods report a boot phase or dependency verdicts on /readyz. A Workflow Engine pod reports only ok, or draining while it shuts down, so for it the restart count and the logs are the signals. An indexer pod answers /healthz and /readyz with 200 while its process runs, and its chain state is not a readiness verdict. A chain it cannot retry is parked: it logs EVM span chain parked; its siblings continue and stops while the pod stays Ready. Ledger indexing stalled owns that diagnosis.

The scheduler, the fourth backend role, runs background jobs and has no runbook of its own. Its /readyz answers like the Workflow Engine's, plus two bank rail cases. A scheduler pod answers ok, or draining while it shuts down. With the bank rail in simulator mode, it answers 503 with bank-fix-simulator-unavailable when the bank FIX simulator could not listen on bankFix.simulator.listenHost and listenPort. The reason names the cause. Free the port or correct the setting, then restart the pod, because the platform does not retry the listen. With the bank rail in external mode, one scheduler pod dials the bank at a time, and it adds a bankFix object: sessions counts the active connections it dials, loggedOn counts those whose Logon the bank has answered, and status is logged-on when the two match and not-logged-on otherwise. A session that is not logged on keeps the pod Ready, because the scheduler keeps dialling on its own, and a restart fixes neither a bank outage nor a refused logon. Read the scheduler log for The bank FIX session to the bank ended; reconnecting. and its reason, as the bank rail guide describes.

Symptoms

  • kubectl get pods shows a Platform API, indexer, Workflow Engine or scheduler pod Running but not 1/1 Ready.
  • A Helm install or upgrade hangs waiting for the rollout, then fails with a progress deadline error.
  • Console or API requests return connection errors or 503 responses while every pod shows Running.
Four checks route a not-Ready pod to its one real cause.A decision tree with four questions. First, is the restart count climbing? Yes means a boot failure: read the previous logs, because this is not a readiness problem. No leads to the second question: does /readyz report draining? Yes means an old pod leaving during a rollout, which is routine and needs no action. No leads to the third question: does /readyz report starting? Yes means the pod is still booting, and the reason field names the phase: building the graph gets up to 10 minutes, and a build past that limit exits the pod, so its restart count climbs. No leads to the final question: which verdict is unhealthy? Postgres means fix the database or the pooler in front of it. Durable means the work-scheduling cluster is down or shutting down. If postgres and indexer both read healthy, capture logs and restart the pod. Indexer means follow the indexing runbook, or on a Platform API pod, check the indexer pod it is waiting for.Four checks route a not-Ready pod to its one real cause.The same decision tree as the desktop variant, drawn portrait. Is the restart count climbing? Yes: a boot failure, read the previous logs. Does /readyz report draining? Yes: an old pod leaving during a rollout, no action. Does /readyz report starting? Yes: still booting, with up to 10 minutes for the graph build, after which the pod exits and its restart count climbs. Otherwise, which verdict is unhealthy: postgres means fix the database or the pooler, durable means the work-scheduling cluster is down or shutting down and a healthy postgres and indexer route to a restart, indexer means follow the indexing runbook or check the indexer pod.

Four checks sort a Platform API pod into a boot failure, a drain, a start in progress or a degraded verdict, and only one path ends in a manual restart.

Checks

  1. Rule out a crash loop first. Signal: the RESTARTS column of kubectl get pods. Healthy value: a count that stays put. A lightweight boot shell binds the HTTP port at start and answers the startup probe within milliseconds, so the startup probe does not end a slow boot. Instead the Platform API process allows up to 10 minutes to build its application graph, and a build past that limit exits so the failure becomes a visible restart. A climbing count therefore means a boot failure: read kubectl logs --previous <pod> for the exit, and stop investigating readiness until the pod stops crash looping.

  2. Read the pod's own answer. Signal: the JSON body of /readyz on the pod's HTTP port, reached with kubectl port-forward or a curl from inside the cluster. For a Platform API pod, the body routes you to the remaining checks: draining is check 3, starting with a reason field is check 4, and degraded with per-dependency fields is check 5. A Workflow Engine pod returns only ok or draining. An indexer pod returns 200 while the process runs, so read its logs for a parked chain. A scheduler pod returns ok, draining or bank-fix-simulator-unavailable. In external bank rail mode the pod that dials the bank adds a bankFix object to its ok. Only bank-fix-simulator-unavailable needs action on the pod. A not-logged-on status sends you to the reconnect warning in the scheduler log.

  3. Decide whether the pod is arriving or leaving. Signal: the pod's age next to a draining status. A pod that is shutting down flips not-Ready on purpose and skips its dependency checks rather than racing a closing connection pool. Every rolling restart produces this state briefly. Platform API rollouts start one new pod before taking any old one away by default, so the old replicas keep serving while the new one boots. Indexer rollouts replace one pod at a time instead, holding each replacement 90 seconds past Ready before moving to the next. Workflow Engine rollouts stop every old pod before the new ones start, so durable work pauses for the length of the boot. Healthy value: at most one Platform API or indexer replica in transition at a time. Investigate when rollout progress stops or capacity falls below the deployment's needs. Losing every Ready Platform API replica is a complete serving outage, so treat it as an incident and work checks 1, 2 and 4 on the newest pod.

  4. Place a starting pod in its boot phase. Signal: the reason field beside status: "starting". One phase appears: application-building, the graph build, bounded at 10 minutes. If it wedges, the pod exits, which sends you back to check 1.

  5. Name the verdict holding a running Platform API pod out. Signal: the postgres, durable and indexer fields in a degraded body, with their reasons under reasons.

    • postgres. The probe dials a dedicated client against a PgBouncer alias reserved for it, an isolated one-connection pool that application traffic never shares, so a saturated app pool cannot make a healthy database look down. A failing verdict names a real problem in Postgres or in the pooler in front of it. The pooler has a shorter rollout deadline than this pod, so a stuck pooler fails first and shows up here instead of looking like a Platform API fault. Check the pooler alongside the database.
    • durable. This is the work-scheduling cluster the Workflow Engine runs on. It lives inside the same pod and stores its state in the same Postgres, so there is no separate service to check. An unhealthy verdict, with the reason runner_shutting_down, means the cluster is shutting down or has not come up.
    • indexer. The pod needs a servable schema, the current version of the indexed database tables, which the indexer publishes. serving_schema_unbound means the pod waits, with no deadline, for the indexer to publish one, including on a fresh install. bump_window_blocked means a release's data rebuild holds the bind. Read the indexer pod's logs before you treat this pod as stuck.

Actions

  • If the restart count is climbing, fix the boot failure from kubectl logs --previous first. Readiness is downstream of a boot that completes.
  • If the postgres verdict is failing, check both the database and the pooler in front of it, and do not restart the pod. A restart only forces the reconnect the pod makes on its own, and the pod rejoins the Service once the database and pooler both answer. The default probe settings pull a pod from rotation after about 45 seconds of sustained readiness failure, which is three failed probes 15 seconds apart, and return it after one successful probe, up to 15 seconds later.
  • If indexing has stopped on a chain, search the indexer logs for EVM span chain parked; its siblings continue and follow Ledger indexing stalled. Restarting or resizing the Platform API deployment changes nothing about indexing.
  • If a pod reports serving_schema_unbound under reasons.indexer (check 5) with no schema ever appearing, the wait itself is not the problem: diagnose the indexer pod, whose background schema work every Platform API pod waits on. Producing that schema, on a fresh install too, is work the indexer does inside its own process, not a separate Kubernetes Job, so kubectl get jobs finds nothing for it. Read the indexer pods' restart counts and logs. Restarting the waiting pod changes nothing, because the wait resumes on the next boot.
  • If the durable verdict is failing while postgres is healthy, and the pod still sits not-Ready, capture kubectl describe pod and its logs first, then delete the pod. A durable failure outside a drain you already expect (check 3) means the wedge case applies, and the wedged state does not survive the restart, so the evidence has to come first.

Verify recovery

kubectl get pods shows the pod 1/1 Ready, its /readyz returns status ok, and Console and API calls succeed.

Escalate

If the same pod wedges again after a restart, escalate to your DALP support contact with the logs, the describe output and the failing verdict. A wedge that repeats points to a platform defect.

Frequently asked questions

Why does Kubernetes leave a not-Ready pod running instead of restarting it?

Readiness and liveness ask different questions. Readiness asks whether traffic should route here now, and its failures are recoverable states that clear on their own, so the right response is to wait out of rotation. Liveness asks whether the process is wedged beyond recovery, where a restart is the only remedy. Combining them would restart every pod whose only problem is a slow dependency, so a short database outage would cold-start the fleet. A Platform API pod's liveness fails only when its heartbeat goes stale for 30 seconds. The indexer uses the same heartbeat on its own liveness port, and the Workflow Engine pod answers liveness from its main loop. On both workers, four failed probes ten seconds apart call the pod wedged. No liveness probe reads a dependency, so a database or pooler outage does not by itself trip one.

Can I point the startup probe at the readiness endpoint to enforce the boot budget?

No. The indexer pod's /readyz is static, so it cannot enforce a boot budget. A Platform API pod already enforces its own 10-minute graph build, which is why the shipped startup probe confirms the port is up instead of using dependency readiness.

Why did my Helm upgrade fail on a deadline while every pod showed Running?

The rollout counts Ready pods. The chart marks a rollout failed after 25 minutes without progress by default, so a pod that never reports Ready fails the upgrade. What that costs you while you diagnose depends on the role. A Platform API rollout starts the new pod before removing any old one, so the old replicas keep serving unaffected and you can read the new pod's /readyz without time pressure. An indexer rollout replaces pods one at a time, so a pod that never becomes Ready has already taken the instance down a replica, and a Workflow Engine rollout stops every old pod first. Diagnose it promptly. Either way, a starting phase or a failing verdict in the new pod's /readyz names the cause directly before you decide whether to roll back.

Is it safe to poll the readiness endpoint myself?

Yes. The kubelet polls the same endpoint. It answers read-only, from in-memory state plus a dedicated Postgres session on an isolated PgBouncer pool, held open across probes rather than opened per call. That session gets 8 seconds to connect and query, covering a fresh reconnect after a dropped connection as well as an ordinary check. The chart pairs it with a 10-second probe timeout and a 15-second period, so a normal kubelet probe has time to finish before its next scheduled check. Avoid rapid additional polling during an incident. The body names each verdict and its reason, which makes it the fastest way to watch a recovery complete. It reports one pod's view only. For the rolled-up operator verdict across the whole instance, use Platform Health in the Console.

Was this page helpful?