Skip to main content

Utila signing agent not signing

Enrol, enable, rotate and recover the customer-run Utila signing agent, and diagnose transactions that wait for its signature.

A Utila transaction that DALP filed waits for the signing agent. If the agent is down, enrolled wrongly, or refused by DALP, the transaction stays in AWAITING_SIGNATURE and nothing reaches the chain. Read the status the agent last got from DALP in the status table under Checks: 200 with signTransaction true means it signs, 200 with signTransaction false cancels the transaction, and every other status leaves it pending. To enrol, rotate or roll back the agent, use the procedure sections after Escalate.

Utila is a third-party custody provider that holds the keys of wallets DALP uses. The agent is the Utila co-signer, run in your cluster. Deploy it with the Utila signing agent deployment page. The agent polls Utila for transactions designated to its service account. For each one it asks DALP whether to sign. Use a service account dedicated to DALP, because a refusal cancels the transaction at Utila.

What each answer from DALP does to the transaction at Utila.The agent asks DALP about a held transaction. Three outcomes branch from the question. An accent arrow for a 200 answer with sign true leads to a box that says the agent signs. A second 200 answer with sign false leads to a box that says Utila cancels the transaction. A dashed arrow for any other status, 401, 409, 413, 400 or 503, leads to a box that says the transaction stays pending and the agent asks again.What each answer from DALP does to the transaction at Utila.The same figure as the desktop variant, drawn portrait. The agent asks DALP about a held transaction and three outcomes branch from the question down a left lane. A 200 answer with sign true means the agent signs and the transaction goes on. A 200 answer with sign false means Utila cancels the transaction and it is final. A dashed branch for any other status, 401, 409, 413, 400 or 503, means the transaction stays pending and the agent asks again.

A 200 answer with signTransaction true leads to a signature, false cancels the transaction at Utila, and every other status leaves it pending.

Symptoms

  • Work that DALP designated to the Utila service account stays in AWAITING_SIGNATURE in the Utila console past the bound you accept.
  • A DALP approval or write through a Utila configuration does not finish, and its approval deadline passes.
  • The co-signer does not show as running under Vault Settings, API, Co-signers in the Utila console.
  • The agent log shows an error from the decision address on every poll.

Checks

  1. Read the pod. kubectl get pods for the label app.kubernetes.io/component=utila-signing-agent must show one pod 1/1 Running. Kubernetes has no health signal for the agent beyond that, because Utila documents no liveness endpoint. A running pod can still be stuck.
  2. Read the agent log for the status of its last decision request, and match it to the status table. The decision address is the URL the agent calls to ask DALP whether to sign, built from dapiInternalUrl and ending in the agent secret, so never paste it into a ticket.
  3. In the Utila console, open Vault Settings, API, Co-signers. The co-signer must show as running. If it does not, the agent is not connected: check the service account key, the network egress to Utila, and the state store.
  4. Check the transaction. It must list the service account in its designated signers. Utila sends only designated transactions to the co-signer, and transfers created in the Utila console are never sent.
  5. In DALP, open Settings, Custody, and read the Utila configuration. It must not be quarantined, and its “Signing agent secret” must be set.
Status the agent sees
Meaning
Action
200 with signTransaction true
DALP filed this request and it matches. The agent signs.
None.
200 with signTransaction false
The held request differs from the one DALP filed, in the field the body names. Utila cancels the transaction.
Do not retry. Compare the field, then file a new request.
401
The secret is wrong, the configuration is unknown, or the configuration is quarantined.
Compare the agent secret with the Console field. Check quarantine.
404
The dapiInternalUrl, the path, or the secret is wrong. A secret with URL-reserved characters (/, ?, #, %) or a trailing newline also gives 404.
Check dapiInternalUrl. Recreate the secret with openssl rand -hex 32.
409
DALP never filed this request through this configuration.
Find who created it. The service account must be dedicated to DALP.
413 or 400
The body is over 64 KiB, or it is not a transaction.
Check the CLI version and any proxy between the agent and DALP.
503
DALP cannot read its records or its secrets.
Check the DALP API health. The agent asks again.

Actions

  • If the pod is not running, read kubectl describe pod and the log. A pod stuck on its volume needs the state store. A pod that cannot reach Utila or the KMS needs an egress rule in utilaSigningAgent.networkPolicy.egress.
  • If the status is 401, fix the secret first. Generate a new secret with openssl rand -hex 32, store it in the cluster Secret, enter the same value in the “Signing agent secret” field of the Utila configuration, and restart the agent. If the configuration is quarantined, find out who quarantined it and why: DALP records the actor and the reason. Lift it only when that cause is fixed and its owner agrees, then select “Test connection” and “Recover execution” with a reason.
  • If the status is 503, fix DALP first. The agent needs no change.
  • If the pod is running and answers 200 but nothing is signed, check that the key shares cover the chain. Utila needs a key downloaded for every network you sign on, so run utila keys download again for the device against the agent's state location.
  • To stop new work while you investigate, turn off “New accounts” for the configuration. To stop all signing through it, select “Quarantine execution” with a reason. Quarantine stops DALP from filing and from answering for the configuration. It does not touch transactions already in Utila.
  • If designated work stays unsigned, DALP parks the approval until its deadline. After the deadline the request ends as expired and you file it again. You can cancel a transaction in the Utila console when you decide it must not sign. A cancelled transaction cannot be restored, so file the request again.

Verify recovery

The co-signer shows as running under Vault Settings, API, Co-signers, and a small test transaction that designates the service account gets a 200 with signTransaction true and signs.

Escalate

Escalate to your DALP support contact when the status is 200 and the pod is running but nothing signs after utila keys download, or when the agent log shows a status the table does not list. Include the configuration id, the transaction's Utila status, the matching status table row and the time window, with the agent log redacted as Redact before you share describes.

Enrol the agent

Do this once per environment, before you enable the provider.

  1. In Utila, create a service account with the Signer role and wait for an admin to approve the role. Use one account only for DALP.
  2. Choose the state location and encryption before you register the device. Utila saves the device key and signing keys there, and changing it later can break the agent's commands.
  3. Register the device with the Utila CLI. It prints a device id and a fingerprint. Record both.
  4. A vault admin approves the device in Vault Settings, Devices, and signs it in the Utila mobile app, and the admin quorum approves it. Record who approved and when.
  5. Download the key shares for every network you need to sign on.
  6. Keep the device id. The chart needs it as deviceId.

Set the agent secret

The secret binds one DALP provider configuration to one agent.

  1. Generate a secret with openssl rand -hex 32. DALP enforces a minimum of 32 characters when it saves the credential. The secret is a URL path segment, so use only letters, digits, ., _, ~ and -. Do not use openssl rand -base64, which can produce /.
  2. Store it in a cluster Secret with kubectl create secret generic <name> --from-literal=secret=<value>, not --from-file, which keeps a trailing newline. Name that Secret in utilaSigningAgent.signingAgentSecret.existingSecret.
  3. Enter the same value in the “Signing agent secret” field of the Utila configuration in Settings, Custody. The credential key is auth.signingAgentSecret.
  4. The decision address carries the secret, so never log it. The expanded argument is visible to anyone who can read the pod's process arguments, so restrict who can exec into the agent pod or read its spec. The address uses utilaSigningAgent.dapiInternalUrl, plain HTTP inside the cluster by default, so keep the chart's NetworkPolicy on, and use a service mesh with mutual TLS if your policy forbids secrets in cleartext on the pod network.

Each provider configuration has its own secret. A second configuration needs its own agent.

Enable the agent

Before you enable, record these values: how many Utila configurations exist, the key source, the state location, and who approved the device.

  1. Confirm that the organisation has a Utila provider configuration in Settings, Custody, with its “Signing agent secret” set, as Utila setup describes. Without that configuration the agent has nothing to answer for.
  2. Deploy the chart with utilaSigningAgent.enabled set to true. Confirm the pod is Running and shows under Co-signers in the Utila console.
  3. Select Test connection on the configuration. It must be healthy with a fresh probe time.
  4. Send one small test transaction. Confirm that Utila lists the service account as the designated signer, and that the agent log shows one decision request answered with 200.

For the first day, check that the configuration stays healthy, and that no designated work stays in AWAITING_SIGNATURE past your bound.

Roll back

Use the smallest action that stops the harm.

  1. Turn off “New accounts” for the configuration to stop new accounts. Existing accounts keep signing.
  2. Select Quarantine execution with a reason to stop all signing through the configuration. DALP records the action, the actor and the reason.
  3. Scale the agent to zero only after you quarantine. If you scale it down first, new writes wait until their deadline and then end as expired.
  4. Roll the image back by digest. The older pod resumes against the existing state.
  5. De-enrol the device in Utila as a separate step. Deleting the workload does not revoke the key share, so a live signing credential would stay behind.

To recover, fix the cause, select “Test connection” on the configuration, and select “Recover execution” with a reason.

Rotate the service account key

DALP and the agent use the same Utila service account, so rotation changes both sides in order. Keep the old key working until the last step.

  1. Add the new key to the service account in Utila as a standby.
  2. In DALP, put the new key in the “Standby private key (PEM)” field of the configuration.
  3. Switch DALP to the new key by replacing the “Service account private key (PEM)”.
  4. Switch the agent. Update the KMS key or the Secret behind UTILA_SA_PRIVATE_KEY, then restart the pod.
  5. Select Test connection and confirm it is healthy, and confirm that a test transaction signs with the new key on both sides.
  6. Only then remove the old key from the service account in Utila. Removing it earlier leaves you no working key if the new one fails, and adding another needs the vault admin quorum.

Recover from lost access

If Utila answers code 1002 after DALP mints a fresh token, Utila no longer accepts the key. Someone revoked it, or it expired. Switch to the standby key: move the standby into the active field in DALP and in the agent. Adding a new key to the service account needs the vault administrator quorum, so allow time for it.

State loss

The agent state holds the device key and the key shares. If you lose it, the agent cannot sign. Recovery is a new enrolment: register a new device, have the admin quorum approve it in the Utila mobile app, and download a new key share. Back up the state store. On a PVC, use volume snapshots. In object storage, turn on versioning.

Redact before you share

Remove these from tickets, logs and screenshots: vault ids, service account emails, wallet resource names, addresses, signature bytes, the agent secret and the decision address. A transaction hash is public and is safe to share.

Run it locally

Use a stub to check the invocation without a Utila vault.

  1. Build the image from Utila's distroless Dockerfile with the CLI binary from the Utila Resource Centre.
  2. Start a stub HTTP server that answers every POST with 200 and {"signTransaction":false}.
  3. Run the image with cosigner run, the vault id, the device id and the stub address as --webhook-url. Use a vault with no value, and never --sign-all-txs against a vault that holds funds.
  4. Create one transaction that designates the service account. Confirm the stub receives the transaction body and that Utila cancels it. Change the stub to answer 503 and confirm the transaction stays pending.

Alert bound for unsigned work

Utila documents that an unreachable endpoint leaves a transaction pending and that the co-signer asks again on the next poll. DALP publishes no measurement of how the agent behaves in a live vault when the decision route is unreachable, or of how many seconds it takes to notice unsigned work. Set your AWAITING_SIGNATURE alert bound from what you observe in your own vault during Enable the agent, step 4.

Was this page helpful?