DALP exports metrics, logs, and sampled traces over the OpenTelemetry Protocol (OTLP). Configure each signal's endpoint to use your collector or the bundled observability stack. The admin debug bundle reads stored logs and traces through a separate query path. Alerts that show a recovery risk lists the alerts that warn when a part of recovery stops working.
Every deployment declares a stable dalpInstance slug and deploymentEnvironment value, stamped on every signal it exports, so a central view can group a whole fleet by instance identity. Each Platform API role process owns one observability instance, with a separate exporter for each signal, so replica identity remains a reliable grouping key.
What exports by default
All three default to the same in-cluster Alloy OTLP receiver at obs-, which only the separate observability chart provides. Without that release or your own endpoint, nothing receives the data. Point these at a different collector by overriding each signal's otlp.endpoint key individually. There is no single global override for all three. This exporter configuration is a separate concern from the admin debug bundle below: the bundle reads telemetry back out of a VictoriaLogs and Tempo query API, while the keys above control what the Platform API pushes in.
Role processes push OTLP traces, metrics and logs to the in-cluster Alloy receiver, while the admin debug bundle reads back on request, for a platform administrator's export only.
Point the exporters at the bundled in-cluster collector when you want zero-configuration observability and are comfortable keeping telemetry inside the cluster. Point them at an external OTLP endpoint instead when a compliance requirement keeps observability data outside the deployment's own cluster, or when you already run a fleet-wide observability backend and want this instance folded into it rather than siloed. Overriding the three endpoints independently is supported. A deployment can send traces to one backend and logs to another.
In the bundled stack's telemetry-forwarding mode, logs pass through an attribute allow list and text redaction before they leave, and metrics and traces are not redacted, as Hosting and access describes. Treat every signal as data that can contain personal data. Telemetry is for operations: the platform's audit record is the audit trail in Security and compliance posture.
During shutdown, the process stops accepting new work and makes bounded attempts to release role-owned resources and flush its telemetry exporters before forced exit. The flush runs last so it can report the shutdown itself. The flush is one bounded delivery attempt and gives no delivery guarantee, so the terminal log record also carries any final condition.
Which signal answers which operational question
The three signals answer different questions.
A webhook delivery attempt that times out, or a Console request that hangs waiting on a database query, are both trace questions.
Because metrics export every 60 seconds (telemetry.) by default, a metrics dashboard shows data with up to one export interval of delay before ingestion. Treat a metric reading during an active incident as slightly stale, and confirm a fast-moving condition against a trace or a direct check instead.
A log carries identifiers, such as a request id or a trace id, that let you pivot from a log line back to the trace or metric around it.
Kubernetes health probes (/healthz and /readyz) are separate from this OTLP pipeline: fast checks the cluster itself polls to decide whether to route traffic to a pod or restart it. They are not exported anywhere and do not appear in VictoriaLogs or Tempo. See Platform API not ready for what each probe checks and what a failure means. In external mode the scheduler's /readyz also reports the bank FIX sessions it dials, and a session that is not logged on does not take the pod out of Ready, as the bank rail guide describes.
The sampling trade-off
The default 10% trace sample ratio is a head sampler: the platform decides whether to keep a given trace before that request finishes, with no knowledge yet of whether it fails or runs slow. Raising the ratio towards 100% keeps every trace, including the large majority that succeed quickly and tell you nothing new, and ingestion and storage cost rises in direct proportion to the ratio you set.
A downstream tail sampler can keep traces according to their outcome or latency. It can only inspect traces that pass the source sampler. To evaluate every trace, raise source sampling to 100% and size the collector for that incoming load before applying selective retention. Measure exporter, collector, network, and storage costs when changing the ratio.
Retention is not controlled from this page
How long a trace, metric, or log line survives in your backend is a property of that backend's own configuration, not of anything in the table above. If your deployment uses the bundled in-cluster stack, its defaults keep logs and traces for 168 hours and metrics for one month. Observability stack covers the values that raise them. An investigation that starts days after an incident can fail because retention has expired. Raise retention if you need a longer window.
Alerts that show a recovery risk
Alerting is off by default. Set grafana. in the observability chart to turn these alerts on.
PostgresDown and ChainUnowned carry critical severity and the other recovery alerts carry warning, so confirm that your routing reaches someone who can act on each. Backup and disaster recovery explains the objectives they protect.
The three Velero alerts read metrics from the Velero server, so they stay silent when no Velero server runs or its metrics are not scraped. PostgresDown reads the PostgreSQL exporter, so it stays silent if you set postgres- to false. Postgres reads replication lag metrics that an exporter reports, either from the primary's replication statistics or from a standby that has its own exporter. It stays silent for a single unreplicated instance and for a managed service whose lag the exporter cannot see.
The admin debug bundle
Console's account menu carries an “Export debug information” item, visible only to a platform administrator or an organisation owner, the same gate the Platform Status page enforces. Selecting it calls a Platform API route that collects the bundle for the window into a zip and downloads it, without giving the requester direct access to the observability cluster. The bundle's recent errors and API requests categories hold only requests made in the requester's active organisation.
Only a platform administrator's export includes the log lines and traces the stack holds for the window, because those carry every organisation's activity on the instance. An organisation owner's export does not query VictoriaLogs or Tempo. Its manifest marks the observability logs and observability traces categories as unavailable, with a note that only a platform administrator's export includes them. The bundle the Console downloads when one of these users stops a recording started from Report an issue follows the same rules.
Set global. when your deployment includes the bundled observability stack, and the debug bundle's in-cluster VictoriaLogs and Tempo query endpoints are derived from that stack's release service DNS, so you set no URL. global. (default monitoring) controls the namespace those derived endpoints target, and has no effect on where the traces, metrics, and logs exporters above send signals: those always target the fixed obs- host unless you override each otlp.endpoint key yourself.
Two settings stay explicit regardless of auto-wiring:
dapi./observability Logs Url dapi.override the derived endpoints, for an out-of-cluster stack or renamed services.observability Tempo Url dapi.sets the Grafana deep-link base, an external route host never auto-derived from in-cluster service DNS.observability Base Url
Alert on how long a dependency is unavailable
A gauge that reports whether an input is connected reads the same for a two-second blip and a two-hour outage. Both show the disconnected value, and only the second is an operator's problem. Alert on how long the input has been unavailable.
The platform's own signals follow this pattern. dalp. tracks how long the chain's newHeads WebSocket has been continuously down. dalp_ tracks the same for a Redis client. Both sit at 0 while connected. Both escalate their log line from WARN to ERROR once the outage passes 30 seconds. The log threshold is separate from an alert's evaluation and hold periods. The bundled sustained-outage alerts take about six minutes to fire. Tune them for your operating requirements. Reconnection can still recover without operator action after either threshold.
If you build your own alert on such a gauge, measure from when the outage started, not from the latest reconnect attempt: a duration that resets on every attempt stays near zero for the outage that does not resolve itself.
The dashboards and alert rules the bundled stack ships with, and its default retention windows, live with the stack itself: see Observability stack. The receiving stack's content and storage policy are not covered here.
The database metrics exporter
The PostgreSQL exporter ships enabled by default: it opens one direct session to your PostgreSQL server and reports the aggregate server backend count, on top of the metrics its own upstream collectors define. The platform adds no custom queries of its own. Set postgres- to false to remove that session.
If you extend it, do not give a custom query a metric name a built-in collector already publishes. A duplicated name makes the exporter reject the whole scrape and serve nothing. The pod stays ready while that happens, because readiness does not read /metrics, so watch up and pg_up for the gap rather than the pod status.
Treat the exporter's own metric names as a fixed contract, and prefer widening what it already collects over declaring a metric again.
Neither probe queries the database. Both hit /, the exporter's static landing page, with a 5-second timeout: readiness every 15 seconds, liveness every 30 seconds. Only the Prometheus scrape of /metrics runs the collection, so the database pays for one collection per scrape interval and for nothing else. A readiness probe on /metrics would double that cost and would also mark the pod unready when the database is slow, which drops the scrape that carries pg_up and the wait metrics exactly when they answer the question.
The per-table collectors, stat_user_tables and statio_user_tables, ship off. stat_user_tables calls pg_indexes_size and pg_table_size once per user table, so one scrape stats every fork and segment of every relation and index in the database. On a schema with about a thousand tables that is seconds of server time per scrape and tens of thousands of series. Set postgres- to true on a small schema where per-table vacuum, bloat and scan counts are worth that price.
Frequently asked questions
What happens if I set global.observability.enabled to true but never install the observability stack?
The debug bundle still derives the in-cluster VictoriaLogs and Tempo URLs from that flag, whether or not the stack is running. Each query carries its own five-second timeout, and an unreachable or erroring endpoint is reported as an unavailable artifact rather than failing the whole export, so the rest of the bundle still lands. The exporters above are unaffected either way, since they send OTLP to obs-alloy independently of this flag.
Can I disable trace export while keeping metrics and logs exporting?
Yes. Traces, metrics, and logs each carry their own enabled key under telemetry, defaulting to true, so turning one off leaves the other two untouched. No single switch disables all three at once: disabling logs, for instance, does nothing to the trace or metric exporters, which keep running on their own schedules and their own endpoints.
Do the three signals authenticate to their collector the same way?
Not necessarily. Each signal's OTLP block carries its own headers key alongside its endpoint, so traces, metrics, and logs can each present a different bearer token or API key, or none at all, to whatever collector they target. That matters most once you split signals across backends: a managed metrics vendor and an in-cluster log store rarely share one credential.
If I set dapi.observabilityLogsUrl explicitly, do I still need global.observability.enabled: true?
No. The debug bundle reads that value directly and uses it whenever it is a non-empty, valid HTTP or HTTPS URL, independent of the flag. global. only controls whether the chart derives a default for you. An explicit URL is used as given, whether the flag is on, off, or the bundled stack was never installed at all.
Can the debug bundle's incident window run past my backend's retention?
Yes, if your backend retains less history than the requested window. The bundle's window defaults to the six hours before the request and is capped at 24 hours of lookback regardless of what a caller asks for, well inside the bundled stack's 168-hour default for logs and traces. Retention only becomes a real risk for a direct backend query days or weeks later, which is a different question from this bundle's own short window.