The observability chart is the receiving side of the platform's telemetry. A DALP instance exports traces, metrics, and logs over OTLP (the OpenTelemetry Protocol) whether or not anything is listening, as Observability describes.
This chart is one Helm release that receives that export, routes each signal to its own store and shows all three in Grafana, with the DALP dashboards provisioned. Install it when you want the deployment observable without assembling a stack yourself. Skip it when a fleet-wide backend already exists: in that case point each signal's export endpoint at your collector instead, and nothing on this page applies.
Installing it changes two things beyond the new pods. First, the platform's zero-configuration path starts working: the dalp chart's export defaults already target this stack's collector in the monitoring namespace, so a platform installed after this chart ships telemetry from its first pod onward with no export values set. Second, you commit to a telemetry identity. The chart refuses to render until you name the deployment, and that name is stamped on every signal the stack handles, so choosing it is the decision here that is hardest to change later.
You can keep part of the stack. Every component sits behind its own enable flag, so a deployment that already runs a fleet-wide log store can disable the bundled one and keep the rest. Disabling a component is not enough on its own: Alloy and Grafana still point at the bundled service names, so keeping your own store also means repointing the matching alloy. endpoint and the corresponding Grafana datasource at it.
Services export OTLP to Alloy, which routes logs, traces and metrics to VictoriaLogs, Tempo and VictoriaMetrics, and Grafana reads all three.
Steps
Name the deployment's telemetry identity
Set
global.dalpInstanceto a stable slug for this deployment: 1 to 63 characters, lowercase letters, digits, and hyphens, starting and ending with a letter or digit. Setglobal.to one ofdeployment Environment local,development,test,staging,demo, orproduction. Both are required and validated at render time, and neither has a default.The pair is stamped on every metric, log, trace, and profile the stack handles, and two deployments sharing one slug merge silently into a single identity in every fleet-level dashboard and alert group, while every signal still flows. Pick a slug you can keep, because changing it later splits the deployment's history into two identities.
The render also refuses the slug
localin any environment other thanlocal, which catches the common copy-paste of a local values file into a staging install.Write the values file
Start the values file with the identity pair, then configure credentials, routing, storage, and retention for your deployment:
global: dalpInstance: "example-primary" deploymentEnvironment: "production"On a restricted or multi-tenant cluster, scope Alloy before you install rather than after. Set
alloy.rbac.namespacesto the namespaces you want the ClusterRole to cover. That list limits pod discovery for stdout scraping. It does not restrict OTLP logs, which can arrive from any pod that can reach the receiver Service. The platform'sdapi,dapp, anddocscontainers export OTLP, so the stdout scrape excludes them. Setalloy.instead when the scrape needs a different scope than the RBAC namespaces.pod Discovery. namespaces Set storage, retention and credentials now, not during an incident. To keep a store you already run elsewhere, disable the bundled component and repoint Alloy and Grafana at your own. Each component has its own enable flag:
victoria-,logs- single. enabled tempo.enabled,victoria-,metrics- single. enabled grafana.enabled,alloy.enabled. Disabling one alone leaves Alloy writing to a Service that no longer exists.Set
alloy.,endpoints. internal. prometheus. url alloy.orendpoints. internal. otel. url alloy.to your store's address, and point the corresponding Grafana datasource at the same place.endpoints. internal. logs. victoria Logs Url To hold telemetry longer than the defaults, raise the retention values now:
victoria-andlogs- single. server. retention Period tempo.tempo.retentionboth default to seven days, andvictoria-to 1 month. A seven-day window cannot answer an investigation into older data, so size retention for the questions you must answer. This stack holds operational telemetry.metrics- single. server. retention Period The platform's audit trail lives in its database, which Backup and recovery covers.
Replace the Grafana administrator login before the release is reachable. The chart ships
grafana.andadmin User: settlemint grafana., which are public, and Grafana can read every log line. Supply unique values through your deployment's secret mechanism.admin Password: dalp Set
alloy.clusternameas well. Alloy stamps it ascluster_nameon every signal that lacks one, and falls back tosettlemintwhen it is empty.Install the release
helm install observability oci://pkg.distribution.settlemint.com/settlemint/observability \ --version <chart version> \ --namespace monitoring --create-namespace \ --values my-values.yamlThe chart fixes its service names whatever the release name:
obs-alloy,obs-logs,obs-tempo,obs-metricsandobs-grafana. The namespace is the placement choice that matters. Thedalpchart's traces, metrics, and logs exporters all default to the Alloy receiver inmonitoring, so installing there means the platform needs no export configuration at all. Installing anywhere else works, and costs two sets of overrides: each signal'sotlp.endpointkey on the platform release, andglobal.on the same release if you enable the debug bundle wiring in the next step.observability. namespace Install the receiving stack before the platform on a new deployment to avoid losing early telemetry. When upgrading both releases, check compatibility between the collector configuration and application exporters before deciding the order.
Connect the platform release
If the platform runs with default export settings and this stack sits in
monitoring, telemetry flows with no further change. One connection is opt-in: setglobal.on theobservability. enabled: true dalprelease and the admin debug bundle derives its VictoriaLogs and Tempo query endpoints from this stack's service DNS.A platform administrator's “Export debug information” action in the Console then pulls raw logs and traces without cluster access. An organisation owner's export leaves them out, because they carry every organisation's activity. The connection defaults to off because the platform cannot assume this chart exists. Explicit
dapi.andobservability Logs Url dapi.values override the derivation for a renamed or out-of-cluster stack.observability Tempo Url For export-side choices, the sample ratio, a different collector per signal, or an external backend, follow Observability.
The chart's default ingress host is a placeholder, and OpenShift Route and Gateway API HTTPRoute variants sit behind their own flags. The provisioned dashboards are grouped into folders named 00-start-here, services, cluster, telemetry, fleet, and local. The local folder ships in every environment and stays sparse outside one.
What the chart deploys
Grafana's three datasources come provisioned and cross-linked: a log line carries a link to its trace, and a trace links back to the logs in its time window, so an investigation moves between signals without hand-building queries.
Alert rules are a deliberate second step
Dashboards work as soon as the release installs. The alert rules ship in the chart but stay dormant until you set grafana., and enabling them forces a second choice: grafana. must be active or shadow. Exactly one Grafana in an estate should be active and deliver notifications. A shadow install evaluates every rule and shows its state, but an always-on mute route stops it paging anyone, which is how a second stack observes the same fleet without doubling every page.
Once enabled, the rule groups cover these areas:
Two families of absence rules fire on silence instead of a threshold: per-service log-ingest rules for streams that stop arriving, and a per-cluster heartbeat for a whole cluster that stops reporting. They exist because a broken telemetry path silences every log-based alert without any signal of its own.
Delivery goes to Slack in one of two modes. Webhook mode posts everything to one channel through grafana.. Bot mode routes critical and warning alerts to separate channels with runbook and dashboard links. It reads its token from the grafana- Secret, which the chart only renders from a chart value as a development convenience. Supply the token through your own secret machinery in a real environment, and never through a values file in version control.
Tune before trusting: the thresholds under dalp.observability are starting points. LowOperatorWalletBalance has no threshold value: it compares each operator wallet with the low-balance mark the platform publishes for each network, 0.1 whole units of the network's native currency whatever its decimal precision, the same mark the Console payer status uses. The chart refuses to render while native is still set.
webhooks. defaults to 500, and webhooks. defaults to 300000 milliseconds (5 minutes). These control the backlog and stalled-delivery alerts. transaction defaults to 900 seconds (15 minutes), the reporting window a regulator transaction report may stay queued before Transaction fires. Set it to your regulator's deadline. kyverno. defaults to 7 seconds, below the admission webhook's 10-second timeout.
webhooks. also carries a ceiling the others do not: the gauge it compares against saturates at 50000, so a threshold at or above that value could never fire, and the chart refuses to render one that high.
Two lists must also match your deployment. The log-ingest rules generate one rule per entry of dalp., whose default names the platform's own service containers: dapi (the Platform API), dapp (the Console), and docs (this documentation site). An install that does not deploy one of them must trim the entry, or that rule fires forever. The heartbeat rules generate one per entry of dalp. and default to none, because the chart cannot infer which clusters are supposed to report into it.
One stack for several clusters
The bundled stack is sized for one instance's telemetry in one cluster. To aggregate a fleet, run one central stack and put edge installs into forwarding mode. Set alloy. and alloy. to the central Alloy's endpoint, and every signal copies there while the local stores keep working, so an edge cluster stays diagnosable on its own even when the link is down. Signals a cluster does not need locally can be switched off individually under alloy.endpoints.internal.
The central side does not authenticate its inbound traffic by default: the standard OTLP receiver on 4317 (gRPC) and 4318 (HTTP) takes unauthenticated traffic from anywhere that can reach it, in-cluster or not. Setting alloy. and alloy. adds a second, authenticated OTLP receiver on ports 4320 (gRPC) and 4321 (HTTP) with basic auth.
The standard 4317 and 4318 stay open and unauthenticated alongside it, because enabling the auth receiver does not close them. Restrict ports 4317 and 4318 to the platform namespaces with a NetworkPolicy when the cluster is reachable from outside.
Both ends read their passwords from Secrets the chart deliberately does not create, observability- on the receiving side and observability- on the sending side. Create each Secret, holding a single password key, before the install that enables it.
Rotating either password needs an Alloy pod restart, because the container reads it from its environment, and the config reloader only watches configuration.
Troubleshooting
helm install fails at render time asking for identity values
Four distinct messages come from the same guard, and each names its own fix. A required error naming
global.dalpInstance or global. means the value is missing. Set it. global. means the slug is set but shaped wrong. It must be lowercase letters, digits, and
hyphens, starting and ending with a letter or digit. global. means the environment value is set but does not match one of those six; typing prod
instead of production is the common miss. A message naming the slug and the environment together, ending “reserved
for local installs”, means the slug local reached a non-local environment. Give the deployment its own stable slug
instead. Fix the value the message names rather than working around it: every one of these failures is a signal that
would otherwise surface as corrupted fleet dashboards weeks later.
helm install fails at render time complaining about webhooks.queueDepthThreshold
dalp. must be a whole number from 0 up to, but not including, 50000. The
gauge it compares against saturates at 50000, so a threshold at or above that cap could never fire. A non-numeric
value would otherwise coerce to 0 and page on the first pending row that lingers 15 minutes, so the chart fails the
render instead. Set the threshold to a whole number below 50000.
The render fails after I supplied my own Alloy extraEnv list
Helm replaces list values wholesale, so your alloy.alloy.extraEnv list dropped the chart's Secret-backed password
entries, RECEIVER_PASSWORD and FORWARDING_PASSWORD. The chart fails the render when receiver auth or forwarding
auth is enabled and its entry is missing, because catching the values mistake at deploy time beats discovering it as
authentication failures on a live receiver. Repeat the chart's entries inside your own list alongside your additions.
Grafana is up but the service dashboards are empty
The stack shows what reaches it. Before the dalp release lands, empty is correct. Once it is installed, check
placement and endpoints: the platform's export defaults target obs-alloy in monitoring, so a stack in a different
namespace receives nothing until each signal's export endpoint on the platform release is overridden to match. Cluster
dashboards are the control here. They draw on Alloy's own scrapes and fill in regardless of the platform. Cluster data
with no service data points at the export path, while no data at all points at the stack itself.
A log-ingest absence alert fires and never resolves
Two different causes generate this alert, and each needs a different fix. dalp.
lists the services the rule expects to hear from. A service the deployment never runs produces a stream that never
arrives, so trim the list to the services deployed. Separately, dalp. (falling
back to alloy., then alloy.rbac.namespaces) names the namespaces each service must report
from, and the rule emits one series per namespace, so it fires when a single namespace goes dark rather than only when
every namespace does. A namespace in that list that never runs the service fires forever regardless of the services
list. Do not confuse either key with alloy., which drops the stdout scrape for the same
container names and must never list a workload that does not export OTLP, the ledger node, proxies, and policy
controllers included. The fix in every case is a values change and an upgrade. Do not silence the alert.
Alerts are enabled but nothing reaches Slack
Check ownership first: a notificationOwner of shadow mutes all delivery by design, and only an active owner
pages. Then check the delivery mode: webhook mode needs grafana. set, and bot mode needs the
token present in the grafana- Secret. A rule that fires in Grafana with nothing delivered points
to one of these two configurations.
Frequently asked questions
What does changing the slug cost once the deployment has history?
The stamped global.dalpInstance value is how every earlier signal ties back to this deployment. Changing the slug starts a second identity. Existing history retains the old slug. New signals use the replacement. Treat the slug as fixed for the life of the deployment.
Can I install this chart after the platform chart instead of before?
Yes. The platform chart is dalp. A DALP instance exports whether or not anything is listening, so the platform drops telemetry until this chart's receiver exists and resumes once obs-alloy answers in the monitoring namespace. The platform release needs no restart. Installing this chart first avoids that gap, so on a new deployment install it first.
Do I need to enable alerting to get value from this chart?
No. Dashboards and their cross-linked datasources come up as soon as the release installs: cluster panels show data immediately, and service panels fill in once the dalp release starts exporting to it. Alerting is a separate step. It stays dormant until grafana. is set, and turning it on forces a second choice, active or shadow, so a working install and a paging install are separate decisions.
What happens to the 30Gi of stored telemetry when I uninstall?
It stays. VictoriaLogs, Tempo, and VictoriaMetrics each use a stateful workload with its own persistent volume claim. helm uninstall removes the release's pods and services but leaves those claims. Kubernetes retains their storage after the workload is removed. The three 10Gi claims stay bound and billable until you delete them with kubectl delete pvc. A reinstall with matching claim names can reuse the retained data. A complete teardown also requires a deliberate decision about those claims and their underlying volumes.