Skip to main content
Every flow view in Celum — traffic topology, forensics, policy insights — is a query against ClickHouse on the supervisor, fed continuously by the flow pipeline. Both are components with their own status, and this page exists mostly for one lesson: when a flow view is empty, check freshness before components — the pipeline’s most deceptive failure mode is every component green while ingestion is stalled.

​
The ingestion path

Every dispatcher writes its own rows straight into ClickHouse; there is no central collector in between. The central flow-collector that used to poll conntrack from every node was retired in October 2026, and the cluster agent’s CT dispatcher took its place. Three pieces, three statuses: Guest clusters run their own collectors pointed at the same supervisor ClickHouse, so one database holds flows for the supervisor and every guest — each row tagged with its cluster name. The pipeline status also reports per-table row counts (cumulative totals per output table) for flows_local, ct_flows_1m_local, tetragon_syscalls_1m_local, tls_sni_1m_local, tls_handshakes_1m_local, ext_ips_fqdn_local and gateway_requests_local. A disabled dispatcher reads as a hard zero (e.g. tetragon_syscalls_1m_local: 0 with the Tetragon dispatcher off) — expected, not a fault. A zero on a table whose dispatcher is enabled is the fault.
The gateway access log matters on a supervisor whose North-South gateway is Envoy Gateway. Hubble receives L7 events only from a proxy Cilium owns, and Envoy Gateway’s data plane is a separate Envoy. Without this dispatcher a project behind the shared gateway shows no external client in Ingress Forensics. Envoy Gateway streams its access log to the cluster agent on each node, which writes gateway_requests_local. It has no effect on a supervisor that runs Cilium’s own Gateway API, which already reports L7. Turn it on in Onboarding → Flow Telemetry.
DNS enrichment is part of the pipeline: a cluster-wide DNS visibility policy feeds the ext_ips_fqdn_local table, which maps external IPs to hostnames in flow views. The pipeline status reports whether that policy is present.

​
Freshness — the first check

The freshness endpoint reports, per cluster writing into ClickHouse, the timestamp of the most recent flow row and its age: A cluster goes stale when its newest row is older than the threshold (staleAfterSeconds, 900 by default). Single-digit ages are normal on an active cluster; ages climbing toward the threshold mean ingestion for that cluster has stopped.
Empty dashboards, green components is a real incident class, not a hypothetical: every HelmRelease ready, every pod running — and no data, because the collectors could not reach or resolve the ClickHouse write endpoint. Component status cannot see that; only freshness can. If a flow view is empty, read freshness first and only then start on components.

​
Triage order

1

Freshness

Which clusters are stale? All of them points at ClickHouse or the write path; one guest points at that guest’s collectors or its network path to the supervisor.
2

Ingestion rates

In the pipeline status, check the per-table row counts. A zero on an enabled dispatcher’s table narrows the fault to that telemetry kind; for is it ingesting right now, freshness (above) is the authoritative signal, not the counts.
3

Pipeline runtime

Collector DaemonSet ready count, enabled dispatchers, and the DSN host. A DSN the collectors cannot resolve or connect to stalls everything while every release stays green.
4

Components

Only now the HelmReleases: operator ready, CHI Completed, schema applied with tablesPresent = tablesExpected. A schema mismatch after an upgrade blocks inserts into the missing tables.

​
Schema chart versioning

The table schema is versioned and shipped as its own chart, independent of the operator: bumping the schema chart rolls out new tables and views without touching ClickHouse itself. The status pins the exact applied version (e.g. clickhouse-schema@1.0.10) and the hash of the schema ConfigMap, so “which schema is this supervisor on” is always answerable. After a schema upgrade, tablesPresent vs tablesExpected confirms the migration completed.

​
Ingestion counters

The security flows status complements freshness with volume: rows written in the last minute and hour, and policy denials in the last hour — a quick sanity check that volumes look plausible for the environment. The Hubble dispatcher records only NEW and REPLY trace events, so a connection is counted at setup and at its reply rather than continuously.

​
Flow store passwords

Celum signs in to the supervisor’s ClickHouse as two users it manages, and gives each flow-telemetry guest that can have one a user of its own. Celum never shows their passwords, in the dashboard or in an API answer. Both Secrets in security carry the keys username and password.

​
In the secret store

When the supervisor has a usable secret store, the Flow Telemetry apply moves both passwords into it without changing them, under clusters/<supervisor>/security/. Usable means the Secret Store step is done and its ClusterSecretStore and canary ExternalSecret are Ready. External Secrets then writes the same two Secrets, with the same names and keys, so nothing that reads them changes.
  • A move deletes the Secret before External Secrets writes it again, and waits up to 20 seconds for that. Meanwhile Celum cannot read that supervisor’s flows. If External Secrets does not write it in time, Celum puts the original Secret back. The apply reports each move in ownCredentials, and a failed one among its warnings.
  • Once the store holds a password, Celum never generates a new one over it. An apply that finds the ExternalSecret but not yet its Secret waits for External Secrets.
  • Without a usable store, both passwords stay ordinary Secrets in the cluster, and Celum cannot rotate them.
  • Uninstalling ClickHouse deletes both Secrets and their ExternalSecrets, but not the store entries: Celum’s store credential can write but not delete.

​
Rotate a password

A rotation gives one of the two users a new random password. Celum writes it to the secret store, and External Secrets writes it into the user’s Secret. A rotation works only on a password the store already holds: until a Flow Telemetry apply has moved it there, the call is refused with 412 and changes nothing. The two users rotate differently, because ClickHouse takes one password for k8sgate_admin and several for flows.
Onboarding → Flow Telemetry → ClickHouse passwords, with a Rotate password button for each user. On a supervisor whose onboarding is complete, choose Change configuration on the onboarding summary and open the Flow Telemetry step, or open /s/<supervisor>/onboarding?step=flow-telemetry.Each rotation asks first and says what it will do. When a flows rotation is refused because something may still use the old password, the step lists each one, and Rotate anyway… asks again and names what stops writing. An unfinished flows rotation shows Finish rotation while the step stays open; after you leave it, rotate flows again to finish. Leaving the page does not stop a rotation, and its result also arrives as a notification. The buttons are disabled while the step is locked, while Flow Telemetry is being applied, and while another rotation runs.Onboarding exists only on Talos supervisors. On any other supervisor the dashboard cannot rotate these passwords: use the API or MCP. The section also needs access to Onboarding, which takes supervisor:Bootstrap or supervisor:Onboard.
A 412 names its cause in its row:
  • The Secret does not exist, or holds no password for the user: apply Flow Telemetry first.
  • The Secret is still an ordinary Secret, and the store is usable: apply Flow Telemetry, which moves it into the store, then call again.
  • The store’s ClusterSecretStore or canary is not Ready: repair the store, then call again.
  • The Secret is an ordinary Secret, and the supervisor has no usable store: Celum cannot rotate it. Change it by hand, as the row says:
    • k8sgate_admin: change the password in security/clickhouse-k8sgate-credentials, then apply Flow Telemetry, which writes its hash into the ClickHouseInstallation. If that apply cannot sign in to ClickHouse, apply it again a minute later, once ClickHouse has loaded the new password.
    • flows: change the password in security/clickhouse-flows-writer, then apply Flow Telemetry, which makes it the only password ClickHouse takes for flows. Then restart the cluster-agent DaemonSet and the celum-dns-honeypot Deployment in kube-system, and the cluster agent of every guest still on flows’ DSN.

​
Rotating the admin password

ClickHouse takes only one password for k8sgate_admin: the one whose hash the ClickHouseInstallation holds. The rotation therefore switches ClickHouse first and the store second.
1

Switch ClickHouse

Celum writes the new password’s hash into hubble-flows. ClickHouse loads it only after the ClickHouse operator has rewritten its users ConfigMap and the kubelet has refreshed the mounted copy, which happens about once a minute by default. Until then Celum signs in with the new password every 3 seconds, for up to 3 minutes.
2

Store the password

Once ClickHouse takes the new password, Celum writes it to the secret store, asks External Secrets to sync, and waits up to 30 seconds for security/clickhouse-k8sgate-credentials to carry it.
3

Reconnect

The backend drops its pooled admin connections, so the next one signs in with the new password.
From the moment ClickHouse takes the new password until External Secrets has written it into the Secret, the backend still holds the old one and cannot sign in as k8sgate_admin. That window lasts a few seconds, and the Flow Telemetry step’s status checks may fail during it. Anything outside Celum that signs in as k8sgate_admin with the old password stops working at once. A rotation usually takes a minute or two. If ClickHouse does not take the new password within 3 minutes, or the store or External Secrets fails, Celum writes the old hash back and waits for ClickHouse to take the old password again. When External Secrets was too slow, Celum first puts the old password back in the store, so a late sync cannot hand the Secret a password the rotation abandoned. The rows say what was put back, and what to do if putting it back failed too.
The rotation leaves failed sign-ins in ClickHouse’s logs. Each try in the first step that ClickHouse refuses (AUTHENTICATION_FAILED) is logged as a failed sign-in for k8sgate_admin, about one every 3 seconds until ClickHouse has loaded the new hash. In rotations measured on 2026-11-01 that was 27 tries in 83 seconds on a long-running supervisor, and a few seconds’ worth on a fresh one. They come from the rotation, not from an attack.

​
Rotating the flows password

A ClickHouse user created in SQL, as flows is, can hold several passwords at once. The rotation uses that to change flows without a moment in which a writer holds a password ClickHouse refuses.
1

Check for writers it would break

Before it changes anything, Celum looks for anything that may still use the current password:
  • a flow-telemetry guest without a writer of its own, or one that an older Celum’s namespace-wide DSN set cluster-agent-clickhouse-dsn still selects;
  • a KubeVirt guest whose pod or VM address inserted as flows in the last 15 minutes;
  • a workload that carries flows’ DSN as a plain value in its environment;
  • the Flow Telemetry DSN override, when it carries flows’ password.
For each one it finds, and each check it cannot make, the answer has a refused row that says how to settle it, and the call is refused with 409.
2

Add the new password

The new password is added beside the current one, which ClickHouse keeps accepting. Celum writes it to the secret store, and External Secrets writes it into security/clickhouse-flows-writer. If the store or External Secrets fails, Celum puts the old password back in the store and removes the new one from ClickHouse. If it cannot put the old one back, ClickHouse keeps taking both, and the row says to call again.
3

Restart the writers

Celum writes the new DSN into kube-system/cluster-agent-clickhouse, then restarts once each supervisor workload that reads that Secret: the cluster agent DaemonSet and the DNS honeypot Deployment. A restart loses about a minute of the supervisor’s aggregated flows. The backend moves to the new password on its own.
4

Retire the old password

Once every pod of those workloads runs on the new password, Celum retires the old one, and flows takes only the new one. It waits up to 5 minutes for that. After that the answer is pending, and the old password stays accepted.
In the rotations measured on 2026-11-01, the cluster agents and the DNS honeypot rolled out in 25 to 40 seconds, the same call retired the old password, and inserts never stopped. Finishing a pending rotation. Call again, or apply Flow Telemetry, which makes the Secret’s password the only one flows takes. While flows takes more than one password, a call finishes the rotation under way instead of starting another. It first has External Secrets refresh the Secret, and is refused with 409 if that does not happen within 30 seconds. Then it restarts only what is not on the new password yet, and retires the old one. Other writers it notices. A noted row lists every other address that inserted as flows in the same 15 minutes, by node or kube-system pod where Celum can name one. Celum’s backend is among them when its external-IP name worker ran, and it moves onto the new password by itself. Anything else listed there, other than the supervisor’s own cluster agent and DNS honeypot, holds the password too and stops writing when it is retired. The check does not refuse on these. Rotating anyway. force: true, or Rotate anyway… in the dashboard, rotates despite the refused rows. What they named keeps the old password and stops writing when it is retired. Guests on flows’ DSN get their DSN sets delivered again with the new password, which each guest’s agent takes at its next restart, and an older namespace-wide set is replaced by a set for each guest. Clear the DSN override, or put the new password in it, before you apply Flow Telemetry again. To finish a forced rotation that answered pending, call again with force.
Anything outside Celum that holds flows’ old password stops working when it is retired, such as an automation job configured with a copy of flows’ DSN, and the check cannot see it. Give such a job a ClickHouse user of its own before you rotate, or give it the new password from security/clickhouse-flows-writer afterwards.

​
While a rotation runs

  • Only one rotation runs on a supervisor at a time. A rotation, a Flow Telemetry apply and a guest-writer pass on the same supervisor refuse one another with 409, because each reads the passwords when it starts and writes them into ClickHouse minutes later.
  • Creating or editing a guest cluster with flow telemetry on that supervisor waits while a rotation changes a password.

​
Guests’ own passwords

A guest’s own password is rotated by another call: POST .../flow-pipeline/guest-writers with {"rotate": true}, for every guest, or for one named by namespace and cluster together. Through MCP, use apply_guest_flow_writers. The dashboard has no control for it. Each old password stays accepted until that guest’s agent runs on the new one, and a later call or Flow Telemetry apply retires it.

​
Permissions

Freshness and flow status are security:Read, not supervisor:GetSummary — they moved in August 2026 along with the rest of the flow surface. The two halves of this table are genuinely different grants: ClickHouse and the pipeline are installed and inspected as monitoring, while the data flowing through them is read as security. An operator who can install the pipeline cannot necessarily read what it collected.

Platform health overview

Why data freshness is the fourth layer of the triage model.

Monitoring

Metrics — the other telemetry pipeline, with its own store.

Logs

Loki and Alloy — same green-but-empty failure shape, different pipeline.

Networking overview

The Cilium datapath the flow telemetry observes.