Skip to main content
Everything in the Security area is a query over one database, and this page is about the machinery that fills it. The Flow Collector tab shows its state. Two onboarding steps set it up, the ClickHouse section of Monitoring and Flow Telemetry, and both need monitoring:Install. The .../security/... routes on this page need security:Onboard to change something and security:Read to look. The read-only health model for this pipeline — components, schema versioning, ingestion counters — is documented in Flow telemetry. This page is the operator’s side: what to click, in what order, and what breaks if you skip a step.

​
Standing it up

1

Install the store

Onboarding → Monitoring → ClickHouse installs the Altinity operator into clickhouse-system and the ClickHouseInstallation hubble-flows into the security namespace (POST .../clickhouse/install, then .../clickhouse/chi/apply). The same step publishes ClickHouse through the Gateway API as clickhouse.<your domain>. POST .../security/onboard installs the operator and the installation from the ClickHouse card on the supervisor’s Monitoring tab instead, and streams its progress as server-sent events.
2

Check the components

GET .../security/status returns the two pieces and their state:
3

Turn on the pipeline

Onboarding → Flow Telemetry (POST .../flow-pipeline/configure) is the step that turns a database into a pipeline. It refuses with 412 until ClickHouse is published through the Gateway API, because every call the backend makes to ClickHouse goes over that route.
4

Confirm rows are arriving

GET .../security/flows/status — counts.rowsLastMinute climbing is the only proof that matters. See no data or no traffic.

​
What the Flow Telemetry Apply does

The Apply is idempotent and does considerably more than flip a flag, which is why re-running it is the standard repair for a half-configured pipeline. It:
  • turns the Hubble dispatcher off when Cilium’s Hubble is not enabled, because the agent would otherwise mount a socket that does not exist and fail to start;
  • ensures flows’ password in security/clickhouse-flows-writer, generated once and reused on every later Apply so the agents’ DSN stays the same. Where the secret store owns that Secret, the Apply waits for External Secrets to write it and never generates a password over it;
  • moves flows’ and k8sgate_admin’s passwords into the secret store, unchanged, when the supervisor has a usable one. See Flow store passwords;
  • applies the clickhouse-schema chart and runs its baseline against the installation;
  • writes k8sgate_admin’s hash into hubble-flows from security/clickhouse-k8sgate-credentials, and makes the Secret’s password the only one flows takes, which retires an old one a rotation left accepted;
  • re-applies the cluster agent with the dispatchers you picked, the cluster name stamped on every row, and the in-cluster DSN;
  • applies the cluster-wide DNS visibility policy. Without it Cilium’s DNS proxy stays transparent and never emits lookups, so external peers show as bare ext/<ip> forever while every other layer looks perfectly healthy;
  • points the shared gateway’s access log at the cluster agents when the Gateway access log dispatcher is on, and takes it away again when it is off;
  • applies the pipeline’s alert rule and the ClickHouse metrics ServiceMonitor, when kube-prometheus-stack is installed;
  • gives each flow-telemetry guest a ClickHouse writer of its own (see Guest clusters and vclusters).
The response reports what each stage did (schemaRelease, schemaBootstrap, clusterAgentRelease, dnsVisibility, gatewayAccessLog, alerts), each with its own warnings. It also has a row for each moved Secret in ownCredentials and for each guest in guestWriters, and repeats a row that needs attention in the top-level warnings. The Apply is refused with 409 while a ClickHouse password rotation runs on the supervisor. Warnings are not failures, but a dnsVisibility warning is the direct cause of nameless egress in FQDN egress.
There used to be a separate enable call, POST .../security/flows/enable, and a central flow-collector Deployment that polled conntrack from every node. Both were retired in October 2026: each node’s cluster agent now writes its own rows. An uninstall still removes a flow-collector Deployment an older supervisor may carry.

​
Trace reasons

The Hubble dispatcher keeps only NEW and REPLY trace events: a connection is counted at setup and at its reply, not continuously. That is the cluster agent’s default, and Celum offers no setting to change it.

​
Schema migrations on their own

POST .../security/flows/migrate runs the incremental schema migrations without the rest of the enable path. It serves exactly one case: a ClickHouse with no flow schema at all, on the legacy direct-connection path.
It will not add a missing view or column to an existing database, and it is not the fix after a schema chart upgrade.The migrations are a history that converges only from an empty database — one of them builds a view on a column a later one removes, and a later one drops flows_local outright. So the moment the migrator sees a flows_local, it stops without writing a byte and reports that the schema belongs to the chart baseline. Callers treat that as “nothing to do”, which is correct and also means the call you just made did nothing.The fix for a missing view or column is a schema chart bump, which re-applies the baseline. Every current cluster got its schema that way.
That guard exists because of what happened without it. The migrator used to reason “dirty ledger → reset and replay”, the first run against a chart-built schema always left the ledger dirty, and the second apply of the Flow Telemetry step therefore dropped a populated flows_local — 248 million rows on a live supervisor. The guard now refuses to touch a table that holds data, and the reset path refuses to drop one. A schema delta belongs in the chart’s baseline plus a chart version bump — never in a new migration file.

​
ClickHouse instances

You manage the ClickHouse store on Overview → Monitoring, in the ClickHouse Platform panel at the foot of the tab. It shows the operator (clickhouse-operator) and the flows instance (clickhouse-instance) as Ready, Starting or Not installed. Each instance is a ClickHouseInstallation in the security namespace. The flows instance is hubble-flows, created by onboarding. The Security pages read it and nothing else, so any other instance is yours: no Celum view reads it.
The panel does not report a failed install, create, edit, delete, exposure or uninstall. The dialog closes and the list reloads either way. If the list did not change, make the same call through the API, which returns the error.

​
Install ClickHouse

When the operator is not installed, the panel offers Install ClickHouse Stack: Storage Size (default 100Gi) and Storage Class (default: the class the supervisor marks as default). Install ClickHouse installs the operator as a Flux HelmRelease, then creates hubble-flows with one replica. Prefer the ClickHouse section of Onboarding → Monitoring, which is the install onboarding runs. Only that install gives hubble-flows its pod template, which pins the server image and places it on the control plane.

​
New Instance

New Instance opens New ClickHouse Instance: Create writes a ClickHouseInstallation with that layout, a volume template of that size and class, and Celum’s admin user, whose password comes from the admin Secret in security. A new instance gets nothing else: no limit on ClickHouse’s own log tables, no metrics endpoint, no control-plane placement, and the operator’s default server image.

​
Edit an instance

The pencil opens the same form without the name. Current Layout and New Layout show shards × replicas and the number of pods each gives. Apply Changes changes only the instance’s shards, replicas and data volume (size and class), and keeps everything else on it. It also keeps the passwordless default user to loopback. An instance without the default cluster or the data-volume volume template was not made by Celum, and is refused with 409. Celum checks neither a change of Storage Class nor a smaller Storage Size; it writes what you enter. What happens next is up to the ClickHouse operator and Kubernetes. Kubernetes does not let an existing volume change its storage class or shrink, and grows a volume only where its storage class allows expansion.
hubble-flows belongs to the ClickHouse section of Onboarding → Monitoring (Storage class, Storage size (GiB), Replicas per shard). An edit here keeps what that step set (log retention, metrics endpoint, pinned server image and placement), but the next Apply of that step sets the layout and storage back to its own values. Before Celum 1.296, an edit here dropped those settings; applying the step again restores them.

​
Delete an instance

The bin asks for confirmation, then deletes the ClickHouseInstallation. For hubble-flows, the confirmation warns that the Security pages show no traffic without it. Celum deletes only the ClickHouseInstallation and leaves its volumes to the operator. An HTTPS exposure of the instance is not removed with it. Remove the exposure first, or it is left pointing at a Service that no longer exists.

​
Expose via HTTPS

The globe on an instance opens Expose ClickHouse, or Change exposure when the instance is already exposed. The globe is green while it is exposed, and hovering over it names the hostname.
  • Hostname is proposed from the supervisor’s subdomain and domain in Settings → Kubeconfigs: clickhouse.<subdomain>.<domain> for the first instance, clickhouse-<instance>.<subdomain>.<domain> for any other. A supervisor without a domain gets an example.com placeholder; replace it. On an instance that is already exposed, the form starts from the live hostname, and saving re-applies that route. The hostname of a live exposure cannot be changed here: remove the exposure, then expose the instance again.
  • Gateway is Create new Gateway (new LoadBalancer IP) or one of the supervisor’s existing Gateways that admit routes from security. A project’s own Gateway admits routes from its own namespace only, so it is not offered, and the API refuses it with 400. A new exposure starts on the shared Gateway; Change exposure keeps the Gateway the exposure already uses.
Expose writes, in security, an HTTPRoute named clickhouse-<instance>-gateway to the Service clickhouse-<instance> on port 8123. It adds an HTTPS listener for the hostname to the Gateway, and an HTTP route that redirects to HTTPS with a 301. A new Gateway uses the gateway class cilium and gets its own LoadBalancer address. The hostname gets its own certificate, route-<hostname>, from the supervisor’s active issuer: the one pinned on the Certificates page, otherwise the one Celum detects. An AD CS issuer works too. Expose does not wait for it to be issued; the endpoint answers HTTPS once cert-manager has issued it. A Gateway the exposure creates carries no cert-manager.io/cluster-issuer annotation. A supervisor with no issuer at all leaves the certificate to an existing Gateway’s annotation, and is refused a new Gateway. A hostname another route already serves is refused with 409 before anything is written. Exposing an instance again repairs an endpoint whose listener serves no certificate, such as one exposed under an AD CS issuer before Celum 1.297: the listener moves to the new certificate. A listener that serves a certificate keeps it. To stop exposing an instance, use the bin beside its HTTPS endpoint. When the exposure created its own Gateway, the confirmation names that Gateway, its address, and any other hostname it serves: all of them go, with their certificates. On a Gateway it shares, the routes go, and so do the hostname’s listeners and certificate unless another route still serves that hostname; the shared Gateway and a project’s own are never deleted. The instance and its data stay. A failed expose or removal shows a notification.

​
Uninstall ClickHouse

Uninstall ClickHouse, at the foot of the panel, is disabled while the panel lists any instance but hubble-flows, and names them: delete them first. Otherwise it asks for confirmation and then:
  1. deletes the retired flow-collector Deployment and Service if they are still there, the mirrored Hubble client certificates, the clickhouse-write NodePort Service, and the ClickHouse writer and admin Secrets in security;
  2. deletes hubble-flows with its data, and waits up to 60 seconds for it to go;
  3. deletes the clickhouse-operator HelmRelease, so helm-controller uninstalls the operator;
  4. deletes the clickhouse-system namespace, and waits up to 60 seconds for it to go.
The API answers with status: uninstalled and a list of warnings, such as an instance or namespace still terminating after 60 seconds; the panel shows them in a notification. See Force-cleanup escape hatches for what to do with them.
Uninstall deletes only hubble-flows, so it refuses while any other instance exists. Without the operator, another instance would keep its pods and volumes with nothing left to remove them. The API answers 409 with the instances it found in security and clickhouse-system, before it deletes anything, and looks again before it removes the operator. An instance in clickhouse-system is not on the panel: delete it with kubectl. While a deleted instance is still terminating it stays listed; reload the page once it is gone.

​
API and MCP

The security:* actions are checked on krn:celum:supervisor:<supervisor>:security:*, and the gateway:* actions on krn:celum:supervisor:<supervisor>:gateway:*.

​
Guest clusters and vclusters

Guest clusters do not get their own database. Their collectors write into the supervisor’s ClickHouse and tag every row with the guest’s cluster name. That is why the flows-source selector on each Security tab (?cluster=) can switch you into a tenant’s traffic without leaving the supervisor. A guest’s cluster agent writes to clickhouse-central.security.svc:31900 inside the guest: a headless Service whose endpoints are the supervisor’s nodes, on NodePort 31900 of the clickhouse-write Service in security. It signs in as a ClickHouse user of its own, guest_<namespace>_<cluster>, which may insert only rows of its own cluster and cannot read. The Flow Telemetry Apply gives every flow-telemetry guest that user, and so do creating a cluster, editing one and turning its flow telemetry on, before its agent starts.
  • The password exists only in the secret store, at clusters/<supervisor>/<namespace>/clickhouse-writer-<cluster>. External Secrets writes it into the Secret clickhouse-writer-<cluster> in the cluster’s namespace.
  • A ClusterResourceSet of the guest’s own, clickhouse-dsn-<cluster>, delivers the DSN into the guest’s kube-system/cluster-agent-clickhouse.
  • A guest keeps writing as the shared flows user, and its row in the Apply’s guestWriters says why, when the supervisor has no usable secret store; when the flow store predates clickhouse-schema 1.0.25, or a materialized view on a guest’s table does not run as its definer; or when the cluster’s name is the supervisor’s own or is used in another namespace too, because rows name a cluster without its namespace.
Rotating the shared flows password is refused while such a guest exists, unless you force it. Both rotations, the supervisor’s and the guests’, are described in Flow store passwords. Collection for a guest is enabled through the cluster’s own configuration rather than from this tab: new clusters get it by default, existing ones are opted in per cluster. Two consequences are worth knowing before you debug a silent guest:
  • Byte accounting depends on a Cilium setting that only takes effect after the agent restarts. The signature is flow rows arriving normally while byte-carrying tables stay at zero.
  • The supervisor’s write endpoint has to be reachable from the guest. When it is not, the guest’s collectors log connection refused and nothing at all appears for that cluster — which freshness shows as one entry going stale while every other cluster stays current.
vclusters need a different treatment entirely. A vcluster has no CNI of its own: the syncer pushes its pods onto the host as ordinary pods in a host namespace, so the namespace the tenant believes it is in does not exist on the supervisor, and filtering by namespace finds nothing. What does exist is a label pair the syncer stamps on every synced pod — one naming the vcluster, one naming the namespace the guest sees. GET .../security/vclusters enumerates those virtual namespaces, discovered from both the flow store (what is talking) and the pod API (what exists but is idle), and the Security scope pickers use them to scope views correctly.
The same host namespace can hold a native workload and a vcluster-synced one with byte-identical labels. Only the two vcluster labels tell them apart. A policy written for the tenant without those labels silently covers the native twin as well. Use the vcluster scope in the authoring surface rather than hand-writing the selector, and see Topology for the origin filter that separates the two worlds on the graph.

​
Tetragon

Tetragon is the optional runtime-security layer: eBPF probes that produce process-level events and real byte counters alongside the network flows. It has no tab today; it is installed and operated through the API below. It installs into its own tetragon namespace so its lifecycle is independent of the ClickHouse stack — you can add or remove it without touching flow collection. Four starter tracing policies ship with the platform: tcp-flows, tcp-accept, process-exec and tls-sni. They are labelled as platform-managed, so reconciliation removes drift among those without ever touching TracingPolicies you created yourself. Two operational notes. Readiness is the DaemonSet count, not the presence of a release — the eBPF programs live in the agent pods, so installed: true with dsReady: 0 means nothing is being observed. And the collectors pick Tetragon up automatically: once the agent’s gRPC Service exists, re-running enable wires the subscription without further configuration.

​
Platform secrets

The Secrets tab appears only where External Secrets Operator is installed. GET .../security/secrets returns the store configuration together with every platform-managed secret and its sync state:
Secret values are never returned — only names, key names and sync state. Unlike the rest of this page, the secrets routes have actions of their own: reading the list needs security:ReadSecrets, and POST .../security/secrets, which creates one, needs security:WriteSecrets rather than security:Onboard. When storeReady is false, every secret reads as failing with the store’s own error in storeMessage; fix the store before investigating individual secrets. On the tab, the tiles count synced and failing secrets, the namespaces consuming them, and the store’s state with its Vault mount. The table lists each target Secret with its namespace, Vault path, keys, last sync and status. Add secret writes a value to Vault and has External Secrets sync it into a namespace:
  • Namespace and secret name decide both the Kubernetes Secret and the Vault path: <mount>/clusters/<supervisor>/<namespace>/<name>, shown in the dialog before you save.
  • Type is Key / value (any keys) or Docker pull secret (registry, user and password, synced as a .dockerconfigjson).
  • Saving again with the same namespace and name replaces the secret entirely: any key not in the new save is removed. The dialog warns when the name already exists. Celum’s Vault credential cannot read secrets back, so it cannot merge; send every key you want to keep.
There is no delete. The platform’s Vault credential can create and update secrets but not remove them, so removing one is two manual steps: delete the path in Vault, then the ExternalSecret and the Secret in the namespace.

​
Tearing it down

POST .../security/uninstall removes the stack in the order that actually works: a leftover flow-collector Deployment if there is one, and the platform-owned Services and Secrets first, then the ClickHouseInstallation with foreground propagation so the operator cleans up its own dependants, waiting for it to disappear — and only then the operator release itself. Removing the operator first would leave a finalizer nobody can process.

​
Force-cleanup escape hatches

Two endpoints exist for when that ordering has already been broken. Both are destructive and neither is part of normal operation.
POST .../security/chi/force-cleanup strips the finalizer from the flows installation and force-deletes it. The finalizer was the operator’s opportunity to clean up, so the pod, Service and PVC are left behind — and the PVC holds the flow data. The endpoint reaps those leftovers by label only when the operator namespace is already empty; otherwise you delete them by hand. Use this only when the operator was removed before the installation finished terminating.
POST .../security/namespace/force-cleanup clears the finalizers on the ClickHouse operator namespace so a namespace stuck in Terminating can complete. Everything still inside that namespace goes with it, and any controller that expected to run cleanup on the way out never gets the chance. Reach for it only when the namespace is genuinely wedged.
Both return already-gone when there is nothing to clean, so a mistaken call against a healthy supervisor is a no-op rather than damage.

Flow telemetry

Component health, schema versioning and the ingestion triage order.

Security overview

What the pipeline feeds, and how to tell no data from no traffic.

Day-2 operations

Where guest-cluster settings, including telemetry, are changed.

Networking overview

The Cilium datapath these collectors observe.
Celum AI reads the same surfaces: get_component_status for whether the pipeline components are installed and ready, get_flow_freshness for whether ingestion is actually alive, and list_platform_secrets for the secrets view and its sync state.