Monitoring

Each supervisor runs its own kube-prometheus-stack — Prometheus, Alertmanager, Grafana, kube-state-metrics, and node-exporter — installed by the onboarding wizard into the monitoring namespace as a Flux HelmRelease. Two pages cover it. Overview → Monitoring is a live read: the release, its versions, the exporters, what Prometheus actually scrapes, and the ClickHouse flow store. Onboarding → Monitoring is where you configure, upgrade and repair the stack.

​

Install state and versions

The status reports the release identity and both version signals: Alongside it, the helmRelease block carries the Ready condition with reason and message — see the status model. The status also carries two configuration blocks: Only the API and the assistant (get_component_status with component=monitoring) show deployedConfig. Onboarding → Monitoring opens with the values from desired: StorageClass, Prometheus storage (GiB), Grafana storage (GiB), Retention (days) and, under Advanced (Prometheus + ServiceMonitors), Prometheus scrape interval. That is where you change them. Overview → Monitoring shows none of them; it shows the release name, chart version, app version and namespace. Re-applying with the same values is a no-op reconcile; changing storage or retention triggers a Helm upgrade with the new values.

​

ServiceMonitors

The status lists every ServiceMonitor on the supervisor — the objects that tell Prometheus what to scrape — with namespace, labels, and endpoint count. A typical supervisor’s inventory spans well beyond the stack’s own namespace: Some of these are created by their own operators (CDI, KubeVirt, Rook); the rest are platform-managed: Celum applies a known set covering Cilium, Hubble, the cluster agent, Ceph, and the hardware exporters. The install form lets you select which platform-managed ServiceMonitors to enable — an empty selection means the full default set. Re-applying the platform ServiceMonitors recreates the entire platform-managed set at any time; it is idempotent and safe to run after an operator upgrade removed one. No page has a button for it: call POST /api/supervisors/<supervisor>/monitoring/service-monitors, or the MCP tool install_service_monitors. Applying Onboarding → Monitoring also applies the ServiceMonitors selected there. The re-apply also creates two headless metrics Services the Cilium agent and operator need, because those pods serve /metrics with no Service of their own. It always writes the full default set and never the opt-in hardware exporters — those only go out through the install form’s exporter selection.

​

Opt-in exporters

Hardware and probe exporters are not part of the default install. Each one is a Helm chart in the exporter catalog: entries of the app catalog with the category exporter. The catalog is one list for the whole platform, and you install from it on each supervisor. Celum seeds five entries, and puts back any of them that is missing every time it starts: On Overview → Monitoring, Installable exporters lists every enabled entry with its chart, and either its pinned version or latest at install time. It counts how many are installed on this supervisor.

• Install writes a Flux HelmRelease named after the entry into the monitoring namespace, with the entry’s values. It also registers the chart’s repository as a Flux HelmRepository. An entry without a version gets the newest version in the repository’s index, pinned in the HelmRelease.
• An installed entry shows its source badge, normally Flux, and Remove. Remove asks first, then deletes the HelmRelease, and helm-controller uninstalls the chart. Metrics already collected stay in Prometheus until they age out of retention.

The status reports each exporter’s installed state, so the list doubles as the inventory of what this supervisor collects. Install only what the hardware justifies: the GPU exporter on a GPU-less supervisor is a DaemonSet doing nothing.

​

Your own exporter

Add exporter, beside the count, adds an entry to the exporter catalog. It installs nothing. Celum also sets the entry’s target namespace to monitoring and its category to exporter, enables it, and sorts it after the five seeded entries. What happens next: Rules that apply to your own entries:

• Celum installs the values exactly as written. It does not render {{ }} placeholders in an exporter’s values. Values that are not a YAML mapping are replaced with serviceMonitor.enabled: true when you install.
• The install passes no credentials, and registers the repository as a Flux HelmRepository of type default, not oci. The chart must therefore come from a repository the supervisor can read anonymously. On an air-gapped site, point Chart repository at a mirror the supervisor reaches.
• A name the catalog already holds is refused.
• You change or delete the entry in the app catalog, where it is listed with the category exporter. Remove the exporter from every supervisor first: once its entry is deleted or disabled, the Monitoring tab no longer lists it, and neither Remove nor the API can uninstall it by name.
• The entry also appears in Install App’s chart list, because that list shows every enabled entry. Install it from the Monitoring tab instead: Install App refuses an entry without a pinned version.

​

Exposed routes

When the supervisor runs a shared Gateway, the Monitoring apply publishes Grafana and the Prometheus push path on it, and the Loki apply publishes the Loki push path. The status lists every route in the monitoring namespace with its hostname, the Gateway’s address, its TLS state and, for a route narrowed to named paths, those paths. See Gateways & Routes for how the shared Gateway works. Celum reads Prometheus and Loki inside the supervisor, through the Kubernetes API server’s service proxy, and Grafana reads them by their in-cluster addresses: nothing reads through the two push routes. On both, the Gateway removes the Authorization header before the request reaches Prometheus or Loki. The Monitoring card on Overview → Monitoring shows Grafana as a link and Prometheus as its hostname, marked /api/v1/write only. If another route already serves one of these hostnames and answers every path, the apply does not add a second route. Its row reads not narrowed and says to withdraw that route and re-apply, or to narrow it to the push path. The Monitoring apply also removes the prometheus-write NodePort Service (31090) that earlier releases wrote, which served the whole Prometheus API in plain HTTP. Guests write through the route instead. A Service of that name that Celum did not create stays in place, with a skipped row.

​

The push credential

On a supervisor with a usable secret store, each project has a telemetry push credential. Its username is the project’s name and its password is random. The secret store holds it, External Secrets writes it into the project namespace as the Secret telemetry-push, and a ClusterResourceSet copies the username and password into monitoring/telemetry-push in every guest of the project that runs the Monitoring or Logs add-on. The guest’s Prometheus agent and Alloy send it as basic auth.

• New projects get the credential when they are created. If that fails, the next Monitoring apply issues it.
• Existing projects get it from the Monitoring apply, which also delivers it to the project’s guests and moves the project’s add-on catalogue so that its guests send it. The catalogue moves to the add-on engine’s chart version, at most capi-addons 0.7.1 and never down; nothing else in it changes, and the row names the move. The project’s guest agents restart once. Nothing moves while the engine is older than capi-addons 0.7.0 (a not wired row: apply the Cluster API step first), and a catalogue older than 0.6.22 is left alone with a skipped row: edit one of the project’s clusters to move it.
• The requirement turns on per route. Each route gets a basic-auth SecurityPolicy, prometheus-push-auth or loki-push-auth, only once every guest that runs the route’s add-on sends the credential: its add-on release is Ready and rendered with the credential, and the credential Secret has been delivered. Until then the policy row is skipped and names up to five guests that do not send it yet. The first apply that delivers the credential usually skips; apply Monitoring again a few minutes later, once the guests’ agents have restarted.
• Once on, it stays on. A guest that sends no credential, or a wrong one, gets 401, and the row reads kept; failing for N guests that do not send the credential. A deleted project’s credential stops being accepted.
• Guests outside a project have no credential. While one runs the add-on, the route stays open and the row names that guest. Once the route requires the credential, such a guest is created without the Prometheus agent or Alloy, and the apply turns the add-on off in every add-on catalogue outside a project namespace.
• Nothing is required on a supervisor whose shared Gateway is a Cilium Gateway, or that has no usable secret store. Both routes stay unauthenticated, and the rows say so: skipped — <route> is not on an Envoy Gateway; pushes stay unauthenticated, or skipped — no usable secret store; pushes stay unauthenticated. The Loki route also keeps no policy while Loki still runs its retired NGINX auth, until the Loki step is applied again.

​

Direct access to Prometheus, Loki and Alertmanager

On a supervisor whose push routes run on an Envoy Gateway, the Monitoring apply ends with the CiliumNetworkPolicy monitoring/telemetry-backends-ingress on the Prometheus, Loki and Alertmanager pods. It admits only:

• the monitoring namespace, such as Grafana and the supervisor’s Alloy;
• the nodes and the API server, for kubelet probes and for Celum’s reads through the service proxy;
• the shared Gateway’s Envoy proxies, on ports 9090 and 3100, and on 8080 while the Loki route still goes to Loki’s retired NGINX gateway.

No workload in any other namespace, in a project or not, reaches these pods directly. Guests are not affected, because they push through the Gateway. Something of your own that reads them from another namespace by their in-cluster address stops working: admit its namespace with a CiliumNetworkPolicy of your own in monitoring. Deleting telemetry-backends-ingress lasts only until the next Monitoring apply. The apply does not create the policy, and removes an existing one, when:

• the push routes run on a Cilium Gateway, which checks each guest’s own identity at the pods;
• in the flow store’s last 15 minutes, a node opened connections to its own pods from an address that Cilium sees as outside the cluster. The policy would refuse the kubelet’s probes; the row names the address and its node. Give each node, and every node added later, its node IP as a /32 on lo. The check runs only when Monitoring is applied.

It does not create the policy either when the flow store cannot be read, or when it holds no connection in that window from each node that runs one of these pods to its pods. In those two cases an existing policy is kept.

​

When the release is not ready

The most common monitoring failure is simply an unready HelmRelease — and the panel surfaces the chart controller’s reason and message directly:

​

A Ready release with missing metrics

This is a different failure class, and the most common one after the release itself. Component status reports a deployment fact — the chart landed. It says nothing about whether anything is being scraped, and the two disagree often. The scrape view answers the second question by reading Prometheus’s own up metric: one row per job, with targets up and down, and the failing targets named. It deliberately returns no health verdict, because the two failure shapes need opposite fixes: The second is easy to misread as a gap in the answer. It is not: a job with zero targets emits no up series at all, so the absence is the diagnosis.

​

Why a platform job goes missing

Almost always because Cilium or Rook was installed after Monitoring. A ServiceMonitor whose target namespace does not exist yet is skipped silently, and nothing ever retries it — so the stack installs clean, the release is Ready, and cilium-agent, cilium-operator, hubble, rook-ceph-mgr, rook-ceph-exporter or cluster-agent simply never appear. Re-applying the platform ServiceMonitor set is the repair, and it does not touch the kube-prometheus-stack release.

​

Permissions

​

Related