Skip to main content
Platform logging is two components, both in the monitoring namespace: Loki without Alloy stores nothing; Alloy without Loki has nowhere to push. The wizard installs them as separate steps so each has its own release state.

​
Install them in this order, and finish with a third step

1

Loki first

Alloy’s default push target is the in-cluster Loki service. Install it with no Loki present and the DaemonSet runs happily, reporting healthy, while dropping every line on the floor.
2

Then Alloy

Once Loki is Ready.
3

Then re-apply Monitoring with Loki enabled

This is the step people miss. Grafana only gains Loki as a datasource when the kube-prometheus-stack release is re-applied with Loki enabled — that wiring lives in the monitoring stack’s values, not in Loki’s own. Skip it and logs are stored, queryable through Celum, and absent from Grafana.
Re-applying Loki replaces its configuration, it does not merge. Omitted fields fall back to defaults, so an empty re-apply resets storage to 50Gi, the StorageClass to the supervisor’s default and the chart to the version Celum pins. Read the current desired configuration from the status first and send it back with your change. Before your first Loki apply on a supervisor, read Logs from before tenancy: that apply deletes logs.
Alloy collects pod container output only. There is no journald or host-log source, so a kernel or kubelet problem visible in dmesg on a node will never appear in Loki. Those stay a node-level investigation.It is also not the guest-cluster path: a tenant’s logs come from the in-guest Alloy add-on, which reports its own status.

​
Configuration that matters

The status reports the desired configuration each release runs with:
  • Loki — storage class and size (e.g. ceph-block, 50Gi). Loki keeps every project’s and cluster’s logs until that volume is full, so its size is the only dial. The status still carries a retention period, which reads 0s (keep forever) once Loki has been applied. The Loki step’s Retention period field is not applied: an apply that sends any value other than 0s adds a Loki retention row that says so.
  • Alloy — the Loki endpoint it pushes to (the in-cluster service, http://loki.monitoring.svc.cluster.local:3100) and the excluded namespaces whose logs are not collected (e.g. flux-system, cilium-secrets, tetragon-system) — high-churn system namespaces that would otherwise dominate the store.

​
Tenants

Loki is multi-tenant. Every write and every read names its tenant in the X-Scope-OrgID header, and Loki refuses a request that names none. Where a stream lands depends on who wrote it: Readers name every tenant the streams can be under. Celum’s read of a guest cluster’s logs names the projects that hold a cluster of that name, the cluster’s own name, the supervisor and fake. Grafana’s Loki datasource names every tenant of the supervisor: the supervisor, every project, every Cluster API cluster and fake. The Loki apply, the kube-prometheus-stack apply, project create and delete, and cluster create and delete keep that list current. Both lists are built from what exists now, so the logs of a deleted cluster or project stay in Loki and are no longer read. Loki refuses writers that name no tenant: the Alloy of a guest whose project’s add-on catalogue is older than capi-addons 0.6.7, and a supervisor Alloy whose configuration has no tenant_id. The Loki apply names them in its Loki writers row. Edit one cluster in each namespace it names to move that catalogue, and apply the Alloy step for the supervisor’s Alloy. An Alloy you point at another supervisor’s Loki (Loki push URL (override)) writes under tenants that supervisor’s Celum and Grafana do not read.

​
Logs from before tenancy

The first Loki apply on a supervisor deletes, for good, the logs Loki kept from before it was multi-tenant. Those logs sit under the tenant fake. Loki gives fake alone a 24-hour retention, so its compactor deletes the fake logs older than a day, then the rest as they turn a day old. Every other tenant keeps all its logs: Loki deletes nothing else, and its delete API is off. If you still need any of the fake logs, save them before you apply Loki.
These apply Loki: Re-apply chart on the kube-prometheus-stack panel does not apply Loki. It still closes the Loki port 8080 in project policies (see below) wherever the Loki route no longer goes to Loki’s retired NGINX gateway.

​
hr-status vs status

Both components expose two reads, and the distinction is the useful part: The split mirrors the three-layer triage: hr-status answers “did the chart land”, status answers “what is it configured to do” — and neither proves logs are actually arriving. For that, query a pod’s logs and check the source (below).

​
The tenant log read layer

Pod logs of a guest cluster, on the cluster’s Resources tab, are served Loki-first with live fallback:
1

Loki first

The backend queries the supervisor’s Loki through the Kubernetes API server’s service proxy, never through the Gateway route, with a selector on cluster, namespace and pod (and container, when one is picked). The query names every tenant the cluster’s streams can be under (see Tenants) and returns up to 72 hours of history, including logs from pods that have since restarted or been rescheduled.
2

Live fallback

When Loki has nothing for the pod (shipping is off, or ingestion has not caught up), or the Loki read fails, the request falls back to the live kubelet log read. You never get less than the plain Kubernetes path would give you.
3

Source badge

The logs view labels each transcript with its source (Loki · history or live), so a silent fallback is visible. A pod that should have history but shows live is your cue to check the shipping path.
Over the API, GET /api/clusters/<cluster>/workload-pod-logs takes source=auto (the default), loki or live. With loki it does not fall back: it returns an empty transcript and a note. Over MCP, get_cluster_pod_logs takes the same source. The supervisor’s own pod logs are not read from Loki: GET /api/supervisors/<supervisor>/pod-logs reads them live, and no page or MCP tool calls that route. Guest clusters ship their logs with their own Alloy collectors pushing to the supervisor’s Loki. That path requires Loki to be published on the shared Gateway — guests are separate clusters and can only reach it via a routable hostname. Celum exposes the route automatically when Loki is installed; see Gateways & Routes. The route forwards /loki/api/v1/push only, to the loki Service on port 3100. It answers no query: Celum and Grafana read Loki inside the supervisor. On a supervisor whose shared Gateway is Envoy Gateway and which has a usable secret store, the route also requires the guest’s project push credential, as basic auth, once every guest that runs the Logs add-on sends it. The Gateway then sets the Loki tenant to the guest’s project and removes the Authorization header before the request reaches Loki. The push credential describes the credential and when the requirement turns on; the Prometheus route follows the same rules.
That route is not cosmetic, and its absence is silent. The guest log-shipping add-on resolves its push URL by reading exactly this loki route. With no route to read, enabling the guest Logs — Alloy toggle does nothing at all — no error, no logs, and a guest-side collector that looks installed.

​
Why a project’s network isolation does not already allow that push

This is the least intuitive thing on the page, and it costs an afternoon every time it is met fresh. A project namespace with network isolation gets an egress rule allowing traffic to world — the public internet. A guest’s collector pushes to the Loki hostname on the shared Gateway, which is an address outside the cluster, so that rule looks like it covers the push. It does not. The Gateway terminates TLS and hairpins the request back into the cluster, and the proxy policy-checks that upstream hop with the original client’s egress policy against the real backend pod identity — the Loki pod in the monitoring namespace, not world. A rule written for world never matches, the push is dropped at L7, and the shape of the failure is distinctive: the proxy answers 403 while the flow record reads verdict=DROPPED with the destination logged as the Gateway’s address. A collector then retries forever, producing a steady stream of denials and no logs. On an Envoy Gateway supervisor, the guest’s first hop is the Gateway’s own Envoy proxy, a pod in envoy-gateway-system. A world rule does not match that pod either, and the isolation policy holds a hole for it as well. The platform maintains that egress hole itself, so on a current release there is nothing to do. Two details matter if you are reading an older namespace or writing policy by hand:
  • The hole opens one port, 3100. That is the port the loki route targets. Port 8080 on the Loki pod belongs to its rules sidecar, not to a Loki API, so the hole leaves it shut. Policies written by older releases still open 8080: the next Loki or Monitoring apply removes it from every project isolation policy and allow-tenant-loki-push hole, and reports it in the */Loki egress row (narrowed in N policies). While the Loki route still goes to Loki’s retired NGINX gateway, which listens on 8080, the row reads kept in N policies until a Loki apply moves the route.
  • Do not hand-write the hole into a namespace that has no isolation policy. A CiliumNetworkPolicy with an empty endpointSelector flips every endpoint in that namespace to egress default-deny — so a “fix” applied to an open namespace creates the very lockout it was meant to relieve. The platform ties the two together for this reason: the hole is applied only where isolation exists, and removed with it.

​
What commonly breaks

The collector DaemonSet needs privileged access — it mounts host log paths on every node. If the namespace it runs in enforces a restrictive Pod Security level, the DaemonSet schedules zero pods while the HelmRelease still reports Ready — the chart deployed fine; the workloads were rejected at admission. A green release with no recent logs: check the collector’s pod count first, then the namespace’s Pod Security labels.
Other recurring failure modes, in triage order:
  1. Alloy not installed — logs views quietly serve live only. The status panel shows the release absent.
  2. Collector pods down — the Pod Security case above, or a node-level issue; either way the release is green and the DaemonSet is not.
  3. Guest push path broken — the guest’s collectors are healthy but the Loki route is missing or unreachable from the guest network; supervisor logs work, guest history does not.
  4. Guest push denied by policy — route present, collector healthy, and every push dropped. The flow record reads DROPPED against the Gateway’s address while the proxy answers 403. See the section above: a world egress rule does not cover a hairpinned push.
  5. Guest push refused with 401: the Loki route requires the project’s push credential, and the guest sends none or a wrong one. The loki-push-auth row of the next Monitoring or Loki apply reads kept; failing for N guests that do not send the credential and names each guest with the reason, such as a guest outside a project, an add-on rendered without the credential, or a credential Secret not delivered yet. See The push credential.
  6. Pushes refused because they name no tenant: a guest whose add-on catalogue is older than capi-addons 0.6.7, or a supervisor Alloy without a tenant_id. The Loki apply’s Loki writers row names them; see Tenants.
  7. History shorter than expected: Loki deletes no tenant’s logs except fake’s, and the Retention period field plays no part. Missing history is either from before tenancy, which a Loki apply deletes once it is a day old, or from a cluster or project that no longer exists, whose logs Loki keeps and Celum and Grafana no longer read.
  8. Logs exist in Celum but not in Grafana — the monitoring stack was never re-applied with Loki enabled, so Grafana has no Loki datasource. Nothing about Loki itself is wrong. If the datasource exists and one cluster or project is missing from it, its tenant is not in the datasource’s tenant list yet. Re-apply chart on the kube-prometheus-stack panel rewrites that list, and so does a Loki apply.
  9. A reader or writer of your own stops working: Loki refuses a request without X-Scope-OrgID. On an Envoy Gateway supervisor, the Monitoring apply also admits only the monitoring namespace, the nodes, the API server and the Gateway’s proxies to the Loki pod; see Direct access to Prometheus, Loki and Alertmanager. That includes Alloy inside a VM (the alpha Send system logs to Grafana), which writes to Loki’s in-cluster address.
  10. Looking for kernel or kubelet messages — they are not collected. Alloy reads pod output only.

​
Permissions

Platform health overview

Release state vs runtime state — the model this page applies.

Monitoring

The metrics half of the observability stack, same namespace.

Flow telemetry

Network flow data — a separate pipeline with its own freshness signal.

Gateways & Routes

The shared Gateway that guest log shipping depends on.