Skip to main content
Cluster problems announce themselves at different depths — a cluster that never provisions fails at the Cluster API layer, a crashing app fails three layers further in. Working top-down keeps you from reading pod logs for what is actually a machine that never booted.

​
The triage order

1

Fleet health — which cluster, how badly

GET /api/clusters/health (the cluster list page) gives per-cluster health across the supervisor. Start here to confirm which cluster is unhealthy and whether its API is reachable at all — an unreachable tenant makes every deeper read fail, and that is your answer, not an obstacle.
2

Machines and phase — does the cluster physically exist

GET /api/clusters/<cluster>/nodes lists the Cluster API machines. A cluster stuck in Provisioning, or a worker count that never converges on the desired replicas, is an infrastructure-provider problem: the cluster’s events page (/s/<supervisor>/clusters/<cluster>/events) carries the provider’s own reason. Nothing inside the tenant can be fixed until the machines run.
3

Events inside the tenant — what Kubernetes itself is complaining about

GET /api/clusters/<cluster>/events is the tenant’s own event stream. Warnings with high counts are the signal — this real excerpt points directly at a crash-looping container:
4

Pod events and logs — the actual failure

Follow the object the event names: GET /api/clusters/<cluster>/workload-pods?podNamespace=<ns> shows status and restart counts, .../workload-pod-events narrows to one pod, and .../workload-pod-logs reads what the container printed before dying. A CrashLoopBackOff with four-digit restarts has been failing for days — the first log lines after startup usually say why.
When no curated view covers the resource you suspect, GET /api/clusters/<cluster>/kubectl-get fetches any object’s full YAML — see the escape hatch.

​
Patterns that look like failures but aren’t

​
Metrics panel empty — available: false

GET /api/clusters/<cluster>/metrics returns 200 with "available": false and zeroed numbers when the tenant has no metrics source installed. This is the documented fallback, not an outage:
Fix by enabling the monitoring addon (or installing metrics-server in the tenant) — not by debugging connectivity.

​
Cluster reachable but views are empty — cold-start partial status

Right after the backend starts (or the first time a supervisor is queried), list endpoints can return 200 with an empty result while caches warm in the background — the health endpoint explicitly serves an empty list on a cold cache rather than blocking on a slow probe. The UI polls and fills in within moments. An empty cluster list or blank health column seconds after a deploy means “not warmed yet”; conclude an outage only if it persists across refreshes.

​
Creation rejected for overlapping CIDRs

Cluster creation validates pod/service CIDRs against the supervisor and every existing cluster, and rejects explicit overlaps — a guest whose pod CIDR collides with the supervisor’s is unreachable from the supervisor’s own pods, so the rejection is protecting you. Leave the CIDR fields blank and the allocator picks free subnets automatically. See Create a cluster.

​
Talosconfig download returns 404

GET /api/clusters/<cluster>/talosconfig returns 404 for any non-Talos cluster — the credential simply does not exist for that provider. Same shape for a vcluster instance kubeconfig whose export was never enabled (details).

​
A node VM was recreated on a new address, and things that dial it stop

This follows a node VM being recreated onto a new IP, and it has two layers that fail independently. Both are self-healing on a current release; the operator-facing question is which one did not heal. Layer one — the Node object itself keeps the old address. This is the counter-intuitive half. On a guest cluster the kubelet does not publish its own addresses: it defers them to the external cloud provider, which sets them once at node registration. Nothing re-reads them afterwards. So a VM that comes back on a different address leaves a Node object still advertising the one it was born with, and the cluster goes on reporting Ready — because the kubelet-to-control-plane direction works, and that is the direction health is measured in. Everything that dials into the node by that address breaks at once, which is what makes the symptom set look unrelated: The platform runs the cloud provider’s node controller, which re-syncs those addresses from the VM every few minutes, so this normally corrects itself. The node-lifecycle controller is deliberately left off — it deletes nodes it cannot map to a VM, which is a far worse failure than a stale address.
On the supervisor side the tell is distinctive: traffic aimed at the dead address no longer resolves to a known workload, gets classified as external, and is dropped by the project’s egress policy. A burst of egress denials naming an address inside a guest’s own range is this, not a policy mistake.
Layer two — the CNI agent keeps the old address in its own node record, even after the Node object is correct, so the overlay’s tunnel endpoints disagree and cross-node traffic is dropped cluster-wide. This layer has its own repair: the cluster agent runs a watchdog that notices the drift and restarts that node’s Cilium agent, on a 60-second loop. So when cross-node traffic is still dead after the addresses are correct, the question is why that watchdog did not run:
  • the guest’s cluster agent is older than the watchdog — the likeliest cause, because add-on versions are sticky and a cluster will otherwise sit on a months-old build indefinitely. Upgrading the guest agent is the repair;
  • the watchdog was explicitly disabled on that agent.
Upgrading a guest’s cluster agent re-renders that namespace’s whole add-on catalog, not just the agent — every other add-on’s enablement is re-read and re-applied, and telemetry endpoints are re-resolved. A project’s guests move together. Worth knowing before doing it to fix one cluster.

​
When the platform layer is the suspect

If every cluster on a supervisor misbehaves the same way, stop debugging tenants and check the supervisor’s components — the CAPI engine, addon engine, and providers all report through the component-status model:
  • GET /api/supervisors/<supervisor>/capi/status — operator and provider readiness with the chart controller’s reason and message.
  • GET /api/supervisors/<supervisor>/capi/addons/status — the CAAPH engine and each HelmChartProxy; a proxy that is not ready explains a missing CNI or agent in new guests.
  • POST /api/supervisors/<supervisor>/capi/recover-rollback — recovers the engine from a stuck Helm rollback state when its status shows a failed release.

​
Permissions

Triage is read-only: cluster:GetHealth, cluster:GetNodes, cluster:GetEvents, and cluster:GetWorkloads cover the whole ladder, with cluster:GetResources for kubectl-get and capi:GetStatus for the engine checks.
Celum AI runs this same ladder on request — get_clusters_health, get_cluster_events, and get_cluster_pod_logs are the assistant’s versions of the steps above, so “why is demo-cl unhealthy?” walks the identical APIs.

Day-2 operations

Every read used above, with fuller payload examples.

Platform health

The component-status model the supervisor-level checks build on.

Create a cluster

The creation-time failure table, including CIDR and quorum mistakes.

Storage overview

The parallel triage ladder for Pending PVCs inside tenants.