Skip to main content
A declared mirror is not one setting. It is two kinds of artefact travelling over four channels to five consumers, and each combination takes effect at a different moment. Most mirror problems are not “the mirror is broken” — they are one combination never having been wired, or having been wired after the thing that needed it was already installed. Registries covers registering one and why reachable is the wrong check. This page maps everything downstream of that: supervisor, guest, node and controller.

​
The two artefacts do not travel together

Knowing which one you are looking at saves most of the debugging. The asymmetry in the last two rows is the one that costs afternoons: a chart can be re-pointed, a node cannot be re-taught. A HelmRelease’s sourceRef is written once and nothing revisits it, so a chart installed before a registry was declared keeps pulling from upstream forever while everything applied afterwards comes from the mirror. The remedy is a re-apply, and it has a button — see the provenance audit on Charts & the mirror.

​
The five consumers

Supervisor nodes take their mirror from a Celum-owned Omni ConfigPatch. Talos applies it live — containerd re-reads its host config, no reboot and no drain. Guest nodes take theirs from the ClusterClass, as a strategic patch built from the registryMirrors and registryTrust variables. Every guest family works this way — talos-kubevirt, talos-vsphere and kamaji-kubevirt alike.
Guest clusters are never registered with Omni. They do not talk to it and it does not manage them, which is why their mirror arrives by a completely different route from the supervisor’s — even though the YAML that lands on the node is identical.
Guest workloads pull through their guest node, so they inherit whatever that node was booted with. Provisioning controllers — CAAPH fetching add-on charts, capvc fetching the vcluster chart — run as pods on the supervisor and pull through its nodes.

​
Two different things are called “the mirror”

This is the distinction most likely to mislead, because both are true and they contradict each other on the surface.

App-catalog pull-through

Supervisor only. Stocks the registry at install time and wires its pull secret onto a supervisor HelmRepository. A guest install pulls with the guest’s own credentials, so there is nothing to attach the secret to — and the request is refused rather than silently ignored.

Platform chart sources

Reaches guests. The add-on charts a guest cluster is built with — CNI, the telemetry agent, the metrics agent, the cloud-controller-manager — resolve through the supervisor’s declaration and are written into each HelmChartProxy as an oci:// URL.
So “my guests pull charts from the mirror” and “pull-through is supervisor-only” are both correct: they are different mechanisms with different owners. The first is a per-install option on a catalog app; the second is platform onboarding state that every guest inherits.

​
What a guest inherits, exactly

A guest declares nothing of its own. It inherits the supervisor’s declaration — deliberately, because two places describing one registry are two places that can disagree.
Only catalogued charts are redirected, and that guard is the point. Chart resolution takes the mirror branch whenever a default registry exists — including for a name the catalogue has never heard of, where the entry is empty. Emitting such a key would point the reconciler at an oci:// URL with no chart behind it. An absent key means “leave the upstream baked into the file”, which is also the safe answer for a second reason: pointing a guest at a registry whose nodes do not trust the certificate turns a working public pull into an x509 failure.

​
What Harbor specifically gives you

The node-side mirror is a pull-through proxy cache: the image reference stays quay.io/…, and containerd is told to fetch it from a project on your registry that proxies that upstream.
The pairs are declared by the operator, not hardcoded: each entry maps one upstream host to one proxy project. The endpoint URL is always https://<mirror host>/v2/<project>, built by a single function so the shape cannot drift between the supervisor channel and the two guest channels.
overridePath: true is what makes it transparent. Without it containerd appends the upstream’s own path to the endpoint and the request misses. With it, the proxy project is the whole path — which is why the mirror needs no change to any image reference anywhere.
A mirror that is warm serves at local-network speed. A cold entry costs whatever the upstream costs, once, and is warm afterwards — so the first build on a new estate is not representative of the second.

​
What is supported, and where

These paths speak the standard distribution API and work against any conforming registry:
  • pulling and pushing charts as OCI artefacts
  • copying images into the mirror (prepull)
  • the push probe behind Test, which starts a blob upload and cancels it
  • the node-side mirrors: declaration, which is containerd’s, not a vendor’s
A registry row carries a type, and it defaults to harbor. Nothing in the platform branches on that value today: the field records intent rather than selecting behaviour. Treat Harbor as the product the platform is built and tested against, and a plain OCI registry as usable for everything except pull-through caching.

​
Credentials: the two ways they surprise you

Registry rights are per PROJECT, not per registry. A robot account can hold push on the project that keeps charts and only pull on the project that keeps images, and a credential table that records one verb per host will read as “cannot push” when it pushes charts perfectly well. Ask the registry’s token service for a push scope on the specific project rather than inferring one answer for the whole host.
The mirror credential does not reach containerd through the mirrors: block. Measured across three configurations — with and without a mirrors: entry, against a private and then a public project — every one pulled anonymously. The CA under the same host key worked in all of them. So a proxy project that requires authentication will fail on the node even when the credential is correct everywhere else; the working shapes are a project that allows anonymous read, or a pull secret delivered to the workload.
The mirror credential is matched by host against the registered registries — so register the mirror first. An unmatched host silently yields an anonymous pull, which works on a registry whose proxy projects allow anonymous read and fails on the first one that does not.

​
Filling the mirror is a separate act from declaring it

Declaring where artefacts come from does not put them there. Two operations do that, and they are separate because charts and images travel differently. Both are long-running and report through a polled job. The chart mirror walks the platform chart catalogue, so a chart that is not catalogued is not copied — which is the same guard that keeps a guest from being pointed at a chart no registry holds. Cataloguing a chart is therefore what puts it in that dialog; there is no separate list to maintain.
A chart is also stocked on demand at install time — but only when the stock check succeeded and found it missing. On a read error the platform deliberately does not push, and lets the reconciler pull instead. That asymmetry exists because a read failure once forced a write, and a pull-only credential then turned a recoverable “version not found” loop into a hard install failure.

​
Three surfaces

set_onboarding_image_registry returning 200 does not mean the nodes were told. The declaration is saved on the supervisor resource first, then rendered into the ConfigPatch that Talos applies. The second half is best-effort: with the machine-management API unreachable the call still returns 200 with the declaration stored. Confirm with the provenance read, which compares declared against actually applied and reports its read failures instead of hiding them.

​
Permissions

Mirroring charts and pre-pulling images share registry:Prepull — both are the same privilege in substance: writing into a registry on the platform’s behalf.

​
What it is worth

Measured on one estate, the same guest cluster built twice — once fetching across the network, once through a warm local mirror: The third row is not an upstream comparison: that image already came from an internal registry, just one on the far side of a site-to-site link. Mirroring it locally was worth 9× anyway, which is the argument for treating “internal” and “local” as different words. The first row is the one worth internalising: it happens before the cluster exists, so no in-cluster setting can influence it. Only the node’s own containerd declaration can. The largest single saving was not an image at all. A chart left unpinned made the client download the entire repository index — tens of megabytes — to discover that “newest” meant a 90 KB chart. Pinning it removed that per-guest cost entirely.

​
Three things the mirror does not cover

  1. The registry’s own certificate. A mirror on an internal CA needs that CA on the nodes, which is the registryTrust half of the declaration. Without it every pull fails x509 and the mirror looks broken when it is merely untrusted.
  2. Anything not in the platform catalogue. Third-party charts named by an app definition carry their own repository URL and are not redirected.
  3. Artefacts that are not container images. VM golden images are PVCs, not registry objects — see Image library.

​
Traps worth knowing before you hit them

Registries

Registering one, the push probe, and why a health endpoint is not the check that matters.

Charts & the mirror

Resolution order, pinned versions, and the provenance audit for releases installed before a registry was declared.

Platform onboarding

Where a supervisor declares its chart and image sources — before anything is installed.

GitOps & Apps overview

The two routes software takes onto a cluster, and why pull-through is a supervisor-only option on one of them.