> ## Documentation Index
> Fetch the complete documentation index at: https://docs.celum.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Registry mirroring in depth

> Charts and images travel separately, over four channels, to five consumers — what reaches a guest cluster, what does not, and which half of a mirror is Harbor-specific.

A declared mirror is not one setting. It is **two kinds of artefact** travelling over **four channels** to **five consumers**, and each combination takes effect at a different moment. Most mirror problems are not "the mirror is broken" — they are one combination never having been wired, or having been wired *after* the thing that needed it was already installed.

[Registries](https://docs.celum.io/gitops/registries.md) covers registering one and why *reachable* is the wrong check. This page maps everything downstream of that: supervisor, guest, node and controller.

## The two artefacts do not travel together

Knowing which one you are looking at saves most of the debugging.

| | **Charts** | **Images** |
| - | - | - |
| what carries it | a `repoURL` written into a `HelmRelease` or `HelmChartProxy` | `machine.registries.mirrors` in a node's containerd config |
| who reads it | helm-controller, CAAPH, capvc | containerd, on every node |
| when it is decided | **at apply time**, once | **at boot time**, once per machine |
| changing it later | a re-apply of the release rewrites it | needs a **new machine** |
| a wrong value looks like | `not a valid chart repository` | a pull that crawls, or `ImagePullBackOff` |

The asymmetry in the last two rows is the one that costs afternoons: **a chart can be re-pointed, a node cannot be re-taught.** A `HelmRelease`'s `sourceRef` is written once and nothing revisits it, so a chart installed before a registry was declared keeps pulling from upstream forever while everything applied afterwards comes from the mirror. The remedy is a re-apply, and it has a button — see the provenance audit on [Charts & the mirror](https://docs.celum.io/gitops/charts.md).

## The five consumers

```text theme={null}
                           charts                    images
supervisor nodes             —                  Omni ConfigPatch
supervisor workloads    HelmRelease             (via their node)
guest nodes                  —                  ClusterClass strategicPatch
guest workloads         HelmChartProxy          (via their guest node)
provisioning controllers  their own repoURL     (via the supervisor node they run on)
```

**Supervisor nodes** take their mirror from a Celum-owned Omni ConfigPatch. Talos applies it live — containerd re-reads its host config, no reboot and no drain.

**Guest nodes** take theirs from the ClusterClass, as a strategic patch built from the `registryMirrors` and `registryTrust` variables. Every guest family works this way — `talos-kubevirt`, `talos-vsphere` and `kamaji-kubevirt` alike.

> [!NOTE]
>
> **Guest clusters are never registered with Omni.** They do not talk to it and it does not manage them, which is why their mirror arrives by a completely different route from the supervisor's — even though the YAML that lands on the node is identical.

**Guest workloads** pull through their guest node, so they inherit whatever that node was booted with. **Provisioning controllers** — CAAPH fetching add-on charts, capvc fetching the vcluster chart — run as pods on the supervisor and pull through *its* nodes.

## Two different things are called "the mirror"

This is the distinction most likely to mislead, because both are true and they contradict each other on the surface.

> [card] App-catalog pull-through
>
> **Supervisor only.** Stocks the registry at install time and wires its pull secret onto a supervisor `HelmRepository`. A guest install pulls with the guest's own credentials, so there is nothing to attach the secret to — and the request is **refused** rather than silently ignored.

> [card] Platform chart sources
>
> **Reaches guests.** The add-on charts a guest cluster is built with — CNI, the telemetry agent, the metrics agent, the cloud-controller-manager — resolve through the *supervisor's* declaration and are written into each `HelmChartProxy` as an `oci://` URL.

So "my guests pull charts from the mirror" and "pull-through is supervisor-only" are both correct: they are different mechanisms with different owners. The first is a per-install option on a catalog app; the second is platform onboarding state that every guest inherits.

## What a guest inherits, exactly

A guest declares nothing of its own. It inherits the supervisor's declaration — deliberately, because two places describing one registry are two places that can disagree.

| add-on | inherits the mirror | note |
| - | - | - |
| CNI (Cilium) | yes | |
| Cluster agent | yes | image **and** chart |
| Metrics agent (Prometheus) | yes | pinned; an unpinned chart makes the client fetch the whole repository index to discover "newest" |
| Log agent (Alloy) | yes | |
| Talos cloud-controller-manager | yes | not installed on hosted-control-plane guests, which get their provider ID from the infrastructure provider |
| vSphere CSI | **no** | its chart is not catalogued, so the key is deliberately withheld |

> [!WARNING]
>
> **Only catalogued charts are redirected, and that guard is the point.** Chart resolution takes the mirror branch whenever a default registry exists — including for a name the catalogue has never heard of, where the entry is empty. Emitting such a key would point the reconciler at an `oci://` URL with no chart behind it. An absent key means "leave the upstream baked into the file", which is also the safe answer for a second reason: pointing a guest at a registry whose **nodes** do not trust the certificate turns a working public pull into an x509 failure.

## What Harbor specifically gives you

The node-side mirror is a **pull-through proxy cache**: the image reference stays `quay.io/…`, and containerd is told to fetch it from a project on your registry that proxies that upstream.

```yaml theme={null}
machine:
  registries:
    mirrors:
      quay.io:
        endpoints:
          - https://registry.example.com/v2/proxy-quay
        overridePath: true
```

The pairs are **declared by the operator**, not hardcoded: each entry maps one upstream host to one proxy project. The endpoint URL is always `https://<mirror host>/v2/<project>`, built by a single function so the shape cannot drift between the supervisor channel and the two guest channels.

> [!NOTE]
>
> **`overridePath: true` is what makes it transparent.** Without it containerd appends the upstream's own path to the endpoint and the request misses. With it, the proxy project is the whole path — which is why the mirror needs no change to any image reference anywhere.

A mirror that is warm serves at local-network speed. A cold entry costs whatever the upstream costs, once, and is warm afterwards — so the first build on a new estate is not representative of the second.

## What is supported, and where

> [tab] Any OCI registry
>
> These paths speak the standard distribution API and work against any conforming registry:
>
> * pulling and pushing charts as OCI artefacts
> * copying images into the mirror (`prepull`)
> * the **push probe** behind **Test**, which starts a blob upload and cancels it
> * the node-side `mirrors:` declaration, which is containerd's, not a vendor's

> [tab] Harbor-specific
>
> Three things assume Harbor and will not do anything useful elsewhere:
>
> * the **proxy-cache project** itself — pull-through caching is a Harbor feature, and other products express it differently or not at all
> * the **health endpoint** consulted when a registry is registered (and it answers for anonymous and for wrong credentials, which is why Test exists)
> * listing **cloud-init template repositories**, which reads Harbor's project/repository API

A registry row carries a `type`, and it defaults to `harbor`. Nothing in the platform branches on that value today: the field records intent rather than selecting behaviour. Treat Harbor as the product the platform is built and tested against, and a plain OCI registry as usable for everything except pull-through caching.

## Credentials: the two ways they surprise you

> [!WARNING]
>
> **Registry rights are per PROJECT, not per registry.** A robot account can hold push on the project that keeps charts and only pull on the project that keeps images, and a credential table that records one verb per host will read as "cannot push" when it pushes charts perfectly well. Ask the registry's token service for a push scope **on the specific project** rather than inferring one answer for the whole host.

> [!WARNING]
>
> **The mirror credential does not reach containerd through the `mirrors:` block.** Measured across three configurations — with and without a `mirrors:` entry, against a private and then a public project — every one pulled **anonymously**. The CA under the same host key worked in all of them. So a proxy project that requires authentication will fail on the node even when the credential is correct everywhere else; the working shapes are a project that allows anonymous read, or a pull secret delivered to the workload.

The mirror credential is matched **by host** against the registered registries — so register the mirror first. An unmatched host silently yields an anonymous pull, which works on a registry whose proxy projects allow anonymous read and fails on the first one that does not.

## Filling the mirror is a separate act from declaring it

Declaring where artefacts come from does not put them there. Two operations do that, and they are separate because charts and images travel differently.

| | Charts | Images |
| - | - | - |
| what it copies | every chart in the platform catalogue | every image needed to bring a cluster up |
| API | `POST /api/registries/{id}/mirror-charts` | `POST /api/registries/{id}/prepull` |
| progress | `GET /api/registries/mirror-charts/{jobId}` | `GET /api/registries/prepull/{jobId}` |
| inventory first | — | `GET /api/registries/required-images` |

Both are long-running and report through a polled job. The chart mirror walks the **platform chart catalogue**, so a chart that is not catalogued is not copied — which is the same guard that keeps a guest from being pointed at a chart no registry holds. Cataloguing a chart is therefore what puts it in that dialog; there is no separate list to maintain.

> [!NOTE]
>
> A chart is also stocked on demand at install time — but only when the stock check **succeeded and found it missing**. On a read error the platform deliberately does not push, and lets the reconciler pull instead. That asymmetry exists because a read failure once forced a write, and a pull-only credential then turned a recoverable "version not found" loop into a hard install failure.

## Three surfaces

| Task | UI | API | MCP |
| - | - | - | - |
| List / register a registry | Settings → Registries | `GET`/`POST /api/registries` | `list_registries`, `create_registry` |
| Update one | Settings → Registries | `PUT /api/registries/{id}` | `update_registry` |
| Delete one | Settings → Registries | `DELETE /api/registries/{id}` | **none** — deleting needs a person |
| Push probe | **Test** | `POST /api/registries/{id}/test` | `test_registry` |
| Chart-side probe | **Test charts** | `POST /api/registries/{id}/test-charts` | `test_registry_charts` |
| Mirror the chart catalogue | **Mirror charts** | `POST /api/registries/{id}/mirror-charts` | `mirror_registry_charts` |
| Pre-pull images | **Pre-pull** | `POST /api/registries/{id}/prepull` | `prepull_registry_images` |
| Required-image inventory | shown in the pre-pull dialog | `GET /api/registries/required-images` | — |
| Declare a supervisor's chart/image sources | Onboarding → registry sources | `GET`/`PUT /api/supervisors/{name}/onboarding/registry-sources` | `get_`/`set_onboarding_registry_sources` |
| Declare the node mirror | Onboarding → image registry | `GET`/`PUT /api/supervisors/{name}/onboarding/image-registry` | `get_`/`set_onboarding_image_registry` |
| Did the nodes actually get it | Onboarding → provenance | `GET /api/supervisors/{name}/onboarding/registry-provenance` | `get_onboarding_registry_provenance` |
| Where does a chart resolve from | mirror dialog | `GET /api/charts/mirror-status?supervisor=…` | `get_chart_mirror_status` |

> [!WARNING]
>
> **`set_onboarding_image_registry` returning 200 does not mean the nodes were told.** The declaration is saved on the supervisor resource first, then rendered into the ConfigPatch that Talos applies. The second half is best-effort: with the machine-management API unreachable the call still returns 200 with the declaration stored. Confirm with the provenance read, which compares *declared* against *actually applied* and reports its read failures instead of hiding them.

## Permissions

| Route | Action | KRN |
| - | - | - |
| `GET /api/registries` | `registry:List` | `krn:celum:registry:*` |
| `POST /api/registries` | `registry:Create` | `krn:celum:registry:*` |
| `GET /api/registries/{id}` | `registry:Get` | `krn:celum:registry:{id}` |
| `PUT /api/registries/{id}` | `registry:Update` | `krn:celum:registry:{id}` |
| `DELETE /api/registries/{id}` | `registry:Delete` | `krn:celum:registry:{id}` |
| `POST /api/registries/{id}/test` | `registry:Test` | `krn:celum:registry:{id}` |
| `POST /api/registries/{id}/prepull` | `registry:Prepull` | `krn:celum:registry:{id}` |
| `POST /api/registries/{id}/mirror-charts` | `registry:Prepull` | `krn:celum:registry:{id}` |

Mirroring charts and pre-pulling images share `registry:Prepull` — both are the same privilege in substance: writing into a registry on the platform's behalf.

## What it is worth

Measured on one estate, the same guest cluster built twice — once fetching across the network, once through a warm local mirror:

| | before | mirrored |
| - | - | - |
| kubelet image, fetched by the OS before Kubernetes exists | 57.2 s | **3.0 s** |
| whole node boot sequence | 67.7 s | **12.1 s** |
| cluster agent, 84 MB | 22.2 s | **2.4 s** |
| a 29 MB system image, from its public upstream | 165 s | **22 s** |

The third row is not an upstream comparison: that image already came from an internal registry, just one on the far side of a site-to-site link. Mirroring it locally was worth 9× anyway, which is the argument for treating "internal" and "local" as different words.

The first row is the one worth internalising: it happens before the cluster exists, so no in-cluster setting can influence it. Only the node's own containerd declaration can.

The largest single saving was not an image at all. A chart left **unpinned** made the client download the entire repository index — tens of megabytes — to discover that "newest" meant a 90 KB chart. Pinning it removed that per-guest cost entirely.

## Three things the mirror does not cover

1. **The registry's own certificate.** A mirror on an internal CA needs that CA on the nodes, which is the `registryTrust` half of the declaration. Without it every pull fails x509 and the mirror looks broken when it is merely untrusted.
2. **Anything not in the platform catalogue.** Third-party charts named by an app definition carry their own repository URL and are not redirected.
3. **Artefacts that are not container images.** VM golden images are PVCs, not registry objects — see [Image library](https://docs.celum.io/vms/image-library.md).

## Traps worth knowing before you hit them

| Symptom | Cause |
| - | - |
| Nodes pull from the internet despite a mirror | the node declaration is per supervisor and separate from the registry catalogue; and an existing node cannot be re-taught — only a new one boots with a new mirror |
| Some charts come from the mirror and some do not | the ones from upstream were installed **before** the registry was declared; a release's source is written once |
| The registry tests reachable and mirroring fails | the credential is pull-only on the project being written to. Read the push result, not the reachability |
| A guest ignores the mirror for one add-on | that chart is not catalogued, so the redirect is withheld on purpose |
| An anonymous pull works in one project and not another | the mirror host did not match a registered registry, so no credential was attached — and the credential would not have reached containerd anyway |
| A component still crosses the network after everything is mirrored | an explicit per-component image override wins over all registry resolution. It is an exact reference, so it is never second-guessed — clear it to let resolution work |

## Related

> [card] [Registries](https://docs.celum.io/gitops/registries.md)
>
> Registering one, the push probe, and why a health endpoint is not the check that matters.

> [card] [Charts & the mirror](https://docs.celum.io/gitops/charts.md)
>
> Resolution order, pinned versions, and the provenance audit for releases installed before a registry was declared.

> [card] [Platform onboarding](https://docs.celum.io/get-started/platform-onboarding.md)
>
> Where a supervisor declares its chart and image sources — before anything is installed.

> [card] [GitOps & Apps overview](https://docs.celum.io/gitops/overview.md)
>
> The two routes software takes onto a cluster, and why pull-through is a supervisor-only option on one of them.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.