LINSTOR

LINSTOR is a software-defined backend that can serve a supervisor two quite different kinds of storage from one installation: replicated local disks over DRBD, and a shared array LUN that every node sees. Both appear as ordinary StorageClasses, and the choice between them is a performance decision rather than an administrative one. The shared pool is the interesting one: there is no replica and no copying, because every node already sees the same device. A volume follows its pod across the cluster, and — since September 2026 — two nodes can hold it at the same time, which is what a KubeVirt live migration needs. Where the local pool gives each node its own disk, this is one disk with three views of it.

​

Installing it

On the wizard’s Storage step, choose the Software-defined family and then the LINSTOR row that matches what you want — with DRBD for replicated local storage, without it for plain node-local volumes. The form asks for:

• Storage pool — the name of the LVM pool satellites carve from.
• Disks — picked from the live inventory, stored as <node>:<by-id path>. Always by-id, never /dev/nvme0n1: a kernel name is a different disk on every node and can move to another one after a reboot.
• Replicas — how many nodes hold a copy. 2 survives one node; 1 is not replicated at all. 2 or more requires enterprise NVMe with power-loss protection — on consumer drives every replicated write waits for flash on two nodes.
• Shared array LUN (optional) — one LUN masked to every node, which creates the second pool and its class.
• Default StorageClass — none, local or shared. One value rather than two switches, because a cluster can have only one default.

Replication rides the network marked storage in the cluster’s own configuration, so the wire is chosen once and every backend that cares finds it there. Where no such role is declared it falls back to the node address, which is a working network but usually the wrong one — a 1500-MTU VLAN rather than the 9000-MTU storage fabric.

​

Which class for which workload

On enterprise NVMe, replicated local storage and the array are close; on consumer NVMe, replication is not usable. The comparison below uses one recipe everywhere — the platform’s e2e disk benchmark: 4k random I/O, queue depth 16, one job, a pre-filled 8 GiB file, 30 s after a 5 s ramp, direct, on ext4 — run inside Windows and Ubuntu VMs and, with the identical fio command, in a pod. Replicated local storage is linstor-drbd with two copies on Samsung PM983 — enterprise NVMe with power-loss protection (pietraszka); the array is linstor-san on a Hitachi VSP over Fibre Channel (talos-waw). Measured 2026-09-24 to 2026-09-26 on quiet clusters (no DRBD resync running): Inside a VM the two land in the same place — the virtual disk path is the narrower part, and a Linux VM on the replicated class measured the same write rate as the Windows one. Only in a pod, where the path is short, does the array’s cache show: it takes writes faster than two NVMe drives and a network round trip can. For VMs, replicated local storage on enterprise drives is a full equal of the array; for write-heavy pods the array is ahead. The two sides are different clusters, so this compares storage classes as deployed, not media in isolation. On talos-waw one node (talos-p7u-3m2) serves the array at half the rate of the other two for reasons still under investigation; the figures above use the other two. The full table — including vSAN on vSphere measured the same way, and the pod runs — is on Storage performance. Consumer NVMe under DRBD — for information only. The same pod benchmark on linstor-drbd with two copies on Samsung 990 PRO (consumer, no power-loss protection, talos-waw) wrote at about 1/50 of the enterprise figure, with a write p99 of 30 ms against 0.49 ms, while its reads were the fastest of any class measured. One run, not a baseline — but it is the reason for the requirement at the top of this page. The mechanism is why the drive class matters and not just the network. DRBD acknowledges a write only once the peer has it on stable storage, and it issues cache flushes to make sure of that. An enterprise drive with power-loss protection completes a flush from its protected cache; a consumer drive has to put the data on flash — every write, on both nodes. Reads never flush, which is why the same drives look excellent on reads and unusable on replicated writes. So:

• Replicated local storage (linstor-drbd, 2+ copies) — only on enterprise NVMe with PLP. On such drives it is the right place for data that must survive a node.
• etcd, database commit logs, anything that fsyncs and waits → linstor-san, or linstor-drbd on PLP drives. One writer waiting on one acknowledgement is where a protected cache wins.
• Image stores, caches, read-mostly datasets → either class; reads are served locally at near-full speed even with replication.
• Consumer NVMe → single-copy pools and scratch data only.

​

Live migration on the shared pool

A VM on a shared array LUN can live-migrate, and getting there meant removing the layer that was assumed to enable it. DRBD exists to replicate a volume between nodes. On a shared LUN there is nothing to replicate — every node already sees the same device — so the layer is not merely redundant: it is what forbids the operation the pool exists for. LINSTOR refuses to activate a DRBD resource in a shared pool on two nodes, and live migration needs exactly that, because for a moment the source and the target both have the disk open. Measured on one cluster, same pool, same class shape, one parameter apart: With the second, a real VM migrated host to host in 22 seconds. The shared class ships layerList: storage for this reason, alongside placementCount: 1 — every replica would land on the same LUN, so a second copy buys nothing and costs capacity. Three things have to be true before a shared-LUN volume allows it, and the symptom changes as each is fixed — which is why this looked like three different bugs: All three ship in the chart from 0.8.0. A supervisor that predates it keeps the DRBD layer on volumes already created — StorageClass parameters are immutable, so the class must be recreated and goldens re-imported before the volumes built from them can migrate.

​

Snapshots

Both pools can snapshot — LINSTOR reports SupportsSnapshots=true for the thin local pool and the thick shared one alike — and the chart creates a VolumeSnapshotClass for each: linstor-drbd-snap and linstor-san-snap. Neither is marked as the cluster default, so name the class explicitly on a VolumeSnapshot. What a thick pool cannot do is grow, which is why allowVolumeExpansion is off on the shared class and on regardless of snapshot support. The controller that serves those classes installs with the backend — see PVCs & snapshots.

​

What LINSTOR says about itself

The status panel reports two views side by side, and the distinction is the point: what Kubernetes sees (are the operator’s pods running) and what LINSTOR sees (are the satellites connected, do the pools have space, is the controller filing errors). A healthy set of pods says nothing about whether a satellite lost its storage network. From LINSTOR itself: each node with its type and connection status, each storage pool with driver, total and free bytes, whether it supports snapshots and whether it is shared, the resource count, and the controller’s error-report count with the newest one and its timestamp. A controller unreachable from the platform is reported as exactly that, rather than as zero of everything. That error count is worth watching on its own — the recurring LINSTOR failures in this platform have all announced themselves there first, minutes before anything else noticed. The supervisor’s Storage tab goes further for day-2: which interface each satellite is controlled over, the replication path and DRBD state for every node pair, the storage NIC’s MTU and VLAN per node, what listens where, and a Run check that dials every peer from the host network — and on the control interface from a pod — and pings at full frame size.

​

Monitoring

LINSTOR is a row in the monitoring step’s scraper catalogue: applying it installs a PodMonitor against the controller in piraeus-datastore, plus alert rules and a dashboard. Where that namespace does not exist the row reports skipped with the namespace named — not a failure, and not silence either.

​

What commonly goes wrong

A satellite is controlled over the wrong network. The controller should reach every satellite on its node address (default-ipv4), and DRBD should replicate over the storage network — two channels, pinned separately. The Storage tab’s Control plane table shows the interface each satellite is connected on right now and flags any that differ; linstor node list shows the same. This page said the opposite — that every satellite should be controlled over the storage network — until linstor-storage 0.10.2 (September 2026). The storage NIC is deliberately not a CNI device, so a pod’s path to it is not a supported one, and on talos-waw it broke for a reason that had nothing to do with storage. The controller — a pod — had reached all three satellites over their storage addresses for 39 hours without a single drop. At 11:10:41 on 2026-09-25 an unrelated NetworkPolicy with ipBlock exceptions was created in another namespace; Cilium 1.20.1 then stamped replies crossing the geneve tunnel with a node-local CIDR identity truncated to 24 bits, which the receiving node read as its own host identity and dropped as INVALID_IDENTITY. The first drop is on the flow record 1.5 seconds after the policy appeared, and the controller fell back to the node address for that satellite 8 seconds later. The chart now pins the control connection to the node address on every apply; replication is pinned to the storage wire by the node-connection paths (linstor node-connection path list <a> <b>) and PrefNic, and runs from the host network, where none of this applies. The reachability check dials every satellite on the control interface from a pod, which is the vantage point that failed. The controller files an error report every minute. LINSTOR 1.35.1 and later export the whole database on a schedule, defaulting the path to /var/lib/linstor — which the operator mounts read-only. The export serialises the entire database under a global write lock before discovering it cannot write, then reschedules. The chart provides a writable volume for it; if you see this, the release predates that fix. The shared pool “does not carry the declared shared-space id”. Read which of the two messages you got, because they sound alike and mean opposite things. What matters is not which id the members carry but whether they all carry the same one — a pool built by hand may carry an LVM volume-group UUID where the chart declares the LUN WWID, and both name the same device. All members agreeing under a foreign id is cosmetic: LINSTOR knows they are one backing store, which is the whole purpose of the field. It costs exactly one thing — a second LUN added under the same pool name could not be told apart. Members disagreeing is the dangerous case. LINSTOR then treats them as separate pools and will happily place two replicas of one volume on one LUN and call that redundancy. The check counts distinct ids and says which case you are in. Disks appear to move between reboots. They do. Kernel names are assigned in probe order, so /dev/nvme0n1 can be a different device after a restart and the 4 TB disk can trade places with the system disk. Everything the platform stores — pool devices, wipe targets — uses /dev/disk/by-id, and anything you write by hand should too. A volume will not schedule where you expect. With allowRemoteVolumeAccess off, a pod must run on a node holding a copy. With it on, a pod can use a volume whose data lives elsewhere, at the cost of reading across the network. Necessary for RWX block and live migration — and, on the shared pool, not sufficient: believing it is the lever there costs an afternoon, because it was already on while migration still failed. See Live migration on the shared pool. A VM on the shared pool will not live-migrate. The target pod sits in Init and the migration stays Scheduling, while the driver answers Cannot deactivate a resource while being in use or Activating resource on two nodes of a shared storage pool is not possible. All three conditions under Live migration on the shared pool have to hold — and the third, the golden image’s own class, is the one that survives a correct-looking StorageClass. Replicated writes are slow on a healthy cluster. Check the drive class before anything else. A two-copy linstor-drbd volume on consumer NVMe wrote at about 1/50 of the same class on enterprise drives, with a 30 ms write p99 — and every resource still reported UpToDate and every connection Connected. Nothing is broken; the drives cannot acknowledge a flush from cache.

​

Related