Kubernetes CSI with shared FC luns (LVM storage, snapshot as a chain)

[rbremer]

Note: in order to keep the other thread clean (https://forum.proxmox.com/threads/p...h-shared-fc-storage.186319/page-2#post-869245) I have moved the conversation over here.

@bbgeek17 as a reply to your message:

Indeed, this CSI has been built with AI as far as I can tell. It does not modify Proxmox source code, however. When I talked to the author about utilizing the experimental API to implement snap shots he came up with this approach. Not really what I want to use. So I forked it and implemented username/password authentication in order to call the API as root@pve.

Anyway, the CSI itself works fine (except the snapshotting) and roughly does that:

1. a K8s PVC claim gets created, referencing the Proxmox Shared LVM CSI
2. the CSI created a new volume belonging to VM 9999 (so no VM in theory, in big clusters that fixed number is definitely an issue)
3. the new volume was originally created as RAW but with the patch introduced in PVE (https://lore.proxmox.com/all/20260716103243.61836-1-e.huhsovitz@proxmox.com/) it can now created it as a QCOW2 volume
4. as soon as a POD referencing the PVC gets scheduled on a K8s node, the volume will be hot-added to the corresponding virtual machine in PVE
5. when the POD gets evicted, the volume will be hot-removed from the corresponding virtual machine in PVE

So basically, the PV is an independent volume hanging around on the Proxmox LVM storage, not associated with any VM. As soon as its needed its attached and can be used by the POD and will be detached if no longer necessary (hence supporting POD scheduling on a different node). As long as all K8s nodes are in the same PVE cluster and the storage is shared it works nicely.

When it comes to snapshot though, the independent volume needs to be snapshotted. At that time in can be either attached or not attached to a K8s node's VM. Even if it would be attached a snapshot of a single volume should not perform a full backup of the K8s node its attached too. So the only way is to call the Storage API to copy/clone a volume, which on a snapshot-as-a-chain LVM should create an QCOW2 snapshot chain. This API is unfortunately a NO-OP at the moment, doing nothing and just returning with SUCCESS.

Do you have any better recommendations for a K8s CSI which supports shared LUN storage on PVE? My research only brought up this CSI.

 

[bbgeek17]

The design is what worries me. Disks owned by a dummy VMID and moved between VMs by editing configs through the API. Clever, but it does not hold up when things fail.

Snapshots: that endpoint is a copy, not a snapshot. It won't give you a chain, and PVE has nothing today that snapshots a single volume outside a VM.

Not trying to be dismissive... but we are not going to spend time debugging AI-generated drivers anymore. Our support team has taken a few calls from community members, trying to help them. Every time it's the same thing: clean code, good docs, and an architecture built on assumptions that fall apart in the real world. AI has lowered the bar for writing correct syntax, but writing the code was never really the hard part. Getting the architecture right isn't something you can easily generate. It's a bit scary, honestly, because the happy path now "just works" with minimal understanding of the system as a whole.

That said, I don't know of a reliable CSI driver that orchestrates storage through Proxmox. The two I'm aware of share the same design. FC is the tricky one, since nothing in the guest can reach it. If your array does iSCSI or NVMe/TCP well enough, use your vendor's supported driver inside the guest. That's how we run k8s on PVE in production: iSCSI/NVMe in the guest, VLANs for network isolation, and a separate storage management domain per tenant for API, quota, and credential isolation.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[rbremer]

Thank you for your input.

When there are no alternative options you start grasping at straws. Kinda the situation we are in with K8s on PVE right now. I agree, it ain't ideal but that doesn't change the fact that SANs are pretty common in enterprise environments and solutions for permanent storage in Kubernetes are more tailored towards NAS (on-prem).
So indeed I was hoping to find a good CSI driver for our situation, but that seems to be quite hard at the moment.

Regarding the design, I am not really sure where the issues would lie "if things fail". The official Proxmox VE API is used to mount a disk to an existing VM, so thats pretty standard to me. I understand the limitation of SCSI being used, so we have a finite amount of disks attachable this way. It doesn't require any software to be installed on the K8s nodes, so everything is handled by the hypervisor. If a disk is mounted to one node, it can't be mounted to another node at the same time, PVE will prevent that.
So IMHO it is indeed a quite clever design, suitable for smaller K8s environments.

Since I am at a dead end right now I will look into software driven solutions a bit more. Longhorn might be something we could implement. It just adds a new layer of complexity, with replication and stuff, which I was hoping to prevent.