pve-fcsan: vCenter-style Fibre Channel SAN management and array provisioning for Proxmox VE (beta)

[tward]

Hi all,

I've published pve-fcsan, an open-source (AGPL-3.0) plugin that brings the vCenter SAN workflow to Proxmox VE 8 and 9 for Fibre Channel (and iSCSI) block storage.

What it does

• A SAN / Fibre Channel panel under Node → Disks: HBAs, targets, LUNs, and multipath state and paths for every node in the cluster
• Cluster-wide rescan, multipath aliases and path policies, and cleanup of dead paths
• Shared-LVM datastores you can create, grow, rename, unmount, reclaim and delete from the GUI, the pvesan CLI or the API
• Optional array integration: create the volume on the array, map it to the cluster, rescan every node and create the datastore from one form. It can also grow and delete volumes on the array.

Supported arrays (provisioning)
IBM FlashSystem / Spectrum Virtualize, Pure Storage FlashArray, INFINIDAT InfiniBox, Dell PowerStore, Dell PowerMax / VMAX, NetApp ONTAP, NetApp E-Series, Hitachi VSP, Fujitsu Eternus

Status: beta
There are 293 automated tests, and the plugin runs on a 6-node Proxmox VE 9.2 cluster with QLogic and Emulex HBAs. The array drivers are written against the vendors' API references and SDKs, but none has been tested against a real array yet. If you have one of these arrays in a lab, please try it with a small test datastore and report what works and what doesn't (with the task log) using the issue template on GitHub. That's the most useful help right now.

Install (on every node)

wget https://github.com/warddynamiccomputing/pve-fcsan/releases/download/v1.3.0/pve-fcsan_1.3.0_all.deb
apt install ./pve-fcsan_1.3.0_all.deb
/usr/share/pve-fcsan/pvesan-integrate status

Then reload the web GUI (Ctrl+F5) and open Node → Disks → SAN / Fibre Channel.

Good to know
Proxmox VE has no plugin hook for new API paths or GUI panels, so the package adds one marked line to each of two pve-manager files (PVE/API2/Nodes.pm and index.html.tpl). A dpkg trigger re-applies it after pve-manager upgrades, and removing the package removes both lines. This isn't supported by Proxmox Server Solutions, so check pvesan-integrate status after upgrades.

Links
Repo: https://github.com/warddynamiccomputing/pve-fcsan
Release: https://github.com/warddynamiccomputing/pve-fcsan/releases/tag/v1.4.0
Runbook: https://github.com/warddynamiccomputing/pve-fcsan/blob/main/docs/pvesan-Runbook.md
How it works: https://github.com/warddynamiccomputing/pve-fcsan/blob/main/docs/pvesan-How-It-Works.md

Feedback, bug reports and pull requests are welcome.

 

[LnxBil]

Nice idea, have you tried to get it into PVE instead of just programming another thing that patches PVE on the fly and can potentially break everything? That is always the dealbreaker for me. I read through the code, so it's just a small change necessary, but still. Hopefully Proxmox can chime in here how to solve this properly.

I have QLogic (we don't use emulex with Linux) and different incarnations of Fujitu Eternus laying around to test this out (if I get the time).

 

[tward]

Nice idea, have you tried to get it into PVE instead of just programming another thing that patches PVE on the fly and can potentially break everything? That is always the dealbreaker for me. I read through the code, so it's just a small change necessary, but still. Hopefully Proxmox can chime in here how to solve this properly.

I have QLogic (we don't use emulex with Linux) and different incarnations of Fujitu Eternus laying around to test this out (if I get the time).

Thanks for reading through the code. That's a fair point, and it was the part I liked least too.

I haven't tried to get it into PVE yet. I wanted it working and in front of people first, but upstream is where this belongs. The storage side already uses the official custom storage plugin interface (PVE::Storage::Custom), so nothing is patched there. The patch exists only because pve-manager has no way for a third-party package to register API paths or load a GUI script. The clean fix would be a small hook in pve-manager, for example loading extra API modules and GUI scripts from a directory the way storage plugins already work. I'd rather propose that on pve-devel than keep patching. If someone from Proxmox reads this: would you accept something like that, or is there a way you'd prefer?

Your post did make me fix the biggest problem with the current approach. Until 1.3.0, the line in Nodes.pm loaded pve-fcsan unguarded, so if its API module ever failed to load, pveproxy and pvedaemon would have gone down with it. As of 1.3.1:

• The line loads pve-fcsan inside an eval. If it fails, only its own /nodes/{node}/san API and GUI panel are missing, the journal says why (pve-fcsan: SAN API disabled - ...), and the rest of the API keeps working.
• The GUI override catches its own errors, so a broken pvesan.js only loses the SAN panel.
• I tested it on my cluster by removing the API module and restarting pveproxy and pvedaemon. Both came back up, the GUI and the rest of the API answered, and only /san was gone. There's also a unit test that loads the patched Nodes.pm with a deliberately broken module.

The other safeguards are unchanged:

• It adds one marked line to each of two files, after a fixed anchor. If the anchor isn't found (for example after a pve-manager change), it refuses to touch the file.
• The edited Nodes.pm is compile-checked and rolled back if it doesn't compile, and the file is written atomically.
• A dpkg trigger re-applies the line after pve-manager upgrades. pvesan-integrate remove or removing the package takes both lines out again.
• pvesan-integrate status shows whether it's applied and whether pve-fcsan's API actually loads.

So it's still a patch, but a broken pve-fcsan can no longer take PVE down with it.

On testing: QLogic is a good fit. That's what most of my test cluster has, so the HBA side is covered. What's missing is time on real arrays. There's no Eternus driver, but everything except creating volumes on the array works with any FC array: discovery, rescan, multipath, and creating, growing and renaming datastores. So even that would be a useful test. If your Eternus models have a REST API, tell me which ones and I'll look at writing a driver. Even a partial report helps; there's an issue template on GitHub for it.

 

[LnxBil]

Your post did make me fix the biggest problem with the current approach.

Glad to be your rubber duck.

There's no Eternus driver, but everything except creating volumes on the array works with any FC array: discovery, rescan, multipath, and creating, growing and renaming datastores. So even that would be a useful test. If your Eternus models have a REST API, tell me which ones and I'll look at writing a driver. Even a partial report helps; there's an issue template on GitHub for it.

I can see that some need this, for me, I only get LUNs and work with them, so I have no need to have some kind of driver for this. A quick AI ask told me that there is an API that can be used, which runs on a port that I never knew. Have to check if it's true. AFAIK the older eternus are somewhat dead and I only have access to one currently supported module, the DX200S5, never laid fingers on the bigger modules. I've a lot on my plate right now, but it's on my list to try.

 

[cfgmgr]

This is awesome! I love me some Fiber Channel!

How does the shared LVM work since PVE has no VMFS? Looks like it locks it to a single node?

Looks fantastic regardless.

 

[tward]

This is awesome! I love me some Fiber Channel!

How does the shared LVM work since PVE has no VMFS? Looks like it locks it to a single node?

Looks fantastic regardless.

Thanks! No, it isn't locked to one node. Every node uses the datastore at the same time.

It's Proxmox's own shared LVM, the same thing you get from the built-in LVM storage with "shared" ticked. pve-fcsan sets it up and keeps it consistent across the nodes. It works without a cluster filesystem like VMFS because nothing ever writes to the same blocks from two nodes:

• One LUN, one volume group, visible on every node. Each VM disk is its own logical volume inside it.
• Metadata changes are serialized by the Proxmox cluster lock (pmxcfs). Creating, resizing or deleting a disk happens on one node at a time, so the LVM metadata can't be corrupted by two nodes at once. No clvm or lvmlockd is needed.
• A disk is active only where its VM runs. Proxmox activates the logical volume on that node. On a live migration it's deactivated on the source and activated on the target. HA works the same way.

So the lock is per disk, not per datastore. Ten VMs on ten nodes can all run from the same LUN.

The trade-offs compared with VMFS:

• Disks are thick logical volumes. There's no thin provisioning at the LVM layer, but the LUN itself can be thin on the array.
• Snapshots: on PVE 9, shared LVM supports snapshots as volume chains (qcow2 on LVM). Before 9 there were none on shared LVM.
• You don't get VMFS's file-level view. A VM disk is a block device, not a file on a shared filesystem.

What pve-fcsan adds on top is the FC side: discovering the LUNs, a consistent multipath setup on every node, and checks that every node actually sees a LUN before it becomes a datastore. With a supported array it can also create the volume and map it to the cluster in one step.

 

[tward]

Glad to be your rubber duck.


I can see that some need this, for me, I only get LUNs and work with them, so I have no need to have some kind of driver for this. A quick AI ask told me that there is an API that can be used, which runs on a port that I never knew. Have to check if it's true. AFAIK the older eternus are somewhat dead and I only have access to one currently supported module, the DX200S5, never laid fingers on the bigger modules. I've a lot on my plate right now, but it's on my list to try.

Thanks, that's useful, and your AI was right. The ETERNUS DX S5 / AF S3 series have a RESTful API on HTTPS port 5665, with controller firmware V11L40 or later. It's off by default: enable it in the Web GUI or CLI and open port 5665 in the array's firewall settings. An upgrade from older firmware closes that port.

On your workflow first: if you get LUNs and work with them, you don't need a driver at all. Add no array, and pve-fcsan discovers the LUNs, sets up multipath and builds the shared LVM datastores the same way. The drivers only add one-step "create volume, map, rescan, datastore", plus grow and delete. I've made that clearer in the README.

For the DX200 S5: I've released 1.4.0 with a fujitsu-eternus driver built on that API, following Fujitsu's RESTful API Reference.

• The API can't create host groups, so it registers each node's HBA port as its own FC host and connects each volume to all of them with the same host LUN.
• The CA ports zoned to the cluster need host affinity enabled.
• It refuses to map if one of the cluster's hosts shares a LUN group with a host outside the cluster.
• Volumes go into a thin provisioning pool.

Release notes: https://github.com/warddynamiccomputing/pve-fcsan/releases/tag/v1.4.0
Install steps for the apt repo: https://warddynamiccomputing.github.io/pve-fcsan/

Once the repo is set up, it's just:

apt update && apt install pve-fcsan

It hasn't touched a real ETERNUS yet. Whenever you get to it, a 10 GiB test datastore (provision, grow, delete) on a non-production pool would tell us a lot. If anything fails, the task log shows the array's own error code, and an issue with that log is all I need.

No rush, I know the feeling of a full plate.