Shared LVM over FC SAN

[whiney]

Hello,

We are currently planning a migration from VMware to Proxmox VE for our production infrastructure.

Our current setup includes:

• 8 Dell servers in a cluster
• 2 FC SAN storage arrays used for VM storage and high-availability

For evaluation purposes, we’ve built a test environment with similar (but older) hardware:

• 8 nodes
• Each node has 2x 110GB SSDs in RAID 1 (Proxmox VE installed with EXT4)
• 2 FC SAN arrays connected via FCoE

So far, I have:

1. Installed Proxmox VE on all 8 nodes
2. Created a single cluster with all nodes
3. Configured fcoe-utils, lldpad, and multipath on all nodes
4. Detected FC LUNs and prepared them (pvcreate, vgcreate)
5. Configured a shared LVM storage across all nodes
6. Successfully tested VM live migration and high-availability

The main issue we’re facing now is the lack of snapshot functionality. As far as we understand, snapshots are not supported with shared LVM storage in Proxmox VE.

I would like to ask:
What would be your recommended approach for our type of infrastructure?
Is there a way to keep using shared FC SAN storage while still having access to snapshot functionality and other features (backups, migrations, HA)?

We’re looking for a solution that:

• Works with shared SAN storage
• Supports snapshot and other features (migration, HA)
• Is stable, maintainable, and ideally supported

I'd highly appreciate your insights or recommendations based on how you would approach this scenario.

 

[bbgeek17]

Hi @whiney , welcome to the forum.

What would be your recommended approach for our type of infrastructure?

At this point in time you are using the primary supported option.

Is there a way to keep using shared FC SAN storage while still having access to snapshot functionality and other features (backups, migrations, HA)?

Today, your only option to have all of the above is to use a 3rd party cluster-aware filesystem, i.e. OCFS. I'd recommend researching its functionality, supportability and viability prior to going into production. There are many threads available on this forum regarding this topic.

We’re looking for a solution that:

• Works with shared SAN storage
• Supports snapshot and other features (migration, HA)
• Is stable, maintainable, and ideally supported

You are trying to retrofit infrastructure investment you made. This is your primary limiting factor. There are solutions out there that can help immediately. However, some of them are not free.
There are also other solutions that deliver natively PVE integrated, stable shared storage infrastructure that would be significantly faster than your existing storage.

Cheers

PS you may also find this KB article useful: https://kb.blockbridge.com/technote/proxmox-lvm-shared-storage

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[whiney]

Thanks a lot for your reply.

There are solutions out there that can help immediately. However, some of them are not free

Would you be able to point me in the right direction regarding the solutions you mentioned that "can help immediately"? Some names or examples of these solutions would be very helpful, I am happy to do my own research from there.

There are also other solutions that deliver natively PVE integrated, stable shared storage infrastructure that would be significantly faster than your existing storage.

So far, the only well-integrated solution with proxmox VE that I'm familiar with is CEPH, is that what you were referring to?

PS you may also find this KB article useful: https://kb.blockbridge.com/technote/proxmox-lvm-shared-storage

Thank you, I will make sure to read through it.

 

[bbgeek17]

So far, the only well-integrated solution with proxmox VE that I'm familiar with is CEPH, is that what you were referring to?

Ceph is an example of built-in PVE storage, its capabilities as the default go-to storage in PVE infrastructure are a given. No, I was not referring to Ceph. It is not suitable for your FC infrastructure re-use.

Would you be able to point me in the right direction regarding the solutions

For the solution I am most familiar with, you'd need to hit certain capacity, performance and budget criteria. You can find the link at the bottom of this post.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[guruevi]

As pointed out above, the hardware RAID shared disk storage systems have gone the way of the dodo, even in VMware clusters, software defined storage is what you will get buying anything new these days. Sorry to say, but you’ve been duped by VMware and Dell into buying cheap hardware with an overpriced badge.

What you can do if you absolutely need to retrofit is share out the physical disk (which you would hope are SSD), as a LUN to each node and then use Ceph over them, supposing a fairly recent dual FC link to each node, you should not see a huge performance issue, you don’t need RAID, since Ceph will take care of the 3-way replication.

If your hardware is a little bit more modern (PowerStore or PowerScale - previously Isilon), those are really just Xeon servers under the hood, you could potentially turn them into regular Ceph clusters (actually, that was a recommendation from another SDS sales person, to use the old Isilon as the host for their software).

Snapshot support on FC storage and hardware RAID is dependent on the hardware. If your RAID controller can do snapshots, then you can set that up independent of the host solution.

 

[whiney]

Hi again and thank you for your reply.

What you can do if you absolutely need to retrofit is share out the physical disk (which you would hope are SSD), as a LUN to each node and then use Ceph over them, supposing a fairly recent dual FC link to each node, you should not see a huge performance issue, you don’t need RAID, since Ceph will take care of the 3-way replication.

Could you clarify something for me?
Are you referring to a scenario where the SAN storage array can expose each physical disk as an individual LUN (1:1 passthrough) - without RAID?

I’m trying to understand if that’s what you had in mind - so that Ceph OSDs would run over those individual LUNs, mapped 1:1 to each node, and we’d avoid RAID entirely, trusting Ceph’s replication. However, this doesn't sound like a supported option and should be avoided, if I understand correctly.

At this point, would you agree that simply living without snapshots and using something like PBS for fast backup and restore would be more practical and maintainable approach?

 

[guruevi]

You don’t have a SAN, you have a FibreChannel disk system right now. Unless your FC system can do independently NFS or something to that end.

FibreChannel systems should be able to share out their disks as individual raw LUN and bypass the RAID. As long as you can get a raw disk as LUN, that would be “supported” as it would be the same as an external SAS enclosure. It’s how our old VMware system worked, we’d actually had to share 2 individual disks from a SAN to do RAID1. Every VM got a set of physical disks (no RAID controller, just a disk management plane)

Ceph will take any block device, whether that is SAS, FC, NVMoF. You are going to be limited by the speed of the FC link to each host, but that is already the case regardless of the option you take. Not sure if it is feasible to migrate the disks into the physical hardware. If you have 8 nodes that are Rx40s, take the SSD out of the enclosure and push them into one of the 16-24 front slots.

Snapshots are not backups, you should always have PBS as a backup regardless of what front end solution you use. Snapshots are useful to quickly roll back, I almost never use them, since we have hourly PBS, but you could do eg hourly snapshot and daily backups, as long as your RTO and RPO allows for that. My RPO is 2 hours, so I do hourly backups.

 

[whiney]

Thanks again for your input.

I was thinking more about the idea of Ceph on top of LUNs. Wouldn't one of the main issues be that if all OSDs are LUNs carved from the same physical device, then even though Ceph performs replication, all the replicas would still live on the same physical hardware, right? That would essentially eliminate the redundancy and fault tolerance that Ceph is designed to provide, or am I missing something? Plus it definitely adds another layer of complexity.

I get that it's possible, but is it really practical for production use? I guess I could try to test that out. Thanks. If anyone else has an experience with similar setup, please let me know.

Regarding snapshots - What I meant earlier was more in the sense of using PBS as a safe rollback point before critical operations, like system upgrades or risky changes. So instead of relying on snapshots, I'd trigger a PBS backup beforehand (as PBS has supposedly fast backups that can be used as an alternative to snapshots). I agree that backups are essential and we already have a backup platform - Commvault. However, Commvault has the same problem with LVM, it can only protect VMs with LVM-thin storage type, not LVM, so that's another challenge I need to address in this setup.

 

[guruevi]

Well, yes, but you’ve deployed your VMware in the past with the same reckless abandon to a SPOF, whether it is Ceph or RAID6 or VMware SDS doesn’t make a difference, that’s just software, your hardware is and always has been a SPOF. It will be slightly less so with Ceph, because if you use raw LUN, then you can theoretically pull the disks and put them in any other enclosure and reconstruct your Ceph array, whereas with proprietary stuff you are stuck having to repair or replace with gear from the same vendor. My suggestion is not to pass through parts of the disk, pass through the entire disk(s). So let’s say you have 24 disks, you map 3 disks to each of the 8 nodes completely, using proper FC you should be able to do that, provided no shenanigans in the firmware.

Is it practical for production, depends on your business needs, you have so far relied on it, if that is now unacceptable, you need to buy new hardware or as I said, move the disk into separate enclosures to separate them from the singular enclosure you do have. And as I said, it is very likely possible to just pull the disks, put them in the servers and use them as ‘regular disks’, just have an empty FC array you can sell on eBay.

Commvault has a Proxmox plugin, I haven’t used it myself, Commvault was too expensive ($3/GB) for the scope of our Proxmox environment (~400TB) for small environments it may work.

 

[whiney]

Thanks a lot for the detailed response.

I’ll definitely discuss these options with my colleagues and possibly test some of them in practice. If we try anything interesting or come to any conclusions, I’ll make sure to follow up here.

 

[bbgeek17]

I’m not aware of any major SAN vendors offering solutions that can or want to expose individual disks directly as part of their standard SAN platforms. These systems are typically designed around aggregated, abstracted storage pools, not raw device access.

Some vendors do offer JBOD solutions, but these are typically marketed separately and targeted toward niche use cases. These usually come from ODMs that actually build the SANs for the OEMs.

I get that it's possible, but is it really practical for production use?

IMHO the answer is no. You will likely see "scorn" from Ceph community and little to zero support from the SAN vendor.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[guruevi]

@bbgeek: not entirely sure we’re talking about the same things. There are SAN platforms that are completely proprietary, as I said, they generally run Xeon processors and are “just computers” with Linux/FreeBSD (eg Dell-EMC Isilon) and a software layer. Those can be converted as-is often by plugging in a USB boot drive and finding the VGA/serial port, as long as they have a network card.

Cheaper RAID-FC “SAN” are just simple RAID controllers with a (minimal) web GUI and in some cases (if you buy VMware’s SDS) just FC-to-SAS expanders. I’ve recovered one of the regular RAID systems in the past, propped all the disks in a SAS enclosure, then they can actually be read by Linux software RAID (it detects the disk format and can read the RAID config). It was painful, since disks were physically damaged, but possible.

I’ve just built a Proxmox environment for dev/test doing exactly the same, someone purchased the wrong thing for VMware and then the rates hiked, the project got cancelled and was going to throw out $50k worth of disks, we’re just passing through the disks to Proxmox over FC. We find it acceptable for dr/dev/test environment.

Of course none of this will be ‘supported’ by the vendor, you purchase a package deal with VMware-lock-in. They will honor the warranty on parts but that’s about it. Ceph doesn’t care what fabric you use, people run (supported) Ceph and other SDS in the cloud. We’re deploying Ceph with SAS expanders, using FC to effectively SAS expander is not ideal, but not impossible.

Depends on your business needs and risk acceptance. If you previously accepted the risk on VMware, being better informed you may not want to continue that risk.

Note that for Dell’s Red Hat Apex Hybrid Cloud, they are selling Red Hat + Kubernetes + Ceph on APEX-MC-750 (rebadged R750s) + Connectrix FC switch + PowerStore 1200T FibreChannel shared storage enclosures. Not kidding, this is what Dell sells and supports as a managed offering when you need more storage than cores.