"Real" VMware ESXi Replacement with Shared Storage -> Snapshots

[inbyteswetrust]

Hello Proxmox Community!

We have lots of costumers running esxi with a shared storage and fibrechannel san, who are willing to replace their hypervisor.

And 99% of these customers have VEEAM in place as a backup solution.

As VEEAM announced support for Proxmox, our customers are really looking for a change to proxmox and replace vmware. Which is great!


Big thing: VMware VMFS filesystem offers lots of comfort actually. Snapshtos for individual VMs on shared (block) storage for example.


Is there any replacement filesystem in linux/proxmox with vmfs features on block storage? Something under developement?
I heard about glusterfs, but it seems to be deprecated.

Our classic customer has 3 ore more esxi nodes with shared storage block storage attached over fc san or iscsi. (ibm/hp/pure)

The only showstopper for migration to proxmox at the moment is the lack of availabilty to do VM-snapshots stored on shared block storage for (future veeam) backups

Of course, ceph is the way to go, but our customers don't want to throw away > 30k block storages during migration ...

Thanks!
Chris

 

[floh8]

Thats the Problem with Linux there exist no good cluster file system. One possability is to use OCFS2. Ontop of that you can USe the qcow2 layer. But u get no support from proxmox.

 

[bbgeek17]

Hi @inbyteswetrust ,

The company behind PVE, Proxmox Server Solutions, is not in the business of developing filesystems. They could potentially adopt one if a good one came along. However, if a new Cluster Aware file system did come out, it would be a long time before it becomes stable and acceptable for production use.

Developing a file system is not cheap. Those that exist now have billion-dollar companies behind them: Glusterfs/RedHat/IBM, OCFS/Oracle. You are correct that GlusterFS now has EoL at the end of 2024. I'd say that OCFS2 is also not under active development.
This wiki page has a somewhat comprehensive list of CFSs. The ones that have stable operations and wide deployment are those that are part of the core business for the company that developed it (MS CSV, VMWare VMFS).

There are other solutions out there, more modern and more nimble. You can get VM snapshots, Thin Provisioning, Thin Clones, and HA. However, they may carry a cost that would make it impractical for customers who bought low-end, low-capacity SAN solutions.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[VictorSTS]

The only showstopper for migration to proxmox at the moment is the lack of availabilty to do VM-snapshots stored on shared block storage for (future veeam) backups

Backup snapshosts != VM snapshots. With PVE + SAN you lose VM snapshots, not backup snapshots.

Don't know how Veeam will implement their backups, but Proxmox Backup Server uses QEMU to backup the VMs in snapshot mode, which is created at QEMU level. This does work with LVM thick, which is what you would use with those SAN.

You can easily reach 30k in pay per use VMWare Foundation licenses for 3 years for a 3 node cluster. Not to mention SAN support contracts, updates or capacity extensions if needed. And you still won't get as many features as with PVE. I'm afraid end customers will have to decide were to invest for their IT.

 

[bbgeek17]

Backup snapshosts != VM snapshots. With PVE + SAN you lose VM snapshots, not backup snapshots.

AFAIK a common Veeam integration with storage is by taking a snapshot/clone of the disk via storage-specific API and then attaching the clone to the backup server. This is quite efficient from a network/CPU/downtime point of view. We are just a few days away from learning the details of the Veeam/PVE integration details.

Overall, I agree with your post.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

AFAIK a common Veeam integration with storage is by taking a snapshot/clone of the disk via storage-specific API and then attaching the clone to the backup server.

An important part of the orchestration is that veeam also directs the hypervisor to take a guest snapshot BEFORE initiating backing store snapshot, which is how it assures quiescence. I'm honestly not sure if an lvm-thick can be a supportable environment for Veeam, but we haven't seen any specifics on this as we're not in the beta program

edit @martin if its not a secret, can you shed some light?

 

[bbgeek17]

edit @martin if its not a secret, can you shed some light?

Doubt they will take the thunder away from VeeamON
You'll know in 3 days.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

probably. but it doesnt hurt to ask

 

[inbyteswetrust]

Backup snapshosts != VM snapshots. With PVE + SAN you lose VM snapshots, not backup snapshots.

Don't know how Veeam will implement their backups, but Proxmox Backup Server uses QEMU to backup the VMs in snapshot mode, which is created at QEMU level. This does work with LVM thick, which is what you would use with those SAN.

You can easily reach 30k in pay per use VMWare Foundation licenses for 3 years for a 3 node cluster. Not to mention SAN support contracts, updates or capacity extensions if needed. And you still won't get as many features as with PVE. I'm afraid end customers will have to decide were to invest for their IT.

Snapshot is not for backup in our case. It is the process to get a readable state of the VM and copy it to the backup location, without interrupting or stopping the VM.
Because you will need LVM for shared-BLOCK-storage usage, there won't be any QEMU snapshotting option. LVM is raw data.

 

[inbyteswetrust]

Thanks for your answers!

Let's see if VEEAM is able to do some magic which helps in the shared-block-storage snapshot thing.

 

[bbgeek17]

Let's see if VEEAM is able to do some magic which helps in the shared-block-storage snapshot thing.

Are your SAN/Veeam/ESXi customers using storage-originated snapshots now in some form? Are they Agent-based? Or do they rely on VMFS snapshots?

If its the latter, you will likely be disappointed. But I could be wrong...

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

Snapshot is not for backup in our case. It is the process to get a readable state of the VM and copy it to the backup location, without interrupting or stopping the VM.

You mean like replication? This is only supported in PVE with ZFS. Could probably be extended to ZFS-over-iSCSI, yet needs more work.

 

[bbgeek17]

You mean like replication? This is only supported in PVE with ZFS. Could probably be extended to ZFS-over-iSCSI, yet needs more work.

My guess, if the backup location is local - then it's just a clone from a snapshot attached to the backup server. If the backup location is remote then it is more like Netapp Snapmirror, i.e. it would be similar to ZFS provided functionality.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[_--James--_]

IMHO PVE and Block level storage is a hit and miss. It works with LVM-Thin over iSCSI but you lose out on snapshots (this affects backup growth and the ability to freeze VM states). But also you could front load your block storage with a filer and export NFS/SMB to PVE and have the "best of both worlds" while maintaining that block storage investment, but you do introduce a middle-failure-point. However, the likes of TrueNAS scale supporting a HA config plays nice in this space. Just spin up the filers, build the HA pair, attach them to your Block storage, build the LUNs and format them and export them via NFS. Then you connect that up to your ProxmoxVE cluster.

 

[bbgeek17]

It works with LVM-Thin over iSCSI but you lose out on snapshots

Perhaps you meant LVM-Thick?

If you forgo shared/HA storage, yes, you can use iSCSI with LVM-Thin by dedicating LUNs to a node. However, that gains you snapshots.

Most people using iSCSI want to have Shared/HA storage. In this case, you MUST use LVM-Thick. Indeed, there are no snapshots in this combination.

But also you could front load your block storage with a filer and export NFS/SMB to PVE and have the "best of both worlds" while maintaining that block storage investment

You are also "gaining" questionable performance and guaranteed VM stunning on QCOW snapshots at large capacities.

If the storage you are trying to use is not properly architected to work with Proxmox and requires middleware (NAS, File system, File Protocol, etc) - one should be cognizant of various "gotchas" being introduced.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[_--James--_]

Perhaps you meant LVM-Thick?

Yes absolutely, thanks!

If you forgo shared/HA storage, yes, you can use iSCSI with LVM-Thin by dedicating LUNs to a node. However, that gains you snapshots.

Most people using iSCSI want to have Shared/HA storage. In this case, you MUST use LVM-Thick. Indeed, there are no snapshots in this combination.


You are also "gaining" questionable performance and guaranteed VM stunning on QCOW snapshots at large capacities.

If the storage you are trying to use is not properly architected to work with Proxmox and requires middleware (NAS, File system, File Protocol, etc) - one should be cognizant of various "gotchas" being introduced.

Sure, if you do not build the back end out correctly yes you will absolutely get stunning. But you can build out LUNs based on right sizing and export them accordingly to reduce IO stunning and such. Also having a good SAN that is either Flash, or has a Flash Cache Tier (Pure/Nimble) helps a ton here.

I have a couple Nimble Arrays connected to TrueNAS Scale in HA over iSCSI, exported to PVE on NFSv4 with some very large (6TB+) QCOWs and very highly transactional (mixed IO) VMs and have no performance issues when compared to native iSCSI when it was on VMFS. Nimble still reports <1ms latency and hits target IOPS (250k+) on those LUNs.

At the same time, have a couple native luns on the same SANs going to PVE direct and LVM-Thick (Sorry about that earlier!) for some extremely latency sensitive SQL systems and performance outside of snapshots/backups is on par as those going through the NFS filer to the same storage. We are testing both scenarios as the final cut still, having the HA pair does add complexity but its been working without issue for 6 months so far.

But its down to features of the attached storage. LVM-Thick has its limits but so does patching LUNs through NFS. Performance will depend on the deployment model and the storage system capabilities.

 

[bbgeek17]

The stunning was reported by the community when taking snapshots of large QCOWs. But if one invests enough money into their infrastructure no problem is unsolvable ;-)

You clearly had a large investment already made into high-performance hardware and it's great you were able to be nimble (pun intended) with it!

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[_--James--_]

You clearly had a large investment already made into high-performance hardware and it's great you were able to be nimble (pun intended) with it!

Yup and that was the driving force behind the "how can we leverage this insanely expensive SAN infrastructure and ProxmoxVE. Funny enough, each PVE host is setup with CEPH on NVMe and 10k SAS too...so we have what....4 or 5 storage tiers attached to our clusters (iSCSI direct, iSCSI->NFS, SMB(DAS) for backups, CEPH-RBD(NVMe), and CEPH-FS(10K)). I wanted to retire the SAN's and sell them off but I was out voted!

 

[Falk R.]

Snapshot is not for backup in our case. It is the process to get a readable state of the VM and copy it to the backup location, without interrupting or stopping the VM.
Because you will need LVM for shared-BLOCK-storage usage, there won't be any QEMU snapshotting option. LVM is raw data.

You have probably not understood the QEMU snapshot functionality correctly.
You can use QEMU Snapshots to take snapshots of VMs on any storage for the backups. The PBS uses this function and I have already successfully migrated several customers with such setups (FC/iSCSI). You only have to retrain the customers once that you do not take a snapshot but a backup of the VM before updates.
The restore is also extremely fast thanks to PBS.

Veeam uses the HottAdd function so that they can use their own transport service. They also take QEMU snapshots in the background to be able to back up the disk consistently. Only when it comes to change block tracking, Veeam doesn't tell you anything.

 

[inbyteswetrust]

You have probably not understood the QEMU snapshot functionality correctly.
You can use QEMU Snapshots to take snapshots of VMs on any storage for the backups. The PBS uses this function and I have already successfully migrated several customers with such setups (FC/iSCSI). You only have to retrain the customers once that you do not take a snapshot but a backup of the VM before updates.
The restore is also extremely fast thanks to PBS.

Veeam uses the HottAdd function so that they can use their own transport service. They also take QEMU snapshots in the background to be able to back up the disk consistently. Only when it comes to change block tracking, Veeam doesn't tell you anything.

But there is no QEMU functionality when i use 2 Proxmox Hosts attached to a shared-block-storage with LVM Thick? I thought LVM Thick is always "raw" without QEMU.





And I really want to use LVM thick to have the built in HA and live-migration feature of Proxmox.

If VEEAM wants to backup a VM with hotadd, it has to trigger a Snapshot and then mounts this Snapshot with a vm-proxy to get the data to the backup location.