Replacing VMware

[Tom Schmidt]

We a running a 15 Node VMware Cluster which we a going to replace with a Proxmox Cluster. The old cluster consist of different generations of servers. We plan on replacing most of them with than 7 Hosts. I'm going to reuse 4 of the "old" servers, which we aquired two years ago (they are still available). These are:
Fujitsu RX 2540 M6 with the following configuration:
2 x XEON Gold 6338 32C
512 GB RAM (new Servers will come with 1TB)
1 x M.2 240GB Boot-SSD
1 x Nic 4x 1GB Intel I350-T4 OCPV3
2 x Nic 2x 25GB SFP28 Intel E810-XXVDA2
TPM 2.0

Do I miss something? Anything I should take a closer look at?

Greetings
Tom

 

[AristonPost]

What exactly are you asking for? If it's all about hardware compatibility there should ne no issues so far.
However, I don't see any storage so I guess you'll have external storage. I suggest to carefully read the Proxmox documentation about storage and supported features as this is not always comparable to VMware.

 

[Tom Schmidt]

What exactly are you asking for? If it's all about hardware compatibility there should ne no issues so far.
However, I don't see any storage so I guess you'll have external storage. I suggest to carefully read the Proxmox documentation about storage and supported features as this is not always comparable to VMware.

Yes, mostly known problems with this configuration. Storage will be NFS, so I don't expect problems there.

 

[AristonPost]

[...] Storage will be NFS, so I don't expect problems there.

VMware supports snapshots on NFS datastores, period.
The story with Proxmox is little bit more complicated. Not a show stopper IMHO but you should know the limits.

 

[louie1961]

if these are going to be production servers, you would be better off having your operating system on a mirrored pair of drives. if possible I would add a second boot drive.

 

[louie1961]

VMware supports snapshots on NFS datastores, period.
The story with Proxmox is little bit more complicated. Not a show stopper IMHO but you should know the limits.

I missed this at first, but agree. While the network connection is fast, you lose a lot of features by putting the VMs on NFS shares as opposed to local ZFS storage

 

[Tom Schmidt]

VMware supports snapshots on NFS datastores, period.
The story with Proxmox is little bit more complicated. Not a show stopper IMHO but you should know the limits.

Works just fine with QEMU/qcow2...

 

[Tom Schmidt]

I missed this at first, but agree. While the network connection is fast, you lose a lot of features by putting the VMs on NFS shares as opposed to local ZFS storage

What do I lose? BTW: The servers we allready have do not have any local storage besides 1 boot drive (never had one failling), Redundancy is not that easy, you'll need a dedicated mirror device since they only have one onboard m.2 slot.
We allready have a HA NFS Store (NetApp), so buying local storage to use ZFS on the Host would become quite expensive...

 

[bbgeek17]

We allready have a HA NFS Store (NetApp), so buying local storage to use ZFS on the Host would become quite expensive...

You should be fine with NetApp NFS. Assuming that its not HDD based of course. NFS/qcow is prone to VM stunning on snapshot operation.
But if your Netapp was fine for ESXi store, it should be ok for Proxmox.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Tom Schmidt]

You should be fine with NetApp NFS. Assuming that its not HDD based of course. NFS/qcow is prone to VM stunning on snapshot operation.
But if your Netapp was fine for ESXi store, it should be ok for Proxmox.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Do you have any sources concerning the NFS/qcow VM stunning? And yes, the NetApp is full flash.

 

[bbgeek17]

Do you have any sources concerning the NFS/qcow VM stunning? And yes, the NetApp is full flash.

There are multiple user reports and discussions in the forum and elsewhere about it.

https://forum.proxmox.com/threads/snapshot-function-does-not-clean-up-properly.139623/#post-625043
https://forum.proxmox.com/threads/p...on-about-disk-format-snapshots-backup.127222/
https://www.reddit.com/r/Proxmox/comments/rh23sj/proxmox_with_truenas_nfs_snapshots_are_slow/

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[nielsh]

Fwiw, we use a NetApp AFF with NVMe as well and we specifically stopped using snapshots on our nfs/qcow2 volumes on the netapp.

It works ok on very small disks, but for example with 500G disks we already see like an interuption of at least a few minutes. We have a VM with 4TB storage and the VM goes down for 10-15 minutes while it is creating a snapshot. In the end we had to cancel the snapshot and reboot the VM because we could not keep the VM down that long.

 

[bbgeek17]

Do I miss something? Anything I should take a closer look at?

One other thing to keep in mind, since you will have a heterogeneous cluster, you will need to be aware of the CPU type used by the VM. The most performant one is "host". However, that is only suitable for homogenous clusters.

You will need to find the lowest common denominator CPU type or split your PVE cluster into node groups for HA purposes.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[justinclift]

@Tom Schmidt You're going to do a proof of concept first yeah, just to shake things out?

 

[Tom Schmidt]

@Tom Schmidt You're going to do a proof of concept first yeah, just to shake things out?

For sure! I allready have a cluster running, but with older hardware. Next step will be a new cluster with new hardware...

 

[Tom Schmidt]

One other thing to keep in mind, since you will have a heterogeneous cluster, you will need to be aware of the CPU type used by the VM. The most performant one is "host". However, that is only suitable for homogenous clusters.

You will need to find the lowest common denominator CPU type or split your PVE cluster into node groups for HA purposes.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

We will have identical hardware in the cluster, but will most likely use QEMU's x86-64-v4 as CPU type.

 

[Tom Schmidt]

Fwiw, we use a NetApp AFF with NVMe as well and we specifically stopped using snapshots on our nfs/qcow2 volumes on the netapp.

It works ok on very small disks, but for example with 500G disks we already see like an interuption of at least a few minutes. We have a VM with 4TB storage and the VM goes down for 10-15 minutes while it is creating a snapshot. In the end we had to cancel the snapshot and reboot the VM because we could not keep the VM down that long.

That's kind of a deal breaker. How are you doing your backup? In my test cluster I'm using the PBS with snapshot...
Maybe using iSCSI with ZFS on top instead of NFS is the way to go. Or even CEPH?

 

[Magnus-mercer]

Do you need virtual machine migration solutions? Vinchin Backup & Recovery can help you migrate virtual machines quickly and safely. It supports 10+ virtualization platforms including VMware and Proxmox, ensuring seamless migration and easy operation. It also provides a free edition for you to experience!

 

[bbgeek17]

That's kind of a deal breaker. How are you doing your backup? In my test cluster I'm using the PBS with snapshot...

There are two different types of snapshots involved. The QCow snapshot is what most people think of when one mentions a snapshot. A point-in-time copy of data located on disk. In Qcow case - Copy on Write.

PVE Backups use "Snapshot" technology as implemented by Qemu. An in-memory dirty bit map of data during backup only. This allows on-demand backup of blocks that are about to change, so client write can proceed. This functionality is in a state of transitioning to a more bullet-proof "fleecing backup" type. You can find more data through a search.

In short, backup snapshots are NOT the same as VM/disk snapshots. Whether that affects your decision on the storage technology is up to you.

Maybe using iSCSI with ZFS on top instead of NFS is the way to go.

Your Netapp would not be part of this setup.

Or even CEPH?

If you are prepared to make an investment in appropriate CPU, Network and additional Storage, Ceph is one way to go.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[nielsh]

That's kind of a deal breaker. How are you doing your backup? In my test cluster I'm using the PBS with snapshot...
Maybe using iSCSI with ZFS on top instead of NFS is the way to go. Or even CEPH?

We use file-level backups with Borg. VM-level backups (i.e. with PBS) are not a requirement for us, so I cannot say how it handles NFS/qcow. But as bbgeek17 mentions, the PBS backup method does not use the same snapshot method for backups so perhaps it is no issue there. I couldn't tell you myself though.