Many people search for a comparison between the last affordable virtualization solution in the market Proxmox PVE and xcp-ng. Lawrence Systems offers a good but incomplete summary.
!!! Important hint to the information in this post !!!!
Besides the technical facts the valuations to this both products are my impression, the opinion of my brain based on the long experience as PVE user, Proxmox forum reader and product tester released with the help of the account "floh8". The ratings refer only to the comparison between the two solutions to each other. Are there still fail information about the product features then post a hint! Please not, that these information are snapshots of the products at the time of post creation. With the ongoing development some of these will change in the future, of course.
The price information refers to a production ready 3 Node Cluster with all important features and the smallest support packet.
Some information to xcp-ng, xostor, xo and Vates:
XCP-ng is a virtualization platform incubated within the Xen Project as XenServer fork and hosted by the Linux Foundation. Vates is the company that develop the xen orchestra (xo) an open source Web-Admin-UI for xcp-ng with additional features like VM replication, VM backup aso. xostore is a project of xcp-ng to offer HCI functionality with Linbits "Linstor" based on DRBD and is included in the xcp-ng product. Vates offer a bundle consisting of a stable release of xo (called xoa) with a Web-Admin-UI for xostor and professional support.
So here are a more detailed feature comparison of these both hypervisor solutions:
Short illumination about the row "snapshot needed space efficiency (block storage)"
When using a shared block storage with PVE or xcp-ng the cluster storage format is LVM Thick. For using it with snapshot support one need an extra storage property like that of vhd or qcow2 . In contrast to vhd the qcow2 usage requires the same amount of free disk space as the vdisk volume of which a snapshot is to be taken. More infos can be found at this link: https://kb.blockbridge.com/technote...m/index.html#lun-sizing-for-thick-allocations. To work around this one must own a thin provisioned block storage that one can over provision.
Hints for the VMware user with block storage (VMFS)
To decide which way to go from VMware to a new hypervisor is awful at the moment especially when your company wants to protect its block storage investment. Only with the new price tag for VMware products by Broadcom Proxmox and xcp-ng start develop further their storage environment to be a real alternative to VMware's VMFS. At the moment both solutions offer only a BETA solution for block storage with snapshot support. xcp-ng already imply a space limit (2 TB) for its long proven vhd storage solution. So if your one do'nt need snapshots of vdisks bigger than 2 TB then xcp-ng is the way to go at the moment. In the xcp-ng forum they discussed to combine several 2 TB vhd vdisks with LVM or BTRFS to one big file system inside the VM. Although Veeam would support such file systems my one is not convinced of an error free workload in production.
But also with the implementation of the qcow2 properties for block storage snapshots that support vdisks bigger than 2 TB your storage should offer thin provisioning (see above information about disadvantage of qcow2 block storage snapshots) to work around its limitation.
My advice is to wait with the migration for 6 - 12 months until the new storage solutions of both hypervisor solutions reach the status "stable". This is also the better decision for Veeam user, which wanne go with xcp-ng, because Veeam support for xcp-ng is also only in BETA status. An other way is to live without snapshot support. Instead of using snapshots one can use incremental and fast vdisk backups with CBT. With the right backup solution (like Proxmox' PBS) in combination with PVE and LVM+raw one get a stable, fast and unlimited vdisk sizeable block storage solution with extreme fast backups.
For the owner of a non thin provisioning block storage appliance its not senseful to use either PVE or xcp-ng. These user should go with Citrix Xenserver because XenServer itself goes an additional way with GSF2 + qcow2 for shared block storage. With it one should get a similar performance as with NFS + qcow2.
Here a overview of the block storage problematic:
From my perspective:
killer features of PVE:
killer features of xcp-ng:
!!! Important hint to the information in this post !!!!
Besides the technical facts the valuations to this both products are my impression, the opinion of my brain based on the long experience as PVE user, Proxmox forum reader and product tester released with the help of the account "floh8". The ratings refer only to the comparison between the two solutions to each other. Are there still fail information about the product features then post a hint! Please not, that these information are snapshots of the products at the time of post creation. With the ongoing development some of these will change in the future, of course.
The price information refers to a production ready 3 Node Cluster with all important features and the smallest support packet.
Some information to xcp-ng, xostor, xo and Vates:
XCP-ng is a virtualization platform incubated within the Xen Project as XenServer fork and hosted by the Linux Foundation. Vates is the company that develop the xen orchestra (xo) an open source Web-Admin-UI for xcp-ng with additional features like VM replication, VM backup aso. xostore is a project of xcp-ng to offer HCI functionality with Linbits "Linstor" based on DRBD and is included in the xcp-ng product. Vates offer a bundle consisting of a stable release of xo (called xoa) with a Web-Admin-UI for xostor and professional support.
So here are a more detailed feature comparison of these both hypervisor solutions:
Short illumination about the row "snapshot needed space efficiency (block storage)"
When using a shared block storage with PVE or xcp-ng the cluster storage format is LVM Thick. For using it with snapshot support one need an extra storage property like that of vhd or qcow2 . In contrast to vhd the qcow2 usage requires the same amount of free disk space as the vdisk volume of which a snapshot is to be taken. More infos can be found at this link: https://kb.blockbridge.com/technote...m/index.html#lun-sizing-for-thick-allocations. To work around this one must own a thin provisioned block storage that one can over provision.
Hints for the VMware user with block storage (VMFS)
To decide which way to go from VMware to a new hypervisor is awful at the moment especially when your company wants to protect its block storage investment. Only with the new price tag for VMware products by Broadcom Proxmox and xcp-ng start develop further their storage environment to be a real alternative to VMware's VMFS. At the moment both solutions offer only a BETA solution for block storage with snapshot support. xcp-ng already imply a space limit (2 TB) for its long proven vhd storage solution. So if your one do'nt need snapshots of vdisks bigger than 2 TB then xcp-ng is the way to go at the moment. In the xcp-ng forum they discussed to combine several 2 TB vhd vdisks with LVM or BTRFS to one big file system inside the VM. Although Veeam would support such file systems my one is not convinced of an error free workload in production.
But also with the implementation of the qcow2 properties for block storage snapshots that support vdisks bigger than 2 TB your storage should offer thin provisioning (see above information about disadvantage of qcow2 block storage snapshots) to work around its limitation.
My advice is to wait with the migration for 6 - 12 months until the new storage solutions of both hypervisor solutions reach the status "stable". This is also the better decision for Veeam user, which wanne go with xcp-ng, because Veeam support for xcp-ng is also only in BETA status. An other way is to live without snapshot support. Instead of using snapshots one can use incremental and fast vdisk backups with CBT. With the right backup solution (like Proxmox' PBS) in combination with PVE and LVM+raw one get a stable, fast and unlimited vdisk sizeable block storage solution with extreme fast backups.
For the owner of a non thin provisioning block storage appliance its not senseful to use either PVE or xcp-ng. These user should go with Citrix Xenserver because XenServer itself goes an additional way with GSF2 + qcow2 for shared block storage. With it one should get a similar performance as with NFS + qcow2.
Here a overview of the block storage problematic:
From my perspective:
killer features of PVE:
- integraded HCI with ceph
- zfs-over-iscsi storage
- integraded firewall for host and VMs
- awesome gui with most functions
- RDMA compatible
killer features of xcp-ng:
- stability
- 24/7 support
- complete load balancing
Last edited:
