XCP-ng 8.0 and Proxmox VE 6.1

[Jarvar]

Proxmox Community

I wondering what peoples thoughts are about these two Hypervisors?
I am drawn to both because of their Open Source ethos and also for ability to enable the smaller people to utilize Virtualization technology.

I initially tried Esxi and it's the most hear of hypervisor I have beene exposed to. Followed by Hyper-V and then VMware player. Eventually I meandered my way to Xenserver and then XCP-ng and then finally Proxmox which I have been using since for the past few months since September.

I would like to know what peoples thoughts are between the two XCP-ng and Proxmox. Honestly, I went with Proxmox because it's Debian based and I already have some familiarity with it. Another thing it had was ZFS support for their system and lastly the Web GUI built it. I think I just found having to install XCP-ng Centre or Xen Orchestra and Xen Orchestra Community Edition after the initial setup. To be fair, I have been setting up XCP-ng as a nested Virtual Machine since I do not presently have a bare metal machine free to test things out otherwise. Also the Proxmox footprint and requirements seem to be smaller than XCP-ng.
Nonetheless, I'm sure XCP-ng is a great Hypervisor as well, just that I cannot understand all of the information related to it.
I would appreciate anybody's feedback or experience with it.
Thank you.

 

[Alwin]

Answers on the Proxmox forum will definitely be biased.

The technologies underneath are for a good part very different. A simple comparison might not do either solution justice. Best, test them and go with what fits best to your needs. Certainly this will be Proxmox VE.

 

[ness1602]

I've worked with almost anything( very little Microsoft Hyper-V), and i can tell you that Proxmox is really something best there is on the hypervisor sky , atleast for me.
Compared to Xen, clustering was really easier, and i really didn't like the whole dom0 machines,which were really partly unusable for me.
Xen uses qemu as Proxmox, and has great snapshot capabilities. As for the backups, Proxmox is clear winner, atleast for me.
Only thing that is currently not available in proxmox i think is some kind of DR on-a-click. I guess you could fix that with multitenant proxmox cluster(or multi-proxmox cluster don't know how it is supposed to be).
I guess best thing is to try Xen, and then come back here :d

 

[Lephisto]

I cannot tell for VMware and HyperV, I have no experience there.

I can just tell about all the way I come from Xen.. been using xen since the first Kernelpatches back in the early 2000's, been using Xenserver and been using XCP1.x, XS6.2->7.2 and XCP-NG in medium Size Cluster settings (traditional iscsi san + N XS/XCP-ng hosts).

Thing is, that XCP is suffering from a innovation lag that Citrix is responsible for and many limitations that haven't been lifted yet.

- Serious orchestration of XS/XCP cannot be done with the Windows Heavyclient.
- The webbased Orchestration "Xen-Orchestra" works, but you're lost without a costly subscription if you run in issues. And you will run in issues. (which is a non issue for me, since I use stuff in business, so it's fine to pay for it)
- The Storage Backend "smapiv2" is seriously flawed and outdated. It does some LVM magic under the hood, snapshots work sometimes, sometimes they don't. Google for "leaf coalescing", and see rivers of tears. The SMAPI is a blackbox, you don't really know what's happening, you can just guess from a few clumsy logfiles / python errors.
- Maximum Disk size ends at 2TB, snapshots are barely working.
- Snapshot exporting (and importing) is terribly slow, so that it's impossible to reach a decent RTO. You only get a fraction of the bang of your up-to-date-hardware. I barely get more then 40MB/s on Backup exports on all-10g hardware which gives my 600-700MB/s from within guests.
- Thoughts about the future: Sure Olivier has put big efforts to save Xenserver after Citrix has decicded to kill all enterprise feature from the free Version, the team is small and I'm not too sure about seeing substantial improvements to core components which are overdue.
- It's not sure for me, that there won't be license Issues and the whole project gets smashed by Citrix.

Sure, Xen still has a few percent performance advantage hypervisor-wise over kvm, but that doesn't play a big role in times you can get 24c server cpu's for around 1k€.

This is not to be meant as a rant against Xen - I stood aside XS/XCP superloyal for so many year and it has worked for me, but I think i'm at a crossroad now. Also Olvier, a nice guy and a few from the IRC Channel help when they can, but when it get's supercomplicated you want to have that subscription. The userbase is tiny, so chances you run into a problem another one has experienced arent too big..

 

[chotaire]

I actually wonder if there is any real speed difference left when using KVM on Kernel 5.* with all-in virtio, host cpu and proper optimization on modern hardware (e.g. 9th gen intel and up) unless you go wild like Amazon and optimize Xen to the core like they did with the recent nitro instances. Any recent sophisticated benchmarks out there?

 

[Jarvar]

Hello Everybody,

I really like Proxmox as well. I actually tried a few Hypervisors before settling upon Proxmox. I just thought I would ask since XCP-ng seems to be similar to Proxmox in that it is Opensource and seems well supported by that community.
I think I settled on Proxmox mainly because I had some familiarity with it's Debain base already and that it had a web gui built in compared to Xen Centre which was Windows Based and their Xen Orchestra which I had to create a separate VM to utilize.
It just seemed like people really liked both Proxmox and XCP-ng and I wanted to find out as to why.
I think one feature I was looking into was backing up offsite, which XCP-ng has an SD-wan plugin.

I appreciate everybody's responses so far. It has been informative and helpful in making my decision. I think initially with XCP-ng, I was impressed with Olivier's response time, but ultimately want to choose the option that best suited my needs. Another thing that seemed to stand out was the ease with which to add nodes use Xen Orchestra. I know Proxmox can do that as well, but I haven't spent too much time in a lab environment setting it up to tinker with it.

Anyways thank you so much and I do value any feedback or input others with experience may have.

 

[chotaire]

You can backup to where you like, Storages, SANs, Backup Clusters or mount some remote NFS, CIFS, SSHFS, S3 etc.. etc.. or if you feel nerdy, even to Google Drive or a public FTP server in Uganda protecting your backups with gocryptfs or ecryptfs. vzdump-hook makes it possible to mount/unmount remote crypto when needed.

I run four backup schemes using local storage, CIFS, SFTP and borgbackup over CIFS.

SD-WAN plugin is awesome though. If that is actual SD-WAN which I doubt, probably just a WAN connector.

 

[Lephisto]

Hello Everybody,

I really like Proxmox as well. I actually tried a few Hypervisors before settling upon Proxmox. I just thought I would ask since XCP-ng seems to be similar to Proxmox in that it is Opensource and seems well supported by that community.
I think I settled on Proxmox mainly because I had some familiarity with it's Debain base already and that it had a web gui built in compared to Xen Centre which was Windows Based and their Xen Orchestra which I had to create a separate VM to utilize.
It just seemed like people really liked both Proxmox and XCP-ng and I wanted to find out as to why.
I think one feature I was looking into was backing up offsite, which XCP-ng has an SD-wan plugin.

I appreciate everybody's responses so far. It has been informative and helpful in making my decision. I think initially with XCP-ng, I was impressed with Olivier's response time, but ultimately want to choose the option that best suited my needs. Another thing that seemed to stand out was the ease with which to add nodes use Xen Orchestra. I know Proxmox can do that as well, but I haven't spent too much time in a lab environment setting it up to tinker with it.

Anyways thank you so much and I do value any feedback or input others with experience may have.

Adding nodes in terms of additional Clusterhosts is more or less them same procedure. Bootstrap a new host, configure network, enter a Cluster join key (or root credentials on XCP). Sync is done with corosync on Proxmox and xapi on Xen. The Cloudinit support with qemu/kvm/proxmox seems more mature in my opinion, when it comes to mass-provisioning guests.

Offsite Backup is a non-issue.. you can use the Proxmox integrated Snapshotting tool and just write to a remote FS/S3bucket/you name it, use async Ceph export or znapzend in case you use ZFS.

 

[fx919]

I really like PVE a lot. It works great!

However what I prefer of citrix XenServer/citrix Hypervisor/XCP-NG is the loose coupled servers. I once had a problem within Proxmox Cluster management with PVE 4.x or 5.x (not sure), some years ago: The Cluster didn't respond any more. No cluster command on the CLI worked. Every command I entered just blocked. The only thing I could do was to shutdown the whole cluster and boot it up again.

That was a big issue and a nightly maintenance had to be planned and carried out.

I fear such an issue coming again. And if the cluster of servers get more and more nodes the issue has dramatically rising impact. If I have to shutdown 100 VMs ore more on all nodes that's horror for me.

Here with XCP-NG I can migrate a VM to a non-cluster server. Despite having a lot of other unpleasant experience with XCP-NG/XenServer I had never had such cluster blocking issues. And that's why I like the loose coupling of that tech.

That's what keeps me from being completeley fascinated with proxmox.

So the main thing for me is the missing feature of...

Live-Migration of VMs to cluster-external Proxmox-Servers.

(And yes there's proxmove out there, but that does no live migration but only offline migration.: See Google: ProxMove)

So I refrain from building large Clusters of PVE Servers and keep them small for isolation of problems to strict boundaries.

 

[tom]

... every command I entered just blocked

If you do not have quorum, access is blocked.

Solution: gain quorum and you can fix your issues.

Best solution for you is the solution where you are familiar with the basic admin tasks.

 

[fx919]

Well the issue was long ago, so I for sure can not remember all details of the case.

What I'm sure of: It was no quorum issue.

I couldn't get even the quorum information out of pvecm status (That blocked too) or trying to override the quorum with pvecm expected 1

... and I'm sure if I had contacted Proxmox support, the problem would have been resolved. But unfortunateley we had no subscription for that server. ;-(

 

[Jarvar]

I really like PVE a lot. It works great!

However what I prefer of citrix XenServer/citrix Hypervisor/XCP-NG is the loose coupled servers. I once had a problem within Proxmox Cluster management with PVE 4.x or 5.x (not sure), some years ago: The Cluster didn't respond any more. No cluster command on the CLI worked. Every command I entered just blocked. The only thing I could do was to shutdown the whole cluster and boot it up again.

That was a big issue and a nightly maintenance had to be planned and carried out.

I fear such an issue coming again. And if the cluster of servers get more and more nodes the issue has dramatically rising impact. If I have to shutdown 100 VMs ore more on all nodes that's horror for me.

Here with XCP-NG I can migrate a VM to a non-cluster server. Despite having a lot of other unpleasant experience with XCP-NG/XenServer I had never had such cluster blocking issues. And that's why I like the loose coupling of that tech.

That's what keeps me from being completeley fascinated with proxmox.

So the main thing for me is the missing feature of...

Live-Migration of VMs to cluster-external Proxmox-Servers.

(And yes there's proxmove out there, but that does no live migration but only offline migration.: See Google: ProxMove)

So I refrain from building large Clusters of PVE Servers and keep them small for isolation of problems to strict boundaries.

Thank you for your feedback. I don't have that much experience with either. I am new to both. Although, I do have slightly more experience with Proxmox than with XCP as that is the one I have been using since August 2019. I've also used Ubuntu and Debian in the past which gives me the edge in familiarity with commands and maintentnance.

Like you, I will take note that adding nodes to clusters and managing them seemed a little easier. With Proxmox I had to create a cluster and then add the nodes to the cluster. However, when removing them, it wasn't as clean and I am supposed to reinstall the Proxmox OS before starting it up on the same network.

Granted, in my envrionment we have not setup a cluster. Similarily, I needed to replace a single node and I wanted to transfer VMs over from the old server. In the end I had an NFS storage setup where I did vzdumps and then restored them on the new node without joining a cluster. Both nodes where able to access the NFS. I backed up from one server and then restored on the other. I did try ZFS send and receive as well, but the backup and restore process with NFS storage was the simplest. At first I forgot to check off the Thin Provisining on the second VM so I was using up a lot of disk storage, but once I discovered the issue it was a pretty simple fix and process.

In the end I am more familiar with Proxmox's Debian structure than I am with XCP-ng. One other thing was the way XCP-Ng uses Xen Orchestra, took me a bit of figuring things out to get it working, I wasn't aware at first that the XOA needed to be run as a separate VM or else use Xen Admin Centre which was similar to the Windows Server Admin Centre. I liked PVE's built in GUI when I need it and then use their shell or ssh for everything else.

 

[Lephisto]

You are right, the possibility to have a Cross-pool migration in XS/XCP is a unique selling point. However, I barely made use of it, since it turned out to be not super-robust. In Lab environments this worked most of the time, in Production, when guests were up for some month, and maybe live migrated within the cluster a few times, it was not very reliable (as unreliable as SXM (Storage Xen Motion)) turned out to be. So in theory this would be nice to have, in reality it is only of very limited use. On Proxmox you have to export/import it offline, but you don't suffer from ancient bandwidth limitations that the XS/XCP smapiv2 gives you.

 

[fx919]

However, I barely made use of it, since it turned out to be not super-robust.

Yes. That's my experience too. It sometimes took ages to get migrations finished and often it just did not work(failed at some point). Sometimes it worked after multiple retries, sometimes it didn't.

 

[Rubin]

I'm not very familiar with Promox, i've used vmware, citrix, xcp-ng, ovirt, and some other platform. And i have tried many backup solutions: Vinchin Backup & Recovery, veeam, vprotect. At last, i chose Vinchin software to backup my citrix and xcp-ng environment, veeam price too high, Vinchin is a good choice for me, support many mainstream platforms, good price and kind service.

 

[ness1602]

If you are an employee of Vinchin,just write that in your profile.