Is Proxmox VE ready for enterprise production environment?

[poetry]

There is an active thread on https://old.reddit.com/r/sysadmin/comments/yzm5hc/shout_out_to_proxmox_ve_devs_best_hypervisor_imo/ about proxmox ve and personally I am interested in drawbacks and limitations vs other enterprise virtualization products. As we are starting to adopt more proxmox in our company I am wondering if comments below are valid and if proxmox staff has any answers for them?
People have been successful in running proxmox in home environments something that is completely different to enterprise production environment.

I want to thank proxmox team for their great work so far. My experience has been great so take all people's comments as feedback where you can improve because you can always do better. That is at least my mentality and if you think some of the comments are just wrong make sure to comment on reddit thread too to let them know they are wrong.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix0sa61/
From nerdyviking88

Proxmox is a fantastic tool, but the problem is, it's not ready for the enterprise. The features it has are not fully baked, nor are they sustainable. Support is 'available', but only business hours in Germany, which again, is not sustainable for many businesses.
I've been using Proxmox since 1.7, and did run it in production. It did everything I needed it to, but I made the choice to use it understanding that I was going it alone, no backup vendor support, no MSP to pull me out, etc. That is not an option for enterprise.
SMB or Home Datacenter, sure. Enterprise prod without a dedicated team? Never.

From: nerdyviking88
https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix0ty4b/

Distributed Switches. DRS. Tanzu/Pod service. Host profiles. Instant clones. Storage class of service. Persistant memory. GPU passthrough.
Just to start.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix15z0w/
From HalfysReddit

IMO it's hard to say one thing is "better" than other definitively, because people have different needs and different values and rarely does one thing do everything better than another.
I'm a big fan of Proxmox myself, but I would be lying if I said it's easy to use. I could give someone with very basic IT skills access to a Hyper-V server (with GUI of course) and expect that they can do 90% of the tasks that go into managing a single hypervisor. Proxmox, I would expect them to make a number of mistakes before reaching that 90% proficiency level.
Not to mention, if you run into a problem with Hyper-V or VMWare, there's a 99.999% chance that other people have had the same issue and documented their fixes somewhere online. Proxmox, you're often times going to have to do some reverse-engineering to get things working the way you want them to, because their simply just isn't easy answers available online.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix0yvkj/
From jayminer

Can it import/export to ovf/ova yet? Don't want to be locked in.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix1f0t5/
From jrdwiz

Adding to this - A much more intuitive datastore experience, ISO, templates, resource information, and networking, to name some. There is also SR-IOV. Maybe I'm biased as I've used ESXi and vCenter longer. I've used Proxmox too, and am actively using it on one host in my lab. I don't like how many resources vCenter needs to run, but I find it a lot more intuitive to use.
Proxmox is better when it comes to consumer gear that is not easily supported with ESXi (granted, this is more homelab focused).

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix166jm/
From EspurrStare

GPU passthrough can be done.
Distributed Switches can be done in Proxmox, Bond+Bridge+VLAN. But the observability of it sucks.
Host profile can be exported by copying the /etc/pve directory, as it holds all configuration (But a button would be nice).
Instant clones is something I find bizarre it lacks. In theory, it only has to do the same process live migration does, just not deleting the original host afterwards. But I guess they have their reasons. You can make a snapshot and then clone, which is close (unless you are using Zvols, then you are going to need to do it manually with zfs send, recv and copying the VM configuration and snapshot).
No kubernetes support, and given that the devs have multiple times come up against native Docker or Kubernetes support, saying that it should be provided in a VM, it's probably never going to happen.
It appears you can use Persistant Memory with Proxmox, but not through the GUI : https://forum.proxmox.com/threads/proxmox-and-intel-optane-dcpmm.87861/ I wouldn't consider this feature as supported and wouldn't consider Proxmox to use it unless I had to.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix0u1wb/
From: davokr

Enterprise support.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix1l2e7/
From: paq12x

network card failover works very well in ESXi, but not so well in Proxmox (it works some of the time).
The ability to reinstall the boot drive and restore the entire host setting with just one command (with a single configuration file). I can do that in ESXi in no time. In proxmox, I'll have to manually edit or restore a bunch of configuration files.
Even w/o any backup, it's very fast to scan and get the VMs up and running. In proxmox, it's not as simple.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix0u671/
From: mastert429

not the person who made the comment, but the biggest annoyance i have with it is not being able to add a hdd from an existing disk image from the web ui.. i haven't tried but i assume it also doesn't have anything like live motion.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix1aw0j/
From: nevergrownup97

Absolutely! One thing I‘m really missing is a stupid simple ootb router feature with DHCP NAT Port Forwarding etc. like in most Level 2 hypervisors.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix1m4qo/
From: Gomm_Jabbar

I pay for VMUG to stay on top of VMWARE technologies cause throughout my very long sysadmin carrier, I haven't yet seen any business using Proxmox. The use ESXi, RHEV, Hyper-V or clouds but not Proxmox.
I know it's great but most of medium and large companies in North America abstain of using it.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix134jf/
From: bcredeur97

I’d kill for proxmox or something else to have a multi-tenancy
So you would have an easy low cost way for a small guy to start his own datacenter and build his own “cloud” so you could easily start small and work your way up to a big infrastructure

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix1vpz0/
From: TheKrister2

While I do mostly like Proxmox, the only thing I dislike is its atrocious documentation. The wiki itself is alright, but it's missing a lot too and the moment it does you'll likely have to hope there are solutions other places than their forum. It's one of the few forums I can wholeheartedly call awful.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix1b6eo/
From: roiki11

While it's really nice, it really doesn't compare to vmware. Which is quite understandable as it has billions in development money in it. Vmware just has a great portfolio that works well on 100+ node scales and the products bring a lot of value add, if you can pay.
VMware also has a lot better automation and customization support. You need cloudinit in proxmox.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix1c3a3/
From: MDSExpro

Some? There are tons of such features. Proxmox is barely at "single site, single cluster" config, mostly computing with minimal storage and networking support.
Hell, Proxmox freezes VM while making snapshots which makes it no go for and serious installation.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix2tc64/
From: finobi

No support from other commercial software like Veeam, Zerto, Avamar, Eaton etc. Doubt that proxmoxs own backup can do Windows shadow copy tricks to keep backup data consistent?

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix3dvei/
From: djgizmo

Proxmox is awesome... it just sucks they haven't sorted an orchestrator down to allow migrations from 1 cluster to another. That's the big hold out for most.

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix3d9zr/
From: Alex_Hauff

key word Home DC

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix109p3/
From: brkdncr

“Home datacenter”

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix2c053/
From: canadian_viking

I would have to agree. I'm in the midst of doing a Hyper-V to Proxmox migration, but the farther this project proceeds, the more I have misgivings about it. It's unsettling migrating to a product that's feels incomplete, and the rest is just "Well, it's in Linux somewhere....figure it out". And the "figure it out" stuff is often "If you slide off the paved road we built, you ain't just offroading...you're falling off the cliff, and you're probably gonna be on your own."

https://old.reddit.com/r/sysadmin/c..._proxmox_ve_devs_best_hypervisor_imo/ix2083m/
From: Bijorak

Tesla devs used proxmox and trusted it. One host went down and the entire cluster failed. They lost everything on it. The entire config was wiped. They were down for quite a while. I knew a guy that helped them bring it back to life. They lost a ton of work.

 

[UdoB]

Thanks for those interesting bits from reddit. It definitely shows that there are always different point-of-views. And yes, PVE is not perfect (yet). Personally I am a heavy debian user - so PVE just feels "native" for me, completely in contrast to those closed source alternatives - which also do not always work flawlessly btw...

I am very happy with Proxmox and I am glad to have the choice. (Both at my employers site and also @home.)

 

[tom]

thousands of companies around the world are using Proxmox VE in production with commercial support. hundred of thousands use it without a support contract, and countless home users love Proxmox VE too.

if Proxmox VE is the right hypervisor for your business? Just test and evaluate - we do not know your business needs nor your company.

Following unknown people threads (some contain wrong assumptions) and their stories is not always the best advice.

=> teach yourself, test your workload and decide if Proxmox VE is your best fit.

 

[t.lamprecht]

There's a lot of name-dropping without context, and often it's pretty obvious that the commenter either did not used PVE ever in a serious manner or that the last time was quite a few years ago. I'll try to add a bit to most points, mostly from top of my head, so don't pin me on wording et al. please.

Distributed Switches. DRS. Tanzu/Pod service. Host profiles. Instant clones. Storage class of service. Persistant memory. GPU passthrough. Just to start.

There's a sdn tech preview, and with Linux networking basically everything is possible without that too, as long as one isn't scared of that. Note that network requirements vary widely from setup to setup and the respective needs, for a lot of them the built-in is working out fine, for the rest SDN is. You can clone, and that instantly if the underlying storage supports it (most do), name-dropping persistent memory makes no sense without context, GPU pass through is supported since a while, the commenter seems to either not checked our PVE too closely or did so years ago.

Can it import/export to ovf/ova yet? Don't want to be locked in.

OVF import works since years (e.g.).

Adding to this - A much more intuitive datastore experience, ISO, templates, resource information, and networking, to name some

Naming some is not really useful if one doesn't state what's wrong/off/... you got ISO, templates and resource information in Proxmox VE's data store model too.

No kubernetes support, and given that the devs have multiple times come up against native Docker or Kubernetes support, saying that it should be provided in a VM, it's probably never going to happen.

Docker makes no sense natively on PVE in our opinion, but I don't recall speaking out against Kubernetes, that's rather orthogonal and can be done and used with PVE.

Enterprise support

There's enterprise support https://www.proxmox.com/proxmox-ve/pricing

not being able to add a hdd from an existing disk image from the web ui..

Trivial from the CLI with qm importdisk, and the node CLI is reachable from the web-interface But there's some semi-active work going on to take importing directly to the GUI as a wizard.

One thing I‘m really missing is a stupid simple ootb router feature with DHCP NAT Port Forwarding etc. like in most Level 2 hypervisors.

SDN has a simple zone exactly for this, once one opt-in into the tech preview by installing the pve-network package one can do so via the web interface.

So you would have an easy low cost way for a small guy to start his own datacenter and build his own “cloud” so you could easily start small and work your way up to a big infrastructure

I don't really know anything that blocks one from doing that with Proxmox VE. You can expand the cluster, and FWIW, there are plans for a datacenter manager, and we're actively worked on its foundation already this year.

While I do mostly like Proxmox, the only thing I dislike is its atrocious documentation. The wiki itself is alright, but it's missing a lot too and the moment it does you'll likely have to hope there are solutions other places than their forum. It's one of the few forums I can wholeheartedly call awful.

The docs can surely be extended, as in every project, but I'm actually not sure if the user saw the reference docs, which are linked to all over the place in the web-interface to the respective specific context via the help button. https://pve.proxmox.com/pve-docs/pve-admin-guide.html
Also, I find the forum community extremely helpful in the general case, and not even thinking of the staff when saying that.

Doubt that proxmoxs own backup can do Windows shadow copy tricks to keep backup data consistent?

Sure we do. With the QEMU guest agent installed we ensure full consistency on Windows and Linux guests. Proxmox Backup Server has dirty-bitmap tracking and can do a full consistent backup in a differential way in a few seconds that way. I doubt that user every tried it

Proxmox is awesome... it just sucks they haven't sorted an orchestrator down to allow migrations from 1 cluster to another. That's the big hold out for most.

See a bit above w.r.t datacenter manager, and as said we're laying foundation for that stuff right now, for example:
https://git.proxmox.com/?p=qemu-server.git;a=commit;h=192bbfda82f82ce828179e5601a9b1c50ac2821d

Tesla devs used proxmox and trusted it. One host went down and the entire cluster failed. They lost everything on it. The entire config was wiped. They were down for quite a while. I knew a guy that helped them bring it back to life. They lost a ton of work.

That I find hard to believe knowing our cluster filesystem and how it works, e.g., we don't touch the data of other nodes and any node basically holds a full copy of the other ones. On top of that, if the entire cluster fails it means that quorum is gone too, and at that point the configuration file system goes in read-only mode. That reads a bit fishy/FUD like tbh., and I'm quite a long time at Proxmox I'd like to think that I'd remember a support case like this, especially if from a prominent company as them.

I would have to agree. I'm in the midst of doing a Hyper-V to Proxmox migration, but the farther this project proceeds, the more I have misgivings about it. It's unsettling migrating to a product that's feels incomplete, and the rest is just "Well, it's in Linux somewhere....figure it out". And the "figure it out" stuff is often "If you slide off the paved road we built, you ain't just offroading...you're falling off the cliff, and you're probably gonna be on your own."

Hyper-V shops will surely have some more accustoming to do, the underlying foundations are quite different. With Enterprise Support nobody really is on their own, without it, well, you may find that you're not always lucky for free help even in our forum and indeed get the help you pay for

key word Home DC

“Home datacenter”

Yeah no, just check the official testimonials: https://www.proxmox.com/en/proxmox-ve/testimonials
And that's just the companies/institutions where both we and them took the time to flesh out an official testimonial, Proxmox VE has lots of small, medium and big enterprise users and institutions.

TLDR; Can Proxmox VE still get better? Sure, yes; as most solutions it in fact can. Are we actively working on that? Yeah, SDN, Datacenter management and "cattle not pets" for huge deployments and polishing UX by bringing more features natively in the API and GUI are things that we know PVE has still potential to grow, and we're actively working on that.

edit: oops, was a bit to slow to reply, but I'll keep it up

 

[RolandK]

> Tesla devs used proxmox and trusted it. One host went down and the entire cluster failed. They lost everything on it.
> The entire config was wiped. They were down for quite a while. I knew a guy that helped them bring it back to life. They lost a ton of work.

this sounds like FUD to me. first off, you won't loose vm config if a cluster node fails, furthermore it's dead simple to backup all the proxmox and VM configuration and furthermore, if you have something in use in production, you should have backup.

so, if they lost a ton of work then it's because THEY did something wrong. not proxmox.

 

[CvH]

furthermore it's dead simple to backup all the proxmox and VM

The backup of VMs is kind of broken https://bugzilla.proxmox.com/show_bug.cgi?id=2874 so it is very possible that all your backups are broken. Ofc this is not happen that frequently but there are already A LOT reports about it. I already hit by that problem twice so if you do not test your VM backups frequently you can expect that some of them might be broken.

if they lost a ton of work then it's because THEY did something wrong. not proxmox

Ofc this is possible, if you have no experience at proxmox/linux you can easily hit pitfalls.

but I don't recall speaking out against Kubernetes

for some odd reasons a lot software focuses exclusively at docker and k8s these days, so having a native support (not inside some random vm) would be very handy

 

[RolandK]

>The backup of VMs is kind of broken https://bugzilla.proxmox.com/show_bug.cgi?id=2874 so it is very possible that all your backups are broken

to be honest i think you are right to some degree and, i have to admit that i'm missing more attention on this to get this resolved by proxmox team, even if there is not a repro case yet.

such bug should not lurk beneath the surface in something which is ready for the enterprise.

if it's not solvable because no repro-case exists, there could at least be added some solution for detection of mbr corruption inside your backups (which would perhaps even help getting to the root cause of that bug, as it would be detected more often)

 

[t.lamprecht]

The backup of VMs is kind of broken https://bugzilla.proxmox.com/show_bug.cgi?id=2874 so it is very possible that all your backups are broken. Ofc this is not happen that frequently but there are already A LOT reports about it. I already hit by that problem twice so if you do not test your VM backups frequently you can expect that some of them might be broken.

They aren't broken for most of our high six figures user base. This is a very specific issue hitting a handful of people; and yes we understand that it's definitively a frustrating problem, and spent considerable time looking into it even though (IIRC) we did not have a single enterprise customer affected; but it's not even established yet that vzdump is the actual cause, and not just a correlation, it well can be a bug from the software running inside the VM (most reports say its always affecting the same VM) or an underlying storage bug/error.

 

[RolandK]

EDIT/NOTE 07.03.2023:
recent insight revealed that bug no. #2874 only seems to affect virtual sata disks (so thomas is right and this bug is very specific) and progress has been made to gather information from affected people. so this post is outdated.



@t.lamprecht, i know that this may not be the right place to discuss, but the mentioned bug above extremly frightens me and also it's saddening me, that it seems there is not put more effort into finding the root cause and fixing it.

i cannot estimate how much effort has been put into it already, but personally i'm missing some activity and discussion how we can progress. i hardly believe it only hit a handful of people, as we know not every user will report/complain in a forum or bugtracker.

"enterprise ready" label has something to do with trust, and to see a bug like this lurking being unresolved for 2 1/2 years and now hitting another proxmox user is causing loss of trust. trust into the product and trust into the company behind it. at least for me (though, there is SO MUCH positive with proxmox) .

data loss or storage corruption is a nightmare for every infrastructure administrator, i have undergone such event in a larger environment and know what it means if your san/storage has hicup , your data getting corrupted/lost and your backup is hosed, too.

i've got another idea how we can make progress here and commented the bugreport.

there could be build a qemu debug package containing the proposed debug option for monitoring writes to sector 0. that would empower people affected to collect more valuable data/logs.

we could also ask qemu devs for help or ideas solving/encircling this issue, if proxmox team has no capacity or is out of ideas.

please let's work together and fix this.

meanwhile - have fun !

https://www.youtube.com/watch?v=MgxgYL5P4z4