Big proxmox installations

[Fathi]

Hello,
We are in the process of migrating our legacy vendro-locked virtualization platform and would like to know if anyone is using proxmox VE at a large scale ? Number of nodes and VMs ? Hardware configs ? Also, has anyone made a comparisson of how much it would cost to host all nodes currently on Proxmox VE on other virtualisation platforms like vmware ?

TIA

 

[tschanness]

The costs depend on what deal you get with PVE and the other vendors.
Even a small company might be able to get a 50% discount, so it's best you do the math yourself.

I can't say much about big installations though.

 

[dietmar]

The costs depend on what deal you get with PVE and the other vendors.

Just want to mention that "licence" costs for PVE are exactly zero! You only pay for support.

 

[adamb]

Hello,
We are in the process of migrating our legacy vendro-locked virtualization platform and would like to know if anyone is using proxmox VE at a large scale ? Number of nodes and VMs ? Hardware configs ? Also, has anyone made a comparisson of how much it would cost to host all nodes currently on Proxmox VE on other virtualisation platforms like vmware ?

TIA

Not sure if we qualify as "Big", but we run 315 CentOS7 VM's across 4 DL 380 Gen9 front ends. This is a pretty tough question as workloads can very so drastically in environments.

 

[Fathi]

Not sure if we qualify as "Big", but we run 315 CentOS7 VM's across 4 DL 380 Gen9 front ends. This is a pretty tough question as workloads can very so drastically in environments.

Thanks for the information,

Could you tell me what are the specs of these servers ? Especially the ratio Number VMs/cpu ? Shared storage ? Multipath links in the previous is true ? Interconnexion bandwith ? Cost of the whole solution ?


Tia.

 

[udo]

Hello,
We are in the process of migrating our legacy vendro-locked virtualization platform and would like to know if anyone is using proxmox VE at a large scale ?

Hi,
at what point begins "large scale"?

Is an 10-node-cluster with dual-cpu and 128GB Ram (or more) large scale?

Amount of VMs to Nodes/Ram is very specific and must sized to your environment. You can have VMs, which need mostly the full power of the host, and then you have 30VMs which run without trouble on a small node (like one with 32GB Ram).
Bottleneck with virtualisation is normaly a) IO (esp. disk), b) Ram and after that cpu.

Udo

 

[badji]

What you think about mine?

 

[alexskysilk]

What you think about mine?

I dont mean to Hijack the thread, but are your mesos nodes virtual machines? what is the benefit of doing so vs bare metal?

 

[badji]

Yes, I deploy dcos/mesos on vms, the advantage is that you can deploy multiple clusters dcos/mesos on the same cluster Proxmoxve, for different departments, without overlapping, and use the security of an IAAS ( Proxmoxve, Openstack), which is not the case if you deploy it bare-metal. The version dcos/mesos OS does not provide security. like managing users on an LDAP-Freeipa cluster ...

 

[Alessandro 123]

Would be interesting (for all nerds reading this forum, I think :-D ) to know the hardware size and specs of these huge install.

 

[badji]

For hardware: ASUS RS700-E8 servers with 2 CPU XEON (12vcpu) and ASUS ESC 4000 G3S (To add GPU, dcos/mesos manage already GPU, soon pve?) + 5 SSD + 64 GB RAM + 2 * 10G / bits Network.

 

[Alessandro 123]

For hardware: ASUS RS700-E8 servers with 2 CPU XEON (12vcpu) and ASUS ESC 4000 G3S (To add GPU, dcos/mesos manage already GPU, soon pve?) + 5 SSD + 64 GB RAM + 2 * 10G / bits Network.

Interesting. Which storage are you using ?

 

[badji]

Ceph and Glusterfs Interne + Externe cluster Ntfs on Centos

 

[badji]

Soon an Cluster external with Ceph or Gluster 4.0, Web-gui for Ovirt (Just web for Gluster not for Virtualization) can automate the deploiment...

 

[badji]

 

[Alessandro 123]

Soon an Cluster external with Ceph or Gluster 4.0, Web-gui for Ovirt (Just web for Gluster not for Virtualization) can automate the deploiment...

No issues with gluster?
I was a strong advocate of gluster but development seems to be out of control and with no quality check
There was a corruption bug for years, fixed more or less 2 years after discovery, now there seems to be another corruption bug

That's a shame, gluster is very interesting

 

[badji]

Ovirt and Redhat use Gluster (me too).
There is a new standard CSI Common Storage Interface ) that is used by other projects CAAS, like the Common Network Interface (CSI) for the network. We will see more. Our friend spirit (Alexandre Derumier -team proxmoxve- ) uses a lot of CEPH.

 

[spirit]

running around 4000vms here.

I'm doing multiples clusters of 16-20 nodes, 370gb memory, 2x12cores, and between 70-100 vms by node.
storage on ceph/rbd (ssd and nvme), on dedicated nodes (not proxmox). No lag, no problem since 3 years. (and with major version upgrade)

(my personal opinion about glusterfs : it's sucks. I known a lot of users with recovery failures. Don't known about version 4.0)

 

[Alessandro 123]

storage on ceph/rbd (ssd and nvme), on dedicated nodes (not proxmox). No lag, no problem since 3 years. (and with major version upgrade)

Could you describe your ceph environments ? How many servers, how many switches and so on. 10GBe ?

(my personal opinion about glusterfs : it's sucks. I known a lot of users with recovery failures. Don't known about version 4.0)

Not exactly right this. Gluster works, but from what I can see in dev mailing list, there isn't a real roadmap to follow, every release add tons of feature, most of the time with tons of bugs. Months ago i've asked to focus to fixes and stability adding less untested features with no success.
The corruption bug (always triggered and well known, just rebalance a sharded volume to loose data) took years to be fixed, during this years, tons of other features where added. This is non-sense, imo.

 

[badji]

The compagny of Alexandre : 4em. Good!!!
Proxmoxve+Ceph : Odiso

http://www.01net.com/services/indicateurs-ip-label/hebergement-en-haute-disponibilite/

Aller Alexandre et Aller l'OM

Thank's.