Interested in node density

A

adamb

Famous Member
Proxmox Subscriber
Mar 1, 2012
1,323
73
113
Hopefully this doesn't get closed just looking to get the community's input on node density.

We have a fairly dense 3 node cluster that is currently running 140+ CentOS7 VM's. So far it has been doing great!

How many of you out there are running such dense setups? How has it been, have you run into any issues?

My only complaint is having to migrate each VM and not having the ability to migrate all HA VM's Although I haven't tested in a few versions, last I knew the migrate all wasn't working with HA enabled VM's.
 
Migration is possible via qm migrate easily. I wrote scripts for our cluster to migrate to two other nodes randomly in a similar setup as you describe (works also with HA VMs):

Code: 
root@proxmox1 ~ > for i in $( qm list | grep running | awk '{ print $1 }' ); do qm migrate $i proxmox$( [ $(( $RANDOM % 2 )) -eq 1 ] && echo "2" || echo "3" ) --online; done

We have over 150+ KVM on 4 nodes, yet there is a lot of free space. We're running mostly Debian and some Windows and Oracle Linux machines.
 
LnxBil said:
Migration is possible via qm migrate easily. I wrote scripts for our cluster to migrate to two other nodes randomly in a similar setup as you describe (works also with HA VMs):

Code: 
root@proxmox1 ~ > for i in $( qm list | grep running | awk '{ print $1 }' ); do qm migrate $i proxmox$( [ $(( $RANDOM % 2 )) -eq 1 ] && echo "2" || echo "3" ) --online; done

We have over 150+ KVM on 4 nodes, yet there is a lot of free space. We're running mostly Debian and some Windows and Oracle Linux machines.
Click to expand...

Very cool to here there is atleast 1 other running with such dense configurations.
 
Hi Adam
If you consolidate such a dense setup, I would advise you to have a look in the summary panel for each node the KSM sharing value.

A high value here means KVM is saving you memory by consolidating memory pages detected as being duplicated between your guests.
say if your 140 CentosVM are running the same version of the cron daemon will, KSM on PVE will detect that and prevent some memory page duplication.

On a setup wuch as yours I would expect the pro node value of KSM to be quite high.
 
I'm running 2x10 cores intel 3,1ghz host with 370G hosts, with around 70-100vms.
Cluster have around 1200vms on 15hosts. (same config)

KSM give me around 80G memory deduplicationby host.

Of course, we monitor carefully all vms. (we do administration of all vms, customers don't have root access, so no too much surprise with crazy process)
 
manu said:
Hi Adam
If you consolidate such a dense setup, I would advise you to have a look in the summary panel for each node the KSM sharing value.

A high value here means KVM is saving you memory by consolidating memory pages detected as being duplicated between your guests.
say if your 140 CentosVM are running the same version of the cron daemon will, KSM on PVE will detect that and prevent some memory page duplication.

On a setup wuch as yours I would expect the pro node value of KSM to be quite high.
Click to expand...

Yep we are currently seeing some significant savings from KSM. Right now we have roughly 170G of KSM memory. Its impressive.
 
spirit said:
I'm running 2x10 cores intel 3,1ghz host with 370G hosts, with around 70-100vms.
Cluster have around 1200vms on 15hosts. (same config)

KSM give me around 80G memory deduplicationby host.

Of course, we monitor carefully all vms. (we do administration of all vms, customers don't have root access, so no too much surprise with crazy process)
Click to expand...

Very cool! Good to hear there are ppl pushing the limits even further. We are growing at a fast pace so its only a matter of time before we start to hit these numbers.
 
adamb said:
Very cool! Good to hear there are ppl pushing the limits even further. We are growing at a fast pace so its only a matter of time before we start to hit these numbers.
Click to expand...

On a side note, just tested "Migrate All" and it seems to be working as expected with the latest packages. Very cool!
 
mir said:
Could anybody give a good reason for migrating stopped VM/CT's?
Click to expand...

If you don't migrate the VMs from e.g. A, then after stopping A, you cannot start any VMs on A. This is only possible if you migrate it before. (start I mean on a another node)
 
robhost said:
Every task which make it unclear if the node will be available when needed:
  • Reinstalling a whole node
  • Upgrade node storage
  • Upgrading node to newer PVE release
Click to expand...
Upgrade node storage and upgrade to newer release does not require that stopped VM's are migrated.
 
LnxBil said:
If you don't migrate the VMs from e.g. A, then after stopping A, you cannot start any VMs on A. This is only possible if you migrate it before. (start I mean on a another node)
Click to expand...
But if a VM is stopped before stopping A why should it not remain stopped after stopping A?

I would prefer an option to be able to decide whether stopped VM/CT's should be migrated as well.
 
mir said:
But if a VM is stopped before stopping A why should it not remain stopped after stopping A?

I would prefer an option to be able to decide whether stopped VM/CT's should be migrated as well.
Click to expand...

I don't know, but you cannot start it if you want because the configuration is on the down-node A, so it is not available in the GUI (at least for 3.x, don't know about 4.x cluster).
 
LnxBil said:
I don't know, but you cannot start it if you want because the configuration is on the down-node A, so it is not available in the GUI (at least for 3.x, don't know about 4.x cluster).
Click to expand...
Configuration for all VM/CT is available on all nodes. This is true for both 3.x and 4.x.
/etc/pve/nodes/{node}/qemu-server and /etc/pve/nodes/{node}/(openvz|lxc)
 
If you copy the config to another node you can start it from the GUI on this node. Eg.
1) Node1 is down so login to node2
2) mv /etc/pve/nodes/node1/qemu-server/vm1.conf /etc/pve/nodes/node2/qemu-server/vm1.conf
3) Start vm1 in the GUI where vm1 will appear stopped on node2
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back