Advice for memory optimisation / overcommit

[Joe Botha]

Hi

I imagine some VPS hosting companies have ways to sell more memory than their servers actually have. How do they do this? Any advice for making the most of your server memory?

A few ideas so far:

* Use containers rather than VMs - pretty obvious
* Make sure KSM (Kernel Samepage Merging) is enabled, and try run the same OS in the VMs / containers
* Give each VM some swap space on a fast disk
* In the case of container hosting, give the hostOS swap space on a fast disk
* Use Intel Optane pmem dimms - probably expensive and only really available in their Scalable Xeon range of CPUs
* Install an Optane SSD for swap space, something like the P5800X / PCIe4 (almost 8GB/s read speeds)

Anybody using some of these ideas or have better solutions? Any guidelines to follow or can you point me at things to read?

I'm thinking a server with 128G RAM and Optane SSD with KSM enabled could maybe run 100 x 2GB containers.

Do you know of any extreme cases of VMs or containers per host?

 

[Dunuin]

* Make sure KSM (Kernel Samepage Merging) is enabled, and try run the same OS in the VMs / containers

Keep in mind that you should disable KSM if you can't trust the software running in the guests. KSM makes it easier to read RAM of other guests.
See discussion here: https://forum.proxmox.com/threads/ksm-sharing-security-implications.104261/

 

[mvs]

I'm thinking a server with 128G RAM and Optane SSD with KSM enabled could maybe run 100 x 2GB containers.

Optane DC SSDs are quite expensive. I doubt that you get any cost savings by using this approach.
Additional 128GB DDR4 Reg. ECC RAM would cost just around $600 - $800.

 

[LnxBil]

AFAIK KSM does not work with LX(C) containers or even normal linux processes due to not using memory that has the MADV_MERGEABLE flag, so you're limite to KVM/QEMU VMs, which does set this flag for alle memory allocated to the guest.

 

[hvisage]

I imagine some VPS hosting companies have ways to sell more memory than their servers actually have. How do they do this?

The same way as ISPs oversell bandwidth by playing a numbers game
ie. expecting that, on average, users will not use more than 50% of RAM, thus 100 x 2G VPS's expecting to not use more than 100*2*0.5GB of RAM.

Some even enforce (inside VMs) echo 3 > /proc/sys/vm/drop_caches and others have (slow) swap with higher swappiness.

Ditto for CPU usage, where some even have clauses about not running crypto mining etc.

 

[harmonyp]

Any progress on this? Is there a equivalent for AMD (intel Optane)

 

[pringlestuffs]

Any progress on this? Is there a equivalent for AMD (intel Optane)

I imagine you're talking about actual server hardware as opposed to consumer hardware, but FWIW I'm running Proxmox on an AMD 5750GE mini desktop PC with an Optane NVMe stick in it that serves as swap for the host and guests. Works great, I'm happy with it.

Edit: I also use Optane in NVMe form factor on a couple other Proxmox installs on Intel-based consumer PCs, and in those cases I'm using it via 'lvmcache' as cache for the LVM thin pool where guest disks live, since that pool is on slower storage. Again, works quite well, though I haven't done rigorous testing.

I've seen some things about how Intel CPUs with built-in support were required for some types of Optane devices, but this doesn't seem to apply to all Optane products.

Get them now, new old stock is selling out and once it's gone it's gone.

 

[LnxBil]

I've seen some things about how Intel CPUs with built-in support were required for some types of Optane devices, but this doesn't seem to apply to all Optane products.

AFAIK, this was also only for Windows-based operating system in which the optane and the local harddisk was "merged" into one disk. We used it a lot back in the days for speeding up harddisk/optane combinations, yet this technology did not work properly with full-ssd systems.