ESXi 8 VM on Proxmox - Memory Usage Question

[stoney777]

I have managed to import and also install ESXi as a VM inside of Proxmox however I have a question with regards to memory usage. My physical host has 32GB of RAM. I wish to install multiple instances of ESXi for lab/troubleshooting purposes and recreating different scenarios within ESXi.

When it comes to memory usage, it appears that Proxmox is allocating the full amount of RAM to the ESXi VM. This means I will quickly run out of RAM as I am assigning 16GB to ESXi VM's. I have enabled memory ballooning and set the minimum to 4GB and max 16GB however this doesn't appear to have made a difference.

When I check the summary view of the ESXi VM it shows as using 98% of the allocated RAM. I know this isn't 100% true as the ESXi VM in theory is sitting idle. If I run ESXTOP from a console session, I can indeed see 16,378MB assigned, but it also shows as 14,686MB free.

Of course, I would like Proxmox to be able to reclaim this free RAM and use it elsewhere. Is this possible?

 

[BobhWasatch]

I have managed to import and also install ESXi as a VM inside of Proxmox however I have a question with regards to memory usage. My physical host has 32GB of RAM. I wish to install multiple instances of ESXi for lab/troubleshooting purposes and recreating different scenarios within ESXi.

You have a problem then.

When it comes to memory usage, it appears that Proxmox is allocating the full amount of RAM to the ESXi VM.

What do you think it should do? More importantly, what do you think it should do after it has been running for a while with the guest allocating memory to its guests?

BTW, if you are passing any PCI devices into a VM, then it _must_ allocate and commit all of the memory up-front because of DMA.

This means I will quickly run out of RAM as I am assigning 16GB to ESXi VM's. I have enabled memory ballooning and set the minimum to 4GB and max 16GB however this doesn't appear to have made a difference.

Ballooning doesn't kick in until the host's memory usage is >= 80%.

But really, you still have a problem. What is supposed to happen if an ESXi VM needs the full 16 GB because it has all been allocated to its VM's?

What you are asking for is to over-commit memory. The only way that can really be done is to enable a lot of swap. But that will be very slow. Using zRAM to compress some of your PVE machine's memory might help a bit as well. Probably not enough to have more than two 16 GB instances of ESXi though.

 

[stoney777]

You have a problem then.


What do you think it should do? More importantly, what do you think it should do after it has been running for a while with the guest allocating memory to its guests?

BTW, if you are passing any PCI devices into a VM, then it _must_ allocate and commit all of the memory up-front because of DMA.


Ballooning doesn't kick in until the host's memory usage is >= 80%.

But really, you still have a problem. What is supposed to happen if an ESXi VM needs the full 16 GB because it has all been allocated to its VM's?

What you are asking for is to over-commit memory. The only way that can really be done is to enable a lot of swap. But that will be very slow. Using zRAM to compress some of your PVE machine's memory might help a bit as well. Probably not enough to have more than two 16 GB instances of ESXi though.

thanks for your informative reply. I guess this is where my knowledge of Proxmox is fairly limited, coming from an ESXi background. However in saying that, there is a reason why these Nucbox/ Mini PCs are so popular with the ESXi homelab community and it’s purely because they pack a fair bit of punch for an affordable price allowing you to run numerous VMware related workloads for lab/training purposes.

Some workloads require multiple ESXi hosts and appliances and are known to run comfortably well on the Mini PCs as they obviously over commit the RAM.

I guess i need to better understand the difference with ballooning and over commitment, and possibly even increasing swap size

 

[BobhWasatch]

thanks for your informative reply. I guess this is where my knowledge of Proxmox is fairly limited, coming from an ESXi background.

There is no magic. You can't use more resources than you have.

However in saying that, there is a reason why these Nucbox/ Mini PCs are so popular with the ESXi homelab community and it’s purely because they pack a fair bit of punch for an affordable price allowing you to run numerous VMware related workloads for lab/training purposes.

PVE runs fine on mini-PC's. Lots of people here do that. You just can't use more resources than you have. Do you really need all of these VM's running at the same time? There's no issue with setting up 50 VM's that need 16 GB each. You just won't be able to start more than one or two of them at a time.

I don't use ESXi but I am quite sure it is also not able to use resources it does not have.

 

[LnxBil]

What you are asking for is to over-commit memory. The only way that can really be done is to enable a lot of swap. But that will be very slow. Using zRAM to compress some of your PVE machine's memory might help a bit as well. Probably not enough to have more than two 16 GB instances of ESXi though.

In addition, there could also KSM kick in in order to have a couple of bytes deduplicated, yet this is also just a couple of drops in the ocean.

 

[stoney777]

There is no magic. You can't use more resources than you have.


PVE runs fine on mini-PC's. Lots of people here do that. You just can't use more resources than you have. Do you really need all of these VM's running at the same time? There's no issue with setting up 50 VM's that need 16 GB each. You just won't be able to start more than one or two of them at a time.

I don't use ESXi but I am quite sure it is also not able to use resources it does not have.

You can’t use resources you dont have but you can heavily over commit.

Right now in proxmox i have 5 ESXi VMs running. Each VM has been assigned 8GB of RAM. So thats 40GB in total. My machine only has 32GB.

theyre all powered on and performance is fine so far…. For now!

 

[Johannes S]

You can’t use resources you dont have but you can heavily over commit.

This is true for CPU and storage space ( allthough limited for storage ) but not for RAM

 

[stoney777]

This is true for CPU and stirage space ( allthough limited for storage )but not for RAM

On ESXi or on Proxmox?

If Proxmox, you can't over commit at all, or you can't heavily over commit?

If ESXi, an excerpt from their documentation:

Because of the memory management techniques the ESXi host uses, your virtual machines can use more virtual RAM than there is physical RAM available on the host. For example, you can have a host with 2GB memory and run four virtual machines with 1GB memory each. In that case, the memory is overcommitted. For instance, if all four virtual machines are idle, the combined consumed memory may be well below 2GB. However, if all 4GB virtual machines are actively consuming memory, then their memory footprint may exceed 2GB and the ESXi host will become overcommitted.

Overcommitment makes sense because, typically, some virtual machines are lightly loaded while others are more heavily loaded, and relative activity levels vary over time.

Let me stress again, my use case here is not production. My use case is not performance. My use case is for troubleshooting and recreating, as close as possible, real life scenarios that appear in production ESXi environments

 

[Johannes S]

On ESXi or on Proxmox?

If Proxmox, you can't over commit at all, or you can't heavily over commit?

In theory you can overcommit in practice overcommiting CPU works but not RAM. For storage space you can overcommit with thin-provisioning, but of course you would then need to take care, that the actual used storage space by the guests is still within the limits of your available storage space.

Proxmox is not ESXi (which has pros and cons), for RAM it's best to give the guests just the needed memory plus a small margin.

 

[BobhWasatch]

Right now in proxmox i have 5 ESXi VMs running. Each VM has been assigned 8GB of RAM. So thats 40GB in total. My machine only has 32GB.

That only works because they aren't actually using all of their RAM (yet). When they do you will either need to have swap space available or the OOM-killer will zap one.

If they don't ever use all of their RAM and everything is fine forever, that just says you gave them too much RAM.