Proxmox server dies when transferring files to VM

[holckj]

Today I tried to transfer a lot of photos (5 GB) from a disk, mounted via sshfs, to one of my VMs. This caused the Proxmox host to put full blow on its fan, and it became totally unresponsive. On the screen it showed messages about being out of memory, and killing and reaping kvm.
After 10 minutes like this. I switched off power to the server.

I could start the servers again - both the host and the VM - and everything seems to be ok now.

The Proxmox host mostly runs ZFS, but the disk with the photos was a HDD with ext4, mounted locally on the host and via sshfs on the VM.

What happened? What's the best way to deal with a situation like this? Should I just wait for the server to come back?

/Jesper, Denmark

 

[Dunuin]

You shouldn't overprovision your RAM in the first place. Keep in mind that ZFS by deault will use up to 50% of your hosts total RAM. Then PVE needs 2 GB RAM and all VMs will need more RAM then you assign to them because of virtualization overhead. A 4GB RAM VM might actually need 5GB of RAM and so on. And you always want some free RAM for caching and to prevent workload spikes to cause the OOM killer to trigger.
If you installed your PVE on a ZFS pool you also don't got swap by default. In case you got some unallocated space on one of your disks you could create a swap partition to prevent such OOM situations.

 

[holckj]

You shouldn't overprovision your RAM in the first place. Keep in mind that ZFS by deault will use up to 50% of your hosts total RAM. Then PVE needs 2 GB RAM and all VMs will need more RAM then you assign to them because of virtualization overhead. A 4GB RAM VM might actually need 5GB of RAM and so on. And you always want some free RAM for caching and to prevent workload spikes to cause the OOM killer to trigger.
If you installed your PVE on a ZFS pool you also don't got swap by default. In case you got some unallocated space on one of your disks you could create a swap partition to prevent such OOM situations.

Thanks for your info. Yes, to little RAM is probably some of the problem. I have 32 GB, that may no be enough but I believe it's all my server can hold (HP Proliant Gen10 Micro Server). I have a HDD with extra space, but now it only holds the ext4 partition. I guess I can reduce this partition and add a swap-partition. What do you think of adding a USB thumb drive and use this for swap?
Do you have suggestions for how to handle the OOM situations? Let the server keep on running or switch if off?

 

[Dunuin]

What do you think of adding a USB thumb drive and use this for swap?

Swap is heavy writing and will kill a USB stick or SD card quite fast.

Do you have suggestions for how to handle the OOM situations?
Let the server keep on running or switch if off?

Add more RAM and if that isn't possible add swap. KSM might also save some GBs of RAM through deduplication. and don't allocate the guests more RAM they actually really need. Guests will waste all RAM for caching you throw at them.

 

[UdoB]

Good morning,

I believe it's all my server can hold (HP Proliant Gen10 Micro Server)

I added one single 32 GiB to the single 16 GiB SIMM I had and it is working fine. Search for "PHS-memory 32GB RAM Speicher passend für HP ProLiant MicroServer Gen10 (G10) X3216 DDR4 UDIMM ECC 2666MHz PC4-2666V-E".

Despite of the high price increasing Ram is the cheapest way to raise the capabilities of my server. I am thinking to go 2*32GiB but sadly one has to trash the payed 16 GiB module then...


Best regards

 

[holckj]

Thanks for all your input. I guess more RAM is the way to go. But on the short term, will it make sense to limit ZFS ARC, in order to keep the VMs running and prevent these OOM situations? Something like

echo "8589934592" > /sys/module/zfs/parameters/zfs_arc_max

I much prefer living with slightly lowered performance to experiencing OOM situations where the server becomes totally unresponsive.

 

[UdoB]

will it make sense to limit ZFS ARC, in order to keep the VMs running

Yes, sure, that might work.

Bear in mind that reducing ARC will reduce "cache hits". But the performance impact might be completely fine for your use case. You may watch arc_summary (look for "hits + misses") before and after setting the limits. (It is a cache - it needs to warm up, so give it some time...)

 

[holckj]

Good morning,


I added one single 32 GiB to the single 16 GiB SIMM I had and it is working fine. Search for "PHS-memory 32GB RAM Speicher passend für HP ProLiant MicroServer Gen10 (G10) X3216 DDR4 UDIMM ECC 2666MHz PC4-2666V-E".

Despite of the high price increasing Ram is the cheapest way to raise the capabilities of my server. I am thinking to go 2*32GiB but sadly one has to trash the payed 16 GiB module then...


Best regards

Thanks for the info regarding RAM for the MicroServer Gen10. I now shifted one of my 16 GB blocks to a new 32 GB. They work perfect together, giving me 48 GB.

It's a bit odd to think about that when I bought the server, I knew from previous that I probably would need all the RAM I could get. So I filled the server to the maximum - 32 GB. Or at least that was what the specs said. But now it hums along with 48 GB ...