SECURITY: PVE can be crashed by user on demand

krzykacz

Member
Nov 14, 2018
8
0
6
43
Hello everyone,

Using swap on ZFS is bad, everyone knows that. But how bad? We can put this to the test!

Let's suppose that PVE node has 32GB of RAM.
Let's suppose that no VM is running on it. If you ARC is free you have plenty of memory available (30GB-31GB).
Install stress tool (apt install stress).
Create unprivileged user and log into his account.
Run stress command to allocate little more memory than is available, this should invoke swapping pages.

$ stress -m 4 --vm-keep --vm-bytes 7700M

And now interesting part: After few seconds you will see machine hard reset. Reset will be without ANYTHING in log files / console etc.

For me this is serious security problem. Anyone can allocate a lot of memory fast. This should lead to swapping and OOM kills - not unexpected host resets!

A was able to invoke this issue on few different hosts. (some of them were in VM). I was using clean 5.2 Proxmox from ISO install or machine was upgraded to recent public version (free repo).

ARC size – seems to be irrelevant (here limited to 8GB)
ZFS group tx timeout was set to 2s/5s
Swap on zfs (rpool/swap) was tested with sync always/disabled, compression zle/off
Swappiness - 0 / 10 / 60

Main problem is that if You want to use ZFS or soft RAID you are forced to use this filesystem. Setup will always create swap on rpool/swap. So default configuration seems to be very very dangerous!

If You turn of swap, host will survive (is some way - OOM). When swap is on regular disk everything works as expected.

Maybe someone know how to tune swap on ZFS to avoid such crashes?

Is there a way to move swap partition outside rpool and put it directly on disks like boot partitions? (I know that ZFS like to have exclusive access but this idea is worth trying)
 
Thanks Tom! This seems to be very similar, if not exactly same problem.

Users needing swap can either leave unpartitioned space, by setting minfree, or create the zvol quite easily after installation

And here I have problem. In Proxmox setup I cannot set "minfree". It dissapears when I choose ZFS filesystem. Any idea how to install PVE with a little bit of free space for swap outside ZFS rpool?
 
Thanks Tom! This seems to be very similar, if not exactly same problem.



And here I have problem. In Proxmox setup I cannot set "minfree". It dissapears when I choose ZFS filesystem. Any idea how to install PVE with a little bit of free space for swap outside ZFS rpool?

You will see this fix in the upcoming Proxmox VE 5.3 ISO, available soon.
 
Last edited:
The installer creates a third small partition at the end of the drive(s) on ZFS installs. It appears it is unused. I wonder if it can be repurposed as swap partition.
 
on a dataset an not on a zvol!
Hi,
but it's doesn't work (sync?)
Code:
root@pvetest:~# mkswap /swapfile
Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)
no label, UUID=62473471-6ec1-4b40-8824-f58965a6ef58
root@pvetest:~# swapon -a
swapon: /swapfile: swapon failed: Invalid argument

root@pvetest:~# swapon /swapfile
swapon: /swapfile: swapon failed: Invalid argument
Udo
 
You will see this fix in the upcoming Proxmox VE 5.3 ISO, available soon.
Hi,
and what is the official recommendation?

To use free space on an single disk as swap looks not very secure for me.
Use an (before not supported) md-raid on the free space?
I've tried swap on md-device and it's work without panic/reboot.

Buy much more ram is often not an alternative... and the swap was mostly used by VMs with data, which are not used (the host has an swapiness of 1).

Udo
 
Does creating a SWAP file or SWAP partition inside KVM VM residing on ZVOL and running memory stress test inside it, also cause the problems on host, if swap is on ZVOL on host?
 
Swapfile on ZFS isnt Supported: https://wiki.archlinux.org/index.php/ZFS#Swap_volume
ZFS does not allow to use swapfiles, but users can use a ZFS volume (ZVOL) as swap.

You can do it as follow (for example):
zfs create -V 12G rpool/swap

mkswap /dev/zvol/rpool/swap
swapon /dev/zvol/rpool/swap

Does creating a SWAP file or SWAP partition inside KVM VM residing on ZVOL and running memory stress test inside it, also cause the problems on host, if swap is on ZVOL on host?
Normally i think not. Only if the RAM of the whole Node are not enough to get it and the Host beginns to use the swap.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!