Out of memory crash during hdd speedtest in vm

[espencer]

Hi,

I'm new to proxmox and currently I have 1 windows 10 vm which I'm using for testing.

When I run a hdd benchmark using crystaldiskmark or ATTO in the windows 10 vm and monitor ram usage on the host with htop I can see ram completely filling up during the write tests.
Because there isn't enough ram available, processes are being killed to free memory.
This makes the pc completely hang and I have to use the physical power button to reset it.
When the host is back up, I can see messages about out of memory in the syslog.

I first thought this was because of zfs caching the writes, but if I understood correctly this shouldn't be the problem because only 22GB should be used and 10GB should still be free. (2GB for proxmox itself + 4GB for the vm + 16GB (by default half the total amount of ram) for ZFS)
So I tried to limit ZFS down to 8GB ram to see if it would make a difference, but the problem still happens.

Then I tried to add a swap zvol to see if this would help, but the host crashes before swap is being used.

The only way I have found to prevent this problem is to use "write back" as the hard disk cache option.
If I run the benchmark again with "write back", I can see that an additional ± 1GB of ram is used.
So in total about 7GB ram is occupied. (2GB for proxmox itself + 4GB for the vm + 1GB write cache?)

I have read https://pve.proxmox.com/wiki/Performance_Tweaks but I still don't fully understand the differences between all the options.
Also write back (unsafe) is missing on that page so I don't know what the difference between "write back" and "write back (unsafe)" is.

After some googling I found that using the "write back" option is not recommended in my situation because I don't have multiple powersupply's, a ups, raid card with battery backed cache, etc.

Can someone tell me:
1) Why using "default (no cache)" completely fills up all ram?
2) What I can do so I can use "Default (no cache)" as the disk cache option?
3) If I configured something wrong or forgot to configure something?

Spoiler: Host specs

Gigabyte x570 aorus master
Ryzen 7 3700x
Corsair Vengeance LPX 32GB (2x16GB)
4x WD Blue HDD 4TB
1x Samsung 850 SSD 250GB
1x Crucial MX500 SSD 250GB

Spoiler: /etc/pve/qemu-server/100.conf with "default (no cache)"

bootdisk: scsi0
cores: 4
ide2: local:iso/Windows10-1909.iso,media=cdrom
memory: 4096
name: test
net0: virtio=8A:BA:13:A8:89:A9,bridge=vmbr0,firewall=1,link_down=1
numa: 0
ostype: win10
sata0: local:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K
scsi0: local-zfs:vm-100-disk-0,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=83de4fd5-dc76-463f-a646-51e3f3956357
sockets: 1
vmgenid: 4129bbac-8512-44a7-80e3-8b4e4dc4150d

Spoiler: /etc/pve/qemu-server/100.conf with "write back"

bootdisk: scsi0
cores: 4
ide2: local:iso/Windows10-1909.iso,media=cdrom
memory: 4096
name: test
net0: virtio=8A:BA:13:A8:89:A9,bridge=vmbr0,firewall=1,link_down=1
numa: 0
ostype: win10
sata0: local:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K
scsi0: local-zfs:vm-100-disk-0,cache=writeback,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=83de4fd5-dc76-463f-a646-51e3f3956357
sockets: 1
vmgenid: 4129bbac-8512-44a7-80e3-8b4e4dc4150d

 

[spaxxilein]

I have the same issue except i have 128GB of Ram.

 

[espencer]

Can someone please help with this issue?

 

[bpaiva]

Me too.. 4TB HDD/ZFS + Windows Server 2019 + Intensive Writes = Proxmox crash and reboot...

 

[spaxxilein]

Me too.. 4TB HDD/ZFS + Windows Server 2019 + Intensive Writes = Proxmox crash and reboot...

Do you have Proxmox installation on the same pool? We fixed proxmox freezing by installing proxmox to seperate volume, which is not affected by the extensive read/write of the pools.

Before we had crashes, now we get VMs stuck for long time, but Proxmox runs just fine.

 

[bpaiva]

Do you have Proxmox installation on the same pool? We fixed proxmox freezing by installing proxmox to seperate volume, which is not affected by the extensive read/write of the pools.

Before we had crashes, now we get VMs stuck for long time, but Proxmox runs just fine.

Yes i have instalation of proxmox on same pool, already disable Balloning but remains the same... will try Proxmox on separated disk to. Thanks

 

[bpaiva]

Same thing with, new 6.2 instalation on HDD, create a 4TB ZFS Mirror, import VM from backups,boot WindowsServer 2019 VM, testspeed to disk and hangs proxmox on Write tests....

 

[pcready.cl]

Same thing with, new 6.2 instalation on HDD, create a 4TB ZFS Mirror, import VM from backups,boot WindowsServer 2019 VM, testspeed to disk and hangs proxmox on Write tests....

Same here proxmox 6.2

 

[DemiNe0]

Same here as well, proxmox 7.0

 

[Felix.]

Can confirm. Since the upgrade to 7.0, the proxmox host is not crashing and the VM does no longer get killed by OOM.
But the VM freezes very easily on "heavy" IO like a multi-gigabyte download of a file.
The VM Storage is separated from the OS install, different ZFS pools, all SSD.

 

[pcready.cl]

Can confirm. Since the upgrade to 7.0, the proxmox host is not crashing and the VM does no longer get killed by OOM.
But the VM freezes very easily on "heavy" IO like a multi-gigabyte download of a file.
The VM Storage is separated from the OS install, different ZFS pools, all SSD.

Same here, all SSD.

 

[spirit]

Same here, all SSD.

ssd model ?

 

[pcready.cl]

Several models:

• Kingston SKC600, 512GB / 1TB / 2TB
• Crucial MX500, 1TB / 2TB
• WD SSD Blue, 1TB / 2TB
• WD SSD Green, 1TB / 2TB
• Kingston A400, 1TB

Thanks.

 

[Felix.]

Mostly Intel SSDSC2BB480G6, 480GB DC SSDs.
Some Crucial CT2000BX500SSD1, 2TB Consumer SSD.
Happens with both of those.

 

[DemiNe0]

So for me this happened on HDD's. My ZFS pool is a zfs raid1 mirror between two 6tb hdd's:
HGST_HUS726T6TALE6L1

root@pve01:~# zpool status
  pool: zfs01
 state: ONLINE
  scan: scrub repaired 0B in 00:02:29 with 0 errors on Sun Sep 12 00:26:30 2021
config:

        NAME                                   STATE     READ WRITE CKSUM
        zfs01                                  ONLINE       0     0     0
          mirror-0                             ONLINE       0     0     0
            ata-HGST_HUS726T6TALE6L1_V9HPGRLL  ONLINE       0     0     0
            ata-HGST_HUS726T6TALE6L1_V9HPAL0L  ONLINE       0     0     0

errors: No known data errors

Thinking back on this... my experience was a bit different. I was running crystalmark on the zfs pool above and noticed that proxmox had locked up. SSH stopped responding as well. IPKVM in and reboot. It hangs at

a start job is running for import zfs pools by cache file

I let it hang up to an hour at one point, however It wouldn't go past that point. Tried chrooting the lvm with a ubuntu liveiso and the proxmox iso, however I couldn't get the lvm root to mount for some reason.

Decided to just wipe it and redo it at that point. Wiped and upon logging in for the first time I get another warning about the zpool failing to load. zpool import would say nothing to import and zpool status would hang. I rebooted and it started hanging again at

a start job is running for import zfs pools by cache file

Frustrated, I went to bed and in the morning I wake up, proxmox is booted and zpool status lists the pool at online. what. the. ffffffffffff. lol This is so weird. I guess the zfspool self healed itself? I noticed that after I reattached the pool to proxmox that vm hard drive file that I was hdd speed testing too was completely full (3tb) and wasn't having it's deleted data discorded. Ended up just deleting the drive since I didn't need it anymore.

Can also add that I tested this later with cache set to writethrough. I don't experience the lockup or crash anymore.

 

[chrcoluk]

I have had OOM's during write phase of VM benchmark, I suggest disabling transparent huge pages, and if you have less than 64 gig of ram, also consider reducing zfs dirty cache size.

 

[pcready.cl]

I have had OOM's during write phase of VM benchmark, I suggest disabling transparent huge pages, and if you have less than 64 gig of ram, also consider reducing zfs dirty cache size.

Hi, how do this:

1. I suggest disabling transparent huge pages
2. consider reducing zfs dirty cache size.

Thanks bro.

 

[chrcoluk]

Hi.

Add 'transparent_hugepage=never' to kernel cmdline. which is /etc/kernel/cmdline if using proxmox-boot-tool, then 'proxmox-boot-tool refresh' to apply changes and reboot.

For zfs write cache you can reduce with the variable 'zfs_dirty_data_max' it defaults to 10% of ram or 4 gigs, whichever is lower. The documents state it should be 10% regardless of amount of ram but my testing has shown it caps at 4gigs, which is why I think on 64 gigs and higher its safer, as 4 gigs on those machines is a lower % of total ram.

To adjust it add 'options zfs zfs_dirty_data_max=value in bytes' to /etc/modprobe.d/zfs.conf, run 'update-initramfs -u -k all' and reboot.

You can also adjust live by 'echo value in bytes > /sys/module/zfs/parameters/zfs_dirty_data_max'

On my local proxmox I capped it to 5%, 1.6 gigs.

Also adding a swap works wonders at mitigating fragmentation based OOM's. If you add a swap you are probably safe with default sized zfs dirty cache, although I would still disable THP.

 

[PLANTROON]

Hi.

Add 'transparent_hugepage=never' to kernel cmdline. which is /etc/kernel/cmdline if using proxmox-boot-tool, then 'proxmox-boot-tool refresh' to apply changes and reboot.

For zfs write cache you can reduce with the variable 'zfs_dirty_data_max' it defaults to 10% of ram or 4 gigs, whichever is lower. The documents state it should be 10% regardless of amount of ram but my testing has shown it caps at 4gigs, which is why I think on 64 gigs and higher its safer, as 4 gigs on those machines is a lower % of total ram.

To adjust it add 'options zfs zfs_dirty_data_max=value in bytes' to /etc/modprobe.d/zfs.conf, run 'update-initramfs -u -k all' and reboot.

You can also adjust live by 'echo value in bytes > /sys/module/zfs/parameters/zfs_dirty_data_max'

On my local proxmox I capped it to 5%, 1.6 gigs.

Also adding a swap works wonders at mitigating fragmentation based OOM's. If you add a swap you are probably safe with default sized zfs dirty cache, although I would still disable THP.

This helps prevent the OOM situation which led to a complete system freeze (unresolvable by any means other than rebooting), but unless you limit the guest's IOPS and/or bandwidth, it will still lead to a stall while the written data is flushed to disk(s) (I didn't debug this thoroughly, so I am just guessing what is happening). Therefore I completely abandoned ZFS and just stuck with md RAID and LVM volumes which do not have these problems. If I had 100% control over what the VM is going to be doing, I'd happily use ZFS.

 

[hhoehle]

Hi all,

I think I see a similar issue: Proxmox system crashes always with OOM when running CrystalDiskMark 8.0.4 write test on VirtIO (virtio cache=default (no cache)) NTFS VM-disk for VM Win2019 server. On the proxmox node 8 GB RAM swap is configured. Even with THP=never in grub config validated with cat /sys/kernel/mm/transparent_hugepage/enabled the problem persists. The vm-disk has 10 TB free space.

However, when using virtio cache= write back for the VM-disk the system does not crashes any more.

I have almost no linux experience, to me it seems that the default IO memory handling for virtio cache=default does not stop the IO-data flow and/or does no manage the IO buffer size vs the available system memory correct.

I am wondering if this is a bug or intended behavior and if there is a way to get this fixed/changed?

If I copy a big (5G) file on that disk it pushes the IO delays in the proxmox node GUI to 90% (red) or more. In this case the data rate goes down for some time even to something close to zero probably until the cache has been written. I would assume that the average file copy datarate is close the limit given by the zfs system and the underlying HW. in my case this is app. 100MB/s. Is this an appropriate value for a zfs RAID1 mirror with 2 HDs with a specified interface data rate of 255Mb/s each?



---------------------------------------------------------------------------------------
Proxmox 7.1-8 on supermicro X11SCL-IF – Xenon E-22260 3.4 GHz – 32 GB RAM – System/VMs on Intel SATA SSD S4510 – data on zfs raid1 mirror with 2 x HDD WD red pro 14 TB