Disk cache/memory issues ZFS on RAID card.

John Morrison

Well-Known Member
Feb 26, 2018
I have a PERC H730 Mini in HBA mode on Dell R730.

I have all 8 disks in HBA, not using the RAID at all.

Set up using ZFS 4 way mirror.

I am using writeback for the vm disks, SCSI virtIO.

When should the VM disk's cache get flushed? As proxmox is constantly running out of physical memory due to big writes of data in the VMs. But Unless you delete that data or reboot the VM machine that wrote that data, the physical memory is returned immediately.

I find this behavior strange.

Is using a RAID card in HBA mode with ZFS on top bad due to this problem with disk caches?

• data will be cached in the host-side page cache if the cache mode is set to

• data will be immediately flushed to the physical disk cache if the cache mode
is set to none;

• data will be immediately flushed to the physical disk platters if the cache mode
is set to writethrough.

Cache-ing mode is only one tiny part of zfs .... ;)

Cache TypeUses Host Page CacheGuest Disk "WCE" on

WCE=write cache enabled

Source: https://wiki.mikejung.biz/KVM_/_Xen

Generally speaking, with only cache=none(and raw disk format, default in PMX+zfs) you will not cache the same data on both zfs cache AND at the guest cache level.

Good luck!
Last edited:
  • Like
Reactions: John Morrison
but by default the disks don't seem to be raw

bootdisk: scsi0
cores: 8
cpu: host
ide2: none,media=cdrom
memory: 131076
name: storage1
net0: virtio=22:97:A8:69:F6:94,bridge=vmbr0,tag=1
net1: virtio=CE:AF:C7:22:C5:1E,bridge=vmbr0,tag=1
numa: 1
onboot: 1
ostype: l26
scsi0: local-zfs:vm-103-disk-1,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=20G,ssd=1
scsi1: /dev/disk/by-id/nvme-INTEL_SSDPEKNW020T8_PHNH928300TU2P0C,size=2000398680K
scsi2: local-zfs:vm-103-disk-5,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi3: local-zfs:vm-103-disk-6,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi4: local-zfs:vm-103-disk-7,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi5: local-zfs:vm-103-disk-9,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi6: local-zfs:vm-103-disk-10,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi7: local-zfs:vm-103-disk-11,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi8: local-zfs:vm-103-disk-0,backup=0,cache=directsync,discard=on,iothread=1,replicate=0,size=5G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=2062fe20-1236-45a7-bc07-0c6756fd1ba5
sockets: 2


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!