Disk cache/memory issues ZFS on RAID card.

[John Morrison]

I have a PERC H730 Mini in HBA mode on Dell R730.

I have all 8 disks in HBA, not using the RAID at all.

Set up using ZFS 4 way mirror.

I am using writeback for the vm disks, SCSI virtIO.

When should the VM disk's cache get flushed? As proxmox is constantly running out of physical memory due to big writes of data in the VMs. But Unless you delete that data or reboot the VM machine that wrote that data, the physical memory is returned immediately.

I find this behavior strange.

Is using a RAID card in HBA mode with ZFS on top bad due to this problem with disk caches?


• data will be cached in the host-side page cache if the cache mode is set to
writeback;

• data will be immediately flushed to the physical disk cache if the cache mode
is set to none;

• data will be immediately flushed to the physical disk platters if the cache mode
is set to writethrough.

 

[guletz]

Hi,

Cache-ing mode is only one tiny part of zfs ....

Cache Type	Uses Host Page Cache	Guest Disk "WCE" on
WriteBack	YES	YES
None	NO	YES
WriteThrough	YES	NO
DirectSync	NO	NO

WCE=write cache enabled

Source: https://wiki.mikejung.biz/KVM_/_Xen

Generally speaking, with only cache=none(and raw disk format, default in PMX+zfs) you will not cache the same data on both zfs cache AND at the guest cache level.

Good luck!

 

[John Morrison]

but by default the disks don't seem to be raw

bootdisk: scsi0
cores: 8
cpu: host
ide2: none,media=cdrom
memory: 131076
name: storage1
net0: virtio=22:97:A8:69:F6:94,bridge=vmbr0,tag=1
net1: virtio=CE:AF:C7:22:C5:1E,bridge=vmbr0,tag=1
numa: 1
onboot: 1
ostype: l26
scsi0: local-zfs:vm-103-disk-1,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=20G,ssd=1
scsi1: /dev/disk/by-id/nvme-INTEL_SSDPEKNW020T8_PHNH928300TU2P0C,size=2000398680K
scsi2: local-zfs:vm-103-disk-5,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi3: local-zfs:vm-103-disk-6,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi4: local-zfs:vm-103-disk-7,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi5: local-zfs:vm-103-disk-9,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi6: local-zfs:vm-103-disk-10,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi7: local-zfs:vm-103-disk-11,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=350G,ssd=1
scsi8: local-zfs:vm-103-disk-0,backup=0,cache=directsync,discard=on,iothread=1,replicate=0,size=5G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=2062fe20-1236-45a7-bc07-0c6756fd1ba5
sockets: 2

 

[guletz]

(and raw disk format

read as block device, sorry for my missleading(my bad english)