ZFS Mirror - very low performance

[ligistx]

Hi,

A few months ago I migrated from a single proxmox instal to a mirrored array using a supermicro pcie dual slot nvme card 9my motherboard doesn't have native physical nvme support), and ever since, I have noticed very slow VM perofmrnace. This is a homelab, so it isn't "really" an issue, but even just apt upgrade on simple ubuntu server VM's takes a while, and if I kick a bunch off at once, things really slow down. This was not an issue at all when I was using a single nvme drive and not using ZFS.

I do notice really high io delay, and being almost entirely a noob, I assume this is related? I store my VM's on the same array as the boot drives, and have reduced ZFS's RAM usage manually since it was eating up all my systems RAM for no real reason (again, this is a homelab, I really don't need ARC acceleration on a pair of NVMe drives that do almost not raeding and writting anyways...). I believe I set ARC to either 4 or 8 GB, which should be plenty...

The screenshot below was while I was running apt update on 3 ubuntu server VM's. Any ideas why or what is causing this? Its certainly not the end of the world, but it is a bit annoying as this machine should not be slow at anything I throw at it.

 

[Azunai333]

What NVMe models and dual slot NVMe card do you use?

 

[Maximiliano]

Hello,

What storage type are you using for your VMs?

You can list the storage types with `pvesm status` and see which of them is being used by a guest's disk in its config, e.g. in `/etc/pve/qemu-server/VM_ID.conf`.

 

[ligistx]

What NVMe models and dual slot NVMe card do you use?

Supermicro AOC-SLG3-2M2 with Samsung 980’s. I was using this same PCIe card and a single 980 prior to switching to ZFS and performance was fantastic. I did have to excitement with different BIOS bifrication settings to get both drives to work, but I’d imagine even if somehow I ended up with 1x PCIe speeds (it shouldn’t have, but assuming the worst…) that should still be plenty for some Ubuntu apt updates to work fine as that’s mostly random IO, and super low bandwidth. But I am clearly just grasping at straws.

Hello,

What storage type are you using for your VMs?

You can list the storage types with `pvesm status` and see which of them is being used by a guest's disk in its config, e.g. in `/etc/pve/qemu-server/VM_ID.conf`.

I am not entirely sure what I am looking for here. I get:

root@pve:~# pvesm status
Name             Type     Status           Total            Used       Available        %
PBS               pbs     active       304745344       164211328       140534016   53.88%
ext4tb            dir     active       387671680         3549952       384121728    0.92%
local             dir     active       387671680         3549952       384121728    0.92%
local-zfs     zfspool     active       467758116        83636184       384121932   17.88%

(Note: ext4tb is not currently connected, local I believe is the local volume proxmox set up, local-ZFS is where I set up all of my proxmox storage to include containers, vm's, ISO's, etc)

I get this when checking one of my VM's (they should all be using the same storage)

agent: 1
balloon: 1536
boot: order=ide2;scsi0;net0
cores: 2
cpu: x86-64-v3
ide2: none,media=cdrom
memory: 2048
meta: creation-qemu=6.1.0,ctime=1648790409
name: xxxxx
net0: virtio=xxxxxx,bridge=vmbr0,firewall=1,tag=xx
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-601-disk-0,discard=on,size=16G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=62389444-4da5-4798-8e6b-74924b8bfd64
sockets: 1
startup: order=5
usb0: host=0bda:2832
vmgenid: 72bd7481-628a-474f-8390-d81cc47d6965

 

[nsc117]

mark，the same issus

 

[VDNKH]

Hi @ligistx, Have you tried making a trim to rpool?

the command is:
zpool trim rpool
to notify of blocks that are no longer in use.

When using ZFS on SSDs without power loss protection, TRIMs must be configured at least once every week, otherwise you will always suffer from very high IO delay.

 

[ligistx]

Hi @ligistx, Have you tried making a trim to rpool?

the command is:
zpool trim rpool
to notify of blocks that are no longer in use.

When using ZFS on SSDs without power loss protection, TRIMs must be configured at least once every week, otherwise you will always suffer from very high IO delay.

Hi,

I have, and it autoruns every week (or two? whatever teh default behavior is). It doens't seem to help.

Thankfully, this really isn't that big of an issue as all VM's seem to run fine and are happy, nothing I do hits the host or VM's very hard IO wise, so it mostly doesn't matter. I was just trying to figure out what it could be.

I had pretty good in depth help on another forum (level1techs), but that also didn't ultimitely go anywhere. Entirely not sure why I am observing the behavior I am observing...