Help with slow ZFS performance?

[79corvette]

I have a single node with 2 ZFS pools. I am trying to restore a 2gb LXC backup which is stored on one of the ZFS pools, and after 15+ min it still does not complete. Its been awhile since I have had to restore a backup but I dont think it was nearly this slow in the past.

I restarted the host, and even that process was slow and took several minutes. It looks like the cause of the slow restart was mostly due to mounting the ZFS pool, see errors below. When I was setting the system up a year or so ago, many restarts and restores were quite fast. I haven't done much in the past 8-9 months and am not sure where to start troubleshooting

I'm still researching and reading various old threads, but would appreciate any tips on what to troubleshoot.

These are the first things I have checked:

root@pve:~# journalctl -b 0 -p 3

Jun 29 17:25:47 pve kernel: x86/cpu: SGX disabled by BIOS.
Jun 29 17:25:47 pve kernel: ACPI BIOS Error (bug): AE_AML_BUFFER_LIMIT, Field [CAP1] at bit offset/length 64/32 exce>
Jun 29 17:25:47 pve kernel: ACPI Error: Aborting method \_SB._OSC due to previous error (AE_AML_BUFFER_LIMIT) (20230>
lines 1-3/3 (END)

root@pve:~# systemd-analyze blame

1min 12.153s zfs-import@bkup_extZFS.service
1min 12.152s zfs-import@dataZFS.service
     19.956s pve-guests.service
     15.648s ifupdown2-pre.service
     15.647s systemd-udev-settle.service
     13.039s smartmontools.service
      5.194s zfs-import-scan.service
      1.495s systemd-modules-load.service
      1.437s lvm2-monitor.service
      1.157s networking.service
      1.015s pve-cluster.service
       941ms pveproxy.service
       754ms pvedaemon.service
       752ms systemd-binfmt.service
       597ms dev-mapper-pve\x2droot.device
       583ms pvescheduler.service
       523ms pvestatd.service
       493ms pve-firewall.service
       368ms zfs-mount.service
       315ms apparmor.service
       301ms systemd-journald.service
       286ms systemd-udev-trigger.service
       252ms postfix@-.service
       248ms spiceproxy.service
       244ms pvebanner.service
       175ms user@0.service
       162ms systemd-sysusers.service
        80ms run-rpc_pipefs.mount
        71ms systemd-journal-flush.service
        58ms systemd-udevd.service
        49ms chrony.service
        49ms systemd-remount-fs.service
        44ms systemd-random-seed.service
        44ms lxc.service
        35ms systemd-logind.service
        34ms kmod-static-nodes.service
        34ms dev-hugepages.mount
        34ms dev-mqueue.mount
        34ms keyboard-setup.service
        34ms sys-kernel-debug.mount
        33ms sys-kernel-tracing.mount
        33ms modprobe@configfs.service
        33ms modprobe@dm_mod.service
        33ms modprobe@drm.service
        32ms modprobe@efi_pstore.service
        32ms modprobe@fuse.service
        32ms modprobe@loop.service
        30ms systemd-fsck@dev-disk-by\x2duuid-E5A5\x2d86D0.service
        28ms rrdcached.service
        28ms e2scrub_reap.service
        28ms ksmtuned.service

 

[UdoB]

Please start by giving us some information, like the copy-n-pasted output of some commands, run on a PVE host, ideally during the problematic situation:

System information:

• pveversion -v
• grep . /proc/pressure/* # Pressure Stall Information = System Load

Storages/Disks:

• lsblk -f | grep -v zd # block devices, skipping "zd123" virtual disks
• pvesm status # and/or cat /etc/pve/storage.cfg for comparison
• zpool list -v # layout and capacity of all pools
• zpool status -v # layout and status of each device
• mount | grep -v subvol- # mounted filesystem with options; skip ZFS datasets
• df -h | grep -v subvol- # disk usage without listing ZFS datasets used for containers

Smart-data:

• smartctl -i /dev/SDX && smartctl -A /dev/SDX # for each relevant block device from the "lsblk" command

Of course you may adjust or extend all the above commands to better fit the actual situation - they are just a first shot.

 

[79corvette]

I took a look at the output from the checks you suggested, full results are copied below.

The system was idling when I ran all checks, so I was surprised to see the pressure stall information reporting some processes were waiting for IO. Is this normal behavior?

The system has 6 disks:

• 2x 14tb 7200 rpm disks (sda and sdb) in a ZFS pool ("dataZFS")
• 2x ~4tb 7200 rpm disks (sdc and sdd) in a ZFS pool ("bkupZFS")
• 2x 1tb nvme in an mdadm raid running the host and all LXC
  • In retrospect this probably was not necessary and I would not do it again (would probably just use one disk and keep good backups), but it has been working well for some time so I have not changed it

'zpool status' says some supported features are not enabled. Is there away to see what these features are and what the consequence of running 'zpool upgrade' will be?

I only copied smartctl output for the 2 devices in the slow ZFS pool (sda and sdb, in "dataZFS" pool). Would it be helpful to see results for the other devices or are they not relevant?

Thanks for taking time to help. I really appreciate it.

~# pveversion -v

proxmox-ve: 8.2.0 (running kernel: 6.8.4-2-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-4
proxmox-kernel-6.8: 6.8.4-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.1
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.2-1
proxmox-backup-file-restore: 3.2.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.6
pve-container: 5.0.11
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.6
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2

~# grep . /proc/pressure/*

/proc/pressure/cpu:some avg10=0.00 avg60=0.00 avg300=0.00 total=100885266
/proc/pressure/cpu:full avg10=0.00 avg60=0.00 avg300=0.00 total=0
/proc/pressure/io:some avg10=20.69 avg60=18.99 avg300=19.24 total=12221420890
/proc/pressure/io:full avg10=20.61 avg60=18.94 avg300=19.13 total=12108270898
/proc/pressure/memory:some avg10=0.00 avg60=0.00 avg300=0.00 total=1261053
/proc/pressure/memory:full avg10=0.00 avg60=0.00 avg300=0.00 total=1191347

~# lsblk -f | grep -v zd

NAME                           FSTYPE            FSVER    LABEL       UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                                                         
├─sda1                         zfs_member        5000     dataZFS     333751729939431349                                   
└─sda9                                                                                                                     
sdb                                                                                                                         
├─sdb1                         zfs_member        5000     dataZFS     333751729939431349                                   
└─sdb9                                                                                                                     
sdc                                                                                                                         
├─sdc1                         zfs_member        5000     bkup_extZFS 15611364751950098079                                 
└─sdc9                                                                                                                     
sdd                                                                                                                         
├─sdd1                         zfs_member        5000     bkup_extZFS 15611364751950098079                                 
└─sdd9                                                                                                                     
nvme0n1                                                                                                                     
├─nvme0n1p1                                                                                                                 
├─nvme0n1p2                    vfat              FAT32                E5A5-86D0                                             
└─nvme0n1p3                    linux_raid_member 1.2      pve:0       a3368cea-be2e-7dd6-bf36-b45ed39e1692                 
  └─md0                        LVM2_member       LVM2 001             Vo0R1t-qVvP-6iNj-ueXK-rZa6-NOOL-sbCkeS               
    ├─pve-swap                 swap              1                    658557ba-6b22-41a3-acd5-a6f6c0282388                  [SWAP]
    ├─pve-root                 ext4              1.0                  c5424633-1cef-4384-aaeb-c3481be05500     83.5G     6% /
    ├─pve-data_tmeta                                                                                                       
    │ └─pve-data-tpool                                                                                                     
    │   ├─pve-data                                                                                                         
    │   ├─pve-vm--102--disk--0 ext4              1.0                  42db2441-e752-4dc1-8567-2bacd8602905                 
    │   ├─pve-vm--105--disk--0 ext4              1.0                  e063f0d9-c4e1-452f-8290-690a474ad347                 
    │   ├─pve-vm--104--disk--0 ext4              1.0                  e1f3cf89-5c26-4d8a-ba4d-c5919a9e8394                 
    │   ├─pve-vm--108--disk--0 ext4              1.0                  5419a45b-7a61-4d20-ba1a-fbcd11eb0d5c                 
    │   ├─pve-vm--110--disk--0 ext4              1.0                  8fa9c936-e2cb-4c78-9096-2a9a2eb45e80                 
    │   ├─pve-vm--112--disk--0 ext4              1.0                  26e963b5-871c-4ecb-8a12-67432345eee1                 
    │   ├─pve-vm--100--disk--1 ext4              1.0                  645bb10f-5ea3-46bb-b74b-6ba1785d0149                 
    │   ├─pve-vm--109--disk--0 ext4              1.0                  e4ab99e9-ea83-487d-9c23-f30dc631a526                 
    │   ├─pve-vm--113--disk--0 ext4              1.0                  0c0ebee3-4ff8-41ce-b4aa-b27219644f42                 
    │   ├─pve-vm--111--disk--0 ext4              1.0                  c0b7fa1c-6c3f-4d26-8ff6-c25a3303f714                 
    │   ├─pve-vm--107--disk--0 ext4              1.0                  2970ff4e-af66-43ec-9689-d18f5219e01f                 
    │   ├─pve-vm--114--disk--0 ext4              1.0                  ba131f96-3c3f-461f-af34-74068cfd108f                 
    │   ├─pve-vm--115--disk--0 ext4              1.0                  b5f2885a-bae1-48f1-ab66-6e2a6136a818                 
    │   └─pve-vm--116--disk--0 ext4              1.0                  c4b1a09d-b335-4b03-9e35-5b2059d86ea6                 
    └─pve-data_tdata                                                                                                       
      └─pve-data-tpool                                                                                                     
        ├─pve-data                                                                                                         
        ├─pve-vm--102--disk--0 ext4              1.0                  42db2441-e752-4dc1-8567-2bacd8602905                 
        ├─pve-vm--105--disk--0 ext4              1.0                  e063f0d9-c4e1-452f-8290-690a474ad347                 
        ├─pve-vm--104--disk--0 ext4              1.0                  e1f3cf89-5c26-4d8a-ba4d-c5919a9e8394                 
        ├─pve-vm--108--disk--0 ext4              1.0                  5419a45b-7a61-4d20-ba1a-fbcd11eb0d5c                 
        ├─pve-vm--110--disk--0 ext4              1.0                  8fa9c936-e2cb-4c78-9096-2a9a2eb45e80                 
        ├─pve-vm--112--disk--0 ext4              1.0                  26e963b5-871c-4ecb-8a12-67432345eee1                 
        ├─pve-vm--100--disk--1 ext4              1.0                  645bb10f-5ea3-46bb-b74b-6ba1785d0149                 
        ├─pve-vm--109--disk--0 ext4              1.0                  e4ab99e9-ea83-487d-9c23-f30dc631a526                 
        ├─pve-vm--113--disk--0 ext4              1.0                  0c0ebee3-4ff8-41ce-b4aa-b27219644f42                 
        ├─pve-vm--111--disk--0 ext4              1.0                  c0b7fa1c-6c3f-4d26-8ff6-c25a3303f714                 
        ├─pve-vm--107--disk--0 ext4              1.0                  2970ff4e-af66-43ec-9689-d18f5219e01f                 
        ├─pve-vm--114--disk--0 ext4              1.0                  ba131f96-3c3f-461f-af34-74068cfd108f                 
        ├─pve-vm--115--disk--0 ext4              1.0                  b5f2885a-bae1-48f1-ab66-6e2a6136a818                 
        └─pve-vm--116--disk--0 ext4              1.0                  c4b1a09d-b335-4b03-9e35-5b2059d86ea6                 
nvme1n1                                                                                                                     
├─nvme1n1p1                                                                                                                 
├─nvme1n1p2                    vfat              FAT32                E5A5-86D0                              1021.6M     0% /boot/efi
└─nvme1n1p3                    linux_raid_member 1.2      pve:0       a3368cea-be2e-7dd6-bf36-b45ed39e1692                 
  └─md0                        LVM2_member       LVM2 001             Vo0R1t-qVvP-6iNj-ueXK-rZa6-NOOL-sbCkeS               
    ├─pve-swap                 swap              1                    658557ba-6b22-41a3-acd5-a6f6c0282388                  [SWAP]
    ├─pve-root                 ext4              1.0                  c5424633-1cef-4384-aaeb-c3481be05500     83.5G     6% /
    ├─pve-data_tmeta                                                                                                       
    │ └─pve-data-tpool                                                                                                     
    │   ├─pve-data                                                                                                         
    │   ├─pve-vm--102--disk--0 ext4              1.0                  42db2441-e752-4dc1-8567-2bacd8602905                 
    │   ├─pve-vm--105--disk--0 ext4              1.0                  e063f0d9-c4e1-452f-8290-690a474ad347                 
    │   ├─pve-vm--104--disk--0 ext4              1.0                  e1f3cf89-5c26-4d8a-ba4d-c5919a9e8394                 
    │   ├─pve-vm--108--disk--0 ext4              1.0                  5419a45b-7a61-4d20-ba1a-fbcd11eb0d5c                 
    │   ├─pve-vm--110--disk--0 ext4              1.0                  8fa9c936-e2cb-4c78-9096-2a9a2eb45e80                 
    │   ├─pve-vm--112--disk--0 ext4              1.0                  26e963b5-871c-4ecb-8a12-67432345eee1                 
    │   ├─pve-vm--100--disk--1 ext4              1.0                  645bb10f-5ea3-46bb-b74b-6ba1785d0149                 
    │   ├─pve-vm--109--disk--0 ext4              1.0                  e4ab99e9-ea83-487d-9c23-f30dc631a526                 
    │   ├─pve-vm--113--disk--0 ext4              1.0                  0c0ebee3-4ff8-41ce-b4aa-b27219644f42                 
    │   ├─pve-vm--111--disk--0 ext4              1.0                  c0b7fa1c-6c3f-4d26-8ff6-c25a3303f714                 
    │   ├─pve-vm--107--disk--0 ext4              1.0                  2970ff4e-af66-43ec-9689-d18f5219e01f                 
    │   ├─pve-vm--114--disk--0 ext4              1.0                  ba131f96-3c3f-461f-af34-74068cfd108f                 
    │   ├─pve-vm--115--disk--0 ext4              1.0                  b5f2885a-bae1-48f1-ab66-6e2a6136a818                 
    │   └─pve-vm--116--disk--0 ext4              1.0                  c4b1a09d-b335-4b03-9e35-5b2059d86ea6                 
    └─pve-data_tdata                                                                                                       
      └─pve-data-tpool                                                                                                     
        ├─pve-data                                                                                                         
        ├─pve-vm--102--disk--0 ext4              1.0                  42db2441-e752-4dc1-8567-2bacd8602905                 
        ├─pve-vm--105--disk--0 ext4              1.0                  e063f0d9-c4e1-452f-8290-690a474ad347                 
        ├─pve-vm--104--disk--0 ext4              1.0                  e1f3cf89-5c26-4d8a-ba4d-c5919a9e8394                 
        ├─pve-vm--108--disk--0 ext4              1.0                  5419a45b-7a61-4d20-ba1a-fbcd11eb0d5c                 
        ├─pve-vm--110--disk--0 ext4              1.0                  8fa9c936-e2cb-4c78-9096-2a9a2eb45e80                 
        ├─pve-vm--112--disk--0 ext4              1.0                  26e963b5-871c-4ecb-8a12-67432345eee1                 
        ├─pve-vm--100--disk--1 ext4              1.0                  645bb10f-5ea3-46bb-b74b-6ba1785d0149                 
        ├─pve-vm--109--disk--0 ext4              1.0                  e4ab99e9-ea83-487d-9c23-f30dc631a526                 
        ├─pve-vm--113--disk--0 ext4              1.0                  0c0ebee3-4ff8-41ce-b4aa-b27219644f42                 
        ├─pve-vm--111--disk--0 ext4              1.0                  c0b7fa1c-6c3f-4d26-8ff6-c25a3303f714                 
        ├─pve-vm--107--disk--0 ext4              1.0                  2970ff4e-af66-43ec-9689-d18f5219e01f                 
        ├─pve-vm--114--disk--0 ext4              1.0                  ba131f96-3c3f-461f-af34-74068cfd108f                 
        ├─pve-vm--115--disk--0 ext4              1.0                  b5f2885a-bae1-48f1-ab66-6e2a6136a818                 
        └─pve-vm--116--disk--0 ext4              1.0                  c4b1a09d-b335-4b03-9e35-5b2059d86ea6

 

[79corvette]

~# pvesm status

Name               Type     Status           Total            Used       Available        %
bkupZFS             dir     active       524288000       257594240       266693760   49.13%
bkup_extZFS     zfspool     active      3771203584        17974460      3753229124    0.48%
dataZFS         zfspool     active     13522436096      6344189448      7178246648   46.92%
local               dir     active        98497780         5856340        87591892    5.95%
local-lvm       lvmthin     active       832884736        37146659       795738076    4.46%
tars                pbs     active      3778108544        36929920      3741178624    0.98%
ut-pbs              pbs     active      2147483648       322520832      1824962816   15.02%

cat /etc/pve/storage.cfg

dir: local
        path /var/lib/vz
        content vztmpl,iso,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

zfspool: dataZFS
        pool dataZFS
        content images
        mountpoint /dataZFS
        nodes pve
        sparse 0

dir: bkupZFS
        path /dataZFS/bkup
        content backup,iso,images,vztmpl
        prune-backups keep-all=1
        shared 0

zfspool: bkup_extZFS
        pool bkup_extZFS
        content rootdir,images
        mountpoint /bkup_extZFS
        nodes pve

pbs: tars
        datastore backup
        server 192.168.1.102
        content backup
        fingerprint ---------------------------------
        port 8007
        prune-backups keep-all=1
        username backup@pbs

pbs: ut-pbs
        datastore pbs_ut
        server 192.168.86.201
        content backup
        fingerprint ---------------------------------
        prune-backups keep-all=1
        username backup@pbs

~# zpool list -v

NAME                                    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bkup_extZFS                            3.62T  17.1G  3.61T        -         -     0%     0%  1.00x    ONLINE  -
  mirror-0                             3.62T  17.1G  3.61T        -         -     0%  0.46%      -    ONLINE
    wwn-0x50014ee6095854e5             3.64T      -      -        -         -      -      -      -    ONLINE
    wwn-0x50014ee609354e25             3.64T      -      -        -         -      -      -      -    ONLINE
dataZFS                                12.7T  5.91T  6.81T        -         -     1%    46%  1.00x    ONLINE  -
  mirror-0                             12.7T  5.91T  6.81T        -         -     1%  46.5%      -    ONLINE
    ata-ST14000NM000J-2TX103_ZR70N8YG  12.7T      -      -        -         -      -      -      -    ONLINE
    ata-ST14000NM000J-2TX103_ZR70N7VQ  12.7T      -      -        -         -      -      -      -    ONLINE
root@pve:~#

~# zpool status -v

NAME                                    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bkup_extZFS                            3.62T  17.1G  3.61T        -         -     0%     0%  1.00x    ONLINE  -
  mirror-0                             3.62T  17.1G  3.61T        -         -     0%  0.46%      -    ONLINE
    wwn-0x50014ee6095854e5             3.64T      -      -        -         -      -      -      -    ONLINE
    wwn-0x50014ee609354e25             3.64T      -      -        -         -      -      -      -    ONLINE
dataZFS                                12.7T  5.91T  6.81T        -         -     1%    46%  1.00x    ONLINE  -
  mirror-0                             12.7T  5.91T  6.81T        -         -     1%  46.5%      -    ONLINE
    ata-ST14000NM000J-2TX103_ZR70N8YG  12.7T      -      -        -         -      -      -      -    ONLINE
    ata-ST14000NM000J-2TX103_ZR70N7VQ  12.7T      -      -        -         -      -      -      -    ONLINE
root@pve:~# zpool status -v
  pool: bkup_extZFS
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:11:24 with 0 errors on Sun Jun  9 00:35:26 2024
config:

        NAME                        STATE     READ WRITE CKSUM
        bkup_extZFS                 ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            wwn-0x50014ee6095854e5  ONLINE       0     0     0
            wwn-0x50014ee609354e25  ONLINE       0     0     0

errors: No known data errors

  pool: dataZFS
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 08:46:13 with 0 errors on Sun Jun  9 09:10:17 2024
config:

        NAME                                   STATE     READ WRITE CKSUM
        dataZFS                                ONLINE       0     0     0
          mirror-0                             ONLINE       0     0     0
            ata-ST14000NM000J-2TX103_ZR70N8YG  ONLINE       0     0     0
            ata-ST14000NM000J-2TX103_ZR70N7VQ  ONLINE       0     0     0

errors: No known data errors

~# mount | grep -v subvol-

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=24295728k,nr_inodes=6073932,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4865956k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2348)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/nvme1n1p2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
bkup_extZFS on /bkup_extZFS type zfs (rw,relatime,xattr,noacl,casesensitive)
dataZFS on /dataZFS type zfs (rw,relatime,xattr,noacl,casesensitive)
dataZFS/bkup on /dataZFS/bkup type zfs (rw,relatime,xattr,noacl,casesensitive)
dataZFS/immich on /dataZFS/immich type zfs (rw,relatime,xattr,noacl,casesensitive)
dataZFS/share on /dataZFS/share type zfs (rw,relatime,xattr,noacl,casesensitive)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=4865952k,nr_inodes=1216488,mode=700,inode64)

~# df -h | grep -v subvol-

Filesystem                     Size  Used Avail Use% Mounted on
udev                            24G     0   24G   0% /dev
tmpfs                          4.7G  1.5M  4.7G   1% /run
/dev/mapper/pve-root            94G  5.6G   84G   7% /
tmpfs                           24G   34M   24G   1% /dev/shm
tmpfs                          5.0M     0  5.0M   0% /run/lock
efivarfs                       150K   82K   64K  57% /sys/firmware/efi/efivars
/dev/nvme1n1p2                1022M  344K 1022M   1% /boot/efi
bkup_extZFS                    3.5T  3.5G  3.5T   1% /bkup_extZFS
dataZFS                        6.8T   23G  6.7T   1% /dataZFS
dataZFS/bkup                   500G  246G  255G  50% /dataZFS/bkup
dataZFS/immich                 400G  205G  196G  52% /dataZFS/immich
dataZFS/share                  8.0T  5.5T  2.6T  69% /dataZFS/share
/dev/fuse                      128M   28K  128M   1% /etc/pve
tmpfs                          4.7G  4.0K  4.7G   1% /run/user/0

smartctl results for the 2 disk drives in the underperforming ZFS pool:

~# smartctl -i /dev/sda && smartctl -A /dev/sda

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.4-2-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST14000NM000J-2TX103
Serial Number:    ZR70N8YG
LU WWN Device Id: 5 000c50 0e54ee611
Firmware Version: SN04
User Capacity:    14,000,519,643,136 bytes [14.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database 7.3/5319
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Jun 30 11:27:40 2024 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.4-2-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   075   064   044    Pre-fail  Always       -       29134704
  3 Spin_Up_Time            0x0003   092   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       26
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   079   060   045    Pre-fail  Always       -       81799810
  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       8703
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       26
 18 Unknown_Attribute       0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   062   048   000    Old_age   Always       -       38 (Min/Max 21/50)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       9
193 Load_Cycle_Count        0x0032   089   089   000    Old_age   Always       -       23965
194 Temperature_Celsius     0x0022   038   052   000    Old_age   Always       -       38 (0 21 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   100   000    Old_age   Offline      -       1556 (155 127 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       36870816187
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       151676884470

~# smartctl -i /dev/sdb && smartctl -A /dev/sdb

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.4-2-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST14000NM000J-2TX103
Serial Number:    ZR70N7VQ
LU WWN Device Id: 5 000c50 0e54ef52d
Firmware Version: SN04
User Capacity:    14,000,519,643,136 bytes [14.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database 7.3/5319
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Jun 30 11:28:44 2024 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.4-2-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   064   044    Pre-fail  Always       -       196514965
  3 Spin_Up_Time            0x0003   092   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       26
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   079   060   045    Pre-fail  Always       -       82266240
  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       8703
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       26
 18 Unknown_Attribute       0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   063   049   000    Old_age   Always       -       37 (Min/Max 20/49)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       7
193 Load_Cycle_Count        0x0032   089   089   000    Old_age   Always       -       23967
194 Temperature_Celsius     0x0022   037   051   000    Old_age   Always       -       37 (0 20 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   100   000    Old_age   Offline      -       1556 (105 113 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       36870816187
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       151356030394

 

[UdoB]

/proc/pressure/io:full avg10=20.61 avg60=18.94 avg300=19.13 total=12108270898

That is high.

In my Homelab I reach the same level when I run a Backup onto PBS with rotating rust as the destination. (Albeit the presence of a "Special Device"; not a recommended setup.)

You can watch the disks in action by zpool iostat -lvq dataZFS 5. Use iotop on process level. Sometimes it is difficult to find the guilty.

Pure guessing: probably a buffer/write-cache is full and it needs a long time to actually deliver the data to the underlying device.

 

[79corvette]

I've been travelling for work, but finally made it home and tried some more troubleshooting.

I noticed the ZFS arc was completely full, and increased the size from 24 to 32 gb. This did not have any impact, and the arc cache quickly grew to the new limit. Is this typical behavior?

~# cat /proc/spl/kstat/zfs/arcstats

c                               4    24913686528
c_min                           4    1557105408
c_max                           4    24913686528

------------------------------------------------------------------------------------
EDIT: I think I figured this out. It was a silly problem.
During a previous restore, the LXC in question was restored to the "dataZFS" ZFS pool (spinning disks) instead of the nvme raid (mdadm raid) that hosts pve and all my containers. iostat showed extremely high utilization on the ZFS disks during the restore process, which seemed wrong. Moving the LXC back to the correct storage results in very fast restores, like what I experienced previously. I do not know why the ZFS pool is so slow (it should be slower than SSD but this seems extreme), but it does not currently affect what I am doing with the system. I will keep trying to troubleshoot but it is no longer a high priority.

 

[leesteken]

I noticed the ZFS arc was completely full, and increased the size from 24 to 32 gb. This did not have any impact, and the arc cache quickly grew to the new limit. Is this typical behavior?

If you're doing I/O, yes. Unused (cache) memory is wasted memory.