Issue with arc_summary after upgrading to zfsutils-linux/stable 2.2.0-pve3

Taylan

Active Member
Oct 19, 2020
94
33
38
55
Hello everyone,

after upgrading the package `zfsutils-linux/stable 2.2.0-pve3` I get following errors with `arc_summary`:
Code:
# arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Sun Nov 19 12:32:54 2023
Linux 6.2.16-19-pve                                          2.1.13-pve1
Machine: Proxmox (x86_64)                                    2.1.13-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 1034, in <module>
    main()
  File "/usr/sbin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
           ~~~~~~~~~^^^^^^^^
KeyError: 'meta'

Original exception was:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 1034, in <module>
    main()
  File "/usr/sbin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
           ~~~~~~~~~^^^^^^^^
KeyError: 'meta'

Package versions:
Code:
proxmox-ve: 8.0.2 (running kernel: 6.2.16-19-pve)
pve-manager: 8.0.9 (running version: 8.0.9/fd1a0ae1b385cdcd)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.5
proxmox-kernel-6.2.16-19-pve: 6.2.16-19
proxmox-kernel-6.2: 6.2.16-19
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx6
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.6
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.10
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.4
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.4-1
proxmox-backup-file-restore: 3.0.4-1
proxmox-kernel-helper: 8.0.5
proxmox-mail-forward: 0.2.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.1.1
pve-cluster: 8.0.5
pve-container: 5.0.5
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.0.7
pve-qemu-kvm: 8.1.2-2
pve-xtermjs: 5.3.0-2
qemu-server: 8.0.8
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.0-pve3

It's the second one (for me) that upgrading the packages breaks this. Last time I had to opt-in to a newer kernel. How can we assure, that the packages are being testet with the stable/current kernels?

Apparently a workaround is to use "arc_summary -r" or upgrade to kernel 6.5, according to here: https://github.com/openzfs/zfs/issues/15490

"arcstat" doesn't work either:
Code:
# arcstat
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
Traceback (most recent call last):
  File "/usr/sbin/arcstat", line 637, in <module>
    main()
  File "/usr/sbin/arcstat", line 621, in main
    calculate()
  File "/usr/sbin/arcstat", line 481, in calculate
    v["iohs"] = d["iohits"] // sint
                ~^^^^^^^^^^
KeyError: 'iohits'

Edit: Here is the previous thread: https://forum.proxmox.com/threads/update-to-5-11-22-7-pve-causes-zfs-issues.99401/

Edit2: I opted in to kernel 6.5 (see https://forum.proxmox.com/threads/o...-2-for-proxmox-ve-8-available-on-test.135635/). The error went away.
 
Last edited:
  • Like
Reactions: leesteken
Hello,

I have exactly the same issue with fresh Proxmox 8 installation and all package upgraded.

ii zfsutils-linux 2.2.0-pve3 amd64 command-line tools to manage OpenZFS filesystems

Code:
root@lab:~# arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Mon Nov 20 18:11:23 2023
Linux 6.2.16-19-pve                                          2.1.13-pve1
Machine: lab (x86_64)                                        2.1.13-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 1034, in <module>
    main()
  File "/usr/sbin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
           ~~~~~~~~~^^^^^^^^
KeyError: 'meta'

Original exception was:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 1034, in <module>
    main()
  File "/usr/sbin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
           ~~~~~~~~~^^^^^^^^
KeyError: 'meta'
root@lab:~# arcstat
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
Traceback (most recent call last):
  File "/usr/sbin/arcstat", line 637, in <module>
    main()
  File "/usr/sbin/arcstat", line 621, in main
    calculate()
  File "/usr/sbin/arcstat", line 481, in calculate
    v["iohs"] = d["iohits"] // sint
                ~^^^^^^^^^^
KeyError: 'iohits'

Pveversion :
Code:
root@lab:~# pveversion -v
proxmox-ve: 8.0.2 (running kernel: 6.2.16-19-pve)
pve-manager: 8.0.9 (running version: 8.0.9/fd1a0ae1b385cdcd)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.5
proxmox-kernel-6.2.16-19-pve: 6.2.16-19
proxmox-kernel-6.2: 6.2.16-19
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx6
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.6
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.10
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.4
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.4-1
proxmox-backup-file-restore: 3.0.4-1
proxmox-kernel-helper: 8.0.5
proxmox-mail-forward: 0.2.1
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.1
pve-cluster: 8.0.5
pve-container: 5.0.5
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.0.7
pve-qemu-kvm: 8.1.2-2
pve-xtermjs: 5.3.0-2
qemu-server: 8.0.8
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.0-pve3

Best Regards
 
can relate. same issue here.

Code:
proxmox-ve: 8.1.0 (running kernel: 6.2.16-15-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5: 6.5.11-8
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.1
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-3
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.5-2
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1