Issue with arc_summary after upgrading to zfsutils-linux/stable 2.2.0-pve3

Taylan

Member
Oct 19, 2020
83
30
23
54
Hello everyone,

after upgrading the package `zfsutils-linux/stable 2.2.0-pve3` I get following errors with `arc_summary`:
Code:
# arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Sun Nov 19 12:32:54 2023
Linux 6.2.16-19-pve                                          2.1.13-pve1
Machine: Proxmox (x86_64)                                    2.1.13-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 1034, in <module>
    main()
  File "/usr/sbin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
           ~~~~~~~~~^^^^^^^^
KeyError: 'meta'

Original exception was:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 1034, in <module>
    main()
  File "/usr/sbin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
           ~~~~~~~~~^^^^^^^^
KeyError: 'meta'

Package versions:
Code:
proxmox-ve: 8.0.2 (running kernel: 6.2.16-19-pve)
pve-manager: 8.0.9 (running version: 8.0.9/fd1a0ae1b385cdcd)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.5
proxmox-kernel-6.2.16-19-pve: 6.2.16-19
proxmox-kernel-6.2: 6.2.16-19
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx6
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.6
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.10
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.4
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.4-1
proxmox-backup-file-restore: 3.0.4-1
proxmox-kernel-helper: 8.0.5
proxmox-mail-forward: 0.2.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.1.1
pve-cluster: 8.0.5
pve-container: 5.0.5
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.0.7
pve-qemu-kvm: 8.1.2-2
pve-xtermjs: 5.3.0-2
qemu-server: 8.0.8
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.0-pve3

It's the second one (for me) that upgrading the packages breaks this. Last time I had to opt-in to a newer kernel. How can we assure, that the packages are being testet with the stable/current kernels?

Apparently a workaround is to use "arc_summary -r" or upgrade to kernel 6.5, according to here: https://github.com/openzfs/zfs/issues/15490

"arcstat" doesn't work either:
Code:
# arcstat
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
Traceback (most recent call last):
  File "/usr/sbin/arcstat", line 637, in <module>
    main()
  File "/usr/sbin/arcstat", line 621, in main
    calculate()
  File "/usr/sbin/arcstat", line 481, in calculate
    v["iohs"] = d["iohits"] // sint
                ~^^^^^^^^^^
KeyError: 'iohits'

Edit: Here is the previous thread: https://forum.proxmox.com/threads/update-to-5-11-22-7-pve-causes-zfs-issues.99401/

Edit2: I opted in to kernel 6.5 (see https://forum.proxmox.com/threads/o...-2-for-proxmox-ve-8-available-on-test.135635/). The error went away.
 
Last edited:
  • Like
Reactions: leesteken
Hello,

I have exactly the same issue with fresh Proxmox 8 installation and all package upgraded.

ii zfsutils-linux 2.2.0-pve3 amd64 command-line tools to manage OpenZFS filesystems

Code:
root@lab:~# arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Mon Nov 20 18:11:23 2023
Linux 6.2.16-19-pve                                          2.1.13-pve1
Machine: lab (x86_64)                                        2.1.13-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 1034, in <module>
    main()
  File "/usr/sbin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
           ~~~~~~~~~^^^^^^^^
KeyError: 'meta'

Original exception was:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 1034, in <module>
    main()
  File "/usr/sbin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
           ~~~~~~~~~^^^^^^^^
KeyError: 'meta'
root@lab:~# arcstat
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
Traceback (most recent call last):
  File "/usr/sbin/arcstat", line 637, in <module>
    main()
  File "/usr/sbin/arcstat", line 621, in main
    calculate()
  File "/usr/sbin/arcstat", line 481, in calculate
    v["iohs"] = d["iohits"] // sint
                ~^^^^^^^^^^
KeyError: 'iohits'

Pveversion :
Code:
root@lab:~# pveversion -v
proxmox-ve: 8.0.2 (running kernel: 6.2.16-19-pve)
pve-manager: 8.0.9 (running version: 8.0.9/fd1a0ae1b385cdcd)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.5
proxmox-kernel-6.2.16-19-pve: 6.2.16-19
proxmox-kernel-6.2: 6.2.16-19
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx6
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.6
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.10
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.4
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.4-1
proxmox-backup-file-restore: 3.0.4-1
proxmox-kernel-helper: 8.0.5
proxmox-mail-forward: 0.2.1
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.1
pve-cluster: 8.0.5
pve-container: 5.0.5
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.0.7
pve-qemu-kvm: 8.1.2-2
pve-xtermjs: 5.3.0-2
qemu-server: 8.0.8
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.0-pve3

Best Regards
 
can relate. same issue here.

Code:
proxmox-ve: 8.1.0 (running kernel: 6.2.16-15-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5: 6.5.11-8
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.1
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-3
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.5-2
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!