Update to 5.11.22-7-pve causes zfs issues

T

Taylan

Active Member
Oct 19, 2020
92
33
38
54
Hello everyone,

I just updated my server from
5.11.22-5-pve
to
5.11.22-7-pve
, and have issues afterwards with the commands "arcstat" and "arc_summary".

They throw following errors now:
Bash: 
root@Proxmox:~# arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
Traceback (most recent call last):
  File "/usr/sbin/arcstat", line 554, in <module>
    main()
  File "/usr/sbin/arcstat", line 538, in main
    calculate()
  File "/usr/sbin/arcstat", line 485, in calculate
    v["el2mfu"] = d["evict_l2_eligible_mfu"] / sint
KeyError: 'evict_l2_eligible_mfu'

Code: 
root@Proxmox:~# arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Tue Nov 09 18:53:50 2021
Linux 5.11.22-7-pve                                           2.0.6-pve1
Machine: Proxmox (x86_64)                                     2.0.6-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                     4.1 %    1.3 GiB
        Target size (adaptive):                         6.2 %    2.0 GiB
        Min size (hard limit):                          6.2 %    2.0 GiB
        Max size (high water):                           16:1   31.3 GiB
        Most Frequently Used (MFU) cache size:         41.9 %  280.6 MiB
        Most Recently Used (MRU) cache size:           58.1 %  389.5 MiB
        Metadata cache size (hard limit):              75.0 %   23.4 GiB
        Metadata cache size (current):                  3.3 %  801.1 MiB
        Dnode cache size (hard limit):                 10.0 %    2.3 GiB
        Dnode cache size (current):                     1.2 %   27.9 MiB

ARC hash breakdown:
        Elements max:                                               6.2M
        Elements current:                             100.0 %       6.2M
        Collisions:                                                 1.8M
        Chain max:                                                     7
        Chains:                                                     1.4M

ARC misc:
        Deleted:                                                    1.4k
        Mutex misses:                                                  0
        Eviction skips:                                             2.0k
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                  21.3 MiB
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 986, in <module>
    main()
  File "/usr/sbin/arc_summary", line 980, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 620, in section_arc
    f_perc(arc_stats['evict_l2_eligible_mfu'],
KeyError: 'evict_l2_eligible_mfu'

Original exception was:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 986, in <module>
    main()
  File "/usr/sbin/arc_summary", line 980, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 620, in section_arc
    f_perc(arc_stats['evict_l2_eligible_mfu'],
KeyError: 'evict_l2_eligible_mfu'

Does anyone have this issue?
 
Last edited:
Hi,

Is this Proxmox VE 7.x and are you using no-subscription repository? If so, can you try upgrading again so that newer ZFS userland 2.1 packages get pulled in, seems working with them here (and they are backward compatible with ZFS 2.0.6 from that 5.11 kerneö.
 
Hi Thomas,

yes, this is PVE 7.0-14 from the community repo with following kernel and zfs:
Bash: 
root@Proxmox:~# pveversion
pve-manager/7.0-14/a9dbe7e3 (running kernel: 5.11.22-7-pve)
root@Proxmox:~# uname -r
5.11.22-7-pve
root@Proxmox:~# zfs -V
zfs-2.1.1-pve1
zfs-kmod-2.0.6-pve1
root@Proxmox:~#

What do you mean with upgrading again? To what exactly?
 
Taylan said:
Bash: 
root@Proxmox:~# zfs -V
zfs-2.1.1-pve1
zfs-kmod-2.0.6-pve1
root@Proxmox:~#
What do you mean with upgrading again? To what exactly?
Click to expand...
you already have 2.1.1 userspace tools, so already upgraded.. Weird that it errors out that way in that case..
 
I checked this out a bit closer, it's only the two ARC related stats tool getting confused as 2.0 does not have any info about the stats evict_l2_eligible_mfu and evict_l2_eligible_mru and the 2.1 version of the tool just tries to access that blindly in a python dict.

We'll see to patch in a guard here to avoid that bogus error. Note that ZFS itself is unafeccted.
 
  • Like
Reactions: Taylan
t.lamprecht said:
I checked this out a bit closer, it's only the two ARC related stats tool getting confused as 2.0 does not have any info about the stats evict_l2_eligible_mfu and evict_l2_eligible_mru and the 2.1 version of the tool just tries to access that blindly in a python dict.

We'll see to patch in a guard here to avoid that bogus error. Note that ZFS itself is unafeccted.
Click to expand...
Just let me know if I can test something.
 
There are newer user space packages on pve-no-subscription since a bit, they carry a fix for your specific situation.

The most relevant package is zfsutils-linux in version 2.1.1-pve3, as that contains the arcstat and arc_summary tools.
 
  • Like
Reactions: Taylan
t.lamprecht said:
There are newer user space packages on pve-no-subscription since a bit, they carry a fix for your specific situation.

The most relevant package is zfsutils-linux in version 2.1.1-pve3, as that contains the arcstat and arc_summary tools.
Click to expand...
I’ll check them out later after work and report back.
 
t.lamprecht said:
There are newer user space packages on pve-no-subscription since a bit, they carry a fix for your specific situation.

The most relevant package is zfsutils-linux in version 2.1.1-pve3, as that contains the arcstat and arc_summary tools.
Click to expand...
So I've updated again and now have another error:

Bash: 
root@Proxmox:~# apt update && apt list --upgradable
Hit:1 http://security.debian.org/debian-security bullseye-security InRelease
Hit:2 http://ftp.debian.org/debian bullseye InRelease
Hit:3 http://download.proxmox.com/debian/pve bullseye InRelease
Hit:4 http://ftp.debian.org/debian bullseye-updates InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Listing... Done
root@Proxmox:~# apt dist-upgrade -y && apt autoremove -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@Proxmox:~# pveversion
pve-manager/7.0-14/a9dbe7e3 (running kernel: 5.11.22-7-pve)
root@Proxmox:~# uname -r
5.11.22-7-pve
root@Proxmox:~# zfs -V
zfs-2.1.1-pve3
zfs-kmod-2.0.6-pve1

Output of arcstat:
Bash: 
@Proxmox:~# arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
Traceback (most recent call last):
  File "/usr/sbin/arcstat", line 554, in <module>
    main()
  File "/usr/sbin/arcstat", line 538, in main
    calculate()
  File "/usr/sbin/arcstat", line 501, in calculate
    v["l2pref"] = cur["l2_prefetch_asize"]
KeyError: 'l2_prefetch_asize'

Output of arc_summary:
Bash: 
root@Proxmox:~# arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Nov 10 20:09:18 2021
Linux 5.11.22-7-pve                                           2.0.6-pve1
Machine: Proxmox (x86_64)                                     2.0.6-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                     4.2 %    1.3 GiB
        Target size (adaptive):                         6.2 %    2.0 GiB
        Min size (hard limit):                          6.2 %    2.0 GiB
        Max size (high water):                           16:1   31.3 GiB
        Most Frequently Used (MFU) cache size:         38.4 %  286.1 MiB
        Most Recently Used (MRU) cache size:           61.6 %  459.0 MiB
        Metadata cache size (hard limit):              75.0 %   23.4 GiB
        Metadata cache size (current):                  3.3 %  788.9 MiB
        Dnode cache size (hard limit):                 10.0 %    2.3 GiB
        Dnode cache size (current):                     1.3 %   30.5 MiB

ARC hash breakdown:
        Elements max:                                               6.0M
        Elements current:                             100.0 %       6.0M
        Collisions:                                                 1.7M
        Chain max:                                                     8
        Chains:                                                     1.3M

ARC misc:
        Deleted:                                                    1.4k
        Mutex misses:                                                  0
        Eviction skips:                                             2.6k
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                  21.3 MiB
        L2 eligible MFU evictions:                      0.0 %    0 Bytes
        L2 eligible MRU evictions:                      0.0 %    0 Bytes
        L2 ineligible evictions:                                72.0 KiB

ARC total accesses (hits + misses):                               743.7k
        Cache hit ratio:                               96.5 %     717.5k
        Cache miss ratio:                               3.5 %      26.2k
        Actual hit ratio (MFU + MRU hits):             95.8 %     712.8k
        Data demand efficiency:                        96.3 %     266.7k
        Data prefetch efficiency:                      13.1 %       2.3k

Cache hits by cache type:
        Most frequently used (MFU):                    59.7 %     428.3k
        Most recently used (MRU):                      39.6 %     284.5k
        Most frequently used (MFU) ghost:               0.0 %          0
        Most recently used (MRU) ghost:                 0.0 %          0
        Anonymously used:                               0.7 %       4.7k

Cache hits by data type:
        Demand data:                                   35.8 %     256.8k
        Demand prefetch data:                         < 0.1 %        297
        Demand metadata:                               63.6 %     456.0k
        Demand prefetch metadata:                       0.6 %       4.5k

Cache misses by data type:
        Demand data:                                   38.2 %      10.0k
        Demand prefetch data:                           7.5 %       2.0k
        Demand metadata:                               29.5 %       7.7k
        Demand prefetch metadata:                      24.8 %       6.5k

DMU prefetch efficiency:                                           74.3k
        Hit ratio:                                      3.4 %       2.5k
        Miss ratio:                                    96.6 %      71.8k

L2ARC status:                                                    HEALTHY
        Low memory aborts:                                             0
        Free on write:                                                 0
        R/W clashes:                                                   0
        Bad checksums:                                                 0
        I/O errors:                                                    0

L2ARC size (adaptive):                                         705.1 GiB
        Compressed:                                    99.3 %  699.9 GiB
        Header size:                                    0.1 %  544.6 MiB
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 986, in <module>
    main()
  File "/usr/sbin/arc_summary", line 980, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 768, in section_l2arc
    f_perc(arc_stats['l2_mfu_asize'], arc_stats['l2_asize']),
KeyError: 'l2_mfu_asize'

Original exception was:
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 986, in <module>
    main()
  File "/usr/sbin/arc_summary", line 980, in main
    section_calls[section](kstats)
  File "/usr/sbin/arc_summary", line 768, in section_l2arc
    f_perc(arc_stats['l2_mfu_asize'], arc_stats['l2_asize']),
KeyError: 'l2_mfu_asize'
 
Did you just remove those dictionary objects or was this a official patch from the maintainers?
 
Taylan said:
So I've updated again and now have another error:
Click to expand...
Seems my test system had no l2arc configured and thus I missed an affected call site.

Taylan said:
Did you just remove those dictionary objects or was this a official patch from the maintainers?
Click to expand...
FWIW, I upstreamed patches to ZFS in the past and plan to submit this one too once finished. :)
FYI, I switched a direct dictionary access to a get one with zero as fallback.

Also, you could switch to the 5.13 kernel now already if you really need the arcstats/_summary, we'll switch over to it as default in a few days to weeks anyway and includes ZFS 2.1 module, obsoleting handling this unfortunate bogus incompat that could have been avoided..
 
Is there any issues with that kernel?

And which one is the newest?
Code: 
pve-kernel-5.13/stable 7.1-4 all
  Latest Proxmox VE Kernel Image

pve-kernel-5.13.14-1-pve/stable 5.13.14-1 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.13.18-1-pve/stable 5.13.18-1 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.13.19-1-pve/stable 5.13.19-2 amd64
  The Proxmox PVE Kernel Image
 
t.lamprecht said:
Nothing known.

I'd recommend installing the pve-kernel-5.13 meta package as it will always depend on the latest ABI version of the 5.13 series.
Click to expand...
I've installed the kernel 5.13.19-1-pve. I can use the arcstat or arc_summary without error. I'm not sure though, if all the relevant informations are there, need to check that later.

It seems there are new zfs features available, that I can upgrade my pools. What are those changes exactly?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom