How to downgrade zfs?

M

Meikel

New Member
Dec 15, 2025
17
0
1
I ran into a kernel level issue with the latest zfs: https://github.com/openzfs/zfs/issues/18648 (I opened the issue)

According to some folks on there the issue is only existent in zfs 2.4.x and downgrading to 2.3.x should be a valid workaround until fixed.

I was able to downgrade zfs to 2.3.4-pve1 but am unable to also downgrade zfs-kmod because it's apparently not an apt package?

I pinned all zfs packages via /etc/apt/preferences.d/zfs using:

Code: 
Package: zfsutils-linux zfs-zed zfs-initramfs libzfs7linux libzpool5linux
Pin: version 2.3.4-pve1
Pin-Priority: 1000

However when I run zfs --versionI get:

Code: 
root@pve-01:~# zfs --version
zfs-2.3.4-pve1
zfs-kmod-2.4.2-pve1
 
The zfs module comes with the Proxmox kernel package.
If I'm not mistaken, ZFS 2.4.0 was included since 6.17.9-1-pve, so downgrade to a kernel before that.
 
Last edited:
You don't need to downgrade your entire Proxmox, you can also run an older kernel on 9.2 like you manually downgraded ZFS.
For example:
apt install proxmox-kernel-6.17.4-2-pve-signed
proxmox-boot-tool kernel pin 6.17.4-2-pve

For which kernel to install, see:
https://github.com/proxmox/pve-kernel/blob/master/debian/changelog

Edit; If I am not mistaken:

proxmox kernel 6.17.13-(1-13)zfs 2.4.1
proxmox kernel 6.17.9-1zfs 2.4.0
proxmox kernel 6.17.4-(1/2)zfs 2.3.4
proxmox kernel 6.17.2-(1/2)zfs 2.3.4
proxmox kernel 6.17.1-1zfs 2.3.4
 
Last edited:
  • Like
Reactions: Meikel
So I tried your commands. Sadly the installer is stuck at 60% (for at least 10min now). Probably the same issue here
1781694545556.png
Should I boot an available older kernel and then install the latest nonbroken one?

Info:

Code: 
root@pve-01:~# pveversion
pve-manager/9.2.3/d0fde103346cf89a (running kernel: 7.0.6-2-pve)

Code: 
root@pve-01:~# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
6.8.12-25-pve
7.0.2-6-pve
7.0.6-2-pve
 
I would definitely check out the journal ouput if there is any relevant messages there first.
If installing a kernel also fails because of your zfs 2.4.x related issue then booting into 6.8.12-25 (i.e. older zfs) should solve that.
 
Nothing spectacular. I will try the 6.8 kernel for now
Code: 
Jun 17 12:57:53 pve-01 chronyd[2489]: Selected source 46.38.241.235 (2.debian.pool.ntp.org)
Jun 17 12:56:13 pve-01 pvedaemon[2858]: <root@pam> successful auth for user 'root@pam'
Jun 17 12:56:00 pve-01 systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Jun 17 12:56:00 pve-01 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Jun 17 12:56:00 pve-01 systemd-tmpfiles[12282]: /usr/lib/tmpfiles.d/legacy.conf:14: Duplicate line for path "/run/lock", ignoring.
Jun 17 12:56:00 pve-01 systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Jun 17 12:51:12 pve-01 systemd[1]: Finished fstrim.service - Discard unused blocks on filesystems from /etc/fstab.
Jun 17 12:51:12 pve-01 systemd[1]: fstrim.service: Deactivated successfully.
Jun 17 12:51:12 pve-01 systemd[1]: Starting fstrim.service - Discard unused blocks on filesystems from /etc/fstab...
Jun 17 12:45:07 pve-01 kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
Jun 17 12:45:07 pve-01 kernel: INFO: task zfs:4535 is blocked on a mutex likely owned by task zpool:2901.
 
So I'm on 6.8.12-25-pve now.

Command is now stuck at

Code: 
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 6.17.4-2-pve /boot/vmlinuz-6.17.4-2-pve

Guess I'm fcked
 
To be clear; your rpool is healthy and you can boot the system using any kernel already installed?
Is anything system/proxmox-related on your zpool or is it just storage?

If your zpool is troubled and fails to import on boot, you might want to prevent it from importing on boot.
See for example: https://forum.proxmox.com/threads/how-to-prevent-zfs-pool-import-on-boot-up.132990/

run-parts: executing /etc/kernel/postinst.d/zz-update-grub stalling indicates an issue with setting up the boot/EFI partition, depending on your system layout.
It might fail because zfs is occupied with your zpool. Or there might be anything else, for which you should check any relevant logfiles/dmesg/journalctl.
 
Last edited:
The proxmox pool is healthy. I can easily boot from it without issue. The zpool storage-01 is the troublechild. The issue even happens on another OS (debian13 live OS).

I disabled the pool import with systemctl disable zfs-import@storage-01 and the system could easily reboot (I usually cannot reboot without -f since this bug).

I did journalctl -f to see what could be happening, but as always with this bug there are no (relevant) logs. Probably because of that namespace deadlock (see github issue).

Code: 
Jun 17 15:33:30 pve-01 systemd[1]: session-2.scope: Still around after SIGKILL. Ignoring.
Jun 17 15:33:30 pve-01 systemd[1]: session-2.scope: Failed with result 'timeout'.
Jun 17 15:33:30 pve-01 systemd[1]: Stopped session-2.scope - Session 2 of User root.
Jun 17 15:33:30 pve-01 systemd[1]: session-2.scope: Consumed 59.343s CPU time, 763M memory peak.
Jun 17 15:33:30 pve-01 systemd[1]: Stopping systemd-user-sessions.service - Permit User Sessions...
Jun 17 15:33:30 pve-01 systemd[1]: Stopping user@0.service - User Manager for UID 0...
Jun 17 15:33:30 pve-01 systemd-logind[2376]: Removed session 2.
Jun 17 15:33:30 pve-01 systemd[9730]: Activating special unit exit.target...
Jun 17 15:33:30 pve-01 systemd[9730]: Stopped target default.target - Main User Target.
Jun 17 15:33:30 pve-01 systemd[9730]: Stopped target basic.target - Basic System.
Jun 17 15:33:30 pve-01 systemd[9730]: Stopped target paths.target - Paths.
Jun 17 15:33:30 pve-01 systemd[9730]: Stopped target sockets.target - Sockets.
Jun 17 15:33:30 pve-01 systemd[9730]: Stopped target timers.target - Timers.
Jun 17 15:33:30 pve-01 systemd[9730]: Closed dbus.socket - D-Bus User Message Bus Socket.
Jun 17 15:33:30 pve-01 systemd[9730]: Closed dirmngr.socket - GnuPG network certificate management daemon.
Jun 17 15:33:30 pve-01 systemd[9730]: Closed gpg-agent-browser.socket - GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jun 17 15:33:30 pve-01 systemd[9730]: Closed gpg-agent-extra.socket - GnuPG cryptographic agent and passphrase cache (restricted).
Jun 17 15:33:30 pve-01 systemd[9730]: Stopping gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation)...
Jun 17 15:33:30 pve-01 systemd[9730]: Stopping gpg-agent.socket - GnuPG cryptographic agent and passphrase cache...
Jun 17 15:33:30 pve-01 systemd[9730]: Closed keyboxd.socket - GnuPG public key management service.
Jun 17 15:33:30 pve-01 systemd[9730]: Stopping ssh-agent.socket - OpenSSH Agent socket...
Jun 17 15:33:30 pve-01 systemd[1]: systemd-user-sessions.service: Deactivated successfully.
Jun 17 15:33:30 pve-01 systemd[1]: Stopped systemd-user-sessions.service - Permit User Sessions.
Jun 17 15:33:30 pve-01 systemd[9730]: Closed gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation).
Jun 17 15:33:30 pve-01 systemd[9730]: Closed gpg-agent.socket - GnuPG cryptographic agent and passphrase cache.
Jun 17 15:33:30 pve-01 systemd[9730]: Closed ssh-agent.socket - OpenSSH Agent socket.
Jun 17 15:33:30 pve-01 systemd[9730]: Removed slice app.slice - User Application Slice.
Jun 17 15:33:30 pve-01 systemd[9730]: Reached target shutdown.target - Shutdown.
Jun 17 15:33:30 pve-01 systemd[9730]: Finished systemd-exit.service - Exit the Session.
Jun 17 15:33:30 pve-01 systemd[9730]: Reached target exit.target - Exit the Session.
Jun 17 15:33:31 pve-01 (sd-pam)[9733]: pam_unix(systemd-user:session): session closed for user root
Jun 17 15:33:31 pve-01 systemd-logind[2376]: Removed session 3.
Jun 17 15:33:31 pve-01 systemd[1]: user@0.service: Deactivated successfully.
Jun 17 15:33:31 pve-01 systemd[1]: Stopped user@0.service - User Manager for UID 0.
Jun 17 15:33:31 pve-01 systemd[1]: Stopping user-runtime-dir@0.service - User Runtime Directory /run/user/0...
Jun 17 15:33:31 pve-01 systemd[1]: run-user-0.mount: Deactivated successfully.
Jun 17 15:33:31 pve-01 systemd[1]: Unmounted run-user-0.mount - /run/user/0.
Jun 17 15:33:31 pve-01 systemd[1]: user-runtime-dir@0.service: Deactivated successfully.
Jun 17 15:33:31 pve-01 systemd[1]: Stopped user-runtime-dir@0.service - User Runtime Directory /run/user/0.
Jun 17 15:33:31 pve-01 systemd[1]: Removed slice user-0.slice - User Slice of UID 0.
Jun 17 15:33:31 pve-01 systemd[1]: user-0.slice: Consumed 59.936s CPU time, 770.6M memory peak.
Jun 17 15:33:31 pve-01 systemd[1]: Stopping dbus.service - D-Bus System Message Bus...
Jun 17 15:33:31 pve-01 systemd[1]: Stopping systemd-logind.service - User Login Management...
Jun 17 15:33:31 pve-01 systemd[1]: dbus.service: Deactivated successfully.
Jun 17 15:33:31 pve-01 systemd[1]: Stopped dbus.service - D-Bus System Message Bus.
Jun 17 15:33:31 pve-01 systemd[1]: systemd-logind.service: Deactivated successfully.
Jun 17 15:33:31 pve-01 systemd[1]: Stopped systemd-logind.service - User Login Management.


This is all the output I'm getting until stuck:
dpkg --configure -a because I have to kill the process and it has to repair itself I guess

Code: 
root@pve-01:~# dpkg --configure -a
Setting up proxmox-kernel-6.17.4-2-pve-signed (6.17.4-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/dkms 6.17.4-2-pve /boot/vmlinuz-6.17.4-2-pve
Automatic installation of modules for kernel 6.17.4-2-pve was skipped since the kernel headers for this kernel do not seem to be installed.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.17.4-2-pve /boot/vmlinuz-6.17.4-2-pve
update-initramfs: Generating /boot/initrd.img-6.17.4-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/2A62-256F
        Copying kernel and creating boot-entry for 6.17.4-2-pve
        Copying kernel and creating boot-entry for 6.8.12-25-pve
        Copying kernel and creating boot-entry for 7.0.2-6-pve
        Copying kernel and creating boot-entry for 7.0.6-2-pve
Copying and configuring kernels on /dev/disk/by-uuid/2A63-A40D
        Copying kernel and creating boot-entry for 6.17.4-2-pve
        Copying kernel and creating boot-entry for 6.8.12-25-pve
        Copying kernel and creating boot-entry for 7.0.2-6-pve
        Copying kernel and creating boot-entry for 7.0.6-2-pve
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 6.17.4-2-pve /boot/vmlinuz-6.17.4-2-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 6.17.4-2-pve /boot/vmlinuz-6.17.4-2-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/2A62-256F
        Copying kernel and creating boot-entry for 6.17.4-2-pve
        Copying kernel and creating boot-entry for 6.8.12-25-pve
        Copying kernel and creating boot-entry for 7.0.2-6-pve
        Copying kernel and creating boot-entry for 7.0.6-2-pve
Copying and configuring kernels on /dev/disk/by-uuid/2A63-A40D
        Copying kernel and creating boot-entry for 6.17.4-2-pve
        Copying kernel and creating boot-entry for 6.8.12-25-pve
        Copying kernel and creating boot-entry for 7.0.2-6-pve
        Copying kernel and creating boot-entry for 7.0.6-2-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 6.17.4-2-pve /boot/vmlinuz-6.17.4-2-pve
 
Ok, so we are now looking at a boot/kernel update issue.
Can you post proxmox-boot-tool status?
Does journalctl -k say anything during the dpkg command?
 
Code: 
root@pve-01:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
2A62-256F is configured with: uefi (versions: 6.17.4-2-pve, 6.8.12-25-pve, 7.0.2-6-pve, 7.0.6-2-pve)
2A63-A40D is configured with: uefi (versions: 6.17.4-2-pve, 6.8.12-25-pve, 7.0.2-6-pve, 7.0.6-2-pve)

journalctl -k does not log anything while the command is running. The only messages are from before:

Code: 
Jun 17 16:08:52 pve-01 kernel: INFO: task zfs:4577 is blocked on a mutex likely owned by task zpool:2954.
Jun 17 16:08:52 pve-01 kernel: task:zpool           state:D stack:0     pid:2954  tgid:2954  ppid:2951   task_flags:0x480100 flags:0x00004002
Jun 17 16:08:52 pve-01 kernel: Call Trace:
Jun 17 16:08:52 pve-01 kernel:  <TASK>
Jun 17 16:08:52 pve-01 kernel:  __schedule+0x468/0x1310
Jun 17 16:08:52 pve-01 kernel:  ? try_to_wake_up+0x392/0x8a0
Jun 17 16:08:52 pve-01 kernel:  schedule+0x27/0xf0
Jun 17 16:08:52 pve-01 kernel:  io_schedule+0x4c/0x80
Jun 17 16:08:52 pve-01 kernel:  cv_wait_common+0xb0/0x140 [spl]
Jun 17 16:08:52 pve-01 kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Jun 17 16:08:52 pve-01 kernel:  __cv_wait_io+0x18/0x30 [spl]
Jun 17 16:08:52 pve-01 kernel:  txg_wait_synced_flags+0xd8/0x130 [zfs]
Jun 17 16:08:52 pve-01 kernel:  txg_wait_synced+0x10/0x60 [zfs]
Jun 17 16:08:52 pve-01 kernel:  spa_config_update+0x17e/0x1a0 [zfs]
Jun 17 16:08:52 pve-01 kernel:  spa_import+0x5d8/0x6c0 [zfs]
Jun 17 16:08:52 pve-01 kernel:  zfs_ioc_pool_import+0x153/0x170 [zfs]
Jun 17 16:08:52 pve-01 kernel:  zfsdev_ioctl_common+0x7c2/0x970 [zfs]
Jun 17 16:08:52 pve-01 kernel:  zfsdev_ioctl+0x57/0xf0 [zfs]
Jun 17 16:08:52 pve-01 kernel:  __x64_sys_ioctl+0xa5/0x100
Jun 17 16:08:52 pve-01 kernel:  x64_sys_call+0x1151/0x2330
Jun 17 16:08:52 pve-01 kernel:  do_syscall_64+0x80/0xa30
Jun 17 16:08:52 pve-01 kernel:  ? __handle_mm_fault+0xb55/0xfd0
Jun 17 16:08:52 pve-01 kernel:  ? count_memcg_events+0xd7/0x1a0
Jun 17 16:08:52 pve-01 kernel:  ? handle_mm_fault+0x254/0x370
Jun 17 16:08:52 pve-01 kernel:  ? do_user_addr_fault+0x2f8/0x830
Jun 17 16:08:52 pve-01 kernel:  ? irqentry_exit_to_user_mode+0x2e/0x290
Jun 17 16:08:52 pve-01 kernel:  ? irqentry_exit+0x43/0x50
Jun 17 16:08:52 pve-01 kernel:  ? exc_page_fault+0x90/0x1b0
Jun 17 16:08:52 pve-01 kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jun 17 16:08:52 pve-01 kernel: RIP: 0033:0x7cfd523f991b
Jun 17 16:08:52 pve-01 kernel: RSP: 002b:00007ffc0078c8a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Jun 17 16:08:52 pve-01 kernel: RAX: ffffffffffffffda RBX: 0000601bfbd37810 RCX: 00007cfd523f991b
Jun 17 16:08:52 pve-01 kernel: RDX: 00007ffc0078c960 RSI: 0000000000005a02 RDI: 0000000000000003
Jun 17 16:08:52 pve-01 kernel: RBP: 00007ffc00791050 R08: 00000000000606f1 R09: 0000000000000001
Jun 17 16:08:52 pve-01 kernel: R10: 00007cfd524d2ac0 R11: 0000000000000246 R12: 0000601bfbd2d2c0
Jun 17 16:08:52 pve-01 kernel: R13: 00007ffc0078c960 R14: 00007cfd18001028 R15: 0000601bfbd93e60
Jun 17 16:08:52 pve-01 kernel:  </TASK>
Jun 17 16:08:52 pve-01 kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
 
So it seems it installed 6.17.4-2-pve after all before stalling on/after postinst.d/zz-update-grub.
Can you also boot it so you have zfs 2.3.4 as both kernel version and userspace tools?

The journal you posted is when booting 6.8.12-25-pve (zfs 2.2.x module if I'm not mistaken) and without importing the faulted zpool?

Meikel said:
Jun 17 16:08:52 pve-01 kernel: INFO: task zfs:4577 is blocked on a mutex likely owned by task zpool:2954. Jun 17 16:08:52 pve-01 kernel: task:zpool state:D stack:0 pid:2954 tgid:2954 ppid:2951 task_flags:0x480100 flags:0x00004002
Click to expand...
Is that 2954 (or similar after reboot) zpool process then related to your rpool?
 
ZFS is on the "correct" version:

Code: 
root@pve-01:~# zfs --version
zfs-2.3.4-pve1
zfs-kmod-2.3.4-pve1

Sadly the issue issue isn't going away. I will try to ask for help in the Github issue again as downgrading doesn't seem to help.

The import service is down as you suggested:

Code: 
root@pve-01:~# systemctl status zfs-import@storage-01
â—‹ zfs-import@storage-01.service - Import ZFS pool storage-01
     Loaded: loaded (/usr/lib/systemd/system/zfs-import@.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:zpool(8)

The issue still persists (even after reboot). I can't even check the pool health via zpool status

I assume the PID is related to pool storage-01 not rpool - how can I check that?
 
Quickly skimming through the thread here and the upstream issue (so I might have missed something).

I think the core issue here is that the broken zpool storage-01 is still configured as storage in PVE? - in that case pvestatd would try to import it repeatedly - which would explain the hung tasks waiting for each of the imports.

This should probably be visible in `ps auxwf` output.

could you please also share the output of:
`zpool status`, `zpool list` and `zpool import -d /dev/disk/by-id` (that should list all pools that are currently not imported )
 
@Stoiko Ivanov The main problem is that no `zpool` command is working because they all hang up (can't even kill -9 them).

The issue also exists on a non-pve system (I tried to boot debian13 on the system and installed zfs manually with the same result).
 
Meikel said:
The main problem is that no `zpool` command is working because they all hang up (can't even kill -9 them).
Click to expand...

hm - if this also happens in a live-system that does not touch zfs before that's a bit odd (if anything or someone in the debian installation, tried running zpool import for all pools that would explain subsequent commands hanging)

as is - restoring the data on the broken storage-01 pool from backup is probably the cleanest way forward...
 
Too bad that downgrading the kernel/zfs didn't solve your issue.

It is unclear to me if your system is still trying to import the storage pool on boot. Like Stoiko said, there might be other triggers than the zfs-import service. That's why I asked if that pool is plain storage only or if it also has Proxmox related parts. Make sure your system/Proxmox doesn't need the storage pool somehow.

If your rpool is healthy like you said, you could go as far as physically disconnecting the storage pool from your system and see if your system then works as it should.

You could then technically put your problematic storage pool to another system and isolate the problem there.
Seems like it is a ZFS issue rather than a Proxmox issue anyway.
 
Last edited:
Yeah the issue is on ZFS' side. I only asked for downgrading guidance here so it's fine. I have to wait for the upstream issue to be resolved.

I tried importing the pool into a new system (live debian13) but the issue was the same there with the latest zfs tools. I will try it with some older tools tho and see how that turns out.
 
You must log in or register to reply here.
Top Bottom
Back