ZFS Trouble, zed utilize 100% cpu

[syn]

Hi! May be somebody know how fix this, after upgrade zed process eat 100% from one cpu core.

Now i on Linux srv1 4.4.83-1-pve #1 SMP PVE 4.4.83-96 (Tue, 19 Sep 2017 10:30:12 +0200) x86_64 GNU/Linux

I think this is because versions of zfs modules and zfs utils mismatch:

root@srv1:/etc/apt# dpkg -l |grep zfs
ii libzfs2linux 0.7.3-pve1~bpo9
ii zfs-initramfs 0.7.3-pve1~bpo9
ii zfs-zed 0.7.3-pve1~bpo9
ii zfsutils-linux 0.7.3-pve1~bpo9

root@srv1:/etc/apt# dmesg|grep ZFS
[ 25.476228] ZFS: Loaded module v0.6.5.9-1, ZFS pool version 5000, ZFS filesystem version 5

How can i upgrade modules version ? or downgrade utils ? For this upgrade i use proxmox no-subscription repo only.
And sure i rebooted my system after upgrade.

 

[fabian]

you are running a kernel from PVE 4 and ZFS user space from PVE 5. reboot into a PVE 5 kernel..

 

[syn]

Thanks!

 

[iprigger]

All,
I do have a similar problem:

ii libzfs2 0.6.5.9-pve15~bpo80 all transitional package
ii libzfs2linux 0.7.3-pve1~bpo9 amd64 OpenZFS filesystem library for Linux
ii zfs-initramfs 0.7.3-pve1~bpo9 all OpenZFS root filesystem capabilities for Linux - initramfs
ii zfs-zed 0.7.3-pve1~bpo9 amd64 OpenZFS Event Daemon
ii zfsutils 0.6.5.9-pve15~bpo80 all transitional package
ii zfsutils-linux 0.7.3-pve1~bpo9 amd64 command-line tools to manage OpenZFS filesystems

proxmox-ve: 5.1-32 (running kernel: 4.13.13-1-pve)
pve-manager: 5.1-41 (running version: 5.1-41/0b958203)
pve-kernel-4.4.40-1-pve: 4.4.40-82
pve-kernel-4.4.35-2-pve: 4.4.35-79
pve-kernel-4.4.13-2-pve: 4.4.13-58
pve-kernel-4.4.24-1-pve: 4.4.24-72
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.4.62-1-pve: 4.4.62-88
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.13.13-2-pve: 4.13.13-32
pve-kernel-4.10.17-4-pve: 4.10.17-24
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.10.17-2-pve: 4.10.17-20
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.10.15-1-pve: 4.10.15-15
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.44-1-pve: 4.4.44-84
pve-kernel-4.4.16-1-pve: 4.4.16-64
pve-kernel-4.4.67-1-pve: 4.4.67-90
pve-kernel-4.13.13-1-pve: 4.13.13-31
pve-kernel-4.10.17-3-pve: 4.10.17-23
pve-kernel-4.4.59-1-pve: 4.4.59-87
libpve-http-server-perl: 2.0-8
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-19
qemu-server: 5.0-18
pve-firmware: 2.0-3
libpve-common-perl: 5.0-25
libpve-guest-common-perl: 2.0-14
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-17
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-3
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-5
pve-container: 2.0-18
pve-firewall: 3.0-5
pve-ha-manager: 2.0-4
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.1-2
lxcfs: 2.0.8-1
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9

I do have six nodes here, only one is acting up...

Any ideas?
Thanks
Tobias

 

[fabian]

check the running kernel on the affected node, and reboot it into the correct one.

 

[iprigger]

check the running kernel on the affected node, and reboot it into the correct one.

Yes, that was the last facepalm in 2017 for me.... it's been the wrong kernel and I checked and didn't see it. In addition, the update to proxmox 5.2 wasn't somehow complete. after an apt-get install proxmox-ve again it works now flawless.

tobias