[SOLVED] PVE 6.1: ZFS Segfault upon system boot

[ThinkAgain]

Hi, upon booting the system (only during boot so far), I get a segfault in ZFS:

[   17.905941] ZFS: Loaded module v0.8.3-pve1, ZFS pool version 5000, ZFS filesystem version 5
[...][   20.804544] zfs[4387]: segfault at 0 ip 00007f565ddde694 sp 00007f5656ff5420 error 4
[   20.804546] zfs[4379]: segfault at 0 ip 00007f565deba01c sp 00007f565d509478 error 4
[   20.804547]  in libc-2.28.so[7f565dd84000+148000]
[   20.805298]  in libc-2.28.so[7f565dd84000+148000]
[   20.805846] Code: 29 f2 41 ff 55 70 48 85 c0 7e 3b 48 8b 93 90 00 00 00 48 01 43 10 48 83 fa ff 74 0a 48 01 d0 48 89 83 90 00 00 00 48 8b 43 08 <0f> b6 00 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3 66 2e 0f 1f
[   20.806365] Code: 29 c8 c5 f8 77 c3 0f 1f 84 00 00 00 00 00 48 85 d2 0f 84 5a 02 00 00 89 f9 c5 f9 6e c6 c4 e2 7d 78 c0 83 e1 3f 83 f9 20 77 44 <c5> fd 74 0f c5 fd d7 c1 85 c0 0f 85 c4 01 00 00 48 83 ea 20 0f 86

I haven't yet noticed any data corruption, but who knows.

 

[wolfgang]

Hi,

please send the output of.

pveversion -v

 

[ThinkAgain]

Sure, here you go:

# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-7
pve-kernel-5.3: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-22
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

 

[wolfgang]

I do not see this here on my Servers.
Do you use the ZFS as rootfs?
What HW do you use with ZFS(SSD/HDD, CPU, ...) ?

 

[ThinkAgain]

Yes, I haven't found out, yet, either, how to reproduce this bug with certainty. It's just weird that this is happening from time to time, and gives me a bit of a bad feeling.

The machine here is an EPYC 7502P on a TYAN platform with 256 GB RAM. It uses NVME and SATA SSDs. Boot is from two mirrored (ZFS, I had meant to use md, but this is not officially supported) Kingston DC1000B SSDs. There are also two mirrored P4800X, 6 Micron 9300 Max in striped mirrors, and 4 SATA SSDs (Samsung, Kingston) in a striped mirror in the system.

 

[wolfgang]

I can't reproduce it here but my setup is quite different.
Please try the pve-kernel-5.4 maybe it solve it.
If not I can hopefully test is next week with a similar HW.

 

[ThinkAgain]

Thanks @wolfgang
This is just to confirm that I haven't seen a segfault anymore since updating to 5.4. So this has probably fixed it!

 

[ThinkAgain]

@wolfgang: I'm sorry that I have to report that the segfault upon boot is back.

#dmesg | grep zfs
[   23.442792] traps: zfs[10790] general protection fault ip:7fbe057054a6 sp:7fbdf7ff7310 error:0 in libc-2.28.so[7fbe056a3000+148000]
[   23.462145] systemd[1]: zfs-mount.service: Main process exited, code=killed, status=11/SEGV
[   23.485228] systemd[1]: zfs-mount.service: Failed with result 'signal'.

pveversion -v:

proxmox-ve: 6.2-1 (running kernel: 5.4.41-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.4.27-1-pve: 5.4.27-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

 

[wolfgang]

This is not the same class of errors.

The error comes now from the zfs-mount.service.
Is there more information in journald?

 

[ThinkAgain]

Indeed, the journal contained a bit more information. This appears to be related to this other issue I had, where a Directory configured in PVE on a ZFS resource populated the mount directory before ZFS could mount the dataset. And once the mount directory contained something, ZFS threw an error when mounting.

I'll mark this thread as solved. Thanks again!