Hi all,
what happened?
A PVE server commissioned in 2018 ceased service after almost 2 years of trouble-free continuous operation; server access was no longer possible, neither PVE-GUI nor other services; ping was still possible.
A login to the physical console was still successful. The subsequent shutdown got stuck on shutdown of the VMs…
Now only a reset remained as an option.
The subsequent boot process will hang when the root system is importing rpool.
Shortly after that a lot of kernel messages appear every two minutes:
After an initial hardware check, a hardware problem can be ruled out.
rpool is on a mirror of 2x SSD Samsung 850 Pro 1TB, another pool on a raidz2 with 4x 4TB HDD, no ZIL, no L2ARC.
CPU: AMD FX-8350
RAM: 32GB DDR3 ECC
Proxmox, probably PVE 5.2
Kernel: 4.15.18-21-pve
The SSDs are okay according to
The big problem: rpool can no longer be imported in any way that I know of, even though a
A
No other behavior:
This was tested on different hardware under different kernel versions – last under the current PVE-6.2-1
The following message is conspicuous:
I have found absolutely no clues on the net in connection with ZFS.
Have we come across a new bug in the ZFS universe here?
Who has a helpful idea to get the server up and running again?
what happened?
A PVE server commissioned in 2018 ceased service after almost 2 years of trouble-free continuous operation; server access was no longer possible, neither PVE-GUI nor other services; ping was still possible.
A login to the physical console was still successful. The subsequent shutdown got stuck on shutdown of the VMs…
Now only a reset remained as an option.
The subsequent boot process will hang when the root system is importing rpool.
Shortly after that a lot of kernel messages appear every two minutes:
After an initial hardware check, a hardware problem can be ruled out.
rpool is on a mirror of 2x SSD Samsung 850 Pro 1TB, another pool on a raidz2 with 4x 4TB HDD, no ZIL, no L2ARC.
CPU: AMD FX-8350
RAM: 32GB DDR3 ECC
Proxmox, probably PVE 5.2
Kernel: 4.15.18-21-pve
The SSDs are okay according to
smartctl.The big problem: rpool can no longer be imported in any way that I know of, even though a
zpool import shows an intact pool:
Code:
# zpool import
pool: rpool
id: 9373167444002024865
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the '-f' flag.
see: http://zfsonlinux.org/msg/ZFS-8000-EY
config:
rpool ONLINE
mirror-0 ONLINE
sda2 ONLINE
sdc2 ONLINEA
zpool import -f rpool does not end with a normal error message, but instead triggers a kernel panic:No other behavior:
zpool import -f -F -m -N -o cachefile=none rpoolThis was tested on different hardware under different kernel versions – last under the current PVE-6.2-1
The following message is conspicuous:
PANIC: blkptr at 000000004ab2be1f has invalid TYPE 140I have found absolutely no clues on the net in connection with ZFS.
Have we come across a new bug in the ZFS universe here?
Who has a helpful idea to get the server up and running again?
