Hi there! 
I've got Hetzner dedicated server with two identical NVMe drives with PVE default zfs (RAID1) installation which was working flawlessly for years. Last time I've upgraded PVE 11 days ago to 8.4.5, everything went well and the server came online after reboot as expected.
Yesterday the server became unavailable suddenly. Reboot which helped in such (rare) cases didn't at this time, but in Recovery mode (kind of Live CD) the server booted just fine, so I've decided to request IP KVM for troubleshooting.
After Recovery mode the server didn't boot because run-init wasn't able to import rpool as it was previously in use from another system (Recovery mode), which is expected.
I've imported it manually as suggested:
I've mounted the filesystems as suggested and exited:
Then I've tried to exit one more time and got kernel panic. Next reboot has been followed by kernel panic.
I've booted into Rescue mode one more time to ensure no hardware errors in
After that all boot attempts stuck at
Didn't find anything relevant on the forum and over Internet. Desperately need adviсe on further troubleshooting!
UPD: Found this thread. I do use Docker in one of my VMs, which virtual disk resides on ZFS storage...
I've got Hetzner dedicated server with two identical NVMe drives with PVE default zfs (RAID1) installation which was working flawlessly for years. Last time I've upgraded PVE 11 days ago to 8.4.5, everything went well and the server came online after reboot as expected.
Yesterday the server became unavailable suddenly. Reboot which helped in such (rare) cases didn't at this time, but in Recovery mode (kind of Live CD) the server booted just fine, so I've decided to request IP KVM for troubleshooting.
After Recovery mode the server didn't boot because run-init wasn't able to import rpool as it was previously in use from another system (Recovery mode), which is expected.
I've imported it manually as suggested:
zpool import rpool -f and got filesystem 'rpool/ROOT/pve-1' cannot be mounted at /root//mnt' due to canonicalization error: No such file or directory message.I've mounted the filesystems as suggested and exited:
zfs mount -a; exit. And got some Free initramfs and switch to another root fs messagess (guess count is equal to the number of filesystems in the rpool) followed by No init found. Try passing init= bootarg error.Then I've tried to exit one more time and got kernel panic. Next reboot has been followed by kernel panic.
I've booted into Rescue mode one more time to ensure no hardware errors in
dmesg output, fine smartctl output for both of drives and to do zpool scrub rpool just in case — no errors found.After that all boot attempts stuck at
cannot be mounted at /root//mnt' due to canonicalization error. Looks like something is missing between slashes... Tried to boot previous Grub2 entry with same result.Didn't find anything relevant on the forum and over Internet. Desperately need adviсe on further troubleshooting!
UPD: Found this thread. I do use Docker in one of my VMs, which virtual disk resides on ZFS storage...
Attachments
Last edited:
