Proxmox ZFS on Root Problems

[ichi]

Hey so thank you ahead of time for reading this. So in my homelab I run a 3 node Proxmox and Ceph cluster with all 3 nodes each running on a 2 SSD drive ZFS mirror. About a week ago one of my nodes had a kernel panic, I'm kicking myself that I didn't take a screen shot but I rebooted and haven't been able to get it to fully boot back up.

When I boot up and I choose the advance boot options for PVE I can boot from 3 different kernels. From those 3 kernels 2 of them end in a kernel panic(screenshot attached) and the last on ends with "error: failure reading sector 0xb4f20 from 'hd0'" (screenshot attached). So I started to do some troubleshooting with booting up a Live environment to see if I could import the zfs pool since it seems like that is the problem.

I tried booting from a USB with PVE using the debug console and a SystemResuce+ZFS as well as pulling the drives from the node and putting them into my Ubuntu test machine and all 3 I have not been able to import the rpool. When I use the command 'zpool import' rpool shows as online and when I use 'zpool import -R /mnt -N -f' it hangs for a bit but seems to complete. The problem is when I use 'zpool status' or 'zfs list' nothing shows up, both return " no pools available" / "no datasets available". The weird thing is that after a reboot or hot unplug/plug the pool will come back up showing as online like the import never happened.

I also ran a SMART test on both of the SSDs to see if anything turned up from that but that just showed that nothing was wrong with them besides having 3+ years of run time on them. So the only other thing I could think of to do is use the -F recovery mode flag on my 'zpool import' command but I have never used that before. So before I use that I was hoping to get some insight on possible things to try or ways to proceed?

Thanks for your time!

 

[fabian]

the pool not being visible after import sounds very strange.. is there anything visible in dmesg? you could try with "zdb" next in a live environment, it's a tool that allows you to examine a zpool without it being imported. or re-try the import with zfs debugging enabled: https://openzfs.github.io/openzfs-d...hooting.html#zfs-kernel-module-debug-messages

 

[ichi]

Hey sorry for the late response have been having late nights at work. So I finally had sometime to troubleshoot. I totally forgot about dmsg so i ran that as well as ran a 'zpool events' on my test machine. Attached are pics of some screenshots because otherwise this post would be too long. Based on Dmsg it seems that one of the disks is forsure dead and possibly/most likely the other is also dead? I'm going to pull the drives out of my test machine and put them back into the server node and run the 'zdb' command from the live environment next.

 

[fabian]

yes, that looks very much like a dead disk/controller/cable/..

 

[ichi]

yes, that looks very much like a dead disk/controller/cable/..

Yeah so I have tried different cables the controller in the system is fine. So it comes down to the SSD drive. Based on dmsg it seems that the drive connected at ata2 is the dead one and I might be able to recover the pool if I remove that drive and try and import the pool in a degraded state. By proceeding that way would it be best if I do all that on the node itself in a PVE debug console Live Environment? Also anything special I should consider doing if importing the degraded pool works out?

 

[ichi]

So a little update to the current situation. Both discs seem to be completely dead. Based on dmesg and everything I have tried to do to be able to try and read these disks between multiple machines, controllers and cables. Pretty unfortunate but anyways looks like a complete reinstall of Proxmox on new SSD's.

So then my other 2 nodes are both PVE: 7.4-3 with Ceph: 16.2.11. Would installing a fresh PVE 8 with the packaged Ceph be fine? Or should I find a PVE 7 iso and go that route? Say could I just update the PVE and Ceph on the current nodes that are running then the downed node could just be started fully updated from the start?

 

[fabian]

yeah, upgrading the existing cluster seems like the way to go. please follow the upgrade guide: https://pve.proxmox.com/wiki/Upgrade_from_7_to_8

please be aware that with only two nodes, rebooting or an outage of services on one of them would cause the other to not be quorate either. so if you have HA enabled, please disarm it first.

 

[ichi]

Okay so currently I dont have any of my VM's configured for HA at this moment. Also i have all my VM's using Ceph as their backend storage for easy fast migration and was going to eventually setup HA just have not gotten around to it yet. I did loose the config for the VM that was on the downed node because this all happened before the back up started for that VM but not the end of the world. I would like to clarify a couple things first.

yeah, upgrading the existing cluster seems like the way to go. please follow the upgrade guide: https://pve.proxmox.com/wiki/Upgrade_from_7_to_8

So should I reinstall the downed node with PVE 8 with Ceph Quincy and reintroduce it into the PVE and Ceph clusters then update the other 2 nodes? Or did you mean update the current 2 online nodes to PVE 8 and Ceph Quincy respectively? Of course either way I would have all VM's shutdown and follow the upgrade path for Ceph Pacific to Quincy and the upgrade path you linked for PVE 7 to 8.

please be aware that with only two nodes, rebooting or an outage of services on one of them would cause the other to not be quorate either. so if you have HA enabled, please disarm it first.

Yes correct and that is why I'm just trying to get all my ducks in a row before I proceed with reinstalling the down node/upgrading the current onine nodes. Since I do not have any of my VM's configured for HA and I'm just using Ceph as a backing storage for them at the moment, even though the nodes were all joined together in a PVE cluster, If I understand correctly the only quorum I have to worry about is Ceph's at the moment? Or am I mistaken in that the PVE cluster also has quorum that I have to worry about when HA is not enable?

Thank you Fabian for your insight and time I really do appreciate it!

 

[fabian]

upgrade first, then extend would be my choice.

PVE also has a quorum requirement without HA, but the failure mode is /etc/pve going read-only and certain/most operations not being available (like start/stop/.. guests, migration, backups, config changes ..) rather than nodes being fenced, like it would be with HA on.