I installed Clevis on my proxmox host in order to unlock an encrypted external USB storeage drive at boot. I put Tang on an old raspberry pi. I rebooted 3 times, and it worked perfectly from the start. Then I shut down my host, and performed a cold boot. I received a bunch of errors in my terminal. First being "245 Cant open /scripts/zfs. The bottom line of my terminal reads "Kernal offset... Relocation range... Kernel Panic...Not syncing....".
So here is what led to the issue
1) First I installed the wrong package. I followed some instructions from Oracle. and Installed <<sudo apt install -y clevis clevis-luks clevis-udisks2 clevis-dracut>>> I didn't know what dracut was at the time.... I received some error messages about a an iscsiuio socket not existing....so when I checked the status of iscsiuio on my system it seemed the socket was enabled but the isciuio service was disabled. It said vendor default was enabled so I enabled it. Well, actually, I'm not certain what service I enabled I checked 4 items with systemctl.....(iscsiuio and iscsid - both services and sockets). I just looked in my systemd folder and it seems iscsid has some dependencies that were updated today. perhaps thats what I enabled I may have updated it by mistake...
2) After that didn't work to get through the errors on the clevis install, I did some research on dracut. I realized there was a clevis initramfs version that I should have installed instead. I uninstalled the 3 clevis packages. I reinstalled clevis, including the clevis-initramfs package. However I did not disable the iscsiuio service that I enabled
3) I configured Clevis and Tang. Then I assumed I needed to update initramfs (based on seeing that in the Oracle docs they updated dracut), so I did an update-initramfs -u
4) It appeared to work the first 3 times when I executed a reboot. However, I didn't see any indication on the terminal of what was happening. However, it did appear to me that it was rebooting, as my drives would power down and power up. When I actually shut down and booted from cold boot, got the errors described.
Thoughts:
I had entries in crypttab and fstab to mount the USB drive manually by typing in the password at boot (my setup before installing clevis). I left the same config. My crypttab has 'none' for the password and creates a devmapper name that I use to mount in fstab. Should have I changed any of that setup?
Or is it some issue with clevis? Or perhaps the dracut package has some leftovers? Or is it the fact I enabled iscsiuio.service and left it enabled when I updated initramfs?
FYI the encrypted is not my proxmox boot drive, nor does it store any volumes, its just a USB Data Drive. My boot drives and volume storeage are zfs, unencrypted.
Questions:
1) Is it possible to recover, and if so, what are the exact steps? i believe I've successfully chrooted into the file system using the proxmox debug installer and importing my zfs and mounting required folder. I just have not a clue on what to do to fix this..
3) FYI I tried booting from the prior proxmox kernel (6.5.11-8) ... It took me to a blank black screen and hung. both 6.5.11-8 and ...6.5.13-1 images have dates as of today. when I do an initramfs$(uname-r).img from chroot, it indicates i'm using 11-8. But also tells me command not found. I find that confusing because its 13-1 that gives me the errors. 11-8 goes to a blank screen. I used the -u option only when I ran update-initramfs (so I thought it was only updating the current version) I really have no knowledge of kernals, versions, etc. This is out of my realm. I just need to get back to a working kernel. I wish I could use systemctl to disable both clevis and iscsid/iscsiuio(or at least look to see which one I might have changed.) I guess its not possible to update the services in chroot. I really don't have the knowledge to know what to try....
Again, my primary errors are
First being "245 Cant open /scripts/zfs no such file or directory.
"Kernal offset xxxxx from xxxxxxx... Relocation range: xxxxxx-xxxxxx)... Kernel Panic....Not Syncing....".
So here is what led to the issue
1) First I installed the wrong package. I followed some instructions from Oracle. and Installed <<sudo apt install -y clevis clevis-luks clevis-udisks2 clevis-dracut>>> I didn't know what dracut was at the time.... I received some error messages about a an iscsiuio socket not existing....so when I checked the status of iscsiuio on my system it seemed the socket was enabled but the isciuio service was disabled. It said vendor default was enabled so I enabled it. Well, actually, I'm not certain what service I enabled I checked 4 items with systemctl.....(iscsiuio and iscsid - both services and sockets). I just looked in my systemd folder and it seems iscsid has some dependencies that were updated today. perhaps thats what I enabled I may have updated it by mistake...
2) After that didn't work to get through the errors on the clevis install, I did some research on dracut. I realized there was a clevis initramfs version that I should have installed instead. I uninstalled the 3 clevis packages. I reinstalled clevis, including the clevis-initramfs package. However I did not disable the iscsiuio service that I enabled
3) I configured Clevis and Tang. Then I assumed I needed to update initramfs (based on seeing that in the Oracle docs they updated dracut), so I did an update-initramfs -u
4) It appeared to work the first 3 times when I executed a reboot. However, I didn't see any indication on the terminal of what was happening. However, it did appear to me that it was rebooting, as my drives would power down and power up. When I actually shut down and booted from cold boot, got the errors described.
Thoughts:
I had entries in crypttab and fstab to mount the USB drive manually by typing in the password at boot (my setup before installing clevis). I left the same config. My crypttab has 'none' for the password and creates a devmapper name that I use to mount in fstab. Should have I changed any of that setup?
Or is it some issue with clevis? Or perhaps the dracut package has some leftovers? Or is it the fact I enabled iscsiuio.service and left it enabled when I updated initramfs?
FYI the encrypted is not my proxmox boot drive, nor does it store any volumes, its just a USB Data Drive. My boot drives and volume storeage are zfs, unencrypted.
Questions:
1) Is it possible to recover, and if so, what are the exact steps? i believe I've successfully chrooted into the file system using the proxmox debug installer and importing my zfs and mounting required folder. I just have not a clue on what to do to fix this..
3) FYI I tried booting from the prior proxmox kernel (6.5.11-8) ... It took me to a blank black screen and hung. both 6.5.11-8 and ...6.5.13-1 images have dates as of today. when I do an initramfs$(uname-r).img from chroot, it indicates i'm using 11-8. But also tells me command not found. I find that confusing because its 13-1 that gives me the errors. 11-8 goes to a blank screen. I used the -u option only when I ran update-initramfs (so I thought it was only updating the current version) I really have no knowledge of kernals, versions, etc. This is out of my realm. I just need to get back to a working kernel. I wish I could use systemctl to disable both clevis and iscsid/iscsiuio(or at least look to see which one I might have changed.) I guess its not possible to update the services in chroot. I really don't have the knowledge to know what to try....
Again, my primary errors are
First being "245 Cant open /scripts/zfs no such file or directory.
"Kernal offset xxxxx from xxxxxxx... Relocation range: xxxxxx-xxxxxx)... Kernel Panic....Not Syncing....".
Last edited: