Hey all,
I just wasted several hours with a customer with this problem and thought I would share it in case anyone else has this problem.
We have a customer using Proxmox VE 8 with several blade servers, all doing iSCSI boot with open-iscsi via iBFT.
The iSCSI LUNs are initially mounted by the server firmware to start Grub then initramfs does its thing.
They have been using this configuration with great success with PVE 8.
Today we decided to upgrade 2 test nodes to PVE 9.
On the reboot initramfs failed to the console with the errors
> iscsistart: version 2.1.11
> iscsistart: Could not make /var/lib/iscsi 2
> iscsistart: exiting due to idbm configuration error
I could manually mount the iSCSI LUN in initramfs from the console and boot the host manually that way, but could not get initramfs to boot on its own without assistance.
Long story short, and several hours of troubleshooting and swearing, I ended up discovering this Debian bug report for open-iscsi
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1103644
The TLDR is open-iscsi 2.1.11-1 has a bug in it that the /var/lib/iscsi directory is missing in initramfs.
This causes the boot failure.
Unfortunately open-iscsi 2.1.11-1 is the version in Trixie 13.0, and as a result PVE 9.0.
This bug has been patched in open-iscsi 2.1.11-2, and there is a manually made deb package available to test linked in that bug report page.
There is also an active merge request with this patch waiting to be medged.
https://salsa.debian.org/linux-blocks-team/open-iscsi/-/merge_requests/22
We tested the patched open-iscsi 2.1.11-2 and it solved the problem right away, the servers booted without a problem with open-iscsi 2.1.11-2.
However we decided to keep everything on PVE 8.4 for now, and stay there until the patch is officially in the repos.
We don't want to be running out of band patched .deb files.
The merge request says they are aiming for Trixie 13.1, so hopefully it won't be too long and this will be fixed.
I just wanted to give anyone a warning to be aware of this iSCSI booting bug until the patch makes it into the repos.
It definitely would have saved me a morning of troubleshooting.
I just wasted several hours with a customer with this problem and thought I would share it in case anyone else has this problem.
We have a customer using Proxmox VE 8 with several blade servers, all doing iSCSI boot with open-iscsi via iBFT.
The iSCSI LUNs are initially mounted by the server firmware to start Grub then initramfs does its thing.
They have been using this configuration with great success with PVE 8.
Today we decided to upgrade 2 test nodes to PVE 9.
On the reboot initramfs failed to the console with the errors
> iscsistart: version 2.1.11
> iscsistart: Could not make /var/lib/iscsi 2
> iscsistart: exiting due to idbm configuration error
I could manually mount the iSCSI LUN in initramfs from the console and boot the host manually that way, but could not get initramfs to boot on its own without assistance.
Long story short, and several hours of troubleshooting and swearing, I ended up discovering this Debian bug report for open-iscsi
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1103644
The TLDR is open-iscsi 2.1.11-1 has a bug in it that the /var/lib/iscsi directory is missing in initramfs.
This causes the boot failure.
Unfortunately open-iscsi 2.1.11-1 is the version in Trixie 13.0, and as a result PVE 9.0.
This bug has been patched in open-iscsi 2.1.11-2, and there is a manually made deb package available to test linked in that bug report page.
There is also an active merge request with this patch waiting to be medged.
https://salsa.debian.org/linux-blocks-team/open-iscsi/-/merge_requests/22
We tested the patched open-iscsi 2.1.11-2 and it solved the problem right away, the servers booted without a problem with open-iscsi 2.1.11-2.
However we decided to keep everything on PVE 8.4 for now, and stay there until the patch is officially in the repos.
We don't want to be running out of band patched .deb files.
The merge request says they are aiming for Trixie 13.1, so hopefully it won't be too long and this will be fixed.
I just wanted to give anyone a warning to be aware of this iSCSI booting bug until the patch makes it into the repos.
It definitely would have saved me a morning of troubleshooting.
