Warning about open-iscsi for admins upgrading Proxmox VE 8 to 9 with iSCSI boot

[Lindsay-Sol1]

Hey all,

I just wasted several hours with a customer with this problem and thought I would share it in case anyone else has this problem.

We have a customer using Proxmox VE 8 with several blade servers, all doing iSCSI boot with open-iscsi via iBFT.
The iSCSI LUNs are initially mounted by the server firmware to start Grub then initramfs does its thing.

They have been using this configuration with great success with PVE 8.

Today we decided to upgrade 2 test nodes to PVE 9.

On the reboot initramfs failed to the console with the errors
> iscsistart: version 2.1.11
> iscsistart: Could not make /var/lib/iscsi 2
> iscsistart: exiting due to idbm configuration error

I could manually mount the iSCSI LUN in initramfs from the console and boot the host manually that way, but could not get initramfs to boot on its own without assistance.

Long story short, and several hours of troubleshooting and swearing, I ended up discovering this Debian bug report for open-iscsi
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1103644

The TLDR is open-iscsi 2.1.11-1 has a bug in it that the /var/lib/iscsi directory is missing in initramfs.
This causes the boot failure.

Unfortunately open-iscsi 2.1.11-1 is the version in Trixie 13.0, and as a result PVE 9.0.

This bug has been patched in open-iscsi 2.1.11-2, and there is a manually made deb package available to test linked in that bug report page.
There is also an active merge request with this patch waiting to be medged.
https://salsa.debian.org/linux-blocks-team/open-iscsi/-/merge_requests/22

We tested the patched open-iscsi 2.1.11-2 and it solved the problem right away, the servers booted without a problem with open-iscsi 2.1.11-2.
However we decided to keep everything on PVE 8.4 for now, and stay there until the patch is officially in the repos.
We don't want to be running out of band patched .deb files.

The merge request says they are aiming for Trixie 13.1, so hopefully it won't be too long and this will be fixed.

I just wanted to give anyone a warning to be aware of this iSCSI booting bug until the patch makes it into the repos.
It definitely would have saved me a morning of troubleshooting.

 

[softworx]

Thanks, I wish I had seen this post a couple of hours ago, I would have waited.

I just got bit by this and had to slog my way through it. I came to see if anyone else ran in this and found your post.

Just to be clear for anyone else the comes by, this is how I resumed the boot process. Fortunately it was easy.

mkdir -p /var/lib/iscsi
iscsistart -b

after that iscsi logged in, saw the boot drives, and everything was there.

finally at the initramfs prompt, i just did ctrl-d to exit and the boot resumed.

I don't want to run out of band debs either, but it's more important that i have this boot safe, so i'm going to check the patched iscsi package out for the short term.

and i'll think again about upgrading the rest of the cluster. heh

After getting it installed, I downloaded the files:
* libopeniscsiusr_2.1.11-2_amd64.deb
* open-iscsi_2.1.11-2_amd64.deb
and installed them both with same command (if doing separate, install lib first).

Now reboots no problem

Thanks again for your post.

 

[kapone]

Ahhh....I got bit by this as well.

Mucho gracias both of you, for the fix.