PVE 8.2 / Kernel 6.8.4-2 does not boot - cannot find root device

pschneider1968 · Jun 27, 2024

A new PVE kernel version 6.8.8-2-pve has been released by Proxmox, and it contains the fix for this problem. I just installed and test-booted it on my machine, and it indeed works fine and fixes the issue, so I'd consider this CLOSED.

If you were affected and install and successfully test this kernel, you will have a line like this in your dmesg output:

Code:

root@linus:~# dmesg | grep VPD
[    3.483058] scsi 0:3:0:0: scsi_get_vpd_size: long VPD page 0 length: 516 bytes

This is the dev_warn_once from Martin's fix, see above.

kabello · Jun 28, 2024

pschneider1968 said:
A new PVE kernel version 6.8.8-2-pve has been released by Proxmox, and it contains the fix for this problem. I just installed and test-booted it on my machine, and it indeed works fine and fixes the issue, so I'd consider this CLOSED.

If you were affected and and install and successfully test this kernel, you will have a line like this in your dmesg output:

Code:

root@linus:~# dmesg | grep VPD [ 3.483058] scsi 0:3:0:0: scsi_get_vpd_size: long VPD page 0 length: 516 bytes

This is the dev_warn_once from Martin's fix, see above.

I updated to 6.8.8-2-pve and now, my PERC H310 (Dell T420 server) is working without any issues!

Thanks pschneider1968 for your support

pschneider1968 · Jun 28, 2024

kabello said:
I updated to 6.8.8-2-pve and now, my PERC H310 (Dell T420 server) is working without any issues!

Thanks pschneider1968 for your support

Great, happy to hear that!

alexvirt · Jul 9, 2024

Apparently the bug is still there. Few days ago I updated the home server with Proxmox and got the same result: no boot.
Buggy kernel: 6.8.8-2-pve
Kernel that works for me: 6.5.13-5-pve
M/B: AsRock Rack EP2C602-4L/D16
HDD: ST6000NE0021 (Seagate SATA 6 Tb)

But I don't have any additional drive adapters, just builtin SATA.

# pve-efiboot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
6.5.13-5-pve
6.8.8-2-pve

# lsscsi
[8:0:0:0] disk ATA ST6000NE0021-2EN EN02 /dev/sda
[14:0:0:0] process Marvell Console 1.01 -

# dmesg | grep "Attached"
[ 3.568778] sd 8:0:0:0: Attached scsi generic sg0 type 0
[ 3.569299] scsi 14:0:0:0: Attached scsi generic sg1 type 3
[ 3.600859] sd 8:0:0:0: [sda] Attached SCSI disk

# sg_vpd --all -HHHH /dev/sg0
00 00 00 08 00 80 83 89 b0 b1 b2 b9
....
00 b2 00 04 00 40 00 00

fetching VPD page failed: Illegal request
sg_vpd failed: Illegal request

Any suggestions????

pschneider1968 · Jul 9, 2024

alexvirt said:
Apparently the bug is still there. Few days ago I updated the home server with Proxmox and got the same result: no boot.
Buggy kernel: 6.8.8-2-pve
Kernel that works for me: 6.5.13-5-pve
M/B: AsRock Rack EP2C602-4L/D16
HDD: ST6000NE0021 (Seagate SATA 6 Tb)

But I don't have any additional drive adapters, just builtin SATA.

# pve-efiboot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
6.5.13-5-pve
6.8.8-2-pve

# lsscsi
[8:0:0:0] disk ATA ST6000NE0021-2EN EN02 /dev/sda
[14:0:0:0] process Marvell Console 1.01 -

# dmesg | grep "Attached"
[ 3.568778] sd 8:0:0:0: Attached scsi generic sg0 type 0
[ 3.569299] scsi 14:0:0:0: Attached scsi generic sg1 type 3
[ 3.600859] sd 8:0:0:0: [sda] Attached SCSI disk

# sg_vpd --all -HHHH /dev/sg0
00 00 00 08 00 80 83 89 b0 b1 b2 b9
....
00 b2 00 04 00 40 00 00

fetching VPD page failed: Illegal request
sg_vpd failed: Illegal request

Any suggestions????

I would suggest that you create a new bug report on the Proxmox Bugzilla, and also write a bug report to the Linux Kernel mailing list with CC linux-scsi, linux-stable and Martin K. Petersen. Probably best as a reply to his mail where he announced the Patch:

https://lore.kernel.org/linux-scsi/20240521023040.2703884-1-martin.petersen@oracle.com/

(It is easiest to follow LKML via the NNTP interface of lore.kernel.org - that's how I read the mailing lists)

I guess Martin and the other SCSI developers would be interested in the exact and full output of your last "sg_vpd" command, as that might show him in which exact way your device is buggy (with respect to the SCSI standard) and his scanning code still cannot deal with.

You should also check if you have any kernel crash log from the failed boot in /var/lib/systemd/pstore/ and if so, include that in the bug report. This might have a full stack trace, which normally scrolls too fast across the console to notice or copy. The Pstore thing might work and be present if your machine has some mechanism like ERST via BIOS to persistently store diagnostic early boot messages across boots even in case of a kernel panic (typically more often found in server hardware though).

tentpiglet · Jul 21, 2024

alexvirt said:
Apparently the bug is still there. Few days ago I updated the home server with Proxmox and got the same result: no boot.
Buggy kernel: 6.8.8-2-pve
Kernel that works for me: 6.5.13-5-pve

I have a Dell PowerEdge 730xd with a PERC H310 Mini. Running an 6.8.8-2 or 6.8.8-3 kernel I encounter this (although not exact) problem. If I revert back to 6.5.13-5 everything works fine. So whatever the "bug" is, its still there -- or more than one bug has been introduced in the 6.8.8 stream somewhere along the line.

Specifically, I get the "Timed out for waiting the udev queue being empty" message on boot and booting can take several (5+) minutes. However, my system boots because my boot disk is on a Dell BOSS-S1 card, so it isn't the boot disk that has the issue.

Where I run into a problem is with the VMs that access VDs on the H310. With the 6.8.8.x kernels I cannot reliably access the H310 virtual disks. Sometimes they will mount, othertimes the system will just hang (at the proxmox level, requiring a cold start via the idrac) attempting to access them. All the H310 VDs are connected to VMs as SCSI drives via passthrough, no sata.

I also have a PERC H800 controller in this system which connects to an MD1200 array, I have no issues at all with those VDs, its just the H310.

Obviously my message is posted about 2 weeks from the last post on this issue because I do not reboot my proxmox servers all that often, so didn't encounter the problem until today when I had to perform some maintenance, and this is where google lead me.

FoodFighter · Aug 14, 2024

R1ch13 said:
In my case , it was my coral TPU (PCIe) . Had to rebuild driver. All is OK now.

Can you explain, how you did that?

I'm facing the same issue since I updated to PVE8.2.4 -> System will not boot anymore.
Kernels 6.8.8-4-pve and 6.8.12-1-pve won't boot

Code:

/dev/root: Can't open blockdev
VFS: cannot open root device "/dev/mapper/pve-root"§ or unknown-block(0,0): error -6

By selecting Kernel 6.5.13-6-pve in grub everything is booting fine.

I also have a Google Coral in my Futro S740 (Mainboard: D3544-S2)
Maybe the Coral is the Problem.

How to rebuild the driver?
Do I have to boot the new kernel anyway to rebuild the driver? (I don't think it makes much sense rebuilding the driver when I'm on the old, working Kernel)
can you explain to me, how you did the rebuild?

FoodFighter · Aug 15, 2024

The problem has been solved for me.
It seems someone has been able to solve this issue here.

If someone needs that:
select an older kernel in grub to boot the System anyway.
than:

Code:

dpkg --list@grep proxmox-kernel

for me the newes one was 6.8.12-1-pve

Code:

sudo update-initramfs -u -k 6.8.12-1-pve

Code:

sudo update-grub

Code:

reboot now

now everything works fine again.

madindehead · Sep 1, 2024

FoodFighter said:
The problem has been solved for me.
It seems someone has been able to solve this issue here.

If someone needs that:
select an older kernel in grub to boot the System anyway.
than:

Code:

dpkg --list@grep proxmox-kernel

for me the newes one was 6.8.12-1-pve

Code:

sudo update-initramfs -u -k 6.8.12-1-pve

Code:

sudo update-grub

Code:

reboot now

now everything works fine again.

Has anyone else verified that this fix is working? Hopefully someone can confirm before I look at going back to one of the newer kernels.

christianh32 · Sep 13, 2024

madindehead said:
Has anyone else verified that this fix is working? Hopefully someone can confirm before I look at going back to one of the newer kernels.

I tried the fix the above and it did not work for me unfortunately. I pinned the older kernel in the meantime. If anyone is still having the issue I would love to find a fix. The issue only happened after upgrading from 8.1-2 to 8.2.-2.
Dell R420 with H310 PERC and SAS drives. During my testing i found the Proxmox installer for 8.1-2 sees the drives and allowed the Proxmox install, but the 8.2-2 installer didn't' see the drives. I installed 8.1-2 then upgraded to 8.2-2 and then had the issue where it "Failed to mount pveroot as root file system" . This is a test server, but would still love to figure it out.

alexvirt · Oct 20, 2024

The latest kernel 6.8.12-2 still does not work at my AsRock (mentioned previously). Need to find out a way how to grab logs and then create a bug request and upload collected logs. So far stick with 6.5.13-5

Search

Search

PVE 8.2 / Kernel 6.8.4-2 does not boot - cannot find root device

pschneider1968

Member

kabello

New Member

pschneider1968

Member

alexvirt

New Member

pschneider1968

Member

tentpiglet

New Member

FoodFighter

New Member

FoodFighter

New Member

madindehead

Member

christianh32

New Member

alexvirt

New Member