Proxmox upgrade 8 to 9 on zfs boot: upgrade fails with grub errors

J

jcol

New Member
Aug 11, 2025
8
1
3
(Thread moved on request @fabian )

Hello, we have upgraded 6 PVE 8 servers via apt successfully and without any glitch or problem. But number 7 gives us headaches. This one is the only server with ZFS as boot / root file system, the others have LVM / ext4 setups. We upgraded to the latest 8.4 version and booted into that successfully. Then we did all the steps outlined in the docs and finally an apt dist-upgrade. This ended with :

run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.14.8-2-pve /boot/vmlinuz-6.14.8-2-pve
update-initramfs: Generating /boot/initrd.img-6.14.8-2-pve

Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/B336-C9A1
Copying kernel 6.14.8-2-pve
Copying kernel 6.8.12-13-pve
/usr/sbin/grub-probe: error: failed to get canonical path of `/dev/disk/by-id/scsi-35002538840116350-part3'.
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1

which means we now have a fully upgraded 9.0 system running on kernel 6.8.12 as shown by the 8to9 script:

Checking proxmox-ve package version..
PASS: already upgraded to Proxmox VE 9

Checking running kernel version..
WARN: unexpected running and installed kernel '6.8.12-13-pve'.

I do not even dare to reboot in this state. The installation is pretty normal, the ZFS status of the "rpool" is fine and I cannot see why it could build a new boot config when upgrading to the latest 8.4 but not when doing 9.0 30 minutes later. Does anybody have a hint how to get out of this short of reinstalling?
I also tried to create the links to the disks manually (/dev/disk/by-id/scsi-35002538840116350-part3 and one other disk) but then grub-probe complains about "# grub-probe /
grub-probe: error: unknown filesystem."

The lsblk output says (the relevant disks are sda and sde, the rest are also ZFS but used for VMs and not relevant for booting)
root@pve04:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 476.9G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 1G 0 part
└─sda3 8:3 0 475.9G 0 part
sdb 8:16 0 1.7T 0 disk
├─sdb1 8:17 0 1.7T 0 part
└─sdb9 8:25 0 8M 0 part
sdc 8:32 0 9.1T 0 disk
├─sdc1 8:33 0 9.1T 0 part
└─sdc9 8:41 0 8M 0 part
sdd 8:48 0 9.1T 0 disk
├─sdd1 8:49 0 9.1T 0 part
└─sdd9 8:57 0 8M 0 part
sde 8:64 0 476.9G 0 disk
├─sde1 8:65 0 1007K 0 part
├─sde2 8:66 0 1G 0 part
└─sde3 8:67 0 475.9G 0 part
sdf 8:80 0 1.7T 0 disk
├─sdf1 8:81 0 1.7T 0 part
└─sdf9 8:89 0 8M 0 part
sr0 11:0 1 1024M 0 rom
zd0 230:0 0 200G 0 disk
└─zd0p1 230:1 0 200G 0 part
zd16 230:16 0 50G 0 disk
zd32 230:32 0 2T 0 disk
├─zd32p1 230:33 0 512M 0 part
├─zd32p2 230:34 0 19.9G 0 part
├─zd32p3 230:35 0 4G 0 part
├─zd32p4 230:36 0 1.9T 0 part
└─zd32p5 230:37 0 1M 0 part
zd48 230:48 0 100G 0 disk
zd64 230:64 0 100G 0 disk
├─zd64p1 230:65 0 1M 0 part
├─zd64p2 230:66 0 2G 0 part
└─zd64p3 230:67 0 98G 0 part
zd80 230:80 0 100G 0 disk
├─zd80p1 230:81 0 1M 0 part
├─zd80p2 230:82 0 2G 0 part
└─zd80p3 230:83 0 98G 0 part
zd96 230:96 0 100G 0 disk
├─zd96p1 230:97 0 1M 0 part
├─zd96p2 230:98 0 1G 0 part
├─zd96p3 230:99 0 59G 0 part
└─zd96p4 230:100 0 40G 0 part
zd112 230:112 0 1M 0 disk
zd128 230:128 0 100G 0 disk
├─zd128p1 230:129 0 100M 0 part
├─zd128p2 230:130 0 16M 0 part
├─zd128p3 230:131 0 99.3G 0 part
└─zd128p4 230:132 0 562M 0 part
zd144 230:144 0 400G 0 disk
├─zd144p1 230:145 0 200G 0 part
└─zd144p2 230:146 0 200G 0 part
zd160 230:160 0 4G 0 disk
├─zd160p1 230:161 0 2.2G 0 part
├─zd160p2 230:162 0 94M 0 part
├─zd160p3 230:163 0 476M 0 part
└─zd160p4 230:164 0 952.7M 0 part
zd176 230:176 0 50G 0 disk
├─zd176p1 230:177 0 1M 0 part
├─zd176p2 230:178 0 2G 0 part
└─zd176p3 230:179 0 48G 0 part
zd192 230:192 0 50G 0 disk
├─zd192p1 230:193 0 1M 0 part
├─zd192p2 230:194 0 2G 0 part
└─zd192p3 230:195 0 48G 0 part
zd208 230:208 0 1M 0 disk
zd224 230:224 0 150G 0 disk
zd240 230:240 0 100G 0 disk
├─zd240p1 230:241 0 1M 0 part
├─zd240p2 230:242 0 2G 0 part
└─zd240p3 230:243 0 98G 0 part
zd256 230:256 0 4M 0 disk
zd272 230:272 0 100G 0 disk
├─zd272p1 230:273 0 1M 0 part
├─zd272p2 230:274 0 2G 0 part
└─zd272p3 230:275 0 98G 0 part
zd288 230:288 0 100G 0 disk
├─zd288p1 230:289 0 600M 0 part
├─zd288p2 230:290 0 1G 0 part
└─zd288p3 230:291 0 98.4G 0 part

zpool status says (for the relevant rpool):
pool: rpool
state: ONLINE
scan: scrub repaired 0B in 00:00:24 with 0 errors on Sun Aug 10 00:24:29 2025
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35002538840116350-part3 ONLINE 0 0 0
scsi-3500253884011633d-part3 ONLINE 0 0 0



Glad for any hints, JC
 
what happens if you run "proxmox-boot-tool refresh" now?
 
Exactly the same result:
proxmox-boot-tool refresh
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/B336-C9A1
Copying kernel 6.14.8-2-pve
Copying kernel 6.8.12-13-pve
/usr/sbin/grub-probe: error: failed to get canonical path of `/dev/disk/by-id/scsi-35002538840116350-part3'.

And actually that link is not there, as the second one for sde. If I create these links to sda3 and sde3 manually, the error messages changes to "error: unknown filesystem."
 
ls -l /dev/disk/by-id/scsi-*
lrwxrwxrwx 1 root root 9 Aug 11 12:33 /dev/disk/by-id/scsi-35000c50094491cb3 -> ../../sdc
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-35000c50094491cb3-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-35000c50094491cb3-part9 -> ../../sdc9
lrwxrwxrwx 1 root root 9 Aug 11 12:33 /dev/disk/by-id/scsi-35000c500944fade7 -> ../../sdd
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-35000c500944fade7-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-35000c500944fade7-part9 -> ../../sdd9
lrwxrwxrwx 1 root root 9 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a950f -> ../../sdb
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a950f-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a950f-part9 -> ../../sdb9
lrwxrwxrwx 1 root root 9 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a95a8 -> ../../sdf
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a95a8-part1 -> ../../sdf1
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a95a8-part9 -> ../../sdf9
 
could you post the output of:

Code: 
for d in /dev/sd? ; do echo; echo $d;echo;udevadm info $d; done
 
# for d in /dev/sd? ; do echo; echo $d;echo;udevadm info $d; done

/dev/sda

P: /devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:10/0:0:10:0/block/sda
M: sda
J: b8:0
U: block
T: disk
D: b 8:0
N: sda
L: 0
S: disk/by-id/wwn-0x5002538840116350
S: disk/by-path/pci-0000:81:00.0-scsi-0:0:10:0
S: disk/by-id/ata-Samsung_SSD_850_PRO_512GB_S250NXAGB26374F
S: disk/by-diskseq/9
Q: 9
E: DEVPATH=/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:10/0:0:10:0/block/sda
E: DEVNAME=/dev/sda
E: DEVTYPE=disk
E: DISKSEQ=9
E: MAJOR=8
E: MINOR=0
E: SUBSYSTEM=block
E: USEC_INITIALIZED=5625814
E: ID_ATA=1
E: ID_TYPE=disk
E: ID_BUS=ata
E: ID_MODEL=Samsung_SSD_850_PRO_512GB
E: ID_MODEL_ENC=Samsung\x20SSD\x20850\x20PRO\x20512GB\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_REVISION=EXM02B6Q
E: ID_SERIAL=Samsung_SSD_850_PRO_512GB_S250NXAGB26374F
E: ID_SERIAL_SHORT=S250NXAGB26374F
E: ID_ATA_WRITE_CACHE=1
E: ID_ATA_WRITE_CACHE_ENABLED=1
E: ID_ATA_READ_LOOKAHEAD=1
E: ID_ATA_READ_LOOKAHEAD_ENABLED=1
E: ID_ATA_FEATURE_SET_HPA=1
E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
E: ID_ATA_FEATURE_SET_PM=1
E: ID_ATA_FEATURE_SET_PM_ENABLED=1
E: ID_ATA_FEATURE_SET_SECURITY=1
E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=2
E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=2
E: ID_ATA_FEATURE_SET_SMART=1
E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
E: ID_ATA_DOWNLOAD_MICROCODE=1
E: ID_ATA_SATA=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
E: ID_ATA_ROTATION_RATE_RPM=0
E: ID_WWN=0x5002538840116350
E: ID_WWN_WITH_EXTENSION=0x5002538840116350
E: ID_ATA_PERIPHERAL_DEVICE_TYPE=0
E: ID_PATH=pci-0000:81:00.0-scsi-0:0:10:0
E: ID_PATH_TAG=pci-0000_81_00_0-scsi-0_0_10_0
E: ID_PART_TABLE_UUID=3aad1dc4-a394-42ab-82a5-ae32be03c6d9
E: ID_PART_TABLE_TYPE=gpt
E: DEVLINKS=/dev/disk/by-id/wwn-0x5002538840116350 /dev/disk/by-path/pci-0000:81:00.0-scsi-0:0:10:0 /dev/disk/by-id/ata-Samsung_SSD_850_PRO_512GB_S250NXAGB26374F /dev/disk/by-diskseq/9
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:


/dev/sdb

P: /devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:11/0:0:11:0/block/sdb
M: sdb
J: b8:16
U: block
T: disk
D: b 8:16
N: sdb
L: 0
S: disk/by-path/pci-0000:81:00.0-scsi-0:0:11:0
S: disk/by-id/scsi-355cd2e41511a950f
S: disk/by-id/wwn-0x55cd2e41511a950f
S: disk/by-diskseq/10
Q: 10
E: DEVPATH=/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:11/0:0:11:0/block/sdb
E: DEVNAME=/dev/sdb
E: DEVTYPE=disk
E: DISKSEQ=10
E: MAJOR=8
E: MINOR=16
E: SUBSYSTEM=block
E: USEC_INITIALIZED=5626865
E: ID_SCSI=1
E: ID_VENDOR=ATA
E: ID_VENDOR_ENC=ATA\x20\x20\x20\x20\x20
E: ID_MODEL=INTEL_SSDSC2KB01
E: ID_MODEL_ENC=INTEL\x20SSDSC2KB01
E: ID_REVISION=0110
E: ID_TYPE=disk
E: ID_SERIAL=355cd2e41511a950f
E: ID_SERIAL_SHORT=55cd2e41511a950f
E: ID_WWN=0x55cd2e41511a950f
E: ID_WWN_WITH_EXTENSION=0x55cd2e41511a950f
E: ID_SCSI_SERIAL=PHYF937400JB1P9DGN
E: ID_BUS=scsi
E: ID_PATH=pci-0000:81:00.0-scsi-0:0:11:0
E: ID_PATH_TAG=pci-0000_81_00_0-scsi-0_0_11_0
E: ID_PART_TABLE_UUID=e98ab545-f87a-7348-b6ec-a7dad7a7cacc
E: ID_PART_TABLE_TYPE=gpt
E: DEVLINKS=/dev/disk/by-path/pci-0000:81:00.0-scsi-0:0:11:0 /dev/disk/by-id/scsi-355cd2e41511a950f /dev/disk/by-id/wwn-0x55cd2e41511a950f /dev/disk/by-diskseq/10
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:


/dev/sdc

P: /devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:13/0:0:13:0/block/sdc
M: sdc
J: b8:32
U: block
T: disk
D: b 8:32
N: sdc
L: 0
S: disk/by-diskseq/11
S: disk/by-path/pci-0000:81:00.0-scsi-0:0:13:0
S: disk/by-id/wwn-0x5000c50094491cb3
S: disk/by-id/scsi-35000c50094491cb3
Q: 11
E: DEVPATH=/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:13/0:0:13:0/block/sdc
E: DEVNAME=/dev/sdc
E: DEVTYPE=disk
E: DISKSEQ=11
E: MAJOR=8
E: MINOR=32
E: SUBSYSTEM=block
E: USEC_INITIALIZED=5746215
E: ID_SCSI=1
E: ID_VENDOR=SEAGATE
E: ID_VENDOR_ENC=SEAGATE\x20
E: ID_MODEL=ST10000NM0216
E: ID_MODEL_ENC=ST10000NM0216\x20\x20\x20
E: ID_REVISION=ET02
E: ID_TYPE=disk
E: ID_SERIAL=35000c50094491cb3
E: ID_SERIAL_SHORT=5000c50094491cb3
E: ID_WWN=0x5000c50094491cb3
E: ID_WWN_WITH_EXTENSION=0x5000c50094491cb3
E: ID_SCSI_SERIAL=ZA223D9M0000C7490Y7V
E: ID_BUS=scsi
E: ID_PATH=pci-0000:81:00.0-scsi-0:0:13:0
E: ID_PATH_TAG=pci-0000_81_00_0-scsi-0_0_13_0
E: ID_PART_TABLE_UUID=d5a812c7-f2dd-3845-b828-35e14b615fc2
E: ID_PART_TABLE_TYPE=gpt
E: DEVLINKS=/dev/disk/by-diskseq/11 /dev/disk/by-path/pci-0000:81:00.0-scsi-0:0:13:0 /dev/disk/by-id/wwn-0x5000c50094491cb3 /dev/disk/by-id/scsi-35000c50094491cb3
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:


/dev/sdd

P: /devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:14/0:0:14:0/block/sdd
M: sdd
J: b8:48
U: block
T: disk
D: b 8:48
N: sdd
L: 0
S: disk/by-id/wwn-0x5000c500944fade7
S: disk/by-diskseq/12
S: disk/by-id/scsi-35000c500944fade7
S: disk/by-path/pci-0000:81:00.0-scsi-0:0:14:0
Q: 12
E: DEVPATH=/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:14/0:0:14:0/block/sdd
E: DEVNAME=/dev/sdd
E: DEVTYPE=disk
E: DISKSEQ=12
E: MAJOR=8
E: MINOR=48
E: SUBSYSTEM=block
E: USEC_INITIALIZED=5753565
E: ID_SCSI=1
E: ID_VENDOR=SEAGATE
E: ID_VENDOR_ENC=SEAGATE\x20
E: ID_MODEL=ST10000NM0216
E: ID_MODEL_ENC=ST10000NM0216\x20\x20\x20
E: ID_REVISION=ET02
E: ID_TYPE=disk
E: ID_SERIAL=35000c500944fade7
E: ID_SERIAL_SHORT=5000c500944fade7
E: ID_WWN=0x5000c500944fade7
E: ID_WWN_WITH_EXTENSION=0x5000c500944fade7
E: ID_SCSI_SERIAL=ZA21Z6Y10000C7497GQ8
E: ID_BUS=scsi
E: ID_PATH=pci-0000:81:00.0-scsi-0:0:14:0
E: ID_PATH_TAG=pci-0000_81_00_0-scsi-0_0_14_0
E: ID_PART_TABLE_UUID=f47faa97-1c8d-f741-aa76-5287548a07b5
E: ID_PART_TABLE_TYPE=gpt
E: DEVLINKS=/dev/disk/by-id/wwn-0x5000c500944fade7 /dev/disk/by-diskseq/12 /dev/disk/by-id/scsi-35000c500944fade7 /dev/disk/by-path/pci-0000:81:00.0-scsi-0:0:14:0
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:


/dev/sde

P: /devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:16/0:0:16:0/block/sde
M: sde
J: b8:64
U: block
T: disk
D: b 8:64
N: sde
L: 0
S: disk/by-id/ata-Samsung_SSD_850_PRO_512GB_S250NXAGB26365E
S: disk/by-path/pci-0000:81:00.0-scsi-0:0:16:0
S: disk/by-diskseq/13
S: disk/by-id/wwn-0x500253884011633d
Q: 13
E: DEVPATH=/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:16/0:0:16:0/block/sde
E: DEVNAME=/dev/sde
E: DEVTYPE=disk
E: DISKSEQ=13
E: MAJOR=8
E: MINOR=64
E: SUBSYSTEM=block
E: USEC_INITIALIZED=5624318
E: ID_ATA=1
E: ID_TYPE=disk
E: ID_BUS=ata
E: ID_MODEL=Samsung_SSD_850_PRO_512GB
E: ID_MODEL_ENC=Samsung\x20SSD\x20850\x20PRO\x20512GB\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_REVISION=EXM02B6Q
E: ID_SERIAL=Samsung_SSD_850_PRO_512GB_S250NXAGB26365E
E: ID_SERIAL_SHORT=S250NXAGB26365E
E: ID_ATA_WRITE_CACHE=1
E: ID_ATA_WRITE_CACHE_ENABLED=1
E: ID_ATA_READ_LOOKAHEAD=1
E: ID_ATA_READ_LOOKAHEAD_ENABLED=1
E: ID_ATA_FEATURE_SET_HPA=1
E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
E: ID_ATA_FEATURE_SET_PM=1
E: ID_ATA_FEATURE_SET_PM_ENABLED=1
E: ID_ATA_FEATURE_SET_SECURITY=1
E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=2
E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=2
E: ID_ATA_FEATURE_SET_SMART=1
E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
E: ID_ATA_DOWNLOAD_MICROCODE=1
E: ID_ATA_SATA=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
E: ID_ATA_ROTATION_RATE_RPM=0
E: ID_WWN=0x500253884011633d
E: ID_WWN_WITH_EXTENSION=0x500253884011633d
E: ID_ATA_PERIPHERAL_DEVICE_TYPE=0
E: ID_PATH=pci-0000:81:00.0-scsi-0:0:16:0
E: ID_PATH_TAG=pci-0000_81_00_0-scsi-0_0_16_0
E: ID_PART_TABLE_UUID=2811e684-304e-4dce-98b5-ff30c4f7ce0d
E: ID_PART_TABLE_TYPE=gpt
E: DEVLINKS=/dev/disk/by-id/ata-Samsung_SSD_850_PRO_512GB_S250NXAGB26365E /dev/disk/by-path/pci-0000:81:00.0-scsi-0:0:16:0 /dev/disk/by-diskseq/13 /dev/disk/by-id/wwn-0x500253884011633d
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:


/dev/sdf

P: /devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:17/0:0:17:0/block/sdf
M: sdf
J: b8:80
U: block
T: disk
D: b 8:80
N: sdf
L: 0
S: disk/by-id/scsi-355cd2e41511a95a8
S: disk/by-id/wwn-0x55cd2e41511a95a8
S: disk/by-diskseq/14
S: disk/by-path/pci-0000:81:00.0-scsi-0:0:17:0
Q: 14
E: DEVPATH=/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:17/0:0:17:0/block/sdf
E: DEVNAME=/dev/sdf
E: DEVTYPE=disk
E: DISKSEQ=14
E: MAJOR=8
E: MINOR=80
E: SUBSYSTEM=block
E: USEC_INITIALIZED=5627017
E: ID_SCSI=1
E: ID_VENDOR=ATA
E: ID_VENDOR_ENC=ATA\x20\x20\x20\x20\x20
E: ID_MODEL=INTEL_SSDSC2KB01
E: ID_MODEL_ENC=INTEL\x20SSDSC2KB01
E: ID_REVISION=0110
E: ID_TYPE=disk
E: ID_SERIAL=355cd2e41511a95a8
E: ID_SERIAL_SHORT=55cd2e41511a95a8
E: ID_WWN=0x55cd2e41511a95a8
E: ID_WWN_WITH_EXTENSION=0x55cd2e41511a95a8
E: ID_SCSI_SERIAL=PHYF937400PZ1P9DGN
E: ID_BUS=scsi
E: ID_PATH=pci-0000:81:00.0-scsi-0:0:17:0
E: ID_PATH_TAG=pci-0000_81_00_0-scsi-0_0_17_0
E: ID_PART_TABLE_UUID=3945df63-a6aa-b345-a206-e434b4835812
E: ID_PART_TABLE_TYPE=gpt
E: DEVLINKS=/dev/disk/by-id/scsi-355cd2e41511a95a8 /dev/disk/by-id/wwn-0x55cd2e41511a95a8 /dev/disk/by-diskseq/14 /dev/disk/by-path/pci-0000:81:00.0-scsi-0:0:17:0
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:
 
okay. please run "udevadm trigger --verbose -t all --settle /dev/sda" and the same for /dev/sde and post the output here. afterwards, please run "ls -lha /dev/disk/by-id/scsi-*" again and also post its output.
 
udevadm trigger --verbose -t all --settle /dev/sda
/sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:10/0:0:10:0/block/sda
/sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:10/0:0:10:0/block/sda/sda1
/sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:10/0:0:10:0/block/sda/sda2
/sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:10/0:0:10:0/block/sda/sda3
settle /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:10/0:0:10:0/block/sda
settle /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:10/0:0:10:0/block/sda/sda1
settle /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:10/0:0:10:0/block/sda/sda2
settle /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:10/0:0:10:0/block/sda/sda3

udevadm trigger --verbose -t all --settle /dev/sde
/sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:16/0:0:16:0/block/sde
/sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:16/0:0:16:0/block/sde/sde1
/sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:16/0:0:16:0/block/sde/sde2
/sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:16/0:0:16:0/block/sde/sde3
settle /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:16/0:0:16:0/block/sde
settle /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:16/0:0:16:0/block/sde/sde1
settle /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:16/0:0:16:0/block/sde/sde2
settle /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:0:16/0:0:16:0/block/sde/sde3

Aaand: no change!
ls -lha /dev/disk/by-id/scsi-*
lrwxrwxrwx 1 root root 9 Aug 11 12:33 /dev/disk/by-id/scsi-35000c50094491cb3 -> ../../sdc
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-35000c50094491cb3-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-35000c50094491cb3-part9 -> ../../sdc9
lrwxrwxrwx 1 root root 9 Aug 11 12:33 /dev/disk/by-id/scsi-35000c500944fade7 -> ../../sdd
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-35000c500944fade7-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-35000c500944fade7-part9 -> ../../sdd9
lrwxrwxrwx 1 root root 9 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a950f -> ../../sdb
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a950f-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a950f-part9 -> ../../sdb9
lrwxrwxrwx 1 root root 9 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a95a8 -> ../../sdf
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a95a8-part1 -> ../../sdf1
lrwxrwxrwx 1 root root 10 Aug 11 12:33 /dev/disk/by-id/scsi-355cd2e41511a95a8-part9 -> ../../sdf9
 
And to emphasize: just before the upgrade to 9.0 we did an upgrade to the latest 8.4, where a new kernel was also installed and configured. That went smoothly. So what has changed here ?
 
could you check the journal covering the time period of the udevadm trigger call and look for any related messages there?

the steps above work for me for recreating missing symlinks, so something else must be missing..
 
Nothing relevant, and not at the specific time where the command was run. But we can create the symlinks by hand, that is not a problem - just that afterwards grub complains that it cannot understand the filesystem there.
 
that is normal for ZFS, and shouldn't matter the specification where / is is encoded in the kernel cmdline for this reason.
 
Ah ok! I am no really familiar with grub and zfs. I now created the links by hand like so:

# ln -sf /dev/sda3 /dev/disk/by-id/scsi-35002538840116350-part3
# ln -sf /dev/sde3 /dev/disk/by-id/scsi-3500253884011633d-part3

and then called proxmox-boot-tool refresh (instead of grub-probe, which gives an error) and voilà:

proxmox-boot-tool refresh
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/B336-C9A1
Copying kernel 6.14.8-2-pve
Copying kernel 6.8.12-13-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.14.8-2-pve
Found initrd image: /boot/initrd.img-6.14.8-2-pve
Found linux image: /boot/vmlinuz-6.8.12-13-pve
Found initrd image: /boot/initrd.img-6.8.12-13-pve
Adding boot menu entry for UEFI Firmware Settings ...
done
Copying and configuring kernels on /dev/disk/by-uuid/B337-975F
Copying kernel 6.14.8-2-pve
Copying kernel 6.8.12-13-pve
Removing old version 6.8.12-11-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.14.8-2-pve
Found initrd image: /boot/initrd.img-6.14.8-2-pve
Found linux image: /boot/vmlinuz-6.8.12-13-pve
Found initrd image: /boot/initrd.img-6.8.12-13-pve
Adding boot menu entry for UEFI Firmware Settings ...
done

So the links were missing, not sure why and since when, but recreating them by hand enables us to get a server with working bootloader. As we will reinstall this thing soon after migrating machines, I will not investigate further why these links are not created automatically. Anyway it is rather a debian than a PVE issue, but I found nothing related to trixie when researching.

Thank you very much for helpnig so fast. We appreciate that very much.
 
  • Like
Reactions: fabian
Same mine. I think this is my fault that I initially created zpool using scsj- like links for ATA disks instead wwn-. I have mixed array of SATA and SAS discs and noticed that only SAS discs are linked using scsi- like names. So with Debian Trixie they made a cleanup in udev rules and wrong configurations were exposed.
 
You must log in or register to reply here.
Top Bottom