UPDATE
Issue seems to be related to the 6.1 Kernel opt in and 4Kb sector sizing on the NVME. Another system with 512b sector sizing works fine on the 6.1 kernel and reverting this machine 5.15.85-1-pve causes storage to go back to working.
ORIGINAL
After PVE updates, VMs running on a specific LVM Thin Pool quit booting. At first I thought I had messed something up during the update and corrupted one of the volumes but a restore from a known good backup still won't boot and is stuck looping on the SeaBios screen.
Here is where it gets really strange. Since I could still see the volumes using lvs I tried having PVE move one of the VM disks to the default lvm thin pool setup on the PVE boot disk. VM boots fine. Tried a couple more that weren't booting and same thing, as soon as I move the disk off of this pool the VM boots fine. Next I dropped the pool and ran a wipefs on the drive, rebooted, and then recreated the pool. Same issue, VM can't boot if it's disk is on this pool. I tried creating a LXC instance and it's able to startup and use the pool just fine. Lastly I took a working Rocky 8 VM and added a brand new disk from this pool, the VM can see the disk but if I try and format it as xfs I get an error `mkfs.xfs: pwrite failed: Remote I/O error`.
At this point I don't know what's going on. The disk is 960gb Samsung PM983 with no smart errors and a wear level of 0%. PVE seems to be able to read and write to the pool since LXC works and moving a disk back and forth from the pool doens't lose any data. Qemu just can't use the pool. I've checked dmesg, /var/log/messages, syslog, kern.log, etc and there are no errors. I have another PVE box with the same updates and the same brand drive that is working just fine.
This is what the VM dumps to the console when I try and run mkfs.xfs I can format and mount the underlying lvm thin volume on PVE just fine
Issue seems to be related to the 6.1 Kernel opt in and 4Kb sector sizing on the NVME. Another system with 512b sector sizing works fine on the 6.1 kernel and reverting this machine 5.15.85-1-pve causes storage to go back to working.
ORIGINAL
After PVE updates, VMs running on a specific LVM Thin Pool quit booting. At first I thought I had messed something up during the update and corrupted one of the volumes but a restore from a known good backup still won't boot and is stuck looping on the SeaBios screen.
Here is where it gets really strange. Since I could still see the volumes using lvs I tried having PVE move one of the VM disks to the default lvm thin pool setup on the PVE boot disk. VM boots fine. Tried a couple more that weren't booting and same thing, as soon as I move the disk off of this pool the VM boots fine. Next I dropped the pool and ran a wipefs on the drive, rebooted, and then recreated the pool. Same issue, VM can't boot if it's disk is on this pool. I tried creating a LXC instance and it's able to startup and use the pool just fine. Lastly I took a working Rocky 8 VM and added a brand new disk from this pool, the VM can see the disk but if I try and format it as xfs I get an error `mkfs.xfs: pwrite failed: Remote I/O error`.
At this point I don't know what's going on. The disk is 960gb Samsung PM983 with no smart errors and a wear level of 0%. PVE seems to be able to read and write to the pool since LXC works and moving a disk back and forth from the pool doens't lose any data. Qemu just can't use the pool. I've checked dmesg, /var/log/messages, syslog, kern.log, etc and there are no errors. I have another PVE box with the same updates and the same brand drive that is working just fine.
This is what the VM dumps to the console when I try and run mkfs.xfs I can format and mount the underlying lvm thin volume on PVE just fine
Code:
[ 1368.209028] sd 0:0:0:1: [sdb] tag#237 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 1368.213642] sd 0:0:0:1: [sdb] tag#237 Sense Key : Illegal Request [current]
[ 1368.217050] sd 0:0:0:1: [sdb] tag#237 Add. Sense: Invalid field in cdb
[ 1368.220224] sd 0:0:0:1: [sdb] tag#237 CDB: Write(10) 2a 00 00 00 00 00 00 00 01 00
[ 1368.223878] blk_update_request: critical target error, dev sdb, sector 0 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
Code:
proxmox-ve: 7.3-1 (running kernel: 6.1.10-1-pve)
pve-manager: 7.3-6 (running version: 7.3-6/723bb6ec)
pve-kernel-6.1: 7.3-4
pve-kernel-helper: 7.3-4
pve-kernel-5.15: 7.3-2
pve-kernel-6.1.10-1-pve: 6.1.10-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
ceph: 17.2.5-pve1
ceph-fuse: 17.2.5-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-2
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-1
lxcfs: 5.0.3-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.5.5
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-3
pve-ha-manager: 3.5.1
pve-i18n: 2.8-2
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1
Last edited:
