Bizarre disk pass-through unavailability, `Could not open '/dev/disk/by-id/nvme-(...)': No such file or directory

B

branpurn

Member
Feb 22, 2023
7
0
6
PVE 9.0.3, lab environment

Did not have this issue using PVE in a past install with same hardware. I am doing a standard disk pass-through (see commands used below). Sometimes the Windows guest will boot, oftentimes not. KVM says the disk is not available and when you check `lsblk`, sure enough, it has disappeared. It shows back up after a reboot. No SMART errors or reason to believe the hardware is defective. Works fine baremetal booting the Windows guest's drive. Sometimes works in PVE too.

Summary:
- Use lsblk to check drive ID; it's shown (twice, I try both intermittently no difference in behavior between the two)
- Add pass-through disk to VM and set it as the only boot option; power on VM
- Sometimes it will boot without issue ...
- Other times, KVM errors out "no such file or directory" ...
- When the latter, returning to lsblk, the drive is no longer shown; it is shown after a reboot.

Relevant Output:
Markdown (GitHub flavored): 
```
root@proxmox:~# lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS DEVICE-ID(S)
sda                            8:0    0  14.6T  0 disk   /dev/disk/by-id/wwn-0x5000c500c70d54c1 /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL28H4M0
sdb                            8:16   0  14.6T  0 disk   /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2NTFD9 /dev/disk/by-id/wwn-0x5000c500e3e26bf9
sdc                            8:32   0  14.6T  0 disk   /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2NNA2K /dev/disk/by-id/wwn-0x5000c500e3b6954d
zd0                          230:0    0   512G  0 disk
zd16                         230:16   0   2.1T  0 disk
nvme0n1                      259:0    0   1.8T  0 disk   /dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04_1 /dev/disk/by-id/nvme-eui.000000000000000100a075223afe5d04 /dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04
nvme1n1                      259:1    0 931.5G  0 disk   /dev/disk/by-id/nvme-eui.00a075014fcad3a3 /dev/disk/by-id/nvme-CT1000P310SSD8_25184FCAD3A3_1 /dev/disk/by-id/nvme-CT1000P310SSD8_25184FCAD3A3
root@proxmox:~#
```
```
update VM 100: -sata1 /dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04
```
```
kvm: -blockdev {"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"host_device","filename":"/dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04","node-name":"ede067feaff0fb3e3ca0399d663bc44","read-only":false},"node-name":"fde067feaff0fb3e3ca0399d663bc44","read-only":false},"node-name":"drive-sata1","read-only":false,"throttle-group":"throttle-drive-sata1"}: Could not open '/dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04': No such file or directory
TASK ERROR: start failed: QEMU exited with code 1
```
```
root@proxmox:~# lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS DEVICE-ID(S)
sda                            8:0    0  14.6T  0 disk   /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL28H4M0 /dev/disk/by-id/wwn-0x5000c500c70d54c1
sdb                            8:16   0  14.6T  0 disk   /dev/disk/by-id/wwn-0x5000c500e3e26bf9 /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2NTFD9
sdc                            8:32   0  14.6T  0 disk   /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2NNA2K /dev/disk/by-id/wwn-0x5000c500e3b6954d
zd0                          230:0    0   512G  0 disk
zd16                         230:16   0   2.1T  0 disk
nvme1n1                      259:0    0 931.5G  0 disk   /dev/disk/by-id/nvme-CT1000P310SSD8_25184FCAD3A3_1 /dev/disk/by-id/nvme-eui.00a075014fcad3a3 /dev/disk/by-id/nvme-CT1000P310SSD8_25184FCAD3A3
root@proxmox:~#
```
```
reboot
```
```
root@proxmox:~# lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS DEVICE-ID(S)
sda                            8:0    0  14.6T  0 disk   /dev/disk/by-id/wwn-0x5000c500c70d54c1 /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL28H4M0
sdb                            8:16   0  14.6T  0 disk   /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2NTFD9 /dev/disk/by-id/wwn-0x5000c500e3e26bf9
sdc                            8:32   0  14.6T  0 disk   /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2NNA2K /dev/disk/by-id/wwn-0x5000c500e3b6954d
zd0                          230:0    0   512G  0 disk
zd16                         230:16   0   2.1T  0 disk
nvme0n1                      259:0    0   1.8T  0 disk   /dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04_1 /dev/disk/by-id/nvme-eui.000000000000000100a075223afe5d04 /dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04
nvme1n1                      259:1    0 931.5G  0 disk   /dev/disk/by-id/nvme-eui.00a075014fcad3a3 /dev/disk/by-id/nvme-CT1000P310SSD8_25184FCAD3A3_1 /dev/disk/by-id/nvme-CT1000P310SSD8_25184FCAD3A3
root@proxmox:~#
```
```
update VM 100: -sata1 /dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04_1
```
```
kvm: -blockdev {"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"host_device","filename":"/dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04_1","node-name":"e36e6ba4c7a5260fcd7be4a2ad503cd","read-only":false},"node-name":"f36e6ba4c7a5260fcd7be4a2ad503cd","read-only":false},"node-name":"drive-sata1","read-only":false,"throttle-group":"throttle-drive-sata1"}: Could not open '/dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04_1': No such file or directory
TASK ERROR: start failed: QEMU exited with code 1
```
```
root@proxmox:~# lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS DEVICE-ID(S)
sda                            8:0    0  14.6T  0 disk   /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL28H4M0 /dev/disk/by-id/wwn-0x5000c500c70d54c1
sdb                            8:16   0  14.6T  0 disk   /dev/disk/by-id/wwn-0x5000c500e3e26bf9 /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2NTFD9
sdc                            8:32   0  14.6T  0 disk   /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2NNA2K /dev/disk/by-id/wwn-0x5000c500e3b6954d
zd0                          230:0    0   512G  0 disk
zd16                         230:16   0   2.1T  0 disk
nvme1n1                      259:0    0 931.5G  0 disk   /dev/disk/by-id/nvme-CT1000P310SSD8_25184FCAD3A3_1 /dev/disk/by-id/nvme-eui.00a075014fcad3a3 /dev/disk/by-id/nvme-CT1000P310SSD8_25184FCAD3A3
root@proxmox:~#
```

VM conf below if you want it; I have CPU on EPYC instead of "host" because "host" breaks GPU passthrough, HAHA (Code 53)-- that's another bizarre (not experienced in past install) matter for another thread, another time.
Markdown (GitHub flavored): 
```
root@proxmox:/etc/pve/qemu-server# cat 100.conf
agent: 1
bios: ovmf
boot: order=sata1
cores: 16
cpu: EPYC
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00,pcie=1,rombar=0,x-vga=1
hostpci1: 0000:33:00,pcie=1,rombar=0
machine: pc-q35-10.0
memory: 65536
meta: creation-qemu=10.0.2,ctime=1758829967
name: workstation
net0: virtio=BC:24:11:A1:8F:26,bridge=vmbr1,firewall=1
net1: e1000=BC:24:11:2C:21:41,bridge=vmbr0,firewall=1,link_down=1
numa: 0
ostype: win11
sata1: /dev/disk/by-id/nvme-CT2000P5PSSD8_22353AFE5D04_1,size=1953514584K
scsihw: virtio-scsi-single
smbios1: uuid=06499af2-f2d5-441c-ae58-2bdf490feb4a
sockets: 1
usb0: host=3384:0001,usb3=1
usb1: host=30fa:0300,usb3=1
usb2: host=1050:0407,usb3=1
usb3: host=0bda:4b8c,usb3=1
vga: none
virtio0: zfs_pool:vm-100-disk-0,iothread=1,size=2176G
vmgenid: aa3749b0-dd0b-4a34-852b-b8e4ec112c1e
root@proxmox:/etc/pve/qemu-server#
```
 
Last edited:
I want to stress sometimes it "just works." It's bizarre. It always boots if baremetal, there's no intermittency there. Again no drive health red flags or strange behavior outside of this.
 
You must log in or register to reply here.
Top Bottom