file system issue with block device passthrough

[Azapa]

I just noticed something on my homelab server the other day, and while i am sure it is an issue, i don't know how much of an issue it might be.

i have an old lenovo minipci running proxmox ve 8 (pve-manager/8.4.19/a68fb383814bb1e6 (running kernel: 6.8.12-23-pve )to be exact)

it has an old usb external drive doing passthrough to one of the VM's and i noticed something weird the other day.

on the server itself, if i look at block devices (lsblk -f) i get this - sdc being the passthrough device:

sdc
└─sdc1 btrfs cb30a...

but if i log in to the vm itself and look at the device with the same command i get this:

sdc
└─sdc1 ext4 1.0 67ad....

i.e. the device has 2 issues:

1. why does the bare metal detect the drive as being BTRFS and the vm using it on passthrough detect it as being EXT4?
2. why does it have a different UUID?

i verified the fstab and the file system is indeed being mounted as ext4. i set this thing up with PVE 6 several years ago and don't remember exactly how i formatted everything... it was my first foray into proxmox and i followed a guide in the forum on setting up passthrough. i cant remember how i initially formatted the drive, or if i reformatted it when it went i got the pass through to work.

im not too concerned about the UUID thing as much as i am concerned about the file system thing. i am currently backing up the drive leaving it in place on the VM as to do my best to keep whatever is on it from getting corrupted in any way, then i plan on reformatting the drive so the file system matches in and out of the virtual environment.

am i right to be concerned here? maybe i dont understand the passthrough process as much as i thought but i am very interested in how this happened in the first place, tho i doubt i will ever figure that out.

 

[Neobin]

What does the VM config (qm config YourVMID) look like?

 

[fiona]

Hi,
are you sure this is the same device? Just because it has the same /dev/sdX letter, that doesn't mean anything if it's not the same machine.

 

[Azapa]

What does the VM config (qm config YourVMID) look like?

bios: ovmf
boot: order=scsi0;ide2;net0
cores: 2
efidisk0: local-lvm:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: local:iso/debian-11.2.0-amd64-netinst.iso,media=cdrom
memory: 8192
meta: creation-qemu=6.1.0,ctime=1643079731
name: grape
net0: virtio=FE:B5:F0:1F:B9:26,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=15G
scsi2: /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0,backup=0,size=5723134M
scsihw: virtio-scsi-pci
smbios1: uuid=40dd359c-eaf3-415a-ba51-eb54bfdaa03b
sockets: 2
usb0: host=0080:a001
vcpus: 4
vga: virtio
vmgenid: 26ce93bc-1cff-4f8a-84c0-c1606f4946ab

 

[Azapa]

Hi,
are you sure this is the same device? Just because it has the same /dev/sdX letter, that doesn't mean anything if it's not the same machine.

there is only one external hard drive attached to the machine, so i dont know how it could be anything else.

i think the drive was originally a WD MYBook or something like that... the controller failed inside the original housing and i yanked the drive and put it into a different external enclosure. i had do update the UUID in the config file so it would keep passing though the drive, but it was pretty seamless from what i remember. that was about 2 years ago tho, so i can't remember if i needed to do anything else at the time.

 

[RolandK]

what does lsblk -f /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0 tell ?

and lsblk --version on host and inside vm ?

also print fdisk -l for usb drive and for mapped drive inside vm

 

[Azapa]

what does lsblk -f /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0 tell ?

on the server:
:~# lsblk -f /dev/disk/by-id/usb-WD_easystore_266A_575838324431324148413836-0\:0
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
sdc
└─sdc1 btrfs cb30a848-1882-4f7f-aae1-1533f52d8783

on the vm:

:~$ sudo lsblk -f /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0\:0
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
sdc
└─sdc1 ext4 1.0 67ad2ce6-f300-4146-8c32-42e7dfb2fe5b

and lsblk --version on host and inside vm ?

on the server:
:~# lsblk --version
lsblk from util-linux 2.38.1

on the vm:
:~$ lsblk --version
lsblk from util-linux 2.36.1

also print fdisk -l for usb drive and for mapped drive inside vm

in the vm:
:~$ lsblk -l
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 5.5T 0 disk
sdb 8:16 0 15G 0 disk
sdb1 8:17 0 512M 0 part /boot/efi
sdb2 8:18 0 13.5G 0 part /
sdb3 8:19 0 976M 0 part [SWAP]
sdc 8:32 0 5.5T 0 disk
sdc1 8:33 0 5.5T 0 part /mnt/external
sr0 11:0 1 378M 0 rom

and on the server:
:~# lsblk -l
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 223.6G 0 disk
sda1 8:1 0 1007K 0 part
sda2 8:2 0 512M 0 part /boot/efi
sda3 8:3 0 223.1G 0 part
sdc 8:32 0 4.5T 0 disk
sdc1 8:33 0 4.5T 0 part
pve-swap 252:0 0 7G 0 lvm [SWAP]
pve-root 252:1 0 55.8G 0 lvm /
pve-data_tmeta 252:2 0 1.4G 0 lvm
pve-data_tdata 252:3 0 141.4G 0 lvm
pve-data-tpool 252:4 0 141.4G 0 lvm
pve-data 252:5 0 141.4G 1 lvm
pve-vm--100--disk--0 252:6 0 15G 0 lvm
pve-vm--100--disk--1 252:7 0 4M 0 lvm
pve-vm--101--disk--0 252:8 0 15G 0 lvm

edit: i think i fixed where i clobbered the formatting.

and in case anyone is interested this is the 100. conf file used to setup the vm

:/etc/pve/qemu-server# cat 100.conf
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 2
efidisk0: local-lvm:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: none,media=cdrom
memory: 8192
meta: creation-qemu=6.1.0,ctime=1643079731
name: grape
net0: virtio=FE:B5:F0:1F:B9:26,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=15G
scsi2: /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0,backup=0,size=5723134M
scsihw: virtio-scsi-pci
smbios1: uuid=40dd359c-eaf3-415a-ba51-eb54bfdaa03b
sockets: 2
usb0: host=0080:a001
vcpus: 4
vga: virtio
vmgenid:

and one more: my storage.cfg. this might be the issue...

/etc/pve# cat storage.cfg
dir: local
path /var/lib/vz
content iso,backup,vztmpl

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

ext4: fef180bd-9990-46ae-bcda-636e47d69bbb
path /mnt/sdb
content images,rootdir
prune-backups keep-all=1

i dont know how safe or not it would be to change the ext4:{uuid} to btrfs: {uuid} without fear of corruping the data.

 

[RolandK]

i think there must be some misunderstanding here or you mix something up.

please post

fdisk (fdisk, not lsblk) -l from

/dev/disk/by-id/usb-WD_easystore_266A_575838324431324148413836-0:0

and

/dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0

and

/dev/sdb

on the HOST, as all devices/paths seem to exist there.

are you really show that /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0 is a path visible inside VM?
that device should be /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsiX inside VM.

also show output of mount command on the host.

mind that device names can change , especially with external attached usb. if disk gets hiccup, it may disappear as "sdbX" and immediatly shows up as "sdbY". that's why we better use static devices names with uuid or something else.

it's weird that lsblk does not show /dev/sdb, which is mounted as datastore

 

[Azapa]

i think there must be some misunderstanding here or you mix something up.

please post

fdisk (fdisk, not lsblk) -l from

/dev/disk/by-id/usb-WD_easystore_266A_575838324431324148413836-0:0

and

/dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0

and

/dev/sdb

on the HOST, as all devices/paths seem to exist there.

this is all from the host:
:~# fdisk -l /dev/disk/by-id/usb-WD_easystore_266A_575838324431324148413836-0:0
Disk /dev/disk/by-id/usb-WD_easystore_266A_575838324431324148413836-0:0: 4.55 TiB, 5000947302400 bytes, 9767475200 sectors
Disk model: easystore 266A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 13AC310F-3B8C-498B-9288-44300AB07204

Device Start End Sectors Size Type
/dev/disk/by-id/usb-WD_easystore_266A_575838324431324148413836-0:0-part1 2048 9767473151 9767471104 4.5T Linux filesys

:~# fdisk -l /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0
fdisk: cannot open /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0: No such file or directory

this only exists inside the VM. i think it has to do with how i initially setup the passthrough.

:~# fdisk -l /dev/sdb
fdisk: cannot open /dev/sdb: No such file or directory

i think the device is actually /dev/sdc tho.

:~# fdisk -l /dev/sdc
Disk /dev/sdc: 4.55 TiB, 5000947302400 bytes, 9767475200 sectors
Disk model: easystore 266A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 13AC310F-3B8C-498B-9288-44300AB07204

Device Start End Sectors Size Type
/dev/sdc1 2048 9767473151 9767471104 4.5T Linux filesystem

i think the sdb/sdc issue occured when the easystore enclosure died. the usb <---> sata controller went bad. i shucked the drive and put it in a different external drive enclosure that i had. i know when i swapped the enclosures i had to manually edit one of the config files to change the uuid of the disk it was sharing.

are you really show that /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0 is a path visible inside VM?
that device should be /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsiX inside VM.

/dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0 is how the drive shows up in the VM. maybe i did USB passthrough instead of storage passthrough? i don't see it in any of the config files tho.

:~$ ls /dev/disk/by-id/
ata-QEMU_DVD-ROM_QM00003 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part3
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part1 usb-TO_Exter_nal_USB_3.0_2015033100081-0:0
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part2 usb-TO_Exter_nal_USB_3.0_2015033100081-0:0-part1

~$ sudo fdisk -l /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0\:0
Disk /dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0: 5.46 TiB, 6001175126016 bytes, 11721045168 sectors
Disk model: nal USB 3.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 0DD7C03B-951C-405B-AA7A-B6392C623CAB

Device Start End Sectors Size Type
/dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0-part1 2048 11720978398 11720976351 5.5T Linux filesystem

also show output of mount command on the host.

mind that device names can change , especially with external attached usb. if disk gets hiccup, it may disappear as "sdbX" and immediatly shows up as "sdbY". that's why we better use static devices names with uuid or something else.

it's weird that lsblk does not show /dev/sdb, which is mounted as datastore

i am not mounting the storage drive on the host at all (sorry for the clobbered formatting... i did what i could)

:~# mount -vl
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=3977132k,nr_inodes=994283,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=802188k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5253)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=802184k,nr_inodes=200546,mode=700,inode64)

and here is the output from mount from inisde the vm

:~$ mount -vl
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=4045912k,nr_inodes=1011478,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=812872k,mode=755)
/dev/sdb2 on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12369)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sdb1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
/dev/sdc1 on /mnt/external type ext4 (rw,relatime,errors=remount-ro)
/dev/sdc1 on /home/gimp/torrents/deluge type ext4 (rw,relatime,errors=remount-ro)
/dev/sdc1 on /home/gimp/torrents/rtorrent type ext4 (rw,relatime,errors=remount-ro)
/dev/sdc1 on /var/lib/navidrome type ext4 (rw,relatime,errors=remount-ro)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
192.168.0.140:/mnt/music on /mnt/nfs/music type nfs4 (rw,noatime,nodiratime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.124,fsc,local_lock=none,addr=192.168.0.140)
tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=812868k,nr_inodes=203217,mode=700,uid=1001,gid=1001)

 

[RolandK]

>/dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0 is how the drive shows up in the VM. maybe i did USB passthrough instead of storage passthrough?

yes

>i don't see it in any of the config files tho.

most likely this entry:

usb0: host=0080:a001


but how can you run a vm with a scsi passtrough disk attached which doesn't exist ?

you probably did not reboot the VM since the old usb enclosure died ? you should get error on cold reboot of vm (i.e. power off - power on)

you will need to either recreate the scsi2 device passtrough mapping and remove usb passtrough or remove usb passtrough and create new mapping

the usb disk and sdc is the same device and you cannot use it on host and inside vm at the same time, when using a normal filesystem on it.

you should better not use devices without partition , as you did with /dev/sdb & ext4

 

[Azapa]

>/dev/disk/by-id/usb-TO_Exter_nal_USB_3.0_2015033100081-0:0 is how the drive shows up in the VM. maybe i did USB passthrough instead of storage passthrough?

yes

>i don't see it in any of the config files tho.

most likely this entry:

usb0: host=0080:a001


but how can you run a vm with a scsi passtrough disk attached which doesn't exist ?

you probably did not reboot the VM since the old usb enclosure died ? you should get error on cold reboot of vm (i.e. power off - power on)

you will need to either recreate the scsi2 device passtrough mapping and remove usb passtrough or remove usb passtrough and create new mapping

the usb disk and sdc is the same device and you cannot use it on host and inside vm at the same time, when using a normal filesystem on it.

you should better not use devices without partition , as you did with /dev/sdb & ext4

yeah... this has been rebooted many times. i think the enclosure died in the end of 2024. so i don't quite know how it is still working.

but i am 99% sure the file system issue existed since before i had the hardware issue. when i swapped the enclosure i had to update *something*, and it had to be that USB line. i remember when i fixed it. i guess i was smart enough to backup the original configuration, and here is the diff output:

:/etc/pve/qemu-server# diff 100.bak 100.conf
5c5
< ide2: local:iso/debian-11.2.0-amd64-netinst.iso,media=cdrom
---
> ide2: none,media=cdrom
18,19c18
< unused0: fef180bd-9990-46ae-bcda-636e47d69bbb:100/vm-100-disk-0.raw
< usb0: host=1949:03f8
---
> usb0: host=0080:a001

edit: here is the date on the backup file, so it would also be the date i did all this
-rw-r----- 1 root www-data 705 2024-05-30 22:27:15.000000000 -0400 100.bak


i just don't get how it shows different file systems on the host and on the vm.... if the *whole device* is being passed through i would think that there is no virtual mapping or whatever to rewrite one file system to another. that is the most confusing thing to me.

i think my best course of action would be to backup the data from the drive, remove all mappings to the VM, reformat it correctly, and pass it though as a storage device instead of usb passthorugh, i.e. actually do it correctly. i was hoping for an easier fix or at least an explanation of why it is working like this... but i fear there probably isn't an easy fix, and the explanation has to be something inside KVM or qeum.

 

[RolandK]

maybe we can see more with

lsusb

and

lsblk -o NAME,TRAN,VENDOR,MODEL,SERIAL,UUID


for me it looks that you have a 4,5TB and 5,5TB disk attached