SSD Passthrough - Vendor details not available in guest

[telesteven]

Hi
I want to use TrueNAS as a guest and used this guide (https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)) to add the disk I want to use (NVMe SSD drives) to the VM.
When configuring TrueNAS it complained that the disk serial numbers are not available and therefore using zfs can be dangerous.
I checked using smartctl and also only see basic info which cannot be used to securely identify the disks:

Venfdor: QEMU
Product: QEMU HARDDISK
Revision: 2.5+
Compliance: SPC-3
User Capacity: 7,681,501,126,656 bytes
Device type: disk
SMART support is: Unavailable

Is there a way to passthrough hardware information (like serial) to the guest?

 

[Dunuin]

Using the disk passthrough your VM only sees a virtual disk. So stuff like SMART won't be available inside the guest as the VM doesn't got access to the real physical hardware. If you want direct access to the physical disk you should use PCI passthrough. But for that to work your NVMe SSDs need to be in a IOMMU group without other devices sharing the same IOMMU group. This is usually only the case when the M.2 slots are directly connected to the CPU and not to the chipset.

 

[leesteken]

I was writing what @Dunuin just wrote...
If you are using only one drive then I would think that identification of the drive is not a problem? Can you put it in a fixed virtual SCSI slot (in the VM configuration) and use something like /dev/disk/by-path/, which is unchanging on Linux, but I don't know that TrueNAS uses.

 

[telesteven]

@Dunuin:
Unfortunately PCI passthrough is limited to 16 devices and I have 20 SSDs so this is not an option
(by the way - is there a way to change this limitation? I can add more devices via qm set but they are not available in the VM and not visible in the UI

@leesteken:
in theory this should work but I have a bad feeling about this relying on a production system.

 

[dcsapak]

When configuring TrueNAS it complained that the disk serial numbers are not available and therefore using zfs can be dangerous.

if it just need the serial numbers, you should be able to set the serial number on the virtual disk e.g.

qm set ID -scsi0 <some disk>,serial=myserial01

 

[telesteven]

Indeed, this works fine. Thanks!
I still would have a better feeling with a complete passthrough of the devices but I think I can work with this solution.

Fun fact: ESXi also as a 16 PCI device for guest limitation. Still not sure why this limit exists...

 

[dcsapak]

Fun fact: ESXi also as a 16 PCI device for guest limitation. Still not sure why this limit exists...

in pve this exists because we manually assign hostpciX entries to pci addresses in the guest, and there has to be some limit. in the past this limit was 5 until some users requested more, then we increased it to 16
you can create a feature request to increase it even more: https://bugzilla.proxmox.com

you could also try to 'cheat' a bit and use our syntax to pass it through as multifunction device:

hostpci0: 00:01.0;00:02.0;00:03.0;...

all devices in such an entry will be passed through as functions (up to 8 per device)

eg as 01:00.0, 01:00.1, 01:00.2 etc.

i cannot guarantee that this will work properly or if the drivers/software in the guest will do the correct thing then though

 

[telesteven]

in pve this exists because we manually assign hostpciX entries to pci addresses in the guest, and there has to be some limit. in the past this limit was 5 until some users requested more, then we increased it to 16
you can create a feature request to increase it even more: https://bugzilla.proxmox.com

I personally have a much better gut feeling when using pci-passthrough instead of disk-passthrough (not sure why) and as more an more servers out there will be equipped with more and more U.2 drives (we already have 48 U.2 servers running), I assume that this limit of 16 will not be enough at some point. But I couldn't tell a number that would...

you could also try to 'cheat' a bit and use our syntax to pass it through as multifunction device:

hostpci0: 00:01.0;00:02.0;00:03.0;...

all devices in such an entry will be passed through as functions (up to 8 per device)

eg as 01:00.0, 01:00.1, 01:00.2 etc.

i cannot guarantee that this will work properly or if the drivers/software in the guest will do the correct thing then though

Seems to work fine. Will do some checks if everything works like expected but it looks like this is the solution I'll go with

 

[left4pve]

@telesteven Thanks for giving it a try. I'm thinking about passing through a PCIe to 4xSATA adapter to the TrueNAS VM so that it can manage the disk directly. But the backup plan is to just use disk passthrough. Have you seen any other issue with disk passthrough?

 

[telesteven]

I think the device that I passed to the VM was not available any more on the host so I wasn't able to check it via SMART (guest also could not use SMART). Not 100% sure, you should check this

 

[stefgrifon]

qm set ID -scsi0 <some disk>,serial=myserial01

Hello i have the same problem.
Can you give an example of this command?

 

[telesteven]

i cannot guarantee that this will work properly or if the drivers/software in the guest will do the correct thing then though

Indeed it looks like it does not work correctly.
Using

hostpci0: 0000:01:00.0;0000:02:00.0;0000:03:00.0;0000:04:00.0
hostpci1: 0000:41:00.0;0000:42:00.0;0000:43:00.0;0000:44:00.0
hostpci2: 0000:81:00.0;0000:82:00.0;0000:83:00.0;0000:84:00.0
hostpci3: 0000:85:00.0;0000:86:00.0;0000:87:00.0;0000:88:00.0
hostpci4: 0000:c1:00.0;0000:c2:00.0;0000:c3:00.0;0000:c4:00.0

I can see massive performance problems with the disks. We're using a raidz2 with 2 VDEVs in the guest and can see a maximum throughput of around 100 MB/s

 

[dcsapak]

I can see massive performance problems with the disks. We're using a raidz2 with 2 VDEVs in the guest and can see a maximum throughput of around 100 MB/s

i mean that's not necessarily a problem with the passthrough though
how do you measure the throughput? what model are the disks?

 

[NOiSEA]

if it just need the serial numbers, you should be able to set the serial number on the virtual disk e.g.

qm set ID -scsi0 <some disk>,serial=myserial01

do you enter that in the proxmox shell or the truenas shell. Sorry if that is a dumb question but I really don't know.

 

[Dunuin]

do you enter that in the proxmox shell or the truenas shell. Sorry if that is a dumb question but I really don't know.

PVE shell. But you can't just copy&paste that command. You will have to understand it in order to modify it to match your VM and virtual disks. See the manual for "qm set" https://pve.proxmox.com/pve-docs/qm.1.html