2 identical disks - only one /dev/disk/by-id ???

[rofo69]

I have 2 identical Seagate EXOs 16 drives inside an external USB JBOD enclosure.
Both work fine, but there is a quirk I'd like to see if its possible to solve.

Both disks are passed to an Ubuntu VM, I've tried via USB / Vendor ID and by port (and a mix of both), and both work that way, but either way, the 'quirk' is still there.

This is the output of LSUSB:-

Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU Tablet
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 2109:0715 VIA Labs, Inc. VL817 SATA Adaptor
Bus 003 Device 003: ID 2109:0715 VIA Labs, Inc. VL817 SATA Adaptor

You can see both disks listed at the bottom
And this is

ls /dev/disk/by-id/

ata-ST16000NM001G-2KK103_ZL2GXMAW
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part1
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part2
scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part3
usb-ST16000N_M001G-2KK103_000000123AE8-0:0
usb-ST16000N_M001G-2KK103_000000123AE8-0:0-part1
wwn-0x5000c500cba1cf40

Only one of the disks shows up:-

usb-ST16000N_M001G-2KK103_000000123AE8-0:0

This has the effect that the built in disks utility can't 'see' the second disk, and I have to use gparted (which does work).
Also, it means I have to use /dev/disk/by-path/ to get both physical disks to work with smartctl (which also works).

If I pass the disks to a windows VM and use Seatools, it also has the same problem, it can only see the first disk, despite the fact that device manager happily shows both.

Is there a way to fix this so that the ubuntu disks utility works as expected, and I can also reference the disk by'by-id' ?

 

[EricPerl]

AFAIK, the output of lsusb contains vendor ID and product ID for the purpose of identifying device types. Eg 2109 is VIA Labs.
It would make sense that identical devices show identical IDs...

What's the output of lsblk?

 

[rofo69]

lsblk shows both disks fine:-

sdb 8:16 0 14.6T 0 disk
8:17 0 14.6T 0 part /media/primary16
sdc 8:32 0 14.6T 0 disk
8:33 0 14.6T 0 part /media/backup16

And disk by path and by uuid all work fine.

Previous to the jbod enclosure, I had two identical external drives that didn't have this problem. I wondering if they were ok because they were plugged into separate usb ports, whereas the jbod enclosure sends all the disks via a single usb c connection.

 

[EricPerl]

So /dev/sdb and /dev/sdc (plain SATA, as indicated by MAJ).

My interpretation of the first output is that the enclosure is acting as a hub (device 001) with the 2 disks connected to it.

Are the mount points something of your choosing?
They imply some sort of backup or mirror, which could explain why the 2nd drive is not "usable".

What's the enclosure model?

 

[rofo69]

Both drives are perfectly usable, and I can partition them fine using gparted, and mount them via uuid in fstab with no issues.
I was merely curious why only 1 shows up in seatools for windows, and the disks utility for ubuntu.

There is no raid or mirror going on, its just a jbod with separate disks. I'm simply backing up the 'primary' disk to a 'backup' using rsync.
The mount points I chose myself are /media/primary16 and /media/backup16

The enclosure is a yottamaster fs5c3

Each disk appears separately when passing through to the VM, and empty bays don't show up in the USB menu in the Proxmox host when passing hardware through to a VM.

I'm sure its something to do with the fact that they are identical disks connected via an internal usb hub in the enclosure. Pretty sure some random googling and I found some references to that. I'll see if I can dig them out again.

 

[EricPerl]

The fact that identical devices are used can't be a complete obstacle. But it obviously prevents you from assigning them by device type.
The proxmox docs actually have an example covering exactly this scenario:
https://pve.proxmox.com/wiki/USB_Devices_in_Virtual_Machines#Assigning_Devices_to_VMs
The device is assigned by bus+port.

 

[rofo69]

So /dev/disk/by-path/ works fine and both disks show up as expected:-

lrwxrwxrwx 1 root root 9 Dec 16 15:27 pci-0000:01:01.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Dec 16 15:27 pci-0000:01:01.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Dec 16 15:27 pci-0000:01:01.0-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Dec 16 15:27 pci-0000:01:01.0-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Dec 16 15:27 pci-0000:02:1b.0-usb-0:1:1.0-scsi-0:0:0:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Dec 16 15:27 pci-0000:02:1b.0-usb-0:1:1.0-scsi-0:0:0:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 Dec 16 15:27 pci-0000:02:1b.0-usb-0:2:1.0-scsi-0:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 Dec 16 15:27 pci-0000:02:1b.0-usb-0:2:1.0-scsi-0:0:0:0-part1 -> ../../sdc1

(sdb and sdc)

And looking at the link you posted, the instructions for USB3 devices is as follows:-

qm set 804 -usb0 host=058f:6387,usb3=yes
qm set 804 -usb1 host=2-1.2.2,usb3=yes

Looks exactly like whats supported in the GUI now:-



or

 

[rofo69]

Also, when using LSUSB -t, from the host, the enclosure is correctly detected as a USB 3.1 gen 2 (10gb) speed:-

/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 10000M
|__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 2: Dev 7, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 1: Dev 4, If 0, Class=Mass Storage, Driver=usbfs, 10000M
|__ Port 2: Dev 5, If 0, Class=Mass Storage, Driver=usbfs, 10000M

But in the VM, its showing as:-

/: Bus 003.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/15p, 5000M
|__ Port 001: Dev 002, If 0, Class=Mass Storage, Driver=uas, 5000M
|__ Port 002: Dev 003, If 0, Class=Mass Storage, Driver=uas, 5000M

 

[EricPerl]

I can't tell if your initial issue is solved or not using assignment by bus+port.

Per https://pve.proxmox.com/wiki/USB_Devices_in_Virtual_Machines#Show_Device_Map_and_Types, you should use the output of 'info usbhost' from the qm monitor prompt for your VM. Note that 5000Mb/s maps to USB 3.0 in that section... I have no clue if that's a consequence of the virtualization or because the doc is outdated.

In any case, it seems the hubs can't be assigned (first line of the previous link), so whatever you're seeing from the VM is likely emulated.
lsusb -t is apparently not super reliable. That's stated about the host. I suspect it applies even more on the guest.

 

[rofo69]

The output of info usbhost on the qm monitor prompt is:-

Bus 2, Addr 5, Port 2.3.2, Speed 5000+ Mb/s
Class 00: USB device 2109:0715, VLI Product String
Bus 2, Addr 4, Port 2.3.1, Speed 5000+ Mb/s
Class 00: USB device 2109:0715, VLI Product String


Which is incorrect as far as the maximum speed for the enclosure.

As far as assigning the disk via bus+port is this not the same as using the 'use usb port' option in the gui ?

 

[EricPerl]

Using the GUI or the CLI shouldn't matter. Both are likely using the same underlying APIs.

That output looks pretty good.
At least it says 5000+. It's probably not that relevant because it exceeds the transfer rate of the drive.
You can perf test on both the host and the VMs...

Given that output, assuming use of the bus-port syntax and exclusive access, if the VMs are still not behaving as expected, I'm at a loss.
It would be even stranger if on the same VM, some components/tools worked properly while others don't.
It would imply they don't enumerate devices the same way...