Passing multiple DVD/Bluray writers to VM

[drbuckingham]

I have 4 internal SATA (connected via two built-in controllers) DVD/Bluray writers that I wish to use in a VM. While I can pass one physical device via "Use physical CD/DVD", it is not presented as a Bluray drive to the VM (a Bluray disc can be read in its entirety, but Bluray software reports it is not a Bluray device). Additionally, specifying 2 (or more) CD/DVD devices as "Use physical CD/DVD" only results in both pointing to the same drive and not separate ones. The net result must allow the software running in the VM to see the DVD/Bluray writers as what they are (though they don't have to appear as their physical models --- just as something that the Bluray reading/writing software recognizes as a Bluray reader/writer).

I found a reference in the past (Proxmox VE 4.3 era) that iscsi-scst/scstadmin could be used to make the DVD writers available to the VM's via iscsi targets --- but neither package is available via supported repositories now (Proxmox VE 6.3). I did locate the current version of iscsi-scst/scstadmin which appears can be built/used on Debian 10, but my question is simply --- is there a better way?

I understand I could:
- use PCIe pass-through to dedicate the two built-in SATA controllers to that VM
- purchase SATA-to-USB adapters and plug into an internal USB controller, then use PCIe pass-through to dedicate it to the VM
- purchase a separate internal 4-port SATA controller and use PCIe pass-through to dedicate that controller to the VM (most likely personal choice absent other options)

I need to keep at least one DVD/Bluray drive available to other VMs when necessary, hence why I don't want to use PCIe pass-through if I don't have to. I am new to Proxmox VE, but I do come from a background of VMWare and Hyper-V environments. I just don't have the depth of knowledge of Proxmox VE yet to be able to determine the best course of action (and searching the forums yielded nothing that applied to my situation). A supported software choice that wouldn't require dedicating the devices is my preference, but I defer to the experienced voices here.

 

[leesteken]

Looks like you did your research. In my experience, only reading (unencrypted) files from DVD/Blu-ray works with the normal QEMU device/disk passthrough, as well as Proxmox webUI. I'm also still looking for alternatives that are easier or work better, but every suggestion I've seen so far does not work for movie playback.

To use the optical device for reading movies or writing data: I have tried iSCSI from Proxmox to VMs but could not get it to work reliably, and I have used PCI passthrough of SATA controller (even via an NVME slot), which worked perfectly but only if the PCIe slot is in a isolated IOMMU group and only for a single VM. Eventually, I settled for a SATA-to-USB-controller, which I can easily hot-plug from one VM to another and works well enough for video and burning at USB2.0 speeds.

Alternatively, if the software that needs the optical devices can instead run inside a container, you can bind-mount the /dev-entries. I have used a container for this that would automatically create backups of inserted discs and put the ISO in a Proxmox storage for mounting by VMs. Note that this is not always legal in all places! And I would use the command line interface of the container to burn ISOs to discs once in a while. Maybe you could install x2go if you need a GUI for the software?

If you know you have an isolated IOMMU group for a PCIe slot and a SATA-controller that is known to work for passthrough, that might be the best approach for 4 devices. PCI passthrough is a bit hit or miss and people often only find out after buying all the hardware that it is not going to work. You could try to expose one of them via iSCSI from inside the VM to other VMs, once the passthrough is working, or create ISOs for other VMs.

 

[drbuckingham]

Looks like you did your research. In my experience, only reading (unencrypted) files from DVD/Blu-ray works with the normal QEMU device/disk passthrough, as well as Proxmox webUI. I'm also still looking for alternatives that are easier or work better, but every suggestion I've seen so far does not work for movie playback.

To use the optical device for reading movies or writing data: I have tried iSCSI from Proxmox to VMs but could not get it to work reliably, and I have used PCI passthrough of SATA controller (even via an NVME slot), which worked perfectly but only if the PCIe slot is in a isolated IOMMU group and only for a single VM. Eventually, I settled for a SATA-to-USB-controller, which I can easily hot-plug from one VM to another and works well enough for video and burning at USB2.0 speeds.

Alternatively, if the software that needs the optical devices can instead run inside a container, you can bind-mount the /dev-entries. I have used a container for this that would automatically create backups of inserted discs and put the ISO in a Proxmox storage for mounting by VMs. Note that this is not always legal in all places! And I would use the command line interface of the container to burn ISOs to discs once in a while. Maybe you could install x2go if you need a GUI for the software?

If you know you have an isolated IOMMU group for a PCIe slot and a SATA-controller that is known to work for passthrough, that might be the best approach for 4 devices. PCI passthrough is a bit hit or miss and people often only find out after buying all the hardware that it is not going to work. You could try to expose one of them via iSCSI from inside the VM to other VMs, once the passthrough is working, or create ISOs for other VMs.

Thank you very much for the detailed reply with additional options/info. I will likely try the to SATA-to-USB controller route in a few days unless someone else presents another option since I am about to attempt a GPU passthrough (AMD Firepor S7150x2) and don't wish to push my luck on enough isolated IOMMU groups.

 

[leesteken]

Thank you very much for the detailed reply with additional options/info. I will likely try the to SATA-to-USB controller route in a few days unless someone else presents another option since I am about to attempt a GPU passthrough (AMD Firepor S7150x2) and don't wish to push my luck on enough isolated IOMMU groups.

You could check (when intel or AMD iommu is enabled) if your two internal SATA controllers are in a group separate from the rest using: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
Here is a post of someone who got iSCSI working to their satisfaction. Here is a post of someone having issues with it.

 

[drbuckingham]

You could check (when intel or AMD iommu is enabled) if your two internal SATA controllers are in a group separate from the rest using: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
Here is a post of someone who got iSCSI working to their satisfaction. Here is a post of someone having issues with it.

Again, thank you very much. I failed to locate those two forum posts, which I will dig into. I will also check to see if one or the other internal SATA controllers are in an isolated group, after I get the AMD Fireport installed/working.

 

[drbuckingham]

You could check (when intel or AMD iommu is enabled) if your two internal SATA controllers are in a group separate from the rest using: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
Here is a post of someone who got iSCSI working to their satisfaction. Here is a post of someone having issues with it.

Using the first post link, I was able to get the 4 drives working correctly with DVD discs (though about 50% slower than when they operated natively under Windows 10 Pro on the same system); however, when using Bluray discs I do encounter the issue raised in the second post link. It is odd that the iSCSI targets work fine with DVD discs in them, but not with Bluray discs (and yes, they are 16x Bluray reader/burners --- two each of different manufacturers/models so it shouldn't be manufacturer/model related).

I have a single SATA-to-USB adapter arriving Friday, which I will give a try.

 

[drbuckingham]

SATA-to-USB failed to work as intended.

A single bluray device connected via SATA-to-USB works fine (reading, no burning tested), but when using all three simultaneously, two of the drives cease transferring data (first one after a minute or so, then the 2nd another minute or two later). If the software in use allows sufficient retry/timeout values, then they do continue sporadically -- but most software isn't that tolerant and I wouldn't dare try burning any DVDs/Blurays.

All three are connected via a USB3 hub into a USB3 port. When functional, the throughput is very slow (33 minutes to read 1400MB = 0.7MB/s) --- where previously (SATA connected running under Win 10) it was 17-29MB/s for each of four drives running simultaneously.

In my VM conf file, I have:
usb0: host=2-4.1.1,usb3=1
usb1: host=2-4.1.2,usb3=1
usb2: host=2-4.1.3,usb3=1

vm monitor/info usb reports:
Device 1.1, Port 1, Speed 5000 Mb/s, Product ASMT1153e, ID: usb0
Device 1.2, Port 2, Speed 5000 Mb/s, Product ASMT1153e, ID: usb1
Device 1.3, Port 3, Speed 5000 Mb/s, Product ASMT1153e, ID: usb2

I had also tried using just "usb0: host-2.4.1" in the conf file, but the VM failed to see the three drives.

I may yet have to try passing through the internal SATA controller after reinstalling the drives back into the computer case. Any suggestions in lieu of that?

 

[leesteken]

I may yet have to try passing through the internal SATA controller after reinstalling the drives back into the computer case. Any suggestions in lieu of that?

Make sure to check whether the SATA controller is in a separate IOMMU group, and then check if it works inside a VM because that is not always the case.

Regarding the USB troubles: You cannot pass a USB-hub as you noticed, only separate ports (and at most 5 via the GUI). It could be a hardware thing as (expensive) USB PCIe cards exist with a separate controller per port, which is necessary for multiple high-bandwidth streams simultaneous such as VR. Those also typically require a free x4 slot on the motherboard. I did not realize before that this might not work for more than 1 device. Have you tried spreading them over ports connected to different USB controllers? You could try PCI passthrough of a whole USB controller instead of the ports, which would remove any QEMU USB passthrough/emulation bottlenecks. Not many (affordable) USB controllers appear to passthrough/reset correctly, as post on various forums will tell you. You'll need one that is known to work and that the manufacturer has not changed the components (without notice) in the meantime. In the end, this would require the same trial and error as passing through a SATA controller...