RAW SCSI device (LTO tape) in qemu

[paulatz]

Hello, I have a HPE LTO tape reader+changer connected via a SAS card that I want to use from inside a qemu VM on proxmox.
After a bit of reading, I have been able to connect it like follows (I put here the full process, in case other people are interested) :
On the pve, I find out which are the "raw" scsi devices that correspond to the tape reader and tape changer:

# ls -l /dev/sg*|grep tape
crw-rw---- 1 root tape  21,  7 11 avril 14:23 /dev/sg7
crw-rw---- 1 root tape  21,  8 11 avril 14:23 /dev/sg8

In order to find which is the reader and which is the changer, I can check with dmesg, although it is not strictly necessary:

# dmesg|grep sg[78]
[101770.048401] st 1:0:2:0: Attached scsi generic sg7 type 1
[101770.721470] ch 1:0:2:1: Attached scsi generic sg8 type 8

So it is st->sg7 (the tape drive) and ch->st8 (the tape changer robot)

I can then add the device by hand in the qemu configuration file, in my case /etc/pve/nodes/pve01/qemu-server/123.conf, by adding the following line (I have the line unrolled it for readability, but it has to be put on a single line, or potentially several "args" lines?):

args: -drive id=footape,
        if=none,
        format=raw,
        readonly=off,
        file=/dev/sg7
    -device virtio-scsi-pci,
        id=scsi2 
    -device scsi-generic,
        bus=scsihw0.0,
        drive=footape
    -drive id=foochgr,
        if=none,
        format=raw,
        readonly=off,
        file=/dev/sg8
    -device virtio-scsi-pci,
        id=scsi3
    -device scsi-generic,
        bus=scsihw0.0,
        drive=foochgr

This works perfectly fine, the two devices appear inside the VM and I can use them.
I have, however, two questions:
1. Is I correct to use the same virtual bus for both? Would it make any difference to use different buses?
2. The configuration is a bit fragile, i.e. the number of the devices can change after a reboot. It usually does not, but it will if, for example, I modify the iSCSI drives which are also attached. Is there a way to make this more solid? I could make a script to modify the file during boot, or poke around with udev configuration, but they both seem dangerous to me.
Also, I use scsi virtual devices 2 and 3, because I already have 0 and 1 configured directly from proxmox interface. If I add a new drive in proxmox and forget to update the file by hand, problems could arise. Ideally, a native proxmox supports for raw scsi device could solve all these problems.

Thank you anyway, and I hope this could be useful to someone else.

 

[fabian]

you could also try passing through those two devices as scsi2/3, e.g. like this:

qm set XXX -scsi2 /dev/sg7

this will either result in scsi-hd (not what you want), or scsi-generic/scsi-block, depending on the scan logic in /usr/share/perl5/PVE/QemuServer/Drive.pm

 

[paulatz]

Thank you Fabian, if I understand correctly what you mean, neither option would work for me, and from a quick test, they do not. The point is that a LTO tape is not a block device, but a character device. I.e. if I setup a loop device, I can attach it in eitehr way:

#qm set 128 --scsi2 /dev/loop0,scsiblock=0
update VM 128: -scsi2 /dev/loop0,scsiblock=0
# qm set 128 --scsi3 /dev/loop0,scsiblock=1
update VM 128: -scsi3 /dev/loop0,scsiblock=1

But with a character device, it will not:

# qm set 128 --scsi2 /dev/sg7,scsiblock=0
update VM 128: -scsi2 /dev/sg7,scsiblock=0
volume /dev/sg7 does not exist

I.e. it does not work either with the dvd r/w, if I target the raw scsi interface (/dev/sg9) or the dvd-writer device file (/dev/sch0), although it works with the read-only interface, /dev/r0, which is a block device.

 

[fabian]

oh, that seems to be just an error from an attempt to query the size when adding a new disk.. if you want, you could try once more by putting it directly into the config to skip that check? if it works, feel free to file a bug to make that check non-fatal for pass-through disks..

 

[paulatz]

Hello, if I naively add
scsi2: /des/sg7
it fails with error

kvm: -drive file=/dev/sg7,if=none,id=drive-scsi2,format=raw,cache=none,aio=io_uring,detect-zeroes=on: Could not open '/dev/sg7': filesystem does not support O_DIRECT
TASK ERROR: start failed: QEMU exited with code 1

I think it is the missing "scsi-generic" option that is the key point. Anyway, I'll see to open a bug report after a few more tries.
thank you for your help

 

[fabian]

could you run

perl -e 'use strict; use warnings; use PVE::QemuServer::Drive; use Data::Dumper; print Dumper(PVE::QemuServeR::Drive::path_is_scsi("/dev/sg7"))';
perl -e 'use strict; use warnings; use PVE::QemuServer::Drive; use Data::Dumper; print Dumper(PVE::QemuServeR::Drive::path_is_scsi("/dev/sg8"))'

and post the output here?

 

[paulatz]

Sure, sorry for the delay, I took a day off:

# perl -e 'use strict; use warnings; use PVE::QemuServer::Drive; use Data::Dumper; print Dumper(PVE::QemuServeR::Dri
ve::path_is_scsi("/dev/sg7"))';
Undefined subroutine &PVE::QemuServeR::Drive::path_is_scsi called at -e line 1, <DATA> line 960.
#perl -e 'use strict; use warnings; use PVE::QemuServer::Drive; use Data::Dumper; print Dumper(PVE::QemuServeR::Dri
ve::path_is_scsi("/dev/sg8"))'
Undefined subroutine &PVE::QemuServeR::Drive::path_is_scsi called at -e line 1, <DATA> line 960.

I'm not sure that it went as expected. We're using proxmox 7.4-17, we're planning to upgrade to v8, but probably not before the summer break.

 

[Darkk]

I'm not sure that it went as expected. We're using proxmox 7.4-17, we're planning to upgrade to v8, but probably not before the summer break.

Yep, we're still running 7.4-17 at work in production as it's been incredibly stable. No issues. Now the dev team just released PVE 8.2.2 (congratz btw!) and was like holy cow. I'm a bit behind. I already upgraded my home lab and test PVE servers to 8.2.2 and it's been running great. Lots of improvements and bug fixes.

I too will upgrade the production servers to 8.2.x series soon to make use of these enhancements.

 

[alexskysilk]

do you have any other devices attached to that SAS hba? Might have better luck passing the whole hba to the vm.

https://pve.proxmox.com/wiki/PCI_Passthrough

 

[fabian]

ah, in that case the command is

perl -e 'use strict; use warnings; use PVE::QemuServer; use Data::Dumper; print Dumper(PVE::QemuServer::path_is_scsi("/dev/sg7"))';

the code is largely the same, just in a different module

 

[fabian]

and yes, passing through the whole HBA is also an option if it is possible, and probably gives you better performance

 

[paulatz]

ah, in that case the command is

perl -e 'use strict; use warnings; use PVE::QemuServer; use Data::Dumper; print Dumper(PVE::QemuServer::path_is_scsi("/dev/sg7"))';

the code is largely the same, just in a different module

Hello, here is the output (the device node changed in the meanwhile, following a reboot of the MSL) :

# perl -e 'use strict; use warnings; use PVE::QemuServer; use Data::Dumper; print Dumper(PVE::QemuServer::path_is_sc
si("/dev/sg17"))';
$VAR1 = {
'vendor' => 'HPE',
'revision' => '3330',
'product' => 'MSL3040',
'type' => 8,
'removable' => 1
};
root@pve01:~# perl -e 'use strict; use warnings; use PVE::QemuServer; use Data::Dumper; print Dumper(PVE::QemuServer::path_is_sc
si("/dev/sg16"))';
$VAR1 = {
'removable' => 1,
'vendor' => 'HPE',
'product' => 'Ultrium 8-SCSI',
'revision' => 'Q387',
'type' => 1
};

And yes, the biggest problem turned out to be the scsi device number changing, but I wrote a new udev rule to generate stable names for all scsi devices (instead of just tape and disk, which would not include the tape changer), and use those instead. So I have everything working, even if it is not supported in the webgui.

 

[paulatz]

do you have any other devices attached to that SAS hba? Might have better luck passing the whole hba to the vm.

https://pve.proxmox.com/wiki/PCI_Passthrough

Thank you, I though about that, but it did not work out of the box «No IOMMU detected, please activate it.». I haven't found this option in the hardware configuration (via iDRAC) but I'll do a proper restart later and check the bios.

 

[fabian]

so the tape itself (type 1) should be passed through using scsi-generic, but the library has type 8 which is not handled at all right now. feel free to add an enhancement request, but probably passing through the whole HBA makes more sense in any case.