PBS - unable to Format or Label cartridges on TL2000 robotic / scsi transport error

[carnyx.io]

Hello world,


I have a PBS server (version 3.2-8) with a Dell TL2000 (LTO5) tape system.



I try to reuse with PBS a TL2000 and cartridges which where previously used by Veeam for VMWare backup. (successfully)


Firstly, when i tried to load a cartridge, i had "load drive failed - Illegal Request, Additionnal sense: invalid element address"

So i changed the drive Number to 0 in the Tape Backup / Drive windows of the PBS GUI.

And then, i can now load cartridges on the drive


But when i try to "Format" a cartridge with ''proxmox-tape format --drive ULT3580-HH5 --fast 0'', i have this error :

unable to read media label: got tape block with unknown magic number - not written by PBS or incompatible LTO version

format anyways

<snip - wait around 10 minutes , during this time, the Drive is effectively WRITING>

TASK ERROR: erase failed - scsi command failed: transport error

Error: task failed (status erase failed - scsi command failed: transport error)




I tried this on 3 ou 4 differents cartridges.


Any idea ?


best regards

GE

 

[l.leahu-vladucu]

Hello carnyx.io! The error

unable to read media label: got tape block with unknown magic number - not written by PBS or incompatible LTO version

makes sense since the tape has not been written with PBS previously. However, the error

TASK ERROR: erase failed - scsi command failed: transport error

sounds like there is some issue with a cable or port, or a bad connection to the tape drive. Could you please try the following:

1. Update the firmware of the tape drive and changer to the latest version. These might contain fixes for your issues.
2. Update PBS to the latest version, since PBS 3.3 includes some improvements for tape drives. (Although it doesn't sound like they would fix your issue, it can be worth updating and trying anyway.)
3. Check that the cable is properly connected on both ends and there are no visible issues. If possible, try with another cable if you have one.

Let me know how this worked out! We might have to dig further to find the actual issue.

 

[carnyx.io]

Hello. Thanks for your answer

I upgraded PBS to 3.3.2. but i have the same error.

I have changed this morning the 2 SAS cable and the HBA SAS card (it's another DELL SAS 6 HBA adapter).

I just checked that I have the latest firmware installed on the HBA SAS adapter, the TL2000 changer, and the IBM drive.


I investigated further in the kernel log. I see some strange errors at boot on the ch driver (SCSI media changer)

Dec 27 12:19:05 srvpbsa kernel: scsi 0:0:0:0: Sequential-Access IBM ULT3580-HH5 H971 PQ: 0 ANSI: 6
Dec 27 12:19:05 srvpbsa kernel: scsi 0:0:0:1: Medium Changer IBM 3573-TL F.11 PQ: 0 ANSI: 5
Dec 27 12:19:05 srvpbsa kernel: scsi 0:0:1:0: Sequential-Access IBM ULT3580-HH5 H971 PQ: 0 ANSI: 6
Dec 27 12:19:05 srvpbsa kernel: scsi 0:0:1:1: Medium Changer IBM 3573-TL F.11 PQ: 0 ANSI: 5
Dec 27 12:19:05 srvpbsa kernel: scsi 0:0:0:0: Attached scsi generic sg3 type 1
Dec 27 12:19:05 srvpbsa kernel: scsi 0:0:0:1: Attached scsi generic sg4 type 8
Dec 27 12:19:05 srvpbsa kernel: scsi 0:0:1:0: Attached scsi generic sg5 type 1
Dec 27 12:19:05 srvpbsa kernel: scsi 0:0:1:1: Attached scsi generic sg6 type 8
Dec 27 12:19:05 srvpbsa kernel: st 0:0:0:0: Attached scsi tape st0
Dec 27 12:19:05 srvpbsa kernel: ch 0:0:0:1: [ch0] ID/LUN unknown
Dec 27 12:19:05 srvpbsa kernel: ch 0:0:0:1: [ch0] ID/LUN unknown
Dec 27 12:19:05 srvpbsa kernel: st 0:0:1:0: Attached scsi tape st1
Dec 27 12:19:05 srvpbsa kernel: ch 0:0:1:1: [ch1] ID/LUN unknown
Dec 27 12:19:05 srvpbsa kernel: ch 0:0:1:1: [ch1] ID/LUN unknown

is it normal or unimportant ?

I also have some questions about the Dell SAS 6 HBA adapter which is an old model card, but it works well in Proxmox servers to manage MD3200 storage bays, so I tell myself that it must be OK for a TL2000 changer (especially since all the same hardware worked well before under Windows with VEEAM for VMWare backups)

I'm looking to remove part of our VMWare hypervisors in favor of Proxmox, but we need to have the same functional level, and it's not so simple... :-(


best regards
GE

 

[l.leahu-vladucu]

It seems that a LUN (Logical Unit Number) is not recognized correctly. You can call lsscsi (or install it using apt install lsscsi if not done already) to see whether all drives and changers that you need are recognized correctly. You can also use the Proxmox tools to list the available changers, as explained here. However, from your screenshots I can see that Linux and the Proxmox Backup Server have already detected it.

The Proxmox Backup Server uses the SCSI Medium Changer protocol, so I expect it to work without issues. Can you please post the output of cat /proc/scsi/scsi?

 

[l.leahu-vladucu]

You might also want to check whether there are any Linux drivers available from the vendor, as there might be a driver issue causing the errors you are seeing.

 

[carnyx.io]

Hello again,

As you probably know Dell only supports paid Linux distributions like RedHat or Suse. Nothing for Proxmox or even Debian. So, I will look but I don't think there's much to find on that side.

Concerning LUNs, indeed it's a concept that is familiar to me, but on storage bays. I must admit that I don't see what it corresponds on an LTO changer.
I looked in the SAS card settings (at boot time), but there is no modifiable parameter to define an ID or a LUN (cf attached photo)
root@srvpbsa:~# lsscsi
[0:0:0:0] tape IBM ULT3580-HH5 H971 /dev/st0
[0:0:0:1] mediumx IBM 3573-TL F.11 /dev/sch0
[0:0:1:0] tape IBM ULT3580-HH5 H971 /dev/st1
[0:0:1:1] mediumx IBM 3573-TL F.11 /dev/sch1
[1:0:0:0] disk ATA Samsung SSD 860 3B6Q /dev/sda
[2:0:0:0] disk ATA ST12000NM005G-2M EAL6 /dev/sdb
[3:0:0:0] disk ATA ST12000NM005G-2M EAL6 /dev/sdc

root@srvpbsa:~# cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: Samsung SSD 860 Rev: 3B6Q
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: ST12000NM005G-2M Rev: EAL6
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: ST12000NM005G-2M Rev: EAL6
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: ULT3580-HH5 Rev: H971
Type: Sequential-Access ANSI SCSI revision: 06
Host: scsi0 Channel: 00 Id: 00 Lun: 01
Vendor: IBM Model: 3573-TL Rev: F.11
Type: Medium Changer ANSI SCSI revision: 05
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: ULT3580-HH5 Rev: H971
Type: Sequential-Access ANSI SCSI revision: 06
Host: scsi0 Channel: 00 Id: 01 Lun: 01
Vendor: IBM Model: 3573-TL Rev: F.11
Type: Medium Changer ANSI SCSI revision: 05
root@srvpbsa:~#


root@srvpbsa:~# proxmox-tape changer scan
┌───────────────────────────────────────┬────────┬─────────┬──────────────────┐
│ path │ vendor │ model │ serial │
╞═══════════════════════════════════════╪════════╪═════════╪══════════════════╡
│ /dev/tape/by-id/scsi-00X2U78CG354_LL0 │ IBM │ 3573-TL │ 00X2U78CG354_LL0 │
├───────────────────────────────────────┼────────┼─────────┼──────────────────┤
│ /dev/tape/by-id/scsi-00X2U78CG354_LL0 │ IBM │ 3573-TL │ 00X2U78CG354_LL0 │
└───────────────────────────────────────┴────────┴─────────┴──────────────────┘

root@srvpbsa:~# proxmox-tape changer list
┌────────────┬──────────────────────────────────── ───┬─────── ───┐
│ name │ path │ vendor │ model │ serial │
╞════════════╪══════════════════════════════════════ ═╪════════╪═ ══╡
│ TL2000Brio │ /dev/tape/by-id/scsi-00X2U78CG354_LL0 │ IBM │ 3573-TL│ 00X2U78CG354_LL0 │
└────────────┴────────────────────────────────────── ─┴────────┴───┘


root@srvpbsa:~# proxmox-tape drive list
┌─────────────┬────────────────────────────────────┬────────────┬────────┬─────────────┬────────────┐
│ name │ path │ changer │ vendor │ model │ serial │
╞═════════════╪════════════════════════════════════╪════════════╪════════╪═════════════╪════════════╡
│ ULT3580-HH5 │ /dev/tape/by-id/scsi-1068097015-sg │ TL2000Brio │ IBM │ ULT3580-HH5 │ 1068097015 │
└─────────────┴────────────────────────────────────┴────────────┴────────┴─────────────┴────────────┘

Please note that I only have one drive in my TL2000 changer in slot 0. The second slot (1) for a second drive is empty.

By default, PBS assigns the drive to the number "1". I changed it in PBS GUI to "0" for it to work in the GUI (cf photo)


Another thing that may be important is that my HBA card is connected to my TL2000 via 2 SAS cables.

Maybe it would be interesting if I tested with a single SAS cable... because the proxmox-tape changer scan show my TL2000 twice with the same paths...

 

[carnyx.io]

same error with only one SAS cable ... GRRRRR