proxmox-tape barcode-label got timeout

G

gothbert

Well-Known Member
Apr 3, 2021
30
5
48
46
I run PBS 4.2.6. A Quantum Superloader 3 is connected via iSCSI. The iSCSI target is a Debian 13.7. PBS web interface shows the library slots and drives. I have a LTO-8 cartridge in the drive which is labeled BN0001L8.

Bildschirmfoto_20260920_115218.png

Bash: 
proxmox-tape barcode-label  -pool bn-continue --drive drive
gives
Code: 
checking/loading media 'BN0001L8'
TASK ERROR: open drive 'drive' (/dev/tape/by-id/scsi-XXXXXXXXXX-sg) failed - wait_until_ready failed - got timeout
Error: task failed (status open drive 'drive' (/dev/tape/by-id/scsi-XXXXXXXXXX-sg) failed - wait_until_ready failed - got timeout)
It takes quite some time for the error to appear and the command to end although the web interface of the Superloader shows the drive idling quite immediately. As a potential alternative
Code: 
proxmox-tape label --drive drive --label-text BN0001L8 -pool bn-continue
for labeling the single tape does not work either.

The tape seems okay, I can tar files to it and read successfully read them back. I even did a long-erase of the tape and cleaned the drive.

dmesg shows repeated messages:
Code: 
connection1:0: detected conn error (1020)
Though, I can still access the library from the PBS web GUI, e.g. unloading the tape from the drive into a slot in the library.

I have seen this thread but not the solution. Chatting with Gemini did not help either.

Any idea what to do? Can I manually write the "Inventory" to the tape?
 
Just a few tips: if you run lsscsi on the pbs server (you may have to install it) you can see which device the tape library and the drives are on. The try mtx -f /dev/sg0 (or whatever device the library is on). Note that the drives and the library are on differenct scsi devices.
 
Thanks for the tips. I already got quite deep into these things recently. The drive and the library on the target are referenced by-id because the generic device names /dev/sg0 /dev/sg1 ... are randomly assigned after a restart of the machine. The devices on the pbs server are correct, as far as I judge from

Code: 
root@pbs:~# proxmox-tape drive list
┌───────┬────────────────────────────────────┬─────────┬────────┬─────────────┬────────────┐
│ name  │ path                               │ changer │ vendor │ model       │ serial     │
╞═══════╪════════════════════════════════════╪═════════╪════════╪═════════════╪════════════╡
│ drive │ /dev/tape/by-id/scsi-XXXXXXXXXX-sg │ qsl3    │ IBM    │ ULTRIUM-HH8 │ XXXXXXXXXX │
└───────┴────────────────────────────────────┴─────────┴────────┴─────────────┴────────────┘
and my experiments with tar cvzf /dev/st0. According to the serial, I have the devices right.
 
Hi,

not sure what is happening, but we'll hopefully find out :)

gothbert said:
Any idea what to do? Can I manually write the "Inventory" to the tape?
Click to expand...
no manually adding will not help since the labeling also includes writing the first block to the tape (our 'media' header)

can you post the output of dmesg after a failed (manual) labeling of the tape?

also, please don't access any of the 'dev/st0' etc. devices during pbs tape operation, otherwise it can happen that the kernel interferes with our scsi commands.
 
That sounds promising, so let's go.

I choose a systematic reproducible approach.

*** 1
Power on the Quantum Superloader 3 (QSL3) and wait until it settles into "System Ready" state.

*** 2
Power on the target host (Debian 13.7) and wait until it has fully booted. Log in as root and check:

Code: 
root@pve-3:~# lsscsi -g
[0:0:0:0]    disk    ATA      Crucial_CCCCCCCC R031  /dev/sda   /dev/sg0
[2:0:0:0]    tape    IBM      ULTRIUM-HH8      S2T5  /dev/st0   /dev/sg1
[2:0:0:1]    mediumx QUANTUM  UHDL             0116  /dev/sch0  /dev/sg2

Though, the iscsi target configuration was incomplete and I had to add the drive again.

*** 3
Log into the Proxmox Backup Server (PBS) initiator host and check:

Code: 
root@pbs:~# lsscsi -g
[0:0:0:0]    disk    ATA      WDC WDWWWWWWWWWW 0A81  /dev/sda   /dev/sg0
[9:0:0:0]    mediumx QUANTUM  UHDL             0116  /dev/sch0  /dev/sg1
[9:0:0:1]    tape    IBM      ULTRIUM-HH8      S2T5  /dev/st0   /dev/sg2
[N:0:1:1]    disk    ADATA SSSSSSSSSSSS                         /dev/nvme0n1  -

I logged out of and back into the iscsi target to make sure that the drive is available.


Corresponding dmesg:
Code: 
[Sep22 19:57] scsi host9: iSCSI Initiator over TCP/IP
[  +0.006998] scsi 9:0:0:0: Medium Changer    QUANTUM  UHDL             0116 PQ: 0 ANSI: 2
[  +0.001488] ch 9:0:0:0: LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
[  +0.110279] ch 9:0:0:0: [ch0] type #1 (mt): 0x0+1 [medium transport]
[  +0.000017] ch 9:0:0:0: [ch0] type #2 (st): 0x100+8 [storage]
[  +0.000011] ch 9:0:0:0: [ch0] type #3 (ie): 0x0+0 [import/export]
[  +0.000008] ch 9:0:0:0: [ch0] type #4 (dt): 0x20+1 [data transfer]
[  +0.124295] ch 9:0:0:0: [ch0] dt 0x20:
[  +0.000007] ch 9:0:0:0: [ch0] ID/LUN unknown
[  +0.000005] ch 9:0:0:0: [ch0] INITIALIZE ELEMENT STATUS, may take some time ...
[ +15.572633] ch 9:0:0:0: [ch0] ... finished
[  +0.000017] ch 9:0:0:0: Attached scsi changer ch0
[  +0.000398] ch 9:0:0:0: Attached scsi generic sg1 type 8
[  +0.005741] scsi 9:0:0:1: Sequential-Access IBM      ULTRIUM-HH8      S2T5 PQ: 0 ANSI: 6
[  +0.026445] st 9:0:0:1: Attached scsi tape st0
[  +0.000008] st 9:0:0:1: st0: try direct i/o: yes (alignment 512 B)
[  +0.000118] st 9:0:0:1: Attached scsi generic sg2 type 1

*** 4
Use the PBS Web GUI to load the tape from slot 1 to the drive.
This gives a timeout error quite early, though, the tape is finally in the drive and the Web GUI shows the exactly that.

dmesg shows:
Code: 
[Sep22 19:24]  connection1:0: detected conn error (1020)


Code: 
root@pbs:~# mtx -f /dev/sch0 status
  Storage Changer /dev/sch0:1 Drives, 8 Slots ( 0 Import/Export )
Data Transfer Element 0:Full (Storage Element 1 Loaded)
      Storage Element 1:Empty
      Storage Element 2:Empty
      Storage Element 3:Empty
      Storage Element 4:Empty
      Storage Element 5:Empty
      Storage Element 6:Empty
      Storage Element 7:Empty
      Storage Element 8:Full :VolumeTag=CLN001CU




I run
Code: 
proxmox-tape label --drive drive --label-text BN0001L8 -pool bn-continue

Corresponding dmesg on PBS:
Code: 
[Sep22 19:59]  connection2:0: detected conn error (1020)
[  +2.017900]  connection2:0: detected conn error (1020)
[  +2.024449]  connection2:0: detected conn error (1020)
[  +2.023055]  connection2:0: detected conn error (1020)
[  +3.024699]  connection2:0: detected conn error (1020)
[  +2.014132]  connection2:0: detected conn error (1020)
[  +2.020736]  connection2:0: detected conn error (1020)
[  +2.023409]  connection2:0: detected conn error (1020)
[  +2.022155]  connection2:0: detected conn error (1020)
[  +2.009352]  connection2:0: detected conn error (1020)
[  +2.023685]  connection2:0: detected conn error (1020)
[  +2.013886]  connection2:0: detected conn error (1020)
[  +2.022936]  connection2:0: detected conn error (1020)
[  +2.022124]  connection2:0: detected conn error (1020)
[  +2.013975]  connection2:0: detected conn error (1020)
[  +2.019390]  connection2:0: detected conn error (1020)
[  +2.023931]  connection2:0: detected conn error (1020)
[  +2.021252]  connection2:0: detected conn error (1020)
[  +2.017758]  connection2:0: detected conn error (1020)
[  +2.015700]  connection2:0: detected conn error (1020)
[  +2.023359]  connection2:0: detected conn error (1020)
[  +2.022505]  connection2:0: detected conn error (1020)
[  +2.023510]  connection2:0: detected conn error (1020)
[  +2.014686]  connection2:0: detected conn error (1020)
[  +2.015895]  connection2:0: detected conn error (1020)
repeating for a long long time...

The corresponding dmesg on the target host (host with the physically attached QSL3):
Code: 
[Sep22 19:59] ITT: 0x00000054 read_data_left is zero!
[  +0.000018] iscsit_get_datain_values failed for ITT: 0x00000054
[  +2.020891] ITT: 0x10000056 read_data_left is zero!
[  +0.000019] iscsit_get_datain_values failed for ITT: 0x10000056
[  +2.024462] ITT: 0x20000058 read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0x20000058
[  +2.022805] ITT: 0x3000005a read_data_left is zero!
[  +0.000019] iscsit_get_datain_values failed for ITT: 0x3000005a
[  +3.024673] ITT: 0x4000005d read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0x4000005d
[  +2.015434] ITT: 0x5000005f read_data_left is zero!
[  +0.000018] iscsit_get_datain_values failed for ITT: 0x5000005f
[  +2.020251] ITT: 0x60000061 read_data_left is zero!
[  +0.000019] iscsit_get_datain_values failed for ITT: 0x60000061
[  +2.022844] ITT: 0x70000064 read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0x70000064
[  +2.022534] ITT: 0x80000066 read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0x80000066
[  +2.009946] ITT: 0x90000068 read_data_left is zero!
[  +0.000015] iscsit_get_datain_values failed for ITT: 0x90000068
[  +2.022623] ITT: 0xa000006a read_data_left is zero!
[  +0.000019] iscsit_get_datain_values failed for ITT: 0xa000006a
[  +2.013777] ITT: 0xb000006c read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0xb000006c
[  +2.022642] ITT: 0xc000006e read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0xc000006e
[  +2.022682] ITT: 0xd0000070 read_data_left is zero!
[  +0.000019] iscsit_get_datain_values failed for ITT: 0xd0000070
[  +2.013601] ITT: 0xe0000072 read_data_left is zero!
[  +0.000019] iscsit_get_datain_values failed for ITT: 0xe0000072
[  +2.015247] ITT: 0xf0000074 read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0xf0000074
[  +2.027210] ITT: 0x00000077 read_data_left is zero!
[  +0.000019] iscsit_get_datain_values failed for ITT: 0x00000077
[  +2.022472] ITT: 0x10000079 read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0x10000079
[  +2.016341] ITT: 0x2000007b read_data_left is zero!
[  +0.000019] iscsit_get_datain_values failed for ITT: 0x2000007b
[  +2.016327] ITT: 0x3000007e read_data_left is zero!
[  +0.000019] iscsit_get_datain_values failed for ITT: 0x3000007e
[  +2.022683] ITT: 0x40000001 read_data_left is zero!
[  +0.000017] iscsit_get_datain_values failed for ITT: 0x40000001
[  +2.023096] ITT: 0x50000003 read_data_left is zero!
[  +0.000019] iscsit_get_datain_values failed for ITT: 0x50000003
[  +2.023674] ITT: 0x60000007 read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0x60000007
[  +2.014573] ITT: 0x70000009 read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0x70000009
[  +2.016274] ITT: 0x8000000a read_data_left is zero!
[  +0.000020] iscsit_get_datain_values failed for ITT: 0x8000000a
[  +2.020403] ITT: 0x9000000c read_data_left is zero!
[  +0.000016] iscsit_get_datain_values failed for ITT: 0x9000000c
[  +2.014128] ITT: 0xa000000e read_data_left is zero!
repeating for a long long time...

And this is where I am lost. QSL3 is idle, the label command hangs and the wait-cursor in the Web GUI continues.

Edit: finally the proxmox-tape command bails out with
Code: 
TASK ERROR: open drive 'drive' (/dev/tape/by-id/scsi-TTTTTTTTT-sg) failed - wait_until_ready failed - got timeout
Error: task failed (status open drive 'drive' (/dev/tape/by-id/scsi-TTTTTTTTT-sg) failed - wait_until_ready failed - got timeout)
an no additional information in the dmesg kernel logs on either end.
 
Last edited:
The tape appears healthy. The main issue is likely the iSCSI connection, especially the connection1:0 detected conn error (1020) messages.

Check:


iscsiadm -m session -P 3

Also verify the tape device with:



sg_turs /dev/tape/by-id/scsi-XXXXXXXXXX-sg

Do not manually write the PBS inventory to the tape. Fix the iSCSI/SCSI timeout first, then retry proxmox-tape barcode-label.
 
ok reading the errors from the target side, it seems this is an issue in our code that was recentyl fixed:
https://lore.proxmox.com/pbs-devel/20260817123128.2813924-1-d.csapak@proxmox.com/T/#t

the patch is applied, but not yet in a bumped package

The patch fixes an issue where we would set up a buffer for the tape/changer/etc. to write in but the commands we send don't have any data to write data to. so I think the target side sees that and errors out.

to verify, could you enable debug logging for iscsi/lio (i assume that's what you using for the iscsi target) ?
 
You must log in or register to reply here.
Top Bottom
Back