[SOLVED] Tape Label TASK ERROR: media read error - read failed - do_scsi_pt failed with err EINVAL: Invalid argument

[sundxplosion]

Hi there,

I am currently trying to label the media of our tape library,
but I'm getting a Task Error

2022-01-04T19:08:13+01:00: TASK ERROR: media read error - read failed - do_scsi_pt failed with err EINVAL: Invalid argument

I think it is related to the hardware we use: The Overland NEO 400s which drive is attached to the backup-server running Proxmox Backup Server.

What can I do to find out more about the error?

I get this error for several other tasks I start that are related to tape media

Library Information
Serial Number	MXA134Z055
Product	NEOseries
Currently Installed Library Firmware	G.83 / 3.10e
Bootcode Firmware Revision	0.80
Barcode Reader	SE625
Library Mode	Manual, Random
WWide Node Name	500143801603F830
Library Controller Version	LCM 1.1

Drive Information	1 (LUN)
Vendor ID	HP
Product ID	Ultrium 5-SCSI
Serial Number	HU1140JVN4
Firmware Revision	Z58W
World Wide ID - Port A	500143801603F832
Physical Drive Slot Number	1
Element Address	1
Library LUN Hosted By Drive	Yes
Data Compression	Yes
Interface Type	SAS

 

[dietmar]

Is the drive firmware up to date?

 

[dietmar]

Also, is the media empty or contains encrypted data? Please can you test with an new/empty tape?

 

[sundxplosion]

Hi
the media do not contain encrypted data, I sadly got no new media at hand right now.

I checked to upgrade the drive firmware, but sadly hp only offers a .exe I search a way to get around this right now.
At first I try to get some non paywalled .exe to extract the firmware of. (easy) then I will come back.

 

[sundxplosion]

I managed to install the hp_ltt software with some tweaks to their proprietary rpm RHEL/Linux package. The tool says that the firmware description cant be read, but also reports that there are no errors with the drive whatsoever.

Because library and drive where used before by another company, i would conclude that the hardware/software is running.

I managed to test with a fresh media and it still shows this problem.

Do I need to install custom drivers on the server in order to use some devices? I read that PBS uses its own drivers..

Is it a problem that the labels follow this enumeration:

 

[sundxplosion]

Okey i managed to get my hands on some firmware version that are a bit more latest. currently trying the firmware upgrade manually over pg_write_puffer, hope this thing will survive this attempt

 

[sundxplosion]

Okey so the firmware of both drive and library are already latest, Im currently upgrading once again to make sure that everything is on the right firmware revision.
Thanks to overland, seems that they know about the bad situation with hp drives:
http://support.overlandstorage.com/jive/servlet/KbServlet/download/3853-102-993/firmware

 

[sundxplosion]

I am able to execute other things just fine with the drive.
For Example:

pmt cartridge-memory

cleanly reads the tapes memory

#pmt erase --drive neos-drive --fast=false

erases the tape as expected.

I checked this by checking the state of the tape library web interface. Drive Status says "erasing" for a couple of minutes.
So I think that at least the drive is functional.

So in conclusion of today: Drive and library firmware is up to date (and reflashed) basic tape operations via pmt are working so far.
I think that my problem has something to do with the labeling mechanism of Proxmox Backup Server.

Do you have any ideas for further troubleshooting?

 

[sundxplosion]

I checked back for the pmt erase: ended like this:

root@srv-backup-1:~# pmt erase --drive neos-drive --fast=false --device /dev/tape/by-id/scsi-HU1140JVN4-sg
using device /dev/tape/by-id/scsi-HU1140JVN4-sg
Error: erase failed - Illegal Request, Additional sense: Invalid command operation code

A look into syslog, tells not really more:

root@srv-backup-1:~# tail -f /var/log/syslog
Jan  6 02:30:43 srv-backup-1 proxmox-backup-proxy[1024]: scsi command failed: transport error
Jan  6 02:31:09 srv-backup-1 proxmox-backup-proxy[1024]: scsi command failed: transport error
Jan  6 02:31:09 srv-backup-1 proxmox-backup-proxy[1024]: unable to get changer 'neos' status - error reading element status: read element status (B8h) failed: scsi command failed: transport error
Jan  6 02:31:48 srv-backup-1 proxmox-backup-proxy[1024]: scsi command failed: transport error

 

[dcsapak]

mhm.. could you try to erase the tape with the 'mt' command instead of pmt ? (you may have to install the 'mt-st' package for that to be available)

 

[sundxplosion]

Thanks for your suggestions, i will try it today after lunch.

I further digged into tape error codes and library event codes and found some errors suggesting a drive clean. which was not mentioned anywhere else on the library interface but I give it a go as well.

22.01.05 16:00:05.80  Event - Drive Warn or Crit Tape Alert flag
     Event Code: 0x84 - tape alert
     Element number: 0x01, 1
     Drive number: 0x01, 1
     Tape Alert Flag: 0x20, 32
22.01.05 15:54:45.66  Event - Drive Warn or Crit Tape Alert flag
     Event Code: 0x84 - tape alert
     Element number: 0x01, 1
     Drive number: 0x01, 1
     Tape Alert Flag: 0x22, 34

 

[sundxplosion]

Hi there,
so after cleaning the drive via the administration hw panel, i erased a tape with fast=false.

But i still cant label the tape. Same error as before.

I provide the .lzt file from the hp-ltt tool
https://cloud.sund-xplosion.de/index.php/s/g4zTpmKsep4AN8s
password:
i5w9qA3R5w

 

[dcsapak]

so after cleaning the drive via the administration hw panel, i erased a tape with fast=false.

so it did successfully erase then?

can you load such a tape into the drive and post the output of

proxmox-tape status --output-format json-pretty

?

 

[sundxplosion]

so it did successfully erase then?

can you load such a tape into the drive and post the output of

proxmox-tape status --output-format json-pretty

?

Ofcourse, here is the output with the erased tape loaded to the drive:

{
  "alert-flags": "(empty)",
  "block-number": 0,
  "blocksize": 0,
  "buffer-mode": 1,
  "bytes-read": 239127756800,
  "bytes-written": 24059665448960,
  "compression": true,
  "density": "LTO5",
  "file-number": 0,
  "manufactured": 1339632000,
  "medium-passes": 922,
  "medium-wearout": 0.057625,
  "product": "Ultrium 5-SCSI",
  "revision": "Z58W",
  "vendor": "HP",
  "volume-mounts": 102
}

 

[dcsapak]

ok, thanks looks good so far (aside from the tape already being older (though 922 passes should not be a problem)

can you do the labeling on the cli (maybe theres a bit more info) and post the output?

proxmox-tape label --label-text "LABEL"

?

 

[sundxplosion]

Hi there, here is what i got so far:

root@srv-backup-1:~# proxmox-tape label --label-text "ABR401L5" --drive neos-drive --output-format=text --pool neos-pool
TASK ERROR: media read error - read failed - do_scsi_pt failed with err EINVAL: Invalid argument

I dug a bit deeper to find our more details again on that omnios Invalid Argument:
From /var/log/syslog:

Jan 12 18:35:51 srv-backup-1 proxmox-backup-proxy[1034]: Illegal Request, Additional sense: Invalid field in cdb

from within the /var/log/proxmox-backup/tasks/archive

UPID:srv-backup-1:0000040A:00000AA6:00000090:61DF109A:label-media:neos\x2ddrive:root@pam: 61DF109A media read error - read failed - do_scsi_pt failed with err EINVAL: Invalid argument

From /proc/scsi/debug/*:

root@srv-backup-1:/proc/scsi/sg# tail *
^[[3~==> allow_dio <==
0

==> debug <==
max_active_device=5  def_reserved_size=32768
 >>> device=sg3 7:0:0:1   em=0 sg_tablesize=60 excl=0 open_cnt=1
   FD(1): timeout=60000ms bufflen=32768 (res)sgat=1 low_dma=0
   cmd_q=1 f_packid=0 k_orphan=0 closed=0
     act: id=0 blen=65536 t_o/elap=300000/2328ms sgat=2 op=0xb8

==> def_reserved_size <==
32768

==> device_hdr <==
host    chan    id      lun   type      opens   qdepth  busy    online

==> devices <==
0 2     0       0       0     1 256     0       1
0 2     1       0       0     1 256     0       1
7 0     0       0       1     1 256     0       1
7 0     0       1       8     1 256     1       1
5 0     0       0       5     1 1       0       1

==> device_strs <==
DELL            PERC H710               3.13
DELL            PERC H710               3.13
HP              Ultrium 5-SCSI          Z58W
HP              MSL G3 Series           G.83
PLDS            DVD-ROM DS-8D3SH        HD51

==> version <==
30536   3.5.36 [20140603]

from /proc/scsi/scsi

root@srv-backup-1:/proc/scsi# cat scsi
Attached devices:
Host: scsi0 Channel: 02 Id: 00 Lun: 00
  Vendor: DELL     Model: PERC H710        Rev: 3.13
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi0 Channel: 02 Id: 01 Lun: 00
  Vendor: DELL     Model: PERC H710        Rev: 3.13
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi7 Channel: 00 Id: 00 Lun: 00
  Vendor: HP       Model: Ultrium 5-SCSI   Rev: Z58W
  Type:   Sequential-Access                ANSI  SCSI revision: 06
Host: scsi7 Channel: 00 Id: 00 Lun: 01
  Vendor: HP       Model: MSL G3 Series    Rev: G.83
  Type:   Medium Changer                   ANSI  SCSI revision: 05
Host: scsi5 Channel: 00 Id: 00 Lun: 00
  Vendor: PLDS     Model: DVD-ROM DS-8D3SH Rev: HD51
  Type:   CD-ROM                           ANSI  SCSI revision: 05

I will check back to the NOC tomorrow to do some hardware maintenance on other servers, totally will check bios settings of that machine, maybe there is something to be done about this.

Best Regards


Markus

 

[sundxplosion]

I just discovered a different driver for the lto-5 drive from hp on the overland FTP server. I flash a new firmware and hope that this will solve the problem for now.

(no big hopes, its uploaded 2014, still 2 years newer but meeh..)

 

[sundxplosion]

So with the newest official firmware from the overland FTP still no success, same behavior for the drive

I tried once again, flashing manually without the web interface, but this firmware, potentially a bit newer gets rejected as well.

Any ideas for bios/hardware settings? Like is there something like a mode or setting in common with scsi?

 

[dcsapak]

mhmm.. ok it seems during the read of the first block, something goes wrong...
to further debug this, can you do the following commands (and post the output)?

pmt rewind --drive DRIVENAME
proxmox-tape scan --drive DRIVENAME

(replace DRIVENAME with the configured one ofc)

 

[dcsapak]

oh and please do a 'pmt format' before...