Tape confusion in PBS 3.1-5

[storelli]

We have a strange problem with PBS.

In the tape loader we have 40 tapes + 2 cleaning.
Two IBM Ultrium-HH8 drives.
We used both drives to label the tapes. They are distributed in 3 media pools named MP_Weekly, MP_Monthly and MP_Yearly.
When labeling tapes all seems to be running good but when we load a tape in one drive and ask to read the label the tape looks mislabeled.
For example :
- In Drive0 the system auto-loaded tape A00000L8 to start a backup and it tells that the tape is labeled A00003L8
- In Drive1 we asked the system to load the tape A00002L8 (from the pool MP_Weekly) when it read the label it say that the tape is the A00039L8 from the pool MP_yearly.

Not sure what we did wrong there but I'm joining two screenshots to detail that.



Please tell me what information are needed and I will be happy to provide them.
Many thanks for all help.

Regards

 

[dcsapak]

what i can imagine happened is that somehow the wrong label ended up on the tapes

can you load 2 tapes into your drive and show the output of

proxmox-tape changer status <changername>
pmtx status --changer <changername>

also i guess the library has some kind of web ui? if yes can you check there which tapes are inserted in the drives ?

 

[storelli]

Hello Dominik,

many thanks for your assistance.

Here are the results with the same tapes as before in the drives :

proxmox-tape changer status <changername>

┌───────────────┬──────────┬────────────┬─────────────┐
│ entry-kind    │ entry-id │ label-text │ loaded-slot │
╞═══════════════╪══════════╪════════════╪═════════════╡
│ drive         │        0 │ A00000L8   │           1 │
├───────────────┼──────────┼────────────┼─────────────┤
│ drive         │        1 │ A00002L8   │           3 │
├───────────────┼──────────┼────────────┼─────────────┤
│ drive         │        2 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ drive         │        3 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ import-export │       51 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ import-export │       52 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ import-export │       53 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ import-export │       54 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │        1 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │        2 │ A00001L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │        3 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │        4 │ A00003L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │        5 │ A00004L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │        6 │ A00005L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │        7 │ A00006L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │        8 │ A00007L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │        9 │ A00008L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       10 │ A00009L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       11 │ A00010L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       12 │ A00011L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       13 │ A00012L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       14 │ A00013L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       15 │ A00014L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       16 │ A00015L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       17 │ A00016L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       18 │ A00017L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       19 │ A00018L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       20 │ A00019L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       21 │ A00020L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       22 │ A00021L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       23 │ A00022L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       24 │ A00023L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       25 │ A00024L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       26 │ A00025L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       27 │ A00026L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       28 │ A00027L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       29 │ A00028L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       30 │ A00029L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       31 │ A00030L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       32 │ A00031L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       33 │ A00032L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       34 │ A00033L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       35 │ A00034L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       36 │ A00035L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       37 │ A00036L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       38 │ A00037L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       39 │ A00038L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       40 │ A00039L8   │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       41 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       42 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       43 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       44 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       45 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       46 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       47 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       48 │            │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       49 │ CLN000L    │             │
├───────────────┼──────────┼────────────┼─────────────┤
│ slot          │       50 │ CLN001L    │             │
└───────────────┴──────────┴────────────┴─────────────┘

pmtx status --changer <changername>

using device /dev/tape/by-id/scsi-2210630100
Transport Element (Griper)      0: Empty
Data Transfer Element (Drive)   0: VolumeTag("A00000L8"), Source: 1, Serial: 10WT097286
Data Transfer Element (Drive)   1: VolumeTag("A00002L8"), Source: 3, Serial: 10WT097275
Data Transfer Element (Drive)   2: Empty
Data Transfer Element (Drive)   3: Empty
  Storage Element   1: Empty
  Storage Element   2: VolumeTag("A00001L8")
  Storage Element   3: Empty
  Storage Element   4: VolumeTag("A00003L8")
  Storage Element   5: VolumeTag("A00004L8")
  Storage Element   6: VolumeTag("A00005L8")
  Storage Element   7: VolumeTag("A00006L8")
  Storage Element   8: VolumeTag("A00007L8")
  Storage Element   9: VolumeTag("A00008L8")
  Storage Element  10: VolumeTag("A00009L8")
  Storage Element  11: VolumeTag("A00010L8")
  Storage Element  12: VolumeTag("A00011L8")
  Storage Element  13: VolumeTag("A00012L8")
  Storage Element  14: VolumeTag("A00013L8")
  Storage Element  15: VolumeTag("A00014L8")
  Storage Element  16: VolumeTag("A00015L8")
  Storage Element  17: VolumeTag("A00016L8")
  Storage Element  18: VolumeTag("A00017L8")
  Storage Element  19: VolumeTag("A00018L8")
  Storage Element  20: VolumeTag("A00019L8")
  Storage Element  21: VolumeTag("A00020L8")
  Storage Element  22: VolumeTag("A00021L8")
  Storage Element  23: VolumeTag("A00022L8")
  Storage Element  24: VolumeTag("A00023L8")
  Storage Element  25: VolumeTag("A00024L8")
  Storage Element  26: VolumeTag("A00025L8")
  Storage Element  27: VolumeTag("A00026L8")
  Storage Element  28: VolumeTag("A00027L8")
  Storage Element  29: VolumeTag("A00028L8")
  Storage Element  30: VolumeTag("A00029L8")
  Storage Element  31: VolumeTag("A00030L8")
  Storage Element  32: VolumeTag("A00031L8")
  Storage Element  33: VolumeTag("A00032L8")
  Storage Element  34: VolumeTag("A00033L8")
  Storage Element  35: VolumeTag("A00034L8")
  Storage Element  36: VolumeTag("A00035L8")
  Storage Element  37: VolumeTag("A00036L8")
  Storage Element  38: VolumeTag("A00037L8")
  Storage Element  39: VolumeTag("A00038L8")
  Storage Element  40: VolumeTag("A00039L8")
  Storage Element  41: Empty
  Storage Element  42: Empty
  Storage Element  43: Empty
  Storage Element  44: Empty
  Storage Element  45: Empty
  Storage Element  46: Empty
  Storage Element  47: Empty
  Storage Element  48: Empty
  Storage Element  49: VolumeTag("CLN000L")
  Storage Element  50: VolumeTag("CLN001L")
  Import/Export    51: Empty
  Import/Export    52: Empty
  Import/Export    53: Empty
  Import/Export    54: Empty

 

[dcsapak]

does the library have a webui/interface? can you check what it says there?

was there maybe some confusion in the past with the driver order of the changer config?

 

[storelli]

Yes there is a very basic web interface.

But from what I see it looks correct :

 

[dcsapak]

ok, what does

proxmox-tape read-label --drive <drive1>
proxmox-tape read-label --drive <drive2>

show?

if they're incorrect, there must have been some drive/tape/label confusion in the past so that during the initial labeling a wrong label identifier was written on the on-tape label data

 

[storelli]

As expected labels are wrong.
Do you think I should format every tapes and the label them again ?
Do you think this problem could appear because we labelled the tapes with the two drives simultaneously ?

proxmox-tape read-label --drive Drive0
┌────────────┬──────────────────────────────────────┐
│ Name       │ Value                                │
╞════════════╪══════════════════════════════════════╡
│ label-text │ A00003L8                             │
├────────────┼──────────────────────────────────────┤
│ uuid       │ 50f37110-7dd4-4d27-a5c9-d513d1a53ee9 │
├────────────┼──────────────────────────────────────┤
│ ctime      │ Tue Apr 23 13:37:54 2024             │
├────────────┼──────────────────────────────────────┤
│ pool       │ MP_Weekly                            │
└────────────┴──────────────────────────────────────┘

proxmox-tape read-label --drive Drive1
┌────────────┬──────────────────────────────────────┐
│ Name       │ Value                                │
╞════════════╪══════════════════════════════════════╡
│ label-text │ A00039L8                             │
├────────────┼──────────────────────────────────────┤
│ uuid       │ 90f131b6-610f-4975-816e-191f52951183 │
├────────────┼──────────────────────────────────────┤
│ ctime      │ Tue Apr 23 17:57:17 2024             │
├────────────┼──────────────────────────────────────┤
│ pool       │ MP_Yearly                            │
└────────────┴──────────────────────────────────────┘

 

[dcsapak]

Do you think I should format every tapes and the label them again ?

yes that will be the only way to get the correct labels on them

Do you think this problem could appear because we labelled the tapes with the two drives simultaneously ?

that should not matter. the only thing that comes to mind how that could happen is that the drives have/had misconfigured slots (so drive 1 had slot 0 and so on). in that case you'd e.g. say 'load tape A in drive 0' and 'load tape B in drive 1' but in reality it would have been tape B in drive 0 and tape A in drive 1.

can you post your tape config, so we can check if there's maybe something wrong: `/etc/proxmox-backup/tape.cfg`

 

[storelli]

Tapes are empty (from a backup perspective) so I definitely can try to format them all and label them again.

Here is the configuration :

cat /etc/proxmox-backup/tape.cfg
changer: Qualstar_RLS-85
        path /dev/tape/by-id/scsi-2210630100

lto: Drive0
        changer Qualstar_RLS-85
        path /dev/tape/by-id/scsi-10WT097275-sg

lto: Drive1
        changer Qualstar_RLS-85
        changer-drivenum 1
        path /dev/tape/by-id/scsi-10WT097286-sg

 

[dcsapak]

mhmm ok the config looks good, did it change at some point (especially did you change the 'changer-drivenum' parameter at some point?)

 

[storelli]

No it was installed like that and all the configuration has been done via the web interface.

 

[dcsapak]

ok, that's weird. do you still have the original task logs of the labeling? if yes, it would be great to post a few of them here.
in the meantime you can try formatting and relabeling the tapes and see if it fixes that

 

[storelli]

Can you please guide on finding the log and the information needed in it ?

I have started the process to format all the tapes and strangely when I formatted tape A00002L8 it staying with the pool MP_Weekly but the pool disappeared on tape A00039L8. I mean I formatted the tape.
Same with tape A00000L8 and A00006L8.

 

[dcsapak]

Can you please guide on finding the log and the information needed in it ?

in the top right, you can click on tasks -> all tasks

there you can filter by 'task type' just enter 'label' and it should show you the label/barcode label tasks, for the log just double click it

I have started the process to format all the tapes and strangely when I formatted tape A00002L8 it staying with the pool MP_Weekly but the pool disappeared on tape A00039L8. I mean I formatted the tape.
Same with tape A00000L8 and A00006L8.

that does not make really make sense to me... can you also post the task logs of those actions?
(maybe also the journal/syslog from that time?)

 

[storelli]

Thanks for the guidance.
Nothing too fancy with the labeling :

2023-08-09T15:40:53+02:00: Label media 'A00002L8' for pool 'MP_weekly'
2023-08-09T15:40:55+02:00: TASK OK

Probably it's not related but if you remember we have this bug with our loader/drives : https://bugzilla.proxmox.com/show_bug.cgi?id=4904

For the last formatting I formatted two times the tapes in Drive1 because pool didn't disappear and I first didn't realized that it disappeared on another tape. But as you can see in the log the label do not correspond to the one reported in the Drives module on my previous screenshot :

Proxmox Backup Server 3.1-5
2024-04-30T09:41:10+02:00: found media 'A00039L8' with uuid '90f131b6-610f-4975-816e-191f52951183'
2024-04-30T09:41:19+02:00: TASK OK

Proxmox Backup Server 3.1-5
2024-04-30T09:42:01+02:00: found empty media - format anyways
2024-04-30T09:42:10+02:00: TASK OK

Same with Drive0. But my bad in previous message I wrote A00006L8 instead of A00003L8.

Proxmox Backup Server 3.1-5
2024-04-30T09:41:44+02:00: found media 'A00003L8' with uuid '50f37110-7dd4-4d27-a5c9-d513d1a53ee9'
2024-04-30T09:41:52+02:00: TASK OK

 

[dcsapak]

ah ok now that makes sense. when the wrong media-labels are on tape and we format, we of course remove the one from the inventory that was actually on the tape, not what the barcode says...
i'd just go through each tape one by one, format it, and then do a barcode label to label all the tapes in one go

then i'd check if they are correct

still not sure what could have happened though, there is not much in our code that could be confused in this way besides from what i already wrote: when the drivenum of the drive in the config does not correspond to the actual config of the changer

 

[storelli]

ah ok now that makes sense. when the wrong media-labels are on tape and we format, we of course remove the one from the inventory that was actually on the tape, not what the barcode says...

Yes that was my thought

i'd just go through each tape one by one, format it, and then do a barcode label to label all the tapes in one go

I foresee two problems with that. First one we have 3 pools of tape and second one tapes are still not ejected automatically for now and we have to use "mt -f" to eject them.

When I have formatted all tapes is there a way to "reset" the internal tape database in PBS?

 

[dcsapak]

I foresee two problems with that. First one we have 3 pools of tape and second one tapes are still not ejected automatically for now and we have to use "mt -f" to eject them.

you could also use 'proxmox-tape eject --drive <drivename>' for that

When I have formatted all tapes is there a way to "reset" the internal tape database in PBS?

if you have formatted all tapes, the inventory should be empty, but you can check `/var/lib/proxmox-backup/tape/inventory.json` (i guess you could delete it, pbs should recreate that, but to be sure i'd only rename it so you could rename it back in case of a problem)

 

[storelli]

you could also use 'proxmox-tape eject --drive <drivename>' for that

Unfortunately it's not working :

proxmox-tape eject --drive Drive0
TASK ERROR: unload drive failed - Illegal Request, ASC=3b, vendor specific qualification ASCQ=90 (hex)
Error: task failed (status unload drive failed - Illegal Request, ASC=3b, vendor specific qualification ASCQ=90 (hex))

if you have formatted all tapes, the inventory should be empty, but you can check `/var/lib/proxmox-backup/tape/inventory.json` (i guess you could delete it, pbs should recreate that, but to be sure i'd only rename it so you could rename it back in case of a problem)

I'm working on it and will post result here. Thanks for helping.

 

[storelli]

So I formatted all the tapes twice.
Loaded the tape in the drive, first format, there was the comment in the log saying "found media XXX with ID XYZ", second format, comment was "found empty media - format anyways".
After formatting the 40 tapes there was two objects left in the file /var/lib/proxmox-backup/tape/inventory.json
I deleted those two objects.

Then I started to label all the tapes again but before and after labeling I clicked on the "read label" button. Two tapes where still holding a label and I had to format them before relabeling.

At this point everything looks good but when launching the first backup it seems that the tapes are mislabeled again.