PBS Datastore is not available

[Ziesie]

Hey everyone,

i have a running PBS with an external USB hard drive mounted under /media/usb-pbs where all backups are stored.
After the system was switched off for two weeks and I changed the static IP address of the PBS, the corresponding data storage is shown as "not available" in the user interface.



Every time I load the data store, the follwing error is displayed in the logs:

Nov 23 20:59:05 pbs2 proxmox-backup-[633]: pbs2 proxmox-backup-proxy[633]: GET /api2/json/admin/datastore/backup-usb/status/?verbose=true: 400 Bad Request: [client [::ffff:192.XXX.XXX.XXX]:45384] ENOTDIR: Not a directory

But the drive is succesfully mounted into the PBS, as you can see here:

root@pbs2:~# ls -all /media/usb-pbs/
total 1084
drwx------  6 backup backup    4096 Aug 13 00:12 .
drwxr-xr-x  3 root   root      4096 Feb 10  2023 ..
drwxr-x---  1 backup backup 1069056 Feb 10  2023 .chunks
drwxr-xr-x 13 backup backup    4096 Oct 26 03:12 ct
-rw-r--r--  1 backup backup     328 Aug 13 00:12 .gc-status
-rw-r--r--  1 backup backup       0 Feb 10  2023 .lock
drwx------  2 root   root     16384 Feb 10  2023 lost+found
drwxr-xr-x  3 backup backup    4096 Feb 11  2023 vm

This is my datastore.cfg:

datastore: backup-usb
        comment
        gc-schedule daily
        path /media/usb-pbs
        prune-schedule daily

My PBS runs in a VM on a PVE where the usb drive is passed through.
PBS: V2.4-3
PVE: v7.4-17

Many thanks for any advice!

 

[fabian]

could you try restart proxmox-backup-proxy and proxmox-backup? anything interesting in the journal?

 

[Ziesie]

I have rebooted the Proxy and the whole system several times.

Here are some lines from the booting:

ov 24 17:45:09 pbs2 systemd[1]: Mounting /media/usb-pbs...
Nov 24 17:45:09 pbs2 systemd[1]: Found device /dev/pbs/swap.
Nov 24 17:45:09 pbs2 systemd[1]: Activating swap /dev/pbs/swap...
Nov 24 17:45:09 pbs2 kernel: Adding 3145724k swap on /dev/mapper/pbs-swap.  Priority:-2 extents:1 across:3145724k FS
Nov 24 17:45:09 pbs2 systemd[1]: Activated swap /dev/pbs/swap.
Nov 24 17:45:09 pbs2 systemd[1]: Reached target Swap.
Nov 24 17:45:09 pbs2 systemd[1]: Finished Wait for udev To Complete Device Initialization.
Nov 24 17:45:09 pbs2 systemd[1]: Condition check resulted in Import ZFS pools by cache file being skipped.
Nov 24 17:45:09 pbs2 systemd[1]: Starting Import ZFS pools by device scanning...
Nov 24 17:45:09 pbs2 kernel: EXT4-fs (sdb1): warning: mounting fs with errors, running e2fsck is recommended
Nov 24 17:45:09 pbs2 systemd[1]: Finished Helper to synchronize boot up for ifupdown.
Nov 24 17:45:09 pbs2 zpool[470]: no pools available to import
Nov 24 17:45:09 pbs2 systemd[1]: Finished Import ZFS pools by device scanning.
Nov 24 17:45:09 pbs2 systemd[1]: Reached target ZFS pool import target.
Nov 24 17:45:09 pbs2 systemd[1]: Starting Mount ZFS filesystems...
Nov 24 17:45:09 pbs2 systemd[1]: Starting Wait for ZFS Volume (zvol) links in /dev...
Nov 24 17:45:09 pbs2 systemd[1]: Finished Mount ZFS filesystems.
Nov 24 17:45:09 pbs2 zvol_wait[489]: No zvols found, nothing to do.
Nov 24 17:45:09 pbs2 systemd[1]: Finished Wait for ZFS Volume (zvol) links in /dev.
Nov 24 17:45:09 pbs2 kernel: EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
Nov 24 17:45:09 pbs2 systemd[1]: Mounted /media/usb-pbs.

I also got this message from the proxmox-backup-proxy:

Nov 24 17:57:34 pbs2 proxmox-backup-proxy[637]: error during snapshot file listing: 'unable to load blob '"/media/usb-pbs/ct/109/2023-10-16T23:08:56Z/index.json.blob"' - Is a directory (os error 21)'

Why am I getting this director error? When I navigate to the directory, the index.json.blob is a file:

root@pbs2:~# ls -all /media/usb-pbs/ct/109/2023-10-16T23\:08\:56Z/
total 24
drwxr-xr-x  2 backup backup 4096 Oct 28 05:00 .
drwxr-xr-x 18 backup backup 4096 Oct 29 05:00 ..
-rw-r--r--  1 backup backup 5256 Oct 17 03:08 catalog.pcat1.didx
-rw-r--r--  1 backup backup  879 Oct 17 03:08 client.log.blob
drwxr-xr-x  2 backup backup 4096 Oct 19 03:04 index.json.blob

 

[Dunuin]

drwxr-xr-x 2 backup backup 4096 Oct 19 03:04 index.json.blob

According to ls it is a directory

Nov 24 17:45:09 pbs2 kernel: EXT4-fs (sdb1): warning: mounting fs with errors, running e2fsck is recommended

Did you run that fsck to repair the filesystem? I guess the sdb with errors is your passthroughed USB disk?

 

[Ziesie]

Ok, I din't expect the filesystem could to have any damage.
E2fck shows several erros (Multiply-claimed blocks, nvalid inode number for '.', incorrect filetype (was 1, should be 2).

I have now fixed all the errors and now the datastore is visible in the GUI too. Thank you very much!!
At the moment the verficiation of all backups looks good. But I think I should reformat the disk, or what you think?

 

[fabian]

you could check past logs for issues - most likely given that it's an USB device it got unplugged while in use at some point.. checking it via smartctl might and watching out for errors would still be advisable.