Backup command fails at opening chunck store

[jlgarnier]

Dear Community,

I've recently noticed an error in the backup process of my Proxmox VE (v7.4-18) + BS server, following a power supply interruption (due to stormy weather). The exact message says

ERROR: Backup of VM 1000 failed - VM 1000 qmp command 'backup' failed - backup connect failed: command error: unable to open chunk store 'datastore1' at "/mnt/datastore/backups/datastore1/.chunks" - No such file or directory (os error 2)

When I parse the BS configuration, I can see that the "unused disk 0 = datastore1:1000/vm-1000-disk-0.qcow2, VM1000 being the BS VM itself.

When I try to have a look at the previous backups, the tab "Backups" displays an error message too: "error listing snapshots - 400 Bad Request (500)", which I don't understand.

Would anyone be kind enough to put me on the right track? I just don't where to start troubleshooting...

Thanks in advance for any help!

 

[Hannes Laimer]

Hey,

could you post the output of ls -lah /mnt/datastore/backups/datastore1?

 

[jlgarnier]

Hi Hannes,

Whether on the PVE or PBS node, I get cannot access '/mnt/datastore/backups/datastore1': No such file or directory... The datastore folder has just vanished.

Thanks for your help!

 

[jlgarnier]

Hi all,

I've launched the command pve7to8 to prepare for upgrade and got the following warning:

proxmox-backup-client failed: Error: unable to open chunk store 'datastore1' at "/mnt/datastore/backups/datastore1/.chunks" - No such file or directory (os error 2)
WARN: storage 'backups' enabled but not active!
PASS: storage 'data' enabled and active.
PASS: storage 'local' enabled and active.
PASS: storage 'local-lvm' enabled and active.
INFO: Checking storage content type configuration..
WARN: VM 1000 - volume 'datastore1:1000/vm-1000-disk-0.qcow2' (in config) - storage does not exist!
WARN: Proxmox VE enforces stricter content type checks since 7.0. The guests above might not work until the storage configuration is fixed.

where VM1000 = PBS.

I don't understand how storage 'backups' got deactivated and don't how to reactivate it. Any advice on this?

 

[Hannes Laimer]

I assume some device was mounted to /mnt/datastore/backups/datastore1, make sure it is again, because it looks like it is not.

 

[jlgarnier]

This is the point where the system stumbles on missing device: there's no /datastore in /mnt! I don't know why it's missing and how to mount it manually... Is there any command in the GUI that could force /mnt/datastore/backups/datastore1 to mount?

 

[Hannes Laimer]

What kind of storage do you have, is it a local disk, ZFS, NFS? How did you setup the Datastore?

 

[jlgarnier]

Here's the information on storage, but unfortunately I don't remember how the datastore was setup (this is a home server I setup 3 years ago and it ran perfectly till the storm)...

 

[Hannes Laimer]

No, I meant on the PBS. This is PVE that has the PBS configured as a backup target. Could you post the output of

ls -lah /etc/systemd/system/mnt-datastore*
# if this lists files, please also post their contents using `cat <file>`

lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,UUID,PARTUUID

# and
/etc/proxmox-backup/datastore.cfg

 

[jlgarnier]

Here's the output of the commands on the PBS server:

root@pbs:~# ls -lah /etc/systemd/system/mnt-datastore*
-rw-r--r-- 1 root root 240 Aug 19  2021 /etc/systemd/system/mnt-datastore-backups.mount

root@pbs:~# cat /etc/systemd/system/mnt-datastore-backups.mount
[Unit]
Description=Mount datatstore 'backups' under '/mnt/datastore/backups'

[Mount]
Options=defaults
Type=ext4
What=/dev/disk/by-uuid/b4d11d35-086a-489b-b5ac-15e1d0ce1f9d
Where=/mnt/datastore/backups

[Install]
WantedBy=multi-user.target


root@pbs:~# lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,UUID,PARTUUID
NAME           SIZE TYPE MOUNTPOINT UUID                                   PARTUUID
sda             32G disk
├─sda1        1007K part                                                   8f693b91-3092-4859-9b1f-3e6d23f07971
├─sda2         512M part            EA27-7129                              e5b0427e-23f2-4bf0-9343-8137fa8a7649
└─sda3        31.5G part            KYiEat-Kvas-BA80-CzBb-T2Fp-VfTs-oRyNBu fd00b5b2-ebe8-4123-9dba-5cecb604c8b4
  ├─pbs-swap   3.9G lvm  [SWAP]     643e1a88-c67c-400b-b289-2cde8241ea8f
  └─pbs-root  23.8G lvm  /          7dc549ca-a5b4-4613-9e1a-8342c4bb06b9
sr0          783.8M rom             2021-07-12-17-27-57-00

root@pbs:~# cat /etc/proxmox-backup/datastore.cfg
datastore: datastore1
        comment
        gc-schedule daily
        path /mnt/datastore/backups/datastore1

Hope this helps...

 

[Hannes Laimer]

Looks like you created the Datastore on a disk, and that disk is not present anymore. Either its connection was damaged, or the disk itself was.

What=/dev/disk/by-uuid/b4d11d35-086a-489b-b5ac-15e1d0ce1f9d
Where=/mnt/datastore/backups

in the .mount file tells us the partition with the UUID b4d11d35-086a-489b-b5ac-15e1d0ce1f9d should be mounted to /mnt/datastore/backups. And since the lsblk output does not contain a partition with this UUID the disk containing it is not connected.

Could you check if the disk is correctly plugged in? But since I assume you didn't unplug things, it is more like the disk (or connection) got damaged.

 

[jlgarnier]

Thanks for this analysis! A damaged disk is highly possible as the issue appeared after a power supply interruption! As far as I remember, the server uses an array of 1TB 2.5" disks working in RAID5 mode, so the damaged disk may have been rebuilt unless it's dead... I'll shut the serve down and open it soon.

By the way, all other services are working on this server: OMV, Nextcloud, a wiki, etc. This is surprising that only PBS would be impacted...

 

[jlgarnier]

Hi all!
Couldn't find time to shutdown and open the server... Is there any way to check the physical disks status from PVE interface?

 

[jlgarnier]

Hi all!
Problem solved! But I didn't do anything! Just upgraded the system some days ago, rebooted and bam! backup is working again!
I'll probably never know what was going on...
Thanks to all for your help and advice!