Issue with missing chunks on large backups.

[dcsapak]

In my case it looks like that if task is taking longer then 24h and GC was run before backup is finished, GC cleans chunks from running backup that are older than 24h which ultimatly result in failed backup when i try to verify or use it.

that explicitly should not happen, since we use the oldest writer start time as cutoff then, or can you reproduce that behaviour (i know, letting a backup run for over 24 hours is tedious... )
but i'll check if there's a bug somewhere

 

[Dariusz Bilewicz]

When i disabled GC i've managed to create successfully HOST backups with PBS Client on CentOS that are usable. I didn't test it on VM backup but scenario was the same for VM and HOST backups with GC enabled. In both cases verify was failed and i wasn't able to access files inside backups with GC running every 24h and only for backups taking longer then 24h.

 

[dcsapak]

mhmm.. can you post your versions ('proxmox-backup-manager versions --verbose') ?
i'll see if i can reproduce that here...

edit: also did you restart/reload the proxmox-backup-proxy inbetween ?

 

[AnNH]

Hi @dcsapak ,

Is there any way for us to restore backup from the VMID that was missing .chunks?

PBS Version: proxmox-backup-server 2.2.1-1 running version: 2.2.1

Thanks and Regards !

 

[Dariusz Bilewicz]

mhmm.. can you post your versions ('proxmox-backup-manager versions --verbose') ?
i'll see if i can reproduce that here...

edit: also did you restart/reload the proxmox-backup-proxy inbetween ?

I didn't restart any of services, they didn't crash either.

● proxmox-backup-proxy.service - Proxmox Backup API Proxy Server
Loaded: loaded (/lib/systemd/system/proxmox-backup-proxy.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-05-08 01:40:14 CEST; 3 weeks 4 days ago

# proxmox-backup-manager versions --verbose
proxmox-backup 2.2-1 running kernel: 5.13.19-1-pve
proxmox-backup-server 2.2.1-1 running version: 2.1.1
pve-kernel-5.13 7.1-9
pve-kernel-helper 7.2-3
pve-kernel-5.13.19-1-pve 5.13.19-3
ifupdown2 3.1.0-1+pmx3
libjs-extjs 7.0.0-1
proxmox-backup-docs 2.2.1-1
proxmox-backup-client 2.2.1-1
proxmox-mini-journalreader 1.2-1
proxmox-widget-toolkit 3.5.1
pve-xtermjs 4.16.0-1
smartmontools 7.2-pve3
zfsutils-linux 2.1.4-pve1


$ proxmox-backup-client version
client version: 2.1.2

My PBS client it it's any help was installed from https://github.com/sg4r/proxmox-bac.../v2.1.2/proxmox-backup-2.1.2-1.x86_64.el7.rpm

 

[dcsapak]

ok, we looked a bit deeper and found some issues that are now fixed in git and on pbstest (proxmox-backup-server 2.2.3 contains the fixes) but it's only really fixed after all old server instances are gone (so if there are still running backup tasks, it's only really fixed when that task is done).

 

[Ronny Aasen]

is there a way to quick and easy delete all backups that have missing chunks ?
GC fail with non-exsisting chunks all the time. and the vm's are unrecoverable anyways.

 

[dcsapak]

i'd delete all snapshots with a failed verification, but no there is no built-in way to do that automatically
(you can probably use the api/cli for that though, get the list of snapshots, filter by verified status, delete)

 

[aydoganatakan]

did your pbs crash or hard stop (so no proper shutdown) at any time in the past? alternatively, does the datastore live on a network share?
these are currently the only possible explanations we have so far (aside from someone manually deleting chunks)

Hi,

I apologize for replying to a topic that has taken a long time.
However I am also getting this error and experienced what you said.
Unfortunately, PBS has been closed several times due to power outages.

What is your suggestion?

 

[Ronny Aasen]

we did end up deleting all snapshots with failed verification, using a combination of oneliners and manually.
unfortunate the one liner is lost in the bash history.