backup register image failed: command error: no previous backup found, cannot do incremental backup

[layer7.net]

Hi,

the PBS server had to be reinstalled because of a filesystem corruption of the OS.

The datastore is backed by ZFS, is fine and was used in the new PBS again.

Now some VMs have issues generating new backups:

```
INFO: starting new backup job: vzdump 118 --remove 0 --mode snapshot --node n2 --storage backup01
INFO: Starting Backup of VM 118 (qemu)
INFO: Backup started at 2022-06-14 21:46:57
INFO: status = running
INFO: VM Name: webhost
INFO: include disk 'virtio0' 'storage01-mc1-fra1-isp4p:118/vm-118-disk-0.qcow2' 700G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/118/2022-06-14T19:46:57Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 118 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 118 failed - VM 118 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: Failed at 2022-06-14 21:46:57
INFO: Backup job finished with errors
TASK ERROR: job errors
```

As you can see on the pictures, in the PVE it will create a new entry with 1B

And on the PBS server it will have this entry and act like its working there something.

Is there a way to force a (new) full backup to overcome this issue?

 

[layer7.net]

This issue was "solved" by deleting in the PBS server the faulty entry PLUS deleting the entry before the faulty entry.

After this, a new backup was possible.

It would be nice, if there would be a button/way to enforce a new full backup and start this way a new reference point for all future increment backups.

That would increase the reliability of the data safetiness. The very first full backup is actually a SPOF. If it becomes corrupt, all following incremental backups will be useless.

But if there is a way to make every N backups a new full backup, then it would remove this SPOF.

Luckily in our case, removing the last (working) incremental backup did the trick. But who wants to rely on luck when its about backups...

 

[dcsapak]

That would increase the reliability of the data safetiness. The very first full backup is actually a SPOF. If it becomes corrupt, all following incremental backups will be useless.

But if there is a way to make every N backups a new full backup, then it would remove this SPOF.

thats not really true, since all backups are 'full' backups, the chunks are deduplicated, you can remove any single backup from the pbs and not corrupt the others...

as to the error message:

at this portion in the code, the qemu vm has a dirty bitmap, but for some reason the server says there is no previous backup. this could happen if the previous backups does not contain the same image (e.g. if multiple vms were backed up into the same group; which should never be done!) or it cannot be downloaded (disk error/permission error for example).

stopping & starting the vm could probably also helped, since then the dirty bitmap will be cleared and no incremental backup will be attempted

 

[layer7.net]

Hi Dominik,

thank you for your time.

But i am sorry, i think you are wrong here.

As far as i can see, you are using restic or something similar. That means there is ONE full copy at the beginning, and everything else is an incremental copy alias snapshot that is deduplicated.

As you can see in your PBS UI, there is one base, which is a full backup. Everything else is a snapshot alias incremental copy, alias NOT a full backup.

You can remove all snapshots, but you can not remove the 1st ( only full ) backup.

Please correct me if i am wrong ( i assume thats very important for anyone to clearify this and being 200% sure about how this REALLY works ).

Greetings
Oliver

 

[dcsapak]

Hi,

As far as i can see, you are using restic or something similar. That means there is ONE full copy at the beginning, and everything else is an incremental copy alias snapshot that is deduplicated.

As you can see in your PBS UI, there is one base, which is a full backup. Everything else is a snapshot alias incremental copy, alias NOT a full backup.

You can remove all snapshots, but you can not remove the 1st ( only full ) backup.

This is not correct. We're not using restic but our own implementation, though it is similar to what restic does.
Here is a high level explanation:

The data is chopped up in chunks, identified by a checksum of the content. Each backup references the chunks it needs. So the first backup will backup all the chunks and subsequent backups will only backup the chunks that are needed,
but since all backups reference the chunks they need, you can remove any backup (even the first), since only the metadata is removed. The real data is removed when you run a garbage collect.

Here is a more in-depth explanation:
https://pbs.proxmox.com/docs/technical-overview.html

 

[Jarvar]

Hi,

This is not correct. We're not using restic but our own implementation, though it is similar to what restic does.
Here is a high level explanation:

The data is chopped up in chunks, identified by a checksum of the content. Each backup references the chunks it needs. So the first backup will backup all the chunks and subsequent backups will only backup the chunks that are needed,
but since all backups reference the chunks they need, you can remove any backup (even the first), since only the metadata is removed. The real data is removed when you run a garbage collect.

Here is a more in-depth explanation:
https://pbs.proxmox.com/docs/technical-overview.html

Would using something like restic be redundant to backup PBS datastore somewhere else? I like restic for it's ability to encrypt, chunk and resume. Rclone could doesn't usually have the ability to resume files if the transfer isn't complete.
Any recommendations? This is an extra fail safe.
I'm also wanting to reinstall the PBS but without losing data so I can restore it from another storage location like a cloud provider or offsite HDD.
Thank you.

 

[dcsapak]

Would using something like restic be redundant to backup PBS datastore somewhere else? I like restic for it's ability to encrypt, chunk and resume. Rclone could doesn't usually have the ability to resume files if the transfer isn't complete.
Any recommendations? This is an extra fail safe.
I'm also wanting to reinstall the PBS but without losing data so I can restore it from another storage location like a cloud provider or offsite HDD.
Thank you.

id probably rather use the built-in sync feature to sync to a different pbs. advantage is that the remote datastore is then consistent, which you can't know when you're just syncing the underlying data (there is locking between the different operations going on, so e.g. it'll make sure that chunks don't get deleted during gc as long as someone read the backup, etc. )

for reinstall you can then either sync back, or if you did not wipe the datastore disks, mount them and add the config manually in the /etc/proxmox-backup/datastore.cfg file