[SOLVED] Incremental VM backups fail

[stsch]

Hey guys,

I'm having issues with backing up VMs using PBS. I'm trying to backup a pfSense VM, with the initial backup succeeding without errors. Then, when trying to do another manual backup, it fails. Here's the relevant info:

Log of initial backup in PVE:

INFO: starting new backup job: vzdump 100 --mode snapshot --remove 0 --node mabel --storage sb-backup2
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-03-02 11:28:55
INFO: status = running
INFO: VM Name: pfsense
INFO: include disk 'virtio0' 'thin0000:vm-100-disk-0' 50G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2022-03-02T10:28:54Z'
INFO: started backup task 'da11dbfa-2cf5-4c1f-9471-ca2eaaca9d30'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: created new
INFO:   0% (280.0 MiB of 50.0 GiB) in 3s, read: 93.3 MiB/s, write: 82.7 MiB/s
INFO:   1% (580.0 MiB of 50.0 GiB) in 6s, read: 100.0 MiB/s, write: 22.7 MiB/s
INFO:   2% (1.1 GiB of 50.0 GiB) in 14s, read: 63.5 MiB/s, write: 23.5 MiB/s
INFO:   3% (1.7 GiB of 50.0 GiB) in 24s, read: 67.6 MiB/s, write: 22.0 MiB/s
INFO:   4% (2.3 GiB of 50.0 GiB) in 33s, read: 63.6 MiB/s, write: 7.6 MiB/s
INFO:   5% (2.9 GiB of 50.0 GiB) in 36s, read: 196.0 MiB/s, write: 2.7 MiB/s
INFO:  24% (12.3 GiB of 50.0 GiB) in 39s, read: 3.2 GiB/s, write: 58.7 MiB/s
INFO:  25% (12.7 GiB of 50.0 GiB) in 42s, read: 130.7 MiB/s, write: 28.0 MiB/s
INFO:  26% (13.1 GiB of 50.0 GiB) in 45s, read: 126.7 MiB/s, write: 12.0 MiB/s
INFO:  33% (17.0 GiB of 50.0 GiB) in 48s, read: 1.3 GiB/s, write: 12.0 MiB/s
INFO:  50% (25.1 GiB of 50.0 GiB) in 51s, read: 2.7 GiB/s, write: 26.7 MiB/s
INFO:  84% (42.4 GiB of 50.0 GiB) in 54s, read: 5.8 GiB/s, write: 49.3 MiB/s
INFO:  90% (45.3 GiB of 50.0 GiB) in 57s, read: 1002.7 MiB/s, write: 21.3 MiB/s
INFO:  91% (45.6 GiB of 50.0 GiB) in 1m, read: 110.7 MiB/s, write: 101.3 MiB/s
INFO:  92% (46.0 GiB of 50.0 GiB) in 1m 13s, read: 30.5 MiB/s, write: 12.0 MiB/s
INFO:  93% (46.6 GiB of 50.0 GiB) in 1m 23s, read: 55.6 MiB/s, write: 11.6 MiB/s
INFO:  94% (47.0 GiB of 50.0 GiB) in 1m 27s, read: 115.0 MiB/s, write: 31.0 MiB/s
INFO: 100% (50.0 GiB of 50.0 GiB) in 1m 30s, read: 1021.3 MiB/s, write: 2.7 MiB/s
INFO: backup is sparse: 47.92 GiB (95%) total zero data
INFO: backup was done incrementally, reused 47.92 GiB (95%)
INFO: transferred 50.00 GiB in 91 seconds (562.6 MiB/s)
INFO: Finished Backup of VM 100 (00:01:33)
INFO: Backup finished at 2022-03-02 11:30:27
INFO: Backup job finished successfully
TASK OK

Log of failing backup in PVE:

INFO: starting new backup job: vzdump 100 --mode snapshot --remove 0 --node mabel --storage sb-backup2
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-03-02 11:31:53
INFO: status = running
INFO: VM Name: pfsense
INFO: include disk 'virtio0' 'thin0000:vm-100-disk-0' 50G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2022-03-02T10:31:53Z'
ERROR: VM 100 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: Failed at 2022-03-02 11:31:54
INFO: Backup job finished with errors
TASK ERROR: job errors

Before doing the backup, I created a fresh datastore and rebooted PVE, PBS and the host system. The datastore is a mounted CIFS share at the PBS. PBS itself is running as a VM inside PVE.

This is what `journal -xef` says, where you can see both backup tasks:

Mar 02 11:28:55 proxmox-backup proxmox-backup-proxy[648]: starting new backup on datastore 'sb-backup2': "vm/100/2022-03-02T10:28:54Z"
Mar 02 11:28:55 proxmox-backup proxmox-backup-proxy[648]: GET /previous: 400 Bad Request: no valid previous backup
Mar 02 11:28:55 proxmox-backup proxmox-backup-proxy[648]: created new fixed index 1 ("vm/100/2022-03-02T10:28:54Z/drive-virtio0.img.fidx")
Mar 02 11:28:56 proxmox-backup proxmox-backup-proxy[648]: add blob "/mnt/sb-backup2/vm/100/2022-03-02T10:28:54Z/qemu-server.conf.blob" (373 bytes, comp: 373)
Mar 02 11:29:08 proxmox-backup proxmox-backup-proxy[648]: error during snapshot file listing: 'unable to load blob '"/mnt/sb-backup2/vm/100/2022-03-02T10:28:54Z/index.json.blob"' - No such file or directory (os error 2)'
Mar 02 11:29:11 proxmox-backup proxmox-backup-proxy[648]: error during snapshot file listing: 'unable to load blob '"/mnt/sb-backup2/vm/100/2022-03-02T10:28:54Z/index.json.blob"' - No such file or directory (os error 2)'
Mar 02 11:29:27 proxmox-backup proxmox-backup-proxy[648]: error during snapshot file listing: 'unable to load blob '"/mnt/sb-backup2/vm/100/2022-03-02T10:28:54Z/index.json.blob"' - No such file or directory (os error 2)'
Mar 02 11:29:52 proxmox-backup proxmox-backup-proxy[648]: error during snapshot file listing: 'unable to load blob '"/mnt/sb-backup2/vm/100/2022-03-02T10:28:54Z/index.json.blob"' - No such file or directory (os error 2)'
Mar 02 11:29:55 proxmox-backup proxmox-backup-proxy[648]: error during snapshot file listing: 'unable to load blob '"/mnt/sb-backup2/vm/100/2022-03-02T10:28:54Z/index.json.blob"' - No such file or directory (os error 2)'
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: Upload statistics for 'drive-virtio0.img.fidx'
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: UUID: b77402a81d2b436cae8ba52c0ac3feb3
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: Checksum: a83976666c4ae5265bb208c0fafabc8a892fc98fc27b867db48152f8e64668a0
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: Size: 53687091200
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: Chunk count: 12800
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: Upload size: 2239758336 (4%)
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: Duplicates: 12266+0 (95%)
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: Compression: 36%
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: successfully closed fixed index 1
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: add blob "/mnt/sb-backup2/vm/100/2022-03-02T10:28:54Z/index.json.blob" (329 bytes, comp: 329)
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: successfully finished backup
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: backup finished successfully
Mar 02 11:30:26 proxmox-backup proxmox-backup-proxy[648]: TASK OK

Mar 02 11:30:27 proxmox-backup proxmox-backup-proxy[648]: Upload backup log to sb-backup2/vm/100/2022-03-02T10:28:54Z/client.log.blob
Mar 02 11:31:53 proxmox-backup proxmox-backup-proxy[648]: starting new backup on datastore 'sb-backup2': "vm/100/2022-03-02T10:31:53Z"
Mar 02 11:31:53 proxmox-backup proxmox-backup-proxy[648]: download 'index.json.blob' from previous backup.
Mar 02 11:31:53 proxmox-backup proxmox-backup-proxy[648]: backup ended and finish failed: backup ended but finished flag is not set.
Mar 02 11:31:53 proxmox-backup proxmox-backup-proxy[648]: removing unfinished backup
Mar 02 11:31:53 proxmox-backup proxmox-backup-proxy[648]: removing backup snapshot "/mnt/sb-backup2/vm/100/2022-03-02T10:31:53Z"
Mar 02 11:31:53 proxmox-backup proxmox-backup-proxy[648]: TASK ERROR: backup ended but finished flag is not set.

And here's the contents of the initial backup directory:

root@proxmox-backup:~# ls -la /mnt/sb-backup2/vm/100/2022-03-02T10\:28\:54Z/
total 37
drwxrwx--- 2 1000 1000      0 Mar  2 11:30 .
drwxrwx--- 2 1000 1000      0 Mar  2 11:31 ..
-rw-rw---- 1 1000 1000    839 Mar  2 11:30 client.log.blob
-rw-rw---- 1 1000 1000 413696 Mar  2 11:30 drive-virtio0.img.fidx
-rw-rw---- 1 1000 1000    403 Mar  2 11:30 index.json.blob
-rw-rw---- 1 1000 1000    373 Mar  2 11:28 qemu-server.conf.blob

I can backup LXC containers without issues. Any help is greatly appreciated.

 

[stsch]

Are there any places I might take a look at to find more information? Any logs or configs? I can't get it to work, and I have no clue where to find more info or what to do at all. :-(

 

[stsch]

Hey,

I'm still having the issue mentioned above. In the meantime, we have installed another Proxmox instance that also provides the PBS. Backing up to that PBS from the 2nd PVE works without issues, so the backup process in general works.

However, when I try to backup from my first PVE instance to the newly installed PBS machine, the same error occurs.

This leads me to the conclusion that there must be something wrong with the first PVE. Do you have any hints for how I can debug this?

 

[stsch]

Solution: Do not use ReiserFS as your /tmp.

 

[Neobin]

Solution: Do not use ReiserFS as your /tmp.

Glad to hear that you solved it.
Can you please mark the thread as "Solved" by editing your first post? Thanks in advance.

[0] https://forum.proxmox.com/threads/n...o-incremental-backup.78340/page-2#post-489996