No previous backup found, cannot do incremental backup

ScIT

Active Member
Apr 13, 2019
22
0
41
32
Hi

First of all, thanks for that awesome proxmox backup server. I currently do my first steps with it and got the most things working properly.

What I noticed is, that the first backup always went trough properly, as soon as I now want do to a second backup of the machine, only lxc is working properly:

Code:
INFO: starting new backup job: vzdump --storage xxx --mode snapshot --mailnotification failure --all 1 --mailto xxx --quiet 1
INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2020-11-01 00:00:02
INFO: status = running
INFO: CT Name: xxx
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
WARNING: You have not turned on protection against thin pools running out of space.
WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
Logical volume "snap_vm-100-disk-0_vzdump" created.
WARNING: Sum of all thin volume sizes (600.00 GiB) exceeds the size of thin pool vg0/data and the size of whole volume group (<475.81 GiB).
INFO: creating Proxmox Backup Server archive 'ct/100/2020-10-31T23:00:02Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp2415_100/etc/vzdump/pct.conf fw.conf:/var/tmp/vzdumptmp2415_100/etc/vzdump/pct.fw root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 100 --backup-time 1604185202 --repository xxx
INFO: Starting backup: ct/100/2020-10-31T23:00:02Z
INFO: Client name: xxx
INFO: Starting backup protocol: Sun Nov 1 00:00:02 2020
INFO: Upload config file '/var/tmp/vzdumptmp2415_100/etc/vzdump/pct.conf' to 'root@pam@xxx' as pct.conf.blob
INFO: Upload config file '/var/tmp/vzdumptmp2415_100/etc/vzdump/pct.fw' to 'root@pam@xxx' as fw.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@xxx' as root.pxar.didx
INFO: root.pxar: had to upload 36.55 GiB of 37.42 GiB in 292.49s, average speed 127.95 MiB/s).
INFO: root.pxar: backup was done incrementally, reused 895.41 MiB (2.3%)
INFO: Uploaded backup catalog (19.04 MiB)
INFO: Duration: 292.98s
INFO: End Time: Sun Nov 1 00:04:55 2020
INFO: running 'proxmox-backup-client prune' for 'ct/100'
INFO: remove vzdump snapshot
Logical volume "snap_vm-100-disk-0_vzdump" successfully removed
INFO: Finished Backup of VM 100 (00:04:56)
INFO: Backup finished at 2020-11-01 00:04:58

But as soon as I want to backup a VM, it fails with "ERROR: VM 101 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup":

Code:
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-11-01 00:04:58
INFO: status = running
INFO: VM Name: xxx
INFO: include disk 'scsi0' 'data:vm-101-disk-0' 120G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2020-10-31T23:04:58Z'
ERROR: VM 101 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: Failed at 2020-11-01 00:04:58
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2020-11-01 00:04:58
INFO: status = running
INFO: VM Name: xxx
INFO: include disk 'virtio0' 'data:vm-102-disk-0' 160G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/102/2020-10-31T23:04:58Z'
ERROR: VM 102 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
ERROR: Backup of VM 102 failed - VM 102 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: Failed at 2020-11-01 00:04:59
INFO: Backup job finished with errors

It looks like I keep missing some small but important part about the vm backup, hope you can enlight me :).
 
Hmm, just to update: Sometimes it works, sometimes not - just started manualy a backup of the vm, which worked. When I trigger the same over Datacenter -> Backup -> Run Job with the same vmids, one is working, the other not. Maybe you could give me an idea, where I can start debuging :).
 
hmm, that should not be a fatal error but just a warning/information message.. could you post the output of pveversion -v? does the problem go away if you stop a VM, and start it again?
 
Good Morning Fabian

First of all sorry, I could have shared the version informations already in the first post...

First standalone host:

Code:
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-14 (running version: 6.2-14/2001b502)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.13-1-pve: 5.3.13-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.4-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-6
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-17
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

A second host, little bit older, but with the same issue:

Code:
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.13-1-pve: 5.3.13-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-1
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

I can confirm, that when I halt the machine, then boot it again - the backup works properly once. This would also explain my "foundings" of the last post, that it works "sometimes" - because I enabled qemu agent for all vms.

Can I provide you anything else, which may help to solve the issue?
 
if it works for all recently started VMs, then it's probably an issue with your running VMs using an outdated version of the proxmox-backup library, and the issue is already fixed.
 
It works just the first time: I halt the machine, start it again, run backup job once - as soon as I run it a second time, the job fails again.

Also I noticed that the error reporting doesnt work properly: I set the e-mail notification to "on failure", as soon as one backup worked (in my case lxc), I don't get a notification - when all backups are failing (2 vms), I get the notification.

The proxmox host is up to date, installed the last available patches yesterday, also updated the proxmox backup server a few minutes ago - the issue is still the same.
 
Hi @fabian, aswell as @Stefan_R

The issue is still pending, the vm backups are working only when I "halt" the machines once. I use a clean debian 10 installation with thin-lvm, on top proxmox - same for proxmox backup server. All system are on the current state.

Is there any advice you could help me out? Would love to take a incremental backup instead full backups the whole time :).
 
I cannot reproduce the issue here (tested on lvm-thin, Debian VM, multiple drives, multiple (incremental) backups)...

First, which version of libproxmox-backup-qemu0 do you have installed? (i.e. apt show libproxmox-backup-qemu0, I also just sent a patch to include that in pveversion :) )

The issue you are seeing should not happen with VMs started using the latest version of that and our QEMU package, as we check for that kind of error beforehand. Do you do anything weird like hotplugging drives or the like? What does your VM config (/etc/pve/qemu-server/<vmid>.conf) look like? Could you post the full log of a working backup as well (so the first one if I understand correctly)?

You said the issue only happens when you trigger it twice and via the "Backup Job" GUI. Does it also happen if you trigger a backup twice manually?
 
Hi @Stefan_R

Thanks for the amazing and detailed answer, below the requested informations:

All 4 hosts have the same version of the libproxmox-backup-qemu0 package.
Code:
root@pm101 ~ # apt show libproxmox-backup-qemu0
Package: libproxmox-backup-qemu0
Version: 0.7.1-1
Priority: optional
Section: admin
Source: proxmox-backup-qemu
Maintainer: Proxmox Support Team <support@proxmox.com>
Installed-Size: 6,339 kB
Depends: libacl1 (>= 2.2.23), libapt-pkg5.0 (>= 0.8.0), libc6 (>= 2.25), libfuse3-3 (>= 3.2.3), libgcc1 (>= 1:4.2), libpam0g (>= 0.99.7.1), libssl1.1 (>= 1.1.0), libstdc++6 (>= 4.1.1), libsystemd0, libudev1 (>= 183), libuuid1 (>= 2.16), libzstd1 (>= 1.3.2)
Homepage: https://www.proxmox.com
Download-Size: 1,594 kB
APT-Sources: http://download.proxmox.com/debian/pve buster/pve-no-subscription amd64 Packages
Description: Proxmox Backup Server client library for QEMU
 This library contains the library to access the Proxmox Backup server from
 within QEMU.

N: There are 12 additional records. Please use the '-a' switch to see them.

Also maybe a bit more informations to the hosts itself, they are mostly Hetzner AX41, installed with their Debian 10 minimal image on a lvm (with a additional thin-lvm for the vm data).
our QEMU package
Can you define what you mean with "our" QEMU package? Currently I use qemu-guest-agent from the debian 10 repository and the windows package from the virtio-win driver iso.

Do you do anything weird like hotplugging drives or the like?
Currently do not have any hotplug drives or other devices.

What does your VM config (/etc/pve/qemu-server/<vmid>.conf) look like?
Code:
root@pm102 ~ # cat /etc/pve/qemu-server/103.conf
agent: 1
bootdisk: scsi0
cores: 8
ide2: none,media=cdrom
memory: 12288
name: web01.xxx.xx
net0: virtio=FE:17:14:34:2C:3F,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: data:vm-103-disk-0,discard=on,size=120G
scsihw: virtio-scsi-pci
smbios1: uuid=63672e20-2fe6-4c2e-b466-62a9d110d7a1
sockets: 1
vmgenid: 9267bd81-a7cd-4040-871d-5bc815e27f45

Code:
root@pm102 ~ # cat /etc/pve/qemu-server/104.conf
agent: 1
bootdisk: scsi0
cores: 6
ide2: none,media=cdrom
memory: 12288
name: pbx001.xx
net0: virtio=26:B7:A9:50:BC:C9,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: data:vm-104-disk-0,discard=on,size=120G
scsihw: virtio-scsi-pci
smbios1: uuid=caf8c880-a4b5-41a6-9137-dd8debee3917
sockets: 1
vmgenid: 38675625-ee4b-456b-b2c8-962a56a23e5a

Could you post the full log of a working backup as well (so the first one if I understand correctly)?
Sure, just did it on a unimportant vm, started the backup once - full log:
Code:
()
INFO: starting new backup job: vzdump 100 --node pm102 --mode snapshot --remove 0 --storage pb101
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2020-11-05 10:46:27
INFO: status = running
INFO: VM Name: sip001.xxx.xx
INFO: include disk 'scsi0' 'data:vm-100-disk-0' 120G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2020-11-05T09:46:27Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '6b7b96ae-1cdd-4b02-8822-bd4762cf50a4'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: created new
INFO:   0% (608.0 MiB of 120.0 GiB) in  3s, read: 202.7 MiB/s, write: 178.7 MiB/s
INFO:   1% (1.2 GiB of 120.0 GiB) in  7s, read: 157.0 MiB/s, write: 156.0 MiB/s
INFO:   2% (2.5 GiB of 120.0 GiB) in 16s, read: 142.7 MiB/s, write: 138.7 MiB/s
INFO:   3% (3.7 GiB of 120.0 GiB) in 25s, read: 136.9 MiB/s, write: 136.9 MiB/s
INFO:   4% (4.8 GiB of 120.0 GiB) in 31s, read: 200.7 MiB/s, write: 195.3 MiB/s
INFO:   5% (6.0 GiB of 120.0 GiB) in 37s, read: 202.0 MiB/s, write: 200.7 MiB/s
INFO:   6% (7.3 GiB of 120.0 GiB) in 43s, read: 215.3 MiB/s, write: 210.7 MiB/s
xxx
INFO:  70% (84.4 GiB of 120.0 GiB) in  4m 51s, read: 403.0 MiB/s, write: 131.0 MiB/s
INFO:  78% (94.3 GiB of 120.0 GiB) in  4m 54s, read: 3.3 GiB/s, write: 84.0 MiB/s
INFO:  94% (114.0 GiB of 120.0 GiB) in  4m 57s, read: 6.6 GiB/s, write: 10.7 MiB/s
INFO: 100% (120.0 GiB of 120.0 GiB) in  4m 59s, read: 3.0 GiB/s, write: 46.0 MiB/s
INFO: backup is sparse: 75.94 GiB (63%) total zero data
INFO: backup was done incrementally, reused 75.99 GiB (63%)
INFO: transferred 120.00 GiB in 299 seconds (411.0 MiB/s)
INFO: Finished Backup of VM 100 (00:05:00)
INFO: Backup finished at 2020-11-05 10:51:27
INFO: Backup job finished successfully
TASK OK

When I now directly try to run a manual backup again, I get the following error output:
Code:
INFO: starting new backup job: vzdump 100 --node pm102 --storage pb101 --remove 0 --mode snapshot
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2020-11-05 10:52:43
INFO: status = running
INFO: VM Name: sip001.pbx.scit.ch
INFO: include disk 'scsi0' 'data:vm-100-disk-0' 120G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2020-11-05T09:52:43Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 100 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: Failed at 2020-11-05 10:52:44
INFO: Backup job finished with errors
TASK ERROR: job errors

When I shutdown/halt the machine and start it freshly, the backup is working again (manualy triggered):
Code:
INFO: starting new backup job: vzdump 100 --node pm102 --storage pb101 --remove 0 --mode snapshot
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2020-11-05 10:54:37
INFO: status = running
INFO: VM Name: sip001.xxx.xx
INFO: include disk 'scsi0' 'data:vm-100-disk-0' 120G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2020-11-05T09:54:37Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'd5227256-062e-4f69-983e-23567ca8cd81'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: created new
INFO:   0% (588.0 MiB of 120.0 GiB) in  3s, read: 196.0 MiB/s, write: 172.0 MiB/s
INFO:   1% (1.2 GiB of 120.0 GiB) in  7s, read: 164.0 MiB/s, write: 163.0 MiB/s
INFO:   2% (2.4 GiB of 120.0 GiB) in 15s, read: 155.5 MiB/s, write: 151.0 MiB/s
INFO:   3% (3.7 GiB of 120.0 GiB) in 24s, read: 147.6 MiB/s, write: 147.6 MiB/s
INFO:   4% (4.9 GiB of 120.0 GiB) in 30s, read: 198.0 MiB/s, write: 192.7 MiB/s
INFO:   5% (6.1 GiB of 120.0 GiB) in 36s, read: 202.0 MiB/s, write: 200.7 MiB/s
xxx
INFO:  40% (48.1 GiB of 120.0 GiB) in  4m  8s, read: 240.0 MiB/s, write: 239.2 MiB/s
INFO:  41% (49.3 GiB of 120.0 GiB) in  4m 13s, read: 241.6 MiB/s, write: 216.8 MiB/s
INFO:  47% (56.8 GiB of 120.0 GiB) in  4m 16s, read: 2.5 GiB/s, write: 177.3 MiB/s
INFO:  55% (67.1 GiB of 120.0 GiB) in  4m 19s, read: 3.5 GiB/s, write: 121.3 MiB/s
INFO:  63% (76.4 GiB of 120.0 GiB) in  4m 22s, read: 3.1 GiB/s, write: 114.7 MiB/s
INFO:  68% (82.5 GiB of 120.0 GiB) in  4m 25s, read: 2.0 GiB/s, write: 128.0 MiB/s
INFO:  69% (82.9 GiB of 120.0 GiB) in  4m 28s, read: 148.0 MiB/s, write: 148.0 MiB/s
INFO:  70% (84.5 GiB of 120.0 GiB) in  4m 32s, read: 409.0 MiB/s, write: 137.0 MiB/s
INFO:  82% (99.6 GiB of 120.0 GiB) in  4m 35s, read: 5.0 GiB/s, write: 46.7 MiB/s
INFO:  97% (116.9 GiB of 120.0 GiB) in  4m 38s, read: 5.8 GiB/s, write: 41.3 MiB/s
INFO: 100% (120.0 GiB of 120.0 GiB) in  4m 39s, read: 3.1 GiB/s, write: 0 B/s
INFO: backup is sparse: 75.94 GiB (63%) total zero data
INFO: backup was done incrementally, reused 76.00 GiB (63%)
INFO: transferred 120.00 GiB in 279 seconds (440.4 MiB/s)
INFO: Finished Backup of VM 100 (00:04:40)
INFO: Backup finished at 2020-11-05 10:59:17
INFO: Backup job finished successfully
TASK OK

And a screenshot of the related datastore in the proxmox backup server:
1604570449091.png

Does it also happen if you trigger a backup twice manually?
As explained above, yes. It doesnt matter if I trigger it over auto backup or manualy.

Thanks for your help and sorry for the long post :).
 
PS: If you like, I could also give you access to an host. I got a empty one which I could prepare with some vms to backup for reproducing the issue.
 
All 4 hosts have the same version of the libproxmox-backup-qemu0 package.
Sorry I believe that command was wrong, it shows the newest known, not the installed version. Try apt list --installed | grep 'libproxmox-backup-qemu0'. Still, if you say you updated everything that shouldn't be an issue. Though just to note it again, the PBS library and QEMU will only be reloaded after a full shutdown of the VM, 'reboot' from inside the VM is not enough.

BTW: Is your PBS install up-to-date as well? (apt list --installed | grep proxmox on there, we don't have a 'pbsversion' yet)

Can you define what you mean with "our" QEMU package? Currently I use qemu-guest-agent from the debian 10 repository and the windows package from the virtio-win driver iso.
I meant the version of QEMU installed on your PVE host, 'pve-qemu-kvm', which contains the patches necessary to work with PBS. The point I was trying to make is that I don't see a codepath leading to the error message you are getting, it is mostly a remnant from earlier versions. The newest version should check for the case the message handles earlier and report something else (well, it shouldn't report anything, it should just work of course ;) ).

Instead of giving us access, could you maybe attempt to design some sort of reproducer? I.e. install a new system in a VM or on a lab machine and see if you can get the error to occur there as well, take the minimal route and describe how you got there.
 
No problem, I'll grab all relevant informations and try to provide a reproduceable step by step guide. Give me time until tomorrow :).
 
Good Morning @Stefan_R

Just to inform, all system have the libproxmox-backup-qemu0 package with version 0.7.0-1 installed, there is a new version 0.7.1-1 available, I'll install them as soon as we get out of productive usage time of the systems.

Also there are a few new pbs packages, which I'll update aswell soon:
Code:
proxmox-backup-client/stable 0.9.6-1 amd64 [upgradable from: 0.9.4-1]
proxmox-backup-docs/stable 0.9.6-1 all [upgradable from: 0.9.4-1]
proxmox-backup-server/stable 0.9.6-1 amd64 [upgradable from: 0.9.4-2]
proxmox-widget-toolkit/stable 2.3-8 all [upgradable from: 2.3-6]

Preparing now docs and the demo server for reproducable steps about the issue.
 
Last edited:
Ok, so let's provide some docs :).

Details of the proxmox standalone host: Hetzner AX41-NVME (https://www.hetzner.com/dedicated-rootserver/ax41-nvme), Debian 10 minimal image
Details of the proxmox backup server host: Hetzner SX62 (https://www.hetzner.com/dedicated-rootserver/sx62), Debian 10 minimal image

Proxmox Standalone Host LVM Setup:
1604647818998.png

Then proceed with the proxmox installation according to the docs (https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Buster):
Code:
echo "deb http://download.proxmox.com/debian/pve buster pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg
apt update && apt dist-upgrade -y
apt install proxmox-ve postfix open-iscsi -y
reboot

After installation, I create directly a thin-lvm data store:
Code:
lvcreate -L 100G -n data vg0
lvconvert --type thin-pool vg0/data
lvresize --poolmetadatasize +924M vg0/data
lvextend -l +100%FREE vg0/data

Then add the datastore to proxmox, aswell as create the pbs storage:
1604648351414.png
1604648733306.png

Setup a test vm (plain debian 10), do first backup:
Code:
INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --storage pb101 --node pm104
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-11-06 08:22:48
INFO: status = running
INFO: VM Name: pbx101.xxx.xx
INFO: include disk 'scsi0' 'data:vm-101-disk-0' 160G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2020-11-06T07:22:48Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'c21ca19d-c5e3-489f-aea2-b301a625b400'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: created new
INFO:   0% (916.0 MiB of 160.0 GiB) in  3s, read: 305.3 MiB/s, write: 264.0 MiB/s
INFO:   1% (1.7 GiB of 160.0 GiB) in  6s, read: 258.7 MiB/s, write: 257.3 MiB/s
xxx
INFO:  84% (135.4 GiB of 160.0 GiB) in 39s, read: 3.9 GiB/s, write: 12.0 MiB/s
INFO:  93% (149.1 GiB of 160.0 GiB) in 42s, read: 4.6 GiB/s, write: 2.7 MiB/s
INFO: 100% (160.0 GiB of 160.0 GiB) in 45s, read: 3.6 GiB/s, write: 2.7 MiB/s
INFO: backup is sparse: 156.76 GiB (97%) total zero data
INFO: backup was done incrementally, reused 156.96 GiB (98%)
INFO: transferred 160.00 GiB in 45 seconds (3.6 GiB/s)
INFO: Finished Backup of VM 101 (00:00:45)
INFO: Backup finished at 2020-11-06 08:23:33
INFO: Backup job finished successfully
TASK OK

Do a second backup, which will fail again:
Code:
INFO: starting new backup job: vzdump 101 --node pm104 --mode snapshot --storage pb101 --remove 0
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-11-06 08:24:25
INFO: status = running
INFO: VM Name: pbx101.xxx.xx
INFO: include disk 'scsi0' 'data:vm-101-disk-0' 160G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2020-11-06T07:24:25Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 101 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: aborting backup job
ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: Failed at 2020-11-06 08:24:25
INFO: Backup job finished with errors
TASK ERROR: job errors

VM Configuration file:
Code:
root@pm104 ~ # cat /etc/pve/qemu-server/101.conf
agent: 1
bootdisk: scsi0
cores: 6
ide2: iso:iso/debian-10.1.0-amd64-netinst.iso,media=cdrom
memory: 8192
name: pbx101.xxx.xx
net0: virtio=DE:44:AB:43:26:08,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: data:vm-101-disk-0,discard=on,size=160G
scsihw: virtio-scsi-pci
smbios1: uuid=f4dd5642-359d-4e33-8e74-9dc9ecf520ee
sockets: 1
vmgenid: 4e32b77e-68f0-4f03-872a-97ed3421d7d8

Both systems are now completly up to date and freshly rebooted.

Proxmox Standalone Host
Code:
root@pm104 ~ # pveversion -v
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-15 (running version: 6.2-15/48bd51b6)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-4
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.6-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-8
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-6
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-19
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

Code:
root@pm104 ~ # apt list --installed | grep 'libproxmox-backup-qemu0'
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
libproxmox-backup-qemu0/stable,now 0.7.1-1 amd64 [installed,automatic]

Proxmox Backup Server Host
Code:
root@pm104 ~ # apt list --installed | grep 'proxmox'

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libproxmox-acme-perl/stable,now 1.0.5 all [installed,automatic]
libproxmox-backup-qemu0/stable,now 0.7.1-1 amd64 [installed,automatic]
proxmox-archive-keyring/stable,now 1.0 all [installed,automatic]
proxmox-backup-client/stable,now 0.9.6-1 amd64 [installed,automatic]
proxmox-mini-journalreader/stable,now 1.1-1 amd64 [installed,automatic]
proxmox-ve/stable,now 6.2-2 all [installed]
proxmox-widget-toolkit/stable,now 2.3-8 all [installed,automatic]
 
Thanks for the detailed info! I followed your setup as closely as possible, but still no reproducer.

Brand new PBS instance, brand new PVE instance, both from debian buster via our guides, configured the LVM-thin the same way, the config of the VM to back up looks almost identical (even used the same 10.1.0 guest image)... And still, backing up works just fine, as often as I like in a row.

Is your backup server really up to date? (The output you posted as "Proxmox Backup Server Host" appears to be from the same host as your PVE 'pm104' and also doesn't contain the proxmox-backup-server package) Maybe try setting up a new PBS instance too (can run in a VM just as well, for testing) and see if backing up to there works fine.
 
Yes, you're right, apt was from the wrong host. PBS was up to date, but you already shipped a new package :):

Code:
root@pb101 ~ # apt list --installed | grep 'proxmox'

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

proxmox-archive-keyring/stable,now 1.0 all [installed,automatic]
proxmox-backup-client/stable,now 0.9.6-1 amd64 [installed,upgradable to: 0.9.7-1]
proxmox-backup-docs/stable,now 0.9.6-1 all [installed,upgradable to: 0.9.7-1]
proxmox-backup-server/stable,now 0.9.6-1 amd64 [installed,upgradable to: 0.9.7-1]
proxmox-backup/stable,now 1.0-4 all [installed]
proxmox-mini-journalreader/stable,now 1.1-1 amd64 [installed,automatic]
proxmox-widget-toolkit/stable,now 2.3-8 all [installed,automatic]

I'll try to run pbs in a vm and let you know the feedback.
 
Freshly installed a debian 10 on the same host in a vm, added two disks, added the second disk as lvm with ext4 fs and mounted it to /backup. Then runned the backup twice and got the same issue again :(.

First backup:
Code:
INFO: starting new backup job: vzdump 101 --node pm104 --mode snapshot --storage pb102 --remove 0
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-11-09 14:20:34
INFO: status = running
INFO: VM Name: pbx101.xxx.xx
INFO: include disk 'scsi0' 'data:vm-101-disk-0' 160G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2020-11-09T13:20:34Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'b588e693-372a-408c-9e1e-ef8dfc27e974'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: created new
INFO:   0% (828.0 MiB of 160.0 GiB) in  3s, read: 276.0 MiB/s, write: 234.7 MiB/s
INFO:   1% (1.6 GiB of 160.0 GiB) in  6s, read: 274.7 MiB/s, write: 274.7 MiB/s
INFO:   2% (3.3 GiB of 160.0 GiB) in 13s, read: 246.9 MiB/s, write: 226.9 MiB/s
...
INFO: 100% (160.0 GiB of 160.0 GiB) in  1m  1s, read: 5.0 GiB/s, write: 4.0 MiB/s
INFO: backup is sparse: 152.47 GiB (95%) total zero data
INFO: backup was done incrementally, reused 152.67 GiB (95%)
INFO: transferred 160.00 GiB in 61 seconds (2.6 GiB/s)
INFO: Finished Backup of VM 101 (00:01:02)
INFO: Backup finished at 2020-11-09 14:21:36
INFO: Backup job finished successfully
TASK OK

Code:
INFO: starting new backup job: vzdump 101 --node pm104 --storage pb102 --mode snapshot --remove 0
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-11-09 14:21:52
INFO: status = running
INFO: VM Name: pbx101.xxx.xx
INFO: include disk 'scsi0' 'data:vm-101-disk-0' 160G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2020-11-09T13:21:52Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 101 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: aborting backup job
ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: Failed at 2020-11-09 14:21:52
INFO: Backup job finished with errors
TASK ERROR: job errors

backup overview on the vm
1604929218624.png

The issue is reporduceable for me on 4 proxmox hosts, 2 backup target servers - do you have any hint, what I could test additionaly?
 
Last edited:
@Stefan_R I would also love to pay to resolve the project, just let me know what's the best way to get in contact :).
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!