Backup does not finish

[upcnuwft]

Hell everyone,

I've tried to setup a backup job to a FreeNAS SMB share.
If I try to run the job manually, it will stay at the following state for 20 minutes and doesn't finish.

INFO: starting new backup job: vzdump --mailnotification always --node proxmox --all 1 --mailto xxx@xxx.xxx --compress zstd --quiet 1 --exclude 103 --mode snapshot --storage backup
INFO: Starting Backup of VM 111 (qemu)
INFO: Backup started at 2020-11-03 20:35:10
INFO: status = running
INFO: VM Name: xxx
INFO: include disk 'scsi0' 'images:vm-111-disk-0' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/backup/dump/vzdump-qemu-111-2020_11_03-20_35_10.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'ae3716d6-e5b6-4872-9c5b-ce6fc27640d7'
INFO: resuming VM again
INFO:   1% (627.1 MiB of 32.0 GiB) in  3s, read: 209.0 MiB/s, write: 139.3 MiB/s
INFO:   3% (1.2 GiB of 32.0 GiB) in  6s, read: 192.1 MiB/s, write: 177.3 MiB/s
INFO:   5% (1.8 GiB of 32.0 GiB) in  9s, read: 200.4 MiB/s, write: 180.8 MiB/s
INFO:  14% (4.7 GiB of 32.0 GiB) in 12s, read: 993.7 MiB/s, write: 26.3 MiB/s
INFO:  25% (8.2 GiB of 32.0 GiB) in 15s, read: 1.2 GiB/s, write: 1.5 MiB/s
INFO:  36% (11.8 GiB of 32.0 GiB) in 18s, read: 1.2 GiB/s, write: 753.3 KiB/s
INFO:  45% (14.7 GiB of 32.0 GiB) in 21s, read: 1001.3 MiB/s, write: 42.7 MiB/s
INFO:  55% (17.7 GiB of 32.0 GiB) in 24s, read: 1.0 GiB/s, write: 39.2 MiB/s
INFO:  63% (20.4 GiB of 32.0 GiB) in 27s, read: 925.0 MiB/s, write: 53.4 MiB/s
INFO:  70% (22.4 GiB of 32.0 GiB) in 30s, read: 675.3 MiB/s, write: 102.1 MiB/s
INFO:  72% (23.1 GiB of 32.0 GiB) in 33s, read: 215.1 MiB/s, write: 215.0 MiB/s
INFO:  74% (23.7 GiB of 32.0 GiB) in 36s, read: 223.5 MiB/s, write: 223.5 MiB/s
INFO:  76% (24.5 GiB of 32.0 GiB) in 39s, read: 252.5 MiB/s, write: 211.3 MiB/s
INFO:  78% (25.1 GiB of 32.0 GiB) in 42s, read: 209.0 MiB/s, write: 209.0 MiB/s
INFO:  83% (26.8 GiB of 32.0 GiB) in 45s, read: 607.0 MiB/s, write: 99.6 MiB/s
INFO:  94% (30.3 GiB of 32.0 GiB) in 48s, read: 1.1 GiB/s, write: 1.7 MiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 50s, read: 893.5 MiB/s, write: 4.4 MiB/s
INFO: backup is sparse: 26.94 GiB (84%) total zero data
INFO: transferred 32.00 GiB in 50 seconds (655.4 MiB/s)

It should backup all VMS but freezes at this state of the first one.
However it created a file:

root@proxmox:~# ls -l /mnt/pve/backup/dump/
total 1
drwxr-xr-x 2 root root          0 Nov  3 20:35 vzdump-qemu-111-2020_11_03-20_35_10.tmp
-rwxr-xr-x 1 root root 3707364686 Nov  3 20:35 vzdump-qemu-111-2020_11_03-20_35_10.vma.dat

I then have to manually kill the process and unfreeze the VM.

 

[dietmar]

Please can you post the VM config?

Is the bug reproducible? Does it work if you backup to another (local) storage?

 

[upcnuwft]

The backup job is now running since 19 hours.
I've killed it manually again...

Please can you post the VM config?

Is it this one you are asking for?

Spoiler: /etc/pve/qemu-server/111.conf

agent: 1
balloon: 1024
boot: c
bootdisk: scsi0
cores: 2
cpu: host
ide2: none,media=cdrom
memory: 2048
name: xxx
net0: virtio=4A:10:41:3A:24:D7,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
parent: Base_config
scsi0: images:vm-111-disk-0,discard=on,size=32G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=3db7cda4-8e5a-40f7-a05a-848d56c07023
sockets: 1
vmgenid: cc1a5b0a-3258-4f25-a2b7-3aa464731213

[Base_config]
agent: 1
balloon: 1024
boot: c
bootdisk: scsi0
cores: 2
cpu: host
ide2: none,media=cdrom
memory: 2048
name: xxx
net0: virtio=4A:10:41:3A:24:D7,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: images:vm-111-disk-0,discard=on,size=32G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=3db7cda4-8e5a-40f7-a05a-848d56c07023
snaptime: 1604226205
sockets: 1
vmgenid: cc1a5b0a-3258-4f25-a2b7-3aa464731213

Is the bug reproducible?

I started the process several times and always had the same issue.
Also did a reboot because I've installed some updates recently.

Does it work if you backup to another (local) storage?

Well... yes actually that works just fine.

 

[t.lamprecht]

Can you please also post pveversion -v output (also available in the UI on the nodes summary panel).

Is the FreeNAS its own physical storage box?

As it works to local storage, did you try to attach a NFS or other storage, to see if that works too.

 

[upcnuwft]

Spoiler: pveversion -v

proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-15 (running version: 6.2-15/48bd51b6)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.4-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-6
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-4
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-18
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

Is the FreeNAS its own physical storage box?

No. It is a VM on that proxmox host. But it is excluded from the backup job.
The VM has its own HBA with HDDs via PCIe passthrough.

As it works to local storage, did you try to attach a NFS or other storage, to see if that works too.

No I didn't try that. I wanted to stick with SMB/CIFS for all the shares.

 

[upcnuwft]

The problem was that I did assign the wrong group to the user accessing the share.
So he didn't have the correct permissions.
Totally my bad. Didn't notice this for like 10 times I checked the settings...

After I fixed that, it worked for several days/weeks like a charm.
Until I few days ago when I received the following message:

can't acquire lock '/var/run/vzdump.lock' - got timeout
VMID          STATUS          TIME          SIZE   FILENAME
  
Detailed backup logs:
vzdump --compress zstd --exclude 103 --storage TrueNAS_Backup --mailto xxx@xxx.xxx  --quiet 1 --mailnotification always --all 1 --mode snapshot

I then went to the Proxmox GUI and saw that the first VM that should be backed up is in a locked state.
And a backup job is still running since 2 days. (Hangs at the same state as mentioned in the first post)

I then killed the job manually and unfreezed the VM manually.

When I now run the Backup job manually, it completes successfully.
But on the next scheduled automatic backup the same error occurs.
I didn't change anything...


... any ideas?