Proxmox VE 6.4-13 Backup problem

[h9lk]

Hello everyone,

Faced such problem - paths to disks and general any symbols disappeared in backup task, backup is simply created incomprehensibly and is written to nowhere, Proxmox VE version has been updated to the latest version. /etc/pve/vzdump.cron & /etc/vzdump.conf files didn't change.
now it looks like this:

INFO starting new backup job vzdump 115 --node server --storage sata2 --compress lzo --mode suspend --remove 0
INFO Starting Backup of VM 115 (qemu)
INFO Backup started at 2021-10-09 054355
INFO status = running
INFO backup mode suspend
INFO ionice priority 7
INFO VM Name sqlserv
INFO include disk 'sata0' 'local-lvmvm-115-disk-0' 800G
INFO include disk 'sata1' 'sata2115vm-115-disk-2.qcow2' 1000G
INFO include disk 'sata2' 'sata2115vm-115-disk-3.qcow2' 1000G
INFO suspending guest
INFO creating vzdump archive 'varlibSATA2dumpvzdump-qemu-115-2021_10_09-05_43_55.vma.lzo'
INFO started backup task '4ef32521-f5db-49ce-aa91-7d5b6b2fa6f1'
INFO resuming VM again after 2 seconds
INFO   0% (391.5 MiB of 2.7 TiB) in 3s, read 130.5 MiBs, write 118.9 MiBs
ERROR interrupted by signal
INFO aborting backup job
INFO resuming VM again
INFO resume vm
ERROR Backup of VM 115 failed - interrupted by signal
INFO Failed at 2021-10-09 063308
ERROR Backup job failed - interrupted by signal
TASK ERROR interrupted by signal

before the update it was like this:

INFO: starting new backup job: vzdump 115 --storage sata2 --mode suspend --node server --quiet 1 --compress lzo --mailnotification always
INFO: Starting Backup of VM 115 (qemu)
INFO: Backup started at 2021-09-30 01:30:02
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: VM Name: sqlserv
INFO: include disk 'sata0' 'local-lvm:vm-115-disk-0' 800G
INFO: include disk 'sata1' 'sata2:115/vm-115-disk-2.qcow2' 1000G
INFO: include disk 'sata2' 'sata2:115/vm-115-disk-3.qcow2' 1000G
INFO: suspending guest
INFO: creating vzdump archive '/var/lib/SATA2//dump/vzdump-qemu-115-2021_09_30-01_30_02.vma.lzo'
INFO: started backup task '5baddfc1-f567-4206-91da-68bb84007e0c'
INFO: resuming VM again after 1 seconds

Is there way change any file backup task, without installation successful backup? Just need to fix this structure, but I don't know how (

 

[fiona]

Hi,
please provide the output of pveversion -v. You might also want to use the debsums tool to verify the installed files. Do you get the same behavior when you start the backup manually from the command line (just copy the command from the first line in the log)?

 

[h9lk]

Hi,
please provide the output of pveversion -v. You might also want to use the debsums tool to verify the installed files. Do you get the same behavior when you start the backup manually from the command line (just copy the command from the first line in the log)?

Hello, what's strange if i make backup manually behavior is different:

INFO: starting new backup job: vzdump 115 --compress lzo --node server --storage sata2 --mode snapshot --remove 0

pveversion -v

proxmox-ve: 6.4-1 (running kernel: 5.4.140-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-6
pve-kernel-helper: 6.4-6
pve-kernel-5.4.140-1-pve: 5.4.140-1
pve-kernel-5.4.124-1-pve: 5.4.124-2
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.13-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-1
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.5-pve1~bpo10+1

tried to find debsums tool, but my searches were unsuccessful - tool itself is not there, i don't exclude that missed something, all that found was algorithms for checking files separately like sha224sum

 

[fiona]

Hello, what's strange if i make backup manually behavior is different:

INFO: starting new backup job: vzdump 115 --compress lzo --node server --storage sata2 --mode snapshot --remove 0

What if you open the task log in the UI afterwards (even if you start a job manually, the task log should show up)? Is this a stand-alone node or is the request proxied to another node in the cluster?

tried to find debsums tool, but my searches were unsuccessful - tool itself is not there, i don't exclude that missed something, all that found was algorithms for checking files separately like sha224sum

It has to be installed first with apt-get install debsums. I suggest running

debsums -s pve-manager libpve-guest-common-perl libpve-common-perl proxmox-widget-toolkit

for likely culprits (or simply debsums -s for everything). If there is no output the installed files should be fine.

 

[h9lk]

What if you open the task log in the UI afterwards (even if you start a job manually, the task log should show up)? Is this a stand-alone node or is the request proxied to another node in the cluster?


It has to be installed first with apt-get install debsums. I suggest running

debsums -s pve-manager libpve-guest-common-perl libpve-common-perl proxmox-widget-toolkit

for likely culprits (or simply debsums -s for everything). If there is no output the installed files should be fine.

1. Yes, there's always log.
2. have 1 server, stand-alone node
3. no output for debsums -s and

debsums -s pve-manager libpve-guest-common-perl libpve-common-perl proxmox-widget-toolkit

 

[fiona]

1. Yes, there's always log.

Does the log of the task started via CLI have the special symbols when you view it in the UI?

 

[h9lk]

Does the log of the task started via CLI have the special symbols when you view it in the UI?

package update came, after that it all appeared, checked - all paths are in place, only backup is carried out for very long time in 2 hours 1%, before this was not what happens. I stopped understanding anything

INFO: starting new backup job: vzdump 115 --compress lzo --quiet 1 --all 0 --node server --storage sata2 --mode suspend --mailnotification always

Did hardware raid 1 on ssd, all disks are working in normal, backups don't differ much now backup should weigh 420-430 gigabytes, but before 2.7 terabytes was processed maximum of 2 hours, now only 1% in 2 hours, it's weird that he's so slow, i run backup at night, when there is no load on server and there are 5 hours for backup but maximum 3% was processed in 5 hours

 

[fiona]

Are there any interesting messages in /var/log/syslog? Could you try booting with an older kernel and see if the speed is better?

 

[h9lk]

Are there any interesting messages in /var/log/syslog? Could you try booting with an older kernel and see if the speed is better?

1) The only thing that was interesting in log:

Oct 21 05:43:39 server pveproxy[1519]: worker 31726 finished
Oct 21 05:43:39 server pveproxy[1519]: starting 1 worker(s)
Oct 21 05:43:39 server pveproxy[1519]: worker 38696 started
Oct 21 05:43:39 server pveproxy[38695]: got inotify poll request in wrong process - disabling inotify

otherwise all tasks completed successfully

2) unfortunately can't make backup to older kernel, this server works 19 hours day even on weekends, there are several databases, u know
making backup remotely is not good idea.

3) tried to put other disks - this also didn't work + result of command:

debsums -s -a
debsums: changed file /etc/issue (from base-files package)