VM stop working when make manual backup on pbs or local disks

badarin

New Member
Jun 28, 2021
9
1
3
40
hello,
since the update to version 7 of one of my proxmox VE, I can no longer manually save my virtual machines in snapshot mode.

Systematically, the VMs remain locked, if I unlock by hand with a "QM UNLOCK VM 100" it unlocks but remains inactive. I don't have shell access.
If the virtual machine is stopped there is no problem to do a local or pbs backup manually.

I have no apparent problem if I run autosaves on my PBS. I have other proxmox servers on the network that are still in version 6 and I have no problems with them.

The machine concerned is a small server (hp proliant ml 110 g5 / 3GHz / 8Go / proxmox on sas raid 1 drive / Vm100 on ssd, clone vm100 on sata drive), I have two VMs on it which are in windows 10 and which manage a kaspersky security center console in a school, they are NEVER on at the same time, the second is simply a clone of the first in case of failure.

If you have an idea, it's not very disturbing but it's true that when I need to make a backup before making changes to my kaspersky server on my Vm, I like to do a manual backup just in case where ...

Thank you.


write with google translate.
 
Hi,
please share the output of pveversion -v and qm config <ID>, as well as the full backup log for a failed backup. You might also want to check if there's anything related in /var/log/syslog around the time of the manual backup.
 
Hi, thank's for your answer

result of pveversion -v

root@pve:~# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.35-2-pve)
pve-manager: 7.2-4 (running version: 7.2-4/ca9d43cc)
pve-kernel-5.15: 7.2-4
pve-kernel-helper: 7.2-4
pve-kernel-5.15.35-2-pve: 5.15.35-5
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-2
libpve-storage-perl: 7.2-4
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.3-1
proxmox-backup-file-restore: 2.2.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-10
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1
root@pve:~#


then result of qm config 100

root@pve:~# qm config 100
balloon: 0
boot: order=net0;sata0
cores: 4
cpu: host
ide2: none,media=cdrom
memory: 6144
name: ksc
net0: e1000=82:D9:6F:20:92:EB,bridge=vmbr0
numa: 0
onboot: 1
ostype: win10
sata0: ssd120:100/vm-100-disk-0.raw,size=101G
scsihw: virtio-scsi-pci
smbios1: uuid=99c35c06-cabf-4aae-ac5d-a99ecd62cc9a
sockets: 1
startup: up=120
tablet: 1
vga: std,memory=32
vmgenid: d0a66383-ff91-4317-8583-1ebe7af18118
vmstatestorage: dell500
root@pve:~#


In syslog, I have this, first we see that the pbs auto backup work great, then I have a lot of error like the last lines of log.

Jun 20 06:58:24 pve pvedaemon[969549]: <root@pam> starting task UPID:pve:000F0D1E:0180AA7F:62AFFE70:vzdump:100:root@pam:
Jun 20 06:58:24 pve pvedaemon[986398]: INFO: starting new backup job: vzdump 100 --node pve --remove 0 --storage pbs-2 --notes-template '{{guestname}}' --mode snapshot
Jun 20 06:58:24 pve pvedaemon[986398]: INFO: Starting Backup of VM 100 (qemu)
Jun 20 07:00:01 pve pvescheduler[986798]: <root@pam> starting task UPID:pve:000F0EAF:0180D06E:62AFFED1:vzdump:100:root@pam:
Jun 20 07:00:02 pve pvescheduler[986799]: INFO: trying to get global lock - waiting...
Jun 20 07:00:15 pve pvedaemon[65700]: worker exit
Jun 20 07:00:15 pve pvedaemon[968]: worker 65700 finished
Jun 20 07:00:15 pve pvedaemon[968]: starting 1 worker(s)
Jun 20 07:00:15 pve pvedaemon[968]: worker 986839 started
Jun 20 07:00:21 pve pvedaemon[972698]: <root@pam> successful auth for user 'root@pam'
Jun 20 07:02:38 pve pvedaemon[969549]: worker exit
Jun 20 07:02:38 pve pvedaemon[968]: worker 969549 finished
Jun 20 07:02:38 pve pvedaemon[968]: starting 1 worker(s)
Jun 20 07:02:38 pve pvedaemon[968]: worker 987418 started
Jun 20 07:03:14 pve pvedaemon[986398]: INFO: Finished Backup of VM 100 (00:04:50)

Jun 20 07:03:14 pve pvedaemon[986398]: INFO: Backup job finished successfully
Jun 20 07:03:14 pve pvescheduler[986799]: INFO: got global lock
Jun 20 07:03:14 pve pvescheduler[986799]: INFO: starting new backup job: vzdump 100 --mailto *****@gmail.com --compress 0 --mailnotification failure --storage dell --quiet 1 --notes-template '{{guestname}}' --mode snapshot
Jun 20 07:03:14 pve pvescheduler[986799]: INFO: Starting Backup of VM 100 (qemu)
Jun 20 07:03:22 pve pvedaemon[986839]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - got timeout
Jun 20 07:03:23 pve pvestatd[943]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - got timeout
Jun 20 07:03:23 pve pvestatd[943]: status update time (6.538 seconds)
Jun 20 07:03:33 pve pvestatd[943]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - got timeout
Jun 20 07:03:34 pve pvestatd[943]: status update time (6.480 seconds)
Jun 20 07:03:42 pve pvedaemon[986839]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - got timeout
Jun 20 07:03:43 pve pvestatd[943]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - got timeout
Jun 20 07:03:43 pve pvestatd[943]: status update time (6.439 seconds)
Jun 20 07:03:53 pve pvestatd[943]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - got timeout
Jun 20 07:03:54 pve pvestatd[943]: status update time (6.458 seconds)
Jun 20 07:04:02 pve pvedaemon[972698]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 31 retries
 
Please also share the task log of the failed backup job (it should show up in the bottom panel in the web UI). Please try if downgrading with apt install pve-qemu-kvm=6.2.0-8 helps. You need to stop/start the VM after this to pick up the newly installed version.
 
Task log of vm backup job (local on "dell" storage)



Task viewer: VM/CT 100 - Sauvegarde

SortieStatut

Stopper
INFO: trying to get global lock - waiting...
INFO: got global lock
INFO: starting new backup job: vzdump 100 --mailto *******@gmail.com --compress 0 --mailnotification failure --storage dell --quiet 1 --notes-template '{{guestname}}' --mode snapshot
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-06-20 07:03:14
INFO: status = running
INFO: VM Name: ksc
INFO: include disk 'sata0' 'ssd120:100/vm-100-disk-0.raw' 101G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/dell/dump/vzdump-qemu-100-2022_06_20-07_03_14.vma'
INFO: started backup task 'e691b6c7-9dc3-49d9-b097-3e154c6608be'
INFO: resuming VM again
INFO: 0% (989.6 MiB of 101.0 GiB) in 7s, read: 141.4 MiB/s, write: 134.5 MiB/s
INFO: 1% (1.5 GiB of 101.0 GiB) in 13s, read: 90.6 MiB/s, write: 89.3 MiB/s
INFO: 2% (2.1 GiB of 101.0 GiB) in 21s, read: 77.0 MiB/s, write: 76.5 MiB/s
INFO: 3% (3.5 GiB of 101.0 GiB) in 38s, read: 82.5 MiB/s, write: 82.1 MiB/s
INFO: 4% (4.2 GiB of 101.0 GiB) in 47s, read: 88.6 MiB/s, write: 87.7 MiB/s
INFO: 5% (5.4 GiB of 101.0 GiB) in 1m 1s, read: 81.8 MiB/s, write: 81.0 MiB/s
INFO: 6% (6.1 GiB of 101.0 GiB) in 1m 9s, read: 90.2 MiB/s, write: 89.8 MiB/s
INFO: 7% (7.5 GiB of 101.0 GiB) in 1m 27s, read: 82.2 MiB/s, write: 81.6 MiB/s
INFO: 8% (8.6 GiB of 101.0 GiB) in 1m 41s, read: 81.3 MiB/s, write: 79.5 MiB/s
INFO: 9% (9.4 GiB of 101.0 GiB) in 1m 50s, read: 88.2 MiB/s, write: 86.9 MiB/s
INFO: 10% (10.8 GiB of 101.0 GiB) in 2m 7s, read: 84.0 MiB/s, write: 83.8 MiB/s
INFO: 11% (11.3 GiB of 101.0 GiB) in 2m 14s, read: 76.6 MiB/s, write: 75.9 MiB/s
INFO: 12% (13.1 GiB of 101.0 GiB) in 2m 23s, read: 197.4 MiB/s, write: 78.1 MiB/s
INFO: 14% (14.3 GiB of 101.0 GiB) in 2m 29s, read: 206.6 MiB/s, write: 92.1 MiB/s
INFO: 15% (15.6 GiB of 101.0 GiB) in 2m 46s, read: 80.8 MiB/s, write: 80.2 MiB/s
INFO: 16% (16.8 GiB of 101.0 GiB) in 3m 1s, read: 79.8 MiB/s, write: 79.4 MiB/s
INFO: 17% (17.5 GiB of 101.0 GiB) in 3m 11s, read: 74.7 MiB/s, write: 74.3 MiB/s
INFO: 18% (18.8 GiB of 101.0 GiB) in 3m 28s, read: 80.2 MiB/s, write: 80.0 MiB/s
INFO: 19% (19.4 GiB of 101.0 GiB) in 3m 34s, read: 89.4 MiB/s, write: 89.0 MiB/s
INFO: 20% (20.6 GiB of 101.0 GiB) in 3m 51s, read: 77.4 MiB/s, write: 77.2 MiB/s
INFO: 21% (21.3 GiB of 101.0 GiB) in 3m 58s, read: 88.6 MiB/s, write: 86.4 MiB/s
INFO: 22% (22.5 GiB of 101.0 GiB) in 4m 14s, read: 81.3 MiB/s, write: 79.5 MiB/s
INFO: 23% (23.8 GiB of 101.0 GiB) in 4m 31s, read: 78.5 MiB/s, write: 77.5 MiB/s
INFO: 24% (24.4 GiB of 101.0 GiB) in 4m 38s, read: 83.4 MiB/s, write: 83.0 MiB/s
INFO: 25% (25.7 GiB of 101.0 GiB) in 4m 55s, read: 77.0 MiB/s, write: 76.9 MiB/s
INFO: 26% (26.9 GiB of 101.0 GiB) in 5m 11s, read: 79.1 MiB/s, write: 78.8 MiB/s
INFO: 27% (27.5 GiB of 101.0 GiB) in 5m 18s, read: 83.9 MiB/s, write: 83.8 MiB/s
INFO: 28% (28.7 GiB of 101.0 GiB) in 5m 34s, read: 79.5 MiB/s, write: 79.3 MiB/s
INFO: 29% (30.0 GiB of 101.0 GiB) in 5m 51s, read: 74.2 MiB/s, write: 73.9 MiB/s
INFO: 30% (30.5 GiB of 101.0 GiB) in 5m 58s, read: 82.3 MiB/s, write: 82.1 MiB/s
INFO: 31% (31.8 GiB of 101.0 GiB) in 6m 14s, read: 79.3 MiB/s, write: 79.2 MiB/s
INFO: 32% (33.0 GiB of 101.0 GiB) in 6m 31s, read: 76.6 MiB/s, write: 76.3 MiB/s
INFO: 33% (33.7 GiB of 101.0 GiB) in 6m 41s, read: 71.8 MiB/s, write: 71.8 MiB/s
INFO: 34% (35.0 GiB of 101.0 GiB) in 6m 57s, read: 79.5 MiB/s, write: 79.1 MiB/s
INFO: 35% (35.5 GiB of 101.0 GiB) in 7m 5s, read: 67.5 MiB/s, write: 67.5 MiB/s
INFO: 36% (36.7 GiB of 101.0 GiB) in 7m 21s, read: 77.9 MiB/s, write: 77.9 MiB/s
INFO: 37% (38.0 GiB of 101.0 GiB) in 7m 38s, read: 75.3 MiB/s, write: 75.0 MiB/s
INFO: 38% (38.5 GiB of 101.0 GiB) in 7m 45s, read: 77.7 MiB/s, write: 77.6 MiB/s
INFO: 39% (39.7 GiB of 101.0 GiB) in 8m 2s, read: 73.8 MiB/s, write: 73.7 MiB/s
INFO: 40% (40.4 GiB of 101.0 GiB) in 8m 11s, read: 78.9 MiB/s, write: 78.6 MiB/s
INFO: 41% (41.7 GiB of 101.0 GiB) in 8m 28s, read: 75.1 MiB/s, write: 74.8 MiB/s
INFO: 42% (42.9 GiB of 101.0 GiB) in 8m 45s, read: 73.1 MiB/s, write: 71.7 MiB/s
INFO: 43% (43.6 GiB of 101.0 GiB) in 8m 54s, read: 78.1 MiB/s, write: 76.8 MiB/s
INFO: 44% (44.8 GiB of 101.0 GiB) in 9m 12s, read: 71.1 MiB/s, write: 71.0 MiB/s
INFO: 45% (45.5 GiB of 101.0 GiB) in 9m 21s, read: 77.3 MiB/s, write: 77.3 MiB/s
INFO: 46% (46.9 GiB of 101.0 GiB) in 9m 41s, read: 70.7 MiB/s, write: 70.5 MiB/s
INFO: 47% (47.6 GiB of 101.0 GiB) in 9m 51s, read: 74.7 MiB/s, write: 73.1 MiB/s
INFO: 48% (49.1 GiB of 101.0 GiB) in 10m 12s, read: 72.7 MiB/s, write: 69.4 MiB/s
INFO: 49% (49.7 GiB of 101.0 GiB) in 10m 21s, read: 74.5 MiB/s, write: 74.5 MiB/s
INFO: 50% (50.5 GiB of 101.0 GiB) in 10m 32s, read: 70.2 MiB/s, write: 70.1 MiB/s
INFO: 51% (51.9 GiB of 101.0 GiB) in 10m 52s, read: 71.6 MiB/s, write: 70.5 MiB/s
INFO: 52% (52.6 GiB of 101.0 GiB) in 11m 1s, read: 75.3 MiB/s, write: 75.3 MiB/s
INFO: 53% (54.1 GiB of 101.0 GiB) in 11m 24s, read: 70.0 MiB/s, write: 69.8 MiB/s
INFO: 54% (54.8 GiB of 101.0 GiB) in 11m 34s, read: 69.2 MiB/s, write: 69.1 MiB/s
INFO: 55% (56.0 GiB of 101.0 GiB) in 11m 52s, read: 69.1 MiB/s, write: 69.1 MiB/s
INFO: 56% (56.8 GiB of 101.0 GiB) in 11m 59s, read: 107.2 MiB/s, write: 70.9 MiB/s
INFO: 57% (58.1 GiB of 101.0 GiB) in 12m 9s, read: 139.2 MiB/s, write: 68.8 MiB/s
INFO: 59% (59.9 GiB of 101.0 GiB) in 12m 18s, read: 206.6 MiB/s, write: 72.6 MiB/s
INFO: 61% (61.7 GiB of 101.0 GiB) in 12m 22s, read: 448.6 MiB/s, write: 65.9 MiB/s
INFO: 62% (62.7 GiB of 101.0 GiB) in 12m 32s, read: 105.8 MiB/s, write: 69.9 MiB/s
INFO: 63% (64.0 GiB of 101.0 GiB) in 12m 49s, read: 76.1 MiB/s, write: 69.2 MiB/s
ERROR: VM 100 qmp command 'query-backup' failed - client closed connection
INFO: aborting backup job
ERROR: VM 100 not running
INFO: resuming VM again
no lock found trying to remove 'backup' lock
ERROR: Backup of VM 100 failed - VM 100 not running
INFO: Failed at 2022-06-20 07:16:16
INFO: Backup job finished with errors
TASK ERROR: job errors




so you want me to downgrade to version 6 ???
 
Task log of vm backup job (local on "dell" storage)



Task viewer: VM/CT 100 - Sauvegarde

SortieStatut

Stopper
INFO: trying to get global lock - waiting...
INFO: got global lock
INFO: starting new backup job: vzdump 100 --mailto *******@gmail.com --compress 0 --mailnotification failure --storage dell --quiet 1 --notes-template '{{guestname}}' --mode snapshot
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-06-20 07:03:14
INFO: status = running
INFO: VM Name: ksc
INFO: include disk 'sata0' 'ssd120:100/vm-100-disk-0.raw' 101G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/dell/dump/vzdump-qemu-100-2022_06_20-07_03_14.vma'
INFO: started backup task 'e691b6c7-9dc3-49d9-b097-3e154c6608be'
INFO: resuming VM again
INFO: 0% (989.6 MiB of 101.0 GiB) in 7s, read: 141.4 MiB/s, write: 134.5 MiB/s
INFO: 1% (1.5 GiB of 101.0 GiB) in 13s, read: 90.6 MiB/s, write: 89.3 MiB/s
INFO: 2% (2.1 GiB of 101.0 GiB) in 21s, read: 77.0 MiB/s, write: 76.5 MiB/s
INFO: 3% (3.5 GiB of 101.0 GiB) in 38s, read: 82.5 MiB/s, write: 82.1 MiB/s
INFO: 4% (4.2 GiB of 101.0 GiB) in 47s, read: 88.6 MiB/s, write: 87.7 MiB/s
INFO: 5% (5.4 GiB of 101.0 GiB) in 1m 1s, read: 81.8 MiB/s, write: 81.0 MiB/s
INFO: 6% (6.1 GiB of 101.0 GiB) in 1m 9s, read: 90.2 MiB/s, write: 89.8 MiB/s
INFO: 7% (7.5 GiB of 101.0 GiB) in 1m 27s, read: 82.2 MiB/s, write: 81.6 MiB/s
INFO: 8% (8.6 GiB of 101.0 GiB) in 1m 41s, read: 81.3 MiB/s, write: 79.5 MiB/s
INFO: 9% (9.4 GiB of 101.0 GiB) in 1m 50s, read: 88.2 MiB/s, write: 86.9 MiB/s
INFO: 10% (10.8 GiB of 101.0 GiB) in 2m 7s, read: 84.0 MiB/s, write: 83.8 MiB/s
INFO: 11% (11.3 GiB of 101.0 GiB) in 2m 14s, read: 76.6 MiB/s, write: 75.9 MiB/s
INFO: 12% (13.1 GiB of 101.0 GiB) in 2m 23s, read: 197.4 MiB/s, write: 78.1 MiB/s
INFO: 14% (14.3 GiB of 101.0 GiB) in 2m 29s, read: 206.6 MiB/s, write: 92.1 MiB/s
INFO: 15% (15.6 GiB of 101.0 GiB) in 2m 46s, read: 80.8 MiB/s, write: 80.2 MiB/s
INFO: 16% (16.8 GiB of 101.0 GiB) in 3m 1s, read: 79.8 MiB/s, write: 79.4 MiB/s
INFO: 17% (17.5 GiB of 101.0 GiB) in 3m 11s, read: 74.7 MiB/s, write: 74.3 MiB/s
INFO: 18% (18.8 GiB of 101.0 GiB) in 3m 28s, read: 80.2 MiB/s, write: 80.0 MiB/s
INFO: 19% (19.4 GiB of 101.0 GiB) in 3m 34s, read: 89.4 MiB/s, write: 89.0 MiB/s
INFO: 20% (20.6 GiB of 101.0 GiB) in 3m 51s, read: 77.4 MiB/s, write: 77.2 MiB/s
INFO: 21% (21.3 GiB of 101.0 GiB) in 3m 58s, read: 88.6 MiB/s, write: 86.4 MiB/s
INFO: 22% (22.5 GiB of 101.0 GiB) in 4m 14s, read: 81.3 MiB/s, write: 79.5 MiB/s
INFO: 23% (23.8 GiB of 101.0 GiB) in 4m 31s, read: 78.5 MiB/s, write: 77.5 MiB/s
INFO: 24% (24.4 GiB of 101.0 GiB) in 4m 38s, read: 83.4 MiB/s, write: 83.0 MiB/s
INFO: 25% (25.7 GiB of 101.0 GiB) in 4m 55s, read: 77.0 MiB/s, write: 76.9 MiB/s
INFO: 26% (26.9 GiB of 101.0 GiB) in 5m 11s, read: 79.1 MiB/s, write: 78.8 MiB/s
INFO: 27% (27.5 GiB of 101.0 GiB) in 5m 18s, read: 83.9 MiB/s, write: 83.8 MiB/s
INFO: 28% (28.7 GiB of 101.0 GiB) in 5m 34s, read: 79.5 MiB/s, write: 79.3 MiB/s
INFO: 29% (30.0 GiB of 101.0 GiB) in 5m 51s, read: 74.2 MiB/s, write: 73.9 MiB/s
INFO: 30% (30.5 GiB of 101.0 GiB) in 5m 58s, read: 82.3 MiB/s, write: 82.1 MiB/s
INFO: 31% (31.8 GiB of 101.0 GiB) in 6m 14s, read: 79.3 MiB/s, write: 79.2 MiB/s
INFO: 32% (33.0 GiB of 101.0 GiB) in 6m 31s, read: 76.6 MiB/s, write: 76.3 MiB/s
INFO: 33% (33.7 GiB of 101.0 GiB) in 6m 41s, read: 71.8 MiB/s, write: 71.8 MiB/s
INFO: 34% (35.0 GiB of 101.0 GiB) in 6m 57s, read: 79.5 MiB/s, write: 79.1 MiB/s
INFO: 35% (35.5 GiB of 101.0 GiB) in 7m 5s, read: 67.5 MiB/s, write: 67.5 MiB/s
INFO: 36% (36.7 GiB of 101.0 GiB) in 7m 21s, read: 77.9 MiB/s, write: 77.9 MiB/s
INFO: 37% (38.0 GiB of 101.0 GiB) in 7m 38s, read: 75.3 MiB/s, write: 75.0 MiB/s
INFO: 38% (38.5 GiB of 101.0 GiB) in 7m 45s, read: 77.7 MiB/s, write: 77.6 MiB/s
INFO: 39% (39.7 GiB of 101.0 GiB) in 8m 2s, read: 73.8 MiB/s, write: 73.7 MiB/s
INFO: 40% (40.4 GiB of 101.0 GiB) in 8m 11s, read: 78.9 MiB/s, write: 78.6 MiB/s
INFO: 41% (41.7 GiB of 101.0 GiB) in 8m 28s, read: 75.1 MiB/s, write: 74.8 MiB/s
INFO: 42% (42.9 GiB of 101.0 GiB) in 8m 45s, read: 73.1 MiB/s, write: 71.7 MiB/s
INFO: 43% (43.6 GiB of 101.0 GiB) in 8m 54s, read: 78.1 MiB/s, write: 76.8 MiB/s
INFO: 44% (44.8 GiB of 101.0 GiB) in 9m 12s, read: 71.1 MiB/s, write: 71.0 MiB/s
INFO: 45% (45.5 GiB of 101.0 GiB) in 9m 21s, read: 77.3 MiB/s, write: 77.3 MiB/s
INFO: 46% (46.9 GiB of 101.0 GiB) in 9m 41s, read: 70.7 MiB/s, write: 70.5 MiB/s
INFO: 47% (47.6 GiB of 101.0 GiB) in 9m 51s, read: 74.7 MiB/s, write: 73.1 MiB/s
INFO: 48% (49.1 GiB of 101.0 GiB) in 10m 12s, read: 72.7 MiB/s, write: 69.4 MiB/s
INFO: 49% (49.7 GiB of 101.0 GiB) in 10m 21s, read: 74.5 MiB/s, write: 74.5 MiB/s
INFO: 50% (50.5 GiB of 101.0 GiB) in 10m 32s, read: 70.2 MiB/s, write: 70.1 MiB/s
INFO: 51% (51.9 GiB of 101.0 GiB) in 10m 52s, read: 71.6 MiB/s, write: 70.5 MiB/s
INFO: 52% (52.6 GiB of 101.0 GiB) in 11m 1s, read: 75.3 MiB/s, write: 75.3 MiB/s
INFO: 53% (54.1 GiB of 101.0 GiB) in 11m 24s, read: 70.0 MiB/s, write: 69.8 MiB/s
INFO: 54% (54.8 GiB of 101.0 GiB) in 11m 34s, read: 69.2 MiB/s, write: 69.1 MiB/s
INFO: 55% (56.0 GiB of 101.0 GiB) in 11m 52s, read: 69.1 MiB/s, write: 69.1 MiB/s
INFO: 56% (56.8 GiB of 101.0 GiB) in 11m 59s, read: 107.2 MiB/s, write: 70.9 MiB/s
INFO: 57% (58.1 GiB of 101.0 GiB) in 12m 9s, read: 139.2 MiB/s, write: 68.8 MiB/s
INFO: 59% (59.9 GiB of 101.0 GiB) in 12m 18s, read: 206.6 MiB/s, write: 72.6 MiB/s
INFO: 61% (61.7 GiB of 101.0 GiB) in 12m 22s, read: 448.6 MiB/s, write: 65.9 MiB/s
INFO: 62% (62.7 GiB of 101.0 GiB) in 12m 32s, read: 105.8 MiB/s, write: 69.9 MiB/s
INFO: 63% (64.0 GiB of 101.0 GiB) in 12m 49s, read: 76.1 MiB/s, write: 69.2 MiB/s
ERROR: VM 100 qmp command 'query-backup' failed - client closed connection
INFO: aborting backup job
ERROR: VM 100 not running
INFO: resuming VM again
no lock found trying to remove 'backup' lock
ERROR: Backup of VM 100 failed - VM 100 not running
INFO: Failed at 2022-06-20 07:16:16
INFO: Backup job finished with errors
TASK ERROR: job errors
What likely is helpful is a backtrace of the crash. To obtain one, please do the following:
  1. apt install pve-qemu-kvm-dbg
  2. Stop/Start the VM
  3. Run gdb --ex 'handle SIGUSR1 nostop noprint' --ex 'pagination off' --ex 'c' -p $(qm status 100 --verbose | grep pid: | cut -d: -f2)
  4. Start the manual backup and wait for the crash.
  5. Issue t a a bt in gdb and share the resulting output here.

so you want me to downgrade to version 6 ???
No, only the QEMU package, with the specific command I mentioned. Please do that after getting the backtrace. The changes between that version and your version should only affect start/end of the backup, so I wouldn't bet on it to help.
 
Sorry I can't execute the backtrace command that you posted (and I'm a newbie and I don't know how it run...)

result of command :

root@pve:~# gdb --ex 'handle SIGUSR1 nostop noprint' --ex 'pagination off' --ex 'c' -p $(qm status 100 --verbose | grep pid: | cut -d: -f2) -bash: gdb: command not found root@pve:~#
 
I don't know if it's good or not....

I have made one backup good with compression on local hdd.
So, reboot server and I try another backup with no compression and it's halt again...

this is the result of gdb :

root@pve:~# gdb --ex 'handle SIGUSR1 nostop noprint' --ex 'pagination off' --ex 'c' -p $(qm status 100 --verbose | grep pid: | cut -d: -f2)
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 1015
[New LWP 1016]
[New LWP 1035]
[New LWP 1036]
[New LWP 1037]
[New LWP 1038]
[New LWP 1041]
[New LWP 2282]
[New LWP 3932]
[New LWP 5569]
[New LWP 14879]
[New LWP 14880]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f3a4d0784f6 in __ppoll (fds=0x561baacab560, nfds=9, timeout=<optimized out>, timeout@entry=0x7ffd90d70b90, sigmask=sigmask@entry=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:44
44 ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
Signal Stop Print Pass to program Description
SIGUSR1 No No Yes User defined signal 1
Undefined command: "pagination". Try "help".
Continuing.
[Thread 0x7f38abdff700 (LWP 14880) exited]
[New Thread 0x7f38abdff700 (LWP 15020)]
[Thread 0x7f38abdff700 (LWP 15020) exited]

Thread 1 "kvm" received signal SIGPIPE, Broken pipe.
__libc_sendmsg (flags=0, msg=0x7ffd90d707e0, fd=25) at ../sysdeps/unix/sysv/linux/sendmsg.c:28
28 ../sysdeps/unix/sysv/linux/sendmsg.c: No such file or directory.
(gdb) t a a bt

Thread 11 (Thread 0x7f38b3dff700 (LWP 14879) "iou-wrk-1037"):
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 10 (Thread 0x7f3a40dff700 (LWP 5569) "iou-wrk-1036"):
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 9 (Thread 0x7f3a41c3d700 (LWP 3932) "iou-wrk-1035"):
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 8 (Thread 0x7f38b31ff700 (LWP 2282) "iou-wrk-1038"):
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 7 (Thread 0x7f38aa9bf700 (LWP 1041) "kvm"):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x561bab0f2c18) at ../sysdeps/nptl/futex-internal.h:186
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x561bab0f2c28, cond=0x561bab0f2bf0) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=cond@entry=0x561bab0f2bf0, mutex=mutex@entry=0x561bab0f2c28) at pthread_cond_wait.c:638
#3 0x0000561ba9c231bb in qemu_cond_wait_impl (cond=0x561bab0f2bf0, mutex=0x561bab0f2c28, file=0x561ba9cab854 "../ui/vnc-jobs.c", line=248) at ../util/qemu-thread-posix.c:195
#4 0x0000561ba9736563 in vnc_worker_thread_loop (queue=0x561bab0f2bf0) at ../ui/vnc-jobs.c:248
#5 0x0000561ba9737228 in vnc_worker_thread (arg=arg@entry=0x561bab0f2bf0) at ../ui/vnc-jobs.c:361
#6 0x0000561ba9c227c9 in qemu_thread_start (args=0x7f38aa9ba3f0) at ../util/qemu-thread-posix.c:556
#7 0x00007f3a4d154ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8 0x00007f3a4d082def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f38b31ff700 (LWP 1038) "kvm"):
#0 0x00007f3a4d079cc7 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1 0x0000561ba9aaff87 in kvm_vcpu_ioctl (cpu=cpu@entry=0x561baaced4a0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3027
#2 0x0000561ba9ab00f1 in kvm_cpu_exec (cpu=cpu@entry=0x561baaced4a0) at ../accel/kvm/kvm-all.c:2853
#3 0x0000561ba9ab15bd in kvm_vcpu_thread_fn (arg=arg@entry=0x561baaced4a0) at ../accel/kvm/kvm-accel-ops.c:49
#4 0x0000561ba9c227c9 in qemu_thread_start (args=0x7f38b31fa3f0) at ../util/qemu-thread-posix.c:556
#5 0x00007f3a4d154ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6 0x00007f3a4d082def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f38b3dff700 (LWP 1037) "kvm"):
#0 0x00007f3a4d079cc7 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1 0x0000561ba9aaff87 in kvm_vcpu_ioctl (cpu=cpu@entry=0x561baacdfab0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3027
#2 0x0000561ba9ab00f1 in kvm_cpu_exec (cpu=cpu@entry=0x561baacdfab0) at ../accel/kvm/kvm-all.c:2853
#3 0x0000561ba9ab15bd in kvm_vcpu_thread_fn (arg=arg@entry=0x561baacdfab0) at ../accel/kvm/kvm-accel-ops.c:49
#4 0x0000561ba9c227c9 in qemu_thread_start (args=0x7f38b3dfa3f0) at ../util/qemu-thread-posix.c:556
#5 0x00007f3a4d154ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
--Type <RET> for more, q to quit, c to continue without paging--c
#6 0x00007f3a4d082def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f3a40dff700 (LWP 1036) "kvm"):
#0 __lll_lock_wait (futex=futex@entry=0x561baa1b86a0 <qemu_global_mutex>, private=0) at lowlevellock.c:52
#1 0x00007f3a4d157843 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x561baa1b86a0 <qemu_global_mutex>) at ../nptl/pthread_mutex_lock.c:80
#2 0x0000561ba9c22984 in qemu_mutex_lock_impl (mutex=0x561baa1b86a0 <qemu_global_mutex>, file=0x561ba9d6be9f "../softmmu/physmem.c", line=2752) at ../util/qemu-thread-posix.c:80
#3 0x0000561ba99c723e in qemu_mutex_lock_iothread_impl (file=file@entry=0x561ba9d6be9f "../softmmu/physmem.c", line=line@entry=2752) at ../softmmu/cpus.c:493
#4 0x0000561ba99ce2b1 in prepare_mmio_access (mr=0x561babb32040) at ../softmmu/physmem.c:2752
#5 flatview_read_continue (fv=fv@entry=0x7f38ac04d770, addr=addr@entry=4272361780, attrs=..., ptr=ptr@entry=0x7f3a4265b028, len=len@entry=4, addr1=<optimized out>, l=<optimized out>, mr=0x561babb32040) at ../softmmu/physmem.c:2844
#6 0x0000561ba99ce3a7 in flatview_read (fv=0x7f38ac04d770, addr=4272361780, attrs=..., buf=0x7f3a4265b028, len=4) at ../softmmu/physmem.c:2885
#7 0x0000561ba99ce806 in address_space_read_full (len=<optimized out>, buf=0x0, attrs=..., addr=<optimized out>, as=<optimized out>) at ../softmmu/physmem.c:2898
#8 address_space_rw (as=<optimized out>, addr=<optimized out>, attrs=..., attrs@entry=..., buf=buf@entry=0x7f3a4265b028, len=<optimized out>, is_write=<optimized out>) at ../softmmu/physmem.c:2926
#9 0x0000561ba9ab0290 in kvm_cpu_exec (cpu=cpu@entry=0x561baacd1f50) at ../accel/kvm/kvm-all.c:2903
#10 0x0000561ba9ab15bd in kvm_vcpu_thread_fn (arg=arg@entry=0x561baacd1f50) at ../accel/kvm/kvm-accel-ops.c:49
#11 0x0000561ba9c227c9 in qemu_thread_start (args=0x7f3a40dfa3f0) at ../util/qemu-thread-posix.c:556
#12 0x00007f3a4d154ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x00007f3a4d082def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f3a41c3d700 (LWP 1035) "kvm"):
#0 0x00007f3a4d079cc7 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1 0x0000561ba9aaff87 in kvm_vcpu_ioctl (cpu=cpu@entry=0x561baac9d6d0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3027
#2 0x0000561ba9ab00f1 in kvm_cpu_exec (cpu=cpu@entry=0x561baac9d6d0) at ../accel/kvm/kvm-all.c:2853
#3 0x0000561ba9ab15bd in kvm_vcpu_thread_fn (arg=arg@entry=0x561baac9d6d0) at ../accel/kvm/kvm-accel-ops.c:49
#4 0x0000561ba9c227c9 in qemu_thread_start (args=0x7f3a41c383f0) at ../util/qemu-thread-posix.c:556
#5 0x00007f3a4d154ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6 0x00007f3a4d082def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f3a4253f700 (LWP 1016) "kvm"):
#0 __lll_lock_wait (futex=futex@entry=0x561baa1b86a0 <qemu_global_mutex>, private=0) at lowlevellock.c:52
#1 0x00007f3a4d157843 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x561baa1b86a0 <qemu_global_mutex>) at ../nptl/pthread_mutex_lock.c:80
#2 0x0000561ba9c22984 in qemu_mutex_lock_impl (mutex=0x561baa1b86a0 <qemu_global_mutex>, file=0x561ba9dfa585 "../util/rcu.c", line=269) at ../util/qemu-thread-posix.c:80
#3 0x0000561ba99c723e in qemu_mutex_lock_iothread_impl (file=file@entry=0x561ba9dfa585 "../util/rcu.c", line=line@entry=269) at ../softmmu/cpus.c:493
#4 0x0000561ba9c2c13e in call_rcu_thread (opaque=opaque@entry=0x0) at ../util/rcu.c:269
#5 0x0000561ba9c227c9 in qemu_thread_start (args=0x7f3a4253a3f0) at ../util/qemu-thread-posix.c:556
#6 0x00007f3a4d154ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7 0x00007f3a4d082def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f3a426aa1c0 (LWP 1015) "kvm"):
#0 __libc_sendmsg (flags=0, msg=0x7ffd90d707e0, fd=25) at ../sysdeps/unix/sysv/linux/sendmsg.c:28
#1 __libc_sendmsg (fd=25, msg=msg@entry=0x7ffd90d707e0, flags=flags@entry=0) at ../sysdeps/unix/sysv/linux/sendmsg.c:25
#2 0x0000561ba9ae3c59 in qio_channel_socket_writev (ioc=<optimized out>, iov=0x7ffd90d708d0, niov=1, fds=0x0, nfds=140727033464800, errp=0x0) at ../io/channel-socket.c:561
#3 0x0000561ba9ba4646 in io_channel_send_full (ioc=0x561bab062e90, buf=buf@entry=0x561baabace40, len=len@entry=123, fds=0x0, nfds=0) at ../chardev/char-io.c:123
#4 0x0000561ba9ba7bb1 in tcp_chr_write (chr=0x561baaa026b0, buf=0x561baabace40 "{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 0, \"minor\": 2, \"major\": 6}, \"package\": \"pve-qemu-kvm_6.2.0\"}, \"capabilities\": []}}\r\n", len=123) at ../chardev/char-socket.c:168
#5 0x0000561ba9baac53 in qemu_chr_write_buffer (s=s@entry=0x561baaa026b0, buf=buf@entry=0x561baabace40 "{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 0, \"minor\": 2, \"major\": 6}, \"package\": \"pve-qemu-kvm_6.2.0\"}, \"capabilities\": []}}\r\n", len=123, offset=offset@entry=0x7ffd90d709e0, write_all=write_all@entry=false) at ../chardev/char.c:121
#6 0x0000561ba9bab070 in qemu_chr_write (s=0x561baaa026b0, buf=0x561baabace40 "{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 0, \"minor\": 2, \"major\": 6}, \"package\": \"pve-qemu-kvm_6.2.0\"}, \"capabilities\": []}}\r\n", len=len@entry=123, write_all=write_all@entry=false) at ../chardev/char.c:173
#7 0x0000561ba9ba35ef in qemu_chr_fe_write (be=<optimized out>, buf=<optimized out>, len=len@entry=123) at ../chardev/char-fe.c:42
#8 0x0000561ba9baec5a in monitor_flush_locked (mon=0x561baabcb4a0) at ../monitor/monitor.c:201
#9 0x0000561ba9baf16e in monitor_flush_locked (mon=0x561baabcb4a0) at ../monitor/monitor.c:233
#10 monitor_puts (mon=mon@entry=0x561baabcb4a0, str=0x561baabb67d0 "{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 0, \"minor\": 2, \"major\": 6}, \"package\": \"pve-qemu-kvm_6.2.0\"}, \"capabilities\": []}}\n") at ../monitor/monitor.c:240
#11 0x0000561ba9bb04ad in qmp_send_response (mon=0x561baabcb4a0, rsp=<optimized out>) at ../monitor/qmp.c:118
#12 0x0000561ba9bb0d53 in monitor_qmp_event (event=CHR_EVENT_OPENED, opaque=0x561baabcb4a0) at ../monitor/qmp.c:463
#13 monitor_qmp_event (opaque=0x561baabcb4a0, event=<optimized out>) at ../monitor/qmp.c:452
#14 0x0000561ba9ba8a54 in tcp_chr_new_client (chr=0x561baaa026b0, sioc=0x561bab062e90) at ../chardev/char-socket.c:931
#15 0x0000561ba9aeaaea in qio_net_listener_channel_func (ioc=<optimized out>, condition=<optimized out>, opaque=<optimized out>) at ../io/net-listener.c:54
#16 0x00007f3a4e4dfd6f in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#17 0x0000561ba9c3df10 in glib_pollfds_poll () at ../util/main-loop.c:232
#18 os_host_main_loop_wait (timeout=0) at ../util/main-loop.c:255
#19 main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:531
#20 0x0000561ba99d6d41 in qemu_main_loop () at ../softmmu/runstate.c:726
#21 0x0000561ba97060ae in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at ../softmmu/main.c:50
(gdb)
 
Unfortunately, it seems that you ran into a (harmless) SIGPIPE first.
Please try again with
Code:
gdb --ex 'handle SIGUSR1 nostop noprint' --ex 'handle SIGPIPE nostop noprint' --ex 'pagination off' --ex 'c' -p $(qm status 100 --verbose | grep pid: | cut -d: -f2)
which makes the debugger not stop on that signal.

If you still have the debugging session open, you can also input
Code:
handle SIGPIPE nostop noprint
c
directly and then issue another backup.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!