Hi. First of all, thanks for the great products!
I have a small lab with PVE. The PVE contains two VMs, one of them is a PBS. The storage for the backup datastore is a separate disk which is passed-through to the PBS VM.
A few days ago I have set up a schedule to backup both VMs to the PBS. Of course the datastore disk is excluded from the backup job, only the "system" disk of the PBS is backed up.
This night the second part of the job (creating backup of the PBS VM) failed. The error is:
There was no this error during previous scheduled nor manual runs of the backup job.
I have spent some time googling for a possible reason but only have found some old (from 2021 or so) threads on the Forum, mentioning that a fix was introduced later then, and some newer threads with suggestions about a network problem.
But there is no physical network between the PVE and the PBS. The PBS is hosted in the very same PVE.
Here you are the log of the latest backup job:
In the "System Log" of the PVE in this timeframe there is:
In the Syslog of the PBS there's nothing unusual, as far as I can see:
Both the PVE (8.4.13) and the PBS (3.4.6) are up-to-date.
Is there anything more that I can show to help find the reason of the error?
Thank you in advance for the help!
I have a small lab with PVE. The PVE contains two VMs, one of them is a PBS. The storage for the backup datastore is a separate disk which is passed-through to the PBS VM.
A few days ago I have set up a schedule to backup both VMs to the PBS. Of course the datastore disk is excluded from the backup job, only the "system" disk of the PBS is backed up.
This night the second part of the job (creating backup of the PBS VM) failed. The error is:
[B]ERROR: VM 101 qmp command 'backup' failed - backup connect failed: command error: http request timed out[/B]There was no this error during previous scheduled nor manual runs of the backup job.
I have spent some time googling for a possible reason but only have found some old (from 2021 or so) threads on the Forum, mentioning that a fix was introduced later then, and some newer threads with suggestions about a network problem.
But there is no physical network between the PVE and the PBS. The PBS is hosted in the very same PVE.
Here you are the log of the latest backup job:
Code:
INFO: starting new backup job: vzdump --mailto someuser@somewhere --notes-template '{{guestname}}' --all 1 --quiet 1 --mode snapshot --fleecing '1,storage=local-lvm' --node pmx1 --storage store1 --notification-mode legacy-sendmail --mailnotification always
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2025-09-18 02:00:12
INFO: status = running
INFO: VM Name: ****
INFO: include disk 'virtio0' 'local-lvm:vm-100-disk-0' 28160M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2025-09-18T00:00:12Z'
Logical volume "vm-100-fleece-0" created.
INFO: drive-virtio0: attaching fleecing image local-lvm:vm-100-fleece-0 to QEMU
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'd93c4a4c-3cb3-4c6c-8332-3ef6a831561f'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: OK (1.5 GiB of 27.5 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 1.5 GiB dirty of 27.5 GiB total
INFO: 6% (100.0 MiB of 1.5 GiB) in 3s, read: 33.3 MiB/s, write: 33.3 MiB/s
INFO: 11% (180.0 MiB of 1.5 GiB) in 6s, read: 26.7 MiB/s, write: 25.3 MiB/s
[...]
INFO: 100% (1.5 GiB of 1.5 GiB) in 1m 4s, read: 24.0 MiB/s, write: 24.0 MiB/s
INFO: backup was done incrementally, reused 26.01 GiB (94%)
INFO: transferred 1.50 GiB in 70 seconds (21.9 MiB/s)
INFO: adding notes to backup
INFO: removing (old) fleecing image 'local-lvm:vm-100-fleece-0'
Logical volume "vm-100-fleece-0" successfully removed.
INFO: Finished Backup of VM 100 (00:01:18)
INFO: Backup finished at 2025-09-18 02:01:30
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2025-09-18 02:01:31
INFO: status = running
INFO: VM Name: pbs1
INFO: include disk 'scsi0' 'local-lvm:vm-101-disk-0' 16G
INFO: exclude disk 'scsi1' '/dev/disk/by-id/ata-HD****' (backup=no)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2025-09-18T00:01:31Z'
Logical volume "vm-101-fleece-0" created.
INFO: drive-scsi0: attaching fleecing image local-lvm:vm-101-fleece-0 to QEMU
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 101 qmp command 'backup' failed - backup connect failed: command error: http request timed out
INFO: aborting backup job
INFO: resuming VM again
INFO: removing (old) fleecing image 'local-lvm:vm-101-fleece-0'
Logical volume "vm-101-fleece-0" successfully removed.
ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup connect failed: command error: http request timed out
INFO: Failed at 2025-09-18 02:03:32
INFO: Backup job finished with errors
INFO: notified via target `<someuser@somewhere>`
TASK ERROR: job errorsIn the "System Log" of the PVE in this timeframe there is:
Code:
Sep 18 02:00:12 pmx1 pvescheduler[1800519]: INFO: Starting Backup of VM 100 (qemu)
Sep 18 02:01:30 pmx1 pvescheduler[1800519]: INFO: Finished Backup of VM 100 (00:01:18)
Sep 18 02:01:31 pmx1 pvescheduler[1800519]: INFO: Starting Backup of VM 101 (qemu)
Sep 18 02:01:48 pmx1 pvestatd[1703]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - got timeout
Sep 18 02:01:55 pmx1 pvestatd[1703]: store1: error fetching datastores - 500 read timeout
Sep 18 02:01:56 pmx1 pvestatd[1703]: status update time (15.227 seconds)
Sep 18 02:02:04 pmx1 pvestatd[1703]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Sep 18 02:02:11 pmx1 pvestatd[1703]: store1: error fetching datastores - 500 read timeout
Sep 18 02:02:11 pmx1 pvestatd[1703]: status update time (15.274 seconds)
Sep 18 02:02:19 pmx1 pvestatd[1703]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Sep 18 02:02:26 pmx1 pvestatd[1703]: store1: error fetching datastores - 500 read timeout
Sep 18 02:02:26 pmx1 pvestatd[1703]: status update time (15.230 seconds)
Sep 18 02:02:34 pmx1 pvestatd[1703]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Sep 18 02:02:41 pmx1 pvestatd[1703]: store1: error fetching datastores - 500 read timeout
Sep 18 02:02:41 pmx1 pvestatd[1703]: status update time (15.240 seconds)
Sep 18 02:02:49 pmx1 pvestatd[1703]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Sep 18 02:02:56 pmx1 pvestatd[1703]: store1: error fetching datastores - 500 read timeout
Sep 18 02:02:56 pmx1 pvestatd[1703]: status update time (15.237 seconds)
Sep 18 02:03:05 pmx1 pvestatd[1703]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Sep 18 02:03:12 pmx1 pvestatd[1703]: store1: error fetching datastores - 500 read timeout
Sep 18 02:03:12 pmx1 pvestatd[1703]: status update time (15.246 seconds)
Sep 18 02:03:20 pmx1 pvestatd[1703]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Sep 18 02:03:27 pmx1 pvestatd[1703]: store1: error fetching datastores - 500 read timeout
Sep 18 02:03:27 pmx1 pvestatd[1703]: status update time (15.233 seconds)
Sep 18 02:03:32 pmx1 pvescheduler[1800519]: VM 101 qmp command failed - VM 101 qmp command 'backup' failed - backup connect failed: command error: http request timed out
Sep 18 02:03:32 pmx1 pvescheduler[1800519]: ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup connect failed: command error: http request timed out
Sep 18 02:03:32 pmx1 pvescheduler[1800519]: INFO: Backup job finished with errorsIn the Syslog of the PBS there's nothing unusual, as far as I can see:
Code:
Sep 18 01:37:09 pbs1 proxmox-backup-proxy[787]: rrd journal successfully committed (25 files in 0.206 seconds)
Sep 18 02:00:18 pbs1 proxmox-backup-proxy[787]: starting new backup on datastore 'store1' from ::ffff:10.x.x.x: "ns/pmx1/vm/100/2025-09-18T00:00:12Z"
Sep 18 02:01:31 pbs1 proxmox-backup-proxy[787]: Upload backup log to datastore 'store1', namespace 'pmx1' vm/100/2025-09-18T00:00:12Z/client.log.blob
Sep 18 02:01:31 pbs1 qemu-ga[530]: info: guest-ping called
Sep 18 02:01:31 pbs1 qemu-ga[530]: info: guest-fsfreeze called
Sep 18 02:07:12 pbs1 proxmox-backup-proxy[787]: rrd journal successfully committed (25 files in 0.104 seconds)
Sep 18 02:17:01 pbs1 CRON[42741]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 18 02:17:02 pbs1 CRON[42742]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 18 02:17:02 pbs1 CRON[42741]: pam_unix(cron:session): session closed for user rootBoth the PVE (8.4.13) and the PBS (3.4.6) are up-to-date.
Code:
pmx1:~# pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.8.12-15-pve)
pve-manager: 8.4.13 (running version: 8.4.13/5b08ebc2823dd9cb)
proxmox-kernel-helper: 8.1.4
proxmox-kernel-6.8: 6.8.12-15
proxmox-kernel-6.8.12-15-pve-signed: 6.8.12-15
proxmox-kernel-6.8.12-14-pve-signed: 6.8.12-14
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
ceph-fuse: 17.2.7-pve1
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
intel-microcode: 3.20250512.1~deb12u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.2
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.2
libpve-cluster-perl: 8.1.2
libpve-common-perl: 8.3.4
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.7
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.6-1
proxmox-backup-file-restore: 3.4.6-1
proxmox-backup-restore-image: 0.7.0
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.4
proxmox-mail-forward: 0.3.3
proxmox-mini-journalreader: 1.5
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.13
pve-cluster: 8.1.2
pve-container: 5.3.2
pve-docs: 8.4.1
pve-edk2-firmware: 4.2025.02-4~bpo12+1
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.2
pve-firmware: 3.16-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.5
pve-qemu-kvm: 9.2.0-7
pve-xtermjs: 5.5.0-2
qemu-server: 8.4.3
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.8-pve1
Code:
pbs1:~# proxmox-backup-manager versions --verbose
proxmox-backup 3.4.0 running kernel: 6.8.12-15-pve
proxmox-backup-server 3.4.6-1 running version: 3.4.6
proxmox-kernel-helper 8.1.4
proxmox-kernel-6.8 6.8.12-15
proxmox-kernel-6.8.12-15-pve-signed 6.8.12-15
proxmox-kernel-6.8.12-13-pve-signed 6.8.12-13
proxmox-kernel-6.8.12-9-pve-signed 6.8.12-9
ifupdown2 3.2.0-1+pmx11
libjs-extjs 7.0.0-5
proxmox-backup-docs 3.4.6-1
proxmox-backup-client 3.4.6-1
proxmox-mail-forward 0.3.3
proxmox-mini-journalreader 1.5
proxmox-offline-mirror-helper 0.6.7
proxmox-widget-toolkit 4.3.13
pve-xtermjs 5.5.0-2
smartmontools 7.3-pve1
zfsutils-linux 2.2.8-pve1Is there anything more that I can show to help find the reason of the error?
Thank you in advance for the help!
