Hello everyone.
We are having a problem with backups going from a PVE to a PBS and I am not finding where the problem is coming from. Let me explain.
We have a PVE with several VMs and LxC. The backup jobs are done during the midnight-morning hours and sent to a PBS. Several days ago there are one or two VMs, one with Windows Server and the other with Windows 8, that randomly fail in their backup tasks, showing the following error in the task:
This error led me to investigate in depth the PVE logs. In /var/log/syslog I find this error repeated all the time:
I don't know if that influences this particular error.
On the other hand, I left a cron running a ping and an mtr to the PBS, but both showed no packet loss or slowness. I also left a cron running vmstat and iostat, if needed I can upload them.
Finally mention that the failure does not happen all the time and I can't determine why. What I can say is that the failure was recorded on 8/2, 12/2, 15/2, 19/2 and 20/2. Also mention that this is a production environment.
Finally, I paste our version of pve.
I would be very grateful if you could help me to find the fault.
We are having a problem with backups going from a PVE to a PBS and I am not finding where the problem is coming from. Let me explain.
We have a PVE with several VMs and LxC. The backup jobs are done during the midnight-morning hours and sent to a PBS. Several days ago there are one or two VMs, one with Windows Server and the other with Windows 8, that randomly fail in their backup tasks, showing the following error in the task:
INFO: Backup started at 2025-02-20 03:00:11INFO: status = runningINFO: VM Name: pdcINFO: include disk 'ide0' 'local-1-4TB:102/vm-102-disk-0.raw' 128GINFO: backup mode: snapshotINFO: ionice priority: 7INFO: creating Proxmox Backup Server archive 'vm/102/2025-02-20T06:00:11Z'ERROR: VM 102 qmp command 'backup' failed - backup connect failed: command error: http upgrade request timed outINFO: aborting backup jobINFO: resuming VM againERROR: Backup of VM 102 failed - VM 102 qmp command 'backup' failed - backup connect failed: command error: http upgrade request timed outINFO: Failed at 2025-02-20 03:02:11This error led me to investigate in depth the PVE logs. In /var/log/syslog I find this error repeated all the time:
Code:
PVE1 lxcfs[1898]: ../src/utils.c: 316: read_file_fuse: Write to cache was truncatedI don't know if that influences this particular error.
On the other hand, I left a cron running a ping and an mtr to the PBS, but both showed no packet loss or slowness. I also left a cron running vmstat and iostat, if needed I can upload them.
Finally mention that the failure does not happen all the time and I can't determine why. What I can say is that the failure was recorded on 8/2, 12/2, 15/2, 19/2 and 20/2. Also mention that this is a production environment.
Finally, I paste our version of pve.
pveversion -v
Code:
proxmox-ve: 8.2.0 (running kernel: 6.5.11-8-pve)
pve-manager: 8.2.7 (running version: 8.2.7/3e0176e6bb2ade3b)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-10
proxmox-kernel-6.8: 6.8.12-2
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
pve-kernel-5.15.136-1-pve: 5.15.136-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx9
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.3
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.1
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.10
libpve-storage-perl: 8.2.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-4
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.2.0
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.0.7
pve-firmware: 3.13-2
pve-ha-manager: 4.0.5
pve-i18n: 3.2.3
pve-qemu-kvm: 9.0.2-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1I would be very grateful if you could help me to find the fault.
Last edited: