What was done during this time.
1. Updated virtio drivers on all Windows machines - network card settings are no longer automatically reset.
2. Installed qemu guest agent on all Windows machines.
3. Updated Proxmox and all packages to the latest versions.
Everything works correctly until the backup of virtual machines is started, which is configured according to the schedule, weekly, on the night from Friday to Saturday. The backup runs without errors, but today we noticed:
1. 2 of the 6 windows machines are not available even through noVNC
Code:
VM 114 qmp command 'change' failed - unable to connect to VM 114 qmp socket - timeout after 600 retrie
TASK ERROR: Failed to run vncproxy
2. 5 out of 6 linux machines (more precisely, Mikrotik CHR) are not accessible even through the console.
3. While we were restoring the operation of the problem machines by stopping/starting, other machines that were working earlier became unavailable.
As a result, we rebooted the hypervisor and after that everything worked. It is logical to conclude that problems arise during the backup process. We can't find a solution on our own. SOS
Code:
# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.103-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-7
pve-kernel-helper: 6.3-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.101-1-pve: 5.4.101-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-6
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-3
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-8
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.3-pve2