VM randomly lost network and qemu-agent on backup

[marti.cardus]

Hi there,


We have a PVE cluster backing up to PBS, and randomly, some VMs lost his network transit and qemu-agent is locked. We cannot connect to these VMs by SSH or KVM Console. When I restart these VMs I can access again, but eventually, they are blocked again some minutes later.
The solution of this issue, is reboot the hypervisor

Any ideas?

Kind regards

 

[Moayad]

Hi,

Anything in dmesg or journalctl?

How looks your network configuration?

 

[marti.cardus]

Hi,

I cannot see anything strange in dmesg or journalctl.

Our network configuration is "simple" (I think), we have one bond for public network access and another for storage network (configuration is attached)

Spoiler: network configuration

auto lo
iface lo inet loopback

auto enp1s0f0
iface enp1s0f0 inet manual
mtu 9000

auto enp1s0f1
iface enp1s0f1 inet manual
mtu 9000

auto eno1
iface eno1 inet manual
mtu 9000

auto eno2
iface eno2 inet manual
mtu 9000

auto bond1
iface bond1 inet manual
bond-slaves enp1s0f0 enp1s0f1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
mtu 9000
bond-lacp-rate 1

auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
mtu 9000
bond-lacp-rate 1

auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0

auto bond1_511
iface bond1_511 inet static
address 10.60.2.X/24
vlan-id 511
vlan-raw-device bond1
mtu 9000

auto bond1_512
iface bond1_512 inet static
address 10.61.2.X24
vlan-id 512
vlan-raw-device bond1
mtu 9000

auto bond1_513
iface bond1_513 inet static
address 10.61.3.X/24
vlan-id 513
vlan-raw-device bond1
mtu 9000

auto bond0_22
iface bond0_22 inet static
address 10.10.2.X/24
gateway 10.10.2.1
vlan-id 22
vlan-raw-device bond0

All VM are connected to vmbr0

Thanks

 

[mailinglists]

It probably locks fs using qemu agent, but then fails to unlock it, due to some reason like unable to create snapshots, or whathever...
If you just wish to get on with your life, you can try and disable qemu agent, as you will probably do not want to disable backups or replication.

 

[marti.cardus]

Hi,

Sounds like an interesting option, but we have locked virtual machines that weren't backed up