unknown node status when vm backup job

[dadadadadadadad]

Help, has anyone encountered a similar problem, a cluster of 3 nodes, one random node freezes, with a VM backup task, restarting networking.service helps and the node comes to life, the VM comes to life after unlocking qm, but the network on the VM does not work if you migrate VM to another node, then the network works or remove and add a network card.
I don’t know if this is connected or not, one node changes the ssh key, you have to delete the key on two nodes and add it again

 

[Tahsin]

I have a similar issue but everything works as normal on the node that says Status: Unknown on all the Containers and VMs.

 

[NiceRath]

We have seen a similar behavior on LXC backups.
On multiple clusters/environments (work, side-business and home-lab, local backup storage and remote PBS).

What I've seen this ocurrs while processing one container.


State:

• The backup-job gets stuck.
  
  
  
  
• The container that is currently processed by the backup-job dies.
• All other containers and VMs are still working without any issues.
• PVE cli tools like `pct` or `qm` are non-responsive
• We found the only way to recover the node without a reboot is to unlock the LXC and kill it:
  

# try to get the LXC-ID
pct list
## if that does not work - you can find the ID via
ls -l /etc/pve/nodes/<NODE>/lxc/
grep hostname < /etc/pve/nodes/<NODE>/lxc/<ID>.conf


# check it is not locked
pct unlock <ID>

# try to stop the container
pct shutdown <ID>
## OR
pct stop <ID>

# if that does not work - get its process and kill it
ps aux | grep <ID> | grep lxc
# should be: '/usr/bin/lxc-start -F -n <ID>'
kill -9 <PID>

After that the node is back to being online and the PVE tools and services work as expected.