Backup randomly freezes node

[carlosmp]

Hi,

We seem to be having an issue with our nodes. We were running nightly backups of all VMs across our 3 node setup. However, we would experience random lockups on a random node that would require a reboot of the node. Our nodes are also running ceph. Primary network adapter for management is a gigabit, virtual networks are on 10Gbit adapter, and ceph/storage/backup are on Infiniband network. backup is to an NFS device that also backs up our promox 3.4 nodes without any issues.

I've even tried to run the backup jobs in two batches for the lxc machines and the kvms. There was no change in the behavior. It would run fine for 1 or two days, and then lockup the node.

pveversion -v

proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.2-1-pve: 4.2.2-16
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-30
qemu-server: 4.0-46
pve-firmware: 1.1-7
libpve-common-perl: 4.0-43
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-21
pve-container: 1.0-37
pve-firewall: 2.0-15
pve-ha-manager: 1.0-18
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
openvswitch-switch: 2.3.2-2

Trying to get to the bottom of this...

Thanks in advance,

Carlos.

 

[fireon]

Have you some syslog from the important times?