Automated snapshot backup issue

[fang0654]

I have a 3-node cluster running Proxmox. I just upgraded it to 2.3 last weekend. Yesterday, when we came in, one of the VMs on one of the cluster was hard locked. We rebooted it, and it came up with the dreaded "NTLOADER MISSING".

Our monitoring showed it failed about an hour after its backup started. The backup itself said it completed without any issues. We restored from a backup a day earlier, and the machine running on the node was horribly slow (disk IO was maxxing at about 5 MB/s).

We finally scrapped it, and restored from a backup onto the main node, and it started running fine.

Today, I came in, and a windows VM sitting on a different node wasn't responding. I connected into it, and there are hundreds of "delayed write failed" error messages sitting on the screen. I rebooted it, and while it did come back up (thank god!) it is running very very slow.

The only thing I can find in the logs is some entries like this around the backup (this is in syslog)

Mar 21 00:30:48 node2 pvestatd[1895]: WARNING: command 'df -P -B 1 /mnt/pve/backup-server' failed: got timeout

All 3 nodes backup to the same NFS share (backup-server)

Not really sure where to go from here to debug the issue. For now I'm going to suspend backups.

 

[malakatronis]

Have you checked the network connection and the health of the NFS hard drives?

 

[fang0654]

Have you checked the network connection and the health of the NFS hard drives?

The NFS drives are running fine. Being that all 3 nodes backup at roughly the same time, I can definitely see the switch getting a little bit overwhelmed, but the backups don't give any error, and those log errors with the nfs drives happen throughout the day.

 

[malakatronis]

It seems to me that the NFS drive is not fast enough to sustain such a high load. Try to space out the backup times for each node and see what happens.

I have no idea why you get those error messages, though :-/