Backup to NFs crashes Nodes and VMS

[gerdnl]

when running full backups to NFS all my nodes go offline and vms to
what could cause this?

even from 1 of my vms all data is gone

 

[mir]

- Network problems
- Slow NFS server
- Slow CPU in proxmox host
etc

If network and/or NFS server is an issue you can try this quick fix:
1) sudo nano /etc/vzdump.conf
2) Add: tmpdir: /var/lib/vz/vztmp

Above setting means that workdir for creating compressed backup is done on the host instead of your NFS server and that the file is first transferred to the destination when the compressed file is ready (Transferring one big file instead of a lot of small fragments increases performance greatly)

 

[gerdnl]

could a to big Jumbo frame setting on the nfs cause this?

 

[mir]

No. By network problems I think of congestion when the backup runs and especially if the backup shares the same nic with cluster communication and storage for other VM's. If backup is network intensive it could be an idea to create a separate network for backup. Anyways, try my suggestion first and see if this solves your problem.

 

[gerdnl]

will do thanks, ill keep you uptodate

 

[gerdnl]

just for info, the NFS is a QNAP connected to a public ip, the NFS transfer rates seemed slow to me indeed.

 

[gerdnl]

on 1 vm im getting no bootable device.
seems like that vm got corrupted somehow

anyway to fix that?

 

[gerdnl]

seems that the vm settings are whiped out

 

[mir]

I also have a Qnap for backup. Unless you have a pro model with a capable CPU NFS speed will not be impressive.

 

[gerdnl]

Jul 31 21:43:01 INFO: status: 33% (120494489600/365072220160), sparse 0% (1635422208), duration 9734, 13/13 MB/s
Jul 31 21:49:09 INFO: status: 34% (124143403008/365072220160), sparse 0% (1687859200), duration 10102, 9/9 MB/s
Jul 31 21:51:58 ERROR: unable to find configuration file for VM 106 - no such machine
Jul 31 21:51:58 INFO: aborting backup job
Jul 31 21:52:00 ERROR: Backup of VM 106 failed - unable to find configuration file for VM 106 - no such machine

 

[gerdnl]

its a ts 859-u

 

[gerdnl]

looks like the backup got intterupted and messed things up, is this fixable?

 

[gerdnl]

/etc/pve/nodes/pve03/qemu-server/106.conf seems empty

 

[mir]

Is the file available on another node?

 

[gerdnl]

the weird thing is that 106 is not on pve03 but on pve01 (the original location)
both conf files are empty do

 

[gerdnl]

which file am i looking for ?
the conf files i find for 106 are all empty

 

[gerdnl]

the disk file should have 106 in it also right, cant find it

 

[mir]

If all copies are empty then you have an issue. Any old backup of the vm (the backup will contain the config file)?

 

[mir]

the weird thing is that 106 is not on pve03 but on pve01 (the original location)
both conf files are empty do

Sounds like a ha-migration was done while doing backup. Does your fencing work?

 

[gerdnl]

mm looks like im out of luck, no disk file anywhere