[SOLVED] Backup to FreeNAS NFS, VM freeze

[SergejA]

Hello.
I installed 3 nodes Proxmox 5.0 cluster.
2 nodes host all needed VMs and 3rd has only one VM with FreeNAS for backups.
FreeNAS has NFS share.
FreeNAS has 10GB RAM, 16GB RAIDZ2
Nodes connected via 1Gb lan.

When I start backup to NFS share, periodicaly I see read/write speed around 0-10MB/s.




At this moment VM, which I backup, freezes. top shows iowait ~ 25
And I see messages like that:



But if I perform backup to local storage, all goes OK, no freezes and speed lags.



Please, help me, what I have to check?

 

[Klaus Steinberger]

This sounds like a problem I had too with vzdump backup -> https://forum.proxmox.com/threads/ha-manager-vms-going-in-error-state.40766/

massive NFS writes (that's not only related to vzdump) tend to fill up all buffers and make machines unresponsible for some seconds. We observe such a behavior also with ubuntu desktops writing large (multi GByte files) to NFS Storages.

I would suggest setting bwlimit inside /etc/vzdump.conf, that helped with my problem. I had even hosts fenced by HW Watchdog during backup

maybe it is better to limit backup bandwith anyway to keep I/O stress down to keep VM's in responsive state. Backup is a good thing of course, but I think it can run in background with lower speed as long as the backups are consistent through snapshotting,

 

[GadgetPig]

Something to try, what if you mount FreeNAS storage as an SMB/CIFS share instead of NFS and retry backup? Do you get the same lags/stutters in this setup?

https://servernetworktech.com/2013/03/proxmox-and-using-samba-for-shared-storage/

 

[SergejA]

I discovered that there was an issue in network cable to NAS server and link was only 100Mb instead of 1Gb.
I changed cable.
And now during 60GB backup there are few freezes about 3-5s. It's already good.

As Klaus Steinberger adviced, I decreased bwlimit and now there are no freezes at all.

This sounds like a problem I had too with vzdump backup -> https://forum.proxmox.com/threads/ha-manager-vms-going-in-error-state.40766/

Thank you!