NFS share dead, backups frozen

Stewge

Renowned Member
Feb 11, 2010
40
3
73
Hi guys,

Noticed an issue with one of our VZ containers over the weekend and tracked the issue back to a frozen backup.

We have a FreeNAS server running to serve a NFS share for backups. The scheduled backup process froze while initiating the backup which has resulted in many issues:

1) NFS mount is stuck: "df -h", "mount" or navigating to the share freezes the console session immediately.
2) Attempting to unmount results in "umount.nfs: /mnt/pve/xxxxxxx: device is busy".
3) Can no longer log into the proxmox web interface (no idea why this is, the login dialog sits on "Please wait..." for a minute and results in "login failed"). SSH still works. Tried Chrome, IE and Firefox.
4) Can't run "qm list". Responds with:
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
5) /var/lib/vz got very full. Could this cause everything to die even though the backups weren't even landing there? (going to NFS)
6) Had to "kill -9" all vzdump processes associated with the NFS share which froze.
7) Can't run "lsof" to see if any other processes are tying up the NFS share, therefore can't umount. Just sits on empty command line.
8) I have to run commands in Screen sessions just in case they freeze up the terminal.

Apart from all this, the VMs are all functioning. I'm hesitant to reboot during the day as this is a client's production machine. On top of that, I've run into similar issues like this which prevented clean rebooting due to the inability to unmount the NFS shares. This is the one reason I hate NFS with a passion, no matter how stable my network seems to be, it's the one protocol which, when it dies, goes in a huge ball of flames.
 
If you only use that for backup, you can try to mount with option 'soft' (see 'man nfs')
 
If you only use that for backup, you can try to mount with option 'soft' (see 'man nfs')

How do I add the soft option to the PVE conf file?

Currently it looks like:
nfs: nfs_xxxxx
path /mnt/pve/nfs_xxxxx
server 10.100.100.100
export /mnt/raid_storage/xxxxx/proxmox
options vers=3
content backup
maxfiles 5

I'm assuming I modify line 5 to something like this? :
"options vers=3,soft"
 
On the same note, what processes are safe to restart to get access to the Proxmox GUI again?

The KVMs are still running and I don't want to cause them to die if it isn't necessary.

I'm guessing /etc/init.d/pve-manager and /etc/init.d/pvedaemon should do it?
 
On another note, we've found that directly backing up to freenas [ version 8.3 beta1 ] is unreliable. Doing so caused at least weekly nfs issues. Instead we backup to a local pve host that has extra storage . Then use a cronjob to rsync the dump directory to freenas for long term backup [ using zfs snapshots ].

I do not know the cause, maybe it is the interaction of the 2.6.32 kernel or rsync libs etc.

On desktops we tried to use freenas for samba and nfs sharing. Sometimes there would be lags and over all the feel was slow. We switched to using samba and nfs in a kvm and have not had any lags since. We do not like slow keyboard or slow network file access .

We use freenas for long term backups and nfs storage for zoneminder and bluecherry video . Freenas has a nice web/gui for zfs and more. In the future we hope it will work reliably with the linux version in pve.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!