NFS Timeout & No backup failure emails

Archmatux

Renowned Member
Oct 18, 2016
16
5
68
32
I am currently using a Synology NAS for nightly backups.

As the NAS is used only for this purpose it enters hibernation to conserve power when not in use.
The NAS wakes up when it detects network traffic, however it can take approximately 10 seconds to do so.

I can see that the default timeout to check whether NFS storage is available before a backup runs is only 2 seconds which means that my backups are failing.

To resolve this I have a few options.
I could just create a cron job that runs a few minutes before my scheduled backups that sends several ping packets to the NAS. Alternatively I could edit the timeout in
Code:
/usr/share/perl5/PVE/Storage/NFSPlugin.pm
to a larger value. However this would affect all NFS storage and not just the storage used for backups.
I could also edit the cronjob that the backup task creates.

None of the above solutions are very elegant as the first requires manual intervention if and when the backup schedule is changed. The second may break due to system updates. The third is likely to break if backup settings are changed in the GUI.

Is there a clean way to increase the NFS timeout for a particular NFS storage target? If not, is there a way to trigger a script as a pre-backup hook that won't break if the backup schedule is edited in the WebGUI?

Ideally it would be nice if the WebGUI allowed optional pre and post commands to be run in the backup configuration.


My other cause for concern is that although the backup failed, Proxmox did not send an email notification.

I have tested my email configuration and it is working correctly.
I have also received other emails from Proxmox recently due to S.M.A.R.T errors on a disk so I know that email is working.

I can see the following in the log:

Code:
Task viewer: Backup
OutputStatus

Stop
TASK ERROR: storage 'SynologyBackup' is not online

However, no email is sent due to this failure.

Version currently installed:

Code:
proxmox-ve: 4.3-66 (running kernel: 4.4.19-1-pve)
pve-manager: 4.3-1 (running version: 4.3-1/e7cdc165)
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-46
qemu-server: 4.0-88
pve-firmware: 1.1-9
libpve-common-perl: 4.0-73
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-61
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6.1-6
pve-container: 1.0-75
pve-firewall: 2.0-29
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.4-1
lxcfs: 2.0.3-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
zfsutils: 0.6.5.7-pve10~bpo80
openvswitch-switch: 2.5.0-1

Any assistance or recommendations are much appreciated.
 
See this: /usr/share/doc/pve-manager/examples/vzdump-hook-script.pl
You should intercept 'job-start' and add a sleep 10.

I suspect that this may not work as the issue appears to occur before vzdump is called?

From what I have read, before Proxmox triggers vzdump it apparently runs:
Code:
/usr/bin/rpcinfo -p $server
to check whether the storage is online with a 2 second timeout.

As this fails vzdump is never triggered. This also explains why no email notification is send as vzdump is also responsible for the notification.

I am unsure if the above is accurate and will implement the hook anyway to test.
 
Will this won't work as the issue appears to occur before vzdump is called?

From what I have read, before Proxmox triggers vzdump it apparently runs:
Code:
/usr/bin/rpcinfo -p $server
to check whether the storage is online with a 2 second timeout.

As this fails vzdump is never triggered. This also explains why no email notification is send as vzdump is also responsible for the notification.
If this is the case I would file a bug against vzdump on bugzilla.proxmox.com since this voids the start-job hook.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!