storage '<storage-name>' is not online

[f4242]

Hello,

What can cause this message? Each night, I do a scheduled vzdump of all VM in the cluster. I scheduled them in order that only one host is doing the task at a time to avoid to overload the NFS server. It works great most of the time. Sometime I have a VM failing to be backed up (timeout issues). Every time, I see theses lines on others server in the cluster when doing backup on another host:

Oct 25 02:06:24 pve1 pvestatd[2772]: storage 'nfs-backup' is not online
Oct 25 02:24:04 pve1 pvestatd[2772]: storage 'nfs-storage' is not online
Oct 25 02:34:54 pve1 pvestatd[2772]: storage 'nfs-backup' is not online
Oct 25 02:41:04 pve1 pvestatd[2772]: unable to activate storage 'nfs-backup' - directory '/mnt/pve/nfs-backup' does not exist or is unreachable
Oct 25 02:50:04 pve1 pvestatd[2772]: storage 'nfs-storage' is not online
Oct 25 03:06:34 pve1 pvestatd[2772]: storage 'nfs-backup' is not online
Oct 25 03:28:34 pve1 pvestatd[2772]: storage 'nfs-storage' is not online
Oct 25 03:31:44 pve1 pvestatd[2772]: storage 'nfs-backup' is not online

The storage nfs-backup is my backup destination. The storage nfs-storage hold my ISO images (not in use when doing the backup). Both are hosted by the same NFS server.

I checked my monitoring system about the server hosting the NFS server (FreeNAS box with a Xeon CPU and 32GB of RAM). The load average is generally at or under 0.5 per core. Plenty of CPU time or RAM available. What's wrong?

Thanks.

 

[Jarvar]

Hello,

What can cause this message? Each night, I do a scheduled vzdump of all VM in the cluster. I scheduled them in order that only one host is doing the task at a time to avoid to overload the NFS server. It works great most of the time. Sometime I have a VM failing to be backed up (timeout issues). Every time, I see theses lines on others server in the cluster when doing backup on another host:

Oct 25 02:06:24 pve1 pvestatd[2772]: storage 'nfs-backup' is not online
Oct 25 02:24:04 pve1 pvestatd[2772]: storage 'nfs-storage' is not online
Oct 25 02:34:54 pve1 pvestatd[2772]: storage 'nfs-backup' is not online
Oct 25 02:41:04 pve1 pvestatd[2772]: unable to activate storage 'nfs-backup' - directory '/mnt/pve/nfs-backup' does not exist or is unreachable
Oct 25 02:50:04 pve1 pvestatd[2772]: storage 'nfs-storage' is not online
Oct 25 03:06:34 pve1 pvestatd[2772]: storage 'nfs-backup' is not online
Oct 25 03:28:34 pve1 pvestatd[2772]: storage 'nfs-storage' is not online
Oct 25 03:31:44 pve1 pvestatd[2772]: storage 'nfs-backup' is not online

The storage nfs-backup is my backup destination. The storage nfs-storage hold my ISO images (not in use when doing the backup). Both are hosted by the same NFS server.

I checked my monitoring system about the server hosting the NFS server (FreeNAS box with a Xeon CPU and 32GB of RAM). The load average is generally at or under 0.5 per core. Plenty of CPU time or RAM available. What's wrong?

Thanks.

Hello.
I have been seen some similar issues with my NFS nightly backups as well.




Task viewer: VM/CT 100 - Backup

OutputStatus

Stop
INFO: starting new backup job: vzdump 100 --mailnotification always --compress gzip --storage Castledental1 --mailtoXXXXXXXXXXXquiet 1 --mode stop
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2019-10-26 02:15:02
INFO: status = running
INFO: update VM 100: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: WS2K19Standard
INFO: include disk 'scsi0' 'local-zfs:vm-100-disk-0' 100G
INFO: include disk 'scsi1' 'local-zfs:vm-100-disk-1' 100G
INFO: include disk 'scsi2' 'local-zfs:vm-100-disk-2' 915200M
INFO: stopping vm
INFO: snapshots found (not included into backup)
INFO: creating archive '/mnt/pve/Castledental1/dump/vzdump-qemu-100-2019_10_26-02_15_02.vma.gz'
INFO: starting kvm to execute backup task
INFO: started backup task 'd69f51cc-5b78-49de-9010-f34fdf246285'
INFO: resuming VM again after 28 seconds
ERROR: VM 100 qmp command 'cont' failed - got timeout
INFO: aborting backup job
ERROR: Backup of VM 100 failed - VM 100 qmp command 'cont' failed - got timeout
INFO: Failed at 2019-10-26 02:15:57
INFO: Backup job finished with errors

TASK ERROR: job errors

If I start the backup manually it usually works, but I have to go and maybe wake up the device. Maybe it has been going to sleep. I have to look into it more.

 

[f4242]

Hi, in my case I'm sure the server never goes to sleep.

 

[f4242]

Seem like this issue only occurs on my cluster upgraded to 6.0. My other cluster at 5.4 doesn't have this issue.

 

[Jarvar]

I tried changing the settings in t he NFS share which is on a Synology NAS box. Although the computer didn’t sleep the HDDs were allowed to hibernate after 20 minutes. I changed the setting and will monitor it for now. Last couple days seemed to work, but I will keep an eye on it.

 

[Jarvar]

Still got this error last night. Looks like when i run the backup manually it works, but not always when it's scheduled. Sometimes it runs, sometimes it doesn't. It's not a consistent pattern. Getting a qmp command 'cont' failed - got timeout

 

[Lowrent]

@f4242

It seems that PVE 6.0 has an issue with Synology NAS.
Here another thread that discuss of the problem:
https://forum.proxmox.com/threads/mount-no-longer-works-in-proxmox-6-nfs-synology.56503/post-274714

 

[f4242]

Interesting. Not sure if it's related. My NAS is running FreeNAS, not Synology. Also, PVE is able to connect most of the time to the NFS server, it just glitch randomly when doing vzdump on night.

 

[rspille]

After struggeling some time I found out, that the firewall on my synology nas blocked the nsf mount. So I deactivated and reactivated nfs and the firewall showed a dialog, that it updated the rules – now I'm able to see the nfs mounts on proxmox again…