[SOLVED] Datastore creation on NFS share times out, store gets created, backups fail

[Mike Lupe]

First of all I would like to thank you for your enormous efforts to create PBS. Btw, I'm not sure if you anticipated Rust maybe getting integrated into the Linux kernel https://hackaday.com/

I'm having problems with backing up to a Datastore residing on a NFS share. Beta2 installed from .iso on a VM (not a PVE host). My network is quite slow, but the problematic NFS share is on the virtualization host where the PBS VM resides (a separate Univention UCS physical machine).

- While creating a new datastore on an mounted NFS share in the PBS WebUI, after 30 secs (every time, seems like a fixed configured 30sec timeout) I get "Connection error - Timeout." After confirming the error the UI drops back to the datastore creation mask with the fields still filled with the values just entered. But in the background, all the .chunk files are getting created, and about 2 mins after the error popped up, creation of the .chunk files is successful and the datastore appears in the PBS UI. All this happens in the background, while the datastore creation mask is still active in the PBS UI.

- In PVE activating that created PBS storage works without problems, but backing an LXC up ends with this error on PBS side (on PVE side the same error more or less):

2020-07-21T09:59:18+02:00: starting new backup on datastore 'Datastore03_UCSHOST-01': "ct/602/2020-07-21T07:59:10Z"
2020-07-21T09:59:18+02:00: GET /previous: 400 Bad Request: no previous backup
2020-07-21T09:59:18+02:00: add blob "/mnt/nfs/pbs_datastores/datastore03_ucshost-01/ct/602/2020-07-21T07:59:10Z/pct.conf.blob" (188 bytes, comp: 188)
2020-07-21T09:59:24+02:00: POST /dynamic_index: 400 Bad Request: unable to get shared lock - ENOLCK: No record locks available
2020-07-21T09:59:24+02:00: backup ended and finish failed: backup ended but finished flag is not set.
2020-07-21T09:59:24+02:00: removing unfinished backup
2020-07-21T09:59:24+02:00: TASK ERROR: backup ended but finished flag is not set.
2020-07-21T09:59:30+02:00: POST /dynamic_index: 400 Bad Request: unable to get shared lock - ENOLCK: No record locks available

By the way, when deleting that NFS datastore on the PBS side, the folder with all files and folders remain (.chunks/*, ct/*, .lock) and surely when next trying to create a datastore with that same name, I get "unable to create chunk store 'Datastore03_UCSHOST-01' subdir "/mnt/nfs/pbs_datastores/datastore03_ucshost-01/.chunks" - EEXIST: File exists".

After removing the entire folder (directly on the NFS sharing host, otherways it takes too long), recreation of the datastore then works.

On the NFS host, the share is given "backup:root" rights.

May it be possible, that some finalizing tasks do not get finished because of the error, eventhough the process of chunks and the datastore being created continues? And where should I crank up that 30 secs timout? Is it PBS internal, or could it be the local NFS client config?

Thank you again for your great work.
regards

 

[t.lamprecht]

Yes, this is known. The datastore initial directory namespace creation for the .chunkstore should be moved in a task so that it avoids errors on storage which are slower to create many directories.

For now, the task continues in the background - you could check the syslog for completion.
As a workaround you could use proxmox-backup-manager datastore create <id> <path> on the server CLI, this avoids the HTTP synchronous request timeout.

 

[t.lamprecht]

And where should I crank up that 30 secs timout?

This is simply the default timeout for synchronous HTTP requests to the API. Rather than increasing that, long running processes should be moved to a task, then a task UPID can be returned which the client can query for the status.

 

[Mike Lupe]

@t.lamprecht Ok, thank you.

 

[Mike Lupe]

@t.lamprecht datastore creation works indeed flawlessly with cli. But I still have the problems backupping (the same CT) to a completely newly created NFS store (so no old "junk" sticking around). Btw, backuping to another local (ext4) datastore on the PBS works without problems.

PVE error:

"[....]
INFO: Error: HTTP Error 400 Bad Request: unable to get shared lock - ENOLCK: No record locks available
ERROR: Backup of VM 602 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp4131696/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 602 --backup-time 1595348945 --repository bkpuser@pbs@pvepbs-01atastore04_UCSHOST-01' failed: exit code 255
INFO: Failed at 2020-07-21 18:29:12
INFO: Backup job finished with errors
TASK ERROR: job errors"

 

[selund]

Ran into the same problem. The problem running backups seems to be related to the lack of locking on nfs.

 

[fabian]

for the locking issue you can either set insecure_locks on the export, or switch to local locking via mount options (depends on your deployment scenario)

 

[Mike Lupe]

for the locking issue you can either set insecure_locks on the export, or switch to local locking via mount options (depends on your deployment scenario)

Thank you Fabian, this made it work. I first had to clean/reinitiate all mounts and datastores, but then it worked.

 

[semira uthsala]

for the locking issue you can either set insecure_locks on the export, or switch to local locking via mount options (depends on your deployment scenario)

Hi,

I'm testing the datastore backed by NFS share from Truenas core and Qnap NAS. I already set NFS options soft while mounting. my backups are running ok. But what is the proper way to mount NFS storage for the datastore? -o soft,local_lock=all ?

Also I'm not sure which options I need to enable from Truenas/Qnap side. I cannot find any insecure_locks option on both devices.

 

[fabian]

local_lock=all should work