Backup failed - Operation not permitted

[alc]

Since creating two W10 VM's, they couldn't backup for days with the same error :

(Example VM '106') :

106

W10x32-G01

FAILED

00:00:00

unable to open file '/mnt/pve/nasa-backup2/dump/vzdump-qemu-106-2022_11_21-02_10_37.tmp/qemu-server.fw.tmp.3577369' - Operation not permitted

Couldn't find the file mentioned in the logs (makes sense because it's a tmp).
These VM each have an active Firewall that uses Security groups (and I'm guessing it's related because of the '.fw.'), but I don't know what to do - I need these firewall rules.

How can I solve this issue ?

 

[fabian]

is your backup target storage a network share? is it by chance exposed via CIFS/SMB, and the NAS providing it is running TrueNAS? if so, try adding the "noserverino" mount option to your mount on the PVE side..

 

[alc]

Hi fabian,
The target storage is indeed on a TrueNAS, but exposed via NFS.
I've added the option in /etc/pve/storage.cfg anyways but I'm guessing the option may not apply to NFS.
Re-running a backup manually for one of these VM's failed with the same error after the edit (but note that I didn't take any action after editing storage.cfg - so I'm unsure if the option has been applied).

 

[fabian]

noserverino is CIFS only. you can check the "DATA AND METADATA COHERENCE" section in "man 5 nfs" for possibly relevant options in the NFS case.

 

[alc]

Ok thanks,
I've tested most options mentioned in the "DATA AND METADATA..." section of 'man 5 nfs' (in the Proxmox VE server), without success.

My procedure was the following :
- Edit storage.cfg, adding an 'options <option>' line to the desired storage entry
- Unmount the target storage with 'umount <mount_path>'
- Launch the backup for one of the faulty VM's menu, in the web interface, using the "Backup now" button.
RESULTS :
- In less than half a second, the backup fails consistently with the error from the original post.

options that were tested (separately, restarting the whole procedure for each) were :
- noac
- nocto
- nolock
- lookupcache = positive
- lookupcache = none

 

[fabian]

what exact truenas version are you running? could you give details about the truenas side settings as well that would help us reproduce?

 

[alc]

Anything you need.

Here is some information :
- TrueNAS is CORE version 12.0-U8.1
- NFS share has 'MapAll User : root' and only allows access to PVE IP's. Other options are default.
- Storage (aka 'pool') options are as follows (on that particular pool being shared to PVE) :

 

[alc]

Also note that disabling or deleting all Firewall rules didn't change anything.

And here is a pastebin link to the Backup log, just in case :
https://pastebin.com/ACzChayV

 

[dcsapak]

could not really reproduce with a default install + pool + share...
did you maybe change the permissions/acls in any way on the truenas install?

 

[alc]

Yes there are ACL's but there is also another VM on the same host, with the same 'local drive' config. and the same backup settings that backs up every night without issues to the same NFS storage.

But again, doesn't the original error log point to the Firewall one way or another ?
Because that is a difference I have between the working VM's and those who don't backup.

 

[dcsapak]

Yes there are ACL's but there is also another VM on the same host, with the same 'local drive' config. and the same backup settings that backs up every night without issues to the same NFS storage.

mhmm.. weird

But again, doesn't the original error log point to the Firewall one way or another ?
Because that is a difference I have between the working VM's and those who don't backup.

maybe, but i tested here too with a vm firewall configured and without, it made no difference
(fwiw, i tested container + vm; with and without firewall)

 

[alc]

Two more details :

1. I think that all VM's that don't backup are Clones of a single W10 VM. The original does backup fine.
   There are the two aformentioned active VM's that don't backup, but 3 others that are not in production (and thus not in scheduled backups) won't backup either, if I launch a Backup manually on them.
   It's actually untrue. Out of 6 VM's, 3 were from one clone and 3 from another. Out of these 6, 5 don't backup. The one that does is one of the original clones, but it doesn't relate to the other group of 3 VM's that don't backup, and it means the second "original clone" fails to backup too.
   The only thing they have in common is being all W10 VM's (but note some W2008Server and W2016Server VM's will backup).
   Sorry for the false lead.
   
   
2. This error does not happen at all when choosing another NFS Export on the same server, that has the same ACL's (be it in the server itself or in /etc/pve/<storagename>), and the same options in TrueNAS and in /etc/pve/storage.cfg. They differ in contents, because the second share is not usually used for backups, I just added the flag for this test.
   Note that deleting my usual NFS backup storage (in PVE) and re-creating it doesn't have an effect on the issue, neither does renaming it.
   I also tried deleting all files in 'dump' that mentionned the VM number I was trying to backup, or the original clone's number.