Problem backing up via PBS

[offerlam]

Hi all,
I have a singel node proxmoxve running serveral VM herunder also a PBS server for which to take backups of the VMs in my proxmox server.
The PBS server have a mounted CIFS connection to a synology NAS.
I have tested the connection and its working.
I have add my PBS in PVE storage as a VZdump target. The target path points to the CIFS mount point.

When i try to do a backup of a VM i get this error

INFO: starting new backup job: vzdump 114 --remove 0 --mode snapshot --storage PBS02 --node pve --mailto support@kun-it.dk
INFO: Starting Backup of VM 114 (qemu)
INFO: Backup started at 2023-11-25 12:24:54
INFO: status = running
INFO: VM Name: FTP02
INFO: include disk 'scsi0' 'SSD03:114/vm-114-disk-0.qcow2' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/114/2023-11-25T11:24:54Z'
ERROR: VM 114 qmp command 'backup' failed - backup connect failed: command error: not a valid user id
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 114 failed - VM 114 qmp command 'backup' failed - backup connect failed: command error: not a valid user id
INFO: Failed at 2023-11-25 12:24:55
INFO: Backup job finished with errors
TASK ERROR: job errors

im not sure what user were are refering to?
im using a root a user for connection on both PBS and PVE

 

[sb-jw]

ERROR: Backup of VM 114 failed - VM 114 qmp command 'backup' failed - backup connect failed: command error: not a valid user id

It seems that you have not stored the PBS correctly. Check the settings again for how you integrated the PBS into PVE.

 

[offerlam]

Hi sb-jw

im not sure what im doing wrong?
also im new at PBS

on PVE i go to storage and click add and chose PBS server
I give it a ID which is just a name
under server i give the ip og PBS server
username is root@pam
password is... the password
datastore is NAS01 which is the name of the storage device in PBS which in turn points to /mnt/NAS01 which connect via CIFS to the synology nas.

fingerprint i get from the PBS dashboard.

All this fly right through and the PBS device is added as storage in PVE.

What am i doing wrong here? or am i misunderstanding you?

 

[zyghom]

" not a valid user id" - that suggests the issue - just I am not sure if that suggest problem on PVE or PBS
go to shell on PBS and check this:

root@pbs:~# cat /etc/proxmox-backup/datastore.cfg
datastore: pbs-nfs
        gc-schedule 19:00
        path /nfs/pbs-nfs

then this:

root@pbs:~# ls -la /nfs/pbs-nfs
total 8295
drwxrwxrwx     5 backup backup     7 Nov 24 20:15 .
drwxr-xr-x     4 root   root       0 Nov 24 13:12 ..
drwxr-x--- 65538 backup backup 65538 Nov 20 06:27 .chunks
drwxr-xr-x     9 backup backup     9 Nov 24 14:14 ct
-rw-r--r--     1 backup backup   312 Nov 24 20:15 .gc-status
-rw-r--r--     1 backup backup     0 Nov 20 06:01 .lock
drwxr-xr-x    18 backup backup    18 Nov 24 14:14 vm

I am 100% sure that share connected to the PBS must have owner of "backup" - so check this first
(in my case it is NFS not Samba but the idea is the same)

 

[offerlam]

" not a valid user id" - that suggests the issue - just I am not sure if that suggest problem on PVE or PBS
go to shell on PBS and check this:

root@pbs:~# cat /etc/proxmox-backup/datastore.cfg
datastore: pbs-nfs
        gc-schedule 19:00
        path /nfs/pbs-nfs

then this:

root@pbs:~# ls -la /nfs/pbs-nfs
total 8295
drwxrwxrwx     5 backup backup     7 Nov 24 20:15 .
drwxr-xr-x     4 root   root       0 Nov 24 13:12 ..
drwxr-x--- 65538 backup backup 65538 Nov 20 06:27 .chunks
drwxr-xr-x     9 backup backup     9 Nov 24 14:14 ct
-rw-r--r--     1 backup backup   312 Nov 24 20:15 .gc-status
-rw-r--r--     1 backup backup     0 Nov 20 06:01 .lock
drwxr-xr-x    18 backup backup    18 Nov 24 14:14 vm

I am 100% sure that share connected to the PBS must have owner of "backup" - so check this first
(in my case it is NFS not Samba but the idea is the same)

Hi and thanks for pitching in

here are the outputs of your requests

root@pbs01:~# cat /etc/proxmox-backup/datastore.cfg
datastore: NAS01
        comment
        gc-schedule daily
        path /mnt/NAS01
        prune-schedule daily

root@pbs01:~# ls -la /mnt/NAS01/
total 4
drwxr-xr-x 2 root root    0 Nov 26 00:01 .
drwxr-xr-x 3 root root 4096 Nov 22 14:43 ..
drwxr-xr-x 2 root root    0 Nov 22 15:01 .chunks
drwxr-xr-x 2 root root    0 Nov 22 14:51 dump
-rwxr-xr-x 1 root root    0 Nov 26 00:01 .gc-status
-rwxr-xr-x 1 root root    0 Nov 22 15:00 .lock
drwxr-xr-x 2 root root    0 Nov 22 15:14 test
drwxr-xr-x 2 root root    0 Nov 25 12:24 vm

You know why its root? cause you have backup as the owner of your NFS storage. Is it the way i added it perhaps? can har chmod it or?

 

[zyghom]

so now, question is: did you share from NAS over NFS or over SMB?

 

[offerlam]

Im using cifs so smb

 

[_gabriel]

cifs datastore for PBS require mount option uid=34,noforceuid,gid=34,noforcegid (= user and group "backup")
and before PBS datastore creation.
You should delete then recreate PBS datastore with correct mount option.

edited: corrected mount options.

 

[offerlam]

cifs datastore for PBS require mount option uid=34,gid=34 (= user and group "backup")
and before PBS datastore creation.
You should delete then recreate PBS datastore with correct mount option.

Hi Gabriel,

I did as you asked. i corrected my FSTAB like this

//172.16.109.139/data/Firma_Backup   /mnt/NAS01   cifs    user=kunit,pass=XXXXXX,,uid=34,gid=34, 0 0

now my ls -la for the mount point looks like this

root@pbs01:~# ls -la /mnt/NAS01/
total 4
drwxr-xr-x 2 backup backup    0 Nov 26 09:51 .
drwxr-xr-x 3 root   root   4096 Nov 22 14:43 ..
drwxr-xr-x 2 backup backup    0 Nov 26 09:45 .chunks
-rwxr-xr-x 1 backup backup    0 Nov 26 09:44 .lock
drwxr-xr-x 2 backup backup    0 Nov 26 09:51 vm

but i still get the error with "not a valid user id"

I have one root left. im not sure if thats why? but also unsure how to change it. Should i delete the entire mount folder and make a new one?

 

[sb-jw]

Maybe this will help? uid=34,noforceuid,gid=34,noforcegid

Should i delete the entire mount folder and make a new one?

What's stopping you from just trying it? It's done in just a few seconds

 

[offerlam]

Hi

You know what you are right im one of those who has to talk to think kind of people. I think that mindset sliped in the post. Because i did try it
I also tried with noforceuid and noforcegid but get the same error.

Would it be better if i used nfs? The reason i skipped it was because speeds was slow and i just needed something to work. So i switched to cifs.

 

[sb-jw]

Would it be better if i used nfs? The reason i skipped it was because speeds was slow and i just needed something to work. So i switched to cifs.

Basically, I would refrain from hanging network drives in the PBS. This can lead to a variety of problems of all kinds.

Take a look at this Guide: https://steven-england.info/2022/10...-share-as-datastore-in-proxmox-backup-server/

 

[offerlam]

Basically, I would refrain from hanging network drives in the PBS. This can lead to a variety of problems of all kinds.

Take a look at this Guide: https://steven-england.info/2022/10...-share-as-datastore-in-proxmox-backup-server/

Hi sb-jw
just so im sure i understand you correctly. When you say network drive you mean CIFS? as opposed to NFS which is not considered a network drive? I'm asking because technically they both connect over the network, so i think im struggling with the terminology "Network Drive"?

 

[wbk]

Hi offerlam,

[NFS] speeds were low and i just needed something to work.

So you had something that just worked?

In addition to @sb-jw 's remark: expect transfer speeds much lower than the maximum your network or your disk can offer. Latency kills the diff-checks when doing incremental backups; network storage (whether NFS or SMB) adds latency.

 

[sb-jw]

Network drives are all drives that are not physically located in the same server that PBS manages.
You operate a backup server, which is supposed to help you get your data back in case of error or even disaster. So if the PBS is not the last station of your data, then you don't have a backup of your NAS and the success of PBS depends on your NAS.

You can do a lot of things and a lot of things work. But that doesn't mean that this is the right way to go and for me the backup server is the final destination of my backups and not just a gateway. Just because I wouldn't do it that way doesn't mean you can't do it anyway. The important thing is that you are aware of the risks and if it suits you, it's okay.

 

[offerlam]

Hi offerlam,

So you had something that just worked?

In addition to @sb-jw 's remark: expect transfer speeds much lower than the maximum your network or your disk can offer. Latency kills the diff-checks when doing incremental backups; network storage (whether NFS or SMB) adds latency.

Now you are just teasing me. I had something working with NFS which was before PBS was introduced. Back then i had just mounted the NAS directly into my PVE server and did backups that way. But the speeds was horrible, close to not functioning. Actually the backup stalled at around 27%.

So i went with SMB to see if that was different. But then along came PBS which had even more restore options then the built in VZdump of PVE. So i tried with that and SMB. And here we are

 

[offerlam]

Network drives are all drives that are not physically located in the same server that PBS manages.
You operate a backup server, which is supposed to help you get your data back in case of error or even disaster. So if the PBS is not the last station of your data, then you don't have a backup of your NAS and the success of PBS depends on your NAS.

You can do a lot of things and a lot of things work. But that doesn't mean that this is the right way to go and for me the backup server is the final destination of my backups and not just a gateway. Just because I wouldn't do it that way doesn't mean you can't do it anyway. The important thing is that you are aware of the risks and if it suits you, it's okay.

Thanks. There are more restore options with PBS then VZdump which is also why im using it. Yes its a gateway to my NAS which will hold the data. But i need the options PBS provides.

When this is working the NAS will offload the data to a second site afterwards so we have two copies. Or maybe that part can be taken care of by PBS aswell. not sure on that one yet.. but the data has to bee offloaded to a second site.

Thanks for the guide i will try and use it. and also see if it works with NFS.

 

[sb-jw]

However, the pure data will only help you to a limited extent. You have to organize the whole thing a bit so that you can use the data again afterwards. For example, you should document permissions or store corresponding notes in the backups so that you know what is behind VM ID 100. Therefore, it may be advisable to back up the PBS or its configurations as well.

 

[wbk]

Now you are just teasing me. I had something working with NFS which was before PBS was introduced.

Ah, I interpreted as that you had PBS configured with an NFS target, my bad.

I'm sure that you thought of it in case it was an option, just to be sure: your NAS does not run Debian or containers? Because in that case, you could install PBS there, instead of on Proxmox itself.

 

[_gabriel]

I should try disabling ext4 journaling to confirm impact, while it's not recommended.