PBS datastore

[inxsible]

Hello,

I have Proxmox & TrueNAS on my network. I created a PBS container on Proxmox and want to use the TrueNAS as the backup datastore for PBS since I have already setup ZFS Replication from TrueNAS to another server. So when I use TrueNAS for PBS, I would get 2 backups for the price of 1.

For this, I created a new NFS share on TrueNAS called pbsDatastore. I added this NFS storage on Proxmox and then used a mountpoint in the PBS container to allow the PBS container to access the share. I have done this using the conf file:

mp0: /mnt/pve/pbsDatastore,mp=/mnt/freenas/pbsDatastore

Then I created a Datastore in PBS by adding the absolute path. Finally I added this datastore into Proxmox by adding a Proxmox Backup Server as the storage and put in the IP and user/password.

However, when I try to backup to this PBS datastore I keep getting the following error:

....
....
INFO: Starting backup protocol: Wed Sep 14 16:32:24 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp2478968_102/etc/vzdump/pct.conf' to 'root@pam@192.168.1.33:8007:pbsDatastore' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@192.168.1.33:8007:pbsDatastore' as root.pxar.didx
INFO: Error: unable to get shared lock - EIO: I/O error
INFO: restarting vm
WARN: old systemd (< v232) detected, container won't run in a pure cgroupv2 environment! Please see documentation -> container -> cgroup version.
INFO: guest is online again after 15 seconds
ERROR: Backup of VM 102 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp2478968_102/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 102 --backup-time 1663191142 --repository root@pam@192.168.1.33:pbsDatastore' failed: exit code 255
INFO: Failed at 2022-09-14 16:32:37
INFO: Backup job finished with errors
TASK ERROR: job errors

What can be wrong here?

 

[Dunuin]

Make sure that the "backup" user inside the LXC got rights to write to that datastore. It should be UID 34 inside the LXC so because of user remapping UID 100034 on the host needs rights to access that NFS share, unless you edited the user remapping.
Also keep in mind that PBS was designed with local SSDs in mind. HDDs are not recommended to use because of the terrible IOPS performance and NFS/SMB adds additional latency screwing up the IOPS performance even more. Might be good enough for a homelab and will work fine, but don't wonder when maintaince tasks will run for hours/days where your NAS might be very badly slowed down.

 

[inxsible]

Make sure that the "backup" user inside the LXC got rights to write to that datastore. It should be UID 34 inside the LXC so because of user remapping UID 100034 on the host needs rights to access that NFS share, unless you edited the user remapping.
Also keep in mind that PBS was designed with local SSDs in mind. HDDs are not recommended to use because of the terrible IOPS performance and NFS/SMB adds additional latency screwing up the IOPS performance even more. Might be good enough for a homelab and will work fine, but don't wonder when maintaince tasks will run for hours/days where your NAS might be very badly slowed down.

Thanks @Dunuin.

I double checked the owner for the pbsDatastore from the Proxmox shell and the owner & group is 100034 as expected.

As for the latency regarding IOPS and network (due to NFS) -- yes I did read about it but then again, this is just my home and I have about 16 CTs and 1 VM. So definitely not a large system. I was hoping to perform the backup jobs late in the night so as to avoid usage during backups. Will the backup jobs be that slow that it will take days even after the first time?

I guess I will find out when I am able to setup my first backup job.

 

[Dunuin]

Thanks @Dunuin.

I double checked the owner for the pbsDatastore from the Proxmox shell and the owner & group is 100034 as expected.

And rights for the NFS share on TrueNAS? Did you set something at "maproot" for example? And is that dataset you share via NFS on TrueNAS owned by a user with UID 100034?

As for the latency regarding IOPS and network (due to NFS) -- yes I did read about it but then again, this is just my home and I have about 16 CTs and 1 VM. So definitely not a large system. I was hoping to perform the backup jobs late in the night so as to avoid usage during backups. Will the backup jobs be that slow that it will take days even after the first time?

I guess I will find out when I am able to setup my first backup job.

The problems are not the backups jobs, but verify and GC jobs. Lets say you got 1TB of backups with a average chunk size of 2MB. Then doing a GC will have to read + write the metadata of half a million of chunk files. HDDs usually only can handle 100-200 IOPS, so doing 1,000,000 IO with 100 IOPS would be 10,000 seconds (so nearly 3 hours) just for freeing up space of backups you already have pruned. A SSD could do that is seconds or a couple of minutes instead of hours.

 

[inxsible]

And rights for the NFS share on TrueNAS? Did you set something at "maproot" for example? And is that dataset you share via NFS on TrueNAS owned by a user with UID 100034?

I did indeed have maproot set. I removed it, restarted the NFS service on TrueNAS, but now I get this error instead.

INFO: Starting backup protocol: Wed Sep 14 20:28:34 2022
INFO: Error: Permission denied (os error 13)
INFO: restarting vm
WARN: old systemd (< v232) detected, container won't run in a pure cgroupv2 environment! Please see documentation -> container -> cgroup version.
INFO: guest is online again after 3 seconds
ERROR: Backup of VM 102 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp2836834_102/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 102 --backup-time 1663205313 --repository root@pam@172.16.5.33:pbsDatastore' failed: exit code 255
INFO: Failed at 2022-09-14 20:28:36
INFO: Backup job finished with errors
TASK ERROR: job errors

I don't have a user with UID/GID 100034 on the TrueNAS box --

The problems are not the backups jobs, but verify and GC jobs.

I see what you are saying. Maybe after my initial evaluation, I can switch to a HP T730 thin client that I have lying around as my PBS. I will have to buy a 1TB m.2 drive for it.

I actually didn't go that route initially because that box can only fit 1 ssd and I thought it would be better to have the PBS OS and datastore on separate drives. So I installed PBS as an LXC on Proxmox itself and was planning to backup the CTs & VMs. and I would have manually backed up the PBS container using the vzdump backup.

Another option I have is to use a Dell Optiplex SFF box with 2x1TB SSDs. I would just need to get the drives and the PSU -- since the PSU for that box is non-functional.

I don't want to create a new user 100034 on TrueNAS only for evaluating PBS. Maybe I'll just get the 1TB m.2 SSD for the T730 and go that route.

 

[Dunuin]

Here it was working with "maproot user: root", "maproot group: wheel" at the NFS share page and dataset owned by "root:wheel" with chmod 777 and without ACLs.

 

[inxsible]

I have the mapAll user as root and mapAll group as wheel and was able to backup 2 containers. so it seems to be working.

I'll set up some regular jobs and see how the IOPS affect it for these 2 test containers and then start prep for moving to SSDs somehow.