[TUTORIAL] PBS on TrueNAS - Have your cake and eat it too

[ashranpewter]

Thought I'd chime in here real quick, I did this but now I'm running into issues because I upgraded to PBS4 and I can't upgrade the LXC container's kernel (as reported by pbs3to4 --full)

Though it doesn't really seem to affect anything now I can see it causing issues in the future. Is there some easy way to migrate to a trixie lxc? or will I have the same issue since it uses or reports the base OS's kernel? I'm not familiar enough with the design of LXCs to navigate this properly.

I'm not super enthused having to reinstall PBS each time a version updates however sparse that upgrade path is.

 

[Johannes S]

I'm not super enthused having to reinstall PBS each time a version updates however sparse that upgrade path is.

Well now you know why it's not recommended to run PBS inside a lxc. You could still run PBS inside a VM in TrueNAS. If the TrueNAS itself a VM I would go with a PBS VM outside of the TrueNAS and mounting an ISCSI share as datastore.

Since an LXC always share the kernel with the host OS it's not possible to update the lxcs kernel. In general lxcs are less isolated and more dependable on the host os than vms.

 

[scyto]

I'm not super enthused having to reinstall PBS each time a version updates however sparse that upgrade path is.

tl;dr you don't need to reinstall pbs every time there is a verison update

just ignore the kernel messages, totally expected, you are in an LXC, LXCs can't change the host kernel they use, even if truenas used EXACTLY the same kernel major revsison and even if there were no differences in the kernel you will still get the messages because the kernels have different version strings

i upgraded mine at the weekend, its working just fine, there is no need to re-install each time, and if you hit an actual kernel issue the reinstall wouldn't help in the slightest anyway, if. you don't like doing it this way your other alternative is to run a pbs VM instead and have that connect over say SMB to the truenas host (this is how i had my PBS running for years on my synology) - the messages you see are mostly warnings, not errors.

i decided to trade the low-level access to ZFS dataset i get with an unpviliged incus LXC container for the risk it might stop working

given it seems incus LXC is going away for libvert managed LXCs in truenas it will be interesting to see what if any migration of the container will be possible - their product management decision framework is an utter mess tbh, or their issue is they let devs decide to run from one interesting-thing-of-the-week to another....

 

[Johannes S]

given it seems incus LXC is going away for libvert managed LXCs in truenas it will be interesting to see what if an migration of the container will be possible - their product management decision frameowrk is an utter mess tbh, or their issue is they let devs decide to run from one interesting thing to another....

First time I heard about the plan to migrate to libvirt-managed LXCs. I assumed the recent switch migration was only for VMs? Then I won't bother with a PBS LXC on TrueNAS, I'm not interested in needing to get my backups working again after an update of TrueNAS.

 

[scyto]

First time I heard about the plan to migrate to libvirt-managed LXCs. I assumed the recent switch migration was only for VMs? Then I won't bother with a PBS LXC on TrueNAS, I'm not interested in needing to get my backups working again after an update of TrueNAS.

yes i was mystified too, here is the post where i learnt it and the reference to Kris on staffwho made the statememt it will be libvirt all the way down

https://forums.truenas.com/t/truena...-enables-virtualization-blog/49312/18?u=scyto

which now means they have to license or build their own HA clustering for LXCs and VMs, rtaher than using incus's for free..... not the product management decision i would have made....

that said, currently goldeye nightlies is the same as latest fangtooth... classic truenas libvirt virtualization, docker based app, and incus containers.....

 

[AlexanderMomchilov]

This guide installs proxmox-backup-server, but not proxmox-backup. Is that intentional?

This gets flagged as a failure by the pbs3to4 utility. The common solution is to just install it.

 

[alpha754293]

(THIS PART COULD USE FEEDBACK FROM THE COMMUNITY, I'M NOT GOOD AT LINUX PERMISSIONS YET, THIS JUST WORKED FOR ME AND MAY NOT BE THE MOST SECURE SETTINGS)
Navigate to the dataset that was created and change the permissions to the preset ACL named "POSIX_OPEN" and set the owner and group to "backup"

This was the part that threw me for a loop when I tried to implement something very similar, with the only differences being that:

1) I was running TrueNAS in a VM, on PVE and passed my LSI 9341-8i into said VM and I was able to create the pool etc.

2) PBS was running as a LXC, on PVE.

3) When I tried to mount the data set /mnt/tank/share/pbs over NFS, PBS couldn't create the datastore due to permissions issues.

I couldn't figure out why, even though the dataset had its own assigned to backup.

Seems so strange to me.

So I ended up deleting the whole thing, started from scratch (of sorts) again, and then just create the ZFS pool directly on PVE and then re-install PBS in a LXC, and then passed the dataset into the LXC as a bind mountpoint.

That seems to be working.

TrueNAS permissions is a bit of a mystery to me. I didn't realise that you had to change it to "POSIX_OPEN" and set the owner and group to "backup" for this to work.

Might try it again after the current sync task is complete.

 

[scyto]

1) I was running TrueNAS in a VM, on PVE and passed my LSI 9341-8i into said VM and I was able to create the pool etc.

this is a huge difference and changes the permissions needed

these are my incus LXC permission on the dataset for PBS datastore mounted into /mnt/pbs in the LXC

the ? is user and group 2147000035, which is the mapping user of backup in the incus container to the host mapped userid/grouid, this used to show a username, seems like truenas no longer does that for some reason....

 

[makkusu]

I run the two primary backup servers for my company like this, for over half a year now, after encourerting performance problems with having a VM connect to dataset via NFS/SMB/iSCSI.
It works incredibly well for me, to the point that I think it would make sense for PBS to officially support this scenario.

The only thing I still haven't figured out for the future is how to backup TrueNAS to our third PBS, which writes to LTO tapes, but I saw other people had success using the PBS Backup Agent on TrueNAS data sets.

 

[Johannes S]

It works incredibly well for me, to the point that I think it would make sense for PBS to officially support this scenario.

Which scenario? Running as lxc from TrueNAS? Then they would have to support a scenario which envolves a software they have no influence over (TrueNAS by Ixsystems) as long as they don't partner with TrueNAS. And (like pointed out by Scyto and me) since Ixsystems seems to rebuild their container/virtualization stack from the ground every year it would be quite cumbersome to catch up with any change made by TrueNAS developers.

 

[makkusu]

Which scenario? Running as lxc from TrueNAS? Then they would have to support a scenario which envolves a software they have no influence over (TrueNAS by Ixsystems) as long as they don't partner with TrueNAS. And (like pointed out by Scyto and me) since Ixsystems seems to rebuild their container/virtualization stack from the ground every year it would be quite cumbersome to catch up with any change made by TrueNAS developers.

We are running PBS in a LXC container mapped to dataset for the backup datastore.

The TrueNAS stable release is pretty conservative with updates, so it might not be that huge of a challenge, but I am hardly deep in Linux development.
This purely speaking from a customers perspective, but TrueNAS is and Proxmox probably have a customer base that overlaps to a considerable degree, and TrueNAS has aquired certification for Veeam compatability, so a partnership might not be super far fetched. But obviously nothing I can controll as a customer.

 

[Johannes S]

The TrueNAS stable release is pretty conservative with updates, so it might not be that huge of a challenge, but I am hardly deep in Linux development.

This is true for their storage stack but sadly (see discussions in this thread) not for their container/vm stack. Regarding a potential partnership I really don't know. On one hand it would be definitively in the interest of many customers but I'm not so sure in regards to the business strategys of both companies.
Ixsystems obviouvsly try to also position TrueNAS as a Hypervisor plattform for small and middle businesses. Although their development is way to fluid that I would consider it personally I can imagine a potential conflict of interests as soon their virtualiation/container stack gets more mature and less fluid.

 

[scyto]

For anyone coming here about migrating to TrueNAS 26 - i did a test upgrade to TrueNAS 26 Beta, it seemed to blow away the incus containers. As such i am unclear there will be an incus\lxc > livirt\lxc migration path - I am sure one could export / import if one was careful and new the commands for incus and libvirt

for me i just recreated the pbs container (in isolated mode, this fixes my complaint earlier in the thread about containers being able to access each other)

however, the UID base shifted going from Incus to libvirt so the old dataset owner no longer maps to backup in the new container, so a host-side chown to the new base+34 is mandatory, and the base must be read from /proc/<pid>/uid_map, not assumed.

it does require that the permissions on the dataset to be made to match either the default or isolated slice number - here were the basic steps

• Create the new container as Isolated. Debian Trixie image, and at create time set ID Mapping = Isolated (it can't be changed later). This gives the container its own private UID slice so other containers can't reach its data. Set autostart on.
• Bind-mount the existing datastore dataset. Filesystem device: source = your dataset (e.g. /mnt/pool/backups/pbs), destination = /mnt/pbs. Don't tick any "delete dataset" option if you ever delete a container that's the only thing that would destroy the backups.
• Find the container's real UID base from the CLI, not the UI. The base is 2147000001 + slice × 65536, but read it authoritatively.
  On the host:

pid=$(midclt call container.query | jq -r '.[]|select(.name=="pbs1")|.status.pid')
cat /proc/$pid/uid_map      # the host-UID on the "0 ..." line = base

make a note of the new PBS's user UID = base + 34

• Chown the dataset to that backup UID on the host. The container can't do this (the files are outside its range until they're owned correctly):

bash
chown -R <base+34>:<base+34> /mnt/pool/backups/pbs

• Strip any ACL and set the modes PBS actually wants. If the dataset has a POSIX ACL (TrueNAS share presets add one), strip it, the ACL mask clamps the mode and PBS rejects it. Then set the modes, which are the standard 755/750/644, not hardened 700 (this is the one everyone gets backwards).
• From inside the container:

setfacl -bR /mnt/pbs 2>/dev/null
chmod 0755 /mnt/pbs
chmod 0750 /mnt/pbs/.chunks
find /mnt/pbs/.chunks -mindepth 1 -type d -exec chmod 0750 {} +
find /mnt/pbs/.chunks -type f -exec chmod 0640 {} +
chmod 0644 /mnt/pbs/.lock

• Verify as the backup user before touching PBS. Root can see things backup can't, so check from backup's eyes:

sudo -u backup stat -c '%a %u:%g' /mnt/pbs          # want 755 34:34
sudo -u backup find /mnt/pbs/.chunks -mindepth 1 -maxdepth 1 -type d \
\( ! -perm 750 -o ! -uid 34 -o ! -gid 34 \) | head # want empty

• Install PBS (keyring → deb822 .sources → apt install proxmox-backup-server), passwd root for the web login, enable both services.
• Add the datastore with reuse via CLI:

proxmox-backup-manager datastore create <name> /mnt/pbs --reuse-datastore true

TASK OK = done. Any "permissions or owner not correct" here means step 3/4/5 is off — re-read the base and recheck modes as backup.

1. Rebuild the jobs PBS lost (they live in config the old install took with it): prune, daily GC offset after prune, weekly verify, plus one verify now to confirm the data rode through intact.

Now maybe this will only affect beta TrueNAS 26 and the final will have an incus > libvirt LXC migrator...... so don't do what i did, WAIT