adding a TrueNAS VM share to unprivelidged LXC

[chenks]

hi, looking for some semi-newbie help

i have PVE 8.2.4 running with various LXCs and VMs. all running good.
one of the VMs is TrueNAS with HBA in passthru mode - again this is running fine.

however, how do i map one (or any) of the NFS shares to PVE so that my LXCs can use it? (LXCs are unprivelidged)
i already have an external synology NAS share mapped via FSTAB on the PVE and then mapped to LXC via the LXC conf file.
obviously this method wouldn't quite work as the FSTAB will already have ran by the time the TrueNAS VM boots.

i don't want to add it as a pve storage location though as i'm not using it for container/template etc storage.

 

[TuxHobo]

There is no direct method to reach into a ZFS pool of the TrueNAS VM to Proxmox LXC containers. ZFS pools are completely isolated from the host because the disk controllers must pass through at the PCI interface to the TrueNAS VM. The disks attached to this controller are not visible from the Proxmox host.

Access to TrueNAS shares is the same as if the NAS was a separate machine. (similar to your Synology shares, however not sure how the fstab entries are interworking) There is a little twist at the end to support unprivelaged containers. This Proxmox procedure does not depend on the type of NFS server.

The setup is a three step process:

1. Create a TrueNAS NFS share. (The NFS server) TrueNAS documentation assumes that the data set to share exists already.
2. Mount an NFS client share on the Proxmox host.
3. Create a bind mount point from the Proxmox host to the LXC container

Note that the above steps do not depend on TrueNAS as a VM within the Proxmox host. The TrueNAS VM has its own IP address which could be an external server. (I think that the same can be done with the Synology shares.) Access between the Proxmox host and the TrueNAS VM is via the Ethernet bridge. I use this method with my Proxmox to TrueNAS VM in my home lab. It works well.

For the second step I use the Proxmox GUI to create the NFS client storage location on the Proxmox host:
DataCenter->Storage->Add->NFS

Fill in the form with:

• ID (used as the name of the local pve storage directory)
• Nodes (I use my local lvm storage location)
• Server (use the IP address of the NFS server)
• Export (The shares advertised by the server are listed here)
• Leave the rest as default and click the "add" button at the lower right.

(errors with the add may need adjustment of the Maproot user name on the TrueNAS NFS share. Set it to root if needed.)

The NFS share is now mounted on the Proxmox host. This can be verified in the host shell:
ls -la /mnt/pve/ID_NAME
The ID_NAME is the name given in the ID field above.

The mount point can also be seen:
mount | grep ID_NAME

The third step is to create a bind mount point to the lxc container as described in Proxmox docs.
From the host shell, the command will look something like:
pct set 100 -mp0 /mnt/pve/ID_NAME, mp=/media

Where:

• 100 is the target container ID
• /mnt/pve/ID_NAME is the directory in the host to bind to the LXC container
• mp=/media is the mount point in the LXC container.

The NFS share should now be visible from the container console: (linux container is assumed)
ls -la /media

I think that the bind mount point is the recommended method for NFS storage access to unprivileged containers.
I have not found a means to create a bind mount point from the GUI. Command line only. It would be a nice feature to add a Bind Mount Point option next to the Mount Point option in the container resources add pull down.

As an alternative, I've seen posts where users simply create a privileged LXC container. Doing so allows installing the NFS client package within the container. Procedures are then similar to the Proxmox VM environment. This method follows documented practices of the target os in the container. I'm not sure how to quantify the security risk of privileged containers for Home Lab deployments especially if they are not exposed to the internet.

 

[chenks]

@TuxHobo sorry i think you've missed the point of my post.
yes i know i could mount the share via "DataCenter->Storage->Add->NFS" however that's not the route i want to use, as it will then create folders in that share relating to the "content" options (which you can't leave blank), and as i have 4 or 5 NFS shares i would end up with unwanted folders in each of those shares.

i am referring to mounting the share via fstab, however as fstab will execute (during pve boot) before the truenas VM has started it will fail to mount.

once the share is mounted via fstab the process would be same as you described (and what i already use for another standalone NAS) - using the bind point in the conf file of the LXC.

 

[TuxHobo]

@TuxHobo sorry i think you've missed the point of my

I've toyed with fstab and found that the shares are not created when listed there.
But I did not reboot the host. I only tried to mount after editing fstab.
Not sure how to do what you want. You may be stuck with the existing method.

 

[chenks]

i'm already using fstab to mount shares on an external NAS and it works fine, fstab is a standard way to mount shares in linux.
if you are having issues mounting using fstab then it's almost certainly a permissions issue.

so back to the original question, as truenas won't have booted at the time fstab runs, how do i get these mounted?

 

[gfngfn256]

so back to the original question, as truenas won't have booted at the time fstab runs, how do i get these mounted?

I believe what you need in your fstab mount is the bg option as shown here.

 

[Ernst T.]

How about a delayed mount using crontab?

@reboot sleep 120; mount -t nfs xxx.xxx.xxx.xxx:/export/nfs /mnt/nfs

 

[chenks]

I believe what you need in your fstab mount is the bg option as shown here.

yes i think that is what i need.
the initial mount will fail (as the truenas VM hasn't started yet), and the bg command will keep it trying to mount which it will once the VM has started.

How about a delayed mount using crontab?

@reboot sleep 120; mount -t nfs xxx.xxx.xxx.xxx:/export/nfs /mnt/nfs

that also looks like an option.
i might try the bg option first and if that seems problematic i'll give the crontab a try.

 

[chenks]

have come across an issue that i don't understand

created the nfs shares in truenas and made sure root (in pve) was mapped to the truenas user with the appropriate permissions.
added the following to /etc/fstab (note these are just test shares and not the final mappings)

192.168.50.16:/mnt/prox-pool/prox-share    /mnt/truenas_pve/          nfs    defaults    0 0
192.168.50.16:/mnt/prox-pool/media         /mnt/truenas_media/        nfs    defaults    0 0
192.168.50.16:/mnt/prox-pool/downloads     /mnt/truenas_downloads/    nfs    defaults    0 0
192.168.50.16:/mnt/prox-pool/other         /mnt/truenas_other/        nfs    defaults    0 0
192.168.50.16:/mnt/prox-pool/data          /mnt/truenas_data/         nfs    defaults    0 0

ran systemctl daemon-reload then mount -a and they are accessible in pve with read/write permission - no issues

however i then go to add the following to /etc/pve/lxc/106.conf

mp0: /mnt/truenas_downloads/,mp=/downloads

save, reboot the LXC and the LXC fails to boot with a pretty generic error

run_buffer: 571 Script exited with status 13
lxc_init: 845 Failed to run lxc.hook.pre-start for container "106"
__lxc_start: 2034 Failed to initialize container "106"
TASK ERROR: startup for container '106' failed

any ideas?

 

[gfngfn256]

mp0: /mnt/truenas_downloads/,mp=/downloads

Try removing that trailing slash, so:

mp0: /mnt/truenas_downloads,mp=/downloads

Thats the way it appears in the wiki here, (under section: "Bind Mount Points").

 

[chenks]

Try removing that trailing slash, so:


Thats the way it appears in the wiki here, (under section: "Bind Mount Points").

that made no difference - same error

 

[gfngfn256]

Try a debug log for that 106 container, so:

lxc-start -n 106 -F -l DEBUG -o /tmp/lxc-CT106.log

# and then post output here of:

cat /tmp/lxc-CT106.log

 

[chenks]

this is the conf file

arch: amd64
cores: 2
features: keyctl=1,nesting=1
hostname: sabnzbd
memory: 2048
mp0: /mnt/truenas_downloads,mp=/downloads
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:97:7B:30,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-106-disk-0,size=8G
startup: order=20
swap: 512
unprivileged: 1

and this is the log

https://pastebin.com/yxd0V0j1 (was too large to paste direct in the post).

note that when i change the mount to use a smbshare posted below (which is mounted in pve also using fstab) then it boots with no issue

mp0: /mnt/truenas_downloads,mp=/downloads

 

[chenks]

@gfngfn256 logs attached

 

[gfngfn256]

I don't have much time now but I skimmed through the logs, and don't really see the issue, it would appear the LXC starts up & then shuts down gracefully (sort of).

Just a point - I assume that the /downloads directory has been created/already exists in the LXC.

 

[chenks]

Just a point - I assume that the /downloads directory has been created/already exists in the LXC.

yes it exists

i can mount a samba share to it with no problems, but when i set the nfs share to mount to it instead, it then fails.
switching back and forward the mount results in 100% success for samba and 100% fail for nfs.

both the samba and nfs shares are mounted in pve fstab, and both are fully accessible in pve host.

 

[gfngfn256]

Does the directory /mnt/truenas_downloads possibly contain symlinks? (As per the above Wiki, they may not).

Anyway have you tried to Bind Mount another subdirectory of that folder?

 

[chenks]

Does the directory /mnt/truenas_downloads possibly contain symlinks? (As per the above Wiki, they may not).

nope, it currently has 1 file in it, nothing else - this is folder when accessing it on the pve host.

root@proxmox:/mnt/truenas_downloads# ls -la
total 5
drwxrwx---  2 root root    3 Aug 23 13:59 .
drwxr-xr-x 10 root root 4096 Aug 23 14:03 ..
-rw-r--r--  1 3000 root    0 Aug 23 13:59 foobar
root@proxmox:/mnt/truenas_downloads#

Anyway have you tried to Bind Mount another subdirectory of that folder?

yes, no matter where i try to mount it, it results in the same outcome (even tried it in a different LXC with the same outcome).
that suggests it's not a problem with the mount location within the LXC.

 

[gfngfn256]

no matter where i try to mount it

That is not what I suggested. I suggested trying to bind mount a subdirectory of /mnt/truenas_downloads to the LXC. So maybe /mnt/truenas_downloads/share to the LXC's /downloads (I'm trying to achieve that it is not the same identical mountpoint used for the original NFS to the PVE host). Give it a try & report back.

Anyway on researching your problem, I'm starting to think that it is probably a permissions issue.

NFS shares can be picky / quirky when it comes to permissions. Since you are running an unprivileged container, I guess this is what needs addressing. I therefore started searching & discovered that indeed permissions need to be addressed in the LXC for this NFS share to work.

Maybe try Option 3, Point 7 from this post.

 

[chenks]

That is not what I suggested. I suggested trying to bind mount a subdirectory of /mnt/truenas_downloads to the LXC. So maybe /mnt/truenas_downloads/share to the LXC's /downloads (I'm trying to achieve that it is not the same identical mountpoint used for the original NFS to the PVE host). Give it a try & report back.

yeah that makes no difference, same error

Anyway on researching your problem, I'm starting to think that it is probably a permissions issue.

NFS shares can be picky / quirky when it comes to permissions. Since you are running an unprivileged container, I guess this is what needs addressing. I therefore started searching & discovered that indeed permissions need to be addressed in the LXC for this NFS share to work.

Maybe try Option 3, Point 7 from this post.

i'll take a look at permissions issues, but i'd have thought pve would have given a more usable error if it was permissions.
with samba shares, the LXC still mounts the share but you get a "permission denied" message when trying to access it.

however, i've seen that tutorial you linked to, and the directory in the LXC already has those permissions applied to it - as the was the same tutorial i used when mounting a samba share in an unprivelidged LXC.