Help installing LXC to shared drive for HA

[tyler72us]

Hi, thank you for visiting my thread today. I appreciate you stopping by. Go ahead and take a seat. Feel free to rest up.

By the way, while you are here, could you help me with a little issue I'm having? Great. So, I'm trying to set up a HA service. I know as part of that I need a shared storage for the VM and containers, but I seem to be having troubles with that. I'm trying to install an LXC on a shared SMB drive. The drive is set up in the GUI so I can store anything there. But when I try to install a pihole instance using the LXC helper scripts, it keeps failing with error:

⠹ Starting LXC Containervolume 'dataone:105/vm-105-disk-0.raw' does not exist
⠧ Starting LXC Container
[ERROR] in line 1265: exit code 0: while executing command pct start "$CTID"

.

So far, I've had no issues with backups, templates, or general access to this drive, so I don't THINK it's a general permissions issue. I can install to any of the local drives for the nodes in my cluster, and have 0 issues. Maybe it's an issue because the unprivileged LXC is trying to access it? Oh, I've tried doing it with an NFS share as well, with no real change. Still fails but with a more generic exit code 0. Is the shared HA storage supposed to be a different from SMB/NFS? I'm mostly just looking for some failover protection if my main server fails (too power hungry for my UPC), would a different type of service be a better choice for me (docker swarm, k8, etc). What are your thoughts?

 

[dakralex]

Hi!

Is the shared HA storage supposed to be a different from SMB/NFS?

For larger clusters, we usually recommend a proper Ceph setup, but for smaller setups a NFS can do OK as a shared storage.

For the rest of the post, it is hard to tell without any configurations and when the error exactly happens. How is the storage setup in PVE? Is it available on all nodes? Or is it a single node HA setup? Can you share the config of the storage (cat /etc/pve/storage.cfg) and the container (with pct config 105)?

 

[tyler72us]

Hi!


For larger clusters, we usually recommend a proper Ceph setup, but for smaller setups a NFS can do OK as a shared storage.

For the rest of the post, it is hard to tell without any configurations and when the error exactly happens. How is the storage setup in PVE? Is it available on all nodes? Or is it a single node HA setup? Can you share the config of the storage (cat /etc/pve/storage.cfg) and the container (with pct config 105)?

Thanks, thats good to know that NFS is at least valid. Yeah I'm just a simple homelabber with a retired enterprise server and some retired laptops. I don't have the infrastructure for real ceph storage.


I set up the shared storages in the Datacenter storage GUI, as far as I can tell that has made it available to every host on my cluster.

Here is my storage.cfg:

dir: local
        path /var/lib/vz
        content iso,backup,vztmpl

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

lvmthin: extra
        thinpool extra
        vgname extra
        content images,rootdir
        nodes pve

cifs: dataone
        path /mnt/pve/dataone
        server 192.168.1.100
        share proxmox
        content iso,snippets,import,rootdir,backup,vztmpl,images
        nodes pve2,pve,pve1
        prune-backups keep-all=1
        username smithfam

zfspool: NVMEdrive
        pool NVMEdrive
        content images,rootdir
        mountpoint /NVMEdrive
        nodes pve

nfs: lxc
        export /mnt/dataone/lxc
        path /mnt/pve/lxc
        server 192.168.1.100
        content vztmpl,snippets,backup,iso,images,rootdir,import
        prune-backups keep-last=3

And the container when installed on smb (dataone above):

arch: amd64
cores: 1
features: keyctl=1,nesting=1,fuse=1
hostname: pihole
memory: 512
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:B6:D0:A8,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: dataone:105/vm-105-disk-0.raw,size=2G
swap: 512
tags: adblock;community-script
unprivileged: 1
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

And I was going to install on the NFS one as well, but of course this morning it installed without a single issue. I'm guessing it was a permissions issue in TrueNAS, because I had some permission fixing yesterday while I was working on not being able to get to my media library.

So if you see anything wrong with the SMB share in general let me know, otherwise thank you, I wouldn't have retried installing if it wasn't for you needing some extra logs.

If some future poor soul comes to looking for help with a similar issue, the best advice is to go with a NFS share, and really make sure your permissions are set up correctly. (linux permissions are the bane of my homelab existence, I swear)


Thanks again @dakralex.