Storage conundrums

[venquessa]

I have a VM running about 5 docker-compose stacks. I want to split it up into multiple CTs. Probably one per stack. Example stacks would be: network-services(primary dhcp, primary dns, mqtt*3 (external, internal and test), zigbee2mqtt, PiHole) and home_auto (about a dozen python MQTT uServices) and homeassistant. etc. etc.

As the majority (or all) will be unpriv'ed CTs they will not be able to mount NFS shares from the priv'd NAS CT.

The NAS CT has a "volume" /srv/fast with no root squash designed for this purpose. Shared container volume drives, or just generally, shared state for VMs, CTs etc. However while it's useful for VMs it's useless to CTs.

I tried different ways, like mounting the NAS NFS Share back onto the PVE host and bind mounting it into the CT. That just feels like a bad idea and did indeed cause shutdown and startup issues with availability ordering.

What I am trying to avoid is having to checkout individual dockerstack meta data onto individual CTs and the manage individual CT volumes all containing a /data volume of their own.

"long finger", I should/could/will move the docker-compose stacks to k8s and use it to manage all the stacks in HA with multiple nodes etc. etc. etc. A long way off. Also expensive as a proper k8s setup will probably need to be VMs so they can share block level / nas points etc.

Here is where I have ended up but I'm not sure it's a good solution.

I created my own subpool folder. rpool/fast on the primary mirrored 1Tb M.2s. Moved the volume (actually renamed) the NAS /srv/fast to that folder. Let ZFS auto mount it in /rpool/fast I can then place docker-stack folders and data folders in that single place from all CTs by bind mounting it into the CTs. This way I can run 5 docker CTs and freely migrate services between them as I please. (IPs, IP aliases and static DHCP foolery asides)

The downsides already is that PVE ignores that volume and won't back it up.

That got me to thinking. Is it just the naming and location of the volume it cares about? Would it accept a bind mount? I mean it's more than just what it's called. It will be mounted by multilpe CTs if they all consider it their own volume bad things will happen when I delete one.

I don't mind backing it up myself. The kind of share it is means it will have pretty specific backup requirements and rsync might be a better idea "off PVE".

 

[venquessa]

While a dedicated NAS host, leaving PVE only managing boot volumes, would solve a lot of this. It's not in the budget yet.

EDIT: I have 2 or 3 DDR3 era eWaste machines. All of which would be capable of running a small 2 or 3 bay NAS of SSDs on a 1Gbit link. Scaling them up to include a 2.5Gb link and taking the 6xSATA expansion card, might work, but they are all SFF or Micro format. The 6 bay SSD 5.25" bay would need to "free float" outside the case etc.

Amazon priced up a 32Gb, B550, 4300G, 2.5G NIC + M.2 pair at nearly £500. Which is cheap for a 6 bay NAS, but I'm trying not to exercise the credit card that much, having just bought a similar but more powerful PVE node this month! I must resist the need to buy more tech, I must!

I did also explore the "Jansbo" option (and others) on AliExpress, however if you are buying "genuine" tech and not scam/bait ware the prices today are not that much better than buying from Amazon with 2 day delivery. The NAS motherboard they sell is still £130. That is your board and CPU, but a B550 board and AMD chip is actually cheaper! The Ali NAS cases are £80+ ... for which you can get a Fractal Design's Core case.