How to setup separate drives for LVMs and LXCs and their respective application data

[acabrera]

Hello. New to the group so please forgive me if I missed where this has already been answered. I have search extensively but I don't believe I have found anyone with the exact same situation. Here goes....

I have built a homelab that will be used for various purposes and VMs (i.e. Frigate, Ollama, Home Assistant, MQTT, NAS, mail server, media store...).

I have multiple NVMe and HDD drives that I intend to use as follows for speed and ease of management, backups, recovery...THEORETICALLY:

- One 1TB PCIe 5 NVMe: Proxmox OS, VMs, Containers...
- One 1TB PCIe 4 NVMe: Data storage for all VMs and Containers with a separate data folder for each
- One 512GB PCIe 4 NVMe: For ISO images...
- One 2TB HDD: Backup 1
- One 2TB HDD: Backup 2

So, for each of the VM/Containers, I intend to mount or otherwise define a location where each can store their logs, databases, docs, other files... Everything from the NVMe drives will periodically be backed up to the HDD drives (alternating, not RAID, for now).

For storing the VM/Container data, here is what the folder structure might look like on the 1TB PCIe 4 NVMe drive (vm-data):

/mnt/vm-data/frigate-data
/mnt/vm-data/ollama-data
/mnt/vm-data/ha-data
/mnt/vm-data/mqtt-data
...

I see that there are a variety of options for adding storage from Datacenter->Storage->Add such as Directory, LVM, NFS, ZFS... Depending on how the drive is setup, we then have options for mounting the drive using the config files such as /etc/pve/qemu-server/100.conf but everything I have tried feels clunky or just plain wrong.

My question is...what is the preferred/best practices way to define/configure a data drive to allow the VMs/Containers to access/deposit their data onto their respective folders while running them on a different drive?

Edit:

After further research on this topic, I can confirm that I am even more confused now. My expectation is not that there is a single perfect answer for this situation but that at least there is a preferred approach that balances complexity with practicality.

So, I will ask my question again but with options, as I understand them, to choose from.

With the goal of keeping the LVM and LXCs (i.e. Frigate, PostgreSQL, Home Assistant...) on one drive and their respective application data (i.e. photos, logs, databases...) on a different drive, which of these options is the preferred approach and how is it accomplished?

- Setup some sort of NAS VM that controls access to the separate "data" drive by limiting each VMs access to specific folders on that drive?
- Use NFS share directly from Proxmox Datacenter for the separate "data" drive that can somehow be seen by each VM by creating some mountpoint?
- Setup a Directory from Datacenter that is somehow accessible by each VM?
- Setup a shared pool from pve?
- Create a separate virtual store for each application (i.e. frigate-data, postgresql-data, ha-data) that will be added as a disk from each VM?
- ......


Thanks in advance for any advice.

 

[Kingneutron]

> - One 512GB PCIe 4 NVMe: For ISO images...

I would use this drive for Proxmox OS, you can avoid the PVE GUI and copy the ISOs to /var/lib/vz/template/iso with e.g. Midnight Commander -- or share them with Samba from somewhere else if you have multiple PCs that need access to them.

https://github.com/kneutron/ansitest/blob/master/proxmox/symlink-samba-isos.sh

Reason for separating OS and data - the PVE installer ISO will wipe the target disk if you ever need to reinstall.

> - One 1TB PCIe 5 NVMe: Proxmox OS, VMs, Containers...

> - One 2TB HDD: Backup 1
> - One 2TB HDD: Backup 2

Consider separating your backup onto separate hardware with Proxmox Backup Server. Dedup alone is worth it, but there are additional features. You want to avoid having a single point of failure (also consider ransomware and misuse of rm), and easy restores if you hit a DR situation.

https://toast.djw.org.uk/tarpipe.html

PROTIP: You can avoid scp and drop your ISOs directly into /var/lib... over an open port (firewall off, or port = allowed) if you have tar and netcat on both sides. Don't bother with compression if you have Gigabit ethernet. Use tar cpvf on the sending end and tar xpvf on the recv to preserve ownership / permissions, or you can just chown/chmod afterwards

 

[acabrera]

> - One 512GB PCIe 4 NVMe: For ISO images...

I would use this drive for Proxmox OS, you can avoid the PVE GUI and copy the ISOs to /var/lib/vz/template/iso with e.g. Midnight Commander -- or share them with Samba from somewhere else if you have multiple PCs that need access to them.

https://github.com/kneutron/ansitest/blob/master/proxmox/symlink-samba-isos.sh

Reason for separating OS and data - the PVE installer ISO will wipe the target disk if you ever need to reinstall.

> - One 1TB PCIe 5 NVMe: Proxmox OS, VMs, Containers...

> - One 2TB HDD: Backup 1
> - One 2TB HDD: Backup 2

Consider separating your backup onto separate hardware with Proxmox Backup Server. Dedup alone is worth it, but there are additional features. You want to avoid having a single point of failure (also consider ransomware and misuse of rm), and easy restores if you hit a DR situation.

https://toast.djw.org.uk/tarpipe.html

PROTIP: You can avoid scp and drop your ISOs directly into /var/lib... over an open port (firewall off, or port = allowed) if you have tar and netcat on both sides. Don't bother with compression if you have Gigabit ethernet. Use tar cpvf on the sending end and tar xpvf on the recv to preserve ownership / permissions, or you can just chown/chmod afterwards

Thanks for info....

For some reason I assumed that having the Proxmox OS on the faster PCIe 5 NVMe would benefit the overall performance. But, let me say back to you what I believe you are suggesting:

- One 1TB PCIe 5 NVMe: VMs and Containers along with their data
- One 1TB PCIe 4 NVMe: Unused?
- One 512GB PCIe 4 NVMe: Proxmox OS (since a reinstall will require wiping entire disk)
- One 2TB HDD: Backup 1
- One 2TB HDD: Backup 2

What I don't get is why it's better to house the VMs/Containers AND their data on the same drive. Backups and recovery for either the VM and/or the videos feel less efficient.

For example, if I use Frigate for storing security videos, I would imagine that restoring the VM in the event of a corruption or DR situation would be extremely difficult and time consuming because you would not only have to restore the VM but all of the videos. Even backing up that setup would seem to be complex, even with snapshots.

Am I missing the point?

 

[Kingneutron]

> Am I missing the point?

Well, yes. Why would you leave an nvme unused? Put VMs/LXCs on one and their data on the other. You're the admin of the system, I'm only making suggestions.

 

[acabrera]

> Am I missing the point?

Well, yes. Why would you leave an nvme unused? Put VMs/LXCs on one and their data on the other. You're the admin of the system, I'm only making suggestions.

I must have missed where you suggested putting VMs/LXCs on one and their data on the other. I thought you were recommending them both reside on one drive. My original question was how best to do that separation.

In other words, if one NVMe drive will house the VMs/LXCs and the other will house each of their data folders, what is the best/preferred way to do that? Do I define an ZFS storage from Datacenter->Storage? Would it be better to create an NFS drive or a Directory and then map/mount that from within the VM/LXC?

It's possible I am asking the wrong question but I'm a bit confused by the various options for setting this up which is why I am hoping someone has done this successfully and can push me in the right direction.

 

[acabrera]

I've edited my original post to clarify my question at the end.