Disclaimer: All this is done in a homelab (non-production) environment with a limited budget.
TLDR: One central ssd zfs pool shared storage (distinct machine) added to proxmox (two other machines) used for VM app storage (not the VM OS itself). I also want PBS/deduplicated backups. How do I go about this without CEPH?
Currently, I'm looking to implement HA and shared disk storage for my VMs. My setup will include 2 PVE nodes and 1 q-device (on bare-metal TrueNAS Scale). My goal (and the focus of this post) is to have one central ssd zfs pool shared over the network (from TrueNAS; perhaps NFS) where all my VM app data will reside (not the OS itself). There are a few considerations:
• app databases shouldn't be used over network shares (risk of db corruption)
• practically, I don't need the VM OS itself to be saved onto the ssd zfs pool; bandwidth limitation and speed (1Gbit) is a another reason for this
• I don't need true HA and frankly I don't have the resources for CEPH or similar solutions; thus, zfs replication every few minutes for the OS disks of the VMs should suffice
• Why not use local ssd zfs pools in each node? Less disks simply and a central location for all data. I realize this is a single point of failure (I have backups on different disks/storage). Feel free to change my mind. Also, less data to replicate to the other host for zfs replication.
• TrueNAS Scale bare-metal (also q-devide), 2x PVE nodes bare-metal, PBS as VM backing up to NFS share in TrueNAS (HDD)
Thus, here is a rough plan and variations of it:
(a) When spinning up a VM, I will set up 2 disks, one will be a local ssd on the physical machine (e.g. 32GB) solely for the OS itself (e.g. Ubuntu Server), and the other will be a shared storage over the network (e.g. NFS, added via the datacenter->storage tab in the GUI). Docker and all the apps used with it should be on this second disk, the bandwidth limitation is acceptable in this case. I will also make a separate VM for all the databases–this one VM will use only local ssd storage to avoid db corruption (backed up via PBS + zfs replication)
(b) Same as above but mount the NFS share directly inside each VM. The issue here is, I probably wouldn't be able to use PBS for the VM app data, only the root disk of the OS, which kind of makes PBS useless if it doesn't back up the data I want it to?
(c) A different solution? Please, do recommend or give thoughts.
As noted earlier I want to ideally use PBS for its lightweight deduplication solution. I'd like to primarily back up VM app data (nfs disk mounted to the Proxmox host over the network via the Datacenter GUI) to it. I'm not as well-versed in TrueNAS Scale and its deduplication regarding zfs pools, but I hear it is much heavier on resources (that I might not have), which is why I prefer PBS right now.
If I should add any more details, please let me know.
Thank you.
TLDR: One central ssd zfs pool shared storage (distinct machine) added to proxmox (two other machines) used for VM app storage (not the VM OS itself). I also want PBS/deduplicated backups. How do I go about this without CEPH?
Currently, I'm looking to implement HA and shared disk storage for my VMs. My setup will include 2 PVE nodes and 1 q-device (on bare-metal TrueNAS Scale). My goal (and the focus of this post) is to have one central ssd zfs pool shared over the network (from TrueNAS; perhaps NFS) where all my VM app data will reside (not the OS itself). There are a few considerations:
• app databases shouldn't be used over network shares (risk of db corruption)
• practically, I don't need the VM OS itself to be saved onto the ssd zfs pool; bandwidth limitation and speed (1Gbit) is a another reason for this
• I don't need true HA and frankly I don't have the resources for CEPH or similar solutions; thus, zfs replication every few minutes for the OS disks of the VMs should suffice
• Why not use local ssd zfs pools in each node? Less disks simply and a central location for all data. I realize this is a single point of failure (I have backups on different disks/storage). Feel free to change my mind. Also, less data to replicate to the other host for zfs replication.
• TrueNAS Scale bare-metal (also q-devide), 2x PVE nodes bare-metal, PBS as VM backing up to NFS share in TrueNAS (HDD)
Thus, here is a rough plan and variations of it:
(a) When spinning up a VM, I will set up 2 disks, one will be a local ssd on the physical machine (e.g. 32GB) solely for the OS itself (e.g. Ubuntu Server), and the other will be a shared storage over the network (e.g. NFS, added via the datacenter->storage tab in the GUI). Docker and all the apps used with it should be on this second disk, the bandwidth limitation is acceptable in this case. I will also make a separate VM for all the databases–this one VM will use only local ssd storage to avoid db corruption (backed up via PBS + zfs replication)
(b) Same as above but mount the NFS share directly inside each VM. The issue here is, I probably wouldn't be able to use PBS for the VM app data, only the root disk of the OS, which kind of makes PBS useless if it doesn't back up the data I want it to?
(c) A different solution? Please, do recommend or give thoughts.
As noted earlier I want to ideally use PBS for its lightweight deduplication solution. I'd like to primarily back up VM app data (nfs disk mounted to the Proxmox host over the network via the Datacenter GUI) to it. I'm not as well-versed in TrueNAS Scale and its deduplication regarding zfs pools, but I hear it is much heavier on resources (that I might not have), which is why I prefer PBS right now.
If I should add any more details, please let me know.
Thank you.