I'm unsure of what is the best way to setup peristant storage for kubernetes via CNI with ProxMox. What I want to do is provision fast NVMe persistant storage to my kubernetes cluster which is running on a single ProxMox node with incremental backups with a Synology DS923+. My goal is to have the ProxMox VM's that are running the kubernetes nodes with Talos, to have access to fast storage for computing and performance, with the assurance of being able to restore all of the Persistent Volumes or just on specific persistent volume through snapshots or incremental backups. I have a Synology DS923+ and TrueNAS Core setup (although I'm not confident with TrueNAS yet.
In summary, I want to:
Any guidance is helpful. I'm also open to alternative solutions or patterns.
What I'm currently not looking to do but would consider if convinced:
In summary, I want to:
- Run a 3 node k8s cluster on a single ProxMox host.
- Provide fast persistent storage to the k8s and be able to backup the persistent storage incrementally.
- User the native k8s CNI such that storage provider can automatically provision PV and PVC from storage class.
- Have the ability to shutdown the k8s nodes and the ProxMox host for maintenance (or power outage), and be able to gracefully start things backup with as few manual steps as possible. (I would account for order of shutdown like stopping applications prior to their databases, but the expectation is that when resuming a database running on the k8s nodes, it would pick up from where it left off. )
- I have a UPS, but in my area a storm can knock the power out for 8+ hours. The ability to gracefully shutdown through NUT is a priority.
- Option 1: Hyperconvergence - I don't mind putting the NVMe's directly in the ProxMox node, but I'm lost to how I would setup them to provision to Kubernetes via the CNI and take incremental backups. Any advice here would be much appreciated. It seems like with the concept of Hyperconverging the compute and storage with the k8s cluster would make sense, although I'm not sure if my pattern works, what type of storage to use (ZFS, LVM...etc), or even if Synology does not make sense in this pattern and I should be looking at ProxMox Backup.
- Option 2: TrueNAS NFS Mount - Alternatively, I was considering putting the NVMe drives in the TrueNAS system, create a new storage pool for just the NVMe's and provision the kubernetes nodes with a NFS mount, then use csi-driver-nfs https://github.com/kubernetes-csi/csi-driver-nfs for providing the storage to kubernetes. Then some how backup the new storage pool. For some reason this seems like a simpler solution in my head from years of IT training, but on paper it is more complicated and in opposition of the Hyperconvergence pattern --> Why would we mount via NFS over the network to serve a single host?
- Option 3: Overthinking it - It may be that I'm missing the most straight forward approach of just putting the NVMe drives in RAID mirror and not worry about the rest. Maybe there is an easy way to back these up with local snapshots and long term backups off of the system to synology or something else.
Any guidance is helpful. I'm also open to alternative solutions or patterns.
What I'm currently not looking to do but would consider if convinced:
- Setup Ceph on ProxMox (While a great option, I don't have a 3rd server for a good cluster. In addition, I still would need to be able to shut down the entire rack if needed (my power can be unreliable during storms.))
- Convert the TrueNAS server into a additional ProxMox Host, extend the Synology DS923+ with DX517 move the former TrueNAS HDD's there. (However, I don't want to invest in any additional hardware at this time)
- ProxMox Host
- ASUS Z9PE-D16
- 2x Intel Xeon E5-2687WV2 3.40 GHz
- 256 GB of ECC PC3-12800 DDR3-1600MHz KVR16R11D4K4/64 Kingston
- 2x 3TB WD HDD
- 2x 500GB Samsung SSD (ZFS mirror local storage)
- 2x 256GB Crucial SSD (ZFS mirror ProxMox host install)
- Synology DS923+
- 4x Western Digital 10TB WD Red Pro NAS
- 2x WD_BLACK SN850X 2TB
- TrueNAS Host
- ASRock Rack EPC612DU4-8R
- Intel E5-2620 @ 3.20 GHz
- 64 GB ECC - Micron 16GB PC4-17000 DDR4-2133MHz
- 5x 4TB WD HDD (RAIDZ1 pool)
- 1x 256GB HDD (TrueNAS host install)
- 1x 500GB SSD cache
- 2x 1TB Crucial PCIe3 NVMe drives with NVMe to PCIe Adapter (M.2 NVME to PCIe 3.0 x4 Adapter)