Hello,
I'm writing this to maybe get some more idea as to why I should invest time and maybe money in building a pve cluster or a shared storage pool.
As of now, I have a setup that mainly does failover and which I quite like :
As of now, if one of my firewall fails, the other takes over, keeping all sessions open, fuly transperent. If one docker host (my LXC) fails, the other keeps the proxy and most important services running. All that works well when one or the other of the PVE host is rebooting or stuck in a misconfigured state. I like that a lot. (haproxy running on both opnSense boxes even switches from the main LXC to the other when needed)
But the way I achieve it is that each LXC is running off of the internal SSD of the pve HOST mainly and mounting NFS shares from the NAS for the media files and such. It means I have a daily routine to take the few containers on the main and rsync their config&files to the failover. That I find quite dirty, since when one LXC fails is usually because I've been tweaking it, and of course the tweaks are not in the backup. Sometime that's a life-saver sometime an annoyance...
Now, I was first thinking of creating a glusterfs storage that could be synced in realtime on the NAS and on both the pve hosts, but given everything on my network is only 1GB/s, even if LAG allow to do 1GB/s to each pve node, I'm afraid it'll be really slowing down my VMs and containers... (I do have an SSD in the NAS as well and could use that to avoid slowing down even more)
But I'm not clear as what benefit I would get from it, I mean I have regular backups of all my VMs on the NAS, I don't realy need high availability... I'm already running some VMs with their root virtual drive being used as cifs by the PVE host onto the NAS, that can already help me alleviate the lack of storage on the failover PVE host, so storage wise I'm not clear what ceph or gluster would bring besides slowing the VMs down and adding HA that I'm not really needing.
Also, I find that having the NAS do mailny file-sharing allows me to actually see and use each file from the NAS which I quite like
Only VM Disks are not so easily opened But I suspect a glusterfs brick running on a docker inside the NAS will be much more opaque to the NAS itself or my windows computers accessing the NAS with CIFS...
But maybe a pve cluster could still be usefull in order to manage all VMs from the same gui, but even that I'm not sure it will help if the main member of the cluster is down, could I easily have the VM automatically started on the other pve host and using the same cifs storage for the VM HDD ?
So if you have any idea of anything that would be fun, likable, or even usefull for me that a pve cluster, ceph or glusterfs could bring, and that is not purely high availability, please tell me that will give me a reason (besides fun) to start playing with it
For the time being, I just discovered that there exists pve-backup-server that seems to allow dedup and single-file restore, that is much more appealing to me (already 2TB of VM backups...) and I think I'll set this up to try it out !
I'm writing this to maybe get some more idea as to why I should invest time and maybe money in building a pve cluster or a shared storage pool.
As of now, I have a setup that mainly does failover and which I quite like :
- 1 miniPC with 6*1GbLAN bonded in PVE for LAG, 32GBRAM, 500GBSSD, running VM opnSense as my main router/fw + 1 LXC container running all my docker images
- another miniPC with also 6*1GbLAN bonded the same way but only 8GB RAM and 125GB SSD, running VM opnSense as CARP failover + 1 LXC container running only the most important docker containers for me (traefik, ...)
- 1 synology DS918+, 2*1GbLAN also in LAG, holding about 20TB, "all" my storage is here, at least big files and backups, not using RAID but hyperBackup (syno custom) regularly to USB drives and other syno NAS at remote places.
As of now, if one of my firewall fails, the other takes over, keeping all sessions open, fuly transperent. If one docker host (my LXC) fails, the other keeps the proxy and most important services running. All that works well when one or the other of the PVE host is rebooting or stuck in a misconfigured state. I like that a lot. (haproxy running on both opnSense boxes even switches from the main LXC to the other when needed)
But the way I achieve it is that each LXC is running off of the internal SSD of the pve HOST mainly and mounting NFS shares from the NAS for the media files and such. It means I have a daily routine to take the few containers on the main and rsync their config&files to the failover. That I find quite dirty, since when one LXC fails is usually because I've been tweaking it, and of course the tweaks are not in the backup. Sometime that's a life-saver sometime an annoyance...
Now, I was first thinking of creating a glusterfs storage that could be synced in realtime on the NAS and on both the pve hosts, but given everything on my network is only 1GB/s, even if LAG allow to do 1GB/s to each pve node, I'm afraid it'll be really slowing down my VMs and containers... (I do have an SSD in the NAS as well and could use that to avoid slowing down even more)
But I'm not clear as what benefit I would get from it, I mean I have regular backups of all my VMs on the NAS, I don't realy need high availability... I'm already running some VMs with their root virtual drive being used as cifs by the PVE host onto the NAS, that can already help me alleviate the lack of storage on the failover PVE host, so storage wise I'm not clear what ceph or gluster would bring besides slowing the VMs down and adding HA that I'm not really needing.
Also, I find that having the NAS do mailny file-sharing allows me to actually see and use each file from the NAS which I quite like
Only VM Disks are not so easily opened But I suspect a glusterfs brick running on a docker inside the NAS will be much more opaque to the NAS itself or my windows computers accessing the NAS with CIFS...But maybe a pve cluster could still be usefull in order to manage all VMs from the same gui, but even that I'm not sure it will help if the main member of the cluster is down, could I easily have the VM automatically started on the other pve host and using the same cifs storage for the VM HDD ?
So if you have any idea of anything that would be fun, likable, or even usefull for me that a pve cluster, ceph or glusterfs could bring, and that is not purely high availability, please tell me that will give me a reason (besides fun) to start playing with it
For the time being, I just discovered that there exists pve-backup-server that seems to allow dedup and single-file restore, that is much more appealing to me (already 2TB of VM backups...) and I think I'll set this up to try it out !
Last edited: