Hello there,
I've been experimenting with the Proxmox HA stack in my lab for the last few months, but I'm now at a point where I need some advice on how to move forward. For reference, I'm running a small homelab with mostly consumer-grade hardware/used server stuff from a few years back. My nodes are all running Sandy-Bridge 4C8T CPUs and 16GB of RAM, my NAS/"SAN" is a low-power FreeNAS Box with a dual-core and 8GB of ECC RAM. The nodes are connected to a central switch via Gigabit-Ethernet, the FreeNAS box has 2-port LAGG bond to the same switch.
As for software, I'm mostly using LXC containers (~20 total) on top of Proxmox 5.2, with a few VMs thrown in. The root filesystems are currently on a NFS share provided by the FreeNAS box, using 2 SSDs in a mirror ZFS array for storage. Not exactly terrific, I know. In fact, the storage solution has given me a few headaches recently and I'm looking for a more elegant and more consistent solution. Essentially, I need a shared FS that can hold both VMs and LXC containers (and can preferably do snapshots)
Looking at this handy storage table, it looks like there are a few options that fit my requirements:
Something iSCSI-based sounds interesting, but I'm not sure about the stability: How well does LVM over iSCSI work within a cluster of nodes? Any issues with nodes updating or anything like that? If anyone has any experiences, I'd be very happy if they shared them.
Ceph seems like a good solution, but perhaps a bit "overkill" for my setup. Since FreeNAS doesn't support it and I'm not planning on adding another box for storage, I'd probably use the built-in PVE Ceph server.
I'm curious to hear what you think. Any suggestions would be appreciated. Cheers!
I've been experimenting with the Proxmox HA stack in my lab for the last few months, but I'm now at a point where I need some advice on how to move forward. For reference, I'm running a small homelab with mostly consumer-grade hardware/used server stuff from a few years back. My nodes are all running Sandy-Bridge 4C8T CPUs and 16GB of RAM, my NAS/"SAN" is a low-power FreeNAS Box with a dual-core and 8GB of ECC RAM. The nodes are connected to a central switch via Gigabit-Ethernet, the FreeNAS box has 2-port LAGG bond to the same switch.
As for software, I'm mostly using LXC containers (~20 total) on top of Proxmox 5.2, with a few VMs thrown in. The root filesystems are currently on a NFS share provided by the FreeNAS box, using 2 SSDs in a mirror ZFS array for storage. Not exactly terrific, I know. In fact, the storage solution has given me a few headaches recently and I'm looking for a more elegant and more consistent solution. Essentially, I need a shared FS that can hold both VMs and LXC containers (and can preferably do snapshots)
Looking at this handy storage table, it looks like there are a few options that fit my requirements:
- NFS/CIFS - What I'm using right now. It works, except when it doesn't
- iSCSI - Doesn't seem to support LXC containers directly, same for ZFS over iSCSI. Guess that makes sense, given that iSCSI devices are pure block devices
- LVM on iSCSI - Seems to fit my criteria, but I'm not sure about the stability
- ZFS over iSCSI - Not sure if this is possible at all
- Ceph - Would need to rebase my VM storage away from FreeNAS, but it seems to be a decent solution otherwise, if somewhat overkill
Something iSCSI-based sounds interesting, but I'm not sure about the stability: How well does LVM over iSCSI work within a cluster of nodes? Any issues with nodes updating or anything like that? If anyone has any experiences, I'd be very happy if they shared them.
Ceph seems like a good solution, but perhaps a bit "overkill" for my setup. Since FreeNAS doesn't support it and I'm not planning on adding another box for storage, I'd probably use the built-in PVE Ceph server.
I'm curious to hear what you think. Any suggestions would be appreciated. Cheers!