Opinion on planned proxmox cluster with CEPH

B

bigups43

New Member
Apr 26, 2025
4
1
3
Hello!

Thanks in advance for any input. Ive been reading a lot of posts about the minimum requirements for Ceph with Promox. Im planning a cluster and would like an opinion on whether or not it would be adequate or cutting it too close. Some say 3 nodes is fine, some say 5 is the minimum so its hard for me to reach a consensus.:

Three node cluster with identical specs per node:

Intel Xeon W1250 / 64GB RAM
OS drive: 250GB M.2
Ceph drives: 2x 4TB Micron 5200 ECO (plp enterprise grade SSD)
Managment network: 1Gb
VM network: 10Gb
Ceph network: 25Gb full mesh direct attached

I would use replication and not erasure coding in this setup.

Im not opposed to ZFS with replication if the specs of this cluster are simply not enough to run Ceph properly.

Thank you!
 
Last edited:
alexskysilk said:
you are putting the cart before the horse.

Why are you speccing hardware before defining what your expectations are? in the simplest terms, what are your IOPS requirements as a minimum to a single guest, and total iops in aggregate.
Click to expand...
It might have helped to say that this is simply a home lab. The cluster doesnt need to be particularly performant and Im mostly building it for fun. I run a media server, a very small k8s cluster, and a few standalone VMs, one of which is my wifi controller. The most demanding VM would be the media server which would be nice to have running at 50k IOPS for the VM disk speed but thats very close to direct hardware speed so I assume thats not possible.
 
Last edited:
media servers are bandwidth dependent, not iops. but that's probably not relevent to the conversation. you would achieve adequate performance with the hardware you specified regardless of storage choice. I would just say that the benefits of ceph are pretty much lost on the use case, so its not worth the equipment and networking overhead or the resulting engineering complexity.

you would likely achieve adequate performance even with a 10 year old pc; since performance isnt the relevant factor, only you can define what is fun.
 
  • Like
Reactions: Johannes S
alexskysilk said:
media servers are bandwidth dependent, not iops. but that's probably not relevent to the conversation. you would achieve adequate performance with the hardware you specified regardless of storage choice. I would just say that the benefits of ceph are pretty much lost on the use case, so its not worth the equipment and networking overhead or the resulting engineering complexity.

you would likely achieve adequate performance even with a 10 year old pc; since performance isnt the relevant factor, only you can define what is fun.
Click to expand...
My goal is to setup a hyper-converged cluster in my home lab. Proxmox+Ceph seems like a good option, but I do understand that it comes with added complexity. I can probably achieve what I want with ZFS + replication + HA. Thanks for your input.
 
I think for a home lab you will be fine. Good choice on the Ceph network. Your CPU may disappoint a bit, as Ceph can be CPU hungry. Replication, not erasure is the right call. The only danger spot is the boot drive. If its not a high endurance drive, you could kill it with log writes. You should end up with about 6 TB of usuable storage and and maybe 45gb of RAM per node. If that supports the VMs you want to run, go for it.
 
  • Like
Reactions: Johannes S
  • Like
Reactions: Johannes S
the other major factor for homelab deployment is that the equipment is physically likely to share space with people. this is where noise, heat, and physical form factor come into play. If it were me, with your given use case, I'd use a different approach generally: 3x minipcs supporting 2x2.5 nics for main cluster members, desktop NAS, and 9+ port 2.5g switch. space efficient, quiet, low power draw (helps with the electric budget AND reduces the floor heater effect.)

It might even outperform your proposed ceph deployment too.
 
  • Like
Reactions: Johannes S
There are several considerations. Ceph is resource (quite) hungry, especially if you would like to get some decent IOPS. The decision to run 3 nodes is good as you can run full mesh network between the nodes = you do not need power hungry 25gbps switch. Ceph also produces a LOT of logs so your bootable drive is in danger. I am using 7.68TB NVMes and they have 16GB of RAM assigned and about 10GB is in use. I guess 1GB for 1TB would be good enough for home lab.

Maybe ZFS and replication for lab is resource wise better. But I understand the motivation to play with Ceph.

There are several default settings that are IMHO not good. One of them is # PG assignment, as the minimum is 32 and the ratio is not defined, you can end up with way too low # PGs. The default setting of 2GB of ram per OSD is also quite low, maybe fine for home lab with smaller drives.
 
  • Like
Reactions: Johannes S
Can you clarify the network configuration? Is the "25Gb full mesh" you're referring to a dual-port NIC host-only setup or something else? Are the 1G and 10G dual ports per host as well and connected to redundant switches?

If you want or need VM-level HA with absolute zero virtual disk data loss when a host fails, then Ceph is probably the best option, although LINSTOR might also be of interest. Ceph of course is directly integrated with PVE, and LINSTOR isn't, but LINSTOR has some attractive features for some workloads/configurations.

If you can live with a minute or so of lost writes in a VM, then ZFS replication could be more performant in this situation as well as simpler to troubleshoot compared to Ceph if there is a major issue. There is a non-zero chance that your three-node Ceph cluster could go down due to software/hardware/user error and now all of your VMs are offline at once. One way to reduce the pain for this could be a fourth PVE host with PBS as a VM and some local storage, along with an aggressive backup schedule (minutes). You can live restore some critical VMs to this backup host while you work to revive the Ceph cluster.

The non-redundant PVE OS drive is slightly less than ideal in my opinion. So I think if I had this hardware and wanted a more straightforward setup, I would remove the M.2 drive and go with mirrored ZFS with the 4TB drives for PVE and VMs, then set up ZFS replication in a circle, such that node 1 replicates to node 2, 2 to 3, and 3 to 1. This allows easy maintenance because you can hot migrate node 1 VMs to node 2 in a few seconds, then back when you're done. Every host and its VMS can stand on its own if necessary.

So ultimately I think if you can provide full network redundancy for the Ceph cluster, then go for it, but if not, ZFS + replication.

Side note, for network redundancy where the fast port count is limited, you can use the slower port as a backup only in a bond. 10G+1G, but 1G is only for backup if your 10G switch dies. Things are painful, but still working.
 
J-Rod said:
Can you clarify the network configuration? Is the "25Gb full mesh" you're referring to a dual-port NIC host-only setup or something else? Are the 1G and 10G dual ports per host as well and connected to redundant switches?

If you want or need VM-level HA with absolute zero virtual disk data loss when a host fails, then Ceph is probably the best option, although LINSTOR might also be of interest. Ceph of course is directly integrated with PVE, and LINSTOR isn't, but LINSTOR has some attractive features for some workloads/configurations.

If you can live with a minute or so of lost writes in a VM, then ZFS replication could be more performant in this situation as well as simpler to troubleshoot compared to Ceph if there is a major issue. There is a non-zero chance that your three-node Ceph cluster could go down due to software/hardware/user error and now all of your VMs are offline at once. One way to reduce the pain for this could be a fourth PVE host with PBS as a VM and some local storage, along with an aggressive backup schedule (minutes). You can live restore some critical VMs to this backup host while you work to revive the Ceph cluster.

The non-redundant PVE OS drive is slightly less than ideal in my opinion. So I think if I had this hardware and wanted a more straightforward setup, I would remove the M.2 drive and go with mirrored ZFS with the 4TB drives for PVE and VMs, then set up ZFS replication in a circle, such that node 1 replicates to node 2, 2 to 3, and 3 to 1. This allows easy maintenance because you can hot migrate node 1 VMs to node 2 in a few seconds, then back when you're done. Every host and its VMS can stand on its own if necessary.

So ultimately I think if you can provide full network redundancy for the Ceph cluster, then go for it, but if not, ZFS + replication.

Side note, for network redundancy where the fast port count is limited, you can use the slower port as a backup only in a bond. 10G+1G, but 1G is only for backup if your 10G switch dies. Things are painful, but still working.
Click to expand...
The 25Gb mesh configuration Im talking about is the full mesh network which I would set up as routed with fallback: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server . The 10Gb ports on each node would be LACP to a pair of stacked or MLAG switches (havent decided on a model yet), which would then be connected to my HA pfSense pair.

I dont want to have a fourth PVE host, although I do understand the benefits. I agree that a single OS drive is not ideal and i would probably have two m.2 drives in RAID1. I would opt for enterprise grade drives for that as well.

I do like the idea of using ZFS and replication in a circular fashion like you mentioned. I do prefer having the OS drives separate from the storage drives, but other than that do you have any other suggestions for that setup?

Thanks for the input!
 
  • Like
Reactions: Johannes S
If you have the ability to run enterprise NVMe drives in each node, then consider using those for Ceph instead of the SATA drives. The lower the latency and higher the throughput for Ceph, the better.

For playing around at home, Ceph is probably a must-have, at least for gaining experience. But it's also restrictive/inconvenient if you've allocated all your storage to it. If you want to redesign your Ceph configuration or make some other major change, you'll need some place to stash your VMs.

If you plan to run pfSense HA VMs on PVE, I wouldn't store both on Ceph, because I consider Ceph a single point of failure. It may survive certain hardware faults, but probably not major user/configuration errors. You'll have to upgrade both PVE and Ceph eventually, and a mistake or glitch could result in PVE and/or Ceph blowing up in your face, and you may not have the skill/time to fix it (or it could be a newly discovered bug that someone will have to fix). So have solid, aggressive backups and a DR plan.
 
You must log in or register to reply here.
Top Bottom
Back