Homelab Real World Ceph Advice For VM/LXC usage

leex12

New Member
Mar 11, 2022
20
1
3
52
I have a four node proxmox cluster for my homelab, tied up with a 10gb network! Essentailly 2 nodes are compute and 2 are storage focsused. All four nodes have docker and VMs but the load is focused on the 2 compute nodes. Using EXOS hd for storage, SSDs for VMs/LXCs. and seperate boot SSDs.

Despite all the advise i went down the consumer SSD route. Had a play and all seemed good After 3-4 months the performance of the ceph SSD pool has gone through the floor, the point of getting timeout in plex and overseer. Only got 10% wear on the SSDs and usage level (% capacity used) has been stable. I am now getiing significantly better performance on the hard drive pool so have move the 'machines' over to HDD for now. So now rethinking my strategy and looking for some real world feedback rather than theortical !

I don't have any wiggle room in the nodes so am limited to being able to have 2 sdds in 'compute' nodes in total. In the storage nodes I have the boot SSD drives mirroed in addition to the SSD for VM/LXC pool.

On the ebay hunt for used enterprise SSD and was just intending to do a like for like replacement, i.e. one SSD per node for the VM/LSX pool. However looking at the 'work' done by the boot SSD its not being hammered by any stretch of the imagination. So have this conflicting thoughts, general expereience is the more drives you have the more performant systems are however I know the guideline is not to mix use devices that are used for ceph.

So straight choice get a single medium enterprise SSD for each node which would be dedicated to the VMs and leave boot SSD on the existing consumer SSD or get two smaller enterprise drives ( one shared boot and caph). To mitigate against the conflict between system and ceph I was thinking of not maximising the space allocated to ceph but to limit it to 250gb for example if it was a 500gb drive.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!