10G write speed in VM | Ceph?

[GameJunkie98]

Hello everyone,

I am currently planning a new Proxmox cluster for my Homelab.

Because my firewall is to run in the cluster, the cluster should have shared storage for the VM disks and HA features.
The firewall cannot run as a cluster, as only one external IP (PPPoE) is available and double NAT should be avoided.

One VM in the cluster should be a file server for ~10TB of data.

Is it possible in a 3 node Ceph cluster to transfer files from a client with the full 10G to the file server (write performance)?

The main goals are:
- full 10G transfer to the fileserver VM (read & write)
- HA / automatic transfer of VMs in the event of a node failure

iSCSI to Synology etc. is again a single point of failure.

What else would you suggest as shared storage?

Thanks in advance

 

[ness1602]

Yes if you use nvme disks, something like 4-6 per node.

 

[GameJunkie98]

Yes if you use nvme disks, something like 4-6 per node.

Do you have any recommendations on which SSDs for ceph?

 

[SteveITS]

Proxmox has a benchmark PDF at https://proxmox.com/en/downloads/proxmox-virtual-environment. Note their network speeds in it.

 

[LnxBil]

Yes if you use nvme disks, something like 4-6 per node.

4-6 SATA SSDs mabe, with NVMe you need normally just one. PCIe 5.0 SSDs is capable of up to 16 GB/s (not Gb). We benchmarked PCIe 4.0 SSDs with 7,1 GB/s, which is 56 Gb ... Therefore you need for a REALLY fast system multiplt 100Gb or even multiple 400Gb. And please ONLY buy enterprise hardware, consumer and prosumer stuff is not worth using in CEPH (or ZFS).

 

[GameJunkie98]

4-6 SATA SSDs mabe, with NVMe you need normally just one. PCIe 5.0 SSDs is capable of up to 16 GB/s (not Gb). We benchmarked PCIe 4.0 SSDs with 7,1 GB/s, which is 56 Gb ... Therefore you need for a REALLY fast system multiplt 100Gb or even multiple 400Gb. And please ONLY buy enterprise hardware, consumer and prosumer stuff is not worth using in CEPH (or ZFS).

Do you mean the whole hardware?
Or only the osds?

I want to use the Minisforum MS-01.

 

[Johannes S]

You might also want to read the relevant part of the reference documentation https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster and UdoBs writeup https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

To be honest I'm not sure whether Ceph will be worth it in a homelab setup for the reasons pointed out by LnxBil and Udo. So I would go with ZFS storage replication instead or a setup where you have one node with a NAS OS and a PBS VM (which also acts as a qdevice) and two compute nodes or a combination of both approaches.