First test cluster setup with Ceph

[osalj]

Hello Everyone,

I come here with a few questions about Proxmox cluster configuration.

What I want to do is for testing purposes.

So I have three HP Elitedesk 800 G4 DM mini computers
Each of them has:
CPU: i5-8600
RAM: 32GB RAM
SSD: 256GB for system
NVMe: 1TB for VMs
NIC1: 1 Gbps RJ45
NIC2: 2.5 Gbps RJ45

I want to create a cluster with Ceph on it.

* File system on the system disk:
I was wondering about the file system for the system disk between EXT4 and ZFS.

My SSDs are consumer and I will probably choose EXT4. On this disk there will be only the system and ISO images.

* Network:
The computers have two network interfaces.
NIC 1Gbps will be the main one on which I want to create vmbr0 (10.40.1.0/24) and it will have access to the internet.

NIC 2.5Gbps I want to use for the cluster and it will not have access to the internet. IP (10.41.41.0/24) will be configured directly on this interface without a bridge.

* Creating a cluster:
When creating a cluster I want to set the 2.5Gbps interface as Link 0.

Then when adding the remaining nodes I will also set Link 0 as the 2.5Gbps interface

* Creating Ceph:
When creating Ceph I should set the 1Gbps interface as the public network and select the 2.5Gbps interface for the cluster network?

Does this configuration look good to you?


I know that a minimum 10 Gbps network is recommended for Ceph, but what I'm doing is for testing and seeing how it works.
I currently have only this equipment to perform these tests.

TIA!

 

[UdoB]

So I have three HP Elitedesk 800 G4 DM mini computers

I want to create a cluster with Ceph on it.

This post is made exactly for you: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

I was wondering about the file system for the system disk between EXT4 and ZFS.

This post is exactly made for you: https://forum.proxmox.com/threads/f...y-a-few-disks-should-i-use-zfs-at-all.160037/

* Network:
The computers have two network interfaces.
NIC 1Gbps will be the main one on which I want to create vmbr0 (10.40.1.0/24) and it will have access to the internet.
NIC 2.5Gbps I want to use for the cluster and it will not have access to the internet. IP (10.41.41.0/24) will be configured directly on this interface without a bridge.

* Creating a cluster:
When creating a cluster I want to set the 2.5Gbps interface as Link 0.
Then when adding the remaining nodes I will also set Link 0 as the 2.5Gbps interface

* Creating Ceph:
When creating Ceph I should set the 1Gbps interface as the public network and select the 2.5Gbps interface for the cluster network?

Does this configuration look good to you?

Everything's looking fine

I know that a minimum 10 Gbps network is recommended for Ceph, but what I'm doing is for testing and seeing how it works.

Yeah. I am running 2.5 GBit/s in my Homelab. It works! But it is really slow. (Possibly because I added some pools using "erasure coding"...)