Mini PC Proxmox cluster with ceph

[bradin]

Hello, I am trying to create a high availability Proxmox cluster with three nodes and three SSD's however I can't seem to get it working. The monitoring on two of the three nodes isn't working and I cannot actually create the VM on the share

 

[dherzig]

Hi bradin!

It somewhat looks as if you haven't added all of your SSDs as OSDs -- is this possible (see the warning `OSD count 1 < osd_pool_default_size 3`). Is this possible?

Best,
Daniel

 

[bradin]

So i have one ssd per machine with a total of three machines. Do i have to set all the machines to one?

 

[UdoB]

So i have one ssd per machine with a total of three machines

Okay. That's the absolute minimum, but okay.

(( Edit: what I mean is: as soon as one device fails you are degraded immediately and forever. There is no chance for self-healing, which would kick in if there were at least one more node available. ))

Do i have to set all the machines to one?

Please elaborate what you mean with this sentence.

After installation of Ceph "the package" on all three Nodes you need to prepare some details. At the end you should

• have one single Ceph pool. (Multiple use of the same name for different things: a ZFS pool and/or a Ressource pool is a completely different animal.)
• have three OSDs
• have three MON = Monitors, one on each node
• have one active MGR = Manager, possibly more on standby
• ignore "Meta Data Servers" for now

The overview is at Datacenter (PVE) --> Ceph.

The creation of the OSDs is done via any one of those Nodes --> Ceph --> OSD, which then should list all three Nodes with one OSD.x each.

Disclaimer: I am not a Ceph specialist...

PS: in your second screenshot are lines with messages. The "+" at the left will open that message, possibly with more information.

PPS: Reference: https://pve.proxmox.com/pve-docs/chapter-pveceph.html

 

[wuwzy]

From my personal opinion:
The number of hard disks is too small, so it is better not to use ceph. It is better to have more than 3 osds per node.
1 for installing the system
1 for DB cache
3 osds
So 5 disk slots are OK.
If there are 3 nodes, please prepare 15 ssds.

 

[XMarcR]

From my personal opinion:
The number of hard disks is too small, so it is better not to use ceph. It is better to have more than 3 osds per node.
1 for installing the system
1 for DB cache
3 osds
So 5 disk slots are OK.
If there are 3 nodes, please prepare 15 ssds.

For production use, I'm with you. but I've run several mini-clusters for years and it's worked well, even with partitioned SSDs. By the way, the N100 Mini-PCs with 2.5G NICs are super cheap (from ~€100) and quite performant. My latest project consists of GKMTec G3 with 256GB SATA SSD and 1TB NVMe and 16GB RAM each. I love it!

 

[wuwzy]

For production use, I'm with you. but I've run several mini-clusters for years and it's worked well, even with partitioned SSDs. By the way, the N100 Mini-PCs with 2.5G NICs are super cheap (from ~€100) and quite performant. My latest project consists of GKMTec G3 with 256GB SATA SSD and 1TB NVMe and 16GB RAM each. I love it!

Yes, I agree with you. I am also looking for a combination of small size, energy saving, cheap and durable. At present, it seems that the 16th generation Intel will be a good choice. Looking forward to it.

 

[Johannes S]

For production use, I'm with you. but I've run several mini-clusters for years and it's worked well, even with partitioned SSDs. By the way, the N100 Mini-PCs with 2.5G NICs are super cheap (from ~€100) and quite performant. My latest project consists of GKMTec G3 with 256GB SATA SSD and 1TB NVMe and 16GB RAM each. I love it!

For a minimal cluster with two to three mini pcs and just one or two disks it's propably better to use the zfs storage replication feature instead of Ceph