Proxmox Ceph network configuration

[zeuxprox]

Hi,

first of all I'm very sorry for my poor English...

In few weeks I have to setup a Proxmox/Ceph cluster of 3 Nodes. Each node is a Ceph node end a Proxmox node in wich I virtualize some VMs and some LXC (Debian 9) containers.
Each node has:
CPU: 2 x Intel Xeon Gold 6130
RAM: 196 GB
SSD: 2 x SSD class enterprise (
NVMe: 1 x 800 GB
HDD: 6 X 8GB HGST Ultrastar HE10
Controller Raid: Adaptec ASR-8405
Controller HBA: LSI 9305-16i
4 x 10Gb nic
4 x 1Gb nic

I'll configure 2 SSD in RAID 1 (proxmox installation). NVMe is for journaling and HDDs for OSD.

My doubts are about network configuration of Ceph and Proxmox Cluster.
I would like reach this goal:

Ceph cluster network: 10.10.1.0/24 (1 x 10 Gb port)
Ceph public network: 10.10.2.0/24 (1 x 10 Gb port)

Proxmox cluster network: 10.10.3.0/24 (1 x 1 Gb port)

Proxmox public network: 2 x 1 Gb ports (this port will be associated/bridged to the VMs/LXC containers)
What I want is that OSDs use Ceph cluster network (10.10.1.0/24) for replication and for all other job they have to do to work correctly while Ceph public network is used by proxmox and other (eventually) clients to read an write in Ceph storage.

Is correct this configuration? If Yes, how can be done in Proxmox?

Thank you very much

 

[wolfgang]

Hi,

the best and easiest way is to change the config /etc/pve/ceph.conf after you call pveceph init.

Your network setting is ok .
But it would be better to use 2 nics with 2 rings for corosync.

 

[zeuxprox]

So, if I have understood well, Proxmox, and eventually other clients, will use Ceph Public network (10.10.2.0/24) to read and write on Ceph Cluster, but if so, I have to add a 10 Gb port to Proxmox and connect it to Ceph Public network, right?

Thank you very much

 

[wolfgang]

I have to add a 10 Gb port to Proxmox

I do not exactly understand this.
You wrote you like a 10 GBit for Ceph cluster and 10GBit for Ceph public.

 

[zeuxprox]

The Ceph network configuration is:
1 x 10 Gb for Ceph Cluster (communication between OSD)
1 x 10 Gb fo Ceph Public (on this interface Ceph "listen" the client requests)

My doubt is: which interface does Proxmox use to connect to Ceph Public? I need to use another 1 x 10 Gb port connected to Ceph Public network to connect Proxmox to Ceph Public network, right?

 

[wolfgang]

If your Ceph and Proxmox Node are the same you need no extra client Nic.

The setup with your requirements work like this.

Create a Proxmox VE cluster with this 3 nodes.

Install ceph with

pveceph install --version luminous

Then set up the init config with the public network

pveceph init --network 10.10.2.0/24

After the initial config is done create on all there nodes a monitor

pveceph createmon

Now you can add the cluster network to the config
add this line under public network in /etc/pve/ceph.conf

cluster network = 10.10.1.0/24

Now you can create your osd witch communicate on 10.10.1.0/24 network.[/CODE]

 

[zeuxprox]

Thank you very much

 

[alexskysilk]

But it would be better to use 2 nics with 2 rings for corosync.

Sorry to Hijack, but I've always used an active/active bond for corosync with a single ring. I'm not suggesting it is a better way, but can you chime in what are the pros and cons of either approach?

 

[wolfgang]

Corosync with two rings can recover partial sections in case of failure.
Two separat ring are more robust in case of interfering in one network.
It is one layer of complexity less because you are using a plain nic and not a bond.
Bond can produce extra latency through the bonding algorithms.

 

[zeuxprox]

What can you say me about a second ring also for Ceph cluster network? I was thinking: there is the possibility to configure a secondary Ceph cluster network that save me in case of the primary Ceph cluster network stops to work (Switch failure) ?

Tahank you

 

[wolfgang]

Ceph has no two rings.
You have to use bonding to make them fault-tolerant.
What you may mean are the cluster and public network?
http://docs.ceph.com/docs/jewel/rados/configuration/network-config-ref/

 

[zeuxprox]

Wolfgang, I know the differences between public and cluster network. My question is: what happen if the 10 Gb switch of the cluster network stops to work?

 

[Leron]

I am currently testing proxmox on a configuration with three nodes and ceph storage on node.
I created a cluster and initialised ceph with a dedicatied network equipement (one NIC par node and a switch) for ceph (public and cluster on the same network).
But this network is a SPOF, if the dedicated switch become down, ceph stops.

On a SAN configuration, two SAN fabric is needed but i know, ceph is not a traditional storage.

So, what's the good practice for not SPOF on our infrastructure ?
May be Bonding ? but with this problem :

Bond can produce extra latency through the bonding algorithms.

A better solution ?
Thx

 

[Dzung]

On this config if I change network
Ceph public network: 10.10.2.0/24 (1 x 10 Gb port)
Ceph cluster network + Proxmox cluster network : 10.10.1.0/24 (2 x 10 Gb port, ring 0, ring 1)
VM public network: 2 x 1 Gb ports (this port will be associated/bridged to the VMs/LXC containers)
Proxmox managerment: 1Gb port.
Is correct this configuration?