switchless 3-node ceph, direct connections

[mouk]

Hi,

Is it possible to use direct 10G connections between my three proxmox/ceph machines for ceph traffic?

The machines have dual 10G nic plus regular gigabit nics, so physically I can connect the machines to each other on 10G. I'm just not sure how to configure the 10G nics. Should I be looking at bonding and/or bridging?

I could use some pointers / ideas...

 

[tom]

Yes, I will send more info later today.

 

[mouk]

That is great. Thanks in advance, looking forward to your info.

 

[tom]

See this example configuration:

http://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

 

[mouk]

Hi Tom,

Great, thanks. Just to make sure I understand things correctly:

* the 10.14.14.50/51/52 on eth3 has nothing to do with the ceph mesh, and (probably? perhaps?) is to restrict access the proxmox management interface from that subnet?

* the 10.15.15.50/51/52 is where the magic happens: you give *the same* ip (.50) to two different nics (I didn't know that was possible!) Then using routing you make one .50 nic to talk to the .51, and the other .50 nic to the .52.

* route -n output will be generated automatically, and is just listed to compare/verify your setup against ours.

Do I understand things correctly?

 

[tom]

Hi Tom,

Great, thanks. Just to make sure I understand things correctly:

* the 10.14.14.50/51/52 on eth3 has nothing to do with the ceph mesh, and (probably? perhaps?) is to restrict access the proxmox management interface from that subnet?

this network was used for the proxmox ve cluster communication.

* the 10.15.15.50/51/52 is where the magic happens: you give *the same* ip (.50) to two different nics (I didn't know that was possible!) Then using routing you make one .50 nic to talk to the .51, and the other .50 nic to the .52.

* route -n output will be generated automatically, and is just listed to compare/verify your setup against ours.

Do I understand things correctly?

yes, the route -n was just added for illustration.

 

[mouk]

Thanks very much for your insight, Tom!

 

[rafafell]

See this example configuration:

http://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

In the "Example" it is written "4 Network cards in each server". It would not be better "4 Network ports in each server"?

The "ens21 LAN (for fileserver etc.) 10.14.14.y" is not necessary for ceph, correct?

Does this "ens21" port go to the router?

If you do not use the "ens21" port, just do not put it in "/etc/network/interface"?

In case of purchase of a network switch. Which features do you recommend?


best

 

[codedmind]

Hello @tom sorry, but i'm trying configure a cluster and find your wiki on full Mesh, but i'm not able to put it to work.

As i have 4 ethernets ports available i pretend using two ports to connect to each server (maybe get 2gb) could this be done? From your post i understand that for each server you bond two ports but one go to node 2 and other to node 3.

Should i do a bond with the 4 ports in node1 and then connect two cables in node2 and node 3 ? Will this be any good?

 

[Alwin]

Should i do a bond with the 4 ports in node1 and then connect two cables in node2 and node 3 ? Will this be any good?

Just try it out. Whichever works best for you.