Proxmox + Ceph = 4 NICS?

gasmanc

New Member
Jul 29, 2019
8
0
1
38
Hi,

I’m in the planning stages of setting up a three node cluster. 2 of the nodes will have dual 10gbe and dual 1gbe NICs, and the other has a single 10gbe Nic and a dual 1gbe Nic.

Reading through the documentation and doing searches it seems that Proxmox wants 2 NICs - 1 for Corosync and one for everything else, and Ceph wants a public and a private network - so 4 NICs??

If I only want to use three NICs, could I use a 1gbe Nic for Corosync, a 10gbe Nic for Proxmox and ceph public network and the the last 10geb (1gbe for one of the nodes) for the ceph osd private network?

Are there any additional steps for sharing the Proxmox/ceph public network?

Thanks
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
100
5
18
Ceph can be split into a public and private network but does not have to. You can run Ceph on one NIC only quite fine. Do not try to run Ceph on a 1G NIC, even on one server. That won't make you happy.
 

gasmanc

New Member
Jul 29, 2019
8
0
1
38
What about separating Corosync network and Ceph osd network for rebalancing? How can I share the Ceph public network and Proxmox public network?
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
100
5
18
What about separating Corosync network and Ceph osd network for rebalancing? How can I share the Ceph public network and Proxmox public network?
Don't confuse the two networks for Ceph (cluster, public) with the corosync cluster network.

You can have both Ceph networks configured in the same network so they both use the one 10G NIC. In the config you would use the same IP for both networks.
Corosync on the other hand does not need a lot of bandwidth but wants low latency. For this don't configure you corosync network on the same physical network on which you run stuff that can demand a lot of the network like any storage traffic.

Run the Corosync on one of the 1G NICs. To have a fallback you can configure a second Link (ring in the config file) on the other 1G NIC (Section in the manual).

TL;DR:
Ceph Cluster and Public networks are both only related to Ceph, need lots of bandwidth and can be run on the same physical network -> 10G NIC
Corosync cluster network is not related to any Ceph networks, likes low latency -> 1G NIC with possible fallback on the second 1G NIC
 
  • Like
Reactions: gasmanc

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!