With ceph network is there a difference in performance if I use 2 network interfaces directly for example eth1 and eth2 vs. bonding the same interfaces into vmbr ?
Ceph network
- Thread starter brucexx
- Start date
I guess I did not ask the question correctly, is it possible to use two interfaces on one node without bonding them either on private or public network ? and if it is, is there a difference in performance ?
yes, you can. simply setup 2 ip address in 2 different network.
But I personnaly prefer to use bonding for redondancy.
No, it wouldnt improve performance. Ceph performance is directly related to the latency of your interface, multiple interfaces with the same latency would still yield the same performance.
As Spirit suggested, best practice is to bond interfaces for fault tolerance (active/backup.)
you can also using bonding with lacp layer3+4.
each osd connect to other osd, and client connect to each osd.
So multiple connection, and it's loadbalancing fine.
each osd connect to other osd, and client connect to each osd.
So multiple connection, and it's loadbalancing fine.
thats true, but unless you have SSDs with multiple 12gig HBAs you're not likely to realize much benefit. ymmv, of course 
...well I have 18 10K 6Gbps SAS drives with 3 dedicated storage servers each with multiple 10Gbps interfaces. Hence I was wondering if better to bond or use dedicated interfaces (if possible) and if I gain any speed benefits and/or only redundancy with bonding. the drives are JBOD but the RAID's controller they are connected to max capacity is 12 Gbps. I like the load balancing idea.
You could bond and dedicate at the same time as long as you use active/backup bonding.
Using 2 vlans, you can create eth1.10 and eth2.10 for Ceph Private network and eth1.20 and eth2.20 for Ceph Public network. bond0 will use eth1.10 and eth2.10 for its interfaces and set eth1.10 as the primary for the bond. bond1 will use eth1.20 and eth2.20 for its interfaces and use eth2.20 as primary.
In a steady state, eth1 will carry Ceph Private traffic and eth2 will carry Ceph Public traffic. If one interface fails, or switch fails, both vlans will be carried across the same interface.
Let me know if you would like to see a network config. I use this all over the place.
Using 2 vlans, you can create eth1.10 and eth2.10 for Ceph Private network and eth1.20 and eth2.20 for Ceph Public network. bond0 will use eth1.10 and eth2.10 for its interfaces and set eth1.10 as the primary for the bond. bond1 will use eth1.20 and eth2.20 for its interfaces and use eth2.20 as primary.
In a steady state, eth1 will carry Ceph Private traffic and eth2 will carry Ceph Public traffic. If one interface fails, or switch fails, both vlans will be carried across the same interface.
Let me know if you would like to see a network config. I use this all over the place.
While I'm not sure what you mean by "Ceph Public Traffic," generally speaking such a configuration is NOT safe for production, precisely because what you pointed out happens on link fault. Ceph should NOT share traffic as it is very sensitive to latency disruption.
NIC ports are cheap and plentiful. No reason to build a potential breakpoint into your design to save those.
I mis-typed. Ceph operates using a Cluster network for data replication between nodes and a Public network for Ceph monitor control and client access. http://docs.ceph.com/docs/jewel/rados/configuration/network-config-ref/
Per your own comment above, "multiple interfaces with the same latency would still yield the same performance", latency does not become an issue in the event of a failure, but rather throughput. You are now sharing the overall pipe with the two networks. Yes, Ceph relies heavily on network performance. While many like to speak in general terms regarding what is recommended, I actually run this configuration on quite a few Ceph clusters. When there is a network outage, there is no noticeable performance loss except in the event of OSD recovery when an OSD fails. Then the only noticeable event is replication will take longer as there is not as much bandwidth available. Also, when an interface fails, that is high priority break-fix item in the environments I operate.
10G nic ports are not cheap and plentiful for everyone. This design works for environments I have installed it in which sound very similar to the OPs environment.
Per your own comment above, "multiple interfaces with the same latency would still yield the same performance", latency does not become an issue in the event of a failure, but rather throughput. You are now sharing the overall pipe with the two networks. Yes, Ceph relies heavily on network performance. While many like to speak in general terms regarding what is recommended, I actually run this configuration on quite a few Ceph clusters. When there is a network outage, there is no noticeable performance loss except in the event of OSD recovery when an OSD fails. Then the only noticeable event is replication will take longer as there is not as much bandwidth available. Also, when an interface fails, that is high priority break-fix item in the environments I operate.
10G nic ports are not cheap and plentiful for everyone. This design works for environments I have installed it in which sound very similar to the OPs environment.
I am not trying to talk you out of your design- I'm only saying its not best practice. You built the config and accepted the quirks specific to it- thats fine. Suggesting the quirks dont exist because they're not a problem for you is... irresponsible.
Well I have 4 x 10Gbps interfaces (2 NICs with 2 ports each) per server so I can separate Ceph public and Ceph private networks completely and with bonding two ports on two different cards achieve redundancy on the switch as well as in case one of the NICs fails.
Hello @valeech,
This message is a bit old, I found it useful. Could you please share a network config example? I would like to adapt this network configuration for my ceph cluster.
This message is a bit old, I found it useful. Could you please share a network config example? I would like to adapt this network configuration for my ceph cluster.
Here is a snippet from one of my interfaces files:
# Ceph Cluster
# Activated on eth2, eth3
# eth2 Primary
auto eth2.200
iface eth2.200 inet manual
bond-master Bond200
bond-primary eth2.200
bond-mode active-backup
bond-miimon 100
bond-updelay 45000
pre-up ifconfig eth2 up
post-up ifconfig $IFACE mtu 9000 up
pre-down ifconfig $IFACE down
auto eth3.200
iface eth3.200 inet manual
bond-master Bond200
bond-primary eth2.200
pre-up ifconfig eth3 up
post-up ifconfig $IFACE mtu 9000 up
pre-down ifconfig $IFACE down
auto Bond200
iface Bond200 inet manual
pre-up ifconfig eth2.200 up && ifconfig eth3.200 up
post-up ifconfig Bond200 mtu 9000
post-down ifconfig eth3.200 down && ifconfig eth2.200 down
auto vmbr200
iface vmbr200 inet static
address 10.XX.X.XX
netmask 255.255.255.192
gateway 10.XX.X.X
bridge_ports Bond200
bridge_stp off
bridge_fd 0
dns-nameservers 10.XX.XX.XX
dns-search de.neustar.com neustar.com
# CEPH Public
# Activated on eth2, eth3
# eth3 Primary
auto eth3.201
iface eth3.201 inet manual
bond-master Bond201
bond-primary eth3.201
bond-mode active-backup
bond-miimon 100
bond-updelay 45000
post-up ifconfig $IFACE mtu 9000 up
pre-down ifconfig $IFACE down
auto eth2.201
iface eth2.201 inet manual
bond-master Bond201
bond-primary eth3.201
post-up ifconfig $IFACE mtu 9000 up
pre-down ifconfig $IFACE down
auto Bond201
iface Bond201 inet manual
pre-up ifconfig eth3.201 up && ifconfig eth2.201 up
post-up ifconfig Bond201 mtu 9000
post-down ifconfig eth2.201 down && ifconfig eth3.201 down
auto vmbr201
iface vmbr201 inet static
address 192.XXX.X.XX
netmask 255.255.255.192
bridge_ports Bond201
bridge_stp off
bridge_fd 0
# Ceph Cluster
# Activated on eth2, eth3
# eth2 Primary
auto eth2.200
iface eth2.200 inet manual
bond-master Bond200
bond-primary eth2.200
bond-mode active-backup
bond-miimon 100
bond-updelay 45000
pre-up ifconfig eth2 up
post-up ifconfig $IFACE mtu 9000 up
pre-down ifconfig $IFACE down
auto eth3.200
iface eth3.200 inet manual
bond-master Bond200
bond-primary eth2.200
pre-up ifconfig eth3 up
post-up ifconfig $IFACE mtu 9000 up
pre-down ifconfig $IFACE down
auto Bond200
iface Bond200 inet manual
pre-up ifconfig eth2.200 up && ifconfig eth3.200 up
post-up ifconfig Bond200 mtu 9000
post-down ifconfig eth3.200 down && ifconfig eth2.200 down
auto vmbr200
iface vmbr200 inet static
address 10.XX.X.XX
netmask 255.255.255.192
gateway 10.XX.X.X
bridge_ports Bond200
bridge_stp off
bridge_fd 0
dns-nameservers 10.XX.XX.XX
dns-search de.neustar.com neustar.com
# CEPH Public
# Activated on eth2, eth3
# eth3 Primary
auto eth3.201
iface eth3.201 inet manual
bond-master Bond201
bond-primary eth3.201
bond-mode active-backup
bond-miimon 100
bond-updelay 45000
post-up ifconfig $IFACE mtu 9000 up
pre-down ifconfig $IFACE down
auto eth2.201
iface eth2.201 inet manual
bond-master Bond201
bond-primary eth3.201
post-up ifconfig $IFACE mtu 9000 up
pre-down ifconfig $IFACE down
auto Bond201
iface Bond201 inet manual
pre-up ifconfig eth3.201 up && ifconfig eth2.201 up
post-up ifconfig Bond201 mtu 9000
post-down ifconfig eth2.201 down && ifconfig eth3.201 down
auto vmbr201
iface vmbr201 inet static
address 192.XXX.X.XX
netmask 255.255.255.192
bridge_ports Bond201
bridge_stp off
bridge_fd 0