Advice on new Ceph cluster

[Florius]

Hi guys.
I'm setting up a new Proxmox cluster with Ceph with 3 Dell PowerEdge R610's.
I'm planning on running an internal network which is only available when connected to a VPN and was planning it for using it to connect to the nodes in the cluster.
And an external network on which customers have access to.

What is the recommended configuration if you lack 10G ports?
Is 1 1G port enough for external connections and 1 bond of 3 NIC's for Ceph and internal traffic?

Any advice is appreciated, thank you!

EDIT: Not sure if useful information, but it will run 40-50 VM's for now in total.

 

[wolfgang]

Hi,

I think you won't be happy with 3 GBit nics this is about 150MB - 200 MB throughput in total.

 

[Florius]

Hi,

Okay. I tried to get it working with 2 NIC's but wasn't able to get it working...:

unfortunately for some reason I am not getting any speed above 100Mb/s.

My config is as follow:

/# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto bond1
iface bond1 inet manual
        slaves eno3 eno4
        bond_miimon 100
        bond_mode 802.3ad

auto bond0
iface bond0 inet manual
        slaves eno1 eno2
        bond_miimon 100
        bond_mode 802.3ad

auto vmbr0
iface vmbr0 inet static
        address  X.X.X.X
        netmask  255.255.255.0
        gateway  X.X.X.X
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet static
        address  10.0.0.1
        netmask  255.255.255.0
        bridge_ports bond1
        bridge_stp off
        bridge_fd 0

Both public (I have removed the IP's for safety reasons!) and internal should both be bonded according to '/proc/net/bonding/bondX':

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: f0:4d:a2:00:58:8f
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 9
        Partner Key: 51
        Partner Mac Address: 00:15:60:fe:46:80

Slave Interface: eno1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: f0:4d:a2:00:58:8f
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: f0:4d:a2:00:58:8f
    port key: 9
    port priority: 255
    port number: 1
    port state: 61
details partner lacp pdu:
    system priority: 18048
    system mac address: 00:15:60:fe:46:80
    oper key: 51
    port priority: 0
    port number: 1
    port state: 61

Slave Interface: eno2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: f0:4d:a2:00:58:91
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: f0:4d:a2:00:58:8f
    port key: 9[code]

port priority: 255
port number: 2
port state: 61
details partner lacp pdu:
system priority: 18048
system mac address: 00:15:60:fe:46:80
oper key: 51
port priority: 0
port number: 2
port state: 61
[/code]

I use a HP Procurve HP2824:

interface 1
   no lacp
exit
interface 2
   no lacp
exit
interface 5
   no lacp
exit
interface 6
   no lacp
exit
trunk 1-2 Trk1 LACP
trunk 5-6 Trk2 LACP
snmp-server community "public" Unrestricted
vlan 1
   name "DEFAULT_VLAN"
   untagged 16-17,23-24
   ip address dhcp-bootp
   no untagged 3-4,7-15,18-22,Trk1-Trk2
   exit
vlan 40
   name "OOB"
   untagged 3-4,7-8,11-12,21
   ip address 10.0.0.254 255.255.255.0
   exit
vlan 300
   name "IPMI"
   untagged 13-15,18-19
   tagged 24
   exit
vlan 133
   name "VLAN133"
   untagged 9-10,22,Trk1-Trk2
   tagged 24
   exit
vlan 135
   name "VLAN135"
   untagged 20
   tagged 24
   exit
spanning-tree Trk1 priority 4
spanning-tree Trk2 priority 4

Config is as above. I test it via scp downloading an ISO file from one server to another, and tried both public and internal. (Perhaps it's the testing method, as it is a single file?)

 

[wolfgang]

This 150-200 was a very optimistic guess.

But first why do you use a vbmr for ceph?
A bond is enough.

What is the network load looks like.
You can use bmon to monitor it
bmon -p 'eno1,eno2' -o format:fmt='$(element:name) $(attr:rxrateackets) $(attr:txrateackets)\n' > netuse.txt

 

[Florius]

I thought VMBR was required by Proxmox to assign IP addresses.
I'm planning on using this with WHMCS to automatically assign IP addresses.
But it doesn't explain the 'slow connection' right?

Currently there are no VM's on the cluster (as I haven't have Ceph yet working), below netuse.txt:

eno1 0.00 0.00
eno2 0.00 0.00
eno1 0.00 0.00
eno2 3.00 1.00
eno1 0.00 0.00
eno2 0.75 0.25
eno1 0.00 0.00
eno2 0.94 0.81
eno1 0.00 0.00
eno2 0.23 0.20
eno1 0.00 0.00
eno2 1.56 0.80
eno1 0.00 0.00
eno2 1.89 1.70
eno1 0.00 0.00
eno2 1.97 1.18
eno1 0.00 0.00
eno2 0.49 0.29
eno1 0.00 0.00
eno2 0.12 0.07
eno1 0.00 0.00
eno2 1.53 0.77
eno1 0.00 0.00
eno2 0.38 0.19
eno1 0.00 0.00
eno2 0.10 0.05
eno1 0.00 0.00
eno2 1.52 1.51
eno1 0.00 1.00
eno2 1.13 1.13
eno1 0.00 0.25
eno2 1.78 1.78
eno1 1.00 0.06
eno2 2.70 0.45
eno1 1.00 0.02
eno2 0.67 0.11
eno1 0.25 0.00
eno2 0.17 0.03
eno1 0.06 0.00
eno2 0.04 0.01
eno1 0.02 0.00
eno2 0.76 0.75
eno1 0.00 0.00
eno2 2.44 1.69
eno1 1.50 0.00
eno2 2.11 1.17
eno1 3.38 0.00
eno2 0.53 0.29

 

[wolfgang]

But it doesn't explain the 'slow connection' right?

Only when the cpu is very busy it will slow down the net speed.

Sorry to be not precise enough, I meant the network speed under benchmark load load.