Hardware configuration e network setup

S

Samuele Bistoletti

Member
Oct 2, 2018
33
1
11
34
Hi,
we are planning a migration to a 3 nodes Proxmox cluster with Ceph.
We have 3 identicals servers with the following specs each:

Dell PowerEdge R640
Intel Xeon Silver 4110 2.1G, 8C/16T
RAM 64 GB

1x SSD 240 GB Intel S4600 for Proxmox OS
2x SSD 480 GB Samsung SM863 -> OSD (SSD Pool)
3x Spinning 1TB 7.2K RPM SATA -> OSD (HDD Pool)

2x Intel X550 10Gb Base-T
2x Intel I350 1Gb Base-T

Also we have two Netgear M7100 switches, with a 4 port LAG (LACP) for switch HA.

I planned to setup the network as following:

Proxmox Management / Cluster Communication / Coro Sync -> VLAN 1050 over 10Gb ethernet bond
Ceph Network -> VLAN 1060 also over 10Gb ethernet bond
VMs Network (LAN) -> VLAN 1055 over 1Gb ethernet bond
Outside Network (DMZ) -> VLAN 1040 over 1Gb ethernet bond

Bond 6 (ALB) for ethernet bonding on the two 10Gb interfaces of nodes is ok?
Bond 6 (ALB) for ethernet bonding on the two 1Gb interfaces of nodes is ok?
OpenVSwitch is ok with such a configuration or should we use another kind of bond mode instead of that?
In your opinion, network configuration (VLAN separation, CEPH + cluster communication) is ok or can we do better?
Coro Sync should be separated from Proxmox management network? In case, a different VLAN is ok for that?

Many thanks for the help!
 
Hi,

Samuele Bistoletti said:
Bond 6 (ALB) for ethernet bonding on the two 10Gb interfaces of nodes is ok?
Bond 6 (ALB) for ethernet bonding on the two 1Gb interfaces of nodes is ok?
Click to expand...
Why do you do not use LACP?

Samuele Bistoletti said:
OpenVSwitch is ok with such a configuration or should we use another kind of bond mode instead of that?
Click to expand...
For what propose you need OVS? I would stay on Linux Network model because it reduces the complexity and OVS can nothing that Linux bride can do.

Samuele Bistoletti said:
In your opinion, network configuration (VLAN separation, CEPH + cluster communication) is ok or can we do better?
Click to expand...
The problem with VLAN is you must guarantee that corosync network has the highest prio and will proceed faster.
The most switches are not capable to provide VLAN and QoS in that quality corosync is needed.
Keep in mind corosync is very latency sensible and VLAN, QoS, MLAG and QoS will take CPU from your switch.
 
wolfgang said:
Why do you do not use LACP?
Click to expand...

We could. It's about complexity, we have two switch that are not stackable (natively speaking), so we should use MLAG, and we are not confident with that. But at this point we will try with LACP and MLAG.

wolfgang said:
For what propose you need OVS? I would stay on Linux Network model because it reduces the complexity and OVS can nothing that Linux bride can do.
Click to expand...

I thought it was better in order to manage multiple VLANs on the same bridge (some VMs have multiple network interfaces attached, with different VLAN tag: we have another VLAN that I have not listed here). For example a VM can have 3 interfaces, one on VLAN 1055, one on VLAN 1040 and one on VLAN 1056. All this interfaces are attached to the same bridge. Do you have a different solution or an example for that with linux bonding?

Thanks
 
Ok thanks.
One last question.
By adding one dual port 1G ethernet adapter per node we could have a dedicated network for Corosync in order to avoid all possible issues connected with cluster sync.
In that case, is it safe to keep cluster management network (access to Proxmox interface) as a VLAN on the same Ceph bond (the one on the only two 10 Gb interfaces)? I read that Proxmox will use cluster management network for live migrations, and we would like these to be as fast as possible, so the 10 Gb interface seems to be a good choice. On the other hands, I'm afraid that moving the cluster management network, as a VLAN, over the 1GB dual nic used for VMs and DMZ, would saturate that communication channel in case of live migrations, isn't it?
Do you have any advice with that hardware configuration?

Thanks
 
Samuele Bistoletti said:
I read that Proxmox will use cluster management network for live migrations
Click to expand...
You can set the migration network.
see man datacenter.conf

I would use a dedicated network for corosync, no vlan.
 
  • Like
Reactions: Patrick987 and Samuele Bistoletti
wolfgang said:
I would use a dedicated network for corosync, no vlan.
Click to expand...

I will reserve 2 NIC for that (one per switch with dual ring for Corosync).

wolfgang said:
You can set the migration network. see man datacenter.conf
Click to expand...

At this point I have no idea on what network use for migration. I have 6 NICs (2x10Gb and 4x1 Gb), I will use the 2x10 Gb (one per switch) for Ceph, 2x1 Gb (also one per switch) for Corosync and 2x1 Gb (still one per switch) for VMs and DMZ with VLANs. What physical network could I use for migrations in your opinion (I would use a VLAN on that in order to separate the networks)?

Thanks
 
I would use the 10GBit network may by with VLAN to make traffic shaping.
Both ceph and migration network are ok if the latency goes up.
 
  • Like
Reactions: Samuele Bistoletti
wolfgang said:
I would use the 10GBit network may by with VLAN to make traffic shaping.
Both ceph and migration network are ok if the latency goes up.
Click to expand...

What about MTU? I read the docs but I can't find any hint.
I've been advised to set MTU to 9000 on Ceph network because of better performance, but I have some questions. If I set the mtu (with mtu directive in /etc/network/interfaces) to 9000 on Ceph bond then I have to also set it on underlying network interfaces (for example eth0 and eth1)? Many guides I found by searching on Google advise about using also pre-up command on bond to set mtu on underlying network interface (such as pre-up ip link set <interface name> mtu 9000), is it really necessary?

Thanks
 
Samuele Bistoletti said:
pre-up command on bond to set mtu on underlying network interface (such as pre-up ip link set <interface name> mtu 9000), is it really necessary?
Click to expand...
You can set the MTU direct on the bond and the nic will get this setting.
 
Hi again,
I made this setup on my /etc/network/interfaces. I have seperated Ceph Public Net from Ceph Cluster Net because I would like to have a VM monitoring Ceph with Prometheus (on the public net). This is the reason why I created vmbr1060, in order to have a VM attached to that network.
First question is: should I set IP address of Ceph public network of that node on the bridge (like in that config) or on the bond itself?
Second question is: at the moment I called all bridges with names like vmbrNNNN like suggested on proxmox web gui, but on proxmox docs it is suggested to call vlan bridges like this vmbr0v5, but this way the bridge is marked as type unknown on web gui. Is there any difference?
Last question: MTU settings on Ceph interfaces are correct in your opinion?

Unfortunately my switch don't support Mlag very well so I preferred to use bond 6 ALB in order to use both interfaces simultaneusly.

Thanks

Code: 
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto enp101s0f0
iface enp101s0f0 inet static
    address  10.65.0.10
    netmask  255.255.255.0
#Corosync Ring 1

auto enp101s0f1
iface enp101s0f1 inet static
    address  10.66.0.10
    netmask  255.255.255.0
#Corosync Ring 2

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode balance-alb
    mtu 9000
#Bond U - 10 Gb Network

iface bond0.1060 inet manual
    mtu 9000
#Bond T - Ceph Public Network

auto bond0.1061
iface bond0.1061 inet static
    address  10.61.0.10
    netmask  255.255.255.0
    mtu 9000
#Bond T - Ceph Cluster Network

auto bond1
iface bond1 inet manual
    bond-slaves eno3 eno4
    bond-miimon 100
    bond-mode balance-alb
#Bond U - 1 Gb Network

iface bond1.1040 inet manual
#Bond T - DMZ Network

iface bond1.1051 inet manual
#Bond T - USB Network

auto vmbr1050
iface vmbr1050 inet static
    address  10.50.0.200
    netmask  255.255.255.0
    gateway  10.50.0.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    mtu 1500
#Bridge - Cluster Management Network

auto vmbr1060
iface vmbr1060 inet static
    address  10.60.0.10
    netmask  255.255.255.0
    bridge-ports bond0.1060
    bridge-stp off
    bridge-fd 0
    mtu 9000
#Bridge - Ceph Public Network

auto vmbr1055
iface vmbr1055 inet manual
    bridge-ports bond1
    bridge-stp off
    bridge-fd 0
#Bridge - VMs Network

auto vmbr1040
iface vmbr1040 inet manual
    bridge-ports bond1.1040
    bridge-stp off
    bridge-fd 0
#Bridge - DMZ Network

auto vmbr1051
iface vmbr1051 inet manual
    bridge-ports bond1.1051
    bridge-stp off
    bridge-fd 0
#Bridge - USB Network
 
I also add that I saw a large number of dropped packet in rx on each bond and underlying slaves (enoX interfaces). The strange thing is that the drops occured in the same identical number on the 2 bonds and on the 2 underlying interfaces (completely different vlans).
Also I have a drop in a rx on all VM. Do you have any idea about that? I attach ip -s link output of proxmox node and a test vm.

Thanks

Code: 
Proxmox Node

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    RX: bytes  packets  errors  dropped overrun mcast   
    1982408630 2427462  0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    1982408630 2427462  0       0       0       0       
2: enp101s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:0a:f7:bc:a1:44 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    301535     2097     0       0       0       2097   
    TX: bytes  packets  errors  dropped carrier collsns
    9410148    54420    0       0       0       0       
3: enp101s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:0a:f7:bc:a1:45 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    304328     2116     0       0       0       2116   
    TX: bytes  packets  errors  dropped carrier collsns
    9479796    54821    0       0       0       0       
4: eno3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4f brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    6638457    19596    0       55      0       3541   
    TX: bytes  packets  errors  dropped carrier collsns
    11480413   108389   0       0       0       0       
5: eno4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4e brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    13804610   93760    0       991     0       61222   
    TX: bytes  packets  errors  dropped carrier collsns
    8306902    103466   0       0       0       0       
6: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4d brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    5098561    12518    0       55      0       1874   
    TX: bytes  packets  errors  dropped carrier collsns
    9664927    98497    0       0       0       0       
7: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4c brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    3064296    34069    0       991     0       31618   
    TX: bytes  packets  errors  dropped carrier collsns
    6331081    94733    0       0       0       0       
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr1050 state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4c brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    8162857    46587    0       1046    0       33492   
    TX: bytes  packets  errors  dropped carrier collsns
    15996008   193230   0       0       0       0       
9: vmbr1050: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4c brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    5041711    42756    0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    4784834    6489     0       0       0       0       
10: bond0.1061@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4c brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    1104       19       0       0       0       19     
    TX: bytes  packets  errors  dropped carrier collsns
    1956       26       0       0       0       0       
11: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1055 state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4e brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    20443067   113356   0       1046    0       64763   
    TX: bytes  packets  errors  dropped carrier collsns
    19787315   211855   0       0       0       0       
12: vmbr1055: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4e brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    4485912    42499    0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    1730       23       0       0       0       0       
13: vmbr1060: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4c brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    916        16       0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    1886       25       0       0       0       0       
14: bond0.1060@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr1060 state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4c brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    916        16       0       0       0       16     
    TX: bytes  packets  errors  dropped carrier collsns
    1886       25       0       0       0       0       
15: bond1.1040@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1040 state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4e brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    4160110    42809    0       0       0       31268   
    TX: bytes  packets  errors  dropped carrier collsns
    6010137    10476    0       0       0       0       
16: vmbr1040: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4e brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    1755652    31350    0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    1886       25       0       0       0       0       
17: vmbr1051: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4e brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    1984       34       0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    1886       25       0       0       0       0       
18: bond1.1051@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1051 state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:16:d0:4e brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    2030       35       0       0       0       34     
    TX: bytes  packets  errors  dropped carrier collsns
    1886       25       0       0       0       0       
21: tap55151i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1055 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 6e:32:90:79:f4:5e brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    77191      399      0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    5105070    42678    0       0       0       0       
22: tap55151i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1040 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 2e:42:0e:ca:aa:1e brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    2399680    3808     0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    3122680    35402    0       0       0       0       
23: tap50220i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1050 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 4e:4f:80:00:26:62 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    16234      217      0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    2658877    32420    0       0       0       0       
24: tap55040i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1055 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 3e:70:1c:1b:a3:9f brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    102116     1492     0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    5039899    41082    0       0       0       0       
25: tap55040i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1040 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ca:7b:29:ff:65:ec brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    3607593    6630     0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    3739753    37917    0       0       0       0       
26: tap55050i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1055 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 92:b4:19:16:28:9c brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    43374394470 2242382  0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    10803215558 2325282  0       0       0       0       
27: tap55051i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1055 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether c6:52:e8:6f:ea:db brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    5592350187 502272   0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    1155009895 598646   0       0       0       0       
28: tap55052i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1055 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 1e:e5:92:ef:f8:45 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    5772222988 847487   0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    12750099104 938599   0       0       0       0       
29: tap55170i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1055 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 26:2d:1c:4d:0b:e8 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    230479116  1275745  0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    30283641109 1177307  0       0       0       0       
30: tap55190i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1055 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether d2:dc:36:9b:26:f6 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    79044      1094     0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    7649881    43754    0       0       0       0


Code: 
Test VM

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    RX: bytes  packets  errors  dropped overrun mcast   
    1071039    5756     0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    1071039    5756     0       0       0       0       
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 2a:64:c7:39:3c:10 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    5116565    41709    0       1064    0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    109262     1569     0       0       0       0       
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 86:61:11:a6:ee:ae brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    3784260    38429    0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    3653185    6718     0       0       0       0

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 2a:64:c7:39:3c:10 brd ff:ff:ff:ff:ff:ff
    inet 10.55.0.40/24 brd 10.55.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::2864:c7ff:fe39:3c10/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 86:61:11:a6:ee:ae brd ff:ff:ff:ff:ff:ff
    inet 10.40.0.10/24 brd 10.40.0.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::8461:11ff:fea6:eeae/64 scope link 
       valid_lft forever preferred_lft forever
 
Drop in the vm seems to happen at this level: 1 drops at __netif_receive_skb_core+68f (0xffffffff8204f67f)

I have not idea what is meaning...
 
Solved! Dropped packets were due to Netgear LLDP and ISDP, for some reason they reached the VMs and a drop occured.

I have only one last problem to solve: on Ceph cluster network (where mtu is set to 9000, also on switch) performance are very bad with iperf 4.8gb/s, if I do the same test on Proxmox bond (mtu 1500) I obtain a value of 9.8gb/s. What could be the reason for that?
 
Samuele Bistoletti said:
First question is: should I set IP address of Ceph public network of that node on the bridge (like in that config) or on the bond itself?
Click to expand...
On the bond direct.
A bridge is a switch and is only needed for VM/CT and brings not needed complexity and a bit overhead.

Samuele Bistoletti said:
Second question is: at the moment I called all bridges with names like vmbrNNNN like suggested on proxmox web gui, but on proxmox docs it is suggested to call vlan bridges like this vmbr0v5, but this way the bridge is marked as type unknown on web gui. Is there any difference?
Click to expand...

You should use vlan aware bridges for VM/CT and there you do not need this naming.
 
wolfgang said:
On the bond direct.
You should use vlan aware bridges for VM/CT and there you do not need this naming.
Click to expand...

Unfortunately is not working, I'm not able to reach VMs with VLAN aware flag.. but with bridge attached to tagged bond is working very well. Isn't that correct same way?

What about MTU? Do you have any idea on why we are having this bad result on 9000 mtu network? I made a ping test with
ping -M do -s 8972 [destinationIP] and it was successfull.

Thanks
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom