Hey all, I've got the following network setup:
I've configured my network as follows on both proxmox1 and proxmox3:
And the Unifi side:
Synology: 10.88.20.1
proxmox1: 10.88.20.10
proxmox3: 10.88.20.13
When I do an Iperf3 test from proxmox1 or proxmox3 to my Synology I get (almost) full 10gbe speed:
However when I run the same test from proxmox1 to proxmox3 or the reverse I only get 2.5gbe:
This also seems to impact the speed of live migrations as they also never go above 2.5gbit.
Is there a way to configure proxmox + networking to allow to bond the speed of the 4x 2.5gbit network cards when communicating to other proxmox nodes?
Edit:
If I do a -R (reverse) Iperf from my NAS to my proxmox nodes I also get 2.5gbit. So for some reason it seems that any traffic flowing TO the proxmox nodes is capped at 2.5gbit, whereas any traffic flowing FROM the proxmox nodes can use the full combined bandwith.
I've configured my network as follows on both proxmox1 and proxmox3:
And the Unifi side:
Synology: 10.88.20.1
proxmox1: 10.88.20.10
proxmox3: 10.88.20.13
When I do an Iperf3 test from proxmox1 or proxmox3 to my Synology I get (almost) full 10gbe speed:
Code:
root@proxmox3:~# iperf3 -c 10.88.20.1 -P 10 -t 1000
Connecting to host 10.88.20.1, port 5201
[ 5] local 10.88.20.13 port 57878 connected to 10.88.20.1 port 5201
[ 7] local 10.88.20.13 port 57888 connected to 10.88.20.1 port 5201
[ 9] local 10.88.20.13 port 57900 connected to 10.88.20.1 port 5201
[ 11] local 10.88.20.13 port 57906 connected to 10.88.20.1 port 5201
[ 13] local 10.88.20.13 port 57910 connected to 10.88.20.1 port 5201
[ 15] local 10.88.20.13 port 57926 connected to 10.88.20.1 port 5201
[ 17] local 10.88.20.13 port 57932 connected to 10.88.20.1 port 5201
[ 19] local 10.88.20.13 port 57936 connected to 10.88.20.1 port 5201
[ 21] local 10.88.20.13 port 57942 connected to 10.88.20.1 port 5201
[ 23] local 10.88.20.13 port 57956 connected to 10.88.20.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 25.4 MBytes 213 Mbits/sec 0 256 KBytes
[ 7] 0.00-1.00 sec 71.8 MBytes 602 Mbits/sec 0 454 KBytes
[ 9] 0.00-1.00 sec 37.3 MBytes 313 Mbits/sec 0 303 KBytes
[ 11] 0.00-1.00 sec 25.7 MBytes 216 Mbits/sec 0 307 KBytes
[ 13] 0.00-1.00 sec 160 MBytes 1.34 Gbits/sec 1 1.80 MBytes
[ 15] 0.00-1.00 sec 35.5 MBytes 298 Mbits/sec 0 305 KBytes
[ 17] 0.00-1.00 sec 159 MBytes 1.33 Gbits/sec 3 1.89 MBytes
[ 19] 0.00-1.00 sec 157 MBytes 1.32 Gbits/sec 1 1.89 MBytes
[ 21] 0.00-1.00 sec 23.4 MBytes 196 Mbits/sec 0 277 KBytes
[ 23] 0.00-1.00 sec 72.9 MBytes 611 Mbits/sec 0 496 KBytes
[SUM] 0.00-1.00 sec 768 MBytes 6.45 Gbits/sec 5
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 22.9 MBytes 192 Mbits/sec 22 126 KBytes
[ 7] 1.00-2.00 sec 68.4 MBytes 573 Mbits/sec 66 180 KBytes
[ 9] 1.00-2.00 sec 38.4 MBytes 322 Mbits/sec 42 165 KBytes
[ 11] 1.00-2.00 sec 21.3 MBytes 179 Mbits/sec 26 93.3 KBytes
[ 13] 1.00-2.00 sec 219 MBytes 1.83 Gbits/sec 249 322 KBytes
[ 15] 1.00-2.00 sec 33.2 MBytes 278 Mbits/sec 38 122 KBytes
[ 17] 1.00-2.00 sec 218 MBytes 1.82 Gbits/sec 256 284 KBytes
[ 19] 1.00-2.00 sec 225 MBytes 1.89 Gbits/sec 260 345 KBytes
[ 21] 1.00-2.00 sec 26.8 MBytes 225 Mbits/sec 17 129 KBytes
[ 23] 1.00-2.00 sec 68.9 MBytes 578 Mbits/sec 49 181 KBytes
[SUM] 1.00-2.00 sec 941 MBytes 7.89 Gbits/sec 1025
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 25.5 MBytes 214 Mbits/sec 0 214 KBytes
[ 7] 2.00-3.00 sec 69.5 MBytes 583 Mbits/sec 5 257 KBytes
[ 9] 2.00-3.00 sec 36.0 MBytes 302 Mbits/sec 5 197 KBytes
[ 11] 2.00-3.00 sec 19.1 MBytes 160 Mbits/sec 7 141 KBytes
[ 13] 2.00-3.00 sec 169 MBytes 1.42 Gbits/sec 30 461 KBytes
[ 15] 2.00-3.00 sec 35.1 MBytes 295 Mbits/sec 3 191 KBytes
[ 17] 2.00-3.00 sec 172 MBytes 1.45 Gbits/sec 23 431 KBytes
[ 19] 2.00-3.00 sec 172 MBytes 1.45 Gbits/sec 30 462 KBytes
[ 21] 2.00-3.00 sec 26.7 MBytes 224 Mbits/sec 0 222 KBytes
[ 23] 2.00-3.00 sec 69.3 MBytes 581 Mbits/sec 8 262 KBytes
[SUM] 2.00-3.00 sec 795 MBytes 6.67 Gbits/sec 111
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 23.5 MBytes 197 Mbits/sec 0 233 KBytes
[ 7] 3.00-4.00 sec 70.2 MBytes 589 Mbits/sec 0 324 KBytes
[ 9] 3.00-4.00 sec 34.9 MBytes 292 Mbits/sec 0 243 KBytes
[ 11] 3.00-4.00 sec 23.0 MBytes 193 Mbits/sec 0 221 KBytes
[ 13] 3.00-4.00 sec 161 MBytes 1.35 Gbits/sec 0 639 KBytes
[ 15] 3.00-4.00 sec 35.4 MBytes 297 Mbits/sec 0 239 KBytes
[ 17] 3.00-4.00 sec 159 MBytes 1.33 Gbits/sec 0 638 KBytes
[ 19] 3.00-4.00 sec 159 MBytes 1.33 Gbits/sec 0 652 KBytes
[ 21] 3.00-4.00 sec 23.7 MBytes 199 Mbits/sec 0 238 KBytes
[ 23] 3.00-4.00 sec 70.6 MBytes 592 Mbits/sec 0 337 KBytes
[SUM] 3.00-4.00 sec 760 MBytes 6.37 Gbits/sec 0However when I run the same test from proxmox1 to proxmox3 or the reverse I only get 2.5gbe:
Code:
root@proxmox3:~# iperf3 -c 10.88.20.10 -P 10 -t 1000
Connecting to host 10.88.20.10, port 5201
[ 5] local 10.88.20.13 port 38608 connected to 10.88.20.10 port 5201
[ 7] local 10.88.20.13 port 38610 connected to 10.88.20.10 port 5201
[ 9] local 10.88.20.13 port 38614 connected to 10.88.20.10 port 5201
[ 11] local 10.88.20.13 port 38622 connected to 10.88.20.10 port 5201
[ 13] local 10.88.20.13 port 38628 connected to 10.88.20.10 port 5201
[ 15] local 10.88.20.13 port 38636 connected to 10.88.20.10 port 5201
[ 17] local 10.88.20.13 port 38646 connected to 10.88.20.10 port 5201
[ 19] local 10.88.20.13 port 38660 connected to 10.88.20.10 port 5201
[ 21] local 10.88.20.13 port 38676 connected to 10.88.20.10 port 5201
[ 23] local 10.88.20.13 port 38688 connected to 10.88.20.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 23.4 MBytes 196 Mbits/sec 65 93.3 KBytes
[ 7] 0.00-1.00 sec 32.0 MBytes 269 Mbits/sec 65 99.0 KBytes
[ 9] 0.00-1.00 sec 22.2 MBytes 186 Mbits/sec 60 76.4 KBytes
[ 11] 0.00-1.00 sec 29.6 MBytes 248 Mbits/sec 77 91.9 KBytes
[ 13] 0.00-1.00 sec 20.6 MBytes 173 Mbits/sec 87 52.3 KBytes
[ 15] 0.00-1.00 sec 36.5 MBytes 306 Mbits/sec 63 93.3 KBytes
[ 17] 0.00-1.00 sec 32.8 MBytes 276 Mbits/sec 44 93.3 KBytes
[ 19] 0.00-1.00 sec 36.6 MBytes 307 Mbits/sec 72 70.7 KBytes
[ 21] 0.00-1.00 sec 29.6 MBytes 248 Mbits/sec 61 84.8 KBytes
[ 23] 0.00-1.00 sec 24.3 MBytes 204 Mbits/sec 69 83.4 KBytes
[SUM] 0.00-1.00 sec 288 MBytes 2.41 Gbits/sec 663
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 22.7 MBytes 191 Mbits/sec 77 133 KBytes
[ 7] 1.00-2.00 sec 23.0 MBytes 193 Mbits/sec 72 42.4 KBytes
[ 9] 1.00-2.00 sec 39.5 MBytes 331 Mbits/sec 59 103 KBytes
[ 11] 1.00-2.00 sec 36.7 MBytes 308 Mbits/sec 70 63.6 KBytes
[ 13] 1.00-2.00 sec 24.9 MBytes 209 Mbits/sec 49 89.1 KBytes
[ 15] 1.00-2.00 sec 28.8 MBytes 241 Mbits/sec 61 87.7 KBytes
[ 17] 1.00-2.00 sec 29.9 MBytes 251 Mbits/sec 26 106 KBytes
[ 19] 1.00-2.00 sec 26.2 MBytes 220 Mbits/sec 66 42.4 KBytes
[ 21] 1.00-2.00 sec 24.7 MBytes 207 Mbits/sec 80 91.9 KBytes
[ 23] 1.00-2.00 sec 24.3 MBytes 204 Mbits/sec 93 69.3 KBytes
[SUM] 1.00-2.00 sec 281 MBytes 2.35 Gbits/sec 653
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 21.8 MBytes 183 Mbits/sec 95 56.6 KBytes
[ 7] 2.00-3.00 sec 30.5 MBytes 256 Mbits/sec 45 100 KBytes
[ 9] 2.00-3.00 sec 25.0 MBytes 210 Mbits/sec 84 53.7 KBytes
[ 11] 2.00-3.00 sec 27.1 MBytes 227 Mbits/sec 76 82.0 KBytes
[ 13] 2.00-3.00 sec 27.7 MBytes 232 Mbits/sec 63 80.6 KBytes
[ 15] 2.00-3.00 sec 40.1 MBytes 336 Mbits/sec 31 110 KBytes
[ 17] 2.00-3.00 sec 26.5 MBytes 222 Mbits/sec 71 55.1 KBytes
[ 19] 2.00-3.00 sec 22.1 MBytes 185 Mbits/sec 65 70.7 KBytes
[ 21] 2.00-3.00 sec 30.0 MBytes 252 Mbits/sec 44 102 KBytes
[ 23] 2.00-3.00 sec 29.9 MBytes 251 Mbits/sec 59 70.7 KBytes
[SUM] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec 633
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-3.35 sec 8.08 MBytes 195 Mbits/sec 23 55.1 KBytes
[ 7] 3.00-3.35 sec 14.5 MBytes 350 Mbits/sec 6 165 KBytes
[ 9] 3.00-3.35 sec 7.77 MBytes 188 Mbits/sec 33 63.6 KBytes
[ 11] 3.00-3.35 sec 8.58 MBytes 207 Mbits/sec 20 87.7 KBytes
[ 13] 3.00-3.35 sec 6.59 MBytes 159 Mbits/sec 37 41.0 KBytes
[ 15] 3.00-3.35 sec 12.2 MBytes 294 Mbits/sec 9 90.5 KBytes
[ 17] 3.00-3.35 sec 11.5 MBytes 278 Mbits/sec 0 144 KBytes
[ 19] 3.00-3.35 sec 10.9 MBytes 263 Mbits/sec 11 107 KBytes
[ 21] 3.00-3.35 sec 6.71 MBytes 162 Mbits/sec 41 39.6 KBytes
[ 23] 3.00-3.35 sec 11.0 MBytes 266 Mbits/sec 18 65.0 KBytes
[SUM] 3.00-3.35 sec 97.7 MBytes 2.36 Gbits/sec 198
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-3.35 sec 76.0 MBytes 191 Mbits/sec 260 sender
[ 5] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiver
[ 7] 0.00-3.35 sec 100 MBytes 251 Mbits/sec 188 sender
[ 7] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiver
[ 9] 0.00-3.35 sec 94.5 MBytes 237 Mbits/sec 236 sender
[ 9] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiver
[ 11] 0.00-3.35 sec 102 MBytes 255 Mbits/sec 243 sender
[ 11] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiver
[ 13] 0.00-3.35 sec 79.8 MBytes 200 Mbits/sec 236 sender
[ 13] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiver
[ 15] 0.00-3.35 sec 118 MBytes 295 Mbits/sec 164 sender
[ 15] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiver
[ 17] 0.00-3.35 sec 101 MBytes 252 Mbits/sec 141 sender
[ 17] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiver
[ 19] 0.00-3.35 sec 95.7 MBytes 240 Mbits/sec 214 sender
[ 19] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiver
[ 21] 0.00-3.35 sec 91.0 MBytes 228 Mbits/sec 226 sender
[ 21] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiver
[ 23] 0.00-3.35 sec 89.5 MBytes 224 Mbits/sec 239 sender
[ 23] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiver
[SUM] 0.00-3.35 sec 947 MBytes 2.37 Gbits/sec 2147 sender
[SUM] 0.00-3.35 sec 0.00 Bytes 0.00 bits/sec receiverThis also seems to impact the speed of live migrations as they also never go above 2.5gbit.
Is there a way to configure proxmox + networking to allow to bond the speed of the 4x 2.5gbit network cards when communicating to other proxmox nodes?
Edit:
If I do a -R (reverse) Iperf from my NAS to my proxmox nodes I also get 2.5gbit. So for some reason it seems that any traffic flowing TO the proxmox nodes is capped at 2.5gbit, whereas any traffic flowing FROM the proxmox nodes can use the full combined bandwith.
Last edited: