Hi! I’m trying to take advantage of a 10G Mellanox ConnectX-3 EN (running the most recent firmware version, 2.42.5000) installed in my Proxmox (5.2) to utilize the full 10G bandwidth between Proxmox and a seperate FreeNAS (11.1) box and I’m running into trouble. My Proxmox host (pve) is on an untagged VLAN (192.168.1.0/24) and is connected via 10G fiber to an Ubiquiti USW-48 switch via SFP+. The FreeNAS is on a tagged VLAN10 (192.168.10.0/24) and is also connected via 10G fiber to the same USW-48 switch via SFP+. FreeNAS also utilizes the same model Mellanox ConnectX-3 EN card running the most recent firmware (2.42.5000). The router/WAN gateway on my network is a Ubiquiti USG3P which has 1Gb/s RJ45 ports.
This is the interfaces file and routing table on pve:
Inside pve, I have guests on 3 different subnets.
When I run iperf2 between FreeNAS and guest #2 above, I get expected 10G speeds because they are on the same subnet.
When i run iperf2 to test the speeds between FreeNAS and pve itself or guests 1 or 3 inside pve, I end up with 1Gb/s speeds because iperf2 is getting routed through the USG3P because I’m crossing subnets.
So my question is, how can I set up my pve networking so that pve and all guests can access FreeNAS at 10G speeds? Do I need to change network settings on the guests or the host (pve)?
I have tried the following on pve without luck, but I’m stumped on what else to do and my Google-Fu just seems to turn up this same idea and it seems to "just work" for others, but not myself, so either I have a special case here or I'm missing something.
Adding additional route and ip address in the same subnet as FreeNAS.
The problem I have with this is that I can’t add a second default gateway, so recycling 192.168.1.1 still routes through the USG3P, giving me 1Gb/s speeds.
As you can probably tell, I’m no networking guru. If anyone has any pointers or needs more info, I’m more than happy to answer questions or provide more info. Thank you in advance.
Mike
This is the interfaces file and routing table on pve:
Code:
$ cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface enp3s0 inet manual
#Realtek RTL8111GR
iface enp1s0 inet manual
#Mellanox ConnectX-3
auto vmbr0
iface vmbr0 inet static
address 192.168.1.10
netmask 255.255.255.0
gateway 192.168.1.1
bridge-ports enp1s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
Code:
root@pve:~# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 vmbr0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0Inside pve, I have guests on 3 different subnets.
When I run iperf2 between FreeNAS and guest #2 above, I get expected 10G speeds because they are on the same subnet.
Code:
root@vpnhole:~# iperf -c 192.168.10.2 -t 30
------------------------------------------------------------
Client connecting to 192.168.10.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.10.4 port 43436 connected with 192.168.10.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-30.0 sec 25.0 GBytes 8.16 Gbits/sec
Code:
root@vpnhole:~# traceroute -d 192.168.10.2
traceroute to 192.168.10.2 (192.168.10.2), 30 hops max, 60 byte packets
1 freenas.lan (192.168.10.2) 0.288 ms 0.205 ms 0.148 ms
Code:
root@vpnhole:~# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.10.1 0.0.0.0 UG 0 0 0 eth0
10.8.0.0 0.0.0.0 255.255.255.0 U 0 0 0 tun0
192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0When i run iperf2 to test the speeds between FreeNAS and pve itself or guests 1 or 3 inside pve, I end up with 1Gb/s speeds because iperf2 is getting routed through the USG3P because I’m crossing subnets.
Code:
root@pve:~# iperf -c 192.168.10.2 -t 30
------------------------------------------------------------
Client connecting to 192.168.10.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.10 port 50362 connected with 192.168.10.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-30.0 sec 1.04 GBytes 889 Mbits/sec
Code:
root@pve:~# traceroute -d 192.168.10.2
traceroute to 192.168.10.2 (192.168.10.2), 30 hops max, 60 byte packets
1 usg3p (192.168.1.1) 0.499 ms 5.534 ms 5.518 ms
2 192.168.10.2 (192.168.10.2) 5.496 ms 1.682 ms 2.100 msSo my question is, how can I set up my pve networking so that pve and all guests can access FreeNAS at 10G speeds? Do I need to change network settings on the guests or the host (pve)?
I have tried the following on pve without luck, but I’m stumped on what else to do and my Google-Fu just seems to turn up this same idea and it seems to "just work" for others, but not myself, so either I have a special case here or I'm missing something.
Adding additional route and ip address in the same subnet as FreeNAS.
Code:
# ip route add 192.168.10.0/24 via 192.168.1.1 dev vmbr0
# ip addr add 192.168.10.200/24 dev vmbr0The problem I have with this is that I can’t add a second default gateway, so recycling 192.168.1.1 still routes through the USG3P, giving me 1Gb/s speeds.
As you can probably tell, I’m no networking guru. If anyone has any pointers or needs more info, I’m more than happy to answer questions or provide more info. Thank you in advance.
Mike