[SOLVED] Crossing Subnets with 10G Speeds (Bypassing Router)

mihanson

Active Member
Nov 1, 2018
34
3
28
48
Hi! I’m trying to take advantage of a 10G Mellanox ConnectX-3 EN (running the most recent firmware version, 2.42.5000) installed in my Proxmox (5.2) to utilize the full 10G bandwidth between Proxmox and a seperate FreeNAS (11.1) box and I’m running into trouble. My Proxmox host (pve) is on an untagged VLAN (192.168.1.0/24) and is connected via 10G fiber to an Ubiquiti USW-48 switch via SFP+. The FreeNAS is on a tagged VLAN10 (192.168.10.0/24) and is also connected via 10G fiber to the same USW-48 switch via SFP+. FreeNAS also utilizes the same model Mellanox ConnectX-3 EN card running the most recent firmware (2.42.5000). The router/WAN gateway on my network is a Ubiquiti USG3P which has 1Gb/s RJ45 ports.

This is the interfaces file and routing table on pve:
Code:
$ cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface enp3s0 inet manual
#Realtek RTL8111GR

iface enp1s0 inet manual
#Mellanox ConnectX-3

auto vmbr0
iface vmbr0 inet static
       address  192.168.1.10
       netmask  255.255.255.0
       gateway  192.168.1.1
       bridge-ports enp1s0
       bridge-stp off
       bridge-fd 0
       bridge-vlan-aware yes                                                                                                                                                                
       bridge-vids 2-4094

Code:
root@pve:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 vmbr0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 vmbr0

Inside pve, I have guests on 3 different subnets.

Proxmox (pve) Guests:
  1. IP/Mask: 192.168.30.2/28 (VLAN30)
    GW: 192.168.30.1
  2. IP/Mask: 192.168.10.4/24 (VLAN10)
    GW: 192.168.10.1
  3. IP/Mask: 192.168.1.11/24 (Untagged VLAN)
    GW: 192.168.1.1

When I run iperf2 between FreeNAS and guest #2 above, I get expected 10G speeds because they are on the same subnet.

Code:
root@vpnhole:~# iperf -c 192.168.10.2 -t 30
------------------------------------------------------------
Client connecting to 192.168.10.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.4 port 43436 connected with 192.168.10.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  25.0 GBytes  8.16 Gbits/sec

Code:
root@vpnhole:~# traceroute -d 192.168.10.2
traceroute to 192.168.10.2 (192.168.10.2), 30 hops max, 60 byte packets
1  freenas.lan (192.168.10.2)  0.288 ms  0.205 ms  0.148 ms

Code:
root@vpnhole:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.10.1    0.0.0.0         UG        0 0          0 eth0
10.8.0.0        0.0.0.0         255.255.255.0   U         0 0          0 tun0
192.168.10.0    0.0.0.0         255.255.255.0   U         0 0          0 eth0

When i run iperf2 to test the speeds between FreeNAS and pve itself or guests 1 or 3 inside pve, I end up with 1Gb/s speeds because iperf2 is getting routed through the USG3P because I’m crossing subnets.

Code:
root@pve:~# iperf -c 192.168.10.2 -t 30
------------------------------------------------------------
Client connecting to 192.168.10.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.10 port 50362 connected with 192.168.10.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  1.04 GBytes   889 Mbits/sec

Code:
root@pve:~# traceroute -d 192.168.10.2
traceroute to 192.168.10.2 (192.168.10.2), 30 hops max, 60 byte packets
1  usg3p (192.168.1.1)  0.499 ms  5.534 ms  5.518 ms
2  192.168.10.2 (192.168.10.2)  5.496 ms  1.682 ms  2.100 ms

So my question is, how can I set up my pve networking so that pve and all guests can access FreeNAS at 10G speeds? Do I need to change network settings on the guests or the host (pve)?

I have tried the following on pve without luck, but I’m stumped on what else to do and my Google-Fu just seems to turn up this same idea and it seems to "just work" for others, but not myself, so either I have a special case here or I'm missing something.

Adding additional route and ip address in the same subnet as FreeNAS.
Code:
# ip route add 192.168.10.0/24 via 192.168.1.1 dev vmbr0
# ip addr add 192.168.10.200/24 dev vmbr0

The problem I have with this is that I can’t add a second default gateway, so recycling 192.168.1.1 still routes through the USG3P, giving me 1Gb/s speeds.

As you can probably tell, I’m no networking guru. If anyone has any pointers or needs more info, I’m more than happy to answer questions or provide more info. Thank you in advance.

Mike
 
why dont you change the

netmask 255.255.0.0

then they will be on the same subnet
Thank you for the reply.
If I change the netmask to /16 (255.255.0.0) I lose connectivity to pve (if I change it there) or any of the guests (if I change it on them).
 
You would need to move the routing function to a device that supports 10G, logically the proxmox host. You could do this either by configuring routing on the host o/s or use a VM to provide the same function. The USG would only come into play for LAN to WAN traffic.
 
You would need to move the routing function to a device that supports 10G, logically the proxmox host. You could do this either by configuring routing on the host o/s or use a VM to provide the same function. The USG would only come into play for LAN to WAN traffic.
Thank you for this. I'm trying to figure out how I accomplish using my Proxmox host (pve) to route itself and guests directly to FreeNAS without the USG sitting in the middle. Looking at the networking wiki page, do I just use the "Routed Configuration?" As I said, I'm not a networking guru and I really appreciate the help. Thank you.
Mike
 
My 1st question would be - why are you using VLAN's?
I use VLANs to isolate networks. i.e. I don't want my IoT devices to communicate with other computers on my network; I don't want my kid's computers to be able to get to mine; I don't want "guest" devices to be able to browse my network; I want to keep traffic from my multimedia computers or VOIP phones isolated.

Mike
 
In that case, I'd be inclined to install pfSense as a VM on Proxmox and use this to manage and route your vlan traffic with the USG being connected to enp3s0 as the gateway of last resort for internet traffic.

Have a look at this you tube video https://www.youtube.com/watch?v=b2w1Ywt081o which outlines the methods.

In theory this should give you FreeNAS to VM speeds of 10G
 
I worked around my issue by giving Proxmox (pve) an IPv4 address in the same subnet as my FreeNAS machine. Here is my working /etc/network/interfaces:
Code:
auto lo
iface lo inet loopback

iface enp3s0 inet manual
#Realtek RTL8111GR

iface enp1s0 inet manual
#Mellanox ConnectX-3

iface enp1s0.10 inet manual

auto vmbr10
iface vmbr10 inet static
    address  192.168.10.3
    netmask  255.255.255.0
    gateway  192.168.10.1
    bridge-ports enp1s0.10
    bridge-stp off
    bridge-fd 0
#Mellanox bug: https://forum.proxmox.com/threads/vlan-with-tag-above-126-problem.46072/ bridge-vids 2-4094

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp1s0
    bridge-stp off
    bridge-fd 0

All containers/VMs in the 192.168.10.0/24 subnet are assigned to vmbr10. All others are assigned to vmbr0. I have a switch port profile set up on my Ubiquiti switch for the port enp1s0 is connected to. The native network to this port is the default untagged LAN network (In my case this is the 192.168.1.0/24 subnet) with VLAN10 (192.168.10.0/24) and VLAN30 (192.168.30.0/28) set as additional tagged networks on the port.

Mike
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!