Crossing Subnets with 10G Speeds (Bypassing Router)

Discussion in 'Proxmox VE: Networking and Firewall' started by mihanson, Nov 1, 2018.

  1. mihanson

    mihanson New Member

    Joined:
    Nov 1, 2018
    Messages:
    4
    Likes Received:
    0
    Hi! I’m trying to take advantage of a 10G Mellanox ConnectX-3 EN (running the most recent firmware version, 2.42.5000) installed in my Proxmox (5.2) to utilize the full 10G bandwidth between Proxmox and a seperate FreeNAS (11.1) box and I’m running into trouble. My Proxmox host (pve) is on an untagged VLAN (192.168.1.0/24) and is connected via 10G fiber to an Ubiquiti USW-48 switch via SFP+. The FreeNAS is on a tagged VLAN10 (192.168.10.0/24) and is also connected via 10G fiber to the same USW-48 switch via SFP+. FreeNAS also utilizes the same model Mellanox ConnectX-3 EN card running the most recent firmware (2.42.5000). The router/WAN gateway on my network is a Ubiquiti USG3P which has 1Gb/s RJ45 ports.

    This is the interfaces file and routing table on pve:
    Code:
    $ cat /etc/network/interfaces
    auto lo
    iface lo inet loopback
    
    iface enp3s0 inet manual
    #Realtek RTL8111GR
    
    iface enp1s0 inet manual
    #Mellanox ConnectX-3
    
    auto vmbr0
    iface vmbr0 inet static
           address  192.168.1.10
           netmask  255.255.255.0
           gateway  192.168.1.1
           bridge-ports enp1s0
           bridge-stp off
           bridge-fd 0
           bridge-vlan-aware yes                                                                                                                                                                
           bridge-vids 2-4094
    
    Code:
    root@pve:~# netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 vmbr0
    192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 vmbr0
    
    Inside pve, I have guests on 3 different subnets.

    When I run iperf2 between FreeNAS and guest #2 above, I get expected 10G speeds because they are on the same subnet.

    Code:
    root@vpnhole:~# iperf -c 192.168.10.2 -t 30
    ------------------------------------------------------------
    Client connecting to 192.168.10.2, TCP port 5001
    TCP window size: 85.0 KByte (default)
    ------------------------------------------------------------
    [  3] local 192.168.10.4 port 43436 connected with 192.168.10.2 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-30.0 sec  25.0 GBytes  8.16 Gbits/sec
    
    Code:
    root@vpnhole:~# traceroute -d 192.168.10.2
    traceroute to 192.168.10.2 (192.168.10.2), 30 hops max, 60 byte packets
    1  freenas.lan (192.168.10.2)  0.288 ms  0.205 ms  0.148 ms
    
    Code:
    root@vpnhole:~# netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         192.168.10.1    0.0.0.0         UG        0 0          0 eth0
    10.8.0.0        0.0.0.0         255.255.255.0   U         0 0          0 tun0
    192.168.10.0    0.0.0.0         255.255.255.0   U         0 0          0 eth0
    
    When i run iperf2 to test the speeds between FreeNAS and pve itself or guests 1 or 3 inside pve, I end up with 1Gb/s speeds because iperf2 is getting routed through the USG3P because I’m crossing subnets.

    Code:
    root@pve:~# iperf -c 192.168.10.2 -t 30
    ------------------------------------------------------------
    Client connecting to 192.168.10.2, TCP port 5001
    TCP window size: 85.0 KByte (default)
    ------------------------------------------------------------
    [  3] local 192.168.1.10 port 50362 connected with 192.168.10.2 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-30.0 sec  1.04 GBytes   889 Mbits/sec
    
    Code:
    root@pve:~# traceroute -d 192.168.10.2
    traceroute to 192.168.10.2 (192.168.10.2), 30 hops max, 60 byte packets
    1  usg3p (192.168.1.1)  0.499 ms  5.534 ms  5.518 ms
    2  192.168.10.2 (192.168.10.2)  5.496 ms  1.682 ms  2.100 ms
    
    So my question is, how can I set up my pve networking so that pve and all guests can access FreeNAS at 10G speeds? Do I need to change network settings on the guests or the host (pve)?

    I have tried the following on pve without luck, but I’m stumped on what else to do and my Google-Fu just seems to turn up this same idea and it seems to "just work" for others, but not myself, so either I have a special case here or I'm missing something.

    Adding additional route and ip address in the same subnet as FreeNAS.
    Code:
    # ip route add 192.168.10.0/24 via 192.168.1.1 dev vmbr0
    # ip addr add 192.168.10.200/24 dev vmbr0
    
    The problem I have with this is that I can’t add a second default gateway, so recycling 192.168.1.1 still routes through the USG3P, giving me 1Gb/s speeds.

    As you can probably tell, I’m no networking guru. If anyone has any pointers or needs more info, I’m more than happy to answer questions or provide more info. Thank you in advance.

    Mike
     
  2. zarlo5899

    zarlo5899 New Member

    Joined:
    Aug 14, 2018
    Messages:
    14
    Likes Received:
    1
    why dont you change the

    netmask 255.255.0.0

    then they will be on the same subnet
     
  3. mihanson

    mihanson New Member

    Joined:
    Nov 1, 2018
    Messages:
    4
    Likes Received:
    0
    Thank you for the reply.
    If I change the netmask to /16 (255.255.0.0) I lose connectivity to pve (if I change it there) or any of the guests (if I change it on them).
     
  4. bobmc

    bobmc Member

    Joined:
    May 17, 2018
    Messages:
    50
    Likes Received:
    8
    You would need to move the routing function to a device that supports 10G, logically the proxmox host. You could do this either by configuring routing on the host o/s or use a VM to provide the same function. The USG would only come into play for LAN to WAN traffic.
     
  5. mihanson

    mihanson New Member

    Joined:
    Nov 1, 2018
    Messages:
    4
    Likes Received:
    0
    Thank you for this. I'm trying to figure out how I accomplish using my Proxmox host (pve) to route itself and guests directly to FreeNAS without the USG sitting in the middle. Looking at the networking wiki page, do I just use the "Routed Configuration?" As I said, I'm not a networking guru and I really appreciate the help. Thank you.
    Mike
     
  6. bobmc

    bobmc Member

    Joined:
    May 17, 2018
    Messages:
    50
    Likes Received:
    8
    My 1st question would be - why are you using VLAN's?
     
  7. mihanson

    mihanson New Member

    Joined:
    Nov 1, 2018
    Messages:
    4
    Likes Received:
    0
    I use VLANs to isolate networks. i.e. I don't want my IoT devices to communicate with other computers on my network; I don't want my kid's computers to be able to get to mine; I don't want "guest" devices to be able to browse my network; I want to keep traffic from my multimedia computers or VOIP phones isolated.

    Mike
     
  8. bobmc

    bobmc Member

    Joined:
    May 17, 2018
    Messages:
    50
    Likes Received:
    8
    In that case, I'd be inclined to install pfSense as a VM on Proxmox and use this to manage and route your vlan traffic with the USG being connected to enp3s0 as the gateway of last resort for internet traffic.

    Have a look at this you tube video https://www.youtube.com/watch?v=b2w1Ywt081o which outlines the methods.

    In theory this should give you FreeNAS to VM speeds of 10G
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice