[SOLVED] Multi-WAN Proxmox: VM 2nd NIC not working

T

Thomas Ganter

New Member
Jul 21, 2024
5
0
1
Prien a Chiemsee, Germany
Hello world,

I have Proxmox set up with two NICs to have IP Addresses in two subnets that each have a dedicated/ separate public IP Address.
I have added a second bridge and bound it to the second NIC, and I can confirm routing works:

Code: 
root@prxmx-01:~# echo Primary $( curl -sX GET --interface vmbr0 ifconfig.me ) ; echo Backup $( curl -sX GET --interface vmbr1 ifconfig.me )
Primary 93.225.17.142
Backup 217.93.126.178
root@prxmx-01:~# ip a s vmbr0
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:6b:00:62:7b:7e brd ff:ff:ff:ff:ff:ff
    inet 172.20.30.11/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::9e6b:ff:fe62:7b7e/64 scope link 
       valid_lft forever preferred_lft forever
root@prxmx-01:~# ip a s vmbr1
15: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 20:7b:d2:8f:b0:4e brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.11/24 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::227b:d2ff:fe8f:b04e/64 scope link 
       valid_lft forever preferred_lft forever
root@prxmx-01:~#

I have a VM running HAProxy which I want to have an IP on both networks as well.

This I have configured:
1721585969902.png

This is what I get inside the VM:
Code: 
prxmx-haproxy:~$ sudo cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

iface eth0 inet static
        address 172.20.30.50/32

auto eth1
iface eth1 inet dhcp

prxmx-haproxy:~$ sudo service networking restart
 * WARNING: you are stopping a boot service
 * Stopping haproxy ... [ ok ]
 * Stopping networking ...
 *   lo ... [ ok ]
 *   eth0 ... [ ok ]
 * Starting networking ...
 *   lo ... [ ok ]
 *   eth0 ...
udhcpc: started, v1.36.1
udhcpc: broadcasting discover
udhcpc: broadcasting select for 172.20.30.203, server 172.20.30.1
udhcpc: lease of 172.20.30.203 obtained from 172.20.30.1, lease time 7200 [ ok ]
 *   eth1 ...
udhcpc: started, v1.36.1
udhcpc: broadcasting discover
udhcpc: broadcasting discover
udhcpc: broadcasting discover
udhcpc: broadcasting discover
udhcpc: broadcasting discover
udhcpc failed to get a DHCP lease
udhcpc: no lease, forking to background [ ok ]
prxmx-haproxy:~$ ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:02:00:00:02:03 brd ff:ff:ff:ff:ff:ff
    inet 172.20.30.203/24 brd 172.20.30.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.20.30.50/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd39:fdfb:7567:172:2:ff:fe00:203/64 scope global dynamic flags 100
       valid_lft forever preferred_lft forever
    inet6 fe80::2:ff:fe00:203/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:45:00:00:02:03 brd ff:ff:ff:ff:ff:ff
    inet6 2003:ef:671c:f00:45:ff:fe00:203/64 scope global dynamic flags 100
       valid_lft 6698sec preferred_lft 1298sec
    inet6 fe80::45:ff:fe00:203/64 scope link
       valid_lft forever preferred_lft forever
prxmx-haproxy:~$

Even if I manually assign an IP to the eth1 interface, all my attempts to make networking work on that interface utterly fail. It should get IP address 192.168.178.50 assigned from the DHCP in the Backup Subnet.

Interestingly, the router on the 192.x network does see and answer the DHCP requests. So I am leaning towards assuming the replies do not reach back to the VM. Since the proxmox host can talk on the 2nd network without any issues, I pressume something is not routing properly on the host.

I am lost.
Please help.

(updated: added more detail)
 
Last edited:
Thomas Ganter said:
auto eth0 iface eth0 inet dhcp iface eth0 inet static address 172.20.30.50/32
Click to expand...
You shouldn't assign DHCP and static IP simultaneously. Please consider removing one of them. Additionally, could you provide me with your /etc/network/interfaces file from your host?
 
thefu21 said:
You shouldn't assign DHCP and static IP simultaneously. Please consider removing one of them. Additionally, could you provide me with your /etc/network/interfaces file from your host?
Click to expand...
The interface with the two IP addresses (DHCP and static) works as expected.

It is the other one that is my worry.

Code: 
root@prxmx-01:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enp0s31f6 inet manual

iface enx207bd28fb04e inet manual
#Backup-WAN

auto vmbr0
iface vmbr0 inet static
        address 172.20.30.11/24
        gateway 172.20.30.1
        bridge-ports enp0s31f6 enp2s0
        bridge-stp off
        bridge-fd 0
        post-up /usr/sbin/ethtool -K enp0s31f6 tso off gso off

iface wlp0s20f3 inet manual

auto vmbr1
iface vmbr1 inet static
        address 192.168.178.11/24
        bridge-ports enx207bd28fb04e
        bridge-stp off
        bridge-fd 0
        post-up  echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up /usr/sbin/route add -net 0.0.0.0 gw 192.168.178.1 metric 200 vmbr1

source /etc/network/interfaces.d/*
root@prxmx-01:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.20.30.1     0.0.0.0         UG    0      0        0 vmbr0
0.0.0.0         192.168.178.1   0.0.0.0         UG    200    0        0 vmbr1
172.20.30.0     0.0.0.0         255.255.255.0   U     0      0        0 vmbr0
192.168.178.0   0.0.0.0         255.255.255.0   U     0      0        0 vmbr1
root@prxmx-01:~#

both interfaces work as expected on the proxmox host:

Code: 
root@prxmx-01:~# curl -I --interface vmbr0 heise.de
HTTP/1.1 301 Moved Permanently
Date: Mon, 22 Jul 2024 07:19:06 GMT
Server: Apache
X-Cobbler: servo65.heise.de
X-Pect: The Spanish Inquisition
X-Clacks-Overhead: GNU Terry Pratchett
X-42: DON'T PANIC
Location: https://www.heise.de/
Connection: close
Content-Type: text/html; charset=iso-8859-1

root@prxmx-01:~# curl -I --interface vmbr1 heise.de
HTTP/1.1 301 Moved Permanently
Date: Mon, 22 Jul 2024 07:19:10 GMT
Server: Apache
X-Cobbler: servo65.heise.de
X-Pect: The Spanish Inquisition
X-Clacks-Overhead: GNU Terry Pratchett
X-42: DON'T PANIC
Location: https://www.heise.de/
Connection: close
Content-Type: text/html; charset=iso-8859-1

root@prxmx-01:~# echo Primary $( curl -sX GET --interface vmbr0 ifconfig.me ) ; echo Backup $( curl -sX GET --interface vmbr1 ifconfig.me )
Primary 93.225.21.26
Backup 217.93.126.178
root@prxmx-01:~#

On the VM however, the 2nd interface does not even activate on it's own ... 8^(
 
bridge-ports enp0s31f6 enp2s0
bridge-stp off
post-up /usr/sbin/ethtool -K enp0s31f6 tso off gso off
Click to expand...
What is the purpose of the second interface (enp2s0), and why have you disabled STP when using two interfaces? Additionally, why have you disabled hardware offloading?
post-up /usr/sbin/route add -net 0.0.0.0 gw 192.168.178.1 metric 200 vmbr1
Click to expand...
Why are you configuring a second gateway, as shown below? This configuration doesn't seem to make much sense.
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
Click to expand...
What is the reason for enabling IP forwarding?

Could you please clarify the goal you are trying to achieve with these settings?
 
thefu21 said:
Could you please clarify the goal you are trying to achieve with these settings?
Click to expand...

My goal is to host VMs on two public IP addresses.
This is my setup:

Code: 
                                                 +----------------------+                                                    
                                                 |                      |                                                    
                                                 |   PROXMOX VE HOST    |                                                    
                                +----------------|   prxmx-01           |                                                    
                                | enp2s0         |                      |                                                    
                                | 2.5GBit Link   |  +----------------+  |                                                    
                                +----------------|  | VM HAProxy     |  |                                                    
PRIMARY     +--------------+    +----------------|  | prxmx-haproxy  |  |-----------------+    +----------------+      BACKUP
INTERNET<---|172.20.30.0/24|<---| enp0s31f6      |<---              --->| enx207bd28fb04e |--->|192.168.178.0/24|--- INTERNET
UPLINK      +--------------+    | GBit Link      |  |                |  | GBit Link       |    +----------------+      UPLINK
                                +----------------|  |                |  |-----------------+                                  
                                 vmbr0           |  +----------------+  |            vmbr1                                   
                                                 |                      |                                                    
                                                 +----------------------+

I have two independent Internet Uplinks and three physical network interfaces on the Host.
vmbr0 was set up by Proxmox (the 2.5GBit is as of yet unused).
The tso/gso was disabled since my console was flooded with erratic NETDEV WATCHDOG errors that this hint here in the forums helped me solve.

vmbr1 I tried to set up to create the 2nd uplink.
As indicated above, the two independent bridges on the host work, I can easily and without any delay get the external IP address of both uplinks:

Code: 
root@prxmx-01:/proc/sys/net/ipv4/conf/vmbr0# echo Primary $( curl -sX GET --interface vmbr0 ifconfig.me ) ; echo Backup $( curl -sX GET --interface vmbr1 ifconfig.me )
Primary 93.225.21.26
Backup 217.93.126.178
root@prxmx-01:/proc/sys/net/ipv4/conf/vmbr0#

The haproxy VM works fine on the vmbr0 bridge, but neither dhcp nor static IP setup on the NIC bound to the vmbr1 bridge works.

I now also created a plain valinna alpine VM with only one NIC connected to vmbr1 — that cannot connect to the LAN at all:
1721651295923.png

This is the VM config:

Code: 
root@prxmx-01:/etc/pve# cat nodes/prxmx-01/qemu-server/204.conf 
boot: order=scsi0;ide2;net0
cores: 1
cpu: x86-64-v2-AES
ide2: local-btrfs:iso/alpine-virt-3.20.1-x86_64.iso,media=cdrom,size=61M
memory: 512
meta: creation-qemu=9.0.0,ctime=1721641759
net0: virtio=02:02:00:F6:3A:28,bridge=vmbr1,firewall=1
numa: 0
ostype: l26
scsi0: local-btrfs:204/vm-204-disk-0.raw,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=f7214912-dfa4-46bd-8c27-dd1bef407a26
sockets: 1
vmgenid: 7b8c465a-134a-4063-90a5-4c19d74240cf
root@prxmx-01:/etc/pve#

Clearly I am missing something. I just do not know what ... 8^(
 
thefu21 said:
Could you please answer my quesitons
Click to expand...
What is the purpose of the second interface (enp2s0), and why have you disabled STP when using two interfaces?
Click to expand...
I left the default configuration as Proxmox had it configured.
I did not change anything there.
Enabling stp does not change anything. (see below)

Why are you configuring a second gateway, as shown below? This configuration doesn't seem to make much sense.
Click to expand...
Adding a secondary gateway with a larger metric should facilitate failover if the primary link is down.
And the setup will even not work on the host without it. (see below)

What is the reason for enabling IP forwarding?
Click to expand...
That was an attempt to make it wirk.
I have removed it since, but the situation is not improved / changed.


After a fresh reboot, this is what I get on the host:
Code: 
root@prxmx-01:~# !echo
echo Primary $( curl -sX GET --interface vmbr0 ifconfig.me ) ; echo Backup $( curl -sX GET --interface vmbr1 ifconfig.me )
Primary 93.225.21.26
Backup
root@prxmx-01:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enp0s31f6 inet manual

iface enx207bd28fb04e inet manual

auto vmbr0
iface vmbr0 inet static
        address 172.20.30.11/24
        gateway 172.20.30.1
        bridge-ports enp0s31f6 enp2s0
        bridge-stp on
        bridge-fd 0
        post-up /usr/sbin/ethtool -K enp0s31f6 tso off gso off

iface wlp0s20f3 inet manual

auto vmbr1
iface vmbr1 inet static
        address 192.168.178.11/24
        metric 200
        bridge-ports enx207bd28fb04e
        bridge-stp   on
        bridge-fd    10
        #post-up  echo 1 > /proc/sys/net/ipv4/ip_forward
        #post-up /usr/sbin/route add -net 0.0.0.0 gw 192.168.178.1 metric 200 vmbr1

source /etc/network/interfaces.d/*
Note: the second gateway is disabled, and the curl through the backup uplink fails.

If I change it and add it again I get an error message from the vmbr0 interface for some reason, but then the curl through the backup uplink works again. So I figure, the gateway does help and not hurt ... :
Code: 
root@prxmx-01:~# vi /etc/network/interfaces
root@prxmx-01:~# ifreload -a 
error: vmbr0: netlink: cannot create bridge or set attributes: operation failed with 'Numerical result out of range' (34)
root@prxmx-01:~# echo Primary $( curl -sX GET --interface vmbr0 ifconfig.me ) ; echo Backup $( curl -sX GET --interface vmbr1 ifconfig.me )
Primary 93.225.21.26
Backup 217.93.126.178
root@prxmx-01:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enp0s31f6 inet manual

iface enx207bd28fb04e inet manual

auto vmbr0
iface vmbr0 inet static
        address 172.20.30.11/24
        gateway 172.20.30.1
        bridge-ports enp0s31f6 enp2s0
        bridge-stp on
        bridge-fd 0
        post-up /usr/sbin/ethtool -K enp0s31f6 tso off gso off

iface wlp0s20f3 inet manual

auto vmbr1
iface vmbr1 inet static
        address 192.168.178.11/24
        metric 200
        bridge-ports enx207bd28fb04e
        bridge-stp   on
        bridge-fd    10
        #post-up  echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up /usr/sbin/route add -net 0.0.0.0 gw 192.168.178.1 metric 200 vmbr1

source /etc/network/interfaces.d/*
root@prxmx-01:~#

Still everything seems broken from within the VM.
 
thefu21 said:
You could try following this guide [1] and have a look at this thread [2]

[1] https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System
[2] https://forum.proxmox.com/threads/proxmox-ve-multiple-gateways.64377/
Click to expand...

Thanks — that did the trick.

Even though my Host already worked, this seems cleaner and I configured exactly as per the guide:
Code: 
root@prxmx-01:~# tail -5 /etc/iproute2/rt_tables
#
# https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System
#
1 backup

root@prxmx-01:~# tail -13 /etc/network/interfaces
auto vmbr1
iface vmbr1 inet static
        address 192.168.178.11/24
        metric 200
        bridge-ports enx207bd28fb04e
        bridge-stp   on
        bridge-fd    10
        post-up /usr/sbin/ip  route  add             192.168.178.0/24 dev vmbr1   src 192.168.178.11    table backup
        post-up /usr/sbin/ip  route  add default via 192.168.178.1    dev vmbr1                         table backup
        post-up /usr/sbin/ip  rule   add                                         from 192.168.178.11/32 table backup
        post-up /usr/sbin/ip  rule   add                                           to 192.168.178.11/32 table backup

source /etc/network/interfaces.d/*

root@prxmx-01:~#

For my alpine VM I had to fiddle a bit but also made it work. First I tried via staticroute, but that had problems with the ip rule ... statements, and then I had a stupid typo, so I ended up writing a small init.d script:
Code: 
prxmx-haproxy:~$ cat /etc/init.d/dualstack 
#!/sbin/openrc-run
#
# This script was inspired by the staticroute init.d script from alpine.

DEV=eth1
TBL=2

description="Configures Dualstack via ${DEV} in routing table ${TBL}"
depend()
{
        after clock
        provide net
        use network
        keyword -jail -prefix -vserver
}

start()
{

        {
                echo "Configuring ${DEV} ... "
                IP=""
                while [ ${IP:-X} = "X" ];
                do
                        sleep 1
                        IP=$( ip a show eth1 | grep ' inet ' | tr '/' ' ' | awk ' {print $2}' )
                        if [ ${IP:-X} = "X" ]; then echo "IP not set. waiting."; sleep 5; fi
                done
                NET=$( echo $IP | cut -d'.' -f1-3 )
                if [ ${NET:-X} = "X" ]; then echo "NET not found. aborting"; exit 1; fi

                ip route add             ${NET}.0/24 dev ${DEV}  src ${IP}    table ${TBL}
                ip route add default via ${NET}.1    dev ${DEV}               table ${TBL}
                ip rule  add                                    from ${IP}/32 table ${TBL}
                ip rule  add                                      to ${IP}/32 table ${TBL}

                printf "\nRules:\n"
                ip rule show

                printf "\nStandard Routes:\n"
                ip route show

                printf "\nRoutes Table ${TBL}:\n"
                ip route show table ${TBL}

                printf "\ndone."

        }  2>&1 | tee /tmp/dualstack.out
}

stop()
{
        echo "not implemented"
}

I figured I would be re-using this anyway on other VMs, so why not?

Now both, my Host and my VM are super-happy and have two independent routes to the internet:
1721743329071.png

Thanks again!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom