how to 2nd bridge interface

[almiller]

Hi (obviously) I'm a proxmox noob and .....

I have an old Dell running pve-manager/7.2-7 with 4 physical ethernet ports eno1-eno4

1st port is connected to switch and has address 172.16.1.101
So far all my VMs are deployed in this subnet (172.16.1.0/24) and everything works fine.

I want to create a 2nd subnet, 172.17.1.0/24 so I connected the 2nd port to a bgp-router using a cross-over cable.

The router is 172.17.1.5 and the Dell is 172.17.1.100
From the router I can ping the Dell and from the Dell I can ping the router.

On the ProxMox server:

root@pve ~ # netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.16.1.1      0.0.0.0         UG        0 0          0 vmbr0
172.16.1.0      0.0.0.0         255.255.255.0   U         0 0          0 vmbr0
172.17.1.0      0.0.0.0         255.255.255.0   U         0 0          0 vmbr1

root@pve ~ # cat  /etc/network/interfaces
auto lo
iface lo inet loopback
iface eno1 inet manual
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual
auto vmbr0
iface vmbr0 inet static
    address 172.16.1.101/24
    gateway 172.16.1.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
auto vmbr1
iface vmbr1 inet static
    address 172.17.1.100/24
    gateway 172.17.1.5
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0

From a VM at 172.17.1.10 I can't ping the proxmox server (or the router IP):

root@k8s-ctlr01:~# ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group d0
    link/ether ee:74:01:7c:61:a1 brd ff:ff:ff:ff:ff:ff
    inet 172.17.1.10/24 brd 172.17.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fde7:9611:2b18:d2b8:ec74:1ff:fe7c:61a1/64 scope global dynamic mngtmpaddr n
       valid_lft 1539sec preferred_lft 1539sec
    inet6 fe80::ec74:1ff:fe7c:61a1/64 scope link
       valid_lft forever preferred_lft forever
root@k8s-ctlr01:~# ip route show
default via 172.17.1.100 dev eth0 proto static
172.17.1.0/24 dev eth0 proto kernel scope link src 172.17.1.10

root@k8s-ctlr01:~# ping 172.17.1.100
PING 172.17.1.100 (172.17.1.100) 56(84) bytes of data.
^C
--- 172.17.1.100 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5115ms

root@k8s-ctlr01:~# networkctl status
●          State: routable                                    
    Online state: online                                      
         Address: 172.17.1.10 on eth0
                  fde7:9611:2b18:d2b8:ec74:1ff:fe7c:61a1 on eth0
                  fe80::ec74:1ff:fe7c:61a1 on eth0
         Gateway: 172.17.1.100 on eth0
             DNS: 172.16.1.222
                  1.1.1.1
  Search Domains: home.lab
                  home.net
[CODE]

If I run tcpdump on the proxmox server I can see the arp requests on the vmbr1 interface.
[CODE]
root@pve ~ # tcpdump -i vmbr1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vmbr1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:17:20.042748 ARP, Request who-has 172.17.1.10 tell 172.17.1.100, length 28
18:17:22.087274 ARP, Request who-has 172.17.1.10 tell 172.17.1.100, length 28
18:17:23.118677 ARP, Request who-has 172.17.1.10 tell 172.17.1.100, length 28
18:17:24.138672 ARP, Request who-has 172.17.1.10 tell 172.17.1.100, length 28
^C
4 packets captured

 

[bobmc]

I think you can only have one gateway, if you want to send external traffic on the second subnet/nic you will need to define some rules using IP tables to make this work.

 

[almiller]

Thanks @bobmc but actually it's working now. All I needed to do was set the correct bridge interface.
I changed it from vmbr0 to vmbr1 and things started working.

root@pve ~ # grep ^net0 /etc/pve/nodes/pve/qemu-server/401.conf
net0: virtio=1A:68:AE:F6:D7:E0,bridge=vmbr1,firewall=1

I created the VM from a template I created from ubuntu cloud init image.
So for the additional settings I just had to update /etc/netplan/50-cloud-init.yaml

network:
    version: 2
    ethernets:
        eth0:
            addresses:
            - 172.17.1.11/24
            gateway4: 172.17.1.5
            match:
                macaddress: 1a:68:ae:f6:d7:e0
            nameservers:
                addresses:
                - 172.16.1.222
                - 1.1.1.1
                search:
                - home.lab
                - home.net
            set-name: eth0

Now everything works. The BGP router at 172.17.1.5 is my default route and I can reach hosts in both subnets.

root@k8s-ctlr02:~# ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group d0
    link/ether 1a:68:ae:f6:d7:e0 brd ff:ff:ff:ff:ff:ff
    inet 172.17.1.11/24 brd 172.17.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1868:aeff:fef6:d7e0/64 scope link
       valid_lft forever preferred_lft forever


root@k8s-ctlr02:~# ip route show
default via 172.17.1.5 dev eth0 proto static
172.17.1.0/24 dev eth0 proto kernel scope link src 172.17.1.11


root@k8s-ctlr02:~# resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub


Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 172.16.1.222
       DNS Servers: 172.16.1.222 1.1.1.1
        DNS Domain: home.lab home.net

root@k8s-ctlr02:~# ssh pi@172.16.1.253
pi@172.16.1.253's password:
Linux pihole 4.14.77-v7+ #1154 SMP Fri Oct 19 16:01:02 BST 2018 armv7l


Last login: Tue Jan 31 10:34:41 2023 from 172.16.1.5
pi@pihole:~$