Routing problem with new Server

toddehb

New Member
Jul 5, 2022
24
0
1
Hi,

have set up a new server with proxmox. On vmbr0 is 192.168.1.0/24. This is a built-in NIC. I have 2 USB/Ethernet adapters connected. On vmbr1 is 192.168.178.0/24 and on vmbr2 192.168.2.0/24. All bridges are connected to a Ubiquiti switch with VLAN in the respective subnet to my Pfsense. Immediately after setup I was able to access all other two networks from the 192.168.178.0 network. Restarted the server and it doesn't work anymore. Funnily enough I can access the VMs via IP. So only the Proxmox server has a routing problem.

1687276367133.png

The gateway seems strange to me for vmbr1 and 2. It cannot be changed on the console. I deleted the route for testing and tried to create a new one

1687276401873.png

1687276514300.png

1687276420636.png



1687277942322.png



I am a little lost here. Would appriciate any help. Thanks
 

Attachments

  • 1687276665903.png
    1687276665903.png
    25.2 KB · Views: 3
  • 1687277216884.png
    1687277216884.png
    35.9 KB · Views: 2
  • 1687277709987.png
    1687277709987.png
    36.7 KB · Views: 2
Last edited:
The bridges have no gateway defined, so no way to route packets.
Additionally you say they are to be in VLAN networks. These packets will need to be made VLAN aware unless they come untagged. I do all my VLANing outside proxmox so I don't know if ticking "VLAN aware" is the way to do it but you need to be clear how the tagging and trunking will be done in your network.
 
The bridges have no gateway defined, so no way to route packets.
There can only be on default gateway. If you need more specific routes, then the command should look like this:
Code:
ip route add 192.168.178.0/24 via x.x.x.x dev eth0

But that does not make any sense in the context of this thread.

@toddehb what is the output of ip route?

For the networks on vmbr1 and vmbr2 they should look something like this:
Code:
default via 192.168.10.1 dev vmbr0 proto kernel onlink
10.10.200.0/24 dev enp130s0 proto kernel scope link src 10.10.200.4
10.10.230.0/24 dev enp129s0f0 proto kernel scope link src 10.10.230.4
192.168.10.0/24 dev vmbr0 proto kernel scope link src 192.168.10.4

The default via an IP, for the network configured on a NIC, the route is the device itself.

You could also post the contents of the /etc/network/interfaces

And please if you can, copy the output inside [CODE][/CODE] tags as screenshots are harder to read, search and copy text from
;)

Edit: changed the ip route add command as the 'via' address needs to be in a range that is reachable and cannot be an address in the range for which the route is added.
 
Last edited:
Ok, true on only one default gateway.
There can only be on default gateway. If you need more specific routes, then the command should look like this:
Code:
ip route add 192.168.178.0/24 via 192.168.178.1 dev eth0

But that does not make any sense in the context of this thread.

@toddehb what is the output of ip route?

For the networks on vmbr1 and vmbr2 they should look something like this:
Code:
default via 192.168.10.1 dev vmbr0 proto kernel onlink
10.10.200.0/24 dev enp130s0 proto kernel scope link src 10.10.200.4
10.10.230.0/24 dev enp129s0f0 proto kernel scope link src 10.10.230.4
192.168.10.0/24 dev vmbr0 proto kernel scope link src 192.168.10.4

The default via an IP, for the network configured on a NIC, the route is the device itself.

You could also post the contents of the /etc/network/interfaces

And please if you can, copy the output inside [CODE][/CODE] tags as screenshots are harder to read, search and copy text from ;)
 
This is my routes

Code:
root@prox1:~# ip route
default via 192.168.1.1 dev vmbr0 proto kernel onlink
192.168.1.0/24 dev vmbr0 proto kernel scope link src 192.168.1.222
192.168.2.0/24 dev vmbr2 proto kernel scope link src 192.168.2.3
192.168.178.0/24 dev vmbr1 proto kernel scope link src 192.168.178.3

The weird thing was, that yesterday it started working for no reason. Today no chance to connect. Don't see the issue. VLAN is untagged and the fact that the VMs inside Proxmox are reachable, at least the ones in the network of 192.168.1.0/24 and 192.168.2.0/24, lets my believe it might be a bug in Proxmox?
 
Last edited:
That looks good. I just noticed, that the NICs used for vmbr1 and 2 are not active in the screenshots.

Can you post the output of ip a?
 
That looks good. I just noticed, that the NICs used for vmbr1 and 2 are not active in the screenshots.

Can you post the output of ip a?
Code:
auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enxa0cec8e6ec62 inet manual

iface enx0023545c7026 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.222/24
        gateway 192.168.1.1
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.178.3/24
        bridge-ports enxa0cec8e6ec62
        bridge-stp off
        bridge-fd 0

auto vmbr2
iface vmbr2 inet static
        address 192.168.2.3/24
        bridge-ports enx0023545c7026
        bridge-stp off
        bridge-fd 0


Code:
root@prox1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 9c:6b:00:17:61:52 brd ff:ff:ff:ff:ff:ff
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:6b:00:17:61:52 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.222/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::9e6b:ff:fe17:6152/64 scope link
       valid_lft forever preferred_lft forever
6: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a0:ce:c8:e6:ec:62 brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.3/24 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::a2ce:c8ff:fee6:ec62/64 scope link
       valid_lft forever preferred_lft forever
7: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:23:54:5c:70:26 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.3/24 scope global vmbr2
       valid_lft forever preferred_lft forever
    inet6 fe80::223:54ff:fe5c:7026/64 scope link
       valid_lft forever preferred_lft forever
9: veth100i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:c0:6f:dd:c3:52 brd ff:ff:ff:ff:ff:ff link-netnsid 0
10: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether e6:ce:20:36:c4:d4 brd ff:ff:ff:ff:ff:ff
11: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr103i0 state UNKNOWN group default qlen 1000
    link/ether a2:25:6d:f0:a5:df brd ff:ff:ff:ff:ff:ff
12: fwbr103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 2e:12:be:c7:06:bb brd ff:ff:ff:ff:ff:ff
13: fwpr103p0@fwln103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether 9e:39:26:06:fd:a6 brd ff:ff:ff:ff:ff:ff
14: fwln103i0@fwpr103p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr103i0 state UP group default qlen 1000
    link/ether 86:1c:1c:c2:53:3f brd ff:ff:ff:ff:ff:ff
15: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 2e:c0:65:25:fa:49 brd ff:ff:ff:ff:ff:ff
16: veth105i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr105i0 state UP group default qlen 1000
    link/ether fe:3f:a3:67:db:62 brd ff:ff:ff:ff:ff:ff link-netnsid 1
17: fwbr105i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 72:2a:93:4e:d6:6f brd ff:ff:ff:ff:ff:ff
18: fwpr105p0@fwln105i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether 82:a6:ad:ea:fb:d6 brd ff:ff:ff:ff:ff:ff
19: fwln105i0@fwpr105p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr105i0 state UP group default qlen 1000
    link/ether 8a:08:5a:50:6f:2a brd ff:ff:ff:ff:ff:ff
20: tap110i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 7e:39:d4:9a:3b:f1 brd ff:ff:ff:ff:ff:ff
22: tap111i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr111i0 state UNKNOWN group default qlen 1000
    link/ether 4e:31:29:f9:93:e7 brd ff:ff:ff:ff:ff:ff
23: fwbr111i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:a0:fb:fb:18:e3 brd ff:ff:ff:ff:ff:ff
24: fwpr111p0@fwln111i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether ae:48:f9:5d:9c:c9 brd ff:ff:ff:ff:ff:ff
25: fwln111i0@fwpr111p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr111i0 state UP group default qlen 1000
    link/ether 86:b0:16:b3:98:e4 brd ff:ff:ff:ff:ff:ff
27: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a0:ce:c8:e6:ec:62 brd ff:ff:ff:ff:ff:ff
root@prox1:~#

Hab vmbr2 gerade mal abgeklemmt, da ich den Adater benötige für Laptop um ins vmbr0 Netz zu kommen.
 
Last edited:
The enxa0cec8e6ec62 and enx0023545c7026 NIC don't show up in the ip a output at all. Are they still connected and do they work? Checking the output of dmesg might shed some light as well.
 
1687354256604.png


Innerhalb einer VM Zugriff ins 192.168.178.0/24 kein Problem. Direkt vom ProxMOX aber schon

1687354330887.png
 
Can you show the config of one such VM? qm config {vmid}
Is it possible that they connect via vmbr0 and the traffic gets routed via the central firewall? tracert {ip address} should show another hop if the FW in included.
 
Can you show the config of one such VM? qm config {vmid}
Is it possible that they connect via vmbr0 and the traffic gets routed via the central firewall? tracert {ip address} should show another hop if the FW in included.
1687354554943.png

1687354633937.png
 
ah for a container it is pct config {vmid}.
And you can see that the traffics goes through 192.168.1.1 and then into the 192.168.178.x network.

Also, it seems like the USB NICs are a bit flaky. You could try to replug them and run ifreload -a on the node. Maybe they work then (for some time, depending on how flaky they are).
 
  • Like
Reactions: toddehb
ah for a container it is pct config {vmid}.
And you can see that the traffics goes through 192.168.1.1 and then into the 192.168.178.x network.

Also, it seems like the USB NICs are a bit flaky. You could try to replug them and run ifreload -a on the node. Maybe they work then (for some time, depending on how flaky they are).

Ich habe es jetzt erstmal anders gelöst und das vmbr01 gelöscht. War eh nur ein Client dran und den habe ich nun umkonfiguriert. Danke für dein Hilfe.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!