Routing problem with new Server

toddehb · Jun 20, 2023

Hi,

have set up a new server with proxmox. On vmbr0 is 192.168.1.0/24. This is a built-in NIC. I have 2 USB/Ethernet adapters connected. On vmbr1 is 192.168.178.0/24 and on vmbr2 192.168.2.0/24. All bridges are connected to a Ubiquiti switch with VLAN in the respective subnet to my Pfsense. Immediately after setup I was able to access all other two networks from the 192.168.178.0 network. Restarted the server and it doesn't work anymore. Funnily enough I can access the VMs via IP. So only the Proxmox server has a routing problem.

The gateway seems strange to me for vmbr1 and 2. It cannot be changed on the console. I deleted the route for testing and tried to create a new one

I am a little lost here. Would appriciate any help. Thanks

pengu1n · Jun 21, 2023

The bridges have no gateway defined, so no way to route packets.
Additionally you say they are to be in VLAN networks. These packets will need to be made VLAN aware unless they come untagged. I do all my VLANing outside proxmox so I don't know if ticking "VLAN aware" is the way to do it but you need to be clear how the tagging and trunking will be done in your network.

aaron · Jun 21, 2023

pengu1n said:
The bridges have no gateway defined, so no way to route packets.

There can only be on default gateway. If you need more specific routes, then the command should look like this:

Code:

ip route add 192.168.178.0/24 via x.x.x.x dev eth0

But that does not make any sense in the context of this thread.

@toddehb what is the output of ip route?

For the networks on vmbr1 and vmbr2 they should look something like this:

Code:

default via 192.168.10.1 dev vmbr0 proto kernel onlink
10.10.200.0/24 dev enp130s0 proto kernel scope link src 10.10.200.4
10.10.230.0/24 dev enp129s0f0 proto kernel scope link src 10.10.230.4
192.168.10.0/24 dev vmbr0 proto kernel scope link src 192.168.10.4

The default via an IP, for the network configured on a NIC, the route is the device itself.

You could also post the contents of the /etc/network/interfaces

And please if you can, copy the output inside [CODE][/CODE] tags as screenshots are harder to read, search and copy text from

Edit: changed the ip route add command as the 'via' address needs to be in a range that is reachable and cannot be an address in the range for which the route is added.

pengu1n · Jun 21, 2023

Ok, true on only one default gateway.

aaron said:
There can only be on default gateway. If you need more specific routes, then the command should look like this:

Code:

ip route add 192.168.178.0/24 via 192.168.178.1 dev eth0

But that does not make any sense in the context of this thread.

@toddehb what is the output of ip route?

For the networks on vmbr1 and vmbr2 they should look something like this:

Code:

default via 192.168.10.1 dev vmbr0 proto kernel onlink 10.10.200.0/24 dev enp130s0 proto kernel scope link src 10.10.200.4 10.10.230.0/24 dev enp129s0f0 proto kernel scope link src 10.10.230.4 192.168.10.0/24 dev vmbr0 proto kernel scope link src 192.168.10.4

The default via an IP, for the network configured on a NIC, the route is the device itself.

You could also post the contents of the /etc/network/interfaces

And please if you can, copy the output inside [CODE][/CODE] tags as screenshots are harder to read, search and copy text from

toddehb · Jun 21, 2023

This is my routes

Code:

root@prox1:~# ip route
default via 192.168.1.1 dev vmbr0 proto kernel onlink
192.168.1.0/24 dev vmbr0 proto kernel scope link src 192.168.1.222
192.168.2.0/24 dev vmbr2 proto kernel scope link src 192.168.2.3
192.168.178.0/24 dev vmbr1 proto kernel scope link src 192.168.178.3

The weird thing was, that yesterday it started working for no reason. Today no chance to connect. Don't see the issue. VLAN is untagged and the fact that the VMs inside Proxmox are reachable, at least the ones in the network of 192.168.1.0/24 and 192.168.2.0/24, lets my believe it might be a bug in Proxmox?

aaron · Jun 21, 2023

That looks good. I just noticed, that the NICs used for vmbr1 and 2 are not active in the screenshots.

Can you post the output of ip a?

toddehb · Jun 21, 2023

aaron said:
That looks good. I just noticed, that the NICs used for vmbr1 and 2 are not active in the screenshots.

Can you post the output of ip a?

Code:

auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enxa0cec8e6ec62 inet manual

iface enx0023545c7026 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.222/24
        gateway 192.168.1.1
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.178.3/24
        bridge-ports enxa0cec8e6ec62
        bridge-stp off
        bridge-fd 0

auto vmbr2
iface vmbr2 inet static
        address 192.168.2.3/24
        bridge-ports enx0023545c7026
        bridge-stp off
        bridge-fd 0

Code:

root@prox1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 9c:6b:00:17:61:52 brd ff:ff:ff:ff:ff:ff
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:6b:00:17:61:52 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.222/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::9e6b:ff:fe17:6152/64 scope link
       valid_lft forever preferred_lft forever
6: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a0:ce:c8:e6:ec:62 brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.3/24 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::a2ce:c8ff:fee6:ec62/64 scope link
       valid_lft forever preferred_lft forever
7: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:23:54:5c:70:26 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.3/24 scope global vmbr2
       valid_lft forever preferred_lft forever
    inet6 fe80::223:54ff:fe5c:7026/64 scope link
       valid_lft forever preferred_lft forever
9: veth100i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:c0:6f:dd:c3:52 brd ff:ff:ff:ff:ff:ff link-netnsid 0
10: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether e6:ce:20:36:c4:d4 brd ff:ff:ff:ff:ff:ff
11: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr103i0 state UNKNOWN group default qlen 1000
    link/ether a2:25:6d:f0:a5:df brd ff:ff:ff:ff:ff:ff
12: fwbr103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 2e:12:be:c7:06:bb brd ff:ff:ff:ff:ff:ff
13: fwpr103p0@fwln103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether 9e:39:26:06:fd:a6 brd ff:ff:ff:ff:ff:ff
14: fwln103i0@fwpr103p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr103i0 state UP group default qlen 1000
    link/ether 86:1c:1c:c2:53:3f brd ff:ff:ff:ff:ff:ff
15: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 2e:c0:65:25:fa:49 brd ff:ff:ff:ff:ff:ff
16: veth105i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr105i0 state UP group default qlen 1000
    link/ether fe:3f:a3:67:db:62 brd ff:ff:ff:ff:ff:ff link-netnsid 1
17: fwbr105i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 72:2a:93:4e:d6:6f brd ff:ff:ff:ff:ff:ff
18: fwpr105p0@fwln105i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether 82:a6:ad:ea:fb:d6 brd ff:ff:ff:ff:ff:ff
19: fwln105i0@fwpr105p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr105i0 state UP group default qlen 1000
    link/ether 8a:08:5a:50:6f:2a brd ff:ff:ff:ff:ff:ff
20: tap110i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 7e:39:d4:9a:3b:f1 brd ff:ff:ff:ff:ff:ff
22: tap111i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr111i0 state UNKNOWN group default qlen 1000
    link/ether 4e:31:29:f9:93:e7 brd ff:ff:ff:ff:ff:ff
23: fwbr111i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:a0:fb:fb:18:e3 brd ff:ff:ff:ff:ff:ff
24: fwpr111p0@fwln111i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether ae:48:f9:5d:9c:c9 brd ff:ff:ff:ff:ff:ff
25: fwln111i0@fwpr111p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr111i0 state UP group default qlen 1000
    link/ether 86:b0:16:b3:98:e4 brd ff:ff:ff:ff:ff:ff
27: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a0:ce:c8:e6:ec:62 brd ff:ff:ff:ff:ff:ff
root@prox1:~#

Hab vmbr2 gerade mal abgeklemmt, da ich den Adater benötige für Laptop um ins vmbr0 Netz zu kommen.

aaron · Jun 21, 2023

The enxa0cec8e6ec62 and enx0023545c7026 NIC don't show up in the ip a output at all. Are they still connected and do they work? Checking the output of dmesg might shed some light as well.

toddehb · Jun 21, 2023

Innerhalb einer VM Zugriff ins 192.168.178.0/24 kein Problem. Direkt vom ProxMOX aber schon

aaron · Jun 21, 2023

Can you show the config of one such VM? qm config {vmid}
Is it possible that they connect via vmbr0 and the traffic gets routed via the central firewall? tracert {ip address} should show another hop if the FW in included.

toddehb · Jun 21, 2023

aaron said:
The enxa0cec8e6ec62 and enx0023545c7026 NIC don't show up in the ip a output at all. Are they still connected and do they work? Checking the output of dmesg might shed some light as well.

toddehb · Jun 21, 2023

aaron said:
Can you show the config of one such VM? qm config {vmid}
Is it possible that they connect via vmbr0 and the traffic gets routed via the central firewall? tracert {ip address} should show another hop if the FW in included.

aaron · Jun 21, 2023

ah for a container it is pct config {vmid}.
And you can see that the traffics goes through 192.168.1.1 and then into the 192.168.178.x network.

Also, it seems like the USB NICs are a bit flaky. You could try to replug them and run ifreload -a on the node. Maybe they work then (for some time, depending on how flaky they are).

toddehb · Jun 23, 2023

aaron said:
ah for a container it is pct config {vmid}.
And you can see that the traffics goes through 192.168.1.1 and then into the 192.168.178.x network.

Also, it seems like the USB NICs are a bit flaky. You could try to replug them and run ifreload -a on the node. Maybe they work then (for some time, depending on how flaky they are).

Ich habe es jetzt erstmal anders gelöst und das vmbr01 gelöscht. War eh nur ein Client dran und den habe ich nun umkonfiguriert. Danke für dein Hilfe.

Search

Search

Routing problem with new Server

toddehb

Member

Attachments

pengu1n

Member

aaron

Proxmox Staff Member

pengu1n

Member

toddehb

Member

aaron

Proxmox Staff Member

toddehb

Member

aaron

Proxmox Staff Member

toddehb

Member

aaron

Proxmox Staff Member

toddehb

Member

toddehb

Member

aaron

Proxmox Staff Member

toddehb

Member