Hi there,
i got a really strange problem. I got my Proxmox VE running and recently i started to use Linux Bonding and VLANs.
My /etc/network/interfaces looks like this
I discovered the problem as i wanted to remove the proxmox ip from vmbr3. I thought i can use 172.21.0.11 on vmbr300 (bond0 -> bond0.300 (VLAN ID 300) -> vmbr300) but as soon as i remove the ip on vmbr3 i could not access the webgui.
I started to dig deeper into this issue and found out that everything thats going to 172.21.0.11 gets answered from 172.on vmbr3.
I cant really explain why this is happening. If i shutdown vmbr3 (e.g. remove the ip or the whole interface) i cant connect to the gui or via ssh nor even pinging. If i add the ip again everything works except for ssh to 172.21.0.11 (172.16.0.11 works). I can see via tcpdump that there is no answer going out if i remove the ip... can some1 explain to me whats going on? I really cant seem to get it
My Setup looks like this:
I got a linux vm gateway which connects to vlans 100 (172.16.0.0/22), 300 (172.21.0.0/24) and a couple more which are not relevant. The firewall rules allow access from vlan100 to 300 and vice versa. My client sits in vlan 100 and the proxmox webgui interface in vlan 300. The physical Proxmox VE Server has 8 NICs where 4 of them bonds to a switch with tagged vlan 100 and 300. The client sits at the switch on an untagged vlan 100 with PVID 100. vmbr3 which is the interface the traffic gets replied on is also connected to the switch but not to the gateway vm with untagged vlan100 and PVID 100. vmbr3 is just an interface thats being used by the proxmox hypervisor
i got a really strange problem. I got my Proxmox VE running and recently i started to use Linux Bonding and VLANs.
My /etc/network/interfaces looks like this
Code:
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet manual
auto eno2
iface eno2 inet manual
auto eno3
iface eno3 inet manual
auto eno4
iface eno4 inet manual
auto enp4s0f0
iface enp4s0f0 inet manual
auto enp4s0f1
iface enp4s0f1 inet manual
auto enp4s0f2
iface enp4s0f2 inet manual
auto enp4s0f3
iface enp4s0f3 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp4s0f0 enp4s0f1 enp4s0f2 enp4s0f3
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
auto bond0.100
iface bond0.100 inet manual
#Home-DMZ
auto bond0.110
iface bond0.110 inet manual
#Guest-DMZ
auto bond0.200
iface bond0.200 inet manual
#IoT-DMZ
auto bond0.300
iface bond0.300 inet manual
#CritServer-DMZ
auto bond0.310
iface bond0.310 inet manual
#MediaServer-DMZ
auto vmbr100
iface vmbr100 inet manual
bridge-ports bond0.100
bridge-stp off
bridge-fd 0
#Home-DMZ Bridge
auto vmbr110
iface vmbr110 inet manual
bridge-ports bond0.110
bridge-stp off
bridge-fd 0
#Guest-DMZ Bridge
auto vmbr200
iface vmbr200 inet manual
bridge-ports bond0.200
bridge-stp off
bridge-fd 0
#IoT-DMZ Bridge
auto vmbr300
iface vmbr300 inet static
address 172.21.0.11/24
bridge-ports bond0.300
bridge-stp off
bridge-fd 0
#CritServer-DMZ Bridge
auto vmbr310
iface vmbr310 inet manual
bridge-ports bond0.310
bridge-stp off
bridge-fd 0
#MediaServer-DMZ
auto vmbr1
iface vmbr1 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0
#Ext1 Bridge
auto vmbr2
iface vmbr2 inet manual
bridge-ports eno2
bridge-stp off
bridge-fd 0
#Ext2 Bridge
auto vmbr3
iface vmbr3 inet manual
address 172.16.0.11/22
bridge-ports eno3
bridge-stp off
bridge-fd 0I discovered the problem as i wanted to remove the proxmox ip from vmbr3. I thought i can use 172.21.0.11 on vmbr300 (bond0 -> bond0.300 (VLAN ID 300) -> vmbr300) but as soon as i remove the ip on vmbr3 i could not access the webgui.
I started to dig deeper into this issue and found out that everything thats going to 172.21.0.11 gets answered from 172.on vmbr3.
Code:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr100, link-type EN10MB (Ethernet), capture size 262144 bytes
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr3, link-type EN10MB (Ethernet), capture size 262144 bytes
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr300, link-type EN10MB (Ethernet), capture size 262144 bytes
[Interface:vmbr100] 10:26:23.068444 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 572, length 40
[Interface:vmbr3] 10:26:23.068599 IP 172.21.0.11 > 172.16.2.175: ICMP echo reply, id 1, seq 572, length 40
[Interface:vmbr300] 10:26:23.068581 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 572, length 40
[Interface:vmbr100] 10:26:24.071885 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 573, length 40
[Interface:vmbr3] 10:26:24.072037 IP 172.21.0.11 > 172.16.2.175: ICMP echo reply, id 1, seq 573, length 40
[Interface:vmbr300] 10:26:24.072018 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 573, length 40
[Interface:vmbr3] 10:26:25.077667 IP 172.21.0.11 > 172.16.2.175: ICMP echo reply, id 1, seq 574, length 40
[Interface:vmbr100] 10:26:25.077400 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 574, length 40
[Interface:vmbr300] 10:26:25.077647 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 574, length 40
[Interface:vmbr300] 10:26:30.125040 ARP, Request who-has 172.21.0.11 tell 172.21.0.254, length 46
[Interface:vmbr300] 10:26:30.125048 ARP, Reply 172.21.0.11 is-at a0:36:9f:03:bd:3c, length 28
[Interface:vmbr300] 10:27:56.935470 IP 172.21.0.11.32916 > 172.21.0.1.123: NTPv4, Client, length 48
[Interface:vmbr300] 10:27:56.935693 IP 172.21.0.1.123 > 172.21.0.11.32916: NTPv4, Server, length 48
[Interface:vmbr300] 10:28:02.061080 ARP, Request who-has 172.21.0.1 tell 172.21.0.11, length 28
[Interface:vmbr300] 10:28:02.061327 ARP, Reply 172.21.0.1 is-at d6:41:aa:3a:30:65, length 46
[Interface:vmbr300] 10:28:02.119202 ARP, Request who-has 172.21.0.11 tell 172.21.0.1, length 46
[Interface:vmbr300] 10:28:02.119208 ARP, Reply 172.21.0.11 is-at a0:36:9f:03:bd:3c, length 28
[Interface:vmbr3] 10:29:02.133654 IP 172.21.0.11 > 172.16.2.175: ICMP echo reply, id 1, seq 575, length 40
[Interface:vmbr100] 10:29:02.133529 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 575, length 40
[Interface:vmbr300] 10:29:02.133635 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 575, length 40
[Interface:vmbr3] 10:29:03.138221 IP 172.21.0.11 > 172.16.2.175: ICMP echo reply, id 1, seq 576, length 40
[Interface:vmbr100] 10:29:03.138023 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 576, length 40
[Interface:vmbr300] 10:29:03.138203 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 576, length 40
[Interface:vmbr100] 10:29:04.142752 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 577, length 40
[Interface:vmbr3] 10:29:04.142872 IP 172.21.0.11 > 172.16.2.175: ICMP echo reply, id 1, seq 577, length 40
[Interface:vmbr300] 10:29:04.142854 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 577, length 40
[Interface:vmbr100] 10:29:05.148609 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 578, length 40
[Interface:vmbr3] 10:29:05.148793 IP 172.21.0.11 > 172.16.2.175: ICMP echo reply, id 1, seq 578, length 40
[Interface:vmbr300] 10:29:05.148775 IP 172.16.2.175 > 172.21.0.11: ICMP echo request, id 1, seq 578, length 40
[Interface:vmbr300] 10:29:07.137329 ARP, Request who-has 172.21.0.11 tell 172.21.0.254, length 46
[Interface:vmbr300] 10:29:07.137336 ARP, Reply 172.21.0.11 is-at a0:36:9f:03:bd:3c, length 28
[Interface:vmbr100] 10:35:27.406781 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [S], seq 617195008, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
[Interface:vmbr100] 10:35:27.408188 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [.], ack 537259376, win 513, length 0
[Interface:vmbr100] 10:35:27.413380 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [P.], seq 0:28, ack 1, win 513, length 28
[Interface:vmbr100] 10:35:27.714185 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [P.], seq 0:28, ack 1, win 513, length 28
[Interface:vmbr100] 10:35:28.314638 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [P.], seq 0:28, ack 1, win 513, length 28
[Interface:vmbr3] 10:35:27.407184 IP 172.21.0.11.22 > 172.16.2.175.60421: Flags [S.], seq 537259375, ack 617195009, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
[Interface:vmbr300] 10:35:27.407159 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [S], seq 617195008, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
[Interface:vmbr3] 10:35:28.429081 IP 172.21.0.11.22 > 172.16.2.175.60421: Flags [S.], seq 537259375, ack 617195009, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
[Interface:vmbr100] 10:35:28.430406 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [.], ack 1, win 513, options [nop,nop,sack 1 {0:1}], length 0
[Interface:vmbr100] 10:35:29.515752 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [P.], seq 0:28, ack 1, win 513, length 28
[Interface:vmbr100] 10:35:30.446191 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [.], ack 1, win 513, options [nop,nop,sack 1 {0:1}], length 0
[Interface:vmbr3] 10:35:30.445070 IP 172.21.0.11.22 > 172.16.2.175.60421: Flags [S.], seq 537259375, ack 617195009, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
[Interface:vmbr100] 10:35:31.916155 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [P.], seq 0:28, ack 1, win 513, length 28
[Interface:vmbr300] 10:35:32.410992 ARP, Request who-has 172.21.0.11 tell 172.21.0.254, length 46
[Interface:vmbr300] 10:35:32.410999 ARP, Reply 172.21.0.11 is-at a0:36:9f:03:bd:3c, length 28
[Interface:vmbr3] 10:35:34.669084 IP 172.21.0.11.22 > 172.16.2.175.60421: Flags [S.], seq 537259375, ack 617195009, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
[Interface:vmbr100] 10:35:34.670226 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [.], ack 1, win 513, options [nop,nop,sack 1 {0:1}], length 0
[Interface:vmbr100] 10:35:36.716453 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [P.], seq 0:28, ack 1, win 513, length 28
[Interface:vmbr3] 10:35:42.861079 IP 172.21.0.11.22 > 172.16.2.175.60421: Flags [S.], seq 537259375, ack 617195009, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
[Interface:vmbr100] 10:35:42.863078 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [.], ack 1, win 513, options [nop,nop,sack 1 {0:1}], length 0
[Interface:vmbr100] 10:35:46.315912 IP 172.16.2.175.60421 > 172.21.0.11.22: Flags [R.], seq 28, ack 1, win 0, length 0
[Interface:vmbr3] 10:35:58.989075 IP 172.21.0.11.22 > 172.16.2.175.60421: Flags [S.], seq 537259375, ack 617195009, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0My Setup looks like this:
I got a linux vm gateway which connects to vlans 100 (172.16.0.0/22), 300 (172.21.0.0/24) and a couple more which are not relevant. The firewall rules allow access from vlan100 to 300 and vice versa. My client sits in vlan 100 and the proxmox webgui interface in vlan 300. The physical Proxmox VE Server has 8 NICs where 4 of them bonds to a switch with tagged vlan 100 and 300. The client sits at the switch on an untagged vlan 100 with PVID 100. vmbr3 which is the interface the traffic gets replied on is also connected to the switch but not to the gateway vm with untagged vlan100 and PVID 100. vmbr3 is just an interface thats being used by the proxmox hypervisor
Last edited:
you have to set a routing point, else there is no traffic.