[SOLVED] vlan on single nic doesn't work?!?

austin17427

Member
Jun 4, 2020
16
0
6
53
I'm trying to setup `vlan 33` with a single nic on my proxmox server. I have the below configuration on my `/etc/network/interfaces` file.
auto lo iface lo inet loopback iface enp2s0 inet manual auto vmbr0.33 iface vmbr0.33 inet static address [redacted]/24 gateway [redacted] auto vmbr0 iface vmbr0 inet static bridge-ports enp2s0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094

Once the above configuration applied, machine can not access anything.

$ ping -c 2 [redacted] PING [redacted] (10.24.0.1) 56(84) bytes of data. From [redacted] icmp_seq=1 Destination Host Unreachable From [redacted] icmp_seq=2 Destination Host Unreachable

IP address looks normal.
$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000 link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff 7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff inet6 fe80::8e89:a5ff:fe5d:9f5a/64 scope link valid_lft forever preferred_lft forever 8: vmbr0.33@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff inet [redacted]/24 scope global vmbr0.33 valid_lft forever preferred_lft forever inet6 fe80::8e89:a5ff:fe5d:9f5a/64 scope link valid_lft forever preferred_lft forever

I tried every permutation on the switch TL-SG108E like below, this machine is connected to the port #3 of this switch. All other vlans (34,35,36) are functional.

VLAN IDVLAN NameMember PortsTagged PortsUntagged PortsDelete
1Default1-41-4
33Dev1-31-3
34IoT1-21-2
35Guest1-21-2
36TVs1,5-815-8

Switch port #1 goes to my main router port #4. My main router vlan table is as below.



VLAN


VID

WAN

Tag
WAN

1

Tag
1

2

Tag
2

3

Tag
3

4

Tag
4

Default
VLAN
Ethernet to
bridge
mapping
11⭐⭐⭐⭐DefaultLAN0 (br0)
22⭐WAN0 bridge
334⭐TaggedLAN1 (br1)
435⭐TaggedLAN2 (br2)
536⭐TaggedLAN3 (br3)
633⭐Tagged


Port #2 goes to my slave router Port #1. Slave router table is like below.


VLAN


VID

WAN

Tag
WAN

1

Tag
1

2

Tag
2

3

Tag
3

4

Tag
4

Default
VLAN
Ethernet to
bridge
mapping
11⭐⭐⭐⭐⭐DefaultLAN0 (br0)
22WAN0 bridge
334⭐TaggedLAN1 (br1)
435⭐TaggedLAN2 (br2)
533⭐Tagged⭐Tagged

Please let me know if you see anything wrong with above config. I really like to make use of Proxmox vlans.
Thank you in advance!!


Edits:
- Looks like tag icons are not retained in the tables above, replaced them with text
 
Last edited:
I'm trying to setup `vlan 33` with a single nic on my proxmox server. I have the below configuration on my `/etc/network/interfaces` file.
auto lo iface lo inet loopback iface enp2s0 inet manual auto vmbr0.33 iface vmbr0.33 inet static address [redacted]/24 gateway [redacted] auto vmbr0 iface vmbr0 inet static bridge-ports enp2s0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094

Once the above configuration applied, machine can not access anything.

$ ping -c 2 [redacted] PING [redacted] (10.24.0.1) 56(84) bytes of data. From [redacted] icmp_seq=1 Destination Host Unreachable From [redacted] icmp_seq=2 Destination Host Unreachable

IP address looks normal.
$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000 link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff 7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff inet6 fe80::8e89:a5ff:fe5d:9f5a/64 scope link valid_lft forever preferred_lft forever 8: vmbr0.33@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff inet [redacted]/24 scope global vmbr0.33 valid_lft forever preferred_lft forever inet6 fe80::8e89:a5ff:fe5d:9f5a/64 scope link valid_lft forever preferred_lft forever

I tried every permutation on the switch TL-SG108E like below, this machine is connected to the port #3 of this switch. All other vlans (34,35,36) are functional.

VLAN IDVLAN NameMember PortsTagged PortsUntagged PortsDelete
1Default1-41-4
33Dev1-31-3
34IoT1-21-2
35Guest1-21-2
36TVs1,5-815-8

Switch port #1 goes to my main router port #4. My main router vlan table is as below.



VLAN


VID

WAN

Tag
WAN

1

Tag
1

2

Tag
2

3

Tag
3

4

Tag
4

Default
VLAN
Ethernet to
bridge
mapping
11⭐⭐⭐⭐DefaultLAN0 (br0)
22⭐WAN0 bridge
334⭐TaggedLAN1 (br1)
435⭐TaggedLAN2 (br2)
536⭐TaggedLAN3 (br3)
633⭐Tagged


Port #2 goes to my slave router Port #1. Slave router table is like below.


VLAN


VID

WAN

Tag
WAN

1

Tag
1

2

Tag
2

3

Tag
3

4

Tag
4

Default
VLAN
Ethernet to
bridge
mapping
11⭐⭐⭐⭐⭐DefaultLAN0 (br0)
22WAN0 bridge
334⭐TaggedLAN1 (br1)
435⭐TaggedLAN2 (br2)
533⭐Tagged⭐Tagged

Please let me know if you see anything wrong with above config. I really like to make use of Proxmox vlans.
Thank you in advance!!


Edits:
- Looks like tag icons are not retained in the tables above, replaced them with text

If WAN port on slave router is not a member of the regular LAN then you have double NAT active. It is not wrong, but double NAT gives more headache when you want to open ports etc.
 
WAN port on the slave router is part of regular LAN, see below;
Code:
# robocfg show
Switch: enabled gigabit
Port 0: 1000FD enabled stp: none vlan: 1 jumbo: off mac: [redacted]
Port 1:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 2:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 3:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 4:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 5:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 8: 1000FD enabled stp: none vlan: 1 jumbo: off mac: [redacted]
VLANs: BCM53115 enabled mac_check mac_hash
   1: vlan1: 0 1 2 3 4 8t
   2: vlan2: 8t
  32: vlan32: 0t 4t 8t
  33: vlan33: 0t 4t 8t
  34: vlan34: 0t 8t
  35: vlan35: 0t 8t

In fact, I can now access the vlan 33 from a workstation but the Proxmox can not ping gateway or access internet.
Note: I'm learning about networking, I don't have previous experience when it comes to setting up networking in Linux.
 
WAN port on the slave router is part of regular LAN, see below;
Code:
# robocfg show
Switch: enabled gigabit
Port 0: 1000FD enabled stp: none vlan: 1 jumbo: off mac: [redacted]
Port 1:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 2:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 3:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 4:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 5:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 8: 1000FD enabled stp: none vlan: 1 jumbo: off mac: [redacted]
VLANs: BCM53115 enabled mac_check mac_hash
   1: vlan1: 0 1 2 3 4 8t
   2: vlan2: 8t
  32: vlan32: 0t 4t 8t
  33: vlan33: 0t 4t 8t
  34: vlan34: 0t 8t
  35: vlan35: 0t 8t

In fact, I can now access the vlan 33 from a workstation but the Proxmox can not ping gateway or access internet.
Note: I'm learning about networking, I don't have previous experience when it comes to setting up networking in Linux.

From Proxmox host what is the output of:
# ip route
# dig debian.org
 
vmbr0 has no Gateway.
Put the Gateway on vmbr0 and not on a vlan.
@Spoonman2002 Do you mean the `/etc/network/interfaces` file on Proxmox server? But every example I looked at, has the Gateway on vlan and NOT on `vmbr0` even the Proxmox documentation. See documentation at https://pve.proxmox.com/wiki/Network_Configuration > VLAN 802.1Q > Vlan on the Host section and the snippet starting with the sentence "Example: Use VLAN 5 for the Proxmox VE management IP with VLAN aware Linux bridge". I have a single NIC on Proxmox and I got this configuration working! I can access the vlan33 on proxmox from a workstation when I have below NW config.

Code:
ip link add link ens9 name ens9.33 type vlan id 33
ip addr add x.x.x.x/24 dev ens9.33
ifconfig ens9.33 up

and then I can ping the Proxmox

Code:
ping -c 2 [redacted]
.....
2 packets transmitted, 2 received, 0% packet loss, time 1009ms

But the Proxmox can't ping the gateway or access Internet. Again, I only have a single NIC, and this should be possible with a single NIC...
 
From Proxmox host what is the output of:
# ip route
# dig debian.org
This is the output
# ip r default via x.x.x.1 dev vmbr0.33 proto kernel onlink x.x.x.0/24 dev vmbr0.33 proto kernel scope link src [redacted]

# dig debian.org ;; communications error to [my-dns-server]#53: timed out

Same thing happens when I use 1.1.1.1 as dns. Proxmox can't access outside.
 
This is the output
# ip r default via x.x.x.1 dev vmbr0.33 proto kernel onlink x.x.x.0/24 dev vmbr0.33 proto kernel scope link src [redacted]

# dig debian.org ;; communications error to [my-dns-server]#53: timed out

Same thing happens when I use 1.1.1.1 as dns. Proxmox can't access outside.
Has VLAN33 got a DNS server configured and a way out to the internet (gateway) (in your main router perhaps)?
I think not, that's why Proxmox cannot resolve debian.org through VLAN33.

Why don't you set the gateway on vmbr0, to make things a bit easier for you to configure?
 
And why do you delete the ip addresses in your posts and replace them with x.x.x.x ?
If it is your local internal network......nobody is going to hack you.
 
Has VLAN33 got a DNS server configured and a way out to the internet (gateway) (in your main router perhaps)?
I think not, that's why Proxmox cannot resolve debian.org through VLAN33.

Why don't you set the gateway on vmbr0, to make things a bit easier for you to configure?
Thank you for your answers. Because I followed this tutorial https://youtu.be/ljq6wlzn4qo and same config is in Proxmox documentation, that's why.
Both puts the address and gateway under vlan. I'm not trying to be stubborn. My objective is to be able use vlans on Proxmox and create separate vlans for Proxmox Management (UI+API) and separate vlans for VMs. I'm trying to get the first vlan (management) working before I add a second vlan for VMs. Have a working config and better understanding of Proxmox vlans and then copy solution to my existing Proxmox VEs.

If you know a better tutorial or if you have a config for accomplishing the above with a single nic, I'll be very curious to look at that.
 
And why do you delete the ip addresses in your posts and replace them with x.x.x.x ?
If it is your local internal network......nobody is going to hack you.
Lol, I'm a noob in networking, that's why... Not very comfortable with my knowledge, so trying not to expose much.
 
FWIW, I have my machine set up the exact same way as you, and my /etc/network/interfaces file is identical to yours and working correctly. The problem is either in your managed switch configuration (trunked ports/proper tagging) or perhaps you have misconfigured the VM? Is the machine your trying to ping from a VM on your node? And if yes, did you tag the VM with a VLAN when you created it?


Screenshot 2023-07-12 214346.jpg
 
@Spoonman2002 I fixed added tagged on the switch for port#3 first (initially it wasn't like that, I edited the question here too) and then fixed the workstation's network settings that was trying to access Proxmox. I understand the whole thread doesn't give much insight and the solution isn't very satisfactory. I'm sorry about that. But the access problem to Proxmox is resolved.

The bigger picture which is not mentioned in the problem statement was, to have a Kubernetes cluster (on Proxmox) that has nodes in the same VLAN. However, same cluster provides network wide services like DNS, Nginx Proxy, and some other services. So as far as that problem goes, I still don't know how to solve it yet. At the same time, I'm not sure if that's a relevant topic for Proxmox forum.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!