[SOLVED] vlan on single nic doesn't work?!?

austin17427 · Jul 12, 2023

I'm trying to setup `vlan 33` with a single nic on my proxmox server. I have the below configuration on my `/etc/network/interfaces` file.


auto lo
iface lo inet loopback

iface enp2s0 inet manual

auto vmbr0.33
iface vmbr0.33 inet static
    address [redacted]/24
    gateway [redacted]

auto vmbr0
iface vmbr0 inet static
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

Once the above configuration applied, machine can not access anything.


$ ping -c 2 [redacted]
PING [redacted] (10.24.0.1) 56(84) bytes of data.
From [redacted] icmp_seq=1 Destination Host Unreachable
From [redacted] icmp_seq=2 Destination Host Unreachable

IP address looks normal.


$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8e89:a5ff:fe5d:9f5a/64 scope link
       valid_lft forever preferred_lft forever
8: vmbr0.33@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff
    inet [redacted]/24 scope global vmbr0.33
       valid_lft forever preferred_lft forever
    inet6 fe80::8e89:a5ff:fe5d:9f5a/64 scope link
       valid_lft forever preferred_lft forever

I tried every permutation on the switch TL-SG108E like below, this machine is connected to the port #3 of this switch. All other vlans (34,35,36) are functional.

VLAN ID	VLAN Name	Member Ports	Tagged Ports	Untagged Ports	Delete
1	Default	1-4		1-4
33	Dev	1-3	1-3
34	IoT	1-2	1-2
35	Guest	1-2	1-2
36	TVs	1,5-8	1	5-8

Switch port #1 goes to my main router port #4. My main router vlan table is as below.

VLAN	VID	WAN	Tag WAN	1	Tag 1	2	Tag 2	3	Tag 3	4	Tag 4	Default VLAN	Ethernet to bridge mapping
1	1											Default	LAN0 (br0)
2	2												WAN0 bridge
3	34										Tagged		LAN1 (br1)
4	35										Tagged		LAN2 (br2)
5	36										Tagged		LAN3 (br3)
6	33										Tagged

Port #2 goes to my slave router Port #1. Slave router table is like below.

VLAN	VID	WAN	Tag WAN	1	Tag 1	2	Tag 2	3	Tag 3	4	Tag 4	Default VLAN	Ethernet to bridge mapping
1	1											Default	LAN0 (br0)
2	2												WAN0 bridge
3	34		Tagged										LAN1 (br1)
4	35		Tagged										LAN2 (br2)
5	33		Tagged								Tagged

Please let me know if you see anything wrong with above config. I really like to make use of Proxmox vlans.
Thank you in advance!!

Edits:
- Looks like tag icons are not retained in the tables above, replaced them with text

tjingeling · Jul 12, 2023

A long shot: Could it be that you put the (default) gateway on a VLAN tagged bridge?

Spoonman2002 · Jul 12, 2023

vmbr0 has no Gateway.
Put the Gateway on vmbr0 and not on a vlan.

Spoonman2002 · Jul 12, 2023

austin17427 said:
I'm trying to setup `vlan 33` with a single nic on my proxmox server. I have the below configuration on my `/etc/network/interfaces` file.
auto lo iface lo inet loopback iface enp2s0 inet manual auto vmbr0.33 iface vmbr0.33 inet static address [redacted]/24 gateway [redacted] auto vmbr0 iface vmbr0 inet static bridge-ports enp2s0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094

Once the above configuration applied, machine can not access anything.

$ ping -c 2 [redacted] PING [redacted] (10.24.0.1) 56(84) bytes of data. From [redacted] icmp_seq=1 Destination Host Unreachable From [redacted] icmp_seq=2 Destination Host Unreachable

IP address looks normal.
$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000 link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff 7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff inet6 fe80::8e89:a5ff:fe5d:9f5a/64 scope link valid_lft forever preferred_lft forever 8: vmbr0.33@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 8c:89:a5:5d:9f:5a brd ff:ff:ff:ff:ff:ff inet [redacted]/24 scope global vmbr0.33 valid_lft forever preferred_lft forever inet6 fe80::8e89:a5ff:fe5d:9f5a/64 scope link valid_lft forever preferred_lft forever

I tried every permutation on the switch TL-SG108E like below, this machine is connected to the port #3 of this switch. All other vlans (34,35,36) are functional.

VLAN ID VLAN Name Member Ports Tagged Ports Untagged Ports Delete
1 Default 1-4 1-4
33 Dev 1-3 1-3
34 IoT 1-2 1-2
35 Guest 1-2 1-2
36 TVs 1,5-8 1 5-8

Switch port #1 goes to my main router port #4. My main router vlan table is as below.

VLAN

VID
WAN
Tag
WAN
1
Tag
1
2
Tag
2
3
Tag
3
4
Tag
4
Default
VLAN Ethernet to
bridge
mapping
1 1 Default LAN0 (br0)
2 2 WAN0 bridge
3 34 Tagged LAN1 (br1)
4 35 Tagged LAN2 (br2)
5 36 Tagged LAN3 (br3)
6 33 Tagged

Port #2 goes to my slave router Port #1. Slave router table is like below.

VLAN

VID
WAN
Tag
WAN
1
Tag
1
2
Tag
2
3
Tag
3
4
Tag
4
Default
VLAN Ethernet to
bridge
mapping
1 1 Default LAN0 (br0)
2 2 WAN0 bridge
3 34 Tagged LAN1 (br1)
4 35 Tagged LAN2 (br2)
5 33 Tagged Tagged

Please let me know if you see anything wrong with above config. I really like to make use of Proxmox vlans.
Thank you in advance!!

Edits:
- Looks like tag icons are not retained in the tables above, replaced them with text

If WAN port on slave router is not a member of the regular LAN then you have double NAT active. It is not wrong, but double NAT gives more headache when you want to open ports etc.

austin17427 · Jul 12, 2023

WAN port on the slave router is part of regular LAN, see below;

Code:

# robocfg show
Switch: enabled gigabit
Port 0: 1000FD enabled stp: none vlan: 1 jumbo: off mac: [redacted]
Port 1:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 2:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 3:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 4:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 5:   DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 8: 1000FD enabled stp: none vlan: 1 jumbo: off mac: [redacted]
VLANs: BCM53115 enabled mac_check mac_hash
   1: vlan1: 0 1 2 3 4 8t
   2: vlan2: 8t
  32: vlan32: 0t 4t 8t
  33: vlan33: 0t 4t 8t
  34: vlan34: 0t 8t
  35: vlan35: 0t 8t

In fact, I can now access the vlan 33 from a workstation but the Proxmox can not ping gateway or access internet.
Note: I'm learning about networking, I don't have previous experience when it comes to setting up networking in Linux.

Spoonman2002 · Jul 12, 2023

austin17427 said:
WAN port on the slave router is part of regular LAN, see below;

Code:

# robocfg show Switch: enabled gigabit Port 0: 1000FD enabled stp: none vlan: 1 jumbo: off mac: [redacted] Port 1: DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00 Port 2: DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00 Port 3: DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00 Port 4: DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00 Port 5: DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00 Port 8: 1000FD enabled stp: none vlan: 1 jumbo: off mac: [redacted] VLANs: BCM53115 enabled mac_check mac_hash 1: vlan1: 0 1 2 3 4 8t 2: vlan2: 8t 32: vlan32: 0t 4t 8t 33: vlan33: 0t 4t 8t 34: vlan34: 0t 8t 35: vlan35: 0t 8t

In fact, I can now access the vlan 33 from a workstation but the Proxmox can not ping gateway or access internet.
Note: I'm learning about networking, I don't have previous experience when it comes to setting up networking in Linux.

From Proxmox host what is the output of:
# ip route
# dig debian.org

austin17427 · Jul 12, 2023

Spoonman2002 said:
vmbr0 has no Gateway.
Put the Gateway on vmbr0 and not on a vlan.

@Spoonman2002 Do you mean the `/etc/network/interfaces` file on Proxmox server? But every example I looked at, has the Gateway on vlan and NOT on `vmbr0` even the Proxmox documentation. See documentation at https://pve.proxmox.com/wiki/Network_Configuration > VLAN 802.1Q > Vlan on the Host section and the snippet starting with the sentence "Example: Use VLAN 5 for the Proxmox VE management IP with VLAN aware Linux bridge". I have a single NIC on Proxmox and I got this configuration working! I can access the vlan33 on proxmox from a workstation when I have below NW config.

Code:

ip link add link ens9 name ens9.33 type vlan id 33
ip addr add x.x.x.x/24 dev ens9.33
ifconfig ens9.33 up

and then I can ping the Proxmox

Code:

ping -c 2 [redacted]
.....
2 packets transmitted, 2 received, 0% packet loss, time 1009ms

But the Proxmox can't ping the gateway or access Internet. Again, I only have a single NIC, and this should be possible with a single NIC...

austin17427 · Jul 12, 2023

Spoonman2002 said:
From Proxmox host what is the output of:
# ip route
# dig debian.org

This is the output


# ip r
default via x.x.x.1 dev vmbr0.33 proto kernel onlink 
x.x.x.0/24 dev vmbr0.33 proto kernel scope link src [redacted]


# dig debian.org
;; communications error to [my-dns-server]#53: timed out

Same thing happens when I use 1.1.1.1 as dns. Proxmox can't access outside.

Spoonman2002 · Jul 12, 2023

austin17427 said:
This is the output
# ip r default via x.x.x.1 dev vmbr0.33 proto kernel onlink x.x.x.0/24 dev vmbr0.33 proto kernel scope link src [redacted]

# dig debian.org ;; communications error to [my-dns-server]#53: timed out

Same thing happens when I use 1.1.1.1 as dns. Proxmox can't access outside.

Has VLAN33 got a DNS server configured and a way out to the internet (gateway) (in your main router perhaps)?
I think not, that's why Proxmox cannot resolve debian.org through VLAN33.

Why don't you set the gateway on vmbr0, to make things a bit easier for you to configure?

Spoonman2002 · Jul 12, 2023

And why do you delete the ip addresses in your posts and replace them with x.x.x.x ?
If it is your local internal network......nobody is going to hack you.

austin17427 · Jul 12, 2023

Spoonman2002 said:
Has VLAN33 got a DNS server configured and a way out to the internet (gateway) (in your main router perhaps)?
I think not, that's why Proxmox cannot resolve debian.org through VLAN33.

Why don't you set the gateway on vmbr0, to make things a bit easier for you to configure?

Thank you for your answers. Because I followed this tutorial https://youtu.be/ljq6wlzn4qo and same config is in Proxmox documentation, that's why.
Both puts the address and gateway under vlan. I'm not trying to be stubborn. My objective is to be able use vlans on Proxmox and create separate vlans for Proxmox Management (UI+API) and separate vlans for VMs. I'm trying to get the first vlan (management) working before I add a second vlan for VMs. Have a working config and better understanding of Proxmox vlans and then copy solution to my existing Proxmox VEs.

If you know a better tutorial or if you have a config for accomplishing the above with a single nic, I'll be very curious to look at that.

austin17427 · Jul 12, 2023

Spoonman2002 said:
And why do you delete the ip addresses in your posts and replace them with x.x.x.x ?
If it is your local internal network......nobody is going to hack you.

Lol, I'm a noob in networking, that's why... Not very comfortable with my knowledge, so trying not to expose much.

louie1961 · Jul 13, 2023

FWIW, I have my machine set up the exact same way as you, and my /etc/network/interfaces file is identical to yours and working correctly. The problem is either in your managed switch configuration (trunked ports/proper tagging) or perhaps you have misconfigured the VM? Is the machine your trying to ping from a VM on your node? And if yes, did you tag the VM with a VLAN when you created it?

austin17427 · Jul 13, 2023

Yes the problem was my managed switch. That's how I fixed the issue.

Spoonman2002 · Jul 13, 2023

austin17427 said:
Yes the problem was my managed switch. That's how I fixed the issue.

Can you explain exactly what you fixed in your switch?
From your screenshot it looks correct.

austin17427 · Jul 13, 2023

@Spoonman2002 I fixed added tagged on the switch for port#3 first (initially it wasn't like that, I edited the question here too) and then fixed the workstation's network settings that was trying to access Proxmox. I understand the whole thread doesn't give much insight and the solution isn't very satisfactory. I'm sorry about that. But the access problem to Proxmox is resolved.

The bigger picture which is not mentioned in the problem statement was, to have a Kubernetes cluster (on Proxmox) that has nodes in the same VLAN. However, same cluster provides network wide services like DNS, Nginx Proxy, and some other services. So as far as that problem goes, I still don't know how to solve it yet. At the same time, I'm not sure if that's a relevant topic for Proxmox forum.

Search

Search

[SOLVED] vlan on single nic doesn't work?!?

austin17427

Member

tjingeling

New Member

Spoonman2002

Active Member

Spoonman2002

Active Member

austin17427

Member

Spoonman2002

Active Member

austin17427

Member

austin17427

Member

Spoonman2002

Active Member

Spoonman2002

Active Member

austin17427

Member

austin17427

Member

louie1961

Active Member

austin17427

Member

Spoonman2002

Active Member

austin17427

Member