[solved] problem with vlans and virtual switch

[BillyIT]

Hi,

I'm new to proxmox and I can't find how to do what I have in mind properly. I know how to do it in virtualbox but no way to do it in proxmox.
Topology :
My proxmox host is connected with one nic enp0s3
I have vmbr0 with enp0s3

vmbr0<>Pfsense<>vmbr3<>my vm

I put one virtual nic of pfsense on vmbr0 and it's ok, i received my ip from my real lan.
I put one virtual nic on vmbr3 and vm too on vmbr3
I set manual ip of 192.168.200.254/24 for my pfsense and 192.168.200.180/24 for my vm with the .254 for gateway.
When i try to ping from pfsense interfaces in the vlan 192.168.200.0 or from my vm. No ping worked.
I have this when i tcpdump the vmbr3 interfaces :

11:02:37.270726 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.200.254 tell 192.168.200.180, length 28
11:02:38.026809 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.200.254 tell 192.168.200.180, length 28

It's like the pfsense and vm are not interconnected

Can someone help me out ?

/etc/network/interfaces is :
auto lo
iface lo inet loopback

iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.10.250/24
gateway 192.168.10.1
bridge-ports enp3s0
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0

auto vmbr2
iface vmbr2 inet static
address 192.168.100.0/24
bridge-ports none
bridge-stp off
bridge-fd 0
#Isolated network

auto vmbr3
iface vmbr3 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
mtu 1500
bridge_ageing 0
#réseau privé
auto vlan10
iface vlan10 inet manual
vlan-raw-device vmbr3
#192.168.200.0/24

auto vlan20
iface vlan20 inet manual
vlan-raw-device vmbr3
#192.168.210.0/24

normally, pfsense and my vm is in vlan 10

I can't find a way to tag them in vlan 10. When i try to do it on the virtual nic (i tried E1000 and paravirtual), i have an error and i can't start my vm. In my memory, something like no physical ethernet card on vmbr3 -i have only one ethernet card on vmb0

 

[aaron]

Hmm, to unterstand it better, why do you want the VM and the pfsense interface using vmbr3 to be in vlan10 if they are already separated?

i have an error and i can't start my vm.

Can you post the actual error message? That could potentially help us to narrow down why

 

[BillyIT]

When starting in webui, the error is not explicit. In console :
root@proxmox:~# qm start 104no physical interface on bridge 'vmbr3'
kvm: -netdev type=tap,id=net0,ifname=tap104i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown: network script /var/lib/qemu-server/pve-bridge failed with status 6400
start failed: QEMU exited with code 1

I'm trying to make this topology :



So i need to have pfsense with one interface to my lan/internet and one interface to my vms on different vlans. For now i'm working on my windows server's vm that will belong to my vlan 10 (192.168.200.0/24)

Pfsense<>Linux mint VM is working find on vmbr1 with ip 10.0.0.0/30
i can't find why i'm not able to do the same with vmbr3 to put my vm and pfsense together (i create rules hyper permissive on the pfsense firewall to test)

 

[spirit]

I don't think that

vlan10/vlan20 is working are vlan-raw-device.
(you are doing internal bridge vlan, like vmbr3.10), i should only be used to setup ip on proxmox itself.

Best way, is to create a vlan-aware vmbr3 (or vmbr1 from your schema), and setup vlan tag on the vm nic gui.

(or create 2 differents vmbrX non-vlan-aware for each vlan, as they are not plugged to a physical interface anyway)

 

[BillyIT]

Spirit,

if I'm trying to make vlan on vmbr3, it said that there is no physical interface.

 

[BillyIT]

When i put vlan tag on the nic of my vm, my vm doesn't have any connection, no nic card !

 

[spirit]

you need to enable vlan-aware option on vmbr3.

 

[spirit]

and about your windows, do you have installed the virtio drivers ?

 

[BillyIT]

oh no, i didn't. I'll try that. (i tried with the E1000 but it wasn't working)

 

[BillyIT]

i can't mount my cdrom with virtio iso on my windows vm
i try to fix the pending message in the vm config file but not working. i tried ide and sata cdrom

 

[BillyIT]

ok, i installed the virtio driver. The card is present on my vm.
But no way to ping my gateway.
tcpdum -i vmbr3 : always the arp request but no reply to give the mac-address of my pfsense interface on the vmbr3

 

[BillyIT]

and about your windows, do you have installed the virtio drivers ?

Hi @spirit

Windows vm has the drivers
I don't have any connection between pfsense<>vmbr3<>windows

If you have any tips ? Do you need more infos?

 

[BillyIT]

Hi all,

When waiting for advices or help, i'm trying to find solutions. I saw somewhere that sometimes (with some nic virtualisation drivers) the vlan can't cover 2-4096 but less... So i tried to put this :
auto vmbr3
iface vmbr3 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 10 20 30 40 50
mtu 1500
bridge_ageing 0

And i retried to capture the trafic on vmbr3 when pinging from my vm windows, tagged normally with tag=10
root@proxmox:~# qm config 104
agent: 1
boot: order=ide0;sata1
cores: 2
ide0: local2:104/vm-104-disk-0.qcow2,size=32G
machine: pc-i440fx-6.0
memory: 4096
name: srv-2019AD
net0: virtio=46:F22:8AC:C3,bridge=vmbr3,tag=10
numa: 0
ostype: win10
sata1: local:iso/virtio.iso,media=cdrom,size=528322K
scsihw: virtio-scsi-pci
smbios1: uuid=a6507048-c027-472d-b630-b25a4a480203
sockets: 1
vmgenid: baa290f1-90eb-4f0d-b026-67e0a351aac5

Now, i don't see anymore the arp request from my windows ! So, there is a real problem with vlans in my vmbr3 or my vm or...
If someone can help with this information ?

 

[BillyIT]

ok, i solved it youhou !! Happy day for me !!!

I delete my vnic on my pfsense and my windows vm
I added my vnic on windows with vlan tag 10 and i did the same for pfsense but, i did a mistake before in putting vlan 10. I don't put vlan tag, because it needs to sniff all vlan traffics (10 20 30 40 50)
I did that with vms stopped and after that to be good, i restart my proxmox node server.

I was always tcpdump the vmbr3 and saw trafics from my windows after the start... So i knew there is some good news in it (internet trafic), i tried to ping and boom. Pfsense to windows ok !!

So next time i need to implement vlan on vmbr :
1) vmbr aware vlan yes
2) add new (delete if exists) vnic with the vlan tag
3) restart my node (maybe not indispensable if we put vnic after the vmbr aware)
4) enjoy !

Thanks @spirit for your help !!