Difficulties with VLAN in network configuration of Proxmox VE 5.4

[Joogser]

Hello Colleagues.

This note added in July 14, 2019
NOTE! This schema works in nested virtualisation, Proxmox installed in Hyper-V 2016

I'm beginner in a Proxmox VE administration, some things are not so clear for me yet.
So, on a picture below, this is my case and my vision how it's need to configure. Please correct me if I have wrong!

If my vision is correct in that case and your will ask, why some things are not so clear for me? Read next ...
I did install Proxmox VE without subscription, from .iso image downloaded from official web-page on a blade server.

Begin configuration, here listing steps of tries to configure my schema:

1. Install Proxmox VE 5.4
2. Install the Open vSwitch packages from Debian repository. Not from Enterprise repository! Just because I don't have subscription.
   

   apt-get install openvswitch-switch

   Open vSwitch comes with v.2.6.6
3. In Proxmox web interface
   
   
   
   • I did remove default Linux bridge
   • I did create OVS Bridge vmbr0, in Bridge port set interface of physical NIC (Network Interface Card) enp0s5 and specified IP address, subnet, gateway
   • I did create OVS Bridge vmbr1, field Bridge port leaved empty, click create
   • I did create OVS IntPort with name vlan2058, choose our new bridge port vmbr1, specified Vlan Tag: 2058 (for untagged mode, right?), click create
     
   • Reboot Proxmox VE server
     

     shutdown -r now

     or just click Reboot button in Proxmox web interface.
   
   So, we have now:
   
   This vmbr0 made for Proxmox addressing and his management.
   
   
   
   
   This OVS IntPort made for TASK 1 - take a look on schema
   Specified Vlan Tag: 2058 (for untagged mode, right?)
   
   
   
   
   This vmbr1 made for TASK 1 - take a look on schema
   
   
   
   
4. Create NIC for VM with Centos 7 OS:
   
   
   
   
   Choose vmbr0 for test scope Proxmox addressing
   
   
   
   Choose vmbr1 for Supercompany.com, according TASK 1
   
   
   
   
   
   Choose vmbr0 for Goodcompany.com, according TASK 2 and set vlan tag 2060 for access to tagged packets
   
   
   
   
   Read more in the next post... I can't attach more than 10 files to one post.

 

[Joogser]

Results:

1. Start my VM and enable each NIC for test network connections in setup menu of Centos 7 OS
   
   Here OK!
   
   
   
   
   In other cases, no IP, it's mean my schema doesn't work or doesn't work proxmox
   
   
   
   
   
   
   
   
   
   Eth1 (vmbr1 for taking untagged traffic from vlan2058), Eth2 (vmbr0 for taking tagged traffic 2060). This interfaces doesn't work in this case. Nothing going. I went to winbox to watch activity in Mikrotik. In Mikrotik no any activity, in DHCP server in Leases tab, persist only 10.1.1.2 from eth0, also in Torch (Mikrotik tool for traffic snifing) no any udp 67,68 ports requests and replies (67,68 ports are for DHCP).
   
   Somewhere I was reading, Proxmox doesn't interact correct with Open vSwitch installed from Debian repository, for correct interaction Proxmox environment need install openvswitch-switch from official repository only!
   
   
   
   
   
   
2. Okay, life is struggle, I did erase my Sata Dom by formating and reinstall Proxmox.
   Then in a fresh Proxmox server I did configure no-subscription repository:
   

   nano /etc/apt/sources.list

   

   nano /etc/apt/sources.list.d/pve-enterprise.list

   
   
   
   

   apt-get update && apt-get upgrade -y
   apt-get dist-upgrade
   reboot

3. I did repeat all configuration that were early, in result I had same situation with single difference!
   This difference is consists in one, in Mikrotik, in Torch (Mikrotik tool) I've got dhpc address request
   
   
   
   Also in DHCP Leases tab I've saw
   
   
   
   Traffic going in one side, I mean outgoing request from VM with Centos 7 passed to Mikrotik, but reply don't going via proxmox, because I think this is issue with OVS trunk.
   
   
4. According to recommendation of Proxmox administration, the repository with no-subscription is not recommended for production. So! I am totally agree with this statements on a practice and my conclusion => VLAN networks aren't supporting without subscription.
   
   This issue I was digging deeper, in one of topic on this forum I've find thread with name Proxmox VE 5.4 released! Click Release notes. In a topic with description of this release, in the end, in last point, click GIT.
   

   • Countless bug fixes and package updates (for all details see bugtracker and GIT)

   
   
   
   
   
   I did try to install this fresh openvswitch version by downloading it from this GIT repo to my test Proxmox environment. I did make every step which were state in attached to this package documentation. In documentation, where has been stated final step, need install one last package (openswitch-datapath) for correct interaction OVS. Okay google, lets go & begin, but after installation of this last package I've got message with error, take look on screnshot below, this error mean my package pve-headers has wrong version (something like this, just find this solution somewhere in i'net) and I need update this package, but this package possible install from enterprise repository only.
   
   
   
   
   If do you have an idea, please share
   
   
5. Also, I can't to say with 100% confidence that VLAN networks aren't supporting without subscription.
   Just because I didn't use Subscription yet & want to ask your opinion, thoughts towards the subject according to my design before buy it!
   
   Guys, share your experience according to my case. If I will buy subscription, my schema(concept) will be workable from a box after updates, or I still will be expect difficulties? It'll be possible adjust via Proxmox web-interface, or it'll be need spend time on a fine tuning via CLI?

 

[czechsys]

Can't help you with all questions. But my servers:
1] vlan's works with no-subs repo
2] using ovs (from pve no-subs repo)
3] all config written directly (not using webgui) - i had problems with configuration from webgui in pve5.3 and older, can't say for pve5.4 and newer
4] not using dkms

 

[Joogser]

The reason has been found. It's happened because this proxmox has been installed in test environment of Hyper-V. I don't know yet why Virtual Switch in Hyper-V 2016 don't delivery tagged traffic into second virtual NIC of VM in proxmox.

Just I didn't tell here about this nuance. I did install Proxmox VE into Sata Dom on real server, in according my schema\concept and no any problem. Works fine. Just I was absolutely sure vSwitch in Hyper-V works in trunk mode.

If somebody known how to fix this issue in vSwitch of Hyper-V, plz, share experience, will be interesting to know it up.

And yes, in Hyper-V server, nested virtualisation doesn't supporting by default, to successfully run inside a virtual machine, these virtualization extensions need to be enabled and presented to the guest operating system.

Make connection to Hyper-V Server, run Power Shell:

• To enable virtualization extensions, use this commands:
  

  Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true

  Get-VMProcessor -VMName | FL ExposeVirtualizationExtensions

• To know what status VM has, true or false (supporting\don't supporting), use this
  

  Get-VMProcessor -VMName | FL ExposeVirtualizationExtensions

Instead the -VMName past here the name of your VM with Proxmox or some another Virtual Environment server.

I did try install it in Parallels VE, this is VE for MacOS, it also had the same issue with network.
So, this issue with vlans persists in nested virtualisation only, on a real hardware works fine.

 

[velocity08]

The reason has been found. It's happened because this proxmox has been installed in test environment of Hyper-V. I don't know yet why Virtual Switch in Hyper-V 2016 don't delivery tagged traffic into second virtual NIC of VM in proxmox.

Just I didn't tell here about this nuance. I did install Proxmox VE into Sata Dom on real server, in according my schema\concept and no any problem. Works fine. Just I was absolutely sure vSwitch in Hyper-V works in trunk mode.

If somebody known how to fix this issue in vSwitch of Hyper-V, plz, share experience, will be interesting to know it up.

And yes, in Hyper-V server, nested virtualisation doesn't supporting by default, to successfully run inside a virtual machine, these virtualization extensions need to be enabled and presented to the guest operating system.

Make connection to Hyper-V Server, run Power Shell:

• To enable virtualization extensions, use this commands:
  

  Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true

  Get-VMProcessor -VMName | FL ExposeVirtualizationExtensions

• To know what status VM has, true or false (supporting\don't supporting), use this
  

  Get-VMProcessor -VMName | FL ExposeVirtualizationExtensions

Instead the -VMName past here the name of your VM with Proxmox or some another Virtual Environment server.

I did try install it in Parallels VE, this is VE for MacOS, it also had the same issue with network.
So, this issue with vlans persists in nested virtualisation only, on a real hardware works fine.

It might have something to do with promiscuous mode of the network not allowing for additional encapsulation of nested network packets for different vLans.

This is a VMware link but would assume the issue is the same on Hyper-V or with any nested hypervisor.

https://kb.vmware.com/s/article/1002934

This would explain why the untagged vLan works but not the defined vLan.

Hope the above helps.

“”Cheers
G

 

[spirit]

do you have tried with vlan-aware bridge instead openvswitch ?

 

[Joogser]

It might have something to do with promiscuous mode of the network not allowing for additional encapsulation of nested network packets for different vLans.

This is a VMware link but would assume the issue is the same on Hyper-V or with any nested hypervisor.

https://kb.vmware.com/s/article/1002934

This would explain why the untagged vLan works but not the defined vLan.

Hope the above helps.

“”Cheers
G

Sounds good... like a workable solution, just need google more about Hyper-V & this thin network configuration. I did remove Hyper-V now, I don't have the capability to try fix this issue, but I'm guessing this articles will help:

do you have tried with vlan-aware bridge instead openvswitch ?

Yes, I did, it also doesn't gave to me positive results, I mean case with nested virtual environment, Proxmox inside of Hyper-V.

To sum up!
during this discussion
All issues in this concept got a resolutions.

However, it would be great if someone shared their thoughts on how to set it up in Parallels VE in Mac OS.

 

[velocity08]

Sounds good... like a workable solution, just need google more about Hyper-V & this thin network configuration. I did remove Hyper-V now, I don't have the capability to try fix this issue, but I'm guessing this articles will help:

Yes, I did, it also doesn't gave to me positive results, I mean case with nested virtual environment, Proxmox inside of Hyper-V.

To sum up!
during this discussion
All issues in this concept got a resolutions.

However, it would be great if someone shared their thoughts on how to set it up in Parallels VE in Mac OS.

the correct solution for nested would be more inline with this:
https://www.jeffreykusters.nl/2018/05/22/vlans-vxlans-ospf-nested-vmware-homelab/

the same principals still apply for all hypervisors.

not sure if i remember reading why you need this to work in Hyper-V is it just for testing ?

""Cheers
G

 

[Joogser]

https://www.jeffreykusters.nl/2018/05/22/vlans-vxlans-ospf-nested-vmware-homelab/
the same principals still apply for all hypervisors.

Yes it is, but I was writing about this difficulties, it doesn't work in Proxmox from a BOX and it's need adjust and make vSwitch configuration.

Okay, in this link, which did you share, there is wrote a way with configuration MTU. I did try use example like example showed for VMWare:

Doesn't work...probably need to make some config on Mikrotik with MTU size, because something went wrong. Kinda like the network stopped working at all, or this issue has been still actual, don't remember accuracy. It would be perfect, if somebody will share own config of vSwitch here according this concept which has been described in first two messages of this thread. Just because all this links are wastepapers and doesn't give clear resolution for this issue. It's only make brain working and time spending and as like result nothing. Probably here need subscription, probably not, just all this recipes for vSwitch cooking didn't work.

not sure if i remember reading why you need this to work in Hyper-V is it just for testing ?

Yes, for testing Proxmox in different configuration, like a Ceph and other....
Just wanna use it for test in specific VLAN network.

 

[velocity08]

Yes it is, but I was writing about this difficulties, it doesn't work in Proxmox from a BOX and it's need adjust and make vSwitch configuration.

Okay, in this link, which did you share, there is wrote a way with configuration MTU. I did try use example like example showed for VMWare:

Doesn't work...probably need to make some config on Mikrotik with MTU size, because something went wrong. Kinda like the network stopped working at all, or this issue has been still actual, don't remember accuracy. It would be perfect, if somebody will share own config of vSwitch here according this concept which has been described in first two messages of this thread. Just because all this links are wastepapers and doesn't give clear resolution for this issue. It's only make brain working and time spending and as like result nothing. Probably here need subscription, probably not, just all this recipes for vSwitch cooking didn't work.


Yes, for testing Proxmox in different configuration, like a Ceph and other....
Just wanna use it for test in specific VLAN network.

Hi Joogser

A few points if I may.

1. Your initial post excluded the primary hyperviser being used which is hyper-v, this is a major exclusion from the problem your asking to be solved.

This is a ProxMox forum not hyper-v if you need help configuring hyper-v this this would be best asked on a hyper-v forum.

2. I replied to your query on why the network was behaving in this manor which was the actual question at hand. This has to do with promiscuous mode being enabled in hyper-v.

This has nothing to do with ProxMox networking and the issue here is not ProxMox but again it’s a hyper-v issue.

ProxMox is a management interface for KVM, networking standards “vxlan, vlan, oVS etc” primary storage Local lvm and Ceph and more.

These are all open source technologies which ProxMox VE uses to deliver a solid virtualisation platform most of which are managed via the GUI.

The issue your experiencing has nothing to do with any of the above technologies but everything to do with Hyper-v or any hypervisor being used to nest within.

3. I took the time to show you that this issue has to do with all hypervisors that are providing nesting so again this isn’t a ProxMox issue it’s a hyper-v issue you’ll need to work out.

Sorry if I or others can’t be of further help with this but it appears your question would be better answered in a Hyper-v forum.

Let me know if there is a ProxMox related question and I promise I’ll do my best to help if I can

“”Cheers
G