Difficulties with VLAN in network configuration of Proxmox VE 5.4

Joogser

Member
Jun 30, 2019
25
4
23
38
Russian Federation
habr.com
Hello Colleagues.

This note added in July 14, 2019
NOTE! This schema works in nested virtualisation, Proxmox installed in Hyper-V 2016


I'm beginner in a Proxmox VE administration, some things are not so clear for me yet.
So, on a picture below, this is my case and my vision how it's need to configure. Please correct me if I have wrong!
proxmox_1-jpg.10750

If my vision is correct in that case and your will ask, why some things are not so clear for me? Read next ...
I did install Proxmox VE without subscription, from .iso image downloaded from official web-page on a blade server.

Begin configuration, here listing steps of tries to configure my schema:
  1. Install Proxmox VE 5.4
  2. Install the Open vSwitch packages from Debian repository. Not from Enterprise repository! Just because I don't have subscription.
    Code:
    apt-get install openvswitch-switch
    Open vSwitch comes with v.2.6.6
  3. In Proxmox web interface

    123-11-jpg.10755
    • I did remove default Linux bridge
    • I did create OVS Bridge vmbr0, in Bridge port set interface of physical NIC (Network Interface Card) enp0s5 and specified IP address, subnet, gateway
    • I did create OVS Bridge vmbr1, field Bridge port leaved empty, click create
    • I did create OVS IntPort with name vlan2058, choose our new bridge port vmbr1, specified Vlan Tag: 2058 (for untagged mode, right?), click create
    • Reboot Proxmox VE server
      Code:
      shutdown -r now
      or just click Reboot button in Proxmox web interface.

    So, we have now:

    This
    vmbr0 made for Proxmox addressing and his management.
    123-13-jpg.10757


    This OVS IntPort made for TASK 1 - take a look on schema
    Specified Vlan Tag: 2058 (for untagged mode, right?)
    123-16-jpg.10761


    This vmbr1 made for TASK 1 - take a look on schema
    123-12-jpg.10756


  4. Create NIC for VM with Centos 7 OS:
    123-14-jpg.10758


    Choose vmbr0 for test scope Proxmox addressing
    123-15-jpg.10760


    Choose vmbr1 for Supercompany.com, according TASK 1

    123-17-jpg.10763


    Choose vmbr0 for Goodcompany.com, according TASK 2 and set vlan tag 2060 for access to tagged packets
    123-18-jpg.10764


    Read more in the next post... I can't attach more than 10 files to one post.
 

Attachments

  • proxmox_1.jpg
    proxmox_1.jpg
    842.1 KB · Views: 289
  • 123-11.jpg
    123-11.jpg
    22.2 KB · Views: 269
  • 123-12.jpg
    123-12.jpg
    25.8 KB · Views: 266
  • 123-13.jpg
    123-13.jpg
    29.7 KB · Views: 273
  • 123-14.jpg
    123-14.jpg
    67.2 KB · Views: 267
  • image.png
    image.png
    137.9 KB · Views: 8
  • 123-15.jpg
    123-15.jpg
    29.8 KB · Views: 262
  • 123-16.jpg
    123-16.jpg
    31.4 KB · Views: 267
  • 123-17.jpg
    123-17.jpg
    48.3 KB · Views: 262
  • 123-18.jpg
    123-18.jpg
    23.4 KB · Views: 267
Last edited:
Results:
  1. Start my VM and enable each NIC for test network connections in setup menu of Centos 7 OS

    Here OK!
    123-19-jpg.10770


    In other cases, no IP, it's mean my schema doesn't work :( or doesn't work proxmox

    123-20-jpg.10771


    123-21-jpg.10772


    Eth1 (vmbr1 for taking untagged traffic from vlan2058), Eth2 (vmbr0 for taking tagged traffic 2060). This interfaces doesn't work in this case. Nothing going. I went to winbox to watch activity in Mikrotik. In Mikrotik no any activity, in DHCP server in Leases tab, persist only 10.1.1.2 from eth0, also in Torch (Mikrotik tool for traffic snifing) no any udp 67,68 ports requests and replies (67,68 ports are for DHCP).

    Somewhere I was reading, Proxmox doesn't interact correct with Open vSwitch installed from Debian repository, for correct interaction Proxmox environment need install openvswitch-switch from official repository only!

    123-26-jpg.10780


  2. Okay, life is struggle, I did erase my Sata Dom by formating and reinstall Proxmox.
    Then in a fresh Proxmox server I did configure no-subscription repository:
    Code:
    nano /etc/apt/sources.list
    123-22-jpg.10773
    Code:
    nano /etc/apt/sources.list.d/pve-enterprise.list
    123-23-jpg.10774


    Code:
    apt-get update && apt-get upgrade -y
    apt-get dist-upgrade
    reboot
  3. I did repeat all configuration that were early, in result I had same situation with single difference!
    This difference is consists in one, in Mikrotik, in Torch (Mikrotik tool) I've got dhpc address request
    123-24-jpg.10776

    Also in DHCP Leases tab I've saw
    123-25-jpg.10777

    Traffic going in one side, I mean outgoing request from VM with Centos 7 passed to Mikrotik, but reply don't going via proxmox, because I think this is issue with OVS trunk.

  4. According to recommendation of Proxmox administration, the repository with no-subscription is not recommended for production. So! I am totally agree with this statements on a practice and my conclusion => VLAN networks aren't supporting without subscription.

    This issue I was digging deeper, in one of topic on this forum I've find thread with name Proxmox VE 5.4 released! Click Release notes. In a topic with description of this release, in the end, in last point, click GIT.
    • Countless bug fixes and package updates (for all details see bugtracker and GIT)

    123-27-jpg.10781


    I did try to install this fresh openvswitch version by downloading it from this GIT repo to my test Proxmox environment. I did make every step which were state in attached to this package documentation. In documentation, where has been stated final step, need install one last package (openswitch-datapath) for correct interaction OVS. Okay google, lets go & begin, but after installation of this last package I've got message with error, take look on screnshot below, this error mean my package pve-headers has wrong version (something like this, just find this solution somewhere in i'net) and I need update this package, but this package possible install from enterprise repository only.

    123-28-jpg.10783

    If do you have an idea, please share ;)

  5. Also, I can't to say with 100% confidence that VLAN networks aren't supporting without subscription.
    Just because I didn't use Subscription yet & want to ask your opinion, thoughts towards the subject according to my design before buy it!

    Guys, share your experience according to my case. If I will buy subscription, my schema(concept) will be workable from a box after updates, or I still will be expect difficulties? It'll be possible adjust via Proxmox web-interface, or it'll be need spend time on a fine tuning via CLI?
 

Attachments

  • 123-19.jpg
    123-19.jpg
    53.4 KB · Views: 260
  • 123-20.jpg
    123-20.jpg
    40 KB · Views: 252
  • 123-21.jpg
    123-21.jpg
    42.2 KB · Views: 249
  • 123-22.jpg
    123-22.jpg
    45.3 KB · Views: 238
  • 123-23.jpg
    123-23.jpg
    30 KB · Views: 233
  • 123-24.jpg
    123-24.jpg
    77.3 KB · Views: 236
  • 123-25.jpg
    123-25.jpg
    48.8 KB · Views: 231
  • 123-26.jpg
    123-26.jpg
    88.5 KB · Views: 246
  • 123-27.jpg
    123-27.jpg
    274 KB · Views: 243
  • 123-28.jpg
    123-28.jpg
    138.4 KB · Views: 238
Last edited:
Can't help you with all questions. But my servers:
1] vlan's works with no-subs repo
2] using ovs (from pve no-subs repo)
3] all config written directly (not using webgui) - i had problems with configuration from webgui in pve5.3 and older, can't say for pve5.4 and newer
4] not using dkms
 
The reason has been found. It's happened because this proxmox has been installed in test environment of Hyper-V. I don't know yet why Virtual Switch in Hyper-V 2016 don't delivery tagged traffic into second virtual NIC of VM in proxmox.

Just I didn't tell here about this nuance. I did install Proxmox VE into Sata Dom on real server, in according my schema\concept and no any problem. Works fine. Just I was absolutely sure vSwitch in Hyper-V works in trunk mode.

If somebody known how to fix this issue in vSwitch of Hyper-V, plz, share experience, will be interesting to know it up.

And yes, in Hyper-V server, nested virtualisation doesn't supporting by default, to successfully run inside a virtual machine, these virtualization extensions need to be enabled and presented to the guest operating system.

Make connection to Hyper-V Server, run Power Shell:
  • To enable virtualization extensions, use this commands:
    Code:
    Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true
    Code:
    Get-VMProcessor -VMName | FL ExposeVirtualizationExtensions
  • To know what status VM has, true or false (supporting\don't supporting), use this
    Code:
    Get-VMProcessor -VMName | FL ExposeVirtualizationExtensions
Instead the -VMName past here the name of your VM with Proxmox or some another Virtual Environment server.

I did try install it in Parallels VE, this is VE for MacOS, it also had the same issue with network.
So, this issue with vlans persists in nested virtualisation only, on a real hardware works fine.
 
Last edited:
The reason has been found. It's happened because this proxmox has been installed in test environment of Hyper-V. I don't know yet why Virtual Switch in Hyper-V 2016 don't delivery tagged traffic into second virtual NIC of VM in proxmox.

Just I didn't tell here about this nuance. I did install Proxmox VE into Sata Dom on real server, in according my schema\concept and no any problem. Works fine. Just I was absolutely sure vSwitch in Hyper-V works in trunk mode.

If somebody known how to fix this issue in vSwitch of Hyper-V, plz, share experience, will be interesting to know it up.

And yes, in Hyper-V server, nested virtualisation doesn't supporting by default, to successfully run inside a virtual machine, these virtualization extensions need to be enabled and presented to the guest operating system.

Make connection to Hyper-V Server, run Power Shell:
  • To enable virtualization extensions, use this commands:
    Code:
    Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true
    Code:
    Get-VMProcessor -VMName | FL ExposeVirtualizationExtensions
  • To know what status VM has, true or false (supporting\don't supporting), use this
    Code:
    Get-VMProcessor -VMName | FL ExposeVirtualizationExtensions
Instead the -VMName past here the name of your VM with Proxmox or some another Virtual Environment server.

I did try install it in Parallels VE, this is VE for MacOS, it also had the same issue with network.
So, this issue with vlans persists in nested virtualisation only, on a real hardware works fine.

It might have something to do with promiscuous mode of the network not allowing for additional encapsulation of nested network packets for different vLans.

This is a VMware link but would assume the issue is the same on Hyper-V or with any nested hypervisor.

https://kb.vmware.com/s/article/1002934

This would explain why the untagged vLan works but not the defined vLan.

Hope the above helps.

“”Cheers
G
 
It might have something to do with promiscuous mode of the network not allowing for additional encapsulation of nested network packets for different vLans.

This is a VMware link but would assume the issue is the same on Hyper-V or with any nested hypervisor.

https://kb.vmware.com/s/article/1002934

This would explain why the untagged vLan works but not the defined vLan.

Hope the above helps.

“”Cheers
G

Sounds good... like a workable solution, just need google more about Hyper-V & this thin network configuration. I did remove Hyper-V now, I don't have the capability to try fix this issue, but I'm guessing this articles will help:

dfw-jpg.10892


df-jpg.10893


do you have tried with vlan-aware bridge instead openvswitch ?

Yes, I did, it also doesn't gave to me positive results, I mean case with nested virtual environment, Proxmox inside of Hyper-V.

To sum up!
during this discussion
All issues in this concept got a resolutions.

However, it would be great if someone shared their thoughts on how to set it up in Parallels VE in Mac OS.
 

Attachments

  • dfw.jpg
    dfw.jpg
    116.9 KB · Views: 179
  • df.jpg
    df.jpg
    152.6 KB · Views: 182
Last edited:
Sounds good... like a workable solution, just need google more about Hyper-V & this thin network configuration. I did remove Hyper-V now, I don't have the capability to try fix this issue, but I'm guessing this articles will help:

dfw-jpg.10892


df-jpg.10893




Yes, I did, it also doesn't gave to me positive results, I mean case with nested virtual environment, Proxmox inside of Hyper-V.

To sum up!
during this discussion
All issues in this concept got a resolutions.

However, it would be great if someone shared their thoughts on how to set it up in Parallels VE in Mac OS.

the correct solution for nested would be more inline with this:
https://www.jeffreykusters.nl/2018/05/22/vlans-vxlans-ospf-nested-vmware-homelab/

the same principals still apply for all hypervisors.

not sure if i remember reading why you need this to work in Hyper-V is it just for testing ?

""Cheers
G
 

Yes it is, but I was writing about this difficulties, it doesn't work in Proxmox from a BOX and it's need adjust and make vSwitch configuration.

Okay, in this link, which did you share, there is wrote a way with configuration MTU. I did try use example like example showed for VMWare:

efsdf-jpg.10921


Doesn't work...probably need to make some config on Mikrotik with MTU size, because something went wrong. Kinda like the network stopped working at all, or this issue has been still actual, don't remember accuracy. It would be perfect, if somebody will share own config of vSwitch here according this concept which has been described in first two messages of this thread. Just because all this links are wastepapers and doesn't give clear resolution for this issue. It's only make brain working and time spending and as like result nothing. Probably here need subscription, probably not, just all this recipes for vSwitch cooking didn't work.

not sure if i remember reading why you need this to work in Hyper-V is it just for testing ?
Yes, for testing Proxmox in different configuration, like a Ceph and other....
Just wanna use it for test in specific VLAN network.
 

Attachments

  • efsdf.jpg
    efsdf.jpg
    290.6 KB · Views: 179
Last edited:
Yes it is, but I was writing about this difficulties, it doesn't work in Proxmox from a BOX and it's need adjust and make vSwitch configuration.

Okay, in this link, which did you share, there is wrote a way with configuration MTU. I did try use example like example showed for VMWare:

efsdf-jpg.10921


Doesn't work...probably need to make some config on Mikrotik with MTU size, because something went wrong. Kinda like the network stopped working at all, or this issue has been still actual, don't remember accuracy. It would be perfect, if somebody will share own config of vSwitch here according this concept which has been described in first two messages of this thread. Just because all this links are wastepapers and doesn't give clear resolution for this issue. It's only make brain working and time spending and as like result nothing. Probably here need subscription, probably not, just all this recipes for vSwitch cooking didn't work.


Yes, for testing Proxmox in different configuration, like a Ceph and other....
Just wanna use it for test in specific VLAN network.

Hi Joogser

A few points if I may.

1. Your initial post excluded the primary hyperviser being used which is hyper-v, this is a major exclusion from the problem your asking to be solved.

This is a ProxMox forum not hyper-v if you need help configuring hyper-v this this would be best asked on a hyper-v forum.

2. I replied to your query on why the network was behaving in this manor which was the actual question at hand. This has to do with promiscuous mode being enabled in hyper-v.

This has nothing to do with ProxMox networking and the issue here is not ProxMox but again it’s a hyper-v issue.

ProxMox is a management interface for KVM, networking standards “vxlan, vlan, oVS etc” primary storage Local lvm and Ceph and more.

These are all open source technologies which ProxMox VE uses to deliver a solid virtualisation platform most of which are managed via the GUI.

The issue your experiencing has nothing to do with any of the above technologies but everything to do with Hyper-v or any hypervisor being used to nest within.

3. I took the time to show you that this issue has to do with all hypervisors that are providing nesting so again this isn’t a ProxMox issue it’s a hyper-v issue you’ll need to work out.

Sorry if I or others can’t be of further help with this but it appears your question would be better answered in a Hyper-v forum.

Let me know if there is a ProxMox related question and I promise I’ll do my best to help if I can :)

“”Cheers
G
 
  • Like
Reactions: Joogser

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!