RESOLVED: Can't get any VMs to communicate on VLANs

Discussion in 'Proxmox VE: Networking and Firewall' started by Scott Zupek, Jan 8, 2019.

  1. Scott Zupek

    Scott Zupek New Member
    Proxmox Subscriber

    Joined:
    Nov 5, 2018
    Messages:
    18
    Likes Received:
    1
    RESOLVED: See last thread as the issue ended up being a switch and firewall config issue, not Proxmox VE. My fault.
    ------ Original thread below ------

    I have kept the trial small.

    1. I installed a new VM on the default bridge (vmbr0) and tagged it for vlan 201 (and bridge was vlan aware)
    2. I was NOT able to ping the firewall ip on VLAN 201
    3. I checked the firewall traffic and nothing with vlan tag was making to the firewall, which means the host/proxmox ve was blocking the traffic
    4. I manually created eno1v201, eno2v201 and attached to a new bridge vmbr0201 (with vlan aware enabled and original vmbr0 UNCHECKed for vlan aware)
    5. rebooted, verified configuration, same exact result...no traffic getting past the host.
    Any advice would be recommended. I don't personally want to manually configure /etc/network/interfaces (or interfaces.new) but I will if that's my only option (as I did for step 4 since you can't create new interfaces through the web interfaces). I DO NOT want access to the Promox Management on vlan 201. Attached is my current /etc/network/interfaces config
    Code:
    cat /etc/network/interfaces
    # network interface settings; autogenerated
    # Please do NOT modify this file directly, unless you know what
    # you're doing.
    #
    # If you want to manage part of the network configuration manually,
    # please utilize the 'source' or 'source-directory' directives to do
    # so.
    # PVE will preserve these directives, but will NOT its network
    # configuration from sourced files, so do not attempt to move any of
    # the PVE managed interfaces into external files!
    
    auto lo
    iface lo inet loopback
    
    iface eno1 inet manual
    
    iface eno2 inet manual
    #interface1 vlan 201
    iface eno1v201 inet manual
    #interface2 vlan 201
    iface eno2v201 inet manual
    
    auto vmbr0
    iface vmbr0 inet static
            address  10.21.32.100
            netmask  255.255.255.0
            gateway  10.21.32.254
            bridge-ports eno1 eno2
            bridge-stp off
            bridge-fd 0
    
    auto vmbr0201
    iface vmbr0201 inet manual
            bridge-ports eno1v201 eno2v201
            bridge-stp off
            bridge-fd 0
            bridge-vlan-aware yes
            bridge-vids 2-4094
     
    #1 Scott Zupek, Jan 8, 2019
    Last edited: Jan 11, 2019
  2. Scott Zupek

    Scott Zupek New Member
    Proxmox Subscriber

    Joined:
    Nov 5, 2018
    Messages:
    18
    Likes Received:
    1
    I noticed I have MANUAL instead of STATIC in the vmbr0201 line in interfaces file. I will update to STATIC even though I don't know if I need an IP ADDRESS (again I don't want the Promox host reachable through my clients VMs/vlans), reboot and report back.
     
  3. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,317
    Likes Received:
    132
    Hi,
    for bridge vlan aware, you only need

    auto vmbr0
    iface vmbr0 inet static
    address 10.21.32.100
    netmask 255.255.255.0
    gateway 10.21.32.254
    bridge-ports eno1 eno2
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094


    (BTW, why do you have 2 eno ? if they are on same physical switch/network, you should use bonding, or you'll have network loop)


    without vlanaware :

    auto vmbr0
    iface vmbr0 inet static
    address 10.21.32.100
    netmask 255.255.255.0
    gateway 10.21.32.254
    bridge-ports eno1 eno2
    bridge-stp off
    bridge-fd 0

    auto vmbr0v201
    iface vmbr0v201 inet manual
    bridge-ports eno1.201 eno2.201
    bridge-stp off
    bridge-fd 0


    (but you only need vmbr0, proxmox will create the vmbr0v201 when you'll start the a vm with vlan tag 201)



    Of course, you need to configure you physical switch to allow vlans. (cisco trunk for example)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. Scott Zupek

    Scott Zupek New Member
    Proxmox Subscriber

    Joined:
    Nov 5, 2018
    Messages:
    18
    Likes Received:
    1
    Thank you for the reply.

    Unfortunately your first example is where I started and it did not work, probably due to starting the VM with a VLAN assigned to the guest VM didn't create the vmbr0v201 as you suggested. I did manually create it and it's adapters and restart to try that and it also did not work.

    The 2nd example hasn't worked either (with vlan aware or without)

    I have verified the trunk ports and router configuration is correct by assigned a PC with a vlan tag associated within the guest itself and verifying connectivity.

    I will redo this process again to verify, but so far it's not working.

    Also ENO1 is primary ENO2 is failover. I don't want or need the interfaces bonded because I lose static redundancy that way. I believe I read this is the default behavior in linux but now you have me 2nd guessing, so I'll look again.

    Thanks,
    Scott
     
  5. Scott Zupek

    Scott Zupek New Member
    Proxmox Subscriber

    Joined:
    Nov 5, 2018
    Messages:
    18
    Likes Received:
    1
    This is getting on my last nerve. This documentation is WAYYYY too vague https://pve.proxmox.com/wiki/Network_Configuration

    I have since created the bond0 and assigned eno1 and eno2 and assigned the bond-type to active-backup and it works great for VLAN1(untagged) traffic just like it always has.

    I have tried creating individual interfaces (en1v201 and eno2v201) and set it to manual along with assigning a physical adapter (eno1 and eno2)

    No luck.

    I tried creating a new bridge and assigning eno1v201 and eno2v201, no luck

    I tried created a bond0v201 and assigning eno1v201 and eno2v201 and then assigned bond0v201 to a bridge on it's own (using the web gui), also didn't work.

    Can someone please update the web interface to either completely exclude the linux default if I am stuck using OVS or better yet UPDATE THE DOCUMENTATION TO INCLUDE EVERYTHING SINGLE SCENARIO.

    I want UNTAGGED traffic to manage the Proxmox box. Everything else (everything tagged) CAN NOT sign in or see the web interface...at all.

    The configuration sample in the official link doesn't do this. It assigned a VLAN and leaves everything else. I tried reversing that scenario and it still didn't work. This isn't rocket science. If you can't have a web interface that works, just remove it all together. (if you think the web interface works, try creating a new interface with LINUX (non ovs) ...you can't because it's not a drop down. Which ALSO means that every single time a client gets assigned a new VLAN I need to restart the host?! Bad design.
     
  6. Scott Zupek

    Scott Zupek New Member
    Proxmox Subscriber

    Joined:
    Nov 5, 2018
    Messages:
    18
    Likes Received:
    1
    Code:
    auto lo
    iface lo inet loopback
    
    iface eno1 inet manual
    
    iface eno2 inet manual
    
    iface eno1v201 inet manual
    
    iface eno2v201 inet manual
    
    auto bond0
    iface bond0 inet manual
            bond-slaves eno1 eno2
            bond-miimon 100
            bond-mode active-backup
    
    auto bond0201
    iface bond0201 inet manual
            bond-slaves eno1v201 eno2v201
            bond-miimon 100
            bond-mode active-backup
            vlan-raw-device bond0
    
    auto vmbr0
    iface vmbr0 inet static
            address  10.21.32.100
            netmask  255.255.255.0
            gateway  10.21.32.254
            bridge-ports bond0
            bridge-stp off
            bridge-fd 0
            bridge-vlan-aware yes
            bridge-vids 2-4094
    
    auto vmbr0201
    iface vmbr0201 inet manual
            bridge-ports bond0201
            bridge-stp off
            bridge-fd 0
            bridge-vlan-aware yes
            bridge-vids 2-4094
    
    I've assign the VM to vmbr0 and vmbr0201 neither can talk to the router and there is absolutely no traffic hitting it because it's being blocked by the Proxmox VE host and I do not use the software firewall built into Proxmox VE
     
  7. sb-jw

    sb-jw Active Member

    Joined:
    Jan 23, 2018
    Messages:
    484
    Likes Received:
    42
    Try to remove: "bridge-vlan-aware yes".
    Normally, if you have complex network config you do it over shell or use OVS.
    In my opinion, the option VLAN aware is not an option you would have in an virtual environment, you would tag the interfaces in your Hypervisor and not in the VM itself. This add an additional layer which you can not control in an easy way. I tried the VLAN aware one time for pfsense, it doesn't worked, all my VMs on this specified node are not reachable anymore. I think there was anything wrong, but I revert my changes and don't check it again.

    On my setup I always use the Shell, because PVE isnt able to do my Config. I create an Bonding device and assign multiple VLANs to bond devices and this I assign on the bridges where they get the IP addresses, and it works without problems.
     
  8. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,317
    Likes Received:
    132

    I really don't undestand your config,

    what are this interfaces ?

    iface eno1v201 inet manual
    iface eno2v201 inet manual



    If you use vlan in the vm configuration, you simply need:

    vlan aware
    ----------------
    Code:
    auto bond0
    iface bond0 inet manual
            bond-slaves eno1 eno2
            bond-miimon 100
            bond-mode active-backup
    
    auto vmbr0
    iface vmbr0 inet static
            address  10.21.32.100
            netmask  255.255.255.0
            gateway  10.21.32.254
            bridge-ports bond0
            bridge-stp off
            bridge-fd 0
            bridge-vlan-aware yes
            bridge-vids 2-4094
    
    (in this case, if you have vm with vlan 201 in the vm config, on vm start,
    proxmox will add vlan tag directly on the vmbr0 port. you can see it with
    "bridge -c vlan show" )



    vlan unaware:
    ----------------------
    Code:
    auto bond0
    iface bond0 inet manual
            bond-slaves eno1 eno2
            bond-miimon 100
            bond-mode active-backup
    
    auto vmbr0
    iface vmbr0 inet static
            address  10.21.32.100
            netmask  255.255.255.0
            gateway  10.21.32.254
            bridge-ports bond0
            bridge-stp off
            bridge-fd 0
     
    (in this case, if you have vm with vlan 201 in the vm config, on vm start,
    proxmox will create a new vmbr0v201 + bond0.201. (Not added in /etc/network/interfaces, but you can see it with brctl show)





    If you don't want to manage vlan config in the vm, you can manually create a vmbr with an interface in the vlan

    Code:
    auto bond0
    iface bond0 inet manual
            bond-slaves eno1 eno2
            bond-miimon 100
            bond-mode active-backup
    
    
    auto vmbr0
    iface vmbr0 inet static
            address  10.21.32.100
            netmask  255.255.255.0
            gateway  10.21.32.254
            bridge-ports bond0
            bridge-stp off
            bridge-fd 0
    
    auto vmbr1
    iface vmbr1 inet manual
            bridge-ports bond0.201
            bridge-stp off
            bridge-fd 0
    
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,317
    Likes Received:
    132
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. Scott Zupek

    Scott Zupek New Member
    Proxmox Subscriber

    Joined:
    Nov 5, 2018
    Messages:
    18
    Likes Received:
    1
    Thank you for the response.

    RESOLVED: Out of all the things it ended up being, it was a faulty switch configuration, NOT my /etc/network/interfaces configuration. To answer your question, I was trying everything under the sun to get it work, but in all honesty, the OUT OF BOX solution works just fine.

    I was able to determine the problem by trying to ping the switch vlan interface to the firewall and wasn't getting a response. Then, there was a hardware firewall issue (SonicWALL) preventing pings on the vlan interface after that (duh!)

    Once I corrected the switch problem the GUI default option worked and for the sake of reference, here is my currently running network config. It should look very familiar as it's all completed in the GUI.
    /etc/network/interfaces
    Code:
    #cat /etc/network/interfaces
    auto lo
    iface lo inet loopback
    
    iface eno1 inet manual
    
    iface eno2 inet manual
    
    auto bond0
    iface bond0 inet manual
            bond-slaves eno1 eno2
            bond-miimon 100
            bond-mode active-backup
    
    auto vmbr0
    iface vmbr0 inet static
            address  10.21.32.100
            netmask  255.255.255.0
            gateway  10.21.32.254
            bridge-ports bond0
            bridge-stp off
            bridge-fd 0
            bridge-vlan-aware yes
            bridge-vids 2-4094
    
    and the brctl show command (not that I adjusted anything)
    Code:
    #brctl show
    bridge name     bridge id               STP enabled     interfaces
    vmbr0           8000.d094669029f5       no              bond0
                                                            tap100i0
                                                            tap101i0
                                                            tap102i0
    
     
  11. tebuu

    tebuu New Member

    Joined:
    Mar 25, 2019
    Messages:
    1
    Likes Received:
    0

    @Scott Zupek
    Would you please share your network switch config or what's issue with it? My Proxmox VE host(s) were configuring similar to yours. However, the VM still is not working.
    Anything I need to change on the switch to make it works.

    RESOLVED:
    Here's network config on the switch.

    #show run int po 10
    interface Port-Channel10
    load-interval 30
    switchport trunk native vlan 2004
    switchport trunk allowed vlan 2001-2005
    switchport mode trunk
    ipv6 access-group rogue-ipv6 in
    storm-control broadcast level 0.5
    spanning-tree portfast
    spanning-tree bpduguard enable

    Of course, you MUST to enter "VLAN Tag" ex: (2001) when creating a new VM for the other VLAN.
     
    #11 tebuu, Mar 26, 2019
    Last edited: Mar 26, 2019
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice