Proxmox Ceph Cluster - VLAN Tagged VM's not Communicating

HomelabHobbyistSK

New Member
Oct 4, 2021
6
1
3
Canada
Hey Everyone,

I am slowly reaching the end of my Ceph Cluster project and the absolute last thing I need to accomplish is setting up my PVE so that I can create VM's for the different VLAN's in my existing environment. So far I can confirm I can create VM's and have them migrate successfully, but I have hit the wall when it comes to getting them to communicate on my existing network. Below is a quick rundown of how my host's networks are configured.

Network Setup:
  • Ethernet Port 1:Management Port on Management VLAN (10.10.100.0/24 - eno1)
    • Proxmox Gateway set to 10.10.100.1
  • Ethernet Port 2: Corosync Ring 1 (Connected to Switch #1 to act as a heartbeat) (172.16.1.0/24 - eno2)
  • Ethernet Port 3: Corosync Ring 2 (Connected to Switch #2 to act as a backup heartbeat) (172.16.2.0/24 - eno3)
  • 10GB Port 1: Proxmox Public Network (Ceph Public Network / LAN Access) (10.10.150.0/24 - enp4s0)
  • 10GB Port 2: Proxmox Private Network (Ceph Replication) (172.16.100.0/24- enp4s0d1)
Code:
auto vmbr0
iface vmbr0 inet static
        address 10.10.100.111/24
        gateway 10.10.100.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
#Management Interface

auto vmbr1
iface vmbr1 inet static
        address 172.16.1.111/24
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
        mtu 1500
#CoroSync 1

auto vmbr2
iface vmbr2 inet static
        address 172.16.2.111/24
        bridge-ports eno3
        bridge-stp off
        bridge-fd 0
        mtu 1500
#CoroSync 2

auto vmbr3
iface vmbr3 inet static
        address 10.10.150.11/24
        bridge-ports enp4s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#Ceph Public

auto vmbr4
iface vmbr4 inet static
        address 172.16.100.111/24
        bridge-ports enp4s0d1
        bridge-stp off
        bridge-fd 0
#Ceph Replication

Right now, the only two VLAN tags I need to apply are for VLAN 150 (10.10.150.0/24 - "Internal Servers") and VLAN 200 (10.10.200.0/24 - "External/Public Servers"). I have confirmed that the switch port going to the 10GB port is properly detecting VLAN 150 and VLAN 200 tags if I connect a separate device straight to the switch port, but the issue seems to be with my Proxmox Ceph host.

I am really new to Ceph and may have misinterpreted what the "Public" network is properly used. I initially believed this to be used for not only Ceph <-> VM OSD traffic, but also act as a connection to the LAN (Which is why I have the public IP assigned to the Bridge on all my hosts within my "Internal Servers" VLAN). Would it make more sense to have the Ceph Public Network IP's in their own separate subnet? I also tried turning on "VLAN Aware Bridge" and sadly that did not get me anywhere.

I have created a test Ubuntu VM to confirm that each VLAN I assign it to will grab the correct DHCP Address from my firewall/router, and here is what I have noticed:
  • When I set it's VirtIO interface to use Vmbr3 (the Proxmox Ceph Cluster Public Network) and configure it to use a VLAN ID of 200, the VM fails to grab an address and it will not even register the MAC Address of the virtual interface on my switch
  • If I remove the VLAN Tag from the VM, the VM receives an IP based on the VLAN configured as the "Default VLAN ID" on my Mikrotik Switch for the port Vmbr3 is bridged to (enp4s0 with the untagged VLAN set to 150) and gains the ability to talk to the network and reach out to the Internet.
Could I have potentially configure my interfaces incorrectly? I am still learning PVE and tried messing about with creating some Linux VLAN's, but sadly none of it worked :(. I am just not sure if maybe it is because my gateway was autoconfigured on interface vmbr0, or if I somehow need to create multiple sub-interfaces on vmbr3 to support both the Public Network and a sub-interface to handle the various VLAN tagged traffic of my VM's.

If anyone could steer me in the right direction and help me get back on track I would greatly appreciate it!
 
Last edited:
Assuming that VLAN 200 is tagged or allowed on the switch interface, have you set an ip-helper address on VLAN 200 to allow dhcp requests to find the dhcp server?

I would also try assigning a static IP in correct subnet for VLAN 200 and verify that the VM can reach it's gateway address and any other client in the same VLAN
 
DHCP Server is hosted on the VLAN 200 Default Gateway. I have tried statically setting the IP Address to one in VLAN 200, but even when it is all configured the MAC Address for the VirtiIO Network Interface on the VM doesn't even populate on the 10GB Mikrotik switch it is directly connected to.

May have to steal a SFP+ -> RJ45 connector from work and try some more testing with my laptop
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!