Hey Everyone,
I am slowly reaching the end of my Ceph Cluster project and the absolute last thing I need to accomplish is setting up my PVE so that I can create VM's for the different VLAN's in my existing environment. So far I can confirm I can create VM's and have them migrate successfully, but I have hit the wall when it comes to getting them to communicate on my existing network. Below is a quick rundown of how my host's networks are configured.
Network Setup:
Right now, the only two VLAN tags I need to apply are for VLAN 150 (10.10.150.0/24 - "Internal Servers") and VLAN 200 (10.10.200.0/24 - "External/Public Servers"). I have confirmed that the switch port going to the 10GB port is properly detecting VLAN 150 and VLAN 200 tags if I connect a separate device straight to the switch port, but the issue seems to be with my Proxmox Ceph host.
I am really new to Ceph and may have misinterpreted what the "Public" network is properly used. I initially believed this to be used for not only Ceph <-> VM OSD traffic, but also act as a connection to the LAN (Which is why I have the public IP assigned to the Bridge on all my hosts within my "Internal Servers" VLAN). Would it make more sense to have the Ceph Public Network IP's in their own separate subnet? I also tried turning on "VLAN Aware Bridge" and sadly that did not get me anywhere.
I have created a test Ubuntu VM to confirm that each VLAN I assign it to will grab the correct DHCP Address from my firewall/router, and here is what I have noticed:
. I am just not sure if maybe it is because my gateway was autoconfigured on interface vmbr0, or if I somehow need to create multiple sub-interfaces on vmbr3 to support both the Public Network and a sub-interface to handle the various VLAN tagged traffic of my VM's.
If anyone could steer me in the right direction and help me get back on track I would greatly appreciate it!
I am slowly reaching the end of my Ceph Cluster project and the absolute last thing I need to accomplish is setting up my PVE so that I can create VM's for the different VLAN's in my existing environment. So far I can confirm I can create VM's and have them migrate successfully, but I have hit the wall when it comes to getting them to communicate on my existing network. Below is a quick rundown of how my host's networks are configured.
Network Setup:
- Ethernet Port 1:Management Port on Management VLAN (10.10.100.0/24 - eno1)
- Proxmox Gateway set to 10.10.100.1
- Ethernet Port 2: Corosync Ring 1 (Connected to Switch #1 to act as a heartbeat) (172.16.1.0/24 - eno2)
- Ethernet Port 3: Corosync Ring 2 (Connected to Switch #2 to act as a backup heartbeat) (172.16.2.0/24 - eno3)
- 10GB Port 1: Proxmox Public Network (Ceph Public Network / LAN Access) (10.10.150.0/24 - enp4s0)
- 10GB Port 2: Proxmox Private Network (Ceph Replication) (172.16.100.0/24- enp4s0d1)
Code:
auto vmbr0
iface vmbr0 inet static
address 10.10.100.111/24
gateway 10.10.100.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
#Management Interface
auto vmbr1
iface vmbr1 inet static
address 172.16.1.111/24
bridge-ports eno2
bridge-stp off
bridge-fd 0
mtu 1500
#CoroSync 1
auto vmbr2
iface vmbr2 inet static
address 172.16.2.111/24
bridge-ports eno3
bridge-stp off
bridge-fd 0
mtu 1500
#CoroSync 2
auto vmbr3
iface vmbr3 inet static
address 10.10.150.11/24
bridge-ports enp4s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#Ceph Public
auto vmbr4
iface vmbr4 inet static
address 172.16.100.111/24
bridge-ports enp4s0d1
bridge-stp off
bridge-fd 0
#Ceph ReplicationRight now, the only two VLAN tags I need to apply are for VLAN 150 (10.10.150.0/24 - "Internal Servers") and VLAN 200 (10.10.200.0/24 - "External/Public Servers"). I have confirmed that the switch port going to the 10GB port is properly detecting VLAN 150 and VLAN 200 tags if I connect a separate device straight to the switch port, but the issue seems to be with my Proxmox Ceph host.
I am really new to Ceph and may have misinterpreted what the "Public" network is properly used. I initially believed this to be used for not only Ceph <-> VM OSD traffic, but also act as a connection to the LAN (Which is why I have the public IP assigned to the Bridge on all my hosts within my "Internal Servers" VLAN). Would it make more sense to have the Ceph Public Network IP's in their own separate subnet? I also tried turning on "VLAN Aware Bridge" and sadly that did not get me anywhere.
I have created a test Ubuntu VM to confirm that each VLAN I assign it to will grab the correct DHCP Address from my firewall/router, and here is what I have noticed:
- When I set it's VirtIO interface to use Vmbr3 (the Proxmox Ceph Cluster Public Network) and configure it to use a VLAN ID of 200, the VM fails to grab an address and it will not even register the MAC Address of the virtual interface on my switch
- If I remove the VLAN Tag from the VM, the VM receives an IP based on the VLAN configured as the "Default VLAN ID" on my Mikrotik Switch for the port Vmbr3 is bridged to (enp4s0 with the untagged VLAN set to 150) and gains the ability to talk to the network and reach out to the Internet.
. I am just not sure if maybe it is because my gateway was autoconfigured on interface vmbr0, or if I somehow need to create multiple sub-interfaces on vmbr3 to support both the Public Network and a sub-interface to handle the various VLAN tagged traffic of my VM's.If anyone could steer me in the right direction and help me get back on track I would greatly appreciate it!
Last edited: