Multiple node networking

[dpearceFL]

I have three Proxmox 8.1.4 nodes. I have one VM (VM1) on node 1 and one VM (VM2) on node 2. I need VM1 to talk to VM2. Each VM is using a network bridge called "vmbr5" defined at both node levels.

Using the SDN functionality, I know using a "Simple" Zone is not the way to go. I am now trying VLAN but without any luck.

Under "Zones", I created a VLAN using vmbr5 as the bridge. All of the other fields left with their defaults. I named this Zone vlan5.

Under "VNets", I hit create and named it vnet5 and used vlan5 as the zone. I add a Tag number and hit OK.

Under SDN I hit apply and all three nodes now have vlan5 listed.

(SELinux and Firewall is disabled on each VM.)

I can not ping VM1 from VM2. What am I doing wrong?

 

[louie1961]

What kind of networking are you using to connect the nodes? Is there a switch involved? if yes, is it a managed switch? An unmanaged switch will strip off the VLAN headers making it impossible for the nodes to communicate properly.

 

[dpearceFL]

No switch between the two nodes. I was trying to establish a network connection between the two VMs internal to the Datacenter.

 

[louie1961]

so just an ethernet cable between the two machines? You said they were separate nodes, so I am assuming two different physical machines, one for each node, is that correct?

 

[dpearceFL]

The VMs are on the vmbr5 network.

 

[dpearceFL]

Maybe I'm thinking Proxmox SDN is capable of doing something it can't do.

I need each VM (on different physical nodes) to be able to connect to each other through a common network.

Each Proxmox node has vmb0 with it's own IP address (external). Each of the Proxmox nodes can talk to each other over the private network. Both nodes are part of a cluster (at the Datacenter level).

Can I create a private network for these two VMs, using a VLAN tag over the vmbr0 connections on the Proxmox hosts?

 

[louie1961]

Right, but until you tell us more about how the physical nodes are connected, it is all a big black box. paste in the contents of your /etc/network/interfaces file here for both machines in question, and tell us about how the NICs for the two machines are configured/connected.

 

[dpearceFL]

Both nodes are connected to a switch via a single NIC. Both use vmbr0. The VMs are connected using vmbr5.

Proxmox node 1

auto vmbr0
iface vmbr0 inet static
        address 172.20.21.229/24
        gateway 172.20.21.1
        bridge-ports enp1s0f0
        bridge-stp off
        bridge-fd 0

auto vmbr5
iface vmbr5 inet static
        address 10.1.1.0/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*

Proxmox node 2

auto vmbr0
iface vmbr0 inet static
        address 172.20.21.228/24
        gateway 172.20.21.1
        bridge-ports enp1s0f0
        bridge-stp off
        bridge-fd 0

auto vmbr5
iface vmbr5 inet static
        address 10.1.1.0/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*

 

[louie1961]

Is that an excerpt from the /etc/network/interfaces file(s) or the entire contents? If it is the entire contents you do not have any actual ethernet NICs configured. As an example this is from one of my proxmox nodes. I have changed the IP addresses for obvious reasons. Note the sections I highlighted in red that you don't seem to have. PLUS, you can't have "brigde-ports none" for vmbr5. You need to have some connection from vmbr5 to an ethernet NIC, either an actual or a virtual NIC. In my example below, I have created two bridges, an actual bridge (vmbr0) and a virtual bridge tied to a VLAN (vmbr0.100) all on the same NIC


# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
bridge-ports enp3s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4092

auto vmbr0.100
iface vmbr0.100 inet static
address 10.10.10.2/24
gateway 10.10.10.1

 

[dpearceFL]

I removed the comments and the NICS that are not being used. otherwise the file is unedited.

So is it possible to have a private network across multiple Proxmox nodes without another physical NIC and an external switch?

 

[louie1961]

Potentially, but you would need to check the documentation. I have done this on my pfSense box, but I haven't done it on Proxmox yet, so its a bit of a guess on my part. I would imagine you can create virtual NICs like "enp1s0f0.1", "enp1s0f0.2" etc. and then you would assign those as bridge ports to your vmbr0 and vmbr5. Again, maybe someone with more smarts or experience than I have will chime in or you can find an example in the Proxmox documentation

 

[louie1961]

something like this could be done. Notice how virtual interface eno1.5 is created.

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno1.5 inet manual

auto vmbr0v5
iface vmbr0v5 inet static
address 10.10.10.2/24
gateway 10.10.10.1
bridge-ports eno1.5
bridge-stp off
bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0





Check the documentation here: https://pve.proxmox.com/wiki/Network_Configuration