network configuration on new cluster

R

Reartu24

New Member
Jul 9, 2024
3
0
1
good afternoon everybody
I am preparing to migrate my infrastructure from VMware to Proxmox.
right now I have a network card that is exposed to the LAN (with ip 192.168.4.x) and an internal LAN for all the VMs (10.150.0.0/20),
my idea is to have 2-3 Vm on the WAN network (192.168.4.x) and then all the production in the 10.150.0.0/20 network.

At the moment I have installed PFsense as a firewall and configured with a card on vtnet0 (network 192.168) and a card on the OVS bridge that I created (with ip 10.150.0.0/20),
Up to now everything is working for me, but if I move a test VM from node 1 to node 2, DHCP can no longer reach me and therefore loses the address), where am I wrong?
 
Last edited:
I take it that first LAN is a typo and is supposed to be WAN (of course with a LAN range for privacy sake?)? ;)

Also, which VM are you moving and there is the DHCP running?
Is the PFsense the VM you're moving and are guests outside of the system not getting new DHCP-leases, or is your PFSense external and is a test-VM that you're moving losing connection?
Also, if configured with static IP's, can you still ping from/to the moved VM?

And could you post the result of these 2 commands of both hosts to see if something looks wrong there?
Code: 
ip a
cat /etc/network/interfaces
 
Hi,
yes, the ip 192 is WAN for pfsense.
my private network is 192.168.4.x/24, and I would like all servers and services to be in another subnet (10.150.0.0/20), pfsense is a VM in the node 1 and have 2 lan attached,
vmbr0 on fisical network ( for me in internal LAN, but for pfsense is WAN)
OVE_Bridge: is the private network for my server and service ( for pfsense is LAN where i have activate the dhcp )

i try to do a ping and don't work,
i have do the commando on host 01
First ip a (node 1)
Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether d4:ae:52:b5:44:bc brd ff:ff:ff:ff:ff:ff
    altname enp2s0f0
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether d4:ae:52:b5:44:bd brd ff:ff:ff:ff:ff:ff
    altname enp2s0f1
    inet6 fe80::d6ae:52ff:feb5:44bd/64 scope link
       valid_lft forever preferred_lft forever
4: enp8s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:10:18:e4:64:60 brd ff:ff:ff:ff:ff:ff
5: enp8s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:10:18:e4:64:61 brd ff:ff:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d4:ae:52:b5:44:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.4.21/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::d6ae:52ff:feb5:44bc/64 scope link
       valid_lft forever preferred_lft forever
70: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr103i0 state UNKNOWN group default qlen 1000
    link/ether a2:4e:c3:60:99:0e brd ff:ff:ff:ff:ff:ff
71: fwbr103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8a:d4:2e:2c:8d:8b brd ff:ff:ff:ff:ff:ff
72: fwpr103p0@fwln103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether ce:83:11:15:31:92 brd ff:ff:ff:ff:ff:ff
73: fwln103i0@fwpr103p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr103i0 state UP group default qlen 1000
    link/ether 8a:d4:2e:2c:8d:8b brd ff:ff:ff:ff:ff:ff
74: tap103i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr103i1 state UNKNOWN group default qlen 1000
    link/ether e2:99:52:37:41:18 brd ff:ff:ff:ff:ff:ff
75: fwbr103i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e2:99:52:37:41:18 brd ff:ff:ff:ff:ff:ff
77: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr102i0 state UNKNOWN group default qlen 1000
    link/ether 46:03:ed:46:bd:58 brd ff:ff:ff:ff:ff:ff
78: fwbr102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 42:ff:32:86:28:ee brd ff:ff:ff:ff:ff:ff
79: fwpr102p0@fwln102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 8a:4c:5e:7a:39:c8 brd ff:ff:ff:ff:ff:ff
80: fwln102i0@fwpr102p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr102i0 state UP group default qlen 1000
    link/ether 42:ff:32:86:28:ee brd ff:ff:ff:ff:ff:ff
81: tap102i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr102i1 state UNKNOWN group default qlen 1000
    link/ether be:dd:36:27:92:38 brd ff:ff:ff:ff:ff:ff
82: fwbr102i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether be:dd:36:27:92:38 brd ff:ff:ff:ff:ff:ff
94: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 72:ff:9c:5c:a0:41 brd ff:ff:ff:ff:ff:ff
95: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ee:52:84:3b:7b:40 brd ff:ff:ff:ff:ff:ff
    inet 10.150.0.0/20 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::ec52:84ff:fe3b:7b40/64 scope link
       valid_lft forever preferred_lft forever
99: veth10001i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr10001i0 state UP group default qlen 1000
    link/ether fe:a3:d4:c5:88:bd brd ff:ff:ff:ff:ff:ff link-netnsid 0
100: fwbr10001i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:a3:d4:c5:88:bd brd ff:ff:ff:ff:ff:ff
101: fwln10001o0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr10001i0 state UNKNOWN group default qlen 1000
    link/ether 02:e3:e1:98:42:ff brd ff:ff:ff:ff:ff:ff


second command (cat /etc/network/interfaces)

Code: 
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

iface enp8s0f0 inet manual

iface enp8s0f1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.4.21/24
        gateway 192.168.4.101
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.150.0.0/20
        ovs_type OVSBridge

source /etc/network/interfaces.d/*

Node 2
ip a
Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether d4:ae:52:b5:43:78 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f0
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether d4:ae:52:b5:43:79 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f1
    inet6 fe80::d6ae:52ff:feb5:4379/64 scope link
       valid_lft forever preferred_lft forever
4: enp8s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:10:18:e4:64:78 brd ff:ff:ff:ff:ff:ff
5: enp8s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:10:18:e4:64:79 brd ff:ff:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d4:ae:52:b5:43:78 brd ff:ff:ff:ff:ff:ff
    inet 192.168.4.22/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::d6ae:52ff:feb5:4378/64 scope link
       valid_lft forever preferred_lft forever
14: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0a:53:31:38:94:45 brd ff:ff:ff:ff:ff:ff
15: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d4:ae:52:b5:43:79 brd ff:ff:ff:ff:ff:ff
    inet 10.150.0.0/20 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::f84d:92ff:fe7c:d641/64 scope link
       valid_lft forever preferred_lft forever
31: veth100i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:17:b3:c3:bb:a3 brd ff:ff:ff:ff:ff:ff link-netnsid 0

second command (cat /etc/network/interfaces)
Code: 
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

iface enp8s0f0 inet manual

iface enp8s0f1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.4.22/24
        gateway 192.168.4.101
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.150.0.0/20
        ovs_type OVSBridge

source /etc/network/interfaces.d/*

thank you so much
 
Last edited:
Looks like you haven't connected a physical port (and optional VLAN) to the vmbr1's, so the two OVSBridge's act like switches only inside of the proxmox-host itself, and not outside it / to other hosts.
Do note, I only work with Linux Bridges myself, but a quick glance through the documentation [1] also shows physical ports attached to it.
I see you still have eno2 free, if you don't want to go over the existing network (and the servers are close-by eachother), just connect a network-cable between the two eno2 ports, and set the eno2 port as the bridge port for both.

[1] https://pve.proxmox.com/wiki/Open_vSwitch#Examples

Edit: Typo's and such
 
Last edited:
oh goodness, I'm lost on the stupidest thing.....
I thought it worked like VMWare and distributed switches, which don't require attached cables, I'm at the office on Friday and I attach them to a switch (there are 3 nodes so I have to use the switch) and I create an isolated Vlan just for this service and I should have Resolved.
Thank you so much
 
No problem and good luck/enjoy.

Also as an alternative / for future reference:
If you can set (tagged) vlan's remotely (leave untagged as-is of course for access) on the switches it is already connected to, you can also;
- just re-use vmbr0 and select a VLAN-Tag in the network-card configuration
or (what I personally use)
- use SDN, create a vlan-zone with the bridge set to vmbr0 and on that create a vnet with the correct VLAN. You can then select that vnet on your network-interface config (which also helps to give the different network-segments a name and you can't forget to set the VLAN for new VM's).

Edit: Added note about tag/untag just-in-case
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back