[SOLVED] Proxmox System with 2 Pub IPs on seperate Hardware Ports

[Barcode-Snow]

At the moment, we have a hardware appliance at a vendor's data center. This has a network card with several ports. The provider has given us 2 public IP addresses. We now want to use one IP and thus also one port only to reach the Proxmox system. The other IP and Port should be used in a MASQUERADE/NAT setup, so that the VMs communicate over this to the outside and we separate management and production.
Currently the configuration looks like this:

[...]
iface enp1s0f0 inet manual

auto enp1s0f1
iface enp1s0f1 inet static
    address XXX.XXX.XXX.XX6/XX
    
auto vmbr0
iface vmbr0 inet static
        address XXX.XXX.XXX.XX1/XX
        gateway XXX.XXX.XXX.1
        bridge-ports enp1s0f0
        bridge-stp off
        bridge-fd 0
        
auto vmbr1
iface vmbr1 inet static
        address 10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp1s0f1 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp1s0f1 -j MASQUERADE

Now to the crazy part:

• I can reach the GUI of Proxmox via both IPs.
• If I ping with the interface enp1s0f1 I don't get an answer ping -I enp1s0f1 google.com -> 100% packet loss
• If I connect a VM to the vmbr1 it does not reach the internet, the Proxmox interface (10.10.10.1) in this network can be reached via ping.
• ip link show up is saying that the state is down enp1s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT
• ethtool enp1s0f1 -> Link detected: no
• /sys/class/net/enp1s0f1/operstate -> down
• ifup enp1s0f1 does not change the state of the interface

As I said: You can reach the GUI of Proxmox via both public IPs.

This is making me crazy

 

[_gabriel]

it's expected. PVE port 8006 listen on all network interfaces.

 

[Barcode-Snow]

And? The problem here is that I can access the GUI on an interface which, if you believe the system, is down. So, how can a system listen on an interface that is down? Furthermore, if the interface is indeed not down: why can’t VMs connected to vmbr1 reach the internet?

 

[_gabriel]

oh sorry, read too fast, don't see one port were connected.
I'm not expert but isn't ip_forward cause ? compare with echo 0 > /proc/sys/net/ipv4/ip_forward

 

[Barcode-Snow]

Configuration is based on the tips given in the Proxmox Admin Guide: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_flexible_networking

Here is how they did it:

auto eno1
#real IP address
iface eno1 inet static
        address  198.51.100.5/24
        gateway  198.51.100.1

auto vmbr0
#private sub network
iface vmbr0 inet static
        address  10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE

 

[_gabriel]

disabling temporary ip_forward is just to diagnose, it will break MASQUERADE/NAT as expected.

 

[Barcode-Snow]

This was after all a issue on the vendors site