Connections to PVE in VLAN timeout

[TheHellSite]

I would like to have my PVE available only via its VLAN_SRV_IP "10.10.73.10". Sadly when accessing it via SSH/SCP/HTTPS on that VLAN/IP the connection times out after about 30 seconds.
When I access it on the untagged LAN IP "10.10.31.10" I don't get these timeouts. However, it can't be an issue with my OPNsense since the access to my TrueNAS VM (SSH/HTTPS) installed on the same PVE never times out!

Here is my network config.

Tagged VLAN
===========
10.10.31.xx Client PCs --> 10.10.73.10 PVE = Timeouts
10.10.31.xx Client PCs --> 10.10.73.20 TrueNAS = NO Timeouts

Untagged LAN
============
10.10.31.xx Client PCs --> 10.10.31.10 PVE = NO Timeouts

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp3s0 inet manual

iface enp1s0 inet manual

iface enp2s0 inet manual

iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp1s0
    bridge-stp off
    bridge-fd 0
#WAN

auto vmbr1
iface vmbr1 inet manual
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0
#MODEM

auto vmbr2
iface vmbr2 inet static
    address 10.10.31.10/24
    bridge-ports enp3s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
#LAN

auto vmbr3
iface vmbr3 inet static
    address 10.10.1.1/24
    bridge-ports enp4s0
    bridge-stp off
    bridge-fd 0
#MGMT

auto vmbr2.73
iface vmbr2.20 inet static
    address 10.10.73.10/24
    gateway 10.10.73.1
#VLAN_SRV

 

[gurubert]

The response packets to your clients in 10.10.31.0/24 will be sent out via vmbr2 with a sender address of 10.10.31.10. The client that receives the answer will not be able to deal with it as it has sent the request to 10.10.73.10.

You need to use policy routing to solve this multi homed server issue. Or tell your clients to always connect to 10.10.31.10 when they want to talk to Proxmox.

 

[TheHellSite]

The response packets to your clients in 10.10.31.0/24 will be sent out via vmbr2 with a sender address of 10.10.31.10.

If I remove the interface address "10.10.31.10/24" from vmbr2 the issue is still present, even though PVE shouldn't be able anymore to directly speak to my "10.10.31.x" clients.

The client that receives the answer will not be able to deal with it as it has sent the request to 10.10.73.10.

You need to use policy routing to solve this multi homed server issue.

Why is it working for all the other services that are in a different VLAN than my LAN clients?
I mean my OPNsense is already doing the "translation" between my server net and client net.
It is just Proxmox that is behaving odd!

 

[gurubert]

You can check with "ip route get 10.10.31.X" on your Proxmox node which route the packets will use when travelling to your client X.

 

[TheHellSite]

interface address "10.10.31.10/24" removed from vmbr2

10.10.31.208 via 10.10.73.1 dev vmbr2.20 src 10.10.73.10 uid 0
    cache

interface address "10.10.31.10/24" on vmbr2

10.10.31.208 via 10.10.31.1 dev vmbr2.20 src 10.10.31.10 uid 0
    cache

Oh man! It seems that I oversaw something when I first removed it. (possibly apply)....
Anyway it is working now.