[SOLVED] Dropped packets from proxmox with two nics

[duindain]

I'm trying to work out why i have many dropped packets between two network cards on the same host pc

I have a proxmox bare metal pc with 2 nics

This is a mockup of my network topology


The 192.168.3.0 network is not meant to access the internet
The 192.168.2.0 network is meant to access 192.168.3.0 and the internet

This is proxmox's /etc/network/interfaces

ip route show table main
default via 192.168.2.1 dev vmbr0 proto kernel onlink
192.168.2.0/24 dev vmbr0 proto kernel scope link src 192.168.2.8
192.168.3.0/24 dev vmbr1 proto kernel scope link src 192.168.3.1

I have a static route in the router to direct any traffic on the lan interface (192.168.2.0) that is trying to go to the 192.168.3.0 network to use a gateway of 192.168.2.8

I have installed kea-dhcp4 on the proxmox host its set to be authorative for 192.168.3.0 network
it has specified host entries for some devices and an ip range of 2-99 for dhcp

When i test the network using mtr I'm getting a substantial number of dropped packets
When i test with a pc that is on the 192.168.3.0 network i get no dropped packets

This is the mtr results the ip target i was testing % of dropped packets and the number is how many packets i tested

Privileged LXE that has access to both cards ip 192.168.2.71
192.168.3.101 - 3.3% - 450 packets
192.168.3.102 - 4.7% - 450 packets
192.168.3.103 - 3.5% - 450 packets
192.168.3.104 - 2.7% - 600 packets
192.168.3.105 - 3.0% - 2641 packets
192.168.3.106 - 3.6% - 775 packets
192.168.3.2   - 3.6% - 825 packets
192.168.3.1   -   0% - 550 packets

Proxmox server ip 192.168.2.8
192.168.3.101 - 81.7% - 450 packets
192.168.3.102 - 86.3% - 569 packets
192.168.3.103 - 86.5% - 2620 packets
192.168.3.104 - 84.1% - 775 packets
192.168.3.105 - 86.5% - 820 packets
192.168.3.106 - 85.4% - 625 packets
192.168.3.2   - 73.9% - 450 packets
192.168.3.1   -    0% - 1215 packets

Physical pc with lan connection to 192.168.3.0 switch ip 192.168.3.2
192.168.3.101 - 0% - 4000 packets
192.168.3.102 - 0% - 4000 packets
192.168.3.103 - 0% - 4000 packets
192.168.3.104 - 0% - 4000 packets
192.168.3.105 - 0% - 4000 packets
192.168.3.106 - 0% - 4000 packets
192.168.3.1   - 0% - 650 packets

So i think the pc wired into the switch is showing there's no layer 2 issues, cabling issues and whatnot

Proxmox and the LXE getting different results is very weird they have the same level of access

Anyway does anyone know what to try next? i guess I'm missing some setting to allow the 2 nics to talk between each other
Though why it works some or most of the time is very odd too

I could try setting 192.168.3.0 to have a gateway to 192.168.2.1 then block from the router instead which might resolve it but when i try adding that gateway in the proxmox ui it prevents it saying "Default gateway already exists on interface 'vmbr0'"

 

[ce3rd]

normally, you have to enable ip forwarding for such a setup, what does this command say?

cat /proc/sys/net/ipv4/ip_forward

if it is 0 then I'm with you here "Though why it works some or most of the time is very odd too"

 

[duindain]

Thanks for helping @ce3rd

This is what i get running that command from the proxmox console
cat /proc/sys/net/ipv4/ip_forward
0

 

[ce3rd]

should be 1 - you can set it like in

https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_network_routed

 

[duindain]

Not quite sure im doing the right thing here but ive added the ipforwarding and arp for vmbr0

auto lo
iface lo inet loopback

auto enp8s0
iface enp8s0 inet manual

auto enp9s0
iface enp9s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.2.8/24
    gateway 192.168.2.1
    bridge-ports enp8s0
    bridge-stp off
    bridge-fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/vmbr0/proxy_arp

auto vmbr1
iface vmbr1 inet static
    address 192.168.3.1/24
    bridge-ports enp9s0
    bridge-stp off
    bridge-fd 0

source /etc/network/interfaces.d/*

cat /proc/sys/net/ipv4/ip_forward is 1 as is proxy_arp

I've defined two firewall rules


Im still getting 50% dropped packets using mtr to a device in the .3 network from the .2 network

 

[duindain]

Ok ive made a bit of progress here

I've removed my gui firewall rules in the proxmox server

In the LXE
Ive added
iptables -A FORWARD -i eth0 -o eth1 -s 192.168.2.0/24 -d 192.168.3.0/24 -j ACCEPT
iptables -A FORWARD -i eth1 -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -P FORWARD DROP

apt install iptables-persistent
iptables-save > /etc/iptables/rules.v4

So now in the LXE I get no packet loss to devices in the .3 network

In the Proxmox shell however i still get substantial packet loss even up to 95% sometimes
as well as other machines on the 2 network trying to contact the 3 network

I attempted adding these same rules for vmbr0 & 1 on the proxmox server and then dhcp requests got intercepted from the .3 network

 

[duindain]

Ok again a bit more progress

So since the LXE seems stable ive swapped more control over to it
I have a redirect on my router to direct any 192.168.3.0 traffic to the proxmox host 192.168.2.8
I've just changed that today to the LXE instead and it seems much more stable

I am still getting packet loss but less

I've also probably worked out why, the proxmox server has both network cards then a virtual bridge for both and an ip for the 3 network 192.168.3.1 on that virtual bridge
The LXE has both those virtual bridges added to it and also has the same ip for that network

When i check them though, the mac addresses are different between the LXE and the proxmox host for the same virtual bridge so i probably have duplicate ip addresses on the network which is probably causing some packets to hit the proxmox host and get lost and some to hit the LXE and get through

 

[duindain]

Ok this is solved now, thanks again for your help @ce3rd
Looks like forwarding and having that duplicate ip was the issue