Connection refused on NAT but happens randomly

[strboul]

Hello! I have a problem with my private network setup I have searched the Internet and forum extensively, I haven't yet found a solution. It's a bit complex to tell but here's my situation. I'd really appreciate if someone helps me out.

---

On my /etc/network/interfaces file, I set up a private network:

auto lo
iface lo inet loopback

iface enp2s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.10.197/24
        gateway 192.168.10.1
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.10.10.1/24
        gateway 10.10.10.1
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
        post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

And I created some LXC containers with IPs from vmbr1. I cannot reach those services often times. It just does happen randomly and have no idea. Usually the subsequent requests come through but this event creates a very bad annoyance/issues in my setup.

When I try to connect to a LXC container (only has a vmbr1 IP) through a jump host (it has two bridges, vmbr0 and vmbr1) from my machine:

ssh -J root@192.168.10.11 root@10.10.10.152
# channel 0: open failed: connect failed: Connection refused
# stdio forwarding failed
# Connection closed by UNKNOWN port 65535

# 1 second later

ssh -J root@192.168.10.11 root@10.10.10.152
# Welcome to Ubuntu 22.04 LTS (GNU/Linux 6.5.11-7-pve x86_64)
# ...

# suddenly works!?

Every 1 out of 10 connection drops like that. But it happens more frequent when I leave more time between the requests.

I also realized this, (I don't know if relevant though), when I try to traceroute an LXC container from the pve host:

root@pve:~# traceroute 10.10.10.186
traceroute to 10.10.10.186 (10.10.10.186), 30 hops max, 60 byte packets
 1  10.10.10.186 (10.10.10.186)  0.072 ms * *
root@pve:~# traceroute 10.10.10.186
traceroute to 10.10.10.186 (10.10.10.186), 30 hops max, 60 byte packets
 1  10.10.10.186 (10.10.10.186)  0.074 ms * *
root@pve:~# traceroute 10.10.10.186
traceroute to 10.10.10.186 (10.10.10.186), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * 10.10.10.186 (10.10.10.186)  0.047 ms  0.013 ms
# The last one took longer to pass, for some reason?

I don't see any logs in the firewall when this happens. Though, if I turn off the firewall everywhere, this still happens.

Could you help with this issue?