For evpn, a different vrf with a different routing table is used for each zone, so I think it could work. (but I never tested it).
Do you have tested to enable 1exit-node (where this node have physical access to 192.168.10.1) + enable s-nat on the subnet ?
I'm not sure about how is working the conntrack with multiple vrf and same ips in different vrf.
Edit:
Found an interesting article
https://blog.oddbit.com/post/2023-02-19-vrf-and-nat/
Seem that it need to implement different conntrack zone. I don't think it's done currently.
(can you open a bugzilla.proxmox.com, I'll try to work on it in coming weeks)
I succeeded in solving this problem using conntrack. Here are my steps：
Test Environment：One PVE host（10.30.2.50），2 Zones：
- Zone1，65001，vnet1 65101，subnet1：10.10.22.1/24，VM1：10.10.22.10;
- Zone2，65001，vnet2 65102，subnet1：10.10.22.1/24，VM1：10.10.22.10; Subnet2：10.10.43.1/24，VM2：10.10.43.10
Step 0：create zones、vnets、subnets、vms，and enable exit node.
Step 1：Create IP 10.30.2.161 and 10.30.2.161
Code:
iface vmbr0:0 inet static
address 10.30.2.161/24
iface vmbr0:1 inet static
address 10.30.2.162/24
Step 2：Add the following Iptable rules：
Code:
iptables -t mangle -A PREROUTING -i vrf_z01 -s 10.10.22.10 -d 0.0.0.0/0 -j CONNMARK --set-mark 12
iptables -t mangle -A PREROUTING -s 0.0.0.0/0 -d 10.30.2.161 -j CONNMARK --set-mark 13
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t nat -A PREROUTING -m connmark --mark 13 -j DNAT --to-destination 10.10.22.10
iptables -t nat -A POSTROUTING -m connmark --mark 12 -j SNAT --to-source 10.30.2.161
ip rule add fwmark 12 lookup 1001 prio 102
ip rule add fwmark 13 lookup 1001 prio 103
iptables -t mangle -A PREROUTING -i vrf_z02 -s 10.10.22.10 -d 0.0.0.0/0 -j CONNMARK --set-mark 14
iptables -t mangle -A PREROUTING -s 0.0.0.0/0 -d 10.30.2.162 -j CONNMARK --set-mark 15
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t nat -A PREROUTING -m connmark --mark 15 -j DNAT --to-destination 10.10.22.10
iptables -t nat -A POSTROUTING -m connmark --mark 14 -j SNAT --to-source 10.30.2.162
ip rule add fwmark 14 lookup 1002 prio 104
ip rule add fwmark 15 lookup 1002 prio 105
Step 3：comment out the following codes generated by PVE SDN in frr.conf and run systemctl restart frr
Code:
frr version 8.5.1
frr defaults datacenter
hostname pvev
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_z01
vni 65001
# ip route 10.10.22.1/24 null0
# ip route 10.10.43.1/24 null0
# ip route 10.10.43.1/24 null0
exit-vrf
!
vrf vrf_z02
vni 65002
# ip route 10.10.22.1/24 null0
# ip route 10.10.43.1/24 null0
exit-vrf
!
vrf vrf_z03
vni 65003
# ip route 10.10.22.1/24 null0
# ip route 10.10.22.1/24 null0
# ip route 10.10.43.1/24 null0
exit-vrf
!
router bgp 65000
bgp router-id 10.30.2.50
...
!
address-family ipv4 unicast
# import vrf vrf_z01
# import vrf vrf_z02
# import vrf vrf_z03
exit-address-family
!
address-family ipv6 unicast
# import vrf vrf_z01
# import vrf vrf_z02
# import vrf vrf_z03
exit-address-family
!
address-family l2vpn evpn
...
Result：
1. external terminal could ping zone1 10.10.22.10 or zone2 10.10.22.10 through 10.30.2.161 or 10.30.2.162
2. 10.10.22.10 in two zones have correct internet access.
3. VMs in zone2 could ping each other without problem.
4. VM in zone1 could not ping VMs in zone2.
I think this article is very informative：
https://blog.oddbit.com/post/2023-02-19-vrf-and-nat/
However, I think it suggested another solution, such as implementing a router VM inside a zone to route traffic, or something like namespace in Openstack.
Also, I think this solution might be better than the current exit node implementation, as
1. there is no need to add many "ip route xxx null0" rules in frr.conf, especially when there are lots of subnets and zones.
2. VRF route information doesn't need to "leak" to the host, the host's route table is clean.
3. Host doesn't have route information to access VRF subnets. It might be more secure.
Is it so?
BTW, is it normal that I have to reset VM's network device (e.g. un-select firewall, click ok and select firewall, click ok) to make the VM could ping its gateway after a "systemctl restart networking" on the host？
Bugzilla ticket：
https://bugzilla.proxmox.com/show_bug.cgi?id=4980
