[SOLVED] SDN VNet subnet specify SNAT interface/IP

[murda]

I've setup a simple zone with automatic DHCP, a vnet and 10.0.0.0/24 subnet with SNAT enabled.

On the host I have 2 bridges configured:
- vmbr0 with public IP 1.1.1.1
- vmbr1 with public IP 1.1.1.2
(Those aren't the real IP addresses, just substituted by me).

When creating the subnet and enabling the SNAT option, it automatically chooses vmbr0. I couldn't find documentation on how to change this to another interface/IP so I manually edited /etc/network/interfaces.d/sdn like so:

#version:15

auto vnet0
iface vnet0
    address 10.0.0.1/24
    post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr1 -j SNAT --to-source 1.1.1.2
    post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr1 -j SNAT --to-source 1.1.1.2
    post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    ip-forward on

But upon `reloadnetworkall` the POSTROUTING rules get overwritten back to `-o vmbr0` and `--to-source 1.1.1.1`.

What's the recommended method to get my desired outcome?

 

[murda]

Additionally, are custom SNAT settings in SDN planned with these roadmap items once they get merged?

 

[shanreich]

Choosing the Interface is currently not possible. It always takes the IP of the interface that provides the default route (or none, if there is no default route).

Better SNAT/DNAT support is something that I am considering implementing with the new nftables firewall, there is already an issue for that in our Bugzilla [1]

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=5239

 

[murda]

Choosing the Interface is currently not possible. It always takes the IP of the interface that provides the default route (or none, if there is no default route).

Better SNAT/DNAT support is something that I am considering implementing with the new nftables firewall, there is already an issue for that in our Bugzilla [1]

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=5239

Thank you for the swift reply Stefan. I'm excited for the new nftables based firewall.

I will manually create the additional interface in /etc/network/interfaces.d/. But is my assumption correct that this entirely opts out of SDN, including automatic DHCP?

 

[shanreich]

I will manually create the additional interface in /etc/network/interfaces.d/.

You mean the vnet0 interface? Yes, sadly this will opt out of SDN entirely - including Automatic DHCP.

I think should be able to manually create the SNAT rules in your interfaces file instead. Turn off SNAT in the SDN config and add the following to your /etc/network/interfaces:

iface vnet0
    post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr1 -j SNAT --to-source 1.1.1.2
    post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr1 -j SNAT --to-source 1.1.1.2
    post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

 

[murda]

You mean the vnet0 interface? Yes, sadly this will opt out of SDN entirely - including Automatic DHCP.

I think should be able to manually create the SNAT rules in your interfaces file instead. Turn off SNAT in the SDN config and add the following to your /etc/network/interfaces:

iface vnet0
    post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr1 -j SNAT --to-source 1.1.1.2
    post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr1 -j SNAT --to-source 1.1.1.2
    post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

Yes, the vnet0 interface. I've added the iptables rules but I'm not getting internet access with `-o vmbr1` and with 1.1.1.2 on vmbr1. Aliasing 1.1.1.2 under vmbr0 and setting `-o vmbr0` did the trick though. Just need to setup DHCP now and I'm set.

Thanks

 

[maarten256]

I wanted to add to this discussion - I was looking for a solution to a similar problem in my setup. I have a simple zone set up with SDN, with a vnet and subnet. For entirely different reasons, my pve nodes in the cluster have two physical NICs and associated bridges.

I also found that SNAT for the subnet was only enabled for the interface with the default route, so a guest attached to the vnet would only be able to communicate out through the host on one of the two NICs.

The change I made to address this was to replicate the SNAT rules in /etc/network/interface.d/sdn and set them up for the second host bridge/NIC, such that traffic routed through either NIC on the host gets SNAT.

This seems to work - the guest gets an IP address from DHCP (defined in SDN), and it's able to communicate through both NICs.

Perhaps my use case is different from the one discussed in this thread, but I did not need to take the interface out of SDN altogether, as was suggested.

 

[murda]

I wanted to add to this discussion - I was looking for a solution to a similar problem in my setup. I have a simple zone set up with SDN, with a vnet and subnet. For entirely different reasons, my pve nodes in the cluster have two physical NICs and associated bridges.

I also found that SNAT for the subnet was only enabled for the interface with the default route, so a guest attached to the vnet would only be able to communicate out through the host on one of the two NICs.

The change I made to address this was to replicate the SNAT rules in /etc/network/interface.d/sdn and set them up for the second host bridge/NIC, such that traffic routed through either NIC on the host gets SNAT.

This seems to work - the guest gets an IP address from DHCP (defined in SDN), and it's able to communicate through both NICs.

Perhaps my use case is different from the one discussed in this thread, but I did not need to take the interface out of SDN altogether, as was suggested.

I thought of doing something similar in my case, keep the SDN but disable SNAT in the GUI and write my own SNAT rules in /etc/network/interface.d/ for the vnet. Haven't tried it but it could very well work, although I opted out of SDN entirely because I'm not sure what potential side-effects exist this way.

 

[w2vy]

Yes, the vnet0 interface. I've added the iptables rules but I'm not getting internet access with `-o vmbr1` and with 1.1.1.2 on vmbr1. Aliasing 1.1.1.2 under vmbr0 and setting `-o vmbr0` did the trick though. Just need to setup DHCP now and I'm set.

Thanks

Could you post the complete final solution?

I think what you have done is very close to what I need.

One physical port to Internet (subnet with 2+ WAN IP)
vmbr0/1 each with a Public IP - how are those setup? 1.1.1.1/32 ? Any Gateway IP set?

And your final networking/interfaces file(s)?

I have been working on this on and off for the past few months and I know the details really really matter... There is NO 'close enough' lol

Tom

 

[w2vy]

I've setup a simple zone with automatic DHCP, a vnet and 10.0.0.0/24 subnet with SNAT enabled.

On the host I have 2 bridges configured:
- vmbr0 with public IP 1.1.1.1
- vmbr1 with public IP 1.1.1.2
(Those aren't the real IP addresses, just substituted by me).

Do vmbr0 and vmbr1 have Bridge Ports assigned to them? Maybe the port connected to the WAN?

Also where would the default gateway be set? Your example does not mention one

In my case I have a handful if IP's 2.2.2.55 - 2.2.2.59 with a default gateway 2.2.2.1
(Obviously not real but mirrors what I have fairly well)

Thanks,
Tom