Isolated private networks with SDN

[dejieres]

Hello,

I have a standalone node (no cluster) on wich I try to configure several private networks, isolated from each other.

While searching for a solution, I found Proxmox's SDN capabilities. So I tried to create 2 zones (Simple plugin), each containing a Vnet, like this :

zone25​

vnet25​

subnet 192.168.25.0/24, SNAT=1​

zone26​

vnet26​

subnet 192.168.26.0/24, SNAT=1​

I have a W10 VM connected to vnet25, and a Debian 12 VM connected to vnet26

Everything seem to work fine, both VM have Internet access, but isolation seems to not be total.

For example, I can ping the Debian from the W10 machine, and I can ssh from W10 to Debian.

So my question is :
is SDN the appropriate solution for what I want to achieve ? Or did I miss something in my configuration ?

Thanks for any suggestion

 

[spirit]

mmm, indeed currently simple zones don't use dedicated vrf.

It's done for bgp-evpn zone, but not yet for simple zone.

I need to check if it could be easy to implement, because with bgp-evpn we have frr routing deamon, and we can easily manage routes to block traffic between differents vnet.

can you open a request in bugzilla.proxmox.com ?


(as workaround the only way is to use proxmox firewall)

 

[dejieres]

Hello spirit,

Thanks for your reply. I opened a request : https://bugzilla.proxmox.com/show_bug.cgi?id=5084

I'll try to fiddle a bit with the firewall. Not sure I'll achieve something, since I'm really not used to it. Creating rules is something mysterious to me
I'll try this on my local test installation, since I'm very likely to mess up networking at a time or another...

 

[dejieres]

Well, after trying a lot (and locking myself out once - fortunately, I had a console open on the node so I could disable the firewall from command line), I think I found the correct rules to isolate fully the virtual nets from each other and from the LAN, while allowing the VM to have Internet access in both direction, so I can do RDP to the VM, for example.​

After creating the SDN, I added four rules at VM level :

Type	Action	Interface	Source	Destination
out	ACCEPT	net0	192.168.25.0/24	192.168.25.0/24
in	ACCEPT	net0
out	ACCEPT	net0		192.168.0.254 (LAN GW)
out	DROP	net0		192.168.0.0/16

I also added some DNAT rules to /etc/network/interface to redirect traffic for various services (RDP, HTTPS, etc) to the appropriate VM: port.
The first rule depends on the subnet, and must be adapated accordingly. The last three can be put in a security group, making it easier to add to each VM.

After enabling proxmox firewall at each level (datacenter, node, VM and NIC), it seems to work so far...

 

[SanderNL]

I think i'm in the same boat. I'm trying to create an isolated network via SDN. For this i've created:
* Zone: Simple, automatic DHCP
* Vnet: name=vnet1
* Subnet within the vnet1: Name=snet1 Subnet=192.168.1.0/24, Gateway=192.168.1.1, NOT snat, Range=192.168.1.20..192.168.1.254
* A Debian container, Network=vnet1,Firewall=enabled
* A datacenter-firewall-rule using macro DNS to ACCEPT-IN DNS traffic to 192.168.1.1 on interface vnet1
* A datacenter-firewall-rule using macro DHCPfwd to ACCEPT-IN DHCP traffic on interface vnet1

In the Debian container i'm able to ping the outside world:
* i can ping my Proxmox host on 192.168.178.2
* i can ping my router on 192.168.178.1
* i can ping Google DNS on 8.8.8.8
* i can ping other Proxmox containers attached to vmbr0 at 192.168.178.9

There seems to be absolutely no network-isolation.

Some logs from my Proxmox host:

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 58:47:ca:71:f6:03 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
...
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 58:47:ca:71:f6:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.2/24 scope global vmbr0
       valid_lft forever preferred_lft forever
5: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 92:44:e1:14:83:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 scope global Desktops
       valid_lft forever preferred_lft forever

root@pve:/etc#  ip route
default via 192.168.178.1 dev vmbr0 proto kernel onlink
192.168.1.0/24 dev Desktops proto kernel scope link src 192.168.1.1          < vnet1
192.168.178.0/24 dev vmbr0 proto kernel scope link src 192.168.178.2      < vmbr0

root@pve:/etc# cat /etc/pve/firewall/cluster.fw
[OPTIONS]

enable: 1

[IPSET well-known-public-ip] # Verzameling van bekende publieke IP#

xxx.xxx.104.210 # iPhoneVanMe IP#
192.168.178.0/24 # LAN IP#
xxx.xxx.xxx.xxx # My publieke IP#

[RULES]

IN DNS(ACCEPT) -i vnet1 -dest 192.168.1.1 -log nolog
IN DHCPfwd(ACCEPT) -i vnet1 -log nolog
IN ACCEPT -p udp -dport 161 -log nolog
IN ACCEPT -source 192.168.178.10 -dest 192.168.178.2 -p tcp -dport 8006 -log nolog   # Zabbix traffic to Proxmox Host
IN ACCEPT -source 192.168.178.10 -dest 192.168.178.2 -p tcp -dport 10050 -log nolog # Zabbix traffic to Proxmox Host
IN ACCEPT -dest 192.168.178.2 -p icmp -log nolog -icmp-type echo-request # Echo request

root@pve:/etc# cat /etc/pve/nodes/pve/host.fw
[RULES]

IN ACCEPT -p tcp -dport 5201 -log nolog # iperf3
IN ACCEPT -dest 192.168.178.2 -p udp -dport 137,138 -log nolog # Netbios traffic
IN ACCEPT -dest 192.168.178.2 -p tcp -dport 445,139 -log nolog # Inbound Samba traffic

root@pve:/etc# cat /etc/pve/firewall/998.fw
[OPTIONS]

log_level_out: warning
log_level_in: warning
enable: 1

root@pve:/etc# pve-firewall status
Status: enabled/running

root@pve:/etc# iptables --list

Chain INPUT (policy ACCEPT)
target     prot opt source               destination          
PVEFW-INPUT  all  --  anywhere             anywhere             

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination          
PVEFW-FORWARD  all  --  anywhere             anywhere             

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination          
PVEFW-OUTPUT  all  --  anywhere             anywhere             

Chain PVEFW-Drop (11 references)
target     prot opt source               destination          
PVEFW-DropBroadcast  all  --  anywhere             anywhere             
ACCEPT     icmp --  anywhere             anywhere             icmp fragmentation-needed
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
DROP       all  --  anywhere             anywhere             ctstate INVALID
DROP       udp  --  anywhere             anywhere             multiport dports 135,445
DROP       udp  --  anywhere             anywhere             udp dpts:netbios-ns:139
DROP       udp  --  anywhere             anywhere             udp spt:netbios-ns dpts:1024:65535
DROP       tcp  --  anywhere             anywhere             multiport dports epmap,netbios-ssn,microsoft-ds
DROP       udp  --  anywhere             anywhere             udp dpt:1900
DROP       tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN
DROP       udp  --  anywhere             anywhere             udp spt:domain
           all  --  anywhere             anywhere             /* PVESIG:83WlR/a4wLbmURFqMQT3uJSgIG8 */

Chain PVEFW-DropBroadcast (2 references)
target     prot opt source               destination          
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type MULTICAST
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type ANYCAST
DROP       all  --  anywhere             base-address.mcast.net/4  
           all  --  anywhere             anywhere             /* PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w */

Chain PVEFW-FORWARD (1 references)
target     prot opt source               destination          
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
PVEFW-FWBR-IN  all  --  anywhere             anywhere             PHYSDEV match --physdev-in fwln+ --physdev-is-bridged
PVEFW-FWBR-OUT  all  --  anywhere             anywhere             PHYSDEV match --physdev-out fwln+ --physdev-is-bridged
           all  --  anywhere             anywhere             /* PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw */

Chain PVEFW-FWBR-IN (1 references)
target     prot opt source               destination          
PVEFW-smurfs  all  --  anywhere             anywhere             ctstate INVALID,NEW
tap100i0-IN  all  --  anywhere             anywhere             PHYSDEV match --physdev-out tap100i0 --physdev-is-bridged
...
veth998i0-IN  all  --  anywhere             anywhere             PHYSDEV match --physdev-out veth998i0 --physdev-is-bridged
           all  --  anywhere             anywhere             /* PVESIG:e4KtH2a7adgEnfiaSvs9pfFK74k */

Chain PVEFW-FWBR-OUT (1 references)
target     prot opt source               destination          
tap100i0-OUT  all  --  anywhere             anywhere             PHYSDEV match --physdev-in tap100i0 --physdev-is-bridged
...
veth998i0-OUT  all  --  anywhere             anywhere             PHYSDEV match --physdev-in veth998i0 --physdev-is-bridged
           all  --  anywhere             anywhere             /* PVESIG:GftVzGrEH/xDpFmHnyMmSqqb+Qg */

Chain PVEFW-HOST-IN (1 references)
target     prot opt source               destination          
ACCEPT     all  --  anywhere             anywhere             
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
PVEFW-smurfs  all  --  anywhere             anywhere             ctstate INVALID,NEW
RETURN     igmp --  anywhere             anywhere             
RETURN     tcp  --  anywhere             anywhere             tcp dpt:5201
RETURN     udp  --  anywhere             pve.home             multiport dports netbios-ns,netbios-dgm
RETURN     tcp  --  anywhere             pve.home             multiport dports microsoft-ds,netbios-ssn
RETURN     udp  --  anywhere             192.168.1.1          udp dpt:domain
RETURN     tcp  --  anywhere             192.168.1.1          tcp dpt:domain
RETURN     udp  --  anywhere             anywhere             udp spts:bootps:bootpc dpts:bootps:bootpc
RETURN     udp  --  anywhere             anywhere             udp dpt:snmp
RETURN     tcp  --  zabbix.lan           pve.home             tcp dpt:8006
RETURN     tcp  --  zabbix.lan           pve.home             tcp dpt:zabbix-agent
RETURN     icmp --  anywhere             pve.home             icmp echo-request
RETURN     tcp  --  anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:8006
RETURN     tcp  --  anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpts:5900:5999
RETURN     tcp  --  anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:3128
RETURN     tcp  --  anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:ssh
RETURN     tcp  --  anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpts:60000:60050
PVEFW-Drop  all  --  anywhere             anywhere             
DROP       all  --  anywhere             anywhere             
           all  --  anywhere             anywhere             /* PVESIG:orYnzPiSW7ZvAbjsRZHctTVfbWs */

Chain PVEFW-HOST-OUT (1 references)
target     prot opt source               destination          
ACCEPT     all  --  anywhere             anywhere             
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
RETURN     igmp --  anywhere             anywhere             
RETURN     tcp  --  anywhere             192.168.178.0/24     tcp dpt:8006
RETURN     tcp  --  anywhere             192.168.178.0/24     tcp dpt:ssh
RETURN     tcp  --  anywhere             192.168.178.0/24     tcp dpts:5900:5999
RETURN     tcp  --  anywhere             192.168.178.0/24     tcp dpt:3128
RETURN     all  --  anywhere             anywhere             
           all  --  anywhere             anywhere             /* PVESIG:8kxbCk6iQ0RJcmbPoyVjFXDUeEc */

Chain PVEFW-INPUT (1 references)
target     prot opt source               destination          
PVEFW-HOST-IN  all  --  anywhere             anywhere             
           all  --  anywhere             anywhere             /* PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk */

Chain PVEFW-OUTPUT (1 references)
target     prot opt source               destination          
PVEFW-HOST-OUT  all  --  anywhere             anywhere             
           all  --  anywhere             anywhere             /* PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0 */

Chain PVEFW-Reject (0 references)
target     prot opt source               destination          
PVEFW-DropBroadcast  all  --  anywhere             anywhere             
ACCEPT     icmp --  anywhere             anywhere             icmp fragmentation-needed
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
DROP       all  --  anywhere             anywhere             ctstate INVALID
PVEFW-reject  udp  --  anywhere             anywhere             multiport dports 135,445
PVEFW-reject  udp  --  anywhere             anywhere             udp dpts:netbios-ns:139
PVEFW-reject  udp  --  anywhere             anywhere             udp spt:netbios-ns dpts:1024:65535
PVEFW-reject  tcp  --  anywhere             anywhere             multiport dports epmap,netbios-ssn,microsoft-ds
DROP       udp  --  anywhere             anywhere             udp dpt:1900
DROP       tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN
DROP       udp  --  anywhere             anywhere             udp spt:domain
           all  --  anywhere             anywhere             /* PVESIG:h3DyALVslgH5hutETfixGP08w7c */

Chain PVEFW-SET-ACCEPT-MARK (26 references)
target     prot opt source               destination          
MARK       all  --  anywhere             anywhere             MARK or 0x80000000
           all  --  anywhere             anywhere             /* PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY */

Chain PVEFW-logflags (5 references)
target     prot opt source               destination          
DROP       all  --  anywhere             anywhere             
           all  --  anywhere             anywhere             /* PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A */

Chain PVEFW-reject (4 references)
target     prot opt source               destination          
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST
DROP       all  --  base-address.mcast.net/4  anywhere             
DROP       icmp --  anywhere             anywhere             
REJECT     tcp  --  anywhere             anywhere             reject-with tcp-reset
REJECT     udp  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     icmp --  anywhere             anywhere             reject-with icmp-host-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited
           all  --  anywhere             anywhere             /* PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc */

Chain PVEFW-smurflog (2 references)
target     prot opt source               destination          
DROP       all  --  anywhere             anywhere             
           all  --  anywhere             anywhere             /* PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk */

Chain PVEFW-smurfs (2 references)
target     prot opt source               destination          
RETURN     all  --  0.0.0.0              anywhere             
PVEFW-smurflog  all  --  anywhere             anywhere            [goto]  ADDRTYPE match src-type BROADCAST
PVEFW-smurflog  all  --  base-address.mcast.net/4  anywhere            [goto]  
           all  --  anywhere             anywhere             /* PVESIG:HssVe5QCBXd5mc9kC88749+7fag */

Chain PVEFW-tcpflags (0 references)
target     prot opt source               destination          
PVEFW-logflags  tcp  --  anywhere             anywhere            [goto]  tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,PSH,URG
PVEFW-logflags  tcp  --  anywhere             anywhere            [goto]  tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE
PVEFW-logflags  tcp  --  anywhere             anywhere            [goto]  tcp flags:SYN,RST/SYN,RST
PVEFW-logflags  tcp  --  anywhere             anywhere            [goto]  tcp flags:FIN,SYN/FIN,SYN
PVEFW-logflags  tcp  --  anywhere             anywhere            [goto]  tcp spt:0 flags:FIN,SYN,RST,ACK/SYN
           all  --  anywhere             anywhere             /* PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo */

...

Chain veth998i0-IN (1 references)
target     prot opt source               destination          
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps dpt:bootpc
PVEFW-Drop  all  --  anywhere             anywhere             
NFLOG      all  --  anywhere             anywhere             limit: avg 1/sec burst 5 nflog-prefix ":998:4:veth998i0-IN: policy DROP: "
DROP       all  --  anywhere             anywhere             
           all  --  anywhere             anywhere             /* PVESIG:b3AZwbX7W/0yCRMvtREZODb0r+M */

Chain veth998i0-OUT (1 references)
target     prot opt source               destination          
PVEFW-SET-ACCEPT-MARK  udp  --  anywhere             anywhere            [goto]  udp spt:bootpc dpt:bootps
DROP       all  --  anywhere             anywhere             MAC ! bc:24:11:fe:c0:b0
MARK       all  --  anywhere             anywhere             MARK and 0x7fffffff
PVEFW-SET-ACCEPT-MARK  all  --  anywhere             anywhere            [goto]  
           all  --  anywhere             anywhere             /* PVESIG:BXHxWH959v+4UqdR5zZZXNNnuuU */

 

[dejieres]

Hello SanderNL,

I forgot to mention, but if your Proxmox version is lower than 8.1, or if it is >= 8.1 but was upgraded from previous versions, you need to install the libpve-network-perl package.

In my case, the goal was to separate different subnets from each other (ie. subnet 25 and subnet 26 can't see each other), but router and internet access is allowed for each subnet.

It seems, if I understood correctly, you just need a guest network : simply create a new vmbrN and connect all you VMs to this interface. This should be enough to have a completely isolated subnet.

 

[andreirapuru]

Using two evpn zones (with one vnet in each zone), you can isolate both network.
However, you will have to configure IP manually (our create a DHCP server).
Also will have to configure routing to get to the physical network.