Proxmox, OPNSense, IPv6, Router Advertisments

[atlan78]

Hi Proxmox-Community,

i have a little question about the following setup.

I have a dedicated server from hetzner and I want to run proxmox, opnsense and some vms on it. I want to use ipv6 for connections. (ipv4 is another topic, but ipv6 is the modern way )

I have 1 public ipv4, and these ipv6 nets:

2a01:xxx:yyy:2c96::/64
2a01:xxx:yyy:fd00::/56

I think I can divide this /56 net into several /64 nets:
2a01:xxx:yyy:fd00::/64
2a01:xxx:yyy:fd01::/64
2a01:xxx:yyy:fd02::/64
2a01:xxx:yyy:fd03::/64
...

So my idea is to use the /64 net to connect to my proxmox-host. Then I want to give the opnsense one ip of the /56 net on the wan-interface and configure the lans with /64 subnets. Here is a picture of this idea:



so i configured this in my /etc/network/interfaces on proxmox-host (removed some ipv4 stuff):

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp0s31f6

iface enp0s31f6 inet6 static
    address 2a01:xxx:yyy:2c96::2/128
    gateway fe80::1

auto vmbr0
iface vmbr0 inet6 static
    address 2a01:xxx:yyy:fd00::1/64
    up ip -6 route add 2a01:xxx:yyy:fd00::/56 via 2a01:xxx:yyy:fd00::2
# public interface wan

auto vmbr1
iface vmbr1 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
iface vmbr1 inet6 manual
# LAN

ip -6 route show gives me:

::1 dev lo proto kernel metric 256 pref medium
2a01:xxx:yyy:2c96::2 dev enp0s31f6 proto kernel metric 256 pref medium
2a01:xxx:yyy:fd00::/64 dev vmbr0 proto kernel metric 256 pref medium
2a01:xxx:yyy:fd00::/56 via 2a01:xxx:yyy:fd00::2 dev vmbr0 metric 1024 pref medium
fe80::/64 dev enp0s31f6 proto kernel metric 256 pref medium
fe80::/64 dev vmbr0 proto kernel metric 256 pref medium
fe80::/64 dev vmbr1 proto kernel metric 256 pref medium
default via fe80::1 dev enp0s31f6 proto kernel metric 1024 onlink pref medium

I connect to the opnsense and configured these addresses:
WAN: 2a01:xxx:yyy:fd00::2/64
LAN: 2a01:xxx:yyy:fd01::1/64

Then I disabled the Router-Advertisments for the LAN Interface.

Then I start a VM with Network-Interface vmbr1 and manually set an ipv6 ip in the vm:
Address: 2a01:xxx:yyy:fd01::2/64
Gateway: 2a01:xxx:yyy:fd01::1

Now everything works fine. I can ping this vm and connect to it via ssh from my local machine. (ssh root@2a01:xxx:yyy:fd01::2)

Everything fine until now.

Next step is to activate router-advertisments and dhcpdv6 for the LAN-Interface to set the ip-adresses automatically. So I set the router-advertisement-mode to "unmanaged". I'll try the next modes with dhcp6 later.

But now, a new route appears on the proxmox host:

::1 dev lo proto kernel metric 256 pref medium
2a01:xxx:yyy:2c96::2 dev enp0s31f6 proto kernel metric 256 pref medium
2a01:xxx:yyy:fd00::/64 dev vmbr0 proto kernel metric 256 pref medium
2a01:xxx:yyy:fd01::/64 dev vmbr1 proto kernel metric 256 expires 86391sec pref medium
2a01:xxx:yyy:fd00::/56 via 2a01:xxx:yyy:fd00::2 dev vmbr0 metric 1024 pref medium
fe80::/64 dev enp0s31f6 proto kernel metric 256 pref medium
fe80::/64 dev vmbr0 proto kernel metric 256 pref medium
fe80::/64 dev vmbr1 proto kernel metric 256 pref medium
default via fe80::1 dev enp0s31f6 proto kernel metric 1024 onlink pref medium
default via fe80::2c30:6bff:fe52:a4a2 dev vmbr1 proto ra metric 1024 expires 1791sec hoplimit 64 pref medium

After that, I can't connect to the vm anymore. I think the reason is this route:

2a01:xxx:yyy:fd01::/64 dev vmbr1 proto kernel metric 256 expires 86391sec pref medium

If I understand it correctly, now packets are routed from outside directly to vmbr1 interface. Firewall now blocks this traffic and says, that it comes from LAN-Interface:

ssh root@2a01:xxx:yyy:fd01:bc0f:c9ff:feac:f9cc (dynamically assigned ipv6)

LAN1        2023-03-16T20:01:37    [2a01:xxx:yyy:fd01:bc0f:c9ff:feac:f9cc]:22    [2003:c1:370a:c800:d41d:ee40:62a1:3c36]:65477    tcp    Default deny / state violation rule   
LAN1        2023-03-16T20:01:36    [2a01:xxx:yyy:fd01:bc0f:c9ff:feac:f9cc]:22    [2003:c1:370a:c800:d41d:ee40:62a1:3c36]:65477    tcp    Default deny / state violation rule   
LAN1        2023-03-16T20:01:35    [2a01:xxx:yyy:fd01:bc0f:c9ff:feac:f9cc]:22    [2003:c1:370a:c800:d41d:ee40:62a1:3c36]:65477    tcp    Default deny / state violation rule   
LAN1        2023-03-16T20:01:34    [2a01:xxx:yyy:fd01:bc0f:c9ff:feac:f9cc]:22    [2003:c1:370a:c800:d41d:ee40:62a1:3c36]:65477    tcp    Default deny / state violation rule   
LAN1        2023-03-16T20:01:33    [2a01:xxx:yyy:fd01:bc0f:c9ff:feac:f9cc]:22    [2003:c1:370a:c800:d41d:ee40:62a1:3c36]:65477    tcp    Default deny / state violation rule   
LAN1        2023-03-16T20:01:32    [2a01:xxx:yyy:fd01:bc0f:c9ff:feac:f9cc]:22    [2003:c1:370a:c800:d41d:ee40:62a1:3c36]:65477    tcp    Default deny / state violation rule

if I manually delete the route on proxmox host, everything works again.

ip -6 route del 2a01:xxx:yyy:fd01::/64 dev vmbr1

I watched videos and read a lot of articles in the last days and can't find a solution. Does anybody have a little hint for me? Did I misunderstood some concepts of these bridges or ipv6 concepts?

Thanks for any help!

Ronny

 

[atlan78]

I think I found a possible solution.

I placed this in /etc/sysctl.d/99-proxmox.conf

net.ipv6.conf.vmbr1.accept_ra=0

Now the host does not accept the router advertisements for interface vmbr1 and my routes stay as they are. I enabled router-advertisements on LAN-Interface (Mode Managed) and dhcpv6 in opnsense. Now my VMs automatically get ipv6 addresses from the configured net, can access the internet and I can access them from my local machine.

I hope that will work. I'll test this for a while.

Do you think, this is ok?