venet like networking for lxc

[hk@]

Hi
the main question is: where could we hook startup and shutdown scripts for containers in proxmox?
we connect containers to a linux-bridge and then add host-routes with "ip rou add 10.11.22.33/32 dev vmbrXX" on the hypervisor, we also add a host-route within the container direct at the bridge-ip and then add a default-route with this ip as a gateway.
This way we get IPv4 to work for containers like the venet-device did, but we would need to automate this and add and delete routes with containers starting up and shutting down, any advise is greatly appreciated.

For IPv6 we could not achieve a similar solution as within the cointainer "ip -6 rou add" is not permitted and we have yet to find a solution for this one. Also on this issue any advise is appreciated.

Regards
hk

 

[hk@]

in the meantime ... found that PVE5 is already helping by configuring container interface post-up and pre-down scripts accordingly.
for IPv6 this is also done, but is not sufficient one can not set eg fe80::1 as gateway, but only obvious non local addresses, funfact also link-local IPv4 as gateway (eg 169.254.100.10) can not be used as gateway-IP.
Using RADVD one can circumvent the IPv6 gateway-issue, but using link-local-addresses in this case would be very much appreciated.
And while IPv4 allows /32 IPs to be used as singe address, IPv6 only allows for /120, yet we'd expect /128 for single containers.
Any comment appreciated.

 

[hk@]

hmm - noone?

 

[wbumiller]

but we would need to automate this and add and delete routes with containers starting up and shutting down

We currently have no startup hooks available directly. But since we start containers via a systemd service you can create a drop-in for pve-container@.service with Exec{Start,Stop}Post actions.
eg. a file `/etc/systemd/system/pve-container@.service.d/hooks.conf`) with

[Service]
ExecStartPost=/data/hooks/container %I started
ExecStopPost=/data/hooks/container %I stopped

For IPv6 we could not achieve a similar solution as within the cointainer "ip -6 rou add" is not permitted and we have yet to find a solution for this one.

What do you mean it is not permitted?

but is not sufficient one can not set eg fe80::1 as gateway

I don't see why not? If you get any errors from specific settings it would be nice to know them. I can type fe80::1 in the GUI here and it works.

Using RADVD one can circumvent the IPv6 gateway-issue, but using link-local-addresses in this case would be very much appreciated.

That should work with and without radvd.

IPv6 only allows for /120,

It seems to generally be discouraged to use prefixes longer than actually 64 bits, although technically 128 should work in a container scenario. For now you'd have to leave the field blank and configure it inside the container. We had such a limit for ipv4 at some point and removed it. I suppose we can do that for ipv6 as well.

 

[hk@]

Dear Wolfgang,
thank you for your reply, we will engage another testinground.
As for the IPv6 prefixes longer than /64 it's a general rule for ip-address-autoassignment to work on a broadcast network you need /64, otherwise this won't work, but for example for point-to-point-links it's always been a rule to use /127 (or sometimes /126 if the equipment can't handle /127), otherwise simple attacks like ping a lot of addresses in the whole /64 will fill up the router (aka "Neighbor Discovery DoS Attack" https://tools.ietf.org/html/rfc3756);
In the end for a cointainer (or VM) most of the time something like /128 up to /120 (already 256 IPv6 addressess) should be quite sufficient and whole /64s would potentially risk the host machine's NDP-cache to be exhausted therefore the proposal would be: allow /128 to be configured .

regards,
hk

 

[hk@]

back again for a single-ip-networking-setup

here is what needs to be done:
hypervisor:
have a bridge (eg vmbr2) and assign it an IP like 10.1.1.1/32
the add an IP route for a container: ip rou add 10.10.10.10/32 dev vmbr2
within the container (assuming eth0 is connected to vmbr2 at the hypervisor) two routes need to be added:
ip rou add 10.1.1.1/32 dev eth0
ip rou add default via 10.1.1.1

assuming also ipforwarding has been enabled on the hypervisor - all is done.

besides needing the same setup for ipv6 it would be very helpful to be able to integrated this into the proxmox gui-setup and gui-configuration as for this to work we currently have to modify multiple configurationfiles and especially the interfaces config on the hypervisor and migration from one node to the other is not possible without again manually editing things.

thx
hk

 

[moxfan]

back again for a single-ip-networking-setup

here is what needs to be done:
hypervisor:
have a bridge (eg vmbr2) and assign it an IP like 10.1.1.1/32
the add an IP route for a container: ip rou add 10.10.10.10/32 dev vmbr2
within the container (assuming eth0 is connected to vmbr2 at the hypervisor) two routes need to be added:
ip rou add 10.1.1.1/32 dev eth0
ip rou add default via 10.1.1.1

assuming also ipforwarding has been enabled on the hypervisor - all is done.

besides needing the same setup for ipv6 it would be very helpful to be able to integrated this into the proxmox gui-setup and gui-configuration as for this to work we currently have to modify multiple configurationfiles and especially the interfaces config on the hypervisor and migration from one node to the other is not possible without again manually editing things.

thx
hk

hk, this looks very promising. Would you mind writing down the precise configs you have in the interfaces on the hypervisor and in route-eth0 in the LXC that enables the venet like networking?
Norman