Static route in proxmox and systemd-networkd failure in container..

[bdcatpcsd]

5.3 install with xfs (should it matter)

the building the server is located in is a 10.20.32.0/20

there is a router there for internet, 10.20.32.250

there is a router for lan connectivity, 10.20.32.1

I'm looking for a 'best practice' for how proxmox works..

should I put the static route for the entire network on the proxmox box itself, or only on the containers that will use it.

Currently I have this in proxmox (and proxmox works just fine)

auto vmbr0
iface vmbr0 inet static
address 10.20.32.100
netmask 255.255.240.0
gateway 10.20.32.250
bridge-ports eno1
bridge-stp off
bridge-fd 0
up ip route add 10.20.0.0/16 via 10.20.32.1 dev vmbr0
up ip route add 172.16.0.0/16 via 10.20.32.1 dev vmbr0

then I created an archlinux container, which *was* fine, now .. not so much and I can't figure out why.

the archlinux machine installed just fine and was fine *when rebooted*, the network worked.

but now since the machine was updated it doesn't work without manual intervention..

(archlinux container)

[Match]
Name = eth0

[Network]
Description = Interface eth0 autoconfigured by PVE
Address = 10.20.32.101/20
Gateway = 10.20.32.250
DHCP = none
IPv6AcceptRA = false

that is created and re-created by proxmox upon reboot..

and systemctl start systemd-networkd fails upon reboot..

but running /usr/lib/systemd/systemd-networkd gets the network to work..

The eth0.network file that I would like to use is this: (this forum does terrible with pasted lines.. ugh)


cat /root/eth0.network

[Match]
Name=eth0
[Network]
Description="Something"
Address=10.20.32.101/20
Gateway=10.20.32.250
[Route]
Gateway=10.20.32.1
Destination=10.20.0.0/16
[Route]
Gateway=10.20.32.1
Destination=172.16.0.0/16

tl;dr:
How can I get systemd networking working again in arch, and what is the most efficient way for static routes? per container? or on the proxmox system itself.

thanks in advance.

 

[oguz]

then I created an archlinux container, which *was* fine, now .. not so much and I can't figure out why.

the archlinux machine installed just fine and was fine *when rebooted*, the network worked.

but now since the machine was updated it doesn't work without manual intervention..

My best guess is you're running into this upstream bug:

https://bugs.archlinux.org/task/61313
https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1811248
https://github.com/lxc/lxc/issues/2778

systemd version is 240? If yes, you can try to downgrade until it's fixed upstream, any version before 240 seems to work for now.

EDIT: I can confirm this version works for now:

core systemd 239.303-1 [installed]

 

[bdcatpcsd]

Following your links it looks like this might also be an lxc issue..

dmesg | grep audit (on the archlinux container)

[archuser@archclnt101 ~]$ dmesg | grep audit
[ 0.036572] audit: initializing netlink subsys (disabled)
[ 0.036572] audit: type=2000 audit(1547825017.036:1): state=initialized audit_enabled=0 res=1
[ 15.056904] audit: type=1400 audit(1547825031.898:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=925 comm="apparmor_parser"
[ 15.118049] audit: type=1400 audit(1547825031.958:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=924 comm="apparmor_parser"
[ 15.118060] audit: type=1400 audit(1547825031.958:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=924 comm="apparmor_parser"
[ 15.118062] audit: type=1400 audit(1547825031.958:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=924 comm="apparmor_parser"
[ 15.118064] audit: type=1400 audit(1547825031.958:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=924 comm="apparmor_parser"
[ 17.723480] audit: type=1400 audit(1547825034.562:7): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/bin/lxc-start" pid=1361 comm="apparmor_parser"
[ 17.729217] audit: type=1400 audit(1547825034.570:8): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default" pid=1365 comm="apparmor_parser"
[ 17.729229] audit: type=1400 audit(1547825034.570:9): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-cgns" pid=1365 comm="apparmor_parser"
[ 17.729231] audit: type=1400 audit(1547825034.570:10): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-with-mounting" pid=1365 comm="apparmor_parser"
[ 17.729232] audit: type=1400 audit(1547825034.570:11): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-with-nesting" pid=1365 comm="apparmor_parser"
[ 27.294695] audit: type=1400 audit(1547825044.134:12): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-101_</var/lib/lxc>" pid=1930 comm="apparmor_parser"
[ 29.643859] audit: type=1400 audit(1547825046.482:13): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-101_</var/lib/lxc>" name="/" pid=2132 comm="(networkd)" flags="rw, rslave"
[ 29.671135] audit: type=1400 audit(1547825046.510:14): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-101_</var/lib/lxc>" name="/" pid=2145 comm="(networkd)" flags="rw, rslave"
[ 29.699107] audit: type=1400 audit(1547825046.538:15): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-101_</var/lib/lxc>" name="/" pid=2152 comm="(networkd)" flags="rw, rslave"
[ 29.727322] audit: type=1400 audit(1547825046.566:16): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-101_</var/lib/lxc>" name="/" pid=2162 comm="(networkd)" flags="rw, rslave"
[ 29.755343] audit: type=1400 audit(1547825046.594:17): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-101_</var/lib/lxc>" name="/" pid=2172 comm="(networkd)" flags="rw, rslave"

I cheated..

I run this @reboot via cron:

[root@archclnt101 ~]# cat /root/eth0.linkup

ip link set eth0 up
ip addr add 10.20.32.101/20 dev eth0
ip route add default via 10.20.32.250
ip route add 10.20.0.0/16 via 10.20.32.1 dev eth0
ip route add 172.16.0.0/16 via 10.20.32.1 dev eth0

[root@archclnt101 ~]# pacman -Q | grep systemd
libsystemd 240.34-3
systemd 240.34-3
systemd-sysvcompat 240.34-3

and I've reverted the proxmox machine to this config:

auto vmbr0
iface vmbr0 inet static
address 10.20.32.100
netmask 255.255.240.0
gateway 10.20.32.1
bridge-ports eno1
bridge-stp off
bridge-fd 0


Also is the normal from the container?

[archuser@archclnt101 ~]$ tracepath -n 9.9.9.9
1?: [LOCALHOST] pmtu 1500
1: 10.20.32.250 0.420ms
1: 10.20.32.250 0.698ms
2: x.x.x.x 1.117ms
3: y.y.y.y 7.432ms
4: no reply
5: 140.222.1.59 4.299ms
6: 152.179.72.42 3.694ms asymm 5
7: 129.250.6.69 3.211ms asymm 6
8: no reply
9: 129.250.198.150 5.614ms !H
Resume: pmtu 1500


See the double 10.20.32.250 entry..

From the proxmox host it doesn't do that.. only the container..

Thanks in advance

 

[oguz]

I cheated..

I run this @reboot via cron:

[root@archclnt101 ~]# cat /root/eth0.linkup

ip link set eth0 up
ip addr add 10.20.32.101/20 dev eth0
ip route add default via 10.20.32.250
ip route add 10.20.0.0/16 via 10.20.32.1 dev eth0
ip route add 172.16.0.0/16 via 10.20.32.1 dev eth0

My advice would be to avoid playing around with that, especially with a hacky solution like this one.

Also is the normal from the container?

[archuser@archclnt101 ~]$ tracepath -n 9.9.9.9
1?: [LOCALHOST] pmtu 1500
1: 10.20.32.250 0.420ms
1: 10.20.32.250 0.698ms
2: x.x.x.x 1.117ms
3: y.y.y.y 7.432ms
4: no reply
5: 140.222.1.59 4.299ms
6: 152.179.72.42 3.694ms asymm 5
7: 129.250.6.69 3.211ms asymm 6
8: no reply
9: 129.250.198.150 5.614ms !H
Resume: pmtu 1500


See the double 10.20.32.250 entry..

From the proxmox host it doesn't do that.. only the container..

I'm getting differing outputs, sometimes with double entries, sometimes without. I guess this is not a container thing but a tracepath thing.