Hello
I have a OVH server with Proxmox 5.2 installed (image from OVH). Server has a single public IP address so I need to configure NAT for VM's (lxc containers). The problem is I'm not able to reach my machines within the internal (NAT) network.
What I did, in order (real IP address of my server is substituted with 1.2.3.215):
1. Fresh install of the proxmox 5.2 on OVH (Debian 9)
2. Changed /etc/network/interfaces to the following (according to https://pve.proxmox.com/wiki/Network_Configuration):
auto lo
iface lo inet loopback
auto eno1
#real IP adress
iface eno1 inet static
address 1.2.3.215
netmask 255.255.255.0
gateway 1.2.3.254
auto vmbr0
#private sub network
iface vmbr0 inet static
address 192.168.2.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '192.168.2.0/24' -o eno1 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.2.0/24' -o eno1 -j MASQUERADE
3. Reboot of the server
4. Results of the ifconfig and route -n after the reboot are as follows. Seems that: inet 1.2.3.215 for vmbr0 - as reported by ifconfig - is not OK as it says inet 1.2.3.215 intead of 192.168.2.1!
root@node03:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 1.2.3.254 0.0.0.0 UG 0 0 0 eno1
0.0.0.0 1.2.3.254 0.0.0.0 UG 0 0 0 vmbr0
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
1.2.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eno1
1.2.3.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
root@node03:~# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.2.3.215 netmask 255.255.255.0 broadcast 1.2.3.255
inet6 fe80::a6bf:1ff:fe1f:c81a prefixlen 64 scopeid 0x20<link>
ether a4:bf:01:1f:c8:1a txqueuelen 1000 (Ethernet)
RX packets 443 bytes 79503 (77.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 473 bytes 110689 (108.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 40 bytes 4138 (4.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40 bytes 4138 (4.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.2.3.215 netmask 255.255.255.0 broadcast 1.2.3.255
inet6 fe80::7418:57ff:fe62:b338 prefixlen 64 scopeid 0x20<link>
inet6 2001:41d0:602:4d7:: prefixlen 64 scopeid 0x0<global>
ether 76:18:57:62:b3:38 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21 bytes 1726 (1.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Seems that it is the main issue that vmbr is not able to get right address but anyway, I tried the following:
4. I've created a new container using default template: ubuntu-18.04-standard_18.04-1_amd64.tar.gz with the following network settings (copied from /etc/pve/lxc/101.conf:
net0: name=eth0,bridge=vmbr0,gw=192.168.2.1,hwaddr=CE:09:A2:39:0C:04,ip=192.168.2.101/24,type=veth
5. I've started the new container
6. Results of the route -n and ifconfig after container is started are as follows. Note that route to 192.168.2.0 disappeared after the container is started...
root@node03:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 1.2.3.254 0.0.0.0 UG 0 0 0 eno1
0.0.0.0 1.2.3.254 0.0.0.0 UG 0 0 0 vmbr0
1.2.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eno1
1.2.3.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
root@node03:~# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.2.3.215 netmask 255.255.255.0 broadcast 1.2.3.255
inet6 fe80::a6bf:1ff:fe1f:c81a prefixlen 64 scopeid 0x20<link>
ether a4:bf:01:1f:c8:1a txqueuelen 1000 (Ethernet)
RX packets 286739 bytes 428535201 (408.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 22047 bytes 4211409 (4.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 1897 bytes 2526171 (2.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1897 bytes 2526171 (2.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth100i0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:1c:30:9a:ef:ea txqueuelen 1000 (Ethernet)
RX packets 5 bytes 446 (446.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3 bytes 306 (306.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.2.3.215 netmask 255.255.255.0 broadcast 1.2.3.255
inet6 fe80::7418:57ff:fe62:b338 prefixlen 64 scopeid 0x20<link>
inet6 2001:41d0:602:4d7:: prefixlen 64 scopeid 0x0<global>
ether fe:1c:30:9a:ef:ea txqueuelen 1000 (Ethernet)
RX packets 5 bytes 376 (376.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 29 bytes 2466 (2.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
6. Ping 192.168.2.100 is not working:
root@node03:~# ping 192.168.2.100
PING 192.168.2.100 (192.168.2.100) 56(84) bytes of data.
^C
--- 192.168.2.100 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4074ms
7. /var/log/syslog shows the following during the start of container 100:
Oct 19 07:47:48 node03 systemd[1]: Starting PVE LXC Container: 100...
Oct 19 07:47:48 node03 kernel: [ 710.261728] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Oct 19 07:47:48 node03 kernel: [ 710.401613] kauditd_printk_skb: 1 callbacks suppressed
Oct 19 07:47:48 node03 kernel: [ 710.401614] audit: type=1400 audit(1539935268.766:13): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=4274 comm="apparmor_parser"
Oct 19 07:47:48 node03 systemd-timesyncd[968]: Network configuration changed, trying to establish connection.
Oct 19 07:47:48 node03 systemd-udevd[4277]: Could not generate persistent MAC address for vethMF1AMR: No such file or directory
Oct 19 07:47:48 node03 kernel: [ 710.409582] IPv6: ADDRCONF(NETDEV_UP): veth100i0: link is not ready
Oct 19 07:47:48 node03 systemd-timesyncd[968]: Synchronized to time server 213.251.128.249:123 (ntp.ovh.net).
Oct 19 07:47:49 node03 kernel: [ 710.648753] vmbr0: port 1(veth100i0) entered blocking state
Oct 19 07:47:49 node03 kernel: [ 710.648964] vmbr0: port 1(veth100i0) entered disabled state
Oct 19 07:47:49 node03 systemd-networkd[902]: vmbr0: Lost carrier
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Network configuration changed, trying to establish connection.
Oct 19 07:47:49 node03 kernel: [ 710.649278] device veth100i0 entered promiscuous mode
Oct 19 07:47:49 node03 kernel: [ 710.738600] eth0: renamed from vethMF1AMR
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Synchronized to time server 213.251.128.249:123 (ntp.ovh.net).
Oct 19 07:47:49 node03 systemd[1]: Started PVE LXC Container: 100.
Oct 19 07:47:49 node03 pvedaemon[1817]: <root@pam> end task UPID:node03:0000108E:00011551:5BC98C24:vzstart:100:root@pam: OK
Oct 19 07:47:49 node03 kernel: [ 711.224052] audit: type=1400 audit(1539935269.586:14): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=4410 comm="(networkd)" flags="
rw, rslave"
Oct 19 07:47:49 node03 kernel: [ 711.228863] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Oct 19 07:47:49 node03 kernel: [ 711.229090] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 19 07:47:49 node03 systemd-networkd[902]: veth100i0: Gained carrier
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Network configuration changed, trying to establish connection.
Oct 19 07:47:49 node03 kernel: [ 711.229369] vmbr0: port 1(veth100i0) entered blocking state
Oct 19 07:47:49 node03 kernel: [ 711.229577] vmbr0: port 1(veth100i0) entered forwarding state
Oct 19 07:47:49 node03 kernel: [ 711.255838] audit: type=1400 audit(1539935269.618:15): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=4444 comm="(resolved)" flags="
rw, rslave"
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Synchronized to time server 213.251.128.249:123 (ntp.ovh.net).
Oct 19 07:47:49 node03 systemd-networkd[902]: vmbr0: Gained carrier
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Network configuration changed, trying to establish connection.
Oct 19 07:47:49 node03 kernel: [ 711.446500] audit: type=1400 audit(1539935269.810:16): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4467 comm="apparmor_parser"
Oct 19 07:47:49 node03 kernel: [ 711.488478] audit: type=1400 audit(1539935269.850:17): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4470 comm="apparmor_parser"
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Synchronized to time server 213.251.128.249:123 (ntp.ovh.net).
Oct 19 07:47:49 node03 kernel: [ 711.527768] audit: type=1400 audit(1539935269.890:18): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4471 comm="apparmor_parser"
Oct 19 07:47:50 node03 kernel: [ 711.649971] audit: type=1400 audit(1539935270.014:19): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4482 comm="apparmor_parser"
Oct 19 07:47:50 node03 kernel: [ 711.691970] audit: type=1400 audit(1539935270.054:20): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4485 comm="apparmor_parser"
Oct 19 07:47:50 node03 kernel: [ 711.714665] audit: type=1400 audit(1539935270.078:21): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4480 comm="apparmor_parser"
Oct 19 07:47:51 node03 systemd-networkd[902]: vmbr0: Configured
Oct 19 07:47:51 node03 systemd-timesyncd[968]: Network configuration changed, trying to establish connection.
Any clues what is wrong? I have another server with Proxmox 4 and almost same configuration (the only difference is eth0 is used instead of eno1) and it works properly.
I have a OVH server with Proxmox 5.2 installed (image from OVH). Server has a single public IP address so I need to configure NAT for VM's (lxc containers). The problem is I'm not able to reach my machines within the internal (NAT) network.
What I did, in order (real IP address of my server is substituted with 1.2.3.215):
1. Fresh install of the proxmox 5.2 on OVH (Debian 9)
2. Changed /etc/network/interfaces to the following (according to https://pve.proxmox.com/wiki/Network_Configuration):
auto lo
iface lo inet loopback
auto eno1
#real IP adress
iface eno1 inet static
address 1.2.3.215
netmask 255.255.255.0
gateway 1.2.3.254
auto vmbr0
#private sub network
iface vmbr0 inet static
address 192.168.2.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '192.168.2.0/24' -o eno1 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.2.0/24' -o eno1 -j MASQUERADE
3. Reboot of the server
4. Results of the ifconfig and route -n after the reboot are as follows. Seems that: inet 1.2.3.215 for vmbr0 - as reported by ifconfig - is not OK as it says inet 1.2.3.215 intead of 192.168.2.1!
root@node03:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 1.2.3.254 0.0.0.0 UG 0 0 0 eno1
0.0.0.0 1.2.3.254 0.0.0.0 UG 0 0 0 vmbr0
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
1.2.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eno1
1.2.3.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
root@node03:~# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.2.3.215 netmask 255.255.255.0 broadcast 1.2.3.255
inet6 fe80::a6bf:1ff:fe1f:c81a prefixlen 64 scopeid 0x20<link>
ether a4:bf:01:1f:c8:1a txqueuelen 1000 (Ethernet)
RX packets 443 bytes 79503 (77.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 473 bytes 110689 (108.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 40 bytes 4138 (4.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40 bytes 4138 (4.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.2.3.215 netmask 255.255.255.0 broadcast 1.2.3.255
inet6 fe80::7418:57ff:fe62:b338 prefixlen 64 scopeid 0x20<link>
inet6 2001:41d0:602:4d7:: prefixlen 64 scopeid 0x0<global>
ether 76:18:57:62:b3:38 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21 bytes 1726 (1.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Seems that it is the main issue that vmbr is not able to get right address but anyway, I tried the following:
4. I've created a new container using default template: ubuntu-18.04-standard_18.04-1_amd64.tar.gz with the following network settings (copied from /etc/pve/lxc/101.conf:
net0: name=eth0,bridge=vmbr0,gw=192.168.2.1,hwaddr=CE:09:A2:39:0C:04,ip=192.168.2.101/24,type=veth
5. I've started the new container
6. Results of the route -n and ifconfig after container is started are as follows. Note that route to 192.168.2.0 disappeared after the container is started...
root@node03:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 1.2.3.254 0.0.0.0 UG 0 0 0 eno1
0.0.0.0 1.2.3.254 0.0.0.0 UG 0 0 0 vmbr0
1.2.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eno1
1.2.3.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
root@node03:~# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.2.3.215 netmask 255.255.255.0 broadcast 1.2.3.255
inet6 fe80::a6bf:1ff:fe1f:c81a prefixlen 64 scopeid 0x20<link>
ether a4:bf:01:1f:c8:1a txqueuelen 1000 (Ethernet)
RX packets 286739 bytes 428535201 (408.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 22047 bytes 4211409 (4.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 1897 bytes 2526171 (2.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1897 bytes 2526171 (2.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth100i0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:1c:30:9a:ef:ea txqueuelen 1000 (Ethernet)
RX packets 5 bytes 446 (446.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3 bytes 306 (306.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.2.3.215 netmask 255.255.255.0 broadcast 1.2.3.255
inet6 fe80::7418:57ff:fe62:b338 prefixlen 64 scopeid 0x20<link>
inet6 2001:41d0:602:4d7:: prefixlen 64 scopeid 0x0<global>
ether fe:1c:30:9a:ef:ea txqueuelen 1000 (Ethernet)
RX packets 5 bytes 376 (376.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 29 bytes 2466 (2.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
6. Ping 192.168.2.100 is not working:
root@node03:~# ping 192.168.2.100
PING 192.168.2.100 (192.168.2.100) 56(84) bytes of data.
^C
--- 192.168.2.100 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4074ms
7. /var/log/syslog shows the following during the start of container 100:
Oct 19 07:47:48 node03 systemd[1]: Starting PVE LXC Container: 100...
Oct 19 07:47:48 node03 kernel: [ 710.261728] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Oct 19 07:47:48 node03 kernel: [ 710.401613] kauditd_printk_skb: 1 callbacks suppressed
Oct 19 07:47:48 node03 kernel: [ 710.401614] audit: type=1400 audit(1539935268.766:13): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=4274 comm="apparmor_parser"
Oct 19 07:47:48 node03 systemd-timesyncd[968]: Network configuration changed, trying to establish connection.
Oct 19 07:47:48 node03 systemd-udevd[4277]: Could not generate persistent MAC address for vethMF1AMR: No such file or directory
Oct 19 07:47:48 node03 kernel: [ 710.409582] IPv6: ADDRCONF(NETDEV_UP): veth100i0: link is not ready
Oct 19 07:47:48 node03 systemd-timesyncd[968]: Synchronized to time server 213.251.128.249:123 (ntp.ovh.net).
Oct 19 07:47:49 node03 kernel: [ 710.648753] vmbr0: port 1(veth100i0) entered blocking state
Oct 19 07:47:49 node03 kernel: [ 710.648964] vmbr0: port 1(veth100i0) entered disabled state
Oct 19 07:47:49 node03 systemd-networkd[902]: vmbr0: Lost carrier
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Network configuration changed, trying to establish connection.
Oct 19 07:47:49 node03 kernel: [ 710.649278] device veth100i0 entered promiscuous mode
Oct 19 07:47:49 node03 kernel: [ 710.738600] eth0: renamed from vethMF1AMR
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Synchronized to time server 213.251.128.249:123 (ntp.ovh.net).
Oct 19 07:47:49 node03 systemd[1]: Started PVE LXC Container: 100.
Oct 19 07:47:49 node03 pvedaemon[1817]: <root@pam> end task UPID:node03:0000108E:00011551:5BC98C24:vzstart:100:root@pam: OK
Oct 19 07:47:49 node03 kernel: [ 711.224052] audit: type=1400 audit(1539935269.586:14): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=4410 comm="(networkd)" flags="
rw, rslave"
Oct 19 07:47:49 node03 kernel: [ 711.228863] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Oct 19 07:47:49 node03 kernel: [ 711.229090] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 19 07:47:49 node03 systemd-networkd[902]: veth100i0: Gained carrier
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Network configuration changed, trying to establish connection.
Oct 19 07:47:49 node03 kernel: [ 711.229369] vmbr0: port 1(veth100i0) entered blocking state
Oct 19 07:47:49 node03 kernel: [ 711.229577] vmbr0: port 1(veth100i0) entered forwarding state
Oct 19 07:47:49 node03 kernel: [ 711.255838] audit: type=1400 audit(1539935269.618:15): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=4444 comm="(resolved)" flags="
rw, rslave"
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Synchronized to time server 213.251.128.249:123 (ntp.ovh.net).
Oct 19 07:47:49 node03 systemd-networkd[902]: vmbr0: Gained carrier
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Network configuration changed, trying to establish connection.
Oct 19 07:47:49 node03 kernel: [ 711.446500] audit: type=1400 audit(1539935269.810:16): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4467 comm="apparmor_parser"
Oct 19 07:47:49 node03 kernel: [ 711.488478] audit: type=1400 audit(1539935269.850:17): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4470 comm="apparmor_parser"
Oct 19 07:47:49 node03 systemd-timesyncd[968]: Synchronized to time server 213.251.128.249:123 (ntp.ovh.net).
Oct 19 07:47:49 node03 kernel: [ 711.527768] audit: type=1400 audit(1539935269.890:18): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4471 comm="apparmor_parser"
Oct 19 07:47:50 node03 kernel: [ 711.649971] audit: type=1400 audit(1539935270.014:19): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4482 comm="apparmor_parser"
Oct 19 07:47:50 node03 kernel: [ 711.691970] audit: type=1400 audit(1539935270.054:20): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4485 comm="apparmor_parser"
Oct 19 07:47:50 node03 kernel: [ 711.714665] audit: type=1400 audit(1539935270.078:21): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined"
pid=4480 comm="apparmor_parser"
Oct 19 07:47:51 node03 systemd-networkd[902]: vmbr0: Configured
Oct 19 07:47:51 node03 systemd-timesyncd[968]: Network configuration changed, trying to establish connection.
Any clues what is wrong? I have another server with Proxmox 4 and almost same configuration (the only difference is eth0 is used instead of eno1) and it works properly.