no containers start after MOX7 upgrade, network.c:netdev_configure_server_empty:1240 - Invalid argument - Custom loopback device names not supported

[jimays]

No containers start after upgrade from Debian10/ProxMOX6.3 to Debian11/ProxMOX7.1.

Surely it has to do with the names of our bridges and veth devices.

lxc-start -n 407083 --logpriority=10 -F lxc-start: 407083: network.c: netdev_configure_server_empty: 1240 Invalid argument - Custom loopback device names not supported lxc-start: 407083: network.c: lxc_create_network_priv: 3413 Invalid argument - Failed to create network device lxc-start: 407083: start.c: lxc_spawn: 1837 Failed to create the network lxc-start: 407083: start.c: __lxc_start: 2068 Failed to spawn container "407083" lxc-start: 407083: tools/lxc_start.c: main: 306 The container failed to start lxc-start: 407083: tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

Code from network.c is very basic:

if (!strequal(netdev->name, "lo")) return syserror_set(-EINVAL, "Custom loopback device names not supported");

A piece of the strace which may indicate the issue is with a veth device:

ioctl(9, SIOCGIFINDEX, {ifr_name="veth407083i1", }) = 0 close(9) = 0 sendmsg(5, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=32, type=RTM_NEWLINK, flags=NLM_F_REQUEST|NLM_F_ACK, seq=0, pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=if_nametoindex("veth407083i1"), ifi_flags=IFF_UP, ifi_change=0x1}}, iov_len=32}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 32 recvmsg(5, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=36, type=NLMSG_ERROR, flags=NLM_F_CAPPED, seq=0, pid=116904}, {error=0, msg={len=32, type=RTM_NEWLINK, flags=NLM_F_REQUEST|NLM_F_ACK, seq=0, pid=0}}}, iov_len=8208}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 36 close(5) = 0 pipe2([5, 9], O_CLOEXEC) = 0 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f95ed995a90) = 118117 close(9) = 0 fcntl(5, F_GETFL) = 0 (flags O_RDONLY) read(5, "", 4095) = 0 wait4(118117, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 118117 close(5) = 0 write(3, "lxc-start 407083 20220402174353.324 ERROR network - network.c:netdev_configure_server_empty:1240 - Invalid argument - Custom loopback device names not supported\n", 164) = 164 write(2, "lxc-start: 407083: ", 19lxc-start: 407083: ) = 19 write(2, "network.c: netdev_configure_server_empty: 1240 ", 47network.c: netdev_configure_server_empty: 1240 ) = 47 write(2, "Invalid argument - Custom loopback device names not supported", 61Invalid argument - Custom loopback device names not supported) = 61 write(2, "\n", 1 ) = 1 write(3, "lxc-start 407083 20220402174353.457 ERROR network - network.c:lxc_create_network_priv:3413 - Invalid argument - Failed to create network device\n", 147) = 147

Example 407083.conf:

arch: amd64 hostname: dns407-083.tru60.net memory: 512 mp0: /tru,mp=/tru,backup=0 mp1: /vz/tmp/407083,mp=/tmp,backup=0 mp2: /vz/var/407083,mp=/var,backup=0 mp3: /vz/var.log/407083,mp=/var/log,backup=0 nameserver: 192.168.60.83 192.168.56.149 net0: name=ethT60,bridge=vzbrT60,gw=192.168.60.254,hwaddr=66:02:56:39:6F:7E,ip=192.168.60.83/23,type=veth net1: name=ethTcore62,bridge=vzbrTcore62,hwaddr=FE:91:34:4E:36:74,ip=192.168.62.83/23,type=veth onboot: 1 ostype: debian rootfs: /vz/root/407083 searchdomain: idsp56.net startup: order=10 swap: 512 tty: 0 lxc.net.9.script.up: /tru/mnt/realm-net-tools/bin/realm-net-sync-lxc-routes lxc.net.0.script.down: /tru/mnt/realm-net-tools/bin/realm-net-sync-lxc-routes

There are many bridges running on the server handling various networks among the containers.
The above example config is for a container that should have a presence on the vzbrT60 and vzbrTcore62 bridges.

ip link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:0d:be:c6 brd ff:ff:ff:ff:ff:ff 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:0d:be:c7 brd ff:ff:ff:ff:ff:ff 4: vzbrR56: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 36:f4:11:14:67:64 brd ff:ff:ff:ff:ff:ff 5: vzbrI56: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 6: vzbrIcore58: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 7: vzbrT60: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 8: vzbrTcore62: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 9: vzbrW56: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 10: vzbrWcore58: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 11: vzbrMNT: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 9e:67:0c:8e:f6:c7 brd ff:ff:ff:ff:ff:ff

 

[jimays]

Solved.
The issue was with:

lxc.net.9.script.up: /tru/mnt/realm-net-tools/bin/realm-net-sync-lxc-routes

In ProxMox 6.3, was able to use lxc.net.9.script.up and the hook would run
after the last network was brought up, regardless of how many nets were specified.

In ProxMox 7.1 it is no longer possible to use lxc.net.9.script.up when the last
network listed is net3. Changed to lxc.net.3.script.up and all is well.