[SOLVED] [OVS] veth shows NO-CARRIER on LXC start

Don Daniello · Aug 1, 2019

The problem is that when starting LXC container first time, creating LXC container OR after PVE host is rebooted (and therefore OVS configuration is reset, since PVE does not use persistent OVS DB), the virtual interface plugged into OVS port (vmbr0 here with VLAN tag 4001) does not work.

Here is a log of such LXC container starting:

Code:

Aug 01 02:35:54 ex2 pvedaemon[17256]: starting CT 301: UPID:ex2:00004368:0004361A:5D4233EA:vzstart:301:root@pam:
Aug 01 02:35:54 ex2 pvedaemon[2062]: <root@pam> starting task UPID:ex2:00004368:0004361A:5D4233EA:vzstart:301:root@pam:
Aug 01 02:35:54 ex2 systemd[1]: Created slice PVE LXC Container Slice.
Aug 01 02:35:54 ex2 systemd[1]: Starting PVE LXC Container: 301...
Aug 01 02:35:55 ex2 audit[17285]: AVC apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-301_</var/lib/lxc>" pid=17285 comm="apparmor_parser"
Aug 01 02:35:55 ex2 kernel: kauditd_printk_skb: 6 callbacks suppressed
Aug 01 02:35:55 ex2 kernel: audit: type=1400 audit(1564619755.142:18): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-301_</var/lib/lxc>" pid=17285 comm="apparmor_parser"
Aug 01 02:35:55 ex2 systemd-udevd[17257]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 01 02:35:55 ex2 systemd-udevd[17257]: Using default interface naming scheme 'v240'.
Aug 01 02:35:55 ex2 systemd-udevd[17257]: Could not generate persistent MAC address for veth5UHAU3: No such file or directory
Aug 01 02:35:55 ex2 systemd-udevd[17260]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 01 02:35:55 ex2 systemd-udevd[17260]: Using default interface naming scheme 'v240'.
Aug 01 02:35:55 ex2 ovs-vsctl[17301]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port veth301i0
Aug 01 02:35:55 ex2 ovs-vsctl[17301]: ovs|00002|db_ctl_base|ERR|no port named veth301i0
Aug 01 02:35:55 ex2 ovs-vsctl[17302]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln301i0
Aug 01 02:35:55 ex2 ovs-vsctl[17302]: ovs|00002|db_ctl_base|ERR|no port named fwln301i0
Aug 01 02:35:55 ex2 ovs-vsctl[17303]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-port vmbr0 veth301i0 tag=4001
Aug 01 02:35:55 ex2 kernel: netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
Aug 01 02:35:55 ex2 kernel: device veth301i0 entered promiscuous mode
Aug 01 02:35:55 ex2 kernel: eth0: renamed from veth5UHAU3
Aug 01 02:35:55 ex2 systemd[1]: Started PVE LXC Container: 301.
Aug 01 02:35:55 ex2 pvedaemon[2062]: <root@pam> end task UPID:ex2:00004368:0004361A:5D4233EA:vzstart:301:root@pam: OK

How it looks on the inside (guest):

Code:

[root@test-vrack ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 6a:8e:cb:a6:ee:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0

How it looks on the outside (host):

Code:

root@ex2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
    link/ether b4:2e:99:47:f9:78 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b62e:99ff:fe47:f978/64 scope link 
       valid_lft forever preferred_lft forever
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3a:a1:e2:af:9d:2f brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether b4:2e:99:47:f9:78 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b62e:99ff:fe47:f978/64 scope link 
       valid_lft forever preferred_lft forever
5: vlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether b4:2e:99:47:f9:78 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/27 brd 1.2.3.127 scope global vlan1
       valid_lft forever preferred_lft forever
    inet6 fe80::8426:21ff:fe63:1705/64 scope link 
       valid_lft forever preferred_lft forever
6: vlan4000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether da:cf:02:67:9d:28 brd ff:ff:ff:ff:ff:ff
    inet 192.168.25.2/24 brd 192.168.25.255 scope global vlan4000
       valid_lft forever preferred_lft forever
    inet6 fe80::d8cf:2ff:fe67:9d28/64 scope link 
       valid_lft forever preferred_lft forever
8: veth301i0@if7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1400 qdisc noqueue master ovs-system state LOWERLAYERDOWN group default qlen 1000
    link/ether fe:33:83:a6:63:32 brd ff:ff:ff:ff:ff:ff link-netnsid 0

Interestingly, when I change any parameter of the network interface (e.g. MAC address), the container's networking is fixed.
syslog:

Code:

Aug 01 02:43:37 ex2 kernel: device veth301i0 left promiscuous mode
Aug 01 02:43:37 ex2 ovs-vsctl[26883]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln301i0
Aug 01 02:43:37 ex2 ovs-vsctl[26883]: ovs|00002|db_ctl_base|ERR|no port named fwln301i0
Aug 01 02:43:37 ex2 ovs-vsctl[26884]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port veth301i0
Aug 01 02:43:37 ex2 systemd-udevd[26873]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 01 02:43:37 ex2 systemd-udevd[26875]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 01 02:43:37 ex2 systemd-udevd[26873]: Using default interface naming scheme 'v240'.
Aug 01 02:43:37 ex2 systemd-udevd[26875]: Using default interface naming scheme 'v240'.
Aug 01 02:43:37 ex2 systemd-udevd[26875]: Could not generate persistent MAC address for veth301i0: No such file or directory
Aug 01 02:43:37 ex2 ovs-vsctl[26901]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port veth301i0
Aug 01 02:43:37 ex2 ovs-vsctl[26901]: ovs|00002|db_ctl_base|ERR|no port named veth301i0
Aug 01 02:43:37 ex2 ovs-vsctl[26902]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln301i0
Aug 01 02:43:37 ex2 ovs-vsctl[26902]: ovs|00002|db_ctl_base|ERR|no port named fwln301i0
Aug 01 02:43:37 ex2 ovs-vsctl[26903]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-port vmbr0 veth301i0 tag=4001
Aug 01 02:43:37 ex2 kernel: netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
Aug 01 02:43:37 ex2 kernel: device veth301i0 entered promiscuous mode
Aug 01 02:43:38 ex2 kernel: eth0: renamed from veth301i0p

Guest networking is immediately fixed:

Code:

[root@test-vrack ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
9: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:8e:cb:a6:ee:1e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 5.6.7.8/28 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::688e:cbff:fea6:ee1e/64 scope link 
       valid_lft forever preferred_lft forever

Host no longer shows "NO-CARRIER" on the veth:

Code:

root@ex2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
    link/ether b4:2e:99:47:f9:78 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b62e:99ff:fe47:f978/64 scope link 
       valid_lft forever preferred_lft forever
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3a:a1:e2:af:9d:2f brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether b4:2e:99:47:f9:78 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b62e:99ff:fe47:f978/64 scope link 
       valid_lft forever preferred_lft forever
5: vlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether b4:2e:99:47:f9:78 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/27 brd 1.2.3.127 scope global vlan1
       valid_lft forever preferred_lft forever
    inet6 fe80::8426:21ff:fe63:1705/64 scope link 
       valid_lft forever preferred_lft forever
6: vlan4000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether da:cf:02:67:9d:28 brd ff:ff:ff:ff:ff:ff
    inet 192.168.25.2/24 brd 192.168.25.255 scope global vlan4000
       valid_lft forever preferred_lft forever
    inet6 fe80::d8cf:2ff:fe67:9d28/64 scope link 
       valid_lft forever preferred_lft forever
10: veth301i0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 2e:12:1f:d6:c6:6d brd ff:ff:ff:ff:ff:ff link-netnsid 0

My conclusion for now is that the setup can work but PVE isn't doing all the right things every time and the network interface isn't fully started. Any ideas of what could be the problem?

Here is /etc/network/interfaces on the host:

Code:

auto lo
iface lo inet loopback

allow-vmbr0 eno1
iface eno1 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr0
        ovs_options vlan_mode=native-untagged

auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports eno1 vlan1 vlan4000
        ovs_extra set interface ${IFACE} other-config:hwaddr=\"$(cat /sys/class/net/eno1/address)\"
        mtu 1400

allow-vmbr0 vlan1
iface vlan1 inet static
        address  1.2.3.4
        netmask  27
        gateway  1.2.3.9
        mtu 1500
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options vlan_mode=access
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif other-config:hwaddr=\"$(cat /sys/class/net/eno1/address)\"
        post-up ovs-vsctl set interface ${IFACE} mac=\"$(cat /sys/class/net/eno1/address)\"

allow-vmbr0 vlan4000
iface vlan4000 inet static
        address  192.168.25.2
        netmask  24
        mtu 1400
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=4000

Alwin · Aug 1, 2019

Don Daniello said:
Aug 01 02:35:55 ex2 ovs-vsctl[17303]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-port vmbr0 veth301i0 tag=4001

I guess it may be that the tag 4001 does not exist (as not in the interfaces file) and is created on first execution. A subsequent call then works.

Don Daniello · Aug 1, 2019

Alwin said:
I guess it may be that the tag 4001 does not exist (as not in the interfaces file) and is created on first execution. A subsequent call then works.

Is there a way I need to "initialize" tag (4001) in OVS specifically? It's true that I don't use this tag on the host, it's only for the instances in this case. I'll bring up an interface on the host with that tag to give it a try.

EDIT: No, adding an interface (vlan4001) on the host and having it "up" before starting the LXC container makes no difference.

Alwin · Aug 2, 2019

Maybe there are more messages to see with increased logging?
http://www.openvswitch.org/support/dist-docs/ovs-appctl.8.txt

Don Daniello · Aug 2, 2019

Alwin said:
Maybe there are more messages to see with increased logging?
http://www.openvswitch.org/support/dist-docs/ovs-appctl.8.txt

Thanks @Alwin for the pointers. It turns out that the behavior is isolated to the latest Archlinux template that PVE downloads. On Ubuntu it works fine, therefore it's not OVS/host network problem.

Alwin · Aug 2, 2019

Don Daniello said:
Thanks @Alwin for the pointers. It turns out that the behavior is isolated to the latest Archlinux template that PVE downloads. On Ubuntu it works fine, therefore it's not OVS/host network problem.

Possible that archlinux networkd has some issues in setting up the interface. It seems the wiki knows already it symptom.
https://wiki.archlinux.org/index.ph...work-connection_with_veth_in_container_config

Search

Search

[SOLVED] [OVS] veth shows NO-CARRIER on LXC start

Don Daniello

Active Member

Alwin

Proxmox Retired Staff

Don Daniello

Active Member

Alwin

Proxmox Retired Staff

Don Daniello

Active Member

Alwin

Proxmox Retired Staff