[SOLVED] PVE 3.4 - Lost VLAN connectivity to CT

[MimCom]

Ran apt-get dist-upgrade on an old node yesterday as part of a migration plan. Upgrade went fine, no issues afterwards. About an hour ago, we lost external connectivity to a mailserver running in CT106 there. Node is connected to the router with a single GigE cable, with the main node address (192.168.44.20) on an untagged VLAN and also using 172.16.11.20 in tagged VLAN 11. CT106 uses 172.16.11.6 on eth0, configured as VLAN 11 in OVS on the node Router has 192.168.44.2 untagged and 172.16.11.1 on VLAN 11, both on the same physical Ethernet. We can access 192.168.44.20 from the router, but are unable to reach either the node or the CT address in VLAN 11.

OVS config appears correct (has not changed in 2+ years). Not sure how to diagnose further, but it does seem strange that this just occurred just a day after the dist-upgrade.

root@pve1:~# ovs-vsctl show
6a34fb44-2008-484b-8ebb-aa047967e02d
    Bridge "vmbr0"
        Port "vmbr0"
            Interface "vmbr0"
                type: internal
        Port "vlan11"
            tag: 11
            Interface "vlan11"
                type: internal
        Port "veth106.0"
            tag: 11
            Interface "veth106.0"
        Port "veth107.0"
            tag: 11
            Interface "veth107.0"
        Port "eth0"
            Interface "eth0"
        Port "vlan55"
            tag: 55
            Interface "vlan55"
                type: internal
    ovs_version: "2.3.2"

root@pve1:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
unifi.murray-ho *               255.255.255.255 UH    0      0        0 venet0
sipx.murray-hot *               255.255.255.255 UH    0      0        0 venet0
ns2.murray-hote *               255.255.255.255 UH    0      0        0 venet0
172.16.55.0     *               255.255.255.0   U     0      0        0 vlan55
192.168.33.0    192.168.44.2    255.255.255.0   UG    0      0        0 vmbr0
192.168.44.0    *               255.255.255.0   U     0      0        0 vmbr0
172.16.11.0     *               255.255.255.0   U     0      0        0 vlan11
default         pfsense.murray- 0.0.0.0         UG    0      0        0 vmbr0

root@pve1:~# ping 172.16.11.20
PING 172.16.11.20 (172.16.11.20) 56(84) bytes of data.
64 bytes from 172.16.11.20: icmp_req=1 ttl=64 time=0.027 ms
64 bytes from 172.16.11.20: icmp_req=2 ttl=64 time=0.023 ms
64 bytes from 172.16.11.20: icmp_req=3 ttl=64 time=0.023 ms
^C
--- 172.16.11.20 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.023/0.024/0.027/0.004 ms
root@pve1:~# ping 172.16.11.6
PING 172.16.11.6 (172.16.11.6) 56(84) bytes of data.
64 bytes from 172.16.11.6: icmp_req=1 ttl=64 time=0.130 ms
64 bytes from 172.16.11.6: icmp_req=2 ttl=64 time=0.026 ms
64 bytes from 172.16.11.6: icmp_req=3 ttl=64 time=0.033 ms
^C
--- 172.16.11.6 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.026/0.063/0.130/0.047 ms
root@pve1:~# ping 172.16.11.1
PING 172.16.11.1 (172.16.11.1) 56(84) bytes of data.
From 172.16.11.20 icmp_seq=2 Destination Host Unreachable
From 172.16.11.20 icmp_seq=3 Destination Host Unreachable
From 172.16.11.20 icmp_seq=4 Destination Host Unreachable
^C
--- 172.16.11.1 ping statistics ---
6 packets transmitted, 0 received, +3 errors, 100% packet loss, time 5000ms
pipe 3

 

[MimCom]

Took some time re-learning the (not quite standard) Cisco SG300 command line syntax for me to figure out that a clueless maintenance guy had moved the server NIC cable from ge3 to ge5. Added vlan11 to that and everything is back online.