OpenvSwitch Failing VE4.4

[Marvin]

Upgraded to VE 4.4 with subscription. System was working fine before. Now openvswitch is failing.
1. vmbr0 is popping in and out of "promiscuous mode" (as seen in /var/log/syslog) causing intermittent connection to console and web gui. I can only get to it now from server's KVM.

Reply from 10.94.100.230: bytes=32 time<1ms TTL=63
Request timed out.
Reply from 10.94.100.230: bytes=32 time<1ms TTL=63
Request timed out.
Reply from 10.94.100.230: bytes=32 time<1ms TTL=63
Request timed out.
Request timed out.
Reply from 10.94.100.230: bytes=32 time<1ms TTL=63
Request timed out.
Request timed out.
Request timed out.
Reply from 10.94.100.230: bytes=32 time<1ms TTL=63
Reply from 10.94.100.230: bytes=32 time<1ms TTL=63
Request timed out.​

2. Several internal vm networks do not exists, vm's fail to start because missing vmbrX's.
3. /etc/network/interfaces file is same as before.
4. ovs-vsctl show does show all the vmbrX interfaces as expected.
5. ovs-vsctl show ovs_version: "2.6.0"
6. /var/log/syslog also shows reoccurring "/etc/openvswitch/conf.db does not exist" -- followed by "creating empty database" every 5 seconds.
7. dpkg -l shows openvswtich-switch and -common 2.6.0-2

This is our production server and i cannot bring it back up! I do not know what else to do, please help.

 

[Jason Hamm]

Have you tried replacing the conf.db file with a known good backup?

 

[Marvin]

Have you tried replacing the conf.db file with a known good backup?

The conf.db appears to be a dynamic system generated file. It is constantly being overwritten. There is no static "known good backup".

 

[Marvin]

Here's something interesting. There are several vm's that also use the same vmbrX interface to connect to the network. If i start the vm's, connection to them is solid. It appear only to be intermittent connecting to the proxmox console interface (and web gui).

Edit: if i log into one of the vm's using the same network interface (same vmbrX), it's connection to the Proxmox console (and GUI) shows the same intermittent connection. So the vm's are fine connected to thru openvswitch. It seems it's only the Proxmox console that has the issue.

Edit: If i ping from the Proxmox console to a vm, the rest of the world looses contact with that vm.

Edit: It's now midnight here. I am finding that vm's that seem to work for a while will suddenly loose all connection. then mysteriously come back. I am very tired and frustrated. Deciding to upgrade from 4.2 to 4.4 was not a good choice. Now nothing works and i will be in trouble in the morning. Is there any help to be had?

 

[Jason Hamm]

Have you tried replacing the conf.db file with a known good backup?

Seems that I have a lot to learn...

 

[fabian]

please post the complete log, output of "pveversion -v", network configuration and guest configuration files.

 

[Marvin]

please post the complete log, output of "pveversion -v", network configuration and guest configuration files.

When you say complete log, do you mean the syslog file? It is currently almost a gig! I tried to gzip but not enough room on the disk.

 

[Marvin]

all except the complete log (because i'm nor sure which you mean).

 

[Marvin]

a few more.

 

[fabian]

When you say complete log, do you mean the syslog file? It is currently almost a gig! I tried to gzip but not enough room on the disk.

I think the part from before the upgrade starting until a little bit into the problems should be enough.

 

[fabian]

also, I remember user reporting issues when upgrading to OVS 2.4, that where solved by purging the package and re-installing it, so you could give this a try (note: over the iKVM console or similar, not over the network). note that this will remove the log files from /var/log/openvswitch, so it makes sense to back them up first.

# mkdir ovs-tmp
# apt-get download openvswitch-switch
# cp -v /var/cache/apt/archives/openvswitch-switch*.deb ovs-tmp
‘/var/cache/apt/archives/openvswitch-switch_2.6.0-2_amd64.deb’ -> ‘ovs-tmp/openvswitch-switch_2.6.0-2_amd64.deb’
# apt-get purge openvswitch-switch
# dpkg -i ovs-tmp/openvswitch-switch_2.6.0-2_amd64.deb

 

[Marvin]

Yes i had seen the post where that was recommended and i tried that. But i can do it again...

 

[fabian]

I just noticed the following in your network configuration, which seems very wrong to me:

...

auto vmbr0
iface vmbr0 inet static
    address  10.94.100.230
    netmask  255.255.255.0
    gateway  10.94.100.1
    ovs_type OVSBridge
    ovs_ports eth0 
    bridge_ports eth0
    bridge_stp off

...

notice how you are mixing linux bridge and OVS bridge parameters there? I think this should be

auto vmbr0
iface vmbr0 inet static
    address  10.94.100.230
    netmask  255.255.255.0
    gateway  10.94.100.1
    ovs_type OVSBridge
    ovs_ports eth0

instead.. (needs a reboot to apply)

 

[Marvin]

YES!! The linux bridge parameters in the OVS parameters was it! It is now running 100%!

Thank you Fabian, you saved my sanity. It is 3:30am here and now i can go home and go to sleep!