PVE8: breaks networking during upgrade leaving node unusable

[dlasher]

Upgraded my tiny lab cluster today - 3 headless nodes of miniPC, (2) with C2930, and (1) with N3160. Identical drives, ram. dual NIC, LACP, managed by open-vswitch, with multiple vlan's (including management) across the bundle.

all had green pve7to8 reports.

All dumped networking at the same spot:

```
Selecting previously unselected package pve-kernel-6.2.16-3-pve.
Preparing to unpack .../121-pve-kernel-6.2.16-3-pve_6.2.16-3_amd64.deb ...
Unpacking pve-kernel-6.2.16-3-pve (6.2.16-3) ...
client_loop: send disconnect: Broken pipe
Mac-Pro:~ $
Progress: [ 55%] [#################################################################################################################..
```

Not knowing the status of the upgrade/etc, I powered cycled one of the boxes. Gave it 5 minutes, no pings/etc. Rebooted the second one, same deal.

Hooked up keyboard to the 1st node, boot failure, cannot load kernel. Same with second box.

Had NOT power cycled the 3rd box, so I hooked up keyboard/hdmi, and it had zero networking, for some reason alll the open-vswitch modules had been disabled/stopped which killed networking, which stopped the install (I was upgrading over SSH) mid way through. I was able to resume dpkg, which completed all the install steps, and after a reboot, and lots more patching, everything appears to be back up.

This leaves me with (2) boxes however, that won't boot any more, same "cannot load kernel" message. I pulled the boot drive, and rsync'd the /boot folder over from the (#3) node, nodes #1/#2 still won't boot with same error message.

So three things:

1. How do I get node #1 or #2 up and running again so I can complete the upgrade? (see below : https://forum.proxmox.com/threads/p...ade-leaving-node-unusable.130055/#post-570201)
2. why is openvswitch being stopped/restarted/etc during the upgrade process?
3. given #2 - I recommend a sanity check, and error thrown during the pve7to8 script, so users understand if they use open-vswitch, and they are not performing the upgrade from the console, they'll be left with an unusable node.




* https://forum.proxmox.com/threads/download-pve8-packages-and-continue-upgrade-offline.129804/

 

[dlasher]

/etc/network/interfaces

auto lo
iface lo inet loopback

auto ens1
iface ens1 inet manual
        mtu 1500
        ovs_mtu 1500

auto enp3s0
iface enp3s0 inet manual
    mtu 1500
    ovs_mtu 1500

auto bond0
iface bond0 inet manual
        ovs_bridge vmbr0
        ovs_type OVSBond
        ovs_bonds enp3s0 ens1
        ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast
        ovs_options tag=1 vlan_mode=native-untagged
        pre-up ifconfig enp3s0 mtu 1500
        pre-up ifconfig ens1 mtu 1500

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0
        ovs_mtu 8896

#MGMT INTERFACE
auto vlan5
iface vlan5 inet static
        address X.X.X.108/24
        gateway X.X.X.1
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=5
        ovs_mtu 1500



auto vlan21
iface vlan21 inet static
        address X.X.X.108/24
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=21
        ovs_mtu 1500



auto vlan500
iface vlan500 inet static
        address X.X.X.108/24
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=500
        ovs_mtu 1500

 

[Dunuin]

PVE restarts services when doing major grades. Thats why there is this warning that you shouldn't do this over SSH as the connection might fail while upgrading:

recommended to have access over a host independent channel like iKVM/IPMI or physical access.
If only SSH is available we recommend testing the upgrade on an identical, but non-production machine first.

 

[dlasher]

>https://pve.proxmox.com/wiki/Upgrade_from_7_to_8 said:
>recommended to have access over a host independent channel like iKVM/IPMI or physical access.
>f only SSH is available we recommend testing the upgrade on an identical, but non-production machine first.


Far too mild. Should say: "If only SSH is available you cannot proceed with this upgrade."

If the upgrade process breaks all networking, which breaks your SSH connection, being on console is no longer recommended, it's required.

 

[dlasher]

Node #1 back online - going to work on node #2 next.

Steps:
* attach drive to other linux machine (or boot off recovery CD/etc)
* mkdir /tmp/1
* mount /dev/mapper/pve-root /tmp/1
* cd /tmp/1
* mount -t proc /proc proc/
* mount --rbind /sys sys/
* mount --rbind /dev dev/
* chroot /tmp/1
* dpkg --configure -a
*** answer the questions as appropriate
* exit
* reboot

Back to life.

EDIT: repeated steps on node #2 - both back in service now.

 

[dlasher]

All of my production nodes are running open-vswitch in a similar fashion to my lab nodes. This means I **must** have kvm/console for every node, which is a change to previous upgrade processes.

 

[Neobin]

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=4801

 

[dlasher]

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=4801

Exactly! Thank you!

 

[JohnR82]

I just installed ProxMox VE 8 after reboot I have the message

installation of package pve-kernel-6.2.16-3-pve_6.2.16-3_amd64.deb failed

ProxMox could not be installed

So I googled the the package and landed here

From reading these posts it states don't use SSH use KVM IPMI

I was using KVM IPMI

This was not an Upgrade but a first install

Also, to the OP I'm not trying to intrude on your post

I just thought it may be relevent I was using KVM IPMI

And Have Simial Issues. I 'm Going to attempt to instal Version 7

Though I am curious to see the final outcome of this Thread Hope maybe the added info helps.