Proxmox 7.1 Upgrade Issues

[bzdigblig]

I upgraded my Proxmox installation to 7.1 today and that seems to have just blown up my Proxmox installation...I can't even access the web interface and multiple services (such as pve-cluster, pve-firewall, pve-guests and more) wouldn't even start unless I disabled IPv6 for some weird reason.

I can supply more detail about the particular issues if anybody wants, but I'm kinda looking past that now.

What's the best way to back up my config so I can nuke this box and start over? Is there a current list of files and whatnot that I can just copy elsewhere?

Alternatively, can I reinstall while keeping my current config? Any other options?

 

[vater]

https://forum.proxmox.com/threads/92852/#post-404375

 

[t.lamprecht]

Hi,

I can supply more detail about the particular issues if anybody wants

A few things could be interesting to know to see if the situation you run into could be avoided for others.

• From what version did you upgrade (some 7.0 or 6.4)
• Cluster or single-node?
• ss -tlpn
• cat /etc/default/pveproxy
• cat /etc/network/interfaces (redact any public IPs)
• Checking the journal (e.g., the one from current boot: journalctl -b) for errors and posting them here. You can also upload the whole journal text output here as attachment for us to check, e.g., create a zstd compressed text file journalctl -b | zstd >"journal-$(date -Is).txt.zst"

 

[pete13]

@t.lamprecht : Unable to start existing Windows VMs with Kernel 5.13: My host hw: Asus WS X570 ACE, CPU AMD Ryzen 9 5900X, 64 GB RAM, 2x 2TB SSD (VM1, VM2), 6TB HDD (datasink)
Attached is an example vm configuration. I tried already to change machine type to 6.0, 6.1 without any success.

 

[bzdigblig]

Hi,

A few things could be interesting to know to see if the situation you run into could be avoided for others.

• From what version did you upgrade (some 7.0 or 6.4)
• Cluster or single-node?
• ss -tlpn
• cat /etc/default/pveproxy
• cat /etc/network/interfaces (redact any public IPs)
• Checking the journal (e.g., the one from current boot: journalctl -b) for errors and posting them here. You can also upload the whole journal text output here as attachment for us to check, e.g., create a zstd compressed text file journalctl -b | zstd >"journal-$(date -Is).txt.zst"

I last updated about a week...week and a half ago, so it was some 7.0 version.

Single node.

I can only access the server directly through the console so it's a little obnoxious to work with.

I've attached a screenshot of ss -tlpn.

Pveproxy only contains the following line, which I only added yesterday to see if anything would behave any differently:

LISTEN_IP="0.0.0.0"

Interfaces contains the following. eno2 is the only connected interface right now. I was going through vmbr0, but I set up vmbr1 to again see if anything behaved differently:

auto lo
iface lo inet loopback

auto eno3
iface eno3 inet manual

auto eno4
iface eno4 inet manual

auto eno1
iface eno1 inet manual
    mtu 9000

auto eno2
iface eno2 inet manual
    mtu 1500

auto enp130s0f0
iface enp130s0f0 inet manual

auto enp130s0f1
iface enp130s0f1 inet manual

auto enp3s0f0
iface enp3s0f0 inet manual

auto enp3s0f1
iface enp3s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
#    address 192.168.2.106/24
#    gateway 192.168.2.1
#    bridge-ports eno1 eno2
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    mtu 1500

auto vmbr1
iface vmbr1 inet static
    address 192.168.2.106/24   
    gateway 192.168.2.1
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    mtu 1500

There's gonna be various iSCSI errors in the journal I've attached...I was working on some multipath stuff and there's only one working path at the moment, but I have no idea what else is actually broken as a result of this borked update

 

[t.lamprecht]

I can only access the server directly through the console so it's a little obnoxious to work with.

Thanks for the details, will need a bit to check this, another question though: network is available? i.e., pinging the local gateway (or other LAN hosts) or some internet address works?

 

[bzdigblig]

Thanks for the details, will need a bit to check this, another question though: network is available? i.e., pinging the local gateway (or other LAN hosts) or some internet address works?

Yes mostly. There's some weird, semi-regular packet loss that's occurring though.

If I do an ifconfig, there's little to no dropped packets on the physical interface, but there's a pile of 'em on the bridge that has this interface as a member. I think that's why I ended up making a second bridge and adding eno2 to it, just to see if there'd be any different behavior.

 

[t.lamprecht]

Yes mostly. There's some weird, semi-regular packet loss that's occurring though.

Duplicate IP address or the like?

 

[bzdigblig]

Duplicate IP address or the like?

Not that I'm aware of, but I'm in the process of either confirming it or ruling that out. I think the only way a duplicate IP could exist is if somebody added another device to the network without my knowledge and chose that same IP address for their device.

EDIT: WOWWWWWW good call. That's exactly what happened. I guess our lack of available network jacks in our shop office got on somebody's nerves and a TP-Link switch made its way onto somebody's desk.

 

[bzdigblig]

As a followup, I got rid of that device, rebooted the server, ran

for package in $(apt list --installed | grep -P ".*(?=/)" -o); do apt install --reinstall -y $package; done

to reinstall all packages, rebooted the server again, undid a lot of the stuff I disabled for troubleshooting purposes, and things seem good to go for the time being.

 

[t.lamprecht]

Not that I'm aware of, but I'm in the process of either confirming it or ruling that out. I think the only way a duplicate IP could exist is if somebody added another device to the network without my knowledge and chose that same IP address for their device.

EDIT: WOWWWWWW good call. That's exactly what happened. I guess our lack of available network jacks in our shop office got on somebody's nerves and a TP-Link switch made its way onto somebody's desk.

Yeah, spotty local network is all I need to hear to check for address duplication, been there too often myself
Anyhow, glad you could solve.

for package in $(apt list --installed | grep -P ".*(?=/)" -o); do apt install --reinstall -y $package; done

to reinstall all packages, rebooted the server again, undid a lot of the stuff I disabled for troubleshooting purposes, and things seem good to go for the time being.

Just a little note regarding that manual reinstallation, it's not bad or the like, just possibly something that's maybe good to have in mind.

With apt the dependencies of packages you install are normally marked as automatically installed, so if such a dependency is not required anymore, e.g., due some newer update that made it superfluous or switched to a newer library ABI version, apt offers to clean that up via the apt autoremove command.

Now if you actively reinstall all that is installed apt will mark everything as manually installed, in which case no auto-remove is offered (the admin explicitly requested those packages after all, they did not just come in transitively). As said, normally not an actual issue, after a few major upgrades and repeating the reinstall-all a few times you may end up with quite a few extra packages that wouldn't be required anymore.

 

[bzdigblig]

Thank you for the information.

This reinstall process isn't anything I plan on regularly doing. I'm doing a bunch of Proxmox testing and once this test environment is running the way I want, I'll probably end up wiping everything and redoing it all from scratch when we put it in production, just to make sure we're running on a clean config.

 

[axsim]

Same here I have a Dell Poweredge R520. It was running Proxmox 6.4.14 just fine. I decided to upgrade to 7 and it blew up on me. The server boots and I can still connect via ssh. But the GUI is dead. I followed the instructions from here https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0#Add_the_Proxmox_VE_7_Package_Repository Everything looked great until I have a now dead Proxmox server.
I saw this "Then remove all packages which are currently installed from that repository. Following this, start the upgrade procedure again." but have no clue how to "remove all packages which are currently installed from that repository"