Proxmox 8.2 broke after upgrade, now proxy error 500

[Joris L.]

After a mere pveupdate+pveupgrade the standalone Proxmox VE is now broken (again)

Things that broke are :

• a kernel module .conf file (fix= `rm /etc/modprobe.d/e1000e.conf`)
• the GPU udev rules suddenly contained GROUP== instead of GROUP= (fix: remove = from ==)
  • not 100% sure but the monitor attached now duplicates the GPU VM output again, thought that was gone due to 'nomodeset'
• pveproxy
  • "Connection error 500: RPCEnvironment init request failed: Unable to load access control list: Connection refused" and also " pvx pveproxy[9734]: ipcc_send_rec[3] failed: Connection refused"
  • this seems like an old sore with each upgrade
  • check /etc/hosts and /etc/hostname, all is well as before
    
    
  • restarted pvestatd pve-cluster pveproxy
  • rebooted the machine

Up to this point there's no sign of improvement.
The state of the VM seems reasonably stable but the UI is messed up.
For the GPU VM it seems the state is unreliably reported, seems running when stopped and vice versa.

Most important is help with the pveproxy issue(s)

Br

JL

 

[gfngfn256]

After a mere pveupdate+pveupgrade

How did you perform the update & upgrade?

 

[Joris L.]

How did you perform the update & upgrade?

I don't understand what you're asking

# pveupdate
# pveupgrade

 

[gfngfn256]

Normally, I would update/upgrade from within the GUI. I assume your repos are correctly set for your Enterprise sub. & not on any devs.

 

[Joris L.]

Normally, I would update/upgrade from within the GUI. I assume your repos are correctly set for your Enterprise sub. & not on any devs.

same as before, don't understand what happened

using CLI is my preferred way of working since mosten often this tells me exactly what happened

this time however, Proxmox just 'flipped'

as I did something incredibly stupid after 'rm -Rf /etc/pve + reboot (not restart)'
I'm not sure what made local-zfs disappear either, local is still there though

either way, most VM are broken now
even with rebuilding the VM they don't boot properly

moving on to new, VM, lesson learned, lucky this is a test environment
and now, now I have backups

 

[gfngfn256]

rm -Rf /etc/pve

After doing that command, you had better reinstall PVE! I can't even imagine what made you enter that command.

using CLI is my preferred way of working since most(en) often this tells me exactly what happened

The above command (beginning of this post) shows your lack of safety when using CLI. Updates & most host-management items you need, are all available from the GUI & also offer the feedback you need. As a rule - don't mess with the host HV if you're not absolutely sure what you are doing.

 

[Joris L.]

After doing that command, you had better reinstall PVE! I can't even imagine what made you enter that command.


The above command (beginning of this post) shows your lack of safety when using CLI. Updates & most host-management items you need, are all available from the GUI & also offer the feedback you need. As a rule - don't mess with the host HV if you're not absolutely sure what you are doing.

that command was actually a recommended approach by a Proxmox team support member. Due to me growing old and being quite tired I had also rebooted the server instead of restarting it. Multi-tasking only works well when you're young I guess. Time to not do that again.

If anything, it shows your lack of understanding, not mine. That folder is actually a fusemount and is composed from a .db file. Erasing that folder and restarting the proxmox service(s) (not rebooting the server) rebuilds that directory.

Though I'm guessing your intention are pure.

It doesn't mean because you're competent at click-ops everyone should align with that non-technical approach to technical matters. Some people, most people, using open-source want to learn by doing. Not just click around. I do appreciate the web-ui like many other people. I'm not sticking to it. Getting ready to work with Proxmox over the CLI and API.

 

[gfngfn256]

As you write, I can confirm that my intentions were completely "pure". I definitely did not mean to take any jab or even criticism at your, or for that matter any user's management approach. Everyone to his own - as they say.

I also realized your intent - to rebuild that directory. However doing that is inherently extremely dangerous - as your experience itself testifies. Do remember that after doing that command if the system reboots (for any reason; kernel freeze, io freeze, you or another operator issues a reboot etc.) before the services are fully started (& directory fully written to disk) you will basically have an unusable HV. The only scenarios for doing such a measure would be in a test dev setup - or you have a complete robust backup of the host itself (not a small feat on its own) & are ready immediately to put that backup to restoration.

Any forum provides two basic types of help: Firstly, immediate answers to those posing the post questions, & secondly (& probably more important) an invaluable library resource for those searching later with a similar problem. I believe more than 90% is of the second nature. I therefore feel my initial answer was a valid comment - yes it may have sounded degrading - but that was intended to warn off anyone from trying anything similar.

In any event - if any offence was taken - I apologize for that & as explained it wasn't intended for anyone specific.

 

[Joris L.]

no need for such an elaborate apology, I'm in year 32 of IT, I've seen way to many forums, communities and newsgroups. I expect/hope stuff would finally just work most of the time. And I'm known to be grumpy.

a helpful attitude is obviously admirable
also consider the promox webui is just a shell
pveupdate and pveupgrade are what is actually ran by clicking in the UI