Proxmox GUI hang - not response

[Frank-75]

Good evening,
I'm also attaching this because I've had these errors since last night and had to reset the server with the power button.
However, the VMs inside are working: I found consecutive logs, but the PVE interface can't be reached from the designated LAN.

Hardware changes? I replaced the CPUs with more powerful models, but now I've gone back. It doesn't seem like overheating because the air coming out behind the server isn't even warm.
Anyway, I restored the previous CPUs and will see if they crash with these too. The difference is that in between I only updated the Debian updates, not the PVE repositories, because after three weeks of forced upgrades due to vulnerabilities, I didn't want to keep moving VMs from one server to another.

May 29 15:33:06 pve-ML350pGen8 pulse-sensor-proxy[2236]: [90m3:33PM[0m [33mWRN[0m [1mControl plane sync failed[0m [36merror=[0m[31m[1m"control plane responded 401: {\"error\":\"Authentication required\"}"[0m[0m
May 29 15:33:21 pve-ML350pGen8 kernel: sd 0:0:1:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
May 29 15:33:21 pve-ML350pGen8 kernel: sd 0:0:1:0: [sda] tag#0 Sense Key : Aborted Command [current]
May 29 15:33:21 pve-ML350pGen8 kernel: sd 0:0:1:0: [sda] tag#0 Add. Sense: No additional sense information
May 29 15:33:21 pve-ML350pGen8 kernel: sd 0:0:1:0: [sda] tag#0 CDB: Write(10) 2a 00 01 21 31 98 00 03 68 00
May 29 15:33:21 pve-ML350pGen8 kernel: I/O error, dev sda, sector 18952600 op 0x1:(WRITE) flags 0x0 phys_seg 11 prio class 2
May 29 15:33:21 pve-ML350pGen8 kernel: EXT4-fs warning (device dm-1): ext4_end_bio:369: I/O error 10 writing to inode 4980756 starting block 9267)
May 29 15:33:21 pve-ML350pGen8 kernel: Buffer I/O error on device dm-1, logical block 9267
May 29 15:33:21 pve-ML350pGen8 kernel: Buffer I/O error on device dm-1, logical block 9268
May 29 15:33:21 pve-ML350pGen8 kernel: Buffer I/O error on device dm-1, logical block 9269
May 29 15:33:21 pve-ML350pGen8 kernel: Buffer I/O error on device dm-1, logical block 9270
May 29 15:33:21 pve-ML350pGen8 kernel: Buffer I/O error on device dm-1, logical block 9271
May 29 15:33:21 pve-ML350pGen8 kernel: Buffer I/O error on device dm-1, logical block 9272
May 29 15:33:21 pve-ML350pGen8 kernel: Buffer I/O error on device dm-1, logical block 9273
May 29 15:33:21 pve-ML350pGen8 kernel: Buffer I/O error on device dm-1, logical block 9274
May 29 15:33:21 pve-ML350pGen8 kernel: Buffer I/O error on device dm-1, logical block 9275
May 29 15:33:21 pve-ML350pGen8 kernel: Buffer I/O error on device dm-1, logical block 9276
May 29 15:33:21 pve-ML350pGen8 sshd-session[218213]: Connection closed by authenticating user root 172.20.20.15 port 59648 [preauth]
May 29 15:33:25 pve-ML350pGen8 pvedaemon[210125]: VM 502 qga command failed - VM 502 qga command 'guest-ping' failed - got timeout
May 29 15:33:28 pve-ML350pGen8 pvedaemon[194571]: VM 502 qga command failed - VM 502 qga command 'guest-ping' failed - got timeout
May 29 15:33:34 pve-ML350pGen8 pvedaemon[206275]: VM 502 qga command failed - VM 502 qga command 'guest-ping' failed - got timeout
May 29 15:33:41 pve-ML350pGen8 pveproxy[173109]: worker exit
-- Reboot --
May 29 16:38:05 pve-ML350pGen8 kernel: Linux version 7.0.2-6-pve (build@proxmox) (gcc (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-6 (2026-05-20T08:55Z) ()
May 29 16:38:05 pve-ML350pGen8 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-7.0.2-6-pve root=/dev/mapper/pve-root ro quiet
May 29 16:38:05 pve-ML350pGen8 kernel: KERNEL supported cpus:
May 29 16:38:05 pve-ML350pGen8 kernel:   Intel GenuineIntel
May 29 16:38:05 pve-ML350pGen8 kernel:   AMD AuthenticAMD
May 29 16:38:05 pve-ML350pGen8 kernel:   Hygon HygonGenuine
May 29 16:38:05 pve-ML350pGen8 kernel:   Centaur CentaurHauls
May 29 16:38:05 pve-ML350pGen8 kernel:   zhaoxin   Shanghai 
May 29 16:38:05 pve-ML350pGen8 kernel: x86/CPU: Running old microcode
May 29 16:38:05 pve-ML350pGen8 kernel: BIOS-provided physical RAM map:

 

[daanw]

Looks like either a disk going bad, a loose cable preventing proper I/O or a mdadm error?
The filesystem might have gone into read-only preventing further damage, explaining the system stall.
What do cat /proc/mdstat and smartctl -a /dev/sda say?

The difference is that in between I only updated the Debian updates, not the PVE repositories

Not sure if this works fine, as Debian and Proxmox versions are kind of entwined?

 

[Frank-75]

Thanks for your reply.
I restored the old CPUs last night, and after rebooting the system today, I haven't encountered any problems so far. Everything has been working normally for a day. With the new CPUs, the problem occurs after a few hours.
I ran the command smartctl -a /dev/sda this morning, which produced the attached file.
The disk is old, and I'm aware of that, but the values are all good: it appears healthy. I previously had a non-enterprise SSD that I had to replace due to smart errors, but these were caused by log entries, as described in the Proxmox technicians' posts. I opted to install on a mechanical SATA disk for greater reliability.
What I don't understand is why, after replacing the CPU and running synthetic tests with positive results, the KVM only damages the blocks on SDA, which is the SATA disk.
The VMs work because they are on a separate CTRL RAID on an SDB device.
The fans work just like the older CPUs, and the IML shows no problems except when I manually shut down the server because it's not accessible via the GUI and the "momentary press" had no effect. Perhaps the system was shutting down the VMs, and the cursor was blinking in the top left corner on the screen.

The only thing I can do right now is to test the CPUs individually by moving all the RAM to one CPU and hoping it's not a design flaw in the location of the motherboard regulators with more power-hungry CPUs.
The cables haven't been touched because they run in front of the 80x80 fans, which are located in front of the processor heatsinks like the SATA drive (so everything is properly cooled). The air behind them is slightly warm, and when I shut down the heatsinks, they're warm, so there's heat exchange.
For your information, the CPUs are matched to HPE, which, as you may have gathered from the previous log, is an ML350pGen8.

Not sure if this works fine, as Debian and Proxmox versions are kind of entwined?

Debian updates are those the system proposed before disabling the no-subscription repository. PVE updates have disappeared, but the basic ones remain.

I understand that AI is discovering vulnerabilities every day, but these are more about fixing instabilities than about adding PVE 9.2.2 + QEMU 11 and forcing continuous upgrades to get fixes. If they're then distributed widely with a wide variety of configurations, they're problems that can become a nuisance and cause problems if the machines are in production.

google translate nuissance but I have wrote blasphemies word fai te! <- for my italians friends

 

[daanw]

Did you also check mdadm?
Edit: My bad, I was mixing md1 (mdadm) with dm-1 (LVM).

Which 2 CPU's are you using exactly?

 

[Frank-75]

Good morning,
I didn't enter the result of cat /proc/mdstat because it doesn't return any results. The sdb disk is mounted as a directory in Proxmox because the logical volume is a hardware array.
The CTRL is hardware-based; it's an HPE P420 2GB cache with a dedicated battery. I'm absolutely sure it's working perfectly. There are no anomalies in the IML (Integrated Management Log) or the dedicated page. The CTRL is robust; the battery was replaced a few weeks ago with a new one because the previous one was visibly damaged due to wear. The battery is behind a fan, so it gets air and doesn't overheat.

The only difference is the CPUs; I went from two E5-2620 v2 (6-core 2.1 GHz) to two E5-2680 v2 (10-core 2.8 GHz). The switch is necessary because I need more MHz and faster core speeds due to software optimized for this feature.

If you have a specific command to check the EXT4 file system, I can run it for you. I apologize but I am new to the Linux world and I don't know all the commands by heart like I do for the Windows part.