New Year, new issue?

[sdf212]

I am at a loss. I have 3 proxmox servers, all with identical hardware.
Each of them has been in service since April 2022. They are not clustered together.
Each of them started experiencing this issue at exactly 00:01am on January 1st 2023.

The average IO wait / IO delay has skyrocketed from <1% to 15-25% constantly.

I have shut down all VM's, and rebooted the hypervisor box, and even with no VM's started iowait / io delay is 6-10%.

I am pulling my hair out.

Each box has a 4TB SSD, Samsung 870.

I figure it has to be a drive firmware issue, or proxmox related bug for all 3 hypervisors to have the exact same issue starting at the exact same time.

Any ideas?

 

[bbgeek17]

Have you checked SMART stats? anything suspicious there?
https://pve.proxmox.com/wiki/Disk_Health_Monitoring

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[sdf212]

Have you checked SMART stats? anything suspicious there?
https://pve.proxmox.com/wiki/Disk_Health_Monitoring

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I have, none show anything concerning. Wear level is between 1% and 4% among the servers.

 

[sdf212]

Just an update, moved all VM's off one of the boxes, and with nothing running, still getting abnormally high io wait. Tonight I am going to do a fresh install on one box, and if that doesn't work I will try a new drive.

 

[sdf212]

Re-installed Proxmox on same hardware, issue went away....