Proxmox unbearable IO without changes made

[higkiller]

Hi all,

First time Poster, long time user.
Current Specs

Dell Poweredge 430
2x CPU E5-2630 v4 @ 2.20GHz (2 Sockets
32GB DDR4
3x 2TB Seagate ST2000NM0001 Running on RAID with PERC H730 Mini​

Currently facing a bit of a tricky issue, I have had my proxmox server runningg for over 30 days without a problem.

Current VMS:
HestiaPC
NextCloud
OPNSense

Containers
Nginx Proxy Manager
Ubuntu with wordpress

Suddently im getting up to 90% IO, im getting massive delays on my vms, even ssh into them is delayed by a few seconds, running a command have to wait around 3 to 10 seconds to get a response which is by it self weird.
This also gets me extreme lag with my connection since im running it on the server.

Temps are around 45°C-50°C
Hard drives tested no problems found same with memories.
My ram is about 20GB Usage, and my CPU is less than 10%, my PVE has 100GB and only 20% is being used.

Yesterday decided to updates proxmox currently running
PVE Version: pve-manager/8.4.5/57892e8e686cb35b
Kernel version: Linux 6.8.12-12-pve

Logs seems clear at a glance, no issues no crashing no problems happened.

At this point in time I am trying to get my nextcloud up and running and my services and simply crashing, no updates occured on the machine.

Im completely at a loss now, I havent done major upgrades, no major changes, so all of the sudden this happening is throwing me off.
Any ideias would be grateful.

 

[Impact]

I'd recommend you look into what drive is affected with iostat and which process causes the IO with iotop-c.
We can then look further into the issue(s). In my opinion HDDs are not a good idea to run even a single OS on. You run multiple.

 

[mjtbrady]

PERC H730 battery status?

 

[waltar]

A hw-raid5 out of 3 hdd will just show 1 virtual disk (eg sda) in iostat.
A failed battery would disable ctrl. cache and is a reason for slow I/O.
Pve send mails for bad smart values even if they are behind a raidctrl. and if still not "plugged" out of a raidset.

 

[higkiller]

PERC H730 battery status?

New and in good status

 

[higkiller]

A hw-raid5 out of 3 hdd will just show 1 virtual disk (eg sda) in iostat.
A failed battery would disable ctrl. cache and is a reason for slow I/O.
Pve send mails for bad smart values even if they are behind a raidctrl. and if still not "plugged" out of a raidset.

Server is fresh, about 2 months old. checked and all good

 

[higkiller]

I'd recommend you look into what drive is affected with iostat and which process causes the IO with iotop-c.
We can then look further into the issue(s). In my opinion HDDs are not a good idea to run even a single OS on. You run multiple.

Currently running SAS 7.2k 6GB loooking after which VM is doing this. THanks, altought i have a weird ideia, if 1 vm can cause this much delay i did have an issue with a hestia update, thanks, will investigate and report back

 

[higkiller]

FOUND THE ISSUE! (I think)
Stopped a few VMs just to be sure, clean reboot after some time off, its the nextcloud, problem is when i boot the machine i start getting 70% IO, will restore a backup from 2 days ago, and test.

Without that machine on im getting 2% IO at rare times hovers around 0.1 to 1% if that

 

[news]

I thing use a new Server, board with amd b550 chip set and a ryzen 5 5600 cpu with 32 GByte ram.
Then test your nextcloud on this server.
I use SATA3 SSDs with ZFS on it. Like the Kingston DC600M Serie Enterprise SATA 3.0 SSD 480gb, 960 gb and so on.
I use the ASRock B550(M) PG RIPTIDE (in Germany).

 

[higkiller]

I thing use a new Server, board with amd b550 chip set and a ryzen 5 5600 cpu with 32 GByte ram.
Then test your nextcloud on this server.
I use SATA3 SSDs with ZFS on it. Like the Kingston DC600M Serie Enterprise SATA 3.0 SSD 480gb, 960 gb and so on.
I use the ASRock B550(M) PG RIPTIDE (in Germany).

It was working flawlessly before, it could be a case of requests on shared folders, or the lastest updates since i had an issues with Hestia Apache update, so restoring from a backup and will test after.