Strange harddisk activity

[dvdmeer]

Hi,

I have recently set up a proxmox system (latest version) and currently am only running my main Windows 10 Pro system of it.
For some time now I have noticed that there is a lot of harddisk activity on one of the drives that holds a data partition for my Windows system.
When I am working in Windows I do not touch this drive so there should be no activity whatsoever.
My main OS (Windows) is installed on a M.2 drive so I do not hear when it accesses the drive but I have some larger 3.5" drives and one of them make a heavy rattling noise like it is reading a lot of data. This continues for 20-25 seconds and then is silent for a minute or so and then it starts again.
This goes on for a long time and sometimes it stops for a little while but it happens every day.
When I shut down my Windows system and only leave Proxmox running the rattling goes on for a few minutes and then it eventually stops.

My OS is installed as a thin provisioned system and I set it up with the defaults (as in the OS is installed with virtual disks so no direct pass-through for any of the harddisks).
The graphs for my node do not show any significant disk activity while the constant harddisk rattling goes on and also the virtual machine task manager does not show any significant disk activity.
I tried running a tool (iotop) on the client but it is not clear to me what is actually causing all this and I see 3 or 4 kvm tasks for my virtual machine popping up at the top but they also do this when there is no significant harddisk activity.

How can I find out why Proxmox is doing this because this does not sound normal to me and I don't want it to affect the durability of my harddisk?


Thanks

 

[wolfgang]

Hi,

if you do not use disk passthrough the disk is managed by Proxmox VE so it is normal that you have activities like updating the atime.

 

[dvdmeer]

And this updating can take 20-30 seconds with very heavy harddisk activity?

 

[Quintin Cardinal]

When you get storage involved, there's many different things at play.

Are you using ZFS? LVM on ext4 or xfs? If using ZFS, are you using the M.2 SSD or zil/l2arc? Does your arc have enough RAM? A hardware RAID Controller? HBA controller? Enterprise disks?
A hardware RAID controller could be doing almost anything in the background without presenting that information to Proxmox. Some enterprise disks (The kind I don't like to deploy anymore, the ones that come from Dell Enterprise orders) make a really thick and solid chunking sound under heavy IO even when brand new. Consumer grade disks generally have a higher pitched and lighter clicking to them. They're still spinning disks, the arm still moves. Then there's the matter of if you're using a hardware RAID controller, are your Logical Drives configured using read/write cache if available? If not available, expect terrible performance.

ZFS is an incredible technology, and it will do lots of hidden things in the background. Everything it does is designed to better optimize your IO, and everything you do will be faster the second time around especially when using zil/l2arc on an SSD.

What type of virtual disks are you using? Are you sure you've got a Windows optimized thin provisioned volume?
VirtIO SCSI controller using the Red Hat vioscsi drivers with Discard and SSD Emulation checked is going to be the only way to get a guest OS to properly thin provision. If you don't do this, Windows will see standard HDD storage and run actual defrags, which will cause your virtual disks to bloat to sizes of up to twice the used space if you're not careful. Then there's the matter of only 50% of the time Windows Disk Optimization actually treats the properly configured thin provision virtual disk as thin provisioned without additional help. My standard configuration consists of disabling Disk Optimization and using Task Scheduler to run a daily "defrag /c /o". This runs all volumes using the best optimization method available for the storage, which is trim and slab consolidation. (Don't ask my why defrag from the cli is smart enough to know your storage, but Disk Optimization GUI isn't. I couldn't answer that haha)

I see you mentioned multiple threads for your VM. Sounds like you may have VirtIO SCSI Single SCSI controller set, which will create a separate IO thread for each virtual disk. This can help improve performance, and on slower hypervisor hardware prevent heavy IO on a non-OS disk from crashing the OS due to disk latency. (I've seen that happen on Server 2016 VMs, because Server 2016 is a pile of junk and it's sad how poorly it performs compared to 2012R2 or 2019.)

Long story short, hypervisor storage and virtual disk configuration is a complicated system with dozens of parts. If it performs well, I honestly wouldn't worry about it. If your disk is less than new, and you aren't entirely confident in it's health, replace it and keep your RAID safe from disaster. If you're not using ZFS, I'll make the recommendation you switch. I've seen it too many times that a PERC RAID controller will watch a disk pile up half a million read errors and just let it sit happy in the array without failing it out. ZFS will never let it get that far out of hand.