Permanently read I/O on ZFS pool without known access

[Th3Dan]

I'm using Proxmox 6.2.4 and I have a ZFS Pool on the server with two 8TB disks (WD Red and Seagate). It was previously used for Proxmox VMS, but now it mainly holds the data of Nextcloud in a Docker instance on the same host. Since the server runs 24/7, I want to take care of standby for the disks since they're not always needed.

It seems that the disks never went to standby. When setting one of the disks from the ZFS Pool (sda) to standby manually, they got waked up just a few seconds later:

# hdparm -y /dev/sda

/dev/sda:
issuing standby command
# hdparm -C /dev/sda

/dev/sda:
drive state is:  standby
# hdparm -C /dev/sda

/dev/sda:
drive state is:  standby
# hdparm -C /dev/sda

/dev/sda:
drive state is:  active/idle

Each command was run with about ~1s waiting time. Since I can't figure out why this happens, I stopped everything (complete docker Daemon) except Proxmox. Now just Proxmox runs with one VM, which is not located on the ZFS pool (so there should be no activitiy on sda). I tried hd-idle in debug mode

hd-idle -i 0 -a /dev/sda -i 120 -l /var/log/hd-idle.log -d

and it shows the i/o activity every few seconds. The ZFS pool disks have read activity on EVERY check:

probing sda: reads: 12760648, writes: 3174104
probing sda: reads: 12763720, writes: 3174104
probing sda: reads: 12765256, writes: 3174104
probing sda: reads: 12766792, writes: 3174104

The same on the other ZFS member sdd. It seems that those reads prevent the discs from standby. Why does this happen?
I can't explain where the reads came from. Does Proxmox some kind of FS checks?

As I said there is nothing on Proxmox that could explain those reads, no VM is running on the ZFS pool. I only have it registered in /etc/pve/storage.cfg because there are ISO images that I don't want to place on the SSDs:

dir: zfs-storage
  path /zfs-storage/proxmox
  content images,iso

But they're not really in use currently. I also tried commenting out those lines from ZFS in /etc/pve/storage.cfg. And I also commented them in again and ran pvesm set zfs-storage --disable 1 but still got a lot of reads on every probe in both cases.

Other things I tried:

• Set use_lvmetad = 1 in global section of /etc/lvm/lvm.conf and reboot the entire server (Read in a post that this could cause the reads: https://forum.proxmox.com/threads/pvestatd-awakes-hdd-immediately.15344/)
• systemctl stop pvestatd because I found this thread where the data collection seems the cause for letting HDDs not sleep: https://forum.proxmox.com/threads/pvestatd-doesnt-let-hdds-go-to-sleep.29727/ After the service was stopped, I saw no data in the web ui (so it should work)

All doesn't work: I still see increased reads in the hd-idle probes... Since they're on all disks (also SSDs without ZFS) it seems that something other from Proxmox or Debian causes this problem.

 

[dietmar]

The same on the other ZFS member sdd. It seems that those reads prevent the discs from standby. Why does this happen?
I can't explain where the reads came from. Does Proxmox some kind of FS checks?

Proxmox runs status checks and records disk usage on any configured storage.

 

[LnxBil]

Yes, I encountered this also just this week. Two servers, identical disk layout (mirrored vdev), one with the problem, the other without it. Nothing running, just installed PVE. iotop -oPak shows a lot of I/O operation in tgx_sync, but no actual block read/write is happening. Other disk activity is negligible.

After increasing zfs_txg_timeout, the problem just went away.

Here is a dstat output while the default of 5 was still active:

----system---- --total-cpu-usage-- --dsk/sda-----dsk/sdb-- ---paging-- ---system-- ---sda-------sdb--- -----------ZFS-ARC----------- --ZFS-ZIL--
     time     |usr sys idl wai stl| read  writ: read  writ|  in   out | int   csw |rawa wawa:rawa wawa| mem   hit   miss reads  hit%|count bytes
31-05 12:31:51|  1   1  97   0   0|   0   174k:   0   174k|   0     0 | 362   492 |13.5 1.34:12.0 1.24|1621M   37     0    37   100 |   0     0
31-05 12:31:56|  1   0  98   0   0|   0   183k:   0   183k|   0     0 | 268   407 |12.0 1.39:12.0 1.24|1621M   21     0    21   100 |   0     0
31-05 12:32:01|  4   3  93   0   0|   0   168k:   0   168k|   0     0 | 491   623 |13.5 1.28:13.0 1.30|1621M  184     0   184   100 |   0     0
31-05 12:32:06|  4   1  95   1   0|   0   231k:   0   231k|   0     0 | 401   528 |13.0 1.41:14.0 1.40|1621M   87     0    87   100 |   0     0
31-05 12:32:11|  1   1  97   0   0|   0   170k:   0   170k|   0     0 | 358   502 |13.0 1.40:10.5 1.23|1621M   33     0    33   100 |   0     0
31-05 12:32:16|  1   1  98   0   0|   0   171k:   0   171k|   0     0 | 283   422 |8.00 1.19:12.0 1.40|1621M   26     0    26   100 |   0     0
31-05 12:32:21|  2   1  97   0   0|   0   172k:   0   172k|   0     0 | 387   533 |14.0 1.21:15.0 1.28|1621M   42     0    42   100 |   0     0
31-05 12:32:26|  1   2  96   1   0|   0   194k:   0   194k|   0     0 | 403   551 |9.00 1.05:13.5 5.32|1621M   41     0    41   100 |   0     0
31-05 12:32:31|  1   1  98   0   0|   0   170k:   0   170k|   0     0 | 352   523 |10.0 1.19:12.5 1.48|1621M   37     0    37   100 |   0     0
31-05 12:32:36|  0   0  99   0   0|   0   189k:   0   189k|   0     0 | 242   394 |11.0 1.27:11.5 1.31|1621M   22     0    22   100 |   0     0

It shows an average of ZFS writes of ca. 170k a second constantly, with only the background PVE daemon noise and nothing else running.

 

[Th3Dan]

It seems that I was too fast. Waiting some time after systemctl stop pvestatd helped: Most of the IO went away. But with the side effects of not having any status information in Proxmox UI any more. I found a better solution using the global_filter:

global_filter = [ "r|/dev/zd.*|", "r|/dev/mapper/pve-.*|" "r|/dev/mapper/.*-(vm|base)--[0-9]+--disk--[0-9]+|" "r|/dev/sda" "r|/dev/sdd"]

I just added /dev/sda and /dev/sdd, where r stands for reject (not regular expression). Both volumes are from my ZFS pool. This has the advantage that I'm still able to see system status in Proxmox.

After a reboot, my disks went to standby as we can see from the hd-idle log of the last night:

date: 2020-05-31, time: 01:41:47, disk: sdd, running: 602, stopped: 1800
date: 2020-05-31, time: 02:01:49, disk: sda, running: 602, stopped: 3003
date: 2020-05-31, time: 02:01:50, disk: sdd, running: 601, stopped: 602
date: 2020-05-31, time: 05:32:53, disk: sda, running: 1202, stopped: 11462
date: 2020-05-31, time: 05:33:02, disk: sdd, running: 1202, stopped: 11470
date: 2020-05-31, time: 06:26:05, disk: sda, running: 670, stopped: 2522
date: 2020-05-31, time: 06:26:06, disk: sdd, running: 662, stopped: 2522
date: 2020-05-31, time: 06:47:09, disk: sda, running: 662, stopped: 602
date: 2020-05-31, time: 06:47:10, disk: sdd, running: 662, stopped: 602
date: 2020-05-31, time: 07:48:13, disk: sda, running: 723, stopped: 2941
date: 2020-05-31, time: 07:48:14, disk: sdd, running: 723, stopped: 2941
date: 2020-05-31, time: 09:46:17, disk: sda, running: 602, stopped: 6482
date: 2020-05-31, time: 09:46:18, disk: sdd, running: 602, stopped: 6482
date: 2020-05-31, time: 10:08:21, disk: sda, running: 602, stopped: 722
date: 2020-05-31, time: 10:08:22, disk: sdd, running: 602, stopped: 722
date: 2020-05-31, time: 11:05:24, disk: sda, running: 602, stopped: 2821
date: 2020-05-31, time: 11:05:25, disk: sdd, running: 602, stopped: 2821
date: 2020-05-31, time: 12:09:27, disk: sda, running: 1442, stopped: 2401
date: 2020-05-31, time: 12:09:28, disk: sdd, running: 1441, stopped: 2402
date: 2020-05-31, time: 12:44:31, disk: sda, running: 1202, stopped: 902
date: 2020-05-31, time: 13:00:33, disk: sda, running: 601, stopped: 361

Now it seems that something is waking up the disks regularly. I currently try to figure it out, both fatrace and ausearch doesn't show me any activity when hd-idle reports disk wakeups. Currently I'm assuming that this is somewhat related to Docker (most of the software runs in Containers on the host).

Proxmox runs status checks and records disk usage on any configured storage.

I'm also wondering why it seems that Proxmox also run stats on the ZFS pool even when it's commented out in the storage configuration. Other posts and docs indicates that it shouldn't do that. Since I excluded my two ZFS disks in the global filter, I noticed that Proxmox UI shows all stats except from the local-kvm. There is a "?" and when I click on it and then on content, it shows

command '/sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --config 'report/time_format="%s"' --options vg_name,lv_name,lv_size,lv_attr,pool_lv,data_percent,metadata_percent,snap_percent,uuid,tags,metadata_size,time' failed: exit code 5 (500)

It seems that Proxmox falsely try to run some operations on one of the ZFS disks, which may caused the standby problems I had even when the pool is not enabled in the storage configuration.

 

[j.io]

Any update on this?

 

[proxwolfe]

It seems that I was too fast. Waiting some time after systemctl stop pvestatd helped: Most of the IO went away. But with the side effects of not having any status information in Proxmox UI any more. I found a better solution using the global_filter:

global_filter = [ "r|/dev/zd.*|", "r|/dev/mapper/pve-.*|" "r|/dev/mapper/.*-(vm|base)--[0-9]+--disk--[0-9]+|" "r|/dev/sda" "r|/dev/sdd"]

I just added /dev/sda and /dev/sdd, where r stands for reject (not regular expression). Both volumes are from my ZFS pool. This has the advantage that I'm still able to see system status in Proxmox.

After a reboot, my disks went to standby as we can see from the hd-idle log of the last night:

date: 2020-05-31, time: 01:41:47, disk: sdd, running: 602, stopped: 1800
date: 2020-05-31, time: 02:01:49, disk: sda, running: 602, stopped: 3003
date: 2020-05-31, time: 02:01:50, disk: sdd, running: 601, stopped: 602
date: 2020-05-31, time: 05:32:53, disk: sda, running: 1202, stopped: 11462
date: 2020-05-31, time: 05:33:02, disk: sdd, running: 1202, stopped: 11470
date: 2020-05-31, time: 06:26:05, disk: sda, running: 670, stopped: 2522
date: 2020-05-31, time: 06:26:06, disk: sdd, running: 662, stopped: 2522
date: 2020-05-31, time: 06:47:09, disk: sda, running: 662, stopped: 602
date: 2020-05-31, time: 06:47:10, disk: sdd, running: 662, stopped: 602
date: 2020-05-31, time: 07:48:13, disk: sda, running: 723, stopped: 2941
date: 2020-05-31, time: 07:48:14, disk: sdd, running: 723, stopped: 2941
date: 2020-05-31, time: 09:46:17, disk: sda, running: 602, stopped: 6482
date: 2020-05-31, time: 09:46:18, disk: sdd, running: 602, stopped: 6482
date: 2020-05-31, time: 10:08:21, disk: sda, running: 602, stopped: 722
date: 2020-05-31, time: 10:08:22, disk: sdd, running: 602, stopped: 722
date: 2020-05-31, time: 11:05:24, disk: sda, running: 602, stopped: 2821
date: 2020-05-31, time: 11:05:25, disk: sdd, running: 602, stopped: 2821
date: 2020-05-31, time: 12:09:27, disk: sda, running: 1442, stopped: 2401
date: 2020-05-31, time: 12:09:28, disk: sdd, running: 1441, stopped: 2402
date: 2020-05-31, time: 12:44:31, disk: sda, running: 1202, stopped: 902
date: 2020-05-31, time: 13:00:33, disk: sda, running: 601, stopped: 361

Now it seems that something is waking up the disks regularly. I currently try to figure it out, both fatrace and ausearch doesn't show me any activity when hd-idle reports disk wakeups. Currently I'm assuming that this is somewhat related to Docker (most of the software runs in Containers on the host).


I'm also wondering why it seems that Proxmox also run stats on the ZFS pool even when it's commented out in the storage configuration. Other posts and docs indicates that it shouldn't do that. Since I excluded my two ZFS disks in the global filter, I noticed that Proxmox UI shows all stats except from the local-kvm. There is a "?" and when I click on it and then on content, it shows

command '/sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --config 'report/time_format="%s"' --options vg_name,lv_name,lv_size,lv_attr,pool_lv,data_percent,metadata_percent,snap_percent,uuid,tags,metadata_size,time' failed: exit code 5 (500)

It seems that Proxmox falsely try to run some operations on one of the ZFS disks, which may caused the standby problems I had even when the pool is not enabled in the storage configuration.

Where (in which file) do you apply the global filter?

Thanks!