pve-cluster / lvm unmounted disk polling

H

heiner

New Member
Nov 25, 2017
3
1
1
57
Hi,

I am trying to debug a strange issue and I am at my wits' (and google skills) end.
Running most recent PVE on Debian Stretch (as per the Wiki) with full disk encryption and remote cryptroot unlock.
Code: 
uname -a

Linux pve 4.13.4-1-pve #1 SMP PVE 4.13.4-26 (Mon, 6 Nov 2017 11:23:55 +0100) x86_64 GNU/Linux

My host has four physical 8TB HDDs (sda - sdd); unmounted and used for direct passthrough and one SSD (sde) as root and VM storage, setup as per the original PVE installer as VG with 3 separate LVs for swap, root and data. The data volume is setup as a thinpool.
Code: 
lsblk

NAME                                    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                       8:0    1   7.3T  0 disk
sdb                                       8:16   1   7.3T  0 disk
sdc                                       8:32   1   7.3T  0 disk
sdd                                       8:48   1   7.3T  0 disk
sde                                       8:64   1 232.9G  0 disk
+-sde1                                    8:65   1   243M  0 part  /boot
+-sde2                                    8:66   1     1K  0 part
+-sde5                                    8:69   1 232.7G  0 part
  +-sda5_crypt                          253:0    0 232.7G  0 crypt
    +-b0x--pve--vg-root                 253:1    0  18.6G  0 lvm   /
    +-b0x--pve--vg-swap                 253:2    0   9.1G  0 lvm   [SWAP]
    +-b0x--pve--vg-data_tmeta           253:3    0   104M  0 lvm
    ¦ +-b0x--pve--vg-data-tpool         253:5    0 204.7G  0 lvm
    ¦   +-b0x--pve--vg-data             253:6    0 204.7G  0 lvm
    ¦   +-b0x--pve--vg-vm--100--disk--1 253:7    0    10G  0 lvm
    ¦   +-b0x--pve--vg-vm--101--disk--1 253:8    0    30G  0 lvm
    ¦   +-b0x--pve--vg-vm--101--disk--2 253:9    0   105G  0 lvm
    +-b0x--pve--vg-data_tdata           253:4    0 204.7G  0 lvm
      +-b0x--pve--vg-data-tpool         253:5    0 204.7G  0 lvm
        +-b0x--pve--vg-data             253:6    0 204.7G  0 lvm
        +-b0x--pve--vg-vm--100--disk--1 253:7    0    10G  0 lvm
        +-b0x--pve--vg-vm--101--disk--1 253:8    0    30G  0 lvm
        +-b0x--pve--vg-vm--101--disk--2 253:9    0   105G  0 lvm

With the server at complete idle (no VMs or any tasks running) the kernel is accessing the disks exactly every 10 seconds, which prevents spindown and is also quite annoying when sitting next to the server :)
Not only the root device is affected, but also the four unmounted HDDs (all LUKS devices).
It also makes no difference if the devices are actively being passed through to a running VM or not.
Code: 
iotop -o -b -d 10

Total DISK READ :       0.00 B/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:      26.14 K/s | Actual DISK WRITE:    1225.97 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 5051 be/0 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [kworker/u17:1]
   16 be/4 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [ksoftirqd/1]
   28 be/4 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [ksoftirqd/3]
    7 be/4 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [ksoftirqd/0]
   22 be/4 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [ksoftirqd/2]
   34 be/4 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [ksoftirqd/4]
   40 be/4 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [ksoftirqd/5]

Disabling pve-ha-lrm and pve-ha-crm via systemctl as per this thread seemingly only reduced the access intervals to the SSD, but not the HDDs.

A block dump shows lvs and vgs reading the same blocks on all devices every 10 seconds:
Code: 
echo 1 > /proc/sys/vm/block_dump
tail -f /var/log/kern.log | grep sd

Nov 25 10:23:40 pve kernel: [150901.104951] vgs(13121): READ block 0 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.105238] vgs(13121): READ block 15628052992 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.105358] vgs(13121): READ block 15628053152 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.105480] vgs(13121): READ block 0 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.105624] vgs(13121): READ block 8 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.105777] vgs(13121): READ block 0 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.111537] vgs(13121): READ block 0 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.111796] vgs(13121): READ block 15628052992 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.111915] vgs(13121): READ block 15628053152 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.112024] vgs(13121): READ block 0 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.112154] vgs(13121): READ block 8 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.112334] vgs(13121): READ block 0 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.112522] vgs(13121): READ block 0 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.112753] vgs(13121): READ block 15628052992 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.112885] vgs(13121): READ block 15628053152 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.113034] vgs(13121): READ block 0 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.113162] vgs(13121): READ block 8 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.113337] vgs(13121): READ block 0 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.113537] vgs(13121): READ block 0 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.113751] vgs(13121): READ block 15628052992 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.113865] vgs(13121): READ block 15628053152 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.113966] vgs(13121): READ block 0 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.114094] vgs(13121): READ block 8 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.114260] vgs(13121): READ block 0 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.114447] vgs(13121): READ block 0 on sde (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.114614] vgs(13121): READ block 497536 on sde1 (2 sectors)
Nov 25 10:23:40 pve kernel: [150901.114650] vgs(13121): READ block 497648 on sde1 (2 sectors)
Nov 25 10:23:40 pve kernel: [150901.114678] vgs(13121): READ block 0 on sde1 (2 sectors)
Nov 25 10:23:40 pve kernel: [150901.114795] vgs(13121): READ block 8 on sde1 (2 sectors)
Nov 25 10:23:40 pve kernel: [150901.114930] vgs(13121): READ block 0 on sde1 (4 sectors)
Nov 25 10:23:40 pve kernel: [150901.115141] vgs(13121): READ block 487894912 on sde5 (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.115266] vgs(13121): READ block 487895024 on sde5 (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.115374] vgs(13121): READ block 0 on sde5 (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.115479] vgs(13121): READ block 8 on sde5 (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.115632] vgs(13121): READ block 0 on sde5 (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.128351] lvs(13122): READ block 0 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.128543] lvs(13122): READ block 15628052992 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.128641] lvs(13122): READ block 15628053152 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.128739] lvs(13122): READ block 0 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.128835] lvs(13122): READ block 8 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.129001] lvs(13122): READ block 0 on sda (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.133275] lvs(13122): READ block 0 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.133445] lvs(13122): READ block 15628052992 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.133534] lvs(13122): READ block 15628053152 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.133628] lvs(13122): READ block 0 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.133723] lvs(13122): READ block 8 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.133858] lvs(13122): READ block 0 on sdb (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.133988] lvs(13122): READ block 0 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.134113] lvs(13122): READ block 15628052992 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.134204] lvs(13122): READ block 15628053152 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.134306] lvs(13122): READ block 0 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.134401] lvs(13122): READ block 8 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.134535] lvs(13122): READ block 0 on sdc (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.134675] lvs(13122): READ block 0 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.134807] lvs(13122): READ block 15628052992 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.134885] lvs(13122): READ block 15628053152 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.134985] lvs(13122): READ block 0 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.135087] lvs(13122): READ block 8 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.135239] lvs(13122): READ block 0 on sdd (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.135374] lvs(13122): READ block 0 on sde (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.135571] lvs(13122): READ block 497536 on sde1 (2 sectors)
Nov 25 10:23:40 pve kernel: [150901.135604] lvs(13122): READ block 497648 on sde1 (2 sectors)
Nov 25 10:23:40 pve kernel: [150901.135631] lvs(13122): READ block 0 on sde1 (2 sectors)
Nov 25 10:23:40 pve kernel: [150901.135738] lvs(13122): READ block 8 on sde1 (2 sectors)
Nov 25 10:23:40 pve kernel: [150901.135867] lvs(13122): READ block 0 on sde1 (4 sectors)
Nov 25 10:23:40 pve kernel: [150901.136091] lvs(13122): READ block 487894912 on sde5 (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.136209] lvs(13122): READ block 487895024 on sde5 (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.136309] lvs(13122): READ block 0 on sde5 (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.136406] lvs(13122): READ block 8 on sde5 (8 sectors)
Nov 25 10:23:40 pve kernel: [150901.136581] lvs(13122): READ block 0 on sde5 (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.168980] vgs(13130): READ block 0 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.169232] vgs(13130): READ block 15628052992 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.169353] vgs(13130): READ block 15628053152 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.169469] vgs(13130): READ block 0 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.169597] vgs(13130): READ block 8 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.169761] vgs(13130): READ block 0 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.175689] vgs(13130): READ block 0 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.175934] vgs(13130): READ block 15628052992 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.176060] vgs(13130): READ block 15628053152 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.176173] vgs(13130): READ block 0 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.176305] vgs(13130): READ block 8 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.176466] vgs(13130): READ block 0 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.176628] vgs(13130): READ block 0 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.176861] vgs(13130): READ block 15628052992 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.176989] vgs(13130): READ block 15628053152 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.177099] vgs(13130): READ block 0 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.177228] vgs(13130): READ block 8 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.177415] vgs(13130): READ block 0 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.177570] vgs(13130): READ block 0 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.177797] vgs(13130): READ block 15628052992 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.177900] vgs(13130): READ block 15628053152 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.178010] vgs(13130): READ block 0 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.178134] vgs(13130): READ block 8 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.178268] vgs(13130): READ block 0 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.178430] vgs(13130): READ block 0 on sde (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.178607] vgs(13130): READ block 497536 on sde1 (2 sectors)
Nov 25 10:23:50 pve kernel: [150911.178639] vgs(13130): READ block 497648 on sde1 (2 sectors)
Nov 25 10:23:50 pve kernel: [150911.178669] vgs(13130): READ block 0 on sde1 (2 sectors)
Nov 25 10:23:50 pve kernel: [150911.178779] vgs(13130): READ block 8 on sde1 (2 sectors)
Nov 25 10:23:50 pve kernel: [150911.178928] vgs(13130): READ block 0 on sde1 (4 sectors)
Nov 25 10:23:50 pve kernel: [150911.179120] vgs(13130): READ block 487894912 on sde5 (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.179240] vgs(13130): READ block 487895024 on sde5 (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.179344] vgs(13130): READ block 0 on sde5 (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.179444] vgs(13130): READ block 8 on sde5 (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.179616] vgs(13130): READ block 0 on sde5 (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.192421] lvs(13131): READ block 0 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.192585] lvs(13131): READ block 15628052992 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.192667] lvs(13131): READ block 15628053152 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.192766] lvs(13131): READ block 0 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.192861] lvs(13131): READ block 8 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.193005] lvs(13131): READ block 0 on sda (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.197204] lvs(13131): READ block 0 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.197362] lvs(13131): READ block 15628052992 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.197443] lvs(13131): READ block 15628053152 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.197583] lvs(13131): READ block 0 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.197679] lvs(13131): READ block 8 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.197859] lvs(13131): READ block 0 on sdb (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.198014] lvs(13131): READ block 0 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.198150] lvs(13131): READ block 15628052992 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.198243] lvs(13131): READ block 15628053152 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.198341] lvs(13131): READ block 0 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.198438] lvs(13131): READ block 8 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.198549] lvs(13131): READ block 0 on sdc (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.198684] lvs(13131): READ block 0 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.198858] lvs(13131): READ block 15628052992 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.198956] lvs(13131): READ block 15628053152 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.199078] lvs(13131): READ block 0 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.199187] lvs(13131): READ block 8 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.199323] lvs(13131): READ block 0 on sdd (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.199443] lvs(13131): READ block 0 on sde (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.199622] lvs(13131): READ block 497536 on sde1 (2 sectors)
Nov 25 10:23:50 pve kernel: [150911.199651] lvs(13131): READ block 497648 on sde1 (2 sectors)
Nov 25 10:23:50 pve kernel: [150911.199678] lvs(13131): READ block 0 on sde1 (2 sectors)
Nov 25 10:23:50 pve kernel: [150911.199786] lvs(13131): READ block 8 on sde1 (2 sectors)
Nov 25 10:23:50 pve kernel: [150911.199923] lvs(13131): READ block 0 on sde1 (4 sectors)
Nov 25 10:23:50 pve kernel: [150911.200135] lvs(13131): READ block 487894912 on sde5 (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.200256] lvs(13131): READ block 487895024 on sde5 (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.200355] lvs(13131): READ block 0 on sde5 (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.200454] lvs(13131): READ block 8 on sde5 (8 sectors)
Nov 25 10:23:50 pve kernel: [150911.200619] lvs(13131): READ block 0 on sde5 (8 sectors)

If I kill pmxcfs the strange access pattern stops immediately.

Any advice on what's causing this strange behavior and how to further debug and/or solve it?


Thanks and regards,
Heiner
 
the daemon 'pvestatd'
checks the status/usage of nodes/vms/storages every 10 seconds
 
Damn.. you hit the nail on the head, Dominik.
Thank you very much :)

Watching the daemon working via pvestatd start --debug shows it is exactly in sync with the access pattern and noise.

For future reference to anyone with the same issue: your hint lead me to this post by spirit detailing how pvestatd is invoking lvm to stat even disks not configured in pve.
Later in the thread rubictus posted the simple, yet effective workaround of using lvmetad to cache lvm metadata and minimize disk polls, which should be applicable to most home or test environments where disk spindown might be an issue.


Thanks again for the great support!
You made my day :)

Regards,
Heiner
 
You must log in or register to reply here.
Top Bottom