drop_caches don't finisched on pveperf

[udo]

Hi,
I have on an server the effect, that dorp_caches don't finisched during an pveperf run.

ps aux | grep echo
root      835293 99.9  0.0   4292   708 pts/3    R+   Jul20 1140:50 sh -c echo 3 > /proc/sys/vm/drop_caches

There run an VM with IO, but the load is quite low and the underlaying storage system are an zfs striped mirror of 4 samsung enterprise disks.

There are no io-waits:

top
top - 07:59:21 up 10 days, 22:03,  6 users,  load average: 4.29, 4.50, 4.93
Tasks: 694 total,   3 running, 469 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.5 us,  4.3 sy,  0.0 ni, 88.0 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 16495880+total, 53446384 free, 10989219+used,  1620224 buff/cache
KiB Swap:  8388604 total,  4656968 free,  3731636 used. 53238628 avail Mem
...

I used not so big values for zfs_arc

cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_min=4294967296
options zfs zfs_arc_max=8589934592

Versions:

pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.17-3-pve)
pve-manager: 5.2-5 (running version: 5.2-5/eb24855a)
pve-kernel-4.15: 5.2-4
pve-kernel-4.15.18-1-pve: 4.15.18-15
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph: 12.2.5-pve1
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-35
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-24
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-28
pve-container: 2.0-24
pve-docs: 5.2-4
pve-firewall: 3.0-13
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-29
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9

Any hint?

Udo

 

[udo]

Info about IO:

zpool iostat 1
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
rpool        473G   415G     50    229   325K  7.33M
rpool        473G   415G    113  1.09K   779K  20.7M
rpool        473G   415G     40    823   164K  15.7M
rpool        473G   415G     89  1.74K   364K  55.2M
rpool        473G   415G     23    889  95.9K  12.4M
rpool        473G   415G     28    980   128K  13.7M
rpool        473G   415G     23    962  95.9K  13.5M
rpool        473G   415G     36    919   148K  12.7M
rpool        473G   415G     35  1.65K   184K  45.3M
rpool        473G   415G     31    871   176K  11.9M
rpool        473G   415G     28    939   120K  12.9M
rpool        473G   415G     28    921   120K  13.7M
rpool        473G   415G     24    933   104K  12.9M
rpool        473G   415G     56  1.69K   280K  47.5M
rpool        473G   415G    527    955  2.07M  20.4M
rpool        473G   415G     35    949   148K  12.8M
rpool        473G   415G     26    849   108K  11.9M
rpool        473G   415G     30    873   132K  12.1M
rpool        473G   415G     28  1.63K   140K  43.2M
rpool        473G   415G     42  1.09K   172K  21.6M
rpool        473G   415G     26    915   116K  15.9M
rpool        473G   415G     39  1.02K   160K  13.6M
rpool        473G   415G     35    859   240K  11.6M
rpool        473G   415G     33  1.69K   148K  48.1M
rpool        473G   415G    198    871  4.67M  12.3M
rpool        473G   415G     31    803   188K  11.1M
rpool        473G   415G     35    943   196K  13.0M
rpool        473G   415G     25  1.03K   108K  14.0M
rpool        473G   415G     28    809   124K  10.9M
^C

Udo

 

[Nemesiz]

It happened to me too. Only reboot (maybe hard reboot) kills the process.

 

[LnxBil]

On the other hand:

Why do you drop your caches?

 

[Nemesiz]

It is #pveperf script

 

[LnxBil]

It is #pveperf script

Oh, that's not good. Sorry, I missed it in the topic description.

 

[udo]

Hi,
after migration the most IO active VMs away, the drop_caches get finished and pveperf work again (but need a (too) long time):

pveperf
CPU BOGOMIPS:      192050.40
REGEX/SECOND:      2483783
HD SIZE:           671.28 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND:     2403.32
DNS EXT:           14.87 ms
DNS INT:           0.86 ms

But imho this should not be an problem, there must something wrong with ZOL.

#edit Perhaps kernel related, because it's don't happens on the cluster member where the VMs are now running, and an node with the same kernel and no load (but hdds, not ssds) take a long time also:

root@pve07:~# time pveperf
CPU BOGOMIPS:      120011.40
REGEX/SECOND:      1625253
HD SIZE:           1303.44 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND:     162.51
DNS EXT:           15.25 ms
DNS INT:           0.99 ms (lv3.metaways.net)

real    1m50.659s
user    0m3.180s
sys     1m42.751s

Linux pve07 4.15.17-3-pve #1 SMP PVE 4.15.17-13 (Mon, 18 Jun 2018 17:15:04 +0200) x86_64 GNU/Linux
top - 04:17:33 up 22 days, 12:02,  2 users,  load average: 0.96, 1.31, 1.31

Udo

 

[Nemesiz]

There are talks about ZFS and drop cache google