I can't to kill the process "echo 3 > /proc/sys/vm/drop_caches"

[sa10]

I suspect it started after my trying to run
pveperf /PoolZFS
The pveperf worked too long and I killed him.
But I can't to kill this process

ps -Flww -p 34740
F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
0 R root 34740 1 99 80 0 - 596 - 764 9 12:45 ? 213504-02:19:01 sh -c echo 3 > /proc/sys/vm/drop_caches

kill -9 34740 has no effect

This process is using 100% of a CPU

Maybe someone has ideas how to kill this process without rebooting the system?



pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.13-2-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-2
pve-kernel-helper: 6.1-2
pve-kernel-5.3.13-2-pve: 5.3.13-2
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.12-pve1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-10
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12+deb10u1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-2
pve-cluster: 6.1-3
pve-container: 3.0-18
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 4.3.0-1
pve-zsync: 2.0-1
qemu-server: 6.1-4
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

 

[sa10]

I found similar problem here
https://github.com/zfsonlinux/spl/issues/388

 

[gerfloim]

Same problem here:

Linux sv-pve 5.3.18-2-pve #1 SMP PVE 5.3.18-2 (Sat, 15 Feb 2020 15:11:52 +0100) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Feb 29 17:31:28 2020 from 192.168.10.134
root@sv-pve:~#  time pveperf /rz2pool
CPU BOGOMIPS:      166049.60
REGEX/SECOND:      1820239
HD SIZE:           3092.23 GB (rz2pool)
FSYNCS/SECOND:     1874.99
DNS EXT:           596.08 ms
DNS INT:           556.12 ms (imedos.de)

real    0m16,011s
user    0m3,552s
sys     0m3,695s
root@sv-pve:~# #running 107
root@sv-pve:~# time pveperf /rz2pool
CPU BOGOMIPS:      166049.60
REGEX/SECOND:      1570540
HD SIZE:           3092.23 GB (rz2pool)
FSYNCS/SECOND:     1787.14
DNS EXT:           577.47 ms
DNS INT:           810.67 ms (imedos.de)

real    0m19,804s
user    0m3,609s
sys     0m7,007s
root@sv-pve:~# #running 107,101
root@sv-pve:~# time pveperf /rz2pool
CPU BOGOMIPS:      166049.60
REGEX/SECOND:      1567023
HD SIZE:           3092.23 GB (rz2pool)
FSYNCS/SECOND:     2180.13
DNS EXT:           29.88 ms
DNS INT:           2.23 ms (imedos.de)

real    0m16,262s
user    0m3,871s
sys     0m7,623s
root@sv-pve:~# #running 107,101,103
root@sv-pve:~# time pveperf /rz2pool
CPU BOGOMIPS:      166049.60
REGEX/SECOND:      1535667
HD SIZE:           3092.23 GB (rz2pool)
FSYNCS/SECOND:     1810.33
DNS EXT:           28.25 ms
DNS INT:           2.04 ms (imedos.de)

real    1m53,397s
user    0m3,818s
sys     1m44,345s
root@sv-pve:~# #running 107,101,103 
root@sv-pve:~# time pveperf /rz2pool
CPU BOGOMIPS:      166049.60
REGEX/SECOND:      1480075
HD SIZE:           3092.23 GB (rz2pool)
FSYNCS/SECOND:     2083.82
DNS EXT:           30.21 ms
DNS INT:           2.13 ms (imedos.de)

real    1m29,123s
user    0m3,774s
sys     1m20,578s
root@sv-pve:~# #running 107,101,103,104
root@sv-pve:~# time pveperf /rz2pool
CPU BOGOMIPS:      166049.60
REGEX/SECOND:      1466412
HD SIZE:           3092.24 GB (rz2pool)

Then this hangs forever and I can see in htop:

sh -c echo 3 > /proc/sys/vm/drop_caches

consuming one CPU with 100%

My fear is, that it not only prevents pveperf from finishing, but also destroyes my server performance!

How can we get rid of this?

 

[sa10]

Do not attempt to perform pveperf with significant disk load on the production system.
This process "echo 3 > /proc/sys/vm/drop_caches" will be run until the disk load be reduced.
To reproduce the situation, run a long benchmark and run this command "echo 3 > /proc/sys/vm/drop_caches" in same time.
You will can't to kill this process until benchmark complete.
I don't think this is normal

 

[rholighaus]

I have the same problem.

 

[UdoB]

Hello,

did anybody manage to actually get rid of that process without rebooting?

Interestingly this problem existed since 2014 and the bug (Github link above) was closed in May 2018...

 

[UdoB]

Okaaaaay, 20 hours later the process is gone. Without reboot.
(Probably it run from 10 a.m. until 5 p.m. - that's when CPU/system load dropped by a constant offset.)