pveperf hangs when doing disk checks.

[nktech1135]

Hello.
Trying to run pveperf on my proxmox host and get the following.

root@chestnut:~# pveperf
CPU BOGOMIPS:      115223.88
REGEX/SECOND:      1572112   
HD SIZE:           93.99 GB (/dev/mapper/pve-root)

After the HD SIZE line the process just hangs. It doesn't respond to ctrl c i have to use kill to get rid of it. I can reproduce this on multiple hosts.
Is it possible it has to do with the fact my systems are running on lvm and zfs instead of having physical disks to interact with?

root@chestnut:~# pveversion -v
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-15 (running version: 6.2-15/48bd51b6)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-4.15: 5.4-9
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-4
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-10
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-10
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-6
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-19
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

 

[oguz]

hi,

Is it possible it has to do with the fact my systems are running on lvm and zfs instead of having physical disks to interact with?

works here on zfs and lvm

do you see anything interesting in dmesg, journalctl and syslog while the hang occurs?

 

[bobmc]

hmm works on my system (6.1.2)

what happens with pveperf /Tank

 

[nktech1135]

hi,


works here on zfs and lvm

do you see anything interesting in dmesg, journalctl and syslog while the hang occurs?

Hi.
No, dmesg and syslog aren't really telling me anything.
Anything else i should check?

 

[bionemesis]

Did you ever get to the bottom of this? I'm running into the same issue.

 

[l_fish]

Same issue here. PVE 6.3-3 on zfs.

 

[oguz]

can you post any logs?

 

[l_fish]

No, same as nktech1135 wrote: dmesg and syslog do not contain any relevant entries.

But I found a subprocess started by pveperf that uses 100% cpu permanently:

sh -c echo 3 > /proc/sys/vm/drop_caches

Maybe this helps?

 

[oguz]

sh -c echo 3 > /proc/sys/vm/drop_caches

this is to drop cached memory, shouldn't cause any hangs.
also write caches to the disk will be flushed by the kernel when this is run, so this might take long if you have slow disks and a lot of memory (although this is non-destructive, read more: https://www.kernel.org/doc/Documentation/sysctl/vm.txt)

dmesg and syslog do not contain any relevant entries.

in that case maybe an strace can be interesting to see:
strace -o pveperf.strace pveperf and then attach the resulting file here

EDIT: correction for cache drop behavior

 

[nktech1135]

Hi.
No i haven't gotten to the bottom of this either. Here is the strace file you wanted.

 

[oguz]

after checking the trace you sent, i saw that your pveperf process hangs right before the fsync tests start. there happens to be a sh -c echo 3 > /proc/sys/vm/drop_caches call before this, and you also said that it uses heavy cpu.

so it could make sense if drop_caches is causing the hang (because maybe your disk is slow), and then pveperf hangs until you kill it with SIGTERM.

how long have you waited at this hang (approximately)?

could you post another trace this time with timestamps? (sorry for not asking earlier)

strace -ft -o pveperf2.strace pveperf

EDIT:

dmesg and syslog do not contain any relevant entries.

could you please post them anyway?

also the output from ps auxfw while its running can be helpful