Highload on proxmox and lvm issue

[sahostking]

Hi

I updated proxmox a few days ago and ever since getting weird errors and high load above 300+.

I noticed this today:

root 26559 0.0 0.0 15832 8720 ? D 08:48 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 26635 0.0 0.0 15832 8600 ? D 07:51 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 26721 0.0 0.0 15852 8416 ? D 04:01 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 26931 0.0 0.0 15832 8684 ? D 06:38 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27044 0.0 0.0 15832 8620 ? D 05:21 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27047 0.0 0.0 15832 8480 ? D 08:49 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27142 0.0 0.0 15832 8632 ? D 04:02 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27157 0.0 0.0 15832 8604 ? D 07:52 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27276 0.0 0.0 15832 8632 ? D 06:39 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27411 0.0 0.0 15832 8744 ? D 08:50 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27459 0.0 0.0 15832 8604 ? D 04:03 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27498 0.0 0.0 15832 8660 ? D 05:22 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27626 0.0 0.0 15832 8628 ? D 07:53 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27666 0.0 0.0 15832 8624 ? D 06:40 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27800 0.0 0.0 15832 8708 ? D 08:51 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27808 0.0 0.0 15832 8604 ? D 04:04 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 27918 0.0 0.0 15832 8608 ? D 05:23 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28086 0.0 0.0 15832 8720 ? D 07:54 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28113 0.0 0.0 15832 8452 ? D 06:41 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28175 0.0 0.0 15832 8744 ? D 04:05 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28185 0.0 0.0 15832 8652 ? D 08:52 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28349 0.0 0.0 15832 8424 ? D 05:24 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28509 0.0 0.0 15852 8620 ? D 04:06 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28512 0.0 0.0 15832 8544 ? D 06:42 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28528 0.0 0.0 15832 8708 ? D 07:55 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28597 0.0 0.0 15832 8640 ? D 08:53 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28788 0.0 0.0 15832 8556 ? D 05:25 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28831 0.0 0.0 15832 8512 ? D 04:07 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28920 0.0 0.0 15832 8532 ? D 06:43 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28991 0.0 0.0 15832 8700 ? D 08:54 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 28993 0.0 0.0 15832 8536 ? D 07:56 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29187 0.0 0.0 15832 8564 ? D 04:08 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29255 0.0 0.0 15832 8384 ? D 05:26 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29350 0.0 0.0 15832 8572 ? D 06:44 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29369 0.0 0.0 15832 8644 ? D 08:55 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29495 0.0 0.0 15832 8612 ? D 07:57 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29538 0.0 0.0 15832 8556 ? D 04:09 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29677 0.0 0.0 15832 8708 ? D 05:27 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29716 0.0 0.0 15832 8560 ? D 02:37 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29751 0.0 0.0 15832 8704 ? D 08:56 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29771 0.0 0.0 15832 8628 ? D 06:45 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29873 0.0 0.0 15832 8660 ? D 04:10 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 29957 0.0 0.0 15832 8604 ? D 07:58 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 30090 0.0 0.0 15832 8660 ? D 05:28 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 30198 0.0 0.0 15832 8608 ? D 06:46 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 30235 0.0 0.0 15832 8604 ? D 04:11 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 30403 0.0 0.0 15832 8536 ? D 02:39 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 30421 0.0 0.0 15832 8684 ? D 07:59 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 30538 0.0 0.0 15832 8640 ? D 05:29 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 30590 0.0 0.0 15832 8568 ? D 04:12 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 30687 0.0 0.0 15832 8652 ? D 06:47 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent
root 30882 0.0 0.0 15832 8676 ? D 08:00 0:00 lvs --noheadings --separator : -o lv_attr,lv_name,data_percent,metadata_percent

 

[sahostking]

Ok we found that disabling our nagios monitoring script that monitors LVM usage caused this as it runs but never gets a result back and hence just stalls.

Strange only on last kernel this happened and this has been running on our servers for years.

But atleast I found a fix and will monitor it manually for awhile