High system charge

[caplam]

Hello,
My installation is made with 2 nodes. They are not very powerful and will be replaced soon with much more powerful one.

My problem is that on one of the nodes i have permanently a high system charge and can't figure out why.
Here is the sum up of the host.

Average cpu usage is around 50%
Average io delay is less than 1%
Hardware is a HP N54L with 8Gb ram. Proxmox and datastore is on a 480Gb ssd (sata).

root@n54:~# pveperf
CPU BOGOMIPS:      8785.62
REGEX/SECOND:      1007582
HD SIZE:           44.04 GB (/dev/mapper/pve-root)
BUFFERED READS:    247.24 MB/sec
AVERAGE SEEK TIME: 0.11 ms
FSYNCS/SECOND:     483.83
DNS EXT:           83.70 ms
DNS INT:           2.02 ms

I'd like to dig to identify the root cause before migrating all my vm and containers on another node.

edit: add ssd map

sda                            8:0    1 465.8G  0 disk
├─sda1                         8:1    1     1M  0 part
├─sda2                         8:2    1   256M  0 part
└─sda3                         8:3    1 465.5G  0 part
  ├─pve-swap                 253:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                 253:1    0    45G  0 lvm  /
  ├─pve-data_tmeta           253:2    0     4G  0 lvm  
  │ └─pve-data-tpool         253:4    0 388.6G  0 lvm  
  │   ├─pve-data             253:5    0 388.6G  0 lvm  
  │   ├─pve-vm--110--disk--0 253:6    0    16G  0 lvm  
  │   └─pve-vm--104--disk--0 253:7    0    32G  0 lvm  
  └─pve-data_tdata           253:3    0 388.6G  0 lvm  
    └─pve-data-tpool         253:4    0 388.6G  0 lvm  
      ├─pve-data             253:5    0 388.6G  0 lvm  
      ├─pve-vm--110--disk--0 253:6    0    16G  0 lvm  
      └─pve-vm--104--disk--0 253:7    0    32G  0 lvm

 

[t.lamprecht]

Hi! As a first simple check you could use top or htop to see what process uses the CPU constantly.

 

[caplam]

top - 11:23:21 up 15 days, 6 min,  1 user,  load average: 1.59, 3.34, 3.82
Tasks: 281 total,   4 running, 187 sleeping,   0 stopped,   0 zombie
%Cpu(s): 19.2 us, 14.8 sy,  0.0 ni, 65.3 id,  0.2 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem :  8039432 total,  3806792 free,  3787720 used,   444920 buff/cache
KiB Swap:  8388604 total,  5717668 free,  2670936 used.  3880008 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                    
 9893 root      20   0 3112800 1.361g   5704 R  24.9 17.8   1181:59 kvm                                                                                                                                                        
 2163 root      20   0 2105168 735468   5600 S  18.3  9.1   5004:12 kvm                                                                                                                                                        
 2275 root      20   0 2063020 225108   5644 S  11.6  2.8   2502:45 kvm                                                                                                                                                        
 3166 root      20   0 3105500 244200   5384 R  11.3  3.0   2351:10 kvm                                                                                                                                                        
 1290 root      rt   0  194248  69420  51484 S   1.7  0.9 318:42.78 corosync                                                                                                                                                  
   29 root      25   5       0      0      0 S   0.7  0.0 126:30.84 ksmd                                                                                                                                                      
 1345 root      20   0  512912  25672   6348 S   0.7  0.3  41:39.25 pve-firewall                                                                                                                                              
 8437 postfix   20   0  322480  59292  28860 R   0.7  0.7   0:06.03 mongod                                                                                                                                                    
 9463 Debian-+  20   0  356300  76908  33816 S   0.7  1.0   0:07.61 mongod                                                                                                                                                    
22268 root      20   0   45056   3932   3112 R   0.7  0.0   0:00.29 top                                                                                                                                                        
32683 110       20   0 1156136   7800    216 S   0.7  0.1   1:50.99 tvheadend                                                                                                                                                  
  384 root      20   0       0      0      0 S   0.3  0.0   0:05.18 kdvb-ad-0-fe-0                                                                                                                                            
 1140 root      20   0  876656   3308   1632 S   0.3  0.0   5:46.92 rrdcached                                                                                                                                                  
 7566 www-data  20   0  564784  43124  12848 S   0.3  0.5   0:05.01 pveproxy worker                                                                                                                                            
 8720 Debian-+  20   0 3589256 240420  22572 S   0.3  3.0   0:35.21 java                                                                                                                                                      
 8848 www-data  20   0  564712  43608  12836 S   0.3  0.5   0:04.51 pveproxy worker

When posting output of lsblk i realized some vm or ct disks where missing and i remembered i moved some to a shared nfs storage. I just moved them back on local lvm. For now charge is still important (around 3).



edit: perhaps a bug in gui.
I moved a ct disk from local-lvm to nfs shared storage. Source disk stayed ( vm-108-disk-0) in local lvm. From what i think it's normal.
When i moved back the disk to local lvm the name became vm-108-disk-1. The old vm-108-disk-0 was there but not listed in gui.
When i went to storage view to delete it i couldn't. A message told me to do it from guest hardware panel what is impossible as disk is not listed.
Destroying old disk had be done with cli (lvremove /pve/ vm-108-disk-0)