Node CPU 100% while VM's idle

[PVTD]

New Proxmox user here,

For a few days now I have been getting 100% CPU load and I have no idea why, how do I go about identifying the issue?
Proxmox is fully updated, no IO delay, the hardware hasn't been changed in the past few weeks so i'm out of idea's

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                    
   2283 root      20   0 2493320   7732   4564 S  2397   0.0  11758:08 accounts-daemon            
2520228 root      20   0   10484   4236   3228 R   0.7   0.0   0:00.08 top                        
    762 root      39  19       0      0      0 S   0.3   0.0   0:00.23 dbuf_evict                  
2516167 www-data  20   0  359852 140740  12460 S   0.3   0.1   0:02.20 pveproxy worker            
2517860 www-data  20   0  355564 132488   8432 S   0.3   0.1   0:00.74 pveproxy worker            
      1 root      20   0  164692  10944   7536 S   0.0   0.0   0:00.74 systemd                    
      2 root      20   0       0      0      0 S   0.0   0.0   1:49.68 kthreadd                    
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp                      
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp                  
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 netns                      
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri
     10 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq                
     11 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_            
     12 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace            
     13 root      20   0       0      0      0 S   0.0   0.0   0:01.71 ksoftirqd/0                
     14 root      20   0       0      0      0 I   0.0   0.0   0:11.97 rcu_sched                  
     15 root      rt   0       0      0      0 S   0.0   0.0   0:00.09 migration/0

If you need anything, let me know, I'm still in the learning process.
PS: VM's and LXC CPU stats are reporting correctly

Le PVTD

 

[bbgeek17]

2283 root 20 0 2493320 7732 4564 S 2397 0.0 11758:08 accounts-daemon

This line is pretty self-explaining. This service has been taking a lot of cpu for the last ~8 days. Since accounts-daemon is part of accountservices package, and that package is not part of Proxmox standard installation, it indicates that you installed it at some point.
Whether it has a known leak that is fixed or an unknown leak exposed by your specific environment - anybody's guess is as good as yours at this point.
Its up to you how to proceed: restart service, reboot host, troubleshoot and report the issue.
Obviously, since its not part of PVE - its unlikely to be fixed by PVE developers.

edit: fixed time

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[PVTD]

This service has been taking a lot of cpu for the last ~489 day. Since accounts-daemon is part of accountservices package, and that package is not part of Proxmox standard installation, it indicates that you installed it at some point.

I started using Proxmox since march the third, not sure where the 489 days come from.
The 100% started happening since the 4th of this month, and that's when I started using LXC containers, using Turnkey templates.
I don't think I installed anything else? At least i can't recall.
How would I go about stopping this?

Edit: I ran apt remove accountsservice then cleaned it. Now waiting to see if its really stopped.

 

[bbgeek17]

yes, the time is minutes so about 8 days

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[PVTD]

yes, the time is minutes so about 8 days

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Makes sense then hehe

Well, i can't get rid of the accounts-daemon thing. I uninstalled accountsservice and it just came back. At least console confirmed the removal.

 

[Dunuin]

Keep in mind that LXCs aren't really isolated like VMs. All you run in a LXC will be run on your host OS. So running htop or top on the PVE host will also show processes that are running in all LXCs. I guess one of your turnkey LXCs is running the "accountsservice" and thats what you are seeing. So check your LXCs for the "accountsservice" and remove/stop it there too.

 

[PVTD]

Keep in mind that LXCs aren't really isolated like VMs. All you run in a LXC will be run on your host OS. So running htop or top on the PVE host will also show processes that are running in all LXCs. I guess one of your turnkey LXCs is running the "accountsservice" and thats what you are seeing. So check your LXCs for the "accountsservice" and remove/stop it there too.

I took your advice, and just removed all LXC's with the idea to run them on a VM with docker, and it worked fine for the entire week.
Yesterday (while nothing changed, and I was at work) suddenly it popped back up.
So, right now I am supper confused. Before I could also reboot Proxmox and it would not show up for a while, now when rebooted, the CPU is pretty much instantly stuck at 100% with accounts-daemon
I changed literally nothing since removing the LXCs to monitor the server.

systemctl stop accounts-daemon gives me accounts-daemon service not loaded

 

[PVTD]

Ok, after tons of googling, I found out htop had more info then top, and ended up finding that the file was located in /var/accounts-daemon/accounts-daemon.conf
I deleted the file (as people claimed it to not be part of Proxmox) and restarted. File has not come back yet, but I'll monitor it.