High CPU use on systemd-journal caused by proxmox-datacenter-api

B

BySempron

New Member
Jul 17, 2024
2
0
1
Spain
Hi!

I recently updated my PDM from an old version to the last version and CPU and memory use has increased a lot:

Captura de pantalla 2026-06-17 140635.jpg

Top says the problem is systemd-journal:

Captura de pantalla 2026-06-17 140715.jpg

And, when I run systemd to view the problem, it gets flooded by this message:

1781698085136.png

datacenter proxmox-datacenter-api[665]: could not read task from archive file: incomplete frame
Click to expand...

Here is my system report and my package versions:

Code: 
## COMMANDS

 $ `date -R`
```
Wed, 17 Jun 2026 14:08:59 +0200
```

$ `proxmox-datacenter-manager-admin versions --verbose`
```
proxmox-datacenter-manager-meta      1.1.0         running kernel: 7.0.6-2-pve
proxmox-datacenter-manager           1.1.4         running version: 1.1.4    
proxmox-kernel-helper                9.2.0                                  
proxmox-kernel-7.0                   7.0.6-2                                
proxmox-kernel-7.0.6-2-pve-signed    7.0.6-2                                
proxmox-kernel-7.0.2-2-pve-signed    7.0.2-2                                
proxmox-kernel-6.17                  6.17.13-13                              
proxmox-kernel-6.17.13-13-pve-signed 6.17.13-13                              
proxmox-kernel-6.17.2-2-pve-signed   6.17.2-2                                
ifupdown2                            3.3.0-1+pmx12                          
proxmox-mail-forward                 1.0.3                                  
proxmox-mini-journalreader           1.6                                    
proxmox-offline-mirror-helper        0.7.4                                  
pve-xtermjs                          6.0.0-1                                
zfsutils-linux                       2.4.2-pve1
```

$ `proxmox-datacenter-manager-admin remote list`
```
Proxmox VE node sol-pve01:
    auth id: root@pam!pdm-admin-datacenter
    token:
    node: sol-pve01.REDACTED.lan:443
Proxmox VE node HomeLab:
    auth id: root@pam!pdm-admin-datacenter
    token:
    nodes:
        pve1.proxmox.lan:443,fingerprint=REDACTED
        pve2.proxmox.lan:443,fingerprint=REDACTED
        pve3.proxmox.lan
Proxmox Backup Server node sol-pbs01:
    auth id: root@pam!pdm-admin-datacenter
    token:
    node: sol-pbs01.REDACTED.lan,fingerprint=REDACTED
Proxmox VE node net1:
    auth id: root@pam!pdm-admin-datacenter
    token:
    node: net1.proxmox.lan,fingerprint=REDACTED
```

$ `proxmox-datacenter-manager-admin remote subscriptions`
```
Remote sol-pve01 subscription status: None
    node sol-pve01: status=notfound,sockets=2,level=None

Remote HomeLab subscription status: None
    node pve2: status=notfound,sockets=1,level=None
    node pve3: status=notfound,sockets=1,level=None
    node pve1: status=notfound,sockets=1,level=None

Remote sol-pbs01 subscription status: None
    node localhost: status=notfound,level=None

Remote net1 subscription status: None
    node net1: status=notfound,sockets=1,level=None
```

$ `proxmox-datacenter-manager-admin support-status get`
```
┌─────────┬────────────────────────────────────────────────────────────────────┐
│ Name    │ Value                                                              │
╞═════════╪════════════════════════════════════════════════════════════════════╡
│ status  │ invalid                                                            │
├─────────┼────────────────────────────────────────────────────────────────────┤
│ message │ Too many remote nodes without active basic or higher subscription! │
├─────────┼────────────────────────────────────────────────────────────────────┤
│ url     │ https://pdm.proxmox.com/faq.html                                   │
└─────────┴────────────────────────────────────────────────────────────────────┘
```

$ `proxmox-boot-tool status`
```
Re-executing '/sbin/proxmox-boot-tool' in new private mount namespace..
```
STDERR:
```
E: /etc/kernel/proxmox-boot-uuids does not exist.

```

$ `df -h -T`
```
Filesystem           Type      Size  Used Avail Use% Mounted on
udev                 devtmpfs  590M     0  590M   0% /dev
tmpfs                tmpfs     105M  668K  105M   1% /run
/dev/mapper/pdm-root ext4      8.3G  5.8G  2.1G  74% /
tmpfs                tmpfs     522M     0  522M   0% /dev/shm
tmpfs                tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                tmpfs     1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs                tmpfs     522M     0  522M   0% /tmp
tmpfs                tmpfs     1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
```

$ `lsblk --ascii -M -o +HOTPLUG,ROTA,PHY-SEC,FSTYPE,MODEL,TRAN`
```
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS HOTPLUG ROTA PHY-SEC FSTYPE      MODEL         TRAN
sda            8:0    0   10G  0 disk                   0    1     512             QEMU HARDDISK
|-sda1         8:1    0 1007K  0 part                   0    1     512                          
|-sda2         8:2    0  512M  0 part                   0    1     512 vfat                    
`-sda3         8:3    0  9.5G  0 part                   0    1     512 LVM2_member              
  |-pdm-swap 252:0    0    1G  0 lvm  [SWAP]            0    1     512 swap                    
  `-pdm-root 252:1    0  8.5G  0 lvm  /                 0    1     512 ext4                    
sr0           11:0    1 1024M  0 rom                    0    0     512             QEMU DVD-ROM  ata
```

$ `ls -l /dev/disk/by-id /dev/disk/by-path`
```
/dev/disk/by-id:
total 0
lrwxrwxrwx 1 root root  9 Jun 17 14:03 ata-QEMU_DVD-ROM_QM00003 -> ../../sr0
lrwxrwxrwx 1 root root 10 Jun 17 14:03 dm-name-pdm-root -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jun 17 14:03 dm-name-pdm-swap -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jun 17 14:03 dm-uuid-LVM-9FyG74IRIz45StsncMUgC7V72q4BwaG36Z5VBz4qTwTad1Mte50afz9lzgVBjjKa -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jun 17 14:03 dm-uuid-LVM-9FyG74IRIz45StsncMUgC7V72q4BwaG3Pm2KlwPNWAelb8jpAxlpFP4fOln9HJlc -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jun 17 14:03 lvm-pv-uuid-Lfgkx8-eqGe-uZrB-f94K-L0QP-ZFro-qgc1k5 -> ../../sda3
lrwxrwxrwx 1 root root  9 Jun 17 14:03 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 17 14:03 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 17 14:03 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 17 14:03 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part3 -> ../../sda3

/dev/disk/by-path:
total 0
lrwxrwxrwx 1 root root   9 Jun 17 14:03 pci-0000:00:01.1-ata-2 -> ../../sr0
lrwxrwxrwx 1 root root   9 Jun 17 14:03 pci-0000:00:01.1-ata-2.0 -> ../../sr0
lrwxrwxrwx 1 root root   9 Jun 17 14:03 pci-0000:01:01.0-scsi-0:0:0:0 -> ../../sda
drwxr-xr-x 5 root root 100 Jun 17 14:03 pci-0000:01:01.0-scsi-0:0:0:0-part
lrwxrwxrwx 1 root root  10 Jun 17 14:03 pci-0000:01:01.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root  10 Jun 17 14:03 pci-0000:01:01.0-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root  10 Jun 17 14:03 pci-0000:01:01.0-scsi-0:0:0:0-part3 -> ../../sda3
```

$ `zpool status`
```

```
STDERR:
```
no pools available

```

$ `zfs list`
```

```
STDERR:
```
no datasets available

```

$ `zarcstat `
```
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
14:08:59     0       0     0       0     0      0    0    960  32.6M   1.3G
```

$ `ip -details -statistics address`
```
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 allmulti 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 524280 tso_max_segs 65535 gro_max_size 65536 gso_ipv4_max_size 65536 gro_ipv4_max_size 65536
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
    RX:  bytes packets errors dropped  missed   mcast          
       7570730    4581      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns          
       7570730    4581      0       0       0       0
2: nic0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:3e:c6:d0 brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 1500 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 gso_ipv4_max_size 65536 gro_ipv4_max_size 65536 parentbus virtio parentdev virtio2
    altname enp0s18
    altname enxbc24113ec6d0
    inet 192.168.1.48/24 scope global nic0
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fe3e:c6d0/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
    RX:  bytes packets errors dropped  missed   mcast          
       1174530    2967      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns          
       6441064    2293      0      11       0       0
```

$ `ip -4 route show`
```
default via 192.168.1.1 dev nic0 proto kernel onlink
192.168.1.0/24 dev nic0 proto kernel scope link src 192.168.1.48
```

$ `ip -6 route show`
```
fe80::/64 dev nic0 proto kernel metric 256 pref medium
```

## FUNCTIONS

#### System Load & Uptime
$ `top -b -c -w512 -n 1 -o TIME`
```
top - 14:08:59 up 5 min,  0 users,  load average: 1.28, 0.91, 0.42
Tasks: 121 total,   2 running, 119 sleeping,   0 stopped,   0 zombie
%Cpu(s): 39.1 us, 26.1 sy,  0.0 ni, 30.4 id,  0.0 wa,  0.0 hi,  4.3 si,  0.0 st
MiB Mem :   1966.2 total,   1305.0 free,    340.7 used,    462.1 buff/cache    
MiB Swap:   1044.0 total,   1044.0 free,      0.0 used.   1625.5 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    242 root      20   0  280076 173640 172324 S  90.9   8.6   4:57.90 /usr/lib/systemd/systemd-journald
    662 www-data  20   0 1073160  84376  33332 S  27.3   4.2   1:39.96 /usr/libexec/proxmox/proxmox-datacenter-api
     12 root      20   0       0      0      0 I   0.0   0.0   0:01.29 [kworker/u8:0-events_power_efficient]
     15 root      20   0       0      0      0 R   0.0   0.0   0:01.18 [rcu_preempt]
      1 root      20   0   23660  14424  10896 S   0.0   0.7   0:01.00 /sbin/init
     45 root      20   0       0      0      0 I   0.0   0.0   0:00.63 [kworker/1:1-events_freezable]
     25 root      20   0       0      0      0 S   0.0   0.0   0:00.47 [ksoftirqd/1]
     14 root      20   0       0      0      0 S   0.0   0.0   0:00.32 [ksoftirqd/0]
    641 nginx     20   0   28444  11212   7960 S   0.0   0.6   0:00.30 nginx: worker process
     11 root      20   0       0      0      0 I   0.0   0.0   0:00.13 [kworker/0:1-events]
    264 root      20   0   37008  10456   7960 S   0.0   0.5   0:00.10 /usr/lib/systemd/systemd-udevd
    131 root      20   0       0      0      0 I   0.0   0.0   0:00.08 [kworker/0:2-events]
     69 root      20   0       0      0      0 I   0.0   0.0   0:00.05 [kworker/u8:5-events_power_efficient]
     35 root      20   0       0      0      0 I   0.0   0.0   0:00.04 [kworker/u8:1-async]
    552 root      20   0   18400   8532   7400 S   0.0   0.4   0:00.04 /usr/lib/systemd/systemd-logind
    648 root      20   0  256840  25344  20624 S   0.0   1.3   0:00.02 /usr/libexec/proxmox/proxmox-datacenter-privileged-api
     65 root      20   0       0      0      0 S   0.0   0.0   0:00.01 [scsi_eh_2]
     67 root      20   0       0      0      0 I   0.0   0.0   0:00.01 [kworker/u8:3-events_unbound]
    549 message+  20   0    8316   4716   4216 S   0.0   0.2   0:00.01 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [kthreadd]
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [pool_workqueue_release]
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/R-rcu_gp]
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/R-sync_wq]
```

## FILES

### General System Info

`$ cat '/etc/hostname'`
```
datacenter
```

`$ cat '/etc/hosts'`
```
127.0.0.1 localhost.localdomain localhost
192.168.1.48 datacenter.lan datacenter

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
```

`$ cat '/etc/network/interfaces'`
```
auto lo
iface lo inet loopback

auto nic0
iface nic0 inet static
    address 192.168.1.48/24
    gateway 192.168.1.1


source /etc/network/interfaces.d/*
```

`$ cat '/etc/apt/sources.list'`
```

```

`$ cat '/etc/apt/sources.list.d/proxmox.sources'`
```
Types: deb
URIs: http://download.proxmox.com/debian/pdm
Suites: trixie
Components: pdm-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
```

`$ cat '/etc/apt/sources.list.d/nginx.list'`
```
deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/debian trixie nginx
```

`$ cat '/etc/apt/sources.list.d/debian.sources'`
```
Types: deb
URIs: http://deb.debian.org/debian/
Suites: trixie trixie-updates
Components: main contrib non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb
URIs: http://security.debian.org/debian-security/
Suites: trixie-security
Components: main contrib non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
```

`$ cat '/etc/apt/sources.list.d/pdm-enterprise.sources'`
```
Types: deb
URIs: https://enterprise.proxmox.com/debian/pdm
Suites: trixie
Components: pdm-enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
Enabled: false
```


`$ cat '/proc/pressure/io'`
```
some avg10=0.08 avg60=0.03 avg300=0.07 total=964343
full avg10=0.00 avg60=0.00 avg300=0.01 total=499661
```

`$ cat '/proc/pressure/cpu'`
```
some avg10=1.85 avg60=1.83 avg300=1.41 total=8524442
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
```

`$ cat '/proc/pressure/memory'`
```
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
```


### User & Access

`$ cat '/etc/proxmox-datacenter-manager/access/user.cfg'`
```
user: root@pam
    email REDACTED

user: sergio@pam
    email REDACTED
    enable true
```

`$ cat '/etc/proxmox-datacenter-manager/access/acl.cfg'`
```
acl:1:/:sergio@pam:Administrator
```

### Others

`$ cat '/etc/proxmox-datacenter-manager/node.cfg'`
```
# file does not exist
```

`$ cat '/etc/proxmox-datacenter-manager/views.cfg'`
```
view: VMs
    include-all true
    layout {"layout":{"layout-type":"rows","rows":[[{"flex":1.0,"widget-type":"nodes","remote-type":"pve"},{"flex":1.0,"widget-type":"nodes","remote-type":"pbs"},{"flex":1.0,"widget-type":"guests"}],[{"flex":1.0,"widget-type":"guests","guest-type":"qemu"},{"flex":1.0,"widget-type":"guests","guest-type":"lxc"},{"flex":1.0,"widget-type":"pbs-datastores"}],[{"widget-type":"leaderboard","leaderboard-type":"guest-cpu"},{"widget-type":"leaderboard","leaderboard-type":"node-cpu"},{"widget-type":"leaderboard","leaderboard-type":"node-memory"}],[{"widget-type":"task-summary","grouping":"category"}]]}}
```

Code: 
proxmox-datacenter-manager-meta: 1.1.0 (running kernel: 7.0.6-2-pve)
proxmox-datacenter-manager: 1.1.4 (running version: 1.1.4)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.6-2
proxmox-kernel-7.0.6-2-pve-signed: 7.0.6-2
proxmox-kernel-7.0.2-2-pve-signed: 7.0.2-2
proxmox-kernel-6.17: 6.17.13-13
proxmox-kernel-6.17.13-13-pve-signed: 6.17.13-13
proxmox-kernel-6.17.2-2-pve-signed: 6.17.2-2
ifupdown2: 3.3.0-1+pmx12
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.4
pve-xtermjs: 6.0.0-1
zfsutils-linux: 2.4.2-pve1

What can I do to troubleshoot this problem?
 
Hi, thanks for the detailed report!

It appears that one of the older, zstd-compressed task archive files is corrupted.

To rectify the situation, it should suffice to simply clear the task archive:

Code: 
systemctl stop proxmox-datacenter-api.service
# Backup the task archive so that we can investigate the potential corruption
mv /var/cache/proxmox-datacenter-manager/remote-tasks /root/remote-tasks
systemctl start proxmox-datacenter-api.service

The task archive is merely a cache of the tasks stored on the remotes, so (re)moving the archive directory will make PDM simply fetch the tasks again from the remotes, which should repair the archive.

For further troubleshooting, it would be interesting which one of the files got corrupted. To check, please report the output of
Code: 
cd /root/remote-tasks
zstd -t *.zst

The location of /root/remote-tasks was chosen arbitrarily, of course you may choose another location for the backup copy of the task archive. You can remove the copy once we are done with troubleshooting this issue.

Also, since this is potentially a disk corruption issue and since at this point we are not sure, if this is a bug in PDM or an issue on your end, please verify that your disks are okay. Have you ever observed any other signs of disk corruption?

Thanks!
 
  • Like
Reactions: BySempron
Hi Lukas! Thanks for you effort!

I followed your recommended steps and the CPU use is now normal, the issue appears to have been resolved.

Here is the output of the .zst files:

1781780748191.png

At the same time, I am investigating a long time task (+2 days verify 1TB backup) in the Backup Server (sol-pbs) that I think are caused due to a failed disk, but I can't verify it until I can get onsite due to rare RAID hardware in the server.

Thanks for solving the issue!!
 
You must log in or register to reply here.
Top Bottom
Back