ksm is needlessly burning cpu because of using vzs and ignoring arcsize

RolandK

RolandK

Renowned Member
Proxmox Subscriber
Mar 5, 2019
924
173
88
50
hello,

see subject.

there are these two bugreports / RFEs which did not get any attention for a long time now

1. RFE: ZFS ARC size not taken into account by pvestatd or ksmtuned
https://bugzilla.proxmox.com/show_bug.cgi?id=3859

2. ksmtuned using vsz instead of rss
https://bugzilla.proxmox.com/show_bug.cgi?id=2635

this both leads to wrong calculation of occupied/free ram ressources and may also lead to constantly running ksm task.

on my system , ksm is constantly hogging cpu, though i have 28gb of ram free (9gb free, 19gb in ZFS ARC)

even if i set KSM_THRES_COEF=1 , ksm is constantly running and burning cpu - for nothing

# tail -f /var/log/syslog|grep ksmtu
Feb 27 17:32:09 pve-bigiron ksmtuned[2299228]: Tue 27 Feb 2024 05:32:09 PM CET: total 131788880
Feb 27 17:32:09 pve-bigiron ksmtuned[2299228]: Tue 27 Feb 2024 05:32:09 PM CET: sleep 12
Feb 27 17:32:09 pve-bigiron ksmtuned[2299228]: Tue 27 Feb 2024 05:32:09 PM CET: thres 1317888
Feb 27 17:32:09 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:09 PM CET: committed 142397948 free 9309156
Feb 27 17:32:09 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:09 PM CET: 143715836 > 131788880, start ksm
Feb 27 17:32:09 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:09 PM CET: 9309156 > 1317888, decay
Feb 27 17:32:09 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:09 PM CET: KSMCTL start 64 12
Feb 27 17:32:19 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:19 PM CET: committed 142553672 free 9297556
Feb 27 17:32:19 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:19 PM CET: 143871560 > 131788880, start ksm
Feb 27 17:32:19 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:19 PM CET: 9297556 > 1317888, decay
Feb 27 17:32:19 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:19 PM CET: KSMCTL start 64 12
Feb 27 17:32:29 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:29 PM CET: committed 142471712 free 9290528
Feb 27 17:32:29 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:29 PM CET: 143789600 > 131788880, start ksm
Feb 27 17:32:29 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:29 PM CET: 9290528 > 1317888, decay
Feb 27 17:32:29 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:29 PM CET: KSMCTL start 64 12
Feb 27 17:32:39 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:39 PM CET: committed 142471712 free 9280792
Feb 27 17:32:39 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:39 PM CET: 143789600 > 131788880, start ksm
Feb 27 17:32:39 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:39 PM CET: 9280792 > 1317888, decay
Feb 27 17:32:39 pve-bigiron ksmtuned[2299234]: Tue 27 Feb 2024 05:32:39 PM CET: KSMCTL start 64 12

i know how i can get rid of this - but what about all the other installations out there who are needlessly wasting cpu and burning energy for nothing ?
 
Last edited:
Code: 
systemctl stop ksmtuned.service
systemctl mask ksmtuned.service
´

will stop and mask it, so it can't be started in the future.
 
narrateourale said:
Code: 
systemctl stop ksmtuned.service
systemctl mask ksmtuned.service
´

will stop and mask it, so it can't be started in the future.
Click to expand...

thanks, but this post is not about getting rid of ksm but about making it more cpu friendly and sorting out issues in conjunction with ZFS, i.e. make it behave better by default.


this is taken from my servers. on more then half of them, ksmd is the top cpu consumer - though ARC size is typically >80GB (from out of 256GB RAM)

there is no need at all for ksmd to try dedup memory

Code: 
top -o TIME

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2323025 root      20   0   21.9g  11.4g  12680 S  21.2   4.5 164968:57 kvm
    266 root      25   5       0      0      0 S  18.9   0.0 121894:51 ksmd
3425228 root      20   0   24.1g  11.4g  14548 S   8.9   4.5  42824:37 kvm
 147827 root      20   0   17.7g  11.4g  11792 S   1.0   4.5  41966:29 kvm
  15127 root      20   0   41.8g  32.8g  14476 S  38.7  13.0  40952:18 kvm

 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    266 root      25   5       0      0      0 S  59.3   0.0 266926:08 ksmd
3788450 root      20   0   18.1g   9.0g  12056 S  15.6   3.6  98673:37 kvm
1504770 root      20   0   42.4g  33.0g  11712 S  21.5  13.1  98487:13 kvm
2302459 root      20   0   21.7g  16.6g  10664 S  17.9   6.6  83551:35 kvm
1646974 root      20   0   22.4g   9.0g  11076 S   3.3   3.6  51007:39 kvm

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    218 root      25   5       0      0      0 S   0.0   0.0 284649:33 ksmd
1801746 root      20   0   42.4g  34.3g  12356 S  13.3  13.6 115118:51 kvm
   5184 root      20   0   29.9g  17.0g  11752 S   1.7   6.8  54265:50 kvm
3532236 root      20   0 6691456   1.9g  10872 S   6.6   0.8  26265:34 kvm
   3128 root      rt   0  567564 174216  53336 S   1.7   0.1  10637:10 corosync
  
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    218 root      25   5       0      0      0 R  50.0   0.0 220553:38 ksmd
 386566 root      20   0   34.2g  25.1g  10152 S  29.5  10.0 167056:42 kvm
4134900 root      20   0   38.5g  32.5g  10552 S  44.4  12.9 113141:55 kvm
1641035 root      20   0   10.2g   2.5g  10248 S   9.6   1.0  77421:22 kvm
1006261 root      20   0 3949448   1.4g  10500 S  10.3   0.6  56958:46 kvm

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    218 root      25   5       0      0      0 S  49.3   0.0 204048:07 ksmd
1675109 root      20   0   17.2g   8.1g   9860 S 133.1   3.2  40494:11 kvm
1256140 root      20   0   15.8g   2.9g  12756 S   1.0   1.2  35107:14 kvm
2824661 root      20   0   14.5g   8.0g  12668 S   1.7   3.2  27715:34 kvm
4096793 root      20   0   21.6g  13.5g  12464 S   4.3   5.4  23502:42 kvm

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     68 root      25   5       0      0      0 R  23.6   0.0  14660:12 ksmd
 514059 root      20   0   13.4g   8.1g  12144 S  11.3  12.9  12316:39 kvm
  86432 root      20   0 7050100   2.2g  11744 S   4.0   3.6   3083:17 kvm
   2779 root      20   0 5903432   1.1g  11768 S   3.0   1.8   2605:56 kvm
 596846 root      20   0   12.8g   9.0g  12220 S   1.3  14.3   1038:52 kvm
 
Last edited:
Are you sure it is not deduplicating ARC? I always had the impression that KSM will deduplicate identical cached data that got cached twice. Once in virtual RAM inside the VM as well as cached again in the hosts RAM by ARC. Because then it would still be useful if KSM woudn't ignore the ARC when accounting for free RAM.
 
Last edited:
>There are some knobs that can limit ksmd, like multiplying KSM_SLEEP_MSEC (put a 0 behind it) and lowering KSM_NPAGES_MAX in /etc/ksmtuned.conf

@leesteken , yes i know, but this post is not about individually tuning KSM , it's about making it work correctly by default

>Are you sure it is not deduplicating ARC? I always had the impression that KSM will deduplicate identical cached data that got cached twice

@Dunuin , yes i'm sure. KSM does not deduplicate ARC.

https://github.com/openzfs/zfs/issues/14279
 
Last edited:
  • Like
Reactions: Dunuin
RolandK said:
>There are some knobs that can limit ksmd, like multiplying KSM_SLEEP_MSEC (put a 0 behind it) and lowering KSM_NPAGES_MAX in /etc/ksmtuned.conf

@leesteken , yes i know, but this post is not about individually tuning KSM , it's about making it work correctly by default
Click to expand...
I fear that the default settings come from the time when 16GB was a decent amount of memory for a server and that Debian needs to change those.
I just wanted to let you know of a way to fix your issue now, using a different knob that you did not yet try (to my knowledge at the time).

EDIT: Maybe you can get Proxmox to fork the package and provide better settings? Or do it yourself and ask Proxmox to adopt it. Okay, I'll shut up now.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back