KSM Memory sharing not working as expected on 6.2.x kernel

aaron said:
Okay, can you compare the versions, especially of the qemu-server package? If the version differs there, this could also be the cause :)
Click to expand...
I actually have updated all my front ends, just haven't rebooted some of them for the changes to take effect.

Is there a easy way to figure out what version they are running right now or was previously installed?
 
Probably by sifting through the apt logs. /var/log/apt/term.log. Look for "Unpacking * over *" lines.
 
aaron said:
Probably by sifting through the apt logs. /var/log/apt/term.log. Look for "Unpacking * over *" lines.
Click to expand...

Here it is.

Unpacking pve-qemu-kvm (7.2.0-8) over (7.1.0-4)

I did not have a chance to reboot the front end back to 5.13.x this week, hoping to get to it early next week.
 
Okay, then I can test next week if the pve-qemu-kvm version might have an impact. :)
 
So, I tested it again the past days. The following kernels and pve-qemu-kvm variants were tested.

kernels:
  • 5.13.19-6-pve (edit: fixed kernel used)
  • 6.2.16-4-bpo11-pve
pve-qemu-kvm versions:
  • 7.0.0-2
  • 7.1.0-4
  • 7.2.0-8
Unfortunately, they all behave very similar. With the 141 test machines (clones of the same template), I end up in the range of ~70-80% RAM usage and KSM is somewhere around 45 GiB, sometimes a bit more or less. On a machine with 64 GiB of RAM total.

Any new insights on your side?
 
Last edited:
If I may, I'd like to join this debate, since I'm also having same or similar KSM symptoms.

I'm not sure what info you'd like me to provide, so I went ahead and added all of the things you already asked from adamb.

Server is: HP Proliant SE326M1.
pve.jpg

It's hosting the following lot:
- 17x Windows Server 2008 R2 (2 GB per VM assigned, no ballooning)
- 2x Windows Server 2012 (4 GB per VM, no ballooning)
- 1x Ubuntu (4 GB assigned, no ballooning)

All VMs are running.

Uptime on the pic above shows when it was upgraded to 6.2.

Before upgrade, it was running latest 7.x PVE and KSM was sharing aprox. 25 GB.
After upgrade to 6.2, KSM only went up to aprox. 3 GB, but it really "prefers" to be at around 2.5 GB most of the time.

I took a note from adamb about tweaking ksmtuned.conf and made the changes as follows:
KSM_NPAGES_MIN=64000000
KSM_NPAGES_MAX=1250000000

With this, KSM gets to 10.15 GB (as pictured above). Adding more zeroes at the end of KSM_NPAGES_x yields diminishing returns.
Indeed, last 2 zeroes I added, only increased KSM by less than 1.5 GB

Below is other info that someone might find useful.

Code: 
pveversion -v

Code: 
proxmox-ve: 8.0.2 (running kernel: 6.2.16-6-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
proxmox-kernel-helper: 8.0.3
pve-kernel-5.15: 7.4-4
pve-kernel-5.13: 7.1-9
proxmox-kernel-6.2.16-6-pve: 6.2.16-7
proxmox-kernel-6.2: 6.2.16-7
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.15.104-1-pve: 5.15.104-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.4
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.7
libpve-guest-common-perl: 5.0.4
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.3
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-4
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1


Code: 
for i in /sys/kernel/mm/ksm/*; do echo "$i:"; cat $i; done

Code: 
/sys/kernel/mm/ksm/full_scans:
94490
/sys/kernel/mm/ksm/max_page_sharing:
256
/sys/kernel/mm/ksm/merge_across_nodes:
1
/sys/kernel/mm/ksm/pages_shared:
269340
/sys/kernel/mm/ksm/pages_sharing:
2665058
/sys/kernel/mm/ksm/pages_to_scan:
64753900
/sys/kernel/mm/ksm/pages_unshared:
220581
/sys/kernel/mm/ksm/pages_volatile:
194130
/sys/kernel/mm/ksm/run:
1
/sys/kernel/mm/ksm/sleep_millisecs:
33
/sys/kernel/mm/ksm/stable_node_chains:
19
/sys/kernel/mm/ksm/stable_node_chains_prune_millisecs:
2000
/sys/kernel/mm/ksm/stable_node_dups:
7679
/sys/kernel/mm/ksm/use_zero_pages:
0
 
@Shomo do you see a much better KSM utilization once you reboot to an older kernel?
 
Yes.

pve_new.jpg

I migrated VMs to a different node (online migration), reset ksmtuned.conf on this node to default and rebooted this node to kernel 5.15.
Then I migrated same VMs back on to it.

Result is expected and is consistent with what I observed before upgrade to 6.2.
 
Hi,

I think I'm hitting same bug.

I'm currently doing tests with 6.2 kernel, migrate alls vm from a 5.15 kernel to a new 6.2 nodes, (using aroun 95% memory on 256GB) .

and after 1h, ksm shared is still around 2-3GB. (Generally I'm around 60GB for this kind of server).

I'm 100% sure that ksm shared memory is increase a lot faster , maybe 5-10min max to reach 30G


New server stats with kernel 6.2, after the vm migration

Code: 
 for i in /sys/kernel/mm/ksm/*; do echo "$i:"; cat $i; done
/sys/kernel/mm/ksm/full_scans:
3
/sys/kernel/mm/ksm/max_page_sharing:
256
/sys/kernel/mm/ksm/merge_across_nodes:
0
/sys/kernel/mm/ksm/pages_shared:
111400
/sys/kernel/mm/ksm/pages_sharing:
682666
/sys/kernel/mm/ksm/pages_to_scan:
900
/sys/kernel/mm/ksm/pages_unshared:
1198053
/sys/kernel/mm/ksm/pages_volatile:
29416836
/sys/kernel/mm/ksm/run:
1
/sys/kernel/mm/ksm/sleep_millisecs:
10
/sys/kernel/mm/ksm/stable_node_chains:
28
/sys/kernel/mm/ksm/stable_node_chains_prune_millisecs:
2000
/sys/kernel/mm/ksm/stable_node_dups:
1039
/sys/kernel/mm/ksm/use_zero_pages:
0

previous server with 5.15 kernel

Code: 
root@kvm11:~#  for i in /sys/kernel/mm/ksm/*; do echo "$i:"; cat $i; done
/sys/kernel/mm/ksm/full_scans:
2025
/sys/kernel/mm/ksm/max_page_sharing:
256
/sys/kernel/mm/ksm/merge_across_nodes:
0
/sys/kernel/mm/ksm/pages_shared:
3473786
/sys/kernel/mm/ksm/pages_sharing:
18222244
/sys/kernel/mm/ksm/pages_to_scan:
64
/sys/kernel/mm/ksm/pages_unshared:
36419825
/sys/kernel/mm/ksm/pages_volatile:
12398786
/sys/kernel/mm/ksm/run:
0
/sys/kernel/mm/ksm/sleep_millisecs:
10
/sys/kernel/mm/ksm/stable_node_chains:
16170
/sys/kernel/mm/ksm/stable_node_chains_prune_millisecs:
2000
/sys/kernel/mm/ksm/stable_node_dups:
71044
/sys/kernel/mm/ksm/use_zero_pages:
0
 
@Shomo The machines are also dual socket ones. I unfortunately don't have a dual socket machine at hand to test right now. Do you have single socket machines that you can use to test this on?

@spirit is this also on a dual socket machine?
 
I am trying to recreate that in a virtualized PVE to see if the sockets are what I need to trigger it myself.
 
Hi,
spirit said:
yes, dual socket intel with 2 numa nodes

(not sure if it could be related, but I just found this commit https://git.kernel.org/pub/scm/linu...5&id=d74943a2f3cdade34e471b36f55f7979be656867)
Click to expand...
the commit it Fixes is present since kernel 6.1, so might be a candidate.

@Shomo @spirit One of the linked mails in the commit says that it affects huge pages. The commit message doesn't sound like it's limited to that though, but for completeness sake: are you using huge pages?
 
fiona said:
Hi,

the commit it Fixes is present since kernel 6.1, so might be a candidate.

@Shomo @spirit One of the linked mails in the commit says that it affects huge pages. The commit message doesn't sound like it's limited to that though, but for completeness sake: are you using huge pages?
Click to expand...

I'm not sure what huges pages even are, but if it's off by default, then I'm not using it.
 
I'm not using static hugepages in the vms, but transparent hugepages are enabled.

cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never


(BTW, I'm only using vms, no ct, and all vms have 2 sockets + numa option)


I'm currently redoing test on 5.15 kernel, remigrating back all the original vms, the ksm shared is already at 20GB after some minutes. (instead 2-3GB stuck on 6.2 kernel)
 
spirit said:
I'm not using static hugepages in the vms, but transparent hugepages are enabled.

cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never


(BTW, I'm only using vms, no ct, and all vms have 2 sockets + numa option)


I'm currently redoing test on 5.15 kernel, remigrating back all the original vms, the ksm shared is already at 20GB after some minutes. (instead 2-3GB stuck on 6.2 kernel)
Click to expand...
Same result on that cat command, also only using VMs. Mine are configured as 1 socket, 4 cores, with NUMA option disabled.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom