ProxMox uses 50% ram without ZFS and won't...

[segobi]

Hi there,

Im running Proxmox on an Intel 14600k System with 64GB Host Memory.
Im running a Windows VM with gpu (nvidia 3800 gtx passthrough).

Im not using ZFS.

When I reboot Proxmox the memory usage is about 50% of the host memory (ca. 32gb). Even when I do not start any VM.

A few weeks ago I was able to run the windows VM with 32 gig of ram.... When I want to start it now I get an out of memory error in dmesg.


I see the host has only 30gb of free memory and when i set the memory of the vm to 29 gb i can start it.

I can not identify where those 30gig of mem go and why qemu can not asign the available mem to the vm.

Maybe someone has an idea ?

 

[Ahmet Bas]

Hi there,

Im running Proxmox on an Intel 14600k System with 64GB Host Memory.
Im running a Windows VM with gpu (nvidia 3800 gtx passthrough).

Im not using ZFS.

When I reboot Proxmox the memory usage is about 50% of the host memory (ca. 32gb). Even when I do not start any VM.

A few weeks ago I was able to run the windows VM with 32 gig of ram.... When I want to start it now I get an out of memory error in dmesg.


I see the host has only 30gb of free memory and when i set the memory of the vm to 29 gb i can start it.

I can not identify where those 30gig of mem go and why qemu can not asign the available mem to the vm.

Maybe someone has an idea ?

Which Proxmox version and kernel are you using at the moment?

 

[segobi]

pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.8.4-2-pve)

 

[segobi]

i went back to 6.8.4-2 because of the problem but it did not help...

 

[LnxBil]

Can you please share more information about this? e.g. the ouput of free and htop to start.

 

[segobi]

free
total used free shared buff/cache available
Mem: 65679100 65457556 348808 24348 389168 221544
Swap: 8388604 2390016 5998588


htop says 61.9g used of 64gig - although im only running the vm with 29 gig.
I can not start it when i set the memory of the vm to 32gig.


hope this helps - otherwise please tell me how to get a better output from htop. thanks!

 

[segobi]

config of vm:
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi1
cores: 10
cpu: x86-64-v2-AES
efidisk0: speicher_sdd:100/vm-100-disk-3.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:01:00,pcie=1
hostpci1: 0000:02:00.0,pcie=1
machine: pc-q35-8.1
memory: 29000
meta: creation-qemu=8.1.5,ctime=1714014363
name: win11vm
net0: virtio=BC:24:11:C3:46:61,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi1: /dev/disk/by-id/ata-CT250MX500SSD1_1849E1DB1E42,size=244198584K,ssd=1
scsi2: /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNF0W704882D,size=7814026584K,ssd=1
scsi3: /dev/disk/by-id/ata-Samsung_SSD_870_QVO_8TB_S5SSNF0W522372A,size=7814026584K,ssd=1
scsi4: /dev/disk/by-id/ata-Samsung_SSD_870_QVO_4TB_S5STNF0R810080B,size=3907018584K,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=a3ea63d6-f2da-4bf8-929f-953bb0e0ec25
sockets: 1
tpmstate0: speicher_sdd:100/vm-100-disk-2.raw,size=16896,version=v2.0
vga: none
vmgenid: 04d9b936-e06b-403c-92e6-b68a508b070b

 

[segobi]

root@z:~# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.4-2-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.4-3
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.2
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.3-1
proxmox-backup-file-restore: 3.2.3-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.6
pve-container: 5.1.10
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.7
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2

 

[segobi]

dmesg when i start the VM with more than 29 gig:

[41698.099549] Out of memory: Killed process 112458 (kvm) total-vm:34603460kB, anon-rss:31432444kB, file-rss:1920kB, shmem-rss:0kB, UID:0 pgtables:61964kB oom_score_adj:0

 

[segobi]

Its running on a Supermicro Motherboard X13SAE-F

 

[justinclift]

With the htop output, would you be ok to screenshot the whole thing?

The formatting here of plain text is making it a real pain in the butt to figure things out, and the htop info you've included didn't have any of the process info.

The process info would actually be super useful here, as if you tell htop to sort by "resident memory" size it'll show you what things are taking up all of the ram. And that's probably the most important thing to know first.

---

Just to be super clear, the htop command needs to be run on the host, not in a VM.

 

[segobi]

This is a htop screenshot without any VM running. As you can see I have less than 32gig free and I can not start my Windows VM when I set it to use 32gigs.

 

[LnxBil]

That looks strange ... please also sort with respect to VIRT and post a screenshot with more information visibile (more columns and more lines). Also post the output of ps auxf in CODE tags or use pastebin for the output.

 

[segobi]

https://pastebin.com/GyDZD9uZ



ps auxf output

 

[segobi]

 

[justinclift]

Oh. In that view, when you see several lines that are mostly identical (ie the pxmcfs, rrdcached, and probably pve-lxc-syscalld ones there), try pressing the 'H' key (capital H) once.

That's a toggle which will remove the duplicate lines that can show up from individual programs (note, it's a simplified explanation), which can help make things a bit more visible. Pressing H multiple times will just toggle that setting on and off. Try it out, and you'll see what I mean. It won't break anything.

 

[justinclift]

https://pastebin.com/GyDZD9uZ

Is that with the VMs running, or without the VMs running?

Asking because the contents of that show what looks like two VMs running (100, 101). Not sure if you're expecting them to be though.

 

[segobi]

Its without running a VM. If id be running my Windows VM it would show about 60gig used. That is the screenshot without the duplicate lines.

 

[justinclift]

Its without running a VM

Ahhh. That goes a long way towards explaining part of what's wrong. You actually do (somehow!) have two virtual machines actively running.

One is called VM 100, and the other is VM 101.

Try doing a ps -ef f | grep -i kvm and see if anything is returned. If it is, please paste it here.

 

[segobi]

I see I have an entry of hugepages=32 in the linux grub default line... I changed that to hugepages=8 and the memory is not taken anymore after a reboot.