[SOLVED] VMs only seeing 900 MB of RAM no matter how much is allocated

[ardentmusician]

I have run into a really strange issue. I have had 4 VMs running for a few months with no issues. A few days ago, I spun up 3 VMs: 2 non-desktop Debian VMs and 1 Mint Cinnamon VM. The other 4 VMs are all Debian, no desktop. The Mint Cinnamon VM is behaving fine, but the two new VMs only show about 900 MB of RAM when I run the free command, and they are behaving like they only have 900 MB of RAM (installing Nextcloud on one of them took literally HOURS and locked up the VM so much that I could not SSH into it). After running updates on Proxmox and Debian on all the VMs and restarting pve multiple times, now ALL of the Debian VMs are reporting 900 or so MB of RAM. The server has 120 GB of RAM, and if every VM was maxed out on balloons, it would still only use up 70 GB. Some of the VMs have balloons, some have no balloon, but all are behaving the same way (except Mint Cinnamon). I tried rolling Debian back to 6.1.0-23-amd64 and
6.1.0-21-amd64 with no change.

Does anyone have any ideas what I can look at to get this resolved? I really don't even know where to start.

# pveversion --verbose
proxmox-ve: 8.2.0 (running kernel: 6.8.12-2-pve)
pve-manager: 8.2.5 (running version: 8.2.5/12c0a59769080547)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-2

 

[dherzig]

Hi ardentmusician,

Could you post the list of vms on your host (qm list) and the configuration of the VM in question (qm config VMID)? This might make it easier to track down configuration issues!

 

[spirit]

seem that you have enabled memory hotplug ?

see: https://pve.proxmox.com/wiki/Hotplug_(qemu_disk,nic,cpu,memory)

 

[ardentmusician]

Hi ardentmusician,

Could you post the list of vms on your host (qm list) and the configuration of the VM in question (qm config VMID)? This might make it easier to track down configuration issues!

Sure!

I also spun up a Home Assistant VM last night (using the HA qcow file) and it is using over 5 GB of RAM just fine.

Please note that vms 100, 104, 105, and 106 ran for over a year with no issues whatsoever. I have played with the configs for 104 and 106 to troubleshoot, but 100 and 105 have the same configs that they have had since they were first spun up.

pve:~# qm config 100
agent: 1
boot: order=scsi0;net0
cores: 6
cpu: x86-64-v2-AES
hotplug: disk,network,usb,memory,cpu
memory: 16384
meta: creation-qemu=8.1.2,ctime=1706346339
name: vm100
net0: virtio=52:54:00:30:72:01,bridge=vmbr0
net1: virtio=52:54:00:30:72:11,bridge=vmbr1
net2: virtio=52:54:00:15:a2:00,bridge=vmbr31
numa: 1
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,iothread=1,size=100G
scsi2: /dev/disk/by-id/wwn-------------------,size=13039G
scsi3: /dev/disk/by-id/wwn-------------------,size=13039G
scsi4: /dev/disk/by-id/wwn-------------------,size=9314G
scsi5: /dev/disk/by-id/wwn-------------------,size=9314G
scsi6: /dev/disk/by-id/wwn-------------------,size=13039G
scsi7: /dev/disk/by-id/wwn-------------------,size=13039G
scsihw: virtio-scsi-single
smbios1: uuid=e923f21b-ab61-481e-a1b3-43c515565deb
sockets: 2
startup: order=1,up=30,down=30
usb0: host=03f0:5d17
vcpus: 12
vga: std
vmgenid: 2b5a103a-7035-4b6e-81f5-59dc7ea22ba5

pve:~# qm config 102
agent: enabled=1
balloon: 8192
boot: order=scsi0
cores: 6
cpu: x86-64-v2-AES
hotplug: disk,network,usb,memory,cpu
memory: 16384
meta: creation-qemu=9.0.2,ctime=1725596236
name: vm102
net0: virtio=52:54:00:30:72:12,bridge=vmbr1,firewall=1
net1: virtio=52:54:00:15:a2:02,bridge=vmbr31,firewall=1
numa: 1
onboot: 1
ostype: l26
scsi0: local-lvm:vm-102-disk-0,iothread=1,size=200G
scsihw: virtio-scsi-single
smbios1: uuid=39300b9b-7f83-4614-816d-f738e9fb87f0
sockets: 2
startup: order=2,up=30,down=30
vcpus: 12
vga: virtio
vmgenid: da076cec-f4c4-4578-8c5a-4cbac8f04ecf

pve:~# qm config 104
agent: 1
boot: order=virtio0
cores: 2
cpu: x86-64-v2-AES
hotplug: disk,network,usb,memory,cpu
memory: 3072
meta: creation-qemu=8.1.2,ctime=1706354523
name: vm104
net0: virtio=52:54:00:30:72:04,bridge=vmbr0
net1: virtio=52:54:00:15:a2:04,bridge=vmbr31
numa: 1
onboot: 1
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=70b3eb40-6953-4051-9a20-88e7ae158ac5
sockets: 2
startup: order=2
vcpus: 4
virtio0: local-lvm:vm-104-disk-1,iothread=1,size=32G
vmgenid: 1f7a6a4b-85ad-4265-8ae0-909defbb751b

pve:~# qm config 105
agent: 1
balloon: 8192
boot: order=virtio0
cores: 12
cpu: x86-64-v2-AES
cpuunits: 100
hotplug: disk,network,usb,memory,cpu
memory: 16384
meta: creation-qemu=8.1.2,ctime=1706354277
name: vm105-plex
net0: virtio=52:54:00:30:72:05,bridge=vmbr0
net1: virtio=52:54:00:30:72:15,bridge=vmbr1
net2: virtio=52:54:00:15:a2:05,bridge=vmbr31
numa: 1
onboot: 1
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=dae5b906-73a9-48b3-9be5-30bce68a4649
sockets: 2
startup: order=2
unused0: local-lvm:vm-105-disk-0
vcpus: 24
vga: virtio
virtio0: local-lvm:vm-105-disk-1,iothread=1,size=64G
vmgenid: c1ebadf8-500f-4acc-a61a-d66e4caafea3

pve:~# qm config 106
agent: 1
boot: order=virtio0
cores: 2
cpu: x86-64-v2-AES
hotplug: disk,network,usb,cpu
memory: 3072
meta: creation-qemu=8.1.2,ctime=1706354472
name: vm106
net0: virtio=52:54:00:30:72:16,bridge=vmbr1
net1: virtio=52:54:00:15:a2:06,bridge=vmbr31
numa: 1
onboot: 1
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=c436e303-206e-4cb3-b670-257c137653a9
sockets: 2
startup: order=2
unused0: local-lvm:vm-106-disk-0
vcpus: 4
virtio0: local-lvm:vm-106-disk-1,iothread=1,size=32G
vmgenid: 0da40127-cc1d-4b53-ba18-335613924060

pve:~# qm config 107
agent: enabled=1
balloon: 4096
boot: order=scsi0;ide2;net0
cores: 2
cpu: x86-64-v2-AES
hotplug: disk,network,usb,memory,cpu
ide2: none,media=cdrom
memory: 8192
meta: creation-qemu=9.0.2,ctime=1725588012
name: vm107-nginx
net0: virtio=52:54:00:30:72:17,bridge=vmbr1,firewall=1
net1: virtio=52:54:00:30:72:07,bridge=vmbr0,firewall=1
net2: virtio=52:54:00:15:a2:07,bridge=vmbr31,firewall=1
numa: 1
onboot: 1
ostype: l26
scsi0: local-lvm:vm-107-disk-0,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=f35d283f-dd17-4dca-97db-260184c3698a
sockets: 2
startup: order=1,up=30,down=30
vga: virtio
vmgenid: 0f1e78c5-a31e-4c66-94ee-43cc8a950bd9

pve:~# qm config 109
agent: enabled=1
balloon: 4096
boot: order=scsi0;ide2
cores: 6
cpu: x86-64-v2-AES
hotplug: disk,network,usb,memory,cpu
ide2: none,media=cdrom
memory: 8192
meta: creation-qemu=9.0.2,ctime=1725588909
name: vm109-Mint
net1: virtio=52:54:00:30:72:19,bridge=vmbr1,firewall=1
net2: virtio=52:54:00:15:a2:09,bridge=vmbr31,firewall=1
numa: 1
onboot: 1
ostype: l26
scsi0: local-lvm:vm-109-disk-0,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=632c8910-8fc5-4ef2-b96e-e92fff6ab75b
sockets: 2
startup: order=5,up=30,down=30
vcpus: 6
vga: qxl
vmgenid: f61dacbf-8bb6-44b7-a35f-495a23d8a634

pve:~# qm config 110
agent: 1
balloon: 4096
bios: ovmf
boot: order=scsi0
cores: 2
cpu: x86-64-v2-AES
efidisk0: local-lvm:vm-110-disk-0,efitype=4m,size=4M
ide2: none,media=cdrom
machine: q35
memory: 8192
meta: creation-qemu=9.0.2,ctime=1727318811
name: vm110-HomeAssistant
net0: virtio=52:54:00:30:72:30,bridge=vmbr1,firewall=1
numa: 1
onboot: 1
ostype: l26
scsi0: local-lvm:vm-110-disk-1,discard=on,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=38cb510a-1e5b-4a28-b01c-e38ed3552668
sockets: 2
startup: order=2,up=10
vga: qxl
vmgenid: 0e147394-d42c-4231-a20a-20f984390c1c

 

[ardentmusician]

This is the RAM that vm100 (which uses no balloon) sees (has 16 GB allocated). All the Debian vms look pretty much like this:

# free --mega
total used free shared buff/cache available
Mem: 938 613 84 8 427 325
Swap: 1022 19 1002

This is what the Mint Cinnamon vm sees (has 4GB/8GB allocated):

# free --mega
total used free shared buff/cache available
Mem: 4159 1309 2194 16 1080 2849
Swap: 2147 0 2147

 

[spirit]

you need to disable memory hotplug if you don't have udev rules or kernel option to activate dynamic memory modules.

 

[dherzig]

Hi!

spirit's answer sums it up, however here's the results from a small comparison of config setups on a smallish debian 12 machine (with less resources assigned and without swap):

| config id | vm memory | visible guest memory |              balloon | memory hotplug pve |       guest udev cpu |       guest udev memory | guest kernel parameter | numa enabled | sockets | cores | vcpus | vm comment           |
|-----------+-----------+----------------------+----------------------+--------------------+----------------------+-------------------------+------------------------+--------------+---------+-------+-------+----------------------|
|         1 |      4096 |                 4105 | 4096 (unset/default) | 0                  |                    0 |                       0 |                      0 | 0            |       1 |     2 | unset | works                |
|         2 |      4096 |            1957-4160 |                 2048 | 0                  |                    0 |                       0 |                      0 | 0            |       1 |     2 | unset | works                |
|         3 |      4096 |            1957-4160 |                 2048 | 0                  |                    0 |                       0 |                      0 | t            |       1 |     2 | unset | works                |
|         4 |      4096 |            1957-4160 |                 2048 | 0                  |                    t |                       0 |                      0 | t            |       1 |     2 | unset | works                |
|         5 |      4096 |            2062-4160 |                 2048 | 0                  |                    t |                       t |                      0 | t            |       1 |     2 | unset | works                |
|-----------+-----------+----------------------+----------------------+--------------------+----------------------+-------------------------+------------------------+--------------+---------+-------+-------+----------------------|
|         6 |      4096 |                   -- |                 2048 | t                  |                    0 |                       0 |                      0 | 0            |       1 |     2 | unset | pve prevents startup |
|         7 |      4096 |                   -- |                 2048 | t                  |                    t |                       0 |                      0 | t            |       1 |     2 | unset | oom killer           |
|         8 |      4096 |                   -- |                 2048 | t                  |                    0 |                       0 |                      0 | t            |       1 |     2 | unset | oom killer           |
|-----------+-----------+----------------------+----------------------+--------------------+----------------------+-------------------------+------------------------+--------------+---------+-------+-------+----------------------|
|         9 |      4096 |            2012-4160 |                 2048 | t                  |                    0 |                       0 |                      t | t            |       1 |     2 | unset | works                |
|        10 |      4096 |            2014-4160 |                 2048 | t                  |                    t |                       0 |                      t | t            |       1 |     2 | unset | works                |
|        11 |      4096 |            2024-4160 |                 2048 | t                  |                    t |                       t |                      t | t            |       1 |     2 | unset | works                |
|        12 |      4096 |            2014-4160 |                 2048 | t                  |                    t |                       t |                      0 | t            |       1 |     2 | unset | works                |
|        13 |      4096 |            2014-4160 |                 2048 | t                  |                    0 |                       t |                      0 | t            |       1 |     2 | unset | works                |
|-----------+-----------+----------------------+----------------------+--------------------+----------------------+-------------------------+------------------------+--------------+---------+-------+-------+----------------------|

Without memory hotplug enabled:

Ballooning itself works well in constellations 1-5. It just takes a while until the guest will see it, if you set minimal memory lower than the assigned memory itself.

With memory hotplug enabled:

Constellation 6 will not work as numa is required for hotplugging.

7-8 do not work at all on this smaller machine, as oom-killer jumps in on boot on this machine -- using a live iso with swap enabled, the guest came up though (but less than satisfying).

9-13 all work. However, regarding administrative intervention, probably only 9 and 10 are desirable (10 if cpu hotplugging is wanted). I did not have to load the kernel modules mentioned in the wiki [0], they seem to be enabled on debian bookworm's kernel 6.1.0-25-amd64 by default.


[0] reusing spirit's link: https://pve.proxmox.com/wiki/Hotplug_(qemu_disk,nic,cpu,memory)

 

[ardentmusician]

you need to disable memory hotplug if you don't have udev rules or kernel option to activate dynamic memory modules.

You are absolutely correct. I disabled hotplug and restarted the VMs and they work great now. What is the purpose of hotplug? Is it something I should enable, or is it more for mission critical/no downtime situations?

 

[dherzig]

For what I understand it's more something that one could enable if the guest OS can properly handle it, rather than should. PVE is leveraging QEMU's hotplugging features there and it's more of a best-effort approach than the proposed one.