VMs take more RAM than they should, breaks Ressource calculation

[rh-xZa2W]

Hey Guys,

we have Guests VMS which are set with 7GB/14GB of RAM. When I look in htop, the VMs take more Virtual RAM:



This breaks our Ressource calculation and forces the Server to use the SWAP File. Sometimes VMs are getting randomly turned off by the Server..

What does it exactly mean? Can we someone turn this "VIRT" overhead off?

 

[Stefan_R]

The memory reported as 'VIRT' is so called virtual memory, that does not take up actual space on the RAM modules. This is almost certainly not the reason for your VMs getting turned off. How much RAM does your machine have? Could you post a full screenshot of htop maybe, sorted by 'RES' (resident memory, the actual memory used - although not quite, since there's also 'SHR', shared memory - this explains it fairly well)? Also post the output of 'free -h', and potentially 'dmesg' and 'journalctl' logs from the point where your VMs crash.

 

[rh-xZa2W]

How much RAM does your machine have?

7168 MB and 14336 MB

Could you post a full screenshot of htop maybe

Also post the output of 'free -h'

root@server_name:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           62Gi        59Gi       495Mi       195Mi       2.8Gi       2.4Gi
Swap:         2.0Gi       2.0Gi       0.0Ki

potentially 'dmesg' and 'journalctl' logs from the point where your VMs crash.

Jun 16 13:02:01 server_name systemd[1]: Started Proxmox VE replication runner.
Jun 16 13:03:00 server_name systemd[1]: Starting Proxmox VE replication runner...
Jun 16 13:03:01 server_name systemd[1]: pvesr.service: Succeeded.
Jun 16 13:03:01 server_name systemd[1]: Started Proxmox VE replication runner.
Jun 16 13:04:00 server_name systemd[1]: Starting Proxmox VE replication runner...
Jun 16 13:04:01 server_name systemd[1]: pvesr.service: Succeeded.
Jun 16 13:04:01 server_name systemd[1]: Started Proxmox VE replication runner.
Jun 16 13:04:51 server_name pvedaemon[4712]: <root@pam> successful auth for user 'root@pam'
Jun 16 13:05:00 server_name systemd[1]: Starting Proxmox VE replication runner...
Jun 16 13:05:01 server_name systemd[1]: pvesr.service: Succeeded.
Jun 16 13:05:01 server_name systemd[1]: Started Proxmox VE replication runner.
Jun 16 13:05:09 server_name pvedaemon[5271]: start VM 900: UPID:server_name:00001497:3919B159:5EE8A765:qmstart:900:root@pam:
Jun 16 13:05:09 server_name pvedaemon[4418]: <root@pam> starting task UPID:server_name:00001497:3919B159:5EE8A765:qmstart:900:root@pam:
Jun 16 13:05:09 server_name systemd[1]: Started 900.scope.
Jun 16 13:05:09 server_name systemd-udevd[5281]: Using default interface naming scheme 'v240'.
Jun 16 13:05:09 server_name systemd-udevd[5281]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 16 13:05:09 server_name systemd-udevd[5281]: Could not generate persistent MAC address for tap900i0: No such file or directory
Jun 16 13:05:10 server_name kernel: [9579793.076063] device tap900i0 entered promiscuous mode
Jun 16 13:05:10 server_name kernel: [9579793.081287] vmbr950: port 1(tap900i0) entered blocking state
Jun 16 13:05:10 server_name kernel: [9579793.081534] vmbr950: port 1(tap900i0) entered disabled state
Jun 16 13:05:10 server_name kernel: [9579793.081826] vmbr950: port 1(tap900i0) entered blocking state
Jun 16 13:05:10 server_name kernel: [9579793.082064] vmbr950: port 1(tap900i0) entered forwarding state
Jun 16 13:05:10 server_name pvedaemon[4418]: <root@pam> end task UPID:server_name:00001497:3919B159:5EE8A765:qmstart:900:root@pam: OK
Jun 16 13:05:13 server_name pvedaemon[4418]: VM 900 qmp command failed - VM 900 qmp command 'guest-ping' failed - got timeout
Jun 16 13:09:12 server_name kernel: [ 0.000000] Linux version 5.3.18-2-pve (build@pve) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.3.18-2 (Sat, 15 Feb 2020 15:11:52 +0100) ()
Jun 16 13:09:12 server_name kernel: [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.3.18-2-pve root=UUID=3b44b2e9-90e4-441b-832b-fb915601832a ro modprobe.blacklist=btrfs

Here the following happened: The User couldn't connect to his VM. It was turned off. After I turned it on, the Server restarted.

Thanks for helping out here!
Greetings

 

[Stefan_R]

root@server_name:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           62Gi        59Gi       495Mi       195Mi       2.8Gi       2.4Gi
Swap:         2.0Gi       2.0Gi       0.0Ki

Your server is awefully low on memory, but it from the htop output it appears that your VMs are not the issue. Is there anything else running on your server? What's your storage configuration like, ZFS, Ceph in use?

I also find it strange that the server decided to just reboot without putting anything in the logs about why (e.g. out of memory situation). That would indicate something else at fault here, possibly even hardware.

 

[rh-xZa2W]

@Stefan_R We are using Directory Type as Storage.

Today we had the same Problem:

I saw that a VM (401) was offline, no Shutdown or Stop Command was running, nothing is shown in the GUI...

I had to start the VM (401) manually at 11:35:08. According to the Proxmox GUI the VM was offline since 10:59.

I will attach the Logfile for this Timeframe.

Thanks

 

[Stefan_R]

That seems like a shutdown from within the VM. Any logs in the guest? Maybe it crashed and powered itself off?

 

[creamers]

Your server is awefully low on memory, but it from the htop output it appears that your VMs are not the issue. Is there anything else running on your server? What's your storage configuration like, ZFS, Ceph in use?

I also find it strange that the server decided to just reboot without putting anything in the logs about why (e.g. out of memory situation). That would indicate something else at fault here, possibly even hardware.

When you over allocate for example 1 vm alot then the proxmox host will reserve this memory for that host.
My host machine 'lost' alot of memory while the vm's weren't actually using it.

proxmoxhost# free -h
              total        used        free      shared  buff/cache   available
Mem:           62Gi        60Gi       564Mi        43Mi       1.2Gi       1.1Gi
Swap:          31Gi       9.9Gi        22Gi

Ballooning solved it somewhat, but setting a vm to 8GB as minimum and 45B max but the vm still got 28GB. ?!
Same for another vm that is using alot of buff/cache and doesnt release back. Proxmox shows this as used ram memory but it actually isnt.

Could it be that you overallocated vm's too much?