KVM: Cannot allocate memory

M

MichaelA

New Member
Jan 4, 2012
13
1
1
Samara, Russia
Hello.

I have KVM virtual machines on host with 64Gb RAM. Now total usage of RAM by them achieved almost 30Gb. And I can't start more:

Code: 
# qm start 213
Failed to allocate 4294967296 B: Cannot allocate memory
start failed: command '/usr/bin/kvm -id 213 -chardev 'socket,id=qmp,path=/var/run/qemu-server/213.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/213.vnc,x509,password -pidfile /var/run/qemu-server/213.pid -daemonize -name qa3.forbet.net -smp 'sockets=2,cores=2' -nodefaults -boot 'menu=on' -vga cirrus -k en-us -m 4096 -usbdevice tablet -drive 'if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/213/vm-213-disk-1.raw,if=none,id=drive-virtio0,aio=native,cache=none' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap213i0,script=/var/lib/qemu-server/pve-bridge' -device 'rtl8139,mac=32:60:85:C0:A6:92,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1
But there are much of RAM yet:
Code: 
# free
             total       used       free     shared    buffers     cached
Mem:      65963420   27951904   38011516          0       9536      58548
-/+ buffers/cache:   27883820   38079600
Swap:      3998712          0    3998712

And Kernel is 64x so can allocate it:
Code: 
# uname -a
Linux vds1 2.6.32-17-pve #1 SMP Wed Nov 28 07:15:55 CET 2012 x86_64 GNU/Linux

There are bunch of similar questions on forum but all of them regarding OpenVZ but not like to mine.
Do you have any suggestions?
 
MichaelA said:
Hello.

I have KVM virtual machines on host with 64Gb RAM. Now total usage of RAM by them achieved almost 30Gb. And I can't start more:

Code: 
# qm start 213
Failed to allocate 4294967296 B: Cannot allocate memory
start failed: command '/usr/bin/kvm -id 213 -chardev 'socket,id=qmp,path=/var/run/qemu-server/213.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/213.vnc,x509,password -pidfile /var/run/qemu-server/213.pid -daemonize -name qa3.forbet.net -smp 'sockets=2,cores=2' -nodefaults -boot 'menu=on' -vga cirrus -k en-us -m 4096 -usbdevice tablet -drive 'if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/213/vm-213-disk-1.raw,if=none,id=drive-virtio0,aio=native,cache=none' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap213i0,script=/var/lib/qemu-server/pve-bridge' -device 'rtl8139,mac=32:60:85:C0:A6:92,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1
But there are much of RAM yet:
Code: 
# free
             total       used       free     shared    buffers     cached
Mem:      65963420   27951904   38011516          0       9536      58548
-/+ buffers/cache:   27883820   38079600
Swap:      3998712          0    3998712

And Kernel is 64x so can allocate it:
Code: 
# uname -a
Linux vds1 2.6.32-17-pve #1 SMP Wed Nov 28 07:15:55 CET 2012 x86_64 GNU/Linux

There are bunch of similar questions on forum but all of them regarding OpenVZ but not like to mine.
Do you have any suggestions?
Click to expand...
Hi,
32-17 is quite old - you should update.

What is the output of "pveversion -v"?

Is ksm running?

Udo
 
Upgraded server - same issue.

New data for discussion:

Code: 
# free
             total       used       free     shared    buffers     cached
Mem:      65963352   10178520   55784832          0      22792      93792
-/+ buffers/cache:   10061936   55901416
Swap:      3998712          0    3998712



Code: 
# qm start 212
Failed to allocate 4294967296 B: Cannot allocate memory
start failed: command '/usr/bin/kvm -id 212 -chardev 'socket,id=qmp,path=/var/run/qemu-server/212.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/212.vnc,x509,password -pidfile /var/run/qemu-server/212.pid -daemonize -name qa2.forbet.net -smp 'sockets=2,cores=2' -nodefaults -boot 'menu=on' -vga cirrus -k en-us -m 4096 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/212/vm-212-disk-1.raw,if=none,id=drive-virtio0,format=raw,aio=native,cache=none' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap212i0,script=/var/lib/qemu-server/pve-bridge' -device 'rtl8139,mac=5E:1C:C2:78:B4:F4,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1



Code: 
# uname -a
Linux vds1 2.6.32-20-pve #1 SMP Wed May 15 08:23:27 CEST 2013 x86_64 GNU/Linux


Code: 
# pveversion -v
pve-manager: 3.0-20 (pve-manager/3.0/0428106c)running kernel: 2.6.32-20-pve
pve-kernel-2.6.32-20-pve: 2.6.32-100
lvm2: 2.02.95-pve3
clvm: 2.02.95-pve3
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-1
pve-cluster: 3.0-4
qemu-server: 3.0-15
pve-firmware: 1.0-22
libpve-common-perl: 3.0-4
libpve-access-control: 3.0-4
libpve-storage-perl: 3.0-6
vncterm: 1.1-3
vzctl: 4.0-1pve3
vzprocps: 2.0.11-2
vzquota: 3.1-2

Code: 
# ps auxf | grep ksm | grep -v grep
root       182  2.5  0.0      0     0 ?        SN   21:08   0:47  \_ [ksmd]
root      2577  0.0  0.0   9232   896 ?        S    21:08   0:00 /bin/bash /usr/sbin/ksmtuned
root      2588  0.0  0.0   4052   532 ?        Ss   21:08   0:00 startpar -f -- ksmtuned
(but I don't actually need KSM. I'm interesting why PVE doesn't allocate entire RAM space)
 
Last edited:
MichaelA said:
Upgraded server - same issue.

...


Code: 
# ps auxf | grep ksm | grep -v grep
root       182  2.5  0.0      0     0 ?        SN   21:08   0:47  \_ [ksmd]
root      2577  0.0  0.0   9232   896 ?        S    21:08   0:00 /bin/bash /usr/sbin/ksmtuned
root      2588  0.0  0.0   4052   532 ?        Ss   21:08   0:00 startpar -f -- ksmtuned
(but I don't actually need KSM. I'm interesting why PVE doesn't allocate entire RAM space)
Click to expand...

Hi,
any further error message in /var/log/messages / syslog if you try to start the VM?

Any change after "service ksmtuned stop"?

Udo
 
In messages and kern.log - nothing, syslog and daemon.log just repeats the same what I'm seeing myself in command line (STDERR).

This is strace part before error:

ioctl(8, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff39059e60) = -1 EINVAL (Invalid argument)lseek(8, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fcntl(8, F_SETFD, FD_CLOEXEC) = 0
connect(8, {sa_family=AF_FILE, path="/dev/log"}, 110) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1448, ...}) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1448, ...}) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1448, ...}) = 0
select(16, [8], NULL, [8], {0, 0}) = 0 (Timeout)
write(8, "<30>May 26 10:54:28 qm[19877]: <"..., 116) = 116
stat("/etc/perl/bytes_heavy.pl", 0x7fff39059ef0) = -1 ENOENT (No such file or directory)
stat("/usr/local/lib/perl/5.14.2/bytes_heavy.pl", 0x7fff39059ef0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/perl/5.14.2/bytes_heavy.pl", 0x7fff39059ef0) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/bytes_heavy.pl", 0x7fff39059ef0) = -1 ENOENT (No such file or directory)
stat("/usr/share/perl5/bytes_heavy.pl", 0x7fff39059ef0) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl/5.14/bytes_heavy.pl", 0x7fff39059ef0) = -1 ENOENT (No such file or directory)
stat("/usr/share/perl/5.14/bytes_heavy.pl", {st_mode=S_IFREG|0644, st_size=758, ...}) = 0
open("/usr/share/perl/5.14/bytes_heavy.pl", O_RDONLY) = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff39059c80) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR) = 0
read(10, "package bytes;\n\nsub length (_) {"..., 8192) = 758
read(10, "", 8192) = 0
close(10) = 0
sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"h", 1}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 1
futex(0x7f3279641010, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 0, {1369551270, 147753666}, ffffffff) = 0
rt_sigaction(SIGALRM, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [ALRM], [], 8) = 0
rt_sigaction(SIGALRM, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [ALRM], [], 8) = 0
rt_sigaction(SIGALRM, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [PIPE], [], 8) = 0
rt_sigaction(SIGPIPE, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, {SIG_IGN, [], SA_RESTORER, 0x7f3278a14030}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
alarm(10) = 0
open("/var/log/pve/tasks/.active.lock", O_WRONLY|O_CREAT|O_APPEND, 0666) = 10
lseek(10, 0, SEEK_END) = 0
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff39059c80) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR) = 0
fstat(10, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC) = 0
flock(10, LOCK_EX|LOCK_NB) = 0
alarm(0) = 10
rt_sigprocmask(SIG_BLOCK, [PIPE], [], 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER, 0x7f3278a14030}, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [ALRM], [], 8) = 0
rt_sigaction(SIGALRM, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
alarm(0) = 0
rt_sigprocmask(SIG_BLOCK, [ALRM], [], 8) = 0
rt_sigaction(SIGALRM, {SIG_DFL, [], SA_RESTORER, 0x7f3278a14030}, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
read(4, 0x7fff39058000, 8192) = -1 EAGAIN (Resource temporarily unavailable)
open("/var/log/pve/tasks/active", O_RDONLY) = 11
ioctl(11, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff39059c80) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(11, 0, SEEK_CUR) = 0
fstat(11, {st_mode=S_IFREG|0644, st_size=19535, ...}) = 0
fcntl(11, F_SETFD, FD_CLOEXEC) = 0
read(11, "UPID:vds1:00004D65:004B627F:51A1"..., 8192) = 8192
brk(0x24d7000) = 0x24d7000
read(11, "enu=on' -vga cirrus -k en-us -m "..., 8192) = 8192
read(11, "one,id=drive-virtio0,format=raw,"..., 8192) = 3151
read(11, "", 8192) = 0
close(11) = 0
open("/proc/19813/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/19755/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/19695/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/19652/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/11440/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/11411/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/9897/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/8200/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/8171/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/8143/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/8106/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/8078/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/8046/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/7871/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/7833/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/7802/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/7755/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/7527/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/7305/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/4986/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/4976/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/4850/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/4070/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/4040/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/4000/stat", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/var/log/pve/tasks/active.tmp.19877", O_WRONLY|O_CREAT, 0644) = 11
ioctl(11, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff39059c90) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(11, 0, SEEK_CUR) = 0
fstat(11, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
fcntl(11, F_SETFD, FD_CLOEXEC) = 0
brk(0x24fc000) = 0x24fc000
brk(0x24f8000) = 0x24f8000
write(11, "UPID:vds1:00004DA6:004B9F2E:51A1"..., 8192) = 8192
write(11, " qa2.forbet.net -smp 'sockets=2,"..., 8192) = 8192
write(11, "0' -drive 'file=/var/lib/vz/imag"..., 3139) = 3139
close(11) = 0
rename("/var/log/pve/tasks/active.tmp.19877", "/var/log/pve/tasks/active") = 0
close(10) = 0
sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\321", 1}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 1
futex(0x7f3279641010, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 0, {1369551270, 155807697}, ffffffff) = 0
rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
rt_sigaction(SIGTERM, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, {SIG_IGN, [], SA_RESTORER, 0x7f3278a14030}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [QUIT], [], 8) = 0
rt_sigaction(SIGQUIT, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, {SIG_IGN, [], SA_RESTORER, 0x7f3278a14030}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
rt_sigaction(SIGINT, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, {SIG_IGN, [], SA_RESTORER, 0x7f3278a14030}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [PIPE], [], 8) = 0
rt_sigaction(SIGPIPE, {0x7f32791429d0, [], SA_RESTORER, 0x7f3278a14030}, {SIG_IGN, [], SA_RESTORER, 0x7f3278a14030}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
ioctl(6, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff39059c80) = -1 EINVAL (Invalid argument)
lseek(6, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fstat(6, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
fcntl(6, F_SETFD, FD_CLOEXEC) = 0
select(8, [6], NULL, NULL, {1, 0}) = 1 (in [6], left {0, 452797})
read(6, "Failed to allocate 4294967296 B:"..., 4096) = 56
write(1, "Failed to allocate 4294967296 B:"..., 56Failed to allocate 4294967296 B: Cannot allocate memory
)
Click to expand...
 
Well. Got it!

Everything were right. For example, we have 10Gb RAM node with 6 virtual machines (2Gb RAM per each).
We start first VM. It allocates all 2Gb but actually uses 1Gb. Command free shows me 9Gb RAM free but actually I have just 8Gb.
If we start 5 VMs which use just 7Gb RAM command free will show 3Gb of available memory. But actually it is 0Gb because all 10Gb are allocated (5x2Gb=10Gb).
 
If VM does not start , despite the fact that you have free memory.

~#
Bash: 
qm start 100
Code: 
kvm: cannot set up guest memory 'pc.ram': Cannot allocate memorystart failed: QEMU exited with code 1

In order to resolve this issue , execute the following

Bash: 
 echo 1 > /proc/sys/vm/overcommit_memory

For more detail visit Configuring System Memory Capacity

This command has resolved the issue by enable the overcommit. It is possible to enable it temporarily and persistently as well.
 

Attachments

  • Screenshot 2024-02-28 at 10.36.24 pm.png
    Screenshot 2024-02-28 at 10.36.24 pm.png
    443 KB · Views: 5
Last edited:
  • Like
Reactions: Kingneutron
Hi,
imtiaz said:
If VM does not start , despite the fact that you have free memory.

~#
Bash: 
qm start 100
Code: 
kvm: cannot set up guest memory 'pc.ram': Cannot allocate memorystart failed: QEMU exited with code 1

In order to resolve this issue , execute the following

Bash: 
 echo 1 > /proc/sys/vm/overcommit_memory

For more detail visit Configuring System Memory Capacity

This command has resolved the issue by enable the overcommit. It is possible to enable it temporarily and persistently as well.
Click to expand...
you'll need to be more careful to monitor the actual usage though. VMs will be killed when the actually available memory runs out.

See also: https://pve.proxmox.com/wiki/Dynamic_Memory_Management
 
  • Like
Reactions: Kingneutron
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back