VM Boot failure: RAM Balloon driver cant get pages

[penetal]

When I start my VMs (Ubuntu 15.04) I often, if not usually get this error message when cold booting.

virtio_balloon virtio0: Out of puff! Can't get 1 pages

The VM seems to settle at 94MB/2GB used RAM when it hangs there.
I don't seem to be able to do anything with the VM through webinterface console, but if I connect via "serial cable" over socket from host I get into initramfs console, output from dmesg at end of post.

If I then reset the VM (not shutting it down completely), it will boot up just fine.

Do I need to do something to the host or VM so that it will get more memory during boot? I currently have the VM set to 256MB ram minimum and 2GB maximum. I have also tried to just set the VM to have 2GB fixed amount, but the problem was the same.

Output of dmesg after failed boot (shortened to fit into post, I cant post links yet but full log on this pastebin id: bH6DX2eR ):

[    7.928681] 262014 pages RAM
[    7.929491] 0 pages HighMem/MovableOnly
[    7.930337] 227731 pages reserved
[    7.931158] 0 pages cma reserved
[    7.931963] 0 pages hwpoisoned
[    7.932780] virtio_balloon virtio0: Out of puff! Can't get 1 pages
[    8.136276] vballoon: page allocation failure: order:0, mode:0x310da
[    8.137323] CPU: 0 PID: 46 Comm: vballoon Not tainted 3.19.0-33-generic #38-Ubuntu
[    8.138344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
[    8.140496]  0000000000000000 ffff880038ea7ba8 ffffffff817c4f9f 0000000000008ffa
[    8.141598]  00000000000310da ffff880038ea7c38 ffffffff81180868 ffff880038ea7c38
[    8.142670]  ffffffff8119157a 0000000000000000 0000000000000020 00000000000310da
[    8.143743] Call Trace:
[    8.144605]  [<ffffffff817c4f9f>] dump_stack+0x45/0x57
[    8.145552]  [<ffffffff81180868>] warn_alloc_failed+0xd8/0x130
[    8.146534]  [<ffffffff8119157a>] ? try_to_free_pages+0xba/0x140
[    8.147486]  [<ffffffff811849b2>] __alloc_pages_nodemask+0x842/0xba0
[    8.148437]  [<ffffffff811cb3c1>] alloc_pages_current+0x91/0x110
[    8.149358]  [<ffffffff811f16e1>] balloon_page_enqueue+0x21/0xa0
[    8.150287]  [<ffffffff81487096>] fill_balloon+0x96/0x110
[    8.151175]  [<ffffffff814878fe>] balloon+0x18e/0x220
[    8.152019]  [<ffffffff810b75f0>] ? __wake_up_sync+0x20/0x20
[    8.152918]  [<ffffffff81487770>] ? virtballoon_probe+0x1c0/0x1c0
[    8.153768]  [<ffffffff81095959>] kthread+0xc9/0xe0
[    8.154548]  [<ffffffff81095890>] ? kthread_create_on_node+0x1c0/0x1c0
[    8.155393]  [<ffffffff817cc018>] ret_from_fork+0x58/0x90
[    8.156198]  [<ffffffff81095890>] ? kthread_create_on_node+0x1c0/0x1c0
[    8.157019] Mem-Info:
[    8.157635] Node 0 DMA per-cpu:
[    8.158278] CPU    0: hi:    0, btch:   1 usd:   0
[    8.158971] Node 0 DMA32 per-cpu:
[    8.159592] CPU    0: hi:  186, btch:  31 usd: 142
[    8.160300] active_anon:222 inactive_anon:14 isolated_anon:0
[    8.160300]  active_file:0 inactive_file:0 isolated_file:0
[    8.160300]  unevictable:14738 dirty:0 writeback:0 unstable:0
[    8.160300]  free:10188 slab_reclaimable:1702 slab_unreclaimable:1785
[    8.160300]  mapped:588 shmem:41 pagetables:32 bounce:0
[    8.160300]  free_cma:0
[    8.164577] Node 0 DMA free:4360kB min:464kB low:580kB high:696kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1072kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:7664kB mlocked:0kB dirty:0kB writeback:0kB mapped:100kB shmem:0kB slab_reclaimable:8kB slab_unreclaimable:48kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1072 all_unreclaimable? yes
[    8.169150] lowmem_reserve[]: 0 975 975 975
[    8.169939] Node 0 DMA32 free:36392kB min:36468kB low:45584kB high:54700kB active_anon:888kB inactive_anon:56kB active_file:0kB inactive_file:0kB unevictable:57880kB isolated(anon):0kB isolated(file):0kB present:1032064kB managed:129468kB mlocked:0kB dirty:0kB writeback:0kB mapped:2252kB shmem:164kB slab_reclaimable:6800kB slab_unreclaimable:7092kB kernel_stack:784kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[    8.174726] lowmem_reserve[]: 0 0 0 0
[    8.175535] Node 0 DMA: 2*4kB (UM) 2*8kB (U) 1*16kB (U) 3*32kB (UM) 2*64kB (M) 2*128kB (M) 3*256kB (UM) 2*512kB (M) 0*1024kB 1*2048kB (R) 0*4096kB = 4360kB
[    8.177605] Node 0 DMA32: 52*4kB (UEM) 41*8kB (UE) 33*16kB (UEM) 14*32kB (UE) 9*64kB (UE) 4*128kB (EM) 4*256kB (UEM) 2*512kB (UE) 3*1024kB (UEM) 2*2048kB (UE) 6*4096kB (MR) = 36392kB
[    8.179863] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[    8.180938] 14779 total pagecache pages
[    8.181800] 0 pages in swap cache
[    8.182636] Swap cache stats: add 0, delete 0, find 0/0
[    8.183561] Free swap  = 0kB
[    8.184387] Total swap = 0kB
[    8.185206] 262014 pages RAM
[    8.186024] 0 pages HighMem/MovableOnly
[    8.186871] 227731 pages reserved
[    8.187689] 0 pages cma reserved
[    8.188504] 0 pages hwpoisoned
[    8.189303] virtio_balloon virtio0: Out of puff! Can't get 1 pages
[    8.392273] vballoon: page allocation failure: order:0, mode:0x310da
[    8.393361] CPU: 0 PID: 46 Comm: vballoon Not tainted 3.19.0-33-generic #38-Ubuntu
[    8.394456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
[    8.396365]  0000000000000000 ffff880038ea7ba8 ffffffff817c4f9f 000000000000945c
[    8.397468]  00000000000310da ffff880038ea7c38 ffffffff81180868 ffff880038ea7c28
[    8.398561]  ffffffff817c71ec ffff880038eb93a0 0000000000014240 ffff880038ea7fd8
[    8.399650] Call Trace:
[    8.400497]  [<ffffffff817c4f9f>] dump_stack+0x45/0x57
[    8.401434]  [<ffffffff81180868>] warn_alloc_failed+0xd8/0x130
[    8.402383]  [<ffffffff817c71ec>] ? __schedule+0x39c/0x900
[    8.403300]  [<ffffffff811849b2>] __alloc_pages_nodemask+0x842/0xba0
[    8.404250]  [<ffffffff811cb3c1>] alloc_pages_current+0x91/0x110
[    8.405166]  [<ffffffff811f16e1>] balloon_page_enqueue+0x21/0xa0
[    8.406073]  [<ffffffff81487096>] fill_balloon+0x96/0x110
[    8.406941]  [<ffffffff814878fe>] balloon+0x18e/0x220
[    8.407773]  [<ffffffff810b75f0>] ? __wake_up_sync+0x20/0x20
[    8.408633]  [<ffffffff81487770>] ? virtballoon_probe+0x1c0/0x1c0
[    8.409478]  [<ffffffff81095959>] kthread+0xc9/0xe0
[    8.410253]  [<ffffffff81095890>] ? kthread_create_on_node+0x1c0/0x1c0
[    8.411093]  [<ffffffff817cc018>] ret_from_fork+0x58/0x90
[    8.411881]  [<ffffffff81095890>] ? kthread_create_on_node+0x1c0/0x1c0
[    8.412714] Mem-Info:
[    8.413328] Node 0 DMA per-cpu:
[    8.413969] CPU    0: hi:    0, btch:   1 usd:   0
[    8.414667] Node 0 DMA32 per-cpu:
[    8.415285] CPU    0: hi:  186, btch:  31 usd: 142
[    8.415964] active_anon:222 inactive_anon:14 isolated_anon:0
[    8.415964]  active_file:0 inactive_file:0 isolated_file:0
[    8.415964]  unevictable:14738 dirty:0 writeback:0 unstable:0
[    8.415964]  free:10188 slab_reclaimable:1702 slab_unreclaimable:1785
[    8.415964]  mapped:588 shmem:41 pagetables:32 bounce:0
[    8.415964]  free_cma:0
[    8.420201] Node 0 DMA free:4360kB min:464kB low:580kB high:696kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1072kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:7664kB mlocked:0kB dirty:0kB writeback:0kB mapped:100kB shmem:0kB slab_reclaimable:8kB slab_unreclaimable:48kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1072 all_unreclaimable? yes
[    8.424694] lowmem_reserve[]: 0 975 975 975
[    8.425475] Node 0 DMA32 free:36392kB min:36468kB low:45584kB high:54700kB active_anon:888kB inactive_anon:56kB active_file:0kB inactive_file:0kB unevictable:57880kB isolated(anon):0kB isolated(file):0kB present:1032064kB managed:129468kB mlocked:0kB dirty:0kB writeback:0kB mapped:2252kB shmem:164kB slab_reclaimable:6800kB slab_unreclaimable:7092kB kernel_stack:784kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[    8.430218] lowmem_reserve[]: 0 0 0 0
[    8.431028] Node 0 DMA: 2*4kB (UM) 2*8kB (U) 1*16kB (U) 3*32kB (UM) 2*64kB (M) 2*128kB (M) 3*256kB (UM) 2*512kB (M) 0*1024kB 1*2048kB (R) 0*4096kB = 4360kB
[    8.433100] Node 0 DMA32: 52*4kB (UEM) 41*8kB (UE) 33*16kB (UEM) 14*32kB (UE) 9*64kB (UE) 4*128kB (EM) 4*256kB (UEM) 2*512kB (UE) 3*1024kB (UEM) 2*2048kB (UE) 6*4096kB (MR) = 36392kB
[    8.435355] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[    8.436427] 14779 total pagecache pages
[    8.437301] 0 pages in swap cache
[    8.438140] Swap cache stats: add 0, delete 0, find 0/0
[    8.439085] Free swap  = 0kB
[    8.439910] Total swap = 0kB
[    8.440733] 262014 pages RAM
[    8.441542] 0 pages HighMem/MovableOnly
[    8.442397] 227731 pages reserved
[    8.443216] 0 pages cma reserved
[    8.444026] 0 pages hwpoisoned
[    8.444828] virtio_balloon virtio0: Out of puff! Can't get 1 pages
[   47.780892] random: nonblocking pool is initialized

 

[robertb]

Hello everyone,

I'm sorry for bringing up this very old thread.
Unfortunately, I'm running into exactly the same issue on multiple servers.

Ubuntu (different versions) shows this behaviour on every other boot during initramfs.

I suspect that the issue is primarily noticeable after rebooting or upgrading a pve node (reloading services).
Could this be related to one of the pve services that manage memory allocation for qemu machines?

Any help would be appreciated!

Thank you

 

[robertb]

Hi,

the issue still persists even with the latest updates, any ideas?

Best regards
Robert

 

[remark]

Can you show your setup configuration, both host and VMs?

 

[robertb]

Hi,
thanks for your reply.

of course I can give some details, even though it's nothing special at all:

The VM is usually configured with:
balloon: 256 (or 512, doesnt matter)
memory: 1024, 2048, 8192, it depends on the use case
Guest OS ranges from Debian 8 & 9 to Ubuntu 16.04 and 17.04

The hypervisor itself is Proxmox 4.4, running the latest updates in a cluster. All nodes are randomly affected.
Memory usage on the hypervisor can be either quite low or high, the problem randomly occurs.

The issue especially occurs when booting an ubuntu live system via PXE in the guest.
During kernel boot, the vm tries to acquire memory and gets the "out of puff" issue, followed by a frozen system.


Any help is appreciated!

 

[remark]

How many VMs on you host?
What is host hw configuration? Is there all of your VMs configured to use memory ballooning?
256 Mb as Min is too small. Try to set at least 1-2 Gb.
Here is thread from qemu-devel mailing list:
https://lists.nongnu.org/archive/html/qemu-devel/2015-06/msg05646.html

 

[robertb]

Hello,

the hypervisors memory ranges from 96 up to 256 GB of DDR3 regECC ram, depending on the cluster nodes.
All systems are dual xeon E5 servers on supermicro mainboards.
Yes, all VMs have ballooning configured. Sometimes this happens on hosts with only 1 virtual machine at that point of time.
Isn't it quite inefficient if there are at least 1 or 2 GB of ram reserved for each guest?

I wonder why those issues arise every now and then.

I will give it a try anyway.

Thank you.

 

[remark]

I have two Proxmox hosts, 196 Gb and 256 Gb RAM, dual Xeon, Supermicro, very similar to your configuration. Very standard servers. VMs with and without ballooning, Windows and Linux, on both nodes and I have never face with your problem.
What is the difference?..
Is your RAM fast enough?

 

[robertb]

yeah, I'd assume that the used ram is quite standard server ram (kingston, samsung). I don't think that there is any bandwidth limit.

I could imagine some condition where maybe pvestatd tries to put pressure on the guests to reduce memory using the balloon driver, before they are completely booted yet.