Cannot start VM with vRAM >128G

[zang3tsu]

Said VM does start with 32, 64, 96 GB vRAM but not with 128GB or higher. Here is the output that I'm getting:

TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=68782ec6-2866-4bfb-88e5-2e4fca313166' -name geostorage -smp '12,sockets=1,cores=12,maxcpus=12' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep -m 131072 -k en-us -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=03:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:f83ee4efa04' -drive 'if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/100/vm-100-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,aio=native,cache=none,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=76:FE:6D:A4:B0:92,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: got timeout

 

[swartz]

Is the kvm process still running after you get this error?
ps -ef | grep kvm

Is it possible you may have hit process limits?
ulimit -a

Just out of curiosity: what kind of hardware and total RAM do you have? Why does your VM need 128GB+ of RAM?

 

[fireon]

Just out of curiosity: what kind of hardware and total RAM do you have? Why does your VM need 128GB+ of RAM?

Interested too.

 

[zang3tsu]

Is the kvm process still running after you get this error?
ps -ef | grep kvm

No, kvm wasn't running.

Is it possible you may have hit process limits?
ulimit -a

Here's the output of ulimit -a:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2063315
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2063315
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

It doesn't seem to be process limits from the looks of it.

Just out of curiosity: what kind of hardware and total RAM do you have? Why does your VM need 128GB+ of RAM?

It's a SuperMicro SuperStorage server with 24 logical CPUs and 256GB total RAM. The 128GB+ RAM is for running a FreeNAS VM with 300TB+ storage.

Upon further testing, I found that it might be because of the PCI passthrough. Booting the VM with 128GB+ vRAM works with PCI passthrough disabled. Enabling it (by adding the following in the configuration file) and booting no longer works:

hostpci0: 04:00.0,driver=vfio

I'm also posting the package versions in case it's needed:

proxmox-ve-2.6.32: 3.3-147 (running kernel: 3.10.0-7-pve)
pve-manager: 3.4-1 (running version: 3.4-1/3f2d890e)
pve-kernel-3.10.0-7-pve: 3.10.0-27
pve-kernel-2.6.32-37-pve: 2.6.32-147
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-16
qemu-server: 3.3-20
pve-firmware: 1.1-3
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-31
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-12
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

 

[zang3tsu]

I also just tried running kvm on the command line and everything works. So it might just be Proxmox timing out too early. Is there anyway to adjust Proxmox's start timeout?

 

[swartz]

I also just tried running kvm on the command line and everything works. So it might just be Proxmox timing out too early. Is there anyway to adjust Proxmox's start timeout?

Do you have the pve-firewall enabled by any chance?

 

[zang3tsu]

Sorry for the late reply. No, I don't have pve-firewall enabled.

 

[swartz]

Sorry for the late reply. No, I don't have pve-firewall enabled.

I had an issue of VMs not starting. Was out of ideas, thought you might be having related issue.

Turned out it was nothing like your problem. Mine was related to glusterfs and improperly configured pve-firewall.

 

[adamb]

I have a bug opened on this as I am having the same issue. Not much activity on it though.

https://bugzilla.proxmox.com/show_bug.cgi?id=607