Running 4-beta 4.0-26/5d4a615b.
I set up the first VM on this machine, which has 64GB of RAM and ZFS backed storage. The VM was given 16GB of RAM to run FreeBSD 10.2. I'm using ZFS send/receive to copy data into the VM from another machine. After it loaded about 14Gb of the data, the machine just stopped. Nothing in the VM's logs nor anything obvious in the Proxmox host logs other than "process exit".
I restarted the VM after giving it 32GB of RAM (dynamically allocated), and this time it was able to copy via send/receive about 25GB of data before just stopping. Again, with nothing notable in any logs. At this point, I cannot get the VM to restart at all.
Here is everything I see in the logs from the last time it ran:
Now every time I try to start it this is all it does:
I restarted the pvedaemon and pveproxy services on the host, but that's kind of the end of my ideas short of reboot.
It is my guess that the VM died because the ZFS ARC ran out of memory -- it was showing continued growth as the copy was proceeding, and the VM memory use chart on proxmox was approaching the max allowed amount. Either that, or something is broken with the KVM on 4.x when it reaches the max memory. I want to try running more tests to limit the arc on the freebsd vm, but at this point nothing will restart... I will try rebooting the proxmox server then see what happens.
I set up the first VM on this machine, which has 64GB of RAM and ZFS backed storage. The VM was given 16GB of RAM to run FreeBSD 10.2. I'm using ZFS send/receive to copy data into the VM from another machine. After it loaded about 14Gb of the data, the machine just stopped. Nothing in the VM's logs nor anything obvious in the Proxmox host logs other than "process exit".
I restarted the VM after giving it 32GB of RAM (dynamically allocated), and this time it was able to copy via send/receive about 25GB of data before just stopping. Again, with nothing notable in any logs. At this point, I cannot get the VM to restart at all.
Here is everything I see in the logs from the last time it ran:
Code:
Aug 23 15:30:02 pve2.int.kcilink.com pvedaemon[4955]: start VM 103: UPID:pve2:0000135B:010ED212:55DA1F3A:qmstart:103:root@pam:
Aug 23 15:30:02 pve2.int.kcilink.com pvedaemon[32110]: <root@pam> starting task UPID:pve2:0000135B:010ED212:55DA1F3A:qmstart:103:root@pam:
Aug 23 15:30:03 pve2.int.kcilink.com kernel: device tap103i0 entered promiscuous mode
Aug 23 15:30:03 pve2.int.kcilink.com kernel: vmbr0: port 2(tap103i0) entered forwarding state
Aug 23 15:30:03 pve2.int.kcilink.com kernel: vmbr0: port 2(tap103i0) entered forwarding state
Aug 23 15:30:04 pve2.int.kcilink.com pvedaemon[5007]: starting vnc proxy UPID:pve2:0000138F:010ED2A0:55DA1F3C:vncproxy:103:root@pam:
Aug 23 15:30:04 pve2.int.kcilink.com pvedaemon[32535]: <root@pam> starting task UPID:pve2:0000138F:010ED2A0:55DA1F3C:vncproxy:103:root@pam:
Aug 23 15:30:05 pve2.int.kcilink.com kernel: kvm: zapping shadow pages for mmio generation wraparound
Aug 23 15:33:05 pve2.int.kcilink.com pvedaemon[32535]: <root@pam> successful auth for user 'root@pam'
Aug 23 15:38:33 pve2.int.kcilink.com pmxcfs[1708]: [dcdb] notice: data verification successful
Aug 23 15:42:00 pve2.int.kcilink.com systemd-timesyncd[1512]: interval/delta/delay/jitter/drift 2048s/-0.001s/0.043s/0.007s/-3ppm
Aug 23 15:43:05 pve2.int.kcilink.com pveproxy[2118]: worker exit
Aug 23 15:43:05 pve2.int.kcilink.com pveproxy[24548]: worker 2118 finished
Aug 23 15:43:05 pve2.int.kcilink.com pveproxy[24548]: starting 1 worker(s)
Aug 23 15:43:05 pve2.int.kcilink.com pveproxy[24548]: worker 28006 started
Aug 23 15:46:53 pve2.int.kcilink.com pvedaemon[32535]: worker exit
Aug 23 15:46:53 pve2.int.kcilink.com pvedaemon[2059]: worker 32535 finished
Aug 23 15:46:53 pve2.int.kcilink.com pvedaemon[2059]: starting 1 worker(s)
Aug 23 15:46:53 pve2.int.kcilink.com pvedaemon[2059]: worker 15248 started
Aug 23 15:47:52 pve2.int.kcilink.com kernel: vmbr0: port 2(tap103i0) entered disabled state
Aug 23 15:47:52 pve2.int.kcilink.com kernel: zd16: p1 p2 p3
Aug 23 15:48:05 pve2.int.kcilink.com pvedaemon[32110]: <root@pam> successful auth for user 'root@pam'
Aug 23 15:49:37 pve2.int.kcilink.com pvedaemon[5007]: command '/bin/nc6 -l -p 5901 -w 10 -e '/usr/sbin/qm vncproxy 103 2>/dev/null'' failed: exit code 1
Now every time I try to start it this is all it does:
Code:
Aug 23 16:17:38 pve2.int.kcilink.com pvedaemon[23958]: <root@pam> starting task UPID:pve2:000063AF:01132DB3:55DA2A62:qmstart:103:root@pam:
Aug 23 16:17:38 pve2.int.kcilink.com pvedaemon[25519]: start VM 103: UPID:pve2:000063AF:01132DB3:55DA2A62:qmstart:103:root@pam:
Aug 23 16:17:38 pve2.int.kcilink.com pvedaemon[25519]: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 103 -p 'CPUShares=1000' /usr/bin/kvm -id 103 -chardev 'socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/103.vnc,x509,password -pidfile /var/run/qemu-server/103.pid -daemonize -smbios 'type=1,uuid=676eee0b-3b28-42be-9890-8cafeb5fe1cc' -name staging -smp '4,sockets=2,cores=2,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 16384 -object 'memory-backend-ram,size=8192M,id=ram-node0' -numa 'node,nodeid=0,cpus=0-1,memdev=ram-node0' -object 'memory-backend-ram,size=8192M,id=ram-node1' -numa 'node,nodeid=1,cpus=2-3,memdev=ram-node1' -k en-us -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:fde03f29af2f' -drive 'file=/dev/zvol/tank/vm-103-disk-2,if=none,id=drive-virtio1,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' -device 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb' -drive 'file=/mnt/pve/filer/template/iso/FreeBSD-10.2-RELEASE-amd64-bootonly.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/dev/zvol/tank/vm-103-disk-1,if=none,id=drive-virtio0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=101' -netdev 'type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=6E:7D:F9:87:BE:A6,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1
I restarted the pvedaemon and pveproxy services on the host, but that's kind of the end of my ideas short of reboot.
It is my guess that the VM died because the ZFS ARC ran out of memory -- it was showing continued growth as the copy was proceeding, and the VM memory use chart on proxmox was approaching the max allowed amount. Either that, or something is broken with the KVM on 4.x when it reaches the max memory. I want to try running more tests to limit the arc on the freebsd vm, but at this point nothing will restart... I will try rebooting the proxmox server then see what happens.