No shutdown for Windows 10 machines

[sigxcpu]

Since upgrading to PVE4b2, the Windows 10 machine does not really shut down. The VM screen goes black and CPU usage jumps to 100% (multiplied by the number of VM cores).
Forced downgrade to QEMU 2.3 and does the same, so I assume it has something to do with 4.2.0 kernel.

Tried a fresh install of Windows and it does the same, so it is not something inside the VM.

strace shows that all the KVM process is doing like crazy is

ioctl(..., KVM_RUN,0)

after shutdown.

Killing the KVM process always works.

 

[spirit]

Maybe this is related to
http://forum.proxmox.com/threads/23548-PVE-4-0-Beta2-qemu-acpi

On this post I have sent a 4.2.1 kernel, can you test it ?

IS it the same with linux guest or other windows (7,8,8.1 ?) ?

 

[sigxcpu]

FreeBSD works fine (if machine != q35, hangs otherwise with interrupt storm on irq19). I don't have a Linux guest.
Does your patched 4.2.1 kernel have ZFS 0.6.5.1? Otherwise there will be breakage with userland tools.

LE: Nope, does not fix it. Still hangs after shutdown (I'm not sure it is a clean shutdown either, because it is too fast).

 

[spirit]

Another possibility (if that occur only on windows),
can you edit the file
/usr/share/perl5/PVE/QemuServer.pm

and comment theses line

                push @$cpuFlags , 'hv_vapic' if !$nokvm;
                push @$cpuFlags , 'hv_time' if !$nokvm;

then restart
/etc/init.d/pvedaemon restart

and stop/start the vm again.


Theses are new hyper-v flags we have enabled for proxmox 4.0.

 

[sigxcpu]

Nope, still doesn't solve it. Here's the kvm process parameters:

/usr/bin/kvm -id 107 -chardev socket,id=qmp,path=/var/run/qemu-server/107.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/107.vnc,x509,password -pidfile /var/run/qemu-server/107.pid -daemonize -smbios type=1,uuid=e29e6d3f-d35d-4c0a-935b-3dc483a38ed6 -name win10547 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga std -no-hpet -cpu kvm64,hv_spinlocks=0x1fff,hv_relaxed,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 2048 -k en-us -object iothread,id=iothread-virtio0 -readconfig /usr/share/qemu-server/pve-q35.cfg -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:ae3e76a8f516 -drive file=/raid0/proxmox/template/iso/en_windows_10_pro_10547_x64_dvd.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/dev/zvol/zones/vm-107-disk-1,if=none,id=drive-virtio0,cache=unsafe,discard=on,format=raw,aio=threads,detect-zeroes=unmap -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,bootindex=100 -drive file=/raid0/proxmox/template/iso/virtio-win-0.1.110.iso,if=none,id=drive-ide0,media=cdrom,aio=threads -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=201 -netdev type=tap,id=net0,ifname=tap107i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=CA:AF:8A:43:59:16,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -machine type=q35 -global kvm-pit.lost_tick_policy=discard

 

[spirit]

[COLOR=#333333]Nope, still doesn't solve it.
[/COLOR]

other thanks.

another new flags are (you can also comment them in QemuServer.pm) :

+kvm_pv_unhalt,
+kvm_pv_eoi


Also, some user reports kernel 4.2 problems,
maybe can you try to downdrage to 4.1 kernel:

http://download.proxmox.com/debian/dists/jessie/pvetest/binary-amd64.beta1/pve-kernel-4.1.3-1-pve_4.1.3-7_amd64.deb

 

[sigxcpu]

Nope, still not working. I've tried with first ones on and remove the last ones and also with all 4 commented out.

Unfortunately I don't think I can test with 4.1.3 as the userland ZFS tools are too new (0.6.5.1) now.

LE: I have another server with L5630 CPU and I will move this VM on that one and check to make sure it is not a CPU issue. The one with problems is a HP Microserver Gen8 with E3-1265L V2.

Nope. The L5630 machine (PVE4b2) does exactly the same so it is not a physical CPU issue.

 

[spirit]

Is is only with windows 10 ? (windows 8.1 is working fine ?)

Does it work with proxmox 3.4 ?

 

[sigxcpu]

Unfortunately I don't have an Windows 8 and neither proxmox 3.4. I've just moved back from SmartOS to Proxmox because of Proxmox 4.0

 

[tom]

On all my testboxes, win10 can shutdown without problems. Do you see this also a fresh Win10 installation?

 

[sigxcpu]

Yes. I did install it from scratch (build 10547). Same thing happens with "stable" build (10240).
As I've said above, FreeBSD VM does the same if machine type is "q35". 100% on all VM CPUs. Maybe it is helpful that is logs an interrupt storm on IRQ19 after shutdown.
The QEMU monitor is dead in this phase. All commands fail with "timeout".

 

[tom]

pls post your VM config:

> qm config VMID

 

[sigxcpu]

# qm config 107
bootdisk: virtio0
cores: 4
ide0: ISO_Templates:iso/virtio-win-0.1.110.iso,media=cdrom,size=55260K
ide2: ISO_Templates:iso/en_windows_10_pro_10547_x64_dvd.iso,media=cdrom
machine: q35
memory: 2048
name: win10547
net0: virtio=CA:AF:8A:43:59:16,bridge=vmbr0
numa: 0
ostype: win8
smbios1: uuid=e29e6d3f-d35d-4c0a-935b-3dc483a38ed6
sockets: 1
virtio0: Zones:vm-107-disk-1,cache=unsafe,discard=on,backup=no,iothread=on,size=64G

Browsing the net I've found an old bug regarding the Balloon driver. I've removed it but shutdown still fails.

 

[sigxcpu]

Extra information:

I've tried the following with same results:

- host cpu vs qemu64
- 2x2 with numa enabled vs 1x4 without numa

The pveversion output:

# pveversion -v
proxmox-ve: 4.0-13 (running kernel: 4.2.1-1-pve)
pve-manager: 4.0-39 (running version: 4.0-39/ab3cc94a)
pve-kernel-4.1.3-1-pve: 4.1.3-7
pve-kernel-4.2.0-1-pve: 4.2.0-13
pve-kernel-4.2.1-1-pve: 4.2.1-14
lvm2: 2.02.116-pve1
corosync-pve: 2.3.5-1
libqb0: 0.17.2-1
pve-cluster: 4.0-19
qemu-server: 4.0-25
pve-firmware: 1.1-7
libpve-common-perl: 4.0-24
libpve-access-control: 4.0-8
libpve-storage-perl: 4.0-23
pve-libspice-server1: 0.12.5-1
vncterm: 1.2-1
pve-qemu-kvm: 2.4-8
pve-container: 0.9-23
pve-firewall: 2.0-11
pve-ha-manager: 1.0-7
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.3-1
lxcfs: 0.9-pve2
cgmanager: 0.37-pve2
criu: 1.6.0-1
zfsutils: 0.6.5-pve1~jessie

 

[tom]

# qm config 107
bootdisk: virtio0
cores: 4
ide0: ISO_Templates:iso/virtio-win-0.1.110.iso,media=cdrom,size=55260K
ide2: ISO_Templates:iso/en_windows_10_pro_10547_x64_dvd.iso,media=cdrom
machine: q35
memory: 2048
name: win10547
net0: virtio=CA:AF:8A:43:59:16,bridge=vmbr0
numa: 0
ostype: win8
smbios1: uuid=e29e6d3f-d35d-4c0a-935b-3dc483a38ed6
sockets: 1
virtio0: Zones:vm-107-disk-1,cache=unsafe,discard=on,backup=no,iothread=on,size=64G

Browsing the net I've found an old bug regarding the Balloon driver. I've removed it but shutdown still fails.

remove

discard=on
iothread=on

and try again (poweroff, start again).

 

[sigxcpu]

It stayed at 100% for few seconds but it shut down successfully after.
With discard=on only it works.

Yes, iothread is the issue. The FreeBSD VM is fine with it enabled, it shutsdown OK.

 

[spirit]

It stayed at 100% for few seconds but it shut down successfully after.
With discard=on only it works.

Yes, iothread is the issue. The FreeBSD VM is fine with it enabled, it shutsdown OK.

Great that you have find the problem.
I never see this with iothread before, but I don't have tested windows 10 yet.

BTW, why do you use cache=unsafe ? (because it's really really unsafe in case of crash)

also why do you use q35 model ? (it's only usefull in some case of pci passthrough)

 

[sigxcpu]

"q35" is to play with.
cache=unsafe and zfs set sync=disabled are things that I always use for throwaway machines. This Windows 10 has a snapshot after being clean installed and I can always revert to it if something gets broken. Its sole purpose is to see what MS is doing nowadays (I'm a Mac user).

 

[spirit]

>>"q35" is to play with.

I ask that because some user have reported hanging problem with q35 recently.

>>cache=unsafe and zfs set sync=disabled are things that I always use for throwaway machines. This Windows 10 has a snapshot after being clean installed and I can always revert to it if something gets broken. Its sole purpose is to see what MS is doing >>nowadays (I'm a Mac user).

Ok, great. (I want to be sure that you don't use that for critical vms