Windows server 2012 R2 freezes

[jompsi]

Hi all,

I wanted to migrate our VOIP(Swyx) server from a physical server to a VM(Windows Server 2012 R2). When I have done this, and the virtual VOIP server was used by the users it froze almost all 10 - 30 minutes and I had to go back to the physical server.

When the server froze I had to stop and start the vm again. And than it worked for a few minutes. In the windows logs I cant find anything helpful. There is no entry until I cold shutdown and started the vm again. Then there is the first entry, which says that the server was shutdown unexpected.

Strange thing is, that the server didnt freeze until he was used productive as VOIP server and afterwards he didnt freeze as well. I now have done some stresstesting on the VM with HeavyLoad and the cpu is used 100% but the server does not freeze. When he was productive as VOIP server the load was only 30% CPU and 2 GB of RAM were used.

My proxmox environment contains 3 servers in a cluster.

pveversion -v
proxmox-ve-2.6.32: 3.4-157 (running kernel: 2.6.32-39-pve)
pve-manager: 3.4-6 (running version: 3.4-6/102d4547)
pve-kernel-2.6.32-39-pve: 2.6.32-157
pve-kernel-2.6.32-37-pve: 2.6.32-150
pve-kernel-2.6.32-34-pve: 2.6.32-140
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-18
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

uname -a
Linux ironman 2.6.32-39-pve #1 SMP Wed Jun 24 06:39:42 CEST 2015 x86_64 GNU/Linux

Config of the vm:

bootdisk: virtio0
cores: 2
ide2: none,media=cdrom
memory: 6241
name: swyx
net0: virtio=F2:6D:75:6E:36:31,bridge=vmbr1
ostype: win8
smbios1: uuid=78cd7979-9927-4d07-8f10-6180a7b7cce8
sockets: 2
virtio0: groot-kvm-lun:vm-451-disk-1,size=50G

Syslog of the hypervisor:

Oct  7 09:13:49 ironman kernel: tap451i0: no IPv6 routers present
Oct  7 09:17:01 ironman /USR/SBIN/CRON[239970]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct  7 09:17:14 ironman pveproxy[214709]: worker exit
Oct  7 09:17:14 ironman pveproxy[3496]: worker 214709 finished
Oct  7 09:17:14 ironman pveproxy[3496]: starting 1 worker(s)
Oct  7 09:17:14 ironman pveproxy[3496]: worker 240000 started
Oct  7 09:21:09 ironman pvedaemon[228188]: worker exit
Oct  7 09:21:09 ironman pvedaemon[3467]: worker 228188 finished
Oct  7 09:21:09 ironman pvedaemon[3467]: starting 1 worker(s)
Oct  7 09:21:09 ironman pvedaemon[3467]: worker 240586 started
Oct  7 09:21:31 ironman pveproxy[214708]: worker exit
Oct  7 09:21:31 ironman pveproxy[3496]: worker 214708 finished
Oct  7 09:21:31 ironman pveproxy[3496]: starting 1 worker(s)
Oct  7 09:21:31 ironman pveproxy[3496]: worker 240639 started
Oct  7 09:21:45 ironman pmxcfs[2991]: [status] notice: received log
Oct  7 09:25:12 ironman pvedaemon[229452]: worker exit
Oct  7 09:25:12 ironman pvedaemon[3467]: worker 229452 finished
Oct  7 09:25:12 ironman pvedaemon[3467]: starting 1 worker(s)
Oct  7 09:25:12 ironman pvedaemon[3467]: worker 241175 started
Oct  7 09:26:44 ironman pvedaemon[241175]: <root@pam> starting  task UPID:ironman:0003AEFE:2482C8A1:5614C934:qmstop:451:root@pam:
Oct  7 09:26:44 ironman pvedaemon[241406]: stop VM 451: UPID:ironman:0003AEFE:2482C8A1:5614C934:qmstop:451:root@pam:
Oct  7 09:26:44 ironman kernel: vmbr1: port 3(tap451i0) entering disabled state
Oct  7 09:26:45 ironman pmxcfs[2991]: [status] notice: received log
Oct  7 09:26:46 ironman ntpd[2929]: Deleting interface #51 tap451i0,  fe80::8cbe:dcff:fe0b:fdbe#123, interface stats: received=0, sent=0,  dropped=0, active_time=783 secs
Oct  7 09:26:46 ironman ntpd[2929]: peers refreshed
Oct  7 09:26:52 ironman pvedaemon[241175]: <root@pam> starting  task UPID:ironman:0003AF27:2482CBE1:5614C93C:qmstart:451:root@pam:
Oct  7 09:26:52 ironman pvedaemon[241447]: start VM 451: UPID:ironman:0003AF27:2482CBE1:5614C93C:qmstart:451:root@pam:
Oct  7 09:26:53 ironman kernel: device tap451i0 entered promiscuous mode
Oct  7 09:26:53 ironman kernel: vmbr1: port 3(tap451i0) entering forwarding state
Oct  7 09:26:55 ironman pmxcfs[2991]: [status] notice: received log
Oct  7 09:26:57 ironman ntpd[2929]: Listen normally on 52 tap451i0 fe80::f4a2:67ff:fe63:f104 UDP 123
Oct  7 09:26:57 ironman ntpd[2929]: peers refreshed
Oct  7 09:27:04 ironman kernel: tap451i0: no IPv6 routers present
Oct  7 09:36:45 ironman pmxcfs[2991]: [status] notice: received log
Oct  7 09:50:37 ironman pveproxy[232344]: worker exit
Oct  7 09:50:37 ironman pveproxy[3496]: worker 232344 finished
Oct  7 09:50:37 ironman pveproxy[3496]: starting 1 worker(s)
Oct  7 09:50:37 ironman pveproxy[3496]: worker 244972 started
Oct  7 09:51:17 ironman pvedaemon[230549]: worker exit
Oct  7 09:51:17 ironman pvedaemon[3467]: worker 230549 finished
Oct  7 09:51:17 ironman pvedaemon[3467]: starting 1 worker(s)
Oct  7 09:51:17 ironman pvedaemon[3467]: worker 245055 started
Oct  7 09:51:45 ironman pmxcfs[2991]: [status] notice: received log
Oct  7 09:55:09 ironman rrdcached[2979]: flushing old values
Oct  7 09:55:09 ironman rrdcached[2979]: rotating journals
Oct  7 09:55:09 ironman rrdcached[2979]: started new journal /var/lib/rrdcached/journal/rrd.journal.1444204509.230686
Oct  7 09:55:09 ironman rrdcached[2979]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1444197309.230727
Oct  7 09:58:40 ironman pveproxy[240639]: worker exit
Oct  7 09:58:40 ironman pvedaemon[241175]: <root@pam> starting  task UPID:ironman:0003C18D:2485B522:5614D0B0:qmstop:451:root@pam:
Oct  7 09:58:40 ironman pvedaemon[246157]: stop VM 451: UPID:ironman:0003C18D:2485B522:5614D0B0:qmstop:451:root@pam:
Oct  7 09:58:40 ironman pveproxy[3496]: worker 240639 finished
Oct  7 09:58:40 ironman pveproxy[3496]: starting 1 worker(s)
Oct  7 09:58:40 ironman pveproxy[3496]: worker 246158 started
Oct  7 09:58:40 ironman kernel: vmbr1: port 3(tap451i0) entering disabled state
Oct  7 09:58:41 ironman pvedaemon[245055]: client closed connection
Oct  7 09:58:41 ironman pmxcfs[2991]: [status] notice: received log
Oct  7 09:58:41 ironman pmxcfs[2991]: [status] notice: received log

I have used the virtio drivers "virtio-win-0.1.102.iso".

Do you know what can cause this behaviour? It is not like a VOIP server needs a lot of ressources. On the physical server, Swyx was and now is running again runs windows server 2008 r2 with 2 GB RAM. I really would like to replace the physical server with the virtual server but I cant have another failure of the phone for 2 hours.

Thanks a lot and kind regards
Joel

 

[acidrop]

Hi

Have you tried with e1000 as a net card?
Also, have you tried to run the vm on another node?

 

[jompsi]

Hi

Thanks for your reply.

No, I havent tried it with e1000 and I havent moved the vm to another node yet.

My problem is, that I cant test it really. I have to switch to the vm, if no one is at work and if I am alone, the server does not freeze. And when the people are at work, the telephony has to work :/

I was hoping I can freeze the vm with HeavyLoad, but unfortunately that didnt work. I try to figure out how to test it without impairing the employees.

So I will try your inputs if I get the opportunity.

Regards
Joel

 

[tom]

If you do not run OpenVZ on this host, try the 3.10 kernel - some users reports better stability of windows VM with this one.

> apt-get install pve-kernel-3.10.0-13-pve

 

[jompsi]

Hi

Unfortunately on this host runs an OpenVZ. If I would update to kernel 3.10 on one node in the cluster, do the other nodes need the same kernel? Do all nodes in a cluster need the same kernel?

Thanks and regards
Joel

 

[Daniel Keller]

Hi,

I had a problem with random freezing windows 2012 r2 in a vm on top of a server with a Intel E5-2620 v3.

I could reproduce the crash with memtest86 and the fix was to change processor type of the vm to default (kvm64).

 

[sigxcpu]

If you have some basic programming skills, you cand create an Asterisk VM and program it to hammer the test (VM) server with calls to simulate the load.

 

[jompsi]

Hi Daniel and sigxcpu

Thanks for your inputs.

I am running memtest86 at this moment and the vm has passed 2 tests yet. And I already use the default cpu(kvm64).

Do you have a little manual or quick start for this, sigxcpu? Because I have done nothing with asterisk till now.

Regards
Joel

 

[sigxcpu]

You install a Linux box with Asterisk PBX that you configure to dial to the Windows VM.
Dialing out is done by placing .call files in /var/spool/asterisk/outgoing. Their format is described here: http://www.voip-info.org/wiki/view/Asterisk+auto-dial+out
Some examples are here: http://the-asterisk-book.com/1.6/call-file.html

You will need to generate call files programatically (with a script) to be able to hammer the Windows machine. Asterisk will be able to initiate and keep few hundred calls without issues using G.711A (alaw) or G.711U(ulaw) codecs so you can give the Windows PBX a big load if you want.

Tip: do not generate the call files in that folder. Generate them in a temporary folder and move them in the spool dir atomically. Otherwise Asterisk will load them partially.

 

[jompsi]

Hi

@sigxcpu: Thanks a lot for the quick start. I didnt realize it yet, but if nothing else helps, I will try this way.

About the memtest. It runs now for 96 hours and the virtual server has passed 70 tests and zero errors occured. So I think there is no problem.

Regards
Joel

 

[spirit]

Hi,

my 2cents, but:

is your bios governor configured to max performance ? (disabling powersaving, dynamic core frequencies, cores shutdown,...)

This can have great impact on clock and windows vms stability.

 

[jompsi]

Hi spirit

Do you mean the bios of the hypervisor or the vm?

Thx
Joel

 

[spirit]

Hi spirit

Do you mean the bios of the hypervisor or the vm?

Thx
Joel

hypervisor, physical server