Random BSOD Server 2008 after p2v

[check-ict]

Hello,

We moved 2 HP servers into 1 server with virtualisation (Proxmox). We used SelfImage and everything boots well. All Virtio drivers are installed and we removed all HP software.

The 2 servers are:
SBS 2008 (with Exchange 2007, AD, file, print, DNS, DHCP)
Server 2008 Standard (With SQL Express / SAP application)

At random times the servers crash with a blue screen. The SBS 2008 crashes more often, this server is used a bit more due to the Exchange software.

Some debug info of the minidumps:
NTFS_FILE_SYSTEM CI.dll
SYSTEM_SERVICE_EXCEPTION ntoskrnl.exe
IRQL_NOT_LESS_OR_EQUAL ntoskrnl.exe
SYSTEM_SERVICE_EXCEPTION ntoskrnl.exe
KMODE_EXCEPTION_NOT_HANDLED ntoskrnl.exe
DRIVER_VERIFIER_DETECTED_VIOLATION crcdisk.sys
PAGE_FAULT_IN_FREED_SPECIAL_POOL ntoskrnl.exe
SYSTEM_SERVICE_EXCEPTION mup.sys
DRIVER_CORRUPTED_EXPOOL ntoskrnl.exe

What i've tried so far:
- Memtest 24 hours
- Replaced hardware with known good test server
- Deleted a old McAfee and Synology driver
- Changed drivers (virtio/ide/e1000/vga)
- Chckdsk /F
- scf /scannow
- Disabled HP leftover drivers from starting (device management, hidden devices)
- Removed any software (except Windows/Microsoft/SAP related), so no anti-virus/back-up/monitoring etc.
- Checked eventlog for problems/errors, nothing special and nothing just before the crash

The timing of the BSOD has nothing to do with any stress, the servers also gets BSOD when doing nothing. It's not like a back-up running or a scheduled task. Also, I'm unable to reproduce the error by creating heavy load on the VM.

The VM's don't get a BSOD together, it's mostly just the SBS server and the APP server will run fine. The SBS server gets a BSOD 3 times a week and the APP server 1 or 2 times a month.

Some info about the currunt host:
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-21 (running version: 3.1-21/93bf03d4)
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-8
libpve-access-control: 3.0-7
libpve-storage-perl: 3.0-17
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-4
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1

Host has 2x 5504 Xeon processor, 16 GB ECC RAM, 4x500GB RAID10 (HP HW RAID). It's a ML350 G6 model.

I don't think this has much to do with Proxmox but maybe someone has experienced this before.

 

[acidrop]

Hello

It looks like something related with your storage subsystem.
What kind of vm disk are you using? (qcow2,raw,lvm)

 

[check-ict]

It's qcow2. We exported with SelfImage to RAW but this was a problem because of the disk space allocated. So we did a convert from raw to qcow2.

 

[udo]

It's qcow2. We exported with SelfImage to RAW but this was a problem because of the disk space allocated. So we did a convert from raw to qcow2.

Hi,
how looks the configs of the two VMs? (which cpu-type and so on).

BTW, you pve version isn't actual - but I don't think, that an update prevent the BSODs...

I would also try to convert from qcow2 back to RAW.

Udo

 

[acidrop]

Also try to disable ballooning if it is already enabled.

 

[check-ict]

I will try to disable ballooning first, see if it works.

The config is:
boot: cdn
bootdisk: ide0
cores: 4
ide0: local:100/vm-100-disk-1.qcow2,format=qcow2,size=488353816K
ide2: none,media=cdrom
memory: 6144
name: SBS
net0: virtio=2E:6D:670:78:BA,bridge=vmbr0
balloon: 0
ostype: w2k8
sockets: 1

(IDE was just to test, it crashed 2 times today with IDE so I guess it only makes things worse).

 

[wahmed]

My take on it would be the problem lies in the first step of converting the physical to an image.

Have you tried clonezilla? It worked for me without much issue. So far i converted 5 Windows Server 2003 and 3 Windows Server 2008 R2 physical machines to Proxmox VM. Following is simplified steps i follow:

1. Use clonezilla to create image of physical server
2. Create a VM with specs of physical machine as much as possible.
3. Boot VM with clonezilla and restore the image on to the VM

You may have already done it, but did you make sure the number of socket and Core are same on the VM as was in the physical? I find it important as first boot. After windows settles down then tinkering can be done.

 

[check-ict]

I've allways used selfimage and it works great. You need to stop all services and do the migration while the system is still running. After the migration I do a chckdsk and sfc /scannow to check everything. I don't think this has anything to do with it, since there is no corruption to the filesystem.

We have used the same sockets/cores and GB memory.

For now, the VM didn't crash since I disabled ballooning. I will report back within a few days if it's still stable. If it works for 1 week, it should be fine/solved.

 

[check-ict]

Disabling ballooning didn't work. I'm converting to RAW now.

 

[mir]

Have you measured disk performance using crystaldiskmark or atto-bench for qcow2 and raw?

 

[check-ict]

You mean a benchmark within the Windows VM?

I just converted to RAW and started the VM, will let you know if it fails again.

 

[mir]

You mean a benchmark within the Windows VM?

Yes, maybe it is of question of inadequate I/O when using Qcow2.

 

[check-ict]

I did benchmarks on the VM's (SBS = RAW and APP = qcow2). They both perform with the same results. It doesn't look like a performance issue.

 

[check-ict]

RAW also crashed, so this doesn't fix the problem. Any suggestions?

 

[check-ict]

I have changed the controller to Virtio, disk to Virtio and disabled tablet pointer. Waiting for the next BSOD.

 

[udo]

RAW also crashed, so this doesn't fix the problem. Any suggestions?

Hi,
you can try CPU -> type -> host

Udo

 

[check-ict]

Hi,
you can try CPU -> type -> host

Udo

I will do this as well

 

[wahmed]

Any device that were attached with the Physical server or any other PCI card that existed at the time of image creation?

Any clue in Event Viewer?

 

[check-ict]

Any device that were attached with the Physical server or any other PCI card that existed at the time of image creation?

Any clue in Event Viewer?

Eventlog has nothing, everything works fine according to eventlog. No special devices, it was a simplex server. We allready removed all drivers/software of any third-party. It's just the default Windows software and on the APP server we have SAP Business.

There is no BSOD since the last change, but it can take up to 1 week.

 

[ebiss]

Whitch virtIO drivres you use for win2008 ?