Still having backup issues with crashing VMs

[svendsen]

Hi,

We are still seeing backup issues with Proxmox. Issues only occour on Windows VMs.
Backup completes 100%, no errors. But inside Windows VM we receive an Atapi event ID 9 (The device, \Device\Ide\IdePort0, did not respond within the timeout period.).
And Windows either get unrespondable or Windows will keep running but database will crash.

It's a Windows 2003 R2 server. Doesn't use Virtio drivers. Any clues?


Here are my VM config:

root@proxmox:~# cat /etc/pve/local/qemu-server/101.conf
bootdisk: ide0
cores: 1
cpu: kvm64
ide0: local:101/vm-101-disk-1.raw,format=raw,size=15G
ide2: cdrom,media=cdrom
memory: 512
name: w2003-dev
net0: e1000=6A:E5:5A:CC:E6:ED,bridge=vmbr0
ostype: wxp
sockets: 1
vga: std

And my pveversion:

root@proxmox:~# pveversion -v
proxmox-ve-2.6.32: 3.2-124 (running kernel: 2.6.32-28-pve)
pve-manager: 3.2-2 (running version: 3.2-2/82599a65)
pve-kernel-2.6.32-27-pve: 2.6.32-121
pve-kernel-2.6.32-28-pve: 2.6.32-124
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-15
pve-firmware: 1.1-2
libpve-common-perl: 3.0-14
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-6
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1

 

[e100]

The symptoms you describe indicate that your IO subsystem is unable to keep up with the backup and the VMs requests at the same time.

Output of pveperf?

What disks/RAID are you using for the backup storage and the VM storage?

 

[svendsen]

root@proxmox:~# pveperf
CPU BOGOMIPS: 47881.56
REGEX/SECOND: 858220
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 470.09 MB/sec
AVERAGE SEEK TIME: 5.17 ms
FSYNCS/SECOND: 3016.44
DNS EXT: 144.45 ms
DNS INT: 62.33 ms (lf.local)

Currently running RAID 6 on 4 x 10k SAS drives on a HP P420i Card with 1GB cache + battery. Will propably upgrade to RAID10 (I'm not fan of RAID5ish)

Backup storage is a older ML350 G6.. or is it G7 - cannot remember. With FreeNAS installed. 1Gb Network between.

Also worth noticing. That only this VM is crashing. All the other (Linux) VMs runs fine..

 

[jacksharing]

I also have the same issue. Any idea?

 

[spirit]

I also have the same issue. Any idea?

pveversion ? vm config ?

 

[jacksharing]

pveversion ? vm config ?

root@proxmox1:~# cat /etc/pve/local/qemu-server/109.conf
balloon: 512
boot: dcn
bootdisk: ide0
cores: 2
ide0: local:109/vm-109-disk-1.qcow2,format=qcow2,size=100G
ide2: cdrom,media=cdrom
memory: 8046
name: spice
net0: e1000=4E:10:1A:4C:4F:E1,bridge=vmbr0
onboot: 1
ostype: wxp
sockets: 1


root@proxmox1:~# pveversion -v
proxmox-ve-2.6.32: 3.2-126 (running kernel: 2.6.32-29-pve)
pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-27-pve: 2.6.32-121
pve-kernel-2.6.32-28-pve: 2.6.32-124
pve-kernel-2.6.32-22-pve: 2.6.32-107
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-16
pve-firmware: 1.1-3
libpve-common-perl: 3.0-18
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-8
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1

 

[spirit]

do you have tried with virtio instead ide in your windows guest ? (ide is really slow and lack of parallelism, and with backup running in background, it can be even worse)

 

[jacksharing]

Changing from IDE to Virtio just gave me blank black screen after windows loading logo. Any idea?

 

[svendsen]

Have you installed Virtio drivers?

 

[jacksharing]

Virtio done. Lets see the result after scheduled backup run.

BTW, the blank black screen is because of Memory Balloon, after installing Memory Balloon driver, reboot give blank screen right after windows loading progress logo.any idea?