VMware to PVE migration - guest locks up

[gob]

Hi

As part of my big plan to rid our company of VMware servers, I migrated my first guest over to Proxmox recently.
This guest is an old Fedora box used as a DNS server (BIND) and is scheduled to be replaced soon so not a huge problem for me if it dies.

I Migrated and converted the vmdk to a qcow2 image and spun it up. All appeared to be working perfectly. However after a few days the VM locked up with CPU running at 100%. Console was not responding and I couldn't ssh in to take a look.
The lockups have started happening more and more freqently, now typically twice a day. Once CPU jumps to 100% it never drops again and I have to stop the guest.

Looking at var/log/messages at the time when it locks up, the log entries jump from the correct date (September) to Oct 8:

Oct  8 17:01:25 ms-005-dns01 kernel: BUG: soft lockup - CPU#1 stuck for 1179869997s! [ata/1:138]

Once I restart the machine the log entries jump back to the correct date again. The time/date on the host is correct.

Any idea what might be locking up this guest?

Here are my versions:

proxmox-ve-2.6.32: 3.4-163 (running kernel: 2.6.32-41-pve)
pve-manager: 3.4-11 (running version: 3.4-11/6502936f)
pve-kernel-2.6.32-40-pve: 2.6.32-160
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-39-pve: 2.6.32-157
pve-kernel-2.6.32-41-pve: 2.6.32-163
pve-kernel-2.6.32-37-pve: 2.6.32-150
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-3
pve-cluster: 3.0-19
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-11
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

... and here is the full log:

Oct  8 17:01:25 ms-005-dns01 kernel: BUG: soft lockup - CPU#1 stuck for 1179869997s! [ata/1:138]
Oct  8 17:01:25 ms-005-dns01 kernel: Modules linked in: sco bridge stp bnep l2cap bluetooth sunrpc iptable_nat nf_nat iptable_mangle ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput i2c_piix4 virtio_balloon e1000 virtio_pci i2c_core joydev floppy pcspkr ata_generic pata_acpi [last unloaded: scsi_wait_scan]
Oct  8 17:01:25 ms-005-dns01 kernel:
Oct  8 17:01:25 ms-005-dns01 kernel: Pid: 138, comm: ata/1 Not tainted (2.6.27.41-170.2.117.fc10.i686 #1) Standard PC (i440FX + PIIX, 1996)
Oct  8 17:01:25 ms-005-dns01 kernel: EIP: 0060:[<c06ac166>] EFLAGS: 00000282 CPU: 1
Oct  8 17:01:25 ms-005-dns01 kernel: EIP is at _spin_unlock_irqrestore+0x2d/0x38
Oct  8 17:01:25 ms-005-dns01 kernel: EAX: 00000282 EBX: 00000282 ECX: 00000000 EDX: 00000282
Oct  8 17:01:25 ms-005-dns01 kernel: ESI: 00000002 EDI: f6b84494 EBP: c08a5fa8 ESP: c08a5fa4
Oct  8 17:01:25 ms-005-dns01 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Oct  8 17:01:25 ms-005-dns01 kernel: CR0: 8005003b CR2: b78b4000 CR3: 31df2000 CR4: 000006d0
Oct  8 17:01:25 ms-005-dns01 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Oct  8 17:01:25 ms-005-dns01 kernel: DR6: ffff0ff0 DR7: 00000400
Oct  8 17:01:25 ms-005-dns01 kernel: [<c05c0e8b>] ata_sff_interrupt+0x86/0xab
Oct  8 17:01:25 ms-005-dns01 kernel: [<c046548f>] handle_IRQ_event+0x2f/0x64
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0466431>] handle_edge_irq+0xb2/0xf4
Oct  8 17:01:25 ms-005-dns01 kernel: [<c046637f>] ? handle_edge_irq+0x0/0xf4
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0406e6e>] do_IRQ+0xc7/0xfe
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0405668>] common_interrupt+0x28/0x30
Oct  8 17:01:25 ms-005-dns01 kernel: [<c06ac166>] ? _spin_unlock_irqrestore+0x2d/0x38
Oct  8 17:01:25 ms-005-dns01 kernel: [<c05c08b2>] ata_sff_hsm_move+0x23a/0x681
Oct  8 17:01:25 ms-005-dns01 kernel: [<c05bf813>] ? ata_sff_check_status+0xb/0xd
Oct  8 17:01:25 ms-005-dns01 kernel: [<c05c0f2b>] ata_pio_task+0x7b/0x87
Oct  8 17:01:25 ms-005-dns01 kernel: [<c043c195>] run_workqueue+0x89/0x113
Oct  8 17:01:25 ms-005-dns01 kernel: [<c05c0eb0>] ? ata_pio_task+0x0/0x87
Oct  8 17:01:25 ms-005-dns01 kernel: [<c043c2eb>] worker_thread+0xcc/0xd8
Oct  8 17:01:25 ms-005-dns01 kernel: [<c043f046>] ? autoremove_wake_function+0x0/0x33
Oct  8 17:01:25 ms-005-dns01 kernel: [<c043c21f>] ? worker_thread+0x0/0xd8
Oct  8 17:01:25 ms-005-dns01 kernel: [<c043eda3>] kthread+0x3b/0x61
Oct  8 17:01:25 ms-005-dns01 kernel: [<c043ed68>] ? kthread+0x0/0x61
Oct  8 17:01:25 ms-005-dns01 kernel: [<c040590b>] kernel_thread_helper+0x7/0x10
Oct  8 17:01:25 ms-005-dns01 kernel: =======================
Oct  8 17:01:25 ms-005-dns01 kernel: BUG: soft lockup - CPU#0 stuck for 1179865901s! [swapper:0]
Oct  8 17:01:25 ms-005-dns01 kernel: Modules linked in: sco bridge stp bnep l2cap bluetooth sunrpc iptable_nat nf_nat iptable_mangle ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput i2c_piix4 virtio_balloon e1000 virtio_pci i2c_core joydev floppy pcspkr ata_generic pata_acpi [last unloaded: scsi_wait_scan]
Oct  8 17:01:25 ms-005-dns01 kernel:
Oct  8 17:01:25 ms-005-dns01 kernel: Pid: 0, comm: swapper Not tainted (2.6.27.41-170.2.117.fc10.i686 #1) Standard PC (i440FX + PIIX, 1996)
Oct  8 17:01:25 ms-005-dns01 kernel: EIP: 0060:[<c06ac166>] EFLAGS: 00000246 CPU: 0
Oct  8 17:01:25 ms-005-dns01 kernel: EIP is at _spin_unlock_irqrestore+0x2d/0x38
Oct  8 17:01:25 ms-005-dns01 kernel: EAX: 00000246 EBX: 00000246 ECX: 000000c0 EDX: 00000246
Oct  8 17:01:25 ms-005-dns01 kernel: ESI: f7826000 EDI: f1c6a700 EBP: c0884cf4 ESP: c0884cf0
Oct  8 17:01:25 ms-005-dns01 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Oct  8 17:01:25 ms-005-dns01 kernel: CR0: 8005003b CR2: 001688b0 CR3: 31fa2000 CR4: 000006d0
Oct  8 17:01:25 ms-005-dns01 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Oct  8 17:01:25 ms-005-dns01 kernel: DR6: ffff0ff0 DR7: 00000400
Oct  8 17:01:25 ms-005-dns01 kernel: [<f887d0ba>] e1000_xmit_frame+0xa6c/0xabe [e1000]
Oct  8 17:01:25 ms-005-dns01 kernel: [<c048ee1d>] ? __slab_alloc+0x56/0x45b
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0653cc6>] ? ip_route_input+0x3e/0xb90
Oct  8 17:01:25 ms-005-dns01 kernel: [<c06347a7>] dev_hard_start_xmit+0x1c2/0x231
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0672ac1>] ? arp_create+0x4d/0x219
Oct  8 17:01:25 ms-005-dns01 kernel: [<c064305e>] __qdisc_run+0xc7/0x1c5
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0632e9a>] qdisc_run+0x2c/0x2f
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0634c5b>] dev_queue_xmit+0x360/0x466
Oct  8 17:01:25 ms-005-dns01 kernel: [<c06722c8>] arp_xmit+0x3d/0x42
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0672cbf>] arp_send+0x32/0x37
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0672fd0>] arp_process+0x30c/0x523
Oct  8 17:01:25 ms-005-dns01 kernel: [<c048ee1d>] ? __slab_alloc+0x56/0x45b
Oct  8 17:01:25 ms-005-dns01 kernel: [<c041c024>] ? pvclock_clocksource_read+0x48/0xa3
Oct  8 17:01:25 ms-005-dns01 kernel: [<c041b6ff>] ? kvm_clock_read+0x16/0x18
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0444740>] ? getnstimeofday+0x54/0xe1
Oct  8 17:01:25 ms-005-dns01 kernel: [<c06732c9>] arp_rcv+0xd8/0xeb
Oct  8 17:01:25 ms-005-dns01 kernel: [<c063435b>] netif_receive_skb+0x371/0x390
Oct  8 17:01:25 ms-005-dns01 kernel: [<f887a3a8>] e1000_clean_rx_irq+0x298/0x348 [e1000]
Oct  8 17:01:25 ms-005-dns01 kernel: [<f887c269>] e1000_clean+0x2c2/0x481 [e1000]
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0632cd2>] net_rx_action+0xbc/0x1d5
Oct  8 17:01:25 ms-005-dns01 kernel: [<c043280b>] __do_softirq+0x84/0x109
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0432787>] ? __do_softirq+0x0/0x109
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0406f1c>] do_softirq+0x77/0xdb
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0466473>] ? handle_fasteoi_irq+0x0/0xc0
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0432472>] irq_exit+0x44/0x83
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0406e8e>] do_IRQ+0xe7/0xfe
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0405668>] common_interrupt+0x28/0x30
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0448102>] ? tick_nohz_restart_sched_tick+0x13f/0x14d
Oct  8 17:01:25 ms-005-dns01 kernel: [<c0403c88>] cpu_idle+0x128/0x134
Oct  8 17:01:25 ms-005-dns01 kernel: [<c069bd32>] rest_init+0x4e/0x50
Oct  8 17:01:25 ms-005-dns01 kernel: =======================
Oct  8 21:20:20 ms-005-dns01 kernel: BUG: soft lockup - CPU#0 stuck for 1232s! [crond:1959]
Oct  8 20:07:02 ms-005-dns01 kernel: Modules linked in: sco bridge stp bnep l2cap bluetooth sunrpc iptable_nat nf_nat iptable_mangle ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput i2c_piix4 virtio_balloon e1000 virtio_pci i2c_core joydev floppy pcspkr ata_generic pata_acpi [last unloaded: scsi_wait_scan]
Oct  8 20:07:02 ms-005-dns01 kernel:
Oct  8 20:07:02 ms-005-dns01 kernel: Pid: 1959, comm: crond Not tainted (2.6.27.41-170.2.117.fc10.i686 #1) Standard PC (i440FX + PIIX, 1996)
Oct  8 20:07:02 ms-005-dns01 kernel: EIP: 0060:[<c04f886e>] EFLAGS: 00000246 CPU: 0
Oct  8 20:07:02 ms-005-dns01 kernel: EIP is at avc_has_perm_noaudit+0x83/0x363
Oct  8 20:07:02 ms-005-dns01 kernel: EAX: 00000040 EBX: f65752d8 ECX: 00000006 EDX: 00000006
Oct  8 20:07:02 ms-005-dns01 kernel: ESI: f1e9df00 EDI: f1daee18 EBP: f1daedfc ESP: f1daeda8
Oct  8 20:07:02 ms-005-dns01 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Oct  8 20:07:02 ms-005-dns01 kernel: CR0: 8005003b CR2: b78b4000 CR3: 31e99000 CR4: 000006d0
Oct  8 20:07:02 ms-005-dns01 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Oct  8 20:07:02 ms-005-dns01 kernel: DR6: ffff0ff0 DR7: 00000400
Oct  8 20:07:02 ms-005-dns01 kernel: [<c04fa795>] ? inode_has_perm+0x5b/0x65
Oct  8 20:07:02 ms-005-dns01 kernel: [<c04f91f4>] avc_has_perm+0x22/0x43
Oct  8 20:07:02 ms-005-dns01 kernel: [<c04fa795>] inode_has_perm+0x5b/0x65
Oct  8 20:07:02 ms-005-dns01 kernel: [<c04a607d>] ? mntput_no_expire+0x1a/0xf7
Oct  8 20:07:02 ms-005-dns01 kernel: [<c0498ede>] ? mntput+0x13/0x15
Oct  8 20:07:02 ms-005-dns01 kernel: [<c0498fba>] ? path_put+0x15/0x18
Oct  8 20:07:02 ms-005-dns01 kernel: [<c04fa7ea>] dentry_has_perm+0x4b/0x53
Oct  8 20:07:02 ms-005-dns01 kernel: [<c04fa835>] selinux_inode_getattr+0x17/0x1c
Oct  8 20:07:02 ms-005-dns01 kernel: [<c04f671a>] security_inode_getattr+0x22/0x29
Oct  8 20:07:02 ms-005-dns01 kernel: [<c0495a14>] vfs_getattr+0x1f/0x58
Oct  8 20:07:02 ms-005-dns01 kernel: [<c0495bf9>] vfs_stat_fd+0x2d/0x40
Oct  8 20:07:02 ms-005-dns01 kernel: [<c0495cbb>] vfs_stat+0x11/0x13
Oct  8 20:07:02 ms-005-dns01 kernel: [<c0495cd1>] sys_stat64+0x14/0x2b
Oct  8 20:07:02 ms-005-dns01 kernel: [<c0461eff>] ? audit_syscall_entry+0xf9/0x123
Oct  8 20:07:02 ms-005-dns01 kernel: [<c040b098>] ? syscall_trace_enter+0xb5/0xc9
Oct  8 20:07:02 ms-005-dns01 kernel: [<c0404c8a>] syscall_call+0x7/0xb
Oct  8 20:07:02 ms-005-dns01 kernel: =======================
Oct  8 21:01:01 ms-005-dns01 kernel: BUG: soft lockup - CPU#1 stuck for 1247s! [kjournald:577]
Oct  8 21:01:01 ms-005-dns01 kernel: Modules linked in: sco bridge stp bnep l2cap bluetooth sunrpc iptable_nat nf_nat iptable_mangle ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput i2c_piix4 virtio_balloon e1000 virtio_pci i2c_core joydev floppy pcspkr ata_generic pata_acpi [last unloaded: scsi_wait_scan]
Oct  8 21:01:01 ms-005-dns01 kernel:
Oct  8 21:01:01 ms-005-dns01 kernel: Pid: 577, comm: kjournald Not tainted (2.6.27.41-170.2.117.fc10.i686 #1) Standard PC (i440FX + PIIX, 1996)
Oct  8 21:01:01 ms-005-dns01 kernel: EIP: 0060:[<c06ac166>] EFLAGS: 00000287 CPU: 1
Oct  8 21:01:01 ms-005-dns01 kernel: EIP is at _spin_unlock_irqrestore+0x2d/0x38
Oct  8 21:01:01 ms-005-dns01 kernel: EAX: 00000287 EBX: 00000287 ECX: c059f475 EDX: 00000287
Oct  8 21:01:01 ms-005-dns01 kernel: ESI: f0480800 EDI: f6b9c800 EBP: f6579dd0 ESP: f6579dcc
Oct  8 21:01:01 ms-005-dns01 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Oct  8 21:01:01 ms-005-dns01 kernel: CR0: 8005003b CR2: 00280328 CR3: 306a5000 CR4: 000006d0
Oct  8 21:01:01 ms-005-dns01 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Oct  8 21:01:01 ms-005-dns01 kernel: DR6: ffff0ff0 DR7: 00000400
Oct  8 21:01:01 ms-005-dns01 kernel: [<c059f713>] scsi_dispatch_cmd+0x19b/0x20b
Oct  8 21:01:01 ms-005-dns01 kernel: [<c05a3f34>] scsi_request_fn+0x2f4/0x3e2
Oct  8 21:01:01 ms-005-dns01 kernel: [<c050fd54>] __generic_unplug_device+0x2a/0x2d
Oct  8 21:01:01 ms-005-dns01 kernel: [<c050ffbc>] generic_unplug_device+0x2c/0x4a
Oct  8 21:01:01 ms-005-dns01 kernel: [<c050ed03>] blk_unplug+0x26/0x2c
Oct  8 21:01:01 ms-005-dns01 kernel: [<c060c494>] dm_table_unplug_all+0x21/0x31
Oct  8 21:01:01 ms-005-dns01 kernel: [<c060c0e4>] dm_unplug_all+0x1a/0x24
Oct  8 21:01:01 ms-005-dns01 kernel: [<c050ed03>] blk_unplug+0x26/0x2c
Oct  8 21:01:01 ms-005-dns01 kernel: [<c050ed14>] blk_backing_dev_unplug+0xb/0xd
Oct  8 21:01:01 ms-005-dns01 kernel: [<c04b05c4>] sync_buffer+0x2e/0x37
Oct  8 21:01:01 ms-005-dns01 kernel: [<c06ab09e>] __wait_on_bit+0x36/0x5d
Oct  8 21:01:01 ms-005-dns01 kernel: [<c04b0596>] ? sync_buffer+0x0/0x37
Oct  8 21:01:01 ms-005-dns01 kernel: [<c06ab170>] out_of_line_wait_on_bit+0xab/0xb3
Oct  8 21:01:01 ms-005-dns01 kernel: [<c04b0596>] ? sync_buffer+0x0/0x37
Oct  8 21:01:01 ms-005-dns01 kernel: [<c043f079>] ? wake_bit_function+0x0/0x43
Oct  8 21:01:01 ms-005-dns01 kernel: [<c04b050c>] __wait_on_buffer+0x29/0x2f
Oct  8 21:01:01 ms-005-dns01 kernel: [<c04e2df5>] wait_on_buffer+0x37/0x3a
Oct  8 21:01:01 ms-005-dns01 kernel: [<c04e330e>] journal_commit_transaction+0x474/0xc92
Oct  8 21:01:01 ms-005-dns01 kernel: [<c04369e2>] ? lock_timer_base+0x1f/0x3e
Oct  8 21:01:01 ms-005-dns01 kernel: [<c0436a49>] ? try_to_del_timer_sync+0x48/0x4f
Oct  8 21:01:01 ms-005-dns01 kernel: [<c04e6212>] kjournald+0xbb/0x1ee
Oct  8 21:01:01 ms-005-dns01 kernel: [<c043f046>] ? autoremove_wake_function+0x0/0x33
Oct  8 21:01:01 ms-005-dns01 kernel: [<c04e6157>] ? kjournald+0x0/0x1ee
Oct  8 21:01:01 ms-005-dns01 kernel: [<c043eda3>] kthread+0x3b/0x61
Oct  8 21:01:01 ms-005-dns01 kernel: [<c043ed68>] ? kthread+0x0/0x61
Oct  8 21:01:01 ms-005-dns01 kernel: [<c040590b>] kernel_thread_helper+0x7/0x10
Oct  8 21:01:01 ms-005-dns01 kernel: =======================
Oct  8 21:35:18 ms-005-dns01 kernel: BUG: soft lockup - CPU#1 stuck for 1423s! [crond:1959]
Oct  8 21:35:18 ms-005-dns01 kernel: Modules linked in: sco bridge stp bnep l2cap bluetooth sunrpc iptable_nat nf_nat iptable_mangle ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput i2c_piix4 virtio_balloon e1000 virtio_pci i2c_core joydev floppy pcspkr ata_generic pata_acpi [last unloaded: scsi_wait_scan]
Oct  8 21:35:18 ms-005-dns01 kernel:
Oct  8 21:35:18 ms-005-dns01 kernel: Pid: 1959, comm: crond Not tainted (2.6.27.41-170.2.117.fc10.i686 #1) Standard PC (i440FX + PIIX, 1996)
Oct  8 21:35:18 ms-005-dns01 kernel: EIP: 0073:[<0019da06>] EFLAGS: 00000256 CPU: 1
Oct  8 21:35:18 ms-005-dns01 kernel: EIP is at 0x19da06
Oct  8 21:35:18 ms-005-dns01 kernel: EAX: 0000076c EBX: 0027fff4 ECX: 0000077f EDX: 00000118
Oct  8 21:35:18 ms-005-dns01 kernel: ESI: 00000640 EDI: 00000013 EBP: bf9d8900 ESP: bf9d88d8
Oct  8 21:35:18 ms-005-dns01 kernel: DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
Oct  8 21:35:18 ms-005-dns01 kernel: CR0: 8005003b CR2: bf9d069c CR3: 31e99000 CR4: 000006d0
Oct  8 21:35:18 ms-005-dns01 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Oct  8 21:35:18 ms-005-dns01 kernel: DR6: ffff0ff0 DR7: 00000400
Oct  8 21:35:18 ms-005-dns01 kernel: =======================
Oct  8 22:11:18 ms-005-dns01 kernel: BUG: soft lockup - CPU#1 stuck for 2012s! [crond:1959]
Oct  8 22:11:18 ms-005-dns01 kernel: Modules linked in: sco bridge stp bnep l2cap bluetooth sunrpc iptable_nat nf_nat iptable_mangle ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput i2c_piix4 virtio_balloon e1000 virtio_pci i2c_core joydev floppy pcspkr ata_generic pata_acpi [last unloaded: scsi_wait_scan]
Oct  8 22:11:18 ms-005-dns01 kernel:
Oct  8 22:11:18 ms-005-dns01 kernel: Pid: 1959, comm: crond Not tainted (2.6.27.41-170.2.117.fc10.i686 #1) Standard PC (i440FX + PIIX, 1996)
Oct  8 22:11:18 ms-005-dns01 kernel: EIP: 0073:[<0804ae08>] EFLAGS: 00000286 CPU: 1
Oct  8 22:11:18 ms-005-dns01 kernel: EIP is at 0x804ae08
Oct  8 22:11:18 ms-005-dns01 kernel: EAX: a183f606 EBX: a183f66c ECX: 00000000 EDX: 00000000
Oct  8 22:11:18 ms-005-dns01 kernel: ESI: a183f606 EDI: 00000067 EBP: bf9d8a48 ESP: bf9d8970
Oct  8 22:11:18 ms-005-dns01 kernel: DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
Oct  8 22:11:18 ms-005-dns01 kernel: CR0: 8005003b CR2: bf9d069c CR3: 31e99000 CR4: 000006d0
Oct  8 22:11:18 ms-005-dns01 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Oct  8 22:11:18 ms-005-dns01 kernel: DR6: ffff0ff0 DR7: 00000400
Oct  8 22:11:18 ms-005-dns01 kernel: =======================
Oct  8 23:15:01 ms-005-dns01 kernel: BUG: soft lockup - CPU#1 stuck for 419s! [crond:1959]
Oct  8 23:15:01 ms-005-dns01 kernel: Modules linked in: sco bridge stp bnep l2cap bluetooth sunrpc iptable_nat nf_nat iptable_mangle ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput i2c_piix4 virtio_balloon e1000 virtio_pci i2c_core joydev floppy pcspkr ata_generic pata_acpi [last unloaded: scsi_wait_scan]
Oct  8 23:15:01 ms-005-dns01 kernel:
Oct  8 23:15:01 ms-005-dns01 kernel: Pid: 1959, comm: crond Not tainted (2.6.27.41-170.2.117.fc10.i686 #1) Standard PC (i440FX + PIIX, 1996)
Oct  8 23:15:01 ms-005-dns01 kernel: EIP: 0073:[<00186e14>] EFLAGS: 00000202 CPU: 1
Oct  8 23:15:01 ms-005-dns01 kernel: EIP is at 0x186e14
Oct  8 23:15:01 ms-005-dns01 kernel: EAX: 0897740c EBX: 0027fff4 ECX: 0897740c EDX: ff535241
Oct  8 23:15:01 ms-005-dns01 kernel: ESI: 08976560 EDI: 00000003 EBP: bf9d88b0 ESP: bf9d8890
Oct  8 23:15:01 ms-005-dns01 kernel: DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
Oct  8 23:15:01 ms-005-dns01 kernel: CR0: 8005003b CR2: b78b4000 CR3: 31e99000 CR4: 000006d0
Oct  8 23:15:01 ms-005-dns01 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Oct  8 23:15:01 ms-005-dns01 kernel: DR6: ffff0ff0 DR7: 00000400
Oct  8 23:15:01 ms-005-dns01 kernel: =======================
Oct  8 23:55:36 ms-005-dns01 kernel: BUG: soft lockup - CPU#1 stuck for 2308s! [crond:1959]
Oct  8 23:55:36 ms-005-dns01 kernel: Modules linked in: sco bridge stp bnep l2cap bluetooth sunrpc iptable_nat nf_nat iptable_mangle ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput i2c_piix4 virtio_balloon e1000 virtio_pci i2c_core joydev floppy pcspkr ata_generic pata_acpi [last unloaded: scsi_wait_scan]
Oct  8 23:55:36 ms-005-dns01 kernel:
Oct  8 23:55:36 ms-005-dns01 kernel: Pid: 1959, comm: crond Not tainted (2.6.27.41-170.2.117.fc10.i686 #1) Standard PC (i440FX + PIIX, 1996)
Oct  8 23:55:36 ms-005-dns01 kernel: EIP: 0060:[<c05d4fdc>] EFLAGS: 00000246 CPU: 1
Oct  8 23:55:36 ms-005-dns01 kernel: EIP is at usb_hcd_irq+0x94/0xa3
Oct  8 23:55:36 ms-005-dns01 kernel: EAX: 00000246 EBX: f6411600 ECX: f64116ec EDX: f6417000
Oct  8 23:55:36 ms-005-dns01 kernel: ESI: 00000001 EDI: 00000246 EBP: c08a5fc0 ESP: c08a5fb4
Oct  8 23:55:36 ms-005-dns01 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Oct  8 23:55:36 ms-005-dns01 kernel: CR0: 8005003b CR2: b78b4000 CR3: 31e99000 CR4: 000006d0
Oct  8 23:55:36 ms-005-dns01 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Oct  8 23:55:36 ms-005-dns01 kernel: DR6: ffff0ff0 DR7: 00000400
Oct  8 23:55:36 ms-005-dns01 kernel: [<c046548f>] handle_IRQ_event+0x2f/0x64
Oct  8 23:55:36 ms-005-dns01 kernel: [<c04664f8>] handle_fasteoi_irq+0x85/0xc0
Oct  8 23:55:36 ms-005-dns01 kernel: [<c0466473>] ? handle_fasteoi_irq+0x0/0xc0
Oct  8 23:55:36 ms-005-dns01 kernel: [<c0406e6e>] do_IRQ+0xc7/0xfe
Oct  8 23:55:36 ms-005-dns01 kernel: [<c0405668>] common_interrupt+0x28/0x30
Oct  8 23:55:36 ms-005-dns01 kernel: =======================

 

[vkhera]

Some people have mentioned here that for stability you should prefer to use virtio disk drivers in the client with qcow2 disk images. Try using virtio or RAW disk image format (or both).

I had one VM crash in a very weird way that corrupted the disk (necessitating a restore from backup). I switched from qcow2 storage file to a ZFS RAW image and have not seen any issues since. That particular VM is an appliance software that does not have a virtio disk driver.

 

[gob]

Thanks for the reply vkhera.
I'll give that a try next time it bombs out!
Typically it's been stable today.