VM's consistanyly giving SATA or VIRTO errors - VM's then go RO

[Jonathon_ND]

Hey Everyone,

We have a single Host setup on a DELL 630 with SAS drives in RAID 10.

We are having some high use VM's and the client machines show disk errors randomly and then drop into read only or loose the information all together.

We have tried disks as VIRTO and SATA for CENTOS and Ubuntu.

PX 3.4

Can anyone please help us shed some light as to what is happening? and what the smoking gun is?

 

[wolfgang]

Hi,
what is the error output? on the host and vm?

 

[Jonathon_ND]

Hi Guys,

Below is the errors we are getting.
We appreciate any assistance or advise.

The disk is setup as SATA qemu.

Nov 4 16:09:10 db kernel: [ 1032.830219] ata3.00: status: { DRDY }
Nov 4 16:09:10 db kernel: [ 1032.831142] ata3.00: failed command: WRITE FPDMA QUEUED
Nov 4 16:09:10 db kernel: [ 1032.832193] ata3.00: cmd 61/08:30:00:b1:c8/00:00:08:00:00/40 tag 6 ncq 4096 out
Nov 4 16:09:10 db kernel: [ 1032.832193] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)

 

[sigxcpu]

What are the errors when you are using virtio?

 

[Jonathon_ND]

We would get

echo 0 > /proc/sys/kernel/hang_tast_timeout_secs
task apt-get:2426 blocked for more than 120 seconds - not tainted 3.19.0-15-generic

and the system would just hang

 

[anigwei]

Hi,

In Proxmox 3 I have a similar behavior in one VM with SATA hard disk configured.

I Usually configure VirtIO, but this VM was P2V'ed and I choose SATA to avoid problems.

The other VM ara working flawlessly. The host does'nt have any disk error.

This is the lastlog:

Thank you!!


[3287660.832361] ata3.00: exception Emask 0x0 SAct 0x78000000 SErr 0x0 action 0x6 frozen
[3287660.832487] ata3.00: failed command: WRITE FPDMA QUEUED
[3287660.832586] ata3.00: cmd 61/08:d8:80:e8:ce/00:00:01:00:00/40 tag 27 ncq 4096 out
[3287660.832586] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[3287660.832775] ata3.00: status: { DRDY }
[3287660.832828] ata3.00: failed command: WRITE FPDMA QUEUED
[3287660.832902] ata3.00: cmd 61/08:e0:00:e9:ce/00:00:01:00:00/40 tag 28 ncq 4096 out
[3287660.832902] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[3287660.833088] ata3.00: status: { DRDY }
[3287660.833140] ata3.00: failed command: WRITE FPDMA QUEUED
[3287660.833214] ata3.00: cmd 61/08:e8:38:e9:ce/00:00:01:00:00/40 tag 29 ncq 4096 out
[3287660.833214] res 40/00:01:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[3287660.833399] ata3.00: status: { DRDY }
[3287660.833451] ata3.00: failed command: WRITE FPDMA QUEUED
[3287660.833536] ata3.00: cmd 61/08:f0:00:e9:cf/00:00:01:00:00/40 tag 30 ncq 4096 out
[3287660.833536] res 40/00:01:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[3287660.833721] ata3.00: status: { DRDY }
[3287660.833796] ata3: hard resetting link
[3287668.848805] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[3287668.850065] ata3.00: configured for UDMA/100
[3287668.850075] ata3.00: device reported invalid CHS sector 0
[3287668.850080] ata3.00: device reported invalid CHS sector 0
[3287668.850084] ata3.00: device reported invalid CHS sector 0
[3287668.850089] ata3.00: device reported invalid CHS sector 0
[3287668.850125] ata3: EH complete
[3287888.864356] ata3.00: exception Emask 0x0 SAct 0x60 SErr 0x0 action 0x6 frozen
[3287888.864484] ata3.00: failed command: WRITE FPDMA QUEUED
[3287888.864562] ata3.00: cmd 61/10:28:78:b4:d5/00:00:02:00:00/40 tag 5 ncq 8192 out
[3287888.864562] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[3287888.864747] ata3.00: status: { DRDY }
[3287888.864800] ata3.00: failed command: WRITE FPDMA QUEUED
[3287888.864874] ata3.00: cmd 61/08:30:b0:e5:d7/00:00:02:00:00/40 tag 6 ncq 4096 out
[3287888.864874] res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[3287888.865059] ata3.00: status: { DRDY }
[3287888.865133] ata3: hard resetting link
[3287902.144797] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[3287902.145799] ata3.00: configured for UDMA/100
[3287902.145818] ata3.00: device reported invalid CHS sector 0
[3287902.145825] ata3.00: device reported invalid CHS sector 0
[3287902.145861] ata3: EH complete
[3287943.840294] ata3.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[3287943.840416] ata3.00: failed command: WRITE FPDMA QUEUED
[3287943.840543] ata3.00: cmd 61/28:00:f8:b4:d5/00:00:02:00:00/40 tag 0 ncq 20480 out
[3287943.840543] res 40/00:01:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[3287943.840731] ata3.00: status: { DRDY }
[3287943.840808] ata3: hard resetting link
[3287959.216081] [sched_delayed] sched: RT throttling activated
[3287959.525006] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[3287959.526424] ata3.00: configured for UDMA/100
[3287959.526435] ata3.00: device reported invalid CHS sector 0
[3287959.526489] ata3: EH complete
[3288226.816474] ata3.00: NCQ disabled due to excessive errors
[3288226.816486] ata3.00: exception Emask 0x0 SAct 0x800 SErr 0x0 action 0x6 frozen
[3288226.816592] ata3.00: failed command: WRITE FPDMA QUEUED
[3288226.816669] ata3.00: cmd 61/48:58:e0:b5:d5/00:00:02:00:00/40 tag 11 ncq 36864 out
[3288226.816669] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[3288226.816857] ata3.00: status: { DRDY }
[3288226.816983] ata3: hard resetting link
[3288303.740981] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[3288303.741696] ata3.00: configured for UDMA/100
[3288303.741706] ata3.00: device reported invalid CHS sector 0
[3288303.741750] ata3: EH complete