proxmox web gui problems

[sarawoot]

i can not acces to my proxmox web interface.

● pvesr.service - Proxmox VE replication runner
Loaded: loaded (/lib/systemd/system/pvesr.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2021-07-21 17:04:00 +07; 22s ago
Process: 31788 ExecStart=/usr/bin/pvesr run --mail 1 (code=exited, status=30)
Main PID: 31788 (code=exited, status=30)

Jul 21 17:04:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Jul 21 17:04:00 proxmox pvesr[31788]: can't lock file '/var/lib/pve-manager/pve-replication-state.lck' - can't open file - Read-only file system
Jul 21 17:04:00 proxmox systemd[1]: pvesr.service: Main process exited, code=exited, status=30/n/a
Jul 21 17:04:00 proxmox systemd[1]: pvesr.service: Failed with result 'exit-code'.
Jul 21 17:04:00 proxmox systemd[1]: Failed to start Proxmox VE replication runner.

● pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-07-09 00:04:10 +07; 1 weeks 5 days ago
Process: 1105 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
Main PID: 1114 (pvestatd)
Tasks: 1 (limit: 4915)
Memory: 64.5M
CGroup: /system.slice/pvestatd.service
└─1114 pvestatd

Jul 21 17:03:49 proxmox pvestatd[1114]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system
Jul 21 17:03:58 proxmox pvestatd[1114]: authkey rotation error: error during cfs-locked 'authkey' operation: got lock request timeout
Jul 21 17:03:58 proxmox pvestatd[1114]: status update time (9.309 seconds)
Jul 21 17:03:59 proxmox pvestatd[1114]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system
Jul 21 17:04:08 proxmox pvestatd[1114]: authkey rotation error: error during cfs-locked 'authkey' operation: got lock request timeout
Jul 21 17:04:08 proxmox pvestatd[1114]: status update time (9.293 seconds)
Jul 21 17:04:09 proxmox pvestatd[1114]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system
Jul 21 17:04:18 proxmox pvestatd[1114]: authkey rotation error: error during cfs-locked 'authkey' operation: got lock request timeout
Jul 21 17:04:18 proxmox pvestatd[1114]: status update time (9.311 seconds)
Jul 21 17:04:19 proxmox pvestatd[1114]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system

● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-07-21 10:09:22 +07; 6h ago
Process: 6074 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=1/FAILURE)
Process: 6076 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 6100 (pveproxy)
Tasks: 4 (limit: 4915)
Memory: 138.3M
CGroup: /system.slice/pveproxy.service
├─ 6100 pveproxy
├─31889 pveproxy worker
├─31890 pveproxy worker
└─31891 pveproxy worker

Jul 21 17:04:22 proxmox pveproxy[31889]: unable to open log file '/var/log/pveproxy/access.log' - Read-only file system
Jul 21 17:04:22 proxmox pveproxy[6100]: worker 31865 finished
Jul 21 17:04:22 proxmox pveproxy[6100]: starting 1 worker(s)
Jul 21 17:04:22 proxmox pveproxy[6100]: worker 31890 started
Jul 21 17:04:22 proxmox pveproxy[31890]: unable to open log file '/var/log/pveproxy/access.log' - Read-only file system
Jul 21 17:04:22 proxmox pveproxy[31866]: worker exit
Jul 21 17:04:22 proxmox pveproxy[6100]: worker 31866 finished
Jul 21 17:04:22 proxmox pveproxy[6100]: starting 1 worker(s)
Jul 21 17:04:22 proxmox pveproxy[6100]: worker 31891 started
Jul 21 17:04:22 proxmox pveproxy[31891]: unable to open log file '/var/log/pveproxy/access.log' - Read-only file system

● pve-ha-lrm.service - PVE Local HA Resource Manager Daemon
Loaded: loaded (/lib/systemd/system/pve-ha-lrm.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-07-09 00:04:12 +07; 1 weeks 5 days ago
Process: 1151 ExecStart=/usr/sbin/pve-ha-lrm start (code=exited, status=0/SUCCESS)
Main PID: 1154 (pve-ha-lrm)
Tasks: 1 (limit: 4915)
Memory: 26.5M
CGroup: /system.slice/pve-ha-lrm.service
└─1154 pve-ha-lrm

Jul 21 17:03:38 proxmox pve-ha-lrm[1154]: unable to write lrm status file - unable to delete old temp file: Input/output error
Jul 21 17:03:43 proxmox pve-ha-lrm[1154]: unable to write lrm status file - unable to delete old temp file: Input/output error
Jul 21 17:03:48 proxmox pve-ha-lrm[1154]: unable to write lrm status file - unable to delete old temp file: Input/output error
Jul 21 17:03:53 proxmox pve-ha-lrm[1154]: unable to write lrm status file - unable to delete old temp file: Input/output error
Jul 21 17:03:58 proxmox pve-ha-lrm[1154]: unable to write lrm status file - unable to delete old temp file: Input/output error
Jul 21 17:04:03 proxmox pve-ha-lrm[1154]: unable to write lrm status file - unable to delete old temp file: Input/output error
Jul 21 17:04:08 proxmox pve-ha-lrm[1154]: unable to write lrm status file - unable to delete old temp file: Input/output error
Jul 21 17:04:13 proxmox pve-ha-lrm[1154]: unable to write lrm status file - unable to delete old temp file: Input/output error
Jul 21 17:04:18 proxmox pve-ha-lrm[1154]: unable to write lrm status file - unable to delete old temp file: Input/output error
Jul 21 17:04:23 proxmox pve-ha-lrm[1154]: unable to write lrm status file - unable to delete old temp file: Input/output error

anyone have an idea pelase?

thank you

 

[oguz]

hi,

- can't open file - Read-only file system

could you post the outputs from:
* df
* mount

when did this start happening? did you do anything before?

you should also check journalctl and /var/log/syslog, there might be hints there pointing to the root cause of the issue

 

[sarawoot]

hi,


could you post the outputs from:
* df
* mount

when did this start happening? did you do anything before?

you should also check journalctl and /var/log/syslog, there might be hints there pointing to the root cause of the issue

outputs from: df, mount

root@proxmox:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 16357924 0 16357924 0% /dev
tmpfs 3277196 329076 2948120 11% /run
/dev/mapper/pve-root 98559220 72662212 20847460 78% /
tmpfs 16385964 21840 16364124 1% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 16385964 0 16385964 0% /sys/fs/cgroup
/dev/sdb2 523248 312 522936 1% /boot/efi
/dev/fuse 30720 20 30700 1% /etc/pve
tmpfs 3277192 0 3277192 0% /run/user/0

root@proxmox:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16357924k,nr_inodes=4089481,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3277196k,mode=755)
/dev/mapper/pve-root on / type ext4 (ro,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=22009)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sdb2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3277192k,mode=700)

 

[oguz]

/dev/mapper/pve-root on / type ext4 (ro,relatime,errors=remount-ro)

here it seems the pve-root is mounted readonly (ro). for anything to work it should be rw (read-write) mounted. since you have errors=remount-ro my guess is that your disk or filesystem had an error and remounted in read-only mode to prevent corruption.

so you should really check the journal and syslog, dmesg might also have error entries. your disk might be at fault here so running smartctl -a /dev/foo can show that (replace /dev/foo with the actual name of your disk, you can find that from lsblk output)

also a good idea would be to make backups to an external disk or network storage so in case something goes wrong you still have your data.

 

[sarawoot]

here it seems the pve-root is mounted readonly (ro). for anything to work it should be rw (read-write) mounted. since you have errors=remount-ro my guess is that your disk or filesystem had an error and remounted in read-only mode to prevent corruption.

so you should really check the journal and syslog, dmesg might also have error entries. your disk might be at fault here so running smartctl -a /dev/foo can show that (replace /dev/foo with the actual name of your disk, you can find that from lsblk output)

also a good idea would be to make backups to an external disk or network storage so in case something goes wrong you still have your data.

syslog :

Jul 13 20:59:00 proxmox kernel: [420912.370430] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Jul 13 20:59:00 proxmox kernel: [420912.370436] ata2.00: irq_stat 0x40000001
Jul 13 20:59:00 proxmox kernel: [420912.370440] ata2.00: failed command: READ DMA
Jul 13 20:59:00 proxmox kernel: [420912.370444] ata2.00: cmd c8/00:00:00:17:f0/00:00:00:00:00/e9 tag 15 dma 131072 in
Jul 13 20:59:00 proxmox kernel: [420912.370444] res 51/40:40:c0:17:f0/00:00:00:00:00/e9 Emask 0x9 (media error)
Jul 13 20:59:00 proxmox kernel: [420912.370449] ata2.00: status: { DRDY ERR }
Jul 13 20:59:00 proxmox kernel: [420912.370450] ata2.00: error: { UNC


/dev/sda :

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.101-1-pve] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: WD Blue and Green SSDs
Device Model: WDC WDS100T2G0A-00JH30
Serial Number: 20453K451410
LU WWN Device Id: 5 001b44 4a7efea7e
Firmware Version: UH510000
User Capacity: 1,000,207,286,272 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Jul 22 09:22:47 2021 +07
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x15) SMART execute Offline immediate.
No Auto Offline data collection support.
Abort Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 182) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 3943
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 91
165 Block_Erase_Count 0x0032 100 100 000 Old_age Always - 214
166 Minimum_PE_Cycles_TLC 0x0032 100 100 --- Old_age Always - 1
167 Max_Bad_Blocks_per_Die 0x0032 100 100 --- Old_age Always - 0
168 Maximum_PE_Cycles_TLC 0x0032 100 100 --- Old_age Always - 3
169 Total_Bad_Blocks 0x0032 100 100 --- Old_age Always - 1146
170 Grown_Bad_Blocks 0x0032 100 100 --- Old_age Always - 0
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Average_PE_Cycles_TLC 0x0032 100 100 000 Old_age Always - 1
174 Unexpected_Power_Loss 0x0032 100 100 000 Old_age Always - 73
184 End-to-End_Error 0x0032 100 100 --- Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 --- Old_age Always - 0
194 Temperature_Celsius 0x0022 055 074 000 Old_age Always - 45 (Min/Max 22/74)
199 UDMA_CRC_Error_Count 0x0032 100 100 --- Old_age Always - 0
230 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0x003d0014003d
232 Available_Reservd_Space 0x0033 100 100 005 Pre-fail Always - 100
233 NAND_GB_Written_TLC 0x0032 100 100 --- Old_age Always - 1559
234 NAND_GB_Written_SLC 0x0032 100 100 000 Old_age Always - 3347
241 Host_Writes_GiB 0x0030 100 100 000 Old_age Offline - 1563
242 Host_Reads_GiB 0x0030 100 100 000 Old_age Offline - 4204
244 Temp_Throttle_Status 0x0032 000 100 --- Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

Selective Self-tests/Logging not supported


/dev/sdb

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.101-1-pve] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: WD Blue and Green SSDs
Device Model: WDC WDS100T2G0A-00JH30
Serial Number: 202981806238
LU WWN Device Id: 5 001b44 8b53fd973
Firmware Version: UH510000
User Capacity: 1,000,207,286,272 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Jul 22 09:23:51 2021 +07
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x15) SMART execute Offline immediate.
No Auto Offline data collection support.
Abort Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 182) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 3841
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 33
165 Block_Erase_Count 0x0032 100 100 000 Old_age Always - 10466
166 Minimum_PE_Cycles_TLC 0x0032 100 100 --- Old_age Always - 5
167 Max_Bad_Blocks_per_Die 0x0032 100 100 --- Old_age Always - 0
168 Maximum_PE_Cycles_TLC 0x0032 100 100 --- Old_age Always - 20
169 Total_Bad_Blocks 0x0032 100 100 --- Old_age Always - 1160
170 Grown_Bad_Blocks 0x0032 100 100 --- Old_age Always - 0
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Average_PE_Cycles_TLC 0x0032 100 100 000 Old_age Always - 5
174 Unexpected_Power_Loss 0x0032 100 100 000 Old_age Always - 34
184 End-to-End_Error 0x0032 100 100 --- Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 2435
188 Command_Timeout 0x0032 100 100 --- Old_age Always - 0
194 Temperature_Celsius 0x0022 052 066 000 Old_age Always - 48 (Min/Max 28/66)
199 UDMA_CRC_Error_Count 0x0032 100 100 --- Old_age Always - 0
230 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0x1d5a01001d5a
232 Available_Reservd_Space 0x0033 100 100 005 Pre-fail Always - 100
233 NAND_GB_Written_TLC 0x0032 100 100 --- Old_age Always - 5391
234 NAND_GB_Written_SLC 0x0032 100 100 000 Old_age Always - 163545
241 Host_Writes_GiB 0x0030 100 100 000 Old_age Offline - 74226
242 Host_Reads_GiB 0x0030 100 100 000 Old_age Offline - 11072
244 Temp_Throttle_Status 0x0032 000 100 --- Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

Selective Self-tests/Logging not supported

 

[oguz]

syslog :

Jul 13 20:59:00 proxmox kernel: [420912.370430] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Jul 13 20:59:00 proxmox kernel: [420912.370436] ata2.00: irq_stat 0x40000001
Jul 13 20:59:00 proxmox kernel: [420912.370440] ata2.00: failed command: READ DMA
Jul 13 20:59:00 proxmox kernel: [420912.370444] ata2.00: cmd c8/00:00:00:17:f0/00:00:00:00:00/e9 tag 15 dma 131072 in
Jul 13 20:59:00 proxmox kernel: [420912.370444] res 51/40:40:c0:17:f0/00:00:00:00:00/e9 Emask 0x9 (media error)
Jul 13 20:59:00 proxmox kernel: [420912.370449] ata2.00: status: { DRDY ERR }
Jul 13 20:59:00 proxmox kernel: [420912.370450] ata2.00: error: { UNC

doesn't look too good. despite the PASSED from smartctl your drive might be starting to fail.

can you post the rest of dmesg output with relevant error messages?

dmesg | grep ata -C 1

 

[sarawoot]

doesn't look too good. despite the PASSED from smartctl your drive might be starting to fail.

can you post the rest of dmesg output with relevant error messages?

dmesg | grep ata -C 1

dmesg output :

[1189456.399662] blk_update_request: I/O error, dev sdb, sector 16350680 op 0x0READ) flags 0x0 phys_seg 1 prio class 0
[1189456.399688] ata2: EH complete
[1189456.481151] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189456.481155] ata2.00: irq_stat 0x40000001
[1189456.481158] ata2.00: failed command: READ DMA
[1189456.481161] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 26 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189456.481164] ata2.00: status: { DRDY ERR }
[1189456.481165] ata2.00: error: { UNC }
[1189456.487225] ata2.00: configured for UDMA/133
[1189456.487239] sd 1:0:0:0: [sdb] tag#26 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
--
[1189456.487248] blk_update_request: I/O error, dev sdb, sector 16350680 op 0x0READ) flags 0x0 phys_seg 1 prio class 0
[1189456.487271] ata2: EH complete
[1189456.569139] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189456.569145] ata2.00: irq_stat 0x40000001
[1189456.569148] ata2.00: failed command: READ DMA
[1189456.569152] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 10 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189456.569156] ata2.00: status: { DRDY ERR }
[1189456.569158] ata2.00: error: { UNC }
[1189456.575241] ata2.00: configured for UDMA/133
[1189456.575259] sd 1:0:0:0: [sdb] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
--
[1189456.575269] blk_update_request: I/O error, dev sdb, sector 16350680 op 0x0READ) flags 0x0 phys_seg 1 prio class 0
[1189456.575298] ata2: EH complete
[1189456.665145] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189456.665152] ata2.00: irq_stat 0x40000001
[1189456.665156] ata2.00: failed command: READ DMA
[1189456.665160] ata2.00: cmd c8/00:08:c0:a1:76/00:00:00:00:00/e9 tag 13 dma 4096 in
res 51/40:08:c0:a1:76/00:00:00:00:00/e9 Emask 0x9 (media error)
[1189456.665165] ata2.00: status: { DRDY ERR }
[1189456.665167] ata2.00: error: { UNC }
[1189456.671457] ata2.00: configured for UDMA/133
[1189456.671470] sd 1:0:0:0: [sdb] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
--
[1189456.671479] blk_update_request: I/O error, dev sdb, sector 158769600 op 0x0READ) flags 0x0 phys_seg 1 prio class 0
[1189456.671502] ata2: EH complete
[1189456.765162] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189456.765176] ata2.00: irq_stat 0x40000001
[1189456.765179] ata2.00: failed command: READ DMA
[1189456.765181] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 24 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189456.765184] ata2.00: status: { DRDY ERR }
[1189456.765185] ata2.00: error: { UNC }
[1189456.771287] ata2.00: configured for UDMA/133
[1189456.771303] ata2: EH complete
[1189456.857158] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189456.857163] ata2.00: irq_stat 0x40000001
[1189456.857165] ata2.00: failed command: READ DMA
[1189456.857168] ata2.00: cmd c8/00:08:c0:a1:76/00:00:00:00:00/e9 tag 23 dma 4096 in
res 51/40:08:c0:a1:76/00:00:00:00:00/e9 Emask 0x9 (media error)
[1189456.857171] ata2.00: status: { DRDY ERR }
[1189456.857172] ata2.00: error: { UNC }
[1189456.863473] ata2.00: configured for UDMA/133
[1189456.863496] ata2: EH complete
[1189456.953144] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189456.953158] ata2.00: irq_stat 0x40000001
[1189456.953160] ata2.00: failed command: READ DMA
[1189456.953163] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 1 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189456.953165] ata2.00: status: { DRDY ERR }
[1189456.953166] ata2.00: error: { UNC }
[1189456.959494] ata2.00: configured for UDMA/133
[1189456.959537] ata2: EH complete
[1189457.301157] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189457.301163] ata2.00: irq_stat 0x40000001
[1189457.301167] ata2.00: failed command: READ DMA
[1189457.301181] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 5 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189457.301186] ata2.00: status: { DRDY ERR }
[1189457.301188] ata2.00: error: { UNC }
[1189457.307467] ata2.00: configured for UDMA/133
[1189457.307504] ata2: EH complete
[1189457.397155] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189457.397166] ata2.00: irq_stat 0x40000001
[1189457.397169] ata2.00: failed command: READ DMA
[1189457.397174] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 9 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189457.397178] ata2.00: status: { DRDY ERR }
[1189457.397179] ata2.00: error: { UNC }
[1189457.403729] ata2.00: configured for UDMA/133
[1189457.403751] ata2: EH complete
[1189457.797161] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189457.797168] ata2.00: irq_stat 0x40000001
[1189457.797172] ata2.00: failed command: READ DMA
[1189457.797176] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 26 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189457.797181] ata2.00: status: { DRDY ERR }
[1189457.797182] ata2.00: error: { UNC }
[1189457.802966] ata2.00: configured for UDMA/133
[1189457.802990] ata2: EH complete
[1189458.025188] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189458.025205] ata2.00: irq_stat 0x40000001
[1189458.025207] ata2.00: failed command: READ DMA
[1189458.025220] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 11 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189458.025230] ata2.00: status: { DRDY ERR }
[1189458.025234] ata2.00: error: { UNC }
[1189458.031686] ata2.00: configured for UDMA/133
[1189458.031704] ata2: EH complete
[1189459.129214] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189459.129219] ata2.00: irq_stat 0x40000001
[1189459.129221] ata2.00: failed command: READ DMA
[1189459.129224] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 6 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189459.129226] ata2.00: status: { DRDY ERR }
[1189459.129227] ata2.00: error: { UNC }
[1189459.135142] ata2.00: configured for UDMA/133
[1189459.135161] ata2: EH complete
[1189459.213165] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189459.213169] ata2.00: irq_stat 0x40000001
[1189459.213181] ata2.00: failed command: READ DMA
[1189459.213184] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 9 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189459.213186] ata2.00: status: { DRDY ERR }
[1189459.213187] ata2.00: error: { UNC }
[1189459.219029] ata2.00: configured for UDMA/133
[1189459.219046] ata2: EH complete
[1189460.253177] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189460.253191] ata2.00: irq_stat 0x40000001
[1189460.253194] ata2.00: failed command: READ DMA
[1189460.253196] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 11 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189460.253199] ata2.00: status: { DRDY ERR }
[1189460.253200] ata2.00: error: { UNC }
[1189460.259520] ata2.00: configured for UDMA/133
[1189460.259558] ata2: EH complete
[1189460.333215] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189460.333219] ata2.00: irq_stat 0x40000001
[1189460.333221] ata2.00: failed command: READ DMA
[1189460.333224] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 17 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189460.333227] ata2.00: status: { DRDY ERR }
[1189460.333228] ata2.00: error: { UNC }
[1189460.339440] ata2.00: configured for UDMA/133
[1189460.339456] ata2: EH complete
[1189461.213172] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189461.213177] ata2.00: irq_stat 0x40000001
[1189461.213179] ata2.00: failed command: READ DMA
[1189461.213182] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 19 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189461.213185] ata2.00: status: { DRDY ERR }
[1189461.213186] ata2.00: error: { UNC }
[1189461.219488] ata2.00: configured for UDMA/133
[1189461.219497] scsi_io_completion_action: 11 callbacks suppressed
--
[1189461.219507] blk_update_request: I/O error, dev sdb, sector 16350680 op 0x0READ) flags 0x0 phys_seg 1 prio class 0
[1189461.219524] ata2: EH complete
[1189461.297186] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[1189461.297200] ata2.00: irq_stat 0x40000001
[1189461.297202] ata2.00: failed command: READ DMA
[1189461.297205] ata2.00: cmd c8/00:08:d8:7d:f9/00:00:00:00:00/e0 tag 12 dma 4096 in
res 51/40:08:d8:7d:f9/00:00:00:00:00/e0 Emask 0x9 (media error)
[1189461.297207] ata2.00: status: { DRDY ERR }
[1189461.297208] ata2.00: error: { UNC }
[1189461.303556] ata2.00: configured for UDMA/133
[1189461.303565] sd 1:0:0:0: [sdb] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
--
[1189461.303572] blk_update_request: I/O error, dev sdb, sector 16350680 op 0x0READ) flags 0x0 phys_seg 1 prio class 0
[1189461.303591] ata2: EH complete