Getting io-error on VM

Dextrode · Feb 26, 2023

Some months ago I have installed TrueNAS on a VM, it was working fine until now, I tried to access my NAS but I couldn't connect to it, once I logged in to proxmox I noticed a yellow triangle on my trueNAS instance, as I'm pretty new at this I don't know where to start, I restarted my instance but still no luck.

I'm on version 7.1-7

fiona · Feb 27, 2023

Hi,
this means that QEMU ran into an IO error for a disk of the VM. Please check your /var/log/syslog from around the time the issue happened for additional information. What kind of storage are the VM's disks on? Is everything alright with the storages and the underlying physical disks (use e.g. smartctl -a /dev/XYZ with the appropriate device path to get information about that)?

Please share the output of pveversion -v and qm config 100.

Dextrode · Mar 6, 2023

Hi fiona, thanks for your answer,

All my VM's are on a 500GB NVME drive, and additionaly the TrueNAS VM has 4 HDD (2TB each) attached to it.

This is the result of the last 100 lines in /var/log/syslog

Bash:

root@dextrode:/var/log# tail -n100 syslog
Mar  5 00:00:07 dextrode systemd[1]: Finished Rotate log files.
Mar  5 00:10:07 dextrode rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="1237" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Mar  5 00:17:01 dextrode CRON[544415]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 00:24:01 dextrode CRON[545564]: (root) CMD (if [ $(date +%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi)
Mar  5 00:55:43 dextrode smartd[1240]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 187 to 193
Mar  5 01:17:01 dextrode CRON[554305]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 01:25:42 dextrode smartd[1240]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 171 to 176
Mar  5 01:25:43 dextrode smartd[1240]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 176 to 181
Mar  5 01:25:43 dextrode smartd[1240]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 193 to 187
Mar  5 01:55:42 dextrode smartd[1240]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 181 to 176
Mar  5 01:55:42 dextrode smartd[1240]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 187 to 193
Mar  5 02:17:01 dextrode CRON[564229]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 02:25:41 dextrode smartd[1240]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 176 to 181
Mar  5 02:25:42 dextrode smartd[1240]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 176 to 181
Mar  5 03:10:01 dextrode CRON[572956]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r)
Mar  5 03:10:40 dextrode systemd[1]: Starting Online ext4 Metadata Check for All Filesystems...
Mar  5 03:10:40 dextrode systemd[1]: e2scrub_all.service: Succeeded.
Mar  5 03:10:40 dextrode systemd[1]: Finished Online ext4 Metadata Check for All Filesystems.
Mar  5 03:17:01 dextrode CRON[574118]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 03:30:01 dextrode CRON[576240]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /usr/lib/x86_64-linux-gnu/e2fsprogs/e2scrub_all_cron)
Mar  5 04:17:01 dextrode CRON[583951]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 04:55:41 dextrode smartd[1240]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 181 to 187
Mar  5 04:55:42 dextrode smartd[1240]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 176 to 181
Mar  5 04:55:42 dextrode smartd[1240]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 181 to 187
Mar  5 04:55:43 dextrode smartd[1240]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 193 to 200
Mar  5 05:17:01 dextrode CRON[593809]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 05:36:40 dextrode systemd[1]: Starting Daily apt download activities...
Mar  5 05:36:40 dextrode systemd[1]: apt-daily.service: Succeeded.
Mar  5 05:36:40 dextrode systemd[1]: Finished Daily apt download activities.
Mar  5 05:40:40 dextrode systemd[1]: Starting Daily PVE download activities...
Mar  5 05:40:42 dextrode pveupdate[597757]: <root@pam> starting task UPID:dextrode:00091F08:01483D8B:64047FBA:aptupdate::root@pam:
Mar  5 05:40:43 dextrode pveupdate[597768]: command 'apt-get update' failed: exit code 100
Mar  5 05:40:43 dextrode pveupdate[597757]: command 'apt-get update' failed: exit code 100
Mar  5 05:40:43 dextrode pveupdate[597757]: <root@pam> end task UPID:dextrode:00091F08:01483D8B:64047FBA:aptupdate::root@pam: command 'apt-get update' failed: exit code 100
Mar  5 05:40:43 dextrode systemd[1]: pve-daily-update.service: Succeeded.
Mar  5 05:40:43 dextrode systemd[1]: Finished Daily PVE download activities.
Mar  5 05:40:43 dextrode systemd[1]: pve-daily-update.service: Consumed 1.168s CPU time.
Mar  5 06:17:01 dextrode CRON[604091]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 06:25:01 dextrode CRON[605398]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Mar  5 06:43:40 dextrode systemd[1]: Starting Daily apt upgrade and clean activities...
Mar  5 06:43:40 dextrode systemd[1]: apt-daily-upgrade.service: Succeeded.
Mar  5 06:43:40 dextrode systemd[1]: Finished Daily apt upgrade and clean activities.
Mar  5 06:45:47 dextrode systemd[1]: Starting Daily apt download activities...
Mar  5 06:45:47 dextrode systemd[1]: apt-daily.service: Succeeded.
Mar  5 06:45:47 dextrode systemd[1]: Finished Daily apt download activities.
Mar  5 06:47:01 dextrode CRON[609103]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ))
Mar  5 07:17:01 dextrode CRON[614018]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 08:17:01 dextrode CRON[623883]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 09:17:01 dextrode CRON[633753]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 10:17:01 dextrode CRON[643597]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 11:17:01 dextrode CRON[653454]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 12:17:01 dextrode CRON[663283]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 12:25:41 dextrode smartd[1240]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 187 to 181
Mar  5 12:25:42 dextrode smartd[1240]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 187 to 181
Mar  5 12:55:42 dextrode smartd[1240]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 181 to 176
Mar  5 12:55:43 dextrode smartd[1240]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 200 to 193
Mar  5 13:17:01 dextrode CRON[673153]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 14:17:01 dextrode CRON[682994]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 14:25:42 dextrode smartd[1240]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 181 to 176
Mar  5 14:55:41 dextrode smartd[1240]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 181 to 176
Mar  5 14:55:41 dextrode smartd[1240]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 176 to 171
Mar  5 15:17:01 dextrode CRON[692788]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 15:25:43 dextrode smartd[1240]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 193 to 187
Mar  5 16:17:01 dextrode CRON[702650]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 17:17:01 dextrode CRON[712562]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 17:22:01 dextrode pvestatd[1603]: auth key pair too old, rotating..
Mar  5 17:25:42 dextrode smartd[1240]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 171 to 166
Mar  5 17:25:42 dextrode smartd[1240]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 176 to 171
Mar  5 17:55:41 dextrode smartd[1240]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 176 to 171
Mar  5 17:55:43 dextrode smartd[1240]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 187 to 181
Mar  5 18:14:07 dextrode systemd[1]: Starting Cleanup of Temporary Directories...
Mar  5 18:14:07 dextrode systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Mar  5 18:14:07 dextrode systemd[1]: Finished Cleanup of Temporary Directories.
Mar  5 18:17:01 dextrode CRON[722422]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 19:17:01 dextrode CRON[732354]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 20:17:01 dextrode CRON[742266]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  5 20:58:23 dextrode pvedaemon[1637]: <root@pam> successful auth for user 'root@pam'
Mar  5 21:00:11 dextrode pvedaemon[1635]: <root@pam> starting task UPID:dextrode:000B6F77:019C6BDF:6405573B:vncshell::root@pam:
Mar  5 21:00:11 dextrode pvedaemon[749431]: starting termproxy UPID:dextrode:000B6F77:019C6BDF:6405573B:vncshell::root@pam:
Mar  5 21:00:11 dextrode pvedaemon[1637]: <root@pam> successful auth for user 'root@pam'
Mar  5 21:00:11 dextrode systemd[1]: Created slice User Slice of UID 0.
Mar  5 21:00:11 dextrode systemd[1]: Starting User Runtime Directory /run/user/0...
Mar  5 21:00:11 dextrode systemd[1]: Finished User Runtime Directory /run/user/0.
Mar  5 21:00:11 dextrode systemd[1]: Starting User Manager for UID 0...
Mar  5 21:00:11 dextrode systemd[749450]: gpgconf: error running '/usr/lib/gnupg/scdaemon': probably not installed
Mar  5 21:00:11 dextrode systemd[749445]: Queued start job for default target Main User Target.
Mar  5 21:00:11 dextrode systemd[749445]: Created slice User Application Slice.
Mar  5 21:00:11 dextrode systemd[749445]: Reached target Paths.
Mar  5 21:00:11 dextrode systemd[749445]: Reached target Timers.
Mar  5 21:00:11 dextrode systemd[749445]: Listening on GnuPG network certificate management daemon.
Mar  5 21:00:11 dextrode systemd[749445]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Mar  5 21:00:11 dextrode systemd[749445]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Mar  5 21:00:11 dextrode systemd[749445]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Mar  5 21:00:11 dextrode systemd[749445]: Listening on GnuPG cryptographic agent and passphrase cache.
Mar  5 21:00:11 dextrode systemd[749445]: Reached target Sockets.
Mar  5 21:00:11 dextrode systemd[749445]: Reached target Basic System.
Mar  5 21:00:11 dextrode systemd[749445]: Reached target Main User Target.
Mar  5 21:00:11 dextrode systemd[749445]: Startup finished in 93ms.
Mar  5 21:00:11 dextrode systemd[1]: Started User Manager for UID 0.
Mar  5 21:00:11 dextrode systemd[1]: Started Session 87 of user root.
root@dextrode:/var/log#

This is the output from pveversion -v

Bash:

root@dextrode:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph: 16.2.7
ceph-fuse: 16.2.7
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

And this is the output from qm config 100

Bash:

root@dextrode:~# qm config 100
boot: order=scsi0;ide2;net0
cores: 6
ide2: local:iso/TrueNAS-12.0-U8.iso,media=cdrom
memory: 16384
meta: creation-qemu=6.1.0,ctime=1644706732
name: truenas
net0: virtio=EA:D8:26:A1:15:ED,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=32G
scsi1: /dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YFHH3L6G,size=1953514584K
scsi2: /dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YGH4DJPC,size=1953514584K
scsi3: /dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YGHTUX5A,size=1953514584K
scsi4: /dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YGHZ7RYA,size=1953514584K
scsihw: virtio-scsi-pci
smbios1: uuid=24546dc4-9581-48bc-84d5-a3d5faa9dd82
sockets: 1
vmgenid: 6238d70a-77e9-404b-b0be-7a44f7d2e3c4

Dextrode · Mar 6, 2023

This is the output for the first HDD

Bash:

root@dextrode:/var/log# smartctl -a /dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YFHH3L6G
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.19-2-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Ultrastar 7K3000
Device Model:     Hitachi HUA723020ALA641
Serial Number:    YFHH3L6G
LU WWN Device Id: 5 000cca 223d4f689
Firmware Version: MK7OA840
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Mar  5 21:13:10 2023 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (19784) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 330) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   133   133   054    Pre-fail  Offline      -       93
  3 Spin_Up_Time            0x0007   127   127   024    Pre-fail  Always       -       496 (Average 485)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       119
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   123   123   020    Pre-fail  Offline      -       31
  9 Power_On_Hours          0x0012   089   089   000    Old_age   Always       -       83890
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       82
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       192
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       192
194 Temperature_Celsius     0x0002   171   171   000    Old_age   Always       -       35 (Min/Max 14/52)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      9512         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

This is the output for the second HDD

Bash:

root@dextrode:/var/log# smartctl -a /dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YGH4DJPC
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.19-2-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Ultrastar 7K3000
Device Model:     Hitachi HUA723020ALA641
Serial Number:    YGH4DJPC
LU WWN Device Id: 5 000cca 224d0180e
Firmware Version: MK7OA840
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Mar  5 21:14:46 2023 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (21112) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 352) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   133   133   054    Pre-fail  Offline      -       92
  3 Spin_Up_Time            0x0007   125   125   024    Pre-fail  Always       -       504 (Average 489)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       100
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   123   123   020    Pre-fail  Offline      -       31
  9 Power_On_Hours          0x0012   089   089   000    Old_age   Always       -       83605
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       74
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       163
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       163
194 Temperature_Celsius     0x0002   166   166   000    Old_age   Always       -       36 (Min/Max 13/53)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      9277         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Dextrode · Mar 6, 2023

This is the output for the third HDD

Bash:

root@dextrode:/var/log# smartctl -a /dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YGHTUX5A
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.19-2-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Ultrastar 7K3000
Device Model:     Hitachi HUA723020ALA641
Serial Number:    YGHTUX5A
LU WWN Device Id: 5 000cca 224d961ab
Firmware Version: MK7OA840
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Mar  5 21:15:44 2023 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (19784) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 330) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   134   134   054    Pre-fail  Offline      -       88
  3 Spin_Up_Time            0x0007   133   133   024    Pre-fail  Always       -       481 (Average 458)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       110
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   123   123   020    Pre-fail  Offline      -       31
  9 Power_On_Hours          0x0012   089   089   000    Old_age   Always       -       82914
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       81
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       166
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       166
194 Temperature_Celsius     0x0002   171   171   000    Old_age   Always       -       35 (Min/Max 14/54)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      8537         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

And this is the output for the last HDD

Bash:

root@dextrode:/var/log# smartctl -a /dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YGHZ7RYA
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.19-2-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Ultrastar 7K3000
Device Model:     Hitachi HUA723020ALA641
Serial Number:    YGHZ7RYA
LU WWN Device Id: 5 000cca 224dbd7b1
Firmware Version: MK7OA840
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Mar  5 21:16:59 2023 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (19784) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 330) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   133   133   054    Pre-fail  Offline      -       90
  3 Spin_Up_Time            0x0007   126   126   024    Pre-fail  Always       -       499 (Average 490)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       127
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   123   123   020    Pre-fail  Offline      -       31
  9 Power_On_Hours          0x0012   089   089   000    Old_age   Always       -       80719
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       81
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       202
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       202
194 Temperature_Celsius     0x0002   181   181   000    Old_age   Always       -       33 (Min/Max 14/52)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      6341         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Thanks in advance!

fiona · Mar 6, 2023

Dextrode said:
Bash:

root@dextrode:/var/log# tail -n100 syslog

You need to check the log from around the time the issue happened, because that's when related messages would be logged. Or did the issue happen again right before that?

Dextrode said:
This is the output from pveversion -v

Bash:

root@dextrode:~# pveversion -v proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)

The version you are using is already more than a year old. I suggest you upgrade to the current version. See
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_package_repositories
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_system_software_updates

From a quick look, the SMART results look good.

Search

Search

Getting io-error on VM

Dextrode

New Member

Attachments

fiona

Proxmox Staff Member

Dextrode

New Member

Dextrode

New Member

Dextrode

New Member

fiona

Proxmox Staff Member