IO-error constantly appears

[newbie57]

Hello, I am currently running Proxmox on an intel nuc (i5-1135G7) running two VMs, TrueNas and Ubuntu. Every couple days or so I will run into an io-error on TrueNas, but all my disks seem to be fine. The log history has no signs of any error, I also tried disabling KSM but the error still kept occurring. Trying to troubleshoot what the issue would be but I am failing. Do I have to replace my m2? Any help on this matter would be greatly appreciated

 

[cwt]

I would check your logs/nvme for errors. NUCs are naturally too small for efficient cooling of ssd/nvme drives. It‘s possible that your drives get too hot and throttle down.

dmesg | grep -i nvme
dmesg | grep -i i/o

In addition (for example if you have two nvme):

smartctl -a /dev/nvme0n1
smartctl -a /dev/nvme1n1

Look for media errors, crc errors and temperature events.

 

[newbie57]

Here are my returns: I don't think I can get temperature information because I am using a terramaster d6-320?

dmesg | grep -i nvme
dmesg | grep -i i/o
[ 1.913405] nvme 0000:01:00.0: platform quirk: setting simple suspend
[ 1.913518] nvme nvme0: pci function 0000:01:00.0
[ 1.940084] nvme nvme0: allocated 64 MiB host memory buffer.
[ 1.957887] nvme nvme0: 8/0/0 default/read/poll queues
[ 1.970036] nvme0n1: p1 p2 p3
[ 196.954745] aesni_intel snd_intel_sdw_acpi drm_buddy crypto_simd snd_hda_codec ttm snd_hda_core cryptd snd_hwdep drm_display_helper snd_pcm intel_pmc_core ucsi_acpi iwlwifi intel_vsec cec snd_timer pmt_telemetry rapl typec_ucsi cmdlinepart intel_cstate pcspkr snd mei_me spi_nor rc_core cfg80211 wmi_bmof ee1004 mtd soundcore i2c_algo_bit mei igen6_edac typec pmt_class acpi_tad acpi_pad mac_hid zfs(PO) spl(O) vhost_net vhost vhost_iotlb tap vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq uas usb_storage hid_generic usbkbd usbmouse usbhid hid dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xhci_pci rtsx_pci_sdmmc xhci_pci_renesas nvme crc32_pclmul rtsx_pci xhci_hcd thunderbolt igc nvme_core intel_lpss_pci ahci spi_intel_pci i2c_i801 intel_lpss spi_intel libahci i2c_smbus idma64 nvme_auth video wmi pinctrl_tigerlake
[ 0.110982] APIC: Switch to symmetric I/O mode setup
[ 1.261146] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 3.433963] sd 2:0:0:0: [sda] Preferred minimum I/O size 4096 bytes
[ 3.624062] sd 3:0:0:0: [sdb] Preferred minimum I/O size 4096 bytes
[ 4.237944] sd 4:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes
[ 4.322963] sd 5:0:0:0: [sdd] Preferred minimum I/O size 4096 bytes
[ 4.408946] sd 6:0:0:0: [sde] Preferred minimum I/O size 4096 bytes
[ 4.495967] sd 7:0:0:0: [sdf] Preferred minimum I/O size 4096 bytes
[45354.473408] I/O error, dev sda, sector 10659195656 op 0x0READ) flags 0x200000 phys_seg 13 prio class 0
[45354.473429] I/O error, dev sda, sector 10659195816 op 0x0READ) flags 0x200000 phys_seg 7 prio class 0
[45354.619434] I/O error, dev sdc, sector 10659195608 op 0x0READ) flags 0x200000 phys_seg 13 prio class 0
[45354.619458] I/O error, dev sdc, sector 10659195768 op 0x0READ) flags 0x200000 phys_seg 7 prio class 0
[45354.777270] I/O error, dev sdd, sector 10659195600 op 0x0READ) flags 0x200000 phys_seg 13 prio class 0
[45354.777307] I/O error, dev sdd, sector 10659195712 op 0x0READ) flags 0x200000 phys_seg 6 prio class 0
[45354.926256] I/O error, dev sde, sector 10659195608 op 0x0READ) flags 0x200000 phys_seg 6 prio class 0
[45354.926272] I/O error, dev sde, sector 10659195664 op 0x0READ) flags 0x200000 phys_seg 13 prio class 0
[45355.074217] I/O error, dev sdf, sector 10659195608 op 0x0READ) flags 0x200000 phys_seg 13 prio class 0
[45355.074238] I/O error, dev sdf, sector 10659195712 op 0x0READ) flags 0x200000 phys_seg 6 prio class 0
[45356.381173] sd 8:0:0:0: [sdg] Preferred minimum I/O size 4096 bytes
[45356.578146] sd 9:0:0:0: [sdh] Preferred minimum I/O size 4096 bytes
[45357.178155] sd 10:0:0:0: [sdi] Preferred minimum I/O size 4096 bytes
[45357.267135] sd 11:0:0:0: [sdj] Preferred minimum I/O size 4096 bytes
[45357.343135] sd 12:0:0:0: [sdk] Preferred minimum I/O size 4096 bytes
[45357.432142] sd 13:0:0:0: [sdl] Preferred minimum I/O size 4096 bytes

 

[cwt]

You have I/O errors on your sdX drives (SSDs or HDDs). Looks like your S-ATA controller has a problem.

 

[newbie57]

thank you for your help and I apologize for my ignorance, what should I be doing with this information? Do I need to replace the nuc?

 

[cwt]

What NUC model is that? And how many drives do you use?

 

[newbie57]

Its the Nuc i5-1135G7, and I am using a Terramaster d6-320 with six hard drives inside

 

[cwt]

Ah, okay. The d6-320 is know for it‘s bugginess in combination with the linux UAS driver. I hope you didn’t setup ZFS on this.

What you can do is a fallback to usbstorage-driver. First you‘ll have to check which VID&PID your enclosure has (for example: 152d:0583j:

lsusb

After that:

echo "options usb-storage quirks=152d:0583:u" > /etc/modprobe.d/terramaster-d6.conf
update-initramfs -u
reboot

Check if the „new“ driver is used after the reboot:

dmesg | grep usb-storage
lsusb -t

But beware that using the more stable usb-storage driver comes with a performance drop of 20-30%.

 

[newbie57]

Thank you again for your help through this! Okay ran those commands rebooted, and checked after. Did I do it right? Here are my results:

root@pve:~# lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 007: ID 8087:0026 Intel Corp. AX201 Bluetooth
Bus 003 Device 002: ID 1ea7:0066 SHARKOON Technologies GmbH [Mediatrack Edge Mini Keyboard]
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 005: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 009: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 008: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 007: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 006: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 004: ID 0bda:0420 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub
Bus 002 Device 003: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 002: ID 0bda:0420 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@pve:~#

echo "options usb-storage quirks=174c:55aa:u" > /etc/modprobe.d/terramaster-d6.conf
update-initramfs -u
reboot

update-initramfs: Generating /boot/initrd.img-6.8.12-9-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
root@pve:~#


root@pve:~# dmesg | grep usb-storage
lsusb -t
[ 3.390740] usb 2-1.1: UAS is ignored for this device, using usb-storage instead
[ 3.390743] usb-storage 2-1.1:1.0: USB Mass Storage device detected
[ 3.390881] usb-storage 2-1.1:1.0: Quirks match for vid 174c pid 55aa: c00000
[ 3.390911] scsi host2: usb-storage 2-1.1:1.0
[ 3.391011] usbcore: registered new interface driver usb-storage
[ 3.580157] usb 2-1.4: UAS is ignored for this device, using usb-storage instead
[ 3.580160] usb-storage 2-1.4:1.0: USB Mass Storage device detected
[ 3.580239] usb-storage 2-1.4:1.0: Quirks match for vid 174c pid 55aa: c00000
[ 3.580266] scsi host3: usb-storage 2-1.4:1.0
[ 4.214556] usb 2-1.3.1: UAS is ignored for this device, using usb-storage instead
[ 4.214560] usb-storage 2-1.3.1:1.0: USB Mass Storage device detected
[ 4.214679] usb-storage 2-1.3.1:1.0: Quirks match for vid 174c pid 55aa: c00000
[ 4.214709] scsi host4: usb-storage 2-1.3.1:1.0
[ 4.296321] usb 2-1.3.2: UAS is ignored for this device, using usb-storage instead
[ 4.296325] usb-storage 2-1.3.2:1.0: USB Mass Storage device detected
[ 4.296420] usb-storage 2-1.3.2:1.0: Quirks match for vid 174c pid 55aa: c00000
[ 4.296445] scsi host5: usb-storage 2-1.3.2:1.0
[ 4.377229] usb 2-1.3.3: UAS is ignored for this device, using usb-storage instead
[ 4.377233] usb-storage 2-1.3.3:1.0: USB Mass Storage device detected
[ 4.377337] usb-storage 2-1.3.3:1.0: Quirks match for vid 174c pid 55aa: c00000
[ 4.377362] scsi host6: usb-storage 2-1.3.3:1.0
[ 4.458396] usb 2-1.3.4: UAS is ignored for this device, using usb-storage instead
[ 4.458400] usb-storage 2-1.3.4:1.0: USB Mass Storage device detected
[ 4.458478] usb-storage 2-1.3.4:1.0: Quirks match for vid 174c pid 55aa: c00000
[ 4.458505] scsi host7: usb-storage 2-1.3.4:1.0
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
|__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 1: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 10: Dev 7, If 0, Class=Wireless, Driver=btusb, 12M
|__ Port 10: Dev 7, If 1, Class=Wireless, Driver=btusb, 12M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 10000M
|__ Port 3: Dev 4, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 1: Dev 6, If 0, Class=Mass Storage, Driver=usb-storage, 10000M
|__ Port 2: Dev 7, If 0, Class=Mass Storage, Driver=usb-storage, 10000M
|__ Port 3: Dev 8, If 0, Class=Mass Storage, Driver=usb-storage, 10000M
|__ Port 4: Dev 9, If 0, Class=Mass Storage, Driver=usb-storage, 10000M
|__ Port 4: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 10000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M

 

[cwt]

Looks good so far. You should watch your logs if the error is gone.

 

[newbie57]

Awesome! Thank you so much, I will update but hopefully this is the last you hear of me!

 

[newbie57]

Welp, the IO error happened again, do you think I need the replace the m2?