Zpool Gets Suspended and All Disks Become Unavailable

[joshontech]

Proxmox VE Version: 7.1-12
Server 1 Hardware:

• Intel i5 quad core 3rd gen,
• dual 120gb ssd in zfs raid1 boot and root disks
• 3, 2tb hdd in raidz1, 2 seagates and one HGST
• 4, 1tb hdd in raidz1, Inside a four disk ProBox usb3 enclosure

Server 2 Hardware

• intel i5 dual core 3rd gen laptop cpu
• single 120gb ssd boot and root drive
• 4, 1tb hdd in raidz1, Inside a four disk ProBox usb3 enclosure

My issue is happening on 2 of my proxmox servers. What is happening is eventually when transferring files from or to my USB enclosure the zpool that is on that will get suspended and all 4 disks will be unavailable untill using the zpool clear command. Than eventually after a while it will happen again and again. The only common thing between these 2 servers is the USB enclosure used. Additionally I created the zpool on the USB enclosure using ubuntu 21.10 and have a hunch that because of that something is not working well with proxmox.

When directly connecting the usb enclosure to either proxmox server everything looks good from the start the pool imports fine and works fine than it will just randomly get suspended with nothing to indicate why. As I have stated before I created the pool on Ubuntu 21.10 so I have a hunch that the version of zfs that proxmox uses does not like the version that is being used when creating with Ubuntu 21.10. I wish I had more information to give but there are literally no logs telling me anything not from proxmox not from zfs and certainly not from the log files in Linux I checked them all. The only other thing indicating any issue was my i/o delay was between 60% and 70% consistently while files were transferring.

Anyways anyone with any knowledge about this I would love some help otherwise I guess consider this a bug report. Thanks in advance for any help!

 

[Dunuin]

You should have a look at zpool status. If there are too many read/write errors for a disk the complete pool be shown as degraded and lock down until you reset the error counter with "zpool clear".
And you are not the first person that has problems with running ZFS pools of such multi disk USB enclosures. Looks like they aren't that reliable. And ZFS is only as reliable as the hardware you run it on. Thats why enterprise grade hardware is recommended if you want a reliable storage.

 

[Dunuin]

Also make sure to use very short and well shielded USB cables. And you don't only get read errors when the data gets currupted on the way from CPU to Disk (CPU -> Chipset -> USB Controller -> USB Cable -> USB-Hub-Chip -> USB-to-SATA-Controller -> Disk), you also get them when the average seek time is too slow and the disk can't answer in time before a timeout occures. I guess all these additional chips in between and sending data of 4 disks through a single USB connection isn't helping there too. ZFS wants to access all the 4 disks at the same time. Maybe that USB-Chip of the enclosure isn't load balancing that load well and data of one disk doesn't get send in time because the other 3 disks are already using all the USB-Bandwidth. So something like that.

 

[joshontech]

Also make sure to use very short and well shielded USB cables. And you don't only get read errors when the data gets currupted on the way from CPU to Disk (CPU -> Chipset -> USB Controller -> USB Cable -> USB-Hub-Chip -> USB-to-SATA-Controller -> Disk), you also get them when the average seek time is too slow and the disk can't answer in time before a timeout occures. I guess all these additional chips in between and sending data of 4 disks through a single USB connection isn't helping there too. ZFS wants to access all the 4 disks at the same time. Maybe that USB-Chip of the enclosure isn't load balancing that load well and data of one disk doesn't get send in time because the other 3 disks are already using all the USB-Bandwidth. So something like that.

I would 100% agree with you and actually believed this was the case except for the fact that I have been using the USB pool for months with no issues at all using Ubuntu 21.10. The minute I attached it to proxmox and than started transferring files it eventually just suspended.

I can't even check for errors on the disks cause zpool status just says the disks are all unavailable not even that they are degraded just fully unavailable.

Thanks for your reply!