Unstable USB drive causes problems

[oiwruk]

I have an 8TB WD usb drive connect to the proxmox host that I then share to containers and VMs. The problem is that it often becomes completely unresponsive and causes the web dashboard to have question marks (communication failure 0) and no communication to the services, which remain running though. The only way to fix this is a complete reboot. The problem existed even before sharing folders from the drive to containers or VMs.

When it gets stuck I cant even cd into the drive through SSH.

What might I be doing wrong with using a USB drive on Proxmox that causes such behaviour?

Many thanks

 

[Dunuin]

What model is that HDD and how did you format it? NTFS for example should be avoided because linux can't handle that natively. And ZFS has problems with SMR drives. And USB is in general not that reliable as a storage.

 

[oiwruk]

What model is that HDD and how did you format it? NTFS for example should be avoided because linux can't handle that natively. And ZFS has problems with SMR drives. And USB is in general not that reliable as a storage.

It's NTFS and CMR (not SMR). I never had problems with it on OMV. Are there any specific logs I should check?

 

[Dunuin]

NTFS is designed to only run on win and its closed source and propietary. Most of it will work with linux but it isn't capable of all the features a typical linux filesystem like ext4 or xfs is offering. So if you want best compability, most features and best performance its better to use a filesystem that is designed with linux in mind.

For the logs you should check the syslog from your host and guest. For your host it should be "/var/log/syslog" and "/var/log/syslog.1".

And you can also run a smart selftest of the drive to see if the drive is maybe causing problems. ("smartctl -t long /dev/yourdrive" on your host to start the selftest and "smartctl -a /dev/yourdrive" to check test progress and results)

 

[oiwruk]

NTFS is designed to only run on win and its closed source and propietary. Most of it will work with linux but it isn't capable of all the features a typical linux filesystem like ext4 or xfs is offering. So if you want best compability, most features and best performance its better to use a filesystem that is designed with linux in mind.

For the logs you should check the syslog from your host and guest. For your host it should be "/var/log/syslog" and "/var/log/syslog.1".

And you can also run a smart selftest of the drive to see if the drive is maybe causing problems. ("smartctl -t long /dev/yourdrive" on your host to start the selftest and "smartctl -a /dev/yourdrive" to check test progress and results)

It happened again... The drives are all fine according to SMART. Here are the logs

Nov  4 01:49:46 proxmox kernel: [703222.829047]       Tainted: P           O      5.11.22-5-pve #1
Nov  4 01:49:46 proxmox kernel: [703222.829058] task:scsi_eh_1       state:D stack:    0 pid:  199 ppid:     2 flags:0x00004000
Nov  4 01:49:46 proxmox kernel: [703222.829072]  __schedule+0x2ca/0x880
Nov  4 01:49:46 proxmox kernel: [703222.829087]  schedule_preempt_disabled+0xe/0x10
Nov  4 01:49:46 proxmox kernel: [703222.829096]  ? bt_tags_iter+0x92/0xd0
Nov  4 01:49:46 proxmox kernel: [703222.829105]  mutex_lock+0x34/0x40
Nov  4 01:49:46 proxmox kernel: [703222.829119]  scsi_eh_ready_devs+0x5f5/0xa50
Nov  4 01:49:46 proxmox kernel: [703222.829131]  scsi_error_handler+0x43f/0x520
Nov  4 01:49:46 proxmox kernel: [703222.829137]  kthread+0x12b/0x150
Nov  4 01:49:46 proxmox kernel: [703222.829146]  ret_from_fork+0x22/0x30
Nov  4 01:49:46 proxmox kernel: [703222.829159]       Tainted: P           O      5.11.22-5-pve #1
Nov  4 01:49:46 proxmox kernel: [703222.829167] task:usb-storage     state:D stack:    0 pid:  201 ppid:     2 flags:0x00004000
Nov  4 01:49:46 proxmox kernel: [703222.829174]  ? usleep_range+0x90/0x90
Nov  4 01:49:46 proxmox kernel: [703222.829182]  ? usleep_range+0x90/0x90
Nov  4 01:49:46 proxmox kernel: [703222.829188]  schedule_timeout+0xfe/0x140
Nov  4 01:49:46 proxmox kernel: [703222.829196]  wait_for_completion+0x24/0x30
Nov  4 01:49:46 proxmox kernel: [703222.829205]  usb_stor_bulk_transfer_sglist+0x8f/0xe0 [usb_storage]
Nov  4 01:49:46 proxmox kernel: [703222.829217]  ? usleep_range+0x90/0x90
Nov  4 01:49:46 proxmox kernel: [703222.829232]  ? try_to_wake_up+0x257/0x5c0
Nov  4 01:49:46 proxmox kernel: [703222.829241]  ? __wait_for_common+0xfb/0x150
Nov  4 01:49:46 proxmox kernel: [703222.829291]  usb_stor_control_thread+0x195/0x290 [usb_storage]
Nov  4 01:49:46 proxmox kernel: [703222.829302]  kthread+0x12b/0x150
Nov  4 01:49:46 proxmox kernel: [703222.829308]  ret_from_fork+0x22/0x30
Nov  4 01:51:46 proxmox kernel: [703343.657279] INFO: task scsi_eh_1:199 blocked for more than 483 seconds.
Nov  4 01:51:46 proxmox kernel: [703343.657303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov  4 01:51:46 proxmox kernel: [703343.657308] task:scsi_eh_1       state:D stack:    0 pid:  199 ppid:     2 flags:0x00004000
Nov  4 01:51:46 proxmox kernel: [703343.657321]  __schedule+0x2ca/0x880
Nov  4 01:51:46 proxmox kernel: [703343.657335]  schedule_preempt_disabled+0xe/0x10
Nov  4 01:51:46 proxmox kernel: [703343.657343]  ? bt_tags_iter+0x92/0xd0
Nov  4 01:51:46 proxmox kernel: [703343.657352]  mutex_lock+0x34/0x40
Nov  4 01:51:46 proxmox kernel: [703343.657365]  scsi_eh_ready_devs+0x5f5/0xa50
Nov  4 01:51:46 proxmox kernel: [703343.657375]  scsi_error_handler+0x43f/0x520
Nov  4 01:51:46 proxmox kernel: [703343.657382]  kthread+0x12b/0x150
Nov  4 01:51:46 proxmox kernel: [703343.657389]  ret_from_fork+0x22/0x30
Nov  4 01:51:46 proxmox kernel: [703343.657402]       Tainted: P           O      5.11.22-5-pve #1
Nov  4 01:51:46 proxmox kernel: [703343.657410] task:usb-storage     state:D stack:    0 pid:  201 ppid:     2 flags:0x00004000
Nov  4 01:51:46 proxmox kernel: [703343.657417]  ? usleep_range+0x90/0x90
Nov  4 01:51:46 proxmox kernel: [703343.657424]  ? usleep_range+0x90/0x90
Nov  4 01:51:46 proxmox kernel: [703343.657431]  schedule_timeout+0xfe/0x140
Nov  4 01:51:46 proxmox kernel: [703343.657438]  wait_for_completion+0x24/0x30
Nov  4 01:51:46 proxmox kernel: [703343.657447]  usb_stor_bulk_transfer_sglist+0x8f/0xe0 [usb_storage]
Nov  4 01:51:46 proxmox kernel: [703343.657459]  ? usleep_range+0x90/0x90
Nov  4 01:51:46 proxmox kernel: [703343.657469]  ? try_to_wake_up+0x257/0x5c0
Nov  4 01:51:46 proxmox kernel: [703343.657478]  ? __wait_for_common+0xfb/0x150
Nov  4 01:51:46 proxmox kernel: [703343.657487]  usb_stor_control_thread+0x195/0x290 [usb_storage]
Nov  4 01:51:46 proxmox kernel: [703343.657499]  kthread+0x12b/0x150
Nov  4 01:51:46 proxmox kernel: [703343.657505]  ret_from_fork+0x22/0x30

Do you have any idea what is happening with the storage?

pvesm status

or cd-ing in the USB storage directory just hangs the server. All VMs and containers are with gray question marks and yet function (they live on the main SSD, not the USB HDD).