[SOLVED] Need help: OMV on Proxmox crashes during rsync

[Tetelestai80]

Hello,

I'm new to Proxmox and I'm looking for some help for an issue that I'm struggling to troubleshoot.

I have been running omv on Proxmox installed on a mini pc for a few months without issues. I decided to set up a new rsync jobs to better fit my use case and now the system is crashing every time I attempt to run this new job. The job will begin and will copy roughly 300-600MB, then it hangs at a seemingly random file, at which point the system powers off. This only seems to happen on large file transfers (my new rsync job needs to transfer ~700GB). Older, pre-existing rsync jobs with only minor changes execute without issues. Nothing has changed in the past two months with my setup other than installing updates.

My setup details:

• pve-manager/7.4x
• Kernel Linux 5.15.102-1
• My hard drives are all SSD and I'm passing them through from Proxmox with the "Use USB port" option.
• My VM has 1 proc and 2 GB of memory.
• My host has a 65w power supply
• Running omv v6.3.6-2

I know I have more than enough power for the drives, but just to rule out a power issue, I did connect the drives to AC power and rebooted my host with the same result. I have also tried increasing the memory to 4GB without success, then decreasing to 1GB just to see if there was a memory utilization/saturation situation and that didn't help.

I'm not sure what else to try next. The only change, aside from system updates, is the creation of the new rsync job. The job itself does not have any abnormal configuration. The only difference that I can see is that the system crashes when there is a larger amount of data to transfer.

Some of the final entries that I see in the omv VM syslog just before the system powers off include:

3/30/2023, 10:51:04 AM kernel: [ 2.036622] sd 2:0:0:0: Power-on or device reset occurred
3/30/2023, 10:51:04 AM kernel: [ 2.035915] usb 1-1: new full-speed USB device number 2 using uhci_hcd
3/30/2023, 10:51:04 AM kernel: [ 1.886979] sr 1:0:0:0: Attached scsi CD-ROM sr0
3/30/2023, 10:51:04 AM kernel: [ 1.871740] virtio_net virtio2 ens18: renamed from eth0
3/30/2023, 10:51:04 AM kernel: [ 1.799014] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
3/30/2023, 10:51:04 AM kernel: [ 1.868102] cdrom: Uniform CD-ROM driver Revision: 3.20
3/30/2023, 10:51:04 AM kernel: [ 2.037102] sd 2:0:0:0: [sda] 62914560 512-byte logical blocks: (32.2 GB/30.0 GiB)
3/30/2023, 10:51:04 AM kernel: [ 1.848028] scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5
3/30/2023, 10:51:04 AM kernel: [ 1.847439] scsi host2: Virtio SCSI HBA
3/30/2023, 10:51:04 AM kernel: [ 1.800216] scsi 1:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ PQ: 0 ANSI: 5
3/30/2023, 10:50:22 AM monit[614]: 'omv' loadavg (1min) of 2.4 matches resource limit [loadavg (1min) > 2.0]
3/30/2023, 10:49:56 AM kernel: [ 132.301237] sd 3:0:0:0: [sdb] tag#19 uas_eh_abort_handler 0 uas-tag 22 inflight: CMD IN
3/30/2023, 10:49:56 AM kernel: [ 132.301242] sd 3:0:0:0: [sdb] tag#19 CDB: Read(10) 28 00 3f 5a 48 48 00 04 00 00
3/30/2023, 10:49:56 AM kernel: [ 132.298471] sd 3:0:0:0: [sdb] tag#20 uas_eh_abort_handler 0 uas-tag 23 inflight: CMD IN
3/30/2023, 10:49:56 AM kernel: [ 132.303837] sd 3:0:0:0: [sdb] tag#18 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD IN
3/30/2023, 10:49:56 AM kernel: [ 132.298477] sd 3:0:0:0: [sdb] tag#20 CDB: Read(10) 28 00 3f 5a 4c 48 00 04 00 00
3/30/2023, 10:49:56 AM kernel: [ 132.303844] sd 3:0:0:0: [sdb] tag#18 CDB: Read(10) 28 00 3f 5a 43 88 00 04 00 00
3/30/2023, 10:49:56 AM kernel: [ 132.268729] sd 3:0:0:0: [sdb] tag#29 uas_eh_abort_handler 0 uas-tag 24 inflight: CMD IN

 

[Tetelestai80]

I was able to find a solution to this.

This article ended up guiding me to the solution. In summary, my SSD drives were using UAS, which I'm suspecting somehow broke for my USB enclosure controllers (Asmedia) with a recent kernel update. Once I blacklisted UAS support for my USB enclosures, everything works as normal.

You can view the details in the article above, plus it links to several other articles with more detail. It was the following actions that resolved the issue for me. Remember to run the lsusb command and replace the 174c:55aa text in the code block below with the ID of your USB enclosure.

echo options usb-storage quirks=174c:55aa:u | sudo tee /etc/modprobe.d/blacklist_uas.conf
sudo update-initramfs -u
sudo reboot

 

[linesquarecube]

Thank you so much for sharing this! After spending several evenings with trial and error, I stumbled upon your post, and it turned out I was also facing the UAS driver problem (using the same SATA adapter).

I truly appreciate that you took the time to post this!