I recently migrated my Ubuntu Server (where I had several Docker containers, including Jellyfin and qBittorrent) to Proxmox because I wanted to use pfSense as my personal router.
Before moving to proxmox I had:
- 2 × 3 TB drives in RAID0
- Stable read/write speeds around 40 MB/s
After setting up pfSense and moving to Proxmox, I reconnected the same two 3 TB drives passed through individually and re-created the RAID0 inside Ubuntu Server as a VM.
At first, upload speeds seemed fine ~30 MB/s, but after a few seconds they dropped exponentially down to ~2 MB/s. These was caused because IO delay spiked, load average increased, and CPU I/O wait went up to 95%. Since my main goal is to share as much as possible from my seedbox via qBittorrent, disk passthrough like this was not viable.
I also tried mounting the RAID directly in Proxmox and sharing via NFS, as well as mounting the disks in ZFS inside Proxmox, both with the same poor results.
My second attempt to reach the 30MB/s upload mark was buying a Marvell 88SE9215 SATA controller which I did PCI passthrough as follows:
The problem with the Marvell was compatibility as it only worked ~10% of the time after a reboot, 90% of the time Proxmox would hang, without any configuration changes. The few times it worked, I had stable ~30 MB/s uploads for hours, performance was fine, but reliability was not.
My third attempt was with an ASM1166 SATA controller. It had much better compatibility, worked on first try with the same passthrough config and never caused Proxmox to hang on reboot. Download speeds were perfect, up to 40 MB/s. But upload issue persisted: after ~30 seconds, load average and CPU I/O wait spiked, and upload dropped to ~5 MB/s (identical to passing the disks individually without a controller).
While testing with the ASM1166, I simulated torrent workloads.
- Upload/download a single 4 GB file had the expected performance.
- Uploading 4 KB random blocks had the same slow performance as with torrents.
So my conclusion is that the bottleneck is the ASM1166's random read performance.
I returned the ASM1166 and am currently passing the RAID directly from Proxmox to the Ubuntu Server VM as a temporary solution so I can keep using Jellyfin.
I’m considering buying an LSI 9207-8i (HBA IT mode) but I’m not 100% sure it will solve the issue, especially since I’ll likely be buying a cheap one from AliExpress, which could be a clone.
Has anyone here experienced similar random read bottlenecks with the ASM1166, and can confirm whether moving to an LSI HBA would fix it for high-IOPS workloads like torrent seeding?
Thanks.
Before moving to proxmox I had:
- 2 × 3 TB drives in RAID0
- Stable read/write speeds around 40 MB/s
After setting up pfSense and moving to Proxmox, I reconnected the same two 3 TB drives passed through individually and re-created the RAID0 inside Ubuntu Server as a VM.
At first, upload speeds seemed fine ~30 MB/s, but after a few seconds they dropped exponentially down to ~2 MB/s. These was caused because IO delay spiked, load average increased, and CPU I/O wait went up to 95%. Since my main goal is to share as much as possible from my seedbox via qBittorrent, disk passthrough like this was not viable.
I also tried mounting the RAID directly in Proxmox and sharing via NFS, as well as mounting the disks in ZFS inside Proxmox, both with the same poor results.
My second attempt to reach the 30MB/s upload mark was buying a Marvell 88SE9215 SATA controller which I did PCI passthrough as follows:
Code:
/etc/modprobe.d/vfio.conf
options vfio-pci ids=8086:5912,1002:ab28 disable_idle_d3=1 disable_vga=1
softdep ahci pre: vfio-pci
My third attempt was with an ASM1166 SATA controller. It had much better compatibility, worked on first try with the same passthrough config and never caused Proxmox to hang on reboot. Download speeds were perfect, up to 40 MB/s. But upload issue persisted: after ~30 seconds, load average and CPU I/O wait spiked, and upload dropped to ~5 MB/s (identical to passing the disks individually without a controller).
While testing with the ASM1166, I simulated torrent workloads.
- Upload/download a single 4 GB file had the expected performance.
- Uploading 4 KB random blocks had the same slow performance as with torrents.
So my conclusion is that the bottleneck is the ASM1166's random read performance.
I returned the ASM1166 and am currently passing the RAID directly from Proxmox to the Ubuntu Server VM as a temporary solution so I can keep using Jellyfin.
I’m considering buying an LSI 9207-8i (HBA IT mode) but I’m not 100% sure it will solve the issue, especially since I’ll likely be buying a cheap one from AliExpress, which could be a clone.
Has anyone here experienced similar random read bottlenecks with the ASM1166, and can confirm whether moving to an LSI HBA would fix it for high-IOPS workloads like torrent seeding?
Thanks.