Proxmox 8.3 + TL-D800s + LSI 9207-8e

R

RejectedGranny

New Member
Jan 5, 2025
7
0
1
Hi, so I'm pretty new here...

My current setup is:
MS-01
TL-D800s
LSI 9207-8e
Virtualizing TrueNAS Scale in VM under Proxmox with PCI passthrough.

for now i'm populating the JBOD with 1 drive (in transition from DSM to TrueNAS) - my proxmox recognizes the drive, but also it lists ALL other bays (the empty ones), so we can see in the picture /dev/sdb as the drive, and 7 "drives" of model ASMT109x-_Config.

also noticed that sometimes on reboots, the drive is mounted as /dev/sda, sometimes as /dev/sdb and so on, i assume depends on the mounting ordered... i'm not sure if it matters that the mounting point changes for the drives (assuming later there's going to be more drives)
Screenshot 2025-01-05 174234.png
Screenshot 2025-01-05 174536.png

The problem is that once I pass the LSI HBA to the VM, the VM doesn't start because lots of `dev sdc sector 520 op...` (dev sdb would not fail) - aka, the actual drive...

Screenshot 2025-01-05 174608.pngScreenshot 2025-01-05 174644.png

Does anybody knows why this happens? how to I configure LSI HBA to "ignore" the empty drive bays? or how to configure TrueNAS Scale to ignore them?

With the QNAP PCI card it did not show the empty bays, but it had other issues with TrueNAS, that's why I chose to go with the LSI HBA...

Any assistance would be highly appreciated!
Thank you.
 
Last edited:
You cannot use (Netapp/EMC ?) disks with sector size 520 bytes and so need to reformat them
like eg. "sg_format --format --size=512 /dev/sdf".
 
Inside the TL-D800s there's only 1 physical drive (4TB), connecting the D800s to LSI-9207 shows 8 "drives" on proxmox...
 
Aah, die TL-D800s ist eine Qnap, die macht (soweit ich das ermitteln kann) und mag daher auch keine 520byte Platten (aus einer Natapp/EMC) ?!
Wenn du nicht explizit in der Doku was davon liest, daß 520byte Sektoren verwendet werden ..., Platte ausbauen, mit sg_format auf 512byte bringen und wieder zurück in die Qnap. (7x virtuelle 105 MB Volumes aus einer 4 TB Platte ...,)
 
RejectedGranny said:
Inside the TL-D800s there's only 1 physical drive (4TB), connecting the D800s to LSI-9207 shows 8 "drives" on proxmox...
Click to expand...
I tried this very same setup, unfortunately only before reading this post, and it failed spectacularly. Fought with it for a solid month but saw frequent SCSI driver power-on/device reset kernel messages, and eventual full enclosure/array drop-outs followed by kernel hangs. A simple scrub task could crash the entire system after some time. Just an unstable mess in TrueNAS 24.10. It was only until I returned the QNAP enclosure+card that I started having success.

Interesting side note though, I came across a TrueNAS forums post describing a similar setup to ours (Proxmox, PCIe passthrough of QNAP SATA card to QNAP enclosure, running in a TrueNAS Scale VM) describing that TrueNAS 23.10 seems to handle these ASMedia chips without crashing, but that any upgrade even to 24.04 (which is newer, but also not current) would render the whole system unstable. I was uncomfortable with sticking to an older version knowing I'd be potentially stuck there forever, so I ditched the QNAP bundle entirely.

The QNAP enclosure, while attractive in looks and price, uses a weak ASMedia SATA port multiplier inside; its bundled QXP-800eS-A1164 PCIe SATA card also uses ASMedia chip, and I think its firmware is programmed to mask those dummy drives. Swapping to a solid LSI HBA like your 9207-8e (I have the same) unhides those dummy drives, but can't remedy the stability issues because of the SATA port multipler inside the enclosure. These ASMedia chips do not do well with the simultaneous disk access demands of ZFS and TrueNAS.

I went with an LSI 9207-8e HBA like yourself, and purchased from Provantage a iStarUSA/Raidage JAGE5BT4HD-DE enclosure and it's been working great; the item on Provantage's site is marked as a "special order" I think because iStarUSA makes them to order, and then drop ships directly to you. The enclosure has tray-less/screw-less slots and uses an SFF-8644 (SAS3) external port for connection to your HBA. A 10GTek or similar brand SFF-8644 to SFF-8088 cable will make the connection between the enclosure and the HBA. This older TrueNAS forums post is what inspired me to go with this brand of enclosure, as they're much harder to find in 2025 what with Chia farming dying off and other factors. The company makes other formfactors and sizes if you need 8-bays like the QNAP had.

Be mindful of your external SAS cable lengths, this is another easy pitfall. I personally use SATA drives, and SATA signals have an overall 1 meter cable length limit that must be respected, else the connection will not work at all or might but with errors and drop-outs. While it would be tempting to order a 1 meter SAS cable to make the connection from enclosure to HBA, the internal cabling inside the enclosure counts towards this 1 meter overall limit, unless the enclosure contains a true SAS expander inside. Since my iStarUSA enclosure only has a simple unpowered adapter to convert between SAS SFF-8643 (internal) to SAS SFF-8644 (external), I assumed the internal cabling was 0.5 meter and used a 0.5 meter SAS cable to stay at the 1 meter cable length limit - this worked well.

Hope you were able to find a solution, otherwise hopefully some of the info I've described here gives you a workable path forward.
 
Last edited:
daedalus01 said:
I tried this very same setup, unfortunately only before reading this post, and it failed spectacularly. Fought with it for a solid month but saw frequent SCSI driver power-on/device reset kernel messages, and eventual full enclosure/array drop-outs followed by kernel hangs. A simple scrub task could crash the entire system after some time. Just an unstable mess in TrueNAS 24.10. It was only until I returned the QNAP enclosure+card that I started having success.

Interesting side note though, I came across a TrueNAS forums post describing a similar setup to ours (Proxmox, PCIe passthrough of QNAP SATA card to QNAP enclosure, running in a TrueNAS Scale VM) describing that TrueNAS 23.10 seems to handle these ASMedia chips without crashing, but that any upgrade even to 24.04 (which is newer, but also not current) would render the whole system unstable. I was uncomfortable with sticking to an older version knowing I'd be potentially stuck there forever, so I ditched the QNAP bundle entirely.

The QNAP enclosure, while attractive in looks and price, uses a weak ASMedia SATA port multiplier inside; its bundled QXP-800eS-A1164 PCIe SATA card also uses ASMedia chip, and I think its firmware is programmed to mask those dummy drives. Swapping to a solid LSI HBA like your 9207-8e (I have the same) unhides those dummy drives, but can't remedy the stability issues because of the SATA port multipler inside the enclosure. These ASMedia chips do not do well with the simultaneous disk access demands of ZFS and TrueNAS.

I went with an LSI 9207-8e HBA like yourself, and purchased from Provantage a iStarUSA/Raidage JAGE5BT4HD-DE enclosure and it's been working great; the item on Provantage's site is marked as a "special order" I think because iStarUSA makes them to order, and then drop ships directly to you. The enclosure has tray-less/screw-less slots and uses an SFF-8644 (SAS3) external port for connection to your HBA. A 10GTek or similar brand SFF-8644 to SFF-8088 cable will make the connection between the enclosure and the HBA. This older TrueNAS forums post is what inspired me to go with this brand of enclosure, as they're much harder to find in 2025 what with Chia farming dying off and other factors. The company makes other formfactors and sizes if you need 8-bays like the QNAP had.

Be mindful of your external SAS cable lengths, this is another easy pitfall. I personally use SATA drives, and SATA signals have an overall 1 meter cable length limit that must be respected, else the connection will not work at all or might but with errors and drop-outs. While it would be tempting to order a 1 meter SAS cable to make the connection from enclosure to HBA, the internal cabling inside the enclosure counts towards this 1 meter overall limit, unless the enclosure contains a true SAS expander inside. Since my iStarUSA enclosure only has a simple unpowered adapter to convert between SAS SFF-8643 (internal) to SAS SFF-8644 (external), I assumed the internal cabling was 0.5 meter and used a 0.5 meter SAS cable to stay at the 1 meter cable length limit - this worked well.

Hope you were able to find a solution, otherwise hopefully some of the info I've described here gives you a workable path forward.
Click to expand...

Hi Daedalus,

I eventually gave up on the setup and reverted back to my (smaller) Synology till i'd come up with another solution... it's quite unfortunate though.
 
RejectedGranny said:
Hi Daedalus,

I eventually gave up on the setup and reverted back to my (smaller) Synology till i'd come up with another solution... it's quite unfortunate though.
Click to expand...
Not sure if I missed it, but what version of TrueNAS are you running? If able, can you upgrade to 25.04 and test if that resolves the issue? If not, please open a bug ticket on Jira (be sure to include a debug file) and link it here for me: https://ixsystems.atlassian.net/jira/software/c/projects/NAS/issues
 
You must log in or register to reply here.
Top Bottom