Pass-Through disks to TrueNAS if I have Direct Attached Storage (DAS) for Data Backup NAS Expansion

[erasmo0284]

Hello everyone! I am new on Proxmox and I'm looking to Pass-Through 4 disks to TrueNAS

https://www.newegg.com/yottamaster-ps500c3-enclosure/p/0VN-067E-00017?Item=9SIBBRKK0F8130

I also want to do Raid10, unless you do not recommend it. The tutorial I have seen has a different serial. can someone guide me in the right direction?
This was the way I tried but it did not work.

qm set 100 -scsi1 /dev/disk/by-id/usb-JMicron_Generic_DISK00_0123456789ABCDEF-0:0
qm set 100 -scsi2 /dev/disk/by-id/usb-JMicron_Generic_DISK01_0123456789ABCDEF-0:1
qm set 100 -scsi3 /dev/disk/by-id/usb-JMicron_Generic_DISK02_0123456789ABCDEF-0:2
qm set 100 -scsi4 /dev/disk/by-id/usb-JMicron_Generic_DISK03_0123456789ABCDEF-0:3




Linux home 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jul 27 03:20:11 EDT 2024 on pts/0
root@home:~# lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS DEVICE-ID(S)
sda 8:0 0 7.3T 0 disk /dev/disk/by-id/usb-JMicron_Generic_DISK00_0123456789ABCDEF-0:0
sdb 8:16 0 7.3T 0 disk /dev/disk/by-id/usb-JMicron_Generic_DISK01_0123456789ABCDEF-0:1
sdc 8:32 0 7.3T 0 disk /dev/disk/by-id/usb-JMicron_Generic_DISK02_0123456789ABCDEF-0:2
sdd 8:48 0 7.3T 0 disk /dev/disk/by-id/usb-JMicron_Generic_DISK03_0123456789ABCDEF-0:3
nvme0n1 259:0 0 3.6T 0 disk /dev/disk/by-id/nvme-SPCC_M.2_PCIe_SSD_230398720041609 /dev/disk/by-id/nvme-SPCC_M.2_PCIe_SSD_230398720041609_1 /dev/disk/by-id/nvme-eui.6479a77fa0000059
root@home:~# lshw -class disk -class storage
*-nvme
description: NVMe device
product: SPCC M.2 PCIe SSD
vendor: Phison Electronics Corporation
physical id: 0
bus info: pci@0000:01:00.0
logical name: /dev/nvme0
version: ELFMC1.0
serial: 230398720041609
width: 64 bits
clock: 33MHz
capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list
configuration: driver=nvme latency=0 nqn=nqn.2020-11.com.phison:nvmeS5021:230398720041609 state=live
resources: irq:16 memory:50800000-50803fff
*-namespace:0
description: NVMe disk
physical id: 0
logical name: hwmon1
*-namespace:1
description: NVMe disk
physical id: 2
logical name: /dev/ng0n1
*-namespace:2
description: NVMe disk
physical id: 1
bus info: nvme@0:1
logical name: /dev/nvme0n1
size: 3726GiB (4TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: guid=d833fc91-87cb-4202-a826-74259815c0ad logicalsectorsize=512 sectorsize=512 wwid=eui.6479a77fa0000059
*-usb:1
description: Mass storage device
product: USB to ATA/ATAPI Bridge
vendor: JMicron
physical id: 2
bus info: usb@2:2
logical name: scsi2
version: 13.24
serial: 0123456789ABCDEF
capabilities: usb-3.00 scsi emulated scsi-host
configuration: driver=usb-storage maxpower=8mA speed=5000Mbit/s
*-disk:0
description: SCSI Disk
product: Generic DISK00
vendor: JMicron
physical id: 0.0.0
bus info: scsi@2:0.0.0
logical name: /dev/sda
version: 0103
serial: 0123456789ABCDEF
size: 7452GiB (8001GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=6 guid=e69cd42f-b3b6-4d76-85ce-96b55c13c117 logicalsectorsize=512 sectorsize=4096
*-disk:1
description: SCSI Disk
product: Generic DISK01
vendor: JMicron
physical id: 0.0.1
bus info: scsi@2:0.0.1
logical name: /dev/sdb
version: 0103
serial: 0123456789ABCDEF
size: 7452GiB (8001GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=6 guid=36bd8cc4-e434-4342-9087-20a7ae80d8ff logicalsectorsize=512 sectorsize=4096
*-disk:2
description: SCSI Disk
product: Generic DISK02
vendor: JMicron
physical id: 0.0.2
bus info: scsi@2:0.0.2
logical name: /dev/sdc
version: 0103
serial: 0123456789ABCDEF
size: 7452GiB (8001GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=6 guid=ad7e3c12-5c5d-4345-8d07-1e4a6501f3ea logicalsectorsize=512 sectorsize=4096
*-disk:3
description: SCSI Disk
product: Generic DISK03
vendor: JMicron
physical id: 0.0.3
bus info: scsi@2:0.0.3
logical name: /dev/sdd
version: 0103
serial: 0123456789ABCDEF
size: 7452GiB (8001GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=6 guid=6835d9cb-716e-4ddc-a35d-25f6d5f41214 logicalsectorsize=512 sectorsize=4096
*-sata
description: SATA controller
product: Alder Lake-P SATA AHCI Controller
vendor: Intel Corporation
physical id: 17
bus info: pci@0000:00:17.0
version: 01
width: 32 bits
clock: 66MHz
capabilities: sata msi pm ahci_1.0 bus_master cap_list
configuration: driver=ahci latency=0
resources: irq:124 memory:50900000-50901fff memory:50903000-509030ff ioport:3090(size=8) ioport:3080(size=4) ioport:3060(size=32) memory:50902000-509027ff
root@home:~#

 

[UdoB]

I am new on Proxmox and I'm looking to Pass-Through 4 disks to TrueNAS

You are starting from zero to 110 percent in one go. That curve is steep!

The only somewhat recommended way to hand over disks to TrueNAS is by PCI-passthrough a controller.

NOT just some disks - the driver handling this is still virtual, even when the bytes written are saved to the hardware disk. TrueNAS has no direct access to the disks with this approach.

Search for it, you are not the first user trying to do this on the cheap...

Good luck! I mean it

 

[erasmo0284]

I have a cluster, which is a full PC case. If I connect all 4 hard disks directly to the motherboard, will that be considered direct access?

 

[UdoB]

If I connect all 4 hard disks directly to the motherboard, will that be considered direct access?

Only if the controller can be passed through. But then... how would the system boot if that controller is not available for the primary OS anymore? So... probably no. Until you add a separate controller which you can "pass through" - if the motherboard/BIOS supports this feature.

More hints: https://pve.proxmox.com/wiki/PCI_Passthrough

From TrueNAS point of view: https://www.truenas.com/blog/yes-you-can-virtualize-freenas/ - note "The key piece of the puzzle is a technology broadly referred to as PCI passthrough..."

 

[erasmo0284]

Thank you for the help.... So in short words. Nobody should be using TrueNAS on Proxmox.

I need to buy a Real NAS

 

[UdoB]

So in short words. Nobody should be using TrueNAS on Proxmox.

Did I say that? There are some pitfalls one should avoid. Following the guides and the well established recommendations will work.

In a homelab you may accept more compromises than in a mission critical company server area - but you should know this and accept the potential consequences you might have to deal with. Example: of course most people (me included) highly recommend ECC Ram - but in my homelab I have several MiniPC without it...

 

[erasmo0284]

Ok, I will set up RAID10 in Proxmox instead of using a NAS. I will manage it with Docker, by mounting and sharing, and then install Plex and other apps to make it function similar to a NAS.

 

[UdoB]

I will manage it with Docker

Yes, Docker is really popular. Just make sure to read https://pve.proxmox.com/wiki/Linux_Container:

"If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox QEMU VM."

----
Edit: my above posts may sound as if I do not recommend PVE as everything is difficult or might not work. The opposite is true: I am really a fan of PVE and I do run a cluster @Work and @Home. PVE is a toolset. There are different toolsets for gardening and for a dentist. Understanding the difference and the specific usecase is essential - or it will hurt.

 

[erasmo0284]

Thank you for providing the documentation. Now I understand why VMs are better than containers. I used to prefer containers because of their speed, but as the documentation explains, there is a security issue with containers since they use the host Linux. If a hacker exploits one of the applications, they could easily gain access to the root of Proxmox..


I'm wondering if I can continue using my 5-bay DAS. The IT guys don't like USB-connected hard drives, but that's my only option because my main server PC is a mini PC. It only allows me to have 1 M.2 and one SSD drive.

 

[UdoB]

I'm wondering if I can continue using my 5-bay DAS.

Sure.

For data-storage? It is fine.
For virtual disks for a VM? I would avoid it, if possible.

The IT guys don't like USB-connected hard drives

For a reason: USB is a complex beast and may be error prone - too many layers, too many cheap manufacturers. And it is usually not really fast, IOPS wise. And some of my USB enclosures hide the actual drives behind a curtain, so PVE (or any other Linux) can not see the drives directly --> can not read SMART-data --> this is bad...

That said...: some of my MiniPC have an external NVMe connected via normal USB3. While my first attempt some years ago was a disaster, my current constellation works for several months now without any problems. I run an Ceph OSD on them. This is really NOT recommended!

Whatever you put on it (or basically on any storage device): make sure to have a backup

As you can see: there are many ways to skin a cat ;-)

 

[erasmo0284]

Hey, thanks a lot for taking the time to answer my questions. I will use the 4 disks for storage and 3 backups locations for VMs and all types of content. These locations will be used specifically for backup purposes. I hope you have a great day!