Proxmox storage setup confusion

[Yaser]

Dears,

I know there is alot of threads that contain lots of information about the best practice storage setup but due to that I was not able to choose whats best for me as Im confused between using motherboard Sata connectors OR having HBA unread (IT Mode).

Note that the use of this setup is for my homelab.

My hardware is like the below:
CPU: Ryzen 7 2700X
Motherboard: ASUS EX-A320M-GAMING
RAM: 64GB (non ECC)
GPU: GeForce 8400 GS
Storage:
NVme 256GB [Proxmox OS]
1TB SSD [ThinProv for VMs]

2x 8TB Seagate IronWolf NAS [Still no setup]
I have an "LSI SAS9211-8i" on IT Mode

I will be using the NAS storage for my VM (along with the 1TB) as well as backing up my vms and NFS among my PC around the house.
I would really love to have this setup for a long-long use, I do not want to lose my data in any case...

Please do let me know whats the best option...

Option 1: Motherboard stock SATA with ZFS
Option 2: Using HBA and passing through PCI to a TrueNAS, and configure ZFS -- It will be use for the same reason VMs (not all), data, backup and NFS
Option 3: Prepare another PC with lower Specs hardware, install TrueNAS OS, prepare the setup as ZFS and connect the 2 servers with Fiber using "MNPA19-XTR 10GB MELLANOX ConnectX"

Thank you.

 

[ubu]

Do NOT use the same storage (NAS) for your data, vms and your backup, if you value your data.
How much Storage do you need and how fast should it be?
You can get about 80 to 90 Megabyte per second over Gigabit from your NAS, local disks will be much faster, ssds are cheap right now.
Install to a zfs mirror on ssd and use the rotating rust as a second mirror for data where speed is less important.

Use the NAS as backup storage

 

[Yaser]

Thank you @ubu for your reply. I understand by your reply you recommend me by going to option 3?

And what do you think about "LSI SAS9211-8i" on IT Mode?

 

[Dunuin]

Depends...

For your VMs/LXCs you really want local SSDs. Virtualizing TrueNAS works fine when passing through the HBA, but keep in mind that your GPU + HBA can't be in the same IOMMU group. Those two or three PCIe 16x slots will probably share the same IOMMU group. You might need a GPU that can fit in a PCIe 4x or 1x slot so it is connected to the chipset. Or otherwise your PVE host will lose the GPU once you passthrough the HBA.

And if you care about your data you want your data in one machine and your backups in another (and best case this other machine is offsite). Or atleast do some backups to external HDDs and store them offsite.

 

[ubu]

Like Dunuin said, it depends

Give us more info what kind of data and how much you have got

1. Do Backup on a separate machine, if you value your data
2. Local Storage will beat external Storage in performance

The questions are, how much Data do you have, what kind of Data.
There is a huge difference between a database and a large video storage.
On a (busy) database you get lots of small random reads and writes (use fast ssd)
on a video storage you mainly get streaming access (hdd should be ok)

 

[Yaser]

Thanks for your proactivity guys on my post. I had to travel and I cameback last weekend to check on this.

- As of now I dont have much of data, its around 600-800GB of sensitive data (images & videos)
- I wont have "heavy" database application that need high-end read and write

* I think I will go with SSD LVM-thin for my VM (OS only)
* Will create proxmox ZFS (RAIDZ) storage with my 2x8TB NAS (without using HBA PCi)
* And I will add another disk storage from the ZFS to my NextCloud (for example) and upload the images and videos there
* Will create NFS directory and use it as share folder for my other PCs

Please do let me know if this practice is good for a long term without having troubles.

Thank you agaaain

 

[RedneckBob]

And what do you think about "LSI SAS9211-8i" on IT Mode?

Quick note on the LSI SAS-9211-8i with IT firmware, I have about a half dozen of these in use for the last 5 to 6 years with zero issues. In a couple dev machines I hang SSDs off them, 4 drives max, with zero issues from the card. Other machines have 8 HDDs in them, all with zero issues from the card.

IMPORTANT: I have firmware version P19 on all of them as P20 did cause issues with SSDs. Don't recall the issue exactly, but no amount of tweaking the BIOS settings in the LSI SAS-9211-8i would solve the issue. Flashed the card with P19 and the issues went away. Have not researched these cards in some time nor sure if P20 issues were isolated to me, so be sure to do your own research. Flashing is a hassle, so bought the cards off ebay with P19 already flashed.