Help with storage solution, i am losing it...

[ukro]

Greetings,
Please help i spend many days researching the best solution for my needs without obvious result.

The thing is that on my desktop home server i have put PCI card, put there 1ssd for PVE, 1ssd for vm's and passed the onboard sata controller and made truenas guest VM and all other vm's.That was easy, working for 4years without any issue.


But the problem is with work server (HP dl380e G8 server with ssd's and hhd's) i don't know the best solution for my needs i want to be able to use the ssd's for VM's and some storage solution for HDD's.
Adding PCI card is not an option as 2 sas connectors i cant split (i thought that 1cable is for 1/2 of the drives and second cable for 2/2 of the drives).
I had some experience in samba LXC which i didn't liked much. I read that somebody is using OMV.

But i would like the power of truenas (samba shares, user management, AD, ZFS cache for read,shadow copies for windows clients, snapshoting) in proxmox(VM/LXC?) ). How can i achieve that?
And another question that i read somewhere , i can passthrough not the PCI card, but hdd to VM or LXC and create btrfs and there are no issues with that, can somebody confirm?
As passing disks to truenas VM's is dangerous.

Thank you for any ideas that can relief my pain.
How are you guyz managing it?

 

[bobmc]

your main problem is that the HP disk controller is a RAID controller not a HBA. It's possible to install a generic LSI card but you would need to purchase a set of internal SAS cables. I've done it, but it's not pretty or cheap.

My goal was to run ZFS so there's no guarantee that you would be able to run TrueNAS with disk or controller pass-through - I didn't test the IOMMU support.

My advice would be to find a supermicro or dell server instead.

 

[ukro]

your main problem is that the HP disk controller is a RAID controller not a HBA. It's possible to install a generic LSI card but you would need to purchase a set of internal SAS cables. I've done it, but it's not pretty or cheap.

My goal was to run ZFS so there's no guarantee that you would be able to run TrueNAS with disk or controller pass-through - I didn't test the IOMMU support.

My advice would be to find a supermicro or dell server instead.

Thank you for reply, the server already bought, can't return. the controller is HP H220 HBA in IT mode.Nevertheless i am not able to pass the seperate disk to the VM...As the proxmox is booted from the drive that is connected to the card. You have some solution that you use?

 

[LnxBil]

You have some solution that you use?

It's always problematic to mix Linux and FreeBSD in such setups, go with one or the other.

LX(C) containers are not able to issue ZFS commands (snapshots), but they can be use them (just pseudo files) and do all other stuff with Samba, the software is the same. You may have to configure it yourself of look at how TrueNAS is doing it.

 

[Dunuin]

Didn't checked the server specs. In case server got a M.2 slot or a free PCIe slot for a PCIe-to-M.2-card you could run PVE and VMs of NVMe SSDs. Better performance and the complete HBA and backplane wouldn't be used by PVE so you could pas through the whole HBA to TrueNAS.
But in case the mainboard doesn't got a M.2 slot you should check first if the UEFI alllows you at all to boot from NVMe. If not you would need to put the bootloader on a USB stick.

 

[ukro]

Didn't checked the server specs. In case server got a M.2 slot or a free PCIe slot for a PCIe-to-M.2-card you could run PVE and VMs of NVMe SSDs. Better performance and the complete HBA and backplane wouldn't be used by PVE so you could pas through the whole HBA to TrueNAS.
But in case the mainboard doesn't got a M.2 slot you should check first if the UEFI alllows you at all to boot from NVMe. If not you would need to put the bootloader on a USB stick.

Thank you for reply, no M2 slot, just internal USB 3 that can be used, i have 2 free PCI slots, the problem is that i could only connect 2 internal drives and only to PCI sata card. As for SAS cables i can't split the drives. So with 2 drives only for Proxmox host and VM's that is not enough.

 

[ukro]

It's always problematic to mix Linux and FreeBSD in such setups, go with one or the other.

LX(C) containers are not able to issue ZFS commands (snapshots), but they can be use them (just pseudo files) and do all other stuff with Samba, the software is the same. You may have to configure it yourself of look at how TrueNAS is doing it.

Yeah, maybe i can just copy paste configs(zfs,samba) from Truenas to proxmox samba guest

 

[LnxBil]

Yeah, maybe i can just copy paste configs(zfs,samba) from Truenas to proxmox samba guest

Yes, that's what I meant.

 

[ukro]

I am now testing maybe a new solution that i dodn't remember reading anywere. So i created Proxmox ZFS volume mirror to let Proxmox figure zfs raid out and in Truenas VM i put 1 hdd in stripe. Working fine, will test performance and stability further. :-D i <3 it when i find a solution(If i did)

 

[bobmc]

But i would like the power of truenas (samba shares, user management, AD, ZFS cache for read,shadow copies for windows clients, snapshoting) in proxmox(VM/LXC?) ). How can i achieve that?

all of that can be done without TrueNAS - its just that the TrueNAS GUI makes it far easier to do. And if that's your core requirement, then install TrueNAS and run VM's under TrueNAS.

Otherwise you can install samba and winbind either directly on the host or on a container but you will need to manually setup shares and edit config files to make everything work.

 

[Dunuin]

I am now testing maybe a new solution that i dodn't remember reading anywere. So i created Proxmox ZFS volume mirror to let Proxmox figure zfs raid out and in Truenas VM i put 1 hdd in stripe. Working fine, will test performance and stability further. :-D i <3 it when i find a solution(If i did)

Thats not a good idea. TrueNAS uses ZFS. ZFS ontop of ZFS is bad, because of massive overhead that is multiplying.

 

[ukro]

Thats not a good idea. TrueNAS uses ZFS. ZFS ontop of ZFS is bad, because of massive overhead that is multiplying.

I tryed an yeah i see -20% -20% = -40% of storage

 

[ukro]

all of that can be done without TrueNAS - its just that the TrueNAS GUI makes it far easier to do. And if that's your core requirement, then install TrueNAS and run VM's under TrueNAS.

Otherwise you can install samba and winbind either directly on the host or on a container but you will need to manually setup shares and edit config files to make everything work.

The last resort is a container. But last resort....

 

[Dunuin]

I tryed an yeah i see -20% -20% = -40% of storage

Not only that. Also performance and disks wear.

You still got the option to use a USB SSD as your boot/system disk. There are quite small USB-to-M.2 enclosures and maybe you even got a internal USB port. Then you could strap that USB SSD somewhere inside the server. Even USB2 should work.

And for VM storage you could still use a PCIe SSD or M.2 on a PCIe-to-M.2-adapter card.

 

[LnxBil]

The last resort is a container. But last resort....

TrueNAS cannot run inside of a container. It's BSD, not Linux

 

[Dunuin]

TrueNAS cannot run inside of a container. It's BSD, not Linux

TrueNAS Scale would be Linux but I don't think it will run in a LXC as its a picky appliance.

 

[LnxBil]

TrueNAS Scale would be Linux but I don't think it will run in a LXC as its a picky appliance.

And isn't it more like on the same level as PVE itself?

 

[ukro]

TrueNAS cannot run inside of a container. It's BSD, not Linux

Sorry i ment Turnkey file server