NAS on Proxmox

[Mineralwasser]

Hello,
I have a small Proxmox home server where I want to host different services for myself.

I also need a NAS for project files and personal data that I can access from different devices.
Also my family should have access to this NAS.

Currently I've set it up like this:
- FreeNAS as a VM
- HBA PCIe passthrough to the FreeNAS VM
- 2x HDDs as ZFS mirror on the HBA
- Shares via NFS to proxmox (for backups)
- Shares via SMB for user access

This works completely fine for almost all aspects.

However, I think this is a unnecessary complicated setup.
Also FreeNAS has too many features that I don't need.

I think to maybe change this setup to something simpler.
But I like to have ZFS for the NAS...

Does anyone have recommendations?

However I don't want to install extra software on Proxmox itself.
It is only the hypervisor and should stay as simple as possible.
Services such as NAS should be separated imo.

 

[narrateourale]

I personally just installed Samba directly on the PVE node and set the config file and samba users manually.

As you I think that the whole passing through disks to a NAS distribution like FreeNAS is too complicated.

The only other way would be to create a normal VM and use that as network share. But then you loose the direct access to ZFS.

 

[DrillSgtErnst]

Well,
I hope my thoughts aren't shit. Please correct me, if I am wrong, but I personally think, you want to use FreeNAS because of the possible Plugins etc. not only the Filesystem, correct?

So how about creating an ZFS Pool in Proxmox, use an VirtIO Block Disk for the VM, give the VM just one Disk an sonfigure it as JBOD. The Storage will be done by the Proxmox and the Applications by FreeNAS. Maybe that's not the best solution, but I think by far the easiest. You could also add DIsks, so every share has it's own (JBOD configured) Virtual Disk.

BUT If you want to use ZFS for the Snapshots etc, this will of course not work, because the FreeNAS isn't aware of the ZFS.
Maybe you could just configure the Single-Drive as XFS, cause AFAIK XFS supports snapshotting. So you would get access to snapshotting.


Maybe let me hear if I'm missing something out, or even I'm bullshitting here.

Hope this helps for your consideration.

 

[narrateourale]

but I personally think, you want to use FreeNAS because of the possible Plugins etc. not only the Filesystem, correct?

I'm curious which feature beside Samba/NFS shares you use.

If all I need is to export some paths as NFS/Samba share and want ZFS, doing it bare metal and manually editing the config files where needed is fine in my opinion.

PVE for my home server is a mainly Debian based Linux distribution that comes with ZFS already built in and the option to manage VMs / containers very easily if I need them. Of course if you need to offer network shares on a cluster with the ability to migrate between the nodes for HA and such, this approach will fail. But then again, passing through an extra HBA for direct access to the disks from the VM will not work too. In that case you will have to use the storage abstraction available to be able to migrate the VM to other nodes.

Maybe you could just configure the Single-Drive as XFS, cause AFAIK XFS supports snapshotting. So you would get access to snapshotting.

Do you mean for the FreeNAS VM? AFAIK FreeNAS will only be happy with ZFS ( https://www.ixsystems.com/documentation/freenas/11.3-U3.2/storage.html#importing-a-disk).

 

[DrillSgtErnst]

I m curious which feature beside Samba/NFS shares you use.

Well maybe Replication and Backup Services Provider like wbadmin, VEEAM, Time-Machine or RSync. Maybe Multimedia Streaming or Collaboration Tools. Just guesses. Or some (like maybe me, like GUIs more, than commandlines, for they are easier to understand at first. The Terminal sure is fine and even better than GUI, as soon as you're familiar with the Material.

But then again, passing through an extra HBA for direct access to the disks from the VM will not work too. In that case you will have to use the storage abstraction available to be able to migrate the VM to other nodes.

I agree on this. Therefore I would find it easier to just present one Virtual Disk to FreeNAS, as your machine can easily be migrated to every other PVE Host via vzdump.

Do you mean for the FreeNAS VM? AFAIK FreeNAS will only be happy with ZFS ( https://www.ixsystems.com/documentation/freenas/11.3-U3.2/storage.html#importing-a-disk).

You are totally right and I was not aware, that FreeNAS is ZFS only. I set up my first FreeNAS Friday, and I only searched for ZFS, So I didn't even look for another FileSystem. Maybe I was a bit too ignorant. Sorry on that.
ZFS can used for SingleDisk as Stripe with Size 1. I don't know shit about the upcoming performance of settling an ZFS Single Disk onto an Virtual Drive running on ZFS RAIDz(1/2/3).


But to be honest, FreeNAS with BSD is a real pain in the ass for me, since I really miss some opportunities on BSD. (Like it does not respect my Proxy Settings at all)

Well this Thread got me stuck on the Idea. Maybe tomorrow in the evening I will reinstall the FreeNAS with a Proxmox, set up ZFS, install an FreeNAS VM with a single Stripe and Test the Performance and will let you know about the outcome. Kinda interested in the outcome right now.

@op thanks for this question, it really appeals to me.

 

[Mineralwasser]

No I don't use plugins etc. on the FreeNAS VM.
I use FreeNAS only for pure storage NFS/SMB shares.

When I think about it now afterwards... how did I come to that brilliant idea to buy a HBA and passthrough the drives to a FreeNAS VM. Just for such a simple thing... lol sounds ridiculous. Guess I was too hyped about FreeNAS.

Setting the shares up on the pve host doesn't sound too bad tbh.
Very lightweight solution and I have the ZFS features.

 

[DrillSgtErnst]

My Last Message was not sent, waiting for the Mod.

I set Up the FreeNAS on Top of PVE.

Just for fun.

Spoiler: Whole Process

root@zfs1:~# zpool import -aN -d /dev/disk/by-id/ -o -f
cannot import 'ZFS2_Pool': pool was previously in use from another system.
Last accessed by HappyFreeNas.company.local (hostid=******) at Fri Jul 3 18:08:31 2020
The pool can be imported, use 'zpool import -f' to import the pool.

root@zfs1:~# zpool import -f -aN -d /dev/disk/by-id/
root@zfs1:~# zpool upgrade
This system supports ZFS pool feature flags.

This is only to show, that my former Pool from FreeNAS would still be readible.

All pools are formatted using feature flags.


Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(5) for details.

POOL FEATURE
---------------
ZFS2_Pool
large_dnode
edonr
userobj_accounting
encryption
project_quota
allocation_classes
resilver_defer
bookmark_v2

root@zfs1:~# zpool upgrade ZFS2_Pool
This system supports ZFS pool feature flags.

Enabled the following features on 'ZFS2_Pool':
large_dnode
edonr
userobj_accounting
encryption
project_quota
allocation_classes
resilver_defer
bookmark_v2

root@zfs1:~# reboot

ADD Storage ID FreeNAS, ZFS2_Pool

But only 24TiB uasble. not an Option

Clean Drives
fdisk /dev/sdX
d
1
d
w

New ZFS Pool
All Drives
Compression LZ4

If you are experimenting with an installation of Proxmox VE inside a VM (Nested Virtualization), don’t use virtio for disks of that VM, since they are not supported by ZFS. Use IDE or SCSI instead (works also with virtio SCSI controller type).

Download and Upload FreeNAS image
Create VM with VirtIO SCSI as Disk, Write back cache
Installation with Swap on Single Drive 24TiB
Base Setup FreeNAS
Yeah delete disk, cause I just added one Disk, need 2. One for System, one for data.

Performance of ZFS on PVE (only sequential)
dd if=/dev/zero of=/myzpool/data/file.out bs=4096 count=10000000
^C875207+0 records in
875206+0 records out
3584843776 bytes (3.6 GB, 3.3 GiB) copied, 9.06472 s, 395 MB/s

dd if=/zpool/file.out of=/dev/null bs=4096
875206+0 records in
875206+0 records out
3584843776 bytes (3.6 GB, 3.3 GiB) copied, 39.012 s, 91.9 MB/s

Performance of ZFS on FreeNAS (only sequential)
root@freenas[/mnt/zpool]# dd if=/dev/zero of=/mnt/zpool/file.out bs=4096 count=10000000
10000000+0 records in
10000000+0 records out
40960000000 bytes transferred in 301.604266 secs (135.807098 MB/sec)

dd if=/mnt/zpool/file.out of=/dev/null bs=4096
10000000+0 records in
10000000+0 records out
40960000000 bytes transferred in 429.566548 secs (95.351931 MB/sec)


it would take more tests to verify these benches, but it seems the performance is impacting mostly on writes.


My biggest Problem so far is, that I can only allocate 12,9TiB. But I used 11x4TB RAIDZ2. I should be able to allocate

ZFS usable storage capacity:

34.444444

37.872067

I am a bit baffled.

trying to figure that one out

 

[Mineralwasser]

But as far as I can see your FreeNAS installation doesn't have direct access to the disks?
Is that "benchmark" even valid then?
Or am I wrong?

 

[Limak]

Hi,
If u want have Raid 1 (i understand), maybe the better solution is creation Raid 1 by mdadm from debian level? it's easy to rebuild it.

Free NAS have better library files, but if you want lighter OS maybe try openmediavault or like narrateoura wrote install debian without kde/gnome and make the Raid during installation or via mdadm. In my case, for debian 9 and 10, I had to remove grub2 and reinstall it again (on both disks).

 

[DrillSgtErnst]

But as far as I can see your FreeNAS installation doesn't have direct access to the disks?
Is that "benchmark" even valid then?
Or am I wrong?

Oh nono I tried my suggestion. Using PVE for the ZFS part and just pointing ONE Virtual DIsk to the VM for Storage. I was curious about the performance loss.

Pasthrough seemed to much of work

Therefore I benched over my PVE installation directly and over my FreeNAS

Well some small changes on my host seem to have increased the Performance of ZFS.
I will report back with the performance of FreeNAS.

maybe this will be an option for you or others. It's kinda easier (for me) administering PVE over FreeNAS.

 

[MarcMorgan]

im looking into this. have Proxmox as my main server. i have a plex vm and on my main server (host) i have 40+ TB of HDDs spare that want to merge in5o one volume or LVM Group so plex (installed on a windows vm) can see this as one volume. i know on windows i can merge them if they all show separately but want them in one group for proxmox to see etc

 

[Maurice]

This might be of your interest:
https://forum.level1techs.com/t/how-to-create-a-nas-using-zfs-and-proxmox-with-pictures/117375