Ubuntu File/Media Server (NAS) on Proxmox

[norsemangrey]

I am building a home Ubuntu Server w/Docker NAS for file and media storage/sharing and I am planning to use Proxmox as I like the flexibility of being able to run other VMs on the server in the future as well (perhaps Home Assistant and pfSense). The hardware I am using is listed below in addition to perhaps a single SSD drive for installing Proxmox.

• Intel Xeon E-2246G
• Supermicro X11SCH-F
• Micron DDR4 2666MHz ECC 32GBG
• Seagate Exos 7E8 8TB (x4)
• Samsung 970 Pro Series 512GB (x2)

The thought is to install Proxmox on a single regular SSD while the Samsung M.2 disk will be set up in a zfs mirror for VMs and the Seagate storage disks in a 2 x 2 mirror vdev zpool. I will make use of the vGPU for hardware transcoding (Plex).

I have some familiarity with Ubuntu and Docker and have bee trying out Proxmox on a test server for a month or two, but I am stuck with how to proceed with the actual configuration for what I am trying to achieve (mainly for the storage). I have never done any setup or mounting or mapping of drives / pools in Linux or Proxmox and I am not sure of the best practices for my purpose. I could really use some input to how to go ahead with this.

All the research I have done tells me something different and nothing seems to be exactly like what I am trying to do. Is I at all recommended to have the Ubuntu NAS as VM or would I be better of running it in bare metal?

 

[Dominic]

Ubuntu NAS as VM or would I be better of running it in bare metal?

If you install Proxmox VE bare metal then I would put everything else in a virtual machine or container. PVE on regular SSD and VM on M.2 is ok for that. This way you can easily wipe PVE or the NAS without affecting the other.

but I am stuck with how to proceed with the actual configuration for what I am trying to achieve (mainly for the storage). I have never done any setup or mounting or mapping of drives / pools

You can create the ZFS mirror in the GUI of Proxmox VE.

The storage is then automatically created. When you create your Ubuntu VM you select the storage (
here "tank"), choose the size of the virtual hard disk and store your files on that virtual disk. This way you don't have to think about passthrough. You can also test this all virtually before buying anything.

 

[norsemangrey]

If you install Proxmox VE bare metal then I would put everything else in a virtual machine or container. PVE on regular SSD and VM on M.2 is ok for that. This way you can easily wipe PVE or the NAS without affecting the other.


You can create the ZFS mirror in the GUI of Proxmox VE.
View attachment 23480
The storage is then automatically created. When you create your Ubuntu VM you select the storage (
here "tank"), choose the size of the virtual hard disk and store your files on that virtual disk. This way you don't have to think about passthrough. You can also test this all virtually before buying anything.

Many thanks for your answer Bying before testing is unfortunately too late, but anyways that is not an issue because I the HW I have chosen should do nicely in many different setups.

In any case, although it is probably poorly formulated, i think my question/issue addresses the step after creation of the zpool. As mentioned the current plan is to run many different microservices in Docker containers on a single Ubuntu VM. Several of these containers will need access to one or more folders/datasets on my storage pool (e.g. Plex, NextCloud, Samba, Sonarr, etc.) Some with need access to the same folders/datasets. I guess what I am looking for is details into best practices for achieving this, and wonder if there are concerns to take into account with regards to the Proxmox -> VM -> Docker setup, ZFS performance, datastore sharing between different services etc.

It might be an easy straight-forward approach to this, but I get confused reading posts mentioning binding, mounting, mapping, using shared NFS in LXC and so on and so on, without any actual guides / tutorials covering this part of which approach is best for which case. Thus I find myself severely stuck and I am looking for a way forward

 

[Dominic]

Using Docker in a VM (instead of LXC) is recommended in our Administration Guide. So in terms of setup this should be the easier option. This way you can ignore every post about LXC.

Several of these containers will need access to one or more folders/datasets on my storage

Do you mean that second 2x2 zpool?

I am not completely sure what you had in mind, but maybe we get closer. When you have 2 zpools


and both are added as storage

then you can place the virtual hard disks of your Ubuntu VM on those different storages

and in your VM you will see two disks

and if you format and mount them then you have them both available as folders in your VM so to say
So in this VM, everything that I place in the folder first is on one pool and everything in the second folder is on the other pool. With dd you can create a dummy file in the folder and you should see the relevant datasets on ZFS grow.


Those screenshots are actually from a live Alpine VM. For your case, you would install Ubuntu on one of those (virtual) disks and mount only the other.

 

[norsemangrey]

Thanks again for taking the time

Using Docker in a VM (instead of LXC) is recommended in our Administration Guide. So in terms of setup this should be the easier option. This way you can ignore every post about LXC.

Yes from what I understand there are some security/isolation trade-offs if using LXC container although they might be more resource effective?

Do you mean that second 2x2 zpool?

No not really. The plan is actually to have to pools, but not for storage. I plan to use the Samsung M.2 disks in a ZFS mirror setup to for hosting VM machines, and then a 2x2 vdev zpool for storage used to store data (files/media) for the Docker micro-services in the Ubuntu VM.

I am not completely sure what you had in mind, but maybe we get closer. When you have 2 zpools

This might be completely wrong because I do not fully understand all of this, but from what I have read I think there are (at least) three different approaches:

1. Create the zpool in Proxmox and mount the zpool on the Ubuntu VM.
   • .... or create datasets on the zpool as well in Proxmox and mount these on the Ubuntu VM (is that even possible?)
2. Mount the individual disks on the Ubuntu VM and create the zpool and datasets on the zpool inside Ubuntu.
3. Create the zpool in Proxmox and mount the zpool on a LXC container create datasets on the zpool and set up som form of NFS service to share the datasets with the Ubuntu VM (or something along those lines).

Is any of this correct or have I misunderstood completely? I if there are in fact different approaches like the ones described above are any recommended above the other (for my purpose at least) and are there any ZFS performance / memory trade-offs for any of them?

 

[norsemangrey]

bump*

 

[bobmc]

I run a setup which is similar to what you seem to be trying to achieve - all of my home services - files shares. plex, emby, transmission, pfsense, all sharing the same hardware resources.

In my setup I have some ssd/nvme storage which is dedicated to Proxmox for running virtual machines and containers. I also have a zfs pool on sata drives on the Proxmox host which provides network storage for all my home users and is the backing store for Plex and Emby

The network shares are managed by samba (again running on the Proxmox host) while at the same time are mounted into the containers running Plex, Emby and Transmission. Any Windows VM's needing access use mapped drives to the Samba shares and while I don't use any Linux VM's, there's no reason why NFS shares wouldn't serve the same purpose. It does mean you have to plan file permissions so that everything works as intended but it works well for me.

I don't run a docker environment currently so I can't comment regarding that.

 

[norsemangrey]

Maybe this illustrates better what I am trying to achieve. From my reading and research I get the impression that many have a setup like this, but I cannot find any detailed guides / explanations or best practices on how to structure, configure and bring the datasets into the VM to make them available to the Docker containers.

 

[bobmc]

I would ask youself first if you *really* need to run docker? It makes things way more complicated on the hosting side of things. I get that docker images are really easy to deploy once you have the environment setup correctly but there isn't anything you are doing here that could not be achieved just using containers instead of docker. There are install guides for Plex and Sonarr and for NextCloud there's a ready-made LXC template available.

 

[norsemangrey]

I would ask youself first if you *really* need to run docker? It makes things way more complicated on the hosting side of things. I get that docker images are really easy to deploy once you have the environment setup correctly but there isn't anything you are doing here that could not be achieved just using containers instead of docker. There are install guides for Plex and Sonarr and for NextCloud there's a ready-made LXC template available.

Thanks for your input. I'm not sure I see any argument against using Docker in your reply though. In any case, does the fact that I choose to use Docker inside the Ubuntu VM affect the questions I have around the storage pool setup?

 

[bobmc]

argument against: introduces complexity and performance overhead where the stated objectives can be achieved without it

argument for: ?

 

[Dunuin]

argument against: introduces complexity and performance overhead where the stated objectives can be achieved without it

argument for: ?

+argument against: If you want to update something you need to throw away that docker container. So you can't individualize your services because everything you edited will also be thrown away.

I only use Docker if the the things I'm trying to do is so super simple that I don't need to individualize anything. Nextcloud and Plex are things I never would want to be dockerized. There are so much tutorials on hardening security and optimizing performance of Nextcloud by using OPcache + Redis as Cache, optimizing MySQL, optimizing PHP FPM, auto creating SSL certificates, setting it up for use with a reverse proxy and so on. If I just use a plug-and-play docker container I don't know how well secured and optimized that webserver is and it is not possible to do it by myself. And for Plex you might want to PCI passthrough a GPU so Plex can use hardware accelleration for encoding and I'm not sure if such things would be possible if Plex is running inside a docker container.

And for your questions in the picture:
You can't passthrough a datataset into a VM. If you want to access your files on the ZFS pool from inside the VM you need to use some kind of network protocol like NFS.

And datasets can only be created on the pool, so you need to do that on your host and not inside the VM. The guest has no access to ZFS at all.

For recordsize: that depends on the workload. If there is for example a DB like MySQL (like your nextcloud is using it) you want the recordsize small because DBs do small writes. But I wouldn't store a DB on a dataset because it would be faster if it is just stored on the virtual HDD and not over a NFS share. If you want to store big files you want a bigger recordsize for better performance.

Using many datasets for different things is a good idea so you can optimize the dataset to fit the workload that dataset is used for.

For the ARC: Rule of thumb is 4GB + 1GB per 1TB of raw storage for the ARC. So you could try it with 37GB RAM for the ARC and run arc_summary to look if you need to increase it or if you even can lower it. I would think just 24GB would run fine too.
You need to test that by yourself for your workloads.

 

[norsemangrey]

+argument against: If you want to update something you need to throw away that docker container. So you can't individualize your services because everything you edited will also be thrown away.

I have not yet used Docker for services like Plex and NextCloud, but quite a few others, and my experience is that any individualized changes and setup are always stored in config files which are not part the container itself. Isn't that kind of the whole point of docker containers? It is then quite an effortless process to update, move, delete, recreate containers and re-use tailored configuration for the specific container service.

I only use Docker if the the things I'm trying to do is so super simple that I don't need to individualize anything. Nextcloud and Plex are things I never would want to be dockerized. There are so much tutorials on hardening security and optimizing performance of Nextcloud by using OPcache + Redis as Cache, optimizing MySQL, optimizing PHP FPM, auto creating SSL certificates, setting it up for use with a reverse proxy and so on. If I just use a plug-and-play docker container I don't know how well secured and optimized that webserver is and it is not possible to do it by myself. And for Plex you might want to PCI passthrough a GPU so Plex can use hardware accelleration for encoding and I'm not sure if such things would be possible if Plex is running inside a docker container.

Are there any guides covering how to set up a typical file and media server with services like Plex, NextCloud, Samba including use of ZFS and tying the services through a reverse proxy? I would really like to try it out.

And for your questions in the picture:
You can't passthrough a datataset into a VM. If you want to access your files on the ZFS pool from inside the VM you need to use some kind of network protocol like NFS.

And datasets can only be created on the pool, so you need to do that on your host and not inside the VM. The guest has no access to ZFS at all.

But I can pass through the whole ZFS pool right? And I I created the datasets in Proxmox as well those datasets would be passed along with the ZFS pool on which they were created?

If using NFS would I then have to create a LXC container, pass the ZFS pool to that container and then use a NFS service in that container to share the datasets with other VMs and LXC containers?

For recordsize: that depends on the workload. If there is for example a DB like MySQL (like your nextcloud is using it) you want the recordsize small because DBs do small writes. But I wouldn't store a DB on a dataset because it would be faster if it is just stored on the virtual HDD and not over a NFS share. If you want to store big files you want a bigger recordsize for better performance.

When you say "virtual HDD" do you mean the same HDD as the VM is running on? Then I guess the DB should be set up for som kind of backup, in case the VM drive crashes.

Using many datasets for different things is a good idea so you can optimize the dataset to fit the workload that dataset is used for.

Good to know

For the ARC: Rule of thumb is 4GB + 1GB per 1TB of raw storage for the ARC. So you could try it with 37GB RAM for the ARC and run arc_summary to look if you need to increase it or if you even can lower it. I would think just 24GB would run fine too.
You need to test that by yourself for your workloads.

I have 32 GB ECC RAM for now and might extend with and additional 32 GB later. But what I was referring to with this question has more to do with ARC tunables like what is being discussed here. However, I have no idea on how to do this in practice on Proxmox.

 

[Dunuin]

I have not yet used Docker for services like Plex and NextCloud, but quite a few others, and my experience is that any individualized changes and setup are always stored in config files which are not part the container itself. Isn't that kind of the whole point of docker containers? It is then quite an effortless process to update, move, delete, recreate containers and re-use tailored configuration for the specific container service.

Yes, some configs will be stored outside in persistent volumes. But what for example if that docker contailer isn't using fail2ban but you want to use it so that nextclouds webserver can't be hacked that easily. Or how do you setup a redis cache if that container isn't using that by default? You can't install new packages to that docker container and because that container isn't using these programs by default, der are no rules set to map these configs to a persistent volume.

Are there any guides covering how to set up a typical file and media server with services like Plex, NextCloud, Samba including use of ZFS and tying the services through a reverse proxy? I would really like to try it out.

You can try NginxProxyManager if you want a easy GUI. But keep in mind that you need to edit each host so header forwarding is working and fail2ban for example wouldn't ban the proxy itself.

But I can pass through the whole ZFS pool right? And I I created the datasets in Proxmox as well those datasets would be passed along with the ZFS pool on which they were created?

No you can't passthrough complete pools, no datasets and no folders inside a dataset. Only thing you can do is storing virtual HDDs as zvols on that pool and use them for a VM or you could use some network protocols like NFS to share folder inside datasets over the network.

If using NFS would I then have to create a LXC container, pass the ZFS pool to that container and then use a NFS service in that container to share the datasets with other VMs and LXC containers?

Yes. That would be an option or you could install a NFS server directly on your host.

When you say "virtual HDD" do you mean the same HDD as the VM is running on? Then I guess the DB should be set up for som kind of backup, in case the VM drive crashes.

Yes, that is what i mean. I would run the OS with all programs like webservers, DBs and so on directly of that virtual HDD and not over a NFS share.

I have 32 GB ECC RAM for now and might extend with and additional 32 GB later. But what I was referring to with this question has more to do with ARC tunables like what is being discussed here. However, I have no idea on how to do this in practice on Proxmox.

By default your ARC is 50% of your total RAM, so 16GB right now. Adding more RAM might be a good idea.