Using PBS with a NAS box (Tutorial)

[MaLe]

Hi there,

first of all, this tutorial is mainly addressed to beginners. As usual with Linux based OS, you can do nearly anything you want. This is only one solution and it does not claim to be the best. There are thousands of others ways, that might bring the same result or even a better. So maybe it could be a good idea to use other hardware or disks, etc..

Sometimes it's an easy way to use NAS boxes for backup because they are small, can be moved to other places and should not be located at the server room in case of fire. Also, they do not have to be mounted in 19" racks and can be placed everywhere, for example on a shelf. And finally, a NAS device is usually way cheaper than a rack mounted server. But as you maybe know, it's not a perfect solution to run a PBS as VM on your PVE and mount external NFS or CIFS shares as backup storage. So my solution is to run the PBS on the NAS itself. Therefore a suitable NAS device will be needed, and I found something that fits pretty good: The TerraMaster 423-series.

Advantage of these devices:

- They have no locked operating system, so you can install whatever you want
- Standard x86 hardware with Intel quad core CPU, not designed for pocket calculators...
- Can be equipped with internal M2 NVMe (2 slots) for the PBS OS
- Can be upgraded with DDR4 memory, which is important for the use with ZFS
- Two network connectors can run up to 2,5 Gbit/s or acting as bond with 5Gbit/s (10G LAN on other models)
- last but not least there is a HDMI connector for the display

These are my suggestions:

- TerraMaster NAS device 423 series (for example F4-423 / 4 bays)
- TerraMaster 16GB DDR4 RAM module (one = 20GB / two = 32GB)
- NAS NVMe SSD M.2 (you can use 2 pieces, so OS is running on RAID)
- SATA 6Gbit SSD/HDD (how much you like, depending on the free bays)

Memory for ZFS: The calculation is 2GB base memory + 1GB/1TB. Example: 16TB = 18GB memory.
SATA Disks: I would recommend to use SSDs. In my experience, verify jobs on large data stores can need several hours on spinning disks instead of minutes on SSD.

How to assemble:

- remove all cables and put the device with front side down on the desk
- remove the back cover with the fan (4 screws)
- unplug the fan from the mainboard (be careful to not damage the cable)
- remove the metal case and slide it in upper direction
- remove the USB dome with the factory OS (on the edge of the inner side of the mainboard)
- plug in the memory module (easily reachable on the back plane)
- plug in one or two of the NVMe M.2 disks (easily reachable on the back plane)
- put metal case back in place and reassemble the back cover
- remove the HDD frames on the front and mount the storage disks
- plug in USB stick with PBS ISO image (can be easily flashed with balenaEtcher)
- plug in power and connect USB keyboard + HDMI to display
- after power on go to BIOS and set up the boot order (I disabled all except the NVMe devices and USB port for installation)
- install PBS on the internal NVMe disk(s)
- reboot the PBS and set up your ZFS RAID on the HDDs

Please take note, this setup is basically working, but I will do some testing during the next weeks to get experiences in terms of speed & reliability. I will post my results as soon as I have new knowledge.

Please feel free to leave your comments or recommendations. If you have suggestions how to improve this setup, I would be happy to know it.

Best Regards,
Martin

 

[Dunuin]

Performance of HDDs would be way better when you would use some of the of the NVMes space as special metdata devices. So for example 1MB grub partition, 512MB ESP, 32GB for the partition the rpool of PBS is installed on and the rest of the capacity as a forth partition added to the HDDs pool as special devices.

 

[MaLe]

Many thanks for your response. Thats a great recommendation, especially due to the fact that todays NVMe have much more storage space than PBS need. So this is a meaningfull use of the remaining space.

 

[misku]

@MaLe thanks for the guide. I've been looking to build a mITX custom NAS, but it would always be physically larger than those proprietary super small NAS boxes. I didn't know you can install Proxmox on TerraMaster - that's awesome!

 

[MaLe]

Thanks for your response. After testing this setup for 3 months, there are no issues so far.

Based on my experiences, here are some additional suggestions:

- in case of using spinning disks, free space on NVMe's can be used for additional partition to speed up the disks (for example as special device)
- NVMe should be data center edition with PLP, to prevent data loss through deleted cache in case of power interruption
- after installing RAM extension on TerraMaster, you have to wait for about 1-2 minutes during first power on because of RAM check. Be not surprised.

Regards, Martin

 

[misku]

Thanks for more tips! I did some research and the thing that worries me is the temps in the MB compartment. The MB sits in a pretty tight space with no fans (just ventilation holes). This space is apparently less than 4cm high, so it would most likely be a problem to fit a 40mm fans there. I've read reviews where people complained about the NVMe SSD temps once loaded.

This amount of space looks the same on T6 and T9, although T9 includes a 1U (flex) PSU inside. I guess one could try to fit 30mm fans in there.
What are your experiences with NVMe SSD/CPU temps at full load?

 

[MaLe]

No issues with the temperature so far. We made a stress test and currently running a daily backup job (3-4 hours). But we are using a F4-423 4-bay desktop device, so I don't have any experiences with the rack mount types. The F4-423 has two massive fans on the back side, that should generate enough airflow through the chassis. No concerns.

 

[misku]

IC, F4-423 is a bit different beast, although the same CPU. Thanks for the info.