New Proxmox server - considerations and disk layout

[gameover-nl]

Hello,

Currently I have an old server at home I wish to upgrade.

This server is an old i5-2500K, with 16GB of RAM, a 120GB SSD for booting and a Areca RAID controller with a 2x500GB mirror and a 6x2TB Raid 6 volume. Current OS is Windows Server 2016.

I use it for the following purposes:

- DHCP
- Local DNS
- Mailserver
- File sharing to my LAN ( about 10 devices, computers, laptops, smartphones)
- HyperV host with currently one Windows 10 VM (Sonarr, Radarr, Sabnzbd)

Seeing that this server is nearing almost 10 years of age, I want to replace it and also get something more energy efficient and silent. Also my disks are filling up. This server will be re-purposed as a backup server for critical files and VM's, so I will power it on once a month or so, as "Cold backup".

The new server will be build around a Fujitsu D3644B mainboard, with 2x16GB of ECC RAM and a Core i3-9100 CPU. It will be powered by a PicoPSU 160XT.

As disks I have the option for a NVME drive and 6 Sata drives.

My idea was to buy a 1TB NVME drive and use this for Proxmox and the VM's. (Something with high TBW, like a PNY XLR8)

I also ordered 5 Seagate Barracuda 5TB 2,5" drives, for my data storage needs. This fills 5 SATA ports. Finally I was thinking about a 500GB/1TB SATA SSD, (Crucial MX500?) as a cache/accelerator for the HDD drives.
This build should be quite energy-efficient when idling (not exceeding 20 Watts, currently my server takes about a 100-110 Watts) and silent, and also powerful enough for my purposes.

I have several kinds of data to store on the server. Some critical (Business administration, documents, photo's) and some less critical like media files and program installers, updates, etc. All these files need to be accessible over the network for the client devices. The client devices don't store any data locally, it all goes on the server. (Well, except the OS, installed software and IMAP mailcache)

Now my main question is, what is the best way to handle the disk layout/file sharing?

2 of the 5 disks in a mirror for the critical files? (5TB usable) and then the 3 other disks for the media? And then somehow add the Sata SSD as cache/accelerator?

Or just make a big ZFS pool with all disks? (like a kind of Raid5)

There are also 1TB USB disks available, which I currently rotate on a 2 weekly schedule, for a "not on the server" backup of the critical files. The one not in use resides on another address, in case my house burns down or something horrible like that.

Advice on how to best handle this is appreciated!

Also, I think there are containers for the the stuff I now have in my Windows 10 VM, like Sonarr, Radarr and sabnzbd, so I can probably ditch the Windows 10 VM. A mail server alternative that can use a smarthost as outbound relay and receive my mail from spamexperts would also be appreciated. Maybe Proxmox Mail Gateway? There are multiple mail domains in use. (I use Spamexperts as inbound filtering host and outbound smarthost, my own IP address changes every night here in Germany...)

I know, a lot of questions, thanks in advance for taking the time to answer some.

 

[wolfgang]

Hi,

CPU: if intel use the 9100t instead the 9100 and if you have the chance to get (not really available) take the 9300t
If you take the 9100 the AMD 3600 would be the better choice.
Same power consumption in avg but 6 cores.

ZIL device: an MX500 is not capable to work as a cache device. You need a device with high 4k sync capabilities this is normally enterprise-grade disks.
But ZFS uses it only for small sync writes.

Raid Mode: I would if you want high redundancy a Raid Z2, there you can lose 2 disks and your pool can operate.
But keep in mind this cost CPU and disk performance but I guess in your home setup this works fine.

The USB disk can be still used for backups.

 

[KatyComputer]

Wolfgang - you introduced a few new concepts to me. I see Servethehome has a few articles discussing ZIL devices & Raid Z2, do you have a favorite tutorial?

AMD 3600 seems to be a good recommendation, what system board(s) do you like for this application, is it worth the extra coin for Z370 based boards.

Sorry for the follow-up, but I have a couple similar projects in the wings.

 

[gameover-nl]

Hi,

CPU: if intel use the 9100t instead the 9100 and if you have the chance to get (not really available) take the 9300t
If you take the 9100 the AMD 3600 would be the better choice.
Same power consumption in avg but 6 cores.

The difference between a 9100t and a "normal" 9100 is the maximum TDP. This is accomplished by clocking down the 9100t to a slower base and maximum speed. (3.1/3.7Ghz vs 3.6/4.2Ghz) During normal operating conditions, the normal 9100 doesn't use more power then a 9100t. It can only go faster when asked, and then it consumes more, but it also does more (faster clocks). So if they're both idling or under 50% load, the power consumption between the two is comparable, because of throttling down (Speedstep) and C states.

For the AMD Ryzen 5-3600 there are not really energy efficient chipsets available. It also has no integrated graphics, so it requires a extra graphics card (I want to be able to troubleshoot and be able to get into the BIOS) and it's more expensive, without the cost for a extra graphics card. I also don't need the extra processing power, so it's just a waste of money and power consumption. The CPU is € 30,- more expensive and the extra graphics card will also be around € 30,-.And then you have the extra power draw of around 15-20 Watts permanently.

ZIL device: an MX500 is not capable to work as a cache device. You need a device with high 4k sync capabilities this is normally enterprise-grade disks.
But ZFS uses it only for small sync writes.

Thanks for clarifying, can you advise a make/model for this purpose? So for the purpose of a "Accelerator" the MX500 is not advisable in a ZFS setup?

Raid Mode: I would if you want high redundancy a Raid Z2, there you can lose 2 disks and your pool can operate.
But keep in mind this cost CPU and disk performance but I guess in your home setup this works fine.

The USB disk can be still used for backups.

If a make a Raid Z1 pool, with a decent SSD for ZIL, then the SSD should "cover up" the performance issues of the normal disk performance, correct? How to accomplish the sharing? Make the ZFS pool in Proxmox and share from Proxmox, or install a VM and passthrough the disks and leave it to the VM? (What OS should I use for that?)

Seeing as everything gets regular backups, it doesn't have to be "ultra safe" so I guess Raid Z1 is good enough, Raid Z2 I think is overkill for my purposes.

Thanks for your replies!

 

[wolfgang]

Hi KatyComputer,

I personally use this here [1]
But if you need IPMI this is maybe an option.[2]

I guess a B450 is enough and saves a bit of energy.
Avoid X570 because they need much more energy.


1.) https://www.asrock.com/mb/AMD/B450 Pro4/index.asp
2.) https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U#Specifications

 

[wolfgang]

This is accomplished by clocking down the 9100t to a slower base and maximum speed. (3.1/3.7Ghz vs 3.6/4.2Ghz) During normal operating conditions, the normal 9100 doesn't use more power then a 9100t.

The t version selected chips of a higher quality so less energy goes in heat over. This is the reason why you get the known T version without problem and the T version is hard to get.

And your right, if you use a GPU for this AMD build the energy efficiency, is no more existing.
But I meant to run this Headless(without GPU).
For debugging you can take your gaming GPU.

Thanks for clarifying, can you advise a make/model for this purpose?

Samsung SATA SM series, Intel DC S4500 series, Intel Optaine Memory
The disk has to be huge 32 Gb is completely enough.

So for the purpose of a "Accelerator" the MX500 is not advisable in a ZFS setup?

It will accelerate but not as you aspect.
The MX500 can do about 1000-2000 IOPS and Intel Optaine Memory can do 80000 IOPS

If a make a Raid Z1 pool, with a decent SSD for ZIL, then the SSD should "cover up" the performance issues of the normal disk performance, correct?

As I wrote the ZIL will ONLY accelerate small sync writes. All blocks witch are written asynchrony or large will be written directly to the datastore.

How to accomplish the sharing?

There are many ways and it depends on some factors like what technic do you use (CT|KVM).
Do you share this data outside the host?
Which OS should access this data?

 

[gameover-nl]

Samsung SATA SM series, Intel DC S4500 series, Intel Optaine Memory
The disk has to be huge 32 Gb is completely enough.

Thanks, I will look into those.

It will accelerate but not as you aspect.
The MX500 can do about 1000-2000 IOPS and Intel Optaine Memory can do 80000 IOPS

Right. I shall look for another device.

As I wrote the ZIL will ONLY accelerate small sync writes. All blocks witch are written asynchrony or large will be written directly to the datastore.

OK, is there a disk setup where the writes first go to the SSD and it then gets distributed to the harddisks? I don't have to use ZFS.

There are many ways and it depends on some factors like what technic do you use (CT|KVM).
Do you share this data outside the host?
Which OS should access this data?

All the data (Except what the VMs themselves need) needs to get shared outside the host. The server is the storage for all the data in the LAN. The clients are mostly WIndows 10 computers.

So the server shares the "My Documents" and all the media files, installers, etc. Like you would use a NAS. The clients don't have any local data, except the OS, installed programs, IMAP cache and temp files.

 

[wolfgang]

All the data (Except what the VMs themselves need) needs to get shared outside the host. The server is the storage for all the data in the LAN. The clients are mostly WIndows 10 computers.

Then use a samba share. You can run this service in a container and use mp to pass the disk to thw CT.

OK, is there a disk setup where the writes first go to the SSD and it then gets distributed to the harddisks? I don't have to use ZFS.

You could use lvm cache or bcache. Both are not supported by Proxmox.
I don't know if they are working well.

 

[gameover-nl]

Hello Wolfgang. Thanks for your replies so far. I will do some testing to see what is possible and performs the best as soon as I have my hardware complete.