Sign off Proxmox Homeserver Hardware

[FabianX2]

Hey,

I´m new to all this and started building a Proxmox Homeserver. Aside from experimenting with virtualizations, it will host FreeNas under Proxmox.

Setup so far:

2*16GB ECC Ram (or should I buy another pair straight away?)
Sandisk Extreme Pro 500gb NVMe
IBM ServeRAID M1015 (IP)
5*8TB N300 (Passthrew woth M1015 to FreeNAS)
R7 2700
X470D4U
BQ Straight Power 11 450 Watt
3x120mm Cooler
Inter-Tech EM-01


So while I am waiting for the last parts to get shipped I am wondering about my drive Setup. And started to do some research. By now I´m completely confused, so I would appreciate some help regarding the following questions.

1. Initially, I intended to use the NVMe as OS Drive for Promox. Now I`m not so sure anymore. I read a lot about SSDs waring super fast due to lots of writing. Not big but very often. As I use a cheap consumer NVMe SSD that’s a problem isn`t it?

Secondly, Proxmox doesn’t need a lot of space and I am wondering if a 500gb NVMe wouldn`t be wasted potential. And to add another point: I did read that the OS drive doesn`t have to be fast. At last I am thinking about using ZFS with two drives to minimizes downtime. I would really appreciate some recommendations as I`m completely in the dark about which direction to go. One or two drives? NVMe? HDD? Sata SSD? And finally which size and Model?


2. I did forget the drives to store and run the VMs. What’s the recommended setup here? Is it better to have more little SSDs or fewer big ones? As I have no Hardware Raid Controller (beside the one in ip mode for FreeNas) I thought about some kind of mirroring. Software Raid 1 or ZFS?


Again thanks for the help. Please do not recommend super fancy expensive business solutions. I´m on a budget here and using it at home for fun. And although not to forget: I´m a total noob.



Fabi

 

[Astraea]

1. I have been using both HDDs and SSDs with Proxmox and have had great reliability so far, my SSDs are coming up on 2 years old and so far going strong. I have been using consumer SSDs as well. I get the Samsung Evo series SSDs for use in both my desktop workstations and my servers. I also use my Proxmox storage for a few VMs that I need to start before my storage servers are running should I need to shut things down for maintenance on either the other Proxmox nodes or on the storage servers and have not had any issues with Proxmox sharing the OS drive for VMs. I have not done a ZFS setup yet and currently, I am running 3 nodes with dual SSDs in a mirror hardware raid and another node with dual HDDs in a mirror hardware raid and another set of 6 drives in a hardware raid 5 setup. Though when I get new server hardware I will be moving to ZFS as I plan to run FreeNAS as well and want to run all my servers and hypervisors under ZFS for data integrity and redundancy.

2. I have most of my machines running their storage from an NFS shared storage server so that I can migrate my VMS between 1 of 4 nodes I have running. If you are wanting to do ZFS you are from what I know better with smaller drives as you want to minimize the time it takes to resliver should a drive fail, but someone with more ZFS experience should be able to provide better direction. If you have multiple NVMe ports or SATA ports on your motherborad I believe Proxmox will allow you to use ZFS on these drives to create a mirror or other ZFS redundant drive pool.


The best advice I can give you would be to once you get all your hardware is to set it up and play around with it before you make it part of your home network/lab. You will want to not only burn in the hardware but also learn how things are configured as you will make errors and sometimes its best to start fresh a few times as you get your head wrapped around how things are working. I specifically have VirtualBox installed on my desktop for this so that I can test ideas out before putting them on the Proxmox stack and when I get new hardware I test it for a good 2 weeks before I put it into my version of production. Take your time and make lots of notes on how you set things up and what commands or processes you need to follow will definitely help you set things up and also help you remember after its all set up and working and you need to make a change or do maintenance.

 

[FabianX2]

I did as told and started playing. I used a basic setup the last 24 hours (Setup as in first post just without the raid controller and the 8Tb HDDs) and am trying to get used to all the new stuff. Some VMs some LXC. Nextcloud, FreeNAS and a few Linux Distros. So far I´m impressed by the performance stability and a bit overwhelmed by the complexity. It´s not only Proxmox things like IPMI, Unifi Controller, IoBroker, PiHole, Nextcloud ...etc. totally blew my mind - in a positive way. Not unexpected as I am a total noob and trying to build a geeky smarthome all at once

The rest of my drives to complete my Proxmox setup will arrive tomorrow. So before it gets serious I would like to lay out my plans:

1.: 2x120gb SSD (ZFS mirror): Just for the Proxmox OS - I know a lot of space will be unused. But I could not get any smaller reasonably priced SSDs. Ideas how to use the rest?
2.: 2x500gb NVMe SSD (ZFS mirror): For the running VMs.
3.: 2x500gb HDD (ZFS mirror) old discs for now. Isos, backups, snapshots etc.
4.: 5x8TB HDD (ZFS RaidZ1) all exclusiv for FreeNAS through a Raid Controller in IP Mode.
5.: 1x4TB HDD: Most of the Space for Nextcloud running as VM on Ubuntu Server in Proxmox.

The Idea is to back up snapshots etc. (3./5.) to FreeNAS (4.). Not sure how to back up Proxmox Os its self (1.). And if I should change the old HDD (2.) for some cheap SSDs. 120Gb should be enough keeping in mind that its just temporary as it will be backed up to FreeNas (4.). Although if to buy a buddy for the last lonely 4TB HDD (5.) and mirror it like the rest.

Thanks so far. Looking forward to the new hobby

 

[Astraea]

1. I would use the rest for ISOs as they normally don't take up a ton of space unless you are running a large variety of operating systems. This allows you to keep a good division between the Proxmox os, ISO images and then having your 2 500GB NVMe drives for VM storage.

I would if possible drop the 2 500GB HDDs as well as the 1 4TB HDD and add 3 more 8TB HDDs for your FreeNAS storage to allow you to run RaidZ2 or even RaidZ3 depending on how much space you are wanting for storage. The downside to RaidZ1 is that should a drive fail you cannot have another failure while the failed drive is replaced and then brought back online. I would use some of the space from your Proxmox OS drives to keep backups of your VMs as they are normally not large files as you are only backing up the VM operating system or even use some of your FreeNAS storage to store the backups after they complete as a second copy. As for Proxmox itself, I don't have backups of the install as I keep greate documentation and setting it up again is pretty quick and with mirrored drives, you have some redundancy there as well. I would export a CIFS share for Nextcloud to use as an external storage as it will keep the size of your VMs storage drive down and allow the data to be protected under a more redundant ZFS pool.

 

[kappclark]

Let us know how you like freenas -- I just created a FreeNas 11 VM with 32G boot disk and 4x50 GB hd in raid 10 ... first time for me using freenas and it seems pretty intuitive , and so many tweaks available ... Sure beats buying the Synology..many ways to share data...many plugins to explore...

I may build a stand-alone freenas box, booting from 8GB USB stick ... I saw a video showing how to install freenas to 2 USB sticks, creating a mirrored pair -- pretty slick ..