Question on sane setup for home

[SanderVeeken]

Hi,

I'm pretty new to Proxmox and Linux in general, I have it running pretty well right now for my needs but it's all super simple; a pihole LXC and some VMs, all running on an old HP EliteDesk. I have to say it's amazing what you can do with an old machine and an advanced tool like Proxmox!

I'm looking to upgrade the machine and would love some advice. I'm staying away from complicated things like HA and Ceph, this is just a very simple homelab and apart from a few "universal" services (like pihole) I'll probably end up using it myself, most of the time.

What I want to get out of it:
- an easy to manage setup
- relatively low idle power, as it would be "doing nothing" or "very little" a lot of the time
- enough memory and CPU power to run some services and an active VM or two at a time.
- a sensible backup scheme that would let me take snapshots of VMs or containers that I could conceivably load in a replacement system in case stuff really goes down the toilet.
- the ability to add storage dynamically, specifically for a TrueNAS type VM.

Things I expect to run:
pihole (container)
TrueNAS (VM?),
a media server (Plex? LXC or VM?),
a Minecraft Server using AMP (just for in house and possible one or two of my kids friends),
Considering PFSense or OpenWRT to replace my aging router, not sure this should be an LXC or VM.
Occasionally a "faffing about" VM. E.g. I had fun installing Kali and practicing stuff with nmap and msf in my network. Or running a clean Ubuntu machine for coding.

It's conceivable at some point I'd want to add a decent graphics card for some AI fun or as a Steam server or however that works. Unless that's too tricky, in that case I'd probably just build a separate gaming system.

I'm doubting a lot on the hardware side of it all. I don't even know whether to go to Intel or AMD. I understand Intel has better idle power at the moment but AMD is lower nm, lower TDP and generally more efficient under load, plus it doesn't complicate things with little.Big.

Next question is storage - I was thinking I'd get a moderately sized drive for Proxmox OS specifically - like 256 or 512. It would hold just Proxmox and some ISOs. Then another reasonably sized drive (1/2TB) for creating VMs on. A third (external) drive to store VM/LXC backups on. And finally, some kind of flexible storage solution for TrueNAS that ideally, I could keep adding disks to (assuming enough ports remain). Is this even possible? Is it a stupid way to go about things?
I hope you have some recommendations! (and that they aren't "buy 3 nodes and a NAS and 10gbps infrastructure")

 

[Kingneutron]

https://www.amazon.com/dp/B0CY27348C?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_1

^ Beelink EQR6, AMD Ryzen. Fantastic deal at under $400 with 24GB RAM. I upgraded mine to 64GB and have extra room now. Add on a 2.5Gbit USB3 adapter and remote desktop is fast enough to be daily-driver unless you want to play 3D games. It's not great for spinning-disk expansion, but I have other servers for that. It has 3x USB3 external SSDs on it tho. Samsung T5, T7 and an SK Hynix 1TB Beetle.

Go for a high-TBW rated drive for OS boot/root. I put in a Lexar NM790 nvme 2280 and it has minimal wear ( ~2% ) after over 1 year of use. Otherwise look for used Enterprise SSD on ebay and the like.

If you're using ZFS, go for CMR NAS-rated spinning drives like Ironwolf or Toshiba N300, and limit your ARC size. 2-4GB is fine for 24GB RAM and you can always add an L2Arc that will survive a reboot (personally I use PNY thumbdrives 32-64GB.)

You didn't mention it, but put everything on UPS power and setup Proxmox Backup Server on separate hardware to take advantage of dedup. It can run on e.g. an old quad-core laptop with 8GB RAM and 1TB SSD.

https://github.com/kneutron/ansitest/tree/master/proxmox
^ Helpful admin scripts

p.s. Buy 1 node, setup your own NAS, and put in 2.5Gbit infrastructure. Equipment came down to gigabit-cheap levels and they use the same CAT5E / CAT6 cables :b

 

[Johannes S]

Go for a high-TBW rated drive for OS boot/root. I put in a Lexar NM790 nvme 2280 and it has minimal wear ( ~2% ) after over 1 year of use. Otherwise look for used Enterprise SSD on ebay and the like.

Hm, did I miss something? The Beelink (although a nice, beefy device homeserver) only has two M.2 2280 slots doesn't it? Then most enterprise ssds won't fit.

I'm pretty new to Proxmox and Linux in general, I have it running pretty well right now for my needs but it's all super simple; a pihole LXC and some VMs, all running on an old HP EliteDesk. I have to say it's amazing what you can do with an old machine and an advanced tool like Proxmox!

Which HP elitedesk model do you have now? It might still have us as a backup server depending on how many disks/ssds it can fit.
The ProxmoxBackupServer doesn't need a lot of power (see https://pbs.proxmox.com/docs/installation.html#minimum-server-requirements-for-evaluation ), but you will want to use SSDs for backup storage.

I'm looking to upgrade the machine and would love some advice. I'm staying away from complicated things like HA and Ceph, this is just a very simple homelab and apart from a few "universal" services (like pihole) I'll probably end up using it myself, most of the time.

What I want to get out of it:
- an easy to manage setup
- relatively low idle power, as it would be "doing nothing" or "very little" a lot of the time
- enough memory and CPU power to run some services and an active VM or two at a time.
- a sensible backup scheme that would let me take snapshots of VMs or containers that I could conceivably load in a replacement system in case stuff really goes down the toilet.

For this I would use a extra device so you can seperate your main server and the backup. For this I would keep your old elitedesk and install ProxmoxBackupServer on it as primary backup target from your ProxmoxVE host. Since you should always have three copies of your data (production, one on different media and another one offsite so not in your home) I would also rent a cheap vserver and install PBS on it too. Then you can setup a sync job to pull your backups from your home pbs to the vserver. See https://pbs.proxmox.com/docs/storage.html#the-3-2-1-rule-with-proxmox-backup-server and https://pbs.proxmox.com/docs/managing-remotes.html for more information how to do this. My offsite vserver at netcup cost me around 10 Euro per month. These are cost I can live with for peace of mind. Of course any other vserver provider would work too. Or use a PBS Cloud provider like tuxis.nl free tier (limited to 150 GB) or Inetts cloud pbs service if you don't want the hassle to maintain a vserver.

- the ability to add storage dynamically, specifically for a TrueNAS type VM.
TrueNAS (VM?),

For TrueNAS you really should have a dedicated sata/storage adapter since TrueNAS recommends to not run TrueNAS as a vm but if you want to do it they only support if you have a dedicated controller for your discs: https://www.truenas.com/community/r...guide-to-not-completely-losing-your-data.212/ See also this thread: https://forum.proxmox.com/threads/best-practice-for-truenas-virtualized-on-proxmox.133536/ and other trheads on the TrueNAS community and Proxmox forums. However: Any storage attached to this controller and managed by TrueNAS can't be managed via ProxmoxVE anymore.
This is what I did for my NAS: I got myself a SATA controller and attached some used datacenter ssds to it which are now the storage used by my TrueNAS vm. There is still room to add some HDDs if I ever need more space.

Of course you don't need to use TrueNAS, you can also use another NAS OS (where you also would want to have your dedicated controller) or just a normal Linux vm/lxc with a samba or nfs server for sharing files if you can live without a shiny web ui (and even that yould be mitigated by using something like webmin). The Linux vm/lxc wouldn't need a dedicated controller card obviouvsly and in that case you could manage the storage dynamically from Proxmox.

a media server (Plex? LXC or VM?),

For a VM I would recommend to get a dedicated graphics card so you still have a working console on your machine for accessing the console even if the WebUI of Proxmox isn't working for some reason. I personally prefer VMs to LXCs in most cases but Plex or Jellyfin are two of the applications I would make an exception since it's easier for them to use the iGPU without blocking it for the rest of the system.
For anything which is run as a docker container I would always use a VM (since LXCs and docker doesn't always play nice in Proxmox after Updates, look up this forum for older threads) but luckily Jellyfin and Plex can both be installed without docker in a lxc so no problem for your usecase

Considering PFSense or OpenWRT to replace my aging router, not sure this should be an LXC or VM.

I would highly recommend to rethink this. If for some reason your homeserver is not working your whole network (or even your internet access) will be broken too. I wouldn't risk this, especially if you don't want to annoy your family Now I know that it can be fun to play around with such stuff so if you want to learn about routing and firewalls ProxmoxVE is a great environment to setup a lab environment where nobody is bothered if you break it.
For OpenWRT, PFSense or OPNSense I would try to put it on your existing router or get a new dedicated one if you really want to use it as the core node of your home network. For the same reason I wouldn't run pihole or adguard on your proxmox node without at least a kind of backup/failover DNS on your main router. I run Pihole on my promox main server but I also have a second Pihole on a second Proxmox node and use keepalived to always have a working DNS server at home: https://delightlylinux.wordpress.co...zed-pi-hole-with-keepalived-and-gravity-sync/

Next question is storage - I was thinking I'd get a moderately sized drive for Proxmox OS specifically - like 256 or 512. It would hold just Proxmox and some ISOs. Then another reasonably sized drive (1/2TB) for creating VMs on. A third (external) drive to store VM/LXC backups on. And finally, some kind of flexible storage solution for TrueNAS that ideally, I could keep adding disks to (assuming enough ports remain). Is this even possible? Is it a stupid way to go about things?

As said above: Your backups should live outside of your main server, be it an external disc or your old elitedesk. It might also be an option to seperate vm/lxc backups and bulk data. I only store my vm/lxc backups on my vserver PBS, my bulk data from my notebook and NAS is backed up to a cloud storage with restic. It might be worth an idea to have a PBS vm and use your external drive as "removable datastore" in PBS so your backups are not at the same place as your actual storage. But personally I would recommend to just use your old elitedesk as PBS (If you are worried about energy cost: Use WakeOverLAN to only start if for the backups and shut it down after finishing them). If you don't want the costs of a vserver you could also attach your external disk to the PBS as "removable datastore". In this case you would have three copies:
- First your production data
- Second: On the internal storage of your elitedesk PBS
- Third: On your external drive which you might also be able to store outside of your place (e.g. at a friend or family members flat) so in case of a fire you still have your data. If you have two external discs and swap them regulary this might be a cheap although manual way to have offsite backup.

For the actual server the optimal setup would be to seperate operating system and vm/lxc/data storage, each one on a sw raid mirror. However depending on the used hardware and budget this is not always feasible. I myself have a small ProxmoxCluster made of used mini-pcs. They only can fit one sata ssd and a M2 nvme. I created a ZFS mirror out of them during the Proxmox install and now everything (Operating systems and vms) have the same pool. This isn't ideal but on the other hand ZFS flexibility make this feasible: Anything not used by the operating system can still be used for VM, lxc and data storage, ZFS is way more flexible than LVM in that regard.
So: If you have space and budget for four ssds/disks I would have two for one ZFS mirror for the operating system and another one for the vms and lxcs. If you don't have that I would just create a mirror out of two ssds and be done with it.