First exp - Multiple VM's and optimization

[Overact]

Hi everyone, first of all i'm sorry if that's a thing that already has been said but i haven't found the anwser i was looking for.

I'll soon build my first PC to make it a NAS and a gaming server for several game.
Can someone tell me if i'm in the good direction ;
- I'll install Proxmox
- Then i'll make a VM to install TRUENAS for plex and for own cloud storage.
- I'll make a VM with a windows OS to run my games servers (for like rust, minecraft, palworld and stuff..)

The questions i got are :
Is this the good way to do it ?
Should i make one VM for each game i'd like to run a server at the same time or just go all in one VM ?

Also, rn i'm looking to buy storage. Can someone tell me the best HDD storage that you guys got, price quality wise ?
Should i buy 5400 or 7200 ?
Can i, in the future, if i make a raid 5, change a upgrade the storage with more powerful/bigger HDD?

Sorry for the "noob" questions and thank you for your time !

 

[kobe_highlander]

Hi Overact, not an Proxmox Expert myself but I am currently building something similar w/o the Gaming part.

1. The first part is perfectly OK
   1. Proxmox basis
   2. VM for TrueNAS
   3. I also have Plex run in a separate VM
2. Second part about Gaming
   1. I do not know anything about that
   2. If it's just Services that needs to run, it might be well suited for an LX Container (LXC)
      1. If that's an option, I'd have one LXC per Game Service
      2. If a service needs complex tunig, set-up, TrueNAS Storage etc, than maybe a VM is better suited
   3. Maybe same Gamers are lurking in the Forum which are able to drop a hint
3. Some Thoughts
   1. I found out that if I share storage from TrueNAS (I mainly use NFS) it is less cumbersome to mount it in a VM
   2. I also utilise a Samba Domain (not hosted on Proxmox) and Kerberised NFS mounts, which also is easier to be done in a VM .. .at least for me
4. HDD topic
   1. I recently found a good deal on "old" WD (ex HGST) 12 TB SAS drives (7200RPM)
      1. I have had very good experiences with drives from WD/HGST, SATA and SAS
      2. Those I found are only SAS 6Gbit (STAT connector), but my use case will rarely utilise the full 12 Gbit SAS potential
      3. There is an option to look for "refurbished" / "recertified" drives, but it's a bit of a gamble
   2. Speed: If anything, the most speed gain comes from the storage set-up type you chose
      1. If money is not much of a concern, most suggest to go for striped mirror set-ups
      2. But in my book, chose your set-up based on your own use case, i.e. fault tolerance, needed speed etc.
   3. If you build a server that sits in a rack or somewhere where you cannot heard it, I'd go fort 7200. They do tend to be a bit louder.
      1. Again it should be based on what you need.
      2. Found this: https://pureinfotech.com/difference-5400-7200-rpm-hard-drives/

Hope that helps a bit

 

[louie1961]

Maybe I am missing the point, but why install TrueNAS on top of Proxmox? It seems to me their are less resource intensive ways to run Plex and ownCloud inside Proxmox. owncloud has an LXC container available for it or you could stand up either a VM or a LXC container to run all your docker apps. TrueNAS is going to want to use a bunch of memory for caching, but you already have ZFS and caching inside of Proxmox. Seems redundant to have it twice. Also, this is my personal preference, but I prefer to have my storage in a separate physical device and serve it up to Proxmox using SMB/CIFS, NFS, etc. That way you also have a perfect backup destination for your VMs.

 

[kobe_highlander]

Louie, you are definitly not missing the point, you have one ;-)

1. Ideally yes, TrueNAS separate serving your Storage needs
   1. also, single point of failure aspects
   2. better utilisation of ressources#
   3. etc
2. In my case I bought an HL15 from 45drives recently and stuffed in an Superrmicro EPYC board (EPYC 3rd gen with 32 cores/64 threads)
3. In my case I judged it to make sense to go for it as such
   1. Proxmox on SATA DOM plus 2TB mirrored NVME 256 GB RAM for all the VMs, LXCs
   2. TrueNAS in a VM is managing the 15 HDD Drives plus 10 NVMEs
      1. Backup Storage: 8 Drives as ZFS RAIDZ2 + mirrored NVME based Log VDEV + Cache on NVME
      2. Data Storage: 7 Drives as striped Mirrors + Spare + mirrored NVME based Log VDEV + Cache on NVME
      3. Fast Data Storage: 4 striped mirrored NVMEs
      4. I hand out Storage as kerberised NFS plus SAMBA for our Windows PCs/Laptops
   3. Proxmox, TrueNAS, VMs are all joined Domain members
4. If I would have installed TrueNAS directly I would have had to ramp-up another beefy Server for Proxmox endevours ... enough money spent so far.

But you never know, I started with a 2 bay qnap and rhaspberry Pi a couple of years ago and played with INTEL NUCs for a while (still running) and this is my newest baby. I might go the separate route in a few years but for now this works for me.

 

[louie1961]

TrueNAS in a VM is managing the 15 HDD Drives plus 10 NVMEs

How are you passing these drives through to TrueNAS?

I started with a 2 bay qnap and rhaspberry Pi

I kind of went the other way. I started out with a Z640 workstation running an e5-2690v3 and 128 gb of ECC ram. I had everything on that box...pfsense, openmediavault, you name it. Every time I screwed something up, the entire network would go down and the wife would be very upset with me. Now I have a very distributed environment: N100 box running pfSense, HP Elitedesk Mini G9 (i5-12500T) as my main proxmox node, Synology NAS device, and another N100 box running proxmox for my backup server (just using a couple of Debian instances {each on different VLANs} running rsync, nothing fancy). VMs get backed up to the NAS (some daily, my wordpress sites get backed up every two hours). Data is stored on the NAS, only the VM images are stored on Proxmox. The NAS gets backed up nightly to both the N100 RSYNC box and AWS Glacier.

I cut my electrical use in half. My total setup including two 2.5 gb switches, a wireless access point, all the hardware mentioned above, and two internet modems (redundant WAN), and my UPS all draw 85 watts at the wall. In my previous set up, the Z640 itself was drawing 100 watts on average)

 

[kobe_highlander]

The famous wife factor ... tell me about it ... "Honey, do we have Internet?" How do they say? Happy wife happy life!

Anyway, you touch another even more important point ... Backup. That saved my butt a couple of times. I still have to figure out my strategy with the new box, as it is still work in progress. It will definitely be a mixture of Snapsots and internal / external backup w/version history for certain data.

Regarding the Drives:

• you can pass through PCIe to a guest VM in Proxmox.
• 8 SAS drives are connected directly to the Motherboard (2 SAS Mini HB connectors running from backplane)
• the other 7 are hooked up to an Broadcom SAS HBA (Mini SAS HD Backplane to Slim-SAS).
• For the NVME's I used cheapish PCIe extension cards and activated bifurcation in BIOS.
• In Proxmox you add PCIe Hardware to the guest VM (individually for the NVMEs and the two SAS Controller - onboard and HBA)
  • You lose access to passed through hardware in Proxmox but gain full control within the VM
  • The same way you can add a graphics card to Plex for encoding support

Last but not least: electrical bill ... that's a good footprint you have ... mine's so much bigger. We plan to go Solar Support in the next two years .. but that's another story.

 

[Dunuin]

Is this the good way to do it ?

Depends...you should learn about ZFS, requirements and passthrough in case you want to use TrueNAS.

Should i make one VM for each game i'd like to run a server at the same time or just go all in one VM ?

Depends...you will have to buy more windows licenses and need beefier hardware if you want to do that. From a management and security standpoint I prefer to run each service, that doesn't belong together, as its own guest.

Also, rn i'm looking to buy storage. Can someone tell me the best HDD storage that you guys got, price quality wise ?
Should i buy 5400 or 7200 ?
Can i, in the future, if i make a raid 5, change a upgrade the storage with more powerful/bigger HDD?

You should get proper SSDs as VM/LXC storage. HDDs are only useful as cold storage. Raid5 is also not great as a VM/LXC storage because of worse IOPS performance. Again, best to use it for cold storage. Make sure to buy CMR HDDs and avoid SMR HDDs. Especially when using CoW like ZFS does.

Maybe same Gamers are lurking in the Forum which are able to drop a hint

Minecraft runs fine on linux. No idea about Rust and Palworld. But security wise I would use a VM anyways as I don't like to run less secure LXCs for anything that you port-forward to, what you probably want when hosting game servers.

There is an option to look for "refurbished" / "recertified" drives, but it's a bit of a gamble

Another gamble is shucking external USB HDDs. But usually the best deal unless you are unlucky getting some SMR disks as USB disks are way cheaper than retail or bulk disks. Like an 8TB WD Elements Desktop for 120€ while a same sized WD Red Plus would be 170€.

Every time I screwed something up, the entire network would go down and the wife would be very upset with me.

Have a look at pfsync. You can run two pfsense VMs on different nodes with failover. If you then kill a node running pfsense your wife wouldn't even notice it with maybe a second of downtime.

 

[alexskysilk]

Proxmox on SATA DOM plus 2TB mirrored NVME 256 GB RAM for all the VMs, LXCs

careful with doms. they're typically designed to house a nonvolatile distro such as vmware, and have pretty poor write endurance. if you insist on running proxmox on a dom, I'd offload the entire /var directory to another drive, at minimum.

TrueNAS in a VM is managing the 15 HDD Drives plus 10 NVMEs

I see people do this all the time, but I dont really know why. What are you getting from Trunas you couldnt host directly on the pve host? If the answer is "security," I can only think "this is for home use, who cares."

Backup Storage: 8 Drives as ZFS RAIDZ2 + mirrored NVME based Log VDEV + Cache on NVME

two things- backup storage serves very little purpose sitting atop the same machine as your production; two- log and cache are probably providing no utility for this usecase. If you still have your qnap it could probably do the trick nicely for backup.

 

[Overact]

The first part is perfectly OK

1. Proxmox basis
2. VM for TrueNAS
3. I also have Plex run in a separate VM

Why would you have plex in a separate VM ?

1. If it's just Services that needs to run, it might be well suited for an LX Container (LXC)
   1. If that's an option, I'd have one LXC per Game Service
   2. If a service needs complex tunig, set-up, TrueNAS Storage etc, than maybe a VM is better suited

I'm not familiar with container at all so rn i don't really know which one's the better

Maybe I am missing the point, but why install TrueNAS on top of Proxmox? It seems to me their are less resource intensive ways to run Plex and ownCloud inside Proxmox. owncloud has an LXC container available for it or you could stand up either a VM or a LXC container to run all your docker apps. TrueNAS is going to want to use a bunch of memory for caching, but you already have ZFS and caching inside of Proxmox. Seems redundant to have it twice. Also, this is my personal preference, but I prefer to have my storage in a separate physical device and serve it up to Proxmox using SMB/CIFS, NFS, etc. That way you also have a perfect backup destination for your VMs.

You may not and that's why i'm asking ahah, i'm a complete begginer with no exp at all in that kind of stuff
But my plan was using TrueNAS for the easy raid configuration and the ZFS that come with it, but i didn't know about Proxmox that could do the same.. It just feel like TrueNAS was the easiest way to begin with. I'm building with a I7 6700K and 32mb of ram. I just assume performance won't be a problem but may i'm wrong?

Also english is not my native so i'm not sure about what you are talking about when you talk about "caching".
Just to be sure... ZFS is like putting HDD on a raid right?

Louie, you are definitly not missing the point, you have one ;-)

1. Ideally yes, TrueNAS separate serving your Storage needs
   1. also, single point of failure aspects
   2. better utilisation of ressources#
   3. etc

Sorry, are you saying Proxmox is good enough on it's own for Plex/Own Storage or are you saying TrueNAS in a VM is better ? I don't get who you were talking too

Depends...you should learn about ZFS, requirements and passthrough in case you want to use TrueNAS.

Like said if i'm right, ZFS is the way to do raid and stuff which is why find TrueNAS to be the easiest way to do it for a begginer
Requierement will be a I7 6700k and 32mb of RAM with no GPU, is it fine?

Correct me if i'm wrong. Passthough is the way to "connect" my physical drive to a VM yeah? if so i've seen videos that explain how to do it and it didn't seem that hard to do.

You should get proper SSDs as VM/LXC storage. HDDs are only useful as cold storage. Raid5 is also not great as a VM/LXC storage because of worse IOPS performance. Again, best to use it for cold storage. Make sure to buy CMR HDDs and avoid SMR HDDs. Especially when using CoW like ZFS does.

My plan was to get NVME for the VM and proxmox installation. Am i wrong in this?


And last question : Imagine i would like to legaly download a torrent movie for my plex server, what's the better way to do it ? I didn't really look to it yet, but i guess there's something i can do with trueNAS to directly download though it ? My plan was to download it with my own computer and send it to TrueNAS but... i feel like that's dumb and there's other way more efficient to do it..

Again i'm sorry i feel stupid because it seems simple for you guys but i'm kinda swimming in the ocean rn xD (and the thing that i don't have the computer yet is not helping)

Thanks!

 

[Dunuin]

I see people do this all the time, but I dont really know why. What are you getting from Trunas you couldnt host directly on the pve host? If the answer is "security," I can only think "this is for home use, who cares."

Keeping chances of human error low because all can be done using the webUI. It supports encryption out of the box. Easy replication and snapshot management to "backup" data from one TrueNAS VM to another one.
Nothing that couldn't be done the hacky way via CLI on PVE with lots of custom scripts. But the learning curve would be very high.

I'm building with a I7 6700K and 32mb of ram. I just assume performance won't be a problem but may i'm wrong?

Thats at the very low end of what you are planning to do. My TrueNAS VM alone got 32GB of RAM and 8 vCPUs.
Then 8GB RAM and 6 vCPUs for every Windows VM, 16 vCPUs and 4GB RAM for the emby VM (that I use instead of plex), 22GB RAM and 11 vCPUs for the Minecraft server...this easily adds up...

Just to be sure... ZFS is like putting HDD on a raid right?

ZFS is more than raid. Its raid, volume management and filesystem in one.

Sorry, are you saying Proxmox is good enough on it's own for Plex/Own Storage or are you saying TrueNAS in a VM is better ? I don't get who you were talking too

Proxmox is no NAS and got no NAS functionalities. But its a full-fledged Linux OS so its possible to do anything that would be possible with a normal Debian server. But you would need to learn tons of stuff as everyting would have to be done by you via CLI!

Correct me if i'm wrong. Passthough is the way to "connect" my physical drive to a VM yeah?

Yes. As a VM can't make use of any of the PVE hosts physical hardware without it!

My plan was to get NVME for the VM and proxmox installation. Am i wrong in this?

Yes. But in case you want to use ZFS there too, keep in mind that only a few enterprise M.2 SSDs exist and they aren't cheap. Recommended for ZFS would be something like a Samsung PM983.

And last question : Imagine i would like to legaly download a torrent movie for my plex server, what's the better way to do it ? I didn't really look to it yet, but i guess there's something i can do with trueNAS to directly download though it ?

Would require installing a jail(container) or VM(not great because of nested virtualization) on TrueNAS. I wouldn't use TrueNAS for anything else than storing your data and sharing it. Would be better to run all your services as LXCs/VMs on PVE. So a dedicated privileged LXC or VM for your torrent server. You probably also want to learn about network stuff like setting up firewall/router/VPN VMs/LXCs first.

 

[kobe_highlander]

careful with doms. they're typically designed to house a nonvolatile distro such as vmware, and have pretty poor write endurance. if you insist on running proxmox on a dom, I'd offload the entire /var directory to another drive, at minimum.

Thanks for the hint ... I have read a-lot about different things over the past weeks. I actually have that at the back of my mind and will chose a different path maybe in the near future ... still fiddling with things and learning.

 

[alexskysilk]

Nothing that couldn't be done the hacky way via CLI on PVE with lots of custom scripts. But the learning curve would be very high.

huh. ok. All I could think of is smb share management, which doesnt... For the use case described I'd just run truenas on metal; the tradeoff of having nested dependencies with a pve parent dont make sense to me.

I'm building with a I7 6700K and 32mb of ram. I just assume performance won't be a problem but may i'm wrong?

you're looking at it backwards. define the load and spec the hardware accordingly, not the other way around. make inventory of the stuff you will have on the machine, eg
1. truenas vm
2. plex vm
3. windows vm
4. etc

you will need to make some educated guesses for what you will need, resources-wise; whatever will be housing your storage will need some ram for zfs- but how much really depends on the type of activity you'd be doing with it; plex requires large block streaming which will not stress the filesystem at all, but other uses might. plex will also want access to some kind of hardware offload, which will mean you will want to run plex in a container so you can map hd graphics to it, which will have the added benefit of being relatively lightweight. if my above example is all you'd be doing with the machine, your cpu/ram will be fine, although I may suggest a small gpu to pass to the windows vm- will make the ux so much better.

If you dont intend to control snapshots and send them to another truenas box, I'd suggest having proxmox manage the storage. that way it can reallocate ram dynamically to zfs and vms depending on load, and will have the added benefit that you could use that same filesystem for other things without a circular dependency.

My plan was to get NVME for the VM and proxmox installation. Am i wrong in this?

Not wrong. just make sure you have two of them.

keep in mind that only a few enterprise M.2 SSDs exist and they aren't cheap. Recommended for ZFS would be something like a Samsung PM983.

not wrong either, but this is like saying you cant ride a bicycle without safety pads. Yes you can, and many do- just understand the risk and be prepared for the inevitable failure (thats why I said have 2.) FTR, the "enterprise" drives fail too, just usually with longer mtbf.

And last question : Imagine i would like to legaly download a torrent movie for my plex server, what's the better way to do it ?

the unifying issue would be accessibility of the target store. If you are serving the target store from truenas, you can run your client anywhere that can access it- eg, it can be any machine on the same network that has the location mapped over nfs, smb, or whatever protocol you use. I would just have the truenas bittorrent client perform the chore so to half the network traffic (and eliminate the network fie transport as a point of failure.)

 

[Dunuin]

huh. ok. All I could think of is smb share management, which doesnt... For the use case described I'd just run truenas on metal; the tradeoff of having nested dependencies with a pve parent dont make sense to me.

If the usecase is a homelab it could be a good reason. The bare metal TrueNAS server idled at 80W and the bare metal PVE at 70W. Moving HBAs and disks from the TrueNAS server to PVE server and virtualizing TrueNAS increased the PVE servers power consumption by 40W to 110W but at the same time I never had to run that 80W bare metal TrueNAS server again saving me 40W idle power consumption and therefore 190€ per year.
But yes, if I wouldn't have to care about electricity bills I would also prefer to run TrueNAS bare metal for less dependencies.

not wrong either, but this is like saying you cant ride a bicycle without safety pads. Yes you can, and many do- just understand the risk and be prepared for the inevitable failure (thats why I said have 2.) FTR, the "enterprise" drives fail too, just usually with longer mtbf.

Yes, and depends on the workload. I wouldn't want to run a big minecraft server backed by MySQL DB for its mods on SSDs without PLP because of the horrible wear and performance when doing sync writes. Might not be that bad when only doing some torrenting with its async writes.

 

[alexskysilk]

If the usecase is a homelab it could be a good reason. The bare metal TrueNAS server idled at 80W and the bare metal PVE at 70W.

On the same hardware? is it because truenas doesnt support the hardware apm? My assertion was that it would make sense to run ONLY truenas, not both- and that they should consume the same amount of power relative each other given the same workload. That said, if power is a concern, I'd just leave the whole thing off and have only a 1L low power device running 24/7 (which is, in rl, exactly what I do. when the 1L server gets full-ish, I wake the nas and offload everything.)

I wouldn't want to run a big minecraft server backed by MySQL DB for its mods on SSDs

I confess I know little about this workload- what kind of sustained iops does this generate per user, and whats a "typical" server host user wise?

 

[Dunuin]

On the same hardware? is it because truenas doesnt support the hardware apm? My assertion was that it would make sense to run ONLY truenas, not both- and that they should consume the same amount of power relative each other given the same workload. That said, if power is a concern, I'd just leave the whole thing off and have only a 1L low power device running 24/7 (which is, in rl, exactly what I do. when the 1L server gets full-ish, I wake the nas and offload everything.)

Yes, same hardware. But running the identical workload on a single server instead of splitted across 2 servers really saved power. Consumption of disks + HBAs I moved from the TrueNAS server to PVE server of cause still is there and the CPU of the PVE server got additional load so a bit more power consumption by that and I had to add more RAM modules. But only needing to run a single 10Gbit NIC, CPU, PSU, mainboard, BMC, case with multiple fans and so on instead of running everything twice really reduced the total power consumption.
Not sure how well the new linux-based TrueNAS Scale handles power saving but the FreeBSD-based TrueNAS Core wasn't good at that. Even after tempering with tunables to enable powerd so the CPU was able to clock down. And that it is an appliance where you are not supposed to temper with the underlying OS as those changes will be wiped on updates isn't really helping when you want to optimize it for better efficiency.

I confess I know little about this workload- what kind of sustained iops does this generate per user, and whats a "typical" server host user wise?

Not sure. Never monitored MySQL there and was never running a big server with lots of active users. But I had to set up MySQL tables for dozens of mods and the Miencraft VM caused enough IO that I wasn't able to run it on HDD raid.
The point was more in general that it really depends what workload you are running as async writes won't hit a consumer SSD as hard as sync writes. What I monitored was the SSD wear and IO when storing all logs and metrics of all VMs and LXC into MongoDB, MariaDB and OpenSearch DB. That caused so much wear that I had to replace all 6 consumer SSDs with enterprise SSDs as the TBW would have been exceeded within a year. Storing logs and metrics into DBs (Zabbix and Graylog) caused more writes than all other VMs combined generating these metrics and logs...