[SOLVED] Good Practice for Disks Setup

[degudejung]

Hi,
in my 2nd attempt to switch over to Proxmox from OMV, I want to move away from an ancient Dell Server with Xeon X3440 that is great for disk storage capacity but terrible on power consumption, noise and thermal exhaust. Therefore, I want to downsize my homelab server to an unused MacMini that will save me about 80-100 Watts in idle mode. However, it will need an external disk storage; in my case a 5-bay USB3 case. Tests have shown it's still enough computing power and transfer speed for what I need here, so that's pretty much checked off.

Anyways, I would love a feedback or recommendation on how to set up the available disks in a smart way. Here's what I can work with:
=== Available Disks ===
2x 4TB HDD
2x 2TB HDD
3x 1TB SSD
2x 128GB SSD
+ Abundance of USB-sticks, older drives etc.

Here's the current setup that worked perfectly (but turns the room into a sauna):
=== Current Setup (in older Dell Server) ===
1TB SSD ___ ZFS Mirror = pool1 1TB, for some smart home VMs, Docker, some more-performance-needy stuff
1TB SSD _/

4TB HDD ___ ZFS Mirror ____ ZFS Stripe = pool2 6TB, for TimeMachines, backups, archives and other slow/old stuff
4TB HDD _/........................./
............................................/
2TB HDD ___ ZFS Mirror /
2TB HDD _/

128GB__ SSDs for Debian/OpenMediaVault (2x, 1 for Backups)

Here's the best I came up with for the new setup:
=== Future Setup (MacMini + 5-bay USB3-case) ===
in MacMini (SATA3):
................. ____ 20GB partition = Proxmox OS
1TB SSD /___ 980GB partition ____ ZFS Mirror = pool1 1TB
1TB SSD ____ 980GB partition __/
............... \____ 20GB partition = Proxmox OS Backup

in external case (USB3):
4TB HDD ___ ZFS Mirror ____ ZFS Stripe1 = pool2 6TB
4TB HDD _/......................... /
............................................/
2TB HDD ___ ZFS Mirror /
2TB HDD _/

1TB SSD __ just more Directory/LVM storage for Proxmox

Alternatively, I thought about making one full 1TB SSD internally the OS-Disk and some local/LVM storage. Then, I would have to mirror the other internal 1 TB SSD with a 1TB SSD externally in the USB3-Case. But I feel like that would give me bad performance and stability.

Another idea was to hook up another (old) drive, SD-card or USB-stick to the MacMini via USB/thunderbolt and try and install Proxmox on that, leaving the two 1TB SSDs completely free for the ZFS-mirror. But having the OS sit on an external disk seems kind of odd, too.

Any feedback and recommendation (OK, maybe apart from "go buy all new stuff!") is appreciated!

 

[LnxBil]

ancient Dell Server with Xeon X3440 that is great for disk storage capacity but terrible on power consumption, noise and thermal exhaust.

Ancient or not ... every enterprise grade server I know is terrible on power consumption, noise and thremal exhaust

in MacMini (SATA3):
................. ____ 20GB partition = Proxmox OS
1TB SSD /___ 980GB partition ____ ZFS Mirror = pool1 1TB
1TB SSD ____ 980GB partition __/
............... \____ 20GB partition = Proxmox OS Backup

Why the backup storage? It's ZFS isn't it? Just snapshot (yes, in this case this is the "same" kind of backup ... none. Backups are ALWAYS external, otherwise it's not a backup, but a snapshot).

Alternatively, I thought about making one full 1TB SSD internally the OS-Disk and some local/LVM storage. Then, I would have to mirror the other internal 1 TB SSD with a 1TB SSD externally in the USB3-Case. But I feel like that would give me bad performance and stability.

I would not mix external and internal drives in a zpool, I tried it and it gives not good iops. USB is way slower delay-wise and had random spikes in slowliness. Maybe it works great for spinning rust, buy my SSD performance really dropped significantly.


What type of usb3 5-bay case is this? How much power does it draw?

 

[degudejung]

Thanks @LnxBil for your consideration!

Why the backup storage?

Well, if I were to mirror the two partitions, it would be limited to 1000GB minus whatever GB I snip off for the Proxmox boot partition, anyways. So I figured let's bring it to some good use. What to do with +/-20GB anyways? I'd rather keep the boot partition on whatever filesystem Proxmox install defaults to (probably not ZFS) and keep the backup unmounted except for the backup procedure itself. That way I would consider it half-way offline. But when worst comes to worst, losing the boot partition should not be a big deal. All I really care about is the data on the ZFS pools.

I would not mix external and internal drives in a zpool, I tried it and it gives not good iops. USB is way slower delay-wise and had random spikes in slowliness. Maybe it works great for spinning rust, buy my SSD performance really dropped significantly.

Glad you agree. So that's pretty much off the table.

What type of usb3 5-bay case is this? How much power does it draw?

It's a random Orico/Yottamaster with a JMicron 394 chip in JBOD-mode. With the rust spinners in idle/sleep - what I expect to see 95% of the time - I assume power consumption of 5 - 10 watts.

Which would you prefer and why?:
a) split SSDs in 980G+20G partitions (as above). make the 980G partitions ZFS pool. make the 20G partition boot Proxmox internally.
b) keep the 1000G as one and make ZFS directly for the full disk. hook up another USB-connected disk to the Mac and make it boot Proxmox from there.

 

[Dunuin]

Also don't install PVE to USB stick/SD card. It will kill that in no time with all of its writes.

And you don't need an extra partition for PVE. By default when choosing to install it as a ZFS mirror it will create 3 partitions:
1.) 1MB grub boot partition
2.) 512MB ESP boot partition
3.) all the remaining space ZFS partition
Then the PVE system (root filesystem) and VM/LXC storage will share the same 3rd partition and you don't need to decide how much storage you want for file and how much storage for guests.

Its also a good idea to keep some space unpartitioned so you could create a swap partition later. PVE won't to that by default. And swap shouldn't be ontop of ZFS.

If you don't need that much performance and don't want to run postgres DBs it might also be an option to create a raidz1 with these 3 SSDs to get 2TB or fast SSD storage. But you would need to increase your volblocksize from 8K to 16K. But if you don't got space for 3 internal disks then this won't work well.

Also read about multi disk USB cases. I only read bad things about them when people are talking about them here. Sounds like they are all very unreliable and causing ZFS errors.

And there are quiet server. My Supermicro tower with its 90mm tower cooler and 120m fans is the most quiet machine here at home. Even my laptop is louder. The problems are more the rackable 1U/2U server or redundant PSUs with 40mm fans.

 

[degudejung]

Thanks @Dunuin that is also some really helpful advice!

Also don't install PVE to USB stick/SD card. It will kill that in no time with all of its writes.

Sure. I'd use a regular 128GB SSD with a SATA-USB3-adapter for that.

And you don't need an extra partition for PVE. By default when choosing to install it as a ZFS mirror...

Oh, I didn't know of that. That's a great advantage.

Its also a good idea to keep some space unpartitioned so you could create a swap partition later. PVE won't to that by default. And swap shouldn't be ontop of ZFS.

If you don't need that much performance and don't want to run postgres DBs it might also be an option to create a raidz1 with these 3 SSDs to get 2TB or fast SSD storage. But you would need to increase your volblocksize from 8K to 16K. But if you don't got space for 3 internal disks then this won't work well.

Unfortunately, with the planned setup I wouldn't find room for 3 SSDs on the same controller. But ZFS mirror/raid1 worked great for me so far.

Also read about multi disk USB cases. I only read bad things about them when people are talking about them here. Sounds like they are all very unreliable and causing ZFS errors.

OK, I'll be prepared. I will certainly have a another backup of everything non-disposable.

And there are quiet server. My Supermicro tower with its 90mm tower cooler and 120m fans is the most quiet machine here at home.

I hear you. What I'm about to do now is trying the no-budget solution with leftover hardware first. If that fails, I would try and rip my Dell T110 tower apart, give it a new mainboard, CPU and RAM and an efficient power supply. The Dell case itself is OK, just the Dell-interior s*cks.

Same question to you: would you rather "outsource" the OS disk to another SSD in order to keep the VMs, CTs and Data separated on other disks?

 

[Dunuin]

Same question to you: would you rather "outsource" the OS disk to another SSD in order to keep the VMs, CTs and Data separated on other disks?

With SSDs it usually should be fine to run PVE and VM storage on the same disk. The main problem would be when guests are creating massive IO so the disk can't handle all the IOPS, PVE gets stuck because nothing can be loaded from the disk anymore, because it is as 100% utilization, and then you won't be able to access the WebUI anymore, to for example kill the VM that is causing all the IO. But you usually only run into this with HDDs and their crappy IOPS performance. SSDs are usually fast enough not to bring the whole PVE down on heavy load.
So yes, if you got more slots/ports than you actually need, then it's nice to use a dedicated PVE system disk. But it's not really required.

 

[degudejung]

Thank you all. I think I have a pretty good idea on how to proceed now. Marked the thread as solved.