Flexible storage options within Proxmox?

[level42]

I'm looking for possible solutions to have "flexible" storage within Proxmox. I'm finding some vague information regarding adding/removing drives to a ZFS pool, I'm confused as to what a VDEV is within a ZFS pool, I'm feeling overwhelmed and I'm hoping for some guidance.

Please allow me to explain my current situation, and desired results... (Optional read, but provides some context...)

Currently, I have a very old (10+ year), QNAP, running 8x 8TB drives in a Raid 5, currently about 95% full. My QNAP is seriously showing it's age, and I fear it's going to die at any moment.

I'm attempting to replace this, and stepped up to buy a new server. I opted for a Dell R730xd, with 18 (16x 3.5 and 2x 2.5) drive bays. The issue is, this has tapped out my budget, and I don't have room for new Hard Drives at this time.

Thus, here's my plan/idea...

I have 8x 4TB drives (which were previously used in my QNAP, before upgrading to the 8TB drives a few years ago). I've installed the 8x 4TB drives into my new Dell server, and setup Proxmox. This is about as far as I've gotten, point being, if I need to redo something, now is the time. My hope was to migrate about 30+TB of data from my old QNAP, to the new server and through some deliciated data juggling at home, slowly decommission the QNAP, and migrate over the additional 8x 8TB drives. The end goal would be 1 ZFS storage pool (would have to be 1 volume as hardlinking is actively leveraged), with 16 drives. 8x 4TB and 8x 8TB drives.

Not only this, but I would like to one day upgrade all the drives in the ZPool, with larger drives. Ideally one day, ending up with a single Zpool, of 16x 30TB drives or something.

Ultimately, my questions are:

- How can I manage and manipulate my storage pool, to allow additions and removals of drives, expansions, growth, etc.?
- Is there any way this is even possible, or am I stuck just buying 16x new drives at the inceptions of the Zpool?
- What other options might I have in order to juggle storage around?

Final thoughts... I read somewhere that I can create a Zpool, say 16 drives large. Could I use a complete mix of drives (some 2TB, some 1TB, some 4TB, etc...) and over time just swap out 1 disk at a time, let the Zpool rebuild/recalculate/expand etc, then move on and replace the next drive? Eventually leaving me with 1 Zpool of 16x drives, all the same large size/capacity? If so, would this likely be my best course of action to initiate the Zpool with 16 drives first, then replace them one by one? Are there caveats to this in the long term in terms of performance that I should be aware of? Obviously, the Zpool in the beginning, will inherit the performance of the weakest drive, but eventually, they'd all be brand new beefy drives. ‍

Please help my anxiety ridden ADHD brain make sense of this, and plan for a safe, healthy, productive future of storage...

Thanks!

For what it's worth, my plans with this Dell server, is to be an "all in one server". It's eventually going to host anything I want to throw at it. It will be running a plethora of services such as:

- FRIGATE
- IMMICH
- ROMM
- PLEX
- HOME AUTOMATION
- RUSTDESK Server
- Various VMs
- DOCKER
- And more as time goes on...

 

[LnxBil]

Is there any way this is even possible, or am I stuck just buying 16x new drives at the inceptions of the Zpool?

That depends on the pool setup. If you e.g. have a 4x4 disk setup, you can simple add another 4 disk vdev to your pool. You have it in your hand. I hope you're not running without any redundancy.

A pool consists of at least one vdev and one disk. Your pool can grow or shrink with adding or removing vdevs. What a vdev consists of is totally in your hand. You can stripe all kind of disks together, at least technically. It does often not make sense to do this, because you get a different response times of each block. The larger disks are hit more often, because ZFS tries to stripe the data over all your vdevs depending of their fullness.

So, ZFS will work, yet it would probably not make a lot of sense to run this due to reliability and performance.

 

[Kingneutron]

> I have 8x 4TB drives (which were previously used in my QNAP, before upgrading to the 8TB drives a few years ago). I've installed the 8x 4TB drives into my new Dell server, and setup Proxmox. This is about as far as I've gotten, point being, if I need to redo something, now is the time. My hope was to migrate about 30+TB of data from my old QNAP, to the new server and through some deliciated data juggling at home, slowly decommission the QNAP, and migrate over the additional 8x 8TB drives. The end goal would be 1 ZFS storage pool (would have to be 1 volume as hardlinking is actively leveraged), with 16 drives. 8x 4TB and 8x 8TB drives.

> Not only this, but I would like to one day upgrade all the drives in the ZPool, with larger drives. Ideally one day, ending up with a single Zpool, of 16x 30TB drives or something.

You can create a zfs pool with the 8x4TB drives; with RAIDZ2, it would have (1) vdev.

When you migrate the additional 8x8TB drives over, you can create a separate ZFS RAIDZ2 pool with those so they're not mismatched.

It's not recommended to use different-sized disks in the same zpool, as your I/O will be unbalanced and the total size of the pool will be limited to the aggregate size of the SMALLEST disks (minus however many disks for parity). This is if you create the whole pool at once though IIRC, and only if you mix disk sizes in the same vdev. You can try test scenarios with a file-backed pool, or use a VM with virtual disks

You can get away with e.g. 2x4TB + 2x8TB if you use mirrors instead of RAIDZx, but for raidz2 you want matching-sized disks.

If you already have an 8x4TB RAIDZ2 vdev and you attempt to add another 8x8TB RAIDZ2 vdev to the same pool, zfs may give you a warning. Not sure, haven't tested.

You have 16x drive bays for 3.5 inch, what about spares and replacements? With my 15-bay disk shelf I went with 14x4TB ZFS DRAID(z2) and left 1 bay free for disk failures.

You can expand the pool by replacing smaller disks with larger (one disk at a time, and wait for resilver to finish), but it generates A LOT of I/O and puts stress on the pool.

You might want to buy an additional 15-bay SAS disk shelf (they're fairly cheap on ebay these days) and use that for expansion when you're ready to replace the smaller disks - e.g. you can add another 8x8TB vdev to your existing 8x8TB pool and just copy over everything from the 8x4TB pool without replacing disks one by one. Then decommission the x4TB pool.

Similarly, when you're ready to upgrade the 14-or-16 disk x8TB pool: you populate the disk shelf with larger disks, create another pool, and copy everything over once instead of replacing disks 1:1. This saves you A LOT of wear and tear on spinning disks

 

[level42]

I appreciate the insight and responses here. Please excuse me while I try to wrap my head around all of this.

I know it's blasphemous to not have additional redundancy and backups, but I simply cannot afford to do so. Thus, my intent is something like a basic RAID5. Enough that I can survive a single drive failure. I understand this is not ideal, not great, and a poor decision, but that's the decision I have to make in my current financial state. (Hence the whole concept in the first place about attempting to salvage the equipment I have currently "the old HDD's from my QNAP")

As well, please excuse my ignorance as ZFS is a brand new technology to me, and I'm learning as I go.

To help me better understand the "pool" and "vdevs" I'd try to compare them to something I am a bit more familiar with.

Correct me if I'm wrong, but it seems that a "vdev" is a collection of drives (example 4x4), and in turn creates a simple RAID for these 4 disks, and the "pool" is essentially a RAID of RAIDs?

The VDEV takes a small 4x4 RAID, and treats it as a single disk? and the POOL is then a collection of these "single disk" VDEVs?

Am I on the right track for understanding here? (in a super high level POV)

Ergo, if I then do as mentioned, and create a VDEV with the 8x4, and later on append the VDEV with the 8x8 to the same pool, it'd essentially be:

Pool [1] ─── VDEV [2] ─┬─ VDEV1: 8x-4tb
                       └─ VDEV2: 8x-8tb

In complete layman's terms, where does the data then get access/written? If I had a folder called "Photos", where does this folder exist? On the POOL? On one of the VDEVS?

If I were to access this from a Windows 10 machine, what would I be accessing, the POOL as a whole? Would this then show as a single volume on the Windows end, and ZFS would simply put the data here/there wherever it felt on whichever VDEV?

As someone administering this, I would then just treat the entire POOL, as two separate raids (The VDEVS)? Ergo, if drive 3 failed on VDEV2, then I would simply replace that one drive (hot swap), and the VDEV would then get rebuilt, and then once the VDEVs are both back up to 100% health, the POOL would then also resync between the two VDEVS? This part leaves me a bit confused.

Given that I don't really know how ZFS works, and how I can't afford additional redundancy, I'm simply trying to make sure I understand:

A.) How this technology works, in a relatable fashion for my broken brain.
B.) What are my "worst case scenarios", and what can/do I have to do when they occur.

Ultimately, I'm just hoping that I don't prep my storage solution, run into a small problem a year or two or three from now, and be left high and dry trying to fix it due to a lack of understanding, and a lack of proper preparation in the first place.

Please excuse my rant, there's just a plethora of nonsense in my brain, and it's a lot to try and juggle everything all at once.

 

[Kingneutron]

ZFS stripes writes/reads across vdevs. However, if you start with e.g. a 2-disk mirror pool and then add another +2 disk mirror column of equivalent-sized disks (or larger) the data does NOT get auto-rebalanced.

This concept also scales to raidz2 - if you have a 1-vdev 6x4TB RAIDZ2 pool and fill it up with data, if you then wait a month and add another 6x4TB vdev to the same pool, ALL of the data is still on the 1st original vdev. New data and rewrites will start going (striped writes) to the additional 2nd vdev.

So if you had "Photos" in a Samba share on the original 2-disk mirror pool, all the data will still be on the 1st (2) original mirror disks until the data starts to be rewritten. Or you do a zfs send/recv, or restore from backup.

When you create a ZFS pool, you also create datasets that have their own set of properties. These datasets are like subdirectories, and are what get shared across the network as SMB / NFS.

As far as redundancy - if you have e.g. a 12-disk zpool with 2x 6-disk RAIDZ2 vdevs, then you can lose up to (4) drives total in the pool before risking data loss. Each vdev has 2-disk redundancy, so you can lose up to (2) disks per-vdev.

Typical Worst-case scenario is: You're not running everything off of a UPS, lightning strikes, server and the whole shebang goes down hard - and you don't have backups. You manage to get the server to boot somehow (possibly by reinstalling / replacing fried hardware) and when you try to import the zpool it says NoWay, restore from backup.

There are a couple of recovery things you can try at that point, no guarantees, maybe mounting the pool R/O works or you can rollback to a previous transaction, but 98% you'll be dead in the water. Nothing Beats Backups. If you value your data, start saving up and buy a used tape drive or something off Ebay. Or do like the rest of us and put it on credit.

My advice, setup a VM and practice changing out failed drives in your pool. Or do it on bare hardware with a minimum set of test data (a couple of TB) before you go "live" and start filling the thing up.

https://klarasystems.com/articles/openzfs-understanding-zfs-vdev-types/

https://arstechnica.com/information...01-understanding-zfs-storage-and-performance/

 

[Dunuin]

I know it's blasphemous to not have additional redundancy and backups, but I simply cannot afford to do so. Thus, my intent is something like a basic RAID5. Enough that I can survive a single drive failure. I understand this is not ideal, not great, and a poor decision, but that's the decision I have to make in my current financial state. (Hence the whole concept in the first place about attempting to salvage the equipment I have currently "the old HDD's from my QNAP")

Also keep the time, needed for resilvering, in mind. Once one of those 8x8TB disks in a raidz1 dies you will have to replace it and resilvering might take some days where your other disks are tortured 24/7 with 100% IO. In that state ZFS can't fix any errors any longer and it is very likely that another disk might fail because of that stress before the resilvering has finished. Especially in case those disks are the same models, were bought at the same time, are already running for some years and were always used together with the same workload. With 8x 8TB disks you usually want to use a raidz2 even if that means losing some more space.

 

[level42]

I appreciate all of the advice and patience from you guys. It's been a huge help to get my head wrapped around all of this.

I'm going to follow a few of the links that @Kingneutron posted, and do some additional reading. If it comes down to it, and I need to just bite the bullet and buy some new Hard Drives, then so be it... it's just may have to wait some time beforehand.

I think I'm going to as well, practice with some VDEVs in a test pool, attempt to break things, merge things, expand, shrink, etc, and literally just get my feet wet on the whole thing.

Thank you all again for your help and feedback! It is greatly appreciated!!! <3