How to add new mirrored disks to an existing zpool?

[waffleireon]

I'm currently running proxmox on two 8TB drives, using zfs, and they're mirrored. My `zpool status` looks like this:

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 1 days 03:13:29 with 0 errors on Mon Nov 14 03:37:30 2022
config:

        NAME                                       STATE     READ WRITE CKSUM
        rpool                                      ONLINE       0     0     0
          mirror-0                                 ONLINE       0     0     0
            ata-ST8000VN004-2M2101_WSDOTAER-part3  ONLINE       0     0     0
            ata-ST8000VN004-2M2101_WSDOUING-part3  ONLINE       0     0     0

errors: No known data errors

I'm looking at getting some 16 or 18TB Ironwolf Pro disks on black friday sales, but before I do that I want to make sure that what I'm thinking is possible? Ideally I'd get two 18TB drives, mirror them as another pair, and just end up with 8 + 18 = 26TB total storage. Therefore I could go to OMV, for example, and increase its disk size from my current 4TB to something like 10 or more.

 

[Dunuin]

I'm looking at getting some 16 or 18TB Ironwolf Pro disks on black friday sales, but before I do that I want to make sure that what I'm thinking is possible? Ideally I'd get two 18TB drives, mirror them as another pair, and just end up with 8 + 18 = 26TB total storage. Therefore I could go to OMV, for example, and increase its disk size from my current 4TB to something like 10 or more.

Jup, correct.

But keep in mind that a ZFS pool should always have 20% free space. So more like (8 + 18) * 80% = 20.8TB of real usable storage.

 

[waffleireon]

Jup, correct.

If I'm understanding it right, would I be adding to the rpool by adding another mirror, most likely mirror-1? Do you know where I can find instructions to do this?

 

[Dunuin]

If I'm understanding it right, would I be adding to the rpool by adding another mirror, most likely mirror-1? Do you know where I can find instructions to do this?

See the "zpool add" command: https://openzfs.github.io/openzfs-docs/man/8/zpool-add.8.html
It also gives an example:

Example 1: Adding a Mirror to a ZFS Storage Pool​

The following command adds two mirrored disks to the pool tank, assuming the pool is already made up of two-way mirrors. The additional space is immediately available to any datasets within the pool.
# zpool add tank mirror sda sdb

 

[waffleireon]

See the "zpool add" command: https://openzfs.github.io/openzfs-docs/man/8/zpool-add.8.html
It also gives an example:

Do I need to do anything special before running the "zpool add" command? For example, do I need to pre-mirror the two new drives, or will they be mirrored by "zpool add"? And since the existing pool is the boot pool, do I need to do anything for that?

Or is it really as simple as just doing "zpool add" and it should all work?

 

[Dunuin]

Do I need to do anything special before running the "zpool add" command? For example, do I need to pre-mirror the two new drives, or will they be mirrored by "zpool add"?

When using the correct command, then no. "zpool add tank mirror sda sdb" will already add two disks as a mirror. That is what the "mirror sda sdb" means.

And since the existing pool is the boot pool, do I need to do anything for that?

Depends. If you want to be able to also boot from the new disks, then you would need to manually partition the new disks first, write the bootloader to them and so on. If you don't care about that, because you already got enough disks with a bootloader, then you could just run the zpool add command.

 

[waffleireon]

When using the correct command, then no. "zpool add tank mirror sda sdb" will already add two disks as a mirror. That is what the "mirror sda sdb" means.

Ok, that makes sense. One last thing... I noticed the existing disks in the existing mirror have long names (as seen in my zpool status in my original post), whereas the new disks I'd be adding are simply "sda" and "sdb"; is there a different ID I should be using to add the new disks? These are the first few lines of "lsblk":

NAME     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda        8:0    0 14.6T  0 disk
sdb        8:16   0 14.6T  0 disk
sdc        8:32   0  7.3T  0 disk
├─sdc1     8:33   0 1007K  0 part
├─sdc2     8:34   0  512M  0 part
└─sdc3     8:35   0  7.3T  0 part
sdd        8:48   0  7.3T  0 disk
├─sdd1     8:49   0 1007K  0 part
├─sdd2     8:50   0  512M  0 part
└─sdd3     8:51   0  7.3T  0 part

sda and sdb are the new disks, sdc and sdd are the existing ones. There's a whole bunch of zdNNpY disks under that too.

 

[Dunuin]

Yes, best would be to add them with unique IDs, like shown in your first post. You can run ls -la /dev/disk/by-id to see the disk IDs.
And then use the IDs that are referring to sda and sdb.

So something like this:
zpool add rpool mirror /dev/disk/by-id/IdOfYourSda /dev/disk/by-id/IdOfYourSdb

 

[waffleireon]

Yes, best would be to add them with unique IDs, like shown in your first post. You can run ls -la /dev/disk/by-id to see the disk IDs.
And then use the IDs that are referring to sda and sdb.

So something like this:
zpool add rpool mirror /dev/disk/by-id/IdOfYourSda /dev/disk/by-id/IdOfYourSdb

Aha, I see... makes sense that the existing disks have 'part3' on them since they're partitioned, but the new ones aren't. I'm guessing that's not an issue though.

 

[Dunuin]

Aha, I see... makes sense that the existing disks have 'part3' on them since they're partitioned, but the new ones aren't. I'm guessing that's not an issue though.

Jup, the existing disks are using the 1st and 2nd partitions for booting and only the third partitions are used for ZFS.
But you don't have to partition the disks you want to add. If you don't partition them and only tell ZFS to use the whole disk, then ZFS will partition it on its own creating a 1st and a 9th partition where the 1st partition will be used for the ZFS pool.

 

[generalproxuser]

I know this is an old thread but it's close to what I am trying to accomplish.

I am working with:
4 2TB ssds
2 8TB hdds

I wanted to combine the 4 2TB ssds to make an 8TB zpool. And then I wanted to use the 2 8TB hdds as a redundant backup to the ssds. Besides doing it the traditional old way of just making two zpools and then using rsync to backup the data from one pool to the other, I wanted to see if I could add the 2 8TB hdds to the 4 2TB zpool instead and just have all the data automatically present on all the disks.

This particular pool is only for storing data. Not vms or cts.

I have tried creating a raidz1 pool with the 4 ssds to make 8TB. But when I go to manually add the hdds to the existing pool as a mirror set it increases the total capacity to 16TB which is not what I wanted.

Is it possible to achieve what I am hoping to achieve?

 

[Dunuin]

I wanted to combine the 4 2TB ssds to make an 8TB zpool.

Not a great idea unless you only plan to store temporary files on it you don't care about. You will highly increase the chances to lose all data since your last backup and ZFS won't protect you from bit rot. Maybe you could get some bit rot protection and minimize data loss by not rsyncing the files between the pools but by using replication via "zfs send ... | zfs recv ..." and then keeping the snapshots longer than what your scrub interval is.

Besides doing it the traditional old way of just making two zpools and then using rsync to backup the data from one pool to the other, I wanted to see if I could add the 2 8TB hdds to the 4 2TB zpool instead and just have all the data automatically present on all the disks.

This will slow down the SSDs to HDD performance.

I have tried creating a raidz1 pool with the 4 ssds to make 8TB.

Raidz1 = Raid5. So you only get 6TB. And keep in mind that you usually don't want to fill that pool more than 80% for best performance and less fragmentation.

Is it possible to achieve what I am hoping to achieve?

No, thats not how it works. The thing that comes closest to it would be to create a mirror using the 2 HDDs and add those 4 SSDs as special devices in a striped mirror. That would give you a single pool with 8TB on the HDDs and 4TB on the SSDs. You could define which data should end on the HDDs and which on the SSDs by setting different special_small_blocks values to different datasets/zvols. But would be a terrible idea to raid0 those special devices. As then lose any of the 4 SSDs will mean losing all data on all 6 disks.
Same for what you have done now. Lose a single SSD and all data on the HDDs is lost too.