[SOLVED] How i can expand an ZFS Raid10 correctly?

fireon

fireon

Distinguished Member
Proxmox Subscriber
Oct 25, 2010
4,511
479
153
Austria/Graz
deepdoc.at
Hello :)

i have here an server with 6 Disks. 2 SSDs in Raid1 and 4 HDDs in Raid10, everything ZFS. Here my config:

Code: 
pool: rpool
 state: ONLINE
  scan: resilvered 4.68M in 0h0m with 0 errors on Fri Apr 24 17:37:52 2015
config:

    NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdf3    ONLINE       0     0     0
        sde3    ONLINE       0     0     0

errors: No known data errors

  pool: v-machines
 state: ONLINE
  scan: none requested
config:

    NAME                                            STATE     READ WRITE CKSUM
    v-machines                                      ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D877YE  ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D0KRWP  ONLINE       0     0     0
      mirror-1                                      ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D688XW  ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D63WM0  ONLINE       0     0     0

errors: No known data errors

So I would like to add 2 and and later another tow disk to the Pool. I read a lot of in forums and documentations. But i think i don't understand. In an normal Linux Softraid 10 i have always one mirror. For example, 4Disks mirrored to the other 4disks. So in my configuration i have the same, 2disks mirrord with 2disks, right?

So i would like to add the next two disks to the ZFS Raid10. So with my understanding: 1 disk to mirror-0 and one disk to mirror1... not right? I don't know. I always found something like this: http://docs.oracle.com/cd/E19253-01/820-2313/gazgw/index.html
But sorry i don't understand it. I tested something in a virtual machine, but i was only able to add another mirror, like this:

Code: 
NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c0t1d0  ONLINE       0     0     0
            c1t1d0  ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c1t2d0  ONLINE       0     0     0
          mirror-2  ONLINE       0     0     0
            c0t3d0  ONLINE       0     0     0
            c1t3d0  ONLINE       0     0     0

So what is right, is Raid10 with ZFS not the same like normaly softraid unter Linux?

Thanks and best Regards
 
Last edited:
I've done another test with a VM.

Code: 
zpool create -f -o ashift=12 testpool mirror sdc sdd sde mirror sdf sdg sdh

Code: 
pool: testpool
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    testpool    ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
        sde     ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        sdf     ONLINE       0     0     0
        sdg     ONLINE       0     0     0
        sdh     ONLINE       0     0     0

errors: No known data errors



I think it should look like this! Or not? So what?
 
If you want to change your ZFS pool from first post example to second post example you will need to recreate pool. There is no other way. You can expand your pool by adding another mirror or raidz but your pool will become dis balanced.
 
I don't think adding an extra vdev to the stripe will make it unbalanced. Adding a new mirrored vdev to the pool will simply create a stripe of 3 mirrored vdev's which will increase performance.
 
You cannot extend already created ZFS pool. ZFS dont have this funkcionality. You must backup you data on current pool, destroy it, and recreate it from scratch with added disks. Linux SW raid is definitely not like ZFS. Its like comparing bike with mercedes :}
 
The problem the last link emphasizes is expanding a pool because it is nearly filled. In this situation expanding is not a good idea but if you want to expand to gain performance and your pool is not close to be filled then it is a different story since ZFS will balance the pool for you in time before the initial vdevs hits 80%. I have done this with success.

The problem in the first link is about devices which has nothing to do with vdevs.
 
PanZvedavy said:
You cannot extend already created ZFS pool. ZFS dont have this funkcionality. You must backup you data on current pool, destroy it, and recreate it from scratch with added disks. Linux SW raid is definitely not like ZFS. Its like comparing bike with mercedes :}
Click to expand...
This is nonsense. You can always extend a pool which is the purpose of the 'add' command.
 
mir ZFS does not do any balancing. If your pool is low usage then you will not sense performance drop, with more and more writes it become "balanced". But for data which was not rewrited it stays as before expanding the pool.

Where was a lot of talks about it. Maybe in the future ZFS will do balancing and new function could appears like:
* fragmentation / balancing
* reduce of pool
 
  • Like
Reactions: chrone
Ok, i understand, right is:

Code: 
pool: v-machines
 state: ONLINE
  scan: none requested
config:

    NAME                                            STATE     READ WRITE CKSUM
    v-machines                                      ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D877YE  ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D0KRWP  ONLINE       0     0     0
      mirror-1                                      ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D688XW  ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D63WM0  ONLINE       0     0     0
      mirror-2                                      ONLINE       0     0     0
        sdg                                         ONLINE       0     0     0
        sdh                                         ONLINE       0     0     0

errors: No known data errors

The other way is also right, but then i have 3 Raid1 but i would like to have my raid10 below. I've added the two new disks with
Code: 
zpool add -f -o ashift=12 v-machines mirror sdg sdh
and i iported the uuids:
Code: 
zpool import -d /dev/disk/by-uuid/ -a
Ok, the poos is bigger, an what is with sync, i have not really high I/O. Here are more infos:
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
Code: 
rpool       27.8G  1.85G  25.9G     6%  1.00x  ONLINE  -
v-machines  5.44T  2.21T  3.23T    40%  1.00x  ONLINE  -
So 6 harddrives with 2TB, it looks likes ok.

Code: 
zfs list
NAME                                           USED  AVAIL  REFER  MOUNTPOINT
rpool                                         5.25G  22.1G   144K  /rpool
rpool/ROOT                                    1.66G  22.1G   144K  /rpool/ROOT
rpool/ROOT/pve-1                              1.66G  22.1G  1.66G  /
rpool/swap                                    3.59G  25.5G   186M  -
v-machines                                    2.39T  2.96T   152K  /v-machines
v-machines/home                               1.96T  2.96T  1.96T  /v-machines/home
v-machines/vm-100-disk-1                      35.1G  2.96T  34.4G  -
v-machines/vm-101-disk-1                      15.5G  2.96T  15.2G  -
v-machines/vm-102-disk-1                      40.3G  2.96T  39.6G  -
v-machines/vm-103-disk-2                      35.1G  2.96T  34.4G  -
v-machines/vm-104-disk-1                      79.8G  2.99T  39.5G  -
v-machines/vm-104-state-vor_check_mk_install  4.53G  2.97T   212M  -
v-machines/vm-105-disk-1                      18.6G  2.98T   519M  -
v-machines/vm-105-disk-2                      18.6G  2.98T   519M  -
v-machines/vm-105-disk-3                      18.6G  2.98T   520M  -
v-machines/vm-105-disk-4                      18.6G  2.98T   520M  -
v-machines/vm-105-disk-5                      18.6G  2.98T  1.19M  -
v-machines/vm-105-disk-6                      18.6G  2.98T  1.19M  -
v-machines/vm-105-disk-7                      17.5G  2.98T  1.04G  -
v-machines/vm-105-disk-8                      17.5G  2.98T  1.04G  -
v-machines/vm-106-disk-1                      41.3G  2.96T  39.4G  -
v-machines/vm-107-disk-1                      40.3G  2.96T  39.4G  -
root@virtu01:~# df -h
Filesystem                          Size  Used Avail Use% Mounted on
rootfs                               24G  1.7G   23G   8% /
udev                                 10M     0   10M   0% /dev
tmpfs                               3.2G  588K  3.2G   1% /run
rpool/ROOT/pve-1                     24G  1.7G   23G   8% /
tmpfs                               5.0M  4.0K  5.0M   1% /run/lock
tmpfs                               7.0G   16M  7.0G   1% /run/shm
/dev/fuse                            30M   28K   30M   1% /etc/pve
v-machines                          3.0T  128K  3.0T   1% /v-machines
v-machines/home                     5.0T  2.0T  3.0T  40% /v-machines/home

So is it ok? Hoply.

Thanks
 
I've found this:

Code: 
----------------------------------------------  -----  -----  -----  -----  -----  -----
v-machines                                      2.22T  3.22T    141    129  1.69M  1011K
  mirror                                        1.11T   722G     70     64   865K   503K
    ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D877YE      -      -     13     13   494K   569K
    ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D0KRWP      -      -     13     13   495K   569K
  mirror                                        1.11T   722G     70     65   867K   506K
    ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D688XW      -      -     13     14   495K   572K
    ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D63WM0      -      -     13     14   496K   572K
  mirror                                        2.96G  1.81T      0     63    447  2.93M
    sdg                                             -      -      0     32    205  2.99M
    sdh                                             -      -      0     32    286  2.99M

So it look like it is syncing. Can give more speed for sync the new harddrives, so about 50mb/s

Best regards
 
  • Like
Reactions: crmspezi
I don't know, why it is not the sam ;( for the first i had 4 2TB harddrives. Today i add another two 2TB harddrives. So what... i've testet it first in VM. And it was ok. It is a production system, so i should hang me? ;(
 
So, i read more about it, and ask an friend. The configuration is ok. ZFS sync data only when the touched next time. So i rebooted the server, and it can't boot anymore. I put out all disk of the v-machines pool. But same result. There was no update no, changes from the system since the last reboot. And the rpool was not touched . Please see the screenshots from attached.

I booted with ubuntu live and ZFS support and i see the rpool. Everything is fine. But i can't mount because i will be mounted on "/". The uuids were imported many boots before. And i had no problems to boot the machine before. No kernel ore other changes.

Must there be an special grubupdate after adding some disks? ... or ...

Thanks for help20150519_221916.jpg20150519_221527.jpg20150519_222023.jpg
 
So, problem is solved. Was on a really hard way! Ok, but the main problem was the ZFSversion 0.6.3.2. The Problem is that hardisk IDs doesn't really work fine. And there is a Bug to attach same harddisk again to the pool. In the new Version 0.6.4.3 (proxmox repository) these Bugs are fixed. After i upgraded to Kernel 3.10.0.10 and Upgraded the Pool everthing works fine. The rpool was lost ;( . I had to reinstall proxmox. After that my v-machines pool was imported autmatcly. During i attached one disk again, the disk give up and was death! Nice. Was an neu WD Red Pro. Maybe material defects. I resilverd the news disk with about 190mb/s. Everything is fine now. Well, hopefully it stays that way.

Code: 
pool: rpool
 state: ONLINE
  scan: none requested
config:

    NAME                                                STATE     READ WRITE CKSUM
    rpool                                               ONLINE       0     0     0
      mirror-0                                          ONLINE       0     0     0
        ata-KINGSTON_SS200S330G_50026B725201E0C7-part3  ONLINE       0     0     0
        ata-KINGSTON_SS200S330G_50026B7252093F6A-part3  ONLINE       0     0     0

errors: No known data errors

  pool: v-machines
 state: ONLINE
  scan: resilvered 1.09T in 4h45m with 0 errors on Sat May 23 02:48:52 2015
config:

    NAME                                            STATE     READ WRITE CKSUM
    v-machines                                      ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D0KRWP  ONLINE       0     0     0
        ata-WDC_WD20EARX-00ZUDB0_WD-WCC1H0343538    ONLINE       0     0     0
      mirror-1                                      ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D688XW  ONLINE       0     0     0
        ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D63WM0  ONLINE       0     0     0
      mirror-2                                      ONLINE       0     0     0
        ata-WDC_WD20EARX-00ZUDB0_WD-WCC1H0381420    ONLINE       0     0     0
        ata-WDC_WD20EURS-63S48Y0_WD-WMAZA9381012    ONLINE       0     0     0

errors: No known data errors

The IDs can you find here: For example:
Code: 
ls  /dev/disk/by-id/ -l | grep sdb
lrwxrwxrwx 1 root root  9 May 23 11:50 ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D688XW -> ../../sdb
lrwxrwxrwx 1 root root 10 May 23 11:50 ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D688XW-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 May 23 11:50 ata-WDC_WD2001FFSX-68JNUN0_WD-WMC5C0D688XW-part9 -> ../../sdb9
lrwxrwxrwx 1 root root  9 May 23 11:50 scsi-SATA_WDC_WD2001FFSX-_WD-WMC5C0D688XW -> ../../sdb
lrwxrwxrwx 1 root root 10 May 23 11:50 scsi-SATA_WDC_WD2001FFSX-_WD-WMC5C0D688XW-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 May 23 11:50 scsi-SATA_WDC_WD2001FFSX-_WD-WMC5C0D688XW-part9 -> ../../sdb9
lrwxrwxrwx 1 root root  9 May 23 11:50 wwn-0x50014ee003f548a4 -> ../../sdb
lrwxrwxrwx 1 root root 10 May 23 11:50 wwn-0x50014ee003f548a4-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 May 23 11:50 wwn-0x50014ee003f548a4-part9 -> ../../sdb9

I used alway the ata*

best regards
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom