[SOLVED] Zpool issue

D

ducs4rs

Member
Apr 13, 2021
36
5
13
66
I have a mirrored zpool with which had 4 disks. So I had a mirror-0 and a mirror-1. See below

root@pve:~# zpool status zones
pool: zones
state: ONLINE
scan: resilvered 11.5G in 00:03:28 with 0 errors on Mon Apr 26 07:40:2
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST8000VN0022-2EL112_ZA1EL30R ONLINE 0 0 0
ata-ST8000VN0022-2EL112_ZA1EJPDN ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0

errors: No known data errors

I decided to add another mirror to the pool, but when I connected the two new drives One shifted the dev order and became sdc. I didn't think the physical position would matter to zfs due to the disk labels but it did. I ended up with

root@pve:~# zpool status
pool: zones
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
scan: resilvered 11.5G in 00:03:28 with 0 errors on Mon Apr 26 07:40:25 2021
config:

NAME STATE READ WRITE CKSUM
zones DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST8000VN0022-2EL112_ZA1EL30R ONLINE 0 0 0
ata-ST8000VN0022-2EL112_ZA1EJPDN ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
sdb ONLINE 0 0 0
5296558483604593766 UNAVAIL 0 0 0 was /dev/sdc1

errors: No known data errors

So I decided to detach sdc and then add the re-positioned sdc (now sdd) to mirror-1. Well this kind of worked. The pool is healthy but mirror-1 is gone. I was able to add my 2 new disks as another mirror now mirror-3 (really should be 2 but what ever)

root@pve:~# zpool status
pool: zones
state: ONLINE
scan: resilvered 11.5G in 00:03:28 with 0 errors on Mon Apr 26 07:40:25 2021
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST8000VN0022-2EL112_ZA1EL30R ONLINE 0 0 0
ata-ST8000VN0022-2EL112_ZA1EJPDN ONLINE 0 0 0
sdb ONLINE 0 0 0
ata-ST8000AS0002-1NA17Z_Z840CZMZ ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
sdc ONLINE 0 0 0
ata-ST8000VN004-2M2101_WSD1D5NP ONLINE 0 0 0

Is there any way to clean this up without having to destroy the zpool?
 
Hi, you should try first to replace your sdc/sdd/sde whatever names with uuid or disknames.
Because the sdc/sdd etc names can change during reboots on some conditions or systems.

As you have mirrors, you should able to do it 1 by 1, without having to destroy anything. Just waiting for resilvering, which will take you overall quite some time.

Maybe your mirror-1 comes back to life after that again, if all other disks aren't mapped by the sdX names. Even if not, it will be later a Problem and can be solved later once you did the first step above.

Cheers
 
Ramalama said:
Hi, you should try first to replace your sdc/sdd/sde whatever names with uuid or disknames.
Because the sdc/sdd etc names can change during reboots on some conditions or systems.

As you have mirrors, you should able to do it 1 by 1, without having to destroy anything. Just waiting for resilvering, which will take you overall quite some time.

Maybe your mirror-1 comes back to life after that again, if all other disks aren't mapped by the sdX names. Even if not, it will be later a Problem and can be solved later once you did the first step above.

Cheers
Click to expand...
suggested commands?
 
so I detached sdc from mirror-3 and when I did mirror-3 like mirror-1 before it went away. I did add sdc with its id but now I have 3 orphaned disks. See below.

Before:
Code: 
root@pve:~# zpool status
  pool: zones
 state: ONLINE
  scan: resilvered 11.5G in 00:03:28 with 0 errors on Mon Apr 26 07:40:25 2021
config:

        NAME                                  STATE     READ WRITE CKSUM
        zones                                 ONLINE       0     0     0
          mirror-0                            ONLINE       0     0     0
            ata-ST8000VN0022-2EL112_ZA1EL30R  ONLINE       0     0     0
            ata-ST8000VN0022-2EL112_ZA1EJPDN  ONLINE       0     0     0
          sdb                                 ONLINE       0     0     0
          ata-ST8000AS0002-1NA17Z_Z840CZMZ    ONLINE       0     0     0
          mirror-3                            ONLINE       0     0     0
            sdc                               ONLINE       0     0     0
            ata-ST8000VN004-2M2101_WSD1D5NP   ONLINE       0     0    0

zpool detach zones sdc

I ended up with

Code: 
root@pve:~# zpool status zones
  pool: zones
 state: ONLINE
  scan: resilvered 11.5G in 00:03:28 with 0 errors on Mon Apr 26 07:40:25 2021
config:

        NAME                                  STATE     READ WRITE CKSUM
        zones                                 ONLINE       0     0     0
          mirror-0                            ONLINE       0     0     0
            ata-ST8000VN0022-2EL112_ZA1EL30R  ONLINE       0     0     0
            ata-ST8000VN0022-2EL112_ZA1EJPDN  ONLINE       0     0     0
          sdb                                 ONLINE       0     0     0
          ata-ST8000AS0002-1NA17Z_Z840CZMZ    ONLINE       0     0     0
          ata-ST8000VN004-2M2101_WSD1D5NP     ONLINE       0     0     0

I then added sdc back using its ID

zpool add zones ata-ST8000AS0002-1NA17Z_Z840EMVD

Ended up with

Code: 
root@pve:~# zpool status
  pool: zones
 state: ONLINE
  scan: resilvered 11.5G in 00:03:28 with 0 errors on Mon Apr 26 07:40:25 2021
config:

        NAME                                  STATE     READ WRITE CKSUM
        zones                                 ONLINE       0     0     0
          mirror-0                            ONLINE       0     0     0
            ata-ST8000VN0022-2EL112_ZA1EL30R  ONLINE       0     0     0
            ata-ST8000VN0022-2EL112_ZA1EJPDN  ONLINE       0     0     0
          sdb                                 ONLINE       0     0     0
          ata-ST8000AS0002-1NA17Z_Z840CZMZ    ONLINE       0     0     0
          ata-ST8000VN004-2M2101_WSD1D5NP     ONLINE       0     0     0
          ata-ST8000AS0002-1NA17Z_Z840EMVD    ONLINE       0     0     0

errors: No known data errors

So I am down 2 mirrors now. I am not sure if I want to touch sdb yet until I I can reestablish the mirrors.
 
After you detached you disk, you can attach it again with:
zpool attach [poolname] [original drive to be mirrored] [new drive]

original drive = just the name that you get from zpool status
new drive = /dev/disk/by-id/ata-ST8000.....

As you see, you can make you mirror again, you have attached the drive simply wrong.

It's normal, that if you detach the drive, it doesn't show as mirror, because one drive isn't a mirror. Hopefully that's understandable.

But there are a ton of guides for zfs commands etc, all over the internet.
 
Ramalama said:
After you detached you disk, you can attach it again with:
zpool attach [poolname] [original drive to be mirrored] [new drive]

original drive = just the name that you get from zpool status
new drive = /dev/disk/by-id/ata-ST8000.....

As you see, you can make you mirror again, you have attached the drive simply wrong.

It's normal, that if you detach the drive, it doesn't show as mirror, because one drive isn't a mirror. Hopefully that's understandable.

But there are a ton of guides for zfs commands etc, all over the internet.
Click to expand...
This looks like its going to work. I removed one of the disks from the new mirror:

Code: 
zpool remove zones ata-ST8000AS0002-1NA17Z_Z840EMVD

Then I created a new mirror, using the existing disk and adding the one I just removed

Code: 
zpool attach zones ata-ST8000VN004-2M2101_WSD1D5NP ata-ST8000AS0002-1NA17Z_Z840EMVD

Once the resilver completed I moved to the next set of disks. I nervously removed sdb (has real data)

Code: 
zpool remove zones sdb

ZFS was happy with the command and is moving data off sdb

Code: 
root@pve:~# zpool status
  pool: zones
 state: ONLINE
  scan: resilvered 1.32G in 00:03:14 with 0 errors on Mon Apr 26 13:25:01 2021
remove: Evacuation of /dev/sdb1 in progress since Mon Apr 26 13:26:22 2021
        66.3G copied out of 2.86T at 104M/s, 2.27% done, 7h47m to go
        31.7K memory used for removed device mappings
config:

        NAME                                  STATE     READ WRITE CKSUM
        zones                                 ONLINE       0     0     0
          mirror-0                            ONLINE       0     0     0
            ata-ST8000VN0022-2EL112_ZA1EL30R  ONLINE       0     0     0
            ata-ST8000VN0022-2EL112_ZA1EJPDN  ONLINE       0     0     0
          sdb                                 ONLINE       0     0     0
          ata-ST8000AS0002-1NA17Z_Z840CZMZ    ONLINE       0     0     0
          mirror-3                            ONLINE       0     0     0
            ata-ST8000VN004-2M2101_WSD1D5NP   ONLINE       0     0     0
            ata-ST8000AS0002-1NA17Z_Z840EMVD  ONLINE       0     0     0

errors: No known data errors

Once the evacuation completes I will recreate the mirror using the disk ID of sdb. Thanks ..
 
  • Like
Reactions: Ramalama
I learned something. If I exported the zpool then imported it using zpool import -d /dev/disk/by-id <zpool id> all my sd devices would have been converted to disk ids. Then I could have added the new disks without the problems.. Hope this helps someone else
 
  • Like
Reactions: Ramalama
Zpool is now in a good state with the two new drives added. All done without data loss which amazes me. Thank you ZFS :)

Code: 
pool: zones
 state: ONLINE
  scan: resilvered 1.32G in 00:03:14 with 0 errors on Mon Apr 26 13:25:01 2021
remove: Removal of vdev 2 copied 982G in 4h51m, completed on Tue Apr 27 09:38:46 2021
        8.89M memory used for removed device mappings
config:

        NAME                                  STATE     READ WRITE CKSUM
        zones                                 ONLINE       0     0     0
          mirror-0                            ONLINE       0     0     0
            ata-ST8000VN0022-2EL112_ZA1EL30R  ONLINE       0     0     0
            ata-ST8000VN0022-2EL112_ZA1EJPDN  ONLINE       0     0     0
          mirror-3                            ONLINE       0     0     0
            ata-ST8000VN004-2M2101_WSD1D5NP   ONLINE       0     0     0
            ata-ST8000AS0002-1NA17Z_Z840EMVD  ONLINE       0     0     0
          mirror-5                            ONLINE       0     0     0
            ata-ST8000AS0002-1NA17Z_Z840CZMZ  ONLINE       0     0     0
            ata-ST8000AS0002-1NA17Z_Z840EVQJ  ONLINE       0     0     0

errors: No known data errors
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom