Howto remove disk from ZFS

[KwikDaFoe]

I wanted to add a disk to my raidz1-0 but instead I added it to rstorage only.

How do I remove it from there? I want to remove usb-Seagate_Expansion_NAAE7KF5-0:0

root@proxmox:~# zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rstorage  18.2T  5.58T  12.6T        -         -     0%    30%  1.00x    ONLINE  -


root@proxmox:~# zpool status
  pool: rstorage
 state: ONLINE
config:


        NAME                                                STATE     READ WRITE CKSUM
        rstorage                                            ONLINE       0     0     0
          raidz1-0                                          ONLINE       0     0     0
            usb-Seagate_One_Touch_HDD_00000000NABN4TNN-0:0  ONLINE       0     0     0
            usb-Seagate_One_Touch_HDD_00000000NABN4TJ2-0:0  ONLINE       0     0     0
            usb-Seagate_Portable_NAAHZAW9-0:0               ONLINE       0     0     0
          usb-Seagate_Expansion_NAAE7KF5-0:0                ONLINE       0     0     0


errors: No known data errors
root@proxmox:~# zpool detach rstorage usb-Seagate_Expansion_NAAE7KF5-0:0
cannot detach usb-Seagate_Expansion_NAAE7KF5-0:0: only applicable to mirror and replacing vdevs
root@proxmox:~# zpool remove rstorage usb-Seagate_Expansion_NAAE7KF5-0:0
cannot remove usb-Seagate_Expansion_NAAE7KF5-0:0: invalid config; all top-level vdevs must have the same sector size and not be raidz.

 

[leesteken]

I wanted to add a disk to my raidz1-0 but instead I added it to rstorage only.

As far as I know, you cannot add vdevs to a Raidz1/2/3.

How do I remove it from there? I want to remove usb-Seagate_Expansion_NAAE7KF5-0:0

As far as I know, recent versions of ZFS support removing vdevs from a stripe.

root@proxmox:~# zpool detach rstorage usb-Seagate_Expansion_NAAE7KF5-0:0
cannot detach usb-Seagate_Expansion_NAAE7KF5-0:0: only applicable to mirror and replacing vdevs
root@proxmox:~# zpool remove rstorage usb-Seagate_Expansion_NAAE7KF5-0:0
cannot remove usb-Seagate_Expansion_NAAE7KF5-0:0: invalid config; all top-level vdevs must have the same sector size and not be raidz.

Detach won't work but I would have expected remove to work. What is the output of pveversion -v?

 

[KwikDaFoe]

proxmox-ve: 8.1.0 (running kernel: 6.5.11-8-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-9
proxmox-kernel-6.5: 6.5.11-8
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
pve-kernel-5.15.131-2-pve: 5.15.131-3
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph: 17.2.7-pve2
ceph-fuse: 17.2.7-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.1
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-3
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.5-2
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1

 

[Himcules]

Could you try offlining the vdev first?

zpool offline VOLUME DEVICE

 

[Nemesiz]

Backup your data and recreate ZFS

 

[KwikDaFoe]

Could you try offlining the vdev first?

zpool offline VOLUME DEVICE

zpool offline rstorage usb-Seagate_Expansion_NAAE7KF5-0:0
cannot offline usb-Seagate_Expansion_NAAE7KF5-0:0: no valid replicas

 

[KwikDaFoe]

Backup your data and recreate ZFS

How would I do that?

I added the hdd, but I didnt use it in any way. Shouldnt it be possible to just remove it again?

 

[Nemesiz]

You cannot remove disk in your situation right now. Plan how to backup your data and recreate ZFS.

 

[KwikDaFoe]

You cannot remove disk in your situation right now. Plan how to backup your data and recreate ZFS.

I understand. Can you explain why it is not possible?

 

[Nemesiz]

ZFS was not designed to allow remove disk. It is very complex system. Yes you can remove disk from the mirror. Maybe one day it will be possible.

 

[KwikDaFoe]

Thanks for clearing that out. So recreation will wipe all my data and I need to restore from a backup?

 

[leesteken]

ZFS was not designed to allow remove disk. It is very complex system. Yes you can remove disk from the mirror. Maybe one day it will be possible.

The latest ZFS version that comes with the latest Proxmox can remove disks from a stripe. This has worked for people on this forum when they accidentally created a stripe instead of a mirror. I don't know why it does not work in this case.

 

[UdoB]

Can you explain why it is not possible?

It is not implemented. (Yet.)

The official statement for the specific installation on my (your) computer is here: man zpool-remove:

---

DESCRIPTION
zpool remove [-npw] pool device…
Removes the specified device from the pool. This command supports removing hot spare, cache, log, and both mirrored and non-redundant primary top-level vdevs, including dedup and special vdevs.

Top-level vdevs can only be removed if the primary pool storage does not contain a top-level raidz vdev, ...

---

Yes, bad news...