Please help with my current zpool

Z

zaedfsh

New Member
Dec 24, 2023
5
0
1
First of all, I'm not sure whether this forum answers questions about zfs or not. If not, please forgive my abruptness and point me to the right direction. Thanks in advance!

I started to use Proxmox from version 5.4 on a 8-bay PC several years ago with a zpool containing one raidz1 vdev of four 8TB hdds. This zpool almost reached its max capacity serveral months ago. So I filled all the remaining bays of the machine with four more 14TB hdds, making another raidz1 vdev to extend the zpool. I also upgraded Proxmox to the lastest version then. Recently, I started to receive warning emails regarding two of the old 8TB hdds as follow

Code: 
The following warning/error was logged by the smartd daemon:

Device: /dev/sdg [SAT], 1 Offline uncorrectable sectors

Device info:
HGST HUH728080ALE600, S/N:VKGLS23X, WWN:5-000cca-254c88546, FW:A4GNT514, 8.00 TB

The current zpool status is shown below.

Code: 
pool: mainpool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 16:07:31 with 0 errors on Sun Dec 10 16:31:32 2023
config:

        NAME                                   STATE     READ WRITE CKSUM
        mainpool                               ONLINE       0     0     0
          raidz1-0                             ONLINE       0     0     0
            ata-HGST_HUH728080ALE600_VKGLS23X  ONLINE       0     0     0
            ata-HGST_HUH728080ALE600_VKGLY24X  ONLINE       0     0     0
            ata-HGST_HUH728080ALE600_VKGMRXVX  ONLINE       0     0     0
            ata-HGST_HUH728080ALE600_VKGMYEVX  ONLINE       0     0     0
          raidz1-1                             ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_9MG6JU8A   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_9MGNBNDU   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_9MGNGUNU   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_9MGNJ36T   ONLINE       0     0     0

errors: No known data errors

Since there's no extra bay available any more, I'm not confident to replace those two defective 8TB hdds. And the capactiy of newly-added raidz1-1 is surely able to cover raidz1-0, is it possible to move all my data from raidz1-0 to raidz1-1 and remove raidz1-0 afterwards? If not, what's the best idea for the current situation of my zpool?

BTW, do I need to run 'zpool upgrade' as suggested in zpool status?

The version info of my Proxmox:

Code: 
proxmox-ve: 8.1.0 (running kernel: 6.2.16-3-pve)
pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5: 6.5.11-7
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
proxmox-kernel-6.2.16-19-pve: 6.2.16-19
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.7-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx7
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.2-1
proxmox-backup-file-restore: 3.1.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.2
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-2
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.1.4
pve-qemu-kvm: 8.1.2-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1
 
Last edited:
zaedfsh said:
And the capactiy of newly-added raidz1-1 is surely able to cover raidz1-0, is it possible to move all my data from raidz1-0 to raidz1-1 and remove raidz1-0 afterwards?
Click to expand...
No, removing vdevs is only possible if there are no raidz1/2/3 vdevs involved.

Replace one disk at a time hope the other disks won't fail while they are stressed by the resilvering. If they do all data of all 8 disks is lost. Thats why you usually want at least a raidz2 when working with big HDDs.
And keep in mind that raid is not a backup. You should still have a backup of everything important so it doesn't matter that much when losing that pool, except for annoying downtime.


zaedfsh said:
BTW, do I need to run 'zpool upgrade' as suggested in zpool status?
Click to expand...
Only if you want to make use of the new ZFS features.
 
@Dunuin Thanks for your kind reply!

Since there's no extra hdd bay on the PC, how to replace the 8TB hdd?

If the data on raidz1-0 can be moved to raid1-1 first, is it more secure to replace the disks of raidz1-0? Or is it also not possible for raidz vdevs?

As you mentioned it's important to backup all the data, is it the best idea to redo the pool after backup?
 
zaedfsh said:
If the data on raidz1-0 can be moved to raid1-1 first,
Click to expand...
nope, the setup is like a "stipe" from the moment you've attached the 2nd vdev (that being raidz1-1) and that means that if a vdev fails, ALL the data in the zpool is basically... gone (safe for your backups). In short, the zpool will balance to an extend, the data, so that (on average) both vdevs will be having the percentage of data.

No, you can't "migrate" data between vdevs, zfs/zpools are not designed that way
Yes, that is an "enterprise" feature of the LVMs/VxVMs of this world, but sorry, that is not ZFS somethings not worth the effort if designed right from the start, and the ide of ZFS that the sysadmins can design correct from the word go.... contrary to enterprises that waffles around and need those features when they changed their minds for all the wrong reasons - personal experiences, apologies

Hendrik's rules of computing:
1. Make A BACKUP
2. Make *another* backup
2b) at one of those *off*provider
3. CHECK those backups for restorability
 
zaedfsh said:
The current zpool status is shown below.

Code: 
pool: mainpool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 16:07:31 with 0 errors on Sun Dec 10 16:31:32 2023
config:

NAME STATE READ WRITE CKSUM
mainpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-HGST_HUH728080ALE600_VKGLS23X ONLINE 0 0 0
ata-HGST_HUH728080ALE600_VKGLY24X ONLINE 0 0 0
ata-HGST_HUH728080ALE600_VKGMRXVX ONLINE 0 0 0
ata-HGST_HUH728080ALE600_VKGMYEVX ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
ata-WDC_WUH721414ALE6L4_9MG6JU8A ONLINE 0 0 0
ata-WDC_WUH721414ALE6L4_9MGNBNDU ONLINE 0 0 0
ata-WDC_WUH721414ALE6L4_9MGNGUNU ONLINE 0 0 0
ata-WDC_WUH721414ALE6L4_9MGNJ36T ONLINE 0 0 0

errors: No known data errors
Click to expand...
hmmm... I wonder if you haven't had a once off event that got "fixed" ??

check/.report on the SMART values for those two HDDs you are complaining is failing...
 
New ZFS on Linux versions do allow for removal of parts of a stripe. You could try removing one raidz1, which will probably take some time, and removing those drives if it succeeds.
 
leesteken said:
New ZFS on Linux versions do allow for removal of parts of a stripe. You could try removing one raidz1, which will probably take some time, and removing those drives if it succeeds.
Click to expand...
YMMV as I see it still as beta and not battle/prod tested ;)
 
leesteken said:
New ZFS on Linux versions do allow for removal of parts of a stripe. You could try removing one raidz1, which will probably take some time, and removing those drives if it succeeds.
Click to expand...
Is it secure as beta, @hvisage says so? What is the command?
 
Last edited:
leesteken said:
I though your top-level is a stripe, and the second-level is raidz1. Did you try it?
Click to expand...

My zpool status is as below

Code: 
pool: mainpool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 16:07:31 with 0 errors on Sun Dec 10 16:31:32 2023
config:

        NAME                                   STATE     READ WRITE CKSUM
        mainpool                               ONLINE       0     0     0
          raidz1-0                             ONLINE       0     0     0
            ata-HGST_HUH728080ALE600_VKGLS23X  ONLINE       0     0     0
            ata-HGST_HUH728080ALE600_VKGLY24X  ONLINE       0     0     0
            ata-HGST_HUH728080ALE600_VKGMRXVX  ONLINE       0     0     0
            ata-HGST_HUH728080ALE600_VKGMYEVX  ONLINE       0     0     0
          raidz1-1                             ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_9MG6JU8A   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_9MGNBNDU   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_9MGNGUNU   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_9MGNJ36T   ONLINE       0     0     0

errors: No known data errors

I haven't try to remove anything, got to find a large hdd to backup my 18TB data first. If I redo the zpool with four 14TB and two 8TB hdds after backup, how to utilize the most capacity while keeping the possibility to remove defective hdds?
 
zaedfsh said:
If I redo the zpool with four 14TB and two 8TB hdds after backup, how to utilize the most capacity while keeping the possibility to remove defective hdds?
Click to expand...
Your questions are about ZFS details and not very specific to Proxmox. Maybe a ZFS on Linux forum could answer those kind of questions better? That's not to say that some people here are very knowledgeable about ZFS and they might be able to answer you here.
 
zaedfsh said:
Since there's no extra bay available any more, I'm not confident to replace those two defective 8TB hdds.
Click to expand...
uhh... you dont need extra bays to replace defective drives. just replace them. btw, no drive has "failed" meaningfully as far as your zpool is concerned so any faults are being corrected on-drive. they may be in prefailure state, but its not a critical issue yet.

zaedfsh said:
got to find a large hdd to backup my 18TB data first.
Click to expand...
a backup is a wise idea.
zaedfsh said:
If I redo the zpool with four 14TB and two 8TB hdds after backup, how to utilize the most capacity while keeping the possibility to remove defective hdds?
Click to expand...
a pool will only use the capacity of the smallest member in the vdev per disk. also, as a general rule, avoid single parity raid. the sanest setup for your pool is striped mirrors if you plan to use it for vm disks, or raidz2 if you dont- regardless of individual disk capacity.
 
You might want to run dmesg|more and take a good long look at the hard disk errors. As your pool currently shows no missing devices, you could have had a "oops the cable was very slightly loose" sort of intermittent issue. Could have been any of a zillion one-off events, or could have been a warning of disk about to die. Also read up on smartctl it is often useful for this sort of thing.

Fully agree with recommendations to just run stripped mirrors. All sorts of odd stuff happens with raidz1 and lots of drives, generally you get to learn stuff about zfs that you didn't want to know :)

Oh and finally, if you don't actually *need* one big pool, create poolA mirror d1 d2 mirror d3 d4 and create poolB mirror d5 d6 mirror d7 d8 gives you two pools, and it is likely that if one goes degraded or down, the other will still be ok. You get to decide where stuff goes (poolA or poolB) but for most of us that isn't really too hard. Two largeish pools lets you rsync critical data to the other in scary situations like this, before even starting to figure out what is wrong.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom