ZFS after disk change - import

[lowcrash]

Hello

My Proxmox server has fault disk with system data. VM's are on second disk in zfs raid. I have changed system disk, install latest Proxmox and boot system. System has found LVM pool and import pool. I have manulay import zpool with cmd zpool import rpool -m and system has booted normaly. But after every reboot I need to manualy import zpool with -m flag. Disk Samsung-SSD-970 is changed, how can I remove this pool and how can I repair cache pool? Do I need to replace logs and how?

Tnx for your help, L.

 

[news]

Hello, do you know the place to be a zfs guru?

# https://docs.oracle.com/en/operatin...manage-zfs/removing-devices-storage-pool.html

Check it.

 

[lowcrash]

I have removed UNAVAIL disk. But there is still issue with corrupted data on cache.

 

[news]

Not replaced? and how? did you managed, that the zfs pool is still alive and clear?

 

[news]

What is the current status of your zfs pool, run

zfs status <pool>

 

[lowcrash]

This is output:

pool: rpool
state: ONLINE
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
scan: scrub repaired 0B in 00:46:38 with 0 errors on Sun May 12 01:10:39 2024
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
cache
nvme0n1p2 FAULTED 0 0 0 corrupted data

errors: No known data errors

 

[news]

So you will remove the zfs cache device nvme0n1p2 also?

 

[lowcrash]

I want to repair this zfs cache. Why is corrupted? On this disk nvme0n1p2 is proxmox system and this is new installation - old disk has been replaced after failure.

 

[justinclift]

Hmmm. Would you be ok to run lsblk -o tran,name,type,size,vendor,model,label,rota,phy-sec and paste the output here?

That'll give the existing structure of your storage, which will help reduce the amount of back and forth questions.

 

[lowcrash]

root@pmx04:~# lsblk -o tran,name,type,size,vendor,model,label,rota,phy-sec
TRAN   NAME               TYPE   SIZE VENDOR   MODEL                          LABEL           ROTA PHY-SEC
sata   sda                disk 931.5G ATA      Samsung SSD 860 EVO 1TB                           0     512
       ├─sda1             part  1007K                                                            0     512
       ├─sda2             part 931.5G                                         rpool              0     512
       └─sda9             part     8M                                                            0     512
sata   sdb                disk 931.5G ATA      Samsung SSD 860 EVO 1TB                           0     512
       ├─sdb1             part  1007K                                                            0     512
       ├─sdb2             part 931.5G                                         rpool              0     512
       └─sdb9             part     8M                                                            0     512
       zd0                disk     8G                                                            0    4096
       zd16               disk   100G                                                            0    8192
       ├─zd16p1           part   500M                                         System Reserved    0    8192
       └─zd16p2           part  99.5G                                                            0    8192
       zd32               disk   150G                                                            0    8192
       ├─zd32p1           part   500M                                                            0    8192
       ├─zd32p2           part  49.5G                                                            0    8192
       └─zd32p3           part   100G                                                            0    8192
       zd48               disk    50G                                                            0    8192
       ├─zd48p1           part   243M                                                            0    8192
       ├─zd48p2           part     1K                                                            0    8192
       └─zd48p5           part  49.8G                                                            0    8192
       zd64               disk   370G                                                            0    8192
       ├─zd64p1           part   500M                                         System Reserved    0    8192
       └─zd64p2           part 369.5G                                                            0    8192
       zd80               disk    16G                                                            0    8192
       ├─zd80p1           part     1G                                                            0    8192
       └─zd80p2           part    15G                                                            0    8192
       zd96               disk   300G                                                            0    8192
       ├─zd96p1           part   128M                                                            0    8192
       └─zd96p2           part 299.9G                                         Data               0    8192
       zd112              disk   100G                                                            0    8192
       ├─zd112p1          part    50M                                         System Reserved    0    8192
       ├─zd112p2          part  99.4G                                                            0    8192
       └─zd112p3          part   522M                                                            0    8192
       zd128              disk   150G                                                            0    8192
       ├─zd128p1          part   500M                                                            0    8192
       ├─zd128p2          part  49.5G                                                            0    8192
       └─zd128p3          part   100G                                                            0    8192
nvme   nvme0n1            disk 465.8G          Samsung SSD 970 EVO Plus 500GB                    0     512
nvme   ├─nvme0n1p1        part  1007K                                                            0     512
nvme   ├─nvme0n1p2        part     1G                                                            0     512
nvme   └─nvme0n1p3        part 464.8G                                                            0     512
         ├─pve-swap       lvm      8G                                                            0     512
         ├─pve-root       lvm     96G                                                            0     512
         ├─pve-data_tmeta lvm    3.4G                                                            0     512
         │ └─pve-data     lvm  337.9G                                                            0     512
         └─pve-data_tdata lvm  337.9G                                                            0     512
           └─pve-data     lvm  337.9G                                                            0     512

 

[justinclift]

Thanks, that's helpful. Please do the same thing for "zpool status" too (in code block, monospace font), as that'll show the ZFS layout for all of your pools and help people understand which member has the fault status.

 

[justinclift]

Oh, it looks like you were running with a single log device, and it went bad?

When you have some free time, it's probably a good idea to read through this:

https://arstechnica.com/information...01-understanding-zfs-storage-and-performance/

 

[lowcrash]

  pool: rpool
 state: ONLINE
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
  scan: scrub repaired 0B in 00:41:27 with 0 errors on Sun Jun  9 01:05:28 2024
config:

        NAME         STATE     READ WRITE CKSUM
        rpool        ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            sda2     ONLINE       0     0     0
            sdb2     ONLINE       0     0     0
        cache
          nvme0n1p2  FAULTED      0     0     0  corrupted data

errors: No known data errors

 

[lowcrash]

Oh, it looks like you were running with a single log device, and it went bad?

When you have some free time, it's probably a good idea to read through this:

https://arstechnica.com/information...01-understanding-zfs-storage-and-performance/

Hello

Tnx, I know. This is an old server, which I need to migrate to new hardware. But ... there is always time problem.

 

[justinclift]

Understood.

You really do need to remove that cache device though, using the instructions in the link which @news provided (here).

If you really, really, really want to use a cache device, then can you find an additional NVMe drive or something so you can run a pair of them?

An individual NVMe cache drive, especially when it's using consumer grade NVMe like you're using, is a very bad idea. If you run a pair of them in a mirror instead, at least you can avoid corrupting the pool if/when one has a bit-flip.