Hello, i am facing an issue with ZFS pool state.
One of the drives in the pool was marked as faulted, but the state of the pool has been kept ONLINE, instead of changing to DEGRADED.
The device of the pool failed previously that day and the pool correctly changed it's state to DEGRADED, but not the second time (i have cleared the faulted state the first time).
EDIT: I have managed to find this github issue https://github.com/openzfs/zfs/issues/16245
One of the drives in the pool was marked as faulted, but the state of the pool has been kept ONLINE, instead of changing to DEGRADED.
The device of the pool failed previously that day and the pool correctly changed it's state to DEGRADED, but not the second time (i have cleared the faulted state the first time).
proxmox-backup: 3.2.0 (running kernel: 6.8.8-2-pve)
proxmox-backup-server: 3.2.6-1 (running version: 3.2.6)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-4
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ifupdown2: 3.2.0-1+pmx8
libjs-extjs: 7.0.0-4
proxmox-backup-docs: 3.2.6-1
proxmox-backup-client: 3.2.6-1
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: not correctly installed
proxmox-widget-toolkit: 4.2.3
pve-xtermjs: 5.3.0-3
smartmontools: 7.3-pve1
zfsutils-linux: 2.2.4-pve1
proxmox-backup-server: 3.2.6-1 (running version: 3.2.6)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-4
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ifupdown2: 3.2.0-1+pmx8
libjs-extjs: 7.0.0-4
proxmox-backup-docs: 3.2.6-1
proxmox-backup-client: 3.2.6-1
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: not correctly installed
proxmox-widget-toolkit: 4.2.3
pve-xtermjs: 5.3.0-3
smartmontools: 7.3-pve1
zfsutils-linux: 2.2.4-pve1
Code:
$ zpool status
pool: rpool
state: ONLINE
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: resilvered 300M in 00:00:34 with 0 errors on Sat Jun 29 14:08:11 2024
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDT06-part3 ONLINE 0 0 0
ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 FAULTED 10 30 0 too many errors
ata-WDC_WD60EFPX-68C5ZN0_WD-WX22A82NERDU-part3 ONLINE 0 0 0
ata-WDC_WD60EFPX-68C5ZN0_WD-WX62D62K6ZA6-part3 ONLINE 0 0 0
ata-WDC_WD60EFPX-68C5ZN0_WD-WXH2D433R8SN-part3 ONLINE 0 0 0
errors: No known data errors
Code:
$ grep 'zed' /var/log/syslog.1 | tail
2024-06-29T22:46:17.570826+02:00 pbs kernel: [126753.634794] traps: agents[157627] general protection fault ip:5a4e550b1b16 sp:7458069fcd98 error:0 in zed[5a4e550a8000+e000]
2024-06-29T22:47:03.254337+02:00 pbs zed: eid=39 class=io pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=12288 offset=3986194722816 priority=2 err=5 flags=0x180180 delay=7134ms bookmark=387:12762736:0:26
2024-06-29T22:47:03.255078+02:00 pbs zed: eid=40 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=8192 offset=270336 priority=1 err=5 flags=0xb00c1 delay=67392ms
2024-06-29T22:47:03.255344+02:00 pbs zed[157631]: ZFS Event Daemon 2.2.4-pve1 (PID 157631)
2024-06-29T22:47:03.255404+02:00 pbs zed[157631]: Processing events since eid=68
2024-06-29T22:47:03.255538+02:00 pbs zed: eid=41 class=io pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=8192 offset=270336 priority=1 err=5 flags=0xb00c1 delay=67392ms
2024-06-29T22:47:03.255640+02:00 pbs zed: eid=42 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=8192 offset=6000099729408 priority=1 err=5 flags=0xb00c1 delay=238224ms
2024-06-29T22:47:03.255735+02:00 pbs zed: eid=43 class=io pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=8192 offset=6000099729408 priority=1 err=5 flags=0xb00c1 delay=238224ms
2024-06-29T22:47:03.255847+02:00 pbs zed: eid=44 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=24576 offset=2766348943360 priority=0 err=0 flags=0x180080 delay=197610ms bookmark=387:5051109:0:7
2024-06-29T22:47:03.255946+02:00 pbs zed: eid=45 class=deadman pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=4096 offset=6000100237312 priority=1 err=0 flags=0x1802c0 delay=126392732ms
2024-06-29T22:47:03.256036+02:00 pbs zed: eid=46 class=deadman pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=4096 offset=6000099975168 priority=1 err=0 flags=0x1802c0 delay=126392732ms
2024-06-29T22:47:03.256123+02:00 pbs zed: eid=47 class=deadman pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=4096 offset=516096 priority=1 err=0 flags=0x1802c0 delay=126392732ms
2024-06-29T22:47:03.256225+02:00 pbs zed: eid=48 class=deadman pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=4096 offset=253952 priority=1 err=5 flags=0x1802c0 delay=195192ms
2024-06-29T22:47:03.256309+02:00 pbs zed: eid=49 class=deadman pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=8192 offset=6000099991552 priority=1 err=0 flags=0xb00c1 delay=126392490ms
2024-06-29T22:47:03.256392+02:00 pbs zed: eid=50 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=8192 offset=6000099991552 priority=1 err=5 flags=0xb00c1 delay=316454ms
2024-06-29T22:47:03.256480+02:00 pbs zed: eid=51 class=io pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=8192 offset=6000099991552 priority=1 err=5 flags=0xb00c1 delay=316454ms
2024-06-29T22:47:03.256564+02:00 pbs zed: eid=52 class=probe_failure pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3
2024-06-29T22:47:03.256648+02:00 pbs zed: eid=53 class=io pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=98304 offset=3928903012352 priority=2 err=5 flags=0x40080480 delay=7150ms
2024-06-29T22:47:03.256743+02:00 pbs zed: eid=54 class=io pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928903077888 priority=0 err=5 flags=0x380180 bookmark=387:12762737:0:21
2024-06-29T22:47:03.256864+02:00 pbs zed: eid=55 class=io pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928903045120 priority=0 err=5 flags=0x380180 bookmark=387:12762737:0:20
2024-06-29T22:47:03.256952+02:00 pbs zed: eid=56 class=io pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928903012352 priority=0 err=5 flags=0x380180 bookmark=387:12762737:0:19
2024-06-29T22:47:03.257036+02:00 pbs zed: eid=57 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=20480 offset=2766108209152 priority=0 err=0 flags=0x180080 delay=35552ms bookmark=387:5051109:0:11
2024-06-29T22:47:03.257120+02:00 pbs zed: eid=58 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=12288 offset=4087836405760 priority=0 err=0 flags=0x180080 delay=291619ms bookmark=387:12744359:0:450
2024-06-29T22:47:03.257211+02:00 pbs zed: eid=59 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912179200 priority=0 err=0 flags=0x180080 delay=35820ms bookmark=387:12762738:0:6
2024-06-29T22:47:03.257310+02:00 pbs zed: eid=60 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928911982592 priority=0 err=0 flags=0x180080 delay=35821ms bookmark=387:12762738:0:1
2024-06-29T22:47:03.257394+02:00 pbs zed: eid=61 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912015360 priority=0 err=0 flags=0x180080 delay=35820ms bookmark=387:12762738:0:2
2024-06-29T22:47:03.257480+02:00 pbs zed: eid=62 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912211968 priority=0 err=0 flags=0x180080 delay=35820ms bookmark=387:12762738:0:7
2024-06-29T22:47:03.257562+02:00 pbs zed: eid=63 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912048128 priority=0 err=0 flags=0x180080 delay=35821ms bookmark=387:12762738:0:0
2024-06-29T22:47:03.257650+02:00 pbs zed: eid=64 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912080896 priority=0 err=0 flags=0x180080 delay=35821ms bookmark=387:12762738:0:3
2024-06-29T22:47:03.257733+02:00 pbs zed: eid=65 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912113664 priority=0 err=0 flags=0x180080 delay=35821ms bookmark=387:12762738:0:5
2024-06-29T22:47:03.257852+02:00 pbs zed: eid=66 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912146432 priority=0 err=0 flags=0x180080 delay=35821ms bookmark=387:12762738:0:4
2024-06-29T22:47:03.257959+02:00 pbs zed: eid=67 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=24576 offset=3487536828416 priority=2 err=0 flags=0x180180 delay=91795ms bookmark=387:12662023:0:3
2024-06-29T22:47:03.258049+02:00 pbs zed: eid=68 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=24576 offset=3487536873472 priority=0 err=0 flags=0x180080 delay=91795ms bookmark=387:12662023:0:2
2024-06-29T22:47:03.258136+02:00 pbs zed: eid=69 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=24576 offset=2766349225984 priority=0 err=0 flags=0x180080 delay=38104ms bookmark=387:5051117:0:12
2024-06-29T22:47:03.258222+02:00 pbs zed: eid=70 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912277504 priority=0 err=0 flags=0x180080 delay=53814ms bookmark=387:12762738:0:11
2024-06-29T22:47:03.258307+02:00 pbs zed: eid=71 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912310272 priority=0 err=0 flags=0x180080 delay=53814ms bookmark=387:12762738:0:10
2024-06-29T22:47:03.258396+02:00 pbs zed: eid=72 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912375808 priority=0 err=0 flags=0x180080 delay=53814ms bookmark=387:12762738:0:13
2024-06-29T22:47:03.258491+02:00 pbs zed: eid=73 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912408576 priority=0 err=0 flags=0x180080 delay=53814ms bookmark=387:12762738:0:12
2024-06-29T22:47:03.258576+02:00 pbs zed: eid=74 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912441344 priority=0 err=0 flags=0x180080 delay=53814ms bookmark=387:12762738:0:14
2024-06-29T22:47:03.258662+02:00 pbs zed: eid=75 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=98304 offset=3928912474112 priority=2 err=0 flags=0x40080480 delay=53814ms
2024-06-29T22:47:03.258747+02:00 pbs zed: eid=76 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=32768 offset=3928912506880 priority=2 err=0 flags=0x180180 delay=53815ms bookmark=387:12762738:0:17
2024-06-29T22:47:03.258876+02:00 pbs zed: eid=77 class=delay pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=4096 offset=2766168104960 priority=2 err=0 flags=0x180180 delay=38169ms bookmark=387:5051109:0:13
2024-06-29T22:47:03.258977+02:00 pbs zed: eid=78 class=probe_failure pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3
2024-06-29T22:47:03.259065+02:00 pbs zed: eid=79 class=io pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 size=20480 offset=2766168059904 priority=0 err=5 flags=0x180080 delay=28338ms bookmark=387:5051109:0:5
2024-06-29T22:47:03.259153+02:00 pbs zed: eid=80 class=statechange pool='rpool' vdev=ata-WDC_WD60EFPX-68C5ZN0_WD-WX12A82CDUVX-part3 vdev_state=FAULTEDEDIT: I have managed to find this github issue https://github.com/openzfs/zfs/issues/16245
Last edited: