Hi,
We have Proxmox (paid and fully updated) set up with 2 nvme drives set up in RAID1 using ZFS. When system restarts I get 2 emails with error message:
ZFS has detected that a device was removed. Both drives are securely mounted, and the ambient temp is around 65F. Any suggestions on how to fix it? This system keeps shutting down randomly, I made BIOS change, it'll take few days to know if that fixed it. Proxmox is set up on MinisForum MS-01 if that makes a difference.
Thank you
EMAIL 1
impact: Fault tolerance of the pool may be compromised.
eid: 7
class: statechange
state: UNAVAIL
host: VM01
time: 2025-07-02 07:42:32-0400
vpath: /dev/nvme1n1p4
vguid: 0x412A95B88377327A
pool: zfs-raid1 (0x68BB09C9D812FB52)
EMAIL 2
ZFS has detected that a device was removed.
impact: Fault tolerance of the pool may be compromised.
eid: 6
class: statechange
state: UNAVAIL
host: VM01
time: 2025-07-02 07:42:32-0400
vpath: /dev/nvme0n1p4
vguid: 0x20B4B493A718D7D5
pool: zfs-raid1 (0x68BB09C9D812FB52)
When I check ZFS
root@VM01:~# zpool status -v
pool: rpool
state: ONLINE
scan: scrub repaired 0B in 00:01:27 with 0 errors on Sun Jun 8 00:25:28 2025
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme-eui.e8238fa6bf530001001b448b4d948316-part3 ONLINE 0 0 0
nvme-eui.e8238fa6bf530001001b448b4d94e111-part3 ONLINE 0 0 0
errors: No known data errors
pool: zfs-raid1
state: ONLINE
scan: scrub repaired 0B in 00:00:01 with 0 errors on Sun Jun 8 00:24:03 2025
config:
NAME STATE READ WRITE CKSUM
zfs-raid1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme1n1p4 ONLINE 0 0 0
nvme0n1p4 ONLINE 0 0 0
errors: No known data errors
root@VM01:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
zd0 230:0 0 32.5G 0 disk
zd16 230:16 0 256G 0 disk
├─zd16p1 230:17 0 100M 0 part
├─zd16p2 230:18 0 16M 0 part
├─zd16p3 230:19 0 255.2G 0 part
└─zd16p4 230:20 0 681M 0 part
zd32 230:32 0 32G 0 disk
├─zd32p1 230:33 0 100M 0 part
├─zd32p2 230:34 0 16M 0 part
├─zd32p3 230:35 0 31.2G 0 part
└─zd32p4 230:36 0 650M 0 part
zd48 230:48 0 32.5G 0 disk
zd64 230:64 0 256G 0 disk
├─zd64p1 230:65 0 100M 0 part
├─zd64p2 230:66 0 16M 0 part
├─zd64p3 230:67 0 255.2G 0 part
└─zd64p4 230:68 0 674M 0 part
zd80 230:80 0 1M 0 disk
zd96 230:96 0 100G 0 disk
├─zd96p1 230:97 0 16M 0 part
└─zd96p2 230:98 0 100G 0 part
zd112 230:112 0 4M 0 disk
zd128 230:128 0 4M 0 disk
zd144 230:144 0 1M 0 disk
zd160 230:160 0 100G 0 disk
├─zd160p1 230:161 0 16M 0 part
└─zd160p2 230:162 0 100G 0 part
nvme1n1 259:0 0 1.8T 0 disk
├─nvme1n1p1 259:4 0 1007K 0 part
├─nvme1n1p2 259:6 0 1G 0 part
├─nvme1n1p3 259:7 0 255G 0 part
└─nvme1n1p4 259:9 0 1.6T 0 part
nvme0n1 259:1 0 1.8T 0 disk
├─nvme0n1p1 259:2 0 1007K 0 part
├─nvme0n1p2 259:3 0 1G 0 part
├─nvme0n1p3 259:5 0 255G 0 part
└─nvme0n1p4 259:8 0 1.6T 0 part
root@VM01:~#
root@VM01:~# fdisk -l
Disk /dev/nvme0n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WD_BLACK SN850X 2000GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 9102246D-8B1B-433A-9CF1-D297EF75AC3C
Device Start End Sectors Size Type
/dev/nvme0n1p1 34 2047 2014 1007K BIOS boot
/dev/nvme0n1p2 2048 2099199 2097152 1G EFI System
/dev/nvme0n1p3 2099200 536870912 534771713 255G Solaris /usr & Apple ZFS
/dev/nvme0n1p4 536872960 3907028991 3370156032 1.6T Linux filesystem
Disk /dev/nvme1n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WD_BLACK SN850X 2000GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0989B109-2728-4359-99F8-470E52986595
Device Start End Sectors Size Type
/dev/nvme1n1p1 34 2047 2014 1007K BIOS boot
/dev/nvme1n1p2 2048 2099199 2097152 1G EFI System
/dev/nvme1n1p3 2099200 536870912 534771713 255G Solaris /usr & Apple ZFS
/dev/nvme1n1p4 536872960 3907028991 3370156032 1.6T Linux filesystem
Disk /dev/zd0: 32.49 GiB, 34884026368 bytes, 68132864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd16: 256 GiB, 274877906944 bytes, 536870912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: 2F959CB7-AAC2-4F8F-9A1E-B1E7ACD93551
Device Start End Sectors Size Type
/dev/zd16p1 2048 206847 204800 100M EFI System
/dev/zd16p2 206848 239615 32768 16M Microsoft reserved
/dev/zd16p3 239616 535472127 535232512 255.2G Microsoft basic data
/dev/zd16p4 535472128 536866815 1394688 681M Windows recovery environment
Disk /dev/zd32: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: 049F6257-1412-427C-96FA-AB2B614427B7
Device Start End Sectors Size Type
/dev/zd32p1 2048 206847 204800 100M EFI System
/dev/zd32p2 206848 239615 32768 16M Microsoft reserved
/dev/zd32p3 239616 65773567 65533952 31.2G Microsoft basic data
/dev/zd32p4 65773568 67104767 1331200 650M Windows recovery environment
Disk /dev/zd48: 32.49 GiB, 34884026368 bytes, 68132864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd64: 256 GiB, 274877906944 bytes, 536870912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: 69349E4E-E3A9-45DE-85D1-6C13283160AC
Device Start End Sectors Size Type
/dev/zd64p1 2048 206847 204800 100M EFI System
/dev/zd64p2 206848 239615 32768 16M Microsoft reserved
/dev/zd64p3 239616 535486463 535246848 255.2G Microsoft basic data
/dev/zd64p4 535486464 536866815 1380352 674M Windows recovery environment
Disk /dev/zd80: 1 MiB, 1048576 bytes, 2048 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd96: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: 90737111-FFD8-4EDD-AE61-A36E184B417C
Device Start End Sectors Size Type
/dev/zd96p1 34 32767 32734 16M Microsoft reserved
/dev/zd96p2 32768 209711103 209678336 100G Microsoft basic data
Partition 1 does not start on physical sector boundary.
Disk /dev/zd112: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd128: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd144: 1 MiB, 1048576 bytes, 2048 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd160: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: 012AE300-3F8D-4012-8639-3D2A2AE22C78
Device Start End Sectors Size Type
/dev/zd160p1 34 32767 32734 16M Microsoft reserved
/dev/zd160p2 32768 209711103 209678336 100G Microsoft basic data
Partition 1 does not start on physical sector boundary.
root@VM01:~# dmesg | grep -i nvme
[ 1.187823] nvme nvme0: pci function 0000:59:00.0
[ 1.187827] nvme nvme1: pci function 0000:58:00.0
[ 1.216320] nvme nvme0: 20/0/0 default/read/poll queues
[ 1.216344] nvme nvme1: 20/0/0 default/read/poll queues
[ 1.220444] nvme0n1: p1 p2 p3 p4
[ 1.220490] nvme1n1: p1 p2 p3 p4
root@VM01:~# journalctl -k | grep -i nvme
Jul 02 07:42:32 VM01-FL kernel: nvme nvme0: pci function 0000:59:00.0
Jul 02 07:42:32 VM01-FL kernel: nvme nvme1: pci function 0000:58:00.0
Jul 02 07:42:32 VM01-FL kernel: nvme nvme0: 20/0/0 default/read/poll queues
Jul 02 07:42:32 VM01-FL kernel: nvme nvme1: 20/0/0 default/read/poll queues
Jul 02 07:42:32 VM01-FL kernel: nvme0n1: p1 p2 p3 p4
Jul 02 07:42:32 VM01-FL kernel: nvme1n1: p1 p2 p3 p4
root@VM01:~#
We have Proxmox (paid and fully updated) set up with 2 nvme drives set up in RAID1 using ZFS. When system restarts I get 2 emails with error message:
ZFS has detected that a device was removed. Both drives are securely mounted, and the ambient temp is around 65F. Any suggestions on how to fix it? This system keeps shutting down randomly, I made BIOS change, it'll take few days to know if that fixed it. Proxmox is set up on MinisForum MS-01 if that makes a difference.
Thank you
EMAIL 1
impact: Fault tolerance of the pool may be compromised.
eid: 7
class: statechange
state: UNAVAIL
host: VM01
time: 2025-07-02 07:42:32-0400
vpath: /dev/nvme1n1p4
vguid: 0x412A95B88377327A
pool: zfs-raid1 (0x68BB09C9D812FB52)
EMAIL 2
ZFS has detected that a device was removed.
impact: Fault tolerance of the pool may be compromised.
eid: 6
class: statechange
state: UNAVAIL
host: VM01
time: 2025-07-02 07:42:32-0400
vpath: /dev/nvme0n1p4
vguid: 0x20B4B493A718D7D5
pool: zfs-raid1 (0x68BB09C9D812FB52)
When I check ZFS
root@VM01:~# zpool status -v
pool: rpool
state: ONLINE
scan: scrub repaired 0B in 00:01:27 with 0 errors on Sun Jun 8 00:25:28 2025
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme-eui.e8238fa6bf530001001b448b4d948316-part3 ONLINE 0 0 0
nvme-eui.e8238fa6bf530001001b448b4d94e111-part3 ONLINE 0 0 0
errors: No known data errors
pool: zfs-raid1
state: ONLINE
scan: scrub repaired 0B in 00:00:01 with 0 errors on Sun Jun 8 00:24:03 2025
config:
NAME STATE READ WRITE CKSUM
zfs-raid1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme1n1p4 ONLINE 0 0 0
nvme0n1p4 ONLINE 0 0 0
errors: No known data errors
root@VM01:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
zd0 230:0 0 32.5G 0 disk
zd16 230:16 0 256G 0 disk
├─zd16p1 230:17 0 100M 0 part
├─zd16p2 230:18 0 16M 0 part
├─zd16p3 230:19 0 255.2G 0 part
└─zd16p4 230:20 0 681M 0 part
zd32 230:32 0 32G 0 disk
├─zd32p1 230:33 0 100M 0 part
├─zd32p2 230:34 0 16M 0 part
├─zd32p3 230:35 0 31.2G 0 part
└─zd32p4 230:36 0 650M 0 part
zd48 230:48 0 32.5G 0 disk
zd64 230:64 0 256G 0 disk
├─zd64p1 230:65 0 100M 0 part
├─zd64p2 230:66 0 16M 0 part
├─zd64p3 230:67 0 255.2G 0 part
└─zd64p4 230:68 0 674M 0 part
zd80 230:80 0 1M 0 disk
zd96 230:96 0 100G 0 disk
├─zd96p1 230:97 0 16M 0 part
└─zd96p2 230:98 0 100G 0 part
zd112 230:112 0 4M 0 disk
zd128 230:128 0 4M 0 disk
zd144 230:144 0 1M 0 disk
zd160 230:160 0 100G 0 disk
├─zd160p1 230:161 0 16M 0 part
└─zd160p2 230:162 0 100G 0 part
nvme1n1 259:0 0 1.8T 0 disk
├─nvme1n1p1 259:4 0 1007K 0 part
├─nvme1n1p2 259:6 0 1G 0 part
├─nvme1n1p3 259:7 0 255G 0 part
└─nvme1n1p4 259:9 0 1.6T 0 part
nvme0n1 259:1 0 1.8T 0 disk
├─nvme0n1p1 259:2 0 1007K 0 part
├─nvme0n1p2 259:3 0 1G 0 part
├─nvme0n1p3 259:5 0 255G 0 part
└─nvme0n1p4 259:8 0 1.6T 0 part
root@VM01:~#
root@VM01:~# fdisk -l
Disk /dev/nvme0n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WD_BLACK SN850X 2000GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 9102246D-8B1B-433A-9CF1-D297EF75AC3C
Device Start End Sectors Size Type
/dev/nvme0n1p1 34 2047 2014 1007K BIOS boot
/dev/nvme0n1p2 2048 2099199 2097152 1G EFI System
/dev/nvme0n1p3 2099200 536870912 534771713 255G Solaris /usr & Apple ZFS
/dev/nvme0n1p4 536872960 3907028991 3370156032 1.6T Linux filesystem
Disk /dev/nvme1n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WD_BLACK SN850X 2000GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0989B109-2728-4359-99F8-470E52986595
Device Start End Sectors Size Type
/dev/nvme1n1p1 34 2047 2014 1007K BIOS boot
/dev/nvme1n1p2 2048 2099199 2097152 1G EFI System
/dev/nvme1n1p3 2099200 536870912 534771713 255G Solaris /usr & Apple ZFS
/dev/nvme1n1p4 536872960 3907028991 3370156032 1.6T Linux filesystem
Disk /dev/zd0: 32.49 GiB, 34884026368 bytes, 68132864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd16: 256 GiB, 274877906944 bytes, 536870912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: 2F959CB7-AAC2-4F8F-9A1E-B1E7ACD93551
Device Start End Sectors Size Type
/dev/zd16p1 2048 206847 204800 100M EFI System
/dev/zd16p2 206848 239615 32768 16M Microsoft reserved
/dev/zd16p3 239616 535472127 535232512 255.2G Microsoft basic data
/dev/zd16p4 535472128 536866815 1394688 681M Windows recovery environment
Disk /dev/zd32: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: 049F6257-1412-427C-96FA-AB2B614427B7
Device Start End Sectors Size Type
/dev/zd32p1 2048 206847 204800 100M EFI System
/dev/zd32p2 206848 239615 32768 16M Microsoft reserved
/dev/zd32p3 239616 65773567 65533952 31.2G Microsoft basic data
/dev/zd32p4 65773568 67104767 1331200 650M Windows recovery environment
Disk /dev/zd48: 32.49 GiB, 34884026368 bytes, 68132864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd64: 256 GiB, 274877906944 bytes, 536870912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: 69349E4E-E3A9-45DE-85D1-6C13283160AC
Device Start End Sectors Size Type
/dev/zd64p1 2048 206847 204800 100M EFI System
/dev/zd64p2 206848 239615 32768 16M Microsoft reserved
/dev/zd64p3 239616 535486463 535246848 255.2G Microsoft basic data
/dev/zd64p4 535486464 536866815 1380352 674M Windows recovery environment
Disk /dev/zd80: 1 MiB, 1048576 bytes, 2048 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd96: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: 90737111-FFD8-4EDD-AE61-A36E184B417C
Device Start End Sectors Size Type
/dev/zd96p1 34 32767 32734 16M Microsoft reserved
/dev/zd96p2 32768 209711103 209678336 100G Microsoft basic data
Partition 1 does not start on physical sector boundary.
Disk /dev/zd112: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd128: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd144: 1 MiB, 1048576 bytes, 2048 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disk /dev/zd160: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: 012AE300-3F8D-4012-8639-3D2A2AE22C78
Device Start End Sectors Size Type
/dev/zd160p1 34 32767 32734 16M Microsoft reserved
/dev/zd160p2 32768 209711103 209678336 100G Microsoft basic data
Partition 1 does not start on physical sector boundary.
root@VM01:~# dmesg | grep -i nvme
[ 1.187823] nvme nvme0: pci function 0000:59:00.0
[ 1.187827] nvme nvme1: pci function 0000:58:00.0
[ 1.216320] nvme nvme0: 20/0/0 default/read/poll queues
[ 1.216344] nvme nvme1: 20/0/0 default/read/poll queues
[ 1.220444] nvme0n1: p1 p2 p3 p4
[ 1.220490] nvme1n1: p1 p2 p3 p4
root@VM01:~# journalctl -k | grep -i nvme
Jul 02 07:42:32 VM01-FL kernel: nvme nvme0: pci function 0000:59:00.0
Jul 02 07:42:32 VM01-FL kernel: nvme nvme1: pci function 0000:58:00.0
Jul 02 07:42:32 VM01-FL kernel: nvme nvme0: 20/0/0 default/read/poll queues
Jul 02 07:42:32 VM01-FL kernel: nvme nvme1: 20/0/0 default/read/poll queues
Jul 02 07:42:32 VM01-FL kernel: nvme0n1: p1 p2 p3 p4
Jul 02 07:42:32 VM01-FL kernel: nvme1n1: p1 p2 p3 p4
root@VM01:~#