emergency mode, help please

[Nathan Davis]

Today I rebooted my server, just to do, and it booted into a emergency mode. Attached is the output of journalctl -xb


I am not sure what I could have done to cause it. The last thing I did before rebooting was to start re-silver a failed drive in my zfs pool.

 

[Miguel Lorenzo]

what happens if you execute zpool list?

 

[Nathan Davis]

root@pve:~# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
storage 97.5T 69.1T 28.4T - 14% 70% 1.00x DEGRADED -
root@pve:~# zpool status
pool: storage
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Mar 23 17:10:12 2018
45.1T scanned out of 68.8T at 916M/s, 7h31m to go
3.36T resilvered, 65.56% done
config:

NAME STATE READ WRITE CKSUM
storage DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
ata-WDC_WD60 ONLINE 0 0 0
ata-WDC_WD60 ONLINE 0 0 0
replacing-2 DEGRADED 0 0 0
ata-WDC_WD60E OFFLINE 0 0 0
ata-WDC_WD60E ONLINE 0 0 0 (resilvering)
ata-WDC_WD60E ONLINE 0 0 0
ata-HGST_HDN726 ONLINE 0 0 0
ata-HGST_HDN726 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
ata-WDC_WD60E ONLINE 0 0 0
ata-HGST_HDN72 ONLINE 0 0 0
ata-HGST_HDN726 ONLINE 0 0 0
ata-HGST_HDN726 ONLINE 0 0 0
ata-HGST_HDN72 ONLINE 0 0 0
ata-HGST_HDN726 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
ata-ST6000 ONLINE 0 0 0
ata-ST6000 ONLINE 0 0 0
ata-ST6000 ONLINE 0 0 0
ata-ST6000 ONLINE 0 0 0
ata-ST6000 ONLINE 0 0 0
ata-ST6000 ONLINE 0 0 0

errors: No known data errors
root@pve:~#

 

[Nathan Davis]

So I think I figured it out. I had a btrfs ssdat /dev/sds mounted to /sdd that seems to have failed or stopped working. Removing it from /etc/fstab and /etc/pve/storage.cfg and then rebooting fixed my issue. I figured it out by just searching through the log files for errors. I saw btrfs complaining a lot. The drive was just configured for backup,rootdir,vztmpl but just used for backups.