I’m currently running a single server, with 3 ZFS pools:
- rpool, mirrored, containing the OS.
- pool0, striped, containing (I thought*), only a PBS datastore.
- pool1, mirrored, containing the VMs.
A drive in pool0 failed a couple of days ago. I was away at the time, and the first I knew of it was that VMs started behaving strangely - SMB server clearly still running but refusing connections, CCTV not showing certain cameras, etc.
I managed to log on via SSH, and with no time to investigate, I did the only thing I could think of - ‘shutdown -h now’. Logs show that the server took well over an hour to actually power down. It wasn’t until I got local access and saw the drive failure in the BMC log, that I realised what had actually happened.
*It turns out that, through my incompetence, pool0 did actually contain one VM, which was running at the time. That’s dead. Fair enough, lesson hopefully learned.
What worries me is that at least 2 of the VMs on pool1 - not directly affected by the drive failure - also have significant data corruption. It appears that none shut down cleanly, judging by the ‘recovering journal’ messages when I got things running again.
So… why did that happen? When I built this server, my thinking was that if a drive in pool0 failed, the worst that would happen is that the pool would fault, I’d lose some backups (that’s what tapes are for), but everything else would continue as normal.
Advice for the future greatly appreciated!
- rpool, mirrored, containing the OS.
- pool0, striped, containing (I thought*), only a PBS datastore.
- pool1, mirrored, containing the VMs.
A drive in pool0 failed a couple of days ago. I was away at the time, and the first I knew of it was that VMs started behaving strangely - SMB server clearly still running but refusing connections, CCTV not showing certain cameras, etc.
I managed to log on via SSH, and with no time to investigate, I did the only thing I could think of - ‘shutdown -h now’. Logs show that the server took well over an hour to actually power down. It wasn’t until I got local access and saw the drive failure in the BMC log, that I realised what had actually happened.
*It turns out that, through my incompetence, pool0 did actually contain one VM, which was running at the time. That’s dead. Fair enough, lesson hopefully learned.
What worries me is that at least 2 of the VMs on pool1 - not directly affected by the drive failure - also have significant data corruption. It appears that none shut down cleanly, judging by the ‘recovering journal’ messages when I got things running again.
So… why did that happen? When I built this server, my thinking was that if a drive in pool0 failed, the worst that would happen is that the pool would fault, I’d lose some backups (that’s what tapes are for), but everything else would continue as normal.
Advice for the future greatly appreciated!
