Failed hard drive = chaos!

[alex591]

I’m currently running a single server, with 3 ZFS pools:
- rpool, mirrored, containing the OS.
- pool0, striped, containing (I thought*), only a PBS datastore.
- pool1, mirrored, containing the VMs.

A drive in pool0 failed a couple of days ago. I was away at the time, and the first I knew of it was that VMs started behaving strangely - SMB server clearly still running but refusing connections, CCTV not showing certain cameras, etc.

I managed to log on via SSH, and with no time to investigate, I did the only thing I could think of - ‘shutdown -h now’. Logs show that the server took well over an hour to actually power down. It wasn’t until I got local access and saw the drive failure in the BMC log, that I realised what had actually happened.

*It turns out that, through my incompetence, pool0 did actually contain one VM, which was running at the time. That’s dead. Fair enough, lesson hopefully learned.

What worries me is that at least 2 of the VMs on pool1 - not directly affected by the drive failure - also have significant data corruption. It appears that none shut down cleanly, judging by the ‘recovering journal’ messages when I got things running again.

So… why did that happen? When I built this server, my thinking was that if a drive in pool0 failed, the worst that would happen is that the pool would fault, I’d lose some backups (that’s what tapes are for), but everything else would continue as normal.

Advice for the future greatly appreciated!

 

[FSIN]

Hi
This seems pretty weird. ZFS pools are normaly logically divided, so from the current context, I don't see the link between one pool and another. How are the disks connected to the server, and how is pool0 mounted to your PBS instance?
zpool status output would be interesting to.

 

[alex591]

Hi,

It's a PowerEdge R730, and the disks are connected via the H330 in HBA mode. Pool0 is just mounted at /zpool/pool0 with a volume under that for PBS and another (now deleted to avoid future mistakes!) for VMs. zpool status probably isn't that interesting now since I destroyed and recreated pool0 following removal of the failed drive, but:

  pool: pool0
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        pool0       ONLINE       0     0     0
          AAAW-03   ONLINE       0     0     0
          AAAK-03   ONLINE       0     0     0
          AAAB-03   ONLINE       0     0     0

errors: No known data errors

  pool: pool1
 state: ONLINE
  scan: scrub repaired 0B in 09:24:34 with 0 errors on Thu Jul 16 05:46:13 2026
remove: Removal of vdev 0 copied 2.44T in 11h13m, completed on Sun May  9 22:13:41 2021
        125K memory used for removed device mappings
config:

        NAME          STATE     READ WRITE CKSUM
        pool1         ONLINE       0     0     0
          mirror-1    ONLINE       0     0     0
            AAAN-03   ONLINE       0     0     0
            AAAM-03   ONLINE       0     0     0
          mirror-2    ONLINE       0     0     0
            AAAO-03   ONLINE       0     0     0
            AAAP-03   ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 00:02:59 with 0 errors on Wed Jul 15 20:21:25 2026
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          AAAU-03   ONLINE       0     0     0
          AAAV-03   ONLINE       0     0     0

errors: No known data errors

Well there is one interesting thing... rpool is actually not mirrored! Some more work for tomorrow, but not relevant to this particular disaster.

 

[leesteken]

*It turns out that, through my incompetence, pool0 did actually contain one VM, which was running at the time. That’s dead. Fair enough, lesson hopefully learned.

This can be prevented by not selecting Disk image for that Storage in PVE. You can prevent certain types of data on each storage in Proxmox.

zpool status probably isn't that interesting now since I destroyed and recreated pool0 following removal of the failed drive

It will be difficult to determine what happened when you destroy evidence.

Well there is one interesting thing... rpool is actually not mirrored! Some more work for tomorrow, but not relevant to this particular disaster.

It is an indication that things are not as you intended or expected. It adds grains of salt to the evidence you are claiming to present, although I think you do mean well.

What worries me is that at least 2 of the VMs on pool1 - not directly affected by the drive failure - also have significant data corruption. It appears that none shut down cleanly, judging by the ‘recovering journal’ messages when I got things running again.

It is worrying that you are experiencing silent data corruption without ZFS noticing. Either you turned checksums off (did you for performance?) or your system has bad memory, which can cause all kinds of weird problems. Did ZFS detect checksum errors before you possibly cleared the information? Did you (every) run a scrub on that pool? Are those drives working correctly or maybe storing corrupt data due to bad memory? Did you run a memtest?

When I built this server, my thinking was that if a drive in pool0 failed, the worst that would happen is that the pool would fault, I’d lose some backups (that’s what tapes are for), but everything else would continue as normal.

That is what is intended to happen but who knows that the actual original situation was and what kind of root process wrecked havoc during the hour of shutting down? Maybe pressing the reset button would have been better in hindsight. I assume you have backups of anything important to you and you can reinstall and restore everything from backups?

 

[alex591]

This can be prevented by not selecting Disk image for that Storage in PVE. You can prevent certain types of data on each storage in Proxmox.

Yes... I did originally want to use that storage as a temporary measure while I was sorting other things out. Selecting it for this particular VM was not intended!

It will be difficult to determine what happened when you destroy evidence.

Agreed, but I'm not sure much useful has been lost there. At the time of the failure, 'zpool status' was showing - as expected - a load of read/write (and checksum? probably) errors related to the failed drive.

It is an indication that things are not as you intended or expected. It adds grains of salt to the evidence you are claiming to present, although I think you do mean well.

That pool was created when I first installed Proxmox on this machine a couple of months ago. I'm sure I selected 'mirrored', but... I guess I didn't, unless there is a recent bug in the installer!

It is worrying that you are experiencing silent data corruption without ZFS noticing. Either you turned checksums off (did you for performance?) or your system has bad memory, which can cause all kinds of weird problems. Did ZFS detect checksum errors before you possibly cleared the information? Did you (every) run a scrub on that pool? Are those drives working correctly or maybe storing corrupt data due to bad memory? Did you run a memtest?

Lots of questions there! No, I didn't turn checksums off - that's pretty much the entire reason I'm using ZFS. The pool was scrubbed clean just a day or two before this happened. Memtest - not recently, and since then all the RAM has been replaced (upgraded). I don't think it is a case of silent data corruption. It seems to me that the VMs running at the time were just starved of resources.

That is what is intended to happen but who knows that the actual original situation was and what kind of root process wrecked havoc during the hour of shutting down? Maybe pressing the reset button would have been better in hindsight. I assume you have backups of anything important to you and you can reinstall and restore everything from backups?

In hindsight, that might well have been better.

Thankfully yes, I was able to restore most things from tape. I only lost a couple of days of not particularly important data that, in theory, I would have been able to recover if I thought it worth the time.