HELP NEEDED. Proxmox Container on XFS after HP RAID Crash

jagzno · Monday at 04:46

The Crash:The HP RAID controller unexpectedly dropped multiple physical disks simultaneously, causing an XFS error.

Code:

TASK ERROR: unable to activate storage 'backup-storage' - directory is expected to be a mount point but is not mounted: '/mnt/pve/backup-storage'

I tried to run xfs_repair -L to force-zero the log so the drive could mount again but failed and now i can't even mount it back again.
The raid is configured to RAID-10 (approx. 13 TB usable). All 7 months worth of data is in there.

Can anyone help please.

cwt · Monday at 07:22

Don‘t run any further xfs_repair commands. -L only clears the log.

Check within the controller if all disks appear within the controller and if any drive is marked faulty or further errors are logged (IML, smart array event log, etc.).

Check the output of

ssacli ctrl all show config detail

jagzno · Monday at 07:49

cwt said:
Don‘t run any further xfs_repair commands. -L only clears the log.

Check within the controller if all disks appear within the controller and if any drive is marked faulty or further errors are logged (IML, smart array event log, etc.).

Check the output of

ssacli ctrl all show config detail

oh no. I've run the command which is why it's now I'd say bad state. seems like it have corrupted the whole RAID 10 which I use it for my LXC storage. I tried a few recovery tools to recover it but it still not working.

The 101.disk is gone.

P822 (Slot 2) — D2700 enclosure at port 4E:

25 drives total, all SAS HDD, 1.2 TB each
24 are active data drives in Array A / Logical Drive 1 (13.10 TB, RAID 1+0), all Status: OK
1 is a hot spare (bay 1), also Status: OK, marked "auto replace spare"

cwt · Monday at 08:13

The fact that the P822 reports all 24 RAID members as OK is good, since it suggests the RAID itself is currently healthy. At this point, it’s important to determine whether this is a filesystem metadata issue, a missing mount, or an actual loss of the VM disk file before attempting any further recovery steps.

Could you share the output of:

lsblk -f
blkid
pvesm status
mount

jagzno · Monday at 08:33

Code:

=== lsblk -f ===
NAME FSTYPE FSVER LABEL        UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
loop0
loop1
     ext4de 18417 䊗k\x9c\xe0i c0f73a72-f929-183f-e71f-93ccde466dd8                
sda                                                                                
├─sda1
├─sda2
│    vfat   FAT32              B3DA-F5A1                                            
└─sda3
  │  LVM2_m LVM2               bEAULa-fjFx-O3Jk-fLvq-EpH4-7aTH-27dYBz              
  ├─pve-swap
  │  swap   1                  99845a15-fe9f-43de-802d-c18c203976ef                  [SWAP]
  ├─pve-root
  │  ext4   1.0                7cbfd07a-627f-46aa-bd97-f0dcecce2198     79.2G    11% /
  ├─pve-data_tmeta
  │ └─pve-data-tpool
  │   ├─pve-data
  │   ├─pve-recovery
  │   │  ext4   1.0                b3d7f39b-73e8-4fe9-9245-5f1a76c94bdf    265.1G    18% /mnt/recovery
  │   └─pve-vm--105--disk--0
  │      ext4   1.0                57c695f9-f358-4323-854a-7f4534808cc6                
  └─pve-data_tdata
    └─pve-data-tpool
      ├─pve-data
      ├─pve-recovery
      │  ext4   1.0                b3d7f39b-73e8-4fe9-9245-5f1a76c94bdf    265.1G    18% /mnt/recovery
      └─pve-vm--105--disk--0
         ext4   1.0                57c695f9-f358-4323-854a-7f4534808cc6                
sdb                                                                                
└─sdb1
     xfs                       a7e5e636-fbd7-48af-8d13-5170310da1bd     12.9T     2% /mnt/test_mount
sr0                                                                                

=== blkid ===
/dev/mapper/pve-root: UUID="7cbfd07a-627f-46aa-bd97-f0dcecce2198" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdb1: UUID="a7e5e636-fbd7-48af-8d13-5170310da1bd" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="eef39a20-f090-43a3-b44a-3143b8e3975f"
/dev/mapper/pve-swap: UUID="99845a15-fe9f-43de-802d-c18c203976ef" TYPE="swap"
/dev/sda2: UUID="B3DA-F5A1" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="cfd0432c-5155-405a-94f6-e4f8a3df0596"
/dev/sda3: UUID="bEAULa-fjFx-O3Jk-fLvq-EpH4-7aTH-27dYBz" TYPE="LVM2_member" PARTUUID="006e62b2-942b-4d0d-a86b-4b491b8f7ca9"
/dev/loop1: LABEL="M-dM-^JM-^WkM-^\M-`i" UUID="c0f73a72-f929-183f-e71f-93ccde466dd8" TYPE="ext4dev"
/dev/mapper/pve-recovery: UUID="b3d7f39b-73e8-4fe9-9245-5f1a76c94bdf" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/pve-vm--105--disk--0: UUID="57c695f9-f358-4323-854a-7f4534808cc6" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sda1: PARTUUID="350ba5b0-8dbc-48ac-94d1-7331986c848f"

=== pvesm status ===
unable to activate storage 'backup-storage' - directory is expected to be a mount point but is not mounted: '/mnt/pve/backup-storage'
Name                  Type     Status     Total (KiB)      Used (KiB) Available (KiB)        %
backup-pve             nfs     active     14062873600     14062872576            1024  100.00%
backup-storage         dir   inactive               0               0               0    0.00%
local                  dir     active        98497780        10381128        83067104   10.54%
local-lvm          lvmthin     active      3355799552       190944994      3164854557    5.69%
pbs                    pbs     active     14062873600     14062873168             432  100.00%

=== mount (Filtered for relevant drives) ===
10.10.10.1:/backup/pve on /mnt/pve/backup-pve type nfs4 (rw,relatime,vers=4.2)
/dev/mapper/pve-recovery on /mnt/recovery type ext4 (rw,relatime,stripe=64)
/dev/sdb1 on /mnt/test_mount type xfs (ro,relatime,norecovery,attr2,inode64,logbufs=8,logbsize=32k,noquota)

cwt said:
The fact that the P822 reports all 24 RAID members as OK is good, since it suggests the RAID itself is currently healthy. At this point, it’s important to determine whether this is a filesystem metadata issue, a missing mount, or an actual loss of the VM disk file before attempting any further recovery steps.

Could you share the output of:

lsblk -f
blkid
pvesm status
mount

/dev/sdb1 is my 13TB RAID array. It is currently safely mounted as Read-Only (ro,norecovery) at /mnt/test_mount. Proxmox reports backup-storage as inactive because I unmounted it to protect it.

It is not a missing mount. The 13TB array (/dev/sdb1) is successfully mounted as XFS (currently in read-only mode to protect it). The directory structure for the LXC containers is visible, but the 8TB disk image file itself is missing.
It is a severe filesystem metadata issue. We had a power loss, and the RAID controller battery was dead, meaning the write cache was lost. This caused massive XFS metadata corruption.
It resulted in the loss of the VM disk file's inode. Because the XFS log was corrupted during the crash, xfs_repair -L was run to force a repair. This cleared the log, but it completely orphaned/deleted the inode for the 8TB vm-101-disk-0.raw file.

cwt · Monday at 09:24

If the directory structure is intact but only the vm-101-disk-0.raw entry is missing, this sounds more like the inode became unreachable after the journal was discarded, rather than the data blocks necessarily being overwritten.

Before attempting any further recovery, I'd strongly recommend creating a sector-by-sector clone of /dev/sdb1 (or the entire RAID logical drive if possible). At this point, the filesystem is in its most valuable state.

One question: have you checked whether the inode was moved to lost+found, or whether xfs_db can still locate orphaned inodes? Also, have you already run xfs_metadump? That could allow deeper analysis without risking further changes to the original filesystem.

jagzno · Monday at 09:58

cwt said:
If the directory structure is intact but only the vm-101-disk-0.raw entry is missing, this sounds more like the inode became unreachable after the journal was discarded, rather than the data blocks necessarily being overwritten.

Before attempting any further recovery, I'd strongly recommend creating a sector-by-sector clone of /dev/sdb1 (or the entire RAID logical drive if possible). At this point, the filesystem is in its most valuable state.

One question: have you checked whether the inode was moved to lost+found, or whether xfs_db can still locate orphaned inodes? Also, have you already run xfs_metadump? That could allow deeper analysis without risking further changes to the original filesystem.

Thanks for the reply.

1. Sector-by-sector clone: I already did exactly this before doing anything else. used ddrescue to create a full 14TB sector-by-sector clone of /dev/sdb1 onto the NFS backup drive.

2. lost+found: Thoroughly checked the lost+found directory. There were some orphaned inodes in there, but they turned out to be legacy files from a 2025 backup. The massive 8TB inode for 2026 data was completely unlinked and did not end up in lost+found.

3. xfs_db and orphaned inodes: used xfs_db to check the free space extents. I found that the 13TB of free space is composed of exactly 12 massive, unbroken extents. The 8TB file's data blocks are sitting safely in the free space and are not heavily fragmented.

4. xfs_metadump: haven't run xfs_metadump yet, but keeping the array mounted strictly as read-only (ro,norecovery) to prevent any metadata changes while deciding on the best extraction tool.

I've ran the xfs_metadump but there were errors

Code:

Unknown directory buffer type!
Metadata corruption detected at 0x61f5706ff2c7, xfs_inode block 0x18013efe8/0x4000
Metadata corruption detected at 0x61f5706ff2c7, xfs_inode block 0x18013f008/0x4000
Unknown directory buffer type!
...

Is the XFS structural "map" itself is severely corrupted now?

Adam Camryn · Monday at 10:22

If multiple disks fail simultaneously, avoid repairs until the RAID array is confirmed healthy. Check the HP RAID controller logs to ensure all drives are online. If the array isn't fully functional, using xfs_repair could worsen the situation. Clone the affected disks before attempting repairs and try recovery on the clone. Remember, RAID is not a backup, so ideally, you should have another backup copy.

jagzno · Monday at 10:36

Adam Camryn said:

If multiple disks fail simultaneously, avoid repairs until the RAID array is confirmed healthy. Check the HP RAID controller logs to ensure all drives are online. If the array isn't fully functional, using xfs_repair could worsen the situation. Clone the affected disks before attempting repairs and try recovery on the clone. Remember, RAID is not a backup, so ideally, you should have another backup copy.

Click to expand...

Hi, yeah, thanks for the warning! Fortunately, I have already taken all of those precautions:

RAID Controller Logs: I used the HP Smart Storage Administrator (ssacli) to verify the physical drives. All 24 SAS drives are completely healthy, online, and the Logical Volume status is 'OK'. The hardware RAID 10 array itself is fully functional.
The xfs_repair mistake: Unfortunately, xfs_repair -L was already run before I realized the severity of the metadata corruption, which is what caused the inode for the VM disk to be orphaned/lost in the first place.
The Clone: I have already created a full 14TB ddrescue clone of the array onto my backup NFS drive. I am not running any tools on the live platters, and the original /dev/sdb1 array is mounted strictly as read-only right now to prevent any further damage.

cwt · Monday at 21:34

The xfs_metadump errors don’t necessarily mean the entire filesystem structure is beyond recovery. They indicate that parts of the XFS metadata are inconsistent or corrupted, which is exactly what you’d expect after a power loss combined with a damaged journal and a forced xfs_repair -L.

The good part is that you’ve already done several things right: you have a complete ddrescue clone, you’re working from a read-only mount, and you’ve identified what appears to be a small number of large, contiguous free extents where the former 8 TB file resided.

One thing I’m curious about is how you determined that those free extents specifically belonged to the missing vm-101-disk-0.raw. Was that conclusion based on xfs_db, a previous extent map, or another forensic tool?

If those extents can indeed be identified with a high degree of confidence, this may no longer be primarily an XFS repair problem but rather a data extraction problem. In other words, the inode may be gone while the underlying data blocks are still intact.

At this stage, I would avoid any further repair attempts on the filesystem itself. Instead, I’d focus on reconstructing the missing file from the cloned image rather than trying to make the XFS metadata consistent again. I think that’s the approach with the highest chance of recovering the 8 TB VM disk.

jagzno · 2026-07-16T02:25:32+0200

cwt said:
The xfs_metadump errors don’t necessarily mean the entire filesystem structure is beyond recovery. They indicate that parts of the XFS metadata are inconsistent or corrupted, which is exactly what you’d expect after a power loss combined with a damaged journal and a forced xfs_repair -L.

The good part is that you’ve already done several things right: you have a complete ddrescue clone, you’re working from a read-only mount, and you’ve identified what appears to be a small number of large, contiguous free extents where the former 8 TB file resided.

One thing I’m curious about is how you determined that those free extents specifically belonged to the missing vm-101-disk-0.raw. Was that conclusion based on xfs_db, a previous extent map, or another forensic tool?

If those extents can indeed be identified with a high degree of confidence, this may no longer be primarily an XFS repair problem but rather a data extraction problem. In other words, the inode may be gone while the underlying data blocks are still intact.

At this stage, I would avoid any further repair attempts on the filesystem itself. Instead, I’d focus on reconstructing the missing file from the cloned image rather than trying to make the XFS metadata consistent again. I think that’s the approach with the highest chance of recovering the 8 TB VM disk.

That's the problem I ran xfs_repair -L then i cloned the image (13tb). The LXC's root storage assigned is only 8Tb.
Any recommendation or suggestion for recovery tools?

HELP NEEDED. Proxmox Container on XFS after HP RAID Crash

jagzno

New Member

cwt

Renowned Member

jagzno

New Member

cwt

Renowned Member

jagzno

New Member

cwt

Renowned Member

jagzno

New Member

Adam Camryn

Member

jagzno

New Member

cwt

Renowned Member

jagzno

New Member

We value your privacy