Restore Failed from proxmox backup

[fandikurnia]

HI Forum,

Thanks all. I have problem with restore 9 TB vm on our proxmox VE ..


new volume ID is 'ceph_pool_vm:vm-124-disk-0'restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@10.30.1.133:datastore1 vm/124/2024-09-13T14:07:43Z drive-sata0.img.fidx 'rbd:ceph_pool_vm/vm-124-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/ceph_pool_vm.keyring' --verbose --format raw --skip-zeroconnecting to repository 'root@pam@10.30.1.133:datastore1'open block backend for target 'rbd:ceph_pool_vm/vm-124-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/ceph_pool_vm.keyring'starting to restore snapshot 'vm/124/2024-09-13T14:07:43Z'download and verify backup indexrestore failed: reading file "/mnt/datastore/datastore1/.chunks/8945/8945905e97bd3a1b327642f236f676d0eab7421b97c7e8b9cf0bc73ae29bada4" failed: No such file or directory (os error 2)Removing image: 1% complete...Removing image: 2% complete...Removing image: 3% complete...Removing image: 4% complete...Removing image: 5% complete...Removing image: 6% complete...Removing image: 7% complete...Removing image: 8% complete...Removing image: 9% complete...Removing image: 10% complete...Removing image: 11% complete...Removing image: 12% complete...Removing image: 13% complete...Removing image: 14% complete...Removing image: 15% complete...Removing image: 16% complete...Removing image: 17% complete...Removing image: 18% complete...Removing image: 19% complete...Removing image: 20% complete...Removing image: 21% complete...Removing image: 22% complete...Removing image: 23% complete...Removing image: 24% complete...Removing image: 25% complete...Removing image: 26% complete...Removing image: 27% complete...Removing image: 28% complete...Removing image: 29% complete...Removing image: 30% complete...Removing image: 31% complete...Removing image: 32% complete...Removing image: 33% complete...Removing image: 34% complete...Removing image: 35% complete...Removing image: 36% complete...Removing image: 37% complete...Removing image: 38% complete...Removing image: 39% complete...Removing image: 40% complete...Removing image: 41% complete...Removing image: 42% complete...Removing image: 43% complete...Removing image: 44% complete...Removing image: 45% complete...Removing image: 46% complete...Removing image: 47% complete...Removing image: 48% complete...Removing image: 49% complete...Removing image: 50% complete...Removing image: 51% complete...Removing image: 52% complete...Removing image: 53% complete...Removing image: 54% complete...Removing image: 55% complete...Removing image: 56% complete...Removing image: 57% complete...Removing image: 58% complete...Removing image: 59% complete...Removing image: 60% complete...Removing image: 61% complete...Removing image: 62% complete...Removing image: 63% complete...Removing image: 64% complete...Removing image: 65% complete...Removing image: 66% complete...Removing image: 67% complete...Removing image: 68% complete...Removing image: 69% complete...Removing image: 70% complete...Removing image: 71% complete...Removing image: 72% complete...Removing image: 73% complete...Removing image: 74% complete...Removing image: 75% complete...Removing image: 76% complete...Removing image: 77% complete...Removing image: 78% complete...Removing image: 79% complete...Removing image: 80% complete...Removing image: 81% complete...Removing image: 82% complete...Removing image: 83% complete...Removing image: 84% complete...Removing image: 85% complete...Removing image: 86% complete...Removing image: 87% complete...Removing image: 88% complete...Removing image: 89% complete...Removing image: 90% complete...Removing image: 91% complete...Removing image: 92% complete...Removing image: 93% complete...Removing image: 94% complete...Removing image: 95% complete...Removing image: 96% complete...Removing image: 97% complete...Removing image: 98% complete...Removing image: 99% complete...Removing image: 100% complete...done.temporary volume 'ceph_pool_vm:vm-124-disk-0' sucessfuly removederror before or during data restore, some or all disks were not completely restored. VM 124 state is NOT cleaned up.TASK ERROR: command '/usr/bin/pbs-restore --repository root@pam@10.30.1.133:datastore1 vm/124/2024-09-13T14:07:43Z drive-sata0.img.fidx 'rbd:ceph_pool_vm/vm-124-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/ceph_pool_vm.keyring' --verbose --format raw --skip-zero' failed: exit code 255

 

[guruevi]

Here is your error:
reading file "/mnt/datastore/datastore1/.chunks/8945/8945905e97bd3a1b327642f236f676d0eab7421b97c7e8b9cf0bc73ae29bada4" failed: No such file or directory (os error 2)

Your backup is not complete or corrupted.

 

[fandikurnia]

So how to restore it? Because we need to solved the data.
The vm already removed btw.


Anyone now how to solve maybe partial data

 

[fandikurnia]

I just try still on progress..
Is the script already correct or need to fix ?
#!/bin/bash

# Variables
VM_ID=124
SNAPSHOT_DATE="2024-09-13T14:07:43Z"
CHUNK_DIR="/mnt/datastore/datastore1/.chunks"
VM_DIR="/mnt/datastore/datastore1/vm/${VM_ID}/${SNAPSHOT_DATE}"
FIDX_FILE="${VM_DIR}/drive-sata0.img.fidx"
# Output image directory and file
OUTPUT_DIR="/mnt/datastore/datastore1/backup_image"
OUTPUT_IMAGE="${OUTPUT_DIR}/partial_disk.img"
CHUNK_LIST="chunk_list.txt"

# Step 1: Check if /mnt/datastore/datastore1/backup_image exists and create it if needed
if [ ! -d "$OUTPUT_DIR" ]; then
echo "Creating backup directory at $OUTPUT_DIR..."
mkdir -p "$OUTPUT_DIR"
fi

# Step 2: Extract the list of chunks from the .fidx file
echo "Extracting chunks from .fidx file..."
proxmox-backup-debug inspect file "$FIDX_FILE" | grep -oE '[a-f0-9]{64}' > "$CHUNK_LIST"

# Step 3: Start restoring chunks to image
echo "Starting chunk restoration process..."

# Remove the old output image if it exists
if [ -f "$OUTPUT_IMAGE" ]; then
echo "Removing old image file..."
rm "$OUTPUT_IMAGE"
fi

# Step 4: Iterate through the chunk list and concatenate available chunks
while read -r chunk; do
chunk_dir=${chunk:0:4} # The first 4 characters define the chunk folder

# Check if the chunk exists
chunk_path="${CHUNK_DIR}/${chunk_dir}/${chunk}"
if [ -f "$chunk_path" ]; then
echo "Chunk $chunk exists. Adding to image..."
cat "$chunk_path" >> "$OUTPUT_IMAGE"
else
echo "Warning: Chunk $chunk is missing!"
fi
done < "$CHUNK_LIST"

echo "Chunk restoration complete. Output image: $OUTPUT_IMAGE"

# Optional: You can add a ddrescue step here to handle errors and further recover data
# ddrescue $OUTPUT_IMAGE /mnt/datastore/datastore1/backup_image/recovered_disk.img /mnt/datastore/datastore1/backup_image/recovery.log

 

[Chris]

Have you already tried to access the backup snapshot by mapping it to a loop device via a proxmox-backup-client map <snapshot> <archive-name> and access the data from there, e.g. by mounting the filesystem or creating a disk image via dd-rescue? What is the filesystem on that disk?

On a side note: Please make sure to have regular verify jobs configured on the datastore to detect such corrupt snapshots early.

 

[fandikurnia]

Hi Chris,

Thanks for your information.

Yes, I have check all.
All System respond the same errors. Yes, we just miss verify jobs on the datasotred. Now, we have enabled it.