Disk Full - can not delete old backups.
- Thread starter mskenderian
- Start date
Well the problem is that I deleted a bunch of chunks, actually a whole dataset. But I still don’t have enough space. It didn’t actually free any space. Even though I deleted the whole folder.
Bash:
zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 928G 899G 28.7G - - 78% 96% 1.00x ONLINE -
mirror-0 928G 899G 28.7G - - 78% 96.9% - ONLINE
nvme-eui.002538b611501529-part3 931G - - - - - - - ONLINE
ata-CT1000MX500SSD1_2311E6BBD0A2-part3 931G - - - - - - - ONLINE
Bash:
zfs list -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
rpool 0B 899G 0B 96K 0B 899G
rpool/ROOT 0B 2.52G 0B 96K 0B 2.52G
rpool/ROOT/pbs-1 0B 2.52G 0B 2.52G 0B 0B
rpool/encrypted_data 0B 897G 0B 897G 0B 0B
Bash:
df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 776M 73M 704M 10% /run
rpool/ROOT/pbs-1 2.6G 2.6G 0 100% /
tmpfs 3.8G 0 3.8G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
rpool 128K 128K 0 100% /rpool
rpool/ROOT 128K 128K 0 100% /rpool/ROOT
rpool/encrypted_data 897G 897G 0 100% /rpool/encrypted_data
tmpfs 776M 0 776M 0% /run/user/1000What dataset did you delete? Your "encrypted_data" dataset is still there with 897G of data on it (which is actually data not hold back by snapshots or missing TRIM). And again, your whole pool is 100% full which should never happen, as ZFS is a Copy-on-Write filesystem so it needs space to write to be able to delete something. That means you can totally brick a pool to a state where it can'T recover without adding more disks when not properly monitoring it or setting quotas.
Last edited:
So inside that folder I have one backup dataset per PVE, I deleted one of those. And backups from other backup datasets, that are also inside that folder.
I made a script to help other out
If you end up filling up your space run this with privileged user like root
Steps:
1. Set the Paths "SRC_DIR", "DEST_DIR" and "BACKUP_FILE"
2. Run the script and select "MOVE", you need to free up at least 100mb
3. Manually run a Garbage Collect Job now in PBS, it should work (ignore the warnings, we will restore them in step 4.)
4. Run this script and select "RESTORE", to restore your moved file as they were.
5. Run a full Verify in PBS
If you end up filling up your space run this with privileged user like root
Steps:
1. Set the Paths "SRC_DIR", "DEST_DIR" and "BACKUP_FILE"
2. Run the script and select "MOVE", you need to free up at least 100mb
3. Manually run a Garbage Collect Job now in PBS, it should work (ignore the warnings, we will restore them in step 4.)
4. Run this script and select "RESTORE", to restore your moved file as they were.
5. Run a full Verify in PBS
Bash:
#!/bin/env bash
# Set Source and destination directories
SRC_DIR="/path/to/datastore/.chunks/"
DEST_DIR="/path/to/move/to/.chunks/"
BACKUP_FILE="/any/path/logs/move_log.txt"
# ----- DO NOT EDIT BELOW ----- #
clear
function_move() {
# warn the user that the script will move files from SRC_DIR to DEST_DIR, make it eye catching:
echo -e "\e[31m----- WARNING: This script will move files -----\e[0m"
echo -e "from: \e[33m'$SRC_DIR'\e[0m"
echo -e "to: \e[32m'$DEST_DIR'\e[0m"
echo -e "\e[31m----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----\e[0m"
# Prompt the user to select the number of latest modified folders to move
echo
echo "Select the number of latest folders to move:"
echo "This is done to free up space for garbage cleaning to be able to run"
echo
options=("<Manually input>" "5" "10" "25" "50" "100")
select opt in "${options[@]}"
do
case $opt in
"<Manually input>")
read -p "Enter the number of items: " NUM_ITEMS
break
;;
"5")
NUM_ITEMS=5
break
;;
"10")
NUM_ITEMS=10
break
;;
"25")
NUM_ITEMS=25
break
;;
"50")
NUM_ITEMS=50
break
;;
"100")
NUM_ITEMS=100
break
;;
*) echo "Invalid option $REPLY";;
esac
done
# Find the latest modified folders or files in SRC_DIR
LATEST_ITEMS=$(ls -t "$SRC_DIR" | head -n "$NUM_ITEMS")
# Output the result
# echo "The latest $NUM_ITEMS modified items are:"
# echo "$LATEST_ITEMS"
# Show human-readable sizes of the selected items
echo "Sizes of the selected items:"
for ITEM in $LATEST_ITEMS;
do
du -sh "$SRC_DIR/$ITEM"
done
# Show the total combined size
echo
echo "Total combined size of the selected items:"
du -ch $(for ITEM in $LATEST_ITEMS; do echo "$SRC_DIR/$ITEM"; done) | grep total
# Ask for confirmation to proceed with the move
while true; do
read -p "Do you want to proceed with moving these items to $DEST_DIR? (y/n): " CONFIRM
case $CONFIRM in
[Yy]* )
break
;;
[Nn]* )
read -p "Do you want to retry or exit? (r/e): " RETRY
case $RETRY in
[Rr]* )
function_move
return
;;
[Ee]* )
echo "Operation cancelled."
exit 1
;;
* )
echo "Invalid option. Please enter 'r' to retry or 'e' to exit."
;;
esac
;;
* )
echo "Invalid option. Please enter 'y' to proceed or 'n' to cancel."
;;
esac
done
# Ensure the destination directory exists
if [ ! -d "$DEST_DIR" ]; then
mkdir -pv "$DEST_DIR"
fi
# Change ownership of the destination directory to 34:34
chown 34:34 "$DEST_DIR"
# Backup 'permissions', 'uid', 'gid', 'File path', 'last modification', 'last access' and 'last status change' to a file
> "$BACKUP_FILE"
for ITEM in $LATEST_ITEMS;
do
find "$SRC_DIR/$ITEM" -exec stat -c "%a %U %G %n %Y %X %Z" {} \; >> "$BACKUP_FILE"
done
# Copy the latest folders or files to DEST_DIR
for ITEM in $LATEST_ITEMS;
do
cp -rpv "$SRC_DIR/$ITEM" "$DEST_DIR"
done
# Restore permissions, timestamps, and ownership from the backup file
while IFS=' ' read -r SRC_PERM SRC_USER SRC_GROUP SRC_PATH SRC_MODTIME SRC_ACCESSTIME SRC_CHANGETIME; do
DEST_PATH="$DEST_DIR/${SRC_PATH#$SRC_DIR/}"
if [ -e "$DEST_PATH" ]; then
DEST_PERM=$(stat -c %a "$DEST_PATH")
if [ "$SRC_PERM" != "$DEST_PERM" ]; then
chmod "$SRC_PERM" "$DEST_PATH"
fi
# Set the timestamps
touch -m -d "@$SRC_MODTIME" "$DEST_PATH"
touch -a -d "@$SRC_ACCESSTIME" "$DEST_PATH"
touch -d "@$SRC_CHANGETIME" "$DEST_PATH"
# Restore ownership
chown "$SRC_USER:$SRC_GROUP" "$DEST_PATH"
fi
done < "$BACKUP_FILE"
# Delete the source items
for ITEM in $LATEST_ITEMS;
do
rm -r "$SRC_DIR/$ITEM"
done
echo
echo "Items copied, permissions, timestamps, and ownership saved, and source items deleted successfully."
echo
echo -e "\e[31m----- DO NOT RUN AGAIN UNTIL FILES ARE RESTORED BACK TO '$SRC_DIR' -----\e[0m"
echo
echo "Next Steps:"
echo
echo " 1). Run a Garbage Collect Job in Proxbox Backup Server to free up space"
echo " 2). Then Run RESTORE with this script to move the files back to '$SRC_DIR'"
echo
}
function_restore() {
# Restore the items from DEST_DIR to SRC_DIR
echo "Restoring items from $DEST_DIR to $SRC_DIR..."
# Copy the items back to SRC_DIR
while IFS=' ' read -r SRC_PERM SRC_USER SRC_GROUP SRC_PATH SRC_MODTIME SRC_ACCESSTIME SRC_CHANGETIME; do
REL_PATH="${SRC_PATH#$SRC_DIR/}"
DEST_PATH="$SRC_DIR/$REL_PATH"
DEST_DIR_PATH=$(dirname "$DEST_PATH")
# Copy the item back
cp -rpv "$DEST_DIR/$REL_PATH" "$DEST_PATH"
# Restore permissions, timestamps, and ownership
if [ -e "$DEST_PATH" ]; then
DEST_PERM=$(stat -c %a "$DEST_PATH")
if [ "$SRC_PERM" != "$DEST_PERM" ]; then
chmod "$SRC_PERM" "$DEST_PATH"
fi
# Set the timestamps
touch -m -d "@$SRC_MODTIME" "$DEST_PATH"
touch -a -d "@$SRC_ACCESSTIME" "$DEST_PATH"
touch -d "@$SRC_CHANGETIME" "$DEST_PATH"
# Restore ownership
chown "$SRC_USER:$SRC_GROUP" "$DEST_PATH"
fi
done < "$BACKUP_FILE"
# Delete the leftover files and folders in DEST_DIR
echo "Deleting leftover files and folders in $DEST_DIR..."
while IFS=' ' read -r _ _ _ SRC_PATH _ _ _; do
REL_PATH="${SRC_PATH#$SRC_DIR/}"
DEST_PATH="$DEST_DIR/$REL_PATH"
if [ -e "$DEST_PATH" ]; then
rm -r "$DEST_PATH"
fi
done < "$BACKUP_FILE"
echo
echo "Items restored, permissions, timestamps, and ownership checked, and leftover files deleted successfully."
echo
}
echo
echo -e "\e[31m----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----\e[0m"
# Start the function_move function or restore function based on user input
echo "Do you want to move items or restore items?"
echo
options=("Move" "Restore")
select opt in "${options[@]}"
do
case $opt in
"Move")
function_move
break
;;
"Restore")
function_restore
break
;;
*) echo "Invalid option $REPLY";;
esac
done
@Anexgohan not sure what I did wrong but the script messed up on restore, as even after freeing up 165Mb the GC job still refused to run. So before trying with more files I did the RESTORE and after failing to move back it still wiped the backed up files. Not placing any blame, just reporting my experience. Looks like I will have to start from scratch again.
Code:
----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----
Do you want to move items or restore items?
1) Move
2) Restore
#? 2
Restoring items from /root/backup/.chunks/ to /mnt/datastore/dpool-01/.chunks/...
cp: cannot create directory '/mnt/datastore/dpool-01/.chunks//661b': No space left on device
'/root/backup/.chunks//661b/661babb92c53e54d46fe945f0ea56e0f2b954773b9169dda0d061ca5f6b22ccb' -> '/mnt/datastore/dpool-01/.chunks//661b/661babb92c53e54d46fe945f0ea56e0f2b954773b9169dda0d061ca5f6b22ccb'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661babb92c53e54d46fe945f0ea56e0f2b954773b9169dda0d061ca5f6b22ccb': No such file or directory
'/root/backup/.chunks//661b/661bca5acf9a2a13f43db972869b014a96c6084851a75c919b90c4f1211bb3a4' -> '/mnt/datastore/dpool-01/.chunks//661b/661bca5acf9a2a13f43db972869b014a96c6084851a75c919b90c4f1211bb3a4'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661bca5acf9a2a13f43db972869b014a96c6084851a75c919b90c4f1211bb3a4': No such file or directory
'/root/backup/.chunks//661b/661b6e1d58bb94651a4174734a3736b62d7e5c0e7cfe17bf83765e048a602970' -> '/mnt/datastore/dpool-01/.chunks//661b/661b6e1d58bb94651a4174734a3736b62d7e5c0e7cfe17bf83765e048a602970'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b6e1d58bb94651a4174734a3736b62d7e5c0e7cfe17bf83765e048a602970': No such file or directory
'/root/backup/.chunks//661b/661b1beb87d0d243c3ba8ae9a2bab2b424eb05c4a2becfd5c779a228d3bdae89' -> '/mnt/datastore/dpool-01/.chunks//661b/661b1beb87d0d243c3ba8ae9a2bab2b424eb05c4a2becfd5c779a228d3bdae89'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b1beb87d0d243c3ba8ae9a2bab2b424eb05c4a2becfd5c779a228d3bdae89': No such file or directory
'/root/backup/.chunks//661b/661b4a18e586a662bc4eedfd54b4718d598099f731b03c90ae4afdac5e723f52' -> '/mnt/datastore/dpool-01/.chunks//661b/661b4a18e586a662bc4eedfd54b4718d598099f731b03c90ae4afdac5e723f52'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b4a18e586a662bc4eedfd54b4718d598099f731b03c90ae4afdac5e723f52': No such file or directory
'/root/backup/.chunks//661b/661bda1a3b62c8ae7eb7c3abff57f453a4db5b897c1f72a73a05cd1543bfd487' -> '/mnt/datastore/dpool-01/.chunks//661b/661bda1a3b62c8ae7eb7c3abff57f453a4db5b897c1f72a73a05cd1543bfd487'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661bda1a3b62c8ae7eb7c3abff57f453a4db5b897c1f72a73a05cd1543bfd487': No such file or directory
'/root/backup/.chunks//661b/661b4fa8d63d7f42032cf426c9a8d04b16c7d7bf833ccd0b9eba7a3395df7904' -> '/mnt/datastore/dpool-01/.chunks//661b/661b4fa8d63d7f42032cf426c9a8d04b16c7d7bf833ccd0b9eba7a3395df7904'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b4fa8d63d7f42032cf426c9a8d04b16c7d7bf833ccd0b9eba7a3395df7904': No such file or directory
'/root/backup/.chunks//661b/661b3c41e0950686640425bd7c89675f3b7d475f004b439661eefe9314e9702c' -> '/mnt/datastore/dpool-01/.chunks//661b/661b3c41e0950686640425bd7c89675f3b7d475f004b439661eefe9314e9702c'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b3c41e0950686640425bd7c89675f3b7d475f004b439661eefe9314e9702c': No such file or directory
'/root/backup/.chunks//661b/661b77f5af019119da6c87d7df35181d123a55b935becfab05d3f79ff397139a' -> '/mnt/datastore/dpool-01/.chunks//661b/661b77f5af019119da6c87d7df35181d123a55b935becfab05d3f79ff397139a'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b77f5af019119da6c87d7df35181d123a55b935becfab05d3f79ff397139a': No such file or directory
cp: cannot create directory '/mnt/datastore/dpool-01/.chunks//5c1e': No space left on device
'/root/backup/.chunks//5c1e/5c1e2d7cb95ab01fd517782b2e331afa061bb445feadb9301957603e6f5c7bc1' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e2d7cb95ab01fd517782b2e331afa061bb445feadb9301957603e6f5c7bc1'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e2d7cb95ab01fd517782b2e331afa061bb445feadb9301957603e6f5c7bc1': No such file or directory
'/root/backup/.chunks//5c1e/5c1e009e7525e563633e92fcc66a23149158331491ccdecccf4ae20cc2a99f03' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e009e7525e563633e92fcc66a23149158331491ccdecccf4ae20cc2a99f03'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e009e7525e563633e92fcc66a23149158331491ccdecccf4ae20cc2a99f03': No such file or directory
'/root/backup/.chunks//5c1e/5c1ed9621ba1bb974d91719cd251b70d8953afaca81633d5ddaa87ec95618366' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1ed9621ba1bb974d91719cd251b70d8953afaca81633d5ddaa87ec95618366'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1ed9621ba1bb974d91719cd251b70d8953afaca81633d5ddaa87ec95618366': No such file or directory
'/root/backup/.chunks//5c1e/5c1e954529f235c9570c856af4892e436a734400f515caa91cbea1c70194d1b4' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e954529f235c9570c856af4892e436a734400f515caa91cbea1c70194d1b4'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e954529f235c9570c856af4892e436a734400f515caa91cbea1c70194d1b4': No such file or directory
'/root/backup/.chunks//5c1e/5c1e24823184fbd0272ee9c257a68f363864fcaa914fd729ff98fcff84c27dc9' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e24823184fbd0272ee9c257a68f363864fcaa914fd729ff98fcff84c27dc9'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e24823184fbd0272ee9c257a68f363864fcaa914fd729ff98fcff84c27dc9': No such file or directory
'/root/backup/.chunks//5c1e/5c1e6ba52e4447b3bf0cfd7625182999a50385da710a494837be83397e49e4a6' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e6ba52e4447b3bf0cfd7625182999a50385da710a494837be83397e49e4a6'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e6ba52e4447b3bf0cfd7625182999a50385da710a494837be83397e49e4a6': No such file or directory
'/root/backup/.chunks//5c1e/5c1ef63a1b0c3744299122210d6d1283d6761041e402e63c80d20224f91f1150' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1ef63a1b0c3744299122210d6d1283d6761041e402e63c80d20224f91f1150'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1ef63a1b0c3744299122210d6d1283d6761041e402e63c80d20224f91f1150': No such file or directory
Deleting leftover files and folders in /root/backup/.chunks/...
Items restored, permissions, timestamps, and ownership checked, and leftover files deleted successfully.
root@lenny:~#I updated my script, from user feedback and added a few failsafes and guardrails:
the script should now give you a more guided experience:
Instructions are same as before
Steps:
Copy the below script as a "anex_pbs_script.sh"
chmod +x ./anex_pbs_script.sh
the script should now give you a more guided experience:
Instructions are same as before
Steps:
- Set the Paths "SRC_DIR", "DEST_DIR" and "BACKUP_FILE"
- follow the "Order of operations"
- Prune Job -- (in PBS GUI)
- Move
- Garbage Collect -- (in PBS GUI)
- Restore
- Garbage Collect -- (in PBS GUI)
chmod +x ./anex_pbs_script.sh
Bash:
#!/bin/env bash
# Set Source and destination directories
# Create the destination directory before running the script, where your PBS files will be moved to clear some space.
# Variables (EDIT BELOW):
SRC_DIR="/path/to/<datastore-name>/.chunks"
DEST_DIR="/path/to/destination"
# ---------------------------------------------------------------------
# DO NOT EDIT BELOW:
# ---------------------------------------------------------------------
# Build paths:
CHUNKS_DIR="$DEST_DIR/backup/.chunks"
BACKUP_FILE="$DEST_DIR/logs/move_log.txt"
LOCK_FILE="$DEST_DIR/.state_lock"
# Pre-flight Configuration Check:
check_config() {
# Ensure log directory exists
mkdir -p "$(dirname "$BACKUP_FILE")"
# Check if CHUNKS_DIR exists, create if not
if [ ! -d "$CHUNKS_DIR" ]; then
echo "Destination directory '$CHUNKS_DIR' does not exist. Attempting to create..."
if ! mkdir -pv "$CHUNKS_DIR"; then
echo -e "\e[31mERROR: Failed to create destination directory '$CHUNKS_DIR'. Aborting.\e[0m"
exit 1
fi
fi
# Set Permissions on CHUNKS_DIR (Must be backup:backup 750)
# This ensures consistency with PBS datastore structure
chown backup:backup "$CHUNKS_DIR"
chmod 750 "$CHUNKS_DIR"
# Check if CHUNKS_DIR is writable
if [ ! -w "$CHUNKS_DIR" ]; then
echo -e "\e[31mCRITICAL ERROR: Destination directory '$CHUNKS_DIR' is NOT writable.\e[0m"
echo "Please fix permissions or check the path."
exit 1
fi
}
clear
function_move() {
# warn the user that the script will move files from SRC_DIR to CHUNKS_DIR, make it eye catching:
echo -e "\e[31m----- WARNING: This script will move files -----\e[0m"
echo -e "from: \e[33m'$SRC_DIR'\e[0m"
echo -e "to: \e[32m'$CHUNKS_DIR'\e[0m"
echo -e "\e[31m----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----\e[0m"
# Prompt the user to select the number of latest modified folders to move
echo
echo "Select the number of latest folders to move:"
echo "This is done to free up space for garbage cleaning to be able to run"
echo
options=("<Manually input>" "5" "10" "25" "50" "100")
select opt in "${options[@]}"
do
case $opt in
"<Manually input>")
read -p "Enter the number of items: " NUM_ITEMS
break
;;
"5")
NUM_ITEMS=5
break
;;
"10")
NUM_ITEMS=10
break
;;
"25")
NUM_ITEMS=25
break
;;
"50")
NUM_ITEMS=50
break
;;
"100")
NUM_ITEMS=100
break
;;
*) echo "Invalid option $REPLY";;
esac
done
# Find the latest modified folders or files in SRC_DIR
LATEST_ITEMS=$(ls -t "$SRC_DIR" | head -n "$NUM_ITEMS")
# Output the result
# echo "The latest $NUM_ITEMS modified items are:"
# echo "$LATEST_ITEMS"
# Show human-readable sizes of the selected items
echo "Sizes of the selected items:"
for ITEM in $LATEST_ITEMS;
do
du -sh "$SRC_DIR/$ITEM"
done
# Show the total combined size
echo
echo "Total combined size of the selected items:"
du -ch $(for ITEM in $LATEST_ITEMS; do echo "$SRC_DIR/$ITEM"; done) | grep total
# Ask for confirmation to proceed with the move
while true; do
read -p "Do you want to proceed with moving these items to $CHUNKS_DIR? (y/n): " CONFIRM
case $CONFIRM in
[Yy]* )
break
;;
[Nn]* )
read -p "Do you want to retry or exit? (r/e): " RETRY
case $RETRY in
[Rr]* )
function_move
return
;;
[Ee]* )
echo "Operation cancelled."
exit 1
;;
* )
echo "Invalid option. Please enter 'r' to retry or 'e' to exit."
;;
esac
;;
* )
echo "Invalid option. Please enter 'y' to proceed or 'n' to cancel."
;;
esac
done
# Ensure the destination directory exists
if [ ! -d "$CHUNKS_DIR" ]; then
mkdir -pv "$CHUNKS_DIR"
fi
# Change ownership of the destination directory to 34:34
chown 34:34 "$CHUNKS_DIR"
# Backup 'permissions', 'uid', 'gid', 'File path', 'last modification', 'last access' and 'last status change' to a file
> "$BACKUP_FILE"
for ITEM in $LATEST_ITEMS;
do
find "$SRC_DIR/$ITEM" -exec stat -c "%a %U %G %n %Y %X %Z" {} \; >> "$BACKUP_FILE"
done
# Copy the latest folders or files to CHUNKS_DIR
# Copy and Verify before Deleting
for ITEM in $LATEST_ITEMS;
do
echo "Moving '$ITEM'..."
# 1. Copy
cp -rpv "$SRC_DIR/$ITEM" "$CHUNKS_DIR"
CP_EXIT_CODE=$?
# 2. Verify
if [ $CP_EXIT_CODE -eq 0 ] && [ -e "$CHUNKS_DIR/$ITEM" ]; then
# Verification Successful
# Backup stats again just to be safe/redundant or relies on earlier bulk backup?
# The bulk backup earlier is fine.
# 3. Delete Source
rm -r "$SRC_DIR/$ITEM"
else
# Verification Failed
echo -e "\e[31mCRITICAL ERROR: Failed to move '$ITEM'. Copy failed or Destination file missing.\e[0m"
echo "Halting operation completely to prevent data loss."
echo "Please check '$CHUNKS_DIR' and manual intervention is required."
return 1 # Exit function, do not continue loop
fi
done
echo
echo "Items moved successfully."
echo "State Locked. You must run Restore to clear the lock."
# Create Lock File
touch "$LOCK_FILE"
echo
echo -e "\e[31m----- DO NOT RUN AGAIN UNTIL FILES ARE RESTORED BACK TO '$SRC_DIR' -----\e[0m"
echo
echo "Next Steps:"
echo
echo -e " 1). Run a \e[32m\"GARBAGE COLLECT\"\e[0m Job in Proxbox Backup Server to free up space"
echo " 2). Then Run RESTORE with this script to move the files back to '$SRC_DIR'"
echo " 3). Or you can Exit now and run this script later to Restore."
echo -e "\e[31mDO NOT RUN \"PRUNE JOB\"\e[0m as it will mark the moved files for deletion and restore wont work then, corrupting the PBS backups"
echo
read -p "Press Enter to return to menu..."
echo
echo "Items copied, permissions, timestamps, and ownership saved, and source items deleted successfully."
echo
echo "--------------------------------------------------------------------------------------------------"
echo
echo -e "\e[31m----- DO NOT RUN AGAIN UNTIL FILES ARE RESTORED BACK TO '$SRC_DIR' -----\e[0m"
echo
echo "Next Steps:"
echo
echo " 1). Run a Garbage Collect Job in Proxbox Backup Server to free up space"
echo " 2). Then Run RESTORE with this script to move the files back to '$SRC_DIR'"
echo
}
function_restore() {
# Restore the items from CHUNKS_DIR to SRC_DIR
echo "Restoring items from $CHUNKS_DIR to $SRC_DIR..."
# Copy the items back to SRC_DIR
while IFS=' ' read -r SRC_PERM SRC_USER SRC_GROUP SRC_PATH SRC_MODTIME SRC_ACCESSTIME SRC_CHANGETIME; do
REL_PATH="${SRC_PATH#$SRC_DIR/}"
DEST_PATH="$SRC_DIR/$REL_PATH"
DEST_DIR_PATH=$(dirname "$DEST_PATH")
# Copy the item back
cp -rpv "$CHUNKS_DIR/$REL_PATH" "$DEST_PATH"
# Restore permissions, timestamps, and ownership
if [ -e "$DEST_PATH" ]; then
DEST_PERM=$(stat -c %a "$DEST_PATH")
if [ "$SRC_PERM" != "$DEST_PERM" ]; then
chmod "$SRC_PERM" "$DEST_PATH"
fi
# Set the timestamps
touch -m -d "@$SRC_MODTIME" "$DEST_PATH"
touch -a -d "@$SRC_ACCESSTIME" "$DEST_PATH"
touch -d "@$SRC_CHANGETIME" "$DEST_PATH"
# Restore ownership
chown "$SRC_USER:$SRC_GROUP" "$DEST_PATH"
fi
done < "$BACKUP_FILE"
# Delete the leftover files and folders in CHUNKS_DIR
# Archival Cleanup (Instead of Deleting)
TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S)
# Ensure archive is a sibling of the chunks folder, not inside it
DEST_ARCHIVE="${CHUNKS_DIR%/}_archive_$TIMESTAMP"
echo "Archiving backup files to '$DEST_ARCHIVE'..."
mkdir -p "$DEST_ARCHIVE"
# Set Permissions on ARCHIVE (Must be backup:backup 750)
chown backup:backup "$DEST_ARCHIVE"
chmod 750 "$DEST_ARCHIVE"
# Archive the log file with matching timestamp
ARCHIVED_LOG_FILE="${BACKUP_FILE%.*}_$TIMESTAMP.txt"
cp "$BACKUP_FILE" "$ARCHIVED_LOG_FILE"
echo "Archived metadata log to '$ARCHIVED_LOG_FILE'"
# Move leftover files to archive
# We use finding from BACKUP_FILE or just move everything remaining in CHUNKS_DIR?
# Strategy: Move everything in CHUNKS_DIR to be safe, as it should only contain what we put there + maybe empty dirs.
# But strictly following the logic, we should only move what we put there.
# Let's move the specific items we restored to the archive folder.
while IFS=' ' read -r _ _ _ SRC_PATH _ _ _; do
REL_PATH="${SRC_PATH#$SRC_DIR/}"
DEST_PATH="$CHUNKS_DIR/$REL_PATH"
if [ -e "$DEST_PATH" ]; then
mv "$DEST_PATH" "$DEST_ARCHIVE/"
fi
done < "$BACKUP_FILE"
# Remove Lock File
if [ -f "$LOCK_FILE" ]; then
rm "$LOCK_FILE"
echo "State Lock removed."
fi
echo
echo "----------------------------------------------------------------"
echo "RESTORE COMPLETED SUCCESSFULLY"
echo "----------------------------------------------------------------"
echo "1. Items restored to Source."
echo "2. Metadata (permissions/ownership) re-applied."
echo "3. Restored files also Archived to (in case of a emergency), you can delete this after confirming proper PBS functionality restored:"
echo -e " -> \e[33m$DEST_ARCHIVE\e[0m"
echo "4. Metadata log preserved at:"
echo -e " -> \e[33m$ARCHIVED_LOG_FILE\e[0m"
echo "----------------------------------------------------------------"
echo -e "\e[32mDONE\e[0m"
echo -e "run another \e[32mGARBAGE COLLECT\e[0m job to fix previous \"GARBAGE COLLECT\" warnings"
echo restarting script for new session...
echo
}
echo
echo -e "\e[31m----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----\e[0m"
# Start the function_move function or restore function based on user input
# Start the function_move function or restore function based on user input
# Main Loop
check_config
while true; do
echo
if [ -f "$LOCK_FILE" ]; then
# Locked State Message
echo -e "\e[32mMove Completed, your files are safe in :\e[0m"
echo -e "\e[33m$CHUNKS_DIR\e[0m"
echo "1. Check the PBS Datastore should now have free space,"
echo -e "2. Next Step: Run a \e[32m\"GARBAGE COLLECT\"\e[0m job only"
echo -e "\e[31mDO NOT RUN \"PRUNE JOB\"\e[0m as it will mark the moved files for deletion and restore wont work then, corrupting the PBS backups"
else
# Unlocked State Message
echo -e "\e[33mMake sure you have atleast run a full 'PRUNE JOB' in pbs before proceding.\e[0m"
echo -e "\e[33mthe prune job will only mark files for deletion nothing is deleted\e[0m"
echo
echo -e "Order of operations:"
echo -e "1). \e[33mPrune Job\e[0m -- (in PBS GUI)"
echo -e "2). Move"
echo -e "3). \e[33mGarbage Collect\e[0m -- (in PBS GUI)"
echo -e "4). Restore"
echo -e "5). \e[33mGarbage Collect\e[0m -- (in PBS GUI)"
fi
echo
echo -e "\e[31m----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----\e[0m"
echo
echo -e "\e[32m SOURCE Directory: $SRC_DIR\e[0m"
echo -e "\e[33m DESTINATION Directory: $CHUNKS_DIR\e[0m"
echo
echo "Do you want to move items or restore items?"
echo
# Dynamic Menu Options
if [ -f "$LOCK_FILE" ]; then
echo "1) Move -- (Locked until Restore is done)"
else
echo "1) Move"
fi
echo "2) Restore"
echo "3) Exit"
read -p "#? " choice
case $choice in
1)
if [ -f "$LOCK_FILE" ]; then
echo
echo -e "\e[31m----- WARNING: LOCKED -----\e[0m"
echo -e "\e[31mA 'Move' operation was already completed. Doing it again implies you want to move MORE files.\e[0m"
echo -e "\e[31mWARNING: This will overwrite the backup log. IF YOU PROCEED WITHOUT RESTORING FIRST, PREVIOUS METADATA LOGS WILL BE LOST.\e[0m"
echo
read -p "Type 'YES' to ignore this warning and proceed (ANYTHING ELSE TO CANCEL): " FORCE_CONFIRM
if [ "$FORCE_CONFIRM" != "YES" ]; then
echo "Cancelled."
continue
fi
fi
function_move
;;
2)
function_restore
;;
3)
echo "Exiting."
exit 0
;;
*)
echo "Invalid option."
;;
esac
done