Disk Full - can not delete old backups.
- Thread starter mskenderian
- Start date
Well the problem is that I deleted a bunch of chunks, actually a whole dataset. But I still don’t have enough space. It didn’t actually free any space. Even though I deleted the whole folder.
Bash:
zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 928G 899G 28.7G - - 78% 96% 1.00x ONLINE -
mirror-0 928G 899G 28.7G - - 78% 96.9% - ONLINE
nvme-eui.002538b611501529-part3 931G - - - - - - - ONLINE
ata-CT1000MX500SSD1_2311E6BBD0A2-part3 931G - - - - - - - ONLINE
Bash:
zfs list -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
rpool 0B 899G 0B 96K 0B 899G
rpool/ROOT 0B 2.52G 0B 96K 0B 2.52G
rpool/ROOT/pbs-1 0B 2.52G 0B 2.52G 0B 0B
rpool/encrypted_data 0B 897G 0B 897G 0B 0B
Bash:
df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 776M 73M 704M 10% /run
rpool/ROOT/pbs-1 2.6G 2.6G 0 100% /
tmpfs 3.8G 0 3.8G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
rpool 128K 128K 0 100% /rpool
rpool/ROOT 128K 128K 0 100% /rpool/ROOT
rpool/encrypted_data 897G 897G 0 100% /rpool/encrypted_data
tmpfs 776M 0 776M 0% /run/user/1000What dataset did you delete? Your "encrypted_data" dataset is still there with 897G of data on it (which is actually data not hold back by snapshots or missing TRIM). And again, your whole pool is 100% full which should never happen, as ZFS is a Copy-on-Write filesystem so it needs space to write to be able to delete something. That means you can totally brick a pool to a state where it can'T recover without adding more disks when not properly monitoring it or setting quotas.
Last edited:
So inside that folder I have one backup dataset per PVE, I deleted one of those. And backups from other backup datasets, that are also inside that folder.
I made a script to help other out
If you end up filling up your space run this with privileged user like root
Steps:
1. Set the Paths "SRC_DIR", "DEST_DIR" and "BACKUP_FILE"
2. Run the script and select "MOVE", you need to free up at least 100mb
3. Manually run a Garbage Collect Job now in PBS, it should work (ignore the warnings, we will restore them in step 4.)
4. Run this script and select "RESTORE", to restore your moved file as they were.
5. Run a full Verify in PBS
If you end up filling up your space run this with privileged user like root
Steps:
1. Set the Paths "SRC_DIR", "DEST_DIR" and "BACKUP_FILE"
2. Run the script and select "MOVE", you need to free up at least 100mb
3. Manually run a Garbage Collect Job now in PBS, it should work (ignore the warnings, we will restore them in step 4.)
4. Run this script and select "RESTORE", to restore your moved file as they were.
5. Run a full Verify in PBS
Bash:
#!/bin/env bash
# Set Source and destination directories
SRC_DIR="/path/to/datastore/.chunks/"
DEST_DIR="/path/to/move/to/.chunks/"
BACKUP_FILE="/any/path/logs/move_log.txt"
# ----- DO NOT EDIT BELOW ----- #
clear
function_move() {
# warn the user that the script will move files from SRC_DIR to DEST_DIR, make it eye catching:
echo -e "\e[31m----- WARNING: This script will move files -----\e[0m"
echo -e "from: \e[33m'$SRC_DIR'\e[0m"
echo -e "to: \e[32m'$DEST_DIR'\e[0m"
echo -e "\e[31m----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----\e[0m"
# Prompt the user to select the number of latest modified folders to move
echo
echo "Select the number of latest folders to move:"
echo "This is done to free up space for garbage cleaning to be able to run"
echo
options=("<Manually input>" "5" "10" "25" "50" "100")
select opt in "${options[@]}"
do
case $opt in
"<Manually input>")
read -p "Enter the number of items: " NUM_ITEMS
break
;;
"5")
NUM_ITEMS=5
break
;;
"10")
NUM_ITEMS=10
break
;;
"25")
NUM_ITEMS=25
break
;;
"50")
NUM_ITEMS=50
break
;;
"100")
NUM_ITEMS=100
break
;;
*) echo "Invalid option $REPLY";;
esac
done
# Find the latest modified folders or files in SRC_DIR
LATEST_ITEMS=$(ls -t "$SRC_DIR" | head -n "$NUM_ITEMS")
# Output the result
# echo "The latest $NUM_ITEMS modified items are:"
# echo "$LATEST_ITEMS"
# Show human-readable sizes of the selected items
echo "Sizes of the selected items:"
for ITEM in $LATEST_ITEMS;
do
du -sh "$SRC_DIR/$ITEM"
done
# Show the total combined size
echo
echo "Total combined size of the selected items:"
du -ch $(for ITEM in $LATEST_ITEMS; do echo "$SRC_DIR/$ITEM"; done) | grep total
# Ask for confirmation to proceed with the move
while true; do
read -p "Do you want to proceed with moving these items to $DEST_DIR? (y/n): " CONFIRM
case $CONFIRM in
[Yy]* )
break
;;
[Nn]* )
read -p "Do you want to retry or exit? (r/e): " RETRY
case $RETRY in
[Rr]* )
function_move
return
;;
[Ee]* )
echo "Operation cancelled."
exit 1
;;
* )
echo "Invalid option. Please enter 'r' to retry or 'e' to exit."
;;
esac
;;
* )
echo "Invalid option. Please enter 'y' to proceed or 'n' to cancel."
;;
esac
done
# Ensure the destination directory exists
if [ ! -d "$DEST_DIR" ]; then
mkdir -pv "$DEST_DIR"
fi
# Change ownership of the destination directory to 34:34
chown 34:34 "$DEST_DIR"
# Backup 'permissions', 'uid', 'gid', 'File path', 'last modification', 'last access' and 'last status change' to a file
> "$BACKUP_FILE"
for ITEM in $LATEST_ITEMS;
do
find "$SRC_DIR/$ITEM" -exec stat -c "%a %U %G %n %Y %X %Z" {} \; >> "$BACKUP_FILE"
done
# Copy the latest folders or files to DEST_DIR
for ITEM in $LATEST_ITEMS;
do
cp -rpv "$SRC_DIR/$ITEM" "$DEST_DIR"
done
# Restore permissions, timestamps, and ownership from the backup file
while IFS=' ' read -r SRC_PERM SRC_USER SRC_GROUP SRC_PATH SRC_MODTIME SRC_ACCESSTIME SRC_CHANGETIME; do
DEST_PATH="$DEST_DIR/${SRC_PATH#$SRC_DIR/}"
if [ -e "$DEST_PATH" ]; then
DEST_PERM=$(stat -c %a "$DEST_PATH")
if [ "$SRC_PERM" != "$DEST_PERM" ]; then
chmod "$SRC_PERM" "$DEST_PATH"
fi
# Set the timestamps
touch -m -d "@$SRC_MODTIME" "$DEST_PATH"
touch -a -d "@$SRC_ACCESSTIME" "$DEST_PATH"
touch -d "@$SRC_CHANGETIME" "$DEST_PATH"
# Restore ownership
chown "$SRC_USER:$SRC_GROUP" "$DEST_PATH"
fi
done < "$BACKUP_FILE"
# Delete the source items
for ITEM in $LATEST_ITEMS;
do
rm -r "$SRC_DIR/$ITEM"
done
echo
echo "Items copied, permissions, timestamps, and ownership saved, and source items deleted successfully."
echo
echo -e "\e[31m----- DO NOT RUN AGAIN UNTIL FILES ARE RESTORED BACK TO '$SRC_DIR' -----\e[0m"
echo
echo "Next Steps:"
echo
echo " 1). Run a Garbage Collect Job in Proxbox Backup Server to free up space"
echo " 2). Then Run RESTORE with this script to move the files back to '$SRC_DIR'"
echo
}
function_restore() {
# Restore the items from DEST_DIR to SRC_DIR
echo "Restoring items from $DEST_DIR to $SRC_DIR..."
# Copy the items back to SRC_DIR
while IFS=' ' read -r SRC_PERM SRC_USER SRC_GROUP SRC_PATH SRC_MODTIME SRC_ACCESSTIME SRC_CHANGETIME; do
REL_PATH="${SRC_PATH#$SRC_DIR/}"
DEST_PATH="$SRC_DIR/$REL_PATH"
DEST_DIR_PATH=$(dirname "$DEST_PATH")
# Copy the item back
cp -rpv "$DEST_DIR/$REL_PATH" "$DEST_PATH"
# Restore permissions, timestamps, and ownership
if [ -e "$DEST_PATH" ]; then
DEST_PERM=$(stat -c %a "$DEST_PATH")
if [ "$SRC_PERM" != "$DEST_PERM" ]; then
chmod "$SRC_PERM" "$DEST_PATH"
fi
# Set the timestamps
touch -m -d "@$SRC_MODTIME" "$DEST_PATH"
touch -a -d "@$SRC_ACCESSTIME" "$DEST_PATH"
touch -d "@$SRC_CHANGETIME" "$DEST_PATH"
# Restore ownership
chown "$SRC_USER:$SRC_GROUP" "$DEST_PATH"
fi
done < "$BACKUP_FILE"
# Delete the leftover files and folders in DEST_DIR
echo "Deleting leftover files and folders in $DEST_DIR..."
while IFS=' ' read -r _ _ _ SRC_PATH _ _ _; do
REL_PATH="${SRC_PATH#$SRC_DIR/}"
DEST_PATH="$DEST_DIR/$REL_PATH"
if [ -e "$DEST_PATH" ]; then
rm -r "$DEST_PATH"
fi
done < "$BACKUP_FILE"
echo
echo "Items restored, permissions, timestamps, and ownership checked, and leftover files deleted successfully."
echo
}
echo
echo -e "\e[31m----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----\e[0m"
# Start the function_move function or restore function based on user input
echo "Do you want to move items or restore items?"
echo
options=("Move" "Restore")
select opt in "${options[@]}"
do
case $opt in
"Move")
function_move
break
;;
"Restore")
function_restore
break
;;
*) echo "Invalid option $REPLY";;
esac
done