Disk Full - can not delete old backups.

You can also set a reservation on your rpool/ROOT/pbs-1
zfs set refreservation=8G rpool/ROOT/pbs-1
 
Well the problem is that I deleted a bunch of chunks, actually a whole dataset. But I still don’t have enough space. It didn’t actually free any space. Even though I deleted the whole folder.
 
Bash: 
zpool list -v
NAME                                         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool                                        928G   899G  28.7G        -         -    78%    96%  1.00x    ONLINE  -
  mirror-0                                   928G   899G  28.7G        -         -    78%  96.9%      -    ONLINE
    nvme-eui.002538b611501529-part3          931G      -      -        -         -      -      -      -    ONLINE
    ata-CT1000MX500SSD1_2311E6BBD0A2-part3   931G      -      -        -         -      -      -      -    ONLINE


Bash: 
zfs list -o space
NAME                  AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool                    0B   899G        0B     96K             0B       899G
rpool/ROOT               0B  2.52G        0B     96K             0B      2.52G
rpool/ROOT/pbs-1         0B  2.52G        0B   2.52G             0B         0B
rpool/encrypted_data     0B   897G        0B    897G             0B         0B

Bash: 
df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                  3.8G     0  3.8G   0% /dev
tmpfs                 776M   73M  704M  10% /run
rpool/ROOT/pbs-1      2.6G  2.6G     0 100% /
tmpfs                 3.8G     0  3.8G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
rpool                 128K  128K     0 100% /rpool
rpool/ROOT            128K  128K     0 100% /rpool/ROOT
rpool/encrypted_data  897G  897G     0 100% /rpool/encrypted_data
tmpfs                 776M     0  776M   0% /run/user/1000
 
mskenderian said:
Well the problem is that I deleted a bunch of chunks, actually a whole dataset. But I still don’t have enough space. It didn’t actually free any space. Even though I deleted the whole folder.
Click to expand...
What dataset did you delete? Your "encrypted_data" dataset is still there with 897G of data on it (which is actually data not hold back by snapshots or missing TRIM). And again, your whole pool is 100% full which should never happen, as ZFS is a Copy-on-Write filesystem so it needs space to write to be able to delete something. That means you can totally brick a pool to a state where it can'T recover without adding more disks when not properly monitoring it or setting quotas.
 
Last edited:
So inside that folder I have one backup dataset per PVE, I deleted one of those. And backups from other backup datasets, that are also inside that folder.
 
Did you disable all backups jobs or set the datastores to maintainance mode to prevent that new data is written? Maybe the freed up space got already filled up again.
 
I made a script to help other out
If you end up filling up your space run this with privileged user like root

Steps:
1. Set the Paths "SRC_DIR", "DEST_DIR" and "BACKUP_FILE"
2. Run the script and select "MOVE", you need to free up at least 100mb
3. Manually run a Garbage Collect Job now in PBS, it should work (ignore the warnings, we will restore them in step 4.)
4. Run this script and select "RESTORE", to restore your moved file as they were.
5. Run a full Verify in PBS

Bash: 
#!/bin/env bash

# Set Source and destination directories

SRC_DIR="/path/to/datastore/.chunks/"
DEST_DIR="/path/to/move/to/.chunks/"
BACKUP_FILE="/any/path/logs/move_log.txt"

# ----- DO NOT EDIT BELOW ----- #
clear
function_move() {

    # warn the user that the script will move files from SRC_DIR to DEST_DIR, make it eye catching:
    echo -e "\e[31m----- WARNING: This script will move files -----\e[0m"
    echo -e "from: \e[33m'$SRC_DIR'\e[0m"
    echo -e "to:   \e[32m'$DEST_DIR'\e[0m"
    echo -e "\e[31m----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----\e[0m"

    # Prompt the user to select the number of latest modified folders to move
    echo
    echo "Select the number of latest folders to move:"
    echo "This is done to free up space for garbage cleaning to be able to run"
    echo
    options=("<Manually input>" "5" "10" "25" "50" "100")
    select opt in "${options[@]}"
    do
        case $opt in
            "<Manually input>")
                read -p "Enter the number of items: " NUM_ITEMS
                break
                ;;
            "5")
                NUM_ITEMS=5
                break
                ;;
            "10")
                NUM_ITEMS=10
                break
                ;;
            "25")
                NUM_ITEMS=25
                break
                ;;
            "50")
                NUM_ITEMS=50
                break
                ;;
            "100")
                NUM_ITEMS=100
                break
                ;;
            *) echo "Invalid option $REPLY";;
        esac
    done

    # Find the latest modified folders or files in SRC_DIR
    LATEST_ITEMS=$(ls -t "$SRC_DIR" | head -n "$NUM_ITEMS")

    # Output the result
    # echo "The latest $NUM_ITEMS modified items are:"
    # echo "$LATEST_ITEMS"

    # Show human-readable sizes of the selected items
    echo "Sizes of the selected items:"
    for ITEM in $LATEST_ITEMS;
    do
      du -sh "$SRC_DIR/$ITEM"
    done

    # Show the total combined size
    echo
    echo "Total combined size of the selected items:"
    du -ch $(for ITEM in $LATEST_ITEMS; do echo "$SRC_DIR/$ITEM"; done) | grep total

    # Ask for confirmation to proceed with the move
    while true; do
        read -p "Do you want to proceed with moving these items to $DEST_DIR? (y/n): " CONFIRM
        case $CONFIRM in
            [Yy]* )
                break
                ;;
            [Nn]* )
                read -p "Do you want to retry or exit? (r/e): " RETRY
                case $RETRY in
                    [Rr]* )
                        function_move
                        return
                        ;;
                    [Ee]* )
                        echo "Operation cancelled."
                        exit 1
                        ;;
                    * )
                        echo "Invalid option. Please enter 'r' to retry or 'e' to exit."
                        ;;
                esac
                ;;
            * )
                echo "Invalid option. Please enter 'y' to proceed or 'n' to cancel."
                ;;
        esac
    done

    # Ensure the destination directory exists
    if [ ! -d "$DEST_DIR" ]; then
        mkdir -pv "$DEST_DIR"
    fi
    # Change ownership of the destination directory to 34:34
    chown 34:34 "$DEST_DIR"

    # Backup 'permissions', 'uid', 'gid', 'File path', 'last modification', 'last access' and 'last status change' to a file
    > "$BACKUP_FILE"
    for ITEM in $LATEST_ITEMS;
    do
    find "$SRC_DIR/$ITEM" -exec stat -c "%a %U %G %n %Y %X %Z" {} \; >> "$BACKUP_FILE"
    done

    # Copy the latest folders or files to DEST_DIR
    for ITEM in $LATEST_ITEMS;
    do
    cp -rpv "$SRC_DIR/$ITEM" "$DEST_DIR"
    done

    # Restore permissions, timestamps, and ownership from the backup file
    while IFS=' ' read -r SRC_PERM SRC_USER SRC_GROUP SRC_PATH SRC_MODTIME SRC_ACCESSTIME SRC_CHANGETIME; do
    DEST_PATH="$DEST_DIR/${SRC_PATH#$SRC_DIR/}"
    if [ -e "$DEST_PATH" ]; then
        DEST_PERM=$(stat -c %a "$DEST_PATH")
        if [ "$SRC_PERM" != "$DEST_PERM" ]; then
        chmod "$SRC_PERM" "$DEST_PATH"
        fi
        # Set the timestamps
        touch -m -d "@$SRC_MODTIME" "$DEST_PATH"
        touch -a -d "@$SRC_ACCESSTIME" "$DEST_PATH"
        touch -d "@$SRC_CHANGETIME" "$DEST_PATH"
        # Restore ownership
        chown "$SRC_USER:$SRC_GROUP" "$DEST_PATH"
    fi
    done < "$BACKUP_FILE"

    # Delete the source items
    for ITEM in $LATEST_ITEMS;
    do
    rm -r "$SRC_DIR/$ITEM"
    done

    echo
    echo "Items copied, permissions, timestamps, and ownership saved, and source items deleted successfully."
    echo
    echo -e "\e[31m----- DO NOT RUN AGAIN UNTIL FILES ARE RESTORED BACK TO '$SRC_DIR' -----\e[0m"
    echo
    echo "Next Steps:"
    echo
    echo " 1). Run a Garbage Collect Job in Proxbox Backup Server to free up space"
    echo " 2). Then Run RESTORE with this script to move the files back to '$SRC_DIR'"
    echo
}

function_restore() {
    # Restore the items from DEST_DIR to SRC_DIR
    echo "Restoring items from $DEST_DIR to $SRC_DIR..."

    # Copy the items back to SRC_DIR
    while IFS=' ' read -r SRC_PERM SRC_USER SRC_GROUP SRC_PATH SRC_MODTIME SRC_ACCESSTIME SRC_CHANGETIME; do
        REL_PATH="${SRC_PATH#$SRC_DIR/}"
        DEST_PATH="$SRC_DIR/$REL_PATH"
        DEST_DIR_PATH=$(dirname "$DEST_PATH")

        # Copy the item back
        cp -rpv "$DEST_DIR/$REL_PATH" "$DEST_PATH"

        # Restore permissions, timestamps, and ownership
        if [ -e "$DEST_PATH" ]; then
            DEST_PERM=$(stat -c %a "$DEST_PATH")
            if [ "$SRC_PERM" != "$DEST_PERM" ]; then
                chmod "$SRC_PERM" "$DEST_PATH"
            fi
            # Set the timestamps
            touch -m -d "@$SRC_MODTIME" "$DEST_PATH"
            touch -a -d "@$SRC_ACCESSTIME" "$DEST_PATH"
            touch -d "@$SRC_CHANGETIME" "$DEST_PATH"
            # Restore ownership
            chown "$SRC_USER:$SRC_GROUP" "$DEST_PATH"
        fi
    done < "$BACKUP_FILE"

    # Delete the leftover files and folders in DEST_DIR
    echo "Deleting leftover files and folders in $DEST_DIR..."
    while IFS=' ' read -r _ _ _ SRC_PATH _ _ _; do
        REL_PATH="${SRC_PATH#$SRC_DIR/}"
        DEST_PATH="$DEST_DIR/$REL_PATH"
        if [ -e "$DEST_PATH" ]; then
            rm -r "$DEST_PATH"
        fi
    done < "$BACKUP_FILE"

    echo
    echo "Items restored, permissions, timestamps, and ownership checked, and leftover files deleted successfully."
    echo
}

echo
echo -e "\e[31m----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----\e[0m"
# Start the function_move function or restore function based on user input
echo "Do you want to move items or restore items?"
echo
options=("Move" "Restore")
select opt in "${options[@]}"
do
    case $opt in
        "Move")
            function_move
            break
            ;;
        "Restore")
            function_restore
            break
            ;;
        *) echo "Invalid option $REPLY";;
    esac
done
 
@Anexgohan not sure what I did wrong but the script messed up on restore, as even after freeing up 165Mb the GC job still refused to run. So before trying with more files I did the RESTORE and after failing to move back it still wiped the backed up files. Not placing any blame, just reporting my experience. Looks like I will have to start from scratch again.

Code: 
----- WARNING: Do Not Run MOVE second time without Running RESTORE first -----
Do you want to move items or restore items?

1) Move
2) Restore
#? 2
Restoring items from /root/backup/.chunks/ to /mnt/datastore/dpool-01/.chunks/...
cp: cannot create directory '/mnt/datastore/dpool-01/.chunks//661b': No space left on device
'/root/backup/.chunks//661b/661babb92c53e54d46fe945f0ea56e0f2b954773b9169dda0d061ca5f6b22ccb' -> '/mnt/datastore/dpool-01/.chunks//661b/661babb92c53e54d46fe945f0ea56e0f2b954773b9169dda0d061ca5f6b22ccb'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661babb92c53e54d46fe945f0ea56e0f2b954773b9169dda0d061ca5f6b22ccb': No such file or directory
'/root/backup/.chunks//661b/661bca5acf9a2a13f43db972869b014a96c6084851a75c919b90c4f1211bb3a4' -> '/mnt/datastore/dpool-01/.chunks//661b/661bca5acf9a2a13f43db972869b014a96c6084851a75c919b90c4f1211bb3a4'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661bca5acf9a2a13f43db972869b014a96c6084851a75c919b90c4f1211bb3a4': No such file or directory
'/root/backup/.chunks//661b/661b6e1d58bb94651a4174734a3736b62d7e5c0e7cfe17bf83765e048a602970' -> '/mnt/datastore/dpool-01/.chunks//661b/661b6e1d58bb94651a4174734a3736b62d7e5c0e7cfe17bf83765e048a602970'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b6e1d58bb94651a4174734a3736b62d7e5c0e7cfe17bf83765e048a602970': No such file or directory
'/root/backup/.chunks//661b/661b1beb87d0d243c3ba8ae9a2bab2b424eb05c4a2becfd5c779a228d3bdae89' -> '/mnt/datastore/dpool-01/.chunks//661b/661b1beb87d0d243c3ba8ae9a2bab2b424eb05c4a2becfd5c779a228d3bdae89'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b1beb87d0d243c3ba8ae9a2bab2b424eb05c4a2becfd5c779a228d3bdae89': No such file or directory
'/root/backup/.chunks//661b/661b4a18e586a662bc4eedfd54b4718d598099f731b03c90ae4afdac5e723f52' -> '/mnt/datastore/dpool-01/.chunks//661b/661b4a18e586a662bc4eedfd54b4718d598099f731b03c90ae4afdac5e723f52'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b4a18e586a662bc4eedfd54b4718d598099f731b03c90ae4afdac5e723f52': No such file or directory
'/root/backup/.chunks//661b/661bda1a3b62c8ae7eb7c3abff57f453a4db5b897c1f72a73a05cd1543bfd487' -> '/mnt/datastore/dpool-01/.chunks//661b/661bda1a3b62c8ae7eb7c3abff57f453a4db5b897c1f72a73a05cd1543bfd487'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661bda1a3b62c8ae7eb7c3abff57f453a4db5b897c1f72a73a05cd1543bfd487': No such file or directory
'/root/backup/.chunks//661b/661b4fa8d63d7f42032cf426c9a8d04b16c7d7bf833ccd0b9eba7a3395df7904' -> '/mnt/datastore/dpool-01/.chunks//661b/661b4fa8d63d7f42032cf426c9a8d04b16c7d7bf833ccd0b9eba7a3395df7904'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b4fa8d63d7f42032cf426c9a8d04b16c7d7bf833ccd0b9eba7a3395df7904': No such file or directory
'/root/backup/.chunks//661b/661b3c41e0950686640425bd7c89675f3b7d475f004b439661eefe9314e9702c' -> '/mnt/datastore/dpool-01/.chunks//661b/661b3c41e0950686640425bd7c89675f3b7d475f004b439661eefe9314e9702c'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b3c41e0950686640425bd7c89675f3b7d475f004b439661eefe9314e9702c': No such file or directory
'/root/backup/.chunks//661b/661b77f5af019119da6c87d7df35181d123a55b935becfab05d3f79ff397139a' -> '/mnt/datastore/dpool-01/.chunks//661b/661b77f5af019119da6c87d7df35181d123a55b935becfab05d3f79ff397139a'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//661b/661b77f5af019119da6c87d7df35181d123a55b935becfab05d3f79ff397139a': No such file or directory
cp: cannot create directory '/mnt/datastore/dpool-01/.chunks//5c1e': No space left on device
'/root/backup/.chunks//5c1e/5c1e2d7cb95ab01fd517782b2e331afa061bb445feadb9301957603e6f5c7bc1' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e2d7cb95ab01fd517782b2e331afa061bb445feadb9301957603e6f5c7bc1'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e2d7cb95ab01fd517782b2e331afa061bb445feadb9301957603e6f5c7bc1': No such file or directory
'/root/backup/.chunks//5c1e/5c1e009e7525e563633e92fcc66a23149158331491ccdecccf4ae20cc2a99f03' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e009e7525e563633e92fcc66a23149158331491ccdecccf4ae20cc2a99f03'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e009e7525e563633e92fcc66a23149158331491ccdecccf4ae20cc2a99f03': No such file or directory
'/root/backup/.chunks//5c1e/5c1ed9621ba1bb974d91719cd251b70d8953afaca81633d5ddaa87ec95618366' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1ed9621ba1bb974d91719cd251b70d8953afaca81633d5ddaa87ec95618366'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1ed9621ba1bb974d91719cd251b70d8953afaca81633d5ddaa87ec95618366': No such file or directory
'/root/backup/.chunks//5c1e/5c1e954529f235c9570c856af4892e436a734400f515caa91cbea1c70194d1b4' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e954529f235c9570c856af4892e436a734400f515caa91cbea1c70194d1b4'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e954529f235c9570c856af4892e436a734400f515caa91cbea1c70194d1b4': No such file or directory
'/root/backup/.chunks//5c1e/5c1e24823184fbd0272ee9c257a68f363864fcaa914fd729ff98fcff84c27dc9' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e24823184fbd0272ee9c257a68f363864fcaa914fd729ff98fcff84c27dc9'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e24823184fbd0272ee9c257a68f363864fcaa914fd729ff98fcff84c27dc9': No such file or directory
'/root/backup/.chunks//5c1e/5c1e6ba52e4447b3bf0cfd7625182999a50385da710a494837be83397e49e4a6' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e6ba52e4447b3bf0cfd7625182999a50385da710a494837be83397e49e4a6'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1e6ba52e4447b3bf0cfd7625182999a50385da710a494837be83397e49e4a6': No such file or directory
'/root/backup/.chunks//5c1e/5c1ef63a1b0c3744299122210d6d1283d6761041e402e63c80d20224f91f1150' -> '/mnt/datastore/dpool-01/.chunks//5c1e/5c1ef63a1b0c3744299122210d6d1283d6761041e402e63c80d20224f91f1150'
cp: cannot create regular file '/mnt/datastore/dpool-01/.chunks//5c1e/5c1ef63a1b0c3744299122210d6d1283d6761041e402e63c80d20224f91f1150': No such file or directory
Deleting leftover files and folders in /root/backup/.chunks/...

Items restored, permissions, timestamps, and ownership checked, and leftover files deleted successfully.

root@lenny:~#
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back