Finding all snapshots in cluster

[Sudodude]

Hello,

I am running a 5-node HA ceph cluster. I like to occasionally look for old snapshots that are no longer needed so I can remove them. I have been trying to figure out a good way to scan the entire cluster for snapshots but have not had good results so far. Currently I run this command on each node:

for x in $(qm list | tail +2 | awk '{print $1}'); do echo -n $x; qm status $x -verbose| grep name | cut -d':' -f2; qm listsnapshot $x | grep -v current; done

It's not pretty, but it works in showing the VM ID, name, and all snapshots that each VM may have.

Does anyone have a suggestion on how I can get all the snapshots from the cluster without having to run this on each node?

Thank you!

 

[bbgeek17]

You should be using API. A good start would be to process output of "pvesh".
i.e.: pvesh get /cluster/resources
Use jq to process json: pvesh get /cluster/resources --output-forma json
You can do all the same stuff you do now, except query entire cluster and any VM or storage.
If you want to run this off the PVE, you can use "curl" to send the APIs.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Sudodude]

I have used pvesh to look at which values are returned, but unless I overlooked it I did not see any snapshot details.

 

[bbgeek17]

There is not a single API that gets you everything.

high level:

for node in "pvesh get /cluster/resources";do
  for vm in " pvesh get /nodes/$node/qemu";do
     pvesh get /nodes/$node/qemu/$vm/snapshot
  done
done

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Sudodude]

Thanks for the command. Unfortunately when I run it in bash, it errors with

400 too many arguments

 

[spirit]

another way, list directly snapshot from ceph

# rbd ls -l -p <poolname>

 

[bbgeek17]

Thanks for the command. Unfortunately when I run it in bash, it errors with

What exactly did you run? The code I provided was "pseudo code" few steps away from being runnable directly. Its missing all the "jq" processing. It was meant to point you in the right direction not as a finaloutput.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[bbgeek17]

You may want to look at this thread for some more hints:
https://forum.proxmox.com/threads/list-all-vms-in-specific-pool-with-pvesh.103690/

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Sudodude]

another way, list directly snapshot from ceph

# rbd ls -l -p <poolname>

Thank you, this worked perfectly! I never thought about using rbd to find them.

 

[ZipTX]

Thanks @bbgeek17 for the hints and your code examples in other places. Here is my rough contribution for those needing a quick snapshot list without the hours of figuring out syntax. FYI - it's a *slow* script having to do an API query to every VM to check if there are snapshots, but the script concisely gets the data needed.

#!/usr/bin/env bash

nodes=$(pvesh get /cluster/resources --type vm  --output-format json |jq -r '.[] | [.vmid, .name, .node] | @json')
printf  "processing data ... \n"
for i in $nodes; do
  #printf "\n i= $i" # -- shows all iterations for troubleshooting
  vmid=$(jq -r '.[0]' <<< $i);
  vmname=$(jq -r '.[1]' <<< $i);
  node=$(jq -r '.[2]' <<< $i);
  snapshots=$(pvesh get "/nodes/$node/qemu/$vmid/snapshot" --output-format=json)
  snapdates=""
  snapdates=$(echo $snapshots | jq -r '[.[] | select(.snaptime) | (.snaptime  | tonumber | todate[:10])] | @csv ');
  if [[ $snapdates ]]; then
    printf "\n $node \t $vmid \t $vmname \t SNAPDATES: $snapdates"
  fi
done;
printf "\n processing complete. \n"

 

[43n12y]

if you have many vms and/or are impatient, here is a faster script:
you have to set the variable pool to the name of your pool

#/bin/bash
pool=<enter your pool name> # "ceph osd pool ls" may help


tmpfile=$(mktemp /tmp/cluster_rescources.XXXXX)
pvesh get /cluster/resources --type vm --output-format json-pretty >> ${tmpfile}
for vmid in $(rbd ls -l -p ${pool}|grep "vm.*@" | cut -c 4-6|uniq); do
    vmname=$(jq -r ".[] | select ( .vmid == ${vmid}) | .name" ${tmpfile})
    node=$(jq -r ".[] | select ( .vmid == ${vmid}) | .node" ${tmpfile})
    filter=".[] | select ( .name != \"current\" ) + {\"vmname\": \"${vmname}\",\"vmid\": \"${vmid}\"} | .snaptime |= strftime(\"%Y-%m-%d\")"
    pvesh get /nodes/${node}/qemu/${vmid}/snapshot --output-format json-pretty | jq "${filter}"
done
rm ${tmpfile}

 

[StK.pm]

Thanks a lot!
Just a note: This only works when the IDs are 3 digits.
change cut -c 4-6 to cut -c 4-7 if you use 4 digits.

to install jq just use
apt install jq

 

[Interpipes]

To reliably get snaps for any ID length, (and to also not match rows for things like templates on datastores which contain "vm") instead consider the following to get the list of ceph snapshots which greps for "vm-.*@" appearing from the start of a line and uses the "-" as a delimiter (as the vmid should, AIUI, always be the second token when delimiting by "-"):

rbd ls -l -p vlt-vms | grep "^vm-.*@" | cut -f 2 -d "-" | uniq

I made some other refinements.. I added back a 'table' output for humans to read.

In table mode it will multi-line snapshot dates and truncate Node name and VM Name if it has to in order to make it fit on the console.

#/bin/bash

# Help Function
Help()
{
        echo
        echo "Show information about VM snapshots that exist on a given Ceph pool"
        echo
        echo "Syntax: $0 -p POOLNAME"
        echo
        echo "Required:"
        echo "-p NAME   Specify the pool name"
        echo
        echo "Options:"
        echo "-h        Print this help message"
        echo "-t        Print out as a table - i.e. don't output as the default JSON"
        echo
        echo "To get a list of pool names, you can use:"
        echo "ceph osd pool ls"
        echo
}

unset -v pool
unset -v text

while getopts hp:t opt; do
        case $opt in
                h) Help ; exit ;;
                p) pool=$OPTARG ;;
                t) text=true ;;
                *) Help ; exit ;;
        esac
done

: ${pool:?-p is required: You must specify the Ceph pool with -p NAME}

tmpfile=$(mktemp /tmp/snapshot-search-$pool.XXXXXX)

pvesh get /cluster/resources --type vm --output-format json-pretty >> ${tmpfile}

if [ $text ]; then
        unset -v tabledata
fi

for vmid in $(rbd ls -l -p $pool | grep "^vm-.*@" | cut -f 2 -d "-" | uniq); do
        vmname=$(jq -r ".[] | select ( .vmid == ${vmid}) | .name" ${tmpfile})
        node=$(jq -r ".[] | select ( .vmid == ${vmid}) | .node" ${tmpfile})
        filter=".[] | select ( .name != \"current\" ) + {\"vmname\": \"${vmname}\",\"vmid\": \"${vmid}\"} | .snaptime |= strftime(\"%Y-%m-%d\")"
        if [ $text ]; then
                unset -v snapdates
                snapdates=$(pvesh get "/nodes/$node/qemu/$vmid/snapshot" --output-format=json | jq -r '[.[] | select(.snaptime) | (.snaptime | tonumber | todate[:10])] | @csv');
                tabledata="$tabledata$node!$vmid!$vmname!$snapdates\n"
        else
                pvesh get /nodes/${node}/qemu/${vmid}/snapshot --output-format json-pretty | jq "${filter}"
        fi
done

if [ $text ]; then
        printf $tabledata | column -t -s "!" -N "Node,VMID,VM Name,Snapshot Dates" -T "Node","VM Name" -W "Snapshot Dates"
fi

rm ${tmpfile}