[TUTORIAL] Automatically Enable/Disable Datastores to a secondary Proxmox Host depending on host availability

Z

Zac

Member
Jan 18, 2022
5
1
8
Hey All,

I just wanted to chime in on this in case anyone else is in a similar situation.

I have a Primary Proxmox host that runs all my VMs and network storage, and a Secondary Proxmox Backup host that I only turn on every so often to backup my Primary host over to, but otherwise keep it off the majority of the time, only really turning it on for a week or so every month or two.

Noticed my journalctl was absolutely full of entries every 10 seconds:

Code: 
Apr 29 08:27:17 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:27:27 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:27:31 OkamiNAS pvedaemon[306572]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:27:37 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:27:47 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:27:51 OkamiNAS pvedaemon[331187]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:27:57 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:08 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:11 OkamiNAS pvedaemon[331187]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:28:17 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:27 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:31 OkamiNAS pvedaemon[331187]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:28:38 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:47 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:51 OkamiNAS pvedaemon[306572]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:28:57 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:08 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:11 OkamiNAS pvedaemon[331187]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:29:17 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:27 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:31 OkamiNAS pvedaemon[306571]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:29:37 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:47 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:51 OkamiNAS pvedaemon[331187]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:29:57 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)

Code: 
Apr 29 09:52:27 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:52:37 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:52:48 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:52:57 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:53:07 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:53:18 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:53:27 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:53:37 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:53:47 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)

So I decided to see what I could do to automate this, as disabling the datastore under Datacenter > Storage will indeed get rid of the errors, but then the Datastore isn't accessible until you re-enable it which is a pain to have to remember to do each time you want to run backups.

I made a bash script that will effectively attempt to ping the backup host, if pings succeed AND the datastore is currently disabled, it enables it. Otherwise if pings fail 5 times in a row AND the datastore is currently enabled, it disables it.

If pings succeed and the datastore is enabled, or if pings fail and the datastore is disabled, then it does nothing.

Bash: 
#!/bin/bash

BACKUP_SERVER_IP="192.168.1.24"
DATASTORE_ID="Remote_ProxmoxBackupServer"
MAX_PING_FAILURES=10

# Function to check if the datastore is enabled
is_datastore_enabled() {
    disable_status=$(pvesh get /storage/$DATASTORE_ID --output-format json-pretty | grep -o '"disable":[0-9]')
    if [ -z "$disable_status" ]; then
        return 0  # Datastore is enabled (disable option not found)
    else
        status=$(echo "$disable_status" | awk -F: '{print $2}')
        if [ "$status" = "0" ]; then
            return 0  # Datastore is enabled
        else
            return 1  # Datastore is disabled
        fi
    fi
}

# Function to ping the backup server multiple times
ping_backup_server() {
    local ping_failures=0
    for ((i=1; i<=MAX_PING_FAILURES; i++)); do
        if ! ping -c 1 $BACKUP_SERVER_IP &> /dev/null; then
            ping_failures=$((ping_failures + 1))
        else
            return 0  # Ping successful, return success
        fi
    done
    return 1  # All pings failed, return failure
}

# Check if the backup server is reachable
if ping_backup_server; then
    # Backup server is reachable
    if ! is_datastore_enabled; then
        # Datastore is disabled, enable it
        pvesh set /storage/$DATASTORE_ID --disable 0
        echo "Datastore $DATASTORE_ID has been enabled."
    fi
else
    # Backup server is not reachable
    if is_datastore_enabled; then
        # Datastore is enabled, disable it
        pvesh set /storage/$DATASTORE_ID --disable 1
        echo "Datastore $DATASTORE_ID has been disabled."
    fi
fi

And the cron.d entry

Bash: 
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user    command
*/2 * * * * root /path/to/your/script/pbs_datastore.sh > /dev/null 2>&1

I'm sure there's room for improvement, but figured I'd share this in case there was anyone else that was in a similar boat with multiple proxmox hosts, where one of them isn't always on, but you want the datastore to be enabled whenever the secondary host is reachable, otherwise disable it.

--Edit--
Made some minor corrections
 
Last edited:
  • Like
Reactions: bbgeek17
gurubert said:
Neat. To parse the JSON output you could use jq:

Bash: 
pvesh get /storage/$DATASTORE_ID --output-format json-pretty | jq .disable
Click to expand...

Thanks! I thought about using jq, but since it's not included by default with Proxmox, I wanted to keep required packages to ones that are already included as not to have to install extras to get it to work, but jq would definitely simplify things. :)

Also expanded it to allow for multiple hosts, with each host containing multiple datastores if required or if someone has a more complex setup with more than one additional Proxmox host:


Bash: 
#!/bin/bash

declare -A SERVER_DATASTORES=(
    ["192.168.1.24"]="Remote_ProxmoxBackupServer"
)
# Additional hosts may be added per line above, and may contain multiple datastores per host.
# Example:
# ["192.168.1.25"]="Datastore1,Datastore2"
# ["192.168.1.26"]="Datastore3,Datastore4,Datastore5,Datastore6"

MAX_PING_FAILURES=5

# Function to check if a datastore is enabled
is_datastore_enabled() {
    local datastore_id=$1
    disable_status=$(pvesh get /storage/$datastore_id --output-format json-pretty | grep -o '"disable":[0-9]')
    if [ -z "$disable_status" ]; then
        return 0  # Datastore is enabled (disable option not found)
    else
        status=$(echo "$disable_status" | awk -F: '{print $2}')
        if [ "$status" = "0" ]; then
            return 0  # Datastore is enabled
        else
            return 1  # Datastore is disabled
        fi
    fi
}

# Function to ping a server multiple times
ping_server() {
    local server_ip=$1
    local ping_failures=0
    for ((i=1; i<=MAX_PING_FAILURES; i++)); do
        if ! ping -c 1 $server_ip &> /dev/null; then
            ping_failures=$((ping_failures + 1))
        else
            return 0  # Ping successful, return success
        fi
    done
    return 1  # All pings failed, return failure
}

# Iterate over the server-datastore pairs
for server_ip in "${!SERVER_DATASTORES[@]}"; do
    datastore_list="${SERVER_DATASTORES[$server_ip]}"

    # Check if the server is reachable
    if ping_server $server_ip; then
        # Server is reachable, enable the datastores
        IFS=',' read -ra datastores <<< "$datastore_list"
        for datastore_id in "${datastores[@]}"; do
            if ! is_datastore_enabled $datastore_id; then
                pvesh set /storage/$datastore_id --disable 0
                echo "Datastore $datastore_id has been enabled."
            fi
        done
    else
        # Server is not reachable, disable the datastores
        IFS=',' read -ra datastores <<< "$datastore_list"
        for datastore_id in "${datastores[@]}"; do
            if is_datastore_enabled $datastore_id; then
                pvesh set /storage/$datastore_id --disable 1
                echo "Datastore $datastore_id has been disabled."
            fi
        done
    fi
done
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back