[TUTORIAL] Automatically Enable/Disable Datastores to a secondary Proxmox Host depending on host availability

[Zac]

Hey All,

I just wanted to chime in on this in case anyone else is in a similar situation.

I have a Primary Proxmox host that runs all my VMs and network storage, and a Secondary Proxmox Backup host that I only turn on every so often to backup my Primary host over to, but otherwise keep it off the majority of the time, only really turning it on for a week or so every month or two.

Noticed my journalctl was absolutely full of entries every 10 seconds:

Apr 29 08:27:17 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:27:27 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:27:31 OkamiNAS pvedaemon[306572]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:27:37 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:27:47 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:27:51 OkamiNAS pvedaemon[331187]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:27:57 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:08 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:11 OkamiNAS pvedaemon[331187]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:28:17 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:27 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:31 OkamiNAS pvedaemon[331187]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:28:38 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:47 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:28:51 OkamiNAS pvedaemon[306572]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:28:57 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:08 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:11 OkamiNAS pvedaemon[331187]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:29:17 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:27 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:31 OkamiNAS pvedaemon[306571]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:29:37 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:47 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 08:29:51 OkamiNAS pvedaemon[331187]: VM 131 qmp command failed - VM 131 qmp command 'guest-ping' failed - got timeout
Apr 29 08:29:57 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)

Apr 29 09:52:27 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:52:37 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:52:48 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:52:57 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:53:07 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:53:18 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:53:27 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:53:37 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)
Apr 29 09:53:47 OkamiNAS pvestatd[5213]: Remote_ProxmoxBackupServer: error fetching datastores - 500 Can't connect to 192.168.1.24:8007 (No route to host)

So I decided to see what I could do to automate this, as disabling the datastore under Datacenter > Storage will indeed get rid of the errors, but then the Datastore isn't accessible until you re-enable it which is a pain to have to remember to do each time you want to run backups.

I made a bash script that will effectively attempt to ping the backup host, if pings succeed AND the datastore is currently disabled, it enables it. Otherwise if pings fail 5 times in a row AND the datastore is currently enabled, it disables it.

If pings succeed and the datastore is enabled, or if pings fail and the datastore is disabled, then it does nothing.

#!/bin/bash

BACKUP_SERVER_IP="192.168.1.24"
DATASTORE_ID="Remote_ProxmoxBackupServer"
MAX_PING_FAILURES=10

# Function to check if the datastore is enabled
is_datastore_enabled() {
    disable_status=$(pvesh get /storage/$DATASTORE_ID --output-format json-pretty | grep -o '"disable":[0-9]')
    if [ -z "$disable_status" ]; then
        return 0  # Datastore is enabled (disable option not found)
    else
        status=$(echo "$disable_status" | awk -F: '{print $2}')
        if [ "$status" = "0" ]; then
            return 0  # Datastore is enabled
        else
            return 1  # Datastore is disabled
        fi
    fi
}

# Function to ping the backup server multiple times
ping_backup_server() {
    local ping_failures=0
    for ((i=1; i<=MAX_PING_FAILURES; i++)); do
        if ! ping -c 1 $BACKUP_SERVER_IP &> /dev/null; then
            ping_failures=$((ping_failures + 1))
        else
            return 0  # Ping successful, return success
        fi
    done
    return 1  # All pings failed, return failure
}

# Check if the backup server is reachable
if ping_backup_server; then
    # Backup server is reachable
    if ! is_datastore_enabled; then
        # Datastore is disabled, enable it
        pvesh set /storage/$DATASTORE_ID --disable 0
        echo "Datastore $DATASTORE_ID has been enabled."
    fi
else
    # Backup server is not reachable
    if is_datastore_enabled; then
        # Datastore is enabled, disable it
        pvesh set /storage/$DATASTORE_ID --disable 1
        echo "Datastore $DATASTORE_ID has been disabled."
    fi
fi

And the cron.d entry

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user    command
*/2 * * * * root /path/to/your/script/pbs_datastore.sh > /dev/null 2>&1

I'm sure there's room for improvement, but figured I'd share this in case there was anyone else that was in a similar boat with multiple proxmox hosts, where one of them isn't always on, but you want the datastore to be enabled whenever the secondary host is reachable, otherwise disable it.

--Edit--
Made some minor corrections

 

[gurubert]

Neat. To parse the JSON output you could use jq:

pvesh get /storage/$DATASTORE_ID --output-format json-pretty | jq .disable

 

[Zac]

Neat. To parse the JSON output you could use jq:

pvesh get /storage/$DATASTORE_ID --output-format json-pretty | jq .disable

Thanks! I thought about using jq, but since it's not included by default with Proxmox, I wanted to keep required packages to ones that are already included as not to have to install extras to get it to work, but jq would definitely simplify things.

Also expanded it to allow for multiple hosts, with each host containing multiple datastores if required or if someone has a more complex setup with more than one additional Proxmox host:

#!/bin/bash

declare -A SERVER_DATASTORES=(
    ["192.168.1.24"]="Remote_ProxmoxBackupServer"
)
# Additional hosts may be added per line above, and may contain multiple datastores per host.
# Example:
# ["192.168.1.25"]="Datastore1,Datastore2"
# ["192.168.1.26"]="Datastore3,Datastore4,Datastore5,Datastore6"

MAX_PING_FAILURES=5

# Function to check if a datastore is enabled
is_datastore_enabled() {
    local datastore_id=$1
    disable_status=$(pvesh get /storage/$datastore_id --output-format json-pretty | grep -o '"disable":[0-9]')
    if [ -z "$disable_status" ]; then
        return 0  # Datastore is enabled (disable option not found)
    else
        status=$(echo "$disable_status" | awk -F: '{print $2}')
        if [ "$status" = "0" ]; then
            return 0  # Datastore is enabled
        else
            return 1  # Datastore is disabled
        fi
    fi
}

# Function to ping a server multiple times
ping_server() {
    local server_ip=$1
    local ping_failures=0
    for ((i=1; i<=MAX_PING_FAILURES; i++)); do
        if ! ping -c 1 $server_ip &> /dev/null; then
            ping_failures=$((ping_failures + 1))
        else
            return 0  # Ping successful, return success
        fi
    done
    return 1  # All pings failed, return failure
}

# Iterate over the server-datastore pairs
for server_ip in "${!SERVER_DATASTORES[@]}"; do
    datastore_list="${SERVER_DATASTORES[$server_ip]}"

    # Check if the server is reachable
    if ping_server $server_ip; then
        # Server is reachable, enable the datastores
        IFS=',' read -ra datastores <<< "$datastore_list"
        for datastore_id in "${datastores[@]}"; do
            if ! is_datastore_enabled $datastore_id; then
                pvesh set /storage/$datastore_id --disable 0
                echo "Datastore $datastore_id has been enabled."
            fi
        done
    else
        # Server is not reachable, disable the datastores
        IFS=',' read -ra datastores <<< "$datastore_list"
        for datastore_id in "${datastores[@]}"; do
            if is_datastore_enabled $datastore_id; then
                pvesh set /storage/$datastore_id --disable 1
                echo "Datastore $datastore_id has been disabled."
            fi
        done
    fi
done

 

[gurubert]

I thought about using jq, but since it's not included by default with Proxmox, I wanted to keep required packages to ones that are already included

Ah, makes sense. We install jq and a bunch of other tools by default on our Proxmox systems.

 

[magingale]

@Zac many thanks for sharing this, this was exactly where I was looking for! I let chatGPT look at the code and it create some enhancements let me write them down below.

• Parsing JSON using grep and awk can be error-prone, especially if the JSON structure changes.
• Solution: Use jq for reliable JSON parsing.

• If the pvesh command fails (e.g., due to network issues or incorrect datastore ID), disable_status may be empty, leading to incorrect assumptions.
• Solution: Check the exit status of the pvesh command.

Verify that required commands (pvesh, jq, ping) are available before executing the main logic.

• Efficiency:
  • The function continues to ping even after a successful ping, which is not necessary.
• Timeout Handling:
  • By default, ping might take longer than desired if the server is unreachable.
  • Solution: Use the -W option to set a timeout for each ping attempt.

Logs are appended to /var/log/datastore_management.log with timestamps for better traceability.

#!/bin/bash


# Configuration
declare -A SERVER_DATASTORES=(
    ["192.168.1.254"]="Datastore1"
    # ["192.168.1.25"]="Datastore1,Datastore2"
    # ["192.168.1.26"]="Datastore3,Datastore4,Datastore5,Datastore6"
)
MAX_PING_FAILURES=5
LOG_FILE="/var/log/datastore_management.log"


# Function to check for required commands
check_dependencies() {
    for cmd in pvesh jq ping; do
        if ! command -v "$cmd" &> /dev/null; then
            echo "ERROR: Required command '$cmd' is not installed." | tee -a "$LOG_FILE" >&2
            exit 1
        fi
    done
}


# Function to check if a datastore is enabled
is_datastore_enabled() {
    local datastore_id="$1"
    # Use jq for JSON parsing
    disable_status=$(pvesh get /storage/"$datastore_id" --output-format json | jq -r '.disable // "0"')
    
    if [ "$disable_status" -eq 0 ]; then
        return 0  # Datastore is enabled
    else
        return 1  # Datastore is disabled
    fi
}


# Function to ping a server multiple times
ping_server() {
    local server_ip="$1"
    local attempt
    for ((attempt=1; attempt<=MAX_PING_FAILURES; attempt++)); do
        if ping -c 1 -W 2 "$server_ip" &> /dev/null; then
            return 0  # Ping successful
        fi
    done
    return 1  # All pings failed
}


# Initialize
check_dependencies


# Iterate over the server-datastore pairs
for server_ip in "${!SERVER_DATASTORES[@]}"; do
    datastore_list="${SERVER_DATASTORES[$server_ip]}"


    # Check if the server is reachable
    if ping_server "$server_ip"; then
        # Server is reachable, enable the datastores
        IFS=',' read -ra datastores <<< "$datastore_list"
        for datastore_id in "${datastores[@]}"; do
            if ! is_datastore_enabled "$datastore_id"; then
                if pvesh set /storage/"$datastore_id" --disable 0 &> /dev/null; then
                    echo "$(date '+%Y-%m-%d %H:%M:%S') - Datastore $datastore_id has been enabled." | tee -a "$LOG_FILE"
                else
                    echo "$(date '+%Y-%m-%d %H:%M:%S') - ERROR: Failed to enable datastore $datastore_id." | tee -a "$LOG_FILE" >&2
                fi
            fi
        done
    else
        # Server is not reachable, disable the datastores
        IFS=',' read -ra datastores <<< "$datastore_list"
        for datastore_id in "${datastores[@]}"; do
            if is_datastore_enabled "$datastore_id"; then
                if pvesh set /storage/"$datastore_id" --disable 1 &> /dev/null; then
                    echo "$(date '+%Y-%m-%d %H:%M:%S') - Datastore $datastore_id has been disabled." | tee -a "$LOG_FILE"
                else
                    echo "$(date '+%Y-%m-%d %H:%M:%S') - ERROR: Failed to disable datastore $datastore_id." | tee -a "$LOG_FILE" >&2
                fi
            fi
        done
    fi
done