Simple reset script - ping

[SamTzu]

I noticed that there does not seem to be a simple reset / reboot script for problematic clients so I made one.
Copy / Paste this to autorestart-vm1100.sh using nano

#! /bin/bash
# Replace VM id number and IP address with your own "VM" id number and IP address
ping -c 1 10.168.100.100 &> /dev/null && echo "Ping OK - VM 1100" || /usr/sbin/qm stop 1100 && /usr/sbin/qm start 1100 && echo "No response: Restarting VM 1100"

Remember to:
chmod +x /root/autorestart-vm1100.sh

Add this "oneliner" to run every hour on cron like this...
On your host use crontab -e to add this line:
@hourly /root/autorestart-vm1100.sh

Restart your cron
service cron restart

 

[SamTzu]

Should be easy to adapt for LXC.

 

[rursache]

I noticed that there does not seem to be a simple reset / reboot script for problematic clients so I made one.

Thanks for your script, great starting point!

I adapted it to my needs and its available here for anyone interested

 

[LnxBil]

If your guest OS (QEMU/KVM VM) really crashes, just use a watchdog device and configure it in your guest. I'm using it for years.

If you have external monitoring set up for your PVE host, why don't you just use it to restart/reboot the machines, e.g. Nagios/Icinga have actions that will trigger if e.g. monitoring cannot query the guest os. Seems a much cleaner solution that creating another DIY script for that.

 

[rursache]

If your guest OS (QEMU/KVM VM) really crashes, just use a watchdog device and configure it in your guest.

I totally agree with you that a proper WatchDog config is 10 times better than a bad bash script. However it was easier for me to just write this quick script until i have time to actually fix the VM crash. I'm still new to Proxmox and hypervisors

 

[SamTzu]

Datacenter menu should include Scripts.
Something simple that could be saved in .sh script under /etc/pve/scripts folder.

 

[rursache]

Datacenter menu should include Scripts.
Something simple that could be saved in .sh script under /etc/pve/scripts folder.

i can't find that entry. i will PM you if you have 10 minutes to help me optimize stuff. thanks!

 

[SamTzu]

If your guest OS (QEMU/KVM VM) really crashes, just use a watchdog device and configure it in your guest. I'm using it for years.

If you have external monitoring set up for your PVE host, why don't you just use it to restart/reboot the machines, e.g. Nagios/Icinga have actions that will trigger if e.g. monitoring cannot query the guest os. Seems a much cleaner solution that creating another DIY script for that.

Most of our VM's are LXC containers.

 

[LnxBil]

Most of our VM's are LXC containers.

You have connectivity problems there? That sounds interessting, I've never encountered those on LX(C) containers. Have you debugged what went wrong?

 

[SamTzu]

Legacy containers (with multiple vulnerabilities) that can't easily be upgraded. We usually just dump them on one host Proxmox and leave them to die slow death.

 

[LnxBil]

Legacy containers (with multiple vulnerabilities) that can't easily be upgraded. We usually just dump them on one host Proxmox and leave them to die slow death.

Thank you. I ran multiple nested virtualization setups just for that.

 

[crazeeEyez]

Thanks for your script, great starting point!

I adapted it to my needs and its available here for anyone interested

Thank you for sharing this script! PVE Watchdog didn't work for me, for some reason, but this watchdog script works perfectly (as did @SamTzu's script).

 

[crazeeEyez]

Thanks for your script, great starting point!

I adapted it to my needs and its available here for anyone interested

Thank you again for your script @rursache! I made a small change to get the VM uptime instead of the PVE uptime

#!/bin/bash
#
# WatchDog script for Proxmox VMs
# v1.0
#
# Install using `crontab -e` like this: * * * * * /root/watchdog.sh
#
# Based on https://forum.proxmox.com/threads/simple-reset-script-ping.49901/
#

# Config
debianVmName="debian"
debianVmId="102"
debianVmIp="192.168.10.209"

haVmName="homeassistant"
haVmId="101"
haVmIp="192.168.10.210"

# Log func
log () {
    output="[WatchDog] $1"
    echo $output && logger $output
}

# Check VM func
checkVM () {
if /usr/bin/ping -c 1 $3 &> /dev/null; then
    vmUptime=$(/usr/sbin/qm status $2 -verbose | grep uptime | cut -f2 -d' ')
    log "$1 seems alive, no action taken.  Current uptime: $vmUptime"
else
    log "$1 Crashed - Restarting now..."
    /usr/sbin/qm stop $2 && /usr/sbin/qm start $2
fi
}

# Logic
checkVM $debianVmName $debianVmId $debianVmIp
checkVM $haVmName $haVmId $haVmIp

 

[gyrex]

This script is awesome! Thanks so much to @rursache for your script.

I use Slack to get alerts/notifications on my mobile phone and I've modified the script to include sending an alert to a Slack channel if a server goes down/becomes unresponsive. To generate your own Slack webhook, visit this page for more info: https://www.howtogeek.com/devops/how-to-send-a-message-to-slack-from-a-bash-script/

#!/bin/bash
#
# WatchDog script for Proxmox VMs
# v1.1
#
# Install using `crontab -e` like this: * * * * * /root/watchdog.sh
#
# Based on https://forum.proxmox.com/threads/simple-reset-script-ping.49901/
#

# Config
slack_url=https://hooks.slack.com/services/<insert your own URL here>

ExampleVmName="Example"
ExampleVmId="100"
ExampleVmIp="xxx.xxx.xxx.xxx"

# Log func
log () {
    output="[WatchDog] $1"
    echo $output && logger $output
}

# Check VM func
checkVM () {
if /usr/bin/ping -c 1 $3 &> /dev/null; then
    vmUptime=$(/usr/sbin/qm status $2 -verbose | grep uptime | cut -f2 -d' ')
    log "$1 seems alive, no action taken. Current uptime: $vmUptime"
else
    log "$1 Crashed - Restarting now..."
    curl -X POST -H 'Content-type: application/json' --data "{'text':'ALERT for Proxmox VM: $1 has crashed, automatically restarting'}" $slack_url;
    /usr/sbin/qm stop $2 && /usr/sbin/qm start $2
fi
}

# Logic
checkVM $ExampleVmName $ExampleVmId $ExampleVmIp