cpu and memory usage measuring to kill container

[Forssux]

Hi There,

I'm running a frigate container that sometimes go crazy for some reasons.

I have a script monitor.sh that should check this and kills the container before it's 92%.

#!/bin/bash


CONTAINER_ID="your_container_id_here"


while true; do
    # Get total CPU time from /proc/stat
    TOTAL_BEFORE=$(cat /proc/stat | grep cpu | awk '{print $2+$3+$4+$5+$6+$7+$8}')
    # Get container's CPU time
    CONTAINER_CPU_BEFORE=$(cat /sys/fs/cgroup/cpu,cpuacct/lxc/$CONTAINER_ID/cpuacct.usage)


    # Sleep for a short interval (e.g., 0.5 seconds)
    sleep 0.5


    # Repeat the above to get current CPU time
    TOTAL_AFTER=$(cat /proc/stat | grep cpu | awk '{print $2+$3+$4+$5+$6+$7+$8}')
    CONTAINER_CPU_AFTER=$(cat /sys/fs/cgroup/cpu,cpuacct/lxc/$CONTAINER_ID/cpuacct.usage)


    # Calculate CPU usage
    TOTAL_DELTA=$((TOTAL_AFTER - TOTAL_BEFORE))
    CONTAINER_CPU_DELTA=$((CONTAINER_CPU_AFTER - CONTAINER_CPU_BEFORE))
    CPU_USAGE=$((100 * CONTAINER_CPU_DELTA / TOTAL_DELTA))


    # Get memory usage and limit
    MEMORY_USAGE=$(cat /sys/fs/cgroup/memory/lxc/$CONTAINER_ID/memory.usage_in_bytes)
    MEMORY_LIMIT=$(cat /sys/fs/cgroup/memory/lxc/$CONTAINER_ID/memory.limit_in_bytes)
    MEM_PERCENTAGE=$((100 * MEMORY_USAGE / MEMORY_LIMIT))


    # Check if either CPU or Memory usage exceeds 91%
    if (( CPU_USAGE > 91 )) || (( MEM_PERCENTAGE > 91 )); then
        # Kill the container
        pct stop $CONTAINER_ID
      
        # Systemd or another tool should be configured to restart the container if it's killed/stopped
    fi
done

I then made sure it's executable and in systemd

chmod +x /path/to/monitor.sh
Create a systemd Service File:Create a new file in /etc/systemd/system/, e.g., monitor.service:

I use systemd to stop start it.
sudo nano /etc/systemd/system/monitor.service
Paste the following:

makefileCopy code
[Unit]
Description=Monitor container resources

[Service]
ExecStart=/path/to/monitor.sh
Restart=always
User=yourusername
Group=yourgroupname

[Install]
WantedBy=multi-user.target

/path/to/monitor.sh should be where the script is. Also, replace yourusername and yourgroupname

sudo systemctl daemon-reload
sudo systemctl enable monitor.service
sudo systemctl start monitor.service
sudo systemctl status monitor.service

However it seems that this line isn't accurate anymore:
CONTAINER_CPU_AFTER=$(cat /sys/fs/cgroup/cpu,cpuacct/lxc/$CONTAINER_ID/cpuacct.usage)

Can somebody point me in the wright direction?

Kind regards
Guy