Guest shows strange commited memory stats

[H4R0]

I have noticed strange memory stats for a docker 10 vm.

The vm itself is set to 6-7gb (balooning) of ram but commits up to 240gb ?

Im trying to figue out what is going on, the vm itself is running without problems.

There are no kernel messages in dmesg. The memory values come directly from /proc/meminfo


I couldnt find a way to list processes by commited memory, does anyone know how to filter them out ?

The vm is running docker with various containers, it is intended to swap about 2-4gb of ram out to disk. (vm.swappiness is set low to 5)

The vm itself is a clone of a template, another clone does not have the problem.

root@docker3:/var/log# cat /proc/meminfo
MemTotal:        7012756 kB
MemFree:         2002656 kB
MemAvailable:    3835588 kB
Buffers:          739924 kB
Cached:           545028 kB
SwapCached:        84380 kB
Active:          2840708 kB
Inactive:         950000 kB
Active(anon):    1828656 kB
Inactive(anon):   762068 kB
Active(file):    1012052 kB
Inactive(file):   187932 kB
Unevictable:       35064 kB
Mlocked:           35064 kB
SwapTotal:       8385532 kB
SwapFree:        6271228 kB
Dirty:              6536 kB
Writeback:             0 kB
AnonPages:       2511736 kB
Mapped:           226416 kB
Shmem:             83788 kB
Slab:            1061936 kB
SReclaimable:     935600 kB
SUnreclaim:       126336 kB
KernelStack:       15840 kB
PageTables:        42484 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    11891908 kB
Committed_AS:   12538940 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:             2064 kB
HardwareCorrupted:     0 kB
AnonHugePages:    487424 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      859768 kB
DirectMap2M:     6477824 kB

root@docker3:/var/log# ps -o pid,user,%mem,command ax | sort -b -k3 -r | head -n 10
  PID USER     %MEM COMMAND
 6034 998       7.3 unicorn worker[1] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru
 6032 998       7.2 unicorn worker[0] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru
 4923 998       5.7 unicorn master -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru
 4738 998       5.6 sidekiq 5.2.7 gitlab-rails [0 of 2 busy]
 2486 Debian-+  2.7 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/x86_64-linux-gnu/mariadb19/plugin --user=mysql --skip-log-error --pid-file=/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock
 3721 postfix   2.5 /usr/sbin/clamd -c /etc/clamav/clamd.conf
 8749 112       2.0 /usr/sbin/amavisd-new (ch7-avail)
 8750 112       1.9 /usr/sbin/amavisd-new (ch6-avail)
 8701 112       1.8 /usr/sbin/amavisd-new (master)

 

[H4R0]

I wrote the following script to compare memory usage before and after a container is run

I found the container commiting the memory with a memory leak

Even though memory and swap are limited by cgroup, commited memory doesnt seem to

Commited memory has now been stable for 3 days

cat << 'EOF' > /root/memoryusage.sh
#!/bin/bash
echo "---- Docker Memory Usage Dump ----"

if docker --version >/dev/null 2>&1; then
    mkdir memoryusage
    chmod 400 memoryusage
    cd memoryusage

    # docker containers
    docker --version > running.txt
    docker ps -a >> running.txt
    docker stats --no-stream | tee -a running.txt

    containers=`docker ps | awk '{if(NR>1) print $NF}'`
    for container in $containers; do
        echo -e "\n\n\ndumping container memory usage of $container "`date`

        cat /proc/meminfo > "$container.after"
       
        docker stop "$container" >/dev/null 2>&1; sleep 1

        cat /proc/meminfo > "$container.before"

        echo -e "\n\nbefore and after memory usage of $container" > "$container.txt"
        diff --suppress-common-lines -y "$container.before" "$container.after" | tee "$container.txt"

        echo -e "\n\nfull diff:" >> "$container.txt"
        diff -y "$container.before" "$container.after" >> "$container.txt"

        rm "$container.before" "$container.after"
    done

    echo -e "\n\n\nstarting containers... "`date`
    for container in $containers; do
        docker start "$container" >/dev/null 2>&1
    done
    echo -e "\n\n\ndone "`date`
else
    echo "error missing docker"

    exit 1
fi
EOF