Memory management problem, Proxmox 3.4

G

Gergo Kapsza

New Member
Dec 14, 2015
4
0
1
50
Hello!

We've got a web server having one VM in Proxmox (v3.4-11). In the VM cPanel/WHM is running (Apache, MySQL, etc.).
The physical node has 32GB of RAM, but memory usage never goes up above 8-9GBs. If we enable mysql to use larger buffers (even 1-2GBs), the server starts to crash randomly. The hardware is tested, RAM, HDD, etc. is OK, the VM can use all RAM, etc.

We guess there is a memory management problem shown on the Munin graph below. On the left hald of the chart memory usage is OK. At the center of the graph active memory usage drops to almost zero, inactive memory usage goes up, disk caches gets flushed and the node starts to swap (swappiness is 10) however it has 20GBs of free RAM. After that the server usually crashes soon (we are trying to solve this issue for 5 weeks now, tried many things like updating, reinstalling proxmox, apache, mysql, etc.)

memory.jpg

We could reproduce the drop of active memory usage by counting all files of the home folder for example: find /home -follow -maxdepth 20 -type f | wc -l
(home folder has 11 million inodes).

Do you have an idea what the problem might be or what can keep active memory usage down and causing swapping when having 20GBs of free RAM?

Many thanks!
 
Hi,

can you describe a bit more your setup, which Filesystem/Storage/Partitioning Layout do you use?
Hows the VM configured, do you use VirtIO?

Does it help if you turn the swap temporary off (as a test) when you configure mysql to use larger buffers?
 
Hello Thomas,

Thank you for replying! I'm sending you the server details:

2xSATA 3GB software raid 1
32GB RAM
CPU: 8 x Intel(R) Xeon(R) CPU E31290 @ 3.60GHz

uname: Linux proxmox 2.6.32-43-pve #1 SMP Tue Oct 27 09:55:55 CET 2015 x86_64 GNU/Linux
Debian version: 7.9

fdisk:
Disk /dev/sda: 3000.6 GB, 3000592982016 bytes
256 heads, 63 sectors/track, 363376 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sda1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.

Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes
256 heads, 63 sectors/track, 363376 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdb1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.

Disk /dev/zd0: 24.7 GB, 24696061952 bytes
16 heads, 63 sectors/track, 47851 cylinders, total 48234496 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


lsblk:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 2.7T 0 disk
sda1 8:1 0 1007K 0 part
sda2 8:2 0 2.7T 0 part
sda9 8:9 0 8M 0 part
sdb 8:16 0 2.7T 0 disk
sdb1 8:17 0 1007K 0 part
sdb2 8:18 0 2.7T 0 part
sdb9 8:25 0 8M 0 part
zd0 230:0 0 23G 0 disk [SWAP]


.vmlist:
{
"version": 1,
"ids": {
"100": { "node": "proxmox", "type": "openvz", "version": 1 }}

}

vz config:
## Global parameters
VIRTUOZZO=yes
LOCKDIR=/var/lib/vz/lock
DUMPDIR=/var/lib/vz/dump
VE0CPUUNITS=1000

## Logging parameters
LOGGING=yes
LOGFILE=/var/log/vzctl.log
LOG_LEVEL=0
VERBOSE=0

## Disk quota parameters
DISK_QUOTA=yes
VZFASTBOOT=no

# Disable module loading. If set, vz initscript does not load any modules.
#MODULES_DISABLED=yes

# The name of the device whose IP address will be used as source IP for CT.
# By default automatically assigned.
#VE_ROUTE_SRC_DEV="eth0"

# Controls which interfaces to send ARP requests and modify ARP tables on.
NEIGHBOUR_DEVS=detect

## Fail if there is another machine in the network with the same IP
ERROR_ON_ARPFAIL="no"

## Template parameters
TEMPLATE=/var/lib/vz/template

## Defaults for containers
VE_ROOT=/var/lib/vz/root/$VEID
VE_PRIVATE=/var/lib/vz/private/$VEID

## Filesystem layout for new CTs: either simfs (default) or ploop
#VE_LAYOUT=ploop

## Load vzwdog module
VZWDOG="no"

## IPv4 iptables kernel modules to be enabled in CTs by default
#IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length"
IPTABLES="ipt_REJECT ipt_recent ipt_owner ipt_REDIRECT ipt_tos ipt_TOS ipt_LOG ip_conntrack ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ipt_state iptable_nat ip_nat_ftp"

## IPv4 iptables kernel modules to be loaded by init.d/vz script
IPTABLES_MODULES="$IPTABLES"

## Enable IPv6
IPV6="yes"

## IPv6 ip6tables kernel modules
IP6TABLES="ip6_tables ip6table_filter ip6table_mangle ip6t_REJECT"


100.cnf:
ONBOOT="yes"

PHYSPAGES="0:6660096"
SWAPPAGES="0:256000"
KMEMSIZE="12399411200:13639876608"
DCACHESIZE="6199181312:6819938304"
LOCKEDPAGES="3330048"
PRIVVMPAGES="unlimited"
SHMPAGES="unlimited"
NUMPROC="unlimited"
VMGUARPAGES="0:unlimited"
OOMGUARPAGES="0:unlimited"
NUMTCPSOCK="unlimited"
NUMFLOCK="unlimited"
NUMPTY="unlimited"
NUMSIGINFO="unlimited"
TCPSNDBUF="unlimited"
TCPRCVBUF="unlimited"
OTHERSOCKBUF="unlimited"
DGRAMRCVBUF="unlimited"
NUMOTHERSOCK="unlimited"
NUMFILE="unlimited"
NUMIPTENT="unlimited"

# Disk quota parameters (in form of softlimit:hardlimit)
DISKSPACE="1048576000:1153433600"
DISKINODES="200000000:220000000"
QUOTATIME="0"
QUOTAUGIDLIMIT="1000"

# CPU fair scheduler parameter
CPUUNITS="1000"
CPUS="8"
HOSTNAME="........."
SEARCHDOMAIN=".............."
NAMESERVER="8.8.8.8"
IP_ADDRESS="..........."
VE_ROOT="/var/lib/vz/root/$VEID"
VE_PRIVATE="/var/lib/vz/private/100"
OSTEMPLATE="centos-6-x86_64.tar.gz"


meminfo:
MemTotal: 32916092 kB
MemFree: 5822376 kB
Buffers: 0 kB
Cached: 980964 kB
SwapCached: 4288 kB
MemCommitted: 27688960 kB
VirtualSwap: 0 kB
Active: 1161504 kB
Inactive: 9700332 kB
Active(anon): 747980 kB
Inactive(anon): 9200144 kB
Active(file): 413524 kB
Inactive(file): 500188 kB
Unevictable: 40336 kB
Mlocked: 0 kB
SwapTotal: 24117244 kB
SwapFree: 24095792 kB
Dirty: 16 kB
Writeback: 1264 kB
AnonPages: 9918272 kB
Mapped: 47304 kB
Shmem: 61288 kB
Slab: 7614136 kB
SReclaimable: 293660 kB
SUnreclaim: 7320476 kB
KernelStack: 11568 kB
PageTables: 44780 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 40575288 kB
Committed_AS: 14499340 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 8773800 kB
VmallocChunk: 34343938336 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6144 kB
DirectMap2M: 33540096 kB

Within the container top reports 20GB of free RAM (see the Munin graph in my previous message that was created in the VM).
I cannot figure out why active memory usage is almost zero whle inactive is high. SUnreclaim seems to be a bit high as well (7-8GB out of 32GB).

Many thanks for your opinion and help!

Kind regards,
Gergo Kapsza
 
Hello Thomas,


The server crashed again due to out of RAM however inside the VM there was at least 10GB of free RAM.
At this moment the server is out of RAM again but inside the VM there is 16GB of free RAM. It seem like the OS/Proxmox is eating up all free RAM resulting server crashes.


Current meminfo:
MemTotal: 32916092 kB
MemFree: 1310368 kB
Buffers: 2724 kB
Cached: 1380664 kB
SwapCached: 744 kB
MemCommitted: 27688960 kB
VirtualSwap: 0 kB
Active: 7648584 kB
Inactive: 1764668 kB
Active(anon): 6906596 kB
Inactive(anon): 1190688 kB
Active(file): 741988 kB
Inactive(file): 573980 kB
Unevictable: 17328 kB
Mlocked: 0 kB
SwapTotal: 24117244 kB
SwapFree: 24114684 kB
Dirty: 36 kB
Writeback: 72 kB
AnonPages: 8046920 kB
Mapped: 76264 kB
Shmem: 61456 kB
Slab: 6368124 kB
SReclaimable: 234996 kB
SUnreclaim: 6133128 kB
KernelStack: 9808 kB
PageTables: 36584 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 40575288 kB
Committed_AS: 10295144 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 16036668 kB
VmallocChunk: 34342840716 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6144 kB
DirectMap2M: 33540096 kB




Current ps output:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Dec19 0:09 [kthreadd]
root 3 0.0 0.0 0 0 ? S Dec19 0:00 [migration/0]

... lot of processes using zero RAM ...

root 926191 0.0 0.0 15932 3648 ? S<L 00:00 0:00 /usr/bin/atop -a -w /var/log/atop/atop_20151220 600
99 451245 0.0 0.0 194480 10616 ? S 11:00 0:00 /usr/local/apache/bin/httpd -DSSL
root 451246 0.0 0.0 76656 11444 ? S 11:00 0:00 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/bin/leechprotect
root 925345 0.0 0.0 24252 11972 ? S<L 00:00 0:05 /usr/bin/atop -a -w /var/log/atop/atop_20151220 600
root 12293 0.0 0.0 84356 17796 ? S Dec19 0:23 tailwatchd
99 925679 0.0 0.0 63328 18964 ? S 00:00 0:00 lfd TEXT messenger
99 925678 0.0 0.0 63328 19008 ? S 00:00 0:00 lfd HTML messenger
root 10711 0.0 0.0 194480 19184 ? Ss Dec19 0:06 /usr/local/apache/bin/httpd -DSSL
root 925673 0.0 0.0 61540 19840 ? Ss 00:00 0:20 lfd - sleeping
root 12233 0.0 0.0 110160 20316 ? S Dec19 0:00 dnsadmin - server mode
root 12177 0.0 0.0 107936 20512 ? S Dec19 0:02 cpsrvd (SSL) - waiting for connections
99 451344 0.3 0.0 2177140 20540 ? Sl 11:00 0:01 /usr/local/apache/bin/httpd -DSSL
99 451263 0.3 0.0 2177140 21204 ? Sl 11:00 0:01 /usr/local/apache/bin/httpd -DSSL
99 451262 0.3 0.0 2177184 21772 ? Sl 11:00 0:01 /usr/local/apache/bin/httpd -DSSL
99 451399 0.3 0.0 2178944 21792 ? Sl 11:00 0:01 /usr/local/apache/bin/httpd -DSSL
32004 453434 0.0 0.0 113144 22124 ? S 11:02 0:00 webmaild - serving 84.3.235.236
32004 449636 0.0 0.0 113532 22424 ? S 10:59 0:00 webmaild - serving 188.6.187.111
root 3609 0.0 0.0 361336 24772 ? Ssl Dec19 0:33 /usr/bin/pmxcfs
root 12791 0.0 0.0 132120 28016 ? S Dec19 0:00 cpdavd - accepting connections on 2077, 2078, 2079, and 2080
root 3896 0.2 0.1 208504 39404 ? Ss Dec19 3:50 pvestatd
root 3841 0.0 0.1 205452 39608 ? Ss Dec19 1:01 pve-firewall
25 10436 0.0 0.1 712756 46996 ? Ssl Dec19 0:39 /usr/sbin/named -u named
root 3873 0.0 0.1 271524 55968 ? Ss Dec19 0:01 pvedaemon
www-data 179856 0.0 0.1 282364 57372 ? S 06:25 0:00 spiceproxy worker
www-data 179878 0.0 0.1 282872 58620 ? S 06:25 0:00 pveproxy worker
www-data 179879 0.0 0.1 282872 58620 ? S 06:25 0:00 pveproxy worker
www-data 179880 0.0 0.1 282872 58620 ? S 06:25 0:00 pveproxy worker
root 3876 0.0 0.1 292912 62068 ? S Dec19 0:01 pvedaemon worker
root 3874 0.0 0.1 293028 62344 ? S Dec19 0:01 pvedaemon worker
www-data 3924 0.0 0.1 279620 62872 ? Ss Dec19 0:02 spiceproxy
root 3875 0.0 0.1 293848 63056 ? S Dec19 0:01 pvedaemon worker
www-data 3901 0.0 0.1 280528 63780 ? Ss Dec19 0:02 pveproxy
root 330715 0.0 0.2 202136 73168 ? Ss 08:58 0:02 /usr/local/cpanel/3rdparty/perl/514/bin/spamd --daemonize --max-spare=1 --max-children=3 --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid -i127.0.0.1
root 347417 0.4 0.2 225308 90364 ? S 09:15 0:27 spamd child
root 1036439 0.1 0.3 425280 113396 ? Ss 02:30 0:36 /opt/cpanel/ea-php54/root/usr/bin/php /backup/incremental/start.php
root 250925 39.1 1.0 426512 330876 ? TNsl 07:45 78:10 /usr/libexec/p7zip/7za a -t7z accounts-!-abc123.20151220074435.7z @filemtimelist-accounts-!-abc123.dat.filelist
498 31290 10.4 20.9 10742728 6910180 ? Sl Dec19 182:53 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/server.netmesterek.net.err --open-files-limit=8192 --pid-file=/var/lib/mysql/server.netmesterek.net.pid




Current top output:

top - 11:06:05 up 1 day, 5:46, 1 user, load average: 2.97, 2.91, 2.94
Tasks: 386 total, 1 running, 384 sleeping, 0 stopped, 1 zombie
%Cpu(s): 8.1 us, 1.8 sy, 2.2 ni, 76.6 id, 11.2 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem: 32916092 total, 31694392 used, 1221700 free, 2724 buffers
KiB Swap: 24117244 total, 2560 used, 24114684 free, 1396944 cached




PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
250925 root 39 19 416m 323m 1204 S 127.8 1.0 78:31.81 7za
607 root 0 -20 0 0 0 S 6.4 0.0 1:10.09 z_wr_int_6
31290 498 20 0 10.2g 6.6g 6348 S 6.4 21.0 183:01.11 mysqld
457046 540 20 0 323m 22m 10m S 6.4 0.1 0:00.07 php-cgi
457048 607 20 0 322m 21m 10m S 6.4 0.1 0:00.06 php-cgi
1 root 20 0 10604 840 704 S 0.0 0.0 0:01.34 init
2 root 20 0 0 0 0 S 0.0 0.0 0:09.70 kthreadd
3 root rt 0 0 0 0 S 0.0 0.0 0:00.55 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:02.78 ksoftirqd/0
5 root rt 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
6 root rt 0 0 0 0 S 0.0 0.0 0:00.08 watchdog/0
7 root rt 0 0 0 0 S 0.0 0.0 0:00.28 migration/1
8 root rt 0 0 0 0 S 0.0 0.0 0:00.00 stopper/1
9 root 20 0 0 0 0 S 0.0 0.0 0:01.09 ksoftirqd/1
10 root rt 0 0 0 0 S 0.0 0.0 0:00.05 watchdog/1
11 root rt 0 0 0 0 S 0.0 0.0 0:00.37 migration/2
12 root rt 0 0 0 0 S 0.0 0.0 0:00.00 stopper/2
13 root 20 0 0 0 0 S 0.0 0.0 0:00.75 ksoftirqd/2
14 root rt 0 0 0 0 S 0.0 0.0 0:00.05 watchdog/2
15 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/3
16 root rt 0 0 0 0 S 0.0 0.0 0:00.00 stopper/3
17 root 20 0 0 0 0 S 0.0 0.0 0:00.64 ksoftirqd/3
18 root rt 0 0 0 0 S 0.0 0.0 0:00.05 watchdog/3
19 root rt 0 0 0 0 S 0.0 0.0 0:01.03 migration/4
20 root rt 0 0 0 0 S 0.0 0.0 0:00.00 stopper/4
21 root 20 0 0 0 0 S 0.0 0.0 0:01.61 ksoftirqd/4
22 root rt 0 0 0 0 S 0.0 0.0 0:00.06 watchdog/4
23 root rt 0 0 0 0 S 0.0 0.0 0:02.01 migration/5
24 root rt 0 0 0 0 S 0.0 0.0 0:00.00 stopper/5
25 root 20 0 0 0 0 S 0.0 0.0 0:00.68 ksoftirqd/5
26 root rt 0 0 0 0 S 0.0 0.0 0:00.05 watchdog/5
27 root rt 0 0 0 0 S 0.0 0.0 0:01.73 migration/6
28 root rt 0 0 0 0 S 0.0 0.0 0:00.00 stopper/6
29 root 20 0 0 0 0 S 0.0 0.0 0:00.54 ksoftirqd/6
30 root rt 0 0 0 0 S 0.0 0.0 0:00.05 watchdog/6
...etc.

What is eating up RAM?

Many thanks,
Gergo
 
We've found the solution. ZFS ARC was used all RAM with its default settings. ZFS should but it does not automatically free allocated RAM when needed... So if anyone would like to use ZFS with Proxmox, ZFS ARC size should be limited, for example add these two lines to /etc/modprobe.d/zfs.conf (memory usage limit values should be adjusted to the amount of RAM, number of files, etc.):
options zfs zfs_arc_max=6442450944
options zfs zfs_arc_meta_limit=4294967296
 
You must log in or register to reply here.
Top Bottom