Process pmxcfs. High RAM and SWAP usage.

[ppo]

One of my cluster's node has been reported by zabbix as having Lack of free swap space:
Free swap space in % (pve1:system.swap.size[,pfree]): 9.98 %

What could cause this and how to resolve?
It seems that others nodes don't have such problem.

Spoiler: Node without problem

  PID USER  PR  NI  VIRT  RES  SHR S  %CPU %MEM  TIME+  COMMAND  SWAP
  2811 root  20  0  265m  53m 1128 S  0.0  0.7  3:50.19 pvedaemon  1404
 285169 root  20  0  287m  61m 6580 S  0.0  0.8  1:05.28 pvedaemon worke  1064
 284460 root  20  0  287m  61m 6660 S  0.0  0.8  1:14.94 pvedaemon worke  1044
 284965 root  20  0  287m  61m 6608 S  0.0  0.8  1:11.13 pvedaemon worke  1032
  2419 root  20  0  362m  33m  31m S  0.0  0.4 142:48.30 pmxcfs  796

Spoiler: Node with high ram and swap usage

  PID USER  PR  NI  VIRT  RES  SHR S  %CPU %MEM  TIME+  COMMAND  SWAP
  3011 root  20  0 6359m 1.8g  19m S  0.0 30.6 220:55.83 pmxcfs  4.2g
  3581 root  20  0 1403m 710m 1760 S  12.3 11.9  32619:14 kvm  330m
  4228 root  20  0 2558m 1.8g 1704 S  18.9 31.2  69696:24 kvm  183m
  3496 root  20  0  265m 2460 1584 S  0.0  0.0  4:26.07 pvedaemon  52m
1042198 root  20  0  288m  29m 4808 S  0.0  0.5  0:46.50 pvedaemon worke  31m

root@pve1:~# pveversion -v
proxmox-ve-2.6.32: 3.4-156 (running kernel: 2.6.32-39-pve)
pve-manager: 3.4-6 (running version: 3.4-6/102d4547)
pve-kernel-2.6.32-39-pve: 2.6.32-156
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-17
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

 

[fireon]

Can you use HTOP and sort by Memory. Idendify the VM that need so much memory, or the process.

Speicherpig is helping too:

#!/usr/bin/perl

use strict;
use warnings;
use Getopt::Long;
use Pod::Usage;
use constant DEFTYPE => 'size';

# pod/help {{{
=head1 NAME

speicherpig - sum up process memory usage based on process names

=head1 SYNOPSIS

speicherpig [--count] [--help] [--byname] [--reverse] [type]

=head1 OPTIONS

    --count     Show count of summed up processes
    --help      Show help and exit
    --byname    Order output by process names, not by sums
    --reverse   Reverse order
                                                                                                                                           
    type        Type of memory usage (passed to ps) to get                                                                                  
                process memory info.  Will use 'size' by                                                                                    
                default.  You may try 'rss'. Use 'vsize' to                                                                                
                include swap information.                                                                                                  
                                                                                                                                           
                Speicherpig expects size outputs in KiB.                                                                                    
                                                                                                                                           
=head1 AUTHOR                                                                                                                              
                                                                                                                                           
Written by Christoph 'Mehdorn' Weber                                                                                                        

=head1 REPORTING BUGS

Report bugs to <kontakt@das-mehdorn.de>

=head1 COPYRIGHT

Copyright 2010 Christoph 'Mehdorn' Weber.  License Creative Commons
Attribution-Share Alike 3.0 Germany
<http://creativecommons.org/licenses/by-sa/3.0/de/>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY.

=head1 TRIVIA

Speicherpig, speicherpig, does whatever a speicherpig does

=head1 SEE ALSO

ps(1), proc(5)

=cut
# }}}

sub pretty_print {
    my ($kbytes) = @_;

    return sprintf "%7.2f GiB", $kbytes/1024/1024 if $kbytes > 1024*1024;
    return sprintf "%7.2f MiB", $kbytes/1024      if $kbytes > 1024;
    return sprintf "%7.2f KiB", $kbytes;
}

# option/parameter handling {{{
# get options
my $help       = 0;
my $order_name = 0;
my $reverse    = 0;
my $with_count = 0;
GetOptions(
    'byname'  => \$order_name,
    'count'   => \$with_count,
    'help'    => \$help,
    'reverse' => \$reverse,
) or pod2usage(-exitval => 1);
pod2usage(-exitval => 0, -verbose => 2) if $help;

# get the only supported parameter
my ($type) = @ARGV;
$type = 'rss' unless $type;
# }}}

# get process list from ps {{{
open(my $ps, 'ps --no-headers -eo '. $type .',comm |') or
    die 'Cannot run ps: ', $!;
my @proclist = <$ps>;
close($ps) or die 'Cannot close ps: ', $!;
chomp @proclist;
# }}}

# sum up information {{{
my %procsums;
my %proccounts;
foreach my $line (@proclist) {
    my ($size, $name) = $line =~ m{^ *(\d+) (.*)$}og;
    $procsums{$name} += $size;
    $proccounts{$name}++ if $with_count;
}
# }}}

# sort output {{{
my @key_order;
if ($order_name) {
    @key_order = sort keys %procsums;
} else {
    @key_order = sort { $procsums{$a} <=> $procsums{$b} } sort keys %procsums;
}
@key_order = reverse @key_order if $reverse;
# }}}

# show output {{{
foreach my $name (@key_order) {
    # don't show empty sets
    next unless $procsums{$name};

    printf "%16s: %s\n",
        pretty_print($procsums{$name}),
        $with_count ? $name .' ('. $proccounts{$name} .')' : $name
}
# }}}