[SOLVED] Proxmox and ram /swap usage

[Valerio Pachera]

I know there's an open topic about zfs that was killing servers for using to much ram.
Right now I'm running proxmox 5.1-43 on a "new" dell R630 server.
Only 3 guests are running.
They have assinged in total 28G of ram.
The server has 48G of ram. I see more than 5G of swap are used.

Do you consider that normal?

By iotop I there's always a z_null_int process at 99%.

free -h
              total        used        free      shared  buff/cache   available
Mem:            47G         21G        1.8G         52M         24G         25G
Swap:          8.0G        5.2G        2.8G


top -o %M

Tasks: 745 total,   1 running, 744 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  0.4 sy,  0.0 ni, 99.2 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 49355708 total,  1935812 free, 22199124 used, 25220772 buff/cache
KiB Swap:  8388604 total,  2918004 free,  5470600 used. 26497116 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                     
 3337 root      20   0 16.892g 0.012t   4836 S   4.0 26.0 352:01.69 kvm                                                                                                                         
31915 root      20   0 9523956 7.346g   4852 S   7.3 15.6   1821:18 kvm                                                                                                                         
 5970 root      20   0 5093528 3.724g   4956 S   8.6  7.9 840:01.19 kvm                                                                                                                         
12547 root       0 -20   58624  41692   3540 S   0.0  0.1   0:55.33 atop                                                                                                                       
 1571 root      20   0  230660  39980  39812 S   0.0  0.1   0:55.14 systemd-journal                                                                                                             
18786 www-data  20   0  557840  39468   9352 S   0.0  0.1   0:04.94 pveproxy worker                                                                                                             
18789 www-data  20   0  545976  34644   9932 S   0.3  0.1   0:05.49 pveproxy worker                                                                                                             
18790 www-data  20   0  545552  31520   9212 S   0.0  0.1   0:04.57 pveproxy worker                                                                                                             
 4620 root      20   0  316216  30924   7200 S   0.0  0.1 173:04.87 pvestatd                                                                                                                   
 4419 root      20   0  544080  27596   8612 S   0.0  0.1   0:06.32 pvedaemon worke                                                                                                             
 4803 root      20   0  325660  27156   6476 S   0.0  0.1   4:21.43 pve-ha-lrm                                                                                                                 
 4619 root      20   0  317804  26420   7028 S   0.0  0.1  38:16.63 pve-firewall                                                                                                               
 4506 root      20   0  686216  26028  20884 S   0.0  0.1  15:05.05 pmxcfs                                                                                                                     
20158 root      20   0  538024  25580   8444 S   0.0  0.1   0:07.52 pvedaemon worke                                                                                                             
25864 root      20   0  537636  24188   8396 S   0.0  0.0   0:08.66 pvedaemon worke                                                                                                             
 4805 www-data  20   0  535164  15824  11884 S   0.0  0.0   0:31.89 pveproxy                                                                                                                   
 4828 www-data  20   0  302496  15304  10432 S   0.0  0.0   0:26.97 spiceproxy                                                                                                                 
18817 www-data  20   0  304832  10376   5328 S   0.0  0.0   0:04.11 spiceproxy work

I don't see hight I/O by atop.

So I wonder if I have to be warry about swap usage as warning indicator of something going wrong.

Thank you.

 

[Tomas Waldow]

Set in /etc/sysctl.conf:

vm.swappiness = 0

The kernel will swap only to avoid an out of memory condition.

Run: sysctl -p to activate.

 

[Vasu Sreekumar]

A node with 3 or more guests, if you see swap usage, that will slow down everything.

We need to avoid swap. vm.swappiness = 0 is the best way.

I even recommend adding more RAM and set swap to 0 in LXC guests.

 

[Valerio Pachera]

Hi, today I've been adding a new guest to the same server.
The guest had 4G of ram and 2 cores.
As soon as I start the guest, the server starts swapping and the guest is so slow, that I have to kill it.

It's absurde such behavior since I still have plenty of free ram.
As of now I have not changed the swappiness but this issue isn't related to swappiness but to zfs:

free -h
              total        used        free      shared  buff/cache   available
Mem:            47G         27G         19G         26M        266M         19G
Swap:          8.0G        2.3G        5.7G

I found one of my previous posts:
https://forum.proxmox.com/threads/host-swapping-when-it-shouldnt.36690/#post-180323

I'm suprised such problem han't been alrady detected and corrected.

 

[tjh]

You should read this.
What's you're seeing isn't a problem, even though you assume it is.

Basically, Linux is moving your memory that hasn't changed/been modified in a while to swap, and using your actual physical RAM for cache/active memory. This is almost always a good thing for performance. There's no point having 3G of RAM used up by a database that is never actually read, so it's moved to swap and the freed memory is then used for caching data so it's quickly accessible.

If you really feel that you're being "ripped off" then change vm.swappiness to 0 and you'll only swap when it's about to go pear shaped.

 

[r.jochum]

You should have a look at this for "vm.swappiness" and proxmox:

https://pve.proxmox.com/pve-docs/chapter-sysadmin.html

 

[Valerio Pachera]

I think hat zfs_arc_max is what causes the abuse of ram.
I'm going to set it to 4G.

 

[Valerio Pachera]

...ok, I see

ZFS depends heavily on memory, so you need at least 8GB to start

 

[Vasu Sreekumar]

I use 4GB for zfs_arc_max

I have no issues, i have 25+ nodes running with 5 to 10 guests.

 

[Valerio Pachera]

I use 4GB for zfs_arc_max

I have no issues, i have 25+ nodes running with 5 to 10 guests.

Thank you for thi info!

 

[fortechitsolutions]

For what it is worth, my general feeling for ZFS and proxmox, is that - you should have a really good reason to use ZFS, otherwise I don't think it would be my recommended pick for "simple,easy,works well". But possibly there is an "SOP" somewhere on how to tweak-tune proxmox so that ZFS becomes as problem-free as vanilla install? (? ie, things like setting 4GB for zfs_arc_max; etc ?)

Admittedly, I only played with ZFS Proxmox ~a few years ago and at that time my general feeling was "I prefer to do unsupported debian manual SW Raid install and add proxmox" rather than "proxmox appliance installer with RaidZ/2 drive ZFS mirror".. But that was just me

Tim