Memory leakage of one of my cluster nodes

jensie · Jun 20, 2024

I have a cluster of 3 machines, of which 2 are somewhat identical. All running pve-manager/8.2.4/faa83925c9641325
The one that has no VMs running actively on it, has a memory usage of approx 50%.
Whereas the other one with 2 VMs, has approx 18%.
Something is clearly wrong but don't know how to find out.

Moayad · Jun 24, 2024

Hi,

Is the 50% memory usage have ZFS storage?

jensie · Jun 24, 2024

Hello,
Thank you for your reply. Yes ZFS. But both are identical servers - how can I check if all is going as expected ?

Moayad · Jun 25, 2024

May you post the output of `top` command?

justinclift · Jun 25, 2024

jensie said:
But both are identical servers - how can I check if all is going as expected ?

Please run the command arcstat on the terminal of all 3 servers, and post the results here.

For example, here's the output from one of mine:

Bash:

# arcstat
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
11:54:14     0       0     0       0     0      0    0    16G    16G   167G

That shows 16GB of ram in use by ZFS cache on that server, which is the maximum amount that server is configured to use (as ZFS cache).

justinclift · Jun 25, 2024

Moayad said:
May you post the output of `top` command?

Maybe htop instead of top?

justinclift · Jun 25, 2024

jensie said:
The one that has no VMs running actively on it, has a memory usage of approx 50%.

Would that node happen to have an Nvidia graphics card physically installed in it?

There was a bug report recently where an Nvidia graphics card (GTX980 I think?) was somehow causing a memory leak. Probably through the noveau kernel module / driver.

jensie · Jun 30, 2024

justinclift said:
Please run the command arcstat on the terminal of all 3 servers, and post the results here.

For example, here's the output from one of mine:

Bash:

# arcstat time read ddread ddh% dmread dmh% pread ph% size c avail 11:54:14 0 0 0 0 0 0 0 16G 16G 167G

That shows 16GB of ram in use by ZFS cache on that server, which is the maximum amount that server is configured to use (as ZFS cache).

Code:

pve1
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
09:19:23     0       0     0       0     0      0    0    50G    53G    56G

pve2
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
09:18:51     0       0     0       0     0      0    0   7.2G   7.3G    15G

pve3
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
09:19:42     0       0     0       0     0      0    0   7.2G    15G    19G

jensie · Jun 30, 2024

Moayad said:
May you post the output of `top` command?

Here you have the output for pve3

Code:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                     
   1824 root      rt   0  560176 167292  52788 S   1.0   0.5 151:59.77 corosync                                   
   1858 root      20   0  156080 117936  17664 S   1.0   0.4  67:19.41 pvestatd                                   
3478043 root      20   0 3240576 629728  14080 S   1.0   1.9   4:30.26 kvm                                         
1929439 root      20   0  613052  67376  49316 S   0.3   0.2  13:27.93 pmxcfs                                     
1937722 root      20   0  220232 114144   4992 S   0.3   0.4   1:26.82 pve-ha-lrm                                 
3783881 www-data  20   0  245196 146188   7552 S   0.3   0.4   0:00.82 pveproxy worker                             
3787174 root      20   0  243584 143732   6656 S   0.3   0.4   0:00.74 pvedaemon worke                             
      1 root      20   0  170148  10496   5632 S   0.0   0.0   0:33.75 systemd                                     
      2 root      20   0       0      0      0 S   0.0   0.0   0:03.22 kthreadd                                   
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqueue_release                     
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_g                             
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_p                             
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-slub_                             
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-netns                             
      9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri                 
     12 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-mm_pe                             
     13 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_kthread                           
     14 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_rude_kthread                     
     15 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_trace_kthread                     
     16 root      20   0       0      0      0 S   0.0   0.0   0:06.76 ksoftirqd/0                                 
     17 root      20   0       0      0      0 I   0.0   0.0   4:15.54 rcu_preempt                                 
     18 root      rt   0       0      0      0 S   0.0   0.0   0:07.92 migration/0                                 
     19 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/0                               
     20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0                                     
     21 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2

pve1

Code:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                     
   3247 root      20   0 9664468   7.8g  14864 S  24.6   6.3     60,57 kvm                                         
2417543 100000    20   0 7527832 478192  55336 S   3.7   0.4  84:21.84 node                                       
2417417 100000    20   0   11.6g 457424  61972 S   2.0   0.3   7:23.18 node                                       
3302581 www-data  20   0  246156 148376   8960 S   1.0   0.1   0:01.50 pveproxy worker                             
   1861 root      rt   0  559424 166388  52572 S   0.7   0.1  87:31.50 corosync                                   
    569 root      20   0       0      0      0 S   0.3   0.0   1:57.68 txg_sync                                   
3181420 100000    20   0 4276912 994300 352016 S   0.3   0.8   0:58.20 python3                                     
3302462 www-data  20   0  249432 154648  11904 S   0.3   0.1   0:01.35 pveproxy worker                             
3303345 www-data  20   0  250048 154360  11008 S   0.3   0.1   0:01.34 pveproxy worker                             
3308003 root      20   0   12960   5376   3328 R   0.3   0.0   0:00.02 top                                         
      1 root      20   0  170136  10240   5504 S   0.0   0.0   0:18.18 systemd                                     
      2 root      20   0       0      0      0 S   0.0   0.0   0:27.67 kthreadd                                   
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqueue_release                     
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_g                             
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_p                             
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-slub_                             
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-netns                             
      9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri                 
     12 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-mm_pe                             
     13 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_kthread                           
     14 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_rude_kthread                     
     15 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_trace_kthread                     
     16 root      20   0       0      0      0 S   0.0   0.0   0:04.96 ksoftirqd/0                                 
     17 root      20   0       0      0      0 I   0.0   0.0   1:39.63 rcu_preempt                                 
     18 root      rt   0       0      0      0 S   0.0   0.0   0:11.56 migration/0

pve2

Code:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                     
   1951 root      20   0  163436 108344   8064 S   1.7   0.3  51:13.93 pvestatd                                   
   1917 root      rt   0  559420 166428  52792 S   1.0   0.5  90:45.81 corosync                                   
 395602 root      20   0 4921472   2.3g   8064 S   1.0   7.4  15:29.52 kvm                                         
 963584 1000      20   0 3701948 271076  98688 S   1.0   0.8  15:16.85 beam.smp                                   
   1996 root      20   0  220272 114144   4992 S   0.7   0.3   1:59.05 pve-ha-lrm                                 
 963733 1000      20   0 3705388 267092  98688 S   0.7   0.8   8:55.56 beam.smp                                   
    470 root       0 -20       0      0      0 S   0.3   0.0   0:23.32 z_wr_int_0                                 
   1818 root      20   0  631556  66976  55108 S   0.3   0.2  12:21.35 pmxcfs                                     
   3629 root      20   0 2021284  35712  16000 S   0.3   0.1   4:13.62 containerd                                 
   5263 root      20   0 1272140  28752  14976 S   0.3   0.1   1:30.40 diun                                       
   5722 root      20   0  275676  42952  16768 S   0.3   0.1   0:28.13 node                                       
1151030 65532     20   0 1261060  28132  12544 S   0.3   0.1   3:43.12 cloudflared                                 
1711786 root      20   0   12960   5376   3328 R   0.3   0.0   0:00.01 top                                         
3275798 root      20   0 3310264 602496  14336 S   0.3   1.8   2:31.64 kvm                                         
      1 root      20   0  168612   8832   5376 S   0.0   0.0   0:11.43 systemd                                     
      2 root      20   0       0      0      0 S   0.0   0.0   0:15.56 kthreadd                                   
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqueue_release                     
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_g                             
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_p                             
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-slub_                             
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-netns                             
      9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri                 
     12 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-mm_pe                             
     13 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_kthread                           
     14 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_rude_kthread

justinclift · Jun 30, 2024

k, yep that's showing the pve1 has 50GB of ram used as ZFS ARC cache:

Code:

pve1
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
09:19:23     0       0     0       0     0      0    0    50G    53G    56G

There's a runtime value you can check to see what the maximum configured setting is on a given system:

Bash:

# cat /sys/module/zfs/parameters/zfs_arc_max
17179869184

You can change that setting dynamically:

Bash:

# echo 1073741824 > /sys/module/zfs/parameters/zfs_arc_max

And you can set a persistent value by creating text file under /etc/modprobe.d/ with the desired value:

Bash:

# cat /etc/modprobe.d/zfs.conf
# 2024.03.16 JC Limit ZFS ARC to 2GB max (needs min set first for it to work with values this low)
options zfs zfs_arc_min=1073741824
options zfs zfs_arc_max=2147483648

There's a bunch of info about this stuff online, so doing some searching should show up a bunch of exact instructions if you need for detail.

Search

Search

Memory leakage of one of my cluster nodes

jensie

Member

Moayad

Proxmox Staff Member

jensie

Member

Moayad

Proxmox Staff Member

justinclift

Active Member

justinclift

Active Member

justinclift

Active Member

jensie

Member

jensie

Member

justinclift

Active Member