smbd memory leak ?

RolandK

Renowned Member
Mar 5, 2019
962
191
88
51
hello, i'm trying to setup 10gigE with samba for backup purpose. debian11 + zfs 2.1.5 + samba 4.13.13

while it runs fine with 1gigE , our new cluster (connecting with smb3_11 in smbstatus) shows something wich looks like a memory leak to me.

during backup, for all related smbd processes ram usage is skyrocketing to high gigabyte levels and then oomkiller kills all of them

has anybody seen something like this ?

it looks as if smbd would largely buffer writes from the network and is not fast enough to write that down to disk, the network (6 cluster nodes are writing in parallel) should deliver data faster as the debian system can write data to disks.

maybe 1gigE is a good "throttle" , so that may be the reason why the problem does show with 1gigE


Code:
top - 15:30:25 up 18 min,  1 user,  load average: 1,48, 10,69, 20,71
Tasks: 238 total,   3 running, 235 sleeping,   0 stopped,   0 zombie
%CPU(s):  2,2 us,  9,7 sy,  0,0 ni, 82,3 id,  3,1 wa,  0,0 hi,  2,1 si,  0,5 st
MiB Spch:  16008,1 total,    736,7 free,  15160,6 used,    110,9 buff/cache
MiB Swap:    975,0 total,    909,3 free,     65,7 used.    626,1 avail Spch

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     ZEIT+ BEFEHL
 104865 pve-clu+  20   0   10,7g   6,7g  11288 S   0,0  42,7   0:38.33 smbd       <--- !!!
 104871 pve-clu+  20   0 1529112  50808  10516 S  15,6   0,3   1:46.78 smbd
    296 root      20   0   95036  40140  39556 S   0,3   0,2   0:05.41 systemd-journal
 104864 pve-clu+  20   0  533700  28232  10524 R  13,6   0,2   1:50.41 smbd
 104870 pve-clu+  20   0  537812  25780  10480 S  14,0   0,2   1:51.19 smbd
 104859 pve-clu+  20   0  718012  25624  10524 S  15,6   0,2   1:56.02 smbd
 104872 pve-clu+  20   0  668884  25528  10412 S  15,3   0,2   1:56.10 smbd
 104857 pve-clu+  20   0  861468  22068  10452 R  13,6   0,1   1:51.22 smbd
 104869 pve-clu+  20   0   99464  13236  10044 S   0,0   0,1   0:00.36 smbd
 104862 pve-clu+  20   0   83012  13200  10104 S   0,0   0,1   0:00.46 smbd
 104867 pve-clu+  20   0   83012  13192  10108 S   0,0   0,1   0:00.47 smbd
 104860 pve-clu+  20   0   83172  13136  10064 S   0,0   0,1   0:00.50 smbd
 104861 pve-clu+  20   0   83012  12992   9900 S   0,7   0,1   0:00.45 smbd
 104858 pve-clu+  20   0   83012  12976   9884 S   0,0   0,1   0:00.44 smbd
 104863 pve-clu+  20   0   83012  12972   9904 S   0,3   0,1   0:00.46 smbd
 
Last edited:
https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html -> Default: oplocks = yes

https://superuser.com/questions/1481407/smb-cifs-oplocks-where-is-the-danger


"To answer your first question: Opportunistic locking is "dangerous" because it gives the user working on a client the feedback that his work has been saved when in reality that saving was done only against the local caching inside the client. I.e. when the client crashes or the network connection to the server experiences problems after the user has received feedback indicating his save operation was successful, he can still loose his work/edits, even if the server implements opportunistic locks perfectly."

i wonder, why samba has dangerous default settings

i remeber i have seen these errors mentioned in the article a LOT in the past:

[15066.212283] CIFS VFS: Close unmatched open
 

Did that work? Because I had this problem after moving from truenas core to truenas scale.
As soon as verification of any backup started, smbd started eating memory on the storage server (TrueNAS SCALE 24.04.2) at an alarming rate.

I managed to deactivate oplocks on the share, which is indeed on for the linux version, but that didn't change a thing. It turned out what I was doing wrong was to mount the samba share from within an LXC container, where I had installed PBS. Everything worked fine with TrueNAS Core regardless but this configuration went bananas when I upgraded to SCALE.

I went back to virtual machine instead, and no memory leak, even with SCALE.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!