Samba under PVE 8 sometimes extremely slow

[meyergru]

I observe a problem on one of my Windows clients since upgrade from PVE 7.4 to PVE 8:

I have defined Samba shares on local ZFS filesystems. The filesystems are also exported to VMs via NFSv4.

When I access these shares from Windows clients, sometimes the transfer speeds are extremely slow (<10 MBit/s). This is best shown when I open an Excel file and save it again. The first few times, this will be just seconds. When I leave the file for one minute and re-save, it takes a few seconds, then a popup shows up with a progress indicator.
Sometimes, after a while, the system speeds up again. I can also copy files back and forth and see slow speeds.

I have tried to eliminate possible other causes up to the point to deinstall my antivirus software. I can rule out network problems because I tested with iperf3 during these "slow" times and I can see no problem there. There is neither high CPU nor network load on either the server nor the client.

However, I am quite sure that it is the Samba daemon as when I restart it via "systemctl restart smbd", the next save is always fast, even when the file in Excel is still open.
Also, when I share the very same folder via NFSv4 to an Ubuntu 22.04 VM and share that via Samba, the problem does not happen.

I can see no obvious problems with the Samba config, I even removed anything pertaining to ZFS snapshots and tried disabling oplocks to no avail.

I am at a complete loss as to what is the problem here, but it started with the PVE upgrade.

Does anyone have an idea?

 

[meyergru]

I found the problem, although that solves nothing for me as my ISP has DS-Lite with CGNAT only:

I tried to configure IPv6 by hand and allowing router advertisements to assign addresses. This seems to have a side-effect on Samba to look up workgroups and sids via IPv6, which obviously fails. This was harming Samba by blocking it for a few seconds now and then, causing the slow speeds. FWIW, disabling IPv6 fixes the problem.

Note: This was no problem before the upgrade to PVE8, so something must have changed in Samba.

I am still waiting for the promised fix that adds router advertisements plus DHCP client role to Proxmox. Currently, one can only assign a fixed IPv4 and fixed IPv6 via Proxmox provided means.

As it looks, enabling IPv6 can have a few more adverse side-effects. One could enable Samba on IPv4 interface adresses, but that works with fixed IPs only.

 

[meyergru]

On another observation, I also had this problem with a machine that did not have IPv6 configured. As soon as I used "bind interfaces only = yes" and configured only a specific IPv4 subnet, the problem went away. Both of the affected machines had multiple (i.e. 4) physical interfaces, so it seems that binding Samba to all of those plus the TAP interfaces implicitely leads to timeouts for naming requests which can stall data transfers. I did not have the problem on a third box with less network interfaces.

This could partly explain why this problem has not been widely experienced. To trigger it, one needs to have;

1. A newer Samba version (like that from Debian 12)
2. A lot of interfaces (like with Proxmox)
3. Samba as a file server (which is not standard for Proxmox)

So neither Debian 12 alone nor Proxmox 8 are prone to have this problem usually, but my specific use-case triggered this.

 

[Mr.Goodcat]

On another observation, I also had this problem with a machine that did not have IPv6 configured. As soon as I used "bind interfaces only = yes" and configured only a specific IPv4 subnet, the problem went away. Both of the affected machines had multiple (i.e. 4) physical interfaces, so it seems that binding Samba to all of those plus the TAP interfaces implicitely leads to timeouts for naming requests which can stall data transfers. I did not have the problem on a third box with less network interfaces.

This could partly explain why this problem has not been widely experienced. To trigger it, one needs to have;

1. A newer Samba version (like that from Debian 12)
2. A lot of interfaces (like with Proxmox)
3. Samba as a file server (which is not standard for Proxmox)

So neither Debian 12 alone nor Proxmox 8 are prone to have this problem usually, but my specific use-case triggered this.

I'm having the same issue. Did you find another solution other than limiting interfaces? Unfortunately that's not a viable solution in my case. Or do you maybe have a pointer where to look to debug this (e.g. where did you see those timeouts)?

 

[meyergru]

You can see the timeouts in the samba logs if you increase verbosity. And no, I did not find any better solution that to limit the interfaces.

 

[UgenPorter]

I'm facing a similar issue with a guest (Ubuntu Server 22.04) which holds my SMB shares - any solution so far?

 

[meyergru]

Limit the interfaces with "bind interfaces only = yes" and "interface = ....". Using vmbr0 did not help in my case, I had to specify an IPv4.

 

[UgenPorter]

Ok I'll try that. Which drawbacks I have? The server is actually only reachable from the one NIC and in the specific network?

 

[meyergru]

Yes. But you could well enumerate multiple interfaces / IPs.

 

[UgenPorter]

Seems to be the speed on my Windows client increased from 10MBps to 240MBps, but still slow. The speed on my linux clients is 1GBps.

And how can we explain, that when I write from my Windows client to my SMB share, that I have a write speed of nearly constant 100MB/s...

 

[meyergru]

Many variables come into play for that:

1. Samba tuning parameters like TCP_NODELAY, multi channel support, minimal logging, These would affect Linux samba servers on bare metal as well. Some of these parameters affect different client types in different ways, e.g. depending on whether they connect via SMB1, 2 or 3 dialect.

2. You have a VM, though, so first you should verify that your network throuhput is decent via iperf. That could depend on how you configure your virtual NICs: emulation type (e.g. virtio, E1000 ...). Depending on were your Linux clienst are (also VMs or physical devices?), they may be faster than a probably physical Windows client.

3. Also relevant for VMs: How is your underlying storage connected, what I/IO throughput can you actually get? NFS? With what version / parameters? ZFS? Is known to be slow on writes. This should not depend on the client type, however.

 

[UgenPorter]

1. I've seen a difference of 30% when I'm mounting the shares as SMB3 under a linux client, but for a Windows client sadly I can not choose the SMB version.

2. Done - with iperf3 the speed between the VM<->VM, VM<->Host, VM<->other Windows/Linux clients, Host<->other Windows/Linux clients. Everything as expected... round about 1GBps.

3. Speed between the HDDs is perfect also between Linux client and Ubuntu Server VM. It is a EXT4 storage.

 

[gustcol]

same problem here. iperf3 25Gbps -- samba - 6Gbps.

 

[UgenPorter]

I've switched from SMB to NFS and now it is even better, but in the end SMB it is a problem in the protocol which can not handle the virtualization environment.

 

[ck_reeses]

I observed the same problem in ProxMox 7.4 VM guest with Linux 8,9.
I also observed that the problem is related to Samba! Because I disabled the "nmb" service and the I/O performance is getting better. Whenever I see the file transfer performance slow down from Windows client (Windows 10) to the Samba share, simply restart the "smb" service (without OS reboot) in Linux and the performance bounces back!

 

[dlasher]

ARRRRRGGGGHHH!! I'm not the only one!!

I'm running two different LXC containers (bind mounts) running Samba -- I've been trying to figure out why the performance has not only gone in the toilet since the 8.x upgrade, but seems to randomly STOP for 30 seconds, then restart.. repeatedly. I've spent hours troubleshooting the network, the disk, etc, and haven't been able to find the solution.

I've built new VM's, used alpine, ubuntu, debian, all have the same issues.

 

[macdet]

ARRRRRGGGGHHH!! I'm not the only one!!

..

I've built new VM's, used alpine, ubuntu, debian, all have the same issues.

thx - this saves my time but now?

 

[dlasher]

I don't know if this : https://forum.proxmox.com/threads/very-poor-lxc-network-performance.139701/

applies - but if I could figure out what they were suggesting, I would give it a try.

 

[ck_reeses]

I don't know if this : https://forum.proxmox.com/threads/very-poor-lxc-network-performance.139701/

applies - but if I could figure out what they were suggesting, I would give it a try.

In the case you mentioned, it is about a performance issue running in LXC (container). In this chat, we are discussing the performance issue of Samba running in VM.

LXC container is sharing the kernel (including network) with ProxMox host!

 

[Defy9673]

Any updates on this issue? Trying to get decent speed when deploying a truenas scale vm behind an opnsense vm. Only solution is to give truenas an interface link to the proxmox interface directly. Tried searching from the opnsense side, but seems to be a vm/smb issue.