Moving ESXi to Proxmox: slow SMB performance from Windows Guest

[jrosseel]

I am experiencing performance issues with SMB file sharing from a Windows VM guest on Proxmox.
I have both a newly configured VM, as well as a VM that was migrated from ESXi.

On the ESXi machine, I easily got wire speed. Meaning: 112MB/sec. which is normal for a 1Gbps interface.

I do not get near that performance on my new Proxmox server, which is ultimately more powerful.

The problem is isolated to SMB read speed (download from VM to regular PC). The tests relate to large file transfers (26 files for 72GB)

hardware: VM disk I'm reading/writing from is on a RAIDZ pool with 4 spinning disks, Proxmox server is Epyc Rome 16 core, 256GB RAM, 10Gbps interface
Client machine that is accessing the server has 2.5GbE interface. So I expect performance somewhere in the 200-250MB/sec region

Tests done:

• ethernet level tests:
  • iperf tests: wire speed when I use more than 1 parallel thread
  • OpenSpeedTest: around 2000gbps
  • So the ethernet drivers (server and client) are OK
• local file read: speeds vary from 200-1000MB/sec.
  • so the disk access is fast enough for read
• FTP server on the VM guest:
  • read: 200-220MB/sec both for sequential (one file at a time) or parallel (5 streams) access
  • write to the VM: slower, initially fast, then levels out to 70-100MB/sec (both sequential and parallel). Not optimal, but doable.
• SMB performance
  • read: levels out at around 20MB/sec
  • write: Starts around wire speed, levels out to around 70-100MB/sec average just like the FTP

Write speed are lower than I hoped for, but that's what I get for a RAIDZ pool, I guess. I can live with that,

What bothers me is the slow read access with SMB. This was way better with the Windows VM on ESXi. I'm getting 20, where I was hoping for >200. And given local file and FTP read performance, that is not an unrealistic expectation. It's really linked to SMB on Windows on Proxmox, as the other tests are fine.

Any thoughts?

Windows guest or Proxmox host tunings I should be looking into?

 

[Kingneutron]

Are you using virtio + multiqueue for the VM NIC?

 

[jrosseel]

Are you using virtio + multiqueue for the VM NIC?

Yes, and no/yes. Meaning: was always on virtio but wasn't using multiqueue, switched it on - no difference.

I've done further digging, and it's probably not Proxmox related. Wireshark traces show quite some packet loss. This seems to disturb SMB more than other protocols. After some packet loss occurred, SMB seems to "freeze" for about 30milliseconds, before continuing. This is very consistent (see attached screenshot).

Packet loss is something I almost never had in my old situation, But then, I was using a server with a 1Gbps interface, towards a switch with all 1Gbps ports. So everything was balanced, and the server could never send more than 1Gbps, and this was automatically traffic limiting.

New situation: 10Gbps interface to server (and switch), and from that switch 2.5Gbps to my PC. So the server can send faster than my PC can receive, and the switch will have to drop packets. This triggers TCP retransmissions. TCP is normally quite good at throttling traffic based on this. But SMB seems also wanting to throttle, and in a much more brutal way. Strange, but could explain why people always believe "SMB is slow".

I'll do some further digging, but I think Proxmox is off the hook here.

 

[Kingneutron]

Interesting. I never ran into packet loss, but I physically isolated all 3 of my network speeds - 10Gbit, 2.5Gbit and 1Gbit - and they are all on a /24 netmask. I use 172.16.25 and 172.16.10 to separate them as well, so none of them conflict; they only "see" each other at the Proxmox host and they all have dedicated same-speed switches (except for the 10Gbit, which is point-to-point, MTU 9000, and only on 3 boxes)

Can you upgrade the PC to 10Gbit, or put the 2.5 on a direct / dedicated 2.5 NIC (or usb3 adapter) on the server so it's bypassing the switch?

 

[jrosseel]

Interesting. I never ran into packet loss, but I physically isolated all 3 of my network speeds - 10Gbit, 2.5Gbit and 1Gbit - and they are all on a /24 netmask. I use 172.16.25 and 172.16.10 to separate them as well, so none of them conflict; they only "see" each other at the Proxmox host and they all have dedicated same-speed switches (except for the 10Gbit, which is point-to-point, MTU 9000, and only on 3 boxes)

Can you upgrade the PC to 10Gbit, or put the 2.5 on a direct / dedicated 2.5 NIC (or usb3 adapter) on the server so it's bypassing the switch?

I enabled rx/tx flow control on the 10gbps port on the switch (it was enabled on the adapter in the Proxmox server).
That gives back pressure on the 10Gbps link when the switch buffers are full. Not ideal as this affects all outgoing 2.5gbps interfaces.

Not ideal, but it helped. For anyone interested, some details on performance tests/analysis I did:

Test with 26 large files, 72GB total.
Ran the tests from an SSD based virtual disk, and from the RAIDZ array.

• From SSD: mostly 230-240MB/sec, sometimes some drop, but not dramatic. Disabling SMB MultiChannel made this drop to 180-190, so MultiChannel on (default for newer Windows OSes and SMBv3) is good for performance
• From RAIDZ spinning rust: constant, around 110MB/sec with my test program, where I get 150-170 if I run the same program locally on the VM. This difference is likely due to different block sizes. SMB works in 256KB chunks, my test program in 16MB chunks. When I stop the test after a few files, and run it again, files come out of (ZFS) cache, and I get 230-240MB/sec initially.

In both of these cases there are still some of these 30 milliseconds where SMB "freezes" due to some packet loss. But it doesn't affect performance too much. It's *way* better than before. See screenshot: first ramp is files out of cache, and then from the disk. Few small pauses (packet loss), but overall acceptable. No TCP (slow) restarts, ramp is wire speed. Speed of first slope is wire speed: 64MB/sec payload until first pause, but there are 4 parallel streams like this. So effectively very close to 2.5Gbps taking into account packet overhead.

Conclusion: it was a network issue, and it is largely solved. I was just expecting a bit better from a RAIDZ HDD array in terms of performance, but that is probably unrealistic.