OsX VM unable to write data to NAS properly

B

borisko

Member
Jun 25, 2019
21
0
21
37
Hi everyone,

I am at an impass. After several iperf3 tests and Blackmagic write/reads, I realized all connections work fine from all the computers at home, except uploads/writes from my Macintosh VM to the NAS (freenas).

As you will see below, I've ran tests from all points (Proxmox node itself to and from several machines including over wifi and ethernet, machines to and from the NAS, etc) and the only thing that doesn't work is from my Mac VM to the NAS.

It feels like the data stream gets saturated when I do so. It starts writing/sending, and everything related to internet begins to slow down: Freenas GUI unresponsive, write stops, etc. You can see through the iperf tests that a first packet is sent, then 0, 0, 0... etc until the end of the test.

I have not set jumbo frames anywhere, tried activating NAT acceleration on my router (ASUS Ac-88u, which is supposed to be a very capable router) and ethernet cable works fine as obviously showing with the proxmox server read/write tests.

Is vmxnet3 the bottleneck? Could there be a setting I missed, or a conflict between Proxmox routing the data and the router receiving it? I am pretty sure I have a double NAT setup, but since there isn't any issue with other machines except my VM, that seems unlikely.

Last thing is that I have set up Syncthing bterween the Mac VM and the NAS a few days ago. I am unsure I had those problems before setting that up (it's a brand new build for both) but uninstalling the whole syncthing dependencies on my mac VM and turning off the jail in freenas did nothing, so I have a feeling the issue is my proxmox setup (correct me if I'm wrong of course! I'm a noob and here to learn).


My machines for this test:
NAS = Freenas machine (LAN connection, 192.168.1.45)
Server = Proxmox machine (Proxmox 5.4, LAN connection, 192.168.1.191)
MacOSvm = My Mac VM (Mojave 10.14.4, VMXNet3 ethernet, 192.168.1.229)
iMacWifi = authentic iMac (Mojave 10.14.5, over wifi 5ghz, 192.168.1.6)

Router: brand new asus AC88u acting as the router and switch (I have a huawei modem to delivery main WAN, supposedly bridged)


Feel free to let me know if I need to run other tests or output other settings to investigate!


Here is my interfaces config on proxmox running cat /etc/network/interfaces :

auto lo
iface lo inet loopback

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.191
netmask 255.255.255.0
gateway 192.168.1.1
bridge_ports eno2
bridge_stp off
bridge_fd 0


Here are the iperf3 tests:

NAS >> Server

[ 5] local 192.168.1.45 port 62731 connected to 192.168.1.191 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 84.9 MBytes 712 Mbits/sec 909 598 KBytes
[ 5] 1.00-2.00 sec 83.4 MBytes 700 Mbits/sec 0 713 KBytes
[ 5] 2.00-3.00 sec 83.5 MBytes 700 Mbits/sec 0 813 KBytes
[ 5] 3.00-4.00 sec 83.4 MBytes 700 Mbits/sec 4 483 KBytes
[ 5] 4.00-5.00 sec 83.4 MBytes 700 Mbits/sec 0 622 KBytes
[ 5] 5.00-6.00 sec 83.5 MBytes 701 Mbits/sec 0 734 KBytes
[ 5] 6.00-7.00 sec 83.5 MBytes 700 Mbits/sec 0 827 KBytes
[ 5] 7.00-8.00 sec 83.4 MBytes 700 Mbits/sec 0 843 KBytes
[ 5] 8.00-9.00 sec 83.5 MBytes 700 Mbits/sec 0 858 KBytes
[ 5] 9.00-10.00 sec 83.5 MBytes 701 Mbits/sec 0 872 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 836 MBytes 701 Mbits/sec 913 sender
[ 5] 0.00-10.00 sec 835 MBytes 701 Mbits/sec receiver

iperf Done.


Server >> NAS

Accepted connection from 192.168.1.191, port 39518
[ 5] local 192.168.1.45 port 5201 connected to 192.168.1.191 port 39520
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 111 MBytes 934 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 937 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 936 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 937 Mbits/sec
[ 5] 5.00-6.00 sec 112 MBytes 937 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 937 Mbits/sec
[ 5] 7.00-8.00 sec 112 MBytes 937 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 936 Mbits/sec
[ 5] 9.00-10.00 sec 112 MBytes 937 Mbits/sec
[ 5] 10.00-10.00 sec 250 KBytes 933 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 1.09 GBytes 936 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------


Server >> MacOSvm

Accepted connection from 192.168.1.191, port 36556
[ 5] local 192.168.1.229 port 5201 connected to 192.168.1.191 port 36558
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 158 MBytes 1.33 Gbits/sec
[ 5] 1.00-2.00 sec 178 MBytes 1.49 Gbits/sec
[ 5] 2.00-3.00 sec 177 MBytes 1.49 Gbits/sec
[ 5] 3.00-4.00 sec 177 MBytes 1.49 Gbits/sec
[ 5] 4.00-5.00 sec 178 MBytes 1.49 Gbits/sec
[ 5] 5.00-6.00 sec 176 MBytes 1.48 Gbits/sec
[ 5] 6.00-7.00 sec 178 MBytes 1.49 Gbits/sec
[ 5] 7.00-8.00 sec 177 MBytes 1.48 Gbits/sec
[ 5] 8.00-9.00 sec 178 MBytes 1.49 Gbits/sec
[ 5] 9.00-10.00 sec 177 MBytes 1.48 Gbits/sec
[ 5] 10.00-10.05 sec 8.33 MBytes 1.51 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.05 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.05 sec 1.72 GBytes 1.47 Gbits/sec


MacOSvm >> Server

Accepted connection from 192.168.1.229, port 64307
[ 5] local 192.168.1.191 port 5201 connected to 192.168.1.229 port 64308
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 195 MBytes 1.64 Gbits/sec
[ 5] 1.00-2.00 sec 874 MBytes 7.34 Gbits/sec
[ 5] 2.00-3.00 sec 455 MBytes 3.82 Gbits/sec
[ 5] 3.00-4.00 sec 204 MBytes 1.71 Gbits/sec
[ 5] 4.00-5.00 sec 977 MBytes 8.20 Gbits/sec
[ 5] 5.00-6.00 sec 867 MBytes 7.27 Gbits/sec
[ 5] 6.00-7.00 sec 1.16 GBytes 10.0 Gbits/sec
[ 5] 7.00-8.00 sec 220 MBytes 1.84 Gbits/sec
[ 5] 8.00-9.00 sec 818 MBytes 6.86 Gbits/sec
[ 5] 9.00-10.00 sec 177 MBytes 1.48 Gbits/sec
[ 5] 10.00-10.00 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.00 sec 5.84 GBytes 5.02 Gbits/sec receiver

MacOSvm >> NAS

Accepted connection from 192.168.1.229, port 64418
[ 5] local 192.168.1.45 port 5201 connected to 192.168.1.229 port 64419
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 56.4 MBytes 469 Mbits/sec
[ 5] 1.01-2.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 2.06-3.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.06-4.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.06-5.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.06-6.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.06-7.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.06-8.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.06-9.05 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.05-10.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 10.06-11.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 11.01-12.05 sec 0.00 Bytes 0.00 bits/sec
[ 5] 12.05-13.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 13.06-13.53 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-13.53 sec 56.4 MBytes 35.0 Mbits/sec receiver

that's where it craps the bed...

NAS >> MacOSvm

Accepted connection from 192.168.1.45, port 24734
[ 5] local 192.168.1.229 port 5201 connected to 192.168.1.45 port 45358
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 55.8 MBytes 468 Mbits/sec
[ 5] 1.00-2.00 sec 83.7 MBytes 702 Mbits/sec
[ 5] 2.00-3.00 sec 83.6 MBytes 702 Mbits/sec
[ 5] 3.00-4.00 sec 83.6 MBytes 701 Mbits/sec
[ 5] 4.00-5.00 sec 83.6 MBytes 701 Mbits/sec
[ 5] 5.00-6.00 sec 83.5 MBytes 701 Mbits/sec
[ 5] 6.00-7.00 sec 83.7 MBytes 702 Mbits/sec
[ 5] 7.00-8.00 sec 83.7 MBytes 702 Mbits/sec
[ 5] 8.00-9.00 sec 83.5 MBytes 701 Mbits/sec
[ 5] 9.00-10.00 sec 83.7 MBytes 702 Mbits/sec
[ 5] 10.00-10.15 sec 12.9 MBytes 701 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.15 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.15 sec 821 MBytes 679 Mbits/sec receiver

all fine the other way !!!


iMacWifi >> MacOSvm

[ 4] local 192.168.1.229 port 65136 connected to 192.168.1.132 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 87.8 MBytes 736 Mbits/sec
[ 4] 1.00-2.00 sec 101 MBytes 844 Mbits/sec
[ 4] 2.00-3.00 sec 103 MBytes 865 Mbits/sec
[ 4] 3.00-4.00 sec 105 MBytes 879 Mbits/sec
[ 4] 4.00-5.00 sec 107 MBytes 898 Mbits/sec
[ 4] 5.00-6.00 sec 82.8 MBytes 695 Mbits/sec
[ 4] 6.00-7.00 sec 78.2 MBytes 656 Mbits/sec
[ 4] 7.00-8.00 sec 84.4 MBytes 708 Mbits/sec
[ 4] 8.00-9.00 sec 88.9 MBytes 746 Mbits/sec
[ 4] 9.00-10.00 sec 94.5 MBytes 793 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 932 MBytes 782 Mbits/sec sender
[ 4] 0.00-10.00 sec 931 MBytes 781 Mbits/sec receiver


iMacWifi >> NAS

Accepted connection from 192.168.1.132, port 54059
[ 5] local 192.168.1.45 port 5201 connected to 192.168.1.132 port 54060
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 51.8 MBytes 434 Mbits/sec
[ 5] 1.00-2.00 sec 59.5 MBytes 500 Mbits/sec
[ 5] 2.00-3.00 sec 60.4 MBytes 507 Mbits/sec
[ 5] 3.00-4.00 sec 60.9 MBytes 511 Mbits/sec
[ 5] 4.00-5.00 sec 61.3 MBytes 514 Mbits/sec
[ 5] 5.00-6.00 sec 61.9 MBytes 519 Mbits/sec
[ 5] 6.00-7.00 sec 62.2 MBytes 521 Mbits/sec
[ 5] 7.00-8.00 sec 62.8 MBytes 527 Mbits/sec
[ 5] 8.00-9.00 sec 62.1 MBytes 521 Mbits/sec
[ 5] 9.00-10.00 sec 61.5 MBytes 516 Mbits/sec
[ 5] 10.00-10.01 sec 441 KBytes 443 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 605 MBytes 507 Mbits/sec receiver
 
Last edited:
Maybe an important note, I tried running iperf3 from MacOSvm to NAS as UDP to test that as well, and that seems to work:

MacOSvm >> NAS (via UDP)

./Downloads/iperf3 -c 192.168.1.45 -u -b 100m -t 10 -i 1 -f m

Connecting to host 192.168.1.45, port 5201

[ 4] local 192.168.1.229 port 55382 connected to 192.168.1.45 port 5201

[ ID] Interval Transfer Bandwidth Total Datagrams

[ 4] 0.00-1.00 sec 10.8 MBytes 90.4 Mbits/sec 1381

[ 4] 1.00-2.00 sec 12.0 MBytes 100 Mbits/sec 1530

[ 4] 2.00-3.00 sec 11.9 MBytes 99.7 Mbits/sec 1521

[ 4] 3.00-4.00 sec 12.0 MBytes 100 Mbits/sec 1532

[ 4] 4.00-5.00 sec 11.9 MBytes 100 Mbits/sec 1525

[ 4] 5.00-6.00 sec 11.9 MBytes 99.7 Mbits/sec 1519

[ 4] 6.00-7.00 sec 12.0 MBytes 100 Mbits/sec 1534

[ 4] 7.00-8.00 sec 11.9 MBytes 99.9 Mbits/sec 1520

[ 4] 8.00-9.00 sec 12.0 MBytes 99.9 Mbits/sec 1530

[ 4] 9.00-10.00 sec 11.9 MBytes 99.6 Mbits/sec 1521

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 4] 0.00-10.00 sec 118 MBytes 99.0 Mbits/sec 0.056 ms 1421/15112 (9.4%)

[ 4] Sent 15112 datagrams

Important packet loss by the way, 9.4% ? That's extremely weird... Anyone has an idea?
 
Bump on this. I tried a LOT of things, including changing completely my Freenas hardware. I am still unable to find a solution. Realtek NICs, Intel NICs, nothing will do. The Proxmox server itself transfers to the NAS and communicates just fine with iperf tests, but the OSX vm does not. One piece of data gets sent, then part of another, then I drop to 0 packets a second after. No explanation possible as to why. Please someone help !
 
Ran more tests. It seems when I set a very small TCP window size that the connection runs smooth. Where could the issue come from? I am really not a network expert, any help would be appreciated...

Code: 
./iperf3 -c 192.168.1.122 -w 2000
Connecting to host 192.168.1.122, port 5201
[  4] local 192.168.1.192 port 62363 connected to 192.168.1.122 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  7.46 MBytes  62.6 Mbits/sec                
[  4]   1.00-2.00   sec  7.39 MBytes  62.0 Mbits/sec                
[  4]   2.00-3.00   sec  7.24 MBytes  60.7 Mbits/sec                
[  4]   3.00-4.00   sec  7.83 MBytes  65.7 Mbits/sec                
[  4]   4.00-5.00   sec  7.51 MBytes  63.0 Mbits/sec                
[  4]   5.00-6.00   sec  7.50 MBytes  62.9 Mbits/sec                
[  4]   6.00-7.00   sec  7.77 MBytes  65.2 Mbits/sec                
[  4]   7.00-8.00   sec  7.62 MBytes  63.9 Mbits/sec                
[  4]   8.00-9.00   sec  7.45 MBytes  62.5 Mbits/sec                
[  4]   9.00-10.00  sec  7.57 MBytes  63.5 Mbits/sec                
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  75.3 MBytes  63.2 Mbits/sec                  sender
[  4]   0.00-10.00  sec  75.3 MBytes  63.2 Mbits/sec                  receiver

iperf Done.
 
I tried something silly tonight, just to see if that would help: Setting the Rate limit to 80MB/s on the VM hardware network through the Proxmox GUI. Since it looked like a saturation of the network from the beginning before (but at the same time I had perfectly working hypervisor > Freenas iperf results) I thought it could help me identify the origin of the issue.

It seems like it did ! I have more consistent transfers now, albeit some times it still hangs, it seems a bit more reliable (see iperf results at the bottom).

So new question now... Where is my bottleneck? Any test I should perform to identify further? Is it the router, my single Ethernet Gigabit port shared with the host, or an I/O problem of some sort? I have limited pci-e availability but if adding an Intel double gigabit card that I passthrough just to that VM will fix my crazy problems for 30$, I'd gladly do that. Your opnion is gladly welcome !

Code: 
./iperf3 -c 192.168.1.28
Connecting to host 192.168.1.28, port 5201
[  4] local 192.168.1.192 port 55983 connected to 192.168.1.28 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  54.0 MBytes   453 Mbits/sec                 
[  4]   1.00-2.00   sec  63.6 MBytes   534 Mbits/sec                 
[  4]   2.00-3.00   sec  65.0 MBytes   546 Mbits/sec                 
[  4]   3.00-4.00   sec  63.5 MBytes   532 Mbits/sec                 
[  4]   4.00-5.00   sec  62.0 MBytes   521 Mbits/sec                 
[  4]   5.00-6.00   sec  66.8 MBytes   560 Mbits/sec                 
[  4]   6.00-7.00   sec  64.6 MBytes   543 Mbits/sec                 
[  4]   7.00-8.00   sec  62.5 MBytes   524 Mbits/sec                 
[  4]   8.00-9.00   sec  63.2 MBytes   530 Mbits/sec                 
[  4]   9.00-10.00  sec  66.2 MBytes   555 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   632 MBytes   530 Mbits/sec                  sender
[  4]   0.00-10.00  sec   631 MBytes   529 Mbits/sec                  receiver
 
I found something that seemed to fix my issues. Note that I changed for an Intel pro/1000 NIC passed through directly to the VM in the meantime, and still was experiencing the same exact problems (actually worse, as it would now kill the network card and needed a reboot to regain any internet access).

Since overloading or saturation seemed to be the issue, I researched further on that. It appears the issue is from OSX itself (who would have thought... ;-)), and TSO no less (TCP Segmentation Offload) which seems to overload any NIC not properly configured, especially on Hackintoshs and MacOS VMs.

So for posterity, steps for the fix:

Verify that your TSO is actually on (on by default in Mojave apparently)

Code: 
sudo sysctl -a | grep net.inet.tcp.tso

This should output:

Code: 
net.inet.tcp.tso: 1

Run this command in OSX terminal (to set TSO off temporarly):

Code: 
sudo sysctl -w net.inet.tcp.tso=0

Check if the problem persists.

If it is fixed, you can add that command to run at startup by either editing your sysctl.conf or creating a new one (works with any text editor, or via command line), adding the following line, and saving it to /private/etc/sysctl.conf :

Code: 
net.inet.tcp.tso: 0
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back