[Update]: Nevermind, you just need to make sure you're watching the right tools to see how much data is being transferred. Completely my fault for not understanding what was going on and not watching the right data to determine the speed that was being achieved.
I just deleted my three node cluster and wiped everything and fresh installed (PVE 8) all three nodes. One is an Intel Nuc 13 Pro, the other two are older MacBook Pros (2011 and 2012). After the fresh install of v8 and updating them, I joined them to a cluster. Two have higher end Samsung SSDs, one has a very fast Samsung NVMe (980 Pro). When I do a migration between nodes on a 1Gb network, it gets about 10k/sec. When I restored the VMs and LCXs from my NAS, it was transferring around 120MB/sec. I've tried setting the migration limits very high, set the migration to insecure, turned off the firewalls, and I can't figure out why these migrations are so slow. I have the cluster operating on VLAN 35 but the migrations still go over the VLAN where the web interface is.
Running iPerf3 between the nodes gives me almost full use of the 1Gb connection (see below for those results).
Clearly there's nothing corrupt as all three have just been fresh installed so it must be some setting somewhere.
Does anyone have any ideas what is causing this? Any help would be appreciated.
Here are the iPerf3 results:
root@node2:~# iperf3 -c 10.99.30.13 -bidir
Connecting to host 10.16.30.13, port 5201
[ 5] local 10.99.30.12 port 36918 connected to 10.99.30.13 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 392 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 392 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 0 413 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 938 Mbits/sec 0 413 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 943 Mbits/sec 0 413 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 0 434 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 0 434 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 939 Mbits/sec 0 434 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 943 Mbits/sec 0 434 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec 0 434 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
iperf Done.
root@node2:~#
I just deleted my three node cluster and wiped everything and fresh installed (PVE 8) all three nodes. One is an Intel Nuc 13 Pro, the other two are older MacBook Pros (2011 and 2012). After the fresh install of v8 and updating them, I joined them to a cluster. Two have higher end Samsung SSDs, one has a very fast Samsung NVMe (980 Pro). When I do a migration between nodes on a 1Gb network, it gets about 10k/sec. When I restored the VMs and LCXs from my NAS, it was transferring around 120MB/sec. I've tried setting the migration limits very high, set the migration to insecure, turned off the firewalls, and I can't figure out why these migrations are so slow. I have the cluster operating on VLAN 35 but the migrations still go over the VLAN where the web interface is.
Running iPerf3 between the nodes gives me almost full use of the 1Gb connection (see below for those results).
Clearly there's nothing corrupt as all three have just been fresh installed so it must be some setting somewhere.
Does anyone have any ideas what is causing this? Any help would be appreciated.
Here are the iPerf3 results:
root@node2:~# iperf3 -c 10.99.30.13 -bidir
Connecting to host 10.16.30.13, port 5201
[ 5] local 10.99.30.12 port 36918 connected to 10.99.30.13 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 392 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 392 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 0 413 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 938 Mbits/sec 0 413 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 943 Mbits/sec 0 413 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 0 434 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 0 434 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 939 Mbits/sec 0 434 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 943 Mbits/sec 0 434 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec 0 434 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
iperf Done.
root@node2:~#
Last edited: