OK, I have been chasing a performance issue transferring data between two proxmox hosts and I cant seem to figure things out. The general issue is that I am seeing really low transfer rates of data between the machines. Copying a large 10GB file is only getting about 80MB/s in transfer. This is regardless of using scp, rsync, nfs, iscsi. Here is the basic layout of the two machines:
Host A
Using iperf3 going host A to B across the bonded bridge, I am getting:
Totally what I expect, one check here. Same results basically going either way between them. The bridge at the host level is working.
On host A, doing some basic disk performance using 'dd' I get:
Overall the disk performance is what I was kinda expecting. Though I was a little surprised about the ram disk performance: /dev/zero about 1.4GB/s and the transfer of the zero and random files to the zpool. Using straight cp and rsync produced low performance transfers, but still within the expected results.
I did the same type of disk benchmarks on host B and got the expected results. Moving data around with dd giving 1+GB/s transfers and cp/rsync around 800MB/s transfers.
So how I created a VM on eahc host with basically the same configuration:
Pretty much expected results as the two ends are using the same bridge on the same host.
How going from the VM on Host A to Host B, I am getting this in iperf3:
That was about 30% slower than expected at 12.1Gb/s. The traffic for this testing is making two hops: VM to host A and then Host A to Host B. Certainly the processing needed to handle the routing from first hop to the second hop cant take the big of a hit. Question 1: What would be causing the drop in performance/through put here?
Now going from VM host A to VM on Host B using iperf3, I am getting the following:
That was about 50% slower than expected at 9.11Gb/s. The traffic for this testing is making three hops: VM to host A, Host A to Host B, and then Host B to VM. Again I can not see the routing taking that big of a hit as it is not that comprex. Question 2: What would be causing the drop in performance/through put here?
Now doing some disk bench marking with dd on a VM with the disk sitting on the host zpool, I am getting:
The write performance in the VM is nowhere close to what is expected. Raw writes from /dev/zero is about 1/3 of the speed and writes from /dev/random is down about 30%. I know that there is overhead with the VM sitting on top of the zpool and handling all the abstration of hardware, but this seems waaaaaaay off. Question 3: What is causing the deduced disk performance from the VM perspective?
Now copying/moving/transfering data from one VM to the VM is the major source of pain where the transfer speeds are only around 80MB/s.
-- using tar, mbuffer, and ssh: in @ 77.9 MiB/s, out @ 77.9 MiB/s, 4202 MiB total, buffer 100% full apps/UpRev/video/06-Evans-Up
-- rysnc: sent 998,511,690 bytes received 35 bytes 60,515,862.12 bytes/sec
One thing I did notice is that the VM receiving the data has at least 50% of the cores pegged at 100% WA. What is causing the huge amount of IO wait? This might be the critical issue.
Host A
- dual Xeon E5-2680v3
- 256GB DDR4-2133 EEC RDIMMs
- dual port 10gb RJ-45 NIC bonded( rr ) added to vmbr1 ( ports, bond, and bridge MTU 9000 )
- zpool 6 iNTEL dc s3700 striped RAID0 compression=on shift=12 blocksize=32k
- 60GB ram disk mounted
- dual Xeon E5-2690v3
- 512GB DDR4-2133 ECC RDIMMs
- dual port 10gb RJ-45 NIC bonded( rr ) added to vmbr1 ( ports, bond, and bridge MTU 9000 )
- 80GB ram disk mounted
- zpool 8 6G 15K SAS2 striped RAID0 compression=on shift=12 blocksize=32k
Using iperf3 going host A to B across the bonded bridge, I am getting:
Code:
[ 5] 0.00-10.00 sec 22.9 GBytes 19.7 Gbits/sec 627 sender
[ 5] 0.00-10.00 sec 22.9 GBytes 19.7 Gbits/sec receiver
Totally what I expect, one check here. Same results basically going either way between them. The bridge at the host level is working.
On host A, doing some basic disk performance using 'dd' I get:
Code:
root@odin:/mnt/pve/ram# dd if=/dev/random of=/vm2-zfs-r0/test/test.dat bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 94.6553 s, 222 MB/s
root@odin:/mnt/pve/ram# dd if=/dev/zero of=/vm2-zfs-r0/test/test1.dat bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 5.78259 s, 3.6 GB/s
root@odin:/mnt/pve/ram# dd if=/dev/random of=/mnt/pve/ram/test2.dat bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 102.238 s, 205 MB/s
root@odin:/mnt/pve/ram# dd if=/dev/zero of=/mnt/pve/ram/test3.dat bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 15.1278 s, 1.4 GB/s
root@odin:/mnt/pve/ram# dd if=test2.dat of=/vm2-zfs-r0/test/test4.dat bs=1M
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 18.981 s, 1.1 GB/s
root@odin:/mnt/pve/ram# dd if=test3.dat of=/vm2-zfs-r0/test/test5.dat bs=1M
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 13.5282 s, 1.6 GB/s
root@odin:/mnt/pve/ram# time cp test2.dat /vm2-zfs-r0/test/test6.dat
real 0m19.271s
user 0m0.156s
sys 0m19.099s
root@odin:/mnt/pve/ram# time cp test3.dat /vm2-zfs-r0/test/test7.dat
real 0m14.861s
user 0m0.112s
sys 0m14.739s
root@odin:/mnt/pve/ram# rsync -avp test2.dat /vm2-zfs-r0/test
sending incremental file list
test2.dat
sent 20,976,640,100 bytes received 35 bytes 856,189,393.27 bytes/sec
total size is 20,971,520,000 speedup is 1.00
root@odin:/mnt/pve/ram# rsync -avp test3.dat /vm2-zfs-r0/test
sending incremental file list
test3.dat
sent 20,976,640,099 bytes received 35 bytes 1,133,872,439.68 bytes/sec
total size is 20,971,520,000 speedup is 1.00
Overall the disk performance is what I was kinda expecting. Though I was a little surprised about the ram disk performance: /dev/zero about 1.4GB/s and the transfer of the zero and random files to the zpool. Using straight cp and rsync produced low performance transfers, but still within the expected results.
I did the same type of disk benchmarks on host B and got the expected results. Moving data around with dd giving 1+GB/s transfers and cp/rsync around 800MB/s transfers.
So how I created a VM on eahc host with basically the same configuration:
- 8 cores, host, numa=1
- 16GB ram
- VirtIO SCSI controller
- 100G disk scsi
- VirtIO NIC to vmbr1 firewall=off ( MTU 9000 set in guest os )
- Fedora 33 x86 guest os
Code:
[ 5] 0.00-10.00 sec 19.8 GBytes 17.0 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 19.8 GBytes 17.0 Gbits/sec receiver
Pretty much expected results as the two ends are using the same bridge on the same host.
How going from the VM on Host A to Host B, I am getting this in iperf3:
Code:
[ 5] 0.00-10.00 sec 14.1 GBytes 12.1 Gbits/sec 30 sender
[ 5] 0.00-10.00 sec 14.1 GBytes 12.1 Gbits/sec receiver
That was about 30% slower than expected at 12.1Gb/s. The traffic for this testing is making two hops: VM to host A and then Host A to Host B. Certainly the processing needed to handle the routing from first hop to the second hop cant take the big of a hit. Question 1: What would be causing the drop in performance/through put here?
Now going from VM host A to VM on Host B using iperf3, I am getting the following:
Code:
[ 5] 0.00-10.00 sec 10.6 GBytes 9.11 Gbits/sec 25 sender
[ 5] 0.00-10.00 sec 10.6 GBytes 9.11 Gbits/sec receiver
That was about 50% slower than expected at 9.11Gb/s. The traffic for this testing is making three hops: VM to host A, Host A to Host B, and then Host B to VM. Again I can not see the routing taking that big of a hit as it is not that comprex. Question 2: What would be causing the drop in performance/through put here?
Now doing some disk bench marking with dd on a VM with the disk sitting on the host zpool, I am getting:
Code:
[root@sauron gondor]# dd if=/dev/zero of=/gondor/test1.dat bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 54.3383 s, 386 MB/s
[root@sauron gondor]# dd if=/dev/random of=/gondor/test2.dat bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 136.695 s, 153 MB/s
Now copying/moving/transfering data from one VM to the VM is the major source of pain where the transfer speeds are only around 80MB/s.
-- using tar, mbuffer, and ssh: in @ 77.9 MiB/s, out @ 77.9 MiB/s, 4202 MiB total, buffer 100% full apps/UpRev/video/06-Evans-Up
-- rysnc: sent 998,511,690 bytes received 35 bytes 60,515,862.12 bytes/sec
One thing I did notice is that the VM receiving the data has at least 50% of the cores pegged at 100% WA. What is causing the huge amount of IO wait? This might be the critical issue.