Network speed slows down during rsync

[Apossum]

Hello everybody!
I have proxmox cluster 2 nodes Virtual Environment 7.4-17 and NAS. Without any cluster FSs. Just EXT2 Dirs and LVM.No ZFS. Network between two nodes and NAS 10Gbit. iperf3 between hosts confirm it.
I have NAS with NFS export an 10 Gbit network. I have NFS mountpoint /mnt/point on VM(Ubuntu server 22 0.4) and when i make rsync --progress /mnt/point/file it starts from about 500-600 Megabytes/sec and rapidly reduces to about 10-20 Mb/sec..(it happens in about 10 seconds).
I have been trying to understand what's wrong for long time but still no idea.
Here's config of my VM:

agent: 1,fstrim_cloned_disks=1
balloon: 0
boot: order=scsi1
cores: 16
cpuunits: 500
description: NewSQL server
memory: 131072
meta: creation-qemu=7.2.0,ctime=1704366252
name: SQLNEW
net0: virtio=42:E9:F2:DD:98:6D,bridge=vmbr1,queues=16
numa: 0
ostype: l26
scsi0: EXT4:132/vm-132-disk-1.qcow2,iothread=1,size=6000G
scsi1: EXT4:132/vm-132-disk-0.qcow2,iothread=1,size=500G
scsihw: virtio-scsi-single
smbios1: uuid=073ff35f-2bdc-4d29-8666-349063ff06a9
sockets: 1
virtio0: EXT4:132/vm-132-disk-2.qcow2,iothread=1,size=650G
virtio1: EXT4:132/vm-132-disk-3.qcow2,iothread=1,size=750G
vmgenid: 1dba6731-2ab3-4064-aba5-11bbb1ae2e6e

Some time rsync starts at once with low speed but if i stops it (Ctrl+C) and starts again the situation mentionet above repeats.

vmbr1 linux bridge is on 10Gbit/s network.

And actualy speed between VMs is very low and unstable even within one node

Please help me to resolve this.. im desperate...If i can't solve this i'll have to switch to Microsoft Hyper-V..but i won't.

 

[spirit]

what is your local storage speed ? is it ssd/nvme ? same question for your nas.

Do you rsync a lot of small files, or some big files ?

rsync could be slow to iterate through a lot of small files, and also as it using ssh, it's only use 1core for encryption. (so, you should be able to reach for than 2-3gbits or 400MB/s

 

[Apossum]

what is your local storage speed ? is it ssd/nvme ? same question for your nas.

Do you rsync a lot of small files, or some big files ?

rsync could be slow to iterate through a lot of small files, and also as it using ssh, it's only use 1core for encryption. (so, you should be able to reach for than 2-3gbits or 400MB/s

Thank you for reply.
My local storage speed is not fast - about 300 Mb/s. it's not ssd/nvme.
NAS:https://www.qnap.com/en/product/ts-h1283xu-rp/specs/hardware
About 5Gb/s

lshw -class disk
root@pve4:~# lshw -class disk
  *-disk:0              
       description: SCSI Disk
       product: ST18000NM004J
       vendor: SEAGATE
       physical id: 0.0.0
       bus info: scsi@0:0.0.0
       logical name: /dev/sda
       version: E004
       serial: ZR5EZ9VH0000C230EMJC
       size: 16TiB (18TB)
       capacity: 46TiB (50TB)
       capabilities: 7200rpm gpt-1.00 partitioned partitioned:gpt
       configuration: ansiversion=7 guid=9266fb74-6b0d-4740-a31f-accc454e765f logicalsectorsize=512 sectorsize=4096
  *-disk:1
       description: SCSI Disk
       product: ST18000NM004J
       vendor: SEAGATE
       physical id: 0.1.0
       bus info: scsi@0:0.1.0
       logical name: /dev/sdb
       version: E004
       serial: ZR5EWPHQ0000C245CN7A
       size: 16TiB (18TB)
       capacity: 45TiB (50TB)
       capabilities: 7200rpm
       configuration: ansiversion=7 logicalsectorsize=512 sectorsize=4096
  *-disk:2
       description: SCSI Disk
       product: ST18000NM004J
       vendor: SEAGATE
       physical id: 0.2.0
       bus info: scsi@0:0.2.0
       logical name: /dev/sdc
       version: E004
       serial: ZR5EWS040000C335ED58
       size: 16TiB (18TB)
       capacity: 46TiB (51TB)
       capabilities: 7200rpm gpt-1.00 partitioned partitioned:gpt
       configuration: ansiversion=7 guid=290aae5c-730a-4d6c-b3b7-d200f8902f32 logicalsectorsize=512 sectorsize=4096
  *-disk:4
       description: SCSI Disk
       product: PERC H730P Mini
       vendor: DELL
       physical id: 2.0.0
       bus info: scsi@0:2.0.0
       logical name: /dev/sde
       version: 4.30
       serial: 007749352046e4172d00a229a9f0a7ce
       size: 1787GiB (1919GB)
       capabilities: gpt-1.00 partitioned partitioned:gpt
       configuration: ansiversion=5 guid=2bac865b-88fe-4e34-a45f-60537af2a584 logicalsectorsize=512 sectorsize=4096

description: SCSI Disk
product: PERC H730P Mini - is SSD RAID1- proxmox installed on this. No storages on this.

rsync i made with file about 150 gigabytes. And the storage of VM is on EXT4 dir sorage /dev/sdc1 17T 5.4T 11T 35% /mnt/pve/EXT4

What else you would you suggest to use for speed tests?))

 

[spirit]

What else you would you suggest to use for speed tests?))

you already have the response:

". iperf3 between hosts confirm it."

^_^


about your slowdown, it's quite possible than you PERC H730p have write cache, so it's fast until the cache is full, and slowdown when it begin to flush to disk.

so don't use disk to test network, don't use an encryption network proto like ssh (cpu) to test network.


just use a network benchmark tool : iperf3 or iperf2.

iperf3 is not multithread, using only 1core, even if you use the -P option. So, iperf2 can be usefull for high bandwith testing >30~40Gb/s.