Slow Downlink/Download speeds in VM after some time.

[dragon2611]

I have strange issue with a couple of cachyos (arch) based virtual machines where once they've been running for an unknown amount of time the download/downlink (TO the VM) is terribly slow. (They will struggle to pull above 30Mbit/s)

The hosts are connected by 2x1Gbit in LACP as it's my personal/lab setup in a colocation facility I didn't have the luxury of adding 10Gbit+ networking at the moment.

Both the VM's that are showing this issue are on different hosts, and using virtio network drivers, the uplink/upload speeds are fine and can easily saturate the physical connections.

I also checked the utilisation of the physical network ports on the host machines to make sure it wasn't simply congestion from something else pulling data.

Network speeds on the host itself are fine and other VMs appeared fine.

Storage is local NVMe, with ZFS replication happening periodically, but that is throttled to ~100Mbit/s

 

[leesteken]

And you're sure it's not the SSD that stores the incoming data that's slowing it down after sustained writes? That happens a lot to people with non-enterprise SSDs. Can you test without writing the downloaded to a disk?

 

[dragon2611]

And you're sure it's not the SSD that stores the incoming data that's slowing it down after sustained writes? That happens a lot to people with non-enterprise SSDs. Can you test without writing the downloaded to a disk?

It shows up on a speed test as well as downloading, but when it next happens I'll try a wget to /dev/null.

I did check the network utilisation at the time but didn't think to check the disk queue, that said the 2VM's are on different hosts, and I've expected that if it was the SSD causing it then the speeds to the host itself to be impacted as the NVM-E is the boot drive.

Those machines have 1x NVM-E and 2 SATA SSDs, although the VMs are mostly on the NVM-E

The disk I/O on those hosts is usually very low as the VM's tend to spend a lot of their life just idling.