ISO Upload Network: On a PVE node with multiple VMBRs/VLANs, which one is used for uploading ISOs to the node?

Sep 1, 2022
475
180
48
41
Hello!

I have a PVE 9 node with two NICs. One is a 1 Gbps NIC I use for the management interface. The other is a 10 Gbps NIC that I've been using for VMs and LXCs to connect to the outside world. (I could have put the management interface on the 10 Gbps NIC, but I like the idea of having an isolated management network that will still function if I have to disconnect the 10 Gbps NIC.)

The PVE node uses the 10 Gbps NIC to connect to an NFS share on my NAS where ISOs live.

Observed Behavior:
When I upload an ISO to the PVE node using the web interface from a client with a 10 Gbps NIC, the upload speed caps and holds steady at ~226 MB/s. I've tested it with multiple ISOs and the speed is constant until the ISO is fully uploaded.

The cp command that moves the ISO from the PVE node to the NAS is basically instant even for a multi-GB ISO, so it must be using the 10 Gbps link between the PVE node and the NAS (this is the only avaialble link between the PVE node and the NAS).

My client computer where I'm logged into the web interface has IP addresses on the same VLANs as the PVE management interface and the NAS's 10Gbps link.

Question:
I had always assumed that ISOs were uploaded over the management network and would be capped to 125 MB/s or so, but seeing a consistent 225 MB/s has me confused.
If it's using the 10 Gbps link, why is it so consistently capped at 226 MB/s?

By default, does the PVE web interface use the management interface (vmbr0) to upload ISOs?
If so, is there a way to override that?


If the answer to the first question is yes, and the ISOs are actually getting uploaded at 1 Gbps (~125 MB/s max), I've got a separate issue with my client misreporting its upload speeds.
 
I don't know about NFS because I didn't create an NFS share, but it's much faster on my smb.

Checking zpool iostat 1 on the host and Truenas, I can see that Truenas is being written to after the host writes.

Writing to local is 425MB/s (3.4Gbps)

Writing to Truenas after Gui's upload completes 250MB/s (2Gbps)

Host from PC doing upload is using all NICs
* In an environment with two NICs, each uses 1.7 Gbps (1.7 to 2.0 Gbps).
 In an environment with one NIC, 3.4 Gbps is used

Only one NIC is used when writing from host to Truenas

It probably depends on the computer performing the upload and the temporary storage location of the upload destination.
 
Last edited: