Hi
i recently finished the 10g connection between Proxmox and TrueNas Scale. My purpose might be a little odd to be considered as a general use case scenario but here it goes nevertheless.
The connection established via NFS and Iscsi and I left async as default so enabled for both connections.
On Prox and TrueNas side set MTU to 9000 in order to enable jumbo frames
What I need to do is add extra storage to the VM for keeping the Active Directory share folder/files in order to save time moving / backing up the VM. I will also add a storage for VM Backups to be stored on Nas.
info: Drives are identical for bot Prox and Nas meaning physical characteristics of them: 512 (logical/physial) sas3 / 12g / 10.000rpm 1.2Tb drives.
On Prox the VM is stored on a raid 10 comprised by 4 drives with ashift=9 / volblock=8 (thinking of 4 and creating Vm again)
On Truenas the shared storage is a dataset coming from a mirrored pool from 2 disks (same specs as mentioned above).
Since this is for testing purposes, later on that shared storage from TrueNas will be bigger and based on raidZ2 since it would be for data and VM backups and not the Vms themselves.
What I thought I would achieve: speeds ( or close enough) to 1000mb/s since it is a 10gb connection.
Of course it is the matter of what transfer speeds according to their raid level the disks themselves can handle,
Maybe it plays role the recordsize of the shared storage since it is by default 128k (or it means up to 128k and it is flexible so no matter as a factor)
So:
On a WinServ19 VM (4 cores / 12g ram based on a 32gb portion of a raid 10 / sas3 12g 10.000rpm drives) I added the following extra storages
- 10gb coming from truenas with the 10gbit connection and added with disk type as raw used NFS connection method between prox-truenas Drive Y:
- 32gb coming from truenas with the 10gbit connection and added with disk type as qcow2 used NFS connection method between prox-truenas Drive Z:
- 60gb coming from truenas with the 10gbit connection used iscsi connection method between prox-truenas Drive D:
File size for copying several ones (one each time) between 2.1 - 4.5gb
I tried copying from c each time and after completion I restarted the Vm again to avoid using the ram buffer which speeds up the process more the second time.
Y: DRIVE / coming from nfs connection / created with disk type raw
C: -> Y: with No Cache (set outside the VM) 45 -560mb
C: -> Y: with Write Back (set outside the VM) 370mb
C: -> Y: with Direct Sync (set outside the VM) 400mb Second try dropped to 35mb (inconsistency even without restarting the VM here)
Z: DRIVE / coming from nfs connection / created with disk type Qcow2 (probably here plays role the record size being set from TrueNas? By the way this is how I would like to setup the second storage since Truenas going to be able to see the files instead of raw)
C: -> Z: with No Cache (set outside the VM) 650-790mb I can t replicate these numbers again.
C: -> Z: with Write Back (set outside the VM) 350mb
C: -> Z: with Direct Sync (set outside the VM) 35mb Second try dropped to 35mb (inconsistency even without restarting the VM here)
D: DRIVE / coming from ISCSI connection
C: -> D: with No Cache (set outside the VM) 400mb
C: -> D: with Write Back (set outside the VM) 380mb
C: -> D: with Direct Sync (set outside the VM) 18mb Second try dropped to 35mb (inconsistency even without restarting the VM here)
I thought afterwards to bench each storage separately with crystal disk to have an idea of what I might expect
You can see in the pic above which disk the bench run on (each time these tests run have way different results but lets assume they are stable)
Do the speeds I am getting seem logical to you?
Is there anything else to do to fine tune it from the Prox or TrueNas side?
PS I have heard of a bug in both WinServ19 and 22 (not in 16) in speed transfers but that was at the middle of copy which reduces the speed to a constant low number like 35 or something and if the file is large enough
i recently finished the 10g connection between Proxmox and TrueNas Scale. My purpose might be a little odd to be considered as a general use case scenario but here it goes nevertheless.
The connection established via NFS and Iscsi and I left async as default so enabled for both connections.
On Prox and TrueNas side set MTU to 9000 in order to enable jumbo frames
What I need to do is add extra storage to the VM for keeping the Active Directory share folder/files in order to save time moving / backing up the VM. I will also add a storage for VM Backups to be stored on Nas.
info: Drives are identical for bot Prox and Nas meaning physical characteristics of them: 512 (logical/physial) sas3 / 12g / 10.000rpm 1.2Tb drives.
On Prox the VM is stored on a raid 10 comprised by 4 drives with ashift=9 / volblock=8 (thinking of 4 and creating Vm again)
On Truenas the shared storage is a dataset coming from a mirrored pool from 2 disks (same specs as mentioned above).
Since this is for testing purposes, later on that shared storage from TrueNas will be bigger and based on raidZ2 since it would be for data and VM backups and not the Vms themselves.
What I thought I would achieve: speeds ( or close enough) to 1000mb/s since it is a 10gb connection.
Of course it is the matter of what transfer speeds according to their raid level the disks themselves can handle,
Maybe it plays role the recordsize of the shared storage since it is by default 128k (or it means up to 128k and it is flexible so no matter as a factor)
So:
On a WinServ19 VM (4 cores / 12g ram based on a 32gb portion of a raid 10 / sas3 12g 10.000rpm drives) I added the following extra storages
- 10gb coming from truenas with the 10gbit connection and added with disk type as raw used NFS connection method between prox-truenas Drive Y:
- 32gb coming from truenas with the 10gbit connection and added with disk type as qcow2 used NFS connection method between prox-truenas Drive Z:
- 60gb coming from truenas with the 10gbit connection used iscsi connection method between prox-truenas Drive D:
File size for copying several ones (one each time) between 2.1 - 4.5gb
I tried copying from c each time and after completion I restarted the Vm again to avoid using the ram buffer which speeds up the process more the second time.
Y: DRIVE / coming from nfs connection / created with disk type raw
C: -> Y: with No Cache (set outside the VM) 45 -560mb
C: -> Y: with Write Back (set outside the VM) 370mb
C: -> Y: with Direct Sync (set outside the VM) 400mb Second try dropped to 35mb (inconsistency even without restarting the VM here)
Z: DRIVE / coming from nfs connection / created with disk type Qcow2 (probably here plays role the record size being set from TrueNas? By the way this is how I would like to setup the second storage since Truenas going to be able to see the files instead of raw)
C: -> Z: with No Cache (set outside the VM) 650-790mb I can t replicate these numbers again.
C: -> Z: with Write Back (set outside the VM) 350mb
C: -> Z: with Direct Sync (set outside the VM) 35mb Second try dropped to 35mb (inconsistency even without restarting the VM here)
D: DRIVE / coming from ISCSI connection
C: -> D: with No Cache (set outside the VM) 400mb
C: -> D: with Write Back (set outside the VM) 380mb
C: -> D: with Direct Sync (set outside the VM) 18mb Second try dropped to 35mb (inconsistency even without restarting the VM here)
I thought afterwards to bench each storage separately with crystal disk to have an idea of what I might expect
You can see in the pic above which disk the bench run on (each time these tests run have way different results but lets assume they are stable)
Do the speeds I am getting seem logical to you?
Is there anything else to do to fine tune it from the Prox or TrueNas side?
PS I have heard of a bug in both WinServ19 and 22 (not in 16) in speed transfers but that was at the middle of copy which reduces the speed to a constant low number like 35 or something and if the file is large enough
Last edited: