TrueNAS Storage Plugin

curruscanis said:
I have just reinstalled my test system as well, I am alos on the "alpha" branch. I have currently only setup NVME connectivity for testing.

I have a single test host with 2 x 10g SFP interfaces - wired to redundant switching - to a TrueNAS mirrored vdev's. in theory with multipathing... haven't fully tested that yet, but set it up.



My /etc/pve/storage.cfg

truenasplugin: truenas-nvme
api_host 10.10.5.10
api_key 3-vVeJEyC29sGwNEjpCfVnRBjHOKei8F3TekksP6bbr1NlStP4OHQ48UVmXl2laDiI
dataset tank/proxmox-nvme
transport_mode nvme-tcp
subsystem_nqn nqn.2011-06.com.truenas:uuid:279fe462-e1d3-4d01-9ce6-05b989731872:proxmox-nvme
api_insecure 1
shared 1
discovery_portal 10.10.5.10:4420
zvol_blocksize 16K
tn_sparse 1
portals 10.10.6.10:4420
hostnqn nqn.2011-06.com.truenas:uuid:279fe462-e1d3-4d01-9ce6-05b989731872
content images


FIO test from the install diagnostics:

FIO Storage Benchmark

Running benchmark on storage: truenas-nvme

FIO installation: ✓ fio-3.39
Storage configuration: ✓ Valid (nvme-tcp mode)
Finding available VM ID: ✓ Using VM ID 990
Allocating 10GB test volume: ✓ truenas-nvme:vol-fio-bench-1764783729-nsa58d227d-01de-4ece-aa57-96b859d97f57
Waiting for device (5s): ✓ Ready
Detecting device path: ✓ /dev/nvme1n1
Validating device is unused: ✓ Device is safe to test

Starting FIO benchmarks (30 tests, 25-30 minutes total)...

Transport mode: nvme-tcp (testing QD=1, 16, 32, 64, 128)

Sequential Read Bandwidth Tests: [1-5/30]
Queue Depth = 1: ✓ 537.10 MB/s
Queue Depth = 16: ✓ 1.10 GB/s
Queue Depth = 32: ✓ 1.10 GB/s
Queue Depth = 64: ✓ 1.10 GB/s
Queue Depth = 128: ✓ 1.10 GB/s

Sequential Write Bandwidth Tests: [6-10/30]
Queue Depth = 1: ✓ 472.18 MB/s
Queue Depth = 16: ✓ 451.78 MB/s
Queue Depth = 32: ✓ 358.25 MB/s
Queue Depth = 64: ✓ 362.28 MB/s
Queue Depth = 128: ✓ 363.43 MB/s

Random Read IOPS Tests: [11-15/30]
Queue Depth = 1: ✓ 5,786 IOPS
Queue Depth = 16: ✓ 78,306 IOPS
Queue Depth = 32: ✓ 103,105 IOPS
Queue Depth = 64: ✓ 112,326 IOPS
Queue Depth = 128: ✓ 122,474 IOPS

Random Write IOPS Tests: [16-20/30]
Queue Depth = 1: ✓ 6,769 IOPS
Queue Depth = 16: ✓ 30,142 IOPS
Queue Depth = 32: ✓ 25,217 IOPS
Queue Depth = 64: ✓ 25,425 IOPS
Queue Depth = 128: ✓ 25,737 IOPS

Random Read Latency Tests: [21-25/30]
Queue Depth = 1: ✓ 175.82 µs
Queue Depth = 16: ✓ 200.85 µs
Queue Depth = 32: ✓ 301.12 µs
Queue Depth = 64: ✓ 531.54 µs
Queue Depth = 128: ✓ 1.01 ms

Mixed 70/30 Workload Tests: [26-30/30]
Queue Depth = 1: ✓ R: 4,289 / W: 1,834 IOPS
Queue Depth = 16: ✓ R: 54,862 / W: 23,532 IOPS
Queue Depth = 32: ✓ R: 63,986 / W: 27,431 IOPS
Queue Depth = 64: ✓ R: 64,147 / W: 27,500 IOPS
Queue Depth = 128: ✓ R: 63,898 / W: 27,393 IOPS

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Benchmark Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Total tests run: 30
Completed: 30

Top Performers:

Sequential Read: 1.10 GB/s (QD=16 )
Sequential Write: 472.18 MB/s (QD=1 )
Random Read IOPS: 122,474 IOPS (QD=128)
Random Write IOPS: 30,142 IOPS (QD=16 )
Lowest Latency: 175.82 µs (QD=1 )

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Press Enter to return to diagnostics menu...
Click to expand...


Here is the reports from the truenas 10g nics, it more or less proves the multipath is working, interesting that one is always send, and the other recieve...

1764794334968.png
 
jt_telrite said:
would you be able to share the MPIO configuration ? i.e. a "safe" output of your /etc/network/interfaces
Click to expand...
Here is the network interfaces on the proxmox server:

I don't have anything specificly configured other than IP / info... VLANs are done at the switching level.

Code: 
root@bob:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto

iface enp87s0 inet manual

auto

iface enp88s0 inet manual

auto enp2s0f0np0
iface enp2s0f0np0 inet static
        address 10.10.5.11/24
#vlan5

auto enp2s0f1np1
iface enp2s0f1np1 inet static
        address 10.10.6.11/24
#vlan6

iface wlp89s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.239.112/24
        gateway 192.168.239.1
        bridge-ports enp87s0
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*
 
  • Like
Reactions: jt_telrite
curruscanis said:
Here is the network interfaces on the proxmox server:

I don't have anything specificly configured other than IP / info... VLANs are done at the switching level.

Code: 
root@bob:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto

iface enp87s0 inet manual

auto

iface enp88s0 inet manual

auto enp2s0f0np0
iface enp2s0f0np0 inet static
        address 10.10.5.11/24
#vlan5

auto enp2s0f1np1
iface enp2s0f1np1 inet static
        address 10.10.6.11/24
#vlan6

iface wlp89s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.239.112/24
        gateway 192.168.239.1
        bridge-ports enp87s0
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*
Click to expand...
is there any other configurations required ? I see several Multipath configuration documents... not sure if they apply here.
 
Last edited:
You must log in or register to reply here.
Top Bottom