That looks great, I have been installing and reinstalling this plugin system for various trials and the various install steps can be tiresome. It will be very nice if you can have it capable of install like this. I also like the idea of the health check tools and backups. I will give it a try this week in a multipath enviorment.
Definitely on the list of todos, haven't had experience with publishing packages though. But it's something to learn!
I don't imagine there's much in the way of updates that a Proxmox update could break unless they totally redo their APIs. But Proxmox is heavily used personally and professionally around here, so always keeping an active look out for big changes.
bash <(curl -sSL https://raw.githubusercontent.com/WarlockSyno/truenasplugin/alpha/install.sh)
## Version 1.1.3 (November 5, 2025)
### **Major Performance Improvements**
#### **List Performance - N+1 Query Pattern Elimination**
- **Dramatic speed improvements for storage listing operations** - Up to 7.5x faster for large deployments
- **10 volumes**: 2.3s → 1.7s (1.4x faster, 28% reduction)
- **50 volumes**: 6.7s → 1.8s (3.7x faster, 73% reduction)
- **100 volumes**: 18.2s → 2.4s (7.5x faster, 87% reduction)
- **Per-volume cost**: 182ms → 24ms (87% reduction)
- **Extrapolated 1000 volumes**: ~182s (3min) → ~24s (8x improvement)
- **Root cause**: `list_images` was making individual `_tn_dataset_get()` API calls for each volume (O(n) API requests)
- **Solution**: Implemented batch dataset fetching with single `pool.dataset.query` API call
- Fetches all child datasets at once with TrueNAS query filter
- Builds O(1) hash lookup table for dataset metadata
- Falls back to individual API calls if batch fetch fails
- **Impact**:
- Small deployments (10 volumes): Modest improvement due to batch fetch overhead
- Large deployments (100+ volumes): Dramatic improvement as N+1 elimination fully realized
- API efficiency: Changed from O(n) API calls to O(1) API call
- Web UI responsiveness: Storage views load 7.5x faster for large environments
- Reduced TrueNAS API load: 87% fewer API calls during list operations
#### **iSCSI Snapshot Deletion Optimization**
- **Brought iSCSI to parity with NVMe recursive deletion** - Consistent ~3 second deletion regardless of snapshot count
- Previously: Sequential snapshot deletion loop (50+ API calls for volumes with many snapshots)
- Now: Single recursive deletion (`recursive => true` flag) deletes all snapshots atomically
- Matches NVMe transport behavior (already optimized)
- Eliminates 50+ API calls for volumes with 50+ snapshots
### ✨ **Code Quality Improvements**
#### **Normalizer Utility Extraction**
- **Eliminated duplicate code across codebase** - Extracted `_normalize_value()` utility function
- Removed 8 duplicate normalizer closures implementing identical logic
- Single source of truth for TrueNAS API value normalization
- Handles mixed response formats: scalars, hash with parsed/raw fields, undefined values
- Bug fixes now apply consistently across all call sites
- Reduced codebase by ~50 lines of duplicate code
#### **Performance Constants Documentation**
- **Documented timing parameters with rationale** - Defined 7 named constants for timeouts and delays
- `UDEV_SETTLE_TIMEOUT_US` (250ms) - udev settle grace period
- `DEVICE_READY_TIMEOUT_US` (100ms) - device availability check
- `DEVICE_RESCAN_DELAY_US` (150ms) - device rescan stabilization
- `DEVICE_SETTLE_DELAY_S` (1s) - post-connection/logout stabilization
- `JOB_POLL_DELAY_S` (1s) - job status polling interval
- `SNAPSHOT_DELETE_TIMEOUT_S` (15s) - snapshot deletion job timeout
- `DATASET_DELETE_TIMEOUT_S` (20s) - dataset deletion job timeout
- **Impact**: Self-documenting code, easier performance tuning, prevents arbitrary value changes
### **Technical Details**
**Modified functions**:
- `_list_images_iscsi()` (lines 3529-3592) - Batch dataset fetching with hash lookup
- `_list_images_nvme()` (lines 3650-3707) - Batch dataset fetching with hash lookup
- `_free_image_iscsi()` - Changed to recursive deletion (matches NVMe behavior)
- `_normalize_value()` (lines 35-44) - New utility function for API response normalization
**Performance testing**:
- Benchmark script created for automated testing with 10/50/100 volumes
- Baseline measurements established before optimization
- Post-optimization measurements confirmed 7.5x improvement for 100 volumes
- All tests validated on TrueNAS SCALE 25.10.0 with NVMe/TCP transport
### **Real-World Impact**
| Deployment Size | Before | After | Time Saved | Speedup |
|-----------------|--------|-------|------------|---------|
| Small (10 VMs) | 2.3s | 1.7s | 0.6s | 1.4x |
| Medium (50 VMs) | 6.7s | 1.8s | 4.9s | 3.7x |
| Large (100 VMs) | 18.2s | 2.4s | 15.8s | 7.5x |
| Enterprise (1000 VMs) | ~182s (3min) | ~24s | ~158s (2.6min) | ~8x |
**User experience improvements**:
- Proxmox Web UI storage view refreshes 7.5x faster for large deployments
- Reduced risk of timeouts in large environments
- Lower API load on TrueNAS servers (87% fewer API calls)
- Better responsiveness during storage operations
---FIO installation: ✓ fio-3.39
Storage configuration: ✓ Valid (nvme-tcp mode)
Finding available VM ID: ✓ Using VM ID 990
Allocating 10GB test volume: ✓ tn-nvme:vol-fio-bench-1762485897-ns07f60098-5449-4d2e-812e-01ca586d78ca
Waiting for device (5s): ✓ Ready
Detecting device path: ✓ /dev/nvme3n20
Validating device is unused: ✓ Device is safe to test
Starting FIO benchmarks (30 tests, 25-30 minutes total)...
Transport mode: nvme-tcp (testing QD=1, 16, 32, 64, 128)
Sequential Read Bandwidth Tests: [1-5/30]
Queue Depth = 1: ✓ 859.56 MB/s
Queue Depth = 16: ✓ 3.26 GB/s
Queue Depth = 32: ✓ 4.02 GB/s
Queue Depth = 64: ✓ 3.91 GB/s
Queue Depth = 128: ✓ 3.18 GB/s
Sequential Write Bandwidth Tests: [6-10/30]
Queue Depth = 1: ✓ 488.73 MB/s
Queue Depth = 16: ✓ 363.00 MB/s
Queue Depth = 32: ✓ 351.03 MB/s
Queue Depth = 64: ✓ 354.49 MB/s
Queue Depth = 128: ✓ 354.86 MB/s
Random Read IOPS Tests: [11-15/30]
Queue Depth = 1: ✓ 7,138 IOPS
Queue Depth = 16: ✓ 66,523 IOPS
Queue Depth = 32: ✓ 70,804 IOPS
Queue Depth = 64: ✓ 68,579 IOPS
Queue Depth = 128: ✓ 70,676 IOPS
Random Write IOPS Tests: [16-20/30]
Queue Depth = 1: ✓ 3,712 IOPS
Queue Depth = 16: ✓ 3,592 IOPS
Queue Depth = 32: ✓ 3,449 IOPS
Queue Depth = 64: ✓ 3,422 IOPS
Queue Depth = 128: ✓ 3,420 IOPS
Random Read Latency Tests: [21-25/30]
Queue Depth = 1: ✓ 127.05 µs
Queue Depth = 16: ✓ 239.92 µs
Queue Depth = 32: ✓ 450.72 µs
Queue Depth = 64: ✓ 915.79 µs
Queue Depth = 128: ✓ 1.88 ms
Mixed 70/30 Workload Tests: [26-30/30]
Queue Depth = 1: ✓ R: 4,615 / W: 1,974 IOPS
Queue Depth = 16: ✓ R: 1,282 / W: 548 IOPS
Queue Depth = 32: ✓ R: 2,193 / W: 932 IOPS
Queue Depth = 64: ✓ R: 2,907 / W: 1,243 IOPS
Queue Depth = 128: ✓ R: 835 / W: 358 IOPS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Benchmark Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total tests run: 30
Completed: 30
Top Performers:
Sequential Read: 859.56 MB/s (QD=1 )
Sequential Write: 488.73 MB/s (QD=1 )
Random Read IOPS: 70,804 IOPS (QD=32 )
Random Write IOPS: 3,712 IOPS (QD=1 )
Lowest Latency: 1.88 ms (QD=128) FIO installation: ✓ fio-3.39
Storage configuration: ✓ Valid (nvme-tcp mode)
Finding available VM ID: ✓ Using VM ID 990
Allocating 10GB test volume: ✓ tn-nvme:vol-fio-bench-1762485897-ns07f60098-5449-4d2e-812e-01ca586d78ca
Waiting for device (5s): ✓ Ready
Detecting device path: ✓ /dev/nvme3n20
Validating device is unused: ✓ Device is safe to test
Starting FIO benchmarks (30 tests, 25-30 minutes total)...
Transport mode: nvme-tcp (testing QD=1, 16, 32, 64, 128)
Sequential Read Bandwidth Tests: [1-5/30]
Queue Depth = 1: ✓ 859.56 MB/s
Queue Depth = 16: ✓ 3.26 GB/s
Queue Depth = 32: ✓ 4.02 GB/s
Queue Depth = 64: ✓ 3.91 GB/s
Queue Depth = 128: ✓ 3.18 GB/s
Sequential Write Bandwidth Tests: [6-10/30]
Queue Depth = 1: ✓ 488.73 MB/s
Queue Depth = 16: ✓ 363.00 MB/s
Queue Depth = 32: ✓ 351.03 MB/s
Queue Depth = 64: ✓ 354.49 MB/s
Queue Depth = 128: ✓ 354.86 MB/s
Random Read IOPS Tests: [11-15/30]
Queue Depth = 1: ✓ 7,138 IOPS
Queue Depth = 16: ✓ 66,523 IOPS
Queue Depth = 32: ✓ 70,804 IOPS
Queue Depth = 64: ✓ 68,579 IOPS
Queue Depth = 128: ✓ 70,676 IOPS
Random Write IOPS Tests: [16-20/30]
Queue Depth = 1: ✓ 3,712 IOPS
Queue Depth = 16: ✓ 3,592 IOPS
Queue Depth = 32: ✓ 3,449 IOPS
Queue Depth = 64: ✓ 3,422 IOPS
Queue Depth = 128: ✓ 3,420 IOPS
Random Read Latency Tests: [21-25/30]
Queue Depth = 1: ✓ 127.05 µs
Queue Depth = 16: ✓ 239.92 µs
Queue Depth = 32: ✓ 450.72 µs
Queue Depth = 64: ✓ 915.79 µs
Queue Depth = 128: ✓ 1.88 ms
Mixed 70/30 Workload Tests: [26-30/30]
Queue Depth = 1: ✓ R: 4,615 / W: 1,974 IOPS
Queue Depth = 16: ✓ R: 1,282 / W: 548 IOPS
Queue Depth = 32: ✓ R: 2,193 / W: 932 IOPS
Queue Depth = 64: ✓ R: 2,907 / W: 1,243 IOPS
Queue Depth = 128: ✓ R: 835 / W: 358 IOPS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Benchmark Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total tests run: 30
Completed: 30
Top Performers:
Sequential Read: 859.56 MB/s (QD=1 )
Sequential Write: 488.73 MB/s (QD=1 )
Random Read IOPS: 70,804 IOPS (QD=32 )
Random Write IOPS: 3,712 IOPS (QD=1 )
Lowest Latency: 1.88 ms (QD=128)FIO Storage Benchmark Running benchmark on storage: truenas-storage FIO installation: ✓ fio-3.39 Storage configuration: ✓ Valid (iscsi mode) Finding available VM ID: ✓ Using VM ID 990 Allocating 10GB test volume: ✓ truenas-storage:vol-fio-bench-1762536334-lun11 Waiting for device (5s): ✓ Ready Detecting device path: ✓ /dev/mapper/mpatho Validating device is unused: ✓ Device is safe to test Starting FIO benchmarks (30 tests, 25-30 minutes total)... Transport mode: iscsi (testing QD=1, 16, 32, 64, 128) Sequential Read Bandwidth Tests: [1-5/30] Queue Depth = 1: ✓ 571.32 MB/s Queue Depth = 16: ✓ 2.11 GB/s Queue Depth = 32: ✓ 2.10 GB/s Queue Depth = 64: ✓ 2.11 GB/s Queue Depth = 128: ✓ 2.14 GB/s Sequential Write Bandwidth Tests: [6-10/30] Queue Depth = 1: ✓ 499.40 MB/s Queue Depth = 16: ✓ 347.63 MB/s Queue Depth = 32: ✓ 329.68 MB/s Queue Depth = 64: ✓ 337.36 MB/s Queue Depth = 128: ✓ 319.98 MB/s Random Read IOPS Tests: [11-15/30] Queue Depth = 1: ✓ 4,854 IOPS Queue Depth = 16: ✓ 57,736 IOPS Queue Depth = 32: ✓ 84,720 IOPS Queue Depth = 64: ✓ 103,969 IOPS Queue Depth = 128: ✓ 105,377 IOPS Random Write IOPS Tests: [16-20/30] Queue Depth = 1: ✓ 3,736 IOPS Queue Depth = 16: ✓ 3,197 IOPS Queue Depth = 32: ✓ 3,131 IOPS Queue Depth = 64: ✓ 3,109 IOPS Queue Depth = 128: ✓ 3,131 IOPS Random Read Latency Tests: [21-25/30] Queue Depth = 1: ✓ 208.30 µs Queue Depth = 16: ✓ 275.67 µs Queue Depth = 32: ✓ 377.25 µs Queue Depth = 64: ✓ 632.02 µs Queue Depth = 128: ✓ 1.24 ms Mixed 70/30 Workload Tests: [26-30/30] Queue Depth = 1: ✓ R: 3,552 / W: 1,519 IOPS Queue Depth = 16: ✓ R: 8,905 / W: 3,822 IOPS Queue Depth = 32: ✓ R: 7,232 / W: 3,110 IOPS Queue Depth = 64: ✓ R: 7,154 / W: 3,077 IOPS Queue Depth = 128: ✓ R: 7,151 / W: 3,076 IOPS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Benchmark Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Total tests run: 30 Completed: 30 Top Performers: Sequential Read: 2.14 GB/s (QD=128) Sequential Write: 499.40 MB/s (QD=1 ) Random Read IOPS: 105,377 IOPS (QD=128) Random Write IOPS: 3,736 IOPS (QD=1 ) Lowest Latency: 208.30 µs (QD=1 ) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ |
roxmoxroxmox-nvmeapparently I was using version 1.0.6... I must have run the install script or something that reverted it, I will manually upgrade. And I will retest.