I have been testing this plug in in on a sigle host using a multipath implementation over redundant 10g links. The plug in seems to work very well and I am getting great performance on my iSCSI access to my TrueNAS test system that is using a 2x4 mirrored vdev storage array shared to iSCSI.
However, I am seeing the pvestatd service crashing regularly and causing the proxmox GUI to have all Grey indicators until the pvestatd service is restarted.
In my setup the proxmox system is managed on 192.168.239.111, and the TrueNAS is managed on 192.168.239.56
The iSCSI setup has each system using SPF+ links via a 10g switch - two VLAN's that are isolated - 172.16.88.x, and 172.16.99.x The Proxmox nics are 172.16.88.20, and 172.16.99.20, the TrueNAS nics are 172.16.88.10, and 172.16.99.10.
Network connectivity is solid and and data trasfer rates tested with iperf3 and fio are in the 10g range - fio confirms parralell path using round robin is functional and working, as well as fault checking ( pulling a cable ) does work as well. Multiple VMs configured with disks on the TrueNAS iSCSI array work fine and performance is good.
Proxmox is fully updated with non-subscription repository - full listing of packages installed below.
TrueNAS is a fresh install with current version 25.04.2.5
I have also provided a partial output of "journalctl -efu pvestatd" below, but I am not an expert so I am not sure what I am seeing with the errors indicated.
Is there way to solve this?
I have configure my test system with the plugin per the install instructions and confgured the following files as such:
/etc/pve/storage.conf
truenasplugin: truenas-storage
api_host 192.168.239.56
api_key 2-VDUWQaV9rQuU6rVfeJ9AU7mzy2o9kJF1saR6SGD3IEp02zVj4MHvDoz2scBmlX22
api_insecure 1
api_transport ws
api_scheme wss
api_retry_max 5
api_retry_delay 2
target_iqn iqn.2005-10.org.freenas.ctl
roxmox
dataset iSCSI_Pool/proxmox
discovery_portal 192.168.239.56:3260
portals 172.16.88.10:3260,172.16.99.10:3260
content images
shared 1
use_multipath 1
/etc/multipath.conf
defaults {
user_friendly_names yes
path_grouping_policy multibus
path_selector "round-robin 0"
rr_min_io_rq 1
failback immediate
no_path_retry queue
find_multipaths no
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode "^sda" # Exclude OS disk (adjust if needed)
}
Optional: Specific device configuration for TrueNAS
devices {
device {
vendor "TrueNAS"
product "iSCSI Disk"
path_grouping_policy multibus
path_selector "round-robin 0"
hardware_handler "0"
rr_weight uniform
rr_min_io_rq 1
}
}
Currently installed Proxmox packages:
proxmox-ve: 9.0.0 (running kernel: 6.14.11-4-pve)
pve-manager: 9.0.11 (running version: 9.0.11/3bf5476b8a4699e2)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.14.11-4-pve-signed: 6.14.11-4
proxmox-kernel-6.14: 6.14.11-4
proxmox-kernel-6.14.8-2-pve-signed: 6.14.8-2
ceph-fuse: 19.2.3-pve1
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.3.1-1+pve4
ifupdown2: 3.3.0-1+pmx10
intel-microcode: 3.20250812.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.3
libpve-apiclient-perl: 3.4.0
libpve-cluster-api-perl: 9.0.6
libpve-cluster-perl: 9.0.6
libpve-common-perl: 9.0.11
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.1.8
libpve-rs-perl: 0.10.10
libpve-storage-perl: 9.0.13
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-1
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.0.16-1
proxmox-backup-file-restore: 4.0.16-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.0
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.2
proxmox-widget-toolkit: 5.0.6
pve-cluster: 9.0.6
pve-container: 6.0.13
pve-docs: 9.0.8
pve-edk2-firmware: 4.2025.02-4
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.3
pve-firmware: 3.17-2
pve-ha-manager: 5.0.5
pve-i18n: 3.6.1
pve-qemu-kvm: 10.0.2-4
pve-xtermjs: 5.5.0-2
qemu-server: 9.0.23
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve2
vncterm: 1.9.1
zfsutils-linux: 2.3.4-pve1
partial output of "journalctl -efu pvestatd"
Oct 23 11:48:18 tarbaby2 pvestatd[19643]: Pre-flight: Failed to map weight extent: JSON-RPC error: {"code":-32602,"message":"Invalid params","data":{"trace":{"class":"ValidationErrors","frames":[{"argspec":["self","app","id_","method","params"],"line":" app.send_truenas_validation_error(id_, sys.exc_info(), list(e))\n","lineno":335,"filename":"/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py","method":"process_method_call","locals":{"method":"<middlewared.api.base.server.method.Method object at 0x7ff0a664bc50>","self":"<middlewared.api.base.server.ws_handler.rpc.RpcWebSocketHandler object at 0x7ff0ac686610>","id_":"131","app":"<middlewared.api.base.server.ws_handler.rpc.RpcWebSocketApp object at 0x7ff083adda10>","params":"[{'lunid': 0, 'target': 2, 'extent': 7}]","e":"ValidationErrors([ValidationError('iscsi_targetextent_create.lunid', 'LUN ID is already being used for this target.', 22)])"}},{"argspec":["self","app","params"],"line":" result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)\n","lineno":52,"method":"call","filename":"/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py","locals":{"self":"<middlewared.api.base.server.method.Method object at 0x7ff0a664bc50>","app":"<middlewared.api.base.server.ws_handler.rpc.RpcWebSocketApp object at 0x7ff083adda10>","methodobj":"<bound method CRUDService.create of <middlewared.plugins.iscsi_.target_to_extent.iSCSITargetToExtentService object at 0x7ff0ac7dadd0>>","params":"[{'lunid': 0, 'target': 2, 'extent': 7}]","mock":"None"}},{"line":" await log_audit_message_for_method(success)\n","argspec":["self","method","serviceobj","methodobj","params","app"],"lineno":922,"keywordspec":"kwargs","locals":{"kwargs":"{}","success":"False","app":"<middlewared.api.base.server.ws_handler.rpc.RpcWebSocketApp object at 0x7ff083adda10>","audit_callback_messages":"['proxmox/0/pve-plugin-weight']","job":"None","params":"[{'lunid': 0, 'target': 2, 'extent': 7}]","methodobj":"<bound method CRUDService.create of <middlewared.plugins.iscsi_.target_to_extent.iSCSITargetToExtentService object at 0x7ff0ac7dadd0>>","log_audit_message_for_method":"<function Middleware.call_with_audit..log_audit_message_for_method at 0x7ff0801d7ba0>","self":"<middlewared.main.Middleware object at 0x7ff0d6a2b590>","method":"'iscsi.targetextent.create'","serviceobj":"<middlewared.plugins.iscsi_.target_to_extent.iSCSITargetToExtentService object at 0x7ff0ac7dadd0>","job_on_finish_cb":"<function Middleware.call_with_audit..job_on_finish_cb at 0x7ff0801d72e0>"},"filename":"/usr/lib/python3/dist-packages/middlewared/main.py","method":"call_with_audit"},{"line":" return await methodobj(
prepared_call.args)\n","argspec":["self","name","serviceobj","methodobj","params"],"lineno":720,"keywordspec":"kwargs","locals":{"prepared_call":"PreparedCall(args=[<middlewared.api.base.server.ws_handler.rpc.RpcWebSocketApp object at 0x7ff083adda10>, <built-in method append of list object at 0x7ff083d6ea40>, {'lunid': 0, 'target': 2, 'extent': 7}], executor=<middlewared.utils.threading.IoThreadPoolExecutor object at 0x7ff0e09397d0>, job=None)","kwargs":"{'app': <middlewared.api.base.server.ws_handler.rpc.RpcWebSocketApp object at 0x7ff083adda10>, 'audit_callback': <built-in method append of list object at 0x7ff083d6ea40>}","self":"<middlewared.main.Middleware object at 0x7ff0d6a2b590>","methodobj":"<bound method CRUDService.create of <middlewared.plugins.iscsi_.target_to_extent.iSCSITargetToExtentService object at 0x7ff0ac7dadd0>>","params":"[{'lunid': 0, 'target': 2, 'extent': 7}]","name":"'iscsi.targetextent.create'","serviceobj":"<middlewared.plugins.iscsi_.target_to_extent.iSCSITargetToExtentService object at 0x7ff0ac7dadd0>"},"filename":"/usr/lib/python3/dist-packages/middlewared/main.py","method":"call"},{"line":" return await self.middleware.call(\n","argspec":["self","app","audit_callback","data"],"locals":{"self":"<middlewared.plugins.iscsi.target_to_extent.iSCSITargetToExtentService object at 0x7ff0ac7dadd0>","app":"<middlewared.api.base.server.ws_handler.rpc.RpcWebSocketApp object at 0x7ff083adda10>","data":"{'lunid': 0, 'target': 2, 'extent': 7}","audit_callback":"<built-in method append of list object at 0x7ff083d6ea40>"},"method":"create","filename":"/usr/lib/python3/dist-packages/middlewared/service/crud_service.py","lineno":256},{"keywordspec":"kwargs","lineno":720,"filename":"/usr/lib/python3/dist-packages/middlewared/main.py","method":"call","locals":{"self":"<middlewared.main.Middleware object at 0x7ff0d6a2b590>","kwargs":"{'app': <middlewared.api.base.server.ws_handler.rpc.RpcWebSocketApp object at 0x7ff083adda10>, 'audit_callback': <built-in method append of list object at 0x7ff083d6ea40>}","prepared_call":"PreparedCall(args=[<built-in method append of list object at 0x7ff083d6ea40>, {'lunid': 0, 'target': 2, 'extent': 7}], executor=<middlewared.utils.threading.IoThreadPoolExecutor object at 0x7ff0e09397d0>, job=None)","methodobj":"<function CRUDService.get_crud_wrapper_func..nf at 0x7ff0801d6e80>","name":"'iscsi.targetextent.create'","params":"[{'lunid': 0, 'target': 2, 'extent': 7}]","serviceobj":"<middlewared.plugins.iscsi.target_to_extent.iSCSITargetToExtentService object at 0x7ff0ac7dadd0>"},"argspec":["self","name","serviceobj","methodobj","params"],"line":" return await methodobj(*prepared_call.args)\n"},{"lineno":287,"keywordspec":"kwargs","locals":{"event_type":"'ADDED'","func":"<bound method iSCSITargetToExtentService.do_create of <middlewared.plugins.iscsi.target_to_extent.iSCSITargetToExtentService object at 0x7ff0ac7dadd0>>","oid":"None","self":"<middlewared.plugins.iscsi.target_to_extent.iSCSITargetToExtentService object at 0x7ff0ac7dadd0>","kwargs":"{}","args":"('', '')","action":"'create'"},"filename":"/usr/lib/python3/dist-packages/middlewared/service/crud_service.py","varargspec":"args","method":"nf","line":" rv = await func(args, kwargs)\n"},{"lineno":93,"locals":{"args_index":"2","func":"<function iSCSITargetToExtentService.do_create at 0x7ff0ad78a200>","args":"('', '', '*')","accepts":"<class 'middlewared.api.v25_04_2.iscsi_target_to_extent.IscsiTargetToExtentCreateArgs'>"},"method":"wrapped","varargspec":"args","filename":"/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py","line":" result = await func(*args)\n"},{"line":" verrors.check()\n","argspec":["self","audit_callback","data"],"locals":{"data":"{'target': 2, 'lunid': 0, 'extent': 7}","audit_callback":"<built-in method append of list object at 0x7ff083d6ea40>","verrors":"ValidationErrors([ValidationError('iscsi_targetextent_create.lunid', 'LUN ID is already being used for this target.', 22)])","self":"<middlewared.plugins.iscsi_.target_to_extent.iSCSITargetToExtentService object at 0x7ff0ac7dadd0>"},"filename":"/usr/lib/python3/dist-packages/middlewared/plugins/iscsi_/target_to_extent.py","method":"do_create","lineno":64},{"line":" raise self\n","argspec":["self"],"locals":{"self":"ValidationErrors([ValidationError('iscsi_targetextent_create.lunid', 'LUN ID is already being used for this target.', 22)])"},"method":"check","filename":"/usr/lib/python3/dist-packages/middlewared/service_exception.py","lineno":72}],"repr":"ValidationErrors([ValidationError('iscsi_targetextent_create.lunid', 'LUN ID is already being used for this target.', 22)])","formatted":"Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 323, in process_method_call\n result = await method.call(app, params)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 52, in call\n result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib/python3/dist-packages/middlewared/main.py", line 911, in call_with_audit\n result = await self._call(method, serviceobj, methodobj, params, app=app,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib/python3/dist-packages/middlewared/main.py", line 720, in _call\n return await methodobj(*prepared_call.args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 256, in create\n return await self.middleware._call(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib/python3/dist-packages/middlewared/main.py", line 720, in
call\n return await methodobj(*prepared_call.args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 287, in nf\n rv = await func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 93, in wrapped\n result = await func(*args)\n ^^^^^^^^^^^^^^^^^\n File "/usr/lib/python3/dist-packages/middlewared/plugins/iscsi/target_to_extent.py", line 64, in do_create\n verrors.check()\n File "/usr/lib/python3/dist-packages/middlewared/service_exception.py", line 72, in check\n raise self\nmiddlewared.service_exception.ValidationErrors: [EINVAL] iscsi_targetextent_create.lunid: LUN ID is already being used for this target.\n\n"},"extra":[["iscsi_targetextent_create.lunid","LUN ID is already being used for this target.",22]],"errname":"EINVAL","error":22,"reason":"[EINVAL] iscsi_targetextent_create.lunid: LUN ID is already being used for this target.\n"}} at /usr/share/perl5/PVE/Storage/Custom/TrueNASPlugin.pm line 656.
Oct 23 11:48:20 tarbaby2 pvestatd[19643]: Pre-flight: Target iqn.2005-10.org.freenas.ctl
roxmox still not discoverable after weight zvol creation
