Info
PVE: Proxmox 8 - 6.8.12-12-pveNAS: FreeBSD 14.3-RELEASE releng/14.3-n271432-8c9ce319fef7 GENERIC amd64
ISCSI PKG: istgt - Version: 20150713_1
Preface
I am setting up a new PVE. Currently it consists of the pve server and a freebsd server to be used for storage. It is possible I have setup the configs for istgt incorrectly, but the/usr/local/etc/istgt/istgt.conf and /usr/local/etc/istgt/auth.conf are pretty straight forward.Research/Docs
I have searched the forum and gone through all the posts (to about 2016) that relate to "zfs over iscsi" and "istgt". There is very little on FreeBSD and proxmox and mostly I see users using Debian and istgt. I could go the route of installing Debian on my storage server, but I had wanted to set up zfs on FreeBSD since the zfs developers write openzfs specifically for bsd and I wanted to manage a bsd box.In the forum I found the following:
- This github repo: freenas-proxmox that people seemed to use (it hasn't had a commit in about 2 years and isn't what I want)
- An old ctld (native iscsi freebsd) patch/implementation repo for managing zfs over iscsi - may work would have to test
I am able to setup and connect successfully to the storage using:
Code:
zfs: freenas
blocksize 4k
target iqn.2025-10.com.<subdomain>.free-nas:vms
pool tank
iscsiprovider istgt
portal 192.168.1.101
content imagesIssue
The issue happens when creating a vm or multiple (not sure about containers, but since they require disks I would assume so). When creating a single vm I am able to create it successfully, but/usr/share/perl5/PVE/Storage/LunCmd/Istgt.pm mangles my /usr/local/etc/istgt/istgt.conf on the freebsd server and creates a broken config.It removes all comments in the "[LogicalUnit1]" section, a few parameters such as "Comment" and "TargetAlias", and adds "QueueDepth" without a number following and that breaks the config. This happens when a single vm is created.
When I then go to create a second vm since things are in a broken state (at that time you don't know it's broken since the proxmox GUI console appears fine), various problems happen like the LUN entry for the disk is written incorrectly, but a zvol/device is created for the disk, and the vm doesn't start giving a generic: "Error: start failed: QEMU exited with code 1".
I cleaned up the disk and configs and tried to start the second vm with the qm command:
Code:
root@pve-01:~# qm start 102
Odd number of elements in anonymous hash at /usr/share/perl5/PVE/Storage/LunCmd/Istgt.pm line 324.
Odd number of elements in anonymous hash at /usr/share/perl5/PVE/Storage/LunCmd/Istgt.pm line 324.
Odd number of elements in anonymous hash at /usr/share/perl5/PVE/Storage/LunCmd/Istgt.pm line 324.
kvm: -drive file=iscsi://192.168.1.101/iqn.2025-10.com.<subdomain>.free-nas:vms/2,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on: iSCSI: Failed to connect to LUN : SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:LOGICAL_UNIT_NOT_SUPPORTED(0x2500)
start failed: QEMU exited with code 1vms/2 for the file=iscsi://.I will need to do some more systematic testing to hone in on what is happening, but I can be sure that QueueDepth is handled incorrectly by the parser sub routine:
Perl:
my $lun_dumper = sub {
my ($lun) = @_;
my $config = '';
$config .= "\n[$lun]\n";
$config .= 'TargetName ' . $SETTINGS->{$lun}->{TargetName} . "\n";
$config .= 'Mapping ' . $SETTINGS->{$lun}->{Mapping} . "\n";
$config .= 'AuthGroup ' . $SETTINGS->{$lun}->{AuthGroup} . "\n";
$config .= 'UnitType ' . $SETTINGS->{$lun}->{UnitType} . "\n";
$config .= 'QueueDepth ' . $SETTINGS->{$lun}->{QueueDepth} . "\n";
foreach my $conf (@{$SETTINGS->{$lun}->{luns}}) {
$config .= "$conf->{lun} Storage " . $conf->{Storage};
$config .= ' ' . $size_with_unit->($conf->{Size}) . "\n";
foreach ($conf->{options}) {
if ($_) {
$config .= "$conf->{lun} Option " . $_ . "\n";
}
}
}
$config .= "\n";
return $config;
};
...
my $parser = sub {
...
}I think there is a regex problem as well. In that same file, we see the base for regex:
Perl:
sub get_base {
return '/dev/zvol';
}then in the parser subroutine:
Perl:
my $base = get_base;
...
if ($storage =~ /^$base\/$scfg->{pool}\/([\w\-]+)$/) {
#print "key: $key, storage: $storage, size: $size, options: @options\n";
$conf = {
lun => $key,
Storage => $storage,
Size => $size,
options => "",
}
}the if conditional and regex should match my zvol:
/dev/zvol/tank/vms and create/delete/add luns in the subsequent directory, but it doesn't as we can see on my freebsd server:
Code:
➜ zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 349G 24.0T 189K /tank
tank/vm-101-disk-0 175G 24.2T 605M -
tank/vm-102-disk-0 175G 24.2T 94.3K -
tank/vms 3.26M 24.0T 94.3K -What Steps Can I Take?
I will delete vms, disks, start over, and troubleshoot in a more thorough way.The source code for this is here and Thomas Lamprecht seems to be the most recent contributor with Fiona Ebner makeing a couple commits. Michael Rasmussen looks to be the original author, but hasn't been active since 2014. Should I reach out and work with Fiona or Thomas?
I am happy to help code and test, but I am unfamiliar with the Debian dependencies since this seems to have cross plugin functionality. Therefore a change that works for FreeBSD could break something for Debian.