ZFS over iSCSI error 500

[milennium]

Hello,

I configured ZFS over iscsi on my Proxmox cluster:

zfs: zfs-iscsi
        blocksize 4k
        iscsiprovider LIO
        pool Tanky/proxmox
        portal 192.168.***.***
        target iqn.2003-01.org.linux-iscsi.plfl03.x8664:sn.ac6ec8d50945
        content images
        lio_tpg tpg1
        nowritecache 1
        sparse 1

When I try to create a new disk I have an error message:

update VM 450105: -scsi1 zfs-iscsi:32,iothread=on
TASK ERROR: Could not open /dev/Tanky/proxmox/vm-450105-disk-0

VM disks were created by Proxmox on the target:

****@plfl03:~$ zfs list -r Tanky/proxmox
NAME                             USED  AVAIL  REFER  MOUNTPOINT
Tanky/proxmox                    152K  10.7T    96K  /mnt/Tanky/proxmox
Tanky/proxmox/vm-450105-disk-0    56K  10.7T    56K  -

There is no folder /dev/Tanky on any node, and creating the folder doesnt change the result.

Did i miss something?

Regards

 

[bbgeek17]

I suspect the error message is coming from the iSCSI target side, not from PVE node. I.e. that path is missing on the ZFS/iSCSI server.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[bbgeek17]

Take a look at sample configurations here: https://pve.proxmox.com/wiki/Storage:_ZFS_over_ISCSI
See if you can drop "Tanky/" in your PVE config, or "proxmox" - not sure what is what in your config.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

portal 192.168.***.***

Security first

There is no folder /dev/Tanky on any node, and creating the folder doesnt change the result.

That is strange indeed. Is there a lowercase tanky?

 

[milennium]

I suspect the error message is coming from the iSCSI target side, not from PVE node. I.e. that path is missing on the ZFS/iSCSI server.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I checked and on Target side, zvol were mounted in /dev/zvol/Tanky. I created a softlink /dev/Tanky and it worked.

Thank you very much.

 

[milennium]

Security first


That is strange indeed. Is there a lowercase tanky?

I double checked this case , but thank you for your help.

 

[gfngfn256]

TASK ERROR: Could not open /dev/Tanky/proxmox/vm-450105-disk-0

A raw block device on ZFS ie ZVOL will be /dev/zvol/pool/name as per docs.
So is this to be seen as a bug with PVE (through iSCSI target) trying to open /dev/pool/name?

I checked and on Target side, zvol were mounted in /dev/zvol/Tanky. I created a softlink /dev/Tanky and it worked.

This is nothing but a crude workaround - shouldn't be necessary & may break.

One possibility why this has happened to OP, maybe because of his pool naming Tanky/proxmox, I wonder if it would happen if he chose a pool name without a forward slash ( / ), such as tanky or proxmox. Possibly a confusion is created by the forward slash.

This should be tested by someone - I personally don't use such a setup.

 

[milennium]

A raw block device on ZFS ie ZVOL will be /dev/zvol/pool/name as per docs.

It s the case on target, but on initiator, proxmox seems to expect /dev/pool/name.

This is nothing but a crude workaround - shouldn't be necessary & may break.

I agree!

One possibility why this has happened to OP, maybe because of his pool naming Tanky/proxmox, I wonder if it would happen if he chose a pool name without a forward slash ( / ), such as tanky or proxmox. Possibly a confusion is created by the forward slash.

I tried both. I began with a forward slash, tried without it and came back to initial setup.

 

[LnxBil]

It s the case on target, but on initiator, proxmox seems to expect /dev/pool/name.

AFAIK, this is the default with OpenZFS on Linux. I cannot say that I've seen it differently, at least I cannot remember. It has always been in /dev/<zpool name>/.

 

[milennium]

AFAIK, this is the default with OpenZFS on Linux. I cannot say that I've seen it differently, at least I cannot remember. It has always been in /dev/<zpool name>/.

It seems that Debian based are a little different on this topic.
In my case, the target is a Ubuntu 24.04.

And you can see it also Debian s documentation

 

[LnxBil]

In my case, the target is a Ubuntu 24.04.

Ah okay ... that could be the case. Ubuntu integrated it differently as Debian does.

 

[gfngfn256]

AFAIK, this is the default with OpenZFS on Linux

I already (above) linked Debian docs to the contrary, here is quote from official OpenZFS docs showing the same expected behavior:

zfs create [-ps] [-b blocksize] [-o property=value]… -V size volumeCreates a volume of the given size. The volume is exported as a block device in /dev/zvol/path, where path is the name of the volume in the ZFS namespace. The size represents the logical size as exported by the device. By default, a reservation of equal size is created.

I checked Oracle (pretty old!) & a few other distros all the same expected behavior.

I checked Ubuntu but couldn't find anything official, however on forums/posts etc. the same expected behavior is shown.

So it should be always
/dev/zvol/pool

Conclusion: A bug report should be opened with Proxmox. This thread should NOT be tagged [SOLVED] because it isn't solved.

 

[LnxBil]

I already (above) linked Debian docs to the contrary, here is quote from official OpenZFS docs showing the same expected behavior:

Thank you for reporting back. You're right and I missunderstood the point.

It's odd that Debian and Proxmox VE has the same "view" of this. They use different OpenZFS versions and implementations (Debian has the upstream dkms module and PVE ships it with its own kernel, that is in fact an Ubuntu LTS kernel). Maybe this is some legacy view on the topic?

I checked Oracle (pretty old!) & a few other distros all the same expected behavior.

I also use (a pretty old) Oracle Linux with ZFS and there IS in fact the directory /dev/<poolname> present and working without any intervention (besides the /dev/zvol/<poolname> of course).

I checked Ubuntu but couldn't find anything official, however on forums/posts etc. the same expected behavior is shown.

So it should be always
/dev/zvol/pool

Conclusion: A bug report should be opened with Proxmox. This thread should NOT be tagged [SOLVED] because it isn't solved.

Yes, I concur and there is already a bugreport about this.

 

[KciNicK]

this also happens if the target is a proxmox ve server