Proxmox with iscsi Setup

AS-eurus · Mar 20, 2024

Hi Guys,

I am very new for proxmox and planning to use it for my private cloud so i have installed proxmox latest version and it is flawless and very smooth. Now i am trying to use iscsi storage option but unable to do.

I have installed a iscsi target on Debian-12 from this

https://www.server-world.info/en/note?os=Debian_12&p=iscsi&f=1

And proxmox able to discovered but unable to connect to iscsi target.

1. I have attached /etc/iscsi/iscsid.conf conf and proxmox web storage target info..
2. CHAP username/password is: admin/admin for > target: iqn.2024-03.19proxmox.srv:dlp.target01
3. CHAP username/password is: username/password > target: iqn.2024-03.20disk02.srv:dlp.target02

Please advise and guide me to configure it!

Many Thanks.
AS

bbgeek17 · Mar 20, 2024

AS-eurus said:
And proxmox able to discovered but unable to connect to iscsi target.

Can you explain what "unable to connect" means exactly? Have you examined the log - "journalctl -n 500" immediately after connection/configuration attempt?

AS-eurus said:
1. I have attached /etc/iscsi/iscsid.conf conf and proxmox web storage target info..
2. CHAP username/password is: admin/admin for > target: iqn.2024-03.19proxmox.srv:dlp.target01
3. CHAP username/password is: username/password > target: iqn.2024-03.20disk02.srv:dlp.target02

At first glance the context of the config file does not match description above.
You only have one set of defined authentication material where you probably should have per target config. I am also not sure the last two lines are legal.
You should refer to the iscsid.conf man page for more information: https://linux.die.net/man/5/iscsi.conf

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

AS-eurus · Mar 20, 2024

Hi bbgeek17,

Thanks for your response and looking into this. I have attached logs.

Thanks,
AS

bbgeek17 · Mar 20, 2024

The logs are pretty explicit:

Code:

Mar 20 12:36:25 proxmox1 pvedaemon[895]: command '/usr/bin/iscsiadm --mode node --targetname iqn.2024-03.20disk02.srv:dlp.target02 --login' failed: exit code 24
Mar 20 12:36:25 proxmox1 iscsid[117502]: Connection15314:0 to [target: iqn.2024-03.20disk02.srv:dlp.target02, portal: 192.168.2.132,3260] through [iface: default] is shutdown.
Mar 20 12:36:25 proxmox1 iscsid[117502]: connection15315:0 login rejected: initiator failed authorization with target
Mar 20 12:36:25 proxmox1 iscsid[117502]: Connection15315:0 to [target: iqn.2024-03.20disk02.srv:dlp.target02, portal: 192.168.2.132,3260] through [iface: default] is shutdown.
Mar 20 12:36:25 proxmox1 iscsid[117502]: connection15316:0 login rejected: initiator failed authorization with target
Mar 20 12:36:25 proxmox1 iscsid[117502]: Connection15316:0 to [target: iqn.2024-03.20disk02.srv:dlp.target02, portal: 192.168.2.132,3260] through [iface: default] is shutdown.
Mar 20 12:36:25 proxmox1 iscsid[117502]: connection15317:0 login rejected: initiator failed authorization with target
Mar 20 12:36:25 proxmox1 iscsid[117502]: Connection15317:0 to [target: iqn.2024-03.20disk02.srv:dlp.target02, portal: 192.168.2.132,3260] through [iface: default] is shutdown.

Authentication induced issues, as suspected.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

AS-eurus · Mar 20, 2024

Yes, but all CHAP auth is fine which is why i am not getting where to look now? Please find CHAP auth settings.

# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
node.session.auth.authmethod = CHAP

# To configure which CHAP algorithms to enable, set
# node.session.auth.chap_algs to a comma separated list.
# The algorithms should be listed in order of decreasing
# preference — in particular, with the most preferred algorithm first.
# Valid values are MD5, SHA1, SHA256, and SHA3-256.
# The default is MD5.
node.session.auth.chap_algs = SHA3-256,SHA256,SHA1,MD5

# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
node.session.auth.username = eurus
node.session.auth.password = eurus

# To set a CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#node.session.auth.username_in = username_in
#node.session.auth.password_in = password_in
#node.session.auth.username_in = eurus
#node.session.auth.password_in = eurus

AS-eurus · Mar 20, 2024

Hi @bbgeek17,

Let me explain once again what i have done till now...

Node-1 - Proxmox installed.
Node-2 (Debian12) - I have installed an iscsi target on Debian-12 from this and configured CHAP auth in Node1 iscsi.conf.

https://www.server-world.info/en/note?os=Debian_12&p=iscsi&f=1

Do i need to do anything else or i am on the right path?

Many Thanks,
AS

bbgeek17 · Mar 20, 2024

You said that you have two separate sets of username/passwords for two targets

AS-eurus said:
2. CHAP username/password is: admin/admin for > target: iqn.2024-03.19proxmox.srv:dlp.target01
3. CHAP username/password is: username/password > target: iqn.2024-03.20disk02.srv:dlp.target02

That means that you should have two Option J entries.
If you have changed the configuration since your original post, then you need to adjust the config file and restart iscsid daemon.

AS-eurus said:
Do i need to do anything else or i am on the right path?

The article you are referring to describes the iSCSI Target configuration. I am assuming that you have done that properly (it is outside of scope of this forum). The issue you are having is on iSCSI Initiator side.

This is not a Proxmox/PVE issue, but basic iSCSI configuration challenge. If you cant connect via "iscsiadm" directly from command line, then PVE (which uses the same exact tool) wont be able to connect either.
Take a look at these (or similar) articles and follow the steps to get your environment in order:
https://access.redhat.com/documenta... to the target with the displayed target IQN:
https://ubuntu.com/server/docs/service-iscsi#:~:text=discovering the targets
https://www.seagate.com/manuals/lyv...=Discover and map the initiator to the target

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

AS-eurus · Mar 20, 2024

bbgeek17 said:
You said that you have two separate sets of username/passwords for two targets

That means that you should have two Option J entries.
If you have changed the configuration since your original post, then you need to adjust the config file and restart iscsid daemon.

The article you are referring to describes the iSCSI Target configuration. I am assuming that you have done that properly (it is outside of scope of this forum). The issue you are having is on iSCSI Initiator side.

This is not a Proxmox/PVE issue, but basic iSCSI configuration challenge. If you cant connect via "iscsiadm" directly from command line, then PVE (which uses the same exact tool) wont be able to connect either.
Take a look at these (or similar) articles and follow the steps to get your environment in order:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_storage_devices/configuring-an-iscsi-initiator_managing-storage-devices#:~:text=Discover the target and log in to the target with the displayed target IQN:
https://ubuntu.com/server/docs/service-iscsi#:~:text=discovering the targets
https://www.seagate.com/manuals/lyve-mobile-rackmount-receiver/iscsi-network-setup-linux-ubuntu-debian/#:~:text=Discover and map the initiator to the target

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Hi bbgeek17,

>> You said that you have two separate sets of username/passwords for two targets

yes, i said because i am trying to different username/password on different targets.

>> If you have changed the configuration since your original post, then you need to adjust the config file and restart iscsid daemon.

Yes, i did.

>> I am not saying it is proxmox issue, i am just looking help to figure out this issue and use Proxmox for my Private cloud.

Many Thanks,
AS

AS-eurus · Mar 21, 2024

Hi @bbgeek17,

I am able to add iscsi target in proxmox. And i want to use this storage for VM but when creating VM it did not show up on the storage allocation list! What could be the problem?

pvesm status
Name Type Status Total Used Available %
Iscsi-01 iscsi active 0 0 0 0.00%
Iscsi-192-Vol_1 lvm active 26210304 0 26210304 0.00%
local dir active 98497780 6023480 87424752 6.12%
local-lvm lvmthin active 833388544 4666975 828721568 0.56%

I have attached screenshots of available stroage and storage allocation list for your reference.

Thanks,
AS

bbgeek17 · Mar 21, 2024

AS-eurus said:
I am able to add iscsi target in proxmox. And i want to use this storage for VM but when creating VM it did not show up on the storage allocation list! What could be the problem?

Probably because you are in ISO/OS selection portion of the wizard where you can pick an ISO file from file system based storage.
Since iSCSI is block based you cant use it here. You can select "none" for the CD and move along to System and then DISK portion where you may be able to pick storage location, if you did all prerequisites properly.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

AS-eurus · Mar 22, 2024

Hi @bbgeek17,

Yes, you are absolutely right. I am able to find iscsi storage now and spin a VM as well.

One more question is that can i add multiple iscsi-storage with different chap auth? Actually, i have tried but unable to get success till now.

Many Thanks,
AS

bbgeek17 · Mar 22, 2024

AS-eurus said:
One more question is that can i add multiple iscsi-storage with different chap auth? Actually, i have tried but unable to get success till now.

see here: https://forum.proxmox.com/threads/iscsi-with-chap.120336/#post-522874

in summary -there is no "easy" way.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

fireon · Sep 19, 2024

I'm on the topic too. But I just can't figure out how and where to configer it. One auth ok, but two? How was it actually solved? Do you have a working example?

fireon · Sep 19, 2024

I've now taken a big step forward. Basically, the targetcli on Debian 12 has a problem with Auth. Because the exact same thing works on a built TrueNAS.
And this is how it looks:

2 targets set up at Truenas:

iqn.2005-10.org.freenas.ctl:target1
iqn.2005-10.org.freenas.ctl:target2

+ 2 different users with different passwords, one user for one target

If i setup in Proxmox chap auth in /etc/iscsi/iscsid.conf it is working perfectly, but yes only for that target that i set username and password. Because this data is then rewritten to the respective defaults every minute:

Code:

ls -l /etc/iscsi/nodes         
total 8
drwx------ 2 root root 4096 Sep 19 16:48 iqn.2005-10.org.freenas.ctl:target1
drwx------ 2 root root 4096 Sep 19 16:48 iqn.2005-10.org.freenas.ctl:target2

Code:

├── initiatorname.iscsi
├── iscsid.conf
├── nodes
│ ├── iqn.2005-10.org.freenas.ctl:target1
│   │   └── 192.168.70.17,3260
│ └── iqn.2005-10.org.freenas.ctl:target2
│       └── 192.168.70.17,3260

The files could be edited manually, but all changes were gone almost 3 seconds later. Only settings in the iscsi.conf are written to the two target configs. As a next attempt, I used iscsiadm to write the chap, username and password into the configs:

Code:

iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:target1 -p 192.168.70.17:3260 --op update -n node.session.auth.authmethod -v CHAP
iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:target1 -p 192.168.70.17:3260 --op update -n node.session.auth.username -v supertux1
iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:target1 -p 192.168.70.17:3260 --op update -n node.session.auth.password -v 123456789012

But even with that, the file was immediately overwritten every time. Next, I stopped the iscsid on Proxmox, changed the file with the above commands and everything was stored cleanly.
Now I changed the permissions of the two target configs to 400. I restarted the iscsi service on Proxmox. And again, everything was overwritten. (really WTF

).

So the question that arises for me is: “How do I configure /etc/iscsi/iscsid.conf with 2 access data for the targets?”

Also tested:

Code:

iqn.2005-10.org.freenas.ctl:target1 = {
  node.session.auth.username = supertux1
  node.session.auth.password = 123456789012
}

iqn.2005-10.org.freenas.ctl:target2 = {
  node.session.auth.username = supertux2
  node.session.auth.password = 12345678901212
}

There must be a way to configure this...

bbgeek17 · Sep 19, 2024

Hi @fireon ,

fireon said:
So the question that arises for me is: “How do I configure /etc/iscsi/iscsid.conf with 2 access data for the targets?”

It's been a while and we dont use manual configurations at Blockbridge, but I am pretty sure I was able to succesfully configure multiple targets with disparate authentications by following these instructions:

Options of Category J:

TargetName=<name> <name> is iSCSI Target name in iqn or eui format. Target specific settings should be entered below the respective TargetName entry. These target-specific settings will be applicable for all iSCSI sessions to the target.

In your example you are using ={}, I dont think thats a valid format. I could be wrong!

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

fireon · Sep 19, 2024

Hi @bbgeek17 and thanks for your answer, I've tested no multible options:

Code:

TargetName=iqn.2005-10.org.freenas.ctl:target1 {
  node.session.auth.username = supertux1
  node.session.auth.password = 123456789012
}

iqn.2005-10.org.freenas.ctl:target1 {
  node.session.auth.username = supertux1
  node.session.auth.password = 123456789012
}

TargetName=iqn.2005-10.org.freenas.ctl:target1
node.session.auth.username = supertux1
node.session.auth.password = 123456789012

If you believe the man page, the last would be the right one...

fireon · Sep 20, 2024

https://bugzilla.proxmox.com/show_bug.cgi?id=5732

Proxmox with iscsi Setup

New Member

Attachments

Distinguished Member

New Member

Attachments

Distinguished Member

New Member

New Member

Distinguished Member

New Member

New Member

Attachments

Distinguished Member

New Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Options of Category J:​

Distinguished Member

Distinguished Member

Options of Category J: