iSCSI disk not active on all clustered nodes

B

Boss

New Member
Apr 22, 2024
24
0
1
Have added iSCSI Disk successfully to Node 1 as Thick LVM and shows as Enabled & Active. Can see it in the web GUI on the other 2 nodes where it shows as Enabled but NOT Active.
  • 1) Checked the storage.cfg file and it's the same on all nodes.
  • 2) Running a pvscan & vgscan on Node 1 displays the disk details OK but is missing on the other 2 nodes.
  • 3) Run lsscsi command shows the disk on Node 1 but not the others

# lsscsi (Node 1)
[2:0:0:0] storage HP P420i 8.32 -
[2:1:0:0] disk HP LOGICAL VOLUME 8.32 /dev/sda
[3:0:0:0] disk QNAP iSCSI Storage 4.0 /dev/sdb
[4:0:0:0] disk QNAP iSCSI Storage 4.0 /dev/sdc


# lsscsi (Other nodes)
[2:0:0:0] storage HP P420i 8.32 -
[2:1:0:0] disk HP LOGICAL VOLUME 8.32 /dev/sda
[3:0:0:0] disk QNAP iSCSI Storage 4.0 /dev/sdb


How do I make the Disk Active on the other clustered nodes so that it can be shared across all three?
Thanks.
 
Last edited:
Boss said:
How do I make the Disk Active on the other clustered nodes so that it can be shared across all three?
Thanks.
Click to expand...
It should become active when you have something on those nodes that uses the disk. Try to migrate a VM to the "bad" node, or create a new VM, backed by an LVM slice on the iSCSI.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Thanks for your tip. I tried migrating a VM from a "bad node" to the "good node" but it doesn't show the iscsi disk in the Target Storage pick box.
At the moment there is no LVM configured on that iscsi node. Should I add LVM to that disk?
 
How did you setup the iSCSI and the storage in PVE?

Normally I would need to
  • login to the portal on each node
  • check that the disks are present on each node
  • create physical volume on one node
  • create volume group on one node
  • add storage as LVM on one node in the PVE Gui and check shared
Did you do something similar?
 
Ahhh... no, I have just attached the raw iscsi lun.
So I have to create the PV & VG manually from the command line first?
As mentioned above pvscan/vgscan only shows the disk on the first node. The same with lsscsi
 
OK - so I have now removed the iscsi disk so I can start afresh.
Is there a step by step doc detailing how to add an iscsi disk and creating the LVM on top of it so that it is accessible from all 3 nodes in the cluster? There is only a single path to the iscsi LUN.
 
Revised steps from above:
  • login to the portal on each node (iscsiadm)
  • check that the disks are present on each node (lsscsi)
  • create physical volume on one node (pvcreate)
  • create volume group on one node (vgcreate)
  • add storage as LVM on one node in the PVE Gui and check shared
 
LnxBil said:
Revised steps from above:
  • login to the portal on each node (iscsiadm)
  • check that the disks are present on each node (lsscsi)
  • create physical volume on one node (pvcreate)
  • create volume group on one node (vgcreate)
  • add storage as LVM on one node in the PVE Gui and check shared
Click to expand...
I followed the steps and then I met the same problem
I hava three nodes I use iscsi on three nodes then pvcreat vgcreate then lvm named pve1 and I also checked shared.but the state of the lvm pve1 is unknown on the other two nodes
 
Last edited:
Boss said:
As mentioned above pvscan/vgscan only shows the disk on the first node. The same with lsscsi
Click to expand...
Are you doing iSCSI connection manually, i.e. directly via iscsiadm, or are you adding Proxmox iSCSI storage pool?
If it's the former, then you need to set up that connection on each node, as @LnxBil mentioned. Don't forget to mark the session to auto-start, or it won't be present on reboot.

If its the latter, please provide the output of :
- cat /etc/pve/storage.cfg
- pvesm status (from each node)
- iscsiadm -m node (from each node)
- iscsiadm -m session (from each node)
- pvesm list [iscsi_storage_name] (from each node)

Boss said:
So I have to create the PV & VG manually from the command line first?
Click to expand...
If you plan to use the iSCSI LUN as shared storage, yes you have to create the LVM structure manually. But only from one node.

Jasonhdx said:
I met the same problem. have you ever solved it?
Click to expand...
You'd need to provide the same output that I listed above. However, it'd be cleaner if you opened a separate thread.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
Are you doing iSCSI connection manually, i.e. directly via iscsiadm, or are you adding Proxmox iSCSI storage pool?
If it's the former, then you need to set up that connection on each node, as @LnxBil mentioned. Don't forget to mark the session to auto-start, or it won't be present on reboot.

If its the latter, please provide the output of :
- cat /etc/pve/storage.cfg
- pvesm status (from each node)
- iscsiadm -m node (from each node)
- iscsiadm -m session (from each node)
- pvesm list [iscsi_storage_name] (from each node)


If you plan to use the iSCSI LUN as shared storage, yes you have to create the LVM structure manually. But only from one node.


You'd need to provide the same output that I listed above. However, it'd be cleaner if you opened a separate thread.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
root@pve1:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,vztmpl,backup

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

lvm: lvm
vgname pve1
content rootdir,images
shared 1
root@pve2:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,vztmpl,backup

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

lvm: lvm
vgname pve1
content rootdir,images
shared 1
root@pve3:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,vztmpl,backup

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

lvm: lvm
vgname pve1
content rootdir,images
shared 1

root@pve1:~# pvesm status
Name Type Status Total Used Available %
local dir active 71017632 11527716 55836696 16.23%
local-lvm lvmthin active 148086784 17059597 131027186 11.52%
lvm lvm active 419426304 104857600 314568704 25.00%
root@pve2:~# pvesm status
Command failed with status code 5.
command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Name Type Status Total Used Available %
local dir active 71017632 7254868 60109544 10.22%
local-lvm lvmthin active 148086784 8648268 139438515 5.84%
lvm lvm inactive 0 0 0 0.00%
root@pve3:~# pvesm status
Command failed with status code 5.
command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Name Type Status Total Used Available %
local dir active 71017632 7255032 60109380 10.22%
local-lvm lvmthin active 148086784 8692694 139394089 5.87%
lvm lvm inactive 0 0 0 0.00%

root@pve3:~# iscsiadm -m node
10.0.1.107:3260,1 iqn.2017-08.zettastor.iqn:1113021025697932295-0
root@pve2:~# iscsiadm -m node
10.0.1.107:3260,1 iqn.2017-08.zettastor.iqn:1113021025697932295-0
root@pve1:~# iscsiadm -m node
10.0.1.107:3260,1 iqn.2017-08.zettastor.iqn:1113021025697932295-0

root@pve1:~# iscsiadm -m session
tcp: [1] 10.0.1.107:3260,1 iqn.2017-08.zettastor.iqn:1113021025697932295-0 (non-flash)

root@pve2:~# iscsiadm -m session
tcp: [4] 10.0.1.107:3260,1 iqn.2017-08.zettastor.iqn:1113021025697932295-0 (non-flash)
root@pve3:~# iscsiadm -m session
tcp: [5] 10.0.1.107:3260,1 iqn.2017-08.zettastor.iqn:1113021025697932295-0 (non-flash)

root@pve3:~# pvesm list lvm
Command failed with status code 5.
command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Volid Format Type Size VMID

root@pve2:~# pvesm list lvm
Command failed with status code 5.
command '/sbin/vgscan --ignorelockingfailure --mknodes' failed: exit code 5
Volid Format Type Size VMID

root@pve1:~# pvesm list lvm
Volid Format Type Size VMID
lvm:vm-103-disk-0 raw images 53687091200 103
lvm:vm-103-disk-1 raw images 53687091200 103
 
Jasonhdx said:
root@pve1:~# cat /etc/pve/storage.cfg
Click to expand...
There was no need to repeat the output of "cat /etc/pve/storage.cfg". In a good cluster that file is located on a special shared file system and is the same on each node.

From your /etc/pve/storage.cfg output I can see that you don't have an iSCSI storage pool defined by PVE. Unless something was preventing you from adding it, it would be easier if you defined one. Then you can make your LVM pool dependent on the iSCSI pool, assuring the start order dependencies are more reliable.

That said, it should work as you have it now. You are solely responsible for making sure that iSCSI sessions are established on each node in your configuration. Please provide "lsscsi" output as requested by @LnxBil .

Note, if your iSCSI is properly configured and the sessions are logged in, the "lsscsi" should list the iSCSI disks. The "vgscan" error sometimes happens when the OS/Kernel is confused about the presence of the disks. In some cases, I've seen a reboot fixing the issue.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Hi all,

I had the same problem after adding iSCSI shared storage via GUI, even if the target was permitting concurrent connections and was apparently configured in all nodes (without any restriction) it was in Available status only in the node used to configure Datacenter Storage (the one I accessed the cluster GUI from).

I noticed all nodes (except the one with the available storage) were missing /etc/iscsi/nodes/iqn* for the specific target and simply after cloning manually such configuration on all other nodes (in my case in a testlab by SCPin' the whole missing directory) the storage became active node-by-node.

Please note this has been done on all nodes suffering the problem, /etc/pve/storage.cfg was already correctly replicated and updated on all of them and no relevant logs found at OS level.

Will try to search for a better solution to the problem, starting from libpve-storage-perl and triggered actions on storage creation, if anyone form the PVE team has better (and faster) solutions, please advice :)

CIao!
 
I'm running into the same issue as @alessiobravi. Only the node handling the web request gets correctly configured after adding a new ISCSI storage at the cluster level (all nodes allowed). `/etc/pve/storage.cfg' is the same in both but `/etc/isci/nodes/iqn*` is missing the configurations.

Adding the missing configurations and running `scan` seems to fix it.
 
alessiobravi said:
I had the same problem after adding iSCSI shared storage via GUI
Click to expand...
Hi @alessiobravi ,
The OP did not use PVE GUI or CLI to add iSCSI storage. It appears that iSCSI was set up manually in OP's case. So your issue is not the same.
If may be somewhat similar to https://forum.proxmox.com/threads/iscsi-not-coming-up-properly-after-host-reboot.149393/ but its hard to say at this point.
I would recommend that you open a new thread with detailed information.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
@bbgeek17 I did some further investigation yesterday and it seems that after adding the ISCSI storage, calling `pvesm scan iscsi <PORTAL>` seems to make the other nodes recognize the storage.

Do you know if that is the expected behavior when dealing with this storage class? I need to continue testing on my end to make sure that this is always the case. I'm soon going to integrate this workflow into my application.
 
bilby91 said:
Do you know if that is the expected behavior when dealing with this storage class?
Click to expand...
Frankly, I don't. We don't use PVE iSCSI definitions with our storage, as we built a native PVE storage plugin. We do recommend restarting several services as part of the driver installation. Theoretically, you shouldn't need to do it with a built-in driver, but it does not hurt - it does not affect running VMs.
I'd give service restart a try on each node in the cluster after adding storage. But it should only be needed on _the first_ set up, not on each reboot.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back