LUN LOST

support@ntt.lu · Oct 10, 2022

Hi,
my target isci is connected to a synology nas.
I've 4 lun ready in the nas.
4 lun were available in promox.

Today only three are avalable in promox and one VM is no more working because the VM is in the lost lun

How can I retrieve the lost lun?

Thanks a lot
Pierre

bbgeek17 · Oct 10, 2022

Is there an error message that happens when you try to start VM?
Can you do "journalctl -f" from the PVE shell, then attempt to start the VM? What does the log say?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

support@ntt.lu · Oct 11, 2022

thanks fir answering.
there you are :
Oct 11 08:17:01 pvers70001 systemd[1]: Started Proxmox VE replication runner.
Oct 11 08:18:00 pvers70001 systemd[1]: Starting Proxmox VE replication runner...
Oct 11 08:18:01 pvers70001 systemd[1]: pvesr.service: Succeeded.
Oct 11 08:18:01 pvers70001 systemd[1]: Started Proxmox VE replication runner.
Oct 11 08:19:00 pvers70001 systemd[1]: Starting Proxmox VE replication runner...
Oct 11 08:19:01 pvers70001 systemd[1]: pvesr.service: Succeeded.
Oct 11 08:19:01 pvers70001 systemd[1]: Started Proxmox VE replication runner.
Oct 11 08:20:00 pvers70001 systemd[1]: Starting Proxmox VE replication runner...
Oct 11 08:20:01 pvers70001 systemd[1]: pvesr.service: Succeeded.
Oct 11 08:20:01 pvers70001 systemd[1]: Started Proxmox VE replication runner.
Oct 11 08:20:44 pvers70001 pvedaemon[16646]: start VM 102: UPID

vers70001:00004106:0A206CBD:63450B3C:qmstart:102:root@pam:
Oct 11 08:20:44 pvers70001 pvedaemon[14258]: <root@pam> starting task UPID

vers70001:00004106:0A206CBD:63450B3C:qmstart:102:root@pam:
Oct 11 08:20:44 pvers70001 pvedaemon[16646]: volume 'DS-CLUSTER:0.0.1.scsi-360014054658df52d682bd4c68d8e4edb' does not exist
Oct 11 08:20:44 pvers70001 pvedaemon[14258]: <root@pam> end task UPID

vers70001:00004106:0A206CBD:63450B3C:qmstart:102:root@pam: volume 'DS-CLUSTER:0.0.1.scsi-360014054658df52d682bd4c68d8e4edb' does not exist
Oct 11 08:21:00 pvers70001 systemd[1]: Starting Proxmox VE replication runner...
Oct 11 08:21:01 pvers70001 systemd[1]: pvesr.service: Succeeded.
Oct 11 08:21:01 pvers70001 systemd[1]: Started Proxmox VE replication runner.

bbgeek17 · Oct 11, 2022

Have you checked that the LUN exists on the backend and its configured in the same exact way as all the others?
Your first screenshot/pdf shows all LUNs under the same target. That means they all have same ACL , if any. The listing there is what is returned by the storage. If the LUN is missing from that list, it means the backend storage did not report it to PVE.
Based on everything you've presented to this point - my educated guess is that the problem is in your storage device.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

support@ntt.lu · Oct 11, 2022

sure they are

bbgeek17 · Oct 12, 2022

What happens when you do:
iscsiadm -m discovery -t st -p [synology IP]
iscsiadm --mode node --target <Synology IQN *3ba*> --portal [synology IP] --login
lsscsi
iscsiadm -m session -P 3 -r 2

use "code>" tags to place the command output in reply rather than pdfs

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

support@ntt.lu · Oct 12, 2022

Code:

root@pvers70001:~# iscsiadm -m discovery -t st -p 172.17.16.235
172.17.16.235:3260,1 iqn.2000-01.com.synology:DS-CLUSTER-BEIR.Target-1.a47378c027
[fe80::211:32ff:febe:bd04]:3260,1 iqn.2000-01.com.synology:DS-CLUSTER-BEIR.Target-1.a47378c027
172.17.16.235:3260,1 iqn.2000-01.com.synology:DS-CLUSTER-BEIR.Target-1.3ba568f5874
[fe80::211:32ff:febe:bd04]:3260,1 iqn.2000-01.com.synology:DS-CLUSTER-BEIR.Target-1.3ba568f5874


iscsiadm --mode node --target iqn.2000-01.com.synology:DS-CLUSTER-BEIR.Target-1.3ba568f5874 *3ba* --portal 172.17.16.235 --login iscsi

root@pvers70001:~# iscsiadm --mode node --target iqn.2000-01.com.synology:DS-CLUSTER-BEIR.Target-1.3ba568f5874 *3ba* --portal 172.17.16.235 --login iscsi
iscsiadm: default: 1 session requested, but 1 already present.
iscsiadm: Could not log into all portals

root@pvers70001:~# iscsiadm -m session -P 3 -r 2
iscsiadm: could not read session targetname: 5
iscsiadm: Could not get session info for sid 2

bbgeek17 · Oct 12, 2022

iscsiadm -m session -info
iscsiadm -m session -P 3 -r 1

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

support@ntt.lu · Oct 12, 2022

root@pvers70001:~# iscsiadm -m session -info
tcp: [3] 172.17.16.235:3260,1 iqn.2000-01.com.synology

S-CLUSTER-BEIR.Target-1.3ba568f5874 (non-flash)
root@pvers70001:~#
root@pvers70001:~# iscsiadm -m session -P 3 -r 1
iscsiadm: could not read session targetname: 5
iscsiadm: Could not get session info for sid 1

Thanks a lot to take time for me

bbgeek17 · Oct 12, 2022

support@ntt.lu said:
tcp: [3]

try 3
iscsiadm -m session -P 3 -r 3

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

support@ntt.lu · Oct 12, 2022

much better

root@pvers70001:~# iscsiadm -m session -P 3 -r 3
iSCSI Transport Class version 2.0-870
version 2.0-874
Target: iqn.2000-01.com.synology

S-CLUSTER-BEIR.Target-1.3ba568f5874 (non-flash)
Current Portal: 172.17.16.235:3260,1
Persistent Portal: 172.17.16.235:3260,1
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1993-08.org.debian:01:7bc8d82d75db
Iface IPaddress: 172.17.20.251
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 3
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 120
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 262144
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 11 State: running
scsi11 Channel 00 Id 0 Lun: 1
Attached scsi disk sdg State: running
scsi11 Channel 00 Id 0 Lun: 2
Attached scsi disk sdf State: running
scsi11 Channel 00 Id 0 Lun: 3
Attached scsi disk sde State: running
scsi11 Channel 00 Id 0 Lun: 4
Attached scsi disk sdd State: running

bbgeek17 · Oct 12, 2022

what is the output of:
lsscsi
blkid
lsblk

lsscsi -i

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

support@ntt.lu · Oct 12, 2022

root@pvers70001:~# lsscsi
-bash: lsscsi: command not found

root@pvers70001:~# blkid
/dev/sda2: UUID="9B01-2CE4" TYPE="vfat" PARTUUID="e199f4c4-14c4-4057-bf4e-1f2897c28c8b"
/dev/sda3: UUID="0sNngx-2Kgd-Uspd-5dm6-uEcb-YsJQ-dlJjje" TYPE="LVM2_member" PARTUUID="2f87cf73-a25b-48b1-95f0-dc9beae4c62d"
/dev/sda1: PARTUUID="1d5ec61d-1285-49c6-8153-9a0cdea54278"
/dev/mapper/pve-root: UUID="f19d9c96-847e-44e9-a830-7e5ccb5d991f" TYPE="ext4"
/dev/mapper/pve-swap: UUID="4db8080a-7133-4cba-9976-16d14e8aff1a" TYPE="swap"
/dev/mapper/pve-vm--108--disk--0: PTUUID="fe4bbcf4" PTTYPE="dos"
/dev/sdf1: UUID="93453529-6712-443a-b2a4-7d61b49bb1e6" TYPE="ext4" PARTUUID="4563975a-01"
/dev/sdf5: UUID="f090ed8e-2c50-40cf-9d22-ec7f2bcedb9d" TYPE="swap" PARTUUID="4563975a-05"
/dev/sdd1: UUID="2215a497-97e8-42a6-a6cb-9c04488293b1" TYPE="ext4" PARTUUID="2ad85998-01"
/dev/sdd5: UUID="1e0d11d6-9b57-4720-98bf-14d16b1dfc25" TYPE="swap" PARTUUID="2ad85998-05"
/dev/sde1: UUID="79bec1b3-f509-4ab7-8fb9-33abd3314d9a" TYPE="ext4" PARTUUID="575a0afb-01"
/dev/sde5: UUID="d3ff7994-e6b1-4ae3-9e86-543f389c5155" TYPE="swap" PARTUUID="575a0afb-05"
/dev/sdg1: UUID="43f38b8e-275c-4fcc-b7cc-608af5960de8" TYPE="ext4" PARTUUID="267f3567-01"
/dev/sdg5: UUID="41a768bb-3583-4077-a122-99175d0f7e38" TYPE="swap" PARTUUID="267f3567-05"

root@pvers70001:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 464.7G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 464.2G 0 part
├─pve-root 253:0 0 96G 0 lvm /
├─pve-swap 253:1 0 8G 0 lvm [SWAP]
├─pve-data_tmeta 253:2 0 3.5G 0 lvm
│ └─pve-data-tpool 253:4 0 337.4G 0 lvm
│ ├─pve-data 253:5 0 337.4G 0 lvm
│ └─pve-vm--108--disk--0 253:6 0 64G 0 lvm
└─pve-data_tdata 253:3 0 337.4G 0 lvm
└─pve-data-tpool 253:4 0 337.4G 0 lvm
├─pve-data 253:5 0 337.4G 0 lvm
└─pve-vm--108--disk--0 253:6 0 64G 0 lvm
sdd 8:48 0 32G 0 disk
├─sdd1 8:49 0 30G 0 part
├─sdd2 8:50 0 1K 0 part
└─sdd5 8:53 0 2G 0 part
sde 8:64 0 2T 0 disk
├─sde1 8:65 0 2T 0 part
├─sde2 8:66 0 1K 0 part
└─sde5 8:69 0 8G 0 part
sdf 8:80 0 500G 0 disk
├─sdf1 8:81 0 492G 0 part
├─sdf2 8:82 0 1K 0 part
└─sdf5 8:85 0 8G 0 part
sdg 8:96 0 500G 0 disk
├─sdg1 8:97 0 492G 0 part
├─sdg2 8:98 0 1K 0 part
└─sdg5 8:101 0 8G 0 part
sr0 11:0 1 1024M 0 rom
sr1 11:1 1 1024M 0 rom

root@pvers70001:~# lsscsi -i
-bash: lsscsi: command not found

bbgeek17 · Oct 12, 2022

"sudo apt install lsscsi" ...

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

support@ntt.lu · Oct 12, 2022

root@pvers70001:~# lsscsi
[0:2:0:0] disk LSI AOC-USAS2LP-H8iR 2.13 /dev/sda
[6:0:0:0] cd/dvd TSSTcorp CDDVDW SN-208BB AS01 /dev/sr0
[8:0:0:0] cd/dvd AMI Virtual CDROM0 1.00 /dev/sr1
[9:0:0:0] disk AMI Virtual Floppy0 1.00 /dev/sdb
[10:0:0:0] disk AMI Virtual HDISK0 1.00 /dev/sdc
[11:0:0:1] disk SYNOLOGY iSCSI Storage 4.0 /dev/sdg
[11:0:0:2] disk SYNOLOGY iSCSI Storage 4.0 /dev/sdf
[11:0:0:3] disk SYNOLOGY iSCSI Storage 4.0 /dev/sde
[11:0:0:4] disk SYNOLOGY iSCSI Storage 4.0 /dev/sdd

root@pvers70001:~# blkid
/dev/sda2: UUID="9B01-2CE4" TYPE="vfat" PARTUUID="e199f4c4-14c4-4057-bf4e-1f2897c28c8b"
/dev/sda3: UUID="0sNngx-2Kgd-Uspd-5dm6-uEcb-YsJQ-dlJjje" TYPE="LVM2_member" PARTUUID="2f87cf73-a25b-48b1-95f0-dc9beae4c62d"
/dev/mapper/pve-root: UUID="f19d9c96-847e-44e9-a830-7e5ccb5d991f" TYPE="ext4"
/dev/mapper/pve-swap: UUID="4db8080a-7133-4cba-9976-16d14e8aff1a" TYPE="swap"
/dev/sdf1: UUID="93453529-6712-443a-b2a4-7d61b49bb1e6" TYPE="ext4" PARTUUID="4563975a-01"
/dev/sdf5: UUID="f090ed8e-2c50-40cf-9d22-ec7f2bcedb9d" TYPE="swap" PARTUUID="4563975a-05"
/dev/sdd1: UUID="2215a497-97e8-42a6-a6cb-9c04488293b1" TYPE="ext4" PARTUUID="2ad85998-01"
/dev/sdd5: UUID="1e0d11d6-9b57-4720-98bf-14d16b1dfc25" TYPE="swap" PARTUUID="2ad85998-05"
/dev/sde1: UUID="79bec1b3-f509-4ab7-8fb9-33abd3314d9a" TYPE="ext4" PARTUUID="575a0afb-01"
/dev/sde5: UUID="d3ff7994-e6b1-4ae3-9e86-543f389c5155" TYPE="swap" PARTUUID="575a0afb-05"
/dev/sdg1: UUID="43f38b8e-275c-4fcc-b7cc-608af5960de8" TYPE="ext4" PARTUUID="267f3567-01"
/dev/sdg5: UUID="41a768bb-3583-4077-a122-99175d0f7e38" TYPE="swap" PARTUUID="267f3567-05"
/dev/sda1: PARTUUID="1d5ec61d-1285-49c6-8153-9a0cdea54278"
/dev/mapper/pve-vm--108--disk--0: PTUUID="fe4bbcf4" PTTYPE="dos"

root@pvers70001:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 464.7G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 464.2G 0 part
├─pve-root 253:0 0 96G 0 lvm /
├─pve-swap 253:1 0 8G 0 lvm [SWAP]
├─pve-data_tmeta 253:2 0 3.5G 0 lvm
│ └─pve-data-tpool 253:4 0 337.4G 0 lvm
│ ├─pve-data 253:5 0 337.4G 0 lvm
│ └─pve-vm--108--disk--0 253:6 0 64G 0 lvm
└─pve-data_tdata 253:3 0 337.4G 0 lvm
└─pve-data-tpool 253:4 0 337.4G 0 lvm
├─pve-data 253:5 0 337.4G 0 lvm
└─pve-vm--108--disk--0 253:6 0 64G 0 lvm
sdd 8:48 0 32G 0 disk
├─sdd1 8:49 0 30G 0 part
├─sdd2 8:50 0 1K 0 part
└─sdd5 8:53 0 2G 0 part
sde 8:64 0 2T 0 disk
├─sde1 8:65 0 2T 0 part
├─sde2 8:66 0 1K 0 part
└─sde5 8:69 0 8G 0 part
sdf 8:80 0 500G 0 disk
├─sdf1 8:81 0 492G 0 part
├─sdf2 8:82 0 1K 0 part
└─sdf5 8:85 0 8G 0 part
sdg 8:96 0 500G 0 disk
├─sdg1 8:97 0 492G 0 part
├─sdg2 8:98 0 1K 0 part
└─sdg5 8:101 0 8G 0 part
sr0 11:0 1 1024M 0 rom
sr1 11:1 1 1024M 0 rom

root@pvers70001:~# lsscsi -i
[0:2:0:0] disk LSI AOC-USAS2LP-H8iR 2.13 /dev/sda 36000ea63b0028800272450ab622a5c9a
[6:0:0:0] cd/dvd TSSTcorp CDDVDW SN-208BB AS01 /dev/sr0 -
[8:0:0:0] cd/dvd AMI Virtual CDROM0 1.00 /dev/sr1 AMI_Virtual_CDROM0_AAAABBBBCCCC1-0:0
[9:0:0:0] disk AMI Virtual Floppy0 1.00 /dev/sdb AMI_Virtual_Floppy0_AAAABBBBCCCC2-0:0
[10:0:0:0] disk AMI Virtual HDISK0 1.00 /dev/sdc AMI_Virtual_HDISK0_AAAABBBBCCCC3-0:0
[11:0:0:1] disk SYNOLOGY iSCSI Storage 4.0 /dev/sdg -
[11:0:0:2] disk SYNOLOGY iSCSI Storage 4.0 /dev/sdf 360014052febc0e6d4304d46f2da5e3dd
[11:0:0:3] disk SYNOLOGY iSCSI Storage 4.0 /dev/sde 3600140574c1dbe6dcb1cd47b9d9b7cd0
[11:0:0:4] disk SYNOLOGY iSCSI Storage 4.0 /dev/sdd 36001405d637faf8d9769d4d4cd8b7bdf

bbgeek17 · Oct 12, 2022

Ok, now that we have all the data - lets summarize what we see:

1) you backend storage presents 4 luns to the Proxmox group:
32GB
2TB
500GB
500GB

2) The host can see them:

support@ntt.lu said:
sdd 8:48 0 32G 0 disk
sde 8:64 0 2T 0 disk
sdf 8:80 0 500G 0 disk
sdg 8:96 0 500G 0 disk

3) we know that the 500G lun is "missing". And we know its SCSI NAA

support@ntt.lu said:
volume 'DS-CLUSTER:0.0.1.scsi-360014054658df52d682bd4c68d8e4edb' does not exist

4) we can see the NAA in "lsscsi -i" output

support@ntt.lu said:
[11:0:0:1] disk SYNOLOGY iSCSI Storage 4.0 /dev/sdg -
[11:0:0:2] disk SYNOLOGY iSCSI Storage 4.0 /dev/sdf 360014052febc0e6d4304d46f2da5e3dd
[11:0:0:3] disk SYNOLOGY iSCSI Storage 4.0 /dev/sde 3600140574c1dbe6dcb1cd47b9d9b7cd0
[11:0:0:4] disk SYNOLOGY iSCSI Storage 4.0 /dev/sdd 36001405d637faf8d9769d4d4cd8b7bdf

5) Its clear that the *edb NAA is not listed. The device that is missing NAA is "sdg". We know from prior output that its the 500GB lun

So, PVE is properly complaining that it cant find LUN with NAA 360014054658df52d682bd4c68d8e4edb.

Next logical steps would be:
a) Determine if the LUN is accessible, ie can you "dd if=/dev/sdg of=/dev/null bs=32k"
b) You can try rebooting the host, removing and re-adding the lun on storage device side, rebooting storage device, etc.

Its possible that you hit some sort of Linux Kernel bug, but much more likely its the fault of the backend storage system.
After you have tried reboots/restarts/etc, if you are still in the same position I recommend that you reach back out to your vendor support, present them with the details and insist on finding out why a Debian based system would be unable to query the storage disk for its NAA. Or you can start with vendor support while the system is in this state.

I suspect there are other issues with the disk as well.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

bbgeek17 · Oct 12, 2022

You can also try to run :

Code:

sg_inq -e -p 0x83 /dev/sdg

For that you may first need to : sudo apt-get install sg3-utils

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

support@ntt.lu · Oct 12, 2022

root@pvers70001:~# sg_inq -e -p 0x83 /dev/sdg
VPD INQUIRY: Device Identification page
Designation descriptor number 1, descriptor length: 20
designator_type: NAA, code_set: Binary
associated with the Addressed logical unit
NAA 6, IEEE Company_id: 0x1405
Vendor Specific Identifier: 0x4658df52d
Vendor Specific Identifier Extension: 0x682bd4c68d8e4edb
[0x60014054658df52d682bd4c68d8e4edb]
Designation descriptor number 2, descriptor length: 63
designator_type: T10 vendor identification, code_set: ASCII
associated with the Addressed logical unit
vendor id: SYNOLOGY
vendor specific: iSCSI Storage:4658df52-682b-4c68-8e4e-bb84c9d43476
Designation descriptor number 3, descriptor length: 8
transport: Internet SCSI (iSCSI)
designator_type: Relative target port, code_set: Binary
associated with the Target port
Relative target port: 0x3
Designation descriptor number 4, descriptor length: 8
transport: Internet SCSI (iSCSI)
designator_type: Target port group, code_set: Binary
associated with the Target port
Target port group: 0x0
Designation descriptor number 5, descriptor length: 8
designator_type: Logical unit group, code_set: Binary
associated with the Addressed logical unit
Logical unit group: 0x0
Designation descriptor number 6, descriptor length: 76
transport: Internet SCSI (iSCSI)
designator_type: SCSI name string, code_set: UTF-8
associated with the Target port
SCSI name string:
iqn.2000-01.com.synology

S-CLUSTER-BEIR.Target-1.3ba568f5874,t,0x0001
Designation descriptor number 7, descriptor length: 68
transport: Internet SCSI (iSCSI)
designator_type: SCSI name string, code_set: UTF-8
associated with the Target device that contains addressed lu
SCSI name string:
iqn.2000-01.com.synology

S-CLUSTER-BEIR.Target-1.3ba568f5874

bbgeek17 · Oct 12, 2022

So, interestingly, NAA is present here.

"lsscsi" (and other programs/API) pull the information from kernel cache in sys/block/sdg/device/vpd_pg83. However, sg_inq sends a direct on-the-wire inquiry to disk. It appears that the information was not available at some point in the past, but perhaps is now. The confirmation might be in the journalctl, which you can carefully study.

There may be ways to refresh the information in the cache, perhaps " echo 1 > /sys/block/sdg/device/rescan ". Reboot of the host should fix it, I dont expect this to be a persistent issue. However, without knowing why it happened you can expect it to happen again.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

support@ntt.lu · Oct 22, 2022

Reboot of synology and pomox everything working fine.
Strange there is no way to reconnect a scsi target in proxmox.

LUN LOST

Member

Attachments

Distinguished Member

Member

Distinguished Member

Member

Attachments

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Distinguished Member

Member

Distinguished Member

Member