creating a LVM on iSCSI storage on TrueNAS

[RockNLol]

hi,
I mounted an iSCSI share from my TrueNAS in Proxmox via Datacenter > Storage > Add > iSCSI which worked fine. I unchecked "Use LUNs directly" and tried to put LVM on the iSCSI disk. This failed, but I cannot remember the error, I think it had something to do with not being able to create a vg because of filters. I was following a guide here. So I thought I did something wrong, I started over with a new ZVOL, and since then I get this error:

create storage failed: pvcreate '/dev/disk/by-id/scsi-STrueNAS_iSCSI_Disk_960000010068006' error: Cannot use device /dev/disk/by-id/scsi-STrueNAS_iSCSI_Disk_960000010068006 with duplicates. (500)

Output of command pvs:

WARNING: Not using device /dev/sdd for PV Afzntf-jHVd-J3rC-QFbi-Ua08-z3Bd-kf4c1u.
  WARNING: Not using device /dev/sdf for PV XVQsX3-i4XL-tf6f-pkPN-wHjB-Ikqt-prh2nn.
  WARNING: PV Afzntf-jHVd-J3rC-QFbi-Ua08-z3Bd-kf4c1u prefers device /dev/sdc because device was seen first.
  WARNING: PV XVQsX3-i4XL-tf6f-pkPN-wHjB-Ikqt-prh2nn prefers device /dev/sde because device was seen first.
  PV         VG Fmt  Attr PSize PFree
  /dev/sdc      lvm2 ---  3.00t 3.00t
  /dev/sde      lvm2 ---  1.00t 1.00t

the 1t and 3t were the two ZVOLs I tried.

how can I cleanly start over?

 

[bbgeek17]

It seems that perhaps you have multiple paths to your storage, but you have not setup multipath?

What does lsblk/lsscsi show?

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[RockNLol]

lsblk shows:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0     1T  0 disk
sdb           8:16   0     1T  0 disk
sdc           8:32   0     3T  0 disk
sdd           8:48   0     3T  0 disk
sde           8:64   0     1T  0 disk
sdf           8:80   0     1T  0 disk
zd0         230:0    0    64G  0 disk
├─zd0p1     230:1    0    60G  0 part
├─zd0p2     230:2    0     1K  0 part
└─zd0p5     230:5    0     4G  0 part
[... zdXX of all my vms]
zd208       230:208  0     1M  0 disk
nvme1n1     259:0    0 465.8G  0 disk
├─nvme1n1p1 259:2    0  1007K  0 part
├─nvme1n1p2 259:4    0   512M  0 part
└─nvme1n1p3 259:7    0 465.3G  0 part
nvme0n1     259:1    0 465.8G  0 disk
├─nvme0n1p1 259:3    0  1007K  0 part
├─nvme0n1p2 259:5    0   512M  0 part
└─nvme0n1p3 259:6    0 465.3G  0 part

the two nvme drives being the physical storage in the proxmox server.

lsscsi shows:

[0:0:0:0]    disk    TrueNAS  iSCSI Disk       0123  /dev/sda
[1:0:0:0]    disk    TrueNAS  iSCSI Disk       0123  /dev/sdb
[2:0:0:0]    disk    TrueNAS  iSCSI Disk       0123  /dev/sdc
[3:0:0:0]    disk    TrueNAS  iSCSI Disk       0123  /dev/sdd
[4:0:0:0]    disk    TrueNAS  iSCSI Disk       0123  /dev/sde
[5:0:0:0]    disk    TrueNAS  iSCSI Disk       0123  /dev/sdf
[N:0:8215:1] disk    WDS500G3X0C-00SJG0__1                      /dev/nvme0n1
[N:1:8215:1] disk    WDS500G3X0C-00SJG0__1                      /dev/nvme1n1

I have two working iSCSI connections where I used the LUNs directly in a VM. So in total there should be 3 connections. 2 where I used the LUNs directly and 1 where I want to create the LVM on.

 

[bbgeek17]

It seems you need to setup multipath: https://pve.proxmox.com/wiki/ISCSI_Multipath
Then use mpathX devices for your LVM/etc.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[RockNLol]

adding a minimal multipath connfiguration did the trick! I followed the proxmox wiki page you linked and added the wwid of /dev/sdc/ (same as /dev/sdd)
After restarting multipath-tools adding the LVM worked without an error.

I don't know, why this is needed, as the TrueNAS only has one NIC, but it is what it is.

Thank you for your help!

 

[bbgeek17]

It could be :
- two nics on client side
- two reachable IPs on TrueNAS
- two IPs on client that can each reach TrueNAS

examine output of: iscsiadm -m target / -m session, it might help

Good luck

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[RockNLol]

iscsiadm -m target / -m session shows one entry with the IP of the TrueNAS and one entry with the domain name of the TrueNAS (pointing to the same IP) for every connected iscsi-disk.

 

[bbgeek17]

curious, somehow whatever mechanism you used to establish ISCSI connection thinks that IP vs host are two paths, something in your config is wonky.
Or perhaps host connection is a different IP, may be external? Have you pinged it?

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[RockNLol]

I might have found the issue while pinging. Reverse DNS of the IP delivers a different domain name than forward DNS request.

*edit: It's always DNS lol