Question about LVM on iSCSI

[Marco2G]

Hi everyone

I could use a bit of clarification. I am trying to compare different ways of attaching shared storage. NFS is working so far (although 220MB/s on a 10G interface seems a tad low)

I would like to try LVM on iSCSI. I have set this up and still have the issue that the second cluster node has question marks both on the iSCSI device and the LVM.

The first host, on the other hand, seems to be working.

So A can I find this LVM as a mount in the shell somewhere so I can make a perf test with dd?
And B, anyone know how to make this accessible to more than one host at a time?

 

[bbgeek17]

So A can I find this LVM as a mount in the shell somewhere so I can make a perf test with dd?

basic linux tools, like "lsblk" and "lsscsi" (needs to be installed) will show you disks with their paths.
You can also utilize "iscsiadm -m node" and "-m session".

And B, anyone know how to make this accessible to more than one host at a time?

This depends entirely on your iSCSI storage, how you configured things, the type of errors you receive in the log and perhaps a few other factors that are escaping me.

A good start would be to :
1) present the context of your /etc/pve/storage.cfg
2) present the output of "pvesm status"
3) output of "lsblk" from each node
4) output of "journalctl -n 500" as attachment, or if you are able to identify the important messages - subset of it
5) describe your infrastructure in detail

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Marco2G]

root@proxhost1:/mnt/pve/prox-nfs# lsscsi
[6:0:0:1]    disk    TrueNAS  iSCSI Disk       0123  /dev/sda
[N:0:5:1]    disk    Samsung SSD 980 500GB__1                   /dev/nvme0n1
root@proxhost1:/mnt/pve/prox-nfs# iscsiadm -m node
192.168.5.110:3260,1 iqn.2018-01.ch.somewhere.ctl:prox-target
root@proxhost1:/mnt/pve/prox-nfs# iscsiadm -m session
tcp: [1] 192.168.5.110:3260,1 iqn.2018-01.ch.somewhere.ctl:prox-target (non-flash)
root@proxhost1:/mnt/pve/prox-nfs# lsblk
NAME                 MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                    8:0    0   100G  0 disk
nvme0n1              259:0    0 465.8G  0 disk
├─nvme0n1p1          259:1    0  1007K  0 part
├─nvme0n1p2          259:2    0     1G  0 part /boot/efi
└─nvme0n1p3          259:3    0 464.8G  0 part
  ├─pve-swap         252:0    0     8G  0 lvm  [SWAP]
  ├─pve-root         252:1    0    96G  0 lvm  /
  ├─pve-data_tmeta   252:2    0   3.4G  0 lvm
  │ └─pve-data-tpool 252:4    0 337.9G  0 lvm
  │   └─pve-data     252:5    0 337.9G  1 lvm
  └─pve-data_tdata   252:3    0 337.9G  0 lvm
    └─pve-data-tpool 252:4    0 337.9G  0 lvm
      └─pve-data     252:5    0 337.9G  1 lvm
root@proxhost1:/mnt/pve/prox-nfs# lsscsi
[6:0:0:1]    disk    TrueNAS  iSCSI Disk       0123  /dev/sda
[N:0:5:1]    disk    Samsung SSD 980 500GB__1                   /dev/nvme0n1
root@proxhost1:/mnt/pve/prox-nfs# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content backup,vztmpl,iso

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

iscsi: prox-iSCSI-1
        portal 192.168.5.110
        target iqn.2018-01.ch.somewhere.ctl:prox-target
        content none

lvm: prox-iSCSI-lvm
        vgname vgiSCSI1
        base prox-iSCSI-1:0.0.1.scsi-36589cfc000000f6fcd6c6a6bc4bfd370
        content images,rootdir
        shared 1

nfs: prox-nfs
        export /mnt/virtualmachines/prox-nfs
        path /mnt/pve/prox-nfs
        server 192.168.5.110
        content images
        prune-backups keep-all=1


root@proxhost2:~# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1                      259:0    0 465.8G  0 disk
├─nvme0n1p1                  259:1    0  1007K  0 part
├─nvme0n1p2                  259:2    0     1G  0 part /boot/efi
└─nvme0n1p3                  259:3    0 464.8G  0 part
  ├─pve-swap                 252:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                 252:1    0    96G  0 lvm  /
  ├─pve-data_tmeta           252:2    0   3.4G  0 lvm 
  │ └─pve-data-tpool         252:4    0 337.9G  0 lvm 
  │   ├─pve-data             252:5    0 337.9G  1 lvm 
  │   └─pve-vm--100--disk--0 252:6    0    32G  0 lvm 
  └─pve-data_tdata           252:3    0 337.9G  0 lvm 
    └─pve-data-tpool         252:4    0 337.9G  0 lvm 
      ├─pve-data             252:5    0 337.9G  1 lvm 
      └─pve-vm--100--disk--0 252:6    0    32G  0 lvm 



root@proxhost1:/mnt/pve/prox-nfs# pvesm status
Name                  Type     Status           Total            Used       Available        %
local                  dir     active        98497780         9764592        83683640    9.91%
local-lvm          lvmthin     active       354275328               0       354275328    0.00%
prox-iSCSI-1         iscsi     active               0               0               0    0.00%
prox-iSCSI-lvm         lvm     active       104849408               0       104849408    0.00%
prox-nfs               nfs     active       536870912         2528640       534342272    0.47%

I have a Mikrotik SFP+ capable switch to which both the TrueNAS box and two proxmox hosts are connected. Management goes over a 1G RJ45link on vmbr0 while vmbr1 handles the fiber. I have several VLANs on the fiber. Relevant for storage is VLAN5 where the TrueNAS box has IP 192.168.5.110 and the proxmox boxes have 192.168.5.2 and 3 respectively.

I have a three gumstick NVMe ZFS pool on which I created a zVOL that is used as a iSCSI LUN.

I'm not quite sure what else could be relevant...

 

[bbgeek17]

I am going to assume that the "second cluster node" is also labeled as such, ie "proxhost2".
You've provided all the information from "first cluster node" (proxhost1) and essentially nothing from the second where you actually have an issue.

I can see the disk is not shown there. Again, there could be many reasons. One of them is that you didnt configure your Initiator access properly on your NAS. Running all the same commands that you did on your "good" node, may help to zoom in on the problem on bad node.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Marco2G]

Sorry, that makes sense, obviously. It's late here and my brain is not on its best...

You helped me a lot though. iscsiadm -m node showed old entries from previous configs. iscsiadm -m session was empty.
I ran a discovery and now it's looking good.

Is this a known phenomenon for proxmox? I've had bits of config remain on the nfs side as well that I had to clean up manually.


As to the remaining question, lsscsi and lsblk show a whole lot of nothing, don't they? is that lvm even mounted anywhere or how exactly does proxmox access it?

 

[bbgeek17]

Is this a known phenomenon for proxmox? I've had bits of config remain on the nfs side as well that I had to clean up manually.

Without knowing exactly how you arrived to the end result that you considered "bad" and had to clean up manually, or having a way to reproduce it, how can one attribute it to anything but admin mistake in general?

As to the remaining question, lsscsi and lsblk show a whole lot of nothing, don't they?

What exactly are you asking here? If you refer to "proxmox1" - it showed me a lot of useful information. If you refer to "proxmox2", then you only presented "lsblk" which indeed did not show a disk. However, its not "nothing", its a confirmation that you have a problem. It is, however, a symptom not a cause.
When you relay description of the problem as a very general symptom (node has question mark) its about the same as a helpdesk ticket "my computer does not work". May be a monitor is not on, or power not plugged in.

is that lvm even mounted anywhere or how exactly does proxmox access it?

You can run LVM specific commands to learn more : pvs, vgs, lvs.

LVM is there :

Feb 17 19:28:21 proxhost1 lvm[1229]: PV /dev/sda online, VG vgiSCSI1 is complete.
Feb 17 19:28:21 proxhost1 systemd[1]: Started lvm-activate-vgiSCSI1.service - /sbin/lvm vgchange -aay --autoactivation event vgiSCSI1.
Feb 17 19:28:21 proxhost1 lvm[1233]:   0 logical volume(s) in volume group "vgiSCSI1" now active
Feb 17 19:28:22 proxhost1 systemd[1]: lvm-activate-vgiSCSI1.service: Deactivated successfully.

The use of LVM in shared access situation is more complicated than with single host. Things are activated as needed. When you have a VM running you will see more.
That said, a question mark on iSCSI indicates that you did not setup that part properly, and since LVM depends on it - LVM wont work until you fix the underlying layer.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Marco2G]

I do not show the second host because at this stage it is irrelevant to the question of where is the LVM mounted to so I could go there and do a dd on it.

As for the LVM commands, I can see that the LVM on iSCSI only created a vg but there is no lv to mount to anywhere. So does that mean lvs aren't used or is that yet another step I am unaware of?

 

[bbgeek17]

As for the LVM commands, I can see that the LVM on iSCSI only created a vg but there is no lv to mount to anywhere. So does that mean lvs aren't used or is that yet another step I am unaware of?

Have you created a disk image in PVE that is located on the LVM storage? Thats the step that allocates LV from a VG.

I do not show the second host because at this stage it is irrelevant to the question of where is the LVM mounted to so I could go there and do a dd on it.

One of your two questions was how to make 2nd host to see the storage. The A and B did not appear to be related.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Marco2G]

And I've told you that I was seeing old entries in iscsiadm -m node. I did a discovery and now it's good.

You accuse me of giving bad info but do not read what I write my friend.

Since I know what I want, namely that proxmox is just like any OSS project not ready for enterprise prime-time unless you are willing to employ a whole cadre of people willing to deep dive into the topic, I will probably no longer need input on this topic. I am going to use ESXi without further patches for as long as it will run, I will not use vCenter to complicate things and I may or may not get rid of Veeam and live with scripted offline backups at home.

That is a much less hassle prone approach than using proxmox.

To you, personally, I would like to say thank you for your efforts to help after what I assume have been years of reading similar complaints and questions. That being said I would like to propose you take a sabbatical from this because your attitude sucks.