Several newbie questions regarding best practices on how to setup a iSCSI SAN

[larsen]

Hello,

we have recently installed a HP MSA 2040 to use with iSCSI. I am a total newbie in this field and decided to just try and learn stuff on the way. Therefore even though it's generally working right now (one host has access to the SAN and I was able to add a mount point to an existing CT), it's probably not configured the way it should be. And I have a lot of questions...

The setup
SAN:
Using a single disk group (RAID5 with 4 SSDs) and only one LUN.
We are using DAC instead of a switch, so every Proxmox host has to use a different IP address for accessing the SAN.

PVE:
We have two hosts in a cluster. I followed https://forum.proxmox.com/threads/iscsi-best-practice.38765/ to create an iSCSI storage and a LVM on top of that.
pve-manager/7.2-3/c743d6c1 (running kernel: 5.13.19-6-pve)

IP configuration:
- SAN Controller A

- Port 1: 10.0.1.100​

- Port 3: 10.0.3.140​

- SAN Controller B

- Port 1: 10.0.1.110​

- Port 3: 10.0.3.150​

- Server 1, vm03 (to A1,B1)

- Port A: 10.0.1.200/24​

- Port B: 10.0.1.210/24​

- Server 2, vm04 (to A3,B3)

- Port A: 10.0.3.240/24​

- Port B: 10.0.3.250/24​

The questions
1) Is a single LUN on the SAN sufficient or do I have to create a LUN for each Proxmox host?
2) If using a single LUN: Do I have to configure the same iSCSI storage on both PVE hosts (with differing IP addresses)?
Will one host know what the other host uses from that LUN?
3) Should I assign a separate subnet for each of the two ports that one host is connected to?
4) When creating the LVM storage, should I enable "Shared"?
5) Afaik, using snapshots is not possible with LVM on iSCSI. Is there a workaround to this? How do you use snapshots in that case or do you simply just not use them?
6) Would ZFS be better suited instead of LVM?
7) How do I migrate the existing containers and VMs (currently using lvm-thin) to the new storage?

 

[bbgeek17]

1) Is a single LUN on the SAN sufficient or do I have to create a LUN for each Proxmox host?

Its one way to go. There are benefits to single LUN approach and multi-LUN. By using two LUNs you could split the ownership between MSA controllers and get "better" performance. Of course if a controller fails you will have to share single controller and your performance can sink. But having more than one LUN means more management: two datastores, extra LVM layers, etc. Its your choice based on your comfort.

2) If using a single LUN: Do I have to configure the same iSCSI storage on both PVE hosts (with differing IP addresses)?

If you want shared storage then all members of PVE cluster must have same access to storage. You will also need to configure multipath.

Will one host know what the other host uses from that LUN?

If you use appropriate storage layer on top of the LUN, i.e. LVM _THICK_

3) Should I assign a separate subnet for each of the two ports that one host is connected to?

matter of preference.

4) When creating the LVM storage, should I enable "Shared"?

If you want to utilize HA capability of PVE cluster - yes, of course. If you dont and access this LUN from two hosts - you will have data corruption.

5) Afaik, using snapshots is not possible with LVM on iSCSI. Is there a workaround to this? How do you use snapshots in that case or do you simply just not use them?

Correct, snapshots are not supported on LVM Thick, which is the only type of LVM you can use with Shared Storage. If you need snapshots - you need to implement different storage than shared Thick LVM on iSCSI.

6) Would ZFS be better suited instead of LVM?

ZFS is not cluster aware. If you want shared storage - you cant use ZFS.

7) How do I migrate the existing containers and VMs (currently using lvm-thin) to the new storage?

You can use "move disk" option available in GUI and CLI.

The table on this page tells what are the supported storage combinations from vanilla PVE install: https://pve.proxmox.com/wiki/Storage
You also have an option of implementing a full Cluster Aware Filesystem on your own: https://en.wikipedia.org/wiki/Clustered_file_system
Then you could use it in combination with Qcow and snapshots, but it will have limited official support from PVE.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[larsen]

Thanks a lot for your detailed answer! I will test and see (and probably come back with more questions next week).

Have a nice weekend! =)

 

[mvs]

3) Should I assign a separate subnet for each of the two ports that one host is connected to?

You need to tell your hosts which interface to use for the connections to MSA.
You can configure separate subnet for each direct link or you can add a routing entries in /etc/network/interfaces:
up ip route add 10.0.1.100/32 dev <NIC_Port_A>
up ip route add 10.0.1.110/32 dev <NIC_Port_B>

 

[larsen]

You need to tell your hosts which interface to use for the connections to MSA.
You can configure separate subnet for each direct link or you can add a routing entries in /etc/network/interfaces:
up ip route add 10.0.1.100/32 dev <NIC_Port_A>
up ip route add 10.0.1.110/32 dev <NIC_Port_B>

Thanks, I found another blog article recommending this, so switched to using a separate subnet not only for each host, but for each port (2 hosts = 4 different subnets). No need for routing or specifying the interface when pinging the target.

 

[larsen]

1) Is a single LUN on the SAN sufficient or do I have to create a LUN for each Proxmox host?
Its one way to go. There are benefits to single LUN approach and multi-LUN. By using two LUNs you could split the ownership between MSA controllers and get "better" performance. Of course if a controller fails you will have to share single controller and your performance can sink. But having more than one LUN means more management: two datastores, extra LVM layers, etc. Its your choice based on your comfort.

1a) And I would need to have a different set of disks for every LUN, as it's probably not possible to have two LUNs on the same RAID/disk group, is it?

2) If using a single LUN: Do I have to configure the same iSCSI storage on both PVE hosts (with differing IP addresses)?
If you want shared storage then all members of PVE cluster must have same access to storage. You will also need to configure multipath.

Multipath is already configured. I have configured the iSCSI storage with an IP address that is available on Proxmox node A. We are using DAC, so node B is using a different IP address to connect to that storage. Node B can see the storage, but e.g. when I try to move a disk to that storage, the shown free space is understandably zero (not available).

2a) How do I have to configure Proxmox so that node B also has access to that storage?

8) So, I have one iSCSI storage, on top of that one LVM storage and this can be used by multiple VMs on multiple nodes?

 

[mvs]

Multipath is already configured. I have configured the iSCSI storage with an IP address that is available on Proxmox node A. We are using DAC, so node B is using a different IP address to connect to that storage. Node B can see the storage, but e.g. when I try to move a disk to that storage, the shown free space is understandably zero (not available).

You can use a hostname instead of an IP Address for your MSA in your storage config (/etc/pve/storage.cfg).
Then just resolve it to propper IP Address on each node (/etc/hosts is unique on each node).

 

[bbgeek17]

1a) And I would need to have a different set of disks for every LUN, as it's probably not possible to have two LUNs on the same RAID/disk group, is it?

Back in 90ies I could configure multiple LUNs on the same RAID group on DG Clariion... You'd have to ping your HP SE for limitations of MSA or if this such setup is recommended by them.

2a) How do I have to configure Proxmox so that node B also has access to that storage?

Like @mvs said you could use hostname, or you could configure iscsi manually on each node using iscsiadm

8) So, I have one iSCSI storage, on top of that one LVM storage and this can be used by multiple VMs on multiple nodes?

PVE will be made aware of the LVM VG and will automatically slice it up as needed. Each VM will have access to one LV off that VG.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[larsen]

You can use a hostname instead of an IP Address for your MSA in your storage config (/etc/pve/storage.cfg).
Then just resolve it to propper IP Address on each node (/etc/hosts is unique on each node).

D'oh... that was a pretty dumb question from me ;-)

How does this work with the two separate subnets? Is multipath taking care of it?

 

[bbgeek17]

How does this work with the two separate subnets? Is multipath taking care of it?

If you plan to use multipath, then you should manually configure iscsi on each node, add multipath and LVM. Then tell PVE about resulting LVM.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[larsen]

If you plan to use multipath, then you should manually configure iscsi on each node, add multipath and LVM. Then tell PVE about resulting LVM.

I have already done so. Configured iSCSI and multipath. Added iSCSI storage on Proxmox and put LVM on top of that.
Should that be sufficient?

 

[bbgeek17]

If you've configured iscsi directly on the host and placed multipath configuration in place , then confirmed that both nodes can see exact same raw storage and created an appropriate (NOT Thin) LVM setup on top of the DM device, then you should not need to configure iscsi in PVE interface, only LVM.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[larsen]

Ok, seems to work fine. Just pulled one cable and the VM using its one and only disk on the SAN is still running.
Thanks for all your help!

 

[larsen]

I just noticed that while the VM is still running, in PVE I have a question mark above the LVM storage "SAN_LVM_LUN01" and get an error message "storage 'SAN' is not online (500)' ("SAN" being the higher level iSCSI storage).

Reason for this probably is that /etc/hosts only contains one of the two IPs/paths there is for the connection to the SAN:

127.0.0.1 localhost.localdomain localhost
...
10.0.1.100 LUN01.san.local

Just a cosmetic problem or is there a better way?

 

[bbgeek17]

Its possible, but highly unlikely that a missing hosts entry is suddenly causing your storage to be marked offline 2 months after initial setup...

Assuming nothing changed since May - you will need to investigate your iSCSI setup/connectivity to determine the cause of the issue. Tools like iscsiadm,lsblk,lssci,dmesg,journalctl will be useful in your troubleshooting.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[larsen]

Sorry, forgot to mention that I was testing again (pulled the cable for which the IP address is registered in the hosts file). So, the cause is clear.
Unclear for me is if that's a problem in some way (as everything seems to be working fine) or if I should configure it differently.

 

[bbgeek17]

You should open a new thread. Post all relevant information (configuration, state, logs, errors, current environment description).

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[larsen]

Well, it's directly related to post #7 in this thread and I already posted all relevant information before (imho).
Maybe also related to your post #12: I have used the documented configuration where you first add a iSCSI storage and then put LVM on top of it. Your post seems to propose adding only a LVM storage via PVE (and I'm not sure how this would work, so link to an example would be greatly appreciated).