Hi, I'm trying to set up SAN storage using fiber cables. I have 5 servers running Proxmox, all of which are in a Proxmox Virtual Environment (PVE) cluster. Could you provide a detailed, step-by-step guide on how to set up SAN storage that will be accessible to all nodes in the cluster.
fc shared storage for all nodes
- Thread starter bablu
- Start date
This has been answered multiple times, yet I coudn't fine a single post that lays out everything, so let's create one:
Let's start with the general entry page in the PVE reference documentation, the storage overview.
FC-based storage has to be block storage and is unfortunately LVM think (setup) so that there are not snapshots.
It also comes most of the time with a multipath setup, so that you need to set it up in the first place. For all steps, you can just search the forums for more information (e.g. specific vendor options and such):
If you want snapshot capability on top of FC-based SAN, you need another solution, like the one described here.
Edits:
Let's start with the general entry page in the PVE reference documentation, the storage overview.
FC-based storage has to be block storage and is unfortunately LVM think (setup) so that there are not snapshots.
It also comes most of the time with a multipath setup, so that you need to set it up in the first place. For all steps, you can just search the forums for more information (e.g. specific vendor options and such):
- [EACH NODE] install multipath-tools (apt install multipath-tools) and configure it (example for iSCSI, yet multipath setup is storage specific)
- [EACH NODE] after that you'll end up with one block device per LUN in /dev/mapper/<name>, let's assume it's file /dev/mapper/san-lun
- [ONE NODE] create a physical volume for LVM on that device with pvcreate /dev/mapper/san-lun (needs to be repeated for each luns)
- [ONE NODE] create a volume group san as vgcreate san /dev/mapper/san-lun
- [ONE NODE] if there are multiple luns, just run this for all other luns vgextend san /dev/mapper/<name>
- [ONE NODE] add the volume group san to PVE (shared must be checked)
If you want snapshot capability on top of FC-based SAN, you need another solution, like the one described here.
Edits:
- TAG added on which node / nodes it has to run
Last edited:
Just a note that these steps need to be done once on one node only.
This is a more advanced step and the end user should understand the reasons and consequences. I.e. will the VG now be spanning two RAID groups on SAN, will the LUNs be sourced from the same group, etc. ? Perhaps, having two independent pools will be more beneficial. This is, of course, beyond, the simple "how to".
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Thank you for the information. I tried this implementation by adding the storage from the database store. However, on other nodes in the cluster, the FC storage shows a capacity of only 285 GB, whereas the host where all actions are performed shows the actual FC capacity. Should I restart the other nodes to update the storage information?
Hi @bablu ,
"database store" is not a standard storage terminology, so it does not mean much to outside readers.
It would be best if you illustrated your concern with command line output. You said that other nodes see 285GB, but do not say what you expect them to see. Could the 285GB be local storage? I.e. the other nodes don't see anything from the SAN?
If you can provide the output of the following commands from each node (as text encoded with CODE tags), it may make it easier for others to help:
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
"database store" is not a standard storage terminology, so it does not mean much to outside readers.
It would be best if you illustrated your concern with command line output. You said that other nodes see 285GB, but do not say what you expect them to see. Could the 285GB be local storage? I.e. the other nodes don't see anything from the SAN?
If you can provide the output of the following commands from each node (as text encoded with CODE tags), it may make it easier for others to help:
lsscsi (may need to be installed via apt install lsscsi)lsblkmultipath -llBlockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Oh yeah, that's a good point. I added it to the post and tried to make the edit clear.
Great news @bablu. Thank you for the update.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
If you have multiple LUNs, you combine them in one big VG? Or would you create a VG per LUN?
Either way is ok. There are benefits and drawback to each.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Could you explain these here? What's the industry best-practices? We started a POC with just 1 LUN backed via Fibre Channel PowerStore Dell Storage Array. I now created a 2nd LUN, but unsure on how to proceed with the VG part. Extend the current VG with the new LUN, or create a new VG. I'm coming from the ESXi world, where every LUN was treated individually, so I guess creating another VG mimics this best? Not sure about the drawbacks about extending the already existing VG.
This has been discussed on the forum more than a few times.
Best practices for one storage vendor will differ from another. This is best asked of your storage vendor.
If you extend, you will continue with single PVE storage pool. You'd be able to extend the LVs onto new space. You will not gain any performance benefit.
If you create a new VG that translates to new PVE storage pool. Potentially more flexibility and isolation.
Thats not strictly correct. You could/can extend a Datastore by concatenating LUNs.
May be. Depends what you meant by "treated individually". New Datastore - yes, VVOLs - no.
Without knowing detailed information about your environment, goals, etc - its hard to recommend any particular approach.
That said, most people will be content with one FC LUN which is as large as their storage allows and single PVE storage pool.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
What other details would you need to recommend a specific approach? This is an enterprise setup, stability and performance are crucial factors. I went over the Dell whitepaper (https://infohub.delltechnologies.co...ying-proxmox-virtual-environment-white-paper/), but it's not really discussing the multiple LUN scenario.
When running out of capacity, to keep things simple, I guess the better option is to just use on big single LUN (by extend the current LUNon the PowerStore)? This also avoids the dilemma whether to go for the multiple VGs, or one single VG.
Enterprise support is generally found via Enterprise Support subscription, whether with the Hypervisor Vendor or your Storage vendor. This is a free community forum.
Call them. Get your dedicated SE or Proxmox SME on the line. Or consider a storage vendor that won't just sell you a box.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox