Hi, I'm trying to set up SAN storage using fiber cables. I have 5 servers running Proxmox, all of which are in a Proxmox Virtual Environment (PVE) cluster. Could you provide a detailed, step-by-step guide on how to set up SAN storage that will be accessible to all nodes in the cluster.
fc shared storage for all nodes
- Thread starter bablu
- Start date
This has been answered multiple times, yet I coudn't fine a single post that lays out everything, so let's create one:
Let's start with the general entry page in the PVE reference documentation, the storage overview.
FC-based storage has to be block storage and is unfortunately LVM think (setup) so that there are not snapshots.
It also comes most of the time with a multipath setup, so that you need to set it up in the first place. For all steps, you can just search the forums for more information (e.g. specific vendor options and such):
If you want snapshot capability on top of FC-based SAN, you need another solution, like the one described here.
Edits:
Let's start with the general entry page in the PVE reference documentation, the storage overview.
FC-based storage has to be block storage and is unfortunately LVM think (setup) so that there are not snapshots.
It also comes most of the time with a multipath setup, so that you need to set it up in the first place. For all steps, you can just search the forums for more information (e.g. specific vendor options and such):
- [EACH NODE] install multipath-tools (apt install multipath-tools) and configure it (example for iSCSI, yet multipath setup is storage specific)
- [EACH NODE] after that you'll end up with one block device per LUN in /dev/mapper/<name>, let's assume it's file /dev/mapper/san-lun
- [ONE NODE] create a physical volume for LVM on that device with pvcreate /dev/mapper/san-lun (needs to be repeated for each luns)
- [ONE NODE] create a volume group san as vgcreate san /dev/mapper/san-lun
- [ONE NODE] if there are multiple luns, just run this for all other luns vgextend san /dev/mapper/<name>
- [ONE NODE] add the volume group san to PVE (shared must be checked)
If you want snapshot capability on top of FC-based SAN, you need another solution, like the one described here.
Edits:
- TAG added on which node / nodes it has to run
Last edited:
Just a note that these steps need to be done once on one node only.
This is a more advanced step and the end user should understand the reasons and consequences. I.e. will the VG now be spanning two RAID groups on SAN, will the LUNs be sourced from the same group, etc. ? Perhaps, having two independent pools will be more beneficial. This is, of course, beyond, the simple "how to".
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Thank you for the information. I tried this implementation by adding the storage from the database store. However, on other nodes in the cluster, the FC storage shows a capacity of only 285 GB, whereas the host where all actions are performed shows the actual FC capacity. Should I restart the other nodes to update the storage information?
Hi @bablu ,
"database store" is not a standard storage terminology, so it does not mean much to outside readers.
It would be best if you illustrated your concern with command line output. You said that other nodes see 285GB, but do not say what you expect them to see. Could the 285GB be local storage? I.e. the other nodes don't see anything from the SAN?
If you can provide the output of the following commands from each node (as text encoded with CODE tags), it may make it easier for others to help:
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
"database store" is not a standard storage terminology, so it does not mean much to outside readers.
It would be best if you illustrated your concern with command line output. You said that other nodes see 285GB, but do not say what you expect them to see. Could the 285GB be local storage? I.e. the other nodes don't see anything from the SAN?
If you can provide the output of the following commands from each node (as text encoded with CODE tags), it may make it easier for others to help:
lsscsi (may need to be installed via apt install lsscsi)lsblkmultipath -llBlockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Oh yeah, that's a good point. I added it to the post and tried to make the edit clear.
Great news @bablu. Thank you for the update.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox