[SOLVED] How to set up VM snapshots on MSA 2060 SAN FC?

[nathanlin1234]

Hi community,

I hope everyone had an awesome 2024, and wish you all a successful 2025.

We have a three-node Proxmox VE cluster running Proxmox VE 8.3.1. All three nodes are connected to a HPE MSA 2060 Fiber Connect storage through HBAs and a FC switch. We're trying to set it up so that we can deploy VMs on the shared MSA storage and have VM snapshots. Is there a way to do this?

I was able to set up multipath, but could not get Directory storage to use the multipathed device. I can set up LVM; it works very well for VM migration but does not work for VM snapshot. I can't seem to have Directory storage on top of LVM.

Any suggestion how to do this?

Thank you.

 

[waltar]

If you have just 1 FC switch which could fail (still we had one who died but there were two that time) you loose storage access, so why not did 2 or direct connect the msa2060fc to all pve directly ? For directory storage you need a filesystem onto the msa volumes created first. When using lvm on these msa volumes you could instead eg. use the snapshot functionality of the msa itself.

 

[bbgeek17]

Hi @nathanlin1234 ,

You are on the right path to your goals.

The only way to achieve a combination of "Legacy FC SAN"+Multipath+Shared Storage+Snapshots is by placing a Cluster File system on top of your FC LUN. There is no officially supported/endorsed CFS in the PVE support matrix. Your primary option is OCFS2. You will need to install, configure, and maintain it yourself.

Once you have a CFS deployed, you can place a "directory" storage pool on top of it. Then you can utilize QCOW as the backend and snapshots will become available.

Hope this helps,

Cheers

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[nathanlin1234]

Thank you @bbgeek17 @walter. These are very helpful information. Does it also mean Directory storage would not work for ext4 file system on top of a logical volume in my logical volume group? Thanks!

 

[bbgeek17]

Does it also mean Directory storage would not work for ext4 file system on top of a logical volume in my logical volume group? Thanks!

Ext4 is not a cluster aware filesystem. Using it in a cluster environment will lead to data loss/corruption.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

The only way to achieve a combination of "Legacy FC SAN"+Multipath+Shared Storage+Snapshots is by placing a Cluster File system on top of your FC LUN.

Not the "only"

The MSA line of controllers have an https api (I believe restful but I could be wrong) and an ssh addressable cli. One could write a snapshot orchestration on PVE that will issue a qm-freeze, snapshot, and qm-thaw. If you're handy with perl you can even write a storage plugin (https://pve.proxmox.com/wiki/Storage_Plugin_Development)

Restoration would be more complicated, but still doable.

 

[bbgeek17]

Not the "only"

The MSA line of controllers have an https api (I believe restful but I could be wrong) and an ssh addressable cli

Absolutely, there’s always more than one way to approach these things. When helping new PVE users, I tend to focus on solutions that are well-documented and have a robust user base for research and troubleshooting.

At Blockbridge, with our experience implementing the Proxmox Plugin, we understand firsthand the importance of continuously testing for compatibility, stability, and scalability.

Regarding snapshots via a plugin, I suspect it may not be feasible to write a plugin limited to that functionality. I’m not aware of a flag or option that allows only snapshots while excluding other actions like create or delete.

That said, it’s software—anything is possible with enough effort! The question would be whether the effort is justified, particularly for an entry-level SAN appliance.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

Oh I get what you're saying. I'm just not at all sure that setting up and maintaining GFS2 on pve is the more beginner friendly option

@nathanlin1234 consider what your use case actually is. Neither option for having snapshots on an external block SAN are "beginner friendly."

 

[nathanlin1234]

Thank you guys! Really appreciate the feedback I'm receiving. We have MSA2060 FC, and it doesn't seem to be capable to install an OS to it, but I will need to dig deeper into its user guide and confirm. The plugin idea is also great. I'm not a storage plugin developer, but I have seen handful of threads on the forum asking about VM snapshot on MSA SAS storages, so it sounds like a really feature to have.

 

[bbgeek17]

We have MSA2060 FC, and it doesn't seem to be capable to install an OS to it, but I will need to dig deeper into its user guide and confirm.

No, you cannot install anything but official HPE software on this unit.

The plugin idea is also great. I'm not a storage plugin developer, but I have seen handle of threads on the forum asking about VM snapshot on MSA SAS storages, so it sounds like a really feature to have.

A vendor-supported plugin is always the preferred option.

An external, community-developed, software package for a proprietary appliance is always at the whim of the vendor and is one change/firmware update away from an outage.

Good luck with your research.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[waltar]

There must be a snapshot volume(s) created first but then ... which could be done in batch mode (by cronjob with expect) with ssh to cli :
Eg create snapshots of volumes V4 and V5.
# create snapshots volumes V4,V5 V4snap,V5snap

And yes as bbgeek17 wrote the msa has it's own firmware-os and you cannot install other.

 

[alexskysilk]

We have MSA2060 FC, and it doesn't seem to be capable to install an OS to it,

HPE MSA is a storage appliance. the "OS" is embedded and not user serviceable except to perform firmware updates. This is not a server in the generic sense.

For your own edification, this particular device controller is actually provided to HP by another OEM- specifically dothill (Seagate.) When looking for plugins for it for various purposes (eg, nagios) you can use dothill or other vendors' plugins that use the same controller if its not available for MSA specifically.

 

[nathanlin1234]

Should I have gone with a MSA2060 with ISCSI storage and use ZFS over ISCSI if I wanted VM snapshots and VM migrations? Thanks.

 

[bbgeek17]

Should I have gone with a MSA2060 with ISCSI storage and use ZFS over ISCSI if I wanted VM snapshots and VM migrations? Thanks.

No, going with a different MSA model would not have helped you.

ZFS over iSCSI scheme requires: a) ability to ssh to the storage appliance as root b) internally storage appliance must be based on ZFS c) storage appliance must allow direct ZFS manipulation d) storage appliance must use one of the approved iSCSI daemons and allow manual iSCSI export manipulation.

While it sounds from the comment above that (a) is possible in some form, b-d are not.
You have a proprietary Block Storage Appliance. Either you use it with FC/iSCSI and LVM and no snapshots, or you need a Cluster Aware Filesystem.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[nathanlin1234]

No, going with a different MSA model would not have helped you.

Thanks @bbgeek17 for breaking it down for me. Would you be so kind and give me an example (or maybe a couple)? I googled around but yeah I'm for sure outside of my comfort zone and don't know what direction to head to. I mean, like, what storage system do I need to get to build a cluster aware filesystem?

I'm gonna keep on researching and learning about it. And thanks a bunch for your help!

 

[bbgeek17]

what storage system do I need to get to build a cluster aware filesystem?

Your current SAN is suitable as a base for installing and configuring CAF
Perhaps this might help: https://forum.proxmox.com/threads/md3200sas-proxmox-ocfs2-not-supported-by-the-proxmox-team.129156/

If this is a production system, perhaps you should reach out to one of the Proxmox solutions partners for consulting help.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[waltar]

You can setup your msa for nfs storage and with up to 4 hosts and pacemaker you can do ha-nfs if you want to too (I would do manual server switch but connect to minimum 2 servers to able to).

 

[alexskysilk]

You can setup your msa for nfs storage

Considering the scope of the original question, maybe you can post a tutorial?

 

[waltar]

Considering the scope of the original question, maybe you can post a tutorial?

What is the unknown ? In general you have inside or outside a host some disk and want to combine them by any raid technologie to survive a disk failure. The raidset then build a virtual disk seen by the os like a real disk which could be used how you want. So if doing a filesystem onto and export it other hosts could mount it and used it as these disk again would be local on the other host.
WIth a storage system like a msa you have the comfort that you could define lots of virtual disks and provide these through 2 controllers with up to 4 hosts same time directly and while this is a FC version with FC switches (or even a fabric) the number of hosts is more just a question of money spend with no real end. If you want an easy to handle system in case of any thinkable error you should do the setup as simple as possible as that is the key for any failure recovery and second mostly better performance instead of having a bunch of separate volumes etc.

 

[waltar]

The cool think with an external storage is that you can online switch volumes from one host to another (or even same time and just mount a fs here or there) to simplify any kind of downtime like repairing a fileserver, exchanging or even switch the os onto. You even could have different os on the connected hosts (wouldn't do that because of thinking ´maybe having different admin's doing somethink nesty ...).