[SOLVED] How to set up VM snapshots on MSA 2060 SAN FC?

[nathanlin1234]

Hi @bbgeek17 I followed the blog on creating a OCFS2 file system, but it's not letting me creating VMs anymore. I have three PVE nodes; I think I created the cluster filesystem correctly, but not sure what I missed. Can you please provide more guidance?

Here is my cluster configurations. I created it via the o2cb tool:

root@pve-node1:/etc/ocfs2# cat /etc/ocfs2/cluster.conf
cluster:
        name = CSTLab
        heartbeat_mode = local
        node_count = 3

node:
        cluster = CSTLab
        number = 0
        ip_port = 7777
        ip_address = 10.xx.xx.xx
        name = pve-node1

node:
        cluster = CSTLab
        number = 1
        ip_port = 7777
        ip_address = 10.xx.xx.xx
        name = pve-node2

node:
        cluster = CSTLab
        number = 2
        ip_port = 7777
        ip_address = 10.xx.xx.xx
        name = pve-node3

And here is my multipath setup:

root@pve-node1:/etc/ocfs2# multipath -ll
mpath0 (3600c0ff000f823c079d2346701000000) dm-5 HPE,MSA 2060 FC
size=8.7T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 1:0:0:1 sdb 8:16 active ready running
| `- 2:0:1:1 sde 8:64 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 2:0:0:1 sdd 8:48 active ready running
  `- 1:0:1:1 sdc 8:32 active ready running
root@pve-node1:/etc/ocfs2#

It looks like the cluster filesystem is seen by all three nodes:

root@pve-node1:/etc/ocfs2# mounted.ocfs2 -d
Device              Stack  Cluster  F  UUID                              Label
/dev/sdb            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/sdc            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/mapper/mpath0  o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/sdd            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/sde            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve

root@pve-node2:~# mounted.ocfs2 -d
Device              Stack  Cluster  F  UUID                              Label
/dev/sdb            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/sdc            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/mapper/mpath0  o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/sdd            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/sde            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
root@pve-node2:~#

root@pve-node3:/etc/ocfs2# mounted.ocfs2 -d
Device              Stack  Cluster  F  UUID                              Label
/dev/sdb            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/sdc            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/mapper/mpath0  o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/sdd            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve
/dev/sde            o2cb               429616C8E41F452AB59C6B4B1381539B  vmstore-lve

I created the ocfs2 from node1, and I try to create a Directory storage at path /media/vmstore, so then I edited /etc/fstab file on node2 and node3, like this:

root@pve-node2:~# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
UUID=4235-CD21 /boot/efi vfat defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
/dev/mapper/mpath0 /media/vmstore ocfs2 _netdev,nointr 0 0
root@pve-node2:~#

root@pve-node3:/etc/ocfs2# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
UUID=C5DE-8982 /boot/efi vfat defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
/dev/mapper/mpath0 /media/vmstore ocfs2 _netdev,nointr 0 0
root@pve-node3:/etc/ocfs2#

However when I try to create a VM I get error on not able to activate my storage because it's not mounted:



On node1 the ocfs2 seems to be mounted at /dlm:

root@pve-node1:/etc/ocfs2# mount -l | grep ocfs2
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw,relatime)

So does this mean I need to create the Directory storage to path /dlm and not just any path I want, like this? What do you think?



I will give it a try again, but wanted to run it by and see if you see anything out of place with my configurations. Thanks.

 

[bbgeek17]

There must be a reason why it's mounted on /dlm. I don't know why, at Blockbridge we don't use OCFS2 or any other file system for shared storage access.
Going over the tutorial again - is a good idea. There may be other documents online that could help.

The error you get is pretty straightforward, the location is not a mount point and you confirmed it by checking system's running state.

Pointing your storage pool configuration to /dlm may be the solution but I would try to understand how you got there first.

I'd also recommend testing various failure scenarios before going to production: node down, path down, controller down, all nodes reboot, one node reboot, etc.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

What is the unknown ?

You can setup your msa for nfs storage

This.

 

[nathanlin1234]

You can setup your msa for nfs storage

@walter is NFS storage something I can deploy onto the MSA like the OCFS? With OCFS I install a OCFS tool (ocfs2-tools) via apt-get my Proxmox node, and then use its to configure my multipathed MSA storage device. Does setting up NFS go something like that?

I googled about setting up NFS on MSA2060 and Google says I can configure NFS by adding volume through the MSA management GUI. My MSA 2060 FC management GUI doesn't give me NFS option, so not sure how much I can trust Google on this. Thanks.

 

[nathanlin1234]

Good luck

Thanks. No joy so far. I updated the mount points and create a new shared Directory storage (vmstorage), so that it's the same configuration across all my Proxmox nodes. I can create new VM, it boots up, but now I get a fsync I/O error at OS installation.



Here is my hardware configuration for the VM. Not sure if I need to do something different for OCFS2 on Proxmox.



All of my three nodes can see file changes made to the new mount point. For example, they all can see the VM disk for this VM 100, and they all can see the Alpine Linux ISO. I think I have set it up correctly now. Not sure what's the issue. With OCFS not being officially supported I guess I just can't expect it to work 100%?

 

[bbgeek17]

There was a known compatibility issue between OCFS2, latest Kernels and certain IO controller settings. I am not sure what is the latest status on it.
I don't have the link to the post at the moment but you might be able to find it through search.

With OCFS not being officially supported I guess I just can't expect it to work 100%?

It should work, but without being officially supported it could get broken by an upgrade of one of the components that did not go through testing of a combination of various technologies. Eventually it will get fixed, you just have to be extra careful with the upgrades and hope that you get in during the "working" cycle.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[waltar]

@walter is NFS storage something I can deploy onto the MSA like the OCFS? With OCFS I install a OCFS tool (ocfs2-tools) via apt-get my Proxmox node, and then use its to configure my multipathed MSA storage device. Does setting up NFS go something like that?

I googled about setting up NFS on MSA2060 and Google says I can configure NFS by adding volume through the MSA management GUI. My MSA 2060 FC management GUI doesn't give me NFS option, so not sure how much I can trust Google on this. Thanks.

You cannot deploy nfs or ocfs2 onto a msa as this is just for creating raidsets and offer defined volumes redundant to host ports.
Don't know how many disks you have but assume you generate 1 volume vol01 on ctrl. A, map to pve1 on the 2 cabled ports (1x A + 1x B).
On the hosts you config multipath (see example in msa docu or here in forum) and get a mpatha (define alias names as you want) volume.
On host pve1 create a filesystem like xfs on /dev/mapper/mpatha and mount maybe on /msav1.
Export /msav1 to your pve hosts in /etc/exports and restart nfs server on pve1.
Now you can add storage typ nfs in pve datacenter.

 

[nathanlin1234]

Thanks @waltar . That sounds pretty straight forward. I will give it a try.

 

[bbgeek17]

Thanks @waltar . That sounds pretty straight forward. I will give it a try.

Don't forget to test what happens to your VMs when the node, that exports NFS, fails.

Or what happens to the NFS exports in your surviving node. You do plan to use hard mounts?

You may also want to research the importance of NFS FSID and mount handle ID preservation on failure for client recovery.

If this is not something your employment depends on, i.e. home-lab, go for it.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[waltar]

Don't forget to test what happens to your VMs when the node, that exports NFS, fails.

When the storage isn't there the vm's cannot write or read further which is logical.
When a zfs pool died all vm's on cannot work either as even if your car didn't start you cannot drive anyway, simply normal.
But you can do pve updates on pve nfs server node and reboot without doing any to the vm's running on other nodes while their disks were living on the nfs share of the rebooting hosts !!

Or what happens to the NFS exports in your surviving node. You do plan to use hard mounts?

See above. Ever use nfs hard mounts which are default !!

You may also want to research the importance of NFS FSID and mount handle ID preservation on failure for client recovery.

That would be important if switching the volume to other pve node, mount and export there (eg for hw defect of pve1).

 

[nathanlin1234]

Thanks @waltar @bbgeek17! Those valuable feedback. I definitely need to plan for what to do when the NFS server goes out.

That wouldn't be a problem for OCFS2 filesystem approach, correct, since the cluster configurations are on all of my Proxmox nodes (they seem to be aware of the OCFS2 filesystem)?

 

[waltar]

Normally if monitoring hw events of hosts you are aware early of nfs server downtime and normally there is no error on enterprise hardware for 5 years. Disks may exchanged on the fly on so normally this single point of failure is just more a theoretically error.

 

[bbgeek17]

. I definitely need to plan for what to do when the NFS server goes out.

Buy a highly available NFS system.

@waltar it seems that we approach high-availabilty , reliability and redundancy with fundamentally different attitudes.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[nathanlin1234]

This.

Hi @alexskysilk if following the thread is still not clear how to set up NFS on MSA, here is what I did on my setup:

I first created multipath for my MSA devices, and then I followed steps 1 - 3 in this blog to set up NFS server on one of my Proxmox node: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-debian-11.

Essentially I use my node 1 as NFS server and then configure NFS storage for node 2 and 3 through the Proxmox management GUI. I followed the Proxmox guide for NFS storage here to do that: https://pve.proxmox.com/wiki/Storage:_NFS.

Hope that's helpful to you, too!

 

[waltar]

it seems that we approach high-availabilty , reliability and redundancy with fundamentally different attitudes.

Don't think so. A solution must be cost effective and each customer has different claims but see here ... most of these pve users are low level and home users were real ha solution are totally overkill. If you operate big clusters with multiple datacenters your storage is ever >1M €|$ and this level isn't discussed here even if blockbridge storage isn't in that range to start.

 

[alexskysilk]

Hi @alexskysilk if following the thread is still not clear how to set up NFS on MSA, here is what I did on my setup:

What you've done is create a SPOF (single point of failure) on whatever device you're using as a head (controller) for the storage to bridge the block device to NFS. It is also substantially less efficient then mapping block devices directly to your vm hosts as doing so necessarily depends on the creation of a middle file system- which would multiply the effect of write amplification stemming from block differences between the guest FS-guest block file-NFS host file system-NFS host block device (tldr- slower writes and waste more space.) This might be an adequate solution for you- I couldn't say as I dont understand the application this is backing (eg, how much money and/or customers will you lose on downtime?)

most of these pve users are low level and home users

Why do you assume this of OP? and if you arent, whats the relevance of different use cases for this post?