Cannot build VM on NFS share.

[mijohnst]

I have a 2 host setup and I want the ability to do live migration. I first tried with gluster. Seemed to work fine for a few weeks until I migrated my VM's to one host rebooted the other. All my VM's locked up and then it came back up in split-brain. That was a pain so I've decided to give up on that and move to NAS.

I have a 10TB NAS made by Buffalo...TeraStation 5400. It mounts fine on both nodes and it graphs the size so I know that it's reading it. I can also see it mounted on '/mnt/pve/nas' and I can created and delete files in it with no problem. But, when I try to create a new VM it run for about 30 seconds and then reports back "TASK ERROR: create failed - unable to create image: got lock timeout - aborting command" What does that mean?? It seems fine everywhere else.

As a test, I rebuilt one of my hosts to XenServer and was able to create a VM on the NAS with no issue, so I don't think it's my NAS. Any idea what's going on here?

 

[t.lamprecht]

Hi,

But, when I try to create a new VM it run for about 30 seconds and then reports back "TASK ERROR: create failed - unable to create image: got lock timeout - aborting command" What does that mean?? It seems fine everywhere else.

We lock the storage cluster wide so that no race conditions can happen, or this to work you need quorum as it happens over our cluster file system. So I guess on host was down?
While this lock may be overkill in a bit of situations it is really needed in some and helps assuring consistence.

Workaround for situations where you gracefully shut down one Node and want to do stuff which affects the cluster none the less is to set the expected vote count lower temporarily.

Do so via:

pvecm expected 1

After that you should be able to do everything even when not quorate.

 

[mijohnst]

I'll give this a try but I've had this same issue with a 1 node cluster and a 2 node cluster. I never had this issue one my gluster or local storage. I'll report back as soon as I get back to that system and run your command. Thanks!

 

[mijohnst]

What if we have a 3 node cluster? Do I need to shut 2 down and then 'pvecm expected 1' one one node? I just added an extra node so my gluster has and arbitrator.

 

[t.lamprecht]

What if we have a 3 node cluster? Do I need to shut 2 down and then 'pvecm expected 1' one one node? I just added an extra node so my gluster has and arbitrator.

Sorry if I misunderstood, but you do not need to shut down nodes.
But if you do then there must be a majority of nodes left and agree, else the cluster is not quorate and he cannot guarantee that there is no concurrent access to shared resources.
This is important to avoid corruption of VMs, Storage and such.

So when you have three nodes one may go completely offline or rogue (e.g. network dead) so the one node has no quorum but the other two still have and can continue with their cluster work.
If now only one node is left, in your case for example as a result that you did not need the power of the other two and shut them down, you may temporarily set the expected votes to one.
This is valid because you know and can ensure that this does not do any harm, the other nodes are gracefully shutdown.
When other nodes come up again the expected vote count will increased automatically again, i.e. there can never be an expected vote number smaller than the actual votes a node got.

If you know that already then cool, I just wanted to ensure that you - or other which read this post - know what a `pvecm ecpected 1` call does and that it should be only done after manually confirming that no possible concurrent access can take place (which a shutdown surely does).

 

[mijohnst]

Thank you very much for taking the time to explain that. What you described above is exactly why I added an a 3rd node. I was having issue with gluster coming up in split-head. Since I've had the 3rd node I haven't had any issues. All is working fine except I want to added other types of VM's to a NAS instead of glusterfs. It just does not seem possible because it errors out when I try to create a VM.

Task viewer: VM 106 - Create

OutputStatus

Stop
TASK ERROR: create failed - unable to create image: got lock timeout - aborting command

Maybe I'm misunderstanding, but shouldn't this just work? I would think that NAS would be much easier to work with then gluster. By the way, I've completely rebuilt all hosts and created a new cluster when I added the 3rd node but I still cannot add a VM to my NAS.

Here is my layout.

root@host-1:/# pvecm status
Quorum information
------------------
Date:             Thu Feb 16 12:53:35 2017
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000002
Ring ID:          2/100
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 192.168.1.17 (local)
0x00000003          1 192.168.1.18
0x00000001          1 192.168.1.19

 

[mijohnst]

So are you saying to change the expected votes to 1, create a VM and then change it back to 3?