Hello
I am managing a cluster with 2 nodes and a qDevice VM to respect the quorum.
Proxmox Server on 9.1.5 version
The IP adresse of Proxmox node are these :
- First node : 192.168.1.1 (MGMT network) 192.168.2.1 (SAN Network)
- Second node : 192.168.1.2 (MGMT network) 192.168.2.2 (SAN Network)
-NFS Server : 192.168.1.150 (MGMT Network) 192.168.2.3 (SAN Network)
The HA is configured on all the VM.
They are Windows-based VMs and Linux-based VMs
The shared storage between the node is a NFS share. This NFS Share is based on a Windows Server 2022.
The creation of VM in the NFS storage is possible, everythings is working fine.
The network link between the node and the Shared storage is a GigabitEthernet cable.
Here is my configuration about my different NFS Share :
I have created 3 NFS Share on Windows Server 2022 :
- proxmox-nfs-storage : only 192.168.2.1 and 192.168.2.2 can access to it (Configured on the NFS Share Properties). But the source of the NFS share is declared using the DNS name of the NFS Server so it might use a different network to access the NFS SHare. NFS Version on proxmox used to connect to : 4.1
- nfs-vm-storage : only 192.168.2.1 and 192.168.2.2 can access to it (Configured on the NFS Share Properties). NFS Version on proxmox used to connect to : Default
- test1 : only 192.168.2.1 and 192.168.2.2 can access to it (Configured on the NFS Share Properties). NFS Version on proxmox used to connect to : Default
I connected both proxmox node to these NFS Shares.
I can successfully create a VM on all of these NFS Share storage.
But I had many problems while trying to move VM between these NFS Share.
It's not really a problem because sometimes it works but I am really confused about some points so here are some tests I conducted if I can have some answers about my hypothesis or maybe it's an undocumented feature
Here are the tests I conducted :
- First test: Moving a Windows-based VM from the proxmox-nfs-storage to test1. Alone, without any other moving process in the mean time. VM not configured in HA
Conclusion : failed to move hard disk.
- Second test : Moving the same VM from the NFS Share proxmox-nfs-storage to local-lvm without any other process. VM not configured in HA
Conclusion : working. No problems.
- Third test : Moving the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. With another process of moving VM hardware (to test1 NFS Share from proxmox-nfs-storage). VM not configured in HA
Conclusion : not working. Same problem as the first test.
- Fourth test : Moving the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. Without any other moving process in the mean time. VM configured in HA
Conclusion : not working. Same problem as the first test
- Fifth test : Moving the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. With another process of moving VM hardware. VM configured in HA.
Conclusion : not working. Same problem as the first test
- Sixth test : Moving a Linux-based VM from proxmox-nfs-storage to nfs-vm-storage. Alone, without any other moving process in the mean time. VM not configured in HA
Conclusion : working, no problems
- Seventh test : Moving a Linux-based VM from proxmox-nfs-storage to local-lvm without any other process. VM not configured in HA
Conclusion : working, no problems
- Eigth test : Moving a the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. With another process of moving VM hardware. VM not configured in HA
Conclusion : failed. Same problem as the first step
- Nineth test : Moving the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. Without any other moving process in the mean time. VM configured in HA
Conclusion : Working
- Tenth test : Moving the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. With another process of moving VM hardware. VM configured in HA.
Conclusion : Not working, same problem as the first test
I have many hypothesis about this unexpected behaviour :
- The HA interferes into the process of moving the hard disk
- The fact that the VMs are currently stored on an NFS storage located on a different network than the destination NFS storage could potentially impact the disk migration process, although I don’t believe this is the root cause.
- Maybe only one process of moving hard disk to another storage is allowed ? That would explain why sometimes it works, sometimes not.
- Maybe the command used to move the hard disk is lock (as it says in the error log) during the process and after the process and can't be used to move another hardware during this time ? But if this is correct, that's strange : why the windows VM did not succeed to move to another NFS Share while there was no other process before and during ?
- Can the machine type be "The THING" that block the process ?? The Windows-base VM is an i440fx machine as the linux a q35.
- Or maybe the problems come from the Windows Server ? But that would be strange that a Windows Server can't handle more than one process related to a NFS Share.
- Maybe this can be a lack of comprehension from me : only one process by port ? and as NFS uses one port : only one process but that would be very limiting.
I have also tried to use iSCSI Storage based on the same Windows Server. But using this technology resulted in a less stable (had some crash (can be explained later but it doesn't help this topic)) and more speedless process (creation, destruction and moves of hardware).
Thanks for reading until this line
and for answering me. This would be very helpful. Maybe this topic was solved but I found no thread about it while researching on google and I found nothing on the wiki (maybe it was written and I'm blind in that case).
Have a nice day
I am managing a cluster with 2 nodes and a qDevice VM to respect the quorum.
Proxmox Server on 9.1.5 version
The IP adresse of Proxmox node are these :
- First node : 192.168.1.1 (MGMT network) 192.168.2.1 (SAN Network)
- Second node : 192.168.1.2 (MGMT network) 192.168.2.2 (SAN Network)
-NFS Server : 192.168.1.150 (MGMT Network) 192.168.2.3 (SAN Network)
The HA is configured on all the VM.
They are Windows-based VMs and Linux-based VMs
The shared storage between the node is a NFS share. This NFS Share is based on a Windows Server 2022.
The creation of VM in the NFS storage is possible, everythings is working fine.
The network link between the node and the Shared storage is a GigabitEthernet cable.
Here is my configuration about my different NFS Share :
I have created 3 NFS Share on Windows Server 2022 :
- proxmox-nfs-storage : only 192.168.2.1 and 192.168.2.2 can access to it (Configured on the NFS Share Properties). But the source of the NFS share is declared using the DNS name of the NFS Server so it might use a different network to access the NFS SHare. NFS Version on proxmox used to connect to : 4.1
- nfs-vm-storage : only 192.168.2.1 and 192.168.2.2 can access to it (Configured on the NFS Share Properties). NFS Version on proxmox used to connect to : Default
- test1 : only 192.168.2.1 and 192.168.2.2 can access to it (Configured on the NFS Share Properties). NFS Version on proxmox used to connect to : Default
I connected both proxmox node to these NFS Shares.
I can successfully create a VM on all of these NFS Share storage.
But I had many problems while trying to move VM between these NFS Share.
It's not really a problem because sometimes it works but I am really confused about some points so here are some tests I conducted if I can have some answers about my hypothesis or maybe it's an undocumented feature
Here are the tests I conducted :
- First test: Moving a Windows-based VM from the proxmox-nfs-storage to test1. Alone, without any other moving process in the mean time. VM not configured in HA
Conclusion : failed to move hard disk.
Code:
create full clone of drive scsi0 (proxmox-nfs-storage:105/vm-105-disk-1.qcow2)
Formatting '/mnt/pve/test1/images/105/vm-105-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=75161927680 lazy_refcounts=off refcount_bits=16
TASK ERROR: storage migration failed: unable to create image: 'storage-test1'-locked command timed out - aborting- Second test : Moving the same VM from the NFS Share proxmox-nfs-storage to local-lvm without any other process. VM not configured in HA
Conclusion : working. No problems.
- Third test : Moving the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. With another process of moving VM hardware (to test1 NFS Share from proxmox-nfs-storage). VM not configured in HA
Conclusion : not working. Same problem as the first test.
- Fourth test : Moving the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. Without any other moving process in the mean time. VM configured in HA
Conclusion : not working. Same problem as the first test
- Fifth test : Moving the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. With another process of moving VM hardware. VM configured in HA.
Conclusion : not working. Same problem as the first test
- Sixth test : Moving a Linux-based VM from proxmox-nfs-storage to nfs-vm-storage. Alone, without any other moving process in the mean time. VM not configured in HA
Conclusion : working, no problems
- Seventh test : Moving a Linux-based VM from proxmox-nfs-storage to local-lvm without any other process. VM not configured in HA
Conclusion : working, no problems
- Eigth test : Moving a the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. With another process of moving VM hardware. VM not configured in HA
Conclusion : failed. Same problem as the first step
- Nineth test : Moving the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. Without any other moving process in the mean time. VM configured in HA
Conclusion : Working
- Tenth test : Moving the same VM from the NFS Share proxmox-nfs-storage to nfs-vm-storage. With another process of moving VM hardware. VM configured in HA.
Conclusion : Not working, same problem as the first test
I have many hypothesis about this unexpected behaviour :
- The HA interferes into the process of moving the hard disk
- The fact that the VMs are currently stored on an NFS storage located on a different network than the destination NFS storage could potentially impact the disk migration process, although I don’t believe this is the root cause.
- Maybe only one process of moving hard disk to another storage is allowed ? That would explain why sometimes it works, sometimes not.
- Maybe the command used to move the hard disk is lock (as it says in the error log) during the process and after the process and can't be used to move another hardware during this time ? But if this is correct, that's strange : why the windows VM did not succeed to move to another NFS Share while there was no other process before and during ?
- Can the machine type be "The THING" that block the process ?? The Windows-base VM is an i440fx machine as the linux a q35.
- Or maybe the problems come from the Windows Server ? But that would be strange that a Windows Server can't handle more than one process related to a NFS Share.
- Maybe this can be a lack of comprehension from me : only one process by port ? and as NFS uses one port : only one process but that would be very limiting.
I have also tried to use iSCSI Storage based on the same Windows Server. But using this technology resulted in a less stable (had some crash (can be explained later but it doesn't help this topic)) and more speedless process (creation, destruction and moves of hardware).
Thanks for reading until this line
and for answering me. This would be very helpful. Maybe this topic was solved but I found no thread about it while researching on google and I found nothing on the wiki (maybe it was written and I'm blind in that case).Have a nice day