Cannot upload ISO or VM to NFS share

[thinkdez]

So I am new to proxmox ve but have worked in VMWare for the last 10 years. I just setup a new 3 node cluster to play with and mounted an NFS share no problem. My issue is when I try to move a VM or upload a ISO to the NFS share the host I am stops communicating with the NFS Server then locks up and I have to hard reboot the node to get access back. It locks up almost as soon as it starts the copy. I tried setting up a SMB share as well and I get the same experieince.

My Journal shows the following:
NFS Share:

Aug 16 16:37:41 pve02 pvedaemon[906]: <root@pam> starting task UPID:pve02:00000972:0000E589:64DD3395:imgcopy::root@pam:
Aug 16 16:38:49 pve02 pvestatd[881]: unable to activate storage 'VirtualMachines' - directory '/mnt/pve/VirtualMachines' does not exist or is unreachable
Aug 16 16:38:59 pve02 pvestatd[881]: unable to activate storage 'VirtualMachines' - directory '/mnt/pve/VirtualMachines' does not exist or is unreachable
Aug 16 16:39:09 pve02 pvescheduler[2622]: jobs: cfs-lock 'file-jobs_cfg' error: got lock request timeout

SMB Share:

Aug 16 16:26:30 pve02 kernel: CIFS: VFS: reconnect tcon failed rc = -2
Aug 16 16:26:32 pve02 kernel: CIFS: VFS: \\10.1.100.20\VirtualMachines BAD_NETWORK_NAME: \\10.1.100.20\VirtualMachines

The interesting part is that when I went into the shell and created a new file on the NFS share it had no issue. But when I tried to do a cp command it would lock up.

The permissions are good as I can create new files from the shell but the upload and cp commands lock it up and disconnect the share.

Any guidance on what I am missing would be appreciated.

Thanks

 

[bbgeek17]

It sounds like a network issue, perhaps packet loss under load. I would concentrate on doing manual mount testing, analyzing network interface errors, iperf type testing, etc. NFS and CIFS are industry standard protocols, PVE is using standard Linux client interfaces to access the Server on the other side. You need to look at all pieces involved (NFS client, network (nic,cable,switch), NFS server). Try to isolate and reduce the number of components.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[thinkdez]

I was doing some testing and it seems that a single NIC was the issue. I am using a Elitedesk G3 Mini and I am still waiting on additional NIC's to install. I had a spare USB Nic for my laptop and I tested it out and it seems that resolved the issue. I think since the managment was on the same interface the storage copy was saturating the link causing the managment to fail and disconnect from the cluster stopping the upload/copy of data. I have more testing to figure out what's the optimum setup but I think I found the culprit.

 

[bbgeek17]

I have more testing to figure out what's the optimum setup but I think I found the culprit.

Good job on identifying the problem piece.

IMHO, there is no way you can saturate even a 1Gbit link with a single cifs/nfs copy to a point that a second lightweight management session would stop functioning. But if swapping NIC fixes your problem, thats all that matters.

good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox