Mount no longer works in Proxmox 6 - NFS - Synology

Vladimir Bulgaru · Jan 27, 2020

Alwin said:
Did you restart the pvedaemon.service?

Yes. Can you try and confirm? Maybe another service needs to be restarted as well?

Alwin · Jan 29, 2020

Vladimir Bulgaru said:
Yes. Can you try and confirm? Maybe another service needs to be restarted as well?

No other daemon needed. Is the path '/home/temp/' writable?

Vladimir Bulgaru · Jan 29, 2020

Alwin said:
No other daemon needed. Is the path '/home/temp/' writable?

Yes, it is. The weird aspect is that neither the CLI output nor file output work for both - successful NFS mount and unsuccessful one. I even started doubting that this is the function responsible for the mount

Alwin · Jan 29, 2020

I tried it on my end and the server:port was written to my test file. Try to umount the nfs storage and run pvesm list (possibly multiple times) to get the storage mounted again.

Vladimir Bulgaru · Jan 30, 2020

Alwin said:
I tried it on my end and the server:port was written to my test file. Try to umount the nfs storage and run pvesm list (possibly multiple times) to get the storage mounted again.

Ok, so the server restart helped for some reason.
Here's a new riddle for you:

when mounting the local NAS, there is output to the text file
when mounting the offsite NAS, there is no output at all (seems that the function is not reached during execution).

Where can it get blocked on the way?

Alwin · Jan 30, 2020

Vladimir Bulgaru said:
when mounting the offsite NAS, there is no output at all (seems that the function is not reached during execution).

Try to print some extra output into the file to verify. The function should be passed.

Vladimir Bulgaru · Feb 5, 2020

Alwin said:
Try to print some extra output into the file to verify. The function should be passed.

No output, no matter how i try.
I get the output only when mounting is successful.
Seems that function is not executed at all.

Alwin · Feb 6, 2020

Hm. Odd, as the nfs_mount method is the one that is called for the mount.

The Plugins are loaded through Storage.pm, that calls Plugin.pm and in turn that executes the method in the plugin.

Code:

/usr/share/perl5/PVE/Storage/Plugin.pm
/usr/share/perl5/PVE/Storage.pm

This will leave you with more debugging. :/ A workaround would be to create a directory storage and set the --is_mountpoint . The mount can then be done through fstab and PVE only checks if it is mounted.
https://pve.proxmox.com/pve-docs/pvesm.1.html

noname · Feb 19, 2020

Alwin said:
Hm. Odd, as the nfs_mount method is the one that is called for the mount.

The Plugins are loaded through Storage.pm, that calls Plugin.pm and in turn that executes the method in the plugin.

Code:

/usr/share/perl5/PVE/Storage/Plugin.pm /usr/share/perl5/PVE/Storage.pm

This will leave you with more debugging. :/ A workaround would be to create a directory storage and set the --is_mountpoint . The mount can then be done through fstab and PVE only checks if it is mounted.
https://pve.proxmox.com/pve-docs/pvesm.1.html

i try also to connect nfs with proxmox 6.1.7 with no lucky, with proxmox 5.4.13 everything okay....any ideas?

Alwin · Feb 20, 2020

noname said:
i try also to connect nfs with proxmox 6.1.7 with no lucky, with proxmox 5.4.13 everything okay....any ideas?

Some as for the others. Read through the thread and try the suggestions. If nothing helps, then explain in detail your setup and start debugging.

Vladimir Bulgaru · Feb 27, 2020

@Alwin
After some long interactions with the ISP and debugging on my end, here are the findings:

NFS fails when check_connection function runs
for some reason it generates the following error command '/sbin/showmount --no-headers --exports XX.XX.XX.XX' failed: got timeout
it works perfectly fine on Proxmox 5
i've tried to run /sbin/showmount --no-headers --exports XX.XX.XX.XX directly in the command line and it times out on Proxmox 6, but works fine on Proxmox 5 (same behavior as when adding storage via pvesm add nfs)
the showmount version in both cases is 1.3.3
i've cheated and removed the failure condition from check_connection. It allowed me to add the storage to Proxmox 6. It shows with a question-mark (probably status function fails as well), but the backup works fine.

Question of the day is - why does showmount fail?

Can there be additional networking settings in Proxmox 6 compared to Proxmox 5? Again - talking about the fresh installation without any configs, settings and apps installed atop.

Update 1: Forgot to mention that rpcinfo -p XX.XX.XX.XX fails the same way as showmount.

Update 2: I seem to be able to telnet both 111 and 2049 ports and i can ssh into the NAS from Proxmox 6.

Vladimir Bulgaru · Mar 9, 2020

Vladimir Bulgaru said:
@Alwin
After some long interactions with the ISP and debugging on my end, here are the findings:

NFS fails when check_connection function runs

for some reason it generates the following error command '/sbin/showmount --no-headers --exports XX.XX.XX.XX' failed: got timeout

it works perfectly fine on Proxmox 5

i've tried to run /sbin/showmount --no-headers --exports XX.XX.XX.XX directly in the command line and it times out on Proxmox 6, but works fine on Proxmox 5 (same behavior as when adding storage via pvesm add nfs)

the showmount version in both cases is 1.3.3

i've cheated and removed the failure condition from check_connection. It allowed me to add the storage to Proxmox 6. It shows with a question-mark (probably status function fails as well), but the backup works fine.

Question of the day is - why does showmount fail?

Can there be additional networking settings in Proxmox 6 compared to Proxmox 5? Again - talking about the fresh installation without any configs, settings and apps installed atop.

Update 1: Forgot to mention that rpcinfo -p XX.XX.XX.XX fails the same way as showmount.

Update 2: I seem to be able to telnet both 111 and 2049 ports and i can ssh into the NAS from Proxmox 6.

Hey!

Any ideas?
Would somebody help me with temporary access to Proxmox 6 to cross out the networking issues?

Alwin · Mar 9, 2020

@Vladimir Bulgaru, I didn't get around to look at it yet. But showmount uses rpc as well and since the rpcinfo fails (as with the other NFS issues in this thread), it well be an issue with the NFS implementation in the kernel.

Vladimir Bulgaru · Mar 11, 2020

Alwin said:
@Vladimir Bulgaru, I didn't get around to look at it yet. But showmount uses rpc as well and since the rpcinfo fails (as with the other NFS issues in this thread), it well be an issue with the NFS implementation in the kernel.

Now i am really confused, which is a good and a bad sign at once

Since you've mentioned that it may be a kernel issue, i was wondering what will the container behaviour be. I've created an Ubuntu 18.04 container and first deployed it on Proxmox 5. sudo rpcinfo -p xx.xx.xx.xx worked fine. I've exported this very same container to Proxmox 6 and to my huge surprise sudo rpcinfo -p xx.xx.xx.xx worked fine.

I don't claim to be an expert on virtualisation, but AFAIK the containers use the same kernel as the hypervisor. Hence the kernel works just fine. Given that the container and the hypervisor have the exactly same environment when it comes to:

firewall and networking
kernel
rpcbind version

i really struggle to understand what else can fail

Alwin · Mar 11, 2020

Hm... good question. Container run in their own cgroup and namespace. And it matters if they are privileged or unprivileged container. But what program version is the showmount / rpcinfo in Ubuntu?

Vladimir Bulgaru · Mar 11, 2020

Alwin said:
Hm... good question. Container run in their own cgroup and namespace. And it matters if they are privileged or unprivileged container. But what program version is the showmount / rpcinfo in Ubuntu?

The container is unpriviledged.
As for the rpcbind version (the main package that allows for rpcinfo), this is for container:

Code:

rpcbind:
  Installed: 0.2.3-0.6
  Candidate: 0.2.3-0.6
  Version table:
 *** 0.2.3-0.6 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

and this is for Proxmox 6:

Code:

rpcbind:
  Installed: 1.2.5-0.3+deb10u1
  Candidate: 1.2.5-0.3+deb10u1
  Version table:
 *** 1.2.5-0.3+deb10u1 500
        500 http://ftp.debian.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status

iamnotarobot · Mar 22, 2020

I have the same setup and the exact same issue. NFS does not work, but iSCSI for example does. Synology NAS and proxmox 6.1-8.

The NFS share worked prior to upgrade from version 5. The network has not been changed so it was the upgrade that broke something. And telneting port 111 from proxmox to NAS works on version 6 as well.

"pvesm list NFS2" for example results in "storage 'NFS2' is not online", and when diagnosing it with various commands from this thread it errors with "no route to host".

iamnotarobot · Mar 22, 2020

Forgot to add, but "rpcinfo -p x.x.x.x" also gives "no route to host".

iamnotarobot · Mar 24, 2020

It would appear this interferes with destruction of machines as well. I put a machine on iSCSI storage and whenever I destroy that the task fails with "storage 'NFS2' is not online"

Vladimir Bulgaru · Mar 24, 2020

iamnotarobot said:
It would appear this interferes with destruction of machines as well. I put a machine on iSCSI storage and whenever I destroy that the task fails with "storage 'NFS2' is not online"

Quick question - have you tried using mount command directly in CLI? In my case it works.
The whole Proxmox 6 attachment routine fails because rpcinfo fails.
Is it identical in your case?

Mount no longer works in Proxmox 6 - NFS - Synology

Renowned Member

Proxmox Retired Staff

Renowned Member

Proxmox Retired Staff

Renowned Member

Proxmox Retired Staff

Renowned Member

Proxmox Retired Staff

Active Member

Proxmox Retired Staff

Renowned Member

Renowned Member

Proxmox Retired Staff

Renowned Member

Proxmox Retired Staff

Renowned Member

New Member

New Member

New Member

Renowned Member