Mount no longer works in Proxmox 6 - NFS - Synology

CookieMonster · Jul 30, 2019

Ey,

First time posting, as I have always been able to find solutions on the forums (great job on that).

The setup:
I have a 3 node Proxmox cluster, and today upgraded them to Proxmox VE 6.
On VE 5.4 all nodes had a nfs mount and everything worked.
Two of the nodes have a two network interfaces connected to NET-1 and NET-2 (and the third only has NET-2)

The problem:
After the upgrade to VE 6 all nodes are unable to mount the nfs share.
The nfs mount was running over NET-2 <-- this no longer works (nothing on the side of the NAS has changed).

Now when I manually try to mount it on any of the nodes it mounts just fine, and I'm able to access all files etc (NET-1 and NET-2 work).

Code:

mount <IP or DNS>:/volume2/wooptest /mnt/test/

When I try to add a NFS sure over NET-1 the Proxmox is able to find the NAS and all nfs shares available. But on NET-2 it does not find anything (either via the GUI of CLI)

Code:

pvesm add nfs woop --path /mnt/test --server <IP OR DNS> --export /volume2/wooptest
create storage failed: error with cfs lock 'file-storage_cfg': storage 'woop' is not online

I found some posts about dns names, all of these where already in place before the upgrade, but does not seem to do anything.

I hope someone has any idea what is going on.

Greetings

Alwin · Jul 31, 2019

CookieMonster said:
Now when I manually try to mount it on any of the nodes it mounts just fine, and I'm able to access all files etc (NET-1 and NET-2 work).

You could check the mount parameters with a PVE 5.x installation, to verify if those may have changed.

To note, it is also very common that the NFS server needs an update, to play nicely with 'modern' clients.

CookieMonster · Aug 1, 2019

The NFS server is a Synology NAS on the latest update (with nfs 4.1 enabled)
There are no special mount parameters.
It would not explain why all of a sudden after the upgrade it stops working on NET-2 but everything works find if I go via NET-1 (the networks are on differed ip rages with a firewall in between)

n1nj4888 · Aug 1, 2019

I had an NFS 4.1 mount to my Synology NAS working under PVE5.4. When I upgraded (actually rebuilt) to PVE6, the same NFS 4.1 mount works to the Synology so assume it must be some config issue in your environment...

I added the NFS mounts through the GUI for both PVE5.4 and PVE6 using the IP of the NAS and, "NET-1" rather than my "NET-2" (corosync LAN in my case)...

CookieMonster · Aug 1, 2019

n1nj4888 said:
I had an NFS 4.1 mount to my Synology NAS working under PVE5.4. When I upgraded (actually rebuilt) to PVE6, the same NFS 4.1 mount works to the Synology so assume it must be some config issue in your environment...

I added the NFS mounts through the GUI for both PVE5.4 and PVE6 using the IP of the NAS and, "NET-1" rather than my "NET-2" (corosync LAN in my case)...

Well, this gives some hope. Now I just have to figure out where in Proxmox it refuses to connect to the NAS.

I'm also trying to add it via the GUI, however when its searching for the export box, it just ends and finds nothing (and on the other network it finds it instantly). I even disabled the firewall on the NAS to test if that was the problem (tho if it was manual mount should not have worked).

Because you mentioned you did a clean install, I'm thinking something went wrong during the upgrade.
Any idea how to further debug this (don't see anything in the logs that could give me more info)?

n1nj4888 · Aug 1, 2019

I assume the IPs didn’t change on your PVE hosts from PVE5.4 to PVE6? For example is the NFS export of the shared folder on the synology configured to enable the share for both incoming NET-1 IP address/subnet (eg x.x.x.0/24) and NET-2 IP address/subnet (eg y.y.y.0/24)?

Also assume that the synology has an IP on your NET-2 range and this hasn’t changed since the PVE6 upgrade?

CookieMonster · Aug 1, 2019

n1nj4888 said:
I assume the IPs didn’t change on your PVE hosts from PVE5.4 to PVE6? For example is the NFS export of the shared folder on the synology configured to enable the share for both incoming NET-1 IP address/subnet (eg x.x.x.0/24) and NET-2 IP address/subnet (eg y.y.y.0/24)?

Also assume that the synology has an IP on your NET-2 range and this hasn’t changed since the PVE6 upgrade?

App ip's are the same. On the Synology I all ip's are added for access (already checked and made sure its correct). Also tested when I remove some of the ip's I fails as it should. Same with firewall settings (even turned it all the way off for testing)

I did configure an iSCSI for now until NFS works again and this connects and works fine over NET-2 and a CIFS mount works fine as well.

Note: Manual mounting of the NFS share over NET-2 still works, but via Proxmox storage it fails (see error in the first post)

CookieMonster · Aug 1, 2019

So for testing, installed a virtual Proxmox on my Proxmox, and tried to connect to the NFS there. The test node has the same 2 networks.
I added the new IP to the NFS permissions in Synology DSM.

- Manual mount works on NET-2 and NET-1
- Proxmox mount does not work on NET-2

Remove NET-1 network card and tried again (did a reboot in between):

- Manual mount works
- Proxmox mount does not work

So at this point, I'm thinkings its the NAS, but why does the manual mount work fine? and why does Proxmox NET-1 works fine as well?
Also its Proxmox, but why does it work on NET-1?

On NET-2 all traffic to each other node is unrestricted (the NAS has a firewall setup, however I disabled it completely while testing)

Code:

    NET-1                NET-2
    192.168.128             192.168.10           

    .5        Synology    .10
    .6        prox-1        .4 (default gateway)
    .7        prox-2        .5 (default gateway)
               prox-3        .6 (default gateway)

    .200    prox-test    .200 (default gateway)

See image for Synology nfs permissions

So either I'm missing something small, or something is broken somewhere but I have no idea what.

n1nj4888 · Aug 1, 2019

“Note: Manual mounting of the NFS share over NET-2 still works, but via Proxmox storage it fails (see error in the first post)”

So, I assume from the above you can mount the synology NFS share via the Proxmox host CLI/SSH (for each host) but it doesn’t work from the Proxmox GUI? If so, this would indicate it’s a PVE GUI issue and not the NAS? Have you tried mounting the same share via the PVE GUI but using different NFS versions?

I’m also confused by the code snippet in your last post where “default gateway” is mentioned under NET-2:

(1) Where did this output originate from?
(2) What is the “default gateway” for NET-2? For example, is it set on the same 192.168.10.x range as NET-2 or something else? If the synology NAS has its own 192.168.10.x IP (assuming a /24 subnet) then there shouldn’t need to be any NET-2 gateway configured for the NET-2 connection?

CookieMonster · Aug 1, 2019

n1nj4888 said:
“Note: Manual mounting of the NFS share over NET-2 still works, but via Proxmox storage it fails (see error in the first post)”

So, I assume from the above you can mount the synology NFS share via the Proxmox host CLI/SSH (for each host) but it doesn’t work from the Proxmox GUI? If so, this would indicate it’s a PVE GUI issue and not the NAS? Have you tried mounting the same share via the PVE GUI but using different NFS versions?

See my first post for mount commands (first one is linux, second is Proxmox) these are both run on CLI/SSH on the nodes.
Tried all the versions, does not make a difference, the "Export" box stays empty on the GUI when using the NET-2 NAS ip

I’m also confused by the code snippet in your last post where “default gateway” is mentioned under NET-2:

(1) Where did this output originate from?
(2) What is the “default gateway” for NET-2? For example, is it set on the same 192.168.10.x range as NET-2 or something else? If the synology NAS has its own 192.168.10.x IP (assuming a /24 subnet) then there shouldn’t need to be any NET-2 gateway configured for the NET-2 connection?

1) Handmade in a text editor
2) NET-2 is the default way to the network / internet for Proxmox. Gateway: 192.168.10.1. Note: Proxmox only allows for 1 gateway config, NET-1 does not have a gateway. Both networks are /24

Alwin · Aug 1, 2019

CookieMonster said:
Note: Manual mounting of the NFS share over NET-2 still works, but via Proxmox storage it fails (see error in the first post)

What are the mount options? And what does 'rpcinfo -p <IP-interface>' show on both?

CookieMonster · Aug 1, 2019

Alwin said:
What are the mount options? And what does 'rpcinfo -p <IP-interface>' show on both?

Result the same on all nodes

rpcinfo -p 192.168.10.10

Code:

192.168.10.10: RPC: Remote system error - No route to host

rpcinfo -p 192.168.128.5

Code:

   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp    892  mountd
    100005    1   tcp    892  mountd
    100005    2   udp    892  mountd
    100005    2   tcp    892  mountd
    100005    3   udp    892  mountd
    100005    3   tcp    892  mountd
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100021    1   udp  44038  nlockmgr
    100021    3   udp  44038  nlockmgr
    100021    4   udp  44038  nlockmgr
    100021    1   tcp  59888  nlockmgr
    100021    3   tcp  59888  nlockmgr
    100021    4   tcp  59888  nlockmgr
    100024    1   udp  44247  status
    100024    1   tcp  41581  status

After seeing this I also tried the

Code:

showmount -e ...

and on NET-1 (128) it shows the mounts and NET-2 (10) it also shows the no route to host.

Tried a ping, this works fine (firewall is of so kinda everything else works now)

Alwin · Aug 2, 2019

Might be an issue with an firewall. See the link for more troubleshooting tips.
http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html

What does 'pve-firewall status' show?

CookieMonster · Aug 2, 2019

Alwin said:
Might be an issue with an firewall. See the link for more troubleshooting tips.
http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html

What does 'pve-firewall status' show?

Ill have a read of the post later today.

pve-firewall status:

Code:

Status: disabled/running

Alwin · Aug 2, 2019

Is the iptable empty?

Code:

iptables -nvL
iptables -nvL -t nat

CookieMonster · Aug 2, 2019

I never did anything with ip tables so this should de default

iptables -nvL

Code:

Chain INPUT (policy ACCEPT 24M packets, 30G bytes)
 pkts bytes target     prot opt in     out     source               destination       

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination       

Chain OUTPUT (policy ACCEPT 20M packets, 60G bytes)
 pkts bytes target     prot opt in     out     source               destination

iptables -nvL -t nat

Code:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination       

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination       

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination       

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Alwin · Aug 3, 2019

CookieMonster said:
rpcinfo -p 192.168.10.10

Is there another host you could run the rpcinfo and see the output? And maybe a different kernel, just to in which direction we should investigate.

CookieMonster · Aug 3, 2019

Alwin said:
Is there another host you could run the rpcinfo and see the output? And maybe a different kernel, just to in which direction we should investigate.

Great minds thing a like? As I just (like 2 mins ago) installed a ubuntu desktop inside Proxmox with only NET-2

The ubuntu machine can see and route to the NAS without issue.

CookieMonster · Aug 4, 2019

Alwin said:
http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html

So looking at everything I'm at the "you are the victim of an overzealous firewall"

So all firewalls:
- Router --> only blocks outside traffic, and traffic to another subnet. But all traffic within an ip range is allowed. (Also did not change before/during/after the upgrade)
- NAS --> disabled for test/debugging, however when enabled allows NFS
- Proxmox --> running but disabled. Never used the Proxmox firewall, and should be all defaults. However made sure its disabled by running "pve-firewall stop" even then get the no route to host error.

As there are no other firewalls, I have no idea at this point. Going to install older version of Proxmox see if that works.

CookieMonster · Aug 4, 2019

So update:

3 clean installs from the ISO installed on the current proxmox cluster.
All have 1 network interface on NET-2

Proxmox 5.3-8 -> Connects to NFS without problems
Proxmox 5.4-3 -> Connects to NFS without problems
Proxmox 6.0-4-> Does not work, rpcinfo gives no route to host
Conclusion do not use proxmox 6.0

@Alwin Any idea's what changed in version 6 that breaks NFS?

Mount no longer works in Proxmox 6 - NFS - Synology

New Member

Proxmox Retired Staff

New Member

Well-Known Member

New Member

Well-Known Member

New Member

New Member

Attachments

Well-Known Member

New Member

Proxmox Retired Staff

New Member

Proxmox Retired Staff

New Member

Proxmox Retired Staff

New Member

Proxmox Retired Staff

New Member

New Member

New Member