Upgrade 5.4 to 6 and NFS

[Blaž Bogataj]

Hello
Upgrade from 5.4 to 6 goes really smooth. Great yob.
After reboot connection to existing NFS in no more available - on NFS system store daily backups.
For test install new system, situation is the same.

I can't get connection from proxmox to NAS:
Situation on proxmox
# rpcinfo -p 10.0.187.68 – this is from proxomx to proxmox machine
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
# rpcinfo -p 10.0.187.207 – this is from proxmox to NAS Synology
10.0.187.207: RPC: Remote system error - No route to host
I can ping NAS.

Situation on NAS:
# rpcinfo -p 10.0.187.68 – this is from NAS to je promox
program vers proto port service
100000 4 tcp 111
100000 3 tcp 111
100000 2 tcp 111
100000 4 udp 111
100000 3 udp 111
100000 2 udp 111
# rpcinfo -p 10.0.187.207 – this is from NAS to NAS
program vers proto port service
100000 4 tcp 111
100000 3 tcp 111
100000 2 tcp 111
100000 4 udp 111
100000 3 udp 111
100000 2 udp 111
100005 1 udp 892
100005 1 tcp 892
100005 2 udp 892
100005 2 tcp 892
100005 3 udp 892
100005 3 tcp 892
100003 2 tcp 2049
100003 3 tcp 2049
100003 4 tcp 2049
100003 2 udp 2049
100003 3 udp 2049
100021 1 udp 54059
100021 3 udp 54059
100021 4 udp 54059
100021 1 tcp 32834
100021 3 tcp 32834
100021 4 tcp 32834
100024 1 udp 39680
100024 1 tcp 54722

Connection from proxmox to another Synology working without troubles. Also connection working to this NAS from the same proxmox until upgrade without troubles.

What I miss, that NFS connection cannot be established. Everything is on the same system, which before upgrade work without any issue.

Thanks in advance

 

[Alwin]

Anything in the syslog/journal? Firewall activated? How does the /etc/pve/storage.cfg look like?

 

[Blaž Bogataj]

In syslog: virttest0 pvestatd[1057]: storage 'nfsvirtualci' is not online
In journal:
virttest0 systemd[1]: nfs-config.service: Succeeded.
virttest0 systemd[1]: Condition check resulted in RPC security service for NFS server being skipped.
virttest0 systemd[1]: Condition check resulted in RPC security service for NFS client and server being skipped.
virttest0 systemd[1]: Reached target NFS client services.

storage.cfg:
root@virttest0:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,backup,vztmpl

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

nfs: nfsvirtualci
export /volume1/proxmox
path /mnt/pve/nfsvirtualci
server 10.0.187.207
content backup,images,iso
maxfiles 3

Firewall:
pve-firewall status
Status: disabled/running

Best regards
Blaž

 

[tin]

Check the obvious if you haven't - IP address on the Proxmox machine is what you think it is, IP address is in the allowed hosts on the NFS server, etc.

 

[Blaž Bogataj]

Tin, thanks for tip.
root@virttest0:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 2c:44:fd:81:18:e4 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether 2c:44:fd:81:18:e5 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 2c:44:fd:81:18:e6 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 2c:44:fd:81:18:e7 brd ff:ff:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 2c:44:fd:81:18:e5 brd ff:ff:ff:ff:ff:ff
inet 10.0.187.68/24 brd 10.0.187.255 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::2e44:fdff:fe81:18e5/64 scope link
valid_lft forever preferred_lft forever


On Squash field also switch to Map all users to admin, result is same.

On this NFS server and on same share I can backup from prox 5.4, which is not upgraded (yet) without problem.

Thanks
Blaž

 

[Alwin]

Did the NFS server block the node? Is there something in the dmesg log?

As a test, could you run a freshly installed Proxmox VE 6 and try if it works with it?

 

[Blaž Bogataj]

All comments, logs, ... in this post are from freshly installed system. Situation on upgraded is same.
Same Synology NFS server work with "unupgradet" proxmox server without any issue.

Thanks Blaž

 

[Blaž Bogataj]

Finally. After 4 days ... with new installed test system NFS start to work as expected.
Solution is add DNS record in our DNS server. After that I can add NFS storage and when enter ip of NFS server proxmox list available nfs shares.
On proxmox servers until ver. 6 this working without this.

Hope this shorten debugging someone.

Thanks
Blaž

 

[Alwin]

I could not reproduce this. NFS sever with IP only, no DNS name, Proxmox VE 6 connects.

 

[Blaž Bogataj]

After adding dns record into AD new, testing prox and also old one, which is upgraded start to work.
On Synology is no log about nfs, so I'cant find anything. Then test with name for NAS server and find that in DNS is no record for Synology. When add record testing prox (I edit storage.cfg and insert nfs part) tail -f stop to complain about nfs.

Thanks,
Blaž

 

[Vladimir Bulgaru]

Confirming the issue. I have the following setup:
NAS connected to the router, which forwards the incoming traffic to NAS. I connect to the router via DDNS (no static ip).
I have 2 servers: 1 running Proxmox 5 and one running Proxmox 6. Both ips of these servers are allowed to connect to NAS.
The Proxmox 5 server connects to NAS, while Proxmox 6 fails. Both are using the fresh Proxmox installs, no firewall settings.
Please help debug and adjust the settings on Proxmox 6.

 

[lmarschk]

Having the same issue here after upgrading to VE6.

I am seeing exactly the same behavior as stated in the first post. Stupid enough, it is possible to mount the nfs via mount.nfs4 directly (mount.nfs4 10.20.11.1:exportName /tmp/mount workd) but not through Proxmox.

I tried to add DNS records as well but that didn't work for me. (Tried forward and reverse records for NFS server).

Regarding to my tcpdump it seems as both servers (NFS and Proxmox) are able to communicate (full TCP connection established and correct teardown as well) during running rpcinfo.

root@s04:~# rpcinfo -p 10.20.11.1
10.20.11.1: RPC: Remote system error - Connection timed out

Have debugged all this for several hours now, it seems to be an issue with newer versions of rpcbind. These newer versions are using GETADDR commands which contain the IP address of the server at application layer. This means in case you have (like me) a NAT between your Proxmox and NFS-Server the Proxmox server will issue a GETADDR, get the "real" address of the server back on application layer and then try to connect to this address (which will obviously fail). As the address comes on application layer, no NAT will rewrite it. At least not mine. Very annoying. (Older versions just issue a DUMP command and then get the exports.)

I don't know if you would like to document this behavior somewhere. It is really not intuitive. FYI: So far I used a Docker container to provide me a little NFS server on one interface of a larger server. Therefore, I had a NAT for the port-forwarding to the docker container and Proxmox tried to contact the internal IP address of the Docker container directly.

tl;dr: get your (mine) networking done right. And don't use NATs in between your Proxmox and NFS/NAS

Hope this helps someone..

 

[Martin Jonáš]

I was ale to "hack" it to work. It seems that ( at lest in our case) mount works but what fails is detection if storage is online. I was able to bypass this by editing file /usr/share/perl5/PVE/Storage/NFSPlugin.pm and replacing:

 my $cmd = ['/sbin/showmount', '--no-headers', '--exports', $server];

by

my $cmd = ['/usr/sbin/rpcinfo', $server];

And then restarting services pvestatd and pvedaemon

It seems that problem occurs when NFS share is behind NAT. Commands showmountand rpcinfo -p fails because they tries to connect to share local address not it public address. Using just rpcinfo solves this problem for us.

 

[Martin Jonáš]

Having the same issue here after upgrading to VE6.

I am seeing exactly the same behavior as stated in the first post. Stupid enough, it is possible to mount the nfs via mount.nfs4 directly (mount.nfs4 10.20.11.1:exportName /tmp/mount workd) but not through Proxmox.

I tried to add DNS records as well but that didn't work for me. (Tried forward and reverse records for NFS server).

Regarding to my tcpdump it seems as both servers (NFS and Proxmox) are able to communicate (full TCP connection established and correct teardown as well) during running rpcinfo.

root@s04:~# rpcinfo -p 10.20.11.1
10.20.11.1: RPC: Remote system error - Connection timed out

Have debugged all this for several hours now, it seems to be an issue with newer versions of rpcbind. These newer versions are using GETADDR commands which contain the IP address of the server at application layer. This means in case you have (like me) a NAT between your Proxmox and NFS-Server the Proxmox server will issue a GETADDR, get the "real" address of the server back on application layer and then try to connect to this address (which will obviously fail). As the address comes on application layer, no NAT will rewrite it. At least not mine. Very annoying. (Older versions just issue a DUMP command and then get the exports.)

I don't know if you would like to document this behavior somewhere. It is really not intuitive. FYI: So far I used a Docker container to provide me a little NFS server on one interface of a larger server. Therefore, I had a NAT for the port-forwarding to the docker container and Proxmox tried to contact the internal IP address of the Docker container directly.

tl;dr: get your (mine) networking done right. And don't use NATs in between your Proxmox and NFS/NAS

Hope this helps someone..

Was you able to find way to access NFS shares hidden behind NAT? We have NFS accessible drive used to off-site backups, but it is in building where local netowrk is behind NAT and we cannot change that. We hacked it by solution psoted above but it is really ugly hack to use.

 

[Alwin]

I was ale to "hack" it to work. It seems that ( at lest in our case) mount works but what fails is detection if storage is online. I was able to bypass this by editing file /usr/share/perl5/PVE/Storage/NFSPlugin.pm and replacing:

Code changes will be lost on update. Best is to mount the NFS share through fstab, add the storage with is_mountpoint and shared flags.