NFS - storage 'NAS' is not online (500) - rpcinfo hangs

[Ninos]

Hey there,

after upgrading from proxmox 5.x to 6.x my NFS mount is not working anymore. Mounting the NAS storage via command all is working fine:
mount -t nfs NAS_IP:/SHAREDIR/ /mnt/pve/NAS/

But adding storage to proxmox I always get following error:

storage 'NAS' is not online (500)

So I tried rpcinfo and it seems, that rpcinfo is not working correctly. rpcinfo -p localhost is working fine, rpcinfo -p ANY_IP_INTERNAL_OR_EXTERNAL is hanging (also Ctrl + C does not quit the command after freezing).

Attached the output of following command:
strace rpcinfo -p NAS_IP

Can someone tell me why rpcinfo is not working anymore and/or how I can solve the problem?

Kindest regards

 

[oguz]

hi,

could you post your /etc/pve/storage.cfg

Can someone tell me why rpcinfo is not working anymore and/or how I can solve the problem?

it's working fine for me here, so my guess is that it's something specific to your setup, possibly networking.

do you have connectivity to the NAS by other means? can you ping it? portscan it?

 

[Ninos]

Yeah for sure:

dir: local
        path /var/lib/vz
        content iso,backup,vztmpl

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

nfs: NAS
        export /NODE
        path /mnt/pve/NAS
        server NAS_IP
        content backup
        maxfiles 4

I can ping and portscan (nmap) every internal and external IP, so also my NAS. Just rpcinfo is not working (hanging on IP other than localhost/127.0.0.1).

Kindest regards

 

[oguz]

what about the rpcclient version? rpcclient -V

also what does the following command return: showmount --no-headers --exports NAS_IP
do you get the shares printed out?

 

[Ninos]

root@NODE:~# rpcclient -V
Version 4.9.5-Debian

showmount --no-headers --exports NAS_IP is also hanging like rpcinfo. Also it seems to be the same error, see attached strace

 

[oguz]

is it possible that you have a firewall setup that's dropping the packets to rpcbind? i tried that and got a similar strace to yours with ENOTCONN

 

[Ninos]

not really, before upgrade to latest proxmox all worked fine. Also I have an ethernet cable to the NAS directly, so no fw between proxmox and NAS (QNAP).

BTW: I also tried rpcinfo on windows, ubuntu and fresh proxmox 6.1 installation to NAS_IP. On Windows & Ubuntu all worked fine, on fresh proxmox 6.1 installation I had the same problems as my proxmox 5->6 upgrade instance. So I think it's a proxmox or proxmox default configuration problem.

BTW2: After long time on proxmox I get following error:
NAS_IP: RPC: Remote system error - Connection timed out

 

[oguz]

BTW: I also tried rpcinfo on windows, ubuntu and fresh proxmox 6.1 installation to NAS_IP. On Windows & Ubuntu all worked fine, on fresh proxmox 6.1 installation I had the same problems as my proxmox 5->6 upgrade instance. So I think it's a proxmox or proxmox default configuration problem.

i don't think it's a proxmox default config problem, since nobody else is reporting this and we can't reproduce it (the only exception being with the firewall DROP rule as i explained in my previous post).

only thing i can suggest you right now is to run tcpdump/wireshark on both PVE and NAS side and see what's going on when you run rpcinfo

 

[Ninos]

tcpdump on Proxmox is responding following:

root@NODEXXX:~# tcpdump -i vmbr0 -nn -s0 -vvv dst NAS_IP
tcpdump: listening on vmbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:46:24.502976 IP (tos 0x0, ttl 64, id 24281, offset 0, flags [DF], proto TCP (6), length 60)
    PROXMOX_IP.706 > NAS_IP.111: Flags [S], cksum 0x6492 (incorrect -> 0x01fa), seq 1867953190, win 64240, options [mss 1460,sackOK,TS val 2082620385 ecr 0
,nop,wscale 7], length 0
17:46:24.503127 IP (tos 0x0, ttl 64, id 24282, offset 0, flags [DF], proto TCP (6), length 52)
    PROXMOX_IP.706 > NAS_IP.111: Flags [.], cksum 0x648a (incorrect -> 0x30fa), seq 1867953191, ack 13616751, win 502, options [nop,nop,TS val 2082620385 e
cr 799015384], length 0
17:46:24.503298 IP (tos 0x0, ttl 64, id 24283, offset 0, flags [DF], proto TCP (6), length 144)
    PROXMOX_IP.706 > NAS_IP.111: Flags [P.], cksum 0x64e6 (incorrect -> 0x89ab), seq 0:92, ack 1, win 502, options [nop,nop,TS val 2082620385 ecr 799015384
], length 92
17:46:24.503865 IP (tos 0x0, ttl 64, id 24284, offset 0, flags [DF], proto TCP (6), length 52)
    PROXMOX_IP.706 > NAS_IP.111: Flags [.], cksum 0x648a (incorrect -> 0x3069), seq 92, ack 53, win 502, options [nop,nop,TS val 2082620385 ecr 799015385],
length 0
17:46:24.503945 IP (tos 0x0, ttl 64, id 24285, offset 0, flags [DF], proto TCP (6), length 52)
    PROXMOX_IP.706 > NAS_IP.111: Flags [F.], cksum 0x648a (incorrect -> 0x3068), seq 92, ack 53, win 502, options [nop,nop,TS val 2082620385 ecr 799015385]
, length 0
17:46:24.504046 IP (tos 0x0, ttl 64, id 24286, offset 0, flags [DF], proto TCP (6), length 52)
    PROXMOX_IP.706 > NAS_IP.111: Flags [.], cksum 0x648a (incorrect -> 0x3066), seq 93, ack 54, win 502, options [nop,nop,TS val 2082620386 ecr 799015385],
length 0
^C
6 packets captured
6 packets received by filter
0 packets dropped

On NAS I cannot use tcpdump because no tool is installed to analyse incoming packages.

Port 111 is open on NAS:

root@NODEXXX:~# nc -z -v NAS_IP 111

nc: NAS_IP (NAS_IP) 111 [sunrpc] open

Hope that helps?

 

[Ninos]

Hey there,

found out, that it's also not working on a clean Debian 10 installation. Seems to be something with the package itself (or something more general). I'll ask in Debian forum, but I'm still hoping for help

 

[oguz]

found out, that it's also not working on a clean Debian 10 installation. Seems to be something with the package itself (or something more general). I'll ask in Debian forum, but I'm still hoping for help

it's still working here. i think it has to be setup-specific.

let us know about the discussion in debian forum

 

[rspille]

From my other post:
After struggeling some time I found out, that the firewall on my synology nas blocked the nsf mount. So I deactivated and reactivated nfs and the firewall showed a dialog, that it updated the rules – now I'm able to see the nfs mounts on proxmox again…