How to configure separate NFS network

[ihr]

Hello,

I've a test environment at home with 5 PCs. One of them is running a NAS server and the other 4 are running on Proxmox. The VMs are all stored in the NAS server and everything run on NFS. All PCs are on the same network 192.168.1.X/24 and everything works fine.

I decided to speed-up a bit the environment by splitting the network in two separate networks. One of the networks will have only NFS traffic between the NAS and the 4 PCs running proxmox. That network will not have internet connection, no router etc. It is just 5 static IP addresses connected to a 10Gb switch.

I've configured the hardware side and the 192.168.1.X/24 network works fine. Then I started to configure the new 192.168.5.X/24 network for the NFS traffic and here is where I start seen things that don't seem to be right.

Each server has 2 network cards. One is 10Gb (the new one).

Server 1, works OK.


From this server I can ping 192.168.5.5 (the NAS server in the new network)


Server 2, doesn't work:

From this server, I can ping 192.168.5.5, but I can't ping back from 192.168.5.5 to 192.168.5.20

Server 3, doesn't work:

Same as server 2, I can ping 192.168.5.5 but I can't ping back from 192.168.5.5 to 192.168.5.30

Server 4, doesn't work either:

Same as server 2 and server 3.

I'm using an unmanaged switch from Netgear XS508M

What can I be doing wrong?

Thanks

 

[ihr]

I found the source of the PING issue. Servers 2,3 and 4 has firewall enabled. Server 1 had no firewall enabled.

Now I'm digging down to the second phase: How can I change the NFS shares ip address? the GUI does not allow me to do it directly so I'll have to find a configuration file...

The NAS server is not configured for any specific ip address so it should be sharing in all interfaces.

 

[ihr]

I managed to change /etc/pve/storage.cfg and update just the server ip address. Then I reboot one server and that didn't work at all. The error was at the NAS server level.

TASK ERROR: mount error: mount.nfs: access denied by server while mounting 192.168.5.5:/mnt/zpool/machines

so I need to do more on the NAS side

 

[ihr]

Just for the records. I managed to configure the NAS server properly and now everything works.

I just wanted to keep a note for others to know (and comment)

I purchased 5 network cards 10G, one with SFP connection.
I purchased 1 Netgear unmanaged switch model XS508M-100EUS to create a subnet just for the storage.

During today's tests and installation process I discovered an issue with the switch. The switch does not resist a power on-off-on cycle. You need to unplug all connected devices. Restart the switch and plug the devices one by one.

I've already opened a support case with Netgear because I need my devices to resist a power on-off-on event and this switch fails if that happens. I've not found in the market a similar product (10G unmanaged or managed switch with 8 ports for small business level) the price I've paid is around 500€ so it is really expensive!

I hope Netgear can move forward and fix/replace my unit.

I keep all this process here in case others want to run a similar configuration.

 

[Whocare007]

I am curently in the process of doing exactly what you did. 10gb backbone network through a 10gb switch not connected to internet (10.10.10.X).

My NFS share are working on my standard network (192.168.2.X) but not on my new network. I added my ip adresses to my NFS share on trueNAS and still have an error 500 (disk is not online)

 

[louie1961]

Here's how I do it:

root@pve-01:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp10s0 inet manual

iface enp10s0d1 inet manual

auto vmbr0
iface vmbr0 inet static
        bridge-ports enp10s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-100

auto vmbr0.100
iface vmbr0.100 inet static
        address 192.168.10.2/24
        gateway 192.168.10.1

auto vmbr1
iface vmbr1 inet static
        address 192.168.3.4/24
        bridge-ports enp10s0d1
        bridge-stp off
        bridge-fd 0


source /etc/network/interfaces.d/*

root@pve-01:~#

 

[Whocare007]

found my problem.

My plan was to have 10gb backbone on only 2 node, with the third being a mini pc, mostly for quorum, nothing runing on it. Once I added that node to the backbone network, everything went fine. I was not able to add my NFS share to my 2 main node only, even though they were selected.