I'm seeing a very odd and difficult to troubleshoot disk share performance issue in case anyone has thoughts.
I have a 4 node PVE cluster, server by a few nodes of FreeNAS with NFS shares. One of the FreeNAS boxes is new, with SSD storage, and as part of bringing it online, I have rearranged the cluser network connections on my Unifi XG switch. Each node has a 10G connection on VLAN100 (192.168.100.0/24) for data and a 10G on VLAN101 (192.168.101.0/24) for SAN/FreeNAS connections. All nodes use the same NFS share for Image files, but 1 node is seeing extremely slow disk access to the share. Iperf3 tests to the share come back at 10G speeds.
I've tried using different ports on the switch, checking that the interfaces are really connected to the right networks, I'm at a complete loss as to what to do. It has to be related to the change (and I can't just 'put it back', I moved enough ports I don't know which ones it was on previous, but the other 3 nodes are fine, and there are no new ports in the configuration, I simply set it up so that the nodes all were in one colum on the switch, with VLAN100 on top for every node and VLAN101 on bottom). I've tried moving the node from SPF+ to native copper ports on the switch with no apparent effect, and I've rebooted the Node multiple times.
Here is some data.
Iperf3 results on FreeNAS share from Node 1 (which is having the slow reads).
Disk performance test from VM running on Node 1
similar test on Node 2
Some stats from Node 1
Same stats from Node 2
I have a 4 node PVE cluster, server by a few nodes of FreeNAS with NFS shares. One of the FreeNAS boxes is new, with SSD storage, and as part of bringing it online, I have rearranged the cluser network connections on my Unifi XG switch. Each node has a 10G connection on VLAN100 (192.168.100.0/24) for data and a 10G on VLAN101 (192.168.101.0/24) for SAN/FreeNAS connections. All nodes use the same NFS share for Image files, but 1 node is seeing extremely slow disk access to the share. Iperf3 tests to the share come back at 10G speeds.
I've tried using different ports on the switch, checking that the interfaces are really connected to the right networks, I'm at a complete loss as to what to do. It has to be related to the change (and I can't just 'put it back', I moved enough ports I don't know which ones it was on previous, but the other 3 nodes are fine, and there are no new ports in the configuration, I simply set it up so that the nodes all were in one colum on the switch, with VLAN100 on top for every node and VLAN101 on bottom). I've tried moving the node from SPF+ to native copper ports on the switch with no apparent effect, and I've rebooted the Node multiple times.
Here is some data.
Iperf3 results on FreeNAS share from Node 1 (which is having the slow reads).
Disk performance test from VM running on Node 1
Code:
bferrell@ntp:~$ sudo hdparm -Tt /dev/sda
[sudo] password for bferrell:
/dev/sda:
Timing cached reads: 2 MB in 32.17 seconds = 63.66 kB/sec
Timing buffered disk reads: 2 MB in 28.34 seconds = 72.26 kB/sec
bferrell@ntp:~$similar test on Node 2
Some stats from Node 1
Same stats from Node 2
