Stability issues with PVE 8.1 and 10gb network shared storage (iScsi and NFS)

[cts1085]

I have an existing SAN+Hypervisor with Freenas + ESXi that has been running for years. I am trying to leverage the same SAN with a 2node PVE cluster. The system details are attached.
What I am seeing is that when I initially boot the nodes - all is good, the shared storage is seen, I can start/stop vm's, run backups, even migrate between nodes. but after a few minutes (and it does not seem to matter if the vm's are running or not), I see one or both nodes move to an unknown status, all of the shared data (iScsi+NFS) shows with question marks, after a few more minutes the NFS shares return but the iScisi+lvm does not even though the nodes still see the targets (per lsscsi and iscsiadm). I have tried restarting services but only a full reboot will allow access to the LVM disks/partitions.

I am sure I misconfigured something and just would like some guidance on how to troubleshoot.
I have reinstalled both nodes several times trying slightly different configurations but I feel I have been stumbling around in the dark.

I really hope someone sees this and let's me know that either I missed something or did something not advised.

At this point I want stability over performance.

Thank you!

 

[bbgeek17]

The first thing that jumps out of the log are iSCSI timeouts. Thats an indication of network problems/misconfiguration.

Your setup has more complexity than average one here and so it requires a bit more troubleshooting from your side. Specifically, isolating parts of the configuration taking them down to bare basics and building back up.

I'd double check that MTU is the same across the iSCSI/NFS link. Its mentioned twice in your config dump and values are different.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[cts1085]

Thank you for this direction.
I have confirmed that on all interfaces I am using MTU 9000, and the switch is setup to use jumbo frames with 9216 as the size.

I tried using tracepath -n 192.168.160.40 and tracepath -n 192.168.152.40 and it indicates the MTU is consistent at 9000
root@prox1:~# tracepath -n 192.168.160.40
1?: [LOCALHOST] pmtu 9000
1: 192.168.160.40 0.255ms reached
1: 192.168.160.40 0.180ms reached
Resume: pmtu 9000 hops 1 back 1
root@prox1:~# tracepath -n 192.168.152.40
1?: [LOCALHOST] pmtu 9000
1: 192.168.152.40 0.482ms reached
1: 192.168.152.40 0.252ms reached
Resume: pmtu 9000 hops 1 back 1

and just for fun....
root@prox2:~# tracepath -4 -n -l 9000 192.168.160.40
1: 192.168.160.40 0.472ms reached
Resume: pmtu 9000 hops 1 back 1
root@prox2:~# tracepath -4 -n -l 9000 192.168.152.40
1: 192.168.152.40 0.523ms reached
Resume: pmtu 9000 hops 1 back 1


Is there something else I should use to test MTU?

 

[bbgeek17]

How about ping? ping -s 8900 -M do

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[cts1085]

Ok - now it is interesting...
Pinging the san - no joy
pinging the other node - works. - Same results from both nodes.

root@prox2:~# ping -s 8900 -M do 192.168.160.40
PING 192.168.160.40 (192.168.160.40) 8900(8928) bytes of data.
^C
--- 192.168.160.40 ping statistics ---
23 packets transmitted, 0 received, 100% packet loss, time 22532ms

root@prox2:~# ping -s 8900 -M do 192.168.160.38
PING 192.168.160.38 (192.168.160.38) 8900(8928) bytes of data.
8908 bytes from 192.168.160.38: icmp_seq=1 ttl=64 time=0.477 ms
8908 bytes from 192.168.160.38: icmp_seq=2 ttl=64 time=0.660 ms
8908 bytes from 192.168.160.38: icmp_seq=3 ttl=64 time=0.591 ms
8908 bytes from 192.168.160.38: icmp_seq=4 ttl=64 time=0.607 ms
8908 bytes from 192.168.160.38: icmp_seq=5 ttl=64 time=0.421 ms
8908 bytes from 192.168.160.38: icmp_seq=6 ttl=64 time=0.484 ms
8908 bytes from 192.168.160.38: icmp_seq=7 ttl=64 time=0.607 ms

However, I started a VM on the esxi box (192.168.160.39) and it was good...
cts@dualnic:~$ ping -s 8600 -M do 192.168.160.40
PING 192.168.160.40 (192.168.160.40) 8600(8628) bytes of data.
8608 bytes from 192.168.160.40: icmp_seq=1 ttl=64 time=0.265 ms
8608 bytes from 192.168.160.40: icmp_seq=2 ttl=64 time=0.358 ms
8608 bytes from 192.168.160.40: icmp_seq=3 ttl=64 time=0.215 ms
8608 bytes from 192.168.160.40: icmp_seq=4 ttl=64 time=0.252 ms
8608 bytes from 192.168.160.40: icmp_seq=5 ttl=64 time=0.280 ms
8608 bytes from 192.168.160.40: icmp_seq=6 ttl=64 time=0.300 ms

This seems to point to an issue on the PVE nodes - correct?

NOTE: I rebooted the nodes and the ping now works..
root@prox2:~# ping -s 8900 -M do 192.168.160.40
PING 192.168.160.40 (192.168.160.40) 8900(8928) bytes of data.
8908 bytes from 192.168.160.40: icmp_seq=1 ttl=64 time=0.344 ms
8908 bytes from 192.168.160.40: icmp_seq=2 ttl=64 time=0.425 ms
8908 bytes from 192.168.160.40: icmp_seq=3 ttl=64 time=0.373 ms
8908 bytes from 192.168.160.40: icmp_seq=4 ttl=64 time=0.390 ms
8908 bytes from 192.168.160.40: icmp_seq=5 ttl=64 time=0.402 ms
8908 bytes from 192.168.160.40: icmp_seq=6 ttl=64 time=0.466 ms
8908 bytes from 192.168.160.40: icmp_seq=7 ttl=64 time=0.493 ms
8908 bytes from 192.168.160.40: icmp_seq=8 ttl=64 time=0.458 ms
8908 bytes from 192.168.160.40: icmp_seq=9 ttl=64 time=0.429 ms


So something is happening on the nodes that is causing the communication channel to go wonky...
Thoughts?

 

[bbgeek17]

PVE is based on Debian Linux with Ubuntu Kernel. Yes, PVE should be treated as an appliance in general because it comes as a single package.
However, the issue you are showing is at the very basic level of TCP/IP connectivity where no PVE functionality is involved.

Because of the relative complexity of your setup I can only help so much. May be your switch port is not set to proper MTU, may be there is some other more exoteric problem. Start reducing complexity, drop down MTU ping size until its working, swap the cables around, turn off other NICs/ports.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[cts1085]

sounds good - I will start with removing jumbo frames and then go from there.

What is concerning to me is that the other devices connected to the switch are operating fine. (san / freenas) I am curious if there is something in the debian/pve stack that is causing this.

But I will start with reducing complexity until it is stable - I will report back then.

Thank you!

 

[cts1085]

ok - just to close the loop on this:
I modified the MTU back to 1500 for all interfaces and the system appears to be stable.
I don't know why changing the MTU to 9000 was causing dropouts but that is a different issue.

Thank you for your thoughtful direction!