Mutil-NIC Proxmox w/ TrueNAS iSCSI

[NateNate]

Hello! Before I posted this I did a search to see if someone else ran into it, and couldn't find one, so here we go.

My setup: 3 Node Proxmox Cluster. Each Node has 2 NICs (2.5gb and 10gb). 2.5 back to the main network, and 10gb directly to my TrueNAS server where the storage resides.
My TrueNAS server has 4 NICs, a single Management port, and three 10gb connections.
(10.10.11.x, 10.10.12.x, and 10.10.13.x)

Each Proxmox Node connects directly to TrueNAS over the 10GB link. No Switch in between.
I, for reasons beyond me, setup iSCSI connections over the Management port (facepalm) and wondered why my VM's were being slow.

Here is the problem:
If I try to connect over the 10GB line, it works as long as I point the specific node to the specific IP of the NAS attached to it. But this means HA is broken.

What I've tried:
Made the Linux Bridge on each of the nodes VLAN Aware.

Results:
Mixed bag. Node 1 can see its connection Plus .13, but not 12. Node 2 can see itself and .11 but not .13, and Node 3 can only see itself.

Goal is to have each node be able to see each of the other network addresses. Maybe its something in TrueNAS I need to enable, or its Proxmox, not sure.
Thanks for whatever direction this awesome community can provide!

 

[bbgeek17]

Hi @NateNate ,

You have not shared your exact configuration of either the networking or the PVE storage subsystem. If my understanding of your description is correct, you are specifying an IP address in storage.cfg:

- if you use the mgmt, the connections are established through it
- if you use one of the storage IPs, the two out of three nodes rightly cannot access it

Your configuration complexity is not something that built-in PVE iSCSI support is designed for. The easiest way to resolve this is to get a SAN network switch. Beyond that, you can skip the iSCSI storage part in PVE and configure iSCSI with iscsiadm directly on each host.

You can try modifying iscsid.conf on each host to set up priority during node discovery. Frankly, I am not sure it's possible.

My advice: If you can't get a switch, configure everything manually: https://kb.blockbridge.com/technote...nderstand-multipath-reporting-for-your-device

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[NateNate]

Was hoping to avoid getting a dedicated switch, but if I must I must.

Other documentation says its possible to do it without a switch using "VLAN Aware". and I have partial success doing that.
Each Node can only see 1 of the other 2 available, which doesn't make sense. Seems it would be all or nothing.

That being said, sorry I didn't provide the exact config, let grab it..