I have 2 Mellanox ConnectX-3 cards running in my 3rd ProxMox 7.4-3 server and my UnRAID 6.11.5 server. Both have identical firmware. In this ProxMox server, I have installed TrueNAS Scale 22.12.2 as a VM as many people do for the NAS capabilities it provides and I have been relatively happy with this setup. HOWEVER.... I am unable to get these Mellanox cards to even so much as PING each other. This has to be user error but I just don't know where to troubleshoot beyond what I have done already.
To the best of my knowledge and a great deal of reading and inference, they are both in ETH mode as they are capable of ib/eth and I dont care about ib mode at all as i don't have the infrastructure for it.
In ProxMox and UnRAID, I have these cards seen and their status is UP.. Green blinky lights on the cards and everything as they do have a direct connection (no switch / no gateway) to each other. From ProxMox, I can PING the local Mellanox NIC, but not the card it is directly connected to on the UnRAID server . The same is true from UnRAID going the opposite direction. This is beyond frustrating because I dont know where to look for log entries for potential issues beyond dmesg, syslog, and kern.log.
How can these cards be connected directly, have the same MTU, the same subnet/mask. Also physically show they are connected and I cant PING either of them from the other???
In ProxMox, I have done the following (PVE3):
'enp1so' is the only port I currently care about. 'enp6s0' is my regular NIC that works fine.
For whatever reason, this setup is not correct and I would love to know where I have gone wrong.
The TrueNAS server does see the added NIC after I add the bridged NIC to the TrueNAS VM in ProxMox as seen below (ens19):
TrueNAS is getting ahead of the situation currently since I cant even PING between these cards via ProxMox/UnRAID, muchless TrueNAS/UnRAID. I have burned up a fair amount of time to warrant a post imho. I do appreciate any help I can get as this should not be that difficult I wouldn't think.
To the best of my knowledge and a great deal of reading and inference, they are both in ETH mode as they are capable of ib/eth and I dont care about ib mode at all as i don't have the infrastructure for it.
In ProxMox and UnRAID, I have these cards seen and their status is UP.. Green blinky lights on the cards and everything as they do have a direct connection (no switch / no gateway) to each other. From ProxMox, I can PING the local Mellanox NIC, but not the card it is directly connected to on the UnRAID server . The same is true from UnRAID going the opposite direction. This is beyond frustrating because I dont know where to look for log entries for potential issues beyond dmesg, syslog, and kern.log.
How can these cards be connected directly, have the same MTU, the same subnet/mask. Also physically show they are connected and I cant PING either of them from the other???
In ProxMox, I have done the following (PVE3):
'enp1so' is the only port I currently care about. 'enp6s0' is my regular NIC that works fine.
For whatever reason, this setup is not correct and I would love to know where I have gone wrong.
The TrueNAS server does see the added NIC after I add the bridged NIC to the TrueNAS VM in ProxMox as seen below (ens19):
TrueNAS is getting ahead of the situation currently since I cant even PING between these cards via ProxMox/UnRAID, muchless TrueNAS/UnRAID. I have burned up a fair amount of time to warrant a post imho. I do appreciate any help I can get as this should not be that difficult I wouldn't think.
Last edited:
