Wow, this actually went sooo smoothly. I am not sure why I was struggling soo badly.
So, I actually have no ethernet ports, only infiniband, on my cards. As such, I assume switching to ethernet mode is not required between two of these cards (on seperate physical machines, right?)
In fact...