So I wanted to give an update, I finally installed a second card in another server but didn't setup SR-IOV, the card works, once I disabled SR-IOV on the other node that card was working too, so it seems to be some sort of a driver/firmware bug. I did attempt to install the mellanox driver but...