First I tried this, but I found out later they weren't persistent:
echo eth >/sys/bus/pci/devices/0000:XX:00.0/mlx4_port1
echo eth >/sys/bus/pci/devices/0000:XX:00.0/mlx4_port2
It's not just not persistent through reboots, or ifdowns.
Basically for drivers on linux, you install the Mellanox...