So I checked for buildin drivers, RoCev2 is not enabled, but the default driver performs for my setup better than the original Mellanox drivers. Latency is below 0.035ms and throughput roudabout 96GBps. I think that is ok. I have no issues with the virtual machines and ceph, no freezing, no delay, no slow ops.
And don't forget to run your cpu at maximum performance without powersaving, c-state, speedstep and so on to geht maximum performance of the mellanox cards.