SR-IOV passthrough or Vmbr for traffic between VMs?

[nva]

I'm planning for a new server and wondering whether i should opt for a NIC capable of SR-IOV.

AFAIK, SR-IOV has obvious advantage of not causing CPU overhead if i give each VM a VF, but VM traffic still need to go all the way up to switch before coming back to same physical NIC.

For virtual bridge, all traffic are internal within physical host so that might be better for stability? Correct me if i'm wrong

Latency is deciding factor for me and i don't know what approach give me better latency. I will be using iSCSI storage VM and a couple of other VMs.

Thanks for help!

 

[mrpops2ko]

I'm planning for a new server and wondering whether i should opt for a NIC capable of SR-IOV.

AFAIK, SR-IOV has obvious advantage of not causing CPU overhead if i give each VM a VF, but VM traffic still need to go all the way up to switch before coming back to same physical NIC.

For virtual bridge, all traffic are internal within physical host so that might be better for stability? Correct me if i'm wrong

Latency is deciding factor for me and i don't know what approach give me better latency. I will be using iSCSI storage VM and a couple of other VMs.

Thanks for help!

Yes you should, and no you dont have to configure it as such that traffic has to flow across the bridge.

You bridge the PF for management only and then assign the VMs with direct SR-IOV capable nics. here is mine.

the two enp are physical ports, one of them is WAN and one is LAN for me. The WAN one is passed to pfsense and the LAN goes to my switch.

I am accessing the proxmox management through the bridge on the PF but thats only for the management. If I attached a VM to the bridge then they wouldn't have an exit path because the traffic has no exit (no WAN).

The VF's that are created from the LAN PF's do though, as the other VM LAN VFs are able to talk to the pfsense LAN VF and then the PFsense LAN VF can talk to the pfsense WAN PF

this is probably one of the best setups if you desire the lowest possible latency outside of going full DPDK / TNSR / VFF [something i need to pick up at some point]

 

[nva]

Yes you should, and no you dont have to configure it as such that traffic has to flow across the bridge.

You bridge the PF for management only and then assign the VMs with direct SR-IOV capable nics. here is mine.

the two enp are physical ports, one of them is WAN and one is LAN for me. The WAN one is passed to pfsense and the LAN goes to my switch.

I am accessing the proxmox management through the bridge on the PF but thats only for the management. If I attached a VM to the bridge then they wouldn't have an exit path because the traffic has no exit (no WAN).

The VF's that are created from the LAN PF's do though, as the other VM LAN VFs are able to talk to the pfsense LAN VF and then the PFsense LAN VF can talk to the pfsense WAN PF

this is probably one of the best setups if you desire the lowest possible latency outside of going full DPDK / TNSR / VFF [something i need to pick up at some point]

I have same problem with bridges once SR-IOV enabled. VM attached to bridge won't communicate with anything except other VMs on same bridge. I tried moving bridge port for vmbr from PF to another VF it still doesn't work.

Kinda wish that proxmox support DPDK for OVS.