At $Dayjob I have managed to with PCI passthrough pass in devices(GPU and IB) into a VM. But for some reason an all_reduce is still slow (1/10 with 2 full nodes in BM). First attempted layout was something like: upstream_port ^ downstream_port ^ ^ ^ ^ ^ GPU1 GPU2 GPU3 .. GPU8 IB1 IB2 IB3 …

Look no further than this link by Demartek: http://www.demartek.com/Demartek_Interface_Comparison.html It has a good picture that shows the difference between the "mini LC", mSFP or whatever it's called and the standard LC connector.