intel iommu causing performance drop in mlx4 ipoib

Hello,

I've been testing various infiniband cards for performance and one of them is the a ConnectX-3: Mellanox Technologies MT27500 Family [ConnectX-3].

I've observed a strange performance pathology with it when running ipoib and using a naive iperf test. My setup has multiple machines with a mix of qlogic/mellanox cards, connected via an QLogic 12300 switch. All of the nodes are running on 4x 10Gbps. When I run a performance test and the mellanox card is a server i.e it is receiving data I get very bad performance. By this I mean I cannot get more than 4 gigabits per second - very low. 'perf top' clearly shows that the culprit is intel_map_page which is being called form the receive path of the mellanox adapter:

84.26% 0.04% ksoftirqd/0 [kernel.kallsyms] [k] intel_map_page

            |
            --- intel_map_page
               |          
               |--98.38%-- ipoib_cm_alloc_rx_skb
               |          ipoib_cm_handle_rx_wc
               |          ipoib_poll
               |          net_rx_action
               |          __do_softirq
               |          run_ksoftirqd
               |          smpboot_thread_fn
               |          kthread
               |          ret_from_fork

When I disable intel_iommu support (By defualt the iommu is not turned on, just compiled, with this performance profile I have compiled out the code altogether) things look very differently:

         86.76%     0.16%  ksoftirqd/0  [kernel.kallsyms]  [k] ipoib_poll                                
            |
            --- ipoib_poll
                net_rx_action
                __do_softirq

Essentially the majority is spent in just receiving the packets and the sustained rate is 26Gbps. So the question why does compiling in (but not enabling intel_iommu=on kills performance) only on the receive side, e.g. if the machine which exhibits poor performance with the mlx card is a client, that is the mellanox driver is sending data the performance is not affected. So far the only workaround is to remove intel iommu support in the kernel altogether.

Regards,
Nikolay
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Are you *sure* it's disabled? Can you be more specific about where the time is spent? intel_map_page() doesn't really do much except calling in to __intel_map_single()... which should return fairly much immediately.

I'm working on improving the per-device DMA ops so that for passthrough devices you don't end up in the IOMMU code at all, but it really shouldn't be taking *that* long... unless you really are doing translation.

Note that even in the case where you're doing translation, there's code which I'm about to ask Linus to pull for 4.7, which will kill most of the performance hit of using the IOMMU.

David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation

Oops, turned out I had intel_iommu=on, though I'm sure some days ago I didn't and performance suffered. I now tried with this option removed and performance is again back to normal. All in all this driver + intel_iommu is a bad combination I guess. Anyway, sorry for the noise and thanks for the prompt reply.

Regards,
Nikolay
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I would be very interested to see your results with the latest code from git://git.infradead.org/intel-iommu.git (and with the IOMMU still enabled).

David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation

Would I be able to apply this cleanly on 4.4, since this is the version I'm currently using?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Yes, it all applies cleanly as long as you also apply commit da972fb13bc5a1baad450c11f9182e4cd0a091f6 first.

dwmw2

On Mon, May 16, 2016 at 1:39 PM, Nikolay Borisov <[email protected]> wrote:
[...]

add iommu=pt to your boot, this is 2009 material [1]

Or.

[1] https://lwn.net/Articles/329174/
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html