XDP performance regression due to CONFIG_RETPOLINE Spectre V2
Subsystems (7)
Heads-up XDP performance nerds!
I got an unpleasant surprise when I updated my GCC compiler (to support the option -mindirect-branch=thunk-extern). My XDP redirect performance numbers when cut in half; from approx 13Mpps to 6Mpps (single CPU core). I've identified the issue, which is caused by kernel CONFIG_RETPOLINE, that only have effect when the GCC compiler have support. This is mitigation of Spectre variant 2 (CVE-2017-5715) related to indirect (function call) branches.
XDP_REDIRECT itself only have two primary (per packet) indirect function calls, ndo_xdp_xmit and invoking bpf_prog, plus any map_lookup_elem calls in the bpf_prog. I PoC implemented bulking for ndo_xdp_xmit, which helped, but not enough. The real root-cause is all the DMA API calls, which uses function pointers extensively.
Mitigation plan
---------------
Implement support for keeping the DMA mapping through the XDP return call, to remove RX map/unmap calls. Implement bulking for XDP ndo_xdp_xmit and XDP return frame API. Bulking allows to perform DMA bulking via scatter-gatter DMA calls, XDP TX need it for DMA map+unmap. The driver RX DMA-sync (to CPU) per packet calls are harder to mitigate (via bulk technique). Ask DMA maintainer for a common case direct call for swiotlb DMA sync call ;-)
Root-cause verification
-----------------------
I have verified that indirect DMA calls are the root-cause, by removing the DMA sync calls from the code (as they for swiotlb does nothing), and manually inlined the DMA map calls (basically calling phys_to_dma(dev, page_to_phys(page)) + offset). For my ixgbe test, performance "returned" to 11Mpps.
Perf reports
------------
It is not easy to diagnose via perf event tool. I'm coordinating with ACME to make it easier to pinpoint the hotspots. Lookout for symbols:__x86_indirect_thunk_r10, __indirect_thunk_start, __x86_indirect_thunk_rdx etc. Be aware that they might not be super high in perf top, but they stop CPU speculation. Thus, instead use perf-stat and see the negative effect of 'insn per cycle'.
Want to understand retpoline at ASM level read this:
https://support.google.com/faqs/answer/7625886
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Quoted 8 lines · Jesper Dangaard Brouer
---------------
Implement support for keeping the DMA mapping through the XDP return call, to remove RX map/unmap calls. Implement bulking for XDPndo_xdp_xmitand XDP return frame API. Bulking allows to perform DMA bulking via scatter-gatter DMA calls, XDP TX need it for DMA map+unmap. The driver RX DMA-sync (to CPU) per packet calls are harder to mitigate (via bulk technique). Ask DMA maintainer for a common case direct call for swiotlb DMA sync call ;-)
Why do you even end up in swiotlb code? Once you bounce buffer your performance is toast anyway..
Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Jesper Dangaard Brouer
---------------
Implement support for keeping the DMA mapping through the XDP return call, to remove RX map/unmap calls. Implement bulking for XDPndo_xdp_xmitand XDP return frame API. Bulking allows to perform DMA bulking via scatter-gatter DMA calls, XDP TX need it for DMA map+unmap. The driver RX DMA-sync (to CPU) per packet calls are harder to mitigate (via bulk technique). Ask DMA maintainer for a common case direct call for swiotlb DMA sync call ;-)
Quoted 2 lines
Why do you even end up in swiotlb code? Once you bounce buffer your performance is toast anyway..
I guess that is because x86 selects it as the default as soon as we have more than 4G memory. That should be solveable fairly easily with the per-device dma ops, though.
Quoted 11 lines · On Thu, 12 Apr 2018 16:56:53 +0200 Christoph Hellwig <[email protected]> wrote:
Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Jesper Dangaard Brouer
---------------
Implement support for keeping the DMA mapping through the XDP return call, to remove RX map/unmap calls. Implement bulking for XDPndo_xdp_xmitand XDP return frame API. Bulking allows to perform DMA bulking via scatter-gatter DMA calls, XDP TX need it for DMA map+unmap. The driver RX DMA-sync (to CPU) per packet calls are harder to mitigate (via bulk technique). Ask DMA maintainer for a common case direct call for swiotlb DMA sync call ;-)
Quoted 5 lines
Quoted 2 lines
Why do you even end up in swiotlb code? Once you bounce buffer your performance is toast anyway..
I guess that is because x86 selects it as the default as soon as we have more than 4G memory.
I were also confused why I ended up using SWIOTLB (SoftWare IO-TLB), that might explain it. And I'm not hitting the bounce-buffer case.
How do I control which DMA engine I use? (So, I can play a little)
Quoted 2 lines
That should be solveable fairly easily with the per-device dma ops, though.
I didn't understand this part.
I wanted to ask your opinion, on a hackish idea I have... Which is howto detect, if I can reuse the RX-DMA map address, for TX-DMA operation on another device (still/only calling sync_single_for_device).
With XDP_REDIRECT we are redirecting between net_device's. Usually we keep the RX-DMA mapping as we recycle the page. On the redirect to TX-device (via ndo_xdp_xmit) we do a new DMA map+unmap for TX. The question is how to avoid this mapping(?). In some cases, with some DMA engines (or lack of) I guess the DMA address is actually the same as the RX-DMA mapping dma_addr_t already known, right? For those cases, would it be possible to just (re)use that address for TX?
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Quoted 7 lines · Jesper Dangaard Brouer
Quoted 2 lines
I guess that is because x86 selects it as the default as soon as we have more than 4G memory.
I were also confused why I ended up using SWIOTLB (SoftWare IO-TLB), that might explain it. And I'm not hitting the bounce-buffer case.
How do I control which DMA engine I use? (So, I can play a little)
At the lowest level you control it by:
(1) setting the dma_ops pointer in struct device
(2) if that is NULL by choosing what is returned from
get_arch_dma_ops()
Quoted 6 lines
Quoted 4 lines
That should be solveable fairly easily with the per-device dma ops, though.
I didn't understand this part.
What I mean with that is that we can start out setting dma_ops to dma_direct_ops for everyone on x86 when we start out (that is assuming we don't have an iommu), and only switching to swiotlb_dma_ops when actually required by either a dma_mask that can't address all memory, or some other special cases like SEV or broken bridges.
Quoted 11 lines
I wanted to ask your opinion, on a hackish idea I have... Which is howto detect, if I can reuse the RX-DMA map address, for TX-DMA operation on another device (still/only calling
sync_single_for_device).With
XDP_REDIRECTwe are redirecting between net_device's. Usually we keep the RX-DMA mapping as we recycle the page. On the redirect to TX-device (viando_xdp_xmit) we do a new DMA map+unmap for TX. The question is how to avoid this mapping(?). In some cases, with some DMA engines (or lack of) I guess the DMA address is actually the same as the RX-DMA mappingdma_addr_talready known, right? For those cases, would it be possible to just (re)use that address for TX?
You can't in any sensible way without breaking a lot of abstractions. For dma direct ops that mapping will be the same unless the devices have different dma_offsets in their struct device, or the architecture overrides phys_to_dma entirely, in which case all bets are off. If you have an iommu it depends on which devices are behind the same iommu.
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Jesper Dangaard Brouer
---------------
Implement support for keeping the DMA mapping through the XDP return call, to remove RX map/unmap calls. Implement bulking for XDPndo_xdp_xmitand XDP return frame API. Bulking allows to perform DMA bulking via scatter-gatter DMA calls, XDP TX need it for DMA map+unmap. The driver RX DMA-sync (to CPU) per packet calls are harder to mitigate (via bulk technique). Ask DMA maintainer for a common case direct call for swiotlb DMA sync call ;-)
Quoted 6 lines
Quoted 2 lines
Why do you even end up in swiotlb code? Once you bounce buffer your performance is toast anyway..
I guess that is because x86 selects it as the default as soon as we have more than 4G memory. That should be solveable fairly easily with the per-device dma ops, though.\
I guess there is nothing we need to do!
On x86, in case of no intel iommu or iommu is disabled, you end up in swiotlb for DMA API calls when system has 4G memory. However, AFAICT, for 64bit DMA capable devices swiotlb DMA APIs do not use bounce buffer until and unless you have swiotlb=force specified in kernel commandline.
e.g. here is the snip:dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
unsigned long offset, size_t size, enum dma_data_direction dir, unsigned long attrs)
{
phys_addr_t map, phys = page_to_phys(page) + offset;
dma_addr_t dev_addr = phys_to_dma(dev, phys);
BUG_ON(dir == DMA_NONE);
/*
* If the address happens to be in the device's DMA window,
* we can safely return the device addr and not worry about bounce
* buffering it.
*/
if (dma_capable(dev, dev_addr, size) && swiotlb_force !=
SWIOTLB_FORCE)
return dev_addr;-Tushar
Quoted 7 lines · Tushar Dave
I guess there is nothing we need to do!
On x86, in case of no intel iommu or iommu is disabled, you end up in swiotlb for DMA API calls when system has 4G memory. However, AFAICT, for 64bit DMA capable devices swiotlb DMA APIs do not use bounce buffer until and unless you have swiotlb=force specified in kernel commandline.
Sure. But that means very sync_*_to_device and sync_*_to_cpu now involves an indirect call to do exactly nothing, which in the workload Jesper is looking at is causing a huge performance degradation due to retpolines.
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Tushar Dave
I guess there is nothing we need to do!
On x86, in case of no intel iommu or iommu is disabled, you end up in swiotlb for DMA API calls when system has 4G memory. However, AFAICT, for 64bit DMA capable devices swiotlb DMA APIs do not use bounce buffer until and unless you have swiotlb=force specified in kernel commandline.
Quoted 4 lines
Sure. But that means very sync_*_to_device and sync_*_to_cpu now involves an indirect call to do exactly nothing, which in the workload Jesper is looking at is causing a huge performance degradation due to retpolines.
We should look at using the
if (dma_ops == swiotlb_dma_ops)
swiotlb_map_page()
else
dma_ops->map_page()
trick for this. Perhaps with alternatives so that when an Intel or AMD IOMMU is detected, it's *that* which is checked for as the special case.
Quoted 10 lines · On Sat, 14 Apr 2018 21:29:26 +0200 David Woodhouse <[email protected]> wrote:
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Tushar Dave
I guess there is nothing we need to do!
On x86, in case of no intel iommu or iommu is disabled, you end up in swiotlb for DMA API calls when system has 4G memory. However, AFAICT, for 64bit DMA capable devices swiotlb DMA APIs do not use bounce buffer until and unless you have swiotlb=force specified in kernel commandline.
Quoted 4 lines
Quoted 4 lines
Sure. But that means very sync_*_to_device and sync_*_to_cpu now involves an indirect call to do exactly nothing, which in the workload Jesper is looking at is causing a huge performance degradation due to retpolines.
Yes, exactly.
Quoted 11 lines
We should look at using the
if (
dma_ops==swiotlb_dma_ops)
swiotlb_map_page()
else
dma_ops->map_page()trick for this. Perhaps with alternatives so that when an Intel or AMD IOMMU is detected, it's *that* which is checked for as the special case.
Yes, this trick is basically what I'm asking for :-)
It did sound like Hellwig wanted to first avoid/fix that x86 end-up defaulting to swiotlb. Thus, we just have to do the same trick with the new default fall-through dma_ops.
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Can you try the following hack which avoids indirect calls entirely for the fast path direct mapping case?
---
Quoted 1 line
From
b256a008c1b305e6a1c2afe7c004c54ad2e96d4bMon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Mon, 16 Apr 2018 14:18:14 +0200
Subject: dma-mapping: bypass dma_ops for direct mappings
Reportedly the retpoline mitigation for spectre causes huge penalties for indirect function calls. This hack bypasses the dma_ops mechanism for simple direct mappings.
--- include/linux/device.h | 1 + include/linux/dma-mapping.h | 53 +++++++++++++++++++++++++++---------- lib/dma-direct.c | 4 +-- 3 files changed, 42 insertions(+), 16 deletions(-)
include/linux/device.h +1 -0
@@ -990,6 +990,7 @@ struct device { bool offline_disabled:1; bool offline:1; bool of_node_reused:1; + bool is_dma_direct:1; };static inline struct device *kobj_to_dev (struct kobject *kobj)
include/linux/dma-mapping.h +39 -14
143 lines not on this page · raw message
lib/dma-direct.c +2 -2
@@ -120,7 +120,7 @@ void dma_direct_free(struct device *dev, size_t size, void *cpu_addr,free_pages ((unsigned long )cpu_addr, page_order); } -static dma_addr_t dma_direct_map_page (struct device *dev,struct page *page, +dma_addr_t dma_direct_map_page (struct device *dev,struct page *page,unsigned long offset,size_t size,enum dma_data_direction dir,unsigned long attrs) {
@@ -131,7 +131,7 @@ static dma_addr_t dma_direct_map_page(struct device *dev, struct page *page,return dma_addr; } -static int dma_direct_map_sg (struct device *dev,struct scatterlist *sgl, +int dma_direct_map_sg (struct device *dev,struct scatterlist *sgl,int nents,enum dma_data_direction dir,unsigned long attrs) {int i;
Signature · 1 line
2.17.0
Quoted 57 lines · Christoph Hellwig
Can you try the following hack which avoids indirect calls entirely for the fast path direct mapping case?
---
Fromb256a008c1b305e6a1c2afe7c004c54ad2e96d4bMon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Mon, 16 Apr 2018 14:18:14 +0200
Subject: dma-mapping: bypassdma_opsfor direct mappingsReportedly the retpoline mitigation for spectre causes huge penalties for indirect function calls. This hack bypasses the
dma_opsmechanism for simple direct mappings.Signed-off-by Christoph Hellwig <[email protected]>--- include/linux/device.h | 1 + include/linux/dma-mapping.h | 53 +++++++++++++++++++++++++++---------- lib/dma-direct.c | 4 +-- 3 files changed, 42 insertions(+), 16 deletions(-)include/linux/device.h +1
-0@@ -990,6 +990,7 @@ struct device { bool offline_disabled:1; bool offline:1; bool of_node_reused:1; + bool is_dma_direct:1; };static inline struct device *kobj_to_dev (struct kobject *kobj)include/linux/dma-mapping.h +7
-3@@ -223,6 +223,13 @@ static inline const struct dma_map_ops *get_dma_ops(struct device *dev) }#endif +/* do not use directly! */ +dma_addr_t dma_direct_map_page (struct device *dev,struct page *page, +unsigned long offset,size_t size,enum dma_data_direction dir, +unsigned long attrs); +int dma_direct_map_sg (struct device *dev,struct scatterlist *sgl, +int nents,enum dma_data_direction dir,unsigned long attrs); + static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr, size_t size, enum dma_data_direction dir,@@ -232,9 +239,13 @@ static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr,dma_addr_t addr;BUG_ON (!valid_dma_direction (dir)); - addr = ops->map_page (dev,virt_to_page (ptr), -offset_in_page (ptr), size, - dir, attrs);
include/linux/dma-mapping.h static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr, · line 251
Quoted 10 lines
include/linux/dma-mapping.h +7
-0+if (dev->is_dma_direct) { + addr =dma_direct_map_page (dev,virt_to_page (ptr), +offset_in_page (ptr), size, dir, attrs); + }else { + addr = ops->map_page (dev,virt_to_page (ptr), +offset_in_page (ptr), size, dir, attrs); + }debug_dma_map_page (dev,virt_to_page (ptr),offset_in_page (ptr), size, dir, addr, true);
I'm not sure if I am really a fan of trying to solve this in this way. It seems like this is going to be optimizing the paths for one case at the detriment of others. Historically mapping and unmapping has always been expensive, especially in the case of IOMMU enabled environments. I would much rather see us focus on having swiotlb_dma_ops replaced with dma_direct_ops in the cases where the device can access all of physical memory.
include/linux/dma-mapping.h static inline void dma_unmap_single_attrs(struct device *dev, dma_addr_t addr, · line 263
Quoted 6 lines
include/linux/dma-mapping.h +1
-1@@ -249,7 +260,7 @@ static inline void dma_unmap_single_attrs(struct device *dev, dma_addr_t addr,const struct dma_map_ops *ops =get_dma_ops (dev);BUG_ON (!valid_dma_direction (dir)); -if (ops->unmap_page) +if (!dev->is_dma_direct && ops->unmap_page)
If I understand correctly this is only needed for the swiotlb case and not the dma_direct case. It would make much more sense to just overwrite the dev->dma_ops pointer with dma_direct_ops to address all of the sync and unmap cases.
Quoted 109 lines
include/linux/dma-mapping.h +18
-10ops->unmap_page (dev, addr, size, dir, attrs);debug_dma_unmap_page (dev, addr, size, dir, true); }@@ -266,7 +277,10 @@ static inline int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg,int ents;BUG_ON (!valid_dma_direction (dir)); - ents = ops->map_sg (dev, sg, nents, dir, attrs); +if (dev->is_dma_direct) + ents =dma_direct_map_sg (dev, sg, nents, dir, attrs); +else + ents = ops->map_sg (dev, sg, nents, dir, attrs);BUG_ON (ents <0 );debug_dma_map_sg (dev, sg, nents, ents, dir);@@ -281,7 +295,7 @@ static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sgBUG_ON (!valid_dma_direction (dir));debug_dma_unmap_sg (dev, sg, nents, dir); -if (ops->unmap_sg) +if (!dev->is_dma_direct && ops->unmap_sg) ops->unmap_sg (dev, sg, nents, dir, attrs); }@@ -295,7 +309,10 @@ static inline dma_addr_t dma_map_page_attrs(struct device *dev,dma_addr_t addr;BUG_ON (!valid_dma_direction (dir)); - addr = ops->map_page (dev, page, offset, size, dir, attrs); +if (dev->is_dma_direct) + addr =dma_direct_map_page (dev, page, offset, size, dir, attrs); +else + addr = ops->map_page (dev, page, offset, size, dir, attrs);debug_dma_map_page (dev, page, offset, size, dir, addr, false);return addr;@@ -309,7 +326,7 @@ static inline void dma_unmap_page_attrs(struct device *dev,const struct dma_map_ops *ops =get_dma_ops (dev);BUG_ON (!valid_dma_direction (dir)); -if (ops->unmap_page) +if (!dev->is_dma_direct && ops->unmap_page) ops->unmap_page (dev, addr, size, dir, attrs);debug_dma_unmap_page (dev, addr, size, dir, false); }@@ -356,7 +373,7 @@ static inline void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr,const struct dma_map_ops *ops =get_dma_ops (dev);BUG_ON (!valid_dma_direction (dir)); -if (ops->sync_single_for_cpu) +if (!dev->is_dma_direct && ops->sync_single_for_cpu) ops->sync_single_for_cpu (dev, addr, size, dir);debug_dma_sync_single_for_cpu (dev, addr, size, dir); }@@ -368,7 +385,7 @@ static inline void dma_sync_single_for_device(struct device *dev,const struct dma_map_ops *ops =get_dma_ops (dev);BUG_ON (!valid_dma_direction (dir)); -if (ops->sync_single_for_device) +if (!dev->is_dma_direct && ops->sync_single_for_device) ops->sync_single_for_device (dev, addr, size, dir);debug_dma_sync_single_for_device (dev, addr, size, dir); }@@ -382,7 +399,7 @@ static inline void dma_sync_single_range_for_cpu(struct device *dev,const struct dma_map_ops *ops =get_dma_ops (dev);BUG_ON (!valid_dma_direction (dir)); -if (ops->sync_single_for_cpu) +if (!dev->is_dma_direct && ops->sync_single_for_cpu) ops->sync_single_for_cpu (dev, addr + offset, size, dir);debug_dma_sync_single_range_for_cpu (dev, addr, offset, size, dir); }@@ -396,7 +413,7 @@ static inline void dma_sync_single_range_for_device(struct device *dev,const struct dma_map_ops *ops =get_dma_ops (dev);BUG_ON (!valid_dma_direction (dir)); -if (ops->sync_single_for_device) +if (!dev->is_dma_direct && ops->sync_single_for_device) ops->sync_single_for_device (dev, addr + offset, size, dir);debug_dma_sync_single_range_for_device (dev, addr, offset, size, dir); }@@ -408,7 +425,7 @@ dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg,const struct dma_map_ops *ops =get_dma_ops (dev);BUG_ON (!valid_dma_direction (dir)); -if (ops->sync_sg_for_cpu) +if (!dev->is_dma_direct && ops->sync_sg_for_cpu) ops->sync_sg_for_cpu (dev, sg, nelems, dir);debug_dma_sync_sg_for_cpu (dev, sg, nelems, dir); }@@ -420,7 +437,7 @@ dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg,const struct dma_map_ops *ops =get_dma_ops (dev);BUG_ON (!valid_dma_direction (dir)); -if (ops->sync_sg_for_device) +if (!dev->is_dma_direct && ops->sync_sg_for_device) ops->sync_sg_for_device (dev, sg, nelems, dir);debug_dma_sync_sg_for_device (dev, sg, nelems, dir);@@ -600,6 +617,8 @@ static inline int dma_supported(struct device *dev, u64 mask)return ops->dma_supported (dev, mask); } +extern const struct dma_map_ops swiotlb_dma_ops; +#ifndef HAVE_ARCH_DMA_SET_MASK static inline int dma_set_mask (struct device *dev,u64 mask) {
include/linux/dma-mapping.h static inline int dma_set_mask(struct device *dev, u64 mask) · line 636
Quoted 9 lines
include/linux/dma-mapping.h +6
-0@@ -609,6 +628,12 @@ static inline int dma_set_mask(struct device *dev, u64 mask)dma_check_mask (dev, mask); *dev->dma_mask = mask; +if (dev->dma_ops == &dma_direct_ops || + (dev->dma_ops == &swiotlb_dma_ops && + mask ==DMA_BIT_MASK (64 ))) + dev->is_dma_direct = true; +else + dev->is_dma_direct = false;
So I am not sure this will work on x86. If I am not mistaken I believe dev->dma_ops is normally not set and instead the default dma operations are pulled via get_arch_dma_ops which returns the global dma_ops pointer.
What you may want to consider as an alternative would be to look at modifying drivers that are using the swiotlb so that you could just overwrite the dev->dma_ops with the dma_direct_ops in the cases where the hardware can support accessing all of physical hardware and where we aren't forcing the use of the bounce buffers in the swiotlb.
Then for the above code you only have to worry about the map calls, and you could just do a check against the dma_direct_ops pointer instead of having to add a new flag.
Quoted 29 lines
include/linux/dma-mapping.h +0
-0return 0; }#endif lib/dma-direct.c +2
-3@@ -120,7 +120,7 @@ void dma_direct_free(struct device *dev, size_t size, void *cpu_addr,free_pages ((unsigned long )cpu_addr, page_order); } -static dma_addr_t dma_direct_map_page (struct device *dev,struct page *page, +dma_addr_t dma_direct_map_page (struct device *dev,struct page *page,unsigned long offset,size_t size,enum dma_data_direction dir,unsigned long attrs) {@@ -131,7 +131,7 @@ static dma_addr_t dma_direct_map_page(struct device *dev, struct page *page,return dma_addr; } -static int dma_direct_map_sg (struct device *dev,struct scatterlist *sgl, +int dma_direct_map_sg (struct device *dev,struct scatterlist *sgl,int nents,enum dma_data_direction dir,unsigned long attrs) {int i; --2.17.0
Quoted 7 lines
I'm not sure if I am really a fan of trying to solve this in this way. It seems like this is going to be optimizing the paths for one case at the detriment of others. Historically mapping and unmapping has always been expensive, especially in the case of IOMMU enabled environments. I would much rather see us focus on having
swiotlb_dma_opsreplaced withdma_direct_opsin the cases where the device can access all of physical memory.
I am definitively not a fan, but IFF indirect calls are such an overhead it makes sense to avoid it for the common and simple case. And the direct mapping is a common case present on just about every architecture, and it is a very simple fast path that just adds an offset to the physical address. So if we want to speed something up, this is it.
Quoted 7 lines
Review
Quoted 2 lines
(quoted diff) +1
-1- if (ops->unmap_page) + if (!dev->is_dma_direct && ops->unmap_page)If I understand correctly this is only needed for the swiotlb case and not the
dma_directcase. It would make much more sense to just overwrite thedev->dma_opspointer withdma_direct_opsto address all of the sync and unmap cases.
Yes.
Quoted 11 lines
include/linux/dma-mapping.h static inline int dma_set_mask(struct device *dev, u64 mask) · line 636
Quoted 6 lines
include/linux/dma-mapping.h +6
-0+if (dev->dma_ops == &dma_direct_ops || + (dev->dma_ops == &swiotlb_dma_ops && + mask ==DMA_BIT_MASK (64 ))) + dev->is_dma_direct = true; +else + dev->is_dma_direct = false;So I am not sure this will work on x86. If I am not mistaken I believe
dev->dma_opsis normally not set and instead the default dma operations are pulled viaget_arch_dma_opswhich returns the globaldma_opspointer.
True, for x86 we'd need to check get_arch_dma_ops().
Quoted 9 lines
What you may want to consider as an alternative would be to look at modifying drivers that are using the swiotlb so that you could just overwrite the
dev->dma_opswith thedma_direct_opsin the cases where the hardware can support accessing all of physical hardware and where we aren't forcing the use of the bounce buffers in the swiotlb.Then for the above code you only have to worry about the map calls, and you could just do a check against the
dma_direct_opspointer instead of having to add a new flag.
That would be the long term plan IFF we go down this route. For now I just wanted a quick hack for performance testing.
kbuild test robot
build botRe: dma-mapping: bypass dma_ops for direct mappings
In reply to Christoph HellwigI love your patch! Yet something to improve:
Hi Christoph,
I love your patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v4.17-rc1 next-20180416]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Christoph-Hellwig/dma-mapping-bypass-dma_ops-for-direct-mappings/20180416-230032
config: i386-defconfig (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0 reproduce:
# save the attached .config to linux build tree
make ARCH=i386 All errors (new ones prefixed by >>):
drivers/char/agp/intel-gtt.o: In function `intel_gmch_probe':
>> intel-gtt.c:(.text+0x11e4): undefined reference to `swiotlb_dma_ops' drivers/ata/ahci.o: In function `ahci_init_one': >> ahci.c:(.text+0x108d): undefined reference to `swiotlb_dma_ops' drivers/net/ethernet/broadcom/bnx2.o: In function `bnx2_init_one': >> bnx2.c:(.text+0x7fe7): undefined reference to `swiotlb_dma_ops' drivers/net/ethernet/broadcom/tg3.o: In function `tg3_init_one': >> tg3.c:(.text+0x13549): undefined reference to `swiotlb_dma_ops' drivers/net/ethernet/intel/e1000/e1000_main.o: In function `e1000_probe': >> e1000_main.c:(.text+0x49b3): undefined reference to `swiotlb_dma_ops'
drivers/net/ethernet/intel/e1000e/netdev.o:netdev.c:(.text+0xa65e): more undefined references to `swiotlb_dma_ops' follow
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Re: dma-mapping: bypass dma_ops for direct mappings
Quoted 3 lines · On Tue, 17 Apr 2018 02:05:12 +0800 kbuild test robot <[email protected]> wrote:
Hi Christoph,
I love your patch! Yet something to improve:
I was just about to complain about the same compile error ;-)
Quoted 28 lines
[auto build test ERROR on linus/master]
[also build test ERROR on v4.17-rc1 next-20180416]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]url: https://github.com/0day-ci/linux/commits/Christoph-Hellwig/dma-mapping-bypass-dma_ops-for-direct-mappings/20180416-230032
config: i386-defconfig (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0 reproduce:# save the attached .config to linux build tree make ARCH=i386All errors (new ones prefixed by >>):
drivers/char/agp/intel-gtt.o: In function `intel_gmch_probe':Quoted 1 line
Quoted 1 line
intel-gtt.c:(.text+0x11e4): undefined reference to `swiotlb_dma_ops'
drivers/ata/ahci.o: In function `ahci_init_one':Quoted 1 line
Quoted 1 line
ahci.c:(.text+0x108d): undefined reference to `swiotlb_dma_ops'
drivers/net/ethernet/broadcom/bnx2.o: In function `bnx2_init_one':Quoted 1 line
Quoted 1 line
bnx2.c:(.text+0x7fe7): undefined reference to `swiotlb_dma_ops'
drivers/net/ethernet/broadcom/tg3.o: In function `tg3_init_one':Quoted 1 line
Quoted 1 line
tg3.c:(.text+0x13549): undefined reference to `swiotlb_dma_ops'
drivers/net/ethernet/intel/e1000/e1000_main.o: In function `e1000_probe':Quoted 1 line
Quoted 1 line
e1000_main.c:(.text+0x49b3): undefined reference to `swiotlb_dma_ops'
drivers/net/ethernet/intel/e1000e/netdev.o:netdev.c:(.text+0xa65e): more undefined references to `swiotlb_dma_ops' follow
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
kbuild test robot
build botRe: dma-mapping: bypass dma_ops for direct mappings
In reply to Christoph HellwigI love your patch! Perhaps something to improve:
Hi Christoph,
I love your patch! Perhaps something to improve:
[auto build test WARNING on linus/master]
[also build test WARNING on v4.17-rc1 next-20180416]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Christoph-Hellwig/dma-mapping-bypass-dma_ops-for-direct-mappings/20180416-230032
reproduce: make htmldocs
All warnings (new ones prefixed by >>):
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
include/net/mac80211.h:2083: warning: bad line: >
net/mac80211/sta_info.h:586: warning: Function parameter or member 'rx_stats_avg' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'rx_stats_avg.signal' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'rx_stats_avg.chain_signal' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'status_stats.filtered' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'status_stats.retry_failed' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'status_stats.retry_count' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'status_stats.lost_packets' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'status_stats.last_tdls_pkt_time' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'status_stats.msdu_retries' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'status_stats.msdu_failed' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'status_stats.last_ack' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'status_stats.last_ack_signal' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'status_stats.ack_signal_filled' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'tx_stats.packets' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'tx_stats.bytes' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'tx_stats.last_rate' not described in 'sta_info' net/mac80211/sta_info.h:586: warning: Function parameter or member 'tx_stats.msdu' not described in 'sta_info' kernel/sched/fair.c:3731: warning: Function parameter or member 'flags' not described in 'attach_entity_load_avg' include/linux/dma-buf.h:307: warning: Function parameter or member 'cb_excl.cb' not described in 'dma_buf' include/linux/dma-buf.h:307: warning: Function parameter or member 'cb_excl.poll' not described in 'dma_buf' include/linux/dma-buf.h:307: warning: Function parameter or member 'cb_excl.active' not described in 'dma_buf' include/linux/dma-buf.h:307: warning: Function parameter or member 'cb_shared.cb' not described in 'dma_buf' include/linux/dma-buf.h:307: warning: Function parameter or member 'cb_shared.poll' not described in 'dma_buf' include/linux/dma-buf.h:307: warning: Function parameter or member 'cb_shared.active' not described in 'dma_buf' include/linux/dma-fence-array.h:54: warning: Function parameter or member 'work' not described in 'dma_fence_array'
Error: Cannot open file drivers/base/firmware_class.c
WARNING: kernel-doc 'scripts/kernel-doc -rst -enable-lineno -function request_firmware drivers/base/firmware_class.c' failed with return code 1
Error: Cannot open file drivers/base/firmware_class.c
WARNING: kernel-doc 'scripts/kernel-doc -rst -enable-lineno -function request_firmware_direct drivers/base/firmware_class.c' failed with return code 1
Error: Cannot open file drivers/base/firmware_class.c
WARNING: kernel-doc 'scripts/kernel-doc -rst -enable-lineno -function request_firmware_into_buf drivers/base/firmware_class.c' failed with return code 1
Error: Cannot open file drivers/base/firmware_class.c
WARNING: kernel-doc 'scripts/kernel-doc -rst -enable-lineno -function request_firmware_nowait drivers/base/firmware_class.c' failed with return code 1
Error: Cannot open file drivers/base/firmware_class.c
WARNING: kernel-doc 'scripts/kernel-doc -rst -enable-lineno -function firmware_request_cache drivers/base/firmware_class.c' failed with return code 1 include/linux/gpio/driver.h:142: warning: Function parameter or member 'request_key' not described in 'gpio_irq_chip' include/linux/iio/iio.h:270: warning: Function parameter or member 'scan_type.sign' not described in 'iio_chan_spec' include/linux/iio/iio.h:270: warning: Function parameter or member 'scan_type.realbits' not described in 'iio_chan_spec' include/linux/iio/iio.h:270: warning: Function parameter or member 'scan_type.storagebits' not described in 'iio_chan_spec' include/linux/iio/iio.h:270: warning: Function parameter or member 'scan_type.shift' not described in 'iio_chan_spec' include/linux/iio/iio.h:270: warning: Function parameter or member 'scan_type.repeat' not described in 'iio_chan_spec' include/linux/iio/iio.h:270: warning: Function parameter or member 'scan_type.endianness' not described in 'iio_chan_spec' include/linux/iio/hw-consumer.h:1: warning: no structured comments found
>> include/linux/device.h:995: warning: Function parameter or member 'is_dma_direct' not described in 'device'
Error: Cannot open file drivers/base/firmware_class.c
Error: Cannot open file drivers/base/firmware_class.c
WARNING: kernel-doc 'scripts/kernel-doc -rst -enable-lineno -export drivers/base/firmware_class.c' failed with return code 2 include/linux/input/sparse-keymap.h:46: warning: Function parameter or member 'sw' not described in 'key_entry' include/linux/mtd/rawnand.h:752: warning: Function parameter or member 'timings.sdr' not described in 'nand_data_interface' include/linux/mtd/rawnand.h:817: warning: Function parameter or member 'buf' not described in 'nand_op_data_instr' include/linux/mtd/rawnand.h:817: warning: Function parameter or member 'buf.in' not described in 'nand_op_data_instr' include/linux/mtd/rawnand.h:817: warning: Function parameter or member 'buf.out' not described in 'nand_op_data_instr' include/linux/mtd/rawnand.h:863: warning: Function parameter or member 'ctx' not described in 'nand_op_instr' include/linux/mtd/rawnand.h:863: warning: Function parameter or member 'ctx.cmd' not described in 'nand_op_instr' include/linux/mtd/rawnand.h:863: warning: Function parameter or member 'ctx.addr' not described in 'nand_op_instr' include/linux/mtd/rawnand.h:863: warning: Function parameter or member 'ctx.data' not described in 'nand_op_instr' include/linux/mtd/rawnand.h:863: warning: Function parameter or member 'ctx.waitrdy' not described in 'nand_op_instr' include/linux/mtd/rawnand.h:1010: warning: Function parameter or member 'ctx' not described in 'nand_op_parser_pattern_elem' include/linux/mtd/rawnand.h:1010: warning: Function parameter or member 'ctx.addr' not described in 'nand_op_parser_pattern_elem' include/linux/mtd/rawnand.h:1010: warning: Function parameter or member 'ctx.data' not described in 'nand_op_parser_pattern_elem' include/linux/mtd/rawnand.h:1313: warning: Function parameter or member 'manufacturer.desc' not described in 'nand_chip' include/linux/mtd/rawnand.h:1313: warning: Function parameter or member 'manufacturer.priv' not described in 'nand_chip' include/linux/regulator/driver.h:222: warning: Function parameter or member 'resume_early' not described in 'regulator_ops' drivers/regulator/core.c:4306: warning: Excess function parameter 'state' description in 'regulator_suspend_late' arch/s390/include/asm/cio.h:245: warning: Function parameter or member 'esw.esw0' not described in 'irb' arch/s390/include/asm/cio.h:245: warning: Function parameter or member 'esw.esw1' not described in 'irb' arch/s390/include/asm/cio.h:245: warning: Function parameter or member 'esw.esw2' not described in 'irb' arch/s390/include/asm/cio.h:245: warning: Function parameter or member 'esw.esw3' not described in 'irb' arch/s390/include/asm/cio.h:245: warning: Function parameter or member 'esw.eadm' not described in 'irb' drivers/usb/typec/mux.c:186: warning: Function parameter or member 'mux' not described in 'typec_mux_unregister' drivers/usb/typec/mux.c:186: warning: Excess function parameter 'sw' description in 'typec_mux_unregister'
vim +995 include/linux/device.h
^1da177e Linus Torvalds 2005-04-16 @995
:::::: The code at line 995 was first introduced by commit :::::: 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 Linux-2.6.12-rc2
:::::: TO: Linus Torvalds <[email protected]> :::::: CC: Linus Torvalds <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Quoted 9 lines · On Mon, 16 Apr 2018 05:27:06 -0700 Christoph Hellwig <[email protected]> wrote:
Can you try the following hack which avoids indirect calls entirely for the fast path direct mapping case?
---
Fromb256a008c1b305e6a1c2afe7c004c54ad2e96d4bMon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Mon, 16 Apr 2018 14:18:14 +0200
Subject: dma-mapping: bypassdma_opsfor direct mappings
Quoted 3 lines
Reportedly the retpoline mitigation for spectre causes huge penalties for indirect function calls. This hack bypasses the
dma_opsmechanism for simple direct mappings.
I did below to get it compiling, and working...
On X86 swiotlb fallback (via get_dma_ops -> get_arch_dma_ops) to use x86_swiotlb_dma_ops, instead of swiotlb_dma_ops. I also included that in below fix patch.
Performance improved to 8.9 Mpps from approx 6.5Mpps.
(This was without my bulking for net_device->ndo_xdp_xmit, so that number should improve more).
---
[PATCH RFC] fixups for Hellwig's DMA avoid retpoline overhead patch
From: Jesper Dangaard Brouer <[email protected]>
Performance improved to 8.9 Mpps
8917613 pkt/s
it was around 6.5 Mpps before.
--- arch/x86/kernel/pci-swiotlb.c | 3 ++- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 1 + include/linux/dma-mapping.h | 14 +++++++++++++- lib/Kconfig | 2 +- lib/Makefile | 1 + lib/dma-direct.c | 2 ++ lib/swiotlb.c | 1 + 7 files changed, 21 insertions(+), 3 deletions(-)
arch/x86/kernel/pci-swiotlb.c +2 -1
@@ -48,7 +48,7 @@ void x86_swiotlb_free_coherent(struct device *dev, size_t size,dma_generic_free_coherent (dev, size, vaddr, dma_addr, attrs); } -static const struct dma_map_ops x86_swiotlb_dma_ops = { +const struct dma_map_ops x86_swiotlb_dma_ops = { .mapping_error = swiotlb_dma_mapping_error, .alloc = x86_swiotlb_alloc_coherent, .free = x86_swiotlb_free_coherent,
@@ -62,6 +62,7 @@ static const struct dma_map_ops x86_swiotlb_dma_ops = { .unmap_page = swiotlb_unmap_page, .dma_supported = NULL, }; +EXPORT_SYMBOL (x86_swiotlb_dma_ops);/* * pci_swiotlb_detect_override - set swiotlb to 1 if necessary
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +1 -0
@@ -10297,6 +10297,7 @@ static int ixgbe_probe(struct pci_dev *pdev, const struct pci_device_id *ent)return err;if (!dma_set_mask_and_coherent (&pdev->dev,DMA_BIT_MASK (64 ))) { +pr_info ("XXX %s() dma_set_mask_and_coherent\n" , __func__); pci_using_dac =1 ; }else { err =dma_set_mask_and_coherent (&pdev->dev,DMA_BIT_MASK (32 ));
include/linux/dma-mapping.h +13 -1
@@ -622,6 +622,7 @@ static inline int dma_supported(struct device *dev, u64 mask) }extern const struct dma_map_ops swiotlb_dma_ops; +extern const struct dma_map_ops x86_swiotlb_dma_ops;#ifndef HAVE_ARCH_DMA_SET_MASK static inline int dma_set_mask (struct device *dev,u64 mask)
@@ -632,12 +633,23 @@ static inline int dma_set_mask(struct device *dev, u64 mask)dma_check_mask (dev, mask); *dev->dma_mask = mask; +#ifdef CONFIG_DMA_DIRECT_OPS if (dev->dma_ops == &dma_direct_ops || +# ifdef CONFIG_SWIOTLB (dev->dma_ops == &swiotlb_dma_ops && - mask ==DMA_BIT_MASK (64 ))) + mask ==DMA_BIT_MASK (64 )) || +# ifdef CONFIG_X86 + (get_dma_ops (dev) == &x86_swiotlb_dma_ops && + mask ==DMA_BIT_MASK (64 )) +# endif /* CONFIG_X86 */ +# endif /* CONFIG_SWIOTLB */ + ) dev->is_dma_direct = true; else +#endif /* CONFIG_DMA_DIRECT_OPS */ dev->is_dma_direct = false; + +pr_info ("XXX: %s() DMA is direct: %d\n" , __func__, dev->is_dma_direct); return 0; }#endif
lib/Kconfig +1 -1
@@ -416,7 +416,7 @@ config SGL_ALLOC config DMA_DIRECT_OPS bool depends on HAS_DMA && (!64BIT || ARCH_DMA_ADDR_T_64BIT) - default n + default y config DMA_VIRT_OPS bool
lib/Makefile +1 -0
@@ -29,6 +29,7 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \ lib-$(CONFIG_MMU) += ioremap.o lib-$(CONFIG_SMP) += cpumask.o lib-$(CONFIG_DMA_DIRECT_OPS) += dma-direct.o +#lib-y += dma-direct.o lib-$(CONFIG_DMA_VIRT_OPS) += dma-virt.o lib-y += kobject.o klist.o
lib/dma-direct.c +2 -0
@@ -107,6 +107,7 @@ dma_addr_t dma_direct_map_page(struct device *dev, struct page *page,return DIRECT_MAPPING_ERROR ;return dma_addr; } +EXPORT_SYMBOL (dma_direct_map_page);int dma_direct_map_sg (struct device *dev,struct scatterlist *sgl,int nents,enum dma_data_direction dir,unsigned long attrs)
@@ -125,6 +126,7 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl,return nents; } +EXPORT_SYMBOL (dma_direct_map_sg);int dma_direct_supported (struct device *dev,u64 mask) {
lib/swiotlb.c +1 -0
@@ -1132,4 +1132,5 @@ const struct dma_map_ops swiotlb_dma_ops = { .unmap_page = swiotlb_unmap_page, .dma_supported = swiotlb_dma_supported, }; +EXPORT_SYMBOL (swiotlb_dma_ops);#endif /* CONFIG_DMA_DIRECT_OPS */
Quoted 3 lines · Jesper Dangaard Brouer
On X86 swiotlb fallback (via
get_dma_ops->get_arch_dma_ops) to usex86_swiotlb_dma_ops, instead ofswiotlb_dma_ops. I also included that in below fix patch.
x86_swiotlb_dma_ops should not exist any mor, and x86 now uses dma_direct_ops. Looks like you are applying it to an old kernel :)
Quoted 4 lines
Performance improved to 8.9 Mpps from approx 6.5Mpps.
(This was without my bulking for
net_device->ndo_xdp_xmit, so that number should improve more).
What is the number for the otherwise comparable setup without repolines?
Quoted 8 lines · On Mon, 16 Apr 2018 23:15:50 -0700 Christoph Hellwig <[email protected]> wrote:
Quoted 3 lines · Jesper Dangaard Brouer
On X86 swiotlb fallback (via
get_dma_ops->get_arch_dma_ops) to usex86_swiotlb_dma_ops, instead ofswiotlb_dma_ops. I also included that in below fix patch.
x86_swiotlb_dma_opsshould not exist any mor, and x86 now usesdma_direct_ops. Looks like you are applying it to an old kernel :)
Quoted 6 lines
Quoted 4 lines
Performance improved to 8.9 Mpps from approx 6.5Mpps.
(This was without my bulking for
net_device->ndo_xdp_xmit, so that number should improve more).What is the number for the otherwise comparable setup without repolines?
Approx 12 Mpps.
You forgot to handle the dma_direct_mapping_error() case, which still used the retpoline in the above 8.9 Mpps measurement, I fixed it up and performance increased to 9.6 Mpps.
Notice, in this test there are still two retpoline/indirect-calls left. The net_device->ndo_xdp_xmit and the invocation of the XDP BPF prog.
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Quoted 6 lines · Jesper Dangaard Brouer
Quoted 3 lines
Quoted 1 line
number should improve more).
What is the number for the otherwise comparable setup without repolines?
Approx 12 Mpps.
Quoted 7 lines
You forgot to handle the
dma_direct_mapping_error()case, which still used the retpoline in the above 8.9 Mpps measurement, I fixed it up and performance increased to 9.6 Mpps.Notice, in this test there are still two retpoline/indirect-calls left. The
net_device->ndo_xdp_xmitand the invocation of the XDP BPF prog.
But that seems like a pretty clear indicator that we want the fast path direct mapping. I'll try to find some time over the next weeks to do a cleaner version of it.