dma-direct: avoid redundant memory sync for swiotlb
PATCH 1/1 dma-direct: avoid redundant memory sync for swiotlb
When we looked into FIO performance with swiotlb enabled in VM, we found swiotlb_bounce() is always called one more time than expected for each DMA read request.
It turns out that the bounce buffer is copied to original DMA buffer twice after the completion of a DMA request (one is done by in dma_direct_sync_single_for_cpu(), the other by swiotlb_tbl_unmap_single()). But the content in bounce buffer actually doesn't change between the two rounds of copy. So, one round of copy is redundant.
Pass DMA_ATTR_SKIP_CPU_SYNC flag to swiotlb_tbl_unmap_single() to skip the memory copy in it.
This fix increases FIO 64KB sequential read throughput in a guest with swiotlb=force by 5.6%.
--- kernel/dma/direct.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/dma/direct.h +2 -1
@@ -114,6 +114,7 @@ static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr,dma_direct_sync_single_for_cpu (dev, addr, size, dir);if (unlikely (is_swiotlb_buffer (dev, phys))) - swiotlb_tbl_unmap_single(dev, phys, size, dir, attrs); +swiotlb_tbl_unmap_single (dev, phys, size, dir, + attrs | DMA_ATTR_SKIP_CPU_SYNC); }#endif /* _KERNEL_DMA_DIRECT_H */
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 10 lines · Chao Gao
When we looked into FIO performance with swiotlb enabled in VM, we found
swiotlb_bounce()is always called one more time than expected for each DMA read request.It turns out that the bounce buffer is copied to original DMA buffer twice after the completion of a DMA request (one is done by in
dma_direct_sync_single_for_cpu(), the other byswiotlb_tbl_unmap_single()). But the content in bounce buffer actually doesn't change between the two rounds of copy. So, one round of copy is redundant.
Quoted 6 lines
Pass
DMA_ATTR_SKIP_CPU_SYNCflag toswiotlb_tbl_unmap_single()to skip the memory copy in it.This fix increases FIO 64KB sequential read throughput in a guest with swiotlb=force by 5.6%.
Sorry. A fixes tag is missing:
Quoted 24 lines
Reported-by Wang Zhaoyang1 <[email protected]>Reported-by Gao Liang <[email protected]>Signed-off-by Chao Gao <[email protected]>Reviewed-by Kevin Tian <[email protected]>---
kernel/dma/direct.h| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)kernel/dma/direct.h +2
-1@@ -114,6 +114,7 @@ static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr,dma_direct_sync_single_for_cpu (dev, addr, size, dir);if (unlikely (is_swiotlb_buffer (dev, phys))) - swiotlb_tbl_unmap_single(dev, phys, size, dir, attrs); +swiotlb_tbl_unmap_single (dev, phys, size, dir, + attrs | DMA_ATTR_SKIP_CPU_SYNC); }#endif /* _KERNEL_DMA_DIRECT_H */ Signature · 1 line
2.25.1
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Robin Murphy
reviewerQuoted 4 lines · Chao Gao
When we looked into FIO performance with swiotlb enabled in VM, we found
swiotlb_bounce()is always called one more time than expected for each DMA read request.
Quoted 8 lines
It turns out that the bounce buffer is copied to original DMA buffer twice after the completion of a DMA request (one is done by in
dma_direct_sync_single_for_cpu(), the other byswiotlb_tbl_unmap_single()). But the content in bounce buffer actually doesn't change between the two rounds of copy. So, one round of copy is redundant.Pass
DMA_ATTR_SKIP_CPU_SYNCflag toswiotlb_tbl_unmap_single()to skip the memory copy in it.
It's still a little suboptimal and non-obvious to call into SWIOTLB twice though - even better might be for SWIOTLB to call arch_sync_dma_for_cpu() at the appropriate place internally, then put the dma_direct_sync in an else path here. I'm really not sure why we have the current disparity between map and unmap in this regard... :/
Robin.
Quoted 16 lines
This fix increases FIO 64KB sequential read throughput in a guest with swiotlb=force by 5.6%.
Reported-by Wang Zhaoyang1 <[email protected]>Reported-by Gao Liang <[email protected]>Signed-off-by Chao Gao <[email protected]>Reviewed-by Kevin Tian <[email protected]>--- kernel/dma/direct.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)kernel/dma/direct.h +0
-0
kernel/dma/direct.h static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, · line 120
Quoted 8 lines
kernel/dma/direct.h +2
-1@@ -114,6 +114,7 @@ static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr,dma_direct_sync_single_for_cpu (dev, addr, size, dir);if (unlikely (is_swiotlb_buffer (dev, phys))) - swiotlb_tbl_unmap_single(dev, phys, size, dir, attrs); +swiotlb_tbl_unmap_single (dev, phys, size, dir, + attrs | DMA_ATTR_SKIP_CPU_SYNC); }#endif /* _KERNEL_DMA_DIRECT_H */
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 11 lines · Robin Murphy
Quoted 10 lines · Chao Gao
When we looked into FIO performance with swiotlb enabled in VM, we found
swiotlb_bounce()is always called one more time than expected for each DMA read request.It turns out that the bounce buffer is copied to original DMA buffer twice after the completion of a DMA request (one is done by in
dma_direct_sync_single_for_cpu(), the other byswiotlb_tbl_unmap_single()). But the content in bounce buffer actually doesn't change between the two rounds of copy. So, one round of copy is redundant.
Quoted 6 lines
Quoted 2 lines
Pass
DMA_ATTR_SKIP_CPU_SYNCflag toswiotlb_tbl_unmap_single()to skip the memory copy in it.It's still a little suboptimal and non-obvious to call into SWIOTLB twice though - even better might be for SWIOTLB to call
arch_sync_dma_for_cpu()at the appropriate place internally,
Hi Robin,
dma_direct_sync_single_for_cpu() also calls arch_sync_dma_for_cpu_all() and arch_dma_mark_clean() in some cases. if SWIOTLB does sync internally, should these two functions be called by SWIOTLB?
Personally, it might be better if swiotlb can just focus on bounce buffer alloc/free. Adding more DMA coherence logic into swiotlb will make it a little complicated.
How about an open-coded version of dma_direct_sync_single_for_cpu in dma_direct_unmap_page with swiotlb_sync_single_for_cpu replaced by swiotlb_tbl_unmap_single?
Quoted 11 lines · Chao Gao
dma_direct_sync_single_for_cpu()also callsarch_sync_dma_for_cpu_all()andarch_dma_mark_clean()in some cases. if SWIOTLB does sync internally, should these two functions be called by SWIOTLB?Personally, it might be better if swiotlb can just focus on bounce buffer alloc/free. Adding more DMA coherence logic into swiotlb will make it a little complicated.
How about an open-coded version of
dma_direct_sync_single_for_cpuindma_direct_unmap_pagewithswiotlb_sync_single_for_cpureplaced byswiotlb_tbl_unmap_single?
I don't think the swiotlb and non-coherent case ever fully worked. Before the merge of swiotlb into dma-direct they obviously were mutally exclusive, and even now all the cache maintainance is done on the physical address of the original data, not the swiotlb buffer.
If we want to fix that properly all the arch dma calls will need to move into swiotlb, but that is a much bigger patch.
So for now I'd be happy with the one liner presented here, but eventually the whole area could use an overhaul.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 2 lines · Christoph Hellwig
So for now I'd be happy with the one liner presented here, but eventually the whole area could use an overhaul.
Thanks. Do you want me to post a new version with the Fixes tag or you will take care of it?
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 6 lines · Chao Gao
Quoted 2 lines · Christoph Hellwig
So for now I'd be happy with the one liner presented here, but eventually the whole area could use an overhaul.
Thanks. Do you want me to post a new version with the Fixes tag or you will take care of it?
I can add the fixes tag. I'll wait another day or two for more feedback, though.
Quoted 2 lines
Fixes 55897af63091 ("dma-direct: merge swiotlb_dma_ops into the dma_direct code")
---end quoted text---
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Robin Murphy
reviewerQuoted 9 lines · Christoph Hellwig
Quoted 8 lines · Chao Gao
dma_direct_sync_single_for_cpu()also callsarch_sync_dma_for_cpu_all()andarch_dma_mark_clean()in some cases. if SWIOTLB does sync internally, should these two functions be called by SWIOTLB?Personally, it might be better if swiotlb can just focus on bounce buffer alloc/free. Adding more DMA coherence logic into swiotlb will make it a little complicated.
Quoted 8 lines
Quoted 3 lines
How about an open-coded version of
dma_direct_sync_single_for_cpuindma_direct_unmap_pagewithswiotlb_sync_single_for_cpureplaced byswiotlb_tbl_unmap_single?I don't think the swiotlb and non-coherent case ever fully worked. Before the merge of swiotlb into dma-direct they obviously were mutally exclusive, and even now all the cache maintainance is done on the physical address of the original data, not the swiotlb buffer.
Are you sure? AFAICS swiotlb_map() does the right thing, and dma_direct_{sync,unmap} are working off the DMA address, which is that of the bounce slot when SWIOTLB is involved (not least, how would the is_swiotlb_buffer() checks work otherwise?)
Quoted 5 lines
If we want to fix that properly all the arch dma calls will need to move into swiotlb, but that is a much bigger patch.
So for now I'd be happy with the one liner presented here, but eventually the whole area could use an overhaul.
Sure, whoever gets round to tackling DMA_ATTR_NO_SNOOP first will need to go through all the cache maintenance hooks anyway, so happy to kick the can down the road until then.
Thanks,
Robin.
Quoted 4 lines · Robin Murphy
Are you sure? AFAICS
swiotlb_map()does the right thing, and dma_direct_{sync,unmap} are working off the DMA address, which is that of the bounce slot when SWIOTLB is involved (not least, how would theis_swiotlb_buffer()checks work otherwise?)
Yeah, actually this should be fine.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu