Skip to contentSkip to conversation

add an API to check if a streamming mapping needs sync calls

PATCH In review Christoph Hellwig 4 participants 13 messages Jun 29, 2020 – Jul 8, 2020

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

Hi all,

this series lifts the somewhat hacky checks in the XSK code if a DMA streaming mapping needs dma_sync_single_for_{device,cpu} calls to the DMA API.

Diffstat:

 Documentation/core-api/dma-api.rst |    8 +++++
 include/linux/dma-direct.h         |    1 
 include/linux/dma-mapping.h        |    5 +++
 include/net/xsk_buff_pool.h        |    6 ++--
 kernel/dma/direct.c                |    6 ++++
 kernel/dma/mapping.c               |   10 ++++++
 net/xdp/xsk_buff_pool.c            |   54 ++-----------------------------------
 7 files changed, 37 insertions(+), 53 deletions(-)

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#1 lore raw

PATCH 1/4 dma-mapping: add a new dma_need_sync API

Add a new API to check if calls to dma_sync_single_for_{device,cpu} are required for a given DMA streaming mapping.

---
 Documentation/core-api/dma-api.rst |  8 ++++++++
 include/linux/dma-direct.h         |  1 +
 include/linux/dma-mapping.h        |  5 +++++
 kernel/dma/direct.c                |  6 ++++++
 kernel/dma/mapping.c               | 10 ++++++++++
 5 files changed, 30 insertions(+)
Documentation/core-api/dma-api.rst +8 -0
index 2d8d2fed731720..f41620439ef349 100644
@@ -204,6 +204,14 @@ Returns the maximum size of a mapping for the device. The size parameter
 of the mapping functions like dma_map_single(), dma_map_page() and
 others should not be larger than the returned value.
 
+::
+
+	bool
+	dma_need_sync(struct device *dev, dma_addr_t dma_addr);
+
+Returns %true if dma_sync_single_for_{device,cpu} calls are required to
+transfer memory ownership.  Returns %false if those calls can be skipped.
+
 ::
 
 	unsigned long
include/linux/dma-direct.h +1 -0
index cdfa400f89b3d3..5184735a0fe8eb 100644
@@ -85,4 +85,5 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma,
 		void *cpu_addr, dma_addr_t dma_addr, size_t size,
 		unsigned long attrs);
 int dma_direct_supported(struct device *dev, u64 mask);
+bool dma_direct_need_sync(struct device *dev, dma_addr_t dma_addr);
 #endif /* _LINUX_DMA_DIRECT_H */
include/linux/dma-mapping.h +5 -0
index 78f677cf45ab69..a33ed3954ed465 100644
@@ -461,6 +461,7 @@ int dma_set_mask(struct device *dev, u64 mask);
 int dma_set_coherent_mask(struct device *dev, u64 mask);
 u64 dma_get_required_mask(struct device *dev);
 size_t dma_max_mapping_size(struct device *dev);
+bool dma_need_sync(struct device *dev, dma_addr_t dma_addr);
 unsigned long dma_get_merge_boundary(struct device *dev);
 #else /* CONFIG_HAS_DMA */
 static inline dma_addr_t dma_map_page_attrs(struct device *dev,
@@ -571,6 +572,10 @@ static inline size_t dma_max_mapping_size(struct device *dev)
 {
 	return 0;
 }
+static inline bool dma_need_sync(struct device *dev, dma_addr_t dma_addr)
+{
+	return false;
+}
 static inline unsigned long dma_get_merge_boundary(struct device *dev)
 {
 	return 0;
kernel/dma/direct.c +6 -0
index 93f578a8e613ba..95866b64758100 100644
@@ -539,3 +539,9 @@ size_t dma_direct_max_mapping_size(struct device *dev)
 		return swiotlb_max_mapping_size(dev);
 	return SIZE_MAX;
 }
+
+bool dma_direct_need_sync(struct device *dev, dma_addr_t dma_addr)
+{
+	return !dev_is_dma_coherent(dev) ||
+		is_swiotlb_buffer(dma_to_phys(dev, dma_addr));
+}
kernel/dma/mapping.c +10 -0
index 98e3d873792ea4..a8c18c9a796fdc 100644
@@ -397,6 +397,16 @@ size_t dma_max_mapping_size(struct device *dev)
 }
 EXPORT_SYMBOL_GPL(dma_max_mapping_size);
 
+bool dma_need_sync(struct device *dev, dma_addr_t dma_addr)
+{
+	const struct dma_map_ops *ops = get_dma_ops(dev);
+
+	if (dma_is_direct(ops))
+		return dma_direct_need_sync(dev, dma_addr);
+	return ops->sync_single_for_cpu || ops->sync_single_for_device;
+}
+EXPORT_SYMBOL_GPL(dma_need_sync);
+
 unsigned long dma_get_merge_boundary(struct device *dev)
 {
 	const struct dma_map_ops *ops = get_dma_ops(dev);
Signature · 6 lines
2.26.2

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#2 lore raw

Re: [PATCH 1/4] dma-mapping: add a new dma_need_sync API

Documentation/core-api/dma-api.rst Returns the maximum size of a mapping for the device. The size parameter · line 213

Quoted 10 lines · Christoph Hellwig

Add a new API to check if calls to dma_sync_single_for_{device,cpu} are required for a given DMA streaming mapping.

+::

Documentation/core-api/dma-api.rst +6 -0
+
+	bool
+	dma_need_sync(struct device *dev, dma_addr_t dma_addr);
+
+Returns %true if dma_sync_single_for_{device,cpu} calls are required to
+transfer memory ownership.  Returns %false if those calls can be skipped.

Hi Christoph -

Thie call above is for a specific dma_addr. For correctness, would I need to check every addr, or can I assume that for a specific memory type (pages returned from malloc), that the answer would be identical?

Signature · 1 line
Jonathan
#3 lore raw

Re: [PATCH 1/4] dma-mapping: add a new dma_need_sync API

Quoted 14 lines · Jonathan Lemon

Documentation/core-api/dma-api.rst Returns the maximum size of a mapping for the device. The size parameter · line 213

Quoted 10 lines · Christoph Hellwig

Add a new API to check if calls to dma_sync_single_for_{device,cpu} are required for a given DMA streaming mapping.

+::

Documentation/core-api/dma-api.rst +6 -0
+
+	bool
+	dma_need_sync(struct device *dev, dma_addr_t dma_addr);
+
+Returns %true if dma_sync_single_for_{device,cpu} calls are required to
+transfer memory ownership.  Returns %false if those calls can be skipped.

Hi Christoph -

Quoted 3 lines

Thie call above is for a specific dma_addr. For correctness, would I need to check every addr, or can I assume that for a specific memory type (pages returned from malloc), that the answer would be identical?

You need to check every mapping. E.g. this API pairs with a dma_map_single/page call. For S/G mappings you'd need to call it for each entry, although if you have a use case for that we really should add a dma_sg_need_sync helper instea of open coding the scatterlist walk.

#4 lore raw

Re: [PATCH 1/4] dma-mapping: add a new dma_need_sync API

Quoted 15 lines · Christoph Hellwig
Quoted 14 lines · Jonathan Lemon

Documentation/core-api/dma-api.rst Returns the maximum size of a mapping for the device. The size parameter · line 213

Quoted 10 lines · Christoph Hellwig

Add a new API to check if calls to dma_sync_single_for_{device,cpu} are required for a given DMA streaming mapping.

+::

Documentation/core-api/dma-api.rst +6 -0
+
+	bool
+	dma_need_sync(struct device *dev, dma_addr_t dma_addr);
+
+Returns %true if dma_sync_single_for_{device,cpu} calls are required to
+transfer memory ownership.  Returns %false if those calls can be skipped.

Hi Christoph -

Quoted 8 lines
Quoted 3 lines

Thie call above is for a specific dma_addr. For correctness, would I need to check every addr, or can I assume that for a specific memory type (pages returned from malloc), that the answer would be identical?

You need to check every mapping. E.g. this API pairs with a dma_map_single/page call. For S/G mappings you'd need to call it for each entry, although if you have a use case for that we really should add a dma_sg_need_sync helper instea of open coding the scatterlist walk.

My use case is setting up a pinned memory area, and caching the dma mappings. I'd like to bypass storing the DMA addresses if they aren't needed. For example:

setup()

{
    if (dma_need_sync(dev, addr, len)) {
        kvmalloc_array(...)
        cache_dma_mappings(...)
    }

dev_get_dma(page)

{
    if (!cache)
        return page_to_phys(page)

    return dma_cache_lookup(...)

The reason for doing it this way is that the page in question may be backed by either system memory, or device memory such as a GPU. For the latter, the GPU provides a table of DMA addresses where data may be accessed, so I'm unable to use the dma_map_page() API.

Signature · 1 line
Jonathan
#5 lore raw

Re: [PATCH 1/4] dma-mapping: add a new dma_need_sync API

Quoted 26 lines · Jonathan Lemon
Quoted 4 lines

You need to check every mapping. E.g. this API pairs with a dma_map_single/page call. For S/G mappings you'd need to call it for each entry, although if you have a use case for that we really should add a dma_sg_need_sync helper instea of open coding the scatterlist walk.

My use case is setting up a pinned memory area, and caching the dma mappings. I'd like to bypass storing the DMA addresses if they aren't needed. For example:

setup()

{
    if (dma_need_sync(dev, addr, len)) {
        kvmalloc_array(...)
        cache_dma_mappings(...)
    }

dev_get_dma(page)

{
    if (!cache)
        return page_to_phys(page)

    return dma_cache_lookup(...)
Quoted 4 lines

The reason for doing it this way is that the page in question may be backed by either system memory, or device memory such as a GPU. For the latter, the GPU provides a table of DMA addresses where data may be accessed, so I'm unable to use the dma_map_page() API.

dma_need_sync doesn't tell you if the unmap needs the dma_addr_t. I've been think about replacing CONFIG_NEED_DMA_MAP_STATE with a runtime for a while, which would give you exattly what you need. For now it isn't very useful as there are very few configs left that do not have CONFIG_NEED_DMA_MAP_STATE set.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#6 lore raw

PATCH 2/4 xsk: replace the cheap_dma flag with a dma_need_sync flag

In reply to Christoph Hellwig

Invert the polarity and better name the flag so that the use case is properly documented.

---
 include/net/xsk_buff_pool.h | 6 +++---
 net/xdp/xsk_buff_pool.c     | 5 ++---
 2 files changed, 5 insertions(+), 6 deletions(-)
include/net/xsk_buff_pool.h +3 -3
index a4ff226505c99c..6842990e2712bd 100644
@@ -40,7 +40,7 @@ struct xsk_buff_pool {
 	u32 headroom;
 	u32 chunk_size;
 	u32 frame_len;
-	bool cheap_dma;
+	bool dma_need_sync;
 	bool unaligned;
 	void *addrs;
 	struct device *dev;
@@ -80,7 +80,7 @@ static inline dma_addr_t xp_get_frame_dma(struct xdp_buff_xsk *xskb)
 void xp_dma_sync_for_cpu_slow(struct xdp_buff_xsk *xskb);
 static inline void xp_dma_sync_for_cpu(struct xdp_buff_xsk *xskb)
 {
-	if (xskb->pool->cheap_dma)
+	if (!xskb->pool->dma_need_sync)
 		return;
 
 	xp_dma_sync_for_cpu_slow(xskb);
@@ -91,7 +91,7 @@ void xp_dma_sync_for_device_slow(struct xsk_buff_pool *pool, dma_addr_t dma,
 static inline void xp_dma_sync_for_device(struct xsk_buff_pool *pool,
 					  dma_addr_t dma, size_t size)
 {
-	if (pool->cheap_dma)
+	if (!pool->dma_need_sync)
 		return;
 
 	xp_dma_sync_for_device_slow(pool, dma, size);
net/xdp/xsk_buff_pool.c +2 -3
index 540ed75e44821c..9fe84c797a7060 100644
@@ -55,7 +55,6 @@ struct xsk_buff_pool *xp_create(struct page **pages, u32 nr_pages, u32 chunks,
 	pool->free_heads_cnt = chunks;
 	pool->headroom = headroom;
 	pool->chunk_size = chunk_size;
-	pool->cheap_dma = true;
 	pool->unaligned = unaligned;
 	pool->frame_len = chunk_size - headroom - XDP_PACKET_HEADROOM;
 	INIT_LIST_HEAD(&pool->free_list);
@@ -195,7 +194,7 @@ int xp_dma_map(struct xsk_buff_pool *pool, struct device *dev,
 		xp_check_dma_contiguity(pool);
 
 	pool->dev = dev;
-	pool->cheap_dma = xp_check_cheap_dma(pool);
+	pool->dma_need_sync = !xp_check_cheap_dma(pool);
 	return 0;
 }
 EXPORT_SYMBOL(xp_dma_map);
@@ -280,7 +279,7 @@ struct xdp_buff *xp_alloc(struct xsk_buff_pool *pool)
 	xskb->xdp.data = xskb->xdp.data_hard_start + XDP_PACKET_HEADROOM;
 	xskb->xdp.data_meta = xskb->xdp.data;
 
-	if (!pool->cheap_dma) {
+	if (pool->dma_need_sync) {
 		dma_sync_single_range_for_device(pool->dev, xskb->dma, 0,
 						 pool->frame_len,
 						 DMA_BIDIRECTIONAL);
Signature · 6 lines
2.26.2

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#7 lore raw

PATCH 3/4 xsk: remove a double pool->dev assignment in xp_dma_map

In reply to Christoph Hellwig

->dev is already assigned at the top of the function, remove the duplicate one at the end.

---
 net/xdp/xsk_buff_pool.c | 1 -
 1 file changed, 1 deletion(-)
net/xdp/xsk_buff_pool.c +0 -1
index 9fe84c797a7060..6733e2c59e4835 100644
@@ -193,7 +193,6 @@ int xp_dma_map(struct xsk_buff_pool *pool, struct device *dev,
 	if (pool->unaligned)
 		xp_check_dma_contiguity(pool);
 
-	pool->dev = dev;
 	pool->dma_need_sync = !xp_check_cheap_dma(pool);
 	return 0;
 }
Signature · 6 lines
2.26.2

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#8 lore raw

PATCH 4/4 xsk: use dma_need_sync instead of reimplenting it

In reply to Christoph Hellwig

Use the dma_need_sync helper instead of (not always entirely correctly) poking into the dma-mapping internals.

---
 net/xdp/xsk_buff_pool.c | 50 +++--------------------------------------
 1 file changed, 3 insertions(+), 47 deletions(-)
net/xdp/xsk_buff_pool.c +3 -47
index 6733e2c59e4835..08b80669f64955 100644
@@ -2,9 +2,6 @@
 
 #include <net/xsk_buff_pool.h>
 #include <net/xdp_sock.h>
-#include <linux/dma-direct.h>
-#include <linux/dma-noncoherent.h>
-#include <linux/swiotlb.h>
 
 #include "xsk_queue.h"
 
@@ -124,48 +121,6 @@ static void xp_check_dma_contiguity(struct xsk_buff_pool *pool)
 	}
 }
 
-static bool __maybe_unused xp_check_swiotlb_dma(struct xsk_buff_pool *pool)
-{
-#if defined(CONFIG_SWIOTLB)
-	phys_addr_t paddr;
-	u32 i;
-
-	for (i = 0; i < pool->dma_pages_cnt; i++) {
-		paddr = dma_to_phys(pool->dev, pool->dma_pages[i]);
-		if (is_swiotlb_buffer(paddr))
-			return false;
-	}
-#endif
-	return true;
-}
-
-static bool xp_check_cheap_dma(struct xsk_buff_pool *pool)
-{
-#if defined(CONFIG_HAS_DMA)
-	const struct dma_map_ops *ops = get_dma_ops(pool->dev);
-
-	if (ops) {
-		return !ops->sync_single_for_cpu &&
-			!ops->sync_single_for_device;
-	}
-
-	if (!dma_is_direct(ops))
-		return false;
-
-	if (!xp_check_swiotlb_dma(pool))
-		return false;
-
-	if (!dev_is_dma_coherent(pool->dev)) {
-#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) ||		\
-	defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) ||	\
-	defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE)
-		return false;
-#endif
-	}
-#endif
-	return true;
-}
-
 int xp_dma_map(struct xsk_buff_pool *pool, struct device *dev,
 	       unsigned long attrs, struct page **pages, u32 nr_pages)
 {
@@ -179,6 +134,7 @@ int xp_dma_map(struct xsk_buff_pool *pool, struct device *dev,
 
 	pool->dev = dev;
 	pool->dma_pages_cnt = nr_pages;
+	pool->dma_need_sync = false;
 
 	for (i = 0; i < pool->dma_pages_cnt; i++) {
 		dma = dma_map_page_attrs(dev, pages[i], 0, PAGE_SIZE,
@@ -187,13 +143,13 @@ int xp_dma_map(struct xsk_buff_pool *pool, struct device *dev,
 			xp_dma_unmap(pool, attrs);
 			return -ENOMEM;
 		}
+		if (dma_need_sync(dev, dma))
+			pool->dma_need_sync = true;
 		pool->dma_pages[i] = dma;
 	}
 
 	if (pool->unaligned)
 		xp_check_dma_contiguity(pool);
-
-	pool->dma_need_sync = !xp_check_cheap_dma(pool);
 	return 0;
 }
 EXPORT_SYMBOL(xp_dma_map);
Signature · 6 lines
2.26.2

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Tested-by Acked-by
#9 lore raw
In reply to Christoph Hellwig
Quoted 6 lines · Christoph Hellwig

Hi all,

this series lifts the somewhat hacky checks in the XSK code if a DMA streaming mapping needs dma_sync_single_for_{device,cpu} calls to the DMA API.

Thanks a lot for working on, and fixing this, Christoph!

I took the series for a spin, and there are (obviously) no performance regressions.

Would the patches go through the net/bpf trees or somewhere else?

For the series:

Björn

Quoted 11 lines

Diffstat:

  Documentation/core-api/dma-api.rst |    8 +++++
  include/linux/dma-direct.h         |    1
  include/linux/dma-mapping.h        |    5 +++
  include/net/xsk_buff_pool.h        |    6 ++--
  kernel/dma/direct.c                |    6 ++++
  kernel/dma/mapping.c               |   10 ++++++
  net/xdp/xsk_buff_pool.c            |   54 ++-----------------------------------
  7 files changed, 37 insertions(+), 53 deletions(-)

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#10 lore raw
Quoted 8 lines · Björn Töpel
Quoted 6 lines · Christoph Hellwig

Hi all,

this series lifts the somewhat hacky checks in the XSK code if a DMA streaming mapping needs dma_sync_single_for_{device,cpu} calls to the DMA API.

Quoted 6 lines

Thanks a lot for working on, and fixing this, Christoph!

I took the series for a spin, and there are (obviously) no performance regressions.

Would the patches go through the net/bpf trees or somewhere else?

either the net tree or (my) dma-mapping tree would be fine with me.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#11 lore raw
In reply to Björn Töpel
Quoted 8 lines · Björn Töpel
Quoted 6 lines · Christoph Hellwig

Hi all,

this series lifts the somewhat hacky checks in the XSK code if a DMA streaming mapping needs dma_sync_single_for_{device,cpu} calls to the DMA API.

Quoted 6 lines

Thanks a lot for working on, and fixing this, Christoph!

I took the series for a spin, and there are (obviously) no performance regressions.

Would the patches go through the net/bpf trees or somewhere else?

Where did this end up? I still don't see it in Linus' tree and this is getting urgent now.

maintainer
#12 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Björn Töpel
Quoted 6 lines · Christoph Hellwig

Hi all,

this series lifts the somewhat hacky checks in the XSK code if a DMA streaming mapping needs dma_sync_single_for_{device,cpu} calls to the DMA API.

Thanks a lot for working on, and fixing this, Christoph!

Quoted 7 lines
Quoted 4 lines

I took the series for a spin, and there are (obviously) no performance regressions.

Would the patches go through the net/bpf trees or somewhere else?

Where did this end up? I still don't see it in Linus' tree and this is getting urgent now.

It was merged into bpf tree and we sent the PR to DaveM which was merged into net tree around a week ago [0]; I assume the PR for net might go to Linus soon this week.

[0] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=e708e2bd55c921f5bb554fa5837d132a878951cf