Skip to contentSkip to conversation

dma-contiguous: try local node first for dma_alloc_contiguous()

PATCH In review Feng Tang 2 participants 3 messages Apr 14 DMA mapping helpers

lore
Share & tools
lore

Feng Tang

author
#0 lore raw

PATCH 1/1 dma-contiguous: try local node first for dma_alloc_contiguous()

There was a bug report on a multi-numa-nodes ARM server that when IOMMU is disabled, the dma_alloc_coherent() function always returns memory from node 0 even for devices attaching to other nodes, while they can get local dma memory when IOMMU is on with the same API.

The reason is, when IOMMU is disabled, the dma_alloc_coherent() will go the direct way and call dma_alloc_contiguous(). The system doesn't have any explicit cma setting (like per-numa cma), and only has a default 64MB cma reserved area (on node 0), where kernel will try first to allocate memory from.

Make the dma allocation more locality friendly by trying first the numa aware allocation alloc_pages_node() before falling back to the reserved cma area.

One more thought is to check the node of the reserved cma area and only call alloc_pages_nodes() when it isn't the same node that the device attaches to.

---
 kernel/dma/contiguous.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
kernel/dma/contiguous.c +10 -2
index c56004d314dc..0180f40f094e 100644
@@ -371,8 +371,9 @@ static struct page *cma_alloc_aligned(struct cma *cma, size_t size, gfp_t gfp)
  */
 struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
 {
-#ifdef CONFIG_DMA_NUMA_CMA
+#ifdef CONFIG_NUMA
 	int nid = dev_to_node(dev);
+	struct page *page;
 #endif
 
 	/* CMA can be used only in the context which permits sleeping */
@@ -386,7 +387,6 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
 #ifdef CONFIG_DMA_NUMA_CMA
 	if (nid != NUMA_NO_NODE && !(gfp & (GFP_DMA | GFP_DMA32))) {
 		struct cma *cma = dma_contiguous_pernuma_area[nid];
-		struct page *page;
 
 		if (cma) {
 			page = cma_alloc_aligned(cma, size, gfp);
@@ -402,6 +402,14 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
 		}
 	}
 #endif
+
+#ifdef CONFIG_NUMA
+	/* Try first to allocate memory on the same node as the device */
+	page = alloc_pages_node(nid, gfp, get_order(size));
+	if (page)
+		return page;
+#endif
+
 	if (!dma_contiguous_default_area)
 		return NULL;
 
Signature · 1 line
2.39.5 (Apple Git-154)
reviewer
#1 lore raw
Quoted 11 lines · Feng Tang

There was a bug report on a multi-numa-nodes ARM server that when IOMMU is disabled, the dma_alloc_coherent() function always returns memory from node 0 even for devices attaching to other nodes, while they can get local dma memory when IOMMU is on with the same API.

The reason is, when IOMMU is disabled, the dma_alloc_coherent() will go the direct way and call dma_alloc_contiguous(). The system doesn't have any explicit cma setting (like per-numa cma), and only has a default 64MB cma reserved area (on node 0), where kernel will try first to allocate memory from.

Quoted 7 lines

Make the dma allocation more locality friendly by trying first the numa aware allocation alloc_pages_node() before falling back to the reserved cma area.

One more thought is to check the node of the reserved cma area and only call alloc_pages_nodes() when it isn't the same node that the device attaches to.

Frankly, no. We literally have a feature specifically created for this use-case already - if you care about NUMA locality of DMA allocations then enable DMA_NUMA_CMA. Or if you don't need CMA at all, then disabling CMA would be even better for the overall performance of your system.

Thanks,
Robin.

Quoted 43 lines
---
  kernel/dma/contiguous.c | 12 ++++++++++--
  1 file changed, 10 insertions(+), 2 deletions(-)
kernel/dma/contiguous.c +10 -2
index c56004d314dc..0180f40f094e 100644
@@ -371,8 +371,9 @@ static struct page *cma_alloc_aligned(struct cma *cma, size_t size, gfp_t gfp)
   */
  struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
  {
-#ifdef CONFIG_DMA_NUMA_CMA
+#ifdef CONFIG_NUMA
  	int nid = dev_to_node(dev);
+	struct page *page;
  #endif
 
  	/* CMA can be used only in the context which permits sleeping */
@@ -386,7 +387,6 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
  #ifdef CONFIG_DMA_NUMA_CMA
  	if (nid != NUMA_NO_NODE && !(gfp & (GFP_DMA | GFP_DMA32))) {
  		struct cma *cma = dma_contiguous_pernuma_area[nid];
-		struct page *page;
 
  		if (cma) {
  			page = cma_alloc_aligned(cma, size, gfp);
@@ -402,6 +402,14 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
  		}
  	}
  #endif
+
+#ifdef CONFIG_NUMA
+	/* Try first to allocate memory on the same node as the device */
+	page = alloc_pages_node(nid, gfp, get_order(size));
+	if (page)
+		return page;
+#endif
+
  	if (!dma_contiguous_default_area)
  		return NULL;
 
author
#2 lore raw

Hi Robin,

Quoted 12 lines · Robin Murphy
Quoted 11 lines · Feng Tang

There was a bug report on a multi-numa-nodes ARM server that when IOMMU is disabled, the dma_alloc_coherent() function always returns memory from node 0 even for devices attaching to other nodes, while they can get local dma memory when IOMMU is on with the same API.

The reason is, when IOMMU is disabled, the dma_alloc_coherent() will go the direct way and call dma_alloc_contiguous(). The system doesn't have any explicit cma setting (like per-numa cma), and only has a default 64MB cma reserved area (on node 0), where kernel will try first to allocate memory from.

Quoted 7 lines
Quoted 7 lines

Make the dma allocation more locality friendly by trying first the numa aware allocation alloc_pages_node() before falling back to the reserved cma area.

One more thought is to check the node of the reserved cma area and only call alloc_pages_nodes() when it isn't the same node that the device attaches to.

Thanks for the prompt review!

Quoted 4 lines

Frankly, no. We literally have a feature specifically created for this use-case already - if you care about NUMA locality of DMA allocations then enable DMA_NUMA_CMA. Or if you don't need CMA at all, then disabling CMA would be even better for the overall performance of your system.

That makes sense. I also recommended using the per-numa cma to the reporter.

One thing is this numa unfriendly behavior happens silently, and I haven't noticed it for long time till I got the bug report, as the normal distribution enable CONFIG_CMA which provides the 64M cma area on node 0 (mostly) by default. I'm afraid many normal users are experiencing the same thing without doing any extra explicit cma setting.

Thanks,
Feng

Quoted 43 lines

Thanks,
Robin.

Quoted 40 lines
---
  kernel/dma/contiguous.c | 12 ++++++++++--
  1 file changed, 10 insertions(+), 2 deletions(-)
kernel/dma/contiguous.c +10 -2
index c56004d314dc..0180f40f094e 100644
@@ -371,8 +371,9 @@ static struct page *cma_alloc_aligned(struct cma *cma, size_t size, gfp_t gfp)
   */
  struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
  {
-#ifdef CONFIG_DMA_NUMA_CMA
+#ifdef CONFIG_NUMA
  	int nid = dev_to_node(dev);
+	struct page *page;
  #endif
  	/* CMA can be used only in the context which permits sleeping */
@@ -386,7 +387,6 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
  #ifdef CONFIG_DMA_NUMA_CMA
  	if (nid != NUMA_NO_NODE && !(gfp & (GFP_DMA | GFP_DMA32))) {
  		struct cma *cma = dma_contiguous_pernuma_area[nid];
-		struct page *page;
  		if (cma) {
  			page = cma_alloc_aligned(cma, size, gfp);
@@ -402,6 +402,14 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
  		}
  	}
  #endif
+
+#ifdef CONFIG_NUMA
+	/* Try first to allocate memory on the same node as the device */
+	page = alloc_pages_node(nid, gfp, get_order(size));
+	if (page)
+		return page;
+#endif
+
  	if (!dma_contiguous_default_area)
  		return NULL;