dma-contiguous: try local node first for dma_alloc_contiguous()
PATCH 1/1 dma-contiguous: try local node first for dma_alloc_contiguous()
There was a bug report on a multi-numa-nodes ARM server that when IOMMU is disabled, the dma_alloc_coherent() function always returns memory from node 0 even for devices attaching to other nodes, while they can get local dma memory when IOMMU is on with the same API.
The reason is, when IOMMU is disabled, the dma_alloc_coherent() will go the direct way and call dma_alloc_contiguous(). The system doesn't have any explicit cma setting (like per-numa cma), and only has a default 64MB cma reserved area (on node 0), where kernel will try first to allocate memory from.
Make the dma allocation more locality friendly by trying first the numa aware allocation alloc_pages_node() before falling back to the reserved cma area.
One more thought is to check the node of the reserved cma area and only call alloc_pages_nodes() when it isn't the same node that the device attaches to.
--- kernel/dma/contiguous.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
kernel/dma/contiguous.c +10 -2
@@ -371,8 +371,9 @@ static struct page *cma_alloc_aligned(struct cma *cma, size_t size, gfp_t gfp)*/ struct page *dma_alloc_contiguous (struct device *dev,size_t size,gfp_t gfp) { -#ifdef CONFIG_DMA_NUMA_CMA +#ifdef CONFIG_NUMA int nid =dev_to_node (dev); +struct page *page;#endif /* CMA can be used only in the context which permits sleeping */
@@ -386,7 +387,6 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)#ifdef CONFIG_DMA_NUMA_CMA if (nid !=NUMA_NO_NODE && !(gfp & (GFP_DMA |GFP_DMA32 ))) {struct cma *cma = dma_contiguous_pernuma_area[nid]; -struct page *page;if (cma) { page =cma_alloc_aligned (cma, size, gfp);
@@ -402,6 +402,14 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp) } }#endif + +#ifdef CONFIG_NUMA +/* Try first to allocate memory on the same node as the device */ + page =alloc_pages_node (nid, gfp,get_order (size)); +if (page) +return page; +#endif +if (!dma_contiguous_default_area)return NULL ;
Signature · 1 line
2.39.5 (Apple Git-154)
Robin Murphy
reviewerQuoted 11 lines · Feng Tang
There was a bug report on a multi-numa-nodes ARM server that when IOMMU is disabled, the
dma_alloc_coherent()function always returns memory from node 0 even for devices attaching to other nodes, while they can get local dma memory when IOMMU is on with the same API.The reason is, when IOMMU is disabled, the
dma_alloc_coherent()will go the direct way and calldma_alloc_contiguous(). The system doesn't have any explicit cma setting (like per-numa cma), and only has a default 64MB cma reserved area (on node 0), where kernel will try first to allocate memory from.
Quoted 7 lines
Make the dma allocation more locality friendly by trying first the numa aware allocation
alloc_pages_node()before falling back to the reserved cma area.One more thought is to check the node of the reserved cma area and only call
alloc_pages_nodes()when it isn't the same node that the device attaches to.
Frankly, no. We literally have a feature specifically created for this use-case already - if you care about NUMA locality of DMA allocations then enable DMA_NUMA_CMA. Or if you don't need CMA at all, then disabling CMA would be even better for the overall performance of your system.
Thanks,
Robin.
Quoted 43 lines
Signed-off-by Feng Tang <[email protected]>--- kernel/dma/contiguous.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)kernel/dma/contiguous.c +10
-2@@ -371,8 +371,9 @@ static struct page *cma_alloc_aligned(struct cma *cma, size_t size, gfp_t gfp)*/ struct page *dma_alloc_contiguous (struct device *dev,size_t size,gfp_t gfp) { -#ifdef CONFIG_DMA_NUMA_CMA +#ifdef CONFIG_NUMA int nid =dev_to_node (dev); +struct page *page;#endif /* CMA can be used only in the context which permits sleeping */ @@ -386,7 +387,6 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)#ifdef CONFIG_DMA_NUMA_CMA if (nid !=NUMA_NO_NODE && !(gfp & (GFP_DMA |GFP_DMA32 ))) {struct cma *cma = dma_contiguous_pernuma_area[nid]; -struct page *page;if (cma) { page =cma_alloc_aligned (cma, size, gfp);@@ -402,6 +402,14 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp) } }#endif + +#ifdef CONFIG_NUMA +/* Try first to allocate memory on the same node as the device */ + page =alloc_pages_node (nid, gfp,get_order (size)); +if (page) +return page; +#endif +if (!dma_contiguous_default_area)return NULL ;
Hi Robin,
Quoted 12 lines · Robin Murphy
Quoted 11 lines · Feng Tang
There was a bug report on a multi-numa-nodes ARM server that when IOMMU is disabled, the
dma_alloc_coherent()function always returns memory from node 0 even for devices attaching to other nodes, while they can get local dma memory when IOMMU is on with the same API.The reason is, when IOMMU is disabled, the
dma_alloc_coherent()will go the direct way and calldma_alloc_contiguous(). The system doesn't have any explicit cma setting (like per-numa cma), and only has a default 64MB cma reserved area (on node 0), where kernel will try first to allocate memory from.
Quoted 7 lines
Quoted 7 lines
Make the dma allocation more locality friendly by trying first the numa aware allocation
alloc_pages_node()before falling back to the reserved cma area.One more thought is to check the node of the reserved cma area and only call
alloc_pages_nodes()when it isn't the same node that the device attaches to.
Thanks for the prompt review!
Quoted 4 lines
Frankly, no. We literally have a feature specifically created for this use-case already - if you care about NUMA locality of DMA allocations then enable
DMA_NUMA_CMA. Or if you don't need CMA at all, then disabling CMA would be even better for the overall performance of your system.
That makes sense. I also recommended using the per-numa cma to the reporter.
One thing is this numa unfriendly behavior happens silently, and I haven't noticed it for long time till I got the bug report, as the normal distribution enable CONFIG_CMA which provides the 64M cma area on node 0 (mostly) by default. I'm afraid many normal users are experiencing the same thing without doing any extra explicit cma setting.
Thanks,
Feng
Quoted 43 lines
Thanks,
Robin.Quoted 40 lines
Signed-off-by Feng Tang <[email protected]>--- kernel/dma/contiguous.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)kernel/dma/contiguous.c +10
-2@@ -371,8 +371,9 @@ static struct page *cma_alloc_aligned(struct cma *cma, size_t size, gfp_t gfp)*/ struct page *dma_alloc_contiguous (struct device *dev,size_t size,gfp_t gfp) { -#ifdef CONFIG_DMA_NUMA_CMA +#ifdef CONFIG_NUMA int nid =dev_to_node (dev); +struct page *page;#endif /* CMA can be used only in the context which permits sleeping */ @@ -386,7 +387,6 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)#ifdef CONFIG_DMA_NUMA_CMA if (nid !=NUMA_NO_NODE && !(gfp & (GFP_DMA |GFP_DMA32 ))) {struct cma *cma = dma_contiguous_pernuma_area[nid]; -struct page *page;if (cma) { page =cma_alloc_aligned (cma, size, gfp);@@ -402,6 +402,14 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp) } }#endif + +#ifdef CONFIG_NUMA +/* Try first to allocate memory on the same node as the device */ + page =alloc_pages_node (nid, gfp,get_order (size)); +if (page) +return page; +#endif +if (!dma_contiguous_default_area)return NULL ;