Skip to contentSkip to conversation

dma-contiguous: support numa CMA for specified node

PATCH Applied Yajun Deng 4 participants 12 messages Jul 12, 2023 – Jul 31, 2023

Subsystems (4)
lore
Share & tools
lore

Yajun Deng

author
#0 lore raw

PATCH v3 1/1 dma-contiguous: support numa CMA for specified node

The kernel parameter 'cma_pernuma=' only supports reserving the same size of CMA area for each node. We need to reserve different sizes of CMA area for specified nodes if these devices belong to different nodes.

Adding another kernel parameter 'numa_cma=' to reserve CMA area for the specified node. If we want to use one of these parameters, we need to enable DMA_NUMA_CMA.

At the same time, print the node id in cma_declare_contiguous_nid() if CONFIG_NUMA is enabled.

---
V2 -> V3: Use nid but not nid_buf in cma_declare_contiguous_nid(). V1 -> V2: Add 'numa_cma=' and keep 'cma_pernuma=' kernel parameter.

---
 .../admin-guide/kernel-parameters.txt         |  11 ++
 kernel/dma/Kconfig                            |   9 +-
 kernel/dma/contiguous.c                       | 101 ++++++++++++++----
 mm/cma.c                                      |  10 +-
 4 files changed, 102 insertions(+), 29 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +11 -0
index bdf0ab6716c8..87ad8154b730 100644
@@ -706,6 +706,17 @@
 			which is located in node nid, if the allocation fails,
 			they will fallback to the global default memory area.
 
+	numa_cma=<node>:nn[MG][,<node>:nn[MG]]
+			[KNL,CMA]
+			Sets the size of kernel numa memory area for
+			contiguous memory allocations. It will reserve CMA
+			area for the specified node.
+
+			With numa CMA enabled, DMA users on node nid will
+			first try to allocate buffer from the numa area
+			which is located in node nid, if the allocation fails,
+			they will fallback to the global default memory area.
+
 	cmo_free_hint=	[PPC] Format: { yes | no }
 			Specify whether pages are marked as being inactive
 			when they are freed.  This is used in CMO environments
kernel/dma/Kconfig +5 -4
index 7afde9bc529f..562463fe30ea 100644
@@ -145,15 +145,16 @@ config DMA_CMA
 
 if  DMA_CMA
 
-config DMA_PERNUMA_CMA
-	bool "Enable separate DMA Contiguous Memory Area for each NUMA Node"
+config DMA_NUMA_CMA
+	bool "Enable separate DMA Contiguous Memory Area for NUMA Node"
 	default NUMA
 	help
-	  Enable this option to get pernuma CMA areas so that NUMA devices
+	  Enable this option to get numa CMA areas so that NUMA devices
 	  can get local memory by DMA coherent APIs.
 
 	  You can set the size of pernuma CMA by specifying "cma_pernuma=size"
-	  on the kernel's command line.
+	  or set the node id and its size of CMA by specifying "numa_cma=
+	  <node>:size[,<node>:size]" on the kernel's command line.
 
 comment "Default contiguous memory area size:"
 
kernel/dma/contiguous.c +79 -22

167 lines not on this page · raw message

mm/cma.c +7 -3
index 4880f72102fa..da2967c6a223 100644
@@ -267,6 +267,9 @@ int __init cma_declare_contiguous_nid(phys_addr_t base,
 	if (alignment && !is_power_of_2(alignment))
 		return -EINVAL;
 
+	if (!IS_ENABLED(CONFIG_NUMA))
+		nid = NUMA_NO_NODE;
+
 	/* Sanitise input arguments. */
 	alignment = max_t(phys_addr_t, alignment, CMA_MIN_ALIGNMENT_BYTES);
 	if (fixed && base & (alignment - 1)) {
@@ -372,14 +375,15 @@ int __init cma_declare_contiguous_nid(phys_addr_t base,
 	if (ret)
 		goto free_mem;
 
-	pr_info("Reserved %ld MiB at %pa\n", (unsigned long)size / SZ_1M,
-		&base);
+	pr_info("Reserved %ld MiB at %pa on node %d\n", (unsigned long)size / SZ_1M,
+		&base, nid);
 	return 0;
 
 free_mem:
 	memblock_phys_free(base, size);
 err:
-	pr_err("Failed to reserve %ld MiB\n", (unsigned long)size / SZ_1M);
+	pr_err("Failed to reserve %ld MiB on node %d\n", (unsigned long)size / SZ_1M,
+	       nid);
 	return ret;
 }
 
Signature · 1 line
2.25.1
#1 lore raw

Thanks.

I'll pick this up for the dma-mapping tree with some minor style fixups.

author
#3 lore raw
Quoted 4 lines · Christoph Hellwig

It turns out this doesn't apply at all. Can you resend it against Linus' latest tree?

It's based on linux-next tree.

This patch should be after my other patch in linux-next tree. a960925a6b23("dma-contiguous: support per-numa CMA for all architectures").

#4 lore raw
Quoted 4 lines · Yajun Deng

It's based on linux-next tree.

This patch should be after my other patch in linux-next tree. a960925a6b23("dma-contiguous: support per-numa CMA for all architectures").

Where did this land? dma patches really should be going through the DMA tree..

#5 lore raw

V Thu, 20 Jul 2023 13:54:08 +0200
Christoph Hellwig <[email protected]> napsáno:

Quoted 7 lines · Yajun Deng
Quoted 4 lines

It's based on linux-next tree.

This patch should be after my other patch in linux-next tree. a960925a6b23("dma-contiguous: support per-numa CMA for all architectures").

Where did this land?

Well... in the linux-next tree:

https://www.kernel.org/doc/man-pages/linux-next.html

Quoted 1 line

dma patches really should be going through the DMA tree..

Indeed. The other patch was also sent to the iommu ML back in May. It's the thread where we were looking for Barry Song's current email address.

Petr T

author
#6 lore raw
Quoted 16 lines · Petr Tesařík

V Thu, 20 Jul 2023 13:54:08 +0200
Christoph Hellwig <[email protected]> napsáno:

Quoted 9 lines · Yajun Deng

It's based on linux-next tree.

This patch should be after my other patch in linux-next tree. a960925a6b23("dma-contiguous: support per-numa CMA for all architectures").

Where did this land?

Well... in the linux-next tree:

Quoted 9 lines

https://www.kernel.org/doc/man-pages/linux-next.html

Quoted 3 lines

dma patches really should be going through the DMA tree..

Indeed. The other patch was also sent to the iommu ML back in May. It's the thread where we were looking for Barry Song's current email address.

Thanks. This link shows what happened at that time:
https://lore.kernel.org/all/[email protected]/T/#u

Quoted 2 lines

Petr T

#7 lore raw
In reply to Petr Tesařík
Quoted 6 lines · Petr Tesařík

V Thu, 20 Jul 2023 13:54:08 +0200
Christoph Hellwig <[email protected]> napsáno:

Quoted 3 lines · Yajun Deng
Quoted 2 lines

It's based on linux-next tree.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This patch should be after my other patch in linux-next tree. a960925a6b23("dma-contiguous: support per-numa CMA for all architectures").

Where did this land?

Well... in the linux-next tree:

https://www.kernel.org/doc/man-pages/linux-next.html

Well, linux-next just pulls in maintainer trees. So nothing lands directly in linux-next. But it looks like Andrew already cleared up the confusion.

maintainer
#8 lore raw
In reply to Christoph Hellwig
Quoted 8 lines · On Thu, 20 Jul 2023 13:54:08 +0200 Christoph Hellwig <[email protected]> wrote:
Quoted 4 lines · Yajun Deng

It's based on linux-next tree.

This patch should be after my other patch in linux-next tree. a960925a6b23("dma-contiguous: support per-numa CMA for all architectures").

Where did this land? dma patches really should be going through the DMA tree..

It's in mm-unstable with a note "hch" :)

I'll drop it.

#9 lore raw
Quoted 6 lines · Andrew Morton
Quoted 2 lines

Where did this land? dma patches really should be going through the DMA tree..

It's in mm-unstable with a note "hch" :)

I'll drop it.

Ah. And looking at the patch is isn't even pure dma but also has a significant mm side. If you're fine with it I'll pick it up in the dma-mapping tree in addition to the follow up.

maintainer
#10 lore raw
Quoted 11 lines · On Fri, 21 Jul 2023 09:42:09 +0200 Christoph Hellwig <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 2 lines

Where did this land? dma patches really should be going through the DMA tree..

It's in mm-unstable with a note "hch" :)

I'll drop it.

Ah. And looking at the patch is isn't even pure dma but also has a significant mm side. If you're fine with it I'll pick it up in the dma-mapping tree in addition to the follow up.

Please do.