Save single pages from CMA area
Subsystems (4)
v2 cover Save single pages from CMA area
This series of patches try to save single pages from CMA area bypassing all CMA single page alloctions and allocating normal pages instead, as all addresses within one single page are contiguous.
We had once applied the PATCH-5 but reverted it as actually not all the callers handled the fallback allocations. Per Robin's suggestion, let's stuff alloc_pages()/free_page() fallbacks to those callers before having PATCH-5.
Changlogv1->v2:
* PATCH-2: Initialized page pointer to NULL
Nicolin Chen (5):
ARM: dma-mapping: Add fallback normal page allocations
dma-remap: Run alloc_pages() on failure
iommu: amd_iommu: Add fallback normal page allocations
arm64: dma-mapping: Add fallback normal page allocations
dma-contiguous: Do not allocate a single page from CMA area
arch/arm/mm/dma-mapping.c | 13 ++++++++++--- arch/arm64/mm/dma-mapping.c | 19 ++++++++++++------- drivers/iommu/amd_iommu.c | 3 +++ kernel/dma/contiguous.c | 22 +++++++++++++++++++--- kernel/dma/remap.c | 4 ++-- 5 files changed, 46 insertions(+), 15 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/5 ARM: dma-mapping: Add fallback normal page allocations
The CMA allocation will skip allocations of single pages to save CMA resource. This requires its callers to rebound those page allocations from normal area. So this patch adds fallback routines.
--- arch/arm/mm/dma-mapping.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
arch/arm/mm/dma-mapping.c +10 -3
@@ -589,6 +589,8 @@ static void *__alloc_from_contiguous(struct device *dev, size_t size,void *ptr =NULL ; page =dma_alloc_from_contiguous (dev, count, order, gfp & __GFP_NOWARN); +if (!page) + page =alloc_pages (gfp, order);if (!page)return NULL ;
@@ -600,7 +602,8 @@ static void *__alloc_from_contiguous(struct device *dev, size_t size,if (PageHighMem (page)) { ptr =__dma_alloc_remap (page, size,GFP_KERNEL , prot, caller);if (!ptr) { -dma_release_from_contiguous (dev, page, count); +if (!dma_release_from_contiguous (dev, page, count)) +__free_pages (page,get_order (size));return NULL ; } }else {
@@ -622,7 +625,8 @@ static void __free_from_contiguous(struct device *dev, struct page *page,else __dma_remap (page, size,PAGE_KERNEL ); } -dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT ); +if (!dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT )) +__free_pages (page,get_order (size)); }static inline pgprot_t __get_dma_pgprot (unsigned long attrs,pgprot_t prot)
@@ -1295,6 +1299,8 @@ static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, page =dma_alloc_from_contiguous (dev, count, order, gfp & __GFP_NOWARN); +if (!page) + page =alloc_pages (gfp, order);if (!page) goto error;
@@ -1369,7 +1375,8 @@ static int __iommu_free_buffer(struct device *dev, struct page **pages,int i;if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) { -dma_release_from_contiguous (dev, pages[0 ], count); +if (!dma_release_from_contiguous (dev, pages[0 ], count)) +__free_pages (page[0 ],get_order (size)); }else {for (i =0 ; i < count; i++)if (pages[i])
Signature · 1 line
2.17.1
Re: [PATCH v2 RFC/RFT 1/5] ARM: dma-mapping: Add fallback normal page allocations
arch/arm/mm/dma-mapping.c static void *__alloc_from_contiguous(struct device *dev, size_t size, · line 593
Quoted 3 lines · Nicolin Chen
arch/arm/mm/dma-mapping.c +2
-0page =dma_alloc_from_contiguous (dev, count, order, gfp & __GFP_NOWARN); +if (!page) + page =alloc_pages (gfp, order);
We have this fallback in most callers already. And with me adding it to the dma-iommu code in one series, and you to arm here I think we really need to take a step back and think of a better way to handle this, and the general mess that dma_alloc_from_contiguous.
So what about:
(1) change the dma_alloc_from_contiguous prototype to be:
struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp);
that is: calculate order and count internally, pass the full gfp_t and mask it internally, and drop the pointless from in the name. I'd also use the oppurtunity to forbid a NULL dev argument and opencode those uses.
(2) handle the alloc_pages fallback internally. Note that we should use alloc_pages_node as we do in dma-direct.
arch/arm/mm/dma-mapping.c static void *__alloc_from_contiguous(struct device *dev, size_t size, · line 606
Quoted 2 lines
arch/arm/mm/dma-mapping.c +2
-0+if (!dma_release_from_contiguous (dev, page, count)) +__free_pages (page,get_order (size));
Same for dma_release_from_contiguous - drop the _from, pass the actual size, and handle the free_pages fallback.
Re: [PATCH v2 RFC/RFT 1/5] ARM: dma-mapping: Add fallback normal page allocations
Quoted 2 lines · Christoph Hellwig
I'd also use the oppurtunity to forbid a NULL dev argument and opencode those uses.
Actually, looking at your last patch again the NULL argument might still fit in ok, so maybe we should keep it.
Re: [PATCH v2 RFC/RFT 1/5] ARM: dma-mapping: Add fallback normal page allocations
In reply to Christoph HellwigHi Christoph,
Quoted 11 lines · Christoph Hellwig
arch/arm/mm/dma-mapping.c static void *__alloc_from_contiguous(struct device *dev, size_t size, · line 593
Quoted 3 lines · Nicolin Chen
arch/arm/mm/dma-mapping.c +2
-0page =dma_alloc_from_contiguous (dev, count, order, gfp & __GFP_NOWARN); +if (!page) + page =alloc_pages (gfp, order);We have this fallback in most callers already. And with me adding it to the dma-iommu code in one series, and you to arm here I think we really need to take a step back and think of a better way to handle this, and the general mess that
dma_alloc_from_contiguous.So what about:
Thanks for the suggestion!
Quoted 11 lines
(1) change the
dma_alloc_from_contiguousprototype to be:struct page *dma_alloc_contiguous(struct device *dev,
size_tsize,gfp_tgfp);that is: calculate order and count internally, pass the full
gfp_tand mask it internally, and drop the pointless from in the name. I'd also use the oppurtunity to forbid a NULL dev argument and opencode those uses.(2) handle the
alloc_pagesfallback internally. Note that we should usealloc_pages_nodeas we do in dma-direct.
I feel it's similar to my previous set, which did most of these internally except the renaming part. But Catalin had a concern that some platforms might have limits on CMA range [1]. Will it be still okay to do the fallback internally?
[1: https://www.spinics.net/lists/arm-kernel/msg714295.html ]
Re: [PATCH v2 RFC/RFT 1/5] ARM: dma-mapping: Add fallback normal page allocations
Quoted 6 lines · Nicolin Chen
I feel it's similar to my previous set, which did most of these internally except the renaming part. But Catalin had a concern that some platforms might have limits on CMA range [1]. Will it be still okay to do the fallback internally?
[1: https://www.spinics.net/lists/arm-kernel/msg714295.html ]
Catalins statement is correct, but I don't see how it applies to your patch. Your patch just ensures that the fallback we have in most callers is uniformly applied everywhere. The non-iommu callers will still need to select a specific zone and/or retry just the page allocator with other flags if the CMA (or fallback) page doesn't match what they need. dma-direct does this correctly and I think the arm32 allocator does as well, although it is a bit hard to follow sometimes.
Re: [PATCH v2 RFC/RFT 1/5] ARM: dma-mapping: Add fallback normal page allocations
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Nicolin Chen
I feel it's similar to my previous set, which did most of these internally except the renaming part. But Catalin had a concern that some platforms might have limits on CMA range [1]. Will it be still okay to do the fallback internally?
[1: https://www.spinics.net/lists/arm-kernel/msg714295.html ]
Quoted 8 lines
Catalins statement is correct, but I don't see how it applies to your patch. Your patch just ensures that the fallback we have in most callers is uniformly applied everywhere. The non-iommu callers will still need to select a specific zone and/or retry just the page allocator with other flags if the CMA (or fallback) page doesn't match what they need. dma-direct does this correctly and I think the arm32 allocator does as well, although it is a bit hard to follow sometimes.
Okay. I will revise and submit the patches then. I think we can still discuss on this topic once we have the changes.
Thanks
Catalin Marinas
maintainerRe: [PATCH v2 RFC/RFT 1/5] ARM: dma-mapping: Add fallback normal page allocations
In reply to Christoph Hellwig(catching up on email)
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Nicolin Chen
I feel it's similar to my previous set, which did most of these internally except the renaming part. But Catalin had a concern that some platforms might have limits on CMA range [1]. Will it be still okay to do the fallback internally?
[1: https://www.spinics.net/lists/arm-kernel/msg714295.html ]
Quoted 8 lines
Catalins statement is correct, but I don't see how it applies to your patch. Your patch just ensures that the fallback we have in most callers is uniformly applied everywhere. The non-iommu callers will still need to select a specific zone and/or retry just the page allocator with other flags if the CMA (or fallback) page doesn't match what they need. dma-direct does this correctly and I think the arm32 allocator does as well, although it is a bit hard to follow sometimes.
My reading of the arm32 __dma_alloc() is that if the conditions are right for the CMA allocator (allows blocking) and there is a default CMA area or a per-device one, the call ends up in cma_alloc() without any fallback if such allocation fails. Whether this is on purpose, I'm not entirely sure. There are a couple of arm32 SoCs which call dma_declare_contiguous() or dma_contiguous_reserve_area() and a few DT files describing a specific CMA range (e.g. arch/arm/boot/dts/sun5i.dtsi with a comment that address must be kept in the lower 256MB).
If ZONE_DMA is set up correctly so that cma_alloc() is (or can be made) interchangeable with alloc_pages(GFP_DMA) from a device DMA capability perspective , I think it should be fine to have such fallback.
Signature · 1 line
Catalin
Re: [PATCH v2 RFC/RFT 1/5] ARM: dma-mapping: Add fallback normal page allocations
Quoted 9 lines · Catalin Marinas
My reading of the arm32
__dma_alloc()is that if the conditions are right for the CMA allocator (allows blocking) and there is a default CMA area or a per-device one, the call ends up incma_alloc()without any fallback if such allocation fails. Whether this is on purpose, I'm not entirely sure. There are a couple of arm32 SoCs which calldma_declare_contiguous()ordma_contiguous_reserve_area()and a few DT files describing a specific CMA range (e.g.arch/arm/boot/dts/sun5i.dtsiwith a comment that address must be kept in the lower 256MB).
Quoted 3 lines
If
ZONE_DMAis set up correctly so thatcma_alloc()is (or can be made) interchangeable withalloc_pages(GFP_DMA)from a device DMA capability perspective , I think it should be fine to have such fallback.
Indeed. I missed arm32 being different from everyone else, but we already addresses that in another thread. Sorry for misleading everyone.
Re: [PATCH v2 RFC/RFT 1/5] ARM: dma-mapping: Add fallback normal page allocations
In reply to Christoph HellwigQuoted 5 lines · Christoph Hellwig
arch/arm/mm/dma-mapping.c static void *__alloc_from_contiguous(struct device *dev, size_t size, · line 606
Quoted 2 lines
arch/arm/mm/dma-mapping.c +2
-0+if (!dma_release_from_contiguous (dev, page, count)) +__free_pages (page,get_order (size));Same for
dma_release_from_contiguous- drop the_from, pass the actual size, and handle thefree_pagesfallback.
What do you think of dma_free_contiguous() instead? I feel "free" is a bit more commonly used (in dma-mapping.h) and it's shorter.
Thanks
Re: [PATCH v2 RFC/RFT 1/5] ARM: dma-mapping: Add fallback normal page allocations
Quoted 2 lines · Nicolin Chen
What do you think of
dma_free_contiguous()instead? I feel "free" is a bit more commonly used (indma-mapping.h) and it's shorter.
Yeah, that sounds good.
PATCH v2 2/5 dma-remap: Run alloc_pages() on failure
In reply to Nicolin ChenThe CMA allocation will skip allocations of single pages to save CMA resource. This requires its callers to rebound those page allocations from normal area.
So this patch moves the alloc_pages() call to the fallback routines.
---
Changlogv1->v2:
* PATCH-2: Initialized page pointer to NULL
kernel/dma/remap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/dma/remap.c +2 -2
@@ -109,14 +109,14 @@ int __init dma_atomic_pool_init(gfp_t gfp, pgprot_t prot) {unsigned int pool_size_order =get_order (atomic_pool_size);unsigned long nr_pages = atomic_pool_size >>PAGE_SHIFT ; -struct page *page; +struct page *page =NULL ;void *addr;int ret;if (dev_get_cma_area (NULL )) page =dma_alloc_from_contiguous (NULL , nr_pages, pool_size_order, false); -else +if (!page) page =alloc_pages (gfp, pool_size_order);if (!page) goto out;
Signature · 1 line
2.17.1
PATCH v2 3/5 iommu: amd_iommu: Add fallback normal page allocations
In reply to Nicolin ChenThe CMA allocation will skip allocations of single pages to save CMA resource. This requires its callers to rebound those page allocations from normal area. So this patch adds fallback routines.
Note: amd_iommu driver uses dma_alloc_from_contiguous() as a fallback
allocation and uses alloc_pages() as its first round allocation. This's in reverse order than other callers. So the alloc_pages() added by this change becomes a second fallback, though it likely won't succeed since the alloc_pages() has failed once.
--- drivers/iommu/amd_iommu.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/iommu/amd_iommu.c +3 -0
@@ -2701,6 +2701,9 @@ static void *alloc_coherent(struct device *dev, size_t size, page =dma_alloc_from_contiguous (dev, size >>PAGE_SHIFT ,get_order (size), flag & __GFP_NOWARN); +if (!page) + page =alloc_pages (flag | __GFP_NOWARN, +get_order (size));if (!page)return NULL ; }
Signature · 1 line
2.17.1
PATCH v2 4/5 arm64: dma-mapping: Add fallback normal page allocations
In reply to Nicolin ChenThe cma allocation will skip allocations of single pages to save CMA resource. This requires its callers to rebound those page allocations from normal area. So this patch adds fallback routines.
--- arch/arm64/mm/dma-mapping.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
arch/arm64/mm/dma-mapping.c +12 -7
@@ -156,17 +156,20 @@ static void *__iommu_alloc_attrs(struct device *dev, size_t size, } }else if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) {pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs); +unsigned long count =PAGE_ALIGN (size) >>PAGE_SHIFT ;struct page *page; - page =dma_alloc_from_contiguous (dev, size >>PAGE_SHIFT , -get_order (size), gfp & __GFP_NOWARN); + page =dma_alloc_from_contiguous (dev, count,get_order (size), + gfp & __GFP_NOWARN); +if (!page) + page =alloc_pages (gfp,get_order (size));if (!page)return NULL ; *handle =iommu_dma_map_page (dev, page,0 , iosize, ioprot);if (*handle ==DMA_MAPPING_ERROR ) { -dma_release_from_contiguous (dev, page, - size >>PAGE_SHIFT ); +if (!dma_release_from_contiguous (dev, page, count)) +__free_pages (page,get_order (size));return NULL ; } addr = dma_common_contiguous_remap(page, size, VM_USERMAP,
@@ -178,8 +181,8 @@ static void *__iommu_alloc_attrs(struct device *dev, size_t size,memset (addr,0 , size); }else {iommu_dma_unmap_page (dev, *handle, iosize,0 , attrs); -dma_release_from_contiguous (dev, page, - size >>PAGE_SHIFT ); +if (!dma_release_from_contiguous (dev, page, count)) +__free_pages (page,get_order (size)); } }else {pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs);
@@ -201,6 +204,7 @@ static void *__iommu_alloc_attrs(struct device *dev, size_t size,static void __iommu_free_attrs (struct device *dev,size_t size,void *cpu_addr,dma_addr_t handle,unsigned long attrs) { +unsigned long count =PAGE_ALIGN (size) >>PAGE_SHIFT ;size_t iosize = size; size =PAGE_ALIGN (size);
@@ -222,7 +226,8 @@ static void __iommu_free_attrs(struct device *dev, size_t size, void *cpu_addr,struct page *page =vmalloc_to_page (cpu_addr);iommu_dma_unmap_page (dev, handle, iosize,0 , attrs); -dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT ); +if (!dma_release_from_contiguous (dev, page, count)) +__free_pages (page,get_order (size));dma_common_free_remap (cpu_addr, size,VM_USERMAP ); }else if (is_vmalloc_addr (cpu_addr)){struct vm_struct *area =find_vm_area (cpu_addr);
Signature · 1 line
2.17.1
PATCH v2 5/5 dma-contiguous: Do not allocate a single page from CMA area
In reply to Nicolin ChenThe addresses within a single page are always contiguous, so it's not so necessary to always allocate one single page from CMA area. Since the CMA area has a limited predefined size of space, it may run out of space in heavy use cases, where there might be quite a lot CMA pages being allocated for single pages.
However, there is also a concern that a device might care where a page comes from -- it might expect the page from CMA area and act differently if the page doesn't.
This patch tries to skip of one-page size allocations and returns NULL so as to let callers allocate normal pages unless the device has its own CMA area. This would save resources from the CMA area for more CMA allocations. And it'd also reduce CMA fragmentations resulted from trivial allocations.
--- kernel/dma/contiguous.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)
kernel/dma/contiguous.c +19 -3
@@ -186,16 +186,32 @@ int __init dma_contiguous_reserve_area(phys_addr_t size, phys_addr_t base,* * This function allocates memory buffer for specified device. It uses * device specific contiguous memory area if available or the default -* global one. Requires architecture specific dev_get_cma_area() helper -* function. +* global one. +* +* However, it skips one-page size of allocations from the global area. +* As the addresses within one page are always contiguous, so there is +* no need to waste CMA pages for that kind; it also helps reduce the +* fragmentations in the CMA area. So a caller should be the rebounder +* in such case to allocate a normal page upon NULL return value. +* +* Requires architecture specific dev_get_cma_area() helper function. */ struct page *dma_alloc_from_contiguous (struct device *dev,size_t count,unsigned int align,bool no_warn) { +struct cma *cma; +if (align >CONFIG_CMA_ALIGNMENT ) align =CONFIG_CMA_ALIGNMENT ; -return cma_alloc (dev_get_cma_area (dev), count, align, no_warn); +if (dev && dev->cma_area) + cma = dev->cma_area; +else if (count >1 ) + cma = dma_contiguous_default_area; +else +return NULL ; + +return cma_alloc (cma, count, align, no_warn); }/**
Signature · 1 line
2.17.1
Quoted 8 lines · Nicolin Chen
This series of patches try to save single pages from CMA area bypassing all CMA single page alloctions and allocating normal pages instead, as all addresses within one single page are contiguous.
We had once applied the PATCH-5 but reverted it as actually not all the callers handled the fallback allocations. Per Robin's suggestion, let's stuff alloc_pages()/free_page() fallbacks to those callers before having PATCH-5.
Given the problems this has caused so far I'd like to see a good explanation of why this optimization is so important that all the churn is even worth it..
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Nicolin Chen
This series of patches try to save single pages from CMA area bypassing all CMA single page alloctions and allocating normal pages instead, as all addresses within one single page are contiguous.
Quoted 8 lines
Quoted 4 lines
We had once applied the PATCH-5 but reverted it as actually not all the callers handled the fallback allocations. Per Robin's suggestion, let's stuff alloc_pages()/free_page() fallbacks to those callers before having PATCH-5.
Given the problems this has caused so far I'd like to see a good explanation of why this optimization is so important that all the churn is even worth it..
With certain downstream user cases, we had run into a CMA exhaustion situation, and this was one of the changes that eased the problem. I have all the reasoning in the PATCH-5 commit message. And what Robin tested can also justify for it:
https://lore.kernel.org/patchwork/patch/1004934/#1190139
Thanks