x86: don't unnecessarily call dma_alloc_from_contiguous()
PATCH 1/2 x86: don't unnecessarily call dma_alloc_from_contiguous()
If CONFIG_DMA_CMA is enabled, dma_generic_alloc_coherent() tries to allocate memory region by dma_alloc_from_contiguous() before trying to use alloc_pages().
This wastes CMA region by small DMA-coherent buffers which can be allocated by alloc_pages(). And it also causes performance degradation, as this is trying to drive _all_ dma mapping allocations through a _very_ small window, reported by Peter Hurley.
This fixes it by trying to allocate by alloc_pages() first in dma_generic_alloc_coherent() as dma_alloc_from_contiguous should be called only for huge allocation.
--- arch/x86/kernel/pci-dma.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
arch/x86/kernel/pci-dma.c +6 -6
@@ -99,20 +99,20 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size, flag &= ~__GFP_ZERO; again: - page =NULL ; + page =alloc_pages_node (dev_to_node (dev), flag | __GFP_NOWARN, +get_order (size));/* CMA can be used only in the context which permits sleeping */ -if (flag & __GFP_WAIT) { +if (!page && (flag & __GFP_WAIT)) { page =dma_alloc_from_contiguous (dev, count,get_order (size));if (page &&page_to_phys (page) + size > dma_mask) {dma_release_from_contiguous (dev, page, count); page =NULL ; } } -/* fallback */ -if (!page) - page =alloc_pages_node (dev_to_node (dev), flag,get_order (size)); -if (!page) +if (!page) { +warn_alloc_failed (flag,get_order (size),NULL );return NULL ; + } addr =page_to_phys (page);if (addr + size > dma_mask) {
Signature · 1 line
1.9.1
PATCH 2/2 intel-iommu: don't unnecessarily call dma_alloc_from_contiguous()
If CONFIG_DMA_CMA is enabled, intel_alloc_coherent() tries to allocate memory region by dma_alloc_from_contiguous() before trying to use alloc_pages().
This wastes CMA region by small DMA-coherent buffers which can be allocated by alloc_pages(). And it also causes performance degradation, as this is trying to drive _all_ dma mapping allocations through a _very_ small window, reported by Peter Hurley.
This fixes it by trying to allocate by alloc_pages() first in intel_alloc_coherent() as dma_alloc_from_contiguous should be called only for huge allocation.
--- drivers/iommu/intel-iommu.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/iommu/intel-iommu.c +5 -4
@@ -3250,7 +3250,8 @@ static void *intel_alloc_coherent(struct device *dev, size_t size, flags |=GFP_DMA32 ; } -if (flags & __GFP_WAIT) { + page =alloc_pages (flags | __GFP_NOWARN, order); +if (!page && (flags & __GFP_WAIT)) {unsigned int count = size >>PAGE_SHIFT ; page =dma_alloc_from_contiguous (dev, count, order);
@@ -3261,10 +3262,10 @@ static void *intel_alloc_coherent(struct device *dev, size_t size, } } -if (!page) - page =alloc_pages (flags, order); -if (!page) +if (!page) { +warn_alloc_failed (flags, order,NULL );return NULL ; + }memset (page_address (page),0 , size); *dma_handle = __intel_map_single(dev, page_to_phys(page), size,
Signature · 1 line
1.9.1
Quoted 36 lines · On Mon, 29 Sep 2014 00:52:03 +0900 Akinobu Mita <[email protected]> wrote:
If
CONFIG_DMA_CMAis enabled,dma_generic_alloc_coherent()tries to allocate memory region bydma_alloc_from_contiguous()before trying to usealloc_pages().This wastes CMA region by small DMA-coherent buffers which can be allocated by
alloc_pages(). And it also causes performance degradation, as this is trying to drive _all_ dma mapping allocations through a _very_ small window, reported by Peter Hurley.This fixes it by trying to allocate by
alloc_pages()first indma_generic_alloc_coherent()asdma_alloc_from_contiguousshould be called only for huge allocation.Signed-off-by Akinobu Mita <[email protected]>Reported-by Peter Hurley <[email protected]>Cc Peter Hurley <[email protected]>Cc Marek Szyprowski <[email protected]>Cc Konrad Rzeszutek Wilk <[email protected]>Cc David Woodhouse <[email protected]>Cc Don Dutile <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Andi Kleen <[email protected]>Cc Yinghai Lu <[email protected]>--- arch/x86/kernel/pci-dma.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)arch/x86/kernel/pci-dma.c +0
-0
arch/x86/kernel/pci-dma.c void *dma_generic_alloc_coherent(struct device *dev, size_t size, · line 103
Quoted 6 lines
arch/x86/kernel/pci-dma.c +2
-1@@ -99,20 +99,20 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size, flag &= ~__GFP_ZERO; again: - page =NULL ; + page =alloc_pages_node (dev_to_node (dev), flag | __GFP_NOWARN, +get_order (size));
Shouldn't you be doing this only in the case where order is relatively small? Like < PAGE_ALLOC_COSTLY_ORDER or so?
Quoted 20 lines
arch/x86/kernel/pci-dma.c +4
-5/* CMA can be used only in the context which permits sleeping */ -if (flag & __GFP_WAIT) { +if (!page && (flag & __GFP_WAIT)) { page =dma_alloc_from_contiguous (dev, count,get_order (size));if (page &&page_to_phys (page) + size > dma_mask) {dma_release_from_contiguous (dev, page, count); page =NULL ; } } -/* fallback */ -if (!page) - page =alloc_pages_node (dev_to_node (dev), flag,get_order (size)); -if (!page) +if (!page) { +warn_alloc_failed (flag,get_order (size),NULL );return NULL ; + } addr =page_to_phys (page);if (addr + size > dma_mask) {
Quoted 36 lines · On Mon, 29 Sep 2014 00:52:03 +0900 Akinobu Mita <[email protected]> wrote:
If
CONFIG_DMA_CMAis enabled,dma_generic_alloc_coherent()tries to allocate memory region bydma_alloc_from_contiguous()before trying to usealloc_pages().This wastes CMA region by small DMA-coherent buffers which can be allocated by
alloc_pages(). And it also causes performance degradation, as this is trying to drive _all_ dma mapping allocations through a _very_ small window, reported by Peter Hurley.This fixes it by trying to allocate by
alloc_pages()first indma_generic_alloc_coherent()asdma_alloc_from_contiguousshould be called only for huge allocation.Signed-off-by Akinobu Mita <[email protected]>Reported-by Peter Hurley <[email protected]>Cc Peter Hurley <[email protected]>Cc Marek Szyprowski <[email protected]>Cc Konrad Rzeszutek Wilk <[email protected]>Cc David Woodhouse <[email protected]>Cc Don Dutile <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Andi Kleen <[email protected]>Cc Yinghai Lu <[email protected]>--- arch/x86/kernel/pci-dma.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)arch/x86/kernel/pci-dma.c +0
-0
arch/x86/kernel/pci-dma.c void *dma_generic_alloc_coherent(struct device *dev, size_t size, · line 103
Quoted 6 lines
arch/x86/kernel/pci-dma.c +2
-1@@ -99,20 +99,20 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size, flag &= ~__GFP_ZERO; again: - page =NULL ; + page =alloc_pages_node (dev_to_node (dev), flag | __GFP_NOWARN, +get_order (size));
Only try small allocs here, like when order < PAGE_ALLOC_COSTLY_ORDER ?
arch/x86/kernel/pci-dma.c void *dma_generic_alloc_coherent(struct device *dev, size_t size, · line 113
Quoted 12 lines
arch/x86/kernel/pci-dma.c +1
-4/* CMA can be used only in the context which permits sleeping */ -if (flag & __GFP_WAIT) { +if (!page && (flag & __GFP_WAIT)) { page =dma_alloc_from_contiguous (dev, count,get_order (size));if (page &&page_to_phys (page) + size > dma_mask) {dma_release_from_contiguous (dev, page, count); page =NULL ; } } -/* fallback */ -if (!page) - page =alloc_pages_node (dev_to_node (dev), flag,get_order (size));
(I forgot to add this in my first reply). I think it should try for a small alloc without CMA first, then try CMA, and then this final fallback for larger allocs.
Quoted 8 lines
arch/x86/kernel/pci-dma.c +3
-1-if (!page) +if (!page) { +warn_alloc_failed (flag,get_order (size),NULL );return NULL ; + } addr =page_to_phys (page);if (addr + size > dma_mask) {
2014-09-29 5:45 GMT+09:00 Chuck Ebbert <[email protected]>:
Quoted 61 lines
Quoted 36 lines · On Mon, 29 Sep 2014 00:52:03 +0900 Akinobu Mita…
If
CONFIG_DMA_CMAis enabled,dma_generic_alloc_coherent()tries to allocate memory region bydma_alloc_from_contiguous()before trying to usealloc_pages().This wastes CMA region by small DMA-coherent buffers which can be allocated by
alloc_pages(). And it also causes performance degradation, as this is trying to drive _all_ dma mapping allocations through a _very_ small window, reported by Peter Hurley.This fixes it by trying to allocate by
alloc_pages()first indma_generic_alloc_coherent()asdma_alloc_from_contiguousshould be called only for huge allocation.Signed-off-by Akinobu Mita <[email protected]>Reported-by Peter Hurley <[email protected]>Cc Peter Hurley <[email protected]>Cc Marek Szyprowski <[email protected]>Cc Konrad Rzeszutek Wilk <[email protected]>Cc David Woodhouse <[email protected]>Cc Don Dutile <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Andi Kleen <andi-Vw/[email protected]>Cc Yinghai Lu <[email protected]>Cc iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org--- arch/x86/kernel/pci-dma.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)arch/x86/kernel/pci-dma.c +0
-0arch/x86/kernel/pci-dma.c void *dma_generic_alloc_coherent(struct device *dev, size_t size, · line 103
Quoted 6 lines
arch/x86/kernel/pci-dma.c +2
-1@@ -99,20 +99,20 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size, flag &= ~__GFP_ZERO; again: - page =NULL ; + page =alloc_pages_node (dev_to_node (dev), flag | __GFP_NOWARN, +get_order (size));Only try small allocs here, like when order <
PAGE_ALLOC_COSTLY_ORDER?arch/x86/kernel/pci-dma.c void *dma_generic_alloc_coherent(struct device *dev, size_t size, · line 113
Quoted 12 lines
arch/x86/kernel/pci-dma.c +1
-4/* CMA can be used only in the context which permits sleeping */ -if (flag & __GFP_WAIT) { +if (!page && (flag & __GFP_WAIT)) { page =dma_alloc_from_contiguous (dev, count,get_order (size));if (page &&page_to_phys (page) + size > dma_mask) {dma_release_from_contiguous (dev, page, count); page =NULL ; } } -/* fallback */ -if (!page) - page =alloc_pages_node (dev_to_node (dev), flag,get_order (size));
Quoted 3 lines
(I forgot to add this in my first reply). I think it should try for a small alloc without CMA first, then try CMA, and then this final fallback for larger allocs.
I'm concerned with the performance problem reported by Peter Hurley. This could be a solution, but I would like to hear Peter's opinion.
For now, I prefer the solution by this patch because it gives less impact on CONFIG_DMA_CMA enabled. But it can be improved later on.