Skip to contentSkip to conversation

x86: don't unnecessarily call dma_alloc_from_contiguous()

PATCH In review Akinobu Mita 2 participants 5 messages Sep 28, 2014 – Sep 29, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 x86: don't unnecessarily call dma_alloc_from_contiguous()

If CONFIG_DMA_CMA is enabled, dma_generic_alloc_coherent() tries to allocate memory region by dma_alloc_from_contiguous() before trying to use alloc_pages().

This wastes CMA region by small DMA-coherent buffers which can be allocated by alloc_pages(). And it also causes performance degradation, as this is trying to drive _all_ dma mapping allocations through a _very_ small window, reported by Peter Hurley.

This fixes it by trying to allocate by alloc_pages() first in dma_generic_alloc_coherent() as dma_alloc_from_contiguous should be called only for huge allocation.

Cc Peter Hurley <[email protected]>
Cc Marek Szyprowski <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>
Cc David Woodhouse <[email protected]>
Cc Don Dutile <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Andi Kleen <andi-Vw/[email protected]>
Cc Yinghai Lu <[email protected]>
Cc iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
---
 arch/x86/kernel/pci-dma.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
arch/x86/kernel/pci-dma.c +6 -6
index a25e202..0402266 100644
@@ -99,20 +99,20 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size,
 
 	flag &= ~__GFP_ZERO;
 again:
-	page = NULL;
+	page = alloc_pages_node(dev_to_node(dev), flag | __GFP_NOWARN,
+				get_order(size));
 	/* CMA can be used only in the context which permits sleeping */
-	if (flag & __GFP_WAIT) {
+	if (!page && (flag & __GFP_WAIT)) {
 		page = dma_alloc_from_contiguous(dev, count, get_order(size));
 		if (page && page_to_phys(page) + size > dma_mask) {
 			dma_release_from_contiguous(dev, page, count);
 			page = NULL;
 		}
 	}
-	/* fallback */
-	if (!page)
-		page = alloc_pages_node(dev_to_node(dev), flag, get_order(size));
-	if (!page)
+	if (!page) {
+		warn_alloc_failed(flag, get_order(size), NULL);
 		return NULL;
+	}
 
 	addr = page_to_phys(page);
 	if (addr + size > dma_mask) {
Signature · 1 line
1.9.1
#1 lore raw

PATCH 2/2 intel-iommu: don't unnecessarily call dma_alloc_from_contiguous()

If CONFIG_DMA_CMA is enabled, intel_alloc_coherent() tries to allocate memory region by dma_alloc_from_contiguous() before trying to use alloc_pages().

This wastes CMA region by small DMA-coherent buffers which can be allocated by alloc_pages(). And it also causes performance degradation, as this is trying to drive _all_ dma mapping allocations through a _very_ small window, reported by Peter Hurley.

This fixes it by trying to allocate by alloc_pages() first in intel_alloc_coherent() as dma_alloc_from_contiguous should be called only for huge allocation.

Cc Peter Hurley <[email protected]>
Cc Marek Szyprowski <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>
Cc David Woodhouse <[email protected]>
Cc Don Dutile <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Andi Kleen <andi-Vw/[email protected]>
Cc Yinghai Lu <[email protected]>
Cc iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
---
 drivers/iommu/intel-iommu.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/iommu/intel-iommu.c +5 -4
index 3441615..f9794fa 100644
@@ -3250,7 +3250,8 @@ static void *intel_alloc_coherent(struct device *dev, size_t size,
 			flags |= GFP_DMA32;
 	}
 
-	if (flags & __GFP_WAIT) {
+	page = alloc_pages(flags | __GFP_NOWARN, order);
+	if (!page && (flags & __GFP_WAIT)) {
 		unsigned int count = size >> PAGE_SHIFT;
 
 		page = dma_alloc_from_contiguous(dev, count, order);
@@ -3261,10 +3262,10 @@ static void *intel_alloc_coherent(struct device *dev, size_t size,
 		}
 	}
 
-	if (!page)
-		page = alloc_pages(flags, order);
-	if (!page)
+	if (!page) {
+		warn_alloc_failed(flags, order, NULL);
 		return NULL;
+	}
 	memset(page_address(page), 0, size);
 
 	*dma_handle = __intel_map_single(dev, page_to_phys(page), size,
Signature · 1 line
1.9.1
#2 lore raw
In reply to Akinobu Mita
Quoted 36 lines · On Mon, 29 Sep 2014 00:52:03 +0900 Akinobu Mita <[email protected]> wrote:

If CONFIG_DMA_CMA is enabled, dma_generic_alloc_coherent() tries to allocate memory region by dma_alloc_from_contiguous() before trying to use alloc_pages().

This wastes CMA region by small DMA-coherent buffers which can be allocated by alloc_pages(). And it also causes performance degradation, as this is trying to drive _all_ dma mapping allocations through a _very_ small window, reported by Peter Hurley.

This fixes it by trying to allocate by alloc_pages() first in dma_generic_alloc_coherent() as dma_alloc_from_contiguous should be called only for huge allocation.

Cc Peter Hurley <[email protected]>
Cc Marek Szyprowski <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>
Cc David Woodhouse <[email protected]>
Cc Don Dutile <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Yinghai Lu <[email protected]>
---
 arch/x86/kernel/pci-dma.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
arch/x86/kernel/pci-dma.c +0 -0
index a25e202..0402266 100644

arch/x86/kernel/pci-dma.c void *dma_generic_alloc_coherent(struct device *dev, size_t size, · line 103

Quoted 6 lines
arch/x86/kernel/pci-dma.c +2 -1
@@ -99,20 +99,20 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size,
 
 	flag &= ~__GFP_ZERO;
 again:
-	page = NULL;
+	page = alloc_pages_node(dev_to_node(dev), flag | __GFP_NOWARN,
+				get_order(size));

Shouldn't you be doing this only in the case where order is relatively small? Like < PAGE_ALLOC_COSTLY_ORDER or so?

Quoted 20 lines
arch/x86/kernel/pci-dma.c +4 -5
 	/* CMA can be used only in the context which permits sleeping */
-	if (flag & __GFP_WAIT) {
+	if (!page && (flag & __GFP_WAIT)) {
 		page = dma_alloc_from_contiguous(dev, count, get_order(size));
 		if (page && page_to_phys(page) + size > dma_mask) {
 			dma_release_from_contiguous(dev, page, count);
 			page = NULL;
 		}
 	}
-	/* fallback */
-	if (!page)
-		page = alloc_pages_node(dev_to_node(dev), flag, get_order(size));
-	if (!page)
+	if (!page) {
+		warn_alloc_failed(flag, get_order(size), NULL);
 		return NULL;
+	}
 
 	addr = page_to_phys(page);
 	if (addr + size > dma_mask) {
#3 lore raw
In reply to Akinobu Mita
Quoted 36 lines · On Mon, 29 Sep 2014 00:52:03 +0900 Akinobu Mita <[email protected]> wrote:

If CONFIG_DMA_CMA is enabled, dma_generic_alloc_coherent() tries to allocate memory region by dma_alloc_from_contiguous() before trying to use alloc_pages().

This wastes CMA region by small DMA-coherent buffers which can be allocated by alloc_pages(). And it also causes performance degradation, as this is trying to drive _all_ dma mapping allocations through a _very_ small window, reported by Peter Hurley.

This fixes it by trying to allocate by alloc_pages() first in dma_generic_alloc_coherent() as dma_alloc_from_contiguous should be called only for huge allocation.

Cc Peter Hurley <[email protected]>
Cc Marek Szyprowski <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>
Cc David Woodhouse <[email protected]>
Cc Don Dutile <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Yinghai Lu <[email protected]>
---
 arch/x86/kernel/pci-dma.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
arch/x86/kernel/pci-dma.c +0 -0
index a25e202..0402266 100644

arch/x86/kernel/pci-dma.c void *dma_generic_alloc_coherent(struct device *dev, size_t size, · line 103

Quoted 6 lines
arch/x86/kernel/pci-dma.c +2 -1
@@ -99,20 +99,20 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size,
 
 	flag &= ~__GFP_ZERO;
 again:
-	page = NULL;
+	page = alloc_pages_node(dev_to_node(dev), flag | __GFP_NOWARN,
+				get_order(size));

Only try small allocs here, like when order < PAGE_ALLOC_COSTLY_ORDER ?

arch/x86/kernel/pci-dma.c void *dma_generic_alloc_coherent(struct device *dev, size_t size, · line 113

Quoted 12 lines
arch/x86/kernel/pci-dma.c +1 -4
 	/* CMA can be used only in the context which permits sleeping */
-	if (flag & __GFP_WAIT) {
+	if (!page && (flag & __GFP_WAIT)) {
 		page = dma_alloc_from_contiguous(dev, count, get_order(size));
 		if (page && page_to_phys(page) + size > dma_mask) {
 			dma_release_from_contiguous(dev, page, count);
 			page = NULL;
 		}
 	}
-	/* fallback */
-	if (!page)
-		page = alloc_pages_node(dev_to_node(dev), flag, get_order(size));

(I forgot to add this in my first reply). I think it should try for a small alloc without CMA first, then try CMA, and then this final fallback for larger allocs.

Quoted 8 lines
arch/x86/kernel/pci-dma.c +3 -1
-	if (!page)
+	if (!page) {
+		warn_alloc_failed(flag, get_order(size), NULL);
 		return NULL;
+	}
 
 	addr = page_to_phys(page);
 	if (addr + size > dma_mask) {
#4 lore raw

2014-09-29 5:45 GMT+09:00 Chuck Ebbert <[email protected]>:

Quoted 61 lines
Quoted 36 lines · On Mon, 29 Sep 2014 00:52:03 +0900 Akinobu Mita…

If CONFIG_DMA_CMA is enabled, dma_generic_alloc_coherent() tries to allocate memory region by dma_alloc_from_contiguous() before trying to use alloc_pages().

This wastes CMA region by small DMA-coherent buffers which can be allocated by alloc_pages(). And it also causes performance degradation, as this is trying to drive _all_ dma mapping allocations through a _very_ small window, reported by Peter Hurley.

This fixes it by trying to allocate by alloc_pages() first in dma_generic_alloc_coherent() as dma_alloc_from_contiguous should be called only for huge allocation.

Cc Peter Hurley <[email protected]>
Cc Marek Szyprowski <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>
Cc David Woodhouse <[email protected]>
Cc Don Dutile <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Andi Kleen <andi-Vw/[email protected]>
Cc Yinghai Lu <[email protected]>
Cc iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
---
 arch/x86/kernel/pci-dma.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
arch/x86/kernel/pci-dma.c +0 -0
index a25e202..0402266 100644

arch/x86/kernel/pci-dma.c void *dma_generic_alloc_coherent(struct device *dev, size_t size, · line 103

Quoted 6 lines
arch/x86/kernel/pci-dma.c +2 -1
@@ -99,20 +99,20 @@ void *dma_generic_alloc_coherent(struct device *dev, size_t size,
 
      flag &= ~__GFP_ZERO;
 again:
-     page = NULL;
+     page = alloc_pages_node(dev_to_node(dev), flag | __GFP_NOWARN,
+                             get_order(size));

Only try small allocs here, like when order < PAGE_ALLOC_COSTLY_ORDER ?

arch/x86/kernel/pci-dma.c void *dma_generic_alloc_coherent(struct device *dev, size_t size, · line 113

Quoted 12 lines
arch/x86/kernel/pci-dma.c +1 -4
      /* CMA can be used only in the context which permits sleeping */
-     if (flag & __GFP_WAIT) {
+     if (!page && (flag & __GFP_WAIT)) {
              page = dma_alloc_from_contiguous(dev, count, get_order(size));
              if (page && page_to_phys(page) + size > dma_mask) {
                      dma_release_from_contiguous(dev, page, count);
                      page = NULL;
              }
      }
-     /* fallback */
-     if (!page)
-             page = alloc_pages_node(dev_to_node(dev), flag, get_order(size));
Quoted 3 lines

(I forgot to add this in my first reply). I think it should try for a small alloc without CMA first, then try CMA, and then this final fallback for larger allocs.

I'm concerned with the performance problem reported by Peter Hurley. This could be a solution, but I would like to hear Peter's opinion.

For now, I prefer the solution by this patch because it gives less impact on CONFIG_DMA_CMA enabled. But it can be improved later on.