Skip to contentSkip to conversation

dma: Fix max PFN arithmetic overflow on 32 bit systems

PATCH Superseded Alexander Dahl 2 participants 7 messages Mar 2, 2020 – May 5, 2020 x86

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dma: Fix max PFN arithmetic overflow on 32 bit systems

For ARCH=x86 (32 bit) when you set CONFIG_IOMMU_INTEL since c5a5dc4cbbf4 ("iommu/vt-d: Don't switch off swiotlb if bounce page is used") there's a dependency on CONFIG_SWIOTLB, which was not necessarily active before.

The init code for swiotlb in 'pci_swiotlb_detect_4gb()' compares something against MAX_DMA32_PFN to decide if it should be active. However that define suffers from an arithmetic overflow since 1b7e03ef7570 ("x86, NUMA: Enable emulation on 32bit too") when it was first made visible to x86_32.

The effect is at boot time 64 MiB (default size) were allocated for bounce buffers now, which is a noticeable amount of memory on small systems. We noticed this effect on the fli4l Linux distribution when migrating from kernel v4.19 (LTS) to v5.4 (LTS) on boards like pcengines ALIX 2D3 with 256 MiB memory for example:

Linux version 5.4.22 (buildroot@buildroot) (gcc version 7.3.0 (Buildroot 2018.02.8)) #1 SMP Mon Nov 26 23:40:00 CET 2018
…
Memory: 183484K/261756K available (4594K kernel code, 393K rwdata, 1660K rodata, 536K init, 456K bss , 78272K reserved, 0K cma-reserved, 0K highmem) …
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
software IO TLB: mapped [mem 0x0bb78000-0x0fb78000] (64MB)

The initial analysis and the suggested fix was done by user 'sourcejedi' at stackoverflow and explicitly marked as GPLv2 for inclusion in the Linux kernel:

https://unix.stackexchange.com/a/520525/50007

---
We tested this in qemu and on real hardware with fli4l on top of v5.4, v5.5, and v5.6-rc kernels, but only as far as the reserved memory goes. The patch itself is based on v5.6-rc3 (IIRC).

A quick grep over the kernel code showed me this define MAX_DMA32_PFN is used in other places as well. I would appreciate feedback on this, because I can not oversee all side effects this might have?!

Thanks again to Alan who proposed the fix, and for his permission to send it upstream.

Greets
Alex

---
 arch/x86/include/asm/dma.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/dma.h +1 -1
index 00f7cf45e699..e25514eca8d6 100644
@@ -74,7 +74,7 @@
 #define MAX_DMA_PFN   ((16UL * 1024 * 1024) >> PAGE_SHIFT)
 
 /* 4GB broken PCI/AGP hardware bus master zone */
-#define MAX_DMA32_PFN ((4UL * 1024 * 1024 * 1024) >> PAGE_SHIFT)
+#define MAX_DMA32_PFN (4UL * ((1024 * 1024 * 1024) >> PAGE_SHIFT))
 
 #ifdef CONFIG_X86_32
 /* The maximum address that we can perform a DMA transfer to on this platform */
Signature · 1 line
2.20.1
#1 lore raw

Hei hei,

gentle ping on this patch from two weeks ago. Did anyone have time to look into it? Did I miss someone in Cc or sent it to the wrong lists maybe?

Quoted 29 lines · Alexander Dahl

For ARCH=x86 (32 bit) when you set CONFIG_IOMMU_INTEL since c5a5dc4cbbf4 ("iommu/vt-d: Don't switch off swiotlb if bounce page is used") there's a dependency on CONFIG_SWIOTLB, which was not necessarily active before.

The init code for swiotlb in 'pci_swiotlb_detect_4gb()' compares something against MAX_DMA32_PFN to decide if it should be active. However that define suffers from an arithmetic overflow since 1b7e03ef7570 ("x86, NUMA: Enable emulation on 32bit too") when it was first made visible to x86_32.

The effect is at boot time 64 MiB (default size) were allocated for bounce buffers now, which is a noticeable amount of memory on small systems. We noticed this effect on the fli4l Linux distribution when migrating from kernel v4.19 (LTS) to v5.4 (LTS) on boards like pcengines ALIX 2D3 with 256 MiB memory for example:

Linux version 5.4.22 (buildroot@buildroot) (gcc version 7.3.0 (Buildroot 2018.02.8)) #1 SMP Mon Nov 26 23:40:00 CET 2018
…
Memory: 183484K/261756K available (4594K kernel code, 393K rwdata, 1660K rodata, 536K init, 456K bss , 78272K reserved, 0K cma-reserved, 0K highmem) …
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
software IO TLB: mapped [mem 0x0bb78000-0x0fb78000] (64MB)

The initial analysis and the suggested fix was done by user 'sourcejedi' at stackoverflow and explicitly marked as GPLv2 for inclusion in the Linux kernel:

https://unix.stackexchange.com/a/520525/50007

Quoted 8 lines

---
We tested this in qemu and on real hardware with fli4l on top of v5.4, v5.5, and v5.6-rc kernels, but only as far as the reserved memory goes. The patch itself is based on v5.6-rc3 (IIRC).

We had no complaints of our fli4l users, since we applied this patch to our distribution kernels.

Thanks & greets
Alex

Quoted 29 lines

A quick grep over the kernel code showed me this define MAX_DMA32_PFN is used in other places as well. I would appreciate feedback on this, because I can not oversee all side effects this might have?!

Thanks again to Alan who proposed the fix, and for his permission to send it upstream.

Greets
Alex

---
 arch/x86/include/asm/dma.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/dma.h +1 -1
index 00f7cf45e699..e25514eca8d6 100644
@@ -74,7 +74,7 @@
 #define MAX_DMA_PFN   ((16UL * 1024 * 1024) >> PAGE_SHIFT)
 
 /* 4GB broken PCI/AGP hardware bus master zone */
-#define MAX_DMA32_PFN ((4UL * 1024 * 1024 * 1024) >> PAGE_SHIFT)
+#define MAX_DMA32_PFN (4UL * ((1024 * 1024 * 1024) >> PAGE_SHIFT))
 
 #ifdef CONFIG_X86_32
 /* The maximum address that we can perform a DMA transfer to on this platform */
Signature · 1 line
2.20.1
Signature · 4 lines
/"\ ASCII RIBBON | »With the first link, the chain is forged. The first
\ / CAMPAIGN     | speech censured, the first thought forbidden, the
 X  AGAINST      | first freedom denied, chains us all irrevocably.«
/ \ HTML MAIL    | (Jean-Luc Picard, quoting Judge Aaron Satie)
#2 lore raw
In reply to Alexander Dahl
Quoted 56 lines · Alexander Dahl

For ARCH=x86 (32 bit) when you set CONFIG_IOMMU_INTEL since c5a5dc4cbbf4 ("iommu/vt-d: Don't switch off swiotlb if bounce page is used") there's a dependency on CONFIG_SWIOTLB, which was not necessarily active before.

The init code for swiotlb in 'pci_swiotlb_detect_4gb()' compares something against MAX_DMA32_PFN to decide if it should be active. However that define suffers from an arithmetic overflow since 1b7e03ef7570 ("x86, NUMA: Enable emulation on 32bit too") when it was first made visible to x86_32.

The effect is at boot time 64 MiB (default size) were allocated for bounce buffers now, which is a noticeable amount of memory on small systems. We noticed this effect on the fli4l Linux distribution when migrating from kernel v4.19 (LTS) to v5.4 (LTS) on boards like pcengines ALIX 2D3 with 256 MiB memory for example:

   Linux version 5.4.22 (buildroot@buildroot) (gcc version 7.3.0 (Buildroot 2018.02.8)) #1 SMP Mon Nov 26 23:40:00 CET 2018
   …
   Memory: 183484K/261756K available (4594K kernel code, 393K rwdata, 1660K rodata, 536K init, 456K bss , 78272K reserved, 0K cma-reserved, 0K highmem)
   …
   PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
   software IO TLB: mapped [mem 0x0bb78000-0x0fb78000] (64MB)

The initial analysis and the suggested fix was done by user 'sourcejedi' at stackoverflow and explicitly marked as GPLv2 for inclusion in the Linux kernel:

   https://unix.stackexchange.com/a/520525/50007

---
We tested this in qemu and on real hardware with fli4l on top of v5.4, v5.5, and v5.6-rc kernels, but only as far as the reserved memory goes. The patch itself is based on v5.6-rc3 (IIRC).

A quick grep over the kernel code showed me this define MAX_DMA32_PFN is used in other places as well. I would appreciate feedback on this, because I can not oversee all side effects this might have?!

Thanks again to Alan who proposed the fix, and for his permission to send it upstream.

Greets
Alex

---
  arch/x86/include/asm/dma.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/dma.h +0 -0
index 00f7cf45e699..e25514eca8d6 100644

arch/x86/include/asm/dma.h line 77

Quoted 5 lines
arch/x86/include/asm/dma.h +1 -1
@@ -74,7 +74,7 @@
  #define MAX_DMA_PFN   ((16UL * 1024 * 1024) >> PAGE_SHIFT)
 
  /* 4GB broken PCI/AGP hardware bus master zone */
-#define MAX_DMA32_PFN ((4UL * 1024 * 1024 * 1024) >> PAGE_SHIFT)
+#define MAX_DMA32_PFN (4UL * ((1024 * 1024 * 1024) >> PAGE_SHIFT))

FWIW, wouldn't s/UL/ULL/ in the original expression suffice? Failing that, rather than awkward parenthesis trickery it might be clearer to just copy the one from arch/mips/include/asm/dma.h.

Robin.

Quoted 4 lines
arch/x86/include/asm/dma.h +0 -0
  #ifdef CONFIG_X86_32
  /* The maximum address that we can perform a DMA transfer to on this platform */
 
#3 lore raw

Hello Robin,

Quoted 5 lines · Robin Murphy
Quoted 4 lines · Alexander Dahl
---
  arch/x86/include/asm/dma.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 13 lines

arch/x86/include/asm/dma.h line 76

Quoted 9 lines
arch/x86/include/asm/dma.h +1 -1
index 00f7cf45e699..e25514eca8d6 100644
@@ -74,7 +74,7 @@
  #define MAX_DMA_PFN   ((16UL * 1024 * 1024) >> PAGE_SHIFT)
  /* 4GB broken PCI/AGP hardware bus master zone */
-#define MAX_DMA32_PFN ((4UL * 1024 * 1024 * 1024) >> PAGE_SHIFT)
+#define MAX_DMA32_PFN (4UL * ((1024 * 1024 * 1024) >> PAGE_SHIFT))

FWIW, wouldn't s/UL/ULL/ in the original expression suffice? Failing that, rather than awkward parenthesis trickery it might be clearer to just copy the one from arch/mips/include/asm/dma.h.

Both of your suggestions yield the correct result, and at least for me both look easier to understand than what Alan proposed in the first place.

I would opt for the variant which is already in arch/mips, because it avoids 64 bit calculation, is most obvious in intent in my eyes, and we have the same calculation twice then instead of two variants.

Thanks for your review, I'll send a v2. :-)

Greets
Alex

Signature · 4 lines
/"\ ASCII RIBBON | »With the first link, the chain is forged. The first
\ / CAMPAIGN     | speech censured, the first thought forbidden, the
 X  AGAINST      | first freedom denied, chains us all irrevocably.«
/ \ HTML MAIL    | (Jean-Luc Picard, quoting Judge Aaron Satie)
#4 lore raw

PATCH v2 1/1 dma: Fix max PFN arithmetic overflow on 32 bit systems

For ARCH=x86 (32 bit) when you set CONFIG_IOMMU_INTEL since c5a5dc4cbbf4 ("iommu/vt-d: Don't switch off swiotlb if bounce page is used") there's a dependency on CONFIG_SWIOTLB, which was not necessarily active before.

The init code for swiotlb in 'pci_swiotlb_detect_4gb()' compares something against MAX_DMA32_PFN to decide if it should be active. However that define suffers from an arithmetic overflow since 1b7e03ef7570 ("x86, NUMA: Enable emulation on 32bit too") when it was first made visible to x86_32.

The effect is at boot time 64 MiB (default size) were allocated for bounce buffers now, which is a noticeable amount of memory on small systems. We noticed this effect on the fli4l Linux distribution when migrating from kernel v4.19 (LTS) to v5.4 (LTS) on boards like pcengines ALIX 2D3 with 256 MiB memory for example:

Linux version 5.4.22 (buildroot@buildroot) (gcc version 7.3.0 (Buildroot 2018.02.8)) #1 SMP Mon Nov 26 23:40:00 CET 2018
…
Memory: 183484K/261756K available (4594K kernel code, 393K rwdata, 1660K rodata, 536K init, 456K bss , 78272K reserved, 0K cma-reserved, 0K highmem) …
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
software IO TLB: mapped [mem 0x0bb78000-0x0fb78000] (64MB)

The initial analysis and the suggested fix was done by user 'sourcejedi' at stackoverflow and explicitly marked as GPLv2 for inclusion in the Linux kernel:

https://unix.stackexchange.com/a/520525/50007

The actual calculation however is the same as for arch/mips now as suggested by Robin Murphy.

---

Notes:
v1 -> v2:
- use the same calculation as with arch/mips (Robin Murphy)

 arch/x86/include/asm/dma.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/dma.h +1 -1
index 00f7cf45e699..8e95aa4b0d17 100644
@@ -74,7 +74,7 @@
 #define MAX_DMA_PFN   ((16UL * 1024 * 1024) >> PAGE_SHIFT)
 
 /* 4GB broken PCI/AGP hardware bus master zone */
-#define MAX_DMA32_PFN ((4UL * 1024 * 1024 * 1024) >> PAGE_SHIFT)
+#define MAX_DMA32_PFN (1UL << (32 - PAGE_SHIFT))
 
 #ifdef CONFIG_X86_32
 /* The maximum address that we can perform a DMA transfer to on this platform */
Signature · 1 line
2.20.1
#5 lore raw

Hello,

now after v5.7-rc1 is out, I would kindly ask, if anyone had time to review this one line patch? Is anything wrong with that fix?

(I added the current fli4l kernel package maintainer Florian to Cc to let him know I'm still having an eye on this.)

Greets
Alex

Quoted 66 lines · Alexander Dahl

For ARCH=x86 (32 bit) when you set CONFIG_IOMMU_INTEL since c5a5dc4cbbf4 ("iommu/vt-d: Don't switch off swiotlb if bounce page is used") there's a dependency on CONFIG_SWIOTLB, which was not necessarily active before.

The init code for swiotlb in 'pci_swiotlb_detect_4gb()' compares something against MAX_DMA32_PFN to decide if it should be active. However that define suffers from an arithmetic overflow since 1b7e03ef7570 ("x86, NUMA: Enable emulation on 32bit too") when it was first made visible to x86_32.

The effect is at boot time 64 MiB (default size) were allocated for bounce buffers now, which is a noticeable amount of memory on small systems. We noticed this effect on the fli4l Linux distribution when migrating from kernel v4.19 (LTS) to v5.4 (LTS) on boards like pcengines ALIX 2D3 with 256 MiB memory for example:

Linux version 5.4.22 (buildroot@buildroot) (gcc version 7.3.0 (Buildroot 2018.02.8)) #1 SMP Mon Nov 26 23:40:00 CET 2018
…
Memory: 183484K/261756K available (4594K kernel code, 393K rwdata, 1660K rodata, 536K init, 456K bss , 78272K reserved, 0K cma-reserved, 0K highmem) …
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
software IO TLB: mapped [mem 0x0bb78000-0x0fb78000] (64MB)

The initial analysis and the suggested fix was done by user 'sourcejedi' at stackoverflow and explicitly marked as GPLv2 for inclusion in the Linux kernel:

https://unix.stackexchange.com/a/520525/50007

The actual calculation however is the same as for arch/mips now as suggested by Robin Murphy.

---

Notes:
v1 -> v2:
- use the same calculation as with arch/mips (Robin Murphy)

 arch/x86/include/asm/dma.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/dma.h +1 -1
index 00f7cf45e699..8e95aa4b0d17 100644
@@ -74,7 +74,7 @@
 #define MAX_DMA_PFN   ((16UL * 1024 * 1024) >> PAGE_SHIFT)
 
 /* 4GB broken PCI/AGP hardware bus master zone */
-#define MAX_DMA32_PFN ((4UL * 1024 * 1024 * 1024) >> PAGE_SHIFT)
+#define MAX_DMA32_PFN (1UL << (32 - PAGE_SHIFT))
 
 #ifdef CONFIG_X86_32
 /* The maximum address that we can perform a DMA transfer to on this platform */
Signature · 6 lines
2.20.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Signature · 4 lines
/"\ ASCII RIBBON | »With the first link, the chain is forged. The first
\ / CAMPAIGN     | speech censured, the first thought forbidden, the
 X  AGAINST      | first freedom denied, chains us all irrevocably.«
/ \ HTML MAIL    | (Jean-Luc Picard, quoting Judge Aaron Satie)

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#6 lore raw

Hei hei,

I would like to kindly ask about the status of this patch.

Quoted 2 lines · Alexander Dahl

now after v5.7-rc1 is out, I would kindly ask, if anyone had time to review this one line patch? Is anything wrong with that fix?

Did it maybe not reach the right maintainers? I used scripts/get_maintainer.pl to get my recipient list.

Quoted 60 lines

(I added the current fli4l kernel package maintainer Florian to Cc to let him know I'm still having an eye on this.)

Greets
Alex

Quoted 53 lines · Alexander Dahl

For ARCH=x86 (32 bit) when you set CONFIG_IOMMU_INTEL since c5a5dc4cbbf4 ("iommu/vt-d: Don't switch off swiotlb if bounce page is used") there's a dependency on CONFIG_SWIOTLB, which was not necessarily active before.

The init code for swiotlb in 'pci_swiotlb_detect_4gb()' compares something against MAX_DMA32_PFN to decide if it should be active. However that define suffers from an arithmetic overflow since 1b7e03ef7570 ("x86, NUMA: Enable emulation on 32bit too") when it was first made visible to x86_32.

The effect is at boot time 64 MiB (default size) were allocated for bounce buffers now, which is a noticeable amount of memory on small systems. We noticed this effect on the fli4l Linux distribution when migrating from kernel v4.19 (LTS) to v5.4 (LTS) on boards like pcengines ALIX 2D3 with 256 MiB memory for example:

Linux version 5.4.22 (buildroot@buildroot) (gcc version 7.3.0 (Buildroot 2018.02.8)) #1 SMP Mon Nov 26 23:40:00 CET 2018
…
Memory: 183484K/261756K available (4594K kernel code, 393K rwdata, 1660K rodata, 536K init, 456K bss , 78272K reserved, 0K cma-reserved, 0K highmem) …
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
software IO TLB: mapped [mem 0x0bb78000-0x0fb78000] (64MB)

The initial analysis and the suggested fix was done by user 'sourcejedi' at stackoverflow and explicitly marked as GPLv2 for inclusion in the Linux kernel:

https://unix.stackexchange.com/a/520525/50007

The actual calculation however is the same as for arch/mips now as suggested by Robin Murphy.

---

Notes:
v1 -> v2:
- use the same calculation as with arch/mips (Robin Murphy)

 arch/x86/include/asm/dma.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/dma.h +0 -0
index 00f7cf45e699..8e95aa4b0d17 100644
@@ -74,7 +74,7 @@
 #define MAX_DMA_PFN   ((16UL * 1024 * 1024) >> PAGE_SHIFT)
 
Quoted 8 lines
Quoted 8 lines
arch/x86/include/asm/dma.h +1 -1
 /* 4GB broken PCI/AGP hardware bus master zone */
-#define MAX_DMA32_PFN ((4UL * 1024 * 1024 * 1024) >> PAGE_SHIFT)
+#define MAX_DMA32_PFN (1UL << (32 - PAGE_SHIFT))
 
 #ifdef CONFIG_X86_32
 /* The maximum address that we can perform a DMA transfer to on this platform */
Signature · 1 line
2.20.1

Greets
Alex

Signature · 4 lines
/"\ ASCII RIBBON | »With the first link, the chain is forged. The first
\ / CAMPAIGN     | speech censured, the first thought forbidden, the
 X  AGAINST      | first freedom denied, chains us all irrevocably.«
/ \ HTML MAIL    | (Jean-Luc Picard, quoting Judge Aaron Satie)