ARM: use phys_addr_t for DMA zone sizes

From: Rob Herring <[email protected]>

In order to specify a DMA zone size of 4GB on LPAE systems, the sizes need to be 64-bit. So make machine_desc.dma_zone_size and arm_dma_zone_size be phys_addr_t instead of unsigned long.

---
 arch/arm/include/asm/mach/arch.h | 2 +-
 arch/arm/mm/init.c               | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

index 441efc4..d91b168 100644
@@ -35,7 +35,7 @@ struct machine_desc {
 	unsigned int		nr_irqs;	/* number of IRQs */
 
 #ifdef CONFIG_ZONE_DMA
-	unsigned long		dma_zone_size;	/* size of DMA-able area */
+	phys_addr_t		dma_zone_size;	/* size of DMA-able area */
 #endif
 
 	unsigned int		video_start;	/* start of video RAM	*/

index 15225d8..c0bb66e 100644
@@ -207,7 +207,7 @@ static void __init arm_bootmem_init(unsigned long start_pfn,
 
 #ifdef CONFIG_ZONE_DMA
 
-unsigned long arm_dma_zone_size __read_mostly;
+phys_addr_t arm_dma_zone_size __read_mostly;
 EXPORT_SYMBOL(arm_dma_zone_size);
 
 /*

1.8.1.2

From: Rob Herring <[email protected]>

LPAE systems may have 64-bit capable DMA, so dma_addr_t needs to be 64-bit.

---
 arch/arm/mm/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 6cacdc8..9f6f3d7 100644
@@ -625,7 +625,7 @@ config ARCH_PHYS_ADDR_T_64BIT
 	def_bool ARM_LPAE
 
 config ARCH_DMA_ADDR_T_64BIT
-	bool
+	def_bool ARM_LPAE
 
 config ARM_THUMB
 	bool "Support Thumb user binaries" if !CPU_THUMBONLY

1.8.1.2

The question I put to you here is: any can your DMA controllers produce more than 32-bits of address?

Yes, physical memory may be mapped at addresses greater than 4GB phys. That doesn't mean that dma_addr_t also becomes 64-bit.

dma_addr_t is the bus address - the address you program into the DMA controller. If you're only ever writing it into a 32-bit address register then you can only address 32-bits of memory, even though that may be in the physical address range of 4GB-8GB. In such cases where that applies to all DMA controllers in the system, dma_addr_t should still be 32-bit.

This is where it's broken to think that DMA addresses are the same as physical addresses. They aren't.

Yes. SATA is 64-bit. And the ones that are 32-bit only have an IOMMU in front of them.

Understood, but it's not that clear once IOMMUs are in the mix. While IOMMU drivers will return 32-bit addresses at the DMA API level, I'd think internally they will need 64-bit addresses.

I'm fine with just selecting this in ARCH_HIGHBANK, but the machine kconfig selects are getting rather long. We'll be cleaning those up at some point if we keep selecting everything at the machine level.

Rob

Given that this breaks PL08x, I think you need to drop this change. We need the root problem fixed, as I have done in my dma-masks patches, rather than endlessly plastering around this.

If we don't solve the root problem first, this issue is just going to get really crappy with workaround on top of more workarounds, and it's going to be impossible to undo all that crap later.

From: Rob Herring <[email protected]>

Some devices are restricted to 32-bit DMA. Thus the platform dma_zone_size needs to be set. Otherwise dma-mapping code is complaining, e.g.

calxedaxgmac fff50000.ethernet: coherent DMA mask 0xffffffff is smaller than system GFP_DMA mask 0xffffffffffffffff

---
 arch/arm/mach-highbank/Kconfig    | 1 +
 arch/arm/mach-highbank/highbank.c | 3 +++
 2 files changed, 4 insertions(+)

index cd9fcb1..efe4961 100644
@@ -18,3 +18,4 @@ config ARCH_HIGHBANK
 	select PL320_MBOX
 	select SPARSE_IRQ
 	select USE_OF
+	select ZONE_DMA if ARM_LPAE

index 8881579..1894dcf 100644
@@ -176,6 +176,9 @@ static const char *highbank_match[] __initconst = {
 };
 
 DT_MACHINE_START(HIGHBANK, "Highbank")
+#ifdef CONFIG_ZONE_DMA
+	.dma_zone_size	= (4ULL * SZ_1G),
+#endif
 	.smp		= smp_ops(highbank_smp_ops),
 	.init_irq	= highbank_init_irq,
 	.init_time	= highbank_timer_init,

1.8.1.2

Note that the patches I posted last week will fix that in a much nicer way, as we will only error if we actually have sufficient memory that the condition becomes problematical.

The other problem here is that we need to get better separation between what's a bus address and what's a physical address.

But in the Midway case it is problematic as we do have more than 4GB with some masters being 32-bit. DMA zone is still the right way to limit allocations to less than 4GB dma address, right?

Right, but for Midway they are actually the same. There is no h/w translation of 32-bit bus addresses to a >4GB physical address other than with an IOMMU.

Rob

From: Rob Herring <[email protected]>

On Midway, the first 4G of memory starts at 0 and the rest of memory (4GB+) starts at 0x2_00000000, so we need to enable pfn_valid checks by selecting ARCH_HAS_HOLES_MEMORYMODEL.

---
 arch/arm/mach-highbank/Kconfig | 1 +
 1 file changed, 1 insertion(+)

index efe4961..8b2948f 100644
@@ -1,6 +1,7 @@
 config ARCH_HIGHBANK
 	bool "Calxeda ECX-1000/2000 (Highbank/Midway)" if ARCH_MULTI_V7
 	select ARCH_HAS_CPUFREQ
+	select ARCH_HAS_HOLES_MEMORYMODEL
 	select ARCH_HAS_OPP
 	select ARCH_WANT_OPTIONAL_GPIOLIB
 	select ARM_AMBA

1.8.1.2

From: Rob Herring <[email protected]>

Ensure necessary errata work-arounds are always enabled for Highbank and Midway platforms. Highbank requires 764369 and 764369. Midway requires 798181, but only the first half of the work-around (no IPI). Support for skipping the IPI is introduced in another commit.

---
 arch/arm/mach-highbank/Kconfig | 3 +++
 1 file changed, 3 insertions(+)

index 8b2948f..386835e 100644
@@ -5,6 +5,9 @@ config ARCH_HIGHBANK
 	select ARCH_HAS_OPP
 	select ARCH_WANT_OPTIONAL_GPIOLIB
 	select ARM_AMBA
+	select ARM_ERRATA_764369
+	select ARM_ERRATA_775420
+	select ARM_ERRATA_798181
 	select ARM_GIC
 	select ARM_TIMER_SP804
 	select CACHE_L2X0

1.8.1.2