Skip to contentSkip to conversation

dra7: sata: Fix SATA with CONFIG_ARM_LPAE enabled

PATCH In review Roger Quadros 4 participants 8 messages Mar 4, 2020 – Mar 6, 2020 Device tree bindings

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dra7: sata: Fix SATA with CONFIG_ARM_LPAE enabled

Even though the TRM says that SATA IP has 36 address bits wired in the SoC, we see bus errors whenever any address greater than 32-bit is given to the controller.

This happens on dra7-EVM with 4G of RAM with CONFIG_ARM_LPAE=y.

As a workaround we limit the DMA address range to 32-bits for SATA.

Cc Christoph Hellwig <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Rob Herring <[email protected]>

---

NOTE: Currently ARM dma-mapping code doesn't account for devices bus_dma_limit. This is fixed in [1].

[1] https://lkml.org/lkml/2020/2/18/712

 arch/arm/boot/dts/dra7.dtsi | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
arch/arm/boot/dts/dra7.dtsi +16 -9
index d78b684e7fca..895462c22d1c 100644
@@ -642,15 +642,22 @@
 		};
 
 		/* OCP2SCP3 */
-		sata: sata@4a141100 {
-			compatible = "snps,dwc-ahci";
-			reg = <0x4a140000 0x1100>, <0x4a141100 0x7>;
-			interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
-			phys = <&sata_phy>;
-			phy-names = "sata-phy";
-			clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
-			ti,hwmods = "sata";
-			ports-implemented = <0x1>;
+		sata_aux_bus {
+			#address-cells = <1>;
+			#size-cells = <2>;
+			compatible = "simple-bus";
+			ranges = <0x0 0x4a140000 0x0 0x1200>;
+			dma-ranges = <0x0 0x0 0x1 0x00000000>;
+			sata: sata@4a141100 {
+				compatible = "snps,dwc-ahci";
+				reg = <0x0 0x0 0x1100>, <0x1100 0x0 0x7>;
+				interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
+				phys = <&sata_phy>;
+				phy-names = "sata-phy";
+				clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
+				ti,hwmods = "sata";
+				ports-implemented = <0x1>;
+			};
 		};
 
 		/* OCP2SCP1 */
Signature · 2 lines
Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki.
Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki
#1 lore raw
Quoted 14 lines · Roger Quadros

Even though the TRM says that SATA IP has 36 address bits wired in the SoC, we see bus errors whenever any address greater than 32-bit is given to the controller.

This happens on dra7-EVM with 4G of RAM with CONFIG_ARM_LPAE=y.

As a workaround we limit the DMA address range to 32-bits for SATA.

Cc Christoph Hellwig <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Rob Herring <[email protected]>
Quoted 6 lines

---

NOTE: Currently ARM dma-mapping code doesn't account for devices bus_dma_limit. This is fixed in [1].

[1] https://lkml.org/lkml/2020/2/18/712

So is this dts patch safe to apply without the series above?

And should this dts patch be applied as a fix or can it wait until the merge window?

Regards,

Tony

Quoted 43 lines
 arch/arm/boot/dts/dra7.dtsi | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
arch/arm/boot/dts/dra7.dtsi +16 -9
index d78b684e7fca..895462c22d1c 100644
@@ -642,15 +642,22 @@
 		};
 
 		/* OCP2SCP3 */
-		sata: sata@4a141100 {
-			compatible = "snps,dwc-ahci";
-			reg = <0x4a140000 0x1100>, <0x4a141100 0x7>;
-			interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
-			phys = <&sata_phy>;
-			phy-names = "sata-phy";
-			clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
-			ti,hwmods = "sata";
-			ports-implemented = <0x1>;
+		sata_aux_bus {
+			#address-cells = <1>;
+			#size-cells = <2>;
+			compatible = "simple-bus";
+			ranges = <0x0 0x4a140000 0x0 0x1200>;
+			dma-ranges = <0x0 0x0 0x1 0x00000000>;
+			sata: sata@4a141100 {
+				compatible = "snps,dwc-ahci";
+				reg = <0x0 0x0 0x1100>, <0x1100 0x0 0x7>;
+				interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
+				phys = <&sata_phy>;
+				phy-names = "sata-phy";
+				clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
+				ti,hwmods = "sata";
+				ports-implemented = <0x1>;
+			};
 		};
 
 		/* OCP2SCP1 */
Signature · 2 lines
Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki.
Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki
#2 lore raw

Tony, Christoph,

Quoted 17 lines · Tony Lindgren
Quoted 16 lines · Roger Quadros

Even though the TRM says that SATA IP has 36 address bits wired in the SoC, we see bus errors whenever any address greater than 32-bit is given to the controller.

This happens on dra7-EVM with 4G of RAM with CONFIG_ARM_LPAE=y.

As a workaround we limit the DMA address range to 32-bits for SATA.

Cc Christoph Hellwig <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Rob Herring <[email protected]>

---

Quoted 7 lines
Quoted 4 lines

NOTE: Currently ARM dma-mapping code doesn't account for devices bus_dma_limit. This is fixed in [1].

[1] https://lkml.org/lkml/2020/2/18/712

So is this dts patch safe to apply without the series above?

Yes. To my surprise this patch fixes the SATA issue even without [1]. Without this patch dev->bus_dma_limit was being set to 0 and with the patch it is being set to 0xffffffff.

Quoted 2 lines

And should this dts patch be applied as a fix or can it wait until the merge window?

I think we should mark it for stable and apply it right away as fix for v5.6.

cheers,
-roger

Quoted 48 lines

Regards,

Tony

Quoted 43 lines
  arch/arm/boot/dts/dra7.dtsi | 25 ++++++++++++++++---------
  1 file changed, 16 insertions(+), 9 deletions(-)
arch/arm/boot/dts/dra7.dtsi +16 -9
index d78b684e7fca..895462c22d1c 100644
@@ -642,15 +642,22 @@
  		};
 
  		/* OCP2SCP3 */
-		sata: sata@4a141100 {
-			compatible = "snps,dwc-ahci";
-			reg = <0x4a140000 0x1100>, <0x4a141100 0x7>;
-			interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
-			phys = <&sata_phy>;
-			phy-names = "sata-phy";
-			clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
-			ti,hwmods = "sata";
-			ports-implemented = <0x1>;
+		sata_aux_bus {
+			#address-cells = <1>;
+			#size-cells = <2>;
+			compatible = "simple-bus";
+			ranges = <0x0 0x4a140000 0x0 0x1200>;
+			dma-ranges = <0x0 0x0 0x1 0x00000000>;
+			sata: sata@4a141100 {
+				compatible = "snps,dwc-ahci";
+				reg = <0x0 0x0 0x1100>, <0x1100 0x0 0x7>;
+				interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
+				phys = <&sata_phy>;
+				phy-names = "sata-phy";
+				clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
+				ti,hwmods = "sata";
+				ports-implemented = <0x1>;
+			};
  		};
 
  		/* OCP2SCP1 */
Signature · 2 lines
Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki.
Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki
Signature · 2 lines
Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki.
Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki
#3 lore raw
Quoted 23 lines · Roger Quadros

Tony, Christoph,

Quoted 20 lines · Tony Lindgren
Quoted 19 lines · Roger Quadros

Even though the TRM says that SATA IP has 36 address bits wired in the SoC, we see bus errors whenever any address greater than 32-bit is given to the controller.

This happens on dra7-EVM with 4G of RAM with CONFIG_ARM_LPAE=y.

As a workaround we limit the DMA address range to 32-bits for SATA.

Cc Christoph Hellwig <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Rob Herring <[email protected]>

---

NOTE: Currently ARM dma-mapping code doesn't account for devices bus_dma_limit. This is fixed in [1].

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

[1] https://lkml.org/lkml/2020/2/18/712

So is this dts patch safe to apply without the series above?

Yes. To my surprise this patch fixes the SATA issue even without [1]. Without this patch dev->bus_dma_limit was being set to 0 and with the patch it is being set to 0xffffffff.

Right - LPAE configs should be getting dma-direct ops since 5.3, which already respect the limit. For non-LPAE configs it means you go from being broken from having no limit at all, to having a limit set but not respected, which is still equally broken, but no *more* so than before.

Robin.

Quoted 56 lines
Quoted 2 lines

And should this dts patch be applied as a fix or can it wait until the merge window?

I think we should mark it for stable and apply it right away as fix for v5.6.

cheers,
-roger

Quoted 46 lines

Regards,

Tony

Quoted 41 lines

  arch/arm/boot/dts/dra7.dtsi | 25 ++++++++++++++++---------

  1 file changed, 16 insertions(+), 9 deletions(-)
arch/arm/boot/dts/dra7.dtsi +16 -10
index d78b684e7fca..895462c22d1c 100644
@@ -642,15 +642,22 @@
           };
           /* OCP2SCP3 */
-        sata: sata@4a141100 {
-            compatible = "snps,dwc-ahci";
-            reg = <0x4a140000 0x1100>, <0x4a141100 0x7>;
-            interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
-            phys = <&sata_phy>;
-            phy-names = "sata-phy";
-            clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
-            ti,hwmods = "sata";
-            ports-implemented = <0x1>;
+        sata_aux_bus {
+            #address-cells = <1>;
+            #size-cells = <2>;
+            compatible = "simple-bus";
+            ranges = <0x0 0x4a140000 0x0 0x1200>;
+            dma-ranges = <0x0 0x0 0x1 0x00000000>;
+            sata: sata@4a141100 {
+                compatible = "snps,dwc-ahci";
+                reg = <0x0 0x0 0x1100>, <0x1100 0x0 0x7>;
+                interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
+                phys = <&sata_phy>;
+                phy-names = "sata-phy";
+                clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
+                ti,hwmods = "sata";
+                ports-implemented = <0x1>;
+            };
           };
           /* OCP2SCP1 */
-- 

Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki

#4 lore raw
Quoted 4 lines · Robin Murphy

Right - LPAE configs should be getting dma-direct ops since 5.3, which already respect the limit. For non-LPAE configs it means you go from being broken from having no limit at all, to having a limit set but not respected, which is still equally broken, but no *more* so than before.

Actually, dma-direct doesn't automatically respect the bus limit. For various architectures including arm it uses the arch_dma_alloc callout, which for arm just uses the existing arm coherent allocator.

Thay being said I'm still looking for reviewers for the arm dma coherent allocator bus limit fix..

#5 lore raw
In reply to Roger Quadros
Quoted 3 lines · Roger Quadros

Even though the TRM says that SATA IP has 36 address bits wired in the SoC, we see bus errors whenever any address greater than 32-bit is given to the controller.

Actually, is it really just SATA? I pulled up a couple of DRA7xx TRMs out of curiosity - thanks for having such easy-to-access documentation by the way :) - and they both give me a clear impression that the entire L3_MAIN interconnect is limited to 32-bit addresses and thus pretty much all the DMA masters should only be able to touch the lower 2GB of DRAM. Especially the bit that explicitly says "This is a high address range (Q8 – Q15) that requires an address greater than 32 bits. This space is visible only for the MPU Subsystem."

Is it in fact the case that the SATA driver happens to be the only one to set a >32-bit DMA mask on your system?

Robin.

Quoted 57 lines

This happens on dra7-EVM with 4G of RAM with CONFIG_ARM_LPAE=y.

As a workaround we limit the DMA address range to 32-bits for SATA.

Cc Christoph Hellwig <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Rob Herring <[email protected]>

---

NOTE: Currently ARM dma-mapping code doesn't account for devices bus_dma_limit. This is fixed in [1].

[1] https://lkml.org/lkml/2020/2/18/712

  arch/arm/boot/dts/dra7.dtsi | 25 ++++++++++++++++---------
  1 file changed, 16 insertions(+), 9 deletions(-)
arch/arm/boot/dts/dra7.dtsi +16 -9
index d78b684e7fca..895462c22d1c 100644
@@ -642,15 +642,22 @@
  		};
 
  		/* OCP2SCP3 */
-		sata: sata@4a141100 {
-			compatible = "snps,dwc-ahci";
-			reg = <0x4a140000 0x1100>, <0x4a141100 0x7>;
-			interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
-			phys = <&sata_phy>;
-			phy-names = "sata-phy";
-			clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
-			ti,hwmods = "sata";
-			ports-implemented = <0x1>;
+		sata_aux_bus {
+			#address-cells = <1>;
+			#size-cells = <2>;
+			compatible = "simple-bus";
+			ranges = <0x0 0x4a140000 0x0 0x1200>;
+			dma-ranges = <0x0 0x0 0x1 0x00000000>;
+			sata: sata@4a141100 {
+				compatible = "snps,dwc-ahci";
+				reg = <0x0 0x0 0x1100>, <0x1100 0x0 0x7>;
+				interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
+				phys = <&sata_phy>;
+				phy-names = "sata-phy";
+				clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
+				ti,hwmods = "sata";
+				ports-implemented = <0x1>;
+			};
  		};
 
  		/* OCP2SCP1 */
#6 lore raw

+Nishanth

Robin,

Quoted 6 lines · Robin Murphy
Quoted 3 lines · Roger Quadros

Even though the TRM says that SATA IP has 36 address bits wired in the SoC, we see bus errors whenever any address greater than 32-bit is given to the controller.

Actually, is it really just SATA? I pulled up a couple of DRA7xx TRMs out of curiosity - thanks for having such easy-to-access documentation by the way :) - and they both give me a clear impression that the entire L3_MAIN interconnect is limited to 32-bit addresses and thus pretty much all the DMA masters should only be able to touch the lower 2GB of DRAM. Especially the bit that explicitly says "This is a high address range (Q8 – Q15) that requires an address greater than 32 bits. This space is visible only for the MPU Subsystem."

You are right that L3 interconnect can only access first 2GB of DRAM. Which means we should add the bus_dma_limit to the entire L3 bus instead of just SATA?

Quoted 2 lines

Is it in fact the case that the SATA driver happens to be the only one to set a >32-bit DMA mask on your system?

This looks like the case. Other device drivers might not be overriding dma_mask at all thus using the default 32-bit dma_mask.

cheers,
-roger

Quoted 58 lines

Robin.

Quoted 55 lines

This happens on dra7-EVM with 4G of RAM with CONFIG_ARM_LPAE=y.

As a workaround we limit the DMA address range to 32-bits for SATA.

Cc Christoph Hellwig <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Rob Herring <[email protected]>

---

NOTE: Currently ARM dma-mapping code doesn't account for devices bus_dma_limit. This is fixed in [1].

[1] https://lkml.org/lkml/2020/2/18/712

  arch/arm/boot/dts/dra7.dtsi | 25 ++++++++++++++++---------

  1 file changed, 16 insertions(+), 9 deletions(-)
arch/arm/boot/dts/dra7.dtsi +16 -9
index d78b684e7fca..895462c22d1c 100644
@@ -642,15 +642,22 @@
           };
           /* OCP2SCP3 */
-        sata: sata@4a141100 {
-            compatible = "snps,dwc-ahci";
-            reg = <0x4a140000 0x1100>, <0x4a141100 0x7>;
-            interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
-            phys = <&sata_phy>;
-            phy-names = "sata-phy";
-            clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
-            ti,hwmods = "sata";
-            ports-implemented = <0x1>;
+        sata_aux_bus {
+            #address-cells = <1>;
+            #size-cells = <2>;
+            compatible = "simple-bus";
+            ranges = <0x0 0x4a140000 0x0 0x1200>;
+            dma-ranges = <0x0 0x0 0x1 0x00000000>;
+            sata: sata@4a141100 {
+                compatible = "snps,dwc-ahci";
+                reg = <0x0 0x0 0x1100>, <0x1100 0x0 0x7>;
+                interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>;
+                phys = <&sata_phy>;
+                phy-names = "sata-phy";
+                clocks = <&l3init_clkctrl DRA7_L3INIT_SATA_CLKCTRL 8>;
+                ti,hwmods = "sata";
+                ports-implemented = <0x1>;
+            };
           };
           /* OCP2SCP1 */
 
Signature · 2 lines
Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki.
Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki
#7 lore raw
Quoted 10 lines · Roger Quadros

+Nishanth

Robin,

Quoted 5 lines · Robin Murphy
Quoted 3 lines · Roger Quadros

Even though the TRM says that SATA IP has 36 address bits wired in the SoC, we see bus errors whenever any address greater than 32-bit is given to the controller.

Quoted 5 lines
Quoted 1 line

Actually, is it really just SATA? I pulled up a couple of DRA7xx TRMs out of curiosity - thanks for having such easy-to-access documentation by the way :) - and they both give me a clear impression that the entire L3_MAIN interconnect is limited to 32-bit addresses and thus pretty much all the DMA masters should only be able to touch the lower 2GB of DRAM. Especially the bit that explicitly says "This is a high address range (Q8 – Q15) that requires an address greater than 32 bits. This space is visible only for the MPU Subsystem."

You are right that L3 interconnect can only access first 2GB of DRAM. Which means we should add the bus_dma_limit to the entire L3 bus instead of just SATA?

OK makes sense to me.

Quoted 4 lines
Quoted 1 line

Is it in fact the case that the SATA driver happens to be the only one to set a >32-bit DMA mask on your system?

This looks like the case. Other device drivers might not be overriding dma_mask at all thus using the default 32-bit dma_mask.

OK

Regards,

Tony