Skip to contentSkip to conversation

ARM: dts: armada-38x: Mark devices as dma-coherent

PATCH In review Marcin Wojtas 4 participants 7 messages Oct 22, 2022 – Oct 24, 2022 Device tree bindings

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ARM: dts: armada-38x: Mark devices as dma-coherent

Armada 38x platforms marks all devices as coherent via mvebu_hwcc_notifier(), whereas the standard way to determine this is by of_dma_is_coherent(). Reflect the hardware capabilities by adding 'dma-coherent' properties to the device tree.

---
 arch/arm/boot/dts/armada-380.dtsi | 1 +
 arch/arm/boot/dts/armada-385.dtsi | 1 +
 arch/arm/boot/dts/armada-38x.dtsi | 1 +
 3 files changed, 3 insertions(+)
arch/arm/boot/dts/armada-380.dtsi +1 -0
index ce1dddb2269b..25d17550e2fc 100644
@@ -38,6 +38,7 @@ pcie {
 			compatible = "marvell,armada-370-pcie";
 			status = "disabled";
 			device_type = "pci";
+			dma-coherent;
 
 			#address-cells = <3>;
 			#size-cells = <2>;
arch/arm/boot/dts/armada-385.dtsi +1 -0
index 83392b92dae2..6fb8c254cbdc 100644
@@ -37,6 +37,7 @@ pciec: pcie {
 			compatible = "marvell,armada-370-pcie";
 			status = "disabled";
 			device_type = "pci";
+			dma-coherent;
 
 			#address-cells = <3>;
 			#size-cells = <2>;
arch/arm/boot/dts/armada-38x.dtsi +1 -0
index 446861b6b17b..5801873dfcbe 100644
@@ -102,6 +102,7 @@ internal-regs {
 			#address-cells = <1>;
 			#size-cells = <1>;
 			ranges = <0 MBUS_ID(0xf0, 0x01) 0 0x100000>;
+			dma-coherent;
 
 			sdramc: sdramc@1400 {
 				compatible = "marvell,armada-xp-sdram-controller";
Signature · 1 line
2.29.0
#1 lore raw
Quoted 4 lines · Marcin Wojtas

Armada 38x platforms marks all devices as coherent via mvebu_hwcc_notifier(), whereas the standard way to determine this is by of_dma_is_coherent(). Reflect the hardware capabilities by adding 'dma-coherent' properties to the device tree.

Hi Marcin

Does this need to go to -rc for 6.0? The DMA issues being reported? If so, please add a Fixed: tag.

Andrew

#2 lore raw
Quoted 10 lines · Andrew Lunn
Quoted 4 lines · Marcin Wojtas

Armada 38x platforms marks all devices as coherent via mvebu_hwcc_notifier(), whereas the standard way to determine this is by of_dma_is_coherent(). Reflect the hardware capabilities by adding 'dma-coherent' properties to the device tree.

Hi Marcin

Does this need to go to -rc for 6.0? The DMA issues being reported? If so, please add a Fixed: tag.

Are we absolutely sure this makes sense?

Looking at atch/arm/mach-mvebu/coherency.c, there are dependencies on stuff such as whether the kernel is in SMP mode or not (because the page tables need to be appropriately marked as shared for coherency with IO to work). We only enable the shared bit if we're in SMP mode because (a) its difficult to do at runtime due to TLB conflicts (requires switching the MMU off, rewriting the page tables and switching the MMU back on), and (b) setting the shared bit for CPUs that don't need it _can_ result in the CPUs basically bypassing their caches and thus kill system performance.

So, if we have Armada 38x platforms that are operated in uniprocessor mode, this patch can cause havoc on such a setup.

I would suggest utmost caution with this approach.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#3 lore raw

niedz., 23 paź 2022 o 18:21 Russell King (Oracle)
<[email protected]> napisał(a):

Quoted 25 lines · Andrew Lunn
Quoted 10 lines · Marcin Wojtas
Quoted 4 lines

Armada 38x platforms marks all devices as coherent via mvebu_hwcc_notifier(), whereas the standard way to determine this is by of_dma_is_coherent(). Reflect the hardware capabilities by adding 'dma-coherent' properties to the device tree.

Hi Marcin

Does this need to go to -rc for 6.0? The DMA issues being reported? If so, please add a Fixed: tag.

Are we absolutely sure this makes sense?

Looking at atch/arm/mach-mvebu/coherency.c, there are dependencies on stuff such as whether the kernel is in SMP mode or not (because the page tables need to be appropriately marked as shared for coherency with IO to work). We only enable the shared bit if we're in SMP mode because (a) its difficult to do at runtime due to TLB conflicts (requires switching the MMU off, rewriting the page tables and switching the MMU back on), and (b) setting the shared bit for CPUs that don't need it _can_ result in the CPUs basically bypassing their caches and thus kill system performance.

Quoted 5 lines

So, if we have Armada 38x platforms that are operated in uniprocessor mode, this patch can cause havoc on such a setup.

I would suggest utmost caution with this approach.

Sure. In such a case the description of 380 variant (single core) should remain untouched.

We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about:
- Don't change current behavior, i.e. perform a necessary kernel configuration in "arm,pl310-cache" driver, arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT
- Satisfy of_dma_is_coherent() by adding dma-coherent; in armada-385.dtsi only (IMO this would describe HW properly)
?

Best regards,
Marcin

#4 lore raw
Quoted 36 lines · On Sun, 23 Oct 2022 23:30:34 +0200 Marcin Wojtas <[email protected]> wrote:

niedz., 23 paź 2022 o 18:21 Russell King (Oracle)
<[email protected]> napisał(a):

Quoted 25 lines · Andrew Lunn
Quoted 10 lines · Marcin Wojtas
Quoted 4 lines

Armada 38x platforms marks all devices as coherent via mvebu_hwcc_notifier(), whereas the standard way to determine this is by of_dma_is_coherent(). Reflect the hardware capabilities by adding 'dma-coherent' properties to the device tree.

Hi Marcin

Does this need to go to -rc for 6.0? The DMA issues being reported? If so, please add a Fixed: tag.

Are we absolutely sure this makes sense?

Looking at atch/arm/mach-mvebu/coherency.c, there are dependencies on stuff such as whether the kernel is in SMP mode or not (because the page tables need to be appropriately marked as shared for coherency with IO to work). We only enable the shared bit if we're in SMP mode because (a) its difficult to do at runtime due to TLB conflicts (requires switching the MMU off, rewriting the page tables and switching the MMU back on), and (b) setting the shared bit for CPUs that don't need it _can_ result in the CPUs basically bypassing their caches and thus kill system performance.

Quoted 5 lines

So, if we have Armada 38x platforms that are operated in uniprocessor mode, this patch can cause havoc on such a setup.

I would suggest utmost caution with this approach.

Sure. In such a case the description of 380 variant (single core) should remain untouched.

Quoted 7 lines

We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about:
- Don't change current behavior, i.e. perform a necessary kernel configuration in "arm,pl310-cache" driver, arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT
- Satisfy of_dma_is_coherent() by adding dma-coherent; in armada-385.dtsi only (IMO this would describe HW properly)
?

It will describe HW properly, but someone running older kernel compiled with no SMP support will see a performance drop. I wonder how many people do that.

Marek

#5 lore raw
Quoted 11 lines · Marek Behún
Quoted 10 lines

Sure. In such a case the description of 380 variant (single core) should remain untouched.

We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about:
- Don't change current behavior, i.e. perform a necessary kernel configuration in "arm,pl310-cache" driver, arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT
- Satisfy of_dma_is_coherent() by adding dma-coherent; in armada-385.dtsi only (IMO this would describe HW properly)
?

Quoted 3 lines

It will describe HW properly, but someone running older kernel compiled with no SMP support will see a performance drop. I wonder how many people do that.

If the kernel is built without SMP support, the page table entries will not have the shared bit set, and the system will _not_ be DMA-coherent. Having DT mark devices as "dma-coherent" in this case will lead to data corruption, because the DMA API will believe them to be DMA-coherent when the page tables are not setup for that to work.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#6 lore raw

pon., 24 paź 2022 o 09:51 Russell King (Oracle)
<[email protected]> napisał(a):

Quoted 17 lines · Marek Behún
Quoted 11 lines
Quoted 10 lines

Sure. In such a case the description of 380 variant (single core) should remain untouched.

We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about:
- Don't change current behavior, i.e. perform a necessary kernel configuration in "arm,pl310-cache" driver, arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT
- Satisfy of_dma_is_coherent() by adding dma-coherent; in armada-385.dtsi only (IMO this would describe HW properly)
?

Quoted 3 lines

It will describe HW properly, but someone running older kernel compiled with no SMP support will see a performance drop. I wonder how many people do that.

Quoted 6 lines

If the kernel is built without SMP support, the page table entries will not have the shared bit set, and the system will _not_ be DMA-coherent. Having DT mark devices as "dma-coherent" in this case will lead to data corruption, because the DMA API will believe them to be DMA-coherent when the page tables are not setup for that to work.

Thanks, for the explanation. Since we're heavily dependent on what happens in the kernel we boot, it will be easier to just drop this patch and keep using the DT as-is.

Best regards,
Marcin