Skip to contentSkip to conversation

arm64: tegra: Prevent early SMMU faults

PATCH Applied Thierry Reding 7 participants 23 messages Jun 3, 2021 – Jun 21, 2021

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

v3 cover arm64: tegra: Prevent early SMMU faults

From: Thierry Reding <[email protected]>

Hi,

this is a set of patches that is the result of earlier discussions regarding early identity mappings that are needed to avoid SMMU faults during early boot.

The goal here is to avoid early identity mappings altogether and instead postpone the need for the identity mappings to when devices are attached to the SMMU. This works by making the SMMU driver coordinate with the memory controller driver on when to start enforcing SMMU translations. This makes Tegra behave in a more standard way and pushes the code to deal with the Tegra-specific programming into the NVIDIA SMMU implementation.

Compared to the original version of these patches, I've split the preparatory work into a separate patch series because it became very large and will be mostly uninteresting for this audience.

Patch 1 provides a mechanism to program SID overrides at runtime. Patch 2 updates the ARM SMMU device tree bindings to include the Tegra186 compatible string as suggested by Robin during review.

Patches 3 and 4 create the fundamentals in the SMMU driver to support this and also make this functionality available on Tegra186. Patch 5 hooks the ARM SMMU up to the memory controller so that the memory client stream ID overrides can be programmed at the right time.

Patch 6 extends this mechanism to Tegra186 and patches 7-9 enable all of this through device tree updates. Patch 10 (that was included in earlier version to show how SMMU will be enabled for display controllers) has been dropped for now while waiting for the identity mappings support to land.

The end result is that various peripherals will have SMMU enabled, while the display controllers will keep using passthrough, as initially set up by firmware. Once the device tree bindings have been accepted and the SMMU driver has been updated to create identity mappings for the display controllers, they can be hooked up to the SMMU and the code in this series will automatically program the SID overrides to enable SMMU translations at the right time.

Will, Krzysztof: as discussed, it'd be best if Krzysztof picked up patch 1 into the memory controller tree on top of v3 of the driver unification series I sent out earlier today and then sent out a PR for Will to merge and apply patches 2-6. I can then take patches 7-9 in via the Tegra tree since there are no hard dependencies.

Changes in v3:
- move hunk from patch 4 to patch 5 to preserve bisectibility

Changes in v2:
- split off the preparatory work into a separate series (that needs to be applied first)
- address review comments by Robin

Thierry

Thierry Reding (9):
memory: tegra: Implement SID override programming
dt-bindings: arm-smmu: Add Tegra186 compatible string iommu/arm-smmu: Implement ->probe_finalize() iommu/arm-smmu: tegra: Detect number of instances at runtime iommu/arm-smmu: tegra: Implement SID override programming iommu/arm-smmu: Use Tegra implementation on Tegra186
arm64: tegra: Use correct compatible string for Tegra186 SMMU
arm64: tegra: Hook up memory controller to SMMU on Tegra186
arm64: tegra: Enable SMMU support on Tegra194

 .../devicetree/bindings/iommu/arm,smmu.yaml   | 11 ++-
 arch/arm64/boot/dts/nvidia/tegra186.dtsi      |  4 +-
 arch/arm64/boot/dts/nvidia/tegra194.dtsi      | 86 ++++++++++++++++++
 drivers/iommu/arm/arm-smmu/arm-smmu-impl.c    |  3 +-
 drivers/iommu/arm/arm-smmu/arm-smmu-nvidia.c  | 90 +++++++++++++++----
 drivers/iommu/arm/arm-smmu/arm-smmu.c         | 13 +++
 drivers/iommu/arm/arm-smmu/arm-smmu.h         |  1 +
 drivers/memory/tegra/mc.c                     |  9 ++
 drivers/memory/tegra/tegra186.c               | 72 +++++++++++++++
 include/soc/tegra/mc.h                        |  3 +
 10 files changed, 269 insertions(+), 23 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH v3 1/9 memory: tegra: Implement SID override programming

From: Thierry Reding <[email protected]>

Instead of programming all SID overrides during early boot, perform the operation on-demand after the SMMU translations have been set up for a device. This reuses data from device tree to match memory clients for a device and programs the SID specified in device tree, which corresponds to the SID used for the SMMU context banks for the device.

---
 drivers/memory/tegra/mc.c       |  9 +++++
 drivers/memory/tegra/tegra186.c | 72 +++++++++++++++++++++++++++++++++
 include/soc/tegra/mc.h          |  3 ++
 3 files changed, 84 insertions(+)
drivers/memory/tegra/mc.c +9 -0
index 11b83de9361c..3c5aae7abf35 100644
@@ -97,6 +97,15 @@ struct tegra_mc *devm_tegra_memory_controller_get(struct device *dev)
 }
 EXPORT_SYMBOL_GPL(devm_tegra_memory_controller_get);
 
+int tegra_mc_probe_device(struct tegra_mc *mc, struct device *dev)
+{
+	if (mc->soc->ops && mc->soc->ops->probe_device)
+		return mc->soc->ops->probe_device(mc, dev);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(tegra_mc_probe_device);
+
 static int tegra_mc_block_dma_common(struct tegra_mc *mc,
 				     const struct tegra_mc_reset *rst)
 {
drivers/memory/tegra/tegra186.c +72 -0

94 lines not on this page · raw message

include/soc/tegra/mc.h +3 -0
index 1bd5aed81868..e19c2504a14b 100644
@@ -180,6 +180,7 @@ struct tegra_mc_ops {
 	int (*suspend)(struct tegra_mc *mc);
 	int (*resume)(struct tegra_mc *mc);
 	irqreturn_t (*handle_irq)(int irq, void *data);
+	int (*probe_device)(struct tegra_mc *mc, struct device *dev);
 };
 
 struct tegra_mc_soc {
@@ -244,4 +245,6 @@ devm_tegra_memory_controller_get(struct device *dev)
 }
 #endif
 
+int tegra_mc_probe_device(struct tegra_mc *mc, struct device *dev);
+
 #endif /* __SOC_TEGRA_MC_H__ */
Signature · 6 lines
2.31.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#2 lore raw

PATCH v3 2/9 dt-bindings: arm-smmu: Add Tegra186 compatible string

In reply to Thierry Reding

From: Thierry Reding <[email protected]>

The ARM SMMU instantiations found on Tegra186 and later need inter- operation with the memory controller in order to correctly program stream ID overrides.

Furthermore, on Tegra194 multiple instances of the SMMU can gang up to achieve higher throughput. In order to do this, they have to be programmed identically so that the memory controller can interleave memory accesses between them.

Add the Tegra186 compatible string to make sure the interoperation with the memory controller can be enabled on that SoC generation.

---
 Documentation/devicetree/bindings/iommu/arm,smmu.yaml | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +9 -2
index 9d27aa5111d4..1181b590db71 100644
@@ -54,8 +54,14 @@ properties:
           - const: arm,mmu-500
       - description: NVIDIA SoCs that program two ARM MMU-500s identically
         items:
+      - description: NVIDIA SoCs that require memory controller interaction
+          and may program multiple ARM MMU-500s identically with the memory
+          controller interleaving translations between multiple instances
+          for improved performance.
+        items:
           - enum:
-              - nvidia,tegra194-smmu
+              - const: nvidia,tegra194-smmu
+              - const: nvidia,tegra186-smmu
           - const: nvidia,smmu-500
       - items:
           - const: arm,mmu-500
@@ -165,10 +171,11 @@ allOf:
           contains:
             enum:
               - nvidia,tegra194-smmu
+              - nvidia,tegra186-smmu
     then:
       properties:
         reg:
-          minItems: 2
+          minItems: 1
           maxItems: 2
     else:
       properties:
Signature · 6 lines
2.31.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
maintainer
#3 lore raw

Re: [PATCH v3 2/9] dt-bindings: arm-smmu: Add Tegra186 compatible string

Quoted 25 lines · Thierry Reding

From: Thierry Reding <[email protected]>

The ARM SMMU instantiations found on Tegra186 and later need inter- operation with the memory controller in order to correctly program stream ID overrides.

Furthermore, on Tegra194 multiple instances of the SMMU can gang up to achieve higher throughput. In order to do this, they have to be programmed identically so that the memory controller can interleave memory accesses between them.

Add the Tegra186 compatible string to make sure the interoperation with the memory controller can be enabled on that SoC generation.

---
 Documentation/devicetree/bindings/iommu/arm,smmu.yaml | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +0 -0
index 9d27aa5111d4..1181b590db71 100644

Documentation/devicetree/bindings/iommu/arm,smmu.yaml properties: · line 57

Quoted 4 lines
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +1 -0
@@ -54,8 +54,14 @@ properties:
           - const: arm,mmu-500
       - description: NVIDIA SoCs that program two ARM MMU-500s identically
         items:
+      - description: NVIDIA SoCs that require memory controller interaction

This is not valid jsonschema:

/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'anyOf' conditional failed, one must be fixed:
None is not of type 'object', 'boolean'
None is not of type 'array'
from schema $id: http://json-schema.org/draft-07/schema#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'oneOf' conditional failed, one must be fixed:
None is not of type 'object'
None is not of type 'array'
from schema $id: http://devicetree.org/meta-schemas/keywords.yaml#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'oneOf' conditional failed, one must be fixed:
None is not of type 'object'
None is not of type 'array'
from schema $id: http://devicetree.org/meta-schemas/string-array.yaml#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:5:items: 'oneOf' conditional failed, one must be fixed:
[{'enum': [{'const': 'nvidia,tegra194-smmu'}, {'const':
'nvidia,tegra186-smmu'}]}, {'const': 'nvidia,smmu-500'}] is not of type 'object'
{'const': 'nvidia,tegra194-smmu'} is not of type 'string'
{'const': 'nvidia,tegra186-smmu'} is not of type 'string'
from schema $id: http://devicetree.org/meta-schemas/string-array.yaml#

This was not reviewed nor tested since the DT list was not Cc'ed.

Rob

#4 lore raw

Re: [PATCH v3 2/9] dt-bindings: arm-smmu: Add Tegra186 compatible string

Quoted 62 lines · Rob Herring
Quoted 25 lines · Thierry Reding

From: Thierry Reding <[email protected]>

The ARM SMMU instantiations found on Tegra186 and later need inter- operation with the memory controller in order to correctly program stream ID overrides.

Furthermore, on Tegra194 multiple instances of the SMMU can gang up to achieve higher throughput. In order to do this, they have to be programmed identically so that the memory controller can interleave memory accesses between them.

Add the Tegra186 compatible string to make sure the interoperation with the memory controller can be enabled on that SoC generation.

---
 Documentation/devicetree/bindings/iommu/arm,smmu.yaml | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +0 -0
index 9d27aa5111d4..1181b590db71 100644

Documentation/devicetree/bindings/iommu/arm,smmu.yaml properties: · line 57

Quoted 4 lines
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +1 -0
@@ -54,8 +54,14 @@ properties:
           - const: arm,mmu-500
       - description: NVIDIA SoCs that program two ARM MMU-500s identically
         items:
+      - description: NVIDIA SoCs that require memory controller interaction

This is not valid jsonschema:

/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'anyOf' conditional failed, one must be fixed:
None is not of type 'object', 'boolean'
None is not of type 'array'
from schema $id: http://json-schema.org/draft-07/schema#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'oneOf' conditional failed, one must be fixed:
None is not of type 'object'
None is not of type 'array'
from schema $id: http://devicetree.org/meta-schemas/keywords.yaml#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'oneOf' conditional failed, one must be fixed:
None is not of type 'object'
None is not of type 'array'
from schema $id: http://devicetree.org/meta-schemas/string-array.yaml#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:5:items: 'oneOf' conditional failed, one must be fixed:
[{'enum': [{'const': 'nvidia,tegra194-smmu'}, {'const':
'nvidia,tegra186-smmu'}]}, {'const': 'nvidia,smmu-500'}] is not of type 'object'
{'const': 'nvidia,tegra194-smmu'} is not of type 'string'
{'const': 'nvidia,tegra186-smmu'} is not of type 'string'
from schema $id: http://devicetree.org/meta-schemas/string-array.yaml#

Quoted 1 line

This was not reviewed nor tested since the DT list was not Cc'ed.

Ugh, I see now weird empty item on a list... and not only DT list was skipped - Thierry did not Cc you either.

My bad, I did not check that patch thoroughly before applying.

Thierry, please Cc folks mentioned by get_maintainer.pl. Either sent a fix or a revert, if fix needs more time.

Additionally, why the patch changes reg to "minItems: 1" for nvidia,tegra194-smmu?

Best regards,
Krzysztof

#5 lore raw

Re: [PATCH v3 2/9] dt-bindings: arm-smmu: Add Tegra186 compatible string

Quoted 63 lines · Krzysztof Kozlowski
Quoted 62 lines · Rob Herring
Quoted 25 lines · Thierry Reding

From: Thierry Reding <[email protected]>

The ARM SMMU instantiations found on Tegra186 and later need inter- operation with the memory controller in order to correctly program stream ID overrides.

Furthermore, on Tegra194 multiple instances of the SMMU can gang up to achieve higher throughput. In order to do this, they have to be programmed identically so that the memory controller can interleave memory accesses between them.

Add the Tegra186 compatible string to make sure the interoperation with the memory controller can be enabled on that SoC generation.

---
 Documentation/devicetree/bindings/iommu/arm,smmu.yaml | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +0 -0
index 9d27aa5111d4..1181b590db71 100644

Documentation/devicetree/bindings/iommu/arm,smmu.yaml properties: · line 57

Quoted 4 lines
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +1 -0
@@ -54,8 +54,14 @@ properties:
           - const: arm,mmu-500
       - description: NVIDIA SoCs that program two ARM MMU-500s identically
         items:
+      - description: NVIDIA SoCs that require memory controller interaction

This is not valid jsonschema:

/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'anyOf' conditional failed, one must be fixed:
None is not of type 'object', 'boolean'
None is not of type 'array'
from schema $id: http://json-schema.org/draft-07/schema#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'oneOf' conditional failed, one must be fixed:
None is not of type 'object'
None is not of type 'array'
from schema $id: http://devicetree.org/meta-schemas/keywords.yaml#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'oneOf' conditional failed, one must be fixed:
None is not of type 'object'
None is not of type 'array'
from schema $id: http://devicetree.org/meta-schemas/string-array.yaml#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:5:items: 'oneOf' conditional failed, one must be fixed:
[{'enum': [{'const': 'nvidia,tegra194-smmu'}, {'const':
'nvidia,tegra186-smmu'}]}, {'const': 'nvidia,smmu-500'}] is not of type 'object'
{'const': 'nvidia,tegra194-smmu'} is not of type 'string'
{'const': 'nvidia,tegra186-smmu'} is not of type 'string'
from schema $id: http://devicetree.org/meta-schemas/string-array.yaml#

Quoted 4 lines
Quoted 1 line

This was not reviewed nor tested since the DT list was not Cc'ed.

Ugh, I see now weird empty item on a list... and not only DT list was skipped - Thierry did not Cc you either.

This seemed like a too trivial addition to waste Rob's time on, so I didn't add him (or the DT list for that matter) on Cc. The ARM SMMU maintainers had reviewed this, which seemed like it was enough for what the DT bindings change was doing.

In any case, I clearly should've checked the DT binding check output more carefully. It's rather messy for Tegra because there's quite a few that we haven't converted yet. I'll have to resume my effort to convert the remaining ones and fixup the device trees so that we can actually run the DT binding and DTB validation checks more usefully.

Quoted 4 lines

My bad, I did not check that patch thoroughly before applying.

Thierry, please Cc folks mentioned by get_maintainer.pl. Either sent a fix or a revert, if fix needs more time.

I've sent out a follow-up fix that removes the two bogus lines. It looks like that was the result of a bad conflict resolution on my part.

Quoted 2 lines

Additionally, why the patch changes reg to "minItems: 1" for nvidia,tegra194-smmu?

This is because originally the Tegra194 SMMU nodes were supposed to only represent a "dual" instance. However, on Tegra194 there are three SMMU instances in total, with the third instance (dedicated for isochronous traffic) being completely separate and having only a single range of registers.

That third instance was previously supposed to be covered by the normal "arm,mmu-500" compatible string, but given that we really need that interoperation between SMMU and memory controller for SID override programming, we need the Tegra-specific compatible for the ISO instance of the SMMU as well. And since that uses only one set of registers, minItems had to become 1.

Thierry

maintainer
#6 lore raw

Re: [PATCH v3 2/9] dt-bindings: arm-smmu: Add Tegra186 compatible string

Quoted 46 lines · Thierry Reding
Quoted 45 lines · Krzysztof Kozlowski
Quoted 43 lines · Rob Herring

Documentation/devicetree/bindings/iommu/arm,smmu.yaml properties: · line 57

Quoted 9 lines · Thierry Reding
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +1 -0
index 9d27aa5111d4..1181b590db71 100644
@@ -54,8 +54,14 @@ properties:
           - const: arm,mmu-500
       - description: NVIDIA SoCs that program two ARM MMU-500s identically
         items:
+      - description: NVIDIA SoCs that require memory controller interaction

This is not valid jsonschema:

/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'anyOf' conditional failed, one must be fixed:
None is not of type 'object', 'boolean'
None is not of type 'array'
from schema $id: http://json-schema.org/draft-07/schema#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'oneOf' conditional failed, one must be fixed:
None is not of type 'object'
None is not of type 'array'
from schema $id: http://devicetree.org/meta-schemas/keywords.yaml#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'oneOf' conditional failed, one must be fixed:
None is not of type 'object'
None is not of type 'array'
from schema $id: http://devicetree.org/meta-schemas/string-array.yaml#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:5:items: 'oneOf' conditional failed, one must be fixed:
[{'enum': [{'const': 'nvidia,tegra194-smmu'}, {'const':
'nvidia,tegra186-smmu'}]}, {'const': 'nvidia,smmu-500'}] is not of type 'object'
{'const': 'nvidia,tegra194-smmu'} is not of type 'string'
{'const': 'nvidia,tegra186-smmu'} is not of type 'string'
from schema $id: http://devicetree.org/meta-schemas/string-array.yaml#

This was not reviewed nor tested since the DT list was not Cc'ed.

Quoted 7 lines
Quoted 2 lines

Ugh, I see now weird empty item on a list... and not only DT list was skipped - Thierry did not Cc you either.

This seemed like a too trivial addition to waste Rob's time on, so I didn't add him (or the DT list for that matter) on Cc. The ARM SMMU maintainers had reviewed this, which seemed like it was enough for what the DT bindings change was doing.

Hmm, I didn't review it. I find the yaml stuff unreadable so I usually wait for the DT folks to ack bindings changes before I queue them in the SMMU tree.

Will

#7 lore raw

Re: [PATCH v3 2/9] dt-bindings: arm-smmu: Add Tegra186 compatible string

Quoted 50 lines · Will Deacon
Quoted 49 lines · Thierry Reding
Quoted 47 lines · Krzysztof Kozlowski
Quoted 42 lines · Rob Herring

Documentation/devicetree/bindings/iommu/arm,smmu.yaml properties: · line 57

Quoted 9 lines · Thierry Reding
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +1 -0
index 9d27aa5111d4..1181b590db71 100644
@@ -54,8 +54,14 @@ properties:
           - const: arm,mmu-500
       - description: NVIDIA SoCs that program two ARM MMU-500s identically
         items:
+      - description: NVIDIA SoCs that require memory controller interaction

This is not valid jsonschema:

/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'anyOf' conditional failed, one must be fixed:
None is not of type 'object', 'boolean'
None is not of type 'array'
from schema $id: http://json-schema.org/draft-07/schema#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'oneOf' conditional failed, one must be fixed:
None is not of type 'object'
None is not of type 'array'
from schema $id: http://devicetree.org/meta-schemas/keywords.yaml#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:4:items: 'oneOf' conditional failed, one must be fixed:
None is not of type 'object'
None is not of type 'array'
from schema $id: http://devicetree.org/meta-schemas/string-array.yaml#
/builds/robherring/linux-dt/Documentation/devicetree/bindings/iommu/arm,smmu.yaml:
properties:compatible:oneOf:5:items: 'oneOf' conditional failed, one must be fixed:
[{'enum': [{'const': 'nvidia,tegra194-smmu'}, {'const':
'nvidia,tegra186-smmu'}]}, {'const': 'nvidia,smmu-500'}] is not of type 'object'
{'const': 'nvidia,tegra194-smmu'} is not of type 'string'
{'const': 'nvidia,tegra186-smmu'} is not of type 'string'
from schema $id: http://devicetree.org/meta-schemas/string-array.yaml#

Quoted 1 line

This was not reviewed nor tested since the DT list was not Cc'ed.

Ugh, I see now weird empty item on a list... and not only DT list was skipped - Thierry did not Cc you either.

Quoted 8 lines
Quoted 4 lines

This seemed like a too trivial addition to waste Rob's time on, so I didn't add him (or the DT list for that matter) on Cc. The ARM SMMU maintainers had reviewed this, which seemed like it was enough for what the DT bindings change was doing.

Hmm, I didn't review it. I find the yaml stuff unreadable so I usually wait for the DT folks to ack bindings changes before I queue them in the SMMU tree.

Alright... in the future I'll make sure to always Cc DT folks, even for trivial stuff like this.

Thierry

#8 lore raw

PATCH v3 3/9 iommu/arm-smmu: Implement ->probe_finalize()

In reply to Thierry Reding

From: Thierry Reding <[email protected]>

Implement a ->probe_finalize() callback that can be used by vendor implementations to perform extra programming necessary after devices have been attached to the SMMU.

---
Changes in v2:
- remove unnecessarily paranoid check

 drivers/iommu/arm/arm-smmu/arm-smmu.c | 13 +++++++++++++
 drivers/iommu/arm/arm-smmu/arm-smmu.h |  1 +
 2 files changed, 14 insertions(+)
drivers/iommu/arm/arm-smmu/arm-smmu.c +13 -0
index 6f72c4d208ca..d20ce4d57df2 100644
@@ -1450,6 +1450,18 @@ static void arm_smmu_release_device(struct device *dev)
 	iommu_fwspec_free(dev);
 }
 
+static void arm_smmu_probe_finalize(struct device *dev)
+{
+	struct arm_smmu_master_cfg *cfg;
+	struct arm_smmu_device *smmu;
+
+	cfg = dev_iommu_priv_get(dev);
+	smmu = cfg->smmu;
+
+	if (smmu->impl->probe_finalize)
+		smmu->impl->probe_finalize(smmu, dev);
+}
+
 static struct iommu_group *arm_smmu_device_group(struct device *dev)
 {
 	struct arm_smmu_master_cfg *cfg = dev_iommu_priv_get(dev);
@@ -1569,6 +1581,7 @@ static struct iommu_ops arm_smmu_ops = {
 	.iova_to_phys		= arm_smmu_iova_to_phys,
 	.probe_device		= arm_smmu_probe_device,
 	.release_device		= arm_smmu_release_device,
+	.probe_finalize		= arm_smmu_probe_finalize,
 	.device_group		= arm_smmu_device_group,
 	.enable_nesting		= arm_smmu_enable_nesting,
 	.set_pgtable_quirks	= arm_smmu_set_pgtable_quirks,
drivers/iommu/arm/arm-smmu/arm-smmu.h +1 -0
index c31a59d35c64..147c95e7c59c 100644
@@ -439,6 +439,7 @@ struct arm_smmu_impl {
 				  struct device *dev, int start);
 	void (*write_s2cr)(struct arm_smmu_device *smmu, int idx);
 	void (*write_sctlr)(struct arm_smmu_device *smmu, int idx, u32 reg);
+	void (*probe_finalize)(struct arm_smmu_device *smmu, struct device *dev);
 };
 
 #define INVALID_SMENDX			-1
Signature · 6 lines
2.31.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#9 lore raw

Re: [PATCH v3 3/9] iommu/arm-smmu: Implement ->probe_finalize()

Hi,

Quoted 7 lines · Thierry Reding

From: Thierry Reding <[email protected]>

Implement a ->probe_finalize() callback that can be used by vendor implementations to perform extra programming necessary after devices have been attached to the SMMU.

This patch landed recently in linux-next as commit 0d97174aeadf ("iommu/arm-smmu: Implement ->probe_finalize()"). It causes the following issue on ARM Juno R1 board:

arm-smmu 2b500000.iommu: probing hardware configuration... arm-smmu 2b500000.iommu: SMMUv1 with:
arm-smmu 2b500000.iommu:         stage 2 translation
arm-smmu 2b500000.iommu:         coherent table walk
arm-smmu 2b500000.iommu:         stream matching with 32 register groups arm-smmu 2b500000.iommu:         4 context banks (4 stage-2 only) arm-smmu 2b500000.iommu:         Supported page sizes: 0x60211000 arm-smmu 2b500000.iommu:         Stage-2: 40-bit IPA -> 40-bit PA arm-smmu 7fb00000.iommu: probing hardware configuration... arm-smmu 7fb00000.iommu: SMMUv1 with:
arm-smmu 7fb00000.iommu:         stage 2 translation
arm-smmu 7fb00000.iommu:         coherent table walk
arm-smmu 7fb00000.iommu:         stream matching with 16 register groups arm-smmu 7fb00000.iommu:         4 context banks (4 stage-2 only) arm-smmu 7fb00000.iommu:         Supported page sizes: 0x60211000 arm-smmu 7fb00000.iommu:         Stage-2: 40-bit IPA -> 40-bit PA arm-smmu 7fb10000.iommu: probing hardware configuration... arm-smmu 7fb10000.iommu: SMMUv1 with:
arm-smmu 7fb10000.iommu:         stage 2 translation
arm-smmu 7fb10000.iommu:         non-coherent table walk arm-smmu 7fb10000.iommu:         (IDR0.CTTW overridden by FW configuration) arm-smmu 7fb10000.iommu:         stream matching with 2 register groups arm-smmu 7fb10000.iommu:         1 context banks (1 stage-2 only) arm-smmu 7fb10000.iommu:         Supported page sizes: 0x60211000 arm-smmu 7fb10000.iommu:         Stage-2: 40-bit IPA -> 40-bit PA arm-smmu 7fb20000.iommu: probing hardware configuration... arm-smmu 7fb20000.iommu: SMMUv1 with:
arm-smmu 7fb20000.iommu:         stage 2 translation
arm-smmu 7fb20000.iommu:         non-coherent table walk arm-smmu 7fb20000.iommu:         (IDR0.CTTW overridden by FW configuration) arm-smmu 7fb20000.iommu:         stream matching with 2 register groups arm-smmu 7fb20000.iommu:         1 context banks (1 stage-2 only) arm-smmu 7fb20000.iommu:         Supported page sizes: 0x60211000 arm-smmu 7fb20000.iommu:         Stage-2: 40-bit IPA -> 40-bit PA arm-smmu 7fb30000.iommu: probing hardware configuration... arm-smmu 7fb30000.iommu: SMMUv1 with:
arm-smmu 7fb30000.iommu:         stage 2 translation
arm-smmu 7fb30000.iommu:         coherent table walk
arm-smmu 7fb30000.iommu:         stream matching with 2 register groups arm-smmu 7fb30000.iommu:         1 context banks (1 stage-2 only) arm-smmu 7fb30000.iommu:         Supported page sizes: 0x60211000 arm-smmu 7fb30000.iommu:         Stage-2: 40-bit IPA -> 40-bit PA tda998x 0-0070: found TDA19988
tda998x 0-0071: found TDA19988
brd: module loaded
loop: module loaded
megasas: 07.714.04.00-rc1
sata_sil24 0000:03:00.0: Adding to iommu group 0

Unable to handle kernel NULL pointer dereference at virtual address 

0000000000000070
Mem abort info:
  ESR = 0x96000004
  EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0
  EA = 0, S1PTW = 0
Data abort info:
  ISV = 0, ISS = 0x00000004
  CM = 0, WnR = 0
[0000000000000070] user address but active_mm is swapper

Internal error: Oops: 96000004 [#1] PREEMPT SMP
Modules linked in:
CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.13.0-rc1+ #3466
Hardware name: ARM Juno development board (r1) (DT)
pstate: 20000005 (nzCv daif -PAN -UAO -TCO BTYPE=--)
pc : arm_smmu_probe_finalize+0x14/0x48
lr : iommu_probe_device+0x74/0x120
...
Call trace:
  arm_smmu_probe_finalize+0x14/0x48
  of_iommu_configure+0xe4/0x1b8
  of_dma_configure_id+0xf8/0x2d8
  pci_dma_configure+0x44/0x88
  really_probe+0xc0/0x3c0
  driver_probe_device+0x60/0xc0
  device_driver_attach+0x6c/0x78
  __driver_attach+0xc0/0x100
  bus_for_each_dev+0x68/0xc8
  driver_attach+0x20/0x28
  bus_add_driver+0x168/0x1f8
  driver_register+0x60/0x110
  __pci_register_driver+0x5c/0x68
  sil24_pci_driver_init+0x20/0x28
  do_one_initcall+0x84/0x450
  kernel_init_freeable+0x2dc/0x334
  kernel_init+0x10/0x110
  ret_from_fork+0x10/0x18
Code: b40001e1 f9405821 f9400023 f9401461 (f9403822)
---[ end trace 561eda4b855861d1 ]---
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

SMP: stopping secondary CPUs
Kernel Offset: disabled
CPU features: 0x00240022,25006086
Memory Limit: none

---[ end Kernel panic - not syncing: Attempted to kill init! 

exitcode=0x0000000b ]---

Quoted 44 lines

---
Changes in v2:
- remove unnecessarily paranoid check

  drivers/iommu/arm/arm-smmu/arm-smmu.c | 13 +++++++++++++
  drivers/iommu/arm/arm-smmu/arm-smmu.h |  1 +
  2 files changed, 14 insertions(+)
drivers/iommu/arm/arm-smmu/arm-smmu.c +13 -0
index 6f72c4d208ca..d20ce4d57df2 100644
@@ -1450,6 +1450,18 @@ static void arm_smmu_release_device(struct device *dev)
  	iommu_fwspec_free(dev);
  }
 
+static void arm_smmu_probe_finalize(struct device *dev)
+{
+	struct arm_smmu_master_cfg *cfg;
+	struct arm_smmu_device *smmu;
+
+	cfg = dev_iommu_priv_get(dev);
+	smmu = cfg->smmu;
+
+	if (smmu->impl->probe_finalize)
+		smmu->impl->probe_finalize(smmu, dev);
+}
+
  static struct iommu_group *arm_smmu_device_group(struct device *dev)
  {
  	struct arm_smmu_master_cfg *cfg = dev_iommu_priv_get(dev);
@@ -1569,6 +1581,7 @@ static struct iommu_ops arm_smmu_ops = {
  	.iova_to_phys		= arm_smmu_iova_to_phys,
  	.probe_device		= arm_smmu_probe_device,
  	.release_device		= arm_smmu_release_device,
+	.probe_finalize		= arm_smmu_probe_finalize,
  	.device_group		= arm_smmu_device_group,
  	.enable_nesting		= arm_smmu_enable_nesting,
  	.set_pgtable_quirks	= arm_smmu_set_pgtable_quirks,
drivers/iommu/arm/arm-smmu/arm-smmu.h +0 -0
index c31a59d35c64..147c95e7c59c 100644

drivers/iommu/arm/arm-smmu/arm-smmu.h struct arm_smmu_impl { · line 445

Quoted 7 lines
drivers/iommu/arm/arm-smmu/arm-smmu.h +1 -0
@@ -439,6 +439,7 @@ struct arm_smmu_impl {
  				  struct device *dev, int start);
  	void (*write_s2cr)(struct arm_smmu_device *smmu, int idx);
  	void (*write_sctlr)(struct arm_smmu_device *smmu, int idx, u32 reg);
+	void (*probe_finalize)(struct arm_smmu_device *smmu, struct device *dev);
  };
 
  #define INVALID_SMENDX			-1

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
reviewer
#10 lore raw

Re: [PATCH v3 3/9] iommu/arm-smmu: Implement ->probe_finalize()

Quoted 14 lines · Marek Szyprowski

Hi,

Quoted 7 lines · Thierry Reding

From: Thierry Reding <[email protected]>

Implement a ->probe_finalize() callback that can be used by vendor implementations to perform extra programming necessary after devices have been attached to the SMMU.

This patch landed recently in linux-next as commit 0d97174aeadf ("iommu/arm-smmu: Implement ->probe_finalize()"). It causes the following issue on ARM Juno R1 board:

[...]

Quoted 9 lines

drivers/iommu/arm/arm-smmu/arm-smmu.c static void arm_smmu_release_device(struct device *dev) · line 1461

Quoted 9 lines
drivers/iommu/arm/arm-smmu/arm-smmu.c +9 -0
+static void arm_smmu_probe_finalize(struct device *dev)
+{
+	struct arm_smmu_master_cfg *cfg;
+	struct arm_smmu_device *smmu;
+
+	cfg = dev_iommu_priv_get(dev);
+	smmu = cfg->smmu;
+
+	if (smmu->impl->probe_finalize)

Oops, indeed that needs to check smmu->impl first.

Robin.

Quoted 3 lines

drivers/iommu/arm/arm-smmu/arm-smmu.c static void arm_smmu_release_device(struct device *dev) · line 1464

Quoted 3 lines
drivers/iommu/arm/arm-smmu/arm-smmu.c +3 -0
+		smmu->impl->probe_finalize(smmu, dev);
+}
+

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

reviewer
#11 lore raw

RE: [PATCH v3 3/9] iommu/arm-smmu: Implement ->probe_finalize()

In reply to Marek Szyprowski
Quoted 1 line

if (smmu->impl->probe_finalize)

The above is the issue. It should be updated as below similar to other instances impl callbacks. if (smmu->impl && smmu->impl->probe_finalize)

-KR

maintainer
#12 lore raw

Re: [PATCH v3 3/9] iommu/arm-smmu: Implement ->probe_finalize()

Quoted 4 lines · Krishna Reddy
Quoted 1 line

if (smmu->impl->probe_finalize)

The above is the issue. It should be updated as below similar to other instances impl callbacks. if (smmu->impl && smmu->impl->probe_finalize)

I'll push a patch on top shortly...

Will

maintainer
#13 lore raw

Re: [PATCH v3 3/9] iommu/arm-smmu: Implement ->probe_finalize()

Quoted 7 lines · Will Deacon
Quoted 4 lines · Krishna Reddy
Quoted 1 line

if (smmu->impl->probe_finalize)

The above is the issue. It should be updated as below similar to other instances impl callbacks. if (smmu->impl && smmu->impl->probe_finalize)

I'll push a patch on top shortly...

Done:

https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=for-thierry/arm-smmu

I'll send this lot to Joerg tomorrow.

Thierry -- feel free to pull in the updated branch if you want the fix sooner, as it may be a few days before this hits -next.

Cheers,

Will

#14 lore raw

PATCH v3 4/9 iommu/arm-smmu: tegra: Detect number of instances at runtime

In reply to Thierry Reding

From: Thierry Reding <[email protected]>

Parse the reg property in device tree and detect the number of instances represented by a device tree node. This is subsequently needed in order to support single-instance SMMUs with the Tegra implementation because additional programming is needed to properly configure the SID override registers in the memory controller.

---
Changes in v3:
- move .probe_finalize initialization to later patch

Changes in v2:
- provide a separate implementation to simplify single instances

 drivers/iommu/arm/arm-smmu/arm-smmu-nvidia.c | 57 ++++++++++++++------
 1 file changed, 41 insertions(+), 16 deletions(-)
drivers/iommu/arm/arm-smmu/arm-smmu-nvidia.c +41 -16
index 29117444e5a0..23889090eb01 100644
@@ -20,13 +20,19 @@
  * The third instance usage is through standard arm-smmu driver itself and
  * is out of scope of this implementation.
  */
-#define NUM_SMMU_INSTANCES 2
+#define MAX_SMMU_INSTANCES 2
 
 struct nvidia_smmu {
-	struct arm_smmu_device	smmu;
-	void __iomem		*bases[NUM_SMMU_INSTANCES];
+	struct arm_smmu_device smmu;
+	void __iomem *bases[MAX_SMMU_INSTANCES];
+	unsigned int num_instances;
 };
 
+static inline struct nvidia_smmu *to_nvidia_smmu(struct arm_smmu_device *smmu)
+{
+	return container_of(smmu, struct nvidia_smmu, smmu);
+}
+
 static inline void __iomem *nvidia_smmu_page(struct arm_smmu_device *smmu,
 					     unsigned int inst, int page)
 {
@@ -47,9 +53,10 @@ static u32 nvidia_smmu_read_reg(struct arm_smmu_device *smmu,
 static void nvidia_smmu_write_reg(struct arm_smmu_device *smmu,
 				  int page, int offset, u32 val)
 {
+	struct nvidia_smmu *nvidia = to_nvidia_smmu(smmu);
 	unsigned int i;
 
-	for (i = 0; i < NUM_SMMU_INSTANCES; i++) {
+	for (i = 0; i < nvidia->num_instances; i++) {
 		void __iomem *reg = nvidia_smmu_page(smmu, i, page) + offset;
 
 		writel_relaxed(val, reg);
@@ -67,9 +74,10 @@ static u64 nvidia_smmu_read_reg64(struct arm_smmu_device *smmu,
 static void nvidia_smmu_write_reg64(struct arm_smmu_device *smmu,
 				    int page, int offset, u64 val)
 {
+	struct nvidia_smmu *nvidia = to_nvidia_smmu(smmu);
 	unsigned int i;
 
-	for (i = 0; i < NUM_SMMU_INSTANCES; i++) {
+	for (i = 0; i < nvidia->num_instances; i++) {
 		void __iomem *reg = nvidia_smmu_page(smmu, i, page) + offset;
 
 		writeq_relaxed(val, reg);
@@ -79,6 +87,7 @@ static void nvidia_smmu_write_reg64(struct arm_smmu_device *smmu,
 static void nvidia_smmu_tlb_sync(struct arm_smmu_device *smmu, int page,
 				 int sync, int status)
 {
+	struct nvidia_smmu *nvidia = to_nvidia_smmu(smmu);
 	unsigned int delay;
 
 	arm_smmu_writel(smmu, page, sync, 0);
@@ -90,7 +99,7 @@ static void nvidia_smmu_tlb_sync(struct arm_smmu_device *smmu, int page,
 			u32 val = 0;
 			unsigned int i;
 
-			for (i = 0; i < NUM_SMMU_INSTANCES; i++) {
+			for (i = 0; i < nvidia->num_instances; i++) {
 				void __iomem *reg;
 
 				reg = nvidia_smmu_page(smmu, i, page) + status;
@@ -112,9 +121,10 @@ static void nvidia_smmu_tlb_sync(struct arm_smmu_device *smmu, int page,
 
 static int nvidia_smmu_reset(struct arm_smmu_device *smmu)
 {
+	struct nvidia_smmu *nvidia = to_nvidia_smmu(smmu);
 	unsigned int i;
 
-	for (i = 0; i < NUM_SMMU_INSTANCES; i++) {
+	for (i = 0; i < nvidia->num_instances; i++) {
 		u32 val;
 		void __iomem *reg = nvidia_smmu_page(smmu, i, ARM_SMMU_GR0) +
 				    ARM_SMMU_GR0_sGFSR;
@@ -157,8 +167,9 @@ static irqreturn_t nvidia_smmu_global_fault(int irq, void *dev)
 	unsigned int inst;
 	irqreturn_t ret = IRQ_NONE;
 	struct arm_smmu_device *smmu = dev;
+	struct nvidia_smmu *nvidia = to_nvidia_smmu(smmu);
 
-	for (inst = 0; inst < NUM_SMMU_INSTANCES; inst++) {
+	for (inst = 0; inst < nvidia->num_instances; inst++) {
 		irqreturn_t irq_ret;
 
 		irq_ret = nvidia_smmu_global_fault_inst(irq, smmu, inst);
@@ -202,11 +213,13 @@ static irqreturn_t nvidia_smmu_context_fault(int irq, void *dev)
 	struct arm_smmu_device *smmu;
 	struct iommu_domain *domain = dev;
 	struct arm_smmu_domain *smmu_domain;
+	struct nvidia_smmu *nvidia;
 
 	smmu_domain = container_of(domain, struct arm_smmu_domain, domain);
 	smmu = smmu_domain->smmu;
+	nvidia = to_nvidia_smmu(smmu);
 
-	for (inst = 0; inst < NUM_SMMU_INSTANCES; inst++) {
+	for (inst = 0; inst < nvidia->num_instances; inst++) {
 		irqreturn_t irq_ret;
 
 		/*
@@ -235,12 +248,16 @@ static const struct arm_smmu_impl nvidia_smmu_impl = {
 	.context_fault = nvidia_smmu_context_fault,
 };
 
+static const struct arm_smmu_impl nvidia_smmu_single_impl = {
+};
+
 struct arm_smmu_device *nvidia_smmu_impl_init(struct arm_smmu_device *smmu)
 {
 	struct resource *res;
 	struct device *dev = smmu->dev;
 	struct nvidia_smmu *nvidia_smmu;
 	struct platform_device *pdev = to_platform_device(dev);
+	unsigned int i;
 
 	nvidia_smmu = devm_krealloc(dev, smmu, sizeof(*nvidia_smmu), GFP_KERNEL);
 	if (!nvidia_smmu)
@@ -248,16 +265,24 @@ struct arm_smmu_device *nvidia_smmu_impl_init(struct arm_smmu_device *smmu)
 
 	/* Instance 0 is ioremapped by arm-smmu.c. */
 	nvidia_smmu->bases[0] = smmu->base;
+	nvidia_smmu->num_instances++;
 
-	res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
-	if (!res)
-		return ERR_PTR(-ENODEV);
+	for (i = 1; i < MAX_SMMU_INSTANCES; i++) {
+		res = platform_get_resource(pdev, IORESOURCE_MEM, i);
+		if (!res)
+			break;
 
-	nvidia_smmu->bases[1] = devm_ioremap_resource(dev, res);
-	if (IS_ERR(nvidia_smmu->bases[1]))
-		return ERR_CAST(nvidia_smmu->bases[1]);
+		nvidia_smmu->bases[i] = devm_ioremap_resource(dev, res);
+		if (IS_ERR(nvidia_smmu->bases[i]))
+			return ERR_CAST(nvidia_smmu->bases[i]);
+
+		nvidia_smmu->num_instances++;
+	}
 
-	nvidia_smmu->smmu.impl = &nvidia_smmu_impl;
+	if (nvidia_smmu->num_instances == 1)
+		nvidia_smmu->smmu.impl = &nvidia_smmu_single_impl;
+	else
+		nvidia_smmu->smmu.impl = &nvidia_smmu_impl;
 
 	return &nvidia_smmu->smmu;
 }
Signature · 6 lines
2.31.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#15 lore raw

PATCH v3 5/9 iommu/arm-smmu: tegra: Implement SID override programming

In reply to Thierry Reding

From: Thierry Reding <[email protected]>

The secure firmware keeps some SID override registers set as passthrough in order to allow devices such as the display controller to operate with no knowledge of SMMU translations until an operating system driver takes over. This is needed in order to seamlessly transition from the firmware framebuffer to the OS framebuffer.

Upon successfully attaching a device to the SMMU and in the process creating identity mappings for memory regions that are being accessed, the Tegra implementation will call into the memory controller driver to program the override SIDs appropriately.

---
Changes in v3:
- move .probe_finalize initialization to this patch

 drivers/iommu/arm/arm-smmu/arm-smmu-nvidia.c | 33 ++++++++++++++++++--
 1 file changed, 30 insertions(+), 3 deletions(-)
drivers/iommu/arm/arm-smmu/arm-smmu-nvidia.c +30 -3
index 23889090eb01..01e9b50b10a1 100644
@@ -7,6 +7,8 @@
 #include <linux/platform_device.h>
 #include <linux/slab.h>
 
+#include <soc/tegra/mc.h>
+
 #include "arm-smmu.h"
 
 /*
@@ -15,10 +17,17 @@
  * interleaved IOVA accesses across them and translates accesses from
  * non-isochronous HW devices.
  * Third one is used for translating accesses from isochronous HW devices.
+ *
+ * In addition, the SMMU driver needs to coordinate with the memory controller
+ * driver to ensure that the right SID override is programmed for any given
+ * memory client. This is necessary to allow for use-case such as seamlessly
+ * handing over the display controller configuration from the firmware to the
+ * kernel.
+ *
  * This implementation supports programming of the two instances that must
- * be programmed identically.
- * The third instance usage is through standard arm-smmu driver itself and
- * is out of scope of this implementation.
+ * be programmed identically and takes care of invoking the memory controller
+ * driver for SID override programming after devices have been attached to an
+ * SMMU instance.
  */
 #define MAX_SMMU_INSTANCES 2
 
@@ -26,6 +35,7 @@ struct nvidia_smmu {
 	struct arm_smmu_device smmu;
 	void __iomem *bases[MAX_SMMU_INSTANCES];
 	unsigned int num_instances;
+	struct tegra_mc *mc;
 };
 
 static inline struct nvidia_smmu *to_nvidia_smmu(struct arm_smmu_device *smmu)
@@ -237,6 +247,17 @@ static irqreturn_t nvidia_smmu_context_fault(int irq, void *dev)
 	return ret;
 }
 
+static void nvidia_smmu_probe_finalize(struct arm_smmu_device *smmu, struct device *dev)
+{
+	struct nvidia_smmu *nvidia = to_nvidia_smmu(smmu);
+	int err;
+
+	err = tegra_mc_probe_device(nvidia->mc, dev);
+	if (err < 0)
+		dev_err(smmu->dev, "memory controller probe failed for %s: %d\n",
+			dev_name(dev), err);
+}
+
 static const struct arm_smmu_impl nvidia_smmu_impl = {
 	.read_reg = nvidia_smmu_read_reg,
 	.write_reg = nvidia_smmu_write_reg,
@@ -246,9 +267,11 @@ static const struct arm_smmu_impl nvidia_smmu_impl = {
 	.tlb_sync = nvidia_smmu_tlb_sync,
 	.global_fault = nvidia_smmu_global_fault,
 	.context_fault = nvidia_smmu_context_fault,
+	.probe_finalize = nvidia_smmu_probe_finalize,
 };
 
 static const struct arm_smmu_impl nvidia_smmu_single_impl = {
+	.probe_finalize = nvidia_smmu_probe_finalize,
 };
 
 struct arm_smmu_device *nvidia_smmu_impl_init(struct arm_smmu_device *smmu)
@@ -263,6 +286,10 @@ struct arm_smmu_device *nvidia_smmu_impl_init(struct arm_smmu_device *smmu)
 	if (!nvidia_smmu)
 		return ERR_PTR(-ENOMEM);
 
+	nvidia_smmu->mc = devm_tegra_memory_controller_get(dev);
+	if (IS_ERR(nvidia_smmu->mc))
+		return ERR_CAST(nvidia_smmu->mc);
+
 	/* Instance 0 is ioremapped by arm-smmu.c. */
 	nvidia_smmu->bases[0] = smmu->base;
 	nvidia_smmu->num_instances++;
Signature · 6 lines
2.31.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#16 lore raw

PATCH v3 6/9 iommu/arm-smmu: Use Tegra implementation on Tegra186

In reply to Thierry Reding

From: Thierry Reding <[email protected]>

Tegra186 requires the same SID override programming as Tegra194 in order to seamlessly transition from the firmware framebuffer to the Linux framebuffer, so the Tegra implementation needs to be used on Tegra186 devices as well.

---
 drivers/iommu/arm/arm-smmu/arm-smmu-impl.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/iommu/arm/arm-smmu/arm-smmu-impl.c +2 -1
index 136872e77195..9f465e146799 100644
@@ -211,7 +211,8 @@ struct arm_smmu_device *arm_smmu_impl_init(struct arm_smmu_device *smmu)
 	if (of_property_read_bool(np, "calxeda,smmu-secure-config-access"))
 		smmu->impl = &calxeda_impl;
 
-	if (of_device_is_compatible(np, "nvidia,tegra194-smmu"))
+	if (of_device_is_compatible(np, "nvidia,tegra194-smmu") ||
+	    of_device_is_compatible(np, "nvidia,tegra186-smmu"))
 		return nvidia_smmu_impl_init(smmu);
 
 	smmu = qcom_smmu_impl_init(smmu);
Signature · 6 lines
2.31.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#17 lore raw

PATCH v3 7/9 arm64: tegra: Use correct compatible string for Tegra186 SMMU

In reply to Thierry Reding

From: Thierry Reding <[email protected]>

The SMMU found on Tegra186 requires interoperation with the memory controller in order to program stream ID overrides. The generic ARM SMMU 500 compatible is therefore inaccurate. Replace it with a more correct, SoC-specific compatible string.

---
 arch/arm64/boot/dts/nvidia/tegra186.dtsi | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/boot/dts/nvidia/tegra186.dtsi +1 -1
index 9f75bbf00cf7..a173f40256ae 100644
@@ -1082,7 +1082,7 @@ pci@3,0 {
 	};
 
 	smmu: iommu@12000000 {
-		compatible = "arm,mmu-500";
+		compatible = "nvidia,tegra186-smmu", "nvidia,smmu-500";
 		reg = <0 0x12000000 0 0x800000>;
 		interrupts = <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
Signature · 6 lines
2.31.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#18 lore raw

PATCH v3 8/9 arm64: tegra: Hook up memory controller to SMMU on Tegra186

In reply to Thierry Reding

From: Thierry Reding <[email protected]>

On Tegra186 and later, the memory controller needs to be programmed in coordination with any of the ARM SMMU instances to configure the stream ID used for each memory client.

To support this, add a phandle reference to the memory controller to the SMMU device tree node.

---
 arch/arm64/boot/dts/nvidia/tegra186.dtsi | 2 ++
 1 file changed, 2 insertions(+)
arch/arm64/boot/dts/nvidia/tegra186.dtsi +2 -0
index a173f40256ae..d02f6bf3e2ca 100644
@@ -1152,6 +1152,8 @@ smmu: iommu@12000000 {
 		stream-match-mask = <0x7f80>;
 		#global-interrupts = <1>;
 		#iommu-cells = <1>;
+
+		nvidia,memory-controller = <&mc>;
 	};
 
 	host1x@13e00000 {
Signature · 6 lines
2.31.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#19 lore raw

PATCH v3 9/9 arm64: tegra: Enable SMMU support on Tegra194

In reply to Thierry Reding

From: Thierry Reding <[email protected]>

Add the device tree node for the dual-SMMU found on Tegra194 and hook up peripherals such as host1x, BPMP, HDA, SDMMC, EQOS and VIC.

---
 arch/arm64/boot/dts/nvidia/tegra194.dtsi | 86 ++++++++++++++++++++++++
 1 file changed, 86 insertions(+)
arch/arm64/boot/dts/nvidia/tegra194.dtsi +86 -0
index ee71e0d9f895..94e1d8f1a79f 100644
@@ -62,6 +62,7 @@ ethernet@2490000 {
 			interconnects = <&mc TEGRA194_MEMORY_CLIENT_EQOSR &emc>,
 					<&mc TEGRA194_MEMORY_CLIENT_EQOSW &emc>;
 			interconnect-names = "dma-mem", "write";
+			iommus = <&smmu TEGRA194_SID_EQOS>;
 			status = "disabled";
 
 			snps,write-requests = <1>;
@@ -733,6 +734,7 @@ sdmmc1: mmc@3400000 {
 			interconnects = <&mc TEGRA194_MEMORY_CLIENT_SDMMCRA &emc>,
 					<&mc TEGRA194_MEMORY_CLIENT_SDMMCWA &emc>;
 			interconnect-names = "dma-mem", "write";
+			iommus = <&smmu TEGRA194_SID_SDMMC1>;
 			nvidia,pad-autocal-pull-up-offset-3v3-timeout =
 									<0x07>;
 			nvidia,pad-autocal-pull-down-offset-3v3-timeout =
@@ -759,6 +761,7 @@ sdmmc3: mmc@3440000 {
 			interconnects = <&mc TEGRA194_MEMORY_CLIENT_SDMMCR &emc>,
 					<&mc TEGRA194_MEMORY_CLIENT_SDMMCW &emc>;
 			interconnect-names = "dma-mem", "write";
+			iommus = <&smmu TEGRA194_SID_SDMMC3>;
 			nvidia,pad-autocal-pull-up-offset-1v8 = <0x00>;
 			nvidia,pad-autocal-pull-down-offset-1v8 = <0x7a>;
 			nvidia,pad-autocal-pull-up-offset-3v3-timeout = <0x07>;
@@ -790,6 +793,7 @@ sdmmc4: mmc@3460000 {
 			interconnects = <&mc TEGRA194_MEMORY_CLIENT_SDMMCRAB &emc>,
 					<&mc TEGRA194_MEMORY_CLIENT_SDMMCWAB &emc>;
 			interconnect-names = "dma-mem", "write";
+			iommus = <&smmu TEGRA194_SID_SDMMC4>;
 			nvidia,pad-autocal-pull-up-offset-hs400 = <0x00>;
 			nvidia,pad-autocal-pull-down-offset-hs400 = <0x00>;
 			nvidia,pad-autocal-pull-up-offset-1v8-timeout = <0x0a>;
@@ -821,6 +825,7 @@ hda@3510000 {
 			interconnects = <&mc TEGRA194_MEMORY_CLIENT_HDAR &emc>,
 					<&mc TEGRA194_MEMORY_CLIENT_HDAW &emc>;
 			interconnect-names = "dma-mem", "write";
+			iommus = <&smmu TEGRA194_SID_HDA>;
 			status = "disabled";
 		};
 
@@ -1300,6 +1305,84 @@ pmc: pmc@c360000 {
 			interrupt-controller;
 		};
 
+		smmu: iommu@12000000 {
+			compatible = "nvidia,tegra194-smmu", "nvidia,smmu-500";
+			reg = <0x12000000 0x800000>,
+			      <0x11000000 0x800000>;
+			interrupts = <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 232 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>;
+			stream-match-mask = <0x7f80>;
+			#global-interrupts = <2>;
+			#iommu-cells = <1>;
+
+			nvidia,memory-controller = <&mc>;
+			status = "okay";
+		};
+
 		host1x@13e00000 {
 			compatible = "nvidia,tegra194-host1x";
 			reg = <0x13e00000 0x10000>,
@@ -1319,6 +1402,7 @@ host1x@13e00000 {
 			ranges = <0x15000000 0x15000000 0x01000000>;
 			interconnects = <&mc TEGRA194_MEMORY_CLIENT_HOST1XDMAR &emc>;
 			interconnect-names = "dma-mem";
+			iommus = <&smmu TEGRA194_SID_HOST1X>;
 
 			display-hub@15200000 {
 				compatible = "nvidia,tegra194-display";
@@ -1430,6 +1514,7 @@ vic@15340000 {
 				interconnects = <&mc TEGRA194_MEMORY_CLIENT_VICSRD &emc>,
 						<&mc TEGRA194_MEMORY_CLIENT_VICSWR &emc>;
 				interconnect-names = "dma-mem", "write";
+				iommus = <&smmu TEGRA194_SID_VIC>;
 			};
 
 			dpaux0: dpaux@155c0000 {
@@ -2136,6 +2221,7 @@ bpmp: bpmp {
 				<&mc TEGRA194_MEMORY_CLIENT_BPMPDMAR &emc>,
 				<&mc TEGRA194_MEMORY_CLIENT_BPMPDMAW &emc>;
 		interconnect-names = "read", "write", "dma-mem", "dma-write";
+		iommus = <&smmu TEGRA194_SID_BPMP>;
 
 		bpmp_i2c: i2c {
 			compatible = "nvidia,tegra186-bpmp-i2c";
Signature · 6 lines
2.31.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#20 lore raw

Re: (subset) [PATCH v3 0/9] arm64: tegra: Prevent early SMMU faults

In reply to Thierry Reding
Quoted 12 lines · Thierry Reding

this is a set of patches that is the result of earlier discussions regarding early identity mappings that are needed to avoid SMMU faults during early boot.

The goal here is to avoid early identity mappings altogether and instead postpone the need for the identity mappings to when devices are attached to the SMMU. This works by making the SMMU driver coordinate with the memory controller driver on when to start enforcing SMMU translations. This makes Tegra behave in a more standard way and pushes the code to deal with the Tegra-specific programming into the NVIDIA SMMU implementation.

Quoted 1 line

[...]

Applied, thanks!

[1/9] memory: tegra: Implement SID override programming

      commit: 393d66fd2cacba3e6aa95d7bb38790bfb7b1cc3a

Best regards,

Signature · 1 line
Krzysztof Kozlowski <[email protected]>
#21 lore raw
In reply to Thierry Reding
Quoted 12 lines · Thierry Reding

this is a set of patches that is the result of earlier discussions regarding early identity mappings that are needed to avoid SMMU faults during early boot.

The goal here is to avoid early identity mappings altogether and instead postpone the need for the identity mappings to when devices are attached to the SMMU. This works by making the SMMU driver coordinate with the memory controller driver on when to start enforcing SMMU translations. This makes Tegra behave in a more standard way and pushes the code to deal with the Tegra-specific programming into the NVIDIA SMMU implementation.

Quoted 1 line

[...]

Applied, thanks!

[1/9] memory: tegra: Implement SID override programming
(no commit info)
[2/9] dt-bindings: arm-smmu: Add Tegra186 compatible string

      commit: 4287861dca9d77490ee50de42aa3ada92da86c9d

[3/9] - skipped

[4/9] iommu/arm-smmu: tegra: Detect number of instances at runtime

      commit: 7ecbf253f8d64c08de28d16a66e3abbe873f6c9f
[5/9] iommu/arm-smmu: tegra: Implement SID override programming
      commit: 8eb68595475ac5fcaaa3718a173283df48cb4ef1
[6/9] iommu/arm-smmu: Use Tegra implementation on Tegra186
      commit: 2c1bc371268862a991a6498e1dddc8971b9076b8

Best regards,

Signature · 1 line
Krzysztof Kozlowski <[email protected]>
author Applied
#22 lore raw
Quoted 22 lines · Krzysztof Kozlowski
Quoted 12 lines · Thierry Reding

this is a set of patches that is the result of earlier discussions regarding early identity mappings that are needed to avoid SMMU faults during early boot.

The goal here is to avoid early identity mappings altogether and instead postpone the need for the identity mappings to when devices are attached to the SMMU. This works by making the SMMU driver coordinate with the memory controller driver on when to start enforcing SMMU translations. This makes Tegra behave in a more standard way and pushes the code to deal with the Tegra-specific programming into the NVIDIA SMMU implementation.

Quoted 1 line

[...]

Applied, thanks!

[1/9] memory: tegra: Implement SID override programming
(no commit info)
[2/9] dt-bindings: arm-smmu: Add Tegra186 compatible string

      commit: 4287861dca9d77490ee50de42aa3ada92da86c9d
Quoted 8 lines

[3/9] - skipped

[4/9] iommu/arm-smmu: tegra: Detect number of instances at runtime

      commit: 7ecbf253f8d64c08de28d16a66e3abbe873f6c9f
[5/9] iommu/arm-smmu: tegra: Implement SID override programming
      commit: 8eb68595475ac5fcaaa3718a173283df48cb4ef1
[6/9] iommu/arm-smmu: Use Tegra implementation on Tegra186
      commit: 2c1bc371268862a991a6498e1dddc8971b9076b8

I've applied patches 7-9 to the Tegra tree.

Thanks Krzysztof and Will for your help in getting this over the finish line!

Thierry

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu