Skip to contentSkip to conversation

Nvidia Arm SMMUv2 Implementation

PATCH Superseded Krishna Reddy 3 participants 15 messages Jun 4, 2020 – Jun 25, 2020

Subsystems (4)

v6 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v6 cover Nvidia Arm SMMUv2 Implementation

Changes in v6:
Restricted the patch set to driver specific patches.
Fixed the cast warning reported by kbuild test robot.
Rebased on git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git next

v5 - https://lkml.org/lkml/2020/5/21/1114
v4 - https://lkml.org/lkml/2019/10/30/1054
v3 - https://lkml.org/lkml/2019/10/18/1601
v2 - https://lkml.org/lkml/2019/9/2/980
v1 - https://lkml.org/lkml/2019/8/29/1588

Krishna Reddy (4):
iommu/arm-smmu: add NVIDIA implementation for dual ARM MMU-500 usage
dt-bindings: arm-smmu: Add binding for Tegra194 SMMU iommu/arm-smmu: Add global/context fault implementation hooks iommu/arm-smmu-nvidia: fix the warning reported by kbuild test robot

 .../devicetree/bindings/iommu/arm,smmu.yaml   |   5 +
 MAINTAINERS                                   |   2 +
 drivers/iommu/Makefile                        |   2 +-
 drivers/iommu/arm-smmu-impl.c                 |   3 +
 drivers/iommu/arm-smmu-nvidia.c               | 261 ++++++++++++++++++
 drivers/iommu/arm-smmu.c                      |  11 +-
 drivers/iommu/arm-smmu.h                      |   4 +
 7 files changed, 285 insertions(+), 3 deletions(-)
 create mode 100644 drivers/iommu/arm-smmu-nvidia.c

base-commit: 431275afdc7155415254aef4bd3816a1b8a2ead0

Signature · 1 line
2.26.2
#1 lore raw

PATCH v6 1/4 iommu/arm-smmu: add NVIDIA implementation for dual ARM MMU-500 usage

NVIDIA's Tegra194 soc uses two ARM MMU-500s together to interleave IOVA accesses across them.
Add NVIDIA implementation for dual ARM MMU-500s and add new compatible string for Tegra194 soc.

---
 MAINTAINERS                     |   2 +
 drivers/iommu/Makefile          |   2 +-
 drivers/iommu/arm-smmu-impl.c   |   3 +
 drivers/iommu/arm-smmu-nvidia.c | 161 ++++++++++++++++++++++++++++++++
 drivers/iommu/arm-smmu.h        |   1 +
 5 files changed, 168 insertions(+), 1 deletion(-)
 create mode 100644 drivers/iommu/arm-smmu-nvidia.c
MAINTAINERS +2 -0
index 50659d76976b7..118da0893c964 100644
@@ -16572,9 +16572,11 @@ F:	drivers/i2c/busses/i2c-tegra.c
 
 TEGRA IOMMU DRIVERS
 M:	Thierry Reding <[email protected]>
+R:	Krishna Reddy <[email protected]>
 L:	[email protected]
 S:	Supported
 F:	drivers/iommu/tegra*
+F:	drivers/iommu/arm-smmu-nvidia.c
 
 TEGRA KBC DRIVER
 M:	Laxman Dewangan <[email protected]>
drivers/iommu/Makefile +1 -1
index 57cf4ba5e27cb..35542df00da72 100644
@@ -15,7 +15,7 @@ obj-$(CONFIG_AMD_IOMMU) += amd_iommu.o amd_iommu_init.o amd_iommu_quirks.o
 obj-$(CONFIG_AMD_IOMMU_DEBUGFS) += amd_iommu_debugfs.o
 obj-$(CONFIG_AMD_IOMMU_V2) += amd_iommu_v2.o
 obj-$(CONFIG_ARM_SMMU) += arm_smmu.o
-arm_smmu-objs += arm-smmu.o arm-smmu-impl.o arm-smmu-qcom.o
+arm_smmu-objs += arm-smmu.o arm-smmu-impl.o arm-smmu-qcom.o arm-smmu-nvidia.o
 obj-$(CONFIG_ARM_SMMU_V3) += arm-smmu-v3.o
 obj-$(CONFIG_DMAR_TABLE) += dmar.o
 obj-$(CONFIG_INTEL_IOMMU) += intel-iommu.o intel-pasid.o
drivers/iommu/arm-smmu-impl.c +3 -0
index c75b9d957b702..52c84c30f83e4 100644
@@ -160,6 +160,9 @@ struct arm_smmu_device *arm_smmu_impl_init(struct arm_smmu_device *smmu)
 	 */
 	switch (smmu->model) {
 	case ARM_MMU500:
+		if (of_device_is_compatible(smmu->dev->of_node,
+					    "nvidia,tegra194-smmu-500"))
+			return nvidia_smmu_impl_init(smmu);
 		smmu->impl = &arm_mmu500_impl;
 		break;
 	case CAVIUM_SMMUV2:
drivers/iommu/arm-smmu-nvidia.c new file +161 -0

161 lines not on this page · raw message

drivers/iommu/arm-smmu.h +1 -0
index d172c024be618..d88374b68da31 100644
@@ -451,6 +451,7 @@ static inline void arm_smmu_writeq(struct arm_smmu_device *smmu, int page,
 
 struct arm_smmu_device *arm_smmu_impl_init(struct arm_smmu_device *smmu);
 struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu);
+struct arm_smmu_device *nvidia_smmu_impl_init(struct arm_smmu_device *smmu);
 
 int arm_mmu500_reset(struct arm_smmu_device *smmu);
 
Signature · 1 line
2.26.2
#2 lore raw

Re: [PATCH v6 1/4] iommu/arm-smmu: add NVIDIA implementation for dual ARM MMU-500 usage

Quoted 1 line · Krishna Reddy

NVIDIA's Tegra194 soc uses two ARM MMU-500s together to interleave

s/soc/SoC/

Quoted 3 lines

IOVA accesses across them.
Add NVIDIA implementation for dual ARM MMU-500s and add new compatible string for Tegra194 soc.

Same here.

Quoted 45 lines
---
 MAINTAINERS                     |   2 +
 drivers/iommu/Makefile          |   2 +-
 drivers/iommu/arm-smmu-impl.c   |   3 +
 drivers/iommu/arm-smmu-nvidia.c | 161 ++++++++++++++++++++++++++++++++
 drivers/iommu/arm-smmu.h        |   1 +
 5 files changed, 168 insertions(+), 1 deletion(-)
 create mode 100644 drivers/iommu/arm-smmu-nvidia.c
MAINTAINERS +2 -0
index 50659d76976b7..118da0893c964 100644
@@ -16572,9 +16572,11 @@ F:	drivers/i2c/busses/i2c-tegra.c
 
 TEGRA IOMMU DRIVERS
 M:	Thierry Reding <[email protected]>
+R:	Krishna Reddy <[email protected]>
 L:	[email protected]
 S:	Supported
 F:	drivers/iommu/tegra*
+F:	drivers/iommu/arm-smmu-nvidia.c
 
 TEGRA KBC DRIVER
 M:	Laxman Dewangan <[email protected]>
drivers/iommu/Makefile +1 -1
index 57cf4ba5e27cb..35542df00da72 100644
@@ -15,7 +15,7 @@ obj-$(CONFIG_AMD_IOMMU) += amd_iommu.o amd_iommu_init.o amd_iommu_quirks.o
 obj-$(CONFIG_AMD_IOMMU_DEBUGFS) += amd_iommu_debugfs.o
 obj-$(CONFIG_AMD_IOMMU_V2) += amd_iommu_v2.o
 obj-$(CONFIG_ARM_SMMU) += arm_smmu.o
-arm_smmu-objs += arm-smmu.o arm-smmu-impl.o arm-smmu-qcom.o
+arm_smmu-objs += arm-smmu.o arm-smmu-impl.o arm-smmu-qcom.o arm-smmu-nvidia.o
 obj-$(CONFIG_ARM_SMMU_V3) += arm-smmu-v3.o
 obj-$(CONFIG_DMAR_TABLE) += dmar.o
 obj-$(CONFIG_INTEL_IOMMU) += intel-iommu.o intel-pasid.o
drivers/iommu/arm-smmu-impl.c +0 -0
index c75b9d957b702..52c84c30f83e4 100644

drivers/iommu/arm-smmu-impl.c struct arm_smmu_device *arm_smmu_impl_init(struct arm_smmu_device *smmu) · line 165

Quoted 6 lines
drivers/iommu/arm-smmu-impl.c +3 -0
@@ -160,6 +160,9 @@ struct arm_smmu_device *arm_smmu_impl_init(struct arm_smmu_device *smmu)
 	 */
 	switch (smmu->model) {
 	case ARM_MMU500:
+		if (of_device_is_compatible(smmu->dev->of_node,
+					    "nvidia,tegra194-smmu-500"))
+			return nvidia_smmu_impl_init(smmu);

Should NVIDIA_TEGRA194_SMMU be a separate value for smmu->model, perhaps? That way we avoid this somewhat odd check here.

drivers/iommu/arm-smmu-nvidia.c

Quoted 4 lines
drivers/iommu/arm-smmu-impl.c +0 -0
 		smmu->impl = &arm_mmu500_impl;
 		break;
 	case CAVIUM_SMMUV2:
drivers/iommu/arm-smmu-nvidia.c +0 -0

I wonder if it would be better to name this arm-smmu-tegra.c to make it clearer that this is for a Tegra chip. We do have regular expressions in MAINTAINERS that catch anything with "tegra" in it to make this easier.

drivers/iommu/arm-smmu-nvidia.c line 2

Quoted 7 lines

new file mode 100644
index 0000000000000..dafc293a45217

drivers/iommu/arm-smmu-nvidia.c new file +2 -0
@@ -0,0 +1,161 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Nvidia ARM SMMU v2 implementation quirks

s/Nvidia/NVIDIA/

drivers/iommu/arm-smmu-nvidia.c line 3

Quoted 1 line
drivers/iommu/arm-smmu-nvidia.c new file +1 -0
+// Copyright (C) 2019 NVIDIA CORPORATION.  All rights reserved.

I suppose this should now also include 2020.

drivers/iommu/arm-smmu-nvidia.c line 5

Quoted 2 lines
drivers/iommu/arm-smmu-nvidia.c new file +2 -0
+
+#define pr_fmt(fmt) "nvidia-smmu: " fmt

Same here. Might be worth making this "tegra-smmu: " for consistency.

drivers/iommu/arm-smmu-nvidia.c line 17

Quoted 12 lines
drivers/iommu/arm-smmu-nvidia.c new file +12 -0
+
+#include <linux/bitfield.h>
+#include <linux/delay.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+
+#include "arm-smmu.h"
+
+/* Tegra194 has three ARM MMU-500 Instances.
+ * Two of them are used together for Interleaved IOVA accesses and
+ * used by Non-Isochronous Hw devices for SMMU translations.

"non-isochronous", s/Hw/HW/

drivers/iommu/arm-smmu-nvidia.c line 18

Quoted 1 line
drivers/iommu/arm-smmu-nvidia.c new file +1 -0
+ * Third one is used for SMMU translations from Isochronous HW devices.

"isochronous"

drivers/iommu/arm-smmu-nvidia.c line 19

Quoted 1 line
drivers/iommu/arm-smmu-nvidia.c new file +1 -0
+ * It is possible to use this Implementation to program either

"implementation"

drivers/iommu/arm-smmu-nvidia.c line 23

Quoted 4 lines
drivers/iommu/arm-smmu-nvidia.c new file +4 -0
+ * all three or two of the instances identically as desired through
+ * DT node.
+ *
+ * Programming all the three instances identically comes with redundant tlb

s/tlb/TLB/

drivers/iommu/arm-smmu-nvidia.c line 24

Quoted 1 line
drivers/iommu/arm-smmu-nvidia.c new file +1 -0
+ * invalidations as all three never need to be tlb invalidated for a HW device.

Same here.

drivers/iommu/arm-smmu-nvidia.c line 26

Quoted 2 lines
drivers/iommu/arm-smmu-nvidia.c new file +2 -0
+ *
+ * When Linux Kernel supports multiple SMMU devices, The SMMU device used for

"kernel" and "..., the SMMU device"

drivers/iommu/arm-smmu-nvidia.c line 27

Quoted 1 line
drivers/iommu/arm-smmu-nvidia.c new file +1 -0
+ * Isochornous HW devices should be added as a separate ARM MMU-500 device

"isochronous"

drivers/iommu/arm-smmu-nvidia.c line 28

Quoted 1 line
drivers/iommu/arm-smmu-nvidia.c new file +1 -0
+ * in DT and be programmed independently for efficient tlb invalidates.

"TLB"

drivers/iommu/arm-smmu-nvidia.c line 33

Quoted 5 lines
drivers/iommu/arm-smmu-nvidia.c new file +5 -0
+ *
+ */
+#define MAX_SMMU_INSTANCES 3
+
+#define TLB_LOOP_TIMEOUT		1000000	/* 1s! */

USEC_PER_SEC?

drivers/iommu/arm-smmu-nvidia.c line 42

Quoted 9 lines
drivers/iommu/arm-smmu-nvidia.c new file +9 -0
+#define TLB_SPIN_COUNT			10
+
+struct nvidia_smmu {
+	struct arm_smmu_device	smmu;
+	unsigned int		num_inst;
+	void __iomem		*bases[MAX_SMMU_INSTANCES];
+};
+
+#define to_nvidia_smmu(s) container_of(s, struct nvidia_smmu, smmu)

Making this static inline can make error messages more readable.

drivers/iommu/arm-smmu-nvidia.c line 46

Quoted 4 lines
drivers/iommu/arm-smmu-nvidia.c new file +4 -0
+
+#define nsmmu_page(smmu, inst, page) \
+	(((inst) ? to_nvidia_smmu(smmu)->bases[(inst)] : smmu->base) + \
+	((page) << smmu->pgshift))

Can we simply define to_nvidia_smmu(smmu)->bases[0] = smmu->base in nvidia_smmu_impl_init()? Then this would become just:

	to_nvidia_smmu(smmu)->bases[inst] + ((page) << (smmu)->pgshift)

Also, the nsmmu_ prefix looks somewhat odd here. You already use struct nvidia_smmu as the name of the structure, so why not be consistent and continue to use nvidia_smmu_ as the prefix for function names?

Or perhaps even use tegra_smmu_ as the prefix to match the filename change I suggested earlier.

Quoted 25 lines
drivers/iommu/arm-smmu-nvidia.c new file +25 -0
+
+static u32 nsmmu_read_reg(struct arm_smmu_device *smmu,
+			      int page, int offset)
+{
+	return readl_relaxed(nsmmu_page(smmu, 0, page) + offset);
+}
+
+static void nsmmu_write_reg(struct arm_smmu_device *smmu,
+			    int page, int offset, u32 val)
+{
+	unsigned int i;
+
+	for (i = 0; i < to_nvidia_smmu(smmu)->num_inst; i++)
+		writel_relaxed(val, nsmmu_page(smmu, i, page) + offset);
+}
+
+static u64 nsmmu_read_reg64(struct arm_smmu_device *smmu,
+				int page, int offset)
+{
+	return readq_relaxed(nsmmu_page(smmu, 0, page) + offset);
+}
+
+static void nsmmu_write_reg64(struct arm_smmu_device *smmu,
+				  int page, int offset, u64 val)
+{

drivers/iommu/arm-smmu-nvidia.c line 81

Quoted 10 lines
drivers/iommu/arm-smmu-nvidia.c new file +10 -0
+	unsigned int i;
+
+	for (i = 0; i < to_nvidia_smmu(smmu)->num_inst; i++)
+		writeq_relaxed(val, nsmmu_page(smmu, i, page) + offset);
+}
+
+static void nsmmu_tlb_sync(struct arm_smmu_device *smmu, int page,
+			   int sync, int status)
+{
+	u32 reg;

I see this is being used to store the value read from a register. I find it clearer to call this "value" or "val" (or in this case perhaps even "status") because whenever I read "reg" I immediately think this is meant to be a register offset, which can then be confusing when I see it used in I/O accessors because it is in the wrong position.

But anyway, that's just my opinion and this is a bit subjective, so feel free to ignore.

drivers/iommu/arm-smmu-nvidia.c line 89

Quoted 8 lines
drivers/iommu/arm-smmu-nvidia.c new file +8 -0
+	unsigned int i;
+	unsigned int spin_cnt, delay;
+
+	arm_smmu_writel(smmu, page, sync, 0);
+
+	for (delay = 1; delay < TLB_LOOP_TIMEOUT; delay *= 2) {
+		for (spin_cnt = TLB_SPIN_COUNT; spin_cnt > 0; spin_cnt--) {
+			reg = 0;

You may want to declare the variable at this scope since you never need it outside. Also, use a space between variable initialization and the for block below for better readability.

drivers/iommu/arm-smmu-nvidia.c line 93

Quoted 4 lines
drivers/iommu/arm-smmu-nvidia.c new file +4 -0
+			for (i = 0; i < to_nvidia_smmu(smmu)->num_inst; i++) {
+				reg |= readl_relaxed(
+					nsmmu_page(smmu, i, page) + status);
+			}

Maybe add a separate variable for the page address so this can be a bit uncluttered. Also, I'd prefer a blank line after the block for readability.

drivers/iommu/arm-smmu-nvidia.c line 96

Quoted 3 lines
drivers/iommu/arm-smmu-nvidia.c new file +3 -0
+			if (!(reg & ARM_SMMU_sTLBGSTATUS_GSACTIVE))
+				return;
+			cpu_relax();

Blank line above cpu_relax() for readability.

drivers/iommu/arm-smmu-nvidia.c line 98

Quoted 2 lines
drivers/iommu/arm-smmu-nvidia.c new file +2 -0
+		}
+		udelay(delay);

Again, a blank line between blocks and subsequent statements can help readability.

drivers/iommu/arm-smmu-nvidia.c line 101

Quoted 3 lines
drivers/iommu/arm-smmu-nvidia.c new file +3 -0
+	}
+	dev_err_ratelimited(smmu->dev,
+			    "TLB sync timed out -- SMMU may be deadlocked\n");

Same here.

Also, is there anything we can do when this happens?

Quoted 32 lines
drivers/iommu/arm-smmu-nvidia.c new file +32 -0
+}
+
+static int nsmmu_reset(struct arm_smmu_device *smmu)
+{
+	u32 reg;
+	unsigned int i;
+
+	for (i = 0; i < to_nvidia_smmu(smmu)->num_inst; i++) {
+		/* clear global FSR */
+		reg = readl_relaxed(nsmmu_page(smmu, i, ARM_SMMU_GR0) +
+				    ARM_SMMU_GR0_sGFSR);
+		writel_relaxed(reg, nsmmu_page(smmu, i, ARM_SMMU_GR0) +
+				    ARM_SMMU_GR0_sGFSR);
+	}
+
+	return 0;
+}
+
+static const struct arm_smmu_impl nvidia_smmu_impl = {
+	.read_reg = nsmmu_read_reg,
+	.write_reg = nsmmu_write_reg,
+	.read_reg64 = nsmmu_read_reg64,
+	.write_reg64 = nsmmu_write_reg64,
+	.reset = nsmmu_reset,
+	.tlb_sync = nsmmu_tlb_sync,
+};
+
+struct arm_smmu_device *nvidia_smmu_impl_init(struct arm_smmu_device *smmu)
+{
+	unsigned int i;
+	struct nvidia_smmu *nsmmu;
+	struct resource *res;

drivers/iommu/arm-smmu-nvidia.c line 143

Quoted 10 lines
drivers/iommu/arm-smmu-nvidia.c new file +10 -0
+	struct device *dev = smmu->dev;
+	struct platform_device *pdev = to_platform_device(smmu->dev);
+
+	nsmmu = devm_kzalloc(smmu->dev, sizeof(*nsmmu), GFP_KERNEL);
+	if (!nsmmu)
+		return ERR_PTR(-ENOMEM);
+
+	nsmmu->smmu = *smmu;
+	/* Instance 0 is ioremapped by arm-smmu.c */
+	nsmmu->num_inst = 1;

Maybe add this here to simplify the nsmmu_page() macro above:

	nsmmu->bases[0] = smmu->base;

drivers/iommu/arm-smmu-nvidia.c line 151

Quoted 8 lines
drivers/iommu/arm-smmu-nvidia.c new file +8 -0
+
+	for (i = 1; i < MAX_SMMU_INSTANCES; i++) {
+		res = platform_get_resource(pdev, IORESOURCE_MEM, i);
+		if (!res)
+			break;
+		nsmmu->bases[i] = devm_ioremap_resource(dev, res);
+		if (IS_ERR(nsmmu->bases[i]))
+			return (struct arm_smmu_device *)nsmmu->bases[i];

ERR_CAST()?

drivers/iommu/arm-smmu-nvidia.c line 153

Quoted 2 lines
drivers/iommu/arm-smmu-nvidia.c new file +2 -0
+		nsmmu->num_inst++;
+	}

More blank lines would make this much easier to read, in my opinion.

drivers/iommu/arm-smmu-nvidia.c line 156

Quoted 3 lines
drivers/iommu/arm-smmu-nvidia.c new file +3 -0
+
+	nsmmu->smmu.impl = &nvidia_smmu_impl;
+	devm_kfree(smmu->dev, smmu);

Maybe a comment here would be useful for readers to immediately understand why you're doing this here.

drivers/iommu/arm-smmu-nvidia.c line 158

Quoted 2 lines
drivers/iommu/arm-smmu-nvidia.c new file +2 -0
+	pr_info("NVIDIA ARM SMMU Implementation, Instances=%d\n",
+		nsmmu->num_inst);

I think I'd just omit this. In general you should only let the user know when things go wrong, but the above is printed when everything goes as expected.

Thierry

reviewer
#3 lore raw

Re: [PATCH v6 1/4] iommu/arm-smmu: add NVIDIA implementation for dual ARM MMU-500 usage

On 2020-06-23 11:29, Thierry Reding wrote:
[...]

Quoted 6 lines
Quoted 6 lines
drivers/iommu/arm-smmu-impl.c +0 -0
index c75b9d957b702..52c84c30f83e4 100644
@@ -160,6 +160,9 @@ struct arm_smmu_device *arm_smmu_impl_init(struct arm_smmu_device *smmu)
  	 */
Quoted 8 lines

drivers/iommu/arm-smmu-impl.c struct arm_smmu_device *arm_smmu_impl_init(struct arm_smmu_device *smmu) · line 165

Quoted 5 lines
drivers/iommu/arm-smmu-impl.c +3 -0
  	switch (smmu->model) {
  	case ARM_MMU500:
+		if (of_device_is_compatible(smmu->dev->of_node,
+					    "nvidia,tegra194-smmu-500"))
+			return nvidia_smmu_impl_init(smmu);

Should NVIDIA_TEGRA194_SMMU be a separate value for smmu->model, perhaps? That way we avoid this somewhat odd check here.

No, this is simply in the wrong place. The design here is that we pick up anything related to the basic SMMU IP (model) first, then make any platform-specific integration checks. That way a platform-specific init function can see the model impl set and subclass it if necessary (although nobody's actually done that yet). The setup for Cavium is just a short-cut since their model is unique to their integration, so the lines get a bit blurred and there's little benefit to trying to separate it out.

In short, put this down below with the other of_device_is_compatible() checks.

Quoted 8 lines

drivers/iommu/arm-smmu-nvidia.c

Quoted 4 lines
drivers/iommu/arm-smmu-impl.c +0 -0
  		smmu->impl = &arm_mmu500_impl;
  		break;
  	case CAVIUM_SMMUV2:
drivers/iommu/arm-smmu-nvidia.c +0 -0

I wonder if it would be better to name this arm-smmu-tegra.c to make it clearer that this is for a Tegra chip. We do have regular expressions in MAINTAINERS that catch anything with "tegra" in it to make this easier.

There was a notion that these would be grouped by vendor, but if there's a strong preference for all NVIDIA-SoC-related stuff to be named "Tegra" then I'm not going to complain too much.

Quoted 10 lines

drivers/iommu/arm-smmu-nvidia.c line 2

Quoted 7 lines

new file mode 100644
index 0000000000000..dafc293a45217

drivers/iommu/arm-smmu-nvidia.c new file +2 -0
@@ -0,0 +1,161 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Nvidia ARM SMMU v2 implementation quirks

s/Nvidia/NVIDIA/

Quoted 8 lines

drivers/iommu/arm-smmu-nvidia.c line 3

Quoted 1 line
drivers/iommu/arm-smmu-nvidia.c new file +1 -0
+// Copyright (C) 2019 NVIDIA CORPORATION.  All rights reserved.

I suppose this should now also include 2020.

drivers/iommu/arm-smmu-nvidia.c line 5

Quoted 2 lines
drivers/iommu/arm-smmu-nvidia.c new file +2 -0
+
+#define pr_fmt(fmt) "nvidia-smmu: " fmt

Same here. Might be worth making this "tegra-smmu: " for consistency.

On the other hand, a log prefix that is literally the name of a completely unrelated driver seems more confusing to users than useful. Same for the function naming - the tegra_smmu_* namespace is already owned by that driver.

Robin.

#4 lore raw

Re: [PATCH v6 1/4] iommu/arm-smmu: add NVIDIA implementation for dual ARM MMU-500 usage

Quoted 37 lines · Robin Murphy

On 2020-06-23 11:29, Thierry Reding wrote:
[...]

Quoted 6 lines
Quoted 6 lines
drivers/iommu/arm-smmu-impl.c +0 -0
index c75b9d957b702..52c84c30f83e4 100644
@@ -160,6 +160,9 @@ struct arm_smmu_device *arm_smmu_impl_init(struct arm_smmu_device *smmu)
  	 */
Quoted 8 lines

drivers/iommu/arm-smmu-impl.c struct arm_smmu_device *arm_smmu_impl_init(struct arm_smmu_device *smmu) · line 165

Quoted 5 lines
drivers/iommu/arm-smmu-impl.c +3 -0
  	switch (smmu->model) {
  	case ARM_MMU500:
+		if (of_device_is_compatible(smmu->dev->of_node,
+					    "nvidia,tegra194-smmu-500"))
+			return nvidia_smmu_impl_init(smmu);

Should NVIDIA_TEGRA194_SMMU be a separate value for smmu->model, perhaps? That way we avoid this somewhat odd check here.

No, this is simply in the wrong place. The design here is that we pick up anything related to the basic SMMU IP (model) first, then make any platform-specific integration checks. That way a platform-specific init function can see the model impl set and subclass it if necessary (although nobody's actually done that yet). The setup for Cavium is just a short-cut since their model is unique to their integration, so the lines get a bit blurred and there's little benefit to trying to separate it out.

In short, put this down below with the other of_device_is_compatible() checks.

Quoted 8 lines

drivers/iommu/arm-smmu-nvidia.c

Quoted 4 lines
drivers/iommu/arm-smmu-impl.c +0 -0
  		smmu->impl = &arm_mmu500_impl;
  		break;
  	case CAVIUM_SMMUV2:
drivers/iommu/arm-smmu-nvidia.c +0 -0

I wonder if it would be better to name this arm-smmu-tegra.c to make it clearer that this is for a Tegra chip. We do have regular expressions in MAINTAINERS that catch anything with "tegra" in it to make this easier.

Quoted 3 lines

There was a notion that these would be grouped by vendor, but if there's a strong preference for all NVIDIA-SoC-related stuff to be named "Tegra" then I'm not going to complain too much.

Maybe I was being overly cautious. I was just trying to avoid adding something called nvidia-arm-smmu which might eventually turn out to be ambiguous if there was ever a non-Tegra chip and the ARM SMMU implementation was not compatible with the one instantiated on Tegra.

Note that I have no knowledge of such a chip being designed, so this may never actually become an issue.

In either case, the compatible string already identifies this as Tegra- specific, so we could always change the driver name later on if we have to.

Quoted 19 lines
Quoted 10 lines

drivers/iommu/arm-smmu-nvidia.c line 2

Quoted 7 lines

new file mode 100644
index 0000000000000..dafc293a45217

drivers/iommu/arm-smmu-nvidia.c new file +2 -0
@@ -0,0 +1,161 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Nvidia ARM SMMU v2 implementation quirks

s/Nvidia/NVIDIA/

Quoted 8 lines

drivers/iommu/arm-smmu-nvidia.c line 3

Quoted 1 line
drivers/iommu/arm-smmu-nvidia.c new file +1 -0
+// Copyright (C) 2019 NVIDIA CORPORATION.  All rights reserved.

I suppose this should now also include 2020.

drivers/iommu/arm-smmu-nvidia.c line 5

Quoted 2 lines
drivers/iommu/arm-smmu-nvidia.c new file +2 -0
+
+#define pr_fmt(fmt) "nvidia-smmu: " fmt

Same here. Might be worth making this "tegra-smmu: " for consistency.

Quoted 4 lines

On the other hand, a log prefix that is literally the name of a completely unrelated driver seems more confusing to users than useful. Same for the function naming - the tegra_smmu_* namespace is already owned by that driver.

The ARM SMMU replaced the Tegra SMMU on Tegra186 and later, so both drivers are never going to run concurrently. The only "problem" might be that both drivers have symbols with the same prefix, but since they don't export any of those symbols I don't see how that would become a real issue.

But then again, the Tegra SMMU is also technically an NVIDIA SMMU, so sticking with the current name might also be confusing.

Perhaps a good compromise would be to use a "tegra194{-,_}smmu" prefix instead, which would make this fairly specific to just Tegra194 (and compatible chips). That's a fairly common pattern we've been following on Tegra, as you can for example see in drivers/gpio/gpio-tegra186.c, drivers/dma/tegra210-adma.c, drivers/memory/tegra/tegra186-emc.c, etc.

Thierry

#5 lore raw

RE: [PATCH v6 1/4] iommu/arm-smmu: add NVIDIA implementation for dual ARM MMU-500 usage

In reply to Thierry Reding
Quoted 1 line

Should NVIDIA_TEGRA194_SMMU be a separate value for smmu->model, perhaps? That way we avoid this somewhat odd check here.

NVIDIA haven't made any changes to arm,mmu-500. It is only used in different topology. New model would be mis-leading here. As suggested by Robin, It can just be moved to end of function.

Quoted 5 lines

drivers/iommu/arm-smmu-nvidia.c

Quoted 2 lines
drivers/iommu/arm-smmu-nvidia.c +0 -0

I wonder if it would be better to name this arm-smmu-tegra.c to make it clearer that this is for a Tegra chip. We do have regular expressions in MAINTAINERS that catch anything with "tegra" in it to make this easier. Also, the nsmmu_ prefix looks somewhat odd here. You already use struct nvidia_smmu as the name of the structure, so why not be consistent and continue to use nvidia_smmu_ as the prefix for function names? Or perhaps even use tegra_smmu_ as the prefix to match the filename change I suggested earlier.

Prefix can be updated to nvidia_smmu as we seem to be okay for now to keep file name as arm-smmu-nvidia.c after the vendor name.

Quoted 2 lines
Quoted 1 line

+#define TLB_LOOP_TIMEOUT 1000000 /* 1s! */

USEC_PER_SEC?

It is not meant for a conversion. Reused Timeout variable from arm-smmu.c for tlb_sync implementation. Can rename it to TLB_LOOP_TIMEOUT_IN_US.

Quoted 5 lines

drivers/iommu/arm-smmu-nvidia.c line 101

Quoted 3 lines
drivers/iommu/arm-smmu-nvidia.c +3 -0
+	}
+	dev_err_ratelimited(smmu->dev,
+			    "TLB sync timed out -- SMMU may be deadlocked\n");

Same here.
Also, is there anything we can do when this happens?

This is never expected to happen on Silicon. This code and message is reused from arm-smmu.c.

Quoted 9 lines

drivers/iommu/arm-smmu-nvidia.c line 46

Quoted 3 lines
drivers/iommu/arm-smmu-nvidia.c +3 -0
+#define nsmmu_page(smmu, inst, page) \
+	(((inst) ? to_nvidia_smmu(smmu)->bases[(inst)] : smmu->base) + \
+	((page) << smmu->pgshift))

Can we simply define to_nvidia_smmu(smmu)->bases[0] = smmu->base in nvidia_smmu_impl_init()? Then this would become just:

	to_nvidia_smmu(smmu)->bases[inst] + ((page) << (smmu)->pgshift)
+
Maybe add this here to simplify the nsmmu_page() macro above:
	nsmmu->bases[0] = smmu->base;

This preferred to avoid the check in nsmmu_page(). But, smmu->base is not yet populated when nvidia_smmu_impl_init() is called. Let me look at the alternative place to set it.

-KR

#6 lore raw

PATCH v6 2/4 dt-bindings: arm-smmu: Add binding for Tegra194 SMMU

In reply to Krishna Reddy

Add binding for NVIDIA's Tegra194 Soc SMMU that is based on ARM MMU-500.

---
 Documentation/devicetree/bindings/iommu/arm,smmu.yaml | 5 +++++
 1 file changed, 5 insertions(+)
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +5 -0
index e3ef1c69d1326..8f7ffd248f303 100644
@@ -37,6 +37,11 @@ properties:
               - qcom,sc7180-smmu-500
               - qcom,sdm845-smmu-500
           - const: arm,mmu-500
+      - description: NVIDIA SoCs that use more than one "arm,mmu-500"
+        items:
+          - enum:
+              - nvdia,tegra194-smmu-500
+          - const: arm,mmu-500
       - items:
           - const: arm,mmu-500
           - const: arm,smmu-v2
Signature · 1 line
2.26.2
#7 lore raw

Re: [PATCH v6 2/4] dt-bindings: arm-smmu: Add binding for Tegra194 SMMU

Quoted 13 lines · Krishna Reddy

Add binding for NVIDIA's Tegra194 Soc SMMU that is based on ARM MMU-500.

---
 Documentation/devicetree/bindings/iommu/arm,smmu.yaml | 5 +++++
 1 file changed, 5 insertions(+)
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +0 -0
index e3ef1c69d1326..8f7ffd248f303 100644

Documentation/devicetree/bindings/iommu/arm,smmu.yaml properties: · line 43

Quoted 7 lines
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +4 -0
@@ -37,6 +37,11 @@ properties:
               - qcom,sc7180-smmu-500
               - qcom,sdm845-smmu-500
           - const: arm,mmu-500
+      - description: NVIDIA SoCs that use more than one "arm,mmu-500"
+        items:
+          - enum:
+              - nvdia,tegra194-smmu-500

The -500 suffix here seems a bit redundant since there's no other type of SMMU in Tegra194, correct?

Thierry

#8 lore raw

RE: [PATCH v6 2/4] dt-bindings: arm-smmu: Add binding for Tegra194 SMMU

Quoted 2 lines
Quoted 1 line

+ - nvdia,tegra194-smmu-500

The -500 suffix here seems a bit redundant since there's no other type of SMMU in Tegra194, correct?

Yeah, there is only one type of SMMU supported in T194. It was added to be synonymous with mmu-500. Can be removed.

-KR

#9 lore raw

PATCH v6 3/4 iommu/arm-smmu: Add global/context fault implementation hooks

In reply to Krishna Reddy

Add global/context fault hooks to allow NVIDIA SMMU implementation handle faults across multiple SMMUs.

---
 drivers/iommu/arm-smmu-nvidia.c | 100 ++++++++++++++++++++++++++++++++
 drivers/iommu/arm-smmu.c        |  11 +++-
 drivers/iommu/arm-smmu.h        |   3 +
 3 files changed, 112 insertions(+), 2 deletions(-)
drivers/iommu/arm-smmu-nvidia.c +100 -0
index dafc293a45217..5999b6a770992 100644
@@ -117,6 +117,104 @@ static int nsmmu_reset(struct arm_smmu_device *smmu)
 	return 0;
 }
 
+static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
+{
+	return container_of(dom, struct arm_smmu_domain, domain);
+}
+
+static irqreturn_t nsmmu_global_fault_inst(int irq,
+					       struct arm_smmu_device *smmu,
+					       int inst)
+{
+	u32 gfsr, gfsynr0, gfsynr1, gfsynr2;
+
+	gfsr = readl_relaxed(nsmmu_page(smmu, inst, 0) + ARM_SMMU_GR0_sGFSR);
+	gfsynr0 = readl_relaxed(nsmmu_page(smmu, inst, 0) +
+				ARM_SMMU_GR0_sGFSYNR0);
+	gfsynr1 = readl_relaxed(nsmmu_page(smmu, inst, 0) +
+				ARM_SMMU_GR0_sGFSYNR1);
+	gfsynr2 = readl_relaxed(nsmmu_page(smmu, inst, 0) +
+				ARM_SMMU_GR0_sGFSYNR2);
+
+	if (!gfsr)
+		return IRQ_NONE;
+
+	dev_err_ratelimited(smmu->dev,
+		"Unexpected global fault, this could be serious\n");
+	dev_err_ratelimited(smmu->dev,
+		"\tGFSR 0x%08x, GFSYNR0 0x%08x, GFSYNR1 0x%08x, GFSYNR2 0x%08x\n",
+		gfsr, gfsynr0, gfsynr1, gfsynr2);
+
+	writel_relaxed(gfsr, nsmmu_page(smmu, inst, 0) + ARM_SMMU_GR0_sGFSR);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t nsmmu_global_fault(int irq, void *dev)
+{
+	int inst;
+	irqreturn_t irq_ret = IRQ_NONE;
+	struct arm_smmu_device *smmu = dev;
+
+	for (inst = 0; inst < to_nvidia_smmu(smmu)->num_inst; inst++) {
+		irq_ret = nsmmu_global_fault_inst(irq, smmu, inst);
+		if (irq_ret == IRQ_HANDLED)
+			return irq_ret;
+	}
+
+	return irq_ret;
+}
+
+static irqreturn_t nsmmu_context_fault_bank(int irq,
+					    struct arm_smmu_device *smmu,
+					    int idx, int inst)
+{
+	u32 fsr, fsynr, cbfrsynra;
+	unsigned long iova;
+
+	fsr = arm_smmu_cb_read(smmu, idx, ARM_SMMU_CB_FSR);
+	if (!(fsr & ARM_SMMU_FSR_FAULT))
+		return IRQ_NONE;
+
+	fsynr = readl_relaxed(nsmmu_page(smmu, inst, smmu->numpage + idx) +
+			      ARM_SMMU_CB_FSYNR0);
+	iova = readq_relaxed(nsmmu_page(smmu, inst, smmu->numpage + idx) +
+			     ARM_SMMU_CB_FAR);
+	cbfrsynra = readl_relaxed(nsmmu_page(smmu, inst, 1) +
+				  ARM_SMMU_GR1_CBFRSYNRA(idx));
+
+	dev_err_ratelimited(smmu->dev,
+	"Unhandled context fault: fsr=0x%x, iova=0x%08lx, fsynr=0x%x, cbfrsynra=0x%x, cb=%d\n",
+			    fsr, iova, fsynr, cbfrsynra, idx);
+
+	writel_relaxed(fsr, nsmmu_page(smmu, inst, smmu->numpage + idx) +
+			    ARM_SMMU_CB_FSR);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t nsmmu_context_fault(int irq, void *dev)
+{
+	int inst, idx;
+	irqreturn_t irq_ret = IRQ_NONE;
+	struct iommu_domain *domain = dev;
+	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+	struct arm_smmu_device *smmu = smmu_domain->smmu;
+
+	for (inst = 0; inst < to_nvidia_smmu(smmu)->num_inst; inst++) {
+		/* Interrupt line shared between all context faults.
+		 * Check for faults across all contexts.
+		 */
+		for (idx = 0; idx < smmu->num_context_banks; idx++) {
+			irq_ret = nsmmu_context_fault_bank(irq, smmu,
+							   idx, inst);
+
+			if (irq_ret == IRQ_HANDLED)
+				return irq_ret;
+		}
+	}
+
+	return irq_ret;
+}
+
 static const struct arm_smmu_impl nvidia_smmu_impl = {
 	.read_reg = nsmmu_read_reg,
 	.write_reg = nsmmu_write_reg,
@@ -124,6 +222,8 @@ static const struct arm_smmu_impl nvidia_smmu_impl = {
 	.write_reg64 = nsmmu_write_reg64,
 	.reset = nsmmu_reset,
 	.tlb_sync = nsmmu_tlb_sync,
+	.global_fault = nsmmu_global_fault,
+	.context_fault = nsmmu_context_fault,
 };
 
 struct arm_smmu_device *nvidia_smmu_impl_init(struct arm_smmu_device *smmu)
drivers/iommu/arm-smmu.c +9 -2
index 243bc4cb2705b..d720e1e191176 100644
@@ -673,6 +673,7 @@ static int arm_smmu_init_domain_context(struct iommu_domain *domain,
 	enum io_pgtable_fmt fmt;
 	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
 	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+	irqreturn_t (*context_fault)(int irq, void *dev);
 
 	mutex_lock(&smmu_domain->init_mutex);
 	if (smmu_domain->smmu)
@@ -835,7 +836,9 @@ static int arm_smmu_init_domain_context(struct iommu_domain *domain,
 	 * handler seeing a half-initialised domain state.
 	 */
 	irq = smmu->irqs[smmu->num_global_irqs + cfg->irptndx];
-	ret = devm_request_irq(smmu->dev, irq, arm_smmu_context_fault,
+	context_fault = (smmu->impl && smmu->impl->context_fault) ?
+			 smmu->impl->context_fault : arm_smmu_context_fault;
+	ret = devm_request_irq(smmu->dev, irq, context_fault,
 			       IRQF_SHARED, "arm-smmu-context-fault", domain);
 	if (ret < 0) {
 		dev_err(smmu->dev, "failed to request context IRQ %d (%u)\n",
@@ -2107,6 +2110,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	struct arm_smmu_device *smmu;
 	struct device *dev = &pdev->dev;
 	int num_irqs, i, err;
+	irqreturn_t (*global_fault)(int irq, void *dev);
 
 	smmu = devm_kzalloc(dev, sizeof(*smmu), GFP_KERNEL);
 	if (!smmu) {
@@ -2193,9 +2197,12 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 		smmu->num_context_irqs = smmu->num_context_banks;
 	}
 
+	global_fault = (smmu->impl && smmu->impl->global_fault) ?
+			smmu->impl->global_fault : arm_smmu_global_fault;
+
 	for (i = 0; i < smmu->num_global_irqs; ++i) {
 		err = devm_request_irq(smmu->dev, smmu->irqs[i],
-				       arm_smmu_global_fault,
+				       global_fault,
 				       IRQF_SHARED,
 				       "arm-smmu global fault",
 				       smmu);
drivers/iommu/arm-smmu.h +3 -0
index d88374b68da31..f8ce34c07d961 100644
@@ -18,6 +18,7 @@
 #include <linux/io-64-nonatomic-hi-lo.h>
 #include <linux/io-pgtable.h>
 #include <linux/iommu.h>
+#include <linux/irqreturn.h>
 #include <linux/mutex.h>
 #include <linux/spinlock.h>
 #include <linux/types.h>
@@ -387,6 +388,8 @@ struct arm_smmu_impl {
 	void (*tlb_sync)(struct arm_smmu_device *smmu, int page, int sync,
 			 int status);
 	int (*def_domain_type)(struct device *dev);
+	irqreturn_t (*global_fault)(int irq, void *dev);
+	irqreturn_t (*context_fault)(int irq, void *dev);
 };
 
 static inline void __iomem *arm_smmu_page(struct arm_smmu_device *smmu, int n)
Signature · 1 line
2.26.2
#10 lore raw

Re: [PATCH v6 3/4] iommu/arm-smmu: Add global/context fault implementation hooks

Quoted 141 lines · Krishna Reddy

Add global/context fault hooks to allow NVIDIA SMMU implementation handle faults across multiple SMMUs.

---
 drivers/iommu/arm-smmu-nvidia.c | 100 ++++++++++++++++++++++++++++++++
 drivers/iommu/arm-smmu.c        |  11 +++-
 drivers/iommu/arm-smmu.h        |   3 +
 3 files changed, 112 insertions(+), 2 deletions(-)
drivers/iommu/arm-smmu-nvidia.c +100 -0
index dafc293a45217..5999b6a770992 100644
@@ -117,6 +117,104 @@ static int nsmmu_reset(struct arm_smmu_device *smmu)
 	return 0;
 }
 
+static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
+{
+	return container_of(dom, struct arm_smmu_domain, domain);
+}
+
+static irqreturn_t nsmmu_global_fault_inst(int irq,
+					       struct arm_smmu_device *smmu,
+					       int inst)
+{
+	u32 gfsr, gfsynr0, gfsynr1, gfsynr2;
+
+	gfsr = readl_relaxed(nsmmu_page(smmu, inst, 0) + ARM_SMMU_GR0_sGFSR);
+	gfsynr0 = readl_relaxed(nsmmu_page(smmu, inst, 0) +
+				ARM_SMMU_GR0_sGFSYNR0);
+	gfsynr1 = readl_relaxed(nsmmu_page(smmu, inst, 0) +
+				ARM_SMMU_GR0_sGFSYNR1);
+	gfsynr2 = readl_relaxed(nsmmu_page(smmu, inst, 0) +
+				ARM_SMMU_GR0_sGFSYNR2);
+
+	if (!gfsr)
+		return IRQ_NONE;
+
+	dev_err_ratelimited(smmu->dev,
+		"Unexpected global fault, this could be serious\n");
+	dev_err_ratelimited(smmu->dev,
+		"\tGFSR 0x%08x, GFSYNR0 0x%08x, GFSYNR1 0x%08x, GFSYNR2 0x%08x\n",
+		gfsr, gfsynr0, gfsynr1, gfsynr2);
+
+	writel_relaxed(gfsr, nsmmu_page(smmu, inst, 0) + ARM_SMMU_GR0_sGFSR);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t nsmmu_global_fault(int irq, void *dev)
+{
+	int inst;
+	irqreturn_t irq_ret = IRQ_NONE;
+	struct arm_smmu_device *smmu = dev;
+
+	for (inst = 0; inst < to_nvidia_smmu(smmu)->num_inst; inst++) {
+		irq_ret = nsmmu_global_fault_inst(irq, smmu, inst);
+		if (irq_ret == IRQ_HANDLED)
+			return irq_ret;
+	}
+
+	return irq_ret;
+}
+
+static irqreturn_t nsmmu_context_fault_bank(int irq,
+					    struct arm_smmu_device *smmu,
+					    int idx, int inst)
+{
+	u32 fsr, fsynr, cbfrsynra;
+	unsigned long iova;
+
+	fsr = arm_smmu_cb_read(smmu, idx, ARM_SMMU_CB_FSR);
+	if (!(fsr & ARM_SMMU_FSR_FAULT))
+		return IRQ_NONE;
+
+	fsynr = readl_relaxed(nsmmu_page(smmu, inst, smmu->numpage + idx) +
+			      ARM_SMMU_CB_FSYNR0);
+	iova = readq_relaxed(nsmmu_page(smmu, inst, smmu->numpage + idx) +
+			     ARM_SMMU_CB_FAR);
+	cbfrsynra = readl_relaxed(nsmmu_page(smmu, inst, 1) +
+				  ARM_SMMU_GR1_CBFRSYNRA(idx));
+
+	dev_err_ratelimited(smmu->dev,
+	"Unhandled context fault: fsr=0x%x, iova=0x%08lx, fsynr=0x%x, cbfrsynra=0x%x, cb=%d\n",
+			    fsr, iova, fsynr, cbfrsynra, idx);
+
+	writel_relaxed(fsr, nsmmu_page(smmu, inst, smmu->numpage + idx) +
+			    ARM_SMMU_CB_FSR);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t nsmmu_context_fault(int irq, void *dev)
+{
+	int inst, idx;
+	irqreturn_t irq_ret = IRQ_NONE;
+	struct iommu_domain *domain = dev;
+	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+	struct arm_smmu_device *smmu = smmu_domain->smmu;
+
+	for (inst = 0; inst < to_nvidia_smmu(smmu)->num_inst; inst++) {
+		/* Interrupt line shared between all context faults.
+		 * Check for faults across all contexts.
+		 */
+		for (idx = 0; idx < smmu->num_context_banks; idx++) {
+			irq_ret = nsmmu_context_fault_bank(irq, smmu,
+							   idx, inst);
+
+			if (irq_ret == IRQ_HANDLED)
+				return irq_ret;
+		}
+	}
+
+	return irq_ret;
+}
+
 static const struct arm_smmu_impl nvidia_smmu_impl = {
 	.read_reg = nsmmu_read_reg,
 	.write_reg = nsmmu_write_reg,
@@ -124,6 +222,8 @@ static const struct arm_smmu_impl nvidia_smmu_impl = {
 	.write_reg64 = nsmmu_write_reg64,
 	.reset = nsmmu_reset,
 	.tlb_sync = nsmmu_tlb_sync,
+	.global_fault = nsmmu_global_fault,
+	.context_fault = nsmmu_context_fault,
 };
 
 struct arm_smmu_device *nvidia_smmu_impl_init(struct arm_smmu_device *smmu)
drivers/iommu/arm-smmu.c +1 -0
index 243bc4cb2705b..d720e1e191176 100644
@@ -673,6 +673,7 @@ static int arm_smmu_init_domain_context(struct iommu_domain *domain,
 	enum io_pgtable_fmt fmt;
 	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
 	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+	irqreturn_t (*context_fault)(int irq, void *dev);
 
 	mutex_lock(&smmu_domain->init_mutex);
 	if (smmu_domain->smmu)

drivers/iommu/arm-smmu.c static int arm_smmu_init_domain_context(struct iommu_domain *domain, · line 840

Quoted 6 lines
drivers/iommu/arm-smmu.c +2 -1
@@ -835,7 +836,9 @@ static int arm_smmu_init_domain_context(struct iommu_domain *domain,
 	 * handler seeing a half-initialised domain state.
 	 */
 	irq = smmu->irqs[smmu->num_global_irqs + cfg->irptndx];
-	ret = devm_request_irq(smmu->dev, irq, arm_smmu_context_fault,
+	context_fault = (smmu->impl && smmu->impl->context_fault) ?
+			 smmu->impl->context_fault : arm_smmu_context_fault;

A simpler way might have been to assign arm_smmu_context_fault to all implementations. That way we wouldn't have to perform this check here and instead just always using smmu->impl->context_fault.

Quoted 13 lines
drivers/iommu/arm-smmu.c +2 -0
+	ret = devm_request_irq(smmu->dev, irq, context_fault,
 			       IRQF_SHARED, "arm-smmu-context-fault", domain);
 	if (ret < 0) {
 		dev_err(smmu->dev, "failed to request context IRQ %d (%u)\n",
@@ -2107,6 +2110,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 	struct arm_smmu_device *smmu;
 	struct device *dev = &pdev->dev;
 	int num_irqs, i, err;
+	irqreturn_t (*global_fault)(int irq, void *dev);
 
 	smmu = devm_kzalloc(dev, sizeof(*smmu), GFP_KERNEL);
 	if (!smmu) {

drivers/iommu/arm-smmu.c static int arm_smmu_device_probe(struct platform_device *pdev) · line 2202

Quoted 6 lines
drivers/iommu/arm-smmu.c +3 -0
@@ -2193,9 +2197,12 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 		smmu->num_context_irqs = smmu->num_context_banks;
 	}
 
+	global_fault = (smmu->impl && smmu->impl->global_fault) ?
+			smmu->impl->global_fault : arm_smmu_global_fault;
+

Same as above.

Thierry

reviewer
#11 lore raw

Re: [PATCH v6 3/4] iommu/arm-smmu: Add global/context fault implementation hooks

On 2020-06-23 09:36, Thierry Reding wrote:
[...]

Quoted 10 lines
Quoted 10 lines
drivers/iommu/arm-smmu.c +1 -0
index 243bc4cb2705b..d720e1e191176 100644
@@ -673,6 +673,7 @@ static int arm_smmu_init_domain_context(struct iommu_domain *domain,
  	enum io_pgtable_fmt fmt;
  	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
  	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+	irqreturn_t (*context_fault)(int irq, void *dev);
 
Quoted 13 lines

drivers/iommu/arm-smmu.c static int arm_smmu_init_domain_context(struct iommu_domain *domain, · line 840

Quoted 9 lines
drivers/iommu/arm-smmu.c +2 -1
  	mutex_lock(&smmu_domain->init_mutex);
  	if (smmu_domain->smmu)
@@ -835,7 +836,9 @@ static int arm_smmu_init_domain_context(struct iommu_domain *domain,
  	 * handler seeing a half-initialised domain state.
  	 */
  	irq = smmu->irqs[smmu->num_global_irqs + cfg->irptndx];
-	ret = devm_request_irq(smmu->dev, irq, arm_smmu_context_fault,
+	context_fault = (smmu->impl && smmu->impl->context_fault) ?
+			 smmu->impl->context_fault : arm_smmu_context_fault;

A simpler way might have been to assign arm_smmu_context_fault to all implementations. That way we wouldn't have to perform this check here and instead just always using smmu->impl->context_fault.

But smmu->impl can still be NULL...

Everything in impl, including the presence of impl itself, is optional, so the notion of overriding a default with the same default doesn't really make much sense, and would go against the pattern everywhere else.

Robin.

Quoted 24 lines
Quoted 13 lines
drivers/iommu/arm-smmu.c +2 -0
+	ret = devm_request_irq(smmu->dev, irq, context_fault,
  			       IRQF_SHARED, "arm-smmu-context-fault", domain);
  	if (ret < 0) {
  		dev_err(smmu->dev, "failed to request context IRQ %d (%u)\n",
@@ -2107,6 +2110,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
  	struct arm_smmu_device *smmu;
  	struct device *dev = &pdev->dev;
  	int num_irqs, i, err;
+	irqreturn_t (*global_fault)(int irq, void *dev);
 
  	smmu = devm_kzalloc(dev, sizeof(*smmu), GFP_KERNEL);
  	if (!smmu) {

drivers/iommu/arm-smmu.c static int arm_smmu_device_probe(struct platform_device *pdev) · line 2202

Quoted 6 lines
drivers/iommu/arm-smmu.c +3 -0
@@ -2193,9 +2197,12 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
  		smmu->num_context_irqs = smmu->num_context_banks;
  	}
 
+	global_fault = (smmu->impl && smmu->impl->global_fault) ?
+			smmu->impl->global_fault : arm_smmu_global_fault;
+

Same as above.

Thierry

#12 lore raw

Re: [PATCH v6 3/4] iommu/arm-smmu: Add global/context fault implementation hooks

Quoted 25 lines · Robin Murphy

On 2020-06-23 09:36, Thierry Reding wrote:
[...]

Quoted 14 lines
Quoted 14 lines
drivers/iommu/arm-smmu.c +1 -0
index 243bc4cb2705b..d720e1e191176 100644
@@ -673,6 +673,7 @@ static int arm_smmu_init_domain_context(struct iommu_domain *domain,
  	enum io_pgtable_fmt fmt;
  	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
  	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+	irqreturn_t (*context_fault)(int irq, void *dev);
  	mutex_lock(&smmu_domain->init_mutex);
  	if (smmu_domain->smmu)
@@ -835,7 +836,9 @@ static int arm_smmu_init_domain_context(struct iommu_domain *domain,
  	 * handler seeing a half-initialised domain state.
  	 */
Quoted 8 lines

drivers/iommu/arm-smmu.c static int arm_smmu_init_domain_context(struct iommu_domain *domain, · line 840

Quoted 4 lines
drivers/iommu/arm-smmu.c +2 -1
  	irq = smmu->irqs[smmu->num_global_irqs + cfg->irptndx];
-	ret = devm_request_irq(smmu->dev, irq, arm_smmu_context_fault,
+	context_fault = (smmu->impl && smmu->impl->context_fault) ?
+			 smmu->impl->context_fault : arm_smmu_context_fault;

A simpler way might have been to assign arm_smmu_context_fault to all implementations. That way we wouldn't have to perform this check here and instead just always using smmu->impl->context_fault.

Quoted 5 lines

But smmu->impl can still be NULL...

Everything in impl, including the presence of impl itself, is optional, so the notion of overriding a default with the same default doesn't really make much sense, and would go against the pattern everywhere else.

True. I had assumed that every implementation would set smmu->impl anyway, in which case there'd be little reason to use these default fallbacks since each implementation could simply directly refer to the exact implementation that it wants.

Perhaps the above could be made a bit more palatable by using a standard if/else rather than the ternary operator? That would also more closely match the pattern elsewhere.

Thierry

#13 lore raw

PATCH v6 4/4 iommu/arm-smmu-nvidia: fix the warning reported by kbuild test robot

In reply to Krishna Reddy
Quoted 2 lines
Quoted 2 lines

drivers/iommu/arm-smmu-nvidia.c:151:33: sparse: sparse: cast removes address space '<asn:2>' of expression

---
 drivers/iommu/arm-smmu-nvidia.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iommu/arm-smmu-nvidia.c +1 -1
index 5999b6a770992..6348d8dc17fc2 100644
@@ -248,7 +248,7 @@ struct arm_smmu_device *nvidia_smmu_impl_init(struct arm_smmu_device *smmu)
 			break;
 		nsmmu->bases[i] = devm_ioremap_resource(dev, res);
 		if (IS_ERR(nsmmu->bases[i]))
-			return (struct arm_smmu_device *)nsmmu->bases[i];
+			return ERR_CAST(nsmmu->bases[i]);
 		nsmmu->num_inst++;
 	}
 
Signature · 1 line
2.26.2
#14 lore raw

Re: [PATCH v6 4/4] iommu/arm-smmu-nvidia: fix the warning reported by kbuild test robot

Quoted 8 lines · Krishna Reddy
Quoted 2 lines
Quoted 2 lines

drivers/iommu/arm-smmu-nvidia.c:151:33: sparse: sparse: cast removes address space '<asn:2>' of expression

---
 drivers/iommu/arm-smmu-nvidia.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

This should be folded into the patch that introduced this error.

Thierry