Skip to contentSkip to conversation

iommu: qcom-iommu-v0 IOMMU

RFC In review Rob Clark 3 participants 3 messages Jul 10, 2014 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore

Rob Clark

author
#0 lore raw

RFC 1/1 iommu: qcom-iommu-v0 IOMMU

So, in it's current form, this is superficially a copy of msm_iommu plus DT conversion. But the pre-DT IOMMU driver had fairly different structure.. ie. psuedo root device, with IOMMU devices hanging off that, and context devices hanging off that. The context devices were what the client device would attach (which was also somewhat non- standard.. see msm_iommu_get_ctx()).

I couldn't really think of some sane way to refactor this and add DT while at the same time keeping compatibility with the old pre-DT msm stuff. So I copied to a new driver.

It was pointed out that nothing upstream actually *used* the msm_iommu driver. So if no one objects to dropping pre-DT support, then I could do some patch rejuggling + sed to make this replace the old driver instead.

I didn't bother documenting the DT bindings at this point. From the looks of it, what we need is rather similar to what exynos needs (and was already nak'd.. see https://lkml.org/lkml/2014/4/27/451)

I tried to leave the parts touching hw and/or building pt's as-is as much as possible. But there is some potential for refactoring there. Just getting rid of the ctx device stuff and funky way of attaching was already a pretty large delta.

Anyways, this is enough to get gpu working without hacks to disable memory protection, so that makes me happy. My work-in-progress branch with this plus other bits needed to get drm/msm running on my ifc6410 (apq8064) is here:

http://cgit.freedesktop.org/~robclark/linux/log/?h=msm-wip-iommu

I guess we couldn't actually merge this until someone sorts out the common IOMMU bindings. And some more cleanup is possible. I'll fill in the missing 'TODO cleanup's, etc. If someone else wants to run with this and get it to point where we can merge it, that is totally fine with me. Otherwise, I'll keep chipping away as I have time.. but it is not exactly like my TODO list is getting empty ;-)

---
 drivers/iommu/Kconfig             |   11 +
 drivers/iommu/Makefile            |    1 +
 drivers/iommu/msm_iommu_hw-8xxx.h |    6 +-
 drivers/iommu/qcom_iommu_v0.c     | 1212 +++++++++++++++++++++++++++++++++++++
 drivers/iommu/qcom_iommu_v0.h     |   95 +++
 5 files changed, 1323 insertions(+), 2 deletions(-)
 create mode 100644 drivers/iommu/qcom_iommu_v0.c
 create mode 100644 drivers/iommu/qcom_iommu_v0.h
drivers/iommu/Kconfig +11 -0
index d260605..a2dccce 100644
@@ -27,6 +27,17 @@ config FSL_PAMU
 	  PAMU can authorize memory access, remap the memory address, and remap I/O
 	  transaction types.
 
+config QCOM_IOMMU_V0
+	bool "Qualcomm IOMMU v0 Support"
+	depends on ARCH_QCOM
+	select IOMMU_API
+	help
+	  Support for the IOMMUs found on certain Qualcomm SOCs.
+	  These IOMMUs allow virtualization of the address space used by most
+	  cores within the multimedia subsystem.
+
+	  If unsure, say N here.
+
 # MSM IOMMU support
 config MSM_IOMMU
 	bool "MSM IOMMU Support"
drivers/iommu/Makefile +1 -0
index 8893bad..b09255f 100644
@@ -2,6 +2,7 @@ obj-$(CONFIG_IOMMU_API) += iommu.o
 obj-$(CONFIG_IOMMU_API) += iommu-traces.o
 obj-$(CONFIG_OF_IOMMU)	+= of_iommu.o
 obj-$(CONFIG_MSM_IOMMU) += msm_iommu.o msm_iommu_dev.o
+obj-$(CONFIG_QCOM_IOMMU_V0) += qcom_iommu_v0.o
 obj-$(CONFIG_AMD_IOMMU) += amd_iommu.o amd_iommu_init.o
 obj-$(CONFIG_AMD_IOMMU_V2) += amd_iommu_v2.o
 obj-$(CONFIG_ARM_SMMU) += arm-smmu.o
drivers/iommu/msm_iommu_hw-8xxx.h +4 -2
index fc16010..84ba573 100644
@@ -61,8 +61,9 @@ do { \
 #define FL_TYPE_TABLE		(1 << 0)
 #define FL_TYPE_SECT		(2 << 0)
 #define FL_SUPERSECTION		(1 << 18)
-#define FL_AP_WRITE		(1 << 10)
-#define FL_AP_READ		(1 << 11)
+#define FL_AP0			(1 << 10)
+#define FL_AP1			(1 << 11)
+#define FL_AP2			(1 << 15)
 #define FL_SHARED		(1 << 16)
 #define FL_BUFFERABLE		(1 << 2)
 #define FL_CACHEABLE		(1 << 3)
@@ -77,6 +78,7 @@ do { \
 #define SL_TYPE_SMALL		(2 << 0)
 #define SL_AP0			(1 << 4)
 #define SL_AP1			(2 << 4)
+#define SL_AP2			(1 << 9)
 #define SL_SHARED		(1 << 10)
 #define SL_BUFFERABLE		(1 << 2)
 #define SL_CACHEABLE		(1 << 3)
drivers/iommu/qcom_iommu_v0.c new file +1212 -0

1,212 lines not on this page · raw message

drivers/iommu/qcom_iommu_v0.h new file +95 -0

95 lines not on this page · raw message

Signature · 1 line
1.9.3
#1 lore raw
Quoted 7 lines · Rob Clark

So, in it's current form, this is superficially a copy of msm_iommu plus DT conversion. But the pre-DT IOMMU driver had fairly different structure.. ie. psuedo root device, with IOMMU devices hanging off that, and context devices hanging off that. The context devices were what the client device would attach (which was also somewhat non- standard.. see msm_iommu_get_ctx()).

Quoted 8 lines

I couldn't really think of some sane way to refactor this and add DT while at the same time keeping compatibility with the old pre-DT msm stuff. So I copied to a new driver.

It was pointed out that nothing upstream actually *used* the msm_iommu driver. So if no one objects to dropping pre-DT support, then I could do some patch rejuggling + sed to make this replace the old driver instead.

+1

drivers/iommu/qcom_iommu_v0.c line 61

Quoted 3 lines
drivers/iommu/qcom_iommu_v0.c new file +3 -0
+
+// TODO any good reason for global lock vs per-iommu lock?
+DEFINE_SPINLOCK(qcom_iommu_lock);

static?

Quoted 8 lines
drivers/iommu/qcom_iommu_v0.c new file +8 -0
+static LIST_HEAD(qcom_iommu_devices);
+
+/* Note that a single iommu_domain can, for devices sitting behind
+ * more than one IOMMU (ie. one per AXI interface) will have more
+ * than one iommu in the iommu_list.  But all are programmed to
+ * point at the same pagetables so from client device perspective
+ * they act as a single IOMMU.
+ */

drivers/iommu/qcom_iommu_v0.c line 79

Quoted 10 lines
drivers/iommu/qcom_iommu_v0.c new file +10 -0
+struct qcom_domain_priv {
+	unsigned long *pgtable;
+	struct list_head iommu_list;  /* list of attached 'struct qcom_iommu' */
+};
+
+static int __enable_clocks(struct qcom_iommu *iommu)
+{
+	int ret;
+
+	ret = clk_prepare_enable(iommu->pclk);

Looks like __enable_clocks() is called from within a spinlock sometimes. Please move the prepare call outside of any atomic sections and also enable CONFIG_PROVE_LOCKING and/or CONFIG_DEBUG_ATOMIC_SLEEP to find such problems.

Quoted 12 lines
drivers/iommu/qcom_iommu_v0.c new file +12 -0
+	if (ret)
+		goto fail;
+
+	if (iommu->clk) {
+		ret = clk_prepare_enable(iommu->clk);
+		if (ret)
+			clk_disable_unprepare(iommu->pclk);
+	}
+fail:
+	return ret;
+}
+
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 17 lines · Stephen Boyd
Quoted 7 lines · Rob Clark

So, in it's current form, this is superficially a copy of msm_iommu plus DT conversion. But the pre-DT IOMMU driver had fairly different structure.. ie. psuedo root device, with IOMMU devices hanging off that, and context devices hanging off that. The context devices were what the client device would attach (which was also somewhat non- standard.. see msm_iommu_get_ctx()).

Quoted 8 lines

I couldn't really think of some sane way to refactor this and add DT while at the same time keeping compatibility with the old pre-DT msm stuff. So I copied to a new driver.

It was pointed out that nothing upstream actually *used* the msm_iommu driver. So if no one objects to dropping pre-DT support, then I could do some patch rejuggling + sed to make this replace the old driver instead.

Quoted 7 lines

+1

drivers/iommu/qcom_iommu_v0.c line 61

Quoted 3 lines
drivers/iommu/qcom_iommu_v0.c new file +3 -0
+
+// TODO any good reason for global lock vs per-iommu lock?
+DEFINE_SPINLOCK(qcom_iommu_lock);

static?

oh, yes.. ofc

I do wonder if we might want to make locking a bit more fine grained to reduce contention (but, otoh, the gpu driver isn't going to contend with itself, and other drivers probably aren't taxing the iommu quite so hard). But I guess it would be ok to leave that as a future optimization.

Quoted 16 lines
Quoted 16 lines
drivers/iommu/qcom_iommu_v0.c new file +15 -0
+static LIST_HEAD(qcom_iommu_devices);
+
+/* Note that a single iommu_domain can, for devices sitting behind
+ * more than one IOMMU (ie. one per AXI interface) will have more
+ * than one iommu in the iommu_list.  But all are programmed to
+ * point at the same pagetables so from client device perspective
+ * they act as a single IOMMU.
+ */
+struct qcom_domain_priv {
+     unsigned long *pgtable;
+     struct list_head iommu_list;  /* list of attached 'struct qcom_iommu' */
+};
+
+static int __enable_clocks(struct qcom_iommu *iommu)
+{
Quoted 8 lines

drivers/iommu/qcom_iommu_v0.c line 79

Quoted 3 lines
drivers/iommu/qcom_iommu_v0.c new file +3 -0
+     int ret;
+
+     ret = clk_prepare_enable(iommu->pclk);

Looks like __enable_clocks() is called from within a spinlock sometimes. Please move the prepare call outside of any atomic sections and also enable CONFIG_PROVE_LOCKING and/or CONFIG_DEBUG_ATOMIC_SLEEP to find such problems.

good point. I haven't gotten around to running this against a debug kernel. It is only at the 'just barely works now' stage, I just wanted to send for early feedback on the approach. (Mainly to make sure no one screams bloody murder about dropping pre-DT support from old driver.)

BR,
-R

Quoted 16 lines

drivers/iommu/qcom_iommu_v0.c line 91

Quoted 12 lines
drivers/iommu/qcom_iommu_v0.c new file +12 -0
+     if (ret)
+             goto fail;
+
+     if (iommu->clk) {
+             ret = clk_prepare_enable(iommu->clk);
+             if (ret)
+                     clk_disable_unprepare(iommu->pclk);
+     }
+fail:
+     return ret;
+}
+

--
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation