iommu: Expose IOMMU information in sysfs

Users want to know the features of their hardware and we need a better way to get it than parsing it out of dmesg. This series adds a simple registration interface for IOMMUs and an example base implementation for intel-iommu.

One key hardware feature for device assignment is the IOMMU support for superpages. In this example, we can parse it out of the "cap" attribute exposed, but I expect we'll want to add more human readable entries for such features. I'd welcome suggestions on what features we should pull out into human friendly attributes and how to format the contents.

I have not attempted to make a common, consistent interface for attributes between various IOMMU types here. I'm not entirely sure such a thing is possible. Perhaps instead we do like I show in the intel-iommu example and provide IOMMU driver specific attribute groups, clearly labeled so that we effectively give each a namespace. We can promote consistency between drivers, but a common namespace is probably best left to userspace tools.

Appreciate any thoughts and comments. Thanks,

Alex

---

Alex Williamson (2):
iommu: Add sysfs support for IOMMUs iommu/intel: Make use of IOMMU sysfs support

 drivers/iommu/Makefile      |    1 
 drivers/iommu/dmar.c        |    8 ++
 drivers/iommu/intel-iommu.c |   75 ++++++++++++++++++++++
 drivers/iommu/iommu-sysfs.c |  147 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/intel-iommu.h |    2 +
 include/linux/iommu.h       |   28 ++++++++
 6 files changed, 260 insertions(+), 1 deletion(-)
 create mode 100644 drivers/iommu/iommu-sysfs.c

IOMMUs currently have no common representation to userspace, most seem to have no representation at all aside from a few printks on bootup. There are, however, features of IOMMUs that are useful to know about. For instance, the IOMMU might support superpages, making use of processor large/huge pages more important in a device assignment scenario. It's also useful to create cross links between devices and IOMMU hardware units, so that users might be able to load balance their devices to avoid thrashing a single hardware unit.

This patch adds a registration and de-registration interface as well as device linking, making it very lightweight for an IOMMU driver to add basic support. IOMMU drivers can provide additional attributes automatically by using an attribute_group.

The attributes exposed are expected to be relatively device specific, the means to retrieve them certainly are. So there are currently no common attributes for the new iommu_class created here.

---
 drivers/iommu/Makefile      |    1 
 drivers/iommu/iommu-sysfs.c |  147 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/iommu.h       |   28 ++++++++
 3 files changed, 176 insertions(+)
 create mode 100644 drivers/iommu/iommu-sysfs.c

index 5d58bf1..437c231 100644
@@ -1,5 +1,6 @@
 obj-$(CONFIG_IOMMU_API) += iommu.o
 obj-$(CONFIG_IOMMU_API) += iommu-traces.o
+obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
 obj-$(CONFIG_OF_IOMMU)	+= of_iommu.o
 obj-$(CONFIG_MSM_IOMMU) += msm_iommu.o msm_iommu_dev.o
 obj-$(CONFIG_AMD_IOMMU) += amd_iommu.o amd_iommu_init.o

index b96a5b2..e9bd3a0 100644
@@ -186,6 +186,12 @@ extern int iommu_domain_get_attr(struct iommu_domain *domain, enum iommu_attr,
 				 void *data);
 extern int iommu_domain_set_attr(struct iommu_domain *domain, enum iommu_attr,
 				 void *data);
+extern int iommu_register_device(struct device *parent, const char *name,
+				 const struct attribute_group **groups,
+				 void *platform_data);
+extern void iommu_unregister_device(void *platform_data);
+extern int iommu_device_link(const void *platform_data, struct device *link);
+extern void iommu_device_unlink(const void *platform_data, struct device *link);
 
 /* Window handling function prototypes */
 extern int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr,

@@ -396,6 +402,28 @@ static inline int iommu_domain_set_attr(struct iommu_domain *domain,
 	return -EINVAL;
 }
 
+static inline int iommu_register_device(struct device *parent, const char *name,
+					const struct attribute_group **groups,
+					void *platform_data)
+{
+	return 0;
+}
+
+static inline void iommu_unregister_device(void *platform_data)
+{
+}
+
+static inline int iommu_device_link(const void *platform_data,
+				    struct device *link)
+{
+	return 0;
+}
+
+static inline void iommu_device_unlink(const void *platform_data,
+				       struct device *link)
+{
+}
+
 #endif /* CONFIG_IOMMU_API */
 
 #endif /* __LINUX_IOMMU_H */

Register our DRHD IOMMUs, cross link devices, and provide a base set of attributes for the IOMMU. Note that IRQ remapping support parses the DMAR table very early in boot, well before the iommu_class can reasonably be setup, so our registration is split between intel_iommu_init(), which occurs later, and alloc_iommu(), which typically occurs much earlier, but may happen at any time later with IOMMU hot-add support.

On a typical desktop system, this provides the following (pruned):

$ find /sys | grep dmar
/sys/devices/virtual/iommu/dmar0
/sys/devices/virtual/iommu/dmar0/devices
/sys/devices/virtual/iommu/dmar0/devices/0000:00:02.0
/sys/devices/virtual/iommu/dmar0/intel-iommu
/sys/devices/virtual/iommu/dmar0/intel-iommu/cap
/sys/devices/virtual/iommu/dmar0/intel-iommu/ecap
/sys/devices/virtual/iommu/dmar0/intel-iommu/address
/sys/devices/virtual/iommu/dmar0/intel-iommu/version
/sys/devices/virtual/iommu/dmar1
/sys/devices/virtual/iommu/dmar1/devices
/sys/devices/virtual/iommu/dmar1/devices/0000:00:00.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:01.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:16.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1a.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1b.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1c.0
...
/sys/devices/virtual/iommu/dmar1/intel-iommu
/sys/devices/virtual/iommu/dmar1/intel-iommu/cap
/sys/devices/virtual/iommu/dmar1/intel-iommu/ecap
/sys/devices/virtual/iommu/dmar1/intel-iommu/address
/sys/devices/virtual/iommu/dmar1/intel-iommu/version
/sys/class/iommu/dmar0
/sys/class/iommu/dmar1

(devices also link back to the dmar units)

Putting these under the "virtual" directory may not be correct since these are physical devices, maybe under /sys/devices/system? Virtual is simply the default for NULL parent.

This makes address, version, capabilities, and extended capabilities available, just like printed on boot. Nearly everything we might want can be parsed from cap/ecap, but I expect we'll want to make some human friendly entries for things that we care about.

---
 drivers/iommu/dmar.c        |    8 +++++
 drivers/iommu/intel-iommu.c |   75 ++++++++++++++++++++++++++++++++++++++++++-
 include/linux/intel-iommu.h |    2 +
 3 files changed, 84 insertions(+), 1 deletion(-)

index 39f8b71..68b3d07 100644
@@ -38,6 +38,7 @@
 #include <linux/tboot.h>
 #include <linux/dmi.h>
 #include <linux/slab.h>
+#include <linux/iommu.h>
 #include <asm/irq_remapping.h>
 #include <asm/iommu_table.h>
 

@@ -980,6 +981,11 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)
 	raw_spin_lock_init(&iommu->register_lock);
 
 	drhd->iommu = iommu;
+
+	if (intel_iommu_enabled)
+		iommu_register_device(NULL, iommu->name,
+				      intel_iommu_groups, iommu);
+
 	return 0;
 
  err_unmap:

@@ -991,6 +997,8 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)
 
 static void free_iommu(struct intel_iommu *iommu)
 {
+	iommu_unregister_device(iommu);
+
 	if (iommu->irq) {
 		free_irq(iommu->irq, iommu);
 		irq_set_handler_data(iommu->irq, NULL);

index 0a2da51..aa6905e 100644
@@ -365,4 +365,6 @@ extern int qi_submit_sync(struct qi_desc *desc, struct intel_iommu *iommu);
 
 extern int dmar_ir_support(void);
 
+extern const struct attribute_group *intel_iommu_groups[];
+
 #endif

No proof of concept is complete with only a single user, so here's an example for AMD IOMMU. This IOMMU is neatly associated with a device, so we don't need to leave the IOMMU device dangling in virtual space. This just has a couple fields from struct amd_iommu cherry picked to expose and looks something like this (pruned):

# find /sys | grep ivhd
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:00.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:02.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:04.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:09.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:11.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:12.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:12.2
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:13.0
...
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/power
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/power/control
...
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/device
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/subsystem
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/amd-iommu
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/amd-iommu/cap
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/amd-iommu/is_iommu_v2
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/amd-iommu/features
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/uevent
/sys/class/iommu/ivhd0

---
 drivers/iommu/amd_iommu.c      |    4 +++
 drivers/iommu/amd_iommu_init.c |   53 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+)

index 4621692..4849580 100644
@@ -346,6 +346,8 @@ static int iommu_init_device(struct device *dev)
 
 	dev->archdata.iommu = dev_data;
 
+	iommu_device_link(amd_iommu_rlookup_table[dev_data->devid], dev);
+
 	return 0;
 }
 

@@ -370,6 +372,8 @@ static void iommu_uninit_device(struct device *dev)
 	if (!dev_data)
 		return;
 
+	iommu_device_unlink(amd_iommu_rlookup_table[dev_data->devid], dev);
+
 	iommu_group_remove_device(dev);
 
 	/* Unlink from alias, it may change if another device is re-plugged */