iommu: Expose IOMMU information in sysfs
cover iommu: Expose IOMMU information in sysfs
Users want to know the features of their hardware and we need a better way to get it than parsing it out of dmesg. This series creates an IOMMU class and allows drivers to create and destroy a device within that class. Drivers may also link and unlink devices to expose the association of a device to a specific hardware unit and create attribute groups to expose further information about the specific IOMMU.
An example use case for this would be to determine whether the IOMMU for a device supports super-pages, which can have performance implications for a device assignment scenario. A user with multiple IOMMU units may also want to factor load per IOMMU into their hardware layout. This helps to expose those connections.
I posted an RFC for this last month, but there were no comments. This version officialy includes AMD-Vi support and removes the kobject used for creating the "devices" subdirectory, instead using an empty attribute group. I've tried not to duplicate any information available elsewhere for the user, for example in the DMAR or IVRS tables. This means that while I think it might be interesting to expose things like RMRR, the user already has access to those by parsing the appropriate firmware table. I welcome any additional IOMMU driver implementaitons. Thanks,
Alex
---
Alex Williamson (3):
iommu: Add sysfs support for IOMMUs iommu/intel: Make use of IOMMU sysfs support iommu/amd: Add sysfs support
Documentation/ABI/testing/sysfs-class-iommu | 17 +++ .../ABI/testing/sysfs-class-iommu-amd-iommu | 14 ++ .../ABI/testing/sysfs-class-iommu-intel-iommu | 32 +++++ drivers/iommu/Makefile | 1 drivers/iommu/amd_iommu.c | 6 + drivers/iommu/amd_iommu_init.c | 38 ++++++ drivers/iommu/amd_iommu_types.h | 3 drivers/iommu/dmar.c | 9 + drivers/iommu/intel-iommu.c | 77 +++++++++++- drivers/iommu/iommu-sysfs.c | 133 ++++++++++++++++++++ include/linux/intel-iommu.h | 3 include/linux/iommu.h | 26 ++++ 12 files changed, 358 insertions(+), 1 deletion(-) create mode 100644 Documentation/ABI/testing/sysfs-class-iommu create mode 100644 Documentation/ABI/testing/sysfs-class-iommu-amd-iommu create mode 100644 Documentation/ABI/testing/sysfs-class-iommu-intel-iommu create mode 100644 drivers/iommu/iommu-sysfs.c
PATCH 1/3 iommu: Add sysfs support for IOMMUs
IOMMUs currently have no common representation to userspace, most seem to have no representation at all aside from a few printks on bootup. There are however features of IOMMUs that are useful to know about. For instance the IOMMU might support superpages, making use of processor large/huge pages more important in a device assignment scenario. It's also useful to create cross links between devices and IOMMU hardware units, so that users might be able to load balance their devices to avoid thrashing a single hardware unit.
This patch adds a device create and destroy interface as well as device linking, making it very lightweight for an IOMMU driver to add basic support. IOMMU drivers can provide additional attributes automatically by using an attribute_group.
The attributes exposed are expected to be relatively device specific, the means to retrieve them certainly are, so there are currently no common attributes for the new class created here.
--- Documentation/ABI/testing/sysfs-class-iommu | 17 +++ drivers/iommu/Makefile | 1 drivers/iommu/iommu-sysfs.c | 133 +++++++++++++++++++++++++++ include/linux/iommu.h | 26 +++++ 4 files changed, 177 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-class-iommu create mode 100644 drivers/iommu/iommu-sysfs.c
Documentation/ABI/testing/sysfs-class-iommu new file +17 -0
@@ -0,0 +1,17 @@ +What: /sys/class/iommu/<iommu>/devices/ +Date: June 2014 +KernelVersion: 3.17 +Contact: Alex Williamson <[email protected]> +Description: + IOMMU drivers are able to link devices managed by a + given IOMMU here to allow association of IOMMU to + device. + +What: /sys/devices/.../iommu +Date: June 2014 +KernelVersion: 3.17 +Contact: Alex Williamson <[email protected]> +Description: + IOMMU drivers are able to link the IOMMU for a + given device here to allow association of device to + IOMMU.
drivers/iommu/Makefile +1 -0
@@ -1,5 +1,6 @@ obj-$(CONFIG_IOMMU_API) += iommu.o obj-$(CONFIG_IOMMU_API) += iommu-traces.o +obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o obj-$(CONFIG_OF_IOMMU) += of_iommu.o obj-$(CONFIG_MSM_IOMMU) += msm_iommu.o msm_iommu_dev.o obj-$(CONFIG_AMD_IOMMU) += amd_iommu.o amd_iommu_init.o
drivers/iommu/iommu-sysfs.c new file +133 -0
133 lines not on this page · raw message
include/linux/iommu.h +26 -0
@@ -187,6 +187,12 @@ extern int iommu_domain_get_attr(struct iommu_domain *domain, enum iommu_attr, void *data);extern int iommu_domain_set_attr (struct iommu_domain *domain,enum iommu_attr ,void *data); +struct device *iommu_device_create (struct device *parent,void *drvdata, +const struct attribute_group **groups, +const char *fmt, ...); +void iommu_device_destroy (struct device *dev); +int iommu_device_link (struct device *dev,struct device *link); +void iommu_device_unlink (struct device *dev,struct device *link);/* Window handling function prototypes */ extern int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr,
@@ -397,6 +403,26 @@ static inline int iommu_domain_set_attr(struct iommu_domain *domain,return -EINVAL ; } +struct device *iommu_device_create (struct device *parent,void *drvdata, +const struct attribute_group **groups, +const char *fmt, ...) +{ +return ERR_PTR (-ENODEV ); +} + +void iommu_device_destroy (struct device *dev) +{ +} + +int iommu_device_link (struct device *dev,struct device *link) +{ +return -EINVAL ; +} + +void iommu_device_unlink (struct device *dev,struct device *link) +{ +} +#endif /* CONFIG_IOMMU_API */ #endif /* __LINUX_IOMMU_H */
PATCH 2/3 iommu/intel: Make use of IOMMU sysfs support
In reply to Alex WilliamsonRegister our DRHD IOMMUs, cross link devices, and provide a base set of attributes for the IOMMU. Note that IRQ remapping support parses the DMAR table very early in boot, well before the iommu_class can reasonably be setup, so our registration is split between intel_iommu_init(), which occurs later, and alloc_iommu(), which typically occurs much earlier, but may happen at any time later with IOMMU hot-add support.
On a typical desktop system, this provides the following (pruned):
$ find /sys | grep dmar /sys/devices/virtual/iommu/dmar0 /sys/devices/virtual/iommu/dmar0/devices /sys/devices/virtual/iommu/dmar0/devices/0000:00:02.0 /sys/devices/virtual/iommu/dmar0/intel-iommu /sys/devices/virtual/iommu/dmar0/intel-iommu/cap /sys/devices/virtual/iommu/dmar0/intel-iommu/ecap /sys/devices/virtual/iommu/dmar0/intel-iommu/address /sys/devices/virtual/iommu/dmar0/intel-iommu/version /sys/devices/virtual/iommu/dmar1 /sys/devices/virtual/iommu/dmar1/devices /sys/devices/virtual/iommu/dmar1/devices/0000:00:00.0 /sys/devices/virtual/iommu/dmar1/devices/0000:00:01.0 /sys/devices/virtual/iommu/dmar1/devices/0000:00:16.0 /sys/devices/virtual/iommu/dmar1/devices/0000:00:1a.0 /sys/devices/virtual/iommu/dmar1/devices/0000:00:1b.0 /sys/devices/virtual/iommu/dmar1/devices/0000:00:1c.0 ... /sys/devices/virtual/iommu/dmar1/intel-iommu /sys/devices/virtual/iommu/dmar1/intel-iommu/cap /sys/devices/virtual/iommu/dmar1/intel-iommu/ecap /sys/devices/virtual/iommu/dmar1/intel-iommu/address /sys/devices/virtual/iommu/dmar1/intel-iommu/version /sys/class/iommu/dmar0 /sys/class/iommu/dmar1
(devices also link back to the dmar units)
This makes address, version, capabilities, and extended capabilities available, just like printed on boot. I've tried not to duplicate data that can be found in the DMAR table, with the exception of the address, which provides an easy way to associate the sysfs device with a DRHD entry in the DMAR. It's tempting to add scopes and RMRR data here, but the full DMAR table is already exposed under /sys/firmware/ and therefore already provides a way for userspace to learn such details.
--- .../ABI/testing/sysfs-class-iommu-intel-iommu | 32 ++++++++ drivers/iommu/dmar.c | 9 ++ drivers/iommu/intel-iommu.c | 77 ++++++++++++++++++++ include/linux/intel-iommu.h | 3 + 4 files changed, 120 insertions(+), 1 deletion(-) create mode 100644 Documentation/ABI/testing/sysfs-class-iommu-intel-iommu
Documentation/ABI/testing/sysfs-class-iommu-intel-iommu new file +32 -0
@@ -0,0 +1,32 @@ +What: /sys/class/iommu/<iommu>/intel-iommu/address +Date: June 2014 +KernelVersion: 3.17 +Contact: Alex Williamson <[email protected]> +Description: + Physical address of the VT-d DRHD for this IOMMU. + Format: %llx. This allows association of a sysfs + intel-iommu with a DMAR DRHD table entry. + +What: /sys/class/iommu/<iommu>/intel-iommu/cap +Date: June 2014 +KernelVersion: 3.17 +Contact: Alex Williamson <[email protected]> +Description: + The cached hardware capability register value + of this DRHD unit. Format: %llx. + +What: /sys/class/iommu/<iommu>/intel-iommu/ecap +Date: June 2014 +KernelVersion: 3.17 +Contact: Alex Williamson <[email protected]> +Description: + The cached hardware extended capability register + value of this DRHD unit. Format: %llx. + +What: /sys/class/iommu/<iommu>/intel-iommu/version +Date: June 2014 +KernelVersion: 3.17 +Contact: Alex Williamson <[email protected]> +Description: + The architecture version as reported from the + VT-d VER_REG. Format: %d:%d, major:minor
drivers/iommu/dmar.c +9 -0
@@ -38,6 +38,7 @@#include <linux/tboot.h> #include <linux/dmi.h> #include <linux/slab.h> +#include <linux/iommu.h> #include <asm/irq_remapping.h> #include <asm/iommu_table.h>
@@ -980,6 +981,12 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)raw_spin_lock_init (&iommu->register_lock); drhd->iommu = iommu; + +if (intel_iommu_enabled) + iommu->iommu_dev =iommu_device_create (NULL , iommu, + intel_iommu_groups, + iommu->name); +return 0 ; err_unmap:
@@ -991,6 +998,8 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)static void free_iommu (struct intel_iommu *iommu) { +iommu_device_destroy (iommu->iommu_dev); +if (iommu->irq) {free_irq (iommu->irq, iommu);irq_set_handler_data (iommu->irq,NULL );
drivers/iommu/intel-iommu.c +76 -1
107 lines not on this page · raw message
include/linux/intel-iommu.h +3 -0
@@ -336,6 +336,7 @@ struct intel_iommu {#ifdef CONFIG_IRQ_REMAP struct ir_table *ir_table;/* Interrupt remapping info */ #endif +struct device *iommu_dev;/* IOMMU-sysfs device */ int node; };
@@ -365,4 +366,6 @@ extern int qi_submit_sync(struct qi_desc *desc, struct intel_iommu *iommu);extern int dmar_ir_support (void ); +extern const struct attribute_group *intel_iommu_groups[]; +#endif
PATCH 3/3 iommu/amd: Add sysfs support
In reply to Alex WilliamsonAMD-Vi support for IOMMU sysfs. This allows us to associate devices with a specific IOMMU device and examine the capabilities and features of that IOMMU. The AMD IOMMU is hosted on and actual PCI device, so we make that device the parent for the IOMMU class device. This initial implementaiton exposes only the capability header and extended features register for the IOMMU.
# find /sys | grep ivhd /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0 /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:00.0 /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:02.0 /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:04.0 /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:09.0 /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:11.0 /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:12.0 /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:12.2 /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:13.0 ... /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/power /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/power/control ... /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/device /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/subsystem /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/amd-iommu /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/amd-iommu/cap /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/amd-iommu/features /sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/uevent /sys/class/iommu/ivhd0
--- .../ABI/testing/sysfs-class-iommu-amd-iommu | 14 +++++++ drivers/iommu/amd_iommu.c | 6 +++ drivers/iommu/amd_iommu_init.c | 38 ++++++++++++++++++++ drivers/iommu/amd_iommu_types.h | 3 ++ 4 files changed, 61 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-class-iommu-amd-iommu
Documentation/ABI/testing/sysfs-class-iommu-amd-iommu new file +14 -0
@@ -0,0 +1,14 @@ +What: /sys/class/iommu/<iommu>/amd-iommu/cap +Date: June 2014 +KernelVersion: 3.17 +Contact: Alex Williamson <[email protected]> +Description: + IOMMU capability header as documented in the AMD IOMMU + specification. Format: %x + +What: /sys/class/iommu/<iommu>/amd-iommu/features +Date: June 2014 +KernelVersion: 3.17 +Contact: Alex Williamson <[email protected]> +Description: + Extended features of the IOMMU. Format: %llx
drivers/iommu/amd_iommu.c +6 -0
@@ -379,6 +379,9 @@ static int iommu_init_device(struct device *dev) dev->archdata.iommu = dev_data; +iommu_device_link (amd_iommu_rlookup_table[dev_data->devid]->iommu_dev, + dev); +return 0 ; }
@@ -403,6 +406,9 @@ static void iommu_uninit_device(struct device *dev)if (!dev_data)return ; +iommu_device_unlink (amd_iommu_rlookup_table[dev_data->devid]->iommu_dev, + dev); +iommu_group_remove_device (dev);/* Unlink from alias, it may change if another device is re-plugged */
drivers/iommu/amd_iommu_init.c +38 -0
@@ -26,6 +26,7 @@#include <linux/msi.h> #include <linux/amd-iommu.h> #include <linux/export.h> +#include <linux/iommu.h> #include <asm/pci-direct.h> #include <asm/iommu.h> #include <asm/gart.h>
@@ -1197,6 +1198,39 @@ static void init_iommu_perf_ctr(struct amd_iommu *iommu) iommu->max_counters = (u8) ((val >>7 ) &0xf ); } +static ssize_t amd_iommu_show_cap (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct amd_iommu *iommu =dev_get_drvdata (dev); +return sprintf (buf,"%x\n" , iommu->cap); +} +static DEVICE_ATTR(cap, S_IRUGO, amd_iommu_show_cap, NULL); + +static ssize_t amd_iommu_show_features (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct amd_iommu *iommu =dev_get_drvdata (dev); +return sprintf (buf,"%llx\n" , iommu->features); +} +static DEVICE_ATTR(features, S_IRUGO, amd_iommu_show_features, NULL); + +static struct attribute *amd_iommu_attrs[] = { + &dev_attr_cap.attr, + &dev_attr_features.attr, +NULL , +}; + +static struct attribute_group amd_iommu_group = { + .name ="amd-iommu" , + .attrs = amd_iommu_attrs, +}; + +static const struct attribute_group *amd_iommu_groups[] = { + &amd_iommu_group, +NULL , +};static int iommu_init_pci (struct amd_iommu *iommu) {
@@ -1297,6 +1331,10 @@ static int iommu_init_pci(struct amd_iommu *iommu)amd_iommu_erratum_746_workaround (iommu); + iommu->iommu_dev =iommu_device_create (&iommu->dev->dev, iommu, + amd_iommu_groups,"ivhd%d" , + iommu->index); +return pci_enable_device (iommu->dev); }
drivers/iommu/amd_iommu_types.h +3 -0
@@ -577,6 +577,9 @@ struct amd_iommu {/* default dma_ops domain for that IOMMU */ struct dma_ops_domain *default_dom; +/* IOMMU sysfs device */ +struct device *iommu_dev; +/* * We can't rely on the BIOS to restore all values on reinit, so we * need to stash them
Joerg Roedel
maintainerQuoted 4 lines · Alex Williamson
Alex Williamson (3):
iommu: Add sysfs support for IOMMUs iommu/intel: Make use of IOMMU sysfs support iommu/amd: Add sysfs support
I like the general approach. But why do you only enable the x86 iommus? Some ARM IOMMUs and PAMU would certainly also benefit from a sysfs representation of which device is translated by which IOMMU.
Joerg
Quoted 9 lines · Joerg Roedel
Quoted 4 lines · Alex Williamson
Alex Williamson (3):
iommu: Add sysfs support for IOMMUs iommu/intel: Make use of IOMMU sysfs support iommu/amd: Add sysfs supportI like the general approach. But why do you only enable the x86 iommus? Some ARM IOMMUs and PAMU would certainly also benefit from a sysfs representation of which device is translated by which IOMMU.
Only because I have no ability to test the others platforms. I'd be thrilled if someone working on those IOMMUs could validate this on non-x86. Thanks,
Alex
Quoted 7 lines · Alex Williamson
Quoted 6 lines · Joerg Roedel
Quoted 4 lines · Alex Williamson
Alex Williamson (3):
iommu: Add sysfs support for IOMMUs iommu/intel: Make use of IOMMU sysfs support iommu/amd: Add sysfs support
Quoted 7 lines
Quoted 3 lines
I like the general approach. But why do you only enable the x86 iommus? Some ARM IOMMUs and PAMU would certainly also benefit from a sysfs representation of which device is translated by which IOMMU.
Only because I have no ability to test the others platforms. I'd be thrilled if someone working on those IOMMUs could validate this on non-x86. Thanks,
Should I re-send this series or are you hoping I'll attempt to piece together a non-x86 user for which I have no hardware and can't test? Varun, do you have any interest in this for the fsl iommu to try a proof of concept? Thanks,
Alex
Hi Alex,
This is on my todo list. I have a requirement for adding sysfs support for PAMU.
Regards
Varun
Quoted 31 lines
-----Original Message-----
From: Alex Williamson [mailto:[email protected]]
Sent: Thursday, July 03, 2014 9:39 PM
To: Joerg Roedel
Cc: iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; [email protected]; linux- [email protected]; Sethi Varun-B16395
Subject: Re: [PATCH 0/3] iommu: Expose IOMMU information in sysfsQuoted 8 lines · Alex Williamson
Quoted 7 lines · Joerg Roedel
Quoted 4 lines · Alex Williamson
Alex Williamson (3):
iommu: Add sysfs support for IOMMUs iommu/intel: Make use of IOMMU sysfs support iommu/amd: Add sysfs supportI like the general approach. But why do you only enable the x86
iommus?
Quoted 6 lines
Quoted 2 lines
Some ARM IOMMUs and PAMU would certainly also benefit from a sysfs representation of which device is translated by which IOMMU.
Only because I have no ability to test the others platforms. I'd be thrilled if someone working on those IOMMUs could validate this on non-x86. Thanks,
Should I re-send this series or are you hoping I'll attempt to piece together a non-x86 user for which I have no hardware and can't test? Varun, do you have any interest in this for the fsl iommu to try a proof of concept? Thanks,
Alex
Joerg Roedel
maintainerQuoted 15 lines · Alex Williamson
Alex Williamson (3):
iommu: Add sysfs support for IOMMUs iommu/intel: Make use of IOMMU sysfs support iommu/amd: Add sysfs supportDocumentation/ABI/testing/sysfs-class-iommu | 17 +++ .../ABI/testing/sysfs-class-iommu-amd-iommu | 14 ++ .../ABI/testing/sysfs-class-iommu-intel-iommu | 32 +++++ drivers/iommu/Makefile | 1 drivers/iommu/amd_iommu.c | 6 + drivers/iommu/amd_iommu_init.c | 38 ++++++ drivers/iommu/amd_iommu_types.h | 3 drivers/iommu/dmar.c | 9 + drivers/iommu/intel-iommu.c | 77 +++++++++++-
Quoted 8 lines
drivers/iommu/iommu-sysfs.c | 133 ++++++++++++++++++++ include/linux/intel-iommu.h | 3 include/linux/iommu.h | 26 ++++ 12 files changed, 358 insertions(+), 1 deletion(-) create mode 100644 Documentation/ABI/testing/sysfs-class-iommu create mode 100644 Documentation/ABI/testing/sysfs-class-iommu-amd-iommu create mode 100644 Documentation/ABI/testing/sysfs-class-iommu-intel-iommu create mode 100644 drivers/iommu/iommu-sysfs.c
Also applied to the core branch, thanks Alex.