Skip to contentSkip to conversation

Support for PRI and PASID capability

PATCH Superseded Joerg Roedel 4 participants 12 messages Sep 22, 2011 – Sep 29, 2011 PCI

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Support for PRI and PASID capability

Hi Jesse,

here is a patch-set that enables support for the PRI and PASID capabilities in the Linux PCI core. These capabilities of a PCIe device are tightly coupled with ATS, but not with IOV. So these patches start by moving the ATS code into its own file: ats.c.

Support for the other two capabilities is based on that change. Please review the patches and let me know of any objections you might have.

Thanks,

Joerg

Diffstat:

 drivers/pci/Kconfig      |   24 +++
 drivers/pci/Makefile     |    1 +
 drivers/pci/ats.c        |  438 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/iov.c        |  142 ---------------
 include/linux/pci-ats.h  |   75 ++++++++
 include/linux/pci_regs.h |   20 ++
 6 files changed, 558 insertions(+), 142 deletions(-)
#1 lore raw

PATCH 1/4 PCI: Move ATS implementation into own file

ATS does not depend on IOV support, so move the code into its own file. This file will also include support for the PRI and PASID capability later.
Also give ATS its own Kconfig variable to allow selecting it without IOV support.

---
 drivers/pci/Kconfig     |    4 +
 drivers/pci/Makefile    |    1 +
 drivers/pci/ats.c       |  155 +++++++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/iov.c       |  142 -------------------------------------------
 include/linux/pci-ats.h |    2 +
 5 files changed, 162 insertions(+), 142 deletions(-)
 create mode 100644 drivers/pci/ats.c
drivers/pci/Kconfig +4 -0
index 0fa466a..1d8ce83 100644
@@ -71,9 +71,13 @@ config HT_IRQ
 
 	   If unsure say Y.
 
+config PCI_ATS
+	bool
+
 config PCI_IOV
 	bool "PCI IOV support"
 	depends on PCI
+	select PCI_ATS
 	help
 	  I/O Virtualization is a PCI feature supported by some devices
 	  which allows them to create virtual devices which share their
drivers/pci/Makefile +1 -0
index 6fadae3..083a49f 100644
@@ -29,6 +29,7 @@ obj-$(CONFIG_PCI_MSI) += msi.o
 # Build the Hypertransport interrupt support
 obj-$(CONFIG_HT_IRQ) += htirq.o
 
+obj-$(CONFIG_PCI_ATS) += ats.o
 obj-$(CONFIG_PCI_IOV) += iov.o
 
 #
drivers/pci/ats.c new file +155 -0

155 lines not on this page · raw message

drivers/pci/iov.c +0 -142

145 lines not on this page · raw message

include/linux/pci-ats.h +2 -0
index 655824f..4eab42b 100644
@@ -1,6 +1,8 @@
 #ifndef LINUX_PCI_ATS_H
 #define LINUX_PCI_ATS_H
 
+#include <linux/pci.h>
+
 /* Address Translation Service */
 struct pci_ats {
 	int pos;        /* capability position */
Signature · 1 line
1.7.4.1
#2 lore raw

PATCH 2/4 PCI: Export ATS functions to modules

In reply to Joerg Roedel

This patch makes the ATS function usable for modules. This will be used by a modules implementing some advanced AMD IOMMU features.

---
 drivers/pci/ats.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
drivers/pci/ats.c +3 -0
index ae4bf87..5ceff3e 100644
@@ -90,6 +90,7 @@ int pci_enable_ats(struct pci_dev *dev, int ps)
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(pci_enable_ats);
 
 /**
  * pci_disable_ats - disable the ATS capability
@@ -120,6 +121,7 @@ void pci_disable_ats(struct pci_dev *dev)
 	if (!dev->is_physfn)
 		ats_free_one(dev);
 }
+EXPORT_SYMBOL_GPL(pci_disable_ats);
 
 /**
  * pci_ats_queue_depth - query the ATS Invalidate Queue Depth
@@ -153,3 +155,4 @@ int pci_ats_queue_depth(struct pci_dev *dev)
 	return PCI_ATS_CAP_QDEP(cap) ? PCI_ATS_CAP_QDEP(cap) :
 				       PCI_ATS_MAX_QDEP;
 }
+EXPORT_SYMBOL_GPL(pci_ats_queue_depth);
Signature · 1 line
1.7.4.1
#3 lore raw

PATCH 3/4 PCI: Add implementation for PRI capability

In reply to Joerg Roedel

Implement the necessary functions to handle PRI capabilities on PCIe devices. With PRI devices behind an IOMMU can signal page fault conditions to software and recover from such faults.

---
 drivers/pci/Kconfig      |    9 +++
 drivers/pci/ats.c        |  167 ++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci-ats.h  |   42 ++++++++++++
 include/linux/pci_regs.h |   12 +++
 4 files changed, 230 insertions(+), 0 deletions(-)
drivers/pci/Kconfig +9 -0
index 1d8ce83..fb1e9707 100644
@@ -85,6 +85,15 @@ config PCI_IOV
 
 	  If unsure, say N.
 
+config PCI_PRI
+	bool "PCI PRI support"
+	select PCI_ATS
+	help
+	  PRI is the PCI Page Request Interface. It allows PCI devices that are
+	  behind an IOMMU to recover from page faults.
+
+	  If unsure, say N.
+
 config PCI_IOAPIC
 	bool
 	depends on PCI
drivers/pci/ats.c +167 -0

179 lines not on this page · raw message

include/linux/pci-ats.h +42 -0

52 lines not on this page · raw message

include/linux/pci_regs.h +12 -0
index e884096..7fc32af 100644
@@ -663,6 +663,18 @@
 #define  PCI_ATS_CTRL_STU(x)	((x) & 0x1f)	/* Smallest Translation Unit */
 #define  PCI_ATS_MIN_STU	12	/* shift of minimum STU block */
 
+/* Page Request Interface */
+#define PCI_PRI_CAP		0x13    /* PRI capability ID */
+#define PCI_PRI_CONTROL_OFF	0x04	/* Offset of control register */
+#define PCI_PRI_STATUS_OFF	0x06	/* Offset of status register */
+#define PCI_PRI_ENABLE		0x0001	/* Enable mask */
+#define PCI_PRI_RESET		0x0002	/* Reset bit mask */
+#define PCI_PRI_STATUS_RF	0x0001  /* Request Failure */
+#define PCI_PRI_STATUS_UPRGI	0x0002  /* Unexpected PRG index */
+#define PCI_PRI_STATUS_STOPPED	0x0100  /* PRI Stopped */
+#define PCI_PRI_MAX_REQ_OFF	0x08	/* Cap offset for max reqs supported */
+#define PCI_PRI_ALLOC_REQ_OFF	0x0c	/* Cap offset for max reqs allowed */
+
 /* Single Root I/O Virtualization */
 #define PCI_SRIOV_CAP		0x04	/* SR-IOV Capabilities */
 #define  PCI_SRIOV_CAP_VFM	0x01	/* VF Migration Capable */
Signature · 1 line
1.7.4.1
#4 lore raw

PATCH 4/4 PCI: Add support for PASID capability

In reply to Joerg Roedel

Devices supporting the PASID capability can use multiple contexts on an IOMMU at the same time. This patch adds support to manage that capability.

---
 drivers/pci/Kconfig      |   11 +++++
 drivers/pci/ats.c        |  113 ++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci-ats.h  |   31 +++++++++++++
 include/linux/pci_regs.h |    8 +++
 4 files changed, 163 insertions(+), 0 deletions(-)
drivers/pci/Kconfig +11 -0
index fb1e9707..ed7c42e 100644
@@ -94,6 +94,17 @@ config PCI_PRI
 
 	  If unsure, say N.
 
+config PCI_PASID
+	bool "PCI PASID support"
+	depends on PCI
+	select PCI_ATS
+	help
+	  Devices supporting PASIDs can access multiple address spaces
+	  on an IOMMU at the same time. This option compiles code for
+	  that capability into the kernel.
+
+	  If unsure, say N.
+
 config PCI_IOAPIC
 	bool
 	depends on PCI
drivers/pci/ats.c +113 -0

122 lines not on this page · raw message

include/linux/pci-ats.h +31 -0
index 0713952..e3d0b38 100644
@@ -93,4 +93,35 @@ static inline int pci_pri_status(struct pci_dev *pdev)
 }
 #endif /* CONFIG_PCI_PRI */
 
+#ifdef CONFIG_PCI_PASID
+
+extern int pci_enable_pasid(struct pci_dev *pdev, int features);
+extern void pci_disable_pasid(struct pci_dev *pdev);
+extern int pci_pasid_features(struct pci_dev *pdev);
+extern int pci_max_pasids(struct pci_dev *pdev);
+
+#else  /* CONFIG_PCI_PASID */
+
+static inline int pci_enable_pasid(struct pci_dev *pdev, int features)
+{
+	return -EINVAL;
+}
+
+static inline void pci_disable_pasid(struct pci_dev *pdev)
+{
+}
+
+static inline int pci_pasid_features(struct pci_dev *pdev)
+{
+	return -EINVAL;
+}
+
+static inline int pci_max_pasids(struct pci_dev *pdev)
+{
+	return -EINVAL;
+}
+
+#endif /* CONFIG_PCI_PASID */
+
+
 #endif /* LINUX_PCI_ATS_H*/
include/linux/pci_regs.h +8 -0
index 7fc32af..b5d9657 100644
@@ -675,6 +675,14 @@
 #define PCI_PRI_MAX_REQ_OFF	0x08	/* Cap offset for max reqs supported */
 #define PCI_PRI_ALLOC_REQ_OFF	0x0c	/* Cap offset for max reqs allowed */
 
+/* PASID capability */
+#define PCI_PASID_CAP		0x1b    /* PASID capability ID */
+#define PCI_PASID_CAP_OFF	0x04    /* PASID feature register */
+#define PCI_PASID_CONTROL_OFF   0x06    /* PASID control register */
+#define PCI_PASID_ENABLE	0x01	/* Enable/Supported bit */
+#define PCI_PASID_EXEC		0x02	/* Exec permissions Enable/Supported */
+#define PCI_PASID_PRIV		0x04	/* Priviledge Mode Enable/Support */
+
 /* Single Root I/O Virtualization */
 #define PCI_SRIOV_CAP		0x04	/* SR-IOV Capabilities */
 #define  PCI_SRIOV_CAP_VFM	0x01	/* VF Migration Capable */
Signature · 1 line
1.7.4.1
maintainer
#5 lore raw

Re: [PATCH 4/4] PCI: Add support for PASID capability

Quoted 3 lines · Joerg Roedel

Devices supporting the PASID capability can use multiple contexts on an IOMMU at the same time. This patch adds support to manage that capability.

These all look reasonable to me.

Is there anything else useful you could say in the PASID Kconfig help text? I'm guessing it stands for "PCI Address Space ID" or something, but I have no idea why I might want to enable it.

I didn't coordinate very well with Jesse. I'm hoping he'll be back before the 3.2 merge window, but I think we have a few patches queued up now, and it might be useful to have them in linux-next for a while before he returns. I'll look into that.

Bjorn

Quoted 227 lines

---
 drivers/pci/Kconfig      |   11 +++++
 drivers/pci/ats.c        |  113 ++++++++++++++++++++++++++++++++++++++++++++++ include/linux/pci-ats.h  |   31 +++++++++++++ include/linux/pci_regs.h |    8 +++

 4 files changed, 163 insertions(+), 0 deletions(-)
drivers/pci/Kconfig +0 -0
index fb1e9707..ed7c42e 100644
@@ -94,6 +94,17 @@ config PCI_PRI
 

         If unsure, say N.

drivers/pci/Kconfig +11 -0
+config PCI_PASID
+       bool "PCI PASID support"
+       depends on PCI
+       select PCI_ATS
+       help
+         Devices supporting PASIDs can access multiple address spaces
+         on an IOMMU at the same time. This option compiles code for
+         that capability into the kernel.
+
+         If unsure, say N.
+

 config PCI_IOAPIC
       bool
       depends on PCI

drivers/pci/ats.c +113 -0
index bf892a0..f727a09 100644
@@ -7,6 +7,7 @@
  * PCI Express I/O Virtualization (IOV) support.
  *   Address Translation Service 1.0
  *   Page Request Interface added by Joerg Roedel <[email protected]>
+ *   PASID support added by Joerg Roedel <[email protected]>
  */
 
  #include <linux/pci-ats.h>
@@ -323,3 +324,115 @@ int pci_pri_status(struct pci_dev *pdev)
  }
  EXPORT_SYMBOL_GPL(pci_pri_status);
  #endif /* CONFIG_PCI_PRI */
+
+#ifdef CONFIG_PCI_PASID
+/**
+ * pci_enable_pasid - Enable the PASID capability
+ * @pdev: PCI device structure
+ * @features: Features to enable
+ *
+ * Returns 0 on success, negative value on error. This function checks
+ * whether the features are actually supported by the device and returns
+ * an error if not.
+ */
+int pci_enable_pasid(struct pci_dev *pdev, int features)
+{
+       u16 control, supported;
+       int pos;
+
+       pos = pci_find_ext_capability(pdev, PCI_PASID_CAP);
+       if (!pos)
+               return -EINVAL;
+
+       pci_read_config_word(pdev, pos + PCI_PASID_CONTROL_OFF, &control);
+       pci_read_config_word(pdev, pos + PCI_PASID_CAP_OFF,     &supported);
+
+       if (!(supported & PCI_PASID_ENABLE))
+               return -EINVAL;
+
+       supported &= PCI_PASID_EXEC | PCI_PASID_PRIV;
+
+       /* User wants to enable anything unsupported? */
+       if ((supported & features) != features)
+               return -EINVAL;
+
+       control = PCI_PASID_ENABLE | features;
+
+       pci_write_config_word(pdev, pos + PCI_PASID_CONTROL_OFF, control);
+
+       return 0;
+}
+EXPORT_SYMBOL_GPL(pci_enable_pasid);
+
+/**
+ * pci_disable_pasid - Disable the PASID capability
+ * @pdev: PCI device structure
+ *
+ */
+void pci_disable_pasid(struct pci_dev *pdev)
+{
+       u16 control = 0;
+       int pos;
+
+       pos = pci_find_ext_capability(pdev, PCI_PASID_CAP);
+       if (!pos)
+               return;
+
+       pci_write_config_word(pdev, pos + PCI_PASID_CONTROL_OFF, control);
+}
+EXPORT_SYMBOL_GPL(pci_disable_pasid);
+
+/**
+ * pci_pasid_features - Check which PASID features are supported
+ * @pdev: PCI device structure
+ *
+ * Returns a negative value when no PASI capability is present.
+ * Otherwise is returns a bitmask with supported features. Current
+ * features reported are:
+ * PCI_PASID_ENABLE - PASID capability can be enabled
+ * PCI_PASID_EXEC - Execute permission supported
+ * PCI_PASID_PRIV - Priviledged mode supported
+ */
+int pci_pasid_features(struct pci_dev *pdev)
+{
+       u16 supported;
+       int pos;
+
+       pos = pci_find_ext_capability(pdev, PCI_PASID_CAP);
+       if (!pos)
+               return -EINVAL;
+
+       pci_read_config_word(pdev, pos + PCI_PASID_CAP_OFF, &supported);
+
+       supported &= PCI_PASID_ENABLE | PCI_PASID_EXEC | PCI_PASID_PRIV;
+
+       return supported;
+}
+EXPORT_SYMBOL_GPL(pci_pasid_features);
+
+#define PASID_NUMBER_SHIFT     8
+#define PASID_NUMBER_MASK      (0x1f << PASID_NUMBER_SHIFT)
+/**
+ * pci_max_pasid - Get maximum number of PASIDs supported by device
+ * @pdev: PCI device structure
+ *
+ * Returns negative value when PASID capability is not present.
+ * Otherwise it returns the numer of supported PASIDs.
+ */
+int pci_max_pasids(struct pci_dev *pdev)
+{
+       u16 supported;
+       int pos;
+
+       pos = pci_find_ext_capability(pdev, PCI_PASID_CAP);
+       if (!pos)
+               return -EINVAL;
+
+       pci_read_config_word(pdev, pos + PCI_PASID_CAP_OFF, &supported);
+
+       supported = (supported & PASID_NUMBER_MASK) >> PASID_NUMBER_SHIFT;
+
+       return (1 << supported);
+}
+EXPORT_SYMBOL_GPL(pci_max_pasids);
+#endif /* CONFIG_PCI_PASID */
include/linux/pci-ats.h +31 -0
index 0713952..e3d0b38 100644
@@ -93,4 +93,35 @@ static inline int pci_pri_status(struct pci_dev *pdev)
  }
  #endif /* CONFIG_PCI_PRI */
 
+#ifdef CONFIG_PCI_PASID
+
+extern int pci_enable_pasid(struct pci_dev *pdev, int features);
+extern void pci_disable_pasid(struct pci_dev *pdev);
+extern int pci_pasid_features(struct pci_dev *pdev);
+extern int pci_max_pasids(struct pci_dev *pdev);
+
+#else  /* CONFIG_PCI_PASID */
+
+static inline int pci_enable_pasid(struct pci_dev *pdev, int features)
+{
+       return -EINVAL;
+}
+
+static inline void pci_disable_pasid(struct pci_dev *pdev)
+{
+}
+
+static inline int pci_pasid_features(struct pci_dev *pdev)
+{
+       return -EINVAL;
+}
+
+static inline int pci_max_pasids(struct pci_dev *pdev)
+{
+       return -EINVAL;
+}
+
+#endif /* CONFIG_PCI_PASID */
+
+
  #endif /* LINUX_PCI_ATS_H*/
include/linux/pci_regs.h +8 -1
index 7fc32af..b5d9657 100644
@@ -675,6 +675,14 @@
  #define PCI_PRI_MAX_REQ_OFF    0x08    /* Cap offset for max reqs supported */
  #define PCI_PRI_ALLOC_REQ_OFF  0x0c    /* Cap offset for max reqs allowed */
 
+/* PASID capability */
+#define PCI_PASID_CAP          0x1b    /* PASID capability ID */
+#define PCI_PASID_CAP_OFF      0x04    /* PASID feature register */
+#define PCI_PASID_CONTROL_OFF   0x06    /* PASID control register */
+#define PCI_PASID_ENABLE       0x01    /* Enable/Supported bit */
+#define PCI_PASID_EXEC         0x02    /* Exec permissions Enable/Supported */
+#define PCI_PASID_PRIV         0x04    /* Priviledge Mode Enable/Support */
+
  /* Single Root I/O Virtualization */
  #define PCI_SRIOV_CAP          0x04    /* SR-IOV Capabilities */
  #define  PCI_SRIOV_CAP_VFM     0x01    /* VF Migration Capable */
--

1.7.4.1

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

#6 lore raw

Re: [PATCH 4/4] PCI: Add support for PASID capability

Quoted 10 lines · Bjorn Helgaas
Quoted 3 lines · Joerg Roedel

Devices supporting the PASID capability can use multiple contexts on an IOMMU at the same time. This patch adds support to manage that capability.

These all look reasonable to me.

Is there anything else useful you could say in the PASID Kconfig help text? I'm guessing it stands for "PCI Address Space ID" or something, but I have no idea why I might want to enable it.

PASID stands for Process Address Space ID, it basically allows a PCI device to access multiple IO address spaces at the same time using an IOMMU. Users should enable it when they want to compile an IOMMU driver into the kernel that supports PASIDs for the translated devices. I'll add all that information to the Kconfig help. Thanks a lot for your feedback :)

Quoted 4 lines

I didn't coordinate very well with Jesse. I'm hoping he'll be back before the 3.2 merge window, but I think we have a few patches queued up now, and it might be useful to have them in linux-next for a while before he returns. I'll look into that.

Great, thanks again.

Regards,

Joerg

#7 lore raw

Re: [PATCH 4/4] PCI: Add support for PASID capability

Quoted 6 lines · Joerg Roedel

PASID stands for Process Address Space ID, it basically allows a PCI device to access multiple IO address spaces at the same time using an IOMMU. Users should enable it when they want to compile an IOMMU driver into the kernel that supports PASIDs for the translated devices. I'll add all that information to the Kconfig help. Thanks a lot for your feedback :)

Is this a case where the PCI device drivers in question should be doing a 'select' or 'depends on' if they support talking to an IOMMU using PASID?

#8 lore raw

Re: [PATCH 4/4] PCI: Add support for PASID capability

Quoted 11 lines · On Wed, Sep 28, 2011 at 03:18:15PM -0400, [email protected] wrote:
Quoted 6 lines · Joerg Roedel

PASID stands for Process Address Space ID, it basically allows a PCI device to access multiple IO address spaces at the same time using an IOMMU. Users should enable it when they want to compile an IOMMU driver into the kernel that supports PASIDs for the translated devices. I'll add all that information to the Kconfig help. Thanks a lot for your feedback :)

Is this a case where the PCI device drivers in question should be doing a 'select' or 'depends on' if they support talking to an IOMMU using PASID?

Exactly. In fact, I plan to put the functionality which handles these capabilities and the fault-handling in a seperate module where all device drivers can use it in an easy way.

Joerg

Signature · 5 lines
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
#9 lore raw
In reply to Joerg Roedel

Hi Jesse,

Quoted 7 lines · Joerg Roedel

here is a patch-set that enables support for the PRI and PASID capabilities in the Linux PCI core. These capabilities of a PCIe device are tightly coupled with ATS, but not with IOV. So these patches start by moving the ATS code into its own file: ats.c.

Support for the other two capabilities is based on that change. Please review the patches and let me know of any objections you might have.

Have you had a chance to look at these patches? Please let me know any questions or objections you might have.

Thanks,

Joerg

maintainer
#10 lore raw
Quoted 9 lines · Joerg Roedel

Hi Jesse,

Quoted 5 lines · Joerg Roedel

here is a patch-set that enables support for the PRI and PASID capabilities in the Linux PCI core. These capabilities of a PCIe device are tightly coupled with ATS, but not with IOV. So these patches start by moving the ATS code into its own file: ats.c.

Quoted 5 lines
Quoted 2 lines

Support for the other two capabilities is based on that change. Please review the patches and let me know of any objections you might have.

Have you had a chance to look at these patches? Please let me know any questions or objections you might have.

Jesse's on vacation, but I'll take a look and give you my opinion (which won't necessarily coincide with Jesse's, but I guess it's a start :)).

Bjorn

#11 lore raw
Quoted 13 lines · Bjorn Helgaas
Quoted 12 lines · Joerg Roedel

Hi Jesse,

Quoted 7 lines · Joerg Roedel

here is a patch-set that enables support for the PRI and PASID capabilities in the Linux PCI core. These capabilities of a PCIe device are tightly coupled with ATS, but not with IOV. So these patches start by moving the ATS code into its own file: ats.c.

Support for the other two capabilities is based on that change. Please review the patches and let me know of any objections you might have.

Quoted 6 lines
Quoted 2 lines

Have you had a chance to look at these patches? Please let me know any questions or objections you might have.

Jesse's on vacation, but I'll take a look and give you my opinion (which won't necessarily coincide with Jesse's, but I guess it's a start :)).

Great. Thanks a lot :)

Joerg