Skip to contentSkip to conversation

iommu: amd/intel: Remove multifunction assumption around grouping

PATCH In review Alex Williamson 3 participants 4 messages May 30, 2013 – Jun 20, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 iommu: amd/intel: Remove multifunction assumption around grouping

If a device is multifunction and does not have ACS enabled then we assume that the entire package lacks ACS and use function 0 as the base of the group. The PCIe spec however states that components are permitted to implement ACS on some, none, or all of their applicable functions. It's therefore conceivable that function 0 may be fully independent and support ACS while other functions do not. Instead use the lowest function of the slot that does not have ACS enabled as the base of the group. This may be the current device, which is intentional. So long as we use a consistent algorithm, all the non-ACS functions will be grouped together and ACS functions will get separate groups.

---
 drivers/iommu/amd_iommu.c   |   25 +++++++++++++++++++------
 drivers/iommu/intel-iommu.c |   25 +++++++++++++++++++------
 2 files changed, 38 insertions(+), 12 deletions(-)
drivers/iommu/amd_iommu.c +19 -6
index 1d84be1..565c745 100644
@@ -287,14 +287,27 @@ static struct pci_dev *get_isolation_root(struct pci_dev *pdev)
 
 	/*
 	 * If it's a multifunction device that does not support our
-	 * required ACS flags, add to the same group as function 0.
+	 * required ACS flags, add to the same group as lowest numbered
+	 * function that also does not suport the required ACS flags.
 	 */
 	if (dma_pdev->multifunction &&
-	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS))
-		swap_pci_ref(&dma_pdev,
-			     pci_get_slot(dma_pdev->bus,
-					  PCI_DEVFN(PCI_SLOT(dma_pdev->devfn),
-					  0)));
+	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS)) {
+		u8 i, slot = PCI_SLOT(dma_pdev->devfn);
+
+		for (i = 0; i < 8; i++) {
+			struct pci_dev *tmp;
+
+			tmp = pci_get_slot(dma_pdev->bus, PCI_DEVFN(slot, i));
+			if (!tmp)
+				continue;
+
+			if (!pci_acs_enabled(tmp, REQ_ACS_FLAGS)) {
+				swap_pci_ref(&dma_pdev, tmp);
+				break;
+			}
+			pci_dev_put(tmp);
+		}
+	}
 
 	/*
 	 * Devices on the root bus go through the iommu.  If that's not us,
drivers/iommu/intel-iommu.c +19 -6
index b4f0e28..eec0d3e 100644
@@ -4182,14 +4182,27 @@ static int intel_iommu_add_device(struct device *dev)
 
 	/*
 	 * If it's a multifunction device that does not support our
-	 * required ACS flags, add to the same group as function 0.
+	 * required ACS flags, add to the same group as lowest numbered
+	 * function that also does not suport the required ACS flags.
 	 */
 	if (dma_pdev->multifunction &&
-	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS))
-		swap_pci_ref(&dma_pdev,
-			     pci_get_slot(dma_pdev->bus,
-					  PCI_DEVFN(PCI_SLOT(dma_pdev->devfn),
-					  0)));
+	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS)) {
+		u8 i, slot = PCI_SLOT(dma_pdev->devfn);
+
+		for (i = 0; i < 8; i++) {
+			struct pci_dev *tmp;
+
+			tmp = pci_get_slot(dma_pdev->bus, PCI_DEVFN(slot, i));
+			if (!tmp)
+				continue;
+
+			if (!pci_acs_enabled(tmp, REQ_ACS_FLAGS)) {
+				swap_pci_ref(&dma_pdev, tmp);
+				break;
+			}
+			pci_dev_put(tmp);
+		}
+	}
 
 	/*
 	 * Devices on the root bus go through the iommu.  If that's not us,
#1 lore raw
Quoted 80 lines

-----Original Message-----
From: [email protected] [mailto:iommu- [email protected]] On Behalf Of Alex Williamson
Sent: Friday, May 31, 2013 12:09 AM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: [PATCH] iommu: amd/intel: Remove multifunction assumption around grouping

If a device is multifunction and does not have ACS enabled then we assume that the entire package lacks ACS and use function 0 as the base of the group. The PCIe spec however states that components are permitted to implement ACS on some, none, or all of their applicable functions. It's therefore conceivable that function 0 may be fully independent and support ACS while other functions do not. Instead use the lowest function of the slot that does not have ACS enabled as the base of the group. This may be the current device, which is intentional. So long as we use a consistent algorithm, all the non-ACS functions will be grouped together and ACS functions will get separate groups.

---
 drivers/iommu/amd_iommu.c   |   25 +++++++++++++++++++------
 drivers/iommu/intel-iommu.c |   25 +++++++++++++++++++------
 2 files changed, 38 insertions(+), 12 deletions(-)
drivers/iommu/amd_iommu.c index renamed from drivers/iommu/amd_iommu.c +0 -0

1d84be1..565c745 100644

drivers/iommu/amd_iommu.c +19 -6
@@ -287,14 +287,27 @@ static struct pci_dev *get_isolation_root(struct
 pci_dev *pdev)
 
 	/*
 	 * If it's a multifunction device that does not support our
-	 * required ACS flags, add to the same group as function 0.
+	 * required ACS flags, add to the same group as lowest numbered
+	 * function that also does not suport the required ACS flags.
 	 */
 	if (dma_pdev->multifunction &&
-	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS))
-		swap_pci_ref(&dma_pdev,
-			     pci_get_slot(dma_pdev->bus,
-					  PCI_DEVFN(PCI_SLOT(dma_pdev->devfn),
-					  0)));
+	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS)) {
+		u8 i, slot = PCI_SLOT(dma_pdev->devfn);
+
+		for (i = 0; i < 8; i++) {
+			struct pci_dev *tmp;
+
+			tmp = pci_get_slot(dma_pdev->bus, PCI_DEVFN(slot, i));
+			if (!tmp)
+				continue;
+
+			if (!pci_acs_enabled(tmp, REQ_ACS_FLAGS)) {
+				swap_pci_ref(&dma_pdev, tmp);
+				break;
+			}
+			pci_dev_put(tmp);
+		}
+	}
 
 	/*
drivers/iommu/amd_iommu.c +0 -0
 	 * Devices on the root bus go through the iommu.  If that's not us,
drivers/iommu/intel-iommu.c +2 -1
index b4f0e28..eec0d3e 100644
@@ -4182,14 +4182,27 @@ static int intel_iommu_add_device(struct device
 *dev)
 
 	/*
 	 * If it's a multifunction device that does not support our
-	 * required ACS flags, add to the same group as function 0.
+	 * required ACS flags, add to the same group as lowest numbered
+	 * function that also does not suport the required ACS flags.
 	 */

drivers/iommu/intel-iommu.c static int intel_iommu_add_device(struct device · line 4193

Quoted 10 lines
drivers/iommu/intel-iommu.c +4 -5
 	if (dma_pdev->multifunction &&
-	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS))
-		swap_pci_ref(&dma_pdev,
-			     pci_get_slot(dma_pdev->bus,
-					  PCI_DEVFN(PCI_SLOT(dma_pdev->devfn),
-					  0)));
+	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS)) {
+		u8 i, slot = PCI_SLOT(dma_pdev->devfn);
+
+		for (i = 0; i < 8; i++) {

[Sethi Varun-B16395] A macro like PCI_MAX_FUNCTIONS would improve code readability.

drivers/iommu/intel-iommu.c static int intel_iommu_add_device(struct device · line 4206

Quoted 13 lines
drivers/iommu/intel-iommu.c +13 -0
+			struct pci_dev *tmp;
+
+			tmp = pci_get_slot(dma_pdev->bus, PCI_DEVFN(slot, i));
+			if (!tmp)
+				continue;
+
+			if (!pci_acs_enabled(tmp, REQ_ACS_FLAGS)) {
+				swap_pci_ref(&dma_pdev, tmp);
+				break;
+			}
+			pci_dev_put(tmp);
+		}
+	}

It would be nice if this code could be represented as a function in a common file like iommu/pci.c.

-Varun

#2 lore raw
Quoted 82 lines · On Mon, 2013-06-03 at 07:28 +0000, Sethi Varun-B16395 wrote:
Quoted 82 lines

-----Original Message-----
From: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org [mailto:iommu- bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org] On Behalf Of Alex Williamson
Sent: Friday, May 31, 2013 12:09 AM
To: iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: [PATCH] iommu: amd/intel: Remove multifunction assumption around
grouping

If a device is multifunction and does not have ACS enabled then we assume that the entire package lacks ACS and use function 0 as the base of the group. The PCIe spec however states that components are permitted to implement ACS on some, none, or all of their applicable functions. It's therefore conceivable that function 0 may be fully independent and support ACS while other functions do not. Instead use the lowest function of the slot that does not have ACS enabled as the base of the group. This may be the current device, which is intentional. So long as we use a consistent algorithm, all the non-ACS functions will be grouped together and ACS functions will get separate groups.

---
 drivers/iommu/amd_iommu.c   |   25 +++++++++++++++++++------
 drivers/iommu/intel-iommu.c |   25 +++++++++++++++++++------
 2 files changed, 38 insertions(+), 12 deletions(-)
drivers/iommu/amd_iommu.c index renamed from drivers/iommu/amd_iommu.c +0 -0

1d84be1..565c745 100644

drivers/iommu/amd_iommu.c +19 -6
@@ -287,14 +287,27 @@ static struct pci_dev *get_isolation_root(struct
 pci_dev *pdev)
 
 	/*
 	 * If it's a multifunction device that does not support our
-	 * required ACS flags, add to the same group as function 0.
+	 * required ACS flags, add to the same group as lowest numbered
+	 * function that also does not suport the required ACS flags.
 	 */
 	if (dma_pdev->multifunction &&
-	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS))
-		swap_pci_ref(&dma_pdev,
-			     pci_get_slot(dma_pdev->bus,
-					  PCI_DEVFN(PCI_SLOT(dma_pdev->devfn),
-					  0)));
+	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS)) {
+		u8 i, slot = PCI_SLOT(dma_pdev->devfn);
+
+		for (i = 0; i < 8; i++) {
+			struct pci_dev *tmp;
+
+			tmp = pci_get_slot(dma_pdev->bus, PCI_DEVFN(slot, i));
+			if (!tmp)
+				continue;
+
+			if (!pci_acs_enabled(tmp, REQ_ACS_FLAGS)) {
+				swap_pci_ref(&dma_pdev, tmp);
+				break;
+			}
+			pci_dev_put(tmp);
+		}
+	}
 
 	/*
drivers/iommu/amd_iommu.c +0 -0
 	 * Devices on the root bus go through the iommu.  If that's not us,
drivers/iommu/intel-iommu.c +2 -3
index b4f0e28..eec0d3e 100644
@@ -4182,14 +4182,27 @@ static int intel_iommu_add_device(struct device
 *dev)
 
 	/*
 	 * If it's a multifunction device that does not support our
-	 * required ACS flags, add to the same group as function 0.
+	 * required ACS flags, add to the same group as lowest numbered
+	 * function that also does not suport the required ACS flags.
 	 */
 	if (dma_pdev->multifunction &&
-	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS))
-		swap_pci_ref(&dma_pdev,
Quoted 8 lines

drivers/iommu/intel-iommu.c static int intel_iommu_add_device(struct device · line 4193

Quoted 7 lines
drivers/iommu/intel-iommu.c +4 -3
-			     pci_get_slot(dma_pdev->bus,
-					  PCI_DEVFN(PCI_SLOT(dma_pdev->devfn),
-					  0)));
+	    !pci_acs_enabled(dma_pdev, REQ_ACS_FLAGS)) {
+		u8 i, slot = PCI_SLOT(dma_pdev->devfn);
+
+		for (i = 0; i < 8; i++) {

[Sethi Varun-B16395] A macro like PCI_MAX_FUNCTIONS would improve code readability.

Ok, I see a couple defines of PCI_MAX_FUNCTION, I guess we could add one to iommu/pci.h too.

Quoted 7 lines
Quoted 7 lines
drivers/iommu/intel-iommu.c +7 -0
+			struct pci_dev *tmp;
+
+			tmp = pci_get_slot(dma_pdev->bus, PCI_DEVFN(slot, i));
+			if (!tmp)
+				continue;
+
+			if (!pci_acs_enabled(tmp, REQ_ACS_FLAGS)) {
Quoted 8 lines

drivers/iommu/intel-iommu.c static int intel_iommu_add_device(struct device · line 4206

Quoted 6 lines
drivers/iommu/intel-iommu.c +6 -0
+				swap_pci_ref(&dma_pdev, tmp);
+				break;
+			}
+			pci_dev_put(tmp);
+		}
+	}

It would be nice if this code could be represented as a function in a common file like iommu/pci.c.

I agree, but I haven't gotten any feedback on iommu/pci.c yet and didn't want this patch to depend on that series. There are probably more opportunities to consolidate there. Thanks,

Alex

maintainer
#3 lore raw
In reply to Alex Williamson
Quoted 12 lines · Alex Williamson

If a device is multifunction and does not have ACS enabled then we assume that the entire package lacks ACS and use function 0 as the base of the group. The PCIe spec however states that components are permitted to implement ACS on some, none, or all of their applicable functions. It's therefore conceivable that function 0 may be fully independent and support ACS while other functions do not. Instead use the lowest function of the slot that does not have ACS enabled as the base of the group. This may be the current device, which is intentional. So long as we use a consistent algorithm, all the non-ACS functions will be grouped together and ACS functions will get separate groups.

Quoted 4 lines
---
 drivers/iommu/amd_iommu.c   |   25 +++++++++++++++++++------
 drivers/iommu/intel-iommu.c |   25 +++++++++++++++++++------

Didn't really fit into any of my existing branches. Applied it to x86/vt-d because it is probably not worth creating a new x86-specific branch just for this patch.

Thanks,

Joerg