Support for Intel Virtualization Technology for Directed I/O

This patch contains basic ACPI parsing and enumeration support.

drivers/pci/search.c +20 -0

+/*
+ * find the upstream PCIE-to-PCI bridge of a PCI device
+ * if the device is PCIE, return NULL
+ * if the device isn't connected to a PCIE bridge (that is its parent is a
+ * legacy PCI bridge and the bridge is directly connected to bus 0), return its
+ * parent
+ */
+struct pci_dev *
+pci_find_upstream_pcie_bridge(struct pci_dev *pdev)
+{
+	struct pci_dev *tmp = NULL;
+
+	if (pdev->is_pcie)
+		return NULL;
+	while (1) {
+		if (!pdev->bus->self)
+			break;
+		pdev = pdev->bus->self;
+		/* a p2p bridge */
+		if (!pdev->is_pcie) {

include/linux/pci.h +1 -0

@@ -126,6 +126,7 @@
 	unsigned short	subsystem_device;
 	unsigned int	class;		/* 3 bytes: (base,sub,prog-if) */
 	u8		hdr_type;	/* PCI header type (`multi' flag masked out) */
+	u8		pcie_type;	/* PCI-E device/port type */
 	u8		rom_base_reg;	/* which config register controls the ROM */
 	u8		pin;  		/* which interrupt pin this device uses */
 

Please help review and provide feedback.

Quoted 1 line · On Monday 09 April 2007 23:55:52 Ashok Raj wrote:

Please help review and provide feedback.

High level question: how did you solve the "user X server needs IOMMU bypass" problem?

Quoted 4 lines · Andi Kleen

Quoted 1 line · On Monday 09 April 2007 23:55:52 Ashok Raj wrote:

Please help review and provide feedback.

Quoted 2 lines

High level question: how did you solve the "user X server needs IOMMU bypass" problem?

Please see patch 5. We have two workarounds for graphics. One is completely bypass dma remapping for gfx, this option only works for gfx card with dedicated dma remapping engine. The other is doing 1-1 mapping for whole physical memory address, then virtual dma address will equal to physical address, so gfx can ignore iommu in the case.

Quoted 24 lines · Ashok Raj

drivers/pci/search.c +24 -0

+/*
+ * find the upstream PCIE-to-PCI bridge of a PCI device
+ * if the device is PCIE, return NULL
+ * if the device isn't connected to a PCIE bridge (that is its parent is a
+ * legacy PCI bridge and the bridge is directly connected to bus 0), return its
+ * parent
+ */
+struct pci_dev *
+pci_find_upstream_pcie_bridge(struct pci_dev *pdev)
+{
+	struct pci_dev *tmp = NULL;
+
+	if (pdev->is_pcie)
+		return NULL;
+	while (1) {
+		if (!pdev->bus->self)
+			break;
+		pdev = pdev->bus->self;
+		/* a p2p bridge */
+		if (!pdev->is_pcie) {
+			tmp = pdev;
+			continue;
+		}
+		/* PCI device should connect to a PCIE bridge */

drivers/pci/search.c line 46

Quoted 6 lines

drivers/pci/search.c +6 -0

+		BUG_ON(pdev->pcie_type != PCI_EXP_TYPE_PCI_BRIDGE);
+		return pdev;
+	}
+
+	return tmp;
+}

No locking while you walk up the bus list?

Quoted 10 lines

include/linux/pci.h +1 -0

@@ -126,6 +126,7 @@
 	unsigned short	subsystem_device;
 	unsigned int	class;		/* 3 bytes: (base,sub,prog-if) */
 	u8		hdr_type;	/* PCI header type (`multi' flag masked out) */
+	u8		pcie_type;	/* PCI-E device/port type */
 	u8		rom_base_reg;	/* which config register controls the ROM */
 	u8		pin;  		/* which interrupt pin this device uses */
 

include/linux/pci.h line 172

Quoted 5 lines

include/linux/pci.h +1 -0

@@ -168,6 +169,7 @@
 	unsigned int 	msi_enabled:1;
 	unsigned int	msix_enabled:1;
 	unsigned int	is_managed:1;
+	unsigned int	is_pcie:1;

Do you really need both fields? Wouldn't just the pcie_type one work (with some NOT_PCIE type being set for it if it isn't I suppose.)

1. Support for ACPI framework to parse and work with DMA Remapping Tables.
2. Add support for PCI infrastructure to search parent relationships.
3. Hardware support for providing DMA remapping support for Intel Chipsets.
4. Supporting Zero Length Reads on DMAR's not able to support ZLR.
5. Graphics driver workarounds to provide unity map since they dont use dma api.
6. Updates to Documentation area for startup options and some basics.
7. Workaround to provide unity map for ISA bridge device to enable floppy disk.
8. Ability to preserve some mappings for devices not able to address entire range.

Quoted 5 lines · Shaohua Li

Quoted 4 lines · Andi Kleen

Quoted 1 line · On Monday 09 April 2007 23:55:52 Ashok Raj wrote:

Please help review and provide feedback.

Quoted 8 lines

Quoted 2 lines

High level question: how did you solve the "user X server needs IOMMU bypass" problem?

Please see patch 5. We have two workarounds for graphics. One is completely bypass dma remapping for gfx, this option only works for gfx card with dedicated dma remapping engine. The other is doing 1-1 mapping for whole physical memory address, then virtual dma address will equal to physical address, so gfx can ignore iommu in the case.

Apologies but I've not had a chance to review the patches yet. Do you special case the BDF of the graphics adpater (via PCI ID?), or does the user need to provide a command line option to specify the graphics adapter?

Quoted 9 lines · Ashok Raj

1. Support for ACPI framework to parse and work with DMA Remapping Tables.
2. Add support for PCI infrastructure to search parent relationships.
3. Hardware support for providing DMA remapping support for Intel Chipsets.
4. Supporting Zero Length Reads on DMAR's not able to support ZLR.
5. Graphics driver workarounds to provide unity map since they dont use dma api.
6. Updates to Documentation area for startup options and some basics.
7. Workaround to provide unity map for ISA bridge device to enable floppy disk.
8. Ability to preserve some mappings for devices not able to address entire range.

I really think it would be best to avoid ACPI for something as critical as DMA remapping!

Most GFX drivers don't call standard PCI DMA APIs to allocate DMA buffer, Such drivers will be broken with IOMMU enabled. To workaround this issue, we added two options.

DMA remapping just uses ACPI table to tell which dma remapping engine a pci device is controlled by at boot time. At run time, DMA remapping hasn't any interactive with ACPI.

Quoted 3 lines

Most GFX drivers don't call standard PCI DMA APIs to allocate DMA buffer, Such drivers will be broken with IOMMU enabled. To workaround this issue, we added two options.

All drm drivers do it. If the usual out of tree crap vendors are too stupid for their own sake it's their fault.

So NACK to this patch.

From: Christoph Hellwig <[email protected]>
Date: Tue, 10 Apr 2007 09:33:51 +0100

Quoted 5 lines · Ashok Raj

Quoted 3 lines

Most GFX drivers don't call standard PCI DMA APIs to allocate DMA buffer, Such drivers will be broken with IOMMU enabled. To workaround this issue, we added two options.

Quoted 4 lines

All drm drivers do it. If the usual out of tree crap vendors are too stupid for their own sake it's their fault.

So NACK to this patch.

That's my feeling as well, everything we care about should be using the proper APIs or else what is the point of them...

Quoted 25 lines · Greg KH

Quoted 24 lines · Ashok Raj

drivers/pci/search.c +24 -0

+/*
+ * find the upstream PCIE-to-PCI bridge of a PCI device
+ * if the device is PCIE, return NULL
+ * if the device isn't connected to a PCIE bridge (that is its parent is a
+ * legacy PCI bridge and the bridge is directly connected to bus 0), return its
+ * parent
+ */
+struct pci_dev *
+pci_find_upstream_pcie_bridge(struct pci_dev *pdev)
+{
+	struct pci_dev *tmp = NULL;
+
+	if (pdev->is_pcie)
+		return NULL;
+	while (1) {
+		if (!pdev->bus->self)
+			break;
+		pdev = pdev->bus->self;
+		/* a p2p bridge */
+		if (!pdev->is_pcie) {
+			tmp = pdev;
+			continue;
+		}
+		/* PCI device should connect to a PCIE bridge */

Quoted 8 lines

drivers/pci/search.c line 46

Quoted 6 lines

drivers/pci/search.c +6 -0

+		BUG_ON(pdev->pcie_type != PCI_EXP_TYPE_PCI_BRIDGE);
+		return pdev;
+	}
+
+	return tmp;
+}

No locking while you walk up the bus list?

hmm, iommu driver didn't support pci hotplug in current stage. But we can add lock here.

Quoted 10 lines

Quoted 10 lines

include/linux/pci.h +1 -0

@@ -126,6 +126,7 @@
 	unsigned short	subsystem_device;
 	unsigned int	class;		/* 3 bytes: (base,sub,prog-if) */
 	u8		hdr_type;	/* PCI header type (`multi' flag masked out) */
+	u8		pcie_type;	/* PCI-E device/port type */
 	u8		rom_base_reg;	/* which config register controls the ROM */
 	u8		pin;  		/* which interrupt pin this device uses */
 

Quoted 8 lines

include/linux/pci.h line 172

Quoted 5 lines

include/linux/pci.h +1 -0

@@ -168,6 +169,7 @@
 	unsigned int 	msi_enabled:1;
 	unsigned int	msix_enabled:1;
 	unsigned int	is_managed:1;
+	unsigned int	is_pcie:1;

Do you really need both fields? Wouldn't just the pcie_type one work (with some NOT_PCIE type being set for it if it isn't I suppose.)

There are some encodings are reserved for future. Just don't want to use reserved bits, as we don't know which one is proper for NOT_PCIE.

Quoted 1 line · Ashok Raj

This patch contains basic ACPI parsing and enumeration support.

AFAICS, ACPI supplies the envelope which delivers the table,
and ACPI has some convenience structure definitions for that
table in include/acpi/actbl1.h (primarily for the acpixtract table dis-assembler), but ACPI is otherwise not involved in IOMMU support.

Indeed, one might argue that all new functions in this patch series with "acpi..." would more appropriately be called "pci...", since a cursory scan of the IOMMU spec seems to suggest it is specific to PCI.

So on first blush, it looks like the only call to a function that begins with "acpi" in this patch series should be acpi_get_table() from some IOMMU specific file outside of drivers/acpi,
and the only modification to any code with an "acpi" in the file path or filename should be any updates to the convenience structure definitions in acpitbl1.h

thanks,
-Len

From: Christoph Hellwig <[email protected]>
Date: Tue, 10 Apr 2007 09:33:51 +0100

Quoted 3 lines · Ashok Raj

Quoted 2 lines

Most GFX drivers don't call standard PCI DMA APIs to allocate DMA buffer,
Such drivers will be broken with IOMMU enabled. To workaround this issue, we added two options.

Quoted 4 lines

All drm drivers do it. If the usual out of tree crap vendors are too stupid for their own sake it's their fault.

So NACK to this patch.

That's my feeling as well, everything we care about should be using the proper APIs or else what is the point of them...

Quoted 1 line · On Monday 09 April 2007 23:55:52 Ashok Raj wrote:

Please help review and provide feedback.

High level question: how did you solve the "user X server needs IOMMU bypass" problem?

-Andi

Quoted 3 lines · Shaohua Li

DMA remapping just uses ACPI table to tell which dma remapping engine a pci device is controlled by at boot time. At run time, DMA remapping hasn't any interactive with ACPI.

The Linux kernel _really_ wants a non-ACPI way to detect this.

Just use the hardware registers themselves, you don't need an ACPI table.

Jeff

Quoted 34 lines · Shaohua Li

Quoted 25 lines · Greg KH

Quoted 24 lines · Ashok Raj

drivers/pci/search.c +24 -0

+/*
+ * find the upstream PCIE-to-PCI bridge of a PCI device
+ * if the device is PCIE, return NULL
+ * if the device isn't connected to a PCIE bridge (that is its parent is a
+ * legacy PCI bridge and the bridge is directly connected to bus 0), return its
+ * parent
+ */
+struct pci_dev *
+pci_find_upstream_pcie_bridge(struct pci_dev *pdev)
+{
+	struct pci_dev *tmp = NULL;
+
+	if (pdev->is_pcie)
+		return NULL;
+	while (1) {
+		if (!pdev->bus->self)
+			break;
+		pdev = pdev->bus->self;
+		/* a p2p bridge */
+		if (!pdev->is_pcie) {
+			tmp = pdev;
+			continue;
+		}
+		/* PCI device should connect to a PCIE bridge */

Quoted 8 lines

drivers/pci/search.c line 46

Quoted 6 lines

drivers/pci/search.c +6 -0

+		BUG_ON(pdev->pcie_type != PCI_EXP_TYPE_PCI_BRIDGE);
+		return pdev;
+	}
+
+	return tmp;
+}

No locking while you walk up the bus list?

Quoted 2 lines

hmm, iommu driver didn't support pci hotplug in current stage. But we can add lock here.

Please do, as PCI-E hotplug is much more common than PCI hotplug these days (think ExpressCard in millions of laptops...)

Quoted 18 lines

Quoted 18 lines

Quoted 11 lines

include/linux/pci.h +1 -0

@@ -126,6 +126,7 @@
 	unsigned short	subsystem_device;
 	unsigned int	class;		/* 3 bytes: (base,sub,prog-if) */
 	u8		hdr_type;	/* PCI header type (`multi' flag masked out) */
+	u8		pcie_type;	/* PCI-E device/port type */
 	u8		rom_base_reg;	/* which config register controls the ROM */
 	u8		pin;  		/* which interrupt pin this device uses */
 

include/linux/pci.h line 172

Quoted 4 lines

include/linux/pci.h +1 -0

@@ -168,6 +169,7 @@
 	unsigned int 	msi_enabled:1;
 	unsigned int	msix_enabled:1;
 	unsigned int	is_managed:1;
+	unsigned int	is_pcie:1;

Do you really need both fields? Wouldn't just the pcie_type one work (with some NOT_PCIE type being set for it if it isn't I suppose.)

Quoted 2 lines

There are some encodings are reserved for future. Just don't want to use reserved bits, as we don't know which one is proper for NOT_PCIE.

What are the current encodings? I don't have my copy of the pci spec availble at the moment...

Quoted 5 lines

Quoted 5 lines · Ashok Raj

Quoted 3 lines

Most GFX drivers don't call standard PCI DMA APIs to allocate DMA buffer, Such drivers will be broken with IOMMU enabled. To workaround this issue, we added two options.

Quoted 7 lines

Quoted 4 lines

All drm drivers do it. If the usual out of tree crap vendors are too stupid for their own sake it's their fault.

So NACK to this patch.

That's my feeling as well, everything we care about should be using the proper APIs or else what is the point of them...

They can't. There is no proper API to do IOMMU mappings from user space. And that is how Xorg on x86 works.

I had some hackish patches to enable mapping on /sys/bus/pci/.../coherent_mem, but you need ioctls to pass out the translated address and it wasn't exactly pretty. Still also not sure that's the right way.

Quoted 59 lines · Greg KH

Quoted 34 lines · Shaohua Li

Quoted 25 lines · Greg KH

Quoted 24 lines · Ashok Raj

drivers/pci/search.c +24 -0

+/*
+ * find the upstream PCIE-to-PCI bridge of a PCI device
+ * if the device is PCIE, return NULL
+ * if the device isn't connected to a PCIE bridge (that is its parent is a
+ * legacy PCI bridge and the bridge is directly connected to bus 0), return its
+ * parent
+ */
+struct pci_dev *
+pci_find_upstream_pcie_bridge(struct pci_dev *pdev)
+{
+	struct pci_dev *tmp = NULL;
+
+	if (pdev->is_pcie)
+		return NULL;
+	while (1) {
+		if (!pdev->bus->self)
+			break;
+		pdev = pdev->bus->self;
+		/* a p2p bridge */
+		if (!pdev->is_pcie) {
+			tmp = pdev;
+			continue;
+		}
+		/* PCI device should connect to a PCIE bridge */

Quoted 8 lines

drivers/pci/search.c line 46

Quoted 6 lines

drivers/pci/search.c +6 -0

+		BUG_ON(pdev->pcie_type != PCI_EXP_TYPE_PCI_BRIDGE);
+		return pdev;
+	}
+
+	return tmp;
+}

No locking while you walk up the bus list?

Quoted 2 lines

hmm, iommu driver didn't support pci hotplug in current stage. But we can add lock here.

Please do, as PCI-E hotplug is much more common than PCI hotplug these days (think ExpressCard in millions of laptops...)

Quoted 18 lines

Quoted 18 lines

Quoted 11 lines

include/linux/pci.h +1 -0

@@ -126,6 +126,7 @@
 	unsigned short	subsystem_device;
 	unsigned int	class;		/* 3 bytes: (base,sub,prog-if) */
 	u8		hdr_type;	/* PCI header type (`multi' flag masked out) */
+	u8		pcie_type;	/* PCI-E device/port type */
 	u8		rom_base_reg;	/* which config register controls the ROM */
 	u8		pin;  		/* which interrupt pin this device uses */
 

include/linux/pci.h line 172

Quoted 4 lines

include/linux/pci.h +1 -0

@@ -168,6 +169,7 @@
 	unsigned int 	msi_enabled:1;
 	unsigned int	msix_enabled:1;
 	unsigned int	is_managed:1;
+	unsigned int	is_pcie:1;

Do you really need both fields? Wouldn't just the pcie_type one work (with some NOT_PCIE type being set for it if it isn't I suppose.)

Quoted 5 lines

Quoted 2 lines

There are some encodings are reserved for future. Just don't want to use reserved bits, as we don't know which one is proper for NOT_PCIE.

What are the current encodings? I don't have my copy of the pci spec availble at the moment...

0 - 1010b is defined. Other bits are reserved. How about the pcie_type's highest bit determines NOT_PCIE? the pcie spec just use 4 bits for pcie type.

Quoted 34 lines · Shaohua Li

Quoted 33 lines · Greg KH

Quoted 20 lines · Ashok Raj

drivers/pci/search.c +20 -0

+/*
+ * find the upstream PCIE-to-PCI bridge of a PCI device
+ * if the device is PCIE, return NULL
+ * if the device isn't connected to a PCIE bridge (that is its parent is a
+ * legacy PCI bridge and the bridge is directly connected to bus 0), return its
+ * parent
+ */
+struct pci_dev *
+pci_find_upstream_pcie_bridge(struct pci_dev *pdev)
+{
+	struct pci_dev *tmp = NULL;
+
+	if (pdev->is_pcie)
+		return NULL;
+	while (1) {
+		if (!pdev->bus->self)
+			break;
+		pdev = pdev->bus->self;
+		/* a p2p bridge */
+		if (!pdev->is_pcie) {

drivers/pci/search.c line 46

Quoted 10 lines

drivers/pci/search.c +10 -0

+			tmp = pdev;
+			continue;
+		}
+		/* PCI device should connect to a PCIE bridge */
+		BUG_ON(pdev->pcie_type != PCI_EXP_TYPE_PCI_BRIDGE);
+		return pdev;
+	}
+
+	return tmp;
+}

No locking while you walk up the bus list?

Quoted 2 lines

hmm, iommu driver didn't support pci hotplug in current stage. But we can add lock here.

Please do, as PCI-E hotplug is much more common than PCI hotplug these days (think ExpressCard in millions of laptops...)