Skip to contentSkip to conversation

Enabling PCI ACS P2P upstream forwarding

PATCH Superseded Allen Kay 3 participants 10 messages Oct 6, 2009 – Oct 7, 2009

Subsystems (10)
Network drivers x86 Block layer PCI +6 more subsystems

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Allen Kay

author
#0 lore raw

PATCH v4 1/1 Enabling PCI ACS P2P upstream forwarding

This patch enables P2P upstream forwarding in ACS capable PCIe switches. It solves two potential problems in virtualization environment where a PCIe device is assigned to a guest domain using a HW iommu such as VT-d:

1) Unintentional failure caused by guest physical address programmed into the device's DMA that happens to match the memory address range of other downstream ports in the same PCIe switch. This causes the PCI transaction to go to the matching downstream port instead of go to the root complex to get translated by VT-d as it should be.

2) Malicious guest software intentionally attacks another downstream PCIe device by programming the DMA address into the assigned device that matches memory address range of the downstream PCIe port.

We are in process of implementing device filtering software in KVM/XEN management software to allow device assignment of PCIe devices behind a PCIe switch only if it has ACS capability and with the P2P upstream forwarding bits enabled. This patch is intended to work for both KVM and Xen environments.

Changes from initial to v1:
- removed #define ACS_ENABLE and dev_info() call
- changed ctrl value setting without using if-condition
- fixed ACS #defines in pci_regs.h

Changes from v2 to v3:
- change #define indention to 2 for PCI reg and 1 for bit position

Changes from v3 to v4:
- enable ACS only if iommu is enabled or if kernel is running as xen dom0
- changed xen_domain_type from enum to #define to allow it to be used in pci/probe.c and to avoid excessive changes to code structure.

---
 arch/x86/include/asm/xen/hypervisor.h |   10 +++-----
 arch/x86/xen/enlighten.c              |    2 +-
 drivers/pci/dmar.c                    |    1 +
 drivers/pci/pci.c                     |   35 +++++++++++++++++++++++++++++++++
 drivers/pci/pci.h                     |   10 +++++++++
 drivers/pci/probe.c                   |   11 ++++++++++
 include/linux/dmar.h                  |    2 +-
 include/linux/pci_regs.h              |   13 ++++++++++++
 8 files changed, 76 insertions(+), 8 deletions(-)
arch/x86/include/asm/xen/hypervisor.h +4 -6
index d5b7e90..a1e65dd 100644
@@ -37,14 +37,12 @@
 extern struct shared_info *HYPERVISOR_shared_info;
 extern struct start_info *xen_start_info;
 
-enum xen_domain_type {
-	XEN_NATIVE,		/* running on bare hardware    */
-	XEN_PV_DOMAIN,		/* running in a PV domain      */
-	XEN_HVM_DOMAIN,		/* running in a Xen hvm domain */
-};
+#define XEN_NATIVE	0	/* running on bare hardware    */
+#define XEN_PV_DOMAIN	1	/* running in a PV domain      */
+#define XEN_HVM_DOMAIN	2	/* running in a Xen hvm domain */
 
 #ifdef CONFIG_XEN
-extern enum xen_domain_type xen_domain_type;
+extern int xen_domain_type;
 #else
 #define xen_domain_type		XEN_NATIVE
 #endif
arch/x86/xen/enlighten.c +1 -1
index 3439616..b263e85 100644
@@ -62,7 +62,7 @@ EXPORT_SYMBOL_GPL(hypercall_page);
 DEFINE_PER_CPU(struct vcpu_info *, xen_vcpu);
 DEFINE_PER_CPU(struct vcpu_info, xen_vcpu_info);
 
-enum xen_domain_type xen_domain_type = XEN_NATIVE;
+int xen_domain_type = XEN_NATIVE;
 EXPORT_SYMBOL_GPL(xen_domain_type);
 
 struct start_info *xen_start_info;
drivers/pci/dmar.c +1 -0
index 22b02c6..814faa3 100644
@@ -35,6 +35,7 @@
 #include <linux/interrupt.h>
 #include <linux/tboot.h>
 #include <linux/dmi.h>
+#include "pci.h"
 
 #define PREFIX "DMAR: "
 
drivers/pci/pci.c +35 -0
index 6edecff..ec61c76 100644
@@ -1533,6 +1533,41 @@ void pci_enable_ari(struct pci_dev *dev)
 }
 
 /**
+ * pci_enable_acs - enable ACS if hardware support it
+ * @dev: the PCI device
+ */
+void pci_enable_acs(struct pci_dev *dev)
+{
+	int pos;
+	u16 cap;
+	u16 ctrl;
+
+	if (!dev->is_pcie)
+		return;
+
+	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS);
+	if (!pos)
+		return;
+
+	pci_read_config_word(dev, pos + PCI_ACS_CAP, &cap);
+	pci_read_config_word(dev, pos + PCI_ACS_CTRL, &ctrl);
+
+	/* Source Validation */
+	ctrl |= (cap & PCI_ACS_SV);
+
+	/* P2P Request Redirect */
+	ctrl |= (cap & PCI_ACS_RR);
+
+	/* P2P Completion Redirect */
+	ctrl |= (cap & PCI_ACS_CR);
+
+	/* Upstream Forwarding */
+	ctrl |= (cap & PCI_ACS_UF);
+
+	pci_write_config_word(dev, pos + PCI_ACS_CTRL, ctrl);
+}
+
+/**
  * pci_swizzle_interrupt_pin - swizzle INTx for device behind bridge
  * @dev: the PCI device
  * @pin: the INTx pin (1=INTA, 2=INTB, 3=INTD, 4=INTD)
drivers/pci/pci.h +10 -0
index d92d195..5b3a8f9 100644
@@ -311,4 +311,14 @@ static inline int pci_resource_alignment(struct pci_dev *dev,
 	return resource_alignment(res);
 }
 
+extern void pci_enable_acs(struct pci_dev *dev);
+
+#ifdef CONFIG_DMAR
+extern int iommu_detected;
+#endif
+
+#ifdef CONFIG_XEN
+extern int xen_domain_type;
+#endif
+
 #endif /* DRIVERS_PCI_H */
drivers/pci/probe.c +11 -0
index 8105e32..32703c7 100644
@@ -1014,6 +1014,17 @@ static void pci_init_capabilities(struct pci_dev *dev)
 
 	/* Single Root I/O Virtualization */
 	pci_iov_init(dev);
+
+	/* Enable ACS P2P upstream forwarding if HW iommu is detected */
+	if (iommu_detected)
+		pci_enable_acs(dev);
+
+#ifdef CONFIG_XEN
+	/* HW iommu is not visible in xen dom0 */
+	if (xen_domain_type)
+		pci_enable_acs(dev);
+#endif
+
 }
 
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
include/linux/dmar.h +1 -1
index 69a6fba..d13c21c 100644
@@ -196,7 +196,7 @@ extern irqreturn_t dmar_fault(int irq, void *dev_id);
 extern int arch_setup_dmar_msi(unsigned int irq);
 
 #ifdef CONFIG_DMAR
-extern int iommu_detected, no_iommu;
+extern int no_iommu;
 extern struct list_head dmar_rmrr_units;
 struct dmar_rmrr_unit {
 	struct list_head list;		/* list of rmrr units	*/
include/linux/pci_regs.h +13 -0
index dd0bed4..d798770 100644
@@ -502,6 +502,7 @@
 #define PCI_EXT_CAP_ID_VC	2
 #define PCI_EXT_CAP_ID_DSN	3
 #define PCI_EXT_CAP_ID_PWR	4
+#define PCI_EXT_CAP_ID_ACS	13
 #define PCI_EXT_CAP_ID_ARI	14
 #define PCI_EXT_CAP_ID_ATS	15
 #define PCI_EXT_CAP_ID_SRIOV	16
@@ -662,4 +663,16 @@
 #define  PCI_SRIOV_VFM_MO	0x2	/* Active.MigrateOut */
 #define  PCI_SRIOV_VFM_AV	0x3	/* Active.Available */
 
+/* Access Control Service */
+#define PCI_ACS_CAP		0x04	/* ACS Capability Register */
+#define  PCI_ACS_SV		0x01	/* Source Validation */
+#define  PCI_ACS_TB		0x02	/* Translation Blocking */
+#define  PCI_ACS_RR		0x04	/* P2P Request Redirect */
+#define  PCI_ACS_CR		0x08	/* P2P Completion Redirect */
+#define  PCI_ACS_UF		0x10	/* Upstream Forwarding */
+#define  PCI_ACS_EC		0x20	/* P2P Egress Control */
+#define  PCI_ACS_DT		0x40	/* Direct Translated P2P */
+#define PCI_ACS_CTRL		0x06	/* ACS Control Register */
+#define PCI_ACS_EGRESS_CTL_V	0x08	/* ACS Egress Control Vector */
+
 #endif /* LINUX_PCI_REGS_H */
Signature · 1 line
1.6.0.6
#1 lore raw
Quoted 5 lines · Allen Kay

Changes from v3 to v4:
- enable ACS only if iommu is enabled or if kernel is running as xen dom0
- changed xen_domain_type from enum to #define to allow it to be used in pci/probe.c and to avoid excessive changes to code structure.

Why can't it be used as an enum?

Quoted 5 lines

+++ b/drivers/pci/pci.h

drivers/pci/pci.h +0 -0
@@ -311,4 +311,14 @@ static inline int pci_resource_alignment(struct pci_dev *dev,
 	return resource_alignment(res);
 }
 

drivers/pci/pci.h static inline int pci_resource_alignment(struct pci_dev *dev, · line 323

Quoted 10 lines
drivers/pci/pci.h +9 -0
+extern void pci_enable_acs(struct pci_dev *dev);
+
+#ifdef CONFIG_DMAR
+extern int iommu_detected;
+#endif
+
+#ifdef CONFIG_XEN
+extern int xen_domain_type;
+#endif
 

That's pretty ugly; duplicate externs are pretty bad idea. Why not just include the right header? If its because its asm-x86, I'm happy to move the definitions to somewhere more common.

Quoted 11 lines
(quoted diff) +1 -0
+

#endif /* DRIVERS_PCI_H */

drivers/pci/probe.c +1 -0
index 8105e32..32703c7 100644
@@ -1014,6 +1014,17 @@ static void pci_init_capabilities(struct pci_dev *dev)
 
 	/* Single Root I/O Virtualization */
 	pci_iov_init(dev);
+

drivers/pci/probe.c static void pci_init_capabilities(struct pci_dev *dev) · line 1027

Quoted 10 lines
drivers/pci/probe.c +9 -0
+	/* Enable ACS P2P upstream forwarding if HW iommu is detected */
+	if (iommu_detected)
+		pci_enable_acs(dev);
+
+#ifdef CONFIG_XEN
+	/* HW iommu is not visible in xen dom0 */
+	if (xen_domain_type)
+		pci_enable_acs(dev);
+#endif
 

The idea is that you're supposed to use xen_domain() to test for Xenness; it expands to a constant 0 if Xen isn't configured, so there's no need to use #ifdefs.

Quoted 37 lines
drivers/pci/probe.c +1 -0
+
 }
 

void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)

include/linux/dmar.h +1 -1
index 69a6fba..d13c21c 100644
@@ -196,7 +196,7 @@ extern irqreturn_t dmar_fault(int irq, void *dev_id);
 extern int arch_setup_dmar_msi(unsigned int irq);
 
 #ifdef CONFIG_DMAR
-extern int iommu_detected, no_iommu;
+extern int no_iommu;
 extern struct list_head dmar_rmrr_units;
 struct dmar_rmrr_unit {
 	struct list_head list;		/* list of rmrr units	*/
include/linux/pci_regs.h +4 -0
index dd0bed4..d798770 100644
@@ -502,6 +502,7 @@
 #define PCI_EXT_CAP_ID_VC	2
 #define PCI_EXT_CAP_ID_DSN	3
 #define PCI_EXT_CAP_ID_PWR	4
+#define PCI_EXT_CAP_ID_ACS	13
 #define PCI_EXT_CAP_ID_ARI	14
 #define PCI_EXT_CAP_ID_ATS	15
 #define PCI_EXT_CAP_ID_SRIOV	16
@@ -662,4 +663,16 @@
 #define  PCI_SRIOV_VFM_MO	0x2	/* Active.MigrateOut */
 #define  PCI_SRIOV_VFM_AV	0x3	/* Active.Available */
 
+/* Access Control Service */
+#define PCI_ACS_CAP		0x04	/* ACS Capability Register */
+#define  PCI_ACS_SV		0x01	/* Source Validation */

include/linux/pci_regs.h line 678

Quoted 10 lines
include/linux/pci_regs.h +9 -0
+#define  PCI_ACS_TB		0x02	/* Translation Blocking */
+#define  PCI_ACS_RR		0x04	/* P2P Request Redirect */
+#define  PCI_ACS_CR		0x08	/* P2P Completion Redirect */
+#define  PCI_ACS_UF		0x10	/* Upstream Forwarding */
+#define  PCI_ACS_EC		0x20	/* P2P Egress Control */
+#define  PCI_ACS_DT		0x40	/* Direct Translated P2P */
+#define PCI_ACS_CTRL		0x06	/* ACS Control Register */
+#define PCI_ACS_EGRESS_CTL_V	0x08	/* ACS Egress Control Vector */
+
 #endif /* LINUX_PCI_REGS_H */

NAK

J

author NAK
#2 lore raw
Quoted 3 lines

That's pretty ugly; duplicate externs are pretty bad idea. Why not just include the right header? If its because its asm-x86, I'm happy to move the definitions to somewhere more common.

The main reason is lack of access of xen header files in driver/pci directory. If we can move xen header files to include/asm-x86, then the ugliness will go away.

Specifically, I would like to access header file containing xen_domain_type and associated enum definition from driver/pci directory.

-----Original Message-----
From: Jeremy Fitzhardinge [mailto:[email protected]]
Sent: Tuesday, October 06, 2009 2:38 PM
To: Kay, Allen M
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH ACS v4 1/1] Enabling PCI ACS P2P upstream forwarding

Quoted 5 lines · Allen Kay

Changes from v3 to v4:
- enable ACS only if iommu is enabled or if kernel is running as xen dom0
- changed xen_domain_type from enum to #define to allow it to be used in pci/probe.c and to avoid excessive changes to code structure.

Why can't it be used as an enum?

Quoted 5 lines

+++ b/drivers/pci/pci.h

drivers/pci/pci.h +0 -0
@@ -311,4 +311,14 @@ static inline int pci_resource_alignment(struct pci_dev *dev,
 	return resource_alignment(res);
 }
 

drivers/pci/pci.h static inline int pci_resource_alignment(struct pci_dev *dev, · line 323

Quoted 10 lines
drivers/pci/pci.h +9 -0
+extern void pci_enable_acs(struct pci_dev *dev);
+
+#ifdef CONFIG_DMAR
+extern int iommu_detected;
+#endif
+
+#ifdef CONFIG_XEN
+extern int xen_domain_type;
+#endif
 

That's pretty ugly; duplicate externs are pretty bad idea. Why not just include the right header? If its because its asm-x86, I'm happy to move the definitions to somewhere more common.

Quoted 11 lines
(quoted diff) +1 -0
+

#endif /* DRIVERS_PCI_H */

drivers/pci/probe.c +1 -0
index 8105e32..32703c7 100644
@@ -1014,6 +1014,17 @@ static void pci_init_capabilities(struct pci_dev *dev)
 
 	/* Single Root I/O Virtualization */
 	pci_iov_init(dev);
+

drivers/pci/probe.c static void pci_init_capabilities(struct pci_dev *dev) · line 1027

Quoted 10 lines
drivers/pci/probe.c +9 -0
+	/* Enable ACS P2P upstream forwarding if HW iommu is detected */
+	if (iommu_detected)
+		pci_enable_acs(dev);
+
+#ifdef CONFIG_XEN
+	/* HW iommu is not visible in xen dom0 */
+	if (xen_domain_type)
+		pci_enable_acs(dev);
+#endif
 

The idea is that you're supposed to use xen_domain() to test for Xenness; it expands to a constant 0 if Xen isn't configured, so there's no need to use #ifdefs.

Quoted 37 lines
drivers/pci/probe.c +1 -0
+
 }
 

void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)

include/linux/dmar.h +1 -1
index 69a6fba..d13c21c 100644
@@ -196,7 +196,7 @@ extern irqreturn_t dmar_fault(int irq, void *dev_id);
 extern int arch_setup_dmar_msi(unsigned int irq);
 
 #ifdef CONFIG_DMAR
-extern int iommu_detected, no_iommu;
+extern int no_iommu;
 extern struct list_head dmar_rmrr_units;
 struct dmar_rmrr_unit {
 	struct list_head list;		/* list of rmrr units	*/
include/linux/pci_regs.h +4 -0
index dd0bed4..d798770 100644
@@ -502,6 +502,7 @@
 #define PCI_EXT_CAP_ID_VC	2
 #define PCI_EXT_CAP_ID_DSN	3
 #define PCI_EXT_CAP_ID_PWR	4
+#define PCI_EXT_CAP_ID_ACS	13
 #define PCI_EXT_CAP_ID_ARI	14
 #define PCI_EXT_CAP_ID_ATS	15
 #define PCI_EXT_CAP_ID_SRIOV	16
@@ -662,4 +663,16 @@
 #define  PCI_SRIOV_VFM_MO	0x2	/* Active.MigrateOut */
 #define  PCI_SRIOV_VFM_AV	0x3	/* Active.Available */
 
+/* Access Control Service */
+#define PCI_ACS_CAP		0x04	/* ACS Capability Register */
+#define  PCI_ACS_SV		0x01	/* Source Validation */

include/linux/pci_regs.h line 678

Quoted 10 lines
include/linux/pci_regs.h +9 -0
+#define  PCI_ACS_TB		0x02	/* Translation Blocking */
+#define  PCI_ACS_RR		0x04	/* P2P Request Redirect */
+#define  PCI_ACS_CR		0x08	/* P2P Completion Redirect */
+#define  PCI_ACS_UF		0x10	/* Upstream Forwarding */
+#define  PCI_ACS_EC		0x20	/* P2P Egress Control */
+#define  PCI_ACS_DT		0x40	/* Direct Translated P2P */
+#define PCI_ACS_CTRL		0x06	/* ACS Control Register */
+#define PCI_ACS_EGRESS_CTL_V	0x08	/* ACS Egress Control Vector */
+
 #endif /* LINUX_PCI_REGS_H */

NAK

J

author NAK
#3 lore raw
In reply to Jeremy Fitzhardinge

Better move to somewhere arch independent as I'm calling this in architectural independent PCI driver code.

-----Original Message-----
From: Kay, Allen M
Sent: Tuesday, October 06, 2009 2:47 PM
To: 'Jeremy Fitzhardinge'
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH ACS v4 1/1] Enabling PCI ACS P2P upstream forwarding

Quoted 3 lines

That's pretty ugly; duplicate externs are pretty bad idea. Why not just include the right header? If its because its asm-x86, I'm happy to move the definitions to somewhere more common.

The main reason is lack of access of xen header files in driver/pci directory. If we can move xen header files to include/asm-x86, then the ugliness will go away.

Specifically, I would like to access header file containing xen_domain_type and associated enum definition from driver/pci directory.

-----Original Message-----
From: Jeremy Fitzhardinge [mailto:[email protected]]
Sent: Tuesday, October 06, 2009 2:38 PM
To: Kay, Allen M
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH ACS v4 1/1] Enabling PCI ACS P2P upstream forwarding

Quoted 5 lines · Allen Kay

Changes from v3 to v4:
- enable ACS only if iommu is enabled or if kernel is running as xen dom0
- changed xen_domain_type from enum to #define to allow it to be used in pci/probe.c and to avoid excessive changes to code structure.

Why can't it be used as an enum?

Quoted 5 lines

+++ b/drivers/pci/pci.h

drivers/pci/pci.h +0 -0
@@ -311,4 +311,14 @@ static inline int pci_resource_alignment(struct pci_dev *dev,
 	return resource_alignment(res);
 }
 

drivers/pci/pci.h static inline int pci_resource_alignment(struct pci_dev *dev, · line 323

Quoted 10 lines
drivers/pci/pci.h +9 -0
+extern void pci_enable_acs(struct pci_dev *dev);
+
+#ifdef CONFIG_DMAR
+extern int iommu_detected;
+#endif
+
+#ifdef CONFIG_XEN
+extern int xen_domain_type;
+#endif
 

That's pretty ugly; duplicate externs are pretty bad idea. Why not just include the right header? If its because its asm-x86, I'm happy to move the definitions to somewhere more common.

Quoted 11 lines
(quoted diff) +1 -0
+

#endif /* DRIVERS_PCI_H */

drivers/pci/probe.c +1 -0
index 8105e32..32703c7 100644
@@ -1014,6 +1014,17 @@ static void pci_init_capabilities(struct pci_dev *dev)
 
 	/* Single Root I/O Virtualization */
 	pci_iov_init(dev);
+

drivers/pci/probe.c static void pci_init_capabilities(struct pci_dev *dev) · line 1027

Quoted 10 lines
drivers/pci/probe.c +9 -0
+	/* Enable ACS P2P upstream forwarding if HW iommu is detected */
+	if (iommu_detected)
+		pci_enable_acs(dev);
+
+#ifdef CONFIG_XEN
+	/* HW iommu is not visible in xen dom0 */
+	if (xen_domain_type)
+		pci_enable_acs(dev);
+#endif
 

The idea is that you're supposed to use xen_domain() to test for Xenness; it expands to a constant 0 if Xen isn't configured, so there's no need to use #ifdefs.

Quoted 37 lines
drivers/pci/probe.c +1 -0
+
 }
 

void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)

include/linux/dmar.h +1 -1
index 69a6fba..d13c21c 100644
@@ -196,7 +196,7 @@ extern irqreturn_t dmar_fault(int irq, void *dev_id);
 extern int arch_setup_dmar_msi(unsigned int irq);
 
 #ifdef CONFIG_DMAR
-extern int iommu_detected, no_iommu;
+extern int no_iommu;
 extern struct list_head dmar_rmrr_units;
 struct dmar_rmrr_unit {
 	struct list_head list;		/* list of rmrr units	*/
include/linux/pci_regs.h +4 -0
index dd0bed4..d798770 100644
@@ -502,6 +502,7 @@
 #define PCI_EXT_CAP_ID_VC	2
 #define PCI_EXT_CAP_ID_DSN	3
 #define PCI_EXT_CAP_ID_PWR	4
+#define PCI_EXT_CAP_ID_ACS	13
 #define PCI_EXT_CAP_ID_ARI	14
 #define PCI_EXT_CAP_ID_ATS	15
 #define PCI_EXT_CAP_ID_SRIOV	16
@@ -662,4 +663,16 @@
 #define  PCI_SRIOV_VFM_MO	0x2	/* Active.MigrateOut */
 #define  PCI_SRIOV_VFM_AV	0x3	/* Active.Available */
 
+/* Access Control Service */
+#define PCI_ACS_CAP		0x04	/* ACS Capability Register */
+#define  PCI_ACS_SV		0x01	/* Source Validation */

include/linux/pci_regs.h line 678

Quoted 10 lines
include/linux/pci_regs.h +9 -0
+#define  PCI_ACS_TB		0x02	/* Translation Blocking */
+#define  PCI_ACS_RR		0x04	/* P2P Request Redirect */
+#define  PCI_ACS_CR		0x08	/* P2P Completion Redirect */
+#define  PCI_ACS_UF		0x10	/* Upstream Forwarding */
+#define  PCI_ACS_EC		0x20	/* P2P Egress Control */
+#define  PCI_ACS_DT		0x40	/* Direct Translated P2P */
+#define PCI_ACS_CTRL		0x06	/* ACS Control Register */
+#define PCI_ACS_EGRESS_CTL_V	0x08	/* ACS Egress Control Vector */
+
 #endif /* LINUX_PCI_REGS_H */

NAK

J

#4 lore raw
Quoted 2 lines · Allen M

Better move to somewhere arch independent as I'm calling this in architectural independent PCI driver code.

I'm prepping a patch for you now, to move it to xen/xen.h.

J

#5 lore raw

Thanks! I will re-spin my patch.

-----Original Message-----
From: Jeremy Fitzhardinge [mailto:[email protected]]
Sent: Tuesday, October 06, 2009 3:05 PM
To: Kay, Allen M
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH ACS v4 1/1] Enabling PCI ACS P2P upstream forwarding

Quoted 2 lines · Allen M

Better move to somewhere arch independent as I'm calling this in architectural independent PCI driver code.

I'm prepping a patch for you now, to move it to xen/xen.h.

J

#6 lore raw

PATCH 1/1 xen: move Xen-testing predicates to common header

In reply to Kay, Allen M

Move xen_domain and related tests out of asm-x86 to xen/xen.h so they can be included whenever they are necessary.

--

 arch/x86/include/asm/xen/hypervisor.h |   27 ---------------------------
 arch/x86/xen/enlighten.c              |    1 +
 drivers/block/xen-blkfront.c          |    1 +
 drivers/char/hvc_xen.c                |    2 ++
 drivers/input/xen-kbdfront.c          |    3 +++
 drivers/net/xen-netfront.c            |    1 +
 drivers/video/xen-fbfront.c           |    3 +++
 drivers/xen/balloon.c                 |    2 ++
 drivers/xen/cpu_hotplug.c             |    1 +
 drivers/xen/evtchn.c                  |    2 ++
 drivers/xen/grant-table.c             |    1 +
 drivers/xen/sys-hypervisor.c          |    1 +
 drivers/xen/xenbus/xenbus_probe.c     |    2 ++
 drivers/xen/xenfs/super.c             |    2 ++
 include/xen/xen.h                     |   32 ++++++++++++++++++++++++++++++++
arch/x86/include/asm/xen/hypervisor.h +0 -27
index d5b7e90..396ff4c 100644
@@ -37,31 +37,4 @@
 extern struct shared_info *HYPERVISOR_shared_info;
 extern struct start_info *xen_start_info;
 
-enum xen_domain_type {
-	XEN_NATIVE,		/* running on bare hardware    */
-	XEN_PV_DOMAIN,		/* running in a PV domain      */
-	XEN_HVM_DOMAIN,		/* running in a Xen hvm domain */
-};
-
-#ifdef CONFIG_XEN
-extern enum xen_domain_type xen_domain_type;
-#else
-#define xen_domain_type		XEN_NATIVE
-#endif
-
-#define xen_domain()		(xen_domain_type != XEN_NATIVE)
-#define xen_pv_domain()		(xen_domain() &&			\
-				 xen_domain_type == XEN_PV_DOMAIN)
-#define xen_hvm_domain()	(xen_domain() &&			\
-				 xen_domain_type == XEN_HVM_DOMAIN)
-
-#ifdef CONFIG_XEN_DOM0
-#include <xen/interface/xen.h>
-
-#define xen_initial_domain()	(xen_pv_domain() && \
-				 xen_start_info->flags & SIF_INITDOMAIN)
-#else  /* !CONFIG_XEN_DOM0 */
-#define xen_initial_domain()	(0)
-#endif	/* CONFIG_XEN_DOM0 */
-
 #endif /* _ASM_X86_XEN_HYPERVISOR_H */
arch/x86/xen/enlighten.c +1 -0
index eb33aaa..f0db2b9 100644
@@ -28,6 +28,7 @@
 #include <linux/highmem.h>
 #include <linux/console.h>
 
+#include <xen/xen.h>
 #include <xen/interface/xen.h>
 #include <xen/interface/version.h>
 #include <xen/interface/physdev.h>
drivers/block/xen-blkfront.c +1 -0
index e532847..1e5baf7 100644
@@ -42,6 +42,7 @@
 #include <linux/module.h>
 #include <linux/scatterlist.h>
 
+#include <xen/xen.h>
 #include <xen/xenbus.h>
 #include <xen/grant_table.h>
 #include <xen/events.h>
drivers/char/hvc_xen.c +2 -0
index eba999f..93d3381 100644
@@ -25,6 +25,8 @@
 #include <linux/types.h>
 
 #include <asm/xen/hypervisor.h>
+
+#include <xen/xen.h>
 #include <xen/page.h>
 #include <xen/events.h>
 #include <xen/interface/io/console.h>
drivers/input/xen-kbdfront.c +3 -0
index b115726..c721c0a 100644
@@ -21,7 +21,10 @@
 #include <linux/errno.h>
 #include <linux/module.h>
 #include <linux/input.h>
+
 #include <asm/xen/hypervisor.h>
+
+#include <xen/xen.h>
 #include <xen/events.h>
 #include <xen/page.h>
 #include <xen/interface/io/fbif.h>
drivers/net/xen-netfront.c +1 -0
index 8d88dae..c2567ad 100644
@@ -42,6 +42,7 @@
 #include <linux/mm.h>
 #include <net/ip.h>
 
+#include <xen/xen.h>
 #include <xen/xenbus.h>
 #include <xen/events.h>
 #include <xen/page.h>
drivers/video/xen-fbfront.c +3 -0
index 54cd916..966b226 100644
@@ -25,7 +25,10 @@
 #include <linux/module.h>
 #include <linux/vmalloc.h>
 #include <linux/mm.h>
+
 #include <asm/xen/hypervisor.h>
+
+#include <xen/xen.h>
 #include <xen/events.h>
 #include <xen/page.h>
 #include <xen/interface/io/fbif.h>
drivers/xen/balloon.c +2 -0
index f5bbd9e..3a27e68 100644
@@ -52,6 +52,8 @@
 
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+
+#include <xen/xen.h>
 #include <xen/interface/xen.h>
 #include <xen/interface/memory.h>
 #include <xen/xenbus.h>
drivers/xen/cpu_hotplug.c +1 -0
index bdfd584..6625ffe 100644
@@ -1,5 +1,6 @@
 #include <linux/notifier.h>
 
+#include <xen/xen.h>
 #include <xen/xenbus.h>
 
 #include <asm/xen/hypervisor.h>
drivers/xen/evtchn.c +2 -0
index af03195..8356843 100644
@@ -49,6 +49,8 @@
 #include <linux/gfp.h>
 #include <linux/mutex.h>
 #include <linux/cpu.h>
+
+#include <xen/xen.h>
 #include <xen/events.h>
 #include <xen/evtchn.h>
 #include <asm/xen/hypervisor.h>
drivers/xen/grant-table.c +1 -0
index 7d8f531..4c6c0bd 100644
@@ -37,6 +37,7 @@
 #include <linux/vmalloc.h>
 #include <linux/uaccess.h>
 
+#include <xen/xen.h>
 #include <xen/interface/xen.h>
 #include <xen/page.h>
 #include <xen/grant_table.h>
drivers/xen/sys-hypervisor.c +1 -0
index 88a60e0..ae5cb05 100644
@@ -14,6 +14,7 @@
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
 
+#include <xen/xen.h>
 #include <xen/xenbus.h>
 #include <xen/interface/xen.h>
 #include <xen/interface/version.h>
drivers/xen/xenbus/xenbus_probe.c +2 -0
index d42e25d..08638ad 100644
@@ -49,6 +49,8 @@
 #include <asm/page.h>
 #include <asm/pgtable.h>
 #include <asm/xen/hypervisor.h>
+
+#include <xen/xen.h>
 #include <xen/xenbus.h>
 #include <xen/events.h>
 #include <xen/page.h>
drivers/xen/xenfs/super.c +2 -0
index 6559e0c..8924d93 100644
@@ -13,6 +13,8 @@
 #include <linux/fs.h>
 #include <linux/magic.h>
 
+#include <xen/xen.h>
+
 #include "xenfs.h"
 
 #include <asm/xen/hypervisor.h>
include/xen/xen.h new file +32 -0
new file mode 100644
index 0000000..a164024
@@ -0,0 +1,32 @@
+#ifndef _XEN_XEN_H
+#define _XEN_XEN_H
+
+enum xen_domain_type {
+	XEN_NATIVE,		/* running on bare hardware    */
+	XEN_PV_DOMAIN,		/* running in a PV domain      */
+	XEN_HVM_DOMAIN,		/* running in a Xen hvm domain */
+};
+
+#ifdef CONFIG_XEN
+extern enum xen_domain_type xen_domain_type;
+#else
+#define xen_domain_type		XEN_NATIVE
+#endif
+
+#define xen_domain()		(xen_domain_type != XEN_NATIVE)
+#define xen_pv_domain()		(xen_domain() &&			\
+				 xen_domain_type == XEN_PV_DOMAIN)
+#define xen_hvm_domain()	(xen_domain() &&			\
+				 xen_domain_type == XEN_HVM_DOMAIN)
+
+#ifdef CONFIG_XEN_DOM0
+#include <xen/interface/xen.h>
+#include <asm/xen/hypervisor.h>
+
+#define xen_initial_domain()	(xen_pv_domain() && \
+				 xen_start_info->flags & SIF_INITDOMAIN)
+#else  /* !CONFIG_XEN_DOM0 */
+#define xen_initial_domain()	(0)
+#endif	/* CONFIG_XEN_DOM0 */
+
+#endif	/* _XEN_XEN_H */
#7 lore raw
In reply to Allen Kay

drivers/pci/pci.h static inline int pci_resource_alignment(struct pci_dev *dev, · line 318

Quoted 3 lines · * Allen Kay ([email protected]) wrote:
drivers/pci/pci.h +3 -0
+#ifdef CONFIG_DMAR
+extern int iommu_detected;
+#endif

This should not be needed.

drivers/pci/pci.h static inline int pci_resource_alignment(struct pci_dev *dev, · line 322

Quoted 4 lines
drivers/pci/pci.h +4 -0
+
+#ifdef CONFIG_XEN
+extern int xen_domain_type;
+#endif

Nor this (there's already a check for is dom0 called xen_initial_domain(), but unclear it's relevant yet for this patch).

drivers/pci/probe.c static void pci_init_capabilities(struct pci_dev *dev) · line 1019

Quoted 3 lines
drivers/pci/probe.c +3 -0
+
+	/* Enable ACS P2P upstream forwarding if HW iommu is detected */
+	if (iommu_detected)

I think you'd want iommu_found() instead. To avoid, e.g., GART triggering this one.

drivers/pci/probe.c static void pci_init_capabilities(struct pci_dev *dev) · line 1025

Quoted 6 lines
drivers/pci/probe.c +6 -0
+		pci_enable_acs(dev);
+
+#ifdef CONFIG_XEN
+	/* HW iommu is not visible in xen dom0 */
+	if (xen_domain_type)
+		pci_enable_acs(dev);

could do this (xen_initial_domain()) above, but it's only relevant for dom0 (so not needed yet?), and really seems to like it should be done by hv.

thanks,
-chris

#8 lore raw
Quoted 8 lines · * Chris Wright ([email protected]) wrote:

drivers/pci/probe.c static void pci_init_capabilities(struct pci_dev *dev) · line 1025

Quoted 4 lines · * Allen Kay ([email protected]) wrote:
drivers/pci/probe.c +4 -0
+#ifdef CONFIG_XEN
+	/* HW iommu is not visible in xen dom0 */
+	if (xen_domain_type)
+		pci_enable_acs(dev);

could do this (xen_initial_domain()) above, but it's only relevant for dom0 (so not needed yet?), and really seems to like it should be done by hv.

Guess it doesn't matter since dom0 could just as easily undo it, so nevermind on the hv bit ;-)

#9 lore raw
In reply to Chris Wright

Thanks Chris, I was not aware of these two functions. I have made the following change:

	if (iommu_found() || xen_initial_domain())
		pci_enable_acs(dev);

xen_intial_domain() should take care of dom0 case. We just make the rough assumption that P2P upstream forwarding is needed if we are running on top of xen HV - similar to what we are doing for native kernel case.

If this is ok, I'm going to re-spin the patch. Separate out xen_initial_domain() modification since it is predicated on Jeremy's xen.h patch.

-----Original Message-----
From: Chris Wright [mailto:[email protected]]
Sent: Tuesday, October 06, 2009 4:42 PM
To: Kay, Allen M
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH ACS v4 1/1] Enabling PCI ACS P2P upstream forwarding

drivers/pci/pci.h static inline int pci_resource_alignment(struct pci_dev *dev, · line 318

Quoted 3 lines · * Allen Kay ([email protected]) wrote:
drivers/pci/pci.h +3 -0
+#ifdef CONFIG_DMAR
+extern int iommu_detected;
+#endif

This should not be needed.

drivers/pci/pci.h static inline int pci_resource_alignment(struct pci_dev *dev, · line 322

Quoted 4 lines
drivers/pci/pci.h +4 -0
+
+#ifdef CONFIG_XEN
+extern int xen_domain_type;
+#endif

Nor this (there's already a check for is dom0 called xen_initial_domain(), but unclear it's relevant yet for this patch).

drivers/pci/probe.c static void pci_init_capabilities(struct pci_dev *dev) · line 1019

Quoted 3 lines
drivers/pci/probe.c +3 -0
+
+	/* Enable ACS P2P upstream forwarding if HW iommu is detected */
+	if (iommu_detected)

I think you'd want iommu_found() instead. To avoid, e.g., GART triggering this one.

drivers/pci/probe.c static void pci_init_capabilities(struct pci_dev *dev) · line 1025

Quoted 6 lines
drivers/pci/probe.c +6 -0
+		pci_enable_acs(dev);
+
+#ifdef CONFIG_XEN
+	/* HW iommu is not visible in xen dom0 */
+	if (xen_domain_type)
+		pci_enable_acs(dev);

could do this (xen_initial_domain()) above, but it's only relevant for dom0 (so not needed yet?), and really seems to like it should be done by hv.

thanks,
-chris