Skip to contentSkip to conversation

PCI: Relax ACS requirement for Intel RCiEP devices.

PATCH In review Ashok Raj 4 participants 8 messages May 28, 2020 – Jun 1, 2020 PCI

lore
Share & tools
lore

Ashok Raj

author
#0 lore raw

PATCH 1/1 PCI: Relax ACS requirement for Intel RCiEP devices.

All Intel platforms guarantee that all root complex implementations must send transactions up to IOMMU for address translations. Hence for RCiEP devices that are Vendor ID Intel, can claim exception for lack of ACS support.

3.16 Root-Complex Peer to Peer Considerations
When DMA remapping is enabled, peer-to-peer requests through the Root-Complex must be handled
as follows:
• The input address in the request is translated (through first-level, second-level or nested translation) to a host physical address (HPA). The address decoding for peer addresses must be done only on the translated HPA. Hardware implementations are free to further limit peer-to-peer accesses to specific host physical address regions (or to completely disallow peer-forwarding of translated requests).
• Since address translation changes the contents (address field) of the PCI Express Transaction Layer Packet (TLP), for PCI Express peer-to-peer requests with ECRC, the Root-Complex hardware must use the new ECRC (re-computed with the translated address) if it decides to forward the TLP as a peer request.
• Root-ports, and multi-function root-complex integrated endpoints, may support additional peerto-peer control features by supporting PCI Express Access Control Services (ACS) capability. Refer to ACS capability in PCI Express specifications for details.

Since Linux didn't give special treatment to allow this exception, certain RCiEP MFD devices are getting grouped in a single iommu group. This doesn't permit a single device to be assigned to a guest for instance.

In one vendor system: Device 14.x were grouped in a single IOMMU group.

/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.3

After the patch:
/sys/kernel/iommu_groups/5/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:14.2 /sys/kernel/iommu_groups/6/devices/0000:00:14.3 <<< new group

14.0 and 14.2 are integrated devices, but legacy end points. Whereas 14.3 was a PCIe compliant RCiEP.

00:14.3 Network controller: Intel Corporation Device 9df0 (rev 30)
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00

This permits assigning this device to a guest VM.

Fixes f096c061f552 ("iommu: Rework iommu_group_get_for_pci_dev()")

To: Joerg Roedel <[email protected]>
To: Bjorn Helgaas <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: Lu Baolu <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Darrel Goeddel <[email protected]>
Cc: Mark Scott <[email protected]>,
Cc: Romil Sharma <[email protected]>
Cc: Ashok Raj <[email protected]>
---
v2: Moved functionality from iommu to pci quirks - Alex Williamson

 drivers/pci/quirks.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
drivers/pci/quirks.c +15 -0
index 28c9a2409c50..63373ca0a3fe 100644
@@ -4682,6 +4682,20 @@ static int pci_quirk_mf_endpoint_acs(struct pci_dev *dev, u16 acs_flags)
 		PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_DT);
 }
 
+static int pci_quirk_rciep_acs(struct pci_dev *dev, u16 acs_flags)
+{
+	/*
+	 * RCiEP's are required to allow p2p only on translated addresses.
+	 * Refer to Intel VT-d specification Section 3.16 Root-Complex Peer
+	 * to Peer Considerations
+	 */
+	if (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_END)
+		return -ENOTTY;
+
+	return pci_acs_ctrl_enabled(acs_flags,
+		PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF);
+}
+
 static int pci_quirk_brcm_acs(struct pci_dev *dev, u16 acs_flags)
 {
 	/*
@@ -4764,6 +4778,7 @@ static const struct pci_dev_acs_enabled {
 	/* I219 */
 	{ PCI_VENDOR_ID_INTEL, 0x15b7, pci_quirk_mf_endpoint_acs },
 	{ PCI_VENDOR_ID_INTEL, 0x15b8, pci_quirk_mf_endpoint_acs },
+	{ PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_rciep_acs },
 	/* QCOM QDF2xxx root ports */
 	{ PCI_VENDOR_ID_QCOM, 0x0400, pci_quirk_qcom_rp_acs },
 	{ PCI_VENDOR_ID_QCOM, 0x0401, pci_quirk_qcom_rp_acs },
Signature · 1 line
2.7.4
Reviewed-by on v1 1/1
#1 lore raw
Quoted 44 lines · On Thu, 28 May 2020 13:57:42 -0700 Ashok Raj <[email protected]> wrote:

All Intel platforms guarantee that all root complex implementations must send transactions up to IOMMU for address translations. Hence for RCiEP devices that are Vendor ID Intel, can claim exception for lack of ACS support.

3.16 Root-Complex Peer to Peer Considerations
When DMA remapping is enabled, peer-to-peer requests through the Root-Complex must be handled
as follows:
• The input address in the request is translated (through first-level, second-level or nested translation) to a host physical address (HPA). The address decoding for peer addresses must be done only on the translated HPA. Hardware implementations are free to further limit peer-to-peer accesses to specific host physical address regions (or to completely disallow peer-forwarding of translated requests).
• Since address translation changes the contents (address field) of the PCI Express Transaction Layer Packet (TLP), for PCI Express peer-to-peer requests with ECRC, the Root-Complex hardware must use the new ECRC (re-computed with the translated address) if it decides to forward the TLP as a peer request.
• Root-ports, and multi-function root-complex integrated endpoints, may support additional peerto-peer control features by supporting PCI Express Access Control Services (ACS) capability. Refer to ACS capability in PCI Express specifications for details.

Since Linux didn't give special treatment to allow this exception, certain RCiEP MFD devices are getting grouped in a single iommu group. This doesn't permit a single device to be assigned to a guest for instance.

In one vendor system: Device 14.x were grouped in a single IOMMU group.

/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.3

After the patch:
/sys/kernel/iommu_groups/5/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:14.2 /sys/kernel/iommu_groups/6/devices/0000:00:14.3 <<< new group

14.0 and 14.2 are integrated devices, but legacy end points. Whereas 14.3 was a PCIe compliant RCiEP.

Quoted 6 lines

00:14.3 Network controller: Intel Corporation Device 9df0 (rev 30)
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00

This permits assigning this device to a guest VM.

Fixes f096c061f552 ("iommu: Rework iommu_group_get_for_pci_dev()")

I don't really understand this Fixes tag. This seems like a feature, not a fix. If you want it in stable releases as a feature, request it via Cc: [email protected]. I'd drop that tag, that's my nit. Otherwise:

Quoted 50 lines

To: Joerg Roedel <[email protected]>
To: Bjorn Helgaas <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: Lu Baolu <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Darrel Goeddel <[email protected]>
Cc: Mark Scott <[email protected]>,
Cc: Romil Sharma <[email protected]>
Cc: Ashok Raj <[email protected]>
---
v2: Moved functionality from iommu to pci quirks - Alex Williamson

 drivers/pci/quirks.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
drivers/pci/quirks.c +15 -0
index 28c9a2409c50..63373ca0a3fe 100644
@@ -4682,6 +4682,20 @@ static int pci_quirk_mf_endpoint_acs(struct pci_dev *dev, u16 acs_flags)
 		PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_DT);
 }
 
+static int pci_quirk_rciep_acs(struct pci_dev *dev, u16 acs_flags)
+{
+	/*
+	 * RCiEP's are required to allow p2p only on translated addresses.
+	 * Refer to Intel VT-d specification Section 3.16 Root-Complex Peer
+	 * to Peer Considerations
+	 */
+	if (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_END)
+		return -ENOTTY;
+
+	return pci_acs_ctrl_enabled(acs_flags,
+		PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF);
+}
+
 static int pci_quirk_brcm_acs(struct pci_dev *dev, u16 acs_flags)
 {
 	/*
@@ -4764,6 +4778,7 @@ static const struct pci_dev_acs_enabled {
 	/* I219 */
 	{ PCI_VENDOR_ID_INTEL, 0x15b7, pci_quirk_mf_endpoint_acs },
 	{ PCI_VENDOR_ID_INTEL, 0x15b8, pci_quirk_mf_endpoint_acs },
+	{ PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_rciep_acs },
 	/* QCOM QDF2xxx root ports */
 	{ PCI_VENDOR_ID_QCOM, 0x0400, pci_quirk_qcom_rp_acs },
 	{ PCI_VENDOR_ID_QCOM, 0x0401, pci_quirk_qcom_rp_acs },
author
#2 lore raw
Quoted 50 lines · Alex Williamson
Quoted 47 lines · On Thu, 28 May 2020 13:57:42 -0700 Ashok Raj <[email protected]> wrote:

All Intel platforms guarantee that all root complex implementations must send transactions up to IOMMU for address translations. Hence for RCiEP devices that are Vendor ID Intel, can claim exception for lack of ACS support.

3.16 Root-Complex Peer to Peer Considerations
When DMA remapping is enabled, peer-to-peer requests through the Root-Complex must be handled
as follows:
• The input address in the request is translated (through first-level, second-level or nested translation) to a host physical address (HPA). The address decoding for peer addresses must be done only on the translated HPA. Hardware implementations are free to further limit peer-to-peer accesses to specific host physical address regions (or to completely disallow peer-forwarding of translated requests).
• Since address translation changes the contents (address field) of the PCI Express Transaction Layer Packet (TLP), for PCI Express peer-to-peer requests with ECRC, the Root-Complex hardware must use the new ECRC (re-computed with the translated address) if it decides to forward the TLP as a peer request.
• Root-ports, and multi-function root-complex integrated endpoints, may support additional peerto-peer control features by supporting PCI Express Access Control Services (ACS) capability. Refer to ACS capability in PCI Express specifications for details.

Since Linux didn't give special treatment to allow this exception, certain RCiEP MFD devices are getting grouped in a single iommu group. This doesn't permit a single device to be assigned to a guest for instance.

In one vendor system: Device 14.x were grouped in a single IOMMU group.

/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.3

After the patch:
/sys/kernel/iommu_groups/5/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:14.2 /sys/kernel/iommu_groups/6/devices/0000:00:14.3 <<< new group

14.0 and 14.2 are integrated devices, but legacy end points. Whereas 14.3 was a PCIe compliant RCiEP.

00:14.3 Network controller: Intel Corporation Device 9df0 (rev 30)
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00

Quoted 8 lines
Quoted 3 lines

This permits assigning this device to a guest VM.

Fixes f096c061f552 ("iommu: Rework iommu_group_get_for_pci_dev()")

I don't really understand this Fixes tag. This seems like a feature, not a fix. If you want it in stable releases as a feature, request it via Cc: [email protected]. I'd drop that tag, that's my nit. Otherwise:

Yes, i should have Cced Stable instead.

Bjorn: Can you massage this in? or i can resend with Alex's Reviewed-by + adding stable in cc list.

Quoted 45 lines
Quoted 42 lines

To: Joerg Roedel <[email protected]>
To: Bjorn Helgaas <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: Lu Baolu <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Darrel Goeddel <[email protected]>
Cc: Mark Scott <[email protected]>,
Cc: Romil Sharma <[email protected]>
Cc: Ashok Raj <[email protected]>
---
v2: Moved functionality from iommu to pci quirks - Alex Williamson

 drivers/pci/quirks.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
drivers/pci/quirks.c +14 -0
index 28c9a2409c50..63373ca0a3fe 100644
@@ -4682,6 +4682,20 @@ static int pci_quirk_mf_endpoint_acs(struct pci_dev *dev, u16 acs_flags)
 		PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_DT);
 }
 
+static int pci_quirk_rciep_acs(struct pci_dev *dev, u16 acs_flags)
+{
+	/*
+	 * RCiEP's are required to allow p2p only on translated addresses.
+	 * Refer to Intel VT-d specification Section 3.16 Root-Complex Peer
+	 * to Peer Considerations
+	 */
+	if (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_END)
+		return -ENOTTY;
+
+	return pci_acs_ctrl_enabled(acs_flags,
+		PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF);
+}
+
 static int pci_quirk_brcm_acs(struct pci_dev *dev, u16 acs_flags)
 {
 	/*
Quoted 9 lines

drivers/pci/quirks.c static const struct pci_dev_acs_enabled { · line 4784

Quoted 8 lines
drivers/pci/quirks.c +1 -0
@@ -4764,6 +4778,7 @@ static const struct pci_dev_acs_enabled {
 	/* I219 */
 	{ PCI_VENDOR_ID_INTEL, 0x15b7, pci_quirk_mf_endpoint_acs },
 	{ PCI_VENDOR_ID_INTEL, 0x15b8, pci_quirk_mf_endpoint_acs },
+	{ PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_rciep_acs },
 	/* QCOM QDF2xxx root ports */
 	{ PCI_VENDOR_ID_QCOM, 0x0400, pci_quirk_qcom_rp_acs },
 	{ PCI_VENDOR_ID_QCOM, 0x0401, pci_quirk_qcom_rp_acs },

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Tested-by
#3 lore raw
In reply to Alex Williamson
Quoted 52 lines · Alex Williamson
Quoted 49 lines · On Thu, 28 May 2020 13:57:42 -0700 Ashok Raj <[email protected]> wrote:

All Intel platforms guarantee that all root complex implementations must send transactions up to IOMMU for address translations. Hence for RCiEP devices that are Vendor ID Intel, can claim exception for lack of ACS support.

3.16 Root-Complex Peer to Peer Considerations
When DMA remapping is enabled, peer-to-peer requests through the Root-Complex must be handled
as follows:
• The input address in the request is translated (through first-level, second-level or nested translation) to a host physical address (HPA). The address decoding for peer addresses must be done only on the translated HPA. Hardware implementations are free to further limit peer-to-peer accesses to specific host physical address regions (or to completely disallow peer-forwarding of translated requests).
• Since address translation changes the contents (address field) of the PCI Express Transaction Layer Packet (TLP), for PCI Express peer-to-peer requests with ECRC, the Root-Complex hardware must use the new ECRC (re-computed with the translated address) if it decides to forward the TLP as a peer request.
• Root-ports, and multi-function root-complex integrated endpoints, may support additional peerto-peer control features by supporting PCI Express Access Control Services (ACS) capability. Refer to ACS capability in PCI Express specifications for details.

Since Linux didn't give special treatment to allow this exception, certain RCiEP MFD devices are getting grouped in a single iommu group. This doesn't permit a single device to be assigned to a guest for instance.

In one vendor system: Device 14.x were grouped in a single IOMMU group.

/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.3

After the patch:
/sys/kernel/iommu_groups/5/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:14.2 /sys/kernel/iommu_groups/6/devices/0000:00:14.3 <<< new group

14.0 and 14.2 are integrated devices, but legacy end points. Whereas 14.3 was a PCIe compliant RCiEP.

00:14.3 Network controller: Intel Corporation Device 9df0 (rev 30)
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00

This permits assigning this device to a guest VM.

Quoted 8 lines
Quoted 1 line
Fixes f096c061f552 ("iommu: Rework iommu_group_get_for_pci_dev()")

I don't really understand this Fixes tag. This seems like a feature, not a fix. If you want it in stable releases as a feature, request it via Cc: [email protected]. I'd drop that tag, that's my nit. Otherwise:

I have tested this patch with 5.6.14 as well as a slightly modified version (without pci_acs_ctrl_enabled()) in a 3.10 enterprise linux kernel.

Quoted 51 lines
Quoted 50 lines

To: Joerg Roedel <[email protected]>
To: Bjorn Helgaas <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: Lu Baolu <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Darrel Goeddel <[email protected]>
Cc: Mark Scott <[email protected]>,
Cc: Romil Sharma <[email protected]>
Cc: Ashok Raj <[email protected]>
---
v2: Moved functionality from iommu to pci quirks - Alex Williamson

  drivers/pci/quirks.c | 15 +++++++++++++++
  1 file changed, 15 insertions(+)
drivers/pci/quirks.c +15 -0
index 28c9a2409c50..63373ca0a3fe 100644
@@ -4682,6 +4682,20 @@ static int pci_quirk_mf_endpoint_acs(struct pci_dev *dev, u16 acs_flags)
  		PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_DT);
  }
 
+static int pci_quirk_rciep_acs(struct pci_dev *dev, u16 acs_flags)
+{
+	/*
+	 * RCiEP's are required to allow p2p only on translated addresses.
+	 * Refer to Intel VT-d specification Section 3.16 Root-Complex Peer
+	 * to Peer Considerations
+	 */
+	if (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_END)
+		return -ENOTTY;
+
+	return pci_acs_ctrl_enabled(acs_flags,
+		PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF);
+}
+
  static int pci_quirk_brcm_acs(struct pci_dev *dev, u16 acs_flags)
  {
  	/*
@@ -4764,6 +4778,7 @@ static const struct pci_dev_acs_enabled {
  	/* I219 */
  	{ PCI_VENDOR_ID_INTEL, 0x15b7, pci_quirk_mf_endpoint_acs },
  	{ PCI_VENDOR_ID_INTEL, 0x15b8, pci_quirk_mf_endpoint_acs },
+	{ PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_rciep_acs },
  	/* QCOM QDF2xxx root ports */
  	{ PCI_VENDOR_ID_QCOM, 0x0400, pci_quirk_qcom_rp_acs },
  	{ PCI_VENDOR_ID_QCOM, 0x0401, pci_quirk_qcom_rp_acs },
#4 lore raw
In reply to Ashok Raj
Quoted 51 lines · Ashok Raj

All Intel platforms guarantee that all root complex implementations must send transactions up to IOMMU for address translations. Hence for RCiEP devices that are Vendor ID Intel, can claim exception for lack of ACS support.

3.16 Root-Complex Peer to Peer Considerations
When DMA remapping is enabled, peer-to-peer requests through the Root-Complex must be handled
as follows:
• The input address in the request is translated (through first-level, second-level or nested translation) to a host physical address (HPA). The address decoding for peer addresses must be done only on the translated HPA. Hardware implementations are free to further limit peer-to-peer accesses to specific host physical address regions (or to completely disallow peer-forwarding of translated requests).
• Since address translation changes the contents (address field) of the PCI Express Transaction Layer Packet (TLP), for PCI Express peer-to-peer requests with ECRC, the Root-Complex hardware must use the new ECRC (re-computed with the translated address) if it decides to forward the TLP as a peer request.
• Root-ports, and multi-function root-complex integrated endpoints, may support additional peerto-peer control features by supporting PCI Express Access Control Services (ACS) capability. Refer to ACS capability in PCI Express specifications for details.

Since Linux didn't give special treatment to allow this exception, certain RCiEP MFD devices are getting grouped in a single iommu group. This doesn't permit a single device to be assigned to a guest for instance.

In one vendor system: Device 14.x were grouped in a single IOMMU group.

/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.3

After the patch:
/sys/kernel/iommu_groups/5/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:14.2 /sys/kernel/iommu_groups/6/devices/0000:00:14.3 <<< new group

14.0 and 14.2 are integrated devices, but legacy end points. Whereas 14.3 was a PCIe compliant RCiEP.

00:14.3 Network controller: Intel Corporation Device 9df0 (rev 30)
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00

This permits assigning this device to a guest VM.

Fixes f096c061f552 ("iommu: Rework iommu_group_get_for_pci_dev()")
Quoted 10 lines

To: Joerg Roedel <[email protected]>
To: Bjorn Helgaas <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: Lu Baolu <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Darrel Goeddel <[email protected]>
Cc: Mark Scott <[email protected]>,
Cc: Romil Sharma <[email protected]>
Cc: Ashok Raj <[email protected]>

Tentatively applied to pci/virtualization for v5.8, thanks!

The spec says this handling must apply "when DMA remapping is enabled". The patch does not check whether DMA remapping is enabled.

Is there any case where DMA remapping is *not* enabled, and we rely on this patch to tell us whether the device is isolated? It sounds like it may give the wrong answer in such a case?

Can you confirm that I don't need to worry about this?

Quoted 42 lines

---
v2: Moved functionality from iommu to pci quirks - Alex Williamson

 drivers/pci/quirks.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
drivers/pci/quirks.c +15 -0
index 28c9a2409c50..63373ca0a3fe 100644
@@ -4682,6 +4682,20 @@ static int pci_quirk_mf_endpoint_acs(struct pci_dev *dev, u16 acs_flags)
 		PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_DT);
 }
 
+static int pci_quirk_rciep_acs(struct pci_dev *dev, u16 acs_flags)
+{
+	/*
+	 * RCiEP's are required to allow p2p only on translated addresses.
+	 * Refer to Intel VT-d specification Section 3.16 Root-Complex Peer
+	 * to Peer Considerations
+	 */
+	if (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_END)
+		return -ENOTTY;
+
+	return pci_acs_ctrl_enabled(acs_flags,
+		PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF);
+}
+
 static int pci_quirk_brcm_acs(struct pci_dev *dev, u16 acs_flags)
 {
 	/*
@@ -4764,6 +4778,7 @@ static const struct pci_dev_acs_enabled {
 	/* I219 */
 	{ PCI_VENDOR_ID_INTEL, 0x15b7, pci_quirk_mf_endpoint_acs },
 	{ PCI_VENDOR_ID_INTEL, 0x15b8, pci_quirk_mf_endpoint_acs },
+	{ PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_rciep_acs },
 	/* QCOM QDF2xxx root ports */
 	{ PCI_VENDOR_ID_QCOM, 0x0400, pci_quirk_qcom_rp_acs },
 	{ PCI_VENDOR_ID_QCOM, 0x0401, pci_quirk_qcom_rp_acs },
Signature · 1 line
2.7.4
author
#5 lore raw
Quoted 65 lines · Bjorn Helgaas
Quoted 51 lines · Ashok Raj

All Intel platforms guarantee that all root complex implementations must send transactions up to IOMMU for address translations. Hence for RCiEP devices that are Vendor ID Intel, can claim exception for lack of ACS support.

3.16 Root-Complex Peer to Peer Considerations
When DMA remapping is enabled, peer-to-peer requests through the Root-Complex must be handled
as follows:
• The input address in the request is translated (through first-level, second-level or nested translation) to a host physical address (HPA). The address decoding for peer addresses must be done only on the translated HPA. Hardware implementations are free to further limit peer-to-peer accesses to specific host physical address regions (or to completely disallow peer-forwarding of translated requests).
• Since address translation changes the contents (address field) of the PCI Express Transaction Layer Packet (TLP), for PCI Express peer-to-peer requests with ECRC, the Root-Complex hardware must use the new ECRC (re-computed with the translated address) if it decides to forward the TLP as a peer request.
• Root-ports, and multi-function root-complex integrated endpoints, may support additional peerto-peer control features by supporting PCI Express Access Control Services (ACS) capability. Refer to ACS capability in PCI Express specifications for details.

Since Linux didn't give special treatment to allow this exception, certain RCiEP MFD devices are getting grouped in a single iommu group. This doesn't permit a single device to be assigned to a guest for instance.

In one vendor system: Device 14.x were grouped in a single IOMMU group.

/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.3

After the patch:
/sys/kernel/iommu_groups/5/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:14.2 /sys/kernel/iommu_groups/6/devices/0000:00:14.3 <<< new group

14.0 and 14.2 are integrated devices, but legacy end points. Whereas 14.3 was a PCIe compliant RCiEP.

00:14.3 Network controller: Intel Corporation Device 9df0 (rev 30)
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00

This permits assigning this device to a guest VM.

Fixes f096c061f552 ("iommu: Rework iommu_group_get_for_pci_dev()")
Quoted 10 lines

To: Joerg Roedel <[email protected]>
To: Bjorn Helgaas <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: Lu Baolu <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Darrel Goeddel <[email protected]>
Cc: Mark Scott <[email protected]>,
Cc: Romil Sharma <[email protected]>
Cc: Ashok Raj <[email protected]>

Tentatively applied to pci/virtualization for v5.8, thanks!

Quoted 8 lines

The spec says this handling must apply "when DMA remapping is enabled". The patch does not check whether DMA remapping is enabled.

Is there any case where DMA remapping is *not* enabled, and we rely on this patch to tell us whether the device is isolated? It sounds like it may give the wrong answer in such a case?

Can you confirm that I don't need to worry about this?

I think all of this makes sense only when DMA remapping is enabled. Otherwise there is no enforcement for isolation.

Cheers,
Ashok

#6 lore raw
Quoted 69 lines · Ashok
Quoted 68 lines · Bjorn Helgaas
Quoted 51 lines · Ashok Raj

All Intel platforms guarantee that all root complex implementations must send transactions up to IOMMU for address translations. Hence for RCiEP devices that are Vendor ID Intel, can claim exception for lack of ACS support.

3.16 Root-Complex Peer to Peer Considerations
When DMA remapping is enabled, peer-to-peer requests through the Root-Complex must be handled
as follows:
• The input address in the request is translated (through first-level, second-level or nested translation) to a host physical address (HPA). The address decoding for peer addresses must be done only on the translated HPA. Hardware implementations are free to further limit peer-to-peer accesses to specific host physical address regions (or to completely disallow peer-forwarding of translated requests).
• Since address translation changes the contents (address field) of the PCI Express Transaction Layer Packet (TLP), for PCI Express peer-to-peer requests with ECRC, the Root-Complex hardware must use the new ECRC (re-computed with the translated address) if it decides to forward the TLP as a peer request.
• Root-ports, and multi-function root-complex integrated endpoints, may support additional peerto-peer control features by supporting PCI Express Access Control Services (ACS) capability. Refer to ACS capability in PCI Express specifications for details.

Since Linux didn't give special treatment to allow this exception, certain RCiEP MFD devices are getting grouped in a single iommu group. This doesn't permit a single device to be assigned to a guest for instance.

In one vendor system: Device 14.x were grouped in a single IOMMU group.

/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.3

After the patch:
/sys/kernel/iommu_groups/5/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:14.2 /sys/kernel/iommu_groups/6/devices/0000:00:14.3 <<< new group

14.0 and 14.2 are integrated devices, but legacy end points. Whereas 14.3 was a PCIe compliant RCiEP.

00:14.3 Network controller: Intel Corporation Device 9df0 (rev 30)
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00

This permits assigning this device to a guest VM.

Fixes f096c061f552 ("iommu: Rework iommu_group_get_for_pci_dev()")
Quoted 10 lines

To: Joerg Roedel <[email protected]>
To: Bjorn Helgaas <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: Lu Baolu <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Darrel Goeddel <[email protected]>
Cc: Mark Scott <[email protected]>,
Cc: Romil Sharma <[email protected]>
Cc: Ashok Raj <[email protected]>

Tentatively applied to pci/virtualization for v5.8, thanks!

The spec says this handling must apply "when DMA remapping is enabled". The patch does not check whether DMA remapping is enabled.

Quoted 8 lines
Quoted 5 lines

Is there any case where DMA remapping is *not* enabled, and we rely on this patch to tell us whether the device is isolated? It sounds like it may give the wrong answer in such a case?

Can you confirm that I don't need to worry about this?

I think all of this makes sense only when DMA remapping is enabled. Otherwise there is no enforcement for isolation.

Yep, without an IOMMU all devices operate in the same IOVA space and we have no isolation. We only enable ACS when an IOMMU driver requests it and it's only used by IOMMU code to determine IOMMU grouping of devices. Thanks,

Alex

#7 lore raw
Quoted 78 lines · Alex Williamson
Quoted 75 lines · Ashok
Quoted 73 lines · Bjorn Helgaas
Quoted 51 lines · Ashok Raj

All Intel platforms guarantee that all root complex implementations must send transactions up to IOMMU for address translations. Hence for RCiEP devices that are Vendor ID Intel, can claim exception for lack of ACS support.

3.16 Root-Complex Peer to Peer Considerations
When DMA remapping is enabled, peer-to-peer requests through the Root-Complex must be handled
as follows:
• The input address in the request is translated (through first-level, second-level or nested translation) to a host physical address (HPA). The address decoding for peer addresses must be done only on the translated HPA. Hardware implementations are free to further limit peer-to-peer accesses to specific host physical address regions (or to completely disallow peer-forwarding of translated requests).
• Since address translation changes the contents (address field) of the PCI Express Transaction Layer Packet (TLP), for PCI Express peer-to-peer requests with ECRC, the Root-Complex hardware must use the new ECRC (re-computed with the translated address) if it decides to forward the TLP as a peer request.
• Root-ports, and multi-function root-complex integrated endpoints, may support additional peerto-peer control features by supporting PCI Express Access Control Services (ACS) capability. Refer to ACS capability in PCI Express specifications for details.

Since Linux didn't give special treatment to allow this exception, certain RCiEP MFD devices are getting grouped in a single iommu group. This doesn't permit a single device to be assigned to a guest for instance.

In one vendor system: Device 14.x were grouped in a single IOMMU group.

/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.3

After the patch:
/sys/kernel/iommu_groups/5/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:14.2 /sys/kernel/iommu_groups/6/devices/0000:00:14.3 <<< new group

14.0 and 14.2 are integrated devices, but legacy end points. Whereas 14.3 was a PCIe compliant RCiEP.

00:14.3 Network controller: Intel Corporation Device 9df0 (rev 30)
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00

This permits assigning this device to a guest VM.

Fixes f096c061f552 ("iommu: Rework iommu_group_get_for_pci_dev()")
Quoted 10 lines

To: Joerg Roedel <[email protected]>
To: Bjorn Helgaas <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: Lu Baolu <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Darrel Goeddel <[email protected]>
Cc: Mark Scott <[email protected]>,
Cc: Romil Sharma <[email protected]>
Cc: Ashok Raj <[email protected]>

Tentatively applied to pci/virtualization for v5.8, thanks!

The spec says this handling must apply "when DMA remapping is enabled". The patch does not check whether DMA remapping is enabled.

Is there any case where DMA remapping is *not* enabled, and we rely on this patch to tell us whether the device is isolated? It sounds like it may give the wrong answer in such a case?

Can you confirm that I don't need to worry about this?

Quoted 7 lines
Quoted 2 lines

I think all of this makes sense only when DMA remapping is enabled. Otherwise there is no enforcement for isolation.

Yep, without an IOMMU all devices operate in the same IOVA space and we have no isolation. We only enable ACS when an IOMMU driver requests it and it's only used by IOMMU code to determine IOMMU grouping of devices. Thanks,

Thanks, Ashok and Alex. I wish it were more obvious from the code, but I am reassured.

I also added a stable tag to help get this backported.