Allow ATS to be always on for certain ATS-capable devices

Controlled by the IOMMU driver, ATS is usually enabled "on demand" when a given PASID on a device is attached to an I/O page table. This is working even when a device has no translation on its RID (i.e., the RID is IOMMU bypassed).

However, certain PCIe devices require non-PASID ATS on their RID even when the RID is IOMMU bypassed. Call this "ATS always on" in IOMMU term.

For example, CXL spec r4.0 notes in sec 3.2.5.13 Memory Type on CXL.cache:
"To source requests on CXL.cache, devices need to get the Host Physical
Address (HPA) from the Host by means of an ATS request on CXL.io."

In other words, the CXL.cache capability requires ATS; otherwise, it can't access host physical memory.

Introduce a new pci_ats_required() helper for the IOMMU driver to scan a PCI device and shift ATS policies between "on demand" and "always on".

Add the support for CXL.cache devices first. Pre-CXL devices will be added in quirks.c file.

Note that pci_ats_required() validates against pci_ats_supported(), so we ensure that untrusted devices (e.g. external ports) will not be always on. This maintains the existing ATS security policy regarding potential side- channel attacks via ATS.

Cc [email protected]

Suggested-by Vikram Sethi <[email protected]>

Suggested-by Jason Gunthorpe <[email protected]>

Reviewed-by Jonathan Cameron <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

Reviewed-by Kevin Tian <[email protected]>

Tested-by Nirmoy Das <[email protected]>

Acked-by Nirmoy Das <[email protected]>

Reviewed-by Dave Jiang <[email protected]>

Signed-off-by Nicolin Chen <[email protected]>

---
 include/linux/pci-ats.h       |  3 +++
 include/uapi/linux/pci_regs.h |  1 +
 drivers/pci/ats.c             | 46 +++++++++++++++++++++++++++++++++++
 3 files changed, 50 insertions(+)

include/linux/pci-ats.h +3 -0

index 75c6c86cf09dc..f3723b6861294 100644
@@ -12,6 +12,7 @@ int pci_prepare_ats(struct pci_dev *dev, int ps);
 void pci_disable_ats(struct pci_dev *dev);
 int pci_ats_queue_depth(struct pci_dev *dev);
 int pci_ats_page_aligned(struct pci_dev *dev);
+bool pci_ats_required(struct pci_dev *dev);
 #else /* CONFIG_PCI_ATS */
 static inline bool pci_ats_supported(struct pci_dev *d)
 { return false; }

@@ -24,6 +25,8 @@ static inline int pci_ats_queue_depth(struct pci_dev *d)
 { return -ENODEV; }
 static inline int pci_ats_page_aligned(struct pci_dev *dev)
 { return 0; }
+static inline bool pci_ats_required(struct pci_dev *dev)
+{ return false; }
 #endif /* CONFIG_PCI_ATS */
 
 #ifdef CONFIG_PCI_PRI

include/uapi/linux/pci_regs.h +1 -0

index 14f634ab9350d..6ac45be1008b8 100644
@@ -1349,6 +1349,7 @@
 /* CXL r4.0, 8.1.3: PCIe DVSEC for CXL Device */
 #define PCI_DVSEC_CXL_DEVICE				0
 #define  PCI_DVSEC_CXL_CAP				0xA
+#define   PCI_DVSEC_CXL_CACHE_CAPABLE			_BITUL(0)
 #define   PCI_DVSEC_CXL_MEM_CAPABLE			_BITUL(2)
 #define   PCI_DVSEC_CXL_HDM_COUNT			__GENMASK(5, 4)
 #define  PCI_DVSEC_CXL_CTRL				0xC

drivers/pci/ats.c +27 -0

index ec6c8dbdc5e9c..ebdf761843867 100644
@@ -205,6 +205,52 @@ int pci_ats_page_aligned(struct pci_dev *pdev)
 	return 0;
 }
 
+/*
+ * CXL r4.0, sec 3.2.5.13 Memory Type on CXL.cache notes: to source requests on
+ * CXL.cache, devices need to get the Host Physical Address (HPA) from the Host
+ * by means of an ATS request on CXL.io.
+ *
+ * In other words, CXL.cache devices cannot access host physical memory without
+ * ATS.
+ *
+ * Check Cache_Capable instead of Cache_Enable because CXL.cache may be enabled
+ * after the caller uses this to make its ATS decision.
+ */
+static bool pci_cxl_ats_required(struct pci_dev *pdev)
+{
+	int offset;
+	u16 cap;
+
+	offset = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					   PCI_DVSEC_CXL_DEVICE);
+	if (!offset)
+		return false;
+
+	if (pci_read_config_word(pdev, offset + PCI_DVSEC_CXL_CAP, &cap))
+		return false;
+
+	return cap & PCI_DVSEC_CXL_CACHE_CAPABLE;
+}
+

drivers/pci/ats.c +8 -0

+	/* A VF inherits its PF's requirement for ATS function */
+	if (pdev->is_virtfn)
+		pdev = pci_physfn(pdev);
+
+	return pci_cxl_ats_required(pdev);
+}
+EXPORT_SYMBOL_GPL(pci_ats_required);
+
 #ifdef CONFIG_PCI_PRI
 void pci_pri_init(struct pci_dev *pdev)
 {

Signature · 1 line

2.43.0

drivers/pci/ats.c int pci_ats_page_aligned(struct pci_dev *pdev) · line 244

Quoted 4 lines · Nicolin Chen

drivers/pci/ats.c +4 -0

+bool pci_ats_required(struct pci_dev *pdev)
+{
+	if (pci_ats_disabled() || !pci_ats_supported(pdev))
+		return false;

I still have the question about whether it's necessary to test pci_ats_disabled() here. I think pci_ats_supported() should return false if pci_ats_disabled() returns true.

Some NVIDIA GPU/NIC devices, though they don't implement CXL config space, have many CXL-like properties. Call this kind "pre-CXL".

Similar to CXL.cache capability, these pre-CXL devices also require the ATS function even when their RIDs are IOMMU bypassed, i.e. keep ATS "always on" v.s. "on demand" when a non-zero PASID line gets enabled in SVA use cases.

Introduce pci_dev_specific_ats_required() quirk function to scan a list of IDs for these devices. Then, include it in pci_ats_required().

Suggested-by Jason Gunthorpe <[email protected]>

Reviewed-by Nirmoy Das <[email protected]>

Tested-by Nirmoy Das <[email protected]>

Reviewed-by Jonathan Cameron <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

Reviewed-by Kevin Tian <[email protected]>

Reviewed-by Dave Jiang <[email protected]>

Signed-off-by Nicolin Chen <[email protected]>

---
 drivers/pci/pci.h    |  9 +++++++++
 drivers/pci/ats.c    |  3 ++-
 drivers/pci/quirks.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 53 insertions(+), 1 deletion(-)

drivers/pci/pci.h +9 -0

index 4a14f88e543a2..e8ad27abb1cfe 100644
@@ -1155,6 +1155,15 @@ static inline int pci_dev_specific_reset(struct pci_dev *dev, bool probe)
 }
 #endif
 
+#if defined(CONFIG_PCI_QUIRKS) && defined(CONFIG_PCI_ATS)
+bool pci_dev_specific_ats_required(struct pci_dev *dev);
+#else
+static inline bool pci_dev_specific_ats_required(struct pci_dev *dev)
+{
+	return false;
+}
+#endif
+
 #if defined(CONFIG_PCI_QUIRKS) && defined(CONFIG_ARM64)
 int acpi_get_rc_resources(struct device *dev, const char *hid, u16 segment,
 			  struct resource *res);

drivers/pci/ats.c +2 -1

index ebdf761843867..3a04d5b04c883 100644
@@ -247,7 +247,8 @@ bool pci_ats_required(struct pci_dev *pdev)
 	if (pdev->is_virtfn)
 		pdev = pci_physfn(pdev);
 
-	return pci_cxl_ats_required(pdev);
+	return pci_cxl_ats_required(pdev) ||
+	       pci_dev_specific_ats_required(pdev);
 }
 EXPORT_SYMBOL_GPL(pci_ats_required);
 

drivers/pci/quirks.c +42 -0

index caaed1a01dc02..c0242f3e9f063 100644
@@ -5715,6 +5715,48 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x1457, quirk_intel_e2000_no_ats);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x1459, quirk_intel_e2000_no_ats);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x145a, quirk_intel_e2000_no_ats);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x145c, quirk_intel_e2000_no_ats);
+
+static bool quirk_nvidia_gpu_ats_required(struct pci_dev *pdev)
+{
+	switch (pdev->device) {
+	case 0x2e00 ... 0x2e3f: /* GB20B */
+		return true;
+	}
+	return false;
+}
+
+static const struct pci_dev_ats_required {
+	u16 vendor;
+	u16 device;
+	bool (*ats_required)(struct pci_dev *dev);
+} pci_dev_ats_required[] = {
+	/* NVIDIA GPUs */
+	{ PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID, quirk_nvidia_gpu_ats_required },
+	/* NVIDIA CX10 Family NVlink-C2C */
+	{ PCI_VENDOR_ID_MELLANOX, 0x2101, NULL },
+	{ 0 }
+};
+
+/*
+ * Some NVIDIA devices do not implement CXL config space, but present as PCIe
+ * devices that can issue CXL-like cache operations like CXL.cache. Thus, they
+ * require ATS to obtain host physical addresses, like pci_cxl_ats_required().
+ */
+bool pci_dev_specific_ats_required(struct pci_dev *pdev)
+{
+	const struct pci_dev_ats_required *i;
+
+	for (i = pci_dev_ats_required; i->vendor; i++) {
+		if (i->vendor != pdev->vendor)
+			continue;
+		if (i->ats_required && i->ats_required(pdev))
+			return true;
+		if (!i->ats_required && i->device == pdev->device)
+			return true;
+	}
+
+	return false;
+}
 #endif /* CONFIG_PCI_ATS */
 
 /* Freescale PCIe doesn't support MSI in RC mode */

Signature · 1 line

2.43.0

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2 -0

@@ -3875,6 +3895,8 @@ static void arm_smmu_attach_dev_ste(struct iommu_domain *domain,
 		.old_domain = old_domain,
 		.ssid = IOMMU_NO_PASID,
 	};
+	bool ats_always_on = master->ats_always_on &&
+			     s1dss != STRTAB_STE_1_S1DSS_TERMINATE;

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +25 -0

+static int arm_smmu_master_prepare_ats(struct arm_smmu_master *master)
+{
+	bool s1p = master->smmu->features & ARM_SMMU_FEAT_TRANS_S1;
+	unsigned int stu = __ffs(master->smmu->pgsize_bitmap);
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!arm_smmu_ats_supported(master))
+		return 0;
+
+	pdev = to_pci_dev(master->dev);
+
+	if (!pci_ats_required(pdev))
+		goto out_prepare;
+
+	/*
+	 * S1DSS is required for ATS to be always on for identity domain cases.
+	 * However, the S1DSS field is ignored if !IDR0_S1P or !IDR1_SSIDSIZE.
+	 */
+	if (!s1p || !master->smmu->ssid_bits) {
+		dev_info_once(master->dev,
+			      "SMMU doesn't support ATS to be always on\n");
+		goto out_prepare;
+	}
+

@ -3870,13 +3870,15 @@ static int arm_smmu_blocking_set_dev_pasid(struct iommu_domain *new_domain,
* When the last user of the CD table goes away downgrade the STE back
* to a non-cd_table one, by re-attaching its sid_domain.
*/

(quoted diff) +1 -2

-       if (!master->ats_always_on &&
-           !arm_smmu_ssids_in_use(&master->cd_table)) {
+       if (!arm_smmu_ssids_in_use(&master->cd_table)) {
                struct iommu_domain *sid_domain =
                        iommu_driver_get_domain_for_dev(master->dev);

Quoted 14 lines · Nicolin Chen

@ -3870,13 +3870,15 @@ static int arm_smmu_blocking_set_dev_pasid(struct iommu_domain *new_domain,
* When the last user of the CD table goes away downgrade the STE back
* to a non-cd_table one, by re-attaching its sid_domain.
*/

(quoted diff) +5 -2

-       if (!master->ats_always_on &&
-           !arm_smmu_ssids_in_use(&master->cd_table)) {
+       if (!arm_smmu_ssids_in_use(&master->cd_table)) {
                struct iommu_domain *sid_domain =
                        iommu_driver_get_domain_for_dev(master->dev);
+               bool ats_always_on = master->ats_always_on &&
+                                    sid_domain->type != IOMMU_DOMAIN_BLOCKED;
+               bool downgrade = sid_domain->type == IOMMU_DOMAIN_IDENTITY ||
+                                sid_domain->type == IOMMU_DOMAIN_BLOCKED;

Review

Quoted 5 lines

(quoted diff) +1 -2

-               if (sid_domain->type == IOMMU_DOMAIN_IDENTITY ||
-                   sid_domain->type == IOMMU_DOMAIN_BLOCKED)
+               if (!ats_always_on && downgrade)
                        sid_domain->ops->attach_dev(sid_domain, dev,
                                                    sid_domain);

Only identity should remain with the CD S1DSS STE, BLOCKED should attach the normal blocking domain still