Skip to contentSkip to conversation

Consolidate domain cache invalidation

PATCH Superseded Lu Baolu 3 participants 44 messages Mar 25, 2024 – Apr 12, 2024

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Lu Baolu

author maintainer
#0 lore raw

cover Consolidate domain cache invalidation

The IOMMU hardware cache needs to be invalidated whenever the mappings in the domain are changed. Currently, domain cache invalidation is scattered across different places, causing several issues:

- IOMMU IOTLB Invalidation: This is done by iterating through the domain IDs of each domain using the following code:

        xa_for_each(&dmar_domain->iommu_array, i, info)
                iommu_flush_iotlb_psi(info->iommu, dmar_domain,
                                      start_pfn, nrpages,
                                      list_empty(&gather->freelist), 0);

This code could theoretically cause a use-after-free problem because there's no lock to protect the "info" pointer within the loop.

- Inconsistent Invalidation Methods: Different domain types implement their own cache invalidation methods, making the code difficult to maintain. For example, the DMA domain, SVA domain, and nested domain have similar cache invalidation code scattered across different files.

- SVA Domain Inconsistency: The SVA domain implementation uses a completely different data structure to track attached devices compared to other domains. This creates unnecessary differences and, even worse, leads to duplicate IOTLB invalidation when an SVA domain is attached to devices belonging to different IOMMU domains.

- Nested Domain Dependency: The special overlap between a nested domain and its parent domain requires a dedicated parent_domain_flush() helper function to be called everywhere the parent domain's mapping changes.

- Limited Debugging Support: There are currently no debugging aids available for domain cache invalidation.

By consolidating domain cache invalidation into a common location, we can address the issues mentioned above and improve the code's maintainability and debuggability.

Jason Gunthorpe (1):
iommu: Add ops->domain_alloc_sva()

Lu Baolu (11):
iommu/vt-d: Add cache tag assignment interface iommu/vt-d: Add cache tag invalidation helpers iommu/vt-d: Add trace events for cache tag interface iommu/vt-d: Use cache_tag_flush_all() in flush_iotlb_all iommu/vt-d: Use cache_tag_flush_range() in tlb_sync iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi() iommu/vt-d: Use cache_tag_flush_range() in cache_invalidate_user iommu/vt-d: Use cache helpers in arch_invalidate_secondary_tlbs iommu/vt-d: Retire intel_svm_dev
iommu/vt-d: Retire struct intel_svm

 include/linux/iommu.h        |   3 +
 drivers/iommu/intel/iommu.h  |  66 +++---
 drivers/iommu/intel/trace.h  |  97 +++++++++
 drivers/iommu/intel/cache.c  | 389 +++++++++++++++++++++++++++++++++++
 drivers/iommu/intel/iommu.c  | 294 ++++----------------------
 drivers/iommu/intel/nested.c |  71 ++-----
 drivers/iommu/intel/svm.c    | 279 ++++++-------------------
 drivers/iommu/iommu-sva.c    |  16 +-
 drivers/iommu/intel/Makefile |   2 +-
 9 files changed, 660 insertions(+), 557 deletions(-)
 create mode 100644 drivers/iommu/intel/cache.c
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH 1/12 iommu/vt-d: Add cache tag assignment interface

Caching tag is a combination of tags used by the hardware to cache various translations. Whenever a mapping in a domain is changed, the IOMMU driver should invalidate the caches with the caching tags. The VT-d specification describes caching tags in section 6.2.1, Tagging of Cached Translations.

Add interface to assign caching tags to an IOMMU domain when attached to a RID or PASID, and unassign caching tags when a domain is detached from a RID or PASID. All caching tags are listed in the per-domain tag list and are protected by a dedicated lock.

In addition to the basic IOTLB and devTLB caching tag types, PARENT_IOTLB and PARENT_DEVTLB tag types are also introduced. These tags are used for caches that store translations for DMA accesses through a nested user domain. They are affected by changes to mappings in the parent domain.

---
 drivers/iommu/intel/iommu.h  |  25 +++++
 drivers/iommu/intel/cache.c  | 192 +++++++++++++++++++++++++++++++++++
 drivers/iommu/intel/iommu.c  |  31 +++++-
 drivers/iommu/intel/nested.c |  21 +++-
 drivers/iommu/intel/svm.c    |  12 ++-
 drivers/iommu/intel/Makefile |   2 +-
 6 files changed, 274 insertions(+), 9 deletions(-)
 create mode 100644 drivers/iommu/intel/cache.c
drivers/iommu/intel/iommu.h +25 -0
index 404d2476a877..e3723b7a0b31 100644
@@ -607,6 +607,9 @@ struct dmar_domain {
 	struct list_head devices;	/* all devices' list */
 	struct list_head dev_pasids;	/* all attached pasids */
 
+	spinlock_t cache_lock;		/* Protect the cache tag list */
+	struct list_head cache_tags;	/* Cache tag list */
+
 	int		iommu_superpage;/* Level of superpages supported:
 					   0 == 4KiB (no superpages), 1 == 2MiB,
 					   2 == 1GiB, 3 == 512GiB, 4 == 1TiB */
@@ -1092,6 +1095,28 @@ struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent,
 					       const struct iommu_user_data *user_data);
 struct device *device_rbtree_find(struct intel_iommu *iommu, u16 rid);
 
+enum cache_tag_type {
+	CACHE_TAG_TYPE_IOTLB,
+	CACHE_TAG_TYPE_DEVTLB,
+	CACHE_TAG_TYPE_PARENT_IOTLB,
+	CACHE_TAG_TYPE_PARENT_DEVTLB,
+};
+
+struct cache_tag {
+	struct list_head node;
+	enum cache_tag_type type;
+	struct intel_iommu *iommu;
+	struct device *dev;
+	u16 domain_id;
+	ioasid_t pasid;
+	int users;
+};
+
+int cache_tag_assign_domain(struct dmar_domain *domain, u16 did,
+			    struct device *dev, ioasid_t pasid);
+void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did,
+			       struct device *dev, ioasid_t pasid);
+
 #ifdef CONFIG_INTEL_IOMMU_SVM
 void intel_svm_check(struct intel_iommu *iommu);
 int intel_svm_enable_prq(struct intel_iommu *iommu);
drivers/iommu/intel/cache.c new file +192 -0

192 lines not on this page · raw message

drivers/iommu/intel/iommu.c +29 -2
index 50eb9aed47cc..b4efbdedccce 100644
@@ -1750,7 +1750,9 @@ static struct dmar_domain *alloc_domain(unsigned int type)
 	domain->has_iotlb_device = false;
 	INIT_LIST_HEAD(&domain->devices);
 	INIT_LIST_HEAD(&domain->dev_pasids);
+	INIT_LIST_HEAD(&domain->cache_tags);
 	spin_lock_init(&domain->lock);
+	spin_lock_init(&domain->cache_lock);
 	xa_init(&domain->iommu_array);
 
 	return domain;
@@ -2322,11 +2324,20 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct intel_iommu *iommu = info->iommu;
 	unsigned long flags;
+	u16 did;
 	int ret;
 
 	ret = domain_attach_iommu(domain, iommu);
 	if (ret)
 		return ret;
+
+	did = domain_id_iommu(domain, iommu);
+	ret = cache_tag_assign_domain(domain, did, dev, IOMMU_NO_PASID);
+	if (ret) {
+		domain_detach_iommu(domain, iommu);
+		return ret;
+	}
+
 	info->domain = domain;
 	spin_lock_irqsave(&domain->lock, flags);
 	list_add(&info->link, &domain->devices);
@@ -3798,6 +3809,7 @@ void device_block_translation(struct device *dev)
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct intel_iommu *iommu = info->iommu;
 	unsigned long flags;
+	u16 did;
 
 	iommu_disable_pci_caps(info);
 	if (!dev_is_real_dma_subdevice(dev)) {
@@ -3815,6 +3827,8 @@ void device_block_translation(struct device *dev)
 	list_del(&info->link);
 	spin_unlock_irqrestore(&info->domain->lock, flags);
 
+	did = domain_id_iommu(info->domain, iommu);
+	cache_tag_unassign_domain(info->domain, did, dev, IOMMU_NO_PASID);
 	domain_detach_iommu(info->domain, iommu);
 	info->domain = NULL;
 }
@@ -4595,10 +4609,12 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
 	struct dmar_domain *dmar_domain;
 	struct iommu_domain *domain;
 	unsigned long flags;
+	u16 did;
 
 	domain = iommu_get_domain_for_dev_pasid(dev, pasid, 0);
 	if (WARN_ON_ONCE(!domain))
 		goto out_tear_down;
+	dmar_domain = to_dmar_domain(domain);
 
 	/*
 	 * The SVA implementation needs to handle its own stuffs like the mm
@@ -4607,10 +4623,11 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
 	 */
 	if (domain->type == IOMMU_DOMAIN_SVA) {
 		intel_svm_remove_dev_pasid(dev, pasid);
+		cache_tag_unassign_domain(dmar_domain,
+					  FLPT_DEFAULT_DID, dev, pasid);
 		goto out_tear_down;
 	}
 
-	dmar_domain = to_dmar_domain(domain);
 	spin_lock_irqsave(&dmar_domain->lock, flags);
 	list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) {
 		if (curr->dev == dev && curr->pasid == pasid) {
@@ -4622,6 +4639,8 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
 	WARN_ON_ONCE(!dev_pasid);
 	spin_unlock_irqrestore(&dmar_domain->lock, flags);
 
+	did = domain_id_iommu(dmar_domain, iommu);
+	cache_tag_unassign_domain(dmar_domain, did, dev, pasid);
 	domain_detach_iommu(dmar_domain, iommu);
 	intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
 	kfree(dev_pasid);
@@ -4638,6 +4657,7 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
 	struct intel_iommu *iommu = info->iommu;
 	struct dev_pasid_info *dev_pasid;
 	unsigned long flags;
+	u16 did;
 	int ret;
 
 	if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev))
@@ -4661,6 +4681,11 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
 	if (ret)
 		goto out_free;
 
+	did = domain_id_iommu(dmar_domain, iommu);
+	ret = cache_tag_assign_domain(dmar_domain, did, dev, pasid);
+	if (ret)
+		goto out_detach_iommu;
+
 	if (domain_type_is_si(dmar_domain))
 		ret = intel_pasid_setup_pass_through(iommu, dev, pasid);
 	else if (dmar_domain->use_first_level)
@@ -4670,7 +4695,7 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
 		ret = intel_pasid_setup_second_level(iommu, dmar_domain,
 						     dev, pasid);
 	if (ret)
-		goto out_detach_iommu;
+		goto out_unassign_tag;
 
 	dev_pasid->dev = dev;
 	dev_pasid->pasid = pasid;
@@ -4682,6 +4707,8 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
 		intel_iommu_debugfs_create_dev_pasid(dev_pasid);
 
 	return 0;
+out_unassign_tag:
+	cache_tag_unassign_domain(dmar_domain, did, dev, pasid);
 out_detach_iommu:
 	domain_detach_iommu(dmar_domain, iommu);
 out_free:
drivers/iommu/intel/nested.c +16 -5
index a7d68f3d518a..85c744099558 100644
@@ -26,6 +26,7 @@ static int intel_nested_attach_dev(struct iommu_domain *domain,
 	struct intel_iommu *iommu = info->iommu;
 	unsigned long flags;
 	int ret = 0;
+	u16 did;
 
 	if (info->domain)
 		device_block_translation(dev);
@@ -52,13 +53,15 @@ static int intel_nested_attach_dev(struct iommu_domain *domain,
 		return ret;
 	}
 
+	did = domain_id_iommu(dmar_domain, iommu);
+	ret = cache_tag_assign_domain(dmar_domain, did, dev, IOMMU_NO_PASID);
+	if (ret)
+		goto detach_iommu;
+
 	ret = intel_pasid_setup_nested(iommu, dev,
 				       IOMMU_NO_PASID, dmar_domain);
-	if (ret) {
-		domain_detach_iommu(dmar_domain, iommu);
-		dev_err_ratelimited(dev, "Failed to setup pasid entry\n");
-		return ret;
-	}
+	if (ret)
+		goto unassign_tag;
 
 	info->domain = dmar_domain;
 	spin_lock_irqsave(&dmar_domain->lock, flags);
@@ -68,6 +71,12 @@ static int intel_nested_attach_dev(struct iommu_domain *domain,
 	domain_update_iotlb(dmar_domain);
 
 	return 0;
+unassign_tag:
+	cache_tag_unassign_domain(dmar_domain, did, dev, IOMMU_NO_PASID);
+detach_iommu:
+	domain_detach_iommu(dmar_domain, iommu);
+
+	return ret;
 }
 
 static void intel_nested_domain_free(struct iommu_domain *domain)
@@ -206,7 +215,9 @@ struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent,
 	domain->domain.type = IOMMU_DOMAIN_NESTED;
 	INIT_LIST_HEAD(&domain->devices);
 	INIT_LIST_HEAD(&domain->dev_pasids);
+	INIT_LIST_HEAD(&domain->cache_tags);
 	spin_lock_init(&domain->lock);
+	spin_lock_init(&domain->cache_lock);
 	xa_init(&domain->iommu_array);
 
 	spin_lock(&s2_domain->s1_lock);
drivers/iommu/intel/svm.c +11 -1
index c1bed89b1026..d706226e84ee 100644
@@ -366,17 +366,25 @@ static int intel_svm_set_dev_pasid(struct iommu_domain *domain,
 			sdev->qdep = 0;
 	}
 
+	ret = cache_tag_assign_domain(to_dmar_domain(domain),
+				      FLPT_DEFAULT_DID, dev, pasid);
+	if (ret)
+		goto free_sdev;
+
 	/* Setup the pasid table: */
 	sflags = cpu_feature_enabled(X86_FEATURE_LA57) ? PASID_FLAG_FL5LP : 0;
 	ret = intel_pasid_setup_first_level(iommu, dev, mm->pgd, pasid,
 					    FLPT_DEFAULT_DID, sflags);
 	if (ret)
-		goto free_sdev;
+		goto unassign_tag;
 
 	list_add_rcu(&sdev->list, &svm->devs);
 
 	return 0;
 
+unassign_tag:
+	cache_tag_unassign_domain(to_dmar_domain(domain),
+				  FLPT_DEFAULT_DID, dev, pasid);
 free_sdev:
 	kfree(sdev);
 free_svm:
@@ -795,6 +803,8 @@ struct iommu_domain *intel_svm_domain_alloc(void)
 	if (!domain)
 		return NULL;
 	domain->domain.ops = &intel_svm_domain_ops;
+	INIT_LIST_HEAD(&domain->cache_tags);
+	spin_lock_init(&domain->cache_lock);
 
 	return &domain->domain;
 }
drivers/iommu/intel/Makefile +1 -1
index 5402b699a122..c8beb0281559 100644
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_DMAR_TABLE) += dmar.o
-obj-$(CONFIG_INTEL_IOMMU) += iommu.o pasid.o nested.o
+obj-$(CONFIG_INTEL_IOMMU) += iommu.o pasid.o nested.o cache.o
 obj-$(CONFIG_DMAR_TABLE) += trace.o cap_audit.o
 obj-$(CONFIG_DMAR_PERF) += perf.o
 obj-$(CONFIG_INTEL_IOMMU_DEBUGFS) += debugfs.o
Signature · 1 line
2.34.1
author maintainer
#13 lore raw

PATCH 2/12 iommu/vt-d: Add cache tag invalidation helpers

In reply to Lu Baolu

Add several helpers to invalidate the caches after mappings in the affected domain are changed.

- cache_tag_flush_range() invalidates a range of caches after mappings within this range are changed. It uses the page-selective cache invalidation methods.

- cache_tag_flush_all() invalidates all caches tagged by a domain ID. It uses the domain-selective cache invalidation methods.

- cache_tag_flush_cm_range() invalidates a range of caches if IOMMU is working in the caching mode and second-only translation is used for the affected domain. It is called when non-present to present mappings are created.

---
 drivers/iommu/intel/iommu.h |   7 ++
 drivers/iommu/intel/cache.c | 189 ++++++++++++++++++++++++++++++++++++
 drivers/iommu/intel/iommu.c |   5 -
 3 files changed, 196 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.h +7 -0
index e3723b7a0b31..d05fa0122d65 100644
@@ -35,6 +35,8 @@
 #define VTD_PAGE_MASK		(((u64)-1) << VTD_PAGE_SHIFT)
 #define VTD_PAGE_ALIGN(addr)	(((addr) + VTD_PAGE_SIZE - 1) & VTD_PAGE_MASK)
 
+#define IOVA_PFN(addr)		((addr) >> PAGE_SHIFT)
+
 #define VTD_STRIDE_SHIFT        (9)
 #define VTD_STRIDE_MASK         (((u64)-1) << VTD_STRIDE_SHIFT)
 
@@ -1116,6 +1118,11 @@ int cache_tag_assign_domain(struct dmar_domain *domain, u16 did,
 			    struct device *dev, ioasid_t pasid);
 void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did,
 			       struct device *dev, ioasid_t pasid);
+void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start,
+			   unsigned long end, int ih);
+void cache_tag_flush_all(struct dmar_domain *domain);
+void cache_tag_flush_cm_range(struct dmar_domain *domain, unsigned long start,
+			      unsigned long end);
 
 #ifdef CONFIG_INTEL_IOMMU_SVM
 void intel_svm_check(struct intel_iommu *iommu);
drivers/iommu/intel/cache.c +189 -0

198 lines not on this page · raw message

drivers/iommu/intel/iommu.c +0 -5
index b4efbdedccce..93e4422c9b10 100644
@@ -54,11 +54,6 @@
 				__DOMAIN_MAX_PFN(gaw), (unsigned long)-1))
 #define DOMAIN_MAX_ADDR(gaw)	(((uint64_t)__DOMAIN_MAX_PFN(gaw)) << VTD_PAGE_SHIFT)
 
-/* IO virtual address start page frame number */
-#define IOVA_START_PFN		(1)
-
-#define IOVA_PFN(addr)		((addr) >> PAGE_SHIFT)
-
 static void __init check_tylersburg_isoch(void);
 static int rwbf_quirk;
 
Signature · 1 line
2.34.1
author maintainer
#19 lore raw

PATCH 3/12 iommu/vt-d: Add trace events for cache tag interface

In reply to Lu Baolu

Add trace events for cache tag assign/unassign/flush operations and trace the events in the interfaces. These trace events will improve debugging capabilities by providing detailed information about cache tag activity. A sample of the traced messages looks like below [messages have been stripped and wrapped to make the line short].

cache_tag_assign: dmar9/0000:00:01.0 type iotlb did 1 pasid 9 ref 1
cache_tag_assign: dmar9/0000:00:01.0 type devtlb did 1 pasid 9 ref 1
cache_tag_flush_all: dmar6/0000:8a:00.0 type iotlb did 7 pasid 0 ref 1
cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9

	[0xeab00000-0xeab1afff] addr 0xeab00000 pages 0x20 mask 0x5
 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9
	[0xeab20000-0xeab31fff] addr 0xeab20000 pages 0x20 mask 0x5
 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9
	[0xeaa40000-0xeaa51fff] addr 0xeaa40000 pages 0x20 mask 0x5
 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9
	[0x98de0000-0x98de4fff] addr 0x98de0000 pages 0x8 mask 0x3
 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9
	[0xe9828000-0xe9828fff] addr 0xe9828000 pages 0x1 mask 0x0

cache_tag_unassign: dmar9/0000:00:01.0 type iotlb did 1 pasid 9 ref 1
cache_tag_unassign: dmar9/0000:00:01.0 type devtlb did 1 pasid 9 ref 1

---
 drivers/iommu/intel/trace.h | 97 +++++++++++++++++++++++++++++++++++++
 drivers/iommu/intel/cache.c |  8 +++
 2 files changed, 105 insertions(+)
drivers/iommu/intel/trace.h +97 -0
index 93d96f93a89b..b8c2ebace8af 100644
@@ -89,6 +89,103 @@ TRACE_EVENT(prq_report,
 				      __entry->dw1, __entry->dw2, __entry->dw3)
 	)
 );
+
+DECLARE_EVENT_CLASS(cache_tag_log,
+	TP_PROTO(struct cache_tag *tag),
+	TP_ARGS(tag),
+	TP_STRUCT__entry(
+		__string(iommu, tag->iommu->name)
+		__string(dev, dev_name(tag->dev))
+		__field(u16, type)
+		__field(u16, domain_id)
+		__field(u32, pasid)
+		__field(u32, users)
+	),
+	TP_fast_assign(
+		__assign_str(iommu, tag->iommu->name);
+		__assign_str(dev, dev_name(tag->dev));
+		__entry->type = tag->type;
+		__entry->domain_id = tag->domain_id;
+		__entry->pasid = tag->pasid;
+		__entry->users = tag->users;
+	),
+	TP_printk("%s/%s type %s did %d pasid %d ref %d",
+		  __get_str(iommu), __get_str(dev),
+		  __print_symbolic(__entry->type,
+			{ CACHE_TAG_TYPE_IOTLB,		"iotlb" },
+			{ CACHE_TAG_TYPE_DEVTLB,	"devtlb" },
+			{ CACHE_TAG_TYPE_PARENT_IOTLB,	"parent_iotlb" },
+			{ CACHE_TAG_TYPE_PARENT_DEVTLB,	"parent_devtlb" }),
+		__entry->domain_id, __entry->pasid, __entry->users
+	)
+);
+
+DEFINE_EVENT(cache_tag_log, cache_tag_assign,
+	TP_PROTO(struct cache_tag *tag),
+	TP_ARGS(tag)
+);
+
+DEFINE_EVENT(cache_tag_log, cache_tag_unassign,
+	TP_PROTO(struct cache_tag *tag),
+	TP_ARGS(tag)
+);
+
+DEFINE_EVENT(cache_tag_log, cache_tag_flush_all,
+	TP_PROTO(struct cache_tag *tag),
+	TP_ARGS(tag)
+);
+
+DECLARE_EVENT_CLASS(cache_tag_flush,
+	TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end,
+		 unsigned long addr, unsigned long pages, unsigned long mask),
+	TP_ARGS(tag, start, end, addr, pages, mask),
+	TP_STRUCT__entry(
+		__string(iommu, tag->iommu->name)
+		__string(dev, dev_name(tag->dev))
+		__field(u16, type)
+		__field(u16, domain_id)
+		__field(u32, pasid)
+		__field(unsigned long, start)
+		__field(unsigned long, end)
+		__field(unsigned long, addr)
+		__field(unsigned long, pages)
+		__field(unsigned long, mask)
+	),
+	TP_fast_assign(
+		__assign_str(iommu, tag->iommu->name);
+		__assign_str(dev, dev_name(tag->dev));
+		__entry->type = tag->type;
+		__entry->domain_id = tag->domain_id;
+		__entry->pasid = tag->pasid;
+		__entry->start = start;
+		__entry->end = end;
+		__entry->addr = addr;
+		__entry->pages = pages;
+		__entry->mask = mask;
+	),
+	TP_printk("%s %s[%d] type %s did %d [0x%lx-0x%lx] addr 0x%lx pages 0x%lx mask 0x%lx",
+		  __get_str(iommu), __get_str(dev), __entry->pasid,
+		  __print_symbolic(__entry->type,
+			{ CACHE_TAG_TYPE_IOTLB,		"iotlb" },
+			{ CACHE_TAG_TYPE_DEVTLB,	"devtlb" },
+			{ CACHE_TAG_TYPE_PARENT_IOTLB,	"parent_iotlb" },
+			{ CACHE_TAG_TYPE_PARENT_DEVTLB,	"parent_devtlb" }),
+		__entry->domain_id, __entry->start, __entry->end,
+		__entry->addr, __entry->pages, __entry->mask
+	)
+);
+
+DEFINE_EVENT(cache_tag_flush, cache_tag_flush_range,
+	TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end,
+		 unsigned long addr, unsigned long pages, unsigned long mask),
+	TP_ARGS(tag, start, end, addr, pages, mask)
+);
+
+DEFINE_EVENT(cache_tag_flush, cache_tag_flush_cm_range,
+	TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end,
+		 unsigned long addr, unsigned long pages, unsigned long mask),
+	TP_ARGS(tag, start, end, addr, pages, mask)
+);
 #endif /* _TRACE_INTEL_IOMMU_H */
 
 /* This part must be outside protection */
drivers/iommu/intel/cache.c +8 -0
index 4c245d39faf2..b35ef2ee2aca 100644
@@ -17,6 +17,7 @@
 
 #include "iommu.h"
 #include "pasid.h"
+#include "trace.h"
 
 /* Checks if an existing cache tag can be reused for a new association. */
 static bool cache_tag_reusable(struct cache_tag *tag, u16 domain_id,
@@ -65,11 +66,13 @@ static int cache_tag_assign(struct dmar_domain *domain, u16 did,
 			temp->users++;
 			spin_unlock_irqrestore(&domain->cache_lock, flags);
 			kfree(tag);
+			trace_cache_tag_assign(temp);
 			return 0;
 		}
 	}
 	list_add_tail(&tag->node, &domain->cache_tags);
 	spin_unlock_irqrestore(&domain->cache_lock, flags);
+	trace_cache_tag_assign(tag);
 
 	return 0;
 }
@@ -87,6 +90,7 @@ static void cache_tag_unassign(struct dmar_domain *domain, u16 did,
 	spin_lock_irqsave(&domain->cache_lock, flags);
 	list_for_each_entry(tag, &domain->cache_tags, node) {
 		if (cache_tag_reusable(tag, did, iommu, dev, pasid, type)) {
+			trace_cache_tag_unassign(tag);
 			if (--tag->users == 0) {
 				list_del(&tag->node);
 				kfree(tag);
@@ -293,6 +297,8 @@ void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start,
 						  IOMMU_NO_PASID, info->ats_qdep);
 			break;
 		}
+
+		trace_cache_tag_flush_range(tag, start, end, addr, pages, mask);
 	}
 	spin_unlock_irqrestore(&domain->cache_lock, flags);
 }
@@ -330,6 +336,7 @@ void cache_tag_flush_all(struct dmar_domain *domain)
 						  IOMMU_NO_PASID, info->ats_qdep);
 			break;
 		}
+		trace_cache_tag_flush_all(tag);
 	}
 	spin_unlock_irqrestore(&domain->cache_lock, flags);
 }
@@ -376,6 +383,7 @@ void cache_tag_flush_cm_range(struct dmar_domain *domain, unsigned long start,
 							 addr, mask,
 							 DMA_TLB_PSI_FLUSH);
 		}
+		trace_cache_tag_flush_cm_range(tag, start, end, addr, pages, mask);
 	}
 	spin_unlock_irqrestore(&domain->cache_lock, flags);
 }
Signature · 1 line
2.34.1
author maintainer
#20 lore raw

PATCH 4/12 iommu/vt-d: Use cache_tag_flush_all() in flush_iotlb_all

In reply to Lu Baolu

The flush_iotlb_all callback is called by the iommu core to flush all caches for the affected domain. Use cache_tag_flush_all() in this callback.

---
 drivers/iommu/intel/iommu.c | 21 +--------------------
 1 file changed, 1 insertion(+), 20 deletions(-)
drivers/iommu/intel/iommu.c +1 -20
index 93e4422c9b10..4ce98f23917c 100644
@@ -1560,26 +1560,7 @@ static void parent_domain_flush(struct dmar_domain *domain,
 
 static void intel_flush_iotlb_all(struct iommu_domain *domain)
 {
-	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
-	struct iommu_domain_info *info;
-	unsigned long idx;
-
-	xa_for_each(&dmar_domain->iommu_array, idx, info) {
-		struct intel_iommu *iommu = info->iommu;
-		u16 did = domain_id_iommu(dmar_domain, iommu);
-
-		if (dmar_domain->use_first_level)
-			domain_flush_pasid_iotlb(iommu, dmar_domain, 0, -1, 0);
-		else
-			iommu->flush.flush_iotlb(iommu, did, 0, 0,
-						 DMA_TLB_DSI_FLUSH);
-
-		if (!cap_caching_mode(iommu->cap))
-			iommu_flush_dev_iotlb(dmar_domain, 0, MAX_AGAW_PFN_WIDTH);
-	}
-
-	if (dmar_domain->nested_parent)
-		parent_domain_flush(dmar_domain, 0, -1, 0);
+	cache_tag_flush_all(to_dmar_domain(domain));
 }
 
 static void iommu_disable_protect_mem_regions(struct intel_iommu *iommu)
Signature · 1 line
2.34.1
author maintainer
#23 lore raw

PATCH 5/12 iommu/vt-d: Use cache_tag_flush_range() in tlb_sync

In reply to Lu Baolu

The tlb_sync callback is called by the iommu core to flush a range of caches for the affected domain. Use cache_tag_flush_range() in this callback.

---
 drivers/iommu/intel/iommu.c | 21 ++-------------------
 1 file changed, 2 insertions(+), 19 deletions(-)
drivers/iommu/intel/iommu.c +2 -19
index 4ce98f23917c..1c03d2dafb9d 100644
@@ -4112,25 +4112,8 @@ static size_t intel_iommu_unmap_pages(struct iommu_domain *domain,
 static void intel_iommu_tlb_sync(struct iommu_domain *domain,
 				 struct iommu_iotlb_gather *gather)
 {
-	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
-	unsigned long iova_pfn = IOVA_PFN(gather->start);
-	size_t size = gather->end - gather->start;
-	struct iommu_domain_info *info;
-	unsigned long start_pfn;
-	unsigned long nrpages;
-	unsigned long i;
-
-	nrpages = aligned_nrpages(gather->start, size);
-	start_pfn = mm_to_dma_pfn_start(iova_pfn);
-
-	xa_for_each(&dmar_domain->iommu_array, i, info)
-		iommu_flush_iotlb_psi(info->iommu, dmar_domain,
-				      start_pfn, nrpages,
-				      list_empty(&gather->freelist), 0);
-
-	if (dmar_domain->nested_parent)
-		parent_domain_flush(dmar_domain, start_pfn, nrpages,
-				    list_empty(&gather->freelist));
+	cache_tag_flush_range(to_dmar_domain(domain), gather->start,
+			      gather->end, list_empty(&gather->freelist));
 	put_pages_list(&gather->freelist);
 }
 
Signature · 1 line
2.34.1
author maintainer
#24 lore raw

PATCH 6/12 iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map

In reply to Lu Baolu

The iotlb_sync_map callback is called by the iommu core after non-present to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Use cache_tag_flush_cm_range() in this callback.

---
 drivers/iommu/intel/iommu.c | 22 +---------------------
 1 file changed, 1 insertion(+), 21 deletions(-)
drivers/iommu/intel/iommu.c +1 -21
index 1c03d2dafb9d..2dcab1e5cd4d 100644
@@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(struct intel_iommu *iommu,
 		iommu_flush_dev_iotlb(domain, addr, mask);
 }
 
-/* Notification for newly created mappings */
-static void __mapping_notify_one(struct intel_iommu *iommu, struct dmar_domain *domain,
-				 unsigned long pfn, unsigned int pages)
-{
-	/*
-	 * It's a non-present to present mapping. Only flush if caching mode
-	 * and second level.
-	 */
-	if (cap_caching_mode(iommu->cap) && !domain->use_first_level)
-		iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1);
-	else
-		iommu_flush_write_buffer(iommu);
-}
-
 /*
  * Flush the relevant caches in nested translation if the domain
  * also serves as a parent
@@ -4549,14 +4535,8 @@ static bool risky_device(struct pci_dev *pdev)
 static int intel_iommu_iotlb_sync_map(struct iommu_domain *domain,
 				      unsigned long iova, size_t size)
 {
-	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
-	unsigned long pages = aligned_nrpages(iova, size);
-	unsigned long pfn = iova >> VTD_PAGE_SHIFT;
-	struct iommu_domain_info *info;
-	unsigned long i;
+	cache_tag_flush_cm_range(to_dmar_domain(domain), iova, iova + size - 1);
 
-	xa_for_each(&dmar_domain->iommu_array, i, info)
-		__mapping_notify_one(info->iommu, dmar_domain, pfn, pages);
 	return 0;
 }
 
Signature · 1 line
2.34.1
author maintainer
#29 lore raw

PATCH 7/12 iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi()

In reply to Lu Baolu

Use cache_tag_flush_range() in switch_to_super_page() to invalidate the necessary caches when switching mappings from normal to super pages. The iommu_flush_iotlb_psi() call in intel_iommu_memory_notifier() is unnecessary since there should be no cache invalidation for the identity domain.

Clean up iommu_flush_iotlb_psi() after the last call site is removed.

---
 drivers/iommu/intel/iommu.c | 175 +-----------------------------------
 1 file changed, 2 insertions(+), 173 deletions(-)
drivers/iommu/intel/iommu.c +2 -173
index 2dcab1e5cd4d..6e019297843b 100644
@@ -1389,161 +1389,6 @@ static void __iommu_flush_dev_iotlb(struct device_domain_info *info,
 	quirk_extra_dev_tlb_flush(info, addr, mask, IOMMU_NO_PASID, qdep);
 }
 
-static void iommu_flush_dev_iotlb(struct dmar_domain *domain,
-				  u64 addr, unsigned mask)
-{
-	struct dev_pasid_info *dev_pasid;
-	struct device_domain_info *info;
-	unsigned long flags;
-
-	if (!domain->has_iotlb_device)
-		return;
-
-	spin_lock_irqsave(&domain->lock, flags);
-	list_for_each_entry(info, &domain->devices, link)
-		__iommu_flush_dev_iotlb(info, addr, mask);
-
-	list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) {
-		info = dev_iommu_priv_get(dev_pasid->dev);
-
-		if (!info->ats_enabled)
-			continue;
-
-		qi_flush_dev_iotlb_pasid(info->iommu,
-					 PCI_DEVID(info->bus, info->devfn),
-					 info->pfsid, dev_pasid->pasid,
-					 info->ats_qdep, addr,
-					 mask);
-	}
-	spin_unlock_irqrestore(&domain->lock, flags);
-}
-
-static void domain_flush_pasid_iotlb(struct intel_iommu *iommu,
-				     struct dmar_domain *domain, u64 addr,
-				     unsigned long npages, bool ih)
-{
-	u16 did = domain_id_iommu(domain, iommu);
-	struct dev_pasid_info *dev_pasid;
-	unsigned long flags;
-
-	spin_lock_irqsave(&domain->lock, flags);
-	list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain)
-		qi_flush_piotlb(iommu, did, dev_pasid->pasid, addr, npages, ih);
-
-	if (!list_empty(&domain->devices))
-		qi_flush_piotlb(iommu, did, IOMMU_NO_PASID, addr, npages, ih);
-	spin_unlock_irqrestore(&domain->lock, flags);
-}
-
-static void __iommu_flush_iotlb_psi(struct intel_iommu *iommu, u16 did,
-				    unsigned long pfn, unsigned int pages,
-				    int ih)
-{
-	unsigned int aligned_pages = __roundup_pow_of_two(pages);
-	unsigned long bitmask = aligned_pages - 1;
-	unsigned int mask = ilog2(aligned_pages);
-	u64 addr = (u64)pfn << VTD_PAGE_SHIFT;
-
-	/*
-	 * PSI masks the low order bits of the base address. If the
-	 * address isn't aligned to the mask, then compute a mask value
-	 * needed to ensure the target range is flushed.
-	 */
-	if (unlikely(bitmask & pfn)) {
-		unsigned long end_pfn = pfn + pages - 1, shared_bits;
-
-		/*
-		 * Since end_pfn <= pfn + bitmask, the only way bits
-		 * higher than bitmask can differ in pfn and end_pfn is
-		 * by carrying. This means after masking out bitmask,
-		 * high bits starting with the first set bit in
-		 * shared_bits are all equal in both pfn and end_pfn.
-		 */
-		shared_bits = ~(pfn ^ end_pfn) & ~bitmask;
-		mask = shared_bits ? __ffs(shared_bits) : BITS_PER_LONG;
-	}
-
-	/*
-	 * Fallback to domain selective flush if no PSI support or
-	 * the size is too big.
-	 */
-	if (!cap_pgsel_inv(iommu->cap) || mask > cap_max_amask_val(iommu->cap))
-		iommu->flush.flush_iotlb(iommu, did, 0, 0,
-					 DMA_TLB_DSI_FLUSH);
-	else
-		iommu->flush.flush_iotlb(iommu, did, addr | ih, mask,
-					 DMA_TLB_PSI_FLUSH);
-}
-
-static void iommu_flush_iotlb_psi(struct intel_iommu *iommu,
-				  struct dmar_domain *domain,
-				  unsigned long pfn, unsigned int pages,
-				  int ih, int map)
-{
-	unsigned int aligned_pages = __roundup_pow_of_two(pages);
-	unsigned int mask = ilog2(aligned_pages);
-	uint64_t addr = (uint64_t)pfn << VTD_PAGE_SHIFT;
-	u16 did = domain_id_iommu(domain, iommu);
-
-	if (WARN_ON(!pages))
-		return;
-
-	if (ih)
-		ih = 1 << 6;
-
-	if (domain->use_first_level)
-		domain_flush_pasid_iotlb(iommu, domain, addr, pages, ih);
-	else
-		__iommu_flush_iotlb_psi(iommu, did, pfn, pages, ih);
-
-	/*
-	 * In caching mode, changes of pages from non-present to present require
-	 * flush. However, device IOTLB doesn't need to be flushed in this case.
-	 */
-	if (!cap_caching_mode(iommu->cap) || !map)
-		iommu_flush_dev_iotlb(domain, addr, mask);
-}
-
-/*
- * Flush the relevant caches in nested translation if the domain
- * also serves as a parent
- */
-static void parent_domain_flush(struct dmar_domain *domain,
-				unsigned long pfn,
-				unsigned long pages, int ih)
-{
-	struct dmar_domain *s1_domain;
-
-	spin_lock(&domain->s1_lock);
-	list_for_each_entry(s1_domain, &domain->s1_domains, s2_link) {
-		struct device_domain_info *device_info;
-		struct iommu_domain_info *info;
-		unsigned long flags;
-		unsigned long i;
-
-		xa_for_each(&s1_domain->iommu_array, i, info)
-			__iommu_flush_iotlb_psi(info->iommu, info->did,
-						pfn, pages, ih);
-
-		if (!s1_domain->has_iotlb_device)
-			continue;
-
-		spin_lock_irqsave(&s1_domain->lock, flags);
-		list_for_each_entry(device_info, &s1_domain->devices, link)
-			/*
-			 * Address translation cache in device side caches the
-			 * result of nested translation. There is no easy way
-			 * to identify the exact set of nested translations
-			 * affected by a change in S2. So just flush the entire
-			 * device cache.
-			 */
-			__iommu_flush_dev_iotlb(device_info, 0,
-						MAX_AGAW_PFN_WIDTH);
-		spin_unlock_irqrestore(&s1_domain->lock, flags);
-	}
-	spin_unlock(&domain->s1_lock);
-}
-
 static void intel_flush_iotlb_all(struct iommu_domain *domain)
 {
 	cache_tag_flush_all(to_dmar_domain(domain));
@@ -1995,9 +1840,7 @@ static void switch_to_super_page(struct dmar_domain *domain,
 				 unsigned long end_pfn, int level)
 {
 	unsigned long lvl_pages = lvl_to_nr_pages(level);
-	struct iommu_domain_info *info;
 	struct dma_pte *pte = NULL;
-	unsigned long i;
 
 	while (start_pfn <= end_pfn) {
 		if (!pte)
@@ -2009,13 +1852,8 @@ static void switch_to_super_page(struct dmar_domain *domain,
 					       start_pfn + lvl_pages - 1,
 					       level + 1);
 
-			xa_for_each(&domain->iommu_array, i, info)
-				iommu_flush_iotlb_psi(info->iommu, domain,
-						      start_pfn, lvl_pages,
-						      0, 0);
-			if (domain->nested_parent)
-				parent_domain_flush(domain, start_pfn,
-						    lvl_pages, 0);
+			cache_tag_flush_range(domain, start_pfn << VTD_PAGE_SHIFT,
+					      end_pfn << VTD_PAGE_SHIFT, 0);
 		}
 
 		pte++;
@@ -3387,18 +3225,9 @@ static int intel_iommu_memory_notifier(struct notifier_block *nb,
 	case MEM_OFFLINE:
 	case MEM_CANCEL_ONLINE:
 		{
-			struct dmar_drhd_unit *drhd;
-			struct intel_iommu *iommu;
 			LIST_HEAD(freelist);
 
 			domain_unmap(si_domain, start_vpfn, last_vpfn, &freelist);
-
-			rcu_read_lock();
-			for_each_active_iommu(iommu, drhd)
-				iommu_flush_iotlb_psi(iommu, si_domain,
-					start_vpfn, mhp->nr_pages,
-					list_empty(&freelist), 0);
-			rcu_read_unlock();
 			put_pages_list(&freelist);
 		}
 		break;
Signature · 1 line
2.34.1
author maintainer
#33 lore raw

PATCH 8/12 iommu/vt-d: Use cache_tag_flush_range() in cache_invalidate_user

In reply to Lu Baolu

The cache_invalidate_user callback is called to invalidate a range of caches for the affected user domain. Use cache_tag_flush_range() in this callback.

---
 drivers/iommu/intel/nested.c | 50 +++---------------------------------
 1 file changed, 3 insertions(+), 47 deletions(-)
drivers/iommu/intel/nested.c +3 -47
index 85c744099558..e251507cfcc0 100644
@@ -90,50 +90,6 @@ static void intel_nested_domain_free(struct iommu_domain *domain)
 	kfree(dmar_domain);
 }
 
-static void nested_flush_dev_iotlb(struct dmar_domain *domain, u64 addr,
-				   unsigned int mask)
-{
-	struct device_domain_info *info;
-	unsigned long flags;
-	u16 sid, qdep;
-
-	spin_lock_irqsave(&domain->lock, flags);
-	list_for_each_entry(info, &domain->devices, link) {
-		if (!info->ats_enabled)
-			continue;
-		sid = info->bus << 8 | info->devfn;
-		qdep = info->ats_qdep;
-		qi_flush_dev_iotlb(info->iommu, sid, info->pfsid,
-				   qdep, addr, mask);
-		quirk_extra_dev_tlb_flush(info, addr, mask,
-					  IOMMU_NO_PASID, qdep);
-	}
-	spin_unlock_irqrestore(&domain->lock, flags);
-}
-
-static void intel_nested_flush_cache(struct dmar_domain *domain, u64 addr,
-				     u64 npages, bool ih)
-{
-	struct iommu_domain_info *info;
-	unsigned int mask;
-	unsigned long i;
-
-	xa_for_each(&domain->iommu_array, i, info)
-		qi_flush_piotlb(info->iommu,
-				domain_id_iommu(domain, info->iommu),
-				IOMMU_NO_PASID, addr, npages, ih);
-
-	if (!domain->has_iotlb_device)
-		return;
-
-	if (npages == U64_MAX)
-		mask = 64 - VTD_PAGE_SHIFT;
-	else
-		mask = ilog2(__roundup_pow_of_two(npages));
-
-	nested_flush_dev_iotlb(domain, addr, mask);
-}
-
 static int intel_nested_cache_invalidate_user(struct iommu_domain *domain,
 					      struct iommu_user_data_array *array)
 {
@@ -166,9 +122,9 @@ static int intel_nested_cache_invalidate_user(struct iommu_domain *domain,
 			break;
 		}
 
-		intel_nested_flush_cache(dmar_domain, inv_entry.addr,
-					 inv_entry.npages,
-					 inv_entry.flags & IOMMU_VTD_INV_FLAGS_LEAF);
+		cache_tag_flush_range(dmar_domain, inv_entry.addr,
+				      inv_entry.npages,
+				      inv_entry.flags & IOMMU_VTD_INV_FLAGS_LEAF);
 		processed++;
 	}
 
Signature · 1 line
2.34.1
author maintainer
#36 lore raw

PATCH 9/12 iommu/vt-d: Use cache helpers in arch_invalidate_secondary_tlbs

In reply to Lu Baolu

The arch_invalidate_secondary_tlbs callback is called in the SVA mm notification path. It invalidates all or a range of caches after the CPU page table is modified. Use the cache tag helps in this path.

---
 drivers/iommu/intel/iommu.h |  1 +
 drivers/iommu/intel/svm.c   | 76 +++----------------------------------
 2 files changed, 6 insertions(+), 71 deletions(-)
drivers/iommu/intel/iommu.h +1 -0
index d05fa0122d65..2575ded9f354 100644
@@ -1148,6 +1148,7 @@ struct intel_svm {
 	struct mm_struct *mm;
 	u32 pasid;
 	struct list_head devs;
+	struct dmar_domain *domain;
 };
 #else
 static inline void intel_svm_check(struct intel_iommu *iommu) {}
drivers/iommu/intel/svm.c +5 -71

105 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#37 lore raw

PATCH 10/12 iommu/vt-d: Retire intel_svm_dev

In reply to Lu Baolu

The intel_svm_dev data structure used in the sva implementation for the Intel IOMMU driver stores information about a device attached to an SVA domain. It is a duplicate of dev_pasid_info that serves the same purpose.

Replace intel_svm_dev with dev_pasid_info and clean up the use of intel_svm_dev.

---
 drivers/iommu/intel/iommu.h |  15 +----
 drivers/iommu/intel/iommu.c |  30 ++++-----
 drivers/iommu/intel/svm.c   | 131 +++++++++++-------------------------
 3 files changed, 55 insertions(+), 121 deletions(-)
drivers/iommu/intel/iommu.h +3 -12
index 2575ded9f354..5dcced6f3634 100644
@@ -650,6 +650,7 @@ struct dmar_domain {
 			struct list_head s2_link;
 		};
 	};
+	struct intel_svm *svm;
 
 	struct iommu_domain domain;	/* generic domain data structure for
 					   iommu core */
@@ -1131,23 +1132,13 @@ int intel_svm_finish_prq(struct intel_iommu *iommu);
 void intel_svm_page_response(struct device *dev, struct iopf_fault *evt,
 			     struct iommu_page_response *msg);
 struct iommu_domain *intel_svm_domain_alloc(void);
-void intel_svm_remove_dev_pasid(struct device *dev, ioasid_t pasid);
+void intel_svm_remove_dev_pasid(struct iommu_domain *domain);
 void intel_drain_pasid_prq(struct device *dev, u32 pasid);
 
-struct intel_svm_dev {
-	struct list_head list;
-	struct rcu_head rcu;
-	struct device *dev;
-	struct intel_iommu *iommu;
-	u16 did;
-	u16 sid, qdep;
-};
-
 struct intel_svm {
 	struct mmu_notifier notifier;
 	struct mm_struct *mm;
 	u32 pasid;
-	struct list_head devs;
 	struct dmar_domain *domain;
 };
 #else
@@ -1158,7 +1149,7 @@ static inline struct iommu_domain *intel_svm_domain_alloc(void)
 	return NULL;
 }
 
-static inline void intel_svm_remove_dev_pasid(struct device *dev, ioasid_t pasid)
+static inline void intel_svm_remove_dev_pasid(struct iommu_domain *domain)
 {
 }
 #endif
drivers/iommu/intel/iommu.c +14 -16
index 6e019297843b..2ac1fc041333 100644
@@ -4384,18 +4384,6 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
 		goto out_tear_down;
 	dmar_domain = to_dmar_domain(domain);
 
-	/*
-	 * The SVA implementation needs to handle its own stuffs like the mm
-	 * notification. Before consolidating that code into iommu core, let
-	 * the intel sva code handle it.
-	 */
-	if (domain->type == IOMMU_DOMAIN_SVA) {
-		intel_svm_remove_dev_pasid(dev, pasid);
-		cache_tag_unassign_domain(dmar_domain,
-					  FLPT_DEFAULT_DID, dev, pasid);
-		goto out_tear_down;
-	}
-
 	spin_lock_irqsave(&dmar_domain->lock, flags);
 	list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) {
 		if (curr->dev == dev && curr->pasid == pasid) {
@@ -4407,10 +4395,20 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
 	WARN_ON_ONCE(!dev_pasid);
 	spin_unlock_irqrestore(&dmar_domain->lock, flags);
 
-	did = domain_id_iommu(dmar_domain, iommu);
-	cache_tag_unassign_domain(dmar_domain, did, dev, pasid);
-	domain_detach_iommu(dmar_domain, iommu);
-	intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
+	/*
+	 * The SVA implementation needs to handle its own stuffs like the mm
+	 * notification. Before consolidating that code into iommu core, let
+	 * the intel sva code handle it.
+	 */
+	if (domain->type == IOMMU_DOMAIN_SVA) {
+		cache_tag_unassign_domain(dmar_domain, FLPT_DEFAULT_DID, dev, pasid);
+		intel_svm_remove_dev_pasid(domain);
+	} else {
+		did = domain_id_iommu(dmar_domain, iommu);
+		cache_tag_unassign_domain(dmar_domain, did, dev, pasid);
+		domain_detach_iommu(dmar_domain, iommu);
+		intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
+	}
 	kfree(dev_pasid);
 out_tear_down:
 	intel_pasid_tear_down_entry(iommu, dev, pasid, false);
drivers/iommu/intel/svm.c +38 -93

210 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#38 lore raw

PATCH 11/12 iommu: Add ops->domain_alloc_sva()

In reply to Lu Baolu

From: Jason Gunthorpe <[email protected]>

Make a new op that receives the device and the mm_struct that the SVA domain should be created for. Unlike domain_alloc_paging() the dev argument is never NULL here.

This allows drivers to fully initialize the SVA domain and allocate the mmu_notifier during allocation. It allows the notifier lifetime to follow the lifetime of the iommu_domain.

Since we have only one call site, upgrade the new op to return ERR_PTR instead of NULL.

---
 include/linux/iommu.h     |  3 +++
 drivers/iommu/iommu-sva.c | 16 +++++++++++-----
 2 files changed, 14 insertions(+), 5 deletions(-)
include/linux/iommu.h +3 -0
index 2e925b5eba53..8aabe83af8f2 100644
@@ -518,6 +518,7 @@ static inline int __iommu_copy_struct_from_user_array(
  *                     Upon failure, ERR_PTR must be returned.
  * @domain_alloc_paging: Allocate an iommu_domain that can be used for
  *                       UNMANAGED, DMA, and DMA_FQ domain types.
+ * @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressing.
  * @probe_device: Add device to iommu driver handling
  * @release_device: Remove device from iommu driver handling
  * @probe_finalize: Do final setup work after the device is added to an IOMMU
@@ -558,6 +559,8 @@ struct iommu_ops {
 		struct device *dev, u32 flags, struct iommu_domain *parent,
 		const struct iommu_user_data *user_data);
 	struct iommu_domain *(*domain_alloc_paging)(struct device *dev);
+	struct iommu_domain *(*domain_alloc_sva)(struct device *dev,
+						 struct mm_struct *mm);
 
 	struct iommu_device *(*probe_device)(struct device *dev);
 	void (*release_device)(struct device *dev);
drivers/iommu/iommu-sva.c +11 -5
index 640acc804e8c..18a35e798b72 100644
@@ -108,8 +108,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
 
 	/* Allocate a new domain and set it on device pasid. */
 	domain = iommu_sva_domain_alloc(dev, mm);
-	if (!domain) {
-		ret = -ENOMEM;
+	if (IS_ERR(domain)) {
+		ret = PTR_ERR(domain);
 		goto out_free_handle;
 	}
 
@@ -283,9 +283,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
 	const struct iommu_ops *ops = dev_iommu_ops(dev);
 	struct iommu_domain *domain;
 
-	domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
-	if (!domain)
-		return NULL;
+	if (ops->domain_alloc_sva) {
+		domain = ops->domain_alloc_sva(dev, mm);
+		if (IS_ERR(domain))
+			return domain;
+	} else {
+		domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
+		if (!domain)
+			return ERR_PTR(-ENOMEM);
+	}
 
 	domain->type = IOMMU_DOMAIN_SVA;
 	mmgrab(mm);
Signature · 1 line
2.34.1
author maintainer
#39 lore raw

PATCH 12/12 iommu/vt-d: Retire struct intel_svm

In reply to Lu Baolu

The struct intel_svm was used for keeping attached devices info for sva domain. Since sva domain is a kind of iommu_domain, the struct dmar_domain should centralize all info of a sva domain, including the info of attached devices. Therefore, retire struct intel_svm to clean up the code.

Besides, allocate sva domain in domain_alloc_sva() callback which allows the memory management notifier lifetime to follow the lifetime of the iommu_domain.

---
 drivers/iommu/intel/iommu.h | 26 ++++------
 drivers/iommu/intel/iommu.c |  9 +---
 drivers/iommu/intel/svm.c   | 94 +++++++++----------------------------
 3 files changed, 32 insertions(+), 97 deletions(-)
drivers/iommu/intel/iommu.h +10 -16
index 5dcced6f3634..7a4645e06bb7 100644
@@ -649,8 +649,12 @@ struct dmar_domain {
 			/* link to parent domain siblings */
 			struct list_head s2_link;
 		};
+
+		/* SVA domain */
+		struct {
+			struct mmu_notifier notifier;
+		};
 	};
-	struct intel_svm *svm;
 
 	struct iommu_domain domain;	/* generic domain data structure for
 					   iommu core */
@@ -1131,26 +1135,16 @@ int intel_svm_enable_prq(struct intel_iommu *iommu);
 int intel_svm_finish_prq(struct intel_iommu *iommu);
 void intel_svm_page_response(struct device *dev, struct iopf_fault *evt,
 			     struct iommu_page_response *msg);
-struct iommu_domain *intel_svm_domain_alloc(void);
-void intel_svm_remove_dev_pasid(struct iommu_domain *domain);
+struct iommu_domain *intel_svm_domain_alloc(struct device *dev,
+					    struct mm_struct *mm);
 void intel_drain_pasid_prq(struct device *dev, u32 pasid);
-
-struct intel_svm {
-	struct mmu_notifier notifier;
-	struct mm_struct *mm;
-	u32 pasid;
-	struct dmar_domain *domain;
-};
 #else
 static inline void intel_svm_check(struct intel_iommu *iommu) {}
 static inline void intel_drain_pasid_prq(struct device *dev, u32 pasid) {}
-static inline struct iommu_domain *intel_svm_domain_alloc(void)
-{
-	return NULL;
-}
-
-static inline void intel_svm_remove_dev_pasid(struct iommu_domain *domain)
+static inline struct iommu_domain *intel_svm_domain_alloc(struct device *dev,
+							  struct mm_struct *mm)
 {
+	return ERR_PTR(-ENODEV);
 }
 #endif
 
drivers/iommu/intel/iommu.c +1 -8
index 2ac1fc041333..cdf7882f58f3 100644
@@ -3687,8 +3687,6 @@ static struct iommu_domain *intel_iommu_domain_alloc(unsigned type)
 		return domain;
 	case IOMMU_DOMAIN_IDENTITY:
 		return &si_domain->domain;
-	case IOMMU_DOMAIN_SVA:
-		return intel_svm_domain_alloc();
 	default:
 		return NULL;
 	}
@@ -4395,14 +4393,8 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
 	WARN_ON_ONCE(!dev_pasid);
 	spin_unlock_irqrestore(&dmar_domain->lock, flags);
 
-	/*
-	 * The SVA implementation needs to handle its own stuffs like the mm
-	 * notification. Before consolidating that code into iommu core, let
-	 * the intel sva code handle it.
-	 */
 	if (domain->type == IOMMU_DOMAIN_SVA) {
 		cache_tag_unassign_domain(dmar_domain, FLPT_DEFAULT_DID, dev, pasid);
-		intel_svm_remove_dev_pasid(domain);
 	} else {
 		did = domain_id_iommu(dmar_domain, iommu);
 		cache_tag_unassign_domain(dmar_domain, did, dev, pasid);
@@ -4631,6 +4623,7 @@ const struct iommu_ops intel_iommu_ops = {
 	.hw_info		= intel_iommu_hw_info,
 	.domain_alloc		= intel_iommu_domain_alloc,
 	.domain_alloc_user	= intel_iommu_domain_alloc_user,
+	.domain_alloc_sva	= intel_svm_domain_alloc,
 	.probe_device		= intel_iommu_probe_device,
 	.probe_finalize		= intel_iommu_probe_finalize,
 	.release_device		= intel_iommu_release_device,
drivers/iommu/intel/svm.c +21 -73

156 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

RE: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

In reply to Lu Baolu

drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1103

Quoted 9 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

drivers/iommu/intel/iommu.h +6 -0
+enum cache_tag_type {
+	CACHE_TAG_TYPE_IOTLB,
+	CACHE_TAG_TYPE_DEVTLB,
+	CACHE_TAG_TYPE_PARENT_IOTLB,
+	CACHE_TAG_TYPE_PARENT_DEVTLB,
+};

'_TYPE_' can be removed to make it shorter

drivers/iommu/intel/cache.c line 23

Quoted 5 lines
drivers/iommu/intel/cache.c new file +5 -0
+
+/* Checks if an existing cache tag can be reused for a new association. */
+static bool cache_tag_reusable(struct cache_tag *tag, u16 domain_id,
+			       struct intel_iommu *iommu, struct device *dev,
+			       ioasid_t pasid, enum cache_tag_type type)

cache_tage_match()

Quoted 4 lines
drivers/iommu/intel/cache.c new file +4 -0
+{
+	if (tag->type != type)
+		return false;
+

drivers/iommu/intel/cache.c line 37

Quoted 10 lines
drivers/iommu/intel/cache.c new file +10 -0
+	if (tag->domain_id != domain_id || tag->pasid != pasid)
+		return false;
+
+	if (type == CACHE_TAG_TYPE_IOTLB)
+		return tag->iommu == iommu;
+
+	if (type == CACHE_TAG_TYPE_DEVTLB)
+		return tag->dev == dev;
+
+	return false;

why do you disallow PARENT_TYPE from reusing? It's not uncommon to have two devices attached to a same nested domain hence with the same parent domain. Disallowing tag reuse implies unnecessarily duplicated cache flushes...

Quoted 9 lines
drivers/iommu/intel/cache.c new file +9 -0
+}
+
+/* Assign a cache tag with specified type to domain. */
+static int cache_tag_assign(struct dmar_domain *domain, u16 did,
+			    struct device *dev, ioasid_t pasid,
+			    enum cache_tag_type type)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+	struct intel_iommu *iommu = info->iommu;

drivers/iommu/intel/cache.c line 56

Quoted 10 lines
drivers/iommu/intel/cache.c new file +10 -0
+	struct cache_tag *tag, *temp;
+	unsigned long flags;
+
+	tag = kzalloc(sizeof(*tag), GFP_KERNEL);
+	if (!tag)
+		return -ENOMEM;
+
+	tag->type = type;
+	tag->iommu = iommu;
+	tag->dev = dev;

should we set tag->dev only for DEVTLB type? It's a bit confusing to set it for IOTLB type which doesn't care about device. Actually doing so is instead misleading as the 1st device creating the tag may have been detached but then it will still show up in the trace when the last device detach destroying the tag.

Review

Quoted 3 lines

+static int __cache_tag_assign_parent_domain(struct dmar_domain *domain, u16 did,

(quoted diff) +1 -0
+					    struct device *dev, ioasid_t pasid)

this pair is similar to the earlier one except the difference on type.

what about keeping just one pair which accepts a 'parent' argument to decide the type internally?

Quoted 3 lines

+/*
+ * Assigns cache tags to a domain when it's associated with a device's
+ * PASID using a specific domain ID.

s/Assigns/Assign/

Quoted 4 lines
drivers/iommu/intel/iommu.c +3 -0
+
+	did = domain_id_iommu(domain, iommu);
+	ret = cache_tag_assign_domain(domain, did, dev,

IOMMU_NO_PASID);

there are many occurrences of this pattern. What about passing in a 'iommu' parameter and getting 'did' inside the helper? for svm it can be specialized internally too.

drivers/iommu/intel/iommu.c static void · line 4628

Quoted 7 lines
drivers/iommu/intel/iommu.c +2 -0
@@ -4607,10 +4623,11 @@ static void
 intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
 	 */
 	if (domain->type == IOMMU_DOMAIN_SVA) {
 		intel_svm_remove_dev_pasid(dev, pasid);
+		cache_tag_unassign_domain(dmar_domain,
+					  FLPT_DEFAULT_DID, dev, pasid);

is it correct to destroy the tag before teardown completes, e.g. iotlb still needs to be flushed in intel_pasid_tear_down_entry()?

#14 lore raw

RE: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers

In reply to Lu Baolu
Quoted 11 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

drivers/iommu/intel/cache.c +7 -0
+
+static unsigned long calculate_psi_aligned_address(unsigned long start,
+						   unsigned long end,
+						   unsigned long *_pages,
+						   unsigned long *_mask)
+{
+	unsigned long pages = (end - start + VTD_PAGE_SIZE - 1) >>

VTD_PAGE_SHIFT;

this doesn't sound correct. You'd want to follow aligned_nrpages().

Quoted 30 lines
drivers/iommu/intel/cache.c +4 -0
+		case CACHE_TAG_TYPE_DEVTLB:
+			if (tag->pasid == IOMMU_NO_PASID)
+				qi_flush_dev_iotlb(iommu, sid, info->pfsid,
+						   info->ats_qdep, addr,

mask);

(quoted diff) +2 -0
+			else
+				qi_flush_dev_iotlb_pasid(iommu, sid, info-
Quoted 1 line

pfsid,

(quoted diff) +1 -0
+							 tag->pasid, info-
Quoted 1 line

ats_qdep,

drivers/iommu/intel/cache.c +3 -0
+							 addr, mask);
+
+			quirk_extra_dev_tlb_flush(info, addr, mask, tag-
Quoted 1 line

pasid, info->ats_qdep);

drivers/iommu/intel/cache.c +10 -0
+			break;
+		case CACHE_TAG_TYPE_PARENT_DEVTLB:
+			/*
+			 * Address translation cache in device side caches the
+			 * result of nested translation. There is no easy way
+			 * to identify the exact set of nested translations
+			 * affected by a change in S2. So just flush the entire
+			 * device cache.
+			 */
+			qi_flush_dev_iotlb(iommu, sid, info->pfsid, info-
Quoted 1 line

ats_qdep,

(quoted diff) +2 -0
+					   0, MAX_AGAW_PFN_WIDTH);
+			quirk_extra_dev_tlb_flush(info, 0,

MAX_AGAW_PFN_WIDTH,

(quoted diff) +1 -0
+						  IOMMU_NO_PASID, info-
Quoted 1 line

ats_qdep);

Review

Quoted 1 line
(quoted diff) +1 -0
+			break;

parent devtlb can has pasid too, though not enabled now. As core helpers probably we can support it now leading to simpler code:

	case CACHE_TAG_TYPE_PARENT_DEVTLB:
		//change addr/mask
		//fallthrough
	case CACHE_TAG_TYPE_DEVTLB:
		//what this patch does
Quoted 33 lines
drivers/iommu/intel/cache.c +8 -0
+void cache_tag_flush_all(struct dmar_domain *domain)
+{
+	struct cache_tag *tag;
+	unsigned long flags;
+
+	spin_lock_irqsave(&domain->cache_lock, flags);
+	list_for_each_entry(tag, &domain->cache_tags, node) {
+		struct device_domain_info *info = dev_iommu_priv_get(tag-
Quoted 1 line

dev);

drivers/iommu/intel/cache.c +11 -0
+		struct intel_iommu *iommu = tag->iommu;
+		u16 sid = PCI_DEVID(info->bus, info->devfn);
+
+		switch (tag->type) {
+		case CACHE_TAG_TYPE_IOTLB:
+		case CACHE_TAG_TYPE_PARENT_IOTLB:
+			if (domain->use_first_level)
+				qi_flush_piotlb(iommu, tag->domain_id,
+						tag->pasid, 0, -1, 0);
+			else
+				iommu->flush.flush_iotlb(iommu, tag-
Quoted 1 line

domain_id,

(quoted diff) +1 -0
+							 0, 0,

DMA_TLB_DSI_FLUSH);

drivers/iommu/intel/cache.c +4 -0
+			break;
+		case CACHE_TAG_TYPE_DEVTLB:
+		case CACHE_TAG_TYPE_PARENT_DEVTLB:
+			qi_flush_dev_iotlb(iommu, sid, info->pfsid, info-
Quoted 1 line

ats_qdep,

(quoted diff) +2 -0
+					   0, MAX_AGAW_PFN_WIDTH);
+			quirk_extra_dev_tlb_flush(info, 0,

MAX_AGAW_PFN_WIDTH,

(quoted diff) +1 -0
+						  IOMMU_NO_PASID, info-
Quoted 1 line

ats_qdep);

drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 296

Quoted 3 lines
drivers/iommu/intel/cache.c +3 -0
+			break;
+		}
+	}

could this loop be consolidated with the one in previous helper? anyway the only difference is on addr/mask...

drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 343

Quoted 9 lines

+/*
+ * Invalidate a range of IOVA when IOMMU is in caching mode and new mappings
+ * are added to the target domain.
+ */
+void cache_tag_flush_cm_range(struct dmar_domain *domain, unsigned long start,

drivers/iommu/intel/cache.c +2 -0
+			      unsigned long end)
+{

I'm also not sure why this is worth a separate helper. why couldn't it be managed by previous flush_range()?

Quoted 18 lines
drivers/iommu/intel/cache.c +12 -0
+	unsigned long pages, mask, addr;
+	struct cache_tag *tag;
+	unsigned long flags;
+
+	addr = calculate_psi_aligned_address(start, end, &pages, &mask);
+
+	spin_lock_irqsave(&domain->cache_lock, flags);
+	list_for_each_entry(tag, &domain->cache_tags, node) {
+		struct intel_iommu *iommu = tag->iommu;
+
+		/*
+		 * When IOMMU is enabled in caching mode some non-

present

drivers/iommu/intel/cache.c +4 -0
+		 * mappings may be cached by the IOTLB if it uses second-
+		 * only translation.
+		 */
+		if (!cap_caching_mode(iommu->cap) || domain-
Quoted 1 line

use_first_level) {

drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 362

Quoted 3 lines
drivers/iommu/intel/cache.c +3 -0
+			iommu_flush_write_buffer(iommu);
+			continue;
+		}

the comment really doesn't tell what this block does. from its intention it probably more suitable to be in the caller side as today.

Quoted 19 lines
drivers/iommu/intel/cache.c +10 -0
+
+		if (tag->type == CACHE_TAG_TYPE_IOTLB ||
+		    tag->type == CACHE_TAG_TYPE_PARENT_IOTLB) {
+			/*
+			 * Fallback to domain selective flush if no
+			 * PSI support or the size is too big.
+			 */
+			if (!cap_pgsel_inv(iommu->cap) ||
+			    mask > cap_max_amask_val(iommu->cap))
+				iommu->flush.flush_iotlb(iommu, tag-
Quoted 1 line

domain_id,

(quoted diff) +1 -0
+							 0, 0,

DMA_TLB_DSI_FLUSH);

(quoted diff) +2 -0
+			else
+				iommu->flush.flush_iotlb(iommu, tag-
Quoted 1 line

domain_id,

(quoted diff) +2 -0
+							 addr, mask,
+

DMA_TLB_PSI_FLUSH);

Review

Quoted 2 lines
(quoted diff) +2 -0
+		}
+	}

You skipped devtlb invalidation. yes it's not necessary based on current nested translation design and this part is inconsistent in different paths.

but as a semantics change you may want to first make removing devtlb invalidation a separate patch and then do this cleanup, so bisect is easier.

#21 lore raw

RE: [PATCH 04/12] iommu/vt-d: Use cache_tag_flush_all() in flush_iotlb_all

In reply to Lu Baolu
Quoted 36 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

The flush_iotlb_all callback is called by the iommu core to flush all caches for the affected domain. Use cache_tag_flush_all() in this callback.

---
 drivers/iommu/intel/iommu.c | 21 +--------------------
 1 file changed, 1 insertion(+), 20 deletions(-)
drivers/iommu/intel/iommu.c +0 -14
index 93e4422c9b10..4ce98f23917c 100644
@@ -1560,26 +1560,7 @@ static void parent_domain_flush(struct
 dmar_domain *domain,
 
 static void intel_flush_iotlb_all(struct iommu_domain *domain)
 {
-	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
-	struct iommu_domain_info *info;
-	unsigned long idx;
-
-	xa_for_each(&dmar_domain->iommu_array, idx, info) {
-		struct intel_iommu *iommu = info->iommu;
-		u16 did = domain_id_iommu(dmar_domain, iommu);
-
-		if (dmar_domain->use_first_level)
-			domain_flush_pasid_iotlb(iommu, dmar_domain, 0,
-1, 0);
-		else
-			iommu->flush.flush_iotlb(iommu, did, 0, 0,
-						 DMA_TLB_DSI_FLUSH);

drivers/iommu/intel/iommu.c static void parent_domain_flush(struct · line 1564

Quoted 10 lines
drivers/iommu/intel/iommu.c +1 -7
-
-		if (!cap_caching_mode(iommu->cap))
-			iommu_flush_dev_iotlb(dmar_domain, 0,
 MAX_AGAW_PFN_WIDTH);
-	}
-
-	if (dmar_domain->nested_parent)
-		parent_domain_flush(dmar_domain, 0, -1, 0);
+	cache_tag_flush_all(to_dmar_domain(domain));
 }

this replacement causes a functional change. Now devtlb is always invalidated while old code doesn't do so for caching mode.

Probably you may want to first clean up all inconsistent devtlb invalidation policies for caching mode before going to this series...

#25 lore raw

RE: [PATCH 06/12] iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map

In reply to Lu Baolu
Quoted 27 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

The iotlb_sync_map callback is called by the iommu core after non-present to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Use cache_tag_flush_cm_range() in this callback.

---
 drivers/iommu/intel/iommu.c | 22 +---------------------
 1 file changed, 1 insertion(+), 21 deletions(-)
drivers/iommu/intel/iommu.c +0 -3
index 1c03d2dafb9d..2dcab1e5cd4d 100644
@@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(struct
 intel_iommu *iommu,
 		iommu_flush_dev_iotlb(domain, addr, mask);
 }
 
-/* Notification for newly created mappings */
-static void __mapping_notify_one(struct intel_iommu *iommu, struct
 dmar_domain *domain,
-				 unsigned long pfn, unsigned int pages)

drivers/iommu/intel/iommu.c static void iommu_flush_iotlb_psi(struct · line 1521

Quoted 10 lines
drivers/iommu/intel/iommu.c +0 -10
-{
-	/*
-	 * It's a non-present to present mapping. Only flush if caching mode
-	 * and second level.
-	 */
-	if (cap_caching_mode(iommu->cap) && !domain->use_first_level)
-		iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1);
-	else
-		iommu_flush_write_buffer(iommu);
-}

iommu_flush_write_buffer is for a quite different issue. it's clearer to keep it separated from the iotlb helpers.

#30 lore raw

RE: [PATCH 07/12] iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi()

In reply to Lu Baolu
Quoted 9 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

Use cache_tag_flush_range() in switch_to_super_page() to invalidate the necessary caches when switching mappings from normal to super pages. The iommu_flush_iotlb_psi() call in intel_iommu_memory_notifier() is unnecessary since there should be no cache invalidation for the identity domain.

what about a software identity domain?

#34 lore raw

RE: [PATCH 08/12] iommu/vt-d: Use cache_tag_flush_range() in cache_invalidate_user

In reply to Lu Baolu
Quoted 6 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

drivers/iommu/intel/nested.c +0 -0
@@ -166,9 +122,9 @@ static int intel_nested_cache_invalidate_user(struct
 iommu_domain *domain,
 			break;

drivers/iommu/intel/nested.c static int intel_nested_cache_invalidate_user(struct · line 130

Quoted 10 lines
drivers/iommu/intel/nested.c +3 -3
 		}
 
-		intel_nested_flush_cache(dmar_domain, inv_entry.addr,
-					 inv_entry.npages,
-					 inv_entry.flags &
 IOMMU_VTD_INV_FLAGS_LEAF);
+		cache_tag_flush_range(dmar_domain, inv_entry.addr,
+				      inv_entry.npages,
+				      inv_entry.flags &
 IOMMU_VTD_INV_FLAGS_LEAF);

the helper requires an 'end' address but 'npages' is incorrectly used here.

#40 lore raw
In reply to Lu Baolu
Quoted 24 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

The IOMMU hardware cache needs to be invalidated whenever the mappings
in the domain are changed. Currently, domain cache invalidation is scattered across different places, causing several issues:

- IOMMU IOTLB Invalidation: This is done by iterating through the domain IDs of each domain using the following code:

        xa_for_each(&dmar_domain->iommu_array, i, info)
                iommu_flush_iotlb_psi(info->iommu, dmar_domain,
                                      start_pfn, nrpages,
                                      list_empty(&gather->freelist), 0);

This code could theoretically cause a use-after-free problem because there's no lock to protect the "info" pointer within the loop.

- Inconsistent Invalidation Methods: Different domain types implement their own cache invalidation methods, making the code difficult to maintain. For example, the DMA domain, SVA domain, and nested domain have similar cache invalidation code scattered across different files.

Quoted 5 lines

- SVA Domain Inconsistency: The SVA domain implementation uses a completely different data structure to track attached devices compared to other domains. This creates unnecessary differences and, even worse, leads to duplicate IOTLB invalidation when an SVA domain is attached to devices belonging to different IOMMU domains.

can you elaborate how duplicated invalidations are caused?

Quoted 6 lines

- Nested Domain Dependency: The special overlap between a nested domain and its parent domain requires a dedicated parent_domain_flush() helper function to be called everywhere the parent domain's mapping changes.

Quoted 7 lines

- Limited Debugging Support: There are currently no debugging aids available for domain cache invalidation.

By consolidating domain cache invalidation into a common location, we can address the issues mentioned above and improve the code's maintainability and debuggability.

overall this is a nice work!

author maintainer
#3 lore raw

Re: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

In reply to Tian, Kevin

Hi Kevin,

Thanks for your review comments.

Quoted 11 lines · Kevin

drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1103

Quoted 9 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

drivers/iommu/intel/iommu.h +6 -0
+enum cache_tag_type {
+	CACHE_TAG_TYPE_IOTLB,
+	CACHE_TAG_TYPE_DEVTLB,
+	CACHE_TAG_TYPE_PARENT_IOTLB,
+	CACHE_TAG_TYPE_PARENT_DEVTLB,
+};

'_TYPE_' can be removed to make it shorter

Okay.

Quoted 8 lines

drivers/iommu/intel/cache.c line 23

Quoted 6 lines
drivers/iommu/intel/cache.c new file +5 -0
+
+/* Checks if an existing cache tag can be reused for a new association. */
+static bool cache_tag_reusable(struct cache_tag *tag, u16 domain_id,
+			       struct intel_iommu *iommu, struct device *dev,
+			       ioasid_t pasid, enum cache_tag_type type)

cache_tage_match()

Okay.

Quoted 12 lines
Quoted 12 lines
drivers/iommu/intel/cache.c new file +11 -0
+{
+	if (tag->type != type)
+		return false;
+
+	if (tag->domain_id != domain_id || tag->pasid != pasid)
+		return false;
+
+	if (type == CACHE_TAG_TYPE_IOTLB)
+		return tag->iommu == iommu;
+
+	if (type == CACHE_TAG_TYPE_DEVTLB)
Quoted 8 lines

drivers/iommu/intel/cache.c line 37

Quoted 3 lines
drivers/iommu/intel/cache.c new file +3 -0
+		return tag->dev == dev;
+
+	return false;

why do you disallow PARENT_TYPE from reusing? It's not uncommon to have two devices attached to a same nested domain hence with the same parent domain. Disallowing tag reuse implies unnecessarily duplicated cache flushes...

PARENT_TYPE could be reused. The new helper looks like the following:

/* Checks if an existing cache tag can be reused for a new association. */ static bool cache_tage_match(struct cache_tag *tag, u16 domain_id,
struct intel_iommu *iommu, struct device
*dev,
ioasid_t pasid, enum cache_tag_type type)

{
        if (tag->type != type)
                return false;

        if (tag->domain_id != domain_id || tag->pasid != pasid)
                return false;

        if (type == CACHE_TAG_IOTLB || type == CACHE_TAG_PARENT_IOTLB)
                return tag->iommu == iommu;

        if (type == CACHE_TAG_DEVTLB || type == CACHE_TAG_PARENT_DEVTLB)
                return tag->dev == dev;

        return false;
}
Quoted 17 lines
Quoted 17 lines
drivers/iommu/intel/cache.c new file +17 -0
+}
+
+/* Assign a cache tag with specified type to domain. */
+static int cache_tag_assign(struct dmar_domain *domain, u16 did,
+			    struct device *dev, ioasid_t pasid,
+			    enum cache_tag_type type)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+	struct intel_iommu *iommu = info->iommu;
+	struct cache_tag *tag, *temp;
+	unsigned long flags;
+
+	tag = kzalloc(sizeof(*tag), GFP_KERNEL);
+	if (!tag)
+		return -ENOMEM;
+
+	tag->type = type;
Quoted 8 lines

drivers/iommu/intel/cache.c line 56

Quoted 2 lines
drivers/iommu/intel/cache.c new file +2 -0
+	tag->iommu = iommu;
+	tag->dev = dev;

should we set tag->dev only for DEVTLB type? It's a bit confusing to set it for IOTLB type which doesn't care about device. Actually doing so is instead misleading as the 1st device creating the tag may have been detached but then it will still show up in the trace when the last device detach destroying the tag.

For IOTLB types, perhaps we could add a struct device pointer for the iommu. This way, the tag->dev could more directly indicate the device implementing the cache.

Quoted 9 lines

Review

Quoted 4 lines

+static int __cache_tag_assign_parent_domain(struct dmar_domain *domain, u16 did,

(quoted diff) +1 -0
+					    struct device *dev, ioasid_t pasid)

this pair is similar to the earlier one except the difference on type.

what about keeping just one pair which accepts a 'parent' argument to decide the type internally?

Okay, let try to refine it.

Quoted 7 lines
Quoted 5 lines

+/*
+ * Assigns cache tags to a domain when it's associated with a device's
+ * PASID using a specific domain ID.

s/Assigns/Assign/

Done.

Quoted 9 lines
Quoted 5 lines
drivers/iommu/intel/iommu.c +3 -0
+
+	did = domain_id_iommu(domain, iommu);
+	ret = cache_tag_assign_domain(domain, did, dev,

IOMMU_NO_PASID);

there are many occurrences of this pattern. What about passing in a 'iommu' parameter and getting 'did' inside the helper? for svm it can be specialized internally too.

Perhaps, let me try it later and see what the code looks like.

Quoted 11 lines

drivers/iommu/intel/iommu.c static void · line 4628

Quoted 8 lines
drivers/iommu/intel/iommu.c +2 -0
@@ -4607,10 +4623,11 @@ static void
 intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
  	 */
  	if (domain->type == IOMMU_DOMAIN_SVA) {
  		intel_svm_remove_dev_pasid(dev, pasid);
+		cache_tag_unassign_domain(dmar_domain,
+					  FLPT_DEFAULT_DID, dev, pasid);

is it correct to destroy the tag before teardown completes, e.g. iotlb still needs to be flushed in intel_pasid_tear_down_entry()?

You are right. iotlb still needs to be there until the teardown completes. I will investigate this more later.

Beset regards,
baolu

author maintainer
#4 lore raw

Re: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

Quoted 12 lines · Baolu Lu
Quoted 11 lines

drivers/iommu/intel/iommu.c static void · line 4628

Quoted 8 lines
drivers/iommu/intel/iommu.c +2 -0
@@ -4607,10 +4623,11 @@ static void
 intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
        */
       if (domain->type == IOMMU_DOMAIN_SVA) {
           intel_svm_remove_dev_pasid(dev, pasid);
+        cache_tag_unassign_domain(dmar_domain,
+                      FLPT_DEFAULT_DID, dev, pasid);

is it correct to destroy the tag before teardown completes, e.g. iotlb still needs to be flushed in intel_pasid_tear_down_entry()?

Quoted 2 lines

You are right. iotlb still needs to be there until the teardown completes. I will investigate this more later.

I reviewed this again. Cache tags are designed specifically for mapping and unmapping paths. Therefore, there is no required order for attaching and detaching paths.

Best regards,
baolu

author maintainer
#15 lore raw

Re: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers

In reply to Tian, Kevin
Quoted 13 lines · Kevin
Quoted 11 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

drivers/iommu/intel/cache.c +7 -0
+
+static unsigned long calculate_psi_aligned_address(unsigned long start,
+						   unsigned long end,
+						   unsigned long *_pages,
+						   unsigned long *_mask)
+{
+	unsigned long pages = (end - start + VTD_PAGE_SIZE - 1) >>

VTD_PAGE_SHIFT;

this doesn't sound correct. You'd want to follow aligned_nrpages().

Okay, I will make it like below.

	unsigned long pages = aligned_nrpages(start, end - start + 1);
Quoted 36 lines
Quoted 31 lines
drivers/iommu/intel/cache.c +4 -0
+		case CACHE_TAG_TYPE_DEVTLB:
+			if (tag->pasid == IOMMU_NO_PASID)
+				qi_flush_dev_iotlb(iommu, sid, info->pfsid,
+						   info->ats_qdep, addr,

mask);

(quoted diff) +2 -0
+			else
+				qi_flush_dev_iotlb_pasid(iommu, sid, info-
Quoted 1 line

pfsid,

(quoted diff) +1 -0
+							 tag->pasid, info-
Quoted 1 line

ats_qdep,

drivers/iommu/intel/cache.c +3 -0
+							 addr, mask);
+
+			quirk_extra_dev_tlb_flush(info, addr, mask, tag-
Quoted 1 line

pasid, info->ats_qdep);

drivers/iommu/intel/cache.c +10 -0
+			break;
+		case CACHE_TAG_TYPE_PARENT_DEVTLB:
+			/*
+			 * Address translation cache in device side caches the
+			 * result of nested translation. There is no easy way
+			 * to identify the exact set of nested translations
+			 * affected by a change in S2. So just flush the entire
+			 * device cache.
+			 */
+			qi_flush_dev_iotlb(iommu, sid, info->pfsid, info-
Quoted 1 line

ats_qdep,

(quoted diff) +2 -0
+					   0, MAX_AGAW_PFN_WIDTH);
+			quirk_extra_dev_tlb_flush(info, 0,

MAX_AGAW_PFN_WIDTH,

(quoted diff) +1 -0
+						  IOMMU_NO_PASID, info-
Quoted 1 line

ats_qdep);

Review

Quoted 1 line
(quoted diff) +1 -0
+			break;

parent devtlb can has pasid too, though not enabled now. As core helpers probably we can support it now leading to simpler code:

Quoted 5 lines
	case CACHE_TAG_TYPE_PARENT_DEVTLB:
		//change addr/mask
		//fallthrough
	case CACHE_TAG_TYPE_DEVTLB:
		//what this patch does

Yes. Yours is better. I will make it like below:

+ case CACHE_TAG_PARENT_DEVTLB:
+ /*

+                        * Address translation cache in device side caches the
+                        * result of nested translation. There is no easy way
+                        * to identify the exact set of nested translations
+                        * affected by a change in S2. So just flush the entire
+                        * device cache.
+                        */
+                       addr = 0;
+                       mask = MAX_AGAW_PFN_WIDTH;
+                       fallthrough;

case CACHE_TAG_DEVTLB:

Quoted 40 lines
Quoted 33 lines
drivers/iommu/intel/cache.c +8 -0
+void cache_tag_flush_all(struct dmar_domain *domain)
+{
+	struct cache_tag *tag;
+	unsigned long flags;
+
+	spin_lock_irqsave(&domain->cache_lock, flags);
+	list_for_each_entry(tag, &domain->cache_tags, node) {
+		struct device_domain_info *info = dev_iommu_priv_get(tag-
Quoted 1 line

dev);

drivers/iommu/intel/cache.c +11 -0
+		struct intel_iommu *iommu = tag->iommu;
+		u16 sid = PCI_DEVID(info->bus, info->devfn);
+
+		switch (tag->type) {
+		case CACHE_TAG_TYPE_IOTLB:
+		case CACHE_TAG_TYPE_PARENT_IOTLB:
+			if (domain->use_first_level)
+				qi_flush_piotlb(iommu, tag->domain_id,
+						tag->pasid, 0, -1, 0);
+			else
+				iommu->flush.flush_iotlb(iommu, tag-
Quoted 1 line

domain_id,

(quoted diff) +1 -0
+							 0, 0,

DMA_TLB_DSI_FLUSH);

drivers/iommu/intel/cache.c +4 -0
+			break;
+		case CACHE_TAG_TYPE_DEVTLB:
+		case CACHE_TAG_TYPE_PARENT_DEVTLB:
+			qi_flush_dev_iotlb(iommu, sid, info->pfsid, info-
Quoted 1 line

ats_qdep,

(quoted diff) +2 -0
+					   0, MAX_AGAW_PFN_WIDTH);
+			quirk_extra_dev_tlb_flush(info, 0,

MAX_AGAW_PFN_WIDTH,

(quoted diff) +1 -0
+						  IOMMU_NO_PASID, info-
Quoted 1 line

ats_qdep);

drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 296

Quoted 3 lines
drivers/iommu/intel/cache.c +3 -0
+			break;
+		}
+	}

could this loop be consolidated with the one in previous helper? anyway the only difference is on addr/mask...

Quoted 12 lines

drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 343

Quoted 9 lines

+/*
+ * Invalidate a range of IOVA when IOMMU is in caching mode and new mappings
+ * are added to the target domain.
+ */
+void cache_tag_flush_cm_range(struct dmar_domain *domain, unsigned long start,

drivers/iommu/intel/cache.c +2 -0
+			      unsigned long end)
+{

I'm also not sure why this is worth a separate helper. why couldn't it be managed by previous flush_range()?

This is only my preference. I'd like to separate things belonging to different paths, so that it's easier for maintenance. For example, if, in the future, we need to add or enhance something for a specific case, we don't need to care about other cases.

Quoted 18 lines
Quoted 18 lines
drivers/iommu/intel/cache.c +12 -0
+	unsigned long pages, mask, addr;
+	struct cache_tag *tag;
+	unsigned long flags;
+
+	addr = calculate_psi_aligned_address(start, end, &pages, &mask);
+
+	spin_lock_irqsave(&domain->cache_lock, flags);
+	list_for_each_entry(tag, &domain->cache_tags, node) {
+		struct intel_iommu *iommu = tag->iommu;
+
+		/*
+		 * When IOMMU is enabled in caching mode some non-

present

drivers/iommu/intel/cache.c +4 -0
+		 * mappings may be cached by the IOTLB if it uses second-
+		 * only translation.
+		 */
+		if (!cap_caching_mode(iommu->cap) || domain-
Quoted 7 lines

drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 362

Quoted 4 lines
Quoted 1 line

use_first_level) {

drivers/iommu/intel/cache.c +3 -0
+			iommu_flush_write_buffer(iommu);
+			continue;
+		}

the comment really doesn't tell what this block does. from its intention it probably more suitable to be in the caller side as today.

Yes, agreed. Will remove it.

Quoted 26 lines
Quoted 20 lines
drivers/iommu/intel/cache.c +10 -0
+
+		if (tag->type == CACHE_TAG_TYPE_IOTLB ||
+		    tag->type == CACHE_TAG_TYPE_PARENT_IOTLB) {
+			/*
+			 * Fallback to domain selective flush if no
+			 * PSI support or the size is too big.
+			 */
+			if (!cap_pgsel_inv(iommu->cap) ||
+			    mask > cap_max_amask_val(iommu->cap))
+				iommu->flush.flush_iotlb(iommu, tag-
Quoted 1 line

domain_id,

(quoted diff) +1 -0
+							 0, 0,

DMA_TLB_DSI_FLUSH);

(quoted diff) +2 -0
+			else
+				iommu->flush.flush_iotlb(iommu, tag-
Quoted 1 line

domain_id,

(quoted diff) +2 -0
+							 addr, mask,
+

DMA_TLB_PSI_FLUSH);

Review

Quoted 2 lines
(quoted diff) +2 -0
+		}
+	}

You skipped devtlb invalidation. yes it's not necessary based on current nested translation design and this part is inconsistent in different paths.

Quoted 2 lines

but as a semantics change you may want to first make removing devtlb invalidation a separate patch and then do this cleanup, so bisect is easier.

This helper is designed for map paths when the IOMMU is in caching mode. Caching mode doesn't impact the device TLB, so we should not invalidate the device TLB here.

I guess the confusing thing is about the code below.

/*
* In caching mode, changes of pages from non-present to
present require
* flush. However, device IOTLB doesn't need to be flushed in
this case.

         */
        if (!cap_caching_mode(iommu->cap) || !map)
                iommu_flush_dev_iotlb(domain, addr, mask);

The real code doesn't match the comments, and I think the comments describe the right thing. So perhaps the code should be changed to match the comments.

	if (!map)
		iommu_flush_dev_iotlb(domain, addr, mask);

??

Best regards,
baolu

author maintainer
#22 lore raw

Re: [PATCH 04/12] iommu/vt-d: Use cache_tag_flush_all() in flush_iotlb_all

In reply to Tian, Kevin
Quoted 41 lines · Kevin
Quoted 41 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

The flush_iotlb_all callback is called by the iommu core to flush all caches for the affected domain. Use cache_tag_flush_all() in this callback.

---
  drivers/iommu/intel/iommu.c | 21 +--------------------
  1 file changed, 1 insertion(+), 20 deletions(-)
drivers/iommu/intel/iommu.c +0 -19
index 93e4422c9b10..4ce98f23917c 100644
@@ -1560,26 +1560,7 @@ static void parent_domain_flush(struct
 dmar_domain *domain,
 
  static void intel_flush_iotlb_all(struct iommu_domain *domain)
  {
-	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
-	struct iommu_domain_info *info;
-	unsigned long idx;
-
-	xa_for_each(&dmar_domain->iommu_array, idx, info) {
-		struct intel_iommu *iommu = info->iommu;
-		u16 did = domain_id_iommu(dmar_domain, iommu);
-
-		if (dmar_domain->use_first_level)
-			domain_flush_pasid_iotlb(iommu, dmar_domain, 0,
-1, 0);
-		else
-			iommu->flush.flush_iotlb(iommu, did, 0, 0,
-						 DMA_TLB_DSI_FLUSH);
-
-		if (!cap_caching_mode(iommu->cap))
-			iommu_flush_dev_iotlb(dmar_domain, 0,
 MAX_AGAW_PFN_WIDTH);
-	}
-
Quoted 8 lines

drivers/iommu/intel/iommu.c static void parent_domain_flush(struct dmar_domain *domain, · line 1564

Quoted 5 lines
drivers/iommu/intel/iommu.c +1 -2
-	if (dmar_domain->nested_parent)
-		parent_domain_flush(dmar_domain, 0, -1, 0);
+	cache_tag_flush_all(to_dmar_domain(domain));
  }

this replacement causes a functional change. Now devtlb is always invalidated while old code doesn't do so for caching mode.

You are right.

As my understanding of the VT-d spec, caching mode has nothing to do with the device TLB, hence we should remove all checks on caching mode before a device TLB invalidation.

I understand that people are leveraging the cache mode to avoid the unnecessary VMEXIT from the guest OS since the host iommu unmap() path ensures that device TLB is also invalidated.

Our direction should be to use the batched queue invalidation requests. As we are consolidating code to a common place, that becomes much easier. :-)

Quoted 2 lines

Probably you may want to first clean up all inconsistent devtlb invalidation policies for caching mode before going to this series...

Sure thing!

Best regards,
baolu

author maintainer
#26 lore raw

Re: [PATCH 06/12] iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map

In reply to Tian, Kevin
Quoted 32 lines · Kevin
Quoted 32 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

The iotlb_sync_map callback is called by the iommu core after non-present to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Use cache_tag_flush_cm_range() in this callback.

---
  drivers/iommu/intel/iommu.c | 22 +---------------------
  1 file changed, 1 insertion(+), 21 deletions(-)
drivers/iommu/intel/iommu.c +0 -8
index 1c03d2dafb9d..2dcab1e5cd4d 100644
@@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(struct
 intel_iommu *iommu,
  		iommu_flush_dev_iotlb(domain, addr, mask);
  }
 
-/* Notification for newly created mappings */
-static void __mapping_notify_one(struct intel_iommu *iommu, struct
 dmar_domain *domain,
-				 unsigned long pfn, unsigned int pages)
-{
-	/*
-	 * It's a non-present to present mapping. Only flush if caching mode
-	 * and second level.
-	 */
Quoted 8 lines

drivers/iommu/intel/iommu.c static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, · line 1519

Quoted 5 lines
drivers/iommu/intel/iommu.c +0 -5
-	if (cap_caching_mode(iommu->cap) && !domain->use_first_level)
-		iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1);
-	else
-		iommu_flush_write_buffer(iommu);
-}

iommu_flush_write_buffer is for a quite different issue. it's clearer to keep it separated from the iotlb helpers.

The VT-d spec describes the write buffer flushing in section 6.8. It states,

1. Write buffer flushing is required only for earlier VT-d hardware implementations. Newer hardware implementations are expected to NOT require this.
2. Write buffer flushing is related to cache invalidation. The hardware performs an implicit write-buffer-flushing as a pre-condition to the invalidation operations.

According to this, write buffer flushing is only required in the map path when IOMMU is not in caching mode and the RWBF iommu capability is set, which perfectly matches the case of cache_tag_flush_cm_range().

By doing it in cache_tag_flush_cm_range(), the code will be simpler and easier for maintenance.

Best regards,
baolu

author maintainer
#31 lore raw

Re: [PATCH 07/12] iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi()

In reply to Tian, Kevin
Quoted 11 lines · Kevin
Quoted 9 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

Use cache_tag_flush_range() in switch_to_super_page() to invalidate the necessary caches when switching mappings from normal to super pages. The iommu_flush_iotlb_psi() call in intel_iommu_memory_notifier() is unnecessary since there should be no cache invalidation for the identity domain.

what about a software identity domain?

Software identity domain is used to fake the hardware passthrough capability, on early VT-d hardware which doesn't implement the passthrough mode. It's not any kind of protection domain, hence the OS is not required to manage the cache synchronization.

Although I hope we can remove it someday and force the DMA domain instead, we still need to carry it nowadays. However, we need to make it consistent with the hardware passthrough. That is, hardware passthrough doesn't require any cache invalidation in memory hot-plug paths, the software passthrough should not either.

Best regards,
baolu

author maintainer
#35 lore raw

Re: [PATCH 08/12] iommu/vt-d: Use cache_tag_flush_range() in cache_invalidate_user

In reply to Tian, Kevin
Quoted 8 lines · Kevin
Quoted 8 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

(quoted diff) +0 -0
@@ -166,9 +122,9 @@ static int intel_nested_cache_invalidate_user(struct
 iommu_domain *domain,
  			break;
  		}
 
Quoted 10 lines

Review static int intel_nested_cache_invalidate_user(struct · line 130

Quoted 8 lines
(quoted diff) +3 -3
-		intel_nested_flush_cache(dmar_domain, inv_entry.addr,
-					 inv_entry.npages,
-					 inv_entry.flags &
 IOMMU_VTD_INV_FLAGS_LEAF);
+		cache_tag_flush_range(dmar_domain, inv_entry.addr,
+				      inv_entry.npages,
+				      inv_entry.flags &
 IOMMU_VTD_INV_FLAGS_LEAF);

the helper requires an 'end' address but 'npages' is incorrectly used here.

Oh yes! Thanks for pointing this out. I will fix it like below.

drivers/iommu/intel/iommu.h +6 -0
index c139524274b9..cef997c07158 100644
@@ -1050,6 +1050,12 @@ static inline unsigned long aligned_nrpages(unsigned long host_addr, size_t size
        return PAGE_ALIGN(host_addr + size) >> VTD_PAGE_SHIFT;
 }
 
+/* Return a size from number of VTD pages. */
+static inline unsigned long nrpages_to_size(unsigned long npages)
+{
+       return npages << VTD_PAGE_SHIFT;
+}
+
 /* Convert value to context PASID directory size field coding. */
 #define context_pdts(pds)      (((pds) & 0x7) << 9)
 
drivers/iommu/intel/nested.c +1 -1
index e251507cfcc0..ffbd8d98a3b8 100644
@@ -123,7 +123,7 @@ static int intel_nested_cache_invalidate_user(struct iommu_domain *domain,
                }
 
                cache_tag_flush_range(dmar_domain, inv_entry.addr,
-                                     inv_entry.npages,
+                                     inv_entry.addr + nrpages_to_size(inv_entry.npages) - 1,
                                      inv_entry.flags & 
IOMMU_VTD_INV_FLAGS_LEAF);
                processed++;
        }

Best regards,
baolu

author maintainer
#41 lore raw
In reply to Tian, Kevin
Quoted 24 lines · Kevin
Quoted 24 lines

From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

The IOMMU hardware cache needs to be invalidated whenever the mappings
in the domain are changed. Currently, domain cache invalidation is scattered across different places, causing several issues:

- IOMMU IOTLB Invalidation: This is done by iterating through the domain IDs of each domain using the following code:

         xa_for_each(&dmar_domain->iommu_array, i, info)
                 iommu_flush_iotlb_psi(info->iommu, dmar_domain,
                                       start_pfn, nrpages,
                                       list_empty(&gather->freelist), 0);

This code could theoretically cause a use-after-free problem because there's no lock to protect the "info" pointer within the loop.

- Inconsistent Invalidation Methods: Different domain types implement their own cache invalidation methods, making the code difficult to maintain. For example, the DMA domain, SVA domain, and nested domain have similar cache invalidation code scattered across different files.

Quoted 6 lines
Quoted 5 lines

- SVA Domain Inconsistency: The SVA domain implementation uses a completely different data structure to track attached devices compared to other domains. This creates unnecessary differences and, even worse, leads to duplicate IOTLB invalidation when an SVA domain is attached to devices belonging to different IOMMU domains.

can you elaborate how duplicated invalidations are caused?

Yes, sure.

Current Intel SVA implementation keeps the bond between mm and a PASID of a device in a list of intel_svm_dev. In the mm notifier callback, it iterates all intel_svam_dev in the list and invalidates the IOTLB and device TLB sequentially.

If multiple devices belong to a single IOMMU, the IOTLB will be flushed multiple times. However, since these devices share the same domain ID and PASID, a single IOTLB cache invalidation is sufficient. The additional flushes are redundant and negatively impact performance.

Best regards,
baolu

#5 lore raw

RE: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

In reply to Baolu Lu
Quoted 17 lines

From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 12:35 PM

Quoted 13 lines · Baolu Lu
Quoted 12 lines

drivers/iommu/intel/iommu.c static void · line 4628

Quoted 8 lines
drivers/iommu/intel/iommu.c +2 -0
@@ -4607,10 +4623,11 @@ static void
 intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
        */
       if (domain->type == IOMMU_DOMAIN_SVA) {
           intel_svm_remove_dev_pasid(dev, pasid);
+        cache_tag_unassign_domain(dmar_domain,
+                      FLPT_DEFAULT_DID, dev, pasid);

is it correct to destroy the tag before teardown completes, e.g. iotlb still
needs to be flushed in intel_pasid_tear_down_entry()?

Quoted 7 lines
Quoted 2 lines

You are right. iotlb still needs to be there until the teardown completes. I will investigate this more later.

I reviewed this again. Cache tags are designed specifically for mapping and unmapping paths. Therefore, there is no required order for attaching and detaching paths.

Okay. intel_pasid_tear_down_entry() directly retrieves the information from the pasid entry instead of relying on the domain cache tag info. so yes destroying the tag at this point is fine.

#16 lore raw

RE: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers

In reply to Baolu Lu
Quoted 19 lines

From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 1:33 PM

Quoted 15 lines · Kevin

drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 343

Quoted 12 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

+/*
+ * Invalidate a range of IOVA when IOMMU is in caching mode and new mappings
+ * are added to the target domain.
+ */
+void cache_tag_flush_cm_range(struct dmar_domain *domain, unsigned long start,

drivers/iommu/intel/cache.c +2 -0
+			      unsigned long end)
+{

I'm also not sure why this is worth a separate helper. why couldn't it be managed by previous flush_range()?

Quoted 4 lines

This is only my preference. I'd like to separate things belonging to different paths, so that it's easier for maintenance. For example, if, in the future, we need to add or enhance something for a specific case, we don't need to care about other cases.

IMHO caching mode is an attribute in low level iommu which can be handled perfectly well within the helper by checking that attribute.

it sounds a bit weird for the caller to know that detail and call different helpers when all paths just want to request to flush a specific range.

Quoted 44 lines
Quoted 25 lines
Quoted 19 lines
drivers/iommu/intel/cache.c +10 -0
+
+		if (tag->type == CACHE_TAG_TYPE_IOTLB ||
+		    tag->type == CACHE_TAG_TYPE_PARENT_IOTLB) {
+			/*
+			 * Fallback to domain selective flush if no
+			 * PSI support or the size is too big.
+			 */
+			if (!cap_pgsel_inv(iommu->cap) ||
+			    mask > cap_max_amask_val(iommu->cap))
+				iommu->flush.flush_iotlb(iommu, tag-
Quoted 1 line

domain_id,

(quoted diff) +1 -0
+							 0, 0,

DMA_TLB_DSI_FLUSH);

(quoted diff) +2 -0
+			else
+				iommu->flush.flush_iotlb(iommu, tag-
Quoted 1 line

domain_id,

(quoted diff) +2 -0
+							 addr, mask,
+

DMA_TLB_PSI_FLUSH);

Review

Quoted 2 lines
(quoted diff) +2 -0
+		}
+	}

You skipped devtlb invalidation. yes it's not necessary based on current nested translation design and this part is inconsistent in different paths.

Quoted 2 lines

but as a semantics change you may want to first make removing devtlb invalidation a separate patch and then do this cleanup, so bisect is easier.

This helper is designed for map paths when the IOMMU is in caching mode. Caching mode doesn't impact the device TLB, so we should not invalidate the device TLB here.

I guess the confusing thing is about the code below.

/*
* In caching mode, changes of pages from non-present to
present require
* flush. However, device IOTLB doesn't need to be flushed in
this case.

          */
         if (!cap_caching_mode(iommu->cap) || !map)
                 iommu_flush_dev_iotlb(domain, addr, mask);
Quoted 9 lines

The real code doesn't match the comments, and I think the comments describe the right thing. So perhaps the code should be changed to match the comments.

	if (!map)
		iommu_flush_dev_iotlb(domain, addr, mask);

??

yes. that should be fixed separately.

#27 lore raw

RE: [PATCH 06/12] iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map

In reply to Baolu Lu
Quoted 57 lines

From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 2:41 PM

Quoted 4 lines · Kevin
Quoted 4 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

The iotlb_sync_map callback is called by the iommu core after non-

present

Quoted 3 lines
Quoted 3 lines

to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Use cache_tag_flush_cm_range() in

this

Quoted 25 lines
Quoted 25 lines

callback.

---
  drivers/iommu/intel/iommu.c | 22 +---------------------
  1 file changed, 1 insertion(+), 21 deletions(-)
drivers/iommu/intel/iommu.c +0 -8
index 1c03d2dafb9d..2dcab1e5cd4d 100644
@@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(struct
 intel_iommu *iommu,
  		iommu_flush_dev_iotlb(domain, addr, mask);
  }
 
-/* Notification for newly created mappings */
-static void __mapping_notify_one(struct intel_iommu *iommu, struct
 dmar_domain *domain,
-				 unsigned long pfn, unsigned int pages)
-{
-	/*
-	 * It's a non-present to present mapping. Only flush if caching mode
-	 * and second level.
-	 */
Quoted 8 lines

drivers/iommu/intel/iommu.c static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, · line 1519

Quoted 5 lines
drivers/iommu/intel/iommu.c +0 -5
-	if (cap_caching_mode(iommu->cap) && !domain->use_first_level)
-		iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1);
-	else
-		iommu_flush_write_buffer(iommu);
-}

iommu_flush_write_buffer is for a quite different issue. it's clearer to keep it separated from the iotlb helpers.

The VT-d spec describes the write buffer flushing in section 6.8. It states,

1. Write buffer flushing is required only for earlier VT-d hardware implementations. Newer hardware implementations are expected to NOT require this.
2. Write buffer flushing is related to cache invalidation. The hardware performs an implicit write-buffer-flushing as a pre-condition to the invalidation operations.

Quoted 7 lines

According to this, write buffer flushing is only required in the map path when IOMMU is not in caching mode and the RWBF iommu capability is set, which perfectly matches the case of cache_tag_flush_cm_range().

By doing it in cache_tag_flush_cm_range(), the code will be simpler and easier for maintenance.

but above applies actually when caching mode is false or when 1st stage is being used. this is the opposite of what flush_cm_range() implies.

if we do it through a general flush_range() helper which accepts a parameter to indicate the map path, then it's reasonable to move this special handling to that helper as that is another trick which the caller doesn't need to know.

author maintainer
#17 lore raw

Re: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers

In reply to Tian, Kevin
Quoted 22 lines · Kevin
Quoted 18 lines

From: Baolu Lu<[email protected]>
Sent: Sunday, April 7, 2024 1:33 PM

Quoted 14 lines · Kevin

drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 343

Quoted 12 lines

From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

+/*
+ * Invalidate a range of IOVA when IOMMU is in caching mode and new mappings
+ * are added to the target domain.
+ */
+void cache_tag_flush_cm_range(struct dmar_domain *domain, unsigned long start,

drivers/iommu/intel/cache.c +2 -0
+			      unsigned long end)
+{

I'm also not sure why this is worth a separate helper. why couldn't it be managed by previous flush_range()?

Quoted 4 lines

This is only my preference. I'd like to separate things belonging to different paths, so that it's easier for maintenance. For example, if, in the future, we need to add or enhance something for a specific case, we don't need to care about other cases.

Quoted 5 lines

IMHO caching mode is an attribute in low level iommu which can be handled perfectly well within the helper by checking that attribute.

it sounds a bit weird for the caller to know that detail and call different helpers when all paths just want to request to flush a specific range.

I see. The helper name is a bit confusing.

Generally speaking, cache_tag_flush_range() and cache_tag_flush_all() are designed to flush caches for mapping change from present to non- present. While cache_tag_flush_cm_range() is designed to flush caches for mapping change from non-present to present.

How about renaming these helpers to

cache_tag_flush_present_range()
cache_tag_flush_present_all()
cache_tag_flush_non_present_range()

?

In cache_tag_flush_non_present_range(),

- if IOMMU is not in caching mode, flush the write buffer if necessary, or it's a no-op.
- if IOMMU is in caching mode, flush the IOTLB caches.

Best regards,
baolu

#32 lore raw

RE: [PATCH 07/12] iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi()

In reply to Baolu Lu
Quoted 22 lines

From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 3:06 PM

Quoted 5 lines · Kevin
Quoted 5 lines

From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

Use cache_tag_flush_range() in switch_to_super_page() to invalidate the necessary caches when switching mappings from normal to super pages.

The

Quoted 6 lines
Quoted 4 lines

iommu_flush_iotlb_psi() call in intel_iommu_memory_notifier() is unnecessary since there should be no cache invalidation for the identity domain.

what about a software identity domain?

Software identity domain is used to fake the hardware passthrough capability, on early VT-d hardware which doesn't implement the passthrough mode. It's not any kind of protection domain, hence the OS is not required to manage the cache synchronization.

Quoted 6 lines

Although I hope we can remove it someday and force the DMA domain instead, we still need to carry it nowadays. However, we need to make it consistent with the hardware passthrough. That is, hardware passthrough doesn't require any cache invalidation in memory hot-plug paths, the software passthrough should not either.

yes, that makes sense.

author maintainer
#28 lore raw

Re: [PATCH 06/12] iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map

In reply to Tian, Kevin
Quoted 62 lines · Kevin
Quoted 55 lines

From: Baolu Lu<[email protected]>
Sent: Sunday, April 7, 2024 2:41 PM

Quoted 4 lines · Kevin
Quoted 4 lines

From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

The iotlb_sync_map callback is called by the iommu core after non-

present

Quoted 3 lines
Quoted 3 lines

to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Use cache_tag_flush_cm_range() in

this

Quoted 16 lines
Quoted 16 lines

callback.

---
   drivers/iommu/intel/iommu.c | 22 +---------------------
   1 file changed, 1 insertion(+), 21 deletions(-)
drivers/iommu/intel/iommu.c +0 -0
index 1c03d2dafb9d..2dcab1e5cd4d 100644
@@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(struct
 intel_iommu *iommu,
   		iommu_flush_dev_iotlb(domain, addr, mask);
   }
 
Quoted 16 lines
Quoted 4 lines
drivers/iommu/intel/iommu.c +0 -3
-/* Notification for newly created mappings */
-static void __mapping_notify_one(struct intel_iommu *iommu, struct
 dmar_domain *domain,
-				 unsigned long pfn, unsigned int pages)

drivers/iommu/intel/iommu.c static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, · line 1519

Quoted 10 lines
drivers/iommu/intel/iommu.c +0 -10
-{
-	/*
-	 * It's a non-present to present mapping. Only flush if caching mode
-	 * and second level.
-	 */
-	if (cap_caching_mode(iommu->cap) && !domain->use_first_level)
-		iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1);
-	else
-		iommu_flush_write_buffer(iommu);
-}

iommu_flush_write_buffer is for a quite different issue. it's clearer to keep it separated from the iotlb helpers.

The VT-d spec describes the write buffer flushing in section 6.8. It states,

1. Write buffer flushing is required only for earlier VT-d hardware implementations. Newer hardware implementations are expected to NOT require this.
2. Write buffer flushing is related to cache invalidation. The hardware performs an implicit write-buffer-flushing as a pre-condition to the invalidation operations.

Quoted 7 lines

According to this, write buffer flushing is only required in the map path when IOMMU is not in caching mode and the RWBF iommu capability is set, which perfectly matches the case of cache_tag_flush_cm_range().

By doing it in cache_tag_flush_cm_range(), the code will be simpler and easier for maintenance.

Quoted 8 lines

but above applies actually when caching mode is false or when 1st stage is being used. this is the opposite of what flush_cm_range() implies.

if we do it through a general flush_range() helper which accepts a parameter to indicate the map path, then it's reasonable to move this special handling to that helper as that is another trick which the caller doesn't need to know.

This could be an option. How about my proposal in another reply?

Best regards,
baolu

#42 lore raw
In reply to Baolu Lu
Quoted 43 lines

From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 3:28 PM

Quoted 16 lines · Kevin
Quoted 16 lines

From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

The IOMMU hardware cache needs to be invalidated whenever the mappings
in the domain are changed. Currently, domain cache invalidation is scattered across different places, causing several issues:

- IOMMU IOTLB Invalidation: This is done by iterating through the domain IDs of each domain using the following code:

         xa_for_each(&dmar_domain->iommu_array, i, info)
                 iommu_flush_iotlb_psi(info->iommu, dmar_domain,
                                       start_pfn, nrpages,
                                       list_empty(&gather->freelist), 0);
Quoted 6 lines
Quoted 6 lines

This code could theoretically cause a use-after-free problem because there's no lock to protect the "info" pointer within the loop.

- Inconsistent Invalidation Methods: Different domain types implement their own cache invalidation methods, making the code difficult to maintain. For example, the DMA domain, SVA domain, and nested

domain

Quoted 8 lines
Quoted 7 lines

have similar cache invalidation code scattered across different files.

- SVA Domain Inconsistency: The SVA domain implementation uses a completely different data structure to track attached devices compared to other domains. This creates unnecessary differences and, even worse, leads to duplicate IOTLB invalidation when an SVA domain is attached to devices belonging to different IOMMU domains.

can you elaborate how duplicated invalidations are caused?

Yes, sure.

Current Intel SVA implementation keeps the bond between mm and a PASID of a device in a list of intel_svm_dev. In the mm notifier callback, it iterates all intel_svam_dev in the list and invalidates the IOTLB and device TLB sequentially.

Quoted 5 lines

If multiple devices belong to a single IOMMU, the IOTLB will be flushed multiple times. However, since these devices share the same domain ID and PASID, a single IOTLB cache invalidation is sufficient. The additional flushes are redundant and negatively impact performance.

yes it's redundant. But what does "devices belonging to different IOMMU domains" in the original context try to convey? From above explanation it sounds irrelevant...

author maintainer
#43 lore raw
Quoted 49 lines · Kevin
Quoted 43 lines

From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 3:28 PM

Quoted 16 lines · Kevin
Quoted 16 lines

From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

The IOMMU hardware cache needs to be invalidated whenever the mappings
in the domain are changed. Currently, domain cache invalidation is scattered across different places, causing several issues:

- IOMMU IOTLB Invalidation: This is done by iterating through the domain

    IDs of each domain using the following code:

          xa_for_each(&dmar_domain->iommu_array, i, info)
                  iommu_flush_iotlb_psi(info->iommu, dmar_domain,
                                        start_pfn, nrpages,
                                        list_empty(&gather->freelist), 0);
Quoted 6 lines
Quoted 6 lines

This code could theoretically cause a use-after-free problem because there's no lock to protect the "info" pointer within the loop.

- Inconsistent Invalidation Methods: Different domain types implement their own cache invalidation methods, making the code difficult to maintain. For example, the DMA domain, SVA domain, and nested

domain

Quoted 8 lines
Quoted 7 lines

have similar cache invalidation code scattered across different files.

- SVA Domain Inconsistency: The SVA domain implementation uses a completely different data structure to track attached devices compared to other domains. This creates unnecessary differences and, even worse, leads to duplicate IOTLB invalidation when an SVA domain is attached to devices belonging to different IOMMU domains.

can you elaborate how duplicated invalidations are caused?

Yes, sure.

Current Intel SVA implementation keeps the bond between mm and a PASID of a device in a list of intel_svm_dev. In the mm notifier callback, it iterates all intel_svam_dev in the list and invalidates the IOTLB and device TLB sequentially.

Quoted 5 lines

If multiple devices belong to a single IOMMU, the IOTLB will be flushed multiple times. However, since these devices share the same domain ID and PASID, a single IOTLB cache invalidation is sufficient. The additional flushes are redundant and negatively impact performance.

Quoted 3 lines

yes it's redundant. But what does "devices belonging to different IOMMU domains" in the original context try to convey? From above explanation it sounds irrelevant...

My typo. :-) Sorry for the confusion.

I should say,

"... attached to devices belonging to a same IOMMU ..."

Best regards,
baolu

#18 lore raw

RE: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers

In reply to Baolu Lu
Quoted 41 lines

From: Baolu Lu <[email protected]>
Sent: Monday, April 8, 2024 10:54 AM

Quoted 9 lines · Kevin
Quoted 9 lines

From: Baolu Lu<[email protected]>
Sent: Sunday, April 7, 2024 1:33 PM

Quoted 5 lines · Kevin
Quoted 5 lines

From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AM

+/*
+ * Invalidate a range of IOVA when IOMMU is in caching mode and

new

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

mappings
+ * are added to the target domain.
+ */
+void cache_tag_flush_cm_range(struct dmar_domain *domain,

unsigned

Quoted 9 lines
Quoted 9 lines
Quoted 5 lines

drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 343

Quoted 3 lines

long start,

drivers/iommu/intel/cache.c +2 -0
+			      unsigned long end)
+{

I'm also not sure why this is worth a separate helper. why couldn't it be managed by previous flush_range()?

This is only my preference. I'd like to separate things belonging to different paths, so that it's easier for maintenance. For example, if, in the future, we need to add or enhance something for a specific case, we don't need to care about other cases.

Quoted 5 lines

IMHO caching mode is an attribute in low level iommu which can be handled perfectly well within the helper by checking that attribute.

it sounds a bit weird for the caller to know that detail and call different helpers when all paths just want to request to flush a specific range.

I see. The helper name is a bit confusing.

Generally speaking, cache_tag_flush_range() and cache_tag_flush_all() are designed to flush caches for mapping change from present to non- present. While cache_tag_flush_cm_range() is designed to flush caches for mapping change from non-present to present.

Quoted 5 lines

How about renaming these helpers to

cache_tag_flush_present_range()
cache_tag_flush_present_all()
cache_tag_flush_non_present_range()

I'll probably keep cache_tag_flush_range/all() as it is because the naming matches the general interpretation of the tlb semantics.

then call the 3rd one as cache_tag_flush_range_np() for various additional requirements on transitions from non-present entry.

Quoted 6 lines

?

In cache_tag_flush_non_present_range(),

- if IOMMU is not in caching mode, flush the write buffer if necessary,

or if using stage-1

Quoted 5 lines

or it's a no-op.
- if IOMMU is in caching mode, flush the IOTLB caches.

Best regards,
baolu

#6 lore raw

Re: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

In reply to Lu Baolu
Quoted 31 lines · Lu Baolu

Caching tag is a combination of tags used by the hardware to cache various translations. Whenever a mapping in a domain is changed, the IOMMU driver should invalidate the caches with the caching tags. The VT-d specification describes caching tags in section 6.2.1, Tagging of Cached Translations.

Add interface to assign caching tags to an IOMMU domain when attached to a RID or PASID, and unassign caching tags when a domain is detached from a RID or PASID. All caching tags are listed in the per-domain tag list and are protected by a dedicated lock.

In addition to the basic IOTLB and devTLB caching tag types, PARENT_IOTLB and PARENT_DEVTLB tag types are also introduced. These tags are used for caches that store translations for DMA accesses through a nested user domain. They are affected by changes to mappings in the parent domain.

---
 drivers/iommu/intel/iommu.h  |  25 +++++
 drivers/iommu/intel/cache.c  | 192 +++++++++++++++++++++++++++++++++++
 drivers/iommu/intel/iommu.c  |  31 +++++-
 drivers/iommu/intel/nested.c |  21 +++-
 drivers/iommu/intel/svm.c    |  12 ++-
 drivers/iommu/intel/Makefile |   2 +-
 6 files changed, 274 insertions(+), 9 deletions(-)
 create mode 100644 drivers/iommu/intel/cache.c
drivers/iommu/intel/iommu.h +0 -0
index 404d2476a877..e3723b7a0b31 100644

drivers/iommu/intel/iommu.h struct dmar_domain { · line 611

Quoted 5 lines
drivers/iommu/intel/iommu.h +2 -0
@@ -607,6 +607,9 @@ struct dmar_domain {
 	struct list_head devices;	/* all devices' list */
 	struct list_head dev_pasids;	/* all attached pasids */
 
+	spinlock_t cache_lock;		/* Protect the cache tag list */
+	struct list_head cache_tags;	/* Cache tag list */

That is quite a neat trick - though building a dedicated invalidation list duplicates data stored in the attached devices list?

You didn't try to make it RCU safe for invalidation?

drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1109

Quoted 5 lines
drivers/iommu/intel/iommu.h +5 -0
+struct cache_tag {
+	struct list_head node;
+	enum cache_tag_type type;
+	struct intel_iommu *iommu;
+	struct device *dev;

iommu and dev probably don't both need to be stored together. We have iommu_get_iommu_dev() now.. I suppose this is probably a union of the two pointers depending on tag. DEVTLB needs the dev and IOTLB needs the iommu.

drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1112

Quoted 3 lines
drivers/iommu/intel/iommu.h +3 -0
+	u16 domain_id;
+	ioasid_t pasid;
+	int users;

unsigned int

drivers/iommu/intel/iommu.h

Quoted 9 lines
drivers/iommu/intel/iommu.h +9 -0
+static int __cache_tag_assign_parent_domain(struct dmar_domain *domain, u16 did,
+					    struct device *dev, ioasid_t pasid)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+	int ret;
+
+	ret = cache_tag_assign(domain, did, dev, pasid, CACHE_TAG_TYPE_PARENT_IOTLB);
+	if (ret || !info->ats_enabled)
+		return ret;

I'm not sure I understood the point of PARENT_IOTLB? I didn't see any different implementation?

Isn't this backwards though? Each domain should have a list of things to invalidate if the domain itself changes.

So the nesting parent should have a list of CHILD_DEVTLB's that need cleaning. That list is changed when the nesting domains are attached to something.

And a list of CHILD_IOTLBs, but the HW doesn't seem to need that?

Jason

#7 lore raw

RE: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

Quoted 19 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 10, 2024 11:42 PM

Quoted 1 line · Lu Baolu

+static int __cache_tag_assign_parent_domain(struct dmar_domain

*domain, u16 did,

drivers/iommu/intel/cache.c

Quoted 6 lines
drivers/iommu/intel/cache.c new file +6 -0
+					    struct device *dev, ioasid_t pasid)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+	int ret;
+
+	ret = cache_tag_assign(domain, did, dev, pasid,

CACHE_TAG_TYPE_PARENT_IOTLB);

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (ret || !info->ats_enabled)
+		return ret;

I'm not sure I understood the point of PARENT_IOTLB? I didn't see any different implementation?

Quoted 7 lines

Isn't this backwards though? Each domain should have a list of things to invalidate if the domain itself changes.

So the nesting parent should have a list of CHILD_DEVTLB's that need cleaning. That list is changed when the nesting domains are attached to something.

probably just a naming confusion. it's called PARENT_IOTLB from the angle that this domain is used as a parent domain but actually it tracks the child tags in nested attach.

author maintainer
#10 lore raw

Re: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

In reply to Jason Gunthorpe
Quoted 35 lines · Jason Gunthorpe
Quoted 34 lines · Lu Baolu

Caching tag is a combination of tags used by the hardware to cache various translations. Whenever a mapping in a domain is changed, the IOMMU driver should invalidate the caches with the caching tags. The VT-d specification describes caching tags in section 6.2.1, Tagging of Cached Translations.

Add interface to assign caching tags to an IOMMU domain when attached to a RID or PASID, and unassign caching tags when a domain is detached from a RID or PASID. All caching tags are listed in the per-domain tag list and are protected by a dedicated lock.

In addition to the basic IOTLB and devTLB caching tag types, PARENT_IOTLB and PARENT_DEVTLB tag types are also introduced. These tags are used for caches that store translations for DMA accesses through a nested user domain. They are affected by changes to mappings in the parent domain.

---
  drivers/iommu/intel/iommu.h  |  25 +++++
  drivers/iommu/intel/cache.c  | 192 +++++++++++++++++++++++++++++++++++
  drivers/iommu/intel/iommu.c  |  31 +++++-
  drivers/iommu/intel/nested.c |  21 +++-
  drivers/iommu/intel/svm.c    |  12 ++-
  drivers/iommu/intel/Makefile |   2 +-
  6 files changed, 274 insertions(+), 9 deletions(-)

create mode 100644 drivers/iommu/intel/cache.c

drivers/iommu/intel/iommu.h +0 -0
index 404d2476a877..e3723b7a0b31 100644
@@ -607,6 +607,9 @@ struct dmar_domain {
  	struct list_head devices;	/* all devices' list */
  	struct list_head dev_pasids;	/* all attached pasids */
 
Quoted 5 lines

drivers/iommu/intel/iommu.h struct dmar_domain { · line 611

Quoted 2 lines
drivers/iommu/intel/iommu.h +2 -0
+	spinlock_t cache_lock;		/* Protect the cache tag list */
+	struct list_head cache_tags;	/* Cache tag list */

That is quite a neat trick - though building a dedicated invalidation list duplicates data stored in the attached devices list?

Yes. The device and dev_pasid lists appear to be duplicate. I am about to remove these two lists later.

Quoted 1 line

You didn't try to make it RCU safe for invalidation?

The queued invalidation interface is a bit complicated, especially when it comes to device TLB invalidation. Device TLB invalidation might result in a timeout, which requires special treatment.

Quoted 11 lines

drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1109

Quoted 5 lines
drivers/iommu/intel/iommu.h +5 -0
+struct cache_tag {
+	struct list_head node;
+	enum cache_tag_type type;
+	struct intel_iommu *iommu;
+	struct device *dev;

iommu and dev probably don't both need to be stored together. We have iommu_get_iommu_dev() now.. I suppose this is probably a union of the two pointers depending on tag. DEVTLB needs the dev and IOTLB needs the iommu.

Quoted 5 lines

drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1112

Quoted 3 lines
drivers/iommu/intel/iommu.h +3 -0
+	u16 domain_id;
+	ioasid_t pasid;
+	int users;

unsigned int

Sure.

Quoted 14 lines

drivers/iommu/intel/iommu.h

Quoted 10 lines
drivers/iommu/intel/iommu.h +9 -0
+static int __cache_tag_assign_parent_domain(struct dmar_domain *domain, u16 did,
+					    struct device *dev, ioasid_t pasid)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+	int ret;
+
+	ret = cache_tag_assign(domain, did, dev, pasid, CACHE_TAG_TYPE_PARENT_IOTLB);
+	if (ret || !info->ats_enabled)
+		return ret;

I'm not sure I understood the point of PARENT_IOTLB? I didn't see any different implementation?

Quoted 8 lines

Isn't this backwards though? Each domain should have a list of things to invalidate if the domain itself changes.

So the nesting parent should have a list of CHILD_DEVTLB's that need cleaning. That list is changed when the nesting domains are attached to something.

And a list of CHILD_IOTLBs, but the HW doesn't seem to need that?

This is a partial replacement of below series.

https://lore.kernel.org/all/[email protected]/

Best regards,
baolu

author maintainer
#11 lore raw

Re: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

In reply to Jason Gunthorpe
Quoted 9 lines · Jason Gunthorpe

drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1109

Quoted 5 lines
drivers/iommu/intel/iommu.h +5 -0
+struct cache_tag {
+	struct list_head node;
+	enum cache_tag_type type;
+	struct intel_iommu *iommu;
+	struct device *dev;

iommu and dev probably don't both need to be stored together. We have iommu_get_iommu_dev() now.. I suppose this is probably a union of the two pointers depending on tag. DEVTLB needs the dev and IOTLB needs the iommu.

I forgot to reply this comment in previous reply. Sorry about it.

struct cache_tag {
        [ ... ]
        struct intel_iommu *iommu;
        struct device *dev;
        [ ... ]
};

I treat @iommu as the queued invalidation interface. All cache invalidation raises to hardware through the invalidation queue.

The @dev field represents the location of the cache. For IOTLB cache, it resides on the IOMMU hardware. In this case, the field stores the device pointer to the IOMMU hardware. For DevTLB cache, it locates in the PCIe endpoint. Here, the field stores the device pointer to that endpoint.

A correctly set @dev pointer allows users to see more accurate trace message.

Best regards,
baolu

author maintainer
#8 lore raw

Re: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

In reply to Tian, Kevin
Quoted 22 lines · Kevin
Quoted 22 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 10, 2024 11:42 PM

Quoted 1 line · Lu Baolu

+static int __cache_tag_assign_parent_domain(struct dmar_domain

*domain, u16 did,

drivers/iommu/intel/cache.c

Quoted 6 lines
drivers/iommu/intel/cache.c new file +6 -0
+					    struct device *dev, ioasid_t pasid)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+	int ret;
+
+	ret = cache_tag_assign(domain, did, dev, pasid,

CACHE_TAG_TYPE_PARENT_IOTLB);

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (ret || !info->ats_enabled)
+		return ret;

I'm not sure I understood the point of PARENT_IOTLB? I didn't see any different implementation?

Isn't this backwards though? Each domain should have a list of things to invalidate if the domain itself changes.

Quoted 8 lines
Quoted 4 lines

So the nesting parent should have a list of CHILD_DEVTLB's that need cleaning. That list is changed when the nesting domains are attached to something.

probably just a naming confusion. it's called PARENT_IOTLB from the angle that this domain is used as a parent domain but actually it tracks the child tags in nested attach.

Is NESTING_IOTLB more readable?

Best regards,
baolu

#9 lore raw

Re: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

Quoted 28 lines · Baolu Lu
Quoted 27 lines · Kevin
Quoted 26 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 10, 2024 11:42 PM

Quoted 1 line · Lu Baolu

+static int __cache_tag_assign_parent_domain(struct dmar_domain

*domain, u16 did,

drivers/iommu/intel/cache.c

Quoted 6 lines
drivers/iommu/intel/cache.c new file +6 -0
+					    struct device *dev, ioasid_t pasid)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+	int ret;
+
+	ret = cache_tag_assign(domain, did, dev, pasid,

CACHE_TAG_TYPE_PARENT_IOTLB);

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (ret || !info->ats_enabled)
+		return ret;

I'm not sure I understood the point of PARENT_IOTLB? I didn't see any different implementation?

Isn't this backwards though? Each domain should have a list of things to invalidate if the domain itself changes.

So the nesting parent should have a list of CHILD_DEVTLB's that need cleaning. That list is changed when the nesting domains are attached to something.

Quoted 5 lines
Quoted 3 lines

probably just a naming confusion. it's called PARENT_IOTLB from the angle that this domain is used as a parent domain but actually it tracks the child tags in nested attach.

Is NESTING_IOTLB more readable?

Yes

Jason

#12 lore raw

RE: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface

In reply to Baolu Lu
Quoted 26 lines

From: Baolu Lu <[email protected]>
Sent: Thursday, April 11, 2024 8:39 PM

Quoted 9 lines · Jason Gunthorpe

drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1109

Quoted 5 lines
drivers/iommu/intel/iommu.h +5 -0
+struct cache_tag {
+	struct list_head node;
+	enum cache_tag_type type;
+	struct intel_iommu *iommu;
+	struct device *dev;

iommu and dev probably don't both need to be stored together. We have iommu_get_iommu_dev() now.. I suppose this is probably a union of the two pointers depending on tag. DEVTLB needs the dev and IOTLB needs the iommu.

I forgot to reply this comment in previous reply. Sorry about it.

struct cache_tag {
         [ ... ]
         struct intel_iommu *iommu;
         struct device *dev;
         [ ... ]
};

I treat @iommu as the queued invalidation interface. All cache invalidation raises to hardware through the invalidation queue.

Quoted 8 lines

The @dev field represents the location of the cache. For IOTLB cache, it resides on the IOMMU hardware. In this case, the field stores the device pointer to the IOMMU hardware. For DevTLB cache, it locates in the PCIe endpoint. Here, the field stores the device pointer to that endpoint.

A correctly set @dev pointer allows users to see more accurate trace message.

it's not a bad to add a comment for @dev here.