Consolidate domain cache invalidation
Subsystems (2)
Lu Baolu
maintainercover Consolidate domain cache invalidation
The IOMMU hardware cache needs to be invalidated whenever the mappings in the domain are changed. Currently, domain cache invalidation is scattered across different places, causing several issues:
- IOMMU IOTLB Invalidation: This is done by iterating through the domain IDs of each domain using the following code:
xa_for_each(&dmar_domain->iommu_array, i, info)
iommu_flush_iotlb_psi(info->iommu, dmar_domain,
start_pfn, nrpages,
list_empty(&gather->freelist), 0);This code could theoretically cause a use-after-free problem because there's no lock to protect the "info" pointer within the loop.
- Inconsistent Invalidation Methods: Different domain types implement their own cache invalidation methods, making the code difficult to maintain. For example, the DMA domain, SVA domain, and nested domain have similar cache invalidation code scattered across different files.
- SVA Domain Inconsistency: The SVA domain implementation uses a completely different data structure to track attached devices compared to other domains. This creates unnecessary differences and, even worse, leads to duplicate IOTLB invalidation when an SVA domain is attached to devices belonging to different IOMMU domains.
- Nested Domain Dependency: The special overlap between a nested domain and its parent domain requires a dedicated parent_domain_flush() helper function to be called everywhere the parent domain's mapping changes.
- Limited Debugging Support: There are currently no debugging aids available for domain cache invalidation.
By consolidating domain cache invalidation into a common location, we can address the issues mentioned above and improve the code's maintainability and debuggability.
Jason Gunthorpe (1):
iommu: Add ops->domain_alloc_sva()
Lu Baolu (11):
iommu/vt-d: Add cache tag assignment interface iommu/vt-d: Add cache tag invalidation helpers iommu/vt-d: Add trace events for cache tag interface iommu/vt-d: Use cache_tag_flush_all() in flush_iotlb_all iommu/vt-d: Use cache_tag_flush_range() in tlb_sync iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi() iommu/vt-d: Use cache_tag_flush_range() in cache_invalidate_user iommu/vt-d: Use cache helpers in arch_invalidate_secondary_tlbs iommu/vt-d: Retire intel_svm_dev
iommu/vt-d: Retire struct intel_svm
include/linux/iommu.h | 3 + drivers/iommu/intel/iommu.h | 66 +++--- drivers/iommu/intel/trace.h | 97 +++++++++ drivers/iommu/intel/cache.c | 389 +++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.c | 294 ++++---------------------- drivers/iommu/intel/nested.c | 71 ++----- drivers/iommu/intel/svm.c | 279 ++++++------------------- drivers/iommu/iommu-sva.c | 16 +- drivers/iommu/intel/Makefile | 2 +- 9 files changed, 660 insertions(+), 557 deletions(-) create mode 100644 drivers/iommu/intel/cache.c
Signature · 1 line
2.34.1
Lu Baolu
maintainerPATCH 1/12 iommu/vt-d: Add cache tag assignment interface
Caching tag is a combination of tags used by the hardware to cache various translations. Whenever a mapping in a domain is changed, the IOMMU driver should invalidate the caches with the caching tags. The VT-d specification describes caching tags in section 6.2.1, Tagging of Cached Translations.
Add interface to assign caching tags to an IOMMU domain when attached to a RID or PASID, and unassign caching tags when a domain is detached from a RID or PASID. All caching tags are listed in the per-domain tag list and are protected by a dedicated lock.
In addition to the basic IOTLB and devTLB caching tag types, PARENT_IOTLB and PARENT_DEVTLB tag types are also introduced. These tags are used for caches that store translations for DMA accesses through a nested user domain. They are affected by changes to mappings in the parent domain.
--- drivers/iommu/intel/iommu.h | 25 +++++ drivers/iommu/intel/cache.c | 192 +++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.c | 31 +++++- drivers/iommu/intel/nested.c | 21 +++- drivers/iommu/intel/svm.c | 12 ++- drivers/iommu/intel/Makefile | 2 +- 6 files changed, 274 insertions(+), 9 deletions(-) create mode 100644 drivers/iommu/intel/cache.c
drivers/iommu/intel/iommu.h +25 -0
@@ -607,6 +607,9 @@ struct dmar_domain {struct list_head devices;/* all devices' list */ struct list_head dev_pasids;/* all attached pasids */ +spinlock_t cache_lock;/* Protect the cache tag list */ +struct list_head cache_tags;/* Cache tag list */ +int iommu_superpage;/* Level of superpages supported: 0 == 4KiB (no superpages), 1 == 2MiB, 2 == 1GiB, 3 == 512GiB, 4 == 1TiB */
@@ -1092,6 +1095,28 @@ struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, const struct iommu_user_data *user_data);struct device *device_rbtree_find (struct intel_iommu *iommu,u16 rid); +enum cache_tag_type { +CACHE_TAG_TYPE_IOTLB , +CACHE_TAG_TYPE_DEVTLB , +CACHE_TAG_TYPE_PARENT_IOTLB , +CACHE_TAG_TYPE_PARENT_DEVTLB , +}; + +struct cache_tag { +struct list_head node; +enum cache_tag_type type; +struct intel_iommu *iommu; +struct device *dev; +u16 domain_id; +ioasid_t pasid; +int users; +}; + +int cache_tag_assign_domain (struct dmar_domain *domain,u16 did, +struct device *dev,ioasid_t pasid); +void cache_tag_unassign_domain (struct dmar_domain *domain,u16 did, +struct device *dev,ioasid_t pasid); +#ifdef CONFIG_INTEL_IOMMU_SVM void intel_svm_check (struct intel_iommu *iommu);int intel_svm_enable_prq (struct intel_iommu *iommu);
drivers/iommu/intel/cache.c new file +192 -0
192 lines not on this page · raw message
drivers/iommu/intel/iommu.c +29 -2
@@ -1750,7 +1750,9 @@ static struct dmar_domain *alloc_domain(unsigned int type) domain->has_iotlb_device = false;INIT_LIST_HEAD (&domain->devices);INIT_LIST_HEAD (&domain->dev_pasids); +INIT_LIST_HEAD (&domain->cache_tags);spin_lock_init (&domain->lock); +spin_lock_init (&domain->cache_lock);xa_init (&domain->iommu_array);return domain;
@@ -2322,11 +2324,20 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,struct device_domain_info *info =dev_iommu_priv_get (dev);struct intel_iommu *iommu = info->iommu;unsigned long flags; +u16 did;int ret; ret =domain_attach_iommu (domain, iommu);if (ret)return ret; + + did =domain_id_iommu (domain, iommu); + ret =cache_tag_assign_domain (domain, did, dev,IOMMU_NO_PASID ); +if (ret) { +domain_detach_iommu (domain, iommu); +return ret; + } + info->domain = domain;spin_lock_irqsave (&domain->lock, flags);list_add (&info->link, &domain->devices);
@@ -3798,6 +3809,7 @@ void device_block_translation(struct device *dev)struct device_domain_info *info =dev_iommu_priv_get (dev);struct intel_iommu *iommu = info->iommu;unsigned long flags; +u16 did;iommu_disable_pci_caps (info);if (!dev_is_real_dma_subdevice (dev)) {
@@ -3815,6 +3827,8 @@ void device_block_translation(struct device *dev)list_del (&info->link);spin_unlock_irqrestore (&info->domain->lock, flags); + did =domain_id_iommu (info->domain, iommu); +cache_tag_unassign_domain (info->domain, did, dev,IOMMU_NO_PASID );domain_detach_iommu (info->domain, iommu); info->domain =NULL ; }
@@ -4595,10 +4609,12 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)struct dmar_domain *dmar_domain;struct iommu_domain *domain;unsigned long flags; +u16 did; domain =iommu_get_domain_for_dev_pasid (dev, pasid,0 );if (WARN_ON_ONCE (!domain)) goto out_tear_down; + dmar_domain =to_dmar_domain (domain);/* * The SVA implementation needs to handle its own stuffs like the mm
@@ -4607,10 +4623,11 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)*/ if (domain->type ==IOMMU_DOMAIN_SVA ) {intel_svm_remove_dev_pasid (dev, pasid); +cache_tag_unassign_domain (dmar_domain, +FLPT_DEFAULT_DID , dev, pasid); goto out_tear_down; } - dmar_domain =to_dmar_domain (domain);spin_lock_irqsave (&dmar_domain->lock, flags);list_for_each_entry (curr, &dmar_domain->dev_pasids, link_domain) {if (curr->dev == dev && curr->pasid == pasid) {
@@ -4622,6 +4639,8 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)WARN_ON_ONCE (!dev_pasid);spin_unlock_irqrestore (&dmar_domain->lock, flags); + did =domain_id_iommu (dmar_domain, iommu); +cache_tag_unassign_domain (dmar_domain, did, dev, pasid);domain_detach_iommu (dmar_domain, iommu);intel_iommu_debugfs_remove_dev_pasid (dev_pasid);kfree (dev_pasid);
@@ -4638,6 +4657,7 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,struct intel_iommu *iommu = info->iommu;struct dev_pasid_info *dev_pasid;unsigned long flags; +u16 did;int ret;if (!pasid_supported (iommu) ||dev_is_real_dma_subdevice (dev))
@@ -4661,6 +4681,11 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,if (ret) goto out_free; + did =domain_id_iommu (dmar_domain, iommu); + ret =cache_tag_assign_domain (dmar_domain, did, dev, pasid); +if (ret) + goto out_detach_iommu; +if (domain_type_is_si (dmar_domain)) ret =intel_pasid_setup_pass_through (iommu, dev, pasid);else if (dmar_domain->use_first_level)
@@ -4670,7 +4695,7 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain, ret =intel_pasid_setup_second_level (iommu, dmar_domain, dev, pasid);if (ret) - goto out_detach_iommu; + goto out_unassign_tag; dev_pasid->dev = dev; dev_pasid->pasid = pasid;
@@ -4682,6 +4707,8 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,intel_iommu_debugfs_create_dev_pasid (dev_pasid);return 0 ; +out_unassign_tag: +cache_tag_unassign_domain (dmar_domain, did, dev, pasid); out_detach_iommu:domain_detach_iommu (dmar_domain, iommu); out_free:
drivers/iommu/intel/nested.c +16 -5
@@ -26,6 +26,7 @@ static int intel_nested_attach_dev(struct iommu_domain *domain,struct intel_iommu *iommu = info->iommu;unsigned long flags;int ret =0 ; +u16 did;if (info->domain)device_block_translation (dev);
@@ -52,13 +53,15 @@ static int intel_nested_attach_dev(struct iommu_domain *domain,return ret; } + did =domain_id_iommu (dmar_domain, iommu); + ret =cache_tag_assign_domain (dmar_domain, did, dev,IOMMU_NO_PASID ); +if (ret) + goto detach_iommu; + ret =intel_pasid_setup_nested (iommu, dev,IOMMU_NO_PASID , dmar_domain); -if (ret) { -domain_detach_iommu (dmar_domain, iommu); -dev_err_ratelimited (dev,"Failed to setup pasid entry\n" ); -return ret; - } +if (ret) + goto unassign_tag; info->domain = dmar_domain;spin_lock_irqsave (&dmar_domain->lock, flags);
@@ -68,6 +71,12 @@ static int intel_nested_attach_dev(struct iommu_domain *domain,domain_update_iotlb (dmar_domain);return 0 ; +unassign_tag: +cache_tag_unassign_domain (dmar_domain, did, dev,IOMMU_NO_PASID ); +detach_iommu: +domain_detach_iommu (dmar_domain, iommu); + +return ret; }static void intel_nested_domain_free (struct iommu_domain *domain)
@@ -206,7 +215,9 @@ struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, domain->domain.type =IOMMU_DOMAIN_NESTED ;INIT_LIST_HEAD (&domain->devices);INIT_LIST_HEAD (&domain->dev_pasids); +INIT_LIST_HEAD (&domain->cache_tags);spin_lock_init (&domain->lock); +spin_lock_init (&domain->cache_lock);xa_init (&domain->iommu_array);spin_lock (&s2_domain->s1_lock);
drivers/iommu/intel/svm.c +11 -1
@@ -366,17 +366,25 @@ static int intel_svm_set_dev_pasid(struct iommu_domain *domain, sdev->qdep =0 ; } + ret =cache_tag_assign_domain (to_dmar_domain (domain), +FLPT_DEFAULT_DID , dev, pasid); +if (ret) + goto free_sdev; +/* Setup the pasid table: */ sflags =cpu_feature_enabled (X86_FEATURE_LA57 ) ?PASID_FLAG_FL5LP :0 ; ret =intel_pasid_setup_first_level (iommu, dev, mm->pgd, pasid,FLPT_DEFAULT_DID , sflags);if (ret) - goto free_sdev; + goto unassign_tag;list_add_rcu (&sdev->list, &svm->devs);return 0 ; +unassign_tag: +cache_tag_unassign_domain (to_dmar_domain (domain), +FLPT_DEFAULT_DID , dev, pasid); free_sdev:kfree (sdev); free_svm:
@@ -795,6 +803,8 @@ struct iommu_domain *intel_svm_domain_alloc(void)if (!domain)return NULL ; domain->domain.ops = &intel_svm_domain_ops; +INIT_LIST_HEAD (&domain->cache_tags); +spin_lock_init (&domain->cache_lock);return &domain->domain; }
drivers/iommu/intel/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_DMAR_TABLE) += dmar.o -obj-$(CONFIG_INTEL_IOMMU) += iommu.o pasid.o nested.o +obj-$(CONFIG_INTEL_IOMMU) += iommu.o pasid.o nested.o cache.o obj-$(CONFIG_DMAR_TABLE) += trace.o cap_audit.o obj-$(CONFIG_DMAR_PERF) += perf.o obj-$(CONFIG_INTEL_IOMMU_DEBUGFS) += debugfs.o
Signature · 1 line
2.34.1
RE: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1103
Quoted 9 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMdrivers/iommu/intel/iommu.h +6
-0+enum cache_tag_type { +CACHE_TAG_TYPE_IOTLB , +CACHE_TAG_TYPE_DEVTLB , +CACHE_TAG_TYPE_PARENT_IOTLB , +CACHE_TAG_TYPE_PARENT_DEVTLB , +};
'_TYPE_' can be removed to make it shorter
drivers/iommu/intel/cache.c line 23
Quoted 5 lines
drivers/iommu/intel/cache.c new file +5
-0+ +/* Checks if an existing cache tag can be reused for a new association. */ +static bool cache_tag_reusable (struct cache_tag *tag,u16 domain_id, +struct intel_iommu *iommu,struct device *dev, +ioasid_t pasid,enum cache_tag_type type)
cache_tage_match()
Quoted 4 lines
drivers/iommu/intel/cache.c new file +4
-0+{ +if (tag->type != type) +return false; +
drivers/iommu/intel/cache.c line 37
Quoted 10 lines
drivers/iommu/intel/cache.c new file +10
-0+if (tag->domain_id != domain_id || tag->pasid != pasid) +return false; + +if (type ==CACHE_TAG_TYPE_IOTLB ) +return tag->iommu == iommu; + +if (type ==CACHE_TAG_TYPE_DEVTLB ) +return tag->dev == dev; + +return false;
why do you disallow PARENT_TYPE from reusing? It's not uncommon to have two devices attached to a same nested domain hence with the same parent domain. Disallowing tag reuse implies unnecessarily duplicated cache flushes...
Quoted 9 lines
drivers/iommu/intel/cache.c new file +9
-0+} + +/* Assign a cache tag with specified type to domain. */ +static int cache_tag_assign (struct dmar_domain *domain,u16 did, +struct device *dev,ioasid_t pasid, +enum cache_tag_type type) +{ +struct device_domain_info *info =dev_iommu_priv_get (dev); +struct intel_iommu *iommu = info->iommu;
drivers/iommu/intel/cache.c line 56
Quoted 10 lines
drivers/iommu/intel/cache.c new file +10
-0+struct cache_tag *tag, *temp; +unsigned long flags; + + tag =kzalloc (sizeof (*tag),GFP_KERNEL ); +if (!tag) +return -ENOMEM ; + + tag->type = type; + tag->iommu = iommu; + tag->dev = dev;
should we set tag->dev only for DEVTLB type? It's a bit confusing to set it for IOTLB type which doesn't care about device. Actually doing so is instead misleading as the 1st device creating the tag may have been detached but then it will still show up in the trace when the last device detach destroying the tag.
Review
Quoted 3 lines
+static int __cache_tag_assign_parent_domain(struct
dmar_domain*domain, u16 did,(quoted diff) +1
-0+ struct device *dev, ioasid_t pasid)
this pair is similar to the earlier one except the difference on type.
what about keeping just one pair which accepts a 'parent' argument to decide the type internally?
Quoted 3 lines
+/*
+ * Assigns cache tags to a domain when it's associated with a device's
+ * PASID using a specific domain ID.
s/Assigns/Assign/
Quoted 4 lines
drivers/iommu/intel/iommu.c +3
-0+ + did =domain_id_iommu (domain, iommu); + ret = cache_tag_assign_domain(domain, did, dev,
IOMMU_NO_PASID);
there are many occurrences of this pattern. What about passing in a 'iommu' parameter and getting 'did' inside the helper? for svm it can be specialized internally too.
drivers/iommu/intel/iommu.c static void · line 4628
Quoted 7 lines
drivers/iommu/intel/iommu.c +2
-0@@ -4607,10 +4623,11 @@ static voidintel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) */ if (domain->type ==IOMMU_DOMAIN_SVA ) {intel_svm_remove_dev_pasid (dev, pasid); +cache_tag_unassign_domain (dmar_domain, +FLPT_DEFAULT_DID , dev, pasid);
is it correct to destroy the tag before teardown completes, e.g. iotlb still needs to be flushed in intel_pasid_tear_down_entry()?
Baolu Lu
maintainerRe: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
Hi Kevin,
Thanks for your review comments.
Quoted 11 lines · Kevin
drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1103
Quoted 9 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMdrivers/iommu/intel/iommu.h +6
-0+enum cache_tag_type { +CACHE_TAG_TYPE_IOTLB , +CACHE_TAG_TYPE_DEVTLB , +CACHE_TAG_TYPE_PARENT_IOTLB , +CACHE_TAG_TYPE_PARENT_DEVTLB , +};'_TYPE_' can be removed to make it shorter
Okay.
Quoted 8 lines
drivers/iommu/intel/cache.c line 23
Quoted 6 lines
drivers/iommu/intel/cache.c new file +5
-0+ +/* Checks if an existing cache tag can be reused for a new association. */ +static bool cache_tag_reusable (struct cache_tag *tag,u16 domain_id, +struct intel_iommu *iommu,struct device *dev, +ioasid_t pasid,enum cache_tag_type type)
cache_tage_match()
Okay.
Quoted 12 lines
Quoted 12 lines
drivers/iommu/intel/cache.c new file +11
-0+{ +if (tag->type != type) +return false; + +if (tag->domain_id != domain_id || tag->pasid != pasid) +return false; + +if (type ==CACHE_TAG_TYPE_IOTLB ) +return tag->iommu == iommu; + +if (type ==CACHE_TAG_TYPE_DEVTLB )
Quoted 8 lines
drivers/iommu/intel/cache.c line 37
Quoted 3 lines
drivers/iommu/intel/cache.c new file +3
-0+return tag->dev == dev; + +return false;why do you disallow
PARENT_TYPEfrom reusing? It's not uncommon to have two devices attached to a same nested domain hence with the same parent domain. Disallowing tag reuse implies unnecessarily duplicated cache flushes...
PARENT_TYPE could be reused. The new helper looks like the following:
/* Checks if an existing cache tag can be reused for a new association. */ static bool cache_tage_match(struct cache_tag *tag, u16 domain_id,
struct intel_iommu *iommu, struct device
*dev,
ioasid_t pasid, enum cache_tag_type type)
{
if (tag->type != type)
return false;
if (tag->domain_id != domain_id || tag->pasid != pasid)
return false;
if (type == CACHE_TAG_IOTLB || type == CACHE_TAG_PARENT_IOTLB)
return tag->iommu == iommu;
if (type == CACHE_TAG_DEVTLB || type == CACHE_TAG_PARENT_DEVTLB)
return tag->dev == dev;
return false;
}Quoted 17 lines
Quoted 17 lines
drivers/iommu/intel/cache.c new file +17
-0+} + +/* Assign a cache tag with specified type to domain. */ +static int cache_tag_assign (struct dmar_domain *domain,u16 did, +struct device *dev,ioasid_t pasid, +enum cache_tag_type type) +{ +struct device_domain_info *info =dev_iommu_priv_get (dev); +struct intel_iommu *iommu = info->iommu; +struct cache_tag *tag, *temp; +unsigned long flags; + + tag =kzalloc (sizeof (*tag),GFP_KERNEL ); +if (!tag) +return -ENOMEM ; + + tag->type = type;
Quoted 8 lines
drivers/iommu/intel/cache.c line 56
Quoted 2 lines
drivers/iommu/intel/cache.c new file +2
-0+ tag->iommu = iommu; + tag->dev = dev;should we set
tag->devonly for DEVTLB type? It's a bit confusing to set it for IOTLB type which doesn't care about device. Actually doing so is instead misleading as the 1st device creating the tag may have been detached but then it will still show up in the trace when the last device detach destroying the tag.
For IOTLB types, perhaps we could add a struct device pointer for the iommu. This way, the tag->dev could more directly indicate the device implementing the cache.
Quoted 9 lines
Review
Quoted 4 lines
+static int __cache_tag_assign_parent_domain(struct
dmar_domain*domain, u16 did,(quoted diff) +1
-0+ struct device *dev, ioasid_t pasid)this pair is similar to the earlier one except the difference on type.
what about keeping just one pair which accepts a 'parent' argument to decide the type internally?
Okay, let try to refine it.
Quoted 7 lines
Quoted 5 lines
+/*
+ * Assigns cache tags to a domain when it's associated with a device's
+ * PASID using a specific domain ID.s/Assigns/Assign/
Done.
Quoted 9 lines
Quoted 5 lines
drivers/iommu/intel/iommu.c +3
-0+ + did =domain_id_iommu (domain, iommu); + ret = cache_tag_assign_domain(domain, did, dev,
IOMMU_NO_PASID);there are many occurrences of this pattern. What about passing in a 'iommu' parameter and getting 'did' inside the helper? for svm it can be specialized internally too.
Perhaps, let me try it later and see what the code looks like.
Quoted 11 lines
drivers/iommu/intel/iommu.c static void · line 4628
Quoted 8 lines
drivers/iommu/intel/iommu.c +2
-0@@ -4607,10 +4623,11 @@ static voidintel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) */ if (domain->type ==IOMMU_DOMAIN_SVA ) {intel_svm_remove_dev_pasid (dev, pasid); +cache_tag_unassign_domain (dmar_domain, +FLPT_DEFAULT_DID , dev, pasid);is it correct to destroy the tag before teardown completes, e.g. iotlb still needs to be flushed in
intel_pasid_tear_down_entry()?
You are right. iotlb still needs to be there until the teardown completes. I will investigate this more later.
Beset regards,
baolu
Baolu Lu
maintainerRe: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
Quoted 12 lines · Baolu Lu
Quoted 11 lines
drivers/iommu/intel/iommu.c static void · line 4628
Quoted 8 lines
drivers/iommu/intel/iommu.c +2
-0@@ -4607,10 +4623,11 @@ static voidintel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) */ if (domain->type == IOMMU_DOMAIN_SVA) { intel_svm_remove_dev_pasid(dev, pasid); + cache_tag_unassign_domain(dmar_domain, + FLPT_DEFAULT_DID, dev, pasid);is it correct to destroy the tag before teardown completes, e.g. iotlb still needs to be flushed in
intel_pasid_tear_down_entry()?
Quoted 2 lines
You are right. iotlb still needs to be there until the teardown completes. I will investigate this more later.
I reviewed this again. Cache tags are designed specifically for mapping and unmapping paths. Therefore, there is no required order for attaching and detaching paths.
Best regards,
baolu
RE: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
Quoted 17 lines
From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 12:35 PMQuoted 13 lines · Baolu Lu
Quoted 12 lines
drivers/iommu/intel/iommu.c static void · line 4628
Quoted 8 lines
drivers/iommu/intel/iommu.c +2
-0@@ -4607,10 +4623,11 @@ static voidintel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) */ if (domain->type == IOMMU_DOMAIN_SVA) { intel_svm_remove_dev_pasid(dev, pasid); + cache_tag_unassign_domain(dmar_domain, + FLPT_DEFAULT_DID, dev, pasid);is it correct to destroy the tag before teardown completes, e.g. iotlb still
needs to be flushed inintel_pasid_tear_down_entry()?
Quoted 7 lines
Quoted 2 lines
You are right. iotlb still needs to be there until the teardown completes. I will investigate this more later.
I reviewed this again. Cache tags are designed specifically for mapping and unmapping paths. Therefore, there is no required order for attaching and detaching paths.
Okay. intel_pasid_tear_down_entry() directly retrieves the information from the pasid entry instead of relying on the domain cache tag info. so yes destroying the tag at this point is fine.
Re: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
In reply to Lu BaoluQuoted 31 lines · Lu Baolu
Caching tag is a combination of tags used by the hardware to cache various translations. Whenever a mapping in a domain is changed, the IOMMU driver should invalidate the caches with the caching tags. The VT-d specification describes caching tags in section 6.2.1, Tagging of Cached Translations.
Add interface to assign caching tags to an IOMMU domain when attached to a RID or PASID, and unassign caching tags when a domain is detached from a RID or PASID. All caching tags are listed in the per-domain tag list and are protected by a dedicated lock.
In addition to the basic IOTLB and devTLB caching tag types,
PARENT_IOTLBandPARENT_DEVTLBtag types are also introduced. These tags are used for caches that store translations for DMA accesses through a nested user domain. They are affected by changes to mappings in the parent domain.Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.h | 25 +++++ drivers/iommu/intel/cache.c | 192 +++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.c | 31 +++++- drivers/iommu/intel/nested.c | 21 +++- drivers/iommu/intel/svm.c | 12 ++- drivers/iommu/intel/Makefile | 2 +- 6 files changed, 274 insertions(+), 9 deletions(-) create mode 100644 drivers/iommu/intel/cache.cdrivers/iommu/intel/iommu.h +0
-0
drivers/iommu/intel/iommu.h struct dmar_domain { · line 611
Quoted 5 lines
drivers/iommu/intel/iommu.h +2
-0@@ -607,6 +607,9 @@ struct dmar_domain {struct list_head devices;/* all devices' list */ struct list_head dev_pasids;/* all attached pasids */ +spinlock_t cache_lock;/* Protect the cache tag list */ +struct list_head cache_tags;/* Cache tag list */
That is quite a neat trick - though building a dedicated invalidation list duplicates data stored in the attached devices list?
You didn't try to make it RCU safe for invalidation?
drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1109
Quoted 5 lines
drivers/iommu/intel/iommu.h +5
-0+struct cache_tag { +struct list_head node; +enum cache_tag_type type; +struct intel_iommu *iommu; +struct device *dev;
iommu and dev probably don't both need to be stored together. We have iommu_get_iommu_dev() now.. I suppose this is probably a union of the two pointers depending on tag. DEVTLB needs the dev and IOTLB needs the iommu.
drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1112
Quoted 3 lines
drivers/iommu/intel/iommu.h +3
-0+u16 domain_id; +ioasid_t pasid; +int users;
unsigned int
drivers/iommu/intel/iommu.h
Quoted 9 lines
drivers/iommu/intel/iommu.h +9
-0+static int __cache_tag_assign_parent_domain (struct dmar_domain *domain,u16 did, +struct device *dev,ioasid_t pasid) +{ +struct device_domain_info *info =dev_iommu_priv_get (dev); +int ret; + + ret =cache_tag_assign (domain, did, dev, pasid,CACHE_TAG_TYPE_PARENT_IOTLB ); +if (ret || !info->ats_enabled) +return ret;
I'm not sure I understood the point of PARENT_IOTLB? I didn't see any different implementation?
Isn't this backwards though? Each domain should have a list of things to invalidate if the domain itself changes.
So the nesting parent should have a list of CHILD_DEVTLB's that need cleaning. That list is changed when the nesting domains are attached to something.
And a list of CHILD_IOTLBs, but the HW doesn't seem to need that?
Jason
RE: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
Quoted 19 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 10, 2024 11:42 PMQuoted 1 line · Lu Baolu
+static int __cache_tag_assign_parent_domain(struct
dmar_domain*domain, u16 did,
drivers/iommu/intel/cache.c
Quoted 6 lines
drivers/iommu/intel/cache.c new file +6
-0+ struct device *dev, ioasid_t pasid) +{ +struct device_domain_info *info =dev_iommu_priv_get (dev); +int ret; + + ret = cache_tag_assign(domain, did, dev, pasid,
CACHE_TAG_TYPE_PARENT_IOTLB);Review
Quoted 2 lines
(quoted diff) +2
-0+ if (ret || !info->ats_enabled) + return ret;I'm not sure I understood the point of
PARENT_IOTLB? I didn't see any different implementation?
Quoted 7 lines
Isn't this backwards though? Each domain should have a list of things to invalidate if the domain itself changes.
So the nesting parent should have a list of CHILD_DEVTLB's that need cleaning. That list is changed when the nesting domains are attached to something.
probably just a naming confusion. it's called PARENT_IOTLB from the angle that this domain is used as a parent domain but actually it tracks the child tags in nested attach.
Baolu Lu
maintainerRe: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
Quoted 22 lines · Kevin
Quoted 22 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 10, 2024 11:42 PMQuoted 1 line · Lu Baolu
+static int __cache_tag_assign_parent_domain(struct
dmar_domain*domain, u16 did,
drivers/iommu/intel/cache.c
Quoted 6 lines
drivers/iommu/intel/cache.c new file +6
-0+ struct device *dev, ioasid_t pasid) +{ +struct device_domain_info *info =dev_iommu_priv_get (dev); +int ret; + + ret = cache_tag_assign(domain, did, dev, pasid,
CACHE_TAG_TYPE_PARENT_IOTLB);Review
Quoted 2 lines
(quoted diff) +2
-0+ if (ret || !info->ats_enabled) + return ret;I'm not sure I understood the point of
PARENT_IOTLB? I didn't see any different implementation?Isn't this backwards though? Each domain should have a list of things to invalidate if the domain itself changes.
Quoted 8 lines
Quoted 4 lines
So the nesting parent should have a list of CHILD_DEVTLB's that need cleaning. That list is changed when the nesting domains are attached to something.
probably just a naming confusion. it's called
PARENT_IOTLBfrom the angle that this domain is used as a parent domain but actually it tracks the child tags in nested attach.
Is NESTING_IOTLB more readable?
Best regards,
baolu
Re: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
Quoted 28 lines · Baolu Lu
Quoted 27 lines · Kevin
Quoted 26 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 10, 2024 11:42 PMQuoted 1 line · Lu Baolu
+static int __cache_tag_assign_parent_domain(struct
dmar_domain*domain, u16 did,
drivers/iommu/intel/cache.c
Quoted 6 lines
drivers/iommu/intel/cache.c new file +6
-0+ struct device *dev, ioasid_t pasid) +{ +struct device_domain_info *info =dev_iommu_priv_get (dev); +int ret; + + ret = cache_tag_assign(domain, did, dev, pasid,
CACHE_TAG_TYPE_PARENT_IOTLB);Review
Quoted 2 lines
(quoted diff) +2
-0+ if (ret || !info->ats_enabled) + return ret;I'm not sure I understood the point of
PARENT_IOTLB? I didn't see any different implementation?Isn't this backwards though? Each domain should have a list of things to invalidate if the domain itself changes.
So the nesting parent should have a list of CHILD_DEVTLB's that need cleaning. That list is changed when the nesting domains are attached to something.
Quoted 5 lines
Quoted 3 lines
probably just a naming confusion. it's called
PARENT_IOTLBfrom the angle that this domain is used as a parent domain but actually it tracks the child tags in nested attach.Is
NESTING_IOTLBmore readable?
Yes
Jason
Baolu Lu
maintainerRe: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
In reply to Jason GunthorpeQuoted 35 lines · Jason Gunthorpe
Quoted 34 lines · Lu Baolu
Caching tag is a combination of tags used by the hardware to cache various translations. Whenever a mapping in a domain is changed, the IOMMU driver should invalidate the caches with the caching tags. The VT-d specification describes caching tags in section 6.2.1, Tagging of Cached Translations.
Add interface to assign caching tags to an IOMMU domain when attached to a RID or PASID, and unassign caching tags when a domain is detached from a RID or PASID. All caching tags are listed in the per-domain tag list and are protected by a dedicated lock.
In addition to the basic IOTLB and devTLB caching tag types,
PARENT_IOTLBandPARENT_DEVTLBtag types are also introduced. These tags are used for caches that store translations for DMA accesses through a nested user domain. They are affected by changes to mappings in the parent domain.Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.h | 25 +++++ drivers/iommu/intel/cache.c | 192 +++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.c | 31 +++++- drivers/iommu/intel/nested.c | 21 +++- drivers/iommu/intel/svm.c | 12 ++- drivers/iommu/intel/Makefile | 2 +- 6 files changed, 274 insertions(+), 9 deletions(-)create mode 100644
drivers/iommu/intel/cache.cdrivers/iommu/intel/iommu.h +0
-0@@ -607,6 +607,9 @@ struct dmar_domain {struct list_head devices;/* all devices' list */ struct list_head dev_pasids;/* all attached pasids */
Quoted 5 lines
drivers/iommu/intel/iommu.h struct dmar_domain { · line 611
Quoted 2 lines
drivers/iommu/intel/iommu.h +2
-0+spinlock_t cache_lock;/* Protect the cache tag list */ +struct list_head cache_tags;/* Cache tag list */ That is quite a neat trick - though building a dedicated invalidation list duplicates data stored in the attached devices list?
Yes. The device and dev_pasid lists appear to be duplicate. I am about to remove these two lists later.
Quoted 1 line
You didn't try to make it RCU safe for invalidation?
The queued invalidation interface is a bit complicated, especially when it comes to device TLB invalidation. Device TLB invalidation might result in a timeout, which requires special treatment.
Quoted 11 lines
drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1109
Quoted 5 lines
drivers/iommu/intel/iommu.h +5
-0+struct cache_tag { +struct list_head node; +enum cache_tag_type type; +struct intel_iommu *iommu; +struct device *dev;iommu and dev probably don't both need to be stored together. We have
iommu_get_iommu_dev()now.. I suppose this is probably a union of the two pointers depending on tag. DEVTLB needs the dev and IOTLB needs the iommu.
Quoted 5 lines
drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1112
Quoted 3 lines
drivers/iommu/intel/iommu.h +3
-0+u16 domain_id; +ioasid_t pasid; +int users;unsigned int
Sure.
Quoted 14 lines
drivers/iommu/intel/iommu.h
Quoted 10 lines
drivers/iommu/intel/iommu.h +9
-0+static int __cache_tag_assign_parent_domain (struct dmar_domain *domain,u16 did, +struct device *dev,ioasid_t pasid) +{ +struct device_domain_info *info =dev_iommu_priv_get (dev); +int ret; + + ret =cache_tag_assign (domain, did, dev, pasid,CACHE_TAG_TYPE_PARENT_IOTLB ); +if (ret || !info->ats_enabled) +return ret;I'm not sure I understood the point of
PARENT_IOTLB? I didn't see any different implementation?
Quoted 8 lines
Isn't this backwards though? Each domain should have a list of things to invalidate if the domain itself changes.
So the nesting parent should have a list of CHILD_DEVTLB's that need cleaning. That list is changed when the nesting domains are attached to something.
And a list of
CHILD_IOTLBs, but the HW doesn't seem to need that?
This is a partial replacement of below series.
https://lore.kernel.org/all/[email protected]/
Best regards,
baolu
Baolu Lu
maintainerRe: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
In reply to Jason GunthorpeQuoted 9 lines · Jason Gunthorpe
drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1109
Quoted 5 lines
drivers/iommu/intel/iommu.h +5
-0+struct cache_tag { +struct list_head node; +enum cache_tag_type type; +struct intel_iommu *iommu; +struct device *dev;iommu and dev probably don't both need to be stored together. We have
iommu_get_iommu_dev()now.. I suppose this is probably a union of the two pointers depending on tag. DEVTLB needs the dev and IOTLB needs the iommu.
I forgot to reply this comment in previous reply. Sorry about it.
struct cache_tag {
[ ... ]
struct intel_iommu *iommu;
struct device *dev;
[ ... ]
};I treat @iommu as the queued invalidation interface. All cache invalidation raises to hardware through the invalidation queue.
The @dev field represents the location of the cache. For IOTLB cache, it resides on the IOMMU hardware. In this case, the field stores the device pointer to the IOMMU hardware. For DevTLB cache, it locates in the PCIe endpoint. Here, the field stores the device pointer to that endpoint.
A correctly set @dev pointer allows users to see more accurate trace message.
Best regards,
baolu
RE: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
Quoted 26 lines
From: Baolu Lu <[email protected]>
Sent: Thursday, April 11, 2024 8:39 PMQuoted 9 lines · Jason Gunthorpe
drivers/iommu/intel/iommu.h struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, · line 1109
Quoted 5 lines
drivers/iommu/intel/iommu.h +5
-0+struct cache_tag { +struct list_head node; +enum cache_tag_type type; +struct intel_iommu *iommu; +struct device *dev;iommu and dev probably don't both need to be stored together. We have
iommu_get_iommu_dev()now.. I suppose this is probably a union of the two pointers depending on tag. DEVTLB needs the dev and IOTLB needs the iommu.I forgot to reply this comment in previous reply. Sorry about it.
struct cache_tag { [ ... ] struct intel_iommu *iommu; struct device *dev; [ ... ] };I treat @iommu as the queued invalidation interface. All cache invalidation raises to hardware through the invalidation queue.
Quoted 8 lines
The @dev field represents the location of the cache. For IOTLB cache, it resides on the IOMMU hardware. In this case, the field stores the device pointer to the IOMMU hardware. For DevTLB cache, it locates in the PCIe endpoint. Here, the field stores the device pointer to that endpoint.
A correctly set @dev pointer allows users to see more accurate trace message.
it's not a bad to add a comment for @dev here.
Lu Baolu
maintainerPATCH 2/12 iommu/vt-d: Add cache tag invalidation helpers
In reply to Lu BaoluAdd several helpers to invalidate the caches after mappings in the affected domain are changed.
- cache_tag_flush_range() invalidates a range of caches after mappings within this range are changed. It uses the page-selective cache invalidation methods.
- cache_tag_flush_all() invalidates all caches tagged by a domain ID. It uses the domain-selective cache invalidation methods.
- cache_tag_flush_cm_range() invalidates a range of caches if IOMMU is working in the caching mode and second-only translation is used for the affected domain. It is called when non-present to present mappings are created.
--- drivers/iommu/intel/iommu.h | 7 ++ drivers/iommu/intel/cache.c | 189 ++++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.c | 5 - 3 files changed, 196 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.h +7 -0
@@ -35,6 +35,8 @@#define VTD_PAGE_MASK (((u64)-1) << VTD_PAGE_SHIFT)#define VTD_PAGE_ALIGN (addr) (((addr) + VTD_PAGE_SIZE - 1) & VTD_PAGE_MASK) +#define IOVA_PFN (addr) ((addr) >> PAGE_SHIFT) +#define VTD_STRIDE_SHIFT (9)#define VTD_STRIDE_MASK (((u64)-1) << VTD_STRIDE_SHIFT)
@@ -1116,6 +1118,11 @@ int cache_tag_assign_domain(struct dmar_domain *domain, u16 did, struct device *dev, ioasid_t pasid);void cache_tag_unassign_domain (struct dmar_domain *domain,u16 did,struct device *dev,ioasid_t pasid); +void cache_tag_flush_range (struct dmar_domain *domain,unsigned long start, +unsigned long end,int ih); +void cache_tag_flush_all (struct dmar_domain *domain); +void cache_tag_flush_cm_range (struct dmar_domain *domain,unsigned long start, +unsigned long end);#ifdef CONFIG_INTEL_IOMMU_SVM void intel_svm_check (struct intel_iommu *iommu);
drivers/iommu/intel/cache.c +189 -0
198 lines not on this page · raw message
drivers/iommu/intel/iommu.c +0 -5
@@ -54,11 +54,6 @@ __DOMAIN_MAX_PFN(gaw), (unsigned long)-1))#define DOMAIN_MAX_ADDR (gaw) (((uint64_t)__DOMAIN_MAX_PFN(gaw)) << VTD_PAGE_SHIFT) -/* IO virtual address start page frame number */ -#define IOVA_START_PFN (1) - -#define IOVA_PFN (addr) ((addr) >> PAGE_SHIFT) -static void __init check_tylersburg_isoch (void );static int rwbf_quirk;
Signature · 1 line
2.34.1
RE: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers
Quoted 11 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMdrivers/iommu/intel/cache.c +7
-0+ +static unsigned long calculate_psi_aligned_address (unsigned long start, +unsigned long end, +unsigned long *_pages, +unsigned long *_mask) +{ + unsigned long pages = (end - start + VTD_PAGE_SIZE - 1) >>
VTD_PAGE_SHIFT;
this doesn't sound correct. You'd want to follow aligned_nrpages().
Quoted 30 lines
drivers/iommu/intel/cache.c +4
-0+case CACHE_TAG_TYPE_DEVTLB : +if (tag->pasid ==IOMMU_NO_PASID ) + qi_flush_dev_iotlb(iommu, sid, info->pfsid, + info->ats_qdep, addr,mask);
(quoted diff) +2
-0+ else + qi_flush_dev_iotlb_pasid(iommu, sid, info-Quoted 1 line
pfsid,
(quoted diff) +1
-0+ tag->pasid, info-Quoted 1 line
ats_qdep,drivers/iommu/intel/cache.c +3
-0+ addr, mask); + + quirk_extra_dev_tlb_flush(info, addr, mask, tag-Quoted 1 line
pasid,
info->ats_qdep);drivers/iommu/intel/cache.c +10
-0+break ; +case CACHE_TAG_TYPE_PARENT_DEVTLB : +/* +* Address translation cache in device side caches the +* result of nested translation. There is no easy way +* to identify the exact set of nested translations +* affected by a change in S2. So just flush the entire +* device cache. +*/ + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info-Quoted 1 line
ats_qdep,(quoted diff) +2
-0+ 0, MAX_AGAW_PFN_WIDTH); + quirk_extra_dev_tlb_flush(info, 0,
MAX_AGAW_PFN_WIDTH,(quoted diff) +1
-0+ IOMMU_NO_PASID, info-Quoted 1 line
ats_qdep);
Review
Quoted 1 line
(quoted diff) +1
-0+ break;
parent devtlb can has pasid too, though not enabled now. As core helpers probably we can support it now leading to simpler code:
case CACHE_TAG_TYPE_PARENT_DEVTLB: //change addr/mask //fallthrough case CACHE_TAG_TYPE_DEVTLB: //what this patch does
Quoted 33 lines
drivers/iommu/intel/cache.c +8
-0+void cache_tag_flush_all (struct dmar_domain *domain) +{ +struct cache_tag *tag; +unsigned long flags; + +spin_lock_irqsave (&domain->cache_lock, flags); +list_for_each_entry (tag, &domain->cache_tags, node) { + struct device_domain_info *info = dev_iommu_priv_get(tag-Quoted 1 line
dev);
drivers/iommu/intel/cache.c +11
-0+struct intel_iommu *iommu = tag->iommu; +u16 sid =PCI_DEVID (info->bus, info->devfn); + +switch (tag->type) { +case CACHE_TAG_TYPE_IOTLB : +case CACHE_TAG_TYPE_PARENT_IOTLB : +if (domain->use_first_level) +qi_flush_piotlb (iommu, tag->domain_id, + tag->pasid,0 ,-1 ,0 ); +else + iommu->flush.flush_iotlb(iommu, tag-Quoted 1 line
domain_id,(quoted diff) +1
-0+ 0, 0,
DMA_TLB_DSI_FLUSH);drivers/iommu/intel/cache.c +4
-0+break ; +case CACHE_TAG_TYPE_DEVTLB : +case CACHE_TAG_TYPE_PARENT_DEVTLB : + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info-Quoted 1 line
ats_qdep,(quoted diff) +2
-0+ 0, MAX_AGAW_PFN_WIDTH); + quirk_extra_dev_tlb_flush(info, 0,
MAX_AGAW_PFN_WIDTH,(quoted diff) +1
-0+ IOMMU_NO_PASID, info-Quoted 1 line
ats_qdep);
drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 296
Quoted 3 lines
drivers/iommu/intel/cache.c +3
-0+break ; + } + }
could this loop be consolidated with the one in previous helper? anyway the only difference is on addr/mask...
drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 343
Quoted 9 lines
+/*
+ * Invalidate a range of IOVA when IOMMU is in caching mode and new mappings
+ * are added to the target domain.
+ */
+void cache_tag_flush_cm_range(structdmar_domain*domain, unsigned long start,drivers/iommu/intel/cache.c +2
-0+ unsigned long end) +{
I'm also not sure why this is worth a separate helper. why couldn't it be managed by previous flush_range()?
Quoted 18 lines
drivers/iommu/intel/cache.c +12
-0+unsigned long pages, mask, addr; +struct cache_tag *tag; +unsigned long flags; + + addr =calculate_psi_aligned_address (start, end, &pages, &mask); + +spin_lock_irqsave (&domain->cache_lock, flags); +list_for_each_entry (tag, &domain->cache_tags, node) { +struct intel_iommu *iommu = tag->iommu; + +/* +* When IOMMU is enabled in caching mode some non- present
drivers/iommu/intel/cache.c +4
-0+* mappings may be cached by the IOTLB if it uses second- +* only translation. +*/ + if (!cap_caching_mode(iommu->cap) || domain-Quoted 1 line
use_first_level) {
drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 362
Quoted 3 lines
drivers/iommu/intel/cache.c +3
-0+iommu_flush_write_buffer (iommu); +continue ; + }
the comment really doesn't tell what this block does. from its intention it probably more suitable to be in the caller side as today.
Quoted 19 lines
drivers/iommu/intel/cache.c +10
-0+ +if (tag->type ==CACHE_TAG_TYPE_IOTLB || + tag->type ==CACHE_TAG_TYPE_PARENT_IOTLB ) { +/* +* Fallback to domain selective flush if no +* PSI support or the size is too big. +*/ +if (!cap_pgsel_inv (iommu->cap) || + mask >cap_max_amask_val (iommu->cap)) + iommu->flush.flush_iotlb(iommu, tag-Quoted 1 line
domain_id,(quoted diff) +1
-0+ 0, 0,
DMA_TLB_DSI_FLUSH);(quoted diff) +2
-0+ else + iommu->flush.flush_iotlb(iommu, tag-Quoted 1 line
domain_id,(quoted diff) +2
-0+ addr, mask, +
DMA_TLB_PSI_FLUSH);
Review
Quoted 2 lines
(quoted diff) +2
-0+ } + }
You skipped devtlb invalidation. yes it's not necessary based on current nested translation design and this part is inconsistent in different paths.
but as a semantics change you may want to first make removing devtlb invalidation a separate patch and then do this cleanup, so bisect is easier.
Baolu Lu
maintainerRe: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers
Quoted 13 lines · Kevin
Quoted 11 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMdrivers/iommu/intel/cache.c +7
-0+ +static unsigned long calculate_psi_aligned_address (unsigned long start, +unsigned long end, +unsigned long *_pages, +unsigned long *_mask) +{ + unsigned long pages = (end - start + VTD_PAGE_SIZE - 1) >>
VTD_PAGE_SHIFT;this doesn't sound correct. You'd want to follow
aligned_nrpages().
Okay, I will make it like below.
unsigned long pages = aligned_nrpages(start, end - start + 1);
Quoted 36 lines
Quoted 31 lines
drivers/iommu/intel/cache.c +4
-0+case CACHE_TAG_TYPE_DEVTLB : +if (tag->pasid ==IOMMU_NO_PASID ) + qi_flush_dev_iotlb(iommu, sid, info->pfsid, + info->ats_qdep, addr,mask);
(quoted diff) +2
-0+ else + qi_flush_dev_iotlb_pasid(iommu, sid, info-Quoted 1 line
pfsid,
(quoted diff) +1
-0+ tag->pasid, info-Quoted 1 line
ats_qdep,drivers/iommu/intel/cache.c +3
-0+ addr, mask); + + quirk_extra_dev_tlb_flush(info, addr, mask, tag-Quoted 1 line
pasid,
info->ats_qdep);drivers/iommu/intel/cache.c +10
-0+break ; +case CACHE_TAG_TYPE_PARENT_DEVTLB : +/* +* Address translation cache in device side caches the +* result of nested translation. There is no easy way +* to identify the exact set of nested translations +* affected by a change in S2. So just flush the entire +* device cache. +*/ + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info-Quoted 1 line
ats_qdep,(quoted diff) +2
-0+ 0, MAX_AGAW_PFN_WIDTH); + quirk_extra_dev_tlb_flush(info, 0,
MAX_AGAW_PFN_WIDTH,(quoted diff) +1
-0+ IOMMU_NO_PASID, info-Quoted 1 line
ats_qdep);Review
Quoted 1 line
(quoted diff) +1
-0+ break;parent devtlb can has pasid too, though not enabled now. As core helpers probably we can support it now leading to simpler code:
Quoted 5 lines
case CACHE_TAG_TYPE_PARENT_DEVTLB: //change addr/mask //fallthrough case CACHE_TAG_TYPE_DEVTLB: //what this patch does
Yes. Yours is better. I will make it like below:
+ case CACHE_TAG_PARENT_DEVTLB:
+ /*
+ * Address translation cache in device side caches the + * result of nested translation. There is no easy way + * to identify the exact set of nested translations + * affected by a change in S2. So just flush the entire + * device cache. + */ + addr = 0; + mask = MAX_AGAW_PFN_WIDTH; + fallthrough;
case CACHE_TAG_DEVTLB:
Quoted 40 lines
Quoted 33 lines
drivers/iommu/intel/cache.c +8
-0+void cache_tag_flush_all (struct dmar_domain *domain) +{ +struct cache_tag *tag; +unsigned long flags; + +spin_lock_irqsave (&domain->cache_lock, flags); +list_for_each_entry (tag, &domain->cache_tags, node) { + struct device_domain_info *info = dev_iommu_priv_get(tag-Quoted 1 line
dev);
drivers/iommu/intel/cache.c +11
-0+struct intel_iommu *iommu = tag->iommu; +u16 sid =PCI_DEVID (info->bus, info->devfn); + +switch (tag->type) { +case CACHE_TAG_TYPE_IOTLB : +case CACHE_TAG_TYPE_PARENT_IOTLB : +if (domain->use_first_level) +qi_flush_piotlb (iommu, tag->domain_id, + tag->pasid,0 ,-1 ,0 ); +else + iommu->flush.flush_iotlb(iommu, tag-Quoted 1 line
domain_id,(quoted diff) +1
-0+ 0, 0,
DMA_TLB_DSI_FLUSH);drivers/iommu/intel/cache.c +4
-0+break ; +case CACHE_TAG_TYPE_DEVTLB : +case CACHE_TAG_TYPE_PARENT_DEVTLB : + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info-Quoted 1 line
ats_qdep,(quoted diff) +2
-0+ 0, MAX_AGAW_PFN_WIDTH); + quirk_extra_dev_tlb_flush(info, 0,
MAX_AGAW_PFN_WIDTH,(quoted diff) +1
-0+ IOMMU_NO_PASID, info-Quoted 1 line
ats_qdep);drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 296
Quoted 3 lines
drivers/iommu/intel/cache.c +3
-0+break ; + } + }could this loop be consolidated with the one in previous helper? anyway the only difference is on addr/mask...
Quoted 12 lines
drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 343
Quoted 9 lines
+/*
+ * Invalidate a range of IOVA when IOMMU is in caching mode and new mappings
+ * are added to the target domain.
+ */
+void cache_tag_flush_cm_range(structdmar_domain*domain, unsigned long start,drivers/iommu/intel/cache.c +2
-0+ unsigned long end) +{I'm also not sure why this is worth a separate helper. why couldn't it be managed by previous
flush_range()?
This is only my preference. I'd like to separate things belonging to different paths, so that it's easier for maintenance. For example, if, in the future, we need to add or enhance something for a specific case, we don't need to care about other cases.
Quoted 18 lines
Quoted 18 lines
drivers/iommu/intel/cache.c +12
-0+unsigned long pages, mask, addr; +struct cache_tag *tag; +unsigned long flags; + + addr =calculate_psi_aligned_address (start, end, &pages, &mask); + +spin_lock_irqsave (&domain->cache_lock, flags); +list_for_each_entry (tag, &domain->cache_tags, node) { +struct intel_iommu *iommu = tag->iommu; + +/* +* When IOMMU is enabled in caching mode some non- present
drivers/iommu/intel/cache.c +4
-0+* mappings may be cached by the IOTLB if it uses second- +* only translation. +*/ + if (!cap_caching_mode(iommu->cap) || domain-
Quoted 7 lines
drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 362
Quoted 4 lines
Quoted 1 line
use_first_level) {drivers/iommu/intel/cache.c +3
-0+iommu_flush_write_buffer (iommu); +continue ; + }the comment really doesn't tell what this block does. from its intention it probably more suitable to be in the caller side as today.
Yes, agreed. Will remove it.
Quoted 26 lines
Quoted 20 lines
drivers/iommu/intel/cache.c +10
-0+ +if (tag->type ==CACHE_TAG_TYPE_IOTLB || + tag->type ==CACHE_TAG_TYPE_PARENT_IOTLB ) { +/* +* Fallback to domain selective flush if no +* PSI support or the size is too big. +*/ +if (!cap_pgsel_inv (iommu->cap) || + mask >cap_max_amask_val (iommu->cap)) + iommu->flush.flush_iotlb(iommu, tag-Quoted 1 line
domain_id,(quoted diff) +1
-0+ 0, 0,
DMA_TLB_DSI_FLUSH);(quoted diff) +2
-0+ else + iommu->flush.flush_iotlb(iommu, tag-Quoted 1 line
domain_id,(quoted diff) +2
-0+ addr, mask, +
DMA_TLB_PSI_FLUSH);Review
Quoted 2 lines
(quoted diff) +2
-0+ } + }You skipped devtlb invalidation. yes it's not necessary based on current nested translation design and this part is inconsistent in different paths.
Quoted 2 lines
but as a semantics change you may want to first make removing devtlb invalidation a separate patch and then do this cleanup, so bisect is easier.
This helper is designed for map paths when the IOMMU is in caching mode. Caching mode doesn't impact the device TLB, so we should not invalidate the device TLB here.
I guess the confusing thing is about the code below.
/*
* In caching mode, changes of pages from non-present to
present require
* flush. However, device IOTLB doesn't need to be flushed in
this case.
*/
if (!cap_caching_mode(iommu->cap) || !map)
iommu_flush_dev_iotlb(domain, addr, mask);The real code doesn't match the comments, and I think the comments describe the right thing. So perhaps the code should be changed to match the comments.
if (!map) iommu_flush_dev_iotlb(domain, addr, mask);
??
Best regards,
baolu
RE: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers
Quoted 19 lines
From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 1:33 PMQuoted 15 lines · Kevin
drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 343
Quoted 12 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM+/*
+ * Invalidate a range of IOVA when IOMMU is in caching mode and new mappings
+ * are added to the target domain.
+ */
+void cache_tag_flush_cm_range(structdmar_domain*domain, unsigned long start,drivers/iommu/intel/cache.c +2
-0+ unsigned long end) +{I'm also not sure why this is worth a separate helper. why couldn't it be managed by previous
flush_range()?
Quoted 4 lines
This is only my preference. I'd like to separate things belonging to different paths, so that it's easier for maintenance. For example, if, in the future, we need to add or enhance something for a specific case, we don't need to care about other cases.
IMHO caching mode is an attribute in low level iommu which can be handled perfectly well within the helper by checking that attribute.
it sounds a bit weird for the caller to know that detail and call different helpers when all paths just want to request to flush a specific range.
Quoted 44 lines
Quoted 25 lines
Quoted 19 lines
drivers/iommu/intel/cache.c +10
-0+ +if (tag->type ==CACHE_TAG_TYPE_IOTLB || + tag->type ==CACHE_TAG_TYPE_PARENT_IOTLB ) { +/* +* Fallback to domain selective flush if no +* PSI support or the size is too big. +*/ +if (!cap_pgsel_inv (iommu->cap) || + mask >cap_max_amask_val (iommu->cap)) + iommu->flush.flush_iotlb(iommu, tag-Quoted 1 line
domain_id,(quoted diff) +1
-0+ 0, 0,
DMA_TLB_DSI_FLUSH);(quoted diff) +2
-0+ else + iommu->flush.flush_iotlb(iommu, tag-Quoted 1 line
domain_id,(quoted diff) +2
-0+ addr, mask, +
DMA_TLB_PSI_FLUSH);Review
Quoted 2 lines
(quoted diff) +2
-0+ } + }You skipped devtlb invalidation. yes it's not necessary based on current nested translation design and this part is inconsistent in different paths.
Quoted 2 lines
but as a semantics change you may want to first make removing devtlb invalidation a separate patch and then do this cleanup, so bisect is easier.
This helper is designed for map paths when the IOMMU is in caching mode. Caching mode doesn't impact the device TLB, so we should not invalidate the device TLB here.
I guess the confusing thing is about the code below.
/*
* In caching mode, changes of pages from non-present to
present require
* flush. However, device IOTLB doesn't need to be flushed in
this case.*/ if (!cap_caching_mode(iommu->cap) || !map) iommu_flush_dev_iotlb(domain, addr, mask);
Quoted 9 lines
The real code doesn't match the comments, and I think the comments describe the right thing. So perhaps the code should be changed to match the comments.
if (!map) iommu_flush_dev_iotlb(domain, addr, mask);??
yes. that should be fixed separately.
Baolu Lu
maintainerRe: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers
Quoted 22 lines · Kevin
Quoted 18 lines
From: Baolu Lu<[email protected]>
Sent: Sunday, April 7, 2024 1:33 PMQuoted 14 lines · Kevin
drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 343
Quoted 12 lines
From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AM+/*
+ * Invalidate a range of IOVA when IOMMU is in caching mode and new mappings
+ * are added to the target domain.
+ */
+void cache_tag_flush_cm_range(structdmar_domain*domain, unsigned long start,drivers/iommu/intel/cache.c +2
-0+ unsigned long end) +{I'm also not sure why this is worth a separate helper. why couldn't it be managed by previous
flush_range()?Quoted 4 lines
This is only my preference. I'd like to separate things belonging to different paths, so that it's easier for maintenance. For example, if, in the future, we need to add or enhance something for a specific case, we don't need to care about other cases.
Quoted 5 lines
IMHO caching mode is an attribute in low level iommu which can be handled perfectly well within the helper by checking that attribute.
it sounds a bit weird for the caller to know that detail and call different helpers when all paths just want to request to flush a specific range.
I see. The helper name is a bit confusing.
Generally speaking, cache_tag_flush_range() and cache_tag_flush_all() are designed to flush caches for mapping change from present to non- present. While cache_tag_flush_cm_range() is designed to flush caches for mapping change from non-present to present.
How about renaming these helpers to
cache_tag_flush_present_range()cache_tag_flush_present_all()cache_tag_flush_non_present_range()
?
In cache_tag_flush_non_present_range(),
- if IOMMU is not in caching mode, flush the write buffer if necessary, or it's a no-op.
- if IOMMU is in caching mode, flush the IOTLB caches.
Best regards,
baolu
RE: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers
Quoted 41 lines
From: Baolu Lu <[email protected]>
Sent: Monday, April 8, 2024 10:54 AMQuoted 9 lines · Kevin
Quoted 9 lines
From: Baolu Lu<[email protected]>
Sent: Sunday, April 7, 2024 1:33 PMQuoted 5 lines · Kevin
Quoted 5 lines
From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AM+/*
+ * Invalidate a range of IOVA when IOMMU is in caching mode andnew
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mappings
+ * are added to the target domain.
+ */
+void cache_tag_flush_cm_range(structdmar_domain*domain,unsigned
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
drivers/iommu/intel/cache.c void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, · line 343
Quoted 3 lines
long start,
drivers/iommu/intel/cache.c +2
-0+ unsigned long end) +{I'm also not sure why this is worth a separate helper. why couldn't it be managed by previous
flush_range()?This is only my preference. I'd like to separate things belonging to different paths, so that it's easier for maintenance. For example, if, in the future, we need to add or enhance something for a specific case, we don't need to care about other cases.
Quoted 5 lines
IMHO caching mode is an attribute in low level iommu which can be handled perfectly well within the helper by checking that attribute.
it sounds a bit weird for the caller to know that detail and call different helpers when all paths just want to request to flush a specific range.
I see. The helper name is a bit confusing.
Generally speaking,
cache_tag_flush_range()andcache_tag_flush_all()are designed to flush caches for mapping change from present to non- present. Whilecache_tag_flush_cm_range()is designed to flush caches for mapping change from non-present to present.
Quoted 5 lines
How about renaming these helpers to
cache_tag_flush_present_range()cache_tag_flush_present_all()cache_tag_flush_non_present_range()
I'll probably keep cache_tag_flush_range/all() as it is because the naming matches the general interpretation of the tlb semantics.
then call the 3rd one as cache_tag_flush_range_np() for various additional requirements on transitions from non-present entry.
Quoted 6 lines
?
In
cache_tag_flush_non_present_range(),- if IOMMU is not in caching mode, flush the write buffer if necessary,
or if using stage-1
Quoted 5 lines
or it's a no-op.
- if IOMMU is in caching mode, flush the IOTLB caches.Best regards,
baolu
Lu Baolu
maintainerPATCH 3/12 iommu/vt-d: Add trace events for cache tag interface
In reply to Lu BaoluAdd trace events for cache tag assign/unassign/flush operations and trace the events in the interfaces. These trace events will improve debugging capabilities by providing detailed information about cache tag activity. A sample of the traced messages looks like below [messages have been stripped and wrapped to make the line short].
cache_tag_assign: dmar9/0000:00:01.0 type iotlb did 1 pasid 9 ref 1
cache_tag_assign: dmar9/0000:00:01.0 type devtlb did 1 pasid 9 ref 1
cache_tag_flush_all: dmar6/0000:8a:00.0 type iotlb did 7 pasid 0 ref 1
cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9
[0xeab00000-0xeab1afff] addr 0xeab00000 pages 0x20 mask 0x5 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xeab20000-0xeab31fff] addr 0xeab20000 pages 0x20 mask 0x5 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xeaa40000-0xeaa51fff] addr 0xeaa40000 pages 0x20 mask 0x5 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0x98de0000-0x98de4fff] addr 0x98de0000 pages 0x8 mask 0x3 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xe9828000-0xe9828fff] addr 0xe9828000 pages 0x1 mask 0x0
cache_tag_unassign: dmar9/0000:00:01.0 type iotlb did 1 pasid 9 ref 1
cache_tag_unassign: dmar9/0000:00:01.0 type devtlb did 1 pasid 9 ref 1
--- drivers/iommu/intel/trace.h | 97 +++++++++++++++++++++++++++++++++++++ drivers/iommu/intel/cache.c | 8 +++ 2 files changed, 105 insertions(+)
drivers/iommu/intel/trace.h +97 -0
@@ -89,6 +89,103 @@ TRACE_EVENT(prq_report, __entry->dw1, __entry->dw2, __entry->dw3) ) ); + +DECLARE_EVENT_CLASS (cache_tag_log, + TP_PROTO(struct cache_tag *tag), +TP_ARGS (tag), +TP_STRUCT__entry ( + __string(iommu, tag->iommu->name) + __string(dev, dev_name(tag->dev)) + __field(u16, type) + __field(u16, domain_id) + __field(u32, pasid) +__field (u32, users) + ), +TP_fast_assign ( + __assign_str(iommu, tag->iommu->name); + __assign_str(dev, dev_name(tag->dev)); + __entry->type = tag->type; + __entry->domain_id = tag->domain_id; + __entry->pasid = tag->pasid; + __entry->users = tag->users; + ), +TP_printk ("%s/%s type %s did %d pasid %d ref %d" , +__get_str (iommu),__get_str (dev), +__print_symbolic (__entry->type, + { CACHE_TAG_TYPE_IOTLB, "iotlb" }, + { CACHE_TAG_TYPE_DEVTLB, "devtlb" }, + { CACHE_TAG_TYPE_PARENT_IOTLB, "parent_iotlb" }, + { CACHE_TAG_TYPE_PARENT_DEVTLB, "parent_devtlb" }), + __entry->domain_id, __entry->pasid, __entry->users + ) +); + +DEFINE_EVENT (cache_tag_log, cache_tag_assign, + TP_PROTO(struct cache_tag *tag), +TP_ARGS (tag) +); + +DEFINE_EVENT (cache_tag_log, cache_tag_unassign, + TP_PROTO(struct cache_tag *tag), +TP_ARGS (tag) +); + +DEFINE_EVENT (cache_tag_log, cache_tag_flush_all, + TP_PROTO(struct cache_tag *tag), +TP_ARGS (tag) +); + +DECLARE_EVENT_CLASS (cache_tag_flush, + TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, + unsigned long addr, unsigned long pages, unsigned long mask), +TP_ARGS (tag, start, end, addr, pages, mask), +TP_STRUCT__entry ( + __string(iommu, tag->iommu->name) + __string(dev, dev_name(tag->dev)) + __field(u16, type) + __field(u16, domain_id) + __field(u32, pasid) + __field(unsigned long, start) + __field(unsigned long, end) + __field(unsigned long, addr) + __field(unsigned long, pages) + __field(unsigned long, mask) + ), +TP_fast_assign ( + __assign_str(iommu, tag->iommu->name); + __assign_str(dev, dev_name(tag->dev)); + __entry->type = tag->type; + __entry->domain_id = tag->domain_id; + __entry->pasid = tag->pasid; + __entry->start = start; + __entry->end = end; + __entry->addr = addr; + __entry->pages = pages; + __entry->mask = mask; + ), +TP_printk ("%s %s[%d] type %s did %d [0x%lx-0x%lx] addr 0x%lx pages 0x%lx mask 0x%lx" , +__get_str (iommu),__get_str (dev), __entry->pasid, +__print_symbolic (__entry->type, + { CACHE_TAG_TYPE_IOTLB, "iotlb" }, + { CACHE_TAG_TYPE_DEVTLB, "devtlb" }, + { CACHE_TAG_TYPE_PARENT_IOTLB, "parent_iotlb" }, + { CACHE_TAG_TYPE_PARENT_DEVTLB, "parent_devtlb" }), + __entry->domain_id, __entry->start, __entry->end, + __entry->addr, __entry->pages, __entry->mask + ) +); + +DEFINE_EVENT (cache_tag_flush, cache_tag_flush_range, + TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, + unsigned long addr, unsigned long pages, unsigned long mask), +TP_ARGS (tag, start, end, addr, pages, mask) +); + +DEFINE_EVENT (cache_tag_flush, cache_tag_flush_cm_range, + TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, + unsigned long addr, unsigned long pages, unsigned long mask), +TP_ARGS (tag, start, end, addr, pages, mask) +);#endif /* _TRACE_INTEL_IOMMU_H */ /* This part must be outside protection */
drivers/iommu/intel/cache.c +8 -0
@@ -17,6 +17,7 @@#include "iommu.h" #include "pasid.h" +#include "trace.h" /* Checks if an existing cache tag can be reused for a new association. */ static bool cache_tag_reusable(struct cache_tag *tag, u16 domain_id,
@@ -65,11 +66,13 @@ static int cache_tag_assign(struct dmar_domain *domain, u16 did, temp->users++;spin_unlock_irqrestore (&domain->cache_lock, flags);kfree (tag); +trace_cache_tag_assign (temp);return 0 ; } }list_add_tail (&tag->node, &domain->cache_tags);spin_unlock_irqrestore (&domain->cache_lock, flags); +trace_cache_tag_assign (tag);return 0 ; }
@@ -87,6 +90,7 @@ static void cache_tag_unassign(struct dmar_domain *domain, u16 did,spin_lock_irqsave (&domain->cache_lock, flags);list_for_each_entry (tag, &domain->cache_tags, node) {if (cache_tag_reusable (tag, did, iommu, dev, pasid, type)) { +trace_cache_tag_unassign (tag);if (--tag->users ==0 ) {list_del (&tag->node);kfree (tag);
@@ -293,6 +297,8 @@ void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start, IOMMU_NO_PASID, info->ats_qdep);break ; } + +trace_cache_tag_flush_range (tag, start, end, addr, pages, mask); }spin_unlock_irqrestore (&domain->cache_lock, flags); }
@@ -330,6 +336,7 @@ void cache_tag_flush_all(struct dmar_domain *domain) IOMMU_NO_PASID, info->ats_qdep);break ; } +trace_cache_tag_flush_all (tag); }spin_unlock_irqrestore (&domain->cache_lock, flags); }
@@ -376,6 +383,7 @@ void cache_tag_flush_cm_range(struct dmar_domain *domain, unsigned long start, addr, mask, DMA_TLB_PSI_FLUSH); } +trace_cache_tag_flush_cm_range (tag, start, end, addr, pages, mask); }spin_unlock_irqrestore (&domain->cache_lock, flags); }
Signature · 1 line
2.34.1
Lu Baolu
maintainerPATCH 4/12 iommu/vt-d: Use cache_tag_flush_all() in flush_iotlb_all
In reply to Lu BaoluThe flush_iotlb_all callback is called by the iommu core to flush all caches for the affected domain. Use cache_tag_flush_all() in this callback.
--- drivers/iommu/intel/iommu.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)
drivers/iommu/intel/iommu.c +1 -20
@@ -1560,26 +1560,7 @@ static void parent_domain_flush(struct dmar_domain *domain,static void intel_flush_iotlb_all (struct iommu_domain *domain) { -struct dmar_domain *dmar_domain =to_dmar_domain (domain); -struct iommu_domain_info *info; -unsigned long idx; - -xa_for_each (&dmar_domain->iommu_array, idx, info) { -struct intel_iommu *iommu = info->iommu; -u16 did =domain_id_iommu (dmar_domain, iommu); - -if (dmar_domain->use_first_level) -domain_flush_pasid_iotlb (iommu, dmar_domain,0 ,-1 ,0 ); -else - iommu->flush.flush_iotlb (iommu, did,0 ,0 , -DMA_TLB_DSI_FLUSH ); - -if (!cap_caching_mode (iommu->cap)) -iommu_flush_dev_iotlb (dmar_domain,0 ,MAX_AGAW_PFN_WIDTH ); - } - -if (dmar_domain->nested_parent) -parent_domain_flush (dmar_domain,0 ,-1 ,0 ); +cache_tag_flush_all (to_dmar_domain (domain)); }static void iommu_disable_protect_mem_regions (struct intel_iommu *iommu)
Signature · 1 line
2.34.1
RE: [PATCH 04/12] iommu/vt-d: Use cache_tag_flush_all() in flush_iotlb_all
Quoted 36 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMThe
flush_iotlb_allcallback is called by the iommu core to flush all caches for the affected domain. Usecache_tag_flush_all()in this callback.Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)drivers/iommu/intel/iommu.c +0
-14@@ -1560,26 +1560,7 @@ static void parent_domain_flush(struct dmar_domain *domain, static void intel_flush_iotlb_all(struct iommu_domain *domain) { - struct dmar_domain *dmar_domain = to_dmar_domain(domain); -struct iommu_domain_info *info; -unsigned long idx; - -xa_for_each (&dmar_domain->iommu_array, idx, info) { -struct intel_iommu *iommu = info->iommu; -u16 did =domain_id_iommu (dmar_domain, iommu); - -if (dmar_domain->use_first_level) -domain_flush_pasid_iotlb (iommu, dmar_domain,0 , -1 ,0 ); -else - iommu->flush.flush_iotlb (iommu, did,0 ,0 , -DMA_TLB_DSI_FLUSH );
drivers/iommu/intel/iommu.c static void parent_domain_flush(struct · line 1564
Quoted 10 lines
drivers/iommu/intel/iommu.c +1
-7- -if (!cap_caching_mode (iommu->cap)) -iommu_flush_dev_iotlb (dmar_domain,0 , MAX_AGAW_PFN_WIDTH); - } - -if (dmar_domain->nested_parent) -parent_domain_flush (dmar_domain,0 ,-1 ,0 ); +cache_tag_flush_all (to_dmar_domain (domain)); }
this replacement causes a functional change. Now devtlb is always invalidated while old code doesn't do so for caching mode.
Probably you may want to first clean up all inconsistent devtlb invalidation policies for caching mode before going to this series...
Baolu Lu
maintainerRe: [PATCH 04/12] iommu/vt-d: Use cache_tag_flush_all() in flush_iotlb_all
Quoted 41 lines · Kevin
Quoted 41 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMThe
flush_iotlb_allcallback is called by the iommu core to flush all caches for the affected domain. Usecache_tag_flush_all()in this callback.Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)drivers/iommu/intel/iommu.c +0
-19@@ -1560,26 +1560,7 @@ static void parent_domain_flush(structdmar_domain *domain, static void intel_flush_iotlb_all(struct iommu_domain *domain) { - struct dmar_domain *dmar_domain = to_dmar_domain(domain); -struct iommu_domain_info *info; -unsigned long idx; - -xa_for_each (&dmar_domain->iommu_array, idx, info) { -struct intel_iommu *iommu = info->iommu; -u16 did =domain_id_iommu (dmar_domain, iommu); - -if (dmar_domain->use_first_level) -domain_flush_pasid_iotlb (iommu, dmar_domain,0 , -1 ,0 ); -else - iommu->flush.flush_iotlb (iommu, did,0 ,0 , -DMA_TLB_DSI_FLUSH ); - -if (!cap_caching_mode (iommu->cap)) -iommu_flush_dev_iotlb (dmar_domain,0 , MAX_AGAW_PFN_WIDTH); - } -
Quoted 8 lines
drivers/iommu/intel/iommu.c static void parent_domain_flush(struct dmar_domain *domain, · line 1564
Quoted 5 lines
drivers/iommu/intel/iommu.c +1
-2-if (dmar_domain->nested_parent) -parent_domain_flush (dmar_domain,0 ,-1 ,0 ); +cache_tag_flush_all (to_dmar_domain (domain)); }this replacement causes a functional change. Now devtlb is always invalidated while old code doesn't do so for caching mode.
You are right.
As my understanding of the VT-d spec, caching mode has nothing to do with the device TLB, hence we should remove all checks on caching mode before a device TLB invalidation.
I understand that people are leveraging the cache mode to avoid the unnecessary VMEXIT from the guest OS since the host iommu unmap() path ensures that device TLB is also invalidated.
Our direction should be to use the batched queue invalidation requests. As we are consolidating code to a common place, that becomes much easier. :-)
Quoted 2 lines
Probably you may want to first clean up all inconsistent devtlb invalidation policies for caching mode before going to this series...
Sure thing!
Best regards,
baolu
Lu Baolu
maintainerPATCH 5/12 iommu/vt-d: Use cache_tag_flush_range() in tlb_sync
In reply to Lu BaoluThe tlb_sync callback is called by the iommu core to flush a range of caches for the affected domain. Use cache_tag_flush_range() in this callback.
--- drivers/iommu/intel/iommu.c | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-)
drivers/iommu/intel/iommu.c +2 -19
@@ -4112,25 +4112,8 @@ static size_t intel_iommu_unmap_pages(struct iommu_domain *domain,static void intel_iommu_tlb_sync (struct iommu_domain *domain,struct iommu_iotlb_gather *gather) { -struct dmar_domain *dmar_domain =to_dmar_domain (domain); -unsigned long iova_pfn =IOVA_PFN (gather->start); -size_t size = gather->end - gather->start; -struct iommu_domain_info *info; -unsigned long start_pfn; -unsigned long nrpages; -unsigned long i; - - nrpages =aligned_nrpages (gather->start, size); - start_pfn =mm_to_dma_pfn_start (iova_pfn); - -xa_for_each (&dmar_domain->iommu_array, i, info) -iommu_flush_iotlb_psi (info->iommu, dmar_domain, - start_pfn, nrpages, -list_empty (&gather->freelist),0 ); - -if (dmar_domain->nested_parent) -parent_domain_flush (dmar_domain, start_pfn, nrpages, -list_empty (&gather->freelist)); +cache_tag_flush_range (to_dmar_domain (domain), gather->start, + gather->end,list_empty (&gather->freelist));put_pages_list (&gather->freelist); }
Signature · 1 line
2.34.1
Lu Baolu
maintainerPATCH 6/12 iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map
In reply to Lu BaoluThe iotlb_sync_map callback is called by the iommu core after non-present to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Use cache_tag_flush_cm_range() in this callback.
--- drivers/iommu/intel/iommu.c | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-)
drivers/iommu/intel/iommu.c +1 -21
@@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(struct intel_iommu *iommu,iommu_flush_dev_iotlb (domain, addr, mask); } -/* Notification for newly created mappings */ -static void __mapping_notify_one (struct intel_iommu *iommu,struct dmar_domain *domain, -unsigned long pfn,unsigned int pages) -{ -/* -* It's a non-present to present mapping. Only flush if caching mode -* and second level. -*/ -if (cap_caching_mode (iommu->cap) && !domain->use_first_level) -iommu_flush_iotlb_psi (iommu, domain, pfn, pages,0 ,1 ); -else -iommu_flush_write_buffer (iommu); -} -/* * Flush the relevant caches in nested translation if the domain * also serves as a parent
@@ -4549,14 +4535,8 @@ static bool risky_device(struct pci_dev *pdev)static int intel_iommu_iotlb_sync_map (struct iommu_domain *domain,unsigned long iova,size_t size) { -struct dmar_domain *dmar_domain =to_dmar_domain (domain); -unsigned long pages =aligned_nrpages (iova, size); -unsigned long pfn = iova >>VTD_PAGE_SHIFT ; -struct iommu_domain_info *info; -unsigned long i; +cache_tag_flush_cm_range (to_dmar_domain (domain), iova, iova + size -1 ); -xa_for_each (&dmar_domain->iommu_array, i, info) -__mapping_notify_one (info->iommu, dmar_domain, pfn, pages);return 0 ; }
Signature · 1 line
2.34.1
RE: [PATCH 06/12] iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map
Quoted 27 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMThe
iotlb_sync_mapcallback is called by the iommu core after non-present to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Usecache_tag_flush_cm_range()in this callback.Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.c | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-)drivers/iommu/intel/iommu.c +0
-3@@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(structintel_iommu *iommu,iommu_flush_dev_iotlb (domain ,addr ,mask ); } -/* Notification for newly created mappings */ -static void __mapping_notify_one (struct intel_iommu *iommu,struct dmar_domain *domain, -unsigned long pfn,unsigned int pages)
drivers/iommu/intel/iommu.c static void iommu_flush_iotlb_psi(struct · line 1521
Quoted 10 lines
drivers/iommu/intel/iommu.c +0
-10-{ -/* -* It's a non-present to present mapping. Only flush if caching mode -* and second level. -*/ -if (cap_caching_mode (iommu->cap) && !domain->use_first_level) -iommu_flush_iotlb_psi (iommu, domain, pfn, pages,0 ,1 ); -else -iommu_flush_write_buffer (iommu); -}
iommu_flush_write_buffer is for a quite different issue. it's clearer to keep it separated from the iotlb helpers.
Baolu Lu
maintainerRe: [PATCH 06/12] iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map
Quoted 32 lines · Kevin
Quoted 32 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMThe
iotlb_sync_mapcallback is called by the iommu core after non-present to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Usecache_tag_flush_cm_range()in this callback.Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.c | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-)drivers/iommu/intel/iommu.c +0
-8@@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(structintel_iommu *iommu,iommu_flush_dev_iotlb (domain ,addr ,mask ); } -/* Notification for newly created mappings */ -static void __mapping_notify_one (struct intel_iommu *iommu,struct dmar_domain *domain, -unsigned long pfn,unsigned int pages) -{ -/* -* It's a non-present to present mapping. Only flush if caching mode -* and second level. -*/
Quoted 8 lines
drivers/iommu/intel/iommu.c static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, · line 1519
Quoted 5 lines
drivers/iommu/intel/iommu.c +0
-5-if (cap_caching_mode (iommu->cap) && !domain->use_first_level) -iommu_flush_iotlb_psi (iommu, domain, pfn, pages,0 ,1 ); -else -iommu_flush_write_buffer (iommu); -}
iommu_flush_write_bufferis for a quite different issue. it's clearer to keep it separated from the iotlb helpers.
The VT-d spec describes the write buffer flushing in section 6.8. It states,
1. Write buffer flushing is required only for earlier VT-d hardware implementations. Newer hardware implementations are expected to NOT require this.
2. Write buffer flushing is related to cache invalidation. The hardware performs an implicit write-buffer-flushing as a pre-condition to the invalidation operations.
According to this, write buffer flushing is only required in the map path when IOMMU is not in caching mode and the RWBF iommu capability is set, which perfectly matches the case of cache_tag_flush_cm_range().
By doing it in cache_tag_flush_cm_range(), the code will be simpler and easier for maintenance.
Best regards,
baolu
RE: [PATCH 06/12] iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map
Quoted 57 lines
From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 2:41 PMQuoted 4 lines · Kevin
Quoted 4 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMThe
iotlb_sync_mapcallback is called by the iommu core after non-present
Quoted 3 lines
Quoted 3 lines
to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Use
cache_tag_flush_cm_range()inthis
Quoted 25 lines
Quoted 25 lines
callback.
Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.c | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-)drivers/iommu/intel/iommu.c +0
-8@@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(structintel_iommu *iommu,iommu_flush_dev_iotlb (domain ,addr ,mask ); } -/* Notification for newly created mappings */ -static void __mapping_notify_one (struct intel_iommu *iommu,struct dmar_domain *domain, -unsigned long pfn,unsigned int pages) -{ -/* -* It's a non-present to present mapping. Only flush if caching mode -* and second level. -*/ Quoted 8 lines
drivers/iommu/intel/iommu.c static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, · line 1519
Quoted 5 lines
drivers/iommu/intel/iommu.c +0
-5-if (cap_caching_mode (iommu->cap) && !domain->use_first_level) -iommu_flush_iotlb_psi (iommu, domain, pfn, pages,0 ,1 ); -else -iommu_flush_write_buffer (iommu); -}
iommu_flush_write_bufferis for a quite different issue. it's clearer to keep it separated from the iotlb helpers.The VT-d spec describes the write buffer flushing in section 6.8. It states,
1. Write buffer flushing is required only for earlier VT-d hardware implementations. Newer hardware implementations are expected to NOT require this.
2. Write buffer flushing is related to cache invalidation. The hardware performs an implicit write-buffer-flushing as a pre-condition to the invalidation operations.
Quoted 7 lines
According to this, write buffer flushing is only required in the map path when IOMMU is not in caching mode and the RWBF iommu capability is set, which perfectly matches the case of
cache_tag_flush_cm_range().By doing it in
cache_tag_flush_cm_range(), the code will be simpler and easier for maintenance.
but above applies actually when caching mode is false or when 1st stage is being used. this is the opposite of what flush_cm_range() implies.
if we do it through a general flush_range() helper which accepts a parameter to indicate the map path, then it's reasonable to move this special handling to that helper as that is another trick which the caller doesn't need to know.
Baolu Lu
maintainerRe: [PATCH 06/12] iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map
Quoted 62 lines · Kevin
Quoted 55 lines
From: Baolu Lu<[email protected]>
Sent: Sunday, April 7, 2024 2:41 PMQuoted 4 lines · Kevin
Quoted 4 lines
From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AMThe
iotlb_sync_mapcallback is called by the iommu core after non-present
Quoted 3 lines
Quoted 3 lines
to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Use
cache_tag_flush_cm_range()inthis
Quoted 16 lines
Quoted 16 lines
callback.
Signed-off-by Lu Baolu<[email protected]>--- drivers/iommu/intel/iommu.c | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-)drivers/iommu/intel/iommu.c +0
-0@@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(structintel_iommu *iommu,iommu_flush_dev_iotlb (domain ,addr ,mask ); }Quoted 16 lines
Quoted 4 lines
drivers/iommu/intel/iommu.c +0
-3-/* Notification for newly created mappings */ -static void __mapping_notify_one (struct intel_iommu *iommu,struct dmar_domain *domain, -unsigned long pfn,unsigned int pages)drivers/iommu/intel/iommu.c static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, · line 1519
Quoted 10 lines
drivers/iommu/intel/iommu.c +0
-10-{ -/* -* It's a non-present to present mapping. Only flush if caching mode -* and second level. -*/ -if (cap_caching_mode (iommu->cap) && !domain->use_first_level) -iommu_flush_iotlb_psi (iommu, domain, pfn, pages,0 ,1 ); -else -iommu_flush_write_buffer (iommu); -}
iommu_flush_write_bufferis for a quite different issue. it's clearer to keep it separated from the iotlb helpers.The VT-d spec describes the write buffer flushing in section 6.8. It states,
1. Write buffer flushing is required only for earlier VT-d hardware implementations. Newer hardware implementations are expected to NOT require this.
2. Write buffer flushing is related to cache invalidation. The hardware performs an implicit write-buffer-flushing as a pre-condition to the invalidation operations.Quoted 7 lines
According to this, write buffer flushing is only required in the map path when IOMMU is not in caching mode and the RWBF iommu capability is set, which perfectly matches the case of
cache_tag_flush_cm_range().By doing it in
cache_tag_flush_cm_range(), the code will be simpler and easier for maintenance.
Quoted 8 lines
but above applies actually when caching mode is false or when 1st stage is being used. this is the opposite of what
flush_cm_range()implies.if we do it through a general
flush_range()helper which accepts a parameter to indicate the map path, then it's reasonable to move this special handling to that helper as that is another trick which the caller doesn't need to know.
This could be an option. How about my proposal in another reply?
Best regards,
baolu
Lu Baolu
maintainerPATCH 7/12 iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi()
In reply to Lu BaoluUse cache_tag_flush_range() in switch_to_super_page() to invalidate the necessary caches when switching mappings from normal to super pages. The iommu_flush_iotlb_psi() call in intel_iommu_memory_notifier() is unnecessary since there should be no cache invalidation for the identity domain.
Clean up iommu_flush_iotlb_psi() after the last call site is removed.
--- drivers/iommu/intel/iommu.c | 175 +----------------------------------- 1 file changed, 2 insertions(+), 173 deletions(-)
drivers/iommu/intel/iommu.c +2 -173
@@ -1389,161 +1389,6 @@ static void __iommu_flush_dev_iotlb(struct device_domain_info *info,quirk_extra_dev_tlb_flush (info, addr, mask,IOMMU_NO_PASID , qdep); } -static void iommu_flush_dev_iotlb (struct dmar_domain *domain, -u64 addr,unsigned mask) -{ -struct dev_pasid_info *dev_pasid; -struct device_domain_info *info; -unsigned long flags; - -if (!domain->has_iotlb_device) -return ; - -spin_lock_irqsave (&domain->lock, flags); -list_for_each_entry (info, &domain->devices, link) -__iommu_flush_dev_iotlb (info, addr, mask); - -list_for_each_entry (dev_pasid, &domain->dev_pasids, link_domain) { - info =dev_iommu_priv_get (dev_pasid->dev); - -if (!info->ats_enabled) -continue ; - -qi_flush_dev_iotlb_pasid (info->iommu, -PCI_DEVID (info->bus, info->devfn), - info->pfsid, dev_pasid->pasid, - info->ats_qdep, addr, - mask); - } -spin_unlock_irqrestore (&domain->lock, flags); -} - -static void domain_flush_pasid_iotlb (struct intel_iommu *iommu, -struct dmar_domain *domain,u64 addr, -unsigned long npages,bool ih) -{ -u16 did =domain_id_iommu (domain, iommu); -struct dev_pasid_info *dev_pasid; -unsigned long flags; - -spin_lock_irqsave (&domain->lock, flags); -list_for_each_entry (dev_pasid, &domain->dev_pasids, link_domain) -qi_flush_piotlb (iommu, did, dev_pasid->pasid, addr, npages, ih); - -if (!list_empty (&domain->devices)) -qi_flush_piotlb (iommu, did,IOMMU_NO_PASID , addr, npages, ih); -spin_unlock_irqrestore (&domain->lock, flags); -} - -static void __iommu_flush_iotlb_psi (struct intel_iommu *iommu,u16 did, -unsigned long pfn,unsigned int pages, -int ih) -{ -unsigned int aligned_pages =__roundup_pow_of_two (pages); -unsigned long bitmask = aligned_pages -1 ; -unsigned int mask =ilog2 (aligned_pages); -u64 addr = (u64 )pfn <<VTD_PAGE_SHIFT ; - -/* -* PSI masks the low order bits of the base address. If the -* address isn't aligned to the mask, then compute a mask value -* needed to ensure the target range is flushed. -*/ -if (unlikely (bitmask & pfn)) { -unsigned long end_pfn = pfn + pages -1 , shared_bits; - -/* -* Since end_pfn <= pfn + bitmask, the only way bits -* higher than bitmask can differ in pfn and end_pfn is -* by carrying. This means after masking out bitmask, -* high bits starting with the first set bit in -* shared_bits are all equal in both pfn and end_pfn. -*/ - shared_bits = ~(pfn ^ end_pfn) & ~bitmask; - mask = shared_bits ?__ffs (shared_bits) :BITS_PER_LONG ; - } - -/* -* Fallback to domain selective flush if no PSI support or -* the size is too big. -*/ -if (!cap_pgsel_inv (iommu->cap) || mask >cap_max_amask_val (iommu->cap)) - iommu->flush.flush_iotlb (iommu, did,0 ,0 , -DMA_TLB_DSI_FLUSH ); -else - iommu->flush.flush_iotlb (iommu, did, addr | ih, mask, -DMA_TLB_PSI_FLUSH ); -} - -static void iommu_flush_iotlb_psi (struct intel_iommu *iommu, -struct dmar_domain *domain, -unsigned long pfn,unsigned int pages, -int ih,int map) -{ -unsigned int aligned_pages =__roundup_pow_of_two (pages); -unsigned int mask =ilog2 (aligned_pages); -uint64_t addr = (uint64_t )pfn <<VTD_PAGE_SHIFT ; -u16 did =domain_id_iommu (domain, iommu); - -if (WARN_ON (!pages)) -return ; - -if (ih) - ih =1 <<6 ; - -if (domain->use_first_level) -domain_flush_pasid_iotlb (iommu, domain, addr, pages, ih); -else -__iommu_flush_iotlb_psi (iommu, did, pfn, pages, ih); - -/* -* In caching mode, changes of pages from non-present to present require -* flush. However, device IOTLB doesn't need to be flushed in this case. -*/ -if (!cap_caching_mode (iommu->cap) || !map) -iommu_flush_dev_iotlb (domain, addr, mask); -} - -/* -* Flush the relevant caches in nested translation if the domain -* also serves as a parent -*/ -static void parent_domain_flush (struct dmar_domain *domain, -unsigned long pfn, -unsigned long pages,int ih) -{ -struct dmar_domain *s1_domain; - -spin_lock (&domain->s1_lock); -list_for_each_entry (s1_domain, &domain->s1_domains, s2_link) { -struct device_domain_info *device_info; -struct iommu_domain_info *info; -unsigned long flags; -unsigned long i; - -xa_for_each (&s1_domain->iommu_array, i, info) -__iommu_flush_iotlb_psi (info->iommu, info->did, - pfn, pages, ih); - -if (!s1_domain->has_iotlb_device) -continue ; - -spin_lock_irqsave (&s1_domain->lock, flags); -list_for_each_entry (device_info, &s1_domain->devices, link) -/* -* Address translation cache in device side caches the -* result of nested translation. There is no easy way -* to identify the exact set of nested translations -* affected by a change in S2. So just flush the entire -* device cache. -*/ -__iommu_flush_dev_iotlb (device_info,0 , -MAX_AGAW_PFN_WIDTH ); -spin_unlock_irqrestore (&s1_domain->lock, flags); - } -spin_unlock (&domain->s1_lock); -} -static void intel_flush_iotlb_all (struct iommu_domain *domain) {cache_tag_flush_all (to_dmar_domain (domain));
@@ -1995,9 +1840,7 @@ static void switch_to_super_page(struct dmar_domain *domain, unsigned long end_pfn, int level) {unsigned long lvl_pages =lvl_to_nr_pages (level); -struct iommu_domain_info *info;struct dma_pte *pte =NULL ; -unsigned long i;while (start_pfn <= end_pfn) {if (!pte)
@@ -2009,13 +1852,8 @@ static void switch_to_super_page(struct dmar_domain *domain, start_pfn + lvl_pages -1 , level + 1); -xa_for_each (&domain->iommu_array, i, info) -iommu_flush_iotlb_psi (info->iommu, domain, - start_pfn, lvl_pages, -0 ,0 ); -if (domain->nested_parent) -parent_domain_flush (domain, start_pfn, - lvl_pages,0 ); +cache_tag_flush_range (domain, start_pfn <<VTD_PAGE_SHIFT , + end_pfn <<VTD_PAGE_SHIFT ,0 ); } pte++;
@@ -3387,18 +3225,9 @@ static int intel_iommu_memory_notifier(struct notifier_block *nb,case MEM_OFFLINE :case MEM_CANCEL_ONLINE : { -struct dmar_drhd_unit *drhd; -struct intel_iommu *iommu;LIST_HEAD (freelist);domain_unmap (si_domain, start_vpfn, last_vpfn, &freelist); - -rcu_read_lock (); -for_each_active_iommu (iommu, drhd) -iommu_flush_iotlb_psi (iommu, si_domain, - start_vpfn, mhp->nr_pages, -list_empty (&freelist),0 ); -rcu_read_unlock ();put_pages_list (&freelist); }break ;
Signature · 1 line
2.34.1
RE: [PATCH 07/12] iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi()
Quoted 9 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMUse
cache_tag_flush_range()inswitch_to_super_page()to invalidate the necessary caches when switching mappings from normal to super pages. Theiommu_flush_iotlb_psi()call inintel_iommu_memory_notifier()is unnecessary since there should be no cache invalidation for the identity domain.
what about a software identity domain?
Baolu Lu
maintainerRe: [PATCH 07/12] iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi()
Quoted 11 lines · Kevin
Quoted 9 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMUse
cache_tag_flush_range()inswitch_to_super_page()to invalidate the necessary caches when switching mappings from normal to super pages. Theiommu_flush_iotlb_psi()call inintel_iommu_memory_notifier()is unnecessary since there should be no cache invalidation for the identity domain.what about a software identity domain?
Software identity domain is used to fake the hardware passthrough capability, on early VT-d hardware which doesn't implement the passthrough mode. It's not any kind of protection domain, hence the OS is not required to manage the cache synchronization.
Although I hope we can remove it someday and force the DMA domain instead, we still need to carry it nowadays. However, we need to make it consistent with the hardware passthrough. That is, hardware passthrough doesn't require any cache invalidation in memory hot-plug paths, the software passthrough should not either.
Best regards,
baolu
RE: [PATCH 07/12] iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi()
Quoted 22 lines
From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 3:06 PMQuoted 5 lines · Kevin
Quoted 5 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMUse
cache_tag_flush_range()inswitch_to_super_page()to invalidate the necessary caches when switching mappings from normal to super pages.The
Quoted 6 lines
Quoted 4 lines
iommu_flush_iotlb_psi()call inintel_iommu_memory_notifier()is unnecessary since there should be no cache invalidation for the identity domain.what about a software identity domain?
Software identity domain is used to fake the hardware passthrough capability, on early VT-d hardware which doesn't implement the passthrough mode. It's not any kind of protection domain, hence the OS is not required to manage the cache synchronization.
Quoted 6 lines
Although I hope we can remove it someday and force the DMA domain instead, we still need to carry it nowadays. However, we need to make it consistent with the hardware passthrough. That is, hardware passthrough doesn't require any cache invalidation in memory hot-plug paths, the software passthrough should not either.
yes, that makes sense.
Lu Baolu
maintainerPATCH 8/12 iommu/vt-d: Use cache_tag_flush_range() in cache_invalidate_user
In reply to Lu BaoluThe cache_invalidate_user callback is called to invalidate a range of caches for the affected user domain. Use cache_tag_flush_range() in this callback.
--- drivers/iommu/intel/nested.c | 50 +++--------------------------------- 1 file changed, 3 insertions(+), 47 deletions(-)
drivers/iommu/intel/nested.c +3 -47
@@ -90,50 +90,6 @@ static void intel_nested_domain_free(struct iommu_domain *domain)kfree (dmar_domain); } -static void nested_flush_dev_iotlb (struct dmar_domain *domain,u64 addr, -unsigned int mask) -{ -struct device_domain_info *info; -unsigned long flags; -u16 sid, qdep; - -spin_lock_irqsave (&domain->lock, flags); -list_for_each_entry (info, &domain->devices, link) { -if (!info->ats_enabled) -continue ; - sid = info->bus <<8 | info->devfn; - qdep = info->ats_qdep; -qi_flush_dev_iotlb (info->iommu, sid, info->pfsid, - qdep, addr, mask); -quirk_extra_dev_tlb_flush (info, addr, mask, -IOMMU_NO_PASID , qdep); - } -spin_unlock_irqrestore (&domain->lock, flags); -} - -static void intel_nested_flush_cache (struct dmar_domain *domain,u64 addr, -u64 npages,bool ih) -{ -struct iommu_domain_info *info; -unsigned int mask; -unsigned long i; - -xa_for_each (&domain->iommu_array, i, info) -qi_flush_piotlb (info->iommu, -domain_id_iommu (domain, info->iommu), -IOMMU_NO_PASID , addr, npages, ih); - -if (!domain->has_iotlb_device) -return ; - -if (npages ==U64_MAX ) - mask =64 -VTD_PAGE_SHIFT ; -else - mask =ilog2 (__roundup_pow_of_two (npages)); - -nested_flush_dev_iotlb (domain, addr, mask); -} -static int intel_nested_cache_invalidate_user (struct iommu_domain *domain,struct iommu_user_data_array *array) {
@@ -166,9 +122,9 @@ static int intel_nested_cache_invalidate_user(struct iommu_domain *domain,break ; } -intel_nested_flush_cache (dmar_domain, inv_entry.addr, - inv_entry.npages, - inv_entry.flags &IOMMU_VTD_INV_FLAGS_LEAF ); +cache_tag_flush_range (dmar_domain, inv_entry.addr, + inv_entry.npages, + inv_entry.flags &IOMMU_VTD_INV_FLAGS_LEAF ); processed++; }
Signature · 1 line
2.34.1
RE: [PATCH 08/12] iommu/vt-d: Use cache_tag_flush_range() in cache_invalidate_user
Quoted 6 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMdrivers/iommu/intel/nested.c +0
-0@@ -166,9 +122,9 @@ static int intel_nested_cache_invalidate_user(structiommu_domain *domain, break;
drivers/iommu/intel/nested.c static int intel_nested_cache_invalidate_user(struct · line 130
Quoted 10 lines
drivers/iommu/intel/nested.c +3
-3} -intel_nested_flush_cache (dmar_domain, inv_entry.addr, - inv_entry.npages, - inv_entry.flags & IOMMU_VTD_INV_FLAGS_LEAF); +cache_tag_flush_range (dmar_domain, inv_entry.addr, + inv_entry.npages, + inv_entry.flags &IOMMU_VTD_INV_FLAGS_LEAF );
the helper requires an 'end' address but 'npages' is incorrectly used here.
Baolu Lu
maintainerRe: [PATCH 08/12] iommu/vt-d: Use cache_tag_flush_range() in cache_invalidate_user
Quoted 8 lines · Kevin
Quoted 8 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AM(quoted diff) +0
-0@@ -166,9 +122,9 @@ static int intel_nested_cache_invalidate_user(struct iommu_domain *domain, break; }
Quoted 10 lines
Review static int intel_nested_cache_invalidate_user(struct · line 130
Quoted 8 lines
(quoted diff) +3
-3- intel_nested_flush_cache(dmar_domain, inv_entry.addr, - inv_entry.npages, - inv_entry.flags & IOMMU_VTD_INV_FLAGS_LEAF); + cache_tag_flush_range(dmar_domain, inv_entry.addr, + inv_entry.npages, + inv_entry.flags & IOMMU_VTD_INV_FLAGS_LEAF);the helper requires an 'end' address but 'npages' is incorrectly used here.
Oh yes! Thanks for pointing this out. I will fix it like below.
drivers/iommu/intel/iommu.h +6 -0
@@ -1050,6 +1050,12 @@ static inline unsigned long aligned_nrpages(unsigned long host_addr, size_t sizereturn PAGE_ALIGN (host_addr + size) >>VTD_PAGE_SHIFT ; } +/* Return a size from number of VTD pages. */ +static inline unsigned long nrpages_to_size (unsigned long npages) +{ +return npages <<VTD_PAGE_SHIFT ; +} +/* Convert value to context PASID directory size field coding. */ #define context_pdts (pds) (((pds) & 0x7) << 9)
drivers/iommu/intel/nested.c +1 -1
@@ -123,7 +123,7 @@ static int intel_nested_cache_invalidate_user(struct iommu_domain *domain, } cache_tag_flush_range(dmar_domain, inv_entry.addr, - inv_entry.npages, + inv_entry.addr + nrpages_to_size(inv_entry.npages) - 1, inv_entry.flags &
IOMMU_VTD_INV_FLAGS_LEAF);
processed++;
}Best regards,
baolu
Lu Baolu
maintainerPATCH 9/12 iommu/vt-d: Use cache helpers in arch_invalidate_secondary_tlbs
In reply to Lu BaoluThe arch_invalidate_secondary_tlbs callback is called in the SVA mm notification path. It invalidates all or a range of caches after the CPU page table is modified. Use the cache tag helps in this path.
--- drivers/iommu/intel/iommu.h | 1 + drivers/iommu/intel/svm.c | 76 +++---------------------------------- 2 files changed, 6 insertions(+), 71 deletions(-)
drivers/iommu/intel/iommu.h +1 -0
@@ -1148,6 +1148,7 @@ struct intel_svm {struct mm_struct *mm;u32 pasid;struct list_head devs; + struct dmar_domain *domain; };#else static inline void intel_svm_check (struct intel_iommu *iommu) {}
drivers/iommu/intel/svm.c +5 -71
105 lines not on this page · raw message
Signature · 1 line
2.34.1
Lu Baolu
maintainerPATCH 10/12 iommu/vt-d: Retire intel_svm_dev
In reply to Lu BaoluThe intel_svm_dev data structure used in the sva implementation for the Intel IOMMU driver stores information about a device attached to an SVA domain. It is a duplicate of dev_pasid_info that serves the same purpose.
Replace intel_svm_dev with dev_pasid_info and clean up the use of intel_svm_dev.
--- drivers/iommu/intel/iommu.h | 15 +---- drivers/iommu/intel/iommu.c | 30 ++++----- drivers/iommu/intel/svm.c | 131 +++++++++++------------------------- 3 files changed, 55 insertions(+), 121 deletions(-)
drivers/iommu/intel/iommu.h +3 -12
@@ -650,6 +650,7 @@ struct dmar_domain {struct list_head s2_link; }; }; +struct intel_svm *svm;struct iommu_domain domain;/* generic domain data structure for iommu core */
@@ -1131,23 +1132,13 @@ int intel_svm_finish_prq(struct intel_iommu *iommu);void intel_svm_page_response (struct device *dev,struct iopf_fault *evt,struct iommu_page_response *msg);struct iommu_domain *intel_svm_domain_alloc (void ); -void intel_svm_remove_dev_pasid (struct device *dev,ioasid_t pasid); +void intel_svm_remove_dev_pasid (struct iommu_domain *domain);void intel_drain_pasid_prq (struct device *dev,u32 pasid); -struct intel_svm_dev { -struct list_head list; -struct rcu_head rcu; -struct device *dev; -struct intel_iommu *iommu; -u16 did; -u16 sid, qdep; -}; -struct intel_svm {struct mmu_notifier notifier;struct mm_struct *mm;u32 pasid; -struct list_head devs;struct dmar_domain *domain; };#else
@@ -1158,7 +1149,7 @@ static inline struct iommu_domain *intel_svm_domain_alloc(void) return NULL; } -static inline void intel_svm_remove_dev_pasid (struct device *dev,ioasid_t pasid) +static inline void intel_svm_remove_dev_pasid (struct iommu_domain *domain) { }#endif
drivers/iommu/intel/iommu.c +14 -16
@@ -4384,18 +4384,6 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) goto out_tear_down; dmar_domain =to_dmar_domain (domain); -/* -* The SVA implementation needs to handle its own stuffs like the mm -* notification. Before consolidating that code into iommu core, let -* the intel sva code handle it. -*/ -if (domain->type ==IOMMU_DOMAIN_SVA ) { -intel_svm_remove_dev_pasid (dev, pasid); -cache_tag_unassign_domain (dmar_domain, -FLPT_DEFAULT_DID , dev, pasid); - goto out_tear_down; - } -spin_lock_irqsave (&dmar_domain->lock, flags);list_for_each_entry (curr, &dmar_domain->dev_pasids, link_domain) {if (curr->dev == dev && curr->pasid == pasid) {
@@ -4407,10 +4395,20 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)WARN_ON_ONCE (!dev_pasid);spin_unlock_irqrestore (&dmar_domain->lock, flags); - did =domain_id_iommu (dmar_domain, iommu); -cache_tag_unassign_domain (dmar_domain, did, dev, pasid); -domain_detach_iommu (dmar_domain, iommu); -intel_iommu_debugfs_remove_dev_pasid (dev_pasid); +/* +* The SVA implementation needs to handle its own stuffs like the mm +* notification. Before consolidating that code into iommu core, let +* the intel sva code handle it. +*/ +if (domain->type ==IOMMU_DOMAIN_SVA ) { +cache_tag_unassign_domain (dmar_domain,FLPT_DEFAULT_DID , dev, pasid); +intel_svm_remove_dev_pasid (domain); + }else { + did =domain_id_iommu (dmar_domain, iommu); +cache_tag_unassign_domain (dmar_domain, did, dev, pasid); +domain_detach_iommu (dmar_domain, iommu); +intel_iommu_debugfs_remove_dev_pasid (dev_pasid); + }kfree (dev_pasid); out_tear_down:intel_pasid_tear_down_entry (iommu, dev, pasid, false);
drivers/iommu/intel/svm.c +38 -93
210 lines not on this page · raw message
Signature · 1 line
2.34.1
Lu Baolu
maintainerPATCH 11/12 iommu: Add ops->domain_alloc_sva()
In reply to Lu BaoluFrom: Jason Gunthorpe <[email protected]>
Make a new op that receives the device and the mm_struct that the SVA domain should be created for. Unlike domain_alloc_paging() the dev argument is never NULL here.
This allows drivers to fully initialize the SVA domain and allocate the mmu_notifier during allocation. It allows the notifier lifetime to follow the lifetime of the iommu_domain.
Since we have only one call site, upgrade the new op to return ERR_PTR instead of NULL.
--- include/linux/iommu.h | 3 +++ drivers/iommu/iommu-sva.c | 16 +++++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-)
include/linux/iommu.h +3 -0
@@ -518,6 +518,7 @@ static inline int __iommu_copy_struct_from_user_array(* Upon failure, ERR_PTR must be returned. * @domain_alloc_paging: Allocate an iommu_domain that can be used for * UNMANAGED, DMA, and DMA_FQ domain types. +* @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressing. * @probe_device: Add device to iommu driver handling * @release_device: Remove device from iommu driver handling * @probe_finalize: Do final setup work after the device is added to an IOMMU
@@ -558,6 +559,8 @@ struct iommu_ops { struct device *dev, u32 flags, struct iommu_domain *parent, const struct iommu_user_data *user_data);struct iommu_domain *(*domain_alloc_paging)(struct device *dev); +struct iommu_domain *(*domain_alloc_sva)(struct device *dev, +struct mm_struct *mm);struct iommu_device *(*probe_device)(struct device *dev);void (*release_device)(struct device *dev);
drivers/iommu/iommu-sva.c +11 -5
@@ -108,8 +108,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm/* Allocate a new domain and set it on device pasid. */ domain =iommu_sva_domain_alloc (dev, mm); -if (!domain) { - ret = -ENOMEM ; +if (IS_ERR (domain)) { + ret =PTR_ERR (domain); goto out_free_handle; }
@@ -283,9 +283,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,const struct iommu_ops *ops =dev_iommu_ops (dev);struct iommu_domain *domain; - domain = ops->domain_alloc (IOMMU_DOMAIN_SVA ); -if (!domain) -return NULL ; +if (ops->domain_alloc_sva) { + domain = ops->domain_alloc_sva (dev, mm); +if (IS_ERR (domain)) +return domain; + }else { + domain = ops->domain_alloc (IOMMU_DOMAIN_SVA ); +if (!domain) +return ERR_PTR (-ENOMEM ); + } domain->type =IOMMU_DOMAIN_SVA ;mmgrab (mm);
Signature · 1 line
2.34.1
Lu Baolu
maintainerPATCH 12/12 iommu/vt-d: Retire struct intel_svm
In reply to Lu BaoluThe struct intel_svm was used for keeping attached devices info for sva domain. Since sva domain is a kind of iommu_domain, the struct dmar_domain should centralize all info of a sva domain, including the info of attached devices. Therefore, retire struct intel_svm to clean up the code.
Besides, allocate sva domain in domain_alloc_sva() callback which allows the memory management notifier lifetime to follow the lifetime of the iommu_domain.
--- drivers/iommu/intel/iommu.h | 26 ++++------ drivers/iommu/intel/iommu.c | 9 +--- drivers/iommu/intel/svm.c | 94 +++++++++---------------------------- 3 files changed, 32 insertions(+), 97 deletions(-)
drivers/iommu/intel/iommu.h +10 -16
@@ -649,8 +649,12 @@ struct dmar_domain {/* link to parent domain siblings */ struct list_head s2_link; }; + +/* SVA domain */ +struct { +struct mmu_notifier notifier; + }; }; -struct intel_svm *svm;struct iommu_domain domain;/* generic domain data structure for iommu core */
@@ -1131,26 +1135,16 @@ int intel_svm_enable_prq(struct intel_iommu *iommu);int intel_svm_finish_prq (struct intel_iommu *iommu);void intel_svm_page_response (struct device *dev,struct iopf_fault *evt,struct iommu_page_response *msg); -struct iommu_domain *intel_svm_domain_alloc (void ); -void intel_svm_remove_dev_pasid (struct iommu_domain *domain); +struct iommu_domain *intel_svm_domain_alloc (struct device *dev, +struct mm_struct *mm);void intel_drain_pasid_prq (struct device *dev,u32 pasid); - -struct intel_svm { -struct mmu_notifier notifier; -struct mm_struct *mm; -u32 pasid; -struct dmar_domain *domain; -};#else static inline void intel_svm_check (struct intel_iommu *iommu) {}static inline void intel_drain_pasid_prq (struct device *dev,u32 pasid) {} -static inline struct iommu_domain *intel_svm_domain_alloc (void ) -{ -return NULL ; -} - -static inline void intel_svm_remove_dev_pasid (struct iommu_domain *domain) +static inline struct iommu_domain *intel_svm_domain_alloc (struct device *dev, +struct mm_struct *mm) { +return ERR_PTR (-ENODEV ); }#endif
drivers/iommu/intel/iommu.c +1 -8
@@ -3687,8 +3687,6 @@ static struct iommu_domain *intel_iommu_domain_alloc(unsigned type)return domain;case IOMMU_DOMAIN_IDENTITY :return &si_domain->domain; -case IOMMU_DOMAIN_SVA : -return intel_svm_domain_alloc ();default :return NULL ; }
@@ -4395,14 +4393,8 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)WARN_ON_ONCE (!dev_pasid);spin_unlock_irqrestore (&dmar_domain->lock, flags); -/* -* The SVA implementation needs to handle its own stuffs like the mm -* notification. Before consolidating that code into iommu core, let -* the intel sva code handle it. -*/ if (domain->type ==IOMMU_DOMAIN_SVA ) {cache_tag_unassign_domain (dmar_domain,FLPT_DEFAULT_DID , dev, pasid); -intel_svm_remove_dev_pasid (domain); }else { did =domain_id_iommu (dmar_domain, iommu);cache_tag_unassign_domain (dmar_domain, did, dev, pasid);
@@ -4631,6 +4623,7 @@ const struct iommu_ops intel_iommu_ops = { .hw_info = intel_iommu_hw_info, .domain_alloc = intel_iommu_domain_alloc, .domain_alloc_user = intel_iommu_domain_alloc_user, + .domain_alloc_sva = intel_svm_domain_alloc, .probe_device = intel_iommu_probe_device, .probe_finalize = intel_iommu_probe_finalize, .release_device = intel_iommu_release_device,
drivers/iommu/intel/svm.c +21 -73
156 lines not on this page · raw message
Signature · 1 line
2.34.1
Quoted 24 lines
From: Lu Baolu <[email protected]>
Sent: Monday, March 25, 2024 10:17 AMThe IOMMU hardware cache needs to be invalidated whenever the mappings
in the domain are changed. Currently, domain cache invalidation is scattered across different places, causing several issues:- IOMMU IOTLB Invalidation: This is done by iterating through the domain IDs of each domain using the following code:
xa_for_each(&dmar_domain->iommu_array, i, info) iommu_flush_iotlb_psi(info->iommu, dmar_domain, start_pfn, nrpages, list_empty(&gather->freelist), 0);This code could theoretically cause a use-after-free problem because there's no lock to protect the "info" pointer within the loop.
- Inconsistent Invalidation Methods: Different domain types implement their own cache invalidation methods, making the code difficult to maintain. For example, the DMA domain, SVA domain, and nested domain have similar cache invalidation code scattered across different files.
Quoted 5 lines
- SVA Domain Inconsistency: The SVA domain implementation uses a completely different data structure to track attached devices compared to other domains. This creates unnecessary differences and, even worse, leads to duplicate IOTLB invalidation when an SVA domain is attached to devices belonging to different IOMMU domains.
can you elaborate how duplicated invalidations are caused?
Quoted 6 lines
- Nested Domain Dependency: The special overlap between a nested domain and its parent domain requires a dedicated
parent_domain_flush()helper function to be called everywhere the parent domain's mapping changes.
Quoted 7 lines
- Limited Debugging Support: There are currently no debugging aids available for domain cache invalidation.
By consolidating domain cache invalidation into a common location, we can address the issues mentioned above and improve the code's maintainability and debuggability.
overall this is a nice work!
Baolu Lu
maintainerQuoted 24 lines · Kevin
Quoted 24 lines
From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AMThe IOMMU hardware cache needs to be invalidated whenever the mappings
in the domain are changed. Currently, domain cache invalidation is scattered across different places, causing several issues:- IOMMU IOTLB Invalidation: This is done by iterating through the domain IDs of each domain using the following code:
xa_for_each(&dmar_domain->iommu_array, i, info) iommu_flush_iotlb_psi(info->iommu, dmar_domain, start_pfn, nrpages, list_empty(&gather->freelist), 0);This code could theoretically cause a use-after-free problem because there's no lock to protect the "info" pointer within the loop.
- Inconsistent Invalidation Methods: Different domain types implement their own cache invalidation methods, making the code difficult to maintain. For example, the DMA domain, SVA domain, and nested domain have similar cache invalidation code scattered across different files.
Quoted 6 lines
Quoted 5 lines
- SVA Domain Inconsistency: The SVA domain implementation uses a completely different data structure to track attached devices compared to other domains. This creates unnecessary differences and, even worse, leads to duplicate IOTLB invalidation when an SVA domain is attached to devices belonging to different IOMMU domains.
can you elaborate how duplicated invalidations are caused?
Yes, sure.
Current Intel SVA implementation keeps the bond between mm and a PASID of a device in a list of intel_svm_dev. In the mm notifier callback, it iterates all intel_svam_dev in the list and invalidates the IOTLB and device TLB sequentially.
If multiple devices belong to a single IOMMU, the IOTLB will be flushed multiple times. However, since these devices share the same domain ID and PASID, a single IOTLB cache invalidation is sufficient. The additional flushes are redundant and negatively impact performance.
Best regards,
baolu
Quoted 43 lines
From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 3:28 PMQuoted 16 lines · Kevin
Quoted 16 lines
From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AMThe IOMMU hardware cache needs to be invalidated whenever the mappings
in the domain are changed. Currently, domain cache invalidation is scattered across different places, causing several issues:- IOMMU IOTLB Invalidation: This is done by iterating through the domain IDs of each domain using the following code:
xa_for_each(&dmar_domain->iommu_array, i, info) iommu_flush_iotlb_psi(info->iommu, dmar_domain, start_pfn, nrpages, list_empty(&gather->freelist), 0);Quoted 6 lines
Quoted 6 lines
This code could theoretically cause a use-after-free problem because there's no lock to protect the "info" pointer within the loop.
- Inconsistent Invalidation Methods: Different domain types implement their own cache invalidation methods, making the code difficult to maintain. For example, the DMA domain, SVA domain, and nested
domain
Quoted 8 lines
Quoted 7 lines
have similar cache invalidation code scattered across different files.
- SVA Domain Inconsistency: The SVA domain implementation uses a completely different data structure to track attached devices compared to other domains. This creates unnecessary differences and, even worse, leads to duplicate IOTLB invalidation when an SVA domain is attached to devices belonging to different IOMMU domains.
can you elaborate how duplicated invalidations are caused?
Yes, sure.
Current Intel SVA implementation keeps the bond between mm and a PASID of a device in a list of
intel_svm_dev. In the mm notifier callback, it iterates allintel_svam_devin the list and invalidates the IOTLB and device TLB sequentially.
Quoted 5 lines
If multiple devices belong to a single IOMMU, the IOTLB will be flushed multiple times. However, since these devices share the same domain ID and PASID, a single IOTLB cache invalidation is sufficient. The additional flushes are redundant and negatively impact performance.
yes it's redundant. But what does "devices belonging to different IOMMU domains" in the original context try to convey? From above explanation it sounds irrelevant...
Baolu Lu
maintainerQuoted 49 lines · Kevin
Quoted 43 lines
From: Baolu Lu <[email protected]>
Sent: Sunday, April 7, 2024 3:28 PMQuoted 16 lines · Kevin
Quoted 16 lines
From: Lu Baolu<[email protected]>
Sent: Monday, March 25, 2024 10:17 AMThe IOMMU hardware cache needs to be invalidated whenever the mappings
in the domain are changed. Currently, domain cache invalidation is scattered across different places, causing several issues:- IOMMU IOTLB Invalidation: This is done by iterating through the domain
IDs of each domain using the following code: xa_for_each(&dmar_domain->iommu_array, i, info) iommu_flush_iotlb_psi(info->iommu, dmar_domain, start_pfn, nrpages, list_empty(&gather->freelist), 0);Quoted 6 lines
Quoted 6 lines
This code could theoretically cause a use-after-free problem because there's no lock to protect the "info" pointer within the loop.
- Inconsistent Invalidation Methods: Different domain types implement their own cache invalidation methods, making the code difficult to maintain. For example, the DMA domain, SVA domain, and nested
domain
Quoted 8 lines
Quoted 7 lines
have similar cache invalidation code scattered across different files.
- SVA Domain Inconsistency: The SVA domain implementation uses a completely different data structure to track attached devices compared to other domains. This creates unnecessary differences and, even worse, leads to duplicate IOTLB invalidation when an SVA domain is attached to devices belonging to different IOMMU domains.
can you elaborate how duplicated invalidations are caused?
Yes, sure.
Current Intel SVA implementation keeps the bond between mm and a PASID of a device in a list of
intel_svm_dev. In the mm notifier callback, it iterates allintel_svam_devin the list and invalidates the IOTLB and device TLB sequentially.Quoted 5 lines
If multiple devices belong to a single IOMMU, the IOTLB will be flushed multiple times. However, since these devices share the same domain ID and PASID, a single IOTLB cache invalidation is sufficient. The additional flushes are redundant and negatively impact performance.
Quoted 3 lines
yes it's redundant. But what does "devices belonging to different IOMMU domains" in the original context try to convey? From above explanation it sounds irrelevant...
My typo. :-) Sorry for the confusion.
I should say,
"... attached to devices belonging to a same IOMMU ..."
Best regards,
baolu