iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF modes
Subsystems (3)
Lu Baolu
maintainerPATCH 1/1 iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF modes
The iotlb_sync_map iommu ops allows drivers to perform necessary cache flushes when new mappings are established. For the Intel iommu driver, this callback specifically serves two purposes:
- To flush caches when a second-stage page table is attached to a device whose iommu is operating in caching mode (CAP_REG.CM==1).
- To explicitly flush internal write buffers to ensure updates to memory- resident remapping structures are visible to hardware (CAP_REG.RWBF==1).
However, in scenarios where neither caching mode nor the RWBF flag is active, the cache_tag_flush_range_np() helper, which is called in the iotlb_sync_map path, effectively becomes a no-op.
Despite being a no-op, cache_tag_flush_range_np() involves iterating through all cache tags of the iommu's attached to the domain, protected by a spinlock. This unnecessary execution path introduces overhead, leading to a measurable I/O performance regression. On systems with NVMes under the same bridge, performance was observed to drop from approximately ~6150 MiB/s down to ~4985 MiB/s.
Introduce a flag in the dmar_domain structure. This flag will only be set when iotlb_sync_map is required (i.e., when CM or RWBF is set). The cache_tag_flush_range_np() is called only for domains where this flag is set.
--- drivers/iommu/intel/iommu.c | 19 ++++++++++++++++++- drivers/iommu/intel/iommu.h | 3 +++ 2 files changed, 21 insertions(+), 1 deletion(-)
drivers/iommu/intel/iommu.c +18 -1
@@ -1796,6 +1796,18 @@ static int domain_setup_first_level(struct intel_iommu *iommu, (pgd_t *)pgd, flags, old); } +static bool domain_need_iotlb_sync_map (struct dmar_domain *domain, +struct intel_iommu *iommu) +{ +if (cap_caching_mode (iommu->cap) && !domain->use_first_level) +return true; + +if (rwbf_quirk ||cap_rwbf (iommu->cap)) +return true; + +return false; +} +static int dmar_domain_attach_device (struct dmar_domain *domain,struct device *dev) {
@@ -1833,6 +1845,8 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,if (ret) goto out_block_translation; + domain->iotlb_sync_map |=domain_need_iotlb_sync_map (domain, iommu); +return 0 ; out_block_translation:
@@ -3945,7 +3959,10 @@ static bool risky_device(struct pci_dev *pdev)static int intel_iommu_iotlb_sync_map (struct iommu_domain *domain,unsigned long iova,size_t size) { -cache_tag_flush_range_np (to_dmar_domain (domain), iova, iova + size -1 ); +struct dmar_domain *dmar_domain =to_dmar_domain (domain); + +if (dmar_domain->iotlb_sync_map) +cache_tag_flush_range_np (dmar_domain, iova, iova + size -1 );return 0 ; }
drivers/iommu/intel/iommu.h +3 -0
@@ -614,6 +614,9 @@ struct dmar_domain { u8 has_mappings:1; /* Has mappings configured through* iommu_map() interface. */ + u8 iotlb_sync_map:1; /* Need to flush IOTLB cache or write +* buffer when creating mappings. +*/ spinlock_t lock;/* Protect device tracking lists */ struct list_head devices;/* all devices' list */
Signature · 1 line
2.43.0
Quoted 16 lines
From: Lu Baolu <[email protected]>
Sent: Thursday, July 3, 2025 11:16 AMThe
iotlb_sync_mapiommu ops allows drivers to perform necessary cache flushes when new mappings are established. For the Intel iommu driver, this callback specifically serves two purposes:- To flush caches when a second-stage page table is attached to a device whose iommu is operating in caching mode (CAP_REG.CM==1).
- To explicitly flush internal write buffers to ensure updates to memory- resident remapping structures are visible to hardware (CAP_REG.RWBF==1).However, in scenarios where neither caching mode nor the RWBF flag is active, the
cache_tag_flush_range_np()helper, which is called in theiotlb_sync_mappath, effectively becomes a no-op.
Quoted 8 lines
Despite being a no-op,
cache_tag_flush_range_np()involves iterating through all cache tags of the iommu's attached to the domain, protected by a spinlock. This unnecessary execution path introduces overhead, leading to a measurable I/O performance regression. On systems with NVMes
under the same bridge, performance was observed to drop from approximately
~6150 MiB/s down to ~4985 MiB/s.
so for the same bridge case two NVMe disks likely are in the same iommu group sharing a domain. Then there is contention on the spinlock from two parallel threads on two disks. when disks come from different bridges they are attached to different domains hence no contention.
is it a correct description for the difference between same vs. different bridge?
drivers/iommu/intel/iommu.c static int dmar_domain_attach_device(struct · line 1852
Quoted 10 lines
drivers/iommu/intel/iommu.c +2
-0@@ -1833,6 +1845,8 @@ static int dmar_domain_attach_device(struct dmar_domain *domain, if (ret) goto out_block_translation; + domain->iotlb_sync_map |=domain_need_iotlb_sync_map (domain, iommu); +return 0 ;
also need to update the flag upon detach.
with it:
Baolu Lu
maintainerQuoted 24 lines · Kevin
Quoted 16 lines
From: Lu Baolu<[email protected]>
Sent: Thursday, July 3, 2025 11:16 AMThe
iotlb_sync_mapiommu ops allows drivers to perform necessary cache flushes when new mappings are established. For the Intel iommu driver, this callback specifically serves two purposes:- To flush caches when a second-stage page table is attached to a device whose iommu is operating in caching mode (CAP_REG.CM==1).
- To explicitly flush internal write buffers to ensure updates to memory- resident remapping structures are visible to hardware (CAP_REG.RWBF==1).However, in scenarios where neither caching mode nor the RWBF flag is active, the
cache_tag_flush_range_np()helper, which is called in theiotlb_sync_mappath, effectively becomes a no-op.Quoted 8 lines
Despite being a no-op,
cache_tag_flush_range_np()involves iterating through all cache tags of the iommu's attached to the domain, protected by a spinlock. This unnecessary execution path introduces overhead, leading to a measurable I/O performance regression. On systems with NVMes
under the same bridge, performance was observed to drop from approximately
~6150 MiB/s down to ~4985 MiB/s.
Quoted 8 lines
so for the same bridge case two NVMe disks likely are in the same iommu group sharing a domain. Then there is contention on the spinlock from two parallel threads on two disks. when disks come from different bridges they are attached to different domains hence no contention.
is it a correct description for the difference between same vs. different bridge?
Yes. I have the same understanding.
Thanks,
baolu
Quoted 31 lines · Lu Baolu
The
iotlb_sync_mapiommu ops allows drivers to perform necessary cache flushes when new mappings are established. For the Intel iommu driver, this callback specifically serves two purposes:- To flush caches when a second-stage page table is attached to a device whose iommu is operating in caching mode (CAP_REG.CM==1).
- To explicitly flush internal write buffers to ensure updates to memory- resident remapping structures are visible to hardware (CAP_REG.RWBF==1).However, in scenarios where neither caching mode nor the RWBF flag is active, the
cache_tag_flush_range_np()helper, which is called in theiotlb_sync_mappath, effectively becomes a no-op.Despite being a no-op,
cache_tag_flush_range_np()involves iterating through all cache tags of the iommu's attached to the domain, protected by a spinlock. This unnecessary execution path introduces overhead, leading to a measurable I/O performance regression. On systems with NVMes under the same bridge, performance was observed to drop from approximately ~6150 MiB/s down to ~4985 MiB/s.Introduce a flag in the
dmar_domainstructure. This flag will only be set wheniotlb_sync_mapis required (i.e., when CM or RWBF is set). Thecache_tag_flush_range_np()is called only for domains where this flag is set.Reported-by Ioanna Alifieraki<[email protected]>Fixes 129dab6e1286 ("iommu/vt-d: Use cache_tag_flush_range_np() in iotlb_sync_map")Signed-off-by Lu Baolu<[email protected]>
Quoted 4 lines
--- drivers/iommu/intel/iommu.c | 19 ++++++++++++++++++- drivers/iommu/intel/iommu.h | 3 +++ 2 files changed, 21 insertions(+), 1 deletion(-)
Queued for linux-next.
--
baolu