Intel IOMMU updates for v7.1
Subsystems (4)
Lu Baolu
maintainerHi Joerg,
The following changes are queued for v7.1-rc1. This update introduces a new feature and several refactorings, including:
- Support for dirty tracking on domains attached to PASID.
- Removal of unnecessary read*()/write*() wrappers.
- Improvements to the invalidation paths.
These patches are based on v7.0-rc6. Please consider them for the iommu/vt-d branch.
Best regards,
baolu
Bjorn Helgaas (2):
iommu/vt-d: Remove dmar_readl() and dmar_readq() iommu/vt-d: Remove dmar_writel() and dmar_writeq()
Jason Gunthorpe (4):
iommu/vt-d: Split piotlb invalidation into range and all iommu/vt-d: Pass size_order to qi_desc_piotlb() not npages iommu/vt-d: Remove the remaining pages along the invalidation path iommu/vt-d: Simplify calculate_psi_aligned_address()
Zhenzhong Duan (4):
iommu/vt-d: Block PASID attachment to nested domain with dirty
tracking
iommu/vt-d: Rename device_set_dirty_tracking() and pass dmar_domain
pointer
iommu/vt-d: Support dirty tracking on PASID iommufd/selftest: Test dirty tracking on PASID
drivers/iommu/intel/iommu.h | 47 +++++------ drivers/iommu/intel/trace.h | 18 ++--- drivers/iommu/intel/cache.c | 100 ++++++++++-------------- drivers/iommu/intel/debugfs.c | 18 ++--- drivers/iommu/intel/dmar.c | 43 ++++------ drivers/iommu/intel/iommu.c | 55 +++++++------ drivers/iommu/intel/irq_remapping.c | 6 +- drivers/iommu/intel/pasid.c | 6 +- drivers/iommu/intel/perfmon.c | 50 ++++++------ drivers/iommu/intel/prq.c | 28 +++---- tools/testing/selftests/iommu/iommufd.c | 27 +++++++ 11 files changed, 193 insertions(+), 205 deletions(-)
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 1/10 iommu/vt-d: Block PASID attachment to nested domain with dirty tracking
From: Zhenzhong Duan <[email protected]>
Kernel lacks dirty tracking support on nested domain attached to PASID, fails the attachment early if nesting parent domain is dirty tracking configured, otherwise dirty pages would be lost.
--- drivers/iommu/intel/nested.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/iommu/intel/nested.c +5 -1
@@ -148,6 +148,7 @@ static int intel_nested_set_dev_pasid(struct iommu_domain *domain, {struct device_domain_info *info =dev_iommu_priv_get (dev);struct dmar_domain *dmar_domain =to_dmar_domain (domain); +struct iommu_domain *s2_domain = &dmar_domain->s2_domain->domain;struct intel_iommu *iommu = info->iommu;struct dev_pasid_info *dev_pasid;int ret;
@@ -155,10 +156,13 @@ static int intel_nested_set_dev_pasid(struct iommu_domain *domain,if (!pasid_supported (iommu) ||dev_is_real_dma_subdevice (dev))return -EOPNOTSUPP ; +if (s2_domain->dirty_ops) +return -EINVAL ; +if (context_copied (iommu, info->bus, info->devfn))return -EBUSY ; - ret =paging_domain_compatible (&dmar_domain->s2_domain->domain, dev); + ret =paging_domain_compatible (s2_domain, dev);if (ret)return ret;
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 2/10 iommu/vt-d: Rename device_set_dirty_tracking() and pass dmar_domain pointer
In reply to Lu BaoluFrom: Zhenzhong Duan <[email protected]>
device_set_dirty_tracking() sets dirty tracking on all devices attached to a domain, also on all PASIDs attached to same domain in subsequent patch.
So rename it as domain_set_dirty_tracking() and pass dmar_domain pointer to better align to what it does.
No functional changes intended.
--- drivers/iommu/intel/iommu.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
drivers/iommu/intel/iommu.c +9 -12
@@ -3684,16 +3684,15 @@ static void *intel_iommu_hw_info(struct device *dev, u32 *length,return vtd; } -/* -* Set dirty tracking for the device list of a domain. The caller must -* hold the domain->lock when calling it. -*/ -static int device_set_dirty_tracking (struct list_head *devices,bool enable) +/* Set dirty tracking for the devices that the domain has been attached. */ +static int domain_set_dirty_tracking (struct dmar_domain *domain,bool enable) {struct device_domain_info *info;int ret =0 ; -list_for_each_entry (info, devices, link) { +lockdep_assert_held (&domain->lock); + +list_for_each_entry (info, &domain->devices, link) { ret =intel_pasid_setup_dirty_tracking (info->iommu, info->dev,IOMMU_NO_PASID , enable);if (ret)
@@ -3713,7 +3712,7 @@ static int parent_domain_set_dirty_tracking(struct dmar_domain *domain,spin_lock (&domain->s1_lock);list_for_each_entry (s1_domain, &domain->s1_domains, s2_link) {spin_lock_irqsave (&s1_domain->lock, flags); - ret =device_set_dirty_tracking (&s1_domain->devices, enable); + ret =domain_set_dirty_tracking (s1_domain, enable);spin_unlock_irqrestore (&s1_domain->lock, flags);if (ret) goto err_unwind;
@@ -3724,8 +3723,7 @@ static int parent_domain_set_dirty_tracking(struct dmar_domain *domain, err_unwind:list_for_each_entry (s1_domain, &domain->s1_domains, s2_link) {spin_lock_irqsave (&s1_domain->lock, flags); -device_set_dirty_tracking (&s1_domain->devices, - domain->dirty_tracking); +domain_set_dirty_tracking (s1_domain, domain->dirty_tracking);spin_unlock_irqrestore (&s1_domain->lock, flags); }spin_unlock (&domain->s1_lock);
@@ -3742,7 +3740,7 @@ static int intel_iommu_set_dirty_tracking(struct iommu_domain *domain,if (dmar_domain->dirty_tracking == enable) goto out_unlock; - ret =device_set_dirty_tracking (&dmar_domain->devices, enable); + ret =domain_set_dirty_tracking (dmar_domain, enable);if (ret) goto err_unwind;
@@ -3759,8 +3757,7 @@ static int intel_iommu_set_dirty_tracking(struct iommu_domain *domain,return 0 ; err_unwind: -device_set_dirty_tracking (&dmar_domain->devices, - dmar_domain->dirty_tracking); +domain_set_dirty_tracking (dmar_domain, dmar_domain->dirty_tracking);spin_unlock (&dmar_domain->lock);return ret; }
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 3/10 iommu/vt-d: Support dirty tracking on PASID
In reply to Lu BaoluFrom: Zhenzhong Duan <[email protected]>
In order to support passthrough device with PASID capability in QEMU, e.g., DSA device, kernel needs to support attaching PASID to a domain.
But attaching is not allowed if the domain is a second stage domain or nested domain with dirty tracking.
The reason is kernel lacking support for dirty tracking on such domain attached to PASID. By adding dirty tracking on PASID, the check can be removed.
--- drivers/iommu/intel/iommu.c | 12 +++++++++--- drivers/iommu/intel/nested.c | 6 +----- 2 files changed, 10 insertions(+), 8 deletions(-)
drivers/iommu/intel/iommu.c +9 -3
@@ -3618,9 +3618,6 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,if (!pasid_supported (iommu) ||dev_is_real_dma_subdevice (dev))return -EOPNOTSUPP ; -if (domain->dirty_ops) -return -EINVAL ; -if (context_copied (iommu, info->bus, info->devfn))return -EBUSY ;
@@ -3688,6 +3685,7 @@ static void *intel_iommu_hw_info(struct device *dev, u32 *length,static int domain_set_dirty_tracking (struct dmar_domain *domain,bool enable) {struct device_domain_info *info; +struct dev_pasid_info *dev_pasid;int ret =0 ;lockdep_assert_held (&domain->lock);
@@ -3695,6 +3693,14 @@ static int domain_set_dirty_tracking(struct dmar_domain *domain, bool enable)list_for_each_entry (info, &domain->devices, link) { ret =intel_pasid_setup_dirty_tracking (info->iommu, info->dev,IOMMU_NO_PASID , enable); +if (ret) +return ret; + } + +list_for_each_entry (dev_pasid, &domain->dev_pasids, link_domain) { + info =dev_iommu_priv_get (dev_pasid->dev); + ret =intel_pasid_setup_dirty_tracking (info->iommu, info->dev, + dev_pasid->pasid, enable);if (ret)break ; }
drivers/iommu/intel/nested.c +1 -5
@@ -148,7 +148,6 @@ static int intel_nested_set_dev_pasid(struct iommu_domain *domain, {struct device_domain_info *info =dev_iommu_priv_get (dev);struct dmar_domain *dmar_domain =to_dmar_domain (domain); -struct iommu_domain *s2_domain = &dmar_domain->s2_domain->domain;struct intel_iommu *iommu = info->iommu;struct dev_pasid_info *dev_pasid;int ret;
@@ -156,13 +155,10 @@ static int intel_nested_set_dev_pasid(struct iommu_domain *domain,if (!pasid_supported (iommu) ||dev_is_real_dma_subdevice (dev))return -EOPNOTSUPP ; -if (s2_domain->dirty_ops) -return -EINVAL ; -if (context_copied (iommu, info->bus, info->devfn))return -EBUSY ; - ret =paging_domain_compatible (s2_domain, dev); + ret =paging_domain_compatible (&dmar_domain->s2_domain->domain, dev);if (ret)return ret;
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 4/10 iommufd/selftest: Test dirty tracking on PASID
In reply to Lu BaoluFrom: Zhenzhong Duan <[email protected]>
Add test case for dirty tracking on a domain attached to PASID, also confirm attachment to PASID fail if device doesn't support dirty tracking.
--- tools/testing/selftests/iommu/iommufd.c | 27 +++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
tools/testing/selftests/iommu/iommufd.c +27 -0
@@ -2275,6 +2275,33 @@ TEST_F(iommufd_dirty_tracking, set_dirty_tracking)test_ioctl_destroy (hwpt_id); } +TEST_F(iommufd_dirty_tracking, pasid_set_dirty_tracking) +{ +uint32_t stddev_id, ioas_id, hwpt_id, pasid =100 ; +uint32_t dev_flags =MOCK_FLAGS_DEVICE_PASID ; + +/* Regular case */ +test_cmd_hwpt_alloc (self->idev_id, self->ioas_id, +IOMMU_HWPT_ALLOC_PASID |IOMMU_HWPT_ALLOC_DIRTY_TRACKING , + &hwpt_id); +test_cmd_mock_domain_flags (hwpt_id, dev_flags, &stddev_id,NULL ,NULL ); +ASSERT_EQ (0 ,_test_cmd_pasid_attach (self->fd, stddev_id, pasid, hwpt_id)); +test_cmd_set_dirty_tracking (hwpt_id, true); +test_cmd_set_dirty_tracking (hwpt_id, false); +ASSERT_EQ (0 ,_test_cmd_pasid_detach (self->fd, stddev_id, pasid)); + +test_ioctl_destroy (stddev_id); + +/* IOMMU device does not support dirty tracking */ + dev_flags |=MOCK_FLAGS_DEVICE_NO_DIRTY ; +test_ioctl_ioas_alloc (&ioas_id); +test_cmd_mock_domain_flags (ioas_id, dev_flags, &stddev_id,NULL ,NULL ); +EXPECT_ERRNO (EINVAL ,_test_cmd_pasid_attach (self->fd, stddev_id, pasid, hwpt_id)); + +test_ioctl_destroy (stddev_id); +test_ioctl_destroy (hwpt_id); +} + TEST_F(iommufd_dirty_tracking, device_dirty_capability) {uint32_t caps =0 ;
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 5/10 iommu/vt-d: Remove dmar_readl() and dmar_readq()
In reply to Lu BaoluFrom: Bjorn Helgaas <[email protected]>
dmar_readl() and dmar_readq() do nothing other than expand to the generic readl() and readq(), and the dmar_read*() wrappers are used inconsistently.
Remove the dmar_read*() wrappers and use readl() and readq() directly.
--- drivers/iommu/intel/iommu.h | 2 -- drivers/iommu/intel/debugfs.c | 18 +++++++++--------- drivers/iommu/intel/dmar.c | 22 +++++++++++----------- drivers/iommu/intel/iommu.c | 10 +++++----- drivers/iommu/intel/irq_remapping.c | 2 +- drivers/iommu/intel/perfmon.c | 28 ++++++++++++++-------------- drivers/iommu/intel/prq.c | 12 ++++++------ 7 files changed, 46 insertions(+), 48 deletions(-)
drivers/iommu/intel/iommu.h +0 -2
@@ -148,9 +148,7 @@#define OFFSET_STRIDE (9) -#define dmar_readq (a) readq(a)#define dmar_writeq (a,v) writeq(v,a) -#define dmar_readl (a) readl(a)#define dmar_writel (a, v) writel(v, a)#define DMAR_VER_MAJOR (v) (((v) & 0xf0) >> 4)
drivers/iommu/intel/debugfs.c +9 -9
@@ -133,13 +133,13 @@ static int iommu_regset_show(struct seq_file *m, void *unused)*/ raw_spin_lock_irqsave (&iommu->register_lock, flag);for (i =0 ; i <ARRAY_SIZE (iommu_regs_32); i++) { - value =dmar_readl (iommu->reg + iommu_regs_32[i].offset); + value =readl (iommu->reg + iommu_regs_32[i].offset);seq_printf (m,"%-16s\t0x%02x\t\t0x%016llx\n" , iommu_regs_32[i].regs, iommu_regs_32[i].offset, value); }for (i =0 ; i <ARRAY_SIZE (iommu_regs_64); i++) { - value =dmar_readq (iommu->reg + iommu_regs_64[i].offset); + value =readq (iommu->reg + iommu_regs_64[i].offset);seq_printf (m,"%-16s\t0x%02x\t\t0x%016llx\n" , iommu_regs_64[i].regs, iommu_regs_64[i].offset, value);
@@ -247,7 +247,7 @@ static void ctx_tbl_walk(struct seq_file *m, struct intel_iommu *iommu, u16 bus) tbl_wlk.ctx_entry = context; m->private = &tbl_wlk; -if (dmar_readq (iommu->reg +DMAR_RTADDR_REG ) &DMA_RTADDR_SMT ) { +if (readq (iommu->reg +DMAR_RTADDR_REG ) &DMA_RTADDR_SMT ) { pasid_dir_ptr = context->lo &VTD_PAGE_MASK ; pasid_dir_size =get_pasid_dir_size (context);pasid_dir_walk (m, pasid_dir_ptr, pasid_dir_size);
@@ -285,7 +285,7 @@ static int dmar_translation_struct_show(struct seq_file *m, void *unused)rcu_read_lock ();for_each_active_iommu (iommu, drhd) { - sts =dmar_readl (iommu->reg +DMAR_GSTS_REG ); + sts =readl (iommu->reg +DMAR_GSTS_REG );if (!(sts &DMA_GSTS_TES )) {seq_printf (m,"DMA Remapping is not enabled on %s\n" , iommu->name);
@@ -364,13 +364,13 @@ static int domain_translation_struct_show(struct seq_file *m,if (seg != iommu->segment)continue ; - sts =dmar_readl (iommu->reg +DMAR_GSTS_REG ); + sts =readl (iommu->reg +DMAR_GSTS_REG );if (!(sts &DMA_GSTS_TES )) {seq_printf (m,"DMA Remapping is not enabled on %s\n" , iommu->name);continue ; } -if (dmar_readq (iommu->reg +DMAR_RTADDR_REG ) &DMA_RTADDR_SMT ) +if (readq (iommu->reg +DMAR_RTADDR_REG ) &DMA_RTADDR_SMT ) scalable = true;else scalable = false;
@@ -538,8 +538,8 @@ static int invalidation_queue_show(struct seq_file *m, void *unused)raw_spin_lock_irqsave (&qi->q_lock, flags);seq_printf (m," Base: 0x%llx\tHead: %lld\tTail: %lld\n" , (u64 )virt_to_phys (qi->desc), -dmar_readq (iommu->reg +DMAR_IQH_REG ) >> shift, -dmar_readq (iommu->reg +DMAR_IQT_REG ) >> shift); +readq (iommu->reg +DMAR_IQH_REG ) >> shift, +readq (iommu->reg +DMAR_IQT_REG ) >> shift);invalidation_queue_entry_show (m, iommu);raw_spin_unlock_irqrestore (&qi->q_lock, flags);seq_putc (m,'\n' );
@@ -620,7 +620,7 @@ static int ir_translation_struct_show(struct seq_file *m, void *unused)seq_printf (m,"Remapped Interrupt supported on IOMMU: %s\n" , iommu->name); - sts =dmar_readl (iommu->reg +DMAR_GSTS_REG ); + sts =readl (iommu->reg +DMAR_GSTS_REG );if (iommu->ir_table && (sts &DMA_GSTS_IRES )) { irta =virt_to_phys (iommu->ir_table->base);seq_printf (m," IR table address:%llx\n" , irta);
drivers/iommu/intel/dmar.c +11 -11
@@ -899,8 +899,8 @@ dmar_validate_one_drhd(struct acpi_dmar_header *entry, void *arg)return -EINVAL ; } - cap =dmar_readq (addr +DMAR_CAP_REG ); - ecap =dmar_readq (addr +DMAR_ECAP_REG ); + cap =readq (addr +DMAR_CAP_REG ); + ecap =readq (addr +DMAR_ECAP_REG );if (arg)iounmap (addr);
@@ -982,8 +982,8 @@ static int map_iommu(struct intel_iommu *iommu, struct dmar_drhd_unit *drhd) goto release; } - iommu->cap =dmar_readq (iommu->reg +DMAR_CAP_REG ); - iommu->ecap =dmar_readq (iommu->reg +DMAR_ECAP_REG ); + iommu->cap =readq (iommu->reg +DMAR_CAP_REG ); + iommu->ecap =readq (iommu->reg +DMAR_ECAP_REG );if (iommu->cap == (uint64_t )-1 && iommu->ecap == (uint64_t )-1 ) { err = -EINVAL ;
@@ -1017,8 +1017,8 @@ static int map_iommu(struct intel_iommu *iommu, struct dmar_drhd_unit *drhd)int i;for (i =0 ; i <DMA_MAX_NUM_ECMDCAP ; i++) { - iommu->ecmdcap[i] =dmar_readq (iommu->reg +DMAR_ECCAP_REG + - i *DMA_ECMD_REG_STEP ); + iommu->ecmdcap[i] =readq (iommu->reg +DMAR_ECCAP_REG + + i *DMA_ECMD_REG_STEP ); } }
@@ -1239,8 +1239,8 @@ static const char *qi_type_string(u8 type)static void qi_dump_fault (struct intel_iommu *iommu,u32 fault) { -unsigned int head =dmar_readl (iommu->reg +DMAR_IQH_REG ); -u64 iqe_err =dmar_readq (iommu->reg +DMAR_IQER_REG ); +unsigned int head =readl (iommu->reg +DMAR_IQH_REG ); +u64 iqe_err =readq (iommu->reg +DMAR_IQER_REG );struct qi_desc *desc = iommu->qi->desc + head;if (fault &DMA_FSTS_IQE )
@@ -1321,7 +1321,7 @@ static int qi_check_fault(struct intel_iommu *iommu, int index, int wait_index)* SID field is valid only when the ITE field is Set in FSTS_REG * see Intel VT-d spec r4.1, section 11.4.9.9 */ - iqe_err =dmar_readq (iommu->reg +DMAR_IQER_REG ); + iqe_err =readq (iommu->reg +DMAR_IQER_REG ); ite_sid =DMAR_IQER_REG_ITESID (iqe_err);writel (DMA_FSTS_ITE , iommu->reg +DMAR_FSTS_REG );
@@ -1980,8 +1980,8 @@ irqreturn_t dmar_fault(int irq, void *dev_id) source_id =dma_frcd_source_id (data); pasid_present =dma_frcd_pasid_present (data); - guest_addr =dmar_readq (iommu->reg + reg + - fault_index *PRIMARY_FAULT_REG_LEN ); + guest_addr =readq (iommu->reg + reg + + fault_index *PRIMARY_FAULT_REG_LEN ); guest_addr =dma_frcd_page_addr (guest_addr); }
drivers/iommu/intel/iommu.c +5 -5
@@ -769,7 +769,7 @@ static void __iommu_flush_context(struct intel_iommu *iommu,/* Make sure hardware complete it */ IOMMU_WAIT_OP (iommu,DMAR_CCMD_REG , - dmar_readq, (!(val &DMA_CCMD_ICC )), val); + readq, (!(val &DMA_CCMD_ICC )), val);raw_spin_unlock_irqrestore (&iommu->register_lock, flag); }
@@ -811,7 +811,7 @@ void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,/* Make sure hardware complete it */ IOMMU_WAIT_OP (iommu, tlb_offset +8 , - dmar_readq, (!(val &DMA_TLB_IVT )), val); + readq, (!(val &DMA_TLB_IVT )), val);raw_spin_unlock_irqrestore (&iommu->register_lock, flag);
@@ -1533,7 +1533,7 @@ static int copy_translation_tables(struct intel_iommu *iommu)int bus, ret;bool new_ext, ext; - rtaddr_reg =dmar_readq (iommu->reg +DMAR_RTADDR_REG ); + rtaddr_reg =readq (iommu->reg +DMAR_RTADDR_REG ); ext = !!(rtaddr_reg &DMA_RTADDR_SMT ); new_ext = !!sm_supported (iommu);
@@ -4188,7 +4188,7 @@ int ecmd_submit_sync(struct intel_iommu *iommu, u8 ecmd, u64 oa, u64 ob)raw_spin_lock_irqsave (&iommu->register_lock, flags); - res =dmar_readq (iommu->reg +DMAR_ECRSP_REG ); + res =readq (iommu->reg +DMAR_ECRSP_REG );if (res &DMA_ECMD_ECRSP_IP ) { ret = -EBUSY ; goto err;
@@ -4204,7 +4204,7 @@ int ecmd_submit_sync(struct intel_iommu *iommu, u8 ecmd, u64 oa, u64 ob)dmar_writeq (iommu->reg +DMAR_ECEO_REG , ob);dmar_writeq (iommu->reg +DMAR_ECMD_REG , ecmd | (oa <<DMA_ECMD_OA_SHIFT )); -IOMMU_WAIT_OP (iommu,DMAR_ECRSP_REG , dmar_readq, +IOMMU_WAIT_OP (iommu,DMAR_ECRSP_REG , readq, !(res &DMA_ECMD_ECRSP_IP ), res);if (res &DMA_ECMD_ECRSP_IP ) {
drivers/iommu/intel/irq_remapping.c +1 -1
@@ -422,7 +422,7 @@ static int iommu_load_old_irte(struct intel_iommu *iommu)u64 irta;/* Check whether the old ir-table has the same size as ours */ - irta =dmar_readq (iommu->reg +DMAR_IRTA_REG ); + irta =readq (iommu->reg +DMAR_IRTA_REG );if ((irta &INTR_REMAP_TABLE_REG_SIZE_MASK ) !=INTR_REMAP_TABLE_REG_SIZE )return -EINVAL ;
drivers/iommu/intel/perfmon.c +14 -14
@@ -307,7 +307,7 @@ static void iommu_pmu_event_update(struct perf_event *event) again: prev_count =local64_read (&hwc->prev_count); - new_count =dmar_readq (iommu_event_base (iommu_pmu, hwc->idx)); + new_count =readq (iommu_event_base (iommu_pmu, hwc->idx));if (local64_xchg (&hwc->prev_count, new_count) != prev_count) goto again;
@@ -340,7 +340,7 @@ static void iommu_pmu_start(struct perf_event *event, int flags) hwc->state =0 ;/* Always reprogram the period */ - count =dmar_readq (iommu_event_base (iommu_pmu, hwc->idx)); + count =readq (iommu_event_base (iommu_pmu, hwc->idx));local64_set ((&hwc->prev_count), count);/*
@@ -496,7 +496,7 @@ static void iommu_pmu_counter_overflow(struct iommu_pmu *iommu_pmu)* Two counters may be overflowed very close. Always check * whether there are more to handle. */ -while ((status =dmar_readq (iommu_pmu->overflow))) { +while ((status =readq (iommu_pmu->overflow))) {for_each_set_bit (i, (unsigned long *)&status, iommu_pmu->num_cntr) {/* * Find the assigned event of the counter.
@@ -518,7 +518,7 @@ static irqreturn_t iommu_pmu_irq_handler(int irq, void *dev_id) {struct intel_iommu *iommu = dev_id; -if (!dmar_readl (iommu->reg +DMAR_PERFINTRSTS_REG )) +if (!readl (iommu->reg +DMAR_PERFINTRSTS_REG ))return IRQ_NONE ;iommu_pmu_counter_overflow (iommu->pmu);
@@ -555,7 +555,7 @@ static int __iommu_pmu_register(struct intel_iommu *iommu)static inline void __iomem *get_perf_reg_address (struct intel_iommu *iommu,u32 offset) { -u32 off =dmar_readl (iommu->reg + offset); +u32 off =readl (iommu->reg + offset);return iommu->reg + off; }
@@ -574,7 +574,7 @@ int alloc_iommu_pmu(struct intel_iommu *iommu)if (!cap_ecmds (iommu->cap))return -ENODEV ; - perfcap =dmar_readq (iommu->reg +DMAR_PERFCAP_REG ); + perfcap =readq (iommu->reg +DMAR_PERFCAP_REG );/* The performance monitoring is not supported. */ if (!perfcap)return -ENODEV ;
@@ -617,8 +617,8 @@ int alloc_iommu_pmu(struct intel_iommu *iommu)for (i =0 ; i < iommu_pmu->num_eg; i++) {u64 pcap; - pcap =dmar_readq (iommu->reg +DMAR_PERFEVNTCAP_REG + - i *IOMMU_PMU_CAP_REGS_STEP ); + pcap =readq (iommu->reg +DMAR_PERFEVNTCAP_REG + + i *IOMMU_PMU_CAP_REGS_STEP ); iommu_pmu->evcap[i] =pecap_es (pcap); }
@@ -651,9 +651,9 @@ int alloc_iommu_pmu(struct intel_iommu *iommu)* Width. */ for (i =0 ; i < iommu_pmu->num_cntr; i++) { - cap =dmar_readl (iommu_pmu->cfg_reg + - i *IOMMU_PMU_CFG_OFFSET + -IOMMU_PMU_CFG_CNTRCAP_OFFSET ); + cap =readl (iommu_pmu->cfg_reg + + i *IOMMU_PMU_CFG_OFFSET + +IOMMU_PMU_CFG_CNTRCAP_OFFSET );if (!iommu_cntrcap_pcc (cap))continue ;
@@ -675,9 +675,9 @@ int alloc_iommu_pmu(struct intel_iommu *iommu)/* Override with per-counter event capabilities */ for (j =0 ; j <iommu_cntrcap_egcnt (cap); j++) { - cap =dmar_readl (iommu_pmu->cfg_reg + i *IOMMU_PMU_CFG_OFFSET + -IOMMU_PMU_CFG_CNTREVCAP_OFFSET + - (j *IOMMU_PMU_OFF_REGS_STEP )); + cap =readl (iommu_pmu->cfg_reg + i *IOMMU_PMU_CFG_OFFSET + +IOMMU_PMU_CFG_CNTREVCAP_OFFSET + + (j *IOMMU_PMU_OFF_REGS_STEP )); iommu_pmu->cntr_evcap[i][iommu_event_group (cap)] =iommu_event_select (cap);/* * Some events may only be supported by a specific counter.
drivers/iommu/intel/prq.c +6 -6
@@ -81,8 +81,8 @@ void intel_iommu_drain_pasid_prq(struct device *dev, u32 pasid)*/ prq_retry:reinit_completion (&iommu->prq_complete); - tail =dmar_readq (iommu->reg +DMAR_PQT_REG ) &PRQ_RING_MASK ; - head =dmar_readq (iommu->reg +DMAR_PQH_REG ) &PRQ_RING_MASK ; + tail =readq (iommu->reg +DMAR_PQT_REG ) &PRQ_RING_MASK ; + head =readq (iommu->reg +DMAR_PQH_REG ) &PRQ_RING_MASK ;while (head != tail) {struct page_req_dsc *req;
@@ -208,8 +208,8 @@ static irqreturn_t prq_event_thread(int irq, void *d)*/ writel (DMA_PRS_PPR , iommu->reg +DMAR_PRS_REG ); - tail =dmar_readq (iommu->reg +DMAR_PQT_REG ) &PRQ_RING_MASK ; - head =dmar_readq (iommu->reg +DMAR_PQH_REG ) &PRQ_RING_MASK ; + tail =readq (iommu->reg +DMAR_PQT_REG ) &PRQ_RING_MASK ; + head =readq (iommu->reg +DMAR_PQH_REG ) &PRQ_RING_MASK ; handled = (head != tail);while (head != tail) { req = &iommu->prq[head /sizeof (*req)];
@@ -268,8 +268,8 @@ static irqreturn_t prq_event_thread(int irq, void *d)if (readl (iommu->reg +DMAR_PRS_REG ) &DMA_PRS_PRO ) {pr_info_ratelimited ("IOMMU: %s: PRQ overflow detected\n" , iommu->name); - head =dmar_readq (iommu->reg +DMAR_PQH_REG ) &PRQ_RING_MASK ; - tail =dmar_readq (iommu->reg +DMAR_PQT_REG ) &PRQ_RING_MASK ; + head =readq (iommu->reg +DMAR_PQH_REG ) &PRQ_RING_MASK ; + tail =readq (iommu->reg +DMAR_PQT_REG ) &PRQ_RING_MASK ;if (head == tail) {iopf_queue_discard_partial (iommu->iopf_queue);writel (DMA_PRS_PRO , iommu->reg +DMAR_PRS_REG );
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 6/10 iommu/vt-d: Remove dmar_writel() and dmar_writeq()
In reply to Lu BaoluFrom: Bjorn Helgaas <[email protected]>
dmar_writel() and dmar_writeq() do nothing other than expand to the generic writel() and writeq(), and the dmar_write*() wrappers are used inconsistently.
Remove the dmar_write*() wrappers and use writel() and writeq() directly.
--- drivers/iommu/intel/iommu.h | 3 --- drivers/iommu/intel/dmar.c | 2 +- drivers/iommu/intel/iommu.c | 12 ++++++------ drivers/iommu/intel/irq_remapping.c | 4 ++-- drivers/iommu/intel/perfmon.c | 22 +++++++++++----------- drivers/iommu/intel/prq.c | 14 +++++++------- 6 files changed, 27 insertions(+), 30 deletions(-)
drivers/iommu/intel/iommu.h +0 -3
@@ -148,9 +148,6 @@#define OFFSET_STRIDE (9) -#define dmar_writeq (a,v) writeq(v,a) -#define dmar_writel (a, v) writel(v, a) -#define DMAR_VER_MAJOR (v) (((v) & 0xf0) >> 4)#define DMAR_VER_MINOR (v) ((v) & 0x0f)
drivers/iommu/intel/dmar.c +1 -1
@@ -1661,7 +1661,7 @@ static void __dmar_enable_qi(struct intel_iommu *iommu)/* write zero to the tail reg */ writel (0 , iommu->reg +DMAR_IQT_REG ); -dmar_writeq (iommu->reg +DMAR_IQA_REG , val); +writeq (val, iommu->reg +DMAR_IQA_REG ); iommu->gcmd |=DMA_GCMD_QIE ;writel (iommu->gcmd, iommu->reg +DMAR_GCMD_REG );
drivers/iommu/intel/iommu.c +6 -6
@@ -697,7 +697,7 @@ static void iommu_set_root_entry(struct intel_iommu *iommu) addr |=DMA_RTADDR_SMT ;raw_spin_lock_irqsave (&iommu->register_lock, flag); -dmar_writeq (iommu->reg +DMAR_RTADDR_REG , addr); +writeq (addr, iommu->reg +DMAR_RTADDR_REG );writel (iommu->gcmd |DMA_GCMD_SRTP , iommu->reg +DMAR_GCMD_REG );
@@ -765,7 +765,7 @@ static void __iommu_flush_context(struct intel_iommu *iommu, val |=DMA_CCMD_ICC ;raw_spin_lock_irqsave (&iommu->register_lock, flag); -dmar_writeq (iommu->reg +DMAR_CCMD_REG , val); +writeq (val, iommu->reg +DMAR_CCMD_REG );/* Make sure hardware complete it */ IOMMU_WAIT_OP(iommu, DMAR_CCMD_REG,
@@ -806,8 +806,8 @@ void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,raw_spin_lock_irqsave (&iommu->register_lock, flag);/* Note: Only uses first TLB reg currently */ if (val_iva) -dmar_writeq (iommu->reg + tlb_offset, val_iva); -dmar_writeq (iommu->reg + tlb_offset +8 , val); +writeq (val_iva, iommu->reg + tlb_offset); +writeq (val, iommu->reg + tlb_offset +8 );/* Make sure hardware complete it */ IOMMU_WAIT_OP(iommu, tlb_offset + 8,
@@ -4201,8 +4201,8 @@ int ecmd_submit_sync(struct intel_iommu *iommu, u8 ecmd, u64 oa, u64 ob)* - It's not invoked in any critical path. The extra MMIO * write doesn't bring any performance concerns. */ -dmar_writeq (iommu->reg +DMAR_ECEO_REG , ob); -dmar_writeq (iommu->reg +DMAR_ECMD_REG , ecmd | (oa <<DMA_ECMD_OA_SHIFT )); +writeq (ob, iommu->reg +DMAR_ECEO_REG ); +writeq (ecmd | (oa <<DMA_ECMD_OA_SHIFT ), iommu->reg +DMAR_ECMD_REG );IOMMU_WAIT_OP (iommu,DMAR_ECRSP_REG , readq, !(res &DMA_ECMD_ECRSP_IP ), res);
drivers/iommu/intel/irq_remapping.c +2 -2
@@ -465,8 +465,8 @@ static void iommu_set_irq_remapping(struct intel_iommu *iommu, int mode)raw_spin_lock_irqsave (&iommu->register_lock, flags); -dmar_writeq (iommu->reg +DMAR_IRTA_REG , - (addr) |IR_X2APIC_MODE (mode) |INTR_REMAP_TABLE_REG_SIZE ); +writeq ((addr) |IR_X2APIC_MODE (mode) |INTR_REMAP_TABLE_REG_SIZE , + iommu->reg +DMAR_IRTA_REG );/* Set interrupt-remapping table pointer */ writel (iommu->gcmd |DMA_GCMD_SIRTP , iommu->reg +DMAR_GCMD_REG );
drivers/iommu/intel/perfmon.c +11 -11
@@ -99,20 +99,20 @@ IOMMU_PMU_ATTR(filter_page_table, "config2:32-36", IOMMU_PMU_FILTER_PAGE_TABLE);#define iommu_pmu_set_filter (_name, _config, _filter, _idx, _econfig) \ { \ if ((iommu_pmu->filter & _filter) && iommu_pmu_en_##_name(_econfig)) { \ - dmar_writel(iommu_pmu->cfg_reg + _idx * IOMMU_PMU_CFG_OFFSET + \ - IOMMU_PMU_CFG_SIZE + \ - (ffs(_filter) - 1) * IOMMU_PMU_CFG_FILTERS_OFFSET, \ - iommu_pmu_get_##_name(_config) | IOMMU_PMU_FILTER_EN);\ + writel(iommu_pmu_get_##_name(_config) | IOMMU_PMU_FILTER_EN, \ + iommu_pmu->cfg_reg + _idx * IOMMU_PMU_CFG_OFFSET + \ + IOMMU_PMU_CFG_SIZE + \ + (ffs(_filter) - 1) * IOMMU_PMU_CFG_FILTERS_OFFSET); \ } \ }#define iommu_pmu_clear_filter (_filter, _idx) \ { \ if (iommu_pmu->filter & _filter) { \ - dmar_writel(iommu_pmu->cfg_reg + _idx * IOMMU_PMU_CFG_OFFSET + \ - IOMMU_PMU_CFG_SIZE + \ - (ffs(_filter) - 1) * IOMMU_PMU_CFG_FILTERS_OFFSET, \ - 0); \ + writel(0, \ + iommu_pmu->cfg_reg + _idx * IOMMU_PMU_CFG_OFFSET + \ + IOMMU_PMU_CFG_SIZE + \ + (ffs(_filter) - 1) * IOMMU_PMU_CFG_FILTERS_OFFSET); \ } \ }
@@ -411,7 +411,7 @@ static int iommu_pmu_assign_event(struct iommu_pmu *iommu_pmu, hwc->idx = idx;/* config events */ -dmar_writeq (iommu_config_base (iommu_pmu, idx), hwc->config); +writeq (hwc->config,iommu_config_base (iommu_pmu, idx)); iommu_pmu_set_filter(requester_id, event->attr.config1, IOMMU_PMU_FILTER_REQUESTER_ID, idx,
@@ -510,7 +510,7 @@ static void iommu_pmu_counter_overflow(struct iommu_pmu *iommu_pmu)iommu_pmu_event_update (event); } -dmar_writeq (iommu_pmu->overflow, status); +writeq (status, iommu_pmu->overflow); } }
@@ -524,7 +524,7 @@ static irqreturn_t iommu_pmu_irq_handler(int irq, void *dev_id)iommu_pmu_counter_overflow (iommu->pmu);/* Clear the status bit */ -dmar_writel (iommu->reg +DMAR_PERFINTRSTS_REG ,DMA_PERFINTRSTS_PIS ); +writel (DMA_PERFINTRSTS_PIS , iommu->reg +DMAR_PERFINTRSTS_REG );return IRQ_HANDLED ; }
drivers/iommu/intel/prq.c +7 -7
@@ -259,7 +259,7 @@ static irqreturn_t prq_event_thread(int irq, void *d) head = (head +sizeof (*req)) &PRQ_RING_MASK ; } -dmar_writeq (iommu->reg +DMAR_PQH_REG , tail); +writeq (tail, iommu->reg +DMAR_PQH_REG );/* * Clear the page request overflow bit and wake up all threads that
@@ -325,9 +325,9 @@ int intel_iommu_enable_prq(struct intel_iommu *iommu) iommu->name); goto free_iopfq; } -dmar_writeq (iommu->reg +DMAR_PQH_REG ,0ULL ); -dmar_writeq (iommu->reg +DMAR_PQT_REG ,0ULL ); -dmar_writeq (iommu->reg +DMAR_PQA_REG ,virt_to_phys (iommu->prq) |PRQ_ORDER ); +writeq (0ULL , iommu->reg +DMAR_PQH_REG ); +writeq (0ULL , iommu->reg +DMAR_PQT_REG ); +writeq (virt_to_phys (iommu->prq) |PRQ_ORDER , iommu->reg +DMAR_PQA_REG );init_completion (&iommu->prq_complete);
@@ -348,9 +348,9 @@ int intel_iommu_enable_prq(struct intel_iommu *iommu)int intel_iommu_finish_prq (struct intel_iommu *iommu) { -dmar_writeq (iommu->reg +DMAR_PQH_REG ,0ULL ); -dmar_writeq (iommu->reg +DMAR_PQT_REG ,0ULL ); -dmar_writeq (iommu->reg +DMAR_PQA_REG ,0ULL ); +writeq (0ULL , iommu->reg +DMAR_PQH_REG ); +writeq (0ULL , iommu->reg +DMAR_PQT_REG ); +writeq (0ULL , iommu->reg +DMAR_PQA_REG );if (iommu->pr_irq) {free_irq (iommu->pr_irq, iommu);
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 7/10 iommu/vt-d: Split piotlb invalidation into range and all
In reply to Lu BaoluFrom: Jason Gunthorpe <[email protected]>
Currently these call chains are muddled up by using npages=-1, but only one caller has the possibility to do both options.
Simplify qi_flush_piotlb() to qi_flush_piotlb_all() since all callers pass npages=-1.
Split qi_batch_add_piotlb() into qi_batch_add_piotlb_all() and related helpers.
--- drivers/iommu/intel/iommu.h | 39 +++++++++++++++++-------------------- drivers/iommu/intel/cache.c | 20 ++++++++++++------- drivers/iommu/intel/dmar.c | 19 ++++-------------- drivers/iommu/intel/pasid.c | 6 +++--- drivers/iommu/intel/prq.c | 2 +- 5 files changed, 39 insertions(+), 47 deletions(-)
drivers/iommu/intel/iommu.h +18 -21
@@ -1077,31 +1077,29 @@ static inline void qi_desc_dev_iotlb(u16 sid, u16 pfsid, u16 qdep, u64 addr, desc->qw3 =0 ; } +/* PASID-selective IOTLB invalidation */ +static inline void qi_desc_piotlb_all (u16 did,u32 pasid,struct qi_desc *desc) +{ + desc->qw0 =QI_EIOTLB_PASID (pasid) |QI_EIOTLB_DID (did) | +QI_EIOTLB_GRAN (QI_GRAN_NONG_PASID ) |QI_EIOTLB_TYPE ; + desc->qw1 =0 ; +} + +/* Page-selective-within-PASID IOTLB invalidation */ static inline void qi_desc_piotlb (u16 did,u32 pasid,u64 addr,unsigned long npages,bool ih,struct qi_desc *desc) { -if (npages ==-1 ) { - desc->qw0 =QI_EIOTLB_PASID (pasid) | -QI_EIOTLB_DID (did) | -QI_EIOTLB_GRAN (QI_GRAN_NONG_PASID ) | -QI_EIOTLB_TYPE ; - desc->qw1 =0 ; - }else { -int mask =ilog2 (__roundup_pow_of_two (npages)); -unsigned long align = (1ULL << (VTD_PAGE_SHIFT + mask)); +int mask =ilog2 (__roundup_pow_of_two (npages)); +unsigned long align = (1ULL << (VTD_PAGE_SHIFT + mask)); -if (WARN_ON_ONCE (!IS_ALIGNED (addr, align))) - addr =ALIGN_DOWN (addr, align); +if (WARN_ON_ONCE (!IS_ALIGNED (addr, align))) + addr =ALIGN_DOWN (addr, align); - desc->qw0 =QI_EIOTLB_PASID (pasid) | -QI_EIOTLB_DID (did) | -QI_EIOTLB_GRAN (QI_GRAN_PSI_PASID ) | -QI_EIOTLB_TYPE ; - desc->qw1 =QI_EIOTLB_ADDR (addr) | -QI_EIOTLB_IH (ih) | -QI_EIOTLB_AM (mask); - } + desc->qw0 =QI_EIOTLB_PASID (pasid) |QI_EIOTLB_DID (did) | +QI_EIOTLB_GRAN (QI_GRAN_PSI_PASID ) |QI_EIOTLB_TYPE ; + desc->qw1 =QI_EIOTLB_ADDR (addr) |QI_EIOTLB_IH (ih) | +QI_EIOTLB_AM (mask); } static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,
@@ -1163,8 +1161,7 @@ void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,void qi_flush_dev_iotlb (struct intel_iommu *iommu,u16 sid,u16 pfsid,u16 qdep,u64 addr,unsigned mask); -void qi_flush_piotlb (struct intel_iommu *iommu,u16 did,u32 pasid,u64 addr, -unsigned long npages,bool ih); +void qi_flush_piotlb_all (struct intel_iommu *iommu,u16 did,u32 pasid); void qi_flush_dev_iotlb_pasid(struct intel_iommu *iommu, u16 sid, u16 pfsid, u32 pasid, u16 qdep, u64 addr,
drivers/iommu/intel/cache.c +13 -7
@@ -330,15 +330,17 @@ static void qi_batch_add_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 pfsidqi_batch_increment_index (iommu, batch); } +static void qi_batch_add_piotlb_all (struct intel_iommu *iommu,u16 did, +u32 pasid,struct qi_batch *batch) +{ +qi_desc_piotlb_all (did, pasid, &batch->descs[batch->index]); +qi_batch_increment_index (iommu, batch); +} +static void qi_batch_add_piotlb (struct intel_iommu *iommu,u16 did,u32 pasid,u64 addr,unsigned long npages,bool ih,struct qi_batch *batch) { -/* -* npages == -1 means a PASID-selective invalidation, otherwise, -* a positive value for Page-selective-within-PASID invalidation. -* 0 is not a valid input. -*/ if (!npages)return ;
@@ -378,8 +380,12 @@ static void cache_tag_flush_iotlb(struct dmar_domain *domain, struct cache_tag *u64 type =DMA_TLB_PSI_FLUSH ;if (intel_domain_use_piotlb (domain)) { -qi_batch_add_piotlb (iommu, tag->domain_id, tag->pasid, addr, - pages, ih, domain->qi_batch); +if (pages ==-1 ) +qi_batch_add_piotlb_all (iommu, tag->domain_id, + tag->pasid, domain->qi_batch); +else +qi_batch_add_piotlb (iommu, tag->domain_id, tag->pasid, + addr, pages, ih, domain->qi_batch);return ; }
drivers/iommu/intel/dmar.c +4 -15
@@ -1550,23 +1550,12 @@ void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid,qi_submit_sync (iommu, &desc,1 ,0 ); } -/* PASID-based IOTLB invalidation */ -void qi_flush_piotlb (struct intel_iommu *iommu,u16 did,u32 pasid,u64 addr, -unsigned long npages,bool ih) +/* PASID-selective IOTLB invalidation */ +void qi_flush_piotlb_all (struct intel_iommu *iommu,u16 did,u32 pasid) { -struct qi_desc desc = {.qw2 =0 , .qw3 =0 }; +struct qi_desc desc = {}; -/* -* npages == -1 means a PASID-selective invalidation, otherwise, -* a positive value for Page-selective-within-PASID invalidation. -* 0 is not a valid input. -*/ -if (WARN_ON (!npages)) { -pr_err ("Invalid input npages = %ld\n" , npages); -return ; - } - -qi_desc_piotlb (did, pasid, addr, npages, ih, &desc); +qi_desc_piotlb_all (did, pasid, &desc);qi_submit_sync (iommu, &desc,1 ,0 ); }
drivers/iommu/intel/pasid.c +3 -3
@@ -282,7 +282,7 @@ void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev,pasid_cache_invalidation_with_pasid (iommu, did, pasid);if (pgtt ==PASID_ENTRY_PGTT_PT || pgtt ==PASID_ENTRY_PGTT_FL_ONLY ) -qi_flush_piotlb (iommu, did, pasid,0 ,-1 ,0 ); +qi_flush_piotlb_all (iommu, did, pasid);else iommu->flush.flush_iotlb (iommu, did,0 ,0 ,DMA_TLB_DSI_FLUSH );
@@ -308,7 +308,7 @@ static void pasid_flush_caches(struct intel_iommu *iommu,if (cap_caching_mode (iommu->cap)) {pasid_cache_invalidation_with_pasid (iommu, did, pasid); -qi_flush_piotlb (iommu, did, pasid,0 ,-1 ,0 ); +qi_flush_piotlb_all (iommu, did, pasid); }else {iommu_flush_write_buffer (iommu); }
@@ -342,7 +342,7 @@ static void intel_pasid_flush_present(struct intel_iommu *iommu,* Addr[63:12]=0x7FFFFFFF_FFFFF) to affected functions */ pasid_cache_invalidation_with_pasid (iommu, did, pasid); -qi_flush_piotlb (iommu, did, pasid,0 ,-1 ,0 ); +qi_flush_piotlb_all (iommu, did, pasid);devtlb_invalidation_with_pasid (iommu, dev, pasid); }
drivers/iommu/intel/prq.c +1 -1
@@ -113,7 +113,7 @@ void intel_iommu_drain_pasid_prq(struct device *dev, u32 pasid)qi_desc_dev_iotlb (sid, info->pfsid, info->ats_qdep,0 ,MAX_AGAW_PFN_WIDTH , &desc[2 ]); }else { -qi_desc_piotlb (did, pasid,0 ,-1 ,0 , &desc[1 ]); +qi_desc_piotlb_all (did, pasid, &desc[1 ]);qi_desc_dev_iotlb_pasid (sid, info->pfsid, pasid, info->ats_qdep,0 ,MAX_AGAW_PFN_WIDTH , &desc[2 ]); }
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 8/10 iommu/vt-d: Pass size_order to qi_desc_piotlb() not npages
In reply to Lu BaoluFrom: Jason Gunthorpe <[email protected]>
It doesn't make sense for the caller to compute mask, throw it away and then have qi_desc_piotlb() compute it again.
--- drivers/iommu/intel/iommu.h | 13 +++++-------- drivers/iommu/intel/cache.c | 10 ++++------ 2 files changed, 9 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.h +5 -8
@@ -1087,19 +1087,16 @@ static inline void qi_desc_piotlb_all(u16 did, u32 pasid, struct qi_desc *desc)/* Page-selective-within-PASID IOTLB invalidation */ static inline void qi_desc_piotlb (u16 did,u32 pasid,u64 addr, -unsigned long npages,bool ih, +unsigned int size_order,bool ih,struct qi_desc *desc) { -int mask =ilog2 (__roundup_pow_of_two (npages)); -unsigned long align = (1ULL << (VTD_PAGE_SHIFT + mask)); - -if (WARN_ON_ONCE (!IS_ALIGNED (addr, align))) - addr =ALIGN_DOWN (addr, align); - +/* +* calculate_psi_aligned_address() must be used for addr and size_order +*/ desc->qw0 =QI_EIOTLB_PASID (pasid) |QI_EIOTLB_DID (did) |QI_EIOTLB_GRAN (QI_GRAN_PSI_PASID ) |QI_EIOTLB_TYPE ; desc->qw1 =QI_EIOTLB_ADDR (addr) |QI_EIOTLB_IH (ih) | -QI_EIOTLB_AM (mask); +QI_EIOTLB_AM (size_order); } static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,
drivers/iommu/intel/cache.c +4 -6
@@ -338,13 +338,11 @@ static void qi_batch_add_piotlb_all(struct intel_iommu *iommu, u16 did, }static void qi_batch_add_piotlb (struct intel_iommu *iommu,u16 did,u32 pasid, -u64 addr,unsigned long npages,bool ih, +u64 addr,unsigned int size_order,bool ih,struct qi_batch *batch) { -if (!npages) -return ; - -qi_desc_piotlb (did, pasid, addr, npages, ih, &batch->descs[batch->index]); +qi_desc_piotlb (did, pasid, addr, size_order, ih, + &batch->descs[batch->index]);qi_batch_increment_index (iommu, batch); }
@@ -385,7 +383,7 @@ static void cache_tag_flush_iotlb(struct dmar_domain *domain, struct cache_tag * tag->pasid, domain->qi_batch); else qi_batch_add_piotlb(iommu, tag->domain_id, tag->pasid, - addr, pages, ih, domain->qi_batch); + addr, mask, ih, domain->qi_batch);return ; }
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 9/10 iommu/vt-d: Remove the remaining pages along the invalidation path
In reply to Lu BaoluFrom: Jason Gunthorpe <[email protected]>
This was only being used to signal that a flush all should be used. Use mask/size_order >= 52 to signal this instead.
--- drivers/iommu/intel/trace.h | 18 ++++++++---------- drivers/iommu/intel/cache.c | 27 +++++++++++---------------- 2 files changed, 19 insertions(+), 26 deletions(-)
drivers/iommu/intel/trace.h +8 -10
@@ -132,8 +132,8 @@ DEFINE_EVENT(cache_tag_log, cache_tag_unassign, DECLARE_EVENT_CLASS(cache_tag_flush, TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, - unsigned long addr, unsigned long pages, unsigned long mask), - TP_ARGS(tag, start, end, addr, pages, mask), + unsigned long addr, unsigned long mask), + TP_ARGS(tag, start, end, addr, mask), TP_STRUCT__entry( __string(iommu, tag->iommu->name) __string(dev, dev_name(tag->dev))
@@ -143,7 +143,6 @@ DECLARE_EVENT_CLASS(cache_tag_flush, __field(unsigned long, start) __field(unsigned long, end) __field(unsigned long, addr) - __field(unsigned long, pages) __field(unsigned long, mask) ), TP_fast_assign(
@@ -155,10 +154,9 @@ DECLARE_EVENT_CLASS(cache_tag_flush, __entry->start = start; __entry->end = end; __entry->addr = addr; - __entry->pages = pages; __entry->mask = mask; ), - TP_printk("%s %s[%d] type %s did %d [0x%lx-0x%lx] addr 0x%lx pages 0x%lx mask 0x%lx", + TP_printk("%s %s[%d] type %s did %d [0x%lx-0x%lx] addr 0x%lx mask 0x%lx", __get_str(iommu), __get_str(dev), __entry->pasid, __print_symbolic(__entry->type, { CACHE_TAG_IOTLB, "iotlb" },
@@ -166,20 +164,20 @@ DECLARE_EVENT_CLASS(cache_tag_flush, { CACHE_TAG_NESTING_IOTLB, "nesting_iotlb" }, { CACHE_TAG_NESTING_DEVTLB, "nesting_devtlb" }), __entry->domain_id, __entry->start, __entry->end, - __entry->addr, __entry->pages, __entry->mask + __entry->addr, __entry->mask ) ); DEFINE_EVENT(cache_tag_flush, cache_tag_flush_range, TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, - unsigned long addr, unsigned long pages, unsigned long mask), - TP_ARGS(tag, start, end, addr, pages, mask) + unsigned long addr, unsigned long mask), + TP_ARGS(tag, start, end, addr, mask) ); DEFINE_EVENT(cache_tag_flush, cache_tag_flush_range_np, TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, - unsigned long addr, unsigned long pages, unsigned long mask), - TP_ARGS(tag, start, end, addr, pages, mask) + unsigned long addr, unsigned long mask), + TP_ARGS(tag, start, end, addr, mask) ); #endif /* _TRACE_INTEL_IOMMU_H */
drivers/iommu/intel/cache.c +11 -16
@@ -255,7 +255,6 @@ void cache_tag_unassign_domain(struct dmar_domain *domain,static unsigned long calculate_psi_aligned_address (unsigned long start,unsigned long end, -unsigned long *_pages,unsigned long *_mask) {unsigned long pages =aligned_nrpages (start, end - start +1 );
@@ -281,10 +280,8 @@ static unsigned long calculate_psi_aligned_address(unsigned long start,*/ shared_bits = ~(pfn ^ end_pfn) & ~bitmask; mask = shared_bits ?__ffs (shared_bits) :MAX_AGAW_PFN_WIDTH ; - aligned_pages =1UL << mask; } - *_pages = aligned_pages; *_mask = mask;return ALIGN_DOWN (start,VTD_PAGE_SIZE << mask);
@@ -371,14 +368,13 @@ static bool intel_domain_use_piotlb(struct dmar_domain *domain) }static void cache_tag_flush_iotlb (struct dmar_domain *domain,struct cache_tag *tag, -unsigned long addr,unsigned long pages, -unsigned long mask,int ih) +unsigned long addr,unsigned long mask,int ih) {struct intel_iommu *iommu = tag->iommu;u64 type =DMA_TLB_PSI_FLUSH ;if (intel_domain_use_piotlb (domain)) { -if (pages ==-1 ) +if (mask >=MAX_AGAW_PFN_WIDTH )qi_batch_add_piotlb_all (iommu, tag->domain_id, tag->pasid, domain->qi_batch);else
@@ -392,7 +388,7 @@ static void cache_tag_flush_iotlb(struct dmar_domain *domain, struct cache_tag ** is too big. */ if (!cap_pgsel_inv (iommu->cap) || - mask >cap_max_amask_val (iommu->cap) || pages ==-1 ) { + mask >cap_max_amask_val (iommu->cap)) { addr =0 ; mask =0 ; ih =0 ;
@@ -441,16 +437,15 @@ void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start, unsigned long end, int ih) {struct intel_iommu *iommu =NULL ; -unsigned long pages, mask, addr; +unsigned long mask, addr;struct cache_tag *tag;unsigned long flags;if (start ==0 && end ==ULONG_MAX ) { addr =0 ; - pages =-1 ; mask =MAX_AGAW_PFN_WIDTH ; }else { - addr =calculate_psi_aligned_address (start, end, &pages, &mask); + addr =calculate_psi_aligned_address (start, end, &mask); }spin_lock_irqsave (&domain->cache_lock, flags);
@@ -462,7 +457,7 @@ void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start,switch (tag->type) {case CACHE_TAG_IOTLB :case CACHE_TAG_NESTING_IOTLB : -cache_tag_flush_iotlb (domain, tag, addr, pages, mask, ih); +cache_tag_flush_iotlb (domain, tag, addr, mask, ih);break ;case CACHE_TAG_NESTING_DEVTLB :/*
@@ -480,7 +475,7 @@ void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start,break ; } -trace_cache_tag_flush_range (tag, start, end, addr, pages, mask); +trace_cache_tag_flush_range (tag, start, end, addr, mask); }qi_batch_flush_descs (iommu, domain->qi_batch);spin_unlock_irqrestore (&domain->cache_lock, flags);
@@ -510,11 +505,11 @@ void cache_tag_flush_range_np(struct dmar_domain *domain, unsigned long start, unsigned long end) {struct intel_iommu *iommu =NULL ; -unsigned long pages, mask, addr; +unsigned long mask, addr;struct cache_tag *tag;unsigned long flags; - addr =calculate_psi_aligned_address (start, end, &pages, &mask); + addr =calculate_psi_aligned_address (start, end, &mask);spin_lock_irqsave (&domain->cache_lock, flags);list_for_each_entry (tag, &domain->cache_tags, node) {
@@ -530,9 +525,9 @@ void cache_tag_flush_range_np(struct dmar_domain *domain, unsigned long start,if (tag->type ==CACHE_TAG_IOTLB || tag->type ==CACHE_TAG_NESTING_IOTLB ) -cache_tag_flush_iotlb (domain, tag, addr, pages, mask,0 ); +cache_tag_flush_iotlb (domain, tag, addr, mask,0 ); -trace_cache_tag_flush_range_np (tag, start, end, addr, pages, mask); +trace_cache_tag_flush_range_np (tag, start, end, addr, mask); }qi_batch_flush_descs (iommu, domain->qi_batch);spin_unlock_irqrestore (&domain->cache_lock, flags);
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 10/10 iommu/vt-d: Simplify calculate_psi_aligned_address()
In reply to Lu BaoluFrom: Jason Gunthorpe <[email protected]>
This is doing far too much math for the simple task of finding a power of 2 that fully spans the given range. Use fls directly on the xor which computes the common binary prefix.
--- drivers/iommu/intel/cache.c | 49 ++++++++++++------------------------- 1 file changed, 16 insertions(+), 33 deletions(-)
drivers/iommu/intel/cache.c +16 -33
@@ -254,37 +254,25 @@ void cache_tag_unassign_domain(struct dmar_domain *domain, }static unsigned long calculate_psi_aligned_address (unsigned long start, -unsigned long end, -unsigned long *_mask) +unsigned long last, +unsigned long *size_order) { -unsigned long pages =aligned_nrpages (start, end - start +1 ); -unsigned long aligned_pages =__roundup_pow_of_two (pages); -unsigned long bitmask = aligned_pages -1 ; -unsigned long mask =ilog2 (aligned_pages); -unsigned long pfn =IOVA_PFN (start); +unsigned int sz_lg2; -/* -* PSI masks the low order bits of the base address. If the -* address isn't aligned to the mask, then compute a mask value -* needed to ensure the target range is flushed. -*/ -if (unlikely (bitmask & pfn)) { -unsigned long end_pfn = pfn + pages -1 , shared_bits; - -/* -* Since end_pfn <= pfn + bitmask, the only way bits -* higher than bitmask can differ in pfn and end_pfn is -* by carrying. This means after masking out bitmask, -* high bits starting with the first set bit in -* shared_bits are all equal in both pfn and end_pfn. -*/ - shared_bits = ~(pfn ^ end_pfn) & ~bitmask; - mask = shared_bits ?__ffs (shared_bits) :MAX_AGAW_PFN_WIDTH ; +/* Compute a sz_lg2 that spans start and last */ + start &=GENMASK (BITS_PER_LONG -1 ,VTD_PAGE_SHIFT ); + sz_lg2 =fls_long (start ^ last); +if (sz_lg2 <=12 ) { + *size_order =0 ; +return start; + } +if (unlikely (sz_lg2 >=MAX_AGAW_PFN_WIDTH )) { + *size_order =MAX_AGAW_PFN_WIDTH ; +return 0 ; } - *_mask = mask; - -return ALIGN_DOWN (start,VTD_PAGE_SIZE << mask); + *size_order = sz_lg2 -VTD_PAGE_SHIFT ; +return start &GENMASK (BITS_PER_LONG -1 , sz_lg2); }static void qi_batch_flush_descs (struct intel_iommu *iommu,struct qi_batch *batch)
@@ -441,12 +429,7 @@ void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start,struct cache_tag *tag;unsigned long flags; -if (start ==0 && end ==ULONG_MAX ) { - addr =0 ; - mask =MAX_AGAW_PFN_WIDTH ; - }else { - addr =calculate_psi_aligned_address (start, end, &mask); - } + addr =calculate_psi_aligned_address (start, end, &mask);spin_lock_irqsave (&domain->cache_lock, flags);list_for_each_entry (tag, &domain->cache_tags, node) {
Signature · 1 line
2.43.0
Baolu Lu
maintainerRe: [PATCH 10/10] iommu/vt-d: Simplify calculate_psi_aligned_address()
Quoted 6 lines · Lu Baolu
From: Jason Gunthorpe<[email protected]>
This is doing far too much math for the simple task of finding a power of 2 that fully spans the given range. Use fls directly on the xor which computes the common binary prefix.
Quoted 6 lines
Signed-off-by Jason Gunthorpe<[email protected]>Signed-off-by Lu Baolu<[email protected]>--- drivers/iommu/intel/cache.c | 49 ++++++++++++------------------------- 1 file changed, 16 insertions(+), 33 deletions(-)
Hi Joerg,
Can you please remove this last patch from the pull request? The AI reviewer reported an issue in this patch.
https://sashiko.dev/#/patchset/20260402065734.1687476-1-baolu.lu%40linux.intel.com
We need more time to evaluate and address this before it is merged into linux-next.
Thanks,
baolu
Joerg Roedel
maintainerRe: [PATCH 10/10] iommu/vt-d: Simplify calculate_psi_aligned_address()
Quoted 17 lines · Baolu Lu
Quoted 6 lines · Lu Baolu
From: Jason Gunthorpe<[email protected]>
This is doing far too much math for the simple task of finding a power of 2 that fully spans the given range. Use fls directly on the xor which computes the common binary prefix.
Quoted 7 lines
Signed-off-by Jason Gunthorpe<[email protected]>Signed-off-by Lu Baolu<[email protected]>--- drivers/iommu/intel/cache.c | 49 ++++++++++++------------------------- 1 file changed, 16 insertions(+), 33 deletions(-)Hi Joerg,
Quoted 7 lines
Can you please remove this last patch from the pull request? The AI reviewer reported an issue in this patch.
https://sashiko.dev/#/patchset/20260402065734.1687476-1-baolu.lu%40linux.intel.com
We need more time to evaluate and address this before it is merged into linux-next.
Removed.
Jason Gunthorpe
maintainerRe: [PATCH 10/10] iommu/vt-d: Simplify calculate_psi_aligned_address()
In reply to Baolu LuQuoted 15 lines · Baolu Lu
Quoted 6 lines · Lu Baolu
From: Jason Gunthorpe<[email protected]>
This is doing far too much math for the simple task of finding a power of 2 that fully spans the given range. Use fls directly on the xor which computes the common binary prefix.
Quoted 7 lines
Signed-off-by Jason Gunthorpe<[email protected]>Signed-off-by Lu Baolu<[email protected]>--- drivers/iommu/intel/cache.c | 49 ++++++++++++------------------------- 1 file changed, 16 insertions(+), 33 deletions(-)
Quoted 6 lines
Hi Joerg,
Can you please remove this last patch from the pull request? The AI reviewer reported an issue in this patch.
https://sashiko.dev/#/patchset/20260402065734.1687476-1-baolu.lu%40linux.intel.com
Yeah, that's an interesting remark. I think this is enough to deal with all of its items:
- if (unlikely(sz_lg2 >= MAX_AGAW_PFN_WIDTH)) {
+ if (unlikely(sz_lg2 >= BITS_PER_LONG)) {
+ /*
+ * MAX_AGAW_PFN_WIDTH triggers full invalidation in all
+ * downstream users.
+ */
*size_order = MAX_AGAW_PFN_WIDTH;
return 0;
}1) Yes AGAW_PFN_WIDTH is in "PFN" not byte notation, so it is off by 12 bits and we would errantly move to full invalidation too soon
2) Yes, if sz_lg2 is BITS_PER_LONG the GENMASK explodes, in this case it should trigger full invalidation even if ulong is 32 bits
3) Yes, we sould retain the 0/ULONG_MASK means full invalidation, but this happens properly now because of the above BITS_PER_LONG check so no need to bring back the other check.
I'll post an updated patch
Jason
Quoted 10 lines · Lu Baolu
Bjorn Helgaas (2):
iommu/vt-d: Removedmar_readl()anddmar_readq()iommu/vt-d: Removedmar_writel()anddmar_writeq()Jason Gunthorpe (4):
iommu/vt-d: Split piotlb invalidation into range and all iommu/vt-d: Passsize_ordertoqi_desc_piotlb()not npages iommu/vt-d: Remove the remaining pages along the invalidation path iommu/vt-d: Simplifycalculate_psi_aligned_address()
Quoted 7 lines
Zhenzhong Duan (4):
iommu/vt-d: Block PASID attachment to nested domain with dirty
tracking
iommu/vt-d: Renamedevice_set_dirty_tracking()and passdmar_domain
pointer
iommu/vt-d: Support dirty tracking on PASID iommufd/selftest: Test dirty tracking on PASID
Applied, thanks Baolu.