Try to reduce lock contention on the SMMUv3 command queue
Subsystems (17)
Will Deacon
maintainerv2 cover Try to reduce lock contention on the SMMUv3 command queue
Hi everyone,
This is a significant rework of the RFC I previously posted here:
https://lkml.kernel.org/r/[email protected]
But this time, it looks like it might actually be worthwhile according to my perf profiles, where __iommu_unmap() falls a long way down the profile for a multi-threaded netperf run. I'm still relying on others to confirm this is useful, however.
Some of the changes since last time are:
* Support for constructing and submitting a list of commands in the driver
* Numerous changes to the IOMMU and io-pgtable APIs so that we can submit commands in batches
* Removal of cmpxchg() from cmdq_shared_lock() fast-path
* Code restructuring and cleanups
This current applies against my iommu/devel branch that Joerg has pulled for 5.3. If you want to test it out, I've put everything here:
https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=iommu/cmdq
Feedback welcome. I appreciate that we're in the merge window, but I wanted to get this on the list for people to look at as an RFC.
Cheers,
Will
--->8
Will Deacon (19):
iommu: Remove empty iommu_tlb_range_add() callback from iommu_ops iommu/io-pgtable-arm: Remove redundant call to io_pgtable_tlb_sync() iommu/io-pgtable: Rename iommu_gather_ops to iommu_flush_ops
iommu: Introduce struct iommu_iotlb_gather for batching TLB flushes
iommu: Introduce iommu_iotlb_gather_add_page()
iommu: Pass struct iommu_iotlb_gather to ->unmap() and ->iotlb_sync() iommu/io-pgtable: Introduce tlb_flush_walk() and tlb_flush_leaf() iommu/io-pgtable: Hook up ->tlb_flush_walk() and ->tlb_flush_leaf() in
drivers
iommu/io-pgtable-arm: Call ->tlb_flush_walk() and ->tlb_flush_leaf() iommu/io-pgtable: Replace ->tlb_add_flush() with ->tlb_add_page() iommu/io-pgtable: Remove unused ->tlb_sync() callback iommu/io-pgtable: Pass struct iommu_iotlb_gather to ->unmap() iommu/io-pgtable: Pass struct iommu_iotlb_gather to ->tlb_add_page() iommu/arm-smmu-v3: Separate s/w and h/w views of prod and cons indexes iommu/arm-smmu-v3: Drop unused 'q' argument from Q_OVF macro iommu/arm-smmu-v3: Move low-level queue fields out of arm_smmu_queue iommu/arm-smmu-v3: Operate directly on low-level queue where possible iommu/arm-smmu-v3: Reduce contention during command-queue insertion iommu/arm-smmu-v3: Defer TLB invalidation until ->iotlb_sync()
drivers/gpu/drm/panfrost/panfrost_mmu.c | 24 +- drivers/iommu/amd_iommu.c | 11 +- drivers/iommu/arm-smmu-v3.c | 856 ++++++++++++++++++++++++-------- drivers/iommu/arm-smmu.c | 103 +++- drivers/iommu/dma-iommu.c | 9 +- drivers/iommu/exynos-iommu.c | 3 +- drivers/iommu/intel-iommu.c | 3 +- drivers/iommu/io-pgtable-arm-v7s.c | 57 +-- drivers/iommu/io-pgtable-arm.c | 48 +- drivers/iommu/iommu.c | 24 +- drivers/iommu/ipmmu-vmsa.c | 28 +- drivers/iommu/msm_iommu.c | 42 +- drivers/iommu/mtk_iommu.c | 45 +- drivers/iommu/mtk_iommu_v1.c | 3 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 44 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 3 +- drivers/iommu/tegra-gart.c | 12 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/vfio/vfio_iommu_type1.c | 27 +- include/linux/io-pgtable.h | 57 ++- include/linux/iommu.h | 92 +++- 23 files changed, 1090 insertions(+), 407 deletions(-)
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 1/19 iommu: Remove empty iommu_tlb_range_add() callback from iommu_ops
Commit add02cfdc9bc ("iommu: Introduce Interface for IOMMU TLB Flushing") added three new TLB flushing operations to the IOMMU API so that the underlying driver operations can be batched when unmapping large regions of IO virtual address space.
However, the ->iotlb_range_add() callback has not been implemented by any IOMMU drivers (amd_iommu.c implements it as an empty function, which incurs the overhead of an indirect branch). Instead, drivers either flush the entire IOTLB in the ->iotlb_sync() callback or perform the necessary invalidation during ->unmap().
Attempting to implement ->iotlb_range_add() for arm-smmu-v3.c revealed two major issues:
1. The page size used to map the region in the page-table is not known, and so it is not generally possible to issue TLB flushes in the most efficient manner.
2. The only mutable state passed to the callback is a pointer to the iommu_domain, which can be accessed concurrently and therefore requires expensive synchronisation to keep track of the outstanding flushes.
Remove the callback entirely in preparation for extending ->unmap() and ->iotlb_sync() to update a token on the caller's stack.
--- drivers/iommu/amd_iommu.c | 6 ------ drivers/iommu/iommu.c | 3 --- drivers/vfio/vfio_iommu_type1.c | 1 - include/linux/iommu.h | 15 --------------- 4 files changed, 25 deletions(-)
drivers/iommu/amd_iommu.c +0 -6
@@ -3196,11 +3196,6 @@ static void amd_iommu_flush_iotlb_all(struct iommu_domain *domain)domain_flush_complete (dom); } -static void amd_iommu_iotlb_range_add (struct iommu_domain *domain, -unsigned long iova,size_t size) -{ -} -const struct iommu_ops amd_iommu_ops = { .capable = amd_iommu_capable, .domain_alloc = amd_iommu_domain_alloc,
@@ -3219,7 +3214,6 @@ const struct iommu_ops amd_iommu_ops = { .is_attach_deferred = amd_iommu_is_attach_deferred, .pgsize_bitmap = AMD_IOMMU_PGSIZES, .flush_iotlb_all = amd_iommu_flush_iotlb_all, - .iotlb_range_add = amd_iommu_iotlb_range_add, .iotlb_sync = amd_iommu_flush_iotlb_all, };
drivers/iommu/iommu.c +0 -3
@@ -1681,9 +1681,6 @@ static size_t __iommu_unmap(struct iommu_domain *domain,if (!unmapped_page)break ; -if (sync && ops->iotlb_range_add) - ops->iotlb_range_add (domain, iova, pgsize); -pr_debug ("unmapped: iova 0x%lx size 0x%zx\n" , iova, unmapped_page);
drivers/vfio/vfio_iommu_type1.c +0 -1
@@ -712,7 +712,6 @@ static size_t unmap_unpin_fast(struct vfio_domain *domain,if (!unmapped) {kfree (entry); }else { -iommu_tlb_range_add (domain->domain, *iova, unmapped); entry->iova = *iova; entry->phys = phys; entry->len = unmapped;
include/linux/iommu.h +0 -15
@@ -204,7 +204,6 @@ struct iommu_sva_ops {* @map: map a physically contiguous memory region to an iommu domain * @unmap: unmap a physically contiguous memory region from an iommu domain * @flush_iotlb_all: Synchronously flush all hardware TLBs for this domain -* @iotlb_range_add: Add a given iova range to the flush queue for this domain * @iotlb_sync_map: Sync mappings created recently using @map to the hardware * @iotlb_sync: Flush all queued ranges from the hardware TLBs and empty flush * queue
@@ -246,8 +245,6 @@ struct iommu_ops {size_t (*unmap)(struct iommu_domain *domain,unsigned long iova,size_t size);void (*flush_iotlb_all)(struct iommu_domain *domain); -void (*iotlb_range_add)(struct iommu_domain *domain, -unsigned long iova,size_t size);void (*iotlb_sync_map)(struct iommu_domain *domain);void (*iotlb_sync)(struct iommu_domain *domain);phys_addr_t (*iova_to_phys)(struct iommu_domain *domain,dma_addr_t iova);
@@ -420,13 +417,6 @@ static inline void iommu_flush_tlb_all(struct iommu_domain *domain) domain->ops->flush_iotlb_all (domain); } -static inline void iommu_tlb_range_add (struct iommu_domain *domain, -unsigned long iova,size_t size) -{ -if (domain->ops->iotlb_range_add) - domain->ops->iotlb_range_add (domain, iova, size); -} -static inline void iommu_tlb_sync (struct iommu_domain *domain) {if (domain->ops->iotlb_sync)
@@ -580,11 +570,6 @@ static inline void iommu_flush_tlb_all(struct iommu_domain *domain) { } -static inline void iommu_tlb_range_add (struct iommu_domain *domain, -unsigned long iova,size_t size) -{ -} -static inline void iommu_tlb_sync (struct iommu_domain *domain) { }
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 2/19 iommu/io-pgtable-arm: Remove redundant call to io_pgtable_tlb_sync()
In reply to Will DeaconCommit b6b65ca20bc9 ("iommu/io-pgtable-arm: Add support for non-strict mode") added an unconditional call to io_pgtable_tlb_sync() immediately after the case where we replace a block entry with a table entry during an unmap() call. This is redundant, since the IOMMU API will call iommu_tlb_sync() on this path and the patch in question mentions this:
| To save having to reason about it too much, make sure the invalidation | in arm_lpae_split_blk_unmap() just performs its own unconditional sync | to minimise the window in which we're technically violating the break- | before-make requirement on a live mapping. This might work out redundant | with an outer-level sync for strict unmaps, but we'll never be splitting | blocks on a DMA fastpath anyway.
However, this sync gets in the way of deferred TLB invalidation for leaf entries and is at best a questionable, unproven hack. Remove it.
--- drivers/iommu/io-pgtable-arm-v7s.c | 1 - drivers/iommu/io-pgtable-arm.c | 1 - 2 files changed, 2 deletions(-)
drivers/iommu/io-pgtable-arm-v7s.c +0 -1
@@ -598,7 +598,6 @@ static size_t arm_v7s_split_blk_unmap(struct arm_v7s_io_pgtable *data, }io_pgtable_tlb_add_flush (&data->iop, iova, size, size, true); -io_pgtable_tlb_sync (&data->iop);return size; }
drivers/iommu/io-pgtable-arm.c +0 -1
@@ -594,7 +594,6 @@ static size_t arm_lpae_split_blk_unmap(struct arm_lpae_io_pgtable *data, tablep =iopte_deref (pte, data); }else if (unmap_idx >=0 ) {io_pgtable_tlb_add_flush (&data->iop, iova, size, size, true); -io_pgtable_tlb_sync (&data->iop);return size; }
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 3/19 iommu/io-pgtable: Rename iommu_gather_ops to iommu_flush_ops
In reply to Will DeaconIn preparation for TLB flush gathering in the IOMMU API, rename the iommu_gather_ops structure in io-pgtable to iommu_flush_ops, which better describes its purpose and avoids the potential for confusion between different levels of the API.
$ find linux/ -type f -name '*.[ch]' | xargs sed -i 's/gather_ops/flush_ops/g'
--- drivers/gpu/drm/panfrost/panfrost_mmu.c | 2 +- drivers/iommu/arm-smmu-v3.c | 4 ++-- drivers/iommu/arm-smmu.c | 8 ++++---- drivers/iommu/io-pgtable-arm-v7s.c | 2 +- drivers/iommu/io-pgtable-arm.c | 2 +- drivers/iommu/ipmmu-vmsa.c | 4 ++-- drivers/iommu/msm_iommu.c | 4 ++-- drivers/iommu/mtk_iommu.c | 4 ++-- drivers/iommu/qcom_iommu.c | 4 ++-- include/linux/io-pgtable.h | 6 +++--- 10 files changed, 20 insertions(+), 20 deletions(-)
drivers/gpu/drm/panfrost/panfrost_mmu.c +1 -1
@@ -249,7 +249,7 @@ static void mmu_tlb_sync_context(void *cookie)// TODO: Wait 1000 GPU cycles for HW_ISSUE_6367/T60X } -static const struct iommu_gather_ops mmu_tlb_ops = { +static const struct iommu_flush_ops mmu_tlb_ops = { .tlb_flush_all = mmu_tlb_inv_context_s1, .tlb_add_flush = mmu_tlb_inv_range_nosync, .tlb_sync = mmu_tlb_sync_context,
drivers/iommu/arm-smmu-v3.c +2 -2
@@ -1597,7 +1597,7 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size, }while (size -= granule); } -static const struct iommu_gather_ops arm_smmu_gather_ops = { +static const struct iommu_flush_ops arm_smmu_flush_ops = { .tlb_flush_all = arm_smmu_tlb_inv_context, .tlb_add_flush = arm_smmu_tlb_inv_range_nosync, .tlb_sync = arm_smmu_tlb_sync,
@@ -1790,7 +1790,7 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain) .ias = ias, .oas = oas, .coherent_walk = smmu->features & ARM_SMMU_FEAT_COHERENCY, - .tlb = &arm_smmu_gather_ops, + .tlb = &arm_smmu_flush_ops, .iommu_dev = smmu->dev, };
drivers/iommu/arm-smmu.c +4 -4
@@ -254,7 +254,7 @@ enum arm_smmu_domain_stage {struct arm_smmu_domain {struct arm_smmu_device *smmu;struct io_pgtable_ops *pgtbl_ops; -const struct iommu_gather_ops *tlb_ops; +const struct iommu_flush_ops *tlb_ops;struct arm_smmu_cfg cfg;enum arm_smmu_domain_stage stage;bool non_strict;
@@ -550,19 +550,19 @@ static void arm_smmu_tlb_inv_vmid_nosync(unsigned long iova, size_t size,writel_relaxed (smmu_domain->cfg.vmid, base +ARM_SMMU_GR0_TLBIVMID ); } -static const struct iommu_gather_ops arm_smmu_s1_tlb_ops = { +static const struct iommu_flush_ops arm_smmu_s1_tlb_ops = { .tlb_flush_all = arm_smmu_tlb_inv_context_s1, .tlb_add_flush = arm_smmu_tlb_inv_range_nosync, .tlb_sync = arm_smmu_tlb_sync_context, }; -static const struct iommu_gather_ops arm_smmu_s2_tlb_ops_v2 = { +static const struct iommu_flush_ops arm_smmu_s2_tlb_ops_v2 = { .tlb_flush_all = arm_smmu_tlb_inv_context_s2, .tlb_add_flush = arm_smmu_tlb_inv_range_nosync, .tlb_sync = arm_smmu_tlb_sync_context, }; -static const struct iommu_gather_ops arm_smmu_s2_tlb_ops_v1 = { +static const struct iommu_flush_ops arm_smmu_s2_tlb_ops_v1 = { .tlb_flush_all = arm_smmu_tlb_inv_context_s2, .tlb_add_flush = arm_smmu_tlb_inv_vmid_nosync, .tlb_sync = arm_smmu_tlb_sync_vmid,
drivers/iommu/io-pgtable-arm-v7s.c +1 -1
@@ -828,7 +828,7 @@ static void dummy_tlb_sync(void *cookie)WARN_ON (cookie != cfg_cookie); } -static const struct iommu_gather_ops dummy_tlb_ops = { +static const struct iommu_flush_ops dummy_tlb_ops = { .tlb_flush_all = dummy_tlb_flush_all, .tlb_add_flush = dummy_tlb_add_flush, .tlb_sync = dummy_tlb_sync,
drivers/iommu/io-pgtable-arm.c +1 -1
@@ -1092,7 +1092,7 @@ static void dummy_tlb_sync(void *cookie)WARN_ON (cookie != cfg_cookie); } -static const struct iommu_gather_ops dummy_tlb_ops__initconst = { +static const struct iommu_flush_ops dummy_tlb_ops__initconst = { .tlb_flush_all = dummy_tlb_flush_all, .tlb_add_flush = dummy_tlb_add_flush, .tlb_sync = dummy_tlb_sync,
drivers/iommu/ipmmu-vmsa.c +2 -2
@@ -360,7 +360,7 @@ static void ipmmu_tlb_add_flush(unsigned long iova, size_t size,/* The hardware doesn't support selective TLB flush. */ } -static const struct iommu_gather_ops ipmmu_gather_ops = { +static const struct iommu_flush_ops ipmmu_flush_ops = { .tlb_flush_all = ipmmu_tlb_flush_all, .tlb_add_flush = ipmmu_tlb_add_flush, .tlb_sync = ipmmu_tlb_flush_all,
@@ -424,7 +424,7 @@ static int ipmmu_domain_init_context(struct ipmmu_vmsa_domain *domain) domain->cfg.pgsize_bitmap =SZ_1G |SZ_2M |SZ_4K ; domain->cfg.ias =32 ; domain->cfg.oas =40 ; - domain->cfg.tlb = &ipmmu_gather_ops; + domain->cfg.tlb = &ipmmu_flush_ops; domain->io_domain.geometry.aperture_end =DMA_BIT_MASK (32 ); domain->io_domain.geometry.force_aperture = true;/*
drivers/iommu/msm_iommu.c +2 -2
@@ -191,7 +191,7 @@ static void __flush_iotlb_sync(void *cookie)*/ } -static const struct iommu_gather_ops msm_iommu_gather_ops = { +static const struct iommu_flush_ops msm_iommu_flush_ops = { .tlb_flush_all = __flush_iotlb, .tlb_add_flush = __flush_iotlb_range, .tlb_sync = __flush_iotlb_sync,
@@ -358,7 +358,7 @@ static int msm_iommu_domain_config(struct msm_priv *priv) .pgsize_bitmap = msm_iommu_ops.pgsize_bitmap, .ias = 32, .oas = 32, - .tlb = &msm_iommu_gather_ops, + .tlb = &msm_iommu_flush_ops, .iommu_dev = priv->dev, };
drivers/iommu/mtk_iommu.c +2 -2
@@ -196,7 +196,7 @@ static void mtk_iommu_tlb_sync(void *cookie) } } -static const struct iommu_gather_ops mtk_iommu_gather_ops = { +static const struct iommu_flush_ops mtk_iommu_flush_ops = { .tlb_flush_all = mtk_iommu_tlb_flush_all, .tlb_add_flush = mtk_iommu_tlb_add_flush_nosync, .tlb_sync = mtk_iommu_tlb_sync,
@@ -275,7 +275,7 @@ static int mtk_iommu_domain_finalise(struct mtk_iommu_domain *dom) .pgsize_bitmap = mtk_iommu_ops.pgsize_bitmap, .ias = 32, .oas = 32, - .tlb = &mtk_iommu_gather_ops, + .tlb = &mtk_iommu_flush_ops, .iommu_dev = data->dev, };
drivers/iommu/qcom_iommu.c +2 -2
@@ -175,7 +175,7 @@ static void qcom_iommu_tlb_inv_range_nosync(unsigned long iova, size_t size, } } -static const struct iommu_gather_ops qcom_gather_ops = { +static const struct iommu_flush_ops qcom_flush_ops = { .tlb_flush_all = qcom_iommu_tlb_inv_context, .tlb_add_flush = qcom_iommu_tlb_inv_range_nosync, .tlb_sync = qcom_iommu_tlb_sync,
@@ -226,7 +226,7 @@ static int qcom_iommu_init_domain(struct iommu_domain *domain, .pgsize_bitmap = qcom_iommu_ops.pgsize_bitmap, .ias = 32, .oas = 40, - .tlb = &qcom_gather_ops, + .tlb = &qcom_flush_ops, .iommu_dev = qcom_iommu->dev, };
include/linux/io-pgtable.h +3 -3
@@ -17,7 +17,7 @@ enum io_pgtable_fmt { };/** -* struct iommu_gather_ops - IOMMU callbacks for TLB and page table management. +* struct iommu_flush_ops - IOMMU callbacks for TLB and page table management. * * @tlb_flush_all: Synchronously invalidate the entire TLB context. * @tlb_add_flush: Queue up a TLB invalidation for a virtual address range.
@@ -28,7 +28,7 @@ enum io_pgtable_fmt {* Note that these can all be called in atomic context and must therefore * not block. */ -struct iommu_gather_ops { +struct iommu_flush_ops {void (*tlb_flush_all)(void *cookie);void (*tlb_add_flush)(unsigned long iova,size_t size,size_t granule,bool leaf,void *cookie);
@@ -84,7 +84,7 @@ struct io_pgtable_cfg {unsigned int ias;unsigned int oas;bool coherent_walk; -const struct iommu_gather_ops *tlb; +const struct iommu_flush_ops *tlb;struct device *iommu_dev;/* Low-level data specific to the table format */
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 4/19 iommu: Introduce struct iommu_iotlb_gather for batching TLB flushes
In reply to Will DeaconTo permit batching of TLB flushes across multiple calls to the IOMMU driver's ->unmap() implementation, introduce a new structure for tracking the address range to be flushed and the granularity at which the flushing is required.
This is hooked into the IOMMU API and its caller are updated to make use of the new structure. Subsequent patches will plumb this into the IOMMU drivers as well, but for now the gathering information is ignored.
--- drivers/iommu/dma-iommu.c | 9 +++++++-- drivers/iommu/iommu.c | 19 +++++++++++------- drivers/vfio/vfio_iommu_type1.c | 26 ++++++++++++++++--------- include/linux/iommu.h | 43 +++++++++++++++++++++++++++++++++++++---- 4 files changed, 75 insertions(+), 22 deletions(-)
drivers/iommu/dma-iommu.c +7 -2
@@ -455,16 +455,21 @@ static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie,static void __iommu_dma_unmap (struct iommu_domain *domain,dma_addr_t dma_addr,size_t size) { +struct iommu_iotlb_gather iotlb_gather;struct iommu_dma_cookie *cookie = domain->iova_cookie;struct iova_domain *iovad = &cookie->iovad;size_t iova_off =iova_offset (iovad, dma_addr); +size_t unmapped; dma_addr -= iova_off; size =iova_align (iovad, size + iova_off); +iommu_iotlb_gather_init (&iotlb_gather); + + unmapped =iommu_unmap_fast (domain, dma_addr, size, &iotlb_gather); +WARN_ON (unmapped != size); -WARN_ON (iommu_unmap_fast (domain, dma_addr, size) != size);if (!cookie->fq_domain) -iommu_tlb_sync (domain); +iommu_tlb_sync (domain, &iotlb_gather);iommu_dma_free_iova (cookie, dma_addr, size); }
drivers/iommu/iommu.c +12 -7
@@ -1640,7 +1640,7 @@ EXPORT_SYMBOL_GPL(iommu_map);static size_t __iommu_unmap (struct iommu_domain *domain,unsigned long iova,size_t size, -bool sync) +struct iommu_iotlb_gather *iotlb_gather) {const struct iommu_ops *ops = domain->ops;size_t unmapped_page, unmapped =0 ;
@@ -1688,9 +1688,6 @@ static size_t __iommu_unmap(struct iommu_domain *domain, unmapped += unmapped_page; } -if (sync && ops->iotlb_sync) - ops->iotlb_sync (domain); -trace_unmap (orig_iova, size, unmapped);return unmapped; }
@@ -1698,14 +1695,22 @@ static size_t __iommu_unmap(struct iommu_domain *domain,size_t iommu_unmap (struct iommu_domain *domain,unsigned long iova,size_t size) { -return __iommu_unmap (domain, iova, size, true); +struct iommu_iotlb_gather iotlb_gather; +size_t ret; + +iommu_iotlb_gather_init (&iotlb_gather); + ret =__iommu_unmap (domain, iova, size, &iotlb_gather); +iommu_tlb_sync (domain, &iotlb_gather); + +return ret; }EXPORT_SYMBOL_GPL (iommu_unmap);size_t iommu_unmap_fast (struct iommu_domain *domain, -unsigned long iova,size_t size) +unsigned long iova,size_t size, +struct iommu_iotlb_gather *iotlb_gather) { -return __iommu_unmap (domain, iova, size, false); +return __iommu_unmap (domain, iova, size, iotlb_gather); }EXPORT_SYMBOL_GPL (iommu_unmap_fast);
drivers/vfio/vfio_iommu_type1.c +17 -9
@@ -666,12 +666,13 @@ static int vfio_iommu_type1_unpin_pages(void *iommu_data, }static long vfio_sync_unpin (struct vfio_dma *dma,struct vfio_domain *domain, -struct list_head *regions) +struct list_head *regions, +struct iommu_iotlb_gather *iotlb_gather) {long unlocked =0 ;struct vfio_regions *entry, *next; -iommu_tlb_sync (domain->domain); +iommu_tlb_sync (domain->domain, iotlb_gather);list_for_each_entry_safe (entry, next, regions, list) { unlocked += vfio_unpin_pages_remote(dma,
@@ -701,13 +702,15 @@ static size_t unmap_unpin_fast(struct vfio_domain *domain, struct vfio_dma *dma, dma_addr_t *iova, size_t len, phys_addr_t phys, long *unlocked, struct list_head *unmapped_list, - int *unmapped_cnt) + int *unmapped_cnt, + struct iommu_iotlb_gather *iotlb_gather) { size_t unmapped = 0;struct vfio_regions *entry =kzalloc (sizeof (*entry),GFP_KERNEL );if (entry) { - unmapped =iommu_unmap_fast (domain->domain, *iova, len); + unmapped =iommu_unmap_fast (domain->domain, *iova, len, + iotlb_gather);if (!unmapped) {kfree (entry);
@@ -727,8 +730,8 @@ static size_t unmap_unpin_fast(struct vfio_domain *domain,* or in case of errors. */ if (*unmapped_cnt >=VFIO_IOMMU_TLB_SYNC_MAX || !unmapped) { - *unlocked +=vfio_sync_unpin (dma, domain, - unmapped_list); + *unlocked +=vfio_sync_unpin (dma, domain, unmapped_list, + iotlb_gather); *unmapped_cnt =0 ; }
@@ -759,6 +762,7 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma,dma_addr_t iova = dma->iova, end = dma->iova + dma->size;struct vfio_domain *domain, *d;LIST_HEAD (unmapped_region_list); +struct iommu_iotlb_gather iotlb_gather;int unmapped_region_cnt =0 ;long unlocked =0 ;
@@ -783,6 +787,7 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma,cond_resched (); } +iommu_iotlb_gather_init (&iotlb_gather);while (iova < end) {size_t unmapped, len;phys_addr_t phys, next;
@@ -811,7 +816,8 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma,*/ unmapped =unmap_unpin_fast (domain, dma, &iova, len, phys, &unlocked, &unmapped_region_list, - &unmapped_region_cnt); + &unmapped_region_cnt, + &iotlb_gather);if (!unmapped) { unmapped =unmap_unpin_slow (domain, dma, &iova, len, phys, &unlocked);
@@ -822,8 +828,10 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma, dma->iommu_mapped = false; -if (unmapped_region_cnt) - unlocked +=vfio_sync_unpin (dma, domain, &unmapped_region_list); +if (unmapped_region_cnt) { + unlocked +=vfio_sync_unpin (dma, domain, &unmapped_region_list, + &iotlb_gather); + }if (do_accounting) {vfio_lock_acct (dma, -unlocked, true);
include/linux/iommu.h +39 -4
94 lines not on this page · raw message
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Joerg Roedel
maintainerRe: [RFC PATCH v2 04/19] iommu: Introduce struct iommu_iotlb_gather for batching TLB flushes
Hi Will,
Quoted 11 lines · Will Deacon
static void __iommu_dma_unmap(struct
iommu_domain*domain,dma_addr_tdma_addr,drivers/iommu/dma-iommu.c +2
-0size_t size) { +struct iommu_iotlb_gather iotlb_gather;struct iommu_dma_cookie *cookie = domain->iova_cookie;struct iova_domain *iovad = &cookie->iovad;size_t iova_off =iova_offset (iovad, dma_addr); +size_t unmapped; dma_addr -= iova_off; size =iova_align (iovad, size + iova_off);
drivers/iommu/dma-iommu.c static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie, · line 473
Quoted 10 lines
drivers/iommu/dma-iommu.c +5
-2+iommu_iotlb_gather_init (&iotlb_gather); + + unmapped =iommu_unmap_fast (domain, dma_addr, size, &iotlb_gather); +WARN_ON (unmapped != size); -WARN_ON (iommu_unmap_fast (domain, dma_addr, size) != size);if (!cookie->fq_domain) -iommu_tlb_sync (domain); +iommu_tlb_sync (domain, &iotlb_gather);iommu_dma_free_iova (cookie, dma_addr, size);
I looked through your patches and was wondering if we can't make the 'struct iotlb_gather' a member of 'struct iommu_domain' and update it transparently for the user? That would make things easier for users of the iommu-api.
Regards,
Joerg
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRe: [RFC PATCH v2 04/19] iommu: Introduce struct iommu_iotlb_gather for batching TLB flushes
Hi Joerg,
Quoted 17 lines · Joerg Roedel
Quoted 16 lines · Will Deacon
static void __iommu_dma_unmap(struct
iommu_domain*domain,dma_addr_tdma_addr,drivers/iommu/dma-iommu.c +6
-0size_t size) { +struct iommu_iotlb_gather iotlb_gather;struct iommu_dma_cookie *cookie = domain->iova_cookie;struct iova_domain *iovad = &cookie->iovad;size_t iova_off =iova_offset (iovad, dma_addr); +size_t unmapped; dma_addr -= iova_off; size =iova_align (iovad, size + iova_off); +iommu_iotlb_gather_init (&iotlb_gather); + + unmapped =iommu_unmap_fast (domain, dma_addr, size, &iotlb_gather); +WARN_ON (unmapped != size);
Quoted 10 lines
drivers/iommu/dma-iommu.c static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie, · line 473
Quoted 5 lines
drivers/iommu/dma-iommu.c +1
-2-WARN_ON (iommu_unmap_fast (domain, dma_addr, size) != size);if (!cookie->fq_domain) -iommu_tlb_sync (domain); +iommu_tlb_sync (domain, &iotlb_gather);iommu_dma_free_iova (cookie, dma_addr, size);I looked through your patches and was wondering if we can't make the 'struct
iotlb_gather' a member of 'structiommu_domain' and update it transparently for the user? That would make things easier for users of the iommu-api.
Thanks for having a look.
My first (private) attempt at this series did exactly what you suggest, but the problem there is that you can have concurrent, disjoint map()/unmap() operations on the same 'struct iommu_domain', so you end up needing either multiple 'iotlb_gather' structures to support these callers or you end up massively over-invalidating. Worse, you can't just make things per-cpu without disabling preemption, so whatever you do you end up re-introducing synchronization to maintain these things. The /huge/ advantage of getting the caller to allocate the 'iotlb_gather' structure on their stack is that you don't have to worry about object lifetime at all, so all of the synchronization disappears. There is also precedent for this when unmapping things on the CPU side -- see the use of 'struct mmu_gather' and tlb_gather_mmu() in unmap_region() [mm/mmap.c], for example.
There aren't tonnes of (direct) users of the iommu-api, and the additional complexity introduced by the 'struct iotlb_gather' only applies to users of iommu_unmap_fast(), which I think is a reasonable trade-off in return for the potential for improved performance.
Will
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Joerg Roedel
maintainerRe: [RFC PATCH v2 04/19] iommu: Introduce struct iommu_iotlb_gather for batching TLB flushes
Hi Will,
Quoted 4 lines · Will Deacon
There aren't tonnes of (direct) users of the iommu-api, and the additional complexity introduced by the 'struct
iotlb_gather' only applies to users ofiommu_unmap_fast(), which I think is a reasonable trade-off in return for the potential for improved performance.
Yeah, that is right. From a scalability point of view this is much better and easier than keeping 'struct iotlb_gather' in the iommu_domain.
Thanks for the explanation.
Regards,
Joerg
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 5/19 iommu: Introduce iommu_iotlb_gather_add_page()
In reply to Will DeaconIntroduce a helper function for drivers to use when updating an iommu_iotlb_gather structure in response to an ->unmap() call, rather than having to open-code the logic in every page-table implementation.
--- include/linux/iommu.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
include/linux/iommu.h +31 -0
@@ -451,6 +451,31 @@ static inline void iommu_tlb_sync(struct iommu_domain *domain,iommu_iotlb_gather_init (iotlb_gather); } +static inline void iommu_iotlb_gather_add_page (struct iommu_domain *domain, +struct iommu_iotlb_gather *gather, +unsigned long iova,size_t size) +{ +unsigned long start = iova, end = start + size; + +/* +* If the new page is disjoint from the current range or is mapped at +* a different granularity, then sync the TLB so that the gather +* structure can be rewritten. +*/ +if (gather->pgsize != size || + end < gather->start || start > gather->end) { +if (gather->pgsize) +iommu_tlb_sync (domain, gather); + gather->pgsize = size; + } + +if (gather->end < end) + gather->end = end; + +if (gather->start > start) + gather->start = start; +} +/* PCI device grouping function */ extern struct iommu_group *pci_device_group (struct device *dev);/* Generic device grouping function */
@@ -760,6 +785,12 @@ static inline void iommu_iotlb_gather_init(struct iommu_iotlb_gather *gather) { } +static inline void iommu_iotlb_gather_add_page (struct iommu_domain *domain, +struct iommu_iotlb_gather *gather, +unsigned long iova,size_t size) +{ +} +static inline void iommu_device_unregister (struct iommu_device *iommu) { }
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 6/19 iommu: Pass struct iommu_iotlb_gather to ->unmap() and ->iotlb_sync()
In reply to Will DeaconTo allow IOMMU drivers to batch up TLB flushing operations and postpone them until ->iotlb_sync() is called, extend the prototypes for the ->unmap() and ->iotlb_sync() IOMMU ops callbacks to take a pointer to the current iommu_iotlb_gather structure.
All affected IOMMU drivers are updated, but there should be no functional change since the extra parameter is ignored for now.
--- drivers/iommu/amd_iommu.c | 11 +++++++++-- drivers/iommu/arm-smmu-v3.c | 7 ++++--- drivers/iommu/arm-smmu.c | 5 +++-- drivers/iommu/exynos-iommu.c | 3 ++- drivers/iommu/intel-iommu.c | 3 ++- drivers/iommu/iommu.c | 2 +- drivers/iommu/ipmmu-vmsa.c | 12 +++++++++--- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 13 ++++++++++--- drivers/iommu/mtk_iommu_v1.c | 3 ++- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 12 +++++++++--- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 3 ++- drivers/iommu/tegra-gart.c | 12 +++++++++--- drivers/iommu/tegra-smmu.c | 2 +- include/linux/iommu.h | 7 ++++--- 17 files changed, 70 insertions(+), 31 deletions(-)
drivers/iommu/amd_iommu.c +9 -2
@@ -3055,7 +3055,8 @@ static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, }static size_t amd_iommu_unmap (struct iommu_domain *dom,unsigned long iova, -size_t page_size) +size_t page_size, +struct iommu_iotlb_gather *gather) {struct protection_domain *domain =to_pdomain (dom);size_t unmap_size;
@@ -3196,6 +3197,12 @@ static void amd_iommu_flush_iotlb_all(struct iommu_domain *domain)domain_flush_complete (dom); } +static void amd_iommu_iotlb_sync (struct iommu_domain *domain, +struct iommu_iotlb_gather *gather) +{ +amd_iommu_flush_iotlb_all (domain); +} +const struct iommu_ops amd_iommu_ops = { .capable = amd_iommu_capable, .domain_alloc = amd_iommu_domain_alloc,
@@ -3214,7 +3221,7 @@ const struct iommu_ops amd_iommu_ops = { .is_attach_deferred = amd_iommu_is_attach_deferred, .pgsize_bitmap = AMD_IOMMU_PGSIZES, .flush_iotlb_all = amd_iommu_flush_iotlb_all, - .iotlb_sync = amd_iommu_flush_iotlb_all, + .iotlb_sync = amd_iommu_iotlb_sync, };/*****************************************************************************
drivers/iommu/arm-smmu-v3.c +4 -3
@@ -1976,8 +1976,8 @@ static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova,return ops->map (ops, iova, paddr, size, prot); } -static size_t -arm_smmu_unmap (struct iommu_domain *domain,unsigned long iova,size_t size) +static size_t arm_smmu_unmap (struct iommu_domain *domain,unsigned long iova, +size_t size,struct iommu_iotlb_gather *gather) {int ret;struct arm_smmu_domain *smmu_domain =to_smmu_domain (domain);
@@ -2001,7 +2001,8 @@ static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain)arm_smmu_tlb_inv_context (smmu_domain); } -static void arm_smmu_iotlb_sync (struct iommu_domain *domain) +static void arm_smmu_iotlb_sync (struct iommu_domain *domain, +struct iommu_iotlb_gather *gather) {struct arm_smmu_device *smmu =to_smmu_domain (domain)->smmu;
drivers/iommu/arm-smmu.c +3 -2
@@ -1304,7 +1304,7 @@ static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, }static size_t arm_smmu_unmap (struct iommu_domain *domain,unsigned long iova, -size_t size) +size_t size,struct iommu_iotlb_gather *gather) {struct io_pgtable_ops *ops =to_smmu_domain (domain)->pgtbl_ops;struct arm_smmu_device *smmu =to_smmu_domain (domain)->smmu;
@@ -1332,7 +1332,8 @@ static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain) } } -static void arm_smmu_iotlb_sync (struct iommu_domain *domain) +static void arm_smmu_iotlb_sync (struct iommu_domain *domain, +struct iommu_iotlb_gather *gather) {struct arm_smmu_domain *smmu_domain =to_smmu_domain (domain);struct arm_smmu_device *smmu = smmu_domain->smmu;
drivers/iommu/exynos-iommu.c +2 -1
@@ -1133,7 +1133,8 @@ static void exynos_iommu_tlb_invalidate_entry(struct exynos_iommu_domain *domain }static size_t exynos_iommu_unmap (struct iommu_domain *iommu_domain, -unsigned long l_iova,size_t size) +unsigned long l_iova,size_t size, +struct iommu_iotlb_gather *gather) {struct exynos_iommu_domain *domain =to_exynos_domain (iommu_domain);sysmmu_iova_t iova = (sysmmu_iova_t )l_iova;
drivers/iommu/intel-iommu.c +2 -1
@@ -5337,7 +5337,8 @@ static int intel_iommu_map(struct iommu_domain *domain, }static size_t intel_iommu_unmap (struct iommu_domain *domain, -unsigned long iova,size_t size) +unsigned long iova,size_t size, +struct iommu_iotlb_gather *gather) {struct dmar_domain *dmar_domain =to_dmar_domain (domain);struct page *freelist =NULL ;
drivers/iommu/iommu.c +1 -1
@@ -1677,7 +1677,7 @@ static size_t __iommu_unmap(struct iommu_domain *domain,while (unmapped < size) {size_t pgsize =iommu_pgsize (domain, iova, size - unmapped); - unmapped_page = ops->unmap (domain, iova, pgsize); + unmapped_page = ops->unmap (domain, iova, pgsize, iotlb_gather);if (!unmapped_page)break ;
drivers/iommu/ipmmu-vmsa.c +9 -3
@@ -719,14 +719,14 @@ static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, }static size_t ipmmu_unmap (struct iommu_domain *io_domain,unsigned long iova, -size_t size) +size_t size,struct iommu_iotlb_gather *gather) {struct ipmmu_vmsa_domain *domain =to_vmsa_domain (io_domain);return domain->iop->unmap (domain->iop, iova, size); } -static void ipmmu_iotlb_sync (struct iommu_domain *io_domain) +static void ipmmu_flush_iotlb_all (struct iommu_domain *io_domain) {struct ipmmu_vmsa_domain *domain =to_vmsa_domain (io_domain);
@@ -734,6 +734,12 @@ static void ipmmu_iotlb_sync(struct iommu_domain *io_domain)ipmmu_tlb_flush_all (domain); } +static void ipmmu_iotlb_sync (struct iommu_domain *io_domain, +struct iommu_iotlb_gather *gather) +{ +ipmmu_flush_iotlb_all (io_domain); +} +static phys_addr_t ipmmu_iova_to_phys (struct iommu_domain *io_domain,dma_addr_t iova) {
@@ -933,7 +939,7 @@ static const struct iommu_ops ipmmu_ops = { .detach_dev = ipmmu_detach_device, .map = ipmmu_map, .unmap = ipmmu_unmap, - .flush_iotlb_all = ipmmu_iotlb_sync, + .flush_iotlb_all = ipmmu_flush_iotlb_all, .iotlb_sync = ipmmu_iotlb_sync, .iova_to_phys = ipmmu_iova_to_phys, .add_device = ipmmu_add_device,
drivers/iommu/msm_iommu.c +1 -1
@@ -522,7 +522,7 @@ static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, }static size_t msm_iommu_unmap (struct iommu_domain *domain,unsigned long iova, -size_t len) +size_t len,struct iommu_iotlb_gather *gather) {struct msm_priv *priv =to_msm_priv (domain);unsigned long flags;
drivers/iommu/mtk_iommu.c +10 -3
@@ -379,7 +379,8 @@ static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, }static size_t mtk_iommu_unmap (struct iommu_domain *domain, -unsigned long iova,size_t size) +unsigned long iova,size_t size, +struct iommu_iotlb_gather *gather) {struct mtk_iommu_domain *dom =to_mtk_domain (domain);unsigned long flags;
@@ -392,7 +393,13 @@ static size_t mtk_iommu_unmap(struct iommu_domain *domain,return unmapsz; } -static void mtk_iommu_iotlb_sync (struct iommu_domain *domain) +static void mtk_iommu_flush_iotlb_all (struct iommu_domain *domain) +{ +mtk_iommu_tlb_sync (mtk_iommu_get_m4u_data ()); +} + +static void mtk_iommu_iotlb_sync (struct iommu_domain *domain, +struct iommu_iotlb_gather *gather) {mtk_iommu_tlb_sync (mtk_iommu_get_m4u_data ()); }
@@ -498,7 +505,7 @@ static const struct iommu_ops mtk_iommu_ops = { .detach_dev = mtk_iommu_detach_device, .map = mtk_iommu_map, .unmap = mtk_iommu_unmap, - .flush_iotlb_all = mtk_iommu_iotlb_sync, + .flush_iotlb_all = mtk_iommu_flush_iotlb_all, .iotlb_sync = mtk_iommu_iotlb_sync, .iova_to_phys = mtk_iommu_iova_to_phys, .add_device = mtk_iommu_add_device,
drivers/iommu/mtk_iommu_v1.c +2 -1
@@ -332,7 +332,8 @@ static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, }static size_t mtk_iommu_unmap (struct iommu_domain *domain, -unsigned long iova,size_t size) +unsigned long iova,size_t size, +struct iommu_iotlb_gather *gather) {struct mtk_iommu_domain *dom =to_mtk_domain (domain);unsigned long flags;
drivers/iommu/omap-iommu.c +1 -1
@@ -1153,7 +1153,7 @@ static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, }static size_t omap_iommu_unmap (struct iommu_domain *domain,unsigned long da, -size_t size) +size_t size,struct iommu_iotlb_gather *gather) {struct omap_iommu_domain *omap_domain =to_omap_domain (domain);struct device *dev = omap_domain->dev;
drivers/iommu/qcom_iommu.c +9 -3
@@ -428,7 +428,7 @@ static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, }static size_t qcom_iommu_unmap (struct iommu_domain *domain,unsigned long iova, -size_t size) +size_t size,struct iommu_iotlb_gather *gather) {size_t ret;unsigned long flags;
@@ -452,7 +452,7 @@ static size_t qcom_iommu_unmap(struct iommu_domain *domain, unsigned long iova,return ret; } -static void qcom_iommu_iotlb_sync (struct iommu_domain *domain) +static void qcom_iommu_flush_iotlb_all (struct iommu_domain *domain) {struct qcom_iommu_domain *qcom_domain =to_qcom_iommu_domain (domain); struct io_pgtable *pgtable = container_of(qcom_domain->pgtbl_ops,
@@ -465,6 +465,12 @@ static void qcom_iommu_iotlb_sync(struct iommu_domain *domain)pm_runtime_put_sync (qcom_domain->iommu->dev); } +static void qcom_iommu_iotlb_sync (struct iommu_domain *domain, +struct iommu_iotlb_gather *gather) +{ +qcom_iommu_flush_iotlb_all (domain); +} +static phys_addr_t qcom_iommu_iova_to_phys (struct iommu_domain *domain,dma_addr_t iova) {
@@ -592,7 +598,7 @@ static const struct iommu_ops qcom_iommu_ops = { .detach_dev = qcom_iommu_detach_dev, .map = qcom_iommu_map, .unmap = qcom_iommu_unmap, - .flush_iotlb_all = qcom_iommu_iotlb_sync, + .flush_iotlb_all = qcom_iommu_flush_iotlb_all, .iotlb_sync = qcom_iommu_iotlb_sync, .iova_to_phys = qcom_iommu_iova_to_phys, .add_device = qcom_iommu_add_device,
drivers/iommu/rockchip-iommu.c +1 -1
@@ -797,7 +797,7 @@ static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, }static size_t rk_iommu_unmap (struct iommu_domain *domain,unsigned long _iova, -size_t size) +size_t size,struct iommu_iotlb_gather *gather) {struct rk_iommu_domain *rk_domain =to_rk_domain (domain);unsigned long flags;
drivers/iommu/s390-iommu.c +2 -1
@@ -314,7 +314,8 @@ static phys_addr_t s390_iommu_iova_to_phys(struct iommu_domain *domain, }static size_t s390_iommu_unmap (struct iommu_domain *domain, -unsigned long iova,size_t size) +unsigned long iova,size_t size, +struct iommu_iotlb_gather *gather) {struct s390_domain *s390_domain =to_s390_domain (domain);int flags =ZPCI_PTE_INVALID ;
drivers/iommu/tegra-gart.c +9 -3
@@ -219,7 +219,7 @@ static inline int __gart_iommu_unmap(struct gart_device *gart, }static size_t gart_iommu_unmap (struct iommu_domain *domain,unsigned long iova, -size_t bytes) +size_t bytes,struct iommu_iotlb_gather *gather) {struct gart_device *gart = gart_handle;int err;
@@ -285,11 +285,17 @@ static int gart_iommu_of_xlate(struct device *dev,return 0 ; } -static void gart_iommu_sync (struct iommu_domain *domain) +static void gart_iommu_sync_map (struct iommu_domain *domain) {FLUSH_GART_REGS (gart_handle); } +static void gart_iommu_sync (struct iommu_domain *domain, +struct iommu_iotlb_gather *gather) +{ +gart_iommu_sync_map (domain); +} +static const struct iommu_ops gart_iommu_ops = { .capable = gart_iommu_capable, .domain_alloc = gart_iommu_domain_alloc,
@@ -304,7 +310,7 @@ static const struct iommu_ops gart_iommu_ops = { .iova_to_phys = gart_iommu_iova_to_phys, .pgsize_bitmap = GART_IOMMU_PGSIZES, .of_xlate = gart_iommu_of_xlate, - .iotlb_sync_map = gart_iommu_sync, + .iotlb_sync_map = gart_iommu_sync_map, .iotlb_sync = gart_iommu_sync, };
drivers/iommu/tegra-smmu.c +1 -1
@@ -683,7 +683,7 @@ static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, }static size_t tegra_smmu_unmap (struct iommu_domain *domain,unsigned long iova, -size_t size) +size_t size,struct iommu_iotlb_gather *gather) {struct tegra_smmu_as *as =to_smmu_as (domain);dma_addr_t pte_dma;
include/linux/iommu.h +4 -3
@@ -260,10 +260,11 @@ struct iommu_ops {int (*map)(struct iommu_domain *domain,unsigned long iova,phys_addr_t paddr,size_t size,int prot);size_t (*unmap)(struct iommu_domain *domain,unsigned long iova, -size_t size); +size_t size,struct iommu_iotlb_gather *iotlb_gather);void (*flush_iotlb_all)(struct iommu_domain *domain);void (*iotlb_sync_map)(struct iommu_domain *domain); -void (*iotlb_sync)(struct iommu_domain *domain); +void (*iotlb_sync)(struct iommu_domain *domain, +struct iommu_iotlb_gather *iotlb_gather);phys_addr_t (*iova_to_phys)(struct iommu_domain *domain,dma_addr_t iova);int (*add_device)(struct device *dev);void (*remove_device)(struct device *dev);
@@ -446,7 +447,7 @@ static inline void iommu_tlb_sync(struct iommu_domain *domain, struct iommu_iotlb_gather *iotlb_gather) {if (domain->ops->iotlb_sync) - domain->ops->iotlb_sync (domain); + domain->ops->iotlb_sync (domain, iotlb_gather);iommu_iotlb_gather_init (iotlb_gather); }
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 7/19 iommu/io-pgtable: Introduce tlb_flush_walk() and tlb_flush_leaf()
In reply to Will DeaconIn preparation for deferring TLB flushes to iommu_tlb_sync(), introduce two new synchronous invalidation helpers to the io-pgtable API, which allow the unmap() code to force invalidation in cases where it cannot be deferred (e.g. when replacing a table with a block or when TLBI_ON_MAP is set).
--- include/linux/io-pgtable.h | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-)
include/linux/io-pgtable.h +19 -5
@@ -19,17 +19,31 @@ enum io_pgtable_fmt {/** * struct iommu_flush_ops - IOMMU callbacks for TLB and page table management. * -* @tlb_flush_all: Synchronously invalidate the entire TLB context. -* @tlb_add_flush: Queue up a TLB invalidation for a virtual address range. -* @tlb_sync: Ensure any queued TLB invalidation has taken effect, and -* any corresponding page table updates are visible to the -* IOMMU. +* @tlb_flush_all: Synchronously invalidate the entire TLB context. +* @tlb_flush_walk: Synchronously invalidate all intermediate TLB state +* (sometimes referred to as the "walk cache") for a virtual +* address range. +* @tlb_flush_leaf: Synchronously invalidate all leaf TLB state for a virtual +* address range. +* @tlb_add_flush: Optional callback to queue up leaf TLB invalidation for a +* virtual address range. This function exists purely as an +* optimisation for IOMMUs that cannot batch TLB invalidation +* operations efficiently and are therefore better suited to +* issuing them early rather than deferring them until +* iommu_tlb_sync(). +* @tlb_sync: Ensure any queued TLB invalidation has taken effect, and +* any corresponding page table updates are visible to the +* IOMMU. * * Note that these can all be called in atomic context and must therefore * not block. */ struct iommu_flush_ops {void (*tlb_flush_all)(void *cookie); +void (*tlb_flush_walk)(unsigned long iova,size_t size,size_t granule, +void *cookie); +void (*tlb_flush_leaf)(unsigned long iova,size_t size,size_t granule, +void *cookie);void (*tlb_add_flush)(unsigned long iova,size_t size,size_t granule,bool leaf,void *cookie);void (*tlb_sync)(void *cookie);
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 8/19 iommu/io-pgtable: Hook up ->tlb_flush_walk() and ->tlb_flush_leaf() in drivers
In reply to Will DeaconHook up ->tlb_flush_walk() and ->tlb_flush_leaf() in drivers using the io-pgtable API so that we can start making use of them in the page-table code. For now, they can just wrap the implementations of ->tlb_add_flush and ->tlb_sync pending future optimisation in each driver.
--- drivers/gpu/drm/panfrost/panfrost_mmu.c | 14 ++++++++++++++ drivers/iommu/arm-smmu-v3.c | 22 ++++++++++++++++++++++ drivers/iommu/arm-smmu.c | 24 ++++++++++++++++++++++++ drivers/iommu/ipmmu-vmsa.c | 8 ++++++++ drivers/iommu/msm_iommu.c | 16 ++++++++++++++++ drivers/iommu/mtk_iommu.c | 16 ++++++++++++++++ drivers/iommu/qcom_iommu.c | 16 ++++++++++++++++ 7 files changed, 116 insertions(+)
drivers/gpu/drm/panfrost/panfrost_mmu.c +14 -0
@@ -249,8 +249,22 @@ static void mmu_tlb_sync_context(void *cookie)// TODO: Wait 1000 GPU cycles for HW_ISSUE_6367/T60X } +static void mmu_tlb_flush_walk (unsigned long iova,size_t size,size_t granule, +void *cookie) +{ +mmu_tlb_sync_context (cookie); +} + +static void mmu_tlb_flush_leaf (unsigned long iova,size_t size,size_t granule, +void *cookie) +{ +mmu_tlb_sync_context (cookie); +} +static const struct iommu_flush_ops mmu_tlb_ops = { .tlb_flush_all = mmu_tlb_inv_context_s1, + .tlb_flush_walk = mmu_tlb_flush_walk, + .tlb_flush_leaf = mmu_tlb_flush_leaf, .tlb_add_flush = mmu_tlb_inv_range_nosync, .tlb_sync = mmu_tlb_sync_context, };
drivers/iommu/arm-smmu-v3.c +22 -0
@@ -1597,8 +1597,30 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size, }while (size -= granule); } +static void arm_smmu_tlb_inv_walk (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +struct arm_smmu_domain *smmu_domain = cookie; +struct arm_smmu_device *smmu = smmu_domain->smmu; + +arm_smmu_tlb_inv_range_nosync (iova, size, granule, false, cookie); +arm_smmu_cmdq_issue_sync (smmu); +} + +static void arm_smmu_tlb_inv_leaf (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +struct arm_smmu_domain *smmu_domain = cookie; +struct arm_smmu_device *smmu = smmu_domain->smmu; + +arm_smmu_tlb_inv_range_nosync (iova, size, granule, true, cookie); +arm_smmu_cmdq_issue_sync (smmu); +} +static const struct iommu_flush_ops arm_smmu_flush_ops = { .tlb_flush_all = arm_smmu_tlb_inv_context, + .tlb_flush_walk = arm_smmu_tlb_inv_walk, + .tlb_flush_leaf = arm_smmu_tlb_inv_leaf, .tlb_add_flush = arm_smmu_tlb_inv_range_nosync, .tlb_sync = arm_smmu_tlb_sync, };
drivers/iommu/arm-smmu.c +24 -0
@@ -550,20 +550,44 @@ static void arm_smmu_tlb_inv_vmid_nosync(unsigned long iova, size_t size,writel_relaxed (smmu_domain->cfg.vmid, base +ARM_SMMU_GR0_TLBIVMID ); } +static void arm_smmu_tlb_inv_walk (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +struct arm_smmu_domain *smmu_domain = cookie; + + smmu_domain->tlb_ops->tlb_add_flush (iova, size, granule, false, cookie); + smmu_domain->tlb_ops->tlb_sync (cookie); +} + +static void arm_smmu_tlb_inv_leaf (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +struct arm_smmu_domain *smmu_domain = cookie; + + smmu_domain->tlb_ops->tlb_add_flush (iova, size, granule, true, cookie); + smmu_domain->tlb_ops->tlb_sync (cookie); +} +static const struct iommu_flush_ops arm_smmu_s1_tlb_ops = { .tlb_flush_all = arm_smmu_tlb_inv_context_s1, + .tlb_flush_walk = arm_smmu_tlb_inv_walk, + .tlb_flush_leaf = arm_smmu_tlb_inv_leaf, .tlb_add_flush = arm_smmu_tlb_inv_range_nosync, .tlb_sync = arm_smmu_tlb_sync_context, };static const struct iommu_flush_ops arm_smmu_s2_tlb_ops_v2 = { .tlb_flush_all = arm_smmu_tlb_inv_context_s2, + .tlb_flush_walk = arm_smmu_tlb_inv_walk, + .tlb_flush_leaf = arm_smmu_tlb_inv_leaf, .tlb_add_flush = arm_smmu_tlb_inv_range_nosync, .tlb_sync = arm_smmu_tlb_sync_context, };static const struct iommu_flush_ops arm_smmu_s2_tlb_ops_v1 = { .tlb_flush_all = arm_smmu_tlb_inv_context_s2, + .tlb_flush_walk = arm_smmu_tlb_inv_walk, + .tlb_flush_leaf = arm_smmu_tlb_inv_leaf, .tlb_add_flush = arm_smmu_tlb_inv_vmid_nosync, .tlb_sync = arm_smmu_tlb_sync_vmid, };
drivers/iommu/ipmmu-vmsa.c +8 -0
@@ -354,6 +354,12 @@ static void ipmmu_tlb_flush_all(void *cookie)ipmmu_tlb_invalidate (domain); } +static void ipmmu_tlb_flush (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +ipmmu_tlb_flush_all (cookie); +} +static void ipmmu_tlb_add_flush (unsigned long iova,size_t size,size_t granule,bool leaf,void *cookie) {
@@ -362,6 +368,8 @@ static void ipmmu_tlb_add_flush(unsigned long iova, size_t size,static const struct iommu_flush_ops ipmmu_flush_ops = { .tlb_flush_all = ipmmu_tlb_flush_all, + .tlb_flush_walk = ipmmu_tlb_flush, + .tlb_flush_leaf = ipmmu_tlb_flush, .tlb_add_flush = ipmmu_tlb_add_flush, .tlb_sync = ipmmu_tlb_flush_all, };
drivers/iommu/msm_iommu.c +16 -0
@@ -191,8 +191,24 @@ static void __flush_iotlb_sync(void *cookie)*/ } +static void __flush_iotlb_walk (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +__flush_iotlb_range (iova, size, granule, false, cookie); +__flush_iotlb_sync (cookie); +} + +static void __flush_iotlb_leaf (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +__flush_iotlb_range (iova, size, granule, true, cookie); +__flush_iotlb_sync (cookie); +} +static const struct iommu_flush_ops msm_iommu_flush_ops = { .tlb_flush_all = __flush_iotlb, + .tlb_flush_walk = __flush_iotlb_walk, + .tlb_flush_leaf = __flush_iotlb_leaf, .tlb_add_flush = __flush_iotlb_range, .tlb_sync = __flush_iotlb_sync, };
drivers/iommu/mtk_iommu.c +16 -0
@@ -196,8 +196,24 @@ static void mtk_iommu_tlb_sync(void *cookie) } } +static void mtk_iommu_tlb_flush_walk (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +mtk_iommu_tlb_add_flush_nosync (iova, size, granule, false, cookie); +mtk_iommu_tlb_sync (cookie); +} + +static void mtk_iommu_tlb_flush_leaf (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +mtk_iommu_tlb_add_flush_nosync (iova, size, granule, true, cookie); +mtk_iommu_tlb_sync (cookie); +} +static const struct iommu_flush_ops mtk_iommu_flush_ops = { .tlb_flush_all = mtk_iommu_tlb_flush_all, + .tlb_flush_walk = mtk_iommu_tlb_flush_walk, + .tlb_flush_leaf = mtk_iommu_tlb_flush_leaf, .tlb_add_flush = mtk_iommu_tlb_add_flush_nosync, .tlb_sync = mtk_iommu_tlb_sync, };
drivers/iommu/qcom_iommu.c +16 -0
@@ -175,8 +175,24 @@ static void qcom_iommu_tlb_inv_range_nosync(unsigned long iova, size_t size, } } +static void qcom_iommu_tlb_flush_walk (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +qcom_iommu_tlb_inv_range_nosync (iova, size, granule, false, cookie); +qcom_iommu_tlb_sync (cookie); +} + +static void qcom_iommu_tlb_flush_leaf (unsigned long iova,size_t size, +size_t granule,void *cookie) +{ +qcom_iommu_tlb_inv_range_nosync (iova, size, granule, true, cookie); +qcom_iommu_tlb_sync (cookie); +} +static const struct iommu_flush_ops qcom_flush_ops = { .tlb_flush_all = qcom_iommu_tlb_inv_context, + .tlb_flush_walk = qcom_iommu_tlb_flush_walk, + .tlb_flush_leaf = qcom_iommu_tlb_flush_leaf, .tlb_add_flush = qcom_iommu_tlb_inv_range_nosync, .tlb_sync = qcom_iommu_tlb_sync, };
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 9/19 iommu/io-pgtable-arm: Call ->tlb_flush_walk() and ->tlb_flush_leaf()
In reply to Will DeaconNow that all IOMMU drivers using the io-pgtable API implement the ->tlb_flush_walk() and ->tlb_flush_leaf() callbacks, we can use them in the io-pgtable code instead of ->tlb_add_flush() immediately followed by ->tlb_sync().
--- drivers/iommu/io-pgtable-arm-v7s.c | 25 +++++++++++++++---------- drivers/iommu/io-pgtable-arm.c | 17 ++++++++++++----- include/linux/io-pgtable.h | 14 ++++++++++++++ 3 files changed, 41 insertions(+), 15 deletions(-)
drivers/iommu/io-pgtable-arm-v7s.c +15 -10
@@ -504,9 +504,8 @@ static int arm_v7s_map(struct io_pgtable_ops *ops, unsigned long iova,* a chance for anything to kick off a table walk for the new iova. */ if (iop->cfg.quirks &IO_PGTABLE_QUIRK_TLBI_ON_MAP ) { -io_pgtable_tlb_add_flush (iop, iova, size, -ARM_V7S_BLOCK_SIZE (2 ), false); -io_pgtable_tlb_sync (iop); +io_pgtable_tlb_flush_walk (iop, iova, size, +ARM_V7S_BLOCK_SIZE (2 )); }else {wmb (); }
@@ -552,8 +551,7 @@ static arm_v7s_iopte arm_v7s_split_cont(struct arm_v7s_io_pgtable *data,__arm_v7s_pte_sync (ptep,ARM_V7S_CONT_PAGES , &iop->cfg); size *=ARM_V7S_CONT_PAGES ; -io_pgtable_tlb_add_flush (iop, iova, size, size, true); -io_pgtable_tlb_sync (iop); +io_pgtable_tlb_flush_leaf (iop, iova, size, size);return pte; }
@@ -648,9 +646,8 @@ static size_t __arm_v7s_unmap(struct arm_v7s_io_pgtable *data,for (i =0 ; i < num_entries; i++) {if (ARM_V7S_PTE_IS_TABLE (pte[i], lvl)) {/* Also flush any partial walks */ -io_pgtable_tlb_add_flush (iop, iova, blk_size, -ARM_V7S_BLOCK_SIZE (lvl +1 ), false); -io_pgtable_tlb_sync (iop); +io_pgtable_tlb_flush_walk (iop, iova, blk_size, +ARM_V7S_BLOCK_SIZE (lvl +1 )); ptep =iopte_deref (pte[i], lvl);__arm_v7s_free_table (ptep, lvl +1 , data); }else if (iop->cfg.quirks &IO_PGTABLE_QUIRK_NON_STRICT ) {
@@ -816,13 +813,19 @@ static void dummy_tlb_flush_all(void *cookie)WARN_ON (cookie != cfg_cookie); } -static void dummy_tlb_add_flush (unsigned long iova,size_t size, -size_t granule,bool leaf,void *cookie) +static void dummy_tlb_flush (unsigned long iova,size_t size,size_t granule, +void *cookie) {WARN_ON (cookie != cfg_cookie);WARN_ON (!(size & cfg_cookie->pgsize_bitmap)); } +static void dummy_tlb_add_flush (unsigned long iova,size_t size, +size_t granule,bool leaf,void *cookie) +{ +dummy_tlb_flush (iova, size, granule, cookie); +} +static void dummy_tlb_sync (void *cookie) {WARN_ON (cookie != cfg_cookie);
@@ -830,6 +833,8 @@ static void dummy_tlb_sync(void *cookie)static const struct iommu_flush_ops dummy_tlb_ops = { .tlb_flush_all = dummy_tlb_flush_all, + .tlb_flush_walk = dummy_tlb_flush, + .tlb_flush_leaf = dummy_tlb_flush, .tlb_add_flush = dummy_tlb_add_flush, .tlb_sync = dummy_tlb_sync, };
drivers/iommu/io-pgtable-arm.c +12 -5
@@ -622,9 +622,8 @@ static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data,if (!iopte_leaf (pte, lvl, iop->fmt)) {/* Also flush any partial walks */ -io_pgtable_tlb_add_flush (iop, iova, size, -ARM_LPAE_GRANULE (data), false); -io_pgtable_tlb_sync (iop); +io_pgtable_tlb_flush_walk (iop, iova, size, +ARM_LPAE_GRANULE (data)); ptep =iopte_deref (pte, data);__arm_lpae_free_pgtable (data, lvl +1 , ptep); }else if (iop->cfg.quirks &IO_PGTABLE_QUIRK_NON_STRICT ) {
@@ -1080,13 +1079,19 @@ static void dummy_tlb_flush_all(void *cookie)WARN_ON (cookie != cfg_cookie); } -static void dummy_tlb_add_flush (unsigned long iova,size_t size, -size_t granule,bool leaf,void *cookie) +static void dummy_tlb_flush (unsigned long iova,size_t size,size_t granule, +void *cookie) {WARN_ON (cookie != cfg_cookie);WARN_ON (!(size & cfg_cookie->pgsize_bitmap)); } +static void dummy_tlb_add_flush (unsigned long iova,size_t size, +size_t granule,bool leaf,void *cookie) +{ +dummy_tlb_flush (iova, size, granule, cookie); +} +static void dummy_tlb_sync (void *cookie) {WARN_ON (cookie != cfg_cookie);
@@ -1094,6 +1099,8 @@ static void dummy_tlb_sync(void *cookie)static const struct iommu_flush_ops dummy_tlb_ops__initconst = { .tlb_flush_all = dummy_tlb_flush_all, + .tlb_flush_walk = dummy_tlb_flush, + .tlb_flush_leaf = dummy_tlb_flush, .tlb_add_flush = dummy_tlb_add_flush, .tlb_sync = dummy_tlb_sync, };
include/linux/io-pgtable.h +14 -0
@@ -198,6 +198,20 @@ static inline void io_pgtable_tlb_flush_all(struct io_pgtable *iop) iop->cfg.tlb->tlb_flush_all (iop->cookie); } +static inline void +io_pgtable_tlb_flush_walk (struct io_pgtable *iop,unsigned long iova, +size_t size,size_t granule) +{ + iop->cfg.tlb->tlb_flush_walk (iova, size, granule, iop->cookie); +} + +static inline void +io_pgtable_tlb_flush_leaf (struct io_pgtable *iop,unsigned long iova, +size_t size,size_t granule) +{ + iop->cfg.tlb->tlb_flush_leaf (iova, size, granule, iop->cookie); +} +static inline void io_pgtable_tlb_add_flush (struct io_pgtable *iop,unsigned long iova,size_t size,size_t granule,bool leaf) {
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 10/19 iommu/io-pgtable: Replace ->tlb_add_flush() with ->tlb_add_page()
In reply to Will DeaconThe ->tlb_add_flush() callback in the io-pgtable API now looks a bit silly:
- It takes a size and a granule, which are always the same
- It takes a 'bool leaf', which is always true
- It only ever flushes a single page
With that in mind, replace it with an optional ->tlb_add_page() callback that drops the useless parameters.
--- drivers/gpu/drm/panfrost/panfrost_mmu.c | 5 -- drivers/iommu/arm-smmu-v3.c | 8 ++- drivers/iommu/arm-smmu.c | 88 +++++++++++++++++++++------------ drivers/iommu/io-pgtable-arm-v7s.c | 12 ++--- drivers/iommu/io-pgtable-arm.c | 11 ++--- drivers/iommu/ipmmu-vmsa.c | 7 --- drivers/iommu/msm_iommu.c | 7 ++- drivers/iommu/mtk_iommu.c | 8 ++- drivers/iommu/qcom_iommu.c | 8 ++- include/linux/io-pgtable.h | 22 ++++----- 10 files changed, 105 insertions(+), 71 deletions(-)
drivers/gpu/drm/panfrost/panfrost_mmu.c +0 -5
@@ -239,10 +239,6 @@ static void mmu_tlb_inv_context_s1(void *cookie)mmu_hw_do_operation (pfdev,0 ,0 , ~0UL ,AS_COMMAND_FLUSH_MEM ); } -static void mmu_tlb_inv_range_nosync (unsigned long iova,size_t size, -size_t granule,bool leaf,void *cookie) -{} -static void mmu_tlb_sync_context (void *cookie) {//struct panfrost_device *pfdev = cookie;
@@ -265,7 +261,6 @@ static const struct iommu_flush_ops mmu_tlb_ops = { .tlb_flush_all = mmu_tlb_inv_context_s1, .tlb_flush_walk = mmu_tlb_flush_walk, .tlb_flush_leaf = mmu_tlb_flush_leaf, - .tlb_add_flush = mmu_tlb_inv_range_nosync, .tlb_sync = mmu_tlb_sync_context, };
drivers/iommu/arm-smmu-v3.c +7 -1
@@ -1597,6 +1597,12 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size, }while (size -= granule); } +static void arm_smmu_tlb_inv_page_nosync (unsigned long iova,size_t granule, +void *cookie) +{ +arm_smmu_tlb_inv_range_nosync (iova, granule, granule, true, cookie); +} +static void arm_smmu_tlb_inv_walk (unsigned long iova,size_t size,size_t granule,void *cookie) {
@@ -1621,7 +1627,7 @@ static const struct iommu_flush_ops arm_smmu_flush_ops = { .tlb_flush_all = arm_smmu_tlb_inv_context, .tlb_flush_walk = arm_smmu_tlb_inv_walk, .tlb_flush_leaf = arm_smmu_tlb_inv_leaf, - .tlb_add_flush = arm_smmu_tlb_inv_range_nosync, + .tlb_add_page = arm_smmu_tlb_inv_page_nosync, .tlb_sync = arm_smmu_tlb_sync, };
drivers/iommu/arm-smmu.c +57 -31
150 lines not on this page · raw message
drivers/iommu/io-pgtable-arm-v7s.c +5 -7
@@ -595,7 +595,7 @@ static size_t arm_v7s_split_blk_unmap(struct arm_v7s_io_pgtable *data,return __arm_v7s_unmap (data, iova, size,2 , tablep); } -io_pgtable_tlb_add_flush (&data->iop, iova, size, size, true); +io_pgtable_tlb_add_page (&data->iop, iova, size);return size; }
@@ -658,8 +658,7 @@ static size_t __arm_v7s_unmap(struct arm_v7s_io_pgtable *data,*/ smp_wmb (); }else { -io_pgtable_tlb_add_flush (iop, iova, blk_size, - blk_size, true); +io_pgtable_tlb_add_page (iop, iova, blk_size); } iova += blk_size; }
@@ -820,10 +819,9 @@ static void dummy_tlb_flush(unsigned long iova, size_t size, size_t granule,WARN_ON (!(size & cfg_cookie->pgsize_bitmap)); } -static void dummy_tlb_add_flush (unsigned long iova,size_t size, -size_t granule,bool leaf,void *cookie) +static void dummy_tlb_add_page (unsigned long iova,size_t granule,void *cookie) { -dummy_tlb_flush (iova, size, granule, cookie); +dummy_tlb_flush (iova, granule, granule, cookie); }static void dummy_tlb_sync (void *cookie)
@@ -835,7 +833,7 @@ static const struct iommu_flush_ops dummy_tlb_ops = { .tlb_flush_all = dummy_tlb_flush_all, .tlb_flush_walk = dummy_tlb_flush, .tlb_flush_leaf = dummy_tlb_flush, - .tlb_add_flush = dummy_tlb_add_flush, + .tlb_add_page = dummy_tlb_add_page, .tlb_sync = dummy_tlb_sync, };
drivers/iommu/io-pgtable-arm.c +5 -6
@@ -593,7 +593,7 @@ static size_t arm_lpae_split_blk_unmap(struct arm_lpae_io_pgtable *data, tablep =iopte_deref (pte, data); }else if (unmap_idx >=0 ) { -io_pgtable_tlb_add_flush (&data->iop, iova, size, size, true); +io_pgtable_tlb_add_page (&data->iop, iova, size);return size; }
@@ -634,7 +634,7 @@ static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data,*/ smp_wmb (); }else { -io_pgtable_tlb_add_flush (iop, iova, size, size, true); +io_pgtable_tlb_add_page (iop, iova, size); }return size;
@@ -1086,10 +1086,9 @@ static void dummy_tlb_flush(unsigned long iova, size_t size, size_t granule,WARN_ON (!(size & cfg_cookie->pgsize_bitmap)); } -static void dummy_tlb_add_flush (unsigned long iova,size_t size, -size_t granule,bool leaf,void *cookie) +static void dummy_tlb_add_page (unsigned long iova,size_t granule,void *cookie) { -dummy_tlb_flush (iova, size, granule, cookie); +dummy_tlb_flush (iova, granule, granule, cookie); }static void dummy_tlb_sync (void *cookie)
@@ -1101,7 +1100,7 @@ static const struct iommu_flush_ops dummy_tlb_ops __initconst = { .tlb_flush_all = dummy_tlb_flush_all, .tlb_flush_walk = dummy_tlb_flush, .tlb_flush_leaf = dummy_tlb_flush, - .tlb_add_flush = dummy_tlb_add_flush, + .tlb_add_page = dummy_tlb_add_page, .tlb_sync = dummy_tlb_sync, };
drivers/iommu/ipmmu-vmsa.c +0 -7
@@ -360,17 +360,10 @@ static void ipmmu_tlb_flush(unsigned long iova, size_t size,ipmmu_tlb_flush_all (cookie); } -static void ipmmu_tlb_add_flush (unsigned long iova,size_t size, -size_t granule,bool leaf,void *cookie) -{ -/* The hardware doesn't support selective TLB flush. */ -} -static const struct iommu_flush_ops ipmmu_flush_ops = { .tlb_flush_all = ipmmu_tlb_flush_all, .tlb_flush_walk = ipmmu_tlb_flush, .tlb_flush_leaf = ipmmu_tlb_flush, - .tlb_add_flush = ipmmu_tlb_add_flush, .tlb_sync = ipmmu_tlb_flush_all, };
drivers/iommu/msm_iommu.c +6 -1
@@ -205,11 +205,16 @@ static void __flush_iotlb_leaf(unsigned long iova, size_t size,__flush_iotlb_sync (cookie); } +static void __flush_iotlb_page (unsigned long iova,size_t granule,void *cookie) +{ +__flush_iotlb_range (iova, granule, granule, true, cookie); +} +static const struct iommu_flush_ops msm_iommu_flush_ops = { .tlb_flush_all = __flush_iotlb, .tlb_flush_walk = __flush_iotlb_walk, .tlb_flush_leaf = __flush_iotlb_leaf, - .tlb_add_flush = __flush_iotlb_range, + .tlb_add_page = __flush_iotlb_page, .tlb_sync = __flush_iotlb_sync, };
drivers/iommu/mtk_iommu.c +7 -1
@@ -210,11 +210,17 @@ static void mtk_iommu_tlb_flush_leaf(unsigned long iova, size_t size,mtk_iommu_tlb_sync (cookie); } +static void mtk_iommu_tlb_flush_page_nosync (unsigned long iova,size_t granule, +void *cookie) +{ +mtk_iommu_tlb_add_flush_nosync (iova, granule, granule, true, cookie); +} +static const struct iommu_flush_ops mtk_iommu_flush_ops = { .tlb_flush_all = mtk_iommu_tlb_flush_all, .tlb_flush_walk = mtk_iommu_tlb_flush_walk, .tlb_flush_leaf = mtk_iommu_tlb_flush_leaf, - .tlb_add_flush = mtk_iommu_tlb_add_flush_nosync, + .tlb_add_page = mtk_iommu_tlb_flush_page_nosync, .tlb_sync = mtk_iommu_tlb_sync, };
drivers/iommu/qcom_iommu.c +7 -1
@@ -189,11 +189,17 @@ static void qcom_iommu_tlb_flush_leaf(unsigned long iova, size_t size,qcom_iommu_tlb_sync (cookie); } +static void qcom_iommu_tlb_add_page (unsigned long iova,size_t granule, +void *cookie) +{ +qcom_iommu_tlb_inv_range_nosync (iova, granule, granule, true, cookie); +} +static const struct iommu_flush_ops qcom_flush_ops = { .tlb_flush_all = qcom_iommu_tlb_inv_context, .tlb_flush_walk = qcom_iommu_tlb_flush_walk, .tlb_flush_leaf = qcom_iommu_tlb_flush_leaf, - .tlb_add_flush = qcom_iommu_tlb_inv_range_nosync, + .tlb_add_page = qcom_iommu_tlb_add_page, .tlb_sync = qcom_iommu_tlb_sync, };
include/linux/io-pgtable.h +11 -11
@@ -25,12 +25,11 @@ enum io_pgtable_fmt {* address range. * @tlb_flush_leaf: Synchronously invalidate all leaf TLB state for a virtual * address range. -* @tlb_add_flush: Optional callback to queue up leaf TLB invalidation for a -* virtual address range. This function exists purely as an -* optimisation for IOMMUs that cannot batch TLB invalidation -* operations efficiently and are therefore better suited to -* issuing them early rather than deferring them until -* iommu_tlb_sync(). +* @tlb_add_page: Optional callback to queue up leaf TLB invalidation for a +* single page. This function exists purely as an optimisation +* for IOMMUs that cannot batch TLB invalidation operations +* efficiently and are therefore better suited to issuing them +* early rather than deferring them until iommu_tlb_sync(). * @tlb_sync: Ensure any queued TLB invalidation has taken effect, and * any corresponding page table updates are visible to the * IOMMU.
@@ -44,8 +43,7 @@ struct iommu_flush_ops { void *cookie);void (*tlb_flush_leaf)(unsigned long iova,size_t size,size_t granule,void *cookie); -void (*tlb_add_flush)(unsigned long iova,size_t size,size_t granule, -bool leaf,void *cookie); +void (*tlb_add_page)(unsigned long iova,size_t granule,void *cookie);void (*tlb_sync)(void *cookie); };
@@ -212,10 +210,12 @@ io_pgtable_tlb_flush_leaf(struct io_pgtable *iop, unsigned long iova, iop->cfg.tlb->tlb_flush_leaf (iova, size, granule, iop->cookie); } -static inline void io_pgtable_tlb_add_flush (struct io_pgtable *iop, -unsigned long iova,size_t size,size_t granule,bool leaf) +static inline void +io_pgtable_tlb_add_page (struct io_pgtable *iop,unsigned long iova, +size_t granule) { - iop->cfg.tlb->tlb_add_flush (iova, size, granule, leaf, iop->cookie); +if (iop->cfg.tlb->tlb_add_page) + iop->cfg.tlb->tlb_add_page (iova, granule, iop->cookie); }static inline void io_pgtable_tlb_sync (struct io_pgtable *iop)
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 11/19 iommu/io-pgtable: Remove unused ->tlb_sync() callback
In reply to Will DeaconThe ->tlb_sync() callback is no longer used, so it can be removed.
--- drivers/gpu/drm/panfrost/panfrost_mmu.c | 1 - drivers/iommu/arm-smmu-v3.c | 8 -------- drivers/iommu/arm-smmu.c | 17 +++++++++-------- drivers/iommu/io-pgtable-arm-v7s.c | 6 ------ drivers/iommu/io-pgtable-arm.c | 6 ------ drivers/iommu/ipmmu-vmsa.c | 1 - drivers/iommu/msm_iommu.c | 20 +++++++------------- drivers/iommu/mtk_iommu.c | 1 - drivers/iommu/qcom_iommu.c | 1 - include/linux/io-pgtable.h | 9 --------- 10 files changed, 16 insertions(+), 54 deletions(-)
drivers/gpu/drm/panfrost/panfrost_mmu.c +0 -1
@@ -261,7 +261,6 @@ static const struct iommu_flush_ops mmu_tlb_ops = { .tlb_flush_all = mmu_tlb_inv_context_s1, .tlb_flush_walk = mmu_tlb_flush_walk, .tlb_flush_leaf = mmu_tlb_flush_leaf, - .tlb_sync = mmu_tlb_sync_context, }; static const char *access_type_name(struct panfrost_device *pfdev,
drivers/iommu/arm-smmu-v3.c +0 -8
@@ -1539,13 +1539,6 @@ static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, }/* IO_PGTABLE API */ -static void arm_smmu_tlb_sync (void *cookie) -{ -struct arm_smmu_domain *smmu_domain = cookie; - -arm_smmu_cmdq_issue_sync (smmu_domain->smmu); -} -static void arm_smmu_tlb_inv_context (void *cookie) {struct arm_smmu_domain *smmu_domain = cookie;
@@ -1628,7 +1621,6 @@ static const struct iommu_flush_ops arm_smmu_flush_ops = { .tlb_flush_walk = arm_smmu_tlb_inv_walk, .tlb_flush_leaf = arm_smmu_tlb_inv_leaf, .tlb_add_page = arm_smmu_tlb_inv_page_nosync, - .tlb_sync = arm_smmu_tlb_sync, };/* IOMMU API */
drivers/iommu/arm-smmu.c +9 -8
@@ -254,7 +254,8 @@ enum arm_smmu_domain_stage {struct arm_smmu_flush_ops {struct iommu_flush_ops tlb;void (*tlb_inv_range)(unsigned long iova,size_t size,size_t granule, -bool leaf,void *cookie) +bool leaf,void *cookie); +void (*tlb_sync)(void *cookie); };struct arm_smmu_domain {
@@ -542,7 +543,7 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size,* On MMU-401 at least, the cost of firing off multiple TLBIVMIDs appears * almost negligible, but the benefit of getting the first one in as far ahead * of the sync as possible is significant, hence we don't just make this a -* no-op and set .tlb_sync to arm_smmu_inv_context_s2() as you might think. +* no-op and set .tlb_sync to arm_smmu_tlb_inv_context_s2() as you might think. */ static void arm_smmu_tlb_inv_vmid_nosync (unsigned long iova,size_t size,size_t granule,bool leaf,void *cookie)
@@ -563,7 +564,7 @@ static void arm_smmu_tlb_inv_walk(unsigned long iova, size_t size,const struct arm_smmu_flush_ops *ops = smmu_domain->flush_ops; ops->tlb_inv_range (iova, size, granule, false, cookie); - ops->tlb.tlb_sync (cookie); + ops->tlb_sync (cookie); } static void arm_smmu_tlb_inv_leaf(unsigned long iova, size_t size,
@@ -573,7 +574,7 @@ static void arm_smmu_tlb_inv_leaf(unsigned long iova, size_t size,const struct arm_smmu_flush_ops *ops = smmu_domain->flush_ops; ops->tlb_inv_range (iova, size, granule, true, cookie); - ops->tlb.tlb_sync (cookie); + ops->tlb_sync (cookie); } static void arm_smmu_tlb_add_page(unsigned long iova, size_t granule,
@@ -591,9 +592,9 @@ static const struct arm_smmu_flush_ops arm_smmu_s1_tlb_ops = { .tlb_flush_walk = arm_smmu_tlb_inv_walk, .tlb_flush_leaf = arm_smmu_tlb_inv_leaf, .tlb_add_page = arm_smmu_tlb_add_page, - .tlb_sync = arm_smmu_tlb_sync_context, }, .tlb_inv_range = arm_smmu_tlb_inv_range_nosync, + .tlb_sync = arm_smmu_tlb_sync_context, }; static const struct arm_smmu_flush_ops arm_smmu_s2_tlb_ops_v2 = {
@@ -602,9 +603,9 @@ static const struct arm_smmu_flush_ops arm_smmu_s2_tlb_ops_v2 = { .tlb_flush_walk = arm_smmu_tlb_inv_walk, .tlb_flush_leaf = arm_smmu_tlb_inv_leaf, .tlb_add_page = arm_smmu_tlb_add_page, - .tlb_sync = arm_smmu_tlb_sync_context, }, .tlb_inv_range = arm_smmu_tlb_inv_range_nosync, + .tlb_sync = arm_smmu_tlb_sync_context, }; static const struct arm_smmu_flush_ops arm_smmu_s2_tlb_ops_v1 = {
@@ -613,9 +614,9 @@ static const struct arm_smmu_flush_ops arm_smmu_s2_tlb_ops_v1 = { .tlb_flush_walk = arm_smmu_tlb_inv_walk, .tlb_flush_leaf = arm_smmu_tlb_inv_leaf, .tlb_add_page = arm_smmu_tlb_add_page, - .tlb_sync = arm_smmu_tlb_sync_vmid, }, .tlb_inv_range = arm_smmu_tlb_inv_vmid_nosync, + .tlb_sync = arm_smmu_tlb_sync_vmid, };static irqreturn_t arm_smmu_context_fault (int irq,void *dev)
@@ -1390,7 +1391,7 @@ static void arm_smmu_iotlb_sync(struct iommu_domain *domain,if (smmu_domain->flush_ops) {arm_smmu_rpm_get (smmu); - smmu_domain->flush_ops->tlb.tlb_sync (smmu_domain); + smmu_domain->flush_ops->tlb_sync (smmu_domain);arm_smmu_rpm_put (smmu); } }
drivers/iommu/io-pgtable-arm-v7s.c +0 -6
@@ -824,17 +824,11 @@ static void dummy_tlb_add_page(unsigned long iova, size_t granule, void *cookie)dummy_tlb_flush (iova, granule, granule, cookie); } -static void dummy_tlb_sync (void *cookie) -{ -WARN_ON (cookie != cfg_cookie); -} -static const struct iommu_flush_ops dummy_tlb_ops = { .tlb_flush_all = dummy_tlb_flush_all, .tlb_flush_walk = dummy_tlb_flush, .tlb_flush_leaf = dummy_tlb_flush, .tlb_add_page = dummy_tlb_add_page, - .tlb_sync = dummy_tlb_sync, };#define __FAIL (ops) ({ \
drivers/iommu/io-pgtable-arm.c +0 -6
@@ -1091,17 +1091,11 @@ static void dummy_tlb_add_page(unsigned long iova, size_t granule, void *cookie)dummy_tlb_flush (iova, granule, granule, cookie); } -static void dummy_tlb_sync (void *cookie) -{ -WARN_ON (cookie != cfg_cookie); -} -static const struct iommu_flush_ops dummy_tlb_ops__initconst = { .tlb_flush_all = dummy_tlb_flush_all, .tlb_flush_walk = dummy_tlb_flush, .tlb_flush_leaf = dummy_tlb_flush, .tlb_add_page = dummy_tlb_add_page, - .tlb_sync = dummy_tlb_sync, };static void __init arm_lpae_dump_ops (struct io_pgtable_ops *ops)
drivers/iommu/ipmmu-vmsa.c +0 -1
@@ -364,7 +364,6 @@ static const struct iommu_flush_ops ipmmu_flush_ops = { .tlb_flush_all = ipmmu_tlb_flush_all, .tlb_flush_walk = ipmmu_tlb_flush, .tlb_flush_leaf = ipmmu_tlb_flush, - .tlb_sync = ipmmu_tlb_flush_all, };/* -----------------------------------------------------------------------------
drivers/iommu/msm_iommu.c +7 -13
@@ -181,28 +181,16 @@ static void __flush_iotlb_range(unsigned long iova, size_t size,return ; } -static void __flush_iotlb_sync (void *cookie) -{ -/* -* Nothing is needed here, the barrier to guarantee -* completion of the tlb sync operation is implicitly -* taken care when the iommu client does a writel before -* kick starting the other master. -*/ -} -static void __flush_iotlb_walk (unsigned long iova,size_t size,size_t granule,void *cookie) {__flush_iotlb_range (iova, size, granule, false, cookie); -__flush_iotlb_sync (cookie); }static void __flush_iotlb_leaf (unsigned long iova,size_t size,size_t granule,void *cookie) {__flush_iotlb_range (iova, size, granule, true, cookie); -__flush_iotlb_sync (cookie); }static void __flush_iotlb_page (unsigned long iova,size_t granule,void *cookie)
@@ -215,7 +203,6 @@ static const struct iommu_flush_ops msm_iommu_flush_ops = { .tlb_flush_walk = __flush_iotlb_walk, .tlb_flush_leaf = __flush_iotlb_leaf, .tlb_add_page = __flush_iotlb_page, - .tlb_sync = __flush_iotlb_sync, };static int msm_iommu_alloc_ctx (unsigned long *map,int start,int end)
@@ -725,6 +712,13 @@ static struct iommu_ops msm_iommu_ops = { .detach_dev = msm_iommu_detach_dev, .map = msm_iommu_map, .unmap = msm_iommu_unmap, +/* +* Nothing is needed here, the barrier to guarantee +* completion of the tlb sync operation is implicitly +* taken care when the iommu client does a writel before +* kick starting the other master. +*/ + .iotlb_sync = NULL, .iova_to_phys = msm_iommu_iova_to_phys, .add_device = msm_iommu_add_device, .remove_device = msm_iommu_remove_device,
drivers/iommu/mtk_iommu.c +0 -1
@@ -221,7 +221,6 @@ static const struct iommu_flush_ops mtk_iommu_flush_ops = { .tlb_flush_walk = mtk_iommu_tlb_flush_walk, .tlb_flush_leaf = mtk_iommu_tlb_flush_leaf, .tlb_add_page = mtk_iommu_tlb_flush_page_nosync, - .tlb_sync = mtk_iommu_tlb_sync, };static irqreturn_t mtk_iommu_isr (int irq,void *dev_id)
drivers/iommu/qcom_iommu.c +0 -1
@@ -200,7 +200,6 @@ static const struct iommu_flush_ops qcom_flush_ops = { .tlb_flush_walk = qcom_iommu_tlb_flush_walk, .tlb_flush_leaf = qcom_iommu_tlb_flush_leaf, .tlb_add_page = qcom_iommu_tlb_add_page, - .tlb_sync = qcom_iommu_tlb_sync, };static irqreturn_t qcom_iommu_fault (int irq,void *dev)
include/linux/io-pgtable.h +0 -9
@@ -30,9 +30,6 @@ enum io_pgtable_fmt {* for IOMMUs that cannot batch TLB invalidation operations * efficiently and are therefore better suited to issuing them * early rather than deferring them until iommu_tlb_sync(). -* @tlb_sync: Ensure any queued TLB invalidation has taken effect, and -* any corresponding page table updates are visible to the -* IOMMU. * * Note that these can all be called in atomic context and must therefore * not block.
@@ -44,7 +41,6 @@ struct iommu_flush_ops {void (*tlb_flush_leaf)(unsigned long iova,size_t size,size_t granule,void *cookie);void (*tlb_add_page)(unsigned long iova,size_t granule,void *cookie); -void (*tlb_sync)(void *cookie); };/**
@@ -218,11 +214,6 @@ io_pgtable_tlb_add_page(struct io_pgtable *iop, unsigned long iova, iop->cfg.tlb->tlb_add_page (iova, granule, iop->cookie); } -static inline void io_pgtable_tlb_sync (struct io_pgtable *iop) -{ - iop->cfg.tlb->tlb_sync (iop->cookie); -} -/** * struct io_pgtable_init_fns - Alloc/free a set of page tables for a * particular format.
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 12/19 iommu/io-pgtable: Pass struct iommu_iotlb_gather to ->unmap()
In reply to Will DeaconUpdate the io-pgtable ->unmap() function to take an iommu_iotlb_gather pointer as an argument, and update the callers as appropriate.
--- drivers/gpu/drm/panfrost/panfrost_mmu.c | 2 +- drivers/iommu/arm-smmu-v3.c | 2 +- drivers/iommu/arm-smmu.c | 2 +- drivers/iommu/io-pgtable-arm-v7s.c | 6 +++--- drivers/iommu/io-pgtable-arm.c | 7 +++---- drivers/iommu/ipmmu-vmsa.c | 2 +- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 2 +- include/linux/io-pgtable.h | 4 +++- 10 files changed, 16 insertions(+), 15 deletions(-)
drivers/gpu/drm/panfrost/panfrost_mmu.c +1 -1
@@ -215,7 +215,7 @@ void panfrost_mmu_unmap(struct panfrost_gem_object *bo)size_t unmapped_page;size_t pgsize =get_pgsize (iova, len - unmapped_len); - unmapped_page = ops->unmap (ops, iova, pgsize); + unmapped_page = ops->unmap (ops, iova, pgsize,NULL );if (!unmapped_page)break ;
drivers/iommu/arm-smmu-v3.c +1 -1
@@ -2006,7 +2006,7 @@ static size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova,if (!ops)return 0 ; - ret = ops->unmap (ops, iova, size); + ret = ops->unmap (ops, iova, size, gather);if (ret &&arm_smmu_atc_inv_domain (smmu_domain,0 , iova, size))return 0 ;
drivers/iommu/arm-smmu.c +1 -1
@@ -1365,7 +1365,7 @@ static size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova,return 0 ;arm_smmu_rpm_get (smmu); - ret = ops->unmap (ops, iova, size); + ret = ops->unmap (ops, iova, size, gather);arm_smmu_rpm_put (smmu);return ret;
drivers/iommu/io-pgtable-arm-v7s.c +3 -3
@@ -677,7 +677,7 @@ static size_t __arm_v7s_unmap(struct arm_v7s_io_pgtable *data, }static size_t arm_v7s_unmap (struct io_pgtable_ops *ops,unsigned long iova, -size_t size) +size_t size,struct iommu_iotlb_gather *gather) {struct arm_v7s_io_pgtable *data =io_pgtable_ops_to_data (ops);
@@ -903,7 +903,7 @@ static int __init arm_v7s_do_selftests(void) size =1UL <<__ffs (cfg.pgsize_bitmap);while (i < loopnr) { iova_start = i *SZ_16M ; -if (ops->unmap (ops, iova_start + size, size) != size) +if (ops->unmap (ops, iova_start + size, size,NULL ) != size)return __FAIL (ops);/* Remap of partial unmap */
@@ -921,7 +921,7 @@ static int __init arm_v7s_do_selftests(void)for_each_set_bit (i, &cfg.pgsize_bitmap,BITS_PER_LONG ) { size =1UL << i; -if (ops->unmap (ops, iova, size) != size) +if (ops->unmap (ops, iova, size,NULL ) != size)return __FAIL (ops);if (ops->iova_to_phys (ops, iova +42 ))
drivers/iommu/io-pgtable-arm.c +3 -4
@@ -23,7 +23,6 @@#include <linux/atomic.h> #include <linux/bitops.h> #include <linux/io-pgtable.h> -#include <linux/iommu.h> #include <linux/kernel.h> #include <linux/sizes.h> #include <linux/slab.h>
@@ -653,7 +652,7 @@ static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data, }static size_t arm_lpae_unmap (struct io_pgtable_ops *ops,unsigned long iova, -size_t size) +size_t size,struct iommu_iotlb_gather *gather) {struct arm_lpae_io_pgtable *data =io_pgtable_ops_to_data (ops);arm_lpae_iopte *ptep = data->pgd;
@@ -1178,7 +1177,7 @@ static int __init arm_lpae_run_tests(struct io_pgtable_cfg *cfg)/* Partial unmap */ size =1UL <<__ffs (cfg->pgsize_bitmap); -if (ops->unmap (ops,SZ_1G + size, size) != size) +if (ops->unmap (ops,SZ_1G + size, size,NULL ) != size)return __FAIL (ops, i);/* Remap of partial unmap */
@@ -1193,7 +1192,7 @@ static int __init arm_lpae_run_tests(struct io_pgtable_cfg *cfg)for_each_set_bit (j, &cfg->pgsize_bitmap,BITS_PER_LONG ) { size =1UL << j; -if (ops->unmap (ops, iova, size) != size) +if (ops->unmap (ops, iova, size,NULL ) != size)return __FAIL (ops, i);if (ops->iova_to_phys (ops, iova +42 ))
drivers/iommu/ipmmu-vmsa.c +1 -1
@@ -723,7 +723,7 @@ static size_t ipmmu_unmap(struct iommu_domain *io_domain, unsigned long iova, {struct ipmmu_vmsa_domain *domain =to_vmsa_domain (io_domain); -return domain->iop->unmap (domain->iop, iova, size); +return domain->iop->unmap (domain->iop, iova, size, gather); }static void ipmmu_flush_iotlb_all (struct iommu_domain *io_domain)
drivers/iommu/msm_iommu.c +1 -1
@@ -536,7 +536,7 @@ static size_t msm_iommu_unmap(struct iommu_domain *domain, unsigned long iova,unsigned long flags;spin_lock_irqsave (&priv->pgtlock, flags); - len = priv->iop->unmap (priv->iop, iova, len); + len = priv->iop->unmap (priv->iop, iova, len, gather);spin_unlock_irqrestore (&priv->pgtlock, flags);return len;
drivers/iommu/mtk_iommu.c +1 -1
@@ -408,7 +408,7 @@ static size_t mtk_iommu_unmap(struct iommu_domain *domain,size_t unmapsz;spin_lock_irqsave (&dom->pgtlock, flags); - unmapsz = dom->iop->unmap (dom->iop, iova, size); + unmapsz = dom->iop->unmap (dom->iop, iova, size, gather);spin_unlock_irqrestore (&dom->pgtlock, flags);return unmapsz;
drivers/iommu/qcom_iommu.c +1 -1
@@ -466,7 +466,7 @@ static size_t qcom_iommu_unmap(struct iommu_domain *domain, unsigned long iova,*/ pm_runtime_get_sync (qcom_domain->iommu->dev);spin_lock_irqsave (&qcom_domain->pgtbl_lock, flags); - ret = ops->unmap (ops, iova, size); + ret = ops->unmap (ops, iova, size, gather);spin_unlock_irqrestore (&qcom_domain->pgtbl_lock, flags);pm_runtime_put_sync (qcom_domain->iommu->dev);
include/linux/io-pgtable.h +3 -1
@@ -1,7 +1,9 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef __IO_PGTABLE_H#define __IO_PGTABLE_H +#include <linux/bitops.h> +#include <linux/iommu.h> /* * Public API for use by IOMMU drivers
@@ -136,7 +138,7 @@ struct io_pgtable_ops {int (*map)(struct io_pgtable_ops *ops,unsigned long iova,phys_addr_t paddr,size_t size,int prot);size_t (*unmap)(struct io_pgtable_ops *ops,unsigned long iova, -size_t size); +size_t size,struct iommu_iotlb_gather *gather);phys_addr_t (*iova_to_phys)(struct io_pgtable_ops *ops,unsigned long iova); };
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 13/19 iommu/io-pgtable: Pass struct iommu_iotlb_gather to ->tlb_add_page()
In reply to Will DeaconWith all the pieces in place, we can finally propagate the iommu_iotlb_gather structure from the call to unmap() down to the IOMMU drivers' implementation of ->tlb_add_page(). Currently everybody ignores it, but the machinery is now there to defer invalidation.
--- drivers/iommu/arm-smmu-v3.c | 3 ++- drivers/iommu/arm-smmu.c | 3 ++- drivers/iommu/io-pgtable-arm-v7s.c | 23 ++++++++++++++--------- drivers/iommu/io-pgtable-arm.c | 22 ++++++++++++++-------- drivers/iommu/msm_iommu.c | 3 ++- drivers/iommu/mtk_iommu.c | 3 ++- drivers/iommu/qcom_iommu.c | 3 ++- include/linux/io-pgtable.h | 16 +++++++++------- 8 files changed, 47 insertions(+), 29 deletions(-)
drivers/iommu/arm-smmu-v3.c +2 -1
@@ -1590,7 +1590,8 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size, }while (size -= granule); } -static void arm_smmu_tlb_inv_page_nosync (unsigned long iova,size_t granule, +static void arm_smmu_tlb_inv_page_nosync (struct iommu_iotlb_gather *gather, +unsigned long iova,size_t granule,void *cookie) {arm_smmu_tlb_inv_range_nosync (iova, granule, granule, true, cookie);
drivers/iommu/arm-smmu.c +2 -1
@@ -577,7 +577,8 @@ static void arm_smmu_tlb_inv_leaf(unsigned long iova, size_t size, ops->tlb_sync (cookie); } -static void arm_smmu_tlb_add_page (unsigned long iova,size_t granule, +static void arm_smmu_tlb_add_page (struct iommu_iotlb_gather *gather, +unsigned long iova,size_t granule,void *cookie) {struct arm_smmu_domain *smmu_domain = cookie;
drivers/iommu/io-pgtable-arm-v7s.c +14 -9
@@ -373,7 +373,8 @@ static bool arm_v7s_pte_is_cont(arm_v7s_iopte pte, int lvl)return false; } -static size_t __arm_v7s_unmap (struct arm_v7s_io_pgtable *,unsigned long , +static size_t __arm_v7s_unmap (struct arm_v7s_io_pgtable *, +struct iommu_iotlb_gather *,unsigned long ,size_t ,int ,arm_v7s_iopte *); static int arm_v7s_init_pte(struct arm_v7s_io_pgtable *data,
@@ -394,7 +395,7 @@ static int arm_v7s_init_pte(struct arm_v7s_io_pgtable *data,size_t sz =ARM_V7S_BLOCK_SIZE (lvl); tblp = ptep -ARM_V7S_LVL_IDX (iova, lvl); -if (WARN_ON (__arm_v7s_unmap (data, iova + i * sz, +if (WARN_ON (__arm_v7s_unmap (data,NULL , iova + i * sz, sz, lvl, tblp) != sz))return -EINVAL ; }else if (ptep[i]) {
@@ -556,6 +557,7 @@ static arm_v7s_iopte arm_v7s_split_cont(struct arm_v7s_io_pgtable *data, }static size_t arm_v7s_split_blk_unmap (struct arm_v7s_io_pgtable *data, +struct iommu_iotlb_gather *gather,unsigned long iova,size_t size,arm_v7s_iopte blk_pte,arm_v7s_iopte *ptep)
@@ -592,14 +594,15 @@ static size_t arm_v7s_split_blk_unmap(struct arm_v7s_io_pgtable *data,return 0 ; tablep =iopte_deref (pte,1 ); -return __arm_v7s_unmap (data, iova, size,2 , tablep); +return __arm_v7s_unmap (data, gather, iova, size,2 , tablep); } -io_pgtable_tlb_add_page (&data->iop, iova, size); +io_pgtable_tlb_add_page (&data->iop, gather, iova, size);return size; }static size_t __arm_v7s_unmap (struct arm_v7s_io_pgtable *data, +struct iommu_iotlb_gather *gather,unsigned long iova,size_t size,int lvl,arm_v7s_iopte *ptep) {
@@ -658,7 +661,7 @@ static size_t __arm_v7s_unmap(struct arm_v7s_io_pgtable *data,*/ smp_wmb (); }else { -io_pgtable_tlb_add_page (iop, iova, blk_size); +io_pgtable_tlb_add_page (iop, gather, iova, blk_size); } iova += blk_size; }
@@ -668,12 +671,13 @@ static size_t __arm_v7s_unmap(struct arm_v7s_io_pgtable *data,* Insert a table at the next level to map the old region, * minus the part we want to unmap */ -return arm_v7s_split_blk_unmap (data, iova, size, pte[0 ], ptep); +return arm_v7s_split_blk_unmap (data, gather, iova, size, pte[0 ], + ptep); }/* Keep on walkin' */ ptep =iopte_deref (pte[0 ], lvl); -return __arm_v7s_unmap (data, iova, size, lvl +1 , ptep); +return __arm_v7s_unmap (data, gather, iova, size, lvl +1 , ptep); } static size_t arm_v7s_unmap(struct io_pgtable_ops *ops, unsigned long iova,
@@ -684,7 +688,7 @@ static size_t arm_v7s_unmap(struct io_pgtable_ops *ops, unsigned long iova,if (WARN_ON (upper_32_bits (iova)))return 0 ; -return __arm_v7s_unmap (data, iova, size,1 , data->pgd); +return __arm_v7s_unmap (data, gather, iova, size,1 , data->pgd); } static phys_addr_t arm_v7s_iova_to_phys(struct io_pgtable_ops *ops,
@@ -819,7 +823,8 @@ static void dummy_tlb_flush(unsigned long iova, size_t size, size_t granule,WARN_ON (!(size & cfg_cookie->pgsize_bitmap)); } -static void dummy_tlb_add_page (unsigned long iova,size_t granule,void *cookie) +static void dummy_tlb_add_page (struct iommu_iotlb_gather *gather, +unsigned long iova,size_t granule,void *cookie) {dummy_tlb_flush (iova, granule, granule, cookie); }
drivers/iommu/io-pgtable-arm.c +14 -8
@@ -300,6 +300,7 @@ static void __arm_lpae_set_pte(arm_lpae_iopte *ptep, arm_lpae_iopte pte, }static size_t __arm_lpae_unmap (struct arm_lpae_io_pgtable *data, +struct iommu_iotlb_gather *gather,unsigned long iova,size_t size,int lvl,arm_lpae_iopte *ptep);
@@ -345,8 +346,10 @@ static int arm_lpae_init_pte(struct arm_lpae_io_pgtable *data,size_t sz =ARM_LPAE_BLOCK_SIZE (lvl, data); tblp = ptep -ARM_LPAE_LVL_IDX (iova, lvl, data); -if (WARN_ON (__arm_lpae_unmap (data, iova, sz, lvl, tblp) != sz)) +if (__arm_lpae_unmap (data,NULL , iova, sz, lvl, tblp) != sz) { +WARN_ON (1 );return -EINVAL ; + } }__arm_lpae_init_pte (data, paddr, prot, lvl, ptep);
@@ -547,6 +550,7 @@ static void arm_lpae_free_pgtable(struct io_pgtable *iop) }static size_t arm_lpae_split_blk_unmap (struct arm_lpae_io_pgtable *data, +struct iommu_iotlb_gather *gather,unsigned long iova,size_t size,arm_lpae_iopte blk_pte,int lvl,arm_lpae_iopte *ptep)
@@ -592,14 +596,15 @@ static size_t arm_lpae_split_blk_unmap(struct arm_lpae_io_pgtable *data, tablep =iopte_deref (pte, data); }else if (unmap_idx >=0 ) { -io_pgtable_tlb_add_page (&data->iop, iova, size); +io_pgtable_tlb_add_page (&data->iop, gather, iova, size);return size; } -return __arm_lpae_unmap (data, iova, size, lvl, tablep); +return __arm_lpae_unmap (data, gather, iova, size, lvl, tablep); }static size_t __arm_lpae_unmap (struct arm_lpae_io_pgtable *data, +struct iommu_iotlb_gather *gather,unsigned long iova,size_t size,int lvl,arm_lpae_iopte *ptep) {
@@ -633,7 +638,7 @@ static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data,*/ smp_wmb (); }else { -io_pgtable_tlb_add_page (iop, iova, size); +io_pgtable_tlb_add_page (iop, gather, iova, size); }return size;
@@ -642,13 +647,13 @@ static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data,* Insert a table at the next level to map the old region, * minus the part we want to unmap */ -return arm_lpae_split_blk_unmap (data, iova, size, pte, +return arm_lpae_split_blk_unmap (data, gather, iova, size, pte, lvl +1 , ptep); }/* Keep on walkin' */ ptep =iopte_deref (pte, data); -return __arm_lpae_unmap (data, iova, size, lvl +1 , ptep); +return __arm_lpae_unmap (data, gather, iova, size, lvl +1 , ptep); } static size_t arm_lpae_unmap(struct io_pgtable_ops *ops, unsigned long iova,
@@ -661,7 +666,7 @@ static size_t arm_lpae_unmap(struct io_pgtable_ops *ops, unsigned long iova,if (WARN_ON (iova >= (1ULL << data->iop.cfg.ias)))return 0 ; -return __arm_lpae_unmap (data, iova, size, lvl, ptep); +return __arm_lpae_unmap (data, gather, iova, size, lvl, ptep); } static phys_addr_t arm_lpae_iova_to_phys(struct io_pgtable_ops *ops,
@@ -1085,7 +1090,8 @@ static void dummy_tlb_flush(unsigned long iova, size_t size, size_t granule,WARN_ON (!(size & cfg_cookie->pgsize_bitmap)); } -static void dummy_tlb_add_page (unsigned long iova,size_t granule,void *cookie) +static void dummy_tlb_add_page (struct iommu_iotlb_gather *gather, +unsigned long iova,size_t granule,void *cookie) {dummy_tlb_flush (iova, granule, granule, cookie); }
drivers/iommu/msm_iommu.c +2 -1
@@ -193,7 +193,8 @@ static void __flush_iotlb_leaf(unsigned long iova, size_t size,__flush_iotlb_range (iova, size, granule, true, cookie); } -static void __flush_iotlb_page (unsigned long iova,size_t granule,void *cookie) +static void __flush_iotlb_page (struct iommu_iotlb_gather *gather, +unsigned long iova,size_t granule,void *cookie) {__flush_iotlb_range (iova, granule, granule, true, cookie); }
drivers/iommu/mtk_iommu.c +2 -1
@@ -210,7 +210,8 @@ static void mtk_iommu_tlb_flush_leaf(unsigned long iova, size_t size,mtk_iommu_tlb_sync (cookie); } -static void mtk_iommu_tlb_flush_page_nosync (unsigned long iova,size_t granule, +static void mtk_iommu_tlb_flush_page_nosync (struct iommu_iotlb_gather *gather, +unsigned long iova,size_t granule,void *cookie) {mtk_iommu_tlb_add_flush_nosync (iova, granule, granule, true, cookie);
drivers/iommu/qcom_iommu.c +2 -1
@@ -189,7 +189,8 @@ static void qcom_iommu_tlb_flush_leaf(unsigned long iova, size_t size,qcom_iommu_tlb_sync (cookie); } -static void qcom_iommu_tlb_add_page (unsigned long iova,size_t granule, +static void qcom_iommu_tlb_add_page (struct iommu_iotlb_gather *gather, +unsigned long iova,size_t granule,void *cookie) {qcom_iommu_tlb_inv_range_nosync (iova, granule, granule, true, cookie);
include/linux/io-pgtable.h +9 -7
@@ -28,10 +28,10 @@ enum io_pgtable_fmt {* @tlb_flush_leaf: Synchronously invalidate all leaf TLB state for a virtual * address range. * @tlb_add_page: Optional callback to queue up leaf TLB invalidation for a -* single page. This function exists purely as an optimisation -* for IOMMUs that cannot batch TLB invalidation operations -* efficiently and are therefore better suited to issuing them -* early rather than deferring them until iommu_tlb_sync(). +* single page. IOMMUs that cannot batch TLB invalidation +* operations efficiently will typically issue them here, but +* others may decide to update the iommu_iotlb_gather structure +* and defer the invalidation until iommu_tlb_sync() instead. * * Note that these can all be called in atomic context and must therefore * not block.
@@ -42,7 +42,8 @@ struct iommu_flush_ops { void *cookie);void (*tlb_flush_leaf)(unsigned long iova,size_t size,size_t granule,void *cookie); -void (*tlb_add_page)(unsigned long iova,size_t granule,void *cookie); +void (*tlb_add_page)(struct iommu_iotlb_gather *gather, +unsigned long iova,size_t granule,void *cookie); };/**
@@ -209,11 +210,12 @@ io_pgtable_tlb_flush_leaf(struct io_pgtable *iop, unsigned long iova, }static inline void -io_pgtable_tlb_add_page (struct io_pgtable *iop,unsigned long iova, +io_pgtable_tlb_add_page (struct io_pgtable *iop, +struct iommu_iotlb_gather * gather,unsigned long iova,size_t granule) {if (iop->cfg.tlb->tlb_add_page) - iop->cfg.tlb->tlb_add_page (iova, granule, iop->cookie); + iop->cfg.tlb->tlb_add_page (gather, iova, granule, iop->cookie); }/**
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 14/19 iommu/arm-smmu-v3: Separate s/w and h/w views of prod and cons indexes
In reply to Will DeaconIn preparation for rewriting the command queue insertion code to use a new algorithm, separate the software and hardware views of the prod and cons indexes so that manipulating the software state doesn't automatically update the hardware state at the same time.
No functional change.
--- drivers/iommu/arm-smmu-v3.c | 36 ++++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 14 deletions(-)
drivers/iommu/arm-smmu-v3.c +22 -14
@@ -691,17 +691,13 @@ static bool queue_empty(struct arm_smmu_queue *q)Q_WRP (q, q->prod) ==Q_WRP (q, q->cons); } -static void queue_sync_cons (struct arm_smmu_queue *q) +static void queue_sync_cons_in (struct arm_smmu_queue *q) { q->cons =readl_relaxed (q->cons_reg); } -static void queue_inc_cons (struct arm_smmu_queue *q) +static void queue_sync_cons_out (struct arm_smmu_queue *q) { -u32 cons = (Q_WRP (q, q->cons) |Q_IDX (q, q->cons)) +1 ; - - q->cons =Q_OVF (q, q->cons) |Q_WRP (q, cons) |Q_IDX (q, cons); -/* * Ensure that all CPU accesses (reads and writes) to the queue * are complete before we update the cons pointer.
@@ -710,7 +706,13 @@ static void queue_inc_cons(struct arm_smmu_queue *q)writel_relaxed (q->cons, q->cons_reg); } -static int queue_sync_prod (struct arm_smmu_queue *q) +static void queue_inc_cons (struct arm_smmu_queue *q) +{ +u32 cons = (Q_WRP (q, q->cons) |Q_IDX (q, q->cons)) +1 ; + q->cons =Q_OVF (q, q->cons) |Q_WRP (q, cons) |Q_IDX (q, cons); +} + +static int queue_sync_prod_in (struct arm_smmu_queue *q) {int ret =0 ;u32 prod =readl_relaxed (q->prod_reg);
@@ -722,12 +724,15 @@ static int queue_sync_prod(struct arm_smmu_queue *q)return ret; } +static void queue_sync_prod_out (struct arm_smmu_queue *q) +{ +writel (q->prod, q->prod_reg); +} +static void queue_inc_prod (struct arm_smmu_queue *q) {u32 prod = (Q_WRP (q, q->prod) |Q_IDX (q, q->prod)) +1 ; - q->prod =Q_OVF (q, q->prod) |Q_WRP (q, prod) |Q_IDX (q, prod); -writel (q->prod, q->prod_reg); }/*
@@ -744,7 +749,7 @@ static int queue_poll_cons(struct arm_smmu_queue *q, bool sync, bool wfe) ARM_SMMU_CMDQ_SYNC_TIMEOUT_US : ARM_SMMU_POLL_TIMEOUT_US); -while (queue_sync_cons (q), (sync ? !queue_empty (q) :queue_full (q))) { +while (queue_sync_cons_in (q), (sync ? !queue_empty (q) :queue_full (q))) {if (ktime_compare (ktime_get (), timeout) >0 )return -ETIMEDOUT ;
@@ -778,6 +783,7 @@ static int queue_insert_raw(struct arm_smmu_queue *q, u64 *ent)queue_write (Q_ENT (q, q->prod), ent, q->ent_dwords);queue_inc_prod (q); +queue_sync_prod_out (q);return 0 ; }
@@ -796,6 +802,7 @@ static int queue_remove_raw(struct arm_smmu_queue *q, u64 *ent)queue_read (ent,Q_ENT (q, q->cons), q->ent_dwords);queue_inc_cons (q); +queue_sync_cons_out (q);return 0 ; }
@@ -1316,7 +1323,7 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev)* Not much we can do on overflow, so scream and pretend we're * trying harder. */ -if (queue_sync_prod (q) == -EOVERFLOW ) +if (queue_sync_prod_in (q) == -EOVERFLOW )dev_err (smmu->dev,"EVTQ overflow detected -- events lost\n" ); }while (!queue_empty (q));
@@ -1373,7 +1380,7 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev)while (!queue_remove_raw (q, evt))arm_smmu_handle_ppr (smmu, evt); -if (queue_sync_prod (q) == -EOVERFLOW ) +if (queue_sync_prod_in (q) == -EOVERFLOW )dev_err (smmu->dev,"PRIQ overflow detected -- requests lost\n" ); }while (!queue_empty (q));
@@ -1557,8 +1564,9 @@ static void arm_smmu_tlb_inv_context(void *cookie)/* * NOTE: when io-pgtable is in non-strict mode, we may get here with * PTEs previously cleared by unmaps on the current CPU not yet visible -* to the SMMU. We are relying on the DSB implicit in queue_inc_prod() -* to guarantee those are observed before the TLBI. Do be careful, 007. +* to the SMMU. We are relying on the DSB implicit in +* queue_sync_prod_out() to guarantee those are observed before the +* TLBI. Do be careful, 007. */ arm_smmu_cmdq_issue_cmd (smmu, &cmd);arm_smmu_cmdq_issue_sync (smmu);
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 15/19 iommu/arm-smmu-v3: Drop unused 'q' argument from Q_OVF macro
In reply to Will DeaconThe Q_OVF macro doesn't need to access the arm_smmu_queue structure, so drop the unused macro argument.
No functional change.
--- drivers/iommu/arm-smmu-v3.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/iommu/arm-smmu-v3.c +6 -6
@@ -184,7 +184,7 @@#define Q_IDX (q, p) ((p) & ((1 << (q)->max_n_shift) - 1))#define Q_WRP (q, p) ((p) & (1 << (q)->max_n_shift))#define Q_OVERFLOW_FLAG (1 << 31) -#define Q_OVF (q, p) ((p) & Q_OVERFLOW_FLAG) +#define Q_OVF (p) ((p) & Q_OVERFLOW_FLAG)#define Q_ENT (q, p) ((q)->base + \ Q_IDX(q, p) * (q)->ent_dwords)
@@ -709,7 +709,7 @@ static void queue_sync_cons_out(struct arm_smmu_queue *q)static void queue_inc_cons (struct arm_smmu_queue *q) {u32 cons = (Q_WRP (q, q->cons) |Q_IDX (q, q->cons)) +1 ; - q->cons =Q_OVF (q, q->cons) |Q_WRP (q, cons) |Q_IDX (q, cons); + q->cons =Q_OVF (q->cons) |Q_WRP (q, cons) |Q_IDX (q, cons); }static int queue_sync_prod_in (struct arm_smmu_queue *q)
@@ -717,7 +717,7 @@ static int queue_sync_prod_in(struct arm_smmu_queue *q)int ret =0 ;u32 prod =readl_relaxed (q->prod_reg); -if (Q_OVF (q, prod) !=Q_OVF (q, q->prod)) +if (Q_OVF (prod) !=Q_OVF (q->prod)) ret = -EOVERFLOW ; q->prod = prod;
@@ -732,7 +732,7 @@ static void queue_sync_prod_out(struct arm_smmu_queue *q)static void queue_inc_prod (struct arm_smmu_queue *q) {u32 prod = (Q_WRP (q, q->prod) |Q_IDX (q, q->prod)) +1 ; - q->prod =Q_OVF (q, q->prod) |Q_WRP (q, prod) |Q_IDX (q, prod); + q->prod =Q_OVF (q->prod) |Q_WRP (q, prod) |Q_IDX (q, prod); }/*
@@ -1328,7 +1328,7 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) }while (!queue_empty (q));/* Sync our overflow flag, as we believe we're up to speed */ - q->cons =Q_OVF (q, q->prod) |Q_WRP (q, q->cons) |Q_IDX (q, q->cons); + q->cons =Q_OVF (q->prod) |Q_WRP (q, q->cons) |Q_IDX (q, q->cons);return IRQ_HANDLED ; }
@@ -1385,7 +1385,7 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev) }while (!queue_empty (q));/* Sync our overflow flag, as we believe we're up to speed */ - q->cons =Q_OVF (q, q->prod) |Q_WRP (q, q->cons) |Q_IDX (q, q->cons); + q->cons =Q_OVF (q->prod) |Q_WRP (q, q->cons) |Q_IDX (q, q->cons);writel (q->cons, q->cons_reg);return IRQ_HANDLED ; }
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 16/19 iommu/arm-smmu-v3: Move low-level queue fields out of arm_smmu_queue
In reply to Will DeaconIn preparation for rewriting the command queue insertion code to use a new algorithm, introduce a new arm_smmu_ll_queue structure which contains only the information necessary to perform queue arithmetic for a queue and will later be extended so that we can perform complex atomic manipulation on some of the fields.
No functional change.
--- drivers/iommu/arm-smmu-v3.c | 88 ++++++++++++++++++++++++--------------------- 1 file changed, 47 insertions(+), 41 deletions(-)
drivers/iommu/arm-smmu-v3.c +47 -41
@@ -181,8 +181,8 @@#define ARM_SMMU_MEMATTR_DEVICE_nGnRE 0x1#define ARM_SMMU_MEMATTR_OIWB 0xf -#define Q_IDX (q, p) ((p) & ((1 << (q)->max_n_shift) - 1)) -#define Q_WRP (q, p) ((p) & (1 << (q)->max_n_shift)) +#define Q_IDX (q, p) ((p) & ((1 << (q)->llq.max_n_shift) - 1)) +#define Q_WRP (q, p) ((p) & (1 << (q)->llq.max_n_shift))#define Q_OVERFLOW_FLAG (1 << 31)#define Q_OVF (p) ((p) & Q_OVERFLOW_FLAG)#define Q_ENT (q, p) ((q)->base + \
@@ -472,7 +472,14 @@ struct arm_smmu_cmdq_ent { }; }; +struct arm_smmu_ll_queue { +u32 prod; +u32 cons; +u32 max_n_shift; +}; +struct arm_smmu_queue { +struct arm_smmu_ll_queue llq;int irq;/* Wired interrupt */ __le64 *base;
@@ -480,9 +487,6 @@ struct arm_smmu_queue {u64 q_base;size_t ent_dwords; -u32 max_n_shift; -u32 prod; -u32 cons;u32 __iomem *prod_reg;u32 __iomem *cons_reg;
@@ -681,19 +685,19 @@ static void parse_driver_options(struct arm_smmu_device *smmu)/* Low-level queue manipulation functions */ static bool queue_full (struct arm_smmu_queue *q) { -return Q_IDX (q, q->prod) ==Q_IDX (q, q->cons) && -Q_WRP (q, q->prod) !=Q_WRP (q, q->cons); +return Q_IDX (q, q->llq.prod) ==Q_IDX (q, q->llq.cons) && +Q_WRP (q, q->llq.prod) !=Q_WRP (q, q->llq.cons); }static bool queue_empty (struct arm_smmu_queue *q) { -return Q_IDX (q, q->prod) ==Q_IDX (q, q->cons) && -Q_WRP (q, q->prod) ==Q_WRP (q, q->cons); +return Q_IDX (q, q->llq.prod) ==Q_IDX (q, q->llq.cons) && +Q_WRP (q, q->llq.prod) ==Q_WRP (q, q->llq.cons); }static void queue_sync_cons_in (struct arm_smmu_queue *q) { - q->cons =readl_relaxed (q->cons_reg); + q->llq.cons =readl_relaxed (q->cons_reg); }static void queue_sync_cons_out (struct arm_smmu_queue *q)
@@ -703,13 +707,13 @@ static void queue_sync_cons_out(struct arm_smmu_queue *q)* are complete before we update the cons pointer. */ mb (); -writel_relaxed (q->cons, q->cons_reg); +writel_relaxed (q->llq.cons, q->cons_reg); }static void queue_inc_cons (struct arm_smmu_queue *q) { -u32 cons = (Q_WRP (q, q->cons) |Q_IDX (q, q->cons)) +1 ; - q->cons =Q_OVF (q->cons) |Q_WRP (q, cons) |Q_IDX (q, cons); +u32 cons = (Q_WRP (q, q->llq.cons) |Q_IDX (q, q->llq.cons)) +1 ; + q->llq.cons =Q_OVF (q->llq.cons) |Q_WRP (q, cons) |Q_IDX (q, cons); }static int queue_sync_prod_in (struct arm_smmu_queue *q)
@@ -717,22 +721,22 @@ static int queue_sync_prod_in(struct arm_smmu_queue *q)int ret =0 ;u32 prod =readl_relaxed (q->prod_reg); -if (Q_OVF (prod) !=Q_OVF (q->prod)) +if (Q_OVF (prod) !=Q_OVF (q->llq.prod)) ret = -EOVERFLOW ; - q->prod = prod; + q->llq.prod = prod;return ret; }static void queue_sync_prod_out (struct arm_smmu_queue *q) { -writel (q->prod, q->prod_reg); +writel (q->llq.prod, q->prod_reg); }static void queue_inc_prod (struct arm_smmu_queue *q) { -u32 prod = (Q_WRP (q, q->prod) |Q_IDX (q, q->prod)) +1 ; - q->prod =Q_OVF (q->prod) |Q_WRP (q, prod) |Q_IDX (q, prod); +u32 prod = (Q_WRP (q, q->llq.prod) |Q_IDX (q, q->llq.prod)) +1 ; + q->llq.prod =Q_OVF (q->llq.prod) |Q_WRP (q, prod) |Q_IDX (q, prod); }/*
@@ -781,7 +785,7 @@ static int queue_insert_raw(struct arm_smmu_queue *q, u64 *ent)if (queue_full (q))return -ENOSPC ; -queue_write (Q_ENT (q, q->prod), ent, q->ent_dwords); +queue_write (Q_ENT (q, q->llq.prod), ent, q->ent_dwords);queue_inc_prod (q);queue_sync_prod_out (q);return 0 ;
@@ -800,7 +804,7 @@ static int queue_remove_raw(struct arm_smmu_queue *q, u64 *ent)if (queue_empty (q))return -EAGAIN ; -queue_read (ent,Q_ENT (q, q->cons), q->ent_dwords); +queue_read (ent,Q_ENT (q, q->llq.cons), q->ent_dwords);queue_inc_cons (q);queue_sync_cons_out (q);return 0 ;
@@ -1328,7 +1332,8 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) }while (!queue_empty (q));/* Sync our overflow flag, as we believe we're up to speed */ - q->cons =Q_OVF (q->prod) |Q_WRP (q, q->cons) |Q_IDX (q, q->cons); + q->llq.cons =Q_OVF (q->llq.prod) |Q_WRP (q, q->llq.cons) | +Q_IDX (q, q->llq.cons);return IRQ_HANDLED ; }
@@ -1385,8 +1390,9 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev) }while (!queue_empty (q));/* Sync our overflow flag, as we believe we're up to speed */ - q->cons =Q_OVF (q->prod) |Q_WRP (q, q->cons) |Q_IDX (q, q->cons); -writel (q->cons, q->cons_reg); + q->llq.cons =Q_OVF (q->llq.prod) |Q_WRP (q, q->llq.cons) | +Q_IDX (q, q->llq.cons); +writel (q->llq.cons, q->cons_reg);return IRQ_HANDLED ; }
@@ -2307,13 +2313,13 @@ static int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,size_t qsz;do { - qsz = ((1 << q->max_n_shift) * dwords) <<3 ; + qsz = ((1 << q->llq.max_n_shift) * dwords) <<3 ; q->base =dmam_alloc_coherent (smmu->dev, qsz, &q->base_dma,GFP_KERNEL );if (q->base || qsz <PAGE_SIZE )break ; - q->max_n_shift--; + q->llq.max_n_shift--; }while (1 );if (!q->base) {
@@ -2325,7 +2331,7 @@ static int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,if (!WARN_ON (q->base_dma & (qsz -1 ))) {dev_info (smmu->dev,"allocated %u entries for %s\n" , -1 << q->max_n_shift, name); +1 << q->llq.max_n_shift, name); } q->prod_reg =arm_smmu_page1_fixup (prod_off, smmu);
@@ -2334,9 +2340,9 @@ static int arm_smmu_init_one_queue(struct arm_smmu_device *smmu, q->q_base =Q_BASE_RWA ; q->q_base |= q->base_dma &Q_BASE_ADDR_MASK ; - q->q_base |=FIELD_PREP (Q_BASE_LOG2SIZE , q->max_n_shift); + q->q_base |=FIELD_PREP (Q_BASE_LOG2SIZE , q->llq.max_n_shift); - q->prod = q->cons =0 ; + q->llq.prod = q->llq.cons =0 ;return 0 ; }
@@ -2729,8 +2735,8 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)/* Command queue */ writeq_relaxed (smmu->cmdq.q.q_base, smmu->base +ARM_SMMU_CMDQ_BASE ); -writel_relaxed (smmu->cmdq.q.prod, smmu->base +ARM_SMMU_CMDQ_PROD ); -writel_relaxed (smmu->cmdq.q.cons, smmu->base +ARM_SMMU_CMDQ_CONS ); +writel_relaxed (smmu->cmdq.q.llq.prod, smmu->base +ARM_SMMU_CMDQ_PROD ); +writel_relaxed (smmu->cmdq.q.llq.cons, smmu->base +ARM_SMMU_CMDQ_CONS ); enables =CR0_CMDQEN ; ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0,
@@ -2757,9 +2763,9 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)/* Event queue */ writeq_relaxed (smmu->evtq.q.q_base, smmu->base +ARM_SMMU_EVTQ_BASE ); -writel_relaxed (smmu->evtq.q.prod, +writel_relaxed (smmu->evtq.q.llq.prod,arm_smmu_page1_fixup (ARM_SMMU_EVTQ_PROD , smmu)); -writel_relaxed (smmu->evtq.q.cons, +writel_relaxed (smmu->evtq.q.llq.cons,arm_smmu_page1_fixup (ARM_SMMU_EVTQ_CONS , smmu)); enables |=CR0_EVTQEN ;
@@ -2774,9 +2780,9 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)if (smmu->features &ARM_SMMU_FEAT_PRI ) {writeq_relaxed (smmu->priq.q.q_base, smmu->base +ARM_SMMU_PRIQ_BASE ); -writel_relaxed (smmu->priq.q.prod, +writel_relaxed (smmu->priq.q.llq.prod,arm_smmu_page1_fixup (ARM_SMMU_PRIQ_PROD , smmu)); -writel_relaxed (smmu->priq.q.cons, +writel_relaxed (smmu->priq.q.llq.cons,arm_smmu_page1_fixup (ARM_SMMU_PRIQ_CONS , smmu)); enables |=CR0_PRIQEN ;
@@ -2930,18 +2936,18 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) }/* Queue sizes, capped to ensure natural alignment */ - smmu->cmdq.q.max_n_shift =min_t (u32,CMDQ_MAX_SZ_SHIFT , -FIELD_GET (IDR1_CMDQS , reg)); -if (!smmu->cmdq.q.max_n_shift) { + smmu->cmdq.q.llq.max_n_shift =min_t (u32,CMDQ_MAX_SZ_SHIFT , +FIELD_GET (IDR1_CMDQS , reg)); +if (!smmu->cmdq.q.llq.max_n_shift) {/* Odd alignment restrictions on the base, so ignore for now */ dev_err (smmu->dev,"unit-length command queue not supported\n" );return -ENXIO ; } - smmu->evtq.q.max_n_shift =min_t (u32,EVTQ_MAX_SZ_SHIFT , -FIELD_GET (IDR1_EVTQS , reg)); - smmu->priq.q.max_n_shift =min_t (u32,PRIQ_MAX_SZ_SHIFT , -FIELD_GET (IDR1_PRIQS , reg)); + smmu->evtq.q.llq.max_n_shift =min_t (u32,EVTQ_MAX_SZ_SHIFT , +FIELD_GET (IDR1_EVTQS , reg)); + smmu->priq.q.llq.max_n_shift =min_t (u32,PRIQ_MAX_SZ_SHIFT , +FIELD_GET (IDR1_PRIQS , reg));/* SID/SSID sizes */ smmu->ssid_bits =FIELD_GET (IDR1_SSIDSIZE , reg);
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 17/19 iommu/arm-smmu-v3: Operate directly on low-level queue where possible
In reply to Will DeaconIn preparation for rewriting the command queue insertion code to use a new algorithm, rework many of our queue macro accessors and manipulation functions so that they operate on the arm_smmu_ll_queue structure where possible. This will allow us to call these helpers on local variables without having to construct a full-blown arm_smmu_queue on the stack.
No functional change.
--- drivers/iommu/arm-smmu-v3.c | 58 ++++++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 27 deletions(-)
drivers/iommu/arm-smmu-v3.c +31 -27
@@ -181,12 +181,13 @@#define ARM_SMMU_MEMATTR_DEVICE_nGnRE 0x1#define ARM_SMMU_MEMATTR_OIWB 0xf -#define Q_IDX (q, p) ((p) & ((1 << (q)->llq.max_n_shift) - 1)) -#define Q_WRP (q, p) ((p) & (1 << (q)->llq.max_n_shift)) +#define Q_IDX (llq, p) ((p) & ((1 << (llq)->max_n_shift) - 1)) +#define Q_WRP (llq, p) ((p) & (1 << (llq)->max_n_shift))#define Q_OVERFLOW_FLAG (1 << 31)#define Q_OVF (p) ((p) & Q_OVERFLOW_FLAG)#define Q_ENT (q, p) ((q)->base + \ - Q_IDX(q, p) * (q)->ent_dwords) + Q_IDX(&((q)->llq), p) * \ + (q)->ent_dwords)#define Q_BASE_RWA (1UL << 62)#define Q_BASE_ADDR_MASK GENMASK_ULL(51, 5)
@@ -683,16 +684,16 @@ static void parse_driver_options(struct arm_smmu_device *smmu) }/* Low-level queue manipulation functions */ -static bool queue_full (struct arm_smmu_queue *q) +static bool queue_full (struct arm_smmu_ll_queue *q) { -return Q_IDX (q, q->llq.prod) ==Q_IDX (q, q->llq.cons) && -Q_WRP (q, q->llq.prod) !=Q_WRP (q, q->llq.cons); +return Q_IDX (q, q->prod) ==Q_IDX (q, q->cons) && +Q_WRP (q, q->prod) !=Q_WRP (q, q->cons); } -static bool queue_empty (struct arm_smmu_queue *q) +static bool queue_empty (struct arm_smmu_ll_queue *q) { -return Q_IDX (q, q->llq.prod) ==Q_IDX (q, q->llq.cons) && -Q_WRP (q, q->llq.prod) ==Q_WRP (q, q->llq.cons); +return Q_IDX (q, q->prod) ==Q_IDX (q, q->cons) && +Q_WRP (q, q->prod) ==Q_WRP (q, q->cons); }static void queue_sync_cons_in (struct arm_smmu_queue *q)
@@ -710,10 +711,10 @@ static void queue_sync_cons_out(struct arm_smmu_queue *q)writel_relaxed (q->llq.cons, q->cons_reg); } -static void queue_inc_cons (struct arm_smmu_queue *q) +static void queue_inc_cons (struct arm_smmu_ll_queue *q) { -u32 cons = (Q_WRP (q, q->llq.cons) |Q_IDX (q, q->llq.cons)) +1 ; - q->llq.cons =Q_OVF (q->llq.cons) |Q_WRP (q, cons) |Q_IDX (q, cons); +u32 cons = (Q_WRP (q, q->cons) |Q_IDX (q, q->cons)) +1 ; + q->cons =Q_OVF (q->cons) |Q_WRP (q, cons) |Q_IDX (q, cons); }static int queue_sync_prod_in (struct arm_smmu_queue *q)
@@ -733,10 +734,10 @@ static void queue_sync_prod_out(struct arm_smmu_queue *q)writel (q->llq.prod, q->prod_reg); } -static void queue_inc_prod (struct arm_smmu_queue *q) +static void queue_inc_prod (struct arm_smmu_ll_queue *q) { -u32 prod = (Q_WRP (q, q->llq.prod) |Q_IDX (q, q->llq.prod)) +1 ; - q->llq.prod =Q_OVF (q->llq.prod) |Q_WRP (q, prod) |Q_IDX (q, prod); +u32 prod = (Q_WRP (q, q->prod) |Q_IDX (q, q->prod)) +1 ; + q->prod =Q_OVF (q->prod) |Q_WRP (q, prod) |Q_IDX (q, prod); }/*
@@ -753,7 +754,8 @@ static int queue_poll_cons(struct arm_smmu_queue *q, bool sync, bool wfe) ARM_SMMU_CMDQ_SYNC_TIMEOUT_US : ARM_SMMU_POLL_TIMEOUT_US); -while (queue_sync_cons_in (q), (sync ? !queue_empty (q) :queue_full (q))) { +while (queue_sync_cons_in (q), + (sync ? !queue_empty (&q->llq) :queue_full (&q->llq))) {if (ktime_compare (ktime_get (), timeout) >0 )return -ETIMEDOUT ;
@@ -782,11 +784,11 @@ static void queue_write(__le64 *dst, u64 *src, size_t n_dwords)static int queue_insert_raw (struct arm_smmu_queue *q,u64 *ent) { -if (queue_full (q)) +if (queue_full (&q->llq))return -ENOSPC ;queue_write (Q_ENT (q, q->llq.prod), ent, q->ent_dwords); -queue_inc_prod (q); +queue_inc_prod (&q->llq);queue_sync_prod_out (q);return 0 ; }
@@ -801,11 +803,11 @@ static void queue_read(__le64 *dst, u64 *src, size_t n_dwords)static int queue_remove_raw (struct arm_smmu_queue *q,u64 *ent) { -if (queue_empty (q)) +if (queue_empty (&q->llq))return -EAGAIN ;queue_read (ent,Q_ENT (q, q->llq.cons), q->ent_dwords); -queue_inc_cons (q); +queue_inc_cons (&q->llq);queue_sync_cons_out (q);return 0 ; }
@@ -1310,6 +1312,7 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev)int i;struct arm_smmu_device *smmu = dev;struct arm_smmu_queue *q = &smmu->evtq.q; +struct arm_smmu_ll_queue *llq = &q->llq;u64 evt[EVTQ_ENT_DWORDS ]; do {
@@ -1329,11 +1332,11 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev)*/ if (queue_sync_prod_in (q) == -EOVERFLOW )dev_err (smmu->dev,"EVTQ overflow detected -- events lost\n" ); - }while (!queue_empty (q)); + }while (!queue_empty (llq));/* Sync our overflow flag, as we believe we're up to speed */ - q->llq.cons =Q_OVF (q->llq.prod) |Q_WRP (q, q->llq.cons) | -Q_IDX (q, q->llq.cons); + llq->cons =Q_OVF (llq->prod) |Q_WRP (llq, llq->cons) | +Q_IDX (llq, llq->cons);return IRQ_HANDLED ; }
@@ -1379,6 +1382,7 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev) {struct arm_smmu_device *smmu = dev;struct arm_smmu_queue *q = &smmu->priq.q; +struct arm_smmu_ll_queue *llq = &q->llq;u64 evt[PRIQ_ENT_DWORDS ]; do {
@@ -1387,12 +1391,12 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev)if (queue_sync_prod_in (q) == -EOVERFLOW )dev_err (smmu->dev,"PRIQ overflow detected -- requests lost\n" ); - }while (!queue_empty (q)); + }while (!queue_empty (llq));/* Sync our overflow flag, as we believe we're up to speed */ - q->llq.cons =Q_OVF (q->llq.prod) |Q_WRP (q, q->llq.cons) | -Q_IDX (q, q->llq.cons); -writel (q->llq.cons, q->cons_reg); + llq->cons =Q_OVF (llq->prod) |Q_WRP (llq, llq->cons) | +Q_IDX (llq, llq->cons); +queue_sync_cons_out (q);return IRQ_HANDLED ; }
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 18/19 iommu/arm-smmu-v3: Reduce contention during command-queue insertion
In reply to Will DeaconThe SMMU command queue is a bottleneck in large systems, thanks to the spin_lock which serialises accesses from all CPUs to the single queue supported by the hardware.
Attempt to improve this situation by moving to a new algorithm for inserting commands into the queue, which is lock-free on the fast-path.
--- drivers/iommu/arm-smmu-v3.c | 673 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 529 insertions(+), 144 deletions(-)
drivers/iommu/arm-smmu-v3.c +529 -144
@@ -183,7 +183,7 @@#define Q_IDX (llq, p) ((p) & ((1 << (llq)->max_n_shift) - 1))#define Q_WRP (llq, p) ((p) & (1 << (llq)->max_n_shift)) -#define Q_OVERFLOW_FLAG (1 << 31) +#define Q_OVERFLOW_FLAG (1U << 31)#define Q_OVF (p) ((p) & Q_OVERFLOW_FLAG)#define Q_ENT (q, p) ((q)->base + \ Q_IDX(&((q)->llq), p) * \
@@ -301,6 +301,8 @@#define CMDQ_ERR_CERROR_ABT_IDX 2#define CMDQ_ERR_CERROR_ATC_INV_IDX 3 +#define CMDQ_PROD_OWNED_FLAG Q_OVERFLOW_FLAG +#define CMDQ_0_OP GENMASK_ULL(7, 0)#define CMDQ_0_SSV (1UL << 11)
@@ -363,9 +365,8 @@#define PRIQ_1_ADDR_MASK GENMASK_ULL(63, 12)/* High-level queue structures */ -#define ARM_SMMU_POLL_TIMEOUT_US 100 -#define ARM_SMMU_CMDQ_SYNC_TIMEOUT_US 1000000/* 1s! */ -#define ARM_SMMU_CMDQ_SYNC_SPIN_COUNT 10 +#define ARM_SMMU_POLL_TIMEOUT_US 1000000/* 1s! */ +#define ARM_SMMU_POLL_SPIN_COUNT 10#define MSI_IOVA_BASE 0x8000000#define MSI_IOVA_LENGTH 0x100000
@@ -467,15 +468,24 @@ struct arm_smmu_cmdq_ent {#define CMDQ_OP_CMD_SYNC 0x46struct { -u32 msidata;u64 msiaddr; } sync; }; };struct arm_smmu_ll_queue { -u32 prod; -u32 cons; +union { +u64 val; +struct { +u32 prod; +u32 cons; + }; +struct { +atomic_t prod; +atomic_t cons; + } atomic; +u8 __pad[SMP_CACHE_BYTES ]; + }____cacheline_aligned_in_smp ;u32 max_n_shift; };
@@ -493,9 +503,18 @@ struct arm_smmu_queue {u32 __iomem *cons_reg; }; +struct arm_smmu_queue_poll { +ktime_t timeout; +unsigned int delay; +unsigned int spin_cnt; +bool wfe; +}; +struct arm_smmu_cmdq {struct arm_smmu_queue q; -spinlock_t lock; +atomic_long_t *valid_map; +atomic_t owner_prod; +atomic_t lock; };struct arm_smmu_evtq {
@@ -575,8 +594,6 @@ struct arm_smmu_device {int gerr_irq;int combined_irq; -u32 sync_nr; -u8 prev_cmd_opcode;unsigned long ias;/* IPA */ unsigned long oas;/* PA */
@@ -595,12 +612,6 @@ struct arm_smmu_device {struct arm_smmu_strtab_cfg strtab_cfg; -/* Hi16xx adds an extra 32 bits of goodness to its MSI payload */ -union { -u32 sync_count; -u64 padding; - }; -/* IOMMU core code handle */ struct iommu_device iommu; };
@@ -684,6 +695,21 @@ static void parse_driver_options(struct arm_smmu_device *smmu) }/* Low-level queue manipulation functions */ +static bool queue_has_space (struct arm_smmu_ll_queue *q,u32 n) +{ +u32 space, prod, cons; + + prod =Q_IDX (q, q->prod); + cons =Q_IDX (q, q->cons); + +if (Q_WRP (q, q->prod) ==Q_WRP (q, q->cons)) + space = (1 << q->max_n_shift) - (prod - cons); +else + space = cons - prod; + +return space >= n; +} +static bool queue_full (struct arm_smmu_ll_queue *q) { return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
@@ -696,9 +722,12 @@ static bool queue_empty(struct arm_smmu_ll_queue *q)Q_WRP (q, q->prod) ==Q_WRP (q, q->cons); } -static void queue_sync_cons_in (struct arm_smmu_queue *q) +static bool queue_consumed (struct arm_smmu_ll_queue *q,u32 prod) { - q->llq.cons =readl_relaxed (q->cons_reg); +return ((Q_WRP (q, q->cons) ==Q_WRP (q, prod)) && + (Q_IDX (q, q->cons) >Q_IDX (q, prod))) || + ((Q_WRP (q, q->cons) !=Q_WRP (q, prod)) && + (Q_IDX (q, q->cons) <=Q_IDX (q, prod))); }static void queue_sync_cons_out (struct arm_smmu_queue *q)
@@ -729,46 +758,34 @@ static int queue_sync_prod_in(struct arm_smmu_queue *q)return ret; } -static void queue_sync_prod_out (struct arm_smmu_queue *q) +static u32 queue_inc_prod_n (struct arm_smmu_ll_queue *q,int n) { -writel (q->llq.prod, q->prod_reg); +u32 prod = (Q_WRP (q, q->prod) |Q_IDX (q, q->prod)) + n; +return Q_OVF (q->prod) |Q_WRP (q, prod) |Q_IDX (q, prod); } -static void queue_inc_prod (struct arm_smmu_ll_queue *q) +static void queue_poll_init (struct arm_smmu_device *smmu, +struct arm_smmu_queue_poll *qp) { -u32 prod = (Q_WRP (q, q->prod) |Q_IDX (q, q->prod)) +1 ; - q->prod =Q_OVF (q->prod) |Q_WRP (q, prod) |Q_IDX (q, prod); + qp->delay =1 ; + qp->spin_cnt =0 ; + qp->wfe = !!(smmu->features &ARM_SMMU_FEAT_SEV ); + qp->timeout =ktime_add_us (ktime_get (),ARM_SMMU_POLL_TIMEOUT_US ); } -/* -* Wait for the SMMU to consume items. If sync is true, wait until the queue -* is empty. Otherwise, wait until there is at least one free slot. -*/ -static int queue_poll_cons (struct arm_smmu_queue *q,bool sync,bool wfe) +static int queue_poll (struct arm_smmu_queue_poll *qp) { -ktime_t timeout; -unsigned int delay =1 , spin_cnt =0 ; - -/* Wait longer if it's a CMD_SYNC */ - timeout =ktime_add_us (ktime_get (), sync ? -ARM_SMMU_CMDQ_SYNC_TIMEOUT_US : -ARM_SMMU_POLL_TIMEOUT_US ); - -while (queue_sync_cons_in (q), - (sync ? !queue_empty (&q->llq) :queue_full (&q->llq))) { -if (ktime_compare (ktime_get (), timeout) >0 ) -return -ETIMEDOUT ; +if (ktime_compare (ktime_get (), qp->timeout) >0 ) +return -ETIMEDOUT ; -if (wfe) { -wfe (); - }else if (++spin_cnt <ARM_SMMU_CMDQ_SYNC_SPIN_COUNT ) { -cpu_relax (); -continue ; - }else { -udelay (delay); - delay *=2 ; - spin_cnt =0 ; - } +if (qp->wfe) { +wfe (); + }else if (++qp->spin_cnt <ARM_SMMU_POLL_SPIN_COUNT ) { +cpu_relax (); + }else { +udelay (qp->delay); + qp->delay *=2 ; + qp->spin_cnt =0 ; }return 0 ;
@@ -782,17 +799,6 @@ static void queue_write(__le64 *dst, u64 *src, size_t n_dwords) *dst++ =cpu_to_le64 (*src++); } -static int queue_insert_raw (struct arm_smmu_queue *q,u64 *ent) -{ -if (queue_full (&q->llq)) -return -ENOSPC ; - -queue_write (Q_ENT (q, q->llq.prod), ent, q->ent_dwords); -queue_inc_prod (&q->llq); -queue_sync_prod_out (q); -return 0 ; -} -static void queue_read (__le64 *dst,u64 *src,size_t n_dwords) {int i;
@@ -875,20 +881,14 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) cmd[1 ] |=FIELD_PREP (CMDQ_PRI_1_RESP , ent->pri.resp);break ;case CMDQ_OP_CMD_SYNC : -if (ent->sync.msiaddr) +if (ent->sync.msiaddr) { cmd[0 ] |=FIELD_PREP (CMDQ_SYNC_0_CS ,CMDQ_SYNC_0_CS_IRQ ); -else + cmd[1 ] |= ent->sync.msiaddr &CMDQ_SYNC_1_MSIADDR_MASK ; + }else { cmd[0 ] |=FIELD_PREP (CMDQ_SYNC_0_CS ,CMDQ_SYNC_0_CS_SEV ); + } cmd[0 ] |=FIELD_PREP (CMDQ_SYNC_0_MSH ,ARM_SMMU_SH_ISH ); cmd[0 ] |=FIELD_PREP (CMDQ_SYNC_0_MSIATTR ,ARM_SMMU_MEMATTR_OIWB ); -/* -* Commands are written little-endian, but we want the SMMU to -* receive MSIData, and thus write it back to memory, in CPU -* byte order, so big-endian needs an extra byteswap here. -*/ - cmd[0 ] |=FIELD_PREP (CMDQ_SYNC_0_MSIDATA , -cpu_to_le32 (ent->sync.msidata)); - cmd[1 ] |= ent->sync.msiaddr &CMDQ_SYNC_1_MSIADDR_MASK ;break ;default :return -ENOENT ;
@@ -897,6 +897,27 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)return 0 ; } +static void arm_smmu_cmdq_build_sync_cmd (u64 *cmd,struct arm_smmu_device *smmu, +u32 prod) +{ +struct arm_smmu_queue *q = &smmu->cmdq.q; +struct arm_smmu_cmdq_ent ent = { + .opcode =CMDQ_OP_CMD_SYNC , + }; + +/* +* Beware that Hi16xx adds an extra 32 bits of goodness to its MSI +* payload, so the write will zero the entire command on that platform. +*/ +if (smmu->features &ARM_SMMU_FEAT_MSI && + smmu->features &ARM_SMMU_FEAT_COHERENCY ) { + ent.sync.msiaddr = q->base_dma +Q_IDX (&q->llq, prod) * + q->ent_dwords *8 ; + } + +arm_smmu_cmdq_build_cmd (cmd, &ent); +} +static void arm_smmu_cmdq_skip_err (struct arm_smmu_device *smmu) {static const char *cerror_str[] = {
@@ -955,109 +976,436 @@ static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu)queue_write (Q_ENT (q, cons), cmd, q->ent_dwords); } -static void arm_smmu_cmdq_insert_cmd (struct arm_smmu_device *smmu,u64 *cmd) +/* +* Command queue locking. +* This is a form of bastardised rwlock with the following major changes: +* +* - The only LOCK routines are exclusive_trylock() and shared_lock(). +* Neither have barrier semantics, and instead provide only a control +* dependency. +* +* - The UNLOCK routines are supplemented with shared_tryunlock(), which +* fails if the caller appears to be the last lock holder (yes, this is +* racy). All successful UNLOCK routines have RELEASE semantics. +*/ +static void arm_smmu_cmdq_shared_lock (struct arm_smmu_cmdq *cmdq) { -struct arm_smmu_queue *q = &smmu->cmdq.q; -bool wfe = !!(smmu->features &ARM_SMMU_FEAT_SEV ); +int val; + +/* +* We can try to avoid the cmpxchg() loop by simply incrementing the +* lock counter. When held in exclusive state, the lock counter is set +* to INT_MIN so these increments won't hurt as the value will remain +* negative. +*/ +if (atomic_fetch_inc_relaxed (&cmdq->lock) >=0 ) +return ; + +do { + val =atomic_cond_read_relaxed (&cmdq->lock,VAL >=0 ); + }while (atomic_cmpxchg_relaxed (&cmdq->lock, val, val +1 ) != val); +} + +static void arm_smmu_cmdq_shared_unlock (struct arm_smmu_cmdq *cmdq) +{ + (void )atomic_dec_return_release (&cmdq->lock); +} + +static bool arm_smmu_cmdq_shared_tryunlock (struct arm_smmu_cmdq *cmdq) +{ +if (atomic_read (&cmdq->lock) ==1 ) +return false; + +arm_smmu_cmdq_shared_unlock (cmdq); +return true; +} + +#define arm_smmu_cmdq_exclusive_trylock_irqsave (cmdq, flags) \ +({ \ + bool __ret; \ + local_irq_save(flags); \ + __ret = !atomic_cmpxchg_relaxed(&cmdq->lock, 0, INT_MIN); \ + if (!__ret) \ + local_irq_restore(flags); \ + __ret; \ +}) + +#define arm_smmu_cmdq_exclusive_unlock_irqrestore (cmdq, flags) \ +({ \ + atomic_set_release(&cmdq->lock, 0); \ + local_irq_restore(flags); \ +}) + + +/* +* Command queue insertion. +* This is made fiddly by our attempts to achieve some sort of scalability +* since there is one queue shared amongst all of the CPUs in the system. If +* you like mixed-size concurrency, dependency ordering and relaxed atomics, +* then you'll *love* this monstrosity. +* +* The basic idea is to split the queue up into ranges of commands that are +* owned by a given CPU; the owner may not have written all of the commands +* itself, but is responsible for advancing the hardware prod pointer when +* the time comes. The algorithm is roughly: +* +* 1. Allocate some space in the queue. At this point we also discover +* whether the head of the queue is currently owned by another CPU, +* or whether we are the owner. +* +* 2. Write our commands into our allocated slots in the queue. +* +* 3. Mark our slots as valid in arm_smmu_cmdq.valid_map. +* +* 4. If we are an owner: +* a. Wait for the previous owner to finish. +* b. Mark the queue head as unowned, which tells us the range +* that we are responsible for publishing. +* c. Wait for all commands in our owned range to become valid. +* d. Advance the hardware prod pointer. +* e. Tell the next owner we've finished. +* +* 5. If we are inserting a CMD_SYNC (we may or may not have been an +* owner), then we need to stick around until it has completed: +* a. If we have MSIs, the SMMU can write back into the CMD_SYNC +* to clear the first 4 bytes. +* b. Otherwise, we spin waiting for the hardware cons pointer to +* advance past our command. +* +* The devil is in the details, particularly the use of locking for handling +* SYNC completion and freeing up space in the queue before we think that it is +* full. +*/ +static void __arm_smmu_cmdq_poll_set_valid_map (struct arm_smmu_cmdq *cmdq, +u32 sprod,u32 eprod,bool set) +{ +u32 swidx, sbidx, ewidx, ebidx; +struct arm_smmu_ll_queue llq = { + .max_n_shift = cmdq->q.llq.max_n_shift, + .prod = sprod, + }; + + ewidx =BIT_WORD (Q_IDX (&llq, eprod)); + ebidx =Q_IDX (&llq, eprod) %BITS_PER_LONG ; + +while (llq.prod != eprod) { +unsigned long mask; +atomic_long_t *ptr; +u32 limit =BITS_PER_LONG ; + + swidx =BIT_WORD (Q_IDX (&llq, llq.prod)); + sbidx =Q_IDX (&llq, llq.prod) %BITS_PER_LONG ; + + ptr = &cmdq->valid_map[swidx]; + +if ((swidx == ewidx) && (sbidx < ebidx)) + limit = ebidx; - smmu->prev_cmd_opcode =FIELD_GET (CMDQ_0_OP , cmd[0 ]); + mask =GENMASK (limit -1 , sbidx); -while (queue_insert_raw (q, cmd) == -ENOSPC ) { -if (queue_poll_cons (q, false, wfe)) -dev_err_ratelimited (smmu->dev,"CMDQ timeout\n" ); +/* +* The valid bit is the inverse of the wrap bit. This means +* that a zero-initialised queue is invalid and, after marking +* all entries as valid, they become invalid again when we +* wrap. +*/ +if (set) { +atomic_long_xor (mask, ptr); + }else {/* Poll */ +unsigned long valid; + + valid = (ULONG_MAX + !!Q_WRP (&llq, llq.prod)) & mask; +atomic_long_cond_read_relaxed (ptr, (VAL & mask) == valid); + } + + llq.prod =queue_inc_prod_n (&llq, limit - sbidx); } } -static void arm_smmu_cmdq_issue_cmd (struct arm_smmu_device *smmu, -struct arm_smmu_cmdq_ent *ent) +/* Mark all entries in the range [sprod, eprod) as valid */ +static void arm_smmu_cmdq_set_valid_map (struct arm_smmu_cmdq *cmdq, +u32 sprod,u32 eprod) +{ +__arm_smmu_cmdq_poll_set_valid_map (cmdq, sprod, eprod, true); +} + +/* Wait for all entries in the range [sprod, eprod) to become valid */ +static void arm_smmu_cmdq_poll_valid_map (struct arm_smmu_cmdq *cmdq, +u32 sprod,u32 eprod) +{ +__arm_smmu_cmdq_poll_set_valid_map (cmdq, sprod, eprod, false); +} + +/* Wait for the command queue to become non-full */ +static int arm_smmu_cmdq_poll_until_not_full (struct arm_smmu_device *smmu, +struct arm_smmu_ll_queue *llq) { -u64 cmd[CMDQ_ENT_DWORDS ];unsigned long flags; +struct arm_smmu_queue_poll qp; +struct arm_smmu_cmdq *cmdq = &smmu->cmdq; +int ret =0 ; -if (arm_smmu_cmdq_build_cmd (cmd, ent)) { -dev_warn (smmu->dev,"ignoring unknown CMDQ opcode 0x%x\n" , - ent->opcode); -return ; +/* +* Try to update our copy of cons by grabbing exclusive cmdq access. If +* that fails, spin until somebody else updates it for us. +*/ +if (arm_smmu_cmdq_exclusive_trylock_irqsave (cmdq, flags)) { +WRITE_ONCE (cmdq->q.llq.cons,readl_relaxed (cmdq->q.cons_reg)); +arm_smmu_cmdq_exclusive_unlock_irqrestore (cmdq, flags); + llq->val =READ_ONCE (cmdq->q.llq.val); +return 0 ; } -spin_lock_irqsave (&smmu->cmdq.lock, flags); -arm_smmu_cmdq_insert_cmd (smmu, cmd); -spin_unlock_irqrestore (&smmu->cmdq.lock, flags); +queue_poll_init (smmu, &qp); +do { + llq->val =READ_ONCE (smmu->cmdq.q.llq.val); +if (!queue_full (llq)) +break ; + + ret =queue_poll (&qp); + }while (!ret); + +return ret; }/* -* The difference between val and sync_idx is bounded by the maximum size of -* a queue at 2^20 entries, so 32 bits is plenty for wrap-safe arithmetic. +* Wait until the SMMU signals a CMD_SYNC completion MSI. +* Must be called with the cmdq lock held in some capacity. */ -static int __arm_smmu_sync_poll_msi (struct arm_smmu_device *smmu,u32 sync_idx) +static int __arm_smmu_cmdq_poll_until_msi (struct arm_smmu_device *smmu, +struct arm_smmu_ll_queue *llq) { -ktime_t timeout; -u32 val; +int ret =0 ; +struct arm_smmu_queue_poll qp; +struct arm_smmu_cmdq *cmdq = &smmu->cmdq; +u32 *cmd = (u32 *)(Q_ENT (&cmdq->q, llq->prod)); - timeout =ktime_add_us (ktime_get (),ARM_SMMU_CMDQ_SYNC_TIMEOUT_US ); - val =smp_cond_load_acquire (&smmu->sync_count, - (int )(VAL - sync_idx) >=0 || - !ktime_before (ktime_get (), timeout)); +queue_poll_init (smmu, &qp); -return (int )(val - sync_idx) <0 ? -ETIMEDOUT :0 ; +/* +* The MSI won't generate an event, since it's being written back +* into the command queue. +*/ + qp.wfe = false; +smp_cond_load_relaxed (cmd, !VAL || (ret =queue_poll (&qp))); + llq->cons = ret ? llq->prod :queue_inc_prod_n (llq,1 ); +return ret; } -static int __arm_smmu_cmdq_issue_sync_msi (struct arm_smmu_device *smmu) +/* +* Wait until the SMMU cons index passes llq->prod. +* Must be called with the cmdq lock held in some capacity. +*/ +static int __arm_smmu_cmdq_poll_until_consumed (struct arm_smmu_device *smmu, +struct arm_smmu_ll_queue *llq) { -u64 cmd[CMDQ_ENT_DWORDS ]; -unsigned long flags; -struct arm_smmu_cmdq_ent ent = { - .opcode =CMDQ_OP_CMD_SYNC , - .sync = { - .msiaddr =virt_to_phys (&smmu->sync_count), - }, - }; +struct arm_smmu_queue_poll qp; +struct arm_smmu_cmdq *cmdq = &smmu->cmdq; +u32 prod = llq->prod; +int ret =0 ; -spin_lock_irqsave (&smmu->cmdq.lock, flags); +queue_poll_init (smmu, &qp); + llq->val =READ_ONCE (smmu->cmdq.q.llq.val); +do { +if (queue_consumed (llq, prod)) +break ; -/* Piggy-back on the previous command if it's a SYNC */ -if (smmu->prev_cmd_opcode ==CMDQ_OP_CMD_SYNC ) { - ent.sync.msidata = smmu->sync_nr; - }else { - ent.sync.msidata = ++smmu->sync_nr; -arm_smmu_cmdq_build_cmd (cmd, &ent); -arm_smmu_cmdq_insert_cmd (smmu, cmd); - } + ret =queue_poll (&qp); -spin_unlock_irqrestore (&smmu->cmdq.lock, flags); +/* +* This needs to be a readl() so that our subsequent call +* to arm_smmu_cmdq_shared_tryunlock() can fail accurately. +* +* Specifically, we need to ensure that we observe all +* shared_lock()s by other CMD_SYNCs that share our owner, +* so that a failing call to tryunlock() means that we're +* the last one out and therefore we can safely advance +* cmdq->q.llq.cons. Roughly speaking: +* +* CPU 0 CPU1 CPU2 (us) +* +* if (sync) +* shared_lock(); +* +* dma_wmb(); +* set_valid_map(); +* +* if (owner) { +* poll_valid_map(); +* <control dependency> +* writel(prod_reg); +* +* readl(cons_reg); +* tryunlock(); +* +* Requires us to see CPU 0's shared_lock() acquisition. +*/ + llq->cons =readl (cmdq->q.cons_reg); + }while (!ret); -return __arm_smmu_sync_poll_msi (smmu, ent.sync.msidata); +return ret; } -static int __arm_smmu_cmdq_issue_sync (struct arm_smmu_device *smmu) +static int arm_smmu_cmdq_poll_until_sync (struct arm_smmu_device *smmu, +struct arm_smmu_ll_queue *llq) { -u64 cmd[CMDQ_ENT_DWORDS ]; +if (smmu->features &ARM_SMMU_FEAT_MSI && + smmu->features &ARM_SMMU_FEAT_COHERENCY ) +return __arm_smmu_cmdq_poll_until_msi (smmu, llq); + +return __arm_smmu_cmdq_poll_until_consumed (smmu, llq); +} + +static void arm_smmu_cmdq_write_entries (struct arm_smmu_cmdq *cmdq,u64 *cmds, +u32 prod,int n) +{ +int i; +struct arm_smmu_ll_queue llq = { + .max_n_shift = cmdq->q.llq.max_n_shift, + .prod = prod, + }; + +for (i =0 ; i < n; ++i) { +u64 *cmd = &cmds[i *CMDQ_ENT_DWORDS ]; + + prod =queue_inc_prod_n (&llq, i); +queue_write (Q_ENT (&cmdq->q, prod), cmd,CMDQ_ENT_DWORDS ); + } +} + +static int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +u64 *cmds,int n,bool sync) +{ +u64 cmd_sync[CMDQ_ENT_DWORDS ]; +u32 prod;unsigned long flags; -bool wfe = !!(smmu->features &ARM_SMMU_FEAT_SEV ); -struct arm_smmu_cmdq_ent ent = { .opcode =CMDQ_OP_CMD_SYNC }; -int ret; +bool owner; +struct arm_smmu_cmdq *cmdq = &smmu->cmdq; +struct arm_smmu_ll_queue llq = { + .max_n_shift = cmdq->q.llq.max_n_shift, + }, head = llq; +int ret =0 ; -arm_smmu_cmdq_build_cmd (cmd, &ent); +/* 1. Allocate some space in the queue */ +local_irq_save (flags); + llq.val =READ_ONCE (cmdq->q.llq.val); +do { +u64 old; + +while (!queue_has_space (&llq, n + sync)) { +local_irq_restore (flags); +if (arm_smmu_cmdq_poll_until_not_full (smmu, &llq)) +dev_err_ratelimited (smmu->dev,"CMDQ timeout\n" ); +local_irq_save (flags); + } + + head.cons = llq.cons; + head.prod =queue_inc_prod_n (&llq, n + sync) | +CMDQ_PROD_OWNED_FLAG ; + + old =cmpxchg_relaxed (&cmdq->q.llq.val, llq.val, head.val); +if (old == llq.val) +break ; + + llq.val = old; + }while (1 ); + owner = !(llq.prod &CMDQ_PROD_OWNED_FLAG ); + +/* +* 2. Write our commands into the queue +* Dependency ordering from the cmpxchg() loop above. +*/ +arm_smmu_cmdq_write_entries (cmdq, cmds, llq.prod, n); +if (sync) { + prod =queue_inc_prod_n (&llq, n); +arm_smmu_cmdq_build_sync_cmd (cmd_sync, smmu, prod); +queue_write (Q_ENT (&cmdq->q, prod), cmd_sync,CMDQ_ENT_DWORDS ); + +/* +* In order to determine completion of our CMD_SYNC, we must +* ensure that the queue can't wrap twice without us noticing. +* We achieve that by taking the cmdq lock as shared before +* marking our slot as valid. +*/ +arm_smmu_cmdq_shared_lock (cmdq); + } + +/* 3. Mark our slots as valid, ensuring commands are visible first */ +dma_wmb (); + prod =queue_inc_prod_n (&llq, n + sync); +arm_smmu_cmdq_set_valid_map (cmdq, llq.prod, prod); + +/* 4. If we are the owner, take control of the SMMU hardware */ +if (owner) { +/* a. Wait for previous owner to finish */ +atomic_cond_read_relaxed (&cmdq->owner_prod,VAL == llq.prod); + +/* b. Stop gathering work by clearing the owned flag */ + prod =atomic_fetch_andnot_relaxed (CMDQ_PROD_OWNED_FLAG , + &cmdq->q.llq.atomic.prod); + prod &= ~CMDQ_PROD_OWNED_FLAG ; + head.prod &= ~CMDQ_PROD_OWNED_FLAG ; + +/* c. Wait for any gathered work to be written to the queue */ +arm_smmu_cmdq_poll_valid_map (cmdq, head.prod, prod); + +/* +* d. Advance the hardware prod pointer +* Control dependency ordering from the entries becoming valid. +*/ +writel_relaxed (prod, cmdq->q.prod_reg); -spin_lock_irqsave (&smmu->cmdq.lock, flags); -arm_smmu_cmdq_insert_cmd (smmu, cmd); - ret =queue_poll_cons (&smmu->cmdq.q, true, wfe); -spin_unlock_irqrestore (&smmu->cmdq.lock, flags); +/* +* e. Tell the next owner we're done +* Make sure we've updated the hardware first, so that we don't +* race to update prod and potentially move it backwards. +*/ +atomic_set_release (&cmdq->owner_prod, prod); + } +/* 5. If we are inserting a CMD_SYNC, we must wait for it to complete */ +if (sync) { + llq.prod =queue_inc_prod_n (&llq, n); + ret =arm_smmu_cmdq_poll_until_sync (smmu, &llq); +if (ret) { +dev_err_ratelimited (smmu->dev, +"CMD_SYNC timeout at 0x%08x [hwprod 0x%08x, hwcons 0x%08x]\n" , + llq.prod, +readl_relaxed (cmdq->q.prod_reg), +readl_relaxed (cmdq->q.cons_reg)); + } + +/* +* Try to unlock the cmq lock. This will fail if we're the last +* reader, in which case we can safely update cmdq->q.llq.cons +*/ +if (!arm_smmu_cmdq_shared_tryunlock (cmdq)) { +WRITE_ONCE (cmdq->q.llq.cons, llq.cons); +arm_smmu_cmdq_shared_unlock (cmdq); + } + } + +local_irq_restore (flags);return ret; } -static int arm_smmu_cmdq_issue_sync (struct arm_smmu_device *smmu) +static int arm_smmu_cmdq_issue_cmd (struct arm_smmu_device *smmu, +struct arm_smmu_cmdq_ent *ent) { -int ret; -bool msi = (smmu->features &ARM_SMMU_FEAT_MSI ) && - (smmu->features &ARM_SMMU_FEAT_COHERENCY ); +u64 cmd[CMDQ_ENT_DWORDS ]; - ret = msi ?__arm_smmu_cmdq_issue_sync_msi (smmu) - :__arm_smmu_cmdq_issue_sync (smmu); -if (ret) -dev_err_ratelimited (smmu->dev,"CMD_SYNC timeout\n" ); -return ret; +if (arm_smmu_cmdq_build_cmd (cmd, ent)) { +dev_warn (smmu->dev,"ignoring unknown CMDQ opcode 0x%x\n" , + ent->opcode); +return -EINVAL ; + } + +return arm_smmu_cmdq_issue_cmdlist (smmu, cmd,1 , false); +} + +static int arm_smmu_cmdq_issue_sync (struct arm_smmu_device *smmu) +{ +return arm_smmu_cmdq_issue_cmdlist (smmu,NULL ,0 , true); }/* Context descriptor manipulation functions */
@@ -1574,9 +1922,9 @@ static void arm_smmu_tlb_inv_context(void *cookie)/* * NOTE: when io-pgtable is in non-strict mode, we may get here with * PTEs previously cleared by unmaps on the current CPU not yet visible -* to the SMMU. We are relying on the DSB implicit in -* queue_sync_prod_out() to guarantee those are observed before the -* TLBI. Do be careful, 007. +* to the SMMU. We are relying on the dma_wmb() implicit during cmd +* insertion to guarantee those are observed before the TLBI. Do be +* careful, 007. */ arm_smmu_cmdq_issue_cmd (smmu, &cmd);arm_smmu_cmdq_issue_sync (smmu);
@@ -2350,18 +2698,49 @@ static int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,return 0 ; } +static void arm_smmu_cmdq_free_bitmap (void *data) +{ +unsigned long *bitmap = data; +bitmap_free (bitmap); +} + +static int arm_smmu_cmdq_init (struct arm_smmu_device *smmu) +{ +int ret =0 ; +struct arm_smmu_cmdq *cmdq = &smmu->cmdq; +unsigned int nents =1 << cmdq->q.llq.max_n_shift; +atomic_long_t *bitmap; + +atomic_set (&cmdq->owner_prod,0 ); +atomic_set (&cmdq->lock,0 ); + + bitmap = (atomic_long_t *)bitmap_zalloc (nents,GFP_KERNEL ); +if (!bitmap) { +dev_err (smmu->dev,"failed to allocate cmdq bitmap\n" ); + ret = -ENOMEM ; + }else { + cmdq->valid_map = bitmap; +devm_add_action (smmu->dev, arm_smmu_cmdq_free_bitmap, bitmap); + } + +return ret; +} +static int arm_smmu_init_queues (struct arm_smmu_device *smmu) {int ret;/* cmdq */ -spin_lock_init (&smmu->cmdq.lock); ret =arm_smmu_init_one_queue (smmu, &smmu->cmdq.q,ARM_SMMU_CMDQ_PROD ,ARM_SMMU_CMDQ_CONS ,CMDQ_ENT_DWORDS ,"cmdq" );if (ret)return ret; + ret =arm_smmu_cmdq_init (smmu); +if (ret) +return ret; +/* evtq */ ret = arm_smmu_init_one_queue(smmu, &smmu->evtq.q, ARM_SMMU_EVTQ_PROD, ARM_SMMU_EVTQ_CONS, EVTQ_ENT_DWORDS,
@@ -2942,9 +3321,15 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)/* Queue sizes, capped to ensure natural alignment */ smmu->cmdq.q.llq.max_n_shift =min_t (u32,CMDQ_MAX_SZ_SHIFT ,FIELD_GET (IDR1_CMDQS , reg)); -if (!smmu->cmdq.q.llq.max_n_shift) { -/* Odd alignment restrictions on the base, so ignore for now */ -dev_err (smmu->dev,"unit-length command queue not supported\n" ); +if (smmu->cmdq.q.llq.max_n_shift <ilog2 (BITS_PER_LONG )) { +/* +* The cmdq valid_map relies on the total number of entries +* being a multiple of BITS_PER_LONG. There's also no way +* we can handle the weird alignment restrictions on the +* base pointer for a unit-length queue. +*/ +dev_err (smmu->dev,"command queue size < %d entries not supported\n" , +BITS_PER_LONG );return -ENXIO ; }
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH v2 18/19] iommu/arm-smmu-v3: Reduce contention during command-queue insertion
Quoted 67 lines
drivers/iommu/arm-smmu-v3.c +61
-4+static int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +u64 *cmds,int n,bool sync) +{ +u64 cmd_sync[CMDQ_ENT_DWORDS ]; +u32 prod;unsigned long flags; -bool wfe = !!(smmu->features &ARM_SMMU_FEAT_SEV ); -struct arm_smmu_cmdq_ent ent = { .opcode =CMDQ_OP_CMD_SYNC }; -int ret; +bool owner; +struct arm_smmu_cmdq *cmdq = &smmu->cmdq; +struct arm_smmu_ll_queue llq = { + .max_n_shift = cmdq->q.llq.max_n_shift, + }, head = llq; +int ret =0 ; -arm_smmu_cmdq_build_cmd (cmd, &ent); +/* 1. Allocate some space in the queue */ +local_irq_save (flags); + llq.val =READ_ONCE (cmdq->q.llq.val); +do { +u64 old; + +while (!queue_has_space (&llq, n + sync)) { +local_irq_restore (flags); +if (arm_smmu_cmdq_poll_until_not_full (smmu, &llq)) +dev_err_ratelimited (smmu->dev,"CMDQ timeout\n" ); +local_irq_save (flags); + } + + head.cons = llq.cons; + head.prod =queue_inc_prod_n (&llq, n + sync) | +CMDQ_PROD_OWNED_FLAG ; + + old =cmpxchg_relaxed (&cmdq->q.llq.val, llq.val, head.val); +if (old == llq.val) +break ; + + llq.val = old; + }while (1 ); + owner = !(llq.prod &CMDQ_PROD_OWNED_FLAG ); + +/* +* 2. Write our commands into the queue +* Dependency ordering from the cmpxchg() loop above. +*/ +arm_smmu_cmdq_write_entries (cmdq, cmds, llq.prod, n); +if (sync) { + prod =queue_inc_prod_n (&llq, n); +arm_smmu_cmdq_build_sync_cmd (cmd_sync, smmu, prod); +queue_write (Q_ENT (&cmdq->q, prod), cmd_sync,CMDQ_ENT_DWORDS ); + +/* +* In order to determine completion of our CMD_SYNC, we must +* ensure that the queue can't wrap twice without us noticing. +* We achieve that by taking the cmdq lock as shared before +* marking our slot as valid. +*/ +arm_smmu_cmdq_shared_lock (cmdq); + } + +/* 3. Mark our slots as valid, ensuring commands are visible first */ +dma_wmb (); + prod =queue_inc_prod_n (&llq, n + sync); +arm_smmu_cmdq_set_valid_map (cmdq, llq.prod, prod); + +/* 4. If we are the owner, take control of the SMMU hardware */
drivers/iommu/arm-smmu-v3.c static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu) · line 1348
Quoted 10 lines
drivers/iommu/arm-smmu-v3.c +10
-0+if (owner) { +/* a. Wait for previous owner to finish */ +atomic_cond_read_relaxed (&cmdq->owner_prod,VAL == llq.prod); + +/* b. Stop gathering work by clearing the owned flag */ + prod =atomic_fetch_andnot_relaxed (CMDQ_PROD_OWNED_FLAG , + &cmdq->q.llq.atomic.prod); + prod &= ~CMDQ_PROD_OWNED_FLAG ; + head.prod &= ~CMDQ_PROD_OWNED_FLAG ; +
Hi Will,
Could it be a minor optimisation to advance the HW producer pointer at this stage for the owner only? We know that its entries are written, and it should be first in the new batch of commands (right?), so we could advance the pointer to at least get the HW started.
Cheers,
Quoted 37 lines
drivers/iommu/arm-smmu-v3.c +30
-4+/* c. Wait for any gathered work to be written to the queue */ +arm_smmu_cmdq_poll_valid_map (cmdq, head.prod, prod); + +/* +* d. Advance the hardware prod pointer +* Control dependency ordering from the entries becoming valid. +*/ +writel_relaxed (prod, cmdq->q.prod_reg); -spin_lock_irqsave (&smmu->cmdq.lock, flags); -arm_smmu_cmdq_insert_cmd (smmu, cmd); - ret =queue_poll_cons (&smmu->cmdq.q, true, wfe); -spin_unlock_irqrestore (&smmu->cmdq.lock, flags); +/* +* e. Tell the next owner we're done +* Make sure we've updated the hardware first, so that we don't +* race to update prod and potentially move it backwards. +*/ +atomic_set_release (&cmdq->owner_prod, prod); + } +/* 5. If we are inserting a CMD_SYNC, we must wait for it to complete */ +if (sync) { + llq.prod =queue_inc_prod_n (&llq, n); + ret =arm_smmu_cmdq_poll_until_sync (smmu, &llq); +if (ret) { +dev_err_ratelimited (smmu->dev, +"CMD_SYNC timeout at 0x%08x [hwprod 0x%08x, hwcons 0x%08x]\n" , + llq.prod, +readl_relaxed (cmdq->q.prod_reg), +readl_relaxed (cmdq->q.cons_reg)); + } + +/* +* Try to unlock the cmq lock. This will fail if we're the last +* reader, in which case we can safely update cmdq->q.llq.cons
drivers/iommu/arm-smmu-v3.c static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu) · line 1390
Quoted 10 lines
drivers/iommu/arm-smmu-v3.c +8
-0+*/ +if (!arm_smmu_cmdq_shared_tryunlock (cmdq)) { +WRITE_ONCE (cmdq->q.llq.cons, llq.cons); +arm_smmu_cmdq_shared_unlock (cmdq); + } + } + +local_irq_restore (flags);return ret; }
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRe: [RFC PATCH v2 18/19] iommu/arm-smmu-v3: Reduce contention during command-queue insertion
Hi John,
Thanks for reading the code!
Quoted 74 lines · John Garry
Quoted 74 lines
drivers/iommu/arm-smmu-v3.c +68
-4+static int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +u64 *cmds,int n,bool sync) +{ +u64 cmd_sync[CMDQ_ENT_DWORDS ]; +u32 prod;unsigned long flags; -bool wfe = !!(smmu->features &ARM_SMMU_FEAT_SEV ); -struct arm_smmu_cmdq_ent ent = { .opcode =CMDQ_OP_CMD_SYNC }; -int ret; +bool owner; +struct arm_smmu_cmdq *cmdq = &smmu->cmdq; +struct arm_smmu_ll_queue llq = { + .max_n_shift = cmdq->q.llq.max_n_shift, + }, head = llq; +int ret =0 ; -arm_smmu_cmdq_build_cmd (cmd, &ent); +/* 1. Allocate some space in the queue */ +local_irq_save (flags); + llq.val =READ_ONCE (cmdq->q.llq.val); +do { +u64 old; + +while (!queue_has_space (&llq, n + sync)) { +local_irq_restore (flags); +if (arm_smmu_cmdq_poll_until_not_full (smmu, &llq)) +dev_err_ratelimited (smmu->dev,"CMDQ timeout\n" ); +local_irq_save (flags); + } + + head.cons = llq.cons; + head.prod =queue_inc_prod_n (&llq, n + sync) | +CMDQ_PROD_OWNED_FLAG ; + + old =cmpxchg_relaxed (&cmdq->q.llq.val, llq.val, head.val); +if (old == llq.val) +break ; + + llq.val = old; + }while (1 ); + owner = !(llq.prod &CMDQ_PROD_OWNED_FLAG ); + +/* +* 2. Write our commands into the queue +* Dependency ordering from the cmpxchg() loop above. +*/ +arm_smmu_cmdq_write_entries (cmdq, cmds, llq.prod, n); +if (sync) { + prod =queue_inc_prod_n (&llq, n); +arm_smmu_cmdq_build_sync_cmd (cmd_sync, smmu, prod); +queue_write (Q_ENT (&cmdq->q, prod), cmd_sync,CMDQ_ENT_DWORDS ); + +/* +* In order to determine completion of our CMD_SYNC, we must +* ensure that the queue can't wrap twice without us noticing. +* We achieve that by taking the cmdq lock as shared before +* marking our slot as valid. +*/ +arm_smmu_cmdq_shared_lock (cmdq); + } + +/* 3. Mark our slots as valid, ensuring commands are visible first */ +dma_wmb (); + prod =queue_inc_prod_n (&llq, n + sync); +arm_smmu_cmdq_set_valid_map (cmdq, llq.prod, prod); + +/* 4. If we are the owner, take control of the SMMU hardware */ +if (owner) { +/* a. Wait for previous owner to finish */ +atomic_cond_read_relaxed (&cmdq->owner_prod,VAL == llq.prod); + +/* b. Stop gathering work by clearing the owned flag */ + prod =atomic_fetch_andnot_relaxed (CMDQ_PROD_OWNED_FLAG , + &cmdq->q.llq.atomic.prod);
Quoted 8 lines
drivers/iommu/arm-smmu-v3.c static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu) · line 1348
Quoted 3 lines
drivers/iommu/arm-smmu-v3.c +3
-0+ prod &= ~CMDQ_PROD_OWNED_FLAG ; + head.prod &= ~CMDQ_PROD_OWNED_FLAG ; +Could it be a minor optimisation to advance the HW producer pointer at this stage for the owner only? We know that its entries are written, and it should be first in the new batch of commands (right?), so we could advance the pointer to at least get the HW started.
I think that would be a valid thing to do, but it depends on the relative cost of writing to prod compared to how long we're likely to wait. Given that everybody has irqs disabled when writing out their commands, I wouldn't expect the waiting to be a big issue, although we could probably optimise arm_smmu_cmdq_write_entries() into a memcpy() if we needed to.
In other words, I think we need numbers to justify that change.
Thanks,
Will
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH v2 18/19] iommu/arm-smmu-v3: Reduce contention during command-queue insertion
Quoted 88 lines · Will Deacon
Hi John,
Thanks for reading the code!
Quoted 74 lines · John Garry
Quoted 74 lines
drivers/iommu/arm-smmu-v3.c +68
-4+static int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +u64 *cmds,int n,bool sync) +{ +u64 cmd_sync[CMDQ_ENT_DWORDS ]; +u32 prod;unsigned long flags; -bool wfe = !!(smmu->features &ARM_SMMU_FEAT_SEV ); -struct arm_smmu_cmdq_ent ent = { .opcode =CMDQ_OP_CMD_SYNC }; -int ret; +bool owner; +struct arm_smmu_cmdq *cmdq = &smmu->cmdq; +struct arm_smmu_ll_queue llq = { + .max_n_shift = cmdq->q.llq.max_n_shift, + }, head = llq; +int ret =0 ; -arm_smmu_cmdq_build_cmd (cmd, &ent); +/* 1. Allocate some space in the queue */ +local_irq_save (flags); + llq.val =READ_ONCE (cmdq->q.llq.val); +do { +u64 old; + +while (!queue_has_space (&llq, n + sync)) { +local_irq_restore (flags); +if (arm_smmu_cmdq_poll_until_not_full (smmu, &llq)) +dev_err_ratelimited (smmu->dev,"CMDQ timeout\n" ); +local_irq_save (flags); + } + + head.cons = llq.cons; + head.prod =queue_inc_prod_n (&llq, n + sync) | +CMDQ_PROD_OWNED_FLAG ; + + old =cmpxchg_relaxed (&cmdq->q.llq.val, llq.val, head.val); +if (old == llq.val) +break ; + + llq.val = old; + }while (1 ); + owner = !(llq.prod &CMDQ_PROD_OWNED_FLAG ); + +/* +* 2. Write our commands into the queue +* Dependency ordering from the cmpxchg() loop above. +*/ +arm_smmu_cmdq_write_entries (cmdq, cmds, llq.prod, n); +if (sync) { + prod =queue_inc_prod_n (&llq, n); +arm_smmu_cmdq_build_sync_cmd (cmd_sync, smmu, prod); +queue_write (Q_ENT (&cmdq->q, prod), cmd_sync,CMDQ_ENT_DWORDS ); + +/* +* In order to determine completion of our CMD_SYNC, we must +* ensure that the queue can't wrap twice without us noticing. +* We achieve that by taking the cmdq lock as shared before +* marking our slot as valid. +*/ +arm_smmu_cmdq_shared_lock (cmdq); + } + +/* 3. Mark our slots as valid, ensuring commands are visible first */ +dma_wmb (); + prod =queue_inc_prod_n (&llq, n + sync); +arm_smmu_cmdq_set_valid_map (cmdq, llq.prod, prod); + +/* 4. If we are the owner, take control of the SMMU hardware */ +if (owner) { +/* a. Wait for previous owner to finish */ +atomic_cond_read_relaxed (&cmdq->owner_prod,VAL == llq.prod); + +/* b. Stop gathering work by clearing the owned flag */ + prod =atomic_fetch_andnot_relaxed (CMDQ_PROD_OWNED_FLAG , + &cmdq->q.llq.atomic.prod);Quoted 8 lines
drivers/iommu/arm-smmu-v3.c static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu) · line 1348
Quoted 3 lines
drivers/iommu/arm-smmu-v3.c +3
-0+ prod &= ~CMDQ_PROD_OWNED_FLAG ; + head.prod &= ~CMDQ_PROD_OWNED_FLAG ; +Could it be a minor optimisation to advance the HW producer pointer at this stage for the owner only? We know that its entries are written, and it should be first in the new batch of commands (right?), so we could advance the pointer to at least get the HW started.
Quoted 4 lines
I think that would be a valid thing to do, but it depends on the relative cost of writing to prod compared to how long we're likely to wait. Given that everybody has irqs disabled when writing out their commands, I wouldn't expect the waiting to be a big issue,
For sure, but I'm thinking of the possible scenario where the the guy(s) we're waiting on have many more commands. Or they just joined the current gathering quite late, just prior to clearing the owner flag.
although we could probably optimise
Quoted 3 lines
arm_smmu_cmdq_write_entries()into amemcpy()if we needed to.In other words, I think we need numbers to justify that change.
Anyway, this is quite minor, and I will see if the change could be justified by numbers.
Cheers,
John
Quoted 7 lines
Thanks,
Will
.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRe: [RFC PATCH v2 18/19] iommu/arm-smmu-v3: Reduce contention during command-queue insertion
Quoted 6 lines · John Garry
Quoted 5 lines · Will Deacon
Quoted 4 lines
Could it be a minor optimisation to advance the HW producer pointer at this stage for the owner only? We know that its entries are written, and it should be first in the new batch of commands (right?), so we could advance the pointer to at least get the HW started.
Quoted 8 lines
Quoted 4 lines
I think that would be a valid thing to do, but it depends on the relative cost of writing to prod compared to how long we're likely to wait. Given that everybody has irqs disabled when writing out their commands, I wouldn't expect the waiting to be a big issue,
For sure, but I'm thinking of the possible scenario where the the guy(s) we're waiting on have many more commands. Or they just joined the current gathering quite late, just prior to clearing the owner flag.
Understood, but a "cacheable" memcpy (assuming the SMMU is coherent) should be pretty quick, even for maximum batch size I think.
Quoted 7 lines
although we could probably optimise
Quoted 3 lines
arm_smmu_cmdq_write_entries()into amemcpy()if we needed to.In other words, I think we need numbers to justify that change.
Anyway, this is quite minor, and I will see if the change could be justified by numbers.
Thanks! If the numbers show it's useful, we can definitely add it.
Will
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH v2 18/19] iommu/arm-smmu-v3: Reduce contention during command-queue insertion
In reply to Will DeaconQuoted 25 lines · Will Deacon
drivers/iommu/arm-smmu-v3.c +19
-4+static int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +u64 *cmds,int n,bool sync) +{ +u64 cmd_sync[CMDQ_ENT_DWORDS ]; +u32 prod;unsigned long flags; -bool wfe = !!(smmu->features &ARM_SMMU_FEAT_SEV ); -struct arm_smmu_cmdq_ent ent = { .opcode =CMDQ_OP_CMD_SYNC }; -int ret; +bool owner; +struct arm_smmu_cmdq *cmdq = &smmu->cmdq; +struct arm_smmu_ll_queue llq = { + .max_n_shift = cmdq->q.llq.max_n_shift, + }, head = llq; +int ret =0 ; -arm_smmu_cmdq_build_cmd (cmd, &ent); +/* 1. Allocate some space in the queue */ +local_irq_save (flags); + llq.val =READ_ONCE (cmdq->q.llq.val); + do { + u64 old; + + while (!queue_has_space(&llq, n + sync)) { + local_irq_restore(flags);
drivers/iommu/arm-smmu-v3.c static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu) · line 1306
Quoted 10 lines
drivers/iommu/arm-smmu-v3.c +10
-0+if (arm_smmu_cmdq_poll_until_not_full (smmu, &llq)) +dev_err_ratelimited (smmu->dev,"CMDQ timeout\n" ); +local_irq_save (flags); + } + + head.cons = llq.cons; + head.prod =queue_inc_prod_n (&llq, n + sync) | +CMDQ_PROD_OWNED_FLAG ; + + old =cmpxchg_relaxed (&cmdq->q.llq.val, llq.val, head.val);
I added some basic debug to the stress test on your branch, and this cmpxchg was failing ~10 times on average on my D06.
So we're not using the spinlock now, but this cmpxchg may lack fairness.
Since we're batching commands, I wonder if it's better to restore the spinlock and send batched commands + CMD_SYNC under the lock, and then wait for the CMD_SYNC completion outside the lock.
I don't know if it improves the queue contetion, but at least the prod pointer would be more closely track the issued commands, such that we're not waiting to kick off many gathered batches of commands, while the SMMU HW may be idle (in terms of command processing).
Cheers,
John
drivers/iommu/arm-smmu-v3.c static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu)
Quoted 6 lines
drivers/iommu/arm-smmu-v3.c +6
-0+if (old == llq.val) +break ; + + llq.val = old; + }while (1 ); + owner = !(llq.prod & CMDQ_PROD_OWNED_F
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRe: [RFC PATCH v2 18/19] iommu/arm-smmu-v3: Reduce contention during command-queue insertion
Hi again, John,
Quoted 40 lines · John Garry
Quoted 25 lines · Will Deacon
drivers/iommu/arm-smmu-v3.c +19
-4+static int arm_smmu_cmdq_issue_cmdlist (struct arm_smmu_device *smmu, +u64 *cmds,int n,bool sync) +{ +u64 cmd_sync[CMDQ_ENT_DWORDS ]; +u32 prod;unsigned long flags; -bool wfe = !!(smmu->features &ARM_SMMU_FEAT_SEV ); -struct arm_smmu_cmdq_ent ent = { .opcode =CMDQ_OP_CMD_SYNC }; -int ret; +bool owner; +struct arm_smmu_cmdq *cmdq = &smmu->cmdq; +struct arm_smmu_ll_queue llq = { + .max_n_shift = cmdq->q.llq.max_n_shift, + }, head = llq; +int ret =0 ; -arm_smmu_cmdq_build_cmd (cmd, &ent); +/* 1. Allocate some space in the queue */ +local_irq_save (flags); + llq.val =READ_ONCE (cmdq->q.llq.val); + do { + u64 old; + + while (!queue_has_space(&llq, n + sync)) { + local_irq_restore(flags);drivers/iommu/arm-smmu-v3.c static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu) · line 1306
Quoted 10 lines
drivers/iommu/arm-smmu-v3.c +10
-0+if (arm_smmu_cmdq_poll_until_not_full (smmu, &llq)) +dev_err_ratelimited (smmu->dev,"CMDQ timeout\n" ); +local_irq_save (flags); + } + + head.cons = llq.cons; + head.prod =queue_inc_prod_n (&llq, n + sync) | +CMDQ_PROD_OWNED_FLAG ; + + old =cmpxchg_relaxed (&cmdq->q.llq.val, llq.val, head.val);I added some basic debug to the stress test on your branch, and this cmpxchg was failing ~10 times on average on my D06.
Quoted 1 line
So we're not using the spinlock now, but this cmpxchg may lack fairness.
It definitely lacks fairness, although that's going to be the case in many other places where locking is elided as well. If it shows up as an issue, we should try to address it, but queueing means serialisation and that largely defeats the point of this patch. I also don't expect it to be a problem unless the cmpxchg() is heavily contended, which shouldn't be the case if we're batching.
Quoted 3 lines
Since we're batching commands, I wonder if it's better to restore the spinlock and send batched commands +
CMD_SYNCunder the lock, and then wait for theCMD_SYNCcompletion outside the lock.
Again, we'd need some numbers, but my concern with that approach is that we're serialising CPUs which is what I've been trying hard to avoid. It also doesn't let you straightforwardly remove the cmpxchg() loop, since the owner clears the OWNED flag and you probably wouldn't want to hold the lock for that.
Quoted 4 lines
I don't know if it improves the queue contetion, but at least the prod pointer would be more closely track the issued commands, such that we're not waiting to kick off many gathered batches of commands, while the SMMU HW may be idle (in terms of command processing).
Again, probably going to need some numbers to change this, although it sounds like your other suggestion about having the owner move prod twice would largely address your concerns. Reintroducing the lock, on the other hand, feels like a big step backwards to me, and the whole reason I started down the current route was because of vague claims that the locking was a problem for large systems.
Thanks,
Will
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH v2 18/19] iommu/arm-smmu-v3: Reduce contention during command-queue insertion
Hi Will,
Quoted 14 lines
Quoted 6 lines
drivers/iommu/arm-smmu-v3.c static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu) · line 1306
Quoted 1 line
drivers/iommu/arm-smmu-v3.c +1
-0+ old =cmpxchg_relaxed (&cmdq->q.llq.val, llq.val, head.val);I added some basic debug to the stress test on your branch, and this cmpxchg was failing ~10 times on average on my D06.
So we're not using the spinlock now, but this cmpxchg may lack fairness.
It definitely lacks fairness, although that's going to be the case in many other places where locking is elided as well. If it shows up as an issue, we should try to address it, but queueing means serialisation and that largely defeats the point of this patch. I also don't expect it to be a problem unless the
cmpxchg()is heavily contended, which shouldn't be the case if we're batching.
Quoted 6 lines
Quoted 3 lines
Since we're batching commands, I wonder if it's better to restore the spinlock and send batched commands +
CMD_SYNCunder the lock, and then wait for theCMD_SYNCcompletion outside the lock.Again, we'd need some numbers, but my concern with that approach is that we're serialising CPUs which is what I've been trying hard to avoid.
That makes sense about the serialisation.
I'm just concerned with the scenario when the queue is heavily contented; here we may be gathering multiple batches of commands, and owners and non-owners a. may delay each other in writing the commands b. also pend the final CMD_SYNC consumption, which is for all gathered commands, and not just the commands which a particular CPU is interested in.
It
Quoted 4 lines
also doesn't let you straightforwardly remove the
cmpxchg()loop, since the owner clears the OWNED flag and you probably wouldn't want to hold the lock for that.
Quoted 8 lines
Quoted 4 lines
I don't know if it improves the queue contetion, but at least the prod pointer would be more closely track the issued commands, such that we're not waiting to kick off many gathered batches of commands, while the SMMU HW may be idle (in terms of command processing).
Again, probably going to need some numbers to change this, although it sounds like your other suggestion about having the owner move prod twice would largely address your concerns.
Limited initial testing shows that it makes little difference...
Reintroducing the lock, on the other
Quoted 3 lines
hand, feels like a big step backwards to me, and the whole reason I started down the current route was because of vague claims that the locking was a problem for large systems.
To me, the biggest improvement is batching of the commands, and it may not make much difference either way in how we deal with submitting them to the queue.
Cheers,
John
Quoted 7 lines
Thanks,
Will
.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerRFC PATCH v2 19/19 iommu/arm-smmu-v3: Defer TLB invalidation until ->iotlb_sync()
In reply to Will DeaconUpdate the iommu_iotlb_gather structure passed to ->tlb_add_page() and use this information to defer all TLB invalidation until ->iotlb_sync(). This drastically reduces contention on the command queue, since we can insert our commands in batches rather than one-by-one.
--- drivers/iommu/arm-smmu-v3.c | 71 +++++++++++++++++++++++++++------------------ 1 file changed, 42 insertions(+), 29 deletions(-)
drivers/iommu/arm-smmu-v3.c +42 -29
@@ -303,6 +303,13 @@#define CMDQ_PROD_OWNED_FLAG Q_OVERFLOW_FLAG +/* +* This is used to size the command queue and therefore must be at least +* BITS_PER_LONG so that the valid_map works correctly (it relies on the +* total number of queue entries being a multiple of BITS_PER_LONG). +*/ +#define CMDQ_BATCH_ENTRIES BITS_PER_LONG +#define CMDQ_0_OP GENMASK_ULL(7, 0)#define CMDQ_0_SSV (1UL << 11)
@@ -1930,15 +1937,17 @@ static void arm_smmu_tlb_inv_context(void *cookie)arm_smmu_cmdq_issue_sync (smmu); } -static void arm_smmu_tlb_inv_range_nosync (unsigned long iova,size_t size, -size_t granule,bool leaf,void *cookie) +static void arm_smmu_tlb_inv_range (unsigned long iova,size_t size, +size_t granule,bool leaf, +struct arm_smmu_domain *smmu_domain) { -struct arm_smmu_domain *smmu_domain = cookie; +u64 cmds[CMDQ_BATCH_ENTRIES *CMDQ_ENT_DWORDS ];struct arm_smmu_device *smmu = smmu_domain->smmu; +unsigned long end = iova + size; +int i =0 ;struct arm_smmu_cmdq_ent cmd = { .tlbi = { .leaf = leaf, - .addr = iova, }, };
@@ -1950,37 +1959,41 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size, cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; } -do { -arm_smmu_cmdq_issue_cmd (smmu, &cmd); - cmd.tlbi.addr += granule; - }while (size -= granule); +while (iova < end) { +if (i ==CMDQ_BATCH_ENTRIES ) { +arm_smmu_cmdq_issue_cmdlist (smmu, cmds, i, false); + i =0 ; + } + + cmd.tlbi.addr = iova; +arm_smmu_cmdq_build_cmd (&cmds[i *CMDQ_ENT_DWORDS ], &cmd); + iova += granule; + i++; + } + +arm_smmu_cmdq_issue_cmdlist (smmu, cmds, i, true); }static void arm_smmu_tlb_inv_page_nosync (struct iommu_iotlb_gather *gather,unsigned long iova,size_t granule,void *cookie) { -arm_smmu_tlb_inv_range_nosync (iova, granule, granule, true, cookie); +struct arm_smmu_domain *smmu_domain = cookie; +struct iommu_domain *domain = &smmu_domain->domain; + +iommu_iotlb_gather_add_page (domain, gather, iova, granule); }static void arm_smmu_tlb_inv_walk (unsigned long iova,size_t size,size_t granule,void *cookie) { -struct arm_smmu_domain *smmu_domain = cookie; -struct arm_smmu_device *smmu = smmu_domain->smmu; - -arm_smmu_tlb_inv_range_nosync (iova, size, granule, false, cookie); -arm_smmu_cmdq_issue_sync (smmu); +arm_smmu_tlb_inv_range (iova, size, granule, false, cookie); }static void arm_smmu_tlb_inv_leaf (unsigned long iova,size_t size,size_t granule,void *cookie) { -struct arm_smmu_domain *smmu_domain = cookie; -struct arm_smmu_device *smmu = smmu_domain->smmu; - -arm_smmu_tlb_inv_range_nosync (iova, size, granule, true, cookie); -arm_smmu_cmdq_issue_sync (smmu); +arm_smmu_tlb_inv_range (iova, size, granule, true, cookie); }static const struct iommu_flush_ops arm_smmu_flush_ops = {
@@ -2391,10 +2404,10 @@ static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain)static void arm_smmu_iotlb_sync (struct iommu_domain *domain,struct iommu_iotlb_gather *gather) { -struct arm_smmu_device *smmu =to_smmu_domain (domain)->smmu; +struct arm_smmu_domain *smmu_domain =to_smmu_domain (domain); -if (smmu) -arm_smmu_cmdq_issue_sync (smmu); +arm_smmu_tlb_inv_range (gather->start, gather->end - gather->start, + gather->pgsize, true, smmu_domain); } static phys_addr_t
@@ -3321,15 +3334,15 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)/* Queue sizes, capped to ensure natural alignment */ smmu->cmdq.q.llq.max_n_shift =min_t (u32,CMDQ_MAX_SZ_SHIFT ,FIELD_GET (IDR1_CMDQS , reg)); -if (smmu->cmdq.q.llq.max_n_shift <ilog2 (BITS_PER_LONG )) { +if (smmu->cmdq.q.llq.max_n_shift <=ilog2 (CMDQ_BATCH_ENTRIES )) {/* -* The cmdq valid_map relies on the total number of entries -* being a multiple of BITS_PER_LONG. There's also no way -* we can handle the weird alignment restrictions on the -* base pointer for a unit-length queue. +* We don't support splitting up batches, so one batch of +* commands plus an extra sync needs to fit inside the command +* queue. There's also no way we can handle the weird alignment +* restrictions on the base pointer for a unit-length queue. */ -dev_err (smmu->dev,"command queue size < %d entries not supported\n" , -BITS_PER_LONG ); +dev_err (smmu->dev,"command queue size <= %d entries not supported\n" , +CMDQ_BATCH_ENTRIES );return -ENXIO ; }
Signature · 6 lines
2.11.0 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Will,
Quoted 24 lines · Will Deacon
Hi everyone,
This is a significant rework of the RFC I previously posted here:
https://lkml.kernel.org/r/[email protected]
But this time, it looks like it might actually be worthwhile according to my perf profiles, where
__iommu_unmap()falls a long way down the profile for a multi-threaded netperf run. I'm still relying on others to confirm this is useful, however.Some of the changes since last time are:
* Support for constructing and submitting a list of commands in the driver
* Numerous changes to the IOMMU and io-pgtable APIs so that we can submit commands in batches
* Removal of
cmpxchg()fromcmdq_shared_lock()fast-path* Code restructuring and cleanups
Quoted 7 lines
This current applies against my iommu/devel branch that Joerg has pulled for 5.3. If you want to test it out, I've put everything here:
https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=iommu/cmdq
Feedback welcome. I appreciate that we're in the merge window, but I wanted to get this on the list for people to look at as an RFC.
I have tried branch iommu/cmdq on ThunderX2. I do see there is drastic reduction in CPU bandwidth consumption(from 15 to 20% to 1 to 2% in perf top) from SMMU CMDQ helper functions, when I run iperf with more than 64 clients(-P 64). However I have not noticed any measurable performance improvement in iperf results. IMO, this might/should help in performance improvement of IO intensive workloads.
FWIW, you can add,
Quoted 67 lines
Cheers,
Will
--->8
Cc Jean-Philippe Brucker <[email protected]>Cc Robin Murphy <[email protected]>Cc Jayachandran Chandrasekharan Nair <[email protected]>Cc Jan Glauber <[email protected]>Cc Jon Masters <[email protected]>Cc Eric Auger <[email protected]>Cc Zhen Lei <[email protected]>Cc Jonathan Cameron <[email protected]>Cc Vijay Kilary <[email protected]>Cc Joerg Roedel <[email protected]>Cc John Garry <[email protected]>Cc Alex Williamson <[email protected]>Will Deacon (19):
iommu: Remove emptyiommu_tlb_range_add()callback fromiommu_opsiommu/io-pgtable-arm: Remove redundant call toio_pgtable_tlb_sync()iommu/io-pgtable: Renameiommu_gather_opstoiommu_flush_ops
iommu: Introduce structiommu_iotlb_gatherfor batching TLB flushes
iommu: Introduceiommu_iotlb_gather_add_page()
iommu: Pass structiommu_iotlb_gatherto ->unmap() and ->iotlb_sync() iommu/io-pgtable: Introducetlb_flush_walk()andtlb_flush_leaf()iommu/io-pgtable: Hook up ->tlb_flush_walk() and ->tlb_flush_leaf() in
drivers
iommu/io-pgtable-arm: Call ->tlb_flush_walk() and ->tlb_flush_leaf() iommu/io-pgtable: Replace ->tlb_add_flush() with ->tlb_add_page() iommu/io-pgtable: Remove unused ->tlb_sync() callback iommu/io-pgtable: Pass structiommu_iotlb_gatherto ->unmap() iommu/io-pgtable: Pass structiommu_iotlb_gatherto ->tlb_add_page() iommu/arm-smmu-v3: Separate s/w and h/w views of prod and cons indexes iommu/arm-smmu-v3: Drop unused 'q' argument fromQ_OVFmacro iommu/arm-smmu-v3: Move low-level queue fields out ofarm_smmu_queueiommu/arm-smmu-v3: Operate directly on low-level queue where possible iommu/arm-smmu-v3: Reduce contention during command-queue insertion iommu/arm-smmu-v3: Defer TLB invalidation until ->iotlb_sync()drivers/gpu/drm/panfrost/panfrost_mmu.c | 24 +- drivers/iommu/amd_iommu.c | 11 +- drivers/iommu/arm-smmu-v3.c | 856 ++++++++++++++++++++++++-------- drivers/iommu/arm-smmu.c | 103 +++- drivers/iommu/dma-iommu.c | 9 +- drivers/iommu/exynos-iommu.c | 3 +- drivers/iommu/intel-iommu.c | 3 +- drivers/iommu/io-pgtable-arm-v7s.c | 57 +-- drivers/iommu/io-pgtable-arm.c | 48 +- drivers/iommu/iommu.c | 24 +- drivers/iommu/ipmmu-vmsa.c | 28 +- drivers/iommu/msm_iommu.c | 42 +- drivers/iommu/mtk_iommu.c | 45 +- drivers/iommu/mtk_iommu_v1.c | 3 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 44 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 3 +- drivers/iommu/tegra-gart.c | 12 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/vfio/vfio_iommu_type1.c | 27 +- include/linux/io-pgtable.h | 57 ++- include/linux/iommu.h | 92 +++- 23 files changed, 1090 insertions(+), 407 deletions(-)
Quoted 3 lines
--
2.11.0
Thanks,
Ganapat
Quoted 4 lines
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerQuoted 37 lines · Ganapatrao Kulkarni
Quoted 21 lines · Will Deacon
This is a significant rework of the RFC I previously posted here:
https://lkml.kernel.org/r/[email protected]
But this time, it looks like it might actually be worthwhile according to my perf profiles, where
__iommu_unmap()falls a long way down the profile for a multi-threaded netperf run. I'm still relying on others to confirm this is useful, however.Some of the changes since last time are:
* Support for constructing and submitting a list of commands in the driver
* Numerous changes to the IOMMU and io-pgtable APIs so that we can submit commands in batches
* Removal of
cmpxchg()fromcmdq_shared_lock()fast-path* Code restructuring and cleanups
Quoted 7 lines
This current applies against my iommu/devel branch that Joerg has pulled for 5.3. If you want to test it out, I've put everything here:
https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=iommu/cmdq
Feedback welcome. I appreciate that we're in the merge window, but I wanted to get this on the list for people to look at as an RFC.
I have tried branch iommu/cmdq on ThunderX2. I do see there is drastic reduction in CPU bandwidth consumption(from 15 to 20% to 1 to 2% in perf top) from SMMU CMDQ helper functions, when I run iperf with more than 64 clients(-P 64). However I have not noticed any measurable performance improvement in iperf results. IMO, this might/should help in performance improvement of IO intensive workloads.
Quoted 2 lines
FWIW, you can add,
Tested-by Ganapatrao Kulkarni <[email protected]>
Brilliant, thanks. Your measurements reflect mine in that I can saturate the NICs I have access to regardless of these changes, but the CPU time is drastically reduced.
Will
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 23 lines · Will Deacon
Hi everyone,
This is a significant rework of the RFC I previously posted here:
https://lkml.kernel.org/r/[email protected]
But this time, it looks like it might actually be worthwhile according to my perf profiles, where
__iommu_unmap()falls a long way down the profile for a multi-threaded netperf run. I'm still relying on others to confirm this is useful, however.Some of the changes since last time are:
* Support for constructing and submitting a list of commands in the driver
* Numerous changes to the IOMMU and io-pgtable APIs so that we can submit commands in batches
* Removal of
cmpxchg()fromcmdq_shared_lock()fast-path* Code restructuring and cleanups
Quoted 8 lines
This current applies against my iommu/devel branch that Joerg has pulled for 5.3. If you want to test it out, I've put everything here:
https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=iommu/cmdq
Feedback welcome. I appreciate that we're in the merge window, but I wanted to get this on the list for people to look at as an RFC.
I tested storage performance on this series, which I think is a better scenario to test than network performance, that being generally limited by the network link speed.
Results:
Baseline performance (will/iommu/devel, commit 9e6ea59f3)
8x SAS disks D05 839K IOPS
1x NVMe D05 454K IOPS
1x NVMe D06 442k IOPS
Patchset performance (will/iommu/cmdq)
8x SAS disk D05 835K IOPS
1x NVMe D05 472K IOPS
1x NVMe D06 459k IOPS
So we see a bit of an NVMe boost, but about the same for 8x disks. No iommu performance is about 918K IOPs for 8x disks, so it is not limited by the medium.
The D06 is a bit memory starved, so that may account for generally lower NVMe performance.
John
Quoted 57 lines
Cheers,
Will
--->8
Cc Jean-Philippe Brucker <[email protected]>Cc Robin Murphy <[email protected]>Cc Jayachandran Chandrasekharan Nair <[email protected]>Cc Jan Glauber <[email protected]>Cc Jon Masters <[email protected]>Cc Eric Auger <[email protected]>Cc Zhen Lei <[email protected]>Cc Jonathan Cameron <[email protected]>Cc Vijay Kilary <[email protected]>Cc Joerg Roedel <[email protected]>Cc John Garry <[email protected]>Cc Alex Williamson <[email protected]>Will Deacon (19):
iommu: Remove emptyiommu_tlb_range_add()callback fromiommu_opsiommu/io-pgtable-arm: Remove redundant call toio_pgtable_tlb_sync()iommu/io-pgtable: Renameiommu_gather_opstoiommu_flush_ops
iommu: Introduce structiommu_iotlb_gatherfor batching TLB flushes
iommu: Introduceiommu_iotlb_gather_add_page()
iommu: Pass structiommu_iotlb_gatherto ->unmap() and ->iotlb_sync() iommu/io-pgtable: Introducetlb_flush_walk()andtlb_flush_leaf()iommu/io-pgtable: Hook up ->tlb_flush_walk() and ->tlb_flush_leaf() in
drivers
iommu/io-pgtable-arm: Call ->tlb_flush_walk() and ->tlb_flush_leaf() iommu/io-pgtable: Replace ->tlb_add_flush() with ->tlb_add_page() iommu/io-pgtable: Remove unused ->tlb_sync() callback iommu/io-pgtable: Pass structiommu_iotlb_gatherto ->unmap() iommu/io-pgtable: Pass structiommu_iotlb_gatherto ->tlb_add_page() iommu/arm-smmu-v3: Separate s/w and h/w views of prod and cons indexes iommu/arm-smmu-v3: Drop unused 'q' argument fromQ_OVFmacro iommu/arm-smmu-v3: Move low-level queue fields out ofarm_smmu_queueiommu/arm-smmu-v3: Operate directly on low-level queue where possible iommu/arm-smmu-v3: Reduce contention during command-queue insertion iommu/arm-smmu-v3: Defer TLB invalidation until ->iotlb_sync()drivers/gpu/drm/panfrost/panfrost_mmu.c | 24 +- drivers/iommu/amd_iommu.c | 11 +- drivers/iommu/arm-smmu-v3.c | 856 ++++++++++++++++++++++++-------- drivers/iommu/arm-smmu.c | 103 +++- drivers/iommu/dma-iommu.c | 9 +- drivers/iommu/exynos-iommu.c | 3 +- drivers/iommu/intel-iommu.c | 3 +- drivers/iommu/io-pgtable-arm-v7s.c | 57 +-- drivers/iommu/io-pgtable-arm.c | 48 +- drivers/iommu/iommu.c | 24 +- drivers/iommu/ipmmu-vmsa.c | 28 +- drivers/iommu/msm_iommu.c | 42 +- drivers/iommu/mtk_iommu.c | 45 +- drivers/iommu/mtk_iommu_v1.c | 3 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 44 +-
Quoted 9 lines
drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 3 +- drivers/iommu/tegra-gart.c | 12 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/vfio/vfio_iommu_type1.c | 27 +- include/linux/io-pgtable.h | 57 ++- include/linux/iommu.h | 92 +++- 23 files changed, 1090 insertions(+), 407 deletions(-)
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerQuoted 27 lines · John Garry
Quoted 26 lines · Will Deacon
This is a significant rework of the RFC I previously posted here:
https://lkml.kernel.org/r/[email protected]
But this time, it looks like it might actually be worthwhile according to my perf profiles, where
__iommu_unmap()falls a long way down the profile for a multi-threaded netperf run. I'm still relying on others to confirm this is useful, however.Some of the changes since last time are:
* Support for constructing and submitting a list of commands in the driver
* Numerous changes to the IOMMU and io-pgtable APIs so that we can submit commands in batches
* Removal of
cmpxchg()fromcmdq_shared_lock()fast-path* Code restructuring and cleanups
This current applies against my iommu/devel branch that Joerg has pulled for 5.3. If you want to test it out, I've put everything here:
https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=iommu/cmdq
Quoted 7 lines
Quoted 3 lines
Feedback welcome. I appreciate that we're in the merge window, but I wanted to get this on the list for people to look at as an RFC.
I tested storage performance on this series, which I think is a better scenario to test than network performance, that being generally limited by the network link speed.
Interesting, thanks for sharing. Do you also see a similar drop in CPU time to the one reported by Ganapat?
Quoted 5 lines
Baseline performance (will/iommu/devel, commit
9e6ea59f3)
8x SAS disks D05 839K IOPS
1x NVMe D05 454K IOPS
1x NVMe D06 442k IOPS
Quoted 8 lines
Patchset performance (will/iommu/cmdq)
8x SAS disk D05 835K IOPS
1x NVMe D05 472K IOPS
1x NVMe D06 459k IOPSSo we see a bit of an NVMe boost, but about the same for 8x disks. No iommu performance is about 918K IOPs for 8x disks, so it is not limited by the medium.
It would be nice to know if this performance gap is because of Linux, or simply because of the translation overhead in the SMMU hardware. Are you able to get a perf profile to see where we're spending time?
Thanks,
Will
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 32 lines · Will Deacon
Quoted 31 lines · John Garry
Quoted 29 lines · Will Deacon
This is a significant rework of the RFC I previously posted here:
https://lkml.kernel.org/r/[email protected]
But this time, it looks like it might actually be worthwhile according to my perf profiles, where
__iommu_unmap()falls a long way down the profile for a multi-threaded netperf run. I'm still relying on others to confirm this is useful, however.Some of the changes since last time are:
* Support for constructing and submitting a list of commands in the driver
* Numerous changes to the IOMMU and io-pgtable APIs so that we can submit commands in batches
* Removal of
cmpxchg()fromcmdq_shared_lock()fast-path* Code restructuring and cleanups
This current applies against my iommu/devel branch that Joerg has pulled for 5.3. If you want to test it out, I've put everything here:
https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=iommu/cmdq
Feedback welcome. I appreciate that we're in the merge window, but I wanted to get this on the list for people to look at as an RFC.
Quoted 6 lines
Quoted 3 lines
I tested storage performance on this series, which I think is a better scenario to test than network performance, that being generally limited by the network link speed.
Interesting, thanks for sharing. Do you also see a similar drop in CPU time to the one reported by Ganapat?
Not really, CPU load reported by fio is mostly the same.
Quoted 11 lines
Quoted 11 lines
Baseline performance (will/iommu/devel, commit
9e6ea59f3)
8x SAS disks D05 839K IOPS
1x NVMe D05 454K IOPS
1x NVMe D06 442k IOPSPatchset performance (will/iommu/cmdq)
8x SAS disk D05 835K IOPS
1x NVMe D05 472K IOPS
1x NVMe D06 459k IOPS
Quoted 7 lines
Quoted 3 lines
So we see a bit of an NVMe boost, but about the same for 8x disks. No iommu performance is about 918K IOPs for 8x disks, so it is not limited by the medium.
It would be nice to know if this performance gap is because of Linux, or simply because of the translation overhead in the SMMU hardware. Are you able to get a perf profile to see where we're spending time?
I'll look to do that, but I'd really expect it to be down to the time linux spends on the DMA map and unmaps.
Cheers,
john
Quoted 7 lines
Thanks,
Will
.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerQuoted 33 lines · John Garry
Quoted 32 lines · Will Deacon
Quoted 31 lines · John Garry
Quoted 29 lines · Will Deacon
This is a significant rework of the RFC I previously posted here:
https://lkml.kernel.org/r/[email protected]
But this time, it looks like it might actually be worthwhile according to my perf profiles, where
__iommu_unmap()falls a long way down the profile for a multi-threaded netperf run. I'm still relying on others to confirm this is useful, however.Some of the changes since last time are:
* Support for constructing and submitting a list of commands in the driver
* Numerous changes to the IOMMU and io-pgtable APIs so that we can submit commands in batches
* Removal of
cmpxchg()fromcmdq_shared_lock()fast-path* Code restructuring and cleanups
This current applies against my iommu/devel branch that Joerg has pulled for 5.3. If you want to test it out, I've put everything here:
https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=iommu/cmdq
Feedback welcome. I appreciate that we're in the merge window, but I wanted to get this on the list for people to look at as an RFC.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I tested storage performance on this series, which I think is a better scenario to test than network performance, that being generally limited by the network link speed.
Interesting, thanks for sharing. Do you also see a similar drop in CPU time to the one reported by Ganapat?
Not really, CPU load reported by fio is mostly the same.
That's a pity. Maybe the cmdq isn't actually getting hit very heavily by fio.
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Baseline performance (will/iommu/devel, commit
9e6ea59f3)
8x SAS disks D05 839K IOPS
1x NVMe D05 454K IOPS
1x NVMe D06 442k IOPSPatchset performance (will/iommu/cmdq)
8x SAS disk D05 835K IOPS
1x NVMe D05 472K IOPS
1x NVMe D06 459k IOPSSo we see a bit of an NVMe boost, but about the same for 8x disks. No iommu performance is about 918K IOPs for 8x disks, so it is not limited by the medium.
Quoted 6 lines
Quoted 3 lines
It would be nice to know if this performance gap is because of Linux, or simply because of the translation overhead in the SMMU hardware. Are you able to get a perf profile to see where we're spending time?
I'll look to do that, but I'd really expect it to be down to the time linux spends on the DMA map and unmaps.
Right, and it would be good to see how much of that is in SMMUv3-specific code. Another interesting thing to try would be reducing the depth of the io-pgtable. We currently key that off VA_BITS which may be much larger than you need (by virtue of being a compile-time value).
Will
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 64 lines · Will Deacon
Quoted 33 lines · John Garry
Quoted 32 lines · Will Deacon
Quoted 31 lines · John Garry
Quoted 29 lines · Will Deacon
This is a significant rework of the RFC I previously posted here:
https://lkml.kernel.org/r/[email protected]
But this time, it looks like it might actually be worthwhile according to my perf profiles, where
__iommu_unmap()falls a long way down the profile for a multi-threaded netperf run. I'm still relying on others to confirm this is useful, however.Some of the changes since last time are:
* Support for constructing and submitting a list of commands in the driver
* Numerous changes to the IOMMU and io-pgtable APIs so that we can submit commands in batches
* Removal of
cmpxchg()fromcmdq_shared_lock()fast-path* Code restructuring and cleanups
This current applies against my iommu/devel branch that Joerg has pulled for 5.3. If you want to test it out, I've put everything here:
https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=iommu/cmdq
Feedback welcome. I appreciate that we're in the merge window, but I wanted to get this on the list for people to look at as an RFC.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I tested storage performance on this series, which I think is a better scenario to test than network performance, that being generally limited by the network link speed.
Interesting, thanks for sharing. Do you also see a similar drop in CPU time to the one reported by Ganapat?
Not really, CPU load reported by fio is mostly the same.
That's a pity. Maybe the cmdq isn't actually getting hit very heavily by fio.
Quoted 18 lines
Quoted 17 lines
Quoted 5 lines
Baseline performance (will/iommu/devel, commit
9e6ea59f3)
8x SAS disks D05 839K IOPS
1x NVMe D05 454K IOPS
1x NVMe D06 442k IOPSQuoted 8 lines
Patchset performance (will/iommu/cmdq)
8x SAS disk D05 835K IOPS
1x NVMe D05 472K IOPS
1x NVMe D06 459k IOPSSo we see a bit of an NVMe boost, but about the same for 8x disks. No iommu performance is about 918K IOPs for 8x disks, so it is not limited by the medium.
It would be nice to know if this performance gap is because of Linux, or simply because of the translation overhead in the SMMU hardware. Are you able to get a perf profile to see where we're spending time?
Quoted 8 lines
Quoted 2 lines
I'll look to do that, but I'd really expect it to be down to the time linux spends on the DMA map and unmaps.
Right, and it would be good to see how much of that is in SMMUv3-specific code. Another interesting thing to try would be reducing the depth of the io-pgtable. We currently key that off
VA_BITSwhich may be much larger than you need (by virtue of being a compile-time value).
The perf call-graph before and after is below. Interesting to see how long the mapping takes. So changing the io-pgtable depth may help here.
The unmapping speed improves - I don't know why it doesn't help the 8x disk scenario throughput, which these profiles are based on.
JFYI, our HW guys are quite concerned with the SMMU programming interface, specifically having a single submission queue which requires locking or some sort of synchronization. Obviously it's not a single culprit for consuming time.
before map
|--3.02%--__iommu_map_sg_attrs
|
--2.99%--iommu_dma_map_sg
|
--2.71%--iommu_map_sg
|
--2.58%--iommu_map
|
--2.50%--arm_smmu_map
|
--2.48%--arm_lpae_map
|
--0.66%--__arm_lpae_map
|
--0.50%--__arm_lpae_mapafter map
--3.25%--__iommu_map_sg_attrs
|
--3.20%--iommu_dma_map_sg
|
--2.77%--iommu_map_sg
|
--2.62%--iommu_map
|
--2.53%--arm_smmu_map
|
--2.49%--arm_lpae_map
|
--0.72%--__arm_lpae_map
|
--0.56%--__arm_lpae_mapbefore unmap
--5.64%--__iommu_unmap_sg_attrs
|
--5.63%--iommu_dma_unmap_sg
|
--5.53%--__iommu_dma_unmap
|
--4.97%--iommu_unmap_fast
|
--4.94%--__iommu_unmap
|
--4.91%--arm_smmu_unmap
|
--4.83%--arm_lpae_unmap
|
--4.80%--__arm_lpae_unmap
|
--4.62%--__arm_lpae_unmap
|
--4.50%--__arm_lpae_unmap
|
--4.33%--__arm_lpae_unmap
|
--4.13%--arm_smmu_tlb_inv_range_nosync
|
--3.97%--arm_smmu_cmdq_issue_cmd
|
--3.74%--_raw_spin_unlock_irqrestoreafter unmap
--2.75%--__iommu_unmap_sg_attrs
|
--2.74%--iommu_dma_unmap_sg
|
--2.59%--__iommu_dma_unmap
|
|--1.82%--arm_smmu_iotlb_sync
| |
| --1.78%--arm_smmu_tlb_inv_range
| |
| --1.75%--arm_smmu_cmdq_issue_cmdlist
|
--0.65%--iommu_unmap_fast
|
--0.64%--__iommu_unmap
|
--0.62%--arm_smmu_unmap
|
--0.55%--arm_lpae_unmap
|
--0.54%--__arm_lpae_unmapI tried to capture a record for direct DMA, but dma_direct_map_sg et al wasn't even showing in graph. That's the target...
John
Quoted 4 lines
Will
.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu