Add PASID support to SMMUv3 unmanaged domains
Subsystems (3)
v5 cover Add PASID support to SMMUv3 unmanaged domains
This patch series implements the set_dev_pasid operation for DMA and UNMANAGED iommu domains. The series depends on the CD table refactor patch series as a pre-requisite.
This patch series is also available on gerrit with Jean's SMMU test
engine patches cherry-picked along with an additional set of tests for
this feature: https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/24770/6
Thanks,
Michael Shavit
Changes in v5:
- Renamed domain_head to list for consistency with other lists
- Renamed attached_domains to attached_ssids to avoid confusion. This is
a list of master/ssid pairs the domain is attached to, not a list of
other domains.
- Fix missing error value return in set_dev_pasid
- Fix issue where nr_attached_pasid_domains isn't updated when
arm_smmu_write_ctx_desc fails
- Fix missing free of the attached_domain node
- Split off the CD table refactor to separate patch series: https://lore.kernel.org/all/[email protected]/
- Link to v4: https://lore.kernel.org/all/[email protected]/
- New commit: Free attached pasid domains on release_device() call
Changes in v4:
- Fix build warning and error on patch 07. The error was introduced during a v1->v2 rebase and hidden by patch 09 which removed the offending line.
- Link to v3: https://lore.kernel.org/all/[email protected]/
Changes in v3:
- Dropped the bulk of the SVA refactoring to re-work as a follow-up series.
- Reworded cover letter to omit dropped changes.
- Rebased on 6.4 tip
- Link to v2: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Reworded cover letter and commits based on v1 feedback.
- Split and reworked iommu/arm-smmu-v3: Move cdtable to arm_smmu_master
- Added SVA clean-up and refactor.
- A few other small bug fixes and cosmetics.
- Link to v1: https://lore.kernel.org/all/[email protected]/
Michael Shavit (6):
iommu/arm-smmu-v3: Simplify arm_smmu_enable_ats iommu/arm-smmu-v3: Keep track of attached ssids iommu/arm-smmu-v3: Add helper for atc invalidation iommu/arm-smmu-v3: Implement set_dev_pasid iommu/arm-smmu-v3: Free pasid domains on iommu release iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 28 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 283 ++++++++++++++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 31 +- 3 files changed, 267 insertions(+), 75 deletions(-)
base-commit: 5d0c230f1de8c7515b6567d9afba1f196fb4e2f4
prerequisite-patch-id: c8d21ff19c2c1dd18799a6b83f483add654d187e
prerequisite-patch-id: bdeb88498393e7049fd22cfc24d2b7674e81bb85
prerequisite-patch-id: b84be729e187aa2d67a7f90ec396e2b878c76243
prerequisite-patch-id: 0ee7da8eaae46e2f8a0a791808f421025e148d79
prerequisite-patch-id: 2d80d99964059ecb31065ec4954130c817b9046c
prerequisite-patch-id: 49910462ec68b834c6af18ae9c58de25982e2752
prerequisite-patch-id: 1daf192523b0b7ed24a670b47ad07366aca6d26d
prerequisite-patch-id: 96101f0f4fd95cdb442cfe0881d80c3a61a93716
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
PATCH v5 1/6 iommu/arm-smmu-v3: Simplify arm_smmu_enable_ats
arm_smmu_enable_ats's call to inv_domain would trigger an invalidation for all masters that a domain is attached to everytime it's attached to another ATS-enabled master. It doesn't seem like those invalidations are necessary, and it's easier to reason about arm_smmu_enable_ats if it only issues invalidation commands for the current master.
---
(no changes since v2)
Changes in v2:
- Fix commit message wrapping
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -1
@@ -2305,7 +2305,7 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master) pdev =to_pci_dev (master->dev);atomic_inc (&smmu_domain->nr_ats_masters); -arm_smmu_atc_inv_domain (smmu_domain,0 ,0 ,0 ); +arm_smmu_atc_inv_master (master);if (pci_enable_ats (pdev, stu))dev_err (master->dev,"Failed to enable ATS (STU %zu)\n" , stu); }
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
PATCH v5 2/6 iommu/arm-smmu-v3: Keep track of attached ssids
In reply to Michael ShavitThe arm-smmu-v3 driver keeps track of all masters that a domain is attached to so that it can re-write their STEs when the domain's ASID is upated by SVA. This tracking is also used to invalidate ATCs on all masters that a domain is attached to.
This change introduces a new data structures to track all the CD entries that a domain is attached to. This change is a pre-requisite to allow domain attachment on non 0 SSIDs.
---
Changes in v5:
- Renamed domain_head to list for consistency with other lists
- Renamed attached_domains to attached_ssids to avoid confusion. This is a list of master/ssid pairs the domain is attached to, not a list of other domains.
Changes in v4:
- Remove reference to the master's domain accidentally re-introduced during a rebase. Make arm_smmu_atc_inv_domain static.
Changes in v2:
- Fix arm_smmu_atc_inv_cmd_set_ssid and other cosmetic changes
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 28 ++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 99 +++++++++++-------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 22 ++++- 3 files changed, 98 insertions(+), 51 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +21 -7
@@ -37,21 +37,35 @@ struct arm_smmu_bond {static DEFINE_MUTEX (sva_lock); +/* +* When ssid is 0, update all the CD entries that this domain is attached to. +* When ssid is non-zero, write the CD into all the masters where this domain is +* the primary domain, with the provided SSID. This is used because SVA still +* piggybacks over the primary domain. +*/ static int arm_smmu_write_ctx_desc_devices (struct arm_smmu_domain *smmu_domain,int ssid,struct arm_smmu_ctx_desc *cd) { +struct arm_smmu_attached_domain *attached_domain;struct arm_smmu_master *master;unsigned long flags;int ret; -spin_lock_irqsave (&smmu_domain->devices_lock, flags); -list_for_each_entry (master, &smmu_domain->devices, domain_head) { - ret =arm_smmu_write_ctx_desc (master, ssid, cd); +spin_lock_irqsave (&smmu_domain->attached_ssids_lock, flags); +list_for_each_entry (attached_domain, &smmu_domain->attached_ssids, + list) { + master = attached_domain->master; +if (ssid && attached_domain->ssid ==0 ) { + ret =arm_smmu_write_ctx_desc (master, ssid, cd); + }else { + ret =arm_smmu_write_ctx_desc ( + master, attached_domain->ssid, cd); + }if (ret)break ; } -spin_unlock_irqrestore (&smmu_domain->devices_lock, flags); +spin_unlock_irqrestore (&smmu_domain->attached_ssids_lock, flags);return ret; }
@@ -222,7 +236,7 @@ static void arm_smmu_mm_invalidate_range(struct mmu_notifier *mn,if (!(smmu_domain->smmu->features &ARM_SMMU_FEAT_BTM ))arm_smmu_tlb_inv_range_asid (start, size, smmu_mn->cd->asid,PAGE_SIZE , false, smmu_domain); -arm_smmu_atc_inv_domain (smmu_domain, mm->pasid, start, size); +arm_smmu_atc_inv_domain_ssid (smmu_domain, mm->pasid, start, size); }static void arm_smmu_mm_release (struct mmu_notifier *mn,struct mm_struct *mm)
@@ -243,7 +257,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)arm_smmu_write_ctx_desc_devices (smmu_domain, mm->pasid, &quiet_cd);arm_smmu_tlb_inv_asid (smmu_domain->smmu, smmu_mn->cd->asid); -arm_smmu_atc_inv_domain (smmu_domain, mm->pasid,0 ,0 ); +arm_smmu_atc_inv_domain_ssid (smmu_domain, mm->pasid,0 ,0 ); smmu_mn->cleared = true;mutex_unlock (&sva_lock);
@@ -333,7 +347,7 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)*/ if (!smmu_mn->cleared) {arm_smmu_tlb_inv_asid (smmu_domain->smmu, cd->asid); -arm_smmu_atc_inv_domain (smmu_domain, mm->pasid,0 ,0 ); +arm_smmu_atc_inv_domain_ssid (smmu_domain, mm->pasid,0 ,0 ); }/* Frees smmu_mn */
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +60 -39
225 lines not on this page · raw message
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +17 -5
@@ -689,11 +689,19 @@ struct arm_smmu_stream {struct rb_node node; }; +/* List of {masters, ssid} that a domain is attached to */ +struct arm_smmu_attached_domain { +struct list_head list; +struct arm_smmu_domain *domain; +struct arm_smmu_master *master; +int ssid; +}; +/* SMMU private data for each master */ struct arm_smmu_master {struct arm_smmu_device *smmu;struct device *dev; -struct arm_smmu_domain *domain; +struct arm_smmu_attached_domain non_pasid_domain;struct list_head domain_head;struct arm_smmu_stream *streams;/* Locked by the iommu core using the group mutex */
@@ -730,8 +738,12 @@ struct arm_smmu_domain {struct iommu_domain domain; -struct list_head devices; -spinlock_t devices_lock; +/* +* List of arm_smmu_attached_domain nodes used to track all the +* {master, ssid} pairs that this domain is attached to. +*/ +struct list_head attached_ssids; +spinlock_t attached_ssids_lock;struct list_head mmu_notifiers; };
@@ -752,8 +764,8 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, struct arm_smmu_domain *smmu_domain);bool arm_smmu_free_asid (struct arm_smmu_ctx_desc *cd); -int arm_smmu_atc_inv_domain (struct arm_smmu_domain *smmu_domain,int ssid, -unsigned long iova,size_t size); +int arm_smmu_atc_inv_domain_ssid (struct arm_smmu_domain *smmu_domain,int ssid, +unsigned long iova,size_t size);#ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported (struct arm_smmu_device *smmu);
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
PATCH v5 3/6 iommu/arm-smmu-v3: Add helper for atc invalidation
In reply to Michael ShavitThis will be used to invalidate ATC entries made on an SSID for a master when detaching a domain with pasid.
---
(no changes since v2)
Changes in v2:
- Make use of arm_smmu_atc_inv_cmd_set_ssid
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +8 -1
@@ -1798,13 +1798,15 @@ arm_smmu_atc_inv_to_cmd(unsigned long iova, size_t size, cmd->atc.size = log2_span; } -static int arm_smmu_atc_inv_master (struct arm_smmu_master *master) +static int arm_smmu_atc_inv_master_ssid (struct arm_smmu_master *master, +int ssid) {int i;struct arm_smmu_cmdq_ent cmd;struct arm_smmu_cmdq_batch cmds;arm_smmu_atc_inv_to_cmd (0 ,0 , &cmd); +arm_smmu_atc_inv_cmd_set_ssid (ssid, &cmd); cmds.num =0 ;for (i =0 ; i < master->num_streams; i++) { cmd.atc.sid = master->streams[i].id;
@@ -1814,6 +1816,11 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)return arm_smmu_cmdq_batch_submit (master->smmu, &cmds); } +static int arm_smmu_atc_inv_master (struct arm_smmu_master *master) +{ +return arm_smmu_atc_inv_master_ssid (master,0 ); +} +/* * If ssid is non-zero, issue atc invalidations with the given ssid instead of * the one the domain is attached to. This is used by SVA since it's pasid
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
PATCH v5 4/6 iommu/arm-smmu-v3: Implement set_dev_pasid
In reply to Michael ShavitThis change enables the use of the iommu_attach_dev_pasid API for UNMANAGED domains. The primary use-case is to allow in-kernel users of the iommu API to manage domains with PASID. This change also allows for future support of pasid in the DMA api.
---
Changes in v5:
- Fix missing error value return in set_dev_pasid
- Fix issue where nr_attached_pasid_domains isn't updated when
arm_smmu_write_ctx_desc fails
- Fix missing free of the attached_domain node
- Split off the CD table refactor to separate patch series: https://lore.kernel.org/all/[email protected]/
- Link to v4: https://lore.kernel.org/all/[email protected]/
- Remove districting change where a NULL master is passed to
arm_smmu_prepare_domain_for_smmu
Changes in v4:
- Fix build warning and error on patch 07. The error was introduced during a v1->v2 rebase and hidden by patch 09 which removed the offending line.
- Link to v3: https://lore.kernel.org/all/[email protected]/
Changes in v3:
- Dropped the bulk of the SVA refactoring to re-work as a follow-up series.
- Reworded cover letter to omit dropped changes.
- Rebased on 6.4 tip
- Link to v2: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Reworded cover letter and commits based on v1 feedback.
- Split and reworked iommu/arm-smmu-v3: Move cdtable to arm_smmu_master
- Added SVA clean-up and refactor.
- A few other small bug fixes and cosmetics.
- Link to v1: https://lore.kernel.org/all/[email protected]/
- Add missing atc invalidation when detaching with pasid
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 156 ++++++++++++++++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 2 files changed, 141 insertions(+), 16 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +140 -16
@@ -2388,6 +2388,11 @@ static int arm_smmu_enable_pasid(struct arm_smmu_master *master)return 0 ; } +static bool arm_smmu_master_has_pasid_domains (struct arm_smmu_master *master) +{ +return master->nr_attached_pasid_domains >0 ; +} +static void arm_smmu_disable_pasid (struct arm_smmu_master *master) {struct pci_dev *pdev;
@@ -2423,6 +2428,25 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master)arm_smmu_install_ste_for_dev (master); } +static int arm_smmu_prepare_domain_for_smmu (struct arm_smmu_device *smmu, +struct arm_smmu_domain *smmu_domain, +struct arm_smmu_master *master) +{ +int ret =0 ; + +mutex_lock (&smmu_domain->init_mutex); +if (!smmu_domain->smmu) { + smmu_domain->smmu = smmu; + ret =arm_smmu_domain_finalise (&smmu_domain->domain, master); +if (ret) + smmu_domain->smmu =NULL ; + }else if (smmu_domain->smmu != smmu) + ret = -EINVAL ; + +mutex_unlock (&smmu_domain->init_mutex); +return ret; +} +static int arm_smmu_attach_dev (struct iommu_domain *domain,struct device *dev) {int ret =0 ;
@@ -2438,6 +2462,10 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master =dev_iommu_priv_get (dev); smmu = master->smmu; + ret =arm_smmu_prepare_domain_for_smmu (smmu, smmu_domain, master); +if (ret) +return ret; +/* * Checking that SVA is disabled ensures that this device isn't bound to * any mm, and can be safely detached from its old domain. Bonds cannot
@@ -2448,21 +2476,17 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)return -EBUSY ; } -arm_smmu_detach_dev (master); - -mutex_lock (&smmu_domain->init_mutex); - -if (!smmu_domain->smmu) { - smmu_domain->smmu = smmu; - ret =arm_smmu_domain_finalise (domain, master); -if (ret) - smmu_domain->smmu =NULL ; - }else if (smmu_domain->smmu != smmu) - ret = -EINVAL ; +/* +* Attaching a bypass or stage 2 domain would break any domains attached +* with pasid. Attaching an S1 domain should be feasible but requires +* more complicated logic to handle. +*/ +if (arm_smmu_master_has_pasid_domains (master)) { +dev_err (dev,"cannot attach - domain attached with pasid\n" ); +return -EBUSY ; + } -mutex_unlock (&smmu_domain->init_mutex); -if (ret) -return ret; +arm_smmu_detach_dev (master);/* * The SMMU does not support enabling ATS with bypass. When the STE is
@@ -2500,6 +2524,72 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)return 0 ; } +static int arm_smmu_set_dev_pasid (struct iommu_domain *domain, +struct device *dev,ioasid_t pasid) +{ +int ret =0 ; +unsigned long flags; +struct iommu_fwspec *fwspec =dev_iommu_fwspec_get (dev); +struct arm_smmu_device *smmu; +struct arm_smmu_domain *smmu_domain =to_smmu_domain (domain); +struct arm_smmu_attached_domain *attached_domain; +struct arm_smmu_master *master; + +if (!fwspec) +return -ENOENT ; + + master =dev_iommu_priv_get (dev); + smmu = master->smmu; + + ret =arm_smmu_prepare_domain_for_smmu (smmu, smmu_domain, master); +if (ret) +return ret; + +if (pasid ==0 ) { +dev_err (dev,"pasid 0 is reserved for the device's primary domain\n" ); +return -ENODEV ; + } + +if (smmu_domain->stage !=ARM_SMMU_DOMAIN_S1 ) { +dev_err (dev,"set_dev_pasid only supports stage 1 domains\n" ); +return -EINVAL ; + } + +if (!master->cd_table.cdtab) +return -EBUSY ; + + attached_domain =kzalloc (sizeof (*attached_domain),GFP_KERNEL ); +if (!attached_domain) +return -ENOMEM ; + + attached_domain->master = master; + attached_domain->domain = smmu_domain; + attached_domain->ssid = pasid; + +/* +* arm_smmu_share_asid may update the cd's asid value and write the +* ctx_desc for every attached_domains in the list. There's a potential +* race here regardless of whether we first write the ctx_desc or +* first insert into the domain's list. Grabbing the asic_lock prevents +* SVA from changing the cd's ASID while the cd is being attached. +*/ +mutex_lock (&arm_smmu_asid_lock); + ret =arm_smmu_write_ctx_desc (master, pasid, &smmu_domain->cd); +if (ret) { +mutex_unlock (&arm_smmu_asid_lock); +kfree (attached_domain); +return ret; + } + +spin_lock_irqsave (&smmu_domain->attached_ssids_lock, flags); +list_add (&attached_domain->list, &smmu_domain->attached_ssids); +spin_unlock_irqrestore (&smmu_domain->attached_ssids_lock, flags); +mutex_unlock (&arm_smmu_asid_lock); + + master->nr_attached_pasid_domains +=1 ; +return 0 ; +} +static int arm_smmu_map_pages (struct iommu_domain *domain,unsigned long iova,phys_addr_t paddr,size_t pgsize,size_t pgcount,int prot,gfp_t gfp,size_t *mapped)
@@ -2738,6 +2828,15 @@ static void arm_smmu_release_device(struct device *dev)if (WARN_ON (arm_smmu_master_sva_enabled (master)))iopf_queue_remove_device (master->smmu->evtq.iopf, dev); +if (WARN_ON (master->nr_attached_pasid_domains !=0 )) { +/* +* TODO: Do we need to handle this case? +* This requires a mechanism to obtain all the pasid domains +* that this master is attached to so that we can clean up the +* domain's attached_domain list. +*/ + } +arm_smmu_detach_dev (master);arm_smmu_disable_pasid (master);arm_smmu_remove_master (master);
@@ -2874,12 +2973,36 @@ static int arm_smmu_def_domain_type(struct device *dev)static void arm_smmu_remove_dev_pasid (struct device *dev,ioasid_t pasid) {struct iommu_domain *domain; +struct arm_smmu_master *master =dev_iommu_priv_get (dev); +struct arm_smmu_domain *smmu_domain; +struct arm_smmu_attached_domain *attached_domain; +unsigned long flags; - domain =iommu_get_domain_for_dev_pasid (dev, pasid,IOMMU_DOMAIN_SVA ); +if (!master || pasid ==0 ) +return ; + + domain =iommu_get_domain_for_dev_pasid (dev, pasid,0 );if (WARN_ON (IS_ERR (domain)) || !domain)return ; +if (domain->type ==IOMMU_DOMAIN_SVA ) +return arm_smmu_sva_remove_dev_pasid (domain, dev, pasid); -arm_smmu_sva_remove_dev_pasid (domain, dev, pasid); + smmu_domain =to_smmu_domain (domain); +mutex_lock (&arm_smmu_asid_lock); +spin_lock_irqsave (&smmu_domain->attached_ssids_lock, flags); +list_for_each_entry (attached_domain, &smmu_domain->attached_ssids, list) { +if (attached_domain->master != master || + attached_domain->ssid != pasid) +continue ; +list_del (&attached_domain->list); + master->nr_attached_pasid_domains -=1 ; +kfree (attached_domain); +break ; + } +spin_unlock_irqrestore (&smmu_domain->attached_ssids_lock, flags); +arm_smmu_write_ctx_desc (master, pasid,NULL ); +arm_smmu_atc_inv_master_ssid (master, pasid); +mutex_unlock (&arm_smmu_asid_lock); }static struct iommu_ops arm_smmu_ops = {
@@ -2899,6 +3022,7 @@ static struct iommu_ops arm_smmu_ops = { .owner = THIS_MODULE, .default_domain_ops = &(const struct iommu_domain_ops ) { .attach_dev = arm_smmu_attach_dev, + .set_dev_pasid = arm_smmu_set_dev_pasid, .map_pages = arm_smmu_map_pages, .unmap_pages = arm_smmu_unmap_pages, .flush_iotlb_all = arm_smmu_flush_iotlb_all,
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +1 -0
@@ -713,6 +713,7 @@ struct arm_smmu_master {bool iopf_enabled;struct list_head bonds;unsigned int ssid_bits; +unsigned int nr_attached_pasid_domains; };/* SMMU private data for an IOMMU domain */
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
PATCH v5 5/6 iommu/arm-smmu-v3: Free pasid domains on iommu release
In reply to Michael ShavitThe iommu core doesn't guarantee that pasid domains will be detached before the device is released.
Track the list of domains that a master is attached to with PASID, so that they can be freed when the iommu is released.
---
Changes in v5:
- New commit: Free attached pasid domains on release_device() call
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 22 +++++++++++++++------ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 10 +++++++++- 2 files changed, 25 insertions(+), 7 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +16 -6
@@ -2587,6 +2587,9 @@ static int arm_smmu_set_dev_pasid(struct iommu_domain *domain,mutex_unlock (&arm_smmu_asid_lock); master->nr_attached_pasid_domains +=1 ; +list_add (&attached_domain->list_in_master, + &master->attached_domains); +return 0 ; }
@@ -2786,6 +2789,7 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->dev = dev; master->smmu = smmu;INIT_LIST_HEAD (&master->bonds); +INIT_LIST_HEAD (&master->attached_domains);dev_iommu_priv_set (dev, master); ret =arm_smmu_insert_master (smmu, master);
@@ -2825,16 +2829,21 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)static void arm_smmu_release_device (struct device *dev) {struct arm_smmu_master *master =dev_iommu_priv_get (dev); +struct arm_smmu_attached_domain *attached_domain; +struct arm_smmu_domain *smmu_domain; +unsigned long flags;if (WARN_ON (arm_smmu_master_sva_enabled (master)))iopf_queue_remove_device (master->smmu->evtq.iopf, dev);if (WARN_ON (master->nr_attached_pasid_domains !=0 )) { -/* -* TODO: Do we need to handle this case? -* This requires a mechanism to obtain all the pasid domains -* that this master is attached to so that we can clean up the -* domain's attached_domain list. -*/ +list_for_each_entry (attached_domain, &master->attached_domains, list_in_master) { + smmu_domain = attached_domain->domain; +spin_lock_irqsave (&smmu_domain->attached_ssids_lock, flags); +list_del (&attached_domain->list); +list_del (&attached_domain->list_in_master); +kfree (&attached_domain->list_in_master); +spin_unlock_irqrestore (&smmu_domain->attached_ssids_lock, flags); + } }arm_smmu_detach_dev (master);
@@ -2995,6 +3004,7 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) attached_domain->ssid != pasid)continue ;list_del (&attached_domain->list); +list_del (&attached_domain->list_in_master); master->nr_attached_pasid_domains -=1 ;kfree (attached_domain);break ;
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +9 -1
@@ -689,9 +689,15 @@ struct arm_smmu_stream {struct rb_node node; }; -/* List of {masters, ssid} that a domain is attached to */ +/* +* List of {masters, ssid} that a domain is attached to, and conversely of +* domains that a master is attached to. +*/ struct arm_smmu_attached_domain { +/* List node arm_smmu_domain*/ struct list_head list; +/* List node in arm_smmu_master*/ +struct list_head list_in_master;struct arm_smmu_domain *domain;struct arm_smmu_master *master;int ssid;
@@ -714,6 +720,8 @@ struct arm_smmu_master {struct list_head bonds;unsigned int ssid_bits;unsigned int nr_attached_pasid_domains; +/* Locked by the iommu core using the group mutex */ +struct list_head attached_domains; };/* SMMU private data for an IOMMU domain */
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
PATCH v5 6/6 iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
In reply to Michael ShavitRemove unused master parameter now that the CD table is allocated elsewhere.
---
(no changes since v1)
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +6 -11
@@ -2129,7 +2129,6 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) }static int arm_smmu_domain_finalise_cd (struct arm_smmu_domain *smmu_domain, -struct arm_smmu_master *master,struct io_pgtable_cfg *pgtbl_cfg) {int ret;
@@ -2167,7 +2166,6 @@ static int arm_smmu_domain_finalise_cd(struct arm_smmu_domain *smmu_domain, }static int arm_smmu_domain_finalise_s2 (struct arm_smmu_domain *smmu_domain, -struct arm_smmu_master *master,struct io_pgtable_cfg *pgtbl_cfg) {int vmid;
@@ -2192,8 +2190,7 @@ static int arm_smmu_domain_finalise_s2(struct arm_smmu_domain *smmu_domain,return 0 ; } -static int arm_smmu_domain_finalise (struct iommu_domain *domain, -struct arm_smmu_master *master) +static int arm_smmu_domain_finalise (struct iommu_domain *domain) {int ret;unsigned long ias, oas;
@@ -2201,7 +2198,6 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain,struct io_pgtable_cfg pgtbl_cfg;struct io_pgtable_ops *pgtbl_ops;int (*finalise_stage_fn)(struct arm_smmu_domain *, -struct arm_smmu_master *,struct io_pgtable_cfg *);struct arm_smmu_domain *smmu_domain =to_smmu_domain (domain);struct arm_smmu_device *smmu = smmu_domain->smmu;
@@ -2253,7 +2249,7 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain, domain->geometry.aperture_end = (1UL << pgtbl_cfg.ias) -1 ; domain->geometry.force_aperture = true; - ret =finalise_stage_fn (smmu_domain, master, &pgtbl_cfg); + ret =finalise_stage_fn (smmu_domain, &pgtbl_cfg);if (ret <0 ) {free_io_pgtable_ops (pgtbl_ops);return ret;
@@ -2429,15 +2425,14 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master) }static int arm_smmu_prepare_domain_for_smmu (struct arm_smmu_device *smmu, -struct arm_smmu_domain *smmu_domain, -struct arm_smmu_master *master) +struct arm_smmu_domain *smmu_domain) {int ret =0 ;mutex_lock (&smmu_domain->init_mutex);if (!smmu_domain->smmu) { smmu_domain->smmu = smmu; - ret =arm_smmu_domain_finalise (&smmu_domain->domain, master); + ret =arm_smmu_domain_finalise (&smmu_domain->domain);if (ret) smmu_domain->smmu =NULL ; }else if (smmu_domain->smmu != smmu)
@@ -2462,7 +2457,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master =dev_iommu_priv_get (dev); smmu = master->smmu; - ret =arm_smmu_prepare_domain_for_smmu (smmu, smmu_domain, master); + ret =arm_smmu_prepare_domain_for_smmu (smmu, smmu_domain);if (ret)return ret;
@@ -2541,7 +2536,7 @@ static int arm_smmu_set_dev_pasid(struct iommu_domain *domain, master =dev_iommu_priv_get (dev); smmu = master->smmu; - ret =arm_smmu_prepare_domain_for_smmu (smmu, smmu_domain, master); + ret =arm_smmu_prepare_domain_for_smmu (smmu, smmu_domain);if (ret)return ret;
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Re: [PATCH v5 6/6] iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
Quoted 3 lines · Michael Shavit
Remove unused master parameter now that the CD table is allocated elsewhere.
This almost certainly belongs in the CD table patch series. I'd like to move it there if this commit looks OK.
Re: [PATCH v5 5/6] iommu/arm-smmu-v3: Free pasid domains on iommu release
In reply to Michael ShavitQuoted 3 lines · Michael Shavit
The iommu core doesn't guarantee that pasid domains will be detached before the device is released.
I'm not certain whether this is possible or not. Is this change really necessary?
Re: [PATCH v5 6/6] iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
In reply to Michael ShavitQuoted 10 lines · Michael Shavit
Remove unused master parameter now that the CD table is allocated elsewhere.
Signed-off-by Michael Shavit <[email protected]>---
(no changes since v1)
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
Agree with moving the commit
Jason
Re: [PATCH v5 5/6] iommu/arm-smmu-v3: Free pasid domains on iommu release
In reply to Michael ShavitQuoted 7 lines · Michael Shavit
Quoted 3 lines · Michael Shavit
The iommu core doesn't guarantee that pasid domains will be detached before the device is released.
I'm not certain whether this is possible or not. Is this change really necessary?
I think we should rely on the core code to detach all PASIDs prior to calling release, drivers should not do this. I suppose that means we have a missing bit in the core code.
FWIW, I'd like to get to a point where the core code can also automatically attach an identity domain so that the driver's release functions don't have to open code that either...
Thus I wouldn't do this patch..
Jason
Re: [PATCH v5 2/6] iommu/arm-smmu-v3: Keep track of attached ssids
In reply to Michael ShavitQuoted 3 lines · Michael Shavit
The arm-smmu-v3 driver keeps track of all masters that a domain is attached to so that it can re-write their STEs when the domain's ASID is upated by SVA.
Wah?
A domain's ASID shouldn't change, why does it change for SVA? Doesn't SVA use CDTE's only? Why would it ever change a STE? I'm confused what you are trying to explain here.
Quoted 6 lines
+/*
+ * When ssid is 0, update all the CD entries that this domain is attached to.
+ * When ssid is non-zero, write the CD into all the masters where this domain is
+ * the primary domain, with the provided SSID. This is used because SVA still
+ * piggybacks over the primary domain.
+ */
What is a "primary domain"? Why can't we fix SVA first so it doesn't have this weird "piggybacks" or:
Quoted 5 lines
+/*
+ * If ssid is non-zero, issue atc invalidations with the given ssid instead of
+ * the one the domain is attached to. This is used by SVA since it's pasid
+ * attachments aren't recorded insmmu_domainyet.
+ */
fails to be recorded?
This patch is not making sense to me, the goal in the commit message seems logical, but why is tracking CD entries introducing this concept of a primary domain and doing special stuff for SSID=0?
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h struct arm_smmu_stream { · line 694
Quoted 11 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +3
-0@@ -689,11 +689,19 @@ struct arm_smmu_stream {struct rb_node node; }; +/* List of {masters, ssid} that a domain is attached to */ +struct arm_smmu_attached_domain { +struct list_head list;
I like to call this something like
attached_ssids_item
To help understand where the list head is and that this is a list element.
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h struct arm_smmu_stream { · line 704
Quoted 11 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +6
-1+struct arm_smmu_domain *domain; +struct arm_smmu_master *master; +int ssid; +}; +/* SMMU private data for each master */ struct arm_smmu_master {struct arm_smmu_device *smmu;struct device *dev; -struct arm_smmu_domain *domain; +struct arm_smmu_attached_domain non_pasid_domain;
Probably should consistently use the word ssid not pasid in driver internals.
The smmu spec talks about the substream ID being optional and the behavior is controleld by the STE.S1DSS (Default substream)
So maybe non_subtream_domain ?
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h struct arm_smmu_domain { · line 746
Quoted 12 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +6
-2@@ -730,8 +738,12 @@ struct arm_smmu_domain {struct iommu_domain domain; -struct list_head devices; -spinlock_t devices_lock; +/* +* List of arm_smmu_attached_domain nodes used to track all the +* {master, ssid} pairs that this domain is attached to. +*/ +struct list_head attached_ssids; +spinlock_t attached_ssids_lock;
So ssid = 0 means that the list entry == master->non_pasid_domain ?
It would be clearer to just test that directly if that is what needs to be determined.
At least these struct changes seem aligned with the commit message :)
Jason
Re: [PATCH v5 2/6] iommu/arm-smmu-v3: Keep track of attached ssids
Quoted 11 lines · Jason Gunthorpe
Quoted 3 lines · Michael Shavit
The arm-smmu-v3 driver keeps track of all masters that a domain is attached to so that it can re-write their STEs when the domain's ASID is upated by SVA.
Wah?
A domain's ASID shouldn't change, why does it change for SVA? Doesn't SVA use CDTE's only? Why would it ever change a STE? I'm confused what you are trying to explain here.
Urh right, I mixed up CD entry and STE here. Before this patch, SVA keeps tracks of all the masters attached to a CD domain, and updates the CD entry 0 in their CD table. Now that a CD domain can be attached on non-zero SSIDs, SVA can't simply update slot 0 in the table; it must know which slot(s) this domain is attached to.
Quoted 4 lines
What is a "primary domain"? Why can't we fix SVA first so it doesn't have this weird "piggybacks" or:
...
Quoted 4 lines
This patch is not making sense to me, the goal in the commit message seems logical, but why is tracking CD entries introducing this concept of a primary domain and doing special stuff for SSID=0?
I'd argue this patch isn't introducing anything the driver isn't already doing. It's just making the status-quo explicit since it has to handle it. I do have a patch series in the works to properly fix SVA, but it's growing quite large and I was trying to get this feature out first. At a high level, the subsequent series:
1. Nests the list of attached masters in a list of SMMUs the domain is installed in. Updates SMMU-level operations (such as invalidations) to iterate over said list.
2. Checks the compatibility of a domain when attaching to a new SMMU instead of outright rejecting, to allow attaching a domain to multiple SMMUs.
3. Thus allowing SVA to alloc a single domain for the MM struct (which the series maps from multiple SVA domains internally, pending support at the iommu core layer)
4. And allowing SVA to use the same set_dev_pasid implementation used here on that domain.
Now having said that, it might be possible to get rid of piggybacking sooner if we create an MM per master instead of per "primary-domain", but I'm not too sure about performance implications. AFAICT, the only downside might be for invalidate_range commands that could no longer be sent as a batched command to the SMMU (since each mmu notifier would be called independently).
Re: [PATCH v5 2/6] iommu/arm-smmu-v3: Keep track of attached ssids
Quoted 9 lines · Michael Shavit
Quoted 8 lines · Jason Gunthorpe
Quoted 3 lines · Michael Shavit
The arm-smmu-v3 driver keeps track of all masters that a domain is attached to so that it can re-write their STEs when the domain's ASID is upated by SVA.
Wah?
Quoted 7 lines
Quoted 3 lines
A domain's ASID shouldn't change, why does it change for SVA? Doesn't SVA use CDTE's only? Why would it ever change a STE? I'm confused what you are trying to explain here.
Urh right, I mixed up CD entry and STE here. Before this patch, SVA keeps tracks of all the masters attached to a CD domain, and updates the CD entry 0 in their CD table.
Because it assumes that if a domain is returned from the ASID lookup it is a RID domain.
Quoted 3 lines
Now that a CD domain can be attached on non-zero SSIDs, SVA can't simply update slot 0 in the table; it must know which slot(s) this domain is attached to.
Yes, so why are you passing in 0 as the ssid argument to arm_smmu_write_ctx_desc_devices() for the ASID change?
I think your commit message is trying to say:
The SMMUv3 driver keeps track of all the iommu_domains that are assigned to an ASID in an xarray. The SVA code needs to re-use the same ASID as the CPU's ASID (presumably only for BTM mode?) so it has a mechanism to reclaim an already used ASID from an existing domain.
This is currently hardwired with the assumption that a domain using an ASID is only on SSID 0.
Add a list to the struct arm_smmu_domain recording each master and SSID that the domain is attached to and update it when any domain is attached/detached.
Make arm_smmu_write_ctx_desc_devices() follow this list when storing the CD table entries for the domain.
Remove 'ssid' as an argument to arm_smmu_write_ctx_desc_devices() since it is always found in the attached_ssids.
Quoted 11 lines
Quoted 3 lines
What is a "primary domain"? Why can't we fix SVA first so it doesn't have this weird "piggybacks" or:
...
Quoted 4 lines
This patch is not making sense to me, the goal in the commit message seems logical, but why is tracking CD entries introducing this concept of a primary domain and doing special stuff for SSID=0?
I'd argue this patch isn't introducing anything the driver isn't already doing.
So this I don't understand:
+ if (ssid && attached_domain->ssid == 0) {
+ ret = arm_smmu_write_ctx_desc(master, ssid, cd);
+ } else {
+ ret = arm_smmu_write_ctx_desc(
+ master, attached_domain->ssid, cd);
+ }
Fix this patch so attached_domain->ssid is never wrong?
Remove ssid as an input to the function.
(I'd ultimately expect to remove CD too)
Quoted 4 lines
it. I do have a patch series in the works to properly fix SVA, but it's growing quite large and I was trying to get this feature out first. At a high level, the subsequent series:
1. Nests the list of attached masters in a list of SMMUs the domain is
Quoted 8 lines
installed in. Updates SMMU-level operations (such as invalidations) to iterate over said list.
2. Checks the compatibility of a domain when attaching to a new SMMU instead of outright rejecting, to allow attaching a domain to multiple SMMUs.
3. Thus allowing SVA to alloc a single domain for the MM struct (which the series maps from multiple SVA domains internally, pending support at the iommu core layer)
This should not be hard for the core code
Quoted 2 lines
4. And allowing SVA to use the same
set_dev_pasidimplementation used here on that domain.
This list all makes alot of sense to me
Quoted 6 lines
Now having said that, it might be possible to get rid of piggybacking sooner if we create an MM per master instead of per "primary-domain", but I'm not too sure about performance implications. AFAICT, the only downside might be for
invalidate_rangecommands that could no longer be sent as a batched command to the SMMU (since each mmu notifier would be called independently).
I'm not sure this series leaves things in a better state than before, now we have two parallel domain attachment paths for PASID :(
Jason
Re: [PATCH v5 4/6] iommu/arm-smmu-v3: Implement set_dev_pasid
In reply to Michael ShavitQuoted 14 lines · Michael Shavit
(quoted diff) +0
-10@@ -2448,21 +2476,17 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return -EBUSY; } - arm_smmu_detach_dev(master); - - mutex_lock(&smmu_domain->init_mutex); - - if (!smmu_domain->smmu) { - smmu_domain->smmu = smmu; - ret = arm_smmu_domain_finalise(domain, master); - if (ret) - smmu_domain->smmu = NULL; - } else if (smmu_domain->smmu != smmu)
Review static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 2487
Quoted 10 lines
(quoted diff) +9
-1- ret = -EINVAL; + /* + * Attaching a bypass or stage 2 domain would break any domains attached + * with pasid. Attaching an S1 domain should be feasible but requires + * more complicated logic to handle. + */ + if (arm_smmu_master_has_pasid_domains(master)) { + dev_err(dev, "cannot attach - domain attached with pasid\n"); + return -EBUSY; + }
Basically don't you just call set_dev_pased(pasid = 0) and it will do that logic?
Review static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
Quoted 13 lines
(quoted diff) +13
-0+static int arm_smmu_set_dev_pasid(struct iommu_domain *domain, + struct device *dev, ioasid_t pasid) +{ + int ret = 0; + unsigned long flags; + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); + struct arm_smmu_device *smmu; + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + struct arm_smmu_attached_domain *attached_domain; + struct arm_smmu_master *master; + + if (!fwspec) + return -ENOENT;
fwspec drivers always have fwspecs, they don't need to check this.
Quoted 5 lines
(quoted diff) +5
-0+ + master = dev_iommu_priv_get(dev); + smmu = master->smmu; + + ret = arm_smmu_prepare_domain_for_smmu(smmu, smmu_domain, master);
Review
Quoted 10 lines
(quoted diff) +10
-0+ if (ret) + return ret; + + if (pasid == 0) { + dev_err(dev, "pasid 0 is reserved for the device's primary domain\n"); + return -ENODEV; + } + + if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) { + dev_err(dev, "set_dev_pasid only supports stage 1 domains\n");
don't print messages, iommufd can trigger these paths from userspace.
Review static void arm_smmu_release_device(struct device *dev) · line 2838
Quoted 12 lines
(quoted diff) +8
-0@@ -2738,6 +2828,15 @@ static void arm_smmu_release_device(struct device *dev) if (WARN_ON(arm_smmu_master_sva_enabled(master))) iopf_queue_remove_device(master->smmu->evtq.iopf, dev); + if (WARN_ON(master->nr_attached_pasid_domains != 0)) { + /* + * TODO: Do we need to handle this case? + * This requires a mechanism to obtain all the pasid domains + * that this master is attached to so that we can clean up the + * domain's attached_domain list. + */ + }
Unnecessary, the core code should handle this, add a patch that does this:
drivers/iommu/iommu.c +7 -0
@@ -438,9 +438,16 @@ static void iommu_deinit_device(struct device *dev) {struct iommu_group *group = dev->iommu_group;const struct iommu_ops *ops =dev_iommu_ops (dev); + struct iommu_domain *{pasid_domain; +unsigned long pasid;lockdep_assert_held (&group->mutex); +xa_for_each (&group->pasid_array, pasid, pasid_domain) { +__iommu_remove_group_pasid (pasid); +xa_erase (&group->pasid_array, pasid); + } +iommu_device_unlink (dev->iommu->iommu_dev, dev);/*
Quoted 6 lines
(quoted diff) +2
-0@@ -2874,12 +2973,36 @@ static int arm_smmu_def_domain_type(struct device *dev) static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) { struct iommu_domain *domain; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct arm_smmu_domain *smmu_domain;
Review static int arm_smmu_def_domain_type(struct device *dev) · line 2986
Quoted 10 lines
(quoted diff) +6
-1+ struct arm_smmu_attached_domain *attached_domain; + unsigned long flags; - domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA); + if (!master || pasid == 0) + return; + + domain = iommu_get_domain_for_dev_pasid(dev, pasid, 0); if (WARN_ON(IS_ERR(domain)) || !domain) return;
I feel like we have made a mistake here to not pass in the current domain from the core code, it already knows what the domain was..
Review static int arm_smmu_def_domain_type(struct device *dev) · line 2988
Quoted 2 lines
(quoted diff) +2
-0+ if (domain->type == IOMMU_DOMAIN_SVA) + return arm_smmu_sva_remove_dev_pasid(domain, dev, pasid);
Yuk.
Jason