iommu: SVA and IOPF refactoring
Subsystems (6)
Lu Baolu
maintainerv4 cover iommu: SVA and IOPF refactoring
Hi folks,
The former part of this series refactors the IOMMU SVA code by assigning an SVA type of iommu_domain to a shared virtual address and replacing sva_bind/unbind iommu ops with attach/detach_dev_pasid domain ops.
The latter part changes the existing I/O page fault handling framework from only serving SVA to a generic one. Any driver or component could handle the I/O page faults for its domain in its own way by installing an I/O page fault handler.
This series has been functionally tested on an x86 machine and compile tested for other architectures.
This series is also available on github:
[2] https://github.com/LuBaolu/intel-iommu/commits/iommu-sva-refactoring-v4
Please review and suggest.
Best regards,
baolu
Change log:
v4:
- Solve the overlap with another series and make this series self-contained.
- No objection to the abstraction of data structure during v3 review. Hence remove the RFC subject prefix.
- Refine the immutable singleton group code according to Kevin's comments.
v3:
- https://lore.kernel.org/linux-iommu/[email protected]/
- Rework iommu_group_singleton_lockdown() by adding a flag to the group that positively indicates the group can never have more than one member, even after hot plug.
- Abstract the data structs used for iommu sva in a separated patches to make it easier for review.
- I still keep the RFC prefix in this series as above two significant changes need at least another round review to be finalized.
- Several misc refinements.
v2:
- https://lore.kernel.org/linux-iommu/[email protected]/
- Add sva domain life cycle management to avoid race between unbind and page fault handling.
- Use a single domain for each mm.
- Return a single sva handler for the same binding.
- Add a new helper to meet singleton group requirement.
- Rework the SVA domain allocation for arm smmu v3 driver and move the pasid_bit initialization to device probe.
- Drop the patch "iommu: Handle IO page faults directly".
- Add mmget_not_zero(mm) in SVA page fault handler.
v1:
- https://lore.kernel.org/linux-iommu/[email protected]/
- Initial post.
Dave Jiang (1):
dmaengine: idxd: Separate user and kernel pasid enabling
Lu Baolu (11):
iommu: Add pasid_bits field in struct dev_iommu
iommu: Add attach/detach_dev_pasid domain ops iommu/sva: Basic data structures for SVA iommu/vt-d: Remove SVM_FLAG_SUPERVISOR_MODE support iommu/vt-d: Add SVA domain support arm-smmu-v3/sva: Add SVA domain support iommu/sva: Use attach/detach_pasid_dev in SVA interfaces
iommu: Remove SVA related callbacks from iommu ops
iommu: Prepare IOMMU domain for IOPF
iommu: Per-domain I/O page fault handling
iommu: Rename iommu-sva-lib.{c,h}
include/linux/intel-iommu.h | 5 +-
include/linux/iommu.h | 106 ++++--
drivers/dma/idxd/idxd.h | 6 +
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 25 +-
.../iommu/{iommu-sva-lib.h => iommu-sva.h} | 8 +-
drivers/dma/idxd/cdev.c | 4 +-
drivers/dma/idxd/init.c | 30 +-
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 85 ++---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 28 +-
drivers/iommu/intel/iommu.c | 20 +-
drivers/iommu/intel/svm.c | 135 +++----
drivers/iommu/io-pgfault.c | 71 +---
drivers/iommu/iommu-sva-lib.c | 71 ----
drivers/iommu/iommu-sva.c | 331 ++++++++++++++++++
drivers/iommu/iommu.c | 219 +++++++-----
drivers/iommu/Makefile | 2 +-
16 files changed, 721 insertions(+), 425 deletions(-)
rename drivers/iommu/{iommu-sva-lib.h => iommu-sva.h} (88%)
delete mode 100644 drivers/iommu/iommu-sva-lib.c
create mode 100644 drivers/iommu/iommu-sva.cSignature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Lu Baolu
maintainerPATCH v4 1/12 dmaengine: idxd: Separate user and kernel pasid enabling
From: Dave Jiang <[email protected]>
The idxd driver always gated the pasid enabling under a single knob and this assumption is incorrect. The pasid used for kernel operation can be independently toggled and has no dependency on the user pasid (and vice versa). Split the two so they are independent "enabled" flags.
--- drivers/dma/idxd/idxd.h | 6 ++++++ drivers/dma/idxd/cdev.c | 4 ++-- drivers/dma/idxd/init.c | 30 ++++++++++++++++++------------ 3 files changed, 26 insertions(+), 14 deletions(-)
drivers/dma/idxd/idxd.h +6 -0
@@ -239,6 +239,7 @@ enum idxd_device_flag {IDXD_FLAG_CONFIGURABLE =0 ,IDXD_FLAG_CMD_RUNNING ,IDXD_FLAG_PASID_ENABLED , + IDXD_FLAG_USER_PASID_ENABLED, }; struct idxd_dma_dev {
@@ -469,6 +470,11 @@ static inline bool device_pasid_enabled(struct idxd_device *idxd)return test_bit (IDXD_FLAG_PASID_ENABLED , &idxd->flags); } +static inline bool device_user_pasid_enabled (struct idxd_device *idxd) +{ +return test_bit (IDXD_FLAG_USER_PASID_ENABLED , &idxd->flags); +} +static inline bool device_swq_supported (struct idxd_device *idxd) {return (support_enqcmd &&device_pasid_enabled (idxd));
drivers/dma/idxd/cdev.c +2 -2
@@ -99,7 +99,7 @@ static int idxd_cdev_open(struct inode *inode, struct file *filp) ctx->wq = wq; filp->private_data = ctx; -if (device_pasid_enabled (idxd)) { +if (device_user_pasid_enabled (idxd)) { sva =iommu_sva_bind_device (dev, current->mm,NULL );if (IS_ERR (sva)) { rc =PTR_ERR (sva);
@@ -152,7 +152,7 @@ static int idxd_cdev_release(struct inode *node, struct file *filep)if (wq_shared (wq)) {idxd_device_drain_pasid (idxd, ctx->pasid); }else { -if (device_pasid_enabled (idxd)) { +if (device_user_pasid_enabled (idxd)) {/* The wq disable in the disable pasid function will drain the wq */ rc =idxd_wq_disable_pasid (wq);if (rc <0 )
drivers/dma/idxd/init.c +18 -12
@@ -513,16 +513,19 @@ static int idxd_probe(struct idxd_device *idxd)if (IS_ENABLED (CONFIG_INTEL_IDXD_SVM ) && sva) { rc =iommu_dev_enable_feature (dev,IOMMU_DEV_FEAT_SVA ); -if (rc ==0 ) { - rc =idxd_enable_system_pasid (idxd); -if (rc <0 ) { -iommu_dev_disable_feature (dev,IOMMU_DEV_FEAT_SVA ); -dev_warn (dev,"Failed to enable PASID. No SVA support: %d\n" , rc); - }else { -set_bit (IDXD_FLAG_PASID_ENABLED , &idxd->flags); - } - }else { +if (rc) { +/* +* Do not bail here since legacy DMA is still +* supported, both user and in-kernel DMA with +* PASID rely on SVA feature. +*/ dev_warn (dev,"Unable to turn on SVA feature.\n" ); + }else { +set_bit (IDXD_FLAG_USER_PASID_ENABLED , &idxd->flags); +if (idxd_enable_system_pasid (idxd)) +dev_warn (dev,"No in-kernel DMA with PASID.\n" ); +else +set_bit (IDXD_FLAG_PASID_ENABLED , &idxd->flags); } }else if (!sva) {dev_warn (dev,"User forced SVA off via module param.\n" );
@@ -561,7 +564,8 @@ static int idxd_probe(struct idxd_device *idxd) err:if (device_pasid_enabled (idxd))idxd_disable_system_pasid (idxd); -iommu_dev_disable_feature (dev,IOMMU_DEV_FEAT_SVA ); +if (device_user_pasid_enabled (idxd) ||device_pasid_enabled (idxd)) +iommu_dev_disable_feature (dev,IOMMU_DEV_FEAT_SVA );return rc; }
@@ -574,7 +578,8 @@ static void idxd_cleanup(struct idxd_device *idxd)idxd_cleanup_internals (idxd);if (device_pasid_enabled (idxd))idxd_disable_system_pasid (idxd); -iommu_dev_disable_feature (dev,IOMMU_DEV_FEAT_SVA ); +if (device_user_pasid_enabled (idxd) ||device_pasid_enabled (idxd)) +iommu_dev_disable_feature (dev,IOMMU_DEV_FEAT_SVA ); }static int idxd_pci_probe (struct pci_dev *pdev,const struct pci_device_id *id)
@@ -691,7 +696,8 @@ static void idxd_remove(struct pci_dev *pdev)free_irq (irq_entry->vector, irq_entry);pci_free_irq_vectors (pdev);pci_iounmap (pdev, idxd->reg_base); -iommu_dev_disable_feature (&pdev->dev,IOMMU_DEV_FEAT_SVA ); +if (device_user_pasid_enabled (idxd) ||device_pasid_enabled (idxd)) +iommu_dev_disable_feature (&pdev->dev,IOMMU_DEV_FEAT_SVA );pci_disable_device (pdev);destroy_workqueue (idxd->wq);perfmon_pmu_remove (idxd);
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Lu Baolu
maintainerPATCH v4 2/12 iommu: Add pasid_bits field in struct dev_iommu
In reply to Lu BaoluUse this field to save the pasid/ssid bits that a device is able to support with its IOMMU hardware. It is a generic attribute of a device and lifting it into the per-device dev_iommu struct makes it possible to allocate a PASID for device without calls into the IOMMU drivers. Any iommu driver which suports PASID related features should set this field before features are enabled on the devices.
For initialization of this field in the VT-d driver, the info->pasid_supported is only set for PCI devices. So the status is that non-PCI SVA hasn't been supported yet. Setting this field only for PCI devices has no functional change.
--- include/linux/iommu.h | 1 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 2 ++ drivers/iommu/intel/iommu.c | 5 ++++- 3 files changed, 7 insertions(+), 1 deletion(-)
include/linux/iommu.h +1 -0
@@ -368,6 +368,7 @@ struct dev_iommu {struct iommu_fwspec *fwspec;struct iommu_device *iommu_dev;void *priv; +unsigned int pasid_bits; }; int iommu_device_register(struct iommu_device *iommu,
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2 -0
@@ -2681,6 +2681,8 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) smmu->features & ARM_SMMU_FEAT_STALL_FORCE) master->stall_enabled = true; + dev->iommu->pasid_bits = master->ssid_bits; +return &smmu->iommu; err_free_master:
drivers/iommu/intel/iommu.c +4 -1
@@ -4593,8 +4593,11 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)if (pasid_supported (iommu)) {int features =pci_pasid_features (pdev); -if (features >=0 ) +if (features >=0 ) { info->pasid_supported = features |1 ; + dev->iommu->pasid_bits = +fls (pci_max_pasids (pdev)) -1 ; + } } if (info->ats_supported && ecap_prs(iommu->ecap) &&
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Lu Baolu
maintainerPATCH v4 3/12 iommu: Add attach/detach_dev_pasid domain ops
In reply to Lu BaoluAttaching an IOMMU domain to a PASID of a device is a generic operation for modern IOMMU drivers which support PASID-granular DMA address translation. Currently visible usage scenarios include (but not limited):
- SVA (Shared Virtual Address)
- kernel DMA with PASID
- hardware-assist mediated device
This adds a pair of common domain ops for this purpose and adds helpers to attach/detach a domain to/from a {device, PASID}. Some buses, like PCI, route packets without considering the PASID value. Thus a DMA target address with PASID might be treated as P2P if the address falls into the MMIO BAR of other devices in the group. To make things simple, these interfaces only apply to devices belonging to the singleton groups, and the singleton is immutable in fabric i.e. not affected by hotplug.
--- include/linux/iommu.h | 21 ++++++++++ drivers/iommu/iommu.c | 95 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+)
include/linux/iommu.h +21 -0
@@ -262,6 +262,8 @@ struct iommu_ops {* struct iommu_domain_ops - domain specific operations * @attach_dev: attach an iommu domain to a device * @detach_dev: detach an iommu domain from a device +* @attach_dev_pasid: attach an iommu domain to a pasid of device +* @detach_dev_pasid: detach an iommu domain from a pasid of device * @map: map a physically contiguous memory region to an iommu domain * @map_pages: map a physically contiguous set of pages of the same size to * an iommu domain.
@@ -279,6 +281,10 @@ struct iommu_ops {struct iommu_domain_ops {int (*attach_dev)(struct iommu_domain *domain,struct device *dev);void (*detach_dev)(struct iommu_domain *domain,struct device *dev); +int (*attach_dev_pasid)(struct iommu_domain *domain, +struct device *dev,ioasid_t pasid); +void (*detach_dev_pasid)(struct iommu_domain *domain, +struct device *dev,ioasid_t pasid);int (*map)(struct iommu_domain *domain,unsigned long iova,phys_addr_t paddr,size_t size,int prot,gfp_t gfp);
@@ -672,6 +678,10 @@ int iommu_group_claim_dma_owner(struct iommu_group *group, void *owner);void iommu_group_release_dma_owner (struct iommu_group *group);bool iommu_group_dma_owner_claimed (struct iommu_group *group); +int iommu_attach_device_pasid (struct iommu_domain *domain, +struct device *dev,ioasid_t pasid); +void iommu_detach_device_pasid (struct iommu_domain *domain, +struct device *dev,ioasid_t pasid);#else /* CONFIG_IOMMU_API */ struct iommu_ops {};
@@ -1040,6 +1050,17 @@ static inline bool iommu_group_dma_owner_claimed(struct iommu_group *group) {return false; } + +static inline int iommu_attach_device_pasid (struct iommu_domain *domain, +struct device *dev,ioasid_t pasid) +{ +return -ENODEV ; +} + +static inline void iommu_detach_device_pasid (struct iommu_domain *domain, +struct device *dev,ioasid_t pasid) +{ +}#endif /* CONFIG_IOMMU_API */ /**
drivers/iommu/iommu.c +95 -0
110 lines not on this page · raw message
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v4 03/12] iommu: Add attach/detach_dev_pasid domain ops
Quoted 17 lines · Lu Baolu
Attaching an IOMMU domain to a PASID of a device is a generic operation for modern IOMMU drivers which support PASID-granular DMA address translation. Currently visible usage scenarios include (but not limited):
- SVA (Shared Virtual Address)
- kernel DMA with PASID
- hardware-assist mediated deviceThis adds a pair of common domain ops for this purpose and adds helpers to attach/detach a domain to/from a {device, PASID}. Some buses, like PCI, route packets without considering the PASID value. Thus a DMA target address with PASID might be treated as P2P if the address falls into the MMIO BAR of other devices in the group. To make things simple, these interfaces only apply to devices belonging to the singleton groups, and the singleton is immutable in fabric i.e. not affected by hotplug.
Signed-off-by Lu Baolu <[email protected]>
[...]
Quoted 26 lines
drivers/iommu/iommu.c +26
-0+/* +* Use standard PCI bus topology, isolation features, and DMA +* alias quirks to check the immutable singleton attribute. If +* the device came from DT, assume it is static and then +* singleton can know from the device count in the group. +*/ +static bool device_group_immutable_singleton (struct device *dev) +{ +struct iommu_group *group =iommu_group_get (dev); +int count; + +if (!group) +return false; + +mutex_lock (&group->mutex); + count =iommu_group_device_count (group); +mutex_unlock (&group->mutex); +iommu_group_put (group); + +if (count !=1 ) +return false; + +if (dev_is_pci (dev)) { +struct pci_dev *pdev =to_pci_dev (dev); + +/*
drivers/iommu/iommu.c bool iommu_group_dma_owner_claimed(struct iommu_group *group) · line 3213
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+* The device could be considered to be fully isolated if +* all devices on the path from the device to the host-PCI +* bridge are protected from peer-to-peer DMA by ACS. +*/ +if (!pci_acs_path_enabled (pdev,NULL ,REQ_ACS_FLAGS )) +return false; + +/* Filter out devices which has any alias device. */ +if (pci_for_each_dma_alias (pdev, has_pci_alias, pdev)) +return false;
Aren't aliases already added to the group by pci_device_group()? If so it's part of the count check above
drivers/iommu/iommu.c bool iommu_group_dma_owner_claimed(struct iommu_group *group) · line 3222
Quoted 9 lines
drivers/iommu/iommu.c +9
-0+ +return true; + } + +/* +* If the device came from DT, assume it is static and then +* singleton can know from the device count in the group. +*/ +return is_of_node (dev_fwnode (dev));
I don't think DT is relevant here because a platform device enumerated through ACPI will also have its own group. It should be safe to stick to what the IOMMU drivers declare with their device_group() callback. Except for PCI those groups should be immutable so we can return true here.
Thanks,
Jean
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Baolu Lu
maintainerRe: [PATCH v4 03/12] iommu: Add attach/detach_dev_pasid domain ops
Quoted 50 lines · Jean-Philippe Brucker
Quoted 17 lines · Lu Baolu
Attaching an IOMMU domain to a PASID of a device is a generic operation for modern IOMMU drivers which support PASID-granular DMA address translation. Currently visible usage scenarios include (but not limited):
- SVA (Shared Virtual Address)
- kernel DMA with PASID
- hardware-assist mediated deviceThis adds a pair of common domain ops for this purpose and adds helpers to attach/detach a domain to/from a {device, PASID}. Some buses, like PCI, route packets without considering the PASID value. Thus a DMA target address with PASID might be treated as P2P if the address falls into the MMIO BAR of other devices in the group. To make things simple, these interfaces only apply to devices belonging to the singleton groups, and the singleton is immutable in fabric i.e. not affected by hotplug.
Signed-off-by Lu Baolu <[email protected]>[...]
Quoted 31 lines
drivers/iommu/iommu.c +31
-0+/* +* Use standard PCI bus topology, isolation features, and DMA +* alias quirks to check the immutable singleton attribute. If +* the device came from DT, assume it is static and then +* singleton can know from the device count in the group. +*/ +static bool device_group_immutable_singleton (struct device *dev) +{ +struct iommu_group *group =iommu_group_get (dev); +int count; + +if (!group) +return false; + +mutex_lock (&group->mutex); + count =iommu_group_device_count (group); +mutex_unlock (&group->mutex); +iommu_group_put (group); + +if (count !=1 ) +return false; + +if (dev_is_pci (dev)) { +struct pci_dev *pdev =to_pci_dev (dev); + +/* +* The device could be considered to be fully isolated if +* all devices on the path from the device to the host-PCI +* bridge are protected from peer-to-peer DMA by ACS. +*/ +if (!pci_acs_path_enabled (pdev,NULL ,REQ_ACS_FLAGS ))
Quoted 8 lines
drivers/iommu/iommu.c bool iommu_group_dma_owner_claimed(struct iommu_group *group) · line 3213
Quoted 5 lines
drivers/iommu/iommu.c +5
-0+return false; + +/* Filter out devices which has any alias device. */ +if (pci_for_each_dma_alias (pdev, has_pci_alias, pdev)) +return false;Aren't aliases already added to the group by
pci_device_group()? If so it's part of the count check above
You are right. pci_device_group() has already covered pci aliases.
Quoted 15 lines
drivers/iommu/iommu.c bool iommu_group_dma_owner_claimed(struct iommu_group *group) · line 3222
Quoted 10 lines
drivers/iommu/iommu.c +9
-0+ +return true; + } + +/* +* If the device came from DT, assume it is static and then +* singleton can know from the device count in the group. +*/ +return is_of_node (dev_fwnode (dev));I don't think DT is relevant here because a platform device enumerated through ACPI will also have its own group. It should be safe to stick to what the IOMMU drivers declare with their
device_group()callback. Except for PCI those groups should be immutable so we can return true here.
Fair enough. My code is too restrict. The group singleton is immutable as long as the fabric is static or ACS (or similar) technology is implemented. Currently we only need to care about PCI as far as I can see.
Quoted 3 lines
Thanks,
Jean
Best regards,
baolu
Lu Baolu
maintainerPATCH v4 4/12 iommu/sva: Basic data structures for SVA
In reply to Lu BaoluUse below data structures for SVA implementation in the IOMMU core:
- struct iommu_sva_ioas
Represent the I/O address space shared with an application CPU address space. This structure has a 1:1 relationship with an mm_struct. It grabs a "mm->mm_count" refcount during creation and drop it on release.
- struct iommu_domain (IOMMU_DOMAIN_SVA type)
Represent a hardware pagetable that the IOMMU hardware could use for SVA translation. Multiple iommu domains could be bound with an SVA ioas and each grabs a refcount from ioas in order to make sure ioas could only be freed after all domains have been unbound.
- struct iommu_sva
Represent a bond relationship between an SVA ioas and an iommu domain. If a bond already exists, it's reused and a reference is taken.
--- include/linux/iommu.h | 14 +++++++++++++- drivers/iommu/iommu-sva-lib.h | 1 + drivers/iommu/iommu-sva-lib.c | 18 ++++++++++++++++++ 3 files changed, 32 insertions(+), 1 deletion(-)
include/linux/iommu.h +13 -1
@@ -42,6 +42,7 @@ struct notifier_block;struct iommu_sva ;struct iommu_fault_event ;struct iommu_dma_cookie ; +struct iommu_sva_ioas ;/* iommu fault flags */ #define IOMMU_FAULT_READ 0x0
@@ -64,6 +65,9 @@ struct iommu_domain_geometry {#define __IOMMU_DOMAIN_PT (1U << 2)/* Domain is identity mapped */ #define __IOMMU_DOMAIN_DMA_FQ (1U << 3)/* DMA-API uses flush queue */ +#define __IOMMU_DOMAIN_SHARED (1U << 4)/* Page table shared from CPU */ +#define __IOMMU_DOMAIN_HOST_VA (1U << 5)/* Host CPU virtual address */ +/* * This are the possible domain-types *
@@ -86,6 +90,8 @@ struct iommu_domain_geometry {#define IOMMU_DOMAIN_DMA_FQ (__IOMMU_DOMAIN_PAGING | \ __IOMMU_DOMAIN_DMA_API | \ __IOMMU_DOMAIN_DMA_FQ) +#define IOMMU_DOMAIN_SVA (__IOMMU_DOMAIN_SHARED | \ + __IOMMU_DOMAIN_HOST_VA)struct iommu_domain {unsigned type;
@@ -95,6 +101,7 @@ struct iommu_domain {void *handler_token;struct iommu_domain_geometry geometry;struct iommu_dma_cookie *iova_cookie; +struct iommu_sva_ioas *sva_ioas; };static inline bool iommu_is_dma_domain (struct iommu_domain *domain)
@@ -622,7 +629,12 @@ struct iommu_fwspec {* struct iommu_sva - handle to a device-mm bond */ struct iommu_sva { -struct device *dev; +struct device *dev; +struct iommu_sva_ioas *sva_ioas; +struct iommu_domain *domain; +/* Link to sva ioas's bonds list */ +struct list_head node; +refcount_t users; }; int iommu_fwspec_init(struct device *dev, struct fwnode_handle *iommu_fwnode,
drivers/iommu/iommu-sva-lib.h +1 -0
@@ -10,6 +10,7 @@int iommu_sva_alloc_pasid (struct mm_struct *mm,ioasid_t min,ioasid_t max);struct mm_struct *iommu_sva_find (ioasid_t pasid); +struct mm_struct *iommu_sva_domain_mm (struct iommu_domain *domain);/* I/O Page fault */ struct device ;
drivers/iommu/iommu-sva-lib.c +18 -0
@@ -3,6 +3,8 @@* Helpers for IOMMU drivers implementing SVA */ #include <linux/mutex.h> +#include <linux/iommu.h> +#include <linux/slab.h> #include <linux/sched/mm.h> #include "iommu-sva-lib.h"
@@ -10,6 +12,22 @@static DEFINE_MUTEX (iommu_sva_lock);static DECLARE_IOASID_SET (iommu_sva_pasid); +struct iommu_sva_ioas { +struct mm_struct *mm; +ioasid_t pasid; + +/* Counter of domains attached to this ioas. */ +refcount_t users; + +/* All bindings are linked here. */ +struct list_head bonds; +}; + +struct mm_struct *iommu_sva_domain_mm (struct iommu_domain *domain) +{ +return domain->sva_ioas->mm; +} +/** * iommu_sva_alloc_pasid - Allocate a PASID for the mm * @mm: the mm
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Lu Baolu
maintainerPATCH v4 5/12 iommu/vt-d: Remove SVM_FLAG_SUPERVISOR_MODE support
In reply to Lu BaoluThe current kernel DMA with PASID support is based on the SVA with a flag SVM_FLAG_SUPERVISOR_MODE. The IOMMU driver binds the kernel memory address space to a PASID of the device. The device driver programs the device with kernel virtual address (KVA) for DMA access. There have been security and functional issues with this approach:
- The lack of IOTLB synchronization upon kernel page table updates. (vmalloc, module/BPF loading, CONFIG_DEBUG_PAGEALLOC etc.)
- Other than slight more protection, using kernel virtual address (KVA) has little advantage over physical address. There are also no use cases yet where DMA engines need kernel virtual addresses for in-kernel DMA.
This removes SVM_FLAG_SUPERVISOR_MODE support in the Intel IOMMU driver. The device driver is suggested to handle kernel DMA with PASID through the kernel DMA APIs.
--- drivers/iommu/intel/svm.c | 53 +++++++++------------------------------ 1 file changed, 12 insertions(+), 41 deletions(-)
drivers/iommu/intel/svm.c +12 -41
@@ -313,8 +313,7 @@ static int pasid_to_svm_sdev(struct device *dev, unsigned int pasid,return 0 ; } -static int intel_svm_alloc_pasid (struct device *dev,struct mm_struct *mm, -unsigned int flags) +static int intel_svm_alloc_pasid (struct device *dev,struct mm_struct *mm) {ioasid_t max_pasid =dev_is_pci (dev) ?pci_max_pasids (to_pci_dev (dev)) : intel_pasid_max_id;
@@ -324,8 +323,7 @@ static int intel_svm_alloc_pasid(struct device *dev, struct mm_struct *mm,static struct iommu_sva *intel_svm_bind_mm (struct intel_iommu *iommu,struct device *dev, -struct mm_struct *mm, -unsigned int flags) +struct mm_struct *mm) {struct device_domain_info *info =dev_iommu_priv_get (dev);unsigned long iflags, sflags;
@@ -341,22 +339,18 @@ static struct iommu_sva *intel_svm_bind_mm(struct intel_iommu *iommu, svm->pasid = mm->pasid; svm->mm = mm; - svm->flags = flags;INIT_LIST_HEAD_RCU (&svm->devs); -if (!(flags &SVM_FLAG_SUPERVISOR_MODE )) { - svm->notifier.ops = &intel_mmuops; - ret =mmu_notifier_register (&svm->notifier, mm); -if (ret) { -kfree (svm); -return ERR_PTR (ret); - } + svm->notifier.ops = &intel_mmuops; + ret =mmu_notifier_register (&svm->notifier, mm); +if (ret) { +kfree (svm); +return ERR_PTR (ret); } ret =pasid_private_add (svm->pasid, svm);if (ret) { -if (svm->notifier.ops) -mmu_notifier_unregister (&svm->notifier, mm); +mmu_notifier_unregister (&svm->notifier, mm);kfree (svm);return ERR_PTR (ret); }
@@ -391,9 +385,7 @@ static struct iommu_sva *intel_svm_bind_mm(struct intel_iommu *iommu, }/* Setup the pasid table: */ - sflags = (flags &SVM_FLAG_SUPERVISOR_MODE ) ? -PASID_FLAG_SUPERVISOR_MODE :0 ; - sflags |=cpu_feature_enabled (X86_FEATURE_LA57 ) ?PASID_FLAG_FL5LP :0 ; + sflags =cpu_feature_enabled (X86_FEATURE_LA57 ) ?PASID_FLAG_FL5LP :0 ;spin_lock_irqsave (&iommu->lock, iflags); ret =intel_pasid_setup_first_level (iommu, dev, mm->pgd, mm->pasid,FLPT_DEFAULT_DID , sflags);
@@ -410,8 +402,7 @@ static struct iommu_sva *intel_svm_bind_mm(struct intel_iommu *iommu,kfree (sdev); free_svm:if (list_empty (&svm->devs)) { -if (svm->notifier.ops) -mmu_notifier_unregister (&svm->notifier, mm); +mmu_notifier_unregister (&svm->notifier, mm);pasid_private_remove (mm->pasid);kfree (svm); }
@@ -817,37 +808,17 @@ static irqreturn_t prq_event_thread(int irq, void *d)struct iommu_sva *intel_svm_bind (struct device *dev,struct mm_struct *mm,void *drvdata) {struct intel_iommu *iommu =device_to_iommu (dev,NULL ,NULL ); -unsigned int flags =0 ;struct iommu_sva *sva;int ret; -if (drvdata) - flags = *(unsigned int *)drvdata; - -if (flags &SVM_FLAG_SUPERVISOR_MODE ) { -if (!ecap_srs (iommu->ecap)) { -dev_err (dev,"%s: Supervisor PASID not supported\n" , - iommu->name); -return ERR_PTR (-EOPNOTSUPP ); - } - -if (mm) { -dev_err (dev,"%s: Supervisor PASID with user provided mm\n" , - iommu->name); -return ERR_PTR (-EINVAL ); - } - - mm = &init_mm; - } -mutex_lock (&pasid_mutex); - ret =intel_svm_alloc_pasid (dev, mm, flags); + ret =intel_svm_alloc_pasid (dev, mm);if (ret) {mutex_unlock (&pasid_mutex);return ERR_PTR (ret); } - sva =intel_svm_bind_mm (iommu, dev, mm, flags); + sva =intel_svm_bind_mm (iommu, dev, mm);mutex_unlock (&pasid_mutex);return sva;
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Lu Baolu
maintainerPATCH v4 6/12 iommu/vt-d: Add SVA domain support
In reply to Lu BaoluAdd support for SVA domain allocation and provide an SVA-specific iommu_domain_ops.
--- include/linux/intel-iommu.h | 1 + drivers/iommu/intel/iommu.c | 10 ++++++++++ drivers/iommu/intel/svm.c | 37 +++++++++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+)
include/linux/intel-iommu.h +1 -0
@@ -744,6 +744,7 @@ void intel_svm_unbind(struct iommu_sva *handle);u32 intel_svm_get_pasid (struct iommu_sva *handle);int intel_svm_page_response (struct device *dev,struct iommu_fault_event *evt,struct iommu_page_response *msg); +extern const struct iommu_domain_ops intel_svm_domain_ops;struct intel_svm_dev {struct list_head list;
drivers/iommu/intel/iommu.c +10 -0
@@ -4324,6 +4324,16 @@ static struct iommu_domain *intel_iommu_domain_alloc(unsigned type)return domain;case IOMMU_DOMAIN_IDENTITY :return &si_domain->domain; +#ifdef CONFIG_INTEL_IOMMU_SVM +case IOMMU_DOMAIN_SVA : + dmar_domain =alloc_domain (type); +if (!dmar_domain) +return NULL ; + domain = &dmar_domain->domain; + domain->ops = &intel_svm_domain_ops; + +return domain; +#endif /* CONFIG_INTEL_IOMMU_SVM */ default :return NULL ; }
drivers/iommu/intel/svm.c +37 -0
@@ -927,3 +927,40 @@ int intel_svm_page_response(struct device *dev,mutex_unlock (&pasid_mutex);return ret; } + +static int intel_svm_attach_dev_pasid (struct iommu_domain *domain, +struct device *dev,ioasid_t pasid) +{ +struct device_domain_info *info =dev_iommu_priv_get (dev); +struct mm_struct *mm =iommu_sva_domain_mm (domain); +struct intel_iommu *iommu = info->iommu; +struct iommu_sva *sva; +int ret =0 ; + +mutex_lock (&pasid_mutex); + sva =intel_svm_bind_mm (iommu, dev, mm); +if (IS_ERR (sva)) + ret =PTR_ERR (sva); +mutex_unlock (&pasid_mutex); + +return ret; +} + +static void intel_svm_detach_dev_pasid (struct iommu_domain *domain, +struct device *dev,ioasid_t pasid) +{ +mutex_lock (&pasid_mutex); +intel_svm_unbind_mm (dev, pasid); +mutex_unlock (&pasid_mutex); +} + +static void intel_svm_domain_free (struct iommu_domain *domain) +{ +kfree (to_dmar_domain (domain)); +} + +const struct iommu_domain_ops intel_svm_domain_ops = { + .attach_dev_pasid = intel_svm_attach_dev_pasid, + .detach_dev_pasid = intel_svm_detach_dev_pasid, + .free = intel_svm_domain_free, +};
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Lu Baolu
maintainerPATCH v4 7/12 arm-smmu-v3/sva: Add SVA domain support
In reply to Lu BaoluAdd support for SVA domain allocation and provide an SVA-specific iommu_domain_ops.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +++++++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 42 +++++++++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 21 ++++++++++ 3 files changed, 77 insertions(+)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +14 -0
@@ -759,6 +759,10 @@ struct iommu_sva *arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,void arm_smmu_sva_unbind (struct iommu_sva *handle);u32 arm_smmu_sva_get_pasid (struct iommu_sva *handle);void arm_smmu_sva_notifier_synchronize (void ); +int arm_smmu_sva_attach_dev_pasid (struct iommu_domain *domain, +struct device *dev,ioasid_t id); +void arm_smmu_sva_detach_dev_pasid (struct iommu_domain *domain, +struct device *dev,ioasid_t id);#else /* CONFIG_ARM_SMMU_V3_SVA */ static inline bool arm_smmu_sva_supported (struct arm_smmu_device *smmu) {
@@ -804,5 +808,15 @@ static inline u32 arm_smmu_sva_get_pasid(struct iommu_sva *handle) }static inline void arm_smmu_sva_notifier_synchronize (void ) {} + +static inline int arm_smmu_sva_attach_dev_pasid (struct iommu_domain *domain, +struct device *dev,ioasid_t id) +{ +return -ENODEV ; +} + +static inline void arm_smmu_sva_detach_dev_pasid (struct iommu_domain *domain, +struct device *dev, +ioasid_t id) {}#endif /* CONFIG_ARM_SMMU_V3_SVA */ #endif /* _ARM_SMMU_V3_H */
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +42 -0
45 lines not on this page · raw message
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +21 -0
@@ -1995,10 +1995,31 @@ static bool arm_smmu_capable(enum iommu_cap cap) } } +static void arm_smmu_sva_domain_free (struct iommu_domain *domain) +{ +kfree (domain); +} + +static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { + .attach_dev_pasid = arm_smmu_sva_attach_dev_pasid, + .detach_dev_pasid = arm_smmu_sva_detach_dev_pasid, + .free = arm_smmu_sva_domain_free, +}; +static struct iommu_domain *arm_smmu_domain_alloc (unsigned type) {struct arm_smmu_domain *smmu_domain; +if (type ==IOMMU_DOMAIN_SVA ) { +struct iommu_domain *domain; + + domain =kzalloc (sizeof (*domain),GFP_KERNEL ); +if (domain) + domain->ops = &arm_smmu_sva_domain_ops; + +return domain; + } + if (type != IOMMU_DOMAIN_UNMANAGED && type != IOMMU_DOMAIN_DMA && type != IOMMU_DOMAIN_DMA_FQ &&
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Lu Baolu
maintainerPATCH v4 8/12 iommu/sva: Use attach/detach_pasid_dev in SVA interfaces
In reply to Lu BaoluThe existing iommu SVA interfaces are implemented by calling the SVA specific iommu ops provided by the IOMMU drivers. There's no need for any SVA specific ops in iommu_ops vector anymore as we can achieve this through the generic attach/detach_dev_pasid domain ops.
This refactors the IOMMU SVA interfaces implementation by using the attach/detach_pasid_dev ops and align them with the concept of the iommu domain. Put the new SVA code in the sva related file in order to make it self-contained.
--- include/linux/iommu.h | 44 ++++---- drivers/iommu/iommu-sva-lib.h | 5 + drivers/iommu/iommu-sva-lib.c | 196 ++++++++++++++++++++++++++++++++++ drivers/iommu/iommu.c | 95 +--------------- 4 files changed, 227 insertions(+), 113 deletions(-)
include/linux/iommu.h +23 -21
@@ -677,12 +677,6 @@ int iommu_dev_enable_feature(struct device *dev, enum iommu_dev_features f);int iommu_dev_disable_feature (struct device *dev,enum iommu_dev_features f);bool iommu_dev_feature_enabled (struct device *dev,enum iommu_dev_features f); -struct iommu_sva *iommu_sva_bind_device (struct device *dev, -struct mm_struct *mm, -void *drvdata); -void iommu_sva_unbind_device (struct iommu_sva *handle); -u32 iommu_sva_get_pasid (struct iommu_sva *handle); -int iommu_device_use_default_domain (struct device *dev);void iommu_device_unuse_default_domain (struct device *dev);
@@ -1019,21 +1013,6 @@ iommu_dev_disable_feature(struct device *dev, enum iommu_dev_features feat)return -ENODEV ; } -static inline struct iommu_sva * -iommu_sva_bind_device (struct device *dev,struct mm_struct *mm,void *drvdata) -{ -return NULL ; -} - -static inline void iommu_sva_unbind_device (struct iommu_sva *handle) -{ -} - -static inline u32 iommu_sva_get_pasid (struct iommu_sva *handle) -{ -return IOMMU_PASID_INVALID ; -} -static inline struct iommu_fwspec *dev_iommu_fwspec_get (struct device *dev) {return NULL ;
@@ -1075,6 +1054,29 @@ static inline void iommu_detach_device_pasid(struct iommu_domain *domain, }#endif /* CONFIG_IOMMU_API */ +#ifdef CONFIG_IOMMU_SVA +struct iommu_sva *iommu_sva_bind_device (struct device *dev, +struct mm_struct *mm, +void *drvdata); +void iommu_sva_unbind_device (struct iommu_sva *handle); +u32 iommu_sva_get_pasid (struct iommu_sva *handle); +#else /* CONFIG_IOMMU_SVA */ +static inline struct iommu_sva * +iommu_sva_bind_device (struct device *dev,struct mm_struct *mm,void *drvdata) +{ +return NULL ; +} + +static inline void iommu_sva_unbind_device (struct iommu_sva *handle) +{ +} + +static inline u32 iommu_sva_get_pasid (struct iommu_sva *handle) +{ +return IOMMU_PASID_INVALID ; +} +#endif /* CONFIG_IOMMU_SVA */ +/** * iommu_map_sgtable - Map the given buffer to the IOMMU domain * @domain: The IOMMU domain to perform the mapping
drivers/iommu/iommu-sva-lib.h +5 -0
@@ -27,6 +27,7 @@ int iopf_queue_flush_dev(struct device *dev);struct iopf_queue *iopf_queue_alloc (const char *name);void iopf_queue_free (struct iopf_queue *queue);int iopf_queue_discard_partial (struct iopf_queue *queue); +void iommu_sva_ioas_put (struct iommu_sva_ioas *ioas);#else /* CONFIG_IOMMU_SVA */ static inline int iommu_queue_iopf (struct iommu_fault *fault,void *cookie)
@@ -64,5 +65,9 @@ static inline int iopf_queue_discard_partial(struct iopf_queue *queue) {return -ENODEV ; } + +static inline void iommu_sva_ioas_put (struct iommu_sva_ioas *ioas) +{ +}#endif /* CONFIG_IOMMU_SVA */ #endif /* _IOMMU_SVA_LIB_H */
drivers/iommu/iommu-sva-lib.c +196 -0
205 lines not on this page · raw message
drivers/iommu/iommu.c +3 -92
113 lines not on this page · raw message
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Lu Baolu
maintainerPATCH v4 9/12 iommu: Remove SVA related callbacks from iommu ops
In reply to Lu BaoluThese ops'es have been replaced with the dev_attach/detach_pasid domain ops'es. There's no need for them anymore. Remove them to avoid dead code.
--- include/linux/intel-iommu.h | 4 -- include/linux/iommu.h | 8 --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 17 ------- drivers/iommu/iommu-sva-lib.h | 1 - .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 41 ---------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 -- drivers/iommu/intel/iommu.c | 3 -- drivers/iommu/intel/svm.c | 49 ------------------- drivers/iommu/iommu-sva-lib.c | 4 +- 9 files changed, 2 insertions(+), 128 deletions(-)
include/linux/intel-iommu.h +0 -4
@@ -738,10 +738,6 @@ struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn);extern void intel_svm_check (struct intel_iommu *iommu);extern int intel_svm_enable_prq (struct intel_iommu *iommu);extern int intel_svm_finish_prq (struct intel_iommu *iommu); -struct iommu_sva *intel_svm_bind (struct device *dev,struct mm_struct *mm, -void *drvdata); -void intel_svm_unbind (struct iommu_sva *handle); -u32 intel_svm_get_pasid (struct iommu_sva *handle);int intel_svm_page_response (struct device *dev,struct iommu_fault_event *evt,struct iommu_page_response *msg);extern const struct iommu_domain_ops intel_svm_domain_ops;
include/linux/iommu.h +0 -8
@@ -213,9 +213,6 @@ struct iommu_iotlb_gather {* @dev_has/enable/disable_feat: per device entries to check/enable/disable * iommu specific features. * @dev_feat_enabled: check enabled feature -* @sva_bind: Bind process address space to device -* @sva_unbind: Unbind process address space from device -* @sva_get_pasid: Get PASID associated to a SVA handle * @page_response: handle page request response * @def_domain_type: device default domain type, return value: * - IOMMU_DOMAIN_IDENTITY: must use an identity domain
@@ -249,11 +246,6 @@ struct iommu_ops {int (*dev_enable_feat)(struct device *dev,enum iommu_dev_features f);int (*dev_disable_feat)(struct device *dev,enum iommu_dev_features f); -struct iommu_sva *(*sva_bind)(struct device *dev,struct mm_struct *mm, -void *drvdata); -void (*sva_unbind)(struct iommu_sva *handle); -u32 (*sva_get_pasid)(struct iommu_sva *handle); -int (*page_response)(struct device *dev,struct iommu_fault_event *evt,struct iommu_page_response *msg);
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0 -17
@@ -754,10 +754,6 @@ bool arm_smmu_master_sva_enabled(struct arm_smmu_master *master);int arm_smmu_master_enable_sva (struct arm_smmu_master *master);int arm_smmu_master_disable_sva (struct arm_smmu_master *master);bool arm_smmu_master_iopf_supported (struct arm_smmu_master *master); -struct iommu_sva *arm_smmu_sva_bind (struct device *dev,struct mm_struct *mm, -void *drvdata); -void arm_smmu_sva_unbind (struct iommu_sva *handle); -u32 arm_smmu_sva_get_pasid (struct iommu_sva *handle);void arm_smmu_sva_notifier_synchronize (void );int arm_smmu_sva_attach_dev_pasid (struct iommu_domain *domain,struct device *dev,ioasid_t id);
@@ -794,19 +790,6 @@ static inline bool arm_smmu_master_iopf_supported(struct arm_smmu_master *masterreturn false; } -static inline struct iommu_sva * -arm_smmu_sva_bind (struct device *dev,struct mm_struct *mm,void *drvdata) -{ -return ERR_PTR (-ENODEV ); -} - -static inline void arm_smmu_sva_unbind (struct iommu_sva *handle) {} - -static inline u32 arm_smmu_sva_get_pasid (struct iommu_sva *handle) -{ -return IOMMU_PASID_INVALID ; -} -static inline void arm_smmu_sva_notifier_synchronize (void ) {} static inline int arm_smmu_sva_attach_dev_pasid(struct iommu_domain *domain,
drivers/iommu/iommu-sva-lib.h +0 -1
@@ -8,7 +8,6 @@#include <linux/ioasid.h> #include <linux/mm_types.h> -int iommu_sva_alloc_pasid (struct mm_struct *mm,ioasid_t min,ioasid_t max);struct mm_struct *iommu_sva_find (ioasid_t pasid);struct mm_struct *iommu_sva_domain_mm (struct iommu_domain *domain);
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +0 -41
53 lines not on this page · raw message
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +0 -3
@@ -2875,9 +2875,6 @@ static struct iommu_ops arm_smmu_ops = { .dev_feat_enabled = arm_smmu_dev_feature_enabled, .dev_enable_feat = arm_smmu_dev_enable_feature, .dev_disable_feat = arm_smmu_dev_disable_feature, - .sva_bind = arm_smmu_sva_bind, - .sva_unbind = arm_smmu_sva_unbind, - .sva_get_pasid = arm_smmu_sva_get_pasid, .page_response = arm_smmu_page_response, .pgsize_bitmap = -1UL, /* Restricted during device attach */ .owner = THIS_MODULE,
drivers/iommu/intel/iommu.c +0 -3
@@ -4896,9 +4896,6 @@ const struct iommu_ops intel_iommu_ops = { .def_domain_type = device_def_domain_type, .pgsize_bitmap =SZ_4K , #ifdef CONFIG_INTEL_IOMMU_SVM - .sva_bind = intel_svm_bind, - .sva_unbind = intel_svm_unbind, - .sva_get_pasid = intel_svm_get_pasid, .page_response = intel_svm_page_response, #endif .default_domain_ops = &(const struct iommu_domain_ops ) {
drivers/iommu/intel/svm.c +0 -49
61 lines not on this page · raw message
drivers/iommu/iommu-sva-lib.c +2 -2
@@ -41,7 +41,8 @@ struct mm_struct *iommu_sva_domain_mm(struct iommu_domain *domain)* * Returns 0 on success and < 0 on error. */ -int iommu_sva_alloc_pasid (struct mm_struct *mm,ioasid_t min,ioasid_t max) +static int iommu_sva_alloc_pasid (struct mm_struct *mm, +ioasid_t min,ioasid_t max) {int ret =0 ;ioasid_t pasid;
@@ -67,7 +68,6 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max)mutex_unlock (&iommu_sva_lock);return ret; } -EXPORT_SYMBOL_GPL (iommu_sva_alloc_pasid);/* ioasid_find getter() requires a void * argument */ static bool __mmget_not_zero (void *mm)
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Lu Baolu
maintainerPATCH v4 10/12 iommu: Prepare IOMMU domain for IOPF
In reply to Lu BaoluThis adds some mechanisms around the iommu_domain so that the I/O page fault handling framework could route a page fault to the domain and call the fault handler from it.
It includes:
- A reference counter for the iommu domain. The page fault handler is a different context from the bind()/unbind() thread. A reference counter is added to struct iommu_domain so that the life cycle of the iommu_domain could be synced between different threads.
- A pointer to the page fault handler and its private data. The fault handler will be called with the private data as a parameter once a page fault is routed to the domain. Any kernel component which owns an iommu domain could install handler and its parameter so that the page fault could be further routed or handled.
This also prepares the SVA implementation to be the first consumer of the per-domain page fault handling model.
--- include/linux/iommu.h | 18 ++++++++++ drivers/iommu/iommu-sva-lib.c | 68 ++++++++++++++++++++++++++++++++++- drivers/iommu/iommu.c | 29 +++++++++++++++ 3 files changed, 114 insertions(+), 1 deletion(-)
include/linux/iommu.h +18 -0
@@ -102,6 +102,10 @@ struct iommu_domain {struct iommu_domain_geometry geometry;struct iommu_dma_cookie *iova_cookie;struct iommu_sva_ioas *sva_ioas; +enum iommu_page_response_code (*iopf_handler)(struct iommu_fault *fault, +void *data); +void *fault_data; +refcount_t async_users; };static inline bool iommu_is_dma_domain (struct iommu_domain *domain)
@@ -680,6 +684,14 @@ int iommu_attach_device_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t pasid);void iommu_detach_device_pasid (struct iommu_domain *domain,struct device *dev,ioasid_t pasid); +struct iommu_domain * +iommu_get_domain_for_dev_pasid_async (struct device *dev,ioasid_t pasid); + +static inline void iommu_domain_put_async (struct iommu_domain *domain) +{ +if (refcount_dec_and_test (&domain->async_users)) +iommu_domain_free (domain); +}#else /* CONFIG_IOMMU_API */ struct iommu_ops {};
@@ -1044,6 +1056,12 @@ static inline void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t pasid) { } + +static inline struct iommu_domain * +iommu_get_domain_for_dev_pasid_async (struct device *dev,ioasid_t pasid) +{ +return NULL ; +}#endif /* CONFIG_IOMMU_API */ #ifdef CONFIG_IOMMU_SVA
drivers/iommu/iommu-sva-lib.c +67 -1
86 lines not on this page · raw message
drivers/iommu/iommu.c +29 -0
@@ -3173,3 +3173,32 @@ void iommu_detach_device_pasid(struct iommu_domain *domain,iommu_group_put (group); } + +/* +* Get the attached domain for asynchronous usage, for example the I/O +* page fault handling framework. The caller get a reference counter +* of the domain automatically on a successful return and should put +* it with iommu_domain_put() after usage. +*/ +struct iommu_domain * +iommu_get_domain_for_dev_pasid_async (struct device *dev,ioasid_t pasid) +{ +struct iommu_domain *domain; +struct iommu_group *group; + +if (!pasid_valid (pasid)) +return NULL ; + + group =iommu_group_get (dev); +if (!group) +return NULL ; + +mutex_lock (&group->mutex); + domain =xa_load (&group->pasid_array, pasid); +if (domain) +refcount_inc (&domain->async_users); +mutex_unlock (&group->mutex); +iommu_group_put (group); + +return domain; +}
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v4 10/12] iommu: Prepare IOMMU domain for IOPF
Hi Baolu,
Quoted 10 lines · Lu Baolu
drivers/iommu/iommu.c +10
-0+/* +* Get the attached domain for asynchronous usage, for example the I/O +* page fault handling framework. The caller get a reference counter +* of the domain automatically on a successful return and should put +* it with iommu_domain_put() after usage. +*/ +struct iommu_domain * +iommu_get_domain_for_dev_pasid_async (struct device *dev,ioasid_t pasid) +{ +struct iommu_domain *domain;
drivers/iommu/iommu.c void iommu_detach_device_pasid(struct iommu_domain *domain, · line 3196
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+struct iommu_group *group; + +if (!pasid_valid (pasid)) +return NULL ; + + group =iommu_group_get (dev); +if (!group) +return NULL ; + +mutex_lock (&group->mutex);
There is a possible deadlock between unbind() and the fault handler:
unbind() iopf_handle_group()
mutex_lock(&group->mutex)
iommu_detach_device_pasid()
iopf_queue_flush_dev() iommu_get_domain_for_dev_pasid_async()
... waits for IOPF work mutex_lock(&group->mutex)
I was wrong in my previous review: we do have a guarantee that the SVA domain does not go away during IOPF handling, because unbind() waits for pending faults with iopf_queue_flush_dev() before freeing the domain (or for Arm stall, knows that there are no pending faults). So we can just get rid of domain->async_users and the group->mutex in IOPF, I think?
Thanks,
Jean
Quoted 11 lines
drivers/iommu/iommu.c +8
-0+ domain =xa_load (&group->pasid_array, pasid); +if (domain) +refcount_inc (&domain->async_users); +mutex_unlock (&group->mutex); +iommu_group_put (group); + +return domain; +}Signature · 1 line
2.25.1
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Baolu Lu
maintainerRe: [PATCH v4 10/12] iommu: Prepare IOMMU domain for IOPF
Hi Jean,
Quoted 26 lines · Jean-Philippe Brucker
Hi Baolu,
Quoted 10 lines · Lu Baolu
drivers/iommu/iommu.c +10
-0+/* +* Get the attached domain for asynchronous usage, for example the I/O +* page fault handling framework. The caller get a reference counter +* of the domain automatically on a successful return and should put +* it with iommu_domain_put() after usage. +*/ +struct iommu_domain * +iommu_get_domain_for_dev_pasid_async (struct device *dev,ioasid_t pasid) +{ +struct iommu_domain *domain;drivers/iommu/iommu.c void iommu_detach_device_pasid(struct iommu_domain *domain, · line 3196
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+struct iommu_group *group; + +if (!pasid_valid (pasid)) +return NULL ; + + group =iommu_group_get (dev); +if (!group) +return NULL ; + +mutex_lock (&group->mutex);There is a possible deadlock between
unbind()and the fault handler:
Quoted 6 lines
unbind()iopf_handle_group()
mutex_lock(&group->mutex)
iommu_detach_device_pasid()iopf_queue_flush_dev() iommu_get_domain_for_dev_pasid_async() ... waits for IOPF work mutex_lock(&group->mutex)
Yes, really.
Quoted 5 lines
I was wrong in my previous review: we do have a guarantee that the SVA domain does not go away during IOPF handling, because
unbind()waits for pending faults withiopf_queue_flush_dev()before freeing the domain (or for Arm stall, knows that there are no pending faults). So we can just get rid ofdomain->async_usersand thegroup->mutexin IOPF, I think?
Agreed with you. The Intel code does the same thing in its unbind().
Thus, the sva domain's life cycle has already synchronized with IOPF handling, there's no need for domain->async.
I will drop it in the next version. Thanks you!
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Lu Baolu
maintainerPATCH v4 11/12 iommu: Per-domain I/O page fault handling
In reply to Lu BaoluTweak the I/O page fault handling framework to route the page faults to the domain and call the page fault handler retrieved from the domain. This makes the I/O page fault handling framework possible to serve more usage scenarios as long as they have an IOMMU domain and install a page fault handler in it. Some unused functions are also removed to avoid dead code.
--- drivers/iommu/iommu-sva-lib.h | 1 - drivers/iommu/io-pgfault.c | 69 ++++++----------------------------- drivers/iommu/iommu-sva-lib.c | 20 ---------- 3 files changed, 12 insertions(+), 78 deletions(-)
drivers/iommu/iommu-sva-lib.h +0 -1
@@ -8,7 +8,6 @@#include <linux/ioasid.h> #include <linux/mm_types.h> -struct mm_struct *iommu_sva_find (ioasid_t pasid);struct mm_struct *iommu_sva_domain_mm (struct iommu_domain *domain);/* I/O Page fault */
drivers/iommu/io-pgfault.c +12 -57
86 lines not on this page · raw message
drivers/iommu/iommu-sva-lib.c +0 -20
@@ -69,26 +69,6 @@ static int iommu_sva_alloc_pasid(struct mm_struct *mm,return ret; } -/* ioasid_find getter() requires a void * argument */ -static bool __mmget_not_zero (void *mm) -{ -return mmget_not_zero (mm); -} - -/** -* iommu_sva_find() - Find mm associated to the given PASID -* @pasid: Process Address Space ID assigned to the mm -* -* On success a reference to the mm is taken, and must be released with mmput(). -* -* Returns the mm corresponding to this PASID, or an error if not found. -*/ -struct mm_struct *iommu_sva_find (ioasid_t pasid) -{ -return ioasid_find (&iommu_sva_pasid, pasid, __mmget_not_zero); -} -EXPORT_SYMBOL_GPL (iommu_sva_find); -/* * Get or put an ioas for a shared memory. */
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v4 11/12] iommu: Per-domain I/O page fault handling
drivers/iommu/io-pgfault.c static void iopf_handle_group(struct work_struct *work) · line 84
Quoted 11 lines · Lu Baolu
static void
iopf_handle_group(struct work_struct *work){drivers/iommu/io-pgfault.c +4
-0struct iopf_group *group;@@ -134,12 +78,23 @@ static void iopf_handle_group(struct work_struct *work) group = container_of(work, struct iopf_group, work);list_for_each_entry_safe (iopf, next, &group->faults, list) { +struct iommu_domain *domain; + + domain =iommu_get_domain_for_dev_pasid_async (group->dev, + iopf->fault.prm.pasid);
Reading the PCIe spec again (v6.0 10.4.1.1 PASID Usage), all faults within the group have the same PASID so we could move the domain fetch out of the loop. It does deviate from the old behavior, though, so we could change it later.
Thanks,
Jean
Quoted 17 lines
drivers/iommu/io-pgfault.c +8
-1+if (!domain || !domain->iopf_handler) + status =IOMMU_PAGE_RESP_INVALID ; +/* * For the moment, errors are sticky: don't handle subsequent * faults in the group if there is an error. */ if (status ==IOMMU_PAGE_RESP_SUCCESS ) - status =iopf_handle_single (iopf); + status = domain->iopf_handler (&iopf->fault, + domain->fault_data); + +if (domain) +iommu_domain_put_async (domain);if (!(iopf->fault.prm.flags &IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE ))
Baolu Lu
maintainerRe: [PATCH v4 11/12] iommu: Per-domain I/O page fault handling
Quoted 7 lines · Jean-Philippe Brucker
Quoted 6 lines · Lu Baolu
static void iopf_handle_group(struct work_struct *work) { struct iopf_group *group;drivers/iommu/io-pgfault.c +0
-0@@ -134,12 +78,23 @@ static void iopf_handle_group(struct work_struct *work) group = container_of(work, struct iopf_group, work);
Quoted 9 lines
drivers/iommu/io-pgfault.c static void iopf_handle_group(struct work_struct *work) · line 84
Quoted 5 lines
drivers/iommu/io-pgfault.c +4
-0list_for_each_entry_safe (iopf, next, &group->faults, list) { +struct iommu_domain *domain; + + domain =iommu_get_domain_for_dev_pasid_async (group->dev, + iopf->fault.prm.pasid);Reading the PCIe spec again (v6.0 10.4.1.1 PASID Usage), all faults within the group have the same PASID so we could move the domain fetch out of the loop. It does deviate from the old behavior, though, so we could change it later.
Perhaps we can add a pasid member in the struct iopf_group and do a sanity check when a new iopf is added to the group? Here, we just fetch the domain with group->pasid.
Best regards,
baolu
Lu Baolu
maintainerPATCH v4 12/12 iommu: Rename iommu-sva-lib.{c,h}
In reply to Lu BaoluRename iommu-sva-lib.c[h] to iommu-sva.c[h] as it contains all code for SVA implementation in iommu core.
---
drivers/iommu/{iommu-sva-lib.h => iommu-sva.h} | 0
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 2 +-
drivers/iommu/intel/iommu.c | 2 +-
drivers/iommu/intel/svm.c | 2 +-
drivers/iommu/io-pgfault.c | 2 +-
drivers/iommu/{iommu-sva-lib.c => iommu-sva.c} | 2 +-
drivers/iommu/iommu.c | 2 +-
drivers/iommu/Makefile | 2 +-
9 files changed, 8 insertions(+), 8 deletions(-)
rename drivers/iommu/{iommu-sva-lib.h => iommu-sva.h} (100%)
rename drivers/iommu/{iommu-sva-lib.c => iommu-sva.c} (99%)drivers/iommu/iommu-sva.h renamed from drivers/iommu/iommu-sva-lib.h +0 -0
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +1 -1
@@ -9,7 +9,7 @@#include <linux/slab.h> #include "arm-smmu-v3.h" -#include "../../iommu-sva-lib.h" +#include "../../iommu-sva.h" #include "../../io-pgtable-arm.h" struct arm_smmu_mmu_notifier {
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -1
@@ -31,7 +31,7 @@#include <linux/amba/bus.h> #include "arm-smmu-v3.h" -#include "../../iommu-sva-lib.h" +#include "../../iommu-sva.h" static bool disable_bypass = true;module_param (disable_bypass, bool,0444 );
drivers/iommu/intel/iommu.c +1 -1
@@ -27,7 +27,7 @@#include <linux/tboot.h> #include "../irq_remapping.h" -#include "../iommu-sva-lib.h" +#include "../iommu-sva.h" #include "pasid.h" #include "cap_audit.h"
drivers/iommu/intel/svm.c +1 -1
@@ -25,7 +25,7 @@#include "pasid.h" #include "perf.h" -#include "../iommu-sva-lib.h" +#include "../iommu-sva.h" static irqreturn_t prq_event_thread (int irq,void *d);static void intel_svm_drain_prq (struct device *dev,u32 pasid);
drivers/iommu/io-pgfault.c +1 -1
@@ -11,7 +11,7 @@#include <linux/slab.h> #include <linux/workqueue.h> -#include "iommu-sva-lib.h" +#include "iommu-sva.h" /** * struct iopf_queue - IO Page Fault queue
drivers/iommu/iommu-sva.c renamed from drivers/iommu/iommu-sva-lib.c +1 -1
@@ -7,7 +7,7 @@#include <linux/slab.h> #include <linux/sched/mm.h> -#include "iommu-sva-lib.h" +#include "iommu-sva.h" static DEFINE_MUTEX (iommu_sva_lock);static DECLARE_IOASID_SET (iommu_sva_pasid);
drivers/iommu/iommu.c +1 -1
@@ -27,7 +27,7 @@#include <linux/cc_platform.h> #include <trace/events/iommu.h> -#include "iommu-sva-lib.h" +#include "iommu-sva.h" static struct kset *iommu_group_kset;static DEFINE_IDA (iommu_group_ida);
drivers/iommu/Makefile +1 -1
@@ -27,6 +27,6 @@ obj-$(CONFIG_FSL_PAMU) += fsl_pamu.o fsl_pamu_domain.o obj-$(CONFIG_S390_IOMMU) += s390-iommu.o obj-$(CONFIG_HYPERV_IOMMU) += hyperv-iommu.o obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o -obj-$(CONFIG_IOMMU_SVA) += iommu-sva-lib.o io-pgfault.o +obj-$(CONFIG_IOMMU_SVA) += iommu-sva.o io-pgfault.o obj-$(CONFIG_SPRD_IOMMU) += sprd-iommu.o obj-$(CONFIG_APPLE_DART) += apple-dart.o
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu