iommufd: Add nesting infrastructure (part 1/2)
Subsystems (5)
v5 cover iommufd: Add nesting infrastructure (part 1/2)
Nested translation is a hardware feature that is supported by many modern IOMMU hardwares. It has two stages of address translations to get access to the physical address. A stage-1 translation table is owned by userspace (e.g. by a guest OS), while a stage-2 is owned by kernel. Any change to a stage-1 translation table should be followed by an IOTLB invalidation.
Take Intel VT-d as an example, the stage-1 translation table is guest I/O page table. As the below diagram shows, the guest I/O page table pointer in GPA (guest physical address) is passed to host and be used to perform a stage-1 translation. Along with it, a modification to present mappings in the guest I/O page table should be followed by an IOTLB invalidation.
.-------------. .---------------------------.
| vIOMMU | | Guest I/O page table |
| | '---------------------------' .----------------/
| PASID Entry |--- PASID cache flush --+
'-------------' |
| | V
| | I/O page table pointer in GPA
'-------------'
Guest
------| Shadow |---------------------------|--------
v v v
Host
.-------------. .------------------------.
| pIOMMU | | FS for GIOVA->GPA |
| | '------------------------'
.----------------/ |
| PASID Entry | V (Nested xlate)
'----------------\.----------------------------------.
| | | SS for GPA->HPA, unmanaged domain|
| | '----------------------------------'
'-------------'Where:
- FS = First stage page tables
- SS = Second stage page tables
<Intel VT-d Nested translation>
In IOMMUFD, all the translation tables are tracked by hw_pagetable (hwpt) and each hwpt is backed by an iommu_domain allocated from an iommu driver. So in this series hw_pagetable and iommu_domain means the same thing if no special note. IOMMUFD has already supported allocating hw_pagetable linked with an IOAS. However, a nesting case requires IOMMUFD to allow allocating hw_pagetable with driver specific parameters and interface to sync stage-1 IOTLB as user owns the stage-1 translation table.
This series is based on the iommu hw info reporting series [1] and nested parent domain allocation [2]. It first extends domain_alloc_user to allocate hwpt with user data by allowing the IOMMUFD internal infrastructure to accept user_data and parent hwpt, relaying the user_data/parent to the iommu core to allocate IOMMU_DOMAIN_NESTED. And it then extends the IOMMU_HWPT_ALLOC ioctl to accept user data and a parent hwpt ID.
Note that this series is the part-1 set of a two-part nesting series. It does not include the cache invalidation interface, which will be added in the part 2.
Complete code can be found in [3], QEMU could can be found in [4].
At last, this is a team work together with Nicolin Chen, Lu Baolu. Thanks them for the help. ^_^. Look forward to your feedbacks.
[1] https://lore.kernel.org/linux-iommu/[email protected]/#r - merged
[2] https://lore.kernel.org/linux-iommu/[email protected]/ - merged
[3] https://github.com/yiliu1765/iommufd/tree/iommufd_nesting
[4] https://github.com/yiliu1765/qemu/tree/zhenzhong/wip/iommufd_nesting_rfcv1
Change log:
v5:
- Split the iommufd nesting series into two parts of alloc_user and invalidation (Jason)
- Split IOMMUFD_OBJ_HW_PAGETABLE to IOMMUFD_OBJ_HWPT_PAGING/_NESTED, and do the same with the structures/alloc()/abort()/destroy(). Reworked the selftest accordingly too. (Jason)
- Move hwpt/data_type into struct iommu_user_data from standalone op arguments. (Jason)
- Rename hwpt_type to be data_type, the HWPT_TYPE to be HWPT_ALLOC_DATA, _TYPE_DEFAULT to be _ALLOC_DATA_NONE (Jason, Kevin)
- Rename iommu_copy_user_data() to iommu_copy_struct_from_user() (Kevin)
- Add macro to the iommu_copy_struct_from_user() to calculate min_size (Jason)
- Fix two bugs spotted by ZhaoYan
v4: https://lore.kernel.org/linux-iommu/[email protected]/
- Separate HWPT alloc/destroy/abort functions between user-managed HWPTs and kernel-managed HWPTs
- Rework invalidate uAPI to be a multi-request array-based design
- Add a struct iommu_user_data_array and a helper for driver to sanitize and copy the entry data from user space invalidation array
- Add a patch fixing TEST_LENGTH() in selftest program
- Drop IOMMU_RESV_IOVA_RANGES patches
- Update kdoc and inline comments
- Drop the code to add IOMMU_RESV_SW_MSI to kernel-managed HWPT in nested translation, this does not change the rule that resv regions should only be added to the kernel-managed HWPT. The IOMMU_RESV_SW_MSI stuff will be added in later series as it is needed only by SMMU so far.
v3: https://lore.kernel.org/linux-iommu/[email protected]/
- Add new uAPI things in alphabetical order
- Pass in "enum iommu_hwpt_type hwpt_type" to op->domain_alloc_user for sanity, replacing the previous op->domain_alloc_user_data_len solution
- Return ERR_PTR from domain_alloc_user instead of NULL
- Only add IOMMU_RESV_SW_MSI to kernel-managed HWPT in nested translation (Kevin)
- Add IOMMU_RESV_IOVA_RANGES to report resv iova ranges to userspace hence userspace is able to exclude the ranges in the stage-1 HWPT (e.g. guest I/O page table). (Kevin)
- Add selftest coverage for the new IOMMU_RESV_IOVA_RANGES ioctl
- Minor changes per Kevin's inputs
v2: https://lore.kernel.org/linux-iommu/[email protected]/
- Add union iommu_domain_user_data to include all user data structures to avoid passing void * in kernel APIs.
- Add iommu op to return user data length for user domain allocation
- Rename struct iommu_hwpt_alloc::data_type to be hwpt_type
- Store the invalidation data length in iommu_domain_ops::cache_invalidate_user_data_len
- Convert cache_invalidate_user op to be int instead of void
- Remove @data_type in struct iommu_hwpt_invalidate
- Remove out_hwpt_type_bitmap in struct iommu_hw_info hence drop patch 08 of v1
v1: https://lore.kernel.org/linux-iommu/[email protected]/
Thanks,
Yi Liu
Jason Gunthorpe (2):
iommufd: Rename IOMMUFD_OBJ_HW_PAGETABLE to IOMMUFD_OBJ_HWPT_PAGING iommufd/device: Wrap IOMMUFD_OBJ_HWPT_PAGING-only configurations
Lu Baolu (1):
iommu: Add IOMMU_DOMAIN_NESTED
Nicolin Chen (6):
iommufd: Derive iommufd_hwpt_paging from iommufd_hw_pagetable
iommufd: Share iommufd_hwpt_alloc with IOMMUFD_OBJ_HWPT_NESTED
iommufd: Add a nested HW pagetable object
iommu: Add iommu_copy_struct_from_user helper iommufd/selftest: Add nested domain allocation for mock domain iommufd/selftest: Add coverage for IOMMU_HWPT_ALLOC with nested HWPTs
Yi Liu (1):
iommu: Pass in parent domain with user_data to domain_alloc_user op
drivers/iommu/intel/iommu.c | 4 +- drivers/iommu/iommufd/device.c | 184 +++++++++----- drivers/iommu/iommufd/hw_pagetable.c | 237 ++++++++++++++---- drivers/iommu/iommufd/iommufd_private.h | 67 +++-- drivers/iommu/iommufd/iommufd_test.h | 18 ++ drivers/iommu/iommufd/main.c | 10 +- drivers/iommu/iommufd/selftest.c | 137 ++++++++-- drivers/iommu/iommufd/vfio_compat.c | 6 +- include/linux/iommu.h | 72 +++++- include/uapi/linux/iommufd.h | 31 ++- tools/testing/selftests/iommu/iommufd.c | 120 +++++++++ .../selftests/iommu/iommufd_fail_nth.c | 3 +- tools/testing/selftests/iommu/iommufd_utils.h | 31 ++- 13 files changed, 752 insertions(+), 168 deletions(-)
Signature · 1 line
2.34.1
PATCH v5 1/10 iommu: Add IOMMU_DOMAIN_NESTED
From: Lu Baolu <[email protected]>
Introduce a new domain type for a user I/O page table, which is nested on top of another user space address represented by a PAGING domain. This new domain can be allocated by the domain_alloc_user op, and attached to a device through the existing iommu_attach_device/group() interfaces.
The mappings of a nested domain are managed by user space software, so it is not necessary to have map/unmap callbacks.
--- include/linux/iommu.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/iommu.h +4 -0
@@ -65,6 +65,9 @@ struct iommu_domain_geometry {#define __IOMMU_DOMAIN_SVA (1U << 4)/* Shared process address space */ +#define __IOMMU_DOMAIN_NESTED (1U << 5)/* User-managed address space nested +on a stage-2 translation */ +#define IOMMU_DOMAIN_ALLOC_FLAGS ~__IOMMU_DOMAIN_DMA_FQ/* * This are the possible domain-types
@@ -91,6 +94,7 @@ struct iommu_domain_geometry { __IOMMU_DOMAIN_DMA_API | \ __IOMMU_DOMAIN_DMA_FQ)#define IOMMU_DOMAIN_SVA (__IOMMU_DOMAIN_SVA) +#define IOMMU_DOMAIN_NESTED (__IOMMU_DOMAIN_NESTED)struct iommu_domain {unsigned type;
Signature · 1 line
2.34.1
PATCH v5 2/10 iommu: Pass in parent domain with user_data to domain_alloc_user op
In reply to Yi Liudomain_alloc_user op already accepts user flags for domain allocation, add a parent domain pointer and a driver specific user data support as well.
Add a struct iommu_user_data as a bundle of data_ptr/data_len/type from an iommufd core uAPI structure. Make the user data opaque to the core, since a userspace driver must match the kernel driver. In the future, if drivers share some common parameter, there would be a generic parameter as well.
Define an enum iommu_hwpt_data_type (with IOMMU_HWPT_DATA_NONE type) for iommu drivers to add their own driver specific user data per hw_pagetable.
--- drivers/iommu/intel/iommu.c | 4 +++- drivers/iommu/iommufd/hw_pagetable.c | 3 ++- drivers/iommu/iommufd/selftest.c | 4 +++- include/linux/iommu.h | 27 ++++++++++++++++++++++++--- include/uapi/linux/iommufd.h | 8 ++++++++ 5 files changed, 40 insertions(+), 6 deletions(-)
drivers/iommu/intel/iommu.c +3 -1
@@ -4075,7 +4075,9 @@ static struct iommu_domain *intel_iommu_domain_alloc(unsigned type) }static struct iommu_domain * -intel_iommu_domain_alloc_user (struct device *dev,u32 flags) +intel_iommu_domain_alloc_user (struct device *dev,u32 flags, +struct iommu_domain *parent, +const struct iommu_user_data *user_data) {struct iommu_domain *domain;struct intel_iommu *iommu;
drivers/iommu/iommufd/hw_pagetable.c +2 -1
@@ -96,7 +96,8 @@ iommufd_hw_pagetable_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, hwpt->ioas = ioas;if (ops->domain_alloc_user) { - hwpt->domain = ops->domain_alloc_user (idev->dev, flags); + hwpt->domain = ops->domain_alloc_user (idev->dev, flags, +NULL ,NULL );if (IS_ERR (hwpt->domain)) { rc =PTR_ERR (hwpt->domain); hwpt->domain =NULL ;
drivers/iommu/iommufd/selftest.c +3 -1
@@ -171,7 +171,9 @@ static struct iommu_domain *mock_domain_alloc(unsigned int iommu_domain_type) }static struct iommu_domain * -mock_domain_alloc_user (struct device *dev,u32 flags) +mock_domain_alloc_user (struct device *dev,u32 flags, +struct iommu_domain *parent, +const struct iommu_user_data *user_data) {struct iommu_domain *domain;
include/linux/iommu.h +24 -3
@@ -231,6 +231,21 @@ struct iommu_iotlb_gather {bool queued; }; +/** +* struct iommu_user_data - iommu driver specific user space data info +* @type: The data type of the user buffer +* @uptr: Pointer to the user buffer for copy_from_user() +* @len: The length of the user buffer in bytes +* +* A user space data is an uAPI that is defined in include/uapi/linux/iommufd.h +* @type, @uptr and @len should be just copied from an iommufd core uAPI struct. +*/ +struct iommu_user_data { +unsigned int type; +void __user *uptr; +size_t len; +}; +/** * struct iommu_ops - iommu ops and capabilities * @capable: check capability
@@ -245,8 +260,12 @@ struct iommu_iotlb_gather {* parameters as defined in include/uapi/linux/iommufd.h. * Unlike @domain_alloc, it is called only by IOMMUFD and * must fully initialize the new domain before return. -* Upon success, a domain is returned. Upon failure, -* ERR_PTR must be returned. +* Upon success, if the @user_data is valid and the @parent +* points to a kernel-managed domain, the new domain must be +* IOMMU_DOMAIN_NESTED type; otherwise, the @parent must be +* NULL while the @user_data can be optionally provided, the +* new domain must support __IOMMU_DOMAIN_PAGING. +* Upon failure, ERR_PTR must be returned. * @probe_device: Add device to iommu driver handling * @release_device: Remove device from iommu driver handling * @probe_finalize: Do final setup work after the device is added to an IOMMU
@@ -279,7 +298,9 @@ struct iommu_ops {/* Domain allocation and freeing by the iommu driver */ struct iommu_domain *(*domain_alloc)(unsigned iommu_domain_type); -struct iommu_domain *(*domain_alloc_user)(struct device *dev,u32 flags); +struct iommu_domain *(*domain_alloc_user)(struct device *dev,u32 flags, +struct iommu_domain *parent, +const struct iommu_user_data *user_data);struct iommu_device *(*probe_device)(struct device *dev);void (*release_device)(struct device *dev);
include/uapi/linux/iommufd.h +8 -0
@@ -356,6 +356,14 @@ enum iommufd_hwpt_alloc_flags { IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0, }; +/** +* enum iommu_hwpt_data_type - IOMMU HWPT Data Type +* @IOMMU_HWPT_DATA_NONE: no data +*/ +enum iommu_hwpt_data_type { +IOMMU_HWPT_DATA_NONE , +}; +/** * struct iommu_hwpt_alloc - ioctl(IOMMU_HWPT_ALLOC) * @size: sizeof(struct iommu_hwpt_alloc)
Signature · 1 line
2.34.1
PATCH v5 3/10 iommufd: Rename IOMMUFD_OBJ_HW_PAGETABLE to IOMMUFD_OBJ_HWPT_PAGING
In reply to Yi LiuFrom: Jason Gunthorpe <[email protected]>
To add a new IOMMUFD_OBJ_HWPT_NESTED, rename the HWPT object to confine it to PAGING hwpts/domains. The following patch will separate the hwpt structure as well.
--- drivers/iommu/iommufd/device.c | 10 +++++----- drivers/iommu/iommufd/hw_pagetable.c | 2 +- drivers/iommu/iommufd/iommufd_private.h | 2 +- drivers/iommu/iommufd/main.c | 2 +- drivers/iommu/iommufd/selftest.c | 2 +- 5 files changed, 9 insertions(+), 9 deletions(-)
drivers/iommu/iommufd/device.c +5 -5
@@ -579,7 +579,7 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id,return PTR_ERR (pt_obj);switch (pt_obj->type) { -case IOMMUFD_OBJ_HW_PAGETABLE : { +case IOMMUFD_OBJ_HWPT_PAGING : {struct iommufd_hw_pagetable *hwpt = container_of(pt_obj, struct iommufd_hw_pagetable, obj);
@@ -617,8 +617,8 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id,/** * iommufd_device_attach - Connect a device to an iommu_domain * @idev: device to attach -* @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HW_PAGETABLE -* Output the IOMMUFD_OBJ_HW_PAGETABLE ID +* @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING +* Output the IOMMUFD_OBJ_HWPT_PAGING ID * * This connects the device to an iommu_domain, either automatically or manually * selected. Once this completes the device could do DMA.
@@ -646,8 +646,8 @@ EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, IOMMUFD);/** * iommufd_device_replace - Change the device's iommu_domain * @idev: device to change -* @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HW_PAGETABLE -* Output the IOMMUFD_OBJ_HW_PAGETABLE ID +* @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING +* Output the IOMMUFD_OBJ_HWPT_PAGING ID * * This is the same as:: *
drivers/iommu/iommufd/hw_pagetable.c +1 -1
@@ -86,7 +86,7 @@ iommufd_hw_pagetable_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,if (flags && !ops->domain_alloc_user)return ERR_PTR (-EOPNOTSUPP ); - hwpt =iommufd_object_alloc (ictx, hwpt,IOMMUFD_OBJ_HW_PAGETABLE ); + hwpt =iommufd_object_alloc (ictx, hwpt,IOMMUFD_OBJ_HWPT_PAGING );if (IS_ERR (hwpt))return hwpt;
drivers/iommu/iommufd/iommufd_private.h +1 -1
@@ -113,7 +113,7 @@ enum iommufd_object_type {IOMMUFD_OBJ_NONE ,IOMMUFD_OBJ_ANY =IOMMUFD_OBJ_NONE ,IOMMUFD_OBJ_DEVICE , -IOMMUFD_OBJ_HW_PAGETABLE , +IOMMUFD_OBJ_HWPT_PAGING ,IOMMUFD_OBJ_IOAS ,IOMMUFD_OBJ_ACCESS , #ifdef CONFIG_IOMMUFD_TEST
drivers/iommu/iommufd/main.c +1 -1
@@ -482,7 +482,7 @@ static const struct iommufd_object_ops iommufd_object_ops[] = { [IOMMUFD_OBJ_IOAS] = { .destroy = iommufd_ioas_destroy, }, - [IOMMUFD_OBJ_HW_PAGETABLE] = { + [IOMMUFD_OBJ_HWPT_PAGING] = { .destroy = iommufd_hw_pagetable_destroy, .abort = iommufd_hw_pagetable_abort, },
drivers/iommu/iommufd/selftest.c +1 -1
@@ -350,7 +350,7 @@ get_md_pagetable(struct iommufd_ucmd *ucmd, u32 mockpt_id,struct iommufd_object *obj; obj =iommufd_get_object (ucmd->ictx, mockpt_id, -IOMMUFD_OBJ_HW_PAGETABLE ); +IOMMUFD_OBJ_HWPT_PAGING );if (IS_ERR (obj))return ERR_CAST (obj); hwpt = container_of(obj, struct iommufd_hw_pagetable, obj);
Signature · 1 line
2.34.1
PATCH v5 4/10 iommufd/device: Wrap IOMMUFD_OBJ_HWPT_PAGING-only configurations
In reply to Yi LiuFrom: Jason Gunthorpe <[email protected]>
Some of the configurations during the attach/replace() should only apply to IOMMUFD_OBJ_HWPT_PAGING. Once IOMMUFD_OBJ_HWPT_NESTED gets introduced in a following patch, keeping them unconditionally in the common routine will not work.
Wrap all of those PAGING-only configurations together into helpers. Do a hwpt_is_paging check whenever calling them or their fallback routines.
Also, move "num_devices++" closer to the place using it.
--- drivers/iommu/iommufd/device.c | 137 ++++++++++++++++-------- drivers/iommu/iommufd/iommufd_private.h | 5 + 2 files changed, 99 insertions(+), 43 deletions(-)
drivers/iommu/iommufd/device.c +94 -43
@@ -325,6 +325,35 @@ static int iommufd_group_setup_msi(struct iommufd_group *igroup,return 0 ; } +static int iommufd_hwpt_paging_attach (struct iommufd_hw_pagetable *hwpt, +struct iommufd_device *idev) +{ +int rc; + +lockdep_assert_held (&idev->igroup->lock); + +/* Try to upgrade the domain we have */ +if (idev->enforce_cache_coherency) { + rc =iommufd_hw_pagetable_enforce_cc (hwpt); +if (rc) +return rc; + } + + rc =iopt_table_enforce_dev_resv_regions (&hwpt->ioas->iopt, idev->dev, + &idev->igroup->sw_msi_start); +if (rc) +return rc; + +if (list_empty (&idev->igroup->device_list)) { + rc =iommufd_group_setup_msi (idev->igroup, hwpt); +if (rc) { +iopt_remove_reserved_iova (&hwpt->ioas->iopt, idev->dev); +return rc; + } + } +return 0 ; +} +int iommufd_hw_pagetable_attach (struct iommufd_hw_pagetable *hwpt,struct iommufd_device *idev) {
@@ -337,18 +366,12 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, goto err_unlock; } -/* Try to upgrade the domain we have */ -if (idev->enforce_cache_coherency) { - rc =iommufd_hw_pagetable_enforce_cc (hwpt); +if (hwpt_is_paging (hwpt)) { + rc =iommufd_hwpt_paging_attach (hwpt, idev);if (rc) goto err_unlock; } - rc =iopt_table_enforce_dev_resv_regions (&hwpt->ioas->iopt, idev->dev, - &idev->igroup->sw_msi_start); -if (rc) - goto err_unlock; -/* * Only attach to the group once for the first device that is in the * group. All the other devices will follow this attachment. The user
@@ -357,10 +380,6 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,* attachment. */ if (list_empty (&idev->igroup->device_list)) { - rc =iommufd_group_setup_msi (idev->igroup, hwpt); -if (rc) - goto err_unresv; - rc =iommu_attach_group (hwpt->domain, idev->igroup->group);if (rc) goto err_unresv;
@@ -371,7 +390,8 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,mutex_unlock (&idev->igroup->lock);return 0 ; err_unresv: -iopt_remove_reserved_iova (&hwpt->ioas->iopt, idev->dev); +if (hwpt_is_paging (hwpt)) +iopt_remove_reserved_iova (&hwpt->ioas->iopt, idev->dev); err_unlock:mutex_unlock (&idev->igroup->lock);return rc;
@@ -388,7 +408,8 @@ iommufd_hw_pagetable_detach(struct iommufd_device *idev)iommu_detach_group (hwpt->domain, idev->igroup->group); idev->igroup->hwpt =NULL ; } -iopt_remove_reserved_iova (&hwpt->ioas->iopt, idev->dev); +if (hwpt_is_paging (hwpt)) +iopt_remove_reserved_iova (&hwpt->ioas->iopt, idev->dev);mutex_unlock (&idev->igroup->lock);/* Caller must destroy hwpt */
@@ -407,40 +428,36 @@ iommufd_device_do_attach(struct iommufd_device *idev,return NULL ; } -static struct iommufd_hw_pagetable * -iommufd_device_do_replace (struct iommufd_device *idev, -struct iommufd_hw_pagetable *hwpt) +static void iommufd_group_remove_reserved_iova (struct iommufd_group *igroup, +struct iommufd_hw_pagetable *hwpt) { -struct iommufd_group *igroup = idev->igroup; -struct iommufd_hw_pagetable *old_hwpt; -unsigned int num_devices =0 ;struct iommufd_device *cur; -int rc; -mutex_lock (&idev->igroup->lock); +lockdep_assert_held (&igroup->lock); -if (igroup->hwpt ==NULL ) { - rc = -EINVAL ; - goto err_unlock; - } +list_for_each_entry (cur, &igroup->device_list, group_item) +iopt_remove_reserved_iova (&hwpt->ioas->iopt, cur->dev); +} -if (hwpt == igroup->hwpt) { -mutex_unlock (&idev->igroup->lock); -return NULL ; - } +static int iommufd_group_do_replace_paging (struct iommufd_group *igroup, +struct iommufd_hw_pagetable *hwpt) +{ +struct iommufd_hw_pagetable *old_hwpt = igroup->hwpt; +struct iommufd_device *cur; +int rc; + +lockdep_assert_held (&igroup->lock);/* Try to upgrade the domain we have */ list_for_each_entry (cur, &igroup->device_list, group_item) { - num_devices++;if (cur->enforce_cache_coherency) { rc =iommufd_hw_pagetable_enforce_cc (hwpt);if (rc) - goto err_unlock; +return rc; } } - old_hwpt = igroup->hwpt; -if (hwpt->ioas != old_hwpt->ioas) { +if (hwpt_is_paging (old_hwpt) && hwpt->ioas != old_hwpt->ioas) {list_for_each_entry (cur, &igroup->device_list, group_item) { rc =iopt_table_enforce_dev_resv_regions ( &hwpt->ioas->iopt, cur->dev,NULL );
@@ -448,23 +465,57 @@ iommufd_device_do_replace(struct iommufd_device *idev, goto err_unresv; } } - - rc =iommufd_group_setup_msi (idev->igroup, hwpt); + rc =iommufd_group_setup_msi (igroup, hwpt);if (rc) goto err_unresv; +return 0 ; + +err_unresv: +iommufd_group_remove_reserved_iova (igroup, hwpt); +return rc; +} + +static struct iommufd_hw_pagetable * +iommufd_device_do_replace (struct iommufd_device *idev, +struct iommufd_hw_pagetable *hwpt) +{ +struct iommufd_group *igroup = idev->igroup; +struct iommufd_hw_pagetable *old_hwpt; +unsigned int num_devices =0 ; +struct iommufd_device *cur; +int rc; + +mutex_lock (&idev->igroup->lock); + +if (igroup->hwpt ==NULL ) { + rc = -EINVAL ; + goto err_unlock; + } + +if (hwpt == igroup->hwpt) { +mutex_unlock (&idev->igroup->lock); +return NULL ; + } + +if (hwpt_is_paging (hwpt)) { + rc =iommufd_group_do_replace_paging (igroup, hwpt); +if (rc) + goto err_unlock; + } rc =iommu_group_replace_domain (igroup->group, hwpt->domain);if (rc) goto err_unresv; -if (hwpt->ioas != old_hwpt->ioas) { -list_for_each_entry (cur, &igroup->device_list, group_item) -iopt_remove_reserved_iova (&old_hwpt->ioas->iopt, - cur->dev); - } + old_hwpt = igroup->hwpt; +if (hwpt_is_paging (old_hwpt) && + (!hwpt_is_paging (hwpt) || hwpt->ioas != old_hwpt->ioas)) +iommufd_group_remove_reserved_iova (igroup, old_hwpt); igroup->hwpt = hwpt; +list_for_each_entry (cur, &igroup->device_list, group_item) + num_devices++;/* * Move the refcounts held by the device_list to the new hwpt. Retain a * refcount for this thread as the caller will free it.
@@ -478,8 +529,8 @@ iommufd_device_do_replace(struct iommufd_device *idev,/* Caller must destroy old_hwpt */ return old_hwpt; err_unresv: -list_for_each_entry (cur, &igroup->device_list, group_item) -iopt_remove_reserved_iova (&hwpt->ioas->iopt, cur->dev); +if (hwpt_is_paging (hwpt)) +iommufd_group_remove_reserved_iova (igroup, hwpt); err_unlock:mutex_unlock (&idev->igroup->lock);return ERR_PTR (rc);
drivers/iommu/iommufd/iommufd_private.h +5 -0
@@ -240,6 +240,11 @@ struct iommufd_hw_pagetable {struct list_head hwpt_item; }; +static inline bool hwpt_is_paging (struct iommufd_hw_pagetable *hwpt) +{ +return hwpt->obj.type ==IOMMUFD_OBJ_HWPT_PAGING ; +} +struct iommufd_hw_pagetable * iommufd_hw_pagetable_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, struct iommufd_device *idev, u32 flags,
Signature · 1 line
2.34.1
PATCH v5 5/10 iommufd: Derive iommufd_hwpt_paging from iommufd_hw_pagetable
In reply to Yi LiuFrom: Nicolin Chen <[email protected]>
To prepare for IOMMUFD_OBJ_HWPT_NESTED, derive struct iommufd_hwpt_paging from struct iommufd_hw_pagetable, by leaving the common members in struct iommufd_hw_pagetable. Add a __iommufd_object_alloc and to_hwpt_paging() helpers for the new structure.
Then, update "hwpt" to "hwpt_paging" throughout the files, accordingly.
--- drivers/iommu/iommufd/device.c | 78 ++++++++++++-------- drivers/iommu/iommufd/hw_pagetable.c | 98 ++++++++++++++----------- drivers/iommu/iommufd/iommufd_private.h | 37 +++++++--- drivers/iommu/iommufd/main.c | 4 +- drivers/iommu/iommufd/vfio_compat.c | 6 +- 5 files changed, 131 insertions(+), 92 deletions(-)
drivers/iommu/iommufd/device.c +46 -32
@@ -293,7 +293,7 @@ u32 iommufd_device_to_id(struct iommufd_device *idev)EXPORT_SYMBOL_NS_GPL (iommufd_device_to_id,IOMMUFD );static int iommufd_group_setup_msi (struct iommufd_group *igroup, -struct iommufd_hw_pagetable *hwpt) +struct iommufd_hwpt_paging *hwpt_paging) {phys_addr_t sw_msi_start = igroup->sw_msi_start;int rc;
@@ -311,8 +311,9 @@ static int iommufd_group_setup_msi(struct iommufd_group *igroup,* matches what the IRQ layer actually expects in a newly created * domain. */ -if (sw_msi_start !=PHYS_ADDR_MAX && !hwpt->msi_cookie) { - rc =iommu_get_msi_cookie (hwpt->domain, sw_msi_start); +if (sw_msi_start !=PHYS_ADDR_MAX && !hwpt_paging->msi_cookie) { + rc =iommu_get_msi_cookie (hwpt_paging->common.domain, + sw_msi_start);if (rc)return rc;
@@ -320,12 +321,12 @@ static int iommufd_group_setup_msi(struct iommufd_group *igroup,* iommu_get_msi_cookie() can only be called once per domain, * it returns -EBUSY on later calls. */ - hwpt->msi_cookie = true; + hwpt_paging->msi_cookie = true; }return 0 ; } -static int iommufd_hwpt_paging_attach (struct iommufd_hw_pagetable *hwpt, +static int iommufd_hwpt_paging_attach (struct iommufd_hwpt_paging *hwpt_paging,struct iommufd_device *idev) {int rc;
@@ -334,20 +335,22 @@ static int iommufd_hwpt_paging_attach(struct iommufd_hw_pagetable *hwpt,/* Try to upgrade the domain we have */ if (idev->enforce_cache_coherency) { - rc =iommufd_hw_pagetable_enforce_cc (hwpt); + rc =iommufd_hwpt_paging_enforce_cc (hwpt_paging);if (rc)return rc; } - rc =iopt_table_enforce_dev_resv_regions (&hwpt->ioas->iopt, idev->dev, + rc =iopt_table_enforce_dev_resv_regions (&hwpt_paging->ioas->iopt, + idev->dev, &idev->igroup->sw_msi_start);if (rc)return rc;if (list_empty (&idev->igroup->device_list)) { - rc =iommufd_group_setup_msi (idev->igroup, hwpt); + rc =iommufd_group_setup_msi (idev->igroup, hwpt_paging);if (rc) { -iopt_remove_reserved_iova (&hwpt->ioas->iopt, idev->dev); +iopt_remove_reserved_iova (&hwpt_paging->ioas->iopt, + idev->dev);return rc; } }
@@ -367,7 +370,7 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, }if (hwpt_is_paging (hwpt)) { - rc =iommufd_hwpt_paging_attach (hwpt, idev); + rc =iommufd_hwpt_paging_attach (to_hwpt_paging (hwpt), idev);if (rc) goto err_unlock; }
@@ -391,7 +394,8 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,return 0 ; err_unresv:if (hwpt_is_paging (hwpt)) -iopt_remove_reserved_iova (&hwpt->ioas->iopt, idev->dev); +iopt_remove_reserved_iova (&to_hwpt_paging (hwpt)->ioas->iopt, + idev->dev); err_unlock:mutex_unlock (&idev->igroup->lock);return rc;
@@ -409,7 +413,8 @@ iommufd_hw_pagetable_detach(struct iommufd_device *idev) idev->igroup->hwpt =NULL ; }if (hwpt_is_paging (hwpt)) -iopt_remove_reserved_iova (&hwpt->ioas->iopt, idev->dev); +iopt_remove_reserved_iova (&to_hwpt_paging (hwpt)->ioas->iopt, + idev->dev);mutex_unlock (&idev->igroup->lock);/* Caller must destroy hwpt */
@@ -428,19 +433,20 @@ iommufd_device_do_attach(struct iommufd_device *idev,return NULL ; } -static void iommufd_group_remove_reserved_iova (struct iommufd_group *igroup, -struct iommufd_hw_pagetable *hwpt) +static void +iommufd_group_remove_reserved_iova (struct iommufd_group *igroup, +struct iommufd_hwpt_paging *hwpt_paging) {struct iommufd_device *cur;lockdep_assert_held (&igroup->lock);list_for_each_entry (cur, &igroup->device_list, group_item) -iopt_remove_reserved_iova (&hwpt->ioas->iopt, cur->dev); +iopt_remove_reserved_iova (&hwpt_paging->ioas->iopt, cur->dev); }static int iommufd_group_do_replace_paging (struct iommufd_group *igroup, -struct iommufd_hw_pagetable *hwpt) +struct iommufd_hwpt_paging *hwpt_paging) {struct iommufd_hw_pagetable *old_hwpt = igroup->hwpt;struct iommufd_device *cur;
@@ -451,27 +457,28 @@ static int iommufd_group_do_replace_paging(struct iommufd_group *igroup,/* Try to upgrade the domain we have */ list_for_each_entry (cur, &igroup->device_list, group_item) {if (cur->enforce_cache_coherency) { - rc =iommufd_hw_pagetable_enforce_cc (hwpt); + rc =iommufd_hwpt_paging_enforce_cc (hwpt_paging);if (rc)return rc; } } -if (hwpt_is_paging (old_hwpt) && hwpt->ioas != old_hwpt->ioas) { +if (hwpt_is_paging (old_hwpt) && + hwpt_paging->ioas !=to_hwpt_paging (old_hwpt)->ioas) {list_for_each_entry (cur, &igroup->device_list, group_item) { rc =iopt_table_enforce_dev_resv_regions ( - &hwpt->ioas->iopt, cur->dev,NULL ); + &hwpt_paging->ioas->iopt, cur->dev,NULL );if (rc) goto err_unresv; } } - rc =iommufd_group_setup_msi (igroup, hwpt); + rc =iommufd_group_setup_msi (igroup, hwpt_paging);if (rc) goto err_unresv;return 0 ; err_unresv: -iommufd_group_remove_reserved_iova (igroup, hwpt); +iommufd_group_remove_reserved_iova (igroup, hwpt_paging);return rc; }
@@ -498,7 +505,8 @@ iommufd_device_do_replace(struct iommufd_device *idev, }if (hwpt_is_paging (hwpt)) { - rc =iommufd_group_do_replace_paging (igroup, hwpt); + rc =iommufd_group_do_replace_paging (igroup, +to_hwpt_paging (hwpt));if (rc) goto err_unlock; }
@@ -509,8 +517,10 @@ iommufd_device_do_replace(struct iommufd_device *idev, old_hwpt = igroup->hwpt;if (hwpt_is_paging (old_hwpt) && - (!hwpt_is_paging (hwpt) || hwpt->ioas != old_hwpt->ioas)) -iommufd_group_remove_reserved_iova (igroup, old_hwpt); + (!hwpt_is_paging (hwpt) || +to_hwpt_paging (hwpt)->ioas !=to_hwpt_paging (old_hwpt)->ioas)) +iommufd_group_remove_reserved_iova (igroup, +to_hwpt_paging (old_hwpt)); igroup->hwpt = hwpt;
@@ -530,7 +540,8 @@ iommufd_device_do_replace(struct iommufd_device *idev,return old_hwpt; err_unresv:if (hwpt_is_paging (hwpt)) -iommufd_group_remove_reserved_iova (igroup, hwpt); +iommufd_group_remove_reserved_iova (igroup, +to_hwpt_paging (old_hwpt)); err_unlock:mutex_unlock (&idev->igroup->lock);return ERR_PTR (rc);
@@ -558,6 +569,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev,*/ bool immediate_attach = do_attach == iommufd_device_do_attach;struct iommufd_hw_pagetable *destroy_hwpt; +struct iommufd_hwpt_paging *hwpt_paging;struct iommufd_hw_pagetable *hwpt;/*
@@ -566,10 +578,11 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev,* other. */ mutex_lock (&ioas->mutex); -list_for_each_entry (hwpt, &ioas->hwpt_list, hwpt_item) { -if (!hwpt->auto_domain) +list_for_each_entry (hwpt_paging, &ioas->hwpt_list, hwpt_item) { +if (!hwpt_paging->auto_domain)continue ; + hwpt = &hwpt_paging->common;if (!iommufd_lock_obj (&hwpt->obj))continue ; destroy_hwpt = (*do_attach)(idev, hwpt);
@@ -590,12 +603,13 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, goto out_unlock; } - hwpt =iommufd_hw_pagetable_alloc (idev->ictx, ioas, idev, -0 , immediate_attach); -if (IS_ERR (hwpt)) { - destroy_hwpt =ERR_CAST (hwpt); + hwpt_paging =iommufd_hwpt_paging_alloc (idev->ictx, ioas, idev, +0 , immediate_attach); +if (IS_ERR (hwpt_paging)) { + destroy_hwpt =ERR_CAST (hwpt_paging); goto out_unlock; } + hwpt = &hwpt_paging->common;if (!immediate_attach) { destroy_hwpt = (*do_attach)(idev, hwpt);
@@ -605,7 +619,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, destroy_hwpt =NULL ; } - hwpt->auto_domain = true; + hwpt_paging->auto_domain = true; *pt_id = hwpt->obj.id;iommufd_object_finalize (idev->ictx, &hwpt->obj);
drivers/iommu/iommufd/hw_pagetable.c +54 -44
172 lines not on this page · raw message
drivers/iommu/iommufd/iommufd_private.h +26 -11
@@ -171,7 +171,7 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx,size_t size, enum iommufd_object_type type); -#define iommufd_object_alloc (ictx, ptr, type) \ +#define __iommufd_object_alloc (ictx, ptr, type, obj) \ container_of(_iommufd_object_alloc( \ ictx, \ sizeof(*(ptr)) + BUILD_BUG_ON_ZERO( \
@@ -180,6 +180,9 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx, type), \ typeof(*(ptr)), obj) +#define iommufd_object_alloc (ictx, ptr, type) \ + __iommufd_object_alloc(ictx, ptr, type, obj) +/* * The IO Address Space (IOAS) pagetable is a virtual page table backed by the * io_pagetable object. It is a user controlled mapping of IOVA -> PFNs. The
@@ -231,8 +234,12 @@ int iommufd_vfio_ioas(struct iommufd_ucmd *ucmd);*/ struct iommufd_hw_pagetable {struct iommufd_object obj; -struct iommufd_ioas *ioas;struct iommu_domain *domain; +}; + +struct iommufd_hwpt_paging { +struct iommufd_hw_pagetable common; +struct iommufd_ioas *ioas;bool auto_domain :1 ;bool enforce_cache_coherency :1 ;bool msi_cookie :1 ;
@@ -245,24 +252,32 @@ static inline bool hwpt_is_paging(struct iommufd_hw_pagetable *hwpt)return hwpt->obj.type ==IOMMUFD_OBJ_HWPT_PAGING ; } -struct iommufd_hw_pagetable * -iommufd_hw_pagetable_alloc (struct iommufd_ctx *ictx,struct iommufd_ioas *ioas, -struct iommufd_device *idev,u32 flags, -bool immediate_attach); -int iommufd_hw_pagetable_enforce_cc (struct iommufd_hw_pagetable *hwpt); +static inline struct iommufd_hwpt_paging * +to_hwpt_paging (struct iommufd_hw_pagetable *hwpt) +{ + return container_of(hwpt, struct iommufd_hwpt_paging, common); +} + +struct iommufd_hwpt_paging * +iommufd_hwpt_paging_alloc (struct iommufd_ctx *ictx,struct iommufd_ioas *ioas, +struct iommufd_device *idev,u32 flags, +bool immediate_attach); +int iommufd_hwpt_paging_enforce_cc (struct iommufd_hwpt_paging *hwpt_paging);int iommufd_hw_pagetable_attach (struct iommufd_hw_pagetable *hwpt,struct iommufd_device *idev);struct iommufd_hw_pagetable *iommufd_hw_pagetable_detach (struct iommufd_device *idev); -void iommufd_hw_pagetable_destroy (struct iommufd_object *obj); -void iommufd_hw_pagetable_abort (struct iommufd_object *obj); +void iommufd_hwpt_paging_destroy (struct iommufd_object *obj); +void iommufd_hwpt_paging_abort (struct iommufd_object *obj);int iommufd_hwpt_alloc (struct iommufd_ucmd *ucmd);static inline void iommufd_hw_pagetable_put (struct iommufd_ctx *ictx,struct iommufd_hw_pagetable *hwpt) { -lockdep_assert_not_held (&hwpt->ioas->mutex); -if (hwpt->auto_domain) +struct iommufd_hwpt_paging *hwpt_paging =to_hwpt_paging (hwpt); + +lockdep_assert_not_held (&hwpt_paging->ioas->mutex); +if (hwpt_paging->auto_domain)iommufd_object_deref_user (ictx, &hwpt->obj);else refcount_dec (&hwpt->obj.users);
drivers/iommu/iommufd/main.c +2 -2
@@ -483,8 +483,8 @@ static const struct iommufd_object_ops iommufd_object_ops[] = { .destroy = iommufd_ioas_destroy, }, [IOMMUFD_OBJ_HWPT_PAGING] = { - .destroy = iommufd_hw_pagetable_destroy, - .abort = iommufd_hw_pagetable_abort, + .destroy = iommufd_hwpt_paging_destroy, + .abort = iommufd_hwpt_paging_abort, }, #ifdef CONFIG_IOMMUFD_TEST [IOMMUFD_OBJ_SELFTEST] = {
drivers/iommu/iommufd/vfio_compat.c +3 -3
@@ -255,7 +255,7 @@ static int iommufd_vfio_unmap_dma(struct iommufd_ctx *ictx, unsigned int cmd,static int iommufd_vfio_cc_iommu (struct iommufd_ctx *ictx) { -struct iommufd_hw_pagetable *hwpt; +struct iommufd_hwpt_paging *hwpt_paging;struct iommufd_ioas *ioas;int rc =1 ;
@@ -264,8 +264,8 @@ static int iommufd_vfio_cc_iommu(struct iommufd_ctx *ictx)return PTR_ERR (ioas);mutex_lock (&ioas->mutex); -list_for_each_entry (hwpt, &ioas->hwpt_list, hwpt_item) { -if (!hwpt->enforce_cache_coherency) { +list_for_each_entry (hwpt_paging, &ioas->hwpt_list, hwpt_item) { +if (!hwpt_paging->enforce_cache_coherency) { rc =0 ;break ; }
Signature · 1 line
2.34.1
PATCH v5 6/10 iommufd: Share iommufd_hwpt_alloc with IOMMUFD_OBJ_HWPT_NESTED
In reply to Yi LiuFrom: Nicolin Chen <[email protected]>
Allow iommufd_hwpt_alloc() to have a common routine but jump to different allocators corresponding to different user input pt_obj types, either an IOMMUFD_OBJ_IOAS for a PAGING hwpt or an IOMMUFD_OBJ_HWPT_PAGING as the parent for a NESTED hwpt.
Also, move the "flags" validation to the hwpt allocator (paging), so that later the hwpt_nested allocator can do its own separate flags validation.
--- drivers/iommu/iommufd/hw_pagetable.c | 43 ++++++++++++++++++---------- 1 file changed, 28 insertions(+), 15 deletions(-)
drivers/iommu/iommufd/hw_pagetable.c +28 -15
@@ -81,6 +81,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, struct iommufd_device *idev, u32 flags, bool immediate_attach) { + const u32 valid_flags = IOMMU_HWPT_ALLOC_NEST_PARENT;const struct iommu_ops *ops =dev_iommu_ops (idev->dev);struct iommufd_hwpt_paging *hwpt_paging;struct iommufd_hw_pagetable *hwpt;
@@ -90,6 +91,8 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,if (flags && !ops->domain_alloc_user)return ERR_PTR (-EOPNOTSUPP ); +if (flags & ~valid_flags) +return ERR_PTR (-EOPNOTSUPP ); hwpt_paging = __iommufd_object_alloc(ictx, hwpt_paging, IOMMUFD_OBJ_HWPT_PAGING,
@@ -161,33 +164,41 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,int iommufd_hwpt_alloc (struct iommufd_ucmd *ucmd) {struct iommu_hwpt_alloc *cmd = ucmd->cmd; -struct iommufd_hwpt_paging *hwpt_paging;struct iommufd_hw_pagetable *hwpt; +struct iommufd_ioas *ioas =NULL ; +struct iommufd_object *pt_obj;struct iommufd_device *idev; -struct iommufd_ioas *ioas;int rc; -if ((cmd->flags & (~IOMMU_HWPT_ALLOC_NEST_PARENT )) || cmd->__reserved) +if (cmd->__reserved)return -EOPNOTSUPP ; idev =iommufd_get_device (ucmd, cmd->dev_id);if (IS_ERR (idev))return PTR_ERR (idev); - ioas =iommufd_get_ioas (ucmd->ictx, cmd->pt_id); -if (IS_ERR (ioas)) { - rc =PTR_ERR (ioas); + pt_obj =iommufd_get_object (ucmd->ictx, cmd->pt_id,IOMMUFD_OBJ_ANY ); +if (IS_ERR (pt_obj)) { + rc = -EINVAL ; goto out_put_idev; } -mutex_lock (&ioas->mutex); - hwpt_paging =iommufd_hwpt_paging_alloc (ucmd->ictx, ioas, idev, - cmd->flags, false); -if (IS_ERR (hwpt_paging)) { - rc =PTR_ERR (hwpt_paging); - goto out_unlock; +if (pt_obj->type ==IOMMUFD_OBJ_IOAS ) { +struct iommufd_hwpt_paging *hwpt_paging; + + ioas = container_of(pt_obj, struct iommufd_ioas, obj); +mutex_lock (&ioas->mutex); + hwpt_paging =iommufd_hwpt_paging_alloc (ucmd->ictx, ioas, idev, + cmd->flags, false); +if (IS_ERR (hwpt_paging)) { + rc =PTR_ERR (hwpt_paging); + goto out_unlock; + } + hwpt = &hwpt_paging->common; + }else { + rc = -EINVAL ; + goto out_put_pt; } - hwpt = &hwpt_paging->common; cmd->out_hwpt_id = hwpt->obj.id; rc =iommufd_ucmd_respond (ucmd,sizeof (*cmd));
@@ -199,8 +210,10 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd) out_hwpt:iommufd_object_abort_and_destroy (ucmd->ictx, &hwpt->obj); out_unlock: -mutex_unlock (&ioas->mutex); -iommufd_put_object (&ioas->obj); +if (ioas) +mutex_unlock (&ioas->mutex); +out_put_pt: +iommufd_put_object (pt_obj); out_put_idev:iommufd_put_object (&idev->obj);return rc;
Signature · 1 line
2.34.1
PATCH v5 7/10 iommufd: Add a nested HW pagetable object
In reply to Yi LiuFrom: Nicolin Chen <[email protected]>
IOMMU_HWPT_ALLOC already supports iommu_domain allocation for usersapce. But it can only allocate a hw_pagetable that associates to a given IOAS, i.e. only a kernel-managed hw_pagetable of IOMMUFD_OBJ_HWPT_PAGING type.
IOMMU drivers can now support user-managed hw_pagetables, for two-stage translation use cases that require user data input from the user space.
Add a new IOMMUFD_OBJ_HWPT_NESTED type with its abort/destroy(). Pair it with a new iommufd_hwpt_nested structure and its to_hwpt_nested() helper. Update the to_hwpt_paging() helper, so a NESTED-type hw_pagetable can be handled in the callers, for example iommufd_hw_pagetable_enforce_rr().
Screen the inputs including the parent PAGING-type hw_pagetable that has a need of a new nest_parent flag in the iommufd_hwpt_paging structure.
Extend the IOMMU_HWPT_ALLOC ioctl to accept an IOMMU driver specific data input. Also, update the @pt_id to accept hwpt_id too besides an ioas_id. Then, use them to allocate a hw_pagetable of IOMMUFD_OBJ_HWPT_NESTED type using the iommufd_hw_pagetable_alloc_nested() allocator.
--- drivers/iommu/iommufd/device.c | 1 + drivers/iommu/iommufd/hw_pagetable.c | 105 ++++++++++++++++++++++++ drivers/iommu/iommufd/iommufd_private.h | 29 +++++-- drivers/iommu/iommufd/main.c | 4 + include/uapi/linux/iommufd.h | 23 +++++- 5 files changed, 154 insertions(+), 8 deletions(-)
drivers/iommu/iommufd/device.c +1 -0
@@ -644,6 +644,7 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id,return PTR_ERR (pt_obj);switch (pt_obj->type) { +case IOMMUFD_OBJ_HWPT_NESTED :case IOMMUFD_OBJ_HWPT_PAGING : {struct iommufd_hw_pagetable *hwpt = container_of(pt_obj, struct iommufd_hw_pagetable, obj);
drivers/iommu/iommufd/hw_pagetable.c +105 -0
135 lines not on this page · raw message
drivers/iommu/iommufd/iommufd_private.h +23 -6
@@ -114,6 +114,7 @@ enum iommufd_object_type {IOMMUFD_OBJ_ANY =IOMMUFD_OBJ_NONE ,IOMMUFD_OBJ_DEVICE ,IOMMUFD_OBJ_HWPT_PAGING , +IOMMUFD_OBJ_HWPT_NESTED ,IOMMUFD_OBJ_IOAS ,IOMMUFD_OBJ_ACCESS , #ifdef CONFIG_IOMMUFD_TEST
@@ -243,10 +244,16 @@ struct iommufd_hwpt_paging { bool auto_domain : 1; bool enforce_cache_coherency : 1; bool msi_cookie : 1; + bool nest_parent : 1;/* Head at iommufd_ioas::hwpt_list */ struct list_head hwpt_item; }; +struct iommufd_hwpt_nested { +struct iommufd_hw_pagetable common; +struct iommufd_hwpt_paging *parent; +}; +static inline bool hwpt_is_paging (struct iommufd_hw_pagetable *hwpt) {return hwpt->obj.type ==IOMMUFD_OBJ_HWPT_PAGING ;
@@ -269,18 +276,28 @@ struct iommufd_hw_pagetable * iommufd_hw_pagetable_detach(struct iommufd_device *idev);void iommufd_hwpt_paging_destroy (struct iommufd_object *obj);void iommufd_hwpt_paging_abort (struct iommufd_object *obj); +void iommufd_hwpt_nested_destroy (struct iommufd_object *obj); +void iommufd_hwpt_nested_abort (struct iommufd_object *obj);int iommufd_hwpt_alloc (struct iommufd_ucmd *ucmd);static inline void iommufd_hw_pagetable_put (struct iommufd_ctx *ictx,struct iommufd_hw_pagetable *hwpt) { -struct iommufd_hwpt_paging *hwpt_paging =to_hwpt_paging (hwpt); +if (WARN_ON (hwpt->obj.type !=IOMMUFD_OBJ_HWPT_PAGING && + hwpt->obj.type !=IOMMUFD_OBJ_HWPT_NESTED )) +return ; -lockdep_assert_not_held (&hwpt_paging->ioas->mutex); -if (hwpt_paging->auto_domain) -iommufd_object_deref_user (ictx, &hwpt->obj); -else -refcount_dec (&hwpt->obj.users); +if (hwpt->obj.type ==IOMMUFD_OBJ_HWPT_PAGING ) { +struct iommufd_hwpt_paging *hwpt_paging =to_hwpt_paging (hwpt); + +lockdep_assert_not_held (&hwpt_paging->ioas->mutex); + +if (hwpt_paging->auto_domain) { +iommufd_object_deref_user (ictx, &hwpt->obj); +return ; + } + } +refcount_dec (&hwpt->obj.users); }struct iommufd_group {
drivers/iommu/iommufd/main.c +4 -0
@@ -486,6 +486,10 @@ static const struct iommufd_object_ops iommufd_object_ops[] = { .destroy = iommufd_hwpt_paging_destroy, .abort = iommufd_hwpt_paging_abort, }, + [IOMMUFD_OBJ_HWPT_NESTED] = { + .destroy = iommufd_hwpt_nested_destroy, + .abort = iommufd_hwpt_nested_abort, + }, #ifdef CONFIG_IOMMUFD_TEST [IOMMUFD_OBJ_SELFTEST] = { .destroy = iommufd_selftest_destroy,
include/uapi/linux/iommufd.h +21 -2
@@ -369,15 +369,31 @@ enum iommu_hwpt_data_type {* @size: sizeof(struct iommu_hwpt_alloc) * @flags: Combination of enum iommufd_hwpt_alloc_flags * @dev_id: The device to allocate this HWPT for -* @pt_id: The IOAS to connect this HWPT to +* @pt_id: The IOAS or HWPT to connect this HWPT to * @out_hwpt_id: The ID of the new HWPT * @__reserved: Must be 0 +* @data_type: One of enum iommu_hwpt_data_type +* @data_len: Length of the type specific data +* @data_uptr: User pointer to the type specific data * * Explicitly allocate a hardware page table object. This is the same object * type that is returned by iommufd_device_attach() and represents the * underlying iommu driver's iommu_domain kernel object. * -* A HWPT will be created with the IOVA mappings from the given IOAS. +* A kernel-managed HWPT will be created with the mappings from the given +* IOAS via the @pt_id. The @data_type for this allocation must be set to +* IOMMU_HWPT_DATA_NONE. The HWPT can be allocated as a parent HWPT for a +* nesting configuration by passing IOMMU_HWPT_ALLOC_NEST_PARENT via @flags. +* +* A user-managed nested HWPT will be created from a given parent HWPT via +* @pt_id, in which the parent HWPT must be allocated previously via the +* same ioctl from a given IOAS (@pt_id). In this case, the @data_type +* must be set to a pre-defined type corresponding to an I/O page table +* type supported by the underlying IOMMU hardware. +* +* If the @data_type is set to IOMMU_HWPT_DATA_NONE, @data_len and +* @data_uptr should be zero. Otherwise, both @data_len and @data_uptr +* must be given. */ struct iommu_hwpt_alloc {__u32 size;
@@ -386,6 +402,9 @@ struct iommu_hwpt_alloc {__u32 pt_id;__u32 out_hwpt_id;__u32 __reserved; +__u32 data_type; +__u32 data_len; +__aligned_u64 data_uptr; };#define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
Signature · 1 line
2.34.1
PATCH v5 8/10 iommu: Add iommu_copy_struct_from_user helper
In reply to Yi LiuFrom: Nicolin Chen <[email protected]>
Wrap up the data type/pointer/len sanity and a copy_struct_from_user call for iommu drivers to copy driver specific data via struct iommu_user_data. And expect it to be used in the domain_alloc_user op for example.
--- include/linux/iommu.h | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
include/linux/iommu.h +41 -0
@@ -246,6 +246,47 @@ struct iommu_user_data {size_t len; }; +/** +* __iommu_copy_struct_from_user - Copy iommu driver specific user space data +* @dst_data: Pointer to an iommu driver specific user data that is defined in +* include/uapi/linux/iommufd.h +* @src_data: Pointer to a struct iommu_user_data for user space data info +* @data_type: The data type of the @dst_data. Must match with @src_data.type +* @data_len: Length of current user data structure, i.e. sizeof(struct _dst) +* @min_len: Initial length of user data structure for backward compatibility. +* This should be offsetofend using the last member in the user data +* struct that was initially added to include/uapi/linux/iommufd.h +*/ +static inline int +__iommu_copy_struct_from_user (void *dst_data, +const struct iommu_user_data *src_data, +unsigned int data_type,size_t data_len, +size_t min_len) +{ +if (src_data->type != data_type) +return -EINVAL ; +if (WARN_ON (!dst_data || !src_data)) +return -EINVAL ; +if (src_data->len < min_len || data_len < src_data->len) +return -EINVAL ; +return copy_struct_from_user (dst_data, data_len, + src_data->uptr, src_data->len); +} + +/** +* iommu_copy_struct_from_user - Copy iommu driver specific user space data +* @kdst: Pointer to an iommu driver specific user data that is defined in +* include/uapi/linux/iommufd.h +* @user_data: Pointer to a struct iommu_user_data for user space data info +* @data_type: The data type of the @kdst. Must match with @user_data->type +* @min_last: The last memember of the data structure @kdst points in the +* initial version. +* Return 0 for success, otherwise -error. +*/ +#define iommu_copy_struct_from_user (kdst, user_data, data_type, min_last) \ + __iommu_copy_struct_from_user(kdst, user_data, data_type, sizeof(*kdst), \ + offsetofend(typeof(*kdst), min_last)) +/** * struct iommu_ops - iommu ops and capabilities * @capable: check capability
Signature · 1 line
2.34.1
PATCH v5 9/10 iommufd/selftest: Add nested domain allocation for mock domain
In reply to Yi LiuFrom: Nicolin Chen <[email protected]>
Add nested domain support in the ->domain_alloc_user op with some proper sanity checks. Then, add a domain_nested_ops for all nested domains and split the get_md_pagetable helper into paging and nested helpers.
Also, add an iotlb as a testing property of a nested domain.
--- drivers/iommu/iommufd/iommufd_test.h | 18 ++++ drivers/iommu/iommufd/selftest.c | 135 +++++++++++++++++++++++---- 2 files changed, 133 insertions(+), 20 deletions(-)
drivers/iommu/iommufd/iommufd_test.h +18 -0
@@ -40,6 +40,11 @@ enum { MOCK_FLAGS_ACCESS_CREATE_NEEDS_PIN_PAGES = 1 << 0, }; +enum { +MOCK_NESTED_DOMAIN_IOTLB_ID_MAX =3 , +MOCK_NESTED_DOMAIN_IOTLB_NUM =4 , +}; +struct iommu_test_cmd {__u32 size;__u32 op;
@@ -109,4 +114,17 @@ struct iommu_test_hw_info { __u32 test_reg; }; +/* Should not be equal to any defined value in enum iommu_hwpt_data_type */ +#define IOMMU_HWPT_DATA_SELFTEST 0xdead + +/** +* struct iommu_hwpt_selftest +* +* @iotlb: default mock iotlb value, IOMMU_TEST_IOTLB_DEFAULT +*/ +struct iommu_hwpt_selftest { +#define IOMMU_TEST_IOTLB_DEFAULT 0xbadbeef +__u32 iotlb; +}; +#endif
drivers/iommu/iommufd/selftest.c +115 -20
186 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v5 10/10 iommufd/selftest: Add coverage for IOMMU_HWPT_ALLOC with nested HWPTs
In reply to Yi LiuFrom: Nicolin Chen <[email protected]>
The IOMMU_HWPT_ALLOC ioctl now supports passing user_data to allocate a user-managed domain for nested HWPTs. Add its coverage for that. Also, update _test_cmd_hwpt_alloc() and add test_cmd/err_hwpt_alloc_nested().
--- tools/testing/selftests/iommu/iommufd.c | 120 ++++++++++++++++++ .../selftests/iommu/iommufd_fail_nth.c | 3 +- tools/testing/selftests/iommu/iommufd_utils.h | 31 ++++- 3 files changed, 146 insertions(+), 8 deletions(-)
tools/testing/selftests/iommu/iommufd.c +120 -0
@@ -264,6 +264,126 @@ TEST_F(iommufd_ioas, ioas_destroy) } } +TEST_F(iommufd_ioas, alloc_hwpt_nested) +{ +const uint32_t min_data_len = + offsetofend(struct iommu_hwpt_selftest, iotlb); +struct iommu_hwpt_selftest data = { + .iotlb =IOMMU_TEST_IOTLB_DEFAULT , + }; +uint32_t nested_hwpt_id[2 ] = {}; +uint32_t parent_hwpt_id =0 ; +uint32_t parent_hwpt_id_not_work =0 ; +uint32_t test_hwpt_id =0 ; + +if (self->device_id) { +/* Negative tests */ +test_err_hwpt_alloc (ENOENT , self->ioas_id, self->device_id, +0 , &test_hwpt_id); +test_err_hwpt_alloc (EINVAL , self->device_id, + self->device_id,0 , &test_hwpt_id); + +test_cmd_hwpt_alloc (self->device_id, self->ioas_id, +IOMMU_HWPT_ALLOC_NEST_PARENT , + &parent_hwpt_id); + +test_cmd_hwpt_alloc (self->device_id, self->ioas_id, +0 , &parent_hwpt_id_not_work); + +/* Negative nested tests */ +test_err_hwpt_alloc_nested (EINVAL , + self->device_id, parent_hwpt_id, +0 , &nested_hwpt_id[0 ], +IOMMU_HWPT_DATA_NONE , + &data,sizeof (data)); +test_err_hwpt_alloc_nested (EOPNOTSUPP , + self->device_id, parent_hwpt_id, +0 , &nested_hwpt_id[0 ], +IOMMU_HWPT_DATA_SELFTEST +1 , + &data,sizeof (data)); +test_err_hwpt_alloc_nested (EINVAL , + self->device_id, parent_hwpt_id, +0 , &nested_hwpt_id[0 ], +IOMMU_HWPT_DATA_SELFTEST , + &data, min_data_len -1 ); +test_err_hwpt_alloc_nested (EFAULT , + self->device_id, parent_hwpt_id, +0 , &nested_hwpt_id[0 ], +IOMMU_HWPT_DATA_SELFTEST , +NULL ,sizeof (data)); +test_err_hwpt_alloc_nested (EOPNOTSUPP , + self->device_id, parent_hwpt_id, +IOMMU_HWPT_ALLOC_NEST_PARENT , + &nested_hwpt_id[0 ], +IOMMU_HWPT_DATA_SELFTEST , + &data,sizeof (data)); +test_err_hwpt_alloc_nested (EINVAL , self->device_id, + parent_hwpt_id_not_work, +0 , &nested_hwpt_id[0 ], +IOMMU_HWPT_DATA_SELFTEST , + &data,sizeof (data)); + +/* Allocate two nested hwpts sharing one common parent hwpt */ +test_cmd_hwpt_alloc_nested (self->device_id, parent_hwpt_id, +0 , &nested_hwpt_id[0 ], +IOMMU_HWPT_DATA_SELFTEST , + &data,sizeof (data)); +test_cmd_hwpt_alloc_nested (self->device_id, parent_hwpt_id, +0 , &nested_hwpt_id[1 ], +IOMMU_HWPT_DATA_SELFTEST , + &data,sizeof (data)); + +/* Negative test: a nested hwpt on top of a nested hwpt */ +test_err_hwpt_alloc_nested (EINVAL , + self->device_id, nested_hwpt_id[0 ], +0 , &test_hwpt_id, +IOMMU_HWPT_DATA_SELFTEST , + &data,sizeof (data)); +/* Negative test: parent hwpt now cannot be freed */ +EXPECT_ERRNO (EBUSY , +_test_ioctl_destroy (self->fd, parent_hwpt_id)); + +/* Attach device to nested_hwpt_id[0] that then will be busy */ +test_cmd_mock_domain_replace (self->stdev_id, + nested_hwpt_id[0 ]); +EXPECT_ERRNO (EBUSY , +_test_ioctl_destroy (self->fd, nested_hwpt_id[0 ])); + +/* Switch from nested_hwpt_id[0] to nested_hwpt_id[1] */ +test_cmd_mock_domain_replace (self->stdev_id, + nested_hwpt_id[1 ]); +EXPECT_ERRNO (EBUSY , +_test_ioctl_destroy (self->fd, nested_hwpt_id[1 ])); +test_ioctl_destroy (nested_hwpt_id[0 ]); + +/* Detach from nested_hwpt_id[1] and destroy it */ +test_cmd_mock_domain_replace (self->stdev_id, parent_hwpt_id); +test_ioctl_destroy (nested_hwpt_id[1 ]); + +/* Detach from the parent hw_pagetable and destroy it */ +test_cmd_mock_domain_replace (self->stdev_id, self->ioas_id); +test_ioctl_destroy (parent_hwpt_id); +test_ioctl_destroy (parent_hwpt_id_not_work); + }else { +test_err_hwpt_alloc (ENOENT , self->device_id, self->ioas_id, +0 , &parent_hwpt_id); +test_err_hwpt_alloc_nested (ENOENT , + self->device_id, parent_hwpt_id, +0 , &nested_hwpt_id[0 ], +IOMMU_HWPT_DATA_SELFTEST , + &data,sizeof (data)); +test_err_hwpt_alloc_nested (ENOENT , + self->device_id, parent_hwpt_id, +0 , &nested_hwpt_id[1 ], +IOMMU_HWPT_DATA_SELFTEST , + &data,sizeof (data)); +test_err_mock_domain_replace (ENOENT , + self->stdev_id, nested_hwpt_id[0 ]); +test_err_mock_domain_replace (ENOENT , + self->stdev_id, nested_hwpt_id[1 ]); + } +} + TEST_F(iommufd_ioas, hwpt_attach) {/* Create a device attached directly to a hwpt */
tools/testing/selftests/iommu/iommufd_fail_nth.c +2 -1
@@ -615,7 +615,8 @@ TEST_FAIL_NTH(basic_fail_nth, device)if (_test_cmd_get_hw_info (self->fd, idev_id, &info,sizeof (info)))return -1 ; -if (_test_cmd_hwpt_alloc (self->fd, idev_id, ioas_id,0 , &hwpt_id)) +if (_test_cmd_hwpt_alloc (self->fd, idev_id, ioas_id,0 , &hwpt_id, +IOMMU_HWPT_DATA_NONE ,0 ,0 ))return -1 ;if (_test_cmd_mock_domain_replace (self->fd, stdev_id, ioas_id2,NULL ))
tools/testing/selftests/iommu/iommufd_utils.h +24 -7
@@ -103,13 +103,17 @@ static int _test_cmd_mock_domain_replace(int fd, __u32 stdev_id, __u32 pt_id, pt_id, NULL))static int _test_cmd_hwpt_alloc (int fd,__u32 device_id,__u32 pt_id, -__u32 flags,__u32 *hwpt_id) +__u32 flags,__u32 *hwpt_id,__u32 data_type, +void *data,size_t data_len) {struct iommu_hwpt_alloc cmd = { .size =sizeof (cmd), .flags = flags, .dev_id = device_id, .pt_id = pt_id, + .data_type = data_type, + .data_len = data_len, + .data_uptr = (uint64_t )data, };int ret;
@@ -121,12 +125,25 @@ static int _test_cmd_hwpt_alloc(int fd, __u32 device_id, __u32 pt_id,return 0 ; } -#define test_cmd_hwpt_alloc (device_id, pt_id, flags, hwpt_id) \ - ASSERT_EQ(0, _test_cmd_hwpt_alloc(self->fd, device_id, \ - pt_id, flags, hwpt_id)) -#define test_err_hwpt_alloc (_errno, device_id, pt_id, flags, hwpt_id) \ - EXPECT_ERRNO(_errno, _test_cmd_hwpt_alloc(self->fd, device_id, \ - pt_id, flags, hwpt_id)) +#define test_cmd_hwpt_alloc (device_id, pt_id, flags, hwpt_id) \ + ASSERT_EQ(0, _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, flags, \ + hwpt_id, IOMMU_HWPT_DATA_NONE, \ + NULL, 0)) +#define test_err_hwpt_alloc (_errno, device_id, pt_id, flags, hwpt_id) \ + EXPECT_ERRNO(_errno, _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, \ + flags, hwpt_id, \ + IOMMU_HWPT_DATA_NONE, \ + NULL, 0)) + +#define test_cmd_hwpt_alloc_nested (device_id, pt_id, flags, hwpt_id, \ + data_type, data, data_len) \ + ASSERT_EQ(0, _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, flags, \ + hwpt_id, data_type, data, data_len)) +#define test_err_hwpt_alloc_nested (_errno, device_id, pt_id, flags, hwpt_id, \ + data_type, data, data_len) \ + EXPECT_ERRNO(_errno, \ + _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, flags, \ + hwpt_id, data_type, data, data_len))static int _test_cmd_access_replace_ioas (int fd,__u32 access_id,unsigned int ioas_id)
Signature · 1 line
2.34.1