iommu: Add support to change default domain of an iommu group
v4 cover iommu: Add support to change default domain of an iommu group
Presently, the default domain of an iommu group is allocated during boot time and it cannot be changed later. So, the device would typically be either in identity (pass_through) mode or the device would be in DMA mode as long as the system is up and running. There is no way to change the default domain type dynamically i.e. after booting, a device cannot switch between identity mode and DMA mode.
Assume a use case wherein the privileged user would want to use the device in pass-through mode when the device is used for host so that it would be high performing. Presently, this is not supported. Hence add support to change the default domain of an iommu group dynamically.
Support this by writing to a sysfs file, namely
"/sys/kernel/iommu_groups/<grp_id>/type".
Testing:
--------
Tested by dynamically changing storage device (nvme) from
1. identity mode to DMA and making sure file transfer works
2. DMA mode to identity mode and making sure file transfer works Tested only for intel_iommu/vt-d. Would appreciate if someone could test on AMD and ARM based machines.
Based on iommu maintainer's 'next' branch.
Changes from V3:
----------------
1. Made changes to commit message as suggested by Baolu.
2. Don't pass "prev_dom" and "dev" as parameters to iommu_change_dev_def_domain(). Instead get them from group.
3. Sanitize the logic to validate user default domain type request. The logic remains same but is implmented differently.
4. Push lot of error checking into iommu_change_dev_def_domain() from iommu_group_store_type().
5. iommu_change_dev_def_domain() takes/releases group mutex as needed. So, it shouldn't be called holding a group mutex.
6. Use pr_err_ratelimited() instead of pr_err() to avoid DOS attack.
Changes from V2:
----------------
1. Change the logic of updating default domain from V2 because ops->probe_finalize() could be used to update dma_ops.
2. Drop 1st and 2nd patch of V2 series because they are no longer needed on iommu maintainer's 'next' branch.
3. Limit this feature to iommu groups with only one device.
4. Hold device_lock and group mutex until the default domain is changed.
Changes from V1:
----------------
1. V1 patch set wasn't updating dma_ops for some vendors (Eg: AMD), hence, change the logic of updating default domain as below (because adding a device to iommu_group automatically updates dma_ops)
a. Allocate a new domain
b. For every device in the group
i. Remove the device from the group
ii. Add the device back to the group
c. Free previous domain
2. Drop 1st patch of V1 (iommu/vt-d: Modify device_def_domain_type() to use at runtime) because "iommu=pt" has no effect on this function anymore.
3. Added a patch to take/release lock while reading iommu_group->default_domain->type because it can be changed any time by user.
4. Before changing default domain type of a group, check if the group is directly assigned for user level access. If so, abort.
5. Sanitize return path (using ternary operator) in iommu_group_store_type()
6. Split 2nd patch of V1 (iommu: Add device_def_domain_type() call back function to iommu_ops) into two patches such that iommu generic changes are now in 1st patch of V2 and vt-d specific changes are in 2nd patch of V2.
7. Rename device_def_domain_type() to dev_def_domain_type()
8. Remove example from documentation
9. Change the value written to file "/sys/kernel/iommu_groups/<grp_id>/type" from "dma" to "DMA".
Changes from RFC:
-----------------
1. Added support for "auto" type, so that kernel selects one among identity or dma mode.
2. Use "system_state" in device_def_domain_type() instead of an argument.
Sai Praneeth Prakhya (3):
iommu: Add support to change default domain of an iommu_group
iommu: Take lock before reading iommu_group default domain type
iommu: Document usage of "/sys/kernel/iommu_groups/<grp_id>/type" file
.../ABI/testing/sysfs-kernel-iommu_groups | 30 +++ drivers/iommu/iommu.c | 217 +++++++++++++++++- 2 files changed, 246 insertions(+), 1 deletion(-)
Signature · 6 lines
2.19.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v4 1/3 iommu: Add support to change default domain of an iommu group
Presently, the default domain of an iommu group is allocated during boot time and it cannot be changed later. So, the device would typically be either in identity (also known as pass_through) mode or the device would be in DMA mode as long as the machine is up and running. There is no way to change the default domain type dynamically i.e. after booting, a device cannot switch between identity mode and DMA mode.
But, assume a use case wherein the user trusts the device and believes that the OS is secure enough and hence wants *only* this device to bypass IOMMU (so that it could be high performing) whereas all the other devices to go through IOMMU (so that the system is protected). Presently, this use case is not supported. It will be helpful if there is some way to change the default domain of an iommu group dynamically. Hence, add such support.
A privileged user could request the kernel to change the default domain type of a iommu group by writing to "/sys/kernel/iommu_groups/<grp_id>/type" file. Presently, only three values are supported
1. identity: all the DMA transactions from the device in this group are *not* translated by the iommu
2. DMA: all the DMA transactions from the device in this group are translated by the iommu
3. auto: change to the type the device was booted with
Note:
1. Default domain of an iommu group with two or more devices cannot be changed.
2. The device in the iommu group shouldn't be bound to any driver.
3. The device shouldn't be assigned to user for direct access.
4. The vendor iommu driver is required to add def_domain_type() callback. The change request will fail if the request type conflicts with that returned from the callback.
Please see "Documentation/ABI/testing/sysfs-kernel-iommu_groups" for more information.
--- drivers/iommu/iommu.c | 215 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 214 insertions(+), 1 deletion(-)
drivers/iommu/iommu.c +214 -1
@@ -93,6 +93,8 @@ static void __iommu_detach_group(struct iommu_domain *domain,static int iommu_create_device_direct_mappings (struct iommu_group *group,struct device *dev);static struct iommu_group *iommu_group_get_for_dev (struct device *dev); +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count);#define IOMMU_GROUP_ATTR (_name, _mode, _show, _store) \ struct iommu_group_attribute iommu_group_attr_##_name = \
@@ -525,7 +527,8 @@ static IOMMU_GROUP_ATTR(name, S_IRUGO, iommu_group_show_name, NULL); static IOMMU_GROUP_ATTR(reserved_regions, 0444, iommu_group_show_resv_regions, NULL); -static IOMMU_GROUP_ATTR(type, 0444, iommu_group_show_type, NULL); +static IOMMU_GROUP_ATTR(type, 0644, iommu_group_show_type, + iommu_group_store_type);static void iommu_group_release (struct kobject *kobj) {
@@ -2838,3 +2841,213 @@ int iommu_sva_get_pasid(struct iommu_sva *handle)return ops->sva_get_pasid (handle); }EXPORT_SYMBOL_GPL (iommu_sva_get_pasid); + +/* +* Changes the default domain of an iommu group +* +* @group: The group for which the default domain should be changed +* @type: The type of the new default domain that gets associated with the group +* +* Returns 0 on success and error code on failure +* +* Note: +* 1. Presently, this function is called only when user requests to change the +* group's default domain type through /sys/kernel/iommu_groups/<grp_id>/type +* Please take a closer look if intended to use for other purposes. +*/ +static int iommu_change_dev_def_domain (struct iommu_group *group,int type) +{ +struct iommu_domain *prev_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops; +int ret, dev_def_dom; +struct device *dev; + +if (!group) +return -EINVAL ; + +mutex_lock (&group->mutex); + +if (group->default_domain != group->domain) { +pr_err_ratelimited ("Group assigned to user level for direct access\n" ); + ret = -EBUSY ; + goto out; + } + +/* +* iommu group wasn't locked while acquiring device lock in +* iommu_group_store_type(). So, make sure that the device count hasn't +* changed while acquiring device lock. +* +* Changing default domain of an iommu group with two or more devices +* isn't supported because there could be a potential deadlock. Consider +* the following scenario. T1 is trying to acquire device locks of all +* the devices in the group and before it could acquire all of them, +* there could be another thread T2 (from different sub-system and use +* case) that has already acquired some of the device locks and might be +* waiting for T1 to release other device locks. +*/ +if (iommu_group_device_count (group) !=1 ) { +pr_err_ratelimited ("Cannot change default domain of a group with two or more devices\n" ); + ret = -EINVAL ; + goto out; + } + +/* Since group has only one device */ +list_for_each_entry (grp_dev, &group->devices, list) + dev = grp_dev->dev; + + prev_dom = group->default_domain; +if (!prev_dom || !prev_dom->ops || !prev_dom->ops->def_domain_type) { +pr_err_ratelimited ("'def_domain_type' call back isn't registered\n" ); + ret = -EINVAL ; + goto out; + } + + ops = prev_dom->ops; + dev_def_dom = ops->def_domain_type (dev); + +/* Check if user requested domain is supported by the device or not */ +if (!type) { +/* +* If the user hasn't requested any specific type of domain and +* if the device supports both the domains, then default to the +* domain the device was booted with +*/ + type = dev_def_dom ?: iommu_def_domain_type; + }else if (dev_def_dom && type != dev_def_dom) { +pr_err_ratelimited ("Device cannot be in %s domain\n" , +iommu_domain_type_str (type)); + ret = -EINVAL ; + goto out; + } + +/* +* Switch to a new domain only if the requested domain type is different +* from the existing default domain type +*/ +if (prev_dom->type == type) { + ret =0 ; + goto out; + } + +/* Sets group->default_domain to the newly allocated domain */ + ret =iommu_group_alloc_default_domain (dev->bus, group, type); +if (ret) + goto out; + + ret =__iommu_attach_device (group->default_domain, dev); +if (ret) + goto free_new_domain; + + group->domain = group->default_domain; + + ret =iommu_create_device_direct_mappings (group, dev); +if (ret) + goto free_new_domain; + +/* +* Release the mutex here because ops->probe_finalize() call-back of +* some vendor IOMMU drivers calls arm_iommu_attach_device() which +* in-turn might call back into IOMMU core code, where it tries to take +* group->mutex, resulting in a deadlock. +*/ +mutex_unlock (&group->mutex); + +/* Make sure dma_ops is appropriatley set */ +iommu_group_do_probe_finalize (dev, group->default_domain); +iommu_domain_free (prev_dom); +return 0 ; + +free_new_domain: +iommu_domain_free (group->default_domain); + group->default_domain = prev_dom; + group->domain = prev_dom; + +out: +mutex_unlock (&group->mutex); + +return ret; +} + +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) +{ +struct group_device *grp_dev; +struct device *dev; +int ret, req_type; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" )) + req_type =0 ; +else +return -EINVAL ; + +/* +* Lock/Unlock the group mutex here before device lock to +* 1. Make sure that the iommu group has only one device (this is a +* prerequisite for step 2) +* 2. Get struct *dev which is needed to lock device +*/ +mutex_lock (&group->mutex); +if (iommu_group_device_count (group) !=1 ) { +mutex_unlock (&group->mutex); +pr_err_ratelimited ("Cannot change default domain of a group with two or more devices\n" ); +return -EINVAL ; + } + +/* Since group has only one device */ +list_for_each_entry (grp_dev, &group->devices, list) + dev = grp_dev->dev; + +/* +* Don't hold the group mutex because taking group mutex first and then +* the device lock could potentially cause a deadlock as below. Assume +* two threads T1 and T2. T1 is trying to change default domain of an +* iommu group and T2 is trying to hot unplug a device or release [1] VF +* of a PCIe device which is in the same iommu group. T1 takes group +* mutex and before it could take device lock assume T2 has taken device +* lock and is yet to take group mutex. Now, both the threads will be +* waiting for the other thread to release lock. Below, lock order was +* suggested. +* device_lock(dev); +* mutex_lock(&group->mutex); +* iommu_change_dev_def_domain(); +* mutex_unlock(&group->mutex); +* device_unlock(dev); +* +* [1] Typical device release path +* device_lock() from device/driver core code +* -> bus_notifier() +* -> iommu_bus_notifier() +* -> iommu_release_device() +* -> ops->release_device() vendor driver calls back iommu core code +* -> mutex_lock() from iommu core code +*/ +mutex_unlock (&group->mutex); + +/* Check if the device in the group still has a driver bound to it */ +device_lock (dev); +if (device_is_bound (dev)) { +pr_err_ratelimited ("Device is still bound to driver\n" ); + ret = -EBUSY ; + goto out; + } + + ret =iommu_change_dev_def_domain (group, req_type); + ret = ret ?: count; + +out: +device_unlock (dev); + +return ret; +}
Signature · 6 lines
2.19.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 33 lines · Sai Praneeth Prakhya
Presently, the default domain of an iommu group is allocated during boot time and it cannot be changed later. So, the device would typically be either in identity (also known as
pass_through) mode or the device would be in DMA mode as long as the machine is up and running. There is no way to change the default domain type dynamically i.e. after booting, a device cannot switch between identity mode and DMA mode.But, assume a use case wherein the user trusts the device and believes that the OS is secure enough and hence wants *only* this device to bypass IOMMU (so that it could be high performing) whereas all the other devices to go through IOMMU (so that the system is protected). Presently, this use case is not supported. It will be helpful if there is some way to change the default domain of an iommu group dynamically. Hence, add such support.
A privileged user could request the kernel to change the default domain type of a iommu group by writing to "/sys/kernel/iommu_groups/<grp_id>/type" file. Presently, only three values are supported
1. identity: all the DMA transactions from the device in this group are *not* translated by the iommu
2. DMA: all the DMA transactions from the device in this group are translated by the iommu
3. auto: change to the type the device was booted withNote:
1. Default domain of an iommu group with two or more devices cannot be changed.
2. The device in the iommu group shouldn't be bound to any driver.
3. The device shouldn't be assigned to user for direct access.
4. The vendor iommu driver is required to adddef_domain_type()callback. The change request will fail if the request type conflicts with that returned from the callback.
Quoted 8 lines
Please see "
Documentation/ABI/testing/sysfs-kernel-iommu_groups" for more information.Cc Christoph Hellwig <[email protected]>Cc Joerg Roedel <[email protected]>Cc Ashok Raj <[email protected]>Cc Will Deacon <[email protected]>Cc Lu Baolu <[email protected]>
This patch looks good to me now.
Best regards,
baolu
Quoted 236 lines
Cc Sohil Mehta <[email protected]>Cc Robin Murphy <[email protected]>Cc Jacob Pan <[email protected]>Signed-off-by Sai Praneeth Prakhya <[email protected]>--- drivers/iommu/iommu.c | 215 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 214 insertions(+), 1 deletion(-)drivers/iommu/iommu.c +204
-1@@ -93,6 +93,8 @@ static void __iommu_detach_group(struct iommu_domain *domain,static int iommu_create_device_direct_mappings (struct iommu_group *group,struct device *dev);static struct iommu_group *iommu_group_get_for_dev (struct device *dev); +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count);#define IOMMU_GROUP_ATTR (_name, _mode, _show, _store) \ struct iommu_group_attribute iommu_group_attr_##_name = \@@ -525,7 +527,8 @@ static IOMMU_GROUP_ATTR(name, S_IRUGO, iommu_group_show_name, NULL); static IOMMU_GROUP_ATTR(reserved_regions, 0444, iommu_group_show_resv_regions, NULL); -static IOMMU_GROUP_ATTR(type, 0444, iommu_group_show_type, NULL); +static IOMMU_GROUP_ATTR(type, 0644, iommu_group_show_type, + iommu_group_store_type);static void iommu_group_release (struct kobject *kobj) {@@ -2838,3 +2841,213 @@ int iommu_sva_get_pasid(struct iommu_sva *handle)return ops->sva_get_pasid (handle); }EXPORT_SYMBOL_GPL (iommu_sva_get_pasid); + +/* +* Changes the default domain of an iommu group +* +* @group: The group for which the default domain should be changed +* @type: The type of the new default domain that gets associated with the group +* +* Returns 0 on success and error code on failure +* +* Note: +* 1. Presently, this function is called only when user requests to change the +* group's default domain type through /sys/kernel/iommu_groups/<grp_id>/type +* Please take a closer look if intended to use for other purposes. +*/ +static int iommu_change_dev_def_domain (struct iommu_group *group,int type) +{ +struct iommu_domain *prev_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops; +int ret, dev_def_dom; +struct device *dev; + +if (!group) +return -EINVAL ; + +mutex_lock (&group->mutex); + +if (group->default_domain != group->domain) { +pr_err_ratelimited ("Group assigned to user level for direct access\n" ); + ret = -EBUSY ; + goto out; + } + +/* +* iommu group wasn't locked while acquiring device lock in +* iommu_group_store_type(). So, make sure that the device count hasn't +* changed while acquiring device lock. +* +* Changing default domain of an iommu group with two or more devices +* isn't supported because there could be a potential deadlock. Consider +* the following scenario. T1 is trying to acquire device locks of all +* the devices in the group and before it could acquire all of them, +* there could be another thread T2 (from different sub-system and use +* case) that has already acquired some of the device locks and might be +* waiting for T1 to release other device locks. +*/ +if (iommu_group_device_count (group) !=1 ) { +pr_err_ratelimited ("Cannot change default domain of a group with two or more devices\n" ); + ret = -EINVAL ; + goto out; + } + +/* Since group has only one device */ +list_for_each_entry (grp_dev, &group->devices, list) + dev = grp_dev->dev; + + prev_dom = group->default_domain; +if (!prev_dom || !prev_dom->ops || !prev_dom->ops->def_domain_type) { +pr_err_ratelimited ("'def_domain_type' call back isn't registered\n" ); + ret = -EINVAL ; + goto out; + } + + ops = prev_dom->ops; + dev_def_dom = ops->def_domain_type (dev); + +/* Check if user requested domain is supported by the device or not */ +if (!type) { +/* +* If the user hasn't requested any specific type of domain and +* if the device supports both the domains, then default to the +* domain the device was booted with +*/ + type = dev_def_dom ?: iommu_def_domain_type; + }else if (dev_def_dom && type != dev_def_dom) { +pr_err_ratelimited ("Device cannot be in %s domain\n" , +iommu_domain_type_str (type)); + ret = -EINVAL ; + goto out; + } + +/* +* Switch to a new domain only if the requested domain type is different +* from the existing default domain type +*/ +if (prev_dom->type == type) { + ret =0 ; + goto out; + } + +/* Sets group->default_domain to the newly allocated domain */ + ret =iommu_group_alloc_default_domain (dev->bus, group, type); +if (ret) + goto out; + + ret =__iommu_attach_device (group->default_domain, dev); +if (ret) + goto free_new_domain; + + group->domain = group->default_domain; + + ret =iommu_create_device_direct_mappings (group, dev); +if (ret) + goto free_new_domain; + +/* +* Release the mutex here because ops->probe_finalize() call-back of +* some vendor IOMMU drivers calls arm_iommu_attach_device() which +* in-turn might call back into IOMMU core code, where it tries to take +* group->mutex, resulting in a deadlock. +*/ +mutex_unlock (&group->mutex); + +/* Make sure dma_ops is appropriatley set */ +iommu_group_do_probe_finalize (dev, group->default_domain); +iommu_domain_free (prev_dom); +return 0 ; + +free_new_domain: +iommu_domain_free (group->default_domain); + group->default_domain = prev_dom; + group->domain = prev_dom; + +out: +mutex_unlock (&group->mutex); + +return ret; +} + +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) +{ +struct group_device *grp_dev; +struct device *dev; +int ret, req_type; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" )) + req_type =0 ; +else +return -EINVAL ; + +/* +* Lock/Unlock the group mutex here before device lock to +* 1. Make sure that the iommu group has only one device (this is a +* prerequisite for step 2) +* 2. Get struct *dev which is needed to lock device +*/ +mutex_lock (&group->mutex); +if (iommu_group_device_count (group) !=1 ) { +mutex_unlock (&group->mutex); +pr_err_ratelimited ("Cannot change default domain of a group with two or more devices\n" ); +return -EINVAL ; + } + +/* Since group has only one device */ +list_for_each_entry (grp_dev, &group->devices, list) + dev = grp_dev->dev; + +/* +* Don't hold the group mutex because taking group mutex first and then +* the device lock could potentially cause a deadlock as below. Assume +* two threads T1 and T2. T1 is trying to change default domain of an +* iommu group and T2 is trying to hot unplug a device or release [1] VF +* of a PCIe device which is in the same iommu group. T1 takes group +* mutex and before it could take device lock assume T2 has taken device +* lock and is yet to take group mutex. Now, both the threads will be +* waiting for the other thread to release lock. Below, lock order was +* suggested. +* device_lock(dev); +* mutex_lock(&group->mutex); +* iommu_change_dev_def_domain(); +* mutex_unlock(&group->mutex); +* device_unlock(dev); +* +* [1] Typical device release path +* device_lock() from device/driver core code +* -> bus_notifier() +* -> iommu_bus_notifier() +* -> iommu_release_device() +* -> ops->release_device() vendor driver calls back iommu core code +* -> mutex_lock() from iommu core code +*/ +mutex_unlock (&group->mutex); + +/* Check if the device in the group still has a driver bound to it */ +device_lock (dev); +if (device_is_bound (dev)) { +pr_err_ratelimited ("Device is still bound to driver\n" ); + ret = -EBUSY ; + goto out;
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 3053
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ } + + ret =iommu_change_dev_def_domain (group, req_type); + ret = ret ?: count; + +out: +device_unlock (dev); + +return ret; +}
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Joerg Roedel
maintainerQuoted 5 lines · Sai Praneeth Prakhya
drivers/iommu/iommu.c +5
-0+static int iommu_change_dev_def_domain (struct iommu_group *group,int type) +{ +struct iommu_domain *prev_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops;
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2872
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+int ret, dev_def_dom; +struct device *dev; + +if (!group) +return -EINVAL ; + +mutex_lock (&group->mutex); + +if (group->default_domain != group->domain) { +pr_err_ratelimited ("Group assigned to user level for direct access\n" );
Make this message: "Group not assigned to default domain\n".
Quoted 9 lines
drivers/iommu/iommu.c +9
-0+ ret = -EBUSY ; + goto out; + } + +/* +* iommu group wasn't locked while acquiring device lock in +* iommu_group_store_type(). So, make sure that the device count hasn't +* changed while acquiring device lock. +*
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2891
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+* Changing default domain of an iommu group with two or more devices +* isn't supported because there could be a potential deadlock. Consider +* the following scenario. T1 is trying to acquire device locks of all +* the devices in the group and before it could acquire all of them, +* there could be another thread T2 (from different sub-system and use +* case) that has already acquired some of the device locks and might be +* waiting for T1 to release other device locks. +*/ +if (iommu_group_device_count (group) !=1 ) { +pr_err_ratelimited ("Cannot change default domain of a group with two or more devices\n" );
"Can not change default domain: Group has more than one device\n"
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2902
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+ ret = -EINVAL ; + goto out; + } + +/* Since group has only one device */ +list_for_each_entry (grp_dev, &group->devices, list) + dev = grp_dev->dev; + + prev_dom = group->default_domain; +if (!prev_dom || !prev_dom->ops || !prev_dom->ops->def_domain_type) { +pr_err_ratelimited ("'def_domain_type' call back isn't registered\n" );
This message isn't needed.
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2947
Quoted 9 lines
drivers/iommu/iommu.c +9
-0+ ret =__iommu_attach_device (group->default_domain, dev); +if (ret) + goto free_new_domain; + + group->domain = group->default_domain; + + ret =iommu_create_device_direct_mappings (group, dev); +if (ret) + goto free_new_domain;
You need to create the direct mappings before you attach the device to the new domain. Otherwise there might be a short time-window where RMRR regions are not mapped.
Quoted 28 lines
drivers/iommu/iommu.c +28
-0+static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) +{ +struct group_device *grp_dev; +struct device *dev; +int ret, req_type; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" )) + req_type =0 ; +else +return -EINVAL ; + +/* +* Lock/Unlock the group mutex here before device lock to +* 1. Make sure that the iommu group has only one device (this is a +* prerequisite for step 2) +* 2. Get struct *dev which is needed to lock device +*/
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 3010
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+mutex_lock (&group->mutex); +if (iommu_group_device_count (group) !=1 ) { +mutex_unlock (&group->mutex); +pr_err_ratelimited ("Cannot change default domain of a group with two or more devices\n" ); +return -EINVAL ; + } + +/* Since group has only one device */ +list_for_each_entry (grp_dev, &group->devices, list) + dev = grp_dev->dev;
Please use list_first_entry().
You also need to take a reference with get_device() and then drop the group->mutex.
After device_lock() you need to verify that the device is still in the same group and that the group has still only one device in it. Then you can call down to iommu_change_dev_def_domain() which does not need to take the group-mutex by itself.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Joerg,
Quoted 22 lines
-----Original Message-----
From: Joerg Roedel <[email protected]>
Sent: Tuesday, June 30, 2020 2:16 AM
To: Prakhya, Sai Praneeth <[email protected]>
Cc: [email protected]; Christoph Hellwig <[email protected]>; Raj, Ashok <[email protected]>; Will Deacon <[email protected]>; Lu Baolu <[email protected]>; Mehta, Sohil <[email protected]>; Robin Murphy <[email protected]>; Jacob Pan <[email protected]>
Subject: Re: [PATCH V4 1/3] iommu: Add support to change default domain of an iommu groupQuoted 10 lines · Sai Praneeth Prakhya
drivers/iommu/iommu.c +10
-0+static int iommu_change_dev_def_domain (struct iommu_group *group,int +type) { +struct iommu_domain *prev_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops; +int ret, dev_def_dom; +struct device *dev; + +if (!group) +return -EINVAL ;
Quoted 8 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 6 lines
drivers/iommu/iommu.c +6
-0+ +mutex_lock (&group->mutex); + +if (group->default_domain != group->domain) { +pr_err_ratelimited ("Group assigned to user level for direct +access\n");Make this message: "Group not assigned to default domain\n".
Sure! I will change it
Quoted 22 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 7 lines
drivers/iommu/iommu.c +7
-0+ ret = -EBUSY ; + goto out; + } + +/* +* iommu group wasn't locked while acquiring device lock in +* iommu_group_store_type(). So, make sure that the device count hasn't
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 3 lines
drivers/iommu/iommu.c +3
-0+* changed while acquiring device lock. +* +* Changing default domain of an iommu group with two or more devices
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 9 lines
drivers/iommu/iommu.c +9
-0+* isn't supported because there could be a potential deadlock. Consider +* the following scenario. T1 is trying to acquire device locks of all +* the devices in the group and before it could acquire all of them, +* there could be another thread T2 (from different sub-system and use +* case) that has already acquired some of the device locks and might be +* waiting for T1 to release other device locks. +*/ +if (iommu_group_device_count (group) !=1 ) { + pr_err_ratelimited("Cannot change default domain of a groupwith
Quoted 3 lines
Quoted 1 line
+two or more devices\n");
"Can not change default domain: Group has more than one device\n"
Ok.. make sense. I will change this.
Quoted 7 lines
Quoted 7 lines
drivers/iommu/iommu.c +7
-0+ ret = -EINVAL ; + goto out; + } + +/* Since group has only one device */ +list_for_each_entry (grp_dev, &group->devices, list) + dev = grp_dev->dev;
Quoted 8 lines
Review
Quoted 6 lines
drivers/iommu/iommu.c +3
-0+ + prev_dom = group->default_domain; + if (!prev_dom || !prev_dom->ops || !prev_dom->ops-
def_domain_type) {(quoted diff) +2
-0+ pr_err_ratelimited("'def_domain_type' call back isn't +registered\n");This message isn't needed.
Ok. I will remove it.
Quoted 5 lines
Quoted 5 lines
drivers/iommu/iommu.c +5
-0+ ret =__iommu_attach_device (group->default_domain, dev); +if (ret) + goto free_new_domain; + + group->domain = group->default_domain;
Quoted 8 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2947
Quoted 4 lines
drivers/iommu/iommu.c +4
-0+ + ret =iommu_create_device_direct_mappings (group, dev); +if (ret) + goto free_new_domain;You need to create the direct mappings before you attach the device to the new domain. Otherwise there might be a short time-window where RMRR regions are not mapped.
Ok.. makes sense. I will change this accordingly.
Quoted 32 lines
Quoted 21 lines
drivers/iommu/iommu.c +21
-0+static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) { +struct group_device *grp_dev; +struct device *dev; +int ret, req_type; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" )) + req_type =0 ; +else +return -EINVAL ; +drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+/* +* Lock/Unlock the group mutex here before device lock to +* 1. Make sure that the iommu group has only one device (this is a +* prerequisite for step 2) +* 2. Get struct *dev which is needed to lock device +*/ +mutex_lock (&group->mutex); +if (iommu_group_device_count (group) !=1 ) { +mutex_unlock (&group->mutex); + pr_err_ratelimited("Cannot change default domain of a groupwith two or more devices\n");
Quoted 8 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2898
Quoted 6 lines
drivers/iommu/iommu.c +6
-0+return -EINVAL ; + } + +/* Since group has only one device */ +list_for_each_entry (grp_dev, &group->devices, list) + dev = grp_dev->dev;Please use
list_first_entry().
Ok.
Quoted 2 lines
You also need to take a reference with
get_device()and then drop thegroup->mutex.
Sure! I will change it.
Quoted 2 lines
After
device_lock()you need to verify that the device is still in the same group and that the group has still only one device in it.
Presently, iommu_change_dev_def_domain() checks if the iommu group still has only one device or not. Hence, checking if iommu group has one device or not is done twice, once before taking device_lock() and the other, after taking device_lock().
I agree that the code isn't checking if the iommu group still has the _same_ device or not. One way, I could think of doing it is by storing "dev" temporarily and checking for it. Do you think that's ok? Or would you rather suggest something else?
Quoted 3 lines
Then you can call down to
iommu_change_dev_def_domain()which does not need to take the group- mutex by itself.
The reason for taking iommu_group->mutex in the beginning of iommu_change_dev_def_domain() is that the function
1. Checks if the group is being directly used by user level drivers (i.e. if (group->default_domain != group->domain))
2. Uses iommu_ops
(prev_dom = group->default_domain;
if (!prev_dom || !prev_dom->ops || !prev_dom->ops->def_domain_type))
3. Sets iomu_group->domain to iommu_group->default_domain
I wanted to make sure that iommu_group->domain and iommu_group->default_domain aren't changed by some other thread
while this thread is working on it. So, please let me know if I misunderstood something.
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Joerg,
Replying again because I noticed that I couldn't find this mail in the external iommu mailing list while I was able to find your comments on my patch. Also, could you please answer my two questions below?
Quoted 151 lines
-----Original Message-----
From: iommu <[email protected]> On Behalf Of Prakhya, Sai Praneeth
Sent: Tuesday, June 30, 2020 8:04 PM
To: Joerg Roedel <[email protected]>
Cc: Raj, Ashok <[email protected]>; Will Deacon <[email protected]>; [email protected]; Robin Murphy <[email protected]>; Christoph Hellwig <[email protected]>
Subject: RE: [PATCH V4 1/3] iommu: Add support to change default domain of an iommu groupHi Joerg,
Quoted 24 lines
-----Original Message-----
From: Joerg Roedel <[email protected]>
Sent: Tuesday, June 30, 2020 2:16 AM
To: Prakhya, Sai Praneeth <[email protected]>
Cc: [email protected]; Christoph Hellwig <[email protected]>; Raj, Ashok <[email protected]>; Will Deacon <[email protected]>; Lu Baolu <[email protected]>; Mehta, Sohil <[email protected]>; Robin Murphy <[email protected]>; Jacob Pan <[email protected]>
Subject: Re: [PATCH V4 1/3] iommu: Add support to change default domain of an iommu groupQuoted 11 lines · Sai Praneeth Prakhya
drivers/iommu/iommu.c +11
-0+static int iommu_change_dev_def_domain (struct iommu_group *group, +int +type) { +struct iommu_domain *prev_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops; +int ret, dev_def_dom; +struct device *dev; + +if (!group) +return -EINVAL ;Quoted 8 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 6 lines
drivers/iommu/iommu.c +6
-0+ +mutex_lock (&group->mutex); + +if (group->default_domain != group->domain) { +pr_err_ratelimited ("Group assigned to user level for direct +access\n");Make this message: "Group not assigned to default domain\n".
Sure! I will change it
Quoted 22 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 7 lines
drivers/iommu/iommu.c +7
-0+ ret = -EBUSY ; + goto out; + } + +/* +* iommu group wasn't locked while acquiring device lock in +* iommu_group_store_type(). So, make sure that the device count hasn't
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 3 lines
drivers/iommu/iommu.c +3
-0+* changed while acquiring device lock. +* +* Changing default domain of an iommu group with two or more devices
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 9 lines
drivers/iommu/iommu.c +9
-0+* isn't supported because there could be a potential deadlock. Consider +* the following scenario. T1 is trying to acquire device locks of all +* the devices in the group and before it could acquire all of them, +* there could be another thread T2 (from different sub-system and use +* case) that has already acquired some of the device locks and might be +* waiting for T1 to release other device locks. +*/ +if (iommu_group_device_count (group) !=1 ) { + pr_err_ratelimited("Cannot change default domain of a groupwith
Quoted 3 lines
Quoted 1 line
+two or more devices\n");
"Can not change default domain: Group has more than one device\n"
Ok.. make sense. I will change this.
Quoted 7 lines
Quoted 7 lines
drivers/iommu/iommu.c +7
-0+ ret = -EINVAL ; + goto out; + } + +/* Since group has only one device */ +list_for_each_entry (grp_dev, &group->devices, list) + dev = grp_dev->dev;Quoted 8 lines
Review
Quoted 6 lines
drivers/iommu/iommu.c +3
-0+ + prev_dom = group->default_domain; + if (!prev_dom || !prev_dom->ops || !prev_dom->ops-
def_domain_type) {(quoted diff) +2
-0+ pr_err_ratelimited("'def_domain_type' call back isn't +registered\n");This message isn't needed.
Ok. I will remove it.
Quoted 5 lines
Quoted 5 lines
drivers/iommu/iommu.c +5
-0+ ret =__iommu_attach_device (group->default_domain, dev); +if (ret) + goto free_new_domain; + + group->domain = group->default_domain;Quoted 8 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2947
Quoted 4 lines
drivers/iommu/iommu.c +4
-0+ + ret =iommu_create_device_direct_mappings (group, dev); +if (ret) + goto free_new_domain;You need to create the direct mappings before you attach the device to the new domain. Otherwise there might be a short time-window where RMRR regions are not mapped.
Ok.. makes sense. I will change this accordingly.
Quoted 32 lines
Quoted 21 lines
drivers/iommu/iommu.c +21
-0+static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) { +struct group_device *grp_dev; +struct device *dev; +int ret, req_type; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" )) + req_type =0 ; +else +return -EINVAL ; +drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+/* +* Lock/Unlock the group mutex here before device lock to +* 1. Make sure that the iommu group has only one device (this is a +* prerequisite for step 2) +* 2. Get struct *dev which is needed to lock device +*/ +mutex_lock (&group->mutex); +if (iommu_group_device_count (group) !=1 ) { +mutex_unlock (&group->mutex); + pr_err_ratelimited("Cannot change default domain of a groupwith two or more devices\n");
Quoted 8 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2898
Quoted 6 lines
drivers/iommu/iommu.c +6
-0+return -EINVAL ; + } + +/* Since group has only one device */ +list_for_each_entry (grp_dev, &group->devices, list) + dev = grp_dev->dev;Please use
list_first_entry().
Quoted 9 lines
Ok.
Quoted 2 lines
You also need to take a reference with
get_device()and then drop thegroup->mutex.Sure! I will change it.
Quoted 2 lines
After
device_lock()you need to verify that the device is still in the same group and that the group has still only one device in it.
Q1:
Quoted 5 lines
Presently,
iommu_change_dev_def_domain()checks if the iommu group still has only one device or not. Hence, checking if iommu group has one device or not is done twice, once before takingdevice_lock()and the other, after takingdevice_lock().
Quoted 9 lines
I agree that the code isn't checking if the iommu group still has the _same_ device or not.
One way, I could think of doing it is by storing "dev" temporarily and checking for it.
Do you think that's ok? Or would you rather suggest something else?Quoted 3 lines
Then you can call down to
iommu_change_dev_def_domain()which does not need to take the group- mutex by itself.
Q2:
Quoted 14 lines
The reason for taking
iommu_group->mutexin the beginning ofiommu_change_dev_def_domain()is that the function1. Checks if the group is being directly used by user level drivers (i.e. if (group-
Quoted 1 line
default_domain!=group->domain))2. Uses
iommu_ops
(prev_dom=group->default_domain;
if (!prev_dom || !prev_dom->ops || !prev_dom->ops->def_domain_type))3. Sets
iomu_group->domaintoiommu_group->default_domainI wanted to make sure that
iommu_group->domainand iommu_group-Quoted 1 line
default_domainaren't changed by some other thread while this thread is
Quoted 8 lines
working on it. So, please let me know if I misunderstood something.
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Joerg Roedel
maintainerQuoted 11 lines · Sai Praneeth
Q1:
Quoted 10 lines
Presently,
iommu_change_dev_def_domain()checks if the iommu group still has only one device or not. Hence, checking if iommu group has one device or not is done twice, once before takingdevice_lock()and the other, after takingdevice_lock().I agree that the code isn't checking if the iommu group still has the _same_ device or not.
One way, I could think of doing it is by storing "dev" temporarily and checking for it.
Do you think that's ok? Or would you rather suggest something else?
That sounds reasonable, get the device from the group, lock it, take group->mutex, and check whether the same device is still alone in the group.
Quoted 16 lines
Q2:
Quoted 15 lines
The reason for taking
iommu_group->mutexin the beginning ofiommu_change_dev_def_domain()is that the function1. Checks if the group is being directly used by user level drivers (i.e. if (group-
Quoted 1 line
default_domain!=group->domain))2. Uses
iommu_ops
(prev_dom=group->default_domain;
if (!prev_dom || !prev_dom->ops || !prev_dom->ops->def_domain_type))3. Sets
iomu_group->domaintoiommu_group->default_domainI wanted to make sure that
iommu_group->domainand iommu_group-Quoted 1 line
default_domainaren't changed by some other thread while this thread isworking on it. So, please let me know if I misunderstood something.
This looks correct as well.
Regards,
Joerg
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Joerg,
Thanks for the reply. I will spin another version of the patch addressing your comments.
Quoted 23 lines
-----Original Message-----
From: Joerg Roedel <[email protected]>
Sent: Wednesday, July 22, 2020 6:53 AM
To: Prakhya, Sai Praneeth <[email protected]>
Cc: Raj, Ashok <[email protected]>; Will Deacon <[email protected]>; [email protected]; Robin Murphy <[email protected]>; Christoph Hellwig <[email protected]>
Subject: Re: [PATCH V4 1/3] iommu: Add support to change default domain of an iommu groupQuoted 11 lines · Sai Praneeth
Q1:
Quoted 10 lines
Presently,
iommu_change_dev_def_domain()checks if the iommu group still has only one device or not. Hence, checking if iommu group has one device or not is done twice, once before takingdevice_lock()and the other, after takingdevice_lock().I agree that the code isn't checking if the iommu group still has the _same_ device or not.
One way, I could think of doing it is by storing "dev" temporarily and checking for it.
Do you think that's ok? Or would you rather suggest something else?
Quoted 3 lines
That sounds reasonable, get the device from the group, lock it, take
group->mutex, and check whether the same device is still alone in the group.
Sounds good! I will implement this.
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v4 2/3 iommu: Take lock before reading iommu group default domain type
In reply to Sai Praneeth Prakhya"/sys/kernel/iommu_groups/<grp_id>/type" file could be read to find out the default domain type of an iommu group. The default domain of an iommu group doesn't change after booting and hence could be read directly. But, after addding support to dynamically change iommu group default domain, the above assumption no longer stays valid.
iommu group default domain type could be changed at any time by writing to "/sys/kernel/iommu_groups/<grp_id>/type". So, take group mutex before reading iommu group default domain type so that the user wouldn't see stale values or iommu_group_show_type() doesn't try to derefernce stale pointers.
--- drivers/iommu/iommu.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/iommu/iommu.c +2 -0
@@ -501,6 +501,7 @@ static ssize_t iommu_group_show_type(struct iommu_group *group, {char *type ="unknown\n" ; +mutex_lock (&group->mutex);if (group->default_domain) {switch (group->default_domain->type) {case IOMMU_DOMAIN_BLOCKED :
@@ -517,6 +518,7 @@ static ssize_t iommu_group_show_type(struct iommu_group *group,break ; } } +mutex_unlock (&group->mutex);strcpy (buf, type);return strlen (type);
Signature · 6 lines
2.19.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V4 2/3] iommu: Take lock before reading iommu group default domain type
Quoted 11 lines · Sai Praneeth Prakhya
"/sys/kernel/iommu_groups/<grp_id>/type" file could be read to find out the default domain type of an iommu group. The default domain of an iommu group doesn't change after booting and hence could be read directly. But, after addding support to dynamically change iommu group default domain, the above assumption no longer stays valid.
iommu group default domain type could be changed at any time by writing to "/sys/kernel/iommu_groups/<grp_id>/type". So, take group mutex before reading iommu group default domain type so that the user wouldn't see stale values or
iommu_group_show_type()doesn't try to derefernce stale pointers.
Quoted 5 lines
Cc Christoph Hellwig <[email protected]>Cc Joerg Roedel <[email protected]>Cc Ashok Raj <[email protected]>Cc Will Deacon <[email protected]>Cc Lu Baolu <[email protected]>
Best regards,
baolu
Quoted 22 lines
Cc Sohil Mehta <[email protected]>Cc Robin Murphy <[email protected]>Cc Jacob Pan <[email protected]>Signed-off-by Sai Praneeth Prakhya <[email protected]>--- drivers/iommu/iommu.c | 2 ++ 1 file changed, 2 insertions(+)drivers/iommu/iommu.c +1
-0@@ -501,6 +501,7 @@ static ssize_t iommu_group_show_type(struct iommu_group *group, {char *type ="unknown\n" ; +mutex_lock (&group->mutex);if (group->default_domain) {switch (group->default_domain->type) {case IOMMU_DOMAIN_BLOCKED :
drivers/iommu/iommu.c static ssize_t iommu_group_show_type(struct iommu_group *group, · line 524
Quoted 7 lines
drivers/iommu/iommu.c +1
-0@@ -517,6 +518,7 @@ static ssize_t iommu_group_show_type(struct iommu_group *group,break ; } } +mutex_unlock (&group->mutex);strcpy (buf, type);return strlen (type);
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v4 3/3 iommu: Document usage of "/sys/kernel/iommu_groups/<grp_id>/type" file
In reply to Sai Praneeth PrakhyaThe default domain type of an iommu group can be changed by writing to "/sys/kernel/iommu_groups/<grp_id>/type" file. Hence, document it's usage and more importantly spell out its limitations.
--- .../ABI/testing/sysfs-kernel-iommu_groups | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+)
Documentation/ABI/testing/sysfs-kernel-iommu_groups +30 -0
@@ -33,3 +33,33 @@ Description: In case an RMRR is used only by graphics or USB devices it is now exposed as "direct-relaxable" instead of "direct". In device assignment use case, for instance, those RMRR are considered to be relaxable and safe. + +What: /sys/kernel/iommu_groups/<grp_id>/type +Date: June 2020 +KernelVersion: v5.8 +Contact: Sai Praneeth Prakhya <[email protected]> +Description: Let the user know the type of default domain in use by iommu + for this group. A privileged user could request kernel to change + the group type by writing to this file. Presently, only three + types are supported + 1. DMA: All the DMA transactions from the device in this group + are translated by the iommu. + 2. identity: All the DMA transactions from the device in this + group are *not* translated by the iommu. + 3. auto: Change to the type the device was booted with. When the + user reads the file he would never see "auto". This is + just a write only value. + Note: + ----- + A group type could be modified only when + 1. The group has *only* one device + 2. The device in the group is not bound to any device driver. + So, the user must first unbind the appropriate driver and + then change the default domain type. + Caution: + -------- + Unbinding a device driver will take away the driver's control + over the device and if done on devices that host root file + system could lead to catastrophic effects (the user might + need to reboot the machine to get it to normal state). So, it's + expected that the user understands what he is doing.
Signature · 6 lines
2.19.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V4 3/3] iommu: Document usage of "/sys/kernel/iommu_groups/<grp_id>/type" file
Quoted 9 lines · Sai Praneeth Prakhya
The default domain type of an iommu group can be changed by writing to "/sys/kernel/iommu_groups/<grp_id>/type" file. Hence, document it's usage and more importantly spell out its limitations.
Cc Christoph Hellwig <[email protected]>Cc Joerg Roedel <[email protected]>Cc Ashok Raj <[email protected]>Cc Will Deacon <[email protected]>Cc Lu Baolu <[email protected]>
Best regards,
baolu
Quoted 37 lines
Cc Sohil Mehta <[email protected]>Cc Robin Murphy <[email protected]>Cc Jacob Pan <[email protected]>Signed-off-by Sai Praneeth Prakhya <[email protected]>--- .../ABI/testing/sysfs-kernel-iommu_groups | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+)Documentation/ABI/testing/sysfs-kernel-iommu_groups +20
-0@@ -33,3 +33,33 @@ Description: In case an RMRR is used only by graphics or USB devices it is now exposed as "direct-relaxable" instead of "direct". In device assignment use case, for instance, those RMRR are considered to be relaxable and safe. + +What: /sys/kernel/iommu_groups/<grp_id>/type +Date: June 2020 +KernelVersion: v5.8 +Contact: Sai Praneeth Prakhya <[email protected]> +Description: Let the user know the type of default domain in use by iommu + for this group. A privileged user could request kernel to change + the group type by writing to this file. Presently, only three + types are supported + 1. DMA: All the DMA transactions from the device in this group + are translated by the iommu. + 2. identity: All the DMA transactions from the device in this + group are *not* translated by the iommu. + 3. auto: Change to the type the device was booted with. When the + user reads the file he would never see "auto". This is + just a write only value. + Note: + ----- + A group type could be modified only when + 1. The group has *only* one device
Documentation/ABI/testing/sysfs-kernel-iommu_groups Description: In case an RMRR is used only by graphics or USB devices · line 65
Quoted 10 lines
Documentation/ABI/testing/sysfs-kernel-iommu_groups +10
-0+ 2. The device in the group is not bound to any device driver. + So, the user must first unbind the appropriate driver and + then change the default domain type. + Caution: + -------- + Unbinding a device driver will take away the driver's control + over the device and if done on devices that host root file + system could lead to catastrophic effects (the user might + need to reboot the machine to get it to normal state). So, it's + expected that the user understands what he is doing.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Joerg,
Quoted 36 lines
-----Original Message-----
From: Prakhya, Sai Praneeth <[email protected]>
Sent: Thursday, June 4, 2020 6:32 PM
To: [email protected]
Cc: Prakhya, Sai Praneeth <[email protected]>; Christoph Hellwig <[email protected]>; Joerg Roedel <[email protected]>; Raj, Ashok <[email protected]>; Will Deacon <[email protected]>; Lu Baolu <[email protected]>; Mehta, Sohil <[email protected]>; Robin Murphy <[email protected]>; Jacob Pan <[email protected]>
Subject: [PATCH V4 0/3] iommu: Add support to change default domain of an iommu groupPresently, the default domain of an iommu group is allocated during boot time and it cannot be changed later. So, the device would typically be either in identity (
pass_through) mode or the device would be in DMA mode as long as the system is up and running. There is no way to change the default domain type dynamically i.e. after booting, a device cannot switch between identity mode and DMA mode.Assume a use case wherein the privileged user would want to use the device in pass-through mode when the device is used for host so that it would be high performing. Presently, this is not supported. Hence add support to change the default domain of an iommu group dynamically.
Support this by writing to a sysfs file, namely
"/sys/kernel/iommu_groups/<grp_id>/type".Testing:
--------
Tested by dynamically changing storage device (nvme) from 1. identity mode to DMA and making sure file transfer works 2. DMA mode to identity mode and making sure file transfer works Tested only for intel_iommu/vt-d. Would appreciate if someone could test on AMD and ARM based machines.Based on iommu maintainer's 'next' branch.
Quoted 13 lines
Changes from V3:
----------------
1. Made changes to commit message as suggested by Baolu.
2. Don't pass "prev_dom" and "dev" as parameters toiommu_change_dev_def_domain(). Instead get them from group.
3. Sanitize the logic to validate user default domain type request. The logic remains same but is implmented differently.
4. Push lot of error checking intoiommu_change_dev_def_domain()fromiommu_group_store_type().
5.iommu_change_dev_def_domain()takes/releases group mutex as needed. So, it
shouldn't be called holding a group mutex.
6. Usepr_err_ratelimited()instead ofpr_err()to avoid DOS attack.
Could you please review this patch set and let me know if you have any comments?
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu