iommu: Add support to change default domain of a group
v2 cover iommu: Add support to change default domain of a group
Presently, the default domain of a group is allocated during boot time and it cannot be changed later. So, the device would typically be either in identity (pass_through) mode or the device would be in DMA mode as long as the system is up and running. There is no way to change the default domain type dynamically i.e. after booting, a device cannot switch between identity mode and DMA mode.
Assume a use case wherein the privileged user would want to use the device in pass-through mode when the device is used for host so that it would be high performing. Presently, this is not supported. Hence add support to change the default domain of a group dynamically.
Support this through a sysfs file, namely "/sys/kernel/iommu_groups/<grp_id>/type".
Hi Joerg,
Sorry! for _huge_ delay in posting a V2 of this patch set. Was stuck with some internal PoC work. Will be consistent from now.
Changes from V1:
----------------
1. V1 patch set wasn't updating dma_ops for some vendors (Eg: AMD), hence, change the logic of updating default domain as below (because adding a device to iommu_group automatically updates dma_ops)
a. Allocate a new domain
b. For every device in the group
i. Remove the device from the group
ii. Add the device back to the group
c. Free previous domain
2. Drop 1st patch of V1 (iommu/vt-d: Modify device_def_domain_type() to use at runtime) because "iommu=pt" has no effect on this function anymore.
3. Added a patch to take/release lock while reading iommu_group->default_domain->type because it can be changed any time by user.
4. Before changing default domain type of a group, check if the group is directly assigned for user level access. If so, abort.
5. Sanitize return path (using ternary operator) in iommu_group_store_type()
6. Split 2nd patch of V1 (iommu: Add device_def_domain_type() call back function to iommu_ops) into two patches such that iommu generic changes are now in 1st patch of V2 and vt-d specific changes are in 2nd patch of V2.
7. Rename device_def_domain_type() to dev_def_domain_type()
8. Remove example from documentation
9. Change the value written to file "/sys/kernel/iommu_groups/<grp_id>/type" from "dma" to "DMA".
Changes from RFC:
-----------------
1. Added support for "auto" type, so that kernel selects one among identity or dma mode.
2. Use "system_state" in device_def_domain_type() instead of an argument.
Testing:
--------
Tested by dynamically changing storage device (nvme) from
1. identity mode to dma and making sure file transfer works
2. dma mode to identity mode and making sure file transfer works Tested only for intel_iommu/vt-d.
Based on iommu maintainer's 'next' branch.
Sai Praneeth Prakhya (5):
iommu: Add dev_def_domain_type() call back function to iommu_ops iommu/vt-d: Rename device_def_domain_type() to
intel_iommu_dev_def_domain_type()
iommu: Add support to change default domain of an iommu_group
iommu: Take lock before reading iommu_group default domain type
iommu: Document usage of "/sys/kernel/iommu_groups/<grp_id>/type" file
.../ABI/testing/sysfs-kernel-iommu_groups | 29 +++ drivers/iommu/intel-iommu.c | 8 +- drivers/iommu/iommu.c | 229 ++++++++++++++++++++- include/linux/iommu.h | 3 + 4 files changed, 265 insertions(+), 4 deletions(-)
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v2 1/5 iommu: Add dev_def_domain_type() call back function to iommu_ops
When user requests kernel to change the default domain type of a group through sysfs, kernel has to make sure that it's ok to change the domain type of every device in the group to the requested domain (every device may not support both the domain types i.e. DMA and identity). Hence, add a call back function that could be implemented per architecture that performs the above check.
--- include/linux/iommu.h | 3 +++ 1 file changed, 3 insertions(+)
include/linux/iommu.h +3 -0
@@ -248,6 +248,7 @@ struct iommu_iotlb_gather {* @cache_invalidate: invalidate translation caches * @sva_bind_gpasid: bind guest pasid and mm * @sva_unbind_gpasid: unbind guest pasid and mm +* @dev_def_domain_type: Return the required default domain type for a device * @pgsize_bitmap: bitmap of all possible supported page sizes * @owner: Driver module providing these ops */
@@ -318,6 +319,8 @@ struct iommu_ops {int (*sva_unbind_gpasid)(struct device *dev,int pasid); +int (*dev_def_domain_type)(struct device *dev); +unsigned long pgsize_bitmap;struct module *owner; };
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v2 2/5 iommu/vt-d: Rename device_def_domain_type() to intel_iommu_dev_def_domain_type()
In reply to Sai Praneeth PrakhyaThe functionality needed for iommu_ops->dev_def_domain_type() is already provided by device_def_domain_type() in intel_iommu.c. But, every call back function in intel_iommu_ops starts with intel_iommu prefix, hence rename device_def_domain_type() to intel_iommu_dev_def_domain_type() so that it follows the same semantics.
--- drivers/iommu/intel-iommu.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/iommu/intel-iommu.c +5 -3
@@ -3034,7 +3034,7 @@ static bool device_is_rmrr_locked(struct device *dev)* - IOMMU_DOMAIN_IDENTITY: device requires an identical mapping domain * - 0: both identity and dynamic domains work for this device */ -static int device_def_domain_type (struct device *dev) +static int intel_iommu_dev_def_domain_type (struct device *dev) {if (dev_is_pci (dev)) {struct pci_dev *pdev =to_pci_dev (dev);
@@ -5796,7 +5796,8 @@ static int intel_iommu_add_device(struct device *dev) domain =iommu_get_domain_for_dev (dev); dmar_domain =to_dmar_domain (domain);if (domain->type ==IOMMU_DOMAIN_DMA ) { -if (device_def_domain_type (dev) ==IOMMU_DOMAIN_IDENTITY ) { +if (intel_iommu_dev_def_domain_type (dev) == +IOMMU_DOMAIN_IDENTITY ) { ret =iommu_request_dm_for_dev (dev);if (ret) {dmar_remove_one_dev_info (dev);
@@ -5807,7 +5808,7 @@ static int intel_iommu_add_device(struct device *dev) } } }else { -if (device_def_domain_type (dev) ==IOMMU_DOMAIN_DMA ) { +if (intel_iommu_dev_def_domain_type (dev) ==IOMMU_DOMAIN_DMA ) { ret =iommu_request_dma_domain_for_dev (dev);if (ret) {dmar_remove_one_dev_info (dev);
@@ -6194,6 +6195,7 @@ const struct iommu_ops intel_iommu_ops = { .dev_enable_feat = intel_iommu_dev_enable_feat, .dev_disable_feat = intel_iommu_dev_disable_feat, .is_attach_deferred = intel_iommu_is_attach_deferred, + .dev_def_domain_type = intel_iommu_dev_def_domain_type, .pgsize_bitmap = INTEL_IOMMU_PGSIZES, };
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v2 3/5 iommu: Add support to change default domain of an iommu_group
In reply to Sai Praneeth PrakhyaPresently, the default domain of an iommu_group is allocated during boot time (i.e. when a device is being added to a group) and it cannot be changed later. So, the device would typically be either in identity (also known as pass_through) mode (controlled by "iommu=pt" kernel command line argument) or the device would be in DMA mode as long as the machine is up and running. There is no way to change the default domain type dynamically i.e. after booting, a device cannot switch between identity mode and DMA mode.
But, assume a use case wherein the user trusts the device and also believes that the OS is secure enough and hence wants *only* this device to bypass IOMMU (so that it could be high performing) whereas all the other devices to go through IOMMU (so that the system is protected). Presently, this use case is not supported. Hence it will be helpful if there is some way to change the default domain of a B:D.F dynamically. Since, linux iommu subsystem prefers to deal at iommu_group level instead of B:D.F level, it might be helpful if there is some way to change the default domain of a *iommu_group* dynamically. Hence, add such support.
A privileged user could request the kernel to change the default domain type of a iommu_group by writing to "/sys/kernel/iommu_groups/<grp_id>/type"
file. Presently, only three values are supported
1. identity: all the DMA transactions from the devices in this group are *not* translated by the iommu
2. DMA: all the DMA transactions from the devices in this group are translated by the iommu
3. auto: kernel decides one among DMA/identity
Also please note that a group type could be modified only when *all* the devices in the group are not binded to any device driver. Please see "Documentation/ABI/testing/sysfs-kernel-iommu_groups" for more information.
--- drivers/iommu/iommu.c | 227 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 226 insertions(+), 1 deletion(-)
drivers/iommu/iommu.c +226 -1
@@ -225,6 +225,8 @@ static int __iommu_attach_group(struct iommu_domain *domain, struct iommu_group *group);static void __iommu_detach_group (struct iommu_domain *domain,struct iommu_group *group); +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count);static int __init iommu_set_def_domain_type (char *str) {
@@ -448,7 +450,8 @@ static IOMMU_GROUP_ATTR(name, S_IRUGO, iommu_group_show_name, NULL); static IOMMU_GROUP_ATTR(reserved_regions, 0444, iommu_group_show_resv_regions, NULL); -static IOMMU_GROUP_ATTR(type, 0444, iommu_group_show_type, NULL); +static IOMMU_GROUP_ATTR(type, 0644, iommu_group_show_type, + iommu_group_store_type);static void iommu_group_release (struct kobject *kobj) {
@@ -2654,3 +2657,225 @@ int iommu_sva_get_pasid(struct iommu_sva *handle)return ops->sva_get_pasid (handle); }EXPORT_SYMBOL_GPL (iommu_sva_get_pasid); + +/* +* Changes the default domain of a group +* +* @group: The group for which the default domain should be changed +* @prv_dom: The previous domain that is being switched from +* @type: The type of the new default domain that gets associated with the group +* +* Returns 0 on success and error code on failure +*/ +static int iommu_group_change_def_domain (struct iommu_group *group, +struct iommu_domain *prv_dom, +int type) +{ +struct group_device *grp_dev, *temp; +struct iommu_domain *new_domain; +int ret; + +/* +* iommu_domain_alloc() takes "struct bus_type" as an argument which is +* a member in "struct device". Changing a group's default domain type +* deals at iommu_group level rather than device level and hence there +* is no straight forward way to get "bus_type" of an iommu_group that +* could be passed to iommu_domain_alloc(). So, instead of directly +* calling iommu_domain_alloc(), use iommu_ops from previous default +* domain. +*/ +if (!prv_dom || !prv_dom->ops || !prv_dom->ops->domain_alloc || !type) +return -EINVAL ; + +/* Allocate a new domain of requested type */ + new_domain = prv_dom->ops->domain_alloc (type); +if (!new_domain) { +pr_err ("Unable to allocate memory for the new domain\n" ); +return -ENOMEM ; + } + + new_domain->type = type; + new_domain->ops = prv_dom->ops; + new_domain->pgsize_bitmap = prv_dom->pgsize_bitmap; + +/* +* Set these upfront here because iommu_group_add_device() and +* iommu_group_create_direct_mappings() needs these to be set +*/ +mutex_lock (&group->mutex); + group->default_domain = new_domain; + group->domain = new_domain; +mutex_unlock (&group->mutex); + +iommu_group_ref_get (group); + +list_for_each_entry_safe (grp_dev, temp, &group->devices, list) { +struct device *dev; + + dev = grp_dev->dev; +iommu_release_device (dev); + + ret =iommu_group_add_device (group, dev); +if (ret) +dev_err (dev,"Failed to add to iommu group %d: %d\n" , + group->id, ret); + + ret = prv_dom->ops->add_device (dev); +if (ret) +dev_err (dev,"Error adding to iommu: %d\n" , ret); + } + +iommu_group_put (group); +iommu_domain_free (prv_dom); +return 0 ; +} + +static int is_driver_bound (struct device *dev,void *not_used) +{ +int ret =0 ; + +device_lock (dev); +if (device_is_bound (dev)) + ret =1 ; +device_unlock (dev); +return ret; +} + +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) +{ +int ret =0 , req_type =0 , req_auto =0 ; +struct iommu_domain *prv_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops; +struct device *dev; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" )) + req_auto =1 ; +else +return -EINVAL ; + +/* +* Check if any device in the group still has a driver binded to it. +* This might race with device driver probing code and unfortunately +* there is no clean way out of that either, locking all devices in the +* group and then do the re-attach will introduce a lock-inversion with +* group->mutex - Joerg. +*/ +if (iommu_group_for_each_dev (group,NULL , is_driver_bound)) { +pr_err ("Active drivers exist for devices in the group\n" ); +return -EBUSY ; + } + +mutex_lock (&group->mutex); + prv_dom = group->default_domain; +if (!prv_dom || !prv_dom->ops || !prv_dom->ops->dev_def_domain_type) { +pr_err ("'dev_def_domain_type' call back isn't registered\n" ); + ret = -EPERM ; + goto out; + } + +/* +* Check if any user level driver (that doesn't use kernel driver like +* VFIO) is directly using the group. VFIO use case is detected by +* is_driver_bound() check above +*/ +if (prv_dom != group->domain) { +pr_err ("Group assigned to user level for direct access\n" ); + ret = -EBUSY ; + goto out; + } + +/* +* If user has requested "auto", kernel has to decide the default domain +* type of a group. Hence, find out individual preferences of a device. +*/ + ops = prv_dom->ops; +if (req_auto) { +int dma_devs =0 , idt_devs =0 , dev_def_dom; + +list_for_each_entry (grp_dev, &group->devices, list) { + dev = grp_dev->dev; + dev_def_dom = ops->dev_def_domain_type (dev); +if (dev_def_dom ==IOMMU_DOMAIN_IDENTITY ) + idt_devs++; +if (dev_def_dom ==IOMMU_DOMAIN_DMA ) + dma_devs++; + } + +/* +* Default domain type of a group is undefined if some devices +* in the group should be in dma domain while other devices +* should be in identity domain +*/ +if (idt_devs && dma_devs) { +pr_err ("Some devices need identity domain while other need dma domain\n" ); + ret = -EPERM ; + goto out; + } + +/* +* Default domain type of a group is identity if +* 1. All the devices in the group need to be in identity domain +* 2. Some devices should be in identity domain while other +* devices could be in either of dma or identity domain +*/ +if (idt_devs && !dma_devs) + req_type =IOMMU_DOMAIN_IDENTITY ; + +/* +* Default domain type of a group is dma if +* 1. All the devices in the group need to be in dma domain +* 2. Some devices should be in dma domain while other devices +* could be in either of dma or identity domain +* 3. All the devices could be in either of the domains (namely +* dma and identity) +*/ +if (!idt_devs) + req_type =IOMMU_DOMAIN_DMA ; + } + +/* +* Switch to a new domain only if the requested domain type is different +* from the existing default domain type +*/ +if (prv_dom->type == req_type) + goto out; + +/* +* Every device may not support both the domain types (namely DMA and +* identity), so check if it's ok to change domain type of every device +* in the group to the requested domain. This check is not required if +* user has requested "auto" because it's already done above implicitly. +*/ +if (!req_auto) { +list_for_each_entry (grp_dev, &group->devices, list) { +int allowed_types; + + dev = grp_dev->dev; + allowed_types = ops->dev_def_domain_type (dev); +if (allowed_types && allowed_types != req_type) { +dev_err (dev,"Cannot be in %s domain\n" , buf); + ret = -EPERM ; + goto out; + } + } + } + +mutex_unlock (&group->mutex); + ret =iommu_group_change_def_domain (group, prv_dom, req_type); +return ret ?: count; +out: +mutex_unlock (&group->mutex); +return ret ?: count; +}
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v2 4/5 iommu: Take lock before reading iommu_group default domain type
In reply to Sai Praneeth PrakhyaSince user could change default domain type of an iommu group through sysfs, reading the default domain type now requires taking the lock first.
--- drivers/iommu/iommu.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/iommu/iommu.c +2 -0
@@ -424,6 +424,7 @@ static ssize_t iommu_group_show_type(struct iommu_group *group, {char *type ="unknown\n" ; +mutex_lock (&group->mutex);if (group->default_domain) {switch (group->default_domain->type) {case IOMMU_DOMAIN_BLOCKED :
@@ -440,6 +441,7 @@ static ssize_t iommu_group_show_type(struct iommu_group *group,break ; } } +mutex_unlock (&group->mutex);strcpy (buf, type);return strlen (type);
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v2 5/5 iommu: Document usage of "/sys/kernel/iommu_groups/<grp_id>/type" file
In reply to Sai Praneeth PrakhyaThe default domain type of an iommu group can be changed using file "/sys/kernel/iommu_groups/<grp_id>/type". Hence, document it's usage and more importantly spell out it's limitations.
--- .../ABI/testing/sysfs-kernel-iommu_groups | 29 ++++++++++++++++++++++ 1 file changed, 29 insertions(+)
Documentation/ABI/testing/sysfs-kernel-iommu_groups +29 -0
@@ -33,3 +33,32 @@ Description: In case an RMRR is used only by graphics or USB devices it is now exposed as "direct-relaxable" instead of "direct". In device assignment use case, for instance, those RMRR are considered to be relaxable and safe. + +What: /sys/kernel/iommu_groups/<grp_id>/type +Date: February 2020 +KernelVersion: v5.6 +Contact: Sai Praneeth Prakhya <[email protected]> +Description: Lets the user know the type of default domain in use by iommu + for this group. A privileged user could request kernel to change + the group type by writing to this file. Presently, only three + types are supported + 1. DMA: All the DMA transactions from the devices in this group + are translated by the iommu. + 2. identity: All the DMA transactions from the devices in this + group are *not* translated by the iommu. + 3. auto: Kernel decides one among DMA/identity mode and hence + when the user reads the file he would never see "auto". + This is just a write only value. + Note: + ----- + A group type could be modified only when *all* the devices in + the group are not binded to any device driver. So, the user has + to first unbind the appropriate drivers and then change the + default domain type. + Caution: + -------- + Unbinding a device driver will take away the drivers control + over the device and if done on devices that host root file + system could lead to catastrophic effects (the user might + need to reboot the machine to get it to normal state). So, it's + expected that the user understands what he is doing.
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 1/5] iommu: Add dev_def_domain_type() call back function to iommu_ops
In reply to Sai Praneeth PrakhyaHi Sai,
Quoted 25 lines · Sai Praneeth Prakhya
When user requests kernel to change the default domain type of a group through sysfs, kernel has to make sure that it's ok to change the domain type of every device in the group to the requested domain (every device may not support both the domain types i.e. DMA and identity). Hence, add a call back function that could be implemented per architecture that performs the above check.
Cc Christoph Hellwig <[email protected]>Cc Joerg Roedel <[email protected]>Cc Ashok Raj <[email protected]>Cc Will Deacon <[email protected]>Cc Lu Baolu <[email protected]>Cc Sohil Mehta <[email protected]>Cc Robin Murphy <[email protected]>Cc Jacob Pan <[email protected]>Signed-off-by Sai Praneeth Prakhya <[email protected]>--- include/linux/iommu.h | 3 +++ 1 file changed, 3 insertions(+)include/linux/iommu.h +0
-0
include/linux/iommu.h struct iommu_iotlb_gather { · line 251
Quoted 4 lines
include/linux/iommu.h +1
-0@@ -248,6 +248,7 @@ struct iommu_iotlb_gather {* @cache_invalidate: invalidate translation caches * @sva_bind_gpasid: bind guest pasid and mm * @sva_unbind_gpasid: unbind guest pasid and mm +* @dev_def_domain_type: Return the required default domain type for a device
Can you please define the return value of this callback?
Best regards,
baolu
include/linux/iommu.h struct iommu_ops { · line 326
Quoted 13 lines
include/linux/iommu.h +2
-0* @pgsize_bitmap: bitmap of all possible supported page sizes * @owner: Driver module providing these ops */ @@ -318,6 +319,8 @@ struct iommu_ops {int (*sva_unbind_gpasid)(struct device *dev,int pasid); +int (*dev_def_domain_type)(struct device *dev); +unsigned long pgsize_bitmap;struct module *owner; };
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH V2 1/5] iommu: Add dev_def_domain_type() call back function to iommu_ops
Quoted 22 lines · Sai Praneeth Prakhya
Quoted 20 lines
When user requests kernel to change the default domain type of a group through sysfs, kernel has to make sure that it's ok to change the domain type of every device in the group to the requested domain (every device may not support both the domain types i.e. DMA and identity). Hence, add a call back function that could be implemented per architecture that performs the above check.
Cc Christoph Hellwig <[email protected]>Cc Joerg Roedel <[email protected]>Cc Ashok Raj <[email protected]>Cc Will Deacon <[email protected]>Cc Lu Baolu <[email protected]>Cc Sohil Mehta <[email protected]>Cc Robin Murphy <[email protected]>Cc Jacob Pan <[email protected]>Signed-off-by Sai Praneeth Prakhya <[email protected]>--- include/linux/iommu.h | 3 +++ 1 file changed, 3 insertions(+)
Quoted 12 lines
include/linux/iommu.h struct iommu_iotlb_gather { · line 252
Quoted 10 lines
include/linux/iommu.h index renamed from include/linux/iommu.h +0
-0
d1b5f4d98569..3f4aaad0aeb7100644include/linux/iommu.h +2
-0@@ -248,6 +248,7 @@ struct iommu_iotlb_gather {* @cache_invalidate: invalidate translation caches * @sva_bind_gpasid: bind guest pasid and mm * @sva_unbind_gpasid: unbind guest pasid and mm +* @dev_def_domain_type: Return the required default domain type for +a device Can you please define the return value of this callback?
Sure! I will add it in the next version
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Joerg,
Quoted 35 lines
Presently, the default domain of a group is allocated during boot time and it cannot be changed later. So, the device would typically be either in identity (
pass_through) mode or the device would be in DMA mode as long as the system is up and running. There is no way to change the default domain type dynamically i.e. after booting, a device cannot switch between identity mode and DMA mode.Assume a use case wherein the privileged user would want to use the device in pass-through mode when the device is used for host so that it would be high performing. Presently, this is not supported. Hence add support to change the default domain of a group dynamically.
Support this through a sysfs file, namely
"/sys/kernel/iommu_groups/<grp_id>/type".Hi Joerg,
Sorry! for _huge_ delay in posting a V2 of this patch set. Was stuck with some internal PoC work. Will be consistent from now.Changes from V1:
----------------
1. V1 patch set wasn't updatingdma_opsfor some vendors (Eg: AMD), hence, change the logic of updating default domain as below (because adding a
device
toiommu_groupautomatically updatesdma_ops)
a. Allocate a new domain
b. For every device in the group
i. Remove the device from the group
ii. Add the device back to the group
c. Free previous domain
2. Drop 1st patch of V1 (iommu/vt-d: Modifydevice_def_domain_type()to use at
runtime) because "iommu=pt" has no effect on this function anymore.
3. Added a patch to take/release lock while reading iommu_group-Quoted 1 line
default_domain->type
Quoted 12 lines
because it can be changed any time by user.
4. Before changing default domain type of a group, check if the group is directly assigned for user level access. If so, abort.
5. Sanitize return path (using ternary operator) iniommu_group_store_type()6. Split 2nd patch of V1 (iommu: Adddevice_def_domain_type()call back function toiommu_ops) into two patches such that iommu generic changes are now in
1st
patch of V2 and vt-d specific changes are in 2nd patch of V2.
7. Renamedevice_def_domain_type()todev_def_domain_type()8. Remove example from documentation 9. Change the value written to file "/sys/kernel/iommu_groups/<grp_id>/type"
from "dma" to "DMA".
Just a gentle reminder, could you please review this patch set and let me know your comments?
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 2/5] iommu/vt-d: Rename device_def_domain_type() to intel_iommu_dev_def_domain_type()
In reply to Sai Praneeth PrakhyaHi,
Quoted 5 lines · Sai Praneeth Prakhya
The functionality needed for
iommu_ops->dev_def_domain_type()is already provided bydevice_def_domain_type()inintel_iommu.c. But, every call back function inintel_iommu_opsstarts withintel_iommuprefix, hence renamedevice_def_domain_type()tointel_iommu_dev_def_domain_type()so that it follows the same semantics.
How about keep device_def_domain_type() and call it in the new intel_iommu_dev_def_domain_type()?
Best regards,
baolu
Quoted 48 lines
Cc Christoph Hellwig <[email protected]>Cc Joerg Roedel <[email protected]>Cc Ashok Raj <[email protected]>Cc Will Deacon <[email protected]>Cc Lu Baolu <[email protected]>Cc Sohil Mehta <[email protected]>Cc Robin Murphy <[email protected]>Cc Jacob Pan <[email protected]>Signed-off-by Sai Praneeth Prakhya <[email protected]>--- drivers/iommu/intel-iommu.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)drivers/iommu/intel-iommu.c +4
-3@@ -3034,7 +3034,7 @@ static bool device_is_rmrr_locked(struct device *dev)* - IOMMU_DOMAIN_IDENTITY: device requires an identical mapping domain * - 0: both identity and dynamic domains work for this device */ -static int device_def_domain_type (struct device *dev) +static int intel_iommu_dev_def_domain_type (struct device *dev) {if (dev_is_pci (dev)) {struct pci_dev *pdev =to_pci_dev (dev);@@ -5796,7 +5796,8 @@ static int intel_iommu_add_device(struct device *dev) domain =iommu_get_domain_for_dev (dev); dmar_domain =to_dmar_domain (domain);if (domain->type ==IOMMU_DOMAIN_DMA ) { -if (device_def_domain_type (dev) ==IOMMU_DOMAIN_IDENTITY ) { +if (intel_iommu_dev_def_domain_type (dev) == +IOMMU_DOMAIN_IDENTITY ) { ret =iommu_request_dm_for_dev (dev);if (ret) {dmar_remove_one_dev_info (dev);@@ -5807,7 +5808,7 @@ static int intel_iommu_add_device(struct device *dev) } } }else { -if (device_def_domain_type (dev) ==IOMMU_DOMAIN_DMA ) { +if (intel_iommu_dev_def_domain_type (dev) ==IOMMU_DOMAIN_DMA ) { ret =iommu_request_dma_domain_for_dev (dev);if (ret) {dmar_remove_one_dev_info (dev);
drivers/iommu/intel-iommu.c const struct iommu_ops intel_iommu_ops = { · line 6201
Quoted 7 lines
drivers/iommu/intel-iommu.c +1
-0@@ -6194,6 +6195,7 @@ const struct iommu_ops intel_iommu_ops = { .dev_enable_feat = intel_iommu_dev_enable_feat, .dev_disable_feat = intel_iommu_dev_disable_feat, .is_attach_deferred = intel_iommu_is_attach_deferred, + .dev_def_domain_type = intel_iommu_dev_def_domain_type, .pgsize_bitmap = INTEL_IOMMU_PGSIZES, };
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH V2 2/5] iommu/vt-d: Rename device_def_domain_type() to intel_iommu_dev_def_domain_type()
Quoted 11 lines · Sai Praneeth Prakhya
Quoted 6 lines
The functionality needed for
iommu_ops->dev_def_domain_type()is already provided bydevice_def_domain_type()inintel_iommu.c. But, every call back function inintel_iommu_opsstarts withintel_iommuprefix, hence renamedevice_def_domain_type()tointel_iommu_dev_def_domain_type()so that it follows the same semantics.How about keep
device_def_domain_type()and call it in the newintel_iommu_dev_def_domain_type()?
Sure! I could but could you please explain the advantages we might get by doing so? Less number of changes is what I could think of.. any other reasons?
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
In reply to Sai Praneeth PrakhyaHi,
Quoted 4 lines · Sai Praneeth Prakhya
Presently, the default domain of an
iommu_groupis allocated during boot time (i.e. when a device is being added to a group) and it cannot be changed later. So, the device would typically be either in identity (also known aspass_through) mode (controlled by "iommu=pt" kernel command line
The default domain type is initialized according to the kernel build option, and could be overrided by several kernel commands like iommu=pt and iommu.passthrough.
Quoted 66 lines
argument) or the device would be in DMA mode as long as the machine is up and running. There is no way to change the default domain type dynamically i.e. after booting, a device cannot switch between identity mode and DMA mode.
But, assume a use case wherein the user trusts the device and also believes that the OS is secure enough and hence wants *only* this device to bypass IOMMU (so that it could be high performing) whereas all the other devices to go through IOMMU (so that the system is protected). Presently, this use case is not supported. Hence it will be helpful if there is some way to change the default domain of a B:D.F dynamically. Since, linux iommu subsystem prefers to deal at
iommu_grouplevel instead of B:D.F level, it might be helpful if there is some way to change the default domain of a *iommu_group* dynamically. Hence, add such support.A privileged user could request the kernel to change the default domain type of a
iommu_groupby writing to "/sys/kernel/iommu_groups/<grp_id>/type"
file. Presently, only three values are supported
1. identity: all the DMA transactions from the devices in this group are *not* translated by the iommu
2. DMA: all the DMA transactions from the devices in this group are translated by the iommu
3. auto: kernel decides one among DMA/identityAlso please note that a group type could be modified only when *all* the devices in the group are not binded to any device driver. Please see "
Documentation/ABI/testing/sysfs-kernel-iommu_groups" for more information.Cc Christoph Hellwig <[email protected]>Cc Joerg Roedel <[email protected]>Cc Ashok Raj <[email protected]>Cc Will Deacon <[email protected]>Cc Lu Baolu <[email protected]>Cc Sohil Mehta <[email protected]>Cc Robin Murphy <[email protected]>Cc Jacob Pan <[email protected]>Signed-off-by Sai Praneeth Prakhya <[email protected]>--- drivers/iommu/iommu.c | 227 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 226 insertions(+), 1 deletion(-)drivers/iommu/iommu.c +4
-1@@ -225,6 +225,8 @@ static int __iommu_attach_group(struct iommu_domain *domain, struct iommu_group *group);static void __iommu_detach_group (struct iommu_domain *domain,struct iommu_group *group); +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count);static int __init iommu_set_def_domain_type (char *str) {@@ -448,7 +450,8 @@ static IOMMU_GROUP_ATTR(name, S_IRUGO, iommu_group_show_name, NULL); static IOMMU_GROUP_ATTR(reserved_regions, 0444, iommu_group_show_resv_regions, NULL); -static IOMMU_GROUP_ATTR(type, 0444, iommu_group_show_type, NULL); +static IOMMU_GROUP_ATTR(type, 0644, iommu_group_show_type, + iommu_group_store_type);static void iommu_group_release (struct kobject *kobj) {
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2665
Quoted 9 lines
drivers/iommu/iommu.c +6
-0@@ -2654,3 +2657,225 @@ int iommu_sva_get_pasid(struct iommu_sva *handle)return ops->sva_get_pasid (handle); }EXPORT_SYMBOL_GPL (iommu_sva_get_pasid); + +/* +* Changes the default domain of a group +* +* @group: The group for which the default domain should be changed +* @prv_dom: The previous domain that is being switched from
The previous domain is still kept in group->default_domain, so it's unnecessary to pass it as a parameter, right?
Quoted 46 lines
drivers/iommu/iommu.c +46
-0+* @type: The type of the new default domain that gets associated with the group +* +* Returns 0 on success and error code on failure +*/ +static int iommu_group_change_def_domain (struct iommu_group *group, +struct iommu_domain *prv_dom, +int type) +{ +struct group_device *grp_dev, *temp; +struct iommu_domain *new_domain; +int ret; + +/* +* iommu_domain_alloc() takes "struct bus_type" as an argument which is +* a member in "struct device". Changing a group's default domain type +* deals at iommu_group level rather than device level and hence there +* is no straight forward way to get "bus_type" of an iommu_group that +* could be passed to iommu_domain_alloc(). So, instead of directly +* calling iommu_domain_alloc(), use iommu_ops from previous default +* domain. +*/ +if (!prv_dom || !prv_dom->ops || !prv_dom->ops->domain_alloc || !type) +return -EINVAL ; + +/* Allocate a new domain of requested type */ + new_domain = prv_dom->ops->domain_alloc (type); +if (!new_domain) { +pr_err ("Unable to allocate memory for the new domain\n" ); +return -ENOMEM ; + } + + new_domain->type = type; + new_domain->ops = prv_dom->ops; + new_domain->pgsize_bitmap = prv_dom->pgsize_bitmap; + +/* +* Set these upfront here because iommu_group_add_device() and +* iommu_group_create_direct_mappings() needs these to be set +*/ +mutex_lock (&group->mutex); + group->default_domain = new_domain; + group->domain = new_domain; +mutex_unlock (&group->mutex); + +iommu_group_ref_get (group); +
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2721
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+list_for_each_entry_safe (grp_dev, temp, &group->devices, list) { +struct device *dev; + + dev = grp_dev->dev; +iommu_release_device (dev); + + ret =iommu_group_add_device (group, dev); +if (ret) +dev_err (dev,"Failed to add to iommu group %d: %d\n" , + group->id, ret);
Need to handle this error case.
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2725
Quoted 4 lines
drivers/iommu/iommu.c +4
-0+ + ret = prv_dom->ops->add_device (dev); +if (ret) +dev_err (dev,"Error adding to iommu: %d\n" , ret);
Ditto.
Quoted 38 lines
drivers/iommu/iommu.c +38
-0+ } + +iommu_group_put (group); +iommu_domain_free (prv_dom); +return 0 ; +} + +static int is_driver_bound (struct device *dev,void *not_used) +{ +int ret =0 ; + +device_lock (dev); +if (device_is_bound (dev)) + ret =1 ; +device_unlock (dev); +return ret; +} + +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) +{ +int ret =0 , req_type =0 , req_auto =0 ; +struct iommu_domain *prv_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops; +struct device *dev; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" ))
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2773
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ req_auto =1 ; + else + return -EINVAL; + +/* +* Check if any device in the group still has a driver binded to it. +* This might race with device driver probing code and unfortunately +* there is no clean way out of that either, locking all devices in the +* group and then do the re-attach will introduce a lock-inversion with +* group->mutex - Joerg.
Do you mean that we can't do below?
mutex_lock(&group->mutex);for_each_group_device()
device_lock(dev);
/* Default domain switch */for_each_group_device()
device_unlock()mutex_unlock(&group->mutex)
Quoted 62 lines
drivers/iommu/iommu.c +62
-0+*/ +if (iommu_group_for_each_dev (group,NULL , is_driver_bound)) { +pr_err ("Active drivers exist for devices in the group\n" ); +return -EBUSY ; + } + +mutex_lock (&group->mutex); + prv_dom = group->default_domain; +if (!prv_dom || !prv_dom->ops || !prv_dom->ops->dev_def_domain_type) { +pr_err ("'dev_def_domain_type' call back isn't registered\n" ); + ret = -EPERM ; + goto out; + } + +/* +* Check if any user level driver (that doesn't use kernel driver like +* VFIO) is directly using the group. VFIO use case is detected by +* is_driver_bound() check above +*/ +if (prv_dom != group->domain) { +pr_err ("Group assigned to user level for direct access\n" ); + ret = -EBUSY ; + goto out; + } + +/* +* If user has requested "auto", kernel has to decide the default domain +* type of a group. Hence, find out individual preferences of a device. +*/ + ops = prv_dom->ops; +if (req_auto) { +int dma_devs =0 , idt_devs =0 , dev_def_dom; + +list_for_each_entry (grp_dev, &group->devices, list) { + dev = grp_dev->dev; + dev_def_dom = ops->dev_def_domain_type (dev); +if (dev_def_dom ==IOMMU_DOMAIN_IDENTITY ) + idt_devs++; +if (dev_def_dom ==IOMMU_DOMAIN_DMA ) + dma_devs++; + } + +/* +* Default domain type of a group is undefined if some devices +* in the group should be in dma domain while other devices +* should be in identity domain +*/ +if (idt_devs && dma_devs) { +pr_err ("Some devices need identity domain while other need dma domain\n" ); + ret = -EPERM ; + goto out; + } + +/* +* Default domain type of a group is identity if +* 1. All the devices in the group need to be in identity domain +* 2. Some devices should be in identity domain while other +* devices could be in either of dma or identity domain +*/ +if (idt_devs && !dma_devs) + req_type =IOMMU_DOMAIN_IDENTITY ; +
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2845
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+/* +* Default domain type of a group is dma if +* 1. All the devices in the group need to be in dma domain +* 2. Some devices should be in dma domain while other devices +* could be in either of dma or identity domain +* 3. All the devices could be in either of the domains (namely +* dma and identity) +*/ +if (!idt_devs) + req_type =IOMMU_DOMAIN_DMA ;
Actually, There are four combinations:
idt_devs==0 idt_devs!=0
dma_devs == 0 iommu_def_domain_type identitydma_devs != 0 DMA unsupported
Quoted 21 lines
drivers/iommu/iommu.c +21
-0+ } + +/* +* Switch to a new domain only if the requested domain type is different +* from the existing default domain type +*/ +if (prv_dom->type == req_type) + goto out; + +/* +* Every device may not support both the domain types (namely DMA and +* identity), so check if it's ok to change domain type of every device +* in the group to the requested domain. This check is not required if +* user has requested "auto" because it's already done above implicitly. +*/ +if (!req_auto) { +list_for_each_entry (grp_dev, &group->devices, list) { +int allowed_types; + + dev = grp_dev->dev; + allowed_types = ops->dev_def_domain_type (dev);
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2876
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+if (allowed_types && allowed_types != req_type) { +dev_err (dev,"Cannot be in %s domain\n" , buf); + ret = -EPERM ; + goto out; + } + } + } + +mutex_unlock (&group->mutex); + ret =iommu_group_change_def_domain (group, prv_dom, req_type);
Why do you want to put group->mutex before do the real domain switching?
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2881
Quoted 6 lines
drivers/iommu/iommu.c +5
-0+return ret ?: count; +out: +mutex_unlock (&group->mutex); +return ret ?: count; +}
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 5/5] iommu: Document usage of "/sys/kernel/iommu_groups/<grp_id>/type" file
In reply to Sai Praneeth PrakhyaHi,
Quoted 22 lines · Sai Praneeth Prakhya
The default domain type of an iommu group can be changed using file "/sys/kernel/iommu_groups/<grp_id>/type". Hence, document it's usage and more importantly spell out it's limitations.
Cc Christoph Hellwig <[email protected]>Cc Joerg Roedel <[email protected]>Cc Ashok Raj <[email protected]>Cc Will Deacon <[email protected]>Cc Lu Baolu <[email protected]>Cc Sohil Mehta <[email protected]>Cc Robin Murphy <[email protected]>Cc Jacob Pan <[email protected]>Signed-off-by Sai Praneeth Prakhya <[email protected]>--- .../ABI/testing/sysfs-kernel-iommu_groups | 29 ++++++++++++++++++++++ 1 file changed, 29 insertions(+)Documentation/ABI/testing/sysfs-kernel-iommu_groups +0
-0
Documentation/ABI/testing/sysfs-kernel-iommu_groups Description: In case an RMRR is used only by graphics or USB devices · line 42
Quoted 10 lines
Documentation/ABI/testing/sysfs-kernel-iommu_groups +7
-0@@ -33,3 +33,32 @@ Description: In case an RMRR is used only by graphics or USB devices it is now exposed as "direct-relaxable" instead of "direct". In device assignment use case, for instance, those RMRR are considered to be relaxable and safe. + +What: /sys/kernel/iommu_groups/<grp_id>/type +Date: February 2020 +KernelVersion: v5.6 +Contact: Sai Praneeth Prakhya <[email protected]> +Description: Lets the user know the type of default domain in use by iommu + for this group. A privileged user could request kernel to change
Let the users know the default domain type of the IOMMU group by reading this file. A privileged user could request to change it by writing to this file.
Documentation/ABI/testing/sysfs-kernel-iommu_groups Description: In case an RMRR is used only by graphics or USB devices · line 50
Quoted 8 lines
Documentation/ABI/testing/sysfs-kernel-iommu_groups +8
-0+ the group type by writing to this file. Presently, only three + types are supported + 1. DMA: All the DMA transactions from the devices in this group + are translated by the iommu. + 2. identity: All the DMA transactions from the devices in this + group are *not* translated by the iommu. + 3. auto: Kernel decides one among DMA/identity mode and hence + when the user reads the file he would never see "auto".
Just out of curiosity, when could a user reads "auto" from this file?
Documentation/ABI/testing/sysfs-kernel-iommu_groups Description: In case an RMRR is used only by graphics or USB devices · line 54
Quoted 4 lines
Documentation/ABI/testing/sysfs-kernel-iommu_groups +4
-0+ This is just a write only value. + Note: + ----- + A group type could be modified only when *all* the devices in
group's default domain type (not group type).
Documentation/ABI/testing/sysfs-kernel-iommu_groups Description: In case an RMRR is used only by graphics or USB devices · line 55
Quoted 1 line
Documentation/ABI/testing/sysfs-kernel-iommu_groups +1
-0+ the group are not binded to any device driver. So, the user has
bound
Documentation/ABI/testing/sysfs-kernel-iommu_groups Description: In case an RMRR is used only by graphics or USB devices · line 64
Quoted 10 lines
Documentation/ABI/testing/sysfs-kernel-iommu_groups +9
-0+ to first unbind the appropriate drivers and then change the + default domain type. + Caution: + -------- + Unbinding a device driver will take away the drivers control + over the device and if done on devices that host root file + system could lead to catastrophic effects (the user might + need to reboot the machine to get it to normal state). So, it's + expected that the user understands what he is doing.
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 2/5] iommu/vt-d: Rename device_def_domain_type() to intel_iommu_dev_def_domain_type()
In reply to Prakhya, Sai PraneethHi,
Quoted 9 lines · Sai Praneeth
Quoted 9 lines · Sai Praneeth Prakhya
Quoted 6 lines
The functionality needed for
iommu_ops->dev_def_domain_type()is already provided bydevice_def_domain_type()inintel_iommu.c. But, every call back function inintel_iommu_opsstarts withintel_iommuprefix, hence renamedevice_def_domain_type()tointel_iommu_dev_def_domain_type()so that it follows the same semantics.
Quoted 6 lines
Quoted 2 lines
How about keep
device_def_domain_type()and call it in the newintel_iommu_dev_def_domain_type()?Sure! I could but could you please explain the advantages we might get by doing so? Less number of changes is what I could think of.. any other reasons?
device_def_domain_type() is a quirk list for devices that must use a specified domain type. intel_iommu_dev_def_domain_type() tells the upper layer whether the device could switch to another type of domain. Put them in separated functions will make it easier for maintenance.
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH V2 5/5] iommu: Document usage of "/sys/kernel/iommu_groups/<grp_id>/type" file
In reply to Lu BaoluQuoted 10 lines
Quoted 9 lines
(quoted diff) +5
-0+What: /sys/kernel/iommu_groups/<grp_id>/type +Date: February 2020 +KernelVersion: v5.6 +Contact: Sai Praneeth Prakhya <[email protected]<mailto:[email protected]>> +Description: Lets the user know the type of default domain in use by iommu
Quoted 7 lines
Quoted 1 line
+ for this group. A privileged user could request kernel to change
Let the users know the default domain type of the IOMMU group by reading this
file. A privileged user could request to change it by writing to this file.
Sure! Will fix it.
Quoted 14 lines
Review
Quoted 13 lines
(quoted diff) +7
-0+ the group type by writing to this file. Presently, only three + types are supported + 1. DMA: All the DMA transactions from the devices in this group + are translated by the iommu. + 2. identity: All the DMA transactions from the devices in this + group are *not* translated by the iommu. + 3. auto: Kernel decides one among DMA/identity mode and
Quoted 7 lines
hence
Quoted 1 line
+ when the user reads the file he would never see "auto".
Just out of curiosity, when could a user reads "auto" from this file?
Never, because when a user requests kernel to change to "auto" mode, kernel will select one among identity or DMA. So, default domain type cannot be auto, it's always either identity or DMA.
"auto" mode is just a write value where user is requesting kernel to select one among identity or DMA.
Quoted 11 lines
Documentation/ABI/testing/sysfs-kernel-iommu_groups Description: In case an RMRR is used only by graphics or USB devices · line 54
Quoted 7 lines
Documentation/ABI/testing/sysfs-kernel-iommu_groups +4
-0+ This is just a write only value. + Note: + ----- + A group type could be modified only when *all* the devices ingroup's default domain type (not group type).
Sure! Makes sense.. will fix it 😊
Quoted 7 lines
Quoted 3 lines
+ the group are not binded to any device driver. So, the user has
bound
Makes sense.. will fix this too.
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
In reply to Lu BaoluQuoted 10 lines · Sai Praneeth Prakhya
Quoted 5 lines
Presently, the default domain of an
iommu_groupis allocated during boot time (i.e. when a device is being added to a group) and it cannot be changed later. So, the device would typically be either in identity (also known aspass_through) mode (controlled by "iommu=pt" kernel command lineThe default domain type is initialized according to the kernel build option, and could be overrided by several kernel commands like iommu=pt and iommu.passthrough.
Yes, that's right. Will change this text.
Quoted 15 lines
Quoted 15 lines
argument) or the device would be in DMA mode as long as the machine is up and running. There is no way to change the default domain type dynamically i.e. after booting, a device cannot switch between identity mode and DMA mode.
But, assume a use case wherein the user trusts the device and also believes that the OS is secure enough and hence wants *only* this device to bypass IOMMU (so that it could be high performing) whereas all the other devices to go through IOMMU (so that the system is protected). Presently, this use case is not supported. Hence it will be helpful if there is some way to change the default domain of a B:D.F dynamically. Since, linux iommu subsystem prefers to deal at
iommu_grouplevel instead of B:D.F level, it might be helpful if there is some way to change the default domain of a
*iommu_group* dynamically. Hence, add such support.
[snipped]
Quoted 8 lines
Quoted 5 lines
+/*
+ * Changes the default domain of a group
+ *
+ * @group: The group for which the default domain should be changed
+ * @prv_dom: The previous domain that is being switched fromThe previous domain is still kept in
group->default_domain, so it's unnecessary to pass it as a parameter, right?
Yes, you are right. I passed it as an argument thinking that I could make one assignment less in this function. The caller of this function iommu_group_store_type() already has prv_dom, so just passed it. But, will change this so that instead of every caller getting and passing the argument, it's easier to do it in this function.
Quoted 52 lines
Quoted 4 lines
+ * @type: The type of the new default domain that gets associated
drivers/iommu/iommu.c +3
-0+with the group +* + * Returns 0 on success and error code on failure */ static intdrivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+iommu_group_change_def_domain(struct iommu_group *group, + struct iommu_domain *prv_dom, + int type) +{ + struct group_device *grp_dev, *temp; +struct iommu_domain *new_domain; +int ret; + +/* +* iommu_domain_alloc() takes "struct bus_type" as an argument which is
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 8 lines
drivers/iommu/iommu.c +8
-0+* a member in "struct device". Changing a group's default domain type +* deals at iommu_group level rather than device level and hence there +* is no straight forward way to get "bus_type" of an iommu_group that +* could be passed to iommu_domain_alloc(). So, instead of directly +* calling iommu_domain_alloc(), use iommu_ops from previous default +* domain. +*/ + if (!prv_dom || !prv_dom->ops || !prv_dom->ops->domain_alloc ||!type)
Quoted 28 lines
drivers/iommu/iommu.c +28
-0+return -EINVAL ; + +/* Allocate a new domain of requested type */ + new_domain = prv_dom->ops->domain_alloc (type); +if (!new_domain) { +pr_err ("Unable to allocate memory for the new domain\n" ); +return -ENOMEM ; + } + + new_domain->type = type; + new_domain->ops = prv_dom->ops; + new_domain->pgsize_bitmap = prv_dom->pgsize_bitmap; + +/* +* Set these upfront here because iommu_group_add_device() and +* iommu_group_create_direct_mappings() needs these to be set +*/ +mutex_lock (&group->mutex); + group->default_domain = new_domain; + group->domain = new_domain; +mutex_unlock (&group->mutex); + +iommu_group_ref_get (group); + +list_for_each_entry_safe (grp_dev, temp, &group->devices, list) { +struct device *dev; + + dev = grp_dev->dev;
Quoted 8 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2721
Quoted 6 lines
drivers/iommu/iommu.c +6
-0+iommu_release_device (dev); + + ret =iommu_group_add_device (group, dev); +if (ret) +dev_err (dev,"Failed to add to iommu group %d: %d\n" , + group->id, ret);Need to handle this error case.
I wasn't sure on how to handle the error ☹
i.e. group's domain/default_domain are already updated to new domain and assume there are 'n' devices in the group and this failed for 'k'th device, I wasn't sure how I could roll back the changes made for k-1 devices. So, I thought probably just alert the user that there was an error while changing default domain type and try updating for other devices in the group (hopefully other devices might succeed). Also, *generally* we shouldn't see any errors here because all these devices were already in the same group earlier (we aren’t adding/removing new devices to the group). We are just changing default domain type and we already made sure that device could be in the requested default domain type.
Quoted 56 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2725
Quoted 4 lines
drivers/iommu/iommu.c +4
-0+ + ret = prv_dom->ops->add_device (dev); +if (ret) +dev_err (dev,"Error adding to iommu: %d\n" , ret);Ditto.
Quoted 36 lines
drivers/iommu/iommu.c +36
-0+ } + +iommu_group_put (group); +iommu_domain_free (prv_dom); +return 0 ; +} + +static int is_driver_bound (struct device *dev,void *not_used) { +int ret =0 ; + +device_lock (dev); +if (device_is_bound (dev)) + ret =1 ; +device_unlock (dev); +return ret; +} + +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) { +int ret =0 , req_type =0 , req_auto =0 ; +struct iommu_domain *prv_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops; +struct device *dev; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" ))drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2773
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ req_auto =1 ; + else + return -EINVAL; + +/* +* Check if any device in the group still has a driver binded to it. +* This might race with device driver probing code and unfortunately +* there is no clean way out of that either, locking all devices in the +* group and then do the re-attach will introduce a lock-inversion with +* group->mutex - Joerg. Do you mean that we can't do below?
Quoted 8 lines
mutex_lock(&group->mutex);for_each_group_device()device_lock(dev);/* Default domain switch */
for_each_group_device()
device_unlock()mutex_unlock(&group->mutex)
I think, Joerg talks about two issues here
1. is_driver_bound() takes/releases device_lock() which could race with device driver probing code i.e. in an other terminal user could try to unload/load the module.
2. We cannot do as you suggested above because the functions (one is mentioned below) that switch default domain need to take iommu_group_mutex lock. So, taking the mutex and then calling iommu functions that switch default domain will dead lock.
Quoted 77 lines
Quoted 40 lines
drivers/iommu/iommu.c +9
-0+*/ +if (iommu_group_for_each_dev (group,NULL , is_driver_bound)) { +pr_err ("Active drivers exist for devices in the group\n" ); +return -EBUSY ; + } + +mutex_lock (&group->mutex); + prv_dom = group->default_domain; + if (!prv_dom || !prv_dom->ops || !prv_dom->ops-
dev_def_domain_type) {drivers/iommu/iommu.c +30
-0+pr_err ("'dev_def_domain_type' call back isn't registered\n" ); + ret = -EPERM ; + goto out; + } + +/* +* Check if any user level driver (that doesn't use kernel driver like +* VFIO) is directly using the group. VFIO use case is detected by +* is_driver_bound() check above +*/ +if (prv_dom != group->domain) { +pr_err ("Group assigned to user level for direct access\n" ); + ret = -EBUSY ; + goto out; + } + +/* +* If user has requested "auto", kernel has to decide the default domain +* type of a group. Hence, find out individual preferences of a device. +*/ + ops = prv_dom->ops; +if (req_auto) { +int dma_devs =0 , idt_devs =0 , dev_def_dom; + +list_for_each_entry (grp_dev, &group->devices, list) { + dev = grp_dev->dev; + dev_def_dom = ops->dev_def_domain_type (dev); +if (dev_def_dom ==IOMMU_DOMAIN_IDENTITY ) + idt_devs++; +if (dev_def_dom ==IOMMU_DOMAIN_DMA )drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ dma_devs++; + } + +/* +* Default domain type of a group is undefined if some devices +* in the group should be in dma domain while other devices +* should be in identity domain +*/ +if (idt_devs && dma_devs) { + pr_err("Some devices need identity domain while otherneed dma domain\n");
Quoted 13 lines
drivers/iommu/iommu.c +13
-0+ ret = -EPERM ; + goto out; + } + +/* +* Default domain type of a group is identity if +* 1. All the devices in the group need to be in identity domain +* 2. Some devices should be in identity domain while other +* devices could be in either of dma or identity domain +*/ +if (idt_devs && !dma_devs) + req_type =IOMMU_DOMAIN_IDENTITY ; +drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2845
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+/* +* Default domain type of a group is dma if +* 1. All the devices in the group need to be in dma domain +* 2. Some devices should be in dma domain while other devices +* could be in either of dma or identity domain +* 3. All the devices could be in either of the domains (namely +* dma and identity) +*/ +if (!idt_devs) + req_type =IOMMU_DOMAIN_DMA ;Actually, There are four combinations:
Quoted 3 lines
idt_devs==0 idt_devs!=0
dma_devs== 0iommu_def_domain_typeidentitydma_devs!= 0 DMA unsupported
Yes.. you are right. All these four combinations are presently handled in the code. The comment above talks about combinations that lead us to select DMA domain.
The cases you mentioned above are handled as:
1. dma_devs == 0 && idt_devs == 0 -> In code, req_type = IOMMU_DOMAIN_DMA; because this means *all* the devices in the group can be in either of identity domain or DMA domain, hence select DMA domain over identity domain.
2. dma_devs != 0 && idt_devs == 0 -> In code, req_type = IOMMU_DOMAIN_DMA; because this means some devices in the group *should* be in DMA domain and other devices in the group could be in either of identity domain or DMA domain.
3. dma_devs == 0 && idt_devs != 0 -> In code, req_type = IOMMU_DOMAIN_IDENTITY; because this means some devices in the group *should* be in identity domain and other devices in the group could be in either of identity domain or DMA domain.
4. dma_devs != 0 && idt_devs != 0 -> In code, return error, because this means some devices in the group *should* be in identity domain and other devices in the group *should* be in DMA domain. This combination is not expected. This situation shouldn't arise because this would have failed during boot itself because these kind of devices aren't grouped together in the first place.
Quoted 26 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+ } + +/* +* Switch to a new domain only if the requested domain type is different +* from the existing default domain type +*/ +if (prv_dom->type == req_type) + goto out; + +/* +* Every device may not support both the domain types (namely DMA and
Quoted 14 lines
drivers/iommu/iommu.c +14
-0+* identity), so check if it's ok to change domain type of every device +* in the group to the requested domain. This check is not required if +* user has requested "auto" because it's already done above implicitly. +*/ +if (!req_auto) { +list_for_each_entry (grp_dev, &group->devices, list) { +int allowed_types; + + dev = grp_dev->dev; + allowed_types = ops->dev_def_domain_type (dev); +if (allowed_types && allowed_types != req_type) { +dev_err (dev,"Cannot be in %s domain\n" , buf); + ret = -EPERM ; + goto out;
Quoted 8 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2876
Quoted 6 lines
drivers/iommu/iommu.c +6
-0+ } + } + } + +mutex_unlock (&group->mutex); + ret =iommu_group_change_def_domain (group, prv_dom, req_type);Why do you want to put
group->mutexbefore do the real domain switching?
Because the below call stack takes iommu_group_mutex. I drop the lock here to facilitate it, otherwise the kernel hangs trying to get the same lock which it is already holding on to
mutex_lock(&group->mutex);iommu_group_remove_device()intel_iommu_remove_device()iommu_release_device()iommu_group_change_def_domain()
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH V2 2/5] iommu/vt-d: Rename device_def_domain_type() to intel_iommu_dev_def_domain_type()
In reply to Lu BaoluQuoted 12 lines
Quoted 8 lines
Quoted 8 lines · Sai Praneeth Prakhya
Quoted 6 lines
The functionality needed for
iommu_ops->dev_def_domain_type()is already provided bydevice_def_domain_type()inintel_iommu.c. But, every call back function inintel_iommu_opsstarts withintel_iommuprefix, hence renamedevice_def_domain_type()tointel_iommu_dev_def_domain_type()so that it follows the same semantics.Quoted 4 lines
Quoted 2 lines
How about keep
device_def_domain_type()and call it in the newintel_iommu_dev_def_domain_type()?Sure! I could but could you please explain the advantages we might get by
Quoted 8 lines
doing so?
Quoted 2 lines
Less number of changes is what I could think of.. any other reasons?
device_def_domain_type()is a quirk list for devices that must use a specified domain type.intel_iommu_dev_def_domain_type()tells the upper layer whether the device could switch to another type of domain. Put them in separated functions will make it easier for maintenance.
Ok. Will fix this.
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
In reply to Prakhya, Sai PraneethHi Sai,
Quoted 11 lines · Sai Praneeth
Quoted 11 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2721
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+list_for_each_entry_safe (grp_dev, temp, &group->devices, list) { +struct device *dev; + + dev = grp_dev->dev; +iommu_release_device (dev); + + ret =iommu_group_add_device (group, dev); +if (ret) +dev_err (dev,"Failed to add to iommu group %d: %d\n" , + group->id, ret);Need to handle this error case.
Quoted 1 line
I wasn't sure on how to handle the error ☹
Just roll back to the state before calling this function and return an appropriate error value.
The likely behavior is detaching the new domains from all devices (if it has already attached), attaching the old domains to all devices in the group, cleaning up all new resources allocated in this function, putting a error message to tell the user why it fails and returning an error code.
Quoted 1 line
i.e. group's domain/default_domain are already updated to new domain and assume there are 'n' devices in the group and this failed for 'k'th device, I wasn't sure how I could roll back the changes made for k-1 devices.
A successful attach could be checked by (group->domain == group->default_domain).
Quoted 60 lines
So, I thought probably just alert the user that there was an error while changing default domain type and try updating for other devices in the group (hopefully other devices might succeed). Also,*generally* we shouldn't see any errors here because all these devices were already in the same group earlier (we aren’t adding/removing new devices to the group). We are just changing default domain type and we already made sure that device could be in the requested default domain type.
Quoted 58 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2725
Quoted 4 lines
drivers/iommu/iommu.c +4
-0+ + ret = prv_dom->ops->add_device (dev); +if (ret) +dev_err (dev,"Error adding to iommu: %d\n" , ret);Ditto.
Quoted 36 lines
drivers/iommu/iommu.c +36
-0+ } + +iommu_group_put (group); +iommu_domain_free (prv_dom); +return 0 ; +} + +static int is_driver_bound (struct device *dev,void *not_used) { +int ret =0 ; + +device_lock (dev); +if (device_is_bound (dev)) + ret =1 ; +device_unlock (dev); +return ret; +} + +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) { +int ret =0 , req_type =0 , req_auto =0 ; +struct iommu_domain *prv_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops; +struct device *dev; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" ))drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2773
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ req_auto =1 ; + else + return -EINVAL; + +/* +* Check if any device in the group still has a driver binded to it. +* This might race with device driver probing code and unfortunately +* there is no clean way out of that either, locking all devices in the +* group and then do the re-attach will introduce a lock-inversion with +* group->mutex - Joerg. Do you mean that we can't do below?
mutex_lock(&group->mutex);for_each_group_device()device_lock(dev);
Quoted 6 lines
Quoted 4 lines
/* Default domain switch */
for_each_group_device()
device_unlock()mutex_unlock(&group->mutex)I think, Joerg talks about two issues here
1.is_driver_bound()takes/releasesdevice_lock()which could race with device driver probing code i.e. in an other terminal user could try to unload/load the module.
So you need to device_lock() all devices during the whole process. (I assume that load/unload device driver requiring this lock as well. Didn't check it in code. Please correct me if I misunderstood it.)
Quoted 1 line
2. We cannot do as you suggested above because the functions (one is mentioned below) that switch default domain need to take
iommu_group_mutexlock. So, taking the mutex and then calling iommu functions that switch default domain will dead lock.
You probably need to move the mutex_lock() in that function out to the caller? And only assert the lock has been held there.
Quoted 77 lines
Quoted 77 lines
Quoted 41 lines
drivers/iommu/iommu.c +9
-0+*/ +if (iommu_group_for_each_dev (group,NULL , is_driver_bound)) { +pr_err ("Active drivers exist for devices in the group\n" ); +return -EBUSY ; + } + +mutex_lock (&group->mutex); + prv_dom = group->default_domain; + if (!prv_dom || !prv_dom->ops || !prv_dom->ops-
dev_def_domain_type) {drivers/iommu/iommu.c +30
-0+pr_err ("'dev_def_domain_type' call back isn't registered\n" ); + ret = -EPERM ; + goto out; + } + +/* +* Check if any user level driver (that doesn't use kernel driver like +* VFIO) is directly using the group. VFIO use case is detected by +* is_driver_bound() check above +*/ +if (prv_dom != group->domain) { +pr_err ("Group assigned to user level for direct access\n" ); + ret = -EBUSY ; + goto out; + } + +/* +* If user has requested "auto", kernel has to decide the default domain +* type of a group. Hence, find out individual preferences of a device. +*/ + ops = prv_dom->ops; +if (req_auto) { +int dma_devs =0 , idt_devs =0 , dev_def_dom; + +list_for_each_entry (grp_dev, &group->devices, list) { + dev = grp_dev->dev; + dev_def_dom = ops->dev_def_domain_type (dev); +if (dev_def_dom ==IOMMU_DOMAIN_IDENTITY ) + idt_devs++; +if (dev_def_dom ==IOMMU_DOMAIN_DMA )drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ dma_devs++; + } + +/* +* Default domain type of a group is undefined if some devices +* in the group should be in dma domain while other devices +* should be in identity domain +*/ +if (idt_devs && dma_devs) { + pr_err("Some devices need identity domain while otherneed dma domain\n");
Quoted 13 lines
drivers/iommu/iommu.c +13
-0+ ret = -EPERM ; + goto out; + } + +/* +* Default domain type of a group is identity if +* 1. All the devices in the group need to be in identity domain +* 2. Some devices should be in identity domain while other +* devices could be in either of dma or identity domain +*/ +if (idt_devs && !dma_devs) + req_type =IOMMU_DOMAIN_IDENTITY ; +drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2845
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+/* +* Default domain type of a group is dma if +* 1. All the devices in the group need to be in dma domain +* 2. Some devices should be in dma domain while other devices +* could be in either of dma or identity domain +* 3. All the devices could be in either of the domains (namely +* dma and identity) +*/ +if (!idt_devs) + req_type =IOMMU_DOMAIN_DMA ;Actually, There are four combinations:
Quoted 4 lines
Quoted 3 lines
idt_devs==0 idt_devs!=0
dma_devs== 0iommu_def_domain_typeidentitydma_devs!= 0 DMA unsupportedYes.. you are right. All these four combinations are presently handled in the code.
At least I didn't see the iommu_def_domain_type is used if both idt_devs and dma_devs are both equal to 0. :-)
Quoted 41 lines
The comment above talks about combinations that lead us to select DMA domain.
The cases you mentioned above are handled as:
1.dma_devs== 0 &&idt_devs== 0 -> In code,req_type=IOMMU_DOMAIN_DMA; because this means*all* the devices in the group can be in either of identity domain or DMA domain, hence select DMA domain over identity domain.
2.dma_devs!= 0 &&idt_devs== 0 -> In code,req_type=IOMMU_DOMAIN_DMA; because this means some devices in the group*should* be in DMA domain and other devices in the group could be in either of identity domain or DMA domain.
3.dma_devs== 0 &&idt_devs!= 0 -> In code,req_type=IOMMU_DOMAIN_IDENTITY; because this means some devices in the group*should* be in identity domain and other devices in the group could be in either of identity domain or DMA domain.
4.dma_devs!= 0 &&idt_devs!= 0 -> In code, return error, because this means some devices in the group*should* be in identity domain and other devices in the group*should* be in DMA domain. This combination is not expected. This situation shouldn't arise because this would have failed during boot itself because these kind of devices aren't grouped together in the first place.Quoted 25 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+ } + +/* +* Switch to a new domain only if the requested domain type is different +* from the existing default domain type +*/ +if (prv_dom->type == req_type) + goto out; + +/* +* Every device may not support both the domain types (namely DMA and
Quoted 13 lines
drivers/iommu/iommu.c +13
-0+* identity), so check if it's ok to change domain type of every device +* in the group to the requested domain. This check is not required if +* user has requested "auto" because it's already done above implicitly. +*/ +if (!req_auto) { +list_for_each_entry (grp_dev, &group->devices, list) { +int allowed_types; + + dev = grp_dev->dev; + allowed_types = ops->dev_def_domain_type (dev); +if (allowed_types && allowed_types != req_type) { +dev_err (dev,"Cannot be in %s domain\n" , buf); + ret = -EPERM ;Quoted 8 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2876
Quoted 7 lines
drivers/iommu/iommu.c +7
-0+ goto out; + } + } + } + +mutex_unlock (&group->mutex); + ret =iommu_group_change_def_domain (group, prv_dom, req_type);Why do you want to put
group->mutexbefore do the real domain switching?
Quoted 7 lines
Because the below call stack takes
iommu_group_mutex. I drop the lock here to facilitate it, otherwise the kernel hangs trying to get the same lock which it is already holding on to
mutex_lock(&group->mutex);iommu_group_remove_device()intel_iommu_remove_device()iommu_release_device()iommu_group_change_def_domain()
Refer to above comment.
Quoted 3 lines
Regards,
Sai
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
Quoted 12 lines · Sai Praneeth
Quoted 11 lines
Quoted 11 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2721
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+list_for_each_entry_safe (grp_dev, temp, &group->devices, list) { +struct device *dev; + + dev = grp_dev->dev; +iommu_release_device (dev); + + ret =iommu_group_add_device (group, dev); +if (ret) +dev_err (dev,"Failed to add to iommu group %d: %d\n" , + group->id, ret);Need to handle this error case.
Quoted 7 lines
Quoted 1 line
I wasn't sure on how to handle the error ☹
Just roll back to the state before calling this function and return an appropriate error value.
The likely behavior is detaching the new domains from all devices (if it has already attached), attaching the old domains to all devices in the group,
And while handling this error case, there is a possibility that attaching to old domain could fail.. so, we might need to handle that error case as well. If we plan to handle error case, since we have removed devices from group above, adding them back could fail too.. that would lead into handling error case for an error case. So, given the probability of these functions failing here are very low, I think, why not bite the bullet and say, add code to handle these error cases if we see that these functions are failing frequently? Otherwise the error handling code is just a dead code.
Quoted 10 lines
cleaning
up all new resources allocated in this function, putting a error message to tell the user why it fails and returning an error code.Quoted 1 line
i.e. group's domain/default_domain are already updated to new domain and
assume there are 'n' devices in the group and this failed for 'k'th device, I wasn't sure how I could roll back the changes made for k-1 devices.
A successful attach could be checked by (
group->domain==group->default_domain).
No.. because I have manually set group->domain == group->default_domain = new_domain (did this because iommu_group_add_device() and iommu_group_create_direct_mappings() needs them) So, probably we might need some other way to check successful attach.
Quoted 70 lines
Quoted 1 line
So, I thought probably just alert the user that there was an error while
changing default domain type and try updating for other devices in the group (hopefully other devices might succeed). Also,*generally* we shouldn't see any errors here because all these devices were already in the same group earlier (we aren’t adding/removing new devices to the group). We are just changing default domain type and we already made sure that device could be in the requested default domain type.
Quoted 63 lines
Quoted 63 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2725
Quoted 5 lines
drivers/iommu/iommu.c +4
-0+ + ret = prv_dom->ops->add_device (dev); +if (ret) +dev_err (dev,"Error adding to iommu: %d\n" , ret);Ditto.
Quoted 36 lines
drivers/iommu/iommu.c +36
-0+ } + +iommu_group_put (group); +iommu_domain_free (prv_dom); +return 0 ; +} + +static int is_driver_bound (struct device *dev,void *not_used) { +int ret =0 ; + +device_lock (dev); +if (device_is_bound (dev)) + ret =1 ; +device_unlock (dev); +return ret; +} + +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) { +int ret =0 , req_type =0 , req_auto =0 ; +struct iommu_domain *prv_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops; +struct device *dev; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" ))drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2773
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ req_auto =1 ; + else + return -EINVAL; + +/* +* Check if any device in the group still has a driver binded to it. +* This might race with device driver probing code and unfortunately +* there is no clean way out of that either, locking all devices in the +* group and then do the re-attach will introduce a lock-inversion with +* group->mutex - Joerg. Do you mean that we can't do below?
mutex_lock(&group->mutex);for_each_group_device()device_lock(dev);/* Default domain switch */
for_each_group_device()
device_unlock()mutex_unlock(&group->mutex)
Quoted 7 lines
Quoted 2 lines
I think, Joerg talks about two issues here 1.
is_driver_bound()takes/releasesdevice_lock()which could race with device driver probing codei.e. in an other terminal user could try to unload/load the module.
So you need to
device_lock()all devices during the whole process. (I assume that load/unload device driver requiring this lock as well. Didn't check it in code. Please correct me if I misunderstood it.)
Ok.. I will check this.
Quoted 7 lines
Quoted 1 line
2. We cannot do as you suggested above because the functions (one is
mentioned below) that switch default domain need to take
iommu_group_mutexlock. So, taking the mutex and then calling iommu functions that switch default domain will dead lock.You probably need to move the
mutex_lock()in that function out to the caller? And only assert the lock has been held there.
I mean, there are many functions that are taking iommu_group_mutex_lock and they are being called as part of changing default domain.
So, we would then need to change all those functions and let callers of those functions know that now it's their responsibility to take the lock and call these functions. Some of those functions are:
iommu_group_add_deviceiommu_group_remove_deviceiommu_group_for_each_dev
So, rather than that, I think it's better to delegate taking/releasing lock to individual functions than giving that responsibility to callers because then some callers might forget to take/release lock.
Quoted 77 lines
Quoted 77 lines
Quoted 77 lines
Quoted 41 lines
drivers/iommu/iommu.c +9
-0+*/ +if (iommu_group_for_each_dev (group,NULL , is_driver_bound)) { +pr_err ("Active drivers exist for devices in the group\n" ); +return -EBUSY ; + } + +mutex_lock (&group->mutex); + prv_dom = group->default_domain; + if (!prv_dom || !prv_dom->ops || !prv_dom->ops-
dev_def_domain_type) {drivers/iommu/iommu.c +30
-0+pr_err ("'dev_def_domain_type' call back isn't registered\n" ); + ret = -EPERM ; + goto out; + } + +/* +* Check if any user level driver (that doesn't use kernel driver like +* VFIO) is directly using the group. VFIO use case is detected by +* is_driver_bound() check above +*/ +if (prv_dom != group->domain) { +pr_err ("Group assigned to user level for direct access\n" ); + ret = -EBUSY ; + goto out; + } + +/* +* If user has requested "auto", kernel has to decide the default domain +* type of a group. Hence, find out individual preferences of a device. +*/ + ops = prv_dom->ops; +if (req_auto) { +int dma_devs =0 , idt_devs =0 , dev_def_dom; + +list_for_each_entry (grp_dev, &group->devices, list) { + dev = grp_dev->dev; + dev_def_dom = ops->dev_def_domain_type (dev); +if (dev_def_dom ==IOMMU_DOMAIN_IDENTITY ) + idt_devs++; +if (dev_def_dom ==IOMMU_DOMAIN_DMA )drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ dma_devs++; + } + +/* +* Default domain type of a group is undefined if some devices +* in the group should be in dma domain while other devices +* should be in identity domain +*/ +if (idt_devs && dma_devs) { + pr_err("Some devices need identity domain while otherneed dma domain\n");
Quoted 13 lines
drivers/iommu/iommu.c +13
-0+ ret = -EPERM ; + goto out; + } + +/* +* Default domain type of a group is identity if +* 1. All the devices in the group need to be in identity domain +* 2. Some devices should be in identity domain while other +* devices could be in either of dma or identity domain +*/ +if (idt_devs && !dma_devs) + req_type =IOMMU_DOMAIN_IDENTITY ; +drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2845
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+/* +* Default domain type of a group is dma if +* 1. All the devices in the group need to be in dma domain +* 2. Some devices should be in dma domain while other devices +* could be in either of dma or identity domain +* 3. All the devices could be in either of the domains (namely +* dma and identity) +*/ +if (!idt_devs) + req_type =IOMMU_DOMAIN_DMA ;Actually, There are four combinations:
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
idt_devs==0 idt_devs!=0
dma_devs== 0iommu_def_domain_typeidentitydma_devs!= 0 DMA unsupportedYes.. you are right. All these four combinations are presently handled in the
code.
At least I didn't see the
iommu_def_domain_typeis used if bothidt_devsanddma_devsare both equal to 0. :-)
Sorry! I didn't get what you meant by "iommu_def_domain_type", so, could you please explain that?
When idt_devs == 0 and dma_devs == 0, it means that all the devices in the group could be in either of the domain. Hence, I default to DMA domain which is covered by this code.
if (!idt_devs) req_type = IOMMU_DOMAIN_DMA;
Quoted 13 lines
Quoted 1 line
The comment above talks about combinations that lead us to select DMA
domain.
Quoted 3 lines
The cases you mentioned above are handled as:
1.dma_devs== 0 &&idt_devs== 0 -> In code,req_type=
IOMMU_DOMAIN_DMA; because this means*all* the devices in the group can be in either of identity domain or DMA domain, hence select DMA domain over identity domain.Quoted 1 line
2.
dma_devs!= 0 &&idt_devs== 0 -> In code,req_type=
IOMMU_DOMAIN_DMA; because this means some devices in the group*should* be in DMA domain and other devices in the group could be in either of identity domain or DMA domain.Quoted 1 line
3.
dma_devs== 0 &&idt_devs!= 0 -> In code,req_type=
Quoted 8 lines
IOMMU_DOMAIN_IDENTITY; because this means some devices in the group*should* be in identity domain and other devices in the group could be in either of identity domain or DMA domain.Quoted 1 line
4.
dma_devs!= 0 &&idt_devs!= 0 -> In code, return error, because this meanssome devices in the group*should* be in identity domain and other devices in the group*should* be in DMA domain. This combination is not expected. This situation shouldn't arise because this would have failed during boot itself because these kind of devices aren't grouped together in the first place.
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
Quoted 14 lines · Sai Praneeth
Quoted 14 lines · Sai Praneeth
Quoted 12 lines
Quoted 11 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2721
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+list_for_each_entry_safe (grp_dev, temp, &group->devices, list) { +struct device *dev; + + dev = grp_dev->dev; +iommu_release_device (dev); + + ret =iommu_group_add_device (group, dev); +if (ret) +dev_err (dev,"Failed to add to iommu group %d: %d\n" , + group->id, ret);Need to handle this error case.
I wasn't sure on how to handle the error ☹
Quoted 7 lines
Quoted 5 lines
Just roll back to the state before calling this function and return an appropriate error value.
The likely behavior is detaching the new domains from all devices (if it has already attached), attaching the old domains to all devices in the group,
And while handling this error case, there is a possibility that attaching to old domain could fail.. so, we might need to handle that error case as well. If we plan to handle error case, since we have removed devices from group above, adding them back could fail too.. that would lead into handling error case for an error case.
We can assume that the old domain should always be attached back. Otherwise, there must be some bugs in the vendor iommu driver.
It must be able to role back, otherwise users have to reboot the system in order to use the devices in the group. This is not acceptable in the production kernel.
Quoted 6 lines
So, given the probability of these functions failing here are very low, I think, why not bite the bullet and say, add code to handle these error cases if we see that these functions are failing frequently? Otherwise the error handling code is just a dead code.
Quoted 4 lines
cleaning
up all new resources allocated in this function, putting a error message to tell the user why it fails and returning an error code.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
i.e. group's domain/default_domain are already updated to new domain and
assume there are 'n' devices in the group and this failed for 'k'th device, I wasn't sure how I could roll back the changes made for k-1 devices.
A successful attach could be checked by (
group->domain==group->default_domain).No.. because I have manually set
group->domain==group->default_domain=new_domain(did this becauseiommu_group_add_device()andiommu_group_create_direct_mappings()needs them)
You could set group->domain to the default domain only after it has been attached to the device successfully, right?
Quoted 94 lines
So, probably we might need some other way to check successful attach.
Quoted 70 lines
Quoted 1 line
So, I thought probably just alert the user that there was an error while
changing default domain type and try updating for other devices in the group (hopefully other devices might succeed). Also,*generally* we shouldn't see any errors here because all these devices were already in the same group earlier (we aren’t adding/removing new devices to the group). We are just changing default domain type and we already made sure that device could be in the requested default domain type.
Quoted 63 lines
Quoted 63 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2725
Quoted 5 lines
drivers/iommu/iommu.c +4
-0+ + ret = prv_dom->ops->add_device (dev); +if (ret) +dev_err (dev,"Error adding to iommu: %d\n" , ret);Ditto.
Quoted 36 lines
drivers/iommu/iommu.c +36
-0+ } + +iommu_group_put (group); +iommu_domain_free (prv_dom); +return 0 ; +} + +static int is_driver_bound (struct device *dev,void *not_used) { +int ret =0 ; + +device_lock (dev); +if (device_is_bound (dev)) + ret =1 ; +device_unlock (dev); +return ret; +} + +static ssize_t iommu_group_store_type (struct iommu_group *group, +const char *buf,size_t count) { +int ret =0 , req_type =0 , req_auto =0 ; +struct iommu_domain *prv_dom; +struct group_device *grp_dev; +const struct iommu_ops *ops; +struct device *dev; + +if (!capable (CAP_SYS_ADMIN ) || !capable (CAP_SYS_RAWIO )) +return -EACCES ; + +if (WARN_ON (!group)) +return -EINVAL ; + +if (sysfs_streq (buf,"identity" )) + req_type =IOMMU_DOMAIN_IDENTITY ; +else if (sysfs_streq (buf,"DMA" )) + req_type =IOMMU_DOMAIN_DMA ; +else if (sysfs_streq (buf,"auto" ))drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2773
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ req_auto =1 ; + else + return -EINVAL; + +/* +* Check if any device in the group still has a driver binded to it. +* This might race with device driver probing code and unfortunately +* there is no clean way out of that either, locking all devices in the +* group and then do the re-attach will introduce a lock-inversion with +* group->mutex - Joerg. Do you mean that we can't do below?
mutex_lock(&group->mutex);for_each_group_device()device_lock(dev);/* Default domain switch */
for_each_group_device()
device_unlock()mutex_unlock(&group->mutex)Quoted 7 lines
Quoted 2 lines
I think, Joerg talks about two issues here 1.
is_driver_bound()takes/releasesdevice_lock()which could race with device driver probing codei.e. in an other terminal user could try to unload/load the module.
So you need to
device_lock()all devices during the whole process. (I assume that load/unload device driver requiring this lock as well. Didn't check it in code. Please correct me if I misunderstood it.)Ok.. I will check this.
Quoted 7 lines
Quoted 1 line
2. We cannot do as you suggested above because the functions (one is
mentioned below) that switch default domain need to take
iommu_group_mutexlock. So, taking the mutex and then calling iommu functions that switch default domain will dead lock.You probably need to move the
mutex_lock()in that function out to the caller? And only assert the lock has been held there.I mean, there are many functions that are taking
iommu_group_mutex_lockand they are being called as part of changing default domain.
So, we would then need to change all those functions and let callers of those functions know that now it's their responsibility to take the lock and call these functions. Some of those functions are:
Quoted 5 lines
iommu_group_add_deviceiommu_group_remove_deviceiommu_group_for_each_devSo, rather than that, I think it's better to delegate taking/releasing lock to individual functions than giving that responsibility to callers because then some callers might forget to take/release lock.
Okay. Fair enough.
Quoted 84 lines
Quoted 84 lines
Quoted 78 lines
Quoted 78 lines
Quoted 42 lines
drivers/iommu/iommu.c +9
-0+*/ +if (iommu_group_for_each_dev (group,NULL , is_driver_bound)) { +pr_err ("Active drivers exist for devices in the group\n" ); +return -EBUSY ; + } + +mutex_lock (&group->mutex); + prv_dom = group->default_domain; + if (!prv_dom || !prv_dom->ops || !prv_dom->ops-
dev_def_domain_type) {drivers/iommu/iommu.c +30
-0+pr_err ("'dev_def_domain_type' call back isn't registered\n" ); + ret = -EPERM ; + goto out; + } + +/* +* Check if any user level driver (that doesn't use kernel driver like +* VFIO) is directly using the group. VFIO use case is detected by +* is_driver_bound() check above +*/ +if (prv_dom != group->domain) { +pr_err ("Group assigned to user level for direct access\n" ); + ret = -EBUSY ; + goto out; + } + +/* +* If user has requested "auto", kernel has to decide the default domain +* type of a group. Hence, find out individual preferences of a device. +*/ + ops = prv_dom->ops; +if (req_auto) { +int dma_devs =0 , idt_devs =0 , dev_def_dom; + +list_for_each_entry (grp_dev, &group->devices, list) { + dev = grp_dev->dev; + dev_def_dom = ops->dev_def_domain_type (dev); +if (dev_def_dom ==IOMMU_DOMAIN_IDENTITY ) + idt_devs++; +if (dev_def_dom ==IOMMU_DOMAIN_DMA )drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle)
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ dma_devs++; + } + +/* +* Default domain type of a group is undefined if some devices +* in the group should be in dma domain while other devices +* should be in identity domain +*/ +if (idt_devs && dma_devs) { + pr_err("Some devices need identity domain while otherneed dma domain\n");
Quoted 13 lines
drivers/iommu/iommu.c +13
-0+ ret = -EPERM ; + goto out; + } + +/* +* Default domain type of a group is identity if +* 1. All the devices in the group need to be in identity domain +* 2. Some devices should be in identity domain while other +* devices could be in either of dma or identity domain +*/ +if (idt_devs && !dma_devs) + req_type =IOMMU_DOMAIN_IDENTITY ; +drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2845
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+/* +* Default domain type of a group is dma if +* 1. All the devices in the group need to be in dma domain +* 2. Some devices should be in dma domain while other devices +* could be in either of dma or identity domain +* 3. All the devices could be in either of the domains (namely +* dma and identity) +*/ +if (!idt_devs) + req_type =IOMMU_DOMAIN_DMA ;Actually, There are four combinations:
Quoted 4 lines
Quoted 3 lines
idt_devs==0 idt_devs!=0
dma_devs== 0iommu_def_domain_typeidentitydma_devs!= 0 DMA unsupportedYes.. you are right. All these four combinations are presently handled in the
code.
Quoted 8 lines
Quoted 2 lines
At least I didn't see the
iommu_def_domain_typeis used if bothidt_devsanddma_devsare both equal to 0. :-)Sorry! I didn't get what you meant by "
iommu_def_domain_type", so, could you please explain that?When
idt_devs== 0 anddma_devs== 0, it means that all the devices in the group could be in either of the domain. Hence, I default to DMA domain which is covered by this code.if (!idt_devs) req_type = IOMMU_DOMAIN_DMA;
iommu_def_domain_type is the system level default domain type defined in iommu.c. If the vendor iommu driver has no special requirement, we should choose the default value.
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
Quoted 28 lines · Sai Praneeth
Quoted 14 lines
Quoted 14 lines · Sai Praneeth
Quoted 12 lines
Quoted 11 lines
drivers/iommu/iommu.c int iommu_sva_get_pasid(struct iommu_sva *handle) · line 2721
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+list_for_each_entry_safe (grp_dev, temp, &group->devices, list) { +struct device *dev; + + dev = grp_dev->dev; +iommu_release_device (dev); + + ret =iommu_group_add_device (group, dev); +if (ret) +dev_err (dev,"Failed to add to iommu group %d: %d\n" , + group->id, ret);Need to handle this error case.
I wasn't sure on how to handle the error ☹
Quoted 8 lines
Quoted 6 lines
Just roll back to the state before calling this function and return an appropriate error value.
The likely behavior is detaching the new domains from all devices (if it has already attached), attaching the old domains to all devices in the group,
And while handling this error case, there is a possibility that attaching to old
domain could fail.. so, we might need to handle that error case as well. If we plan to handle error case, since we have removed devices from group above, adding them back could fail too.. that would lead into handling error case for an error case.
Quoted 5 lines
We can assume that the old domain should always be attached back. Otherwise, there must be some bugs in the vendor iommu driver.
It must be able to role back, otherwise users have to reboot the system in order to use the devices in the group. This is not acceptable in the production kernel.
I agree that we should be able to roll back but I am afraid that the error handling code could become complex than the usual code that gets to run. For example, assume there are 'n' devices in the group, 'k' of them are successfully processed (i.e. default domain type has been changed) and if k+1 fails while changing default domain type, to roll back state of k devices, we need to maintain a list of processed devices so that we now roll back state for devices in this list. The present code does not have any list because it's processing sequentially, it takes a device from the group, changes it domain and moves to other device and hence does not require a list.
All said, I could give this a try and see how complex the code could turn out to be. I hope I am wrong (i.e. turns out implementing error handling is simple).
Quoted 17 lines
Quoted 1 line
So, given the probability of these functions failing here are very low, I think,
why not bite the bullet and say, add code to handle these error cases if we see that these functions are failing frequently? Otherwise the error handling code is just a dead code.
Quoted 13 lines
Quoted 12 lines
cleaning
up all new resources allocated in this function, putting a error message to tell the user why it fails and returning an error code.Quoted 2 lines
i.e. group's domain/default_domain are already updated to new domain and
assume there are 'n' devices in the group and this failed for 'k'th device, I wasn't sure how I could roll back the changes made for k-1 devices.
A successful attach could be checked by (
group->domain==group->default_domain).
Quoted 7 lines
Quoted 4 lines
No.. because I have manually set
group->domain==group->default_domain=new_domain(did this becauseiommu_group_add_device()andiommu_group_create_direct_mappings()needs them)You could set
group->domainto the default domain only after it has been attached to the device successfully, right?
Will reorder things and see how this could be handled.
Quoted 11 lines
Quoted 10 lines
So, probably we might need some other way to check successful attach.
Quoted 8 lines
Quoted 2 lines
So, I thought probably just alert the user that there was an error while
changing default domain type and try updating for other devices in the group (hopefully other devices might succeed). Also,*generally* we shouldn't see any errors here because all these devices were already in the same group earlier (we aren’t adding/removing new devices to the group). We are just changing default domain type and we already made sure that device could be in the requested default domain
type.
[snipped]
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
At least I didn't see the
iommu_def_domain_typeis used if bothidt_devsanddma_devsare both equal to 0. :-)Sorry! I didn't get what you meant by "
iommu_def_domain_type", so, couldyou please explain that?
Quoted 2 lines
When
idt_devs== 0 anddma_devs== 0, it means that all the devices in the
Quoted 8 lines
group could be in either of the domain. Hence, I default to DMA domain which is covered by this code.
Quoted 2 lines
if (!idt_devs) req_type = IOMMU_DOMAIN_DMA;
iommu_def_domain_typeis the system level default domain type defined iniommu.c. If the vendor iommu driver has no special requirement, we should choose the default value.
Ok.. got it, thanks for explaining. I will change this as suggested.
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
Quoted 26 lines · Sai Praneeth
Quoted 26 lines · Sai Praneeth
Quoted 12 lines
Quoted 12 lines · Sai Praneeth
Quoted 11 lines
Quoted 11 lines
Review
Quoted 10 lines
(quoted diff) +10
-0+ list_for_each_entry_safe(grp_dev, temp, &group->devices, list) { + struct device *dev; + + dev = grp_dev->dev; + iommu_release_device(dev); + + ret = iommu_group_add_device(group, dev); + if (ret) + dev_err(dev, "Failed to add to iommu group %d: %d\n", + group->id, ret);Need to handle this error case.
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
I wasn't sure on how to handle the error ☹
Just roll back to the state before calling this function and return an appropriate error value.
The likely behavior is detaching the new domains from all devices (if it has already attached), attaching the old domains to all devices in the group,
And while handling this error case, there is a possibility that attaching to old
domain could fail.. so, we might need to handle that error case as well. If we plan to handle error case, since we have removed devices from group above, adding them back could fail too.. that would lead into handling error case for an error case.
Quoted 9 lines
Quoted 5 lines
We can assume that the old domain should always be attached back. Otherwise, there must be some bugs in the vendor iommu driver.
It must be able to role back, otherwise users have to reboot the system in order to use the devices in the group. This is not acceptable in the production kernel.
I agree that we should be able to roll back but I am afraid that the error handling code could become complex than the usual code that gets to run. For example, assume there are 'n' devices in the group, 'k' of them are successfully processed (i.e. default domain type has been changed) and if k+1 fails while changing default domain type, to roll back state of k devices, we need to maintain a list of processed devices so that we now roll back state for devices in this list. The present code does not have any list because it's processing sequentially, it takes a device from the group, changes it domain and moves to other device and hence does not require a list.
All said, I could give this a try and see how complex the code could turn out to be. I hope I am wrong (i.e. turns out implementing error handling is simple).
I think something like below should work.
static int iommu_group_do_attach_device(struct device *dev, void *data)
{
struct iommu_domain *domain = data;
return __iommu_attach_device(domain, dev);
}static int __iommu_attach_group(struct iommu_domain *domain,
struct iommu_group *group)
{
int ret;
if (group->default_domain && group->domain !=
group->default_domain)
return -EBUSY;
ret = __iommu_group_for_each_dev(group, domain,
iommu_group_do_attach_device);
if (ret == 0)
group->domain = domain;
return ret;
}The vendor iommu driver should always deprecate the old domain if it exists. Add a comment there.
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
Quoted 61 lines · Lu Baolu
Quoted 27 lines · Sai Praneeth
Quoted 27 lines · Sai Praneeth
Quoted 13 lines
Quoted 13 lines · Sai Praneeth
Quoted 12 lines
Quoted 12 lines
Review
Quoted 11 lines
+
list_for_each_entry_safe(grp_dev, temp, &group->devices, list){
+ struct device *dev;(quoted diff) +8
-0+ + dev = grp_dev->dev; + iommu_release_device(dev); + + ret = iommu_group_add_device(group, dev); + if (ret) + dev_err(dev, "Failed to add to iommu group %d: %d\n", + group->id, ret);Need to handle this error case.
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
I wasn't sure on how to handle the error ☹
Just roll back to the state before calling this function and return an appropriate error value.
The likely behavior is detaching the new domains from all devices (if it has already attached), attaching the old domains to all devices in the group,
And while handling this error case, there is a possibility that attaching to old
domain could fail.. so, we might need to handle that error case as well. If we plan to handle error case, since we have removed devices from group above, adding them back could fail too.. that would lead into handling error case for an error case.
Quoted 9 lines
Quoted 5 lines
We can assume that the old domain should always be attached back. Otherwise, there must be some bugs in the vendor iommu driver.
It must be able to role back, otherwise users have to reboot the system in order to use the devices in the group. This is not acceptable in the production kernel.
I agree that we should be able to roll back but I am afraid that the error handling code could become complex than the usual code that gets to run. For example, assume there are 'n' devices in the group, 'k' of them are successfully processed (i.e. default domain type has been changed) and if k+1 fails while changing default domain type, to roll back state of k devices, we need to maintain a list of processed devices so that we now roll back state for devices in this list. The present code does not have any list because it's processing sequentially, it takes a device from the group, changes it domain and moves to other device and hence does not require a list.
All said, I could give this a try and see how complex the code could turn out to be. I hope I am wrong (i.e. turns out implementing error handling is simple).
I think something like below should work.
static int
iommu_group_do_attach_device(struct device *dev, void *data){
structiommu_domain*domain = data;return
__iommu_attach_device(domain, dev);
}static int __iommu_attach_group(struct
iommu_domain*domain,
structiommu_group*group)
{
int ret;if (
group->default_domain&&group->domain!=group->default_domain)
return-EBUSY;ret =
__iommu_group_for_each_dev(group, domain, iommu_group_do_attach_device); if (ret == 0)
group->domain= domain;
Quoted 6 lines
return ret;
}The vendor iommu driver should always deprecate the old domain if it exists. Add a comment there.
By the way, this is the expected behavior. Please check __iommu_detach_group().
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
Quoted 79 lines · Lu Baolu
Quoted 78 lines · Sai Praneeth
Quoted 47 lines
Quoted 29 lines · Sai Praneeth
Quoted 15 lines
Quoted 15 lines · Sai Praneeth
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
+
list_for_each_entry_safe(grp_dev, temp, &group->devices, list){
+ struct device *dev;
+
+ dev =grp_dev->dev;Quoted 8 lines
Quoted 7 lines
+
iommu_release_device(dev);
+
+ ret =iommu_group_add_device(group, dev); + if (ret)
+ dev_err(dev, "Failed to add to iommu group %d:
+%d\n",
+group->id, ret);Need to handle this error case.
I wasn't sure on how to handle the error ☹
Quoted 8 lines
Quoted 6 lines
Just roll back to the state before calling this function and return an appropriate error value.
The likely behavior is detaching the new domains from all devices (if it has already attached), attaching the old domains to all devices in the group,
And while handling this error case, there is a possibility that attaching to old
domain could fail.. so, we might need to handle that error case as well. If we plan to handle error case, since we have removed devices from group above, adding them back could fail too.. that would lead into handling error case for an error case.
Quoted 6 lines
We can assume that the old domain should always be attached back. Otherwise, there must be some bugs in the vendor iommu driver.
It must be able to role back, otherwise users have to reboot the system in order to use the devices in the group. This is not acceptable in the production kernel.
I agree that we should be able to roll back but I am afraid that the error handling code could become complex than the usual code that gets to run. For example, assume there are 'n' devices in the group, 'k' of them are successfully processed (i.e. default domain type has been
changed) and if k+1 fails while changing default domain type, to roll back state of k devices, we need to maintain a list of processed devices so that we now roll back state for devices in this list. The present code does not have any list because it's processing sequentially, it takes a device from the group, changes it domain and moves to other device and hence does not require a list.Quoted 4 lines
All said, I could give this a try and see how complex the code could turn out to be. I hope I am wrong (i.e. turns out implementing error handling is simple).
I think something like below should work.
static int
iommu_group_do_attach_device(struct device *dev, void *data){
structiommu_domain*domain = data;return
__iommu_attach_device(domain, dev); }static int __iommu_attach_group(struct
iommu_domain*domain,
structiommu_group*group) { int ret;if (
group->default_domain&&group->domain!=group->default_domain)
return-EBUSY;ret = __iommu_group_for_each_dev(group, domain,
iommu_group_do_attach_device);
if (ret == 0)
group->domain= domain;return ret;
}
Quoted 6 lines
Quoted 3 lines
The vendor iommu driver should always deprecate the old domain if it exists. Add a comment there.
By the way, this is the expected behavior. Please check
__iommu_detach_group().
Ok.. I will look into it.
Regards,
Sai
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Joerg Roedel
maintainerRe: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
In reply to Sai Praneeth PrakhyaHello Sai, Baolu,
Quoted 5 lines · Sai Praneeth Prakhya
Hence it will be helpful if there is some way to change the default domain of a B:D.F dynamically. Since, linux iommu subsystem prefers to deal at
iommu_grouplevel instead of B:D.F level, it might be helpful if there is some way to change the default domain of a *iommu_group* dynamically. Hence, add such support.
The question is how this plays together with the per-device private domains in the Intel VT-d driver. I recently debugged an issue there and I think there are more. The overall code for this seems to be pretty fragile, so I had the idea to make the private default domains more general.
IOMMU default domains don't necessarily need to stick to the iommu-group granularity, because the default domain is used by in-kernel drivers only, and the kernel trusts itself.
So my idea was to make the private-domain concept of the VT-d driver more generic and move it to the iommu core code. With that we can configure real per-device default domain types and don't have the race condition with driver probing when changing the default domain of multiple devices. We have to limit the ability to change default domain types to devices with no PCI aliases, but that should not be a problem for the intended use-case.
What do you think?
Regards,
Joerg
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
Hi Joerg,
Quoted 27 lines · Joerg Roedel
Hello Sai, Baolu,
Quoted 5 lines · Sai Praneeth Prakhya
Hence it will be helpful if there is some way to change the default domain of a B:D.F dynamically. Since, linux iommu subsystem prefers to deal at
iommu_grouplevel instead of B:D.F level, it might be helpful if there is some way to change the default domain of a *iommu_group* dynamically. Hence, add such support.The question is how this plays together with the per-device private domains in the Intel VT-d driver. I recently debugged an issue there and I think there are more. The overall code for this seems to be pretty fragile, so I had the idea to make the private default domains more general.
IOMMU default domains don't necessarily need to stick to the iommu-group granularity, because the default domain is used by in-kernel drivers only, and the kernel trusts itself.
So my idea was to make the private-domain concept of the VT-d driver more generic and move it to the iommu core code. With that we can configure real per-device default domain types and don't have the race condition with driver probing when changing the default domain of multiple devices. We have to limit the ability to change default domain types to devices with no PCI aliases, but that should not be a problem for the intended use-case.
Quoted 2 lines
What do you think?
Theoretically speaking, per-device default domain is impractical. PCI aliased devices (PCI bridge and all devices beneath it, VMD devices and various devices quirked with pci_add_dma_alias()) must use the same domain. It's likely that we have to introduce something like a sub-group with all PCI aliased devices staying in it. Current private-domain implementation in the vt-d driver was introduced for compatible purpose and I wanted to abandon it from the first day. :-)
On Intel platforms, there are only rare devices which require a specific default domain: some graphic devices (identity), a specific model of AZALIA (identity) and external devices connected through thunderbolt (dma). They are not supposed to belong to a same group. Hence, if we are able to configure per-group default domain type, we don't need to keep private domain anymore.
Probably, we are able to configure per-group default domain type with below two interfaces.
- (ops->)dev_def_domain_type: Return the required default domain type for a device. It returns
- IOMMU_DOMAIN_DMA (device must use a DMA domain), unlikely
- IOMMU_DOMAIN_IDENTITY (device must use an Identity domain), unlikely
- 0 (both are okay), likely
- iommu_group_change_def_domain: Change the default domain of a group Works only when all devices have no driver bond.
[Sai's patch set has already included these two interfaces.]
In iommu_probe_device(),
dev_def_type = ops->dev_def_domain_type(dev)
if (dev_def_type && dev_def_type != group->default_domain->type) {
ret = iommu_group_change_def_domain(...)
if (ret)
return -EINVAL;
}This should work during boot since iommu_probe_device() always happens before device driver binding. We need to further consider the hot-plug cases.
- Hardware initiated device hotplug
We should always use DMA domain for devices connected through an external port to avoid DMA attacking from malicious devices. And such devices shouldn't share a group with internal (trusted) devices. Hence, I can't see any problems here.
- Software initiated device hotplug
The default domain type won't change before and after device hotplug so there's no problem as well.
This is what I have for the private domain in vt-d driver. Just for discussion.
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Joerg Roedel
maintainerRe: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
Hi Baolu,
Quoted 7 lines · Lu Baolu
Theoretically speaking, per-device default domain is impractical. PCI aliased devices (PCI bridge and all devices beneath it, VMD devices and various devices quirked with
pci_add_dma_alias()) must use the same domain. It's likely that we have to introduce something like a sub-group with all PCI aliased devices staying in it. Current private-domain implementation in the vt-d driver was introduced for compatible purpose and I wanted to abandon it from the first day. :-)
What hinders you from removing it now? I looked a bit closer into these private default domain implementation and it is very fragile. If it can be removed, then better do so sooner than later.
Quoted 8 lines
Probably, we are able to configure per-group default domain type with below two interfaces.
- (ops->)dev_def_domain_type: Return the required default domain type for a device. It returns
-IOMMU_DOMAIN_DMA(device must use a DMA domain), unlikely
-IOMMU_DOMAIN_IDENTITY(device must use an Identity domain), unlikely
- 0 (both are okay), likely
If we stay at the group level, this interface should work on the group level too, and not on the device level.
Quoted 2 lines
-
iommu_group_change_def_domain: Change the default domain of a group Works only when all devices have no driver bond.
Btw, I have no objections about the concept of private default domains for devices, but the implementation should be moved to generic IOMMU code so that the behavior is consistent accross differnet IOMMU platforms, and of course be robust.
Regards,
Joerg
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V2 3/5] iommu: Add support to change default domain of an iommu_group
Hi Joerg,
Quoted 11 lines · Joerg Roedel
Hi Baolu,
Quoted 7 lines · Lu Baolu
Theoretically speaking, per-device default domain is impractical. PCI aliased devices (PCI bridge and all devices beneath it, VMD devices and various devices quirked with
pci_add_dma_alias()) must use the same domain. It's likely that we have to introduce something like a sub-group with all PCI aliased devices staying in it. Current private-domain implementation in the vt-d driver was introduced for compatible purpose and I wanted to abandon it from the first day. :-)
Quoted 4 lines
What hinders you from removing it now? I looked a bit closer into these private default domain implementation and it is very fragile. If it can be removed, then better do so sooner than later.
I will soon send out the patches for review.
Quoted 12 lines
Quoted 8 lines
Probably, we are able to configure per-group default domain type with below two interfaces.
- (ops->)dev_def_domain_type: Return the required default domain type for a device. It returns
-IOMMU_DOMAIN_DMA(device must use a DMA domain), unlikely
-IOMMU_DOMAIN_IDENTITY(device must use an Identity domain), unlikely
- 0 (both are okay), likelyIf we stay at the group level, this interface should work on the group level too, and not on the device level.
Quoted 8 lines
Quoted 2 lines
-
iommu_group_change_def_domain: Change the default domain of a group Works only when all devices have no driver bond.Btw, I have no objections about the concept of private default domains for devices, but the implementation should be moved to generic IOMMU code so that the behavior is consistent accross differnet IOMMU platforms, and of course be robust.
Yes. I agree with you.
Quoted 5 lines
Regards,
Joerg
Best regards,
baolu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu