Prevent RESV_DIRECT devices from user assignment
Subsystems (3)
Lu Baolu
maintainercover Prevent RESV_DIRECT devices from user assignment
These are follow-up patches on this discussion:
I just summarized the ideas and code into a real patch series. Please help to review and merge.
Best regards,
baolu
Lu Baolu (2):
iommu: Prevent RESV_DIRECT devices from blocking domains iommu/vt-d: Remove rmrr check in domain attaching device path
include/linux/iommu.h | 2 ++ drivers/iommu/intel/iommu.c | 58 ------------------------------------- drivers/iommu/iommu.c | 39 ++++++++++++++++++------- 3 files changed, 31 insertions(+), 68 deletions(-)
Signature · 1 line
2.34.1
Lu Baolu
maintainerPATCH 1/2 iommu: Prevent RESV_DIRECT devices from blocking domains
The IOMMU_RESV_DIRECT flag indicates that a memory region must be mapped 1:1 at all times. This means that the region must always be accessible to the device, even if the device is attached to a blocking domain. This is equal to saying that IOMMU_RESV_DIRECT flag prevents devices from being attached to blocking domains.
This also implies that devices that implement RESV_DIRECT regions will be prevented from being assigned to user space since taking the DMA ownership immediately switches to a blocking domain.
The rule of preventing devices with the IOMMU_RESV_DIRECT regions from being assigned to user space has existed in the Intel IOMMU driver for a long time. Now, this rule is being lifted up to a general core rule, as other architectures like AMD and ARM also have RMRR-like reserved regions. This has been discussed in the community mailing list and refer to below link for more details.
Other places using unmanaged domains for kernel DMA must follow the iommu_get_resv_regions() and setup IOMMU_RESV_DIRECT - we do not restrict them in the core code.
--- include/linux/iommu.h | 2 ++ drivers/iommu/iommu.c | 39 +++++++++++++++++++++++++++++---------- 2 files changed, 31 insertions(+), 10 deletions(-)
include/linux/iommu.h +2 -0
@@ -409,6 +409,7 @@ struct iommu_fault_param {* @priv: IOMMU Driver private data * @max_pasids: number of PASIDs this device can consume * @attach_deferred: the dma domain attachment is deferred +* @requires_direct: The driver requested IOMMU_RESV_DIRECT * * TODO: migrate other per device data pointers under iommu_dev_data, e.g. * struct iommu_group *iommu_group;
@@ -422,6 +423,7 @@ struct dev_iommu {void *priv;u32 max_pasids; u32 attach_deferred:1; + u32 requires_direct:1; }; int iommu_device_register(struct iommu_device *iommu,
drivers/iommu/iommu.c +29 -10
@@ -959,12 +959,7 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain,unsigned long pg_size;int ret =0 ; -if (!iommu_is_dma_domain (domain)) -return 0 ; - -BUG_ON (!domain->pgsize_bitmap); - - pg_size =1UL <<__ffs (domain->pgsize_bitmap); + pg_size = domain->pgsize_bitmap ?1UL <<__ffs (domain->pgsize_bitmap) :0 ;INIT_LIST_HEAD (&mappings);iommu_get_resv_regions (dev, &mappings);
@@ -974,13 +969,22 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain,dma_addr_t start, end, addr;size_t map_size =0 ; +if (entry->type ==IOMMU_RESV_DIRECT ) + dev->iommu->requires_direct =1 ; + +if ((entry->type !=IOMMU_RESV_DIRECT && + entry->type !=IOMMU_RESV_DIRECT_RELAXABLE ) || + !iommu_is_dma_domain (domain)) +continue ; + +if (WARN_ON_ONCE (!pg_size)) { + ret = -EINVAL ; + goto out; + } + start =ALIGN (entry->start, pg_size); end =ALIGN (entry->start + entry->length, pg_size); -if (entry->type !=IOMMU_RESV_DIRECT && - entry->type !=IOMMU_RESV_DIRECT_RELAXABLE ) -continue ; -for (addr = start; addr <= end; addr += pg_size) {phys_addr_t phys_addr;
@@ -2121,6 +2125,21 @@ static int __iommu_device_set_domain(struct iommu_group *group, {int ret; +/* +* If the driver has requested IOMMU_RESV_DIRECT then we cannot allow +* the blocking domain to be attached as it does not contain the +* required 1:1 mapping. This test effectively exclusive the device from +* being used with iommu_group_claim_dma_owner() which will block vfio +* and iommufd as well. +*/ +if (dev->iommu->requires_direct && + (new_domain->type ==IOMMU_DOMAIN_BLOCKED || + new_domain == group->blocking_domain)) { +dev_warn (dev, +"Firmware has requested this device have a 1:1 IOMMU mapping, rejecting configuring the device without a 1:1 mapping. Contact your platform vendor.\n" ); +return -EINVAL ; + } +if (dev->iommu->attach_deferred) {if (new_domain == group->default_domain)return 0 ;
Signature · 1 line
2.34.1
Re: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 57 lines · Lu Baolu
The
IOMMU_RESV_DIRECTflag indicates that a memory region must be mapped 1:1 at all times. This means that the region must always be accessible to the device, even if the device is attached to a blocking domain. This is equal to saying thatIOMMU_RESV_DIRECTflag prevents devices from being attached to blocking domains.This also implies that devices that implement
RESV_DIRECTregions will be prevented from being assigned to user space since taking the DMA ownership immediately switches to a blocking domain.The rule of preventing devices with the
IOMMU_RESV_DIRECTregions from being assigned to user space has existed in the Intel IOMMU driver for a long time. Now, this rule is being lifted up to a general core rule, as other architectures like AMD and ARM also have RMRR-like reserved regions. This has been discussed in the community mailing list and refer to below link for more details.Other places using unmanaged domains for kernel DMA must follow the
iommu_get_resv_regions()and setupIOMMU_RESV_DIRECT- we do not restrict them in the core code.Cc Robin Murphy <[email protected]>Cc Alex Williamson <[email protected]>Cc Kevin Tian <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>Link https://lore.kernel.org/linux-iommu/BN9PR11MB5276E84229B5BD952D78E9598C639@BN9PR11MB5276.namprd11.prod.outlook.com open hereSigned-off-by Lu Baolu <[email protected]>--- include/linux/iommu.h | 2 ++ drivers/iommu/iommu.c | 39 +++++++++++++++++++++++++++++---------- 2 files changed, 31 insertions(+), 10 deletions(-)include/linux/iommu.h +2
-0@@ -409,6 +409,7 @@ struct iommu_fault_param {* @priv: IOMMU Driver private data * @max_pasids: number of PASIDs this device can consume * @attach_deferred: the dma domain attachment is deferred +* @requires_direct: The driver requested IOMMU_RESV_DIRECT * * TODO: migrate other per device data pointers under iommu_dev_data, e.g. * struct iommu_group *iommu_group; @@ -422,6 +423,7 @@ struct dev_iommu {void *priv;u32 max_pasids; u32 attach_deferred:1; + u32 requires_direct:1; }; int iommu_device_register(struct iommu_device *iommu,drivers/iommu/iommu.c +0
-0
drivers/iommu/iommu.c static int iommu_create_device_direct_mappings(struct iommu_domain *domain, · line 962
Quoted 10 lines
drivers/iommu/iommu.c +1
-6@@ -959,12 +959,7 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain,unsigned long pg_size;int ret =0 ; -if (!iommu_is_dma_domain (domain)) -return 0 ; - -BUG_ON (!domain->pgsize_bitmap); - - pg_size =1UL <<__ffs (domain->pgsize_bitmap); + pg_size = domain->pgsize_bitmap ?1UL <<__ffs (domain->pgsize_bitmap) :0 ;
Would it be better to add the following check here? if (WARN_ON(!pg_size))
return -EINVAL;
Instead of checking latter in the loop as follows. if (WARN_ON_ONCE(!pg_size)) {
ret = -EINVAL;
goto out;
}
Thanks,
Jingqi
Quoted 52 lines
drivers/iommu/iommu.c +28
-4INIT_LIST_HEAD (&mappings);iommu_get_resv_regions (dev, &mappings);@@ -974,13 +969,22 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain,dma_addr_t start, end, addr;size_t map_size =0 ; +if (entry->type ==IOMMU_RESV_DIRECT ) + dev->iommu->requires_direct =1 ; + +if ((entry->type !=IOMMU_RESV_DIRECT && + entry->type !=IOMMU_RESV_DIRECT_RELAXABLE ) || + !iommu_is_dma_domain (domain)) +continue ; + +if (WARN_ON_ONCE (!pg_size)) { + ret = -EINVAL ; + goto out; + } + start =ALIGN (entry->start, pg_size); end =ALIGN (entry->start + entry->length, pg_size); -if (entry->type !=IOMMU_RESV_DIRECT && - entry->type !=IOMMU_RESV_DIRECT_RELAXABLE ) -continue ; -for (addr = start; addr <= end; addr += pg_size) {phys_addr_t phys_addr;@@ -2121,6 +2125,21 @@ static int __iommu_device_set_domain(struct iommu_group *group, {int ret; +/* +* If the driver has requested IOMMU_RESV_DIRECT then we cannot allow +* the blocking domain to be attached as it does not contain the +* required 1:1 mapping. This test effectively exclusive the device from +* being used with iommu_group_claim_dma_owner() which will block vfio +* and iommufd as well. +*/ +if (dev->iommu->requires_direct && + (new_domain->type ==IOMMU_DOMAIN_BLOCKED || + new_domain == group->blocking_domain)) { +dev_warn (dev, +"Firmware has requested this device have a 1:1 IOMMU mapping, rejecting configuring the device without a 1:1 mapping. Contact your platform vendor.\n" ); +return -EINVAL ; + } +if (dev->iommu->attach_deferred) {if (new_domain == group->default_domain)return 0 ;
Baolu Lu
maintainerRe: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 72 lines · Jingqi
Quoted 57 lines · Lu Baolu
The
IOMMU_RESV_DIRECTflag indicates that a memory region must be mapped 1:1 at all times. This means that the region must always be accessible to the device, even if the device is attached to a blocking domain. This is equal to saying thatIOMMU_RESV_DIRECTflag prevents devices from being attached to blocking domains.This also implies that devices that implement
RESV_DIRECTregions will be prevented from being assigned to user space since taking the DMA ownership immediately switches to a blocking domain.The rule of preventing devices with the
IOMMU_RESV_DIRECTregions from being assigned to user space has existed in the Intel IOMMU driver for a long time. Now, this rule is being lifted up to a general core rule, as other architectures like AMD and ARM also have RMRR-like reserved regions. This has been discussed in the community mailing list and refer to below link for more details.Other places using unmanaged domains for kernel DMA must follow the
iommu_get_resv_regions()and setupIOMMU_RESV_DIRECT- we do not restrict them in the core code.Cc Robin Murphy <[email protected]>Cc Alex Williamson <[email protected]>Cc Kevin Tian <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>Link https://lore.kernel.org/linux-iommu/BN9PR11MB5276E84229B5BD952D78E9598C639@BN9PR11MB5276.namprd11.prod.outlook.com open hereSigned-off-by Lu Baolu <[email protected]>---
include/linux/iommu.h| 2 ++
drivers/iommu/iommu.c| 39 +++++++++++++++++++++++++++++----------2 files changed, 31 insertions(+), 10 deletions(-)include/linux/iommu.h +2
-0@@ -409,6 +409,7 @@ struct iommu_fault_param { * @priv: IOMMU Driver private data * @max_pasids: number of PASIDs this device can consume * @attach_deferred: the dma domain attachment is deferred + * @requires_direct: The driver requested IOMMU_RESV_DIRECT * * TODO: migrate other per device data pointers under iommu_dev_data, e.g. * struct iommu_group *iommu_group;@@ -422,6 +423,7 @@ struct dev_iommu { void *priv; u32 max_pasids; u32 attach_deferred:1; + u32 requires_direct:1; }; int iommu_device_register(struct iommu_device *iommu,drivers/iommu/iommu.c +0
-0@@ -959,12 +959,7 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain,drivers/iommu/iommu.c static int · line 963
Quoted 10 lines
drivers/iommu/iommu.c +1
-6unsigned long pg_size; int ret = 0; - if (!iommu_is_dma_domain(domain)) - return 0; - - BUG_ON(!domain->pgsize_bitmap); - - pg_size = 1UL << __ffs(domain->pgsize_bitmap); + pg_size = domain->pgsize_bitmap ? 1UL <<__ffs (domain->pgsize_bitmap) :0 ;Would it be better to add the following check here?
if (WARN_ON(!pg_size))
return-EINVAL;
Quoted 5 lines
Instead of checking latter in the loop as follows.
if (WARN_ON_ONCE(!pg_size)) {
ret =-EINVAL;
goto out;
}
I am afraid no. Only the paging domains need a valid pg_size. That's the reason why I put it after the iommu_is_dma_domain() check. The previous code has the same behavior too.
Best regards,
baolu
RE: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 21 lines
From: Baolu Lu <[email protected]>
Sent: Tuesday, June 13, 2023 11:15 AMQuoted 11 lines · Jingqi
Quoted 6 lines · Lu Baolu
drivers/iommu/iommu.c +1
-4- - BUG_ON(!domain->pgsize_bitmap); - - pg_size = 1UL << __ffs(domain->pgsize_bitmap); + pg_size = domain->pgsize_bitmap ? 1UL <<
__ffs(domain->pgsize_bitmap): 0;Would it be better to add the following check here?
if (WARN_ON(!pg_size))
return-EINVAL;Quoted 5 lines
Instead of checking latter in the loop as follows.
if (WARN_ON_ONCE(!pg_size)) {
ret =-EINVAL;
goto out;
}
Quoted 4 lines
I am afraid no. Only the paging domains need a valid
pg_size. That's the reason why I put it after theiommu_is_dma_domain()check. The previous code has the same behavior too.
You could also add the dma domain check here. pg_size is static then it makes more sense to verify it once instead of in a loop.
Baolu Lu
maintainerRe: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 21 lines · Kevin
Quoted 21 lines
From: Baolu Lu <[email protected]>
Sent: Tuesday, June 13, 2023 11:15 AMQuoted 16 lines · Jingqi
Quoted 6 lines · Lu Baolu
(quoted diff) +1
-4- - BUG_ON(!domain->pgsize_bitmap); - - pg_size = 1UL << __ffs(domain->pgsize_bitmap); + pg_size = domain->pgsize_bitmap ? 1UL <<
__ffs(domain->pgsize_bitmap): 0;Would it be better to add the following check here?
if (WARN_ON(!pg_size))
return-EINVAL;Instead of checking latter in the loop as follows.
if (WARN_ON_ONCE(!pg_size)) {
ret =-EINVAL;
goto out;
}
Quoted 7 lines
Quoted 4 lines
I am afraid no. Only the paging domains need a valid
pg_size. That's the reason why I put it after theiommu_is_dma_domain()check. The previous code has the same behavior too.You could also add the dma domain check here.
pg_sizeis static then it makes more sense to verify it once instead of in a loop.
Agreed. Does below additional change make sense?
drivers/iommu/iommu.c +0 -0
@@ -962,6 +962,9 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain, pg_size = domain->pgsize_bitmap ? 1UL <<
__ffs(domain->pgsize_bitmap) : 0;
INIT_LIST_HEAD(&mappings);+ if (WARN_ON_ONCE((domain->type & __IOMMU_DOMAIN_PAGING) && !pg_size))
+ return -EINVAL;
+
iommu_get_resv_regions(dev, &mappings);
/* We need to consider overlapping regions for different devices */
@@ -977,11 +980,6 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain,
!iommu_is_dma_domain(domain))
continue;- if (WARN_ON_ONCE(!pg_size)) {
- ret = -EINVAL;
- goto out;
- }
-
start = ALIGN(entry->start, pg_size);
end = ALIGN(entry->start + entry->length, pg_size);Best regards,
baolu
RE: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 47 lines
From: Baolu Lu <[email protected]>
Sent: Tuesday, June 27, 2023 4:01 PMQuoted 15 lines · Kevin
Quoted 15 lines
From: Baolu Lu <[email protected]>
Sent: Tuesday, June 13, 2023 11:15 AMQuoted 11 lines · Jingqi
Quoted 6 lines · Lu Baolu
(quoted diff) +1
-4- - BUG_ON(!domain->pgsize_bitmap); - - pg_size = 1UL << __ffs(domain->pgsize_bitmap); + pg_size = domain->pgsize_bitmap ? 1UL <<
__ffs(domain->pgsize_bitmap): 0;Would it be better to add the following check here?
if (WARN_ON(!pg_size))
return-EINVAL;Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Instead of checking latter in the loop as follows.
if (WARN_ON_ONCE(!pg_size)) {
ret =-EINVAL;
goto out;
}I am afraid no. Only the paging domains need a valid
pg_size. That's the reason why I put it after theiommu_is_dma_domain()check. Theprevious
Quoted 5 lines
Quoted 2 lines
code has the same behavior too.
You could also add the dma domain check here.
pg_sizeis static then it makes more sense to verify it once instead of in a loop.Agreed. Does below additional change make sense?
drivers/iommu/iommu.c +1
-0@@ -962,6 +962,9 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain, pg_size = domain->pgsize_bitmap ?1UL <<__ffs (domain->pgsize_bitmap) :0 ;INIT_LIST_HEAD (&mappings); + if (WARN_ON_ONCE((domain->type & __IOMMU_DOMAIN_PAGING)
Quoted 3 lines
&&
!pg_size))
+ return-EINVAL;
what's the reason of not using iommu_is_dma_domain()? this is called in the probe path only for the default domain. Otherwise if you change like this then you also want to change the check in the loop later to be consistent.
Quoted 19 lines
drivers/iommu/iommu.c +1
-5+iommu_get_resv_regions (dev, &mappings);/* We need to consider overlapping regions for different devices */ @@ -977,11 +980,6 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain, !iommu_is_dma_domain(domain)) continue; -if (WARN_ON_ONCE (!pg_size)) { - ret = -EINVAL ; - goto out; - } - start =ALIGN (entry->start, pg_size); end =ALIGN (entry->start + entry->length, pg_size);Best regards,
baolu
Baolu Lu
maintainerRe: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 43 lines · Kevin
Quoted 43 lines
From: Baolu Lu<[email protected]>
Sent: Tuesday, June 27, 2023 4:01 PMQuoted 15 lines · Kevin
Quoted 15 lines
From: Baolu Lu<[email protected]>
Sent: Tuesday, June 13, 2023 11:15 AMQuoted 11 lines · Jingqi
Quoted 6 lines · Lu Baolu
(quoted diff) +1
-4- - BUG_ON(!domain->pgsize_bitmap); - - pg_size = 1UL << __ffs(domain->pgsize_bitmap); + pg_size = domain->pgsize_bitmap ? 1UL <<
__ffs(domain->pgsize_bitmap): 0;Would it be better to add the following check here?
if (WARN_ON(!pg_size))
return-EINVAL;Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Instead of checking latter in the loop as follows.
if (WARN_ON_ONCE(!pg_size)) { ret = -EINVAL; goto out; }I am afraid no. Only the paging domains need a valid
pg_size. That's the reason why I put it after theiommu_is_dma_domain()check. Theprevious
Quoted 4 lines
Quoted 2 lines
code has the same behavior too.
You could also add the dma domain check here.
pg_sizeis static then it makes more sense to verify it once instead of in a loop.Agreed. Does below additional change make sense?
drivers/iommu/iommu.c +0
-0@@ -962,6 +962,9 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain, pg_size = domain->pgsize_bitmap ?1UL <<__ffs (domain->pgsize_bitmap) :0 ;INIT_LIST_HEAD (&mappings);
Quoted 9 lines
Quoted 4 lines
drivers/iommu/iommu.c +1
-0+ if (WARN_ON_ONCE((domain->type & __IOMMU_DOMAIN_PAGING)&&
!pg_size))
+ return-EINVAL;what's the reason of not using
iommu_is_dma_domain()? this is called in the probe path only for the default domain. Otherwise if you change like this then you also want to change the check in the loop later to be consistent.
Yes. iommu_is_dma_domain() is better.
Best regards,
baolu
Re: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
In reply to Baolu Ludrivers/iommu/iommu.c static int iommu_create_device_direct_mappings(struct · line 969
Quoted 13 lines · Baolu Lu
drivers/iommu/iommu.c +2
-0@@ -962,6 +962,9 @@ static int iommu_create_device_direct_mappings(structiommu_domain *domain, pg_size = domain->pgsize_bitmap ?1UL <<__ffs (domain->pgsize_bitmap) :0 ;INIT_LIST_HEAD (&mappings); +if (WARN_ON_ONCE ((domain->type & __IOMMU_DOMAIN_PAGING) && !pg_size)) +return -EINVAL ;
Calling this function with an identity domain is expected, it must return 0.
Jason
Robin Murphy
reviewerRe: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
In reply to Lu BaoluQuoted 100 lines · Lu Baolu
The
IOMMU_RESV_DIRECTflag indicates that a memory region must be mapped 1:1 at all times. This means that the region must always be accessible to the device, even if the device is attached to a blocking domain. This is equal to saying thatIOMMU_RESV_DIRECTflag prevents devices from being attached to blocking domains.This also implies that devices that implement
RESV_DIRECTregions will be prevented from being assigned to user space since taking the DMA ownership immediately switches to a blocking domain.The rule of preventing devices with the
IOMMU_RESV_DIRECTregions from being assigned to user space has existed in the Intel IOMMU driver for a long time. Now, this rule is being lifted up to a general core rule, as other architectures like AMD and ARM also have RMRR-like reserved regions. This has been discussed in the community mailing list and refer to below link for more details.Other places using unmanaged domains for kernel DMA must follow the
iommu_get_resv_regions()and setupIOMMU_RESV_DIRECT- we do not restrict them in the core code.Cc Robin Murphy <[email protected]>Cc Alex Williamson <[email protected]>Cc Kevin Tian <[email protected]>Signed-off-by Jason Gunthorpe <[email protected]>Link https://lore.kernel.org/linux-iommu/BN9PR11MB5276E84229B5BD952D78E9598C639@BN9PR11MB5276.namprd11.prod.outlook.com open hereSigned-off-by Lu Baolu <[email protected]>--- include/linux/iommu.h | 2 ++ drivers/iommu/iommu.c | 39 +++++++++++++++++++++++++++++---------- 2 files changed, 31 insertions(+), 10 deletions(-)include/linux/iommu.h +2
-0@@ -409,6 +409,7 @@ struct iommu_fault_param {* @priv: IOMMU Driver private data * @max_pasids: number of PASIDs this device can consume * @attach_deferred: the dma domain attachment is deferred +* @requires_direct: The driver requested IOMMU_RESV_DIRECT * * TODO: migrate other per device data pointers under iommu_dev_data, e.g. * struct iommu_group *iommu_group; @@ -422,6 +423,7 @@ struct dev_iommu {void *priv;u32 max_pasids; u32 attach_deferred:1; + u32 requires_direct:1; }; int iommu_device_register(struct iommu_device *iommu,drivers/iommu/iommu.c +14
-10@@ -959,12 +959,7 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain,unsigned long pg_size;int ret =0 ; -if (!iommu_is_dma_domain (domain)) -return 0 ; - -BUG_ON (!domain->pgsize_bitmap); - - pg_size =1UL <<__ffs (domain->pgsize_bitmap); + pg_size = domain->pgsize_bitmap ?1UL <<__ffs (domain->pgsize_bitmap) :0 ;INIT_LIST_HEAD (&mappings);iommu_get_resv_regions (dev, &mappings);@@ -974,13 +969,22 @@ static int iommu_create_device_direct_mappings(struct iommu_domain *domain,dma_addr_t start, end, addr;size_t map_size =0 ; +if (entry->type ==IOMMU_RESV_DIRECT ) + dev->iommu->requires_direct =1 ; + +if ((entry->type !=IOMMU_RESV_DIRECT && + entry->type !=IOMMU_RESV_DIRECT_RELAXABLE ) || + !iommu_is_dma_domain (domain)) +continue ; + +if (WARN_ON_ONCE (!pg_size)) { + ret = -EINVAL ; + goto out; + } + start =ALIGN (entry->start, pg_size); end =ALIGN (entry->start + entry->length, pg_size); -if (entry->type !=IOMMU_RESV_DIRECT && - entry->type !=IOMMU_RESV_DIRECT_RELAXABLE ) -continue ; -for (addr = start; addr <= end; addr += pg_size) {phys_addr_t phys_addr;@@ -2121,6 +2125,21 @@ static int __iommu_device_set_domain(struct iommu_group *group, {int ret;
drivers/iommu/iommu.c static int __iommu_device_set_domain(struct iommu_group *group, · line 2136
Quoted 10 lines
drivers/iommu/iommu.c +9
-0+/* +* If the driver has requested IOMMU_RESV_DIRECT then we cannot allow +* the blocking domain to be attached as it does not contain the +* required 1:1 mapping. This test effectively exclusive the device from +* being used with iommu_group_claim_dma_owner() which will block vfio +* and iommufd as well. +*/ + if (dev->iommu->requires_direct && + (new_domain->type == IOMMU_DOMAIN_BLOCKED ||
Given the notion elsewhere that we want to use the blocking domain as a last resort to handle an attach failure, at face value it looks suspect that failing to attach to a blocking domain could also be a thing. I guess technically this is failing at a slightly different level so maybe it does work out OK, but it's still smelly.
The main thing, though, is that not everything implements the IOMMU_DOMAIN_BLOCKED optimisation, so a nominal blocking domain could be IOMMU_DOMAIN_UNMANAGED as well. FWIW I'd prefer to make the RESV_DIRECT check explicit in __iommu_take_dma_ownership() rather than hide it in an implementation detail; that's going to be a lot clearer to reason about as time goes on.
Thanks,
Robin.
Quoted 9 lines
drivers/iommu/iommu.c +6
-0+ new_domain == group->blocking_domain)) { +dev_warn (dev, +"Firmware has requested this device have a 1:1 IOMMU mapping, rejecting configuring the device without a 1:1 mapping. Contact your platform vendor.\n" ); +return -EINVAL ; + } +if (dev->iommu->attach_deferred) {if (new_domain == group->default_domain)return 0 ;
Re: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 7 lines · Robin Murphy
Quoted 7 lines
drivers/iommu/iommu.c +4
-0@@ -2121,6 +2125,21 @@ static int __iommu_device_set_domain(struct iommu_group *group, {int ret; +/* +* If the driver has requested IOMMU_RESV_DIRECT then we cannot allow +* the blocking domain to be attached as it does not contain the +* required 1:1 mapping. This test effectively exclusive the device from
Quoted 8 lines
drivers/iommu/iommu.c static int __iommu_device_set_domain(struct iommu_group *group, · line 2136
Quoted 5 lines
drivers/iommu/iommu.c +5
-0+* being used with iommu_group_claim_dma_owner() which will block vfio +* and iommufd as well. +*/ + if (dev->iommu->requires_direct && + (new_domain->type == IOMMU_DOMAIN_BLOCKED ||Given the notion elsewhere that we want to use the blocking domain as a last resort to handle an attach failure,
We shouldn't do that for cases where requires_direct is true, the last resort will have to be the static identity domain.
Quoted 4 lines
at face value it looks suspect that failing to attach to a blocking domain could also be a thing. I guess technically this is failing at a slightly different level so maybe it does work out OK, but it's still smelly.
It basically says that this driver doesn't support blocking domains on this device. What we don't want is for the driver to fail blocking or identity attaches.
Quoted 3 lines
The main thing, though, is that not everything implements the
IOMMU_DOMAIN_BLOCKEDoptimisation, so a nominal blocking domain could beIOMMU_DOMAIN_UNMANAGEDas well.
Yes, it should check new_domain == group->blocking_domain as well.
Quoted 4 lines
FWIW I'd prefer to make the
RESV_DIRECTcheck explicit in__iommu_take_dma_ownership()rather than hide it in an implementation detail; that's going to be a lot clearer to reason about as time goes on.
We want to completely forbid blocking domains at all on these devices because they are not supported (by FW request). I don't really like the idea that we go and assume the only users of blocking domains are also using take_dma_ownership() - that feels like a future bug waiting to happen.
Jason
Robin Murphy
reviewerRe: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 20 lines · Jason Gunthorpe
Quoted 7 lines · Robin Murphy
Quoted 7 lines
drivers/iommu/iommu.c +4
-0@@ -2121,6 +2125,21 @@ static int __iommu_device_set_domain(struct iommu_group *group, {int ret; +/* +* If the driver has requested IOMMU_RESV_DIRECT then we cannot allow +* the blocking domain to be attached as it does not contain the +* required 1:1 mapping. This test effectively exclusive the device from Quoted 8 lines
drivers/iommu/iommu.c static int __iommu_device_set_domain(struct iommu_group *group, · line 2136
Quoted 5 lines
drivers/iommu/iommu.c +5
-0+* being used with iommu_group_claim_dma_owner() which will block vfio +* and iommufd as well. +*/ + if (dev->iommu->requires_direct && + (new_domain->type == IOMMU_DOMAIN_BLOCKED ||Given the notion elsewhere that we want to use the blocking domain as a last resort to handle an attach failure,
We shouldn't do that for cases where
requires_directis true, the last resort will have to be the static identity domain.
Quoted 8 lines
Quoted 4 lines
at face value it looks suspect that failing to attach to a blocking domain could also be a thing. I guess technically this is failing at a slightly different level so maybe it does work out OK, but it's still smelly.
It basically says that this driver doesn't support blocking domains on this device. What we don't want is for the driver to fail blocking or identity attaches.
Is that really the relevant semantic though? I thought the point was to prevent userspace (or anyone else for that matter) taking ownership of a device with reserved regions which we can't trust them to honour. Not least because the series is entitled "Prevent RESV_DIRECT devices from user assignment", not anything about attaching to blocking domains. Plus the existing intel-iommu behaviour being generalised is specific to IOMMU_DOMAIN_UNMANAGED.
Quoted 11 lines
Quoted 3 lines
The main thing, though, is that not everything implements the
IOMMU_DOMAIN_BLOCKEDoptimisation, so a nominal blocking domain could beIOMMU_DOMAIN_UNMANAGEDas well.Yes, it should check
new_domain==group->blocking_domainas well.Quoted 4 lines
FWIW I'd prefer to make the
RESV_DIRECTcheck explicit in__iommu_take_dma_ownership()rather than hide it in an implementation detail; that's going to be a lot clearer to reason about as time goes on.
Quoted 5 lines
We want to completely forbid blocking domains at all on these devices because they are not supported (by FW request). I don't really like the idea that we go and assume the only users of blocking domains are also using
take_dma_ownership()- that feels like a future bug waiting to happen.
On reflection, I don't think that aspect actually matters anyway - nobody can explicitly request attachment to a blocking domain, so if the only time they're used is when the IOMMU driver has already had a catastrophic internal failure such that we decide to declare the device toasted and deliberately put it into an unusable state, blocking its reserved regions doesn't seem like a big deal. In fact if anything it kind of feels like the right thing to do for that situation. We're saying that we want the device to stop accessing memory because things might be in an inconsistent state which we can't trust; who says that mappings of RESV_DIRECT regions haven't also gone wonky? Having BLOCKED mean that the device truly cannot access - and thus potentially corrupt - *any* memory anywhere seems like the most robust and useful behaviour.
Thanks,
Robin.
Re: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 26 lines · Robin Murphy
Quoted 25 lines · Jason Gunthorpe
Quoted 7 lines · Robin Murphy
Quoted 7 lines
drivers/iommu/iommu.c +4
-0@@ -2121,6 +2125,21 @@ static int __iommu_device_set_domain(struct iommu_group *group, {int ret; +/* +* If the driver has requested IOMMU_RESV_DIRECT then we cannot allow +* the blocking domain to be attached as it does not contain the +* required 1:1 mapping. This test effectively exclusive the device from Quoted 8 lines
drivers/iommu/iommu.c static int __iommu_device_set_domain(struct iommu_group *group, · line 2136
Quoted 5 lines
drivers/iommu/iommu.c +5
-0+* being used with iommu_group_claim_dma_owner() which will block vfio +* and iommufd as well. +*/ + if (dev->iommu->requires_direct && + (new_domain->type == IOMMU_DOMAIN_BLOCKED ||Given the notion elsewhere that we want to use the blocking domain as a last resort to handle an attach failure,
We shouldn't do that for cases where
requires_directis true, the last resort will have to be the static identity domain.Quoted 4 lines
at face value it looks suspect that failing to attach to a blocking domain could also be a thing. I guess technically this is failing at a slightly different level so maybe it does work out OK, but it's still smelly.
Quoted 5 lines
Quoted 3 lines
It basically says that this driver doesn't support blocking domains on this device. What we don't want is for the driver to fail blocking or identity attaches.
Is that really the relevant semantic though?
It is the semantic I have had in mind. The end goal:
1) Driver never fails identity or blocking attachments
2) Identity and blocking domains are global static so always available
3) Core code puts restrictions on when identity and blocking domains can be used (eg trusted, reserved regions)
4) Catastrophic error recovery ends up moving to either blocking (preferred) or identity.
Quoted 3 lines
I thought the point was to prevent userspace (or anyone else for that matter) taking ownership of a device with reserved regions which we can't trust them to honour.
Yes, this is the practical side effect of enforcing the rules.
The other side effect is that it removes another special case where a driver has special behavior tied to IOMMU_DOMAIN_DMA.
Quoted 3 lines
assignment", not anything about attaching to blocking domains. Plus the existing intel-iommu behaviour being generalised is specific to
IOMMU_DOMAIN_UNMANAGED.
Being specific to UNMANAGED is a driver mistake, IMHO. It is a hack to make it only work with VFIO and avoid dma-iommu.
Quoted 2 lines
On reflection, I don't think that aspect actually matters anyway - nobody can explicitly request attachment to a blocking domain
They are used as part of the ownership mechanism, blocking domains are effectively explicitly requested by detaching domains from owned groups.
Yes, take_ownership is very carefully a sufficient place to put the check with today's design, but it feels wrong because the blocking domain compatability has conceptually nothing to do with ownership. If we use blocking domains in the future then the check will be in the wrong place.
Quoted 4 lines
so if the only time they're used is when the IOMMU driver has already had a catastrophic internal failure such that we decide to declare the device toasted and deliberately put it into an unusable state, blocking its reserved regions doesn't seem like a big deal.
I think we should discuss then when we get to actually implementing the error recovery flow we want. I do like blocking in general for the reasons you give, and that was my first plan.. But if the BIOS will crash or something if we don't do the reserved region maps that isn't so good either. IDK would like to hear from the people using this BIOS feature.
Thanks,
Jason
RE: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 10 lines
From: Jason Gunthorpe <[email protected]>
Sent: Monday, June 19, 2023 11:30 PMQuoted 4 lines · Robin Murphy
so if the only time they're used is when the IOMMU driver has already had a catastrophic internal failure such that we decide to declare the device toasted and deliberately put it into an unusable state, blocking its reserved regions doesn't seem like a big deal.
Quoted 7 lines
I think we should discuss then when we get to actually implementing the error recovery flow we want. I do like blocking in general for the reasons you give, and that was my first plan.. But if the BIOS will crash or something if we don't do the reserved region maps that isn't so good either. IDK would like to hear from the people using this BIOS feature.
The only devices with RMRR which I'm aware of on Intel platforms are GPU and USB. However they are all RESV_DIRECT_RELAXABLE type.
Here is one reference from the Xen hypervisor. It has a concept called quarantine domain (similar to blocking domain) when a device is de-assigned w/o an owner. The quarantine domain has no mappings except the ones identity-mapped for RMRR types. I'm not sure whether they observed real examples of RMRR devices which are not GPU/USB.
I guess the reason of not going fully identity or fully blocked is from the same worry that the BIOS may go insane while Xen still wants to quarantine the device as much as possible.
Re: [PATCH 1/2] iommu: Prevent RESV_DIRECT devices from blocking domains
Quoted 18 lines · Kevin
Quoted 10 lines
From: Jason Gunthorpe <[email protected]>
Sent: Monday, June 19, 2023 11:30 PMQuoted 4 lines · Robin Murphy
so if the only time they're used is when the IOMMU driver has already had a catastrophic internal failure such that we decide to declare the device toasted and deliberately put it into an unusable state, blocking its reserved regions doesn't seem like a big deal.
Quoted 7 lines
I think we should discuss then when we get to actually implementing the error recovery flow we want. I do like blocking in general for the reasons you give, and that was my first plan.. But if the BIOS will crash or something if we don't do the reserved region maps that isn't so good either. IDK would like to hear from the people using this BIOS feature.
Quoted 8 lines
The only devices with RMRR which I'm aware of on Intel platforms are GPU and USB. However they are all
RESV_DIRECT_RELAXABLEtype.Here is one reference from the Xen hypervisor. It has a concept called quarantine domain (similar to blocking domain) when a device is de-assigned w/o an owner. The quarantine domain has no mappings except the ones identity-mapped for RMRR types. I'm not sure whether they observed real examples of RMRR devices which are not GPU/USB.
I guess, it seems a bit hard core, but we could spend the cost to build such a domain during probe..
At least for our cases we already do expect that DMA is halted before we start messing with the domains so identity may not be the same issue as with Xen..
Jason
Lu Baolu
maintainerPATCH 2/2 iommu/vt-d: Remove rmrr check in domain attaching device path
In reply to Lu BaoluThe core code now prevents devices with RMRR regions from being assigned to user space. There is no need to check for this condition in individual drivers. Remove it to avoid duplicate code.
--- drivers/iommu/intel/iommu.c | 58 ------------------------------------- 1 file changed, 58 deletions(-)
drivers/iommu/intel/iommu.c +0 -58
@@ -2458,30 +2458,6 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,return 0 ; } -static bool device_has_rmrr (struct device *dev) -{ -struct dmar_rmrr_unit *rmrr; -struct device *tmp; -int i; - -rcu_read_lock (); -for_each_rmrr_units (rmrr) { -/* -* Return TRUE if this RMRR contains the device that -* is passed in. -*/ -for_each_active_dev_scope (rmrr->devices, - rmrr->devices_cnt, i, tmp) -if (tmp == dev || -is_downstream_to_pci_bridge (dev, tmp)) { -rcu_read_unlock (); -return true; - } - } -rcu_read_unlock (); -return false; -} -/** * device_rmrr_is_relaxable - Test whether the RMRR of this device * is relaxable (ie. is allowed to be not enforced under some conditions)
@@ -2511,34 +2487,6 @@ static bool device_rmrr_is_relaxable(struct device *dev)return false; } -/* -* There are a couple cases where we need to restrict the functionality of -* devices associated with RMRRs. The first is when evaluating a device for -* identity mapping because problems exist when devices are moved in and out -* of domains and their respective RMRR information is lost. This means that -* a device with associated RMRRs will never be in a "passthrough" domain. -* The second is use of the device through the IOMMU API. This interface -* expects to have full control of the IOVA space for the device. We cannot -* satisfy both the requirement that RMRR access is maintained and have an -* unencumbered IOVA space. We also have no ability to quiesce the device's -* use of the RMRR space or even inform the IOMMU API user of the restriction. -* We therefore prevent devices associated with an RMRR from participating in -* the IOMMU API, which eliminates them from device assignment. -* -* In both cases, devices which have relaxable RMRRs are not concerned by this -* restriction. See device_rmrr_is_relaxable comment. -*/ -static bool device_is_rmrr_locked (struct device *dev) -{ -if (!device_has_rmrr (dev)) -return false; - -if (device_rmrr_is_relaxable (dev)) -return false; - -return true; -} -/* * Return the required default domain type for a specific device. *
@@ -4146,12 +4094,6 @@ static int intel_iommu_attach_device(struct iommu_domain *domain,struct device_domain_info *info =dev_iommu_priv_get (dev);int ret; -if (domain->type ==IOMMU_DOMAIN_UNMANAGED && -device_is_rmrr_locked (dev)) { -dev_warn (dev,"Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.\n" ); -return -EPERM ; - } -if (info->domain)device_block_translation (dev);
Signature · 1 line
2.34.1
Re: [PATCH 2/2] iommu/vt-d: Remove rmrr check in domain attaching device path
Quoted 8 lines · Lu Baolu
The core code now prevents devices with RMRR regions from being assigned to user space. There is no need to check for this condition in individual drivers. Remove it to avoid duplicate code.
Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.c | 58 ------------------------------------- 1 file changed, 58 deletions(-)
I'm happy to see this leave drive code!
Jason