Skip to contentSkip to conversation

iommu: amd: Set page size bitmap during V2 domain allocation

PATCH Applied Jerry Snitselaar 4 participants 5 messages Apr 4, 2023 – Apr 13, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 iommu: amd: Set page size bitmap during V2 domain allocation

With the addition of the V2 page table support, the domain page size bitmap needs to be set prior to iommu core setting up direct mappings for reserved regions. When reserved regions are mapped, if this is not done, it will be looking at the V1 page size bitmap when determining the page size to use in iommu_pgsize(). When it gets into the actual amd mapping code, a check of see if the page size is supported can fail, because at that point it is checking it against the V2 page size bitmap which only supports 4K, 2M, and 1G.

Add a check to __iommu_domain_alloc() to not override the bitmap if it was already set by the iommu ops domain_alloc() code path.

Cc Vasant Hegde <[email protected]>
Cc Suravee Suthikulpanit <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Will Deacon <[email protected]>
Cc Joerg Roedel <[email protected]>
Fixes 4db6c41f0946 ("iommu/amd: Add support for using AMD IOMMU v2 page table for DMA-API")
---
 drivers/iommu/amd/iommu.c | 6 ++----
 drivers/iommu/iommu.c     | 9 +++++++--
 2 files changed, 9 insertions(+), 6 deletions(-)
drivers/iommu/amd/iommu.c +2 -4
index 5a505ba5467e..167da5b1a5e3 100644
@@ -1666,10 +1666,6 @@ static void do_attach(struct iommu_dev_data *dev_data,
 	domain->dev_iommu[iommu->index] += 1;
 	domain->dev_cnt                 += 1;
 
-	/* Override supported page sizes */
-	if (domain->flags & PD_GIOV_MASK)
-		domain->domain.pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
-
 	/* Update device table */
 	set_dte_entry(iommu, dev_data->devid, domain,
 		      ats, dev_data->iommu_v2);
@@ -2048,6 +2044,8 @@ static int protection_domain_init_v2(struct protection_domain *domain)
 
 	domain->flags |= PD_GIOV_MASK;
 
+	domain->domain.pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
+
 	if (domain_enable_v2(domain, 1)) {
 		domain_id_free(domain->id);
 		return -ENOMEM;
drivers/iommu/iommu.c +7 -2
index 10db680acaed..256a38371120 100644
@@ -1964,8 +1964,13 @@ static struct iommu_domain *__iommu_domain_alloc(struct bus_type *bus,
 		return NULL;
 
 	domain->type = type;
-	/* Assume all sizes by default; the driver may override this later */
-	domain->pgsize_bitmap = bus->iommu_ops->pgsize_bitmap;
+	/*
+	 * If not already set, assume all sizes by default; the driver
+	 * may override this later
+	 */
+	if (!domain->pgsize_bitmap)
+		domain->pgsize_bitmap = bus->iommu_ops->pgsize_bitmap;
+
 	if (!domain->ops)
 		domain->ops = bus->iommu_ops->default_domain_ops;
 
Signature · 1 line
2.38.1
#1 lore raw
Quoted 12 lines · Jerry Snitselaar

With the addition of the V2 page table support, the domain page size bitmap needs to be set prior to iommu core setting up direct mappings for reserved regions. When reserved regions are mapped, if this is not done, it will be looking at the V1 page size bitmap when determining the page size to use in iommu_pgsize(). When it gets into the actual amd mapping code, a check of see if the page size is supported can fail, because at that point it is checking it against the V2 page size bitmap which only supports 4K, 2M, and 1G.

Add a check to __iommu_domain_alloc() to not override the bitmap if it was already set by the iommu ops domain_alloc() code path.

Quoted 7 lines
Cc Vasant Hegde <[email protected]>
Cc Suravee Suthikulpanit <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Will Deacon <[email protected]>
Cc Joerg Roedel <[email protected]>
Fixes 4db6c41f0946 ("iommu/amd: Add support for using AMD IOMMU v2 page table for DMA-API")

I'm still not sure this is the best solution. Feels odd with adding a check to core code to handle something one of the drivers has done. Another thought was something like arm does, with amd_iommu_ops dropping the const and setting the default page size bitmap in iommu_init_pci(), but I think that would still require adding something in the protection domain/init code to deal with it forcing v1, so it would still require a check in the core code.

Would adding an op make more sense, with a generic op doing what the core code currently does for setting the default? Or am I overthinking this?

snits

Quoted 52 lines
---
 drivers/iommu/amd/iommu.c | 6 ++----
 drivers/iommu/iommu.c     | 9 +++++++--
 2 files changed, 9 insertions(+), 6 deletions(-)
drivers/iommu/amd/iommu.c +2 -4
index 5a505ba5467e..167da5b1a5e3 100644
@@ -1666,10 +1666,6 @@ static void do_attach(struct iommu_dev_data *dev_data,
 	domain->dev_iommu[iommu->index] += 1;
 	domain->dev_cnt                 += 1;
 
-	/* Override supported page sizes */
-	if (domain->flags & PD_GIOV_MASK)
-		domain->domain.pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
-
 	/* Update device table */
 	set_dte_entry(iommu, dev_data->devid, domain,
 		      ats, dev_data->iommu_v2);
@@ -2048,6 +2044,8 @@ static int protection_domain_init_v2(struct protection_domain *domain)
 
 	domain->flags |= PD_GIOV_MASK;
 
+	domain->domain.pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
+
 	if (domain_enable_v2(domain, 1)) {
 		domain_id_free(domain->id);
 		return -ENOMEM;
drivers/iommu/iommu.c +7 -2
index 10db680acaed..256a38371120 100644
@@ -1964,8 +1964,13 @@ static struct iommu_domain *__iommu_domain_alloc(struct bus_type *bus,
 		return NULL;
 
 	domain->type = type;
-	/* Assume all sizes by default; the driver may override this later */
-	domain->pgsize_bitmap = bus->iommu_ops->pgsize_bitmap;
+	/*
+	 * If not already set, assume all sizes by default; the driver
+	 * may override this later
+	 */
+	if (!domain->pgsize_bitmap)
+		domain->pgsize_bitmap = bus->iommu_ops->pgsize_bitmap;
+
 	if (!domain->ops)
 		domain->ops = bus->iommu_ops->default_domain_ops;
 
Signature · 1 line
2.38.1
maintainer Reviewed-by
#2 lore raw

Jerry,

Quoted 23 lines · Jerry Snitselaar
Quoted 12 lines · Jerry Snitselaar

With the addition of the V2 page table support, the domain page size bitmap needs to be set prior to iommu core setting up direct mappings for reserved regions. When reserved regions are mapped, if this is not done, it will be looking at the V1 page size bitmap when determining the page size to use in iommu_pgsize(). When it gets into the actual amd mapping code, a check of see if the page size is supported can fail, because at that point it is checking it against the V2 page size bitmap which only supports 4K, 2M, and 1G.

Add a check to __iommu_domain_alloc() to not override the bitmap if it was already set by the iommu ops domain_alloc() code path.

Quoted 7 lines
Cc Vasant Hegde <[email protected]>
Cc Suravee Suthikulpanit <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Will Deacon <[email protected]>
Cc Joerg Roedel <[email protected]>
Fixes 4db6c41f0946 ("iommu/amd: Add support for using AMD IOMMU v2 page table for DMA-API")
Quoted 8 lines

I'm still not sure this is the best solution. Feels odd with adding a check to core code to handle something one of the drivers has done. Another thought was something like arm does, with amd_iommu_ops dropping the const and setting the default page size bitmap in iommu_init_pci(), but I think that would still require adding something in the protection domain/init code to deal with it forcing v1, so it would still require a check in the core code.

I am not familiar with how arm paging works. But in AMD case we decide the page table during protection domain allocation. Also we can have different page size for different domains. (like one domain in v1 and another in v2 page mode).

Quoted 3 lines

Would adding an op make more sense, with a generic op doing what the core code currently does for setting the default? Or am I overthinking this?

IMO adding another ops is not required. I think current fix is good enough. Thanks for debugging/fixing this issue.

-Vasant

Quoted 56 lines

snits

Quoted 52 lines
---
 drivers/iommu/amd/iommu.c | 6 ++----
 drivers/iommu/iommu.c     | 9 +++++++--
 2 files changed, 9 insertions(+), 6 deletions(-)
drivers/iommu/amd/iommu.c +2 -4
index 5a505ba5467e..167da5b1a5e3 100644
@@ -1666,10 +1666,6 @@ static void do_attach(struct iommu_dev_data *dev_data,
 	domain->dev_iommu[iommu->index] += 1;
 	domain->dev_cnt                 += 1;
 
-	/* Override supported page sizes */
-	if (domain->flags & PD_GIOV_MASK)
-		domain->domain.pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
-
 	/* Update device table */
 	set_dte_entry(iommu, dev_data->devid, domain,
 		      ats, dev_data->iommu_v2);
@@ -2048,6 +2044,8 @@ static int protection_domain_init_v2(struct protection_domain *domain)
 
 	domain->flags |= PD_GIOV_MASK;
 
+	domain->domain.pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
+
 	if (domain_enable_v2(domain, 1)) {
 		domain_id_free(domain->id);
 		return -ENOMEM;
drivers/iommu/iommu.c +7 -2
index 10db680acaed..256a38371120 100644
@@ -1964,8 +1964,13 @@ static struct iommu_domain *__iommu_domain_alloc(struct bus_type *bus,
 		return NULL;
 
 	domain->type = type;
-	/* Assume all sizes by default; the driver may override this later */
-	domain->pgsize_bitmap = bus->iommu_ops->pgsize_bitmap;
+	/*
+	 * If not already set, assume all sizes by default; the driver
+	 * may override this later
+	 */
+	if (!domain->pgsize_bitmap)
+		domain->pgsize_bitmap = bus->iommu_ops->pgsize_bitmap;
+
 	if (!domain->ops)
 		domain->ops = bus->iommu_ops->default_domain_ops;
 
Signature · 1 line
2.38.1
reviewer
#3 lore raw
In reply to Jerry Snitselaar
Quoted 31 lines · Jerry Snitselaar
Quoted 12 lines · Jerry Snitselaar

With the addition of the V2 page table support, the domain page size bitmap needs to be set prior to iommu core setting up direct mappings for reserved regions. When reserved regions are mapped, if this is not done, it will be looking at the V1 page size bitmap when determining the page size to use in iommu_pgsize(). When it gets into the actual amd mapping code, a check of see if the page size is supported can fail, because at that point it is checking it against the V2 page size bitmap which only supports 4K, 2M, and 1G.

Add a check to __iommu_domain_alloc() to not override the bitmap if it was already set by the iommu ops domain_alloc() code path.

Quoted 7 lines
Cc Vasant Hegde <[email protected]>
Cc Suravee Suthikulpanit <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Will Deacon <[email protected]>
Cc Joerg Roedel <[email protected]>
Fixes 4db6c41f0946 ("iommu/amd: Add support for using AMD IOMMU v2 page table for DMA-API")

I'm still not sure this is the best solution. Feels odd with adding a check to core code to handle something one of the drivers has done. Another thought was something like arm does, with amd_iommu_ops dropping the const and setting the default page size bitmap in iommu_init_pci(), but I think that would still require adding something in the protection domain/init code to deal with it forcing v1, so it would still require a check in the core code.

Quoted 3 lines

Would adding an op make more sense, with a generic op doing what the core code currently does for setting the default? Or am I overthinking this?

I think this is fine as-is. TBH it's probably high time ops->pgsize_bitmap finished going away entirely - as the Arm SMMU bodges prove, it's no longer a good fit for the multi-instance model which we've evolved, and for the sake of only this one assignment now, it really doesn't seem worthwhile any more. Since I'm still reworking domain_alloc, and already have plans to clean up the horrible default_domain_ops thing (honestly, what possessed me to think that was a good idea at the time!?), I can throw pgsize_bitmap in there as well.

Cheers,
Robin.

Quoted 56 lines

snits

Quoted 52 lines
---
  drivers/iommu/amd/iommu.c | 6 ++----
  drivers/iommu/iommu.c     | 9 +++++++--
  2 files changed, 9 insertions(+), 6 deletions(-)
drivers/iommu/amd/iommu.c +2 -4
index 5a505ba5467e..167da5b1a5e3 100644
@@ -1666,10 +1666,6 @@ static void do_attach(struct iommu_dev_data *dev_data,
  	domain->dev_iommu[iommu->index] += 1;
  	domain->dev_cnt                 += 1;
 
-	/* Override supported page sizes */
-	if (domain->flags & PD_GIOV_MASK)
-		domain->domain.pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
-
  	/* Update device table */
  	set_dte_entry(iommu, dev_data->devid, domain,
  		      ats, dev_data->iommu_v2);
@@ -2048,6 +2044,8 @@ static int protection_domain_init_v2(struct protection_domain *domain)
 
  	domain->flags |= PD_GIOV_MASK;
 
+	domain->domain.pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
+
  	if (domain_enable_v2(domain, 1)) {
  		domain_id_free(domain->id);
  		return -ENOMEM;
drivers/iommu/iommu.c +7 -2
index 10db680acaed..256a38371120 100644
@@ -1964,8 +1964,13 @@ static struct iommu_domain *__iommu_domain_alloc(struct bus_type *bus,
  		return NULL;
 
  	domain->type = type;
-	/* Assume all sizes by default; the driver may override this later */
-	domain->pgsize_bitmap = bus->iommu_ops->pgsize_bitmap;
+	/*
+	 * If not already set, assume all sizes by default; the driver
+	 * may override this later
+	 */
+	if (!domain->pgsize_bitmap)
+		domain->pgsize_bitmap = bus->iommu_ops->pgsize_bitmap;
+
  	if (!domain->ops)
  		domain->ops = bus->iommu_ops->default_domain_ops;
 
Signature · 1 line
2.38.1
maintainer Applied
#4 lore raw
In reply to Jerry Snitselaar
Quoted 19 lines · Jerry Snitselaar

With the addition of the V2 page table support, the domain page size bitmap needs to be set prior to iommu core setting up direct mappings for reserved regions. When reserved regions are mapped, if this is not done, it will be looking at the V1 page size bitmap when determining the page size to use in iommu_pgsize(). When it gets into the actual amd mapping code, a check of see if the page size is supported can fail, because at that point it is checking it against the V2 page size bitmap which only supports 4K, 2M, and 1G.

Add a check to __iommu_domain_alloc() to not override the bitmap if it was already set by the iommu ops domain_alloc() code path.

Cc Vasant Hegde <[email protected]>
Cc Suravee Suthikulpanit <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Will Deacon <[email protected]>
Cc Joerg Roedel <[email protected]>
Fixes 4db6c41f0946 ("iommu/amd: Add support for using AMD IOMMU v2 page table for DMA-API")
Quoted 4 lines
---
 drivers/iommu/amd/iommu.c | 6 ++----
 drivers/iommu/iommu.c     | 9 +++++++--
 2 files changed, 9 insertions(+), 6 deletions(-)

Applied, thanks.