Skip to contentSkip to conversation

fsl_pamu: Use 40-bits for addressing where appropriate

PATCH In review Ben Collins 3 participants 5 messages Apr 22, 2025 – Apr 23, 2025 IOMMU

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fsl_pamu: Use 40-bits for addressing where appropriate

On 64-bit QorIQ platforms like T4240, the CPU supports 40-bit addressing and it's safe to move resources to the upper bounds of the 1TiB limit to make room for > 64GiB of memory. The PAMU driver does not account for this, however.

Setup fsl,pamu driver to make use of the full 40-bit addressing space when configuring liodn's that may have been configured in this range. Specifically the e5500 and e6500 CPUs.

---
 drivers/iommu/fsl_pamu.c        | 5 +++--
 drivers/iommu/fsl_pamu.h        | 7 +++++++
 drivers/iommu/fsl_pamu_domain.c | 5 +++--
 3 files changed, 13 insertions(+), 4 deletions(-)
drivers/iommu/fsl_pamu.c +3 -2
index f37d3b0441318..ceb352f824010 100644
@@ -198,7 +198,7 @@ int pamu_config_ppaace(int liodn, u32 omi, u32 stashid, int prot)
 
 	/* window size is 2^(WSE+1) bytes */
 	set_bf(ppaace->addr_bitfields, PPAACE_AF_WSE,
-	       map_addrspace_size_to_wse(1ULL << 36));
+	       map_addrspace_size_to_wse(1ULL << PAMU_MAX_PHYS_BITS));
 
 	pamu_init_ppaace(ppaace);
 
@@ -475,7 +475,8 @@ static void setup_liodns(void)
 			ppaace = pamu_get_ppaace(liodn);
 			pamu_init_ppaace(ppaace);
 			/* window size is 2^(WSE+1) bytes */
-			set_bf(ppaace->addr_bitfields, PPAACE_AF_WSE, 35);
+			set_bf(ppaace->addr_bitfields, PPAACE_AF_WSE,
+			       (PAMU_MAX_PHYS_BITS - 1));
 			ppaace->wbah = 0;
 			set_bf(ppaace->addr_bitfields, PPAACE_AF_WBAL, 0);
 			set_bf(ppaace->impl_attr, PAACE_IA_ATM,
drivers/iommu/fsl_pamu.h +7 -0
index 36df7975ff64d..5d88871610cfd 100644
@@ -42,6 +42,13 @@ struct pamu_mmap_regs {
 	u32 olal;
 };
 
+/* Physical addressing capability */
+#if defined(CONFIG_E6500_CPU) || defined(CONFIG_E5500_CPU)
+#define PAMU_MAX_PHYS_BITS	40
+#else
+#define PAMU_MAX_PHYS_BITS	36
+#endif
+
 /* PAMU Error Registers */
 #define PAMU_POES1 0x0040
 #define PAMU_POES2 0x0044
drivers/iommu/fsl_pamu_domain.c +3 -2
index 30be786bff11e..a4bc6482a00f7 100644
@@ -214,9 +214,10 @@ static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type)
 	INIT_LIST_HEAD(&dma_domain->devices);
 	spin_lock_init(&dma_domain->domain_lock);
 
-	/* default geometry 64 GB i.e. maximum system address */
+	/* Set default geometry based on physical address limit. */
 	dma_domain->iommu_domain. geometry.aperture_start = 0;
-	dma_domain->iommu_domain.geometry.aperture_end = (1ULL << 36) - 1;
+	dma_domain->iommu_domain.geometry.aperture_end =
+		(1ULL << PAMU_MAX_PHYS_BITS) - 1;
 	dma_domain->iommu_domain.geometry.force_aperture = true;
 
 	return &dma_domain->iommu_domain;
Signature · 9 lines
2.49.0


-- 
 Ben Collins
 https://libjwt.io
 https://github.com/benmcollins
 --
 3EC9 7598 1672 961A 1139  173A 5D5A 57C7 242B 22CF
#1 lore raw
Quoted 5 lines · Ben Collins
drivers/iommu/fsl_pamu_domain.c +0 -0
index 30be786bff11e..a4bc6482a00f7 100644

drivers/iommu/fsl_pamu_domain.c static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type) · line 221

Quoted 10 lines
drivers/iommu/fsl_pamu_domain.c +3 -2
@@ -214,9 +214,10 @@ static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type)
 	INIT_LIST_HEAD(&dma_domain->devices);
 	spin_lock_init(&dma_domain->domain_lock);
 
-	/* default geometry 64 GB i.e. maximum system address */
+	/* Set default geometry based on physical address limit. */
 	dma_domain->iommu_domain. geometry.aperture_start = 0;
-	dma_domain->iommu_domain.geometry.aperture_end = (1ULL << 36) - 1;
+	dma_domain->iommu_domain.geometry.aperture_end =
+		(1ULL << PAMU_MAX_PHYS_BITS) - 1;
 	dma_domain->iommu_domain.geometry.force_aperture = true;

What on earth does this even do? There is no map_range() callback in this driver, so nothing should be reading geometry..

Jason

#2 lore raw
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Ben Collins
drivers/iommu/fsl_pamu_domain.c +0 -0
index 30be786bff11e..a4bc6482a00f7 100644
@@ -214,9 +214,10 @@ static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type)
 	INIT_LIST_HEAD(&dma_domain->devices);
 	spin_lock_init(&dma_domain->domain_lock);
 
Quoted 10 lines

drivers/iommu/fsl_pamu_domain.c static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type) · line 221

Quoted 7 lines
drivers/iommu/fsl_pamu_domain.c +3 -2
-	/* default geometry 64 GB i.e. maximum system address */
+	/* Set default geometry based on physical address limit. */
 	dma_domain->iommu_domain. geometry.aperture_start = 0;
-	dma_domain->iommu_domain.geometry.aperture_end = (1ULL << 36) - 1;
+	dma_domain->iommu_domain.geometry.aperture_end =
+		(1ULL << PAMU_MAX_PHYS_BITS) - 1;
 	dma_domain->iommu_domain.geometry.force_aperture = true;

What on earth does this even do? There is no map_range() callback in this driver, so nothing should be reading geometry..

I dunno, but your "FIXME this is broken" comments are all over it from a year and a half ago:

Author: Jason Gunthorpe <[email protected]>
Date:   Wed Sep 13 10:43:38 2023 -0300

iommu/fsl_pamu: Implement a PLATFORM domain

/*
* FIXME: This isn't creating an unmanaged domain since the
* default_domain_ops do not have any map/unmap function it doesn't meet
* the requirements for __IOMMU_DOMAIN_PAGING. The only purpose seems to
* allow drivers/soc/fsl/qbman/qman_portal.c to do
* fsl_pamu_configure_l1_stash()
*/

The logic hasn't really been touched in 10 years.

Signature · 5 lines
 Ben Collins
 https://libjwt.io
 https://github.com/benmcollins
 --
 3EC9 7598 1672 961A 1139  173A 5D5A 57C7 242B 22CF
#3 lore raw
Quoted 21 lines · Ben Collins
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Ben Collins
drivers/iommu/fsl_pamu_domain.c +0 -0
index 30be786bff11e..a4bc6482a00f7 100644
@@ -214,9 +214,10 @@ static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type)
 	INIT_LIST_HEAD(&dma_domain->devices);
 	spin_lock_init(&dma_domain->domain_lock);
 
Quoted 10 lines

drivers/iommu/fsl_pamu_domain.c static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type) · line 221

Quoted 7 lines
drivers/iommu/fsl_pamu_domain.c +3 -2
-	/* default geometry 64 GB i.e. maximum system address */
+	/* Set default geometry based on physical address limit. */
 	dma_domain->iommu_domain. geometry.aperture_start = 0;
-	dma_domain->iommu_domain.geometry.aperture_end = (1ULL << 36) - 1;
+	dma_domain->iommu_domain.geometry.aperture_end =
+		(1ULL << PAMU_MAX_PHYS_BITS) - 1;
 	dma_domain->iommu_domain.geometry.force_aperture = true;

What on earth does this even do? There is no map_range() callback in this driver, so nothing should be reading geometry..

Quoted 2 lines

I dunno, but your "FIXME this is broken" comments are all over it from a year and a half ago:

Yes, I know, but you are changing this - are you changing it because something is broken without making this change, if so what, or are you changing it because it looked like it needed changing?

Quoted 1 line

The logic hasn't really been touched in 10 years.

Yeah, so I'm surprised someone still cares about it :)

Jason

#4 lore raw
Quoted 25 lines · Jason Gunthorpe
Quoted 21 lines · Ben Collins
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Ben Collins
drivers/iommu/fsl_pamu_domain.c +0 -0
index 30be786bff11e..a4bc6482a00f7 100644
@@ -214,9 +214,10 @@ static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type)
 	INIT_LIST_HEAD(&dma_domain->devices);
 	spin_lock_init(&dma_domain->domain_lock);
 
Quoted 10 lines

drivers/iommu/fsl_pamu_domain.c static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type) · line 221

Quoted 7 lines
drivers/iommu/fsl_pamu_domain.c +3 -2
-	/* default geometry 64 GB i.e. maximum system address */
+	/* Set default geometry based on physical address limit. */
 	dma_domain->iommu_domain. geometry.aperture_start = 0;
-	dma_domain->iommu_domain.geometry.aperture_end = (1ULL << 36) - 1;
+	dma_domain->iommu_domain.geometry.aperture_end =
+		(1ULL << PAMU_MAX_PHYS_BITS) - 1;
 	dma_domain->iommu_domain.geometry.force_aperture = true;

What on earth does this even do? There is no map_range() callback in this driver, so nothing should be reading geometry..

Quoted 2 lines

I dunno, but your "FIXME this is broken" comments are all over it from a year and a half ago:

Quoted 7 lines

Yes, I know, but you are changing this - are you changing it because something is broken without making this change, if so what, or are you changing it because it looked like it needed changing?

Quoted 1 line

The logic hasn't really been touched in 10 years.

Yeah, so I'm surprised someone still cares about it :)

Ironically, this patch sat collecting dust for 10 years until recently when I revived my T4240 system :)

The change is mostly to be "correct" in as much as the code can be correct when it's a little broken. Does it fix anything? It does. PAMU gets a little miffed about my liodn tags being up near the 1TiB boundary.

If it makes you feel any better about it, I've added fsl_pamu to the list of things I'm fixing for this board, for the hell of it. I have a pex8724 to program to get 2 of the 4 nvme slots up, a DPAA driver to build to get the RIO up (and likely some tweaking to program the IDT SRIO switch to work across the fabric).

IOW, I think I can maybe help get rid of your FIXMEs. For reference:

https://www.manualslib.com/manual/1061198/Vvdn-T4mfcs-Scaleout.html

Plus I have a couple P4080 systems for regression testing with a 36-bit addressing space.

Signature · 5 lines
 Ben Collins
 https://libjwt.io
 https://github.com/benmcollins
 --
 3EC9 7598 1672 961A 1139  173A 5D5A 57C7 242B 22CF