fsl_pamu: Use 40-bits for addressing where appropriate
PATCH 1/1 fsl_pamu: Use 40-bits for addressing where appropriate
On 64-bit QorIQ platforms like T4240, the CPU supports 40-bit addressing and it's safe to move resources to the upper bounds of the 1TiB limit to make room for > 64GiB of memory. The PAMU driver does not account for this, however.
Setup fsl,pamu driver to make use of the full 40-bit addressing space when configuring liodn's that may have been configured in this range. Specifically the e5500 and e6500 CPUs.
--- drivers/iommu/fsl_pamu.c | 5 +++-- drivers/iommu/fsl_pamu.h | 7 +++++++ drivers/iommu/fsl_pamu_domain.c | 5 +++-- 3 files changed, 13 insertions(+), 4 deletions(-)
drivers/iommu/fsl_pamu.c +3 -2
@@ -198,7 +198,7 @@ int pamu_config_ppaace(int liodn, u32 omi, u32 stashid, int prot)/* window size is 2^(WSE+1) bytes */ set_bf (ppaace->addr_bitfields,PPAACE_AF_WSE , -map_addrspace_size_to_wse (1ULL <<36 )); +map_addrspace_size_to_wse (1ULL <<PAMU_MAX_PHYS_BITS ));pamu_init_ppaace (ppaace);
@@ -475,7 +475,8 @@ static void setup_liodns(void) ppaace =pamu_get_ppaace (liodn);pamu_init_ppaace (ppaace);/* window size is 2^(WSE+1) bytes */ -set_bf (ppaace->addr_bitfields,PPAACE_AF_WSE ,35 ); +set_bf (ppaace->addr_bitfields,PPAACE_AF_WSE , + (PAMU_MAX_PHYS_BITS -1 )); ppaace->wbah =0 ;set_bf (ppaace->addr_bitfields,PPAACE_AF_WBAL ,0 ); set_bf(ppaace->impl_attr, PAACE_IA_ATM,
drivers/iommu/fsl_pamu.h +7 -0
@@ -42,6 +42,13 @@ struct pamu_mmap_regs {u32 olal; }; +/* Physical addressing capability */ +#if defined(CONFIG_E6500_CPU ) || defined(CONFIG_E5500_CPU ) +#define PAMU_MAX_PHYS_BITS 40 +#else +#define PAMU_MAX_PHYS_BITS 36 +#endif +/* PAMU Error Registers */ #define PAMU_POES1 0x0040#define PAMU_POES2 0x0044
drivers/iommu/fsl_pamu_domain.c +3 -2
@@ -214,9 +214,10 @@ static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type)INIT_LIST_HEAD (&dma_domain->devices);spin_lock_init (&dma_domain->domain_lock); -/* default geometry 64 GB i.e. maximum system address */ +/* Set default geometry based on physical address limit. */ dma_domain->iommu_domain. geometry.aperture_start =0 ; - dma_domain->iommu_domain.geometry.aperture_end = (1ULL <<36 ) -1 ; + dma_domain->iommu_domain.geometry.aperture_end = + (1ULL <<PAMU_MAX_PHYS_BITS ) -1 ; dma_domain->iommu_domain.geometry.force_aperture = true;return &dma_domain->iommu_domain;
Signature · 9 lines
2.49.0 -- Ben Collins https://libjwt.io https://github.com/benmcollins -- 3EC9 7598 1672 961A 1139 173A 5D5A 57C7 242B 22CF
Quoted 5 lines · Ben Collins
drivers/iommu/fsl_pamu_domain.c +0
-0
drivers/iommu/fsl_pamu_domain.c static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type) · line 221
Quoted 10 lines
drivers/iommu/fsl_pamu_domain.c +3
-2@@ -214,9 +214,10 @@ static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type)INIT_LIST_HEAD (&dma_domain->devices);spin_lock_init (&dma_domain->domain_lock); -/* default geometry 64 GB i.e. maximum system address */ +/* Set default geometry based on physical address limit. */ dma_domain->iommu_domain. geometry.aperture_start =0 ; - dma_domain->iommu_domain.geometry.aperture_end = (1ULL <<36 ) -1 ; + dma_domain->iommu_domain.geometry.aperture_end = + (1ULL <<PAMU_MAX_PHYS_BITS ) -1 ; dma_domain->iommu_domain.geometry.force_aperture = true;
What on earth does this even do? There is no map_range() callback in this driver, so nothing should be reading geometry..
Jason
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Ben Collins
drivers/iommu/fsl_pamu_domain.c +0
-0@@ -214,9 +214,10 @@ static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type)INIT_LIST_HEAD (&dma_domain->devices);spin_lock_init (&dma_domain->domain_lock);
Quoted 10 lines
drivers/iommu/fsl_pamu_domain.c static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type) · line 221
Quoted 7 lines
drivers/iommu/fsl_pamu_domain.c +3
-2-/* default geometry 64 GB i.e. maximum system address */ +/* Set default geometry based on physical address limit. */ dma_domain->iommu_domain. geometry.aperture_start =0 ; - dma_domain->iommu_domain.geometry.aperture_end = (1ULL <<36 ) -1 ; + dma_domain->iommu_domain.geometry.aperture_end = + (1ULL <<PAMU_MAX_PHYS_BITS ) -1 ; dma_domain->iommu_domain.geometry.force_aperture = true;What on earth does this even do? There is no
map_range()callback in this driver, so nothing should be reading geometry..
I dunno, but your "FIXME this is broken" comments are all over it from a year and a half ago:
Author: Jason Gunthorpe <[email protected]> Date: Wed Sep 13 10:43:38 2023 -0300
iommu/fsl_pamu: Implement a PLATFORM domain
/*
* FIXME: This isn't creating an unmanaged domain since the
* default_domain_ops do not have any map/unmap function it doesn't meet
* the requirements for __IOMMU_DOMAIN_PAGING. The only purpose seems to
* allow drivers/soc/fsl/qbman/qman_portal.c to do
* fsl_pamu_configure_l1_stash()
*/
The logic hasn't really been touched in 10 years.
Signature · 5 lines
Ben Collins https://libjwt.io https://github.com/benmcollins -- 3EC9 7598 1672 961A 1139 173A 5D5A 57C7 242B 22CF
Quoted 21 lines · Ben Collins
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Ben Collins
drivers/iommu/fsl_pamu_domain.c +0
-0@@ -214,9 +214,10 @@ static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type)INIT_LIST_HEAD (&dma_domain->devices);spin_lock_init (&dma_domain->domain_lock);Quoted 10 lines
drivers/iommu/fsl_pamu_domain.c static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type) · line 221
Quoted 7 lines
drivers/iommu/fsl_pamu_domain.c +3
-2-/* default geometry 64 GB i.e. maximum system address */ +/* Set default geometry based on physical address limit. */ dma_domain->iommu_domain. geometry.aperture_start =0 ; - dma_domain->iommu_domain.geometry.aperture_end = (1ULL <<36 ) -1 ; + dma_domain->iommu_domain.geometry.aperture_end = + (1ULL <<PAMU_MAX_PHYS_BITS ) -1 ; dma_domain->iommu_domain.geometry.force_aperture = true;What on earth does this even do? There is no
map_range()callback in this driver, so nothing should be reading geometry..
Quoted 2 lines
I dunno, but your "FIXME this is broken" comments are all over it from a year and a half ago:
Yes, I know, but you are changing this - are you changing it because something is broken without making this change, if so what, or are you changing it because it looked like it needed changing?
Quoted 1 line
The logic hasn't really been touched in 10 years.
Yeah, so I'm surprised someone still cares about it :)
Jason
Quoted 25 lines · Jason Gunthorpe
Quoted 21 lines · Ben Collins
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Ben Collins
drivers/iommu/fsl_pamu_domain.c +0
-0@@ -214,9 +214,10 @@ static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type)INIT_LIST_HEAD (&dma_domain->devices);spin_lock_init (&dma_domain->domain_lock);Quoted 10 lines
drivers/iommu/fsl_pamu_domain.c static struct iommu_domain *fsl_pamu_domain_alloc(unsigned type) · line 221
Quoted 7 lines
drivers/iommu/fsl_pamu_domain.c +3
-2-/* default geometry 64 GB i.e. maximum system address */ +/* Set default geometry based on physical address limit. */ dma_domain->iommu_domain. geometry.aperture_start =0 ; - dma_domain->iommu_domain.geometry.aperture_end = (1ULL <<36 ) -1 ; + dma_domain->iommu_domain.geometry.aperture_end = + (1ULL <<PAMU_MAX_PHYS_BITS ) -1 ; dma_domain->iommu_domain.geometry.force_aperture = true;What on earth does this even do? There is no
map_range()callback in this driver, so nothing should be reading geometry..Quoted 2 lines
I dunno, but your "FIXME this is broken" comments are all over it from a year and a half ago:
Quoted 7 lines
Yes, I know, but you are changing this - are you changing it because something is broken without making this change, if so what, or are you changing it because it looked like it needed changing?
Quoted 1 line
The logic hasn't really been touched in 10 years.
Yeah, so I'm surprised someone still cares about it :)
Ironically, this patch sat collecting dust for 10 years until recently when I revived my T4240 system :)
The change is mostly to be "correct" in as much as the code can be correct when it's a little broken. Does it fix anything? It does. PAMU gets a little miffed about my liodn tags being up near the 1TiB boundary.
If it makes you feel any better about it, I've added fsl_pamu to the list of things I'm fixing for this board, for the hell of it. I have a pex8724 to program to get 2 of the 4 nvme slots up, a DPAA driver to build to get the RIO up (and likely some tweaking to program the IDT SRIO switch to work across the fabric).
IOW, I think I can maybe help get rid of your FIXMEs. For reference:
https://www.manualslib.com/manual/1061198/Vvdn-T4mfcs-Scaleout.html
Plus I have a couple P4080 systems for regression testing with a 36-bit addressing space.
Signature · 5 lines
Ben Collins https://libjwt.io https://github.com/benmcollins -- 3EC9 7598 1672 961A 1139 173A 5D5A 57C7 242B 22CF