arm64: mm: Create gigabyte kernel logical mappings where possible
PATCH 1/1 arm64: mm: Create gigabyte kernel logical mappings where possible
We have the capability to map 1GB level 1 blocks when using a 4K granule.
This patch adjusts the create_mapping logic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries.
---
Hello,
This patch has been tested on the FastModel for 4K and 64K pages. Also, this has been tested with Jungseok's 4 level patch.
I put in the explicit check for PAGE_SHIFT, as I am anticipating a three level 64KB configuration at some point.
With two level 64K, a PUD is equivalent to a PMD which is equivalent to a PGD, and these are all level 2 descriptors.
Under three level 64K, a PUD would be equivalent to a PGD which would be a level 1 descriptor thus may not be a block.
Comments/critique/testers welcome.
Cheers,
Signature · 4 lines
Steve --- arch/arm64/mm/mmu.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
arch/arm64/mm/mmu.c +14 -1
@@ -234,7 +234,20 @@ static void __init alloc_init_pud(pgd_t *pgd, unsigned long addr, pud =pud_offset (pgd, addr);do { next =pud_addr_end (addr, end); -alloc_init_pmd (pud, addr, next, phys); + +/* +* For 4K granule only, attempt to put down a 1GB block +*/ +if ((PAGE_SHIFT ==12 ) && + ((addr | next | phys) & ~PUD_MASK ) ==0 ) { +pud_t old_pud = *pud; +set_pud (pud,__pud (phys | prot_sect_kernel)); + +if (!pud_none (old_pud)) +flush_tlb_all (); + }else { +alloc_init_pmd (pud, addr, next, phys); + } phys += next - addr; }while (pud++, addr = next, addr != end); }
Signature · 1 line
1.8.1.4
PATCH 1/1 arm64: mm: Create gigabyte kernel logical mappings where possible
Quoted 18 lines · On Wednesday 30 April 2014 12:36:22 Steve Capper wrote:
We have the capability to map 1GB level 1 blocks when using a 4K granule.
This patch adjusts the
create_mappinglogic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries.Signed-off-by Steve Capper <[email protected]>---
Hello,
This patch has been tested on the FastModel for 4K and 64K pages. Also, this has been tested with Jungseok's 4 level patch.I put in the explicit check for
PAGE_SHIFT, as I am anticipating a three level 64KB configuration at some point.
Quoted 7 lines
With two level 64K, a PUD is equivalent to a PMD which is equivalent to a PGD, and these are all level 2 descriptors.
Under three level 64K, a PUD would be equivalent to a PGD which would be a level 1 descriptor thus may not be a block.
Comments/critique/testers welcome.
It seems like a great idea. I have to admit that I don't understand the existing code, but what are the page sizes used here?
Does the code always use the largest possible page size, or does it just use either small pages or 1G pages?
In combination with the contiguous page hint, we should be able to theoretically support 4KB/64KB/2M/32M/1G/16G TLBs in any combination for boot-time mappings on a 4K page size kernel, or 64KB/1M/512M/8G on a 64KB page size kernel.
Arnd
PATCH 1/1 arm64: mm: Create gigabyte kernel logical mappings where possible
Quoted 22 lines · Arnd Bergmann
Quoted 21 lines · On Wednesday 30 April 2014 12:36:22 Steve Capper wrote:
We have the capability to map 1GB level 1 blocks when using a 4K granule.
This patch adjusts the
create_mappinglogic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries.Signed-off-by Steve Capper <[email protected]>---
Hello,
This patch has been tested on the FastModel for 4K and 64K pages. Also, this has been tested with Jungseok's 4 level patch.I put in the explicit check for
PAGE_SHIFT, as I am anticipating a three level 64KB configuration at some point.With two level 64K, a PUD is equivalent to a PMD which is equivalent to a PGD, and these are all level 2 descriptors.
Quoted 7 lines
Quoted 4 lines
Under three level 64K, a PUD would be equivalent to a PGD which would be a level 1 descriptor thus may not be a block.
Comments/critique/testers welcome.
It seems like a great idea. I have to admit that I don't understand the existing code, but what are the page sizes used here?
Actually, I think it was your idea ;-). I remember you talking about increasing the mapping size when 4-level page tables were being discussed. (I think I should have added a Reported-by, would be happy to if you want?).
With a 64KB granule, we'll map 512MB blocks if possible, otherwise 64K. And with a 4KB granule, the original code will map 2MB blocks if possible, and 4KB otherwise.
The patch will make the 4KB granule case also map 1GB blocks if possible.
Quoted 3 lines
Does the code always use the largest possible page size, or does it just use either small pages or 1G pages?
The code will put down the largest mappings it can. As the physical memory sizes/address are very likely to be aligned to whatever block size we use; we are likely to achieve the maximum size for our mappings.
Quoted 6 lines
In combination with the contiguous page hint, we should be able to theoretically support 4KB/64KB/2M/32M/1G/16G TLBs in any combination for boot-time mappings on a 4K page size kernel, or 64KB/1M/512M/8G on a 64KB page size kernel.
A contiguous hint could be applied to these mappings. The logic would be a bit more complicated though when we consider different granules. For 4KB we chain together 16 entries, for 64KB we use 32. If/when we adopt a 16KB granule, we use 32 entries for a level 2 lookup and 128 entries for a level 3 lookup...
The largest TLB entry sizes that I am aware of in play are the block sizes (i.e. 2MB, 512MB, 1GB). So I don't think we'll get any benefit at the moment for adding the contiguous logic.
Cheers,
Signature · 1 line
Steve
Quoted 1 line
Arnd
PATCH 1/1 arm64: mm: Create gigabyte kernel logical mappings where possible
Quoted 28 lines · On Thursday 01 May 2014 09:54:12 Steve Capper wrote:
Quoted 27 lines · Arnd Bergmann
Quoted 25 lines · On Wednesday 30 April 2014 12:36:22 Steve Capper wrote:
We have the capability to map 1GB level 1 blocks when using a 4K granule.
This patch adjusts the
create_mappinglogic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries.Signed-off-by Steve Capper <[email protected]>---
Hello,
This patch has been tested on the FastModel for 4K and 64K pages. Also, this has been tested with Jungseok's 4 level patch.I put in the explicit check for
PAGE_SHIFT, as I am anticipating a three level 64KB configuration at some point.With two level 64K, a PUD is equivalent to a PMD which is equivalent to a PGD, and these are all level 2 descriptors.
Under three level 64K, a PUD would be equivalent to a PGD which would be a level 1 descriptor thus may not be a block.
Comments/critique/testers welcome.
Quoted 7 lines
Quoted 2 lines
It seems like a great idea. I have to admit that I don't understand the existing code, but what are the page sizes used here?
Actually, I think it was your idea ;-). I remember you talking about increasing the mapping size when 4-level page tables were being discussed. (I think I should have added a Reported-by, would be happy to if you want?).
I completely forgot we had talked about this.
Quoted 6 lines
With a 64KB granule, we'll map 512MB blocks if possible, otherwise 64K. And with a 4KB granule, the original code will map 2MB blocks if possible, and 4KB otherwise.
The patch will make the 4KB granule case also map 1GB blocks if possible.
Ok.
Quoted 11 lines
Quoted 4 lines
In combination with the contiguous page hint, we should be able to theoretically support 4KB/64KB/2M/32M/1G/16G TLBs in any combination for boot-time mappings on a 4K page size kernel, or 64KB/1M/512M/8G on a 64KB page size kernel.
A contiguous hint could be applied to these mappings. The logic would be a bit more complicated though when we consider different granules. For 4KB we chain together 16 entries, for 64KB we use 32. If/when we adopt a 16KB granule, we use 32 entries for a level 2 lookup and 128 entries for a level 3 lookup...
Quoted 3 lines
The largest TLB entry sizes that I am aware of in play are the block sizes (i.e. 2MB, 512MB, 1GB). So I don't think we'll get any benefit at the moment for adding the contiguous logic.
Is that an architecture limit, or specific to the Cortex-A53/A57 implementations?
Arnd
PATCH 1/1 arm64: mm: Create gigabyte kernel logical mappings where possible
Quoted 59 lines · Arnd Bergmann
Quoted 28 lines · On Thursday 01 May 2014 09:54:12 Steve Capper wrote:
Quoted 27 lines · Arnd Bergmann
Quoted 25 lines · On Wednesday 30 April 2014 12:36:22 Steve Capper wrote:
We have the capability to map 1GB level 1 blocks when using a 4K granule.
This patch adjusts the
create_mappinglogic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries.Signed-off-by Steve Capper <[email protected]>---
Hello,
This patch has been tested on the FastModel for 4K and 64K pages. Also, this has been tested with Jungseok's 4 level patch.I put in the explicit check for
PAGE_SHIFT, as I am anticipating a three level 64KB configuration at some point.With two level 64K, a PUD is equivalent to a PMD which is equivalent to a PGD, and these are all level 2 descriptors.
Under three level 64K, a PUD would be equivalent to a PGD which would be a level 1 descriptor thus may not be a block.
Comments/critique/testers welcome.
Quoted 7 lines
Quoted 2 lines
It seems like a great idea. I have to admit that I don't understand the existing code, but what are the page sizes used here?
Actually, I think it was your idea ;-). I remember you talking about increasing the mapping size when 4-level page tables were being discussed. (I think I should have added a Reported-by, would be happy to if you want?).
I completely forgot we had talked about this.
Quoted 6 lines
With a 64KB granule, we'll map 512MB blocks if possible, otherwise 64K. And with a 4KB granule, the original code will map 2MB blocks if possible, and 4KB otherwise.
The patch will make the 4KB granule case also map 1GB blocks if possible.
Ok.
Quoted 11 lines
Quoted 4 lines
In combination with the contiguous page hint, we should be able to theoretically support 4KB/64KB/2M/32M/1G/16G TLBs in any combination for boot-time mappings on a 4K page size kernel, or 64KB/1M/512M/8G on a 64KB page size kernel.
A contiguous hint could be applied to these mappings. The logic would be a bit more complicated though when we consider different granules. For 4KB we chain together 16 entries, for 64KB we use 32. If/when we adopt a 16KB granule, we use 32 entries for a level 2 lookup and 128 entries for a level 3 lookup...
Quoted 6 lines
Quoted 3 lines
The largest TLB entry sizes that I am aware of in play are the block sizes (i.e. 2MB, 512MB, 1GB). So I don't think we'll get any benefit at the moment for adding the contiguous logic.
Is that an architecture limit, or specific to the Cortex-A53/A57 implementations?
Those are the TLBs that are documented for the Cortex-A53 and Cortex-A57. I have an idea of what the architectural limit is, but I will need to seek confirmation on it.
Cheers,
Signature · 1 line
Steve
Quoted 2 lines
Arnd
PATCH 1/1 arm64: mm: Create gigabyte kernel logical mappings where possible
In reply to Steve CapperQuoted 16 lines · Steve Capper
We have the capability to map 1GB level 1 blocks when using a 4K granule.
This patch adjusts the
create_mappinglogic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries.Signed-off-by Steve Capper <[email protected]>---
Hello,
This patch has been tested on the FastModel for 4K and 64K pages. Also, this has been tested with Jungseok's 4 level patch.I put in the explicit check for
PAGE_SHIFT, as I am anticipating a three level 64KB configuration at some point.
Quoted 7 lines
With two level 64K, a PUD is equivalent to a PMD which is equivalent to a PGD, and these are all level 2 descriptors.
Under three level 64K, a PUD would be equivalent to a PGD which would be a level 1 descriptor thus may not be a block.
Comments/critique/testers welcome.
Hi, Steve
I've tested on my platform, and it works well.
If SoC design follows "Principles of ARM Memory Maps", PA should be supposed to be 1GB aligned. Thus, I think this patch is effective against them.
Best Regards
Jungseok Lee
PATCH 1/1 arm64: mm: Create gigabyte kernel logical mappings where possible
Quoted 20 lines · Jungseok Lee
Quoted 19 lines · Steve Capper
We have the capability to map 1GB level 1 blocks when using a 4K granule.
This patch adjusts the
create_mappinglogic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries.Signed-off-by Steve Capper <[email protected]>---
Hello,
This patch has been tested on the FastModel for 4K and 64K pages. Also, this has been tested with Jungseok's 4 level patch.I put in the explicit check for
PAGE_SHIFT, as I am anticipating a three level 64KB configuration at some point.With two level 64K, a PUD is equivalent to a PMD which is equivalent to a PGD, and these are all level 2 descriptors.
Quoted 9 lines
Quoted 4 lines
Under three level 64K, a PUD would be equivalent to a PGD which would be a level 1 descriptor thus may not be a block.
Comments/critique/testers welcome.
Hi, Steve
I've tested on my platform, and it works well.
Thanks for giving this a go!
Quoted 7 lines
If SoC design follows "Principles of ARM Memory Maps", PA should be supposed to be 1GB aligned. Thus, I think this patch is effective against them.
Best Regards
Jungseok Lee
Catalin Marinas
maintainerPATCH 1/1 arm64: mm: Create gigabyte kernel logical mappings where possible
In reply to Steve CapperQuoted 10 lines · Steve Capper
arch/arm64/mm/mmu.c +1
-1@@ -234,7 +234,20 @@ static void __init alloc_init_pud(pgd_t *pgd, unsigned long addr, pud =pud_offset (pgd, addr); do { next = pud_addr_end(addr, end); - alloc_init_pmd(pud, addr, next, phys); +
arch/arm64/mm/mmu.c static void __init alloc_init_pud(pgd_t *pgd, unsigned long addr, · line 247
Quoted 10 lines
arch/arm64/mm/mmu.c +10
-0+/* +* For 4K granule only, attempt to put down a 1GB block +*/ +if ((PAGE_SHIFT ==12 ) && + ((addr | next | phys) & ~PUD_MASK ) ==0 ) { +pud_t old_pud = *pud; +set_pud (pud,__pud (phys | prot_sect_kernel)); + +if (!pud_none (old_pud)) +flush_tlb_all ();
We could even free the original pmd here. I think a memblock_free(pud_pfn(old_pud) << PAGE_SHIFT, PAGE_SIZE) should do (untested, and you need to define pud_pfn).
Signature · 1 line
Catalin
PATCH 1/1 arm64: mm: Create gigabyte kernel logical mappings where possible
Quoted 17 lines · Catalin Marinas
Quoted 16 lines · Steve Capper
arch/arm64/mm/mmu.c +7
-1@@ -234,7 +234,20 @@ static void __init alloc_init_pud(pgd_t *pgd, unsigned long addr, pud =pud_offset (pgd, addr); do { next = pud_addr_end(addr, end); - alloc_init_pmd(pud, addr, next, phys); + + /* + * For 4K granule only, attempt to put down a 1GB block + */ + if ((PAGE_SHIFT == 12) && + ((addr | next | phys) & ~PUD_MASK) == 0) { + pud_t old_pud = *pud;
Quoted 8 lines
arch/arm64/mm/mmu.c static void __init alloc_init_pud(pgd_t *pgd, unsigned long addr, · line 247
Quoted 4 lines
arch/arm64/mm/mmu.c +4
-0+set_pud (pud,__pud (phys | prot_sect_kernel)); + +if (!pud_none (old_pud)) +flush_tlb_all ();We could even free the original pmd here. I think a
memblock_free(pud_pfn(old_pud) << PAGE_SHIFT, PAGE_SIZE)should do (untested, and you need to definepud_pfn).
I see what you mean, we will potentially have an unused page in our swapper_pg_dir array.
I'll have a think, and add some logic to remove the redundant page.
Cheers,
Signature · 1 line
Steve
Quoted 3 lines
Signature · 2 lines
-- Catalin