PPC32 : Huge-page support for ppc440

From: Edi Shmueli [email protected]

This patch enables applications to exploit the PPC440 TLB support for huge-page mapping, to minimize TLB thrashing. Applications with large memory footprint that exploit this support, experience minimal TLB misses, and boost in performance. NAS benchmarks performed with this patch indicate hundreds of percent of improvement in performance for some application. Known limitations: because each PMD covers 2MB of the process address space, the minimal huge-page size supported by this patch is 2M.

-----

Benchmarks and Implementation comments
======================================
Below is the NAS IS benchmark results, executing under Linux, with and without this huge-page mapping support.
IS Benchmark 4KB pages 16MB pages
==============================================
Class = A A
Size = 8388608 8388608
Iterations = 10 10
Time in seconds = 24.44 6.38
Mop/s total = 3.43 13.15
Operation type = keys ranked keys ranked
Verification = SUCCESSFUL SUCCESSFUL
Implementation details:
====================
This patch is ppc440 architecture-specific. It enables the use of huge-pages by processes executing under the 2.6.16 kernel on the ppc440 processors.
Huge-pages are accessible to user processes using either the hugetlbfs or using shared memory. See Documentation/vs/hugetlbpage.txt.
This patch supports huge-pages of size 16MB and 256MB.
==
The ppc 32bit kernel uses 64bit PTEs (set by CONFIG_PTE_64BIT).
I exploit a "hole" of 4 unused bits in the PTE MS word (bits 24-27) and code the page size information in those bits.
I then modified the TLB miss handler (head_44x.S) to stop using the constant PPC44x_TLB_4K to set the page size in the TLB,
Instead, when a TLB miss happens, the miss handler reads the size information from the PTE and sets that size in the TLB entry.
This way, different TLB entries get to map different page sizes (e.g., 4KB or 16MB).
The TLB replacement policy remains RR. This means that a huge-page entry in the TLB may be overwritten if not used for a long time, but when accessed it will be set again by the TLB miss handler, with the correct size, as set in the PTE. ==
In arch/ppc/mm/hugetlbpage.c is where page table entries are set to map huge pages:
By default , each process has two-level page-tables.
It has 2048, 32bit PMD (or PGD) entries at the higher-level, each maps 2MB of the process address spase, and 512, 64bit PTE entries in each lower-level page table.
When a TLB miss happens and no PTE is found by the miss handler to offer the translation, a check is made (memory.c) on whether the faulting address belongs to a huge-page VM region.
If so, the code in set_huge_pte_at() will set 8 PMD (8 x 2M=16MB, for the 16MB page case) entries. All will point to the *same* lower-level PTE page table.
It will also set all 512 PTEs in that page table to point to the *same* physical huge-page. All these PTEs will be *identical* and have 16MB size coded in their MS word as described above.
==
Once the TLB miss handler copies the mapping (and the size) from the PTE into on of the TLB entries, the process will not suffer any TLB misses for that huge-page.
If the mapping was overwritten by the TLB RR replacement policy, it will be re-loaded again (possibly in a different TLB entry) when the process re-access that huge-page.
Because all 8 PMDs point to the same PTE page-table, and all 512 PTEs point to the same physical address, it makes no difference where the process accesses that huge-page.
For any address in this huge-pages, the process will get the same translation in the TLB.

diff -uprP -X linux-2.6.16.29-vanilla/Documentation/dontdiff linux-2.6.16.29-vanilla/arch/ppc/kernel/head_44x.S my_linux/arch/ppc/kernel/head_44x.S
@@ -23,12 +23,14 @@
 *            [email protected]
 *    Copyright 2002-2005 MontaVista Software, Inc.
 *      PowerPC 44x support, Matt Porter <[email protected]>
+ *    Copyright (C) 2006 Edi Shmueli, IBM Corporation.
+ *    PowerPC 44x handling of huge-page misses.
 *
 * This program is free software; you can redistribute  it and/or modify it
 * under  the terms of  the GNU General  Public License as published by the
 * Free Software Foundation;  either version 2 of the  License, or (at your
 * option) any later version.
- */
+*/
 

#include <linux/config.h>
#include <asm/processor.h>
@@ -657,19 +659,22 @@ finish_tlb_load:

    lis    r11, tlb_44x_index@ha
    stw    r13, tlb_44x_index@l(r11)

-    lwz    r11, 0(r12)            /* Get MS word of PTE */
-    lwz    r12, 4(r12)            /* Get LS word of PTE */
-    rlwimi    r11, r12, 0, 0 , 19        /* Insert RPN */
-    tlbwe    r11, r13, PPC44x_TLB_XLAT    /* Write XLAT */

-
+ /*huge-page support*/
/*
* Create PAGEID. This is the faulting address,
* page size, and valid flag.
- */

-    li    r11, PPC44x_TLB_VALID | PPC44x_TLB_4K
-    rlwimi    r10, r11, 0, 20, 31        /* Insert valid and page size */
+    */
+    lwz    r11, 0(r12)            /* Get MS word of PTE */       +    andi.   r11, r11, 0x000000F0        /* Leave only the size */
+    ori     r11, r11, PPC44x_TLB_VALID | PPC44x_TLB_4K /* Insert valid and page size */
+    rlwimi    r10, r11, 0, 20, 31        /* Combine EA from r10 with flags from r11 */

tlbwe r10, r13, PPC44x_TLB_PAGEID /* Write PAGEID */

+    lwz    r11, 0(r12)            /* Get MS word of PTE */
+    lwz    r12, 4(r12)            /* Get LS word of PTE */
+    rlwimi    r11, r12, 0, 0 , 19        /* Insert RPN */
+    tlbwe    r11, r13, PPC44x_TLB_XLAT    /* Write XLAT */
+
    li    r10, PPC44x_TLB_SR@l        /* Set SR */
    rlwimi    r10, r12, 0, 30, 30        /* Set SW = _PAGE_RW */
    rlwimi    r10, r12, 29, 29, 29        /* SX = _PAGE_HWEXEC */

diff -uprP -X linux-2.6.16.29-vanilla/Documentation/dontdiff linux-2.6.16.29-vanilla/arch/ppc/mm/Makefile my_linux/arch/ppc/mm/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_PPC_STD_MMU)    += hashtable.o

obj-$(CONFIG_40x) += 4xx_mmu.o
obj-$(CONFIG_44x) += 44x_mmu.o
obj-$(CONFIG_FSL_BOOKE) += fsl_booke_mmu.o
+obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o

diff -uprP -X linux-2.6.16.29-vanilla/Documentation/dontdiff linux-2.6.16.29-vanilla/fs/Kconfig my_linux/fs/Kconfig
@@ -840,7 +840,7 @@ config TMPFS
 

config HUGETLBFS
bool "HugeTLB file system support"

-    depends X86 || IA64 || PPC64 || SPARC64 || SUPERH || BROKEN
+    depends X86 || IA64 || PPC || PPC64 || SPARC64 || SUPERH || BROKEN

config HUGETLB_PAGE
def_bool HUGETLBFS

diff -uprP -X linux-2.6.16.29-vanilla/Documentation/dontdiff linux-2.6.16.29-vanilla/include/asm-ppc/page.h my_linux/include/asm-ppc/page.h
@@ -8,6 +8,12 @@

#define PAGE_SHIFT 12
#define PAGE_SIZE (ASM_CONST(1) << PAGE_SHIFT)

+#ifdef CONFIG_HUGETLB_PAGE

+#define HPAGE_SHIFT     24
+#define HPAGE_SIZE      ((1UL) << HPAGE_SHIFT)
+#define HPAGE_MASK      (~(HPAGE_SIZE - 1))

+#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT)
+#endif
/*
* Subtle: this is an int (not an unsigned long) and so it
* gets extended to 64 bits the way want (i.e. with 1s). -- paulus

diff -uprP -X linux-2.6.16.29-vanilla/Documentation/dontdiff linux-2.6.16.29-vanilla/include/asm-ppc/pgtable.h my_linux/include/asm-ppc/pgtable.h
@@ -264,6 +264,24 @@ extern unsigned long ioremap_bot, iorema

#define _PAGE_NO_CACHE 0x00000400 /* H: I bit */
#define _PAGE_WRITETHRU 0x00000800 /* H: W bit */

+#if HPAGE_SHIFT == 10
+#define _PAGE_HUGE 0x0000000000000000ULL /* H: SIZE=1K bytes */
+#elif HPAGE_SHIFT == 12
+#define _PAGE_HUGE 0x0000001000000000ULL /* H: SIZE=4K bytes */
+#elif HPAGE_SHIFT == 14
+#define _PAGE_HUGE 0x0000002000000000ULL /* H: SIZE=16K bytes */
+#elif HPAGE_SHIFT == 16
+#define _PAGE_HUGE 0x0000003000000000ULL /* H: SIZE=64K bytes */
+#elif HPAGE_SHIFT == 18
+#define _PAGE_HUGE 0x0000004000000000ULL /* H: SIZE=256K bytes */
+#elif HPAGE_SHIFT == 20
+#define _PAGE_HUGE 0x0000005000000000ULL /* H: SIZE=1M bytes */
+#elif HPAGE_SHIFT == 24
+#define _PAGE_HUGE 0x0000007000000000ULL /* H: SIZE=16M bytes */
+#elif HPAGE_SHIFT == 28
+#define _PAGE_HUGE 0x0000009000000000ULL /* H: SIZE=256M bytes */
+#endif
+
/* TODO: Add large page lowmem mapping support */
#define _PMD_PRESENT 0
#define _PMD_PRESENT_MASK (PAGE_MASK)
@@ -491,7 +509,12 @@ extern unsigned long bad_call_to_PMD_PAG
#define PFN_SHIFT_OFFSET (PAGE_SHIFT)
#endif

-#define pte_pfn(x) (pte_val(x) >> PFN_SHIFT_OFFSET)
+/* v */
+/* huge-page fix: since now the PTE (MS) contains the size of the page e.g., 0x0000007000000000ULL */ +/* which was once assumed to be 0, we must zero it, otherwise we get extreamly huge (and wrong) pfn */
+//#define pte_pfn(x) ( pte_val(x) >> PFN_SHIFT_OFFSET )
+#define pte_pfn(x) ( ( pte_val(x) & 0x0000000fffffffffULL ) >> PFN_SHIFT_OFFSET )
+
#define pte_page(x) pfn_to_page(pte_pfn(x))

#define pfn_pte(pfn, prot) __pte(((pte_basic_t)(pfn) << PFN_SHIFT_OFFSET) |\
@@ -540,6 +563,7 @@ static inline int pte_exec(pte_t pte) {
static inline int pte_dirty(pte_t pte) { return pte_val(pte) & _PAGE_DIRTY; }
static inline int pte_young(pte_t pte) { return pte_val(pte) & _PAGE_ACCESSED; }
static inline int pte_file(pte_t pte) { return pte_val(pte) & _PAGE_FILE; }
+static inline int pte_huge(pte_t pte) { return (pte_val(pte) & _PAGE_HUGE)?1:0; }

static inline void pte_uncache(pte_t pte) { pte_val(pte) |= _PAGE_NO_CACHE; }
static inline void pte_cache(pte_t pte) { pte_val(pte) &= ~_PAGE_NO_CACHE; }
@@ -565,6 +589,8 @@ static inline pte_t pte_mkdirty(pte_t pt

    pte_val(pte) |= _PAGE_DIRTY; return pte; }
static inline pte_t pte_mkyoung(pte_t pte) {
    pte_val(pte) |= _PAGE_ACCESSED; return pte; }

+static inline pte_t pte_mkhuge(pte_t pte) {
+ pte_val(pte) |= _PAGE_HUGE; return pte; }

static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) {

diff -uprP -X linux-2.6.16.29-vanilla/Documentation/dontdiff linux-2.6.16.29-vanilla/include/linux/hugetlb.h my_linux/include/linux/hugetlb.h
@@ -2,6 +2,7 @@

#define _LINUX_HUGETLB_H

#ifdef CONFIG_HUGETLB_PAGE
+#define ARCH_HAS_SETCLEAR_HUGE_PTE

#include <linux/mempolicy.h>
#include <asm/tlbflush.h>

diff -uprP -X linux-2.6.16.29-vanilla/Documentation/dontdiff linux-2.6.16.29-vanilla/include/linux/mmzone.h my_linux/include/linux/mmzone.h
@@ -17,7 +17,7 @@
 

/* Free memory management - zoned buddy allocator. */ #ifndef CONFIG_FORCE_MAX_ZONEORDER
-#define MAX_ORDER 11
+#define MAX_ORDER 13
#else
#define MAX_ORDER CONFIG_FORCE_MAX_ZONEORDER #endif

Hi Edi,

Your patch is line wrapped and the whitespace has been corrupted. It's also against a bit older version of the kernel.

Could you try to fixup these issues and resubmit? Paul's powerpc.git tree would probably be best to diff against.

josh

Also, while it's great to have somebody do this work, I doubt there is much interest in merging it for arch/ppc...

Ben.

On that subject, what's the latest on moving ppc4xx to arch/powerpc? At the kernel summit you told me to chill out and wait, but I'll repeat my offer to help out....

- R.

Well, I told you that Matt Porter was about to do it :-) Now I haven't heard from him since. Feel free to jump in, just make sure you keep in sync with Josh and Matt.

One of the reason waiting was a good idea too is that we didn't have a proper mecanism for stuffing device-trees in zImage wrappers, which we have now.

Also, I've started doing some work around 4xx devices used on MMIO platforms (like the Axon southbridge on Cell), and while doing that, started moving EMAC to be an of_platform_device (or rather a set of them since EMAC is made of multiple devices). (Based on initial work by Shaun).

You can see some of that stuff in the big patch set I posted to the list yesterday, though I already fixed various issues in some of those patches, so I suppose I'll have to post a new one soon :)

Cheers,
Ben.

I've gotten this mostly working for 44x. I need to rebase and do a few cleanups on the patches though.

That'll come in handy soon. Right now, the zImage wrapper uncompresses the kernel and jumps to it, but I don't get any further output. It's just a matter of getting time and I need to sit down and work on this some more.

josh

Is this 44x work at the point where having more people would help? I have a 440SPe eval board that I'd like to get going with arch/powerpc. I could work on PCI Express or something like that...

On the other hand, I'm happy to wait and work on other stuff...

- R.

Count me in for the 44x work. Will be happy to contribute my 440GP board for this purpose :)

Er, wait a minute. :) I did some work then got busy again. I handed my starting point stuff to Josh who has been continuing on with some work. So there is progress happening there on the basic enablement, that is, outside of your contributions on mmio dcr handling etc.

-Matt