Skip to content

THP support for PPC64

Aneesh Kumar K.V linux-mm linuxppc-dev 1 version In review

Get series with b4
b4 am 1365055083-31956-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com

Versions newest first

  1. v1 Latest25 patches 47 repliesIn review

Changes from V4 from the cover letter

* Fix bad page error in page_table_alloc

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/25powerpc: Use signed formatting when printing errorarch/powerpc/platforms/pseries/lpar.c+1 -10 replies
2/25powerpc: Save DAR and DSISR in pt_regs on MCEarch/powerpc/kernel/exceptions-64s.S arch/powerpc/xmon/xmon.c+10 -10 replies
3/25powerpc: Don't hard code the size of pte pagearch/powerpc/include/asm/pgtable.h arch/powerpc/mm/hash_low_64.S+8 -20 replies
4/25powerpc: Reduce the PTE_INDEX_SIZEarch/powerpc/include/asm/pgtable-ppc64-64k.h+2 -21 reply
5/25powerpc: Move the pte free routines from common headerarch/powerpc/include/asm/pgalloc-32.h arch/powerpc/include/asm/pgalloc-64.h arch/powerpc/include/asm/pgalloc.h+189 -590 replies
6/25powerpc: Reduce PTE table memory wastagearch/powerpc/include/asm/mmu-book3e.h arch/powerpc/include/asm/mmu-hash64.h arch/powerpc/include/asm/page.h +4 more+202 -5810 replies
7/25powerpc: Use encode avpn where we need only avpn valuesarch/powerpc/mm/hash_native_64.c arch/powerpc/platforms/cell/beat_htab.c arch/powerpc/platforms/ps3/htab.c+10 -100 replies
8/25powerpc: Decode the pte-lp-encoding bits correctly.arch/powerpc/include/asm/machdep.h arch/powerpc/include/asm/mmu-hash64.h arch/powerpc/kvm/book3s_hv.c +7 more+230 -1174 replies
9/25powerpc: Fix hpte_decode to use the correct decoding for page sizesarch/powerpc/mm/hash_native_64.c+24 -141 reply
10/25powerpc: print both base and actual page size on hash failurearch/powerpc/include/asm/mmu-hash64.h arch/powerpc/mm/hash_utils_64.c arch/powerpc/mm/hugetlbpage-hash64.c+10 -71 reply
11/25powerpc: Print page size info during bootarch/powerpc/mm/hash_utils_64.c+5 -50 replies
12/25powerpc: Return all the valid pte ecndoing in KVM_PPC_GET_SMMU_INFO ioctlarch/powerpc/kvm/book3s_hv.c+12 -23 replies
13/25powerpc: Update tlbie/tlbiel as per ISA docarch/powerpc/mm/hash_native_64.c+28 -24 replies
14/25mm/THP: HPAGE_SHIFT is not a #define on some archinclude/linux/huge_mm.h mm/huge_memory.c+6 -61 reply
15/25mm/THP: Add pmd args to pgtable deposit and withdraw APIsarch/s390/include/asm/pgtable.h arch/s390/mm/pgtable.c arch/sparc/include/asm/pgtable_64.h +4 more+27 -211 reply
16/25mm/THP: withdraw the pgtable after pmdp related operationsmm/huge_memory.c+2 -10 replies
17/25powerpc/THP: Implement transparent hugepages for ppc64arch/powerpc/include/asm/page.h arch/powerpc/include/asm/pgtable-ppc64-64k.h arch/powerpc/include/asm/pgtable-ppc64.h +4 more+572 -35 replies
18/25powerpc/THP: Double the PMD table size for THParch/powerpc/include/asm/pgalloc-64.h arch/powerpc/include/asm/pgtable-ppc64.h arch/powerpc/mm/init_64.c+14 -71 reply
19/25powerpc/THP: Differentiate THP PMD entries from HUGETLB PMD entriesarch/powerpc/include/asm/pgtable.h arch/powerpc/mm/pgtable.c arch/powerpc/mm/pgtable_64.c+18 -53 replies
20/25powerpc/THP: Add code to handle HPTE faults for large pagesarch/powerpc/include/asm/mmu-hash64.h arch/powerpc/include/asm/pgtable-ppc64.h arch/powerpc/kernel/io-workarounds.c +10 more+286 -421 reply
21/25powerpc: Handle hugepage in perf callchainarch/powerpc/perf/callchain.c+21 -112 replies
22/25powerpc/THP: get_user_pages_fast changesarch/powerpc/mm/gup.c+82 -21 reply
23/25powerpc/THP: Enable THP on PPC64arch/powerpc/include/asm/pgtable.h+29 -20 replies
24/25powerpc: Optimize hugepage invalidatearch/powerpc/include/asm/machdep.h arch/powerpc/mm/hash_native_64.c arch/powerpc/mm/pgtable.c +1 more+210 -103 replies
25/25powerpc: Handle hugepages in kvmarch/powerpc/include/asm/kvm_book3s_64.h arch/powerpc/kvm/book3s_64_mmu_hv.c arch/powerpc/kvm/book3s_hv_rm_mmu.c+97 -210 replies

Reviews trailers per patch

10 of 25 patches have a Reviewed-by or Acked-by.

PatchPaul MackerrasDavid Gibson
1/25A-b
2/25A-b
3/25A-b
4/25A-b
5/25A-b
6/25
7/25A-b
8/25
9/25A-b R-b
10/25R-b
11/25
12/25
13/25
14/25R-b
15/25R-b
16/25
17/25
18/25
19/25
20/25
21/25
22/25
23/25
24/25
25/25

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH -V5 00/25] THP support for PPC64

Hi,

This patchset adds transparent hugepage support for PPC64.

TODO:
* hash preload support in update_mmu_cache_pmd (we don't do that for hugetlb)

Some numbers:

The latency measurements code from Anton found at
http://ozlabs.org/~anton/junkcode/latency2001.c

THP disabled 64K page size
------------------------
[root@llmp24l02 ~]# ./latency2001 8G
8589934592 731.73 cycles 205.77 ns
[root@llmp24l02 ~]# ./latency2001 8G
8589934592 743.39 cycles 209.05 ns
[root@llmp24l02 ~]#

THP disabled large page via hugetlbfs
-------------------------------------
[root@llmp24l02 ~]# ./latency2001 -l 8G
8589934592 416.09 cycles 117.01 ns
[root@llmp24l02 ~]# ./latency2001 -l 8G
8589934592 415.74 cycles 116.91 ns

THP enabled 64K page size.
----------------
[root@llmp24l02 ~]# ./latency2001 8G
8589934592 405.07 cycles 113.91 ns
[root@llmp24l02 ~]# ./latency2001 8G
8589934592 411.82 cycles 115.81 ns
[root@llmp24l02 ~]#

We are close to hugetlbfs in latency and we can achieve this with zero config/page reservation. Most of the allocations above are fault allocated.

Another test that does 50000000 random access over 1GB area goes from 2.65 seconds to 1.07 seconds with this patchset.

split_huge_page impact:
---------------------
To look at the performance impact of large page invalidate, I tried the below experiment. The test involved, accessing a large contiguous region of memory location as below

    for (i = 0; i < size; i += PAGE_SIZE)
	data[i] = i;

We wanted to access the data in sequential order so that we look at the worst case THP performance. Accesing the data in sequential order implies we have the Page table cached and overhead of TLB miss is as minimal as possible. We also don't touch the entire page, because that can result in cache evict.

After we touched the full range as above, we now call mprotect on each of that page. A mprotect will result in a hugepage split. This should allow us to measure the impact of hugepage split.

    for (i = 0; i < size; i += PAGE_SIZE)
	 mprotect(&data[i], PAGE_SIZE, PROT_READ);

Split hugepage impact:
---------------------
THP enabled: 2.851561705 seconds for test completion THP disable: 3.599146098 seconds for test completion

We are 20.7% better than non THP case even when we have all the large pages split.

Detailed output:

THP enabled:
---------------------------------------
[root@llmp24l02 ~]# cat /proc/vmstat | grep thp
thp_fault_alloc 0
thp_fault_fallback 0
thp_collapse_alloc 0
thp_collapse_alloc_failed 0
thp_split 0
thp_zero_page_alloc 0
thp_zero_page_alloc_failed 0
[root@llmp24l02 ~]# /root/thp/tools/perf/perf stat -e page-faults,dTLB-load-misses ./split-huge-page-mpro 20G time taken to touch all the data in ns: 2763096913

Performance counter stats for './split-huge-page-mpro 20G':

             1,581 page-faults                                                 
             3,159 dTLB-load-misses                                            

       2.851561705 seconds time elapsed

[root@llmp24l02 ~]#
[root@llmp24l02 ~]# cat /proc/vmstat | grep thp
thp_fault_alloc 1279
thp_fault_fallback 0
thp_collapse_alloc 0
thp_collapse_alloc_failed 0
thp_split 1279
thp_zero_page_alloc 0
thp_zero_page_alloc_failed 0
[root@llmp24l02 ~]#

    77.05%  split-huge-page  [kernel.kallsyms]     [k] .clear_user_page                        
     7.10%  split-huge-page  [kernel.kallsyms]     [k] .perf_event_mmap_ctx                    
     1.51%  split-huge-page  split-huge-page-mpro  [.] 0x0000000000000a70                      
     0.96%  split-huge-page  [unknown]             [H] 0x000000000157e3bc                      
     0.81%  split-huge-page  [kernel.kallsyms]     [k] .up_write                               
     0.76%  split-huge-page  [kernel.kallsyms]     [k] .perf_event_mmap                        
     0.76%  split-huge-page  [kernel.kallsyms]     [k] .down_write                             
     0.74%  split-huge-page  [kernel.kallsyms]     [k] .lru_add_page_tail                      
     0.61%  split-huge-page  [kernel.kallsyms]     [k] .split_huge_page                        
     0.59%  split-huge-page  [kernel.kallsyms]     [k] .change_protection                      
     0.51%  split-huge-page  [kernel.kallsyms]     [k] .release_pages                          
     0.96%  split-huge-page  [unknown]             [H] 0x000000000157e3bc                      
            |          
            |--79.44%-- reloc_start
            |          |          
            |          |--86.54%-- .__pSeries_lpar_hugepage_invalidate
            |          |          .pSeries_lpar_hugepage_invalidate
            |          |          .hpte_need_hugepage_flush
            |          |          .split_huge_page
            |          |          .__split_huge_page_pmd
            |          |          .vma_adjust
            |          |          .vma_merge
            |          |          .mprotect_fixup
            |          |          .SyS_mprotect

THP disabled:
---------------
[root@llmp24l02 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled [root@llmp24l02 ~]# /root/thp/tools/perf/perf stat -e page-faults,dTLB-load-misses ./split-huge-page-mpro 20G time taken to touch all the data in ns: 3513767220

Performance counter stats for './split-huge-page-mpro 20G':

          3,27,726 page-faults                                                 
          3,29,654 dTLB-load-misses                                            

       3.599146098 seconds time elapsed

[root@llmp24l02 ~]#

Changes from V4:
* Fix bad page error in page_table_alloc

  BUG: Bad page state in process stream  pfn:f1a59
  page:f0000000034dc378 count:1 mapcount:0 mapping:          (null) index:0x0
  [c000000f322c77d0] [c00000000015e198] .bad_page+0xe8/0x140
  [c000000f322c7860] [c00000000015e3c4] .free_pages_prepare+0x1d4/0x1e0
  [c000000f322c7910] [c000000000160450] .free_hot_cold_page+0x50/0x230
  [c000000f322c79c0] [c00000000003ad18] .page_table_alloc+0x168/0x1c0

Changes from V3:
* PowerNV boot fixes

Change from V2:
* Change patch "powerpc: Reduce PTE table memory wastage" to use much simpler approach for PTE page sharing.
* Changes to handle huge pages in KVM code.
* Address other review comments

Changes from V1
* Address review comments
* More patch split
* Add batch hpte invalidate for hugepages.

Changes from RFC V2:
* Address review comments
* More code cleanup and patch split

Changes from RFC V1:
* HugeTLB fs now works
* Compile issues fixed
* rebased to v3.8
* Patch series reorded so that ppc64 cleanups and MM THP changes are moved early in the series. This should help in picking those patches early.

Thanks,
-aneesh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>