Skip to content

THP support for PPC64

Aneesh Kumar K.V linuxppc-dev 1 version New

Get series with b4
b4 am 1368375265-19593-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com

Versions newest first

  1. v1 Latest18 patches 0 repliesNew

Changes from V7 from the cover letter

* Address review feedback.
* mm/ patches also posted as a seperate series to linux-mm list
* Fixes for races against split/collapse and page table walk.
* Updated comments regarding locking details

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/18powerpc/mm: Use the correct mask value when looking at pgtable addressarch/powerpc/include/asm/pgalloc-64.h+1 -10 replies
2/18mm/THP: Use pmd_populate to update the pmd with pgtable_t pointermm/huge_memory.c+1 -10 replies
3/18mm/THP: Use the right function when updating access flagsmm/huge_memory.c+3 -10 replies
4/18mm/THP: Add pmd args to pgtable deposit and withdraw APIsarch/s390/include/asm/pgtable.h arch/s390/mm/pgtable.c arch/sparc/include/asm/pgtable_64.h +4 more+27 -210 replies
5/18mm/THP: withdraw the pgtable after pmdp related operationsmm/huge_memory.c+7 -10 replies
6/18mm/THP: Don't use HPAGE_SHIFT in transparent hugepage codeinclude/linux/huge_mm.h+3 -70 replies
7/18mm/THP: deposit the transpare huge pgtable before set_pmdmm/huge_memory.c+4 -40 replies
8/18powerpc/THP: Double the PMD table size for THParch/powerpc/include/asm/pgalloc-64.h arch/powerpc/include/asm/pgtable-ppc64-64k.h arch/powerpc/include/asm/pgtable-ppc64.h +1 more+16 -80 replies
9/18powerpc/THP: Implement transparent hugepages for ppc64arch/powerpc/include/asm/pgtable-ppc64.h arch/powerpc/include/asm/pgtable.h arch/powerpc/include/asm/tlbflush.h +3 more+599 -20 replies
10/18powerpc: move find_linux_pte_or_hugepte and gup_hugepte to common codearch/powerpc/include/asm/hugetlb.h arch/powerpc/include/asm/pgtable-ppc64.h arch/powerpc/include/asm/pgtable.h +2 more+138 -1380 replies
11/18powerpc: Update find_linux_pte_or_hugepte to handle transparent hugepagesarch/powerpc/mm/hugetlbpage.c+26 -60 replies
12/18powerpc: Replace find_linux_pte with find_linux_pte_or_hugeptearch/powerpc/include/asm/pgtable-ppc64.h arch/powerpc/kernel/io-workarounds.c arch/powerpc/kvm/book3s_hv_rm_mmu.c +4 more+35 -330 replies
13/18powerpc: Update gup_pmd_range to handle transparent hugepagesarch/powerpc/mm/gup.c+13 -20 replies
14/18powerpc/THP: Add code to handle HPTE faults for hugepagesarch/powerpc/include/asm/mmu-hash64.h arch/powerpc/mm/Makefile arch/powerpc/mm/hash_utils_64.c +1 more+199 -40 replies
15/18powerpc: Make linux pagetable walk safe with THP enabledarch/powerpc/mm/hash_utils_64.c arch/powerpc/mm/hugepage-hash64.c arch/powerpc/mm/hugetlbpage.c +1 more+68 -380 replies
16/18powerpc/THP: Enable THP on PPC64arch/powerpc/include/asm/pgtable-ppc64.h arch/powerpc/mm/pgtable_64.c+30 -20 replies
17/18powerpc: Optimize hugepage invalidatearch/powerpc/include/asm/machdep.h arch/powerpc/mm/hash_native_64.c arch/powerpc/mm/pgtable_64.c +1 more+202 -90 replies
18/18powerpc: disable assert_pte_locked for collapse_huge_pagearch/powerpc/mm/pgtable.c+8 -00 replies

Reviews trailers per patch

8 of 18 patches have a Reviewed-by or Acked-by.

PatchDavid GibsonAndrea Arcangeli
1/18
2/18
3/18
4/18R-b R-b
5/18R-b
6/18
7/18
8/18R-b
9/18
10/18R-b
11/18R-b
12/18
13/18R-b
14/18R-b
15/18
16/18R-b
17/18
18/18

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH -V8 00/18] THP support for PPC64

Hi,

This is the second patchset needed to support THP on ppc64. Some of the changes included in this series are tricky in that it changes the powerpc linux page table walk subtly. We also overload few of the pte flags for ptes at PMD level (huge page PTEs).

The below patches are also sent as a seperate patch series to linux-mm

[PATCH -V8 02/18] mm/THP: Use pmd_populate to update the pmd with [PATCH -V8 03/18] mm/THP: Use the right function when updating access [PATCH -V8 04/18] mm/THP: Add pmd args to pgtable deposit and [PATCH -V8 05/18] mm/THP: withdraw the pgtable after pmdp related [PATCH -V8 06/18] mm/THP: Don't use HPAGE_SHIFT in transparent [PATCH -V8 07/18] mm/THP: deposit the transpare huge pgtable before

The below fixes is also sent to linuxppc-devel as an independent patch. [PATCH -V8 01/18] powerpc/mm: Use the correct mask value when looking

I am adding the above patches in this series for easier testing.

Some numbers:

The latency measurements code from Anton found at
http://ozlabs.org/~anton/junkcode/latency2001.c

64K page size (With THP support)
--------------------------
[root@llmp24l02 test]# ./latency2001 8G
8589934592 428.49 cycles 120.50 ns
[root@llmp24l02 test]# ./latency2001 -l 8G
8589934592 471.16 cycles 132.50 ns
[root@llmp24l02 test]# echo never > /sys/kernel/mm/transparent_hugepage/enabled [root@llmp24l02 test]# ./latency2001 8G
8589934592 766.52 cycles 215.56 ns
[root@llmp24l02 test]#

4K page size (No THP support for 4K)
----------------------------
[root@llmp24l02 test]# ./latency2001 8G
8589934592 814.88 cycles 229.16 ns
[root@llmp24l02 test]# ./latency2001 -l 8G
8589934592 463.69 cycles 130.40 ns
[root@llmp24l02 test]#

We are close to hugetlbfs in latency and we can achieve this with zero config/page reservation. Most of the allocations above are fault allocated.

Another test that does 50000000 random access over 1GB area goes from 2.65 seconds to 1.07 seconds with this patchset.

split_huge_page impact:
---------------------
To look at the performance impact of large page invalidate, I tried the below experiment. The test involved, accessing a large contiguous region of memory location as below

    for (i = 0; i < size; i += PAGE_SIZE)
	data[i] = i;

We wanted to access the data in sequential order so that we look at the worst case THP performance. Accesing the data in sequential order implies we have the Page table cached and overhead of TLB miss is as minimal as possible. We also don't touch the entire page, because that can result in cache evict.

After we touched the full range as above, we now call mprotect on each of that page. A mprotect will result in a hugepage split. This should allow us to measure the impact of hugepage split.

    for (i = 0; i < size; i += PAGE_SIZE)
	 mprotect(&data[i], PAGE_SIZE, PROT_READ);

Split hugepage impact:
---------------------
THP enabled: 2.851561705 seconds for test completion THP disable: 3.599146098 seconds for test completion

We are 20.7% better than non THP case even when we have all the large pages split.

Detailed output:

THP enabled:
---------------------------------------
[root@llmp24l02 ~]# cat /proc/vmstat | grep thp
thp_fault_alloc 0
thp_fault_fallback 0
thp_collapse_alloc 0
thp_collapse_alloc_failed 0
thp_split 0
thp_zero_page_alloc 0
thp_zero_page_alloc_failed 0
[root@llmp24l02 ~]# /root/thp/tools/perf/perf stat -e page-faults,dTLB-load-misses ./split-huge-page-mpro 20G time taken to touch all the data in ns: 2763096913

Performance counter stats for './split-huge-page-mpro 20G':

	     1,581 page-faults                                                 
	     3,159 dTLB-load-misses                                            

       2.851561705 seconds time elapsed

[root@llmp24l02 ~]#
[root@llmp24l02 ~]# cat /proc/vmstat | grep thp
thp_fault_alloc 1279
thp_fault_fallback 0
thp_collapse_alloc 0
thp_collapse_alloc_failed 0
thp_split 1279
thp_zero_page_alloc 0
thp_zero_page_alloc_failed 0
[root@llmp24l02 ~]#

    77.05%  split-huge-page  [kernel.kallsyms]     [k] .clear_user_page                        
     7.10%  split-huge-page  [kernel.kallsyms]     [k] .perf_event_mmap_ctx                    
     1.51%  split-huge-page  split-huge-page-mpro  [.] 0x0000000000000a70                      
     0.96%  split-huge-page  [unknown]             [H] 0x000000000157e3bc                      
     0.81%  split-huge-page  [kernel.kallsyms]     [k] .up_write                               
     0.76%  split-huge-page  [kernel.kallsyms]     [k] .perf_event_mmap                        
     0.76%  split-huge-page  [kernel.kallsyms]     [k] .down_write                             
     0.74%  split-huge-page  [kernel.kallsyms]     [k] .lru_add_page_tail                      
     0.61%  split-huge-page  [kernel.kallsyms]     [k] .split_huge_page                        
     0.59%  split-huge-page  [kernel.kallsyms]     [k] .change_protection                      
     0.51%  split-huge-page  [kernel.kallsyms]     [k] .release_pages                          

0.96% split-huge-page [unknown] [H] 0x000000000157e3bc
|
|--79.44%-- reloc_start
| |
| |--86.54%-- .__pSeries_lpar_hugepage_invalidate
| | .pSeries_lpar_hugepage_invalidate
| | .hpte_need_hugepage_flush
| | .split_huge_page
| | .__split_huge_page_pmd
| | .vma_adjust
| | .vma_merge
| | .mprotect_fixup
| | .SyS_mprotect

THP disabled:
---------------
[root@llmp24l02 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled [root@llmp24l02 ~]# /root/thp/tools/perf/perf stat -e page-faults,dTLB-load-misses ./split-huge-page-mpro 20G time taken to touch all the data in ns: 3513767220

Performance counter stats for './split-huge-page-mpro 20G':

	  3,27,726 page-faults                                                 
	  3,29,654 dTLB-load-misses                                            

       3.599146098 seconds time elapsed

[root@llmp24l02 ~]#

Changes from V7:
* Address review feedback.
* mm/ patches also posted as a seperate series to linux-mm list
* Fixes for races against split/collapse and page table walk.
* Updated comments regarding locking details

Changes from V6:
* split the patch series into two patchset.
* Address review feedback.

Changes from V5:
* Address review comments
* Added new patch to not use hugepd for explcit hugepages. Explicit hugepaes now use PTE format similar to transparent hugepages.
* We don't use page->_mapcount for tracking free PTE frags in a PTE page.
* rebased to a86d52667d8eda5de39393ce737794403bdce1eb
* Tested with libhugetlbfs test suite

Changes from V4:
* Fix bad page error in page_table_alloc

  BUG: Bad page state in process stream  pfn:f1a59
  page:f0000000034dc378 count:1 mapcount:0 mapping:          (null) index:0x0
  [c000000f322c77d0] [c00000000015e198] .bad_page+0xe8/0x140
  [c000000f322c7860] [c00000000015e3c4] .free_pages_prepare+0x1d4/0x1e0
  [c000000f322c7910] [c000000000160450] .free_hot_cold_page+0x50/0x230
  [c000000f322c79c0] [c00000000003ad18] .page_table_alloc+0x168/0x1c0

Changes from V3:
* PowerNV boot fixes

Change from V2:
* Change patch "powerpc: Reduce PTE table memory wastage" to use much simpler approach for PTE page sharing.
* Changes to handle huge pages in KVM code.
* Address other review comments

Changes from V1
* Address review comments
* More patch split
* Add batch hpte invalidate for hugepages.

Changes from RFC V2:
* Address review comments
* More code cleanup and patch split

Changes from RFC V1:
* HugeTLB fs now works
* Compile issues fixed
* rebased to v3.8
* Patch series reorded so that ppc64 cleanups and MM THP changes are moved early in the series. This should help in picking those patches early.

Thanks,
-aneesh