THP support for PPC64
Subsystems (10)
cover THP support for PPC64
Hi,
This is the second patchset needed to support THP on ppc64. Some of the changes included in this series are tricky in that it changes the powerpc linux page table walk subtly. We also overload few of the pte flags for ptes at PMD level (huge page PTEs).
mm/ changes are already merged to Andrew's -mm tree. I am including them in this series for easy testing and review.
Some numbers:
The latency measurements code from Anton found at
http://ozlabs.org/~anton/junkcode/latency2001.c
64K page size (With THP support)
--------------------------
[root@llmp24l02 test]# ./latency2001 8G
8589934592 428.49 cycles 120.50 ns
[root@llmp24l02 test]# ./latency2001 -l 8G
8589934592 471.16 cycles 132.50 ns
[root@llmp24l02 test]# echo never > /sys/kernel/mm/transparent_hugepage/enabled [root@llmp24l02 test]# ./latency2001 8G
8589934592 766.52 cycles 215.56 ns
[root@llmp24l02 test]#
4K page size (No THP support for 4K)
----------------------------
[root@llmp24l02 test]# ./latency2001 8G
8589934592 814.88 cycles 229.16 ns
[root@llmp24l02 test]# ./latency2001 -l 8G
8589934592 463.69 cycles 130.40 ns
[root@llmp24l02 test]#
We are close to hugetlbfs in latency and we can achieve this with zero config/page reservation. Most of the allocations above are fault allocated.
Another test that does 50000000 random access over 1GB area goes from 2.65 seconds to 1.07 seconds with this patchset.
split_huge_page impact:
---------------------
To look at the performance impact of large page invalidate, I tried the below experiment. The test involved, accessing a large contiguous region of memory location as below
for (i = 0; i < size; i += PAGE_SIZE) data[i] = i;
We wanted to access the data in sequential order so that we look at the worst case THP performance. Accesing the data in sequential order implies we have the Page table cached and overhead of TLB miss is as minimal as possible. We also don't touch the entire page, because that can result in cache evict.
After we touched the full range as above, we now call mprotect on each of that page. A mprotect will result in a hugepage split. This should allow us to measure the impact of hugepage split.
for (i = 0; i < size; i += PAGE_SIZE) mprotect(&data[i], PAGE_SIZE, PROT_READ);
Split hugepage impact:
---------------------
THP enabled: 2.851561705 seconds for test completion THP disable: 3.599146098 seconds for test completion
We are 20.7% better than non THP case even when we have all the large pages split.
Detailed output:
THP enabled:
---------------------------------------
[root@llmp24l02 ~]# cat /proc/vmstat | grep thpthp_fault_alloc 0thp_fault_fallback 0thp_collapse_alloc 0thp_collapse_alloc_failed 0thp_split 0thp_zero_page_alloc 0thp_zero_page_alloc_failed 0
[root@llmp24l02 ~]# /root/thp/tools/perf/perf stat -e page-faults,dTLB-load-misses ./split-huge-page-mpro 20G time taken to touch all the data in ns: 2763096913
Performance counter stats for './split-huge-page-mpro 20G':
1,581 page-faults
3,159 dTLB-load-misses
2.851561705 seconds time elapsed[root@llmp24l02 ~]#
[root@llmp24l02 ~]# cat /proc/vmstat | grep thpthp_fault_alloc 1279thp_fault_fallback 0thp_collapse_alloc 0thp_collapse_alloc_failed 0thp_split 1279thp_zero_page_alloc 0thp_zero_page_alloc_failed 0
[root@llmp24l02 ~]#
77.05% split-huge-page [kernel.kallsyms] [k] .clear_user_page
7.10% split-huge-page [kernel.kallsyms] [k] .perf_event_mmap_ctx
1.51% split-huge-page split-huge-page-mpro [.] 0x0000000000000a70
0.96% split-huge-page [unknown] [H] 0x000000000157e3bc
0.81% split-huge-page [kernel.kallsyms] [k] .up_write
0.76% split-huge-page [kernel.kallsyms] [k] .perf_event_mmap
0.76% split-huge-page [kernel.kallsyms] [k] .down_write
0.74% split-huge-page [kernel.kallsyms] [k] .lru_add_page_tail
0.61% split-huge-page [kernel.kallsyms] [k] .split_huge_page
0.59% split-huge-page [kernel.kallsyms] [k] .change_protection
0.51% split-huge-page [kernel.kallsyms] [k] .release_pages 0.96% split-huge-page [unknown] [H] 0x000000000157e3bc
|
|--79.44%-- reloc_start
| |
| |--86.54%-- .__pSeries_lpar_hugepage_invalidate
| | .pSeries_lpar_hugepage_invalidate
| | .hpte_need_hugepage_flush
| | .split_huge_page
| | .__split_huge_page_pmd
| | .vma_adjust
| | .vma_merge
| | .mprotect_fixup
| | .SyS_mprotect
THP disabled:
---------------
[root@llmp24l02 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled [root@llmp24l02 ~]# /root/thp/tools/perf/perf stat -e page-faults,dTLB-load-misses ./split-huge-page-mpro 20G time taken to touch all the data in ns: 3513767220
Performance counter stats for './split-huge-page-mpro 20G':
3,27,726 page-faults
3,29,654 dTLB-load-misses
3.599146098 seconds time elapsed[root@llmp24l02 ~]#
Changes from V8:
* rebase to 3.10-rc2
* make subpage protection syscall work with transparent hugepage.
* Add proper barriers when reading pgtable content stashed in the second half of PMD.
Changes from V7:
* Address review feedback.
* mm/ patches also posted as a seperate series to linux-mm list
* Fixes for races against split and page table walk.
* Updated comments regarding locking details
Changes from V6:
* split the patch series into two patchset.
* Address review feedback.
Changes from V5:
* Address review comments
* Added new patch to not use hugepd for explcit hugepages. Explicit hugepaes now use PTE format similar to transparent hugepages.
* We don't use page->_mapcount for tracking free PTE frags in a PTE page.
* rebased to a86d52667d8eda5de39393ce737794403bdce1eb
* Tested with libhugetlbfs test suite
Changes from V4:
* Fix bad page error in page_table_alloc
BUG: Bad page state in process stream pfn:f1a59 page:f0000000034dc378 count:1 mapcount:0 mapping: (null) index:0x0 [c000000f322c77d0] [c00000000015e198] .bad_page+0xe8/0x140 [c000000f322c7860] [c00000000015e3c4] .free_pages_prepare+0x1d4/0x1e0 [c000000f322c7910] [c000000000160450] .free_hot_cold_page+0x50/0x230 [c000000f322c79c0] [c00000000003ad18] .page_table_alloc+0x168/0x1c0
Changes from V3:
* PowerNV boot fixes
Change from V2:
* Change patch "powerpc: Reduce PTE table memory wastage" to use much simpler approach for PTE page sharing.
* Changes to handle huge pages in KVM code.
* Address other review comments
Changes from V1
* Address review comments
* More patch split
* Add batch hpte invalidate for hugepages.
Changes from RFC V2:
* Address review comments
* More code cleanup and patch split
Changes from RFC V1:
* HugeTLB fs now works
* Compile issues fixed
* rebased to v3.8
* Patch series reorded so that ppc64 cleanups and MM THP changes are moved early in the series. This should help in picking those patches early.
Thanks,
-aneesh
PATCH 1/20 mm/THP: Use pmd_populate to update the pmd with pgtable_t pointer
From: "Aneesh Kumar K.V" <[email protected]>
We should not use set_pmd_at to update pmd_t with pgtable_t pointer. set_pmd_at is used to set pmd with huge pte entries and architectures like ppc64, clear few flags from the pte when saving a new entry. Without this change we observe bad pte errors like below on ppc64 with THP enabled.
BUG: Bad page map in process ld mm=0xc000001ee39f4780 pte:7fc3f37848000001 pmd:c000001ec0000000
--- mm/huge_memory.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/huge_memory.c +6 -1
@@ -2325,7 +2325,12 @@ static void collapse_huge_page(struct mm_struct *mm,pte_unmap (pte);spin_lock (&mm->page_table_lock);BUG_ON (!pmd_none (*pmd)); -set_pmd_at (mm, address, pmd, _pmd); +/* +* We can only use set_pmd_at when establishing +* hugepmds and never for establishing regular pmds that +* points to regular pagetables. Use pmd_populate for that +*/ +pmd_populate (mm, pmd,pmd_pgtable (_pmd));spin_unlock (&mm->page_table_lock);anon_vma_unlock_write (vma->anon_vma); goto out;
Signature · 1 line
1.8.1.2
PATCH 2/20 mm/THP: Use the right function when updating access flags
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
We should use pmdp_set_access_flags to update access flags. Archs like powerpc use extra checks(_PAGE_BUSY) when updating a hugepage PTE. A set_pmd_at doesn't do those checks. We should use set_pmd_at only when updating a none hugepage PTE.
--- mm/huge_memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/huge_memory.c +3 -1
@@ -1265,7 +1265,9 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,* young bit, instead of the current set_pmd_at. */ _pmd =pmd_mkyoung (pmd_mkdirty (*pmd)); -set_pmd_at (mm, addr &HPAGE_PMD_MASK , pmd, _pmd); +if (pmdp_set_access_flags (vma, addr &HPAGE_PMD_MASK , + pmd, _pmd,1 )) +update_mmu_cache_pmd (vma, addr, pmd); }if ((flags &FOLL_MLOCK ) && (vma->vm_flags &VM_LOCKED )) {if (page->mapping &&trylock_page (page)) {
Signature · 1 line
1.8.1.2
PATCH 3/20 mm/THP: Add pmd args to pgtable deposit and withdraw APIs
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
This will be later used by powerpc THP support. In powerpc we want to use pgtable for storing the hash index values. So instead of adding them to mm_context list, we would like to store them in the second half of pmd
--- arch/s390/include/asm/pgtable.h | 5 +++-- arch/s390/mm/pgtable.c | 5 +++-- arch/sparc/include/asm/pgtable_64.h | 5 +++-- arch/sparc/mm/tlb.c | 5 +++-- include/asm-generic/pgtable.h | 5 +++-- mm/huge_memory.c | 18 +++++++++--------- mm/pgtable-generic.c | 5 +++-- 7 files changed, 27 insertions(+), 21 deletions(-)
arch/s390/include/asm/pgtable.h +3 -2
@@ -1351,10 +1351,11 @@ static inline pmd_t pmd_mkwrite(pmd_t pmd)#ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PGTABLE_DEPOSIT -extern void pgtable_trans_huge_deposit (struct mm_struct *mm,pgtable_t pgtable); +extern void pgtable_trans_huge_deposit(struct mm_struct *mm, , pmd_t *pmdp, +pgtable_t pgtable);#define __HAVE_ARCH_PGTABLE_WITHDRAW -extern pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm); +extern pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp);static inline int pmd_trans_splitting (pmd_t pmd) {
arch/s390/mm/pgtable.c +3 -2
@@ -1118,7 +1118,8 @@ void pmdp_splitting_flush(struct vm_area_struct *vma, unsigned long address, } } -void pgtable_trans_huge_deposit (struct mm_struct *mm,pgtable_t pgtable) +void pgtable_trans_huge_deposit (struct mm_struct *mm,pmd_t *pmdp, +pgtable_t pgtable) {struct list_head *lh = (struct list_head *) pgtable;
@@ -1132,7 +1133,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pgtable_t pgtable) mm->pmd_huge_pte = pgtable; } -pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm) +pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp) {struct list_head *lh;pgtable_t pgtable;
arch/sparc/include/asm/pgtable_64.h +3 -2
@@ -853,10 +853,11 @@ extern void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd);#define __HAVE_ARCH_PGTABLE_DEPOSIT -extern void pgtable_trans_huge_deposit (struct mm_struct *mm,pgtable_t pgtable); +extern void pgtable_trans_huge_deposit (struct mm_struct *mm,pmd_t *pmdp, +pgtable_t pgtable);#define __HAVE_ARCH_PGTABLE_WITHDRAW -extern pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm); +extern pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp);#endif /* Encode and de-code a swap entry */
arch/sparc/mm/tlb.c +3 -2
@@ -188,7 +188,8 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr, } } -void pgtable_trans_huge_deposit (struct mm_struct *mm,pgtable_t pgtable) +void pgtable_trans_huge_deposit (struct mm_struct *mm,pmd_t *pmdp, +pgtable_t pgtable) {struct list_head *lh = (struct list_head *) pgtable;
@@ -202,7 +203,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pgtable_t pgtable) mm->pmd_huge_pte = pgtable; } -pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm) +pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp) {struct list_head *lh;pgtable_t pgtable;
include/asm-generic/pgtable.h +3 -2
@@ -173,11 +173,12 @@ extern void pmdp_splitting_flush(struct vm_area_struct *vma,#endif #ifndef __HAVE_ARCH_PGTABLE_DEPOSIT -extern void pgtable_trans_huge_deposit (struct mm_struct *mm,pgtable_t pgtable); +extern void pgtable_trans_huge_deposit (struct mm_struct *mm,pmd_t *pmdp, +pgtable_t pgtable);#endif #ifndef __HAVE_ARCH_PGTABLE_WITHDRAW -extern pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm); +extern pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp);#endif #ifndef __HAVE_ARCH_PMDP_INVALIDATE
mm/huge_memory.c +9 -9
@@ -730,7 +730,7 @@ static int __do_huge_pmd_anonymous_page(struct mm_struct *mm, entry =mk_huge_pmd (page, vma);page_add_new_anon_rmap (page, vma, haddr);set_pmd_at (mm, haddr, pmd, entry); -pgtable_trans_huge_deposit (mm, pgtable); +pgtable_trans_huge_deposit (mm, pmd, pgtable);add_mm_counter (mm,MM_ANONPAGES ,HPAGE_PMD_NR ); mm->nr_ptes++;spin_unlock (&mm->page_table_lock);
@@ -772,7 +772,7 @@ static bool set_huge_zero_page(pgtable_t pgtable, struct mm_struct *mm, entry =pmd_wrprotect (entry); entry =pmd_mkhuge (entry);set_pmd_at (mm, haddr, pmd, entry); -pgtable_trans_huge_deposit (mm, pgtable); +pgtable_trans_huge_deposit (mm, pmd, pgtable); mm->nr_ptes++;return true; }
@@ -917,7 +917,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,pmdp_set_wrprotect (src_mm, addr, src_pmd); pmd =pmd_mkold (pmd_wrprotect (pmd));set_pmd_at (dst_mm, addr, dst_pmd, pmd); -pgtable_trans_huge_deposit (dst_mm, pgtable); +pgtable_trans_huge_deposit (dst_mm, dst_pmd, pgtable); dst_mm->nr_ptes++; ret =0 ;
@@ -987,7 +987,7 @@ static int do_huge_pmd_wp_zero_page_fallback(struct mm_struct *mm,pmdp_clear_flush (vma, haddr, pmd);/* leave pmd empty until pte is filled */ - pgtable =pgtable_trans_huge_withdraw (mm); + pgtable =pgtable_trans_huge_withdraw (mm, pmd);pmd_populate (mm, &_pmd, pgtable);for (i =0 ; i <HPAGE_PMD_NR ; i++, haddr +=PAGE_SIZE ) {
@@ -1085,7 +1085,7 @@ static int do_huge_pmd_wp_page_fallback(struct mm_struct *mm,pmdp_clear_flush (vma, haddr, pmd);/* leave pmd empty until pte is filled */ - pgtable =pgtable_trans_huge_withdraw (mm); + pgtable =pgtable_trans_huge_withdraw (mm, pmd);pmd_populate (mm, &_pmd, pgtable);for (i =0 ; i <HPAGE_PMD_NR ; i++, haddr +=PAGE_SIZE ) {
@@ -1360,7 +1360,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,struct page *page;pgtable_t pgtable;pmd_t orig_pmd; - pgtable =pgtable_trans_huge_withdraw (tlb->mm); + pgtable =pgtable_trans_huge_withdraw (tlb->mm, pmd); orig_pmd =pmdp_get_and_clear (tlb->mm, addr, pmd);tlb_remove_pmd_tlb_entry (tlb, pmd, addr);if (is_huge_zero_pmd (orig_pmd)) {
@@ -1693,7 +1693,7 @@ static int __split_huge_page_map(struct page *page, pmd =page_check_address_pmd (page, mm, address,PAGE_CHECK_ADDRESS_PMD_SPLITTING_FLAG );if (pmd) { - pgtable =pgtable_trans_huge_withdraw (mm); + pgtable =pgtable_trans_huge_withdraw (mm, pmd);pmd_populate (mm, &_pmd, pgtable); haddr = address;
@@ -2363,7 +2363,7 @@ static void collapse_huge_page(struct mm_struct *mm,page_add_new_anon_rmap (new_page, vma, address);set_pmd_at (mm, address, pmd, _pmd);update_mmu_cache_pmd (vma, address, pmd); -pgtable_trans_huge_deposit (mm, pgtable); +pgtable_trans_huge_deposit (mm, pmd, pgtable);spin_unlock (&mm->page_table_lock); *hpage =NULL ;
@@ -2669,7 +2669,7 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma,pmdp_clear_flush (vma, haddr, pmd);/* leave pmd empty until pte is filled */ - pgtable =pgtable_trans_huge_withdraw (mm); + pgtable =pgtable_trans_huge_withdraw (mm, pmd);pmd_populate (mm, &_pmd, pgtable);for (i =0 ; i <HPAGE_PMD_NR ; i++, haddr +=PAGE_SIZE ) {
mm/pgtable-generic.c +3 -2
@@ -124,7 +124,8 @@ void pmdp_splitting_flush(struct vm_area_struct *vma, unsigned long address, #ifndef __HAVE_ARCH_PGTABLE_DEPOSIT #ifdef CONFIG_TRANSPARENT_HUGEPAGE -void pgtable_trans_huge_deposit(struct mm_struct *mm, pgtable_t pgtable) +void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, + pgtable_t pgtable) { assert_spin_locked(&mm->page_table_lock);
@@ -141,7 +142,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pgtable_t pgtable) #ifndef __HAVE_ARCH_PGTABLE_WITHDRAW #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* no "address" argument so destroys page coloring of some arch */ -pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm) +pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) { pgtable_t pgtable;
Signature · 1 line
1.8.1.2
PATCH 4/20 mm/THP: withdraw the pgtable after pmdp related operations
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
For architectures like ppc64 we look at deposited pgtable when calling pmdp_get_and_clear. So do the pgtable_trans_huge_withdraw after finishing pmdp related operations.
--- mm/huge_memory.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
mm/huge_memory.c +7 -1
@@ -1360,9 +1360,15 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,struct page *page;pgtable_t pgtable;pmd_t orig_pmd; - pgtable =pgtable_trans_huge_withdraw (tlb->mm, pmd); +/* +* For architectures like ppc64 we look at deposited pgtable +* when calling pmdp_get_and_clear. So do the +* pgtable_trans_huge_withdraw after finishing pmdp related +* operations. +*/ orig_pmd =pmdp_get_and_clear (tlb->mm, addr, pmd);tlb_remove_pmd_tlb_entry (tlb, pmd, addr); + pgtable =pgtable_trans_huge_withdraw (tlb->mm, pmd);if (is_huge_zero_pmd (orig_pmd)) { tlb->mm->nr_ptes--;spin_unlock (&tlb->mm->page_table_lock);
Signature · 1 line
1.8.1.2
PATCH 5/20 mm/THP: Don't use HPAGE_SHIFT in transparent hugepage code
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
For architectures like powerpc that support multiple explicit hugepage sizes, HPAGE_SHIFT indicate the default explicit hugepage shift. For THP to work the hugepage size should be same as PMD_SIZE. So use PMD_SHIFT directly. So move the define outside CONFIG_TRANSPARENT_HUGEPAGE #ifdef because we want to use these defines in generic code with if (pmd_trans_huge()) conditional.
--- include/linux/huge_mm.h | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)
include/linux/huge_mm.h +3 -7
@@ -58,12 +58,11 @@ extern pmd_t *page_check_address_pmd(struct page *page,#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)#define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER) +#define HPAGE_PMD_SHIFT PMD_SHIFT +#define HPAGE_PMD_SIZE ((1UL) << HPAGE_PMD_SHIFT) +#define HPAGE_PMD_MASK (~(HPAGE_PMD_SIZE - 1))#ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define HPAGE_PMD_SHIFT HPAGE_SHIFT -#define HPAGE_PMD_MASK HPAGE_MASK -#define HPAGE_PMD_SIZE HPAGE_SIZE -extern bool is_vma_temporary_stack (struct vm_area_struct *vma);#define transparent_hugepage_enabled (__vma) \
@@ -181,9 +180,6 @@ extern int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vm unsigned long addr, pmd_t pmd, pmd_t *pmdp);#else /* CONFIG_TRANSPARENT_HUGEPAGE */ -#define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) -#define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; }) -#define HPAGE_PMD_SIZE ({ BUILD_BUG(); 0; })#define hpage_nr_pages (x) 1
Signature · 1 line
1.8.1.2
PATCH 6/20 mm/THP: deposit the transpare huge pgtable before set_pmd
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
Architectures like powerpc use the deposited pgtable to store hash index values. We need to make the deposted pgtable visible to other cpus before we are ready to take a hash fault.
--- mm/huge_memory.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
mm/huge_memory.c +4 -4
@@ -729,8 +729,8 @@ static int __do_huge_pmd_anonymous_page(struct mm_struct *mm,pmd_t entry; entry =mk_huge_pmd (page, vma);page_add_new_anon_rmap (page, vma, haddr); -set_pmd_at (mm, haddr, pmd, entry);pgtable_trans_huge_deposit (mm, pmd, pgtable); +set_pmd_at (mm, haddr, pmd, entry);add_mm_counter (mm,MM_ANONPAGES ,HPAGE_PMD_NR ); mm->nr_ptes++;spin_unlock (&mm->page_table_lock);
@@ -771,8 +771,8 @@ static bool set_huge_zero_page(pgtable_t pgtable, struct mm_struct *mm, entry =mk_pmd (zero_page, vma->vm_page_prot); entry =pmd_wrprotect (entry); entry =pmd_mkhuge (entry); -set_pmd_at (mm, haddr, pmd, entry);pgtable_trans_huge_deposit (mm, pmd, pgtable); +set_pmd_at (mm, haddr, pmd, entry); mm->nr_ptes++;return true; }
@@ -916,8 +916,8 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,pmdp_set_wrprotect (src_mm, addr, src_pmd); pmd =pmd_mkold (pmd_wrprotect (pmd)); -set_pmd_at (dst_mm, addr, dst_pmd, pmd);pgtable_trans_huge_deposit (dst_mm, dst_pmd, pgtable); +set_pmd_at (dst_mm, addr, dst_pmd, pmd); dst_mm->nr_ptes++; ret =0 ;
@@ -2367,9 +2367,9 @@ static void collapse_huge_page(struct mm_struct *mm,spin_lock (&mm->page_table_lock);BUG_ON (!pmd_none (*pmd));page_add_new_anon_rmap (new_page, vma, address); +pgtable_trans_huge_deposit (mm, pmd, pgtable);set_pmd_at (mm, address, pmd, _pmd);update_mmu_cache_pmd (vma, address, pmd); -pgtable_trans_huge_deposit (mm, pmd, pgtable);spin_unlock (&mm->page_table_lock); *hpage =NULL ;
Signature · 1 line
1.8.1.2
PATCH 7/20 powerpc/THP: Double the PMD table size for THP
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
THP code does PTE page allocation along with large page request and deposit them for later use. This is to ensure that we won't have any failures when we split hugepages to regular pages.
On powerpc we want to use the deposited PTE page for storing hash pte slot and secondary bit information for the HPTEs. We use the second half of the pmd table to save the deposted PTE page.
--- arch/powerpc/include/asm/pgalloc-64.h | 6 +++--- arch/powerpc/include/asm/pgtable-ppc64-64k.h | 3 ++- arch/powerpc/include/asm/pgtable-ppc64.h | 6 +++++- arch/powerpc/mm/init_64.c | 9 ++++++--- 4 files changed, 16 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/pgalloc-64.h +3 -3
@@ -221,17 +221,17 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table,static inline pmd_t *pmd_alloc_one (struct mm_struct *mm,unsigned long addr) { -return kmem_cache_alloc (PGT_CACHE (PMD_INDEX_SIZE ), +return kmem_cache_alloc (PGT_CACHE (PMD_CACHE_INDEX ),GFP_KERNEL |__GFP_REPEAT); }static inline void pmd_free (struct mm_struct *mm,pmd_t *pmd) { -kmem_cache_free (PGT_CACHE (PMD_INDEX_SIZE ), pmd); +kmem_cache_free (PGT_CACHE (PMD_CACHE_INDEX ), pmd); }#define __pmd_free_tlb (tlb, pmd, addr) \ - pgtable_free_tlb(tlb, pmd, PMD_INDEX_SIZE) + pgtable_free_tlb(tlb, pmd, PMD_CACHE_INDEX)#ifndef CONFIG_PPC_64K_PAGES #define __pud_free_tlb (tlb, pud, addr) \ pgtable_free_tlb(tlb, pud, PUD_INDEX_SIZE)
arch/powerpc/include/asm/pgtable-ppc64-64k.h +2 -1
@@ -33,7 +33,8 @@#define PGDIR_MASK (~(PGDIR_SIZE-1))/* Bits to mask out from a PMD to get to the PTE page */ -#define PMD_MASKED_BITS 0x1ff +/* PMDs point to PTE table fragments which are 4K aligned. */ +#define PMD_MASKED_BITS 0xfff/* Bits to mask out from a PGD/PUD to get to the PMD page */ #define PUD_MASKED_BITS 0x1ff
arch/powerpc/include/asm/pgtable-ppc64.h +5 -1
@@ -20,7 +20,11 @@ PUD_INDEX_SIZE + PGD_INDEX_SIZE + PAGE_SHIFT)#define PGTABLE_RANGE (ASM_CONST(1) << PGTABLE_EADDR_SIZE) - +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define PMD_CACHE_INDEX (PMD_INDEX_SIZE + 1) +#else +#define PMD_CACHE_INDEX PMD_INDEX_SIZE +#endif /* * Define the address range of the kernel non-linear virtual area */
arch/powerpc/mm/init_64.c +6 -3
@@ -88,7 +88,11 @@ static void pgd_ctor(void *addr)static void pmd_ctor (void *addr) { +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +memset (addr,0 ,PMD_TABLE_SIZE *2 ); +#else memset (addr,0 ,PMD_TABLE_SIZE ); +#endif }struct kmem_cache *pgtable_cache[MAX_PGTABLE_INDEX_SIZE ];
@@ -137,10 +141,9 @@ void pgtable_cache_add(unsigned shift, void (*ctor)(void *))void pgtable_cache_init (void ) {pgtable_cache_add (PGD_INDEX_SIZE , pgd_ctor); -pgtable_cache_add (PMD_INDEX_SIZE , pmd_ctor); -if (!PGT_CACHE (PGD_INDEX_SIZE ) || !PGT_CACHE (PMD_INDEX_SIZE )) +pgtable_cache_add (PMD_CACHE_INDEX , pmd_ctor); +if (!PGT_CACHE (PGD_INDEX_SIZE ) || !PGT_CACHE (PMD_CACHE_INDEX ))panic ("Couldn't allocate pgtable caches" ); -/* In all current configs, when the PUD index exists it's the * same size as either the pgd or pmd index. Verify that the * initialization above has also created a PUD cache. This
Signature · 1 line
1.8.1.2
PATCH 8/20 powerpc/THP: Implement transparent hugepages for ppc64
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
We now have pmd entries covering 16MB range and the PMD table double its original size. We use the second half of the PMD table to deposit the pgtable (PTE page). The depoisted PTE page is further used to track the HPTE information. The information include [ secondary group | 3 bit hidx | valid ]. We use one byte per each HPTE entry. With 16MB hugepage and 64K HPTE we need 256 entries and with 4K HPTE we need 4096 entries. Both will fit in a 4K PTE page. On hugepage invalidate we need to walk the PTE page and invalidate all valid HPTEs.
This patch implements necessary arch specific functions for THP support and also hugepage invalidate logic. These PMD related functions are intentionally kept similar to their PTE counter-part.
--- arch/powerpc/include/asm/pgtable-ppc64.h | 200 ++++++++++++++++- arch/powerpc/include/asm/pgtable.h | 4 + arch/powerpc/include/asm/tlbflush.h | 3 +- arch/powerpc/mm/pgtable_64.c | 364 +++++++++++++++++++++++++++++++ arch/powerpc/mm/tlb_hash64.c | 27 +++ arch/powerpc/platforms/Kconfig.cputype | 1 + 6 files changed, 597 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/pgtable-ppc64.h +199 -1
@@ -154,7 +154,7 @@#define pmd_present (pmd) (pmd_val(pmd) != 0)#define pmd_clear (pmdp) (pmd_val(*(pmdp)) = 0)#define pmd_page_vaddr (pmd) (pmd_val(pmd) & ~PMD_MASKED_BITS) -#define pmd_page (pmd) virt_to_page(pmd_page_vaddr(pmd)) +extern struct page *pmd_page (pmd_t pmd);#define pud_set (pudp, pudval) (pud_val(*(pudp)) = (pudval))#define pud_none (pud) (!pud_val(pud))
@@ -382,4 +382,202 @@ static inline pte_t *find_linux_pte_or_hugepte(pgd_t *pgdir, unsigned long ea,#endif /* __ASSEMBLY__ */ +/* +* THP pages can't be special. So use the _PAGE_SPECIAL +*/ +#define _PAGE_SPLITTING _PAGE_SPECIAL + +/* +* We need to differentiate between explicit huge page and THP huge +* page, since THP huge page also need to track real subpage details +*/ +#define _PAGE_THP_HUGE _PAGE_4K_PFN + +/* +* set of bits not changed in pmd_modify. +*/ +#define _HPAGE_CHG_MASK (PTE_RPN_MASK | _PAGE_HPTEFLAGS | \ + _PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_SPLITTING | \ + _PAGE_THP_HUGE) + +#ifndef __ASSEMBLY__ +/* +* The linux hugepage PMD now include the pmd entries followed by the address +* to the stashed pgtable_t. The stashed pgtable_t contains the hpte bits. +* [ 1 bit secondary | 3 bit hidx | 1 bit valid | 000]. We use one byte per +* each HPTE entry. With 16MB hugepage and 64K HPTE we need 256 entries and +* with 4K HPTE we need 4096 entries. Both will fit in a 4K pgtable_t. +* +* The last three bits are intentionally left to zero. This memory location +* are also used as normal page PTE pointers. So if we have any pointers +* left around while we collapse a hugepage, we need to make sure +* _PAGE_PRESENT and _PAGE_FILE bits of that are zero when we look at them +*/ +static inline unsigned int hpte_valid (unsigned char *hpte_slot_array,int index) +{ +return (hpte_slot_array[index] >>3 ) &0x1 ; +} + +static inline unsigned int hpte_hash_index (unsigned char *hpte_slot_array, +int index) +{ +return hpte_slot_array[index] >>4 ; +} + +static inline void mark_hpte_slot_valid (unsigned char *hpte_slot_array, +unsigned int index,unsigned int hidx) +{ + hpte_slot_array[index] = hidx <<4 |0x1 <<3 ; +} + +extern void hpte_do_hugepage_flush (struct mm_struct *mm,unsigned long addr, +pmd_t *pmdp); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +extern pmd_t pfn_pmd (unsigned long pfn,pgprot_t pgprot); +extern pmd_t mk_pmd (struct page *page,pgprot_t pgprot); +extern pmd_t pmd_modify (pmd_t pmd,pgprot_t newprot); +extern void set_pmd_at (struct mm_struct *mm,unsigned long addr, +pmd_t *pmdp,pmd_t pmd); +extern void update_mmu_cache_pmd (struct vm_area_struct *vma,unsigned long addr, +pmd_t *pmd); + +static inline int pmd_trans_huge (pmd_t pmd) +{ +/* +* leaf pte for huge page, bottom two bits != 00 +*/ +return (pmd_val (pmd) &0x3 ) && (pmd_val (pmd) & _PAGE_THP_HUGE); +} + +static inline int pmd_large (pmd_t pmd) +{ +/* +* leaf pte for huge page, bottom two bits != 00 +*/ +if (pmd_trans_huge (pmd)) +return pmd_val (pmd) & _PAGE_PRESENT; +return 0 ; +} + +static inline int pmd_trans_splitting (pmd_t pmd) +{ +if (pmd_trans_huge (pmd)) +return pmd_val (pmd) & _PAGE_SPLITTING; +return 0 ; +} + +/* We will enable it in the last patch */ +#define has_transparent_hugepage () 0 +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +static inline pte_t pmd_pte (pmd_t pmd) +{ +return __pte (pmd_val (pmd)); +} + +static inline pmd_t pte_pmd (pte_t pte) +{ +return __pmd (pte_val (pte)); +} + +static inline pte_t *pmdp_ptep (pmd_t *pmd) +{ +return (pte_t *)pmd; +} + +#define pmd_pfn (pmd) pte_pfn(pmd_pte(pmd)) +#define pmd_young (pmd) pte_young(pmd_pte(pmd)) +#define pmd_mkold (pmd) pte_pmd(pte_mkold(pmd_pte(pmd))) +#define pmd_wrprotect (pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd))) +#define pmd_mkdirty (pmd) pte_pmd(pte_mkdirty(pmd_pte(pmd))) +#define pmd_mkyoung (pmd) pte_pmd(pte_mkyoung(pmd_pte(pmd))) +#define pmd_mkwrite (pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd))) + +#define __HAVE_ARCH_PMD_WRITE +#define pmd_write (pmd) pte_write(pmd_pte(pmd)) + +static inline pmd_t pmd_mkhuge (pmd_t pmd) +{ +/* Do nothing, mk_pmd() does this part. */ +return pmd; +} + +static inline pmd_t pmd_mknotpresent (pmd_t pmd) +{ +pmd_val (pmd) &= ~_PAGE_PRESENT; +return pmd; +} + +static inline pmd_t pmd_mksplitting (pmd_t pmd) +{ +pmd_val (pmd) |= _PAGE_SPLITTING; +return pmd; +} + +#define __HAVE_ARCH_PMD_SAME +static inline int pmd_same (pmd_t pmd_a,pmd_t pmd_b) +{ +return (((pmd_val (pmd_a) ^pmd_val (pmd_b)) & ~_PAGE_HPTEFLAGS) ==0 ); +} + +#define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS +extern int pmdp_set_access_flags (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmdp, +pmd_t entry,int dirty); + +extern unsigned long pmd_hugepage_update (struct mm_struct *mm, +unsigned long addr, +pmd_t *pmdp,unsigned long clr); + +static inline int __pmdp_test_and_clear_young (struct mm_struct *mm, +unsigned long addr,pmd_t *pmdp) +{ +unsigned long old; + +if ((pmd_val (*pmdp) & (_PAGE_ACCESSED | _PAGE_HASHPTE)) ==0 ) +return 0 ; + old =pmd_hugepage_update (mm, addr, pmdp, _PAGE_ACCESSED); +return ((old & _PAGE_ACCESSED) !=0 ); +} + +#define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG +extern int pmdp_test_and_clear_young (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmdp); +#define __HAVE_ARCH_PMDP_CLEAR_YOUNG_FLUSH +extern int pmdp_clear_flush_young (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmdp); + +#define __HAVE_ARCH_PMDP_GET_AND_CLEAR +extern pmd_t pmdp_get_and_clear (struct mm_struct *mm, +unsigned long addr,pmd_t *pmdp); + +#define __HAVE_ARCH_PMDP_CLEAR_FLUSH +extern pmd_t pmdp_clear_flush (struct vm_area_struct *vma,unsigned long address, +pmd_t *pmdp); + +#define __HAVE_ARCH_PMDP_SET_WRPROTECT +static inline void pmdp_set_wrprotect (struct mm_struct *mm,unsigned long addr, +pmd_t *pmdp) +{ + +if ((pmd_val (*pmdp) & _PAGE_RW) ==0 ) +return ; + +pmd_hugepage_update (mm, addr, pmdp, _PAGE_RW); +} + +#define __HAVE_ARCH_PMDP_SPLITTING_FLUSH +extern void pmdp_splitting_flush (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmdp); + +#define __HAVE_ARCH_PGTABLE_DEPOSIT +extern void pgtable_trans_huge_deposit (struct mm_struct *mm,pmd_t *pmdp, +pgtable_t pgtable); +#define __HAVE_ARCH_PGTABLE_WITHDRAW +extern pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp); + +#define __HAVE_ARCH_PMDP_INVALIDATE +extern void pmdp_invalidate (struct vm_area_struct *vma,unsigned long address, +pmd_t *pmdp); +#endif /* __ASSEMBLY__ */ #endif /* _ASM_POWERPC_PGTABLE_PPC64_H_ */
arch/powerpc/include/asm/pgtable.h +4 -0
@@ -220,6 +220,10 @@ extern int gup_hugepd(hugepd_t *hugepd, unsigned pdshift, unsigned long addr,extern int gup_hugepte (pte_t *ptep,unsigned long sz,unsigned long addr,unsigned long end,int write,struct page **pages,int *nr); +#ifndef CONFIG_TRANSPARENT_HUGEPAGE +#define pmd_large (pmd) 0 +#define has_transparent_hugepage () 0 +#endif #endif /* __ASSEMBLY__ */ #endif /* __KERNEL__ */
arch/powerpc/include/asm/tlbflush.h +2 -1
@@ -165,7 +165,8 @@ static inline void flush_tlb_kernel_range(unsigned long start,/* Private function for use by PCI IO mapping code */ extern void __flush_hash_table_range (struct mm_struct *mm,unsigned long start,unsigned long end); - +extern void flush_tlb_pmd_range (struct mm_struct *mm,pmd_t *pmd, +unsigned long addr);#else #error Unsupported MMU type#endif
arch/powerpc/mm/pgtable_64.c +364 -0
373 lines not on this page · raw message
arch/powerpc/mm/tlb_hash64.c +27 -0
@@ -219,3 +219,30 @@ void __flush_hash_table_range(struct mm_struct *mm, unsigned long start,arch_leave_lazy_mmu_mode ();local_irq_restore (flags); } + +void flush_tlb_pmd_range (struct mm_struct *mm,pmd_t *pmd,unsigned long addr) +{ +pte_t *pte; +pte_t *start_pte; +unsigned long flags; + + addr =_ALIGN_DOWN (addr,PMD_SIZE ); +/* Note: Normally, we should only ever use a batch within a +* PTE locked section. This violates the rule, but will work +* since we don't actually modify the PTEs, we just flush the +* hash while leaving the PTEs intact (including their reference +* to being hashed). This is not the most performance oriented +* way to do things but is fine for our needs here. +*/ +local_irq_save (flags); +arch_enter_lazy_mmu_mode (); + start_pte =pte_offset_map (pmd, addr); +for (pte = start_pte; pte < start_pte +PTRS_PER_PTE ; pte++) { +unsigned long pteval =pte_val (*pte); +if (pteval & _PAGE_HASHPTE) +hpte_need_flush (mm, addr, pte, pteval,0 ); + addr +=PAGE_SIZE ; + } +arch_leave_lazy_mmu_mode (); +local_irq_restore (flags); +}
arch/powerpc/platforms/Kconfig.cputype +1 -0
@@ -71,6 +71,7 @@ config PPC_BOOK3S_64 select PPC_FPU select PPC_HAVE_PMU_SUPPORT select SYS_SUPPORTS_HUGETLBFS + select HAVE_ARCH_TRANSPARENT_HUGEPAGE if PPC_64K_PAGES config PPC_BOOK3E_64 bool "Embedded processors"
Signature · 1 line
1.8.1.2
PATCH 9/20 powerpc: move find_linux_pte_or_hugepte and gup_hugepte to common code
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
We will use this in the later patch for handling THP pages
--- arch/powerpc/include/asm/hugetlb.h | 8 +- arch/powerpc/include/asm/pgtable-ppc64.h | 13 -- arch/powerpc/include/asm/pgtable.h | 2 + arch/powerpc/mm/Makefile | 2 +- arch/powerpc/mm/hugetlbpage.c | 251 ++++++++++++++++--------------- 5 files changed, 138 insertions(+), 138 deletions(-)
arch/powerpc/include/asm/hugetlb.h +7 -1
@@ -191,8 +191,14 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr) { } -#endif /* CONFIG_HUGETLB_PAGE */ +#define hugepd_shift (x) 0 +static inline pte_t *hugepte_offset (hugepd_t *hpdp,unsigned long addr, +unsigned pdshift) +{ +return 0 ; +} +#endif /* CONFIG_HUGETLB_PAGE */ /* * FSL Book3E platforms require special gpage handling - the gpages
arch/powerpc/include/asm/pgtable-ppc64.h +0 -13
@@ -367,19 +367,6 @@ static inline pte_t *find_linux_pte(pgd_t *pgdir, unsigned long ea) return pt; } -#ifdef CONFIG_HUGETLB_PAGE -pte_t *find_linux_pte_or_hugepte (pgd_t *pgdir,unsigned long ea, -unsigned *shift); -#else -static inline pte_t *find_linux_pte_or_hugepte (pgd_t *pgdir,unsigned long ea, -unsigned *shift) -{ -if (shift) - *shift =0 ; -return find_linux_pte (pgdir, ea); -} -#endif /* !CONFIG_HUGETLB_PAGE */ -#endif /* __ASSEMBLY__ */ /*
arch/powerpc/include/asm/pgtable.h +2 -0
@@ -224,6 +224,8 @@ extern int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,#define pmd_large (pmd) 0#define has_transparent_hugepage () 0#endif +pte_t *find_linux_pte_or_hugepte (pgd_t *pgdir,unsigned long ea, +unsigned *shift);#endif /* __ASSEMBLY__ */ #endif /* __KERNEL__ */
arch/powerpc/mm/Makefile +1 -1
@@ -28,8 +28,8 @@ obj-$(CONFIG_44x) += 44x_mmu.o obj-$(CONFIG_PPC_FSL_BOOK3E) += fsl_booke_mmu.o obj-$(CONFIG_NEED_MULTIPLE_NODES) += numa.o obj-$(CONFIG_PPC_MM_SLICES) += slice.o -ifeq ($(CONFIG_HUGETLB_PAGE),y) obj-y += hugetlbpage.o +ifeq ($(CONFIG_HUGETLB_PAGE),y) obj-$(CONFIG_PPC_STD_MMU_64) += hugetlbpage-hash64.o obj-$(CONFIG_PPC_BOOK3E_MMU) += hugetlbpage-book3e.o endif
arch/powerpc/mm/hugetlbpage.c +128 -123
272 lines not on this page · raw message
Signature · 1 line
1.8.1.2
PATCH 10/20 powerpc: Update find_linux_pte_or_hugepte to handle transparent hugepages
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
--- arch/powerpc/mm/hugetlbpage.c | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-)
arch/powerpc/mm/hugetlbpage.c +26 -6
@@ -936,30 +936,50 @@ pte_t *find_linux_pte_or_hugepte(pgd_t *pgdir, unsigned long ea, unsigned *shift pg = pgdir +pgd_index (ea); -if (pgd_huge (*pg)) { +/* +* we should first check for none. That takes care of a +* a parallel hugetlb or THP pagefault moving none entries +* to respective types. +*/ +if (pgd_none (*pg)) +return NULL ; +else if (pgd_huge (*pg)) { ret_pte = (pte_t *) pg; goto out; }else if (is_hugepd (pg)) hpdp = (hugepd_t *)pg; -else if (!pgd_none (*pg)) { +else { pdshift =PUD_SHIFT ; pu =pud_offset (pg, ea); -if (pud_huge (*pu)) { +if (pud_none (*pu)) +return NULL ; +else if (pud_huge (*pu)) { ret_pte = (pte_t *) pu; goto out; }else if (is_hugepd (pu)) hpdp = (hugepd_t *)pu; -else if (!pud_none (*pu)) { +else { pdshift =PMD_SHIFT ; pm =pmd_offset (pu, ea); +/* +* A hugepage collapse is captured by pmd_none, because +* it mark the pmd none and do a hpte invalidate. +* +* A hugepage split is captured by pmd_trans_splitting +* because we mark the pmd trans splitting and do a +* hpte invalidate +* +*/ +if (pmd_none (*pm) ||pmd_trans_splitting (*pm)) +return NULL ; -if (pmd_huge (*pm)) { +if (pmd_huge (*pm) ||pmd_large (*pm)) { ret_pte = (pte_t *) pm; goto out; }else if (is_hugepd (pm)) hpdp = (hugepd_t *)pm; -else if (!pmd_none (*pm)) +else return pte_offset_kernel (pm, ea); } }
Signature · 1 line
1.8.1.2
PATCH 11/20 powerpc: Replace find_linux_pte with find_linux_pte_or_hugepte
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
Replace find_linux_pte with find_linux_pte_or_hugepte and explicitly document why we don't need to handle transparent hugepages at callsites.
--- arch/powerpc/include/asm/pgtable-ppc64.h | 24 ------------------------ arch/powerpc/kernel/io-workarounds.c | 10 ++++++++-- arch/powerpc/kvm/book3s_hv_rm_mmu.c | 2 +- arch/powerpc/mm/hash_utils_64.c | 8 +++++++- arch/powerpc/mm/hugetlbpage.c | 8 ++++++-- arch/powerpc/mm/tlb_hash64.c | 9 +++++++-- arch/powerpc/platforms/pseries/eeh.c | 7 ++++++- 7 files changed, 35 insertions(+), 33 deletions(-)
arch/powerpc/include/asm/pgtable-ppc64.h +0 -24
@@ -343,30 +343,6 @@ static inline void __ptep_set_access_flags(pte_t *ptep, pte_t entry)void pgtable_cache_add (unsigned shift,void (*ctor)(void *));void pgtable_cache_init (void ); - -/* -* find_linux_pte returns the address of a linux pte for a given -* effective address and directory. If not found, it returns zero. -*/ -static inline pte_t *find_linux_pte (pgd_t *pgdir,unsigned long ea) -{ -pgd_t *pg; -pud_t *pu; -pmd_t *pm; -pte_t *pt =NULL ; - - pg = pgdir +pgd_index (ea); -if (!pgd_none (*pg)) { - pu =pud_offset (pg, ea); -if (!pud_none (*pu)) { - pm =pmd_offset (pu, ea); -if (pmd_present (*pm)) - pt =pte_offset_kernel (pm, ea); - } - } -return pt; -} -#endif /* __ASSEMBLY__ */ /*
arch/powerpc/kernel/io-workarounds.c +8 -2
@@ -55,6 +55,7 @@ static struct iowa_bus *iowa_pci_find(unsigned long vaddr, unsigned long paddr)struct iowa_bus *iowa_mem_find_bus (const PCI_IO_ADDR addr) { +unsigned shift;struct iowa_bus *bus;int token;
@@ -70,11 +71,16 @@ struct iowa_bus *iowa_mem_find_bus(const PCI_IO_ADDR addr)if (vaddr <PHB_IO_BASE || vaddr >=PHB_IO_END )return NULL ; - ptep =find_linux_pte (init_mm.pgd, vaddr); + ptep =find_linux_pte_or_hugepte (init_mm.pgd, vaddr, &shift);if (ptep ==NULL ) paddr =0 ; -else +else { +/* +* we don't have hugepages backing iomem +*/ +BUG_ON (shift); paddr =pte_pfn (*ptep) <<PAGE_SHIFT ; + } bus =iowa_pci_find (vaddr, paddr);if (bus ==NULL )
arch/powerpc/kvm/book3s_hv_rm_mmu.c +1 -1
@@ -27,7 +27,7 @@ static void *real_vmalloc_addr(void *x)unsigned long addr = (unsigned long ) x;pte_t *p; - p =find_linux_pte (swapper_pg_dir, addr); + p =find_linux_pte_or_hugepte (swapper_pg_dir, addr,NULL );if (!p || !pte_present (*p))return NULL ;/* assume we don't have huge pages in vmalloc space... */
arch/powerpc/mm/hash_utils_64.c +7 -1
@@ -1145,6 +1145,7 @@ EXPORT_SYMBOL_GPL(hash_page);void hash_preload (struct mm_struct *mm,unsigned long ea,unsigned long access,unsigned long trap) { +int shift;unsigned long vsid;pgd_t *pgdir;pte_t *ptep;
@@ -1166,10 +1167,15 @@ void hash_preload(struct mm_struct *mm, unsigned long ea, pgdir = mm->pgd;if (pgdir ==NULL )return ; - ptep =find_linux_pte (pgdir, ea); +/* +* THP pages use update_mmu_cache_pmd. We don't do +* hash preload there. Hence can ignore THP here +*/ + ptep =find_linux_pte_or_hugepte (pgdir, ea, &shift);if (!ptep)return ; +BUG_ON (shift);#ifdef CONFIG_PPC_64K_PAGES /* If either _PAGE_4K_PFN or _PAGE_NO_CACHE is set (and we are on * a 64K kernel), then we don't preload, hash_page() will take
arch/powerpc/mm/hugetlbpage.c +6 -2
@@ -105,6 +105,7 @@ int pgd_huge(pgd_t pgd)pte_t *huge_pte_offset (struct mm_struct *mm,unsigned long addr) { +/* Only called for hugetlbfs pages, hence can ignore THP */ return find_linux_pte_or_hugepte (mm->pgd, addr,NULL ); }
@@ -673,11 +674,14 @@ follow_huge_addr(struct mm_struct *mm, unsigned long address, int write)struct page *page;unsigned shift;unsigned long mask; - +/* +* Transparent hugepages are handled by generic code. We can skip them +* here. +*/ ptep =find_linux_pte_or_hugepte (mm->pgd, address, &shift);/* Verify it is a huge page else bail. */ -if (!ptep || !shift) +if (!ptep || !shift ||pmd_trans_huge (*(pmd_t *)ptep))return ERR_PTR (-EINVAL ); mask = (1UL << shift) -1 ;
arch/powerpc/mm/tlb_hash64.c +7 -2
@@ -189,6 +189,7 @@ void tlb_flush(struct mmu_gather *tlb)void __flush_hash_table_range (struct mm_struct *mm,unsigned long start,unsigned long end) { +int hugepage_shift;unsigned long flags; start =_ALIGN_DOWN (start,PAGE_SIZE );
@@ -206,7 +207,8 @@ void __flush_hash_table_range(struct mm_struct *mm, unsigned long start,local_irq_save (flags);arch_enter_lazy_mmu_mode ();for (; start < end; start +=PAGE_SIZE ) { -pte_t *ptep =find_linux_pte (mm->pgd, start); +pte_t *ptep =find_linux_pte_or_hugepte (mm->pgd, start, + &hugepage_shift);unsigned long pte;if (ptep ==NULL )
@@ -214,7 +216,10 @@ void __flush_hash_table_range(struct mm_struct *mm, unsigned long start, pte =pte_val (*ptep);if (!(pte & _PAGE_HASHPTE))continue ; -hpte_need_flush (mm, start, ptep, pte,0 ); +if (unlikely (hugepage_shift &&pmd_trans_huge (*(pmd_t *)pte))) +hpte_do_hugepage_flush (mm, start, (pmd_t *)pte); +else +hpte_need_flush (mm, start, ptep, pte,0 ); }arch_leave_lazy_mmu_mode ();local_irq_restore (flags);
arch/powerpc/platforms/pseries/eeh.c +6 -1
@@ -258,12 +258,17 @@ void eeh_slot_error_detail(struct eeh_pe *pe, int severity)*/ static inline unsigned long eeh_token_to_phys (unsigned long token) { +int shift;pte_t *ptep;unsigned long pa; - ptep =find_linux_pte (init_mm.pgd, token); +/* +* We won't find hugepages here, iomem +*/ + ptep =find_linux_pte_or_hugepte (init_mm.pgd, token, &shift);if (!ptep)return token; +BUG_ON (shift); pa =pte_pfn (*ptep) <<PAGE_SHIFT ;return pa | (token & (PAGE_SIZE -1 ));
Signature · 1 line
1.8.1.2
PATCH 12/20 powerpc: Update gup_pmd_range to handle transparent hugepages
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
--- arch/powerpc/mm/gup.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
arch/powerpc/mm/gup.c +13 -2
@@ -66,9 +66,20 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,pmd_t pmd = *pmdp; next =pmd_addr_end (addr, end); -if (pmd_none (pmd)) +/* +* The pmd_trans_splitting() check below explains why +* pmdp_splitting_flush has to flush the tlb, to stop +* this gup-fast code from running while we set the +* splitting bit in the pmd. Returning zero will take +* the slow path that will call wait_split_huge_page() +* if the pmd is still in splitting state. gup-fast +* can't because it has irq disabled and +* wait_split_huge_page() would never return as the +* tlb flush IPI wouldn't run. +*/ +if (pmd_none (pmd) ||pmd_trans_splitting (pmd))return 0 ; -if (pmd_huge (pmd)) { +if (pmd_huge (pmd) ||pmd_large (pmd)) {if (!gup_hugepte ((pte_t *)pmdp,PMD_SIZE , addr, next, write, pages, nr))return 0 ;
Signature · 1 line
1.8.1.2
PATCH 13/20 powerpc/THP: Add code to handle HPTE faults for hugepages
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
The deposted PTE page in the second half of the PMD table is used to track the state on hash PTEs. After updating the HPTE, we mark the coresponding slot in the deposted PTE page valid.
--- arch/powerpc/include/asm/mmu-hash64.h | 13 +++ arch/powerpc/mm/Makefile | 1 + arch/powerpc/mm/hash_utils_64.c | 21 +++- arch/powerpc/mm/hugepage-hash64.c | 176 ++++++++++++++++++++++++++++++++++ 4 files changed, 207 insertions(+), 4 deletions(-) create mode 100644 arch/powerpc/mm/hugepage-hash64.c
arch/powerpc/include/asm/mmu-hash64.h +13 -0
@@ -340,6 +340,19 @@ extern int hash_page(unsigned long ea, unsigned long access, unsigned long trap)int __hash_page_huge (unsigned long ea,unsigned long access,unsigned long vsid,pte_t *ptep,unsigned long trap,int local,int ssize,unsigned int shift,unsigned int mmu_psize); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +extern int __hash_page_thp (unsigned long ea,unsigned long access, +unsigned long vsid,pmd_t *pmdp,unsigned long trap, +int local,int ssize,unsigned int psize); +#else +static inline int __hash_page_thp (unsigned long ea,unsigned long access, +unsigned long vsid,pmd_t *pmdp, +unsigned long trap,int local, +int ssize,unsigned int psize) +{ +BUG (); +} +#endif extern void hash_failure_debug(unsigned long ea, unsigned long access, unsigned long vsid, unsigned long trap, int ssize, int psize, int lpsize,
arch/powerpc/mm/Makefile +1 -0
@@ -33,6 +33,7 @@ ifeq ($(CONFIG_HUGETLB_PAGE),y) obj-$(CONFIG_PPC_STD_MMU_64) += hugetlbpage-hash64.o obj-$(CONFIG_PPC_BOOK3E_MMU) += hugetlbpage-book3e.o endif +obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += hugepage-hash64.o obj-$(CONFIG_PPC_SUBPAGE_PROT) += subpage-prot.o obj-$(CONFIG_NOT_COHERENT_CACHE) += dma-noncoherent.o obj-$(CONFIG_HIGHMEM) += highmem.o
arch/powerpc/mm/hash_utils_64.c +17 -4
@@ -1050,13 +1050,26 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap) goto bail; } -#ifdef CONFIG_HUGETLB_PAGE if (hugeshift) { - rc =__hash_page_huge (ea, access, vsid, ptep, trap, local, - ssize, hugeshift, psize); +if (pmd_trans_huge (*(pmd_t *)ptep)) + rc =__hash_page_thp (ea, access, vsid, (pmd_t *)ptep, + trap, local, ssize, psize); +#ifdef CONFIG_HUGETLB_PAGE + else + rc =__hash_page_huge (ea, access, vsid, ptep, trap, + local, ssize, hugeshift, psize); +#else + else { +/* +* if we have hugeshift, and is not transhuge with +* hugetlb disabled, something is really wrong. +*/ + rc =1 ; +WARN_ON (1 ); + } +#endif goto bail; } -#endif /* CONFIG_HUGETLB_PAGE */ #ifndef CONFIG_PPC_64K_PAGES DBG_LOW (" i-pte: %016lx\n" ,pte_val (*ptep));
arch/powerpc/mm/hugepage-hash64.c new file +176 -0
176 lines not on this page · raw message
Signature · 1 line
1.8.1.2
PATCH 14/20 powerpc: Make linux pagetable walk safe with THP enabled
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
We need to have irqs disabled to handle all the possible parallel update for linux page table without holding locks.
Events that we are intersted in while walking page tables are
1) Page fault
2) umap
3) THP split
4) THP collapse
A) local_irq_disabled:
------------------------
1) page fault:
A none to valid transition via page fault is not an issue because we would either see a none or valid. If it is none, we would error out the page table walk. We may need to use on stack values when checking for type of page table elements, because if we do
if (!is_hugepd()) {
if (!pmd_none() {
if (pmd_bad() {We could take that bad condition because the pmd got converted to a hugepd after the !is_hugepd check via a hugetlb fault.
The right way would be to check for pmd_none higher up or use on stack value.
2) A valid to none conversion via unmap:
We can safely walk the upper level table, because we don't remove the the page table entries until rcu grace period. So even if we followed a wrong pointer we still have the pointer valid till the grace period.
A PTE pointer returned need to be atomically checked for _PAGE_PRESENT and
_PAGE_BUSY. A valid pointer returned could becoming none later. To preventpte_clear we take _PAGE_BUSY.
3) THP split:
A valid transparent hugepage is converted to nomal page. Before we split we do pmd_splitting_flush, which sets the hugepage PTE to _PAGE_SPLITTING So when walking page table we need to check for pmd_trans_splitting and handle that. The pte returned should also need to be checked for _PAGE_SPLITTING before setting _PAGE_BUSY similar to _PAGE_PRESENT. We save the value of PTE on stack and check for the flag in the local pte value. If we don't have the value set we can safely operate on the local pte value and we atomicaly set _PAGE_BUSY.
4) THP collapse:
A normal page gets converted to hugepage. In the collapse path, we mark the pmd none early (pmdp_clear_flush). With irq disabled, if we are aleady walking page table we would see the pmd_none and won't continue. If we see a valid PMD, we should still check for _PAGE_PRESENT before setting _PAGE_BUSY, to make sure we didn't collapse the PTE to a Huge PTE.
--- arch/powerpc/mm/hash_utils_64.c | 27 ++++++++------- arch/powerpc/mm/hugepage-hash64.c | 3 ++ arch/powerpc/mm/hugetlbpage.c | 72 +++++++++++++++++++++++++-------------- arch/powerpc/mm/mem.c | 4 +++ 4 files changed, 68 insertions(+), 38 deletions(-)
arch/powerpc/mm/hash_utils_64.c +15 -12
@@ -1180,13 +1180,25 @@ void hash_preload(struct mm_struct *mm, unsigned long ea, pgdir = mm->pgd;if (pgdir ==NULL )return ; + +/* Get VSID */ + ssize =user_segment_size (ea); + vsid =get_vsid (mm->context.id, ea, ssize); +if (!vsid) +return ; +/* +* Hash doesn't like irqs. Walking linux page table with irq disabled +* saves us from holding multiple locks. +*/ +local_irq_save (flags); +/* * THP pages use update_mmu_cache_pmd. We don't do * hash preload there. Hence can ignore THP here */ ptep =find_linux_pte_or_hugepte (pgdir, ea, &shift);if (!ptep) -return ; + goto out_exit;BUG_ON (shift);#ifdef CONFIG_PPC_64K_PAGES
@@ -1197,18 +1209,9 @@ void hash_preload(struct mm_struct *mm, unsigned long ea,* page size demotion here */ if (pte_val (*ptep) & (_PAGE_4K_PFN | _PAGE_NO_CACHE)) -return ; + goto out_exit;#endif /* CONFIG_PPC_64K_PAGES */ -/* Get VSID */ - ssize =user_segment_size (ea); - vsid =get_vsid (mm->context.id, ea, ssize); -if (!vsid) -return ; - -/* Hash doesn't like irqs */ -local_irq_save (flags); -/* Is that local to this CPU ? */ if (cpumask_equal (mm_cpumask (mm),cpumask_of (smp_processor_id ()))) local =1 ;
@@ -1230,7 +1233,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea, mm->context.user_psize, mm->context.user_psize, pte_val(*ptep)); - +out_exit:local_irq_restore (flags); }
arch/powerpc/mm/hugepage-hash64.c +3 -0
@@ -37,6 +37,9 @@ int __hash_page_thp(unsigned long ea, unsigned long access, unsigned long vsid,/* If PMD busy, retry the access */ if (unlikely (old_pmd & _PAGE_BUSY))return 0 ; +/* If PMD is trans splitting retry the access */ +if (unlikely (old_pmd & _PAGE_SPLITTING)) +return 0 ;/* If PMD permissions don't match, take page fault */ if (unlikely (access & ~old_pmd))return 1 ;
arch/powerpc/mm/hugetlbpage.c +46 -26
114 lines not on this page · raw message
arch/powerpc/mm/mem.c +4 -0
@@ -508,6 +508,10 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) {#ifdef CONFIG_PPC_STD_MMU +/* +* We don't need to worry about _PAGE_PRESENT here because we are +* called with either mm->page_table_lock held or ptl lock held +*/ unsigned long access =0 , trap;/* We only want HPTEs for linux PTEs that have _PAGE_ACCESSED set */
Signature · 1 line
1.8.1.2
PATCH 15/20 powerpc: Prevent gcc to re-read the pagetables
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
GCC is very likely to read the pagetables just once and cache them in the local stack or in a register, but it is can also decide to re-read the pagetables. The problem is that the pagetable in those places can change from under gcc.
With THP/hugetlbfs the pmd (and pgd for hugetlbfs giga pages) can change under gup_fast. The pages won't be freed untill we finish gup fast because we have irq disabled and we free these pages via rcu callback.
--- arch/powerpc/mm/gup.c | 8 ++++---- arch/powerpc/mm/hugetlbpage.c | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-)
arch/powerpc/mm/gup.c +4 -4
@@ -34,7 +34,7 @@ static noinline int gup_pte_range(pmd_t pmd, unsigned long addr, ptep =pte_offset_kernel (&pmd, addr); do { - pte_t pte = *ptep; + pte_t pte = ACCESS_ONCE(*ptep); struct page *page; if ((pte_val(pte) & mask) != result)
@@ -63,7 +63,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, pmdp =pmd_offset (&pud, addr); do { - pmd_t pmd = *pmdp; + pmd_t pmd = ACCESS_ONCE(*pmdp); next = pmd_addr_end(addr, end); /*
@@ -102,7 +102,7 @@ static int gup_pud_range(pgd_t pgd, unsigned long addr, unsigned long end, pudp =pud_offset (&pgd, addr); do { - pud_t pud = *pudp; + pud_t pud = ACCESS_ONCE(*pudp); next = pud_addr_end(addr, end); if (pud_none(pud))
@@ -165,7 +165,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, pgdp =pgd_offset (mm, addr); do { - pgd_t pgd = *pgdp; + pgd_t pgd = ACCESS_ONCE(*pgdp); pr_devel(" %016lx: normal pgd %p\n", addr, (void *)pgd_val(pgd));
arch/powerpc/mm/hugetlbpage.c +1 -1
@@ -1024,7 +1024,7 @@ int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,if (pte_end < end) end = pte_end; - pte = *ptep; + pte =ACCESS_ONCE (*ptep); mask = _PAGE_PRESENT | _PAGE_USER;if (write) mask |= _PAGE_RW;
Signature · 1 line
1.8.1.2
PATCH 16/20 powerpc/THP: Enable THP on PPC64
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
We enable only if the we support 16MB page size.
--- arch/powerpc/include/asm/pgtable-ppc64.h | 3 +-- arch/powerpc/mm/pgtable_64.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/pgtable-ppc64.h +1 -2
@@ -429,8 +429,7 @@ static inline int pmd_trans_splitting(pmd_t pmd) return 0; } -/* We will enable it in the last patch */ -#define has_transparent_hugepage () 0 +extern int has_transparent_hugepage (void );#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline pte_t pmd_pte (pmd_t pmd)
arch/powerpc/mm/pgtable_64.c +29 -0
@@ -818,4 +818,33 @@ pmd_t pmdp_get_and_clear(struct mm_struct *mm,memset (pgtable,0 ,PTE_FRAG_SIZE ); return old_pmd; } + +int has_transparent_hugepage (void ) +{ +if (!mmu_has_feature (MMU_FTR_16M_PAGE )) +return 0 ; +/* +* We support THP only if PMD_SIZE is 16MB. +*/ +if (mmu_psize_defs[MMU_PAGE_16M ].shift !=PMD_SHIFT ) +return 0 ; +/* +* We need to make sure that we support 16MB hugepage in a segement +* with base page size 64K or 4K. We only enable THP with a PAGE_SIZE +* of 64K. +*/ +/* +* If we have 64K HPTE, we will be using that by default +*/ +if (mmu_psize_defs[MMU_PAGE_64K ].shift && + (mmu_psize_defs[MMU_PAGE_64K ].penc[MMU_PAGE_16M ] ==-1 )) +return 0 ; +/* +* Ok we only have 4K HPTE +*/ +if (mmu_psize_defs[MMU_PAGE_4K ].penc[MMU_PAGE_16M ] ==-1 ) +return 0 ; + +return 1 ; +}#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
Signature · 1 line
1.8.1.2
PATCH 17/20 powerpc: Optimize hugepage invalidate
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
Hugepage invalidate involves invalidating multiple hpte entries. Optimize the operation using H_BULK_REMOVE on lpar platforms. On native, reduce the number of tlb flush.
--- arch/powerpc/include/asm/machdep.h | 3 + arch/powerpc/mm/hash_native_64.c | 74 +++++++++++++++++++++ arch/powerpc/mm/pgtable_64.c | 12 +++- arch/powerpc/platforms/pseries/lpar.c | 122 ++++++++++++++++++++++++++++++++-- 4 files changed, 202 insertions(+), 9 deletions(-)
arch/powerpc/include/asm/machdep.h +3 -0
@@ -57,6 +57,9 @@ struct machdep_calls {void (*hpte_removebolted)(unsigned long ea,int psize,int ssize);void (*flush_hash_range)(unsigned long number,int local); +void (*hugepage_invalidate)(struct mm_struct *mm, +unsigned char *hpte_slot_array, +unsigned long addr,int psize);/* special for kexec, to be called in real mode, linear mapping is * destroyed as well */
arch/powerpc/mm/hash_native_64.c +74 -0
84 lines not on this page · raw message
arch/powerpc/mm/pgtable_64.c +10 -2
@@ -692,6 +692,7 @@ void hpte_do_hugepage_flush(struct mm_struct *mm, unsigned long addr, {int ssize, i;unsigned long s_addr; +int max_hpte_count;unsigned int psize, valid;unsigned char *hpte_slot_array;unsigned long hidx, vpn, vsid, hash, shift, slot;
@@ -715,9 +716,16 @@ void hpte_do_hugepage_flush(struct mm_struct *mm, unsigned long addr,/* get the base page size */ psize =get_slice_psize (mm, s_addr); - shift = mmu_psize_defs[psize].shift; -for (i =0 ; i < (HPAGE_PMD_SIZE >> shift); i++) { +if (ppc_md.hugepage_invalidate) +return ppc_md.hugepage_invalidate (mm, hpte_slot_array, + s_addr, psize); +/* +* No bluk hpte removal support, invalidate each entry +*/ + shift = mmu_psize_defs[psize].shift; + max_hpte_count =HPAGE_PMD_SIZE >> shift; +for (i =0 ; i < max_hpte_count; i++) {/* * 8 bits per each hpte entries * 000| [ secondary group (one bit) | hidx (3 bits) | valid bit]
arch/powerpc/platforms/pseries/lpar.c +115 -7
146 lines not on this page · raw message
Signature · 1 line
1.8.1.2
PATCH 18/20 powerpc: disable assert_pte_locked for collapse_huge_page
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
With THP we set pmd to none, before we do pte_clear. Hence we can't walk page table to get the pte lock ptr and verify whether it is locked. THP do take pte lock before calling pte_clear. So we don't change the locking rules here. It is that we can't use page table walking to check whether pte locks are held with THP.
--- arch/powerpc/mm/pgtable.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/powerpc/mm/pgtable.c +8 -0
@@ -235,6 +235,14 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) pud =pud_offset (pgd, addr);BUG_ON (pud_none (*pud)); pmd =pmd_offset (pud, addr); +/* +* khugepaged to collapse normal pages to hugepage, first set +* pmd to none to force page fault/gup to take mmap_sem. After +* pmd is set to none, we do a pte_clear which does this assertion +* so if we find pmd none, return. +*/ +if (pmd_none (*pmd)) +return ;BUG_ON (!pmd_present (*pmd));assert_spin_locked (pte_lockptr (mm, pmd)); }
Signature · 1 line
1.8.1.2
PATCH 19/20 powerpc: use smp_rmb when looking at deposisted pgtable to store hash index
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
We need to use smb_rmb when looking at hpte slot array. Otherwise we could reorder the hpte_slot array load bfore even we marked the pmd trans huge. Related smb_wmb()s is done in pgtable_trans_huge_deposit when we deposit a pgtable.
--- arch/powerpc/include/asm/pgtable-ppc64.h | 15 +++++++++++++++ arch/powerpc/mm/hugepage-hash64.c | 6 +----- arch/powerpc/mm/pgtable_64.c | 6 +----- 3 files changed, 17 insertions(+), 10 deletions(-)
arch/powerpc/include/asm/pgtable-ppc64.h +15 -0
@@ -10,6 +10,7 @@#else #include <asm/pgtable-ppc64-4k.h> #endif +#include <asm/barrier.h> #define FIRST_USER_ADDRESS 0
@@ -393,6 +394,20 @@ static inline void mark_hpte_slot_valid(unsigned char *hpte_slot_array, hpte_slot_array[index] = hidx <<4 |0x1 <<3 ; } +static inline char *get_hpte_slot_array (pmd_t *pmdp) +{ +/* +* The hpte hindex is stored in the pgtable whose address is in the +* second half of the PMD +* +* Order this load with the test for pmd_trans_huge in the caller +*/ +smp_rmb (); +return *(char **)(pmdp +PTRS_PER_PMD ); + + +} +extern void hpte_do_hugepage_flush (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp);#ifdef CONFIG_TRANSPARENT_HUGEPAGE
arch/powerpc/mm/hugepage-hash64.c +1 -5
@@ -84,11 +84,7 @@ int __hash_page_thp(unsigned long ea, unsigned long access, unsigned long vsid, vpn =hpt_vpn (ea, vsid, ssize); hash =hpt_hash (vpn, shift, ssize); -/* -* The hpte hindex are stored in the pgtable whose address is in the -* second half of the PMD -*/ - hpte_slot_array = *(char **)(pmdp +PTRS_PER_PMD ); + hpte_slot_array =get_hpte_slot_array (pmdp); valid =hpte_valid (hpte_slot_array, index);if (valid) {
arch/powerpc/mm/pgtable_64.c +1 -5
@@ -701,11 +701,7 @@ void hpte_do_hugepage_flush(struct mm_struct *mm, unsigned long addr,* Flush all the hptes mapping this hugepage */ s_addr = addr &HPAGE_PMD_MASK ; -/* -* The hpte hindex are stored in the pgtable whose address is in the -* second half of the PMD -*/ - hpte_slot_array = *(char **)(pmdp +PTRS_PER_PMD ); + hpte_slot_array =get_hpte_slot_array (pmdp);/* * IF we try to do a HUGE PTE update after a withdraw is done. * we will find the below NULL. This happens when we do
Signature · 1 line
1.8.1.2
PATCH 20/20 powerpc: split hugepage when using subpage protection
In reply to Aneesh Kumar K.VFrom: "Aneesh Kumar K.V" <[email protected]>
We find all the overlapping vma and mark them such that we don't allocate hugepage in that range. Also we split existing huge page so that the normal page hash can be invalidated and new page faulted in with new protection bits.
--- arch/powerpc/mm/subpage-prot.c | 48 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)
arch/powerpc/mm/subpage-prot.c +48 -0
@@ -130,6 +130,53 @@ static void subpage_prot_clear(unsigned long addr, unsigned long len)up_write (&mm->mmap_sem); } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static int subpage_walk_pmd_entry (pmd_t *pmd,unsigned long addr, +unsigned long end,struct mm_walk *walk) +{ +struct vm_area_struct *vma = walk->private; +split_huge_page_pmd (vma, addr, pmd); +return 0 ; +} + +static void subpage_mark_vma_nohuge (struct mm_struct *mm,unsigned long addr, +unsigned long len) +{ +struct vm_area_struct *vma; +struct mm_walk subpage_proto_walk = { + .mm = mm, + .pmd_entry = subpage_walk_pmd_entry, + }; + +/* +* We don't try too hard, we just mark all the vma in that range +* VM_NOHUGEPAGE and split them. +*/ + vma =find_vma (mm, addr); +/* +* If the range is in unmapped range, just return +*/ +if (vma && ((addr + len) <= vma->vm_start)) +return ; + +while (vma) { +if (vma->vm_start >= (addr + len)) +break ; + vma->vm_flags |=VM_NOHUGEPAGE ; + subpage_proto_walk.private = vma; +walk_page_range (vma->vm_start, vma->vm_end, + &subpage_proto_walk); + vma = vma->vm_next; + } +} +#else +static void subpage_mark_vma_nohuge (struct mm_struct *mm,unsigned long addr, +unsigned long len) +{ +return ; +} +#endif +/* * Copy in a subpage protection map for an address range. * The map has 2 bits per 4k subpage, so 32 bits per 64k page.
@@ -168,6 +215,7 @@ long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map)return -EFAULT ;down_write (&mm->mmap_sem); +subpage_mark_vma_nohuge (mm, addr, len);for (limit = addr + len; addr < limit; addr = next) { next =pmd_addr_end (addr, limit); err = -ENOMEM ;
Signature · 1 line
1.8.1.2