perf/mm: Fix PERF_SAMPLE_*_PAGE_SIZE
Subsystems (8)
Peter Zijlstra
maintainerv2 cover perf/mm: Fix PERF_SAMPLE_*_PAGE_SIZE
Hi,
These patches provide generic infrastructure to determine TLB page size from page table entries alone. Perf will use this (for either data or code address) to aid in profiling TLB issues.
While most architectures only have page table aligned large pages, some (notably ARM64, Sparc64 and Power) provide non page table aligned large pages and need to provide their own implementation of these functions.
I've provided (completely untested) implementations for ARM64, Sparc64 and Power/8xxx (it looks like I'm still missing Power/Book3s64/hash support).
Changes since -v1:
- Changed wording to reflect these are page-table sizes; actual TLB sizes might vary.
- added Power/8xx
Barring any objections I'll queue these in tip/perf/core, as these patches fix the code that's currently in there.
Peter Zijlstra
maintainerPATCH v2 1/6 mm/gup: Provide gup_get_pte() more generic
In order to write another lockless page-table walker, we need gup_get_pte() exposed. While doing that, rename it to ptep_get_lockless() to match the existing ptep_get() naming.
--- include/linux/pgtable.h | 55 +++++++++++++++++++++++++++++++++++++++++++++ mm/gup.c | 58 ------------------------------------------------ 2 files changed, 56 insertions(+), 57 deletions(-)
include/linux/pgtable.h +55 -0
@@ -258,6 +258,61 @@ static inline pte_t ptep_get(pte_t *ptep }#endif +#ifdef CONFIG_GUP_GET_PTE_LOW_HIGH +/* +* WARNING: only to be used in the get_user_pages_fast() implementation. +* +* With get_user_pages_fast(), we walk down the pagetables without taking any +* locks. For this we would like to load the pointers atomically, but sometimes +* that is not possible (e.g. without expensive cmpxchg8b on x86_32 PAE). What +* we do have is the guarantee that a PTE will only either go from not present +* to present, or present to not present or both -- it will not switch to a +* completely different present page without a TLB flush in between; something +* that we are blocking by holding interrupts off. +* +* Setting ptes from not present to present goes: +* +* ptep->pte_high = h; +* smp_wmb(); +* ptep->pte_low = l; +* +* And present to not present goes: +* +* ptep->pte_low = 0; +* smp_wmb(); +* ptep->pte_high = 0; +* +* We must ensure here that the load of pte_low sees 'l' IFF pte_high sees 'h'. +* We load pte_high *after* loading pte_low, which ensures we don't see an older +* value of pte_high. *Then* we recheck pte_low, which ensures that we haven't +* picked up a changed pte high. We might have gotten rubbish values from +* pte_low and pte_high, but we are guaranteed that pte_low will not have the +* present bit set *unless* it is 'l'. Because get_user_pages_fast() only +* operates on present ptes we're safe. +*/ +static inline pte_t ptep_get_lockless (pte_t *ptep) +{ +pte_t pte; + +do { + pte.pte_low = ptep->pte_low; +smp_rmb (); + pte.pte_high = ptep->pte_high; +smp_rmb (); + }while (unlikely (pte.pte_low != ptep->pte_low)); + +return pte; +} +#else /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +/* +* We require that the PTE can be read atomically. +*/ +static inline pte_t ptep_get_lockless (pte_t *ptep) +{ +return ptep_get (ptep); +} +#endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE #ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
mm/gup.c +1 -57
70 lines not on this page · raw message
Re: [PATCH v2 1/6] mm/gup: Provide gup_get_pte() more generic
include/linux/pgtable.h static inline pte_t ptep_get(pte_t *ptep · line 271
Quoted 10 lines · Peter Zijlstra
include/linux/pgtable.h +10
-0+#ifdef CONFIG_GUP_GET_PTE_LOW_HIGH +/* +* WARNING: only to be used in the get_user_pages_fast() implementation. +* With get_user_pages_fast(), we walk down the pagetables without taking any +* locks. For this we would like to load the pointers atomically, but sometimes +* that is not possible (e.g. without expensive cmpxchg8b on x86_32 PAE). What +* we do have is the guarantee that a PTE will only either go from not present +* to present, or present to not present or both -- it will not switch to a +* completely different present page without a TLB flush in between; something +* that we are blocking by holding interrupts off.
I feel like this comment needs some love. How about:
* For walking the pagetables without holding any locks. Some architectures
* (eg x86-32 PAE) cannot load the entries atomically without using
* expensive instructions. We are guaranteed that a PTE will only either go
* from not present to present, or present to not present -- it will not
* switch to a completely different present page without a TLB flush
* inbetween; which we are blocking by holding interrupts off.
And it would be nice to have an assertion that interrupts are disabled in the code. Because comments are nice, but nobody reads them.
Quoted 13 lines
include/linux/pgtable.h +13
-0+static inline pte_t ptep_get_lockless (pte_t *ptep) +{ +pte_t pte; + +do { + pte.pte_low = ptep->pte_low; +smp_rmb (); + pte.pte_high = ptep->pte_high; +smp_rmb (); + }while (unlikely (pte.pte_low != ptep->pte_low)); + +return pte; +}
Peter Zijlstra
maintainerRe: [PATCH v2 1/6] mm/gup: Provide gup_get_pte() more generic
Quoted 21 lines · Matthew Wilcox
include/linux/pgtable.h static inline pte_t ptep_get(pte_t *ptep · line 271
Quoted 10 lines · Peter Zijlstra
include/linux/pgtable.h +10
-0+#ifdef CONFIG_GUP_GET_PTE_LOW_HIGH +/* +* WARNING: only to be used in the get_user_pages_fast() implementation. +* With get_user_pages_fast(), we walk down the pagetables without taking any +* locks. For this we would like to load the pointers atomically, but sometimes +* that is not possible (e.g. without expensive cmpxchg8b on x86_32 PAE). What +* we do have is the guarantee that a PTE will only either go from not present +* to present, or present to not present or both -- it will not switch to a +* completely different present page without a TLB flush in between; something +* that we are blocking by holding interrupts off. I feel like this comment needs some love. How about:
* For walking the pagetables without holding any locks. Some architectures
* (eg x86-32 PAE) cannot load the entries atomically without using
* expensive instructions. We are guaranteed that a PTE will only either go
* from not present to present, or present to not present -- it will not
* switch to a completely different present page without a TLB flush
* inbetween; which we are blocking by holding interrupts off.
Quoted 2 lines
And it would be nice to have an assertion that interrupts are disabled in the code. Because comments are nice, but nobody reads them.
Quite agreed, I'll stick a separate patch on with the updated comment and a lockdep_assert_irqs_disabled() in. I'm afraid that latter will make for header soup though :/
We'll see, let the robots have it.
[tip: perf/core] mm/gup: Provide gup_get_pte() more generic
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 10c48d60a6aab1e174426f4de33d876e48c1b200 Gitweb: https://git.kernel.org/tip/10c48d60a6aab1e174426f4de33d876e48c1b200 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 13 Nov 2020 11:41:40 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:00:30 +01:00
mm/gup: Provide gup_get_pte() more generic
In order to write another lockless page-table walker, we need gup_get_pte() exposed. While doing that, rename it to ptep_get_lockless() to match the existing ptep_get() naming.
--- include/linux/pgtable.h | 55 ++++++++++++++++++++++++++++++++++++++- mm/gup.c | 58 +---------------------------------------- 2 files changed, 56 insertions(+), 57 deletions(-)
include/linux/pgtable.h +55 -0
@@ -258,6 +258,61 @@ static inline pte_t ptep_get(pte_t *ptep) }#endif +#ifdef CONFIG_GUP_GET_PTE_LOW_HIGH +/* +* WARNING: only to be used in the get_user_pages_fast() implementation. +* +* With get_user_pages_fast(), we walk down the pagetables without taking any +* locks. For this we would like to load the pointers atomically, but sometimes +* that is not possible (e.g. without expensive cmpxchg8b on x86_32 PAE). What +* we do have is the guarantee that a PTE will only either go from not present +* to present, or present to not present or both -- it will not switch to a +* completely different present page without a TLB flush in between; something +* that we are blocking by holding interrupts off. +* +* Setting ptes from not present to present goes: +* +* ptep->pte_high = h; +* smp_wmb(); +* ptep->pte_low = l; +* +* And present to not present goes: +* +* ptep->pte_low = 0; +* smp_wmb(); +* ptep->pte_high = 0; +* +* We must ensure here that the load of pte_low sees 'l' IFF pte_high sees 'h'. +* We load pte_high *after* loading pte_low, which ensures we don't see an older +* value of pte_high. *Then* we recheck pte_low, which ensures that we haven't +* picked up a changed pte high. We might have gotten rubbish values from +* pte_low and pte_high, but we are guaranteed that pte_low will not have the +* present bit set *unless* it is 'l'. Because get_user_pages_fast() only +* operates on present ptes we're safe. +*/ +static inline pte_t ptep_get_lockless (pte_t *ptep) +{ +pte_t pte; + +do { + pte.pte_low = ptep->pte_low; +smp_rmb (); + pte.pte_high = ptep->pte_high; +smp_rmb (); + }while (unlikely (pte.pte_low != ptep->pte_low)); + +return pte; +} +#else /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +/* +* We require that the PTE can be read atomically. +*/ +static inline pte_t ptep_get_lockless (pte_t *ptep) +{ +return ptep_get (ptep); +} +#endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE #ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
mm/gup.c +1 -57
70 lines not on this page · raw message
[tip: perf/core] mm/gup: Provide gup_get_pte() more generic
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 2a4a06da8a4b93dd189171eed7a99fffd38f42f3 Gitweb: https://git.kernel.org/tip/2a4a06da8a4b93dd189171eed7a99fffd38f42f3 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 13 Nov 2020 11:41:40 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:14:50 +01:00
mm/gup: Provide gup_get_pte() more generic
In order to write another lockless page-table walker, we need gup_get_pte() exposed. While doing that, rename it to ptep_get_lockless() to match the existing ptep_get() naming.
--- include/linux/pgtable.h | 55 ++++++++++++++++++++++++++++++++++++++- mm/gup.c | 58 +---------------------------------------- 2 files changed, 56 insertions(+), 57 deletions(-)
include/linux/pgtable.h +55 -0
@@ -258,6 +258,61 @@ static inline pte_t ptep_get(pte_t *ptep) }#endif +#ifdef CONFIG_GUP_GET_PTE_LOW_HIGH +/* +* WARNING: only to be used in the get_user_pages_fast() implementation. +* +* With get_user_pages_fast(), we walk down the pagetables without taking any +* locks. For this we would like to load the pointers atomically, but sometimes +* that is not possible (e.g. without expensive cmpxchg8b on x86_32 PAE). What +* we do have is the guarantee that a PTE will only either go from not present +* to present, or present to not present or both -- it will not switch to a +* completely different present page without a TLB flush in between; something +* that we are blocking by holding interrupts off. +* +* Setting ptes from not present to present goes: +* +* ptep->pte_high = h; +* smp_wmb(); +* ptep->pte_low = l; +* +* And present to not present goes: +* +* ptep->pte_low = 0; +* smp_wmb(); +* ptep->pte_high = 0; +* +* We must ensure here that the load of pte_low sees 'l' IFF pte_high sees 'h'. +* We load pte_high *after* loading pte_low, which ensures we don't see an older +* value of pte_high. *Then* we recheck pte_low, which ensures that we haven't +* picked up a changed pte high. We might have gotten rubbish values from +* pte_low and pte_high, but we are guaranteed that pte_low will not have the +* present bit set *unless* it is 'l'. Because get_user_pages_fast() only +* operates on present ptes we're safe. +*/ +static inline pte_t ptep_get_lockless (pte_t *ptep) +{ +pte_t pte; + +do { + pte.pte_low = ptep->pte_low; +smp_rmb (); + pte.pte_high = ptep->pte_high; +smp_rmb (); + }while (unlikely (pte.pte_low != ptep->pte_low)); + +return pte; +} +#else /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +/* +* We require that the PTE can be read atomically. +*/ +static inline pte_t ptep_get_lockless (pte_t *ptep) +{ +return ptep_get (ptep); +} +#endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE #ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
mm/gup.c +1 -57
70 lines not on this page · raw message
Peter Zijlstra
maintainerPATCH v2 2/6 mm: Introduce pXX_leaf_size()
In reply to Peter ZijlstraA number of architectures have non-pagetable aligned huge/large pages. For such architectures a leaf can actually be part of a larger entry.
Provide generic helpers to determine the size of a page-table leaf.
--- include/linux/pgtable.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
include/linux/pgtable.h +16 -0
@@ -1536,4 +1536,20 @@ typedef unsigned int pgtbl_mod_mask;#define pmd_leaf (x) 0#endif +#ifndef pgd_leaf_size +#define pgd_leaf_size (x) (1ULL << PGDIR_SHIFT) +#endif +#ifndef p4d_leaf_size +#define p4d_leaf_size (x) P4D_SIZE +#endif +#ifndef pud_leaf_size +#define pud_leaf_size (x) PUD_SIZE +#endif +#ifndef pmd_leaf_size +#define pmd_leaf_size (x) PMD_SIZE +#endif +#ifndef pte_leaf_size +#define pte_leaf_size (x) PAGE_SIZE +#endif +#endif /* _LINUX_PGTABLE_H */
Re: [PATCH v2 2/6] mm: Introduce pXX_leaf_size()
Quoted 6 lines · Peter Zijlstra
A number of architectures have non-pagetable aligned huge/large pages. For such architectures a leaf can actually be part of a larger entry.
Provide generic helpers to determine the size of a page-table leaf.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
[tip: perf/core] mm: Introduce pXX_leaf_size()
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 93aec63b945579b679234ff5e5d7837baf2c7018 Gitweb: https://git.kernel.org/tip/93aec63b945579b679234ff5e5d7837baf2c7018 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 13 Nov 2020 11:45:36 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:00:30 +01:00
mm: Introduce pXX_leaf_size()
A number of architectures have non-pagetable aligned huge/large pages. For such architectures a leaf can actually be part of a larger entry.
Provide generic helpers to determine the size of a page-table leaf.
--- include/linux/pgtable.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
include/linux/pgtable.h +16 -0
@@ -1549,4 +1549,20 @@ typedef unsigned int pgtbl_mod_mask;#define pmd_leaf (x) 0#endif +#ifndef pgd_leaf_size +#define pgd_leaf_size (x) (1ULL << PGDIR_SHIFT) +#endif +#ifndef p4d_leaf_size +#define p4d_leaf_size (x) P4D_SIZE +#endif +#ifndef pud_leaf_size +#define pud_leaf_size (x) PUD_SIZE +#endif +#ifndef pmd_leaf_size +#define pmd_leaf_size (x) PMD_SIZE +#endif +#ifndef pte_leaf_size +#define pte_leaf_size (x) PAGE_SIZE +#endif +#endif /* _LINUX_PGTABLE_H */
[tip: perf/core] mm: Introduce pXX_leaf_size()
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 560dabbdf68bb15f9e241af8f828b1c8c38d6c6f Gitweb: https://git.kernel.org/tip/560dabbdf68bb15f9e241af8f828b1c8c38d6c6f Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 13 Nov 2020 11:45:36 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:14:50 +01:00
mm: Introduce pXX_leaf_size()
A number of architectures have non-pagetable aligned huge/large pages. For such architectures a leaf can actually be part of a larger entry.
Provide generic helpers to determine the size of a page-table leaf.
--- include/linux/pgtable.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
include/linux/pgtable.h +16 -0
@@ -1536,4 +1536,20 @@ typedef unsigned int pgtbl_mod_mask;#define pmd_leaf (x) 0#endif +#ifndef pgd_leaf_size +#define pgd_leaf_size (x) (1ULL << PGDIR_SHIFT) +#endif +#ifndef p4d_leaf_size +#define p4d_leaf_size (x) P4D_SIZE +#endif +#ifndef pud_leaf_size +#define pud_leaf_size (x) PUD_SIZE +#endif +#ifndef pmd_leaf_size +#define pmd_leaf_size (x) PMD_SIZE +#endif +#ifndef pte_leaf_size +#define pte_leaf_size (x) PAGE_SIZE +#endif +#endif /* _LINUX_PGTABLE_H */
Peter Zijlstra
maintainerPATCH v2 3/6 perf/core: Fix arch_perf_get_page_size()
In reply to Peter ZijlstraThe (new) page-table walker in arch_perf_get_page_size() is broken in various ways. Specifically while it is used in a lockless manner, it doesn't depend on CONFIG_HAVE_FAST_GUP nor uses the proper _lockless offset methods, nor is careful to only read each entry only once.
Also the hugetlb support is broken due to calling pte_page() without first checking pte_special().
Rewrite the whole thing to be a proper lockless page-table walker and employ the new pXX_leaf_size() pgtable functions to determine the pagetable size without looking at the page-frames.
--- arch/arm64/include/asm/pgtable.h | 3 + arch/sparc/include/asm/pgtable_64.h | 13 ++++ arch/sparc/mm/hugetlbpage.c | 19 ++++-- include/linux/pgtable.h | 16 +++++ kernel/events/core.c | 102 +++++++++++++----------------------- 5 files changed, 82 insertions(+), 71 deletions(-)
kernel/events/core.c +38 -65
@@ -52,6 +52,7 @@#include <linux/mount.h> #include <linux/min_heap.h> #include <linux/highmem.h> +#include <linux/pgtable.h> #include "internal.h"
@@ -7001,90 +7001,62 @@ static u64 perf_virt_to_phys(u64 virt)return phys_addr; } -#ifdef CONFIG_MMU -/* -* Return the MMU page size of a given virtual address. -* -* This generic implementation handles page-table aligned huge pages, as well -* as non-page-table aligned hugetlbfs compound pages. -* -* If an architecture supports and uses non-page-table aligned pages in their -* kernel mapping it will need to provide it's own implementation of this -* function. +* Return the pagetable size of a given virtual address. */ -__weak u64 arch_perf_get_page_size (struct mm_struct *mm,unsigned long addr) +static u64 perf_get_pgtable_size (struct mm_struct *mm,unsigned long addr) { -struct page *page; -pgd_t *pgd; -p4d_t *p4d; -pud_t *pud; -pmd_t *pmd; -pte_t *pte; +u64 size =0 ; - pgd =pgd_offset (mm, addr); -if (pgd_none (*pgd)) -return 0 ; +#ifdef CONFIG_HAVE_FAST_GUP +pgd_t *pgdp, pgd; +p4d_t *p4dp, p4d; +pud_t *pudp, pud; +pmd_t *pmdp, pmd; +pte_t *ptep, pte; - p4d =p4d_offset (pgd, addr); -if (!p4d_present (*p4d)) + pgdp =pgd_offset (mm, addr); + pgd =READ_ONCE (*pgdp); +if (pgd_none (pgd))return 0 ; -if (p4d_leaf (*p4d)) -return 1ULL <<P4D_SHIFT ; +if (pgd_leaf (pgd)) +return pgd_leaf_size (pgd); - pud =pud_offset (p4d, addr); -if (!pud_present (*pud)) + p4dp =p4d_offset_lockless (pgdp, pgd, addr); + p4d =READ_ONCE (*p4dp); +if (!p4d_present (p4d))return 0 ; -if (pud_leaf (*pud)) { -#ifdef pud_page - page =pud_page (*pud); -if (PageHuge (page)) -return page_size (compound_head (page)); -#endif -return 1ULL <<PUD_SHIFT ; - } +if (p4d_leaf (p4d)) +return p4d_leaf_size (p4d); - pmd =pmd_offset (pud, addr); -if (!pmd_present (*pmd)) + pudp =pud_offset_lockless (p4dp, p4d, addr); + pud =READ_ONCE (*pudp); +if (!pud_present (pud))return 0 ; -if (pmd_leaf (*pmd)) { -#ifdef pmd_page - page =pmd_page (*pmd); -if (PageHuge (page)) -return page_size (compound_head (page)); -#endif -return 1ULL <<PMD_SHIFT ; - } +if (pud_leaf (pud)) +return pud_leaf_size (pud); - pte =pte_offset_map (pmd, addr); -if (!pte_present (*pte)) { -pte_unmap (pte); + pmdp =pmd_offset_lockless (pudp, pud, addr); + pmd =READ_ONCE (*pmdp); +if (!pmd_present (pmd))return 0 ; - } - page =pte_page (*pte); -if (PageHuge (page)) { -u64 size =page_size (compound_head (page)); -pte_unmap (pte); -return size; - } - -pte_unmap (pte); -return PAGE_SIZE ; -} +if (pmd_leaf (pmd)) +return pmd_leaf_size (pmd); -#else + ptep =pte_offset_map (&pmd, addr); + pte =ptep_get_lockless (ptep); +if (pte_present (pte)) + size =pte_leaf_size (pte); +pte_unmap (ptep); +#endif /* CONFIG_HAVE_FAST_GUP */ -static u64 arch_perf_get_page_size (struct mm_struct *mm,unsigned long addr) -{ -return 0 ; +return size; } -#endif -static u64 perf_get_page_size (unsigned long addr) {struct mm_struct *mm;
@@ -7109,7 +7081,7 @@ static u64 perf_get_page_size(unsigned l mm = &init_mm; } - size =arch_perf_get_page_size (mm, addr); + size =perf_get_pgtable_size (mm, addr);local_irq_restore (flags);
Re: [PATCH v2 3/6] perf/core: Fix arch_perf_get_page_size()
Quoted 16 lines · Peter Zijlstra
The (new) page-table walker in
arch_perf_get_page_size()is broken in various ways. Specifically while it is used in a lockless manner, it doesn't depend onCONFIG_HAVE_FAST_GUPnor uses the proper_locklessoffset methods, nor is careful to only read each entry only once.Also the hugetlb support is broken due to calling
pte_page()without first checkingpte_special().Rewrite the whole thing to be a proper lockless page-table walker and employ the new
pXX_leaf_size()pgtable functions to determine the pagetable size without looking at the page-frames.Fixes 51b646b2d9f8 ("perf,mm: Handle non-page-table-aligned hugetlbfs")Fixes 8d97e71811aa ("perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE")Signed-off-by Peter Zijlstra (Intel) <[email protected]>Tested-by Kan Liang <[email protected]>
Quoted 7 lines
--- arch/arm64/include/asm/pgtable.h | 3 + arch/sparc/include/asm/pgtable_64.h | 13 ++++ arch/sparc/mm/hugetlbpage.c | 19 ++++-- include/linux/pgtable.h | 16 +++++ kernel/events/core.c | 102 +++++++++++++----------------------- 5 files changed, 82 insertions(+), 71 deletions(-)
This diffstat doesn't match the patch in this email ...
Quoted 148 lines
kernel/events/core.c +38
-65@@ -52,6 +52,7 @@#include <linux/mount.h> #include <linux/min_heap.h> #include <linux/highmem.h> +#include <linux/pgtable.h> #include "internal.h" @@ -7001,90 +7001,62 @@ static u64 perf_virt_to_phys(u64 virt)return phys_addr; } -#ifdef CONFIG_MMU -/* -* Return the MMU page size of a given virtual address. -* -* This generic implementation handles page-table aligned huge pages, as well -* as non-page-table aligned hugetlbfs compound pages. -* -* If an architecture supports and uses non-page-table aligned pages in their -* kernel mapping it will need to provide it's own implementation of this -* function. +* Return the pagetable size of a given virtual address. */ -__weak u64 arch_perf_get_page_size (struct mm_struct *mm,unsigned long addr) +static u64 perf_get_pgtable_size (struct mm_struct *mm,unsigned long addr) { -struct page *page; -pgd_t *pgd; -p4d_t *p4d; -pud_t *pud; -pmd_t *pmd; -pte_t *pte; +u64 size =0 ; - pgd =pgd_offset (mm, addr); -if (pgd_none (*pgd)) -return 0 ; +#ifdef CONFIG_HAVE_FAST_GUP +pgd_t *pgdp, pgd; +p4d_t *p4dp, p4d; +pud_t *pudp, pud; +pmd_t *pmdp, pmd; +pte_t *ptep, pte; - p4d =p4d_offset (pgd, addr); -if (!p4d_present (*p4d)) + pgdp =pgd_offset (mm, addr); + pgd =READ_ONCE (*pgdp); +if (pgd_none (pgd))return 0 ; -if (p4d_leaf (*p4d)) -return 1ULL <<P4D_SHIFT ; +if (pgd_leaf (pgd)) +return pgd_leaf_size (pgd); - pud =pud_offset (p4d, addr); -if (!pud_present (*pud)) + p4dp =p4d_offset_lockless (pgdp, pgd, addr); + p4d =READ_ONCE (*p4dp); +if (!p4d_present (p4d))return 0 ; -if (pud_leaf (*pud)) { -#ifdef pud_page - page =pud_page (*pud); -if (PageHuge (page)) -return page_size (compound_head (page)); -#endif -return 1ULL <<PUD_SHIFT ; - } +if (p4d_leaf (p4d)) +return p4d_leaf_size (p4d); - pmd =pmd_offset (pud, addr); -if (!pmd_present (*pmd)) + pudp =pud_offset_lockless (p4dp, p4d, addr); + pud =READ_ONCE (*pudp); +if (!pud_present (pud))return 0 ; -if (pmd_leaf (*pmd)) { -#ifdef pmd_page - page =pmd_page (*pmd); -if (PageHuge (page)) -return page_size (compound_head (page)); -#endif -return 1ULL <<PMD_SHIFT ; - } +if (pud_leaf (pud)) +return pud_leaf_size (pud); - pte =pte_offset_map (pmd, addr); -if (!pte_present (*pte)) { -pte_unmap (pte); + pmdp =pmd_offset_lockless (pudp, pud, addr); + pmd =READ_ONCE (*pmdp); +if (!pmd_present (pmd))return 0 ; - } - page =pte_page (*pte); -if (PageHuge (page)) { -u64 size =page_size (compound_head (page)); -pte_unmap (pte); -return size; - } - -pte_unmap (pte); -return PAGE_SIZE ; -} +if (pmd_leaf (pmd)) +return pmd_leaf_size (pmd); -#else + ptep =pte_offset_map (&pmd, addr); + pte =ptep_get_lockless (ptep); +if (pte_present (pte)) + size =pte_leaf_size (pte); +pte_unmap (ptep); +#endif /* CONFIG_HAVE_FAST_GUP */ -static u64 arch_perf_get_page_size (struct mm_struct *mm,unsigned long addr) -{ -return 0 ; +return size; } -#endif -static u64 perf_get_page_size (unsigned long addr) {struct mm_struct *mm;@@ -7109,7 +7081,7 @@ static u64 perf_get_page_size(unsigned l mm = &init_mm; } - size =arch_perf_get_page_size (mm, addr); + size =perf_get_pgtable_size (mm, addr);local_irq_restore (flags);
Re: [PATCH v2 3/6] perf/core: Fix arch_perf_get_page_size()
Quoted 25 lines · Matthew Wilcox
Quoted 16 lines · Peter Zijlstra
The (new) page-table walker in
arch_perf_get_page_size()is broken in various ways. Specifically while it is used in a lockless manner, it doesn't depend onCONFIG_HAVE_FAST_GUPnor uses the proper_locklessoffset methods, nor is careful to only read each entry only once.Also the hugetlb support is broken due to calling
pte_page()without first checkingpte_special().Rewrite the whole thing to be a proper lockless page-table walker and employ the new
pXX_leaf_size()pgtable functions to determine the pagetable size without looking at the page-frames.Fixes 51b646b2d9f8 ("perf,mm: Handle non-page-table-aligned hugetlbfs")Fixes 8d97e71811aa ("perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE")Signed-off-by Peter Zijlstra (Intel) <[email protected]>Tested-by Kan Liang <[email protected]>Quoted 7 lines
--- arch/arm64/include/asm/pgtable.h | 3 + arch/sparc/include/asm/pgtable_64.h | 13 ++++ arch/sparc/mm/hugetlbpage.c | 19 ++++-- include/linux/pgtable.h | 16 +++++ kernel/events/core.c | 102 +++++++++++++----------------------- 5 files changed, 82 insertions(+), 71 deletions(-)
Quoted 1 line
This diffstat doesn't match the patch in this email ...
Urgh, no idea how I did that... I must've edited the diff and not done a quilt-refresh. Updated below.
---
Subject: perf/core: Fix arch_perf_get_page_size()
From: Peter Zijlstra <[email protected]>
Date: Wed, 11 Nov 2020 13:43:57 +0100
The (new) page-table walker in arch_perf_get_page_size() is broken in various ways. Specifically while it is used in a lockless manner, it doesn't depend on CONFIG_HAVE_FAST_GUP nor uses the proper _lockless offset methods, nor is careful to only read each entry only once.
Also the hugetlb support is broken due to calling pte_page() without first checking pte_special().
Rewrite the whole thing to be a proper lockless page-table walker and employ the new pXX_leaf_size() pgtable functions to determine the pagetable size without looking at the page-frames.
--- kernel/events/core.c | 105 ++++++++++++++++++--------------------------------- 1 file changed, 39 insertions(+), 66 deletions(-)
kernel/events/core.c +39 -66
@@ -52,6 +52,7 @@#include <linux/mount.h> #include <linux/min_heap.h> #include <linux/highmem.h> +#include <linux/pgtable.h> #include "internal.h"
@@ -7001,90 +7002,62 @@ static u64 perf_virt_to_phys(u64 virt)return phys_addr; } -#ifdef CONFIG_MMU -/* -* Return the MMU page size of a given virtual address. -* -* This generic implementation handles page-table aligned huge pages, as well -* as non-page-table aligned hugetlbfs compound pages. -* -* If an architecture supports and uses non-page-table aligned pages in their -* kernel mapping it will need to provide it's own implementation of this -* function. +* Return the pagetable size of a given virtual address. */ -__weak u64 arch_perf_get_page_size (struct mm_struct *mm,unsigned long addr) +static u64 perf_get_pgtable_size (struct mm_struct *mm,unsigned long addr) { -struct page *page; -pgd_t *pgd; -p4d_t *p4d; -pud_t *pud; -pmd_t *pmd; -pte_t *pte; - - pgd =pgd_offset (mm, addr); -if (pgd_none (*pgd)) -return 0 ; +u64 size =0 ; - p4d =p4d_offset (pgd, addr); -if (!p4d_present (*p4d)) +#ifdef CONFIG_HAVE_FAST_GUP +pgd_t *pgdp, pgd; +p4d_t *p4dp, p4d; +pud_t *pudp, pud; +pmd_t *pmdp, pmd; +pte_t *ptep, pte; + + pgdp =pgd_offset (mm, addr); + pgd =READ_ONCE (*pgdp); +if (pgd_none (pgd))return 0 ; -if (p4d_leaf (*p4d)) -return 1ULL <<P4D_SHIFT ; +if (pgd_leaf (pgd)) +return pgd_leaf_size (pgd); - pud =pud_offset (p4d, addr); -if (!pud_present (*pud)) + p4dp =p4d_offset_lockless (pgdp, pgd, addr); + p4d =READ_ONCE (*p4dp); +if (!p4d_present (p4d))return 0 ; -if (pud_leaf (*pud)) { -#ifdef pud_page - page =pud_page (*pud); -if (PageHuge (page)) -return page_size (compound_head (page)); -#endif -return 1ULL <<PUD_SHIFT ; - } +if (p4d_leaf (p4d)) +return p4d_leaf_size (p4d); - pmd =pmd_offset (pud, addr); -if (!pmd_present (*pmd)) + pudp =pud_offset_lockless (p4dp, p4d, addr); + pud =READ_ONCE (*pudp); +if (!pud_present (pud))return 0 ; -if (pmd_leaf (*pmd)) { -#ifdef pmd_page - page =pmd_page (*pmd); -if (PageHuge (page)) -return page_size (compound_head (page)); -#endif -return 1ULL <<PMD_SHIFT ; - } +if (pud_leaf (pud)) +return pud_leaf_size (pud); - pte =pte_offset_map (pmd, addr); -if (!pte_present (*pte)) { -pte_unmap (pte); + pmdp =pmd_offset_lockless (pudp, pud, addr); + pmd =READ_ONCE (*pmdp); +if (!pmd_present (pmd))return 0 ; - } - page =pte_page (*pte); -if (PageHuge (page)) { -u64 size =page_size (compound_head (page)); -pte_unmap (pte); -return size; - } +if (pmd_leaf (pmd)) +return pmd_leaf_size (pmd); -pte_unmap (pte); -return PAGE_SIZE ; -} - -#else + ptep =pte_offset_map (&pmd, addr); + pte =ptep_get_lockless (ptep); +if (pte_present (pte)) + size =pte_leaf_size (pte); +pte_unmap (ptep); +#endif /* CONFIG_HAVE_FAST_GUP */ -static u64 arch_perf_get_page_size (struct mm_struct *mm,unsigned long addr) -{ -return 0 ; +return size; } -#endif -static u64 perf_get_page_size (unsigned long addr) {struct mm_struct *mm;
@@ -7109,7 +7082,7 @@ static u64 perf_get_page_size(unsigned l mm = &init_mm; } - size =arch_perf_get_page_size (mm, addr); + size =perf_get_pgtable_size (mm, addr);local_irq_restore (flags);
Re: [PATCH v2 3/6] perf/core: Fix arch_perf_get_page_size()
kernel/events/core.c static u64 perf_virt_to_phys(u64 virt) · line 7020
Quoted 2 lines · Peter Zijlstra
kernel/events/core.c +2
-0+ pgdp =pgd_offset (mm, addr); + pgd =READ_ONCE (*pgdp);
I forget how x86-32-PAE maps to Linux's PGD/P4D/PUD/PMD scheme, but according to volume 3, section 4.4.2, PAE paging uses a 64-bit PDE, so whether a PDE is a PGD or a PMD, we're only reading it with READ_ONCE rather than the lockless-retry method used by ptep_get_lockless(). So it's potentially racy? Do we need a pmdp_get_lockless() or pgdp_get_lockless()?
[...]
Quoted 8 lines
kernel/events/core.c +5
-0+ pmdp =pmd_offset_lockless (pudp, pud, addr); + pmd =READ_ONCE (*pmdp); +if (!pmd_present (pmd))return 0 ; +if (pmd_leaf (pmd)) +return pmd_leaf_size (pmd);
Peter Zijlstra
maintainerRe: [PATCH v2 3/6] perf/core: Fix arch_perf_get_page_size()
Quoted 10 lines · Matthew Wilcox
kernel/events/core.c static u64 perf_virt_to_phys(u64 virt) · line 7020
Quoted 2 lines · Peter Zijlstra
kernel/events/core.c +2
-0+ pgdp =pgd_offset (mm, addr); + pgd =READ_ONCE (*pgdp);I forget how x86-32-PAE maps to Linux's PGD/P4D/PUD/PMD scheme, but according to volume 3, section 4.4.2, PAE paging uses a 64-bit PDE, so whether a PDE is a PGD or a PMD, we're only reading it with
READ_ONCErather than the lockless-retry method used byptep_get_lockless(). So it's potentially racy? Do we need apmdp_get_lockless()orpgdp_get_lockless()?
Oh gawd... this isn't new here though, right? Current gup_fast also gets that wrong, if it is in deed wrong.
I suppose it's a race far more likely today, with THP and all, than it ever was back then.
Re: [PATCH v2 3/6] perf/core: Fix arch_perf_get_page_size()
Quoted 12 lines · Peter Zijlstra
Quoted 10 lines · Matthew Wilcox
kernel/events/core.c static u64 perf_virt_to_phys(u64 virt) · line 7020
Quoted 2 lines · Peter Zijlstra
kernel/events/core.c +2
-0+ pgdp =pgd_offset (mm, addr); + pgd =READ_ONCE (*pgdp);I forget how x86-32-PAE maps to Linux's PGD/P4D/PUD/PMD scheme, but according to volume 3, section 4.4.2, PAE paging uses a 64-bit PDE, so whether a PDE is a PGD or a PMD, we're only reading it with
READ_ONCErather than the lockless-retry method used byptep_get_lockless(). So it's potentially racy? Do we need apmdp_get_lockless()orpgdp_get_lockless()?
Quoted 5 lines
Oh gawd... this isn't new here though, right? Current
gup_fastalso gets that wrong, if it is in deed wrong.I suppose it's a race far more likely today, with THP and all, than it ever was back then.
Right, it's not new. I wouldn't block this patchset for that fix. Just want to get the problem on your radar ;-) I just never reviewed the gup fast codepath before, and this jumped out at me.
[tip: perf/core] perf/core: Fix arch_perf_get_page_size()
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 96078bdfdc40a83501c5415dba56339f7661b3d1 Gitweb: https://git.kernel.org/tip/96078bdfdc40a83501c5415dba56339f7661b3d1 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 11 Nov 2020 13:43:57 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:00:31 +01:00
perf/core: Fix arch_perf_get_page_size()
The (new) page-table walker in arch_perf_get_page_size() is broken in various ways. Specifically while it is used in a lockless manner, it doesn't depend on CONFIG_HAVE_FAST_GUP nor uses the proper _lockless offset methods, nor is careful to only read each entry only once.
Also the hugetlb support is broken due to calling pte_page() without first checking pte_special().
Rewrite the whole thing to be a proper lockless page-table walker and employ the new pXX_leaf_size() pgtable functions to determine the pagetable size without looking at the page-frames.
--- kernel/events/core.c | 103 +++++++++++++++--------------------------- 1 file changed, 38 insertions(+), 65 deletions(-)
kernel/events/core.c +38 -65
@@ -52,6 +52,7 @@#include <linux/mount.h> #include <linux/min_heap.h> #include <linux/highmem.h> +#include <linux/pgtable.h> #include "internal.h"
@@ -7001,90 +7002,62 @@ static u64 perf_virt_to_phys(u64 virt)return phys_addr; } -#ifdef CONFIG_MMU -/* -* Return the MMU page size of a given virtual address. -* -* This generic implementation handles page-table aligned huge pages, as well -* as non-page-table aligned hugetlbfs compound pages. -* -* If an architecture supports and uses non-page-table aligned pages in their -* kernel mapping it will need to provide it's own implementation of this -* function. +* Return the pagetable size of a given virtual address. */ -__weak u64 arch_perf_get_page_size (struct mm_struct *mm,unsigned long addr) +static u64 perf_get_pgtable_size (struct mm_struct *mm,unsigned long addr) { -struct page *page; -pgd_t *pgd; -p4d_t *p4d; -pud_t *pud; -pmd_t *pmd; -pte_t *pte; +u64 size =0 ; - pgd =pgd_offset (mm, addr); -if (pgd_none (*pgd)) -return 0 ; +#ifdef CONFIG_HAVE_FAST_GUP +pgd_t *pgdp, pgd; +p4d_t *p4dp, p4d; +pud_t *pudp, pud; +pmd_t *pmdp, pmd; +pte_t *ptep, pte; - p4d =p4d_offset (pgd, addr); -if (!p4d_present (*p4d)) + pgdp =pgd_offset (mm, addr); + pgd =READ_ONCE (*pgdp); +if (pgd_none (pgd))return 0 ; -if (p4d_leaf (*p4d)) -return 1ULL <<P4D_SHIFT ; +if (pgd_leaf (pgd)) +return pgd_leaf_size (pgd); - pud =pud_offset (p4d, addr); -if (!pud_present (*pud)) + p4dp =p4d_offset_lockless (pgdp, pgd, addr); + p4d =READ_ONCE (*p4dp); +if (!p4d_present (p4d))return 0 ; -if (pud_leaf (*pud)) { -#ifdef pud_page - page =pud_page (*pud); -if (PageHuge (page)) -return page_size (compound_head (page)); -#endif -return 1ULL <<PUD_SHIFT ; - } +if (p4d_leaf (p4d)) +return p4d_leaf_size (p4d); - pmd =pmd_offset (pud, addr); -if (!pmd_present (*pmd)) + pudp =pud_offset_lockless (p4dp, p4d, addr); + pud =READ_ONCE (*pudp); +if (!pud_present (pud))return 0 ; -if (pmd_leaf (*pmd)) { -#ifdef pmd_page - page =pmd_page (*pmd); -if (PageHuge (page)) -return page_size (compound_head (page)); -#endif -return 1ULL <<PMD_SHIFT ; - } +if (pud_leaf (pud)) +return pud_leaf_size (pud); - pte =pte_offset_map (pmd, addr); -if (!pte_present (*pte)) { -pte_unmap (pte); + pmdp =pmd_offset_lockless (pudp, pud, addr); + pmd =READ_ONCE (*pmdp); +if (!pmd_present (pmd))return 0 ; - } - page =pte_page (*pte); -if (PageHuge (page)) { -u64 size =page_size (compound_head (page)); -pte_unmap (pte); -return size; - } +if (pmd_leaf (pmd)) +return pmd_leaf_size (pmd); -pte_unmap (pte); -return PAGE_SIZE ; -} + ptep =pte_offset_map (&pmd, addr); + pte =ptep_get_lockless (ptep); +if (pte_present (pte)) + size =pte_leaf_size (pte); +pte_unmap (ptep); +#endif /* CONFIG_HAVE_FAST_GUP */ -#else - -static u64 arch_perf_get_page_size (struct mm_struct *mm,unsigned long addr) -{ -return 0 ; +return size; } -#endif -static u64 perf_get_page_size (unsigned long addr) {struct mm_struct *mm;
@@ -7109,7 +7082,7 @@ static u64 perf_get_page_size(unsigned long addr) mm = &init_mm; } - size =arch_perf_get_page_size (mm, addr); + size =perf_get_pgtable_size (mm, addr);local_irq_restore (flags);
[tip: perf/core] perf/core: Fix arch_perf_get_page_size()
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 8af26be062721e52eba1550caf50b712f774c5fd Gitweb: https://git.kernel.org/tip/8af26be062721e52eba1550caf50b712f774c5fd Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 11 Nov 2020 13:43:57 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:14:51 +01:00
perf/core: Fix arch_perf_get_page_size()
The (new) page-table walker in arch_perf_get_page_size() is broken in various ways. Specifically while it is used in a lockless manner, it doesn't depend on CONFIG_HAVE_FAST_GUP nor uses the proper _lockless offset methods, nor is careful to only read each entry only once.
Also the hugetlb support is broken due to calling pte_page() without first checking pte_special().
Rewrite the whole thing to be a proper lockless page-table walker and employ the new pXX_leaf_size() pgtable functions to determine the pagetable size without looking at the page-frames.
--- kernel/events/core.c | 103 +++++++++++++++--------------------------- 1 file changed, 38 insertions(+), 65 deletions(-)
kernel/events/core.c +38 -65
@@ -52,6 +52,7 @@#include <linux/mount.h> #include <linux/min_heap.h> #include <linux/highmem.h> +#include <linux/pgtable.h> #include "internal.h"
@@ -7001,90 +7002,62 @@ static u64 perf_virt_to_phys(u64 virt)return phys_addr; } -#ifdef CONFIG_MMU -/* -* Return the MMU page size of a given virtual address. -* -* This generic implementation handles page-table aligned huge pages, as well -* as non-page-table aligned hugetlbfs compound pages. -* -* If an architecture supports and uses non-page-table aligned pages in their -* kernel mapping it will need to provide it's own implementation of this -* function. +* Return the pagetable size of a given virtual address. */ -__weak u64 arch_perf_get_page_size (struct mm_struct *mm,unsigned long addr) +static u64 perf_get_pgtable_size (struct mm_struct *mm,unsigned long addr) { -struct page *page; -pgd_t *pgd; -p4d_t *p4d; -pud_t *pud; -pmd_t *pmd; -pte_t *pte; +u64 size =0 ; - pgd =pgd_offset (mm, addr); -if (pgd_none (*pgd)) -return 0 ; +#ifdef CONFIG_HAVE_FAST_GUP +pgd_t *pgdp, pgd; +p4d_t *p4dp, p4d; +pud_t *pudp, pud; +pmd_t *pmdp, pmd; +pte_t *ptep, pte; - p4d =p4d_offset (pgd, addr); -if (!p4d_present (*p4d)) + pgdp =pgd_offset (mm, addr); + pgd =READ_ONCE (*pgdp); +if (pgd_none (pgd))return 0 ; -if (p4d_leaf (*p4d)) -return 1ULL <<P4D_SHIFT ; +if (pgd_leaf (pgd)) +return pgd_leaf_size (pgd); - pud =pud_offset (p4d, addr); -if (!pud_present (*pud)) + p4dp =p4d_offset_lockless (pgdp, pgd, addr); + p4d =READ_ONCE (*p4dp); +if (!p4d_present (p4d))return 0 ; -if (pud_leaf (*pud)) { -#ifdef pud_page - page =pud_page (*pud); -if (PageHuge (page)) -return page_size (compound_head (page)); -#endif -return 1ULL <<PUD_SHIFT ; - } +if (p4d_leaf (p4d)) +return p4d_leaf_size (p4d); - pmd =pmd_offset (pud, addr); -if (!pmd_present (*pmd)) + pudp =pud_offset_lockless (p4dp, p4d, addr); + pud =READ_ONCE (*pudp); +if (!pud_present (pud))return 0 ; -if (pmd_leaf (*pmd)) { -#ifdef pmd_page - page =pmd_page (*pmd); -if (PageHuge (page)) -return page_size (compound_head (page)); -#endif -return 1ULL <<PMD_SHIFT ; - } +if (pud_leaf (pud)) +return pud_leaf_size (pud); - pte =pte_offset_map (pmd, addr); -if (!pte_present (*pte)) { -pte_unmap (pte); + pmdp =pmd_offset_lockless (pudp, pud, addr); + pmd =READ_ONCE (*pmdp); +if (!pmd_present (pmd))return 0 ; - } - page =pte_page (*pte); -if (PageHuge (page)) { -u64 size =page_size (compound_head (page)); -pte_unmap (pte); -return size; - } +if (pmd_leaf (pmd)) +return pmd_leaf_size (pmd); -pte_unmap (pte); -return PAGE_SIZE ; -} + ptep =pte_offset_map (&pmd, addr); + pte =ptep_get_lockless (ptep); +if (pte_present (pte)) + size =pte_leaf_size (pte); +pte_unmap (ptep); +#endif /* CONFIG_HAVE_FAST_GUP */ -#else - -static u64 arch_perf_get_page_size (struct mm_struct *mm,unsigned long addr) -{ -return 0 ; +return size; } -#endif -static u64 perf_get_page_size (unsigned long addr) {struct mm_struct *mm;
@@ -7109,7 +7082,7 @@ static u64 perf_get_page_size(unsigned long addr) mm = &init_mm; } - size =arch_perf_get_page_size (mm, addr); + size =perf_get_pgtable_size (mm, addr);local_irq_restore (flags);
Peter Zijlstra
maintainerPATCH v2 4/6 arm64/mm: Implement pXX_leaf_size() support
In reply to Peter ZijlstraARM64 has non-pagetable aligned large page support with PTE_CONT, when this bit is set the page is part of a super-page. Match the hugetlb code and support these super pages for PTE and PMD levels.
This enables PERF_SAMPLE_{DATA,CODE}_PAGE_SIZE to report accurate pagetable leaf sizes.
--- arch/arm64/include/asm/pgtable.h | 3 +++ 1 file changed, 3 insertions(+)
arch/arm64/include/asm/pgtable.h +3 -0
@@ -503,6 +503,9 @@ extern pgprot_t phys_mem_access_prot(str PMD_TYPE_SECT)#define pmd_leaf (pmd) pmd_sect(pmd) +#define pmd_leaf_size (pmd) (pmd_cont(pmd) ? CONT_PMD_SIZE : PMD_SIZE) +#define pte_leaf_size (pte) (pte_cont(pte) ? CONT_PTE_SIZE : PAGE_SIZE) +#if defined(CONFIG_ARM64_64K_PAGES ) ||CONFIG_PGTABLE_LEVELS <3 static inline bool pud_sect (pud_t pud) {return false; }static inline bool pud_table (pud_t pud) {return true; }
Peter Zijlstra
maintainerRe: [PATCH v2 4/6] arm64/mm: Implement pXX_leaf_size() support
Now with pmd_cont() defined...
---
Subject: arm64/mm: Implement pXX_leaf_size() support
From: Peter Zijlstra <[email protected]>
Date: Fri Nov 13 11:46:06 CET 2020
ARM64 has non-pagetable aligned large page support with PTE_CONT, when this bit is set the page is part of a super-page. Match the hugetlb code and support these super pages for PTE and PMD levels.
This enables PERF_SAMPLE_{DATA,CODE}_PAGE_SIZE to report accurate pagetable leaf sizes.
--- arch/arm64/include/asm/pgtable.h | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/include/asm/pgtable.h +4 -0
@@ -407,6 +407,7 @@ static inline int pmd_trans_huge(pmd_t p#define pmd_dirty (pmd) pte_dirty(pmd_pte(pmd))#define pmd_young (pmd) pte_young(pmd_pte(pmd))#define pmd_valid (pmd) pte_valid(pmd_pte(pmd)) +#define pmd_cont (pmd) pte_cont(pmd_pte(pmd))#define pmd_wrprotect (pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd)))#define pmd_mkold (pmd) pte_pmd(pte_mkold(pmd_pte(pmd)))#define pmd_mkwrite (pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd)))
@@ -503,6 +504,9 @@ extern pgprot_t phys_mem_access_prot(str PMD_TYPE_SECT)#define pmd_leaf (pmd) pmd_sect(pmd) +#define pmd_leaf_size (pmd) (pmd_cont(pmd) ? CONT_PMD_SIZE : PMD_SIZE) +#define pte_leaf_size (pte) (pte_cont(pte) ? CONT_PTE_SIZE : PAGE_SIZE) +#if defined(CONFIG_ARM64_64K_PAGES ) ||CONFIG_PGTABLE_LEVELS <3 static inline bool pud_sect (pud_t pud) {return false; }static inline bool pud_table (pud_t pud) {return true; }
Re: [PATCH v2 4/6] arm64/mm: Implement pXX_leaf_size() support
Quoted 31 lines · Peter Zijlstra
Now with
pmd_cont()defined...---
Subject: arm64/mm: ImplementpXX_leaf_size()support
From: Peter Zijlstra <[email protected]>
Date: Fri Nov 13 11:46:06 CET 2020ARM64 has non-pagetable aligned large page support with
PTE_CONT, when this bit is set the page is part of a super-page. Match the hugetlb code and support these super pages for PTE and PMD levels.This enables PERF_SAMPLE_{DATA,CODE}_PAGE_SIZE to report accurate pagetable leaf sizes.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/arm64/include/asm/pgtable.h | 4 ++++ 1 file changed, 4 insertions(+)arch/arm64/include/asm/pgtable.h +1
-0@@ -407,6 +407,7 @@ static inline int pmd_trans_huge(pmd_t p#define pmd_dirty (pmd) pte_dirty(pmd_pte(pmd))#define pmd_young (pmd) pte_young(pmd_pte(pmd))#define pmd_valid (pmd) pte_valid(pmd_pte(pmd)) +#define pmd_cont (pmd) pte_cont(pmd_pte(pmd))#define pmd_wrprotect (pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd)))#define pmd_mkold (pmd) pte_pmd(pte_mkold(pmd_pte(pmd)))#define pmd_mkwrite (pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd)))
arch/arm64/include/asm/pgtable.h extern pgprot_t phys_mem_access_prot(str · line 512
Quoted 9 lines
arch/arm64/include/asm/pgtable.h +3
-0@@ -503,6 +504,9 @@ extern pgprot_t phys_mem_access_prot(str PMD_TYPE_SECT)#define pmd_leaf (pmd) pmd_sect(pmd) +#define pmd_leaf_size (pmd) (pmd_cont(pmd) ? CONT_PMD_SIZE : PMD_SIZE) +#define pte_leaf_size (pte) (pte_cont(pte) ? CONT_PTE_SIZE : PAGE_SIZE) +#if defined(CONFIG_ARM64_64K_PAGES ) ||CONFIG_PGTABLE_LEVELS <3 static inline bool pud_sect (pud_t pud) {return false; }static inline bool pud_table (pud_t pud) {return true; }
I'm still highly dubious about the utility of this feature in perf, since the TLB entry size is pretty much independent of the page-table configuration, but that's a problem for all architectures I suspect.
Will
[tip: perf/core] arm64/mm: Implement pXX_leaf_size() support
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 311c656945eccc9304aa4e8f3dee7cbfdbabb72d Gitweb: https://git.kernel.org/tip/311c656945eccc9304aa4e8f3dee7cbfdbabb72d Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 13 Nov 2020 11:46:06 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:00:31 +01:00
arm64/mm: Implement pXX_leaf_size() support
ARM64 has non-pagetable aligned large page support with PTE_CONT, when this bit is set the page is part of a super-page. Match the hugetlb code and support these super pages for PTE and PMD levels.
This enables PERF_SAMPLE_{DATA,CODE}_PAGE_SIZE to report accurate pagetable leaf sizes.
--- arch/arm64/include/asm/pgtable.h | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/include/asm/pgtable.h +4 -0
@@ -415,6 +415,7 @@ static inline int pmd_trans_huge(pmd_t pmd)#define pmd_dirty (pmd) pte_dirty(pmd_pte(pmd))#define pmd_young (pmd) pte_young(pmd_pte(pmd))#define pmd_valid (pmd) pte_valid(pmd_pte(pmd)) +#define pmd_cont (pmd) pte_cont(pmd_pte(pmd))#define pmd_wrprotect (pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd)))#define pmd_mkold (pmd) pte_pmd(pte_mkold(pmd_pte(pmd)))#define pmd_mkwrite (pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd)))
@@ -511,6 +512,9 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, PMD_TYPE_SECT)#define pmd_leaf (pmd) pmd_sect(pmd) +#define pmd_leaf_size (pmd) (pmd_cont(pmd) ? CONT_PMD_SIZE : PMD_SIZE) +#define pte_leaf_size (pte) (pte_cont(pte) ? CONT_PTE_SIZE : PAGE_SIZE) +#if defined(CONFIG_ARM64_64K_PAGES ) ||CONFIG_PGTABLE_LEVELS <3 static inline bool pud_sect (pud_t pud) {return false; }static inline bool pud_table (pud_t pud) {return true; }
[tip: perf/core] arm64/mm: Implement pXX_leaf_size() support
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: d55863db1dfec8845067f5625f1b0ab18c8948be Gitweb: https://git.kernel.org/tip/d55863db1dfec8845067f5625f1b0ab18c8948be Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 13 Nov 2020 11:46:06 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:14:51 +01:00
arm64/mm: Implement pXX_leaf_size() support
ARM64 has non-pagetable aligned large page support with PTE_CONT, when this bit is set the page is part of a super-page. Match the hugetlb code and support these super pages for PTE and PMD levels.
This enables PERF_SAMPLE_{DATA,CODE}_PAGE_SIZE to report accurate pagetable leaf sizes.
--- arch/arm64/include/asm/pgtable.h | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/include/asm/pgtable.h +4 -0
@@ -407,6 +407,7 @@ static inline int pmd_trans_huge(pmd_t pmd)#define pmd_dirty (pmd) pte_dirty(pmd_pte(pmd))#define pmd_young (pmd) pte_young(pmd_pte(pmd))#define pmd_valid (pmd) pte_valid(pmd_pte(pmd)) +#define pmd_cont (pmd) pte_cont(pmd_pte(pmd))#define pmd_wrprotect (pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd)))#define pmd_mkold (pmd) pte_pmd(pte_mkold(pmd_pte(pmd)))#define pmd_mkwrite (pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd)))
@@ -503,6 +504,9 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, PMD_TYPE_SECT)#define pmd_leaf (pmd) pmd_sect(pmd) +#define pmd_leaf_size (pmd) (pmd_cont(pmd) ? CONT_PMD_SIZE : PMD_SIZE) +#define pte_leaf_size (pte) (pte_cont(pte) ? CONT_PTE_SIZE : PAGE_SIZE) +#if defined(CONFIG_ARM64_64K_PAGES ) ||CONFIG_PGTABLE_LEVELS <3 static inline bool pud_sect (pud_t pud) {return false; }static inline bool pud_table (pud_t pud) {return true; }
Peter Zijlstra
maintainerPATCH v2 5/6 sparc64/mm: Implement pXX_leaf_size() support
In reply to Peter ZijlstraSparc64 has non-pagetable aligned large page support; wire up the pXX_leaf_size() functions to report the correct pagetable page size.
This enables PERF_SAMPLE_{DATA,CODE}_PAGE_SIZE to report accurate pagetable leaf sizes.
--- arch/sparc/include/asm/pgtable_64.h | 13 +++++++++++++ arch/sparc/mm/hugetlbpage.c | 19 +++++++++++++------ 2 files changed, 26 insertions(+), 6 deletions(-)
arch/sparc/include/asm/pgtable_64.h +13 -0
@@ -1121,6 +1121,19 @@ extern unsigned long cmdline_memory_size asmlinkage void do_sparc64_fault(struct pt_regs *regs); +#ifdef CONFIG_HUGETLB_PAGE + +#define pud_leaf_size pud_leaf_size +extern unsigned long pud_leaf_size (pud_t pud); + +#define pmd_leaf_size pmd_leaf_size +extern unsigned long pmd_leaf_size (pmd_t pmd); + +#define pte_leaf_size pte_leaf_size +extern unsigned long pte_leaf_size (pte_t pte); + +#endif /* CONFIG_HUGETLB_PAGE */ +#endif /* !(__ASSEMBLY__) */ #endif /* !(_SPARC64_PGTABLE_H) */
arch/sparc/mm/hugetlbpage.c +13 -6
@@ -247,14 +247,17 @@ static unsigned int sun4u_huge_tte_to_shreturn shift; } -static unsigned int huge_tte_to_shift (pte_t entry) +static unsigned long tte_to_shift (pte_t entry) { -unsigned long shift; -if (tlb_type = hypervisor) - shift =sun4v_huge_tte_to_shift (entry); -else - shift =sun4u_huge_tte_to_shift (entry); +return sun4v_huge_tte_to_shift (entry); + +return sun4u_huge_tte_to_shift (entry); +} + +static unsigned int huge_tte_to_shift (pte_t entry) +{ +unsigned long shift =tte_to_shift (entry);if (shift =PAGE_SHIFT ) WARN_ONCE(1, "tto_to_shift: invalid hugepage tte=0x%lx\n",
@@ -272,6 +275,10 @@ static unsigned long huge_tte_to_size(ptreturn size; } +unsigned long pud_leaf_size (pud_t pud) {return 1UL <<tte_to_shift ((pte_t )pud); } +unsigned long pmd_leaf_size (pmd_t pmd) {return 1UL <<tte_to_shift ((pte_t )pmd); } +unsigned long pte_leaf_size (pte_t pte) {return 1UL <<tte_to_shift ((pte_t )pte); } +pte_t *huge_pte_alloc (struct mm_struct *mm,unsigned long addr,unsigned long sz) {
[tip: perf/core] sparc64/mm: Implement pXX_leaf_size() support
The following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 58aeee3cecc5fd4922bbd21c294905267baf4edd Gitweb: https://git.kernel.org/tip/58aeee3cecc5fd4922bbd21c294905267baf4edd Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 13 Nov 2020 11:46:23 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:00:31 +01:00
sparc64/mm: Implement pXX_leaf_size() support
Sparc64 has non-pagetable aligned large page support; wire up the pXX_leaf_size() functions to report the correct pagetable page size.
This enables PERF_SAMPLE_{DATA,CODE}_PAGE_SIZE to report accurate pagetable leaf sizes.
--- arch/sparc/include/asm/pgtable_64.h | 13 +++++++++++++ arch/sparc/mm/hugetlbpage.c | 19 +++++++++++++------ 2 files changed, 26 insertions(+), 6 deletions(-)
arch/sparc/include/asm/pgtable_64.h +13 -0
@@ -1121,6 +1121,19 @@ extern unsigned long cmdline_memory_size; asmlinkage void do_sparc64_fault(struct pt_regs *regs); +#ifdef CONFIG_HUGETLB_PAGE + +#define pud_leaf_size pud_leaf_size +extern unsigned long pud_leaf_size (pud_t pud); + +#define pmd_leaf_size pmd_leaf_size +extern unsigned long pmd_leaf_size (pmd_t pmd); + +#define pte_leaf_size pte_leaf_size +extern unsigned long pte_leaf_size (pte_t pte); + +#endif /* CONFIG_HUGETLB_PAGE */ +#endif /* !(__ASSEMBLY__) */ #endif /* !(_SPARC64_PGTABLE_H) */
arch/sparc/mm/hugetlbpage.c +13 -6
@@ -247,14 +247,17 @@ static unsigned int sun4u_huge_tte_to_shift(pte_t entry)return shift; } -static unsigned int huge_tte_to_shift (pte_t entry) +static unsigned long tte_to_shift (pte_t entry) { -unsigned long shift; -if (tlb_type == hypervisor) - shift =sun4v_huge_tte_to_shift (entry); -else - shift =sun4u_huge_tte_to_shift (entry); +return sun4v_huge_tte_to_shift (entry); + +return sun4u_huge_tte_to_shift (entry); +} + +static unsigned int huge_tte_to_shift (pte_t entry) +{ +unsigned long shift =tte_to_shift (entry);if (shift ==PAGE_SHIFT ) WARN_ONCE(1, "tto_to_shift: invalid hugepage tte=0x%lx\n",
@@ -272,6 +275,10 @@ static unsigned long huge_tte_to_size(pte_t pte)return size; } +unsigned long pud_leaf_size (pud_t pud) {return 1UL <<tte_to_shift ((pte_t )pud); } +unsigned long pmd_leaf_size (pmd_t pmd) {return 1UL <<tte_to_shift ((pte_t )pmd); } +unsigned long pte_leaf_size (pte_t pte) {return 1UL <<tte_to_shift ((pte_t )pte); } +pte_t *huge_pte_alloc (struct mm_struct *mm,unsigned long addr,unsigned long sz) {
[tip: perf/core] sparc64/mm: Implement pXX_leaf_size() support
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 974821786fbc9c5c94ae75d96246c58bc0dc67bb Gitweb: https://git.kernel.org/tip/974821786fbc9c5c94ae75d96246c58bc0dc67bb Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 13 Nov 2020 11:46:23 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:14:51 +01:00
sparc64/mm: Implement pXX_leaf_size() support
Sparc64 has non-pagetable aligned large page support; wire up the pXX_leaf_size() functions to report the correct pagetable page size.
This enables PERF_SAMPLE_{DATA,CODE}_PAGE_SIZE to report accurate pagetable leaf sizes.
--- arch/sparc/include/asm/pgtable_64.h | 13 +++++++++++++ arch/sparc/mm/hugetlbpage.c | 19 +++++++++++++------ 2 files changed, 26 insertions(+), 6 deletions(-)
arch/sparc/include/asm/pgtable_64.h +13 -0
@@ -1121,6 +1121,19 @@ extern unsigned long cmdline_memory_size; asmlinkage void do_sparc64_fault(struct pt_regs *regs); +#ifdef CONFIG_HUGETLB_PAGE + +#define pud_leaf_size pud_leaf_size +extern unsigned long pud_leaf_size (pud_t pud); + +#define pmd_leaf_size pmd_leaf_size +extern unsigned long pmd_leaf_size (pmd_t pmd); + +#define pte_leaf_size pte_leaf_size +extern unsigned long pte_leaf_size (pte_t pte); + +#endif /* CONFIG_HUGETLB_PAGE */ +#endif /* !(__ASSEMBLY__) */ #endif /* !(_SPARC64_PGTABLE_H) */
arch/sparc/mm/hugetlbpage.c +13 -6
@@ -247,14 +247,17 @@ static unsigned int sun4u_huge_tte_to_shift(pte_t entry)return shift; } -static unsigned int huge_tte_to_shift (pte_t entry) +static unsigned long tte_to_shift (pte_t entry) { -unsigned long shift; -if (tlb_type == hypervisor) - shift =sun4v_huge_tte_to_shift (entry); -else - shift =sun4u_huge_tte_to_shift (entry); +return sun4v_huge_tte_to_shift (entry); + +return sun4u_huge_tte_to_shift (entry); +} + +static unsigned int huge_tte_to_shift (pte_t entry) +{ +unsigned long shift =tte_to_shift (entry);if (shift ==PAGE_SHIFT ) WARN_ONCE(1, "tto_to_shift: invalid hugepage tte=0x%lx\n",
@@ -272,6 +275,10 @@ static unsigned long huge_tte_to_size(pte_t pte)return size; } +unsigned long pud_leaf_size (pud_t pud) {return 1UL <<tte_to_shift ((pte_t )pud); } +unsigned long pmd_leaf_size (pmd_t pmd) {return 1UL <<tte_to_shift ((pte_t )pmd); } +unsigned long pte_leaf_size (pte_t pte) {return 1UL <<tte_to_shift ((pte_t )pte); } +pte_t *huge_pte_alloc (struct mm_struct *mm,unsigned long addr,unsigned long sz) {
[tip: perf/core] sparc64/mm: Implement pXX_leaf_size() support
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: e6e4f42eb773c1da869af4bad544c26c89cd01ab Gitweb: https://git.kernel.org/tip/e6e4f42eb773c1da869af4bad544c26c89cd01ab Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 13 Nov 2020 11:46:23 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 09 Dec 2020 17:08:56 +01:00
sparc64/mm: Implement pXX_leaf_size() support
Sparc64 has non-pagetable aligned large page support; wire up the pXX_leaf_size() functions to report the correct pagetable page size.
This enables PERF_SAMPLE_{DATA,CODE}_PAGE_SIZE to report accurate pagetable leaf sizes.
--- arch/sparc/include/asm/pgtable_64.h | 13 +++++++++++++ arch/sparc/mm/hugetlbpage.c | 19 +++++++++++++------ 2 files changed, 26 insertions(+), 6 deletions(-)
arch/sparc/include/asm/pgtable_64.h +13 -0
@@ -1121,6 +1121,19 @@ extern unsigned long cmdline_memory_size; asmlinkage void do_sparc64_fault(struct pt_regs *regs); +#ifdef CONFIG_HUGETLB_PAGE + +#define pud_leaf_size pud_leaf_size +extern unsigned long pud_leaf_size (pud_t pud); + +#define pmd_leaf_size pmd_leaf_size +extern unsigned long pmd_leaf_size (pmd_t pmd); + +#define pte_leaf_size pte_leaf_size +extern unsigned long pte_leaf_size (pte_t pte); + +#endif /* CONFIG_HUGETLB_PAGE */ +#endif /* !(__ASSEMBLY__) */ #endif /* !(_SPARC64_PGTABLE_H) */
arch/sparc/mm/hugetlbpage.c +13 -6
@@ -247,14 +247,17 @@ static unsigned int sun4u_huge_tte_to_shift(pte_t entry)return shift; } -static unsigned int huge_tte_to_shift (pte_t entry) +static unsigned long tte_to_shift (pte_t entry) { -unsigned long shift; -if (tlb_type == hypervisor) - shift =sun4v_huge_tte_to_shift (entry); -else - shift =sun4u_huge_tte_to_shift (entry); +return sun4v_huge_tte_to_shift (entry); + +return sun4u_huge_tte_to_shift (entry); +} + +static unsigned int huge_tte_to_shift (pte_t entry) +{ +unsigned long shift =tte_to_shift (entry);if (shift ==PAGE_SHIFT ) WARN_ONCE(1, "tto_to_shift: invalid hugepage tte=0x%lx\n",
@@ -272,6 +275,10 @@ static unsigned long huge_tte_to_size(pte_t pte)return size; } +unsigned long pud_leaf_size (pud_t pud) {return 1UL <<tte_to_shift (*(pte_t *)&pud); } +unsigned long pmd_leaf_size (pmd_t pmd) {return 1UL <<tte_to_shift (*(pte_t *)&pmd); } +unsigned long pte_leaf_size (pte_t pte) {return 1UL <<tte_to_shift (pte); } +pte_t *huge_pte_alloc (struct mm_struct *mm,unsigned long addr,unsigned long sz) {
Peter Zijlstra
maintainerPATCH v2 6/6 powerpc/8xx: Implement pXX_leaf_size() support
In reply to Peter ZijlstraQuoted 15 lines · Christophe Leroy
I can help with powerpc 8xx. It is a 32 bits powerpc. The PGD has 1024 entries, that means each entry maps 4M.
Page sizes are 4k, 16k, 512k and 8M.
For the 8M pages we use hugepd with a single entry. The two related PGD entries point to the same hugepd.
For the other sizes, they are in standard page tables. 16k pages appear 4 times in the page table. 512k entries appear 128 times in the page table.
When the PGD entry has
_PMD_PAGE_8Mbits, the PMD entry points to a hugepd with holds the single 8M entry.
Quoted 7 lines
In the PTE, we have two bits:
_PAGE_SPSand_PAGE_HUGE
_PAGE_HUGEmeans it is a 512k page_PAGE_SPSmeans it is not a 4k pageThe kernel can by build either with 4k pages as standard page size, or 16k pages. It doesn't change the page table layout though.
--- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
arch/powerpc/include/asm/nohash/32/pte-8xx.h +23 -0
@@ -135,6 +135,29 @@ static inline pte_t pte_mkhuge(pte_t pte }#define pte_mkhuge pte_mkhuge + +static inline unsigned long pgd_leaf_size (pgd_t pgd) +{ +if (pgd_val (pgd) & _PMD_PAGE_8M) +return SZ_8M ; +return SZ_4M ; +} + +#define pgd_leaf_size pgd_leaf_size + +static inline unsigned long pte_leaf_size (pte_t pte) +{ +pte_basic_t val =pte_val (pte); + +if (val & _PAGE_HUGE) +return SZ_512K ; +if (val & _PAGE_SPS) +return SZ_16K ; +return SZ_4K ; +} + +#define pte_leaf_size pte_leaf_size +#endif #endif /* __KERNEL__ */
[tip: perf/core] powerpc/8xx: Implement pXX_leaf_size() support
The following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 9e7d61ae8acc62ef6dd5fc5f4033ed9420372599 Gitweb: https://git.kernel.org/tip/9e7d61ae8acc62ef6dd5fc5f4033ed9420372599 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 26 Nov 2020 11:53:33 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:00:32 +01:00
powerpc/8xx: Implement pXX_leaf_size() support
Quoted 15 lines · Christophe Leroy
I can help with powerpc 8xx. It is a 32 bits powerpc. The PGD has 1024 entries, that means each entry maps 4M.
Page sizes are 4k, 16k, 512k and 8M.
For the 8M pages we use hugepd with a single entry. The two related PGD entries point to the same hugepd.
For the other sizes, they are in standard page tables. 16k pages appear 4 times in the page table. 512k entries appear 128 times in the page table.
When the PGD entry has
_PMD_PAGE_8Mbits, the PMD entry points to a hugepd with holds the single 8M entry.
Quoted 7 lines
In the PTE, we have two bits:
_PAGE_SPSand_PAGE_HUGE
_PAGE_HUGEmeans it is a 512k page_PAGE_SPSmeans it is not a 4k pageThe kernel can by build either with 4k pages as standard page size, or 16k pages. It doesn't change the page table layout though.
--- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 23 +++++++++++++++++++- 1 file changed, 23 insertions(+)
arch/powerpc/include/asm/nohash/32/pte-8xx.h +23 -0
@@ -135,6 +135,29 @@ static inline pte_t pte_mkhuge(pte_t pte) }#define pte_mkhuge pte_mkhuge + +static inline unsigned long pgd_leaf_size (pgd_t pgd) +{ +if (pgd_val (pgd) & _PMD_PAGE_8M) +return SZ_8M ; +return SZ_4M ; +} + +#define pgd_leaf_size pgd_leaf_size + +static inline unsigned long pte_leaf_size (pte_t pte) +{ +pte_basic_t val =pte_val (pte); + +if (val & _PAGE_HUGE) +return SZ_512K ; +if (val & _PAGE_SPS) +return SZ_16K ; +return SZ_4K ; +} + +#define pte_leaf_size pte_leaf_size +#endif #endif /* __KERNEL__ */
[tip: perf/core] powerpc/8xx: Implement pXX_leaf_size() support
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: c88a82f668cff457561272632a06a4a63dbf2fe0 Gitweb: https://git.kernel.org/tip/c88a82f668cff457561272632a06a4a63dbf2fe0 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 26 Nov 2020 11:53:33 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 03 Dec 2020 10:14:52 +01:00
powerpc/8xx: Implement pXX_leaf_size() support
Quoted 15 lines · Christophe Leroy
I can help with powerpc 8xx. It is a 32 bits powerpc. The PGD has 1024 entries, that means each entry maps 4M.
Page sizes are 4k, 16k, 512k and 8M.
For the 8M pages we use hugepd with a single entry. The two related PGD entries point to the same hugepd.
For the other sizes, they are in standard page tables. 16k pages appear 4 times in the page table. 512k entries appear 128 times in the page table.
When the PGD entry has
_PMD_PAGE_8Mbits, the PMD entry points to a hugepd with holds the single 8M entry.
Quoted 7 lines
In the PTE, we have two bits:
_PAGE_SPSand_PAGE_HUGE
_PAGE_HUGEmeans it is a 512k page_PAGE_SPSmeans it is not a 4k pageThe kernel can by build either with 4k pages as standard page size, or 16k pages. It doesn't change the page table layout though.
--- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 23 +++++++++++++++++++- 1 file changed, 23 insertions(+)
arch/powerpc/include/asm/nohash/32/pte-8xx.h +23 -0
@@ -135,6 +135,29 @@ static inline pte_t pte_mkhuge(pte_t pte) }#define pte_mkhuge pte_mkhuge + +static inline unsigned long pgd_leaf_size (pgd_t pgd) +{ +if (pgd_val (pgd) & _PMD_PAGE_8M) +return SZ_8M ; +return SZ_4M ; +} + +#define pgd_leaf_size pgd_leaf_size + +static inline unsigned long pte_leaf_size (pte_t pte) +{ +pte_basic_t val =pte_val (pte); + +if (val & _PAGE_HUGE) +return SZ_512K ; +if (val & _PAGE_SPS) +return SZ_16K ; +return SZ_4K ; +} + +#define pte_leaf_size pte_leaf_size +#endif #endif /* __KERNEL__ */
[tip: perf/core] powerpc/8xx: Implement pXX_leaf_size() support
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: c5eecbb58f65bf1c4effab9a7f283184b469768c Gitweb: https://git.kernel.org/tip/c5eecbb58f65bf1c4effab9a7f283184b469768c Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 26 Nov 2020 11:53:33 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 09 Dec 2020 17:08:56 +01:00
powerpc/8xx: Implement pXX_leaf_size() support
Quoted 15 lines · Christophe Leroy
I can help with powerpc 8xx. It is a 32 bits powerpc. The PGD has 1024 entries, that means each entry maps 4M.
Page sizes are 4k, 16k, 512k and 8M.
For the 8M pages we use hugepd with a single entry. The two related PGD entries point to the same hugepd.
For the other sizes, they are in standard page tables. 16k pages appear 4 times in the page table. 512k entries appear 128 times in the page table.
When the PGD entry has
_PMD_PAGE_8Mbits, the PMD entry points to a hugepd with holds the single 8M entry.
Quoted 7 lines
In the PTE, we have two bits:
_PAGE_SPSand_PAGE_HUGE
_PAGE_HUGEmeans it is a 512k page_PAGE_SPSmeans it is not a 4k pageThe kernel can by build either with 4k pages as standard page size, or 16k pages. It doesn't change the page table layout though.
--- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 23 +++++++++++++++++++- 1 file changed, 23 insertions(+)
arch/powerpc/include/asm/nohash/32/pte-8xx.h +23 -0
@@ -135,6 +135,29 @@ static inline pte_t pte_mkhuge(pte_t pte) }#define pte_mkhuge pte_mkhuge + +static inline unsigned long pgd_leaf_size (pgd_t pgd) +{ +if (pgd_val (pgd) & _PMD_PAGE_8M) +return SZ_8M ; +return SZ_4M ; +} + +#define pgd_leaf_size pgd_leaf_size + +static inline unsigned long pte_leaf_size (pte_t pte) +{ +pte_basic_t val =pte_val (pte); + +if (val & _PAGE_HUGE) +return SZ_512K ; +if (val & _PAGE_SPS) +return SZ_16K ; +return SZ_4K ; +} + +#define pte_leaf_size pte_leaf_size +#endif #endif /* __KERNEL__ */