Generic page walk and ptdump
Subsystems (19)
v17 cover Generic page walk and ptdump
This version is two patches lighter than v16:
mm: pagewalk: Add test_p?d callbacks
x86: mm: Convert ptdump_walk_pgd_level_core() to take an mm_struct
There is a new enum page_walk_action in struct mm_walk which allows the callbacks to decide whether to recurse (ACTION_SUBTREE - the default), skip (ACTION_CONTINUE) or to be called again (ACTION_AGAIN). These are introduced in patch 11 and used for hmm.c to adapt it to the new behaviour of the pud_entry() callback.
The new page_walk_action means that we no longer need to test_p?d() callbacks as the p?d_entry() callbacks can simply return ACTION_CONTINUE to skip parts of the tree.
The x86 patch is dropped because I've updated patch 20 to always use a 'fake_mm' structure which simplifies the code and avoids issues with potentially double locking the mmap_sem of the real mm.
Previous description for the series
-----------------------------------
Many architectures current have a debugfs file for dumping the kernel page tables. Currently each architecture has to implement custom functions for this because the details of walking the page tables used by the kernel are different between architectures.
This series extends the capabilities of walk_page_range() so that it can deal with the page tables of the kernel (which have no VMAs and can contain larger huge pages than exist for user space). A generic PTDUMP implementation is the implemented making use of the new functionality of walk_page_range() and finally arm64 and x86 are switch to using it, removing the custom table walkers.
To enable a generic page table walker to walk the unusual mappings of the kernel we need to implement a set of functions which let us know when the walker has reached the leaf entry. After a suggestion from Will Deacon I've chosen the name p?d_leaf() as this (hopefully) describes the purpose (and is a new name so has no historic baggage). Some architectures have p?d_large macros but this is easily confused with "large pages".
This series ends with a generic PTDUMP implemention for arm64 and x86.
Mostly this is a clean up and there should be very little functional change. The exceptions are:
* arm64 PTDUMP debugfs now displays pages which aren't present (patch 22).
* arm64 has the ability to efficiently process KASAN pages (which previously only x86 implemented). This means that the combination of KASAN and DEBUG_WX is now useable.
Also available as a git tree:git://linux-arm.org/linux-sp.git walk_page_range/v17
Changes since v16:
https://lore.kernel.org/lkml/[email protected]/
* Introduced the ACTION_{SUBTREE,CONTINUE,AGAIN} concept suggested by Thomas Hellström
* Reworked the KASAN support to avoid the need for test_p?d() callbacks by using ACTION_CONTINUE instead
* powerpc: Use p?d_is_leaf rather than adding new functions
* Simplify x86 changes by leaving ptdump_walk_pgd_level_core() taking a pgd
Changes since v15:
https://lore.kernel.org/lkml/[email protected]/
* Rebased onto Linus' tree, which includes the conflicting commit:
ace88f1018b8 ("mm: pagewalk: Take the pagetable lock in walk_pte_range()")
* New patch fixing conflict with above patch
* Squashed in fix for ordering of "static const"
* Squashed in fix checking p*d_present()
* New patch fixing termination condition for walk_pte_range()
Changes since v14:
https://lore.kernel.org/lkml/[email protected]/
* Switch walk_page_range() into two functions, the existing walk_page_range() now still requires VMAs (and treats areas without a VMA as a 'hole'). The new walk_page_range_novma() ignores VMAs and will report the actual page table layout. This fixes the previous breakage of /proc/<pid>/pagemap
* New patch at the end of the series which reduces the 'level' numbers by 1 to simplify the code slightly
* Added tags
Changes since v13:
https://lore.kernel.org/lkml/[email protected]/
* Fixed typo in arc definition of pmd_leaf() spotted by the kbuild test robot
* Added tags
Changes since v12:
https://lore.kernel.org/lkml/[email protected]/
* Correct code format in riscv pud_leaf()/pmd_leaf()
* v12 may not have reached everyone because of mail server problems (which are now hopefully resolved!)
Changes since v11:
https://lore.kernel.org/lkml/[email protected]/
* Use "-1" as dummy depth parameter in patch 14.
Changes since v10:
https://lore.kernel.org/lkml/[email protected]/
* Rebased to v5.4-rc1 - mainly various updates to deal with the splitting out of ops from struct mm_walk.
* Deal with PGD_LEVEL_MULT not always being constant on x86.
Changes since v9:
https://lore.kernel.org/lkml/[email protected]/
* Moved generic macros to first page in the series and explained the macro naming in the commit message.
* mips: Moved macros to pgtable.h as they are now valid for both 32 and 64 bit
* x86: Dropped patch which changed the debugfs output for x86, instead we have...
* new patch adding 'depth' parameter to pte_hole. This is used to provide the necessary information to output lines for 'holes' in the debugfs files
* new patch changing arm64 debugfs output to include holes to match x86
* generic ptdump KASAN handling has been simplified and now works with CONFIG_DEBUG_VIRTUAL.
Changes since v8:
https://lore.kernel.org/lkml/[email protected]/
* Rename from p?d_large() to p?d_leaf()
* Dropped patches migrating arm64/x86 custom walkers to walk_page_range() in favour of adding a generic PTDUMP implementation and migrating arm64/x86 to that instead.
* Rebased to v5.3-rc1
Steven Price (23):
mm: Add generic p?d_leaf() macros
arc: mm: Add p?d_leaf() definitions
arm: mm: Add p?d_leaf() definitions
arm64: mm: Add p?d_leaf() definitions
mips: mm: Add p?d_leaf() definitions
powerpc: mm: Add p?d_leaf() definitions
riscv: mm: Add p?d_leaf() definitions
s390: mm: Add p?d_leaf() definitions
sparc: mm: Add p?d_leaf() definitions
x86: mm: Add p?d_leaf() definitions
mm: pagewalk: Add p4d_entry() and pgd_entry()
mm: pagewalk: Allow walking without vma
mm: pagewalk: Don't lock PTEs for walk_page_range_novma()
mm: pagewalk: fix termination condition in walk_pte_range()
mm: pagewalk: Add 'depth' parameter to pte_hole
x86: mm: Point to struct seq_file from struct pg_state
x86: mm+efi: Convert ptdump_walk_pgd_level() to take a mm_struct
x86: mm: Convert ptdump_walk_pgd_level_debugfs() to take an mm_struct
mm: Add generic ptdump
x86: mm: Convert dump_pagetables to use walk_page_range
arm64: mm: Convert mm/dump.c to use walk_page_range()
arm64: mm: Display non-present entries in ptdump
mm: ptdump: Reduce level numbers by 1 in note_page()
arch/arc/include/asm/pgtable.h | 1 + arch/arm/include/asm/pgtable-2level.h | 1 + arch/arm/include/asm/pgtable-3level.h | 1 + arch/arm64/Kconfig | 1 + arch/arm64/Kconfig.debug | 19 +- arch/arm64/include/asm/pgtable.h | 2 + arch/arm64/include/asm/ptdump.h | 8 +- arch/arm64/mm/Makefile | 4 +- arch/arm64/mm/dump.c | 148 ++++----- arch/arm64/mm/mmu.c | 4 +- arch/arm64/mm/ptdump_debugfs.c | 2 +- arch/mips/include/asm/pgtable.h | 5 + arch/powerpc/include/asm/book3s/64/pgtable.h | 3 + arch/riscv/include/asm/pgtable-64.h | 7 + arch/riscv/include/asm/pgtable.h | 7 + arch/s390/include/asm/pgtable.h | 2 + arch/sparc/include/asm/pgtable_64.h | 2 + arch/x86/Kconfig | 1 + arch/x86/Kconfig.debug | 20 +- arch/x86/include/asm/pgtable.h | 10 +- arch/x86/mm/Makefile | 4 +- arch/x86/mm/debug_pagetables.c | 8 +- arch/x86/mm/dump_pagetables.c | 320 +++++-------------- arch/x86/platform/efi/efi_32.c | 2 +- arch/x86/platform/efi/efi_64.c | 4 +- drivers/firmware/efi/arm-runtime.c | 2 +- fs/proc/task_mmu.c | 4 +- include/asm-generic/pgtable.h | 20 ++ include/linux/pagewalk.h | 46 ++- include/linux/ptdump.h | 22 ++ mm/Kconfig.debug | 21 ++ mm/Makefile | 1 + mm/hmm.c | 59 ++-- mm/migrate.c | 5 +- mm/mincore.c | 1 + mm/pagewalk.c | 156 ++++++--- mm/ptdump.c | 139 ++++++++ 37 files changed, 596 insertions(+), 466 deletions(-) create mode 100644 include/linux/ptdump.h create mode 100644 mm/ptdump.c
Signature · 1 line
2.20.1
PATCH v17 1/23 mm: Add generic p?d_leaf() macros
Exposing the pud/pgd levels of the page tables to walk_page_range() means we may come across the exotic large mappings that come with large areas of contiguous memory (such as the kernel's linear map).
For architectures that don't provide all p?d_leaf() macros, provide generic do nothing default that are suitable where there cannot be leaf pages at that level. Futher patches will add implementations for individual architectures.
The name p?d_leaf() is chosen to minimize the confusion with existing uses of "large" pages and "huge" pages which do not necessary mean that the entry is a leaf (for example it may be a set of contiguous entries that only take 1 TLB slot). For the purpose of walking the page tables we don't need to know how it will be represented in the TLB, but we do need to know for sure if it is a leaf of the tree.
--- include/asm-generic/pgtable.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
include/asm-generic/pgtable.h +20 -0
@@ -1238,4 +1238,24 @@ static inline bool arch_has_pfn_modify_check(void)#define mm_pmd_folded (mm) __is_defined(__PAGETABLE_PMD_FOLDED)#endif +/* +* p?d_leaf() - true if this entry is a final mapping to a physical address. +* This differs from p?d_huge() by the fact that they are always available (if +* the architecture supports large pages at the appropriate level) even +* if CONFIG_HUGETLB_PAGE is not defined. +* Only meaningful when called on a valid entry. +*/ +#ifndef pgd_leaf +#define pgd_leaf (x) 0 +#endif +#ifndef p4d_leaf +#define p4d_leaf (x) 0 +#endif +#ifndef pud_leaf +#define pud_leaf (x) 0 +#endif +#ifndef pmd_leaf +#define pmd_leaf (x) 0 +#endif +#endif /* _ASM_GENERIC_PGTABLE_H */
Signature · 1 line
2.20.1
Michael Ellerman
reviewerRe: [PATCH v17 01/23] mm: Add generic p?d_leaf() macros
Quoted 39 lines · Steven Price
Exposing the pud/pgd levels of the page tables to
walk_page_range()means we may come across the exotic large mappings that come with large areas of contiguous memory (such as the kernel's linear map).For architectures that don't provide all p?d_leaf() macros, provide generic do nothing default that are suitable where there cannot be leaf pages at that level. Futher patches will add implementations for individual architectures.
The name p?d_leaf() is chosen to minimize the confusion with existing uses of "large" pages and "huge" pages which do not necessary mean that the entry is a leaf (for example it may be a set of contiguous entries that only take 1 TLB slot). For the purpose of walking the page tables we don't need to know how it will be represented in the TLB, but we do need to know for sure if it is a leaf of the tree.
Signed-off-by Steven Price <[email protected]>Acked-by Mark Rutland <[email protected]>--- include/asm-generic/pgtable.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)include/asm-generic/pgtable.h +9
-0@@ -1238,4 +1238,24 @@ static inline bool arch_has_pfn_modify_check(void)#define mm_pmd_folded (mm) __is_defined(__PAGETABLE_PMD_FOLDED)#endif +/* +* p?d_leaf() - true if this entry is a final mapping to a physical address. +* This differs from p?d_huge() by the fact that they are always available (if +* the architecture supports large pages at the appropriate level) even +* if CONFIG_HUGETLB_PAGE is not defined. +* Only meaningful when called on a valid entry. +*/ +#ifndef pgd_leaf +#define pgd_leaf (x) 0
include/asm-generic/pgtable.h static inline bool arch_has_pfn_modify_check(void) · line 1259
Quoted 10 lines
include/asm-generic/pgtable.h +10
-0+#endif +#ifndef p4d_leaf +#define p4d_leaf (x) 0 +#endif +#ifndef pud_leaf +#define pud_leaf (x) 0 +#endif +#ifndef pmd_leaf +#define pmd_leaf (x) 0 +#endif
Any reason you made these #defines rather than static inlines?
cheers
Re: [PATCH v17 01/23] mm: Add generic p?d_leaf() macros
Quoted 44 lines · Michael Ellerman
Quoted 43 lines · Steven Price
Exposing the pud/pgd levels of the page tables to
walk_page_range()means we may come across the exotic large mappings that come with large areas of contiguous memory (such as the kernel's linear map).For architectures that don't provide all p?d_leaf() macros, provide generic do nothing default that are suitable where there cannot be leaf pages at that level. Futher patches will add implementations for individual architectures.
The name p?d_leaf() is chosen to minimize the confusion with existing uses of "large" pages and "huge" pages which do not necessary mean that the entry is a leaf (for example it may be a set of contiguous entries that only take 1 TLB slot). For the purpose of walking the page tables we don't need to know how it will be represented in the TLB, but we do need to know for sure if it is a leaf of the tree.
Signed-off-by Steven Price <[email protected]>Acked-by Mark Rutland <[email protected]>--- include/asm-generic/pgtable.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)include/asm-generic/pgtable.h +13
-0@@ -1238,4 +1238,24 @@ static inline bool arch_has_pfn_modify_check(void)#define mm_pmd_folded (mm) __is_defined(__PAGETABLE_PMD_FOLDED)#endif +/* +* p?d_leaf() - true if this entry is a final mapping to a physical address. +* This differs from p?d_huge() by the fact that they are always available (if +* the architecture supports large pages at the appropriate level) even +* if CONFIG_HUGETLB_PAGE is not defined. +* Only meaningful when called on a valid entry. +*/ +#ifndef pgd_leaf +#define pgd_leaf (x) 0 +#endif +#ifndef p4d_leaf +#define p4d_leaf (x) 0 +#endif
Quoted 8 lines
include/asm-generic/pgtable.h static inline bool arch_has_pfn_modify_check(void) · line 1259
Quoted 6 lines
include/asm-generic/pgtable.h +6
-0+#ifndef pud_leaf +#define pud_leaf (x) 0 +#endif +#ifndef pmd_leaf +#define pmd_leaf (x) 0 +#endif Any reason you made these #defines rather than static inlines?
No strong reason - but these have to be #defines in the arch overrides so the #ifndef works, so I was being consistent here. I guess a static inline might avoid warnings although I haven't seen any.
Steve
Michael Ellerman
reviewerRe: [PATCH v17 01/23] mm: Add generic p?d_leaf() macros
Quoted 54 lines · Steven Price
Quoted 44 lines · Michael Ellerman
Quoted 43 lines · Steven Price
Exposing the pud/pgd levels of the page tables to
walk_page_range()means we may come across the exotic large mappings that come with large areas of contiguous memory (such as the kernel's linear map).For architectures that don't provide all p?d_leaf() macros, provide generic do nothing default that are suitable where there cannot be leaf pages at that level. Futher patches will add implementations for individual architectures.
The name p?d_leaf() is chosen to minimize the confusion with existing uses of "large" pages and "huge" pages which do not necessary mean that the entry is a leaf (for example it may be a set of contiguous entries that only take 1 TLB slot). For the purpose of walking the page tables we don't need to know how it will be represented in the TLB, but we do need to know for sure if it is a leaf of the tree.
Signed-off-by Steven Price <[email protected]>Acked-by Mark Rutland <[email protected]>--- include/asm-generic/pgtable.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)include/asm-generic/pgtable.h +13
-0@@ -1238,4 +1238,24 @@ static inline bool arch_has_pfn_modify_check(void)#define mm_pmd_folded (mm) __is_defined(__PAGETABLE_PMD_FOLDED)#endif +/* +* p?d_leaf() - true if this entry is a final mapping to a physical address. +* This differs from p?d_huge() by the fact that they are always available (if +* the architecture supports large pages at the appropriate level) even +* if CONFIG_HUGETLB_PAGE is not defined. +* Only meaningful when called on a valid entry. +*/ +#ifndef pgd_leaf +#define pgd_leaf (x) 0 +#endif +#ifndef p4d_leaf +#define p4d_leaf (x) 0 +#endif Quoted 8 lines
include/asm-generic/pgtable.h static inline bool arch_has_pfn_modify_check(void) · line 1259
Quoted 6 lines
include/asm-generic/pgtable.h +6
-0+#ifndef pud_leaf +#define pud_leaf (x) 0 +#endif +#ifndef pmd_leaf +#define pmd_leaf (x) 0 +#endif Any reason you made these #defines rather than static inlines?
Quoted 2 lines
No strong reason - but these have to be #defines in the arch overrides so the #ifndef works, so I was being consistent here.
We handle that usually just with eg:
static inline bool pgd_leaf(pgd_t pgd)
{
...
}#define pgd_leaf pgd_leaf
Quoted 2 lines
I guess a static inline might avoid warnings although I haven't seen any.
If anything I'd expect it to cause warnings, for example if someone is doing pgd_leaf(pmd), but that would be good to catch.
cheers
PATCH v17 2/23 arc: mm: Add p?d_leaf() definitions
In reply to Steven Pricewalk_page_range() is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information will be provided by the p?d_leaf() functions/macros.
For arc, we only have two levels, so only pmd_leaf() is needed.
--- arch/arc/include/asm/pgtable.h | 1 + 1 file changed, 1 insertion(+)
arch/arc/include/asm/pgtable.h +1 -0
@@ -273,6 +273,7 @@ static inline void pmd_set(pmd_t *pmdp, pte_t *ptep)#define pmd_none (x) (!pmd_val(x))#define pmd_bad (x) ((pmd_val(x) & ~PAGE_MASK))#define pmd_present (x) (pmd_val(x)) +#define pmd_leaf (x) (pmd_val(x) & _PAGE_HW_SZ)#define pmd_clear (xp) do { pmd_val(*(xp)) = 0; } while (0)#define pte_page (pte) pfn_to_page(pte_pfn(pte))
Signature · 1 line
2.20.1
PATCH v17 3/23 arm: mm: Add p?d_leaf() definitions
In reply to Steven Pricewalk_page_range() is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information is provided by the p?d_leaf() functions/macros.
For arm pmd_large() already exists and does what we want. So simply provide the generic pmd_leaf() name.
--- arch/arm/include/asm/pgtable-2level.h | 1 + arch/arm/include/asm/pgtable-3level.h | 1 + 2 files changed, 2 insertions(+)
arch/arm/include/asm/pgtable-2level.h +1 -0
@@ -189,6 +189,7 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr) }#define pmd_large (pmd) (pmd_val(pmd) & 2) +#define pmd_leaf (pmd) (pmd_val(pmd) & 2)#define pmd_bad (pmd) (pmd_val(pmd) & 2)#define pmd_present (pmd) (pmd_val(pmd))
arch/arm/include/asm/pgtable-3level.h +1 -0
@@ -134,6 +134,7 @@#define pmd_sect (pmd) ((pmd_val(pmd) & PMD_TYPE_MASK) == \ PMD_TYPE_SECT)#define pmd_large (pmd) pmd_sect(pmd) +#define pmd_leaf (pmd) pmd_sect(pmd)#define pud_clear (pudp) \ do { \
Signature · 1 line
2.20.1
PATCH v17 4/23 arm64: mm: Add p?d_leaf() definitions
In reply to Steven Pricewalk_page_range() is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information will be provided by the p?d_leaf() functions/macros.
For arm64, we already have p?d_sect() macros which we can reuse for p?d_leaf().
pud_sect() is defined as a dummy function when CONFIG_PGTABLE_LEVELS < 3 or CONFIG_ARM64_64K_PAGES is defined. However when the kernel is configured this way then architecturally it isn't allowed to have a large page at this level, and any code using these page walking macros is implicitly relying on the page size/number of levels being the same as the kernel. So it is safe to reuse this for p?d_leaf() as it is an architectural restriction.
--- arch/arm64/include/asm/pgtable.h | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/include/asm/pgtable.h +2 -0
@@ -445,6 +445,7 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, PMD_TYPE_TABLE)#define pmd_sect (pmd) ((pmd_val(pmd) & PMD_TYPE_MASK) == \ PMD_TYPE_SECT) +#define pmd_leaf (pmd) pmd_sect(pmd)#if defined(CONFIG_ARM64_64K_PAGES ) ||CONFIG_PGTABLE_LEVELS <3 static inline bool pud_sect (pud_t pud) {return false; }
@@ -529,6 +530,7 @@ static inline void pte_unmap(pte_t *pte) { }#define pud_none (pud) (!pud_val(pud))#define pud_bad (pud) (!(pud_val(pud) & PUD_TABLE_BIT))#define pud_present (pud) pte_present(pud_pte(pud)) +#define pud_leaf (pud) pud_sect(pud)#define pud_valid (pud) pte_valid(pud_pte(pud))static inline void set_pud (pud_t *pudp,pud_t pud)
Signature · 1 line
2.20.1
PATCH v17 5/23 mips: mm: Add p?d_leaf() definitions
In reply to Steven Pricewalk_page_range() is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information is provided by the p?d_leaf() functions/macros.
If _PAGE_HUGE is defined we can simply look for it. When not defined we can be confident that there are no leaf pages in existence and fall back on the generic implementation (added in a later patch) which returns 0.
--- arch/mips/include/asm/pgtable.h | 5 +++++ 1 file changed, 5 insertions(+)
arch/mips/include/asm/pgtable.h +5 -0
@@ -639,6 +639,11 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#ifdef _PAGE_HUGE +#define pmd_leaf (pmd) ((pmd_val(pmd) & _PAGE_HUGE) != 0) +#define pud_leaf (pud) ((pud_val(pud) & _PAGE_HUGE) != 0) +#endif +#define gup_fast_permitted (start, end) (!cpu_has_dc_aliases)#include <asm-generic/pgtable.h>
Signature · 1 line
2.20.1
PATCH v17 6/23 powerpc: mm: Add p?d_leaf() definitions
In reply to Steven Pricewalk_page_range() is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information is provided by the p?d_leaf() functions/macros.
For powerpc p?d_is_leaf() functions already exist. Export them using the new p?d_leaf() name.
--- arch/powerpc/include/asm/book3s/64/pgtable.h | 3 +++ 1 file changed, 3 insertions(+)
arch/powerpc/include/asm/book3s/64/pgtable.h +3 -0
@@ -1355,18 +1355,21 @@ static inline bool is_pte_rw_upgrade(unsigned long old_val, unsigned long new_va* Like pmd_huge() and pmd_large(), but works regardless of config options */ #define pmd_is_leaf pmd_is_leaf +#define pmd_leaf pmd_is_leafstatic inline bool pmd_is_leaf (pmd_t pmd) {return !!(pmd_raw (pmd) &cpu_to_be64 (_PAGE_PTE)); }#define pud_is_leaf pud_is_leaf +#define pud_leaf pud_is_leafstatic inline bool pud_is_leaf (pud_t pud) {return !!(pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)); }#define pgd_is_leaf pgd_is_leaf +#define pgd_leaf pgd_is_leafstatic inline bool pgd_is_leaf (pgd_t pgd) {return !!(pgd_raw (pgd) &cpu_to_be64 (_PAGE_PTE));
Signature · 1 line
2.20.1
Re: [PATCH v17 06/23] powerpc: mm: Add p?d_leaf() definitions
Quoted 14 lines · Steven Price
walk_page_range()is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information is provided by the p?d_leaf() functions/macros.For powerpc p?d_is_leaf() functions already exist. Export them using the new p?d_leaf() name.
CC Benjamin Herrenschmidt <[email protected]>CC Paul Mackerras <[email protected]>CC Michael Ellerman <[email protected]>Signed-off-by Steven Price <[email protected]>
Quoted 3 lines
--- arch/powerpc/include/asm/book3s/64/pgtable.h | 3 +++ 1 file changed, 3 insertions(+)
We have fallback versions of our pmd_is_leaf() etc. in arch/powerpc/include/asm/pgtable.h, eg:
#ifndef pmd_is_leaf
#define pmd_is_leaf pmd_is_leaf
static inline bool pmd_is_leaf(pmd_t pmd)
{
return false;
}#endif
Because we support several different MMUs and most of them don't need to do anything.
So we could put the compatibility #defines to your names along with the fallback versions in asm/pgtable.h, rather than in asm/book3s/64/pgtable.h
But I see you also have fallbacks for your versions, so it probably doesn't matter either way.
So I'm OK with this version, unless you think there's a compelling reason to do the compatibility #defines in our asm/pgtable.h
cheers
Quoted 28 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +3
-0@@ -1355,18 +1355,21 @@ static inline bool is_pte_rw_upgrade(unsigned long old_val, unsigned long new_va* Like pmd_huge() and pmd_large(), but works regardless of config options */ #define pmd_is_leaf pmd_is_leaf +#define pmd_leaf pmd_is_leafstatic inline bool pmd_is_leaf (pmd_t pmd) {return !!(pmd_raw (pmd) &cpu_to_be64 (_PAGE_PTE)); }#define pud_is_leaf pud_is_leaf +#define pud_leaf pud_is_leafstatic inline bool pud_is_leaf (pud_t pud) {return !!(pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)); }#define pgd_is_leaf pgd_is_leaf +#define pgd_leaf pgd_is_leafstatic inline bool pgd_is_leaf (pgd_t pgd) {return !!(pgd_raw (pgd) &cpu_to_be64 (_PAGE_PTE));Signature · 1 line
2.20.1
Re: [PATCH v17 06/23] powerpc: mm: Add p?d_leaf() definitions
Quoted 37 lines · Michael Ellerman
Quoted 14 lines · Steven Price
walk_page_range()is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information is provided by the p?d_leaf() functions/macros.For powerpc p?d_is_leaf() functions already exist. Export them using the new p?d_leaf() name.
CC Benjamin Herrenschmidt <[email protected]>CC Paul Mackerras <[email protected]>CC Michael Ellerman <[email protected]>Signed-off-by Steven Price <[email protected]>Quoted 3 lines
--- arch/powerpc/include/asm/book3s/64/pgtable.h | 3 +++ 1 file changed, 3 insertions(+)We have fallback versions of our
pmd_is_leaf()etc. inarch/powerpc/include/asm/pgtable.h, eg:#ifndef
pmd_is_leaf
#definepmd_is_leafpmd_is_leaf
static inline boolpmd_is_leaf(pmd_t pmd){ return false; }#endif
Because we support several different MMUs and most of them don't need to do anything.
So we could put the compatibility #defines to your names along with the fallback versions in
asm/pgtable.h, rather than inasm/book3s/64/pgtable.h
Quoted 5 lines
But I see you also have fallbacks for your versions, so it probably doesn't matter either way.
So I'm OK with this version, unless you think there's a compelling reason to do the compatibility #defines in our
asm/pgtable.h
I was thinking that (assuming this series actually gets merged this time), the p?d_is_leaf() versions could be removed and replaced by p?d_leaf() in the future. Since the fallbacks are in the asm-generic code it makes sense for the pmd_leaf() definitions to be next to the non-fallback versions.
Quoted 1 line
Acked-by Michael Ellerman <[email protected]>
Thanks!
Steve
Quoted 37 lines
cheers
Quoted 28 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +3
-0@@ -1355,18 +1355,21 @@ static inline bool is_pte_rw_upgrade(unsigned long old_val, unsigned long new_va* Like pmd_huge() and pmd_large(), but works regardless of config options */ #define pmd_is_leaf pmd_is_leaf +#define pmd_leaf pmd_is_leafstatic inline bool pmd_is_leaf (pmd_t pmd) {return !!(pmd_raw (pmd) &cpu_to_be64 (_PAGE_PTE)); }#define pud_is_leaf pud_is_leaf +#define pud_leaf pud_is_leafstatic inline bool pud_is_leaf (pud_t pud) {return !!(pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)); }#define pgd_is_leaf pgd_is_leaf +#define pgd_leaf pgd_is_leafstatic inline bool pgd_is_leaf (pgd_t pgd) {return !!(pgd_raw (pgd) &cpu_to_be64 (_PAGE_PTE));Signature · 1 line
2.20.1_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v17 7/23 riscv: mm: Add p?d_leaf() definitions
In reply to Steven Pricewalk_page_range() is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information is provided by the p?d_leaf() functions/macros.
For riscv a page is a leaf page when it has a read, write or execute bit set on it.
--- arch/riscv/include/asm/pgtable-64.h | 7 +++++++ arch/riscv/include/asm/pgtable.h | 7 +++++++ 2 files changed, 14 insertions(+)
arch/riscv/include/asm/pgtable-64.h +7 -0
@@ -43,6 +43,13 @@ static inline int pud_bad(pud_t pud)return !pud_present (pud); } +#define pud_leaf pud_leaf +static inline int pud_leaf (pud_t pud) +{ +return pud_present (pud) && + (pud_val (pud) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)); +} +static inline void set_pud (pud_t *pudp,pud_t pud) { *pudp = pud;
arch/riscv/include/asm/pgtable.h +7 -0
@@ -105,6 +105,13 @@ static inline int pmd_bad(pmd_t pmd)return !pmd_present (pmd); } +#define pmd_leaf pmd_leaf +static inline int pmd_leaf (pmd_t pmd) +{ +return pmd_present (pmd) && + (pmd_val (pmd) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)); +} +static inline void set_pmd (pmd_t *pmdp,pmd_t pmd) { *pmdp = pmd;
Signature · 1 line
2.20.1
PATCH v17 8/23 s390: mm: Add p?d_leaf() definitions
In reply to Steven Pricewalk_page_range() is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information is provided by the p?d_leaf() functions/macros.
For s390, pud_large() and pmd_large() are already implemented as static inline functions. Add a macro to provide the p?d_leaf names for the generic code to use.
--- arch/s390/include/asm/pgtable.h | 2 ++ 1 file changed, 2 insertions(+)
arch/s390/include/asm/pgtable.h +2 -0
@@ -673,6 +673,7 @@ static inline int pud_none(pud_t pud)return pud_val (pud) == _REGION3_ENTRY_EMPTY; } +#define pud_leaf pud_largestatic inline int pud_large (pud_t pud) {if ((pud_val (pud) & _REGION_ENTRY_TYPE_MASK) != _REGION_ENTRY_TYPE_R3)
@@ -690,6 +691,7 @@ static inline unsigned long pud_pfn(pud_t pud)return (pud_val (pud) & origin_mask) >>PAGE_SHIFT ; } +#define pmd_leaf pmd_largestatic inline int pmd_large (pmd_t pmd) {return (pmd_val (pmd) & _SEGMENT_ENTRY_LARGE) !=0 ;
Signature · 1 line
2.20.1
PATCH v17 9/23 sparc: mm: Add p?d_leaf() definitions
In reply to Steven Pricewalk_page_range() is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information is provided by the p?d_leaf() functions/macros.
For sparc 64 bit, pmd_large() and pud_large() are already provided, so add macros to provide the p?d_leaf names required by the generic code.
--- arch/sparc/include/asm/pgtable_64.h | 2 ++ 1 file changed, 2 insertions(+)
arch/sparc/include/asm/pgtable_64.h +2 -0
@@ -683,6 +683,7 @@ static inline unsigned long pte_special(pte_t pte)return pte_val (pte) & _PAGE_SPECIAL; } +#define pmd_leaf pmd_largestatic inline unsigned long pmd_large (pmd_t pmd) {pte_t pte =__pte (pmd_val (pmd));
@@ -867,6 +868,7 @@ static inline unsigned long pud_page_vaddr(pud_t pud)/* only used by the stubbed out hugetlb gup code, should never be called */ #define pgd_page (pgd) NULL +#define pud_leaf pud_largestatic inline unsigned long pud_large (pud_t pud) {pte_t pte =__pte (pud_val (pud));
Signature · 1 line
2.20.1
PATCH v17 10/23 x86: mm: Add p?d_leaf() definitions
In reply to Steven Pricewalk_page_range() is going to be allowed to walk page tables other than those of user space. For this it needs to know when it has reached a 'leaf' entry in the page tables. This information is provided by the p?d_leaf() functions/macros.
For x86 we already have p?d_large() functions, so simply add macros to provide the generic p?d_leaf() names for the generic code.
--- arch/x86/include/asm/pgtable.h | 5 +++++ 1 file changed, 5 insertions(+)
arch/x86/include/asm/pgtable.h +5 -0
@@ -239,6 +239,7 @@ static inline unsigned long pgd_pfn(pgd_t pgd)return (pgd_val (pgd) &PTE_PFN_MASK ) >>PAGE_SHIFT ; } +#define p4d_leaf p4d_largestatic inline int p4d_large (p4d_t p4d) {/* No 512 GiB pages yet */
@@ -247,6 +248,7 @@ static inline int p4d_large(p4d_t p4d)#define pte_page (pte) pfn_to_page(pte_pfn(pte)) +#define pmd_leaf pmd_largestatic inline int pmd_large (pmd_t pte) {return pmd_flags (pte) & _PAGE_PSE;
@@ -874,6 +876,7 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address)return (pmd_t *)pud_page_vaddr (*pud) +pmd_index (address); } +#define pud_leaf pud_largestatic inline int pud_large (pud_t pud) { return (pud_val(pud) & (_PAGE_PSE | _PAGE_PRESENT)) ==
@@ -885,6 +888,7 @@ static inline int pud_bad(pud_t pud) return (pud_flags(pud) & ~(_KERNPG_TABLE | _PAGE_USER)) != 0; }#else +#define pud_leaf pud_largestatic inline int pud_large (pud_t pud) {return 0 ;
@@ -1233,6 +1237,7 @@ static inline bool pgdp_maps_userspace(void *__ptr)return (((ptr & ~PAGE_MASK ) /sizeof (pgd_t)) <PGD_KERNEL_START ); } +#define pgd_leaf pgd_largestatic inline int pgd_large (pgd_t pgd) {return 0 ; }#ifdef CONFIG_PAGE_TABLE_ISOLATION
Signature · 1 line
2.20.1
PATCH v17 11/23 mm: pagewalk: Add p4d_entry() and pgd_entry()
In reply to Steven Pricepgd_entry() and pud_entry() were removed by commit 0b1fbfe50006c410 ("mm/pagewalk: remove pgd_entry() and pud_entry()") because there were no users. We're about to add users so reintroduce them, along with p4d_entry() as we now have 5 levels of tables.
Note that commit a00cc7d9dd93d66a ("mm, x86: add support for PUD-sized transparent hugepages") already re-added pud_entry() but with different semantics to the other callbacks. This commit reverts the semantics back to match the other callbacks.
To support hmm.c which now uses the new semantics of pud_entry() a new member ('action') of struct mm_walk is added which allows the callbacks to either descend (ACTION_SUBTREE, the default), skip (ACTION_CONTINUE) or repeat the callback (ACTION_AGAIN). hmm.c is then updated to call pud_trans_huge_lock() itself and make use of the splitting/retry logic of the core code.
After this change pud_entry() is called for all entries, not just transparent huge pages.
--- include/linux/pagewalk.h | 34 ++++++++++++++++++++----- mm/hmm.c | 55 ++++++++++++++++++++++------------------ mm/pagewalk.c | 50 +++++++++++++++++++++++++----------- 3 files changed, 94 insertions(+), 45 deletions(-)
include/linux/pagewalk.h +28 -6
@@ -8,15 +8,15 @@ struct mm_walk;/** * mm_walk_ops - callbacks for walk_page_range -* @pud_entry: if set, called for each non-empty PUD (2nd-level) entry -* this handler should only handle pud_trans_huge() puds. -* the pmd_entry or pte_entry callbacks will be used for -* regular PUDs. -* @pmd_entry: if set, called for each non-empty PMD (3rd-level) entry +* @pgd_entry: if set, called for each non-empty PGD (top-level) entry +* @p4d_entry: if set, called for each non-empty P4D entry +* @pud_entry: if set, called for each non-empty PUD entry +* @pmd_entry: if set, called for each non-empty PMD entry * this handler is required to be able to handle * pmd_trans_huge() pmds. They may simply choose to * split_huge_page() instead of handling it explicitly. -* @pte_entry: if set, called for each non-empty PTE (4th-level) entry +* @pte_entry: if set, called for each non-empty PTE (lowest-level) +* entry * @pte_hole: if set, called for each hole at all levels * @hugetlb_entry: if set, called for each hugetlb entry * @test_walk: caller specific callback function to determine whether
@@ -27,8 +27,15 @@ struct mm_walk;* @pre_vma: if set, called before starting walk on a non-null vma. * @post_vma: if set, called after a walk on a non-null vma, provided * that @pre_vma and the vma walk succeeded. +* +* p?d_entry callbacks are called even if those levels are folded on a +* particular architecture/configuration. */ struct mm_walk_ops { +int (*pgd_entry)(pgd_t *pgd,unsigned long addr, +unsigned long next,struct mm_walk *walk); +int (*p4d_entry)(p4d_t *p4d,unsigned long addr, +unsigned long next,struct mm_walk *walk);int (*pud_entry)(pud_t *pud,unsigned long addr,unsigned long next,struct mm_walk *walk); int (*pmd_entry)(pmd_t *pmd, unsigned long addr,
@@ -47,11 +54,25 @@ struct mm_walk_ops {void (*post_vma)(struct mm_walk *walk); }; +/* +* Action for pud_entry / pmd_entry callbacks. +* ACTION_SUBTREE is the default +*/ +enum page_walk_action { +/* Descend to next level, splitting huge pages if needed and possible */ +ACTION_SUBTREE =0 , +/* Continue to next entry at this level (ignoring any subtree) */ +ACTION_CONTINUE =1 , +/* Call again for this entry */ +ACTION_AGAIN =2 +}; +/** * mm_walk - walk_page_range data * @ops: operation to call during the walk * @mm: mm_struct representing the target process of page table walk * @vma: vma currently walked (NULL if walking outside vmas) +* @action: next action to perform (see enum page_walk_action) * @private: private data for callbacks' usage * * (see the comment on walk_page_range() for more details)
@@ -60,6 +81,7 @@ struct mm_walk {const struct mm_walk_ops *ops;struct mm_struct *mm;struct vm_area_struct *vma; +enum page_walk_action action;void *private; };
mm/hmm.c +31 -24
89 lines not on this page · raw message
mm/pagewalk.c +35 -15
95 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v17 11/23] mm: pagewalk: Add p4d_entry() and pgd_entry()
Hi, Steven,
Quoted 17 lines · Steven Price
pgd_entry()andpud_entry()were removed by commit0b1fbfe50006c410("mm/pagewalk: removepgd_entry()andpud_entry()") because there were no users. We're about to add users so reintroduce them, along withp4d_entry()as we now have 5 levels of tables.Note that commit
a00cc7d9dd93d66a("mm, x86: add support for PUD-sized transparent hugepages") already re-addedpud_entry()but with different semantics to the other callbacks. This commit reverts the semantics back to match the other callbacks.To support
hmm.cwhich now uses the new semantics ofpud_entry()a new member ('action') of structmm_walkis added which allows the callbacks to either descend (ACTION_SUBTREE, the default), skip (ACTION_CONTINUE) or repeat the callback (ACTION_AGAIN).hmm.cis then updated to callpud_trans_huge_lock()itself and make use of the splitting/retry logic of the core code.
Quoted 4 lines
After this change
pud_entry()is called for all entries, not just transparent huge pages.Signed-off-by Steven Price <[email protected]>
I have a couple of comments that are actually mostly related to pre-existing bugs, but that affect code that are touched by this patch.
Perhaps this is most cleanly addressed by a follow-up patch. Up to you to decide.
Since these problems are pre-existing,
Thanks,
Thomas
Quoted 87 lines
--- include/linux/pagewalk.h | 34 ++++++++++++++++++++----- mm/hmm.c | 55 ++++++++++++++++++++++------------------ mm/pagewalk.c | 50 +++++++++++++++++++++++++----------- 3 files changed, 94 insertions(+), 45 deletions(-)include/linux/pagewalk.h +28
-6@@ -8,15 +8,15 @@ struct mm_walk;/** * mm_walk_ops - callbacks for walk_page_range -* @pud_entry: if set, called for each non-empty PUD (2nd-level) entry -* this handler should only handle pud_trans_huge() puds. -* the pmd_entry or pte_entry callbacks will be used for -* regular PUDs. -* @pmd_entry: if set, called for each non-empty PMD (3rd-level) entry +* @pgd_entry: if set, called for each non-empty PGD (top-level) entry +* @p4d_entry: if set, called for each non-empty P4D entry +* @pud_entry: if set, called for each non-empty PUD entry +* @pmd_entry: if set, called for each non-empty PMD entry * this handler is required to be able to handle * pmd_trans_huge() pmds. They may simply choose to * split_huge_page() instead of handling it explicitly. -* @pte_entry: if set, called for each non-empty PTE (4th-level) entry +* @pte_entry: if set, called for each non-empty PTE (lowest-level) +* entry * @pte_hole: if set, called for each hole at all levels * @hugetlb_entry: if set, called for each hugetlb entry * @test_walk: caller specific callback function to determine whether @@ -27,8 +27,15 @@ struct mm_walk;* @pre_vma: if set, called before starting walk on a non-null vma. * @post_vma: if set, called after a walk on a non-null vma, provided * that @pre_vma and the vma walk succeeded. +* +* p?d_entry callbacks are called even if those levels are folded on a +* particular architecture/configuration. */ struct mm_walk_ops { +int (*pgd_entry)(pgd_t *pgd,unsigned long addr, +unsigned long next,struct mm_walk *walk); +int (*p4d_entry)(p4d_t *p4d,unsigned long addr, +unsigned long next,struct mm_walk *walk);int (*pud_entry)(pud_t *pud,unsigned long addr,unsigned long next,struct mm_walk *walk); int (*pmd_entry)(pmd_t *pmd, unsigned long addr,@@ -47,11 +54,25 @@ struct mm_walk_ops {void (*post_vma)(struct mm_walk *walk); }; +/* +* Action for pud_entry / pmd_entry callbacks. +* ACTION_SUBTREE is the default +*/ +enum page_walk_action { +/* Descend to next level, splitting huge pages if needed and possible */ +ACTION_SUBTREE =0 , +/* Continue to next entry at this level (ignoring any subtree) */ +ACTION_CONTINUE =1 , +/* Call again for this entry */ +ACTION_AGAIN =2 +}; +/** * mm_walk - walk_page_range data * @ops: operation to call during the walk * @mm: mm_struct representing the target process of page table walk * @vma: vma currently walked (NULL if walking outside vmas) +* @action: next action to perform (see enum page_walk_action) * @private: private data for callbacks' usage * * (see the comment on walk_page_range() for more details) @@ -60,6 +81,7 @@ struct mm_walk {const struct mm_walk_ops *ops;struct mm_struct *mm;struct vm_area_struct *vma; +enum page_walk_action action;void *private; };mm/hmm.c +0
-0
mm/hmm.c static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, · line 484
Quoted 9 lines
mm/hmm.c +5
-1@@ -477,20 +477,30 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end,unsigned long addr = start, next;pmd_t *pmdp;pud_t pud; -int ret; +int ret =0 ; +spinlock_t *ptl =pud_trans_huge_lock (pudp, walk->vma); + +if (!ptl) +return 0 ;
Since if we didn't get the PTL here, the pud might be unstable in which case we want to retry:
if (!ptl) {
if (pud_trans_unstable(pudp))
walk->action = ACTION_AGAIN;
return 0;
}mm/hmm.c static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, · line 489
Quoted 6 lines
mm/hmm.c +3
-1+ +/* Normally we don't want to split the huge page */ + walk->action =ACTION_CONTINUE ; -again: pud =READ_ONCE (*pudp);
We have the lock now, so READ_ONCE could probably be a simple dereference.
mm/hmm.c static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, · line 493
Quoted 6 lines
mm/hmm.c +4
-2-if (pud_none (pud)) -return hmm_vma_walk_hole (start, end, walk); +if (pud_none (pud)) { + ret =hmm_vma_walk_hole (start, end, walk); + goto out_unlock; + }
pud_trans_huge_lock() successful return means pud_none() is always false.
mm/hmm.c static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, · line 503
Quoted 12 lines
mm/hmm.c +4
-2if (pud_huge (pud) &&pud_devmap (pud)) {unsigned long i, npages, pfn;uint64_t *pfns, cpu_flags;bool fault, write_fault; -if (!pud_present (pud)) -return hmm_vma_walk_hole (start, end, walk); +if (!pud_present (pud)) { + ret =hmm_vma_walk_hole (start, end, walk); + goto out_unlock; + }
Can't see !pud_present happening either after a succesful pud_trans_huge_lock().
Quoted 115 lines
mm/hmm.c +15
-18i = (addr - range->start) >>PAGE_SHIFT ; npages = (end - addr) >>PAGE_SHIFT ;@@ -499,16 +509,20 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, cpu_flags =pud_to_hmm_pfn_flags (range, pud);hmm_range_need_fault (hmm_vma_walk, pfns, npages, cpu_flags, &fault, &write_fault); -if (fault || write_fault) -return hmm_vma_walk_hole_ (addr, end, fault, - write_fault, walk); +if (fault || write_fault) { + ret =hmm_vma_walk_hole_ (addr, end, fault, + write_fault, walk); + goto out_unlock; + } pfn =pud_pfn (pud) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT );for (i =0 ; i < npages; ++i, ++pfn) { hmm_vma_walk->pgmap =get_dev_pagemap (pfn, hmm_vma_walk->pgmap); -if (unlikely (!hmm_vma_walk->pgmap)) -return -EBUSY ; +if (unlikely (!hmm_vma_walk->pgmap)) { + ret = -EBUSY ; + goto out_unlock; + } pfns[i] =hmm_device_entry_from_pfn (range, pfn) | cpu_flags; }@@ -517,22 +531,15 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, hmm_vma_walk->pgmap = NULL; } hmm_vma_walk->last = end; -return 0 ; + goto out_unlock; } -split_huge_pud (walk->vma, pudp, addr); -if (pud_none (*pudp)) - goto again; +/* Ask for the PUD to be split */ + walk->action =ACTION_SUBTREE ; - pmdp =pmd_offset (pudp, addr); -do { - next =pmd_addr_end (addr, end); - ret =hmm_vma_walk_pmd (pmdp, addr, next, walk); -if (ret) -return ret; - }while (pmdp++, addr = next, addr != end); - -return 0 ; +out_unlock: +spin_unlock (ptl); +return ret; }#else #define hmm_vma_walk_pud NULLmm/pagewalk.c +14
-11@@ -46,6 +46,9 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,break ;continue ; } + + walk->action =ACTION_SUBTREE ; +/* * This implies that each ->pmd_entry() handler * needs to know about pmd_trans_huge() pmds @@ -55,16 +58,21 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,if (err)break ; +if (walk->action ==ACTION_AGAIN ) + goto again; +/* * Check this here so we only break down trans_huge * pages when we _need_ to */ -if (!ops->pte_entry) +if (walk->action ==ACTION_CONTINUE || + !(ops->pte_entry))continue ;split_huge_pmd (walk->vma, pmd, addr);if (pmd_trans_unstable (pmd)) goto again; + err =walk_pte_range (pmd, addr, next, walk);if (err)break ;@@ -93,24 +101,25 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,continue ; } -if (ops->pud_entry) { -spinlock_t *ptl =pud_trans_huge_lock (pud, walk->vma); + walk->action =ACTION_SUBTREE ; -if (ptl) { - err = ops->pud_entry (pud, addr, next, walk); -spin_unlock (ptl); -if (err) -break ; -continue ; - } - } +if (ops->pud_entry) + err = ops->pud_entry (pud, addr, next, walk); +if (err) +break ;
mm/pagewalk.c static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, · line 119
Quoted 10 lines
mm/pagewalk.c +7
-0+ +if (walk->action ==ACTION_AGAIN ) + goto again; + +if (walk->action ==ACTION_CONTINUE || + !(ops->pmd_entry || ops->pte_entry)) +continue ;split_huge_pud (walk->vma, pud, addr);if (pud_none (*pud))
Here we should really have
if (pud_trans_unstable(pud))
Thanks,
Thomas
Re: [PATCH v17 11/23] mm: pagewalk: Add p4d_entry() and pgd_entry()
Quoted 29 lines · Thomas Hellström (VMware)
Hi, Steven,
Quoted 17 lines · Steven Price
pgd_entry()andpud_entry()were removed by commit0b1fbfe50006c410("mm/pagewalk: removepgd_entry()andpud_entry()") because there were no users. We're about to add users so reintroduce them, along withp4d_entry()as we now have 5 levels of tables.Note that commit
a00cc7d9dd93d66a("mm, x86: add support for PUD-sized transparent hugepages") already re-addedpud_entry()but with different semantics to the other callbacks. This commit reverts the semantics back to match the other callbacks.To support
hmm.cwhich now uses the new semantics ofpud_entry()a new member ('action') of structmm_walkis added which allows the callbacks to either descend (ACTION_SUBTREE, the default), skip (ACTION_CONTINUE) or repeat the callback (ACTION_AGAIN).hmm.cis then updated to callpud_trans_huge_lock()itself and make use of the splitting/retry logic of the core code.Quoted 4 lines
After this change
pud_entry()is called for all entries, not just transparent huge pages.Signed-off-by Steven Price <[email protected]>I have a couple of comments that are actually mostly related to pre-existing bugs, but that affect code that are touched by this patch.
Quoted 6 lines
Perhaps this is most cleanly addressed by a follow-up patch. Up to you to decide.
Since these problems are pre-existing,
Reviewed-by Thomas Hellstrom <[email protected]>
Thanks, I'll send out an extra patch with these changes.
Steve
Quoted 288 lines
Thanks,
Thomas
Quoted 94 lines
---
include/linux/pagewalk.h| 34 ++++++++++++++++++++----- mm/hmm.c | 55 ++++++++++++++++++++++------------------ mm/pagewalk.c | 50 +++++++++++++++++++++++++-----------3 files changed, 94 insertions(+), 45 deletions(-)include/linux/pagewalk.h +0
-1@@ -8,15 +8,15 @@ struct mm_walk; /*** mm_walk_ops - callbacks for walk_page_range -* @pud_entry: if set, called for each non-empty PUD (2nd-level) entry
- * this handler should only handlepud_trans_huge()puds.
- * thepmd_entryorpte_entrycallbacks will be used for
- * regular PUDs.
- * @pmd_entry: if set, called for each non-empty PMD (3rd-level) entry
+ * @pgd_entry: if set, called for each non-empty PGD (top-level) entry
+ * @p4d_entry: if set, called for each non-empty P4D entry
+ * @pud_entry: if set, called for each non-empty PUD entry
+ * @pmd_entry: if set, called for each non-empty PMD entry * this handler is required to be able to handle *pmd_trans_huge()pmds. They may simply choose to *split_huge_page()instead of handling it explicitly.
- * @pte_entry: if set, called for each non-empty PTE (4th-level) entry
+ * @pte_entry: if set, called for each non-empty PTE (lowest-level)
+ * entry
* @pte_hole: if set, called for each hole at all levels
* @hugetlb_entry: if set, called for each hugetlb entry
* @test_walk: caller specific callback function to determine whetherinclude/linux/pagewalk.h +0
-0@@ -27,8 +27,15 @@ struct mm_walk; * @pre_vma: if set, called before starting walk on anon-null vma.
* @post_vma: if set, called after a walk on a non-null vma, provided
* that @pre_vma and the vma walk succeeded.
+ *
+ * p?d_entry callbacks are called even if those levels are folded on a
+ * particular architecture/configuration.
*/
structmm_walk_ops{
+ int (*pgd_entry)(pgd_t *pgd, unsigned long addr,
+ unsigned long next, structmm_walk*walk);
+ int (*p4d_entry)(p4d_t *p4d, unsigned long addr,
+ unsigned long next, structmm_walk*walk);
int (*pud_entry)(pud_t *pud, unsigned long addr, unsigned long next, structmm_walk*walk); int (*pmd_entry)(pmd_t *pmd, unsigned long addr,include/linux/pagewalk.h +0
-0@@ -47,11 +54,25 @@ struct mm_walk_ops { void (*post_vma)(struct mm_walk *walk); };+/*
+ * Action forpud_entry/pmd_entrycallbacks.
+ *ACTION_SUBTREEis the default
+ */
+enumpage_walk_action{
+ /* Descend to next level, splitting huge pages if needed and possible */
+ACTION_SUBTREE= 0,
+ /* Continue to next entry at this level (ignoring any subtree) */ +ACTION_CONTINUE= 1,
+ /* Call again for this entry */
+ACTION_AGAIN= 2include/linux/pagewalk.h +2
-0+}; +/**
*mm_walk-walk_page_rangedata
* @ops: operation to call during the walk
* @mm:mm_structrepresenting the target process of page table walk
* @vma: vma currently walked (NULL if walking outside vmas)
+ * @action: next action to perform (see enumpage_walk_action)
* @private: private data for callbacks' usage
*
* (see the comment onwalk_page_range()for more details)include/linux/pagewalk.h +1
-0@@ -60,6 +81,7 @@ struct mm_walk { const struct mm_walk_ops *ops; struct mm_struct *mm; struct vm_area_struct *vma; + enum page_walk_action action; void *private; };diff --git
a/mm/hmm.cb/mm/hmm.c
indexd379cb6496ae..05241c82e05c100644mm/hmm.c static int hmm_vma_walk_pud(pud_t *pudp, · line 485
Quoted 10 lines
mm/hmm.c +5
-1@@ -477,20 +477,30 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, unsigned long addr = start, next; pmd_t *pmdp; pud_t pud; - int ret; + int ret = 0; + spinlock_t *ptl = pud_trans_huge_lock(pudp, walk->vma); + + if (!ptl) + return 0;Since if we didn't get the PTL here, the pud might be unstable in which case we want to retry:
if (!ptl) {
if (pud_trans_unstable(pudp))
walk->action=ACTION_AGAIN;
return 0;
}Quoted 5 lines
mm/hmm.c +3
-0+ + /* Normally we don't want to split the huge page */ + walk->action = ACTION_CONTINUE;-again:
pud =READ_ONCE(*pudp);We have the lock now, so
READ_ONCEcould probably be a simple dereference.Quoted 6 lines
- if (
pud_none(pud))
- returnhmm_vma_walk_hole(start, end, walk); + if (pud_none(pud)) {
+ ret =hmm_vma_walk_hole(start, end, walk); + gotoout_unlock;
+ }
pud_trans_huge_lock()successful return meanspud_none()is always false.Quoted 10 lines
if (
pud_huge(pud)&&pud_devmap(pud)) { unsigned long i, npages, pfn;uint64_t*pfns,cpu_flags;
bool fault,write_fault;
- if (!pud_present(pud))
- returnhmm_vma_walk_hole(start, end, walk); + if (!pud_present(pud)) {
+ ret =hmm_vma_walk_hole(start, end, walk); + gotoout_unlock;
+ }Can't see !pud_present happening either after a succesful
pud_trans_huge_lock().Quoted 111 lines
i = (addr -
range->start) >>PAGE_SHIFT; npages = (end - addr) >>PAGE_SHIFT;mm/hmm.c +9
-5@@ -499,16 +509,20 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, cpu_flags = pud_to_hmm_pfn_flags(range, pud); hmm_range_need_fault(hmm_vma_walk, pfns, npages, cpu_flags, &fault, &write_fault); - if (fault || write_fault) - return hmm_vma_walk_hole_(addr, end, fault, - write_fault, walk); + if (fault || write_fault) { + ret = hmm_vma_walk_hole_(addr, end, fault, + write_fault, walk); + goto out_unlock; + } pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); for (i = 0; i < npages; ++i, ++pfn) { hmm_vma_walk->pgmap = get_dev_pagemap(pfn, hmm_vma_walk->pgmap); - if (unlikely(!hmm_vma_walk->pgmap)) - return -EBUSY; + if (unlikely(!hmm_vma_walk->pgmap)) { + ret = -EBUSY; + goto out_unlock; + }pfns[i] =
hmm_device_entry_from_pfn(range, pfn)|cpu_flags;
}mm/hmm.c +6
-11@@ -517,22 +531,15 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, hmm_vma_walk->pgmap = NULL; } hmm_vma_walk->last = end; - return 0; + goto out_unlock; } - split_huge_pud(walk->vma, pudp, addr); - if (pud_none(*pudp)) - goto again; + /* Ask for the PUD to be split */ + walk->action = ACTION_SUBTREE; - pmdp = pmd_offset(pudp, addr); - do { - next = pmd_addr_end(addr, end); - ret = hmm_vma_walk_pmd(pmdp, addr, next, walk); - if (ret) - return ret; - } while (pmdp++, addr = next, addr != end); - - return 0; +out_unlock: + spin_unlock(ptl); + return ret; } #else#define hmm_vma_walk_pud NULL
mm/pagewalk.c +3
-0@@ -46,6 +46,9 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, break; continue; } + + walk->action = ACTION_SUBTREE; + /** This implies that each ->pmd_entry() handler * needs to know about
pmd_trans_huge()pmdsmm/pagewalk.c +9
-9@@ -55,16 +58,21 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, if (err) break; + if (walk->action == ACTION_AGAIN) + goto again; + /* * Check this here so we only break down trans_huge * pages when we _need_ to */ - if (!ops->pte_entry) + if (walk->action == ACTION_CONTINUE || + !(ops->pte_entry)) continue; split_huge_pmd(walk->vma, pmd, addr); if (pmd_trans_unstable(pmd)) goto again; + err = walk_pte_range(pmd, addr, next, walk); if (err) break;@@ -93,24 +101,25 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, continue; } - if (ops->pud_entry) { - spinlock_t *ptl = pud_trans_huge_lock(pud, walk->vma); + walk->action = ACTION_SUBTREE; - if (ptl) { - err = ops->pud_entry(pud, addr, next, walk); - spin_unlock(ptl); - if (err) - break; - continue; - } - } + if (ops->pud_entry) + err = ops->pud_entry(pud, addr, next, walk); + if (err)mm/pagewalk.c static int walk_pud_range(p4d_t *p4d, unsigned · line 119
Quoted 10 lines
mm/pagewalk.c +8
-0+ break; + + if (walk->action == ACTION_AGAIN) + goto again; + + if (walk->action == ACTION_CONTINUE || + !(ops->pmd_entry || ops->pte_entry)) + continue; split_huge_pud(walk->vma, pud, addr); if (pud_none(*pud))Here we should really have
if (
pud_trans_unstable(pud))Thanks,
Thomas
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v17 12/23 mm: pagewalk: Allow walking without vma
In reply to Steven PriceSince 48684a65b4e3: "mm: pagewalk: fix misbehavior of walk_page_range for vma(VM_PFNMAP)", page_table_walk() will report any kernel area as a hole, because it lacks a vma.
This means each arch has re-implemented page table walking when needed, for example in the per-arch ptdump walker.
Remove the requirement to have a vma in the generic code and add a new function walk_page_range_novma() which ignores the VMAs and simply walks the page tables.
--- include/linux/pagewalk.h | 5 +++++ mm/pagewalk.c | 40 ++++++++++++++++++++++++++++++++-------- 2 files changed, 37 insertions(+), 8 deletions(-)
include/linux/pagewalk.h +5 -0
@@ -73,6 +73,7 @@ enum page_walk_action {* @mm: mm_struct representing the target process of page table walk * @vma: vma currently walked (NULL if walking outside vmas) * @action: next action to perform (see enum page_walk_action) +* @no_vma: walk ignoring vmas (vma will always be NULL) * @private: private data for callbacks' usage * * (see the comment on walk_page_range() for more details)
@@ -82,12 +83,16 @@ struct mm_walk {struct mm_struct *mm;struct vm_area_struct *vma;enum page_walk_action action; +bool no_vma;void *private; };int walk_page_range (struct mm_struct *mm,unsigned long start,unsigned long end,const struct mm_walk_ops *ops,void *private); +int walk_page_range_novma (struct mm_struct *mm,unsigned long start, +unsigned long end,const struct mm_walk_ops *ops, +void *private);int walk_page_vma (struct vm_area_struct *vma,const struct mm_walk_ops *ops,void *private); int walk_page_mapping(struct address_space *mapping, pgoff_t first_index,
mm/pagewalk.c +32 -8
@@ -39,7 +39,7 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, do { again: next = pmd_addr_end(addr, end); - if (pmd_none(*pmd) || !walk->vma) { + if (pmd_none(*pmd) || (!walk->vma && !walk->no_vma)) { if (ops->pte_hole) err = ops->pte_hole(addr, next, walk); if (err)
@@ -65,13 +65,16 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,* Check this here so we only break down trans_huge * pages when we _need_ to */ -if (walk->action ==ACTION_CONTINUE || +if ((!walk->vma && (pmd_leaf (*pmd) || !pmd_present (*pmd))) || + walk->action ==ACTION_CONTINUE || !(ops->pte_entry))continue ; -split_huge_pmd (walk->vma, pmd, addr); -if (pmd_trans_unstable (pmd)) - goto again; +if (walk->vma) { +split_huge_pmd (walk->vma, pmd, addr); +if (pmd_trans_unstable (pmd)) + goto again; + } err =walk_pte_range (pmd, addr, next, walk);if (err)
@@ -93,7 +96,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, do { again: next = pud_addr_end(addr, end); - if (pud_none(*pud) || !walk->vma) { + if (pud_none(*pud) || (!walk->vma && !walk->no_vma)) { if (ops->pte_hole) err = ops->pte_hole(addr, next, walk); if (err)
@@ -111,11 +114,13 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,if (walk->action ==ACTION_AGAIN ) goto again; -if (walk->action ==ACTION_CONTINUE || +if ((!walk->vma && (pud_leaf (*pud) || !pud_present (*pud))) || + walk->action ==ACTION_CONTINUE || !(ops->pmd_entry || ops->pte_entry))continue ; -split_huge_pud (walk->vma, pud, addr); +if (walk->vma) +split_huge_pud (walk->vma, pud, addr);if (pud_none (*pud)) goto again;
@@ -389,6 +394,25 @@ int walk_page_range(struct mm_struct *mm, unsigned long start,return err; } +int walk_page_range_novma (struct mm_struct *mm,unsigned long start, +unsigned long end,const struct mm_walk_ops *ops, +void *private) +{ +struct mm_walk walk = { + .ops = ops, + .mm = mm, + .private = private, + .no_vma = true + }; + +if (start >= end || !walk.mm) +return -EINVAL ; + +lockdep_assert_held (&walk.mm->mmap_sem); + +return __walk_page_range (start, end, &walk); +} +int walk_page_vma (struct vm_area_struct *vma,const struct mm_walk_ops *ops,void *private) {
Signature · 1 line
2.20.1
PATCH v17 13/23 mm: pagewalk: Don't lock PTEs for walk_page_range_novma()
In reply to Steven Pricewalk_page_range_novma() can be used to walk page tables or the kernel or for firmware. These page tables may contain entries that are not backed by a struct page and so it isn't (in general) possible to take the PTE lock for the pte_entry() callback. So update walk_pte_range() to only take the lock when no_vma==false by splitting out the inner loop to a separate function and add a comment explaining the difference to walk_page_range_novma().
--- mm/pagewalk.c | 35 ++++++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-)
mm/pagewalk.c +28 -7
@@ -4,15 +4,12 @@#include <linux/sched.h> #include <linux/hugetlb.h> -static int walk_pte_range (pmd_t *pmd,unsigned long addr,unsigned long end, -struct mm_walk *walk) +static int walk_pte_range_inner (pte_t *pte,unsigned long addr, +unsigned long end,struct mm_walk *walk) { -pte_t *pte; -int err =0 ;const struct mm_walk_ops *ops = walk->ops; -spinlock_t *ptl; +int err =0 ; - pte =pte_offset_map_lock (walk->mm, pmd, addr, &ptl);for (;;) { err = ops->pte_entry (pte, addr, addr +PAGE_SIZE , walk);if (err)
@@ -22,8 +19,26 @@ static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,break ; pte++; } +return err; +} + +static int walk_pte_range (pmd_t *pmd,unsigned long addr,unsigned long end, +struct mm_walk *walk) +{ +pte_t *pte; +int err =0 ; +spinlock_t *ptl; + +if (walk->no_vma) { + pte =pte_offset_map (pmd, addr); + err =walk_pte_range_inner (pte, addr, end, walk); +pte_unmap (pte); + }else { + pte =pte_offset_map_lock (walk->mm, pmd, addr, &ptl); + err =walk_pte_range_inner (pte, addr, end, walk); +pte_unmap_unlock (pte, ptl); + } -pte_unmap_unlock (pte, ptl);return err; }
@@ -394,6 +409,12 @@ int walk_page_range(struct mm_struct *mm, unsigned long start,return err; } +/* +* Similar to walk_page_range() but can walk any page tables even if they are +* not backed by VMAs. Because 'unusual' entries may be walked this function +* will also not lock the PTEs for the pte_entry() callback. This is useful for +* walking the kernel pages tables or page tables for firmware. +*/ int walk_page_range_novma (struct mm_struct *mm,unsigned long start,unsigned long end,const struct mm_walk_ops *ops,void *private)
Signature · 1 line
2.20.1
PATCH v17 14/23 mm: pagewalk: fix termination condition in walk_pte_range()
In reply to Steven PriceIf walk_pte_range() is called with a 'end' argument that is beyond the last page of memory (e.g. ~0UL) then the comparison between 'addr' and 'end' will always fail and the loop will be infinite. Instead change the comparison to >= while accounting for overflow.
--- mm/pagewalk.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/pagewalk.c +2 -2
@@ -14,9 +14,9 @@ static int walk_pte_range_inner(pte_t *pte, unsigned long addr, err = ops->pte_entry (pte, addr, addr +PAGE_SIZE , walk);if (err)break ; - addr +=PAGE_SIZE ; -if (addr == end) +if (addr >= end -PAGE_SIZE )break ; + addr +=PAGE_SIZE ; pte++; }return err;
Signature · 1 line
2.20.1
PATCH v17 15/23 mm: pagewalk: Add 'depth' parameter to pte_hole
In reply to Steven PriceThe pte_hole() callback is called at multiple levels of the page tables. Code dumping the kernel page tables needs to know what at what depth the missing entry is. Add this is an extra parameter to pte_hole(). When the depth isn't know (e.g. processing a vma) then -1 is passed.
The depth that is reported is the actual level where the entry is missing (ignoring any folding that is in place), i.e. any levels where PTRS_PER_P?D is set to 1 are ignored.
Note that depth starts at 0 for a PGD so that PUD/PMD/PTE retain their natural numbers as levels 2/3/4.
--- fs/proc/task_mmu.c | 4 ++-- include/linux/pagewalk.h | 7 +++++-- mm/hmm.c | 8 ++++---- mm/migrate.c | 5 +++-- mm/mincore.c | 1 + mm/pagewalk.c | 31 +++++++++++++++++++++++++------ 6 files changed, 40 insertions(+), 16 deletions(-)
fs/proc/task_mmu.c +2 -2
@@ -505,7 +505,7 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,#ifdef CONFIG_SHMEM static int smaps_pte_hole (unsigned long addr,unsigned long end, -struct mm_walk *walk) +__always_unused int depth,struct mm_walk *walk) {struct mem_size_stats *mss = walk->private;
@@ -1282,7 +1282,7 @@ static int add_to_pagemap(unsigned long addr, pagemap_entry_t *pme, }static int pagemap_pte_hole (unsigned long start,unsigned long end, -struct mm_walk *walk) +__always_unused int depth,struct mm_walk *walk) {struct pagemapread *pm = walk->private;unsigned long addr = start;
include/linux/pagewalk.h +5 -2
@@ -17,7 +17,10 @@ struct mm_walk;* split_huge_page() instead of handling it explicitly. * @pte_entry: if set, called for each non-empty PTE (lowest-level) * entry -* @pte_hole: if set, called for each hole at all levels +* @pte_hole: if set, called for each hole at all levels, +* depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD +* 4:PTE. Any folded depths (where PTRS_PER_P?D is equal +* to 1) are skipped. * @hugetlb_entry: if set, called for each hugetlb entry * @test_walk: caller specific callback function to determine whether * we walk over the current vma or not. Returning 0 means
@@ -43,7 +46,7 @@ struct mm_walk_ops {int (*pte_entry)(pte_t *pte,unsigned long addr,unsigned long next,struct mm_walk *walk);int (*pte_hole)(unsigned long addr,unsigned long next, -struct mm_walk *walk); +int depth,struct mm_walk *walk);int (*hugetlb_entry)(pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long next,struct mm_walk *walk);
mm/hmm.c +4 -4
@@ -186,7 +186,7 @@ static void hmm_range_need_fault(const struct hmm_vma_walk *hmm_vma_walk, }static int hmm_vma_walk_hole (unsigned long addr,unsigned long end, -struct mm_walk *walk) +__always_unused int depth,struct mm_walk *walk) {struct hmm_vma_walk *hmm_vma_walk = walk->private;struct hmm_range *range = hmm_vma_walk->range;
@@ -380,7 +380,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, again: pmd =READ_ONCE (*pmdp);if (pmd_none (pmd)) -return hmm_vma_walk_hole (start, end, walk); +return hmm_vma_walk_hole (start, end,-1 , walk);if (thp_migration_supported () &&is_pmd_migration_entry (pmd)) {bool fault, write_fault;
@@ -488,7 +488,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, pud =READ_ONCE (*pudp);if (pud_none (pud)) { - ret =hmm_vma_walk_hole (start, end, walk); + ret =hmm_vma_walk_hole (start, end,-1 , walk); goto out_unlock; }
@@ -498,7 +498,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end,bool fault, write_fault;if (!pud_present (pud)) { - ret =hmm_vma_walk_hole (start, end, walk); + ret =hmm_vma_walk_hole (start, end,-1 , walk); goto out_unlock; }
mm/migrate.c +3 -2
@@ -2119,6 +2119,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,#ifdef CONFIG_DEVICE_PRIVATE static int migrate_vma_collect_hole (unsigned long start,unsigned long end, +__always_unused int depth,struct mm_walk *walk) {struct migrate_vma *migrate = walk->private;
@@ -2163,7 +2164,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, again:if (pmd_none (*pmdp)) -return migrate_vma_collect_hole (start, end, walk); +return migrate_vma_collect_hole (start, end,-1 , walk);if (pmd_trans_huge (*pmdp)) {struct page *page;
@@ -2196,7 +2197,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,return migrate_vma_collect_skip (start, end, walk);if (pmd_none (*pmdp)) -return migrate_vma_collect_hole (start, end, +return migrate_vma_collect_hole (start, end,-1 , walk); } }
mm/mincore.c +1 -0
@@ -112,6 +112,7 @@ static int __mincore_unmapped_range(unsigned long addr, unsigned long end, }static int mincore_unmapped_range (unsigned long addr,unsigned long end, +__always_unused int depth,struct mm_walk *walk) { walk->private += __mincore_unmapped_range(addr, end,
mm/pagewalk.c +25 -6
@@ -4,6 +4,22 @@#include <linux/sched.h> #include <linux/hugetlb.h> +/* +* We want to know the real level where a entry is located ignoring any +* folding of levels which may be happening. For example if p4d is folded then +* a missing entry found at level 1 (p4d) is actually at level 0 (pgd). +*/ +static int real_depth (int depth) +{ +if (depth ==3 &&PTRS_PER_PMD ==1 ) + depth =2 ; +if (depth ==2 &&PTRS_PER_PUD ==1 ) + depth =1 ; +if (depth ==1 &&PTRS_PER_P4D ==1 ) + depth =0 ; +return depth; +} +static int walk_pte_range_inner (pte_t *pte,unsigned long addr,unsigned long end,struct mm_walk *walk) {
@@ -49,6 +65,7 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,unsigned long next;const struct mm_walk_ops *ops = walk->ops;int err =0 ; +int depth =real_depth (3 ); pmd =pmd_offset (pud, addr); do {
@@ -56,7 +73,7 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, next =pmd_addr_end (addr, end);if (pmd_none (*pmd) || (!walk->vma && !walk->no_vma)) {if (ops->pte_hole) - err = ops->pte_hole (addr, next, walk); + err = ops->pte_hole (addr, next, depth, walk);if (err)break ;continue ;
@@ -106,6 +123,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,unsigned long next;const struct mm_walk_ops *ops = walk->ops;int err =0 ; +int depth =real_depth (2 ); pud =pud_offset (p4d, addr); do {
@@ -113,7 +131,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, next =pud_addr_end (addr, end);if (pud_none (*pud) || (!walk->vma && !walk->no_vma)) {if (ops->pte_hole) - err = ops->pte_hole (addr, next, walk); + err = ops->pte_hole (addr, next, depth, walk);if (err)break ;continue ;
@@ -154,13 +172,14 @@ static int walk_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,unsigned long next;const struct mm_walk_ops *ops = walk->ops;int err =0 ; +int depth =real_depth (1 ); p4d =p4d_offset (pgd, addr); do { next = p4d_addr_end(addr, end); if (p4d_none_or_clear_bad(p4d)) { if (ops->pte_hole) - err = ops->pte_hole(addr, next, walk); + err = ops->pte_hole(addr, next, depth, walk); if (err) break; continue;
@@ -192,7 +211,7 @@ static int walk_pgd_range(unsigned long addr, unsigned long end, next =pgd_addr_end (addr, end);if (pgd_none_or_clear_bad (pgd)) {if (ops->pte_hole) - err = ops->pte_hole (addr, next, walk); + err = ops->pte_hole (addr, next,0 , walk);if (err)break ;continue ;
@@ -239,7 +258,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end,if (pte) err = ops->hugetlb_entry (pte, hmask, addr, next, walk);else if (ops->pte_hole) - err = ops->pte_hole (addr, next, walk); + err = ops->pte_hole (addr, next,-1 , walk);if (err)break ;
@@ -283,7 +302,7 @@ static int walk_page_test(unsigned long start, unsigned long end,if (vma->vm_flags &VM_PFNMAP ) {int err =1 ;if (ops->pte_hole) - err = ops->pte_hole (start, end, walk); + err = ops->pte_hole (start, end,-1 , walk);return err ? err :1 ; }return 0 ;
Signature · 1 line
2.20.1
PATCH v17 16/23 x86: mm: Point to struct seq_file from struct pg_state
In reply to Steven Pricemm/dump_pagetables.c passes both struct seq_file and struct pg_state down the chain of walk_*_level() functions to be passed to note_page(). Instead place the struct seq_file in struct pg_state and access it from struct pg_state (which is private to this file) in note_page().
--- arch/x86/mm/dump_pagetables.c | 69 ++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 34 deletions(-)
arch/x86/mm/dump_pagetables.c +35 -34
@@ -36,6 +36,7 @@ struct pg_state {bool to_dmesg;bool check_wx;unsigned long wx_pages; +struct seq_file *seq; };struct addr_marker {
@@ -265,11 +266,12 @@ static void note_wx(struct pg_state *st)* of PTE entries; the next one is different so we need to * print what we collected so far. */ -static void note_page (struct seq_file *m,struct pg_state *st, -pgprot_t new_prot,pgprotval_t new_eff,int level) +static void note_page (struct pg_state *st,pgprot_t new_prot, +pgprotval_t new_eff,int level) {pgprotval_t prot, cur, eff;static const char units[] ="BKMGTPE" ; +struct seq_file *m = st->seq;/* * If we have a "break" in the series, we need to flush the state that
@@ -354,8 +356,8 @@ static inline pgprotval_t effective_prot(pgprotval_t prot1, pgprotval_t prot2) ((prot1 | prot2) & _PAGE_NX); } -static void walk_pte_level (struct seq_file *m,struct pg_state *st,pmd_t addr, -pgprotval_t eff_in,unsigned long P ) +static void walk_pte_level (struct pg_state *st,pmd_t addr,pgprotval_t eff_in, +unsigned long P ) {int i;pte_t *pte;
@@ -366,7 +368,7 @@ static void walk_pte_level(struct seq_file *m, struct pg_state *st, pmd_t addr, pte =pte_offset_map (&addr, st->current_address); prot =pte_flags (*pte); eff =effective_prot (eff_in, prot); -note_page (m, st,__pgprot (prot), eff,5 ); +note_page (st,__pgprot (prot), eff,5 );pte_unmap (pte); } }
@@ -379,22 +381,20 @@ static void walk_pte_level(struct seq_file *m, struct pg_state *st, pmd_t addr,* us dozens of seconds (minutes for 5-level config) while checking for * W+X mapping or reading kernel_page_tables debugfs file. */ -static inline bool kasan_page_table (struct seq_file *m,struct pg_state *st, -void *pt) +static inline bool kasan_page_table (struct pg_state *st,void *pt) {if (__pa (pt) ==__pa (kasan_early_shadow_pmd) || (pgtable_l5_enabled () &&__pa (pt) ==__pa (kasan_early_shadow_p4d)) ||__pa (pt) ==__pa (kasan_early_shadow_pud)) {pgprotval_t prot =pte_flags (kasan_early_shadow_pte[0 ]); -note_page (m, st,__pgprot (prot),0 ,5 ); +note_page (st,__pgprot (prot),0 ,5 );return true; }return false; }#else -static inline bool kasan_page_table (struct seq_file *m,struct pg_state *st, -void *pt) +static inline bool kasan_page_table (struct pg_state *st,void *pt) {return false; }
@@ -402,7 +402,7 @@ static inline bool kasan_page_table(struct seq_file *m, struct pg_state *st,#if PTRS_PER_PMD >1 -static void walk_pmd_level (struct seq_file *m,struct pg_state *st,pud_t addr, +static void walk_pmd_level (struct pg_state *st,pud_t addr,pgprotval_t eff_in,unsigned long P ) {int i;
@@ -416,27 +416,27 @@ static void walk_pmd_level(struct seq_file *m, struct pg_state *st, pud_t addr, prot =pmd_flags (*start); eff =effective_prot (eff_in, prot);if (pmd_large (*start) || !pmd_present (*start)) { -note_page (m, st,__pgprot (prot), eff,4 ); - }else if (!kasan_page_table (m, st, pmd_start)) { -walk_pte_level (m, st, *start, eff, +note_page (st,__pgprot (prot), eff,4 ); + }else if (!kasan_page_table (st, pmd_start)) { +walk_pte_level (st, *start, eff,P + i *PMD_LEVEL_MULT ); } }else -note_page (m, st,__pgprot (0 ),0 ,4 ); +note_page (st,__pgprot (0 ),0 ,4 ); start++; } }#else -#define walk_pmd_level (m,s,a,e,p) walk_pte_level(m,s,__pmd(pud_val(a)),e,p) +#define walk_pmd_level (s,a,e,p) walk_pte_level(s,__pmd(pud_val(a)),e,p)#define pud_large (a) pmd_large(__pmd(pud_val(a)))#define pud_none (a) pmd_none(__pmd(pud_val(a)))#endif #if PTRS_PER_PUD >1 -static void walk_pud_level (struct seq_file *m,struct pg_state *st,p4d_t addr, -pgprotval_t eff_in,unsigned long P ) +static void walk_pud_level (struct pg_state *st,p4d_t addr,pgprotval_t eff_in, +unsigned long P ) {int i;pud_t *start, *pud_start;
@@ -450,33 +450,33 @@ static void walk_pud_level(struct seq_file *m, struct pg_state *st, p4d_t addr, prot =pud_flags (*start); eff =effective_prot (eff_in, prot);if (pud_large (*start) || !pud_present (*start)) { -note_page (m, st,__pgprot (prot), eff,3 ); - }else if (!kasan_page_table (m, st, pud_start)) { -walk_pmd_level (m, st, *start, eff, +note_page (st,__pgprot (prot), eff,3 ); + }else if (!kasan_page_table (st, pud_start)) { +walk_pmd_level (st, *start, eff,P + i *PUD_LEVEL_MULT ); } }else -note_page (m, st,__pgprot (0 ),0 ,3 ); +note_page (st,__pgprot (0 ),0 ,3 ); start++; } }#else -#define walk_pud_level (m,s,a,e,p) walk_pmd_level(m,s,__pud(p4d_val(a)),e,p) +#define walk_pud_level (s,a,e,p) walk_pmd_level(s,__pud(p4d_val(a)),e,p)#define p4d_large (a) pud_large(__pud(p4d_val(a)))#define p4d_none (a) pud_none(__pud(p4d_val(a)))#endif -static void walk_p4d_level (struct seq_file *m,struct pg_state *st,pgd_t addr, -pgprotval_t eff_in,unsigned long P ) +static void walk_p4d_level (struct pg_state *st,pgd_t addr,pgprotval_t eff_in, +unsigned long P ) {int i;p4d_t *start, *p4d_start;pgprotval_t prot, eff;if (PTRS_PER_P4D ==1 ) -return walk_pud_level (m, st,__p4d (pgd_val (addr)), eff_in,P ); +return walk_pud_level (st,__p4d (pgd_val (addr)), eff_in,P ); p4d_start = start = (p4d_t *)pgd_page_vaddr (addr);
@@ -486,13 +486,13 @@ static void walk_p4d_level(struct seq_file *m, struct pg_state *st, pgd_t addr, prot =p4d_flags (*start); eff =effective_prot (eff_in, prot);if (p4d_large (*start) || !p4d_present (*start)) { -note_page (m, st,__pgprot (prot), eff,2 ); - }else if (!kasan_page_table (m, st, p4d_start)) { -walk_pud_level (m, st, *start, eff, +note_page (st,__pgprot (prot), eff,2 ); + }else if (!kasan_page_table (st, p4d_start)) { +walk_pud_level (st, *start, eff,P + i *P4D_LEVEL_MULT ); } }else -note_page (m, st,__pgprot (0 ),0 ,2 ); +note_page (st,__pgprot (0 ),0 ,2 ); start++; }
@@ -529,6 +529,7 @@ static void ptdump_walk_pgd_level_core(struct seq_file *m, pgd_t *pgd, } st.check_wx = checkwx; + st.seq = m;if (checkwx) st.wx_pages =0 ;
@@ -542,13 +543,13 @@ static void ptdump_walk_pgd_level_core(struct seq_file *m, pgd_t *pgd, eff = prot;#endif if (pgd_large (*start) || !pgd_present (*start)) { -note_page (m, &st,__pgprot (prot), eff,1 ); +note_page (&st,__pgprot (prot), eff,1 ); }else { -walk_p4d_level (m, &st, *start, eff, +walk_p4d_level (&st, *start, eff, i *PGD_LEVEL_MULT ); } }else -note_page (m, &st,__pgprot (0 ),0 ,1 ); +note_page (&st,__pgprot (0 ),0 ,1 );cond_resched (); start++;
@@ -556,7 +557,7 @@ static void ptdump_walk_pgd_level_core(struct seq_file *m, pgd_t *pgd,/* Flush out the last page */ st.current_address =normalize_addr (PTRS_PER_PGD *PGD_LEVEL_MULT ); -note_page (m, &st,__pgprot (0 ),0 ,0 ); +note_page (&st,__pgprot (0 ),0 ,0 );if (!checkwx)return ;if (st.wx_pages)
Signature · 1 line
2.20.1
PATCH v17 17/23 x86: mm+efi: Convert ptdump_walk_pgd_level() to take a mm_struct
In reply to Steven PriceTo enable x86 to use the generic walk_page_range() function, the callers of ptdump_walk_pgd_level() need to pass an mm_struct rather than the raw pgd_t pointer. Luckily since commit 7e904a91bf60 ("efi: Use efi_mm in x86 as well as ARM") we now have an mm_struct for EFI on x86.
--- arch/x86/include/asm/pgtable.h | 2 +- arch/x86/mm/dump_pagetables.c | 4 ++-- arch/x86/platform/efi/efi_32.c | 2 +- arch/x86/platform/efi/efi_64.c | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -29,7 +29,7 @@extern pgd_t early_top_pgt[PTRS_PER_PGD ];int __init __early_make_pgtable (unsigned long address,pmdval_t pmd); -void ptdump_walk_pgd_level (struct seq_file *m,pgd_t *pgd); +void ptdump_walk_pgd_level (struct seq_file *m,struct mm_struct *mm);void ptdump_walk_pgd_level_debugfs (struct seq_file *m,pgd_t *pgd,bool user);void ptdump_walk_pgd_level_checkwx (void );void ptdump_walk_user_pgd_level_checkwx (void );
arch/x86/mm/dump_pagetables.c +2 -2
@@ -567,9 +567,9 @@ static void ptdump_walk_pgd_level_core(struct seq_file *m, pgd_t *pgd,pr_info ("x86/mm: Checked W+X mappings: passed, no W+X pages found.\n" ); } -void ptdump_walk_pgd_level (struct seq_file *m,pgd_t *pgd) +void ptdump_walk_pgd_level (struct seq_file *m,struct mm_struct *mm) { -ptdump_walk_pgd_level_core (m, pgd, false, true); +ptdump_walk_pgd_level_core (m, mm->pgd, false, true); }void ptdump_walk_pgd_level_debugfs (struct seq_file *m,pgd_t *pgd,bool user)
arch/x86/platform/efi/efi_32.c +1 -1
@@ -49,7 +49,7 @@ void efi_sync_low_kernel_mappings(void) {}void __init efi_dump_pagetable (void ) {#ifdef CONFIG_EFI_PGT_DUMP -ptdump_walk_pgd_level (NULL , swapper_pg_dir); +ptdump_walk_pgd_level (NULL , &init_mm);#endif }
arch/x86/platform/efi/efi_64.c +2 -2
@@ -614,9 +614,9 @@ void __init efi_dump_pagetable(void) {#ifdef CONFIG_EFI_PGT_DUMP if (efi_enabled (EFI_OLD_MEMMAP )) -ptdump_walk_pgd_level (NULL , swapper_pg_dir); +ptdump_walk_pgd_level (NULL , &init_mm);else -ptdump_walk_pgd_level (NULL , efi_mm.pgd); +ptdump_walk_pgd_level (NULL , &efi_mm);#endif }
Signature · 1 line
2.20.1
PATCH v17 18/23 x86: mm: Convert ptdump_walk_pgd_level_debugfs() to take an mm_struct
In reply to Steven PriceTo enable x86 to use the generic walk_page_range() function, the callers of ptdump_walk_pgd_level_debugfs() need to pass in the mm_struct.
This means that ptdump_walk_pgd_level_core() is now always passed a valid pgd, so drop the support for pgd==NULL.
--- arch/x86/include/asm/pgtable.h | 3 ++- arch/x86/mm/debug_pagetables.c | 8 ++++---- arch/x86/mm/dump_pagetables.c | 14 ++++++-------- 3 files changed, 12 insertions(+), 13 deletions(-)
arch/x86/include/asm/pgtable.h +2 -1
@@ -30,7 +30,8 @@ extern pgd_t early_top_pgt[PTRS_PER_PGD];int __init __early_make_pgtable (unsigned long address,pmdval_t pmd);void ptdump_walk_pgd_level (struct seq_file *m,struct mm_struct *mm); -void ptdump_walk_pgd_level_debugfs (struct seq_file *m,pgd_t *pgd,bool user); +void ptdump_walk_pgd_level_debugfs (struct seq_file *m,struct mm_struct *mm, +bool user);void ptdump_walk_pgd_level_checkwx (void );void ptdump_walk_user_pgd_level_checkwx (void );
arch/x86/mm/debug_pagetables.c +4 -4
@@ -7,7 +7,7 @@static int ptdump_show (struct seq_file *m,void *v) { -ptdump_walk_pgd_level_debugfs (m,NULL , false); +ptdump_walk_pgd_level_debugfs (m, &init_mm, false);return 0 ; }
@@ -17,7 +17,7 @@ static int ptdump_curknl_show(struct seq_file *m, void *v) {if (current->mm->pgd) {down_read (¤t->mm->mmap_sem); -ptdump_walk_pgd_level_debugfs (m, current->mm->pgd, false); +ptdump_walk_pgd_level_debugfs (m, current->mm, false);up_read (¤t->mm->mmap_sem); }return 0 ;
@@ -30,7 +30,7 @@ static int ptdump_curusr_show(struct seq_file *m, void *v) {if (current->mm->pgd) {down_read (¤t->mm->mmap_sem); -ptdump_walk_pgd_level_debugfs (m, current->mm->pgd, true); +ptdump_walk_pgd_level_debugfs (m, current->mm, true);up_read (¤t->mm->mmap_sem); }return 0 ;
@@ -43,7 +43,7 @@ DEFINE_SHOW_ATTRIBUTE(ptdump_curusr);static int ptdump_efi_show (struct seq_file *m,void *v) {if (efi_mm.pgd) -ptdump_walk_pgd_level_debugfs (m, efi_mm.pgd, false); +ptdump_walk_pgd_level_debugfs (m, &efi_mm, false);return 0 ; }
arch/x86/mm/dump_pagetables.c +6 -8
@@ -518,16 +518,12 @@ static inline bool is_hypervisor_range(int idx)static void ptdump_walk_pgd_level_core (struct seq_file *m,pgd_t *pgd,bool checkwx,bool dmesg) { -pgd_t *start =INIT_PGD ; +pgd_t *start = pgd;pgprotval_t prot, eff;int i;struct pg_state st = {}; -if (pgd) { - start = pgd; - st.to_dmesg = dmesg; - } - + st.to_dmesg = dmesg; st.check_wx = checkwx; st.seq = m;if (checkwx)
@@ -572,8 +568,10 @@ void ptdump_walk_pgd_level(struct seq_file *m, struct mm_struct *mm)ptdump_walk_pgd_level_core (m, mm->pgd, false, true); } -void ptdump_walk_pgd_level_debugfs (struct seq_file *m,pgd_t *pgd,bool user) +void ptdump_walk_pgd_level_debugfs (struct seq_file *m,struct mm_struct *mm, +bool user) { +pgd_t *pgd = mm->pgd;#ifdef CONFIG_PAGE_TABLE_ISOLATION if (user &&boot_cpu_has (X86_FEATURE_PTI )) pgd =kernel_to_user_pgdp (pgd);
@@ -599,7 +597,7 @@ void ptdump_walk_user_pgd_level_checkwx(void)void ptdump_walk_pgd_level_checkwx (void ) { -ptdump_walk_pgd_level_core (NULL ,NULL , true, false); +ptdump_walk_pgd_level_core (NULL ,INIT_PGD , true, false); }static int __init pt_dump_init (void )
Signature · 1 line
2.20.1
PATCH v17 19/23 mm: Add generic ptdump
In reply to Steven PriceAdd a generic version of page table dumping that architectures can opt-in to
--- include/linux/ptdump.h | 21 +++++++ mm/Kconfig.debug | 21 +++++++ mm/Makefile | 1 + mm/ptdump.c | 139 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 182 insertions(+) create mode 100644 include/linux/ptdump.h create mode 100644 mm/ptdump.c
include/linux/ptdump.h new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _LINUX_PTDUMP_H +#define _LINUX_PTDUMP_H + +#include <linux/mm_types.h> + +struct ptdump_range { +unsigned long start; +unsigned long end; +}; + +struct ptdump_state { +void (*note_page)(struct ptdump_state *st,unsigned long addr, +int level,unsigned long val); +const struct ptdump_range *range; +}; + +void ptdump_walk_pgd (struct ptdump_state *st,struct mm_struct *mm); + +#endif /* _LINUX_PTDUMP_H */
mm/Kconfig.debug +21 -0
@@ -117,3 +117,24 @@ config DEBUG_RODATA_TEST depends on STRICT_KERNEL_RWX ---help--- This option enables a testcase for the setting rodata read-only. + +config GENERIC_PTDUMP + bool + +config PTDUMP_CORE + bool + +config PTDUMP_DEBUGFS + bool "Export kernel pagetable layout to userspace via debugfs" + depends on DEBUG_KERNEL + depends on DEBUG_FS + depends on GENERIC_PTDUMP + select PTDUMP_CORE + help + Say Y here if you want to show the kernel pagetable layout in a + debugfs file. This information is only useful for kernel developers + who are working in architecture specific areas of the kernel. + It is probably not a good idea to enable this feature in a production + kernel. + + If in doubt, say N.
mm/Makefile +1 -0
@@ -108,3 +108,4 @@ obj-$(CONFIG_ZONE_DEVICE) += memremap.o obj-$(CONFIG_HMM_MIRROR) += hmm.o obj-$(CONFIG_MEMFD_CREATE) += memfd.o obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o +obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
mm/ptdump.c new file +139 -0
139 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v17 20/23 x86: mm: Convert dump_pagetables to use walk_page_range
In reply to Steven PriceMake use of the new functionality in walk_page_range to remove the arch page walking code and use the generic code to walk the page tables.
The effective permissions are passed down the chain using new fields in struct pg_state.
The KASAN optimisation is implemented by setting action=CONTINUE in the callbacks to skip an entire tree of entries.
--- arch/x86/Kconfig | 1 + arch/x86/Kconfig.debug | 20 +-- arch/x86/mm/Makefile | 4 +- arch/x86/mm/dump_pagetables.c | 292 ++++++++-------------------------- 4 files changed, 70 insertions(+), 247 deletions(-)
arch/x86/Kconfig +1 -0
@@ -119,6 +119,7 @@ config X86 select GENERIC_IRQ_RESERVATION_MODE select GENERIC_IRQ_SHOW select GENERIC_PENDING_IRQ if SMP + select GENERIC_PTDUMP select GENERIC_SMP_IDLE_THREAD select GENERIC_STRNCPY_FROM_USER select GENERIC_STRNLEN_USER
arch/x86/Kconfig.debug +2 -18
@@ -62,26 +62,10 @@ config EARLY_PRINTK_USB_XDBC config MCSAFE_TEST def_bool n -config X86_PTDUMP_CORE - def_bool n - -config X86_PTDUMP - tristate "Export kernel pagetable layout to userspace via debugfs" - depends on DEBUG_KERNEL - select DEBUG_FS - select X86_PTDUMP_CORE - ---help--- - Say Y here if you want to show the kernel pagetable layout in a - debugfs file. This information is only useful for kernel developers - who are working in architecture specific areas of the kernel. - It is probably not a good idea to enable this feature in a production - kernel. - If in doubt, say "N" - config EFI_PGT_DUMP bool "Dump the EFI pagetable" depends on EFI - select X86_PTDUMP_CORE + select PTDUMP_CORE ---help--- Enable this if you want to dump the EFI page table before enabling virtual mode. This can be used to debug miscellaneous
@@ -90,7 +74,7 @@ config EFI_PGT_DUMP config DEBUG_WX bool "Warn on W+X mappings at boot" - select X86_PTDUMP_CORE + select PTDUMP_CORE ---help--- Generate a warning if any W+X mappings are found at boot.
arch/x86/mm/Makefile +2 -2
@@ -28,8 +28,8 @@ obj-$(CONFIG_X86_PAT) += pat_interval.o obj-$(CONFIG_X86_32) += pgtable_32.o iomap_32.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o -obj-$(CONFIG_X86_PTDUMP_CORE) += dump_pagetables.o -obj-$(CONFIG_X86_PTDUMP) += debug_pagetables.o +obj-$(CONFIG_PTDUMP_CORE) += dump_pagetables.o +obj-$(CONFIG_PTDUMP_DEBUGFS) += debug_pagetables.o obj-$(CONFIG_HIGHMEM) += highmem_32.o
arch/x86/mm/dump_pagetables.c +65 -227
394 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v17 21/23 arm64: mm: Convert mm/dump.c to use walk_page_range()
In reply to Steven PriceNow walk_page_range() can walk kernel page tables, we can switch the arm64 ptdump code over to using it, simplifying the code.
--- arch/arm64/Kconfig | 1 + arch/arm64/Kconfig.debug | 19 +---- arch/arm64/include/asm/ptdump.h | 8 +- arch/arm64/mm/Makefile | 4 +- arch/arm64/mm/dump.c | 117 ++++++++++------------------- arch/arm64/mm/mmu.c | 4 +- arch/arm64/mm/ptdump_debugfs.c | 2 +- drivers/firmware/efi/arm-runtime.c | 2 +- 8 files changed, 50 insertions(+), 107 deletions(-)
arch/arm64/Kconfig +1 -0
@@ -103,6 +103,7 @@ config ARM64 select GENERIC_IRQ_SHOW select GENERIC_IRQ_SHOW_LEVEL select GENERIC_PCI_IOMAP + select GENERIC_PTDUMP select GENERIC_SCHED_CLOCK select GENERIC_SMP_IDLE_THREAD select GENERIC_STRNCPY_FROM_USER
arch/arm64/Kconfig.debug +1 -18
@@ -1,22 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only -config ARM64_PTDUMP_CORE - def_bool n - -config ARM64_PTDUMP_DEBUGFS - bool "Export kernel pagetable layout to userspace via debugfs" - depends on DEBUG_KERNEL - select ARM64_PTDUMP_CORE - select DEBUG_FS - help - Say Y here if you want to show the kernel pagetable layout in a - debugfs file. This information is only useful for kernel developers - who are working in architecture specific areas of the kernel. - It is probably not a good idea to enable this feature in a production - kernel. - - If in doubt, say N. - config PID_IN_CONTEXTIDR bool "Write the current PID to the CONTEXTIDR register" help
@@ -42,7 +25,7 @@ config ARM64_RANDOMIZE_TEXT_OFFSET config DEBUG_WX bool "Warn on W+X mappings at boot" - select ARM64_PTDUMP_CORE + select PTDUMP_CORE ---help--- Generate a warning if any W+X mappings are found at boot.
arch/arm64/include/asm/ptdump.h +4 -4
@@ -5,7 +5,7 @@#ifndef __ASM_PTDUMP_H#define __ASM_PTDUMP_H -#ifdef CONFIG_ARM64_PTDUMP_CORE +#ifdef CONFIG_PTDUMP_CORE #include <linux/mm_types.h> #include <linux/seq_file.h>
@@ -21,15 +21,15 @@ struct ptdump_info { unsigned long base_addr; }; -void ptdump_walk_pgd (struct seq_file *s,struct ptdump_info *info); -#ifdef CONFIG_ARM64_PTDUMP_DEBUGFS +void ptdump_walk (struct seq_file *s,struct ptdump_info *info); +#ifdef CONFIG_PTDUMP_DEBUGFS void ptdump_debugfs_register (struct ptdump_info *info,const char *name);#else static inline void ptdump_debugfs_register (struct ptdump_info *info,const char *name) { }#endif void ptdump_check_wx (void ); -#endif /* CONFIG_ARM64_PTDUMP_CORE */ +#endif /* CONFIG_PTDUMP_CORE */ #ifdef CONFIG_DEBUG_WX #define debug_checkwx () ptdump_check_wx()
arch/arm64/mm/Makefile +2 -2
@@ -4,8 +4,8 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o -obj-$(CONFIG_ARM64_PTDUMP_CORE) += dump.o -obj-$(CONFIG_ARM64_PTDUMP_DEBUGFS) += ptdump_debugfs.o +obj-$(CONFIG_PTDUMP_CORE) += dump.o +obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o obj-$(CONFIG_NUMA) += numa.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o KASAN_SANITIZE_physaddr.o += n
arch/arm64/mm/dump.c +38 -79
166 lines not on this page · raw message
arch/arm64/mm/mmu.c +2 -2
@@ -943,13 +943,13 @@ int __init arch_ioremap_pud_supported(void)* SW table walks can't handle removal of intermediate entries. */ return IS_ENABLED (CONFIG_ARM64_4K_PAGES ) && - !IS_ENABLED (CONFIG_ARM64_PTDUMP_DEBUGFS ); + !IS_ENABLED (CONFIG_PTDUMP_DEBUGFS ); }int __init arch_ioremap_pmd_supported (void ) {/* See arch_ioremap_pud_supported() */ -return !IS_ENABLED (CONFIG_ARM64_PTDUMP_DEBUGFS ); +return !IS_ENABLED (CONFIG_PTDUMP_DEBUGFS ); }int pud_set_huge (pud_t *pudp,phys_addr_t phys,pgprot_t prot)
arch/arm64/mm/ptdump_debugfs.c +1 -1
@@ -7,7 +7,7 @@static int ptdump_show (struct seq_file *m,void *v) {struct ptdump_info *info = m->private; -ptdump_walk_pgd (m, info); +ptdump_walk (m, info);return 0 ; }DEFINE_SHOW_ATTRIBUTE (ptdump);
drivers/firmware/efi/arm-runtime.c +1 -1
@@ -27,7 +27,7 @@extern u64 efi_system_table; -#ifdef CONFIG_ARM64_PTDUMP_DEBUGFS +#if defined(CONFIG_PTDUMP_DEBUGFS ) && defined(CONFIG_ARM64 )#include <asm/ptdump.h> static struct ptdump_info efi_ptdump_info = {
Signature · 1 line
2.20.1
Ard Biesheuvel
maintainerRe: [PATCH v17 21/23] arm64: mm: Convert mm/dump.c to use walk_page_range()
Quoted 6 lines · Steven Price
Now
walk_page_range()can walk kernel page tables, we can switch the arm64 ptdump code over to using it, simplifying the code.Reviewed-by Catalin Marinas <[email protected]>Signed-off-by Steven Price <[email protected]>
I did not realize this at the time, but this patch removes the ability to dump the EFI page tables on 32-bit ARM. Was that intentional?
Quoted 331 lines
--- arch/arm64/Kconfig | 1 + arch/arm64/Kconfig.debug | 19 +---- arch/arm64/include/asm/ptdump.h | 8 +- arch/arm64/mm/Makefile | 4 +- arch/arm64/mm/dump.c | 117 ++++++++++------------------- arch/arm64/mm/mmu.c | 4 +- arch/arm64/mm/ptdump_debugfs.c | 2 +- drivers/firmware/efi/arm-runtime.c | 2 +- 8 files changed, 50 insertions(+), 107 deletions(-)arch/arm64/Kconfig +1
-0@@ -103,6 +103,7 @@ config ARM64 select GENERIC_IRQ_SHOW select GENERIC_IRQ_SHOW_LEVEL select GENERIC_PCI_IOMAP + select GENERIC_PTDUMP select GENERIC_SCHED_CLOCK select GENERIC_SMP_IDLE_THREAD select GENERIC_STRNCPY_FROM_USERarch/arm64/Kconfig.debug +1
-18@@ -1,22 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only -config ARM64_PTDUMP_CORE - def_bool n - -config ARM64_PTDUMP_DEBUGFS - bool "Export kernel pagetable layout to userspace via debugfs" - depends on DEBUG_KERNEL - select ARM64_PTDUMP_CORE - select DEBUG_FS - help - Say Y here if you want to show the kernel pagetable layout in a - debugfs file. This information is only useful for kernel developers - who are working in architecture specific areas of the kernel. - It is probably not a good idea to enable this feature in a production - kernel. - - If in doubt, say N. - config PID_IN_CONTEXTIDR bool "Write the current PID to the CONTEXTIDR register" help@@ -42,7 +25,7 @@ config ARM64_RANDOMIZE_TEXT_OFFSET config DEBUG_WX bool "Warn on W+X mappings at boot" - select ARM64_PTDUMP_CORE + select PTDUMP_CORE ---help--- Generate a warning if any W+X mappings are found at boot.arch/arm64/include/asm/ptdump.h +4
-4@@ -5,7 +5,7 @@#ifndef __ASM_PTDUMP_H#define __ASM_PTDUMP_H -#ifdef CONFIG_ARM64_PTDUMP_CORE +#ifdef CONFIG_PTDUMP_CORE #include <linux/mm_types.h> #include <linux/seq_file.h> @@ -21,15 +21,15 @@ struct ptdump_info { unsigned long base_addr; }; -void ptdump_walk_pgd (struct seq_file *s,struct ptdump_info *info); -#ifdef CONFIG_ARM64_PTDUMP_DEBUGFS +void ptdump_walk (struct seq_file *s,struct ptdump_info *info); +#ifdef CONFIG_PTDUMP_DEBUGFS void ptdump_debugfs_register (struct ptdump_info *info,const char *name);#else static inline void ptdump_debugfs_register (struct ptdump_info *info,const char *name) { }#endif void ptdump_check_wx (void ); -#endif /* CONFIG_ARM64_PTDUMP_CORE */ +#endif /* CONFIG_PTDUMP_CORE */ #ifdef CONFIG_DEBUG_WX #define debug_checkwx () ptdump_check_wx()arch/arm64/mm/Makefile +2
-2@@ -4,8 +4,8 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o -obj-$(CONFIG_ARM64_PTDUMP_CORE) += dump.o -obj-$(CONFIG_ARM64_PTDUMP_DEBUGFS) += ptdump_debugfs.o +obj-$(CONFIG_PTDUMP_CORE) += dump.o +obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o obj-$(CONFIG_NUMA) += numa.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o KASAN_SANITIZE_physaddr.o += narch/arm64/mm/dump.c +38
-79@@ -15,6 +15,7 @@#include <linux/io.h> #include <linux/init.h> #include <linux/mm.h> +#include <linux/ptdump.h> #include <linux/sched.h> #include <linux/seq_file.h> @@ -75,10 +76,11 @@ static struct addr_marker address_markers[] = {* dumps out a description of the range. */ struct pg_state { +struct ptdump_state ptdump;struct seq_file *seq;const struct addr_marker *marker;unsigned long start_address; -unsigned level; +int level;u64 current_prot;bool check_wx;unsigned long wx_pages;@@ -179,6 +181,10 @@ static struct pg_level pg_level[] = { .name = "PGD", .bits = pte_bits, .num = ARRAY_SIZE(pte_bits), + }, { /* p4d */ + .name = "P4D", + .bits = pte_bits, + .num = ARRAY_SIZE(pte_bits), }, { /* pud */ .name = (CONFIG_PGTABLE_LEVELS > 3) ? "PUD" : "PGD", .bits = pte_bits,@@ -241,11 +247,15 @@ static void note_prot_wx(struct pg_state *st, unsigned long addr) st->wx_pages += (addr - st->start_address) /PAGE_SIZE ; } -static void note_page (struct pg_state *st,unsigned long addr,unsigned level, -u64 val) +static void note_page (struct ptdump_state *pt_st,unsigned long addr,int level, +unsigned long val) { + struct pg_state *st = container_of(pt_st, struct pg_state, ptdump);static const char units[] ="KMGTPE" ; -u64 prot = val & pg_level[level].mask; +u64 prot =0 ; + +if (level >=0 ) + prot = val & pg_level[level].mask;if (!st->level) { st->level = level;@@ -293,85 +303,27 @@ static void note_page(struct pg_state *st, unsigned long addr, unsigned level, } -static void walk_pte (struct pg_state *st,pmd_t *pmdp,unsigned long start, -unsigned long end) -{ -unsigned long addr = start; -pte_t *ptep =pte_offset_kernel (pmdp, start); - -do { -note_page (st, addr,4 ,READ_ONCE (pte_val (*ptep))); - }while (ptep++, addr +=PAGE_SIZE , addr != end); -} - -static void walk_pmd (struct pg_state *st,pud_t *pudp,unsigned long start, -unsigned long end) -{ -unsigned long next, addr = start; -pmd_t *pmdp =pmd_offset (pudp, start); - -do { -pmd_t pmd =READ_ONCE (*pmdp); - next =pmd_addr_end (addr, end); - -if (pmd_none (pmd) ||pmd_sect (pmd)) { -note_page (st, addr,3 ,pmd_val (pmd)); - }else { -BUG_ON (pmd_bad (pmd)); -walk_pte (st, pmdp, addr, next); - } - }while (pmdp++, addr = next, addr != end); -} - -static void walk_pud (struct pg_state *st,pgd_t *pgdp,unsigned long start, -unsigned long end) +void ptdump_walk (struct seq_file *s,struct ptdump_info *info) { -unsigned long next, addr = start; -pud_t *pudp =pud_offset (pgdp, start); - -do { -pud_t pud =READ_ONCE (*pudp); - next =pud_addr_end (addr, end); - -if (pud_none (pud) ||pud_sect (pud)) { -note_page (st, addr,2 ,pud_val (pud)); - }else { -BUG_ON (pud_bad (pud)); -walk_pmd (st, pudp, addr, next); - } - }while (pudp++, addr = next, addr != end); -} +unsigned long end = ~0UL ; +struct pg_state st; -static void walk_pgd (struct pg_state *st,struct mm_struct *mm, -unsigned long start) -{ -unsigned long end = (start <TASK_SIZE_64 ) ?TASK_SIZE_64 :0 ; -unsigned long next, addr = start; -pgd_t *pgdp =pgd_offset (mm, start); - -do { -pgd_t pgd =READ_ONCE (*pgdp); - next =pgd_addr_end (addr, end); - -if (pgd_none (pgd)) { -note_page (st, addr,1 ,pgd_val (pgd)); - }else { -BUG_ON (pgd_bad (pgd)); -walk_pud (st, pgdp, addr, next); - } - }while (pgdp++, addr = next, addr != end); -} +if (info->base_addr <TASK_SIZE_64 ) + end =TASK_SIZE_64 ; -void ptdump_walk_pgd (struct seq_file *m,struct ptdump_info *info) -{ -struct pg_state st = { - .seq = m, + st = (struct pg_state ){ + .seq = s, .marker = info->markers, + .ptdump = { + .note_page = note_page, + .range = (struct ptdump_range []){ + {info->base_addr, end}, + {0 ,0 } + } + } }; -walk_pgd (&st, info->mm, info->base_addr); - -note_page (&st,0 ,0 ,0 ); +ptdump_walk_pgd (&st.ptdump, info->mm); }static void ptdump_initialize (void )@@ -399,10 +351,17 @@ void ptdump_check_wx(void) { -1, NULL}, }, .check_wx = true, + .ptdump = { + .note_page = note_page, + .range = (struct ptdump_range[]) { + {PAGE_OFFSET , ~0UL }, + {0 ,0 } + } + } }; -walk_pgd (&st, &init_mm,PAGE_OFFSET ); -note_page (&st,0 ,0 ,0 ); +ptdump_walk_pgd (&st.ptdump, &init_mm); +if (st.wx_pages || st.uxn_pages)pr_warn ("Checked W+X mappings: FAILED, %lu W+X pages found, %lu non-UXN pages found\n" , st.wx_pages, st.uxn_pages);arch/arm64/mm/mmu.c +2
-2@@ -943,13 +943,13 @@ int __init arch_ioremap_pud_supported(void)* SW table walks can't handle removal of intermediate entries. */ return IS_ENABLED (CONFIG_ARM64_4K_PAGES ) && - !IS_ENABLED (CONFIG_ARM64_PTDUMP_DEBUGFS ); + !IS_ENABLED (CONFIG_PTDUMP_DEBUGFS ); }int __init arch_ioremap_pmd_supported (void ) {/* See arch_ioremap_pud_supported() */ -return !IS_ENABLED (CONFIG_ARM64_PTDUMP_DEBUGFS ); +return !IS_ENABLED (CONFIG_PTDUMP_DEBUGFS ); }int pud_set_huge (pud_t *pudp,phys_addr_t phys,pgprot_t prot)arch/arm64/mm/ptdump_debugfs.c +1
-1@@ -7,7 +7,7 @@static int ptdump_show (struct seq_file *m,void *v) {struct ptdump_info *info = m->private; -ptdump_walk_pgd (m, info); +ptdump_walk (m, info);return 0 ; }DEFINE_SHOW_ATTRIBUTE (ptdump);drivers/firmware/efi/arm-runtime.c +1
-2@@ -27,7 +27,7 @@extern u64 efi_system_table; -#ifdef CONFIG_ARM64_PTDUMP_DEBUGFS +#if defined(CONFIG_PTDUMP_DEBUGFS ) && defined(CONFIG_ARM64 )#include <asm/ptdump.h> static struct ptdump_info efi_ptdump_info = { --2.20.1
Re: [PATCH v17 21/23] arm64: mm: Convert mm/dump.c to use walk_page_range()
Quoted 10 lines · Ard Biesheuvel
Quoted 6 lines · Steven Price
Now
walk_page_range()can walk kernel page tables, we can switch the arm64 ptdump code over to using it, simplifying the code.Reviewed-by Catalin Marinas <[email protected]>Signed-off-by Steven Price <[email protected]>I did not realize this at the time, but this patch removes the ability to dump the EFI page tables on 32-bit ARM. Was that intentional?
No that wasn't intentional, but I can't instantly see how this change affects 32-bit ARM.
<snip (files in arch/arm64)>
Quoted 10 lines
drivers/firmware/efi/arm-runtime.c line 30
Quoted 10 lines
drivers/firmware/efi/arm-runtime.c +1
-1@@ -27,7 +27,7 @@extern u64 efi_system_table; -#ifdef CONFIG_ARM64_PTDUMP_DEBUGFS +#if defined(CONFIG_PTDUMP_DEBUGFS ) && defined(CONFIG_ARM64 )
The previous define was *ARM64* so should never have been true when building for arm. The new condition should be equivalent (arm64 && ptdump enabled).
Am I missing something?
Steve
Ard Biesheuvel
maintainerRe: [PATCH v17 21/23] arm64: mm: Convert mm/dump.c to use walk_page_range()
Quoted 25 lines · Steven Price
Quoted 10 lines · Ard Biesheuvel
Quoted 6 lines · Steven Price
Now
walk_page_range()can walk kernel page tables, we can switch the arm64 ptdump code over to using it, simplifying the code.Reviewed-by Catalin Marinas <[email protected]>Signed-off-by Steven Price <[email protected]>I did not realize this at the time, but this patch removes the ability to dump the EFI page tables on 32-bit ARM. Was that intentional?
No that wasn't intentional, but I can't instantly see how this change affects 32-bit ARM.
<snip (files in
arch/arm64)>Quoted 8 lines
Quoted 8 lines
drivers/firmware/efi/arm-runtime.c +0
-0@@ -27,7 +27,7 @@extern u64 efi_system_table;
Quoted 9 lines
Quoted 2 lines
drivers/firmware/efi/arm-runtime.c line 30
Quoted 2 lines
drivers/firmware/efi/arm-runtime.c +1
-1-#ifdef CONFIG_ARM64_PTDUMP_DEBUGFS +#if defined(CONFIG_PTDUMP_DEBUGFS ) && defined(CONFIG_ARM64 )The previous define was *ARM64* so should never have been true when building for arm. The new condition should be equivalent (arm64 && ptdump enabled).
Am I missing something?
Not at all, I just got confused.
IIRC we did have support for dumping the EFI pages tables on 32-bit ARM at *some* point, but it obviously wasn't your patch that removed it.
Apologies for the noise.
PATCH v17 22/23 arm64: mm: Display non-present entries in ptdump
In reply to Steven PricePreviously the /sys/kernel/debug/kernel_page_tables file would only show lines for entries present in the page tables. However it is useful to also show non-present entries as this makes the size and level of the holes more visible. This aligns the behaviour with x86 which also shows holes.
--- arch/arm64/mm/dump.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-)
arch/arm64/mm/dump.c +13 -12
@@ -270,21 +270,22 @@ static void note_page(struct ptdump_state *pt_st, unsigned long addr, int level,if (st->current_prot) {note_prot_uxn (st, addr);note_prot_wx (st, addr); -pt_dump_seq_printf (st->seq,"0x%016lx-0x%016lx " , + } + +pt_dump_seq_printf (st->seq,"0x%016lx-0x%016lx " , st->start_address, addr); - delta = (addr - st->start_address) >>10 ; -while (!(delta &1023 ) && unit[1 ]) { - delta >>=10 ; - unit++; - } -pt_dump_seq_printf (st->seq,"%9lu%c %s" , delta, *unit, - pg_level[st->level].name); -if (pg_level[st->level].bits) -dump_prot (st, pg_level[st->level].bits, - pg_level[st->level].num); -pt_dump_seq_puts (st->seq,"\n" ); + delta = (addr - st->start_address) >>10 ; +while (!(delta &1023 ) && unit[1 ]) { + delta >>=10 ; + unit++; } +pt_dump_seq_printf (st->seq,"%9lu%c %s" , delta, *unit, + pg_level[st->level].name); +if (st->current_prot && pg_level[st->level].bits) +dump_prot (st, pg_level[st->level].bits, + pg_level[st->level].num); +pt_dump_seq_puts (st->seq,"\n" );if (addr >= st->marker[1 ].start_address) { st->marker++;
Signature · 1 line
2.20.1
PATCH v17 23/23 mm: ptdump: Reduce level numbers by 1 in note_page()
In reply to Steven PriceRather than having to increment the 'depth' number by 1 in ptdump_hole(), let's change the meaning of 'level' in note_page() since that makes the code simplier.
Note that for x86, the level numbers were previously increased by 1 in commit 45dcd2091363 ("x86/mm/dump_pagetables: Fix printout of p4d level") and the comment "Bit 7 has a different meaning" was not updated, so this change also makes the code match the comment again.
--- arch/arm64/mm/dump.c | 6 +++--- arch/x86/mm/dump_pagetables.c | 19 ++++++++++--------- include/linux/ptdump.h | 1 + mm/ptdump.c | 16 ++++++++-------- 4 files changed, 22 insertions(+), 20 deletions(-)
arch/arm64/mm/dump.c +3 -3
@@ -176,8 +176,7 @@ struct pg_level { };static struct pg_level pg_level[] = { - { - }, {/* pgd */ + {/* pgd */ .name ="PGD" , .bits = pte_bits, .num =ARRAY_SIZE (pte_bits),
@@ -257,7 +256,7 @@ static void note_page(struct ptdump_state *pt_st, unsigned long addr, int level,if (level >=0 ) prot = val & pg_level[level].mask; -if (!st->level) { +if (st->level ==-1 ) { st->level = level; st->current_prot = prot; st->start_address = addr;
@@ -351,6 +350,7 @@ void ptdump_check_wx(void) { 0, NULL}, { -1, NULL}, }, + .level = -1, .check_wx = true, .ptdump = { .note_page = note_page,
arch/x86/mm/dump_pagetables.c +10 -9
@@ -180,7 +180,7 @@ static struct addr_marker address_markers[] = {static void printk_prot (struct seq_file *m,pgprotval_t pr,int level,bool dmsg) {static const char *const level_name[] = - {"cr3" ,"pgd" ,"p4d" ,"pud" ,"pmd" ,"pte" }; + {"pgd" ,"p4d" ,"pud" ,"pmd" ,"pte" };if (!(pr & _PAGE_PRESENT)) {/* Not present */
@@ -204,12 +204,12 @@ static void printk_prot(struct seq_file *m, pgprotval_t pr, int level, bool dmsgpt_dump_cont_printf (m, dmsg," " );/* Bit 7 has a different meaning on level 3 vs 4 */ -if (level <=4 && pr & _PAGE_PSE) +if (level <=3 && pr & _PAGE_PSE)pt_dump_cont_printf (m, dmsg,"PSE " );else pt_dump_cont_printf (m, dmsg," " ); -if ((level ==5 && pr & _PAGE_PAT) || - ((level ==4 || level ==3 ) && pr & _PAGE_PAT_LARGE)) +if ((level ==4 && pr & _PAGE_PAT) || + ((level ==3 || level ==2 ) && pr & _PAGE_PAT_LARGE))pt_dump_cont_printf (m, dmsg,"PAT " );else pt_dump_cont_printf (m, dmsg," " );
@@ -271,15 +271,15 @@ static void note_page(struct ptdump_state *pt_st, unsigned long addr, int level, new_prot = val &PTE_FLAGS_MASK ; -if (level >1 ) { - new_eff =effective_prot (st->prot_levels[level -2 ], +if (level >0 ) { + new_eff =effective_prot (st->prot_levels[level -1 ], new_prot); }else { new_eff = new_prot; } -if (level >0 ) - st->prot_levels[level -1 ] = new_eff; +if (level >=0 ) + st->prot_levels[level] = new_eff;/* * If we have a "break" in the series, we need to flush the state that
@@ -289,7 +289,7 @@ static void note_page(struct ptdump_state *pt_st, unsigned long addr, int level, cur = st->current_prot; eff = st->effective_prot; -if (!st->level) { +if (st->level ==-1 ) {/* First entry */ st->current_prot = new_prot; st->effective_prot = new_eff;
@@ -380,6 +380,7 @@ static void ptdump_walk_pgd_level_core(struct seq_file *m, pgd_t *pgd, .note_page = note_page, .range = ptdump_ranges }, + .level = -1, .to_dmesg = dmesg, .check_wx = checkwx, .seq = m
include/linux/ptdump.h +1 -0
@@ -11,6 +11,7 @@ struct ptdump_range { };struct ptdump_state { +/* level is 0:PGD to 4:PTE, or -1 if unknown */ void (*note_page)(struct ptdump_state *st,unsigned long addr,int level,unsigned long val);const struct ptdump_range *range;
mm/ptdump.c +8 -8
@@ -17,7 +17,7 @@ static inline int note_kasan_page_table(struct mm_walk *walk, {struct ptdump_state *st = walk->private; - st->note_page (st, addr,5 ,pte_val (kasan_early_shadow_pte[0 ])); + st->note_page (st, addr,4 ,pte_val (kasan_early_shadow_pte[0 ])); walk->action =ACTION_CONTINUE ;
@@ -37,7 +37,7 @@ static int ptdump_pgd_entry(pgd_t *pgd, unsigned long addr,#endif if (pgd_leaf (val)) - st->note_page (st, addr,1 ,pgd_val (val)); + st->note_page (st, addr,0 ,pgd_val (val));return 0 ; }
@@ -54,7 +54,7 @@ static int ptdump_p4d_entry(p4d_t *p4d, unsigned long addr,#endif if (p4d_leaf (val)) - st->note_page (st, addr,2 ,p4d_val (val)); + st->note_page (st, addr,1 ,p4d_val (val));return 0 ; }
@@ -71,7 +71,7 @@ static int ptdump_pud_entry(pud_t *pud, unsigned long addr,#endif if (pud_leaf (val)) - st->note_page (st, addr,3 ,pud_val (val)); + st->note_page (st, addr,2 ,pud_val (val));return 0 ; }
@@ -88,7 +88,7 @@ static int ptdump_pmd_entry(pmd_t *pmd, unsigned long addr,#endif if (pmd_leaf (val)) - st->note_page (st, addr,4 ,pmd_val (val)); + st->note_page (st, addr,3 ,pmd_val (val));return 0 ; }
@@ -98,7 +98,7 @@ static int ptdump_pte_entry(pte_t *pte, unsigned long addr, {struct ptdump_state *st = walk->private; - st->note_page (st, addr,5 ,pte_val (READ_ONCE (*pte))); + st->note_page (st, addr,4 ,pte_val (READ_ONCE (*pte)));return 0 ; }
@@ -108,7 +108,7 @@ static int ptdump_hole(unsigned long addr, unsigned long next, {struct ptdump_state *st = walk->private; - st->note_page (st, addr, depth +1 ,0 ); + st->note_page (st, addr, depth,0 );return 0 ; }
@@ -135,5 +135,5 @@ void ptdump_walk_pgd(struct ptdump_state *st, struct mm_struct *mm)up_read (&mm->mmap_sem);/* Flush out the last page */ - st->note_page (st,0 ,0 ,0 ); + st->note_page (st,0 ,-1 ,0 ); }
Signature · 1 line
2.20.1