mm: Add __GFP_UNMAPPED
Subsystems (14)
cover mm: Add __GFP_UNMAPPED
.:: What? Why?
This series adds support for efficiently allocating pages that are not present in the direct map. This is instrumental to two different immediate goals:
1. This supports the effort to remove guest_memfd memory from the direct map [0]. One of the challenges faced in that effort has been efficiently eliminating TLB entries, this series offers a solution to that problem
2. Address Space Isolation (ASI) [1] also needs an efficient way to allocate pages that are missing from the direct map. Although for ASI the needs are slightly different (in that case, the pages need only be removed from ASI's special pagetables), the most interesting mm challenges are basically the same.
So, __GFP_UNMAPPED serves as a Trojan horse to get the page allocator into a state where adding ASI's features "Should Be Easy".
This series _also_ serves as a Trojan horse for the "mermap" (details below) which is also a key building block for making ASI efficient.
Longer term, there are a wide range of security techniques unlocked by being able to efficiently remove pages from the kernel's address space.
There may also be non-security usecases for this feature, for example at LPC Sumit Garg presented an issue with memory-firewalled client devices that could he remediated by __GFP_UNMAPPED [2].
.:: Design
The key design elements introduced here are just repurposed from previous attempts to directly introduce ASI's needs to the page allocator [3]. The only real difference is that now these support totally unmapping stuff from the direct map, instead of only unmapping it from ASI's special pagetables.
.:::: Design: Introducing "freetypes"
The biggest challenge for efficiently getting stuff out of the direct map is TLB flushing. Pushing this problem into the page allocator turns out to enable amortising that flush cost into almost nothing. The core idea is to have pools of already-unmapped pages. We'd like those pages to be physically contiguous so they don't unduly fragment the pagetables around them, and we'd like to be able to efficiently look up these already-unmapped pages during allocation. The page allocator already has deeply-ingrained functionality for physically grouping pages by a certain attribute, and then indexing free pages by that attribute, this mechanism is: migratetypes.
So basically, this series extends the concepts of migratetypes in the allocator so that as well as just representing mobility, they can represent other properties of the page too. (Actually, migratetypes are already sort of overloaded, but the main extension is to be able to represent _orthogonal_ properties). In order to avoid further overloading the concept of a migratetype, this extension is done by adding a new concept on top of migratetype: the _freetype_. A freetype is basically just a migratetype plus some flags, and it replaces migratetypes wherever the latter is currently used as to index free pages.
The first freetype flag is then added, which marks the pages it indexes as being absent from the direct map. This is then used to implement the new __GFP_UNMAPPED flag, which allocates pages from pageblocks that have the new flag, or unmaps pages if no existing ones are already available.
.:::: Design: Introducing the "mermap"
Sharp readers might by now be asking how __GFP_UNMAPPED interacts with __GFP_ZERO. If pages aren't in the direct map, how can the page allocator zero them? The solution is the "mermap", short for "epheMERal mapping". The mermap provides an efficient way to temporarily map pages into the local address space, and the allocator uses these mappings to zero pages.
Using the mermap securely requires some knowledge about the usage of the pages. One slightly awkward part of this design is that the page allocator's usage of the mermap then "leaks" out so that callers who allocate with __GFP_UNMAPPED|__GFP_ZERO need to be aware of the mermap's security implications. For the guest_memfd unmapping usecase, that means when guest_memfd.c makes these special allocations, it is only safe because the pages will belong to the current process. In other words, the use of the mermap potentially allows that process to leak the pages via CPU sidechannels (unless more holistic/expensive mitigations are enabled).
Since this cover letter is already too long I won't describe most details of the mermap here, please see the patch that introduces it.
But one key detail is that it requires a kernel-space but mm-local virtual address region. So... this series adds that too (for x86). This is called the mm-local region and is implemented by "just" extending and generalising the LDT remap area.
.:: Outline of the patchset
- Patches 1 -> 2 introduce the mm-local region for x86
- Patches 3 -> 5 introduce the mermap
- Patches 6 -> 14 introduce freetypes
- Patch 8 in particular is the big annoying switch-over which changes a whole bunch of code from "migratetype" to "freetype". In order to try and have the compiler help out with catching bugs, this is done with an annoying typedef. I'm sorry that this patch is so annoying, but I think if we do want to extend the allocator along these lines then a typedef + big annoying patch is probably the safest way.
- Patches 15 -> 20 introduce __GFP_UNMAPPED
.:: Why [RFC]?
I really wanted to stop sending RFC and start sending PATCHes but getting this series out has taken months longer than I expected, so it's time to get something on the list. The known issues here are:
1. __GFP_UNMAPPED isn't useful yet until guest_memfd unmapping support
[0] gets merged.
2. Apparently while implementing the mm-local region, I totally forgot that KPTI existed on 32-bit systems. I expect the 0-day bot to fire a failure on that patch.
There is also one really nasty hack in mermap.c, namely set_unmapped_pte(). This is basically a symptom of the problem I propose to discuss at LSF/MM/BPF [3], i.e. the fact that there are lots of pagetable libraries yet none of them are flexible enough to do anything new (in this case the "new thing" is pre-allocating pagetables then subsequently populating them in a separate context). Whether this particular hack should block merging the mermap is not clear to me, I'd be interested to hear opinions.
.:: Performance
In [4] is a branch containing:
1. This series.
2. All the key kernel patches from the Firecracker team's "secret-free" effort, which includes guest_memfd unmapping ([0]).
3. Some prototype patches to switch guest_memfd over from an ad-hoc unmapping logic to use of __GFP_UNMAPPED (plus direct use of the mermap to implement write()).
I benchmarked this using Firecracker's own performance tests [4], which measure the time required to populate the VM guest's memory. This population happens via write() so it exercises the mermap. I ran this on a Sapphire Rapids machine [5]. The baseline here is just the secret-free patches on their own. "gfp_unmapped" is the branch described above. "skip-flush" provides a reference against an implementation that just skips flushing the TLB when unmapping guest_memfd pages, which serves as an upper-bound on performance.
metric: populate_latency (ms) | test: firecracker-perf-tests-wrapped
+---------------+---------+----------+----------+------------------------+----------+--------+ | nixos_variant | samples | mean | min | histogram | max | Δμ | +---------------+---------+----------+----------+------------------------+----------+--------+ | | 30 | 1.04s | 1.02s | █ | 1.10s | | | gfp_unmapped | 30 | 313.02ms | 299.48ms | █ | 343.25ms | -70.0% | | skip-flush | 30 | 325.80ms | 307.91ms | █ | 333.30ms | -68.8% | +---------------+---------+----------+----------+------------------------+----------+--------+
Conclusion: it's close to the best case performance for this particular workload. (Note in the sample above the mean is actually faster - that's noise, this isn't a consistent observation).
[0] [PATCH v10 00/15] Direct Map Removal Support for guest_memfd
https://lore.kernel.org/all/[email protected]/
[2] https://lpc.events/event/19/contributions/2095/
[3] https://lore.kernel.org/all/[email protected]/
[5] https://github.com/bjackman/aethelred/blob/eb0dd0e99ee08fa0534733113e93b89499affe91
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: Sumit Garg <[email protected]>
To: Borislav Petkov <[email protected]>
To: Dave Hansen <[email protected]>
To: Peter Zijlstra <[email protected]>
To: Andrew Morton <[email protected]>
To: David Hildenbrand <[email protected]>
To: Lorenzo Stoakes <[email protected]>
To: Vlastimil Babka <[email protected]>
To: Mike Rapoport <[email protected]>
To: Wei Xu <[email protected]>
To: Johannes Weiner <[email protected]>
To: Zi Yan <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: Will Deacon <[email protected]>
Cc: [email protected]
Cc: "Kalyazin, Nikita" <[email protected]>
Cc: [email protected]
Cc: "Itazuri, Takahiro" <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: David Kaplan <[email protected]>
Cc: Thomas Gleixner <[email protected]>
---
Brendan Jackman (19):
x86/mm: split out preallocate_sub_pgd()
x86/mm: Generalize LDT remap into "mm-local region" x86/tlb: Expose some flush function declarations to modules x86/mm: introduce the mermap
mm: KUnit tests for the mermap
mm: introduce for_each_free_list()
mm/page_alloc: don't overload migratetype in find_suitable_fallback()
mm: introduce freetype_t
mm: move migratetype definitions to freetype.h
mm: add definitions for allocating unmapped pages
mm: rejig pageblock mask definitions
mm: encode freetype flags in pageblock flags mm/page_alloc: remove ifdefs from pindex helpers mm/page_alloc: separate pcplists by freetype flags mm/page_alloc: rename ALLOC_NON_BLOCK back to _HARDER mm/page_alloc: introduce ALLOC_NOBLOCK
mm/page_alloc: implement __GFP_UNMAPPED allocations mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations
mm: Minimal KUnit tests for some new page_alloc logic
Documentation/arch/x86/x86_64/mm.rst | 4 +- arch/x86/Kconfig | 3 + arch/x86/include/asm/mermap.h | 23 + arch/x86/include/asm/mmu_context.h | 71 ++- arch/x86/include/asm/pgalloc.h | 33 ++ arch/x86/include/asm/pgtable_64_types.h | 19 +- arch/x86/include/asm/pgtable_types.h | 2 + arch/x86/include/asm/tlbflush.h | 43 +- arch/x86/kernel/ldt.c | 137 ++---- arch/x86/mm/init_64.c | 44 +- arch/x86/mm/pgtable.c | 3 + include/linux/freetype.h | 147 ++++++ include/linux/gfp.h | 25 +- include/linux/gfp_types.h | 26 ++ include/linux/mermap.h | 63 +++ include/linux/mermap_types.h | 43 ++ include/linux/mm.h | 13 + include/linux/mm_types.h | 6 + include/linux/mmzone.h | 84 ++-- include/linux/pageblock-flags.h | 16 +- include/trace/events/mmflags.h | 9 +- kernel/fork.c | 6 + kernel/panic.c | 2 + kernel/power/snapshot.c | 8 +- mm/Kconfig | 41 ++ mm/Makefile | 3 + mm/compaction.c | 36 +- mm/init-mm.c | 3 + mm/internal.h | 43 +- mm/mermap.c | 323 +++++++++++++ mm/mm_init.c | 11 +- mm/page_alloc.c | 782 +++++++++++++++++++++++--------- mm/page_isolation.c | 2 +- mm/page_owner.c | 7 +- mm/page_reporting.c | 4 +- mm/pgalloc-track.h | 6 + mm/show_mem.c | 4 +- mm/tests/mermap_kunit.c | 231 ++++++++++ mm/tests/page_alloc_kunit.c | 250 ++++++++++ 39 files changed, 2099 insertions(+), 477 deletions(-)
---
base-commit: 44982d352c33767cd8d19f8044e7e1161a587ff7
change-id: 20260112-page_alloc-unmapped-944fe5d7b55c
Best regards,
Signature · 1 line
Brendan Jackman <[email protected]>
RFC PATCH 1/19 x86/mm: split out preallocate_sub_pgd()
This code will be needed elsewhere in a following patch. Split out the trivial code move for easy review.
This changes the logging slightly: instead of panic() directly reporting the level of the failure, there is now a generic panic message which will be preceded by a separate warn that reports the level of the failure. This is a simple way to have this helper suit the needs of its new user as well as the existing one.
Other than logging, no functional change intended.
--- arch/x86/include/asm/pgalloc.h | 33 +++++++++++++++++++++++++++++++ arch/x86/mm/init_64.c | 44 +++++++----------------------------------- 2 files changed, 40 insertions(+), 37 deletions(-)
arch/x86/include/asm/pgalloc.h +33 -0
@@ -2,6 +2,7 @@#ifndef _ASM_X86_PGALLOC_H#define _ASM_X86_PGALLOC_H +#include <linux/printk.h> #include <linux/threads.h> #include <linux/mm.h> /* for struct page */ #include <linux/pagemap.h>
@@ -128,6 +129,38 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,___pud_free_tlb (tlb, pud); } +/* Allocate a pagetable pointed to by the top hardware level. */ +static inline int preallocate_sub_pgd (struct mm_struct *mm,unsigned long addr) +{ +const char *lvl; +p4d_t *p4d; +pud_t *pud; + + lvl ="p4d" ; + p4d =p4d_alloc (mm,pgd_offset_pgd (mm->pgd, addr), addr); +if (!p4d) + goto failed; + +if (pgtable_l5_enabled ()) +return 0 ; + +/* +* On 4-level systems, the P4D layer is folded away and +* the above code does no preallocation. Below, go down +* to the pud _software_ level to ensure the second +* hardware level is allocated on 4-level systems too. +*/ + lvl ="pud" ; + pud =pud_alloc (mm, p4d, addr); +if (!pud) + goto failed; +return 0 ; + +failed: +pr_warn_ratelimited ("Failed to preallocate %s\n" , lvl); +return -ENOMEM ; +} +#if CONFIG_PGTABLE_LEVELS >4 static inline void pgd_populate (struct mm_struct *mm,pgd_t *pgd,p4d_t *p4d) {
arch/x86/mm/init_64.c +7 -37
53 lines not on this page · raw message
Signature · 1 line
2.51.2
RFC PATCH 2/19 x86/mm: Generalize LDT remap into "mm-local region"
In reply to Brendan JackmanVarious security features benefit from having process-local address mappings. Examples include no-direct-map guest_memfd [2] significant optimizations for ASI [1].
As pointed out by Andy in [0], x86 already has a PGD entry that is local to the mm, which is used for the LDT.
So, simply redefine that entry's region as "the mm-local region" and then redefine the LDT region as a sub-region of that.
With the currently-envisaged usecases, there will be many situations where almost no processes have any need for the mm-local region. Therefore, avoid its overhead (memory cost of pagetables, alloc/free overhead during fork/exit) for processes that don't use it by requiring its users to explicitly initialize it via the new mm_local_* API.
Freeing the pagetables in this region is left to the mm_local_* API implementation and deferred until process exit. This means that the LDT remap code can be simplified:
1. map_ldt_struct_to_user() is now a NOP on 64-bit, since the mm-local region is defined as already being mapped into the user pagetables.
3. free_ldt_pgtables() is no long required at all, it's handled in the core mm teardown logic in both PAE and KPTI cases now.
2. The sanity-check logic is unified: in both cases just walk to the PMD and use presence of that as the proxy for whether an LDT mapping is present. This requires an extra null-check since the page walk will generally terminate early in the KPTI case.
TODO: Agh, this is broken under PAE, looks like I had totally forgotten that KPTI supports 32-bit? Even though there is 32-bit KPTI code modified here. Oops.
[0] https://lore.kernel.org/linux-mm/CALCETrXHbS9VXfZ80kOjiTrreM2EbapYeGp68mvJPbosUtorYA@mail.gmail.com/
[1] https://linuxasi.dev/
[2] https://lore.kernel.org/all/[email protected]
--- Documentation/arch/x86/x86_64/mm.rst | 4 +- arch/x86/Kconfig | 2 + arch/x86/include/asm/mmu_context.h | 71 ++++++++++++++++- arch/x86/include/asm/pgtable_64_types.h | 13 ++- arch/x86/kernel/ldt.c | 137 +++++++++++--------------------- arch/x86/mm/pgtable.c | 3 + include/linux/mm.h | 13 +++ include/linux/mm_types.h | 2 + kernel/fork.c | 1 + mm/Kconfig | 7 ++ 10 files changed, 155 insertions(+), 98 deletions(-)
Documentation/arch/x86/x86_64/mm.rst +2 -2
@@ -53,7 +53,7 @@ Complete virtual memory map with 4-level page tables ____________________________________________________________|___________________________________________________________ | | | | ffff800000000000 | -128 TB | ffff87ffffffffff | 8 TB | ... guard hole, also reserved for hypervisor - ffff880000000000 | -120 TB | ffff887fffffffff | 0.5 TB | LDT remap for PTI + ffff880000000000 | -120 TB | ffff887fffffffff | 0.5 TB | MM-local kernel data. Includes LDT remap for PTI ffff888000000000 | -119.5 TB | ffffc87fffffffff | 64 TB | direct mapping of all physical memory (page_offset_base) ffffc88000000000 | -55.5 TB | ffffc8ffffffffff | 0.5 TB | ... unused hole ffffc90000000000 | -55 TB | ffffe8ffffffffff | 32 TB | vmalloc/ioremap space (vmalloc_base)
@@ -123,7 +123,7 @@ Complete virtual memory map with 5-level page tables ____________________________________________________________|___________________________________________________________ | | | | ff00000000000000 | -64 PB | ff0fffffffffffff | 4 PB | ... guard hole, also reserved for hypervisor - ff10000000000000 | -60 PB | ff10ffffffffffff | 0.25 PB | LDT remap for PTI + ff10000000000000 | -60 PB | ff10ffffffffffff | 0.25 PB | MM-local kernel data. Includes LDT remap for PTI ff11000000000000 | -59.75 PB | ff90ffffffffffff | 32 PB | direct mapping of all physical memory (page_offset_base) ff91000000000000 | -27.75 PB | ff9fffffffffffff | 3.75 PB | ... unused hole ffa0000000000000 | -24 PB | ffd1ffffffffffff | 12.5 PB | vmalloc/ioremap space (vmalloc_base)
arch/x86/Kconfig +2 -0
@@ -133,6 +133,7 @@ config X86 select ARCH_SUPPORTS_RT select ARCH_SUPPORTS_AUTOFDO_CLANG select ARCH_SUPPORTS_PROPELLER_CLANG if X86_64 + select ARCH_SUPPORTS_MM_LOCAL_REGION if X86_64 select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_CMPXCHG_LOCKREF if X86_CX8 select ARCH_USE_MEMTEST
@@ -2320,6 +2321,7 @@ config CMDLINE_OVERRIDE config MODIFY_LDT_SYSCALL bool "Enable the LDT (local descriptor table)" if EXPERT default y + select MM_LOCAL_REGION if MITIGATION_PAGE_TABLE_ISOLATION help Linux can allow user programs to install a per-process x86 Local Descriptor Table (LDT) using the modify_ldt(2) system
arch/x86/include/asm/mmu_context.h +68 -3
100 lines not on this page · raw message
arch/x86/include/asm/pgtable_64_types.h +10 -3
@@ -5,8 +5,11 @@#include <asm/sparsemem.h> #ifndef __ASSEMBLER__ +#include <linux/build_bug.h> #include <linux/types.h> #include <asm/kaslr.h> +#include <asm/page_types.h> +#include <uapi/asm/ldt.h> /* * These are used to make use of C type-checking..
@@ -100,9 +103,13 @@ extern unsigned int ptrs_per_p4d;#define GUARD_HOLE_BASE_ADDR (GUARD_HOLE_PGD_ENTRY << PGDIR_SHIFT)#define GUARD_HOLE_END_ADDR (GUARD_HOLE_BASE_ADDR + GUARD_HOLE_SIZE) -#define LDT_PGD_ENTRY -240UL -#define LDT_BASE_ADDR (LDT_PGD_ENTRY << PGDIR_SHIFT) -#define LDT_END_ADDR (LDT_BASE_ADDR + PGDIR_SIZE) +#define MM_LOCAL_PGD_ENTRY -240UL +#define MM_LOCAL_BASE_ADDR (MM_LOCAL_PGD_ENTRY << PGDIR_SHIFT) +#define MM_LOCAL_END_ADDR ((MM_LOCAL_PGD_ENTRY + 1) << PGDIR_SHIFT) + +#define LDT_BASE_ADDR MM_LOCAL_BASE_ADDR +#define LDT_REMAP_SIZE PMD_SIZE +#define LDT_END_ADDR (LDT_BASE_ADDR + LDT_REMAP_SIZE)#define __VMALLOC_BASE_L4 0xffffc90000000000UL#define __VMALLOC_BASE_L5 0xffa0000000000000UL
arch/x86/kernel/ldt.c +47 -90
198 lines not on this page · raw message
arch/x86/mm/pgtable.c +3 -0
@@ -375,6 +375,9 @@ pgd_t *pgd_alloc(struct mm_struct *mm)void pgd_free (struct mm_struct *mm,pgd_t *pgd) { +/* Should be cleaned up in mmap exit path. */ +VM_WARN_ON_ONCE (mm_local_region_used (mm)); +pgd_mop_up_pmds (mm, pgd);pgd_dtor (pgd);paravirt_pgd_free (mm, pgd);
include/linux/mm.h +13 -0
@@ -904,6 +904,19 @@ static inline void mm_flags_clear_all(struct mm_struct *mm)bitmap_zero (ACCESS_PRIVATE (&mm->flags, __mm_flags),NUM_MM_FLAG_BITS ); } +#ifdef CONFIG_MM_LOCAL_REGION +static inline bool mm_local_region_used (struct mm_struct *mm) +{ +return mm_flags_test (MMF_LOCAL_REGION_USED , mm); +} +#else +static inline bool mm_local_region_used (struct mm_struct *mm) +{ +VM_WARN_ON_ONCE (mm_flags_test (MMF_LOCAL_REGION_USED , mm)); +return false; +} +#endif +extern const struct vm_operations_struct vma_dummy_vm_ops;static inline void vma_init (struct vm_area_struct *vma,struct mm_struct *mm)
include/linux/mm_types.h +2 -0
@@ -1919,6 +1919,8 @@ enum {#define MMF_TOPDOWN 31/* mm searches top down by default */ #define MMF_TOPDOWN_MASK BIT(MMF_TOPDOWN) +#define MMF_LOCAL_REGION_USED 32 +#define MMF_INIT_LEGACY_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\ MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK)
kernel/fork.c +1 -0
@@ -1139,6 +1139,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, fail_nocontext:mm_free_id (mm); fail_noid: +WARN_ON_ONCE (mm_local_region_used (mm));mm_free_pgd (mm); fail_nopgd:futex_hash_free (mm);
mm/Kconfig +7 -0
@@ -1471,6 +1471,13 @@ config LAZY_MMU_MODE_KUNIT_TEST If unsure, say N. +config ARCH_SUPPORTS_MM_LOCAL_REGION + def_bool n + +config MM_LOCAL_REGION + def_bool n + depends on ARCH_SUPPORTS_MM_LOCAL_REGION + source "mm/damon/Kconfig" endmenu
Signature · 1 line
2.51.2
RFC PATCH 3/19 x86/tlb: Expose some flush function declarations to modules
In reply to Brendan JackmanIn commit bfe3d8f6313d ("x86/tlb: Restrict access to tlbstate") some low-level logic (the important detail here is flush_tlb_info) was hidden from modules, along with functions associated with that data.
Later, the set of functions defined here changed and there are now a bunch of flush_tlb_*() functions that do not depend on x86 internals like flush_tlb_info.
This leads to some build fragility: KVM (which can be a module) cares about TLB flushing and includes {linux->asm}/mmu_context.h which includes asm/tlb.h and asm/tlbflush.h. This x86 TLB code expects these helpers to be defined (e.g. tlb_flush() calls flush_tlb_mm_range()).
Modules probably shouldn't call these helpers - luckily this is already enforced by the lack of EXPORT_SYMBOL(). Therefore keep things simple and just expose the declarations anyway to prevent build failures.
--- arch/x86/include/asm/tlbflush.h | 43 +++++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 21 deletions(-)
arch/x86/include/asm/tlbflush.h +22 -21
@@ -229,7 +229,6 @@ struct flush_tlb_info {u8 trim_cpumask; }; -void flush_tlb_local (void );void flush_tlb_one_user (unsigned long addr);void flush_tlb_one_kernel (unsigned long addr); void flush_tlb_multi(const struct cpumask *cpumask,
@@ -303,26 +302,6 @@ static inline void mm_clear_asid_transition(struct mm_struct *mm) { }static inline bool mm_in_asid_transition (struct mm_struct *mm) {return false; }#endif /* CONFIG_BROADCAST_TLB_FLUSH */ -#define flush_tlb_mm (mm) \ - flush_tlb_mm_range(mm, 0UL, TLB_FLUSH_ALL, 0UL, true) - -#define flush_tlb_range (vma, start, end) \ - flush_tlb_mm_range((vma)->vm_mm, start, end, \ - ((vma)->vm_flags & VM_HUGETLB) \ - ? huge_page_shift(hstate_vma(vma)) \ - : PAGE_SHIFT, true) - -extern void flush_tlb_all (void ); -extern void flush_tlb_mm_range (struct mm_struct *mm,unsigned long start, -unsigned long end,unsigned int stride_shift, -bool freed_tables); -extern void flush_tlb_kernel_range (unsigned long start,unsigned long end); - -static inline void flush_tlb_page (struct vm_area_struct *vma,unsigned long a) -{ -flush_tlb_mm_range (vma->vm_mm, a, a +PAGE_SIZE ,PAGE_SHIFT , false); -} -static inline bool arch_tlbbatch_should_defer (struct mm_struct *mm) {bool should_defer = false;
@@ -487,4 +466,26 @@ static inline void __native_tlb_flush_global(unsigned long cr4)native_write_cr4 (cr4 ^X86_CR4_PGE ); native_write_cr4(cr4); } + +#define flush_tlb_mm (mm) \ + flush_tlb_mm_range(mm, 0UL, TLB_FLUSH_ALL, 0UL, true) + +#define flush_tlb_range (vma, start, end) \ + flush_tlb_mm_range((vma)->vm_mm, start, end, \ + ((vma)->vm_flags & VM_HUGETLB) \ + ? huge_page_shift(hstate_vma(vma)) \ + : PAGE_SHIFT, true) + +void flush_tlb_local (void ); +extern void flush_tlb_all (void ); +extern void flush_tlb_mm_range (struct mm_struct *mm,unsigned long start, +unsigned long end,unsigned int stride_shift, +bool freed_tables); +extern void flush_tlb_kernel_range (unsigned long start,unsigned long end); + +static inline void flush_tlb_page (struct vm_area_struct *vma,unsigned long a) +{ +flush_tlb_mm_range (vma->vm_mm, a, a +PAGE_SIZE ,PAGE_SHIFT , false); +} +#endif /* _ASM_X86_TLBFLUSH_H */
Signature · 1 line
2.51.2
RFC PATCH 4/19 x86/mm: introduce the mermap
In reply to Brendan JackmanThe mermap provides a fast way to create ephemeral mm-local mappings of physical pages. The purpose of this is to access pages that have been removed from the direct map. Potential use cases are:
1. For zeroing __GFP_UNMAPPED pages (added in a later patch).
2. For populating guest_memfd pages that are protected by the GUEST_MEMFD_NO_DIRECT_MAP feature [0].
3. For efficient access of pages protected by Address Space Isolation [1].
[0] https://lore.kernel.org/all/[email protected]/
[1] https://linuxasi.dev
The details of this mechanism are described in the API comments. However the key idea is to use CPU-local virtual regions to avoid a need for synchronizing. On x86, this can also be used to prevent TLB shootdowns.
Because the virtual region is CPU-local, allocating from the mermap disables migration. The caller is forbidden to use the returned value from any other context, and migration is re-enabled when it's freed.
One might notice that mermap_get() bears a strong similarity to kmap_local_page(). The most important differences between mermap_get() and kmap_local_page() are:
1. mermap_get() allows mapping variable sizes while kmap_local_page() specifically maps a single order-0 page.
2. As a consequence of 1 (combined with the need for mermap_get() to be an extremely simple allocator), mermap_get() should be expected to fail, while kmap_local_page() is guaranteed to work up to a certain degree of nesting.
3. While the mappings provided by kmap_local_page() are _logically_ local to the calling context (it's a bug for software to access them from elsewhere), they are _physically_ installed into the shared kernel pagetables. This means their locality doesn't provide any protection from hardware attacks. In contrast, the mermap is physically local to the creating mm, taking advantage of the new mm-local kernel address region.
So that the mermap is available even in contexts where failure is not tolerable there is also a _reserved() variant, which is fixed at allocating a single base page. This is useful, for example, for zeroing __GFP_UNMAPPED pages, where handling failure would be extremely inconvenient. The _reserved() variant is simply implemented by leaving one base-page space unavailable for non-_reserved allocations, and requiring an atomic context.
This mechanism obviously requires manipulating pagetables. The kernel doesn't have a "library" that is 100% suitable for the mermap's needs here. This is resolved with a hack, namely exploiting apply_to[_existing_page_range(), which is _almost_ suitable for the requirements. This will need some later refactoring (perhaps creating the "library") to resolve the hacks it introduces, which are:
1. It introduces an indirect branch, which is likely to be pretty slow on some platforms.
2. It uses a magic sentinel pagetable value, instead of pte_none(), for unmapped regions, to trick apply_to_existing_page_range() into operating on them (while still ensuring no pagetable allocations take place).
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/mermap.h | 23 +++ arch/x86/include/asm/pgtable_64_types.h | 8 +- include/linux/mermap.h | 63 +++++++ include/linux/mermap_types.h | 43 +++++ include/linux/mm_types.h | 4 + kernel/fork.c | 5 + mm/Kconfig | 11 ++ mm/Makefile | 1 + mm/mermap.c | 319 ++++++++++++++++++++++++++++++++ mm/pgalloc-track.h | 6 + 11 files changed, 483 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
@@ -37,6 +37,7 @@ config X86_64 select ZONE_DMA32 select EXECMEM if DYNAMIC_FTRACE select ACPI_MRRM if ACPI + select ARCH_SUPPORTS_MERMAP config FORCE_DYNAMIC_FTRACE def_bool y
arch/x86/include/asm/mermap.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_MERMAP_H +#define _ASM_X86_MERMAP_H + +#include <asm/tlbflush.h> + +static inline void arch_mermap_flush_tlb (void ) +{ +/* +* No shootdown allowed, IRQs may be off. Luckily other CPUs are not +* allowed to access our region so the stale mappings are harmless, as +* long as they still point to data belonging to this process. +*/ +__flush_tlb_all (); +} + +static inline bool arch_mermap_pgprot_allowed (pgprot_t prot) +{ +/* Mermap is mm-local so global mappings would be a bug. */ +return !(pgprot_val (prot) & _PAGE_GLOBAL); +} + +#endif /* _ASM_X86_MERMAP_H */
arch/x86/include/asm/pgtable_64_types.h +7 -1
@@ -105,12 +105,18 @@ extern unsigned int ptrs_per_p4d;#define MM_LOCAL_PGD_ENTRY -240UL#define MM_LOCAL_BASE_ADDR (MM_LOCAL_PGD_ENTRY << PGDIR_SHIFT) -#define MM_LOCAL_END_ADDR ((MM_LOCAL_PGD_ENTRY + 1) << PGDIR_SHIFT) +#define MM_LOCAL_START_ADDR ((MM_LOCAL_PGD_ENTRY) << PGDIR_SHIFT) +#define MM_LOCAL_END_ADDR (MM_LOCAL_START_ADDR + (1UL << PGDIR_SHIFT))#define LDT_BASE_ADDR MM_LOCAL_BASE_ADDR#define LDT_REMAP_SIZE PMD_SIZE#define LDT_END_ADDR (LDT_BASE_ADDR + LDT_REMAP_SIZE) +#define MERMAP_BASE_ADDR LDT_END_ADDR +#define MERMAP_CPU_REGION_SIZE PMD_SIZE +#define MERMAP_SIZE (MERMAP_CPU_REGION_SIZE * NR_CPUS) +#define MERMAP_END_ADDR (MERMAP_BASE_ADDR + (NR_CPUS * MERMAP_CPU_REGION_SIZE)) +#define __VMALLOC_BASE_L4 0xffffc90000000000UL#define __VMALLOC_BASE_L5 0xffa0000000000000UL
include/linux/mermap.h new file +63 -0
63 lines not on this page · raw message
include/linux/mermap_types.h new file +43 -0
43 lines not on this page · raw message
include/linux/mm_types.h +4 -0
@@ -7,6 +7,7 @@#include <linux/auxvec.h> #include <linux/kref.h> #include <linux/list.h> +#include <linux/mermap_types.h> #include <linux/spinlock.h> #include <linux/rbtree.h> #include <linux/maple_tree.h>
@@ -34,6 +35,7 @@struct address_space ;struct futex_private_hash ;struct mem_cgroup ; +struct mermap ; typedef struct { unsigned long f;
@@ -1159,6 +1161,8 @@ struct mm_struct {atomic_t membarrier_state;#endif +struct mermap mermap; +/** * @mm_users: The number of users including userspace. *
kernel/fork.c +5 -0
@@ -13,6 +13,7 @@*/ #include <linux/anon_inodes.h> +#include <linux/mermap.h> #include <linux/slab.h> #include <linux/sched/autogroup.h> #include <linux/sched/mm.h>
@@ -1130,6 +1131,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm->user_ns =get_user_ns (user_ns);lru_gen_init_mm (mm); + +mermap_mm_init (mm); +return mm; fail_pcpu:
@@ -1173,6 +1177,7 @@ static inline void __mmput(struct mm_struct *mm)ksm_exit (mm);khugepaged_exit (mm);/* must run before exit_mmap */ exit_mmap (mm); +mermap_mm_teardown (mm);mm_put_huge_zero_folio (mm);set_mm_exe_file (mm,NULL );if (!list_empty (&mm->mmlist)) {
mm/Kconfig +11 -0
@@ -1480,4 +1480,15 @@ config MM_LOCAL_REGION source "mm/damon/Kconfig" +config ARCH_SUPPORTS_MERMAP + bool + +config MERMAP + bool "Support for epheMERal mappings within the kernel" + default COMPILE_TEST + depends on ARCH_SUPPORTS_MERMAP + select MM_LOCAL_REGION + help + Support for epheMERal mappings within the kernel. + endmenu
mm/Makefile +1 -0
@@ -150,3 +150,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o +obj-$(CONFIG_MERMAP) += mermap.o
mm/mermap.c new file +319 -0
319 lines not on this page · raw message
mm/pgalloc-track.h +6 -0
@@ -2,6 +2,12 @@#ifndef _LINUX_PGALLOC_TRACK_H#define _LINUX_PGALLOC_TRACK_H +#include <linux/mm.h> +#include <linux/pgalloc.h> +#include <linux/pgtable.h> + +#include "internal.h" +#if defined(CONFIG_MMU ) static inline p4d_t *p4d_alloc_track(struct mm_struct *mm, pgd_t *pgd, unsigned long address,
Signature · 1 line
2.51.2
Re: [PATCH RFC 04/19] x86/mm: introduce the mermap
Relaying some code review from an AI that I wasn't able to run before sending...
(This isn't the AI's verbatim output I'm filtering it and rephrasing).
Quoted 75 lines · Brendan Jackman
mm/mermap.c new file +75
-0+ +/* Call with migration disabled. */ +static inline struct mermap_alloc *mermap_alloc (struct mm_struct *mm, +unsigned long size,bool use_reserve) +{ +int cpu =raw_smp_processor_id (); +struct mermap_cpu *mc =this_cpu_ptr (mm->mermap.cpu); +unsigned long cpu_end =mermap_cpu_end (cpu); +struct mermap_alloc *alloc =NULL ; + +/* +* This is an extremely stupid allocator, there can only ever be a small +* number of allocations so everything just works on linear search. +* +* Allocations are "in order", i.e. if the whole region is free it +* allocates from the beginning. If there are any existing allocations +* it allocates from right after the last (highest address) one. Any +* free space before that goes unused. +* +* Once an allocation has been freed, the space it occupied must be flushed +* from the TLB before it can be reused. +* +* Visual example of how this is suppose to behave (A for allocated, T for +* TLB-flush-pending): +* +* _______________ Start with everything free. +* AaaA___________ Allocate something. +* TttT___________ Free it. (Region needs a TLB flush now). +* TttTAaaaaaaaA__ Allocate something else. +* TttTAaaaaaaaAAA Allocate the remaining space. +* TttTTtttttttTAA Free the allocation before last. +* ^^^^^^^^^^^^^ This could all be reused now but for simplicity it +* isn't. Another allocation at this point will fail. +* TttTTtttttttTTT Free the last allocation. +* _______________ Next time we allocate, first flush the TLB +* AA_____________ Now we're back at the beginning. +*/ + +if (use_reserve) { +if (WARN_ON_ONCE (size !=PAGE_SIZE )) +return NULL ; +lockdep_assert_preemption_disabled (); + }else { + cpu_end -=PAGE_SIZE ; + } + +if (WARN_ON_ONCE (!in_task ())) +return NULL ; +guard (preempt)(); + +/* Out of already-available space? */ +if (mc->next_addr + size > cpu_end) { +unsigned long new_next =mermap_cpu_base (cpu); + +/* Would we have space after a TLB flush? */ +for (int i =0 ; i <ARRAY_SIZE (mc->allocs); i++) { +struct mermap_alloc *alloc = &mc->allocs[i]; + +/* +* The space between the uppermost allocated alloc->end +* (or the base of the CPU's region if there are no +* current allocations) and mc->next_addr has been +* unmapped in the pagetables, but not flushed from the +* TLB. Set new_next to point to the beginning of that +* space. +*/ +if (READ_ONCE (alloc->in_use)) + new_next =max (new_next, alloc->end); + } +if (size > cpu_end - new_next) +return NULL ; + +mermap_flush_tlb (cpu, mc); + mc->next_addr = new_next; + }
mm/mermap.c line 152
Quoted 10 lines
mm/mermap.c new file +10
-0+ +/* Find an alloc-tracking structure to use */ +for (int i =0 ; i <ARRAY_SIZE (mc->allocs); i++) { +if (!READ_ONCE (mc->allocs[i].in_use)) { + alloc = &mc->allocs[i]; +break ; + } + } +if (!alloc) +return NULL ;
Oops, I forgot to account for @use_reserve here. The alloc-tracking structures should have a reservation like how the virtual address space does otherwise allocations can fail where they aren't supposed to.
Quoted 40 lines
mm/mermap.c new file +40
-0+ alloc->in_use = true; + alloc->base = mc->next_addr; + alloc->end = alloc->base + size; + mc->next_addr += size; + +return alloc; +} + +struct set_pte_ctx { +pgprot_t prot; +unsigned long next_pfn; +}; + +static inline int do_set_pte (pte_t *pte,unsigned long addr,void *data) +{ +struct set_pte_ctx *ctx = data; + +set_pte (pte,pfn_pte (ctx->next_pfn, ctx->prot)); + ctx->next_pfn++; + +return 0 ; +} + +static struct mermap_alloc * +__mermap_get (struct mm_struct *mm,struct page *page, +unsigned long size,pgprot_t prot,bool use_reserve) +{ +struct mermap_alloc *alloc =NULL ; +struct set_pte_ctx ctx; +int err; + +if (size >MERMAP_CPU_REGION_SIZE ||WARN_ON_ONCE (!mm || !mm->mermap.cpu)) +return NULL ; +if (WARN_ON_ONCE (!arch_mermap_pgprot_allowed (prot))) +return NULL ; + + size =PAGE_ALIGN (size); + +migrate_disable (); +
mm/mermap.c line 202
Quoted 10 lines
mm/mermap.c new file +10
-0+ alloc =mermap_alloc (mm, size, use_reserve); +if (!alloc) { +migrate_enable (); +return NULL ; + } + +/* This probably wants to be optimised. */ + ctx.prot = prot; + ctx.next_pfn =page_to_pfn (page); + err =apply_to_existing_page_range (mm, alloc->base, size, do_set_pte, &ctx);
This takes a PTE lock, and we may have preemption off here, so this may be broken on PREEMPT_RT? Haven't checked. (Maybe I can just test this by running it with PREEMPT_RT and lockdep enabled?)
If that's indeed broken, this is yet another point to discuss in about requirements for a pagetable library [0]. The lock is not needed at all here - we need a way to modify pagetables that lets you take advantage of higher-level synchronization.
mm/mermap.c line 205
Quoted 3 lines
mm/mermap.c new file +3
-0+if (err) { +WRITE_ONCE (alloc->in_use, false); +return NULL ;
Forgot migrate_enable().
(Is there a way to prevent this with the lovely new __attribute__((cleanup)) magic?)
Quoted 13 lines
mm/mermap.c new file +13
-0+ } + +return alloc; +} + +/* +* Allocate a region of virtual memory, and map the page into it. This tries +* pretty hard to be fast but doesn't try very hard at all to actually succeed. +* +* The returned region is physically local to the current mm. It is _logically_ +* local to the current CPU but this is not enforced by hardware so it can be +* exploited to mitigate CPU vulns. This means the caller must not map memory +* here that doesn't belong to the current process. The caller must also perform
mm/mermap.c line 228
Quoted 10 lines
mm/mermap.c new file +10
-0+* a full TLB flush of the region before freeing the pages that have been mapped +* here. +* +* This may only be called from process context, and the caller must arrange to +* first call mermap_mm_prepare(). (It would be possible to support this in IRQ, +* but it seems unlikely there's a valid usecase given the TLB flushing +* requirements). If it succeeds, it disables migration until you call +* mermap_put(). +* +* This is guaranteed not to allocate.
This one isn't from AI, but I just realised that's a pretty confusing thing for an allocator to say. It should say it's guaranteed not to call the page allocator. (This is important coz I want to use it from the page allocator).
Quoted 4 lines
+ *
+ * Usemermap_addr()to get the actual address of the mapped region.
+ */mm/mermap.c new file +1
-0+struct mermap_alloc *mermap_get (struct page *page,unsigned long size,pgprot_t prot)
mm/mermap.c line 242
Quoted 10 lines
mm/mermap.c new file +10
-0+{ +return __mermap_get (current->mm, page, size, prot, false); +} +EXPORT_SYMBOL (mermap_get); + +/* +* Allocate a single PAGE_SIZE page via mermap_get(), requiring preemption to be +* off until it is freed. This always succeeds. +*/ +void *mermap_get_reserved (struct page *page,pgprot_t prot)
Oops, that should return mermap_alloc *.
mm/mermap.c line 312
Quoted 10 lines
mm/mermap.c new file +10
-0+/* Clean up mermap stuff on mm teardown. */ +void mermap_mm_teardown (struct mm_struct *mm) +{ +int cpu; + +if (!mm->mermap.cpu) +return ; + +for_each_possible_cpu (cpu) { +struct mermap_cpu *mc =this_cpu_ptr (mm->mermap.cpu);
Oops, this should be per_cpu_ptr(..., cpu) or whatever it is.
RFC PATCH 5/19 mm: KUnit tests for the mermap
In reply to Brendan JackmanSome simple smoke-tests for the mermap. Mainly aiming to test:
1. That there aren't any silly off-by-ones.
2. That the pagetables are not completely broken.
3. That the TLB appears to get flushed basically when expected.
This last point requires a bit of ifdeffery to detect when the flushing has been performed.
--- include/linux/mermap_types.h | 2 +- mm/Kconfig | 11 +++ mm/Makefile | 1 + mm/mermap.c | 14 ++- mm/tests/mermap_kunit.c | 231 +++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 253 insertions(+), 6 deletions(-)
include/linux/mermap_types.h +1 -1
@@ -23,7 +23,7 @@ struct mermap_cpu {/* Next address immediately available for alloc (no TLB flush needed). */ unsigned long next_addr;struct mermap_alloc allocs[4 ]; -#ifdef CONFIG_MERMAP_KUNIT_TEST +#if IS_ENABLED (CONFIG_MERMAP_KUNIT_TEST )u64 tlb_flushes;#endif };
mm/Kconfig +11 -0
@@ -1491,4 +1491,15 @@ config MERMAP help Support for epheMERal mappings within the kernel. +config MERMAP_KUNIT_TEST + tristate "KUnit tests for the mermap" if !KUNIT_ALL_TESTS + depends on ARCH_SUPPORTS_MERMAP + depends on KUNIT + select MERMAP + default KUNIT_ALL_TESTS + help + KUnit test for the mermap. + + If unsure, say N. + endmenu
mm/Makefile +1 -0
@@ -151,3 +151,4 @@ obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o obj-$(CONFIG_MERMAP) += mermap.o +obj-$(CONFIG_MERMAP_KUNIT_TEST) += tests/mermap_kunit.o
mm/mermap.c +9 -5
@@ -24,7 +24,7 @@ static inline int set_unmapped_pte(pte_t *ptep, unsigned long addr, void *data)return 0 ; } -static void __mermap_put (struct mm_struct *mm,struct mermap_alloc *alloc) +VISIBLE_IF_KUNIT void __mermap_put(struct mm_struct *mm, struct mermap_alloc *alloc) {unsigned long size =PAGE_ALIGN (alloc->end - alloc->base);
@@ -37,6 +37,7 @@ static void __mermap_put(struct mm_struct *mm, struct mermap_alloc *alloc)migrate_enable (); } +EXPORT_SYMBOL_IF_KUNIT (__mermap_put);/* Return a region allocated by mermap_get(). */ void mermap_put (struct mermap_alloc *alloc)
@@ -45,22 +46,24 @@ void mermap_put(struct mermap_alloc *alloc) }EXPORT_SYMBOL (mermap_put); -static inline unsigned long mermap_cpu_base (int cpu) +VISIBLE_IF_KUNIT inline unsigned long mermap_cpu_base(int cpu) {return MERMAP_BASE_ADDR + (cpu *MERMAP_CPU_REGION_SIZE ); } +EXPORT_SYMBOL_IF_KUNIT (mermap_cpu_base);/* Non-inclusive */ -static inline unsigned long mermap_cpu_end (int cpu) +VISIBLE_IF_KUNIT inline unsigned long mermap_cpu_end(int cpu) {return MERMAP_BASE_ADDR + ((cpu +1 ) *MERMAP_CPU_REGION_SIZE ); } +EXPORT_SYMBOL_IF_KUNIT (mermap_cpu_end);static inline void mermap_flush_tlb (int cpu,struct mermap_cpu *mc) { -#ifdef CONFIG_MERMAP_KUNIT_TEST +#if IS_ENABLED (CONFIG_MERMAP_KUNIT_TEST ) mc->tlb_flushes++;#endif arch_mermap_flush_tlb ();
@@ -173,7 +176,7 @@ static inline int do_set_pte(pte_t *pte, unsigned long addr, void *data)return 0 ; } -static struct mermap_alloc * +VISIBLE_IF_KUNIT struct mermap_alloc *__mermap_get (struct mm_struct *mm,struct page *page,unsigned long size,pgprot_t prot,bool use_reserve) {
@@ -207,6 +210,7 @@ __mermap_get(struct mm_struct *mm, struct page *page,return alloc; } +EXPORT_SYMBOL_IF_KUNIT (__mermap_get);/* * Allocate a region of virtual memory, and map the page into it. This tries
mm/tests/mermap_kunit.c new file +231 -0
231 lines not on this page · raw message
Signature · 1 line
2.51.2
RFC PATCH 6/19 mm: introduce for_each_free_list()
In reply to Brendan JackmanLater patches will rearrange the free areas, but there are a couple of places that iterate over them with the assumption that they have the current structure.
It seems ideally, code outside of mm should not be directly aware of struct free_area in the first place, but that awareness seems relatively harmless so just make the minimal change.
Now instead of letting users manually iterate over the free lists, just provide a macro to do that. Then adopt that macro in a couple of places.
--- include/linux/mmzone.h | 7 +++++-- kernel/power/snapshot.c | 8 ++++---- mm/mm_init.c | 11 +++++++---- 3 files changed, 16 insertions(+), 10 deletions(-)
include/linux/mmzone.h +5 -2
@@ -123,9 +123,12 @@ static inline bool migratetype_is_mergeable(int mt)return mt <MIGRATE_PCPTYPES ; } -#define for_each_migratetype_order (order, type) \ +#define for_each_free_list (list, zone, order) \ for (order = 0; order < NR_PAGE_ORDERS; order++) \ - for (type = 0; type < MIGRATE_TYPES; type++) + for (unsigned int type = 0; \ + list = &zone->free_area[order].free_list[type], \ + type < MIGRATE_TYPES; \ + type++) \extern int page_group_by_mobility_disabled;
kernel/power/snapshot.c +4 -4
@@ -1244,8 +1244,9 @@ unsigned int snapshot_additional_pages(struct zone *zone)static void mark_free_pages (struct zone *zone) {unsigned long pfn, max_zone_pfn, page_count =WD_PAGE_COUNT ; +struct list_head *free_list;unsigned long flags; -unsigned int order, t; +unsigned int order;struct page *page;if (zone_is_empty (zone))
@@ -1269,9 +1270,8 @@ static void mark_free_pages(struct zone *zone)swsusp_unset_page_free (page); } -for_each_migratetype_order (order, t) { -list_for_each_entry (page, - &zone->free_area[order].free_list[t], buddy_list) { +for_each_free_list (free_list, zone, order) { +list_for_each_entry (page, free_list, buddy_list) {unsigned long i; pfn =page_to_pfn (page);
mm/mm_init.c +7 -4
@@ -1432,11 +1432,14 @@ static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx,static void __meminit zone_init_free_lists (struct zone *zone) { -unsigned int order, t; -for_each_migratetype_order (order, t) { -INIT_LIST_HEAD (&zone->free_area[order].free_list[t]); +struct list_head *list; +unsigned int order; + +for_each_free_list (list, zone, order) +INIT_LIST_HEAD (list); + +for (order =0 ; order <NR_PAGE_ORDERS ; order++) zone->free_area[order].nr_free =0 ; - }#ifdef CONFIG_UNACCEPTED_MEMORY INIT_LIST_HEAD (&zone->unaccepted_pages);
Signature · 1 line
2.51.2
RFC PATCH 7/19 mm/page_alloc: don't overload migratetype in find_suitable_fallback()
In reply to Brendan JackmanThis function currently returns a signed integer that encodes status in-band, as negative numbers, along with a migratetype.
This function is about to be updated to a mode where this in-band signaling no longer makes sense. Therefore, switch to a more explicit/verbose style that encodes the status and migratetype separately.
In the spirit of making things more explicit, also create an enum to avoid using magic integer literals with special meanings. This enables documenting the values at their definition instead of in one of the callers.
--- mm/compaction.c | 3 ++- mm/internal.h | 14 +++++++++++--- mm/page_alloc.c | 40 +++++++++++++++++++++++----------------- 3 files changed, 36 insertions(+), 21 deletions(-)
mm/compaction.c +2 -1
@@ -2323,7 +2323,8 @@ static enum compact_result __compact_finished(struct compact_control *cc)* Job done if allocation would steal freepages from * other migratetype buddy lists. */ -if (find_suitable_fallback (area, order, migratetype, true) >=0 ) +if (find_suitable_fallback (area, order, migratetype, true,NULL ) + ==FALLBACK_FOUND )/* * Movable pages are OK in any pageblock. If we are * stealing for a non-movable allocation, make sure
mm/internal.h +11 -3
@@ -1028,9 +1028,17 @@ static inline void init_cma_pageblock(struct page *page) }#endif - -int find_suitable_fallback (struct free_area *area,unsigned int order, -int migratetype,bool claimable); +enum fallback_result { +/* Found suitable migratetype, *mt_out is valid. */ +FALLBACK_FOUND , +/* No fallback found in requested order. */ +FALLBACK_EMPTY , +/* Passed @claimable, but claiming whole block is a bad idea. */ +FALLBACK_NOCLAIM , +}; +enum fallback_result +find_suitable_fallback (struct free_area *area,unsigned int order, +int migratetype,bool claimable,unsigned int *mt_out);static inline bool free_area_empty (struct free_area *area,int migratetype) {
mm/page_alloc.c +23 -17
@@ -2280,25 +2280,29 @@ static bool should_try_claim_block(unsigned int order, int start_mt)* we would do this whole-block claiming. This would help to reduce * fragmentation due to mixed migratetype pages in one pageblock. */ -int find_suitable_fallback (struct free_area *area,unsigned int order, -int migratetype,bool claimable) +enum fallback_result +find_suitable_fallback (struct free_area *area,unsigned int order, +int migratetype,bool claimable,unsigned int *mt_out) {int i;if (claimable && !should_try_claim_block (order, migratetype)) -return -2 ; +return FALLBACK_NOCLAIM ;if (area->nr_free ==0 ) -return -1 ; +return FALLBACK_EMPTY ;for (i =0 ; i <MIGRATE_PCPTYPES -1 ; i++) {int fallback_mt = fallbacks[migratetype][i]; -if (!free_area_empty (area, fallback_mt)) -return fallback_mt; +if (!free_area_empty (area, fallback_mt)) { +if (mt_out) + *mt_out = fallback_mt; +return FALLBACK_FOUND ; + } } -return -1 ; +return FALLBACK_EMPTY ; }/*
@@ -2408,16 +2412,16 @@ __rmqueue_claim(struct zone *zone, int order, int start_migratetype,*/ for (current_order =MAX_PAGE_ORDER ; current_order >= min_order; --current_order) { - area = &(zone->free_area[current_order]); - fallback_mt =find_suitable_fallback (area, current_order, - start_migratetype, true); +enum fallback_result result; -/* No block in that order */ -if (fallback_mt ==-1 ) + area = &(zone->free_area[current_order]); + result =find_suitable_fallback (area, current_order, + start_migratetype, true, &fallback_mt); + +if (result ==FALLBACK_EMPTY )continue ; -/* Advanced into orders too low to claim, abort */ -if (fallback_mt ==-2 ) +if (result ==FALLBACK_NOCLAIM )break ; page =get_page_from_free_area (area, fallback_mt);
@@ -2447,10 +2451,12 @@ __rmqueue_steal(struct zone *zone, int order, int start_migratetype)int fallback_mt;for (current_order = order; current_order <NR_PAGE_ORDERS ; current_order++) { +enum fallback_result result; + area = &(zone->free_area[current_order]); - fallback_mt =find_suitable_fallback (area, current_order, - start_migratetype, false); -if (fallback_mt ==-1 ) + result =find_suitable_fallback (area, current_order, start_migratetype, + false, &fallback_mt); +if (result ==FALLBACK_EMPTY )continue ; page =get_page_from_free_area (area, fallback_mt);
Signature · 1 line
2.51.2
RFC PATCH 8/19 mm: introduce freetype_t
In reply to Brendan JackmanThis is preparation for teaching the page allocator to break up free pages according to properties that have nothing to do with mobility. For example it can be used to allocate pages that are non-present in the physmap, or pages that are sensitive in ASI.
For these usecases, certain allocator behaviours are desirable:
- A "pool" of pages with the given property is usually available, so that pages can be provided with the correct sensitivity without zeroing/TLB flushing.
- Pages are physically grouped by the property, so that large allocations rarely have to alter the pagetables due to ASI.
- The properties can be forced to vary only at a certain fixed address granularity, so that the pagetables can all be pre-allocated. This is desirable because the page allocator will be changing mappings:
pre-allocation is a straightforward way to avoid recursive allocations (of pagetables).
It seems that the existing infrastructure for grouping pages by mobility, i.e. pageblocks and migratetypes, serves this purpose pretty nicely. However, overloading migratetype itself for this purpose looks like a road to maintenance hell. In particular, as soon as such properties become orthogonal to migratetypes, it would start to require "doubling" the migratetypes.
Therefore, introduce a new higher-level concept, called "freetype" (because it is used to index "free"lists) that can encode extra properties, orthogonally to mobility, via flags.
Since freetypes and migratetypes would be very easy to mix up, freetypes are (at least for now) stored in a struct typedef similar to atomic_t. This provides type-safety, but comes at the expense of being pretty annoying to code with. For instance, freetype_t cannot be compared with the == operator. Once this code matures, if the freetype/migratetype distinction gets less confusing, it might be wise to drop this struct and just use ints.
Because this will eventually be needed from pageblock-flags.h, put this in its own header instead of directly in mmzone.h.
To try and reduce review pain for such a churny patch, first introduce freetypes as nothing but an indirection over migratetypes. The helpers concerned with the flags are defined, but only as stubs. Convert everything over to using freetypes wherever they are needed to index freelists, but maintain references to migratetypes in code that really only cares specifically about mobility.
--- include/linux/freetype.h | 38 +++++ include/linux/gfp.h | 16 +- include/linux/mmzone.h | 49 +++++- mm/compaction.c | 35 +++-- mm/internal.h | 17 ++- mm/page_alloc.c | 388 +++++++++++++++++++++++++++++------------------ mm/page_isolation.c | 2 +- mm/page_owner.c | 7 +- mm/page_reporting.c | 4 +- mm/show_mem.c | 4 +- 10 files changed, 370 insertions(+), 190 deletions(-)
include/linux/freetype.h new file +38 -0
@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_FREETYPE_H +#define _LINUX_FREETYPE_H + +#include <linux/types.h> + +/* +* A freetype is the index used to identify free lists. This consists of a +* migratetype, and other bits which encode orthogonal properties of memory. +*/ +typedef struct { +int migratetype; +}freetype_t ; + +/* +* Return a dense linear index for freetypes that have lists in the free area. +* Return -1 for other freetypes. +*/ +static inline int freetype_idx (freetype_t freetype) +{ +return freetype.migratetype; +} + +/* No freetype flags actually exist yet. */ +#define NR_FREETYPE_IDXS MIGRATE_TYPES + +static inline unsigned int freetype_flags (freetype_t freetype) +{ +/* No flags supported yet. */ +return 0 ; +} + +static inline bool freetypes_equal (freetype_t a,freetype_t b) +{ +return a.migratetype == b.migratetype; +} + +#endif /* _LINUX_FREETYPE_H */
include/linux/gfp.h +11 -5
@@ -17,8 +17,10 @@ struct mempolicy;#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)#define GFP_MOVABLE_SHIFT 3 -static inline int gfp_migratetype (const gfp_t gfp_flags) +static inline freetype_t gfp_freetype (const gfp_t gfp_flags) { +int migratetype; +VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK );BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE);BUILD_BUG_ON ((___GFP_MOVABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_MOVABLE );
@@ -26,11 +28,15 @@ static inline int gfp_migratetype(const gfp_t gfp_flags)BUILD_BUG_ON (((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >>GFP_MOVABLE_SHIFT ) !=MIGRATE_HIGHATOMIC ); -if (unlikely (page_group_by_mobility_disabled)) -return MIGRATE_UNMOVABLE ; +if (unlikely (page_group_by_mobility_disabled)) { + migratetype =MIGRATE_UNMOVABLE ; + }else { +/* Group based on mobility */ + migratetype = (__force unsigned long )(gfp_flags &GFP_MOVABLE_MASK ) + >>GFP_MOVABLE_SHIFT ; + } -/* Group based on mobility */ -return (__force unsigned long )(gfp_flags &GFP_MOVABLE_MASK ) >>GFP_MOVABLE_SHIFT ; +return migrate_to_freetype (migratetype,0 ); }#undef GFP_MOVABLE_MASK#undef GFP_MOVABLE_SHIFT
include/linux/mmzone.h +44 -5
74 lines not on this page · raw message
mm/compaction.c +19 -16
@@ -1359,7 +1359,7 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn,static bool suitable_migration_source (struct compact_control *cc,struct page *page) { -int block_mt; +freetype_t block_ft;if (pageblock_skip_persistent (page))return false;
@@ -1367,12 +1367,12 @@ static bool suitable_migration_source(struct compact_control *cc,if ((cc->mode !=MIGRATE_ASYNC ) || !cc->direct_compaction)return true; - block_mt =get_pageblock_migratetype (page); + block_ft =get_pageblock_freetype (page); -if (cc->migratetype ==MIGRATE_MOVABLE ) -return is_migrate_movable (block_mt); +if (free_to_migratetype (cc->freetype) ==MIGRATE_MOVABLE ) +return is_migrate_movable (free_to_migratetype (block_ft));else -return block_mt == cc->migratetype; +return freetypes_equal (block_ft, cc->freetype); }/* Returns true if the page is within a block suitable for migration to */
@@ -1963,7 +1963,8 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc)* reduces the risk that a large movable pageblock is freed for * an unmovable/reclaimable small allocation. */ -if (cc->direct_compaction && cc->migratetype !=MIGRATE_MOVABLE ) +if (cc->direct_compaction && +free_to_migratetype (cc->freetype) !=MIGRATE_MOVABLE )return pfn;/*
@@ -2234,7 +2235,7 @@ static bool should_proactive_compact_node(pg_data_t *pgdat)static enum compact_result __compact_finished (struct compact_control *cc) {unsigned int order; -const int migratetype = cc->migratetype; +const freetype_t freetype = cc->freetype;int ret;/* Compaction run completes if the migrate and free scanner meet */
@@ -2309,25 +2310,27 @@ static enum compact_result __compact_finished(struct compact_control *cc)for (order = cc->order; order <NR_PAGE_ORDERS ; order++) {struct free_area *area = &cc->zone->free_area[order]; -/* Job done if page is free of the right migratetype */ -if (!free_area_empty (area, migratetype)) +/* Job done if page is free of the right freetype */ +if (!free_area_empty (area, freetype))return COMPACT_SUCCESS ;#ifdef CONFIG_CMA /* MIGRATE_MOVABLE can fallback on MIGRATE_CMA */ -if (migratetype ==MIGRATE_MOVABLE && - !free_area_empty (area,MIGRATE_CMA )) +if (free_to_migratetype (freetype) ==MIGRATE_MOVABLE && + !free_area_empty (area,freetype_with_migrate (cc->freetype, +MIGRATE_CMA )))return COMPACT_SUCCESS ;#endif /* * Job done if allocation would steal freepages from -* other migratetype buddy lists. +* other freetype buddy lists. */ -if (find_suitable_fallback (area, order, migratetype, true,NULL ) +if (find_suitable_fallback (area, order, freetype, true,NULL ) ==FALLBACK_FOUND )/* -* Movable pages are OK in any pageblock. If we are -* stealing for a non-movable allocation, make sure +* Movable pages are OK in any pageblock of the right +* sensitivity. If we are * stealing for a +* non-movable allocation, make sure * we finish compacting the current pageblock first * (which is assured by the above migrate_pfn align * check) so it is as free as possible and we won't
@@ -2532,7 +2535,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)INIT_LIST_HEAD (&cc->freepages[order]);INIT_LIST_HEAD (&cc->migratepages); - cc->migratetype =gfp_migratetype (cc->gfp_mask); + cc->freetype =gfp_freetype (cc->gfp_mask);if (!is_via_compact_memory (cc->order)) { ret = compaction_suit_allocation_order(cc->zone, cc->order,
mm/internal.h +9 -8
@@ -10,6 +10,7 @@#include <linux/fs.h> #include <linux/khugepaged.h> #include <linux/mm.h> +#include <linux/mmzone.h> #include <linux/mm_inline.h> #include <linux/pagemap.h> #include <linux/pagewalk.h>
@@ -658,7 +659,7 @@ struct alloc_context {struct zonelist *zonelist;nodemask_t *nodemask;struct zoneref *preferred_zoneref; -int migratetype; +freetype_t freetype;/* * highest_zoneidx represents highest usable zone index of
@@ -809,8 +810,8 @@ static inline void clear_zone_contiguous(struct zone *zone) }extern int __isolate_free_page (struct page *page,unsigned int order); -extern void __putback_isolated_page (struct page *page,unsigned int order, -int mt); +void __putback_isolated_page (struct page *page,unsigned int order, +freetype_t freetype);extern void memblock_free_pages (unsigned long pfn,unsigned int order);extern void __free_pages_core (struct page *page,unsigned int order,enum meminit_context context);
@@ -968,7 +969,7 @@ struct compact_control {short search_order;/* order to start a fast search at */ const gfp_t gfp_mask;/* gfp mask of a direct compactor */ int order;/* order a direct compactor needs */ -int migratetype;/* migratetype of direct compactor */ +freetype_t freetype;/* freetype of direct compactor */ const unsigned int alloc_flags;/* alloc flags of a direct compactor */ const int highest_zoneidx;/* zone index of a direct compactor */ enum migrate_mode mode;/* Async or sync migration mode */
@@ -1029,7 +1030,7 @@ static inline void init_cma_pageblock(struct page *page)#endif enum fallback_result { -/* Found suitable migratetype, *mt_out is valid. */ +/* Found suitable fallback, *ft_out is valid. */ FALLBACK_FOUND ,/* No fallback found in requested order. */ FALLBACK_EMPTY ,
@@ -1038,11 +1039,11 @@ enum fallback_result { };enum fallback_result find_suitable_fallback (struct free_area *area,unsigned int order, -int migratetype,bool claimable,unsigned int *mt_out); +freetype_t freetype,bool claimable,freetype_t *ft_out); -static inline bool free_area_empty (struct free_area *area,int migratetype) +static inline bool free_area_empty (struct free_area *area,freetype_t freetype) { -return list_empty (&area->free_list[migratetype]); +return list_empty (free_area_list (area, freetype)); }/* mm/util.c */
mm/page_alloc.c +239 -149
1,030 lines not on this page · raw message
mm/page_isolation.c +1 -1
@@ -276,7 +276,7 @@ static void unset_migratetype_isolate(struct page *page)WARN_ON_ONCE (!pageblock_unisolate_and_move_free_pages (zone, page)); }else {clear_pageblock_isolate (page); -__putback_isolated_page (page, order,get_pageblock_migratetype (page)); +__putback_isolated_page (page, order,get_pageblock_freetype (page)); } zone->nr_isolate_pageblock--; out:
mm/page_owner.c +4 -3
@@ -481,7 +481,8 @@ void pagetypeinfo_showmixedcount_print(struct seq_file *m, goto ext_put_continue; page_owner =get_page_owner (page_ext); - page_mt =gfp_migratetype (page_owner->gfp_mask); + page_mt =free_to_migratetype ( +gfp_freetype (page_owner->gfp_mask));if (pageblock_mt != page_mt) {if (is_migrate_cma (pageblock_mt)) count[MIGRATE_MOVABLE ]++;
@@ -566,7 +567,7 @@ print_page_owner(char __user *buf, size_t count, unsigned long pfn,/* Print information relevant to grouping pages by mobility */ pageblock_mt =get_pageblock_migratetype (page); - page_mt =gfp_migratetype (page_owner->gfp_mask); + page_mt =free_to_migratetype (gfp_freetype (page_owner->gfp_mask)); ret += scnprintf(kbuf + ret, count - ret, "PFN 0x%lx type %s Block %lu type %s Flags %pGp\n", pfn,
@@ -617,7 +618,7 @@ void __dump_page_owner(const struct page *page) page_owner =get_page_owner (page_ext); gfp_mask = page_owner->gfp_mask; - mt =gfp_migratetype (gfp_mask); + mt =free_to_migratetype (gfp_freetype (gfp_mask));if (!test_bit (PAGE_EXT_OWNER , &page_ext->flags)) {pr_alert ("page_owner info is not present (never set?)\n" );
mm/page_reporting.c +2 -2
@@ -113,10 +113,10 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,*/ do { struct page *page = sg_page(sg); - int mt = get_pageblock_migratetype(page); + freetype_t ft = get_pageblock_freetype(page); unsigned int order = get_order(sg->length); - __putback_isolated_page(page, order, mt); + __putback_isolated_page(page, order, ft); /* If the pages were not reported due to error skip flagging */ if (!reported)
mm/show_mem.c +3 -1
@@ -373,7 +373,9 @@ static void show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_z types[order] =0 ;for (type =0 ; type <MIGRATE_TYPES ; type++) { -if (!free_area_empty (area, type)) +freetype_t ft =migrate_to_freetype (type,0 ); + +if (!free_area_empty (area, ft)) types[order] |=1 << type; } }
Signature · 1 line
2.51.2
RFC PATCH 9/19 mm: move migratetype definitions to freetype.h
In reply to Brendan JackmanSince migratetype are a sub-element of freetype, move the pure definitions into the new freetype.h.
This will enable referring to these raw types from pageblock-flags.h.
--- include/linux/freetype.h | 84 ++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/mmzone.h | 73 ----------------------------------------- 2 files changed, 84 insertions(+), 73 deletions(-)
include/linux/freetype.h +84 -0
@@ -3,6 +3,66 @@#define _LINUX_FREETYPE_H#include <linux/types.h> +#include <linux/mmdebug.h> + +/* +* A migratetype is the part of a freetype that encodes the mobility +* requirements for the allocations the freelist is intended to serve. +* +* It's also currently overloaded to encode page isolation state. +*/ +enum migratetype { +MIGRATE_UNMOVABLE , +MIGRATE_MOVABLE , +MIGRATE_RECLAIMABLE , +MIGRATE_PCPTYPES ,/* the number of types on the pcp lists */ +MIGRATE_HIGHATOMIC =MIGRATE_PCPTYPES , +#ifdef CONFIG_CMA +/* +* MIGRATE_CMA migration type is designed to mimic the way +* ZONE_MOVABLE works. Only movable pages can be allocated +* from MIGRATE_CMA pageblocks and page allocator never +* implicitly change migration type of MIGRATE_CMA pageblock. +* +* The way to use it is to change migratetype of a range of +* pageblocks to MIGRATE_CMA which can be done by +* __free_pageblock_cma() function. +*/ +MIGRATE_CMA , + __MIGRATE_TYPE_END =MIGRATE_CMA , +#else + __MIGRATE_TYPE_END =MIGRATE_HIGHATOMIC , +#endif +#ifdef CONFIG_MEMORY_ISOLATION +MIGRATE_ISOLATE ,/* can't allocate from here */ +#endif +MIGRATE_TYPES +}; + +/* In mm/page_alloc.c; keep in sync also with show_migration_types() there */ +extern const char *const migratetype_names[MIGRATE_TYPES ]; + +#ifdef CONFIG_CMA +# define is_migrate_cma (migratetype) unlikely((migratetype) == MIGRATE_CMA) +#else +# define is_migrate_cma (migratetype) false +#endif + +static inline bool is_migrate_movable (int mt) +{ +return is_migrate_cma (mt) || mt ==MIGRATE_MOVABLE ; +} + +/* +* Check whether a migratetype can be merged with another migratetype. +* +* It is only mergeable when it can fall back to other migratetypes for +* allocation. See fallbacks[MIGRATE_TYPES][3] in page_alloc.c. +*/ +static inline bool migratetype_is_mergeable (int mt) +{ +return mt <MIGRATE_PCPTYPES ; +}/* * A freetype is the index used to identify free lists. This consists of a
@@ -35,4 +95,28 @@ static inline bool freetypes_equal(freetype_t a, freetype_t b) return a.migratetype == b.migratetype; } +static inline freetype_t migrate_to_freetype (enum migratetype mt, +unsigned int flags) +{ +freetype_t freetype; + +/* No flags supported yet. */ +VM_WARN_ON_ONCE (flags); + + freetype.migratetype = mt; +return freetype; +} + +static inline enum migratetype free_to_migratetype (freetype_t freetype) +{ +return freetype.migratetype; +} + +/* Convenience helper, return the freetype modified to have the migratetype. */ +static inline freetype_t freetype_with_migrate (freetype_t freetype, +enum migratetype migratetype) +{ +return migrate_to_freetype (migratetype,freetype_flags (freetype)); +} +#endif /* _LINUX_FREETYPE_H */
include/linux/mmzone.h +0 -73
96 lines not on this page · raw message
Signature · 1 line
2.51.2
RFC PATCH 10/19 mm: add definitions for allocating unmapped pages
In reply to Brendan JackmanCreate __GFP_UNMAPPED, which requests pages that are not present in the direct map. Since this feature has a cost (e.g. more freelists), it's behind a kconfig. Unlike other conditionally-defined GFP flags, it doesn't fall back to being 0. This prevents building code that uses __GFP_UNMAPPED but doesn't depend on the necessary kconfig, since that would lead to invisible security issues.
Create a freetype flag to record that pages on the freelists with this flag are unmapped. This is currently only needed for MIGRATE_UNMOVABLE pages, so the freetype encoding remains trivial.
Also create the corresponding pageblock flag to record the same thing.
To keep patches from being too overwhelming, the actual implementation is added separately, this is just types, Kconfig boilerplate, etc.
--- include/linux/freetype.h | 41 +++++++++++++++++++++++++++++++++-------- include/linux/gfp_types.h | 26 ++++++++++++++++++++++++++ include/trace/events/mmflags.h | 9 ++++++++- mm/Kconfig | 4 ++++ 4 files changed, 71 insertions(+), 9 deletions(-)
include/linux/freetype.h +33 -8
@@ -2,6 +2,7 @@#ifndef _LINUX_FREETYPE_H#define _LINUX_FREETYPE_H +#include <linux/log2.h> #include <linux/types.h> #include <linux/mmdebug.h>
@@ -64,30 +65,56 @@ static inline bool migratetype_is_mergeable(int mt)return mt <MIGRATE_PCPTYPES ; } +enum { +/* Defined unconditionally as a hack to avoid a zero-width bitfield. */ +FREETYPE_UNMAPPED_BIT , +NUM_FREETYPE_FLAGS , +}; +/* * A freetype is the index used to identify free lists. This consists of a * migratetype, and other bits which encode orthogonal properties of memory. */ typedef struct { -int migratetype; +unsigned int migratetype :order_base_2 (MIGRATE_TYPES ); +unsigned int flags :NUM_FREETYPE_FLAGS ; }freetype_t ; +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +#define FREETYPE_UNMAPPED BIT(FREETYPE_UNMAPPED_BIT) +#define NUM_UNMAPPED_FREETYPES 1 +#else +#define FREETYPE_UNMAPPED 0 +#define NUM_UNMAPPED_FREETYPES 0 +#endif +/* * Return a dense linear index for freetypes that have lists in the free area. * Return -1 for other freetypes. */ static inline int freetype_idx (freetype_t freetype) { +/* For FREETYPE_UNMAPPED, only MIGRATE_UNMOVABLE has an index. */ +if (freetype.flags &FREETYPE_UNMAPPED ) { +VM_WARN_ON_ONCE (freetype.flags & ~FREETYPE_UNMAPPED ); +if (!IS_ENABLED (CONFIG_PAGE_ALLOC_UNMAPPED )) +return -1 ; +if (freetype.migratetype !=MIGRATE_UNMOVABLE ) +return -1 ; +return MIGRATE_TYPES ; + } +/* No other flags are supported. */ +VM_WARN_ON_ONCE (freetype.flags); +return freetype.migratetype; } -/* No freetype flags actually exist yet. */ -#define NR_FREETYPE_IDXS MIGRATE_TYPES +/* One for each migratetype, plus one for MIGRATE_UNMOVABLE-FREETYPE_UNMAPPED */ +#define NR_FREETYPE_IDXS (MIGRATE_TYPES + NUM_UNMAPPED_FREETYPES)static inline unsigned int freetype_flags (freetype_t freetype) { -/* No flags supported yet. */ -return 0 ; +return freetype.flags; }static inline bool freetypes_equal (freetype_t a,freetype_t b)
@@ -100,10 +127,8 @@ static inline freetype_t migrate_to_freetype(enum migratetype mt, {freetype_t freetype; -/* No flags supported yet. */ -VM_WARN_ON_ONCE (flags); - freetype.migratetype = mt; + freetype.flags = flags;return freetype; }
include/linux/gfp_types.h +26 -0
@@ -56,6 +56,9 @@ enum { ___GFP_NOLOCKDEP_BIT, #endif ___GFP_NO_OBJ_EXT_BIT, +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED + ___GFP_UNMAPPED_BIT, +#endif ___GFP_LAST_BIT };
@@ -97,6 +100,10 @@ enum {#define ___GFP_NOLOCKDEP 0#endif #define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT) +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +#define ___GFP_UNMAPPED BIT(___GFP_UNMAPPED_BIT) +/* No #else - __GFP_UNMAPPED should never be a nop. Break the build if it isn't supported. */ +#endif /* * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -293,6 +300,25 @@ enum {/* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) +/* +* Allocate pages that aren't present in the direct map. If the caller changes +* direct map presence, it must be restored to the previous state before freeing +* the page. (This is true regardless of __GFP_UNMAPPED). +* +* This uses the mermap (when __GFP_ZERO), so it's only valid to allocate with +* this flag where that's valid, namely from process context after the mermap +* has been initialised for that process. This also means that the allocator +* leaves behind stale TLB entries in the mermap region. The caller is +* responsible for ensuring they are flushed as needed. +* +* This is currently incompatible with __GFP_MOVABLE and __GFP_RECLAIMABLE, but +* only because of allocator implementation details, if a usecase arises this +* restriction could be dropped. +*/ +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +#define __GFP_UNMAPPED ((__force gfp_t)___GFP_UNMAPPED) +#endif +/* Room for N __GFP_FOO bits */ #define __GFP_BITS_SHIFT ___GFP_LAST_BIT#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
include/trace/events/mmflags.h +8 -1
@@ -61,11 +61,18 @@# define TRACE_GFP_FLAGS_SLAB #endif +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +# define TRACE_GFP_FLAGS_UNMAPPED TRACE_GFP_EM(UNMAPPED) +#else +# define TRACE_GFP_FLAGS_UNMAPPED +#endif +#define TRACE_GFP_FLAGS \ TRACE_GFP_FLAGS_GENERAL \ TRACE_GFP_FLAGS_KASAN \ TRACE_GFP_FLAGS_LOCKDEP \ - TRACE_GFP_FLAGS_SLAB + TRACE_GFP_FLAGS_SLAB \ + TRACE_GFP_FLAGS_UNMAPPED#undef TRACE_GFP_EM#define TRACE_GFP_EM (a) TRACE_DEFINE_ENUM(___GFP_##a##_BIT);
mm/Kconfig +4 -0
@@ -1503,3 +1503,7 @@ config MERMAP_KUNIT_TEST If unsure, say N. endmenu + +config PAGE_ALLOC_UNMAPPED + bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST + default COMPILE_TEST
Signature · 1 line
2.51.2
RFC PATCH 11/19 mm: rejig pageblock mask definitions
In reply to Brendan JackmanA later patch will complicate the definition of these masks, this is a preparatory patch to make that patch easier to review.
- More masks will be needed, so add a PAGEBLOCK_ prefix to the names to avoid polluting the "global namespace" too much.
- This makes MIGRATETYPE_AND_ISO_MASK start to look pretty long. Well, that global mask only exists for quite a specific purpose so just drop it and take advantage of the newly-defined PAGEBLOCK_ISO_MASK.
--- include/linux/pageblock-flags.h | 6 +++--- mm/page_alloc.c | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-)
include/linux/pageblock-flags.h +3 -3
@@ -36,12 +36,12 @@ enum pageblock_bits {#define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS)) -#define MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2)) +#define PAGEBLOCK_MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2))#ifdef CONFIG_MEMORY_ISOLATION -#define MIGRATETYPE_AND_ISO_MASK (MIGRATETYPE_MASK | BIT(PB_migrate_isolate)) +#define PAGEBLOCK_ISO_MASK BIT(PB_migrate_isolate)#else -#define MIGRATETYPE_AND_ISO_MASK MIGRATETYPE_MASK +#define PAGEBLOCK_ISO_MASK 0#endif #if defined(CONFIG_HUGETLB_PAGE )
mm/page_alloc.c +6 -6
@@ -397,7 +397,7 @@ get_pfnblock_bitmap_bitidx(const struct page *page, unsigned long pfn,#else BUILD_BUG_ON (NR_PAGEBLOCK_BITS !=4 );#endif -BUILD_BUG_ON (__MIGRATE_TYPE_END >MIGRATETYPE_MASK ); +BUILD_BUG_ON (__MIGRATE_TYPE_END >PAGEBLOCK_MIGRATETYPE_MASK );VM_BUG_ON_PAGE (!zone_spans_pfn (page_zone (page), pfn), page); bitmap =get_pageblock_bitmap (page, pfn);
@@ -501,7 +501,7 @@ get_pfnblock_freetype(const struct page *page, unsigned long pfn) __always_inline enum migratetypeget_pfnblock_migratetype (const struct page *page,unsigned long pfn) { -unsigned long mask =MIGRATETYPE_AND_ISO_MASK ; +unsigned long mask =PAGEBLOCK_MIGRATETYPE_MASK |PAGEBLOCK_ISO_MASK ;unsigned long flags; flags =__get_pfnblock_flags_mask (page, pfn, mask);
@@ -510,7 +510,7 @@ get_pfnblock_migratetype(const struct page *page, unsigned long pfn)if (flags &BIT (PB_migrate_isolate))return MIGRATE_ISOLATE ;#endif -return flags &MIGRATETYPE_MASK ; +return flags &PAGEBLOCK_MIGRATETYPE_MASK ; }/**
@@ -598,11 +598,11 @@ static void set_pageblock_migratetype(struct page *page, }VM_WARN_ONCE (get_pageblock_isolate (page),"Use clear_pageblock_isolate() to unisolate pageblock" ); -/* MIGRATETYPE_AND_ISO_MASK clears PB_migrate_isolate if it is set */ +/* PAGEBLOCK_ISO_MASK clears PB_migrate_isolate if it is set */ #endif __set_pfnblock_flags_mask (page,page_to_pfn (page), (unsigned long )migratetype, -MIGRATETYPE_AND_ISO_MASK ); +PAGEBLOCK_MIGRATETYPE_MASK |PAGEBLOCK_ISO_MASK ); } void __meminit init_pageblock_migratetype(struct page *page,
@@ -628,7 +628,7 @@ void __meminit init_pageblock_migratetype(struct page *page, flags |=BIT (PB_migrate_isolate);#endif __set_pfnblock_flags_mask (page,page_to_pfn (page), flags, -MIGRATETYPE_AND_ISO_MASK ); +PAGEBLOCK_MIGRATETYPE_MASK |PAGEBLOCK_ISO_MASK ); }#ifdef CONFIG_DEBUG_VM
Signature · 1 line
2.51.2
RFC PATCH 12/19 mm: encode freetype flags in pageblock flags
In reply to Brendan JackmanIn preparation for implementing allocation from FREETYPE_UNMAPPED lists.
Since it works nicely with the existing allocator logic, and also offers a simple way to amortize TLB flushing costs, __GFP_UNMAPPED will be implemented by changing mappings at pageblock granularity. Therefore, encode the mapping state in the pageblock flags.
Also add the necessary logic to record this from a freetype, and reconstruct a freetype from the pageblock flags.
--- include/linux/pageblock-flags.h | 10 ++++++++++ mm/page_alloc.c | 33 +++++++++++++++++++++++++-------- 2 files changed, 35 insertions(+), 8 deletions(-)
include/linux/pageblock-flags.h +10 -0
@@ -11,6 +11,8 @@#ifndef PAGEBLOCK_FLAGS_H #define PAGEBLOCK_FLAGS_H +#include <linux/freetype.h> +#include <linux/log2.h> #include <linux/types.h> /* Bit indices that affect a whole block of pages */
@@ -18,6 +20,9 @@ enum pageblock_bits { PB_migrate_0, PB_migrate_1, PB_migrate_2, + PB_freetype_flags, + PB_freetype_flags_end = PB_freetype_flags +NUM_FREETYPE_FLAGS -1 , + PB_compact_skip,/* If set the block is skipped by compaction */ #ifdef CONFIG_MEMORY_ISOLATION
@@ -37,6 +42,7 @@ enum pageblock_bits {#define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS))#define PAGEBLOCK_MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2)) +#define PAGEBLOCK_FREETYPE_FLAGS_MASK (((1 << NUM_FREETYPE_FLAGS) - 1) << PB_freetype_flags)#ifdef CONFIG_MEMORY_ISOLATION #define PAGEBLOCK_ISO_MASK BIT(PB_migrate_isolate)
@@ -44,6 +50,10 @@ enum pageblock_bits {#define PAGEBLOCK_ISO_MASK 0#endif +#define PAGEBLOCK_FREETYPE_MASK (PAGEBLOCK_MIGRATETYPE_MASK | \ + PAGEBLOCK_ISO_MASK | \ + PAGEBLOCK_FREETYPE_FLAGS_MASK) +#if defined(CONFIG_HUGETLB_PAGE )#ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE
mm/page_alloc.c +25 -8
@@ -392,11 +392,8 @@ get_pfnblock_bitmap_bitidx(const struct page *page, unsigned long pfn,unsigned long *bitmap;unsigned long word_bitidx; -#ifdef CONFIG_MEMORY_ISOLATION -BUILD_BUG_ON (NR_PAGEBLOCK_BITS !=8 ); -#else -BUILD_BUG_ON (NR_PAGEBLOCK_BITS !=4 ); -#endif +/* NR_PAGEBLOCK_BITS must divide word size. */ +BUILD_BUG_ON (NR_PAGEBLOCK_BITS !=4 &&NR_PAGEBLOCK_BITS !=8 );BUILD_BUG_ON (__MIGRATE_TYPE_END >PAGEBLOCK_MIGRATETYPE_MASK );VM_BUG_ON_PAGE (!zone_spans_pfn (page_zone (page), pfn), page);
@@ -469,9 +466,20 @@ __always_inline freetype_t __get_pfnblock_freetype(const struct page *page, unsigned long pfn, bool ignore_iso) { - int mt = get_pfnblock_migratetype(page, pfn); + unsigned long mask = PAGEBLOCK_FREETYPE_MASK; +enum migratetype migratetype; +unsigned int ft_flags; +unsigned long flags; -return migrate_to_freetype (mt,0 ); + flags =__get_pfnblock_flags_mask (page, pfn, mask); + ft_flags = (flags &PAGEBLOCK_FREETYPE_FLAGS_MASK ) >> PB_freetype_flags; + + migratetype = flags &PAGEBLOCK_MIGRATETYPE_MASK ; +#ifdef CONFIG_MEMORY_ISOLATION +if (!ignore_iso && flags &BIT (PB_migrate_isolate)) + migratetype =MIGRATE_ISOLATE ; +#endif +return migrate_to_freetype (migratetype, ft_flags); }/**
@@ -605,6 +613,15 @@ static void set_pageblock_migratetype(struct page *page, PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK); } +static inline void set_pageblock_freetype_flags (struct page *page, +unsigned int ft_flags) +{ +unsigned int flags = ft_flags << PB_freetype_flags; + +__set_pfnblock_flags_mask (page,page_to_pfn (page), flags, +PAGEBLOCK_FREETYPE_FLAGS_MASK ); +} +void __meminit init_pageblock_migratetype (struct page *page,enum migratetype migratetype,bool isolate)
@@ -628,7 +645,7 @@ void __meminit init_pageblock_migratetype(struct page *page, flags |=BIT (PB_migrate_isolate);#endif __set_pfnblock_flags_mask (page,page_to_pfn (page), flags, -PAGEBLOCK_MIGRATETYPE_MASK |PAGEBLOCK_ISO_MASK ); +PAGEBLOCK_FREETYPE_MASK ); }#ifdef CONFIG_DEBUG_VM
Signature · 1 line
2.51.2
RFC PATCH 13/19 mm/page_alloc: remove ifdefs from pindex helpers
In reply to Brendan JackmanThe ifdefs are not technically needed here, everything used here is always defined.
They aren't doing much harm right now but a following patch will complicate these functions. Switching to IS_ENABLED() makes the code a bit less tiresome to read.
--- mm/page_alloc.c | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-)
mm/page_alloc.c +14 -16
@@ -731,19 +731,17 @@ static void bad_page(struct page *page, const char *reason)static inline unsigned int order_to_pindex (int migratetype,int order) { +if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) { +bool movable = migratetype ==MIGRATE_MOVABLE ; -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -bool movable; -if (order >PAGE_ALLOC_COSTLY_ORDER ) { -VM_BUG_ON (order !=HPAGE_PMD_ORDER ); +if (order >PAGE_ALLOC_COSTLY_ORDER ) { +VM_BUG_ON (order !=HPAGE_PMD_ORDER ); - movable = migratetype ==MIGRATE_MOVABLE ; - -return NR_LOWORDER_PCP_LISTS + movable; +return NR_LOWORDER_PCP_LISTS + movable; + } + }else { +VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); } -#else -VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); -#endif return (MIGRATE_PCPTYPES * order) + migratetype; }
@@ -752,12 +750,12 @@ static inline int pindex_to_order(unsigned int pindex) {int order = pindex /MIGRATE_PCPTYPES ; -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (pindex >=NR_LOWORDER_PCP_LISTS ) - order =HPAGE_PMD_ORDER ; -#else -VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); -#endif +if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) { +if (pindex >=NR_LOWORDER_PCP_LISTS ) + order =HPAGE_PMD_ORDER ; + }else { +VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); + }return order; }
Signature · 1 line
2.51.2
RFC PATCH 14/19 mm/page_alloc: separate pcplists by freetype flags
In reply to Brendan JackmanThe normal freelists are already separated by this flag, so now update the pcplists accordingly. This follows the most "obvious" design where __GFP_UNMAPPED is supported at arbitrary orders.
If necessary, it would be possible to avoid the proliferation of pcplists by restricting orders that can be allocated from them with this FREETYPE_UNMAPPED.
On the other hand, there's currently no usecase for movable/reclaimable unmapped memory, and constraining the migratetype doesn't have any tricky plumbing implications. So, take advantage of that and assume that FREETYPE_UNMAPPED implies MIGRATE_UNMOVABLE.
Overall, this just takes the existing space of pindices and tacks another bank on the end. For !THP this is just 4 more lists, with THP there is a single additional list for hugepages.
--- include/linux/mmzone.h | 11 ++++++++++- mm/page_alloc.c | 44 +++++++++++++++++++++++++++++++++----------- 2 files changed, 43 insertions(+), 12 deletions(-)
include/linux/mmzone.h +10 -1
@@ -692,8 +692,17 @@ enum zone_watermarks {#else #define NR_PCP_THP 0#endif +/* +* FREETYPE_UNMAPPED can currently only be used with MIGRATE_UNMOVABLE, no for +* those there's no need to encode the migratetype in the pindex. +*/ +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +#define NR_UNMAPPED_PCP_LISTS (PAGE_ALLOC_COSTLY_ORDER + 1 + !!NR_PCP_THP) +#else +#define NR_UNMAPPED_PCP_LISTS 0 +#endif #define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1)) -#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP) +#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP + NR_UNMAPPED_PCP_LISTS)/* * Flags used in pcp->flags field.
mm/page_alloc.c +33 -11
80 lines not on this page · raw message
Signature · 1 line
2.51.2
RFC PATCH 15/19 mm/page_alloc: rename ALLOC_NON_BLOCK back to _HARDER
In reply to Brendan JackmanCommit 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves") renamed ALLOC_HARDER to ALLOC_NON_BLOCK because the former is "a vague description".
However, vagueness is accurate here, this is a vague flag. It is not set for __GFP_NOMEMALLOC. It doesn't really mean "allocate without blocking" but rather "allow dipping into atomic reserves, _because_ of the need not to block".
A later commit will need an alloc flag that really means "don't block here", so go back to the flag's old name and update the commentary to try and give it a slightly clearer meaning.
--- mm/internal.h | 9 +++++---- mm/page_alloc.c | 8 ++++---- 2 files changed, 9 insertions(+), 8 deletions(-)
mm/internal.h +5 -4
@@ -1372,9 +1372,10 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_OOM ALLOC_NO_WATERMARKS#endif -#define ALLOC_NON_BLOCK 0x10/* Caller cannot block. Allow access -* to 25% of the min watermark or -* 62.5% if __GFP_HIGH is set. +#define ALLOC_HARDER 0x10/* Because the caller cannot block, +* allow access * to 25% of the min +* watermark or 62.5% if __GFP_HIGH is +* set. */ #define ALLOC_MIN_RESERVE 0x20/* __GFP_HIGH set. Allow access to 50% * of the min watermark.
@@ -1391,7 +1392,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ -#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) +#define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)enum ttu_flags ;struct tlbflush_unmap_batch ;
mm/page_alloc.c +4 -4
@@ -3392,7 +3392,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,* reserves as failing now is worse than failing a * high-order atomic allocation in the future. */ -if (!page && (alloc_flags & (ALLOC_OOM |ALLOC_NON_BLOCK ))) +if (!page && (alloc_flags & (ALLOC_OOM |ALLOC_HARDER ))) page =__rmqueue_smallest (zone, order, ft_high);if (!page) {
@@ -3755,7 +3755,7 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,* or (GFP_KERNEL & ~__GFP_DIRECT_RECLAIM) do not get * access to the min reserve. */ -if (alloc_flags &ALLOC_NON_BLOCK ) +if (alloc_flags &ALLOC_HARDER ) min -= min /4 ; }
@@ -4640,7 +4640,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)* The caller may dip into page reserves a bit more if the caller * cannot run direct reclaim, or if the caller has realtime scheduling * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will -* set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). +* set both ALLOC_HARDER and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |= (__force int ) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM));
@@ -4651,7 +4651,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)* if it can't schedule. */ if (!(gfp_mask & __GFP_NOMEMALLOC)) { - alloc_flags |=ALLOC_NON_BLOCK ; + alloc_flags |=ALLOC_HARDER ;if (order >0 && (alloc_flags &ALLOC_MIN_RESERVE )) alloc_flags |=ALLOC_HIGHATOMIC ;
Signature · 1 line
2.51.2
RFC PATCH 16/19 mm/page_alloc: introduce ALLOC_NOBLOCK
In reply to Brendan JackmanThis flag is set unless we can be sure the caller isn't in an atomic context.
The allocator will soon start needing to call set_direct_map_* APIs which cannot be called with IRQs off. It will need to do this even before direct reclaim is possible.
Despite the fact that, in principle, ALLOC_NOBLOCK is distinct from __GFP_DIRECT_RECLAIM, in order to avoid introducing a GFP flag, just infer the former based on whether the caller set the latter. This means that, in practice, ALLOC_NOBLOCK is just !__GFP_DIRECT_RECLAIM, except that it is not influenced by gfp_allowed_mask. This could change later, though.
Call it ALLOC_NOBLOCK in order to try and mitigate confusion vs the recently-removed ALLOC_NON_BLOCK, which meant something different.
--- mm/internal.h | 1 + mm/page_alloc.c | 29 ++++++++++++++++++++++------- 2 files changed, 23 insertions(+), 7 deletions(-)
mm/internal.h +1 -0
@@ -1390,6 +1390,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_TRYLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#define ALLOC_NOBLOCK 0x1000/* Caller may be atomic */ /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
mm/page_alloc.c +22 -7
@@ -4646,6 +4646,8 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM));if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) { + alloc_flags |=ALLOC_NOBLOCK ; +/* * Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule.
@@ -4839,14 +4841,13 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)static inline struct page *__alloc_pages_slowpath (gfp_t gfp_mask,unsigned int order, -struct alloc_context *ac) +struct alloc_context *ac,unsigned int alloc_flags) {bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;bool can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask);bool nofail = gfp_mask & __GFP_NOFAIL;const bool costly_order = order >PAGE_ALLOC_COSTLY_ORDER ;struct page *page =NULL ; -unsigned int alloc_flags;unsigned long did_some_progress;enum compact_priority compact_priority;enum compact_result compact_result;
@@ -4898,7 +4899,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* kswapd needs to be woken up, and to avoid the cost of setting up * alloc_flags precisely. So we do that now. */ - alloc_flags =gfp_to_alloc_flags (gfp_mask, order); + alloc_flags |=gfp_to_alloc_flags (gfp_mask, order);/* * We need to recalculate the starting point for the zonelist iterator
@@ -5124,6 +5125,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,return page; } +static inline unsigned int init_alloc_flags (gfp_t gfp_mask,unsigned int flags) +{ +/* +* If the caller allowed __GFP_DIRECT_RECLAIM, they can't be atomic. +* Note this is a separate determination from whether direct reclaim is +* actually allowed, it must happen before applying gfp_allowed_mask. +*/ +if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) + flags |=ALLOC_NOBLOCK ; +return flags; +} + static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, int preferred_nid, nodemask_t *nodemask, struct alloc_context *ac, gfp_t *alloc_gfp,
@@ -5205,7 +5218,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,struct list_head *pcp_list;struct alloc_context ac;gfp_t alloc_gfp; -unsigned int alloc_flags =ALLOC_WMARK_LOW ; +unsigned int alloc_flags =init_alloc_flags (gfp,ALLOC_WMARK_LOW );int nr_populated =0 , nr_account =0 ;/*
@@ -5346,7 +5359,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { struct page *page; -unsigned int alloc_flags =ALLOC_WMARK_LOW ; +unsigned int alloc_flags =init_alloc_flags (gfp,ALLOC_WMARK_LOW );gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ struct alloc_context ac = { };
@@ -5391,7 +5404,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,*/ ac.nodemask = nodemask; - page =__alloc_pages_slowpath (alloc_gfp, order, &ac); + page =__alloc_pages_slowpath (alloc_gfp, order, &ac, alloc_flags); out: if (memcg_kmem_online() && (gfp & __GFP_ACCOUNT) && page &&
@@ -7911,11 +7924,13 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned*/ gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | __GFP_COMP | gfp_flags; -unsigned int alloc_flags =ALLOC_TRYLOCK ; +unsigned int alloc_flags =init_alloc_flags (alloc_gfp,ALLOC_TRYLOCK );struct alloc_context ac = { };struct page *page;VM_WARN_ON_ONCE (gfp_flags & ~__GFP_ACCOUNT); +VM_WARN_ON_ONCE (!(alloc_flags &ALLOC_NOBLOCK )); +/* * In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is * unsafe in NMI. If spin_trylock() is called from hard IRQ the current
Signature · 1 line
2.51.2
RFC PATCH 17/19 mm/page_alloc: implement __GFP_UNMAPPED allocations
In reply to Brendan JackmanCurrently __GFP_UNMAPPED allocs will always fail because, although the lists exist to hold them, there is no way to actually create an unmapped page block. This commit adds one, and also the logic to map it back again when that's needed.
Doing this at pageblock granularity ensures that the pageblock flags can be used to infer which freetype a page belongs to. It also provides nice batching of TLB flushes, and also avoids creating too much unnecessary TLB fragmentation in the physmap.
There are some functional requirements for flipping a block:
- Unmapping requires a TLB shootdown, meaning IRQs must be enabled.
- Because the main usecase of this feature is to protect against CPU exploits, when a block is mapped it needs to be zeroed to ensure no residual data is available to attackers. Zeroing a block with a spinlock held seems undesirable.
- Updating the pagetables might require allocating a pagetable to break down a huge page. This would deadlock if the zone lock was held.
This makes allocations that need to change sensitivity _somewhat_ similar to those that need to fallback to a different migratetype. But, the locking requirements mean that this can't just be squashed into the existing "fallback" allocator logic, instead a new allocator path just for this purpose is needed.
The new path is assumed to be much cheaper than the really heavyweight stuff like compaction and reclaim. But at present it is treated as less desirable than the mobility-related "fallback" and "stealing" logic. This might turn out to need revision (in particular, maybe it's a problem that __rmqueue_steal(), which causes fragmentation, happens before __rmqueue_direct_map()), but that should be treated as a subsequent optimisation project.
This currently forbids __GFP_ZERO, this is just to keep the patch from getting too large, the next patch will remove this restriction.
--- include/linux/gfp.h | 11 +++- mm/Kconfig | 4 +- mm/page_alloc.c | 163 ++++++++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 164 insertions(+), 14 deletions(-)
include/linux/gfp.h +10 -1
@@ -20,6 +20,7 @@ struct mempolicy;static inline freetype_t gfp_freetype (const gfp_t gfp_flags) {int migratetype; +unsigned int ft_flags =0 ;VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK );BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE);
@@ -36,7 +37,15 @@ static inline freetype_t gfp_freetype(const gfp_t gfp_flags) >> GFP_MOVABLE_SHIFT; } -return migrate_to_freetype (migratetype,0 ); +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +if (gfp_flags & __GFP_UNMAPPED) { +if (WARN_ON_ONCE (migratetype !=MIGRATE_UNMOVABLE )) + migratetype =MIGRATE_UNMOVABLE ; + ft_flags |=FREETYPE_UNMAPPED ; + } +#endif + +return migrate_to_freetype (migratetype, ft_flags); }#undef GFP_MOVABLE_MASK#undef GFP_MOVABLE_SHIFT
mm/Kconfig +2 -2
@@ -1502,8 +1502,8 @@ config MERMAP_KUNIT_TEST If unsure, say N. -endmenu - config PAGE_ALLOC_UNMAPPED bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST default COMPILE_TEST + +endmenu
mm/page_alloc.c +152 -11
225 lines not on this page · raw message
Signature · 1 line
2.51.2
Re: [PATCH RFC 17/19] mm/page_alloc: implement __GFP_UNMAPPED allocations
More bugs found by AI...
(I am not relaying all of the issues, just the interesting ones/ones in the most interesting bits of code).
Quoted 61 lines · Brendan Jackman
mm/page_alloc.c +61
-0+/* Try to allocate a page by mapping/unmapping a block from the direct map. */ +static inline struct page * +__rmqueue_direct_map (struct zone *zone,unsigned int request_order, +unsigned int alloc_flags,freetype_t freetype) +{ +unsigned int ft_flags_other =freetype_flags (freetype) ^FREETYPE_UNMAPPED ; +freetype_t ft_other =migrate_to_freetype (free_to_migratetype (freetype), + ft_flags_other); +bool want_mapped = !(freetype_flags (freetype) &FREETYPE_UNMAPPED ); +enum rmqueue_mode rmqm =RMQUEUE_NORMAL ; +unsigned long irq_flags; +int nr_pageblocks; +struct page *page; +int alloc_order; +int err; + +if (freetype_idx (ft_other) <0 ) +return NULL ; + +/* +* Might need a TLB shootdown. Even if IRQs are on this isn't +* safe if the caller holds a lock (in case the other CPUs need that +* lock to handle the shootdown IPI). +*/ +if (alloc_flags &ALLOC_NOBLOCK ) +return NULL ; + +if (!can_set_direct_map ()) +return NULL ; + +lockdep_assert (!irqs_disabled () ||unlikely (early_boot_irqs_disabled)); + +/* +* Need to [un]map a whole pageblock (otherwise it might require +* allocating pagetables). First allocate it. +*/ + alloc_order =max (request_order, pageblock_order); + nr_pageblocks =1 << (alloc_order - pageblock_order); +spin_lock_irqsave (&zone->lock, irq_flags); + page =__rmqueue (zone, alloc_order, ft_other, alloc_flags, &rmqm); +spin_unlock_irqrestore (&zone->lock, irq_flags); +if (!page) +return NULL ; + +/* +* Now that IRQs are on it's safe to do a TLB shootdown, and now that we +* released the zone lock it's possible to allocate a pagetable if +* needed to split up a huge page. +* +* Note that modifying the direct map may need to allocate pagetables. +* What about unbounded recursion? Here are the assumptions that make it +* safe: +* +* - The direct map starts out fully mapped at boot. (This is not really +* an assumption" as its in direct control of page_alloc.c). +* +* - Once pages in the direct map are broken down, they are not +* re-aggregated into larger pages again. +* +* - Pagetables are never allocated with __GFP_UNMAPPED. +*
mm/page_alloc.c static inline void zone_statistics(struct zone *preferred_zone, struct zone *z, · line 3456
Quoted 10 lines
mm/page_alloc.c +10
-0+* Under these assumptions, a pagetable might need to be allocated while +* _unmapping_ stuff from the direct map during a __GFP_UNMAPPED +* allocation. But, the allocation of that pagetable never requires +* allocating a further pagetable. +*/ + err =set_direct_map_valid_noflush (page, + nr_pageblocks << pageblock_order, want_mapped); +if (err == -ENOMEM ||WARN_ONCE (err,"err=%d\n" , err)) { +__free_one_page (page,page_to_pfn (page), zone, + alloc_order, freetype,FPI_SKIP_REPORT_NOTIFY );
Forgot to take the zone lock.
Quoted 31 lines
mm/page_alloc.c +31
-0+return NULL ; + } + +if (!want_mapped) { +unsigned long start = (unsigned long )page_address (page); +unsigned long end = start + (nr_pageblocks << (pageblock_order +PAGE_SHIFT )); + +flush_tlb_kernel_range (start, end); + } + +for (int i =0 ; i < nr_pageblocks; i++) { +struct page *block_page = page + (pageblock_nr_pages * i); + +set_pageblock_freetype_flags (block_page,freetype_flags (freetype)); + } + +if (request_order >= alloc_order) +return page; + +/* Free any remaining pages in the block. */ +spin_lock_irqsave (&zone->lock, irq_flags); +for (unsigned int i = request_order; i < alloc_order; i++) { +struct page *page_to_free = page + (1 << i); + +__free_one_page (page_to_free,page_to_pfn (page_to_free), zone, + i, freetype,FPI_SKIP_REPORT_NOTIFY ); + } +spin_unlock_irqrestore (&zone->lock, irq_flags); + +return page; +}
RFC PATCH 18/19 mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations
In reply to Brendan JackmanThe pages being zeroed here are unmapped, so they can't be zeroed via the direct map. Temporarily mapping them in the direct map is not possible because:
- In general this requires allocating pagetables,
- Unmapping them would require a TLB shootdown, which can't be done in general from the allocator (x86 requires IRQs on).
Therefore, use the new mermap mechanism to zero these pages.
The main mermap API is expected to fail very often. In order to avoid needing to fail allocations when that happens, instead fallback to the special mermap_get_reserved() variant, which is less efficient.
--- arch/x86/include/asm/pgtable_types.h | 2 + mm/Kconfig | 12 +++++- mm/page_alloc.c | 76 +++++++++++++++++++++++++++++++----- 3 files changed, 79 insertions(+), 11 deletions(-)
arch/x86/include/asm/pgtable_types.h +2 -0
@@ -223,6 +223,7 @@ enum page_cache_mode {#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G)#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G)#define __PAGE_KERNEL (__PP|__RW| 0|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_NOGLOBAL (__PP|__RW| 0|___A|__NX|___D| 0| 0)#define __PAGE_KERNEL_EXEC (__PP|__RW| 0|___A| 0|___D| 0|___G)#define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC)#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G)
@@ -245,6 +246,7 @@ enum page_cache_mode {#define __pgprot_mask (x) __pgprot((x) & __default_kernel_pte_mask)#define PAGE_KERNEL __pgprot_mask(__PAGE_KERNEL | _ENC) +#define PAGE_KERNEL_NOGLOBAL __pgprot_mask(__PAGE_KERNEL_NOGLOBAL | _ENC)#define PAGE_KERNEL_NOENC __pgprot_mask(__PAGE_KERNEL | 0)#define PAGE_KERNEL_RO __pgprot_mask(__PAGE_KERNEL_RO | _ENC)#define PAGE_KERNEL_EXEC __pgprot_mask(__PAGE_KERNEL_EXEC | _ENC)
mm/Kconfig +10 -2
@@ -1503,7 +1503,15 @@ config MERMAP_KUNIT_TEST If unsure, say N. config PAGE_ALLOC_UNMAPPED - bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST - default COMPILE_TEST + bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST || KUNIT + default COMPILE_TEST || KUNIT + depends on MERMAP + +config PAGE_ALLOC_KUNIT_TESTS + tristate "KUnit tests for the page allocator" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Builds KUnit tests for the page allocator. endmenu
mm/page_alloc.c +67 -9
96 lines not on this page · raw message
Signature · 1 line
2.51.2
Re: [PATCH RFC 18/19] mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations
Quoted 13 lines · Brendan Jackman
mm/page_alloc.c +13
-0+static inline void clear_page_mermap (struct page *page,unsigned int numpages) +{ +void *mermap; + +BUILD_BUG_ON (IS_ENABLED (CONFIG_HIGHMEM )); + +/* Fast path: single mapping (may fail under preemption). */ + mermap =mermap_get (page, numpages <<PAGE_SHIFT ,PAGE_KERNEL_NOGLOBAL ); +if (mermap) { +void *buf =kasan_reset_tag (mermap_addr (mermap)); + +for (int i =0 ; i < numpages; i++) +clear_page (buf + (i <<PAGE_SHIFT ));
mm/page_alloc.c static inline bool should_skip_kasan_poison(struct page *page) · line 1397
Quoted 10 lines
mm/page_alloc.c +10
-0+mermap_put (mermap); +return ; + } + +/* Slow path, map each page individually (always succeeds). */ +for (int i =0 ; i < numpages; i++) { +unsigned long flags; + +local_irq_save (flags); + mermap =mermap_get_reserved (page + i,PAGE_KERNEL );
Oh, that should be PAGE_KERNEL_NONGLOBAL. I think this means I never tested this path as this will crash - mermap_get_reserved() will WARN() and return NULL and then we'll dereference that below.
Maybe that's a bad idea, maybe it should WARN() but serve the request anyway.
Anyway maybe I need to support failure injection to test the fallback path.
Quoted 19 lines
mm/page_alloc.c +19
-0+clear_page (kasan_reset_tag (mermap_addr (mermap))); +mermap_put (mermap); + local_irq_restore(flags); + } +} +#else +static inline bool pageblock_unmapped (struct page *page) +{ +return false; +} + +static inline void clear_page_mermap (struct page *page,unsigned int numpages) +{ +BUG (); +} +#endif + +static void kernel_init_pages (struct page *page,unsigned int numpages) +{
mm/page_alloc.c static inline bool should_skip_kasan_poison(struct page *page) · line 1426
Quoted 10 lines
mm/page_alloc.c +10
-0+int num_blocks =DIV_ROUND_UP (numpages, pageblock_nr_pages); + +for (int block =0 ; block < num_blocks; block++) { +struct page *block_page = page + (block << pageblock_order); +bool unmapped =pageblock_unmapped (block_page); + +/* s390's use of memset() could override KASAN redzones. */ +kasan_disable_current (); +if (unmapped) { +clear_page_mermap (page, numpages);
That should be clearing block_page.
Quoted 9 lines
mm/page_alloc.c +8
-0+ }else { +for (int i =0 ; i <min (numpages, pageblock_nr_pages); i++) +clear_highpage_kasan_tagged (block_page + i); + } +kasan_enable_current (); + + numpages -= pageblock_nr_pages; + } }
RFC PATCH 19/19 mm: Minimal KUnit tests for some new page_alloc logic
In reply to Brendan JackmanAdd a simple smoke test for __GFP_UNMAPPED that tries to exercise flipping pageblocks between mapped/unmapped state.
Also add some basic tests for some freelist-indexing helpers.
Simplest way to run these on x86:
tools/testing/kunit/kunit.py run --arch=x86_64 "page_alloc.*" \ --kconfig_add CONFIG_MERMAP=y --kconfig_add CONFIG_PAGE_ALLOC_UNMAPPED=y
--- kernel/panic.c | 2 + mm/Kconfig | 2 +- mm/Makefile | 1 + mm/init-mm.c | 3 + mm/internal.h | 6 ++ mm/page_alloc.c | 11 +- mm/tests/page_alloc_kunit.c | 250 ++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 271 insertions(+), 4 deletions(-)
kernel/panic.c +2 -0
@@ -39,6 +39,7 @@#include <linux/sys_info.h> #include <trace/events/error_report.h> #include <asm/sections.h> +#include <kunit/visibility.h> #define PANIC_TIMER_STEP 100#define PANIC_BLINK_SPD 18
@@ -900,6 +901,7 @@ unsigned long get_taint(void) {return tainted_mask; } +EXPORT_SYMBOL_IF_KUNIT (get_taint);/** * add_taint: add a taint flag if not already set.
mm/Kconfig +1 -1
@@ -1507,7 +1507,7 @@ config PAGE_ALLOC_UNMAPPED default COMPILE_TEST || KUNIT depends on MERMAP -config PAGE_ALLOC_KUNIT_TESTS +config PAGE_ALLOC_KUNIT_TEST tristate "KUnit tests for the page allocator" if !KUNIT_ALL_TESTS depends on KUNIT default KUNIT_ALL_TESTS
mm/Makefile +1 -0
@@ -152,3 +152,4 @@ obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o obj-$(CONFIG_MERMAP) += mermap.o obj-$(CONFIG_MERMAP_KUNIT_TEST) += tests/mermap_kunit.o +obj-$(CONFIG_PAGE_ALLOC_KUNIT_TEST) += tests/page_alloc_kunit.o
mm/init-mm.c +3 -0
@@ -13,6 +13,8 @@#include <linux/iommu.h> #include <asm/mmu.h> +#include <kunit/visibility.h> +#ifndef INIT_MM_CONTEXT #define INIT_MM_CONTEXT (name)#endif
@@ -50,6 +52,7 @@ struct mm_struct init_mm = { .flexible_array = MM_STRUCT_FLEXIBLE_ARRAY_INIT, INIT_MM_CONTEXT(init_mm) }; +EXPORT_SYMBOL_IF_KUNIT (init_mm);void setup_initial_init_mm (void *start_code,void *end_code,void *end_data,void *brk)
mm/internal.h +6 -0
@@ -1781,4 +1781,10 @@ static inline int io_remap_pfn_range_complete(struct vm_area_struct *vma, return remap_pfn_range_complete(vma, addr, pfn, size, prot); } +#if IS_ENABLED (CONFIG_KUNIT ) +unsigned int order_to_pindex (freetype_t freetype,int order); +int pindex_to_order (unsigned int pindex); +bool pcp_allowed_order (unsigned int order); +#endif /* IS_ENABLED(CONFIG_KUNIT) */ +#endif /* __MM_INTERNAL_H */
mm/page_alloc.c +8 -3
@@ -57,6 +57,7 @@#include <linux/cacheinfo.h> #include <linux/pgalloc_tag.h> #include <asm/div64.h> +#include <kunit/visibility.h> #include "internal.h" #include "shuffle.h" #include "page_reporting.h"
@@ -496,6 +497,7 @@ get_pfnblock_freetype(const struct page *page, unsigned long pfn) {return __get_pfnblock_freetype (page, pfn,0 ); } +EXPORT_SYMBOL_IF_KUNIT (get_pfnblock_freetype);/**
@@ -731,7 +733,7 @@ static void bad_page(struct page *page, const char *reason)add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); } -static inline unsigned int order_to_pindex (freetype_t freetype,int order) +VISIBLE_IF_KUNIT inline unsigned int order_to_pindex(freetype_t freetype, int order) {int migratetype =free_to_migratetype (freetype);
@@ -759,8 +761,9 @@ static inline unsigned int order_to_pindex(freetype_t freetype, int order)return (MIGRATE_PCPTYPES * order) + migratetype; } +EXPORT_SYMBOL_IF_KUNIT (order_to_pindex); -static inline int pindex_to_order (unsigned int pindex) +VISIBLE_IF_KUNIT int pindex_to_order(unsigned int pindex) {unsigned int unmapped_base =NR_LOWORDER_PCP_LISTS +NR_PCP_THP ;int order;
@@ -783,8 +786,9 @@ static inline int pindex_to_order(unsigned int pindex)return order; } +EXPORT_SYMBOL_IF_KUNIT (pindex_to_order); -static inline bool pcp_allowed_order (unsigned int order) +VISIBLE_IF_KUNIT inline bool pcp_allowed_order(unsigned int order) {if (order <=PAGE_ALLOC_COSTLY_ORDER )return true;
@@ -794,6 +798,7 @@ static inline bool pcp_allowed_order(unsigned int order)#endif return false; } +EXPORT_SYMBOL_IF_KUNIT (pcp_allowed_order);/* * Higher-order pages are called "compound pages". They are structured thusly:
mm/tests/page_alloc_kunit.c new file +250 -0
250 lines not on this page · raw message
Signature · 1 line
2.51.2
Vlastimil Babka (SUSE)
maintainerQuoted 7 lines · Brendan Jackman
.:: What? Why?
.:: Why [RFC]?I really wanted to stop sending RFC and start sending PATCHes but getting this series out has taken months longer than I expected, so it's time to get something on the list. The known issues here are:
Quoted 6 lines
1.
__GFP_UNMAPPEDisn't useful yet untilguest_memfdunmapping support
[0] gets merged.2. Apparently while implementing the mm-local region, I totally forgot that KPTI existed on 32-bit systems. I expect the 0-day bot to fire a failure on that patch.
I don't think you mentioned (at least in the cover letter) the mm resistance to add new gfp flags due to number of them being uncomfortably close to 32 already. But I see you've put the new one behind a config. Together with point 2 I wonder if this is where we can start making some flags and associated functionality 64-bit only and change gfp_t to unsigned long?
Hey Vlastimil, sorry for the delay I've been unexpectedly out of office.
Quoted 15 lines · Vlastimil Babka (SUSE)
Quoted 7 lines · Brendan Jackman
.:: What? Why?
.:: Why [RFC]?I really wanted to stop sending RFC and start sending PATCHes but getting this series out has taken months longer than I expected, so it's time to get something on the list. The known issues here are:
Quoted 6 lines
1.
__GFP_UNMAPPEDisn't useful yet untilguest_memfdunmapping support
[0] gets merged.2. Apparently while implementing the mm-local region, I totally forgot that KPTI existed on 32-bit systems. I expect the 0-day bot to fire a failure on that patch.
Quoted 5 lines
I don't think you mentioned (at least in the cover letter) the mm resistance to add new gfp flags due to number of them being uncomfortably close to 32 already. But I see you've put the new one behind a config. Together with point 2 I wonder if this is where we can start making some flags and associated functionality 64-bit only and change
gfp_tto unsigned long?
Yeah, making __GFP_UNMAPPED 64bit-only would be fine with me.
Ultimately the fact that we have KPTI for 32-bit makes it sound like we would also want ASI for 32-bit, so I guess I would still want to add a GFP flag to support that on 32-bit. But that's a pretty futuristic problem, I would say we should focus on __GFP_UNMAPPED in isolation right now.
(Just to be clear regarding point 2 - that bug still matters, even if __GFP_UNMAPPED itself is 64-bit only the mm-local region is separate and needs to be correct on 32-bit).
Borislav Petkov
maintainerQuoted 5 lines · Brendan Jackman
Ultimately the fact that we have KPTI for 32-bit makes it sound like we would also want ASI for 32-bit, so I guess I would still want to add a GFP flag to support that on 32-bit. But that's a pretty futuristic problem, I would say we should focus on
__GFP_UNMAPPEDin isolation right now.
I'd wait until someone really really presents a valid use case to not move to 64-bit. And then justify the effort for adding and supporting ASI on 32-bit.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 24 lines · Brendan Jackman
.:::: Design: Introducing "freetypes"
The biggest challenge for efficiently getting stuff out of the direct map is TLB flushing. Pushing this problem into the page allocator turns out to enable amortising that flush cost into almost nothing. The core idea is to have pools of already-unmapped pages. We'd like those pages to be physically contiguous so they don't unduly fragment the pagetables around them, and we'd like to be able to efficiently look up these already-unmapped pages during allocation. The page allocator already has deeply-ingrained functionality for physically grouping pages by a certain attribute, and then indexing free pages by that attribute, this mechanism is: migratetypes.
So basically, this series extends the concepts of migratetypes in the allocator so that as well as just representing mobility, they can represent other properties of the page too. (Actually, migratetypes are already sort of overloaded, but the main extension is to be able to represent _orthogonal_ properties). In order to avoid further overloading the concept of a migratetype, this extension is done by adding a new concept on top of migratetype: the _freetype_. A freetype is basically just a migratetype plus some flags, and it replaces migratetypes wherever the latter is currently used as to index free pages.
Quoted 4 lines
The first freetype flag is then added, which marks the pages it indexes as being absent from the direct map. This is then used to implement the new
__GFP_UNMAPPEDflag, which allocates pages from pageblocks that have the new flag, or unmaps pages if no existing ones are already available.
This approach seems very interesting to me, and I wonder if it could be applied to another use-case.
I am working on a security feature to protect page table pages (PTPs) using pkeys [1]. This relies on all PTPs being mapped with a specific pkey (in the direct map). That requires changing a mapping attribute rather than making it invalid, but AFAICT this is essentially the same problem as the one you're trying to solve.
There are however extra challenges with mapping PTPs with special attributes. The main one, which you mention in patch 17, is that splitting the direct map may require allocating PTPs, which may lead to recursion.
[1] introduces a dedicated page table allocator on top of the buddy allocator, which attempts to cache PMD-sized blocks if possible. It ensures that no recursion occurs by using a special flag when allocating PTPs while splitting the direct map, and keeping a reserve of pages specifically for that situation (patch 15 and 24). There is also special handling for early page tables (essentially keeping track of them and setting their pkey once we can split the direct map).
Do you think that this freetype infrastructure could be used for that purpose, instead of introducing a layer on top of the buddy allocator? I expect that much of the special handling for allocating PTPs can be kept separate. Ensuring that protected pages are always available to split the direct map may be difficult though... This is deeply embedded in the allocator I proposed.
- Kevin
[1]
https://lore.kernel.org/linux-hardening/[email protected]/
Quoted 30 lines · Kevin Brodsky
Quoted 24 lines · Brendan Jackman
.:::: Design: Introducing "freetypes"
The biggest challenge for efficiently getting stuff out of the direct map is TLB flushing. Pushing this problem into the page allocator turns out to enable amortising that flush cost into almost nothing. The core idea is to have pools of already-unmapped pages. We'd like those pages to be physically contiguous so they don't unduly fragment the pagetables around them, and we'd like to be able to efficiently look up these already-unmapped pages during allocation. The page allocator already has deeply-ingrained functionality for physically grouping pages by a certain attribute, and then indexing free pages by that attribute, this mechanism is: migratetypes.
So basically, this series extends the concepts of migratetypes in the allocator so that as well as just representing mobility, they can represent other properties of the page too. (Actually, migratetypes are already sort of overloaded, but the main extension is to be able to represent _orthogonal_ properties). In order to avoid further overloading the concept of a migratetype, this extension is done by adding a new concept on top of migratetype: the _freetype_. A freetype is basically just a migratetype plus some flags, and it replaces migratetypes wherever the latter is currently used as to index free pages.
Quoted 4 lines
The first freetype flag is then added, which marks the pages it indexes as being absent from the direct map. This is then used to implement the new
__GFP_UNMAPPEDflag, which allocates pages from pageblocks that have the new flag, or unmaps pages if no existing ones are already available.
Quoted 8 lines
This approach seems very interesting to me, and I wonder if it could be applied to another use-case.
I am working on a security feature to protect page table pages (PTPs) using pkeys [1]. This relies on all PTPs being mapped with a specific pkey (in the direct map). That requires changing a mapping attribute rather than making it invalid, but AFAICT this is essentially the same problem as the one you're trying to solve.
Yeah, I think so:
1. The fragmentation issues seem exactly the same.
2. The TLB flushing issues are probably also basically the same, I assume you need to flush the TLB when you convert a page to use for pagetables, and without allocator integration that can happen pretty often and in hot paths. Correct?
Quoted 10 lines
There are however extra challenges with mapping PTPs with special attributes. The main one, which you mention in patch 17, is that splitting the direct map may require allocating PTPs, which may lead to recursion.
[1] introduces a dedicated page table allocator on top of the buddy allocator, which attempts to cache PMD-sized blocks if possible. It ensures that no recursion occurs by using a special flag when allocating PTPs while splitting the direct map, and keeping a reserve of pages specifically for that situation (patch 15 and 24).
Right, and actually just today someone pointed out mm/execmem.c to me, I think execmem_cache_populate() is basically doing the same thing (although it's also creating a separate virtual mapping).
Quoted 6 lines
There is also special
handling for early page tables (essentially keeping track of them and setting their pkey once we can split the direct map).Do you think that this freetype infrastructure could be used for that purpose, instead of introducing a layer on top of the buddy allocator?
Yes!!! 100% definitely, my code certainly solves all your problems...
Quoted 5 lines
I
expect that much of the special handling for allocating PTPs can be kept separate. Ensuring that protected pages are always available to split the direct map may be difficult though... This is deeply embedded in the allocator I proposed.
...Oh, hm, well, um, good point. Thinking aloud a bit...
The way this series dodges the question is (copying from the code comments in patch 17 for convenient reading):
1) - The direct map starts out fully mapped at boot. (This is not really
* an assumption" as its in direct control of page_alloc.c).
*
2) - Once pages in the direct map are broken down, they are not
* re-aggregated into larger pages again.
*
3) - Pagetables are never allocated with __GFP_UNMAPPED.
*
* Under these assumptions, a pagetable might need to be allocated while
* _unmapping_ stuff from the direct map during a __GFP_UNMAPPED
* allocation. But, the allocation of that pagetable never requires
* allocating a further pagetable.
In other words, we might need to allocate while we allocate (which is fine because I have to do locking shenanigans anyway due to x86 TLB shootdown requirements), but there's no further recursion after that.
Can we come up with an analogue for protected PTPs? Point 3) is the inflexible one, and we obviously can't say "PTPs are never allocated as PTPs". But if we invert it and _also_ invert point 1) I think we get something that works in principle:
1) The direct map starts out _fully protected_ (i.e. we treat everything as if it's a pagetable at first).
2) We assume the direct map doesn't get reaggregated once we've broken things down to serve PTP allocations
3) PTPs are always PTPs...
But... this is a bit silly, since what it means is we'll then go through ~all the pagetblocks in the system (except the ones that _are_ actually used for PTPs) and flip their pkey, breaking down the physmap to pageblock granularity as we go. And... if we're gonna do that, we might as well just say the physmap has to be at pageblock granularity to begin with.
(Could we do that? Maybe - Mike Rapoport has previously argued that physmap fragmentation is not a very big deal, so I guess the question is whether we're ready to really lean into that analysis, it would be quite painful if it turned out to be wrong).
Another potential "dodge": Is it really important that the PTPs are always protected from the very moment they are created?
Coz this feature still seems pretty useful even if there's an awkward fallback case where, under specific memory pressure patterns, we temporarily use unprotected pagetables to set up protected pagetables. That still makes exploiting a pagetable overwrite an order of magnitude harder than before, right? Similar to how there's probably ways to exploit bugs if you can get them to race with the intended pagetable update paths that flip the pkey register, or if you can get a ROP chain to flip that register for you or whatever.
Quoted 5 lines · Brendan Jackman
Quoted 4 lines · Kevin Brodsky
[...]
This approach seems very interesting to me, and I wonder if it could be applied to another use-case.
Quoted 8 lines
Quoted 5 lines
I am working on a security feature to protect page table pages (PTPs) using pkeys [1]. This relies on all PTPs being mapped with a specific pkey (in the direct map). That requires changing a mapping attribute rather than making it invalid, but AFAICT this is essentially the same problem as the one you're trying to solve.
Yeah, I think so:
1. The fragmentation issues seem exactly the same.
I believe so.
Quoted 4 lines
2. The TLB flushing issues are probably also basically the same, I assume you need to flush the TLB when you convert a page to use for pagetables, and without allocator integration that can happen pretty often and in hot paths. Correct?
Indeed. Up until v5 [2] no special allocator was used - the pkey was set at the page level every time a PTP was allocated or freed. Clearly suboptimal, and doesn't work at all if large mappings are used due to the risk of recursion.
Quoted 5 lines
Quoted 5 lines
There are however extra challenges with mapping PTPs with special attributes. The main one, which you mention in patch 17, is that splitting the direct map may require allocating PTPs, which may lead to recursion.
Quoted 8 lines
Quoted 5 lines
[1] introduces a dedicated page table allocator on top of the buddy allocator, which attempts to cache PMD-sized blocks if possible. It ensures that no recursion occurs by using a special flag when allocating PTPs while splitting the direct map, and keeping a reserve of pages specifically for that situation (patch 15 and 24).
Right, and actually just today someone pointed out
mm/execmem.cto me, I thinkexecmem_cache_populate()is basically doing the same thing (although it's also creating a separate virtual mapping).
Ah interesting I didn't know about that cache. It does have similarities, and the motivation seems similar too.
Quoted 7 lines
Quoted 6 lines
There is also special
handling for early page tables (essentially keeping track of them and setting their pkey once we can split the direct map).Do you think that this freetype infrastructure could be used for that purpose, instead of introducing a layer on top of the buddy allocator?
Yes!!! 100% definitely, my code certainly solves all your problems...
Almost ;)
Quoted 38 lines
Quoted 5 lines
I
expect that much of the special handling for allocating PTPs can be kept separate. Ensuring that protected pages are always available to split the direct map may be difficult though... This is deeply embedded in the allocator I proposed....Oh, hm, well, um, good point. Thinking aloud a bit...
The way this series dodges the question is (copying from the code comments in patch 17 for convenient reading):
1) - The direct map starts out fully mapped at boot. (This is not really
* an assumption" as its in direct control ofpage_alloc.c).
*
2) - Once pages in the direct map are broken down, they are not
* re-aggregated into larger pages again.
*
3) - Pagetables are never allocated with__GFP_UNMAPPED.
*
* Under these assumptions, a pagetable might need to be allocated while
* _unmapping_ stuff from the direct map during a__GFP_UNMAPPED
* allocation. But, the allocation of that pagetable never requires
* allocating a further pagetable.In other words, we might need to allocate while we allocate (which is fine because I have to do locking shenanigans anyway due to x86 TLB shootdown requirements), but there's no further recursion after that.
Can we come up with an analogue for protected PTPs? Point 3) is the inflexible one, and we obviously can't say "PTPs are never allocated as PTPs". But if we invert it and _also_ invert point 1) I think we get something that works in principle:
1) The direct map starts out
_fullyprotected_(i.e. we treat everything as if it's a pagetable at first).2) We assume the direct map doesn't get reaggregated once we've broken things down to serve PTP allocations
Quoted 8 lines
3) PTPs are always PTPs...
But... this is a bit silly, since what it means is we'll then go through ~all the pagetblocks in the system (except the ones that _are_ actually used for PTPs) and flip their pkey, breaking down the physmap to pageblock granularity as we go. And... if we're gonna do that, we might as well just say the physmap has to be at pageblock granularity to begin with.
Having to change the pkey of every pageblock when allocating it for anything but page tables seems rather unreasonable... And in case of memory pressure, where fragmentation is high, we may not have any protected pageblock left. The allocator I proposed falls back to order-2 allocations if necessary (which is sufficient to replenish the page reserve even if PMD+PTE pages are allocated for splitting).
Quoted 5 lines
(Could we do that? Maybe - Mike Rapoport has previously argued that physmap fragmentation is not a very big deal, so I guess the question is whether we're ready to really lean into that analysis, it would be quite painful if it turned out to be wrong).
Quoted 10 lines
Another potential "dodge": Is it really important that the PTPs are always protected from the very moment they are created?
Coz this feature still seems pretty useful even if there's an awkward fallback case where, under specific memory pressure patterns, we temporarily use unprotected pagetables to set up protected pagetables. That still makes exploiting a pagetable overwrite an order of magnitude harder than before, right? Similar to how there's probably ways to exploit bugs if you can get them to race with the intended pagetable update paths that flip the pkey register, or if you can get a ROP chain to flip that register for you or whatever.
I considered this - I agree that having page tables unprotected inside a small window may be acceptable, considering that this is hardening and not bullet-proof isolation. That said, I'm not sure it helps all that much. You'd need a mechanism to defer setting the pkey for those PTPs. Once you decide to set the pkey, you may very well end up splitting the direct map again, deferring new PTPs... This could go on, and every time fragmentation increases. I think it is really desirable to have that reserve of pages so that splitting the direct map does not become recursive (whether deferred or not).
- Kevin
[2]
https://lore.kernel.org/linux-hardening/[email protected]/
Mike Rapoport
maintainerQuoted 8 lines · Kevin Brodsky
Quoted 3 lines · Brendan Jackman
Right, and actually just today someone pointed out
mm/execmem.cto me, I thinkexecmem_cache_populate()is basically doing the same thing (although it's also creating a separate virtual mapping).Ah interesting I didn't know about that cache. It does have similarities, and the motivation seems similar too.
The motivation for execmem cache is slightly different. The goal there was to ensure kernel's executable memory (modules, kprobes, ftrace and potentially BPF) is mapped at PMD level at vmalloc address space. And the removal of the direct map alias for execmem is rather a side effect :)
But sure, there are similarities.
Signature · 2 lines
Sincerely yours, Mike.
Quoted 1 line · Brendan Jackman
__GFP_UNMAPPED
Haven't looked at this in detail, but there was some previous work that even used the same flag name. In the end, the discussion leaned towards a dedicated API instead of a flag. Not saying the flag approach is dead, but might useful to explain how it fits in with that discussion.
Hi Rick, sorry I was just scanning over this thread again while prepping the next version and noticed I never replied to this.
Quoted 9 lines · Rick P Edgecombe
Quoted 1 line · Brendan Jackman
__GFP_UNMAPPEDHaven't looked at this in detail, but there was some previous work that even used the same flag name. In the end, the discussion leaned towards a dedicated API instead of a flag. Not saying the flag approach is dead, but might useful to explain how it fits in with that discussion.
I am not at all wed to using a GFP flag for this, but a key difference between this and Mike's original __GFP_UNMAPPED is that this is integrated directly into the page allocator itself, and that's a load bearing element.
Technically speaking, in this series __GFP_UNMAPPED is only supported for unmovable allocations, but that's just to avoid bloating the data structures (there isn't a user for that type of allocation yet, so there's no point in creating freelists for it). But, in principle, the goal here is to support all the fancy stuff that the mm does for this memory. That's important because for the real usecases I have in mind here, the vast majority of memory in the system should eventually be relying on the page allocator to unmap it (either completely as in __GFP_UNMAPPED, or just from the special ASI pagetables as in __GFP_SENSITIVE, which will be added later).
So, yeah we can always have a special API but that would be a bit of a roundabout way to just save a bit in a an enum, it wouldn't actually represent any simplification of the page allocator's API.
Anyway thanks for pointing this out, I will neded to explain this in the next version's cover letter, but in the meantime there's a quick braindump of my thinking.