mm: Add __GFP_UNMAPPED
Subsystems (15)
v2 cover mm: Add __GFP_UNMAPPED
.:: What? Why?
This series adds support for efficiently allocating pages that are not present in the direct map. This is instrumental to two different immediate goals:
1. This supports the effort to remove guest_memfd memory from the direct map [0]. One of the challenges faced in that effort has been efficiently eliminating TLB entries, this series offers a solution to that problem
2. Address Space Isolation (ASI) [1] also needs an efficient way to allocate pages that are missing from the direct map. Although for ASI the needs are slightly different (in that case, the pages need only be removed from ASI's special pagetables), the most interesting mm challenges are basically the same.
So, __GFP_UNMAPPED serves as a Trojan horse to get the page allocator into a state where adding ASI's features "Should Be Easy".
This series _also_ serves as a Trojan horse for the "mermap" (details below) which is also a key building block for making ASI efficient.
Longer term, there are a wide range of security techniques unlocked by being able to efficiently remove pages from the direct map. This ranges from straightforward nice-to-haves like removing unused aliases of the vmalloc area, to more ambitious ideas like totally unmapping _all_ user memory.
There may also be non-security usecases for this feature, for example at LPC Sumit Garg presented an issue with memory-firewalled client devices that could be remediated by __GFP_UNMAPPED [2]. It seems also likely to be a key step towards dropping execmem as a distinct allocation layer.
.:: Design
The key design elements introduced here are just repurposed from previous attempts to directly introduce ASI's needs to the page allocator [3]. The only real difference is that now these support totally unmapping stuff from the direct map, instead of only unmapping it from ASI's special pagetables.
Note that it's important that allocating unmapped memory be fully scalable, in principle supporting basically everything the mm can do - previous __GFP_UNMAPPED proposals have implemented it as a cache on top of the page allocator. That won't do here because the end goal here is protecting user data. That means that ultimately, the vast majority of memory in the system ought to be allocated through a mechanism in the vein of __GFP_UNMAPPED, so a really full-featured allocator is required. (In this series, __GFP_UNMAPPED is only supported for unmovable allocations, but that's just to save space in datastructures: the implementation is supposed to be fully generic, eventually allowing reclaim and compaction for the unmapped pages).
Because of the above, I'm proposing a new GFP flag. However the only part of that I'm really attached to is that it's an interface to the real page allocator. For now I still assume a GFP flag is the cleanest way to get that but in principle I'm not opposed to alloc_unmapped_pages() or whatever. Alternatively, there's some discussion in [9] about how to get more GFP space back, which I'm happy to help with.
.:::: Design: Introducing "freetypes"
The biggest challenge for efficiently getting stuff out of the direct map is TLB flushing. Pushing this problem into the page allocator turns out to enable amortising that flush cost into almost nothing. The core idea is to have pools of already-unmapped pages. We'd like those pages to be physically contiguous so they don't unduly fragment the pagetables around them, and we'd like to be able to efficiently look up these already-unmapped pages during allocation. The page allocator already has deeply-ingrained functionality for physically grouping pages by a certain attribute, and then indexing free pages by that attribute, this mechanism is: migratetypes.
So basically, this series extends the concepts of migratetypes in the allocator so that as well as just representing mobility, they can represent other properties of the page too. (Actually, migratetypes are already sort of overloaded, but the main extension is to be able to represent _orthogonal_ properties). In order to avoid further
Because of these ambitious usecases, it's core to this proposal that the feature
overloading the concept of a migratetype, this extension is done by adding a new concept on top of migratetype: the _freetype_. A freetype is basically just a migratetype plus some flags, and it replaces migratetypes wherever the latter is currently used as to index free pages.
The first freetype flag is then added, which marks the pages it indexes as being absent from the direct map. This is then used to implement the new __GFP_UNMAPPED flag, which allocates pages from pageblocks that have the new flag, or unmaps pages if no existing ones are already available.
.:::: Design: Introducing the "mermap"
Sharp readers might by now be asking how __GFP_UNMAPPED interacts with __GFP_ZERO. If pages aren't in the direct map, how can the page allocator zero them? The solution is the "mermap", short for "epheMERal mapping". The mermap provides an efficient way to temporarily map pages into the local address space, and the allocator uses these mappings to zero pages.
Using the mermap securely requires some knowledge about the usage of the pages. One slightly awkward part of this design is that the page allocator's usage of the mermap then "leaks" out so that callers who allocate with __GFP_UNMAPPED|__GFP_ZERO need to be aware of the mermap's security implications. For the guest_memfd unmapping usecase, that means when guest_memfd.c makes these special allocations, it is only safe because the pages will belong to the current process. In other words, the use of the mermap potentially allows that process to leak the pages via CPU sidechannels (unless more holistic/expensive mitigations are enabled).
Since this cover letter is already too long I won't describe most details of the mermap here, please see the patch that introduces it.
But one key detail is that it requires a kernel-space but mm-local virtual address region. So... this series adds that too (for x86). This is called the mm-local region and is implemented by "just" extending and generalising the LDT remap area.
.:: Outline of the patchset
- Patches 1 -> 2 introduce the mm-local region for x86
- Patches 3 -> 5 are prep patches for the mermap
- Patches 6 -> 7 introduce the mermap
- Patches 8 -> 11 introduce freetypes
- Patch 8 in particular is the big annoying switch-over which changes a whole bunch of code from "migratetype" to "freetype". In order to try and have the compiler help out with catching bugs, this is done with an annoying typedef. I'm sorry that this patch is so annoying, but I think if we do want to extend the allocator along these lines then a typedef + big annoying patch is probably the safest way.
- Patches 12 -> 21 introduce __GFP_UNMAPPED
- Patch 22 adopts __GFP_UNMAPPED as for secretmem.
.:: Hacky bits
.:::: Hacky bits: mermap pagetable management
As discussed in [8], I can't find any existing pagetable management code that serves the needs of the mermap. I have responded to this with some extensions to __apply_to_page_range(). Those extensions make it serve the mermap's needs re avoiding allocation and locking. But they don't offer an efficient way to preallocate empty pagetables in mermap_mm_prepare(). For now I think the inefficient way fine but if a workload emerges that is sensitive to the latency of mermap setup, it will need to be optimised. I think that would best be done as part of the general pagetable library I hope to discuss in [8].
.:::: Hacky bits: simplistic secretmem integration
The secretmem integration leaves the mmain optimisations on the table; the security-required flushes of the mermap areas are implemented via distinct tlb_flush_mm() calls. It should be possible to amortize the mermap TLB flushes completely into the normal VMA flushing. However, as far as I know there is no performance-sensitive usecase for secretmem. So, I've just implemented the minimal adoption. This will at least avoid fragmentation of the direct map, even if it doesn't reduce TLB flushing. If anyone knows of a workload that might benefit from dropping that flushing, let me know!
.:: Performance
In [4] is a branch containing:
1. This series.
2. All the key kernel patches from the Firecracker team's "secret-free" effort, which includes guest_memfd unmapping ([0]).
3. Some prototype patches to switch guest_memfd over from an ad-hoc unmapping logic to use of __GFP_UNMAPPED (plus direct use of the mermap to implement write()).
I benchmarked this using Firecracker's own performance tests [4], which measure the time required to populate the VM guest's memory. This population happens via write() so it exercises the mermap. I ran this on a Sapphire Rapids machine [5]. The baseline here is just the secret-free patches on their own. "gfp_unmapped" is the branch described above. "skip-flush" provides a reference against an implementation that just skips flushing the TLB when unmapping guest_memfd pages, which serves as an upper-bound on performance.
metric: populate_latency (ms) | test: firecracker-perf-tests-wrapped
+---------------+---------+----------+----------+------------------------+----------+--------+ | nixos_variant | samples | mean | min | histogram | max | Δμ | +---------------+---------+----------+----------+------------------------+----------+--------+ | | 30 | 1.04s | 1.02s | █ | 1.10s | | | gfp_unmapped | 30 | 313.02ms | 299.48ms | █ | 343.25ms | -70.0% | | skip-flush | 30 | 325.80ms | 307.91ms | █ | 333.30ms | -68.8% | +---------------+---------+----------+----------+------------------------+----------+--------+
Conclusion: it's close to the best case performance for this particular workload. (Note in the sample above the mean is actually faster - that's noise, this isn't a consistent observation).
[0] [PATCH v10 00/15] Direct Map Removal Support for guest_memfd
https://lore.kernel.org/all/[email protected]/
[2] https://lpc.events/event/19/contributions/2095/
[3] https://lore.kernel.org/lkml/[email protected]/
[5] https://github.com/bjackman/aethelred/blob/eb0dd0e99ee08fa0534733113e93b89499affe91
[7] https://lore.kernel.org/lkml/[email protected]/
[8] https://lore.kernel.org/all/[email protected]/
[9] https://lore.kernel.org/lkml/[email protected]/
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: Sumit Garg <[email protected]>
To: Borislav Petkov <[email protected]>
To: Dave Hansen <[email protected]>
To: Peter Zijlstra <[email protected]>
To: Andrew Morton <[email protected]>
To: David Hildenbrand <[email protected]>
To: Lorenzo Stoakes <[email protected]>
To: Vlastimil Babka <[email protected]>
To: Mike Rapoport <[email protected]>
To: Wei Xu <[email protected]>
To: Johannes Weiner <[email protected]>
To: Zi Yan <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: Will Deacon <[email protected]>
Cc: [email protected]
Cc: "Kalyazin, Nikita" <[email protected]>
Cc: [email protected]
Cc: "Itazuri, Takahiro" <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: David Kaplan <[email protected]>
Cc: Thomas Gleixner <[email protected]>
---
Changes in v2:
- Adopted it for secretmem
- Fixed mm-local region under KPTI on PAE
- Added __apply_to_get_range() mess
- Added fault injection for mermap_get(), fixed bug this uncovered
- Fixed various silly bugs
- Link to v1 (RFC): https://lore.kernel.org/r/[email protected]
---
Brendan Jackman (22):
x86/mm: split out preallocate_sub_pgd()
x86/mm: Generalize LDT remap into "mm-local region" x86/tlb: Expose some flush function declarations to modules
mm: Create flags arg for __apply_to_page_range()
mm: Add more flags for __apply_to_page_range()
x86/mm: introduce the mermap
mm: KUnit tests for the mermap
mm: introduce for_each_free_list()
mm/page_alloc: don't overload migratetype in find_suitable_fallback()
mm: introduce freetype_t
mm: move migratetype definitions to freetype.h
mm: add definitions for allocating unmapped pages
mm: rejig pageblock mask definitions
mm: encode freetype flags in pageblock flags mm/page_alloc: remove ifdefs from pindex helpers mm/page_alloc: separate pcplists by freetype flags mm/page_alloc: rename ALLOC_NON_BLOCK back to _HARDER mm/page_alloc: introduce ALLOC_NOBLOCK
mm/page_alloc: implement __GFP_UNMAPPED allocations mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations
mm: Minimal KUnit tests for some new page_alloc logic mm/secretmem: Use __GFP_UNMAPPED when available
Documentation/arch/x86/x86_64/mm.rst | 4 +- arch/x86/Kconfig | 3 + arch/x86/include/asm/mermap.h | 23 + arch/x86/include/asm/mmu_context.h | 119 ++++- arch/x86/include/asm/page.h | 32 ++ arch/x86/include/asm/pgalloc.h | 33 ++ arch/x86/include/asm/pgtable_32_areas.h | 9 +- arch/x86/include/asm/pgtable_64_types.h | 18 +- arch/x86/include/asm/pgtable_types.h | 2 + arch/x86/include/asm/tlbflush.h | 43 +- arch/x86/kernel/ldt.c | 130 +----- arch/x86/mm/init_64.c | 44 +- arch/x86/mm/pgtable.c | 32 +- include/linux/freetype.h | 147 ++++++ include/linux/gfp.h | 25 +- include/linux/gfp_types.h | 26 ++ include/linux/mermap.h | 63 +++ include/linux/mermap_types.h | 44 ++ include/linux/mm.h | 13 + include/linux/mm_types.h | 6 + include/linux/mmzone.h | 84 ++-- include/linux/pageblock-flags.h | 16 +- include/trace/events/mmflags.h | 9 +- kernel/fork.c | 6 + kernel/panic.c | 2 + kernel/power/snapshot.c | 8 +- mm/Kconfig | 43 ++ mm/Makefile | 3 + mm/compaction.c | 36 +- mm/init-mm.c | 3 + mm/internal.h | 70 ++- mm/memory.c | 76 +-- mm/mermap.c | 334 ++++++++++++++ mm/mm_init.c | 11 +- mm/page_alloc.c | 788 +++++++++++++++++++++++--------- mm/page_isolation.c | 2 +- mm/page_owner.c | 7 +- mm/page_reporting.c | 4 +- mm/pgalloc-track.h | 6 + mm/secretmem.c | 87 +++- mm/show_mem.c | 4 +- mm/tests/mermap_kunit.c | 250 ++++++++++ mm/tests/page_alloc_kunit.c | 250 ++++++++++ 43 files changed, 2341 insertions(+), 574 deletions(-)
---
base-commit: b5d083a3ed1e2798396d5e491432e887da8d4a06
change-id: 20260112-page_alloc-unmapped-944fe5d7b55c
Best regards,
Signature · 1 line
Brendan Jackman <[email protected]>
PATCH v2 1/22 x86/mm: split out preallocate_sub_pgd()
This code will be needed elsewhere in a following patch. Split out the trivial code move for easy review.
This changes the logging slightly: instead of panic() directly reporting the level of the failure, there is now a generic panic message which will be preceded by a separate warn that reports the level of the failure. This is a simple way to have this helper suit the needs of its new user as well as the existing one.
Other than logging, no functional change intended.
--- arch/x86/include/asm/pgalloc.h | 33 +++++++++++++++++++++++++++++++ arch/x86/mm/init_64.c | 44 +++++++----------------------------------- 2 files changed, 40 insertions(+), 37 deletions(-)
arch/x86/include/asm/pgalloc.h +33 -0
@@ -2,6 +2,7 @@#ifndef _ASM_X86_PGALLOC_H#define _ASM_X86_PGALLOC_H +#include <linux/printk.h> #include <linux/threads.h> #include <linux/mm.h> /* for struct page */ #include <linux/pagemap.h>
@@ -128,6 +129,38 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,___pud_free_tlb (tlb, pud); } +/* Allocate a pagetable pointed to by the top hardware level. */ +static inline int preallocate_sub_pgd (struct mm_struct *mm,unsigned long addr) +{ +const char *lvl; +p4d_t *p4d; +pud_t *pud; + + lvl ="p4d" ; + p4d =p4d_alloc (mm,pgd_offset_pgd (mm->pgd, addr), addr); +if (!p4d) + goto failed; + +if (pgtable_l5_enabled ()) +return 0 ; + +/* +* On 4-level systems, the P4D layer is folded away and +* the above code does no preallocation. Below, go down +* to the pud _software_ level to ensure the second +* hardware level is allocated on 4-level systems too. +*/ + lvl ="pud" ; + pud =pud_alloc (mm, p4d, addr); +if (!pud) + goto failed; +return 0 ; + +failed: +pr_warn_ratelimited ("Failed to preallocate %s\n" , lvl); +return -ENOMEM ; +} +#if CONFIG_PGTABLE_LEVELS >4 static inline void pgd_populate (struct mm_struct *mm,pgd_t *pgd,p4d_t *p4d) {
arch/x86/mm/init_64.c +7 -37
53 lines not on this page · raw message
Signature · 1 line
2.51.2
Re: [PATCH v2 01/22] x86/mm: split out preallocate_sub_pgd()
arch/x86/mm/init_64.c static void __init register_page_bootmem_info(void) · line 1339
Quoted 4 lines · Brendan Jackman
arch/x86/mm/init_64.c +0
-4-/* -* The goal here is to allocate all possibly required -* hardware page tables pointed to by the top hardware -* level.
This comment is pretty important, IMNHO, and you zapped it.
The problem here is that the per-MM carved out space is PGD-sized. You want to make sure there are page tables allocated for that space. But, if you say "go allocate a p4d" then that will collapse down to doing nothing on a 4-level system.
So, this is effectively:
Go allocate a p4d or pud, depending on if it's 4 or 5 level. Basically, always allocate the level that the hardware PGD points to.
Could we put a comment to that effect around somewhere, please?
Re: [PATCH v2 01/22] x86/mm: split out preallocate_sub_pgd()
Quoted 13 lines · Dave Hansen
arch/x86/mm/init_64.c static void __init register_page_bootmem_info(void) · line 1339
Quoted 4 lines · Brendan Jackman
arch/x86/mm/init_64.c +0
-4-/* -* The goal here is to allocate all possibly required -* hardware page tables pointed to by the top hardware -* level. This comment is pretty important, IMNHO, and you zapped it.
The problem here is that the per-MM carved out space is PGD-sized. You want to make sure there are page tables allocated for that space. But, if you say "go allocate a p4d" then that will collapse down to doing nothing on a 4-level system.
Quoted 7 lines
So, this is effectively:
Go allocate a p4d or pud, depending on if it's 4 or 5 level. Basically, always allocate the level that the hardware PGD points to.
Could we put a comment to that effect around somewhere, please?
Hm I kinda thought the comments I left in there captured all this stuff, but yeah I can see this is a bit of a weird function so more commentary makes sense. How about I just put a few more words into the top comment:
/*
* Allocate all possibly requried hardware page tables pointed to ths
* top hardware level. In other words, allocate a p4d on 5-level or a
* pud on 4-level.
*/
And then just leave the internal one as it is:
/*
* On 4-level systems, the P4D layer is folded away and
* the above code does no preallocation. Below, go down
* to the pud _software_ level to ensure the second
* hardware level is allocated on 4-level systems too. */
Borislav Petkov
maintainerRe: [PATCH v2 01/22] x86/mm: split out preallocate_sub_pgd()
In reply to Brendan JackmanQuoted 8 lines · Brendan Jackman
This code will be needed elsewhere in a following patch. Split out the trivial code move for easy review.
This changes the logging slightly: instead of
panic()directly reporting the level of the failure, there is now a generic panic message which will be preceded by a separate warn that reports the level of the failure. This is a simple way to have this helper suit the needs of its new user as well as the existing one.
"Describe your changes in imperative mood, e.g. “make xyzzy do frotz” instead of “[This patch] makes xyzzy do frotz” or “[I] changed xyzzy to do frotz”, as if you are giving orders to the codebase to change its behaviour."
arch/x86/include/asm/pgalloc.h
Quoted 12 lines
Other than logging, no functional change intended.
Signed-off-by Brendan Jackman <[email protected]>--- arch/x86/include/asm/pgalloc.h | 33 +++++++++++++++++++++++++++++++ arch/x86/mm/init_64.c | 44 +++++++----------------------------------- 2 files changed, 40 insertions(+), 37 deletions(-)arch/x86/include/asm/pgalloc.h +0
-0
Why in a header?
That function is kinda bigger than the rest of the oneliners there...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v2 01/22] x86/mm: split out preallocate_sub_pgd()
Quoted 10 lines · Borislav Petkov
Quoted 8 lines · Brendan Jackman
This code will be needed elsewhere in a following patch. Split out the trivial code move for easy review.
This changes the logging slightly: instead of
panic()directly reporting the level of the failure, there is now a generic panic message which will be preceded by a separate warn that reports the level of the failure. This is a simple way to have this helper suit the needs of its new user as well as the existing one.
Quoted 3 lines
"Describe your changes in imperative mood, e.g. “make xyzzy do frotz” instead of “[This patch] makes xyzzy do frotz” or “[I] changed xyzzy to do frotz”, as if you are giving orders to the codebase to change its behaviour."
Yeah there are a bunch of comments in this series where I've written "we do XYZ" instead of "do XYZ", I've set up an AI prompt [0] to try and catch it for me before I send you guys crazy with yet another series that makes the same mistake over and over.
[0] https://lore.kernel.org/all/[email protected]/
TBH for this particular case it seems borderline? The referent of the "this" is the previous paragraph, it's trying to show that the logging change is a side effect of the code movement rather than a part of the patch's intent. But there are more explicit ways to carry that message so I'll rewrite it.
Quoted 8 lines
Quoted 8 lines
Other than logging, no functional change intended.
Signed-off-by Brendan Jackman <[email protected]>--- arch/x86/include/asm/pgalloc.h | 33 +++++++++++++++++++++++++++++++ arch/x86/mm/init_64.c | 44 +++++++----------------------------------- 2 files changed, 40 insertions(+), 37 deletions(-)
Quoted 8 lines
arch/x86/include/asm/pgalloc.h
Quoted 4 lines
arch/x86/include/asm/pgalloc.h +0
-0Why in a header?
That function is kinda bigger than the rest of the oneliners there...
Hm, I suspect I wanted to avoid creating new ifdefs in the C file? Seems harmless though. Looks fine in arch/x86/mm/pgtable.c.
PATCH v2 2/22 x86/mm: Generalize LDT remap into "mm-local region"
In reply to Brendan JackmanVarious security features benefit from having process-local address mappings. Examples include no-direct-map guest_memfd [2] and significant optimizations for ASI [1].
As pointed out by Andy in [0], x86 already has a PGD entry that is local to the mm, which is used for the LDT.
So, simply redefine that entry's region as "the mm-local region" and then redefine the LDT region as a sub-region of that.
With the currently-envisaged usecases, there will be many situations where almost no processes have any need for the mm-local region. Therefore, avoid its overhead (memory cost of pagetables, alloc/free overhead during fork/exit) for processes that don't use it by requiring its users to explicitly initialize it via the new mm_local_* API.
This means that the LDT remap code can be simplified:
1. map_ldt_struct_to_user() and free_ldt_pgtables() are no longer required as the mm_local core code handles that automatically.
2. The sanity-check logic is unified: in both cases just walk the pagetables via a generic mechanism. This slightly relaxes the sanity-checking since lookup_address_in_pgd() is more flexible than pgd_to_pmd_walk(), but this seems to be worth it for the simplified code.
On 64-bit, the mm-local region gets a whole PGD. On 32-bit, it just i.e. one PMD, i.e. it is completely consumed by the LDT remap - no investigation has been done into whether it's feasible to expand the region on 32-bit. Most likely there is no strong usecase for that anyway.
In both cases, in order to combine the need for an on-demand mm initialisation, combined with the desire to transparently handle propagating mappings to userspace under KPTI, the user and kernel pagetables are shared at the highest level possible. For PAE that means the PTE table is shared and for 64-bit the P4D/PUD. This is implemented by pre-allocating the first shared table when the mm-local region is first initialised.
The PAE implementation of mm_local_map_to_user() does not allocate pagetables, it assumes the PMD has been preallocated. To make that assumption safer, expose PREALLOCATED_PMDs in the arch headers so that mm_local_map_to_user() can have a BUILD_BUG_ON().
[0] https://lore.kernel.org/linux-mm/CALCETrXHbS9VXfZ80kOjiTrreM2EbapYeGp68mvJPbosUtorYA@mail.gmail.com/
[1] https://linuxasi.dev/
[2] https://lore.kernel.org/all/[email protected]
--- Documentation/arch/x86/x86_64/mm.rst | 4 +- arch/x86/Kconfig | 2 + arch/x86/include/asm/mmu_context.h | 119 ++++++++++++++++++++++++++++- arch/x86/include/asm/page.h | 32 ++++++++ arch/x86/include/asm/pgtable_32_areas.h | 9 ++- arch/x86/include/asm/pgtable_64_types.h | 12 ++- arch/x86/kernel/ldt.c | 130 +++++--------------------------- arch/x86/mm/pgtable.c | 32 +------- include/linux/mm.h | 13 ++++ include/linux/mm_types.h | 2 + kernel/fork.c | 1 + mm/Kconfig | 11 +++ 12 files changed, 217 insertions(+), 150 deletions(-)
Documentation/arch/x86/x86_64/mm.rst +2 -2
@@ -53,7 +53,7 @@ Complete virtual memory map with 4-level page tables ____________________________________________________________|___________________________________________________________ | | | | ffff800000000000 | -128 TB | ffff87ffffffffff | 8 TB | ... guard hole, also reserved for hypervisor - ffff880000000000 | -120 TB | ffff887fffffffff | 0.5 TB | LDT remap for PTI + ffff880000000000 | -120 TB | ffff887fffffffff | 0.5 TB | MM-local kernel data. Includes LDT remap for PTI ffff888000000000 | -119.5 TB | ffffc87fffffffff | 64 TB | direct mapping of all physical memory (page_offset_base) ffffc88000000000 | -55.5 TB | ffffc8ffffffffff | 0.5 TB | ... unused hole ffffc90000000000 | -55 TB | ffffe8ffffffffff | 32 TB | vmalloc/ioremap space (vmalloc_base)
@@ -123,7 +123,7 @@ Complete virtual memory map with 5-level page tables ____________________________________________________________|___________________________________________________________ | | | | ff00000000000000 | -64 PB | ff0fffffffffffff | 4 PB | ... guard hole, also reserved for hypervisor - ff10000000000000 | -60 PB | ff10ffffffffffff | 0.25 PB | LDT remap for PTI + ff10000000000000 | -60 PB | ff10ffffffffffff | 0.25 PB | MM-local kernel data. Includes LDT remap for PTI ff11000000000000 | -59.75 PB | ff90ffffffffffff | 32 PB | direct mapping of all physical memory (page_offset_base) ff91000000000000 | -27.75 PB | ff9fffffffffffff | 3.75 PB | ... unused hole ffa0000000000000 | -24 PB | ffd1ffffffffffff | 12.5 PB | vmalloc/ioremap space (vmalloc_base)
arch/x86/Kconfig +2 -0
@@ -133,6 +133,7 @@ config X86 select ARCH_SUPPORTS_RT select ARCH_SUPPORTS_AUTOFDO_CLANG select ARCH_SUPPORTS_PROPELLER_CLANG if X86_64 + select ARCH_SUPPORTS_MM_LOCAL_REGION if X86_64 || X86_PAE select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_CMPXCHG_LOCKREF if X86_CX8 select ARCH_USE_MEMTEST
@@ -2323,6 +2324,7 @@ config CMDLINE_OVERRIDE config MODIFY_LDT_SYSCALL bool "Enable the LDT (local descriptor table)" if EXPERT default y + select MM_LOCAL_REGION if MITIGATION_PAGE_TABLE_ISOLATION || X86_PAE help Linux can allow user programs to install a per-process x86 Local Descriptor Table (LDT) using the modify_ldt(2) system
arch/x86/include/asm/mmu_context.h +116 -3
148 lines not on this page · raw message
arch/x86/include/asm/page.h +32 -0
@@ -78,6 +78,38 @@ static __always_inline u64 __is_canonical_address(u64 vaddr, u8 vaddr_bits) return __canonical_address(vaddr, vaddr_bits) == vaddr; } +#ifdef CONFIG_X86_PAE + +/* +* In PAE mode, we need to do a cr3 reload (=tlb flush) when +* updating the top-level pagetable entries to guarantee the +* processor notices the update. Since this is expensive, and +* all 4 top-level entries are used almost immediately in a +* new process's life, we just pre-populate them here. +*/ +#define PREALLOCATED_PMDS PTRS_PER_PGD +/* +* "USER_PMDS" are the PMDs for the user copy of the page tables when +* PTI is enabled. They do not exist when PTI is disabled. Note that +* this is distinct from the user _portion_ of the kernel page tables +* which always exists. +* +* We allocate separate PMDs for the kernel part of the user page-table +* when PTI is enabled. We need them to map the per-process LDT into the +* user-space page-table. +*/ +#define PREALLOCATED_USER_PMDS (boot_cpu_has(X86_FEATURE_PTI) ? KERNEL_PGD_PTRS : 0) +#define MAX_PREALLOCATED_USER_PMDS KERNEL_PGD_PTRS + +#else /* !CONFIG_X86_PAE */ + +/* No need to prepopulate any pagetable entries in non-PAE modes. */ +#define PREALLOCATED_PMDS 0 +#define PREALLOCATED_USER_PMDS 0 +#define MAX_PREALLOCATED_USER_PMDS 0 + +#endif /* CONFIG_X86_PAE */ +#endif /* __ASSEMBLER__ */ #include <asm-generic/memory_model.h>
arch/x86/include/asm/pgtable_32_areas.h +7 -2
@@ -30,9 +30,14 @@ extern bool __vmalloc_start_set; /* set once high_memory is set */#define CPU_ENTRY_AREA_BASE \ ((FIXADDR_TOT_START - PAGE_SIZE*(CPU_ENTRY_AREA_PAGES+1)) & PMD_MASK) -#define LDT_BASE_ADDR \ - ((CPU_ENTRY_AREA_BASE - PAGE_SIZE) & PMD_MASK) +/* +* On 32-bit the mm-local region is currently completely consumed by the LDT +* remap. +*/ +#define MM_LOCAL_BASE_ADDR ((CPU_ENTRY_AREA_BASE - PAGE_SIZE) & PMD_MASK) +#define MM_LOCAL_END_ADDR (MM_LOCAL_BASE_ADDR + PMD_SIZE) +#define LDT_BASE_ADDR MM_LOCAL_BASE_ADDR#define LDT_END_ADDR (LDT_BASE_ADDR + PMD_SIZE)#define PKMAP_BASE \
arch/x86/include/asm/pgtable_64_types.h +9 -3
@@ -5,8 +5,11 @@#include <asm/sparsemem.h> #ifndef __ASSEMBLER__ +#include <linux/build_bug.h> #include <linux/types.h> #include <asm/kaslr.h> +#include <asm/page_types.h> +#include <uapi/asm/ldt.h> /* * These are used to make use of C type-checking..
@@ -100,9 +103,12 @@ extern unsigned int ptrs_per_p4d;#define GUARD_HOLE_BASE_ADDR (GUARD_HOLE_PGD_ENTRY << PGDIR_SHIFT)#define GUARD_HOLE_END_ADDR (GUARD_HOLE_BASE_ADDR + GUARD_HOLE_SIZE) -#define LDT_PGD_ENTRY -240UL -#define LDT_BASE_ADDR (LDT_PGD_ENTRY << PGDIR_SHIFT) -#define LDT_END_ADDR (LDT_BASE_ADDR + PGDIR_SIZE) +#define MM_LOCAL_PGD_ENTRY -240UL +#define MM_LOCAL_BASE_ADDR (MM_LOCAL_PGD_ENTRY << PGDIR_SHIFT) +#define MM_LOCAL_END_ADDR ((MM_LOCAL_PGD_ENTRY + 1) << PGDIR_SHIFT) + +#define LDT_BASE_ADDR MM_LOCAL_BASE_ADDR +#define LDT_END_ADDR (LDT_BASE_ADDR + PMD_SIZE)#define __VMALLOC_BASE_L4 0xffffc90000000000UL#define __VMALLOC_BASE_L5 0xffa0000000000000UL
arch/x86/kernel/ldt.c +19 -111
196 lines not on this page · raw message
arch/x86/mm/pgtable.c +3 -29
@@ -111,29 +111,6 @@ static void pgd_dtor(pgd_t *pgd)*/ #ifdef CONFIG_X86_PAE -/* -* In PAE mode, we need to do a cr3 reload (=tlb flush) when -* updating the top-level pagetable entries to guarantee the -* processor notices the update. Since this is expensive, and -* all 4 top-level entries are used almost immediately in a -* new process's life, we just pre-populate them here. -*/ -#define PREALLOCATED_PMDS PTRS_PER_PGD - -/* -* "USER_PMDS" are the PMDs for the user copy of the page tables when -* PTI is enabled. They do not exist when PTI is disabled. Note that -* this is distinct from the user _portion_ of the kernel page tables -* which always exists. -* -* We allocate separate PMDs for the kernel part of the user page-table -* when PTI is enabled. We need them to map the per-process LDT into the -* user-space page-table. -*/ -#define PREALLOCATED_USER_PMDS (boot_cpu_has(X86_FEATURE_PTI) ? \ - KERNEL_PGD_PTRS : 0) -#define MAX_PREALLOCATED_USER_PMDS KERNEL_PGD_PTRS -void pud_populate (struct mm_struct *mm,pud_t *pudp,pmd_t *pmd) {paravirt_alloc_pmd (mm,__pa (pmd) >>PAGE_SHIFT );
@@ -150,12 +127,6 @@ void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmd)*/ flush_tlb_mm(mm); } -#else /* !CONFIG_X86_PAE */ - -/* No need to prepopulate any pagetable entries in non-PAE modes. */ -#define PREALLOCATED_PMDS 0 -#define PREALLOCATED_USER_PMDS 0 -#define MAX_PREALLOCATED_USER_PMDS 0#endif /* CONFIG_X86_PAE */ static void free_pmds (struct mm_struct *mm,pmd_t *pmds[],int count)
@@ -375,6 +346,9 @@ pgd_t *pgd_alloc(struct mm_struct *mm)void pgd_free (struct mm_struct *mm,pgd_t *pgd) { +/* Should be cleaned up in mmap exit path. */ +VM_WARN_ON_ONCE (mm_local_region_used (mm)); +pgd_mop_up_pmds (mm, pgd);pgd_dtor (pgd);paravirt_pgd_free (mm, pgd);
include/linux/mm.h +13 -0
@@ -906,6 +906,19 @@ static inline void mm_flags_clear_all(struct mm_struct *mm)bitmap_zero (ACCESS_PRIVATE (&mm->flags, __mm_flags),NUM_MM_FLAG_BITS ); } +#ifdef CONFIG_MM_LOCAL_REGION +static inline bool mm_local_region_used (struct mm_struct *mm) +{ +return mm_flags_test (MMF_LOCAL_REGION_USED , mm); +} +#else +static inline bool mm_local_region_used (struct mm_struct *mm) +{ +VM_WARN_ON_ONCE (mm_flags_test (MMF_LOCAL_REGION_USED , mm)); +return false; +} +#endif +extern const struct vm_operations_struct vma_dummy_vm_ops;static inline void vma_init (struct vm_area_struct *vma,struct mm_struct *mm)
include/linux/mm_types.h +2 -0
@@ -1944,6 +1944,8 @@ enum {#define MMF_USER_HWCAP 32/* user-defined HWCAPs */ +#define MMF_LOCAL_REGION_USED 33 +#define MMF_INIT_LEGACY_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\ MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK)
kernel/fork.c +1 -0
@@ -1153,6 +1153,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, fail_nocontext:mm_free_id (mm); fail_noid: +WARN_ON_ONCE (mm_local_region_used (mm));mm_free_pgd (mm); fail_nopgd:futex_hash_free (mm);
mm/Kconfig +11 -0
@@ -1319,6 +1319,10 @@ config SECRETMEM default y bool "Enable memfd_secret() system call" if EXPERT depends on ARCH_HAS_SET_DIRECT_MAP + # Soft dependency, for optimisation. + imply MM_LOCAL_REGION + imply MERMAP + imply PAGE_ALLOC_UNMAPPED help Enable the memfd_secret() system call with the ability to create memory areas visible only in the context of the owning process and
@@ -1471,6 +1475,13 @@ config LAZY_MMU_MODE_KUNIT_TEST If unsure, say N. +config ARCH_SUPPORTS_MM_LOCAL_REGION + def_bool n + +config MM_LOCAL_REGION + bool + depends on ARCH_SUPPORTS_MM_LOCAL_REGION + source "mm/damon/Kconfig" endmenu
Signature · 1 line
2.51.2
Re: [PATCH v2 02/22] x86/mm: Generalize LDT remap into "mm-local region"
Quoted 6 lines · Brendan Jackman
arch/x86/include/asm/mmu_context.h +5
-0+#ifdef CONFIG_MM_LOCAL_REGION +static inline void mm_local_region_free (struct mm_struct *mm) +{ +if (mm_local_region_used (mm)) { +struct mmu_gather tlb;
arch/x86/include/asm/mmu_context.h static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) · line 240
Quoted 10 lines
arch/x86/include/asm/mmu_context.h +10
-0+unsigned long start =MM_LOCAL_BASE_ADDR ; +unsigned long end =MM_LOCAL_END_ADDR ; + +/* +* Although free_pgd_range() is intended for freeing user +* page-tables, it also works out for kernel mappings on x86. +* We use tlb_gather_mmu_fullmm() to avoid confusing the +* range-tracking logic in __tlb_adjust_range(). +*/ +tlb_gather_mmu_fullmm (&tlb, mm);
These are superficial nits and I need to go through this series in actual detail, but here are the nits:
Indentation is bad. What you have here double-indents the whole function.
Do this:
struct mmu_gather tlb; unsigned long start = MM_LOCAL_BASE_ADDR; unsigned long end = MM_LOCAL_END_ADDR; if (!mm_local_region_used(mm)) return; ... rest of code here
arch/x86/include/asm/mmu_context.h static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) · line 239
Quoted 3 lines
arch/x86/include/asm/mmu_context.h +3
-0+* We use tlb_gather_mmu_fullmm() to avoid confusing the +* range-tracking logic in __tlb_adjust_range(). +*/
Imperative voice, please.
And a meta-comment:
Quoted 7 lines
Documentation/arch/x86/x86_64/mm.rst | 4 +- arch/x86/Kconfig | 2 + arch/x86/include/asm/mmu_context.h | 119 ++++++++++++++++++++++++++++- arch/x86/include/asm/page.h | 32 ++++++++ arch/x86/include/asm/pgtable_32_areas.h | 9 ++- arch/x86/include/asm/pgtable_64_types.h | 12 ++- arch/x86/kernel/ldt.c | 130 +++++---------------------------
This is too big and there's too much going on here. This is doing a few logical things like both introducing mm-local regions *and* making the LDT remap one of them.
Re: [PATCH v2 02/22] x86/mm: Generalize LDT remap into "mm-local region"
Quoted 12 lines · Dave Hansen
Quoted 11 lines · Brendan Jackman
arch/x86/include/asm/mmu_context.h +10
-0+#ifdef CONFIG_MM_LOCAL_REGION +static inline void mm_local_region_free (struct mm_struct *mm) +{ +if (mm_local_region_used (mm)) { +struct mmu_gather tlb; +unsigned long start =MM_LOCAL_BASE_ADDR ; +unsigned long end =MM_LOCAL_END_ADDR ; + +/* +* Although free_pgd_range() is intended for freeing user
Quoted 8 lines
arch/x86/include/asm/mmu_context.h static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) · line 240
Quoted 5 lines
arch/x86/include/asm/mmu_context.h +5
-0+* page-tables, it also works out for kernel mappings on x86. +* We use tlb_gather_mmu_fullmm() to avoid confusing the +* range-tracking logic in __tlb_adjust_range(). +*/ +tlb_gather_mmu_fullmm (&tlb, mm);These are superficial nits and I need to go through this series in actual detail,
Thanks, this series is pretty brutal so I'm very happy to receive incremental reviews!
Quoted 6 lines
but here are the nits:
Indentation is bad. What you have here double-indents the whole function.
Do this:
Quoted 8 lines
struct mmu_gather tlb; unsigned long start = MM_LOCAL_BASE_ADDR; unsigned long end = MM_LOCAL_END_ADDR; if (!mm_local_region_used(mm)) return; ... rest of code here
Ack
Quoted 6 lines
arch/x86/include/asm/mmu_context.h static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) · line 239
Quoted 4 lines
arch/x86/include/asm/mmu_context.h +3
-0+* We use tlb_gather_mmu_fullmm() to avoid confusing the +* range-tracking logic in __tlb_adjust_range(). +*/ Imperative voice, please.
Yeah I don't think I'm ever gonna stop making this mistake. Any LLM should be able to catch this for me, I think it's time to find a way to get that into my pre-mail workflow.
Quoted 10 lines
And a meta-comment:
Quoted 7 lines
Documentation/arch/x86/x86_64/mm.rst | 4 +- arch/x86/Kconfig | 2 + arch/x86/include/asm/mmu_context.h | 119 ++++++++++++++++++++++++++++- arch/x86/include/asm/page.h | 32 ++++++++ arch/x86/include/asm/pgtable_32_areas.h | 9 ++- arch/x86/include/asm/pgtable_64_types.h | 12 ++- arch/x86/kernel/ldt.c | 130 +++++---------------------------
Quoted 3 lines
This is too big and there's too much going on here. This is doing a few logical things like both introducing mm-local regions *and* making the LDT remap one of them.
IIRC I tried this but having both an mm-local region and a separate LDT remap at the same time is annoying, it would mean reviewing temporary code to deal with both existing at the same time.
However I just had an idea: I will try creating the mm-local region but with size 0. Then in a separate patch I'll expand it and simultaneously move the LDT remap into it.
Re: [PATCH v2 02/22] x86/mm: Generalize LDT remap into "mm-local region"
TANGENT - off topic, removing most people from CC.
Quoted 11 lines · Brendan Jackman
Quoted 5 lines · Dave Hansen
arch/x86/include/asm/mmu_context.h static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) · line 239
Quoted 3 lines
arch/x86/include/asm/mmu_context.h +3
-0+* We use tlb_gather_mmu_fullmm() to avoid confusing the +* range-tracking logic in __tlb_adjust_range(). +*/ Imperative voice, please.
Yeah I don't think I'm ever gonna stop making this mistake. Any LLM should be able to catch this for me, I think it's time to find a way to get that into my pre-mail workflow.
Just dumping what I learned from briefly looking into this:
It looks like Sashiko [0] and Chris Mason's review-prompts are not really well geared-up to deal with trivialities like this right now, they are still evolving fast and focussing on much more advanced topics, and AFAICS they don't have a standardised way for the agent to "shell out" to a cheap model to do simple checks like this. So for now until that stuff crystallises a bit more I'll just use a dumb standalone script.
I wrote a quick prompt to check for these particular rules and found that the "pro" model worked perfectly but took ages (and probably an obscene amount of energy) while gemini-2.5-flash-lite was instant but very unreliable. Then I asked the pro model to rework the prompt for the benefit of the small model. Its version made the small model works reliably.
I'll paste the prompt below. The command to run it using Google's stuff is:
gemini --prompt "$(cat check_patch.md) $(git show)" --model "gemini-3.1-flash-lite-preview"
I assume open models that fit on a laptop can handle this task too but I haven't tried it as Google's tooling seems to be hardcoded to funnel you to the cloud service. Yuck, something to figure out on the weekend I suppose.
(Alternatively I bet a plain old NLTK script can handle these particular rules. But that will run into limiations quickly while dumb LLMs are generic).
[0] https://lwn.net/ml/all/[email protected]/
---
You are a strict code reviewer. You will be given a patch file, formatted email, or Git diff.
Your only task is to review the English style of newly added code comments (lines starting with '+' that are comments, e.g., '+ //', '+ /*', '+ *', or '+ #'). Ignore all actual code, variable names, and removed lines.
Flag a comment if it violates either of these two rules:
1. Avoid personal pronouns. For example: Do not use: I, we, you, our, us, my, your. Other pronouns such as "it" are fine.
2. Use the imperative mood to describe what the code does. (e.g., Use "Return the value" instead of "Returns the value" or "This returns the value").
Output format:
If there are no violations, output exactly: "LGTM".
If there are violations, output the snippet from the input where the violation occurs. Prefix each line with a '>' character, followed by a brief description of the violated rule.
### Example 1 (Pronoun Violation) ###
Input Patch:
+ /*
+ * Although free_pgd_range() is intended for freeing user + * page-tables, it also works out for kernel mappings on x86. + * We use tlb_gather_mmu_fullmm() to avoid confusing the + * range-tracking logic in __tlb_adjust_range(). + */
+ tlb_gather_mmu_fullmm(&tlb, mm);
Output:
arch/x86/include/asm/mmu_context.h static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) · line 238
Quoted 2 lines
arch/x86/include/asm/mmu_context.h +2
-0+* We use tlb_gather_mmu_fullmm() to avoid confusing the +* range-tracking logic in __tlb_adjust_range().
Avoid personal pronouns ("We").
### Example 2 (Imperative Mood Violation) ### Input Patch:
+ // Initializes the counter and prepares the struct.
+ counter = 0;
Output:
Quoted 1 line
+ // Initializes the counter and prepares the struct.
Use the imperative mood (e.g., "Initialize the counter...").
<END INSTRUCTIONS>
<BEGIN PATCH FOR REVIEW>
Re: [PATCH v2 02/22] x86/mm: Generalize LDT remap into "mm-local region"
In reply to Brendan JackmanSummarizing Sashiko review [0] so all the comments are in the same place...
https://sashiko.dev/#/patchset/20260320-page_alloc-unmapped-v2-0-28bf1bd54f41%40google.com
Quoted 400 lines · Brendan Jackman
Various security features benefit from having process-local address mappings. Examples include no-direct-map
guest_memfd[2] and significant optimizations for ASI [1].As pointed out by Andy in [0], x86 already has a PGD entry that is local to the mm, which is used for the LDT.
So, simply redefine that entry's region as "the mm-local region" and then redefine the LDT region as a sub-region of that.
With the currently-envisaged usecases, there will be many situations where almost no processes have any need for the mm-local region. Therefore, avoid its overhead (memory cost of pagetables, alloc/free overhead during fork/exit) for processes that don't use it by requiring its users to explicitly initialize it via the new mm_local_* API.
This means that the LDT remap code can be simplified:
1.
map_ldt_struct_to_user()andfree_ldt_pgtables()are no longer required as themm_localcore code handles that automatically.2. The sanity-check logic is unified: in both cases just walk the pagetables via a generic mechanism. This slightly relaxes the sanity-checking since
lookup_address_in_pgd()is more flexible thanpgd_to_pmd_walk(), but this seems to be worth it for the simplified code.On 64-bit, the mm-local region gets a whole PGD. On 32-bit, it just i.e. one PMD, i.e. it is completely consumed by the LDT remap - no investigation has been done into whether it's feasible to expand the region on 32-bit. Most likely there is no strong usecase for that anyway.
In both cases, in order to combine the need for an on-demand mm initialisation, combined with the desire to transparently handle propagating mappings to userspace under KPTI, the user and kernel pagetables are shared at the highest level possible. For PAE that means the PTE table is shared and for 64-bit the P4D/PUD. This is implemented by pre-allocating the first shared table when the mm-local region is first initialised.
The PAE implementation of
mm_local_map_to_user()does not allocate pagetables, it assumes the PMD has been preallocated. To make that assumption safer, exposePREALLOCATED_PMDsin the arch headers so thatmm_local_map_to_user()can have aBUILD_BUG_ON().[0] https://lore.kernel.org/linux-mm/CALCETrXHbS9VXfZ80kOjiTrreM2EbapYeGp68mvJPbosUtorYA@mail.gmail.com/
[1] https://linuxasi.dev/
[2] https://lore.kernel.org/all/[email protected]Signed-off-by Brendan Jackman <[email protected]>--- Documentation/arch/x86/x86_64/mm.rst | 4 +- arch/x86/Kconfig | 2 + arch/x86/include/asm/mmu_context.h | 119 ++++++++++++++++++++++++++++- arch/x86/include/asm/page.h | 32 ++++++++ arch/x86/include/asm/pgtable_32_areas.h | 9 ++- arch/x86/include/asm/pgtable_64_types.h | 12 ++- arch/x86/kernel/ldt.c | 130 +++++--------------------------- arch/x86/mm/pgtable.c | 32 +------- include/linux/mm.h | 13 ++++ include/linux/mm_types.h | 2 + kernel/fork.c | 1 + mm/Kconfig | 11 +++ 12 files changed, 217 insertions(+), 150 deletions(-)Documentation/arch/x86/x86_64/mm.rst +2
-2@@ -53,7 +53,7 @@ Complete virtual memory map with 4-level page tables ____________________________________________________________|___________________________________________________________ | | | | ffff800000000000 | -128 TB | ffff87ffffffffff | 8 TB | ... guard hole, also reserved for hypervisor - ffff880000000000 | -120 TB | ffff887fffffffff | 0.5 TB | LDT remap for PTI + ffff880000000000 | -120 TB | ffff887fffffffff | 0.5 TB | MM-local kernel data. Includes LDT remap for PTI ffff888000000000 | -119.5 TB | ffffc87fffffffff | 64 TB | direct mapping of all physical memory (page_offset_base) ffffc88000000000 | -55.5 TB | ffffc8ffffffffff | 0.5 TB | ... unused hole ffffc90000000000 | -55 TB | ffffe8ffffffffff | 32 TB | vmalloc/ioremap space (vmalloc_base)@@ -123,7 +123,7 @@ Complete virtual memory map with 5-level page tables ____________________________________________________________|___________________________________________________________ | | | | ff00000000000000 | -64 PB | ff0fffffffffffff | 4 PB | ... guard hole, also reserved for hypervisor - ff10000000000000 | -60 PB | ff10ffffffffffff | 0.25 PB | LDT remap for PTI + ff10000000000000 | -60 PB | ff10ffffffffffff | 0.25 PB | MM-local kernel data. Includes LDT remap for PTI ff11000000000000 | -59.75 PB | ff90ffffffffffff | 32 PB | direct mapping of all physical memory (page_offset_base) ff91000000000000 | -27.75 PB | ff9fffffffffffff | 3.75 PB | ... unused hole ffa0000000000000 | -24 PB | ffd1ffffffffffff | 12.5 PB | vmalloc/ioremap space (vmalloc_base)arch/x86/Kconfig +2
-0@@ -133,6 +133,7 @@ config X86 select ARCH_SUPPORTS_RT select ARCH_SUPPORTS_AUTOFDO_CLANG select ARCH_SUPPORTS_PROPELLER_CLANG if X86_64 + select ARCH_SUPPORTS_MM_LOCAL_REGION if X86_64 || X86_PAE select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_CMPXCHG_LOCKREF if X86_CX8 select ARCH_USE_MEMTEST@@ -2323,6 +2324,7 @@ config CMDLINE_OVERRIDE config MODIFY_LDT_SYSCALL bool "Enable the LDT (local descriptor table)" if EXPERT default y + select MM_LOCAL_REGION if MITIGATION_PAGE_TABLE_ISOLATION || X86_PAE help Linux can allow user programs to install a per-process x86 Local Descriptor Table (LDT) using the modify_ldt(2) systemarch/x86/include/asm/mmu_context.h +116
-3@@ -8,8 +8,10 @@#include <trace/events/tlb.h> +#include <asm/tlb.h> #include <asm/tlbflush.h> #include <asm/paravirt.h> +#include <asm/pgalloc.h> #include <asm/debugreg.h> #include <asm/gsseg.h> #include <asm/desc.h> @@ -59,7 +61,6 @@ static inline void init_new_context_ldt(struct mm_struct *mm) }int ldt_dup_context (struct mm_struct *oldmm,struct mm_struct *mm);void destroy_context_ldt (struct mm_struct *mm); -void ldt_arch_exit_mmap (struct mm_struct *mm);#else /* CONFIG_MODIFY_LDT_SYSCALL */ static inline void init_new_context_ldt (struct mm_struct *mm) { } static inline int ldt_dup_context(struct mm_struct *oldmm,@@ -68,7 +69,6 @@ static inline int ldt_dup_context(struct mm_struct *oldmm, return 0; }static inline void destroy_context_ldt (struct mm_struct *mm) { } -static inline void ldt_arch_exit_mmap (struct mm_struct *mm) { }#endif #ifdef CONFIG_MODIFY_LDT_SYSCALL @@ -223,10 +223,123 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm)return ldt_dup_context (oldmm, mm); } +#ifdef CONFIG_MM_LOCAL_REGION +static inline void mm_local_region_free (struct mm_struct *mm) +{ +if (mm_local_region_used (mm)) { +struct mmu_gather tlb; +unsigned long start =MM_LOCAL_BASE_ADDR ; +unsigned long end =MM_LOCAL_END_ADDR ; + +/* +* Although free_pgd_range() is intended for freeing user +* page-tables, it also works out for kernel mappings on x86. +* We use tlb_gather_mmu_fullmm() to avoid confusing the +* range-tracking logic in __tlb_adjust_range(). +*/ +tlb_gather_mmu_fullmm (&tlb, mm); +free_pgd_range (&tlb, start, end, start, end); +tlb_finish_mmu (&tlb); + +mm_flags_clear (MMF_LOCAL_REGION_USED , mm); + } +} + +#if defined(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION ) && defined(CONFIG_X86_PAE ) +static inline pmd_t *pgd_to_pmd_walk (pgd_t *pgd,unsigned long va) +{ +p4d_t *p4d; +pud_t *pud; + +if (pgd->pgd ==0 ) +return NULL ; + + p4d =p4d_offset (pgd, va); +if (p4d_none (*p4d)) +return NULL ; + + pud =pud_offset (p4d, va); +if (pud_none (*pud)) +return NULL ; + +return pmd_offset (pud, va); +} + +static inline int mm_local_map_to_user (struct mm_struct *mm) +{ +BUILD_BUG_ON (!PREALLOCATED_PMDS ); +pgd_t *k_pgd =pgd_offset (mm,MM_LOCAL_BASE_ADDR ); +pgd_t *u_pgd =kernel_to_user_pgdp (k_pgd); +pmd_t *k_pmd, *u_pmd; +int err; + + k_pmd =pgd_to_pmd_walk (k_pgd,MM_LOCAL_BASE_ADDR ); + u_pmd =pgd_to_pmd_walk (u_pgd,MM_LOCAL_BASE_ADDR ); + +BUILD_BUG_ON (MM_LOCAL_END_ADDR -MM_LOCAL_BASE_ADDR >PMD_SIZE ); + +/* Preallocate the PTE table so it can be shared. */ + err =pte_alloc (mm, k_pmd); +if (err) +return err; + +/* Point the userspace PMD at the same PTE as the kernel PMD. */ +set_pmd (u_pmd, *k_pmd); +return 0 ; +} +#elif defined(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION ) +static inline int mm_local_map_to_user (struct mm_struct *mm) +{ +pgd_t *pgd; +int err; + + err =preallocate_sub_pgd (mm,MM_LOCAL_BASE_ADDR ); +if (err) +return err; + + pgd =pgd_offset (mm,MM_LOCAL_BASE_ADDR ); +set_pgd (kernel_to_user_pgdp (pgd), *pgd); +return 0 ; +} +#else +static inline int mm_local_map_to_user (struct mm_struct *mm) +{ +WARN_ONCE (1 ,"mm_local_map_to_user() not implemented" ); +return -EINVAL ; +} +#endif + +/* +* Do initial setup of the user-local region. Call from process context. +* +* Under PTI, userspace shares the pagetables for the mm-local region with the +* kernel (if you map stuff here, it's immediately mapped into userspace too). +* LDT remap. It's assuming nothing gets mapped in here that needs to be +* protected from Meltdown-type attacks from the current process. +*/ +static inline int mm_local_region_init (struct mm_struct *mm) +{ +int err; + +if (boot_cpu_has (X86_FEATURE_PTI )) { + err =mm_local_map_to_user (mm); +if (err) +return err; + } + +mm_flags_set (MMF_LOCAL_REGION_USED , mm); + +return 0 ; +} + +#else +static inline void mm_local_region_free (struct mm_struct *mm) { } +#endif /* CONFIG_MM_LOCAL_REGION */ +static inline void arch_exit_mmap (struct mm_struct *mm) {paravirt_arch_exit_mmap (mm); -ldt_arch_exit_mmap (mm); +mm_local_region_free (mm); }#ifdef CONFIG_X86_64 arch/x86/include/asm/page.h +32
-0@@ -78,6 +78,38 @@ static __always_inline u64 __is_canonical_address(u64 vaddr, u8 vaddr_bits) return __canonical_address(vaddr, vaddr_bits) == vaddr; } +#ifdef CONFIG_X86_PAE + +/* +* In PAE mode, we need to do a cr3 reload (=tlb flush) when +* updating the top-level pagetable entries to guarantee the +* processor notices the update. Since this is expensive, and +* all 4 top-level entries are used almost immediately in a +* new process's life, we just pre-populate them here. +*/ +#define PREALLOCATED_PMDS PTRS_PER_PGD +/* +* "USER_PMDS" are the PMDs for the user copy of the page tables when +* PTI is enabled. They do not exist when PTI is disabled. Note that +* this is distinct from the user _portion_ of the kernel page tables +* which always exists. +* +* We allocate separate PMDs for the kernel part of the user page-table +* when PTI is enabled. We need them to map the per-process LDT into the +* user-space page-table. +*/ +#define PREALLOCATED_USER_PMDS (boot_cpu_has(X86_FEATURE_PTI) ? KERNEL_PGD_PTRS : 0) +#define MAX_PREALLOCATED_USER_PMDS KERNEL_PGD_PTRS + +#else /* !CONFIG_X86_PAE */ + +/* No need to prepopulate any pagetable entries in non-PAE modes. */ +#define PREALLOCATED_PMDS 0 +#define PREALLOCATED_USER_PMDS 0 +#define MAX_PREALLOCATED_USER_PMDS 0 + +#endif /* CONFIG_X86_PAE */ +#endif /* __ASSEMBLER__ */ #include <asm-generic/memory_model.h> arch/x86/include/asm/pgtable_32_areas.h +7
-2@@ -30,9 +30,14 @@ extern bool __vmalloc_start_set; /* set once high_memory is set */#define CPU_ENTRY_AREA_BASE \ ((FIXADDR_TOT_START - PAGE_SIZE*(CPU_ENTRY_AREA_PAGES+1)) & PMD_MASK) -#define LDT_BASE_ADDR \ - ((CPU_ENTRY_AREA_BASE - PAGE_SIZE) & PMD_MASK) +/* +* On 32-bit the mm-local region is currently completely consumed by the LDT +* remap. +*/ +#define MM_LOCAL_BASE_ADDR ((CPU_ENTRY_AREA_BASE - PAGE_SIZE) & PMD_MASK) +#define MM_LOCAL_END_ADDR (MM_LOCAL_BASE_ADDR + PMD_SIZE) +#define LDT_BASE_ADDR MM_LOCAL_BASE_ADDR#define LDT_END_ADDR (LDT_BASE_ADDR + PMD_SIZE)#define PKMAP_BASE \arch/x86/include/asm/pgtable_64_types.h +9
-3@@ -5,8 +5,11 @@#include <asm/sparsemem.h> #ifndef __ASSEMBLER__ +#include <linux/build_bug.h> #include <linux/types.h> #include <asm/kaslr.h> +#include <asm/page_types.h> +#include <uapi/asm/ldt.h> /* * These are used to make use of C type-checking.. @@ -100,9 +103,12 @@ extern unsigned int ptrs_per_p4d;#define GUARD_HOLE_BASE_ADDR (GUARD_HOLE_PGD_ENTRY << PGDIR_SHIFT)#define GUARD_HOLE_END_ADDR (GUARD_HOLE_BASE_ADDR + GUARD_HOLE_SIZE) -#define LDT_PGD_ENTRY -240UL -#define LDT_BASE_ADDR (LDT_PGD_ENTRY << PGDIR_SHIFT) -#define LDT_END_ADDR (LDT_BASE_ADDR + PGDIR_SIZE) +#define MM_LOCAL_PGD_ENTRY -240UL +#define MM_LOCAL_BASE_ADDR (MM_LOCAL_PGD_ENTRY << PGDIR_SHIFT) +#define MM_LOCAL_END_ADDR ((MM_LOCAL_PGD_ENTRY + 1) << PGDIR_SHIFT) + +#define LDT_BASE_ADDR MM_LOCAL_BASE_ADDR +#define LDT_END_ADDR (LDT_BASE_ADDR + PMD_SIZE)#define __VMALLOC_BASE_L4 0xffffc90000000000UL#define __VMALLOC_BASE_L5 0xffa0000000000000ULarch/x86/kernel/ldt.c +13
-5@@ -31,6 +31,8 @@#include <xen/xen.h> +/* LDTs are double-buffered, the buffers are called slots. */ +#define LDT_NUM_SLOTS 2/* This is a multiple of PAGE_SIZE. */ #define LDT_SLOT_STRIDE (LDT_ENTRIES * LDT_ENTRY_SIZE)@@ -186,100 +188,36 @@ static struct ldt_struct *alloc_ldt_struct(unsigned int num_entries)#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION -static void do_sanity_check (struct mm_struct *mm, -bool had_kernel_mapping, -bool had_user_mapping) +static void sanity_check_ldt_mapping (struct mm_struct *mm) { +pgd_t *k_pgd =pgd_offset (mm,LDT_BASE_ADDR ); +pgd_t *u_pgd =kernel_to_user_pgdp (k_pgd); +unsigned int k_level, u_level; +bool had_kernel, had_user; + + had_kernel =lookup_address_in_pgd (k_pgd,LDT_BASE_ADDR , &k_level); + had_user =lookup_address_in_pgd (u_pgd,LDT_BASE_ADDR , &u_level); +if (mm->context.ldt) {/* * We already had an LDT. The top-level entry should already * have been allocated and synchronized with the usermode * tables. */ -WARN_ON (!had_kernel_mapping); +WARN_ON (!had_kernel);if (boot_cpu_has (X86_FEATURE_PTI )) -WARN_ON (!had_user_mapping); +WARN_ON (!had_user);
arch/x86/kernel/ldt.c static struct ldt_struct *alloc_ldt_struct(unsigned int num_entries) · line 217
Quoted 10 lines
arch/x86/kernel/ldt.c +2
-2}else {/* * This is the first time we're mapping an LDT for this process. * Sync the pgd to the usermode tables. */ -WARN_ON (had_kernel_mapping); +WARN_ON (had_kernel);if (boot_cpu_has (X86_FEATURE_PTI )) -WARN_ON (had_user_mapping); +WARN_ON (had_user);
But under PAE the PTE is preallocated. lookup_address_in_pgd() returns NULL if the address is unmapped at a higher level but for 4K specifically it returns a non-NULL pointer to a non-present PTE.
This WARNs immediately when I run the selftests so I suspect I broke this and then forgot to retest with PTI.
Quoted 77 lines
arch/x86/kernel/ldt.c +0
-70} } -#ifdef CONFIG_X86_PAE - -static pmd_t *pgd_to_pmd_walk (pgd_t *pgd,unsigned long va) -{ -p4d_t *p4d; -pud_t *pud; - -if (pgd->pgd ==0 ) -return NULL ; - - p4d =p4d_offset (pgd, va); -if (p4d_none (*p4d)) -return NULL ; - - pud =pud_offset (p4d, va); -if (pud_none (*pud)) -return NULL ; - -return pmd_offset (pud, va); -} - -static void map_ldt_struct_to_user (struct mm_struct *mm) -{ -pgd_t *k_pgd =pgd_offset (mm,LDT_BASE_ADDR ); -pgd_t *u_pgd =kernel_to_user_pgdp (k_pgd); -pmd_t *k_pmd, *u_pmd; - - k_pmd =pgd_to_pmd_walk (k_pgd,LDT_BASE_ADDR ); - u_pmd =pgd_to_pmd_walk (u_pgd,LDT_BASE_ADDR ); - -if (boot_cpu_has (X86_FEATURE_PTI ) && !mm->context.ldt) -set_pmd (u_pmd, *k_pmd); -} - -static void sanity_check_ldt_mapping (struct mm_struct *mm) -{ -pgd_t *k_pgd =pgd_offset (mm,LDT_BASE_ADDR ); -pgd_t *u_pgd =kernel_to_user_pgdp (k_pgd); -bool had_kernel, had_user; -pmd_t *k_pmd, *u_pmd; - - k_pmd =pgd_to_pmd_walk (k_pgd,LDT_BASE_ADDR ); - u_pmd =pgd_to_pmd_walk (u_pgd,LDT_BASE_ADDR ); - had_kernel = (k_pmd->pmd !=0 ); - had_user = (u_pmd->pmd !=0 ); - -do_sanity_check (mm, had_kernel, had_user); -} - -#else /* !CONFIG_X86_PAE */ - -static void map_ldt_struct_to_user (struct mm_struct *mm) -{ -pgd_t *pgd =pgd_offset (mm,LDT_BASE_ADDR ); - -if (boot_cpu_has (X86_FEATURE_PTI ) && !mm->context.ldt) -set_pgd (kernel_to_user_pgdp (pgd), *pgd); -} - -static void sanity_check_ldt_mapping (struct mm_struct *mm) -{ -pgd_t *pgd =pgd_offset (mm,LDT_BASE_ADDR ); -bool had_kernel = (pgd->pgd !=0 ); -bool had_user = (kernel_to_user_pgdp (pgd)->pgd !=0 ); - -do_sanity_check (mm, had_kernel, had_user); -} - -#endif /* CONFIG_X86_PAE */ -/* * If PTI is enabled, this maps the LDT into the kernelmode and * usermode tables for the given mm.
arch/x86/kernel/ldt.c map_ldt_struct(struct mm_struct *mm, struct ldt_struct *ldt, int slot) · line 236
Quoted 4 lines
arch/x86/kernel/ldt.c +1
-0@@ -295,6 +233,8 @@ map_ldt_struct(struct mm_struct *mm, struct ldt_struct *ldt, int slot)if (!boot_cpu_has (X86_FEATURE_PTI ))return 0 ; +mm_local_region_init (mm);
Need to handle errors...
It also seems to think there's a path where we allocate a pagetable in mm_local_region_init(), then fail without setting MMF_LOCAL_REGION_USED, and don't free the pagetable. I can't see the path it's talking about though.
Quoted 64 lines
arch/x86/kernel/ldt.c +1
-31+/* * Any given ldt_struct should have map_ldt_struct() called at most * once. @@ -339,9 +279,6 @@ map_ldt_struct(struct mm_struct *mm, struct ldt_struct *ldt, int slot)pte_unmap_unlock (ptep, ptl); } -/* Propagate LDT mapping to the user page-table */ -map_ldt_struct_to_user (mm); - ldt->slot = slot;return 0 ; }@@ -390,28 +327,6 @@ static void unmap_ldt_struct(struct mm_struct *mm, struct ldt_struct *ldt) }#endif /* CONFIG_MITIGATION_PAGE_TABLE_ISOLATION */ -static void free_ldt_pgtables (struct mm_struct *mm) -{ -#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION -struct mmu_gather tlb; -unsigned long start =LDT_BASE_ADDR ; -unsigned long end =LDT_END_ADDR ; - -if (!boot_cpu_has (X86_FEATURE_PTI )) -return ; - -/* -* Although free_pgd_range() is intended for freeing user -* page-tables, it also works out for kernel mappings on x86. -* We use tlb_gather_mmu_fullmm() to avoid confusing the -* range-tracking logic in __tlb_adjust_range(). -*/ -tlb_gather_mmu_fullmm (&tlb, mm); -free_pgd_range (&tlb, start, end, start, end); -tlb_finish_mmu (&tlb); -#endif -} -/* After calling this, the LDT is immutable. */ static void finalize_ldt_struct (struct ldt_struct *ldt) {@@ -472,7 +387,6 @@ int ldt_dup_context(struct mm_struct *old_mm, struct mm_struct *mm) retval =map_ldt_struct (mm, new_ldt,0 );if (retval) { -free_ldt_pgtables (mm);free_ldt_struct (new_ldt); goto out_unlock; }@@ -494,11 +408,6 @@ void destroy_context_ldt(struct mm_struct *mm) mm->context.ldt =NULL ; } -void ldt_arch_exit_mmap (struct mm_struct *mm) -{ -free_ldt_pgtables (mm); -} -static int read_ldt (void __user *ptr,unsigned long bytecount) {struct mm_struct *mm = current->mm;
arch/x86/kernel/ldt.c static int write_ldt(void __user *ptr, unsigned long bytecount, int oldmode) · line 559
Quoted 9 lines
arch/x86/kernel/ldt.c +2
-3@@ -645,10 +554,9 @@ static int write_ldt(void __user *ptr, unsigned long bytecount, int oldmode)/* * This only can fail for the first LDT setup. If an LDT is * already installed then the PTE page is already -* populated. Mop up a half populated page table. +* populated. */ -if (!WARN_ON_ONCE (old_ldt)) -free_ldt_pgtables (mm); +WARN_ON_ONCE (!old_ldt);
That should be WARN_ON_ONCE(old_ldt);
Quoted 138 lines
arch/x86/kernel/ldt.c +0
-0free_ldt_struct (new_ldt); goto out_unlock; }arch/x86/mm/pgtable.c +3
-29@@ -111,29 +111,6 @@ static void pgd_dtor(pgd_t *pgd)*/ #ifdef CONFIG_X86_PAE -/* -* In PAE mode, we need to do a cr3 reload (=tlb flush) when -* updating the top-level pagetable entries to guarantee the -* processor notices the update. Since this is expensive, and -* all 4 top-level entries are used almost immediately in a -* new process's life, we just pre-populate them here. -*/ -#define PREALLOCATED_PMDS PTRS_PER_PGD - -/* -* "USER_PMDS" are the PMDs for the user copy of the page tables when -* PTI is enabled. They do not exist when PTI is disabled. Note that -* this is distinct from the user _portion_ of the kernel page tables -* which always exists. -* -* We allocate separate PMDs for the kernel part of the user page-table -* when PTI is enabled. We need them to map the per-process LDT into the -* user-space page-table. -*/ -#define PREALLOCATED_USER_PMDS (boot_cpu_has(X86_FEATURE_PTI) ? \ - KERNEL_PGD_PTRS : 0) -#define MAX_PREALLOCATED_USER_PMDS KERNEL_PGD_PTRS -void pud_populate (struct mm_struct *mm,pud_t *pudp,pmd_t *pmd) {paravirt_alloc_pmd (mm,__pa (pmd) >>PAGE_SHIFT );@@ -150,12 +127,6 @@ void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmd)*/ flush_tlb_mm(mm); } -#else /* !CONFIG_X86_PAE */ - -/* No need to prepopulate any pagetable entries in non-PAE modes. */ -#define PREALLOCATED_PMDS 0 -#define PREALLOCATED_USER_PMDS 0 -#define MAX_PREALLOCATED_USER_PMDS 0#endif /* CONFIG_X86_PAE */ static void free_pmds (struct mm_struct *mm,pmd_t *pmds[],int count)@@ -375,6 +346,9 @@ pgd_t *pgd_alloc(struct mm_struct *mm)void pgd_free (struct mm_struct *mm,pgd_t *pgd) { +/* Should be cleaned up in mmap exit path. */ +VM_WARN_ON_ONCE (mm_local_region_used (mm)); +pgd_mop_up_pmds (mm, pgd);pgd_dtor (pgd);paravirt_pgd_free (mm, pgd);include/linux/mm.h +13
-0@@ -906,6 +906,19 @@ static inline void mm_flags_clear_all(struct mm_struct *mm)bitmap_zero (ACCESS_PRIVATE (&mm->flags, __mm_flags),NUM_MM_FLAG_BITS ); } +#ifdef CONFIG_MM_LOCAL_REGION +static inline bool mm_local_region_used (struct mm_struct *mm) +{ +return mm_flags_test (MMF_LOCAL_REGION_USED , mm); +} +#else +static inline bool mm_local_region_used (struct mm_struct *mm) +{ +VM_WARN_ON_ONCE (mm_flags_test (MMF_LOCAL_REGION_USED , mm)); +return false; +} +#endif +extern const struct vm_operations_struct vma_dummy_vm_ops;static inline void vma_init (struct vm_area_struct *vma,struct mm_struct *mm)include/linux/mm_types.h +2
-0@@ -1944,6 +1944,8 @@ enum {#define MMF_USER_HWCAP 32/* user-defined HWCAPs */ +#define MMF_LOCAL_REGION_USED 33 +#define MMF_INIT_LEGACY_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\ MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK)kernel/fork.c +1
-0@@ -1153,6 +1153,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, fail_nocontext:mm_free_id (mm); fail_noid: +WARN_ON_ONCE (mm_local_region_used (mm));mm_free_pgd (mm); fail_nopgd:futex_hash_free (mm);mm/Kconfig +11
-0@@ -1319,6 +1319,10 @@ config SECRETMEM default y bool "Enable memfd_secret() system call" if EXPERT depends on ARCH_HAS_SET_DIRECT_MAP + # Soft dependency, for optimisation. + imply MM_LOCAL_REGION + imply MERMAP + imply PAGE_ALLOC_UNMAPPED help Enable the memfd_secret() system call with the ability to create memory areas visible only in the context of the owning process and@@ -1471,6 +1475,13 @@ config LAZY_MMU_MODE_KUNIT_TEST If unsure, say N. +config ARCH_SUPPORTS_MM_LOCAL_REGION + def_bool n + +config MM_LOCAL_REGION + bool + depends on ARCH_SUPPORTS_MM_LOCAL_REGION + source "mm/damon/Kconfig" endmenu
PATCH v2 3/22 x86/tlb: Expose some flush function declarations to modules
In reply to Brendan JackmanIn commit bfe3d8f6313d ("x86/tlb: Restrict access to tlbstate") some low-level logic (the important detail here is flush_tlb_info) was hidden from modules, along with functions associated with that data.
Later, the set of functions defined here changed and there are now a bunch of flush_tlb_*() functions that do not depend on x86 internals like flush_tlb_info.
This leads to some build fragility: KVM (which can be a module) cares about TLB flushing and includes {linux->asm}/mmu_context.h which includes asm/tlb.h and asm/tlbflush.h. This x86 TLB code expects these helpers to be defined (e.g. tlb_flush() calls flush_tlb_mm_range()).
Modules probably shouldn't call these helpers - luckily this is already enforced by the lack of EXPORT_SYMBOL(). Therefore keep things simple and just expose the declarations anyway to prevent build failures.
--- arch/x86/include/asm/tlbflush.h | 43 +++++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 21 deletions(-)
arch/x86/include/asm/tlbflush.h +22 -21
@@ -251,7 +251,6 @@ struct flush_tlb_info {u8 trim_cpumask; }; -void flush_tlb_local (void );void flush_tlb_one_user (unsigned long addr);void flush_tlb_one_kernel (unsigned long addr); void flush_tlb_multi(const struct cpumask *cpumask,
@@ -325,26 +324,6 @@ static inline void mm_clear_asid_transition(struct mm_struct *mm) { }static inline bool mm_in_asid_transition (struct mm_struct *mm) {return false; }#endif /* CONFIG_BROADCAST_TLB_FLUSH */ -#define flush_tlb_mm (mm) \ - flush_tlb_mm_range(mm, 0UL, TLB_FLUSH_ALL, 0UL, true) - -#define flush_tlb_range (vma, start, end) \ - flush_tlb_mm_range((vma)->vm_mm, start, end, \ - ((vma)->vm_flags & VM_HUGETLB) \ - ? huge_page_shift(hstate_vma(vma)) \ - : PAGE_SHIFT, true) - -extern void flush_tlb_all (void ); -extern void flush_tlb_mm_range (struct mm_struct *mm,unsigned long start, -unsigned long end,unsigned int stride_shift, -bool freed_tables); -extern void flush_tlb_kernel_range (unsigned long start,unsigned long end); - -static inline void flush_tlb_page (struct vm_area_struct *vma,unsigned long a) -{ -flush_tlb_mm_range (vma->vm_mm, a, a +PAGE_SIZE ,PAGE_SHIFT , false); -} -static inline bool arch_tlbbatch_should_defer (struct mm_struct *mm) {bool should_defer = false;
@@ -513,4 +492,26 @@ static inline void __native_tlb_flush_global(unsigned long cr4)native_write_cr4 (cr4 ^X86_CR4_PGE ); native_write_cr4(cr4); } + +#define flush_tlb_mm (mm) \ + flush_tlb_mm_range(mm, 0UL, TLB_FLUSH_ALL, 0UL, true) + +#define flush_tlb_range (vma, start, end) \ + flush_tlb_mm_range((vma)->vm_mm, start, end, \ + ((vma)->vm_flags & VM_HUGETLB) \ + ? huge_page_shift(hstate_vma(vma)) \ + : PAGE_SHIFT, true) + +void flush_tlb_local (void ); +extern void flush_tlb_all (void ); +extern void flush_tlb_mm_range (struct mm_struct *mm,unsigned long start, +unsigned long end,unsigned int stride_shift, +bool freed_tables); +extern void flush_tlb_kernel_range (unsigned long start,unsigned long end); + +static inline void flush_tlb_page (struct vm_area_struct *vma,unsigned long a) +{ +flush_tlb_mm_range (vma->vm_mm, a, a +PAGE_SIZE ,PAGE_SHIFT , false); +} +#endif /* _ASM_X86_TLBFLUSH_H */
Signature · 1 line
2.51.2
PATCH v2 4/22 mm: Create flags arg for __apply_to_page_range()
In reply to Brendan JackmanPreparatory patch, no functional change intended.
To prepare for making this function more generic, convert the boolean "create" arg into a flags arg with a single flag that has the same meaning.
--- mm/internal.h | 10 ++++++++++ mm/memory.c | 29 +++++++++++++++++------------ 2 files changed, 27 insertions(+), 12 deletions(-)
mm/internal.h +10 -0
@@ -1870,4 +1870,14 @@ static inline int get_sysctl_max_map_count(void) return READ_ONCE(sysctl_max_map_count); } +/* +* Create a mapping if it doesn't exist. (Otherwise, skip regions with no +* existing mapping, and return an error for regions with no leaf pagetable). +*/ +#define PGRANGE_CREATE (1 << 0) + +int __apply_to_page_range (struct mm_struct *mm,unsigned long addr, +unsigned long size,pte_fn_t fn, +void *data,unsigned int flags); +#endif /* __MM_INTERNAL_H */
mm/memory.c +17 -12
@@ -3208,9 +3208,10 @@ EXPORT_SYMBOL(vm_iomap_memory);static int apply_to_pte_range (struct mm_struct *mm,pmd_t *pmd,unsigned long addr,unsigned long end, -pte_fn_t fn,void *data,bool create, +pte_fn_t fn,void *data,unsigned int flags,pgtbl_mod_mask *mask) { +bool create = flags &PGRANGE_CREATE ;pte_t *pte, *mapped_pte;int err =0 ;spinlock_t *ptl;
@@ -3251,10 +3252,11 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,static int apply_to_pmd_range (struct mm_struct *mm,pud_t *pud,unsigned long addr,unsigned long end, -pte_fn_t fn,void *data,bool create, +pte_fn_t fn,void *data,unsigned int flags,pgtbl_mod_mask *mask) {pmd_t *pmd; +bool create = flags &PGRANGE_CREATE ;unsigned long next;int err =0 ;
@@ -3279,7 +3281,7 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud,pmd_clear_bad (pmd); } err =apply_to_pte_range (mm, pmd, addr, next, - fn, data, create, mask); + fn, data, flags, mask);if (err)break ; }while (pmd++, addr = next, addr != end);
@@ -3289,10 +3291,11 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud,static int apply_to_pud_range (struct mm_struct *mm,p4d_t *p4d,unsigned long addr,unsigned long end, -pte_fn_t fn,void *data,bool create, +pte_fn_t fn,void *data,unsigned int flags,pgtbl_mod_mask *mask) {pud_t *pud; +bool create = flags &PGRANGE_CREATE ;unsigned long next;int err =0 ;
@@ -3325,10 +3328,11 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d,static int apply_to_p4d_range (struct mm_struct *mm,pgd_t *pgd,unsigned long addr,unsigned long end, -pte_fn_t fn,void *data,bool create, +pte_fn_t fn,void *data,unsigned int flags,pgtbl_mod_mask *mask) {p4d_t *p4d; +bool create = flags &PGRANGE_CREATE ;unsigned long next;int err =0 ;
@@ -3351,7 +3355,7 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd,p4d_clear_bad (p4d); } err =apply_to_pud_range (mm, p4d, addr, next, - fn, data, create, mask); + fn, data, flags, mask);if (err)break ; }while (p4d++, addr = next, addr != end);
@@ -3359,11 +3363,12 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd,return err; } -static int __apply_to_page_range (struct mm_struct *mm,unsigned long addr, -unsigned long size,pte_fn_t fn, -void *data,bool create) +int __apply_to_page_range (struct mm_struct *mm,unsigned long addr, +unsigned long size,pte_fn_t fn, +void *data,unsigned int flags) {pgd_t *pgd; +bool create = flags &PGRANGE_CREATE ;unsigned long start = addr, next;unsigned long end = addr + size;pgtbl_mod_mask mask =0 ;
@@ -3387,7 +3392,7 @@ static int __apply_to_page_range(struct mm_struct *mm, unsigned long addr,pgd_clear_bad (pgd); } err =apply_to_p4d_range (mm, pgd, addr, next, - fn, data, create, &mask); + fn, data, flags, &mask);if (err)break ; }while (pgd++, addr = next, addr != end);
@@ -3405,7 +3410,7 @@ static int __apply_to_page_range(struct mm_struct *mm, unsigned long addr,int apply_to_page_range (struct mm_struct *mm,unsigned long addr,unsigned long size,pte_fn_t fn,void *data) { -return __apply_to_page_range (mm, addr, size, fn, data, true); +return __apply_to_page_range (mm, addr, size, fn, data,PGRANGE_CREATE ); }EXPORT_SYMBOL_GPL (apply_to_page_range);
@@ -3419,7 +3424,7 @@ EXPORT_SYMBOL_GPL(apply_to_page_range);int apply_to_existing_page_range (struct mm_struct *mm,unsigned long addr,unsigned long size,pte_fn_t fn,void *data) { -return __apply_to_page_range (mm, addr, size, fn, data, false); +return __apply_to_page_range (mm, addr, size, fn, data,0 ); }/*
Signature · 1 line
2.51.2
PATCH v2 5/22 mm: Add more flags for __apply_to_page_range()
In reply to Brendan JackmanAdd two flags to make this API more generic:
1. Separate "create" into two levels - one to allow creating new mappings without allocating pagetables, and one for the current behaviour that allows both of these.
2. Create a new flag to report that the caller has taken care of synchronization and no locks are required.
Both of these will serve to allow calling this API from restricted contexts where allocation and pagetable locking are not possible.
--- mm/internal.h | 19 ++++++++++++++++++- mm/memory.c | 59 ++++++++++++++++++++++++++++++++++------------------------- 2 files changed, 52 insertions(+), 26 deletions(-)
mm/internal.h +18 -1
@@ -1872,9 +1872,26 @@ static inline int get_sysctl_max_map_count(void)/* * Create a mapping if it doesn't exist. (Otherwise, skip regions with no -* existing mapping, and return an error for regions with no leaf pagetable). +* existing mapping). Most users will want PGRANGE_ALLOC or 0 instead. */ #define PGRANGE_CREATE (1 << 0) +/* +* Allocate a pagetable if one is missing. (Otherwise, return an error for +* regions with no leaf pagetable). Also implies PGRANGE_CREATE. +*/ +#define PGRANGE_ALLOC (1 << 1) +/* +* Do not take any locks. This means the caller has taken care of +* synchronisation. This is incompatible with PGRANGE_ALLOC and also with +* mm=&init_mm. +*/ +#define PGRANGE_NOLOCK (1 << 2) + + +static inline bool pgrange_create (unsigned int flags) +{ +return flags & (PGRANGE_CREATE |PGRANGE_ALLOC ); +} int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, unsigned long size, pte_fn_t fn,
mm/memory.c +34 -25
180 lines not on this page · raw message
Signature · 1 line
2.51.2
Re: [PATCH v2 05/22] mm: Add more flags for __apply_to_page_range()
Looking at the next AI review from [0]...
[0] https://sashiko.dev/#/patchset/20260320-page_alloc-unmapped-v2-0-28bf1bd54f41%40google.com
I'll keep it short this time: the patch is complete rubbish. In case anyone is reading this please skip it for now. The details of how this is implemented are orthogonal to the rest of the patchset anyway.
PATCH v2 6/22 x86/mm: introduce the mermap
In reply to Brendan JackmanThe mermap provides a fast way to create ephemeral mm-local mappings of physical pages. The purpose of this is to access pages that have been removed from the direct map. Potential use cases are:
1. For zeroing __GFP_UNMAPPED pages (added in a later patch).
2. For populating guest_memfd pages that are protected by the GUEST_MEMFD_NO_DIRECT_MAP feature [0].
3. For efficient access of pages protected by Address Space Isolation [1].
[0] https://lore.kernel.org/all/[email protected]/
[1] https://linuxasi.dev
The details of this mechanism are described in the API comments. However the key idea is to use CPU-local virtual regions to avoid a need for synchronizing. On x86, this can also be used to prevent TLB shootdowns.
Because the virtual region is CPU-local, allocating from the mermap disables migration. The caller is forbidden to use the returned value from any other context, and migration is re-enabled when it's freed.
One might notice that mermap_get() bears a strong similarity to kmap_local_page(). The most important differences between mermap_get() and kmap_local_page() are:
1. mermap_get() allows mapping variable sizes while kmap_local_page() specifically maps a single order-0 page.
2. As a consequence of 1 (combined with the need for mermap_get() to be an extremely simple allocator), mermap_get() should be expected to fail, while kmap_local_page() is guaranteed to work up to a certain degree of nesting.
3. While the mappings provided by kmap_local_page() are _logically_ local to the calling context (it's a bug for software to access them from elsewhere), they are _physically_ installed into the shared kernel pagetables. This means their locality doesn't provide any protection from hardware attacks. In contrast, the mermap is physically local to the creating mm, taking advantage of the new mm-local kernel address region.
So that the mermap is available even in contexts where failure is not tolerable there is also a _reserved() variant, which is fixed at allocating a single base page. This is useful, for example, for zeroing __GFP_UNMAPPED pages, where handling failure would be extremely inconvenient. The _reserved() variant is simply implemented by leaving one base-page space unavailable for non-_reserved allocations, and requiring an atomic context.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/mermap.h | 23 +++ arch/x86/include/asm/pgtable_64_types.h | 8 +- include/linux/mermap.h | 63 ++++++ include/linux/mermap_types.h | 41 ++++ include/linux/mm_types.h | 4 + kernel/fork.c | 5 + mm/Kconfig | 10 + mm/Makefile | 1 + mm/mermap.c | 334 ++++++++++++++++++++++++++++++++ mm/pgalloc-track.h | 6 + 11 files changed, 495 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
@@ -37,6 +37,7 @@ config X86_64 select ZONE_DMA32 select EXECMEM if DYNAMIC_FTRACE select ACPI_MRRM if ACPI + select ARCH_SUPPORTS_MERMAP config FORCE_DYNAMIC_FTRACE def_bool y
arch/x86/include/asm/mermap.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_MERMAP_H +#define _ASM_X86_MERMAP_H + +#include <asm/tlbflush.h> + +static inline void arch_mermap_flush_tlb (void ) +{ +/* +* No shootdown allowed, IRQs may be off. Luckily other CPUs are not +* allowed to access our region so the stale mappings are harmless, as +* long as they still point to data belonging to this process. +*/ +__flush_tlb_all (); +} + +static inline bool arch_mermap_pgprot_allowed (pgprot_t prot) +{ +/* Mermap is mm-local so global mappings would be a bug. */ +return !(pgprot_val (prot) & _PAGE_GLOBAL); +} + +#endif /* _ASM_X86_MERMAP_H */
arch/x86/include/asm/pgtable_64_types.h +7 -1
@@ -105,11 +105,17 @@ extern unsigned int ptrs_per_p4d;#define MM_LOCAL_PGD_ENTRY -240UL#define MM_LOCAL_BASE_ADDR (MM_LOCAL_PGD_ENTRY << PGDIR_SHIFT) -#define MM_LOCAL_END_ADDR ((MM_LOCAL_PGD_ENTRY + 1) << PGDIR_SHIFT) +#define MM_LOCAL_START_ADDR ((MM_LOCAL_PGD_ENTRY) << PGDIR_SHIFT) +#define MM_LOCAL_END_ADDR (MM_LOCAL_START_ADDR + (1UL << PGDIR_SHIFT))#define LDT_BASE_ADDR MM_LOCAL_BASE_ADDR#define LDT_END_ADDR (LDT_BASE_ADDR + PMD_SIZE) +#define MERMAP_BASE_ADDR LDT_END_ADDR +#define MERMAP_CPU_REGION_SIZE PMD_SIZE +#define MERMAP_SIZE (MERMAP_CPU_REGION_SIZE * NR_CPUS) +#define MERMAP_END_ADDR (MERMAP_BASE_ADDR + (NR_CPUS * MERMAP_CPU_REGION_SIZE)) +#define __VMALLOC_BASE_L4 0xffffc90000000000UL#define __VMALLOC_BASE_L5 0xffa0000000000000UL
include/linux/mermap.h new file +63 -0
63 lines not on this page · raw message
include/linux/mermap_types.h new file +41 -0
41 lines not on this page · raw message
include/linux/mm_types.h +4 -0
@@ -7,6 +7,7 @@#include <linux/auxvec.h> #include <linux/kref.h> #include <linux/list.h> +#include <linux/mermap_types.h> #include <linux/spinlock.h> #include <linux/rbtree.h> #include <linux/maple_tree.h>
@@ -34,6 +35,7 @@struct address_space ;struct futex_private_hash ;struct mem_cgroup ; +struct mermap ; typedef struct { unsigned long f;
@@ -1172,6 +1174,8 @@ struct mm_struct {atomic_t membarrier_state;#endif +struct mermap mermap; +/** * @mm_users: The number of users including userspace. *
kernel/fork.c +5 -0
@@ -13,6 +13,7 @@*/ #include <linux/anon_inodes.h> +#include <linux/mermap.h> #include <linux/slab.h> #include <linux/sched/autogroup.h> #include <linux/sched/mm.h>
@@ -1144,6 +1145,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm->user_ns =get_user_ns (user_ns);lru_gen_init_mm (mm); + +mermap_mm_init (mm); +return mm; fail_pcpu:
@@ -1187,6 +1191,7 @@ static inline void __mmput(struct mm_struct *mm)ksm_exit (mm);khugepaged_exit (mm);/* must run before exit_mmap */ exit_mmap (mm); +mermap_mm_teardown (mm);mm_put_huge_zero_folio (mm);set_mm_exe_file (mm,NULL );if (!list_empty (&mm->mmlist)) {
mm/Kconfig +10 -0
@@ -1484,4 +1484,14 @@ config MM_LOCAL_REGION source "mm/damon/Kconfig" +config ARCH_SUPPORTS_MERMAP + bool + +config MERMAP + bool "Support for epheMERal mappings within the kernel" + depends on ARCH_SUPPORTS_MERMAP + depends on MM_LOCAL_REGION + help + Support for epheMERal mappings within the kernel. + endmenu
mm/Makefile +1 -0
@@ -150,3 +150,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o +obj-$(CONFIG_MERMAP) += mermap.o
mm/mermap.c new file +334 -0
334 lines not on this page · raw message
mm/pgalloc-track.h +6 -0
@@ -2,6 +2,12 @@#ifndef _LINUX_PGALLOC_TRACK_H#define _LINUX_PGALLOC_TRACK_H +#include <linux/mm.h> +#include <linux/pgalloc.h> +#include <linux/pgtable.h> + +#include "internal.h" +#if defined(CONFIG_MMU ) static inline p4d_t *p4d_alloc_track(struct mm_struct *mm, pgd_t *pgd, unsigned long address,
Signature · 1 line
2.51.2
PATCH v2 7/22 mm: KUnit tests for the mermap
In reply to Brendan JackmanSome simple smoke-tests for the mermap. Mainly aiming to test:
1. That there aren't any silly off-by-ones.
2. That the pagetables are not completely broken.
3. That the TLB appears to get flushed basically when expected.
This last point requires a bit of ifdeffery to detect when the flushing has been performed.
--- include/linux/mermap_types.h | 3 + mm/Kconfig | 11 ++ mm/Makefile | 1 + mm/tests/mermap_kunit.c | 250 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 265 insertions(+)
include/linux/mermap_types.h +3 -0
@@ -24,6 +24,9 @@ struct mermap_cpu {unsigned long next_addr;struct mermap_alloc normal_allocs[3 ];struct mermap_alloc reserve_alloc; +#if IS_ENABLED (CONFIG_MERMAP_KUNIT_TEST ) +u64 tlb_flushes; +#endif };struct mermap {
mm/Kconfig +11 -0
@@ -1494,4 +1494,15 @@ config MERMAP help Support for epheMERal mappings within the kernel. +config MERMAP_KUNIT_TEST + tristate "KUnit tests for the mermap" if !KUNIT_ALL_TESTS + depends on ARCH_SUPPORTS_MERMAP + depends on KUNIT + depends on MERMAP + default KUNIT_ALL_TESTS + help + KUnit test for the mermap. + + If unsure, say N. + endmenu
mm/Makefile +1 -0
@@ -151,3 +151,4 @@ obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o obj-$(CONFIG_MERMAP) += mermap.o +obj-$(CONFIG_MERMAP_KUNIT_TEST) += tests/mermap_kunit.o
mm/tests/mermap_kunit.c new file +250 -0
250 lines not on this page · raw message
Signature · 1 line
2.51.2
kernel test robot
build botRe: [PATCH v2 07/22] mm: KUnit tests for the mermap
kernel test robot noticed the following build warnings:
Hi Brendan,
kernel test robot noticed the following build warnings:
[auto build test WARNING on b5d083a3ed1e2798396d5e491432e887da8d4a06]
url: https://github.com/intel-lab-lkp/linux/commits/Brendan-Jackman/x86-mm-split-out-preallocate_sub_pgd/20260321-042521 base: b5d083a3ed1e2798396d5e491432e887da8d4a06 patch link: https://lore.kernel.org/r/20260320-page_alloc-unmapped-v2-7-28bf1bd54f41%40google.com patch subject: [PATCH v2 07/22] mm: KUnit tests for the mermap config: x86_64-randconfig-101-20260324 (https://download.01.org/0day-ci/archive/20260324/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
cocci warnings: (new ones prefixed by >>)
>> mm/tests/mermap_kunit.c:156:2-3: Unneeded semicolon
vim +156 mm/tests/mermap_kunit.c
131
132 static void test_multiple_allocs(struct kunit *test)
133 {
134 struct __mermap_put_args *argss[NR_NORMAL_ALLOCS] = { };
135 struct page *pages[NR_NORMAL_ALLOCS + 1];
136 struct mermap_alloc *reserved_alloc;
137 struct mm_struct *mm = get_mm(test);
138 int magic = 0xE4A4;
139
140 for (int i = 0; i < ARRAY_SIZE(pages); i++) {
141 pages[i] = alloc_page_wrapper(test, GFP_KERNEL);
142 WRITE_ONCE(*(int *)page_to_virt(pages[i]), magic + i);
143 }
144
145 for (int i = 0; i < ARRAY_SIZE(argss); i++) {
146 unsigned long base = mermap_cpu_base(raw_smp_processor_id());
147 unsigned long end = mermap_cpu_end(raw_smp_processor_id());
148 unsigned long addr;
149
150 argss[i] = __mermap_get_wrapper(test, mm, pages[i], PAGE_SIZE, PAGE_KERNEL);
151 KUNIT_ASSERT_NOT_NULL_MSG(test, argss[i], "alloc %d failed", i);
152
153 addr = (unsigned long) mermap_addr(argss[i]->alloc);
154 KUNIT_EXPECT_GE_MSG(test, addr, base, "alloc %d out of range", i);
155 KUNIT_EXPECT_LT_MSG(test, addr, end, "alloc %d out of range", i);
> 156 };
157
158 /*
159 * Read through the mappings to try and detect if they point to the
160 * pages we wrote earlier.
161 */
162 kthread_use_mm(mm);
163 for (int i = 0; i < ARRAY_SIZE(pages) - 1; i++) {
164 int *ptr = (int *)mermap_addr(argss[i]->alloc);
165
166 KUNIT_EXPECT_EQ(test, *ptr, magic + i);
167 }
168
169 /* Run out of alloc structures, only reserved allocs should succeed now. */
170 KUNIT_ASSERT_NULL(test, __mermap_get(mm, pages[NR_NORMAL_ALLOCS],
171 PAGE_SIZE, PAGE_KERNEL, false));
172 preempt_disable();
173 reserved_alloc = __mermap_get(mm, pages[NR_NORMAL_ALLOCS],
174 PAGE_SIZE, PAGE_KERNEL, true);
175 KUNIT_EXPECT_NOT_NULL(test, reserved_alloc);
176 /* Also check if this mapping seems correct*/
177 if (reserved_alloc) {
178 int *ptr = (int *)mermap_addr(reserved_alloc);
179
180 KUNIT_EXPECT_EQ(test, *ptr, magic + NR_NORMAL_ALLOCS);
181
182 mermap_put(reserved_alloc);
183 }
184 preempt_enable();
185
186 kthread_unuse_mm(mm);
187 }
188
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v2 8/22 mm: introduce for_each_free_list()
In reply to Brendan JackmanLater patches will rearrange the free areas, but there are a couple of places that iterate over them with the assumption that they have the current structure.
It seems ideally, code outside of mm should not be directly aware of struct free_area in the first place, but that awareness seems relatively harmless so just make the minimal change.
Now instead of letting users manually iterate over the free lists, just provide a macro to do that. Then adopt that macro in a couple of places.
--- include/linux/mmzone.h | 7 +++++-- kernel/power/snapshot.c | 8 ++++---- mm/mm_init.c | 11 +++++++---- 3 files changed, 16 insertions(+), 10 deletions(-)
include/linux/mmzone.h +5 -2
@@ -177,9 +177,12 @@ static inline bool migratetype_is_mergeable(int mt)return mt <MIGRATE_PCPTYPES ; } -#define for_each_migratetype_order (order, type) \ +#define for_each_free_list (list, zone, order) \ for (order = 0; order < NR_PAGE_ORDERS; order++) \ - for (type = 0; type < MIGRATE_TYPES; type++) + for (unsigned int type = 0; \ + list = &zone->free_area[order].free_list[type], \ + type < MIGRATE_TYPES; \ + type++) \extern int page_group_by_mobility_disabled;
kernel/power/snapshot.c +4 -4
@@ -1245,8 +1245,9 @@ unsigned int snapshot_additional_pages(struct zone *zone)static void mark_free_pages (struct zone *zone) {unsigned long pfn, max_zone_pfn, page_count =WD_PAGE_COUNT ; +struct list_head *free_list;unsigned long flags; -unsigned int order, t; +unsigned int order;struct page *page;if (zone_is_empty (zone))
@@ -1270,9 +1271,8 @@ static void mark_free_pages(struct zone *zone)swsusp_unset_page_free (page); } -for_each_migratetype_order (order, t) { -list_for_each_entry (page, - &zone->free_area[order].free_list[t], buddy_list) { +for_each_free_list (free_list, zone, order) { +list_for_each_entry (page, free_list, buddy_list) {unsigned long i; pfn =page_to_pfn (page);
mm/mm_init.c +7 -4
@@ -1445,11 +1445,14 @@ static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx,static void __meminit zone_init_free_lists (struct zone *zone) { -unsigned int order, t; -for_each_migratetype_order (order, t) { -INIT_LIST_HEAD (&zone->free_area[order].free_list[t]); +struct list_head *list; +unsigned int order; + +for_each_free_list (list, zone, order) +INIT_LIST_HEAD (list); + +for (order =0 ; order <NR_PAGE_ORDERS ; order++) zone->free_area[order].nr_free =0 ; - }#ifdef CONFIG_UNACCEPTED_MEMORY INIT_LIST_HEAD (&zone->unaccepted_pages);
Signature · 1 line
2.51.2
Re: [PATCH v2 08/22] mm: introduce for_each_free_list()
Quoted 7 lines · Brendan Jackman
Later patches will rearrange the free areas, but there are a couple of places that iterate over them with the assumption that they have the current structure.
It seems ideally, code outside of mm should not be directly aware of struct
free_areain the first place, but that awareness seems relatively harmless so just make the minimal change.
I think we should lift the code from kernel/power/snapshot.c to under mm/ eventually, ISTR discussing it somewhere recently. But doesn't have to be in this series.
Quoted 4 lines
Now instead of letting users manually iterate over the free lists, just provide a macro to do that. Then adopt that macro in a couple of places.
Signed-off-by Brendan Jackman <[email protected]>
Quoted 76 lines
--- include/linux/mmzone.h | 7 +++++-- kernel/power/snapshot.c | 8 ++++---- mm/mm_init.c | 11 +++++++---- 3 files changed, 16 insertions(+), 10 deletions(-)include/linux/mmzone.h +5
-2@@ -177,9 +177,12 @@ static inline bool migratetype_is_mergeable(int mt)return mt <MIGRATE_PCPTYPES ; } -#define for_each_migratetype_order (order, type) \ +#define for_each_free_list (list, zone, order) \ for (order = 0; order < NR_PAGE_ORDERS; order++) \ - for (type = 0; type < MIGRATE_TYPES; type++) + for (unsigned int type = 0; \ + list = &zone->free_area[order].free_list[type], \ + type < MIGRATE_TYPES; \ + type++) \extern int page_group_by_mobility_disabled;kernel/power/snapshot.c +4
-4@@ -1245,8 +1245,9 @@ unsigned int snapshot_additional_pages(struct zone *zone)static void mark_free_pages (struct zone *zone) {unsigned long pfn, max_zone_pfn, page_count =WD_PAGE_COUNT ; +struct list_head *free_list;unsigned long flags; -unsigned int order, t; +unsigned int order;struct page *page;if (zone_is_empty (zone))@@ -1270,9 +1271,8 @@ static void mark_free_pages(struct zone *zone)swsusp_unset_page_free (page); } -for_each_migratetype_order (order, t) { -list_for_each_entry (page, - &zone->free_area[order].free_list[t], buddy_list) { +for_each_free_list (free_list, zone, order) { +list_for_each_entry (page, free_list, buddy_list) {unsigned long i; pfn =page_to_pfn (page);mm/mm_init.c +7
-4@@ -1445,11 +1445,14 @@ static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx,static void __meminit zone_init_free_lists (struct zone *zone) { -unsigned int order, t; -for_each_migratetype_order (order, t) { -INIT_LIST_HEAD (&zone->free_area[order].free_list[t]); +struct list_head *list; +unsigned int order; + +for_each_free_list (list, zone, order) +INIT_LIST_HEAD (list); + +for (order =0 ; order <NR_PAGE_ORDERS ; order++) zone->free_area[order].nr_free =0 ; - }#ifdef CONFIG_UNACCEPTED_MEMORY INIT_LIST_HEAD (&zone->unaccepted_pages);
PATCH v2 9/22 mm/page_alloc: don't overload migratetype in find_suitable_fallback()
In reply to Brendan JackmanThis function currently returns a signed integer that encodes status in-band, as negative numbers, along with a migratetype.
This function is about to be updated to a mode where this in-band signaling no longer makes sense. Therefore, switch to a more explicit/verbose style that encodes the status and migratetype separately.
In the spirit of making things more explicit, also create an enum to avoid using magic integer literals with special meanings. This enables documenting the values at their definition instead of in one of the callers.
--- mm/compaction.c | 3 ++- mm/internal.h | 14 +++++++++++--- mm/page_alloc.c | 40 +++++++++++++++++++++++----------------- 3 files changed, 36 insertions(+), 21 deletions(-)
mm/compaction.c +2 -1
@@ -2358,7 +2358,8 @@ static enum compact_result __compact_finished(struct compact_control *cc)* Job done if allocation would steal freepages from * other migratetype buddy lists. */ -if (find_suitable_fallback (area, order, migratetype, true) >=0 ) +if (find_suitable_fallback (area, order, migratetype, true,NULL ) + ==FALLBACK_FOUND )/* * Movable pages are OK in any pageblock. If we are * stealing for a non-movable allocation, make sure
mm/internal.h +11 -3
@@ -1059,9 +1059,17 @@ static inline void init_cma_pageblock(struct page *page) }#endif - -int find_suitable_fallback (struct free_area *area,unsigned int order, -int migratetype,bool claimable); +enum fallback_result { +/* Found suitable migratetype, *mt_out is valid. */ +FALLBACK_FOUND , +/* No fallback found in requested order. */ +FALLBACK_EMPTY , +/* Passed @claimable, but claiming whole block is a bad idea. */ +FALLBACK_NOCLAIM , +}; +enum fallback_result +find_suitable_fallback (struct free_area *area,unsigned int order, +int migratetype,bool claimable,unsigned int *mt_out);static inline bool free_area_empty (struct free_area *area,int migratetype) {
mm/page_alloc.c +23 -17
@@ -2248,25 +2248,29 @@ static bool should_try_claim_block(unsigned int order, int start_mt)* we would do this whole-block claiming. This would help to reduce * fragmentation due to mixed migratetype pages in one pageblock. */ -int find_suitable_fallback (struct free_area *area,unsigned int order, -int migratetype,bool claimable) +enum fallback_result +find_suitable_fallback (struct free_area *area,unsigned int order, +int migratetype,bool claimable,unsigned int *mt_out) {int i;if (claimable && !should_try_claim_block (order, migratetype)) -return -2 ; +return FALLBACK_NOCLAIM ;if (area->nr_free ==0 ) -return -1 ; +return FALLBACK_EMPTY ;for (i =0 ; i <MIGRATE_PCPTYPES -1 ; i++) {int fallback_mt = fallbacks[migratetype][i]; -if (!free_area_empty (area, fallback_mt)) -return fallback_mt; +if (!free_area_empty (area, fallback_mt)) { +if (mt_out) + *mt_out = fallback_mt; +return FALLBACK_FOUND ; + } } -return -1 ; +return FALLBACK_EMPTY ; }/*
@@ -2376,16 +2380,16 @@ __rmqueue_claim(struct zone *zone, int order, int start_migratetype,*/ for (current_order =MAX_PAGE_ORDER ; current_order >= min_order; --current_order) { - area = &(zone->free_area[current_order]); - fallback_mt =find_suitable_fallback (area, current_order, - start_migratetype, true); +enum fallback_result result; -/* No block in that order */ -if (fallback_mt ==-1 ) + area = &(zone->free_area[current_order]); + result =find_suitable_fallback (area, current_order, + start_migratetype, true, &fallback_mt); + +if (result ==FALLBACK_EMPTY )continue ; -/* Advanced into orders too low to claim, abort */ -if (fallback_mt ==-2 ) +if (result ==FALLBACK_NOCLAIM )break ; page =get_page_from_free_area (area, fallback_mt);
@@ -2415,10 +2419,12 @@ __rmqueue_steal(struct zone *zone, int order, int start_migratetype)int fallback_mt;for (current_order = order; current_order <NR_PAGE_ORDERS ; current_order++) { +enum fallback_result result; + area = &(zone->free_area[current_order]); - fallback_mt =find_suitable_fallback (area, current_order, - start_migratetype, false); -if (fallback_mt ==-1 ) + result =find_suitable_fallback (area, current_order, start_migratetype, + false, &fallback_mt); +if (result ==FALLBACK_EMPTY )continue ; page =get_page_from_free_area (area, fallback_mt);
Signature · 1 line
2.51.2
Re: [PATCH v2 09/22] mm/page_alloc: don't overload migratetype in find_suitable_fallback()
Quoted 8 lines · Brendan Jackman
This function currently returns a signed integer that encodes status in-band, as negative numbers, along with a migratetype.
This function is about to be updated to a mode where this in-band signaling no longer makes sense. Therefore, switch to a more explicit/verbose style that encodes the status and migratetype separately.
Quoted 6 lines
In the spirit of making things more explicit, also create an enum to avoid using magic integer literals with special meanings. This enables documenting the values at their definition instead of in one of the callers.
Signed-off-by Brendan Jackman <[email protected]>
Re: [PATCH v2 09/22] mm/page_alloc: don't overload migratetype in find_suitable_fallback()
Quoted 9 lines · Vlastimil Babka (SUSE)
Quoted 8 lines · Brendan Jackman
This function currently returns a signed integer that encodes status in-band, as negative numbers, along with a migratetype.
This function is about to be updated to a mode where this in-band signaling no longer makes sense. Therefore, switch to a more explicit/verbose style that encodes the status and migratetype separately.
Quoted 8 lines
Quoted 6 lines
In the spirit of making things more explicit, also create an enum to avoid using magic integer literals with special meanings. This enables documenting the values at their definition instead of in one of the callers.
Signed-off-by Brendan Jackman <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>
Thanks,
This and the prior patch could arguably just be considered independent cleanups, shall I send them on their own?
Equally if they feel like "churn" I'm happy to keep them in this patchset.
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 09/22] mm/page_alloc: don't overload migratetype in find_suitable_fallback()
Quoted 15 lines · Brendan Jackman
Quoted 14 lines · Vlastimil Babka (SUSE)
Quoted 13 lines · Brendan Jackman
This function currently returns a signed integer that encodes status in-band, as negative numbers, along with a migratetype.
This function is about to be updated to a mode where this in-band signaling no longer makes sense. Therefore, switch to a more explicit/verbose style that encodes the status and migratetype separately.
In the spirit of making things more explicit, also create an enum to avoid using magic integer literals with special meanings. This enables documenting the values at their definition instead of in one of the callers.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Brendan Jackman <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Thanks,
This and the prior patch could arguably just be considered independent cleanups, shall I send them on their own?
I guess why not, fewer patches in the patchset :) You're probably going to need to rebase this on current mm tree anyway for the next posting?
Quoted 2 lines
Equally if they feel like "churn" I'm happy to keep them in this patchset.
Re: [PATCH v2 09/22] mm/page_alloc: don't overload migratetype in find_suitable_fallback()
In reply to Brendan JackmanQuoted 9 lines
This function currently returns a signed integer that encodes status in-band, as negative numbers, along with a migratetype.
This function is about to be updated to a mode where this in-band signaling no longer makes sense. Therefore, switch to a more explicit/verbose style that encodes the status and migratetype separately.
Quoted 6 lines
In the spirit of making things more explicit, also create an enum to avoid using magic integer literals with special meanings. This enables documenting the values at their definition instead of in one of the callers.
Signed-off-by Brendan Jackman <[email protected]>
Seems independent patch with no functional change.
Quoted 127 lines
--- mm/compaction.c | 3 ++- mm/internal.h | 14 +++++++++++--- mm/page_alloc.c | 40 +++++++++++++++++++++++----------------- 3 files changed, 36 insertions(+), 21 deletions(-)mm/compaction.c +2
-1@@ -2358,7 +2358,8 @@ static enum compact_result __compact_finished(struct compact_control *cc)* Job done if allocation would steal freepages from * other migratetype buddy lists. */ -if (find_suitable_fallback (area, order, migratetype, true) >=0 ) +if (find_suitable_fallback (area, order, migratetype, true,NULL ) + ==FALLBACK_FOUND )/* * Movable pages are OK in any pageblock. If we are * stealing for a non-movable allocation, make sure mm/internal.h +11
-3@@ -1059,9 +1059,17 @@ static inline void init_cma_pageblock(struct page *page) }#endif - -int find_suitable_fallback (struct free_area *area,unsigned int order, -int migratetype,bool claimable); +enum fallback_result { +/* Found suitable migratetype, *mt_out is valid. */ +FALLBACK_FOUND , +/* No fallback found in requested order. */ +FALLBACK_EMPTY , +/* Passed @claimable, but claiming whole block is a bad idea. */ +FALLBACK_NOCLAIM , +}; +enum fallback_result +find_suitable_fallback (struct free_area *area,unsigned int order, +int migratetype,bool claimable,unsigned int *mt_out);static inline bool free_area_empty (struct free_area *area,int migratetype) {mm/page_alloc.c +23
-17@@ -2248,25 +2248,29 @@ static bool should_try_claim_block(unsigned int order, int start_mt)* we would do this whole-block claiming. This would help to reduce * fragmentation due to mixed migratetype pages in one pageblock. */ -int find_suitable_fallback (struct free_area *area,unsigned int order, -int migratetype,bool claimable) +enum fallback_result +find_suitable_fallback (struct free_area *area,unsigned int order, +int migratetype,bool claimable,unsigned int *mt_out) {int i;if (claimable && !should_try_claim_block (order, migratetype)) -return -2 ; +return FALLBACK_NOCLAIM ;if (area->nr_free ==0 ) -return -1 ; +return FALLBACK_EMPTY ;for (i =0 ; i <MIGRATE_PCPTYPES -1 ; i++) {int fallback_mt = fallbacks[migratetype][i]; -if (!free_area_empty (area, fallback_mt)) -return fallback_mt; +if (!free_area_empty (area, fallback_mt)) { +if (mt_out) + *mt_out = fallback_mt; +return FALLBACK_FOUND ; + } } -return -1 ; +return FALLBACK_EMPTY ; }/* @@ -2376,16 +2380,16 @@ __rmqueue_claim(struct zone *zone, int order, int start_migratetype,*/ for (current_order =MAX_PAGE_ORDER ; current_order >= min_order; --current_order) { - area = &(zone->free_area[current_order]); - fallback_mt =find_suitable_fallback (area, current_order, - start_migratetype, true); +enum fallback_result result; -/* No block in that order */ -if (fallback_mt ==-1 ) + area = &(zone->free_area[current_order]); + result =find_suitable_fallback (area, current_order, + start_migratetype, true, &fallback_mt); + +if (result ==FALLBACK_EMPTY )continue ; -/* Advanced into orders too low to claim, abort */ -if (fallback_mt ==-2 ) +if (result ==FALLBACK_NOCLAIM )break ; page =get_page_from_free_area (area, fallback_mt);@@ -2415,10 +2419,12 @@ __rmqueue_steal(struct zone *zone, int order, int start_migratetype)int fallback_mt;for (current_order = order; current_order <NR_PAGE_ORDERS ; current_order++) { +enum fallback_result result; + area = &(zone->free_area[current_order]); - fallback_mt =find_suitable_fallback (area, current_order, - start_migratetype, false); -if (fallback_mt ==-1 ) + result =find_suitable_fallback (area, current_order, start_migratetype, + false, &fallback_mt); +if (result ==FALLBACK_EMPTY )continue ; page =get_page_from_free_area (area, fallback_mt);
Brendan Jackman
reviewerRe: [PATCH v2 09/22] mm/page_alloc: don't overload migratetype in find_suitable_fallback()
Quoted 14 lines · Pankaj Gupta
Quoted 14 lines
This function currently returns a signed integer that encodes status in-band, as negative numbers, along with a migratetype.
This function is about to be updated to a mode where this in-band signaling no longer makes sense. Therefore, switch to a more explicit/verbose style that encodes the status and migratetype separately.
In the spirit of making things more explicit, also create an enum to avoid using magic integer literals with special meanings. This enables documenting the values at their definition instead of in one of the callers.
Quoted 5 lines
Quoted 1 line
Signed-off-by Brendan Jackman <[email protected]>Seems independent patch with no functional change.
Reviewed-by Pankaj Gupta <[email protected]>
Thanks Pankaj - also BTW this and some other patches were already split out into a separate series and they are now in mm-unstable:
Re: [PATCH v2 09/22] mm/page_alloc: don't overload migratetype in find_suitable_fallback()
Quoted 4 lines
Quoted 4 lines
Thanks Pankaj - also BTW this and some other patches were already split out into a separate series and they are now in mm-unstable:
Thanks for sharing this. I’m reviewing the main GFP_UNMAPPED series and will also go through the separate series you posted.
I am also interested in reviewing and testing the current state of guest_memfd support on top of GFP_UNMAPPED.
Thanks,
Pankaj
PATCH v2 10/22 mm: introduce freetype_t
In reply to Brendan JackmanThis is preparation for teaching the page allocator to break up free pages according to properties that have nothing to do with mobility. For example it can be used to allocate pages that are non-present in the physmap, or pages that are sensitive in ASI.
For these usecases, certain allocator behaviours are desirable:
- A "pool" of pages with the given property is usually available, so that pages can be provided with the correct sensitivity without zeroing/TLB flushing.
- Pages are physically grouped by the property, so that large allocations rarely have to alter the pagetables due to ASI.
- The properties can be forced to vary only at a certain fixed address granularity, so that the pagetables can all be pre-allocated. This is desirable because the page allocator will be changing mappings:
pre-allocation is a straightforward way to avoid recursive allocations (of pagetables).
It seems that the existing infrastructure for grouping pages by mobility, i.e. pageblocks and migratetypes, serves this purpose pretty nicely. However, overloading migratetype itself for this purpose looks like a road to maintenance hell. In particular, as soon as such properties become orthogonal to migratetypes, it would start to require "doubling" the migratetypes.
Therefore, introduce a new higher-level concept, called "freetype" (because it is used to index "free"lists) that can encode extra properties, orthogonally to mobility, via flags.
Since freetypes and migratetypes would be very easy to mix up, freetypes are (at least for now) stored in a struct typedef similar to atomic_t. This provides type-safety, but comes at the expense of being pretty annoying to code with. For instance, freetype_t cannot be compared with the == operator. Once this code matures, if the freetype/migratetype distinction gets less confusing, it might be wise to drop this struct and just use ints.
Because this will eventually be needed from pageblock-flags.h, put this in its own header instead of directly in mmzone.h.
To try and reduce review pain for such a churny patch, first introduce freetypes as nothing but an indirection over migratetypes. The helpers concerned with the flags are defined, but only as stubs. Convert everything over to using freetypes wherever they are needed to index freelists, but maintain references to migratetypes in code that really only cares specifically about mobility.
--- include/linux/freetype.h | 38 +++++ include/linux/gfp.h | 16 +- include/linux/mmzone.h | 49 +++++- mm/compaction.c | 35 +++-- mm/internal.h | 17 ++- mm/page_alloc.c | 390 +++++++++++++++++++++++++++++------------------ mm/page_isolation.c | 2 +- mm/page_owner.c | 7 +- mm/page_reporting.c | 4 +- mm/show_mem.c | 4 +- 10 files changed, 371 insertions(+), 191 deletions(-)
include/linux/freetype.h new file +38 -0
@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_FREETYPE_H +#define _LINUX_FREETYPE_H + +#include <linux/types.h> + +/* +* A freetype is the index used to identify free lists. This consists of a +* migratetype, and other bits which encode orthogonal properties of memory. +*/ +typedef struct { +int migratetype; +}freetype_t ; + +/* +* Return a dense linear index for freetypes that have lists in the free area. +* Return -1 for other freetypes. +*/ +static inline int freetype_idx (freetype_t freetype) +{ +return freetype.migratetype; +} + +/* No freetype flags actually exist yet. */ +#define NR_FREETYPE_IDXS MIGRATE_TYPES + +static inline unsigned int freetype_flags (freetype_t freetype) +{ +/* No flags supported yet. */ +return 0 ; +} + +static inline bool freetypes_equal (freetype_t a,freetype_t b) +{ +return a.migratetype == b.migratetype; +} + +#endif /* _LINUX_FREETYPE_H */
include/linux/gfp.h +11 -5
@@ -21,8 +21,10 @@ struct mempolicy;#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)#define GFP_MOVABLE_SHIFT 3 -static inline int gfp_migratetype (const gfp_t gfp_flags) +static inline freetype_t gfp_freetype (const gfp_t gfp_flags) { +int migratetype; +VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK );BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE);BUILD_BUG_ON ((___GFP_MOVABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_MOVABLE );
@@ -30,11 +32,15 @@ static inline int gfp_migratetype(const gfp_t gfp_flags)BUILD_BUG_ON (((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >>GFP_MOVABLE_SHIFT ) !=MIGRATE_HIGHATOMIC ); -if (unlikely (page_group_by_mobility_disabled)) -return MIGRATE_UNMOVABLE ; +if (unlikely (page_group_by_mobility_disabled)) { + migratetype =MIGRATE_UNMOVABLE ; + }else { +/* Group based on mobility */ + migratetype = (__force unsigned long )(gfp_flags &GFP_MOVABLE_MASK ) + >>GFP_MOVABLE_SHIFT ; + } -/* Group based on mobility */ -return (__force unsigned long )(gfp_flags &GFP_MOVABLE_MASK ) >>GFP_MOVABLE_SHIFT ; +return migrate_to_freetype (migratetype,0 ); }#undef GFP_MOVABLE_MASK#undef GFP_MOVABLE_SHIFT
include/linux/mmzone.h +44 -5
74 lines not on this page · raw message
mm/compaction.c +19 -16
@@ -1394,7 +1394,7 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn,static bool suitable_migration_source (struct compact_control *cc,struct page *page) { -int block_mt; +freetype_t block_ft;if (pageblock_skip_persistent (page))return false;
@@ -1402,12 +1402,12 @@ static bool suitable_migration_source(struct compact_control *cc,if ((cc->mode !=MIGRATE_ASYNC ) || !cc->direct_compaction)return true; - block_mt =get_pageblock_migratetype (page); + block_ft =get_pageblock_freetype (page); -if (cc->migratetype ==MIGRATE_MOVABLE ) -return is_migrate_movable (block_mt); +if (free_to_migratetype (cc->freetype) ==MIGRATE_MOVABLE ) +return is_migrate_movable (free_to_migratetype (block_ft));else -return block_mt == cc->migratetype; +return freetypes_equal (block_ft, cc->freetype); }/* Returns true if the page is within a block suitable for migration to */
@@ -1998,7 +1998,8 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc)* reduces the risk that a large movable pageblock is freed for * an unmovable/reclaimable small allocation. */ -if (cc->direct_compaction && cc->migratetype !=MIGRATE_MOVABLE ) +if (cc->direct_compaction && +free_to_migratetype (cc->freetype) !=MIGRATE_MOVABLE )return pfn;/*
@@ -2269,7 +2270,7 @@ static bool should_proactive_compact_node(pg_data_t *pgdat)static enum compact_result __compact_finished (struct compact_control *cc) {unsigned int order; -const int migratetype = cc->migratetype; +const freetype_t freetype = cc->freetype;int ret;/* Compaction run completes if the migrate and free scanner meet */
@@ -2344,25 +2345,27 @@ static enum compact_result __compact_finished(struct compact_control *cc)for (order = cc->order; order <NR_PAGE_ORDERS ; order++) {struct free_area *area = &cc->zone->free_area[order]; -/* Job done if page is free of the right migratetype */ -if (!free_area_empty (area, migratetype)) +/* Job done if page is free of the right freetype */ +if (!free_area_empty (area, freetype))return COMPACT_SUCCESS ;#ifdef CONFIG_CMA /* MIGRATE_MOVABLE can fallback on MIGRATE_CMA */ -if (migratetype ==MIGRATE_MOVABLE && - !free_area_empty (area,MIGRATE_CMA )) +if (free_to_migratetype (freetype) ==MIGRATE_MOVABLE && + !free_area_empty (area,freetype_with_migrate (cc->freetype, +MIGRATE_CMA )))return COMPACT_SUCCESS ;#endif /* * Job done if allocation would steal freepages from -* other migratetype buddy lists. +* other freetype buddy lists. */ -if (find_suitable_fallback (area, order, migratetype, true,NULL ) +if (find_suitable_fallback (area, order, freetype, true,NULL ) ==FALLBACK_FOUND )/* -* Movable pages are OK in any pageblock. If we are -* stealing for a non-movable allocation, make sure +* Movable pages are OK in any pageblock of the right +* sensitivity. If we are * stealing for a +* non-movable allocation, make sure * we finish compacting the current pageblock first * (which is assured by the above migrate_pfn align * check) so it is as free as possible and we won't
@@ -2567,7 +2570,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)INIT_LIST_HEAD (&cc->freepages[order]);INIT_LIST_HEAD (&cc->migratepages); - cc->migratetype =gfp_migratetype (cc->gfp_mask); + cc->freetype =gfp_freetype (cc->gfp_mask);if (!is_via_compact_memory (cc->order)) { ret = compaction_suit_allocation_order(cc->zone, cc->order,
mm/internal.h +9 -8
@@ -10,6 +10,7 @@#include <linux/fs.h> #include <linux/khugepaged.h> #include <linux/mm.h> +#include <linux/mmzone.h> #include <linux/mm_inline.h> #include <linux/mmu_notifier.h> #include <linux/pagemap.h>
@@ -681,7 +682,7 @@ struct alloc_context {struct zonelist *zonelist;nodemask_t *nodemask;struct zoneref *preferred_zoneref; -int migratetype; +freetype_t freetype;/* * highest_zoneidx represents highest usable zone index of
@@ -832,8 +833,8 @@ static inline void clear_zone_contiguous(struct zone *zone) }extern int __isolate_free_page (struct page *page,unsigned int order); -extern void __putback_isolated_page (struct page *page,unsigned int order, -int mt); +void __putback_isolated_page (struct page *page,unsigned int order, +freetype_t freetype);extern void memblock_free_pages (unsigned long pfn,unsigned int order);extern void __free_pages_core (struct page *page,unsigned int order,enum meminit_context context);
@@ -999,7 +1000,7 @@ struct compact_control {short search_order;/* order to start a fast search at */ const gfp_t gfp_mask;/* gfp mask of a direct compactor */ int order;/* order a direct compactor needs */ -int migratetype;/* migratetype of direct compactor */ +freetype_t freetype;/* freetype of direct compactor */ const unsigned int alloc_flags;/* alloc flags of a direct compactor */ const int highest_zoneidx;/* zone index of a direct compactor */ enum migrate_mode mode;/* Async or sync migration mode */
@@ -1060,7 +1061,7 @@ static inline void init_cma_pageblock(struct page *page)#endif enum fallback_result { -/* Found suitable migratetype, *mt_out is valid. */ +/* Found suitable fallback, *ft_out is valid. */ FALLBACK_FOUND ,/* No fallback found in requested order. */ FALLBACK_EMPTY ,
@@ -1069,11 +1070,11 @@ enum fallback_result { };enum fallback_result find_suitable_fallback (struct free_area *area,unsigned int order, -int migratetype,bool claimable,unsigned int *mt_out); +freetype_t freetype,bool claimable,freetype_t *ft_out); -static inline bool free_area_empty (struct free_area *area,int migratetype) +static inline bool free_area_empty (struct free_area *area,freetype_t freetype) { -return list_empty (&area->free_list[migratetype]); +return list_empty (free_area_list (area, freetype)); }/* mm/util.c */
mm/page_alloc.c +240 -150
1,036 lines not on this page · raw message
mm/page_isolation.c +1 -1
@@ -277,7 +277,7 @@ static void unset_migratetype_isolate(struct page *page)WARN_ON_ONCE (!pageblock_unisolate_and_move_free_pages (zone, page)); }else {clear_pageblock_isolate (page); -__putback_isolated_page (page, order,get_pageblock_migratetype (page)); +__putback_isolated_page (page, order,get_pageblock_freetype (page)); } zone->nr_isolate_pageblock--; out:
mm/page_owner.c +4 -3
@@ -481,7 +481,8 @@ void pagetypeinfo_showmixedcount_print(struct seq_file *m, goto ext_put_continue; page_owner =get_page_owner (page_ext); - page_mt =gfp_migratetype (page_owner->gfp_mask); + page_mt =free_to_migratetype ( +gfp_freetype (page_owner->gfp_mask));if (pageblock_mt != page_mt) {if (is_migrate_cma (pageblock_mt)) count[MIGRATE_MOVABLE ]++;
@@ -566,7 +567,7 @@ print_page_owner(char __user *buf, size_t count, unsigned long pfn,/* Print information relevant to grouping pages by mobility */ pageblock_mt =get_pageblock_migratetype (page); - page_mt =gfp_migratetype (page_owner->gfp_mask); + page_mt =free_to_migratetype (gfp_freetype (page_owner->gfp_mask)); ret += scnprintf(kbuf + ret, count - ret, "PFN 0x%lx type %s Block %lu type %s Flags %pGp\n", pfn,
@@ -617,7 +618,7 @@ void __dump_page_owner(const struct page *page) page_owner =get_page_owner (page_ext); gfp_mask = page_owner->gfp_mask; - mt =gfp_migratetype (gfp_mask); + mt =free_to_migratetype (gfp_freetype (gfp_mask));if (!test_bit (PAGE_EXT_OWNER , &page_ext->flags)) {pr_alert ("page_owner info is not present (never set?)\n" );
mm/page_reporting.c +2 -2
@@ -114,10 +114,10 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,*/ do { struct page *page = sg_page(sg); - int mt = get_pageblock_migratetype(page); + freetype_t ft = get_pageblock_freetype(page); unsigned int order = get_order(sg->length); - __putback_isolated_page(page, order, mt); + __putback_isolated_page(page, order, ft); /* If the pages were not reported due to error skip flagging */ if (!reported)
mm/show_mem.c +3 -1
@@ -374,7 +374,9 @@ static void show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_z types[order] =0 ;for (type =0 ; type <MIGRATE_TYPES ; type++) { -if (!free_area_empty (area, type)) +freetype_t ft =migrate_to_freetype (type,0 ); + +if (!free_area_empty (area, ft)) types[order] |=1 << type; } }
Signature · 1 line
2.51.2
Re: [PATCH v2 10/22] mm: introduce freetype_t
Quoted 42 lines · Brendan Jackman
This is preparation for teaching the page allocator to break up free pages according to properties that have nothing to do with mobility. For example it can be used to allocate pages that are non-present in the physmap, or pages that are sensitive in ASI.
For these usecases, certain allocator behaviours are desirable:
- A "pool" of pages with the given property is usually available, so that pages can be provided with the correct sensitivity without zeroing/TLB flushing.
- Pages are physically grouped by the property, so that large allocations rarely have to alter the pagetables due to ASI.
- The properties can be forced to vary only at a certain fixed address granularity, so that the pagetables can all be pre-allocated. This is desirable because the page allocator will be changing mappings:
pre-allocation is a straightforward way to avoid recursive allocations (of pagetables).It seems that the existing infrastructure for grouping pages by mobility, i.e. pageblocks and migratetypes, serves this purpose pretty nicely. However, overloading migratetype itself for this purpose looks like a road to maintenance hell. In particular, as soon as such properties become orthogonal to migratetypes, it would start to require "doubling" the migratetypes.
Therefore, introduce a new higher-level concept, called "freetype" (because it is used to index "free"lists) that can encode extra properties, orthogonally to mobility, via flags.
Since freetypes and migratetypes would be very easy to mix up, freetypes are (at least for now) stored in a struct typedef similar to
atomic_t. This provides type-safety, but comes at the expense of being pretty annoying to code with. For instance,freetype_tcannot be compared with the == operator. Once this code matures, if the freetype/migratetype distinction gets less confusing, it might be wise to drop this struct and just use ints.Because this will eventually be needed from
pageblock-flags.h, put this in its own header instead of directly inmmzone.h.
Quoted 8 lines
To try and reduce review pain for such a churny patch, first introduce freetypes as nothing but an indirection over migratetypes. The helpers concerned with the flags are defined, but only as stubs. Convert everything over to using freetypes wherever they are needed to index freelists, but maintain references to migratetypes in code that really only cares specifically about mobility.
Signed-off-by Brendan Jackman <[email protected]>
Seems mechanistic enough.
Some nits:
Quoted 7 lines
mm/page_alloc.c +0
-0@@ -422,6 +422,37 @@ bool get_pfnblock_bit(const struct page *page, unsigned long pfn,return test_bit (bitidx + pb_bit, bitmap_word); }
mm/page_alloc.c bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · line 433
Quoted 10 lines
mm/page_alloc.c +9
-0+/** +* __get_pfnblock_freetype - Return the freetype of a pageblock, optionally +* ignoring the fact that it's currently isolated. +* @page: The page within the block of interest +* @pfn: The target page frame number +* @ignore_iso: If isolated, return the migratetype that the block had before +* isolation. +*/ +__always_inline freetype_t
'static' too?
Quoted 6 lines
mm/page_alloc.c +6
-0+__get_pfnblock_freetype(const struct page *page, unsigned long pfn, + bool ignore_iso) +{ + int mt = get_pfnblock_migratetype(page, pfn); + +return migrate_to_freetype (mt,0 );
mm/page_alloc.c bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · line 449
Quoted 10 lines
mm/page_alloc.c +10
-0+} + +/** +* get_pfnblock_migratetype - Return the freetype of a pageblock +* @page: The page within the block of interest +* @pfn: The target page frame number +* +* Return: The freetype of the pageblock +*/ +__always_inline freetype_t
And this is declared in a header so the __always_inline is not really applicable?
(seems we should fix up get_pfnblock_migratetype too)
Quoted 15 lines
mm/page_alloc.c +7
-0+get_pfnblock_freetype(const struct page *page, unsigned long pfn) +{ +return __get_pfnblock_freetype (page, pfn,0 ); +} + +/** * get_pfnblock_migratetype - Return the migratetype of a pageblock * @page: The page within the block of interest @@ -2262,10 +2323,18 @@ find_suitable_fallback(struct free_area *area, unsigned int order,for (i =0 ; i <MIGRATE_PCPTYPES -1 ; i++) {int fallback_mt = fallbacks[migratetype][i]; +/*
mm/page_alloc.c find_suitable_fallback(struct free_area *area, unsigned int order, · line 2333
Quoted 10 lines
mm/page_alloc.c +6
-3+* Fallback to different migratetypes, but currently always with +* the same freetype flags. +*/ +freetype_t fallback_ft =freetype_with_migrate (freetype, fallback_mt); -if (!free_area_empty (area, fallback_mt)) { -if (mt_out) - *mt_out = fallback_mt; +if (freetype_idx (fallback_ft) <0 ) +continue ;
How can this happen? Is it preparatory?
Quoted 7 lines
mm/page_alloc.c +4
-0+ +if (!free_area_empty (area, fallback_ft)) { +if (ft_out) + *ft_out = fallback_ft;return FALLBACK_FOUND ; } }
Re: [PATCH v2 10/22] mm: introduce freetype_t
Quoted 11 lines · Vlastimil Babka (SUSE)
mm/page_alloc.c bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · line 433
Quoted 9 lines
mm/page_alloc.c +9
-0+/** +* __get_pfnblock_freetype - Return the freetype of a pageblock, optionally +* ignoring the fact that it's currently isolated. +* @page: The page within the block of interest +* @pfn: The target page frame number +* @ignore_iso: If isolated, return the migratetype that the block had before +* isolation. +*/ +__always_inline freetype_t'static' too?
Yup thanks
Quoted 21 lines
Quoted 7 lines
mm/page_alloc.c +6
-0+__get_pfnblock_freetype(const struct page *page, unsigned long pfn, + bool ignore_iso) +{ + int mt = get_pfnblock_migratetype(page, pfn); + +return migrate_to_freetype (mt,0 );mm/page_alloc.c bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · line 449
Quoted 10 lines
mm/page_alloc.c +10
-0+} + +/** +* get_pfnblock_migratetype - Return the freetype of a pageblock +* @page: The page within the block of interest +* @pfn: The target page frame number +* +* Return: The freetype of the pageblock +*/ +__always_inline freetype_tAnd this is declared in a header so the
__always_inlineis not really applicable?
Quoted 1 line
(seems we should fix up
get_pfnblock_migratetypetoo)
Um, I think it probably still forces inlining in calls within the same translation unit?
Anyway I am pretty meh about this, I suspect humans and compilers are equally bad at making this decision, I was just trying to be consistent with the code it's replacing.
Quoted 6 lines
Quoted 6 lines
mm/page_alloc.c +5
-0+/* +* Fallback to different migratetypes, but currently always with +* the same freetype flags. +*/ +freetype_t fallback_ft =freetype_with_migrate (freetype, fallback_mt);
Quoted 7 lines
mm/page_alloc.c find_suitable_fallback(struct free_area *area, unsigned int order, · line 2333
Quoted 5 lines
mm/page_alloc.c +2
-3-if (!free_area_empty (area, fallback_mt)) { -if (mt_out) - *mt_out = fallback_mt; +if (freetype_idx (fallback_ft) <0 ) +continue ;How can this happen? Is it preparatory?
Oops, yeah looks like I need to clean up how this happens in the history and clarify the commit messages.
In a later patch I add an optimisation where we avoid having freelists for freetypes that never arise in practice. And in those cases freetype_idx() returns -1.
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 10/22] mm: introduce freetype_t
Quoted 36 lines · Brendan Jackman
Quoted 11 lines · Vlastimil Babka (SUSE)
mm/page_alloc.c bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · line 433
Quoted 9 lines
mm/page_alloc.c +9
-0+/** +* __get_pfnblock_freetype - Return the freetype of a pageblock, optionally +* ignoring the fact that it's currently isolated. +* @page: The page within the block of interest +* @pfn: The target page frame number +* @ignore_iso: If isolated, return the migratetype that the block had before +* isolation. +*/ +__always_inline freetype_t'static' too?
Yup thanks
Quoted 20 lines
Quoted 7 lines
mm/page_alloc.c +6
-0+__get_pfnblock_freetype(const struct page *page, unsigned long pfn, + bool ignore_iso) +{ + int mt = get_pfnblock_migratetype(page, pfn); + +return migrate_to_freetype (mt,0 );mm/page_alloc.c bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · line 449
Quoted 10 lines
mm/page_alloc.c +10
-0+} + +/** +* get_pfnblock_migratetype - Return the freetype of a pageblock +* @page: The page within the block of interest +* @pfn: The target page frame number +* +* Return: The freetype of the pageblock +*/ +__always_inline freetype_tAnd this is declared in a header so the
__always_inlineis not really applicable?
Quoted 4 lines
Quoted 1 line
(seems we should fix up
get_pfnblock_migratetypetoo)Um, I think it probably still forces inlining in calls within the same translation unit?
True but I don't think we try to do that consciously, seems like it was just an oversight for get_pfnblock_migratetype, maybe Zi Yan remembers?
Quoted 3 lines
Anyway I am pretty meh about this, I suspect humans and compilers are equally bad at making this decision, I was just trying to be consistent with the code it's replacing.
True.
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 10/22] mm: introduce freetype_t
In reply to Brendan JackmanQuoted 2 lines · Brendan Jackman
+/**
+ *get_pfnblock_migratetype- Return the freetype of a pageblock
Also just noticed the wrong name here
Quoted 10 lines
+ * @page: The page within the block of interest
+ * @pfn: The target page frame number
+ *
+ * Return: The freetype of the pageblock
+ */mm/page_alloc.c +5
-0+__always_inline freetype_t +get_pfnblock_freetype (const struct page *page,unsigned long pfn) +{ +return __get_pfnblock_freetype (page, pfn,0 ); +}
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 10/22] mm: introduce freetype_t
In reply to Brendan JackmanQuoted 7 lines · Brendan Jackman
mm/page_alloc.c +1
-0@@ -422,6 +422,37 @@ bool get_pfnblock_bit(const struct page *page, unsigned long pfn,return test_bit (bitidx + pb_bit, bitmap_word); } +/**
mm/page_alloc.c bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · line 435
Quoted 10 lines
mm/page_alloc.c +10
-0+* __get_pfnblock_freetype - Return the freetype of a pageblock, optionally +* ignoring the fact that it's currently isolated. +* @page: The page within the block of interest +* @pfn: The target page frame number +* @ignore_iso: If isolated, return the migratetype that the block had before +* isolation. +*/ +__always_inline freetype_t +__get_pfnblock_freetype (const struct page *page,unsigned long pfn, +bool ignore_iso)
Hm I also noticed ignore_iso is ... ignored until later patch.
Quoted 7 lines
mm/page_alloc.c +7
-0+{ +int mt =get_pfnblock_migratetype (page, pfn); + +return migrate_to_freetype (mt,0 ); +} + +/**
mm/page_alloc.c bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · line 452
Quoted 10 lines
mm/page_alloc.c +10
-0+* get_pfnblock_migratetype - Return the freetype of a pageblock +* @page: The page within the block of interest +* @pfn: The target page frame number +* +* Return: The freetype of the pageblock +*/ +__always_inline freetype_t +get_pfnblock_freetype (const struct page *page,unsigned long pfn) +{ +return __get_pfnblock_freetype (page, pfn,0 );
And here it passes 0 to bool.
Quoted 3 lines
mm/page_alloc.c +3
-0+} + +
Brendan Jackman
reviewerRe: [PATCH v2 10/22] mm: introduce freetype_t
Quoted 7 lines · Vlastimil Babka (SUSE)
Quoted 6 lines · Brendan Jackman
mm/page_alloc.c +0
-0@@ -422,6 +422,37 @@ bool get_pfnblock_bit(const struct page *page, unsigned long pfn,return test_bit (bitidx + pb_bit, bitmap_word); }
Quoted 13 lines
mm/page_alloc.c bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · line 435
Quoted 11 lines
mm/page_alloc.c +11
-0+/** +* __get_pfnblock_freetype - Return the freetype of a pageblock, optionally +* ignoring the fact that it's currently isolated. +* @page: The page within the block of interest +* @pfn: The target page frame number +* @ignore_iso: If isolated, return the migratetype that the block had before +* isolation. +*/ +__always_inline freetype_t +__get_pfnblock_freetype (const struct page *page,unsigned long pfn, +bool ignore_iso)Hm I also noticed
ignore_isois ... ignored until later patch.
BTW just to note, Sashiko spotted this bug too:
https://sashiko.dev/#/patchset/20260320-page_alloc-unmapped-v2-0-28bf1bd54f41%40google.com
PATCH v2 11/22 mm: move migratetype definitions to freetype.h
In reply to Brendan JackmanSince migratetypes are a sub-element of freetype, move the pure definitions into the new freetype.h.
This will enable referring to these raw types from pageblock-flags.h.
--- include/linux/freetype.h | 84 ++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/mmzone.h | 73 ----------------------------------------- 2 files changed, 84 insertions(+), 73 deletions(-)
include/linux/freetype.h +84 -0
@@ -3,6 +3,66 @@#define _LINUX_FREETYPE_H#include <linux/types.h> +#include <linux/mmdebug.h> + +/* +* A migratetype is the part of a freetype that encodes the mobility +* requirements for the allocations the freelist is intended to serve. +* +* It's also currently overloaded to encode page isolation state. +*/ +enum migratetype { +MIGRATE_UNMOVABLE , +MIGRATE_MOVABLE , +MIGRATE_RECLAIMABLE , +MIGRATE_PCPTYPES ,/* the number of types on the pcp lists */ +MIGRATE_HIGHATOMIC =MIGRATE_PCPTYPES , +#ifdef CONFIG_CMA +/* +* MIGRATE_CMA migration type is designed to mimic the way +* ZONE_MOVABLE works. Only movable pages can be allocated +* from MIGRATE_CMA pageblocks and page allocator never +* implicitly change migration type of MIGRATE_CMA pageblock. +* +* The way to use it is to change migratetype of a range of +* pageblocks to MIGRATE_CMA which can be done by +* __free_pageblock_cma() function. +*/ +MIGRATE_CMA , + __MIGRATE_TYPE_END =MIGRATE_CMA , +#else + __MIGRATE_TYPE_END =MIGRATE_HIGHATOMIC , +#endif +#ifdef CONFIG_MEMORY_ISOLATION +MIGRATE_ISOLATE ,/* can't allocate from here */ +#endif +MIGRATE_TYPES +}; + +/* In mm/page_alloc.c; keep in sync also with show_migration_types() there */ +extern const char *const migratetype_names[MIGRATE_TYPES ]; + +#ifdef CONFIG_CMA +# define is_migrate_cma (migratetype) unlikely((migratetype) == MIGRATE_CMA) +#else +# define is_migrate_cma (migratetype) false +#endif + +static inline bool is_migrate_movable (int mt) +{ +return is_migrate_cma (mt) || mt ==MIGRATE_MOVABLE ; +} + +/* +* Check whether a migratetype can be merged with another migratetype. +* +* It is only mergeable when it can fall back to other migratetypes for +* allocation. See fallbacks[MIGRATE_TYPES][3] in page_alloc.c. +*/ +static inline bool migratetype_is_mergeable (int mt) +{ +return mt <MIGRATE_PCPTYPES ; +}/* * A freetype is the index used to identify free lists. This consists of a
@@ -35,4 +95,28 @@ static inline bool freetypes_equal(freetype_t a, freetype_t b) return a.migratetype == b.migratetype; } +static inline freetype_t migrate_to_freetype (enum migratetype mt, +unsigned int flags) +{ +freetype_t freetype; + +/* No flags supported yet. */ +VM_WARN_ON_ONCE (flags); + + freetype.migratetype = mt; +return freetype; +} + +static inline enum migratetype free_to_migratetype (freetype_t freetype) +{ +return freetype.migratetype; +} + +/* Convenience helper, return the freetype modified to have the migratetype. */ +static inline freetype_t freetype_with_migrate (freetype_t freetype, +enum migratetype migratetype) +{ +return migrate_to_freetype (migratetype,freetype_flags (freetype)); +} +#endif /* _LINUX_FREETYPE_H */
include/linux/mmzone.h +0 -73
96 lines not on this page · raw message
Signature · 1 line
2.51.2
Re: [PATCH v2 11/22] mm: move migratetype definitions to freetype.h
Quoted 6 lines · Brendan Jackman
Since migratetypes are a sub-element of freetype, move the pure definitions into the new
freetype.h.This will enable referring to these raw types from
pageblock-flags.h.Signed-off-by Brendan Jackman <[email protected]>
git coloring agrees it's just moves
PATCH v2 12/22 mm: add definitions for allocating unmapped pages
In reply to Brendan JackmanCreate __GFP_UNMAPPED, which requests pages that are not present in the direct map. Since this feature has a cost (e.g. more freelists), it's behind a kconfig. Unlike other conditionally-defined GFP flags, it doesn't fall back to being 0. This prevents building code that uses __GFP_UNMAPPED but doesn't depend on the necessary kconfig, since that would lead to invisible security issues.
Create a freetype flag to record that pages on the freelists with this flag are unmapped. This is currently only needed for MIGRATE_UNMOVABLE pages, so the freetype encoding remains trivial.
Also create the corresponding pageblock flag to record the same thing.
To keep patches from being too overwhelming, the actual implementation is added separately, this is just types, Kconfig boilerplate, etc.
--- include/linux/freetype.h | 43 +++++++++++++++++++++++++++++++++--------- include/linux/gfp_types.h | 26 +++++++++++++++++++++++++ include/trace/events/mmflags.h | 9 ++++++++- mm/Kconfig | 4 ++++ 4 files changed, 72 insertions(+), 10 deletions(-)
include/linux/freetype.h +34 -9
@@ -2,6 +2,7 @@#ifndef _LINUX_FREETYPE_H#define _LINUX_FREETYPE_H +#include <linux/log2.h> #include <linux/types.h> #include <linux/mmdebug.h>
@@ -64,35 +65,61 @@ static inline bool migratetype_is_mergeable(int mt)return mt <MIGRATE_PCPTYPES ; } +enum { +/* Defined unconditionally as a hack to avoid a zero-width bitfield. */ +FREETYPE_UNMAPPED_BIT , +NUM_FREETYPE_FLAGS , +}; +/* * A freetype is the index used to identify free lists. This consists of a * migratetype, and other bits which encode orthogonal properties of memory. */ typedef struct { -int migratetype; +unsigned int migratetype :order_base_2 (MIGRATE_TYPES ); +unsigned int flags :NUM_FREETYPE_FLAGS ; }freetype_t ; +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +#define FREETYPE_UNMAPPED BIT(FREETYPE_UNMAPPED_BIT) +#define NUM_UNMAPPED_FREETYPES 1 +#else +#define FREETYPE_UNMAPPED 0 +#define NUM_UNMAPPED_FREETYPES 0 +#endif +/* * Return a dense linear index for freetypes that have lists in the free area. * Return -1 for other freetypes. */ static inline int freetype_idx (freetype_t freetype) { +/* For FREETYPE_UNMAPPED, only MIGRATE_UNMOVABLE has an index. */ +if (freetype.flags &FREETYPE_UNMAPPED ) { +VM_WARN_ON_ONCE (freetype.flags & ~FREETYPE_UNMAPPED ); +if (!IS_ENABLED (CONFIG_PAGE_ALLOC_UNMAPPED )) +return -1 ; +if (freetype.migratetype !=MIGRATE_UNMOVABLE ) +return -1 ; +return MIGRATE_TYPES ; + } +/* No other flags are supported. */ +VM_WARN_ON_ONCE (freetype.flags); +return freetype.migratetype; } -/* No freetype flags actually exist yet. */ -#define NR_FREETYPE_IDXS MIGRATE_TYPES +/* One for each migratetype, plus one for MIGRATE_UNMOVABLE-FREETYPE_UNMAPPED */ +#define NR_FREETYPE_IDXS (MIGRATE_TYPES + NUM_UNMAPPED_FREETYPES)static inline unsigned int freetype_flags (freetype_t freetype) { -/* No flags supported yet. */ -return 0 ; +return freetype.flags; }static inline bool freetypes_equal (freetype_t a,freetype_t b) { -return a.migratetype == b.migratetype; +return a.migratetype == b.migratetype && a.flags == b.flags; } static inline freetype_t migrate_to_freetype(enum migratetype mt,
@@ -100,10 +127,8 @@ static inline freetype_t migrate_to_freetype(enum migratetype mt, {freetype_t freetype; -/* No flags supported yet. */ -VM_WARN_ON_ONCE (flags); - freetype.migratetype = mt; + freetype.flags = flags;return freetype; }
include/linux/gfp_types.h +26 -0
@@ -56,6 +56,9 @@ enum { ___GFP_NOLOCKDEP_BIT, #endif ___GFP_NO_OBJ_EXT_BIT, +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED + ___GFP_UNMAPPED_BIT, +#endif ___GFP_LAST_BIT };
@@ -97,6 +100,10 @@ enum {#define ___GFP_NOLOCKDEP 0#endif #define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT) +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +#define ___GFP_UNMAPPED BIT(___GFP_UNMAPPED_BIT) +/* No #else - __GFP_UNMAPPED should never be a nop. Break the build if it isn't supported. */ +#endif /* * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -295,6 +302,25 @@ enum {/* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) +/* +* Allocate pages that aren't present in the direct map. If the caller changes +* direct map presence, it must be restored to the previous state before freeing +* the page. (This is true regardless of __GFP_UNMAPPED). +* +* This uses the mermap (when __GFP_ZERO), so it's only valid to allocate with +* this flag where that's valid, namely from process context after the mermap +* has been initialised for that process. This also means that the allocator +* leaves behind stale TLB entries in the mermap region. The caller is +* responsible for ensuring they are flushed as needed. +* +* This is currently incompatible with __GFP_MOVABLE and __GFP_RECLAIMABLE, but +* only because of allocator implementation details, if a usecase arises this +* restriction could be dropped. +*/ +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +#define __GFP_UNMAPPED ((__force gfp_t)___GFP_UNMAPPED) +#endif +/* Room for N __GFP_FOO bits */ #define __GFP_BITS_SHIFT ___GFP_LAST_BIT#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
include/trace/events/mmflags.h +8 -1
@@ -61,11 +61,18 @@# define TRACE_GFP_FLAGS_SLAB #endif +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +# define TRACE_GFP_FLAGS_UNMAPPED TRACE_GFP_EM(UNMAPPED) +#else +# define TRACE_GFP_FLAGS_UNMAPPED +#endif +#define TRACE_GFP_FLAGS \ TRACE_GFP_FLAGS_GENERAL \ TRACE_GFP_FLAGS_KASAN \ TRACE_GFP_FLAGS_LOCKDEP \ - TRACE_GFP_FLAGS_SLAB + TRACE_GFP_FLAGS_SLAB \ + TRACE_GFP_FLAGS_UNMAPPED#undef TRACE_GFP_EM#define TRACE_GFP_EM (a) TRACE_DEFINE_ENUM(___GFP_##a##_BIT);
mm/Kconfig +4 -0
@@ -1506,3 +1506,7 @@ config MERMAP_KUNIT_TEST If unsure, say N. endmenu + +config PAGE_ALLOC_UNMAPPED + bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST + default COMPILE_TEST
Signature · 1 line
2.51.2
Re: [PATCH v2 12/22] mm: add definitions for allocating unmapped pages
Quoted 11 lines · Brendan Jackman
Create
__GFP_UNMAPPED, which requests pages that are not present in the direct map. Since this feature has a cost (e.g. more freelists), it's behind a kconfig. Unlike other conditionally-defined GFP flags, it doesn't fall back to being 0. This prevents building code that uses__GFP_UNMAPPEDbut doesn't depend on the necessary kconfig, since that would lead to invisible security issues.Create a freetype flag to record that pages on the freelists with this flag are unmapped. This is currently only needed for
MIGRATE_UNMOVABLEpages, so the freetype encoding remains trivial.
Quoted 6 lines
Also create the corresponding pageblock flag to record the same thing.
To keep patches from being too overwhelming, the actual implementation is added separately, this is just types,
Kconfigboilerplate, etc.Signed-off-by Brendan Jackman <[email protected]>
Aside from the gfp vs internal flag decision, seems fine.
PATCH v2 13/22 mm: rejig pageblock mask definitions
In reply to Brendan JackmanA later patch will complicate the definition of these masks, this is a preparatory patch to make that patch easier to review.
- More masks will be needed, so add a PAGEBLOCK_ prefix to the names to avoid polluting the "global namespace" too much.
- This makes MIGRATETYPE_AND_ISO_MASK start to look pretty long. Well, that global mask only exists for quite a specific purpose so just drop it and take advantage of the newly-defined PAGEBLOCK_ISO_MASK.
--- include/linux/pageblock-flags.h | 6 +++--- mm/page_alloc.c | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-)
include/linux/pageblock-flags.h +3 -3
@@ -36,12 +36,12 @@ enum pageblock_bits {#define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS)) -#define MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2)) +#define PAGEBLOCK_MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2))#ifdef CONFIG_MEMORY_ISOLATION -#define MIGRATETYPE_AND_ISO_MASK (MIGRATETYPE_MASK | BIT(PB_migrate_isolate)) +#define PAGEBLOCK_ISO_MASK BIT(PB_migrate_isolate)#else -#define MIGRATETYPE_AND_ISO_MASK MIGRATETYPE_MASK +#define PAGEBLOCK_ISO_MASK 0#endif #if defined(CONFIG_HUGETLB_PAGE )
mm/page_alloc.c +6 -6
@@ -362,7 +362,7 @@ get_pfnblock_bitmap_bitidx(const struct page *page, unsigned long pfn,#else BUILD_BUG_ON (NR_PAGEBLOCK_BITS !=4 );#endif -BUILD_BUG_ON (__MIGRATE_TYPE_END >MIGRATETYPE_MASK ); +BUILD_BUG_ON (__MIGRATE_TYPE_END >PAGEBLOCK_MIGRATETYPE_MASK );VM_BUG_ON_PAGE (!zone_spans_pfn (page_zone (page), pfn), page); bitmap =get_pageblock_bitmap (page, pfn);
@@ -466,7 +466,7 @@ get_pfnblock_freetype(const struct page *page, unsigned long pfn) __always_inline enum migratetypeget_pfnblock_migratetype (const struct page *page,unsigned long pfn) { -unsigned long mask =MIGRATETYPE_AND_ISO_MASK ; +unsigned long mask =PAGEBLOCK_MIGRATETYPE_MASK |PAGEBLOCK_ISO_MASK ;unsigned long flags; flags =__get_pfnblock_flags_mask (page, pfn, mask);
@@ -475,7 +475,7 @@ get_pfnblock_migratetype(const struct page *page, unsigned long pfn)if (flags &BIT (PB_migrate_isolate))return MIGRATE_ISOLATE ;#endif -return flags &MIGRATETYPE_MASK ; +return flags &PAGEBLOCK_MIGRATETYPE_MASK ; }/**
@@ -563,11 +563,11 @@ static void set_pageblock_migratetype(struct page *page, }VM_WARN_ONCE (get_pageblock_isolate (page),"Use clear_pageblock_isolate() to unisolate pageblock" ); -/* MIGRATETYPE_AND_ISO_MASK clears PB_migrate_isolate if it is set */ +/* PAGEBLOCK_ISO_MASK clears PB_migrate_isolate if it is set */ #endif __set_pfnblock_flags_mask (page,page_to_pfn (page), (unsigned long )migratetype, -MIGRATETYPE_AND_ISO_MASK ); +PAGEBLOCK_MIGRATETYPE_MASK |PAGEBLOCK_ISO_MASK ); } void __meminit init_pageblock_migratetype(struct page *page,
@@ -593,7 +593,7 @@ void __meminit init_pageblock_migratetype(struct page *page, flags |=BIT (PB_migrate_isolate);#endif __set_pfnblock_flags_mask (page,page_to_pfn (page), flags, -MIGRATETYPE_AND_ISO_MASK ); +PAGEBLOCK_MIGRATETYPE_MASK |PAGEBLOCK_ISO_MASK ); }#ifdef CONFIG_DEBUG_VM
Signature · 1 line
2.51.2
Re: [PATCH v2 13/22] mm: rejig pageblock mask definitions
Quoted 11 lines · Brendan Jackman
A later patch will complicate the definition of these masks, this is a preparatory patch to make that patch easier to review.
- More masks will be needed, so add a
PAGEBLOCK_prefix to the names to avoid polluting the "global namespace" too much.- This makes
MIGRATETYPE_AND_ISO_MASKstart to look pretty long. Well, that global mask only exists for quite a specific purpose so just drop it and take advantage of the newly-definedPAGEBLOCK_ISO_MASK.Signed-off-by Brendan Jackman <[email protected]>
LGTM. Could be also part of the immediate cleanup series?
PATCH v2 14/22 mm: encode freetype flags in pageblock flags
In reply to Brendan JackmanIn preparation for implementing allocation from FREETYPE_UNMAPPED lists.
Since it works nicely with the existing allocator logic, and also offers a simple way to amortize TLB flushing costs, __GFP_UNMAPPED will be implemented by changing mappings at pageblock granularity. Therefore, encode the mapping state in the pageblock flags.
Also add the necessary logic to record this from a freetype, and reconstruct a freetype from the pageblock flags.
--- include/linux/pageblock-flags.h | 10 ++++++++++ mm/page_alloc.c | 33 +++++++++++++++++++++++++-------- 2 files changed, 35 insertions(+), 8 deletions(-)
include/linux/pageblock-flags.h +10 -0
@@ -11,6 +11,8 @@#ifndef PAGEBLOCK_FLAGS_H #define PAGEBLOCK_FLAGS_H +#include <linux/freetype.h> +#include <linux/log2.h> #include <linux/types.h> /* Bit indices that affect a whole block of pages */
@@ -18,6 +20,9 @@ enum pageblock_bits { PB_migrate_0, PB_migrate_1, PB_migrate_2, + PB_freetype_flags, + PB_freetype_flags_end = PB_freetype_flags +NUM_FREETYPE_FLAGS -1 , + PB_compact_skip,/* If set the block is skipped by compaction */ #ifdef CONFIG_MEMORY_ISOLATION
@@ -37,6 +42,7 @@ enum pageblock_bits {#define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS))#define PAGEBLOCK_MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2)) +#define PAGEBLOCK_FREETYPE_FLAGS_MASK (((1 << NUM_FREETYPE_FLAGS) - 1) << PB_freetype_flags)#ifdef CONFIG_MEMORY_ISOLATION #define PAGEBLOCK_ISO_MASK BIT(PB_migrate_isolate)
@@ -44,6 +50,10 @@ enum pageblock_bits {#define PAGEBLOCK_ISO_MASK 0#endif +#define PAGEBLOCK_FREETYPE_MASK (PAGEBLOCK_MIGRATETYPE_MASK | \ + PAGEBLOCK_ISO_MASK | \ + PAGEBLOCK_FREETYPE_FLAGS_MASK) +#if defined(CONFIG_HUGETLB_PAGE )#ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE
mm/page_alloc.c +25 -8
@@ -357,11 +357,8 @@ get_pfnblock_bitmap_bitidx(const struct page *page, unsigned long pfn,unsigned long *bitmap;unsigned long word_bitidx; -#ifdef CONFIG_MEMORY_ISOLATION -BUILD_BUG_ON (NR_PAGEBLOCK_BITS !=8 ); -#else -BUILD_BUG_ON (NR_PAGEBLOCK_BITS !=4 ); -#endif +/* NR_PAGEBLOCK_BITS must divide word size. */ +BUILD_BUG_ON (NR_PAGEBLOCK_BITS !=4 &&NR_PAGEBLOCK_BITS !=8 );BUILD_BUG_ON (__MIGRATE_TYPE_END >PAGEBLOCK_MIGRATETYPE_MASK );VM_BUG_ON_PAGE (!zone_spans_pfn (page_zone (page), pfn), page);
@@ -434,9 +431,20 @@ __always_inline freetype_t __get_pfnblock_freetype(const struct page *page, unsigned long pfn, bool ignore_iso) { - int mt = get_pfnblock_migratetype(page, pfn); + unsigned long mask = PAGEBLOCK_FREETYPE_MASK; +enum migratetype migratetype; +unsigned int ft_flags; +unsigned long flags; -return migrate_to_freetype (mt,0 ); + flags =__get_pfnblock_flags_mask (page, pfn, mask); + ft_flags = (flags &PAGEBLOCK_FREETYPE_FLAGS_MASK ) >> PB_freetype_flags; + + migratetype = flags &PAGEBLOCK_MIGRATETYPE_MASK ; +#ifdef CONFIG_MEMORY_ISOLATION +if (!ignore_iso && flags &BIT (PB_migrate_isolate)) + migratetype =MIGRATE_ISOLATE ; +#endif +return migrate_to_freetype (migratetype, ft_flags); }/**
@@ -570,6 +578,15 @@ static void set_pageblock_migratetype(struct page *page, PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK); } +static inline void set_pageblock_freetype_flags (struct page *page, +unsigned int ft_flags) +{ +unsigned int flags = ft_flags << PB_freetype_flags; + +__set_pfnblock_flags_mask (page,page_to_pfn (page), flags, +PAGEBLOCK_FREETYPE_FLAGS_MASK ); +} +void __meminit init_pageblock_migratetype (struct page *page,enum migratetype migratetype,bool isolate)
@@ -593,7 +610,7 @@ void __meminit init_pageblock_migratetype(struct page *page, flags |=BIT (PB_migrate_isolate);#endif __set_pfnblock_flags_mask (page,page_to_pfn (page), flags, -PAGEBLOCK_MIGRATETYPE_MASK |PAGEBLOCK_ISO_MASK ); +PAGEBLOCK_FREETYPE_MASK ); }#ifdef CONFIG_DEBUG_VM
Signature · 1 line
2.51.2
Re: [PATCH v2 14/22] mm: encode freetype flags in pageblock flags
Quoted 11 lines · Brendan Jackman
In preparation for implementing allocation from
FREETYPE_UNMAPPEDlists.Since it works nicely with the existing allocator logic, and also offers a simple way to amortize TLB flushing costs,
__GFP_UNMAPPEDwill be implemented by changing mappings at pageblock granularity. Therefore, encode the mapping state in the pageblock flags.Also add the necessary logic to record this from a freetype, and reconstruct a freetype from the pageblock flags.
Signed-off-by Brendan Jackman <[email protected]>
nit:
Quoted 7 lines
mm/page_alloc.c +2
-1@@ -434,9 +431,20 @@ __always_inline freetype_t __get_pfnblock_freetype(const struct page *page, unsigned long pfn, bool ignore_iso) { - int mt = get_pfnblock_migratetype(page, pfn); + unsigned long mask = PAGEBLOCK_FREETYPE_MASK; +enum migratetype migratetype;
mm/page_alloc.c __always_inline freetype_t · line 444
Quoted 10 lines
mm/page_alloc.c +8
-1+unsigned int ft_flags; +unsigned long flags; -return migrate_to_freetype (mt,0 ); + flags =__get_pfnblock_flags_mask (page, pfn, mask); + ft_flags = (flags &PAGEBLOCK_FREETYPE_FLAGS_MASK ) >> PB_freetype_flags; + + migratetype = flags &PAGEBLOCK_MIGRATETYPE_MASK ; +#ifdef CONFIG_MEMORY_ISOLATION +if (!ignore_iso && flags &BIT (PB_migrate_isolate))
(flags & BIT(PB_migrate_isolate)) ?
Quoted 32 lines
mm/page_alloc.c +13
-1+ migratetype =MIGRATE_ISOLATE ; +#endif +return migrate_to_freetype (migratetype, ft_flags); }/** @@ -570,6 +578,15 @@ static void set_pageblock_migratetype(struct page *page, PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK); } +static inline void set_pageblock_freetype_flags (struct page *page, +unsigned int ft_flags) +{ +unsigned int flags = ft_flags << PB_freetype_flags; + +__set_pfnblock_flags_mask (page,page_to_pfn (page), flags, +PAGEBLOCK_FREETYPE_FLAGS_MASK ); +} +void __meminit init_pageblock_migratetype (struct page *page,enum migratetype migratetype,bool isolate)@@ -593,7 +610,7 @@ void __meminit init_pageblock_migratetype(struct page *page, flags |=BIT (PB_migrate_isolate);#endif __set_pfnblock_flags_mask (page,page_to_pfn (page), flags, -PAGEBLOCK_MIGRATETYPE_MASK |PAGEBLOCK_ISO_MASK ); +PAGEBLOCK_FREETYPE_MASK ); }#ifdef CONFIG_DEBUG_VM
PATCH v2 15/22 mm/page_alloc: remove ifdefs from pindex helpers
In reply to Brendan JackmanThe ifdefs are not technically needed here, everything used here is always defined.
They aren't doing much harm right now but a following patch will complicate these functions. Switching to IS_ENABLED() makes the code a bit less tiresome to read.
--- mm/page_alloc.c | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-)
mm/page_alloc.c +14 -16
@@ -696,19 +696,17 @@ static void bad_page(struct page *page, const char *reason)static inline unsigned int order_to_pindex (int migratetype,int order) { +if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) { +bool movable = migratetype ==MIGRATE_MOVABLE ; -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -bool movable; -if (order >PAGE_ALLOC_COSTLY_ORDER ) { -VM_BUG_ON (!is_pmd_order (order)); +if (order >PAGE_ALLOC_COSTLY_ORDER ) { +VM_BUG_ON (!is_pmd_order (order)); - movable = migratetype ==MIGRATE_MOVABLE ; - -return NR_LOWORDER_PCP_LISTS + movable; +return NR_LOWORDER_PCP_LISTS + movable; + } + }else { +VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); } -#else -VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); -#endif return (MIGRATE_PCPTYPES * order) + migratetype; }
@@ -717,12 +715,12 @@ static inline int pindex_to_order(unsigned int pindex) {int order = pindex /MIGRATE_PCPTYPES ; -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (pindex >=NR_LOWORDER_PCP_LISTS ) - order =HPAGE_PMD_ORDER ; -#else -VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); -#endif +if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) { +if (pindex >=NR_LOWORDER_PCP_LISTS ) + order =HPAGE_PMD_ORDER ; + }else { +VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); + }return order; }
Signature · 1 line
2.51.2
Re: [PATCH v2 15/22] mm/page_alloc: remove ifdefs from pindex helpers
Quoted 8 lines · Brendan Jackman
The ifdefs are not technically needed here, everything used here is always defined.
They aren't doing much harm right now but a following patch will complicate these functions. Switching to
IS_ENABLED()makes the code a bit less tiresome to read.Signed-off-by Brendan Jackman <[email protected]>
Also good for prep series?
Re: [PATCH v2 15/22] mm/page_alloc: remove ifdefs from pindex helpers
Quoted 8 lines · Vlastimil Babka (SUSE)
Quoted 7 lines · Brendan Jackman
The ifdefs are not technically needed here, everything used here is always defined.
They aren't doing much harm right now but a following patch will complicate these functions. Switching to
IS_ENABLED()makes the code a bit less tiresome to read.
Quoted 5 lines
Quoted 1 line
Signed-off-by Brendan Jackman <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Also good for prep series?
Um, shrug. I guess I will send a series with all the things that could _arguably_ considered "cleanups", and then I will let you/David/whoever else chimes in decide on which to pick and which ones would be "churn".
PATCH v2 16/22 mm/page_alloc: separate pcplists by freetype flags
In reply to Brendan JackmanThe normal freelists are already separated by this flag, so now update the pcplists accordingly. This follows the most "obvious" design where __GFP_UNMAPPED is supported at arbitrary orders.
If necessary, it would be possible to avoid the proliferation of pcplists by restricting orders that can be allocated from them with this FREETYPE_UNMAPPED.
On the other hand, there's currently no usecase for movable/reclaimable unmapped memory, and constraining the migratetype doesn't have any tricky plumbing implications. So, take advantage of that and assume that FREETYPE_UNMAPPED implies MIGRATE_UNMOVABLE.
Overall, this just takes the existing space of pindices and tacks another bank on the end. For !THP this is just 4 more lists, with THP there is a single additional list for hugepages.
--- include/linux/mmzone.h | 11 ++++++++++- mm/page_alloc.c | 44 +++++++++++++++++++++++++++++++++----------- 2 files changed, 43 insertions(+), 12 deletions(-)
include/linux/mmzone.h +10 -1
@@ -778,8 +778,17 @@ enum zone_watermarks {#else #define NR_PCP_THP 0#endif +/* +* FREETYPE_UNMAPPED can currently only be used with MIGRATE_UNMOVABLE, no for +* those there's no need to encode the migratetype in the pindex. +*/ +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +#define NR_UNMAPPED_PCP_LISTS (PAGE_ALLOC_COSTLY_ORDER + 1 + !!NR_PCP_THP) +#else +#define NR_UNMAPPED_PCP_LISTS 0 +#endif #define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1)) -#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP) +#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP + NR_UNMAPPED_PCP_LISTS)/* * Flags used in pcp->flags field.
mm/page_alloc.c +33 -11
80 lines not on this page · raw message
Signature · 1 line
2.51.2
Re: [PATCH v2 16/22] mm/page_alloc: separate pcplists by freetype flags
Quoted 13 lines · Brendan Jackman
The normal freelists are already separated by this flag, so now update the pcplists accordingly. This follows the most "obvious" design where
__GFP_UNMAPPEDis supported at arbitrary orders.If necessary, it would be possible to avoid the proliferation of pcplists by restricting orders that can be allocated from them with this
FREETYPE_UNMAPPED.On the other hand, there's currently no usecase for movable/reclaimable unmapped memory, and constraining the migratetype doesn't have any tricky plumbing implications. So, take advantage of that and assume that
FREETYPE_UNMAPPEDimpliesMIGRATE_UNMOVABLE.
Quoted 5 lines
Overall, this just takes the existing space of pindices and tacks another bank on the end. For !THP this is just 4 more lists, with THP there is a single additional list for hugepages.
Signed-off-by Brendan Jackman <[email protected]>
Nit:
Quoted 10 lines
--- include/linux/mmzone.h | 11 ++++++++++- mm/page_alloc.c | 44 +++++++++++++++++++++++++++++++++----------- 2 files changed, 43 insertions(+), 12 deletions(-)include/linux/mmzone.h +0
-0
include/linux/mmzone.h enum zone_watermarks { · line 782
Quoted 5 lines
include/linux/mmzone.h +2
-0@@ -778,8 +778,17 @@ enum zone_watermarks {#else #define NR_PCP_THP 0#endif +/* +* FREETYPE_UNMAPPED can currently only be used with MIGRATE_UNMOVABLE, no for
so ^
Quoted 15 lines
include/linux/mmzone.h +8
-1+* those there's no need to encode the migratetype in the pindex. +*/ +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +#define NR_UNMAPPED_PCP_LISTS (PAGE_ALLOC_COSTLY_ORDER + 1 + !!NR_PCP_THP) +#else +#define NR_UNMAPPED_PCP_LISTS 0 +#endif #define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1)) -#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP) +#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP + NR_UNMAPPED_PCP_LISTS)/* * Flags used in pcp->flags field. mm/page_alloc.c +0
-0
PATCH v2 17/22 mm/page_alloc: rename ALLOC_NON_BLOCK back to _HARDER
In reply to Brendan JackmanCommit 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves") renamed ALLOC_HARDER to ALLOC_NON_BLOCK because the former is "a vague description".
However, vagueness is accurate here, this is a vague flag. It is not set for __GFP_NOMEMALLOC. It doesn't really mean "allocate without blocking" but rather "allow dipping into atomic reserves, _because_ of the need not to block".
A later commit will need an alloc flag that really means "don't block here", so go back to the flag's old name and update the commentary to try and give it a slightly clearer meaning.
--- mm/internal.h | 9 +++++---- mm/page_alloc.c | 8 ++++---- 2 files changed, 9 insertions(+), 8 deletions(-)
mm/internal.h +5 -4
@@ -1413,9 +1413,10 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_OOM ALLOC_NO_WATERMARKS#endif -#define ALLOC_NON_BLOCK 0x10/* Caller cannot block. Allow access -* to 25% of the min watermark or -* 62.5% if __GFP_HIGH is set. +#define ALLOC_HARDER 0x10/* Because the caller cannot block, +* allow access * to 25% of the min +* watermark or 62.5% if __GFP_HIGH is +* set. */ #define ALLOC_MIN_RESERVE 0x20/* __GFP_HIGH set. Allow access to 50% * of the min watermark.
@@ -1432,7 +1433,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ -#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) +#define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)enum ttu_flags ;struct tlbflush_unmap_batch ;
mm/page_alloc.c +4 -4
@@ -3355,7 +3355,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,* reserves as failing now is worse than failing a * high-order atomic allocation in the future. */ -if (!page && (alloc_flags & (ALLOC_OOM |ALLOC_NON_BLOCK ))) +if (!page && (alloc_flags & (ALLOC_OOM |ALLOC_HARDER ))) page =__rmqueue_smallest (zone, order, ft_high);if (!page) {
@@ -3717,7 +3717,7 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,* or (GFP_KERNEL & ~__GFP_DIRECT_RECLAIM) do not get * access to the min reserve. */ -if (alloc_flags &ALLOC_NON_BLOCK ) +if (alloc_flags &ALLOC_HARDER ) min -= min /4 ; }
@@ -4602,7 +4602,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)* The caller may dip into page reserves a bit more if the caller * cannot run direct reclaim, or if the caller has realtime scheduling * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will -* set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). +* set both ALLOC_HARDER and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |= (__force int ) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM));
@@ -4613,7 +4613,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)* if it can't schedule. */ if (!(gfp_mask & __GFP_NOMEMALLOC)) { - alloc_flags |=ALLOC_NON_BLOCK ; + alloc_flags |=ALLOC_HARDER ;if (order >0 && (alloc_flags &ALLOC_MIN_RESERVE )) alloc_flags |=ALLOC_HIGHATOMIC ;
Signature · 1 line
2.51.2
PATCH v2 18/22 mm/page_alloc: introduce ALLOC_NOBLOCK
In reply to Brendan JackmanThis flag is set unless we can be sure the caller isn't in an atomic context.
The allocator will soon start needing to call set_direct_map_* APIs which cannot be called with IRQs off. It will need to do this even before direct reclaim is possible.
Despite the fact that, in principle, ALLOC_NOBLOCK is distinct from __GFP_DIRECT_RECLAIM, in order to avoid introducing a GFP flag, just infer the former based on whether the caller set the latter. This means that, in practice, ALLOC_NOBLOCK is just !__GFP_DIRECT_RECLAIM, except that it is not influenced by gfp_allowed_mask. This could change later, though.
Call it ALLOC_NOBLOCK in order to try and mitigate confusion vs the recently-removed ALLOC_NON_BLOCK, which meant something different.
--- mm/internal.h | 1 + mm/page_alloc.c | 29 ++++++++++++++++++++++------- 2 files changed, 23 insertions(+), 7 deletions(-)
mm/internal.h +1 -0
@@ -1431,6 +1431,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_TRYLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#define ALLOC_NOBLOCK 0x1000/* Caller may be atomic */ /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
mm/page_alloc.c +22 -7
@@ -4608,6 +4608,8 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM));if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) { + alloc_flags |=ALLOC_NOBLOCK ; +/* * Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule.
@@ -4801,14 +4803,13 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)static inline struct page *__alloc_pages_slowpath (gfp_t gfp_mask,unsigned int order, -struct alloc_context *ac) +struct alloc_context *ac,unsigned int alloc_flags) {bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;bool can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask);bool nofail = gfp_mask & __GFP_NOFAIL;const bool costly_order = order >PAGE_ALLOC_COSTLY_ORDER ;struct page *page =NULL ; -unsigned int alloc_flags;unsigned long did_some_progress;enum compact_priority compact_priority;enum compact_result compact_result;
@@ -4860,7 +4861,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* kswapd needs to be woken up, and to avoid the cost of setting up * alloc_flags precisely. So we do that now. */ - alloc_flags =gfp_to_alloc_flags (gfp_mask, order); + alloc_flags |=gfp_to_alloc_flags (gfp_mask, order);/* * We need to recalculate the starting point for the zonelist iterator
@@ -5086,6 +5087,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,return page; } +static inline unsigned int init_alloc_flags (gfp_t gfp_mask,unsigned int flags) +{ +/* +* If the caller allowed __GFP_DIRECT_RECLAIM, they can't be atomic. +* Note this is a separate determination from whether direct reclaim is +* actually allowed, it must happen before applying gfp_allowed_mask. +*/ +if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) + flags |=ALLOC_NOBLOCK ; +return flags; +} + static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, int preferred_nid, nodemask_t *nodemask, struct alloc_context *ac, gfp_t *alloc_gfp,
@@ -5166,7 +5179,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,struct list_head *pcp_list;struct alloc_context ac;gfp_t alloc_gfp; -unsigned int alloc_flags =ALLOC_WMARK_LOW ; +unsigned int alloc_flags =init_alloc_flags (gfp,ALLOC_WMARK_LOW );int nr_populated =0 , nr_account =0 ;/*
@@ -5307,7 +5320,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { struct page *page; -unsigned int alloc_flags =ALLOC_WMARK_LOW ; +unsigned int alloc_flags =init_alloc_flags (gfp,ALLOC_WMARK_LOW );gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ struct alloc_context ac = { };
@@ -5352,7 +5365,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,*/ ac.nodemask = nodemask; - page =__alloc_pages_slowpath (alloc_gfp, order, &ac); + page =__alloc_pages_slowpath (alloc_gfp, order, &ac, alloc_flags); out: if (memcg_kmem_online() && (gfp & __GFP_ACCOUNT) && page &&
@@ -7872,11 +7885,13 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned*/ gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | __GFP_COMP | gfp_flags; -unsigned int alloc_flags =ALLOC_TRYLOCK ; +unsigned int alloc_flags =init_alloc_flags (alloc_gfp,ALLOC_TRYLOCK );struct alloc_context ac = { };struct page *page;VM_WARN_ON_ONCE (gfp_flags & ~__GFP_ACCOUNT); +VM_WARN_ON_ONCE (!(alloc_flags &ALLOC_NOBLOCK )); +/* * In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is * unsafe in NMI. If spin_trylock() is called from hard IRQ the current
Signature · 1 line
2.51.2
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 18/22] mm/page_alloc: introduce ALLOC_NOBLOCK
Quoted 7 lines · Brendan Jackman
This flag is set unless we can be sure the caller isn't in an atomic context.
The allocator will soon start needing to call set_direct_map_* APIs which cannot be called with IRQs off. It will need to do this even before direct reclaim is possible.
Quoted 6 lines
Despite the fact that, in principle,
ALLOC_NOBLOCKis distinct from__GFP_DIRECT_RECLAIM, in order to avoid introducing a GFP flag, just infer the former based on whether the caller set the latter. This means that, in practice,ALLOC_NOBLOCKis just !__GFP_DIRECT_RECLAIM, except that it is not influenced bygfp_allowed_mask. This could change later, though.
I don't think it should change later? We wouldn't want false positives during boot, or what do you have in mind?
I wonder if the implementation of the "not influenced" is correct though...
Quoted 26 lines
Call it
ALLOC_NOBLOCKin order to try and mitigate confusion vs the recently-removedALLOC_NON_BLOCK, which meant something different.Signed-off-by Brendan Jackman <[email protected]>--- mm/internal.h | 1 + mm/page_alloc.c | 29 ++++++++++++++++++++++------- 2 files changed, 23 insertions(+), 7 deletions(-)mm/internal.h +1
-0@@ -1431,6 +1431,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_TRYLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#define ALLOC_NOBLOCK 0x1000/* Caller may be atomic */ /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)mm/page_alloc.c +0
-0
mm/page_alloc.c gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) · line 4611
Quoted 4 lines
mm/page_alloc.c +1
-0@@ -4608,6 +4608,8 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM));if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) { + alloc_flags |=ALLOC_NOBLOCK ;
When this is called from __alloc_pages_slowpath(), gfp_allowed_mask is already applied, so it will be influenced.
Quoted 21 lines
mm/page_alloc.c +2
-2+/* * Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule. @@ -4801,14 +4803,13 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)static inline struct page *__alloc_pages_slowpath (gfp_t gfp_mask,unsigned int order, -struct alloc_context *ac) +struct alloc_context *ac,unsigned int alloc_flags) {bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;bool can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask);bool nofail = gfp_mask & __GFP_NOFAIL;const bool costly_order = order >PAGE_ALLOC_COSTLY_ORDER ;struct page *page =NULL ; -unsigned int alloc_flags;unsigned long did_some_progress;enum compact_priority compact_priority;enum compact_result compact_result;
mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4864
Quoted 5 lines
mm/page_alloc.c +1
-1@@ -4860,7 +4861,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* kswapd needs to be woken up, and to avoid the cost of setting up * alloc_flags precisely. So we do that now. */ - alloc_flags =gfp_to_alloc_flags (gfp_mask, order); + alloc_flags |=gfp_to_alloc_flags (gfp_mask, order);
Is it safe to just combine them? You come with ALLOC_WMARK_LOW and combine with ALLOC_WMARK_MIN from gfp_to_alloc_flags() but these are not bit flags, I think you end up with ALLOC_WMARK_LOW effectively.
Probably you need to pass the old alloc_flags to gfp_to_alloc_flags, mask only ALLOC_NOBLOCK from it and combine with newly calculated alloc_flags. By not recomputing ALLOC_NOBLOCK you also avoid the problem pointed out above?
(or we decide to not use gfp flag but a new function and then it's more like what alloc_frozen_pages_nolock_noprof() does).
Quoted 65 lines
mm/page_alloc.c +18
-4/* * We need to recalculate the starting point for the zonelist iterator @@ -5086,6 +5087,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,return page; } +static inline unsigned int init_alloc_flags (gfp_t gfp_mask,unsigned int flags) +{ +/* +* If the caller allowed __GFP_DIRECT_RECLAIM, they can't be atomic. +* Note this is a separate determination from whether direct reclaim is +* actually allowed, it must happen before applying gfp_allowed_mask. +*/ +if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) + flags |=ALLOC_NOBLOCK ; +return flags; +} + static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, int preferred_nid, nodemask_t *nodemask, struct alloc_context *ac, gfp_t *alloc_gfp,@@ -5166,7 +5179,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,struct list_head *pcp_list;struct alloc_context ac;gfp_t alloc_gfp; -unsigned int alloc_flags =ALLOC_WMARK_LOW ; +unsigned int alloc_flags =init_alloc_flags (gfp,ALLOC_WMARK_LOW );int nr_populated =0 , nr_account =0 ;/* @@ -5307,7 +5320,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { struct page *page; -unsigned int alloc_flags =ALLOC_WMARK_LOW ; +unsigned int alloc_flags =init_alloc_flags (gfp,ALLOC_WMARK_LOW );gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ struct alloc_context ac = { };@@ -5352,7 +5365,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,*/ ac.nodemask = nodemask; - page =__alloc_pages_slowpath (alloc_gfp, order, &ac); + page =__alloc_pages_slowpath (alloc_gfp, order, &ac, alloc_flags); out: if (memcg_kmem_online() && (gfp & __GFP_ACCOUNT) && page &&@@ -7872,11 +7885,13 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned*/ gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | __GFP_COMP | gfp_flags; -unsigned int alloc_flags =ALLOC_TRYLOCK ; +unsigned int alloc_flags =init_alloc_flags (alloc_gfp,ALLOC_TRYLOCK );struct alloc_context ac = { };struct page *page;VM_WARN_ON_ONCE (gfp_flags & ~__GFP_ACCOUNT); +VM_WARN_ON_ONCE (!(alloc_flags &ALLOC_NOBLOCK )); +/* * In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is * unsafe in NMI. If spin_trylock() is called from hard IRQ the current
Re: [PATCH v2 18/22] mm/page_alloc: introduce ALLOC_NOBLOCK
Quoted 15 lines · Vlastimil Babka (SUSE)
Quoted 7 lines · Brendan Jackman
This flag is set unless we can be sure the caller isn't in an atomic context.
The allocator will soon start needing to call set_direct_map_* APIs which cannot be called with IRQs off. It will need to do this even before direct reclaim is possible.
Quoted 6 lines
Despite the fact that, in principle,
ALLOC_NOBLOCKis distinct from__GFP_DIRECT_RECLAIM, in order to avoid introducing a GFP flag, just infer the former based on whether the caller set the latter. This means that, in practice,ALLOC_NOBLOCKis just !__GFP_DIRECT_RECLAIM, except that it is not influenced bygfp_allowed_mask. This could change later, though.
Quoted 2 lines
I don't think it should change later? We wouldn't want false positives during boot, or what do you have in mind?
I don't think I had anything specific in mind or any reason to _want_ to change it. But I think (??) there are reasons to clear __GFP_DIRECT_RECLAIM even if you are not atomic? Like some sort of generalisation of __GFP_NOIO/NOFS. So all I'm getting at here is: I'm using __GFP_DIRECT_RECLAIM to set ALLOC_NOBLOCK, but I think of that as a total implementation detail and these two flags should conceptually be decoupled.
Quoted 1 line
I wonder if the implementation of the "not influenced" is correct though...
This has been broken in several local iterations of this patchset so I would not be surprised...
Quoted 28 lines
Quoted 28 lines
Call it
ALLOC_NOBLOCKin order to try and mitigate confusion vs the recently-removedALLOC_NON_BLOCK, which meant something different.Signed-off-by Brendan Jackman <[email protected]>--- mm/internal.h | 1 + mm/page_alloc.c | 29 ++++++++++++++++++++++------- 2 files changed, 23 insertions(+), 7 deletions(-)mm/internal.h +1
-0@@ -1431,6 +1431,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_TRYLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#define ALLOC_NOBLOCK 0x1000/* Caller may be atomic */ /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)mm/page_alloc.c +0
-0@@ -4608,6 +4608,8 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM));
Quoted 5 lines
mm/page_alloc.c gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) · line 4611
Quoted 2 lines
mm/page_alloc.c +1
-0if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) { + alloc_flags |=ALLOC_NOBLOCK ;When this is called from
__alloc_pages_slowpath(),gfp_allowed_maskis already applied, so it will be influenced.
... yep.
I have tried to generally refactor the flag setup in here to make these kinda mistakes harder but I didn't have any good ideas (this was when I spotted [0]). Maybe I was being too timid, I will try again.
[0] https://lore.kernel.org/linux-mm/20260331-b4-prepare_alloc_pages-flags-v1-1-ea2416def698@google.com/
Quoted 22 lines
Quoted 22 lines
mm/page_alloc.c +2
-2+/* * Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule. @@ -4801,14 +4803,13 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)static inline struct page *__alloc_pages_slowpath (gfp_t gfp_mask,unsigned int order, -struct alloc_context *ac) +struct alloc_context *ac,unsigned int alloc_flags) {bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;bool can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask);bool nofail = gfp_mask & __GFP_NOFAIL;const bool costly_order = order >PAGE_ALLOC_COSTLY_ORDER ;struct page *page =NULL ; -unsigned int alloc_flags;unsigned long did_some_progress;enum compact_priority compact_priority;enum compact_result compact_result;@@ -4860,7 +4861,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* kswapd needs to be woken up, and to avoid the cost of setting up
Quoted 8 lines
mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4864
Quoted 4 lines
mm/page_alloc.c +1
-1* alloc_flags precisely. So we do that now. */ - alloc_flags =gfp_to_alloc_flags (gfp_mask, order); + alloc_flags |=gfp_to_alloc_flags (gfp_mask, order);Is it safe to just combine them? You come with
ALLOC_WMARK_LOWand combine withALLOC_WMARK_MINfromgfp_to_alloc_flags()but these are not bit flags, I think you end up withALLOC_WMARK_LOWeffectively.
Ah, thanks, I do remember thinking about this and deciding that it was safe but I probably just misunderstood the watermark code.
This makes me a bit more attracted to the idea of a struct like Gregory suggested in [1]. Then this could be captured in the type system.
Quoted 3 lines
Probably you need to pass the old
alloc_flagstogfp_to_alloc_flags, mask onlyALLOC_NOBLOCKfrom it and combine with newly calculatedalloc_flags. By not recomputingALLOC_NOBLOCKyou also avoid the problem pointed out above?
Nice, thanks for the pointer.
Quoted 2 lines
(or we decide to not use gfp flag but a new function and then it's more like what
alloc_frozen_pages_nolock_noprof()does).
Re: [PATCH v2 18/22] mm/page_alloc: introduce ALLOC_NOBLOCK
Quoted 6 lines · Brendan Jackman
Quoted 4 lines · Vlastimil Babka (SUSE)
Is it safe to just combine them? You come with
ALLOC_WMARK_LOWand combine withALLOC_WMARK_MINfromgfp_to_alloc_flags()but these are not bit flags, I think you end up withALLOC_WMARK_LOWeffectively.
Quoted 6 lines
Ah, thanks, I do remember thinking about this and deciding that it was safe but I probably just misunderstood the watermark code.
This makes me a bit more attracted to the idea of a struct like Gregory suggested in [1]. Then this could be captured in the type system.
This might actually be a good precursor before pulling alloc_context any further out. Since there's multiple paths into the lower layers with alloc_context, maybe there should already be a sanity check of the alloc flags anyway.
~Gregory
PATCH v2 19/22 mm/page_alloc: implement __GFP_UNMAPPED allocations
In reply to Brendan JackmanCurrently __GFP_UNMAPPED allocs will always fail because, although the lists exist to hold them, there is no way to actually create an unmapped page block. This commit adds one, and also the logic to map it back again when that's needed.
Doing this at pageblock granularity ensures that the pageblock flags can be used to infer which freetype a page belongs to. It also provides nice batching of TLB flushes, and also avoids creating too much unnecessary TLB fragmentation in the physmap.
There are some functional requirements for flipping a block:
- Unmapping requires a TLB shootdown, meaning IRQs must be enabled.
- Because the main usecase of this feature is to protect against CPU exploits, when a block is mapped it needs to be zeroed to ensure no residual data is available to attackers. Zeroing a block with a spinlock held seems undesirable.
- Updating the pagetables might require allocating a pagetable to break down a huge page. This would deadlock if the zone lock was held.
This makes allocations that need to change sensitivity _somewhat_ similar to those that need to fallback to a different migratetype. But, the locking requirements mean that this can't just be squashed into the existing "fallback" allocator logic, instead a new allocator path just for this purpose is needed.
The new path is assumed to be much cheaper than the really heavyweight stuff like compaction and reclaim. But at present it is treated as less desirable than the mobility-related "fallback" and "stealing" logic. This might turn out to need revision (in particular, maybe it's a problem that __rmqueue_steal(), which causes fragmentation, happens before __rmqueue_direct_map()), but that should be treated as a subsequent optimisation project.
This currently forbids __GFP_ZERO, this is just to keep the patch from getting too large, the next patch will remove this restriction.
--- include/linux/gfp.h | 11 +++- mm/Kconfig | 4 +- mm/page_alloc.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 170 insertions(+), 16 deletions(-)
include/linux/gfp.h +10 -1
@@ -24,6 +24,7 @@ struct mempolicy;static inline freetype_t gfp_freetype (const gfp_t gfp_flags) {int migratetype; +unsigned int ft_flags =0 ;VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK );BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE);
@@ -40,7 +41,15 @@ static inline freetype_t gfp_freetype(const gfp_t gfp_flags) >> GFP_MOVABLE_SHIFT; } -return migrate_to_freetype (migratetype,0 ); +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +if (gfp_flags & __GFP_UNMAPPED) { +if (WARN_ON_ONCE (migratetype !=MIGRATE_UNMOVABLE )) + migratetype =MIGRATE_UNMOVABLE ; + ft_flags |=FREETYPE_UNMAPPED ; + } +#endif + +return migrate_to_freetype (migratetype, ft_flags); }#undef GFP_MOVABLE_MASK#undef GFP_MOVABLE_SHIFT
mm/Kconfig +2 -2
@@ -1505,8 +1505,8 @@ config MERMAP_KUNIT_TEST If unsure, say N. -endmenu - config PAGE_ALLOC_UNMAPPED bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST default COMPILE_TEST + +endmenu
mm/page_alloc.c +158 -13
245 lines not on this page · raw message
Signature · 1 line
2.51.2
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 19/22] mm/page_alloc: implement __GFP_UNMAPPED allocations
Quoted 10 lines · Brendan Jackman
Currently
__GFP_UNMAPPEDallocs will always fail because, although the lists exist to hold them, there is no way to actually create an unmapped page block. This commit adds one, and also the logic to map it back again when that's needed.Doing this at pageblock granularity ensures that the pageblock flags can be used to infer which freetype a page belongs to. It also provides nice batching of TLB flushes, and also avoids creating too much unnecessary TLB fragmentation in the physmap.
Quoted 8 lines
There are some functional requirements for flipping a block:
- Unmapping requires a TLB shootdown, meaning IRQs must be enabled.
- Because the main usecase of this feature is to protect against CPU exploits, when a block is mapped it needs to be zeroed to ensure no residual data is available to attackers. Zeroing a block with a spinlock held seems undesirable.
Did I overlook something or this patch doesn't do this whole block zeroing? Or is it handled by set_direct_map_valid_noflush itself?
Quoted 11 lines
- Updating the pagetables might require allocating a pagetable to break down a huge page. This would deadlock if the zone lock was held.
This makes allocations that need to change sensitivity _somewhat_ similar to those that need to fallback to a different migratetype. But, the locking requirements mean that this can't just be squashed into the existing "fallback" allocator logic, instead a new allocator path just for this purpose is needed.
The new path is assumed to be much cheaper than the really heavyweight stuff like compaction and reclaim. But at present it is treated as less
Uhh, speaking of compaction and reclaim... we rely on finding a whole free pageblock in order to flip it. If that doesn't exist, the whole get_page_from_freelist() will fail, and we might enter the reclaim/compaction cycle in __allow_pages_slowpath(). But since we might ultimately want an order-0 allocation, there won't be any compaction attempted, because that code won't know we failed to flip a pageblock. And the watermarks might look good and prevent reclaim as well I think? We should somehow indicate this, and handle accordingly. Might not be trivial. Or maybe reuse pageblock isolation code to do the migrations directly in __rmqueue_direct_map?
Quoted 136 lines
desirable than the mobility-related "fallback" and "stealing" logic. This might turn out to need revision (in particular, maybe it's a problem that
__rmqueue_steal(), which causes fragmentation, happens before__rmqueue_direct_map()), but that should be treated as a subsequent optimisation project.This currently forbids
__GFP_ZERO, this is just to keep the patch from getting too large, the next patch will remove this restriction.Signed-off-by Brendan Jackman <[email protected]>--- include/linux/gfp.h | 11 +++- mm/Kconfig | 4 +- mm/page_alloc.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 170 insertions(+), 16 deletions(-)include/linux/gfp.h +10
-1@@ -24,6 +24,7 @@ struct mempolicy;static inline freetype_t gfp_freetype (const gfp_t gfp_flags) {int migratetype; +unsigned int ft_flags =0 ;VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK );BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE);@@ -40,7 +41,15 @@ static inline freetype_t gfp_freetype(const gfp_t gfp_flags) >> GFP_MOVABLE_SHIFT; } -return migrate_to_freetype (migratetype,0 ); +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +if (gfp_flags & __GFP_UNMAPPED) { +if (WARN_ON_ONCE (migratetype !=MIGRATE_UNMOVABLE )) + migratetype =MIGRATE_UNMOVABLE ; + ft_flags |=FREETYPE_UNMAPPED ; + } +#endif + +return migrate_to_freetype (migratetype, ft_flags); }#undef GFP_MOVABLE_MASK#undef GFP_MOVABLE_SHIFTmm/Kconfig +2
-2@@ -1505,8 +1505,8 @@ config MERMAP_KUNIT_TEST If unsure, say N. -endmenu - config PAGE_ALLOC_UNMAPPED bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST default COMPILE_TEST + +endmenumm/page_alloc.c +29
-4@@ -34,6 +34,7 @@#include <linux/folio_batch.h> #include <linux/memory_hotplug.h> #include <linux/nodemask.h> +#include <linux/set_memory.h> #include <linux/vmstat.h> #include <linux/fault-inject.h> #include <linux/compaction.h> @@ -1002,6 +1003,26 @@ static void change_pageblock_range(struct page *pageblock_page, } } +/* +* Can pages of these two freetypes be combined into a single higher-order free +* page? +*/ +static inline bool can_merge_freetypes (freetype_t a,freetype_t b) +{ +if (freetypes_equal (a, b)) +return true; + +if (!migratetype_is_mergeable (free_to_migratetype (a)) || + !migratetype_is_mergeable (free_to_migratetype (b))) +return false; + +/* +* Mustn't "just" merge pages with different freetype flags, changing +* those requires updating pagetables. +*/ +return freetype_flags (a) ==freetype_flags (b); +} +/* * Freeing function for a buddy system allocator. * @@ -1070,9 +1091,7 @@ static inline void __free_one_page(struct page *page, buddy_ft =get_pfnblock_freetype (buddy, buddy_pfn); buddy_mt =free_to_migratetype (buddy_ft); -if (migratetype != buddy_mt && - (!migratetype_is_mergeable (migratetype) || - !migratetype_is_mergeable (buddy_mt))) +if (!can_merge_freetypes (freetype, buddy_ft)) goto done_merging; }@@ -1089,7 +1108,9 @@ static inline void __free_one_page(struct page *page,/* * Match buddy type. This ensures that an * expand() down the line puts the sub-blocks -* on the right freelists. +* on the right freelists. Freetype flags are +* already set correctly because of +* can_merge_freetypes(). */ change_pageblock_range (buddy, order, migratetype); }@@ -1982,6 +2003,9 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order,struct free_area *area;struct page *page; +if (freetype_idx (freetype) <0 ) +return NULL ; +/* Find a page of the appropriate size in the preferred list */ for (current_order = order; current_order <NR_PAGE_ORDERS ; ++current_order) {enum migratetype migratetype =free_to_migratetype (freetype);@@ -3324,6 +3348,119 @@ static inline void zone_statistics(struct zone *preferred_zone, struct zone *z,#endif } +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED
mm/page_alloc.c static inline void zone_statistics(struct zone *preferred_zone, struct zone *z, · line 3361
Quoted 10 lines
mm/page_alloc.c +10
-0+/* Try to allocate a page by mapping/unmapping a block from the direct map. */ +static inline struct page * +__rmqueue_direct_map (struct zone *zone,unsigned int request_order, +unsigned int alloc_flags,freetype_t freetype) +{ +unsigned int ft_flags_other =freetype_flags (freetype) ^FREETYPE_UNMAPPED ; +freetype_t ft_other =migrate_to_freetype (free_to_migratetype (freetype), + ft_flags_other); +bool want_mapped = !(freetype_flags (freetype) &FREETYPE_UNMAPPED ); +enum rmqueue_mode rmqm =RMQUEUE_NORMAL ;
Why not RMQUEUE_CLAIM? We want to change the migratetype to ours as well, not just the unmapped flag?
Quoted 80 lines
mm/page_alloc.c +80
-0+unsigned long irq_flags; +int nr_pageblocks; +struct page *page; +int alloc_order; +int err; + +if (freetype_idx (ft_other) <0 ) +return NULL ; + +/* +* Might need a TLB shootdown. Even if IRQs are on this isn't +* safe if the caller holds a lock (in case the other CPUs need that +* lock to handle the shootdown IPI). +*/ +if (alloc_flags &ALLOC_NOBLOCK ) +return NULL ; + +if (!can_set_direct_map ()) +return NULL ; + +lockdep_assert (!irqs_disabled () ||unlikely (early_boot_irqs_disabled)); + +/* +* Need to [un]map a whole pageblock (otherwise it might require +* allocating pagetables). First allocate it. +*/ + alloc_order =max (request_order, pageblock_order); + nr_pageblocks =1 << (alloc_order - pageblock_order); +zone_lock_irqsave (zone, irq_flags); + page =__rmqueue (zone, alloc_order, ft_other, alloc_flags, &rmqm); +zone_unlock_irqrestore (zone, irq_flags); +if (!page) +return NULL ; + +/* +* Now that IRQs are on it's safe to do a TLB shootdown, and now that we +* released the zone lock it's possible to allocate a pagetable if +* needed to split up a huge page. +* +* Note that modifying the direct map may need to allocate pagetables. +* What about unbounded recursion? Here are the assumptions that make it +* safe: +* +* - The direct map starts out fully mapped at boot. (This is not really +* an assumption" as its in direct control of page_alloc.c). +* +* - Once pages in the direct map are broken down, they are not +* re-aggregated into larger pages again. +* +* - Pagetables are never allocated with __GFP_UNMAPPED. +* +* Under these assumptions, a pagetable might need to be allocated while +* _unmapping_ stuff from the direct map during a __GFP_UNMAPPED +* allocation. But, the allocation of that pagetable never requires +* allocating a further pagetable. +*/ + err =set_direct_map_valid_noflush (page, + nr_pageblocks << pageblock_order, want_mapped); +if (err == -ENOMEM ||WARN_ONCE (err,"err=%d\n" , err)) { +zone_lock_irqsave (zone, irq_flags); +__free_one_page (page,page_to_pfn (page), zone, + alloc_order, freetype,FPI_SKIP_REPORT_NOTIFY ); +zone_unlock_irqrestore (zone, irq_flags); +return NULL ; + } + +if (!want_mapped) { +unsigned long start = (unsigned long )page_address (page); +unsigned long end = start + (nr_pageblocks << (pageblock_order +PAGE_SHIFT )); + +flush_tlb_kernel_range (start, end); + } + +for (int i =0 ; i < nr_pageblocks; i++) { +struct page *block_page = page + (pageblock_nr_pages * i); + +set_pageblock_freetype_flags (block_page,freetype_flags (freetype)); + } + +if (request_order >= alloc_order)
mm/page_alloc.c static inline void zone_statistics(struct zone *preferred_zone, struct zone *z, · line 3451
Quoted 10 lines
mm/page_alloc.c +10
-0+return page; + +/* Free any remaining pages in the block. */ +zone_lock_irqsave (zone, irq_flags); +for (unsigned int i = request_order; i < alloc_order; i++) { +struct page *page_to_free = page + (1 << i); + +__free_one_page (page_to_free,page_to_pfn (page_to_free), zone, + i, freetype,FPI_SKIP_REPORT_NOTIFY ); + }
Could expand() be used here?
Quoted 86 lines
mm/page_alloc.c +29
-9+zone_unlock_irqrestore (zone, irq_flags); + +return page; +} +#else /* CONFIG_PAGE_ALLOC_UNMAPPED */ +static inline struct page *__rmqueue_direct_map (struct zone *zone,unsigned int request_order, +unsigned int alloc_flags,freetype_t freetype) +{ +return NULL ; +} +#endif /* CONFIG_PAGE_ALLOC_UNMAPPED */ +static __always_inline struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, unsigned int order, unsigned int alloc_flags,@@ -3331,8 +3468,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, {struct page *page;unsigned long flags; -freetype_t ft_high =freetype_with_migrate (freetype, -MIGRATE_HIGHATOMIC ); +freetype_t ft_high =freetype_with_migrate (freetype,MIGRATE_HIGHATOMIC ); do { page = NULL;@@ -3357,13 +3493,15 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,*/ if (!page && (alloc_flags & (ALLOC_OOM |ALLOC_HARDER ))) page =__rmqueue_smallest (zone, order, ft_high); - -if (!page) { -zone_unlock_irqrestore (zone, flags); -return NULL ; - } }zone_unlock_irqrestore (zone, flags); + +/* Try changing direct map, now we've released the zone lock */ +if (!page) + page =__rmqueue_direct_map (zone, order, alloc_flags, freetype); +if (!page) +return NULL ; + }while (check_new_pages (page, order));__count_zid_vm_events (PGALLOC ,page_zonenum (page),1 << order);@@ -3587,6 +3725,8 @@ static void reserve_highatomic_pageblock(struct page *page, int order,static bool unreserve_highatomic_pageblock (const struct alloc_context *ac,bool force) { +freetype_t ft_high =freetype_with_migrate (ac->freetype, +MIGRATE_HIGHATOMIC );struct zonelist *zonelist = ac->zonelist;unsigned long flags;struct zoneref *z;@@ -3595,6 +3735,9 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,int order;int ret; +if (freetype_idx (ft_high) <0 ) +return false; +for_each_zone_zonelist_nodemask (zone, z, zonelist, ac->highest_zoneidx, ac->nodemask) {/* @@ -3608,8 +3751,6 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,zone_lock_irqsave (zone, flags);for (order =0 ; order <NR_PAGE_ORDERS ; order++) {struct free_area *area = &(zone->free_area[order]); -freetype_t ft_high =freetype_with_migrate (ac->freetype, -MIGRATE_HIGHATOMIC );unsigned long size; page =get_page_from_free_area (area, ft_high);@@ -5109,6 +5250,10 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, ac->nodemask = nodemask; ac->freetype =gfp_freetype (gfp_mask); +/* Not implemented yet. */ +if (freetype_flags (ac->freetype) &FREETYPE_UNMAPPED && gfp_mask & __GFP_ZERO) +return false; +if (cpusets_enabled ()) { *alloc_gfp |= __GFP_HARDWALL;/*
Re: [PATCH v2 19/22] mm/page_alloc: implement __GFP_UNMAPPED allocations
Quoted 15 lines · Vlastimil Babka (SUSE)
Quoted 14 lines · Brendan Jackman
Currently
__GFP_UNMAPPEDallocs will always fail because, although the lists exist to hold them, there is no way to actually create an unmapped page block. This commit adds one, and also the logic to map it back again when that's needed.Doing this at pageblock granularity ensures that the pageblock flags can be used to infer which freetype a page belongs to. It also provides nice batching of TLB flushes, and also avoids creating too much unnecessary TLB fragmentation in the physmap.
There are some functional requirements for flipping a block:
- Unmapping requires a TLB shootdown, meaning IRQs must be enabled.
Quoted 7 lines
Quoted 4 lines
- Because the main usecase of this feature is to protect against CPU exploits, when a block is mapped it needs to be zeroed to ensure no residual data is available to attackers. Zeroing a block with a spinlock held seems undesirable.
Did I overlook something or this patch doesn't do this whole block zeroing? Or is it handled by
set_direct_map_valid_noflushitself?
Oops. At some point I was planning to defer the zeroing to another series. I changed my mind about that but, apparently I forgot to actually add the code back.
The code I deleted was in __rmqueue_direct_map() like this:
if (want_mapped) {
<zero the block>
} else {
unsigned long start = (unsigned long)page_address(page);
unsigned long end = start + (nr_pageblocks << (pageblock_order + PAGE_SHIFT));
flush_tlb_kernel_range(start, end);
}But actually I'm not sure that's what we want: At the moment, there's actually a race condition when allocating __GFP_UNMAPPED|__GFP_ZERO:
1. Take page off freelist
2. Mermap it
3. Zero it
4. Mer-unmap it
I don't know, but some sort of CPU attack might support exploiting the gap between 2 and 3 to leak any data left behind from a prior allocation. (Like, maybe you can get the data into a uarch buffer during the race window, then leak that data afterwards at leisure).
To mitigate that, we might want to effectively enforce want_init_on_free() for unmapped blocks. And, if we do that, we don't actually need to zero the block when flipping it back to mapped, since there shouldn't be any user data in there.
Any thoughts on that? I have not tried to implement it yet, I might be missing something that makes it impractical. Also I haven't read that series that's doing zeroing through user addresses either, this might have an interesting interaction with that.
Quoted 12 lines
Quoted 11 lines
- Updating the pagetables might require allocating a pagetable to break down a huge page. This would deadlock if the zone lock was held.
This makes allocations that need to change sensitivity _somewhat_ similar to those that need to fallback to a different migratetype. But, the locking requirements mean that this can't just be squashed into the existing "fallback" allocator logic, instead a new allocator path just for this purpose is needed.
The new path is assumed to be much cheaper than the really heavyweight stuff like compaction and reclaim. But at present it is treated as less
Quoted 10 lines
Uhh, speaking of compaction and reclaim... we rely on finding a whole free pageblock in order to flip it. If that doesn't exist, the whole
get_page_from_freelist()will fail, and we might enter the reclaim/compaction cycle in__allow_pages_slowpath(). But since we might ultimately want an order-0 allocation, there won't be any compaction attempted, because that code won't know we failed to flip a pageblock. And the watermarks might look good and prevent reclaim as well I think? We should somehow indicate this, and handle accordingly. Might not be trivial. Or maybe reuse pageblock isolation code to do the migrations directly in__rmqueue_direct_map?
Ah, thanks, I suspect you are right.
I did fear there would be some sort of case where this "not-quite reclaim" interacted badly with the actual reclaim, and I tried to test it by running some stuff in parallel with stress-ng (allocating __GFP_UNMAPPED via secretmem), and I didn't see a difference in the effective availability of memory. However, I suspect testing this is quite a deep art my "run these two commands that I copy pasted from an LLM suggestion" test was just crap.
Do you have any workloads you can suggest for evaluating this kinda thing? We would definitely see it in Google prod (I think we see this kind of issue with our shrinker-based internal version of ASI distorting reclaim behaviour in ways even more subtle than this) but that is not a very practical experimental cycle...
Quoted 15 lines
mm/page_alloc.c static inline void zone_statistics(struct zone *preferred_zone, struct zone *z, · line 3361
Quoted 12 lines
mm/page_alloc.c +11
-0+#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +/* Try to allocate a page by mapping/unmapping a block from the direct map. */ +static inline struct page * +__rmqueue_direct_map (struct zone *zone,unsigned int request_order, +unsigned int alloc_flags,freetype_t freetype) +{ +unsigned int ft_flags_other =freetype_flags (freetype) ^FREETYPE_UNMAPPED ; +freetype_t ft_other =migrate_to_freetype (free_to_migratetype (freetype), + ft_flags_other); +bool want_mapped = !(freetype_flags (freetype) &FREETYPE_UNMAPPED ); +enum rmqueue_mode rmqm =RMQUEUE_NORMAL ;Why not
RMQUEUE_CLAIM? We want to change the migratetype to ours as well, not just the unmapped flag?
Oh right, actually I think we need to do RMQUEUE_CLAIM _and_ RMQUEUE_NORMAL (or, some variant of RMQUEUE_CLAIM that also supports allocating from blocks that already have the requested migratetype).
If we just switch it over to just RMQUEUE_CLAIM right now, while only one migrateteype supports FREETYPE_UNMAPPED, I think that would actually be broken: When allocating an unmapped block, (want_mapped=true) we would always hit the freetype_idx<0 case in find_suitable_fallback().
But yeah we do need to do RMQUEUE_CLAIM too otherwise we'll miss opportunities to allocate from other unmapped freetypes once those exist.
Quoted 84 lines
Quoted 84 lines
mm/page_alloc.c +84
-0+unsigned long irq_flags; +int nr_pageblocks; +struct page *page; +int alloc_order; +int err; + +if (freetype_idx (ft_other) <0 ) +return NULL ; + +/* +* Might need a TLB shootdown. Even if IRQs are on this isn't +* safe if the caller holds a lock (in case the other CPUs need that +* lock to handle the shootdown IPI). +*/ +if (alloc_flags &ALLOC_NOBLOCK ) +return NULL ; + +if (!can_set_direct_map ()) +return NULL ; + +lockdep_assert (!irqs_disabled () ||unlikely (early_boot_irqs_disabled)); + +/* +* Need to [un]map a whole pageblock (otherwise it might require +* allocating pagetables). First allocate it. +*/ + alloc_order =max (request_order, pageblock_order); + nr_pageblocks =1 << (alloc_order - pageblock_order); +zone_lock_irqsave (zone, irq_flags); + page =__rmqueue (zone, alloc_order, ft_other, alloc_flags, &rmqm); +zone_unlock_irqrestore (zone, irq_flags); +if (!page) +return NULL ; + +/* +* Now that IRQs are on it's safe to do a TLB shootdown, and now that we +* released the zone lock it's possible to allocate a pagetable if +* needed to split up a huge page. +* +* Note that modifying the direct map may need to allocate pagetables. +* What about unbounded recursion? Here are the assumptions that make it +* safe: +* +* - The direct map starts out fully mapped at boot. (This is not really +* an assumption" as its in direct control of page_alloc.c). +* +* - Once pages in the direct map are broken down, they are not +* re-aggregated into larger pages again. +* +* - Pagetables are never allocated with __GFP_UNMAPPED. +* +* Under these assumptions, a pagetable might need to be allocated while +* _unmapping_ stuff from the direct map during a __GFP_UNMAPPED +* allocation. But, the allocation of that pagetable never requires +* allocating a further pagetable. +*/ + err =set_direct_map_valid_noflush (page, + nr_pageblocks << pageblock_order, want_mapped); +if (err == -ENOMEM ||WARN_ONCE (err,"err=%d\n" , err)) { +zone_lock_irqsave (zone, irq_flags); +__free_one_page (page,page_to_pfn (page), zone, + alloc_order, freetype,FPI_SKIP_REPORT_NOTIFY ); +zone_unlock_irqrestore (zone, irq_flags); +return NULL ; + } + +if (!want_mapped) { +unsigned long start = (unsigned long )page_address (page); +unsigned long end = start + (nr_pageblocks << (pageblock_order +PAGE_SHIFT )); + +flush_tlb_kernel_range (start, end); + } + +for (int i =0 ; i < nr_pageblocks; i++) { +struct page *block_page = page + (pageblock_nr_pages * i); + +set_pageblock_freetype_flags (block_page,freetype_flags (freetype)); + } + +if (request_order >= alloc_order) +return page; + +/* Free any remaining pages in the block. */ +zone_lock_irqsave (zone, irq_flags);
Quoted 8 lines
mm/page_alloc.c static inline void zone_statistics(struct zone *preferred_zone, struct zone *z, · line 3451
Quoted 6 lines
mm/page_alloc.c +6
-0+for (unsigned int i = request_order; i < alloc_order; i++) { +struct page *page_to_free = page + (1 << i); + +__free_one_page (page_to_free,page_to_pfn (page_to_free), zone, + i, freetype,FPI_SKIP_REPORT_NOTIFY ); + }Could
expand()be used here?
Hm, good point. It should probably look like what try_to_claim_block() does...
Instead of figuring that out right now I'll just say this: if that works I'll do it, if I find a reason why it doesn't I will add a comment explaining it in the next version.
BTW my thinking is that clarity is the only important factor here, I am confident that any speedup from this would disappear in the noise of the TLB flushing etc. But, if it works then yeah I think it would actually be clearer.
Thanks very much for this review, I really appreciate it!
Brendan Jackman
reviewerRe: [PATCH v2 19/22] mm/page_alloc: implement __GFP_UNMAPPED allocations
Quoted 1 line · Brendan Jackman
On Wed May 13, 2026 at 3:43 PM UTC, Vlastimil Babka (SUSE) wrote:
[...]
Quoted 21 lines
Quoted 10 lines
Uhh, speaking of compaction and reclaim... we rely on finding a whole free pageblock in order to flip it. If that doesn't exist, the whole
get_page_from_freelist()will fail, and we might enter the reclaim/compaction cycle in__allow_pages_slowpath(). But since we might ultimately want an order-0 allocation, there won't be any compaction attempted, because that code won't know we failed to flip a pageblock. And the watermarks might look good and prevent reclaim as well I think? We should somehow indicate this, and handle accordingly. Might not be trivial. Or maybe reuse pageblock isolation code to do the migrations directly in__rmqueue_direct_map?Ah, thanks, I suspect you are right.
I did fear there would be some sort of case where this "not-quite reclaim" interacted badly with the actual reclaim, and I tried to test it by running some stuff in parallel with stress-ng (allocating
__GFP_UNMAPPEDvia secretmem), and I didn't see a difference in the effective availability of memory. However, I suspect testing this is quite a deep art my "run these two commands that I copy pasted from an LLM suggestion" test was just crap.
Quoted 5 lines
Do you have any workloads you can suggest for evaluating this kinda thing? We would definitely see it in Google prod (I think we see this kind of issue with our shrinker-based internal version of ASI distorting reclaim behaviour in ways even more subtle than this) but that is not a very practical experimental cycle...
I slop-coded a benchmark:
https://github.com/bjackman/kernel-benchmarks-nix/tree/master/packages/benchmarks/secretmem-vs-frag
It does some mmap/munmap patterns to try and generate fragmentation, then spams secretmem allocations until it gets OOM-killed.
With this series, I see the OOM-kills happening noticeably sooner on a 1GiB VM:
metric: secretmem_allocated_bytes (B) | test: secretmem-vs-frag
+---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+ | kernel_release | samples | mean | min | histogram | max | Δμ | +---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+ | 7.0.0-rc4-next-20260319 | 4 | 683,147,264 | 643,825,664 | █ | 715,128,832 | | | 7.0.0-rc4-next-20260319-00028-gf00246eb72cd | 3 | 623,553,195 | 551,550,976 | ███ | 692,060,160 | -8.7% | +---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+
So... I think maybe I've reproduced the issue you pointed out? I will try and fix it and see if this degradation goes away.
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 19/22] mm/page_alloc: implement __GFP_UNMAPPED allocations
Quoted 48 lines · Brendan Jackman
Quoted 1 line · Brendan Jackman
On Wed May 13, 2026 at 3:43 PM UTC, Vlastimil Babka (SUSE) wrote:
[...]
Quoted 21 lines
Quoted 10 lines
Uhh, speaking of compaction and reclaim... we rely on finding a whole free pageblock in order to flip it. If that doesn't exist, the whole
get_page_from_freelist()will fail, and we might enter the reclaim/compaction cycle in__allow_pages_slowpath(). But since we might ultimately want an order-0 allocation, there won't be any compaction attempted, because that code won't know we failed to flip a pageblock. And the watermarks might look good and prevent reclaim as well I think? We should somehow indicate this, and handle accordingly. Might not be trivial. Or maybe reuse pageblock isolation code to do the migrations directly in__rmqueue_direct_map?Ah, thanks, I suspect you are right.
I did fear there would be some sort of case where this "not-quite reclaim" interacted badly with the actual reclaim, and I tried to test it by running some stuff in parallel with stress-ng (allocating
__GFP_UNMAPPEDvia secretmem), and I didn't see a difference in the effective availability of memory. However, I suspect testing this is quite a deep art my "run these two commands that I copy pasted from an LLM suggestion" test was just crap.Quoted 5 lines
Do you have any workloads you can suggest for evaluating this kinda thing? We would definitely see it in Google prod (I think we see this kind of issue with our shrinker-based internal version of ASI distorting reclaim behaviour in ways even more subtle than this) but that is not a very practical experimental cycle...
I slop-coded a benchmark:
https://github.com/bjackman/kernel-benchmarks-nix/tree/master/packages/benchmarks/secretmem-vs-frag
It does some mmap/munmap patterns to try and generate fragmentation, then spams secretmem allocations until it gets OOM-killed.
With this series, I see the OOM-kills happening noticeably sooner on a 1GiB VM:
metric:
secretmem_allocated_bytes(B) | test: secretmem-vs-frag+---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+ | kernel_release | samples | mean | min | histogram | max | Δμ | +---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+ | 7.0.0-rc4-next-20260319 | 4 | 683,147,264 | 643,825,664 | █ | 715,128,832 | | | 7.0.0-rc4-next-20260319-00028-gf00246eb72cd | 3 | 623,553,195 | 551,550,976 | ███ | 692,060,160 | -8.7% | +---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+
Quoted 2 lines
So... I think maybe I've reproduced the issue you pointed out? I will try and fix it and see if this degradation goes away.
Since I assume the fragmentating allocations are movable allocations, it might be the case, yeah.
Brendan Jackman
reviewerRe: [PATCH v2 19/22] mm/page_alloc: implement __GFP_UNMAPPED allocations
Quoted 49 lines · Vlastimil Babka (SUSE)
Quoted 48 lines · Brendan Jackman
Quoted 1 line · Brendan Jackman
On Wed May 13, 2026 at 3:43 PM UTC, Vlastimil Babka (SUSE) wrote:
[...]
Quoted 21 lines
Quoted 10 lines
Uhh, speaking of compaction and reclaim... we rely on finding a whole free pageblock in order to flip it. If that doesn't exist, the whole
get_page_from_freelist()will fail, and we might enter the reclaim/compaction cycle in__allow_pages_slowpath(). But since we might ultimately want an order-0 allocation, there won't be any compaction attempted, because that code won't know we failed to flip a pageblock. And the watermarks might look good and prevent reclaim as well I think? We should somehow indicate this, and handle accordingly. Might not be trivial. Or maybe reuse pageblock isolation code to do the migrations directly in__rmqueue_direct_map?Ah, thanks, I suspect you are right.
I did fear there would be some sort of case where this "not-quite reclaim" interacted badly with the actual reclaim, and I tried to test it by running some stuff in parallel with stress-ng (allocating
__GFP_UNMAPPEDvia secretmem), and I didn't see a difference in the effective availability of memory. However, I suspect testing this is quite a deep art my "run these two commands that I copy pasted from an LLM suggestion" test was just crap.Quoted 5 lines
Do you have any workloads you can suggest for evaluating this kinda thing? We would definitely see it in Google prod (I think we see this kind of issue with our shrinker-based internal version of ASI distorting reclaim behaviour in ways even more subtle than this) but that is not a very practical experimental cycle...
I slop-coded a benchmark:
https://github.com/bjackman/kernel-benchmarks-nix/tree/master/packages/benchmarks/secretmem-vs-frag
It does some mmap/munmap patterns to try and generate fragmentation, then spams secretmem allocations until it gets OOM-killed.
With this series, I see the OOM-kills happening noticeably sooner on a 1GiB VM:
metric:
secretmem_allocated_bytes(B) | test: secretmem-vs-frag+---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+ | kernel_release | samples | mean | min | histogram | max | Δμ | +---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+ | 7.0.0-rc4-next-20260319 | 4 | 683,147,264 | 643,825,664 | █ | 715,128,832 | | | 7.0.0-rc4-next-20260319-00028-gf00246eb72cd | 3 | 623,553,195 | 551,550,976 | ███ | 692,060,160 | -8.7% | +---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+
Quoted 5 lines
Quoted 2 lines
So... I think maybe I've reproduced the issue you pointed out? I will try and fix it and see if this degradation goes away.
Since I assume the fragmentating allocations are movable allocations, it might be the case, yeah.
Alright, so I tried splitting NR_FREE_PAGES_BLOCKS into two counters to track mapped vs unmapped blocks. Then I gave compaction_suit_allocation_order() an 'unmapped' flag:
@@ -2510,19 +2510,39 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
static enum compact_result
compaction_suit_allocation_order(struct zone *zone, unsigned int order,
int highest_zoneidx, unsigned int alloc_flags,
- bool async, bool kcompactd)
+ bool unmapped, bool async, bool kcompactd)
{
unsigned long free_pages;
unsigned long watermark;- if (kcompactd && defrag_mode)
+ /*
+ * Might need to generate a whole free block regardless of the actual
+ * allocation order:
+ *
+ * - When allocating an unmapped page, because the allocator only unmaps
+ * whole blocks at a time.
+ *
+ * Why doesn't this apply to the other way around too? (Mightn't we
+ * need to _map_ a whole block?) This is a temporary simplification:
+ * currently, unmapped blocks don't contain movable pages, so
+ * compaction isn't going to free up one of those.
+ *
+ * - In defrag_mode, because the allocator is unwilling to "steal" pages
+ * from the "wrong" block.
+ *
+ * Why is this only under kcompactd?
+ *
+ * Temporary simplification: unmapped pageblocks are currently
+ * nonmovable. So if the compactor is trying to service a
+ */
+ if (unmapped)
+ free_pages = zone_page_state(zone, NR_FREE_PAGES_BLOCKS_MAPPED);
+ else if (kcompactd && defrag_mode)
free_pages = zone_free_pages_blocks(zone);
else
free_pages = zone_page_state(zone, NR_FREE_PAGES);... Then, I changed __alloc_pages_direct_compact() to try to try to compact for a whole block whenever we are trying to allocate an unmapped page (note I think there's an orthogonal bug here where it leaks memory when there's a "captured" compaction):
index 4f04e897c5374..7eed22f3b26eb 100644
mm/page_alloc.c +17 -3
@@ -824,6 +824,9 @@ compaction_capture(struct capture_control *capc, struct page *page, capc_mt != MIGRATE_MOVABLE)return false; +if (freetype_flags (freetype) !=freetype_flags (capc->cc->freetype)) +return false; +if (migratetype != capc_mt)trace_mm_page_alloc_extfrag (page, capc->cc->order, order, capc_mt, migratetype);
@@ -4469,20 +4472,27 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,struct page *page =NULL ;unsigned long pflags;unsigned int noreclaim_flag; +unsigned int compact_order = order; -if (!order) +// TODO: Is it OK to always run compaction like this? +/* +* Unmapped allocations benefit from compaction even at order 0, because the +* allocator will actually grab a whole block. +*/ +if (freetype_flags (ac->freetype) &FREETYPE_UNMAPPED ) + compact_order = pageblock_order; + +if (!compact_order)return NULL ;psi_memstall_enter (&pflags);delayacct_compact_start (); noreclaim_flag =memalloc_noreclaim_save (); - *compact_result =try_to_compact_pages (gfp_mask, order, alloc_flags, ac, - prio, &page); +// TODO: deal with captured page, if we changed the order it will have the +// wrong order. Also check it respects the freetype flags. + *compact_result =try_to_compact_pages (gfp_mask, compact_order, + alloc_flags, ac, prio, &page);memalloc_noreclaim_restore (noreclaim_flag);psi_memstall_leave (&pflags);
Full code:
https://github.com/bjackman/linux/tree/page_alloc-unmapped-2026-06-11
This makes the regression above (faster OOMs) go away, but it seems like a pretty blunt approach. But then I'm realising I don't really know why it matters? The main thing is presumably that we are more likely to pointlessly attempt compaction or compact more than we need. But in that case, aren't we already in a desperately slow path? Does a little bit of extra work in __alloc_pages_direct_compact() really matter? I couldn't measure it in a benchmark (kernel compilation alongside stress-ng --secretmem).
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 19/22] mm/page_alloc: implement __GFP_UNMAPPED allocations
Quoted 151 lines · Brendan Jackman
Quoted 49 lines · Vlastimil Babka (SUSE)
Quoted 48 lines · Brendan Jackman
Quoted 1 line · Brendan Jackman
On Wed May 13, 2026 at 3:43 PM UTC, Vlastimil Babka (SUSE) wrote:
[...]
Quoted 21 lines
Quoted 10 lines
Uhh, speaking of compaction and reclaim... we rely on finding a whole free pageblock in order to flip it. If that doesn't exist, the whole
get_page_from_freelist()will fail, and we might enter the reclaim/compaction cycle in__allow_pages_slowpath(). But since we might ultimately want an order-0 allocation, there won't be any compaction attempted, because that code won't know we failed to flip a pageblock. And the watermarks might look good and prevent reclaim as well I think? We should somehow indicate this, and handle accordingly. Might not be trivial. Or maybe reuse pageblock isolation code to do the migrations directly in__rmqueue_direct_map?Ah, thanks, I suspect you are right.
I did fear there would be some sort of case where this "not-quite reclaim" interacted badly with the actual reclaim, and I tried to test it by running some stuff in parallel with stress-ng (allocating
__GFP_UNMAPPEDvia secretmem), and I didn't see a difference in the effective availability of memory. However, I suspect testing this is quite a deep art my "run these two commands that I copy pasted from an LLM suggestion" test was just crap.Quoted 5 lines
Do you have any workloads you can suggest for evaluating this kinda thing? We would definitely see it in Google prod (I think we see this kind of issue with our shrinker-based internal version of ASI distorting reclaim behaviour in ways even more subtle than this) but that is not a very practical experimental cycle...
I slop-coded a benchmark:
https://github.com/bjackman/kernel-benchmarks-nix/tree/master/packages/benchmarks/secretmem-vs-frag
It does some mmap/munmap patterns to try and generate fragmentation, then spams secretmem allocations until it gets OOM-killed.
With this series, I see the OOM-kills happening noticeably sooner on a 1GiB VM:
metric:
secretmem_allocated_bytes(B) | test: secretmem-vs-frag+---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+ | kernel_release | samples | mean | min | histogram | max | Δμ | +---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+ | 7.0.0-rc4-next-20260319 | 4 | 683,147,264 | 643,825,664 | █ | 715,128,832 | | | 7.0.0-rc4-next-20260319-00028-gf00246eb72cd | 3 | 623,553,195 | 551,550,976 | ███ | 692,060,160 | -8.7% | +---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+Quoted 5 lines
Quoted 2 lines
So... I think maybe I've reproduced the issue you pointed out? I will try and fix it and see if this degradation goes away.
Since I assume the fragmentating allocations are movable allocations, it might be the case, yeah.
Alright, so I tried splitting
NR_FREE_PAGES_BLOCKSinto two counters to track mapped vs unmapped blocks. Then I gavecompaction_suit_allocation_order()an 'unmapped' flag:(quoted diff) +24
-2@@ -2510,19 +2510,39 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, static enum compact_result compaction_suit_allocation_order(struct zone *zone, unsigned int order, int highest_zoneidx, unsigned int alloc_flags, - bool async, bool kcompactd) + bool unmapped, bool async, bool kcompactd) { unsigned long free_pages; unsigned long watermark; - if (kcompactd && defrag_mode) + /* + * Might need to generate a whole free block regardless of the actual + * allocation order: + * + * - When allocating an unmapped page, because the allocator only unmaps + * whole blocks at a time. + * + * Why doesn't this apply to the other way around too? (Mightn't we + * need to _map_ a whole block?) This is a temporary simplification: + * currently, unmapped blocks don't contain movable pages, so + * compaction isn't going to free up one of those. + * + * - In defrag_mode, because the allocator is unwilling to "steal" pages + * from the "wrong" block. + * + * Why is this only under kcompactd? + * + * Temporary simplification: unmapped pageblocks are currently + * nonmovable. So if the compactor is trying to service a + */ + if (unmapped) + free_pages = zone_page_state(zone, NR_FREE_PAGES_BLOCKS_MAPPED); + else if (kcompactd && defrag_mode) free_pages = zone_free_pages_blocks(zone); else free_pages = zone_page_state(zone, NR_FREE_PAGES);... Then, I changed
__alloc_pages_direct_compact()to try to try to compact for a whole block whenever we are trying to allocate an unmapped page (note I think there's an orthogonal bug here where it leaks memory when there's a "captured" compaction):index
4f04e897c5374..7eed22f3b26eb100644mm/page_alloc.c +17
-3@@ -824,6 +824,9 @@ compaction_capture(struct capture_control *capc, struct page *page, capc_mt != MIGRATE_MOVABLE)return false; +if (freetype_flags (freetype) !=freetype_flags (capc->cc->freetype)) +return false; +if (migratetype != capc_mt)trace_mm_page_alloc_extfrag (page, capc->cc->order, order, capc_mt, migratetype);@@ -4469,20 +4472,27 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,struct page *page =NULL ;unsigned long pflags;unsigned int noreclaim_flag; +unsigned int compact_order = order; -if (!order) +// TODO: Is it OK to always run compaction like this? +/* +* Unmapped allocations benefit from compaction even at order 0, because the +* allocator will actually grab a whole block. +*/ +if (freetype_flags (ac->freetype) &FREETYPE_UNMAPPED ) + compact_order = pageblock_order; + +if (!compact_order)return NULL ;psi_memstall_enter (&pflags);delayacct_compact_start (); noreclaim_flag =memalloc_noreclaim_save (); - *compact_result =try_to_compact_pages (gfp_mask, order, alloc_flags, ac, - prio, &page); +// TODO: deal with captured page, if we changed the order it will have the +// wrong order. Also check it respects the freetype flags. + *compact_result =try_to_compact_pages (gfp_mask, compact_order, + alloc_flags, ac, prio, &page);memalloc_noreclaim_restore (noreclaim_flag);psi_memstall_leave (&pflags);
Quoted 6 lines
Full code:
https://github.com/bjackman/linux/tree/page_alloc-unmapped-2026-06-11This makes the regression above (faster OOMs) go away, but it seems like a pretty blunt approach. But then I'm realising I don't really know why it matters?
You mean, why does it matter that we don't OOM prematurely? I'd say that matters a lot.
Quoted 2 lines
The main thing is presumably that we are more likely to pointlessly attempt compaction or compact more than we need. But in that
I don't understand why that would be the case? If compaction thinks our goal is order-0, there won't be any?
Or you mean that it doesn't matter that your approach above is blunt, and are talking about the consequences of that blunt approach?
Quoted 4 lines
case, aren't we already in a desperately slow path? Does a little bit of extra work in
__alloc_pages_direct_compact()really matter? I couldn't measure it in a benchmark (kernel compilation alongside stress-ng --secretmem).
Brendan Jackman
reviewerRe: [PATCH v2 19/22] mm/page_alloc: implement __GFP_UNMAPPED allocations
Quoted 162 lines · Vlastimil Babka (SUSE)
Quoted 151 lines · Brendan Jackman
Quoted 49 lines · Vlastimil Babka (SUSE)
Quoted 48 lines · Brendan Jackman
Quoted 1 line · Brendan Jackman
On Wed May 13, 2026 at 3:43 PM UTC, Vlastimil Babka (SUSE) wrote:
[...]
Quoted 21 lines
Quoted 10 lines
Uhh, speaking of compaction and reclaim... we rely on finding a whole free pageblock in order to flip it. If that doesn't exist, the whole
get_page_from_freelist()will fail, and we might enter the reclaim/compaction cycle in__allow_pages_slowpath(). But since we might ultimately want an order-0 allocation, there won't be any compaction attempted, because that code won't know we failed to flip a pageblock. And the watermarks might look good and prevent reclaim as well I think? We should somehow indicate this, and handle accordingly. Might not be trivial. Or maybe reuse pageblock isolation code to do the migrations directly in__rmqueue_direct_map?Ah, thanks, I suspect you are right.
I did fear there would be some sort of case where this "not-quite reclaim" interacted badly with the actual reclaim, and I tried to test it by running some stuff in parallel with stress-ng (allocating
__GFP_UNMAPPEDvia secretmem), and I didn't see a difference in the effective availability of memory. However, I suspect testing this is quite a deep art my "run these two commands that I copy pasted from an LLM suggestion" test was just crap.Quoted 5 lines
Do you have any workloads you can suggest for evaluating this kinda thing? We would definitely see it in Google prod (I think we see this kind of issue with our shrinker-based internal version of ASI distorting reclaim behaviour in ways even more subtle than this) but that is not a very practical experimental cycle...
I slop-coded a benchmark:
https://github.com/bjackman/kernel-benchmarks-nix/tree/master/packages/benchmarks/secretmem-vs-frag
It does some mmap/munmap patterns to try and generate fragmentation, then spams secretmem allocations until it gets OOM-killed.
With this series, I see the OOM-kills happening noticeably sooner on a 1GiB VM:
metric:
secretmem_allocated_bytes(B) | test: secretmem-vs-frag+---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+ | kernel_release | samples | mean | min | histogram | max | Δμ | +---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+ | 7.0.0-rc4-next-20260319 | 4 | 683,147,264 | 643,825,664 | █ | 715,128,832 | | | 7.0.0-rc4-next-20260319-00028-gf00246eb72cd | 3 | 623,553,195 | 551,550,976 | ███ | 692,060,160 | -8.7% | +---------------------------------------------+---------+-------------+-------------+-----------------+-------------+-------+Quoted 5 lines
Quoted 2 lines
So... I think maybe I've reproduced the issue you pointed out? I will try and fix it and see if this degradation goes away.
Since I assume the fragmentating allocations are movable allocations, it might be the case, yeah.
Alright, so I tried splitting
NR_FREE_PAGES_BLOCKSinto two counters to track mapped vs unmapped blocks. Then I gavecompaction_suit_allocation_order()an 'unmapped' flag:(quoted diff) +24
-2@@ -2510,19 +2510,39 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, static enum compact_result compaction_suit_allocation_order(struct zone *zone, unsigned int order, int highest_zoneidx, unsigned int alloc_flags, - bool async, bool kcompactd) + bool unmapped, bool async, bool kcompactd) { unsigned long free_pages; unsigned long watermark; - if (kcompactd && defrag_mode) + /* + * Might need to generate a whole free block regardless of the actual + * allocation order: + * + * - When allocating an unmapped page, because the allocator only unmaps + * whole blocks at a time. + * + * Why doesn't this apply to the other way around too? (Mightn't we + * need to _map_ a whole block?) This is a temporary simplification: + * currently, unmapped blocks don't contain movable pages, so + * compaction isn't going to free up one of those. + * + * - In defrag_mode, because the allocator is unwilling to "steal" pages + * from the "wrong" block. + * + * Why is this only under kcompactd? + * + * Temporary simplification: unmapped pageblocks are currently + * nonmovable. So if the compactor is trying to service a + */ + if (unmapped) + free_pages = zone_page_state(zone, NR_FREE_PAGES_BLOCKS_MAPPED); + else if (kcompactd && defrag_mode) free_pages = zone_free_pages_blocks(zone); else free_pages = zone_page_state(zone, NR_FREE_PAGES);... Then, I changed
__alloc_pages_direct_compact()to try to try to compact for a whole block whenever we are trying to allocate an unmapped page (note I think there's an orthogonal bug here where it leaks memory when there's a "captured" compaction):index
4f04e897c5374..7eed22f3b26eb100644mm/page_alloc.c +17
-3@@ -824,6 +824,9 @@ compaction_capture(struct capture_control *capc, struct page *page, capc_mt != MIGRATE_MOVABLE)return false; +if (freetype_flags (freetype) !=freetype_flags (capc->cc->freetype)) +return false; +if (migratetype != capc_mt)trace_mm_page_alloc_extfrag (page, capc->cc->order, order, capc_mt, migratetype);@@ -4469,20 +4472,27 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,struct page *page =NULL ;unsigned long pflags;unsigned int noreclaim_flag; +unsigned int compact_order = order; -if (!order) +// TODO: Is it OK to always run compaction like this? +/* +* Unmapped allocations benefit from compaction even at order 0, because the +* allocator will actually grab a whole block. +*/ +if (freetype_flags (ac->freetype) &FREETYPE_UNMAPPED ) + compact_order = pageblock_order; + +if (!compact_order)return NULL ;psi_memstall_enter (&pflags);delayacct_compact_start (); noreclaim_flag =memalloc_noreclaim_save (); - *compact_result =try_to_compact_pages (gfp_mask, order, alloc_flags, ac, - prio, &page); +// TODO: deal with captured page, if we changed the order it will have the +// wrong order. Also check it respects the freetype flags. + *compact_result =try_to_compact_pages (gfp_mask, compact_order, + alloc_flags, ac, prio, &page);memalloc_noreclaim_restore (noreclaim_flag);psi_memstall_leave (&pflags);Quoted 6 lines
Full code:
https://github.com/bjackman/linux/tree/page_alloc-unmapped-2026-06-11This makes the regression above (faster OOMs) go away, but it seems like a pretty blunt approach. But then I'm realising I don't really know why it matters?
You mean, why does it matter that we don't OOM prematurely? I'd say that matters a lot.
Quoted 8 lines
Quoted 2 lines
The main thing is presumably that we are more likely to pointlessly attempt compaction or compact more than we need. But in that
I don't understand why that would be the case? If compaction thinks our goal is order-0, there won't be any?
Or you mean that it doesn't matter that your approach above is blunt, and are talking about the consequences of that blunt approach?
Yeah exactly this - it seems to fix the issue, but I dunno if it's cheating. It _feels_ like cheating but I actually don't know what it would break - maybe nothing?
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 19/22] mm/page_alloc: implement __GFP_UNMAPPED allocations
Quoted 18 lines · Brendan Jackman
Quoted 17 lines · Vlastimil Babka (SUSE)
Quoted 7 lines
Full code:
https://github.com/bjackman/linux/tree/page_alloc-unmapped-2026-06-11This makes the regression above (faster OOMs) go away, but it seems like a pretty blunt approach. But then I'm realising I don't really know why it matters?
You mean, why does it matter that we don't OOM prematurely? I'd say that matters a lot.
Quoted 2 lines
The main thing is presumably that we are more likely to pointlessly attempt compaction or compact more than we need. But in that
I don't understand why that would be the case? If compaction thinks our goal is order-0, there won't be any?
Quoted 6 lines
Quoted 2 lines
Or you mean that it doesn't matter that your approach above is blunt, and are talking about the consequences of that blunt approach?
Yeah exactly this - it seems to fix the issue, but I dunno if it's cheating. It _feels_ like cheating but I actually don't know what it would break - maybe nothing?
Ah ok, now I understand. Will have to check more closely if it can break something. Maybe Johannes has some insight as he mentioned working on a similar thing for the defrag mode?
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 19/22] mm/page_alloc: implement __GFP_UNMAPPED allocations
In reply to Brendan JackmanQuoted 47 lines · Brendan Jackman
Quoted 15 lines · Vlastimil Babka (SUSE)
Quoted 14 lines · Brendan Jackman
Currently
__GFP_UNMAPPEDallocs will always fail because, although the lists exist to hold them, there is no way to actually create an unmapped page block. This commit adds one, and also the logic to map it back again when that's needed.Doing this at pageblock granularity ensures that the pageblock flags can be used to infer which freetype a page belongs to. It also provides nice batching of TLB flushes, and also avoids creating too much unnecessary TLB fragmentation in the physmap.
There are some functional requirements for flipping a block:
- Unmapping requires a TLB shootdown, meaning IRQs must be enabled.
Quoted 7 lines
Quoted 4 lines
- Because the main usecase of this feature is to protect against CPU exploits, when a block is mapped it needs to be zeroed to ensure no residual data is available to attackers. Zeroing a block with a spinlock held seems undesirable.
Did I overlook something or this patch doesn't do this whole block zeroing? Or is it handled by
set_direct_map_valid_noflushitself?Oops. At some point I was planning to defer the zeroing to another series. I changed my mind about that but, apparently I forgot to actually add the code back.
The code I deleted was in
__rmqueue_direct_map()like this:if (want_mapped) { <zero the block> } else { unsigned long start = (unsigned long)page_address(page); unsigned long end = start + (nr_pageblocks << (pageblock_order + PAGE_SHIFT)); flush_tlb_kernel_range(start, end); }But actually I'm not sure that's what we want: At the moment, there's actually a race condition when allocating __GFP_UNMAPPED|__GFP_ZERO:
1. Take page off freelist
2. Mermap it
3. Zero it
4. Mer-unmap it
Quoted 4 lines
I don't know, but some sort of CPU attack might support exploiting the gap between 2 and 3 to leak any data left behind from a prior allocation. (Like, maybe you can get the data into a uarch buffer during the race window, then leak that data afterwards at leisure).
I can't imagine how it would work, but then novel CPU attacks might be beyond my imagination :) But I think we can ignore hypothetical CPU attacks for now?
Quoted 9 lines
To mitigate that, we might want to effectively enforce
want_init_on_free()for unmapped blocks. And, if we do that, we don't actually need to zero the block when flipping it back to mapped, since there shouldn't be any user data in there.Any thoughts on that? I have not tried to implement it yet, I might be missing something that makes it impractical. Also I haven't read that series that's doing zeroing through user addresses either, this might have an interesting interaction with that.
I think let's try the simplest way first.
Quoted 33 lines
Quoted 12 lines
Quoted 11 lines
- Updating the pagetables might require allocating a pagetable to break down a huge page. This would deadlock if the zone lock was held.
This makes allocations that need to change sensitivity _somewhat_ similar to those that need to fallback to a different migratetype. But, the locking requirements mean that this can't just be squashed into the existing "fallback" allocator logic, instead a new allocator path just for this purpose is needed.
The new path is assumed to be much cheaper than the really heavyweight stuff like compaction and reclaim. But at present it is treated as less
Quoted 10 lines
Uhh, speaking of compaction and reclaim... we rely on finding a whole free pageblock in order to flip it. If that doesn't exist, the whole
get_page_from_freelist()will fail, and we might enter the reclaim/compaction cycle in__allow_pages_slowpath(). But since we might ultimately want an order-0 allocation, there won't be any compaction attempted, because that code won't know we failed to flip a pageblock. And the watermarks might look good and prevent reclaim as well I think? We should somehow indicate this, and handle accordingly. Might not be trivial. Or maybe reuse pageblock isolation code to do the migrations directly in__rmqueue_direct_map?Ah, thanks, I suspect you are right.
I did fear there would be some sort of case where this "not-quite reclaim" interacted badly with the actual reclaim, and I tried to test it by running some stuff in parallel with stress-ng (allocating
__GFP_UNMAPPEDvia secretmem), and I didn't see a difference in the effective availability of memory. However, I suspect testing this is quite a deep art my "run these two commands that I copy pasted from an LLM suggestion" test was just crap.
Quoted 5 lines
Do you have any workloads you can suggest for evaluating this kinda thing? We would definitely see it in Google prod (I think we see this kind of issue with our shrinker-based internal version of ASI distorting reclaim behaviour in ways even more subtle than this) but that is not a very practical experimental cycle...
Your test seems a good way to start.
I realized afterwards that the solution might be something similar to how we handle ALLOC_NOFRAGMENT.
Quoted 16 lines
Quoted 15 lines
mm/page_alloc.c static inline void zone_statistics(struct zone *preferred_zone, struct zone *z, · line 3361
Quoted 12 lines
mm/page_alloc.c +11
-0+#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +/* Try to allocate a page by mapping/unmapping a block from the direct map. */ +static inline struct page * +__rmqueue_direct_map (struct zone *zone,unsigned int request_order, +unsigned int alloc_flags,freetype_t freetype) +{ +unsigned int ft_flags_other =freetype_flags (freetype) ^FREETYPE_UNMAPPED ; +freetype_t ft_other =migrate_to_freetype (free_to_migratetype (freetype), + ft_flags_other); +bool want_mapped = !(freetype_flags (freetype) &FREETYPE_UNMAPPED ); +enum rmqueue_mode rmqm =RMQUEUE_NORMAL ;Why not
RMQUEUE_CLAIM? We want to change the migratetype to ours as well, not just the unmapped flag?
Quoted 8 lines
Oh right, actually I think we need to do
RMQUEUE_CLAIM_and_RMQUEUE_NORMAL(or, some variant ofRMQUEUE_CLAIMthat also supports allocating from blocks that already have the requested migratetype).If we just switch it over to just
RMQUEUE_CLAIMright now, while only one migrateteype supportsFREETYPE_UNMAPPED, I think that would actually be broken: When allocating an unmapped block, (want_mapped=true) we would always hit the freetype_idx<0 case infind_suitable_fallback().
Right.
Quoted 97 lines
But yeah we do need to do
RMQUEUE_CLAIMtoo otherwise we'll miss opportunities to allocate from other unmapped freetypes once those exist.Quoted 84 lines
Quoted 84 lines
mm/page_alloc.c +84
-0+unsigned long irq_flags; +int nr_pageblocks; +struct page *page; +int alloc_order; +int err; + +if (freetype_idx (ft_other) <0 ) +return NULL ; + +/* +* Might need a TLB shootdown. Even if IRQs are on this isn't +* safe if the caller holds a lock (in case the other CPUs need that +* lock to handle the shootdown IPI). +*/ +if (alloc_flags &ALLOC_NOBLOCK ) +return NULL ; + +if (!can_set_direct_map ()) +return NULL ; + +lockdep_assert (!irqs_disabled () ||unlikely (early_boot_irqs_disabled)); + +/* +* Need to [un]map a whole pageblock (otherwise it might require +* allocating pagetables). First allocate it. +*/ + alloc_order =max (request_order, pageblock_order); + nr_pageblocks =1 << (alloc_order - pageblock_order); +zone_lock_irqsave (zone, irq_flags); + page =__rmqueue (zone, alloc_order, ft_other, alloc_flags, &rmqm); +zone_unlock_irqrestore (zone, irq_flags); +if (!page) +return NULL ; + +/* +* Now that IRQs are on it's safe to do a TLB shootdown, and now that we +* released the zone lock it's possible to allocate a pagetable if +* needed to split up a huge page. +* +* Note that modifying the direct map may need to allocate pagetables. +* What about unbounded recursion? Here are the assumptions that make it +* safe: +* +* - The direct map starts out fully mapped at boot. (This is not really +* an assumption" as its in direct control of page_alloc.c). +* +* - Once pages in the direct map are broken down, they are not +* re-aggregated into larger pages again. +* +* - Pagetables are never allocated with __GFP_UNMAPPED. +* +* Under these assumptions, a pagetable might need to be allocated while +* _unmapping_ stuff from the direct map during a __GFP_UNMAPPED +* allocation. But, the allocation of that pagetable never requires +* allocating a further pagetable. +*/ + err =set_direct_map_valid_noflush (page, + nr_pageblocks << pageblock_order, want_mapped); +if (err == -ENOMEM ||WARN_ONCE (err,"err=%d\n" , err)) { +zone_lock_irqsave (zone, irq_flags); +__free_one_page (page,page_to_pfn (page), zone, + alloc_order, freetype,FPI_SKIP_REPORT_NOTIFY ); +zone_unlock_irqrestore (zone, irq_flags); +return NULL ; + } + +if (!want_mapped) { +unsigned long start = (unsigned long )page_address (page); +unsigned long end = start + (nr_pageblocks << (pageblock_order +PAGE_SHIFT )); + +flush_tlb_kernel_range (start, end); + } + +for (int i =0 ; i < nr_pageblocks; i++) { +struct page *block_page = page + (pageblock_nr_pages * i); + +set_pageblock_freetype_flags (block_page,freetype_flags (freetype)); + } + +if (request_order >= alloc_order) +return page; + +/* Free any remaining pages in the block. */ +zone_lock_irqsave (zone, irq_flags);Quoted 8 lines
mm/page_alloc.c static inline void zone_statistics(struct zone *preferred_zone, struct zone *z, · line 3451
Quoted 6 lines
mm/page_alloc.c +6
-0+for (unsigned int i = request_order; i < alloc_order; i++) { +struct page *page_to_free = page + (1 << i); + +__free_one_page (page_to_free,page_to_pfn (page_to_free), zone, + i, freetype,FPI_SKIP_REPORT_NOTIFY ); + }Could
expand()be used here?
Quoted 6 lines
Hm, good point. It should probably look like what
try_to_claim_block()does...Instead of figuring that out right now I'll just say this: if that works I'll do it, if I find a reason why it doesn't I will add a comment explaining it in the next version.
Sounds good.
Quoted 4 lines
BTW my thinking is that clarity is the only important factor here, I am confident that any speedup from this would disappear in the noise of the TLB flushing etc. But, if it works then yeah I think it would actually be clearer.
Sure clarity is important, but also if we have multiple functions doing similar thing instead of sharing code, there's a risk a future change to the more common code will miss the new one, etc.
Quoted 1 line
Thanks very much for this review, I really appreciate it!
PATCH v2 20/22 mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations
In reply to Brendan JackmanThe pages being zeroed here are unmapped, so they can't be zeroed via the direct map. Temporarily mapping them in the direct map is not possible because:
- In general this requires allocating pagetables,
- Unmapping them would require a TLB shootdown, which can't be done in general from the allocator (x86 requires IRQs on).
Therefore, use the new mermap mechanism to zero these pages.
The main mermap API is expected to fail very often. In order to avoid needing to fail allocations when that happens, instead fallback to the special mermap_get_reserved() variant, which is less efficient.
--- arch/x86/include/asm/pgtable_types.h | 2 + mm/Kconfig | 11 +++++- mm/page_alloc.c | 76 +++++++++++++++++++++++++++++++----- 3 files changed, 78 insertions(+), 11 deletions(-)
arch/x86/include/asm/pgtable_types.h +2 -0
@@ -223,6 +223,7 @@ enum page_cache_mode {#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G)#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G)#define __PAGE_KERNEL (__PP|__RW| 0|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_NOGLOBAL (__PP|__RW| 0|___A|__NX|___D| 0| 0)#define __PAGE_KERNEL_EXEC (__PP|__RW| 0|___A| 0|___D| 0|___G)#define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC)#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G)
@@ -245,6 +246,7 @@ enum page_cache_mode {#define __pgprot_mask (x) __pgprot((x) & __default_kernel_pte_mask)#define PAGE_KERNEL __pgprot_mask(__PAGE_KERNEL | _ENC) +#define PAGE_KERNEL_NOGLOBAL __pgprot_mask(__PAGE_KERNEL_NOGLOBAL | _ENC)#define PAGE_KERNEL_NOENC __pgprot_mask(__PAGE_KERNEL | 0)#define PAGE_KERNEL_RO __pgprot_mask(__PAGE_KERNEL_RO | _ENC)#define PAGE_KERNEL_EXEC __pgprot_mask(__PAGE_KERNEL_EXEC | _ENC)
mm/Kconfig +9 -2
@@ -1506,7 +1506,14 @@ config MERMAP_KUNIT_TEST If unsure, say N. config PAGE_ALLOC_UNMAPPED - bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST - default COMPILE_TEST + bool "Support allocating pages that aren't in the direct map" + depends on MERMAP + +config PAGE_ALLOC_KUNIT_TESTS + tristate "KUnit tests for the page allocator" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Builds KUnit tests for the page allocator. endmenu
mm/page_alloc.c +67 -9
96 lines not on this page · raw message
Signature · 1 line
2.51.2
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v2 20/22] mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations
Quoted 35 lines · Brendan Jackman
The pages being zeroed here are unmapped, so they can't be zeroed via the direct map. Temporarily mapping them in the direct map is not possible because:
- In general this requires allocating pagetables,
- Unmapping them would require a TLB shootdown, which can't be done in general from the allocator (x86 requires IRQs on).
Therefore, use the new mermap mechanism to zero these pages.
The main mermap API is expected to fail very often. In order to avoid needing to fail allocations when that happens, instead fallback to the special
mermap_get_reserved()variant, which is less efficient.Signed-off-by Brendan Jackman <[email protected]>--- arch/x86/include/asm/pgtable_types.h | 2 + mm/Kconfig | 11 +++++- mm/page_alloc.c | 76 +++++++++++++++++++++++++++++++----- 3 files changed, 78 insertions(+), 11 deletions(-)arch/x86/include/asm/pgtable_types.h +1
-0@@ -223,6 +223,7 @@ enum page_cache_mode {#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G)#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G)#define __PAGE_KERNEL (__PP|__RW| 0|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_NOGLOBAL (__PP|__RW| 0|___A|__NX|___D| 0| 0)#define __PAGE_KERNEL_EXEC (__PP|__RW| 0|___A| 0|___D| 0|___G)#define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC)#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G)
arch/x86/include/asm/pgtable_types.h enum page_cache_mode { · line 252
Quoted 7 lines
arch/x86/include/asm/pgtable_types.h +1
-0@@ -245,6 +246,7 @@ enum page_cache_mode {#define __pgprot_mask (x) __pgprot((x) & __default_kernel_pte_mask)#define PAGE_KERNEL __pgprot_mask(__PAGE_KERNEL | _ENC) +#define PAGE_KERNEL_NOGLOBAL __pgprot_mask(__PAGE_KERNEL_NOGLOBAL | _ENC)#define PAGE_KERNEL_NOENC __pgprot_mask(__PAGE_KERNEL | 0)#define PAGE_KERNEL_RO __pgprot_mask(__PAGE_KERNEL_RO | _ENC)#define PAGE_KERNEL_EXEC __pgprot_mask(__PAGE_KERNEL_EXEC | _ENC)
Should this be part of earlier mermap x86 patches?
Quoted 9 lines
mm/Kconfig +0
-1@@ -1506,7 +1506,14 @@ config MERMAP_KUNIT_TEST If unsure, say N. config PAGE_ALLOC_UNMAPPED - bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST
mm/Kconfig config MERMAP_KUNIT_TEST · line 1517
Quoted 10 lines
mm/Kconfig +9
-1- default COMPILE_TEST + bool "Support allocating pages that aren't in the direct map" + depends on MERMAP + +config PAGE_ALLOC_KUNIT_TESTS + tristate "KUnit tests for the page allocator" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Builds KUnit tests for the page allocator.
This belongs to the next patch?
Quoted 105 lines
mm/Kconfig +0
-0endmenumm/page_alloc.c +67
-9@@ -14,6 +14,7 @@* (lots of bits borrowed from Ingo Molnar & Andrew Morton) */ +#include <linux/mermap.h> #include <linux/stddef.h> #include <linux/mm.h> #include <linux/highmem.h> @@ -1327,15 +1328,72 @@ static inline bool should_skip_kasan_poison(struct page *page)return page_kasan_tag (page) ==KASAN_TAG_KERNEL ; } -static void kernel_init_pages (struct page *page,int numpages) +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +static inline bool pageblock_unmapped (struct page *page) { -int i; +return freetype_flags (get_pageblock_freetype (page)) &FREETYPE_UNMAPPED ; +} -/* s390's use of memset() could override KASAN redzones. */ -kasan_disable_current (); -for (i =0 ; i < numpages; i++) -clear_highpage_kasan_tagged (page + i); -kasan_enable_current (); +static inline void clear_page_mermap (struct page *page,unsigned int numpages) +{ +void *mermap; + +BUILD_BUG_ON (IS_ENABLED (CONFIG_HIGHMEM )); + +/* Fast path: single mapping (may fail under preemption). */ + mermap =mermap_get (page, numpages <<PAGE_SHIFT ,PAGE_KERNEL_NOGLOBAL ); +if (mermap) { +void *buf =kasan_reset_tag (mermap_addr (mermap)); + +for (int i =0 ; i < numpages; i++) +clear_page (buf + (i <<PAGE_SHIFT )); +mermap_put (mermap); +return ; + } + +/* Slow path, map each page individually (always succeeds). */ +for (int i =0 ; i < numpages; i++) { +unsigned long flags; + +local_irq_save (flags); + mermap =mermap_get_reserved (page + i,PAGE_KERNEL_NOGLOBAL ); +clear_page (kasan_reset_tag (mermap_addr (mermap))); +mermap_put (mermap); +local_irq_restore (flags); + } +} +#else +static inline bool pageblock_unmapped (struct page *page) +{ +return false; +} + +static inline void clear_page_mermap (struct page *page,unsigned int numpages) +{ +BUG (); +} +#endif + +static void kernel_init_pages (struct page *page,unsigned int numpages) +{ +int num_blocks =DIV_ROUND_UP (numpages, pageblock_nr_pages); + +for (int block =0 ; block < num_blocks; block++) { +struct page *block_page = page + (block << pageblock_order); +bool unmapped =pageblock_unmapped (block_page); + +/* s390's use of memset() could override KASAN redzones. */ +kasan_disable_current (); +if (unmapped) { +clear_page_mermap (block_page, numpages); + }else { +for (int i =0 ; i <min (numpages, pageblock_nr_pages); i++) +clear_highpage_kasan_tagged (block_page + i); + } +kasan_enable_current (); + + numpages -= pageblock_nr_pages; + } }#ifdef CONFIG_MEM_ALLOC_PROFILING @@ -5250,8 +5308,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, ac->nodemask = nodemask; ac->freetype =gfp_freetype (gfp_mask); -/* Not implemented yet. */ -if (freetype_flags (ac->freetype) &FREETYPE_UNMAPPED && gfp_mask & __GFP_ZERO) +if (freetype_flags (ac->freetype) &FREETYPE_UNMAPPED && +WARN_ON (!mermap_ready ()))return false;if (cpusets_enabled ()) {
Re: [PATCH v2 20/22] mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations
Quoted 7 lines · Vlastimil Babka (SUSE)
Quoted 5 lines
#define
PAGE_KERNEL__pgprot_mask(__PAGE_KERNEL | _ENC)
+#definePAGE_KERNEL_NOGLOBAL__pgprot_mask(__PAGE_KERNEL_NOGLOBAL | _ENC)
#definePAGE_KERNEL_NOENC__pgprot_mask(__PAGE_KERNEL | 0)
#definePAGE_KERNEL_RO__pgprot_mask(__PAGE_KERNEL_RO | _ENC)
#definePAGE_KERNEL_EXEC__pgprot_mask(__PAGE_KERNEL_EXEC | _ENC)Should this be part of earlier mermap x86 patches?
Maybe it could be in the patch that implements arch_mermap_pgprot_allowed()? This is the first patch that actually uses PAGE_KERNEL_NOGLOBAL though since mermap code just cares that you didn't set _PAGE_GLOBAL.
Quoted 7 lines
Quoted 7 lines
mm/Kconfig +0
-0@@ -1506,7 +1506,14 @@ config MERMAP_KUNIT_TEST If unsure, say N.
Quoted 14 lines
mm/Kconfig config MERMAP_KUNIT_TEST · line 1517
Quoted 12 lines
mm/Kconfig +9
-2config PAGE_ALLOC_UNMAPPED - bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST - default COMPILE_TEST + bool "Support allocating pages that aren't in the direct map" + depends on MERMAP + +config PAGE_ALLOC_KUNIT_TESTS + tristate "KUnit tests for the page allocator" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Builds KUnit tests for the page allocator.This belongs to the next patch?
Oops, thanks.
Re: [PATCH v2 20/22] mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations
In reply to Brendan JackmanQuoted 89 lines
The pages being zeroed here are unmapped, so they can't be zeroed via the direct map. Temporarily mapping them in the direct map is not possible because:
- In general this requires allocating pagetables,
- Unmapping them would require a TLB shootdown, which can't be done in general from the allocator (x86 requires IRQs on).
Therefore, use the new mermap mechanism to zero these pages.
The main mermap API is expected to fail very often. In order to avoid needing to fail allocations when that happens, instead fallback to the special
mermap_get_reserved()variant, which is less efficient.Signed-off-by Brendan Jackman <[email protected]>--- arch/x86/include/asm/pgtable_types.h | 2 + mm/Kconfig | 11 +++++- mm/page_alloc.c | 76 +++++++++++++++++++++++++++++++----- 3 files changed, 78 insertions(+), 11 deletions(-)arch/x86/include/asm/pgtable_types.h +2
-0@@ -223,6 +223,7 @@ enum page_cache_mode {#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G)#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G)#define __PAGE_KERNEL (__PP|__RW| 0|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_NOGLOBAL (__PP|__RW| 0|___A|__NX|___D| 0| 0)#define __PAGE_KERNEL_EXEC (__PP|__RW| 0|___A| 0|___D| 0|___G)#define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC)#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G)@@ -245,6 +246,7 @@ enum page_cache_mode {#define __pgprot_mask (x) __pgprot((x) & __default_kernel_pte_mask)#define PAGE_KERNEL __pgprot_mask(__PAGE_KERNEL | _ENC) +#define PAGE_KERNEL_NOGLOBAL __pgprot_mask(__PAGE_KERNEL_NOGLOBAL | _ENC)#define PAGE_KERNEL_NOENC __pgprot_mask(__PAGE_KERNEL | 0)#define PAGE_KERNEL_RO __pgprot_mask(__PAGE_KERNEL_RO | _ENC)#define PAGE_KERNEL_EXEC __pgprot_mask(__PAGE_KERNEL_EXEC | _ENC)mm/Kconfig +9
-2@@ -1506,7 +1506,14 @@ config MERMAP_KUNIT_TEST If unsure, say N. config PAGE_ALLOC_UNMAPPED - bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST - default COMPILE_TEST + bool "Support allocating pages that aren't in the direct map" + depends on MERMAP + +config PAGE_ALLOC_KUNIT_TESTS + tristate "KUnit tests for the page allocator" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Builds KUnit tests for the page allocator. endmenumm/page_alloc.c +5
-3@@ -14,6 +14,7 @@* (lots of bits borrowed from Ingo Molnar & Andrew Morton) */ +#include <linux/mermap.h> #include <linux/stddef.h> #include <linux/mm.h> #include <linux/highmem.h> @@ -1327,15 +1328,72 @@ static inline bool should_skip_kasan_poison(struct page *page)return page_kasan_tag (page) ==KASAN_TAG_KERNEL ; } -static void kernel_init_pages (struct page *page,int numpages) +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +static inline bool pageblock_unmapped (struct page *page) { -int i; +return freetype_flags (get_pageblock_freetype (page)) &FREETYPE_UNMAPPED ; +} -/* s390's use of memset() could override KASAN redzones. */
mm/page_alloc.c static inline bool should_skip_kasan_poison(struct page *page) · line 1342
Quoted 10 lines
mm/page_alloc.c +6
-4-kasan_disable_current (); -for (i =0 ; i < numpages; i++) -clear_highpage_kasan_tagged (page + i); -kasan_enable_current (); +static inline void clear_page_mermap (struct page *page,unsigned int numpages) +{ +void *mermap; + +BUILD_BUG_ON (IS_ENABLED (CONFIG_HIGHMEM )); +
I was thinking — would the fast path read better if we moved migrate_disable()/migrate_enable() up a level, into clear_page_mermap() itself? i.e
migrate_disable() -> here
mm/page_alloc.c static inline bool should_skip_kasan_poison(struct page *page) · line 1350
Quoted 8 lines
mm/page_alloc.c +8
-0+/* Fast path: single mapping (may fail under preemption). */ + mermap =mermap_get (page, numpages <<PAGE_SHIFT ,PAGE_KERNEL_NOGLOBAL ); +if (mermap) { +void *buf =kasan_reset_tag (mermap_addr (mermap)); + +for (int i =0 ; i < numpages; i++) +clear_page (buf + (i <<PAGE_SHIFT )); +mermap_put (mermap);
migrate_enable() -> here
Right now we split the migrate_disable/enable() a layer below across mermap_get() and mermap_put().
Would it make sense to embed that in the clear_page_mermap() API itself — mirroring how we already disable IRQs at this level for the mermap_get_reserved()?
Thanks,
Pankaj
Quoted 61 lines
mm/page_alloc.c +48
-2+ return; + } + +/* Slow path, map each page individually (always succeeds). */ +for (int i =0 ; i < numpages; i++) { +unsigned long flags; + +local_irq_save (flags); + mermap =mermap_get_reserved (page + i,PAGE_KERNEL_NOGLOBAL ); +clear_page (kasan_reset_tag (mermap_addr (mermap))); +mermap_put (mermap); +local_irq_restore (flags); + } +} +#else +static inline bool pageblock_unmapped (struct page *page) +{ +return false; +} + +static inline void clear_page_mermap (struct page *page,unsigned int numpages) +{ +BUG (); +} +#endif + +static void kernel_init_pages (struct page *page,unsigned int numpages) +{ +int num_blocks =DIV_ROUND_UP (numpages, pageblock_nr_pages); + +for (int block =0 ; block < num_blocks; block++) { +struct page *block_page = page + (block << pageblock_order); +bool unmapped =pageblock_unmapped (block_page); + +/* s390's use of memset() could override KASAN redzones. */ +kasan_disable_current (); +if (unmapped) { +clear_page_mermap (block_page, numpages); + }else { +for (int i =0 ; i <min (numpages, pageblock_nr_pages); i++) +clear_highpage_kasan_tagged (block_page + i); + } +kasan_enable_current (); + + numpages -= pageblock_nr_pages; + } }#ifdef CONFIG_MEM_ALLOC_PROFILING @@ -5250,8 +5308,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, ac->nodemask = nodemask; ac->freetype =gfp_freetype (gfp_mask); -/* Not implemented yet. */ -if (freetype_flags (ac->freetype) &FREETYPE_UNMAPPED && gfp_mask & __GFP_ZERO) +if (freetype_flags (ac->freetype) &FREETYPE_UNMAPPED && +WARN_ON (!mermap_ready ()))return false;if (cpusets_enabled ()) {
Brendan Jackman
reviewerRe: [PATCH v2 20/22] mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations
Hi Pankaj,
Thanks for taking a look!
On Wed Jun 17, 2026 at 11:32 AM UTC, Pankaj Gupta wrote:
...
Quoted 24 lines
mm/page_alloc.c static inline bool should_skip_kasan_poison(struct page *page) · line 1342
Quoted 6 lines
mm/page_alloc.c +6
-0+static inline void clear_page_mermap (struct page *page,unsigned int numpages) +{ +void *mermap; + +BUILD_BUG_ON (IS_ENABLED (CONFIG_HIGHMEM )); +I was thinking — would the fast path read better if we moved migrate_disable()/migrate_enable() up a level, into
clear_page_mermap()itself? i.e
migrate_disable()-> heremm/page_alloc.c static inline bool should_skip_kasan_poison(struct page *page) · line 1350
Quoted 8 lines
mm/page_alloc.c +8
-0+/* Fast path: single mapping (may fail under preemption). */ + mermap =mermap_get (page, numpages <<PAGE_SHIFT ,PAGE_KERNEL_NOGLOBAL ); +if (mermap) { +void *buf =kasan_reset_tag (mermap_addr (mermap)); + +for (int i =0 ; i < numpages; i++) +clear_page (buf + (i <<PAGE_SHIFT )); +mermap_put (mermap);
migrate_enable()-> here
Quoted 7 lines
Right now we split the migrate_disable/enable() a layer below across
mermap_get()andmermap_put().Would it make sense to embed that in the
clear_page_mermap()API itself — mirroring how we already disable IRQs at this level for themermap_get_reserved()?
Hm, I guess the symmetry would make sense. The reason it's this way is because the mermap_{get,put}() API mirrors kmap_[un]local_page() etc which do the migrate_disable() internally while there is no precedence (AFAIK) for an API that automatically {dis,en}ables IRQs. That might also be a side-effect of the fact that local_irq_save() requires a local variable while migrate_disble() stores its state globally. Similarly, there is no equivalent of lockdep_assert_preemption_disabled().
Quoted 4 lines
Thanks,
Pankaj
Quoted 12 lines
mm/page_alloc.c static inline bool should_skip_kasan_poison(struct page *page) · line 1362
Quoted 12 lines
mm/page_alloc.c +12
-0+return ; + } + +/* Slow path, map each page individually (always succeeds). */ +for (int i =0 ; i < numpages; i++) { +unsigned long flags; + +local_irq_save (flags); + mermap =mermap_get_reserved (page + i,PAGE_KERNEL_NOGLOBAL ); +clear_page (kasan_reset_tag (mermap_addr (mermap))); +mermap_put (mermap); +local_irq_restore (flags);
BTW, writing the above just made me realise: we don't need to disable IRQs here, only preemption.
In a WIP version I did have mermap_get_reserved() requiring irqsave, so that you could theoretically use it from an IRQ. But actually I don't think that makes any sense since it depends on state in current->mm.
Re: [PATCH v2 20/22] mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations
Hi Brendan,
Quoted 35 lines
Hi Pankaj,
Thanks for taking a look!
On Wed Jun 17, 2026 at 11:32 AM UTC, Pankaj Gupta wrote:
...Quoted 22 lines
mm/page_alloc.c static inline bool should_skip_kasan_poison(struct page *page) · line 1342
Quoted 6 lines
mm/page_alloc.c +6
-0+static inline void clear_page_mermap (struct page *page,unsigned int numpages) +{ +void *mermap; + +BUILD_BUG_ON (IS_ENABLED (CONFIG_HIGHMEM )); +I was thinking — would the fast path read better if we moved migrate_disable()/migrate_enable() up a level, into
clear_page_mermap()itself? i.e
migrate_disable()-> heremm/page_alloc.c static inline bool should_skip_kasan_poison(struct page *page) · line 1350
Quoted 8 lines
mm/page_alloc.c +8
-0+/* Fast path: single mapping (may fail under preemption). */ + mermap =mermap_get (page, numpages <<PAGE_SHIFT ,PAGE_KERNEL_NOGLOBAL ); +if (mermap) { +void *buf =kasan_reset_tag (mermap_addr (mermap)); + +for (int i =0 ; i < numpages; i++) +clear_page (buf + (i <<PAGE_SHIFT )); +mermap_put (mermap);
migrate_enable()-> hereQuoted 7 lines
Right now we split the migrate_disable/enable() a layer below across
mermap_get()andmermap_put().Would it make sense to embed that in the
clear_page_mermap()API itself — mirroring how we already disable IRQs at this level for themermap_get_reserved()?
Quoted 7 lines
Hm, I guess the symmetry would make sense. The reason it's this way is because the mermap_{get,put}() API mirrors kmap_[un]local_page() etc which do the
migrate_disable()internally while there is no precedence (AFAIK) for an API that automatically {dis,en}ables IRQs. That might also be a side-effect of the fact thatlocal_irq_save()requires a local variable whilemigrate_disble()stores its state globally. Similarly, there is no equivalent oflockdep_assert_preemption_disabled().
Makes sense. There is cant_migrate() though, maybe in current code we could use in mermap_put()?
Quoted 15 lines
Quoted 4 lines
Quoted 4 lines
mm/page_alloc.c +4
-0+return ; + } + +/* Slow path, map each page individually (always succeeds). */ Quoted 8 lines
mm/page_alloc.c static inline bool should_skip_kasan_poison(struct page *page) · line 1362
Quoted 8 lines
mm/page_alloc.c +8
-0+for (int i =0 ; i < numpages; i++) { +unsigned long flags; + +local_irq_save (flags); + mermap =mermap_get_reserved (page + i,PAGE_KERNEL_NOGLOBAL ); +clear_page (kasan_reset_tag (mermap_addr (mermap))); +mermap_put (mermap); +local_irq_restore (flags);BTW, writing the above just made me realise: we don't need to disable IRQs here, only preemption.
Quoted 3 lines
In a WIP version I did have
mermap_get_reserved()requiring irqsave, so that you could theoretically use it from an IRQ. But actually I don't think that makes any sense since it depends on state incurrent->mm.
Agree.
Thanks,
Pankaj
Quoted 2 lines
PATCH v2 21/22 mm: Minimal KUnit tests for some new page_alloc logic
In reply to Brendan JackmanAdd a simple smoke test for __GFP_UNMAPPED that tries to exercise flipping pageblocks between mapped/unmapped state.
Also add some basic tests for some freelist-indexing helpers.
Simplest way to run these on x86:
tools/testing/kunit/kunit.py run --arch=x86_64 "page_alloc.*" \ --kconfig_add CONFIG_MERMAP=y --kconfig_add CONFIG_PAGE_ALLOC_UNMAPPED=y
--- kernel/panic.c | 2 + mm/Kconfig | 2 +- mm/Makefile | 1 + mm/init-mm.c | 3 + mm/internal.h | 6 ++ mm/page_alloc.c | 11 +- mm/tests/page_alloc_kunit.c | 250 ++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 271 insertions(+), 4 deletions(-)
kernel/panic.c +2 -0
@@ -39,6 +39,7 @@#include <linux/sys_info.h> #include <trace/events/error_report.h> #include <asm/sections.h> +#include <kunit/visibility.h> #define PANIC_TIMER_STEP 100#define PANIC_BLINK_SPD 18
@@ -941,6 +942,7 @@ unsigned long get_taint(void) {return tainted_mask; } +EXPORT_SYMBOL_IF_KUNIT (get_taint);/** * add_taint: add a taint flag if not already set.
mm/Kconfig +1 -1
@@ -1509,7 +1509,7 @@ config PAGE_ALLOC_UNMAPPED bool "Support allocating pages that aren't in the direct map" depends on MERMAP -config PAGE_ALLOC_KUNIT_TESTS +config PAGE_ALLOC_KUNIT_TEST tristate "KUnit tests for the page allocator" if !KUNIT_ALL_TESTS depends on KUNIT default KUNIT_ALL_TESTS
mm/Makefile +1 -0
@@ -152,3 +152,4 @@ obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o obj-$(CONFIG_MERMAP) += mermap.o obj-$(CONFIG_MERMAP_KUNIT_TEST) += tests/mermap_kunit.o +obj-$(CONFIG_PAGE_ALLOC_KUNIT_TEST) += tests/page_alloc_kunit.o
mm/init-mm.c +3 -0
@@ -13,6 +13,8 @@#include <linux/iommu.h> #include <asm/mmu.h> +#include <kunit/visibility.h> +#ifndef INIT_MM_CONTEXT #define INIT_MM_CONTEXT (name)#endif
@@ -50,6 +52,7 @@ struct mm_struct init_mm = { .flexible_array = MM_STRUCT_FLEXIBLE_ARRAY_INIT, INIT_MM_CONTEXT(init_mm) }; +EXPORT_SYMBOL_IF_KUNIT (init_mm);void setup_initial_init_mm (void *start_code,void *end_code,void *end_data,void *brk)
mm/internal.h +6 -0
@@ -1908,4 +1908,10 @@ int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, unsigned long size, pte_fn_t fn, void *data, unsigned int flags); +#if IS_ENABLED (CONFIG_KUNIT ) +unsigned int order_to_pindex (freetype_t freetype,int order); +int pindex_to_order (unsigned int pindex); +bool pcp_allowed_order (unsigned int order); +#endif /* IS_ENABLED(CONFIG_KUNIT) */ +#endif /* __MM_INTERNAL_H */
mm/page_alloc.c +8 -3
@@ -58,6 +58,7 @@#include <linux/pgalloc_tag.h> #include <linux/mmzone_lock.h> #include <asm/div64.h> +#include <kunit/visibility.h> #include "internal.h" #include "shuffle.h" #include "page_reporting.h"
@@ -461,6 +462,7 @@ get_pfnblock_freetype(const struct page *page, unsigned long pfn) {return __get_pfnblock_freetype (page, pfn,0 ); } +EXPORT_SYMBOL_IF_KUNIT (get_pfnblock_freetype);/**
@@ -696,7 +698,7 @@ static void bad_page(struct page *page, const char *reason)add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); } -static inline unsigned int order_to_pindex (freetype_t freetype,int order) +VISIBLE_IF_KUNIT inline unsigned int order_to_pindex(freetype_t freetype, int order) {int migratetype =free_to_migratetype (freetype);
@@ -724,8 +726,9 @@ static inline unsigned int order_to_pindex(freetype_t freetype, int order)return (MIGRATE_PCPTYPES * order) + migratetype; } +EXPORT_SYMBOL_IF_KUNIT (order_to_pindex); -static inline int pindex_to_order (unsigned int pindex) +VISIBLE_IF_KUNIT int pindex_to_order(unsigned int pindex) {unsigned int unmapped_base =NR_LOWORDER_PCP_LISTS +NR_PCP_THP ;int order;
@@ -748,8 +751,9 @@ static inline int pindex_to_order(unsigned int pindex)return order; } +EXPORT_SYMBOL_IF_KUNIT (pindex_to_order); -static inline bool pcp_allowed_order (unsigned int order) +VISIBLE_IF_KUNIT inline bool pcp_allowed_order(unsigned int order) {if (order <=PAGE_ALLOC_COSTLY_ORDER )return true;
@@ -759,6 +763,7 @@ static inline bool pcp_allowed_order(unsigned int order)#endif return false; } +EXPORT_SYMBOL_IF_KUNIT (pcp_allowed_order);/* * Higher-order pages are called "compound pages". They are structured thusly:
mm/tests/page_alloc_kunit.c new file +250 -0
250 lines not on this page · raw message
Signature · 1 line
2.51.2
PATCH v2 22/22 mm/secretmem: Use __GFP_UNMAPPED when available
In reply to Brendan JackmanThis is the simplest possible way to adopt __GFP_UNMAPPED. Use it to allocate pages when it's available, meaning the set_direct_map_invalid_noflush() call is no longer needed.
--- mm/secretmem.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 74 insertions(+), 13 deletions(-)
mm/secretmem.c +74 -13
@@ -6,6 +6,7 @@*/ #include <linux/mm.h> +#include <linux/mermap.h> #include <linux/fs.h> #include <linux/swap.h> #include <linux/mount.h>
@@ -47,13 +48,78 @@ bool secretmem_active(void)return !!atomic_read (&secretmem_users); } +/* +* If it's supported, allocate using __GFP_UNMAPPED. This lets the page +* allocator amortize TLB flushes and avoids direct map fragmentation. +*/ +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +static inline struct folio *secretmem_folio_alloc (gfp_t gfp,unsigned int order) +{ +int err; + +/* Required for __GFP_UNMAPPED|__GFP_ZERO. */ + err =mermap_mm_prepare (current->mm); +if (err) +return ERR_PTR (err); + +return folio_alloc (gfp | __GFP_UNMAPPED, order); +} + +static inline void secretmem_vma_close (struct vm_area_struct *area) +{ +/* +* Because the folio was allocated with __GFP_UNMAPPED|__GFP_ZERO, a TLB +* shootdown is required for the mermap in order to prevent CPU attacks +* from leaking the content. This is the simplest possible way to +* achieve that, but obviously it's inefficient - it should really be +* amortized against the normal flushing that happened during the VMA +* teardown. +*/ +flush_tlb_mm (area->vm_mm); +} + +/* Used __GFP_UNMAPPED so no need to restore direct map or flush TLB. */ +static inline void secretmem_folio_restore (struct folio *folio) { } +static inline void secretmem_folio_flush (struct folio *folio) { } + +#else +static inline struct folio *secretmem_folio_alloc (gfp_t gfp,unsigned int order) +{ +struct folio *folio; +int err; + + folio =folio_alloc (gfp, order); +if (!folio) +return NULL ; + + err =set_direct_map_invalid_noflush (folio_page (folio,0 )); +if (err) { +folio_put (folio); +return ERR_PTR (err); + } + +return folio; +} + +static inline void secretmem_folio_restore (struct folio *folio) +{ +set_direct_map_default_noflush (folio_page (folio,0 )); +} + +static inline void secretmem_folio_flush (struct folio *folio) +{ +unsigned long addr = (unsigned long )folio_address (folio); + +flush_tlb_kernel_range (addr, addr +PAGE_SIZE ); +} +#endif +static vm_fault_t secretmem_fault (struct vm_fault *vmf) {struct address_space *mapping = vmf->vma->vm_file->f_mapping;struct inode *inode =file_inode (vmf->vma->vm_file);pgoff_t offset = vmf->pgoff;gfp_t gfp = vmf->gfp_mask; -unsigned long addr;struct folio *folio;vm_fault_t ret;int err;
@@ -66,16 +132,9 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) retry: folio =filemap_lock_folio (mapping, offset);if (IS_ERR (folio)) { - folio =folio_alloc (gfp | __GFP_ZERO,0 ); -if (!folio) { - ret =VM_FAULT_OOM ; - goto out; - } - - err =set_direct_map_invalid_noflush (folio_page (folio,0 )); -if (err) { -folio_put (folio); - ret =vmf_error (err); + folio =secretmem_folio_alloc (gfp | __GFP_ZERO,0 ); +if (IS_ERR_OR_NULL (folio)) { + ret = folio ?vmf_error (PTR_ERR (folio)) :VM_FAULT_OOM ; goto out; }
@@ -96,8 +155,7 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) goto out; } - addr = (unsigned long )folio_address (folio); -flush_tlb_kernel_range (addr, addr +PAGE_SIZE ); +secretmem_folio_flush (folio); } vmf->page =folio_file_page (folio, vmf->pgoff);
@@ -110,6 +168,9 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf)static const struct vm_operations_struct secretmem_vm_ops = { .fault = secretmem_fault, +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED + .close = secretmem_vma_close, +#endif };static int secretmem_release (struct inode *inode,struct file *file)
Signature · 1 line
2.51.2
Re: [PATCH v2 22/22] mm/secretmem: Use __GFP_UNMAPPED when available
I skipped on posting most of the intervening AI reviews coz there's just so much stuff and most of it is pretty boring, but this one is interesting.
https://sashiko.dev/#/patchset/20260320-page_alloc-unmapped-v2-0-28bf1bd54f41%40google.com
Quoted 28 lines · Brendan Jackman
This is the simplest possible way to adopt
__GFP_UNMAPPED. Use it to allocate pages when it's available, meaning theset_direct_map_invalid_noflush()call is no longer needed.Signed-off-by Brendan Jackman <[email protected]>--- mm/secretmem.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 74 insertions(+), 13 deletions(-)mm/secretmem.c +4
-0@@ -6,6 +6,7 @@*/ #include <linux/mm.h> +#include <linux/mermap.h> #include <linux/fs.h> #include <linux/swap.h> #include <linux/mount.h> @@ -47,13 +48,78 @@ bool secretmem_active(void)return !!atomic_read (&secretmem_users); } +/* +* If it's supported, allocate using __GFP_UNMAPPED. This lets the page +* allocator amortize TLB flushes and avoids direct map fragmentation.
mm/secretmem.c bool secretmem_active(void) · line 63
Quoted 10 lines
mm/secretmem.c +10
-0+*/ +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +static inline struct folio *secretmem_folio_alloc (gfp_t gfp,unsigned int order) +{ +int err; + +/* Required for __GFP_UNMAPPED|__GFP_ZERO. */ + err =mermap_mm_prepare (current->mm); +if (err) +return ERR_PTR (err);
Sashiko:
Quoted 3 lines
mm/secretmem.c +0
-0In remote access paths such as process_vm_readv or io_uring worker threads,
current->mmmight be NULL or point to a different address space than the faulted VMA. Should this usevmf->vma->vm_mminstead?
This can't happen, right? I think the assumption is good; doing a secretmem fault in a kthread or any process that doesn't have the file mmap()'d would be a bug? But this definitely feels like something I could be wrong about.
AI slop to empirically check the two examples mentioned by the AI fail early (slop for slop, it's slop all the way down...):
- process_vm_readv(): https://paste.debian.net/hidden/5625ef2e
- io_uring: https://paste.debian.net/hidden/e7763ad2
[...]
mm/secretmem.c bool secretmem_active(void) · line 107
Quoted 4 lines
mm/secretmem.c +4
-0+static inline void secretmem_folio_restore (struct folio *folio) +{ +set_direct_map_default_noflush (folio_page (folio,0 )); +}
I defined a lovely helper here but neglected to actually call it. Sashiko says "this isn't a bug" but that's wrong, calling set_direct_map_default_noflush() before freeing a __GFP_UNMAPPED page is not OK.
(Should it be? I think no. We _could_ define "default" so that it checks the pageblock flags and does the right thing for you. But then we'd be baking in the assumption that the page allocator can efficiently look that up. This would be rather tricky if we decided we need to mix mapped an unmapped pages in the same block).
And this was hiding the more important bug which was that I forgot to do the mermap dance for zeroing the page.
Quoted 39 lines
mm/secretmem.c +11
-11+static inline void secretmem_folio_flush (struct folio *folio) +{ +unsigned long addr = (unsigned long )folio_address (folio); + +flush_tlb_kernel_range (addr, addr +PAGE_SIZE ); +} +#endif +static vm_fault_t secretmem_fault (struct vm_fault *vmf) {struct address_space *mapping = vmf->vma->vm_file->f_mapping;struct inode *inode =file_inode (vmf->vma->vm_file);pgoff_t offset = vmf->pgoff;gfp_t gfp = vmf->gfp_mask; -unsigned long addr;struct folio *folio;vm_fault_t ret;int err;@@ -66,16 +132,9 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) retry: folio =filemap_lock_folio (mapping, offset);if (IS_ERR (folio)) { - folio =folio_alloc (gfp | __GFP_ZERO,0 ); -if (!folio) { - ret =VM_FAULT_OOM ; - goto out; - } - - err =set_direct_map_invalid_noflush (folio_page (folio,0 )); -if (err) { -folio_put (folio); - ret =vmf_error (err); + folio =secretmem_folio_alloc (gfp | __GFP_ZERO,0 ); +if (IS_ERR_OR_NULL (folio)) { + ret = folio ?vmf_error (PTR_ERR (folio)) :VM_FAULT_OOM ; goto out; }
Quoted 11 lines
err = filemap_add_folio(mapping, folio, offset, gfp);
if (unlikely(err)) {
/*
* If a split of large page was required, it
* already happened when we marked the page invalid
* which guarantees that this call won't fail
*/
set_direct_map_default_noflush(folio_page(folio, 0));
folio_put(folio);
if (err == -EEXIST)
goto retry;This failure path leaks mermap TLB entries on ther CPUs. So if an attacker can trigger this path, and cause another CPU to populate its TLB for the mermap region, and then cause a victim to allocate the page they just freed, they can use those entries for a sidechannel attack.
I did call out in the cover letter that there's some jank with the "if you use __GFP_UNMAPPED|__GFP_ZERO then you need to think about the TLB" thing. But, this shows it's worse than I thought. I need to think about how to mitigate this.
Quoted 9 lines · Brendan Jackman
Because of these ambitious usecases, it's core to this proposal that the feature
overloading the concept of a migratetype, this extension is done by adding a new concept on top of migratetype: the _freetype_. A freetype is basically just a migratetype plus some flags, and it replaces migratetypes wherever the latter is currently used as to index free pages.
I'm a bit confused why the need for additional level of indirection instead of just adding a new migratetype. You still end up increasing the migratetype matrix, just with a new dimension.
(apologies if this was covered in prior work or discussions, just now
plugging myself into the series).
Why not simply have an unmapped migratetype, for example, and on steal you convert it to movable or whatever the preference is?
Quoted 11 lines
.:::: Hacky bits: simplistic secretmem integration
The secretmem integration leaves the mmain optimisations on the table; the security-required flushes of the mermap areas are implemented via distinct
tlb_flush_mm()calls. It should be possible to amortize the mermap TLB flushes completely into the normal VMA flushing. However, as far as I know there is no performance-sensitive usecase for secretmem. So, I've just implemented the minimal adoption. This will at least avoid fragmentation of the direct map, even if it doesn't reduce TLB flushing. If anyone knows of a workload that might benefit from dropping that flushing, let me know!
Crossing a couple streams here, I wonder if there's some mechanisms introduced by MST's latest multi-zeroing-avoidance [1] code that might help deal with the problem here.
MST wired up an optional user_addr into the buddy that allows us to sink the zeroing step for folio_zero_user (or folio_user_zero or whatever) into the post_alloc_hook - which includes some cache flushing.
That conveniently gives you what you need for a TLB flush AND an indicator that the allocation is intended for userland.
Unless I'm fundamentally misunderstanding something, the pattern at least seems similar.
In that sense, does this just become a post_alloc_hook that unmaps the memory after zeroing and allocation?
I get the intent is to have the majority of memory unmapped by default, and then steal those blocks and map them as the kernel requires more memory, but I wonder if it's cleaner to do it the other way and simply have the buddy unmap on alloc after zeroing, and remap on free.
Seems like the free path would be trivial, check if the page is in the direct map and if not, remap it and move on. Entirely hidden from existing users.
So, maybe a stupid question: Was the opposite mechanism considered (unmap on alloc sunk into the buddy), and if so was it rejected for some other reason?
[1] https://lore.kernel.org/linux-mm/[email protected]/
~Gregory
Quoted 15 lines · Gregory Price
Quoted 9 lines · Brendan Jackman
Because of these ambitious usecases, it's core to this proposal that the feature
overloading the concept of a migratetype, this extension is done by adding a new concept on top of migratetype: the _freetype_. A freetype is basically just a migratetype plus some flags, and it replaces migratetypes wherever the latter is currently used as to index free pages.I'm a bit confused why the need for additional level of indirection instead of just adding a new migratetype. You still end up increasing the migratetype matrix, just with a new dimension.
Quoted 5 lines
(apologies if this was covered in prior work or discussions, just now
plugging myself into the series).Why not simply have an unmapped migratetype, for example, and on steal you convert it to movable or whatever the preference is?
Because the fact that only one migratetype currently supports being unmapped is a temporary happenstance of the guest_memfd usecase. In general, this needs to support having unmapped variants of ~arbitrary migratetypes.
Quoted 20 lines
Quoted 11 lines
.:::: Hacky bits: simplistic secretmem integration
The secretmem integration leaves the mmain optimisations on the table; the security-required flushes of the mermap areas are implemented via distinct
tlb_flush_mm()calls. It should be possible to amortize the mermap TLB flushes completely into the normal VMA flushing. However, as far as I know there is no performance-sensitive usecase for secretmem. So, I've just implemented the minimal adoption. This will at least avoid fragmentation of the direct map, even if it doesn't reduce TLB flushing. If anyone knows of a workload that might benefit from dropping that flushing, let me know!Crossing a couple streams here, I wonder if there's some mechanisms introduced by MST's latest multi-zeroing-avoidance [1] code that might help deal with the problem here.
MST wired up an optional
user_addrinto the buddy that allows us to sink the zeroing step forfolio_zero_user(orfolio_user_zeroor whatever) into thepost_alloc_hook- which includes some cache flushing.
Quoted 5 lines
That conveniently gives you what you need for a TLB flush AND an indicator that the allocation is intended for userland.
Unless I'm fundamentally misunderstanding something, the pattern at least seems similar.
Yeah, I actually only noticed that yesterday due to your posts on that thread! I need to investigate it further. My assumption has always been that this isn't a general solution because we don't always _have_ a user address (e.g. for guest_memfd it's important that we can populate the memory via write(), so there's no user address), but it's pretty likely I'm missing something there.
Quoted 7 lines
In that sense, does this just become a
post_alloc_hookthat unmaps the memory after zeroing and allocation?I get the intent is to have the majority of memory unmapped by default, and then steal those blocks and map them as the kernel requires more memory, but I wonder if it's cleaner to do it the other way and simply have the buddy unmap on alloc after zeroing, and remap on free.
That would be cleaner indeed, but the key question here isn't about the default state of memory here, it's about batching.
The reason we need to do it at the block granularity is that a TLB flush every time we allocate one of these pages is a performance nonstarter - that's actually the entire point of this series. If you can afford a TLB flush per allocation then you don't need __GFP_UNMAPPED for the guest_memfd usecase, the existing direct map removal series [0] is already fine.
[0] https://lore.kernel.org/all/[email protected]/
Quoted 7 lines
Seems like the free path would be trivial, check if the page is in the direct map and if not, remap it and move on. Entirely hidden from existing users.
So, maybe a stupid question: Was the opposite mechanism considered (unmap on alloc sunk into the buddy), and if so was it rejected for some other reason?
Hopefully my prev paragraph explained that it's not viable anyway, but:
if we _did_ do the [un]mapping on a per-allocation basis, the disadvantage of unmap-on-alloc is just that we expect most pages in the system to be unmapped. So the majority of map-unmap cycles are pointless (map on free, but we're probably gonna unmap again on allloc).
Cheers,
Brendan
Quoted 10 lines · Brendan Jackman
Quoted 3 lines · Gregory Price
Why not simply have an unmapped migratetype, for example, and on steal you convert it to movable or whatever the preference is?
Because the fact that only one migratetype currently supports being unmapped is a temporary happenstance of the
guest_memfdusecase. In general, this needs to support having unmapped variants of ~arbitrary migratetypes.
Ah I see, that tracks. Thank you for the context.
Quoted 11 lines
Quoted 3 lines
Unless I'm fundamentally misunderstanding something, the pattern at least seems similar.
Yeah, I actually only noticed that yesterday due to your posts on that thread! I need to investigate it further. My assumption has always been that this isn't a general solution because we don't always _have_ a user address (e.g. for
guest_memfdit's important that we can populate the memory viawrite(), so there's no user address), but it's pretty likely I'm missing something there.
Hm. I'm not quite wrapping my head around the TLB issue fully.
If there's no kernel direct mapping, and there's no userland mapping, the stale TLB entry comes from... the page formerly being present in the page tables and a stale TLB entry lying about after the page is freed?
If that's the case, that sounds more like someone isn't flushing the TLB entry correctly when the page is freed or unmapped (for a transient mermap situation), rather than an issue to be handled by the allocator.
I think I just need to spend a little more time understanding the TLB issue.
Quoted 8 lines
The reason we need to do it at the block granularity is that a TLB flush every time we allocate one of these pages is a performance nonstarter - that's actually the entire point of this series. If you can afford a TLB flush per allocation then you don't need
__GFP_UNMAPPEDfor theguest_memfdusecase, the existing direct map removal series [0] is already fine.
That tracks. So whatever the case, it seems like you prefer block-granularity map/unmap operations, rather than per-allocation.
Do you have numbers on what the cost of map/unmap on a page vs block is?
Whatever the case is - you're pushing that cost onto *someone*, so having that data would at least let us know the value of pushing most memory to be unmapped by default rather than the opposite.
~Gregory
Vlastimil Babka (SUSE)
maintainerQuoted 6 lines · Gregory Price
Hm. I'm not quite wrapping my head around the TLB issue fully.
If there's no kernel direct mapping, and there's no userland mapping, the stale TLB entry comes from... the page formerly being present in the page tables and a stale TLB entry lying about after the page is freed?
It's the direct mapping, we assume it's always there and unchanged, and only kernel can access the contents through it. So nobody flushes it when freeing any pages. Userspace processes can't exploit anything stale there, in absence of kernel's UAF bugs (or e.g. Meltdown like cpu bugs).
Quoted 7 lines
If that's the case, that sounds more like someone isn't flushing the TLB entry correctly when the page is freed or unmapped (for a transient mermap situation), rather than an issue to be handled by the allocator.
I think I just need to spend a little more time understanding the TLB issue.
Quoted 4 lines · Vlastimil Babka (SUSE)
Quoted 3 lines · Gregory Price
Hm. I'm not quite wrapping my head around the TLB issue fully.
Quoted 9 lines
Quoted 3 lines
If there's no kernel direct mapping, and there's no userland mapping, the stale TLB entry comes from... the page formerly being present in the page tables and a stale TLB entry lying about after the page is freed?
It's the direct mapping, we assume it's always there and unchanged, and only kernel can access the contents through it. So nobody flushes it when freeing any pages. Userspace processes can't exploit anything stale there, in absence of kernel's UAF bugs (or e.g. Meltdown like cpu bugs).
Ah, I follow.
If everything is default-unmapped, then you don't have to worry about this issue - except when a stolen block is returned or an ephemeral mapping is unmapped after the operation.
pivoting...
On the GFP front, i wonder if you could factor out the core of alloc_frozen_pages_noprof() and add alloc_unmapped_pages_noprof() which adds (alloc_flags |= ALLOC_UNMAPPED) instead of adding __GFP_UNMAPPED.
I have been considering something similar for __GFP_PRIVATE, but this has the added downside of increasing the surface of the buddy for each new narrow use case (in my case, private nodes, in this case unmapped allocations).
unless of course we nip that in the bud with something like
struct page *alloc_pages_special(enum buddy_context ctxt, gfp_t gfp_mask, ...)
{
switch (ctxt) {
... internal-only details about how that case is handled ...
}
}and just go ahead and allow the buddy to grow internally without adding new gfp flags or an infinite number of interfaces.
Of course that means users have to know the context in which they're being allocated. Right now you can kind of "transiently cheat" by passing a GFP flag through a bunch of interfaces and that makes certain allocations reachable - but maybe we should not be encouraging that kind of design for these kinds of allocator extensions?
Just spitballing here on the GFP issue since that seems to be coming to a head on multiple fronts.
~Gregory
Quoted 37 lines · Gregory Price
Quoted 4 lines · Vlastimil Babka (SUSE)
Quoted 3 lines · Gregory Price
Hm. I'm not quite wrapping my head around the TLB issue fully.
Quoted 9 lines
Quoted 3 lines
If there's no kernel direct mapping, and there's no userland mapping, the stale TLB entry comes from... the page formerly being present in the page tables and a stale TLB entry lying about after the page is freed?
It's the direct mapping, we assume it's always there and unchanged, and only kernel can access the contents through it. So nobody flushes it when freeing any pages. Userspace processes can't exploit anything stale there, in absence of kernel's UAF bugs (or e.g. Meltdown like cpu bugs).
Ah, I follow.
If everything is default-unmapped, then you don't have to worry about this issue - except when a stolen block is returned or an ephemeral mapping is unmapped after the operation.
pivoting...
On the GFP front, i wonder if you could factor out the core of
alloc_frozen_pages_noprof()and addalloc_unmapped_pages_noprof()which adds (alloc_flags|=ALLOC_UNMAPPED) instead of adding__GFP_UNMAPPED.I have been considering something similar for
__GFP_PRIVATE, but this has the added downside of increasing the surface of the buddy for each new narrow use case (in my case, private nodes, in this case unmapped allocations).unless of course we nip that in the bud with something like
struct page *
alloc_pages_special(enum buddy_context ctxt, gfp_t gfp_mask, ...)
Quoted 8 lines
{ switch (ctxt) { ... internal-only details about how that case is handled ... } }and just go ahead and allow the buddy to grow internally without adding new gfp flags or an infinite number of interfaces.
Yeah, this is what I'm thinking too. I don't think growing the interface is such a big deal if we can put it in mm/internal.h. For __GFP_UNMAPPED and ASI's equivalent, we would eventually want to expose the functionality outside of mm/, but that doesn't mean we have to directly expose the page allocator interface itself. Do you think it's a similar story for __GFP_PRIVATE?
Anyway my initial thought was a variant of alloc_pages that lets you directly specify alloc flags alongside/instead of GFP flags. This is actually a bit fiddly though since the GFP flags -> alloc flags thing isn't a clean division. Maybe it should be?
Quoted 5 lines
Of course that means users have to know the context in which they're being allocated. Right now you can kind of "transiently cheat" by passing a GFP flag through a bunch of interfaces and that makes certain allocations reachable - but maybe we should not be encouraging that kind of design for these kinds of allocator extensions?
Hm, for __GFP_UNMAPPED (and __GFP_SENSITIVE in the future), it is nothing to do with the allocation context. It's really expressing something about the page, i.e:
- __GFP_SENSITIVE means "We might put user data in this page"
- __GFP_UNMAPPED means "We might put user data in this page, and I know the kernel doesn't need to access it in the direct map"
So, for those cases, I think a GFP flag is actually conceptually correct, the only reason I can see to avoid it is because of bitmap space.
Quoted 13 lines · Brendan Jackman
Quoted 11 lines · Gregory Price
struct page *
alloc_pages_special(enum buddy_context ctxt, gfp_t gfp_mask, ...){ switch (ctxt) { ... internal-only details about how that case is handled ... } }and just go ahead and allow the buddy to grow internally without adding new gfp flags or an infinite number of interfaces.
Quoted 7 lines
Yeah, this is what I'm thinking too. I don't think growing the interface is such a big deal if we can put it in
mm/internal.h. For__GFP_UNMAPPEDand ASI's equivalent, we would eventually want to expose the functionality outside ofmm/, but that doesn't mean we have to directly expose the page allocator interface itself. Do you think it's a similar story for__GFP_PRIVATE?
The important thing is that the API growth, if there is one, needs to be extensible and not specific to a particular user case.
alloc_pages_unmapped() is just replacing a GFP flag with a function call. Now we've encourage alloc_pages_foobarbazbazinga() to be implemented.
So we should give some careful thought to whether growing the allocator interface is a good idea at all - and if so very intentionally.
I made a similar comment on MST's virtio series [0] about trying not to increase the buddy surface - because it makes it harder to test and easier for people to do the wrong thing.
So if we are going to increase the surface, maybe the intent should be to have an eventual migration, rather than a permanent surface increase.
[0] https://lore.kernel.org/linux-mm/aeozRZ9AvW2jTQkX@gourry-fedora-PF4VCD3F/
Quoted 5 lines
Anyway my initial thought was a variant of
alloc_pagesthat lets you directly specify alloc flags alongside/instead of GFP flags. This is actually a bit fiddly though since the GFP flags -> alloc flags thing isn't a clean division. Maybe it should be?
This is actually what i meant about some GFP flags may more accurately be described as alloc flags. They say something about where to allocate from rather than something about the page itself (ZONE_DMA vs ZONE_CMA vs ZONE_MOVABLE, etc).
Quoted 6 lines
Quoted 5 lines
Of course that means users have to know the context in which they're being allocated. Right now you can kind of "transiently cheat" by passing a GFP flag through a bunch of interfaces and that makes certain allocations reachable - but maybe we should not be encouraging that kind of design for these kinds of allocator extensions?
Quoted 6 lines
Hm, for
__GFP_UNMAPPED(and__GFP_SENSITIVEin the future), it is nothing to do with the allocation context. It's really expressing something about the page, i.e:-
__GFP_SENSITIVEmeans "We might put user data in this page"
That sounds more like an allocation context which implies a set of GFP flags, rather a specific directive to give the allocator.
Quoted 3 lines
-
__GFP_UNMAPPEDmeans "We might put user data in this page, and I know the kernel doesn't need to access it in the direct map"
I don't think this necessarily implies user data, it just implies wanting an unmapped page. The caller's intent isn't necessarily encoded by this flag (although we can derive it from existing use cases, who knows what crazy stuff people will come up with).
Quoted 4 lines
So, for those cases, I think a GFP flag is actually conceptually correct, the only reason I can see to avoid it is because of bitmap space.
I don't know that we can call any particular idea "conceptionally correct" when GFP flags themselves are a mishmash of ideas already :] which is actually the core of the issue isn't it? n_n;;;
~Gregory