mm: Some cleanups for page allocator APIs
Subsystems (26)
v3 cover mm: Some cleanups for page allocator APIs
This is based on mm-new, it depends on moving alloc_tag to mm/ [0].
[0] https://lore.kernel.org/all/aj5QBtJcphPElczI@lucifer/
Some tweaks and cleanups for page allocator entrypoint and flags. This is motivated by preparation for __GFP_UNMAPPED [1] (which will probably become ALLOC_UNMAPPED in its next iteration), but all this is supposed to be an improvement to the codebase in its own right: unifying code paths, reducing API surface, and removing GFP flags.
[1] https://lore.kernel.org/all/[email protected]/
This started with unifying __alloc_frozen_pages[_nolock]_noprof() and expanded from there.
Unifying the nolock allocator entrypoint with the normal allocator entrypoint means adding an alloc_flags argument to the later (only exposed within mm/). This presents an opportunity to take advantage of that arg to remove some GFP flags, if we add that alloc_flags arg a bit more broadly to allocator entrypoints.
To distinguish between mm-internal and "public" allocator entrypoints, it makes sense to use the __ prefix. There are already some public APIs with that prefix. For *alloc_pages*, just removing those variants seems like a nice cleanup anyway, so do that. For get_free_pages, the "__" variant is the _only_ variant and it's very widely used, so it doesn't seem worthwhile to modify that. Therefore, scope this "__" change specifically to the *alloc_pages* API, which means we leave the *folio_alloc* API untouched too, even though that could probably be cleaned up if so desired.
Tested:
- KVM, mm, and BPF selftests in a QEMU VM
- kunit.py on x86_64
- For the ALLOC_NO_CODETAG bits I just booted a VM and read /proc/allocinfo. I confirmed that if I remove ALLOC_NO_CODETAG, the kernel crashes in early boot, so I was at least booting code that depends on this logic.
I used Google's internal version of Antigravity (AI coding harness) to do the repetitive bits, those commits are marked with Assisted-by, the rest is manual.
---
Changes in v3:
- Created mm/page_alloc.h
- Fixed EXPORT_SYMBOL() issues
- Reworded commit messages per Sashiko's pointers
- Dropped rename of alloc_flags arg in prepare_alloc_pages() (Suren)
- Renamed gfp_to_alloc_flags_nonblocking() too after rebasing onto:
https://lore.kernel.org/all/[email protected]/
- Link to v2: https://patch.msgid.link/[email protected]
Changes in v2:
- Fixed up whitespace in nolock unification patch
- Introduced ALLOC_DEFAULT to replace literal 0 for alloc_flags
- All other patches are new
- Link to v1: https://patch.msgid.link/[email protected]
---
Brendan Jackman (16):
mm/page_alloc: rename ALLOC_TRYLOCK -> ALLOC_NOLOCK mm/page_alloc: some renames to clarify alloc_flags scopes
mm: name some args in a function declaration
mm: Split out internal page_alloc.h mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof() mm/page_alloc: relax GFP WARN in nolock allocs
mm: move some stuff to mm/page_alloc.h perf/x86/intel: Use higher-level allocator API
KVM: VMX: Use higher-level allocator API x86/virt: Use higher-level allocator API
sgi-xp: Use higher-level allocator API net/funeth: Switch to higher-level allocator API
mm: Remove __alloc_pages_node()
mm: Move __alloc_pages() to mm/page_alloc.h
mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
mm: remove the __GFP_NO_OBJ_EXT flag
Documentation/admin-guide/cgroup-v1/cpusets.rst | 2 +- Documentation/admin-guide/mm/transhuge.rst | 2 +- MAINTAINERS | 1 + arch/x86/events/intel/ds.c | 6 +- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/virt/hw.c | 2 +- drivers/misc/sgi-xp/xpc_uv.c | 2 +- drivers/net/ethernet/fungible/funeth/funeth_rx.c | 2 +- include/linux/gfp.h | 54 +--- mm/alloc_tag.c | 22 +- mm/compaction.c | 5 +- mm/hugetlb.c | 4 +- mm/internal.h | 253 ------------------ mm/khugepaged.c | 1 + mm/memory-failure.c | 1 + mm/memory_hotplug.c | 1 + mm/mempolicy.c | 11 +- mm/mm_init.c | 1 + mm/page_alloc.c | 259 ++++++++++--------- mm/page_alloc.h | 316 +++++++++++++++++++++++ mm/page_frag_cache.c | 6 +- mm/page_isolation.c | 1 + mm/page_owner.c | 2 +- mm/show_mem.c | 1 + mm/slub.c | 7 +- mm/swap.c | 1 + mm/vmscan.c | 1 + mm/vmstat.c | 1 + tools/include/linux/gfp_types.h | 7 - 29 files changed, 506 insertions(+), 468 deletions(-)
---
base-commit: ec005628e9dbcef26b761fa860f264fe6e5fd690
change-id: 20260617-alloc-trylock-14ad37dab337
Best regards,
Signature · 1 line
Brendan Jackman <[email protected]>
PATCH v3 1/16 mm/page_alloc: rename ALLOC_TRYLOCK -> ALLOC_NOLOCK
It's confusing that the function is called "nolock" but the flag is called "trylock", align them.
The function's terminology is more visible and has more mindshare so use that.
--- mm/internal.h | 2 +- mm/page_alloc.c | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-)
mm/internal.h +1 -1
@@ -1480,7 +1480,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_NOFRAGMENT 0x0#endif #define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ -#define ALLOC_TRYLOCK 0x400/* Only use spin_trylock in allocation path */ +#define ALLOC_NOLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */
mm/page_alloc.c +5 -5
@@ -2530,7 +2530,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,unsigned long flags;int i; -if (unlikely (alloc_flags &ALLOC_TRYLOCK )) { +if (unlikely (alloc_flags &ALLOC_NOLOCK )) {if (!spin_trylock_irqsave (&zone->lock, flags))return 0 ; }else {
@@ -3218,7 +3218,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - if (unlikely(alloc_flags & ALLOC_TRYLOCK)) { + if (unlikely(alloc_flags & ALLOC_NOLOCK)) { if (!spin_trylock_irqsave(&zone->lock, flags)) return NULL; } else {
@@ -5059,7 +5059,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,* Don't invoke should_fail logic, since it may call * get_random_u32() and printk() which need to spin_lock. */ -if (!(*alloc_flags &ALLOC_TRYLOCK ) && +if (!(*alloc_flags &ALLOC_NOLOCK ) &&should_fail_alloc_page (gfp_mask, order))return false;
@@ -7804,7 +7804,7 @@ static bool cond_accept_memory(struct zone *zone, unsigned int order,return false;/* Bailout, since try_to_accept_memory_one() needs to take a lock */ -if (alloc_flags &ALLOC_TRYLOCK ) +if (alloc_flags &ALLOC_NOLOCK )return false; wmark =promo_wmark_pages (zone);
@@ -7896,7 +7896,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned*/ gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | __GFP_COMP | gfp_flags; -unsigned int alloc_flags =ALLOC_TRYLOCK ; +unsigned int alloc_flags =ALLOC_NOLOCK ;struct alloc_context ac = { };struct page *page;
Signature · 1 line
2.54.0
Re: [PATCH v3 01/16] mm/page_alloc: rename ALLOC_TRYLOCK -> ALLOC_NOLOCK
Quoted 10 lines · Brendan Jackman
It's confusing that the function is called "nolock" but the flag is called "trylock", align them.
The function's terminology is more visible and has more mindshare so use that.
Suggested-by "Vlastimil Babka (SUSE)" <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Harry Yoo (Oracle) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
PATCH v3 2/16 mm/page_alloc: some renames to clarify alloc_flags scopes
In reply to Brendan JackmanIt's pretty confusing that:
- The slowpath and fastpath have a totally distinct set of alloc_flags.
- gfp_to_alloc_flags() sounds generic but it only influences the slowpath.
Rename some variables to highlight which alloc_flags are fastpath-specific. Rename gfp_to_alloc_flags() to highlight that it's slowpath-specific.
gfp_to_alloc_flags_cma() and gfp_to_alloc_flags_nonblocking() currently have perfectly harmless names, but to keep the naming consistent also rename those to the alloc_flags_*() pattern (which already exists for alloc_flags_nofragment()).
--- mm/page_alloc.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
mm/page_alloc.c +14 -14
@@ -3774,8 +3774,8 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) }/* Must be called after current_gfp_context() which can change gfp_mask */ -static inline unsigned int gfp_to_alloc_flags_cma (gfp_t gfp_mask, -unsigned int alloc_flags) +static inline unsigned int alloc_flags_cma (gfp_t gfp_mask, +unsigned int alloc_flags) {#ifdef CONFIG_CMA if (gfp_migratetype (gfp_mask) ==MIGRATE_MOVABLE )
@@ -4474,7 +4474,7 @@ static void wake_all_kswapds(unsigned int order, gfp_t gfp_mask, }static inline unsigned int -gfp_to_alloc_flags_nonblocking (gfp_t gfp_mask,unsigned int order) +alloc_flags_nonblocking (gfp_t gfp_mask,unsigned int order) {unsigned int alloc_flags =0 ;
@@ -4497,7 +4497,7 @@ gfp_to_alloc_flags_nonblocking(gfp_t gfp_mask, unsigned int order) }static inline unsigned int -gfp_to_alloc_flags (gfp_t gfp_mask,unsigned int order) +alloc_flags_slowpath (gfp_t gfp_mask,unsigned int order) {unsigned int alloc_flags =ALLOC_WMARK_MIN |ALLOC_CPUSET ;
@@ -4512,7 +4512,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)if (gfp_mask & __GFP_KSWAPD_RECLAIM) alloc_flags |=ALLOC_KSWAPD ; - alloc_flags |=gfp_to_alloc_flags_nonblocking (gfp_mask, order); + alloc_flags |=alloc_flags_nonblocking (gfp_mask, order);if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) {/*
@@ -4525,7 +4525,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) }else if (unlikely (rt_or_dl_task (current)) &&in_task ()) alloc_flags |=ALLOC_MIN_RESERVE ; - alloc_flags =gfp_to_alloc_flags_cma (gfp_mask, alloc_flags); + alloc_flags =alloc_flags_cma (gfp_mask, alloc_flags);if (defrag_mode) alloc_flags |=ALLOC_NOFRAGMENT ;
@@ -4791,7 +4791,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* kswapd needs to be woken up, and to avoid the cost of setting up * alloc_flags precisely. So we do that now. */ - alloc_flags =gfp_to_alloc_flags (gfp_mask, order); + alloc_flags =alloc_flags_slowpath (gfp_mask, order);/* * We need to recalculate the starting point for the zonelist iterator
@@ -4832,7 +4832,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags =__gfp_pfmemalloc_flags (gfp_mask);if (reserve_flags) - alloc_flags =gfp_to_alloc_flags_cma (gfp_mask, reserve_flags) | + alloc_flags =alloc_flags_cma (gfp_mask, reserve_flags) | (alloc_flags &ALLOC_KSWAPD );/*
@@ -5063,7 +5063,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, should_fail_alloc_page(gfp_mask, order))return false; - *alloc_flags =gfp_to_alloc_flags_cma (gfp_mask, *alloc_flags); + *alloc_flags =alloc_flags_cma (gfp_mask, *alloc_flags);/* Dirty zone balancing only done in the fast path */ ac->spread_dirty_pages = (gfp_mask & __GFP_WRITE);
@@ -5277,7 +5277,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { struct page *page; -unsigned int alloc_flags =ALLOC_WMARK_LOW ; +unsigned int fastpath_alloc_flags =ALLOC_WMARK_LOW ;gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ struct alloc_context ac = { };
@@ -5299,18 +5299,18 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, gfp =current_gfp_context (gfp); alloc_gfp = gfp;if (!prepare_alloc_pages (gfp, order, preferred_nid, nodemask, &ac, - &alloc_gfp, &alloc_flags)) + &alloc_gfp, &fastpath_alloc_flags))return NULL ;/* * Forbid the first pass from falling back to types that fragment * memory until all local zones are considered. */ - alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); - alloc_flags |=gfp_to_alloc_flags_nonblocking (gfp, order) &ALLOC_HIGHATOMIC ; + fastpath_alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); + fastpath_alloc_flags |=alloc_flags_nonblocking (gfp, order) &ALLOC_HIGHATOMIC ;/* First allocation attempt */ - page =get_page_from_freelist (alloc_gfp, order, alloc_flags, &ac); + page =get_page_from_freelist (alloc_gfp, order, fastpath_alloc_flags, &ac);if (likely (page)) goto out;
Signature · 1 line
2.54.0
Re: [PATCH v3 02/16] mm/page_alloc: some renames to clarify alloc_flags scopes
Quoted 7 lines · Brendan Jackman
It's pretty confusing that:
- The slowpath and fastpath have a totally distinct set of
alloc_flags.-
gfp_to_alloc_flags()sounds generic but it only influences the slowpath.
Quoted 8 lines
Rename some variables to highlight which
alloc_flagsare fastpath-specific. Renamegfp_to_alloc_flags()to highlight that it's slowpath-specific.
gfp_to_alloc_flags_cma()andgfp_to_alloc_flags_nonblocking()currently have perfectly harmless names, but to keep the naming consistent also rename those to the alloc_flags_*() pattern (which already exists foralloc_flags_nofragment()).
How annoying that alloc_flags_nofragment() doesn't have gfp as the first parameter, unlike others.
Oh well, must resist too much OCD :)
Uh, more annoyingly, alloc_flags_cma() takes alloc_flags and returns augmented alloc flags, so there's stuff like
*alloc_flags = alloc_flags_cma(gfp_mask, *alloc_flags);
Since we're unifying, it could be make to work additively like others? Then:
*alloc_flags |= alloc_flags_cma(gfp_mask);
Quoted 1 line
Signed-off-by Brendan Jackman <[email protected]>
Otherwise, LGTM.
Brendan Jackman
reviewerRe: [PATCH v3 02/16] mm/page_alloc: some renames to clarify alloc_flags scopes
Quoted 21 lines · Vlastimil Babka (SUSE)
Quoted 7 lines · Brendan Jackman
It's pretty confusing that:
- The slowpath and fastpath have a totally distinct set of
alloc_flags.-
gfp_to_alloc_flags()sounds generic but it only influences the slowpath.Quoted 8 lines
Rename some variables to highlight which
alloc_flagsare fastpath-specific. Renamegfp_to_alloc_flags()to highlight that it's slowpath-specific.
gfp_to_alloc_flags_cma()andgfp_to_alloc_flags_nonblocking()currently have perfectly harmless names, but to keep the naming consistent also rename those to the alloc_flags_*() pattern (which already exists foralloc_flags_nofragment()).How annoying that
alloc_flags_nofragment()doesn't have gfp as the first parameter, unlike others.
Oh well, must resist too much OCD :)
Quoted 8 lines
Uh, more annoyingly,
alloc_flags_cma()takesalloc_flagsand returns augmented alloc flags, so there's stuff like
*alloc_flags=alloc_flags_cma(gfp_mask, *alloc_flags);Since we're unifying, it could be make to work additively like others? Then:
*alloc_flags|=alloc_flags_cma(gfp_mask);
Sure, I can chuck this on as an extra patch.
Re: [PATCH v3 02/16] mm/page_alloc: some renames to clarify alloc_flags scopes
In reply to Brendan JackmanQuoted 44 lines · Brendan Jackman
It's pretty confusing that:
- The slowpath and fastpath have a totally distinct set of
alloc_flags.-
gfp_to_alloc_flags()sounds generic but it only influences the slowpath.Rename some variables to highlight which
alloc_flagsare fastpath-specific. Renamegfp_to_alloc_flags()to highlight that it's slowpath-specific.
gfp_to_alloc_flags_cma()andgfp_to_alloc_flags_nonblocking()currently have perfectly harmless names, but to keep the naming consistent also rename those to the alloc_flags_*() pattern (which already exists foralloc_flags_nofragment()).Signed-off-by Brendan Jackman <[email protected]>--- mm/page_alloc.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)mm/page_alloc.c +3
-3@@ -3774,8 +3774,8 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) }/* Must be called after current_gfp_context() which can change gfp_mask */ -static inline unsigned int gfp_to_alloc_flags_cma (gfp_t gfp_mask, -unsigned int alloc_flags) +static inline unsigned int alloc_flags_cma (gfp_t gfp_mask, +unsigned int alloc_flags) {#ifdef CONFIG_CMA if (gfp_migratetype (gfp_mask) ==MIGRATE_MOVABLE )@@ -4474,7 +4474,7 @@ static void wake_all_kswapds(unsigned int order, gfp_t gfp_mask, }static inline unsigned int -gfp_to_alloc_flags_nonblocking (gfp_t gfp_mask,unsigned int order) +alloc_flags_nonblocking (gfp_t gfp_mask,unsigned int order) {unsigned int alloc_flags =0 ;
[...]
PATCH v3 3/16 mm: name some args in a function declaration
In reply to Brendan JackmanCheckpatch complains about this, a later patch will move the code, fix it so that checkpatch doesn't complain about that patch. Do it in a separate patch so the "move the code" patch is trivial to review using Git's diff colouring.
--- mm/internal.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/internal.h +2 -2
@@ -919,8 +919,8 @@ extern bool free_pages_prepare(struct page *page, unsigned int order);extern int user_min_free_kbytes; -struct page *__alloc_frozen_pages_noprof (gfp_t ,unsigned int order,int nid, -nodemask_t *); +struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid, +nodemask_t *nodemask);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order);
Signature · 1 line
2.54.0
Re: [PATCH v3 03/16] mm: name some args in a function declaration
Quoted 6 lines · Brendan Jackman
Checkpatch complains about this, a later patch will move the code, fix it so that checkpatch doesn't complain about that patch. Do it in a separate patch so the "move the code" patch is trivial to review using Git's diff colouring.
Signed-off-by Brendan Jackman <[email protected]>
Quoted 20 lines
--- mm/internal.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)mm/internal.h +2
-2@@ -919,8 +919,8 @@ extern bool free_pages_prepare(struct page *page, unsigned int order);extern int user_min_free_kbytes; -struct page *__alloc_frozen_pages_noprof (gfp_t ,unsigned int order,int nid, -nodemask_t *); +struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid, +nodemask_t *nodemask);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order);
PATCH v3 4/16 mm: Split out internal page_alloc.h
In reply to Brendan Jackmaninternal.h is a bit bloated, seems like time for a page_alloc.h.
Where it wasn't obvious, the heuristic for deciding what goes into this new header was "does it support/correspond to a definition in mm/page_alloc.c?"
Only need to include it from 15 .c files out of 164 so this does seem like a genuine reduction in scopes, which is nice. And there's no circular internal.h<->page_alloc.h dependency, so it seems worthwhile to split this up before that inevitably emerges!
--- MAINTAINERS | 1 + mm/compaction.c | 1 + mm/hugetlb.c | 1 + mm/internal.h | 252 ----------------------------------------------- mm/khugepaged.c | 1 + mm/memory-failure.c | 1 + mm/memory_hotplug.c | 1 + mm/mempolicy.c | 1 + mm/mm_init.c | 1 + mm/page_alloc.c | 1 + mm/page_alloc.h | 269 +++++++++++++++++++++++++++++++++++++++++++++++++++ mm/page_frag_cache.c | 2 +- mm/page_isolation.c | 1 + mm/page_owner.c | 2 +- mm/show_mem.c | 1 + mm/slub.c | 1 + mm/swap.c | 1 + mm/vmscan.c | 1 + 18 files changed, 285 insertions(+), 254 deletions(-)
MAINTAINERS +1 -0
@@ -17171,6 +17171,7 @@ F: mm/debug_page_alloc.c F: mm/debug_page_ref.c F: mm/fail_page_alloc.c F: mm/page_alloc.c +F: mm/page_alloc.h F: mm/page_ext.c F: mm/page_frag_cache.c F: mm/page_isolation.c
mm/compaction.c +1 -0
@@ -24,6 +24,7 @@#include <linux/page_owner.h> #include <linux/psi.h> #include <linux/cpuset.h> +#include "page_alloc.h" #include "internal.h" #ifdef CONFIG_COMPACTION
mm/hugetlb.c +1 -0
@@ -47,6 +47,7 @@#include <linux/node.h> #include <linux/page_owner.h> #include "internal.h" +#include "page_alloc.h" #include "hugetlb_vmemmap.h" #include "hugetlb_cma.h" #include "hugetlb_internal.h"
mm/internal.h +0 -252
282 lines not on this page · raw message
mm/khugepaged.c +1 -0
@@ -26,6 +26,7 @@#include <asm/tlb.h> #include "internal.h" +#include "page_alloc.h" #include "mm_slot.h" enum scan_result {
mm/memory-failure.c +1 -0
@@ -66,6 +66,7 @@#include <trace/events/memory-failure.h> #include "swap.h" +#include "page_alloc.h" #include "internal.h" static int sysctl_memory_failure_early_kill__read_mostly ;
mm/memory_hotplug.c +1 -0
@@ -40,6 +40,7 @@#include <asm/tlbflush.h> #include "internal.h" +#include "page_alloc.h" #include "shuffle.h" enum {
mm/mempolicy.c +1 -0
@@ -119,6 +119,7 @@#include <linux/memory.h> #include "internal.h" +#include "page_alloc.h" /* Internal flags */ #define MPOL_MF_DISCONTIG_OK (MPOL_MF_INTERNAL << 0)/* Skip checks for continuous vmas */
mm/mm_init.c +1 -0
@@ -33,6 +33,7 @@#include <linux/kexec_handover.h> #include <linux/hugetlb.h> #include "internal.h" +#include "page_alloc.h" #include "slab.h" #include "shuffle.h"
mm/page_alloc.c +1 -0
@@ -56,6 +56,7 @@#include <linux/pgalloc_tag.h> #include <asm/div64.h> #include "internal.h" +#include "page_alloc.h" #include "shuffle.h" #include "page_reporting.h"
mm/page_alloc.h new file +269 -0
269 lines not on this page · raw message
mm/page_frag_cache.c +1 -1
@@ -18,7 +18,7 @@#include <linux/init.h> #include <linux/mm.h> #include <linux/page_frag_cache.h> -#include "internal.h" +#include "page_alloc.h" static unsigned long encoded_page_create (struct page *page,unsigned int order,bool pfmemalloc)
mm/page_isolation.c +1 -0
@@ -11,6 +11,7 @@#include <linux/page_owner.h> #include <linux/migrate.h> #include "internal.h" +#include "page_alloc.h" #define CREATE_TRACE_POINTS #include <trace/events/page_isolation.h>
mm/page_owner.c +1 -1
@@ -13,7 +13,7 @@#include <linux/memcontrol.h> #include <linux/sched/clock.h> -#include "internal.h" +#include "page_alloc.h" /* * TODO: teach PAGE_OWNER_STACK_DEPTH (__dump_page_owner and save_stack)
mm/show_mem.c +1 -0
@@ -16,6 +16,7 @@#include <linux/vmstat.h> #include "internal.h" +#include "page_alloc.h" #include "swap.h" atomic_long_t _totalram_pages__read_mostly ;
mm/slub.c +1 -0
@@ -53,6 +53,7 @@#include <trace/events/kmem.h> #include "internal.h" +#include "page_alloc.h" /* * Lock order:
mm/swap.c +1 -0
@@ -39,6 +39,7 @@#include <linux/buffer_head.h> #include "internal.h" +#include "page_alloc.h" #define CREATE_TRACE_POINTS #include <trace/events/pagemap.h>
mm/vmscan.c +1 -0
@@ -66,6 +66,7 @@#include <linux/sched/sysctl.h> #include "internal.h" +#include "page_alloc.h" #include "swap.h" #define CREATE_TRACE_POINTS
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v3 04/16] mm: Split out internal page_alloc.h
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Missing inclusion of `page_alloc.h` causes build failures for multiple configurations. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Missing inclusion of page_alloc.h causes build failures for multiple configurations. --
commit 34a7c3e538f8ec840a6b68715addef408e469ff0
Author: Brendan Jackman <[email protected]>
mm: Split out internal page_alloc.h
internal.h is a bit bloated, seems like time for a page_alloc.h. This moves definitions supporting mm/page_alloc.c into the new header.
Quoted 4 lines
mm/internal.h +0
-0
[ ... ]
mm/internal.h extern int defrag_mode; · line 708
Quoted 5 lines
mm/internal.h +0
-5-static inline unsigned int buddy_order (struct page *page) -{ -/* PageBuddy() must be checked by the caller */ -return page_private (page); -}
[Severity: Low]
Will this cause a build failure when shuffle page allocator or page reporting features are enabled?
It looks like shuffle_valid_page() in mm/shuffle.c and page_reporting_drain() in mm/page_reporting.c use buddy_order() but don't include the new page_alloc.h.
[ ... ]
mm/internal.h extern int __isolate_free_page(struct page *page, unsigned int order); · line 835
Quoted 2 lines
mm/internal.h +0
-2-extern void __free_pages_core (struct page *page,unsigned int order, -enum meminit_context context);
[Severity: Low]
Does mm/kmsan/init.c need to include page_alloc.h now?
It seems do_collection() uses __free_pages_core() and might fail to build when the kernel is configured for kmsan if the header is missing.
[ ... ]
Quoted 1 line
-#define
ALLOC_CMA0x80/* allow allocations from CMA areas */
[Severity: Low]
Could this break the build for migration and page reporting configurations?
I am noticing that migrate_balanced_pgdat() in mm/migrate.c and page_reporting_process_zone() in mm/page_reporting.c use ALLOC_CMA but neither of these files appears to include page_alloc.h.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Re: [PATCH v3 04/16] mm: Split out internal page_alloc.h
In reply to Brendan JackmanQuoted 6 lines · Brendan Jackman
internal.his a bit bloated, seems like time for apage_alloc.h.Where it wasn't obvious, the heuristic for deciding what goes into this new header was "does it support/correspond to a definition in
mm/page_alloc.c?"
Quoted 8 lines
Only need to include it from 15 .c files out of 164 so this does seem like a genuine reduction in scopes, which is nice. And there's no circular internal.h<->page_alloc.h dependency, so it seems worthwhile to split this up before that inevitably emerges!
Cool.
PATCH v3 5/16 mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
In reply to Brendan JackmanCurrently the core allocator code is controlled by ALLOC_NOLOCK, but the main entry point function is significantly different from the normal __alloc_frozen_pages_nolock(), this is tiring when reading the code.
Plumb the ALLOC_NOLOCK control one layer up in the call stack: create an alloc_flags argument to __alloc_frozen_pages_nolock() (which is only exposed to mm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handling NUMA_NO_NODE, similar to how some of the wrappers in gfp.h do).
Rationale that this doesn't change anything:
1. Simple bits: A bunch of the nolock-specific handling is just moved to the new alloc_order_allowed(), alloc_trylock_allowed() and gfp_trylock.
2. __alloc_frozen_pages_noprof() has some extra logic that wasn't previously in the nolock variant:
a. Application of gfp_allowed_mask; this only affects early boot, and
only flags that affect the slowpath get changed here.
b. Application of current_gfp_context() - also only affects the
slowpath
3. The slowpath itself: this is now just explicitly skipped under !ALLOC_TRYLOCK.
Ulterior motive: adding an alloc_flags arg to the allocator's mm-internal entrypoint can later be used to do more allocation customisation without needing to create new GFP flags.
While adding this flag to a bunch of places, create ALLOC_DEFAULT to avoid a mysterious literal 0 in most places. alloc_frozen_pages_noprof() is defined above the alloc flags so just leave that as a slightly messy exception instead of trying to fully reorder mm/internal.h for that one case.
No functional change intended.
--- mm/hugetlb.c | 3 +- mm/mempolicy.c | 10 ++-- mm/page_alloc.c | 178 +++++++++++++++++++++++++++++--------------------------- mm/page_alloc.h | 6 +- mm/slub.c | 6 +- 5 files changed, 108 insertions(+), 95 deletions(-)
mm/hugetlb.c +2 -1
@@ -1806,7 +1806,8 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,if (alloc_try_hard) gfp_mask |= __GFP_RETRY_MAYFAIL; - folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask); + folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask, +ALLOC_DEFAULT );/* * If we did not specify __GFP_RETRY_MAYFAIL, but still got a
mm/mempolicy.c +6 -4
@@ -2426,9 +2426,11 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,*/ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page =__alloc_frozen_pages_noprof (preferred_gfp, order, nid, nodemask); + page =__alloc_frozen_pages_noprof (preferred_gfp, order, nid, nodemask, +ALLOC_DEFAULT );if (!page) - page =__alloc_frozen_pages_noprof (gfp, order, nid,NULL ); + page =__alloc_frozen_pages_noprof (gfp, order, nid,NULL , +ALLOC_DEFAULT );return page; }
@@ -2476,7 +2478,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,*/ page =__alloc_frozen_pages_noprof ( gfp | __GFP_THISNODE | __GFP_NORETRY, order, - nid,NULL ); + nid,NULL ,ALLOC_DEFAULT );if (page || !(gfp & __GFP_DIRECT_RECLAIM))return page;/*
@@ -2488,7 +2490,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } } - page =__alloc_frozen_pages_noprof (gfp, order, nid, nodemask); + page =__alloc_frozen_pages_noprof (gfp, order, nid, nodemask,ALLOC_DEFAULT );if (unlikely (pol->mode ==MPOL_INTERLEAVE || pol->mode ==MPOL_WEIGHTED_INTERLEAVE ) && page) {
mm/page_alloc.c +92 -86
226 lines not on this page · raw message
mm/page_alloc.h +4 -2
@@ -11,6 +11,7 @@#include <linux/nodemask.h> #include <linux/types.h> +#define ALLOC_DEFAULT 0/* The ALLOC_WMARK bits are used as an index to zone->watermark */ #define ALLOC_WMARK_MIN WMARK_MIN#define ALLOC_WMARK_LOW WMARK_LOW
@@ -219,7 +220,7 @@ extern bool free_pages_prepare(struct page *page, unsigned int order);extern int user_min_free_kbytes;struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid, -nodemask_t *nodemask); +nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order);
@@ -230,7 +231,8 @@ struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order);#else static inline struct page *alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order) { -return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL ); +return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL , +0 /* ALLOC_DEFAULT */ ); }#endif
mm/slub.c +4 -2
@@ -3292,7 +3292,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,else if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages (flags, order);else - page =__alloc_frozen_pages (flags, order, node,NULL ); + page =__alloc_frozen_pages (flags, order, node,NULL , +ALLOC_DEFAULT );if (!page)return NULL ;
@@ -5302,7 +5303,8 @@ static void *___kmalloc_large_node(size_t size, gfp_t flags, int node)if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages_noprof (flags, order);else - page =__alloc_frozen_pages_noprof (flags, order, node,NULL ); + page =__alloc_frozen_pages_noprof (flags, order, node,NULL , +ALLOC_DEFAULT );if (page) { ptr =page_address (page);
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Spurious allocation failures for ALLOC_NOLOCK on fragmented systems --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Spurious allocation failures for ALLOC_NOLOCK on fragmented systems --
commit e9001ddaf8c952ebe08e5042f720b09e0b4a5da8
Author: Brendan Jackman <[email protected]>
mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
This commit unifies the core allocator entry points by pulling the ALLOC_NOLOCK control up into __alloc_frozen_pages_noprof(). It turns the nolock variant into a thin wrapper that just sets the ALLOC_NOLOCK flag and handles NUMA_NO_NODE.
Quoted 4 lines
mm/page_alloc.c +0
-0
[ ... ]
mm/page_alloc.c struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, · line 5390
Quoted 10 lines
mm/page_alloc.c +2
-2@@ -5310,9 +5384,9 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, fastpath_alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); fastpath_alloc_flags |=alloc_flags_nonblocking (gfp, order) &ALLOC_HIGHATOMIC ; -/* First allocation attempt */ +/* First allocation attempt (or, for nolock, only attempt) */ page =get_page_from_freelist (alloc_gfp, order, fastpath_alloc_flags, &ac); -if (likely (page)) +if (likely (page) || (alloc_flags &ALLOC_NOLOCK )) goto out;
[Severity: High]
Could this lead to spurious allocation failures for ALLOC_NOLOCK requests on fragmented systems?
In the pre-patch code, alloc_frozen_pages_nolock_noprof() bypassed alloc_flags_nofragment(), allowing it to successfully steal pages from other migratetypes when its preferred migratetype was empty.
The new unified __alloc_frozen_pages_noprof() fastpath unconditionally applies the ALLOC_NOFRAGMENT policy to fastpath_alloc_flags. When the requested migratetype is empty, __rmqueue() respects this flag by skipping __rmqueue_steal() and returning NULL.
Since normal allocations fall back to the slowpath (which clears ALLOC_NOFRAGMENT and retries), they succeed. However, ALLOC_NOLOCK explicitly bails out after the first attempt here, permanently failing the allocation. This might cause systems using bpf or tracing facilities under memory fragmentation to silently drop allocations when hit from atomic contexts.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Re: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
Quoted 37 lines · On Mon, 29 Jun 2026 at 16:29, <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Spurious allocation failures forALLOC_NOLOCKon fragmented systems --commit
e9001ddaf8c952ebe08e5042f720b09e0b4a5da8
Author: Brendan Jackman <[email protected]>
mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()This commit unifies the core allocator entry points by pulling the
ALLOC_NOLOCKcontrol up into__alloc_frozen_pages_noprof(). It turns the nolock variant into a thin wrapper that just sets theALLOC_NOLOCKflag and handlesNUMA_NO_NODE.Quoted 4 lines
mm/page_alloc.c +0
-0[ ... ]
mm/page_alloc.c struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, · line 5390
Quoted 10 lines
mm/page_alloc.c +2
-2@@ -5310,9 +5384,9 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, fastpath_alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); fastpath_alloc_flags |=alloc_flags_nonblocking (gfp, order) &ALLOC_HIGHATOMIC ; -/* First allocation attempt */ +/* First allocation attempt (or, for nolock, only attempt) */ page =get_page_from_freelist (alloc_gfp, order, fastpath_alloc_flags, &ac); -if (likely (page)) +if (likely (page) || (alloc_flags &ALLOC_NOLOCK )) goto out;[Severity: High]
Could this lead to spurious allocation failures forALLOC_NOLOCKrequests on fragmented systems?
Quoted 3 lines
In the pre-patch code,
alloc_frozen_pages_nolock_noprof()bypassedalloc_flags_nofragment(), allowing it to successfully steal pages from other migratetypes when its preferred migratetype was empty.
Oh, yep. Given "no functional change intended" this should not set ALLOC_NOFRAGMENT when ac.alloc_flags has ALLOC_NOLOCK.
(Is this intentional? Do we really want to allow "opportunistic" allocations, which are very much allowed to fail, to fragment memory? It doesn't look like this was discussed at when the nolock functionality was added).
Harry Yoo
maintainerRe: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
In reply to Brendan JackmanQuoted 10 lines · Brendan Jackman
Currently the core allocator code is controlled by
ALLOC_NOLOCK, but the main entry point function is significantly different from the normal__alloc_frozen_pages_nolock(), this is tiring when reading the code.Plumb the
ALLOC_NOLOCKcontrol one layer up in the call stack: create analloc_flagsargument to__alloc_frozen_pages_nolock()(which is only exposed tomm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handlingNUMA_NO_NODE, similar to how some of the wrappers ingfp.hdo).
Quoted 5 lines
Rationale that this doesn't change anything:
1. Simple bits: A bunch of the nolock-specific handling is just moved to the new
alloc_order_allowed(),alloc_trylock_allowed()andgfp_trylock.
Right.
Quoted 5 lines
2.
__alloc_frozen_pages_noprof()has some extra logic that wasn't previously in the nolock variant:a. Application of
gfp_allowed_mask; this only affects early boot, and
only flags that affect the slowpath get changed here.
gfp_allowed_mask clears __GFP_RECLAIM, and that means now allocations with GFP_KERNEL during early boot would see gfpflags_allow_spinning() = false.
The helper is not used in in the page allocator, but used in memcg/stackdepot/page_owner.
Quoted 2 lines
b. Application of
current_gfp_context()- also only affects the
slowpath
PF_MEMALLOC_PIN affects the fast path, but ALLOC_NOLOCK users won't be affected.
What about alloc_flags_nofragment/nonblocking()?
Quoted 2 lines
3. The slowpath itself: this is now just explicitly skipped under !ALLOC_TRYLOCK.
Right.
Quoted 8 lines
Ulterior motive: adding an
alloc_flagsarg to the allocator's mm-internal entrypoint can later be used to do more allocation customisation without needing to create new GFP flags.While adding this flag to a bunch of places, create
ALLOC_DEFAULTto avoid a mysterious literal 0 in most places.
alloc_frozen_pages_noprof()is defined above the alloc flags
The function is defined below the alloc flags, no?
Quoted 32 lines
so just leave that as a slightly messy
exception instead of trying to fully reordermm/internal.hfor that one case.No functional change intended.
Signed-off-by Brendan Jackman <[email protected]>--- mm/hugetlb.c | 3 +- mm/mempolicy.c | 10 ++-- mm/page_alloc.c | 178 +++++++++++++++++++++++++++++--------------------------- mm/page_alloc.h | 6 +- mm/slub.c | 6 +- 5 files changed, 108 insertions(+), 95 deletions(-)mm/page_alloc.c +9
-0@@ -5271,24 +5271,98 @@ void free_pages_bulk(struct page **page_array, unsigned long nr_pages) } } +static inline bool alloc_trylock_allowed (void ) +{ +/* +* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is +* unsafe in NMI. If spin_trylock() is called from hard IRQ the current +* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will +* mark the task as the owner of another rt_spin_lock which will +* confuse PI logic, so return immediately if called from hard IRQ or +* NMI.
mm/page_alloc.c void free_pages_bulk(struct page **page_array, unsigned long nr_pages) · line 5292
Quoted 10 lines
mm/page_alloc.c +10
-0+* +* Note, irqs_disabled() case is ok. This function can be called +* from raw_spin_lock_irqsave region. +*/ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) +return false; + +/* On UP, spin_trylock() always succeeds even when it is locked */ +if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +return false;
Except for deferred_pages_enabled(), it's not specific to the page allocator. SLUB has
/* * See the comment for the same check in * alloc_frozen_pages_nolock_noprof() */
... and repeats the same thing as above.
Perhaps let's factor it out into a helper
rather than trying not to forget to update the other place?
Quoted 6 lines
mm/page_alloc.c +6
-0+/* Bailout, since _deferred_grow_zone() needs to take a lock */ +if (deferred_pages_enabled ()) +return false; + +return true; +}
Signature · 2 lines
Cheers, Harry / Hyeonggon
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
Quoted 24 lines · Harry Yoo
Quoted 10 lines · Brendan Jackman
Currently the core allocator code is controlled by
ALLOC_NOLOCK, but the main entry point function is significantly different from the normal__alloc_frozen_pages_nolock(), this is tiring when reading the code.Plumb the
ALLOC_NOLOCKcontrol one layer up in the call stack: create analloc_flagsargument to__alloc_frozen_pages_nolock()(which is only exposed tomm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handlingNUMA_NO_NODE, similar to how some of the wrappers ingfp.hdo).Quoted 5 lines
Rationale that this doesn't change anything:
1. Simple bits: A bunch of the nolock-specific handling is just moved to the new
alloc_order_allowed(),alloc_trylock_allowed()andgfp_trylock.Right.
Quoted 3 lines
2.
__alloc_frozen_pages_noprof()has some extra logic that wasn't previously in the nolock variant:
Quoted 6 lines
Quoted 2 lines
a. Application of
gfp_allowed_mask; this only affects early boot, and
only flags that affect the slowpath get changed here.
gfp_allowed_maskclears__GFP_RECLAIM, and that means now allocations withGFP_KERNELduring early boot would seegfpflags_allow_spinning()= false.
Is it a problem though? non-nolock allocations were affected before (the masking existed for those already) and will be affected now the same, and _nolock() allocations don't pass __GFP_RECLAIM in the first place, so the masking can't affect them?
Quoted 8 lines
The helper is not used in in the page allocator, but used in memcg/stackdepot/page_owner.
Quoted 2 lines
b. Application of
current_gfp_context()- also only affects the
slowpath
PF_MEMALLOC_PINaffects the fast path, butALLOC_NOLOCKusers won't be affected.
And it wouldn't be wrong if they were? It only clears __GFP_MOVABLE?
Quoted 1 line
What about alloc_flags_nofragment/nonblocking()?
ALLOC_NOFRAGMENT due to e.g. defrag_mode could be a problem indeed, if there's no slowpath. Make ALLOC_NOLOCK override it?
nonblocking() is probably fine?
Quoted 1 line
Brendan Jackman
reviewerRe: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
Quoted 28 lines · Vlastimil Babka (SUSE)
Quoted 27 lines · Harry Yoo
Quoted 10 lines · Brendan Jackman
Currently the core allocator code is controlled by
ALLOC_NOLOCK, but the main entry point function is significantly different from the normal__alloc_frozen_pages_nolock(), this is tiring when reading the code.Plumb the
ALLOC_NOLOCKcontrol one layer up in the call stack: create analloc_flagsargument to__alloc_frozen_pages_nolock()(which is only exposed tomm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handlingNUMA_NO_NODE, similar to how some of the wrappers ingfp.hdo).Quoted 5 lines
Rationale that this doesn't change anything:
1. Simple bits: A bunch of the nolock-specific handling is just moved to the new
alloc_order_allowed(),alloc_trylock_allowed()andgfp_trylock.Right.
Quoted 5 lines
2.
__alloc_frozen_pages_noprof()has some extra logic that wasn't previously in the nolock variant:a. Application of
gfp_allowed_mask; this only affects early boot, and
only flags that affect the slowpath get changed here.
Quoted 8 lines
Quoted 3 lines
gfp_allowed_maskclears__GFP_RECLAIM, and that means now allocations withGFP_KERNELduring early boot would seegfpflags_allow_spinning()= false.Is it a problem though? non-nolock allocations were affected before (the masking existed for those already) and will be affected now the same, and
_nolock()allocations don't pass__GFP_RECLAIMin the first place, so the masking can't affect them?
This was my thinking too.
Quoted 9 lines
Quoted 8 lines
The helper is not used in in the page allocator, but used in memcg/stackdepot/page_owner.
Quoted 2 lines
b. Application of
current_gfp_context()- also only affects the
slowpath
PF_MEMALLOC_PINaffects the fast path, butALLOC_NOLOCKusers won't be affected.
Quoted 6 lines
And it wouldn't be wrong if they were? It only clears
__GFP_MOVABLE?Quoted 1 line
What about alloc_flags_nofragment/nonblocking()?
ALLOC_NOFRAGMENTdue to e.g.defrag_modecould be a problem indeed, if there's no slowpath. MakeALLOC_NOLOCKoverride it?
Yeah calling alloc_flags_nofragment() here is a bug in the patch, and Sashiko also complained:
https://lore.kernel.org/all/[email protected]/
Like I said in the reply to that thread I think maybe we _do_ want to set ALLOC_NOFRAGMENT for nolock allocations? But, that is a functional change, it doesn't belong in this series.
Quoted 1 line
nonblocking()is probably fine?
Yeah, I believe this is fine.
Harry Yoo
maintainerRe: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
Quoted 33 lines · Brendan Jackman
Quoted 32 lines · Vlastimil Babka (SUSE)
Quoted 30 lines · Harry Yoo
Quoted 10 lines · Brendan Jackman
Currently the core allocator code is controlled by
ALLOC_NOLOCK, but the main entry point function is significantly different from the normal__alloc_frozen_pages_nolock(), this is tiring when reading the code.Plumb the
ALLOC_NOLOCKcontrol one layer up in the call stack: create analloc_flagsargument to__alloc_frozen_pages_nolock()(which is only exposed tomm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handlingNUMA_NO_NODE, similar to how some of the wrappers ingfp.hdo).Quoted 5 lines
Rationale that this doesn't change anything:
1. Simple bits: A bunch of the nolock-specific handling is just moved to the new
alloc_order_allowed(),alloc_trylock_allowed()andgfp_trylock.Right.
Quoted 5 lines
2.
__alloc_frozen_pages_noprof()has some extra logic that wasn't previously in the nolock variant:a. Application of
gfp_allowed_mask; this only affects early boot, and
only flags that affect the slowpath get changed here.
gfp_allowed_maskclears__GFP_RECLAIM, and that means now allocations withGFP_KERNELduring early boot would seegfpflags_allow_spinning()= false.
Quoted 6 lines
Quoted 4 lines
Is it a problem though? non-nolock allocations were affected before (the masking existed for those already) and will be affected now the same, and
_nolock()allocations don't pass__GFP_RECLAIMin the first place, so the masking can't affect them?This was my thinking too.
Oh! You guys are right :) I was confused.
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
The helper is not used in in the page allocator, but used in memcg/stackdepot/page_owner.
Quoted 2 lines
b. Application of
current_gfp_context()- also only affects the
slowpath
PF_MEMALLOC_PINaffects the fast path, butALLOC_NOLOCKusers won't be affected.And it wouldn't be wrong if they were? It only clears
__GFP_MOVABLE?
Nothing wrong, but wanted to mention things that were not mentioned in the changelog as it argues there's no functional change intended.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
What about alloc_flags_nofragment/nonblocking()?
ALLOC_NOFRAGMENTdue to e.g.defrag_modecould be a problem indeed, if there's no slowpath. MakeALLOC_NOLOCKoverride it?
Quoted 8 lines
Yeah calling
alloc_flags_nofragment()here is a bug in the patch, and Sashiko also complained:https://lore.kernel.org/all/[email protected]/
Like I said in the reply to that thread I think maybe we _do_ want to set
ALLOC_NOFRAGMENTfor nolock allocations? But, that is a functional change, it doesn't belong in this series.
I don't think it was intentional to bypass ALLOC_NOFRAGMENT for nolock allocations. But yeah that's a functional change.
Quoted 3 lines
Quoted 1 line
nonblocking()is probably fine?Yeah, I believe this is fine.
Agreed.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Brendan Jackman
reviewerRe: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
In reply to Harry YooQuoted 10 lines · Harry Yoo
Quoted 8 lines
Ulterior motive: adding an
alloc_flagsarg to the allocator's mm-internal entrypoint can later be used to do more allocation customisation without needing to create new GFP flags.While adding this flag to a bunch of places, create
ALLOC_DEFAULTto avoid a mysterious literal 0 in most places.
alloc_frozen_pages_noprof()is defined above the alloc flagsThe function is defined below the alloc flags, no?
Yep this paragraph is stale since I created mm/page_alloc.h, will remove it.
Quoted 51 lines
Quoted 32 lines
so just leave that as a slightly messy
exception instead of trying to fully reordermm/internal.hfor that one case.No functional change intended.
Signed-off-by Brendan Jackman <[email protected]>--- mm/hugetlb.c | 3 +- mm/mempolicy.c | 10 ++-- mm/page_alloc.c | 178 +++++++++++++++++++++++++++++--------------------------- mm/page_alloc.h | 6 +- mm/slub.c | 6 +- 5 files changed, 108 insertions(+), 95 deletions(-)mm/page_alloc.c +9
-0@@ -5271,24 +5271,98 @@ void free_pages_bulk(struct page **page_array, unsigned long nr_pages) } } +static inline bool alloc_trylock_allowed (void ) +{ +/* +* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is +* unsafe in NMI. If spin_trylock() is called from hard IRQ the current +* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will +* mark the task as the owner of another rt_spin_lock which will +* confuse PI logic, so return immediately if called from hard IRQ or +* NMI. mm/page_alloc.c void free_pages_bulk(struct page **page_array, unsigned long nr_pages) · line 5292
Quoted 10 lines
mm/page_alloc.c +10
-0+* +* Note, irqs_disabled() case is ok. This function can be called +* from raw_spin_lock_irqsave region. +*/ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) +return false; + +/* On UP, spin_trylock() always succeeds even when it is locked */ +if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +return false;Except for
deferred_pages_enabled(), it's not specific to the page allocator. SLUB has/* * See the comment for the same check in * alloc_frozen_pages_nolock_noprof() */
Quoted 4 lines
... and repeats the same thing as above.
Perhaps let's factor it out into a helper
rather than trying not to forget to update the other place?
Hm, not sure about this. I think I would say it's a "coincidence" that these two bits of code look the same? Like, page_alloc.c uses spin_trylock() so you can't do alloc_pages_nolock() from IRQ on PREEMPT_RT. slub.c ALSO uses spin_trylock(), so you ALSO can't use kmalloc_nolock() in those scenarios. But those are two different facts that just happen to be isomorphic? Putting them into a shared helper would kinda imply that these are part of a single system with inherently coupled constraints.
I dunno I'm being a bit of a ponderous philosopher there, I don't have particularly strong feelings. But I'd lean towards leaving this out of the patchset since the potential deduplication isn't really related to the other cleanups anyway.
Harry Yoo
maintainerRe: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
Quoted 41 lines · Brendan Jackman
Quoted 36 lines
Quoted 17 lines
mm/page_alloc.c +9
-0@@ -5271,24 +5271,98 @@ void free_pages_bulk(struct page **page_array, unsigned long nr_pages) } } +static inline bool alloc_trylock_allowed (void ) +{ +/* +* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is +* unsafe in NMI. If spin_trylock() is called from hard IRQ the current +* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will +* mark the task as the owner of another rt_spin_lock which will +* confuse PI logic, so return immediately if called from hard IRQ or +* NMI. mm/page_alloc.c void free_pages_bulk(struct page **page_array, unsigned long nr_pages) · line 5292
Quoted 10 lines
mm/page_alloc.c +10
-0+* +* Note, irqs_disabled() case is ok. This function can be called +* from raw_spin_lock_irqsave region. +*/ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) +return false; + +/* On UP, spin_trylock() always succeeds even when it is locked */ +if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +return false;Except for
deferred_pages_enabled(), it's not specific to the page allocator. SLUB has/* * See the comment for the same check in * alloc_frozen_pages_nolock_noprof() */Quoted 4 lines
... and repeats the same thing as above.
Perhaps let's factor it out into a helper
rather than trying not to forget to update the other place?
Quoted 8 lines
Hm, not sure about this. I think I would say it's a "coincidence" that these two bits of code look the same? Like,
page_alloc.cusesspin_trylock()so you can't doalloc_pages_nolock()from IRQ onPREEMPT_RT.slub.cALSO usesspin_trylock(), so you ALSO can't usekmalloc_nolock()in those scenarios. But those are two different facts that just happen to be isomorphic? Putting them into a shared helper would kinda imply that these are part of a single system with inherently coupled constraints.
But as long as they use spinlocks and can be called in unknown contexts, they are supposed to have the same constraint?
And actually, the only reason free_pages_nolock() or kfree_nolock() don't have these checks is just because we don't allow kmalloc() -> kfree_nolock() or alloc_pages() -> free_pages_nolock(). Once we allow them, we'll need to repeat this again.
I think it doesn't even belong page/slab allocators, probably should be spin_trylock_allowed()?
Quoted 1 line
But I'd lean towards leaving this out of> the patchset since the
potential deduplication isn't really related to
Quoted 1 line
the other cleanups anyway.
Ack.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Brendan Jackman
reviewerRe: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
Quoted 56 lines · Harry Yoo
Quoted 41 lines · Brendan Jackman
Quoted 36 lines
Quoted 17 lines
mm/page_alloc.c +9
-0@@ -5271,24 +5271,98 @@ void free_pages_bulk(struct page **page_array, unsigned long nr_pages) } } +static inline bool alloc_trylock_allowed (void ) +{ +/* +* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is +* unsafe in NMI. If spin_trylock() is called from hard IRQ the current +* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will +* mark the task as the owner of another rt_spin_lock which will +* confuse PI logic, so return immediately if called from hard IRQ or +* NMI. mm/page_alloc.c void free_pages_bulk(struct page **page_array, unsigned long nr_pages) · line 5292
Quoted 10 lines
mm/page_alloc.c +10
-0+* +* Note, irqs_disabled() case is ok. This function can be called +* from raw_spin_lock_irqsave region. +*/ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) +return false; + +/* On UP, spin_trylock() always succeeds even when it is locked */ +if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +return false;Except for
deferred_pages_enabled(), it's not specific to the page allocator. SLUB has/* * See the comment for the same check in * alloc_frozen_pages_nolock_noprof() */Quoted 4 lines
... and repeats the same thing as above.
Perhaps let's factor it out into a helper
rather than trying not to forget to update the other place?Quoted 8 lines
Hm, not sure about this. I think I would say it's a "coincidence" that these two bits of code look the same? Like,
page_alloc.cusesspin_trylock()so you can't doalloc_pages_nolock()from IRQ onPREEMPT_RT.slub.cALSO usesspin_trylock(), so you ALSO can't usekmalloc_nolock()in those scenarios. But those are two different facts that just happen to be isomorphic? Putting them into a shared helper would kinda imply that these are part of a single system with inherently coupled constraints.But as long as they use spinlocks and can be called in unknown contexts, they are supposed to have the same constraint?
Quoted 7 lines
And actually, the only reason
free_pages_nolock()orkfree_nolock()don't have these checks is just because we don't allowkmalloc()->kfree_nolock()oralloc_pages()->free_pages_nolock(). Once we allow them, we'll need to repeat this again.I think it doesn't even belong page/slab allocators, probably should be
spin_trylock_allowed()?
You are right. I was imagining a function called alloc_nolock_allowed() but actually now I see the name spin_trylock_allowed() I realise it's not about the allocators (which both just happen to use spin_trylock()) at all, it makes perfect sense.
Quoted 5 lines
Quoted 1 line
But I'd lean towards leaving this out of> the patchset since the
potential deduplication isn't really related to
Quoted 1 line
the other cleanups anyway.
Ack.
Patchset is getting pretty big already but most stuff is already acked and also most of the patches are trivial so maybe I'll tack it on after all...
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
In reply to Brendan JackmanQuoted 2 lines · Brendan Jackman
Currently the core allocator code is controlled by
ALLOC_NOLOCK, but the main entry point function is significantly different from the normal
Let's mention it explicitly, alloc_frozen_pages_nolock_noprof().
Quoted 1 line
__alloc_frozen_pages_nolock(), this is tiring when reading the code.
You mean __alloc_frozen_pages_noprof()?
Quoted 3 lines
Plumb the
ALLOC_NOLOCKcontrol one layer up in the call stack: create analloc_flagsargument to__alloc_frozen_pages_nolock()(which is only
Again __alloc_frozen_pages_noprof()
Quoted 9 lines
exposed to
mm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handlingNUMA_NO_NODE, similar to how some of the wrappers ingfp.hdo).Rationale that this doesn't change anything:
1. Simple bits: A bunch of the nolock-specific handling is just moved to the new
alloc_order_allowed(),alloc_trylock_allowed()andgfp_trylock.
Should be alloc_nolock_allowed() and gfp_nolock
Quoted 5 lines
2.
__alloc_frozen_pages_noprof()has some extra logic that wasn't previously in the nolock variant:a. Application of
gfp_allowed_mask; this only affects early boot, and
only flags that affect the slowpath get changed here.
As discussed in reply to Harry, I'd mention the flags excluded by GFP_BOOT_MASK are not usable by _nolock() anyway.
Quoted 5 lines
b. Application of
current_gfp_context()- also only affects the
slowpath3. The slowpath itself: this is now just explicitly skipped under !ALLOC_TRYLOCK.
ALLOC_NOLOCK.
Quoted 5 lines
Ulterior motive: adding an
alloc_flagsarg to the allocator's mm-internal entrypoint can later be used to do more allocation customisation without needing to create new GFP flags.
Quoted 8 lines
While adding this flag to a bunch of places, create
ALLOC_DEFAULTto avoid a mysterious literal 0 in most places.
alloc_frozen_pages_noprof()
is defined above the alloc flags so just leave that as a slightly messy exception instead of trying to fully reordermm/internal.hfor that one case.
This no longer applies in v3?
Quoted 45 lines
No functional change intended.
Signed-off-by Brendan Jackman <[email protected]>--- mm/hugetlb.c | 3 +- mm/mempolicy.c | 10 ++-- mm/page_alloc.c | 178 +++++++++++++++++++++++++++++--------------------------- mm/page_alloc.h | 6 +- mm/slub.c | 6 +- 5 files changed, 108 insertions(+), 95 deletions(-)mm/hugetlb.c +0
-0mm/page_alloc.c +5
-10@@ -5222,7 +5222,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, } nr_account++; -prep_new_page (page,0 , gfp,0 ); +prep_new_page (page,0 , gfp,ALLOC_DEFAULT );set_page_refcounted (page); page_array[nr_populated++] = page; }@@ -5271,24 +5271,98 @@ void free_pages_bulk(struct page **page_array, unsigned long nr_pages) } } -/* -* This is the 'heart' of the zoned buddy allocator. -*/ -struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order, -int preferred_nid,nodemask_t *nodemask) +static inline bool alloc_order_allowed (gfp_t gfp,unsigned int order, +unsigned int alloc_flags) { -struct page *page; -unsigned int fastpath_alloc_flags =ALLOC_WMARK_LOW ; -gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ -struct alloc_context ac = { }; +if (alloc_flags &ALLOC_NOLOCK ) +return pcp_allowed_order (order);
mm/page_alloc.c void free_pages_bulk(struct page **page_array, unsigned long nr_pages) · line 5287
Quoted 10 lines
mm/page_alloc.c +4
-1/* * There are several places where we assume that the order value is sane * so bail out early if the request is out of bound. */ -if (WARN_ON_ONCE_GFP (order >MAX_PAGE_ORDER , gfp)) +return !(WARN_ON_ONCE_GFP (order >MAX_PAGE_ORDER , gfp)); +} + +static inline bool alloc_trylock_allowed (void )
alloc_nolock_allowed()
Quoted 64 lines
mm/page_alloc.c +64
-0+{ +/* +* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is +* unsafe in NMI. If spin_trylock() is called from hard IRQ the current +* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will +* mark the task as the owner of another rt_spin_lock which will +* confuse PI logic, so return immediately if called from hard IRQ or +* NMI. +* +* Note, irqs_disabled() case is ok. This function can be called +* from raw_spin_lock_irqsave region. +*/ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) +return false; + +/* On UP, spin_trylock() always succeeds even when it is locked */ +if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +return false; + +/* Bailout, since _deferred_grow_zone() needs to take a lock */ +if (deferred_pages_enabled ()) +return false; + +return true; +} + +/* +* GFP flags to set for ALLOC_NOLOCK i.e. alloc_pages_nolock(). +* +* Do not specify __GFP_DIRECT_RECLAIM, since direct claim is not allowed. +* Do not specify __GFP_KSWAPD_RECLAIM either, since wake up of kswapd +* is not safe in arbitrary context. +* +* These two are the conditions for gfpflags_allow_spinning() being true. +* +* Specify __GFP_NOWARN since failing alloc_pages_nolock() is not a reason +* to warn. Also warn would trigger printk() which is unsafe from +* various contexts. We cannot use printk_deferred_enter() to mitigate, +* since the running context is unknown. +* +* Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below +* is safe in any context. Also zeroing the page is mandatory for +* BPF use cases. +* +* Though __GFP_NOMEMALLOC is not checked in the code path below, +* specify it here to highlight that alloc_pages_nolock() +* doesn't want to deplete reserves. +*/ +static const gfp_t gfp_nolock = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | + __GFP_COMP; + +/* +* This is the 'heart' of the zoned buddy allocator. +*/ +struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order, +int preferred_nid,nodemask_t *nodemask,unsigned int alloc_flags) +{ +struct page *page; +gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ +struct alloc_context ac = { }; +unsigned int fastpath_alloc_flags = alloc_flags; + +/* Other flags could be supported later if needed. */ +if (WARN_ON (alloc_flags & ~ALLOC_NOLOCK ))
mm/page_alloc.c void free_pages_bulk(struct page **page_array, unsigned long nr_pages) · line 5361
Quoted 10 lines
mm/page_alloc.c +8
-0return NULL ; +if (!alloc_order_allowed (gfp, order, alloc_flags)) +return NULL ; + +if (alloc_flags &ALLOC_NOLOCK ) { +VM_WARN_ON_ONCE (gfp & ~__GFP_ACCOUNT); +if (!alloc_trylock_allowed ()) +return NULL ; + gfp |= gfp_nolock;
I think we could do a
fastpath_alloc_flags |= ALLOC_WMARK_MIN;
to make it explicit, even though it's a no-op (the value is 0) and alloc_frozen_pages_nolock_noprof() didn't do it.
Quoted 143 lines
mm/page_alloc.c +11
-75+ }else { + fastpath_alloc_flags |=ALLOC_WMARK_LOW ; + } + gfp &= gfp_allowed_mask;/* * Apply scoped allocation constraints. This is mainly about GFP_NOFS @@ -5310,9 +5384,9 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, fastpath_alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); fastpath_alloc_flags |=alloc_flags_nonblocking (gfp, order) &ALLOC_HIGHATOMIC ; -/* First allocation attempt */ +/* First allocation attempt (or, for nolock, only attempt) */ page =get_page_from_freelist (alloc_gfp, order, fastpath_alloc_flags, &ac); -if (likely (page)) +if (likely (page) || (alloc_flags &ALLOC_NOLOCK )) goto out; alloc_gfp = gfp;@@ -5329,7 +5403,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, out:if (memcg_kmem_online () && (gfp & __GFP_ACCOUNT) && page &&unlikely (__memcg_kmem_charge_page (page, gfp, order) !=0 )) { -free_frozen_pages (page, order); +__free_frozen_pages (page, order, + alloc_flags &ALLOC_NOLOCK ?FPI_TRYLOCK :0 ); page =NULL ; }@@ -5345,7 +5420,8 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, {struct page *page; - page =__alloc_frozen_pages_noprof (gfp, order, preferred_nid, nodemask); + page =__alloc_frozen_pages_noprof (gfp, order, preferred_nid, nodemask, +ALLOC_DEFAULT );if (page)set_page_refcounted (page);return page;@@ -7875,80 +7951,10 @@ static bool __free_unaccepted(struct page *page)struct page *alloc_frozen_pages_nolock_noprof (gfp_t gfp_flags,int nid,unsigned int order) { -/* -* Do not specify __GFP_DIRECT_RECLAIM, since direct claim is not allowed. -* Do not specify __GFP_KSWAPD_RECLAIM either, since wake up of kswapd -* is not safe in arbitrary context. -* -* These two are the conditions for gfpflags_allow_spinning() being true. -* -* Specify __GFP_NOWARN since failing alloc_pages_nolock() is not a reason -* to warn. Also warn would trigger printk() which is unsafe from -* various contexts. We cannot use printk_deferred_enter() to mitigate, -* since the running context is unknown. -* -* Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below -* is safe in any context. Also zeroing the page is mandatory for -* BPF use cases. -* -* Though __GFP_NOMEMALLOC is not checked in the code path below, -* specify it here to highlight that alloc_pages_nolock() -* doesn't want to deplete reserves. -*/ -gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | __GFP_COMP - | gfp_flags; -unsigned int alloc_flags =ALLOC_NOLOCK ; -struct alloc_context ac = { }; -struct page *page; - -VM_WARN_ON_ONCE (gfp_flags & ~__GFP_ACCOUNT); -/* -* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is -* unsafe in NMI. If spin_trylock() is called from hard IRQ the current -* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will -* mark the task as the owner of another rt_spin_lock which will -* confuse PI logic, so return immediately if called from hard IRQ or -* NMI. -* -* Note, irqs_disabled() case is ok. This function can be called -* from raw_spin_lock_irqsave region. -*/ -if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) -return NULL ; - -/* On UP, spin_trylock() always succeeds even when it is locked */ -if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) -return NULL ; - -if (!pcp_allowed_order (order)) -return NULL ; - -/* Bailout, since _deferred_grow_zone() needs to take a lock */ -if (deferred_pages_enabled ()) -return NULL ; -if (nid ==NUMA_NO_NODE ) nid =numa_node_id (); -prepare_alloc_pages (alloc_gfp, order, nid,NULL , &ac, - &alloc_gfp, &alloc_flags); - -/* -* Best effort allocation from percpu free list. -* If it's empty attempt to spin_trylock zone->lock. -*/ - page =get_page_from_freelist (alloc_gfp, order, alloc_flags, &ac); - -/* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ - -if (memcg_kmem_online () && page && (gfp_flags & __GFP_ACCOUNT) && -unlikely (__memcg_kmem_charge_page (page, alloc_gfp, order) !=0 )) { -__free_frozen_pages (page, order,FPI_TRYLOCK ); - page =NULL ; - } -trace_mm_page_alloc (page, order, alloc_gfp, ac.migratetype); -kmsan_alloc_page (page, order, alloc_gfp); -return page; +return __alloc_frozen_pages_noprof (gfp_flags, order, nid,NULL ,ALLOC_NOLOCK ); }/** * alloc_pages_nolock - opportunistic reentrant allocation from any context mm/page_alloc.h +2
-1@@ -11,6 +11,7 @@#include <linux/nodemask.h> #include <linux/types.h> +#define ALLOC_DEFAULT 0/* The ALLOC_WMARK bits are used as an index to zone->watermark */ #define ALLOC_WMARK_MIN WMARK_MIN#define ALLOC_WMARK_LOW WMARK_LOW@@ -219,7 +220,7 @@ extern bool free_pages_prepare(struct page *page, unsigned int order);extern int user_min_free_kbytes;struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid, -nodemask_t *nodemask); +nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order);
mm/page_alloc.h struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order); · line 235
Quoted 6 lines
mm/page_alloc.h +2
-1@@ -230,7 +231,8 @@ struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order);#else static inline struct page *alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order) { -return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL ); +return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL , +0 /* ALLOC_DEFAULT */ );
Can use ALLOC_DEFAULT now.
Quoted 3 lines
mm/page_alloc.h +0
-0}#endif
Brendan Jackman
reviewerRe: [PATCH v3 05/16] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
Quoted 336 lines · Vlastimil Babka (SUSE)
Quoted 2 lines · Brendan Jackman
Currently the core allocator code is controlled by
ALLOC_NOLOCK, but the main entry point function is significantly different from the normalLet's mention it explicitly,
alloc_frozen_pages_nolock_noprof().Quoted 1 line
__alloc_frozen_pages_nolock(), this is tiring when reading the code.You mean
__alloc_frozen_pages_noprof()?Quoted 3 lines
Plumb the
ALLOC_NOLOCKcontrol one layer up in the call stack: create analloc_flagsargument to__alloc_frozen_pages_nolock()(which is onlyAgain
__alloc_frozen_pages_noprof()Quoted 9 lines
exposed to
mm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handlingNUMA_NO_NODE, similar to how some of the wrappers ingfp.hdo).Rationale that this doesn't change anything:
1. Simple bits: A bunch of the nolock-specific handling is just moved to the new
alloc_order_allowed(),alloc_trylock_allowed()andgfp_trylock.Should be
alloc_nolock_allowed()andgfp_nolockQuoted 5 lines
2.
__alloc_frozen_pages_noprof()has some extra logic that wasn't previously in the nolock variant:a. Application of
gfp_allowed_mask; this only affects early boot, and
only flags that affect the slowpath get changed here.As discussed in reply to Harry, I'd mention the flags excluded by
GFP_BOOT_MASKare not usable by_nolock()anyway.Quoted 5 lines
b. Application of
current_gfp_context()- also only affects the
slowpath3. The slowpath itself: this is now just explicitly skipped under !ALLOC_TRYLOCK.
ALLOC_NOLOCK.Quoted 5 lines
Ulterior motive: adding an
alloc_flagsarg to the allocator's mm-internal entrypoint can later be used to do more allocation customisation without needing to create new GFP flags.Quoted 8 lines
While adding this flag to a bunch of places, create
ALLOC_DEFAULTto avoid a mysterious literal 0 in most places.
alloc_frozen_pages_noprof()
is defined above the alloc flags so just leave that as a slightly messy exception instead of trying to fully reordermm/internal.hfor that one case.This no longer applies in v3?
Quoted 45 lines
No functional change intended.
Signed-off-by Brendan Jackman <[email protected]>--- mm/hugetlb.c | 3 +- mm/mempolicy.c | 10 ++-- mm/page_alloc.c | 178 +++++++++++++++++++++++++++++--------------------------- mm/page_alloc.h | 6 +- mm/slub.c | 6 +- 5 files changed, 108 insertions(+), 95 deletions(-)mm/hugetlb.c +0
-0mm/page_alloc.c +5
-10@@ -5222,7 +5222,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, } nr_account++; -prep_new_page (page,0 , gfp,0 ); +prep_new_page (page,0 , gfp,ALLOC_DEFAULT );set_page_refcounted (page); page_array[nr_populated++] = page; }@@ -5271,24 +5271,98 @@ void free_pages_bulk(struct page **page_array, unsigned long nr_pages) } } -/* -* This is the 'heart' of the zoned buddy allocator. -*/ -struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order, -int preferred_nid,nodemask_t *nodemask) +static inline bool alloc_order_allowed (gfp_t gfp,unsigned int order, +unsigned int alloc_flags) { -struct page *page; -unsigned int fastpath_alloc_flags =ALLOC_WMARK_LOW ; -gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ -struct alloc_context ac = { }; +if (alloc_flags &ALLOC_NOLOCK ) +return pcp_allowed_order (order);mm/page_alloc.c void free_pages_bulk(struct page **page_array, unsigned long nr_pages) · line 5287
Quoted 10 lines
mm/page_alloc.c +4
-1/* * There are several places where we assume that the order value is sane * so bail out early if the request is out of bound. */ -if (WARN_ON_ONCE_GFP (order >MAX_PAGE_ORDER , gfp)) +return !(WARN_ON_ONCE_GFP (order >MAX_PAGE_ORDER , gfp)); +} + +static inline bool alloc_trylock_allowed (void )
alloc_nolock_allowed()Quoted 64 lines
mm/page_alloc.c +64
-0+{ +/* +* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is +* unsafe in NMI. If spin_trylock() is called from hard IRQ the current +* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will +* mark the task as the owner of another rt_spin_lock which will +* confuse PI logic, so return immediately if called from hard IRQ or +* NMI. +* +* Note, irqs_disabled() case is ok. This function can be called +* from raw_spin_lock_irqsave region. +*/ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) +return false; + +/* On UP, spin_trylock() always succeeds even when it is locked */ +if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +return false; + +/* Bailout, since _deferred_grow_zone() needs to take a lock */ +if (deferred_pages_enabled ()) +return false; + +return true; +} + +/* +* GFP flags to set for ALLOC_NOLOCK i.e. alloc_pages_nolock(). +* +* Do not specify __GFP_DIRECT_RECLAIM, since direct claim is not allowed. +* Do not specify __GFP_KSWAPD_RECLAIM either, since wake up of kswapd +* is not safe in arbitrary context. +* +* These two are the conditions for gfpflags_allow_spinning() being true. +* +* Specify __GFP_NOWARN since failing alloc_pages_nolock() is not a reason +* to warn. Also warn would trigger printk() which is unsafe from +* various contexts. We cannot use printk_deferred_enter() to mitigate, +* since the running context is unknown. +* +* Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below +* is safe in any context. Also zeroing the page is mandatory for +* BPF use cases. +* +* Though __GFP_NOMEMALLOC is not checked in the code path below, +* specify it here to highlight that alloc_pages_nolock() +* doesn't want to deplete reserves. +*/ +static const gfp_t gfp_nolock = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | + __GFP_COMP; + +/* +* This is the 'heart' of the zoned buddy allocator. +*/ +struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order, +int preferred_nid,nodemask_t *nodemask,unsigned int alloc_flags) +{ +struct page *page; +gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ +struct alloc_context ac = { }; +unsigned int fastpath_alloc_flags = alloc_flags; + +/* Other flags could be supported later if needed. */ +if (WARN_ON (alloc_flags & ~ALLOC_NOLOCK ))mm/page_alloc.c void free_pages_bulk(struct page **page_array, unsigned long nr_pages) · line 5361
Quoted 10 lines
mm/page_alloc.c +8
-0return NULL ; +if (!alloc_order_allowed (gfp, order, alloc_flags)) +return NULL ; + +if (alloc_flags &ALLOC_NOLOCK ) { +VM_WARN_ON_ONCE (gfp & ~__GFP_ACCOUNT); +if (!alloc_trylock_allowed ()) +return NULL ; + gfp |= gfp_nolock;I think we could do a
fastpath_alloc_flags |= ALLOC_WMARK_MIN;to make it explicit, even though it's a no-op (the value is 0) and
alloc_frozen_pages_nolock_noprof()didn't do it.Quoted 136 lines
mm/page_alloc.c +11
-75+ }else { + fastpath_alloc_flags |=ALLOC_WMARK_LOW ; + } + gfp &= gfp_allowed_mask;/* * Apply scoped allocation constraints. This is mainly about GFP_NOFS @@ -5310,9 +5384,9 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, fastpath_alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); fastpath_alloc_flags |=alloc_flags_nonblocking (gfp, order) &ALLOC_HIGHATOMIC ; -/* First allocation attempt */ +/* First allocation attempt (or, for nolock, only attempt) */ page =get_page_from_freelist (alloc_gfp, order, fastpath_alloc_flags, &ac); -if (likely (page)) +if (likely (page) || (alloc_flags &ALLOC_NOLOCK )) goto out; alloc_gfp = gfp;@@ -5329,7 +5403,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, out:if (memcg_kmem_online () && (gfp & __GFP_ACCOUNT) && page &&unlikely (__memcg_kmem_charge_page (page, gfp, order) !=0 )) { -free_frozen_pages (page, order); +__free_frozen_pages (page, order, + alloc_flags &ALLOC_NOLOCK ?FPI_TRYLOCK :0 ); page =NULL ; }@@ -5345,7 +5420,8 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, {struct page *page; - page =__alloc_frozen_pages_noprof (gfp, order, preferred_nid, nodemask); + page =__alloc_frozen_pages_noprof (gfp, order, preferred_nid, nodemask, +ALLOC_DEFAULT );if (page)set_page_refcounted (page);return page;@@ -7875,80 +7951,10 @@ static bool __free_unaccepted(struct page *page)struct page *alloc_frozen_pages_nolock_noprof (gfp_t gfp_flags,int nid,unsigned int order) { -/* -* Do not specify __GFP_DIRECT_RECLAIM, since direct claim is not allowed. -* Do not specify __GFP_KSWAPD_RECLAIM either, since wake up of kswapd -* is not safe in arbitrary context. -* -* These two are the conditions for gfpflags_allow_spinning() being true. -* -* Specify __GFP_NOWARN since failing alloc_pages_nolock() is not a reason -* to warn. Also warn would trigger printk() which is unsafe from -* various contexts. We cannot use printk_deferred_enter() to mitigate, -* since the running context is unknown. -* -* Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below -* is safe in any context. Also zeroing the page is mandatory for -* BPF use cases. -* -* Though __GFP_NOMEMALLOC is not checked in the code path below, -* specify it here to highlight that alloc_pages_nolock() -* doesn't want to deplete reserves. -*/ -gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | __GFP_COMP - | gfp_flags; -unsigned int alloc_flags =ALLOC_NOLOCK ; -struct alloc_context ac = { }; -struct page *page; - -VM_WARN_ON_ONCE (gfp_flags & ~__GFP_ACCOUNT); -/* -* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is -* unsafe in NMI. If spin_trylock() is called from hard IRQ the current -* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will -* mark the task as the owner of another rt_spin_lock which will -* confuse PI logic, so return immediately if called from hard IRQ or -* NMI. -* -* Note, irqs_disabled() case is ok. This function can be called -* from raw_spin_lock_irqsave region. -*/ -if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) -return NULL ; - -/* On UP, spin_trylock() always succeeds even when it is locked */ -if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) -return NULL ; - -if (!pcp_allowed_order (order)) -return NULL ; - -/* Bailout, since _deferred_grow_zone() needs to take a lock */ -if (deferred_pages_enabled ()) -return NULL ; -if (nid ==NUMA_NO_NODE ) nid =numa_node_id (); -prepare_alloc_pages (alloc_gfp, order, nid,NULL , &ac, - &alloc_gfp, &alloc_flags); - -/* -* Best effort allocation from percpu free list. -* If it's empty attempt to spin_trylock zone->lock. -*/ - page =get_page_from_freelist (alloc_gfp, order, alloc_flags, &ac); - -/* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ - -if (memcg_kmem_online () && page && (gfp_flags & __GFP_ACCOUNT) && -unlikely (__memcg_kmem_charge_page (page, alloc_gfp, order) !=0 )) { -__free_frozen_pages (page, order,FPI_TRYLOCK ); - page =NULL ; - } -trace_mm_page_alloc (page, order, alloc_gfp, ac.migratetype); -kmsan_alloc_page (page, order, alloc_gfp); -return page; +return __alloc_frozen_pages_noprof (gfp_flags, order, nid,NULL ,ALLOC_NOLOCK ); }/** * alloc_pages_nolock - opportunistic reentrant allocation from any context mm/page_alloc.h +1
-0@@ -11,6 +11,7 @@#include <linux/nodemask.h> #include <linux/types.h> +#define ALLOC_DEFAULT 0/* The ALLOC_WMARK bits are used as an index to zone->watermark */ #define ALLOC_WMARK_MIN WMARK_MIN#define ALLOC_WMARK_LOW WMARK_LOW@@ -219,7 +220,7 @@ extern bool free_pages_prepare(struct page *page, unsigned int order);extern int user_min_free_kbytes;
Quoted 15 lines
mm/page_alloc.h struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order); · line 235
Quoted 13 lines
mm/page_alloc.h +3
-2struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid, -nodemask_t *nodemask); +nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order);@@ -230,7 +231,8 @@ struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order);#else static inline struct page *alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order) { -return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL ); +return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL , +0 /* ALLOC_DEFAULT */ );Can use
ALLOC_DEFAULTnow.
Thanks and ack to all of these.
Will mention the ALLOC_WMARK_MIN thing in the commit message too.
PATCH v3 6/16 mm/page_alloc: relax GFP WARN in nolock allocs
In reply to Brendan JackmanThis WARN forbids setting other flags than __GFP_ACCOUNT but we unconditionally set the ones in gfp_nolock so they are certainly fine for the caller to set.
There are other GFP flags that are almost certainly fine to set here; Willy noted GFP_HIGHMEM, GFP_DMA, GFP_MOVABLE and GFP_HARDWALL. But, nolock allocation is rather special, so be conservative to try and ensure we have a chance to think carefully before nontrivial new usecases arise.
--- mm/page_alloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/page_alloc.c +2 -1
@@ -5355,7 +5355,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,return NULL ;if (alloc_flags &ALLOC_NOLOCK ) { -VM_WARN_ON_ONCE (gfp & ~__GFP_ACCOUNT); +/* Certain other flags could be supported later if needed. */ +VM_WARN_ON_ONCE (gfp & ~(__GFP_ACCOUNT | gfp_nolock));if (!alloc_trylock_allowed ())return NULL ; gfp |= gfp_nolock;
Signature · 1 line
2.54.0
Re: [PATCH v3 06/16] mm/page_alloc: relax GFP WARN in nolock allocs
Quoted 10 lines · Brendan Jackman
This WARN forbids setting other flags than
__GFP_ACCOUNTbut we unconditionally set the ones ingfp_nolockso they are certainly fine for the caller to set.There are other GFP flags that are almost certainly fine to set here; Willy noted
GFP_HIGHMEM,GFP_DMA,GFP_MOVABLEandGFP_HARDWALL. But, nolock allocation is rather special, so be conservative to try and ensure we have a chance to think carefully before nontrivial new usecases arise.
Quoted 5 lines
Suggested-by Matthew Wilcox <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Brendan Jackman <[email protected]>---
Signature · 2 lines
Cheers, Harry / Hyeonggon
Re: [PATCH v3 06/16] mm/page_alloc: relax GFP WARN in nolock allocs
In reply to Brendan JackmanQuoted 10 lines · Brendan Jackman
This WARN forbids setting other flags than
__GFP_ACCOUNTbut we unconditionally set the ones ingfp_nolockso they are certainly fine for the caller to set.There are other GFP flags that are almost certainly fine to set here; Willy noted
GFP_HIGHMEM,GFP_DMA,GFP_MOVABLEandGFP_HARDWALL. But, nolock allocation is rather special, so be conservative to try and ensure we have a chance to think carefully before nontrivial new usecases arise.
Quoted 4 lines
Suggested-by Matthew Wilcox <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 19 lines
--- mm/page_alloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/page_alloc.c +2
-1@@ -5355,7 +5355,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,return NULL ;if (alloc_flags &ALLOC_NOLOCK ) { -VM_WARN_ON_ONCE (gfp & ~__GFP_ACCOUNT); +/* Certain other flags could be supported later if needed. */ +VM_WARN_ON_ONCE (gfp & ~(__GFP_ACCOUNT | gfp_nolock));if (!alloc_trylock_allowed ())return NULL ; gfp |= gfp_nolock;
PATCH v3 7/16 mm: move some stuff to mm/page_alloc.h
In reply to Brendan JackmanSome of this stuff in the public header is only used internally so shrink the scope to avoid silently growing new users.
drain_local_pages() is still used from kernel/power/snapshot.c so that needs to stay behind.
--- include/linux/gfp.h | 26 -------------------------- mm/page_alloc.h | 28 ++++++++++++++++++++++++++++ mm/vmstat.c | 1 + 3 files changed, 29 insertions(+), 26 deletions(-)
include/linux/gfp.h +0 -26
@@ -17,28 +17,6 @@ struct mempolicy;#define __default_gfp (a,b,...) b#define default_gfp (...) __default_gfp(,##__VA_ARGS__,GFP_KERNEL) -/* Convert GFP flags to their corresponding migrate type */ -#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) -#define GFP_MOVABLE_SHIFT 3 - -static inline int gfp_migratetype (const gfp_t gfp_flags) -{ -VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK ); -BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE); -BUILD_BUG_ON ((___GFP_MOVABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_MOVABLE ); -BUILD_BUG_ON ((___GFP_RECLAIMABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_RECLAIMABLE ); -BUILD_BUG_ON (((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> -GFP_MOVABLE_SHIFT ) !=MIGRATE_HIGHATOMIC ); - -if (unlikely (page_group_by_mobility_disabled)) -return MIGRATE_UNMOVABLE ; - -/* Group based on mobility */ -return (__force unsigned long )(gfp_flags &GFP_MOVABLE_MASK ) >>GFP_MOVABLE_SHIFT ; -} -#undef GFP_MOVABLE_MASK -#undef GFP_MOVABLE_SHIFT -static inline bool gfpflags_allow_blocking (const gfp_t gfp_flags) {return !!(gfp_flags & __GFP_DIRECT_RECLAIM);
@@ -395,10 +373,6 @@ extern void free_pages(unsigned long addr, unsigned int order);#define __free_page (page) __free_pages((page), 0)#define free_page (addr) free_pages((addr), 0) -void page_alloc_init_cpuhp (void ); -bool decay_pcp_high (struct zone *zone,struct per_cpu_pages *pcp); -void drain_zone_pages (struct zone *zone,struct per_cpu_pages *pcp); -void drain_all_pages (struct zone *zone);void drain_local_pages (struct zone *zone);void page_alloc_init_late (void );
mm/page_alloc.h +28 -0
@@ -266,6 +266,34 @@ static inline bool free_area_empty(struct free_area *area, int migratetype) return list_empty(&area->free_list[migratetype]); } +/* Convert GFP flags to their corresponding migrate type */ +#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) +#define GFP_MOVABLE_SHIFT 3 + +static inline int gfp_migratetype (const gfp_t gfp_flags) +{ +VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK ); +BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE); +BUILD_BUG_ON ((___GFP_MOVABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_MOVABLE ); +BUILD_BUG_ON ((___GFP_RECLAIMABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_RECLAIMABLE ); +BUILD_BUG_ON (((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> +GFP_MOVABLE_SHIFT ) !=MIGRATE_HIGHATOMIC ); + +if (unlikely (page_group_by_mobility_disabled)) +return MIGRATE_UNMOVABLE ; + +/* Group based on mobility */ +return (__force unsigned long )(gfp_flags &GFP_MOVABLE_MASK ) >>GFP_MOVABLE_SHIFT ; +} +#undef GFP_MOVABLE_MASK +#undef GFP_MOVABLE_SHIFT + +bool decay_pcp_high (struct zone *zone,struct per_cpu_pages *pcp); +void drain_zone_pages (struct zone *zone,struct per_cpu_pages *pcp); +void drain_all_pages (struct zone *zone); +void drain_local_pages (struct zone *zone); + +void page_alloc_init_cpuhp (void );void page_alloc_sysctl_init (void );#endif /* __MM_PAGE_ALLOC_H */
mm/vmstat.c +1 -0
@@ -30,6 +30,7 @@#include <linux/sched/isolation.h> #include "internal.h" +#include "page_alloc.h" #ifdef CONFIG_PROC_FS #ifdef CONFIG_NUMA
Signature · 1 line
2.54.0
Re: [PATCH v3 07/16] mm: move some stuff to mm/page_alloc.h
Quoted 7 lines · Brendan Jackman
Some of this stuff in the public header is only used internally so shrink the scope to avoid silently growing new users.
drain_local_pages()is still used fromkernel/power/snapshot.cso that needs to stay behind.Signed-off-by Brendan Jackman <[email protected]>
Quoted 102 lines
--- include/linux/gfp.h | 26 -------------------------- mm/page_alloc.h | 28 ++++++++++++++++++++++++++++ mm/vmstat.c | 1 + 3 files changed, 29 insertions(+), 26 deletions(-)include/linux/gfp.h +0
-26@@ -17,28 +17,6 @@ struct mempolicy;#define __default_gfp (a,b,...) b#define default_gfp (...) __default_gfp(,##__VA_ARGS__,GFP_KERNEL) -/* Convert GFP flags to their corresponding migrate type */ -#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) -#define GFP_MOVABLE_SHIFT 3 - -static inline int gfp_migratetype (const gfp_t gfp_flags) -{ -VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK ); -BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE); -BUILD_BUG_ON ((___GFP_MOVABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_MOVABLE ); -BUILD_BUG_ON ((___GFP_RECLAIMABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_RECLAIMABLE ); -BUILD_BUG_ON (((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> -GFP_MOVABLE_SHIFT ) !=MIGRATE_HIGHATOMIC ); - -if (unlikely (page_group_by_mobility_disabled)) -return MIGRATE_UNMOVABLE ; - -/* Group based on mobility */ -return (__force unsigned long )(gfp_flags &GFP_MOVABLE_MASK ) >>GFP_MOVABLE_SHIFT ; -} -#undef GFP_MOVABLE_MASK -#undef GFP_MOVABLE_SHIFT -static inline bool gfpflags_allow_blocking (const gfp_t gfp_flags) {return !!(gfp_flags & __GFP_DIRECT_RECLAIM);@@ -395,10 +373,6 @@ extern void free_pages(unsigned long addr, unsigned int order);#define __free_page (page) __free_pages((page), 0)#define free_page (addr) free_pages((addr), 0) -void page_alloc_init_cpuhp (void ); -bool decay_pcp_high (struct zone *zone,struct per_cpu_pages *pcp); -void drain_zone_pages (struct zone *zone,struct per_cpu_pages *pcp); -void drain_all_pages (struct zone *zone);void drain_local_pages (struct zone *zone);void page_alloc_init_late (void );mm/page_alloc.h +28
-0@@ -266,6 +266,34 @@ static inline bool free_area_empty(struct free_area *area, int migratetype) return list_empty(&area->free_list[migratetype]); } +/* Convert GFP flags to their corresponding migrate type */ +#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) +#define GFP_MOVABLE_SHIFT 3 + +static inline int gfp_migratetype (const gfp_t gfp_flags) +{ +VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK ); +BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE); +BUILD_BUG_ON ((___GFP_MOVABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_MOVABLE ); +BUILD_BUG_ON ((___GFP_RECLAIMABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_RECLAIMABLE ); +BUILD_BUG_ON (((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> +GFP_MOVABLE_SHIFT ) !=MIGRATE_HIGHATOMIC ); + +if (unlikely (page_group_by_mobility_disabled)) +return MIGRATE_UNMOVABLE ; + +/* Group based on mobility */ +return (__force unsigned long )(gfp_flags &GFP_MOVABLE_MASK ) >>GFP_MOVABLE_SHIFT ; +} +#undef GFP_MOVABLE_MASK +#undef GFP_MOVABLE_SHIFT + +bool decay_pcp_high (struct zone *zone,struct per_cpu_pages *pcp); +void drain_zone_pages (struct zone *zone,struct per_cpu_pages *pcp); +void drain_all_pages (struct zone *zone); +void drain_local_pages (struct zone *zone); + +void page_alloc_init_cpuhp (void );void page_alloc_sysctl_init (void );#endif /* __MM_PAGE_ALLOC_H */ mm/vmstat.c +1
-0@@ -30,6 +30,7 @@#include <linux/sched/isolation.h> #include "internal.h" +#include "page_alloc.h" #ifdef CONFIG_PROC_FS #ifdef CONFIG_NUMA
PATCH v3 8/16 perf/x86/intel: Use higher-level allocator API
In reply to Brendan JackmanThe difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- arch/x86/events/intel/ds.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/events/intel/ds.c +3 -3
@@ -832,7 +832,7 @@ static void *dsalloc_pages(size_t size, gfp_t flags, int cpu)int node =cpu_to_node (cpu);struct page *page; - page =__alloc_pages_node (node, flags | __GFP_ZERO, order); + page =alloc_pages_node (node, flags | __GFP_ZERO, order);return page ?page_address (page) :NULL ; }
@@ -1088,9 +1088,9 @@ void init_arch_pebs_on_cpu(int cpu)/* * 4KB-aligned pointer of the output buffer -* (__alloc_pages_node() return page aligned address) +* (alloc_pages_node() returns page aligned address) * Buffer Size = 4KB * 2^SIZE -* contiguous physical buffer (__alloc_pages_node() with order) +* contiguous physical buffer (alloc_pages_node() with order) */ arch_pebs_base =virt_to_phys (cpuc->pebs_vaddr) |PEBS_BUFFER_SHIFT ;wrmsrq_on_cpu (cpu,MSR_IA32_PEBS_BASE , arch_pebs_base);
Signature · 1 line
2.54.0
Re: [PATCH v3 08/16] perf/x86/intel: Use higher-level allocator API
Quoted 7 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Quoted 13 lines
Cc Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Namhyung Kim <[email protected]>Cc Mark Rutland <[email protected]>Cc Alexander Shishkin <[email protected]>Cc Jiri Olsa <[email protected]>Cc Ian Rogers <[email protected]>Cc Adrian Hunter <[email protected]>Cc James Clark <[email protected]>Assisted-by Gemini:unknown-versionReviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 30 lines
--- arch/x86/events/intel/ds.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)arch/x86/events/intel/ds.c +3
-3@@ -832,7 +832,7 @@ static void *dsalloc_pages(size_t size, gfp_t flags, int cpu)int node =cpu_to_node (cpu);struct page *page; - page =__alloc_pages_node (node, flags | __GFP_ZERO, order); + page =alloc_pages_node (node, flags | __GFP_ZERO, order);return page ?page_address (page) :NULL ; }@@ -1088,9 +1088,9 @@ void init_arch_pebs_on_cpu(int cpu)/* * 4KB-aligned pointer of the output buffer -* (__alloc_pages_node() return page aligned address) +* (alloc_pages_node() returns page aligned address) * Buffer Size = 4KB * 2^SIZE -* contiguous physical buffer (__alloc_pages_node() with order) +* contiguous physical buffer (alloc_pages_node() with order) */ arch_pebs_base =virt_to_phys (cpuc->pebs_vaddr) |PEBS_BUFFER_SHIFT ;wrmsrq_on_cpu (cpu,MSR_IA32_PEBS_BASE , arch_pebs_base);
PATCH v3 9/16 KVM: VMX: Use higher-level allocator API
In reply to Brendan JackmanThe difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- arch/x86/kvm/vmx/vmx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -3028,7 +3028,7 @@ struct vmcs *alloc_vmcs_cpu(bool shadow, int cpu, gfp_t flags)struct page *pages;struct vmcs *vmcs; - pages =__alloc_pages_node (node, flags,0 ); + pages =alloc_pages_node (node, flags,0 );if (!pages)return NULL ; vmcs =page_address (pages);
Signature · 1 line
2.54.0
RE: -EXT-[PATCH v3 09/16] KVM: VMX: Use higher-level allocator API
Can someone please remove me from this list. I've requested a couple of times.
Thanks,
David
-----Original Message-----
From: Brendan Jackman <[email protected]>
Sent: Monday, June 29, 2026 7:12 AM
To: Andrew Morton <[email protected]>; Vlastimil Babka <[email protected]>; Suren Baghdasaryan <[email protected]>; Michal Hocko <[email protected]>; Johannes Weiner <[email protected]>; Zi Yan <[email protected]>; Muchun Song <[email protected]>; Oscar Salvador <[email protected]>; David Hildenbrand <[email protected]>; Lorenzo Stoakes <[email protected]>; Liam R. Howlett <[email protected]>; Mike Rapoport <[email protected]>; Matthew Brost <[email protected]>; Joshua Hahn <[email protected]>; Rakie Kim <[email protected]>; Byungchul Park <[email protected]>; Ying Huang <[email protected]>; Alistair Popple <[email protected]>; Hao Li <[email protected]>; Christoph Lameter <[email protected]>; David Rientjes <[email protected]>; Roman Gushchin <[email protected]>; Sebastian Andrzej Siewior <[email protected]>; Clark Williams <[email protected]>; Steven Rostedt <[email protected]>
Cc: Harry Yoo (Oracle) <[email protected]>; Gregory Price <[email protected]>; Johannes Weiner <[email protected]>; Alexei Starovoitov <[email protected]>; Matthew Wilcox <[email protected]>; Hao Ge <[email protected]>; [email protected]; [email protected]; [email protected]; Brendan Jackman <[email protected]>; Sean Christopherson <[email protected]>; Paolo Bonzini <[email protected]>; [email protected]
Subject: -EXT-[PATCH v3 09/16] KVM: VMX: Use higher-level allocator API
WARNING: This message is from an external source. Evaluate the message carefully BEFORE clicking on links or opening attachments.
The difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- arch/x86/kvm/vmx/vmx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -3028,7 +3028,7 @@ struct vmcs *alloc_vmcs_cpu(bool shadow, int cpu, gfp_t flags)struct page *pages;struct vmcs *vmcs; - pages =__alloc_pages_node (node, flags,0 ); + pages =alloc_pages_node (node, flags,0 );if (!pages)return NULL ; vmcs =page_address (pages);
--
2.54.0
Re: [PATCH v3 09/16] KVM: VMX: Use higher-level allocator API
In reply to Brendan JackmanQuoted 5 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
Quoted 8 lines
No functional change intended.
Cc Sean Christopherson <[email protected]>Cc Paolo Bonzini <[email protected]>Assisted-by Gemini:unknown-versionReviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 18 lines
--- arch/x86/kvm/vmx/vmx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kvm/vmx/vmx.c +1
-1@@ -3028,7 +3028,7 @@ struct vmcs *alloc_vmcs_cpu(bool shadow, int cpu, gfp_t flags)struct page *pages;struct vmcs *vmcs; - pages =__alloc_pages_node (node, flags,0 ); + pages =alloc_pages_node (node, flags,0 );if (!pages)return NULL ; vmcs =page_address (pages);
PATCH v3 10/16 x86/virt: Use higher-level allocator API
In reply to Brendan JackmanThe difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- arch/x86/virt/hw.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/virt/hw.c +1 -1
@@ -196,7 +196,7 @@ static __init int __x86_vmx_init(void)struct page *page;struct vmcs *vmcs; - page =__alloc_pages_node (node,GFP_KERNEL | __GFP_ZERO,0 ); + page =alloc_pages_node (node,GFP_KERNEL | __GFP_ZERO,0 );if (WARN_ON_ONCE (!page)) {x86_vmx_exit ();return -ENOMEM ;
Signature · 1 line
2.54.0
Re: [PATCH v3 10/16] x86/virt: Use higher-level allocator API
Quoted 7 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Quoted 9 lines
Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Cc "H. Peter Anvin" <[email protected]>Assisted-by Gemini:unknown-versionReviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 18 lines
--- arch/x86/virt/hw.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/virt/hw.c +1
-1@@ -196,7 +196,7 @@ static __init int __x86_vmx_init(void)struct page *page;struct vmcs *vmcs; - page =__alloc_pages_node (node,GFP_KERNEL | __GFP_ZERO,0 ); + page =alloc_pages_node (node,GFP_KERNEL | __GFP_ZERO,0 );if (WARN_ON_ONCE (!page)) {x86_vmx_exit ();return -ENOMEM ;
PATCH v3 11/16 sgi-xp: Use higher-level allocator API
In reply to Brendan JackmanThe difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- drivers/misc/sgi-xp/xpc_uv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/misc/sgi-xp/xpc_uv.c +1 -1
@@ -170,7 +170,7 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, mq->mmr_blade =uv_cpu_to_blade_id (cpu); nid =cpu_to_node (cpu); - page =__alloc_pages_node (nid, + page =alloc_pages_node (nid,GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE, pg_order);if (page ==NULL ) {
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v3 11/16] sgi-xp: Use higher-level allocator API
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] xpc_init_mq_node() incorrectly passes the node ID (nid) instead of the loop variable (cpu) to xpc_create_gru_mq_uv(). --
commit 364bc05827d5201ce117b2c0d83f21a9b64f85f3
Author: Brendan Jackman <[email protected]>
sgi-xp: Use higher-level allocator API
This commit replaces __alloc_pages_node() with alloc_pages_node() in xpc_create_gru_mq_uv(). The __alloc_pages_node() function is being phased out, and alloc_pages_node() works identically here while allowing NUMA_NO_NODE if it were needed.
drivers/misc/sgi-xp/xpc_uv.c xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, · line 172
Quoted 8 lines
drivers/misc/sgi-xp/xpc_uv.c +0
-0@@ -170,7 +170,7 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, mq->mmr_blade =uv_cpu_to_blade_id (cpu); nid =cpu_to_node (cpu);
[Severity: High]
This is a pre-existing issue, but while reviewing this code, I noticed a potential out-of-bounds access issue.
In xpc_init_mq_node(), the node ID (nid) is incorrectly passed instead of the loop variable (cpu) to xpc_create_gru_mq_uv():
drivers/misc/sgi-xp/xpc_uv.c:xpc_init_mq_node() {
...
xpc_activate_mq_uv =
xpc_create_gru_mq_uv(XPC_ACTIVATE_MQ_SIZE_UV, nid,
XPC_ACTIVATE_IRQ_NAME,
xpc_handle_activate_IRQ_uv);
...
}Because xpc_create_gru_mq_uv() expects a valid CPU index, passing a NUMA node ID means the cpu parameter here contains a node ID. Since NUMA node IDs and CPU IDs are in different ID spaces, cpu_to_node() will access the numa_node per-CPU variable using a node ID instead of a CPU ID.
If the node ID is greater than or equal to nr_cpu_ids, will this result in an out-of-bounds access causing a kernel panic during module load?
Even if it is within bounds, does this result in routing interrupts and allocating memory using the topology of the wrong CPU?
Quoted 5 lines
drivers/misc/sgi-xp/xpc_uv.c +1
-1- page =__alloc_pages_node (nid, + page =alloc_pages_node (nid,GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE, pg_order);if (page ==NULL ) {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Re: [PATCH v3 11/16] sgi-xp: Use higher-level allocator API
In reply to Brendan JackmanQuoted 36 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Cc Robin Holt <[email protected]>Cc Steve Wahl <[email protected]>Cc Arnd Bergmann <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Assisted-by Gemini:unknown-modelReviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- drivers/misc/sgi-xp/xpc_uv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/misc/sgi-xp/xpc_uv.c +1
-1@@ -170,7 +170,7 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, mq->mmr_blade =uv_cpu_to_blade_id (cpu); nid =cpu_to_node (cpu); - page =__alloc_pages_node (nid, + page =alloc_pages_node (nid,GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE, pg_order);if (page ==NULL ) {Signature · 2 lines
-- 2.54.0
Signature · 1 line
Steve Wahl, Hewlett Packard Enterprise
Re: [PATCH v3 11/16] sgi-xp: Use higher-level allocator API
In reply to Brendan JackmanQuoted 7 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Quoted 8 lines
Cc Robin Holt <[email protected]>Cc Steve Wahl <[email protected]>Cc Arnd Bergmann <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Assisted-by Gemini:unknown-modelReviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Nit:
Quoted 9 lines
--- drivers/misc/sgi-xp/xpc_uv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/misc/sgi-xp/xpc_uv.c +0
-0
drivers/misc/sgi-xp/xpc_uv.c xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, · line 175
Quoted 7 lines
drivers/misc/sgi-xp/xpc_uv.c +1
-1@@ -170,7 +170,7 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, mq->mmr_blade =uv_cpu_to_blade_id (cpu); nid =cpu_to_node (cpu); - page =__alloc_pages_node (nid, + page =alloc_pages_node (nid,GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE, pg_order);
gfp flags can now fit on the line above?
Quoted 2 lines
drivers/misc/sgi-xp/xpc_uv.c +0
-0if (page ==NULL ) {
Brendan Jackman
reviewerRe: [PATCH v3 11/16] sgi-xp: Use higher-level allocator API
Quoted 32 lines · Vlastimil Babka (SUSE)
Quoted 7 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Quoted 8 lines
Cc Robin Holt <[email protected]>Cc Steve Wahl <[email protected]>Cc Arnd Bergmann <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Assisted-by Gemini:unknown-modelReviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Brendan Jackman <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Nit:
Quoted 11 lines
--- drivers/misc/sgi-xp/xpc_uv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/misc/sgi-xp/xpc_uv.c +0
-0@@ -170,7 +170,7 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, mq->mmr_blade =uv_cpu_to_blade_id (cpu);
Quoted 7 lines
drivers/misc/sgi-xp/xpc_uv.c xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, · line 175
Quoted 5 lines
drivers/misc/sgi-xp/xpc_uv.c +1
-1nid =cpu_to_node (cpu); - page =__alloc_pages_node (nid, + page =alloc_pages_node (nid,GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE, pg_order);gfp flags can now fit on the line above?
Ack.
PATCH v3 12/16 net/funeth: Switch to higher-level allocator API
In reply to Brendan JackmanThe difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- drivers/net/ethernet/fungible/funeth/funeth_rx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/fungible/funeth/funeth_rx.c +1 -1
@@ -103,7 +103,7 @@ static int funeth_alloc_page(struct funeth_rxq *q, struct funeth_rxbuf *rb,if (cache_get (q, rb))return 0 ; - p =__alloc_pages_node (node, gfp | __GFP_NOWARN,0 ); + p =alloc_pages_node (node, gfp | __GFP_NOWARN,0 );if (unlikely (!p))return -ENOMEM ;
Signature · 1 line
2.54.0
Re: [PATCH v3 12/16] net/funeth: Switch to higher-level allocator API
Quoted 7 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Quoted 9 lines
Cc Dimitris Michailidis <[email protected]>Cc Andrew Lunn <[email protected]>Cc "David S. Miller" <[email protected]>Cc Eric Dumazet <[email protected]>Cc Jakub Kicinski <[email protected]>Cc Paolo Abeni <[email protected]>Assisted-by Gemini:unknown-versionReviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 18 lines
--- drivers/net/ethernet/fungible/funeth/funeth_rx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/net/ethernet/fungible/funeth/funeth_rx.c +1
-1@@ -103,7 +103,7 @@ static int funeth_alloc_page(struct funeth_rxq *q, struct funeth_rxbuf *rb,if (cache_get (q, rb))return 0 ; - p =__alloc_pages_node (node, gfp | __GFP_NOWARN,0 ); + p =alloc_pages_node (node, gfp | __GFP_NOWARN,0 );if (unlikely (!p))return -ENOMEM ;
PATCH v3 13/16 mm: Remove __alloc_pages_node()
In reply to Brendan JackmanThere were only a few users, which have been removed. The only advantage of this API over alloc_pages_node() is avoiding a single conditional branch. The disadvantages are:
1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of __alloc_pages_node() were using the result of cpu_to_node(); if the CPU gets hotplugged out this will return NUMA_NO_NODE. If one of these paths fails to protect against a concurrent hotplug then page_alloc.c will use NUMA_NO_NODE as an index into NODE_DATA() and cause some horrible memory corruption or other. With alloc_pages_node(), the code might just work fine.
Ulterior motive: this frees up the __* variants of the allocator APIs to serve specifically for use as mm-internal API.
--- include/linux/gfp.h | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)
include/linux/gfp.h +4 -16
@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {
@@ -293,7 +278,10 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,if (nid ==NUMA_NO_NODE ) nid =numa_mem_id (); -return __alloc_pages_node_noprof (nid, gfp_mask, order); +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); +warn_if_node_offline (nid, gfp_mask); + +return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); }#define alloc_pages_node (...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__))
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v3 13/16] mm: Remove __alloc_pages_node()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Incomplete API refactoring leaves __folio_alloc_node() and its external callers vulnerable to the CPU hotplug hazard the patch intends to fix. --
commit 1c3f10897a9f9fbcd231ced2981c00e238f22315
Author: Brendan Jackman <[email protected]>
mm: Remove __alloc_pages_node()
This commit removes the __alloc_pages_node() API to reduce the API surface and prevent issues where callers using cpu_to_node() pass NUMA_NO_NODE during CPU hotplug, leading to memory corruption or panics.
Quoted 4 lines
include/linux/gfp.h +0
-0
[ ... ]
Quoted 12 lines
include/linux/gfp.h +0
-8@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );
include/linux/gfp.h static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) · line 261
Quoted 10 lines
include/linux/gfp.h +0
-7-warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {
[Severity: High]
This is a pre-existing issue, but does __folio_alloc_node_noprof() and its wrappers need the same treatment?
I noticed it still lacks the NUMA_NO_NODE fallback, which can be triggered by existing external callers. For instance, in net/sunrpc/svc.c, svc_pool_map_get_node() uses cpu_to_node(), which can return NUMA_NO_NODE (-1) during CPU hotplug:
net/sunrpc/svc.c:svc_pool_map_get_node() {
...
if (m->mode == SVC_POOL_PERCPU)
return cpu_to_node(m->pool_to[pidx]);
...
}This returned node ID is then passed directly to __folio_alloc_node() in svc_prepare_thread():
net/sunrpc/svc.c:svc_prepare_thread() {
...
rqstp->rq_scratch_folio = __folio_alloc_node(GFP_KERNEL, 0, node);
if (!rqstp->rq_scratch_folio)
goto out_enomem;
...
}Could this trigger the same VM_BUG_ON(nid < 0) or out-of-bounds array access that this commit aims to resolve?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Re: [PATCH v3 13/16] mm: Remove __alloc_pages_node()
In reply to Brendan JackmanQuoted 14 lines · Brendan Jackman
There were only a few users, which have been removed. The only advantage of this API over
alloc_pages_node()is avoiding a single conditional branch. The disadvantages are:1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of
__alloc_pages_node()were using the result ofcpu_to_node(); if the CPU gets hotplugged out this will returnNUMA_NO_NODE. If one of these paths fails to protect against a concurrent hotplug thenpage_alloc.cwill useNUMA_NO_NODEas an index intoNODE_DATA()and cause some horrible memory corruption or other. Withalloc_pages_node(), the code might just work fine.
Quoted 5 lines
Ulterior motive: this frees up the __* variants of the allocator APIs to serve specifically for use as mm-internal API.
Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 43 lines
--- include/linux/gfp.h | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)include/linux/gfp.h +4
-16@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {@@ -293,7 +278,10 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,if (nid ==NUMA_NO_NODE ) nid =numa_mem_id (); -return __alloc_pages_node_noprof (nid, gfp_mask, order); +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); +warn_if_node_offline (nid, gfp_mask); + +return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); }#define alloc_pages_node (...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__))
PATCH v3 14/16 mm: Move __alloc_pages() to mm/page_alloc.h
In reply to Brendan JackmanIt's no longer used outside of mm/.
Since this means __alloc_pages_noprof() is no longer visible from gfp.h, this also means moving the definition of alloc_pages_node_noprof into the .c file.
Also remove references to this API from the documentation tree - referring to the specific function name was already questionable but now the function is not even public it definitely seems wrong.
--- Documentation/admin-guide/cgroup-v1/cpusets.rst | 2 +- Documentation/admin-guide/mm/transhuge.rst | 2 +- include/linux/gfp.h | 16 +--------------- mm/page_alloc.c | 13 ++++++++++++- mm/page_alloc.h | 4 ++++ 5 files changed, 19 insertions(+), 18 deletions(-)
Documentation/admin-guide/cgroup-v1/cpusets.rst +1 -1
@@ -284,7 +284,7 @@ take action. ==> Unless this feature is enabled by writing "1" to the special file /dev/cpuset/memory_pressure_enabled, the hook in the rebalance - code of __alloc_pages() for this metric reduces to simply noticing + code of the page allocator for this metric reduces to simply noticing that the cpuset_memory_pressure_enabled flag is zero. So only systems that enable this feature will compute the metric.
Documentation/admin-guide/mm/transhuge.rst +1 -1
@@ -761,7 +761,7 @@ compact_fail but failed. It is possible to establish how long the stalls were using the function -tracer to record how long was spent in __alloc_pages() and +tracer to record how long was spent in the page allocator and using the mm_page_alloc tracepoint to identify which allocations were for huge pages.
include/linux/gfp.h +1 -15
@@ -204,10 +204,6 @@ static inline void arch_free_page(struct page *page, int order) { }static inline void arch_alloc_page (struct page *page,int order) { }#endif -struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid, -nodemask_t *nodemask); -#define __alloc_pages (...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) -struct folio *__folio_alloc_noprof (gfp_t gfp,unsigned int order,int preferred_nid,nodemask_t *nodemask);#define __folio_alloc (...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))
@@ -272,17 +268,7 @@ struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)* prefer the current CPU's closest node. Otherwise node must be valid and * online. */ -static inline struct page *alloc_pages_node_noprof (int nid,gfp_t gfp_mask, -unsigned int order) -{ -if (nid ==NUMA_NO_NODE ) - nid =numa_mem_id (); - -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} +struct page *alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order);#define alloc_pages_node (...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__))
mm/page_alloc.c +12 -1
@@ -5427,7 +5427,18 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order,set_page_refcounted (page);return page; } -EXPORT_SYMBOL (__alloc_pages_noprof); + +struct page *alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) +{ +if (nid ==NUMA_NO_NODE ) + nid =numa_mem_id (); + +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); +warn_if_node_offline (nid, gfp_mask); + +return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); +} +EXPORT_SYMBOL (alloc_pages_node_noprof);struct folio *__folio_alloc_noprof (gfp_t gfp,unsigned int order,int preferred_nid,nodemask_t *nodemask)
mm/page_alloc.h +4 -0
@@ -244,6 +244,10 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsignedalloc_hooks (alloc_frozen_pages_nolock_noprof(__VA_ARGS__ )) void free_frozen_pages_nolock(struct page *page, unsigned int order); +struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid, +nodemask_t *nodemask); +#define __alloc_pages (...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) +extern void zone_pcp_reset (struct zone *zone);extern void zone_pcp_disable (struct zone *zone);extern void zone_pcp_enable (struct zone *zone);
Signature · 1 line
2.54.0
Re: [PATCH v3 14/16] mm: Move __alloc_pages() to mm/page_alloc.h
Quoted 11 lines · Brendan Jackman
It's no longer used outside of
mm/.Since this means
__alloc_pages_noprof()is no longer visible fromgfp.h, this also means moving the definition ofalloc_pages_node_noprofinto the .c file.Also remove references to this API from the documentation tree - referring to the specific function name was already questionable but now the function is not even public it definitely seems wrong.
Signed-off-by Brendan Jackman <[email protected]>
Less inline bloat also good.
Quoted 108 lines
--- Documentation/admin-guide/cgroup-v1/cpusets.rst | 2 +- Documentation/admin-guide/mm/transhuge.rst | 2 +- include/linux/gfp.h | 16 +--------------- mm/page_alloc.c | 13 ++++++++++++- mm/page_alloc.h | 4 ++++ 5 files changed, 19 insertions(+), 18 deletions(-)Documentation/admin-guide/cgroup-v1/cpusets.rst +1
-1@@ -284,7 +284,7 @@ take action. ==> Unless this feature is enabled by writing "1" to the special file /dev/cpuset/memory_pressure_enabled, the hook in the rebalance - code of __alloc_pages() for this metric reduces to simply noticing + code of the page allocator for this metric reduces to simply noticing that the cpuset_memory_pressure_enabled flag is zero. So only systems that enable this feature will compute the metric.Documentation/admin-guide/mm/transhuge.rst +1
-1@@ -761,7 +761,7 @@ compact_fail but failed. It is possible to establish how long the stalls were using the function -tracer to record how long was spent in __alloc_pages() and +tracer to record how long was spent in the page allocator and using the mm_page_alloc tracepoint to identify which allocations were for huge pages.include/linux/gfp.h +1
-15@@ -204,10 +204,6 @@ static inline void arch_free_page(struct page *page, int order) { }static inline void arch_alloc_page (struct page *page,int order) { }#endif -struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid, -nodemask_t *nodemask); -#define __alloc_pages (...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) -struct folio *__folio_alloc_noprof (gfp_t gfp,unsigned int order,int preferred_nid,nodemask_t *nodemask);#define __folio_alloc (...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))@@ -272,17 +268,7 @@ struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)* prefer the current CPU's closest node. Otherwise node must be valid and * online. */ -static inline struct page *alloc_pages_node_noprof (int nid,gfp_t gfp_mask, -unsigned int order) -{ -if (nid ==NUMA_NO_NODE ) - nid =numa_mem_id (); - -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} +struct page *alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order);#define alloc_pages_node (...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__))mm/page_alloc.c +12
-1@@ -5427,7 +5427,18 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order,set_page_refcounted (page);return page; } -EXPORT_SYMBOL (__alloc_pages_noprof); + +struct page *alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) +{ +if (nid ==NUMA_NO_NODE ) + nid =numa_mem_id (); + +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); +warn_if_node_offline (nid, gfp_mask); + +return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); +} +EXPORT_SYMBOL (alloc_pages_node_noprof);struct folio *__folio_alloc_noprof (gfp_t gfp,unsigned int order,int preferred_nid,nodemask_t *nodemask)mm/page_alloc.h +4
-0@@ -244,6 +244,10 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsignedalloc_hooks (alloc_frozen_pages_nolock_noprof(__VA_ARGS__ )) void free_frozen_pages_nolock(struct page *page, unsigned int order); +struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid, +nodemask_t *nodemask); +#define __alloc_pages (...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) +extern void zone_pcp_reset (struct zone *zone);extern void zone_pcp_disable (struct zone *zone);extern void zone_pcp_enable (struct zone *zone);
PATCH v3 15/16 mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
In reply to Brendan JackmanNow that alloc_pages has an entrypoint that allows passing alloc_flags, we can take advantage of this to start removing GFP flags that are only used for mm-internal stuff.
This requires also plumbing the alloc_flags into some more of the allocator code, in particular __alloc_pages[_noprof]() gets an alloc_flags arg to go along with its callees, and we now need to pass those flags deeper into the allocator so they can reach the alloc_tag code.
No functional change intended.
--- mm/alloc_tag.c | 22 ++++++---------------- mm/compaction.c | 4 ++-- mm/internal.h | 1 - mm/page_alloc.c | 42 ++++++++++++++++++++++++------------------ mm/page_alloc.h | 17 +++++++++++++++-- mm/page_frag_cache.c | 4 ++-- 6 files changed, 49 insertions(+), 41 deletions(-)
mm/alloc_tag.c +6 -16
@@ -15,6 +15,8 @@#include <linux/vmalloc.h> #include <linux/kmemleak.h> +#include "internal.h" +#define ALLOCINFO_FILE_NAME "allocinfo"#define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag))#define SECTION_START (NAME ) (CODETAG_SECTION_START_PREFIX NAME)
@@ -783,19 +785,6 @@ struct pfn_pool {#define PFN_POOL_SIZE ((PAGE_SIZE - offsetof(struct pfn_pool, pfns)) / \ sizeof(unsigned long)) - -/* -* Skip early PFN recording for a page allocation. Reuses the -* %__GFP_NO_OBJ_EXT bit. Used by __alloc_tag_add_early_pfn() to avoid -* recursion when allocating pages for the early PFN tracking list -* itself. -* -* Codetags of the pages allocated with __GFP_NO_CODETAG should be -* cleared (via clear_page_tag_ref()) before freeing the pages to prevent -* alloc_tag_sub_check() from triggering a warning. -*/ -#define __GFP_NO_CODETAG __GFP_NO_OBJ_EXT -static struct pfn_pool *current_pfn_pool__initdata ;static void __init __alloc_tag_add_early_pfn (unsigned long pfn)
@@ -806,7 +795,8 @@ static void __init __alloc_tag_add_early_pfn(unsigned long pfn) do { pool = READ_ONCE(current_pfn_pool); if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE) { - struct page *new_page = alloc_page(__GFP_HIGH | __GFP_NO_CODETAG); + struct page *new_page = __alloc_pages(__GFP_HIGH, 0, numa_mem_id(), + NULL, ALLOC_NO_CODETAG); struct pfn_pool *new; if (!new_page) {
@@ -837,7 +827,7 @@ typedef void alloc_tag_add_func(unsigned long pfn);static alloc_tag_add_func __rcu *alloc_tag_add_early_pfn_ptr__refdata =RCU_INITIALIZER (__alloc_tag_add_early_pfn); -void alloc_tag_add_early_pfn (unsigned long pfn,gfp_t gfp_flags) +void alloc_tag_add_early_pfn (unsigned long pfn,unsigned int alloc_flags) {alloc_tag_add_func *alloc_tag_add;
@@ -845,7 +835,7 @@ void alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags)return ;/* Skip allocations for the tracking list itself to avoid recursion. */ -if (gfp_flags & __GFP_NO_CODETAG) +if (alloc_flags &ALLOC_NO_CODETAG )return ;rcu_read_lock ();
mm/compaction.c +2 -2
@@ -83,7 +83,7 @@ static inline bool is_via_compact_memory(int order) { return false; }static struct page *mark_allocated_noprof (struct page *page,unsigned int order,gfp_t gfp_flags) { -post_alloc_hook (page, order, __GFP_MOVABLE); +post_alloc_hook (page, order, __GFP_MOVABLE,ALLOC_DEFAULT );set_page_refcounted (page);return page; }
@@ -1851,7 +1851,7 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long da } dst = (struct folio *)freepage; -post_alloc_hook (&dst->page, order, __GFP_MOVABLE); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE,ALLOC_DEFAULT );set_page_refcounted (&dst->page);if (order)prep_compound_page (&dst->page, order);
mm/internal.h +0 -1
@@ -1237,7 +1237,6 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,enum ttu_flags ;struct tlbflush_unmap_batch ; -/* * only for MM internal work items which do not depend on * any allocations or locks which might depend on allocations
mm/page_alloc.c +24 -18
141 lines not on this page · raw message
mm/page_alloc.h +15 -2
@@ -49,6 +49,16 @@#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_NOLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +/* +* Skip early PFN recording for a page allocation. Used by +* __alloc_tag_add_early_pfn() to avoid recursion when allocating pages for the +* early PFN tracking list itself. +* +* Codetags of the pages allocated with __GFP_NO_CODETAG should be +* cleared (via clear_page_tag_ref()) before freeing the pages to prevent +* alloc_tag_sub_check() from triggering a warning. +*/ +#define ALLOC_NO_CODETAG 0x1000/* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
@@ -84,6 +94,8 @@ struct alloc_context {*/ enum zone_type highest_zoneidx;bool spread_dirty_pages; +/* Only flags that are global to the whole allocation go here. */ +unsigned int alloc_flags; };/*
@@ -214,7 +226,8 @@ static inline struct page *pageblock_pfn_to_page(unsigned long start_pfn,extern void __free_pages_core (struct page *page,unsigned int order,enum meminit_context context); -void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags); +void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags, +unsigned int alloc_flags);extern bool free_pages_prepare (struct page *page,unsigned int order);extern int user_min_free_kbytes;
@@ -245,7 +258,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsignedvoid free_frozen_pages_nolock (struct page *page,unsigned int order);struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid, -nodemask_t *nodemask); +nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_pages (...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__))extern void zone_pcp_reset (struct zone *zone);
mm/page_frag_cache.c +2 -2
@@ -57,10 +57,10 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, gfp_mask = (gfp_mask & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; page =__alloc_pages (gfp_mask,PAGE_FRAG_CACHE_MAX_ORDER , -numa_mem_id (),NULL ); +numa_mem_id (),NULL ,ALLOC_DEFAULT );#endif if (unlikely (!page)) { - page =__alloc_pages (gfp,0 ,numa_mem_id (),NULL ); + page =__alloc_pages (gfp,0 ,numa_mem_id (),NULL ,ALLOC_DEFAULT ); order =0 ; }
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v3 15/16] mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [Low] The fallback #else definition of pgalloc_tag_add was not updated to accept unsigned int alloc_flags.
Pre-existing issues:
- [Critical] TOCTOU race in __alloc_tag_add_early_pfn leading to an out-of-bounds array write. --
commit ccee7f6d4bcf9d00da38c878e97dd852c5e649b3
Author: Brendan Jackman <[email protected]>
mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
This commit replaces the mm-internal __GFP_NO_CODETAG flag with a new alloc_flags bit, ALLOC_NO_CODETAG. It plumbs alloc_flags through several allocator functions to allow passing this flag directly to the alloc_tag code.
Quoted 4 lines
mm/alloc_tag.c +0
-0
[ ... ]
mm/alloc_tag.c static void __init __alloc_tag_add_early_pfn(unsigned long pfn) · line 800
Quoted 8 lines
mm/alloc_tag.c +2
-1@@ -806,7 +795,8 @@ static void __init __alloc_tag_add_early_pfn(unsigned long pfn) do { pool = READ_ONCE(current_pfn_pool); if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE) { - struct page *new_page = alloc_page(__GFP_HIGH | __GFP_NO_CODETAG); + struct page *new_page = __alloc_pages(__GFP_HIGH, 0, numa_mem_id(), + NULL, ALLOC_NO_CODETAG); struct pfn_pool *new;
[Severity: Critical]
This is a pre-existing issue, but is there a time-of-check to time-of-use race condition here?
If multiple CPUs are active during the deferred init window, could they concurrently observe the same remaining capacity?
CPU1:
if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE)
CPU2:
if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE)
If both see 1 remaining slot, won't they both proceed to call atomic_fetch_add() on the pool counter further down in this function? This could cause an out-of-bounds array write at pool->pfns[PFN_POOL_SIZE].
[ ... ]
Quoted 4 lines
mm/page_alloc.c +0
-0
[ ... ]
Quoted 5 lines
mm/page_alloc.c +0
-0@@ -1263,17 +1263,17 @@ void __pgalloc_tag_add(struct page *page, struct task_struct *task,alloc_tag_add_early_pfn (page_to_pfn (page), alloc_flags);if (task->alloc_tag)alloc_tag_set_inaccurate (task->alloc_tag); }
mm/page_alloc.c void __pgalloc_tag_add(struct page *page, struct task_struct *task, · line 1274
Quoted 10 lines
mm/page_alloc.c +2
-2}static inline void pgalloc_tag_add (struct page *page,struct task_struct *task, -unsigned int nr,gfp_t gfp_flags) +unsigned int nr,unsigned int alloc_flags) {if (mem_alloc_profiling_enabled ()) -__pgalloc_tag_add (page, task, nr, gfp_flags); +__pgalloc_tag_add (page, task, nr, alloc_flags); }
[Severity: Low]
Does this change miss updating the fallback definition of pgalloc_tag_add() when memory allocation profiling is disabled?
If the kernel is compiled without mem-alloc profiling, the dummy function signature appears to still use the old type:
static inline void pgalloc_tag_add(struct page *page, struct task_struct *task,
unsigned int nr, gfp_t gfp_flags) {}Since callers like post_alloc_hook() are now passing an unsigned int for alloc_flags, won't this cause a Sparse type mismatch warning because gfp_t is a restricted bitwise type?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Hao Ge
reviewerRe: [PATCH v3 15/16] mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
CC Andrew and Suren
Quoted 40 lines · On 2026/6/29 23:56, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [Low] The fallback#elsedefinition ofpgalloc_tag_addwas not updated to acceptunsigned int alloc_flags.Pre-existing issues:
- [Critical] TOCTOU race in__alloc_tag_add_early_pfnleading to an out-of-bounds array write. --commit
ccee7f6d4bcf9d00da38c878e97dd852c5e649b3
Author: Brendan Jackman <[email protected]>mm: replace
__GFP_NO_CODETAGwithALLOC_NO_CODETAGThis commit replaces the mm-internal
__GFP_NO_CODETAGflag with a newalloc_flagsbit,ALLOC_NO_CODETAG. It plumbsalloc_flagsthrough several allocator functions to allow passing this flag directly to thealloc_tagcode.Quoted 4 lines
mm/alloc_tag.c +0
-0[ ... ]
mm/alloc_tag.c static void __init __alloc_tag_add_early_pfn(unsigned long pfn) · line 800
Quoted 8 lines
mm/alloc_tag.c +2
-1@@ -806,7 +795,8 @@ static void __init __alloc_tag_add_early_pfn(unsigned long pfn) do { pool = READ_ONCE(current_pfn_pool); if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE) { - struct page *new_page = alloc_page(__GFP_HIGH | __GFP_NO_CODETAG); + struct page *new_page = __alloc_pages(__GFP_HIGH, 0, numa_mem_id(), + NULL, ALLOC_NO_CODETAG); struct pfn_pool *new;[Severity: Critical]
This is a pre-existing issue, but is there a time-of-check to time-of-use race condition here?If multiple CPUs are active during the deferred init window, could they concurrently observe the same remaining capacity?
CPU1:
Quoted 8 lines
if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE)CPU2:
if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE)If both see 1 remaining slot, won't they both proceed to call
atomic_fetch_add()on the pool counter further down in this function? This could cause an out-of-bounds array write at pool->pfns[PFN_POOL_SIZE].
It's interesting.
I didn't get this warning from Sashiko when I submitted the patch earlier,
I'm not sure what changed on its end to trigger it now.
No worries though, unless I've overlooked some detail,
I believe this is a false positive
Even if both CPUs see 1 remaining slot and both skip the pool-growing branch,
they still have to go through atomic_cmpxchg to actually claim it.
static void __init __alloc_tag_add_early_pfn(unsigned long pfn) {
struct pfn_pool *pool;
int idx;
do {
pool = READ_ONCE(current_pfn_pool);
if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE) {
.........
}
idx = atomic_read(&pool->count);
if (idx >= PFN_POOL_SIZE)
continue;
if (atomic_cmpxchg(&pool->count, idx, idx + 1) == idx) break;
} while (1);
pool->pfns[idx] = pfn;
}
Only one wins, the other gets a CAS failure and loops back.
There's no path where both write to pfns[PFN_POOL_SIZE].
Given that, I think we can ignore this Sashiko report and
move forward with Brendan's patch.
Thanks
Best Regards
Hao
Quoted 35 lines
[ ... ]
Quoted 4 lines
mm/page_alloc.c +0
-0[ ... ]
Quoted 5 lines
mm/page_alloc.c +0
-0@@ -1263,17 +1263,17 @@ void __pgalloc_tag_add(struct page *page, struct task_struct *task,alloc_tag_add_early_pfn (page_to_pfn (page), alloc_flags);if (task->alloc_tag)alloc_tag_set_inaccurate (task->alloc_tag); }mm/page_alloc.c void __pgalloc_tag_add(struct page *page, struct task_struct *task, · line 1274
Quoted 10 lines
mm/page_alloc.c +2
-2}static inline void pgalloc_tag_add (struct page *page,struct task_struct *task, -unsigned int nr,gfp_t gfp_flags) +unsigned int nr,unsigned int alloc_flags) {if (mem_alloc_profiling_enabled ()) -__pgalloc_tag_add (page, task, nr, gfp_flags); +__pgalloc_tag_add (page, task, nr, alloc_flags); }[Severity: Low]
Does this change miss updating the fallback definition ofpgalloc_tag_add()when memory allocation profiling is disabled?If the kernel is compiled without mem-alloc profiling, the dummy function signature appears to still use the old type:
static inline void pgalloc_tag_add(struct page *page, struct
task_struct*task,unsigned int nr, gfp_t gfp_flags) {}Since callers like
post_alloc_hook()are now passing an unsigned int foralloc_flags, won't this cause a Sparse type mismatch warning becausegfp_tis a restricted bitwise type?
Hao Ge
reviewerRe: [PATCH v3 15/16] mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
In reply to Brendan JackmanHi Brendan
Quoted 27 lines · Brendan Jackman
Now that
alloc_pageshas an entrypoint that allows passingalloc_flags, we can take advantage of this to start removing GFP flags that are only used for mm-internal stuff.This requires also plumbing the
alloc_flagsinto some more of the allocator code, in particular __alloc_pages[_noprof]() gets analloc_flagsarg to go along with its callees, and we now need to pass those flags deeper into the allocator so they can reach thealloc_tagcode.No functional change intended.
Signed-off-by Brendan Jackman <[email protected]>--- mm/alloc_tag.c | 22 ++++++---------------- mm/compaction.c | 4 ++-- mm/internal.h | 1 - mm/page_alloc.c | 42 ++++++++++++++++++++++++------------------ mm/page_alloc.h | 17 +++++++++++++++-- mm/page_frag_cache.c | 4 ++-- 6 files changed, 49 insertions(+), 41 deletions(-)mm/alloc_tag.c +0
-0
mm/alloc_tag.c line 18
Quoted 4 lines
mm/alloc_tag.c +1
-0@@ -15,6 +15,8 @@#include <linux/vmalloc.h> #include <linux/kmemleak.h> +#include "internal.h"
Should we include page_alloc.h here, as we call __alloc_pages later in this file?
Quoted 35 lines
mm/alloc_tag.c +3
-14+#define ALLOCINFO_FILE_NAME "allocinfo"#define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag))#define SECTION_START (NAME ) (CODETAG_SECTION_START_PREFIX NAME)@@ -783,19 +785,6 @@ struct pfn_pool {#define PFN_POOL_SIZE ((PAGE_SIZE - offsetof(struct pfn_pool, pfns)) / \ sizeof(unsigned long)) - -/* -* Skip early PFN recording for a page allocation. Reuses the -* %__GFP_NO_OBJ_EXT bit. Used by __alloc_tag_add_early_pfn() to avoid -* recursion when allocating pages for the early PFN tracking list -* itself. -* -* Codetags of the pages allocated with __GFP_NO_CODETAG should be -* cleared (via clear_page_tag_ref()) before freeing the pages to prevent -* alloc_tag_sub_check() from triggering a warning. -*/ -#define __GFP_NO_CODETAG __GFP_NO_OBJ_EXT -static struct pfn_pool *current_pfn_pool__initdata ;static void __init __alloc_tag_add_early_pfn (unsigned long pfn)@@ -806,7 +795,8 @@ static void __init __alloc_tag_add_early_pfn(unsigned long pfn) do { pool = READ_ONCE(current_pfn_pool); if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE) { - struct page *new_page = alloc_page(__GFP_HIGH | __GFP_NO_CODETAG); + struct page *new_page = __alloc_pages(__GFP_HIGH, 0, numa_mem_id(), + NULL, ALLOC_NO_CODETAG); struct pfn_pool *new; if (!new_page) {
mm/alloc_tag.c typedef void alloc_tag_add_func(unsigned long pfn); · line 830
Quoted 5 lines
mm/alloc_tag.c +1
-1@@ -837,7 +827,7 @@ typedef void alloc_tag_add_func(unsigned long pfn);static alloc_tag_add_func __rcu *alloc_tag_add_early_pfn_ptr__refdata =RCU_INITIALIZER (__alloc_tag_add_early_pfn); -void alloc_tag_add_early_pfn (unsigned long pfn,gfp_t gfp_flags) +void alloc_tag_add_early_pfn (unsigned long pfn,unsigned int alloc_flags)
alloc_tag_add_early_pfn() has three occurrences across the codebase:
1. Definition in mm/alloc_tag.c:830:
void alloc_tag_add_early_pfn(unsigned long pfn, unsigned int alloc_flags)
2. Declaration in include/linux/alloc_tag.h:166:
void alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags)
3. Static inline stub in include/linux/alloc_tag.h:170:
static inline void alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags) {}
This patch updates the definition in alloc_tag.c to take unsigned int alloc_flags,
but the two declarations in alloc_tag.h are left with the old gfp_t gfp_flags signature
These should be updated to match.
Quoted 63 lines
mm/alloc_tag.c +1
-1{alloc_tag_add_func *alloc_tag_add;@@ -845,7 +835,7 @@ void alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags)return ;/* Skip allocations for the tracking list itself to avoid recursion. */ -if (gfp_flags & __GFP_NO_CODETAG) +if (alloc_flags &ALLOC_NO_CODETAG )return ;rcu_read_lock ();mm/compaction.c +2
-2@@ -83,7 +83,7 @@ static inline bool is_via_compact_memory(int order) { return false; }static struct page *mark_allocated_noprof (struct page *page,unsigned int order,gfp_t gfp_flags) { -post_alloc_hook (page, order, __GFP_MOVABLE); +post_alloc_hook (page, order, __GFP_MOVABLE,ALLOC_DEFAULT );set_page_refcounted (page);return page; }@@ -1851,7 +1851,7 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long da } dst = (struct folio *)freepage; -post_alloc_hook (&dst->page, order, __GFP_MOVABLE); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE,ALLOC_DEFAULT );set_page_refcounted (&dst->page);if (order)prep_compound_page (&dst->page, order);mm/internal.h +0
-1@@ -1237,7 +1237,6 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,enum ttu_flags ;struct tlbflush_unmap_batch ; -/* * only for MM internal work items which do not depend on * any allocations or locks which might depend on allocations mm/page_alloc.c +1
-1@@ -1249,7 +1249,7 @@ void __clear_page_tag_ref(struct page *page)/* Should be called only if mem_alloc_profiling_enabled() */ static noinline void __pgalloc_tag_add(struct page *page, struct task_struct *task, -unsigned int nr,gfp_t gfp_flags) +unsigned int nr,unsigned int alloc_flags) {union pgtag_ref_handle handle;union codetag_ref ref;@@ -1263,17 +1263,17 @@ void __pgalloc_tag_add(struct page *page, struct task_struct *task,* page_ext is not available yet, record the pfn so we can * clear the tag ref later when page_ext is initialized. */
mm/page_alloc.c void __pgalloc_tag_add(struct page *page, struct task_struct *task, · line 1273
Quoted 10 lines
mm/page_alloc.c +2
-2-alloc_tag_add_early_pfn (page_to_pfn (page), gfp_flags); +alloc_tag_add_early_pfn (page_to_pfn (page), alloc_flags);if (task->alloc_tag)alloc_tag_set_inaccurate (task->alloc_tag); } }static inline void pgalloc_tag_add (struct page *page,struct task_struct *task, -unsigned int nr,gfp_t gfp_flags) +unsigned int nr,unsigned int alloc_flags)
Same situation as alloc_tag_add_early_pfn() — the #else stub
at mm/page_alloc.c:1309 still uses gfp_t gfp_flags instead of unsigned int
alloc_flags.
Quoted 138 lines
mm/page_alloc.c +21
-15{if (mem_alloc_profiling_enabled ()) -__pgalloc_tag_add (page, task, nr, gfp_flags); +__pgalloc_tag_add (page, task, nr, alloc_flags); }/* Should be called only if mem_alloc_profiling_enabled() */ @@ -1810,7 +1810,7 @@ static inline bool should_skip_init(gfp_t flags) }inline void post_alloc_hook (struct page *page,unsigned int order, -gfp_t gfp_flags) +gfp_t gfp_flags,unsigned int alloc_flags) {const bool zero_tags = gfp_flags & __GFP_ZEROTAGS; bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) &&@@ -1861,13 +1861,13 @@ inline void post_alloc_hook(struct page *page, unsigned int order,set_page_owner (page, order, gfp_flags);page_table_check_alloc (page, order); -pgalloc_tag_add (page, current,1 << order, gfp_flags); +pgalloc_tag_add (page, current,1 << order, alloc_flags); }static void prep_new_page (struct page *page,unsigned int order,gfp_t gfp_flags,unsigned int alloc_flags) { -post_alloc_hook (page, order, gfp_flags); +post_alloc_hook (page, order, gfp_flags, alloc_flags);if (order && (gfp_flags & __GFP_COMP))prep_compound_page (page, order);@@ -4791,8 +4791,12 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* The fast path uses conservative alloc_flags to succeed only until * kswapd needs to be woken up, and to avoid the cost of setting up * alloc_flags precisely. So we do that now. +* +* Can't just or alloc_flags if it contains WMARK bits, but those flags +* shouldn't be set in ac->alloc_flags. */ - alloc_flags =alloc_flags_slowpath (gfp_mask, order); +VM_WARN_ON (ac->alloc_flags &ALLOC_WMARK_MASK ); + alloc_flags = ac->alloc_flags |alloc_flags_slowpath (gfp_mask, order);/* * We need to recalculate the starting point for the zonelist iterator @@ -4834,7 +4838,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags =__gfp_pfmemalloc_flags (gfp_mask);if (reserve_flags) alloc_flags =alloc_flags_cma (gfp_mask, reserve_flags) | - (alloc_flags &ALLOC_KSWAPD ); + ac->alloc_flags | (alloc_flags &ALLOC_KSWAPD );/* * Reset the nodemask and zonelist iterators if memory policies can be @@ -5236,7 +5240,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,return nr_populated; failed: - page =__alloc_pages_noprof (gfp,0 , preferred_nid, nodemask); + page =__alloc_pages_noprof (gfp,0 , preferred_nid, nodemask,ALLOC_DEFAULT );if (page) page_array[nr_populated++] = page; goto out;@@ -5344,11 +5348,13 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, {struct page *page;gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ -struct alloc_context ac = { }; +struct alloc_context ac = { + .alloc_flags = alloc_flags, + };unsigned int fastpath_alloc_flags = alloc_flags;/* Other flags could be supported later if needed. */ -if (WARN_ON (alloc_flags & ~ALLOC_NOLOCK )) +if (WARN_ON (alloc_flags & ~(ALLOC_NOLOCK |ALLOC_NO_CODETAG )))return NULL ;if (!alloc_order_allowed (gfp, order, alloc_flags))@@ -5417,12 +5423,12 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,EXPORT_SYMBOL (__alloc_frozen_pages_noprof);struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order, -int preferred_nid,nodemask_t *nodemask) +int preferred_nid,nodemask_t *nodemask,unsigned int alloc_flags) {struct page *page; page =__alloc_frozen_pages_noprof (gfp, order, preferred_nid, nodemask, -ALLOC_DEFAULT ); + alloc_flags);if (page)set_page_refcounted (page);return page;@@ -5436,7 +5442,7 @@ struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int orderVM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );warn_if_node_offline (nid, gfp_mask); -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); +return __alloc_pages_noprof (gfp_mask, order, nid,NULL ,ALLOC_DEFAULT ); }EXPORT_SYMBOL (alloc_pages_node_noprof);@@ -5444,7 +5450,7 @@ struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_ nodemask_t *nodemask) {struct page *page =__alloc_pages_noprof (gfp | __GFP_COMP, order, - preferred_nid, nodemask); + preferred_nid, nodemask,ALLOC_DEFAULT );return page_rmappable_folio (page); }EXPORT_SYMBOL (__folio_alloc_noprof);@@ -7126,7 +7132,7 @@ static void split_free_frozen_pages(struct list_head *list, gfp_t gfp_mask)list_for_each_entry_safe (page, next, &list[order], lru) {int i; -post_alloc_hook (page, order, gfp_mask); +post_alloc_hook (page, order, gfp_mask,ALLOC_DEFAULT );if (!order)continue ;@@ -7331,7 +7337,7 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,struct page *head =pfn_to_page (start);check_new_pages (head, order); -prep_new_page (head, order, gfp_mask,0 ); +prep_new_page (head, order, gfp_mask,ALLOC_DEFAULT ); }else { ret = -EINVAL ; WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n",mm/page_alloc.h +0
-0@@ -49,6 +49,16 @@#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_NOLOCK 0x400/* Only use spin_trylock in allocation path */
mm/page_alloc.h line 60
Quoted 10 lines
mm/page_alloc.h +9
-0#define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +/* +* Skip early PFN recording for a page allocation. Used by +* __alloc_tag_add_early_pfn() to avoid recursion when allocating pages for the +* early PFN tracking list itself. +* +* Codetags of the pages allocated with __GFP_NO_CODETAG should be +* cleared (via clear_page_tag_ref()) before freeing the pages to prevent +* alloc_tag_sub_check() from triggering a warning. +*/
I originally wrote this lengthy comment because the logic lives inside alloc_tag.c.
I wanted to document all the context to avoid confusion when revisiting this code later on.
We've since replaced __GFP_NO_CODETAG with ALLOC_NO_CODETAG, a generic alloc_flags bit defined in page_alloc.h.
The original long comment is no longer accurate:
Given that, I suggest updating it to the following:
/*
* Avoid alloc_tag recursion for internal allocations.
* Callers must clear_page_tag_ref() before
* freeing to avoid warnings from alloc_tag_sub_check().
*/
Thanks
Best Regards
Hao
Quoted 50 lines
mm/page_alloc.h +6
-2+#define ALLOC_NO_CODETAG 0x1000/* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)@@ -84,6 +94,8 @@ struct alloc_context {*/ enum zone_type highest_zoneidx;bool spread_dirty_pages; +/* Only flags that are global to the whole allocation go here. */ +unsigned int alloc_flags; };/* @@ -214,7 +226,8 @@ static inline struct page *pageblock_pfn_to_page(unsigned long start_pfn,extern void __free_pages_core (struct page *page,unsigned int order,enum meminit_context context); -void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags); +void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags, +unsigned int alloc_flags);extern bool free_pages_prepare (struct page *page,unsigned int order);extern int user_min_free_kbytes;@@ -245,7 +258,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsignedvoid free_frozen_pages_nolock (struct page *page,unsigned int order);struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid, -nodemask_t *nodemask); +nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_pages (...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__))extern void zone_pcp_reset (struct zone *zone);mm/page_frag_cache.c +2
-2@@ -57,10 +57,10 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, gfp_mask = (gfp_mask & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; page =__alloc_pages (gfp_mask,PAGE_FRAG_CACHE_MAX_ORDER , -numa_mem_id (),NULL ); +numa_mem_id (),NULL ,ALLOC_DEFAULT );#endif if (unlikely (!page)) { - page =__alloc_pages (gfp,0 ,numa_mem_id (),NULL ); + page =__alloc_pages (gfp,0 ,numa_mem_id (),NULL ,ALLOC_DEFAULT ); order =0 ; }
Brendan Jackman
reviewerRe: [PATCH v3 15/16] mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
Quoted 34 lines · Hao Ge
Hi Brendan
Quoted 30 lines · Brendan Jackman
Now that
alloc_pageshas an entrypoint that allows passingalloc_flags, we can take advantage of this to start removing GFP flags that are only used for mm-internal stuff.This requires also plumbing the
alloc_flagsinto some more of the allocator code, in particular __alloc_pages[_noprof]() gets analloc_flagsarg to go along with its callees, and we now need to pass those flags deeper into the allocator so they can reach thealloc_tagcode.No functional change intended.
Signed-off-by Brendan Jackman <[email protected]>--- mm/alloc_tag.c | 22 ++++++---------------- mm/compaction.c | 4 ++-- mm/internal.h | 1 - mm/page_alloc.c | 42 ++++++++++++++++++++++++------------------ mm/page_alloc.h | 17 +++++++++++++++-- mm/page_frag_cache.c | 4 ++-- 6 files changed, 49 insertions(+), 41 deletions(-)mm/alloc_tag.c +0
-0@@ -15,6 +15,8 @@#include <linux/vmalloc.h> #include <linux/kmemleak.h>
Quoted 5 lines
mm/alloc_tag.c line 18
Quoted 1 line
mm/alloc_tag.c +1
-0+#include "internal.h" Should we include
page_alloc.hhere, as we call__alloc_pageslater in this file?
Yeah, there are a few build failures due to me not doing a broad enough build. From now on I will just wait for allmodconfig instead of trying to be clever with my build tests, sorry about this.
Also, this suggests that I have not actually re-tested the alloc_tag code since v3 so I must repeat the test described in my cover letter (I just manually enable the feature and check the kernel boots) for v4.
Quoted 57 lines
Quoted 35 lines
mm/alloc_tag.c +3
-14+#define ALLOCINFO_FILE_NAME "allocinfo"#define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag))#define SECTION_START (NAME ) (CODETAG_SECTION_START_PREFIX NAME)@@ -783,19 +785,6 @@ struct pfn_pool {#define PFN_POOL_SIZE ((PAGE_SIZE - offsetof(struct pfn_pool, pfns)) / \ sizeof(unsigned long)) - -/* -* Skip early PFN recording for a page allocation. Reuses the -* %__GFP_NO_OBJ_EXT bit. Used by __alloc_tag_add_early_pfn() to avoid -* recursion when allocating pages for the early PFN tracking list -* itself. -* -* Codetags of the pages allocated with __GFP_NO_CODETAG should be -* cleared (via clear_page_tag_ref()) before freeing the pages to prevent -* alloc_tag_sub_check() from triggering a warning. -*/ -#define __GFP_NO_CODETAG __GFP_NO_OBJ_EXT -static struct pfn_pool *current_pfn_pool__initdata ;static void __init __alloc_tag_add_early_pfn (unsigned long pfn)@@ -806,7 +795,8 @@ static void __init __alloc_tag_add_early_pfn(unsigned long pfn) do { pool = READ_ONCE(current_pfn_pool); if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE) { - struct page *new_page = alloc_page(__GFP_HIGH | __GFP_NO_CODETAG); + struct page *new_page = __alloc_pages(__GFP_HIGH, 0, numa_mem_id(), + NULL, ALLOC_NO_CODETAG); struct pfn_pool *new; if (!new_page) {mm/alloc_tag.c typedef void alloc_tag_add_func(unsigned long pfn); · line 830
Quoted 5 lines
mm/alloc_tag.c +1
-1@@ -837,7 +827,7 @@ typedef void alloc_tag_add_func(unsigned long pfn);static alloc_tag_add_func __rcu *alloc_tag_add_early_pfn_ptr__refdata =RCU_INITIALIZER (__alloc_tag_add_early_pfn); -void alloc_tag_add_early_pfn (unsigned long pfn,gfp_t gfp_flags) +void alloc_tag_add_early_pfn (unsigned long pfn,unsigned int alloc_flags)
alloc_tag_add_early_pfn()has three occurrences across the codebase:1. Definition in
mm/alloc_tag.c:830:void
alloc_tag_add_early_pfn(unsigned long pfn, unsigned int alloc_flags)2. Declaration in
include/linux/alloc_tag.h:166:void
alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags)3. Static inline stub in
include/linux/alloc_tag.h:170:static inline void
alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags){}
Quoted 7 lines
This patch updates the definition in
alloc_tag.cto take unsigned intalloc_flags,but the two declarations in
alloc_tag.hare left with the oldgfp_tgfp_flagssignatureThese should be updated to match.
Yeah ditto, sorry about this and thanks for the review.
Hao Ge
reviewerRe: [PATCH v3 15/16] mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
Hi Brendan
Quoted 35 lines · Brendan Jackman
Quoted 34 lines · Hao Ge
Hi Brendan
Quoted 30 lines · Brendan Jackman
Now that
alloc_pageshas an entrypoint that allows passingalloc_flags, we can take advantage of this to start removing GFP flags that are only used for mm-internal stuff.This requires also plumbing the
alloc_flagsinto some more of the allocator code, in particular __alloc_pages[_noprof]() gets analloc_flagsarg to go along with its callees, and we now need to pass those flags deeper into the allocator so they can reach thealloc_tagcode.No functional change intended.
Signed-off-by Brendan Jackman <[email protected]>--- mm/alloc_tag.c | 22 ++++++---------------- mm/compaction.c | 4 ++-- mm/internal.h | 1 - mm/page_alloc.c | 42 ++++++++++++++++++++++++------------------ mm/page_alloc.h | 17 +++++++++++++++-- mm/page_frag_cache.c | 4 ++-- 6 files changed, 49 insertions(+), 41 deletions(-)mm/alloc_tag.c +0
-0@@ -15,6 +15,8 @@#include <linux/vmalloc.h> #include <linux/kmemleak.h>
Quoted 7 lines
Quoted 4 lines
mm/alloc_tag.c line 18
Quoted 1 line
mm/alloc_tag.c +1
-0+#include "internal.h" Should we include
page_alloc.hhere, as we call__alloc_pageslater in this file?Yeah, there are a few build failures due to me not doing a broad enough build. From now on I will just wait for allmodconfig instead of trying to be clever with my build tests, sorry about this.
No worries at all. For the alloc_tag build error, it depends on whether
CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled.
Quoted 4 lines
Also, this suggests that I have not actually re-tested the
alloc_tagcode since v3 so I must repeat the test described in my cover letter (I just manually enable the feature and check the kernel boots) for v4.
Thanks a lot for your work. I merged your patch locally via my automation bot,
and verified the basic functionality of alloc_tag.
The alloc_tag feature works well so far.
By the way, you may need to rebase your patchset on top of mm-new.
This patch has been merged into mm-new and will cause a minor conflict,
assuming the mm-new tree has not made further modifications to this patch:
Thanks
Best Regards
Hao
Quoted 64 lines
Quoted 56 lines
Quoted 35 lines
mm/alloc_tag.c +3
-14+#define ALLOCINFO_FILE_NAME "allocinfo"#define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag))#define SECTION_START (NAME ) (CODETAG_SECTION_START_PREFIX NAME)@@ -783,19 +785,6 @@ struct pfn_pool {#define PFN_POOL_SIZE ((PAGE_SIZE - offsetof(struct pfn_pool, pfns)) / \ sizeof(unsigned long)) - -/* -* Skip early PFN recording for a page allocation. Reuses the -* %__GFP_NO_OBJ_EXT bit. Used by __alloc_tag_add_early_pfn() to avoid -* recursion when allocating pages for the early PFN tracking list -* itself. -* -* Codetags of the pages allocated with __GFP_NO_CODETAG should be -* cleared (via clear_page_tag_ref()) before freeing the pages to prevent -* alloc_tag_sub_check() from triggering a warning. -*/ -#define __GFP_NO_CODETAG __GFP_NO_OBJ_EXT -static struct pfn_pool *current_pfn_pool__initdata ;static void __init __alloc_tag_add_early_pfn (unsigned long pfn)@@ -806,7 +795,8 @@ static void __init __alloc_tag_add_early_pfn(unsigned long pfn) do { pool = READ_ONCE(current_pfn_pool); if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE) { - struct page *new_page = alloc_page(__GFP_HIGH | __GFP_NO_CODETAG); + struct page *new_page = __alloc_pages(__GFP_HIGH, 0, numa_mem_id(), + NULL, ALLOC_NO_CODETAG); struct pfn_pool *new; if (!new_page) {mm/alloc_tag.c typedef void alloc_tag_add_func(unsigned long pfn); · line 830
Quoted 5 lines
mm/alloc_tag.c +1
-1@@ -837,7 +827,7 @@ typedef void alloc_tag_add_func(unsigned long pfn);static alloc_tag_add_func __rcu *alloc_tag_add_early_pfn_ptr__refdata =RCU_INITIALIZER (__alloc_tag_add_early_pfn); -void alloc_tag_add_early_pfn (unsigned long pfn,gfp_t gfp_flags) +void alloc_tag_add_early_pfn (unsigned long pfn,unsigned int alloc_flags)
alloc_tag_add_early_pfn()has three occurrences across the codebase:1. Definition in
mm/alloc_tag.c:830:void
alloc_tag_add_early_pfn(unsigned long pfn, unsigned int alloc_flags)2. Declaration in
include/linux/alloc_tag.h:166:void
alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags)3. Static inline stub in
include/linux/alloc_tag.h:170:static inline void
alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags){}Quoted 7 lines
This patch updates the definition in
alloc_tag.cto take unsigned intalloc_flags,but the two declarations in
alloc_tag.hare left with the oldgfp_tgfp_flagssignatureThese should be updated to match.
Yeah ditto, sorry about this and thanks for the review.
Zi Yan
reviewerRe: [PATCH v3 15/16] mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
Quoted 66 lines · Hao Ge
Hi Brendan
Quoted 35 lines · Brendan Jackman
Quoted 34 lines · Hao Ge
Hi Brendan
Quoted 30 lines · Brendan Jackman
Now that
alloc_pageshas an entrypoint that allows passingalloc_flags, we can take advantage of this to start removing GFP flags that are only used for mm-internal stuff.This requires also plumbing the
alloc_flagsinto some more of the allocator code, in particular __alloc_pages[_noprof]() gets analloc_flagsarg to go along with its callees, and we now need to pass those flags deeper into the allocator so they can reach thealloc_tagcode.No functional change intended.
Signed-off-by Brendan Jackman <[email protected]>--- mm/alloc_tag.c | 22 ++++++---------------- mm/compaction.c | 4 ++-- mm/internal.h | 1 - mm/page_alloc.c | 42 ++++++++++++++++++++++++------------------ mm/page_alloc.h | 17 +++++++++++++++-- mm/page_frag_cache.c | 4 ++-- 6 files changed, 49 insertions(+), 41 deletions(-)mm/alloc_tag.c +0
-0@@ -15,6 +15,8 @@#include <linux/vmalloc.h> #include <linux/kmemleak.h> Quoted 7 lines
Quoted 4 lines
mm/alloc_tag.c line 18
Quoted 1 line
mm/alloc_tag.c +1
-0+#include "internal.h" Should we include
page_alloc.hhere, as we call__alloc_pageslater in this file?Yeah, there are a few build failures due to me not doing a broad enough build. From now on I will just wait for allmodconfig instead of trying to be clever with my build tests, sorry about this.
No worries at all. For the
alloc_tagbuild error, it depends on whether
CONFIG_MEM_ALLOC_PROFILING_DEBUGis enabled.Quoted 4 lines
Also, this suggests that I have not actually re-tested the
alloc_tagcode since v3 so I must repeat the test described in my cover letter (I just manually enable the feature and check the kernel boots) for v4.Thanks a lot for your work. I merged your patch locally via my automation bot,
and verified the basic functionality of
alloc_tag.The
alloc_tagfeature works well so far.
Quoted 7 lines
By the way, you may need to rebase your patchset on top of mm-new.
This patch has been merged into mm-new and will cause a minor conflict,
assuming the mm-new tree has not made further modifications to this patch:
Do not worry about it. Based on the feedback, it might be dropped/rewritten soon. :)
Signature · 2 lines
Best Regards, Yan, Zi
Brendan Jackman
reviewerRe: [PATCH v3 15/16] mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
In reply to Hao GeQuoted 24 lines · Hao Ge
Quoted 9 lines
+/*
+ * Skip early PFN recording for a page allocation. Used by
+ *__alloc_tag_add_early_pfn()to avoid recursion when allocating pages for the
+ * early PFN tracking list itself.
+ *
+ * Codetags of the pages allocated with__GFP_NO_CODETAGshould be
+ * cleared (viaclear_page_tag_ref()) before freeing the pages to prevent
+ *alloc_tag_sub_check()from triggering a warning.
+ */I originally wrote this lengthy comment because the logic lives inside
alloc_tag.c.I wanted to document all the context to avoid confusion when revisiting this code later on.
We've since replaced
__GFP_NO_CODETAGwithALLOC_NO_CODETAG, a genericalloc_flagsbit defined inpage_alloc.h.The original long comment is no longer accurate:
Given that, I suggest updating it to the following:
Quoted 8 lines
/*
* Avoid
alloc_tagrecursion for internal allocations.* Callers must
clear_page_tag_ref()before
* freeing to avoid warnings fromalloc_tag_sub_check().*/
Thanks for the context, pasting this in for v4!
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v3 15/16] mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
In reply to Brendan JackmanQuoted 10 lines · Brendan Jackman
Now that
alloc_pageshas an entrypoint that allows passingalloc_flags, we can take advantage of this to start removing GFP flags that are only used for mm-internal stuff.This requires also plumbing the
alloc_flagsinto some more of the allocator code, in particular __alloc_pages[_noprof]() gets analloc_flagsarg to go along with its callees, and we now need to pass those flags deeper into the allocator so they can reach thealloc_tagcode.
Quoted 3 lines
No functional change intended.
Signed-off-by Brendan Jackman <[email protected]>
broadly LGTM, will wait for the updated version due to Hao Ge's feedback.
PATCH v3 16/16 mm: remove the __GFP_NO_OBJ_EXT flag
In reply to Brendan JackmanAll users of the flag are converted to SLAB_ALLOC_NO_RECURSE or ALLOC_NO_CODETAG (from __GFP_NO_CODETAG which reused the NO_OBJ_EXT bit). Free up the flag bit.
--- tools/include/linux/gfp_types.h | 7 ------- 1 file changed, 7 deletions(-)
tools/include/linux/gfp_types.h +0 -7
@@ -55,7 +55,6 @@ enum {#ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, #endif - ___GFP_NO_OBJ_EXT_BIT, ___GFP_LAST_BIT };
@@ -96,7 +95,6 @@ enum {#else #define ___GFP_NOLOCKDEP 0#endif -#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT)/* * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -137,17 +135,12 @@ enum {* node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. -* -* %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. -* mark_obj_codetag_empty() should be called upon freeing for objects allocated -* with this flag to indicate that their NULL tags are expected and normal. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE)#define __GFP_WRITE ((__force gfp_t)___GFP_WRITE)#define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL)#define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE)#define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) -#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT)/** * DOC: Watermark modifiers
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v3 16/16] mm: remove the __GFP_NO_OBJ_EXT flag
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Incomplete removal of `__GFP_NO_OBJ_EXT` flag leaves the bit reserved in the core kernel. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Incomplete removal of __GFP_NO_OBJ_EXT flag leaves the bit reserved in the core kernel. --
commit d2006cc547e302003b4c27943f81bc40cdd90223
Author: Brendan Jackman <[email protected]>
mm: remove the __GFP_NO_OBJ_EXT flag
This commit removes the __GFP_NO_OBJ_EXT flag from the tools copy of gfp_types.h. Its previous users have been converted to use SLAB_ALLOC_NO_RECURSE or ALLOC_NO_CODETAG instead.
tools/include/linux/gfp_types.h enum { · line 58
Quoted 9 lines
tools/include/linux/gfp_types.h +0
-1@@ -55,7 +55,6 @@ enum {#ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT,#endif - ___GFP_NO_OBJ_EXT_BIT,
[Severity: Medium]
Is it possible that the core kernel header updates were missed? The changes in this patch only modify the tools copy of the header, which causes it to diverge from the actual kernel definitions.
Looking at include/linux/gfp_types.h, ___GFP_NO_OBJ_EXT_BIT remains reserved in the core kernel. Additionally, the trace event mapping TRACE_GFP_EM(NO_OBJ_EXT) is left behind as dead code in include/trace/events/mmflags.h.
Could the removal be applied to those files as well so the bit is fully freed in the core kernel page allocator as described in the commit message?
Quoted 2 lines
tools/include/linux/gfp_types.h +0
-0___GFP_LAST_BIT };
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
Brendan Jackman
reviewerRe: [PATCH v3 16/16] mm: remove the __GFP_NO_OBJ_EXT flag
Quoted 28 lines · sashiko-bot
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Incomplete removal of__GFP_NO_OBJ_EXTflag leaves the bit reserved in the core kernel. --commit
d2006cc547e302003b4c27943f81bc40cdd90223
Author: Brendan Jackman <[email protected]>mm: remove the
__GFP_NO_OBJ_EXTflagThis commit removes the
__GFP_NO_OBJ_EXTflag from the tools copy ofgfp_types.h. Its previous users have been converted to useSLAB_ALLOC_NO_RECURSEorALLOC_NO_CODETAGinstead.tools/include/linux/gfp_types.h enum { · line 58
Quoted 9 lines
tools/include/linux/gfp_types.h +0
-1@@ -55,7 +55,6 @@ enum {#ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT,#endif - ___GFP_NO_OBJ_EXT_BIT,[Severity: Medium]
Is it possible that the core kernel header updates were missed? The changes in this patch only modify the tools copy of the header, which causes it to diverge from the actual kernel definitions.
Quoted 2 lines
Looking at
include/linux/gfp_types.h,___GFP_NO_OBJ_EXT_BITremains reserved in the core kernel.
LOL, Sashiko is correct, I did not actually delete __GFP_NO_OBJ_EXT in this patch.
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v3 16/16] mm: remove the __GFP_NO_OBJ_EXT flag
In reply to Brendan JackmanQuoted 5 lines · Brendan Jackman
All users of the flag are converted to
SLAB_ALLOC_NO_RECURSEorALLOC_NO_CODETAG(from__GFP_NO_CODETAGwhich reused theNO_OBJ_EXTbit). Free up the flag bit.Signed-off-by Vlastimil Babka (SUSE) <[email protected]>
For a correct process, there should be also a matching From:
Thanks!
Vlastimil
Quoted 45 lines
[Rebased onto __GFP_NO_CODETAG removal]Signed-off-by Brendan Jackman <[email protected]>--- tools/include/linux/gfp_types.h | 7 ------- 1 file changed, 7 deletions(-)tools/include/linux/gfp_types.h +0
-7@@ -55,7 +55,6 @@ enum {#ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, #endif - ___GFP_NO_OBJ_EXT_BIT, ___GFP_LAST_BIT };@@ -96,7 +95,6 @@ enum {#else #define ___GFP_NOLOCKDEP 0#endif -#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT)/* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -137,17 +135,12 @@ enum {* node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. -* -* %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. -* mark_obj_codetag_empty() should be called upon freeing for objects allocated -* with this flag to indicate that their NULL tags are expected and normal. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE)#define __GFP_WRITE ((__force gfp_t)___GFP_WRITE)#define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL)#define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE)#define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) -#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT)/** * DOC: Watermark modifiers
Brendan Jackman
reviewerRe: [PATCH v3 16/16] mm: remove the __GFP_NO_OBJ_EXT flag
Quoted 8 lines · Vlastimil Babka (SUSE)
Quoted 5 lines · Brendan Jackman
All users of the flag are converted to
SLAB_ALLOC_NO_RECURSEorALLOC_NO_CODETAG(from__GFP_NO_CODETAGwhich reused theNO_OBJ_EXTbit). Free up the flag bit.Signed-off-by Vlastimil Babka (SUSE) <[email protected]>For a correct process, there should be also a matching From:
Hm, that's weird. Maybe whatever weird idiocy I did that caused me to not apply the whole patch, also caused me not to set the Author correctly in Git? Anyway since then I've reapplied it and the From: is showing up when I format it with b4 send -o /tmp/presend so hopefully v4 will be correct...
Mike Rapoport
maintainerHi Brendan,
Quoted 9 lines · Brendan Jackman
Some tweaks and cleanups for page allocator entrypoint and flags. This is motivated by preparation for
__GFP_UNMAPPED[1] (which will probably becomeALLOC_UNMAPPEDin its next iteration), but all this is supposed to be an improvement to the codebase in its own right: unifying code paths, reducing API surface, and removing GFP flags.Tested:
Quoted 8 lines
- KVM, mm, and BPF selftests in a QEMU VM
-
kunit.pyonx86_64- For the
ALLOC_NO_CODETAGbits I just booted a VM and read/proc/allocinfo. I confirmed that if I removeALLOC_NO_CODETAG, the kernel crashes in early boot, so I was at least booting code that depends on this logic.
Heads up before the full kbuild report:
CI has tested the following submission:
Status: FAILURE Name: [v3,00/16] mm: Some cleanups for page allocator APIs Patchwork: https://patchwork.kernel.org/project/linux-mm/list/?series=1118244&state=* Matrix: https://github.com/linux-mm/linux-mm/actions/runs/28375636866
Quoted 6 lines
I used Google's internal version of Antigravity (AI coding harness) to do the repetitive bits, those commits are marked with Assisted-by, the rest is manual.
Signed-off-by Brendan Jackman <[email protected]>---
Signature · 2 lines
Sincerely yours, Mike.
Quoted 22 lines · Mike Rapoport
Hi Brendan,
Quoted 9 lines · Brendan Jackman
Some tweaks and cleanups for page allocator entrypoint and flags. This is motivated by preparation for
__GFP_UNMAPPED[1] (which will probably becomeALLOC_UNMAPPEDin its next iteration), but all this is supposed to be an improvement to the codebase in its own right: unifying code paths, reducing API surface, and removing GFP flags.Tested:
Quoted 8 lines
- KVM, mm, and BPF selftests in a QEMU VM
-
kunit.pyonx86_64- For the
ALLOC_NO_CODETAGbits I just booted a VM and read/proc/allocinfo. I confirmed that if I removeALLOC_NO_CODETAG, the kernel crashes in early boot, so I was at least booting code that depends on this logic.
Quoted 7 lines
Heads up before the full kbuild report:
CI has tested the following submission:
Status: FAILURE Name: [v3,00/16] mm: Some cleanups for page allocator APIs Patchwork: https://patchwork.kernel.org/project/linux-mm/list/?series=1118244&state=* Matrix: https://github.com/linux-mm/linux-mm/actions/runs/28375636866
Agh, thanks, I broke the build for CMA.
I thought I had this covered in my local test scripts. I will fix that first then I'll send a fixup for the patch.
Quoted 32 lines · Brendan Jackman
Quoted 22 lines · Mike Rapoport
Hi Brendan,
Quoted 9 lines · Brendan Jackman
Some tweaks and cleanups for page allocator entrypoint and flags. This is motivated by preparation for
__GFP_UNMAPPED[1] (which will probably becomeALLOC_UNMAPPEDin its next iteration), but all this is supposed to be an improvement to the codebase in its own right: unifying code paths, reducing API surface, and removing GFP flags.Tested:
Quoted 8 lines
- KVM, mm, and BPF selftests in a QEMU VM
-
kunit.pyonx86_64- For the
ALLOC_NO_CODETAGbits I just booted a VM and read/proc/allocinfo. I confirmed that if I removeALLOC_NO_CODETAG, the kernel crashes in early boot, so I was at least booting code that depends on this logic.Quoted 7 lines
Heads up before the full kbuild report:
CI has tested the following submission:
Status: FAILURE Name: [v3,00/16] mm: Some cleanups for page allocator APIs Patchwork: https://patchwork.kernel.org/project/linux-mm/list/?series=1118244&state=* Matrix: https://github.com/linux-mm/linux-mm/actions/runs/28375636866
Quoted 4 lines
Agh, thanks, I broke the build for CMA.
I thought I had this covered in my local test scripts. I will fix that first then I'll send a fixup for the patch.
OK my scripts are indeed checking CMA, the issue is that I didn't build with NUMA_BALANCING.
I guess Suren was right[0] and I really should build allmodconfig (at least vmlinux) before sending patches. I was a bit skeptical that this was an especially useful config to build, but now I realise it just maximises coverage, even if it does so under a pretty arbitrary configuration.
[0] https://lore.kernel.org/all/CAJuCfpHAMaK2sZYSgS750CvgksCKEfOmLzZasXdBup+CrS-3Pg@mail.gmail.com/
Here are the fixes:
mm/migrate.c +1 -0
@@ -49,6 +49,7 @@#include <trace/events/migrate.h> #include "internal.h" +#include "page_alloc.h" #include "swap.h" static const struct movable_operations *offline_movable_ops;
mm/page_reporting.c +1 -0
@@ -8,6 +8,7 @@#include <linux/delay.h> #include <linux/scatterlist.h> +#include "page_alloc.h" #include "page_reporting.h" #include "internal.h"
mm/shuffle.c +1 -0
@@ -7,6 +7,7 @@#include <linux/random.h> #include <linux/moduleparam.h> #include "internal.h" +#include "page_alloc.h" #include "shuffle.h" DEFINE_STATIC_KEY_FALSE (page_alloc_shuffle_key);