mm: Some cleanups for page allocator APIs
Subsystems (29)
v5 cover mm: Some cleanups for page allocator APIs
Based on mm-new.
This depends on moving alloc_tag to mm/:
https://lore.kernel.org/all/aj5QBtJcphPElczI@lucifer/
Some tweaks and cleanups for page allocator entrypoint and flags. This is motivated by preparation for __GFP_UNMAPPED [1] (which will probably become ALLOC_UNMAPPED in its next iteration), but all this is supposed to be an improvement to the codebase in its own right: unifying code paths, reducing API surface, and removing GFP flags.
[1] https://lore.kernel.org/all/[email protected]/
This started with unifying __alloc_frozen_pages[_nolock]_noprof() and expanded from there.
Unifying the nolock allocator entrypoint with the normal allocator entrypoint means adding an alloc_flags argument to the later (only exposed within mm/). This presents an opportunity to take advantage of that arg to remove some GFP flags, if we add that alloc_flags arg a bit more broadly to allocator entrypoints.
To distinguish between mm-internal and "public" allocator entrypoints, it makes sense to use the __ prefix. There are already some public APIs with that prefix. For *alloc_pages*, just removing those variants seems like a nice cleanup anyway, so do that. For get_free_pages, the "__" variant is the _only_ variant and it's very widely used, so it doesn't seem worthwhile to modify that. Therefore, scope this "__" change specifically to the *alloc_pages* API, which means we leave the *folio_alloc* API untouched too, even though that could probably be cleaned up if so desired.
Tested:
- KVM, mm, and BPF selftests in a QEMU VM
- kunit.py on x86_64
- For the ALLOC_NO_CODETAG bits I just booted a VM and read /proc/allocinfo. I confirmed that if I remove ALLOC_NO_CODETAG, the kernel crashes in early boot, so I was at least booting code that depends on this logic.
I used Google's internal version of Antigravity (AI coding harness) to do the repetitive bits, those commits are marked with Assisted-by, the rest is manual.
---
Changes in v5:
- Just trivial non-functional fixes.
- Link to v4: https://patch.msgid.link/[email protected]
Changes in v4:
- Fixed some (harmless) missing applications of ac->alloc_flags (local Sashiko)
- Fixed various build issues.
- Note that Sashiko pointed out a KMSAN build issue [0], I have
fixed it but KMSAN builds are currently broken by objtool [1]. At least
mm/kmsan/init.c compiles.
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/t/#u- Avoided setting ALLOC_NOFRAGMENT under ALLOC_NOLOCK (Sashiko, Harry)
- Added patch to tweak alloc_flags_cma() interface (Vlastimil)
- More commit messages fixups (various)
- Added patch to create can_spin_trylock() (Harry)
- Link to v3: https://patch.msgid.link/[email protected]
Changes in v3:
- Created mm/page_alloc.h
- Fixed EXPORT_SYMBOL() issues
- Reworded commit messages per Sashiko's pointers
- Dropped rename of alloc_flags arg in prepare_alloc_pages() (Suren)
- Renamed gfp_to_alloc_flags_nonblocking() too after rebasing onto:
https://lore.kernel.org/all/[email protected]/
- Link to v2: https://patch.msgid.link/[email protected]
Changes in v2:
- Fixed up whitespace in nolock unification patch
- Introduced ALLOC_DEFAULT to replace literal 0 for alloc_flags
- All other patches are new
- Link to v1: https://patch.msgid.link/[email protected]
---
Brendan Jackman (17):
mm/page_alloc: rename ALLOC_TRYLOCK -> ALLOC_NOLOCK mm/page_alloc: some renames to clarify alloc_flags scopes
mm: name some args in a function declaration
mm: Split out internal page_alloc.h mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof() mm/page_alloc: relax GFP WARN in nolock allocs
mm: move some stuff to mm/page_alloc.h perf/x86/intel: Use higher-level allocator API
KVM: VMX: Use higher-level allocator API x86/virt: Use higher-level allocator API
sgi-xp: Use higher-level allocator API net/funeth: Switch to higher-level allocator API
mm: Remove __alloc_pages_node()
mm: Move __alloc_pages() to mm/page_alloc.h
mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG mm/page_alloc: drop alloc_flags arg from alloc_flags_cma()
mm: factor out can_spin_trylock()
Vlastimil Babka (SUSE) (1):
mm: remove the __GFP_NO_OBJ_EXT flag
Documentation/admin-guide/cgroup-v1/cpusets.rst | 2 +- Documentation/admin-guide/mm/transhuge.rst | 2 +- MAINTAINERS | 1 + arch/x86/events/intel/ds.c | 6 +- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/virt/hw.c | 2 +- drivers/misc/sgi-xp/xpc_uv.c | 5 +- drivers/net/ethernet/fungible/funeth/funeth_rx.c | 2 +- include/linux/alloc_tag.h | 4 +- include/linux/gfp.h | 54 +--- include/linux/gfp_types.h | 7 - include/linux/skbuff.h | 2 +- include/trace/events/mmflags.h | 10 +- mm/alloc_tag.c | 23 +- mm/compaction.c | 5 +- mm/hugetlb.c | 4 +- mm/internal.h | 275 ++------------------ mm/khugepaged.c | 1 + mm/kmsan/init.c | 2 +- mm/memory-failure.c | 1 + mm/memory_hotplug.c | 1 + mm/mempolicy.c | 11 +- mm/migrate.c | 1 + mm/mm_init.c | 1 + mm/page_alloc.c | 269 ++++++++++--------- mm/page_alloc.h | 312 +++++++++++++++++++++++ mm/page_frag_cache.c | 6 +- mm/page_isolation.c | 1 + mm/page_owner.c | 2 +- mm/page_reporting.c | 1 + mm/show_mem.c | 1 + mm/shuffle.c | 1 + mm/slub.c | 17 +- mm/swap.c | 1 + mm/vmscan.c | 1 + mm/vmstat.c | 1 + tools/include/linux/gfp_types.h | 7 - 37 files changed, 536 insertions(+), 508 deletions(-)
---
base-commit: 32af3ff0925368eff29b2fed62f154150eb5dc10
change-id: 20260617-alloc-trylock-14ad37dab337
Best regards,
Signature · 1 line
Brendan Jackman <[email protected]>
PATCH v5 1/18 mm/page_alloc: rename ALLOC_TRYLOCK -> ALLOC_NOLOCK
It's confusing that the function is called "nolock" but the flag is called "trylock", align them.
The function's terminology is more visible and has more mindshare so use that.
--- mm/internal.h | 2 +- mm/page_alloc.c | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-)
mm/internal.h +1 -1
@@ -1480,7 +1480,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_NOFRAGMENT 0x0#endif #define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ -#define ALLOC_TRYLOCK 0x400/* Only use spin_trylock in allocation path */ +#define ALLOC_NOLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */
mm/page_alloc.c +5 -5
@@ -2530,7 +2530,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,unsigned long flags;int i; -if (unlikely (alloc_flags &ALLOC_TRYLOCK )) { +if (unlikely (alloc_flags &ALLOC_NOLOCK )) {if (!spin_trylock_irqsave (&zone->lock, flags))return 0 ; }else {
@@ -3218,7 +3218,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - if (unlikely(alloc_flags & ALLOC_TRYLOCK)) { + if (unlikely(alloc_flags & ALLOC_NOLOCK)) { if (!spin_trylock_irqsave(&zone->lock, flags)) return NULL; } else {
@@ -5059,7 +5059,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,* Don't invoke should_fail logic, since it may call * get_random_u32() and printk() which need to spin_lock. */ -if (!(*alloc_flags &ALLOC_TRYLOCK ) && +if (!(*alloc_flags &ALLOC_NOLOCK ) &&should_fail_alloc_page (gfp_mask, order))return false;
@@ -7804,7 +7804,7 @@ static bool cond_accept_memory(struct zone *zone, unsigned int order,return false;/* Bailout, since try_to_accept_memory_one() needs to take a lock */ -if (alloc_flags &ALLOC_TRYLOCK ) +if (alloc_flags &ALLOC_NOLOCK )return false; wmark =promo_wmark_pages (zone);
@@ -7896,7 +7896,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned*/ gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | __GFP_COMP | gfp_flags; -unsigned int alloc_flags =ALLOC_TRYLOCK ; +unsigned int alloc_flags =ALLOC_NOLOCK ;struct alloc_context ac = { };struct page *page;
Signature · 1 line
2.54.0
Re: [PATCH v5 01/18] mm/page_alloc: rename ALLOC_TRYLOCK -> ALLOC_NOLOCK
Quoted 21 lines · Brendan Jackman
It's confusing that the function is called "nolock" but the flag is called "trylock", align them.
The function's terminology is more visible and has more mindshare so use that.
Suggested-by "Vlastimil Babka (SUSE)" <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Harry Yoo (Oracle) <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- mm/internal.h | 2 +- mm/page_alloc.c | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-)mm/internal.h +0
-0
mm/internal.h unsigned int reclaim_clean_pages_from_list(struct zone *zone, · line 1483
Quoted 5 lines
mm/internal.h +1
-1@@ -1480,7 +1480,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_NOFRAGMENT 0x0#endif #define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ -#define ALLOC_TRYLOCK 0x400/* Only use spin_trylock in allocation path */ +#define ALLOC_NOLOCK 0x400/* Only use spin_trylock in allocation path */
It is strange to me that _NOLOCK uses spin_trylock. Lock or no lock? :) But it matches the _nolock function. Anyway,
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 2/18 mm/page_alloc: some renames to clarify alloc_flags scopes
In reply to Brendan JackmanIt's pretty confusing that:
- The slowpath and fastpath have a totally distinct set of alloc_flags.
- gfp_to_alloc_flags() sounds generic but it only influences the slowpath.
Rename some variables to highlight which alloc_flags are fastpath-specific. Rename gfp_to_alloc_flags() to highlight that it's slowpath-specific.
gfp_to_alloc_flags_cma() and gfp_to_alloc_flags_nonblocking() currently have perfectly harmless names, but to keep the naming consistent also rename those to the alloc_flags_*() pattern (which already exists for alloc_flags_nofragment()).
--- include/linux/skbuff.h | 2 +- mm/page_alloc.c | 28 ++++++++++++++-------------- 2 files changed, 15 insertions(+), 15 deletions(-)
include/linux/skbuff.h +1 -1
@@ -3573,7 +3573,7 @@ static inline struct page *__dev_alloc_pages_noprof(gfp_t gfp_mask,* 3. If requesting a order 0 page it will not be compound * due to the check to see if order has a value in prep_new_page * 4. __GFP_MEMALLOC is ignored if __GFP_NOMEMALLOC is set due to -* code in gfp_to_alloc_flags that should be enforcing this. +* code in alloc_flags_slowpath() that should be enforcing this. */ gfp_mask |= __GFP_COMP | __GFP_MEMALLOC;
mm/page_alloc.c +14 -14
@@ -3774,8 +3774,8 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) }/* Must be called after current_gfp_context() which can change gfp_mask */ -static inline unsigned int gfp_to_alloc_flags_cma (gfp_t gfp_mask, -unsigned int alloc_flags) +static inline unsigned int alloc_flags_cma (gfp_t gfp_mask, +unsigned int alloc_flags) {#ifdef CONFIG_CMA if (gfp_migratetype (gfp_mask) ==MIGRATE_MOVABLE )
@@ -4474,7 +4474,7 @@ static void wake_all_kswapds(unsigned int order, gfp_t gfp_mask, }static inline unsigned int -gfp_to_alloc_flags_nonblocking (gfp_t gfp_mask,unsigned int order) +alloc_flags_nonblocking (gfp_t gfp_mask,unsigned int order) {unsigned int alloc_flags =0 ;
@@ -4497,7 +4497,7 @@ gfp_to_alloc_flags_nonblocking(gfp_t gfp_mask, unsigned int order) }static inline unsigned int -gfp_to_alloc_flags (gfp_t gfp_mask,unsigned int order) +alloc_flags_slowpath (gfp_t gfp_mask,unsigned int order) {unsigned int alloc_flags =ALLOC_WMARK_MIN |ALLOC_CPUSET ;
@@ -4512,7 +4512,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)if (gfp_mask & __GFP_KSWAPD_RECLAIM) alloc_flags |=ALLOC_KSWAPD ; - alloc_flags |=gfp_to_alloc_flags_nonblocking (gfp_mask, order); + alloc_flags |=alloc_flags_nonblocking (gfp_mask, order);if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) {/*
@@ -4525,7 +4525,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) }else if (unlikely (rt_or_dl_task (current)) &&in_task ()) alloc_flags |=ALLOC_MIN_RESERVE ; - alloc_flags =gfp_to_alloc_flags_cma (gfp_mask, alloc_flags); + alloc_flags =alloc_flags_cma (gfp_mask, alloc_flags);if (defrag_mode) alloc_flags |=ALLOC_NOFRAGMENT ;
@@ -4791,7 +4791,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* kswapd needs to be woken up, and to avoid the cost of setting up * alloc_flags precisely. So we do that now. */ - alloc_flags =gfp_to_alloc_flags (gfp_mask, order); + alloc_flags =alloc_flags_slowpath (gfp_mask, order);/* * We need to recalculate the starting point for the zonelist iterator
@@ -4832,7 +4832,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags =__gfp_pfmemalloc_flags (gfp_mask);if (reserve_flags) - alloc_flags =gfp_to_alloc_flags_cma (gfp_mask, reserve_flags) | + alloc_flags =alloc_flags_cma (gfp_mask, reserve_flags) | (alloc_flags &ALLOC_KSWAPD );/*
@@ -5063,7 +5063,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, should_fail_alloc_page(gfp_mask, order))return false; - *alloc_flags =gfp_to_alloc_flags_cma (gfp_mask, *alloc_flags); + *alloc_flags =alloc_flags_cma (gfp_mask, *alloc_flags);/* Dirty zone balancing only done in the fast path */ ac->spread_dirty_pages = (gfp_mask & __GFP_WRITE);
@@ -5277,7 +5277,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { struct page *page; -unsigned int alloc_flags =ALLOC_WMARK_LOW ; +unsigned int fastpath_alloc_flags =ALLOC_WMARK_LOW ;gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ struct alloc_context ac = { };
@@ -5299,18 +5299,18 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, gfp =current_gfp_context (gfp); alloc_gfp = gfp;if (!prepare_alloc_pages (gfp, order, preferred_nid, nodemask, &ac, - &alloc_gfp, &alloc_flags)) + &alloc_gfp, &fastpath_alloc_flags))return NULL ;/* * Forbid the first pass from falling back to types that fragment * memory until all local zones are considered. */ - alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); - alloc_flags |=gfp_to_alloc_flags_nonblocking (gfp, order) &ALLOC_HIGHATOMIC ; + fastpath_alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); + fastpath_alloc_flags |=alloc_flags_nonblocking (gfp, order) &ALLOC_HIGHATOMIC ;/* First allocation attempt */ - page =get_page_from_freelist (alloc_gfp, order, alloc_flags, &ac); + page =get_page_from_freelist (alloc_gfp, order, fastpath_alloc_flags, &ac);if (likely (page)) goto out;
Signature · 1 line
2.54.0
Re: [PATCH v5 02/18] mm/page_alloc: some renames to clarify alloc_flags scopes
Quoted 16 lines · Brendan Jackman
It's pretty confusing that:
- The slowpath and fastpath have a totally distinct set of
alloc_flags.-
gfp_to_alloc_flags()sounds generic but it only influences the slowpath.Rename some variables to highlight which
alloc_flagsare fastpath-specific. Renamegfp_to_alloc_flags()to highlight that it's slowpath-specific.
gfp_to_alloc_flags_cma()andgfp_to_alloc_flags_nonblocking()currently have perfectly harmless names, but to keep the naming consistent also rename those to the alloc_flags_*() pattern (which already exists foralloc_flags_nofragment()).
Quoted 8 lines
Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Acked-by JP Kobryn <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- include/linux/skbuff.h | 2 +- mm/page_alloc.c | 28 ++++++++++++++-------------- 2 files changed, 15 insertions(+), 15 deletions(-)
LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 3/18 mm: name some args in a function declaration
In reply to Brendan JackmanCheckpatch complains about this, a later patch will move the code, fix it so that checkpatch doesn't complain about that patch. Do it in a separate patch so the "move the code" patch is trivial to review using Git's diff colouring.
--- mm/internal.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/internal.h +2 -2
@@ -919,8 +919,8 @@ extern bool free_pages_prepare(struct page *page, unsigned int order);extern int user_min_free_kbytes; -struct page *__alloc_frozen_pages_noprof (gfp_t ,unsigned int order,int nid, -nodemask_t *); +struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid, +nodemask_t *nodemask);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order);
Signature · 1 line
2.54.0
Re: [PATCH v5 03/18] mm: name some args in a function declaration
Quoted 11 lines · Brendan Jackman
Checkpatch complains about this, a later patch will move the code, fix it so that checkpatch doesn't complain about that patch. Do it in a separate patch so the "move the code" patch is trivial to review using Git's diff colouring.
Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- mm/internal.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 4/18 mm: Split out internal page_alloc.h
In reply to Brendan Jackmaninternal.h is a bit bloated, seems like time for a page_alloc.h.
Where it wasn't obvious, the heuristic for deciding what goes into this new header was "does it support/correspond to a definition in mm/page_alloc.c?"
Only need to include it from ~20 .c files out of ~150 so this does seem like a genuine reduction in scopes, which is nice. And there's no circular internal.h<->page_alloc.h dependency, so it seems worthwhile to split this up before that inevitably emerges!
--- MAINTAINERS | 1 + mm/compaction.c | 1 + mm/hugetlb.c | 1 + mm/internal.h | 252 ----------------------------------------------- mm/khugepaged.c | 1 + mm/kmsan/init.c | 2 +- mm/memory-failure.c | 1 + mm/memory_hotplug.c | 1 + mm/mempolicy.c | 1 + mm/migrate.c | 1 + mm/mm_init.c | 1 + mm/page_alloc.c | 1 + mm/page_alloc.h | 269 +++++++++++++++++++++++++++++++++++++++++++++++++++ mm/page_frag_cache.c | 2 +- mm/page_isolation.c | 1 + mm/page_owner.c | 2 +- mm/page_reporting.c | 1 + mm/show_mem.c | 1 + mm/shuffle.c | 1 + mm/slub.c | 1 + mm/swap.c | 1 + mm/vmscan.c | 1 + 22 files changed, 289 insertions(+), 255 deletions(-)
MAINTAINERS +1 -0
@@ -17171,6 +17171,7 @@ F: mm/debug_page_alloc.c F: mm/debug_page_ref.c F: mm/fail_page_alloc.c F: mm/page_alloc.c +F: mm/page_alloc.h F: mm/page_ext.c F: mm/page_frag_cache.c F: mm/page_isolation.c
mm/compaction.c +1 -0
@@ -24,6 +24,7 @@#include <linux/page_owner.h> #include <linux/psi.h> #include <linux/cpuset.h> +#include "page_alloc.h" #include "internal.h" #ifdef CONFIG_COMPACTION
mm/hugetlb.c +1 -0
@@ -47,6 +47,7 @@#include <linux/node.h> #include <linux/page_owner.h> #include "internal.h" +#include "page_alloc.h" #include "hugetlb_vmemmap.h" #include "hugetlb_cma.h" #include "hugetlb_internal.h"
mm/internal.h +0 -252
282 lines not on this page · raw message
mm/khugepaged.c +1 -0
@@ -26,6 +26,7 @@#include <asm/tlb.h> #include "internal.h" +#include "page_alloc.h" #include "mm_slot.h" enum scan_result {
mm/kmsan/init.c +1 -1
@@ -13,7 +13,7 @@#include <linux/mm.h> #include <linux/memblock.h> -#include "../internal.h" +#include "../page_alloc.h" #define NUM_FUTURE_RANGES 128struct start_end_pair {
mm/memory-failure.c +1 -0
@@ -66,6 +66,7 @@#include <trace/events/memory-failure.h> #include "swap.h" +#include "page_alloc.h" #include "internal.h" static int sysctl_memory_failure_early_kill__read_mostly ;
mm/memory_hotplug.c +1 -0
@@ -40,6 +40,7 @@#include <asm/tlbflush.h> #include "internal.h" +#include "page_alloc.h" #include "shuffle.h" enum {
mm/mempolicy.c +1 -0
@@ -119,6 +119,7 @@#include <linux/memory.h> #include "internal.h" +#include "page_alloc.h" /* Internal flags */ #define MPOL_MF_DISCONTIG_OK (MPOL_MF_INTERNAL << 0)/* Skip checks for continuous vmas */
mm/migrate.c +1 -0
@@ -49,6 +49,7 @@#include <trace/events/migrate.h> #include "internal.h" +#include "page_alloc.h" #include "swap.h" static const struct movable_operations *offline_movable_ops;
mm/mm_init.c +1 -0
@@ -33,6 +33,7 @@#include <linux/kexec_handover.h> #include <linux/hugetlb.h> #include "internal.h" +#include "page_alloc.h" #include "slab.h" #include "shuffle.h"
mm/page_alloc.c +1 -0
@@ -56,6 +56,7 @@#include <linux/pgalloc_tag.h> #include <asm/div64.h> #include "internal.h" +#include "page_alloc.h" #include "shuffle.h" #include "page_reporting.h"
mm/page_alloc.h new file +269 -0
269 lines not on this page · raw message
mm/page_frag_cache.c +1 -1
@@ -18,7 +18,7 @@#include <linux/init.h> #include <linux/mm.h> #include <linux/page_frag_cache.h> -#include "internal.h" +#include "page_alloc.h" static unsigned long encoded_page_create (struct page *page,unsigned int order,bool pfmemalloc)
mm/page_isolation.c +1 -0
@@ -11,6 +11,7 @@#include <linux/page_owner.h> #include <linux/migrate.h> #include "internal.h" +#include "page_alloc.h" #define CREATE_TRACE_POINTS #include <trace/events/page_isolation.h>
mm/page_owner.c +1 -1
@@ -13,7 +13,7 @@#include <linux/memcontrol.h> #include <linux/sched/clock.h> -#include "internal.h" +#include "page_alloc.h" /* * TODO: teach PAGE_OWNER_STACK_DEPTH (__dump_page_owner and save_stack)
mm/page_reporting.c +1 -0
@@ -8,6 +8,7 @@#include <linux/delay.h> #include <linux/scatterlist.h> +#include "page_alloc.h" #include "page_reporting.h" #include "internal.h"
mm/show_mem.c +1 -0
@@ -16,6 +16,7 @@#include <linux/vmstat.h> #include "internal.h" +#include "page_alloc.h" #include "swap.h" atomic_long_t _totalram_pages__read_mostly ;
mm/shuffle.c +1 -0
@@ -7,6 +7,7 @@#include <linux/random.h> #include <linux/moduleparam.h> #include "internal.h" +#include "page_alloc.h" #include "shuffle.h" DEFINE_STATIC_KEY_FALSE (page_alloc_shuffle_key);
mm/slub.c +1 -0
@@ -53,6 +53,7 @@#include <trace/events/kmem.h> #include "internal.h" +#include "page_alloc.h" /* * Lock order:
mm/swap.c +1 -0
@@ -39,6 +39,7 @@#include <linux/buffer_head.h> #include "internal.h" +#include "page_alloc.h" #define CREATE_TRACE_POINTS #include <trace/events/pagemap.h>
mm/vmscan.c +1 -0
@@ -66,6 +66,7 @@#include <linux/sched/sysctl.h> #include "internal.h" +#include "page_alloc.h" #include "swap.h" #define CREATE_TRACE_POINTS
Signature · 1 line
2.54.0
Re: [PATCH v5 04/18] mm: Split out internal page_alloc.h
Quoted 31 lines · Brendan Jackman
internal.his a bit bloated, seems like time for apage_alloc.h.Where it wasn't obvious, the heuristic for deciding what goes into this new header was "does it support/correspond to a definition in
mm/page_alloc.c?"Only need to include it from ~20 .c files out of ~150 so this does seem like a genuine reduction in scopes, which is nice. And there's no circular internal.h<->page_alloc.h dependency, so it seems worthwhile to split this up before that inevitably emerges!
Suggested-by "David Hildenbrand (Arm)" <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- MAINTAINERS | 1 + mm/compaction.c | 1 + mm/hugetlb.c | 1 + mm/internal.h | 252 ----------------------------------------------- mm/khugepaged.c | 1 + mm/kmsan/init.c | 2 +- mm/memory-failure.c | 1 + mm/memory_hotplug.c | 1 + mm/mempolicy.c | 1 + mm/migrate.c | 1 + mm/mm_init.c | 1 + mm/page_alloc.c | 1 + mm/page_alloc.h | 269 +++++++++++++++++++++++++++++++++++++++++++++++++++ mm/page_frag_cache.c | 2 +- mm/page_isolation.c | 1 +
Quoted 9 lines
mm/page_owner.c | 2 +- mm/page_reporting.c | 1 + mm/show_mem.c | 1 + mm/shuffle.c | 1 + mm/slub.c | 1 + mm/swap.c | 1 + mm/vmscan.c | 1 + 22 files changed, 289 insertions(+), 255 deletions(-)
Thank you for the cleanup.
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 5/18 mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
In reply to Brendan JackmanCurrently the core allocator code is controlled by ALLOC_NOLOCK, but the main entry point function is significantly different from the normal __alloc_frozen_pages_nolock(), this is tiring when reading the code.
Plumb the ALLOC_NOLOCK control one layer up in the call stack: create an alloc_flags argument to __alloc_frozen_pages_nolock() (which is only exposed to mm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handling NUMA_NO_NODE, similar to how some of the wrappers in gfp.h do).
For consistency, set ALLOC_WMARK_MIN explicitly in fastpath_alloc_flags for the new ALLOC_NOLOCK path. This was already "done" silently in __alloc_frozen_pages_nolock_noprof(): ALLOC_WMARK_MIN is 0.
Rationale that this doesn't change anything:
1. Simple bits: A bunch of the nolock-specific handling is just moved to the new alloc_order_allowed(), alloc_nolock_allowed() and gfp_nolock.
2. __alloc_frozen_pages_noprof() has some extra logic that wasn't previously in the nolock variant:
a. Application of gfp_allowed_mask; this only affects early boot,
only flags that affect the slowpath get changed here, and the nolock allocation path isn't allowed to the GFP_BOOT_MASK flags.
b. Application of current_gfp_context() - also only affects the
slowpath
3. The slowpath itself: this is now just explicitly skipped under !ALLOC_TRYLOCK.
Ulterior motive: adding an alloc_flags arg to the allocator's mm-internal entrypoint can later be used to do more allocation customisation without needing to create new GFP flags.
No functional change intended.
--- mm/hugetlb.c | 3 +- mm/mempolicy.c | 10 +-- mm/page_alloc.c | 192 +++++++++++++++++++++++++++++--------------------------- mm/page_alloc.h | 6 +- mm/slub.c | 6 +- 5 files changed, 117 insertions(+), 100 deletions(-)
mm/hugetlb.c +2 -1
@@ -1790,7 +1790,8 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,if (alloc_try_hard) gfp_mask |= __GFP_RETRY_MAYFAIL; - folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask); + folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask, +ALLOC_DEFAULT );/* * If we did not specify __GFP_RETRY_MAYFAIL, but still got a
mm/mempolicy.c +6 -4
@@ -2417,9 +2417,11 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,*/ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page =__alloc_frozen_pages_noprof (preferred_gfp, order, nid, nodemask); + page =__alloc_frozen_pages_noprof (preferred_gfp, order, nid, nodemask, +ALLOC_DEFAULT );if (!page) - page =__alloc_frozen_pages_noprof (gfp, order, nid,NULL ); + page =__alloc_frozen_pages_noprof (gfp, order, nid,NULL , +ALLOC_DEFAULT );return page; }
@@ -2467,7 +2469,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,*/ page =__alloc_frozen_pages_noprof ( gfp | __GFP_THISNODE | __GFP_NORETRY, order, - nid,NULL ); + nid,NULL ,ALLOC_DEFAULT );if (page || !(gfp & __GFP_DIRECT_RECLAIM))return page;/*
@@ -2479,7 +2481,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } } - page =__alloc_frozen_pages_noprof (gfp, order, nid, nodemask); + page =__alloc_frozen_pages_noprof (gfp, order, nid, nodemask,ALLOC_DEFAULT );if (unlikely (pol->mode ==MPOL_INTERLEAVE || pol->mode ==MPOL_WEIGHTED_INTERLEAVE ) && page) {
mm/page_alloc.c +101 -91
242 lines not on this page · raw message
mm/page_alloc.h +4 -2
@@ -11,6 +11,7 @@#include <linux/nodemask.h> #include <linux/types.h> +#define ALLOC_DEFAULT 0/* The ALLOC_WMARK bits are used as an index to zone->watermark */ #define ALLOC_WMARK_MIN WMARK_MIN#define ALLOC_WMARK_LOW WMARK_LOW
@@ -219,7 +220,7 @@ extern bool free_pages_prepare(struct page *page, unsigned int order);extern int user_min_free_kbytes;struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid, -nodemask_t *nodemask); +nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order);
@@ -230,7 +231,8 @@ struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order);#else static inline struct page *alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order) { -return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL ); +return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL , +ALLOC_DEFAULT ); }#endif
mm/slub.c +4 -2
@@ -3292,7 +3292,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,else if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages (flags, order);else - page =__alloc_frozen_pages (flags, order, node,NULL ); + page =__alloc_frozen_pages (flags, order, node,NULL , +ALLOC_DEFAULT );if (!page)return NULL ;
@@ -5302,7 +5303,8 @@ static void *___kmalloc_large_node(size_t size, gfp_t flags, int node)if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages_noprof (flags, order);else - page =__alloc_frozen_pages_noprof (flags, order, node,NULL ); + page =__alloc_frozen_pages_noprof (flags, order, node,NULL , +ALLOC_DEFAULT );if (page) { ptr =page_address (page);
Signature · 1 line
2.54.0
Re: [PATCH v5 05/18] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
Quoted 27 lines · Brendan Jackman
Currently the core allocator code is controlled by
ALLOC_NOLOCK, but the main entry point function is significantly different from the normal__alloc_frozen_pages_nolock(), this is tiring when reading the code.Plumb the
ALLOC_NOLOCKcontrol one layer up in the call stack: create analloc_flagsargument to__alloc_frozen_pages_nolock()(which is only exposed tomm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handlingNUMA_NO_NODE, similar to how some of the wrappers ingfp.hdo).For consistency, set
ALLOC_WMARK_MINexplicitly infastpath_alloc_flagsfor the newALLOC_NOLOCKpath. This was already "done" silently in__alloc_frozen_pages_nolock_noprof():ALLOC_WMARK_MINis 0.Rationale that this doesn't change anything:
1. Simple bits: A bunch of the nolock-specific handling is just moved to the new
alloc_order_allowed(),alloc_nolock_allowed()andgfp_nolock.2.
__alloc_frozen_pages_noprof()has some extra logic that wasn't previously in the nolock variant:a. Application of
gfp_allowed_mask; this only affects early boot,
only flags that affect the slowpath get changed here, and the nolock allocation path isn't allowed to theGFP_BOOT_MASKflags.
Quoted 5 lines
b. Application of
current_gfp_context()- also only affects the
slowpath3. The slowpath itself: this is now just explicitly skipped under !ALLOC_TRYLOCK.
s/TRYLOCK/NOLOCK
Quoted 17 lines
Ulterior motive: adding an
alloc_flagsarg to the allocator's mm-internal entrypoint can later be used to do more allocation customisation without needing to create new GFP flags.No functional change intended.
Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- mm/hugetlb.c | 3 +- mm/mempolicy.c | 10 +-- mm/page_alloc.c | 192 +++++++++++++++++++++++++++++--------------------------- mm/page_alloc.h | 6 +- mm/slub.c | 6 +- 5 files changed, 117 insertions(+), 100 deletions(-)
<snip>
Quoted 12 lines
mm/page_alloc.c +12
-0+/* +* This is the 'heart' of the zoned buddy allocator. +*/ +struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order, +int preferred_nid,nodemask_t *nodemask,unsigned int alloc_flags) +{ +struct page *page; +gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ +struct alloc_context ac = { }; +unsigned int fastpath_alloc_flags = alloc_flags; + +/* Other flags could be supported later if needed. */
mm/page_alloc.c void free_pages_bulk(struct page **page_array, unsigned long nr_pages) · line 5360
Quoted 10 lines
mm/page_alloc.c +8
-0+if (WARN_ON (alloc_flags & ~ALLOC_NOLOCK ))return NULL ; +if (!alloc_order_allowed (gfp, order, alloc_flags)) +return NULL ; + +if (alloc_flags &ALLOC_NOLOCK ) { +VM_WARN_ON_ONCE (gfp & ~__GFP_ACCOUNT); +if (!alloc_nolock_allowed ()) +return NULL ;
At first look, I wonder why __alloc_frozen_pages_noprof() needs to care about alloc_nolock_allowed(). But the patch's idea is to centralize all allocation policies, so it makes sense.
Ideally, I would want alloc_frozen_pages_nolock_noprof() to filter as much as possible, so that __alloc_frozen_pages_noprof() has minimal/no awareness of ALLOC_NOLOCK. But ALLOC_NOLOCK has different preferences compared to the default __alloc_frozen_pages_noprof() policy like ALLOC_WMARK_MIN vs ALLOC_WMARK_LOW, skip slowpath, and more. Maybe we could do something like:
__alloc_frozen_pages_noprof()
{
alloc_fastpath();
alloc_slowpath();
}alloc_frozen_pages_nolock_noprof()
{
alloc_order_allowed();
alloc_nolock_allow();
alloc_fastpath();
}But it still cannot remove ALLOC_NOLOCK completely from __alloc_frozen_pages_noprof(), like the nofragment skip. Anyway, this patch is a reasonable cleanup. Thanks.
Signature · 2 lines
Best Regards, Yan, Zi
Brendan Jackman
reviewerRe: [PATCH v5 05/18] mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()
Quoted 28 lines · Zi Yan
Quoted 27 lines · Brendan Jackman
Currently the core allocator code is controlled by
ALLOC_NOLOCK, but the main entry point function is significantly different from the normal__alloc_frozen_pages_nolock(), this is tiring when reading the code.Plumb the
ALLOC_NOLOCKcontrol one layer up in the call stack: create analloc_flagsargument to__alloc_frozen_pages_nolock()(which is only exposed tomm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handlingNUMA_NO_NODE, similar to how some of the wrappers ingfp.hdo).For consistency, set
ALLOC_WMARK_MINexplicitly infastpath_alloc_flagsfor the newALLOC_NOLOCKpath. This was already "done" silently in__alloc_frozen_pages_nolock_noprof():ALLOC_WMARK_MINis 0.Rationale that this doesn't change anything:
1. Simple bits: A bunch of the nolock-specific handling is just moved to the new
alloc_order_allowed(),alloc_nolock_allowed()andgfp_nolock.2.
__alloc_frozen_pages_noprof()has some extra logic that wasn't previously in the nolock variant:a. Application of
gfp_allowed_mask; this only affects early boot,
only flags that affect the slowpath get changed here, and the nolock allocation path isn't allowed to theGFP_BOOT_MASKflags.
Quoted 7 lines
Quoted 5 lines
b. Application of
current_gfp_context()- also only affects the
slowpath3. The slowpath itself: this is now just explicitly skipped under !ALLOC_TRYLOCK.
s/TRYLOCK/NOLOCK
Thanks - Andrew would you mind fixing this up in mm-new?
Quoted 67 lines
Quoted 17 lines
Ulterior motive: adding an
alloc_flagsarg to the allocator's mm-internal entrypoint can later be used to do more allocation customisation without needing to create new GFP flags.No functional change intended.
Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- mm/hugetlb.c | 3 +- mm/mempolicy.c | 10 +-- mm/page_alloc.c | 192 +++++++++++++++++++++++++++++--------------------------- mm/page_alloc.h | 6 +- mm/slub.c | 6 +- 5 files changed, 117 insertions(+), 100 deletions(-)<snip>
Quoted 12 lines
mm/page_alloc.c +12
-0+/* +* This is the 'heart' of the zoned buddy allocator. +*/ +struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order, +int preferred_nid,nodemask_t *nodemask,unsigned int alloc_flags) +{ +struct page *page; +gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ +struct alloc_context ac = { }; +unsigned int fastpath_alloc_flags = alloc_flags; + +/* Other flags could be supported later if needed. */ mm/page_alloc.c void free_pages_bulk(struct page **page_array, unsigned long nr_pages) · line 5360
Quoted 10 lines
mm/page_alloc.c +8
-0+if (WARN_ON (alloc_flags & ~ALLOC_NOLOCK ))return NULL ; +if (!alloc_order_allowed (gfp, order, alloc_flags)) +return NULL ; + +if (alloc_flags &ALLOC_NOLOCK ) { +VM_WARN_ON_ONCE (gfp & ~__GFP_ACCOUNT); +if (!alloc_nolock_allowed ()) +return NULL ;At first look, I wonder why
__alloc_frozen_pages_noprof()needs to care aboutalloc_nolock_allowed(). But the patch's idea is to centralize all allocation policies, so it makes sense.Ideally, I would want
alloc_frozen_pages_nolock_noprof()to filter as much as possible, so that__alloc_frozen_pages_noprof()has minimal/no awareness ofALLOC_NOLOCK. ButALLOC_NOLOCKhas different preferences compared to the default__alloc_frozen_pages_noprof()policy likeALLOC_WMARK_MINvsALLOC_WMARK_LOW, skip slowpath, and more. Maybe we could do something like:
__alloc_frozen_pages_noprof(){ alloc_fastpath(); alloc_slowpath(); }
alloc_frozen_pages_nolock_noprof(){ alloc_order_allowed(); alloc_nolock_allow(); alloc_fastpath(); }
Quoted 3 lines
But it still cannot remove
ALLOC_NOLOCKcompletely from__alloc_frozen_pages_noprof(), like the nofragment skip. Anyway, this patch is a reasonable cleanup. Thanks.
Yeah, thanks this does look sensible but I think it's a separate cleanup.
PATCH v5 6/18 mm/page_alloc: relax GFP WARN in nolock allocs
In reply to Brendan JackmanThis WARN forbids setting other flags than __GFP_ACCOUNT but we unconditionally set the ones in gfp_nolock so they are certainly fine for the caller to set.
There are other GFP flags that are almost certainly fine to set here; Willy noted GFP_HIGHMEM, GFP_DMA, GFP_MOVABLE and GFP_HARDWALL. But, nolock allocation is rather special, so be conservative to try and ensure we have a chance to think carefully before nontrivial new usecases arise.
--- mm/page_alloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/page_alloc.c +2 -1
@@ -5355,7 +5355,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,return NULL ;if (alloc_flags &ALLOC_NOLOCK ) { -VM_WARN_ON_ONCE (gfp & ~__GFP_ACCOUNT); +/* Certain other flags could be supported later if needed. */ +VM_WARN_ON_ONCE (gfp & ~(__GFP_ACCOUNT | gfp_nolock));if (!alloc_nolock_allowed ())return NULL ; gfp |= gfp_nolock;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 06/18] mm/page_alloc: relax GFP WARN in nolock allocs
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] The kernel-doc comment for the public API `alloc_pages_nolock` was not updated to reflect the…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The kernel-doc comment for the public API alloc_pages_nolock was not updated to reflect the relaxed GFP flags. --
commit 311d28c3e8fc444b6e80f8dca47557fd6cfebc1f
Author: Brendan Jackman <[email protected]>
mm/page_alloc: relax GFP WARN in nolock allocs
This commit relaxes the VM_WARN_ON_ONCE check in nolock allocations to allow flags included in gfp_nolock. It updates the validation to permit callers to provide a wider set of flags safely.
mm/page_alloc.c struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, · line 5359
Quoted 11 lines
mm/page_alloc.c +2
-1@@ -5355,7 +5355,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,return NULL ;if (alloc_flags &ALLOC_NOLOCK ) { -VM_WARN_ON_ONCE (gfp & ~__GFP_ACCOUNT); +/* Certain other flags could be supported later if needed. */ +VM_WARN_ON_ONCE (gfp & ~(__GFP_ACCOUNT | gfp_nolock));
[Severity: Low]
Does the kernel-doc comment for alloc_pages_nolock() also need to be updated to reflect this relaxation?
Currently in mm/page_alloc.c, the documentation still states:
* @gfp_flags: GFP flags. Only __GFP_ACCOUNT allowed.
This might mislead callers who refer to the API documentation into believing they can strictly only pass __GFP_ACCOUNT, even though the implementation now permits other flags included in gfp_nolock.
Quoted 3 lines
mm/page_alloc.c +0
-0if (!alloc_nolock_allowed ())return NULL ; gfp |= gfp_nolock;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Re: [PATCH v5 06/18] mm/page_alloc: relax GFP WARN in nolock allocs
In reply to Brendan JackmanQuoted 16 lines · Brendan Jackman
This WARN forbids setting other flags than
__GFP_ACCOUNTbut we unconditionally set the ones ingfp_nolockso they are certainly fine for the caller to set.There are other GFP flags that are almost certainly fine to set here; Willy noted
GFP_HIGHMEM,GFP_DMA,GFP_MOVABLEandGFP_HARDWALL. But, nolock allocation is rather special, so be conservative to try and ensure we have a chance to think carefully before nontrivial new usecases arise.Suggested-by Matthew Wilcox <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 4 lines
--- mm/page_alloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 7/18 mm: move some stuff to mm/page_alloc.h
In reply to Brendan JackmanSome of this stuff in the public header is only used internally so shrink the scope to avoid silently growing new users.
drain_local_pages() is still used from kernel/power/snapshot.c so that needs to stay behind.
--- include/linux/gfp.h | 26 -------------------------- mm/page_alloc.h | 27 +++++++++++++++++++++++++++ mm/vmstat.c | 1 + 3 files changed, 28 insertions(+), 26 deletions(-)
include/linux/gfp.h +0 -26
@@ -17,28 +17,6 @@ struct mempolicy;#define __default_gfp (a,b,...) b#define default_gfp (...) __default_gfp(,##__VA_ARGS__,GFP_KERNEL) -/* Convert GFP flags to their corresponding migrate type */ -#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) -#define GFP_MOVABLE_SHIFT 3 - -static inline int gfp_migratetype (const gfp_t gfp_flags) -{ -VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK ); -BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE); -BUILD_BUG_ON ((___GFP_MOVABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_MOVABLE ); -BUILD_BUG_ON ((___GFP_RECLAIMABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_RECLAIMABLE ); -BUILD_BUG_ON (((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> -GFP_MOVABLE_SHIFT ) !=MIGRATE_HIGHATOMIC ); - -if (unlikely (page_group_by_mobility_disabled)) -return MIGRATE_UNMOVABLE ; - -/* Group based on mobility */ -return (__force unsigned long )(gfp_flags &GFP_MOVABLE_MASK ) >>GFP_MOVABLE_SHIFT ; -} -#undef GFP_MOVABLE_MASK -#undef GFP_MOVABLE_SHIFT -static inline bool gfpflags_allow_blocking (const gfp_t gfp_flags) {return !!(gfp_flags & __GFP_DIRECT_RECLAIM);
@@ -395,10 +373,6 @@ extern void free_pages(unsigned long addr, unsigned int order);#define __free_page (page) __free_pages((page), 0)#define free_page (addr) free_pages((addr), 0) -void page_alloc_init_cpuhp (void ); -bool decay_pcp_high (struct zone *zone,struct per_cpu_pages *pcp); -void drain_zone_pages (struct zone *zone,struct per_cpu_pages *pcp); -void drain_all_pages (struct zone *zone);void drain_local_pages (struct zone *zone);void page_alloc_init_late (void );
mm/page_alloc.h +27 -0
@@ -266,6 +266,33 @@ static inline bool free_area_empty(struct free_area *area, int migratetype) return list_empty(&area->free_list[migratetype]); } +/* Convert GFP flags to their corresponding migrate type */ +#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) +#define GFP_MOVABLE_SHIFT 3 + +static inline int gfp_migratetype (const gfp_t gfp_flags) +{ +VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK ); +BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE); +BUILD_BUG_ON ((___GFP_MOVABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_MOVABLE ); +BUILD_BUG_ON ((___GFP_RECLAIMABLE >>GFP_MOVABLE_SHIFT ) !=MIGRATE_RECLAIMABLE ); +BUILD_BUG_ON (((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> +GFP_MOVABLE_SHIFT ) !=MIGRATE_HIGHATOMIC ); + +if (unlikely (page_group_by_mobility_disabled)) +return MIGRATE_UNMOVABLE ; + +/* Group based on mobility */ +return (__force unsigned long )(gfp_flags &GFP_MOVABLE_MASK ) >>GFP_MOVABLE_SHIFT ; +} +#undef GFP_MOVABLE_MASK +#undef GFP_MOVABLE_SHIFT + +bool decay_pcp_high (struct zone *zone,struct per_cpu_pages *pcp); +void drain_zone_pages (struct zone *zone,struct per_cpu_pages *pcp); +void drain_all_pages (struct zone *zone); + +void page_alloc_init_cpuhp (void );void page_alloc_sysctl_init (void );#endif /* __MM_PAGE_ALLOC_H */
mm/vmstat.c +1 -0
@@ -30,6 +30,7 @@#include <linux/sched/isolation.h> #include "internal.h" +#include "page_alloc.h" #ifdef CONFIG_PROC_FS #ifdef CONFIG_NUMA
Signature · 1 line
2.54.0
Re: [PATCH v5 07/18] mm: move some stuff to mm/page_alloc.h
Quoted 6 lines · Brendan Jackman
Some of this stuff in the public header is only used internally so shrink the scope to avoid silently growing new users.
drain_local_pages()is still used fromkernel/power/snapshot.cso that needs to stay behind.
Quoted 8 lines
Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- include/linux/gfp.h | 26 -------------------------- mm/page_alloc.h | 27 +++++++++++++++++++++++++++ mm/vmstat.c | 1 + 3 files changed, 28 insertions(+), 26 deletions(-)
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 8/18 perf/x86/intel: Use higher-level allocator API
In reply to Brendan JackmanThe difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- arch/x86/events/intel/ds.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/events/intel/ds.c +3 -3
@@ -832,7 +832,7 @@ static void *dsalloc_pages(size_t size, gfp_t flags, int cpu)int node =cpu_to_node (cpu);struct page *page; - page =__alloc_pages_node (node, flags | __GFP_ZERO, order); + page =alloc_pages_node (node, flags | __GFP_ZERO, order);return page ?page_address (page) :NULL ; }
@@ -1088,9 +1088,9 @@ void init_arch_pebs_on_cpu(int cpu)/* * 4KB-aligned pointer of the output buffer -* (__alloc_pages_node() return page aligned address) +* (alloc_pages_node() returns page aligned address) * Buffer Size = 4KB * 2^SIZE -* contiguous physical buffer (__alloc_pages_node() with order) +* contiguous physical buffer (alloc_pages_node() with order) */ arch_pebs_base =virt_to_phys (cpuc->pebs_vaddr) |PEBS_BUFFER_SHIFT ;wrmsrq_on_cpu (cpu,MSR_IA32_PEBS_BASE , arch_pebs_base);
Signature · 1 line
2.54.0
Re: [PATCH v5 08/18] perf/x86/intel: Use higher-level allocator API
Quoted 21 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Cc Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Namhyung Kim <[email protected]>Cc Mark Rutland <[email protected]>Cc Alexander Shishkin <[email protected]>Cc Jiri Olsa <[email protected]>Cc Ian Rogers <[email protected]>Cc Adrian Hunter <[email protected]>Cc James Clark <[email protected]>Assisted-by Gemini:unknown-versionReviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 4 lines
--- arch/x86/events/intel/ds.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 9/18 KVM: VMX: Use higher-level allocator API
In reply to Brendan JackmanThe difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- arch/x86/kvm/vmx/vmx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -3028,7 +3028,7 @@ struct vmcs *alloc_vmcs_cpu(bool shadow, int cpu, gfp_t flags)struct page *pages;struct vmcs *vmcs; - pages =__alloc_pages_node (node, flags,0 ); + pages =alloc_pages_node (node, flags,0 );if (!pages)return NULL ; vmcs =page_address (pages);
Signature · 1 line
2.54.0
Re: [PATCH v5 09/18] KVM: VMX: Use higher-level allocator API
Quoted 14 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Cc Sean Christopherson <[email protected]>Cc Paolo Bonzini <[email protected]>Assisted-by Gemini:unknown-versionReviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 4 lines
--- arch/x86/kvm/vmx/vmx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH v5 09/18] KVM: VMX: Use higher-level allocator API
In reply to Brendan JackmanQuoted 7 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Quoted 8 lines
Cc Sean Christopherson <[email protected]>Cc Paolo Bonzini <[email protected]>Assisted-by Gemini:unknown-versionReviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>---
Feel free to take this through mm/, I can't imagine this will conflict with anything going through the KVM tree. Or, holler at me if you want me to take this through KVM x86.
PATCH v5 10/18 x86/virt: Use higher-level allocator API
In reply to Brendan JackmanThe difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- arch/x86/virt/hw.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/virt/hw.c +1 -1
@@ -196,7 +196,7 @@ static __init int __x86_vmx_init(void)struct page *page;struct vmcs *vmcs; - page =__alloc_pages_node (node,GFP_KERNEL | __GFP_ZERO,0 ); + page =alloc_pages_node (node,GFP_KERNEL | __GFP_ZERO,0 );if (WARN_ON_ONCE (!page)) {x86_vmx_exit ();return -ENOMEM ;
Signature · 1 line
2.54.0
Re: [PATCH v5 10/18] x86/virt: Use higher-level allocator API
Quoted 17 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Cc "H. Peter Anvin" <[email protected]>Assisted-by Gemini:unknown-versionReviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 4 lines
--- arch/x86/virt/hw.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 11/18 sgi-xp: Use higher-level allocator API
In reply to Brendan JackmanThe difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- drivers/misc/sgi-xp/xpc_uv.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/misc/sgi-xp/xpc_uv.c +2 -3
@@ -170,9 +170,8 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, mq->mmr_blade =uv_cpu_to_blade_id (cpu); nid =cpu_to_node (cpu); - page =__alloc_pages_node (nid, -GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE, - pg_order); + page =alloc_pages_node (nid,GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE, + pg_order);if (page ==NULL ) {dev_err (xpc_part,"xpc_create_gru_mq_uv() failed to alloc %d " "bytes of memory on nid=%d for GRU mq\n" , mq_size, nid);
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 11/18] sgi-xp: Use higher-level allocator API
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] In xpc_init_mq_node(), the loop iterates over CPUs but mistakenly passes the nid (node ID) variable to xpc_create_gru_mq_uv() instead of the cpu iterator. --
commit 60dc55f94c78f2b8722b9d132aa85a4603393bdd
Author: Brendan Jackman <[email protected]>
sgi-xp: Use higher-level allocator API
The difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
drivers/misc/sgi-xp/xpc_uv.c xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, · line 172
Quoted 8 lines
drivers/misc/sgi-xp/xpc_uv.c +0
-0@@ -170,9 +170,8 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, mq->mmr_blade =uv_cpu_to_blade_id (cpu); nid =cpu_to_node (cpu);
[Severity: High]
This isn't a bug introduced by this patch, but while looking at how the cpu argument is handled here in xpc_create_gru_mq_uv(), I noticed an issue in its caller.
In xpc_init_mq_node():
for_each_cpu(cpu, cpumask_of_node(nid)) {
xpc_activate_mq_uv =
xpc_create_gru_mq_uv(XPC_ACTIVATE_MQ_SIZE_UV, nid,
XPC_ACTIVATE_IRQ_NAME,
xpc_handle_activate_IRQ_uv);Is it possible that the node ID (nid) is mistakenly passed into xpc_create_gru_mq_uv() instead of the cpu iterator? This also happens a few lines later when allocating xpc_notify_mq_uv.
Passing a node ID where a CPU index is expected will cause per-CPU lookups like uv_cpu_to_blade_id() and cpu_to_node() to access data for the wrong CPU. If the node ID exceeds nr_cpu_ids, this could also cause out of bounds memory accesses.
Quoted 8 lines
drivers/misc/sgi-xp/xpc_uv.c +2
-3- page =__alloc_pages_node (nid, -GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE, - pg_order); + page =alloc_pages_node (nid,GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE, + pg_order);if (page ==NULL ) {dev_err (xpc_part,"xpc_create_gru_mq_uv() failed to alloc %d " "bytes of memory on nid=%d for GRU mq\n" , mq_size, nid);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Re: [PATCH v5 11/18] sgi-xp: Use higher-level allocator API
In reply to Brendan JackmanQuoted 17 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Cc Robin Holt <[email protected]>Cc Steve Wahl <[email protected]>Cc Arnd Bergmann <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Assisted-by Gemini:unknown-modelReviewed-by Suren Baghdasaryan <[email protected]>Acked-by Steve Wahl <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 4 lines
--- drivers/misc/sgi-xp/xpc_uv.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 12/18 net/funeth: Switch to higher-level allocator API
In reply to Brendan JackmanThe difference between __alloc_pages_node() and alloc_pages_node() is that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
--- drivers/net/ethernet/fungible/funeth/funeth_rx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/fungible/funeth/funeth_rx.c +1 -1
@@ -103,7 +103,7 @@ static int funeth_alloc_page(struct funeth_rxq *q, struct funeth_rxbuf *rb,if (cache_get (q, rb))return 0 ; - p =__alloc_pages_node (node, gfp | __GFP_NOWARN,0 ); + p =alloc_pages_node (node, gfp | __GFP_NOWARN,0 );if (unlikely (!p))return -ENOMEM ;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 12/18] net/funeth: Switch to higher-level allocator API
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [Critical] Untrusted packet length from hardware CQE leads to stack buffer overflow and OOB memory corruption. --
commit fd0faaa7ff432d936af6fe6e5da6c09cbf2f16eb
Author: Brendan Jackman <[email protected]>
net/funeth: Switch to higher-level allocator API
This commit replaces the deprecated __alloc_pages_node() with alloc_pages_node(). The new API supports passing NUMA_NO_NODE. No functional changes are intended by this switch.
drivers/net/ethernet/fungible/funeth/funeth_rx.c static int funeth_alloc_page(struct funeth_rxq *q, struct funeth_rxbuf *rb, · line 106
Quoted 10 lines
drivers/net/ethernet/fungible/funeth/funeth_rx.c +1
-1@@ -103,7 +103,7 @@ static int funeth_alloc_page(struct funeth_rxq *q, struct funeth_rxbuf *rb,if (cache_get (q, rb))return 0 ; - p =__alloc_pages_node (node, gfp | __GFP_NOWARN,0 ); + p =alloc_pages_node (node, gfp | __GFP_NOWARN,0 );
[Severity: Critical]
This is a pre-existing issue, but while reviewing this file I noticed a potential stack buffer overflow when handling untrusted packet lengths from the hardware CQE.
In fun_handle_cqe_pkt(), the untrusted packet length and fragment count are read from the CQE:
unsigned int i, tot_len, pkt_len = be32_to_cpu(rxreq->pkt_len);
...
tot_len = pkt_len;
...
ref_ok = fun_gather_pkt(q, tot_len, frags);
...
skb = napi_build_skb(va, ALIGN(tot_len, FUN_EPRQ_PKT_ALIGN));
...
si->nr_frags = rxreq->nsgl;
for (i = 0; i < si->nr_frags; i++)
si->frags[i] = frags[i];Does the loop in fun_gather_pkt() iterate until tot_len is exhausted without validating the size?
for (;;) {
...
skb_frag_fill_page_desc(frags++, buf->page, q->buf_offset, frag_len);
tot_len -= frag_len;
if (!tot_len)
break;
}If rxreq->pkt_len is maliciously large, could this increment the frags array pointer past the bounds of the stack-allocated frags array (size 4), resulting in a stack buffer overflow?
Furthermore, could napi_build_skb() use this untrusted length to place skb_shared_info at an out-of-bounds offset, corrupting unrelated memory?
Quoted 3 lines
drivers/net/ethernet/fungible/funeth/funeth_rx.c +0
-0if (unlikely (!p))return -ENOMEM ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Re: [PATCH v5 12/18] net/funeth: Switch to higher-level allocator API
In reply to Brendan JackmanQuoted 17 lines · Brendan Jackman
The difference between
__alloc_pages_node()andalloc_pages_node()is that the latter allows you to passNUMA_NO_NODE.The former is going away and the latter works fine here so switch over.
No functional change intended.
Cc Dimitris Michailidis <[email protected]>Cc Andrew Lunn <[email protected]>Cc "David S. Miller" <[email protected]>Cc Eric Dumazet <[email protected]>Cc Jakub Kicinski <[email protected]>Cc Paolo Abeni <[email protected]>Assisted-by Gemini:unknown-versionReviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 4 lines
--- drivers/net/ethernet/fungible/funeth/funeth_rx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 13/18 mm: Remove __alloc_pages_node()
In reply to Brendan JackmanThere were only a few users, which have been removed. The only advantage of this API over alloc_pages_node() is avoiding a single conditional branch. The disadvantages are:
1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of __alloc_pages_node() were using the result of cpu_to_node(); if the CPU gets hotplugged out this will return NUMA_NO_NODE. If one of these paths fails to protect against a concurrent hotplug then page_alloc.c will use NUMA_NO_NODE as an index into NODE_DATA() and cause some horrible memory corruption or other. With alloc_pages_node(), the code might just work fine.
Ulterior motive: this frees up the __* variants of the allocator APIs to serve specifically for use as mm-internal API.
--- include/linux/gfp.h | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)
include/linux/gfp.h +4 -16
@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {
@@ -293,7 +278,10 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,if (nid ==NUMA_NO_NODE ) nid =numa_mem_id (); -return __alloc_pages_node_noprof (nid, gfp_mask, order); +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); +warn_if_node_offline (nid, gfp_mask); + +return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); }#define alloc_pages_node (...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__))
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 13/18] mm: Remove __alloc_pages_node()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [Critical] The patch removes __alloc_pages_node() to fix CPU hotplug bugs but completely misses the equivalent __folio_alloc_node() API, leaving it exposed with the same NUMA_NO_NODE vulnerability. --
commit 4b93237c4920d8046053a32bcd0656e3236e97c9
Author: Brendan Jackman <[email protected]>
mm: Remove __alloc_pages_node()
There were only a few users, which have been removed. The only advantage of this API over alloc_pages_node() is avoiding a single conditional branch.
Quoted 4 lines
include/linux/gfp.h +0
-0
[ ... ]
Quoted 12 lines
include/linux/gfp.h +0
-8@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );
include/linux/gfp.h static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) · line 261
Quoted 10 lines
include/linux/gfp.h +0
-7-warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {
[Severity: Critical]
This is a pre-existing issue, but does __folio_alloc_node_noprof() suffer from the exact same NUMA_NO_NODE vulnerability that you're fixing here?
This function lacks the fallback safety check for NUMA_NO_NODE that alloc_pages_node() provides.
For example, looking at svc_prepare_thread() in net/sunrpc/svc.c, it calls __folio_alloc_node() using a node ID that can be derived from cpu_to_node():
net/sunrpc/svc.c:svc_prepare_thread() {
...
rqstp->rq_scratch_folio = __folio_alloc_node(GFP_KERNEL, 0, node);
if (!rqstp->rq_scratch_folio)
goto out_enomem;
...
}If a CPU hotplug event causes the node to be NUMA_NO_NODE, couldn't this trigger the same out-of-bounds array access into NODE_DATA() and result in the memory corruption mentioned in the commit message?
Should __folio_alloc_node() also be removed or replaced with a safe folio_alloc_node() wrapper?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Re: [PATCH v5 13/18] mm: Remove __alloc_pages_node()
In reply to Brendan JackmanQuoted 51 lines · Brendan Jackman
There were only a few users, which have been removed. The only advantage of this API over
alloc_pages_node()is avoiding a single conditional branch. The disadvantages are:1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of
__alloc_pages_node()were using the result ofcpu_to_node(); if the CPU gets hotplugged out this will returnNUMA_NO_NODE. If one of these paths fails to protect against a concurrent hotplug thenpage_alloc.cwill useNUMA_NO_NODEas an index intoNODE_DATA()and cause some horrible memory corruption or other. Withalloc_pages_node(), the code might just work fine.Ulterior motive: this frees up the __* variants of the allocator APIs to serve specifically for use as mm-internal API.
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- include/linux/gfp.h | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)include/linux/gfp.h +0
-15@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {
include/linux/gfp.h static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, · line 281
Quoted 5 lines
include/linux/gfp.h +1
-1@@ -293,7 +278,10 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,if (nid ==NUMA_NO_NODE ) nid =numa_mem_id (); -return __alloc_pages_node_noprof (nid, gfp_mask, order); +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );
Could this become a VM_WARN_ON?
Anyway,
Quoted 6 lines
include/linux/gfp.h +3
-0+warn_if_node_offline (nid, gfp_mask); + +return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); }#define alloc_pages_node (...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__))
Signature · 2 lines
Best Regards, Yan, Zi
Brendan Jackman
reviewerRe: [PATCH v5 13/18] mm: Remove __alloc_pages_node()
Quoted 55 lines · Zi Yan
Quoted 54 lines · Brendan Jackman
There were only a few users, which have been removed. The only advantage of this API over
alloc_pages_node()is avoiding a single conditional branch. The disadvantages are:1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of
__alloc_pages_node()were using the result ofcpu_to_node(); if the CPU gets hotplugged out this will returnNUMA_NO_NODE. If one of these paths fails to protect against a concurrent hotplug thenpage_alloc.cwill useNUMA_NO_NODEas an index intoNODE_DATA()and cause some horrible memory corruption or other. Withalloc_pages_node(), the code might just work fine.Ulterior motive: this frees up the __* variants of the allocator APIs to serve specifically for use as mm-internal API.
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- include/linux/gfp.h | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)include/linux/gfp.h +0
-15@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {@@ -293,7 +278,10 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,if (nid ==NUMA_NO_NODE ) nid =numa_mem_id ();
Quoted 4 lines
include/linux/gfp.h static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, · line 281
Quoted 2 lines
include/linux/gfp.h +1
-1-return __alloc_pages_node_noprof (nid, gfp_mask, order); +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );Could this become a
VM_WARN_ON?
Er, it will almost certainly crash later if this is violated. Personally if I'm enabling DEBUG_VM I'd rather it crashes with a helpful stacktrace instead of chasing random poihnters, likely corrupting memory, and then crashing later in a completely undebuggable way instead.
TBH I don't really understand the "don't BUG" culture we have in Linux. I usually just go along with it coz it doesn't seem important enough to argue about. But now I'm wondering: if we really don't like VM_BUG_ON() why not get rid of it completely? I wonder if there's some case-by-case difference that I'm not picking up on.
Zi Yan
reviewerRe: [PATCH v5 13/18] mm: Remove __alloc_pages_node()
Quoted 56 lines · Brendan Jackman
Quoted 55 lines · Zi Yan
Quoted 54 lines · Brendan Jackman
There were only a few users, which have been removed. The only advantage of this API over
alloc_pages_node()is avoiding a single conditional branch. The disadvantages are:1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of
__alloc_pages_node()were using the result ofcpu_to_node(); if the CPU gets hotplugged out this will returnNUMA_NO_NODE. If one of these paths fails to protect against a concurrent hotplug thenpage_alloc.cwill useNUMA_NO_NODEas an index intoNODE_DATA()and cause some horrible memory corruption or other. Withalloc_pages_node(), the code might just work fine.Ulterior motive: this frees up the __* variants of the allocator APIs to serve specifically for use as mm-internal API.
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- include/linux/gfp.h | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)include/linux/gfp.h +0
-15@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {@@ -293,7 +278,10 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,if (nid ==NUMA_NO_NODE ) nid =numa_mem_id ();
Quoted 6 lines
Quoted 4 lines
include/linux/gfp.h static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, · line 281
Quoted 2 lines
include/linux/gfp.h +1
-1-return __alloc_pages_node_noprof (nid, gfp_mask, order); +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );Could this become a
VM_WARN_ON?Er, it will almost certainly crash later if this is violated. Personally
You mean in warn_if_node_offline() when accessing the nodemask?
Quoted 3 lines
if I'm enabling
DEBUG_VMI'd rather it crashes with a helpful stacktrace instead of chasing random poihnters, likely corrupting memory, and then crashing later in a completely undebuggable way instead.
In this case, VM_WARN_ON + return NULL could work, right?
Quoted 6 lines
TBH I don't really understand the "don't BUG" culture we have in Linux. I usually just go along with it coz it doesn't seem important enough to argue about. But now I'm wondering: if we really don't like
VM_BUG_ON()why not get rid of it completely? I wonder if there's some case-by-case difference that I'm not picking up on.
My understanding is that issues like the above one should be caught at development phase, so VM_WARN_ON() is sufficient.
The source of “don’t BUG”: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1cfd9d7e43d5a1cf739d1420b10b1e65feb02f88
You can read the commit message to know when BUG_ON() can be used.
Best Regards,
Yan, Zi
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v5 13/18] mm: Remove __alloc_pages_node()
Quoted 65 lines · Zi Yan
Quoted 56 lines · Brendan Jackman
Quoted 55 lines · Zi Yan
Quoted 54 lines · Brendan Jackman
There were only a few users, which have been removed. The only advantage of this API over
alloc_pages_node()is avoiding a single conditional branch. The disadvantages are:1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of
__alloc_pages_node()were using the result ofcpu_to_node(); if the CPU gets hotplugged out this will returnNUMA_NO_NODE. If one of these paths fails to protect against a concurrent hotplug thenpage_alloc.cwill useNUMA_NO_NODEas an index intoNODE_DATA()and cause some horrible memory corruption or other. Withalloc_pages_node(), the code might just work fine.Ulterior motive: this frees up the __* variants of the allocator APIs to serve specifically for use as mm-internal API.
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- include/linux/gfp.h | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)include/linux/gfp.h +0
-15@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {@@ -293,7 +278,10 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,if (nid ==NUMA_NO_NODE ) nid =numa_mem_id ();Quoted 6 lines
Quoted 4 lines
include/linux/gfp.h static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, · line 281
Quoted 2 lines
include/linux/gfp.h +1
-1-return __alloc_pages_node_noprof (nid, gfp_mask, order); +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );Could this become a
VM_WARN_ON?Er, it will almost certainly crash later if this is violated. Personally
Quoted 7 lines
You mean in
warn_if_node_offline()when accessing the nodemask?Quoted 3 lines
if I'm enabling
DEBUG_VMI'd rather it crashes with a helpful stacktrace instead of chasing random poihnters, likely corrupting memory, and then crashing later in a completely undebuggable way instead.In this case,
VM_WARN_ON+ return NULL could work, right?
See below.
Quoted 7 lines
Quoted 6 lines
TBH I don't really understand the "don't BUG" culture we have in Linux. I usually just go along with it coz it doesn't seem important enough to argue about. But now I'm wondering: if we really don't like
VM_BUG_ON()why not get rid of it completely? I wonder if there's some case-by-case difference that I'm not picking up on.
Quoted 5 lines
My understanding is that issues like the above one should be caught at development phase, so
VM_WARN_ON()is sufficient.The source of “don’t BUG”: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1cfd9d7e43d5a1cf739d1420b10b1e65feb02f88
You can read the commit message to know whenBUG_ON()can be used.
Quoted from there:
I'd prefer to keep all these warnings 'simple' - i.e. no attempted recovery & control flow, unless we ever expect these to trigger.
[5]
So I wouldn't attempt recovery here.
Quoted 3 lines
Best Regards,
Yan, Zi
Zi Yan
reviewerRe: [PATCH v5 13/18] mm: Remove __alloc_pages_node()
Quoted 89 lines · Vlastimil Babka (SUSE)
Quoted 65 lines · Zi Yan
Quoted 56 lines · Brendan Jackman
Quoted 55 lines · Zi Yan
Quoted 54 lines · Brendan Jackman
There were only a few users, which have been removed. The only advantage of this API over
alloc_pages_node()is avoiding a single conditional branch. The disadvantages are:1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of
__alloc_pages_node()were using the result ofcpu_to_node(); if the CPU gets hotplugged out this will returnNUMA_NO_NODE. If one of these paths fails to protect against a concurrent hotplug thenpage_alloc.cwill useNUMA_NO_NODEas an index intoNODE_DATA()and cause some horrible memory corruption or other. Withalloc_pages_node(), the code might just work fine.Ulterior motive: this frees up the __* variants of the allocator APIs to serve specifically for use as mm-internal API.
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- include/linux/gfp.h | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)include/linux/gfp.h +0
-15@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {@@ -293,7 +278,10 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,if (nid ==NUMA_NO_NODE ) nid =numa_mem_id ();Quoted 6 lines
Quoted 4 lines
include/linux/gfp.h static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, · line 281
Quoted 2 lines
include/linux/gfp.h +1
-1-return __alloc_pages_node_noprof (nid, gfp_mask, order); +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );Could this become a
VM_WARN_ON?Er, it will almost certainly crash later if this is violated. Personally
Quoted 7 lines
You mean in
warn_if_node_offline()when accessing the nodemask?Quoted 3 lines
if I'm enabling
DEBUG_VMI'd rather it crashes with a helpful stacktrace instead of chasing random poihnters, likely corrupting memory, and then crashing later in a completely undebuggable way instead.In this case,
VM_WARN_ON+ return NULL could work, right?See below.
Quoted 7 lines
Quoted 6 lines
TBH I don't really understand the "don't BUG" culture we have in Linux. I usually just go along with it coz it doesn't seem important enough to argue about. But now I'm wondering: if we really don't like
VM_BUG_ON()why not get rid of it completely? I wonder if there's some case-by-case difference that I'm not picking up on.Quoted 5 lines
My understanding is that issues like the above one should be caught at development phase, so
VM_WARN_ON()is sufficient.The source of “don’t BUG”: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1cfd9d7e43d5a1cf739d1420b10b1e65feb02f88
You can read the commit message to know whenBUG_ON()can be used.
Quoted 7 lines
Quoted from there:
I'd prefer to keep all these warnings 'simple' - i.e. no attempted recovery & control flow, unless we ever expect these to trigger.
[5]So I wouldn't attempt recovery here.
Ah, missed that part. Yes, for this case, we do not expect the conditions in VM_BUG_ON().
Best Regards,
Yan, Zi
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v5 13/18] mm: Remove __alloc_pages_node()
In reply to Brendan JackmanQuoted 56 lines · Brendan Jackman
Quoted 55 lines · Zi Yan
Quoted 54 lines · Brendan Jackman
There were only a few users, which have been removed. The only advantage of this API over
alloc_pages_node()is avoiding a single conditional branch. The disadvantages are:1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of
__alloc_pages_node()were using the result ofcpu_to_node(); if the CPU gets hotplugged out this will returnNUMA_NO_NODE. If one of these paths fails to protect against a concurrent hotplug thenpage_alloc.cwill useNUMA_NO_NODEas an index intoNODE_DATA()and cause some horrible memory corruption or other. Withalloc_pages_node(), the code might just work fine.Ulterior motive: this frees up the __* variants of the allocator APIs to serve specifically for use as mm-internal API.
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- include/linux/gfp.h | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)include/linux/gfp.h +0
-15@@ -256,21 +256,6 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)dump_stack (); } -/* -* Allocate pages, preferring the node given as nid. The node must be valid and -* online. For more general interface, see alloc_pages_node(). -*/ -static inline struct page * -__alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) -{ -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} - -#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) -static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {@@ -293,7 +278,10 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,if (nid ==NUMA_NO_NODE ) nid =numa_mem_id ();
Quoted 6 lines
Quoted 4 lines
include/linux/gfp.h static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, · line 281
Quoted 2 lines
include/linux/gfp.h +1
-1-return __alloc_pages_node_noprof (nid, gfp_mask, order); +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );Could this become a
VM_WARN_ON?Er, it will almost certainly crash later if this is violated. Personally
Agreed.
Quoted 5 lines
if I'm enabling
DEBUG_VMI'd rather it crashes with a helpful stacktrace instead of chasing random poihnters, likely corrupting memory, and then crashing later in a completely undebuggable way instead.TBH I don't really understand the "don't BUG" culture we have in Linux.
It comes from Linus. The main motivation AFAIU are cases that would be recoverable otherwise. This is almost certainly not the one, but it's hard sell anyway.
I think he would also argue that the "almost certainly crash" you argue about above should give you enough info for you to debug it :) It's likely not the case that would silently corrupt something and then you crash in completely different context.
Quoted 4 lines
I usually just go along with it coz it doesn't seem important enough to argue about. But now I'm wondering: if we really don't like
VM_BUG_ON()why not get rid of it completely? I wonder if there's some case-by-case difference that I'm not picking up on.
IIRC we did agree on getting rid of it, but there was no concerted effort yet. So we're in this "if you touch something, remove it" mode for a while.
Here I think you can just remove it here completely, and not replace it with a warn. Realistically it was only potentially useful for an unexpected NUMA_NO_NODE passed to __alloc_pages_node_noprof() and that's gone with your patch. So keeping that check would be an arbitrary paranoia at this point.
PATCH v5 14/18 mm: Move __alloc_pages() to mm/page_alloc.h
In reply to Brendan JackmanIt's no longer used outside of mm/.
Since this means __alloc_pages_noprof() is no longer visible from gfp.h, this also means moving the definition of alloc_pages_node_noprof into the .c file.
Also remove references to this API from the documentation tree - referring to the specific function name was already questionable but now the function is not even public it definitely seems wrong.
--- Documentation/admin-guide/cgroup-v1/cpusets.rst | 2 +- Documentation/admin-guide/mm/transhuge.rst | 2 +- include/linux/gfp.h | 16 +--------------- mm/page_alloc.c | 13 ++++++++++++- mm/page_alloc.h | 4 ++++ 5 files changed, 19 insertions(+), 18 deletions(-)
Documentation/admin-guide/cgroup-v1/cpusets.rst +1 -1
@@ -284,7 +284,7 @@ take action. ==> Unless this feature is enabled by writing "1" to the special file /dev/cpuset/memory_pressure_enabled, the hook in the rebalance - code of __alloc_pages() for this metric reduces to simply noticing + code of the page allocator for this metric reduces to simply noticing that the cpuset_memory_pressure_enabled flag is zero. So only systems that enable this feature will compute the metric.
Documentation/admin-guide/mm/transhuge.rst +1 -1
@@ -761,7 +761,7 @@ compact_fail but failed. It is possible to establish how long the stalls were using the function -tracer to record how long was spent in __alloc_pages() and +tracer to record how long was spent in the page allocator and using the mm_page_alloc tracepoint to identify which allocations were for huge pages.
include/linux/gfp.h +1 -15
@@ -204,10 +204,6 @@ static inline void arch_free_page(struct page *page, int order) { }static inline void arch_alloc_page (struct page *page,int order) { }#endif -struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid, -nodemask_t *nodemask); -#define __alloc_pages (...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) -struct folio *__folio_alloc_noprof (gfp_t gfp,unsigned int order,int preferred_nid,nodemask_t *nodemask);#define __folio_alloc (...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))
@@ -272,17 +268,7 @@ struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)* prefer the current CPU's closest node. Otherwise node must be valid and * online. */ -static inline struct page *alloc_pages_node_noprof (int nid,gfp_t gfp_mask, -unsigned int order) -{ -if (nid ==NUMA_NO_NODE ) - nid =numa_mem_id (); - -VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); -warn_if_node_offline (nid, gfp_mask); - -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); -} +struct page *alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order);#define alloc_pages_node (...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__))
mm/page_alloc.c +12 -1
@@ -5431,7 +5431,18 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order,set_page_refcounted (page);return page; } -EXPORT_SYMBOL (__alloc_pages_noprof); + +struct page *alloc_pages_node_noprof (int nid,gfp_t gfp_mask,unsigned int order) +{ +if (nid ==NUMA_NO_NODE ) + nid =numa_mem_id (); + +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); +warn_if_node_offline (nid, gfp_mask); + +return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); +} +EXPORT_SYMBOL (alloc_pages_node_noprof);struct folio *__folio_alloc_noprof (gfp_t gfp,unsigned int order,int preferred_nid,nodemask_t *nodemask)
mm/page_alloc.h +4 -0
@@ -244,6 +244,10 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsignedalloc_hooks (alloc_frozen_pages_nolock_noprof(__VA_ARGS__ )) void free_frozen_pages_nolock(struct page *page, unsigned int order); +struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid, +nodemask_t *nodemask); +#define __alloc_pages (...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) +extern void zone_pcp_reset (struct zone *zone);extern void zone_pcp_disable (struct zone *zone);extern void zone_pcp_enable (struct zone *zone);
Signature · 1 line
2.54.0
Re: [PATCH v5 14/18] mm: Move __alloc_pages() to mm/page_alloc.h
Quoted 25 lines · Brendan Jackman
It's no longer used outside of
mm/.Since this means
__alloc_pages_noprof()is no longer visible fromgfp.h, this also means moving the definition ofalloc_pages_node_noprofinto the .c file.Also remove references to this API from the documentation tree - referring to the specific function name was already questionable but now the function is not even public it definitely seems wrong.
Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- Documentation/admin-guide/cgroup-v1/cpusets.rst | 2 +- Documentation/admin-guide/mm/transhuge.rst | 2 +- include/linux/gfp.h | 16 +--------------- mm/page_alloc.c | 13 ++++++++++++- mm/page_alloc.h | 4 ++++ 5 files changed, 19 insertions(+), 18 deletions(-)Documentation/admin-guide/cgroup-v1/cpusets.rst +0
-0
Documentation/admin-guide/cgroup-v1/cpusets.rst take action. · line 290
Quoted 8 lines
Documentation/admin-guide/cgroup-v1/cpusets.rst +1
-1@@ -284,7 +284,7 @@ take action. ==> Unless this feature is enabled by writing "1" to the special file /dev/cpuset/memory_pressure_enabled, the hook in the rebalance - code of __alloc_pages() for this metric reduces to simply noticing + code of the page allocator for this metric reduces to simply noticing that the cpuset_memory_pressure_enabled flag is zero. So only systems that enable this feature will compute the metric.
kernel/cgroup/cpuset.c still has 3 references to __alloc_pages(). They can be converted as well. Otherwise,
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 15/18 mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG
In reply to Brendan JackmanNow that alloc_pages has an entrypoint that allows passing alloc_flags, we can take advantage of this to start removing GFP flags that are only used for mm-internal stuff.
This requires also plumbing the alloc_flags into some more of the allocator code, in particular __alloc_pages[_noprof]() gets an alloc_flags arg to go along with its callees, and we now need to pass those flags deeper into the allocator so they can reach the alloc_tag code.
While moving the flag definition into page_alloc.h, also update the comment per Hao's suggestion.
No functional change intended.
--- include/linux/alloc_tag.h | 4 ++-- mm/alloc_tag.c | 23 +++++++-------------- mm/compaction.c | 4 ++-- mm/page_alloc.c | 52 +++++++++++++++++++++++++++-------------------- mm/page_alloc.h | 14 +++++++++++-- mm/page_frag_cache.c | 4 ++-- 6 files changed, 55 insertions(+), 46 deletions(-)
include/linux/alloc_tag.h +2 -2
@@ -163,11 +163,11 @@ static inline void alloc_tag_sub_check(union codetag_ref *ref) {WARN_ONCE (ref && !ref->ct,"alloc_tag was not set\n" ); } -void alloc_tag_add_early_pfn (unsigned long pfn,gfp_t gfp_flags); +void alloc_tag_add_early_pfn (unsigned long pfn,unsigned int alloc_flags);#else static inline void alloc_tag_add_check (union codetag_ref *ref,struct alloc_tag *tag) {}static inline void alloc_tag_sub_check (union codetag_ref *ref) {} -static inline void alloc_tag_add_early_pfn (unsigned long pfn,gfp_t gfp_flags) {} +static inline void alloc_tag_add_early_pfn (unsigned long pfn,unsigned int alloc_flags) {}#endif /* Caller should verify both ref and tag to be valid */
mm/alloc_tag.c +7 -16
@@ -15,6 +15,9 @@#include <linux/vmalloc.h> #include <linux/kmemleak.h> +#include "internal.h" +#include "page_alloc.h" +#define ALLOCINFO_FILE_NAME "allocinfo"#define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag))#define SECTION_START (NAME ) (CODETAG_SECTION_START_PREFIX NAME)
@@ -783,19 +786,6 @@ struct pfn_pool {#define PFN_POOL_SIZE ((PAGE_SIZE - offsetof(struct pfn_pool, pfns)) / \ sizeof(unsigned long)) - -/* -* Skip early PFN recording for a page allocation. Reuses the -* %__GFP_NO_OBJ_EXT bit. Used by __alloc_tag_add_early_pfn() to avoid -* recursion when allocating pages for the early PFN tracking list -* itself. -* -* Codetags of the pages allocated with __GFP_NO_CODETAG should be -* cleared (via clear_page_tag_ref()) before freeing the pages to prevent -* alloc_tag_sub_check() from triggering a warning. -*/ -#define __GFP_NO_CODETAG __GFP_NO_OBJ_EXT -static struct pfn_pool *current_pfn_pool__initdata ;static void __init __alloc_tag_add_early_pfn (unsigned long pfn)
@@ -806,7 +796,8 @@ static void __init __alloc_tag_add_early_pfn(unsigned long pfn) do { pool = READ_ONCE(current_pfn_pool); if (!pool || atomic_read(&pool->count) >= PFN_POOL_SIZE) { - struct page *new_page = alloc_page(__GFP_HIGH | __GFP_NO_CODETAG); + struct page *new_page = __alloc_pages(__GFP_HIGH, 0, numa_mem_id(), + NULL, ALLOC_NO_CODETAG); struct pfn_pool *new; if (!new_page) {
@@ -837,7 +828,7 @@ typedef void alloc_tag_add_func(unsigned long pfn);static alloc_tag_add_func __rcu *alloc_tag_add_early_pfn_ptr__refdata =RCU_INITIALIZER (__alloc_tag_add_early_pfn); -void alloc_tag_add_early_pfn (unsigned long pfn,gfp_t gfp_flags) +void alloc_tag_add_early_pfn (unsigned long pfn,unsigned int alloc_flags) {alloc_tag_add_func *alloc_tag_add;
@@ -845,7 +836,7 @@ void alloc_tag_add_early_pfn(unsigned long pfn, gfp_t gfp_flags)return ;/* Skip allocations for the tracking list itself to avoid recursion. */ -if (gfp_flags & __GFP_NO_CODETAG) +if (alloc_flags &ALLOC_NO_CODETAG )return ;rcu_read_lock ();
mm/compaction.c +2 -2
@@ -83,7 +83,7 @@ static inline bool is_via_compact_memory(int order) { return false; }static struct page *mark_allocated_noprof (struct page *page,unsigned int order,gfp_t gfp_flags) { -post_alloc_hook (page, order, __GFP_MOVABLE); +post_alloc_hook (page, order, __GFP_MOVABLE,ALLOC_DEFAULT );set_page_refcounted (page);return page; }
@@ -1851,7 +1851,7 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long da } dst = (struct folio *)freepage; -post_alloc_hook (&dst->page, order, __GFP_MOVABLE); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE,ALLOC_DEFAULT );set_page_refcounted (&dst->page);if (order)prep_compound_page (&dst->page, order);
mm/page_alloc.c +30 -22
181 lines not on this page · raw message
mm/page_alloc.h +12 -2
@@ -49,6 +49,13 @@#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_NOLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +/* +* Avoid alloc_tag recursion for internal allocations. +* +* Callers must clear_page_tag_ref() before freeing to avoid warnings from +* alloc_tag_sub_check(). +*/ +#define ALLOC_NO_CODETAG 0x1000/* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
@@ -84,6 +91,8 @@ struct alloc_context {*/ enum zone_type highest_zoneidx;bool spread_dirty_pages; +/* Only flags that are global to the whole allocation go here. */ +unsigned int alloc_flags; };/*
@@ -214,7 +223,8 @@ static inline struct page *pageblock_pfn_to_page(unsigned long start_pfn,extern void __free_pages_core (struct page *page,unsigned int order,enum meminit_context context); -void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags); +void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags, +unsigned int alloc_flags);extern bool free_pages_prepare (struct page *page,unsigned int order);extern int user_min_free_kbytes;
@@ -245,7 +255,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsignedvoid free_frozen_pages_nolock (struct page *page,unsigned int order);struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid, -nodemask_t *nodemask); +nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_pages (...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__))extern void zone_pcp_reset (struct zone *zone);
mm/page_frag_cache.c +2 -2
@@ -57,10 +57,10 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, gfp_mask = (gfp_mask & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; page =__alloc_pages (gfp_mask,PAGE_FRAG_CACHE_MAX_ORDER , -numa_mem_id (),NULL ); +numa_mem_id (),NULL ,ALLOC_DEFAULT );#endif if (unlikely (!page)) { - page =__alloc_pages (gfp,0 ,numa_mem_id (),NULL ); + page =__alloc_pages (gfp,0 ,numa_mem_id (),NULL ,ALLOC_DEFAULT ); order =0 ; }
Signature · 1 line
2.54.0
PATCH v5 16/18 mm: remove the __GFP_NO_OBJ_EXT flag
In reply to Brendan JackmanFrom: "Vlastimil Babka (SUSE)" <[email protected]>
All users of the flag are converted to SLAB_ALLOC_NO_RECURSE or ALLOC_NO_CODETAG (from __GFP_NO_CODETAG which reused the NO_OBJ_EXT bit). Free up the flag bit.
--- include/linux/gfp_types.h | 7 ------- include/trace/events/mmflags.h | 10 +--------- tools/include/linux/gfp_types.h | 7 ------- 3 files changed, 1 insertion(+), 23 deletions(-)
include/linux/gfp_types.h +0 -7
@@ -55,7 +55,6 @@ enum {#ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, #endif - ___GFP_NO_OBJ_EXT_BIT, ___GFP_LAST_BIT };
@@ -96,7 +95,6 @@ enum {#else #define ___GFP_NOLOCKDEP 0#endif -#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT)/* * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -138,17 +136,12 @@ enum {* * %__GFP_ACCOUNT causes the allocation to be accounted to the active * cgroup context. -* -* %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. -* mark_obj_codetag_empty() should be called upon freeing for objects allocated -* with this flag to indicate that their NULL tags are expected and normal. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE)#define __GFP_WRITE ((__force gfp_t)___GFP_WRITE)#define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL)#define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE)#define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) -#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT)/** * DOC: Watermark modifiers
include/trace/events/mmflags.h +1 -9
@@ -54,18 +54,10 @@# define TRACE_GFP_FLAGS_LOCKDEP #endif -#ifdef CONFIG_SLAB_OBJ_EXT -# define TRACE_GFP_FLAGS_SLAB \ - TRACE_GFP_EM(NO_OBJ_EXT) -#else -# define TRACE_GFP_FLAGS_SLAB -#endif -#define TRACE_GFP_FLAGS \ TRACE_GFP_FLAGS_GENERAL \ TRACE_GFP_FLAGS_KASAN \ - TRACE_GFP_FLAGS_LOCKDEP \ - TRACE_GFP_FLAGS_SLAB + TRACE_GFP_FLAGS_LOCKDEP#undef TRACE_GFP_EM#define TRACE_GFP_EM (a) TRACE_DEFINE_ENUM(___GFP_##a##_BIT);
tools/include/linux/gfp_types.h +0 -7
@@ -55,7 +55,6 @@ enum {#ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, #endif - ___GFP_NO_OBJ_EXT_BIT, ___GFP_LAST_BIT };
@@ -96,7 +95,6 @@ enum {#else #define ___GFP_NOLOCKDEP 0#endif -#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT)/* * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -137,17 +135,12 @@ enum {* node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. -* -* %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. -* mark_obj_codetag_empty() should be called upon freeing for objects allocated -* with this flag to indicate that their NULL tags are expected and normal. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE)#define __GFP_WRITE ((__force gfp_t)___GFP_WRITE)#define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL)#define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE)#define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) -#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT)/** * DOC: Watermark modifiers
Signature · 1 line
2.54.0
Re: [PATCH v5 16/18] mm: remove the __GFP_NO_OBJ_EXT flag
Quoted 9 lines · Brendan Jackman
From: "Vlastimil Babka (SUSE)" <[email protected]>
All users of the flag are converted to
SLAB_ALLOC_NO_RECURSEorALLOC_NO_CODETAG(from__GFP_NO_CODETAGwhich reused theNO_OBJ_EXTbit). Free up the flag bit.Signed-off-by Vlastimil Babka (SUSE) <[email protected]>[Rebased onto __GFP_NO_CODETAG removal]Signed-off-by Brendan Jackman <[email protected]>
Quoted 107 lines
--- include/linux/gfp_types.h | 7 ------- include/trace/events/mmflags.h | 10 +--------- tools/include/linux/gfp_types.h | 7 ------- 3 files changed, 1 insertion(+), 23 deletions(-)include/linux/gfp_types.h +0
-7@@ -55,7 +55,6 @@ enum {#ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, #endif - ___GFP_NO_OBJ_EXT_BIT, ___GFP_LAST_BIT };@@ -96,7 +95,6 @@ enum {#else #define ___GFP_NOLOCKDEP 0#endif -#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT)/* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -138,17 +136,12 @@ enum {* * %__GFP_ACCOUNT causes the allocation to be accounted to the active * cgroup context. -* -* %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. -* mark_obj_codetag_empty() should be called upon freeing for objects allocated -* with this flag to indicate that their NULL tags are expected and normal. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE)#define __GFP_WRITE ((__force gfp_t)___GFP_WRITE)#define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL)#define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE)#define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) -#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT)/** * DOC: Watermark modifiers include/trace/events/mmflags.h +1
-9@@ -54,18 +54,10 @@# define TRACE_GFP_FLAGS_LOCKDEP #endif -#ifdef CONFIG_SLAB_OBJ_EXT -# define TRACE_GFP_FLAGS_SLAB \ - TRACE_GFP_EM(NO_OBJ_EXT) -#else -# define TRACE_GFP_FLAGS_SLAB -#endif -#define TRACE_GFP_FLAGS \ TRACE_GFP_FLAGS_GENERAL \ TRACE_GFP_FLAGS_KASAN \ - TRACE_GFP_FLAGS_LOCKDEP \ - TRACE_GFP_FLAGS_SLAB + TRACE_GFP_FLAGS_LOCKDEP#undef TRACE_GFP_EM#define TRACE_GFP_EM (a) TRACE_DEFINE_ENUM(___GFP_##a##_BIT);tools/include/linux/gfp_types.h +0
-7@@ -55,7 +55,6 @@ enum {#ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, #endif - ___GFP_NO_OBJ_EXT_BIT, ___GFP_LAST_BIT };@@ -96,7 +95,6 @@ enum {#else #define ___GFP_NOLOCKDEP 0#endif -#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT)/* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -137,17 +135,12 @@ enum {* node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. -* -* %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. -* mark_obj_codetag_empty() should be called upon freeing for objects allocated -* with this flag to indicate that their NULL tags are expected and normal. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE)#define __GFP_WRITE ((__force gfp_t)___GFP_WRITE)#define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL)#define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE)#define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) -#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT)/** * DOC: Watermark modifiers
PATCH v5 17/18 mm/page_alloc: drop alloc_flags arg from alloc_flags_cma()
In reply to Brendan JackmanTo align the style with other alloc_flags_*() functions, drop this additive argument and just have the callers do that themselves.
Note you can't always freely or alloc_flags like these callers do (because of the WMARK bits that encode an enum) but this is fine for ALLOC_CMA, just like it's fine for e.g. ALLOC_NON_BLOCK returned by alloc_flags_nonblocking() and or'd by its caller.
--- mm/page_alloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
mm/page_alloc.c +6 -7
@@ -3775,14 +3775,13 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) }/* Must be called after current_gfp_context() which can change gfp_mask */ -static inline unsigned int alloc_flags_cma (gfp_t gfp_mask, -unsigned int alloc_flags) +static inline unsigned int alloc_flags_cma (gfp_t gfp_mask) {#ifdef CONFIG_CMA if (gfp_migratetype (gfp_mask) ==MIGRATE_MOVABLE ) - alloc_flags |=ALLOC_CMA ; +return ALLOC_CMA ;#endif -return alloc_flags; +return ALLOC_DEFAULT ; }/*
@@ -4526,7 +4525,7 @@ alloc_flags_slowpath(gfp_t gfp_mask, unsigned int order) }else if (unlikely (rt_or_dl_task (current)) &&in_task ()) alloc_flags |=ALLOC_MIN_RESERVE ; - alloc_flags =alloc_flags_cma (gfp_mask, alloc_flags); + alloc_flags |=alloc_flags_cma (gfp_mask);if (defrag_mode) alloc_flags |=ALLOC_NOFRAGMENT ;
@@ -4837,7 +4836,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags =__gfp_pfmemalloc_flags (gfp_mask);if (reserve_flags) - alloc_flags =alloc_flags_cma (gfp_mask, reserve_flags) | + alloc_flags =alloc_flags_cma (gfp_mask) | reserve_flags | ac->alloc_flags | (alloc_flags &ALLOC_KSWAPD );/*
@@ -5070,7 +5069,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, should_fail_alloc_page(gfp_mask, order))return false; - *alloc_flags =alloc_flags_cma (gfp_mask, *alloc_flags); + *alloc_flags |=alloc_flags_cma (gfp_mask);/* Dirty zone balancing only done in the fast path */ ac->spread_dirty_pages = (gfp_mask & __GFP_WRITE);
Signature · 1 line
2.54.0
Re: [PATCH v5 17/18] mm/page_alloc: drop alloc_flags arg from alloc_flags_cma()
Quoted 8 lines · Brendan Jackman
To align the style with other alloc_flags_*() functions, drop this additive argument and just have the callers do that themselves.
Note you can't always freely or
alloc_flagslike these callers do (because of the WMARK bits that encode an enum) but this is fine forALLOC_CMA, just like it's fine for e.g.ALLOC_NON_BLOCKreturned byalloc_flags_nonblocking()and or'd by its caller.
Quoted 8 lines
Suggested-by Vlastimil Babka (SUSE) <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>--- mm/page_alloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
Nice cleanup.
Signature · 2 lines
Best Regards, Yan, Zi
PATCH v5 18/18 mm: factor out can_spin_trylock()
In reply to Brendan JackmanDeduplicate checks for whether the current context is safe for spin_trylock().
Does this function really belong in mm/internal.h or is it generic? Not sure. If someone ends up duplicating this logic elsewhere in the kernel, that would be a shame. But if it goes in some generic header, someone treats it as documentation about where it's guaranteed safe to spin_trylock(), and then it emerges that there are other subtle preconditions that didn't affect the mm usecase, that would be worse. So, just be conservative and keep it local.
--- mm/internal.h | 23 +++++++++++++++++++++++ mm/page_alloc.c | 17 +---------------- mm/slub.c | 10 +--------- 3 files changed, 25 insertions(+), 25 deletions(-)
mm/internal.h +23 -0
@@ -1716,4 +1716,27 @@ static inline void mm_prepare_for_swap_entries(struct mm_struct *mm) } } +static inline bool can_spin_trylock (void ) +{ +/* +* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is +* unsafe in NMI. If spin_trylock() is called from hard IRQ the current +* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will +* mark the task as the owner of another rt_spin_lock which will +* confuse PI logic, so return immediately if called from hard IRQ or +* NMI. +* +* Note, irqs_disabled() case is ok. spin_trylock() can be called +* from raw_spin_lock_irqsave region. +*/ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) +return false; + +/* On UP, spin_trylock() always succeeds even when it is locked */ +if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +return false; + +return true; +} +#endif /* __MM_INTERNAL_H */
mm/page_alloc.c +1 -16
@@ -5291,22 +5291,7 @@ static inline bool alloc_order_allowed(gfp_t gfp, unsigned int order,static inline bool alloc_nolock_allowed (void ) { -/* -* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is -* unsafe in NMI. If spin_trylock() is called from hard IRQ the current -* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will -* mark the task as the owner of another rt_spin_lock which will -* confuse PI logic, so return immediately if called from hard IRQ or -* NMI. -* -* Note, irqs_disabled() case is ok. This function can be called -* from raw_spin_lock_irqsave region. -*/ -if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) -return false; - -/* On UP, spin_trylock() always succeeds even when it is locked */ -if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +if (!can_spin_trylock ())return false;/* Bailout, since _deferred_grow_zone() needs to take a lock */
mm/slub.c +1 -9
@@ -5408,15 +5408,7 @@ static void *__kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_fif (unlikely (!size))return ZERO_SIZE_PTR ; -/* -* See the comment for the same check in -* alloc_frozen_pages_nolock_noprof() -*/ -if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) -return NULL ; - -/* On UP, spin_trylock() always succeeds even when it is locked */ -if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +if (!can_spin_trylock ())return NULL ; retry:
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 18/18] mm: factor out can_spin_trylock()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Unsafe trylock from NMI on UP systems due to missed deduplication in free_unref_page_commit() --
commit b53c6ec11ab7f82add433847a77e54a590090942
Author: Brendan Jackman <[email protected]>
mm: factor out can_spin_trylock()
Deduplicate checks for whether the current context is safe for spin_trylock().
Quoted 4 lines
mm/page_alloc.c +0
-0
[ ... ]
Quoted 2 lines
static inline bool
alloc_nolock_allowed(void)
{
[ ... ]
mm/page_alloc.c static inline bool alloc_order_allowed(gfp_t gfp, unsigned int order, · line 5295
Quoted 7 lines
mm/page_alloc.c +1
-5-if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) -return false; - -/* On UP, spin_trylock() always succeeds even when it is locked */ -if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +if (!can_spin_trylock ())return false;
[Severity: High]
This is a pre-existing issue, but could this deduplication also be applied to free_unref_page_commit() in mm/page_alloc.c?
It looks like free_unref_page_commit() still uses an open-coded check:
if (unlikely((fpi_flags & FPI_TRYLOCK) && IS_ENABLED(CONFIG_PREEMPT_RT)
&& (in_nmi() || in_hardirq()))) {
add_page_to_zone_llist(zone, page, order);
return;
}Since the open-coded check in free_unref_page_commit() doesn't check for UP systems, would an asynchronous page free executed from an NMI context on a uniprocessor system (e.g., via a BPF memory allocator) incorrectly proceed past this check?
On UP systems, spin_trylock_irqsave() only disables interrupts and unconditionally returns true, providing no mutual exclusion against the interrupted context. Could this allow both the NMI and the interrupted process to concurrently modify the zone list structures and corrupt memory?
Applying !can_spin_trylock() here seems like it would fulfill the intent of this patch and resolve the UP + NMI case.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
Re: [PATCH v5 18/18] mm: factor out can_spin_trylock()
In reply to Brendan JackmanQuoted 16 lines · Brendan Jackman
Deduplicate checks for whether the current context is safe for
spin_trylock().Does this function really belong in
mm/internal.hor is it generic? Not sure. If someone ends up duplicating this logic elsewhere in the kernel, that would be a shame. But if it goes in some generic header, someone treats it as documentation about where it's guaranteed safe tospin_trylock(), and then it emerges that there are other subtle preconditions that didn't affect the mm usecase, that would be worse. So, just be conservative and keep it local.Suggested-by Harry Yoo <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Reviewed-by Harry Yoo (Oracle) <[email protected]>Signed-off-by Brendan Jackman <[email protected]>
Quoted 6 lines
--- mm/internal.h | 23 +++++++++++++++++++++++ mm/page_alloc.c | 17 +---------------- mm/slub.c | 10 +--------- 3 files changed, 25 insertions(+), 25 deletions(-)
Signature · 2 lines
Best Regards, Yan, Zi
Vlastimil Babka (SUSE)
maintainerQuoted 1 line · Brendan Jackman
Based on mm-new.
Seems ready for mm-new to me. I suspect most further non-trivial page allocator work will conflict, so better have it in mm to be based upon.
I was worried about Johannes' defrag fixes [1] that were supposed to be stable bacported and thus would be better to add (the next revision) to mm-new first. But seems like if I apply this series on mm-new, and then [1] on top, nothing conflicts, so the ordering shouldn't matter.
[1] https://lore.kernel.org/all/[email protected]/
Quoted 150 lines
This depends on moving
alloc_tagtomm/:
https://lore.kernel.org/all/aj5QBtJcphPElczI@lucifer/Some tweaks and cleanups for page allocator entrypoint and flags. This is motivated by preparation for
__GFP_UNMAPPED[1] (which will probably becomeALLOC_UNMAPPEDin its next iteration), but all this is supposed to be an improvement to the codebase in its own right: unifying code paths, reducing API surface, and removing GFP flags.[1] https://lore.kernel.org/all/[email protected]/
This started with unifying __alloc_frozen_pages[_nolock]_noprof() and expanded from there.
Unifying the nolock allocator entrypoint with the normal allocator entrypoint means adding an
alloc_flagsargument to the later (only exposed withinmm/). This presents an opportunity to take advantage of that arg to remove some GFP flags, if we add thatalloc_flagsarg a bit more broadly to allocator entrypoints.To distinguish between mm-internal and "public" allocator entrypoints, it makes sense to use the __ prefix. There are already some public APIs with that prefix. For *alloc_pages*, just removing those variants seems like a nice cleanup anyway, so do that. For
get_free_pages, the "__" variant is the _only_ variant and it's very widely used, so it doesn't seem worthwhile to modify that. Therefore, scope this "__" change specifically to the *alloc_pages* API, which means we leave the *folio_alloc* API untouched too, even though that could probably be cleaned up if so desired.Tested:
- KVM, mm, and BPF selftests in a QEMU VM
-
kunit.pyonx86_64- For the
ALLOC_NO_CODETAGbits I just booted a VM and read/proc/allocinfo. I confirmed that if I removeALLOC_NO_CODETAG, the kernel crashes in early boot, so I was at least booting code that depends on this logic.I used Google's internal version of Antigravity (AI coding harness) to do the repetitive bits, those commits are marked with Assisted-by, the rest is manual.
Signed-off-by Brendan Jackman <[email protected]>---
Changes in v5:
- Just trivial non-functional fixes.
- Link to v4: https://patch.msgid.link/[email protected]Changes in v4:
- Fixed some (harmless) missing applications ofac->alloc_flags(local Sashiko)
- Fixed various build issues.
- Note that Sashiko pointed out a KMSAN build issue [0], I havefixed it but KMSAN builds are currently broken by objtool [1]. At least mm/kmsan/init.c compiles. [2] https://lore.kernel.org/all/[email protected]/ [3] https://lore.kernel.org/all/[email protected]/t/#u- Avoided setting
ALLOC_NOFRAGMENTunderALLOC_NOLOCK(Sashiko, Harry)
- Added patch to tweakalloc_flags_cma()interface (Vlastimil)
- More commit messages fixups (various)
- Added patch to createcan_spin_trylock()(Harry)
- Link to v3: https://patch.msgid.link/[email protected]Changes in v3:
- Createdmm/page_alloc.h
- FixedEXPORT_SYMBOL()issues
- Reworded commit messages per Sashiko's pointers
- Dropped rename ofalloc_flagsarg inprepare_alloc_pages()(Suren)
- Renamedgfp_to_alloc_flags_nonblocking()too after rebasing onto:
https://lore.kernel.org/all/[email protected]/
- Link to v2: https://patch.msgid.link/[email protected]Changes in v2:
- Fixed up whitespace in nolock unification patch
- IntroducedALLOC_DEFAULTto replace literal 0 foralloc_flags
- All other patches are new
- Link to v1: https://patch.msgid.link/[email protected]---
Brendan Jackman (17):
mm/page_alloc: renameALLOC_TRYLOCK->ALLOC_NOLOCKmm/page_alloc: some renames to clarifyalloc_flagsscopes
mm: name some args in a function declaration
mm: Split out internalpage_alloc.hmm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof()mm/page_alloc: relax GFP WARN in nolock allocs
mm: move some stuff tomm/page_alloc.hperf/x86/intel: Use higher-level allocator API
KVM: VMX: Use higher-level allocator API x86/virt: Use higher-level allocator API
sgi-xp: Use higher-level allocator API net/funeth: Switch to higher-level allocator API
mm: Remove__alloc_pages_node()
mm: Move__alloc_pages()tomm/page_alloc.h
mm: replace__GFP_NO_CODETAGwithALLOC_NO_CODETAGmm/page_alloc: dropalloc_flagsarg fromalloc_flags_cma()
mm: factor outcan_spin_trylock()Vlastimil Babka (SUSE) (1):
mm: remove the__GFP_NO_OBJ_EXTflagDocumentation/admin-guide/cgroup-v1/cpusets.rst | 2 +- Documentation/admin-guide/mm/transhuge.rst | 2 +- MAINTAINERS | 1 + arch/x86/events/intel/ds.c | 6 +- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/virt/hw.c | 2 +- drivers/misc/sgi-xp/xpc_uv.c | 5 +- drivers/net/ethernet/fungible/funeth/funeth_rx.c | 2 +- include/linux/alloc_tag.h | 4 +- include/linux/gfp.h | 54 +--- include/linux/gfp_types.h | 7 - include/linux/skbuff.h | 2 +- include/trace/events/mmflags.h | 10 +- mm/alloc_tag.c | 23 +- mm/compaction.c | 5 +- mm/hugetlb.c | 4 +- mm/internal.h | 275 ++------------------ mm/khugepaged.c | 1 + mm/kmsan/init.c | 2 +- mm/memory-failure.c | 1 + mm/memory_hotplug.c | 1 + mm/mempolicy.c | 11 +- mm/migrate.c | 1 + mm/mm_init.c | 1 + mm/page_alloc.c | 269 ++++++++++--------- mm/page_alloc.h | 312 +++++++++++++++++++++++ mm/page_frag_cache.c | 6 +- mm/page_isolation.c | 1 + mm/page_owner.c | 2 +- mm/page_reporting.c | 1 + mm/show_mem.c | 1 + mm/shuffle.c | 1 + mm/slub.c | 17 +- mm/swap.c | 1 + mm/vmscan.c | 1 + mm/vmstat.c | 1 + tools/include/linux/gfp_types.h | 7 - 37 files changed, 536 insertions(+), 508 deletions(-)---
base-commit:32af3ff0925368eff29b2fed62f154150eb5dc10
change-id: 20260617-alloc-trylock-14ad37dab337Best regards,
Signature · 1 line
Brendan Jackman <[email protected]>
Andrew Morton
maintainerQuoted 5 lines · On Fri, 3 Jul 2026 14:47:32 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…
Quoted 1 line · Brendan Jackman
Based on mm-new.
Seems ready for mm-new to me. I suspect most further non-trivial page allocator work will conflict, so better have it in mm to be based upon.
OK, thanks, added.
I also added some little notes to track Zi Yan's comments.
Sashiko had things to say, mainly pre-existing things.
https://sashiko.dev/#/patchset/[email protected]
the xpc_create_gru_mq_uv() issues look like facepalms.
Quoted 4 lines · Andrew Morton
Quoted 1 line · On Fri, 3 Jul 2026 14:47:32 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…
On 7/3/26 14:31, Brendan Jackman wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Based on mm-new.
Seems ready for mm-new to me. I suspect most further non-trivial page allocator work will conflict, so better have it in mm to be based upon.
OK, thanks, added.
I also added some little notes to track Zi Yan's comments.
Thank Andrew. In this instance, what's the best way to send updates, I guess now they're in mm-new it's best to just send them as fixups for you to squash? In reply to the thread?
Quoted 5 lines
Sashiko had things to say, mainly pre-existing things.
https://sashiko.dev/#/patchset/[email protected]the
xpc_create_gru_mq_uv()issues look like facepalms.
Oh actually I didn't see the point about the alloc_pages_nolock() comment before. Other than that, yeah all of those comments have been discussed during the reviews, or are brokenness in totally unrelated code.
There is this where we should probably make a decision (also a pre-existing issue but this time in mm/page_alloc.c):
Vlastimil Babka (SUSE)
maintainerQuoted 29 lines · Brendan Jackman
Quoted 4 lines · Andrew Morton
Quoted 1 line · On Fri, 3 Jul 2026 14:47:32 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…
On 7/3/26 14:31, Brendan Jackman wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Based on mm-new.
Seems ready for mm-new to me. I suspect most further non-trivial page allocator work will conflict, so better have it in mm to be based upon.
OK, thanks, added.
I also added some little notes to track Zi Yan's comments.
Thank Andrew. In this instance, what's the best way to send updates, I guess now they're in mm-new it's best to just send them as fixups for you to squash? In reply to the thread?
Quoted 5 lines
Sashiko had things to say, mainly pre-existing things.
https://sashiko.dev/#/patchset/[email protected]the
xpc_create_gru_mq_uv()issues look like facepalms.Oh actually I didn't see the point about the
alloc_pages_nolock()comment before. Other than that, yeah all of those comments have been discussed during the reviews, or are brokenness in totally unrelated code.
Quoted 4 lines
There is this where we should probably make a decision (also a pre-existing issue but this time in
mm/page_alloc.c):
I think we can fix that in a follow-up patch. Tag it with Fixes: and stable and see what happens. We shouldn't have to rearrange everything for small fixes first just if it's just about !SMP.