vmallloc and non-blocking GFPs
Subsystems (4)
Uladzislau Rezki (Sony)
maintainercover vmallloc and non-blocking GFPs
This is a small series tends to support non-blocking GFP flags such as GFP_ATOMIC or GFP_NOWAIT. This is a draft version and it should be improved or changed.
For example there are still hard-coded GFP flags in the:
kmsan_vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
if the kernel is build with KMSAN support. There are other parts which should be fixed. But i tested this series with a fresh non-block-alloc test together with CONFIG_DEBUG_ATOMIC_SLEEP=y to track sleep in atomic issues.
Based on:
VERSION = 6
PATCHLEVEL = 16
SUBLEVEL = 0
EXTRAVERSION = -rc1
Uladzislau Rezki (Sony) (7):
lib/test_vmalloc: Add non-block-alloc-test case mm/vmalloc: Support non-blocking GFP flags in alloc_vmap_area() mm/vmalloc: Avoid cond_resched() when blocking is not permitted mm/kasan, mm/vmalloc: Respect GFP flags in kasan_populate_vmalloc() mm/vmalloc: Defer freeing partly initialized vm_struct mm/vmalloc: Support non-blocking GFP flags in __vmalloc_area_node()
mm: Drop __GFP_DIRECT_RECLAIM flag if PF_MEMALLOC is set
include/linux/kasan.h | 6 +-- include/linux/sched/mm.h | 7 ++- lib/test_vmalloc.c | 27 ++++++++++++ mm/kasan/shadow.c | 22 +++++++--- mm/vmalloc.c | 93 +++++++++++++++++++++++++++++++++------- 5 files changed, 129 insertions(+), 26 deletions(-)
Signature · 1 line
2.39.5
Uladzislau Rezki (Sony)
maintainerRFC 1/7 lib/test_vmalloc: Add non-block-alloc-test case
--- lib/test_vmalloc.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
lib/test_vmalloc.c +27 -0
@@ -54,6 +54,7 @@ __param(int, run_test_mask, INT_MAX,"\t\tid: 256, name: kvfree_rcu_1_arg_vmalloc_test\n" "\t\tid: 512, name: kvfree_rcu_2_arg_vmalloc_test\n" "\t\tid: 1024, name: vm_map_ram_test\n" +"\t\tid: 2048, name: no_block_alloc_test\n" /* Add a new test case description here. */ );
@@ -283,6 +284,31 @@ static int fix_size_alloc_test(void)return 0 ; } +static DEFINE_SPINLOCK (no_block_alloc_lock); + +static int no_block_alloc_test (void ) +{ +void *ptr; +u8 rnd; +int i; + +for (i =0 ; i < test_loop_count; i++) { + rnd =get_random_u8 (); + +spin_lock (&no_block_alloc_lock); + ptr =__vmalloc (PAGE_SIZE , (rnd %2 ) ?GFP_NOWAIT :GFP_ATOMIC ); +spin_unlock (&no_block_alloc_lock); + +if (!ptr) +return -1 ; + + *((__u8 *)ptr) =0 ; +vfree (ptr); + } + +return 0 ; +} +static int pcpu_alloc_test (void ) {
@@ -410,6 +436,7 @@ static struct test_case_desc test_case_array[] = { { "kvfree_rcu_1_arg_vmalloc_test", kvfree_rcu_1_arg_vmalloc_test }, { "kvfree_rcu_2_arg_vmalloc_test", kvfree_rcu_2_arg_vmalloc_test }, { "vm_map_ram_test", vm_map_ram_test }, + { "no_block_alloc_test", no_block_alloc_test }, /* Add a new test case here. */ };
Signature · 1 line
2.39.5
RE: [External] [RFC 1/7] lib/test_vmalloc: Add non-block-alloc-test case
Hi Uladzislau,
Quoted 32 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Uladzislau Rezki (Sony)
Sent: Friday, July 4, 2025 11:26 PM
To: [email protected]; Andrew Morton <[email protected]>
Cc: Michal Hocko <[email protected]>; LKML <[email protected]>; Baoquan He <[email protected]>; Uladzislau Rezki <[email protected]>
Subject: [External] [RFC 1/7]lib/test_vmalloc: Add non-block-alloc-test caseSigned-off-by Uladzislau Rezki (Sony) <[email protected]>--- lib/test_vmalloc.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)lib/test_vmalloc.c index renamed from lib/test_vmalloc.c +0
-0
1b0b59549aaf..9e3429dfe176100644lib/test_vmalloc.c +3
-0@@ -54,6 +54,7 @@ __param(int, run_test_mask, INT_MAX,"\t\tid: 256, name: kvfree_rcu_1_arg_vmalloc_test\n" "\t\tid: 512, name: kvfree_rcu_2_arg_vmalloc_test\n" "\t\tid: 1024, name: vm_map_ram_test\n" +"\t\tid: 2048, name: no_block_alloc_test\n" /* Add a new test case description here. */ );@@ -283,6 +284,31 @@ static int fix_size_alloc_test(void)return 0 ; } +static DEFINE_SPINLOCK (no_block_alloc_lock); +
lib/test_vmalloc.c static int fix_size_alloc_test(void) · line 298
Quoted 10 lines
lib/test_vmalloc.c +10
-0+static int no_block_alloc_test (void ) +{ +void *ptr; +u8 rnd; +int i; + +for (i =0 ; i < test_loop_count; i++) { + rnd =get_random_u8 (); + +spin_lock (&no_block_alloc_lock);
Since there is no shared data to be protected, do we need this lock for serialization? Any concerns?
It spent 18 minutes for this test (256-core server):
# time modprobe test_vmalloc nr_threads=$(nproc) run_test_mask=0x800 real 18m6.099s user 0m0.002s sys 0m4.555s
Without the lock, it spent 41 seconds (Have run for 300+ tests without any failure: 256-core server):
# time modprobe test_vmalloc nr_threads=$(nproc) run_test_mask=0x800 real 0m41.367s user 0m0.003s sys 0m5.758s
Would it be better to run this test concurrently? That said, it can also verify the scalability problem when the number of CPUs grow.
Quoted 28 lines
lib/test_vmalloc.c +13
-0+ ptr =__vmalloc (PAGE_SIZE , (rnd %2 ) ?GFP_NOWAIT :GFP_ATOMIC ); +spin_unlock (&no_block_alloc_lock); + +if (!ptr) +return -1 ; + + *((__u8 *)ptr) =0 ; +vfree (ptr); + } + +return 0 ; +} +static int pcpu_alloc_test (void ){
lib/test_vmalloc.c +1
-1@@ -410,6 +436,7 @@ static struct test_case_desc test_case_array[] = { { "kvfree_rcu_1_arg_vmalloc_test", kvfree_rcu_1_arg_vmalloc_test }, { "kvfree_rcu_2_arg_vmalloc_test", kvfree_rcu_2_arg_vmalloc_test }, { "vm_map_ram_test", vm_map_ram_test }, + { "no_block_alloc_test", no_block_alloc_test }, /* Add a new test case here. */ }; --2.39.5
Uladzislau Rezki
maintainerRe: [External] [RFC 1/7] lib/test_vmalloc: Add non-block-alloc-test case
Hello, Adrian!
Quoted 52 lines
Quoted 33 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Uladzislau Rezki (Sony)
Sent: Friday, July 4, 2025 11:26 PM
To: [email protected]; Andrew Morton <[email protected]>
Cc: Michal Hocko <[email protected]>; LKML <[email protected]>; Baoquan He <[email protected]>; Uladzislau Rezki <[email protected]>
Subject: [External] [RFC 1/7]lib/test_vmalloc: Add non-block-alloc-test caseSigned-off-by Uladzislau Rezki (Sony) <[email protected]>--- lib/test_vmalloc.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)lib/test_vmalloc.c index renamed from lib/test_vmalloc.c +0
-0
1b0b59549aaf..9e3429dfe176100644lib/test_vmalloc.c +3
-0@@ -54,6 +54,7 @@ __param(int, run_test_mask, INT_MAX,"\t\tid: 256, name: kvfree_rcu_1_arg_vmalloc_test\n" "\t\tid: 512, name: kvfree_rcu_2_arg_vmalloc_test\n" "\t\tid: 1024, name: vm_map_ram_test\n" +"\t\tid: 2048, name: no_block_alloc_test\n" /* Add a new test case description here. */ );@@ -283,6 +284,31 @@ static int fix_size_alloc_test(void)return 0 ; } +static DEFINE_SPINLOCK (no_block_alloc_lock); +lib/test_vmalloc.c static int fix_size_alloc_test(void) · line 298
Quoted 10 lines
lib/test_vmalloc.c +10
-0+static int no_block_alloc_test (void ) +{ +void *ptr; +u8 rnd; +int i; + +for (i =0 ; i < test_loop_count; i++) { + rnd =get_random_u8 (); + +spin_lock (&no_block_alloc_lock);Since there is no shared data to be protected, do we need this lock for serialization? Any concerns?
It spent 18 minutes for this test (256-core server):
# time modprobe test_vmalloc nr_threads=$(nproc) run_test_mask=0x800 real 18m6.099s user 0m0.002s sys 0m4.555s
Quoted 8 lines
Without the lock, it spent 41 seconds (Have run for 300+ tests without any failure: 256-core server):
# time modprobe test_vmalloc nr_threads=$(nproc) run_test_mask=0x800 real 0m41.367s user 0m0.003s sys 0m5.758sWould it be better to run this test concurrently? That said, it can also verify the scalability problem when the number of CPUs grow.
It was added just to track the sleep-in-atomic issues. We do not need that spin-lock in fact. Instead we can just invoke preempt_disable/enable() to simulate the context which is not allowed to trigger any schedule(), i.e. sleeping.
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerRFC 2/7 mm/vmalloc: Support non-blocking GFP flags in alloc_vmap_area()
In reply to Uladzislau Rezki (Sony)alloc_vmap_area() currently assumes that sleeping is allowed during allocation. This is not true for callers which pass non-blocking GFP flags, such as GFP_ATOMIC or GFP_NOWAIT.
This patch adds logic to detect whether the given gfp_mask permits blocking. It avoids invoking might_sleep() or falling back to reclaim path if blocking is not allowed.
This makes alloc_vmap_area() safer for use in non-sleeping contexts, where previously it could hit unexpected sleeps, trigger warnings.
--- mm/vmalloc.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
mm/vmalloc.c +13 -3
@@ -2009,6 +2009,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,unsigned long freed;unsigned long addr;unsigned int vn_id; +bool allow_block;int purged =0 ;int ret;
@@ -2018,7 +2019,9 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,if (unlikely (!vmap_initialized))return ERR_PTR (-EBUSY ); -might_sleep (); + allow_block =gfpflags_allow_blocking (gfp_mask); +if (allow_block) +might_sleep ();/* * If a VA is obtained from a global heap(if it fails here)
@@ -2030,7 +2033,8 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,*/ va =node_alloc (size, align, vstart, vend, &addr, &vn_id);if (!va) { - gfp_mask = gfp_mask &GFP_RECLAIM_MASK ; +if (allow_block) + gfp_mask = gfp_mask &GFP_RECLAIM_MASK ; va =kmem_cache_alloc_node (vmap_area_cachep, gfp_mask, node);if (unlikely (!va))
@@ -2057,8 +2061,14 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,* If an allocation fails, the error value is * returned. Therefore trigger the overflow path. */ -if (IS_ERR_VALUE (addr)) +if (IS_ERR_VALUE (addr)) { +if (!allow_block) { +kmem_cache_free (vmap_area_cachep, va); +return ERR_PTR (-ENOMEM ); + } + goto overflow; + } va->va_start = addr; va->va_end = addr + size;
Signature · 1 line
2.39.5
Re: [RFC 2/7] mm/vmalloc: Support non-blocking GFP flags in alloc_vmap_area()
On Fri 04-07-25 17:25:32, Uladzislau Rezki wrote:
[...]
mm/vmalloc.c static struct vmap_area *alloc_vmap_area(unsigned long size, · line 2037
Quoted 7 lines
mm/vmalloc.c +2
-1@@ -2030,7 +2033,8 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,*/ va =node_alloc (size, align, vstart, vend, &addr, &vn_id);if (!va) { - gfp_mask = gfp_mask &GFP_RECLAIM_MASK ; +if (allow_block) + gfp_mask = gfp_mask &GFP_RECLAIM_MASK ;
I don't follow here and is this even correct?
mm/vmalloc.c static struct vmap_area *alloc_vmap_area(unsigned long size, · line 2067
Quoted 12 lines
mm/vmalloc.c +4
-1va =kmem_cache_alloc_node (vmap_area_cachep, gfp_mask, node);if (unlikely (!va))@@ -2057,8 +2061,14 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,* If an allocation fails, the error value is * returned. Therefore trigger the overflow path. */ -if (IS_ERR_VALUE (addr)) +if (IS_ERR_VALUE (addr)) { +if (!allow_block) { +kmem_cache_free (vmap_area_cachep, va); +return ERR_PTR (-ENOMEM );
I would suggest to add a comment for this. Something like
for blockable requests trigger the overflow paths because that relies on vmap_purge_lock mutex and blocking notifiers.
Quoted 9 lines
mm/vmalloc.c +3
-0+ } + goto overflow; + } va->va_start = addr; va->va_end = addr + size;Signature · 1 line
2.39.5
Signature · 2 lines
Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerRe: [RFC 2/7] mm/vmalloc: Support non-blocking GFP flags in alloc_vmap_area()
Quoted 12 lines · Michal Hocko
On Fri 04-07-25 17:25:32, Uladzislau Rezki wrote:
[...]mm/vmalloc.c static struct vmap_area *alloc_vmap_area(unsigned long size, · line 2037
Quoted 7 lines
mm/vmalloc.c +2
-1@@ -2030,7 +2033,8 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,*/ va =node_alloc (size, align, vstart, vend, &addr, &vn_id);if (!va) { - gfp_mask = gfp_mask &GFP_RECLAIM_MASK ; +if (allow_block) + gfp_mask = gfp_mask &GFP_RECLAIM_MASK ;I don't follow here and is this even correct?
Allow nested flags to follow a user request if there is a request to not block. For example if we apply GFP_RECLAIM_MASK to GFP_ATOMIC GFP_ATOMIC is converted to zero, thus to GFP_NOWAIT.
Quoted 15 lines
mm/vmalloc.c static struct vmap_area *alloc_vmap_area(unsigned long size, · line 2067
Quoted 12 lines
mm/vmalloc.c +4
-1va =kmem_cache_alloc_node (vmap_area_cachep, gfp_mask, node);if (unlikely (!va))@@ -2057,8 +2061,14 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,* If an allocation fails, the error value is * returned. Therefore trigger the overflow path. */ -if (IS_ERR_VALUE (addr)) +if (IS_ERR_VALUE (addr)) { +if (!allow_block) { +kmem_cache_free (vmap_area_cachep, va); +return ERR_PTR (-ENOMEM );I would suggest to add a comment for this. Something like
Quoted 3 lines
for blockable requests trigger the overflow paths because that relies on
vmap_purge_lockmutex and blocking notifiers.
Thanks, i can do it easily. Also, this is an RFC i think it should be split and improved. Maybe to move out some functionality into a separate function.
--
Uladzislau Rezki
Re: [RFC 2/7] mm/vmalloc: Support non-blocking GFP flags in alloc_vmap_area()
Quoted 13 lines · Uladzislau Rezki
Quoted 12 lines · Michal Hocko
On Fri 04-07-25 17:25:32, Uladzislau Rezki wrote:
[...]mm/vmalloc.c static struct vmap_area *alloc_vmap_area(unsigned long size, · line 2037
Quoted 7 lines
mm/vmalloc.c +2
-1@@ -2030,7 +2033,8 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,*/ va =node_alloc (size, align, vstart, vend, &addr, &vn_id);if (!va) { - gfp_mask = gfp_mask &GFP_RECLAIM_MASK ; +if (allow_block) + gfp_mask = gfp_mask &GFP_RECLAIM_MASK ;I don't follow here and is this even correct?
Quoted 3 lines
Allow nested flags to follow a user request if there is a request to not block. For example if we apply
GFP_RECLAIM_MASKtoGFP_ATOMICGFP_ATOMICis converted to zero, thus toGFP_NOWAIT.
I still do not follow. The aim of this code is to filter out all non-reclaim related flags. Why that should work differently for non-waiting allocations?
Btw. if you had GPP_ATOMIC the resulting mask will be still GFP_ATOMIC as both __GFP_HIGH|__GFP_KSWAPD_RECLAIM are part of GFP_RECLAIM_MASK.
Signature · 2 lines
Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerRe: [RFC 2/7] mm/vmalloc: Support non-blocking GFP flags in alloc_vmap_area()
Quoted 18 lines · Michal Hocko
Quoted 13 lines · Uladzislau Rezki
Quoted 12 lines · Michal Hocko
On Fri 04-07-25 17:25:32, Uladzislau Rezki wrote:
[...]mm/vmalloc.c static struct vmap_area *alloc_vmap_area(unsigned long size, · line 2037
Quoted 7 lines
mm/vmalloc.c +2
-1@@ -2030,7 +2033,8 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,*/ va =node_alloc (size, align, vstart, vend, &addr, &vn_id);if (!va) { - gfp_mask = gfp_mask &GFP_RECLAIM_MASK ; +if (allow_block) + gfp_mask = gfp_mask &GFP_RECLAIM_MASK ;I don't follow here and is this even correct?
Quoted 3 lines
Allow nested flags to follow a user request if there is a request to not block. For example if we apply
GFP_RECLAIM_MASKtoGFP_ATOMICGFP_ATOMICis converted to zero, thus toGFP_NOWAIT.
Quoted 6 lines
I still do not follow. The aim of this code is to filter out all non-reclaim related flags. Why that should work differently for non-waiting allocations?
Btw. if you hadGPP_ATOMICthe resulting mask will be stillGFP_ATOMICas both __GFP_HIGH|__GFP_KSWAPD_RECLAIM are part ofGFP_RECLAIM_MASK.
Right. I misread the GFP_RECLAIM_MASK, i thought that GFP_ATOMIC and GFP_NOWAIT are not part of it. They allow reclaim, but not direct, i.e. it is OK to wake-up a kswapd.
So, they should not work differently. Thank you for the comment!
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerRFC 3/7 mm/vmalloc: Avoid cond_resched() when blocking is not permitted
In reply to Uladzislau Rezki (Sony)The vm_area_alloc_pages() function uses cond_resched() to yield the CPU during potentially long-running loops. However, yielding should only be done if the given GFP flags allow blocking.
This patch avoids calling cond_resched() when the allocation context is non-blocking(GFP_ATOMIC, GFP_NOWAIT).
--- mm/vmalloc.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
mm/vmalloc.c +6 -2
@@ -3624,7 +3624,9 @@ vm_area_alloc_pages(gfp_t gfp, int nid, pages + nr_allocated); nr_allocated += nr; -cond_resched (); + +if (gfpflags_allow_blocking (gfp)) +cond_resched ();/* * If zero or pages were obtained partly,
@@ -3666,7 +3668,9 @@ vm_area_alloc_pages(gfp_t gfp, int nid,for (i =0 ; i < (1U << order); i++) pages[nr_allocated + i] = page + i; -cond_resched (); +if (gfpflags_allow_blocking (gfp)) +cond_resched (); + nr_allocated +=1U << order; }
Signature · 1 line
2.39.5
Re: [RFC 3/7] mm/vmalloc: Avoid cond_resched() when blocking is not permitted
Quoted 6 lines · Uladzislau Rezki
The
vm_area_alloc_pages()function usescond_resched()to yield the CPU during potentially long-running loops. However, yielding should only be done if the given GFP flags allow blocking.This patch avoids calling
cond_resched()when the allocation context is non-blocking(GFP_ATOMIC,GFP_NOWAIT).
Do we even need those cond_resched calls? Both of them are called shortly after memory allocator which already yields CPU when allowed.
Quoted 34 lines
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)mm/vmalloc.c +6
-2@@ -3624,7 +3624,9 @@ vm_area_alloc_pages(gfp_t gfp, int nid, pages + nr_allocated); nr_allocated += nr; -cond_resched (); + +if (gfpflags_allow_blocking (gfp)) +cond_resched ();/* * If zero or pages were obtained partly, @@ -3666,7 +3668,9 @@ vm_area_alloc_pages(gfp_t gfp, int nid,for (i =0 ; i < (1U << order); i++) pages[nr_allocated + i] = page + i; -cond_resched (); +if (gfpflags_allow_blocking (gfp)) +cond_resched (); + nr_allocated +=1U << order; }Signature · 1 line
2.39.5
Signature · 2 lines
Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerRe: [RFC 3/7] mm/vmalloc: Avoid cond_resched() when blocking is not permitted
Quoted 11 lines · Michal Hocko
Quoted 6 lines · Uladzislau Rezki
The
vm_area_alloc_pages()function usescond_resched()to yield the CPU during potentially long-running loops. However, yielding should only be done if the given GFP flags allow blocking.This patch avoids calling
cond_resched()when the allocation context is non-blocking(GFP_ATOMIC,GFP_NOWAIT).Do we even need those
cond_reschedcalls? Both of them are called shortly after memory allocator which already yields CPU when allowed.
I think it can be just dropped.
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerRFC 4/7 mm/kasan, mm/vmalloc: Respect GFP flags in kasan_populate_vmalloc()
In reply to Uladzislau Rezki (Sony)The function kasan_populate_vmalloc() internally allocates a page using a hardcoded GFP_KERNEL flag. This is not safe in contexts where non-blocking allocation flags are required, such as GFP_ATOMIC or GFP_NOWAIT, for example during atomic vmalloc paths.
This patch modifies kasan_populate_vmalloc() and its helpers to accept a gfp_mask argument to use it for a page allocation. It allows the caller to specify the correct allocation context.
Also, when non-blocking flags are used, memalloc_noreclaim_save/restore() is used around apply_to_page_range() to suppress potential reclaim behavior that may otherwise violate atomic constraints.
--- include/linux/kasan.h | 6 +++--- mm/kasan/shadow.c | 22 +++++++++++++++------- mm/vmalloc.c | 4 ++-- 3 files changed, 20 insertions(+), 12 deletions(-)
include/linux/kasan.h +3 -3
@@ -562,7 +562,7 @@ static inline void kasan_init_hw_tags(void) { }#if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS )void kasan_populate_early_vm_area_shadow (void *start,unsigned long size); -int kasan_populate_vmalloc (unsigned long addr,unsigned long size); +int kasan_populate_vmalloc (unsigned long addr,unsigned long size,gfp_t gfp_mask); void kasan_release_vmalloc(unsigned long start, unsigned long end, unsigned long free_region_start, unsigned long free_region_end,
@@ -574,7 +574,7 @@ static inline void kasan_populate_early_vm_area_shadow(void *start, unsigned long size) { }static inline int kasan_populate_vmalloc (unsigned long start, -unsigned long size) +unsigned long size,gfp_t gfp_mask) {return 0 ; }
@@ -610,7 +610,7 @@ static __always_inline void kasan_poison_vmalloc(const void *start,static inline void kasan_populate_early_vm_area_shadow (void *start,unsigned long size) { }static inline int kasan_populate_vmalloc (unsigned long start, -unsigned long size) +unsigned long size,gfp_t gfp_mask) {return 0 ; }
mm/kasan/shadow.c +15 -7
@@ -335,13 +335,13 @@ static void ___free_pages_bulk(struct page **pages, int nr_pages) } } -static int ___alloc_pages_bulk (struct page **pages,int nr_pages) +static int ___alloc_pages_bulk (struct page **pages,int nr_pages,gfp_t gfp_mask) {unsigned long nr_populated, nr_total = nr_pages;struct page **page_array = pages;while (nr_pages) { - nr_populated =alloc_pages_bulk (GFP_KERNEL , nr_pages, pages); + nr_populated =alloc_pages_bulk (gfp_mask, nr_pages, pages);if (!nr_populated) {___free_pages_bulk (page_array, nr_total - nr_pages);return -ENOMEM ;
@@ -353,25 +353,33 @@ static int ___alloc_pages_bulk(struct page **pages, int nr_pages)return 0 ; } -static int __kasan_populate_vmalloc (unsigned long start,unsigned long end) +static int __kasan_populate_vmalloc (unsigned long start,unsigned long end,gfp_t gfp_mask) {unsigned long nr_pages, nr_total =PFN_UP (end - start); +bool noblock = !gfpflags_allow_blocking (gfp_mask);struct vmalloc_populate_data data; +unsigned int flags;int ret =0 ; - data.pages = (struct page **)__get_free_page (GFP_KERNEL | __GFP_ZERO); + data.pages = (struct page **)__get_free_page (gfp_mask | __GFP_ZERO);if (!data.pages)return -ENOMEM ;while (nr_total) { nr_pages =min (nr_total,PAGE_SIZE /sizeof (data.pages[0 ])); - ret =___alloc_pages_bulk (data.pages, nr_pages); + ret =___alloc_pages_bulk (data.pages, nr_pages, gfp_mask);if (ret)break ; data.start = start; +if (noblock) + flags =memalloc_noreclaim_save (); + ret =apply_to_page_range (&init_mm, start, nr_pages *PAGE_SIZE , kasan_populate_vmalloc_pte, &data); +if (noblock) +memalloc_noreclaim_restore (flags); +___free_pages_bulk (data.pages, nr_pages);if (ret)break ;
@@ -385,7 +393,7 @@ static int __kasan_populate_vmalloc(unsigned long start, unsigned long end)return ret; } -int kasan_populate_vmalloc (unsigned long addr,unsigned long size) +int kasan_populate_vmalloc (unsigned long addr,unsigned long size,gfp_t gfp_mask) {unsigned long shadow_start, shadow_end;int ret;
@@ -414,7 +422,7 @@ int kasan_populate_vmalloc(unsigned long addr, unsigned long size) shadow_start =PAGE_ALIGN_DOWN (shadow_start); shadow_end =PAGE_ALIGN (shadow_end); - ret =__kasan_populate_vmalloc (shadow_start, shadow_end); + ret =__kasan_populate_vmalloc (shadow_start, shadow_end, gfp_mask);if (ret)return ret;
mm/vmalloc.c +2 -2
@@ -2091,7 +2091,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,BUG_ON (va->va_start < vstart);BUG_ON (va->va_end > vend); - ret =kasan_populate_vmalloc (addr, size); + ret =kasan_populate_vmalloc (addr, size, gfp_mask);if (ret) {free_vmap_area (va);return ERR_PTR (ret);
@@ -4832,7 +4832,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,/* populate the kasan shadow space */ for (area =0 ; area < nr_vms; area++) { -if (kasan_populate_vmalloc (vas[area]->va_start, sizes[area])) +if (kasan_populate_vmalloc (vas[area]->va_start, sizes[area],GFP_KERNEL )) goto err_free_shadow; }
Signature · 1 line
2.39.5
Re: [RFC 4/7] mm/kasan, mm/vmalloc: Respect GFP flags in kasan_populate_vmalloc()
On 07/04/25 at 05:25pm, Uladzislau Rezki (Sony) wrote:
......snip.......
Quoted 41 lines
mm/kasan/shadow.c +6
-5@@ -335,13 +335,13 @@ static void ___free_pages_bulk(struct page **pages, int nr_pages) } } -static int ___alloc_pages_bulk (struct page **pages,int nr_pages) +static int ___alloc_pages_bulk (struct page **pages,int nr_pages,gfp_t gfp_mask) {unsigned long nr_populated, nr_total = nr_pages;struct page **page_array = pages;while (nr_pages) { - nr_populated =alloc_pages_bulk (GFP_KERNEL , nr_pages, pages); + nr_populated =alloc_pages_bulk (gfp_mask, nr_pages, pages);if (!nr_populated) {___free_pages_bulk (page_array, nr_total - nr_pages);return -ENOMEM ;@@ -353,25 +353,33 @@ static int ___alloc_pages_bulk(struct page **pages, int nr_pages)return 0 ; } -static int __kasan_populate_vmalloc (unsigned long start,unsigned long end) +static int __kasan_populate_vmalloc (unsigned long start,unsigned long end,gfp_t gfp_mask) {unsigned long nr_pages, nr_total =PFN_UP (end - start); +bool noblock = !gfpflags_allow_blocking (gfp_mask);struct vmalloc_populate_data data; +unsigned int flags;int ret =0 ; - data.pages = (struct page **)__get_free_page (GFP_KERNEL | __GFP_ZERO); + data.pages = (struct page **)__get_free_page (gfp_mask | __GFP_ZERO);if (!data.pages)return -ENOMEM ;while (nr_total) { nr_pages =min (nr_total,PAGE_SIZE /sizeof (data.pages[0 ])); - ret =___alloc_pages_bulk (data.pages, nr_pages);
mm/kasan/shadow.c static int ___alloc_pages_bulk(struct page **pages, int nr_pages) · line 379
Quoted 10 lines
mm/kasan/shadow.c +4
-0+ ret =___alloc_pages_bulk (data.pages, nr_pages, gfp_mask);if (ret)break ; data.start = start; +if (noblock) + flags =memalloc_noreclaim_save (); + ret =apply_to_page_range (&init_mm, start, nr_pages *PAGE_SIZE , kasan_populate_vmalloc_pte, &data);
This series is a great enhancement, thanks.
When checking code, seems apply_to_page_range() will lead to page table allocation which uses GFP_PGTABLE_KERNEL. Not sure if we need to handle this either.
Quoted 6 lines
mm/kasan/shadow.c +3
-0+if (noblock) +memalloc_noreclaim_restore (flags); +___free_pages_bulk (data.pages, nr_pages);if (ret)break ;
...snip...
Re: [RFC 4/7] mm/kasan, mm/vmalloc: Respect GFP flags in kasan_populate_vmalloc()
Quoted 56 lines · Baoquan He
On 07/04/25 at 05:25pm, Uladzislau Rezki (Sony) wrote:
......snip.......Quoted 41 lines
mm/kasan/shadow.c +6
-5@@ -335,13 +335,13 @@ static void ___free_pages_bulk(struct page **pages, int nr_pages) } } -static int ___alloc_pages_bulk (struct page **pages,int nr_pages) +static int ___alloc_pages_bulk (struct page **pages,int nr_pages,gfp_t gfp_mask) {unsigned long nr_populated, nr_total = nr_pages;struct page **page_array = pages;while (nr_pages) { - nr_populated =alloc_pages_bulk (GFP_KERNEL , nr_pages, pages); + nr_populated =alloc_pages_bulk (gfp_mask, nr_pages, pages);if (!nr_populated) {___free_pages_bulk (page_array, nr_total - nr_pages);return -ENOMEM ;@@ -353,25 +353,33 @@ static int ___alloc_pages_bulk(struct page **pages, int nr_pages)return 0 ; } -static int __kasan_populate_vmalloc (unsigned long start,unsigned long end) +static int __kasan_populate_vmalloc (unsigned long start,unsigned long end,gfp_t gfp_mask) {unsigned long nr_pages, nr_total =PFN_UP (end - start); +bool noblock = !gfpflags_allow_blocking (gfp_mask);struct vmalloc_populate_data data; +unsigned int flags;int ret =0 ; - data.pages = (struct page **)__get_free_page (GFP_KERNEL | __GFP_ZERO); + data.pages = (struct page **)__get_free_page (gfp_mask | __GFP_ZERO);if (!data.pages)return -ENOMEM ;while (nr_total) { nr_pages =min (nr_total,PAGE_SIZE /sizeof (data.pages[0 ])); - ret =___alloc_pages_bulk (data.pages, nr_pages);mm/kasan/shadow.c static int ___alloc_pages_bulk(struct page **pages, int nr_pages) · line 379
Quoted 10 lines
mm/kasan/shadow.c +4
-0+ ret =___alloc_pages_bulk (data.pages, nr_pages, gfp_mask);if (ret)break ; data.start = start; +if (noblock) + flags =memalloc_noreclaim_save (); + ret =apply_to_page_range (&init_mm, start, nr_pages *PAGE_SIZE , kasan_populate_vmalloc_pte, &data);This series is a great enhancement, thanks.
Quoted 3 lines
When checking code, seems
apply_to_page_range()will lead to page table allocation which usesGFP_PGTABLE_KERNEL. Not sure if we need to handle this either.
I am fool, didn't see the obvious added scope between memalloc_noreclaim_save/srestore(). Please ignore this noise.
Quoted 9 lines
Quoted 7 lines
mm/kasan/shadow.c +3
-0+if (noblock) +memalloc_noreclaim_restore (flags); +___free_pages_bulk (data.pages, nr_pages);if (ret)break ;...snip...
Uladzislau Rezki
maintainerRe: [RFC 4/7] mm/kasan, mm/vmalloc: Respect GFP flags in kasan_populate_vmalloc()
Quoted 57 lines · Baoquan He
Quoted 56 lines · Baoquan He
On 07/04/25 at 05:25pm, Uladzislau Rezki (Sony) wrote:
......snip.......Quoted 41 lines
mm/kasan/shadow.c +6
-5@@ -335,13 +335,13 @@ static void ___free_pages_bulk(struct page **pages, int nr_pages) } } -static int ___alloc_pages_bulk (struct page **pages,int nr_pages) +static int ___alloc_pages_bulk (struct page **pages,int nr_pages,gfp_t gfp_mask) {unsigned long nr_populated, nr_total = nr_pages;struct page **page_array = pages;while (nr_pages) { - nr_populated =alloc_pages_bulk (GFP_KERNEL , nr_pages, pages); + nr_populated =alloc_pages_bulk (gfp_mask, nr_pages, pages);if (!nr_populated) {___free_pages_bulk (page_array, nr_total - nr_pages);return -ENOMEM ;@@ -353,25 +353,33 @@ static int ___alloc_pages_bulk(struct page **pages, int nr_pages)return 0 ; } -static int __kasan_populate_vmalloc (unsigned long start,unsigned long end) +static int __kasan_populate_vmalloc (unsigned long start,unsigned long end,gfp_t gfp_mask) {unsigned long nr_pages, nr_total =PFN_UP (end - start); +bool noblock = !gfpflags_allow_blocking (gfp_mask);struct vmalloc_populate_data data; +unsigned int flags;int ret =0 ; - data.pages = (struct page **)__get_free_page (GFP_KERNEL | __GFP_ZERO); + data.pages = (struct page **)__get_free_page (gfp_mask | __GFP_ZERO);if (!data.pages)return -ENOMEM ;while (nr_total) { nr_pages =min (nr_total,PAGE_SIZE /sizeof (data.pages[0 ])); - ret =___alloc_pages_bulk (data.pages, nr_pages);mm/kasan/shadow.c static int ___alloc_pages_bulk(struct page **pages, int nr_pages) · line 379
Quoted 10 lines
mm/kasan/shadow.c +4
-0+ ret =___alloc_pages_bulk (data.pages, nr_pages, gfp_mask);if (ret)break ; data.start = start; +if (noblock) + flags =memalloc_noreclaim_save (); + ret =apply_to_page_range (&init_mm, start, nr_pages *PAGE_SIZE , kasan_populate_vmalloc_pte, &data);This series is a great enhancement, thanks.
Quoted 7 lines
Quoted 3 lines
When checking code, seems
apply_to_page_range()will lead to page table allocation which usesGFP_PGTABLE_KERNEL. Not sure if we need to handle this either.I am fool, didn't see the obvious added scope between memalloc_noreclaim_save/srestore(). Please ignore this noise.
No worries :)
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerRFC 5/7 mm/vmalloc: Defer freeing partly initialized vm_struct
In reply to Uladzislau Rezki (Sony)__vmalloc_area_node() may call free_vmap_area() or vfree() on error paths, both of which can sleep. This becomes problematic if the function is invoked from an atomic context, such as when GFP_ATOMIC or GFP_NOWAIT is passed via gfp_mask.
To fix this, unify error paths and defer the cleanup of partly initialized vm_struct objects to a workqueue. This ensures that freeing happens in a process context and avoids invalid sleeps in atomic regions.
--- mm/vmalloc.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-)
mm/vmalloc.c +32 -3
@@ -3677,6 +3677,36 @@ vm_area_alloc_pages(gfp_t gfp, int nid,return nr_allocated; } +static LLIST_HEAD (pending_vm_area_cleanup); + +static void cleanup_vm_area_work (struct work_struct *work) +{ +struct llist_node *node, *next; +struct vm_struct *vm; + +llist_for_each_safe (node, next,llist_del_all (&pending_vm_area_cleanup)) { + vm = (void *) node - offsetof(struct vm_struct , next); + +if (!vm->nr_pages) +free_vm_area (vm); +else +vfree (vm->addr); + } +} + +static DECLARE_WORK(cleanup_vm_area, cleanup_vm_area_work); + +/* +* Helper for __vmalloc_area_node() to defer cleanup +* of partially initialized vm_struct in error paths. +*/ +static void +defer_vm_area_cleanup (struct vm_struct *area) +{ +if (llist_add ((struct llist_node *) &area->next, &pending_vm_area_cleanup)) +schedule_work (&cleanup_vm_area); +} +static void *__vmalloc_area_node (struct vm_struct *area,gfp_t gfp_mask,pgprot_t prot,unsigned int page_shift,int node)
@@ -3708,8 +3738,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,warn_alloc (gfp_mask,NULL ,"vmalloc error: size %lu, failed to allocated page array size %lu" , nr_small_pages *PAGE_SIZE , array_size); -free_vm_area (area); -return NULL ; + goto fail; }set_vm_area_page_order (area, page_shift -PAGE_SHIFT );
@@ -3786,7 +3815,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,return area->addr; fail: -vfree (area->addr); +defer_vm_area_cleanup (area);return NULL ; }
Signature · 1 line
2.39.5
Uladzislau Rezki (Sony)
maintainerRFC 6/7 mm/vmalloc: Support non-blocking GFP flags in __vmalloc_area_node()
In reply to Uladzislau Rezki (Sony)This patch makes __vmalloc_area_node() to correctly handle non-blocking allocation requests, such as GFP_ATOMIC and GFP_NOWAIT. Main changes:
- nested_gfp flag follows the same non-blocking constraints as the primary gfp_mask, ensuring consistency and avoiding sleeping allocations in atomic contexts.
- if blocking is not allowed, __GFP_NOFAIL is forcibly cleared and warning is issued if it was set, since __GFP_NOFAIL is incompatible with non-blocking contexts;
- Add a __GFP_HIGHMEM to gfp_mask only for blocking requests if there are no DMA constraints.
- in non-blocking mode we use memalloc_noreclaim_save/restore() to prevent reclaim related operations that may sleep while setting up page tables or mapping pages.
This is particularly important for page table allocations that internally use GFP_PGTABLE_KERNEL, which may sleep unless such scope restrictions are applied. For example:
<snip>
#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO)
__pte_alloc_kernel()
pte_alloc_one_kernel(&init_mm);
pagetable_alloc_noprof(GFP_PGTABLE_KERNEL & ~__GFP_HIGHMEM, 0);<snip>
--- mm/vmalloc.c | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-)
mm/vmalloc.c +25 -5
@@ -3711,7 +3711,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, pgprot_t prot, unsigned int page_shift, int node) { - const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; + gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO;bool nofail = gfp_mask & __GFP_NOFAIL;unsigned long addr = (unsigned long )area->addr;unsigned long size =get_vm_area_size (area);
@@ -3719,12 +3719,28 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,unsigned int nr_small_pages = size >>PAGE_SHIFT ;unsigned int page_order;unsigned int flags; +bool noblock;int ret; array_size = (unsigned long )nr_small_pages *sizeof (struct page *); + noblock = !gfpflags_allow_blocking (gfp_mask); -if (!(gfp_mask & (GFP_DMA |GFP_DMA32 ))) - gfp_mask |= __GFP_HIGHMEM; +if (noblock) { +/* __GFP_NOFAIL is incompatible with non-blocking contexts. */ +WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL); + gfp_mask &= ~__GFP_NOFAIL; + +/* +* In non-sleeping contexts, ensure nested allocations follow +* same non-blocking rules. +*/ + nested_gfp = gfp_mask | __GFP_ZERO; + nofail = false; + }else { +/* Allow highmem allocations if there are no DMA constraints. */ +if (!(gfp_mask & (GFP_DMA |GFP_DMA32 ))) + gfp_mask |= __GFP_HIGHMEM; + }/* Please note that the recursion is strictly bounded. */ if (array_size >PAGE_SIZE ) {
@@ -3788,7 +3804,9 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,* page tables allocations ignore external gfp mask, enforce it * by the scope API */ -if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) +if (noblock) + flags =memalloc_noreclaim_save (); +else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) flags =memalloc_nofs_save ();else if ((gfp_mask & (__GFP_FS | __GFP_IO)) ==0 ) flags =memalloc_noio_save ();
@@ -3800,7 +3818,9 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,schedule_timeout_uninterruptible (1 ); }while (nofail && (ret <0 )); -if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) +if (noblock) +memalloc_noreclaim_restore (flags); +else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO)memalloc_nofs_restore (flags);else if ((gfp_mask & (__GFP_FS | __GFP_IO)) ==0 )memalloc_noio_restore (flags);
Signature · 1 line
2.39.5
Re: [RFC 6/7] mm/vmalloc: Support non-blocking GFP flags in __vmalloc_area_node()
Quoted 22 lines · Uladzislau Rezki
This patch makes
__vmalloc_area_node()to correctly handle non-blocking allocation requests, such asGFP_ATOMICandGFP_NOWAIT. Main changes:-
nested_gfpflag follows the same non-blocking constraints as the primarygfp_mask, ensuring consistency and avoiding sleeping allocations in atomic contexts.- if blocking is not allowed,
__GFP_NOFAILis forcibly cleared and warning is issued if it was set, since__GFP_NOFAILis incompatible with non-blocking contexts;- Add a
__GFP_HIGHMEMtogfp_maskonly for blocking requests if there are no DMA constraints.- in non-blocking mode we use memalloc_noreclaim_save/restore() to prevent reclaim related operations that may sleep while setting up page tables or mapping pages.
This is particularly important for page table allocations that internally use
GFP_PGTABLE_KERNEL, which may sleep unless such scope restrictions are applied. For example:
Quoted 7 lines
<snip>
#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO) __pte_alloc_kernel() pte_alloc_one_kernel(&init_mm); pagetable_alloc_noprof(GFP_PGTABLE_KERNEL & ~__GFP_HIGHMEM, 0);<snip>
The changelog doesn't explain the actual implementation and that is really crucial here. You rely on memalloc_noreclaim_save (i.e. PF_MEMALLOC) to never trigger memory reclaim but you are not explaining how do you prevent from the biggest caveat of this interface. Let me quote the documentation
* Users of this scope have to be extremely careful to not deplete the reserves
* completely and implement a throttling mechanism which controls the
* consumption of the reserve based on the amount of freed memory. Usage of a
* pre-allocated pool (e.g. mempool) should be always considered before using
* this scope.
Unless I am missing something _any_ vmalloc(GFP_NOWAIT|GFP_ATOMIC) user would get practically unbound access to the whole available memory. This is not really acceptable.
Quoted 74 lines
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-)mm/vmalloc.c +25
-5@@ -3711,7 +3711,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, pgprot_t prot, unsigned int page_shift, int node) { - const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; + gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO;bool nofail = gfp_mask & __GFP_NOFAIL;unsigned long addr = (unsigned long )area->addr;unsigned long size =get_vm_area_size (area);@@ -3719,12 +3719,28 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,unsigned int nr_small_pages = size >>PAGE_SHIFT ;unsigned int page_order;unsigned int flags; +bool noblock;int ret; array_size = (unsigned long )nr_small_pages *sizeof (struct page *); + noblock = !gfpflags_allow_blocking (gfp_mask); -if (!(gfp_mask & (GFP_DMA |GFP_DMA32 ))) - gfp_mask |= __GFP_HIGHMEM; +if (noblock) { +/* __GFP_NOFAIL is incompatible with non-blocking contexts. */ +WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL); + gfp_mask &= ~__GFP_NOFAIL; + +/* +* In non-sleeping contexts, ensure nested allocations follow +* same non-blocking rules. +*/ + nested_gfp = gfp_mask | __GFP_ZERO; + nofail = false; + }else { +/* Allow highmem allocations if there are no DMA constraints. */ +if (!(gfp_mask & (GFP_DMA |GFP_DMA32 ))) + gfp_mask |= __GFP_HIGHMEM; + }/* Please note that the recursion is strictly bounded. */ if (array_size >PAGE_SIZE ) {@@ -3788,7 +3804,9 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,* page tables allocations ignore external gfp mask, enforce it * by the scope API */ -if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) +if (noblock) + flags =memalloc_noreclaim_save (); +else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) flags =memalloc_nofs_save ();else if ((gfp_mask & (__GFP_FS | __GFP_IO)) ==0 ) flags =memalloc_noio_save ();@@ -3800,7 +3818,9 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,schedule_timeout_uninterruptible (1 ); }while (nofail && (ret <0 )); -if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) +if (noblock) +memalloc_noreclaim_restore (flags); +else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO)memalloc_nofs_restore (flags);else if ((gfp_mask & (__GFP_FS | __GFP_IO)) ==0 )memalloc_noio_restore (flags);Signature · 1 line
2.39.5
Signature · 2 lines
Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerRe: [RFC 6/7] mm/vmalloc: Support non-blocking GFP flags in __vmalloc_area_node()
Quoted 31 lines · Michal Hocko
Quoted 22 lines · Uladzislau Rezki
This patch makes
__vmalloc_area_node()to correctly handle non-blocking allocation requests, such asGFP_ATOMICandGFP_NOWAIT. Main changes:-
nested_gfpflag follows the same non-blocking constraints as the primarygfp_mask, ensuring consistency and avoiding sleeping allocations in atomic contexts.- if blocking is not allowed,
__GFP_NOFAILis forcibly cleared and warning is issued if it was set, since__GFP_NOFAILis incompatible with non-blocking contexts;- Add a
__GFP_HIGHMEMtogfp_maskonly for blocking requests if there are no DMA constraints.- in non-blocking mode we use memalloc_noreclaim_save/restore() to prevent reclaim related operations that may sleep while setting up page tables or mapping pages.
This is particularly important for page table allocations that internally use
GFP_PGTABLE_KERNEL, which may sleep unless such scope restrictions are applied. For example:Quoted 7 lines
<snip>
#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO) __pte_alloc_kernel() pte_alloc_one_kernel(&init_mm); pagetable_alloc_noprof(GFP_PGTABLE_KERNEL & ~__GFP_HIGHMEM, 0);<snip>
Quoted 11 lines
The changelog doesn't explain the actual implementation and that is really crucial here. You rely on
memalloc_noreclaim_save(i.e.PF_MEMALLOC) to never trigger memory reclaim but you are not explaining how do you prevent from the biggest caveat of this interface. Let me quote the documentation
* Users of this scope have to be extremely careful to not deplete the reserves
* completely and implement a throttling mechanism which controls the
* consumption of the reserve based on the amount of freed memory. Usage of a
* pre-allocated pool (e.g. mempool) should be always considered before using
* this scope.
I am aware about that comment. I had same concern about this, but it looks like i/you may overshot here. Yes, we have access to memory resrves but this only for page-table manipulations, i.e. to allocate a page for 5-level page table structure. We have PGD, P4D, PUD, PMD and PTE which is the lowest level and which needs pages the most.
As i see we do not free pages at least on PTE level, it means that an address space is populated forward only and never shrink back. Most of the time you do not need to allocate, this mostly occurs initially after the boot.
Quoted 5 lines
Unless I am missing something _any_
vmalloc(GFP_NOWAIT|GFP_ATOMIC)user would get practically unbound access to the whole available memory. This is not really acceptable.
See above comment. If there is a big concern about this, i can add memalloc_noblock_save() memalloc_noblock_restore() pair to eliminate that concern. The context will be converted in a way that it drops __GFP_DIRECT_RECLAIM flag.
Thank you for your comments and input i appreciate it.
--
Uladzislau Rezki
Re: [RFC 6/7] mm/vmalloc: Support non-blocking GFP flags in __vmalloc_area_node()
Quoted 49 lines · Uladzislau Rezki
Quoted 31 lines · Michal Hocko
Quoted 22 lines · Uladzislau Rezki
This patch makes
__vmalloc_area_node()to correctly handle non-blocking allocation requests, such asGFP_ATOMICandGFP_NOWAIT. Main changes:-
nested_gfpflag follows the same non-blocking constraints as the primarygfp_mask, ensuring consistency and avoiding sleeping allocations in atomic contexts.- if blocking is not allowed,
__GFP_NOFAILis forcibly cleared and warning is issued if it was set, since__GFP_NOFAILis incompatible with non-blocking contexts;- Add a
__GFP_HIGHMEMtogfp_maskonly for blocking requests if there are no DMA constraints.- in non-blocking mode we use memalloc_noreclaim_save/restore() to prevent reclaim related operations that may sleep while setting up page tables or mapping pages.
This is particularly important for page table allocations that internally use
GFP_PGTABLE_KERNEL, which may sleep unless such scope restrictions are applied. For example:Quoted 7 lines
<snip>
#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO) __pte_alloc_kernel() pte_alloc_one_kernel(&init_mm); pagetable_alloc_noprof(GFP_PGTABLE_KERNEL & ~__GFP_HIGHMEM, 0);<snip>
Quoted 11 lines
The changelog doesn't explain the actual implementation and that is really crucial here. You rely on
memalloc_noreclaim_save(i.e.PF_MEMALLOC) to never trigger memory reclaim but you are not explaining how do you prevent from the biggest caveat of this interface. Let me quote the documentation
* Users of this scope have to be extremely careful to not deplete the reserves
* completely and implement a throttling mechanism which controls the
* consumption of the reserve based on the amount of freed memory. Usage of a
* pre-allocated pool (e.g. mempool) should be always considered before using
* this scope.I am aware about that comment. I had same concern about this, but it looks like i/you may overshot here. Yes, we have access to memory resrves but this only for page-table manipulations, i.e. to allocate a page for 5-level page table structure. We have PGD, P4D, PUD, PMD and PTE which is the lowest level and which needs pages the most.
Quoted 4 lines
As i see we do not free pages at least on PTE level, it means that an address space is populated forward only and never shrink back. Most of the time you do not need to allocate, this mostly occurs initially after the boot.
You are right, I have misread the patch. I thought this includes vm_area_alloc_pages as well but you are right this is only for page tables and that seems much more reasonable. Having that outlined in the changelog would have helped ;)
Signature · 2 lines
Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerRe: [RFC 6/7] mm/vmalloc: Support non-blocking GFP flags in __vmalloc_area_node()
Quoted 55 lines · Michal Hocko
Quoted 49 lines · Uladzislau Rezki
Quoted 31 lines · Michal Hocko
Quoted 22 lines · Uladzislau Rezki
This patch makes
__vmalloc_area_node()to correctly handle non-blocking allocation requests, such asGFP_ATOMICandGFP_NOWAIT. Main changes:-
nested_gfpflag follows the same non-blocking constraints as the primarygfp_mask, ensuring consistency and avoiding sleeping allocations in atomic contexts.- if blocking is not allowed,
__GFP_NOFAILis forcibly cleared and warning is issued if it was set, since__GFP_NOFAILis incompatible with non-blocking contexts;- Add a
__GFP_HIGHMEMtogfp_maskonly for blocking requests if there are no DMA constraints.- in non-blocking mode we use memalloc_noreclaim_save/restore() to prevent reclaim related operations that may sleep while setting up page tables or mapping pages.
This is particularly important for page table allocations that internally use
GFP_PGTABLE_KERNEL, which may sleep unless such scope restrictions are applied. For example:Quoted 7 lines
<snip>
#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO) __pte_alloc_kernel() pte_alloc_one_kernel(&init_mm); pagetable_alloc_noprof(GFP_PGTABLE_KERNEL & ~__GFP_HIGHMEM, 0);<snip>
Quoted 11 lines
The changelog doesn't explain the actual implementation and that is really crucial here. You rely on
memalloc_noreclaim_save(i.e.PF_MEMALLOC) to never trigger memory reclaim but you are not explaining how do you prevent from the biggest caveat of this interface. Let me quote the documentation
* Users of this scope have to be extremely careful to not deplete the reserves
* completely and implement a throttling mechanism which controls the
* consumption of the reserve based on the amount of freed memory. Usage of a
* pre-allocated pool (e.g. mempool) should be always considered before using
* this scope.I am aware about that comment. I had same concern about this, but it looks like i/you may overshot here. Yes, we have access to memory resrves but this only for page-table manipulations, i.e. to allocate a page for 5-level page table structure. We have PGD, P4D, PUD, PMD and PTE which is the lowest level and which needs pages the most.
Quoted 4 lines
As i see we do not free pages at least on PTE level, it means that an address space is populated forward only and never shrink back. Most of the time you do not need to allocate, this mostly occurs initially after the boot.
Quoted 5 lines
You are right, I have misread the patch. I thought this includes
vm_area_alloc_pagesas well but you are right this is only for page tables and that seems much more reasonable. Having that outlined in the changelog would have helped ;)
I will update the commit message in more detail in my next version.
Thank you for!
--
Uladzislau Rezki
Re: [RFC 6/7] mm/vmalloc: Support non-blocking GFP flags in __vmalloc_area_node()
In reply to Uladzislau Rezki (Sony)Quoted 54 lines · Uladzislau Rezki
This patch makes
__vmalloc_area_node()to correctly handle non-blocking allocation requests, such asGFP_ATOMICandGFP_NOWAIT. Main changes:-
nested_gfpflag follows the same non-blocking constraints as the primarygfp_mask, ensuring consistency and avoiding sleeping allocations in atomic contexts.- if blocking is not allowed,
__GFP_NOFAILis forcibly cleared and warning is issued if it was set, since__GFP_NOFAILis incompatible with non-blocking contexts;- Add a
__GFP_HIGHMEMtogfp_maskonly for blocking requests if there are no DMA constraints.- in non-blocking mode we use memalloc_noreclaim_save/restore() to prevent reclaim related operations that may sleep while setting up page tables or mapping pages.
This is particularly important for page table allocations that internally use
GFP_PGTABLE_KERNEL, which may sleep unless such scope restrictions are applied. For example:<snip>
#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO) __pte_alloc_kernel() pte_alloc_one_kernel(&init_mm); pagetable_alloc_noprof(GFP_PGTABLE_KERNEL & ~__GFP_HIGHMEM, 0);<snip>
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-)mm/vmalloc.c +2
-1@@ -3711,7 +3711,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, pgprot_t prot, unsigned int page_shift, int node) { - const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; + gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO;bool nofail = gfp_mask & __GFP_NOFAIL;unsigned long addr = (unsigned long )area->addr;unsigned long size =get_vm_area_size (area);@@ -3719,12 +3719,28 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,unsigned int nr_small_pages = size >>PAGE_SHIFT ;unsigned int page_order;unsigned int flags; +bool noblock;int ret;
mm/vmalloc.c static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, · line 3731
Quoted 10 lines
mm/vmalloc.c +5
-2array_size = (unsigned long )nr_small_pages *sizeof (struct page *); + noblock = !gfpflags_allow_blocking (gfp_mask); -if (!(gfp_mask & (GFP_DMA |GFP_DMA32 ))) - gfp_mask |= __GFP_HIGHMEM; +if (noblock) { +/* __GFP_NOFAIL is incompatible with non-blocking contexts. */ +WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL); + gfp_mask &= ~__GFP_NOFAIL;
Btw. we already ignore GFP_NOFAIL for atomic allocations and warn about that at the page allocator level (__alloc_pages_slowpath)
What we can do though is to add a pr_warn + dump_stack for request with size that would require (in the worst case) page tables allocation larger than a portion of min_free_kbytes (to scale with different memory sizes). That should be plenty for any reasonable non blocking vmalloc. We would have means to catch abusers in that way.
Signature · 2 lines
Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerRe: [RFC 6/7] mm/vmalloc: Support non-blocking GFP flags in __vmalloc_area_node()
Quoted 59 lines · Michal Hocko
Quoted 58 lines · Uladzislau Rezki
This patch makes
__vmalloc_area_node()to correctly handle non-blocking allocation requests, such asGFP_ATOMICandGFP_NOWAIT. Main changes:-
nested_gfpflag follows the same non-blocking constraints as the primarygfp_mask, ensuring consistency and avoiding sleeping allocations in atomic contexts.- if blocking is not allowed,
__GFP_NOFAILis forcibly cleared and warning is issued if it was set, since__GFP_NOFAILis incompatible with non-blocking contexts;- Add a
__GFP_HIGHMEMtogfp_maskonly for blocking requests if there are no DMA constraints.- in non-blocking mode we use memalloc_noreclaim_save/restore() to prevent reclaim related operations that may sleep while setting up page tables or mapping pages.
This is particularly important for page table allocations that internally use
GFP_PGTABLE_KERNEL, which may sleep unless such scope restrictions are applied. For example:<snip>
#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO) __pte_alloc_kernel() pte_alloc_one_kernel(&init_mm); pagetable_alloc_noprof(GFP_PGTABLE_KERNEL & ~__GFP_HIGHMEM, 0);<snip>
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-)mm/vmalloc.c +3
-1@@ -3711,7 +3711,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, pgprot_t prot, unsigned int page_shift, int node) { - const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; + gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO;bool nofail = gfp_mask & __GFP_NOFAIL;unsigned long addr = (unsigned long )area->addr;unsigned long size =get_vm_area_size (area);@@ -3719,12 +3719,28 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,unsigned int nr_small_pages = size >>PAGE_SHIFT ;unsigned int page_order;unsigned int flags; +bool noblock;int ret; array_size = (unsigned long )nr_small_pages *sizeof (struct page *); + noblock = !gfpflags_allow_blocking (gfp_mask);
Quoted 10 lines
mm/vmalloc.c static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, · line 3731
Quoted 6 lines
mm/vmalloc.c +4
-2-if (!(gfp_mask & (GFP_DMA |GFP_DMA32 ))) - gfp_mask |= __GFP_HIGHMEM; +if (noblock) { +/* __GFP_NOFAIL is incompatible with non-blocking contexts. */ +WARN_ON_ONCE (gfp_mask & __GFP_NOFAIL); + gfp_mask &= ~__GFP_NOFAIL;Btw. we already ignore
GFP_NOFAILfor atomic allocations and warn about that at the page allocator level (__alloc_pages_slowpath)
Thank you. I will comment this!
Quoted 7 lines
What we can do though is to add a
pr_warn+dump_stackfor request with size that would require (in the worst case) page tables allocation larger than a portion ofmin_free_kbytes(to scale with different memory sizes). That should be plenty for any reasonable non blocking vmalloc. We would have means to catch abusers in that way.
OK, i will add it. I assume you mean:
unsigned long pages_min = min_free_kbytes >> (PAGE_SHIFT - 10);
if (request_pages > pages_min)
dump();--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerRFC 7/7 mm: Drop __GFP_DIRECT_RECLAIM flag if PF_MEMALLOC is set
In reply to Uladzislau Rezki (Sony)The memory allocator already avoids reclaim when PF_MEMALLOC is set. Clear __GFP_DIRECT_RECLAIM explicitly to suppress might_alloc() warnings to make more correct behavior.
--- include/linux/sched/mm.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
include/linux/sched/mm.h +6 -1
@@ -246,12 +246,14 @@ static inline bool in_vfork(struct task_struct *tsk)* PF_MEMALLOC_NOIO implies GFP_NOIO * PF_MEMALLOC_NOFS implies GFP_NOFS * PF_MEMALLOC_PIN implies !GFP_MOVABLE +* PF_MEMALLOC implies !__GFP_DIRECT_RECLAIM */ static inline gfp_t current_gfp_context (gfp_t flags) {unsigned int pflags =READ_ONCE (current->flags); -if (unlikely (pflags & (PF_MEMALLOC_NOIO |PF_MEMALLOC_NOFS |PF_MEMALLOC_PIN ))) { +if (unlikely (pflags & (PF_MEMALLOC_NOIO |PF_MEMALLOC_NOFS | +PF_MEMALLOC_PIN |PF_MEMALLOC ))) {/* * NOIO implies both NOIO and NOFS and it is a weaker context * so always make sure it makes precedence
@@ -263,6 +265,9 @@ static inline gfp_t current_gfp_context(gfp_t flags)if (pflags &PF_MEMALLOC_PIN ) flags &= ~__GFP_MOVABLE; + +if (pflags &PF_MEMALLOC ) + flags &= ~__GFP_DIRECT_RECLAIM; }return flags; }
Signature · 1 line
2.39.5