memcg,slab: kmalloc_nolock() fixes
Subsystems (5)
Harry Yoo (Meta)
maintainerv3 cover memcg,slab: kmalloc_nolock() fixes
This patchset fixes three kmalloc_nolock() bugs.
1. obj_cgroup_put() takes spinlocks in the release path when it drops the last reference, but it can be called from any context.
2. A spinlock may be taken in the following path and may lead to deadlock:
kmalloc_nolock()
-> slab_post_alloc_hook()
-> memcg_slab_post_alloc_hook()
-> memcg_alloc_abort_single().
3. irq_work_sync() is called without synchronization for rcuwait (on PREEMPT_RT or some architectures), potentially causing a hang.
Bug 1 was reported by lockdep, and bugs 2 [1] and 3 [2] were reported by Sashiko.
This series fixes the bugs by:
- 1. Deferring obj_cgroup_release() to an irq work.
- 2. Making slab_free_hook() and memcg_alloc_abort_single() respect SLAB_*_NOLOCK
- 3. Serializing irq_work_sync() with a mutex
Patch 1, 4, 5 are bugfixes and others are refactoring and cleanup patches.
Changes in v3
=============
- Rebased onto the latest slab/for-next.
- New patches and changes to existing patches were written and verified with AI assistance and carefully reviewed by Harry Yoo.
- Patch 1: added Vlastimil's Reviewed-by (thanks!).
- Patch 2 (new): pass free_flags to slab_free_hook() instead of adding a third bool, as suggested by Vlastimil.
- Patch 3 (patch 2 in v2):
- Now handles SLAB_FREE_NOLOCK in free_flags, so callers no longer need the allow_spin local variables.
- Updated the subject accordingly.
- Addressed v2 review comments from Vlastimil (thanks!):
- dropped the memcg_alloc_abort_single() functional change from the changelog, as it does not change until patch 4.
- Patch 4 (patch 4 in v2):
- Try freeing the object to the percpu sheaves first in both contexts, as suggested by Alexei and Vlastimil. Dropped Vlastimil's Reviewed-by accordingly.
- memcg_alloc_abort_single() now takes alloc_flags instead of a bool allow_spin, and converts them with a new to_free_flags() helper.
- Patch 5 (patch 5 in v2): added Vlastimil's Acked-by.
- Patch 6 (new): Make free_to_pcs() call can_free_to_pcs() and take free_flags instead of allow_spin.
- Patch 7 (patch 3 in v2): moved to the end of the series, after the fixes. Reworded the comment.
- Link to v2: https://lore.kernel.org/[email protected]
[1] https://sashiko.dev/#/patchset/20260610-slab_alloc_flags-v2-0-7190909db118%40kernel.org?part=9
[2] https://sashiko.dev/#/patchset/20260615-kfree_rcu_nolock-v3-0-70a54f3775bb%40kernel.org?part=5
---
Harry Yoo (Meta) (7):
mm/memcontrol: defer obj_cgroup_release() to irq_work
mm/slab: pass free_flags to slab_free_hook()
mm/slab: handle SLAB_FREE_NOLOCK in slab_free_hook() instead of open coding mm/slab: fix a deadlock in memcg_alloc_abort_single()
mm/slab: serialize deferred_work_barrier()
mm/slab: refactor free_to_pcs()
mm/slab: call debug_locks and KCSAN checks in kfree_nolock()
include/linux/memcontrol.h | 2 + mm/memcontrol.c | 27 ++++- mm/slab.h | 14 ++- mm/slub.c | 247 ++++++++++++++++++++++++++------------------- 4 files changed, 180 insertions(+), 110 deletions(-)
---
base-commit: a7cdd0af963a420fc0e6f20b7c8bc8d4784098db
change-id: 20260624-kmalloc-nolock-fixes-c97675328773
Best regards,
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo (Meta)
maintainerPATCH v3 1/7 mm/memcontrol: defer obj_cgroup_release() to irq_work
obj_cgroup_put() can be called from any context. __memcg_slab_free_hook() puts a per-object reference on every accounted free, and draining a per-cpu objcg stock slot puts the cached reference. When it drops the last reference, obj_cgroup_release() takes objcg_lock, frees percpu memory (pcpu_lock, via percpu_ref_exit()) and calls kfree_rcu(). Each of these can deadlock when the current context interrupted a holder of the respective lock.
Defer the release to irq_work. llist_add() and irq_work_queue() are NMI-safe, and the irq_work runs in a context where acquiring a spinlock is allowed.
Add a separate llist_node instead of reusing the list/rcu union because the list stays linked until the deferred release. struct obj_cgroup grows from 56 to 64 bytes, but both are served from kmalloc-64, so no memory is additionally consumed.
--- include/linux/memcontrol.h | 2 ++ mm/memcontrol.c | 27 ++++++++++++++++++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-)
include/linux/memcontrol.h +2 -0
@@ -185,6 +185,8 @@ struct obj_cgroup {struct list_head list;/* protected by objcg_lock */ struct rcu_head rcu; }; +/* Used to defer obj_cgroup_release() to irq_work */ +struct llist_node release_node;bool is_root; };
mm/memcontrol.c +24 -3
@@ -147,9 +147,8 @@ static void memcg_uncharge_kmem(struct mem_cgroup *memcg, unsigned int nr_pages)memcg_uncharge (memcg, nr_pages); } -static void obj_cgroup_release (struct percpu_ref *ref) +static void __obj_cgroup_release (struct obj_cgroup *objcg) { - struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt);unsigned int nr_bytes;unsigned int nr_pages;unsigned long flags;
@@ -190,10 +189,32 @@ static void obj_cgroup_release(struct percpu_ref *ref)list_del (&objcg->list);spin_unlock_irqrestore (&objcg_lock, flags); -percpu_ref_exit (ref); +percpu_ref_exit (&objcg->refcnt);kfree_rcu (objcg, rcu); } +static LLIST_HEAD (objcg_release_llist); + +static void objcg_release_workfn (struct irq_work *work) +{ +struct llist_node *head =llist_del_all (&objcg_release_llist); +struct obj_cgroup *objcg, *tmp; + +llist_for_each_entry_safe (objcg, tmp, head, release_node) +__obj_cgroup_release (objcg); +} + +static DEFINE_IRQ_WORK(objcg_release_work, objcg_release_workfn); + +static void obj_cgroup_release (struct percpu_ref *ref) +{ + struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt); + +/* The last put may come from a context that cannot take spinlocks */ +if (llist_add (&objcg->release_node, &objcg_release_llist)) +irq_work_queue (&objcg_release_work); +} +static struct obj_cgroup *obj_cgroup_alloc (void ) {struct obj_cgroup *objcg;
Signature · 1 line
2.54.0
Harry Yoo (Meta)
maintainerPATCH v3 2/7 mm/slab: pass free_flags to slab_free_hook()
In reply to Harry Yoo (Meta)slab_free_hook() takes two bools, init and after_rcu_delay, and a later patch needs to tell it whether spinning is allowed.
Instead of adding a third bool, make it take free_flags. Add SLAB_FREE_INIT and SLAB_FREE_AFTER_RCU_DELAY, and convert the callers.
No functional change intended.
--- mm/slab.h | 6 ++++-- mm/slub.c | 47 ++++++++++++++++++++++++++++++++++------------- 2 files changed, 38 insertions(+), 15 deletions(-)
mm/slab.h +4 -2
@@ -24,8 +24,10 @@#define SLAB_ALLOC_NO_RECURSE 0x04/* prevent kmalloc() recursion */ #define SLAB_ALLOC_NO_OBJ_EXT 0x08/* prevent obj_exts array allocation */ -#define SLAB_FREE_DEFAULT 0x00/* no flags */ -#define SLAB_FREE_NOLOCK 0x01/* spinning not allowed */ +#define SLAB_FREE_DEFAULT 0x00/* no flags */ +#define SLAB_FREE_NOLOCK 0x01/* spinning not allowed */ +#define SLAB_FREE_INIT 0x02/* zero the object on free */ +#define SLAB_FREE_AFTER_RCU_DELAY 0x04/* RCU grace period has elapsed */ static inline unsigned int to_alloc_flags (unsigned int free_flags) {
mm/slub.c +34 -13
136 lines not on this page · raw message
Signature · 1 line
2.54.0
Harry Yoo (Meta)
maintainerPATCH v3 3/7 mm/slab: handle SLAB_FREE_NOLOCK in slab_free_hook() instead of open coding
In reply to Harry Yoo (Meta)kfree_nolock() deliberately skips slab free hooks that should not be called in unknown context. However, there is one more place that needs the same thing: memcg_alloc_abort_single().
Instead of open coding slab_free_hook(), let it handle SLAB_FREE_NOLOCK. This is required to correctly handle allocation failures due to memcg charging.
kfree_nolock() now respects init_on_free which has been ignored. Besides that, no functional change intended.
--- mm/slub.c | 76 +++++++++++++++++++++++++++++++-------------------------------- 1 file changed, 37 insertions(+), 39 deletions(-)
mm/slub.c +37 -39
@@ -2684,11 +2684,14 @@ struct rcu_delayed_free {* - memcg_slab_post_alloc_hook() * - alloc_tagging_slab_alloc_hook() * +* slab_free_hook() assumes that memcg_slab_free_hook() and +* alloc_tagging_slab_free_hook() are already invoked. +* * Free hooks that must handle missing corresponding alloc hooks: * - kmemleak_free_recursive() * - kfence_free() * -* Free hooks that have no alloc hook counterpart, and thus safe to call: +* Free hooks that have no alloc hook counterpart: * - debug_check_no_locks_freed() * - debug_check_no_obj_freed() * - __kcsan_check_access()
@@ -2698,28 +2701,39 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags) {bool init = free_flags &SLAB_FREE_INIT ;bool after_rcu_delay = free_flags &SLAB_FREE_AFTER_RCU_DELAY ; +bool allow_spin =free_flags_allow_spinning (free_flags);/* Are the object contents still accessible? */ bool still_accessible = (s->flags &SLAB_TYPESAFE_BY_RCU ) && !after_rcu_delay; -kmemleak_free_recursive (x, s->flags);kmsan_slab_free (s, x); -debug_check_no_locks_freed (x, s->object_size); +/* Hooks that acquire locks or kmalloc_nolock() skips alloc hook counterpart */ +if (likely (allow_spin)) { +kmemleak_free_recursive (x, s->flags); +debug_check_no_locks_freed (x, s->object_size); -if (!(s->flags &SLAB_DEBUG_OBJECTS )) -debug_check_no_obj_freed (x, s->object_size); +if (!(s->flags &SLAB_DEBUG_OBJECTS )) +debug_check_no_obj_freed (x, s->object_size); -/* Use KCSAN to help debug racy use-after-free. */ -if (!still_accessible) -__kcsan_check_access (x, s->object_size, -KCSAN_ACCESS_WRITE |KCSAN_ACCESS_ASSERT ); +/* Use KCSAN to help debug racy use-after-free. */ +if (!still_accessible) +__kcsan_check_access (x, s->object_size, +KCSAN_ACCESS_WRITE |KCSAN_ACCESS_ASSERT ); -if (kfence_free (x)) -return false; +if (kfence_free (x)) +return false; + }/* * Give KASAN a chance to notice an invalid free operation before we * modify the object. +* +* If KASAN finds a bug it will do kasan_report_invalid_free() which +* will call raw_spin_lock_irqsave() which is technically unsafe from +* NMI, but take chance and report kernel bug. The sequence of +* kasan_report_invalid_free() -> raw_spin_lock_irqsave() -> NMI +* -> kfree_nolock() -> kasan_report_invalid_free() on the same CPU +* is double buggy and deserves to deadlock. */ if (kasan_slab_pre_free (s, x))return false;
@@ -2728,6 +2742,7 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)if (still_accessible) {struct rcu_delayed_free *delayed_free; +VM_WARN_ON_ONCE (!allow_spin); delayed_free =kmalloc_obj (*delayed_free,GFP_NOWAIT );if (delayed_free) {/*
@@ -2775,8 +2790,11 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)set_orig_size (s, x, orig_size); } -/* KASAN might put x into memory quarantine, delaying its reuse. */ -return !kasan_slab_free (s, x, init, still_accessible, false); +/* +* KASAN might put x into memory quarantine, delaying its reuse. +* Skip quarantine when spinning is not allowed as it uses a spinlock. +*/ +return !kasan_slab_free (s, x, init, still_accessible, !allow_spin); } static __fastpath_inline
@@ -6935,6 +6953,7 @@ void kfree_nolock(const void *object)struct slab *slab;struct kmem_cache *s;void *x = (void *)object; +unsigned int free_flags =SLAB_FREE_NOLOCK ;if (unlikely (ZERO_OR_NULL_PTR (object)))return ;
@@ -6949,33 +6968,12 @@ void kfree_nolock(const void *object)memcg_slab_free_hook (s, slab, &x,1 );alloc_tagging_slab_free_hook (s, slab, &x,1 ); -/* -* Unlike slab_free() do NOT call the following: -* kmemleak_free_recursive(x, s->flags); -* debug_check_no_locks_freed(x, s->object_size); -* debug_check_no_obj_freed(x, s->object_size); -* __kcsan_check_access(x, s->object_size, ..); -* kfence_free(x); -* since they take spinlocks or not safe from any context. -*/ -kmsan_slab_free (s, x); -/* -* If KASAN finds a kernel bug it will do kasan_report_invalid_free() -* which will call raw_spin_lock_irqsave() which is technically -* unsafe from NMI, but take chance and report kernel bug. -* The sequence of -* kasan_report_invalid_free() -> raw_spin_lock_irqsave() -> NMI -* -> kfree_nolock() -> kasan_report_invalid_free() on the same CPU -* is double buggy and deserves to deadlock. -*/ -if (kasan_slab_pre_free (s, x)) + +if (slab_want_init_on_free (s)) + free_flags |=SLAB_FREE_INIT ; + +if (unlikely (!slab_free_hook (s, x, free_flags)))return ; -/* -* memcg, kasan_slab_pre_free are done for 'x'. -* The only thing left is kasan_poison without quarantine, -* since kasan quarantine takes locks and not supported from NMI. -*/ -kasan_slab_free (s, x, false, false,/* skip quarantine */ true);if (likely (can_free_to_pcs (slab)) &&likely (free_to_pcs (s, x, false)))return ;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH for-next v3 3/7] mm/slab: handle SLAB_FREE_NOLOCK in slab_free_hook() instead of open coding
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/slab: kfree_nolock() leaves KASAN free_meta in init_on_free objects --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/slab: kfree_nolock() leaves KASAN free_meta in init_on_free objects
--
commit f5f03c7c446bcbe50fa191f0cf60accc8ad01ac9
Author: Harry Yoo (Meta) <[email protected]>
mm/slab: handle SLAB_FREE_NOLOCK in slab_free_hook() instead of open coding
This commit moves SLAB_FREE_NOLOCK handling into slab_free_hook() and replaces the open-coded free hooks in kfree_nolock() with a call to slab_free_hook(), enabling SLAB_FREE_INIT when slab_want_init_on_free() is true.
Quoted 4 lines
mm/slub.c +0
-0
[ ... ]
Quoted 12 lines
mm/slub.c +5
-2@@ -2775,8 +2790,11 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)set_orig_size (s, x, orig_size); } -/* KASAN might put x into memory quarantine, delaying its reuse. */ -return !kasan_slab_free (s, x, init, still_accessible, false); +/* +* KASAN might put x into memory quarantine, delaying its reuse. +* Skip quarantine when spinning is not allowed as it uses a spinlock. +*/ +return !kasan_slab_free (s, x, init, still_accessible, !allow_spin); }
[ ... ]
Quoted 4 lines
mm/slub.c +0
-0@@ -6949,33 +6968,12 @@ void kfree_nolock(const void *object)memcg_slab_free_hook (s, slab, &x,1 );alloc_tagging_slab_free_hook (s, slab, &x,1 );
[ ... ]
mm/slub.c void kfree_nolock(const void *object) · line 6976
Quoted 6 lines
mm/slub.c +5
-0+ +if (slab_want_init_on_free (s)) + free_flags |=SLAB_FREE_INIT ; + +if (unlikely (!slab_free_hook (s, x, free_flags)))return ;
[Severity: Low]
When CONFIG_KASAN_GENERIC and init_on_free are enabled, does kfree_nolock() leave non-zero KASAN free_meta in the object after slab_free_hook() zeroes it?
After slab_free_hook() zeroes the object for SLAB_FREE_INIT, it calls kasan_slab_free() with no_quarantine set to true (!allow_spin):
mm/kasan/common.c:__kasan_slab_free() {
...
poison_slab_object(cache, object, init);
if (no_quarantine)
return false;
...
}In Generic KASAN, poison_slab_object() -> kasan_save_free_info() writes struct kasan_free_meta at offset 0 of the object when free_meta_offset == 0. Normal frees clear this in-object metadata upon quarantine eviction in qlink_free(), but returning early for no_quarantine skips quarantine and never zeroes free_meta before the object is reused.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Harry Yoo (Meta)
maintainerPATCH v3 4/7 mm/slab: fix a deadlock in memcg_alloc_abort_single()
In reply to Harry Yoo (Meta)When kmalloc_nolock() successfully grabs a slab object but memcg aborts the allocation, the object is freed via __slab_free(). Calling __slab_free() in unknown context is not allowed and can lead to a deadlock.
Free the object via defer_free() when spinning is not allowed.
While at it, try freeing the object to the percpu sheaves first in both contexts, as kmem_cache_free_bulk() already does for bulk allocations. This also avoids queueing an irq_work for every failed charge when spinning is not allowed.
--- mm/slab.h | 8 ++++++++ mm/slub.c | 22 +++++++++++++++++----- 2 files changed, 25 insertions(+), 5 deletions(-)
mm/slab.h +8 -0
@@ -37,6 +37,14 @@ static inline unsigned int to_alloc_flags(unsigned int free_flags)return SLAB_ALLOC_DEFAULT ; } +static inline unsigned int to_free_flags (unsigned int alloc_flags) +{ +if (alloc_flags &SLAB_ALLOC_NOLOCK ) +return SLAB_FREE_NOLOCK ; +else +return SLAB_FREE_DEFAULT ; +} +static inline bool alloc_flags_allow_spinning (const unsigned int alloc_flags) {return !(alloc_flags &SLAB_ALLOC_NOLOCK );
mm/slub.c +17 -5
@@ -2531,7 +2531,8 @@ static inline void slab_obj_ext_has_codetag_init(void)#ifdef CONFIG_MEMCG -static void memcg_alloc_abort_single (struct kmem_cache *s,void *object); +static void memcg_alloc_abort_single (struct kmem_cache *s,void *object, +unsigned int alloc_flags); static __fastpath_inline bool memcg_slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags,
@@ -2549,7 +2550,7 @@ bool memcg_slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags,return true;if (likely (size ==1 )) { -memcg_alloc_abort_single (s, *p); +memcg_alloc_abort_single (s, *p, ac->alloc_flags); *p =NULL ; }else {kmem_cache_free_bulk (s, size, p);
@@ -6647,18 +6648,29 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object,#ifdef CONFIG_MEMCG /* Do not inline the rare memcg charging failed path into the allocation path */ static noinline -void memcg_alloc_abort_single(struct kmem_cache *s, void *object) +void memcg_alloc_abort_single(struct kmem_cache *s, void *object, +unsigned int alloc_flags) {struct slab *slab =virt_to_slab (object); -unsigned int free_flags =SLAB_FREE_DEFAULT ; +bool allow_spin =alloc_flags_allow_spinning (alloc_flags); +unsigned int free_flags =to_free_flags (alloc_flags);if (slab_want_init_on_free (s)) free_flags |=SLAB_FREE_INIT ;alloc_tagging_slab_free_hook (s, slab, &object,1 ); -if (likely (slab_free_hook (s, object, free_flags))) +if (unlikely (!slab_free_hook (s, object, free_flags))) +return ; + +if (likely (can_free_to_pcs (slab)) && +likely (free_to_pcs (s, object, allow_spin))) +return ; + +if (likely (allow_spin))__slab_free (s, slab, object, object,1 , _RET_IP_); +else +defer_free (s, object); }#endif
Signature · 1 line
2.54.0
Harry Yoo (Meta)
maintainerPATCH v3 5/7 mm/slab: serialize deferred_work_barrier()
In reply to Harry Yoo (Meta)irq_work_sync() uses rcuwait instead of busy waiting in two cases:
1. The kernel is using PREEMPT_RT and the irq work does not run in a hardirq context.
2. The architecture cannot send inter-processor interrupts to make busy waiting reasonably short.
However, rcuwait.h says:
Quoted 3 lines
The caller is responsible for locking around
rcuwait_wait_event(), and [prepare_to/finish]_rcuwait() such that writes to @task are properly serialized.
Since deferred_work_barrier() calls irq_work_sync() without any locks, it can potentially cause a hang as writes to @task are not serialized.
Fix this by serializing deferred_work_barrier() with a dedicated mutex. We cannot reuse slab_mutex because e.g.) some users of kvfree_rcu_barrier() are outside mm/.
--- mm/slub.c | 4 ++++ 1 file changed, 4 insertions(+)
mm/slub.c +4 -0
@@ -6617,8 +6617,12 @@ void defer_kfree_rcu(struct kvfree_rcu_head *head)/* Must be called before flush_rcu_sheaves_on_cache() */ void deferred_work_barrier (void ) { +static DEFINE_MUTEX (barrier_mutex);int cpu; +/* irq_work_sync() may use rcuwait that requires serialization */ +guard (mutex)(&barrier_mutex); +for_each_possible_cpu (cpu)irq_work_sync (&per_cpu_ptr (&deferred_percpu_work, cpu)->work); }
Signature · 1 line
2.54.0
Harry Yoo (Meta)
maintainerPATCH v3 6/7 mm/slab: refactor free_to_pcs()
In reply to Harry Yoo (Meta)Every caller of free_to_pcs() has to call can_free_to_pcs() before it. Refactor this by adding a slab parameter and calling can_free_to_pcs() from free_to_pcs(). To do this, move can_free_to_pcs() above free_to_pcs().
Note that can_free_to_pcs() cannot be folded into free_to_pcs() as free_to_pcs_bulk() still needs it.
While at it, let free_to_pcs() take free_flags instead of allow_spin.
No functional change intended.
--- mm/slub.c | 119 ++++++++++++++++++++++++++++++++------------------------------ 1 file changed, 61 insertions(+), 58 deletions(-)
mm/slub.c +61 -58
@@ -6148,15 +6148,71 @@ __pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,return pcs; } +static __always_inline bool can_free_to_pcs(struct slab *slab) +{ +int slab_node; +int numa_node; + +if (!IS_ENABLED (CONFIG_NUMA )) + goto check_pfmemalloc; + + slab_node =slab_nid (slab); + +#ifdef CONFIG_HAVE_MEMORYLESS_NODES +/* +* numa_mem_id() points to the closest node with memory so only allow +* objects from that node to the percpu sheaves +*/ + numa_node =numa_mem_id (); + +if (likely (slab_node == numa_node)) + goto check_pfmemalloc; +#else + +/* +* numa_mem_id() is only a wrapper to numa_node_id() which is where this +* cpu belongs to, but it might be a memoryless node anyway. We don't +* know what the closest node is. +*/ + numa_node =numa_node_id (); + +/* freed object is from this cpu's node, proceed */ +if (likely (slab_node == numa_node)) + goto check_pfmemalloc; + +/* +* Freed object isn't from this cpu's node, but that node is memoryless +* or only has ZONE_MOVABLE memory, which slab cannot allocate from. +* Proceed as it's better to cache remote objects than falling back to +* the slowpath for everything. The allocation side can never obtain +* a local object anyway, if none exist. We don't have numa_mem_id() to +* point to the closest node as we would on a proper memoryless node +* setup. +*/ +if (unlikely (!node_state (numa_node,N_NORMAL_MEMORY ))) + goto check_pfmemalloc; +#endif + +return false; + +check_pfmemalloc: +return likely (!slab_test_pfmemalloc (slab)); +} +/* * Free an object to the percpu sheaves. * The object is expected to have passed slab_free_hook() already. */ static __fastpath_inline -bool free_to_pcs(struct kmem_cache *s, void *object, bool allow_spin) +bool free_to_pcs(struct kmem_cache *s, struct slab *slab, void *object, +unsigned int free_flags) { +bool allow_spin =free_flags_allow_spinning (free_flags);struct slub_percpu_sheaves *pcs; +if (unlikely (!can_free_to_pcs (slab))) +return false; +if (!local_trylock (&s->cpu_sheaves->lock))return false;
@@ -6355,57 +6411,6 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)return false; } -static __always_inline bool can_free_to_pcs(struct slab *slab) -{ -int slab_node; -int numa_node; - -if (!IS_ENABLED (CONFIG_NUMA )) - goto check_pfmemalloc; - - slab_node =slab_nid (slab); - -#ifdef CONFIG_HAVE_MEMORYLESS_NODES -/* -* numa_mem_id() points to the closest node with memory so only allow -* objects from that node to the percpu sheaves -*/ - numa_node =numa_mem_id (); - -if (likely (slab_node == numa_node)) - goto check_pfmemalloc; -#else - -/* -* numa_mem_id() is only a wrapper to numa_node_id() which is where this -* cpu belongs to, but it might be a memoryless node anyway. We don't -* know what the closest node is. -*/ - numa_node =numa_node_id (); - -/* freed object is from this cpu's node, proceed */ -if (likely (slab_node == numa_node)) - goto check_pfmemalloc; - -/* -* Freed object isn't from this cpu's node, but that node is memoryless -* or only has ZONE_MOVABLE memory, which slab cannot allocate from. -* Proceed as it's better to cache remote objects than falling back to -* the slowpath for everything. The allocation side can never obtain -* a local object anyway, if none exist. We don't have numa_mem_id() to -* point to the closest node as we would on a proper memoryless node -* setup. -*/ -if (unlikely (!node_state (numa_node,N_NORMAL_MEMORY ))) - goto check_pfmemalloc; -#endif - -return false; - -check_pfmemalloc: -return likely (!slab_test_pfmemalloc (slab)); -} -/* * Try to free as many objects (already processed by free hooks) as possible to * a single per-cpu sheaf.
@@ -6642,7 +6647,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object,if (unlikely (!slab_free_hook (s, object, free_flags)))return ; -if (likely (can_free_to_pcs (slab)) &&likely (free_to_pcs (s, object, true))) +if (likely (free_to_pcs (s, slab, object, free_flags)))return ;__slab_free (s, slab, object, object,1 , addr);
@@ -6656,7 +6661,6 @@ void memcg_alloc_abort_single(struct kmem_cache *s, void *object, unsigned int alloc_flags) {struct slab *slab =virt_to_slab (object); -bool allow_spin =alloc_flags_allow_spinning (alloc_flags);unsigned int free_flags =to_free_flags (alloc_flags);if (slab_want_init_on_free (s))
@@ -6667,11 +6671,10 @@ void memcg_alloc_abort_single(struct kmem_cache *s, void *object,if (unlikely (!slab_free_hook (s, object, free_flags)))return ; -if (likely (can_free_to_pcs (slab)) && -likely (free_to_pcs (s, object, allow_spin))) +if (likely (free_to_pcs (s, slab, object, free_flags)))return ; -if (likely (allow_spin)) +if (likely (free_flags_allow_spinning (free_flags)))__slab_free (s, slab, object, object,1 , _RET_IP_);else defer_free (s, object);
@@ -6991,7 +6994,7 @@ void kfree_nolock(const void *object)if (unlikely (!slab_free_hook (s, x, free_flags)))return ; -if (likely (can_free_to_pcs (slab)) &&likely (free_to_pcs (s, x, false))) +if (likely (free_to_pcs (s, slab, x, free_flags)))return ;/*
Signature · 1 line
2.54.0
Harry Yoo (Meta)
maintainerPATCH v3 7/7 mm/slab: call debug_locks and KCSAN checks in kfree_nolock()
In reply to Harry Yoo (Meta)kfree_nolock() skips free hooks that may acquire spinlocks because it can be called from any context. debug_check_no_locks_freed() and __kcsan_check_access() are skipped as well.
However, they acquire a spinlock only when they report a bug. debug_check_no_locks_freed() disables interrupts and walks current->held_locks, and reaches printk only when a held lock is inside the freed object. KCSAN takes report_lock only when it observes a race.
Just like KASAN, take the risk of potential deadlock for bug reporting. Call debug_check_no_locks_freed() and __kcsan_check_access() regardless of allow_spin.
--- mm/slub.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-)
mm/slub.c +12 -13
@@ -2708,19 +2708,25 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)kmsan_slab_free (s, x); +/* +* debug_locks, KCSAN, KASAN may acquire a spinlock to report a bug, +* but otherwise cannot deadlock. Take the risk of deadlock for +* reporting. +*/ +debug_check_no_locks_freed (x, s->object_size); + +/* Use KCSAN to help debug racy use-after-free. */ +if (!still_accessible) +__kcsan_check_access (x, s->object_size, +KCSAN_ACCESS_WRITE |KCSAN_ACCESS_ASSERT ); +/* Hooks that acquire locks or kmalloc_nolock() skips alloc hook counterpart */ if (likely (allow_spin)) {kmemleak_free_recursive (x, s->flags); -debug_check_no_locks_freed (x, s->object_size);if (!(s->flags &SLAB_DEBUG_OBJECTS ))debug_check_no_obj_freed (x, s->object_size); -/* Use KCSAN to help debug racy use-after-free. */ -if (!still_accessible) -__kcsan_check_access (x, s->object_size, -KCSAN_ACCESS_WRITE |KCSAN_ACCESS_ASSERT ); -if (kfence_free (x))return false; }
@@ -2728,13 +2734,6 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)/* * Give KASAN a chance to notice an invalid free operation before we * modify the object. -* -* If KASAN finds a bug it will do kasan_report_invalid_free() which -* will call raw_spin_lock_irqsave() which is technically unsafe from -* NMI, but take chance and report kernel bug. The sequence of -* kasan_report_invalid_free() -> raw_spin_lock_irqsave() -> NMI -* -> kfree_nolock() -> kasan_report_invalid_free() on the same CPU -* is double buggy and deserves to deadlock. */ if (kasan_slab_pre_free (s, x))return false;
Signature · 1 line
2.54.0