Skip to contentSkip to conversation

memcg,slab: kmalloc_nolock() fixes

PATCH New Harry Yoo (Meta) 2 participants 9 messages Nov 15 – Nov 16

Subsystems (5)
lore
Share & tools
lore

Harry Yoo (Meta)

author maintainer
#0 lore raw

v3 cover memcg,slab: kmalloc_nolock() fixes

This patchset fixes three kmalloc_nolock() bugs.

1. obj_cgroup_put() takes spinlocks in the release path when it drops the last reference, but it can be called from any context.

2. A spinlock may be taken in the following path and may lead to deadlock:

kmalloc_nolock()
-> slab_post_alloc_hook()
-> memcg_slab_post_alloc_hook()
-> memcg_alloc_abort_single().

3. irq_work_sync() is called without synchronization for rcuwait (on PREEMPT_RT or some architectures), potentially causing a hang.

Bug 1 was reported by lockdep, and bugs 2 [1] and 3 [2] were reported by Sashiko.

This series fixes the bugs by:
- 1. Deferring obj_cgroup_release() to an irq work.
- 2. Making slab_free_hook() and memcg_alloc_abort_single() respect SLAB_*_NOLOCK
- 3. Serializing irq_work_sync() with a mutex

Patch 1, 4, 5 are bugfixes and others are refactoring and cleanup patches.

Changes in v3
=============

- Rebased onto the latest slab/for-next.

- New patches and changes to existing patches were written and verified with AI assistance and carefully reviewed by Harry Yoo.

- Patch 1: added Vlastimil's Reviewed-by (thanks!).

- Patch 2 (new): pass free_flags to slab_free_hook() instead of adding a third bool, as suggested by Vlastimil.

- Patch 3 (patch 2 in v2):
- Now handles SLAB_FREE_NOLOCK in free_flags, so callers no longer need the allow_spin local variables.
- Updated the subject accordingly.
- Addressed v2 review comments from Vlastimil (thanks!):
- dropped the memcg_alloc_abort_single() functional change from the changelog, as it does not change until patch 4.

- Patch 4 (patch 4 in v2):
- Try freeing the object to the percpu sheaves first in both contexts, as suggested by Alexei and Vlastimil. Dropped Vlastimil's Reviewed-by accordingly.
- memcg_alloc_abort_single() now takes alloc_flags instead of a bool allow_spin, and converts them with a new to_free_flags() helper.

- Patch 5 (patch 5 in v2): added Vlastimil's Acked-by.

- Patch 6 (new): Make free_to_pcs() call can_free_to_pcs() and take free_flags instead of allow_spin.

- Patch 7 (patch 3 in v2): moved to the end of the series, after the fixes. Reworded the comment.

- Link to v2: https://lore.kernel.org/[email protected]

[1] https://sashiko.dev/#/patchset/20260610-slab_alloc_flags-v2-0-7190909db118%40kernel.org?part=9
[2] https://sashiko.dev/#/patchset/20260615-kfree_rcu_nolock-v3-0-70a54f3775bb%40kernel.org?part=5

---
Harry Yoo (Meta) (7):
mm/memcontrol: defer obj_cgroup_release() to irq_work
mm/slab: pass free_flags to slab_free_hook()
mm/slab: handle SLAB_FREE_NOLOCK in slab_free_hook() instead of open coding mm/slab: fix a deadlock in memcg_alloc_abort_single()
mm/slab: serialize deferred_work_barrier()
mm/slab: refactor free_to_pcs()
mm/slab: call debug_locks and KCSAN checks in kfree_nolock()

 include/linux/memcontrol.h |   2 +
 mm/memcontrol.c            |  27 ++++-
 mm/slab.h                  |  14 ++-
 mm/slub.c                  | 247 ++++++++++++++++++++++++++-------------------
 4 files changed, 180 insertions(+), 110 deletions(-)

---
base-commit: a7cdd0af963a420fc0e6f20b7c8bc8d4784098db
change-id: 20260624-kmalloc-nolock-fixes-c97675328773

Best regards,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author maintainer
#1 lore raw

PATCH v3 1/7 mm/memcontrol: defer obj_cgroup_release() to irq_work

obj_cgroup_put() can be called from any context. __memcg_slab_free_hook() puts a per-object reference on every accounted free, and draining a per-cpu objcg stock slot puts the cached reference. When it drops the last reference, obj_cgroup_release() takes objcg_lock, frees percpu memory (pcpu_lock, via percpu_ref_exit()) and calls kfree_rcu(). Each of these can deadlock when the current context interrupted a holder of the respective lock.

Defer the release to irq_work. llist_add() and irq_work_queue() are NMI-safe, and the irq_work runs in a context where acquiring a spinlock is allowed.

Add a separate llist_node instead of reusing the list/rcu union because the list stays linked until the deferred release. struct obj_cgroup grows from 56 to 64 bytes, but both are served from kmalloc-64, so no memory is additionally consumed.

Fixes af92793e52c3 ("slab: Introduce kmalloc_nolock() and kfree_nolock().")
---
 include/linux/memcontrol.h |  2 ++
 mm/memcontrol.c            | 27 ++++++++++++++++++++++++---
 2 files changed, 26 insertions(+), 3 deletions(-)
include/linux/memcontrol.h +2 -0
index 69e1bcc67c58..4d5c634daef8 100644
@@ -185,6 +185,8 @@ struct obj_cgroup {
 		struct list_head list; /* protected by objcg_lock */
 		struct rcu_head rcu;
 	};
+	/* Used to defer obj_cgroup_release() to irq_work */
+	struct llist_node release_node;
 	bool is_root;
 };
 
mm/memcontrol.c +24 -3
index a05d9a8baa1b..c53147a88580 100644
@@ -147,9 +147,8 @@ static void memcg_uncharge_kmem(struct mem_cgroup *memcg, unsigned int nr_pages)
 		memcg_uncharge(memcg, nr_pages);
 }
 
-static void obj_cgroup_release(struct percpu_ref *ref)
+static void __obj_cgroup_release(struct obj_cgroup *objcg)
 {
-	struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt);
 	unsigned int nr_bytes;
 	unsigned int nr_pages;
 	unsigned long flags;
@@ -190,10 +189,32 @@ static void obj_cgroup_release(struct percpu_ref *ref)
 	list_del(&objcg->list);
 	spin_unlock_irqrestore(&objcg_lock, flags);
 
-	percpu_ref_exit(ref);
+	percpu_ref_exit(&objcg->refcnt);
 	kfree_rcu(objcg, rcu);
 }
 
+static LLIST_HEAD(objcg_release_llist);
+
+static void objcg_release_workfn(struct irq_work *work)
+{
+	struct llist_node *head = llist_del_all(&objcg_release_llist);
+	struct obj_cgroup *objcg, *tmp;
+
+	llist_for_each_entry_safe(objcg, tmp, head, release_node)
+		__obj_cgroup_release(objcg);
+}
+
+static DEFINE_IRQ_WORK(objcg_release_work, objcg_release_workfn);
+
+static void obj_cgroup_release(struct percpu_ref *ref)
+{
+	struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt);
+
+	/* The last put may come from a context that cannot take spinlocks */
+	if (llist_add(&objcg->release_node, &objcg_release_llist))
+		irq_work_queue(&objcg_release_work);
+}
+
 static struct obj_cgroup *obj_cgroup_alloc(void)
 {
 	struct obj_cgroup *objcg;
Signature · 1 line
2.54.0
author maintainer
#2 lore raw

PATCH v3 2/7 mm/slab: pass free_flags to slab_free_hook()

In reply to Harry Yoo (Meta)

slab_free_hook() takes two bools, init and after_rcu_delay, and a later patch needs to tell it whether spinning is allowed.

Instead of adding a third bool, make it take free_flags. Add SLAB_FREE_INIT and SLAB_FREE_AFTER_RCU_DELAY, and convert the callers.

No functional change intended.

---
 mm/slab.h |  6 ++++--
 mm/slub.c | 47 ++++++++++++++++++++++++++++++++++-------------
 2 files changed, 38 insertions(+), 15 deletions(-)
mm/slab.h +4 -2
index e94ac63d10dc..15b0d98ae1d4 100644
@@ -24,8 +24,10 @@
 #define SLAB_ALLOC_NO_RECURSE	0x04 /* prevent kmalloc() recursion */
 #define SLAB_ALLOC_NO_OBJ_EXT	0x08 /* prevent obj_exts array allocation */
 
-#define SLAB_FREE_DEFAULT	0x00 /* no flags */
-#define SLAB_FREE_NOLOCK	0x01 /* spinning not allowed */
+#define SLAB_FREE_DEFAULT		0x00 /* no flags */
+#define SLAB_FREE_NOLOCK		0x01 /* spinning not allowed */
+#define SLAB_FREE_INIT			0x02 /* zero the object on free */
+#define SLAB_FREE_AFTER_RCU_DELAY	0x04 /* RCU grace period has elapsed */
 
 static inline unsigned int to_alloc_flags(unsigned int free_flags)
 {
mm/slub.c +34 -13

136 lines not on this page · raw message

Signature · 1 line
2.54.0
author maintainer
#3 lore raw

PATCH v3 3/7 mm/slab: handle SLAB_FREE_NOLOCK in slab_free_hook() instead of open coding

In reply to Harry Yoo (Meta)

kfree_nolock() deliberately skips slab free hooks that should not be called in unknown context. However, there is one more place that needs the same thing: memcg_alloc_abort_single().

Instead of open coding slab_free_hook(), let it handle SLAB_FREE_NOLOCK. This is required to correctly handle allocation failures due to memcg charging.

kfree_nolock() now respects init_on_free which has been ignored. Besides that, no functional change intended.

Fixes af92793e52c3 ("slab: Introduce kmalloc_nolock() and kfree_nolock().")
---
 mm/slub.c | 76 +++++++++++++++++++++++++++++++--------------------------------
 1 file changed, 37 insertions(+), 39 deletions(-)
mm/slub.c +37 -39
index 761dc2affea2..7c0adc104988 100644
@@ -2684,11 +2684,14 @@ struct rcu_delayed_free {
  * - memcg_slab_post_alloc_hook()
  * - alloc_tagging_slab_alloc_hook()
  *
+ * slab_free_hook() assumes that memcg_slab_free_hook() and
+ * alloc_tagging_slab_free_hook() are already invoked.
+ *
  * Free hooks that must handle missing corresponding alloc hooks:
  * - kmemleak_free_recursive()
  * - kfence_free()
  *
- * Free hooks that have no alloc hook counterpart, and thus safe to call:
+ * Free hooks that have no alloc hook counterpart:
  * - debug_check_no_locks_freed()
  * - debug_check_no_obj_freed()
  * - __kcsan_check_access()
@@ -2698,28 +2701,39 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)
 {
 	bool init = free_flags & SLAB_FREE_INIT;
 	bool after_rcu_delay = free_flags & SLAB_FREE_AFTER_RCU_DELAY;
+	bool allow_spin = free_flags_allow_spinning(free_flags);
 	/* Are the object contents still accessible? */
 	bool still_accessible = (s->flags & SLAB_TYPESAFE_BY_RCU) && !after_rcu_delay;
 
-	kmemleak_free_recursive(x, s->flags);
 	kmsan_slab_free(s, x);
 
-	debug_check_no_locks_freed(x, s->object_size);
+	/* Hooks that acquire locks or kmalloc_nolock() skips alloc hook counterpart */
+	if (likely(allow_spin)) {
+		kmemleak_free_recursive(x, s->flags);
+		debug_check_no_locks_freed(x, s->object_size);
 
-	if (!(s->flags & SLAB_DEBUG_OBJECTS))
-		debug_check_no_obj_freed(x, s->object_size);
+		if (!(s->flags & SLAB_DEBUG_OBJECTS))
+			debug_check_no_obj_freed(x, s->object_size);
 
-	/* Use KCSAN to help debug racy use-after-free. */
-	if (!still_accessible)
-		__kcsan_check_access(x, s->object_size,
-				     KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT);
+		/* Use KCSAN to help debug racy use-after-free. */
+		if (!still_accessible)
+			__kcsan_check_access(x, s->object_size,
+					     KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT);
 
-	if (kfence_free(x))
-		return false;
+		if (kfence_free(x))
+			return false;
+	}
 
 	/*
 	 * Give KASAN a chance to notice an invalid free operation before we
 	 * modify the object.
+	 *
+	 * If KASAN finds a bug it will do kasan_report_invalid_free() which
+	 * will call raw_spin_lock_irqsave() which is technically unsafe from
+	 * NMI, but take chance and report kernel bug. The sequence of
+	 * kasan_report_invalid_free() -> raw_spin_lock_irqsave() -> NMI
+	 * -> kfree_nolock() -> kasan_report_invalid_free() on the same CPU
+	 *  is double buggy and deserves to deadlock.
 	 */
 	if (kasan_slab_pre_free(s, x))
 		return false;
@@ -2728,6 +2742,7 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)
 	if (still_accessible) {
 		struct rcu_delayed_free *delayed_free;
 
+		VM_WARN_ON_ONCE(!allow_spin);
 		delayed_free = kmalloc_obj(*delayed_free, GFP_NOWAIT);
 		if (delayed_free) {
 			/*
@@ -2775,8 +2790,11 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)
 		set_orig_size(s, x, orig_size);
 
 	}
-	/* KASAN might put x into memory quarantine, delaying its reuse. */
-	return !kasan_slab_free(s, x, init, still_accessible, false);
+	/*
+	 * KASAN might put x into memory quarantine, delaying its reuse.
+	 * Skip quarantine when spinning is not allowed as it uses a spinlock.
+	 */
+	return !kasan_slab_free(s, x, init, still_accessible, !allow_spin);
 }
 
 static __fastpath_inline
@@ -6935,6 +6953,7 @@ void kfree_nolock(const void *object)
 	struct slab *slab;
 	struct kmem_cache *s;
 	void *x = (void *)object;
+	unsigned int free_flags = SLAB_FREE_NOLOCK;
 
 	if (unlikely(ZERO_OR_NULL_PTR(object)))
 		return;
@@ -6949,33 +6968,12 @@ void kfree_nolock(const void *object)
 
 	memcg_slab_free_hook(s, slab, &x, 1);
 	alloc_tagging_slab_free_hook(s, slab, &x, 1);
-	/*
-	 * Unlike slab_free() do NOT call the following:
-	 * kmemleak_free_recursive(x, s->flags);
-	 * debug_check_no_locks_freed(x, s->object_size);
-	 * debug_check_no_obj_freed(x, s->object_size);
-	 * __kcsan_check_access(x, s->object_size, ..);
-	 * kfence_free(x);
-	 * since they take spinlocks or not safe from any context.
-	 */
-	kmsan_slab_free(s, x);
-	/*
-	 * If KASAN finds a kernel bug it will do kasan_report_invalid_free()
-	 * which will call raw_spin_lock_irqsave() which is technically
-	 * unsafe from NMI, but take chance and report kernel bug.
-	 * The sequence of
-	 * kasan_report_invalid_free() -> raw_spin_lock_irqsave() -> NMI
-	 *  -> kfree_nolock() -> kasan_report_invalid_free() on the same CPU
-	 * is double buggy and deserves to deadlock.
-	 */
-	if (kasan_slab_pre_free(s, x))
+
+	if (slab_want_init_on_free(s))
+		free_flags |= SLAB_FREE_INIT;
+
+	if (unlikely(!slab_free_hook(s, x, free_flags)))
 		return;
-	/*
-	 * memcg, kasan_slab_pre_free are done for 'x'.
-	 * The only thing left is kasan_poison without quarantine,
-	 * since kasan quarantine takes locks and not supported from NMI.
-	 */
-	kasan_slab_free(s, x, false, false, /* skip quarantine */true);
 
 	if (likely(can_free_to_pcs(slab)) && likely(free_to_pcs(s, x, false)))
 		return;
Signature · 1 line
2.54.0
#4 lore raw

Re: [PATCH for-next v3 3/7] mm/slab: handle SLAB_FREE_NOLOCK in slab_free_hook() instead of open coding

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/slab: kfree_nolock() leaves KASAN free_meta in init_on_free objects --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/slab: kfree_nolock() leaves KASAN free_meta in init_on_free objects
--

commit f5f03c7c446bcbe50fa191f0cf60accc8ad01ac9
Author: Harry Yoo (Meta) <[email protected]>

mm/slab: handle SLAB_FREE_NOLOCK in slab_free_hook() instead of open coding

This commit moves SLAB_FREE_NOLOCK handling into slab_free_hook() and replaces the open-coded free hooks in kfree_nolock() with a call to slab_free_hook(), enabling SLAB_FREE_INIT when slab_want_init_on_free() is true.

Quoted 4 lines
mm/slub.c +0 -0
index 761dc2affea27..7c0adc1049880 100644

[ ... ]

Quoted 12 lines
mm/slub.c +5 -2
@@ -2775,8 +2790,11 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)
 		set_orig_size(s, x, orig_size);
 
 	}
-	/* KASAN might put x into memory quarantine, delaying its reuse. */
-	return !kasan_slab_free(s, x, init, still_accessible, false);
+	/*
+	 * KASAN might put x into memory quarantine, delaying its reuse.
+	 * Skip quarantine when spinning is not allowed as it uses a spinlock.
+	 */
+	return !kasan_slab_free(s, x, init, still_accessible, !allow_spin);
 }

[ ... ]

Quoted 4 lines
mm/slub.c +0 -0
@@ -6949,33 +6968,12 @@ void kfree_nolock(const void *object)
 
 	memcg_slab_free_hook(s, slab, &x, 1);
 	alloc_tagging_slab_free_hook(s, slab, &x, 1);

[ ... ]

mm/slub.c void kfree_nolock(const void *object) · line 6976

Quoted 6 lines
mm/slub.c +5 -0
+
+	if (slab_want_init_on_free(s))
+		free_flags |= SLAB_FREE_INIT;
+
+	if (unlikely(!slab_free_hook(s, x, free_flags)))
 		return;

[Severity: Low]
When CONFIG_KASAN_GENERIC and init_on_free are enabled, does kfree_nolock() leave non-zero KASAN free_meta in the object after slab_free_hook() zeroes it?

After slab_free_hook() zeroes the object for SLAB_FREE_INIT, it calls kasan_slab_free() with no_quarantine set to true (!allow_spin):

mm/kasan/common.c:__kasan_slab_free() {
    ...
	poison_slab_object(cache, object, init);

	if (no_quarantine)
		return false;
    ...
}

In Generic KASAN, poison_slab_object() -> kasan_save_free_info() writes struct kasan_free_meta at offset 0 of the object when free_meta_offset == 0. Normal frees clear this in-object metadata upon quarantine eviction in qlink_free(), but returning early for no_quarantine skips quarantine and never zeroes free_meta before the object is reused.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author maintainer
#5 lore raw

PATCH v3 4/7 mm/slab: fix a deadlock in memcg_alloc_abort_single()

In reply to Harry Yoo (Meta)

When kmalloc_nolock() successfully grabs a slab object but memcg aborts the allocation, the object is freed via __slab_free(). Calling __slab_free() in unknown context is not allowed and can lead to a deadlock.

Free the object via defer_free() when spinning is not allowed.

While at it, try freeing the object to the percpu sheaves first in both contexts, as kmem_cache_free_bulk() already does for bulk allocations. This also avoids queueing an irq_work for every failed charge when spinning is not allowed.

Fixes af92793e52c3 ("slab: Introduce kmalloc_nolock() and kfree_nolock().")
---
 mm/slab.h |  8 ++++++++
 mm/slub.c | 22 +++++++++++++++++-----
 2 files changed, 25 insertions(+), 5 deletions(-)
mm/slab.h +8 -0
index 15b0d98ae1d4..fb80a72ad952 100644
@@ -37,6 +37,14 @@ static inline unsigned int to_alloc_flags(unsigned int free_flags)
 		return SLAB_ALLOC_DEFAULT;
 }
 
+static inline unsigned int to_free_flags(unsigned int alloc_flags)
+{
+	if (alloc_flags & SLAB_ALLOC_NOLOCK)
+		return SLAB_FREE_NOLOCK;
+	else
+		return SLAB_FREE_DEFAULT;
+}
+
 static inline bool alloc_flags_allow_spinning(const unsigned int alloc_flags)
 {
 	return !(alloc_flags & SLAB_ALLOC_NOLOCK);
mm/slub.c +17 -5
index 7c0adc104988..8252c0b681cc 100644
@@ -2531,7 +2531,8 @@ static inline void slab_obj_ext_has_codetag_init(void)
 
 #ifdef CONFIG_MEMCG
 
-static void memcg_alloc_abort_single(struct kmem_cache *s, void *object);
+static void memcg_alloc_abort_single(struct kmem_cache *s, void *object,
+				     unsigned int alloc_flags);
 
 static __fastpath_inline
 bool memcg_slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags,
@@ -2549,7 +2550,7 @@ bool memcg_slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags,
 		return true;
 
 	if (likely(size == 1)) {
-		memcg_alloc_abort_single(s, *p);
+		memcg_alloc_abort_single(s, *p, ac->alloc_flags);
 		*p = NULL;
 	} else {
 		kmem_cache_free_bulk(s, size, p);
@@ -6647,18 +6648,29 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object,
 #ifdef CONFIG_MEMCG
 /* Do not inline the rare memcg charging failed path into the allocation path */
 static noinline
-void memcg_alloc_abort_single(struct kmem_cache *s, void *object)
+void memcg_alloc_abort_single(struct kmem_cache *s, void *object,
+			      unsigned int alloc_flags)
 {
 	struct slab *slab = virt_to_slab(object);
-	unsigned int free_flags = SLAB_FREE_DEFAULT;
+	bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
+	unsigned int free_flags = to_free_flags(alloc_flags);
 
 	if (slab_want_init_on_free(s))
 		free_flags |= SLAB_FREE_INIT;
 
 	alloc_tagging_slab_free_hook(s, slab, &object, 1);
 
-	if (likely(slab_free_hook(s, object, free_flags)))
+	if (unlikely(!slab_free_hook(s, object, free_flags)))
+		return;
+
+	if (likely(can_free_to_pcs(slab)) &&
+	    likely(free_to_pcs(s, object, allow_spin)))
+		return;
+
+	if (likely(allow_spin))
 		__slab_free(s, slab, object, object, 1, _RET_IP_);
+	else
+		defer_free(s, object);
 }
 #endif
 
Signature · 1 line
2.54.0
author maintainer
#6 lore raw

PATCH v3 5/7 mm/slab: serialize deferred_work_barrier()

In reply to Harry Yoo (Meta)

irq_work_sync() uses rcuwait instead of busy waiting in two cases:

1. The kernel is using PREEMPT_RT and the irq work does not run in a hardirq context.

2. The architecture cannot send inter-processor interrupts to make busy waiting reasonably short.

However, rcuwait.h says:

Quoted 3 lines

The caller is responsible for locking around rcuwait_wait_event(), and [prepare_to/finish]_rcuwait() such that writes to @task are properly serialized.

Since deferred_work_barrier() calls irq_work_sync() without any locks, it can potentially cause a hang as writes to @task are not serialized.

Fix this by serializing deferred_work_barrier() with a dedicated mutex. We cannot reuse slab_mutex because e.g.) some users of kvfree_rcu_barrier() are outside mm/.

---
 mm/slub.c | 4 ++++
 1 file changed, 4 insertions(+)
mm/slub.c +4 -0
index 8252c0b681cc..aec9fb6e2235 100644
@@ -6617,8 +6617,12 @@ void defer_kfree_rcu(struct kvfree_rcu_head *head)
 /* Must be called before flush_rcu_sheaves_on_cache() */
 void deferred_work_barrier(void)
 {
+	static DEFINE_MUTEX(barrier_mutex);
 	int cpu;
 
+	/* irq_work_sync() may use rcuwait that requires serialization */
+	guard(mutex)(&barrier_mutex);
+
 	for_each_possible_cpu(cpu)
 		irq_work_sync(&per_cpu_ptr(&deferred_percpu_work, cpu)->work);
 }
Signature · 1 line
2.54.0
author maintainer
#7 lore raw

PATCH v3 6/7 mm/slab: refactor free_to_pcs()

In reply to Harry Yoo (Meta)

Every caller of free_to_pcs() has to call can_free_to_pcs() before it. Refactor this by adding a slab parameter and calling can_free_to_pcs() from free_to_pcs(). To do this, move can_free_to_pcs() above free_to_pcs().

Note that can_free_to_pcs() cannot be folded into free_to_pcs() as free_to_pcs_bulk() still needs it.

While at it, let free_to_pcs() take free_flags instead of allow_spin.

No functional change intended.

---
 mm/slub.c | 119 ++++++++++++++++++++++++++++++++------------------------------
 1 file changed, 61 insertions(+), 58 deletions(-)
mm/slub.c +61 -58
index aec9fb6e2235..fd3de3bee1de 100644
@@ -6148,15 +6148,71 @@ __pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	return pcs;
 }
 
+static __always_inline bool can_free_to_pcs(struct slab *slab)
+{
+	int slab_node;
+	int numa_node;
+
+	if (!IS_ENABLED(CONFIG_NUMA))
+		goto check_pfmemalloc;
+
+	slab_node = slab_nid(slab);
+
+#ifdef CONFIG_HAVE_MEMORYLESS_NODES
+	/*
+	 * numa_mem_id() points to the closest node with memory so only allow
+	 * objects from that node to the percpu sheaves
+	 */
+	numa_node = numa_mem_id();
+
+	if (likely(slab_node == numa_node))
+		goto check_pfmemalloc;
+#else
+
+	/*
+	 * numa_mem_id() is only a wrapper to numa_node_id() which is where this
+	 * cpu belongs to, but it might be a memoryless node anyway. We don't
+	 * know what the closest node is.
+	 */
+	numa_node = numa_node_id();
+
+	/* freed object is from this cpu's node, proceed */
+	if (likely(slab_node == numa_node))
+		goto check_pfmemalloc;
+
+	/*
+	 * Freed object isn't from this cpu's node, but that node is memoryless
+	 * or only has ZONE_MOVABLE memory, which slab cannot allocate from.
+	 * Proceed as it's better to cache remote objects than falling back to
+	 * the slowpath for everything. The allocation side can never obtain
+	 * a local object anyway, if none exist. We don't have numa_mem_id() to
+	 * point to the closest node as we would on a proper memoryless node
+	 * setup.
+	 */
+	if (unlikely(!node_state(numa_node, N_NORMAL_MEMORY)))
+		goto check_pfmemalloc;
+#endif
+
+	return false;
+
+check_pfmemalloc:
+	return likely(!slab_test_pfmemalloc(slab));
+}
+
 /*
  * Free an object to the percpu sheaves.
  * The object is expected to have passed slab_free_hook() already.
  */
 static __fastpath_inline
-bool free_to_pcs(struct kmem_cache *s, void *object, bool allow_spin)
+bool free_to_pcs(struct kmem_cache *s, struct slab *slab, void *object,
+		 unsigned int free_flags)
 {
+	bool allow_spin = free_flags_allow_spinning(free_flags);
 	struct slub_percpu_sheaves *pcs;
 
+	if (unlikely(!can_free_to_pcs(slab)))
+		return false;
+
 	if (!local_trylock(&s->cpu_sheaves->lock))
 		return false;
 
@@ -6355,57 +6411,6 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
 	return false;
 }
 
-static __always_inline bool can_free_to_pcs(struct slab *slab)
-{
-	int slab_node;
-	int numa_node;
-
-	if (!IS_ENABLED(CONFIG_NUMA))
-		goto check_pfmemalloc;
-
-	slab_node = slab_nid(slab);
-
-#ifdef CONFIG_HAVE_MEMORYLESS_NODES
-	/*
-	 * numa_mem_id() points to the closest node with memory so only allow
-	 * objects from that node to the percpu sheaves
-	 */
-	numa_node = numa_mem_id();
-
-	if (likely(slab_node == numa_node))
-		goto check_pfmemalloc;
-#else
-
-	/*
-	 * numa_mem_id() is only a wrapper to numa_node_id() which is where this
-	 * cpu belongs to, but it might be a memoryless node anyway. We don't
-	 * know what the closest node is.
-	 */
-	numa_node = numa_node_id();
-
-	/* freed object is from this cpu's node, proceed */
-	if (likely(slab_node == numa_node))
-		goto check_pfmemalloc;
-
-	/*
-	 * Freed object isn't from this cpu's node, but that node is memoryless
-	 * or only has ZONE_MOVABLE memory, which slab cannot allocate from.
-	 * Proceed as it's better to cache remote objects than falling back to
-	 * the slowpath for everything. The allocation side can never obtain
-	 * a local object anyway, if none exist. We don't have numa_mem_id() to
-	 * point to the closest node as we would on a proper memoryless node
-	 * setup.
-	 */
-	if (unlikely(!node_state(numa_node, N_NORMAL_MEMORY)))
-		goto check_pfmemalloc;
-#endif
-
-	return false;
-
-check_pfmemalloc:
-	return likely(!slab_test_pfmemalloc(slab));
-}
-
 /*
  * Try to free as many objects (already processed by free hooks) as possible to
  * a single per-cpu sheaf.
@@ -6642,7 +6647,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object,
 	if (unlikely(!slab_free_hook(s, object, free_flags)))
 		return;
 
-	if (likely(can_free_to_pcs(slab)) && likely(free_to_pcs(s, object, true)))
+	if (likely(free_to_pcs(s, slab, object, free_flags)))
 		return;
 
 	__slab_free(s, slab, object, object, 1, addr);
@@ -6656,7 +6661,6 @@ void memcg_alloc_abort_single(struct kmem_cache *s, void *object,
 			      unsigned int alloc_flags)
 {
 	struct slab *slab = virt_to_slab(object);
-	bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
 	unsigned int free_flags = to_free_flags(alloc_flags);
 
 	if (slab_want_init_on_free(s))
@@ -6667,11 +6671,10 @@ void memcg_alloc_abort_single(struct kmem_cache *s, void *object,
 	if (unlikely(!slab_free_hook(s, object, free_flags)))
 		return;
 
-	if (likely(can_free_to_pcs(slab)) &&
-	    likely(free_to_pcs(s, object, allow_spin)))
+	if (likely(free_to_pcs(s, slab, object, free_flags)))
 		return;
 
-	if (likely(allow_spin))
+	if (likely(free_flags_allow_spinning(free_flags)))
 		__slab_free(s, slab, object, object, 1, _RET_IP_);
 	else
 		defer_free(s, object);
@@ -6991,7 +6994,7 @@ void kfree_nolock(const void *object)
 	if (unlikely(!slab_free_hook(s, x, free_flags)))
 		return;
 
-	if (likely(can_free_to_pcs(slab)) && likely(free_to_pcs(s, x, false)))
+	if (likely(free_to_pcs(s, slab, x, free_flags)))
 		return;
 
 	/*
Signature · 1 line
2.54.0
author maintainer
#8 lore raw

PATCH v3 7/7 mm/slab: call debug_locks and KCSAN checks in kfree_nolock()

In reply to Harry Yoo (Meta)

kfree_nolock() skips free hooks that may acquire spinlocks because it can be called from any context. debug_check_no_locks_freed() and __kcsan_check_access() are skipped as well.

However, they acquire a spinlock only when they report a bug. debug_check_no_locks_freed() disables interrupts and walks current->held_locks, and reaches printk only when a held lock is inside the freed object. KCSAN takes report_lock only when it observes a race.

Just like KASAN, take the risk of potential deadlock for bug reporting. Call debug_check_no_locks_freed() and __kcsan_check_access() regardless of allow_spin.

---
 mm/slub.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)
mm/slub.c +12 -13
index fd3de3bee1de..fad19ec6af4b 100644
@@ -2708,19 +2708,25 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)
 
 	kmsan_slab_free(s, x);
 
+	/*
+	 * debug_locks, KCSAN, KASAN may acquire a spinlock to report a bug,
+	 * but otherwise cannot deadlock. Take the risk of deadlock for
+	 * reporting.
+	 */
+	debug_check_no_locks_freed(x, s->object_size);
+
+	/* Use KCSAN to help debug racy use-after-free. */
+	if (!still_accessible)
+		__kcsan_check_access(x, s->object_size,
+				     KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT);
+
 	/* Hooks that acquire locks or kmalloc_nolock() skips alloc hook counterpart */
 	if (likely(allow_spin)) {
 		kmemleak_free_recursive(x, s->flags);
-		debug_check_no_locks_freed(x, s->object_size);
 
 		if (!(s->flags & SLAB_DEBUG_OBJECTS))
 			debug_check_no_obj_freed(x, s->object_size);
 
-		/* Use KCSAN to help debug racy use-after-free. */
-		if (!still_accessible)
-			__kcsan_check_access(x, s->object_size,
-					     KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT);
-
 		if (kfence_free(x))
 			return false;
 	}
@@ -2728,13 +2734,6 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)
 	/*
 	 * Give KASAN a chance to notice an invalid free operation before we
 	 * modify the object.
-	 *
-	 * If KASAN finds a bug it will do kasan_report_invalid_free() which
-	 * will call raw_spin_lock_irqsave() which is technically unsafe from
-	 * NMI, but take chance and report kernel bug. The sequence of
-	 * kasan_report_invalid_free() -> raw_spin_lock_irqsave() -> NMI
-	 * -> kfree_nolock() -> kasan_report_invalid_free() on the same CPU
-	 *  is double buggy and deserves to deadlock.
 	 */
 	if (kasan_slab_pre_free(s, x))
 		return false;
Signature · 1 line
2.54.0