Skip to contentSkip to conversation

mm: preserve nolock context through memcg cleanup

PATCH In review Karl Mehltretter 2 participants 5 messages Oct 1 – Oct 9

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover mm: preserve nolock context through memcg cleanup

The no-lock allocation and free APIs can still enter regular locking through two memcg cleanup paths.

Patch 1 handles rollback after a post-allocation charge failure. If kmalloc_nolock() obtains an object but its memcg charge fails, memcg_slab_post_alloc_hook() frees the object through the regular SLUB path. Preserve the selected allocation mode by using kfree_nolock() when the allocation flags disallow spinning.

Patch 2 handles the final reference to a killed object cgroup. kfree_nolock() and free_pages_nolock() can drop that reference in the caller's context, after which obj_cgroup_release() takes regular locks. Queue the objcg on a lockless list and finish its teardown from normal irq_work. On PREEMPT_RT that work runs in irq_workd task context.

These paths were found while discussing the following RFC and during the subsequent investigation of no-lock allocation and free paths:

https://lore.kernel.org/r/[email protected]

Based on 40288c9206c17. Each patch's forced cleanup test passed on x86-64 release and lockdep PREEMPT_RT QEMU, arm64 and ARM32 SMP QEMU, and Pi 400 hardware. The objcg test accounted for all 256 queued and deferred releases. The series also passed non-RT builds and arena tests.

Karl Mehltretter (2):
mm/slub: preserve no-lock freeing after memcg charge failure mm/memcontrol: defer final objcg release from no-lock frees

 include/linux/memcontrol.h |  1 +
 mm/memcontrol.c            | 29 ++++++++++++++++++++++++++---
 mm/slub.c                  |  5 ++++-
 3 files changed, 31 insertions(+), 4 deletions(-)

base-commit: 40288c9206c17eb66a603262e06a58d300d0f279

Signature · 1 line
2.53.0
#1 lore raw

PATCH 1/2 mm/slub: preserve no-lock freeing after memcg charge failure

kmalloc_nolock() can obtain an object before its memcg post-allocation charge fails. For a single object, memcg_slab_post_alloc_hook() rolls the allocation back through memcg_alloc_abort_single(). That enters the regular SLUB free path, which can take a sleeping list_lock on PREEMPT_RT even though the caller selected a no-lock allocation.

Use kfree_nolock() when the allocation flags disallow spinning. This keeps the SLUB object rollback on the no-lock free path. The failed allocation continues to return NULL.

Fixes af92793e52c3 ("slab: Introduce kmalloc_nolock() and kfree_nolock().")
Assisted-by LLM
---
 mm/slub.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
mm/slub.c +4 -1
index 54ec125033571..a1f08338102e2 100644
@@ -2519,7 +2519,10 @@ bool memcg_slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags,
 		return true;
 
 	if (likely(size == 1)) {
-		memcg_alloc_abort_single(s, *p);
+		if (alloc_flags_allow_spinning(ac->alloc_flags))
+			memcg_alloc_abort_single(s, *p);
+		else
+			kfree_nolock(*p);
 		*p = NULL;
 	} else {
 		kmem_cache_free_bulk(s, size, p);
Signature · 1 line
2.53.0
maintainer
#2 lore raw

Re: [PATCH 1/2] mm/slub: preserve no-lock freeing after memcg charge failure

Quoted 10 lines · Karl Mehltretter

kmalloc_nolock() can obtain an object before its memcg post-allocation charge fails. For a single object, memcg_slab_post_alloc_hook() rolls the allocation back through memcg_alloc_abort_single(). That enters the regular SLUB free path, which can take a sleeping list_lock on PREEMPT_RT even though the caller selected a no-lock allocation.

Use kfree_nolock() when the allocation flags disallow spinning. This keeps the SLUB object rollback on the no-lock free path. The failed allocation continues to return NULL.

When this happens, the object is not charged by memcg.

The kernel should not invoke memcg_slab_free_hook() (called by kfree_nolock()) for a slab object that is not charged by memcg.

Quoted 25 lines
Fixes af92793e52c3 ("slab: Introduce kmalloc_nolock() and kfree_nolock().")
Assisted-by LLM
---
 mm/slub.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
mm/slub.c +4 -1
index 54ec125033571..a1f08338102e2 100644
@@ -2519,7 +2519,10 @@ bool memcg_slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags,
 		return true;
 
 	if (likely(size == 1)) {
-		memcg_alloc_abort_single(s, *p);
+		if (alloc_flags_allow_spinning(ac->alloc_flags))
+			memcg_alloc_abort_single(s, *p);
+		else
+			kfree_nolock(*p);
 		*p = NULL;
 	} else {
 		kmem_cache_free_bulk(s, size, p);
Signature · 1 line
2.53.0
Signature · 2 lines
Cheers,
Harry / Hyeonggon
#3 lore raw

Re: [PATCH 1/2] mm/slub: preserve no-lock freeing after memcg charge failure

Quoted 10 lines · Harry Yoo
Quoted 4 lines

Use kfree_nolock() when the allocation flags disallow spinning. This keeps the SLUB object rollback on the no-lock free path. The failed allocation continues to return NULL.

When this happens, the object is not charged by memcg.

The kernel should not invoke memcg_slab_free_hook() (called by kfree_nolock()) for a slab object that is not charged by memcg.

Agreed. I'll avoid the memcg free hook on charge abort.

One question: kfree_nolock() currently skips init_on_free, while the existing charge-abort path honors it. Was that intentional? I couldn't find it discussed in the introduction threads.

If kfree_nolock() should honor init_on_free, I'll include that fix as the first patch in the revised series. The charge-abort fix will then reuse the shared initialization helper.

Karl

#4 lore raw

PATCH 2/2 mm/memcontrol: defer final objcg release from no-lock frees

In reply to Karl Mehltretter

kfree_nolock() and free_pages_nolock() can drop the last reference to a killed object cgroup. percpu_ref then invokes obj_cgroup_release() in the caller's context. The callback can uncharge pages. It then takes objcg_lock, exits the percpu reference, and schedules an RCU free. A no-lock free can therefore enter regular locking. On PREEMPT_RT this can take a sleeping lock while the caller holds a raw scheduler lock.

Make the release callback add the object cgroup to an NMI-safe lockless list and queue normal irq_work. Drain the list outside the no-lock caller's context. On PREEMPT_RT normal irq work runs in irq_workd task context. On non-RT the work remains safe to run from hard interrupt context, as required by the existing release path.

Fixes af92793e52c3 ("slab: Introduce kmalloc_nolock() and kfree_nolock().")
Assisted-by LLM
---
 include/linux/memcontrol.h |  1 +
 mm/memcontrol.c            | 29 ++++++++++++++++++++++++++---
 2 files changed, 27 insertions(+), 3 deletions(-)
include/linux/memcontrol.h +1 -0
index 7d1c0ce189a88..c755d946430b6 100644
@@ -186,6 +186,7 @@ struct obj_cgroup {
 	struct percpu_ref refcnt;
 	struct mem_cgroup *memcg;
 	atomic_t nr_charged_bytes;
+	struct llist_node release_node;
 	union {
 		struct list_head list; /* protected by objcg_lock */
 		struct rcu_head rcu;
mm/memcontrol.c +26 -3
index 856a7d07586cc..63b18c7f1965f 100644
@@ -33,6 +33,8 @@
 #include <linux/sched/mm.h>
 #include <linux/shmem_fs.h>
 #include <linux/hugetlb.h>
+#include <linux/irq_work.h>
+#include <linux/llist.h>
 #include <linux/pagemap.h>
 #include <linux/folio_batch.h>
 #include <linux/vm_event_item.h>
@@ -146,9 +148,12 @@ static void memcg_uncharge_kmem(struct mem_cgroup *memcg, unsigned int nr_pages)
 		memcg_uncharge(memcg, nr_pages);
 }
 
-static void obj_cgroup_release(struct percpu_ref *ref)
+static LLIST_HEAD(objcg_release_list);
+static void obj_cgroup_release_workfn(struct irq_work *work);
+static DEFINE_IRQ_WORK(objcg_release_work, obj_cgroup_release_workfn);
+
+static void obj_cgroup_release_one(struct obj_cgroup *objcg)
 {
-	struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt);
 	unsigned int nr_bytes;
 	unsigned int nr_pages;
 	unsigned long flags;
@@ -189,10 +194,28 @@ static void obj_cgroup_release(struct percpu_ref *ref)
 	list_del(&objcg->list);
 	spin_unlock_irqrestore(&objcg_lock, flags);
 
-	percpu_ref_exit(ref);
+	percpu_ref_exit(&objcg->refcnt);
 	kfree_rcu(objcg, rcu);
 }
 
+static void obj_cgroup_release_workfn(struct irq_work *work)
+{
+	struct llist_node *node;
+	struct obj_cgroup *objcg, *next;
+
+	node = llist_del_all(&objcg_release_list);
+	llist_for_each_entry_safe(objcg, next, node, release_node)
+		obj_cgroup_release_one(objcg);
+}
+
+static void obj_cgroup_release(struct percpu_ref *ref)
+{
+	struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt);
+
+	llist_add(&objcg->release_node, &objcg_release_list);
+	irq_work_queue(&objcg_release_work);
+}
+
 static struct obj_cgroup *obj_cgroup_alloc(void)
 {
 	struct obj_cgroup *objcg;
Signature · 1 line
2.53.0