Skip to contentSkip to conversation

cgroup: nmi safe css_rstat_updated

PATCH Superseded Shakeel Butt 3 participants 11 messages Jun 11, 2025 – Jun 17, 2025

Subsystems (5)
Memory management BPF Tracing cgroup +1 more subsystems

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Shakeel Butt

author maintainer
#0 lore raw

v2 cover cgroup: nmi safe css_rstat_updated

BPF programs can run in nmi context and may trigger memcg charged memory allocation in such context. Recently linux added support to nmi safe page allocation along with memcg charging of such allocations. However the kmalloc/slab support and corresponding memcg charging is still lacking,

To provide nmi safe support for memcg charging for kmalloc/slab allocations, we need nmi safe memcg stats because for kernel memory charging and stats happen together. At the moment, memcg charging and memcg stats are nmi safe and the only thing which is not nmi safe is adding the cgroup to the per-cpu rstat update tree. i.e. css_rstat_updated() which this series is doing.

This series made css_rstat_updated by using per-cpu lockless lists whose node in embedded in individual struct cgroup_subsys_state and the per-cpu head is placed in struct cgroup_subsys. For rstat users without cgroup_subsys, a global per-cpu lockless list head is created. The main challenge to use lockless in this scenario was the potential multiple inserters from the stacked context i.e. process, softirq, hardirq & nmi, potentially using the same per-cpu lockless node of a given cgroup_subsys_state. The normal lockless list does not protect against such scenario.

The multiple stacked inserters using potentially same lockless node was resolved by making one of them succeed on reset the lockless node and the winner gets to insert the lockless node in the corresponding lockless list. The losers can assume the lockless list insertion will eventually succeed and continue their operation.

Changelog since v2:
- Add more clear explanation in cover letter and in the comment as suggested by Andrew, Michal & Tejun.
- Use this_cpu_cmpxchg() instead of try_cmpxchg() as suggested by Tejun.
- Remove the per-cpu ss locks as they are not needed anymore.

Changelog since v1:
- Based on Yosry's suggestion always use llist on the update side and create the update tree on flush side

[v1] https://lore.kernel.org/cgroups/[email protected]/

Shakeel Butt (4):
cgroup: support to enable nmi-safe css_rstat_updated
cgroup: make css_rstat_updated nmi safe
cgroup: remove per-cpu per-subsystem locks
memcg: cgroup: call css_rstat_updated irrespective of in_nmi()

 include/linux/cgroup-defs.h   |  11 +--
 include/trace/events/cgroup.h |  47 ----------
 kernel/cgroup/rstat.c         | 169 +++++++++++++---------------------
 mm/memcontrol.c               |  10 +-
 4 files changed, 74 insertions(+), 163 deletions(-)
Signature · 1 line
2.47.1
author maintainer
#1 lore raw

PATCH v2 1/4 cgroup: support to enable nmi-safe css_rstat_updated

Add necessary infrastructure to enable the nmi-safe execution of css_rstat_updated(). Currently css_rstat_updated() takes a per-cpu per-css raw spinlock to add the given css in the per-cpu per-css update tree. However the kernel can not spin in nmi context, so we need to remove the spinning on the raw spinlock in css_rstat_updated().

To support lockless css_rstat_updated(), let's add necessary data structures in the css and ss structures.

---
 include/linux/cgroup-defs.h |  4 ++++
 kernel/cgroup/rstat.c       | 23 +++++++++++++++++++++--
 2 files changed, 25 insertions(+), 2 deletions(-)
include/linux/cgroup-defs.h +4 -0
index e61687d5e496..45860fe5dd0c 100644
@@ -384,6 +384,9 @@ struct css_rstat_cpu {
 	 */
 	struct cgroup_subsys_state *updated_children;
 	struct cgroup_subsys_state *updated_next;	/* NULL if not on the list */
+
+	struct llist_node lnode;		/* lockless list for update */
+	struct cgroup_subsys_state *owner;	/* back pointer */
 };
 
 /*
@@ -822,6 +825,7 @@ struct cgroup_subsys {
 
 	spinlock_t rstat_ss_lock;
 	raw_spinlock_t __percpu *rstat_ss_cpu_lock;
+	struct llist_head __percpu *lhead; /* lockless update list head */
 };
 
 extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem;
kernel/cgroup/rstat.c +21 -2
index cbeaa499a96a..a5608ae2be27 100644
@@ -11,6 +11,7 @@
 
 static DEFINE_SPINLOCK(rstat_base_lock);
 static DEFINE_PER_CPU(raw_spinlock_t, rstat_base_cpu_lock);
+static DEFINE_PER_CPU(struct llist_head, rstat_backlog_list);
 
 static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu);
 
@@ -45,6 +46,13 @@ static spinlock_t *ss_rstat_lock(struct cgroup_subsys *ss)
 	return &rstat_base_lock;
 }
 
+static inline struct llist_head *ss_lhead_cpu(struct cgroup_subsys *ss, int cpu)
+{
+	if (ss)
+		return per_cpu_ptr(ss->lhead, cpu);
+	return per_cpu_ptr(&rstat_backlog_list, cpu);
+}
+
 static raw_spinlock_t *ss_rstat_cpu_lock(struct cgroup_subsys *ss, int cpu)
 {
 	if (ss) {
@@ -468,7 +476,8 @@ int css_rstat_init(struct cgroup_subsys_state *css)
 	for_each_possible_cpu(cpu) {
 		struct css_rstat_cpu *rstatc = css_rstat_cpu(css, cpu);
 
-		rstatc->updated_children = css;
+		rstatc->owner = rstatc->updated_children = css;
+		init_llist_node(&rstatc->lnode);
 
 		if (is_self) {
 			struct cgroup_rstat_base_cpu *rstatbc;
@@ -532,9 +541,19 @@ int __init ss_rstat_init(struct cgroup_subsys *ss)
 			return -ENOMEM;
 	}
 
+	if (ss) {
+		ss->lhead = alloc_percpu(struct llist_head);
+		if (!ss->lhead) {
+			free_percpu(ss->rstat_ss_cpu_lock);
+			return -ENOMEM;
+		}
+	}
+
 	spin_lock_init(ss_rstat_lock(ss));
-	for_each_possible_cpu(cpu)
+	for_each_possible_cpu(cpu) {
 		raw_spin_lock_init(ss_rstat_cpu_lock(ss, cpu));
+		init_llist_head(ss_lhead_cpu(ss, cpu));
+	}
 
 	return 0;
 }
Signature · 1 line
2.47.1
author maintainer
#2 lore raw

PATCH v2 2/4 cgroup: make css_rstat_updated nmi safe

In reply to Shakeel Butt

To make css_rstat_updated() able to safely run in nmi context, let's move the rstat update tree creation at the flush side and use per-cpu lockless lists in struct cgroup_subsys to track the css whose stats are updated on that cpu.

The struct cgroup_subsys_state now has per-cpu lnode which needs to be inserted into the corresponding per-cpu lhead of struct cgroup_subsys. Since we want the insertion to be nmi safe, there can be multiple inserters on the same cpu for the same lnode. Here multiple inserters are from stacked contexts like softirq, hardirq and nmi.

The current llist does not provide function to protect against the scenario where multiple inserters can use the same lnode. So, using llist_node() out of the box is not safe for this scenario.

However we can protect against multiple inserters using the same lnode by using the fact llist node points to itself when not on the llist and atomically reset it and select the winner as the single inserter.

---
Changes since v1:
- More clear code comment as suggested by Tejun.

 kernel/cgroup/rstat.c | 65 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 53 insertions(+), 12 deletions(-)
kernel/cgroup/rstat.c +53 -12
index a5608ae2be27..a7550961dd12 100644
@@ -138,13 +138,16 @@ void _css_rstat_cpu_unlock(struct cgroup_subsys_state *css, int cpu,
  * @css: target cgroup subsystem state
  * @cpu: cpu on which rstat_cpu was updated
  *
- * @css's rstat_cpu on @cpu was updated. Put it on the parent's matching
- * rstat_cpu->updated_children list. See the comment on top of
- * css_rstat_cpu definition for details.
+ * Atomically inserts the css in the ss's llist for the given cpu. This is
+ * reentrant safe i.e. safe against softirq, hardirq and nmi. The ss's llist
+ * will be processed at the flush time to create the update tree.
  */
 __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
 {
-	unsigned long flags;
+	struct llist_head *lhead;
+	struct css_rstat_cpu *rstatc;
+	struct css_rstat_cpu __percpu *rstatc_pcpu;
+	struct llist_node *self;
 
 	/*
 	 * Since bpf programs can call this function, prevent access to
@@ -153,19 +156,44 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
 	if (!css_uses_rstat(css))
 		return;
 
+	lockdep_assert_preemption_disabled();
+
+	/*
+	 * For archs withnot nmi safe cmpxchg or percpu ops support, ignore
+	 * the requests from nmi context.
+	 */
+	if ((!IS_ENABLED(CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG) ||
+	     !IS_ENABLED(CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS)) && in_nmi())
+		return;
+
+	rstatc = css_rstat_cpu(css, cpu);
+	/* If already on list return. */
+	if (llist_on_list(&rstatc->lnode))
+		return;
+
 	/*
-	 * Speculative already-on-list test. This may race leading to
-	 * temporary inaccuracies, which is fine.
+	 * This function can be renentered by irqs and nmis for the same cgroup
+	 * and may try to insert the same per-cpu lnode into the llist. Note
+	 * that llist_add() does not protect against such scenarios.
 	 *
-	 * Because @parent's updated_children is terminated with @parent
-	 * instead of NULL, we can tell whether @css is on the list by
-	 * testing the next pointer for NULL.
+	 * To protect against such stacked contexts of irqs/nmis, we use the
+	 * fact that lnode points to itself when not on a list and then use
+	 * this_cpu_cmpxchg() to atomically set to NULL to select the winner
+	 * which will call llist_add(). The losers can assume the insertion is
+	 * successful and the winner will eventually add the per-cpu lnode to
+	 * the llist.
 	 */
-	if (data_race(css_rstat_cpu(css, cpu)->updated_next))
+	self = &rstatc->lnode;
+	rstatc_pcpu = css->rstat_cpu;
+	if (this_cpu_cmpxchg(rstatc_pcpu->lnode.next, self, NULL) != self)
 		return;
 
-	flags = _css_rstat_cpu_lock(css, cpu, true);
+	lhead = ss_lhead_cpu(css->ss, cpu);
+	llist_add(&rstatc->lnode, lhead);
+}
 
+static void __css_process_update_tree(struct cgroup_subsys_state *css, int cpu)
+{
 	/* put @css and all ancestors on the corresponding updated lists */
 	while (true) {
 		struct css_rstat_cpu *rstatc = css_rstat_cpu(css, cpu);
@@ -191,8 +219,19 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
 
 		css = parent;
 	}
+}
+
+static void css_process_update_tree(struct cgroup_subsys *ss, int cpu)
+{
+	struct llist_head *lhead = ss_lhead_cpu(ss, cpu);
+	struct llist_node *lnode;
+
+	while ((lnode = llist_del_first_init(lhead))) {
+		struct css_rstat_cpu *rstatc;
 
-	_css_rstat_cpu_unlock(css, cpu, flags, true);
+		rstatc = container_of(lnode, struct css_rstat_cpu, lnode);
+		__css_process_update_tree(rstatc->owner, cpu);
+	}
 }
 
 /**
@@ -300,6 +339,8 @@ static struct cgroup_subsys_state *css_rstat_updated_list(
 
 	flags = _css_rstat_cpu_lock(root, cpu, false);
 
+	css_process_update_tree(root->ss, cpu);
+
 	/* Return NULL if this subtree is not on-list */
 	if (!rstatc->updated_next)
 		goto unlock_ret;
Signature · 1 line
2.47.1
author maintainer
#3 lore raw

PATCH v2 3/4 cgroup: remove per-cpu per-subsystem locks

In reply to Shakeel Butt

The rstat update side used to insert the cgroup whose stats are updated in the update tree and the read side flush the update tree to get the latest uptodate stats. The per-cpu per-subsystem locks were used to synchronize the update and flush side. However now the update side does not access update tree but uses per-cpu lockless lists. So there is no need for locks to synchronize update and flush side. Let's remove them.

---
 include/linux/cgroup-defs.h   |   7 ---
 include/trace/events/cgroup.h |  47 ---------------
 kernel/cgroup/rstat.c         | 107 ++--------------------------------
 3 files changed, 4 insertions(+), 157 deletions(-)
include/linux/cgroup-defs.h +0 -7
index 45860fe5dd0c..bca3562e3df4 100644
@@ -375,12 +375,6 @@ struct css_rstat_cpu {
 	 * Child cgroups with stat updates on this cpu since the last read
 	 * are linked on the parent's ->updated_children through
 	 * ->updated_next. updated_children is terminated by its container css.
-	 *
-	 * In addition to being more compact, singly-linked list pointing to
-	 * the css makes it unnecessary for each per-cpu struct to point back
-	 * to the associated css.
-	 *
-	 * Protected by per-cpu css->ss->rstat_ss_cpu_lock.
 	 */
 	struct cgroup_subsys_state *updated_children;
 	struct cgroup_subsys_state *updated_next;	/* NULL if not on the list */
@@ -824,7 +818,6 @@ struct cgroup_subsys {
 	unsigned int depends_on;
 
 	spinlock_t rstat_ss_lock;
-	raw_spinlock_t __percpu *rstat_ss_cpu_lock;
 	struct llist_head __percpu *lhead; /* lockless update list head */
 };
 
include/trace/events/cgroup.h +0 -47

53 lines not on this page · raw message

kernel/cgroup/rstat.c +4 -103

149 lines not on this page · raw message

Signature · 1 line
2.47.1
author maintainer
#4 lore raw

PATCH v2 4/4 memcg: cgroup: call css_rstat_updated irrespective of in_nmi()

In reply to Shakeel Butt

css_rstat_updated() is nmi safe, so there is no need to avoid it in in_nmi(), so remove the check.

---
 mm/memcontrol.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
mm/memcontrol.c +5 -5
index 902da8a9c643..d122bfe33e98 100644
@@ -573,9 +573,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val,
 	if (!val)
 		return;
 
-	/* TODO: add to cgroup update tree once it is nmi-safe. */
-	if (!in_nmi())
-		css_rstat_updated(&memcg->css, cpu);
+	css_rstat_updated(&memcg->css, cpu);
 	statc_pcpu = memcg->vmstats_percpu;
 	for (; statc_pcpu; statc_pcpu = statc->parent_pcpu) {
 		statc = this_cpu_ptr(statc_pcpu);
@@ -2530,7 +2528,8 @@ static inline void account_slab_nmi_safe(struct mem_cgroup *memcg,
 	} else {
 		struct mem_cgroup_per_node *pn = memcg->nodeinfo[pgdat->node_id];
 
-		/* TODO: add to cgroup update tree once it is nmi-safe. */
+		/* preemption is disabled in_nmi(). */
+		css_rstat_updated(&memcg->css, smp_processor_id());
 		if (idx == NR_SLAB_RECLAIMABLE_B)
 			atomic_add(nr, &pn->slab_reclaimable);
 		else
@@ -2753,7 +2752,8 @@ static inline void account_kmem_nmi_safe(struct mem_cgroup *memcg, int val)
 	if (likely(!in_nmi())) {
 		mod_memcg_state(memcg, MEMCG_KMEM, val);
 	} else {
-		/* TODO: add to cgroup update tree once it is nmi-safe. */
+		/* preemption is disabled in_nmi(). */
+		css_rstat_updated(&memcg->css, smp_processor_id());
 		atomic_add(val, &memcg->kmem_stat);
 	}
 }
Signature · 1 line
2.47.1
maintainer
#5 lore raw
In reply to Shakeel Butt

Hello,

Quoted 5 lines · Shakeel Butt

Shakeel Butt (4):
cgroup: support to enable nmi-safe css_rstat_updated
cgroup: make css_rstat_updated nmi safe
cgroup: remove per-cpu per-subsystem locks
memcg: cgroup: call css_rstat_updated irrespective of in_nmi()

The patches look good to me. How should it be routed? Should I take all four, just the first three or would it better to route all through -mm?

Thanks.

Signature · 1 line
tejun
author maintainer
#6 lore raw
Quoted 12 lines · Tejun Heo

Hello,

Quoted 5 lines · Shakeel Butt

Shakeel Butt (4):
cgroup: support to enable nmi-safe css_rstat_updated
cgroup: make css_rstat_updated nmi safe
cgroup: remove per-cpu per-subsystem locks
memcg: cgroup: call css_rstat_updated irrespective of in_nmi()

The patches look good to me. How should it be routed? Should I take all four, just the first three or would it better to route all through -mm?

I would like all four to be together and since most of the code is in cgroup, cgroup tree makes more sense unless Andrew has different opinion.

thanks,
Shakeel

Tested-by on the v2 series
#9 lore raw
In reply to Shakeel Butt
Quoted 49 lines · Shakeel Butt

BPF programs can run in nmi context and may trigger memcg charged memory allocation in such context. Recently linux added support to nmi safe page allocation along with memcg charging of such allocations. However the kmalloc/slab support and corresponding memcg charging is still lacking,

To provide nmi safe support for memcg charging for kmalloc/slab allocations, we need nmi safe memcg stats because for kernel memory charging and stats happen together. At the moment, memcg charging and memcg stats are nmi safe and the only thing which is not nmi safe is adding the cgroup to the per-cpu rstat update tree. i.e. css_rstat_updated() which this series is doing.

This series made css_rstat_updated by using per-cpu lockless lists whose node in embedded in individual struct cgroup_subsys_state and the per-cpu head is placed in struct cgroup_subsys. For rstat users without cgroup_subsys, a global per-cpu lockless list head is created. The main challenge to use lockless in this scenario was the potential multiple inserters from the stacked context i.e. process, softirq, hardirq & nmi, potentially using the same per-cpu lockless node of a given cgroup_subsys_state. The normal lockless list does not protect against such scenario.

The multiple stacked inserters using potentially same lockless node was resolved by making one of them succeed on reset the lockless node and the winner gets to insert the lockless node in the corresponding lockless list. The losers can assume the lockless list insertion will eventually succeed and continue their operation.

Changelog since v2:
- Add more clear explanation in cover letter and in the comment as suggested by Andrew, Michal & Tejun.
- Use this_cpu_cmpxchg() instead of try_cmpxchg() as suggested by Tejun.
- Remove the per-cpu ss locks as they are not needed anymore.

Changelog since v1:
- Based on Yosry's suggestion always use llist on the update side and create the update tree on flush side

[v1] https://lore.kernel.org/cgroups/[email protected]/

Shakeel Butt (4):
cgroup: support to enable nmi-safe css_rstat_updated
cgroup: make css_rstat_updated nmi safe
cgroup: remove per-cpu per-subsystem locks
memcg: cgroup: call css_rstat_updated irrespective of in_nmi()

Quoted 6 lines
  include/linux/cgroup-defs.h   |  11 +--
  include/trace/events/cgroup.h |  47 ----------
  kernel/cgroup/rstat.c         | 169 +++++++++++++---------------------
  mm/memcontrol.c               |  10 +-
  4 files changed, 74 insertions(+), 163 deletions(-)

I tested this series by doing some updates/flushes on a cgroup hierarchy with four levels. This tag can be added to the patches in this series.

author maintainer
#10 lore raw
Quoted 56 lines · JP Kobryn
Quoted 48 lines · Shakeel Butt

BPF programs can run in nmi context and may trigger memcg charged memory allocation in such context. Recently linux added support to nmi safe page allocation along with memcg charging of such allocations. However the kmalloc/slab support and corresponding memcg charging is still lacking,

To provide nmi safe support for memcg charging for kmalloc/slab allocations, we need nmi safe memcg stats because for kernel memory charging and stats happen together. At the moment, memcg charging and memcg stats are nmi safe and the only thing which is not nmi safe is adding the cgroup to the per-cpu rstat update tree. i.e. css_rstat_updated() which this series is doing.

This series made css_rstat_updated by using per-cpu lockless lists whose node in embedded in individual struct cgroup_subsys_state and the per-cpu head is placed in struct cgroup_subsys. For rstat users without cgroup_subsys, a global per-cpu lockless list head is created. The main challenge to use lockless in this scenario was the potential multiple inserters from the stacked context i.e. process, softirq, hardirq & nmi, potentially using the same per-cpu lockless node of a given cgroup_subsys_state. The normal lockless list does not protect against such scenario.

The multiple stacked inserters using potentially same lockless node was resolved by making one of them succeed on reset the lockless node and the winner gets to insert the lockless node in the corresponding lockless list. The losers can assume the lockless list insertion will eventually succeed and continue their operation.

Changelog since v2:
- Add more clear explanation in cover letter and in the comment as suggested by Andrew, Michal & Tejun.
- Use this_cpu_cmpxchg() instead of try_cmpxchg() as suggested by Tejun.
- Remove the per-cpu ss locks as they are not needed anymore.

Changelog since v1:
- Based on Yosry's suggestion always use llist on the update side and create the update tree on flush side

[v1] https://lore.kernel.org/cgroups/[email protected]/

Shakeel Butt (4):
cgroup: support to enable nmi-safe css_rstat_updated
cgroup: make css_rstat_updated nmi safe
cgroup: remove per-cpu per-subsystem locks
memcg: cgroup: call css_rstat_updated irrespective of in_nmi()

Quoted 6 lines
  include/linux/cgroup-defs.h   |  11 +--
  include/trace/events/cgroup.h |  47 ----------
  kernel/cgroup/rstat.c         | 169 +++++++++++++---------------------
  mm/memcontrol.c               |  10 +-
  4 files changed, 74 insertions(+), 163 deletions(-)
Quoted 5 lines

I tested this series by doing some updates/flushes on a cgroup hierarchy with four levels. This tag can be added to the patches in this series.

Thanks a lot.

maintainer
#7 lore raw
In reply to Shakeel Butt
Quoted 10 lines · Shakeel Butt
Quoted 9 lines · Tejun Heo

Hello,

Quoted 5 lines · Shakeel Butt

Shakeel Butt (4):
cgroup: support to enable nmi-safe css_rstat_updated
cgroup: make css_rstat_updated nmi safe
cgroup: remove per-cpu per-subsystem locks
memcg: cgroup: call css_rstat_updated irrespective of in_nmi()

Quoted 7 lines
Quoted 3 lines

The patches look good to me. How should it be routed? Should I take all four, just the first three or would it better to route all through -mm?

I would like all four to be together and since most of the code is in cgroup, cgroup tree makes more sense unless Andrew has different opinion.

Okay, I'll route them through cgroup. The patches don't apply cleanly on cgroup/for-6.17. Can you please send a refreshed set?

Thanks.

Signature · 1 line
tejun
author maintainer
#8 lore raw
Quoted 15 lines · Tejun Heo
Quoted 14 lines · Shakeel Butt
Quoted 12 lines · Tejun Heo

Hello,

Quoted 5 lines · Shakeel Butt

Shakeel Butt (4):
cgroup: support to enable nmi-safe css_rstat_updated
cgroup: make css_rstat_updated nmi safe
cgroup: remove per-cpu per-subsystem locks
memcg: cgroup: call css_rstat_updated irrespective of in_nmi()

The patches look good to me. How should it be routed? Should I take all four, just the first three or would it better to route all through -mm?

Quoted 6 lines
Quoted 3 lines

I would like all four to be together and since most of the code is in cgroup, cgroup tree makes more sense unless Andrew has different opinion.

Okay, I'll route them through cgroup. The patches don't apply cleanly on cgroup/for-6.17. Can you please send a refreshed set?

Yup, I will do asap.