Skip to contentSkip to conversation

mm: kmem: optimize obj_cgroup pointer retrieval

RFC PATCH In review Roman Gushchin 3 participants 9 messages Dec 20, 2022 – Jan 2, 2023

Subsystems (4)
lore
Share & tools
lore

Roman Gushchin

author maintainer
#0 lore raw

cover mm: kmem: optimize obj_cgroup pointer retrieval

This patchset improves the performance of get_obj_cgroup_from_current(), which is used to get an objcg pointer on the kernel memory allocation fast path.

Results (1M 8-bytes accounted allocations):

| version         | accounted (us) | delta | unaccounted (us) |  delta |
|-----------------+----------------+-------+------------------+--------|
| baseline (6.1+) |          81042 |       |            45269 |        |
| patch 1         |          78756 | -2.8% |            42731 |  -5.6% |
| patch 2         |          73650 | -9.1% |            30662 | -32.3% |

Unaccounted allocations were performed from a user's task belonging to the root cgroup, so savings are particularly large because previously the root_mem_cgroup pointer was obtained first just to learn that it's corresponding objcg is NULL.

Roman Gushchin (2):
mm: kmem: optimize get_obj_cgroup_from_current()
mm: kmem: add direct objcg pointer to task_struct

 include/linux/sched.h |   4 ++
 mm/memcontrol.c       | 102 ++++++++++++++++++++++++++++++++----------
 2 files changed, 83 insertions(+), 23 deletions(-)
Signature · 1 line
2.39.0
author maintainer
#1 lore raw

RFC PATCH 1/2 mm: kmem: optimize get_obj_cgroup_from_current()

Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings.

Also add a likely() macro to __get_obj_cgroup_from_memcg():
obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path.

---
 mm/memcontrol.c | 34 ++++++++++++++--------------------
 1 file changed, 14 insertions(+), 20 deletions(-)
mm/memcontrol.c +14 -20
index bafd3cde4507..82828c51d2ea 100644
@@ -1047,19 +1047,6 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm)
 }
 EXPORT_SYMBOL(get_mem_cgroup_from_mm);
 
-static __always_inline bool memcg_kmem_bypass(void)
-{
-	/* Allow remote memcg charging from any context. */
-	if (unlikely(active_memcg()))
-		return false;
-
-	/* Memcg to charge can't be determined. */
-	if (!in_task() || !current->mm || (current->flags & PF_KTHREAD))
-		return true;
-
-	return false;
-}
-
 /**
  * mem_cgroup_iter - iterate over memory cgroup hierarchy
  * @root: hierarchy root
@@ -3004,7 +2991,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg)
 
 	for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
 		objcg = rcu_dereference(memcg->objcg);
-		if (objcg && obj_cgroup_tryget(objcg))
+		if (likely(objcg && obj_cgroup_tryget(objcg)))
 			break;
 		objcg = NULL;
 	}
@@ -3013,16 +3000,23 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg)
 
 __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void)
 {
-	struct obj_cgroup *objcg = NULL;
 	struct mem_cgroup *memcg;
+	struct obj_cgroup *objcg;
 
-	if (memcg_kmem_bypass())
-		return NULL;
+	if (in_task()) {
+		memcg = current->active_memcg;
+
+		/* Memcg to charge can't be determined. */
+		if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD)))
+			return NULL;
+	} else {
+		memcg = this_cpu_read(int_active_memcg);
+		if (likely(!memcg))
+			return NULL;
+	}
 
 	rcu_read_lock();
-	if (unlikely(active_memcg()))
-		memcg = active_memcg();
-	else
+	if (!memcg)
 		memcg = mem_cgroup_from_task(current);
 	objcg = __get_obj_cgroup_from_memcg(memcg);
 	rcu_read_unlock();
Signature · 1 line
2.39.0
author maintainer
#4 lore raw

RFC PATCH 2/2 mm: kmem: add direct objcg pointer to task_struct

In reply to Roman Gushchin

To charge a freshly allocated kernel object to a memory cgroup, the kernel needs to obtain an objcg pointer. Currently it does it indirectly by obtaining the memcg pointer first and then calling to __get_obj_cgroup_from_memcg().

Usually tasks spend their entire life belonging to the same object cgroup. So it makes sense to save the objcg pointer on task_struct directly, so it can be obtained faster. It requires some work on fork, exit and cgroup migrate paths, but these paths are way colder.

The old indirect way is still used for remote memcg charging.

---
 include/linux/sched.h |  4 +++
 mm/memcontrol.c       | 84 +++++++++++++++++++++++++++++++++++++------
 2 files changed, 77 insertions(+), 11 deletions(-)
include/linux/sched.h +4 -0
index 853d08f7562b..e17be609cbcb 100644
@@ -1435,6 +1435,10 @@ struct task_struct {
 	struct mem_cgroup		*active_memcg;
 #endif
 
+#ifdef CONFIG_MEMCG_KMEM
+	struct obj_cgroup		*objcg;
+#endif
+
 #ifdef CONFIG_BLK_CGROUP
 	struct request_queue		*throttle_queue;
 #endif
mm/memcontrol.c +73 -11

128 lines not on this page · raw message

Signature · 1 line
2.39.0
Acked-by on v1 1/2
#2 lore raw

Re: [PATCH RFC 1/2] mm: kmem: optimize get_obj_cgroup_from_current()

In reply to Roman Gushchin
Quoted 10 lines · Roman Gushchin

Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings.

Also add a likely() macro to __get_obj_cgroup_from_memcg():
obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path.

Can you please add your performance experiment setup and result of this patch in the commit description of this patch as well?

#5 lore raw

Re: [PATCH RFC 2/2] mm: kmem: add direct objcg pointer to task_struct

In reply to Roman Gushchin
Quoted 5 lines · Roman Gushchin

To charge a freshly allocated kernel object to a memory cgroup, the kernel needs to obtain an objcg pointer. Currently it does it indirectly by obtaining the memcg pointer first and then calling to __get_obj_cgroup_from_memcg().

Quoted 8 lines

Usually tasks spend their entire life belonging to the same object cgroup. So it makes sense to save the objcg pointer on task_struct directly, so it can be obtained faster. It requires some work on fork, exit and cgroup migrate paths, but these paths are way colder.

The old indirect way is still used for remote memcg charging.

This looks good too. Few comments below:

[...]

Quoted 14 lines
mm/memcontrol.c +14 -0
+
+#ifdef CONFIG_MEMCG_KMEM
+static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset)
+{
+	struct task_struct *task;
+	struct cgroup_subsys_state *css;
+
+	cgroup_taskset_for_each(task, css, tset) {
+		struct mem_cgroup *memcg;
+
+		if (task->objcg)
+			obj_cgroup_put(task->objcg);
+
+		rcu_read_lock();

mm/memcontrol.c static void mem_cgroup_attach(struct cgroup_taskset *tset) · line 6391

Quoted 10 lines
mm/memcontrol.c +10 -0
+		memcg = container_of(css, struct mem_cgroup, css);
+		task->objcg = __get_obj_cgroup_from_memcg(memcg);
+		rcu_read_unlock();
+	}
+}
+#else
+static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) {}
+#endif /* CONFIG_MEMCG_KMEM */
+
+#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEMCG_KMEM)

I think you want CONFIG_LRU_GEN in the above check.

Quoted 11 lines

static void mem_cgroup_attach(struct cgroup_taskset *tset) {

mm/memcontrol.c +3 -1
+	mem_cgroup_lru_gen_attach(tset);
+	mem_cgroup_kmem_attach(tset);
 }
-#endif /* CONFIG_LRU_GEN */
+#endif
 
 static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value)
 {

mm/memcontrol.c struct cgroup_subsys memory_cgrp_subsys = { · line 6875

Quoted 4 lines
mm/memcontrol.c +1 -0
@@ -6816,9 +6872,15 @@ struct cgroup_subsys memory_cgrp_subsys = {
 	.css_reset = mem_cgroup_css_reset,
 	.css_rstat_flush = mem_cgroup_css_rstat_flush,
 	.can_attach = mem_cgroup_can_attach,
+#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEMCG_KMEM)

Same here.

Quoted 14 lines
mm/memcontrol.c +5 -0
 	.attach = mem_cgroup_attach,
+#endif
 	.cancel_attach = mem_cgroup_cancel_attach,
 	.post_attach = mem_cgroup_move_task,
+#ifdef CONFIG_MEMCG_KMEM
+	.fork = mem_cgroup_fork,
+	.exit = mem_cgroup_exit,
+#endif
 	.dfl_cftypes = memory_files,
 	.legacy_cftypes = mem_cgroup_legacy_files,
 	.early_init = 0,
Signature · 1 line
2.39.0
author maintainer
#3 lore raw

Re: [PATCH RFC 1/2] mm: kmem: optimize get_obj_cgroup_from_current()

In reply to Shakeel Butt
Quoted 7 lines · Shakeel Butt
Quoted 5 lines · Roman Gushchin

Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings.

Quoted 8 lines
Quoted 5 lines

Also add a likely() macro to __get_obj_cgroup_from_memcg():
obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path.

Can you please add your performance experiment setup and result of this patch in the commit description of this patch as well?

Sure. I used a small hack to just do a bunch of allocations in a raw and measured the time. Will include it into the commit message.

Also will fix the #ifdef thing from the second patch, thanks for spotting it.

Quoted 2 lines

Thank you for taking a look!

maintainer
#6 lore raw

Re: [PATCH RFC 2/2] mm: kmem: add direct objcg pointer to task_struct

In reply to Roman Gushchin
Quoted 4 lines · Roman Gushchin

To charge a freshly allocated kernel object to a memory cgroup, the kernel needs to obtain an objcg pointer. Currently it does it indirectly by obtaining the memcg pointer first and then calling to __get_obj_cgroup_from_memcg().

Jinx [1].

You report additional 7% improvement with this patch (focused on allocations only). I didn't see impressive numbers (different benchmark in [1]), so it looked as a microoptimization without big benefit to me.

My 0.02€ to RFC,
Michal

[1] https://bugzilla.kernel.org/show_bug.cgi?id=216038#c5

author maintainer
#7 lore raw

Re: [PATCH RFC 2/2] mm: kmem: add direct objcg pointer to task_struct

Quoted 11 lines · Michal Koutný
Quoted 4 lines · Roman Gushchin

To charge a freshly allocated kernel object to a memory cgroup, the kernel needs to obtain an objcg pointer. Currently it does it indirectly by obtaining the memcg pointer first and then calling to __get_obj_cgroup_from_memcg().

Jinx [1].

You report additional 7% improvement with this patch (focused on allocations only). I didn't see impressive numbers (different benchmark in [1]), so it looked as a microoptimization without big benefit to me.

Hi Michal!

Thank you for taking a look.
Do you have any numbers to share?

In general, I agree that it's a micro-optimization, but:
1) some people periodically complain that accounted allocations are slow in comparison to non-accounted and slower than they were with page-based accounting,
2) I don't see any particular hot point or obviously non-optimal place on the allocation path.

so if we want to make it faster, we have to micro-optimize it here and there, no other way. It's basically the question how many cache lines we touch.

Btw, I'm working on a patch 3 for this series, which in early tests brings additional ~25% improvement in my benchmark, hopefully will post it soon as a part of v1.

Thanks!

maintainer
#8 lore raw

Re: [PATCH RFC 2/2] mm: kmem: add direct objcg pointer to task_struct

Hello.

Quoted 1 line · Roman Gushchin

Do you have any numbers to share?

The numbers are in bko#216038, let me explain them here a bit. I used the will-it-scale benchmark that repeatedly locks/unlocks a file and runs in parallel.

The final numbers were:
sample metric δ δ_cg
no accounting implemented 32307750 0 % ­
accounting in cg 2.49577e+07 -23 % 0 %
accounting in cg + cache 2.51642e+07 -22 % +1 %

Hence my result was only 1% improvement.

(But it was a very simple try, not delving into any of the CPU cache statistics.)

Question: Were your measurements multi-threaded?

Quoted 3 lines

1) some people periodically complain that accounted allocations are slow in comparison to non-accounted and slower than they were with page-based accounting,

My result above would not likely satisfy those complainers I know about. But if your additional changes are better the additional code complexity may be justified in the end.

Quoted 3 lines

Btw, I'm working on a patch 3 for this series, which in early tests brings additional ~25% improvement in my benchmark, hopefully will post it soon as a part of v1.

Please send it with more details about your benchmark to put the numbers into context.

Michal