Skip to contentSkip to conversation

mm, memcg: Report number of memcg caches in slabinfo

PATCH In review Waiman Long 2 participants 7 messages Jun 17, 2019 – Jun 18, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm, memcg: Report number of memcg caches in slabinfo

There are concerns about memory leaks from extensive use of memory cgroups as each memory cgroup creates its own set of kmem caches. There is a possiblity that the memcg kmem caches may remain even after the memory cgroup removal.

Therefore, it will be useful to show how many memcg caches are present for each of the kmem caches. As slabinfo reporting code has to iterate through all the memcg caches to get the final numbers anyway, there is no additional cost in reporting the number of memcg caches available.

The slabinfo version is bumped up to 2.2 as a new "<num_caches>" column is added at the end.

---
 mm/slab_common.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)
mm/slab_common.c +16 -8
index 58251ba63e4a..c7aa47a99b2b 100644
@@ -1308,13 +1308,13 @@ static void print_slabinfo_header(struct seq_file *m)
 	 * without _too_ many complaints.
 	 */
 #ifdef CONFIG_DEBUG_SLAB
-	seq_puts(m, "slabinfo - version: 2.1 (statistics)\n");
+	seq_puts(m, "slabinfo - version: 2.2 (statistics)\n");
 #else
-	seq_puts(m, "slabinfo - version: 2.1\n");
+	seq_puts(m, "slabinfo - version: 2.2\n");
 #endif
 	seq_puts(m, "# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab>");
 	seq_puts(m, " : tunables <limit> <batchcount> <sharedfactor>");
-	seq_puts(m, " : slabdata <active_slabs> <num_slabs> <sharedavail>");
+	seq_puts(m, " : slabdata <active_slabs> <num_slabs> <sharedavail> <num_caches>");
 #ifdef CONFIG_DEBUG_SLAB
 	seq_puts(m, " : globalstat <listallocs> <maxobjs> <grown> <reaped> <error> <maxfreeable> <nodeallocs> <remotefrees> <alienoverflow>");
 	seq_puts(m, " : cpustat <allochit> <allocmiss> <freehit> <freemiss>");
@@ -1338,14 +1338,18 @@ void slab_stop(struct seq_file *m, void *p)
 	mutex_unlock(&slab_mutex);
 }
 
-static void
+/*
+ * Return number of memcg caches.
+ */
+static unsigned int
 memcg_accumulate_slabinfo(struct kmem_cache *s, struct slabinfo *info)
 {
 	struct kmem_cache *c;
 	struct slabinfo sinfo;
+	unsigned int cnt = 0;
 
 	if (!is_root_cache(s))
-		return;
+		return 0;
 
 	for_each_memcg_cache(c, s) {
 		memset(&sinfo, 0, sizeof(sinfo));
@@ -1356,17 +1360,20 @@ memcg_accumulate_slabinfo(struct kmem_cache *s, struct slabinfo *info)
 		info->shared_avail += sinfo.shared_avail;
 		info->active_objs += sinfo.active_objs;
 		info->num_objs += sinfo.num_objs;
+		cnt++;
 	}
+	return cnt;
 }
 
 static void cache_show(struct kmem_cache *s, struct seq_file *m)
 {
 	struct slabinfo sinfo;
+	unsigned int nr_memcg_caches;
 
 	memset(&sinfo, 0, sizeof(sinfo));
 	get_slabinfo(s, &sinfo);
 
-	memcg_accumulate_slabinfo(s, &sinfo);
+	nr_memcg_caches = memcg_accumulate_slabinfo(s, &sinfo);
 
 	seq_printf(m, "%-17s %6lu %6lu %6u %4u %4d",
 		   cache_name(s), sinfo.active_objs, sinfo.num_objs, s->size,
@@ -1374,8 +1381,9 @@ static void cache_show(struct kmem_cache *s, struct seq_file *m)
 
 	seq_printf(m, " : tunables %4u %4u %4u",
 		   sinfo.limit, sinfo.batchcount, sinfo.shared);
-	seq_printf(m, " : slabdata %6lu %6lu %6lu",
-		   sinfo.active_slabs, sinfo.num_slabs, sinfo.shared_avail);
+	seq_printf(m, " : slabdata %6lu %6lu %6lu %3u",
+		   sinfo.active_slabs, sinfo.num_slabs, sinfo.shared_avail,
+		   nr_memcg_caches);
 	slabinfo_show_stats(m, s);
 	seq_putc(m, '\n');
 }
Signature · 1 line
2.18.1
#1 lore raw

[Cc linux-api]

Quoted 7 lines · Waiman Long

There are concerns about memory leaks from extensive use of memory cgroups as each memory cgroup creates its own set of kmem caches. There is a possiblity that the memcg kmem caches may remain even after the memory cgroup removal.

Therefore, it will be useful to show how many memcg caches are present for each of the kmem caches.

How is a user going to use that information? Btw. Don't we have an interface to display the number of (dead) cgroups?

Keeping the rest of the email for the reference.

Quoted 90 lines

As slabinfo reporting code has to iterate
through all the memcg caches to get the final numbers anyway, there is no additional cost in reporting the number of memcg caches available.

The slabinfo version is bumped up to 2.2 as a new "<num_caches>" column is added at the end.

---
 mm/slab_common.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)
mm/slab_common.c +16 -8
index 58251ba63e4a..c7aa47a99b2b 100644
@@ -1308,13 +1308,13 @@ static void print_slabinfo_header(struct seq_file *m)
 	 * without _too_ many complaints.
 	 */
 #ifdef CONFIG_DEBUG_SLAB
-	seq_puts(m, "slabinfo - version: 2.1 (statistics)\n");
+	seq_puts(m, "slabinfo - version: 2.2 (statistics)\n");
 #else
-	seq_puts(m, "slabinfo - version: 2.1\n");
+	seq_puts(m, "slabinfo - version: 2.2\n");
 #endif
 	seq_puts(m, "# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab>");
 	seq_puts(m, " : tunables <limit> <batchcount> <sharedfactor>");
-	seq_puts(m, " : slabdata <active_slabs> <num_slabs> <sharedavail>");
+	seq_puts(m, " : slabdata <active_slabs> <num_slabs> <sharedavail> <num_caches>");
 #ifdef CONFIG_DEBUG_SLAB
 	seq_puts(m, " : globalstat <listallocs> <maxobjs> <grown> <reaped> <error> <maxfreeable> <nodeallocs> <remotefrees> <alienoverflow>");
 	seq_puts(m, " : cpustat <allochit> <allocmiss> <freehit> <freemiss>");
@@ -1338,14 +1338,18 @@ void slab_stop(struct seq_file *m, void *p)
 	mutex_unlock(&slab_mutex);
 }
 
-static void
+/*
+ * Return number of memcg caches.
+ */
+static unsigned int
 memcg_accumulate_slabinfo(struct kmem_cache *s, struct slabinfo *info)
 {
 	struct kmem_cache *c;
 	struct slabinfo sinfo;
+	unsigned int cnt = 0;
 
 	if (!is_root_cache(s))
-		return;
+		return 0;
 
 	for_each_memcg_cache(c, s) {
 		memset(&sinfo, 0, sizeof(sinfo));
@@ -1356,17 +1360,20 @@ memcg_accumulate_slabinfo(struct kmem_cache *s, struct slabinfo *info)
 		info->shared_avail += sinfo.shared_avail;
 		info->active_objs += sinfo.active_objs;
 		info->num_objs += sinfo.num_objs;
+		cnt++;
 	}
+	return cnt;
 }
 
 static void cache_show(struct kmem_cache *s, struct seq_file *m)
 {
 	struct slabinfo sinfo;
+	unsigned int nr_memcg_caches;
 
 	memset(&sinfo, 0, sizeof(sinfo));
 	get_slabinfo(s, &sinfo);
 
-	memcg_accumulate_slabinfo(s, &sinfo);
+	nr_memcg_caches = memcg_accumulate_slabinfo(s, &sinfo);
 
 	seq_printf(m, "%-17s %6lu %6lu %6u %4u %4d",
 		   cache_name(s), sinfo.active_objs, sinfo.num_objs, s->size,
@@ -1374,8 +1381,9 @@ static void cache_show(struct kmem_cache *s, struct seq_file *m)
 
 	seq_printf(m, " : tunables %4u %4u %4u",
 		   sinfo.limit, sinfo.batchcount, sinfo.shared);
-	seq_printf(m, " : slabdata %6lu %6lu %6lu",
-		   sinfo.active_slabs, sinfo.num_slabs, sinfo.shared_avail);
+	seq_printf(m, " : slabdata %6lu %6lu %6lu %3u",
+		   sinfo.active_slabs, sinfo.num_slabs, sinfo.shared_avail,
+		   nr_memcg_caches);
 	slabinfo_show_stats(m, s);
 	seq_putc(m, '\n');
 }
Signature · 1 line
2.18.1
Signature · 2 lines
Michal Hocko
SUSE Labs
#2 lore raw
Quoted 8 lines · Michal Hocko

[Cc linux-api]

Quoted 5 lines · Waiman Long

There are concerns about memory leaks from extensive use of memory cgroups as each memory cgroup creates its own set of kmem caches. There is a possiblity that the memcg kmem caches may remain even after the memory cgroup removal.

Quoted 4 lines
Quoted 2 lines

Therefore, it will be useful to show how many memcg caches are present for each of the kmem caches.

How is a user going to use that information? Btw. Don't we have an interface to display the number of (dead) cgroups?

The interface to report dead cgroups is for cgroup v2 (cgroup.stat) only. I don't think there is a way to find that for cgroup v1. Also the number of memcg kmem caches may not be the same as the number of memcg's. It can range from 0 to above the number of memcg's.  So it is an interesting number by itself.

Quoted 1 line

From the user perspective, if the numbers is way above the number of

memcg's, there is probably something wrong there.

Cheers,
Longman

#3 lore raw
Quoted 9 lines · Waiman Long
Quoted 8 lines · Michal Hocko

[Cc linux-api]

Quoted 5 lines · Waiman Long

There are concerns about memory leaks from extensive use of memory cgroups as each memory cgroup creates its own set of kmem caches. There is a possiblity that the memcg kmem caches may remain even after the memory cgroup removal.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Therefore, it will be useful to show how many memcg caches are present for each of the kmem caches.

How is a user going to use that information? Btw. Don't we have an interface to display the number of (dead) cgroups?

The interface to report dead cgroups is for cgroup v2 (cgroup.stat) only. I don't think there is a way to find that for cgroup v1.

Doesn't debug_legacy_files provide the information for both cgroups APIs?

Quoted 4 lines

Also the
number of memcg kmem caches may not be the same as the number of memcg's. It can range from 0 to above the number of memcg's.  So it is an interesting number by itself.

Is this useful enough to put into slabinfo? Doesn't this sound more like a debugfs kinda a thing?

Quoted 2 lines

From the user perspective, if the numbers is way above the number of memcg's, there is probably something wrong there.

Signature · 2 lines
Michal Hocko
SUSE Labs
#4 lore raw
Quoted 10 lines · Michal Hocko
Quoted 9 lines · Waiman Long
Quoted 8 lines · Michal Hocko

[Cc linux-api]

Quoted 5 lines · Waiman Long

There are concerns about memory leaks from extensive use of memory cgroups as each memory cgroup creates its own set of kmem caches. There is a possiblity that the memcg kmem caches may remain even after the memory cgroup removal.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Therefore, it will be useful to show how many memcg caches are present for each of the kmem caches.

How is a user going to use that information? Btw. Don't we have an interface to display the number of (dead) cgroups?

The interface to report dead cgroups is for cgroup v2 (cgroup.stat) only. I don't think there is a way to find that for cgroup v1.

Doesn't debug_legacy_files provide the information for both cgroups APIs?

Not really. The debug controller doesn't provide information about the number of dead cgroups, for instance. Of course, we can always add those information there. Also the debug controller is not typically configured into a production kernel.

Quoted 6 lines
Quoted 4 lines

Also the
number of memcg kmem caches may not be the same as the number of memcg's. It can range from 0 to above the number of memcg's.  So it is an interesting number by itself.

Is this useful enough to put into slabinfo? Doesn't this sound more like a debugfs kinda a thing?

I guess it is probably more on the debug side of things. I add it to slabinfo as the data is readily available. It will be much more work if we need to export the data via debugfs.

We are seeing the kmem_cache slab growing continuously overtime when running a container-based workloads. Roman's kmem_cache reparenting patch will hopefully solve a major part of the problem, but we still need a way to confirm that by looking at how many memcg kmem_caches are associated with each root kmem_cache.

Cheers,
Longman

#5 lore raw
Quoted 1 line · Waiman Long

On 6/18/19 8:37 AM, Michal Hocko wrote:

[...]

Quoted 7 lines
Quoted 2 lines

Is this useful enough to put into slabinfo? Doesn't this sound more like a debugfs kinda a thing?

I guess it is probably more on the debug side of things. I add it to slabinfo as the data is readily available. It will be much more work if we need to export the data via debugfs.

Quoted 5 lines

We are seeing the kmem_cache slab growing continuously overtime when running a container-based workloads. Roman's kmem_cache reparenting patch will hopefully solve a major part of the problem, but we still need a way to confirm that by looking at how many memcg kmem_caches are associated with each root kmem_cache.

I am not disputing usefulness. Dead memcgs are showing up as a problem for a longer time and having a more debugging information is definitely useful. I am just not really sure that /proc/slabinfo is the proper vehicle for that information. It might be just easier to stick it there but that is not the best justification for adding something we will have to maintain for ever. Not to mention that the number of dead memcgs might not be enough to debug further when we can easily end up needing to provide more in something less "carved in stone" kinda interface like debugfs.

Signature · 2 lines
Michal Hocko
SUSE Labs
#6 lore raw
Quoted 14 lines · Michal Hocko
Quoted 1 line · Waiman Long

On 6/18/19 8:37 AM, Michal Hocko wrote:

[...]

Quoted 11 lines
Quoted 2 lines

Is this useful enough to put into slabinfo? Doesn't this sound more like a debugfs kinda a thing?

I guess it is probably more on the debug side of things. I add it to slabinfo as the data is readily available. It will be much more work if we need to export the data via debugfs.

We are seeing the kmem_cache slab growing continuously overtime when running a container-based workloads. Roman's kmem_cache reparenting patch will hopefully solve a major part of the problem, but we still need a way to confirm that by looking at how many memcg kmem_caches are associated with each root kmem_cache.

Quoted 10 lines

I am not disputing usefulness. Dead memcgs are showing up as a problem for a longer time and having a more debugging information is definitely useful. I am just not really sure that /proc/slabinfo is the proper vehicle for that information. It might be just easier to stick it there but that is not the best justification for adding something we will have to maintain for ever. Not to mention that the number of dead memcgs might not be enough to debug further when we can easily end up needing to provide more in something less "carved in stone" kinda interface like debugfs.

Fair enough.

I will rework the patch and expose the information via debugfs then.

Cheers,
Longman