slab+slub accounting for memcg

From: Suleiman Souhlal <[email protected]>

Signed-off-by Suleiman Souhlal <[email protected]>

From: Suleiman Souhlal <[email protected]>

mem_cgroup_do_charge() was written before slab accounting, and expects three cases: being called for 1 page, being called for a stock of 32 pages, or being called for a hugepage. If we call for 2 pages (and several slabs used in process creation are such, at least with the debug options I had), it assumed it's being called for stock and just retried without reclaiming.

Fix that by passing down a minsize argument in addition to the csize.

And what to do about that (csize == PAGE_SIZE && ret) retry? If it's needed at all (and presumably is since it's there, perhaps to handle races), then it should be extended to more than PAGE_SIZE, yet how far?

And should there be a retry count limit, of what? For now retry up to COSTLY_ORDER (as page_alloc.c does), stay safe with a cond_resched(), and make sure not to do it if __GFP_NORETRY.

Signed-off-by Suleiman Souhlal <[email protected]>

---
 mm/memcontrol.c |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)

mm/memcontrol.c +11 -7

index 4b94b2d..cbffc4c 100644
@@ -2187,7 +2187,8 @@ enum {
 };
 
 static int mem_cgroup_do_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,
-				unsigned int nr_pages, bool oom_check)
+				unsigned int nr_pages, unsigned int min_pages,
+				bool oom_check)
 {
 	unsigned long csize = nr_pages * PAGE_SIZE;
 	struct mem_cgroup *mem_over_limit;

@@ -2210,18 +2211,18 @@ static int mem_cgroup_do_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	} else
 		mem_over_limit = mem_cgroup_from_res_counter(fail_res, res);
 	/*
-	 * nr_pages can be either a huge page (HPAGE_PMD_NR), a batch
-	 * of regular pages (CHARGE_BATCH), or a single regular page (1).
-	 *
 	 * Never reclaim on behalf of optional batching, retry with a
 	 * single page instead.
 	 */
-	if (nr_pages == CHARGE_BATCH)
+	if (nr_pages > min_pages)
 		return CHARGE_RETRY;
 
 	if (!(gfp_mask & __GFP_WAIT))
 		return CHARGE_WOULDBLOCK;
 
+	if (gfp_mask & __GFP_NORETRY)
+		return CHARGE_NOMEM;
+
 	ret = mem_cgroup_reclaim(mem_over_limit, gfp_mask, flags);
 	if (mem_cgroup_margin(mem_over_limit) >= nr_pages)
 		return CHARGE_RETRY;

@@ -2234,8 +2235,10 @@ static int mem_cgroup_do_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	 * unlikely to succeed so close to the limit, and we fall back
 	 * to regular pages anyway in case of failure.
 	 */
-	if (nr_pages == 1 && ret)
+	if (nr_pages <= (PAGE_SIZE << PAGE_ALLOC_COSTLY_ORDER) && ret) {
+		cond_resched();
 		return CHARGE_RETRY;
+	}
 
 	/*
 	 * At task move, charge accounts can be doubly counted. So, it's

@@ -2369,7 +2372,8 @@ again:
 			nr_oom_retries = MEM_CGROUP_RECLAIM_RETRIES;
 		}
 
-		ret = mem_cgroup_do_charge(memcg, gfp_mask, batch, oom_check);
+		ret = mem_cgroup_do_charge(memcg, gfp_mask, batch, nr_pages,
+		    oom_check);
 		switch (ret) {
 		case CHARGE_OK:
 			break;

This is just a cleanup patch for clarity of expression. In earlier submissions, people asked it to be in a separate patch, so here it is.

Signed-off-by Glauber Costa <[email protected]>

CC Michal Hocko <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

Since we will succeed with the allocation no matter what, there isn't the need to use __must_check with it. It can very well be optional.

Signed-off-by Glauber Costa <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

CC Michal Hocko <[email protected]>

This patch adds the basic infrastructure for the accounting of the slab caches. To control that, the following files are created:

* memory.kmem.usage_in_bytes
* memory.kmem.limit_in_bytes
* memory.kmem.failcnt
* memory.kmem.max_usage_in_bytes

They have the same meaning of their user memory counterparts. They reflect the state of the "kmem" res_counter.

The code is not enabled until a limit is set. This can be tested by the flag "kmem_accounted". This means that after the patch is applied, no behavioral changes exists for whoever is still using memcg to control their memory usage.

We always account to both user and kernel resource_counters. This effectively means that an independent kernel limit is in place when the limit is set to a lower value than the user memory. A equal or higher value means that the user limit will always hit first, meaning that kmem is effectively unlimited.

People who want to track kernel memory but not limit it, can set this limit to a very high number (like RESOURCE_MAX - 1page - that no one will ever hit, or equal to the user memory)

Signed-off-by Glauber Costa <[email protected]>

CC Michal Hocko <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

Quoted 11 lines · Glauber Costa

This patch adds the basic infrastructure for the accounting of the slab caches. To control that, the following files are created:

* memory.kmem.usage_in_bytes
* memory.kmem.limit_in_bytes
* memory.kmem.failcnt
* memory.kmem.max_usage_in_bytes

They have the same meaning of their user memory counterparts. They reflect the state of the "kmem" res_counter.

Quoted 4 lines

The code is not enabled until a limit is set. This can be tested by the flag "kmem_accounted". This means that after the patch is applied, no behavioral changes exists for whoever is still using memcg to control their memory usage.

Hmm, res_counter never goes naeative ?

Quoted 18 lines · KAMEZAWA Hiroyuki

Quoted 15 lines · Glauber Costa

This patch adds the basic infrastructure for the accounting of the slab caches. To control that, the following files are created:

* memory.kmem.usage_in_bytes
* memory.kmem.limit_in_bytes
* memory.kmem.failcnt
* memory.kmem.max_usage_in_bytes

They have the same meaning of their user memory counterparts. They reflect the state of the "kmem" res_counter.

The code is not enabled until a limit is set. This can be tested by the flag "kmem_accounted". This means that after the patch is applied, no behavioral changes exists for whoever is still using memcg to control their memory usage.

Quoted 1 line

Hmm, res_counter never goes naeative ?

Why would it?

This one has more or less the same logic as the sock buffers.

If we are not accounted, the caches don't get created. If the caches don't get created, we don't release them. (this is modulo bugs, of course)

This patch adds the basic infrastructure for the accounting of the slab caches. To control that, the following files are created:

 * memory.kmem.usage_in_bytes
 * memory.kmem.limit_in_bytes
 * memory.kmem.failcnt
 * memory.kmem.max_usage_in_bytes

They have the same meaning of their user memory counterparts. They reflect the state of the "kmem" res_counter.

The code is not enabled until a limit is set. This can be tested by the flag "kmem_accounted". This means that after the patch is applied, no behavioral changes exists for whoever is still using memcg to control their memory usage.

We always account to both user and kernel resource_counters. This effectively means that an independent kernel limit is in place when the limit is set to a lower value than the user memory. A equal or higher value means that the user limit will always hit first, meaning that kmem is effectively unlimited.

People who want to track kernel memory but not limit it, can set this limit to a very high number (like RESOURCE_MAX - 1page - that no one will ever hit, or equal to the user memory)

Signed-off-by Glauber Costa <[email protected]>

CC Michal Hocko <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

---
 mm/memcontrol.c |   80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-

 1 files changed, 79 insertions(+), 1 deletions(-)

mm/memcontrol.c +10 -0

index 2810228..36f1e6b 100644
@@ -252,6 +252,10 @@ struct mem_cgroup {
        };
 
        /*
+        * the counter to account for kernel memory usage.
+        */
+       struct res_counter kmem;
+       /*
         * Per cgroup active and inactive list, similar to the
         * per zone LRU lists.
         */

@@ -266,6 +270,7 @@ struct mem_cgroup {
         * Should the accounting and control be hierarchical, per subtree?
         */
        bool use_hierarchy;
+       bool kmem_accounted;
 
        bool            oom_lock;
        atomic_t        under_oom;

@@ -378,6 +383,7 @@ enum res_type {
        _MEM,
        _MEMSWAP,
        _OOM_TYPE,
+       _KMEM,
  };
 
  #define MEMFILE_PRIVATE(x, val)        (((x) << 16) | (val))

@@ -1470,6 +1476,10 @@ done:
                res_counter_read_u64(&memcg->memsw, RES_USAGE) >> 10,
                res_counter_read_u64(&memcg->memsw, RES_LIMIT) >> 10,
                res_counter_read_u64(&memcg->memsw, RES_FAILCNT));
+       printk(KERN_INFO "kmem: usage %llukB, limit %llukB, failcnt %llu\n",
+               res_counter_read_u64(&memcg->kmem, RES_USAGE) >> 10,
+               res_counter_read_u64(&memcg->kmem, RES_LIMIT) >> 10,
+               res_counter_read_u64(&memcg->kmem, RES_FAILCNT));
  }
 
  /*

@@ -3914,6 +3924,11 @@ static ssize_t mem_cgroup_read(struct cgroup *cont, struct cftype *cft,

               else
                       val = res_counter_read_u64(&memcg->memsw, name); break;

mm/memcontrol.c +5 -0

+#ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+       case _KMEM:
+               val = res_counter_read_u64(&memcg->kmem, name);
+               break;
+#endif

       default:
               BUG();
       }

mm/memcontrol.c +19 -1

@@ -3951,8 +3966,26 @@ static int mem_cgroup_write(struct cgroup *cont, struct cftype *cft,
                        break;
                if (type == _MEM)
                        ret = mem_cgroup_resize_limit(memcg, val);
-               else
+               else if (type == _MEMSWAP)
                        ret = mem_cgroup_resize_memsw_limit(memcg, val);
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+               else if (type == _KMEM) {
+                       ret = res_counter_set_limit(&memcg->kmem, val);
+                       if (ret)
+                               break;
+                       /*
+                        * Once enabled, can't be disabled. We could in theory
+                        * disable it if we haven't yet created any caches, or
+                        * if we can shrink them all to death.
+                        *
+                        * But it is not worth the trouble
+                        */
+                       if (!memcg->kmem_accounted && val != RESOURCE_MAX)
+                               memcg->kmem_accounted = true;
+               }
+#endif
+               else
+                       return -EINVAL;
                break;

       case RES_SOFT_LIMIT:
               ret = res_counter_memparse_write_strategy(buffer, &val);

Quoted 23 lines

mm/memcontrol.c +16 -1

@@ -3951,8 +3966,26 @@ static int mem_cgroup_write(struct cgroup *cont, struct cftype *cft,
                        break;
                if (type == _MEM)
                        ret = mem_cgroup_resize_limit(memcg, val);
-               else
+               else if (type == _MEMSWAP)
                        ret = mem_cgroup_resize_memsw_limit(memcg, val);
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+               else if (type == _KMEM) {
+                       ret = res_counter_set_limit(&memcg->kmem, val);
+                       if (ret)
+                               break;
+                       /*
+                        * Once enabled, can't be disabled. We could in theory
+                        * disable it if we haven't yet created any caches, or
+                        * if we can shrink them all to death.
+                        *
+                        * But it is not worth the trouble
+                        */
+                       if (!memcg->kmem_accounted&&  val != RESOURCE_MAX)
+                               memcg->kmem_accounted = true;
+               }

mm/memcontrol.c static int mem_cgroup_write(struct cgroup *cont, struct cftype *cft,

Quoted 6 lines

mm/memcontrol.c +3 -0

+#endif
+               else
+                       return -EINVAL;
                break;
        case RES_SOFT_LIMIT:
                ret = res_counter_memparse_write_strategy(buffer,&val);

Why is RESOURCE_MAX special?

For the kmem slab controller, we need to record some extra information in the kmem_cache structure.

Signed-off-by Glauber Costa <[email protected]>

CC Christoph Lameter <[email protected]>

CC Pekka Enberg <[email protected]>

CC Michal Hocko <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

CC Suleiman Souhlal <[email protected]>

---
 include/linux/slab.h     |   15 +++++++++++++++
 include/linux/slab_def.h |    4 ++++
 include/linux/slub_def.h |    3 +++

 3 files changed, 22 insertions(+), 0 deletions(-)

include/linux/slab.h +2 -0

index a595dce..a5127e1 100644
@@ -153,6 +153,21 @@ unsigned int kmem_cache_size(struct kmem_cache *);
  #define ARCH_SLAB_MINALIGN __alignof__(unsigned long long)
  #endif
 
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+struct mem_cgroup_cache_params {

mm/slub.c +2 -1

index 2652e7c..86e40cc 100644
@@ -32,6 +32,7 @@
 #include <linux/prefetch.h>
 
 #include <trace/events/kmem.h>
+#include <linux/memcontrol.h>
 
 /*
  * Lock order:

@@ -3880,7 +3881,7 @@ static int slab_unmergeable(struct kmem_cache *s)
 	return 0;
 }
 
-static struct kmem_cache *find_mergeable(size_t size,
+static struct kmem_cache *find_mergeable(struct mem_cgroup *memcg, size_t size,
 		size_t align, unsigned long flags, const char *name,
 		void (*ctor)(void *))
 {

mm/slub.c +6 -1

+			size_t align, unsigned long flags, void (*ctor)(void *))
 {
 	struct kmem_cache *s;
 
 	if (WARN_ON(!name))
 		return NULL;
 
+#ifndef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+	WARN_ON(memcg != NULL);
+#endif
+
 	down_write(&slub_lock);
-	s = find_mergeable(size, align, flags, name, ctor);
+	s = find_mergeable(memcg, size, align, flags, name, ctor);
 	if (s) {
 		s->refcount++;
 		/*

Quoted 25 lines · Glauber Costa

mm/slub.c +2 -1

index 2652e7c..86e40cc 100644
@@ -32,6 +32,7 @@
  #include<linux/prefetch.h>
 
  #include<trace/events/kmem.h>
+#include<linux/memcontrol.h>
 
  /*
   * Lock order:

@@ -3880,7 +3881,7 @@ static int slab_unmergeable(struct kmem_cache *s)
  	return 0;
  }
 
-static struct kmem_cache *find_mergeable(size_t size,
+static struct kmem_cache *find_mergeable(struct mem_cgroup *memcg, size_t size,
  		size_t align, unsigned long flags, const char *name,
  		void (*ctor)(void *))
  {

@@ -3916,21 +3917,29 @@ static struct kmem_cache *find_mergeable(size_t size,
  		if (s->size - size>= sizeof(void *))
  			continue;
 

mm/slub.c static struct kmem_cache *find_mergeable(size_t size, · line 3922

Quoted 3 lines

mm/slub.c +3 -0

+		if (memcg&&  s->memcg_params.memcg != memcg)
+			continue;
+

This probably won't build without CONFIG_CGROUP_MEM_RES_CTLR_KMEM ?

Quoted 6 lines

mm/slub.c +0 -0

  		return s;
  	}
  	return NULL;
  }
 

mm/slub.c static struct kmem_cache *find_mergeable(size_t size, · line 3929

Quoted 4 lines

mm/slub.c +2 -2

-struct kmem_cache *kmem_cache_create(const char *name, size_t size,
-		size_t align, unsigned long flags, void (*ctor)(void *))
+struct kmem_cache *
+kmem_cache_create_memcg(struct mem_cgroup *memcg, const char *name, size_t size,

Does that build without CONFIG_CGROUP_MEM_RES_CTLR ?

Quoted 17 lines

mm/slub.c +6 -1

+			size_t align, unsigned long flags, void (*ctor)(void *))
  {
  	struct kmem_cache *s;
 
  	if (WARN_ON(!name))
  		return NULL;
 
+#ifndef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+	WARN_ON(memcg != NULL);
+#endif
+
  	down_write(&slub_lock);
-	s = find_mergeable(size, align, flags, name, ctor);
+	s = find_mergeable(memcg, size, align, flags, name, ctor);
  	if (s) {
  		s->refcount++;

mm/slub.c struct kmem_cache *kmem_cache_create(const char *name, size_t size, · line 3966

Quoted 6 lines

mm/slub.c +1 -0

  		/*

@@ -3954,12 +3963,15 @@ struct kmem_cache *kmem_cache_create(const char *name, size_t size,
  				size, align, flags, ctor)) {
  			list_add(&s->list,&slab_caches);
  			up_write(&slub_lock);
+			mem_cgroup_register_cache(memcg, s);

How do you handle when the memcg cgroup gets destroyed?

Also that means only one
memcg cgroup can be accounted for a given slab cache?

What if that memcg cgroup has
children? Hmm, perhaps this is handled in a further patch in the series, I saw a patch title with "children" inside :)

Allow a memcg parameter to be passed during cache creation. The slub allocator will only merge caches that belong to the same memcg.

Default function is created as a wrapper, passing NULL to the memcg version. We only merge caches that belong to the same memcg.

Quoted 1 line

From the memcontrol.c side, 3 helper functions are created:

1) memcg_css_id: because slub needs a unique cache name for sysfs. Since this is visible, but not the canonical location for slab data, the cache name is not used, the css_id should suffice.

2) mem_cgroup_register_cache: is responsible for assigning a unique index to each cache, and other general purpose setup. The index is only assigned for the root caches. All others are assigned index == -1.

3) mem_cgroup_release_cache: can be called from the root cache destruction, and will release the index for other caches.

This index mechanism was developed by Suleiman Souhlal.

Signed-off-by Glauber Costa <[email protected]>

CC Christoph Lameter <[email protected]>

CC Pekka Enberg <penberg-bbCR+/[email protected]>

CC Michal Hocko <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

CC Suleiman Souhlal <[email protected]>

---
 include/linux/memcontrol.h |   14 ++++++++++++++
 include/linux/slab.h       |    6 ++++++
 mm/memcontrol.c            |   29 +++++++++++++++++++++++++++++
 mm/slub.c                  |   31 +++++++++++++++++++++++++++----
 4 files changed, 76 insertions(+), 4 deletions(-)

include/linux/memcontrol.h +14 -0

index f94efd2..99e14b9 100644
@@ -26,6 +26,7 @@ struct mem_cgroup;
 struct page_cgroup;
 struct page;
 struct mm_struct;
+struct kmem_cache;
 
 /* Stats that can be updated by kernel. */
 enum mem_cgroup_page_stat_item {

@@ -440,7 +441,20 @@ struct sock;
 #ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
 void sock_update_memcg(struct sock *sk);
 void sock_release_memcg(struct sock *sk);
+int memcg_css_id(struct mem_cgroup *memcg);
+void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+				      struct kmem_cache *s);
+void mem_cgroup_release_cache(struct kmem_cache *cachep);
 #else
+static inline void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+					     struct kmem_cache *s)
+{
+}
+
+static inline void mem_cgroup_release_cache(struct kmem_cache *cachep)
+{
+}
+
 static inline void sock_update_memcg(struct sock *sk)
 {
 }

include/linux/slab.h +0 -0

index a5127e1..c7a7e05 100644

include/linux/slab.h +0 -0

 #ifdef CONFIG_NUMA
 /*
  * kmalloc_node_track_caller is a special version of kmalloc_node that

mm/memcontrol.c +10 -0

index 36f1e6b..0015ed0 100644
@@ -323,6 +323,11 @@ struct mem_cgroup {
 #endif
 };
 
+int memcg_css_id(struct mem_cgroup *memcg)
+{
+	return css_id(&memcg->css);
+}
+
 /* Stuffs for move charges at task migration. */
 /*
  * Types of charges to be moved. "move_charge_at_immitgrate" is treated as a

@@ -461,6 +466,30 @@ struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg)
 }
 EXPORT_SYMBOL(tcp_proto_cgroup);
 #endif /* CONFIG_INET */
+
+/* Bitmap used for allocating the cache id numbers. */
+static DECLARE_BITMAP(cache_types, MAX_KMEM_CACHE_TYPES);
+
+void mem_cgroup_register_cache(struct mem_cgroup *memcg,

mm/memcontrol.c +9 -0

+	} else
+		INIT_LIST_HEAD(&cachep->memcg_params.destroyed_list);
+	cachep->memcg_params.id = id;
+}
+
+void mem_cgroup_release_cache(struct kmem_cache *cachep)
+{
+	__clear_bit(cachep->memcg_params.id, cache_types);
+}
 #endif /* CONFIG_CGROUP_MEM_RES_CTLR_KMEM */
 
 static void drain_all_stock_async(struct mem_cgroup *memcg);

mm/slub.c +7 -3

index 2652e7c..86e40cc 100644
@@ -32,6 +32,7 @@
 #include <linux/prefetch.h>
 
 #include <trace/events/kmem.h>
+#include <linux/memcontrol.h>
 
 /*
  * Lock order:

@@ -3880,7 +3881,7 @@ static int slab_unmergeable(struct kmem_cache *s)
 	return 0;
 }
 
-static struct kmem_cache *find_mergeable(size_t size,
+static struct kmem_cache *find_mergeable(struct mem_cgroup *memcg, size_t size,
 		size_t align, unsigned long flags, const char *name,
 		void (*ctor)(void *))
 {

@@ -3916,21 +3917,29 @@ static struct kmem_cache *find_mergeable(size_t size,
 		if (s->size - size >= sizeof(void *))
 			continue;
 
+		if (memcg && s->memcg_params.memcg != memcg)
+			continue;
+
 		return s;
 	}
 	return NULL;
 }
 
-struct kmem_cache *kmem_cache_create(const char *name, size_t size,
-		size_t align, unsigned long flags, void (*ctor)(void *))
+struct kmem_cache *
+kmem_cache_create_memcg(struct mem_cgroup *memcg, const char *name, size_t size,

Quoted 80 lines · Glauber Costa

Allow a memcg parameter to be passed during cache creation. The slub allocator will only merge caches that belong to the same memcg.

Default function is created as a wrapper, passing NULL to the memcg version. We only merge caches that belong to the same memcg.

Quoted 1 line

From the memcontrol.c side, 3 helper functions are created:

1) memcg_css_id: because slub needs a unique cache name for sysfs. Since this is visible, but not the canonical location for slab data, the cache name is not used, the css_id should suffice.

2) mem_cgroup_register_cache: is responsible for assigning a unique index to each cache, and other general purpose setup. The index is only assigned for the root caches. All others are assigned index == -1.

3) mem_cgroup_release_cache: can be called from the root cache destruction, and will release the index for other caches.

This index mechanism was developed by Suleiman Souhlal.

Signed-off-by Glauber Costa<[email protected]>

CC Christoph Lameter<[email protected]>

CC Pekka Enberg<[email protected]>

CC Michal Hocko<[email protected]>

CC Kamezawa Hiroyuki<[email protected]>

CC Johannes Weiner<[email protected]>

CC Suleiman Souhlal<[email protected]>

---
  include/linux/memcontrol.h |   14 ++++++++++++++
  include/linux/slab.h       |    6 ++++++
  mm/memcontrol.c            |   29 +++++++++++++++++++++++++++++
  mm/slub.c                  |   31 +++++++++++++++++++++++++++----
  4 files changed, 76 insertions(+), 4 deletions(-)

include/linux/memcontrol.h +14 -0

index f94efd2..99e14b9 100644
@@ -26,6 +26,7 @@ struct mem_cgroup;
  struct page_cgroup;
  struct page;
  struct mm_struct;
+struct kmem_cache;
 
  /* Stats that can be updated by kernel. */
  enum mem_cgroup_page_stat_item {

@@ -440,7 +441,20 @@ struct sock;
  #ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
  void sock_update_memcg(struct sock *sk);
  void sock_release_memcg(struct sock *sk);
+int memcg_css_id(struct mem_cgroup *memcg);
+void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+				      struct kmem_cache *s);
+void mem_cgroup_release_cache(struct kmem_cache *cachep);
  #else
+static inline void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+					     struct kmem_cache *s)
+{
+}
+
+static inline void mem_cgroup_release_cache(struct kmem_cache *cachep)
+{
+}
+
  static inline void sock_update_memcg(struct sock *sk)
  {
  }

include/linux/slab.h +0 -0

index a5127e1..c7a7e05 100644
@@ -321,6 +321,12 @@ extern void *__kmalloc_track_caller(size_t, gfp_t, unsigned long);
  	__kmalloc(size, flags)
  #endif /* DEBUG_SLAB */
 

include/linux/slab.h extern void *__kmalloc_track_caller(size_t, gfp_t, unsigned long); · line 329

Quoted 6 lines

include/linux/slab.h +6 -0

+#ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+#define MAX_KMEM_CACHE_TYPES 400
+#else
+#define MAX_KMEM_CACHE_TYPES 0
+#endif /* CONFIG_CGROUP_MEM_RES_CTLR_KMEM */
+

why 400 ?

Quoted 10 lines

include/linux/slab.h +10 -0

+/* Bitmap used for allocating the cache id numbers. */
+static DECLARE_BITMAP(cache_types, MAX_KMEM_CACHE_TYPES);
+
+void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+			       struct kmem_cache *cachep)
+{
+	int id = -1;
+
+	cachep->memcg_params.memcg = memcg;
+

include/linux/slab.h

Quoted 4 lines

include/linux/slab.h +4 -0

+	if (!memcg) {
+		id = find_first_zero_bit(cache_types, MAX_KMEM_CACHE_TYPES);
+		BUG_ON(id<  0 || id>= MAX_KMEM_CACHE_TYPES);
+		__set_bit(id, cache_types);

No lock here ? you need find_first_zero_bit_and_set_atomic() or some. Rather than that, I think you can use lib/idr.c::ida_simple_get().

Quoted 32 lines

include/linux/slab.h +7 -3

@@ -3880,7 +3881,7 @@ static int slab_unmergeable(struct kmem_cache *s)
  	return 0;
  }
 
-static struct kmem_cache *find_mergeable(size_t size,
+static struct kmem_cache *find_mergeable(struct mem_cgroup *memcg, size_t size,
  		size_t align, unsigned long flags, const char *name,
  		void (*ctor)(void *))
  {

@@ -3916,21 +3917,29 @@ static struct kmem_cache *find_mergeable(size_t size,
  		if (s->size - size>= sizeof(void *))
  			continue;
 
+		if (memcg&&  s->memcg_params.memcg != memcg)
+			continue;
+
  		return s;
  	}
  	return NULL;
  }
 
-struct kmem_cache *kmem_cache_create(const char *name, size_t size,
-		size_t align, unsigned long flags, void (*ctor)(void *))
+struct kmem_cache *
+kmem_cache_create_memcg(struct mem_cgroup *memcg, const char *name, size_t size,
+			size_t align, unsigned long flags, void (*ctor)(void *))
  {
  	struct kmem_cache *s;
 
  	if (WARN_ON(!name))
  		return NULL;
 

include/linux/slab.h static struct kmem_cache *find_mergeable(size_t size, · line 3939

Quoted 3 lines

include/linux/slab.h +3 -0

+#ifndef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+	WARN_ON(memcg != NULL);
+#endif

I'm sorry what's is this warning for ?

include/linux/slab.h static char *create_unique_id(struct kmem_cache *s) · line 5293

Quoted 13 lines

include/linux/slab.h +5 -0

@@ -5265,6 +5283,11 @@ static char *create_unique_id(struct kmem_cache *s)
  	if (p != name + 1)
  		*p++ = '-';
  	p += sprintf(p, "%07d", s->size);
+
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+	if (s->memcg_params.memcg)
+		p += sprintf(p, "-%08d", memcg_css_id(s->memcg_params.memcg));
+#endif
  	BUG_ON(p>  name + ID_STR_LENGTH - 1);
  	return name;
  }

So, you use 'id' in user interface. Should we provide 'id' as memory.id file ?

Allow a memcg parameter to be passed during cache creation. The slub allocator will only merge caches that belong to the same memcg.

Default function is created as a wrapper, passing NULL to the memcg version. We only merge caches that belong to the same memcg.

From the memcontrol.c side, 3 helper functions are created:

 1) memcg_css_id: because slub needs a unique cache name for sysfs. Since this is visible, but not the canonical location for slab data, the cache name is not used, the css_id should suffice.

 2) mem_cgroup_register_cache: is responsible for assigning a unique index to each cache, and other general purpose setup. The index is only assigned for the root caches. All others are assigned index == -1.

 3) mem_cgroup_release_cache: can be called from the root cache destruction, and will release the index for other caches.

This index mechanism was developed by Suleiman Souhlal.

Signed-off-by Glauber Costa <[email protected]>

CC Christoph Lameter <[email protected]>

CC Pekka Enberg <[email protected]>

CC Michal Hocko <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

CC Suleiman Souhlal <[email protected]>

---
 include/linux/memcontrol.h |   14 ++++++++++++++ include/linux/slab.h       |    6 ++++++ mm/memcontrol.c            |   29 +++++++++++++++++++++++++++++ mm/slub.c                  |   31 +++++++++++++++++++++++++++----

 4 files changed, 76 insertions(+), 4 deletions(-)

include/linux/memcontrol.h +14 -0

index f94efd2..99e14b9 100644
@@ -26,6 +26,7 @@ struct mem_cgroup;
  struct page_cgroup;
  struct page;
  struct mm_struct;
+struct kmem_cache;
 
  /* Stats that can be updated by kernel. */
  enum mem_cgroup_page_stat_item {

@@ -440,7 +441,20 @@ struct sock;
  #ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
  void sock_update_memcg(struct sock *sk);
  void sock_release_memcg(struct sock *sk);
+int memcg_css_id(struct mem_cgroup *memcg);
+void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+                                     struct kmem_cache *s);
+void mem_cgroup_release_cache(struct kmem_cache *cachep);
  #else
+static inline void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+                                            struct kmem_cache *s)
+{
+}
+
+static inline void mem_cgroup_release_cache(struct kmem_cache *cachep)
+{
+}
+
  static inline void sock_update_memcg(struct sock *sk)
  {
  }

include/linux/slab.h +6 -0

index a5127e1..c7a7e05 100644
@@ -321,6 +321,12 @@ extern void *__kmalloc_track_caller(size_t, gfp_t, unsigned long);
        __kmalloc(size, flags)
  #endif /* DEBUG_SLAB */
 
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+#define MAX_KMEM_CACHE_TYPES 400
+#else
+#define MAX_KMEM_CACHE_TYPES 0
+#endif /* CONFIG_CGROUP_MEM_RES_CTLR_KMEM */
+
  #ifdef CONFIG_NUMA
  /*
  * kmalloc_node_track_caller is a special version of kmalloc_node that

mm/memcontrol.c +29 -0

index 36f1e6b..0015ed0 100644
@@ -323,6 +323,11 @@ struct mem_cgroup {
  #endif
  };
 
+int memcg_css_id(struct mem_cgroup *memcg)
+{
+       return css_id(&memcg->css);
+}
+
  /* Stuffs for move charges at task migration. */
  /*
  * Types of charges to be moved. "move_charge_at_immitgrate" is treated as a

@@ -461,6 +466,30 @@ struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg)
  }
  EXPORT_SYMBOL(tcp_proto_cgroup);
  #endif /* CONFIG_INET */
+
+/* Bitmap used for allocating the cache id numbers. */
+static DECLARE_BITMAP(cache_types, MAX_KMEM_CACHE_TYPES);
+
+void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+                              struct kmem_cache *cachep)
+{
+       int id = -1;
+
+       cachep->memcg_params.memcg = memcg;
+
+       if (!memcg) {
+               id = find_first_zero_bit(cache_types, MAX_KMEM_CACHE_TYPES);
+               BUG_ON(id < 0 || id >= MAX_KMEM_CACHE_TYPES);
+               __set_bit(id, cache_types);
+       } else
+               INIT_LIST_HEAD(&cachep->memcg_params.destroyed_list);
+       cachep->memcg_params.id = id;
+}
+
+void mem_cgroup_release_cache(struct kmem_cache *cachep)
+{
+       __clear_bit(cachep->memcg_params.id, cache_types);
+}
  #endif /* CONFIG_CGROUP_MEM_RES_CTLR_KMEM */
 
  static void drain_all_stock_async(struct mem_cgroup *memcg);

mm/slub.c +13 -4

index 2652e7c..86e40cc 100644
@@ -32,6 +32,7 @@
  #include <linux/prefetch.h>
 
  #include <trace/events/kmem.h>
+#include <linux/memcontrol.h>
 
  /*
  * Lock order:

@@ -3880,7 +3881,7 @@ static int slab_unmergeable(struct kmem_cache *s)
        return 0;
  }
 
-static struct kmem_cache *find_mergeable(size_t size,
+static struct kmem_cache *find_mergeable(struct mem_cgroup *memcg, size_t size,
                size_t align, unsigned long flags, const char *name,
                void (*ctor)(void *))
  {

@@ -3916,21 +3917,29 @@ static struct kmem_cache *find_mergeable(size_t size,
                if (s->size - size >= sizeof(void *))
                        continue;
 
+               if (memcg && s->memcg_params.memcg != memcg)
+                       continue;
+
                return s;
        }
        return NULL;
  }
 
-struct kmem_cache *kmem_cache_create(const char *name, size_t size,
-               size_t align, unsigned long flags, void (*ctor)(void *))
+struct kmem_cache *
+kmem_cache_create_memcg(struct mem_cgroup *memcg, const char *name, size_t size,
+                       size_t align, unsigned long flags, void (*ctor)(void *))
  {
        struct kmem_cache *s;
 
        if (WARN_ON(!name))
                return NULL;
 
+#ifndef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+       WARN_ON(memcg != NULL);
+#endif
+
        down_write(&slub_lock);
-       s = find_mergeable(size, align, flags, name, ctor);
+       s = find_mergeable(memcg, size, align, flags, name, ctor);
        if (s) {
                s->refcount++;
                /*

Quoted 189 lines · Glauber Costa

Allow a memcg parameter to be passed during cache creation. The slub allocator will only merge caches that belong to the same memcg.

Default function is created as a wrapper, passing NULL to the memcg version. We only merge caches that belong to the same memcg.

From the memcontrol.c side, 3 helper functions are created:

1) memcg_css_id: because slub needs a unique cache name for sysfs. Since this is visible, but not the canonical location for slab data, the cache name is not used, the css_id should suffice.

2) mem_cgroup_register_cache: is responsible for assigning a unique index to each cache, and other general purpose setup. The index is only assigned for the root caches. All others are assigned index == -1.

3) mem_cgroup_release_cache: can be called from the root cache destruction, and will release the index for other caches.

This index mechanism was developed by Suleiman Souhlal.

Signed-off-by Glauber Costa<[email protected]>

CC Christoph Lameter<[email protected]>

CC Pekka Enberg<[email protected]>

CC Michal Hocko<[email protected]>

CC Kamezawa Hiroyuki<[email protected]>

CC Johannes Weiner<[email protected]>

CC Suleiman Souhlal<[email protected]>

---
  include/linux/memcontrol.h |   14 ++++++++++++++
  include/linux/slab.h       |    6 ++++++
  mm/memcontrol.c            |   29 +++++++++++++++++++++++++++++
  mm/slub.c                  |   31 +++++++++++++++++++++++++++----
  4 files changed, 76 insertions(+), 4 deletions(-)

include/linux/memcontrol.h +14 -0

index f94efd2..99e14b9 100644
@@ -26,6 +26,7 @@ struct mem_cgroup;
  struct page_cgroup;
  struct page;
  struct mm_struct;
+struct kmem_cache;
 
  /* Stats that can be updated by kernel. */
  enum mem_cgroup_page_stat_item {

@@ -440,7 +441,20 @@ struct sock;
  #ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
  void sock_update_memcg(struct sock *sk);
  void sock_release_memcg(struct sock *sk);
+int memcg_css_id(struct mem_cgroup *memcg);
+void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+                                     struct kmem_cache *s);
+void mem_cgroup_release_cache(struct kmem_cache *cachep);
  #else
+static inline void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+                                            struct kmem_cache *s)
+{
+}
+
+static inline void mem_cgroup_release_cache(struct kmem_cache *cachep)
+{
+}
+
  static inline void sock_update_memcg(struct sock *sk)
  {
  }

include/linux/slab.h +6 -0

index a5127e1..c7a7e05 100644
@@ -321,6 +321,12 @@ extern void *__kmalloc_track_caller(size_t, gfp_t, unsigned long);
        __kmalloc(size, flags)
  #endif /* DEBUG_SLAB */
 
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+#define MAX_KMEM_CACHE_TYPES 400
+#else
+#define MAX_KMEM_CACHE_TYPES 0
+#endif /* CONFIG_CGROUP_MEM_RES_CTLR_KMEM */
+
  #ifdef CONFIG_NUMA
  /*
  * kmalloc_node_track_caller is a special version of kmalloc_node that

mm/memcontrol.c +29 -0

index 36f1e6b..0015ed0 100644
@@ -323,6 +323,11 @@ struct mem_cgroup {
  #endif
  };
 
+int memcg_css_id(struct mem_cgroup *memcg)
+{
+       return css_id(&memcg->css);
+}
+
  /* Stuffs for move charges at task migration. */
  /*
  * Types of charges to be moved. "move_charge_at_immitgrate" is treated as a

@@ -461,6 +466,30 @@ struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg)
  }
  EXPORT_SYMBOL(tcp_proto_cgroup);
  #endif /* CONFIG_INET */
+
+/* Bitmap used for allocating the cache id numbers. */
+static DECLARE_BITMAP(cache_types, MAX_KMEM_CACHE_TYPES);
+
+void mem_cgroup_register_cache(struct mem_cgroup *memcg,
+                              struct kmem_cache *cachep)
+{
+       int id = -1;
+
+       cachep->memcg_params.memcg = memcg;
+
+       if (!memcg) {
+               id = find_first_zero_bit(cache_types, MAX_KMEM_CACHE_TYPES);
+               BUG_ON(id<  0 || id>= MAX_KMEM_CACHE_TYPES);
+               __set_bit(id, cache_types);
+       } else
+               INIT_LIST_HEAD(&cachep->memcg_params.destroyed_list);
+       cachep->memcg_params.id = id;
+}
+
+void mem_cgroup_release_cache(struct kmem_cache *cachep)
+{
+       __clear_bit(cachep->memcg_params.id, cache_types);
+}
  #endif /* CONFIG_CGROUP_MEM_RES_CTLR_KMEM */
 
  static void drain_all_stock_async(struct mem_cgroup *memcg);

mm/slub.c +13 -4

index 2652e7c..86e40cc 100644
@@ -32,6 +32,7 @@
  #include<linux/prefetch.h>
 
  #include<trace/events/kmem.h>
+#include<linux/memcontrol.h>
 
  /*
  * Lock order:

@@ -3880,7 +3881,7 @@ static int slab_unmergeable(struct kmem_cache *s)
        return 0;
  }
 
-static struct kmem_cache *find_mergeable(size_t size,
+static struct kmem_cache *find_mergeable(struct mem_cgroup *memcg, size_t size,
                size_t align, unsigned long flags, const char *name,
                void (*ctor)(void *))
  {

@@ -3916,21 +3917,29 @@ static struct kmem_cache *find_mergeable(size_t size,
                if (s->size - size>= sizeof(void *))
                        continue;
 
+               if (memcg&&  s->memcg_params.memcg != memcg)
+                       continue;
+
                return s;
        }
        return NULL;
  }
 
-struct kmem_cache *kmem_cache_create(const char *name, size_t size,
-               size_t align, unsigned long flags, void (*ctor)(void *))
+struct kmem_cache *
+kmem_cache_create_memcg(struct mem_cgroup *memcg, const char *name, size_t size,
+                       size_t align, unsigned long flags, void (*ctor)(void *))
  {
        struct kmem_cache *s;
 
        if (WARN_ON(!name))
                return NULL;
 
+#ifndef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+       WARN_ON(memcg != NULL);
+#endif
+
        down_write(&slub_lock);
-       s = find_mergeable(size, align, flags, name, ctor);
+       s = find_mergeable(memcg, size, align, flags, name, ctor);
        if (s) {
                s->refcount++;

mm/slub.c struct kmem_cache *kmem_cache_create(const char *name, size_t size, · line 3966

Quoted 6 lines

mm/slub.c +1 -0

                /*

@@ -3954,12 +3963,15 @@ struct kmem_cache *kmem_cache_create(const char *name, size_t size,
                                size, align, flags, ctor)) {
                        list_add(&s->list,&slab_caches);
                        up_write(&slub_lock);
+                       mem_cgroup_register_cache(memcg, s);

Do the kmalloc caches get their id registered correctly?

Allow a memcg parameter to be passed during cache creation.

Default function is created as a wrapper, passing NULL to the memcg version. We only merge caches that belong to the same memcg.

This code was mostly written by Suleiman Souhlal and only adapted to my patchset, plus a couple of simplifications

Signed-off-by Glauber Costa <[email protected]>

CC Christoph Lameter <[email protected]>

CC Pekka Enberg <[email protected]>

CC Michal Hocko <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

CC Suleiman Souhlal <[email protected]>

---
 mm/slab.c |   38 +++++++++++++++++++++++++++++---------

 1 files changed, 29 insertions(+), 9 deletions(-)

mm/slab.c +6 -6

index a0d51dd..362bb6e 100644
@@ -2287,14 +2287,15 @@ static int __init_refok setup_cpu_cache(struct kmem_cache *cachep, gfp_t gfp)
  * cacheline.  This can be beneficial if you're counting cycles as closely
  * as davem.
  */
-struct kmem_cache *
-kmem_cache_create (const char *name, size_t size, size_t align,
-       unsigned long flags, void (*ctor)(void *))
+static struct kmem_cache *
+__kmem_cache_create(struct mem_cgroup *memcg, const char *name, size_t size,
+                   size_t align, unsigned long flags, void (*ctor)(void *))
  {
-       size_t left_over, slab_size, ralign;
+       size_t left_over, orig_align, ralign, slab_size;
        struct kmem_cache *cachep = NULL, *pc;
        gfp_t gfp;
 
+       orig_align = align;
        /*
         * Sanity checks... these are all serious usage bugs.
         */

@@ -2311,7 +2312,6 @@ kmem_cache_create (const char *name, size_t size, size_t align,
         */
        if (slab_is_available()) {
                get_online_cpus();
-               mutex_lock(&cache_chain_mutex);
        }
 
        list_for_each_entry(pc, &cache_chain, next) {

@@ -2331,9 +2331,9 @@ kmem_cache_create (const char *name, size_t size, size_t align,
                        continue;
                }
 
-               if (!strcmp(pc->name, name)) {
+               if (!strcmp(pc->name, name) && !memcg) {

                       printk(KERN_ERR
-                              "kmem_cache_create: duplicate cache %s\n", name);
+                       "kmem_cache_create: duplicate cache %s\n", name); dump_stack();
                       goto oops;
               }

mm/slab.c +3 -0

@@ -2434,6 +2434,9 @@ kmem_cache_create (const char *name, size_t size, size_t align,
        cachep->nodelists = (struct kmem_list3 **)&cachep->array[nr_cpu_ids];
 
        set_obj_size(cachep, size);
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+       cachep->memcg_params.orig_align = orig_align;
+#endif
  #if DEBUG
 
        /*

This patch provides kmem_cache_dup(), that duplicates a cache for a memcg, preserving its creation properties. Object size, alignment and flags are all respected.

When a duplicate cache is created, the parent cache cannot be destructed during the child lifetime. To assure this, its reference count is increased if the cache creation succeeds.

Signed-off-by Glauber Costa <[email protected]>

CC Christoph Lameter <[email protected]>

CC Pekka Enberg <penberg-bbCR+/[email protected]>

CC Michal Hocko <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

CC Suleiman Souhlal <[email protected]>

---
 include/linux/memcontrol.h |    3 +++
 include/linux/slab.h       |    3 +++
 mm/memcontrol.c            |   44 ++++++++++++++++++++++++++++++++++++++++++++
 mm/slub.c                  |   37 +++++++++++++++++++++++++++++++++++++
 4 files changed, 87 insertions(+), 0 deletions(-)

include/linux/memcontrol.h +3 -0

index 99e14b9..493ecdd 100644
@@ -445,6 +445,9 @@ int memcg_css_id(struct mem_cgroup *memcg);
 void mem_cgroup_register_cache(struct mem_cgroup *memcg,
 				      struct kmem_cache *s);
 void mem_cgroup_release_cache(struct kmem_cache *cachep);
+extern char *mem_cgroup_cache_name(struct mem_cgroup *memcg,
+				   struct kmem_cache *cachep);
+
 #else
 static inline void mem_cgroup_register_cache(struct mem_cgroup *memcg,
 					     struct kmem_cache *s)

include/linux/slab.h +3 -0

index c7a7e05..909b508 100644
@@ -323,6 +323,9 @@ extern void *__kmalloc_track_caller(size_t, gfp_t, unsigned long);
 
 #ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
 #define MAX_KMEM_CACHE_TYPES 400
+extern struct kmem_cache *kmem_cache_dup(struct mem_cgroup *memcg,
+					 struct kmem_cache *cachep);
+void kmem_cache_drop_ref(struct kmem_cache *cachep);
 #else
 #define MAX_KMEM_CACHE_TYPES 0
 #endif /* CONFIG_CGROUP_MEM_RES_CTLR_KMEM */

mm/memcontrol.c +7 -0

index 0015ed0..e881d83 100644
@@ -467,6 +467,50 @@ struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg)
 EXPORT_SYMBOL(tcp_proto_cgroup);
 #endif /* CONFIG_INET */
 
+/*
+ * This is to prevent races againt the kmalloc cache creations.
+ * Should never be used outside the core memcg code. Therefore,
+ * copy it here, instead of letting it in lib/
+ */
+static char *kasprintf_no_account(gfp_t gfp, const char *fmt, ...)
+{

mm/memcontrol.c +14 -0

+	if (!p)
+		goto out;
+
+	vsnprintf(p, len+1, fmt, ap);
+
+out:
+	va_end(ap);
+	return p;
+}
+
+char *mem_cgroup_cache_name(struct mem_cgroup *memcg, struct kmem_cache *cachep)
+{
+	char *name;
+	struct dentry *dentry = memcg->css.cgroup->dentry;

mm/memcontrol.c +3 -0

+	return name;
+}
+
 /* Bitmap used for allocating the cache id numbers. */
 static DECLARE_BITMAP(cache_types, MAX_KMEM_CACHE_TYPES);
 

mm/slub.c +37 -0

index 86e40cc..2285a96 100644
@@ -3993,6 +3993,43 @@ struct kmem_cache *kmem_cache_create(const char *name, size_t size,
 }
 EXPORT_SYMBOL(kmem_cache_create);
 
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
+struct kmem_cache *kmem_cache_dup(struct mem_cgroup *memcg,
+				  struct kmem_cache *s)
+{
+	char *name;
+	struct kmem_cache *new;
+
+	name = mem_cgroup_cache_name(memcg, s);
+	if (!name)
+		return NULL;
+
+	new = kmem_cache_create_memcg(memcg, name, s->objsize, s->align,
+				      s->allocflags, s->ctor);
+
+	/*
+	 * We increase the reference counter in the parent cache, to
+	 * prevent it from being deleted. If kmem_cache_destroy() is
+	 * called for the root cache before we call it for a child cache,
+	 * it will be queued for destruction when we finally drop the
+	 * reference on the child cache.
+	 */
+	if (new) {
+		down_write(&slub_lock);
+		s->refcount++;
+		up_write(&slub_lock);
+	}
+
+	return new;
+}
+
+void kmem_cache_drop_ref(struct kmem_cache *s)
+{
+	BUG_ON(s->memcg_params.id != -1);
+	kmem_cache_destroy(s);
+}
+#endif
+
 #ifdef CONFIG_SMP
 /*
  * Use the cpu notifier to insure that the cpu slabs are flushed when

Signature · 7 lines

1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Quoted 71 lines · Glauber Costa

This patch provides kmem_cache_dup(), that duplicates a cache for a memcg, preserving its creation properties. Object size, alignment and flags are all respected.

When a duplicate cache is created, the parent cache cannot be destructed during the child lifetime. To assure this, its reference count is increased if the cache creation succeeds.

Signed-off-by Glauber Costa<[email protected]>

CC Christoph Lameter<[email protected]>

CC Pekka Enberg<[email protected]>

CC Michal Hocko<[email protected]>

CC Kamezawa Hiroyuki<[email protected]>

CC Johannes Weiner<[email protected]>

CC Suleiman Souhlal<[email protected]>

---
  include/linux/memcontrol.h |    3 +++
  include/linux/slab.h       |    3 +++
  mm/memcontrol.c            |   44 ++++++++++++++++++++++++++++++++++++++++++++
  mm/slub.c                  |   37 +++++++++++++++++++++++++++++++++++++
  4 files changed, 87 insertions(+), 0 deletions(-)

include/linux/memcontrol.h +3 -0

index 99e14b9..493ecdd 100644
@@ -445,6 +445,9 @@ int memcg_css_id(struct mem_cgroup *memcg);
  void mem_cgroup_register_cache(struct mem_cgroup *memcg,
  				      struct kmem_cache *s);
  void mem_cgroup_release_cache(struct kmem_cache *cachep);
+extern char *mem_cgroup_cache_name(struct mem_cgroup *memcg,
+				   struct kmem_cache *cachep);
+
  #else
  static inline void mem_cgroup_register_cache(struct mem_cgroup *memcg,
  					     struct kmem_cache *s)

include/linux/slab.h +3 -0

index c7a7e05..909b508 100644
@@ -323,6 +323,9 @@ extern void *__kmalloc_track_caller(size_t, gfp_t, unsigned long);
 
  #ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
  #define MAX_KMEM_CACHE_TYPES 400
+extern struct kmem_cache *kmem_cache_dup(struct mem_cgroup *memcg,
+					 struct kmem_cache *cachep);
+void kmem_cache_drop_ref(struct kmem_cache *cachep);
  #else
  #define MAX_KMEM_CACHE_TYPES 0
  #endif /* CONFIG_CGROUP_MEM_RES_CTLR_KMEM */

mm/memcontrol.c +12 -0

index 0015ed0..e881d83 100644
@@ -467,6 +467,50 @@ struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg)
  EXPORT_SYMBOL(tcp_proto_cgroup);
  #endif /* CONFIG_INET */
 
+/*
+ * This is to prevent races againt the kmalloc cache creations.
+ * Should never be used outside the core memcg code. Therefore,
+ * copy it here, instead of letting it in lib/
+ */
+static char *kasprintf_no_account(gfp_t gfp, const char *fmt, ...)
+{
+	unsigned int len;
+	char *p = NULL;
+	va_list ap, aq;
+
+	va_start(ap, fmt);

mm/memcontrol.c struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg) · line 486

Quoted 5 lines

mm/memcontrol.c +5 -0

+	va_copy(aq, ap);
+	len = vsnprintf(NULL, 0, fmt, aq);
+	va_end(aq);
+
+	p = kmalloc_no_account(len+1, gfp);

I can't seem to find kmalloc_no_account() in this patch or may be I missed it in a previous one?

Quoted 18 lines

mm/memcontrol.c +18 -0

+	if (!p)
+		goto out;
+
+	vsnprintf(p, len+1, fmt, ap);
+
+out:
+	va_end(ap);
+	return p;
+}
+
+char *mem_cgroup_cache_name(struct mem_cgroup *memcg, struct kmem_cache *cachep)
+{
+	char *name;
+	struct dentry *dentry = memcg->css.cgroup->dentry;
+
+	BUG_ON(dentry == NULL);
+
+	/* Preallocate the space for "dead" at the end */

mm/memcontrol.c struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg) · line 510

Quoted 6 lines

mm/memcontrol.c +6 -0

+	name = kasprintf_no_account(GFP_KERNEL, "%s(%d:%s)dead",
+	    cachep->name, css_id(&memcg->css), dentry->d_name.name);
+
+	if (name)
+		/* Remove "dead" */
+		name[strlen(name) - 4] = '\0';

Why this space for "dead" ?

Quoted 72 lines · Frederic Weisbecker

Quoted 71 lines · Glauber Costa

This patch provides kmem_cache_dup(), that duplicates a cache for a memcg, preserving its creation properties. Object size, alignment and flags are all respected.

When a duplicate cache is created, the parent cache cannot be destructed during the child lifetime. To assure this, its reference count is increased if the cache creation succeeds.

Signed-off-by Glauber Costa<[email protected]>

CC Christoph Lameter<[email protected]>

CC Pekka Enberg<penberg-bbCR+/[email protected]>

CC Michal Hocko<[email protected]>

CC Kamezawa Hiroyuki<[email protected]>

CC Johannes Weiner<[email protected]>

CC Suleiman Souhlal<[email protected]>

---
 include/linux/memcontrol.h |    3 +++
 include/linux/slab.h       |    3 +++
 mm/memcontrol.c            |   44 ++++++++++++++++++++++++++++++++++++++++++++
 mm/slub.c                  |   37 +++++++++++++++++++++++++++++++++++++
 4 files changed, 87 insertions(+), 0 deletions(-)

include/linux/memcontrol.h +3 -0

index 99e14b9..493ecdd 100644
@@ -445,6 +445,9 @@ int memcg_css_id(struct mem_cgroup *memcg);
 void mem_cgroup_register_cache(struct mem_cgroup *memcg,
 				      struct kmem_cache *s);
 void mem_cgroup_release_cache(struct kmem_cache *cachep);
+extern char *mem_cgroup_cache_name(struct mem_cgroup *memcg,
+				   struct kmem_cache *cachep);
+
 #else
 static inline void mem_cgroup_register_cache(struct mem_cgroup *memcg,
 					     struct kmem_cache *s)

include/linux/slab.h +3 -0

index c7a7e05..909b508 100644
@@ -323,6 +323,9 @@ extern void *__kmalloc_track_caller(size_t, gfp_t, unsigned long);
 
 #ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
 #define MAX_KMEM_CACHE_TYPES 400
+extern struct kmem_cache *kmem_cache_dup(struct mem_cgroup *memcg,
+					 struct kmem_cache *cachep);
+void kmem_cache_drop_ref(struct kmem_cache *cachep);
 #else
 #define MAX_KMEM_CACHE_TYPES 0
 #endif /* CONFIG_CGROUP_MEM_RES_CTLR_KMEM */

mm/memcontrol.c +12 -0

index 0015ed0..e881d83 100644
@@ -467,6 +467,50 @@ struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg)
 EXPORT_SYMBOL(tcp_proto_cgroup);
 #endif /* CONFIG_INET */
 
+/*
+ * This is to prevent races againt the kmalloc cache creations.
+ * Should never be used outside the core memcg code. Therefore,
+ * copy it here, instead of letting it in lib/
+ */
+static char *kasprintf_no_account(gfp_t gfp, const char *fmt, ...)
+{
+	unsigned int len;
+	char *p = NULL;
+	va_list ap, aq;
+
+	va_start(ap, fmt);

Quoted 8 lines

mm/memcontrol.c struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg) · line 486

Quoted 5 lines

mm/memcontrol.c +5 -0

+	va_copy(aq, ap);
+	len = vsnprintf(NULL, 0, fmt, aq);
+	va_end(aq);
+
+	p = kmalloc_no_account(len+1, gfp);

I can't seem to find kmalloc_no_account() in this patch or may be I missed it in a previous one?

It is in a previous one (actually two, one for the slab, one for the slub). They are bundled in the cache creation, but I could separate it
for clarity, if you prefer.

Quoted 20 lines

Quoted 20 lines

mm/memcontrol.c +18 -0

+	if (!p)
+		goto out;
+
+	vsnprintf(p, len+1, fmt, ap);
+
+out:
+	va_end(ap);
+	return p;
+}
+
+char *mem_cgroup_cache_name(struct mem_cgroup *memcg, struct kmem_cache *cachep)
+{
+	char *name;
+	struct dentry *dentry = memcg->css.cgroup->dentry;
+
+	BUG_ON(dentry == NULL);
+
+	/* Preallocate the space for "dead" at the end */

Quoted 8 lines

mm/memcontrol.c struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg) · line 510

Quoted 6 lines

mm/memcontrol.c +6 -0

+	name = kasprintf_no_account(GFP_KERNEL, "%s(%d:%s)dead",
+	    cachep->name, css_id(&memcg->css), dentry->d_name.name);
+
+	if (name)
+		/* Remove "dead" */
+		name[strlen(name) - 4] = '\0';

Why this space for "dead" ?

Ok, sorry. Since I didn't include the destruction part, it got too easy for whoever wasn't following the last discussion on this to get lost - My bad. So here it is:

When we destroy the memcg, some objects may still hold the cache in memory. It is like a reference count, in a sense, which each object being a reference.

In typical cases, like non-shrinkable caches that has create - destroy patterns, the caches will go away as soon as the tasks using them.

But in cache-like structure like the dentry cache, the objects may hang around until a shrinker pass takes them out. And even then, some of them will live on.

In this case, we will display them with "dead" in the name.

We could hide them, but then it gets weirder because it would be hard to understand where is your used memory when you need to inspect your system.

Creating another file, slabinfo_deadcaches, and keeping the names, is also a possibility, if people think that the string append is way too ugly.

This patch provides kmem_cache_dup(), that duplicates a cache for a memcg, preserving its creation properties. Object size, alignment and flags are all respected.

When a duplicate cache is created, the parent cache cannot be destructed during the child lifetime. To assure this, its reference count is increased if the cache creation succeeds.

Signed-off-by Glauber Costa <[email protected]>

CC Christoph Lameter <[email protected]>

CC Pekka Enberg <[email protected]>

CC Michal Hocko <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

CC Suleiman Souhlal <[email protected]>

---
 include/linux/memcontrol.h |    3 +++
 include/linux/slab.h       |    3 +++
 mm/memcontrol.c            |   44 ++++++++++++++++++++++++++++++++++++++++++++ mm/slub.c                  |   37 +++++++++++++++++++++++++++++++++++++

 4 files changed, 87 insertions(+), 0 deletions(-)

include/linux/memcontrol.h +3 -0

index 99e14b9..493ecdd 100644
@@ -445,6 +445,9 @@ int memcg_css_id(struct mem_cgroup *memcg);
  void mem_cgroup_register_cache(struct mem_cgroup *memcg,
                                      struct kmem_cache *s);
  void mem_cgroup_release_cache(struct kmem_cache *cachep);
+extern char *mem_cgroup_cache_name(struct mem_cgroup *memcg,
+                                  struct kmem_cache *cachep);
+
  #else
  static inline void mem_cgroup_register_cache(struct mem_cgroup *memcg,
                                             struct kmem_cache *s)

include/linux/slab.h +3 -0

index c7a7e05..909b508 100644
@@ -323,6 +323,9 @@ extern void *__kmalloc_track_caller(size_t, gfp_t, unsigned long);
 
  #ifdef CONFIG_CGROUP_MEM_RES_CTLR_KMEM
  #define MAX_KMEM_CACHE_TYPES 400
+extern struct kmem_cache *kmem_cache_dup(struct mem_cgroup *memcg,
+                                        struct kmem_cache *cachep);
+void kmem_cache_drop_ref(struct kmem_cache *cachep);
  #else
  #define MAX_KMEM_CACHE_TYPES 0
  #endif /* CONFIG_CGROUP_MEM_RES_CTLR_KMEM */

mm/memcontrol.c +21 -0

index 0015ed0..e881d83 100644
@@ -467,6 +467,50 @@ struct cg_proto *tcp_proto_cgroup(struct mem_cgroup *memcg)
  EXPORT_SYMBOL(tcp_proto_cgroup);
  #endif /* CONFIG_INET */
 
+/*
+ * This is to prevent races againt the kmalloc cache creations.
+ * Should never be used outside the core memcg code. Therefore,
+ * copy it here, instead of letting it in lib/
+ */
+static char *kasprintf_no_account(gfp_t gfp, const char *fmt, ...)
+{
+       unsigned int len;
+       char *p = NULL;
+       va_list ap, aq;
+
+       va_start(ap, fmt);
+       va_copy(aq, ap);
+       len = vsnprintf(NULL, 0, fmt, aq);
+       va_end(aq);
+
+       p = kmalloc_no_account(len+1, gfp);
+       if (!p)
+               goto out;
+
+       vsnprintf(p, len+1, fmt, ap);

Some allocations need to be accounted to the root memcg regardless of their context. One trivial example, is the allocations we do during the memcg slab cache creation themselves. Strictly speaking, they could go to the parent, but it is way easier to bill them to the root cgroup.

Only generic kmalloc allocations are allowed to be bypassed.

The function is not exported, because drivers code should always be accounted.

This code is mosly written by Suleiman Souhlal.

Signed-off-by Glauber Costa <[email protected]>

CC Christoph Lameter <[email protected]>

CC Pekka Enberg <[email protected]>

CC Michal Hocko <[email protected]>

CC Kamezawa Hiroyuki <[email protected]>

CC Johannes Weiner <[email protected]>

CC Suleiman Souhlal <[email protected]>

---
 include/linux/slab_def.h |    1 +
 mm/slab.c                |   23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+), 0 deletions(-)

include/linux/slab_def.h +1 -0

index 06e4a3e..54d25d7 100644
@@ -114,6 +114,7 @@ extern struct cache_sizes malloc_sizes[];
 
 void *kmem_cache_alloc(struct kmem_cache *, gfp_t);
 void *__kmalloc(size_t size, gfp_t flags);
+void *kmalloc_no_account(size_t size, gfp_t flags);
 
 #ifdef CONFIG_TRACING
 extern void *kmem_cache_alloc_trace(size_t size,

mm/slab.c +16 -0

index c4ef684..13948c3 100644
@@ -3960,6 +3960,29 @@ void *__kmalloc(size_t size, gfp_t flags)
 }
 EXPORT_SYMBOL(__kmalloc);
 
+static __always_inline void *__do_kmalloc_no_account(size_t size, gfp_t flags,
+						     void *caller)
+{
+	struct kmem_cache *cachep;
+	void *ret;
+
+	cachep = __find_general_cachep(size, flags);
+	if (unlikely(ZERO_OR_NULL_PTR(cachep)))
+		return cachep;
+
+	ret = __cache_alloc(cachep, flags, caller);
+	trace_kmalloc((unsigned long)caller, ret, size,
+		      cachep->buffer_size, flags);
+
+	return ret;
+}