kmemcg shrinkers

Using destroy_super() in alloc_super() fail path is bad, because:

* It will trigger WARN_ON(!list_empty(&s->s_mounts)) since s_mounts is initialized after several 'goto fail's.

* It will call kfree_rcu() to free the super block although kfree() is obviously enough there.
* The list_lru structure was initially implemented without the ability to destroy an uninitialized object in mind.

I'm going to replace the conventional list_lru with per-memcg lru to implement per-memcg slab reclaim. This new structure will fail destruction of objects that haven't been properly initialized so let's inline appropriate snippets from destroy_super() to alloc_super() fail path instead of using the whole function there.

Quoted 4 lines · Vladimir Davydov

Using destroy_super() in alloc_super() fail path is bad, because:

* It will trigger WARN_ON(!list_empty(&s->s_mounts)) since s_mounts is initialized after several 'goto fail's.

So let's fix that.

Quoted 10 lines

* It will call kfree_rcu() to free the super block although kfree() is obviously enough there.
* The list_lru structure was initially implemented without the ability to destroy an uninitialized object in mind.

I'm going to replace the conventional list_lru with per-memcg lru to implement per-memcg slab reclaim. This new structure will fail destruction of objects that haven't been properly initialized so let's inline appropriate snippets from destroy_super() to alloc_super() fail path instead of using the whole function there.

You're basically undoing the change made in commit 7eb5e88 ("uninline destroy_super(), consolidate alloc_super()") which was done less than a month ago. :/

The code as it stands works just fine - the list-lru structures in the superblock are actually initialised (to zeros) - and so calling list_lru_destroy() on it works just fine in that state as the pointers that are freed are NULL. Yes, unexpected, but perfectly valid code.

I haven't looked at the internals of the list_lru changes you've made yet, but it surprises me that we can't handle this case internally to list_lru_destroy().

Actually, I'm not going to modify the list_lru structure, because I think it's good as it is. I'd like to substitute it with a new structure, memcg_list_lru, only in those places where this functionality (per-memcg scanning) is really needed. This new structure would look like this:

struct memcg_list_lru {
    struct list_lru global_lru;
    struct list_lru **memcg_lrus;
    struct list_head list;
    void *old_lrus;
}

Since old_lrus and memcg_lrus can be NULL under normal operation, in memcg_list_lru_destroy() I'd have to check either the list or the global_lru field, i.e. it would look like:

if (!list.next)

    /* has not been initialized */
    return;

or

Quoted 17 lines · Vladimir Davydov

Actually, I'm not going to modify the list_lru structure, because I think it's good as it is. I'd like to substitute it with a new structure, memcg_list_lru, only in those places where this functionality (per-memcg scanning) is really needed. This new structure would look like this:

struct memcg_list_lru {
    struct list_lru global_lru;
    struct list_lru **memcg_lrus;
    struct list_head list;
    void *old_lrus;
}

Since old_lrus and memcg_lrus can be NULL under normal operation, in memcg_list_lru_destroy() I'd have to check either the list or the global_lru field, i.e. it would look like:

Quoted 5 lines

if (!list.next)

    /* has not been initialized */
    return;

or

... or just use hlist_head.

Currently in addition to a shrink_control struct shrink_slab() takes two arguments, nr_pages_scanned and lru_pages, which are used for balancing slab reclaim versus page reclaim - roughly speaking, shrink_slab() will try to scan nr_pages_scanned/lru_pages fraction of all slab objects.

However, shrink_slab() is not always called after page cache reclaim. For example, drop_slab() uses shrink_slab() to drop as many slab objects as possible and thus has to pass phony values 1000/1000 to it, which do not make sense for nr_pages_scanned/lru_pages. Moreover, as soon as

kmemcg reclaim is introduced, we will have to make up phony values for nr_pages_scanned and lru_pages again when doing kmem-only reclaim for a memory cgroup, which is possible if the cgroup has its kmem limit less than the total memory limit.

Quoted 4 lines

kmemcg reclaim is introduced, we will have to make up phony values for nr_pages_scanned and lru_pages again when doing kmem-only reclaim for a memory cgroup, which is possible if the cgroup has its kmem limit less than the total memory limit.

I'm missing something here - why would memcg reclaim require passing phony values? How are you going to keep slab caches in balance with memory pressure generated by the page cache?

Quoted 4 lines · Dave Chinner

Quoted 4 lines

kmemcg reclaim is introduced, we will have to make up phony values for nr_pages_scanned and lru_pages again when doing kmem-only reclaim for a memory cgroup, which is possible if the cgroup has its kmem limit less than the total memory limit.

Quoted 3 lines

I'm missing something here - why would memcg reclaim require passing phony values? How are you going to keep slab caches in balance with memory pressure generated by the page cache?

To perform kmem-only reclaim - please see patch 13. When the kmem limit is less than the user limit, we can reach only the kmem limit. It is no use trying to reclaim page cache then.

This patch makes direct reclaim path shrink slabs not only on global memory pressure, but also when we reach memory cgroup limit. To achieve that, it introduces a new per-shrinker flag, SHRINKER_MEMCG_AWARE, which should be set if the shrinker can handle per-memcg reclaim. For such shrinkers, shrink_slab() will iterate over all eligible memory cgroups (i.e. the cgroup that triggered the reclaim and all its descendants) and pass the current memory cgroup to the shrinker in shrink_control.memcg just like it passes the current NUMA node to NUMA-aware shrinkers. It is completely up to memcg-aware shrinkers how to organize objects in order to provide required functionality. Currently none of the existing shrinkers is memcg-aware, but next patches will introduce per-memcg list_lru, which will facilitate the process of turning shrinkers that use list_lru to be memcg-aware.

The number of slab objects scanned on memcg pressure is calculated in the same way as on global pressure - it is proportional to the number of pages scanned over the number of pages eligible for reclaim (i.e. the number of on-LRU pages in the target memcg and all its descendants) - except we do not employ the nr_deferred per-shrinker counter to avoid memory cgroup isolation issues. Ideally, this counter should be made per-memcg.

mm/vmscan.c +2 -3

@@ -236,11 +236,17 @@ shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker,
 		return 0;
 
 	/*
-	 * copy the current shrinker scan count into a local variable
-	 * and zero it so that other concurrent shrinker invocations
-	 * don't also do this scanning work.
+	 * Do not touch global counter of deferred objects on memcg pressure to
+	 * avoid isolation issues. Ideally the counter should be per-memcg.

mm/vmscan.c +4 -7

 	total_scan = nr;
 	delta = (4 * fraction) / shrinker->seeks;

@@ -296,21 +302,46 @@ shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker,
 		cond_resched();
 	}
 
-	/*
-	 * move the unused scan count back into the shrinker in a
-	 * manner that handles concurrent updates. If we exhausted the
-	 * scan, there is no need to do an update.
-	 */
-	if (total_scan > 0)
-		new_nr = atomic_long_add_return(total_scan,
+	if (!shrinkctl->target_mem_cgroup) {
+		/*
+		 * move the unused scan count back into the shrinker in a
+		 * manner that handles concurrent updates. If we exhausted the

mm/vmscan.c +7 -6

@@ -352,18 +383,23 @@ unsigned long shrink_slab(struct shrink_control *shrinkctl,
 	}
 
 	list_for_each_entry(shrinker, &shrinker_list, list) {
-		for_each_node_mask(shrinkctl->nid, shrinkctl->nodes_to_scan) {
-			if (!node_online(shrinkctl->nid))
-				continue;
-
-			if (!(shrinker->flags & SHRINKER_NUMA_AWARE) &&
-			    (shrinkctl->nid != 0))
+		shrinkctl->memcg = shrinkctl->target_mem_cgroup;
+		do {
+			if (!(shrinker->flags & SHRINKER_MEMCG_AWARE) &&
+			    (shrinkctl->memcg != NULL)) {
+				mem_cgroup_iter_break(
+						shrinkctl->target_mem_cgroup,
+						shrinkctl->memcg);
 				break;

Quoted 9 lines

mm/vmscan.c +2 -3

@@ -236,11 +236,17 @@ shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker,
 		return 0;
 
 	/*
-	 * copy the current shrinker scan count into a local variable
-	 * and zero it so that other concurrent shrinker invocations
-	 * don't also do this scanning work.
+	 * Do not touch global counter of deferred objects on memcg pressure to
+	 * avoid isolation issues. Ideally the counter should be per-memcg.

mm/vmscan.c shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker, · line 249

Quoted 10 lines

mm/vmscan.c +8 -1

 	 */
-	nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
+	if (!shrinkctl->target_mem_cgroup) {
+		/*
+		 * copy the current shrinker scan count into a local variable
+		 * and zero it so that other concurrent shrinker invocations
+		 * don't also do this scanning work.
+		 */
+		nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
+	}

That's ugly. Effectively it means that memcg reclaim is going to be completely ineffective when large numbers of allocations and hence reclaim attempts are done under GFP_NOFS context.

The only thing that keeps filesystem caches in balance when there is lots of filesystem work going on (i.e. lots of GFP_NOFS allocations) is the deferal of reclaim work to a context that can do something about it.

Quoted 17 lines

mm/vmscan.c +4 -7

 	total_scan = nr;
 	delta = (4 * fraction) / shrinker->seeks;

@@ -296,21 +302,46 @@ shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker,
 		cond_resched();
 	}
 
-	/*
-	 * move the unused scan count back into the shrinker in a
-	 * manner that handles concurrent updates. If we exhausted the
-	 * scan, there is no need to do an update.
-	 */
-	if (total_scan > 0)
-		new_nr = atomic_long_add_return(total_scan,
+	if (!shrinkctl->target_mem_cgroup) {
+		/*
+		 * move the unused scan count back into the shrinker in a
+		 * manner that handles concurrent updates. If we exhausted the

mm/vmscan.c shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker, · line 316

Quoted 10 lines

mm/vmscan.c +7 -2

+		 * scan, there is no need to do an update.
+		 */
+		if (total_scan > 0)
+			new_nr = atomic_long_add_return(total_scan,
 						&shrinker->nr_deferred[nid]);
-	else
-		new_nr = atomic_long_read(&shrinker->nr_deferred[nid]);
+		else
+			new_nr = atomic_long_read(&shrinker->nr_deferred[nid]);
+	}

So, if the memcg can't make progress, why wouldn't you defer the work to the global scan? Or can't a global scan trim memcg LRUs? And if it can't, then isn't that a major design flaw? Why not just allow kswapd to walk memcg LRUs in the background?

/me just looked at patch 13

Yeah, this goes some way to explaining why something like patch 13 is necessary - slab shrinkers are not keeping up with page cache reclaim because of GFP_NOFS allocations, and so the page cache empties only leaving slab caches to be trimmed....

mm/vmscan.c shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker, · line 324

Quoted 3 lines

mm/vmscan.c +3 -0

+static unsigned long
+shrink_slab_memcg(struct shrink_control *shrinkctl, struct shrinker *shrinker,
+		  unsigned long fraction, unsigned long denominator)

what's this function got to do with memcgs? Why did you rename it from the self explanitory shrink_slab_one() name that Glauber gave it?

mm/vmscan.c shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker, · line 329

Quoted 5 lines

mm/vmscan.c +5 -0

+{
+	unsigned long freed = 0;
+
+	if (shrinkctl->memcg && !memcg_kmem_is_active(shrinkctl->memcg))
+		return 0;

Why here? why not check that in the caller where memcg's are being iterated?

Quoted 5 lines

mm/vmscan.c +5 -0

+
+	for_each_node_mask(shrinkctl->nid, shrinkctl->nodes_to_scan) {
+		if (!node_online(shrinkctl->nid))
+			continue;
+

mm/vmscan.c shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker, · line 337

Quoted 3 lines

mm/vmscan.c +3 -0

+		if (!(shrinker->flags & SHRINKER_NUMA_AWARE) &&
+		    (shrinkctl->nid != 0))
+			break;

Hmmm - this looks broken. Nothing guarantees that node 0 in shrinkctl->nodes_to_scan is ever set, so non-numa aware shrinkers will do nothing when the first node in the mask is not set. For non-numa aware shrinkers, the shrinker should always be called once with a node id of 0.

That's what earlier versions of the numa aware shrinker patch set did, and it seems to have been lost along the way. Yeah, there's the last version from Glauber's tree that I saw:

static unsigned long
shrink_slab_one(struct shrink_control *shrinkctl, struct shrinker *shrinker,
unsigned long nr_pages_scanned, unsigned long lru_pages)

{
       unsigned long freed = 0;

       if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) {
               shrinkctl->nid = 0;

               return shrink_slab_node(shrinkctl, shrinker,
                        nr_pages_scanned, lru_pages,
                        &shrinker->nr_deferred);
       }

       for_each_node_mask(shrinkctl->nid, shrinkctl->nodes_to_scan)

               if (!node_online(shrinkctl->nid))
                       continue;

               freed += shrink_slab_node(shrinkctl, shrinker,
                        nr_pages_scanned, lru_pages,
			 &shrinker->nr_deferred_node[shrinkctl->nid]);
       }

       return freed;
}

So, that's likely to be another reason that all the non-numa slab caches are not being shrunk appropriately and need to be hit with a bit hammer...

Quoted 20 lines

mm/vmscan.c +8 -6

@@ -352,18 +383,23 @@ unsigned long shrink_slab(struct shrink_control *shrinkctl,
 	}
 
 	list_for_each_entry(shrinker, &shrinker_list, list) {
-		for_each_node_mask(shrinkctl->nid, shrinkctl->nodes_to_scan) {
-			if (!node_online(shrinkctl->nid))
-				continue;
-
-			if (!(shrinker->flags & SHRINKER_NUMA_AWARE) &&
-			    (shrinkctl->nid != 0))
+		shrinkctl->memcg = shrinkctl->target_mem_cgroup;
+		do {
+			if (!(shrinker->flags & SHRINKER_MEMCG_AWARE) &&
+			    (shrinkctl->memcg != NULL)) {
+				mem_cgroup_iter_break(
+						shrinkctl->target_mem_cgroup,
+						shrinkctl->memcg);
 				break;
+			}
 

mm/vmscan.c unsigned long shrink_slab(struct shrink_control *shrinkctl, · line 402

Quoted 8 lines

mm/vmscan.c +6 -2

-			freed += shrink_slab_node(shrinkctl, shrinker,
-						  fraction, denominator);
+			freed += shrink_slab_memcg(shrinkctl, shrinker,
+						   fraction, denominator);
+			shrinkctl->memcg = mem_cgroup_iter(
+						shrinkctl->target_mem_cgroup,
+						shrinkctl->memcg, NULL);
+		} while (shrinkctl->memcg);

Glauber's tree also had a bunch of comments explaining what was going on here. I've got no idea what the hell this code is doing, and why the hell we are iterating memcgs here and how and why the normal, non-memcg scan and shrinkers still worked.

This is now just a bunch of memcg gobbledegook with no explanations to tell us what it is supposed to be doing. Comments are important - you might not think they are necessary, but seeing comments like this:

(quoted diff) +11 -0

+               /*
+                * In a hierarchical chain, it might be that not all memcgs are
+                * kmem active. kmemcg design mandates that when one memcg is
+                * active, its children will be active as well. But it is
+                * perfectly possible that its parent is not.
+                *
+                * We also need to make sure we scan at least once, for the
+                * global case. So if we don't have a target memcg (saved in
+                * root), we proceed normally and expect to break in the next
+                * round.
+                */

in Glauber's tree helped an awful lot to explain the mess that the memcg stuff was making of the code...

I'm liking this patch set less and less as I work my way through it...

Quoted 23 lines · Dave Chinner

Quoted 9 lines

mm/vmscan.c +2 -3

@@ -236,11 +236,17 @@ shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker,
 		return 0;
 
 	/*
-	 * copy the current shrinker scan count into a local variable
-	 * and zero it so that other concurrent shrinker invocations
-	 * don't also do this scanning work.
+	 * Do not touch global counter of deferred objects on memcg pressure to
+	 * avoid isolation issues. Ideally the counter should be per-memcg.

mm/vmscan.c shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker, · line 249

Quoted 10 lines

mm/vmscan.c +8 -1

 	 */
-	nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
+	if (!shrinkctl->target_mem_cgroup) {
+		/*
+		 * copy the current shrinker scan count into a local variable
+		 * and zero it so that other concurrent shrinker invocations
+		 * don't also do this scanning work.
+		 */
+		nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
+	}

That's ugly. Effectively it means that memcg reclaim is going to be completely ineffective when large numbers of allocations and hence reclaim attempts are done under GFP_NOFS context.

Quoted 4 lines

The only thing that keeps filesystem caches in balance when there is lots of filesystem work going on (i.e. lots of GFP_NOFS allocations) is the deferal of reclaim work to a context that can do something about it.

Imagine the situation: a memcg issues a GFP_NOFS allocation and goes to shrink_slab() where it defers them to the global counter; then another memcg issues a GFP_KERNEL allocation, also goes to shrink_slab() where it sees a huge number of deferred objects and starts shrinking them, which is not good IMHO.

I understand that nr_deferred is necessary, but
I think it should be per-memcg. What do you think about moving it to list_lru?

Quoted 13 lines

So, if the memcg can't make progress, why wouldn't you defer the work to the global scan? Or can't a global scan trim memcg LRUs? And if it can't, then isn't that a major design flaw? Why not just allow kswapd to walk memcg LRUs in the background?

/me just looked at patch 13

Yeah, this goes some way to explaining why something like patch 13 is necessary - slab shrinkers are not keeping up with page cache reclaim because of GFP_NOFS allocations, and so the page cache empties only leaving slab caches to be trimmed....

Quoted 6 lines

mm/vmscan.c shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker, · line 324

Quoted 3 lines

mm/vmscan.c +3 -0

+static unsigned long
+shrink_slab_memcg(struct shrink_control *shrinkctl, struct shrinker *shrinker,
+		  unsigned long fraction, unsigned long denominator)

what's this function got to do with memcgs? Why did you rename it from the self explanitory shrink_slab_one() name that Glauber gave it?

When I sent the previous version, Johannes Weiner disliked the name that was why I renamed it, now you don't like the new name and ask for the old one :-) But why do you think that shrink_slab_one() is self-explanatory while shrink_slab_memcg() is not? I mean shrink_slab_memcg() means "shrink slab accounted to a memcg" just like

shrink_slab_node() means "shrink slab on the node" while seeing shrink_slab_one() I would ask "one what?".

Quoted 28 lines · Vladimir Davydov

Quoted 23 lines · Dave Chinner

Quoted 9 lines

mm/vmscan.c +2 -3

@@ -236,11 +236,17 @@ shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker,
 		return 0;
 
 	/*
-	 * copy the current shrinker scan count into a local variable
-	 * and zero it so that other concurrent shrinker invocations
-	 * don't also do this scanning work.
+	 * Do not touch global counter of deferred objects on memcg pressure to
+	 * avoid isolation issues. Ideally the counter should be per-memcg.

mm/vmscan.c shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker, · line 249

Quoted 10 lines

mm/vmscan.c +8 -1

 	 */
-	nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
+	if (!shrinkctl->target_mem_cgroup) {
+		/*
+		 * copy the current shrinker scan count into a local variable
+		 * and zero it so that other concurrent shrinker invocations
+		 * don't also do this scanning work.
+		 */
+		nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
+	}

That's ugly. Effectively it means that memcg reclaim is going to be completely ineffective when large numbers of allocations and hence reclaim attempts are done under GFP_NOFS context.

Quoted 4 lines

The only thing that keeps filesystem caches in balance when there is lots of filesystem work going on (i.e. lots of GFP_NOFS allocations) is the deferal of reclaim work to a context that can do something about it.

Quoted 5 lines

Imagine the situation: a memcg issues a GFP_NOFS allocation and goes to shrink_slab() where it defers them to the global counter; then another memcg issues a GFP_KERNEL allocation, also goes to shrink_slab() where it sees a huge number of deferred objects and starts shrinking them, which is not good IMHO.

That's exactly what the deferred mechanism is for - we know we have to do the work, but we can't do it right now so let someone else do it who can.

In most cases, deferral is handled by kswapd, because when a filesystem workload is causing memory pressure then most allocations are done in GFP_NOFS conditions. Hence the only memory reclaim that can make progress here is kswapd.

Right now, you aren't deferring any of this memory pressure to some other agent, so it just does not get done. That's a massive problem
- it's a design flaw - and instead I see lots of crazy hacks being added to do stuff that should simply be deferred to kswapd like is done for global memory pressure.

Hell, kswapd shoul dbe allowed to walk memcg LRU lists and trim them, just like it does for the global lists. We only need a single "deferred work" counter per node for that - just let kswapd proportion the deferred work over the per-node LRU and the memcgs....

Quoted 49 lines · Dave Chinner

Quoted 28 lines · Vladimir Davydov

Quoted 23 lines · Dave Chinner

Quoted 9 lines

mm/vmscan.c +2 -3

@@ -236,11 +236,17 @@ shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker,
 		return 0;
 
 	/*
-	 * copy the current shrinker scan count into a local variable
-	 * and zero it so that other concurrent shrinker invocations
-	 * don't also do this scanning work.
+	 * Do not touch global counter of deferred objects on memcg pressure to
+	 * avoid isolation issues. Ideally the counter should be per-memcg.

mm/vmscan.c shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker, · line 249

Quoted 10 lines

mm/vmscan.c +8 -1

 	 */
-	nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
+	if (!shrinkctl->target_mem_cgroup) {
+		/*
+		 * copy the current shrinker scan count into a local variable
+		 * and zero it so that other concurrent shrinker invocations
+		 * don't also do this scanning work.
+		 */
+		nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
+	}

That's ugly. Effectively it means that memcg reclaim is going to be completely ineffective when large numbers of allocations and hence reclaim attempts are done under GFP_NOFS context.

Quoted 4 lines

The only thing that keeps filesystem caches in balance when there is lots of filesystem work going on (i.e. lots of GFP_NOFS allocations) is the deferal of reclaim work to a context that can do something about it.

Quoted 5 lines

Imagine the situation: a memcg issues a GFP_NOFS allocation and goes to shrink_slab() where it defers them to the global counter; then another memcg issues a GFP_KERNEL allocation, also goes to shrink_slab() where it sees a huge number of deferred objects and starts shrinking them, which is not good IMHO.

That's exactly what the deferred mechanism is for - we know we have to do the work, but we can't do it right now so let someone else do it who can.

In most cases, deferral is handled by kswapd, because when a filesystem workload is causing memory pressure then most allocations are done in GFP_NOFS conditions. Hence the only memory reclaim that can make progress here is kswapd.

Right now, you aren't deferring any of this memory pressure to some other agent, so it just does not get done. That's a massive problem
- it's a design flaw - and instead I see lots of crazy hacks being added to do stuff that should simply be deferred to kswapd like is done for global memory pressure.

Quoted 5 lines

Hell, kswapd shoul dbe allowed to walk memcg LRU lists and trim them, just like it does for the global lists. We only need a single "deferred work" counter per node for that - just let kswapd proportion the deferred work over the per-node LRU and the memcgs....

Seems I misunderstand :-(

Let me try. You mean we have the only nr_deferred counter per-node, and kswapd scans

nr_deferred*memcg_kmem_size/total_kmem_size

objects in each memcg, right?

Then if there were a lot of objects deferred on memcg (not global) pressure due to a memcg issuing a lot of GFP_NOFS allocations, kswapd will reclaim objects from all, even unlimited, memcgs. This looks like an isolation issue :-/

Currently we have a per-node nr_deferred counter for each shrinker. If we add per-memcg reclaim, we have to make it per-memcg per-node, don't we?

Quoted 60 lines · Vladimir Davydov

Quoted 49 lines · Dave Chinner

Quoted 28 lines · Vladimir Davydov

Quoted 23 lines · Dave Chinner

Quoted 9 lines

mm/vmscan.c +2 -3

@@ -236,11 +236,17 @@ shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker,
 		return 0;
 
 	/*
-	 * copy the current shrinker scan count into a local variable
-	 * and zero it so that other concurrent shrinker invocations
-	 * don't also do this scanning work.
+	 * Do not touch global counter of deferred objects on memcg pressure to
+	 * avoid isolation issues. Ideally the counter should be per-memcg.

mm/vmscan.c shrink_slab_node(struct shrink_control *shrinkctl, struct shrinker *shrinker, · line 249

Quoted 10 lines

mm/vmscan.c +8 -1

 	 */
-	nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
+	if (!shrinkctl->target_mem_cgroup) {
+		/*
+		 * copy the current shrinker scan count into a local variable
+		 * and zero it so that other concurrent shrinker invocations
+		 * don't also do this scanning work.
+		 */
+		nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
+	}

That's ugly. Effectively it means that memcg reclaim is going to be completely ineffective when large numbers of allocations and hence reclaim attempts are done under GFP_NOFS context.

Quoted 4 lines

The only thing that keeps filesystem caches in balance when there is lots of filesystem work going on (i.e. lots of GFP_NOFS allocations) is the deferal of reclaim work to a context that can do something about it.

Quoted 5 lines

Imagine the situation: a memcg issues a GFP_NOFS allocation and goes to shrink_slab() where it defers them to the global counter; then another memcg issues a GFP_KERNEL allocation, also goes to shrink_slab() where it sees a huge number of deferred objects and starts shrinking them, which is not good IMHO.

That's exactly what the deferred mechanism is for - we know we have to do the work, but we can't do it right now so let someone else do it who can.

In most cases, deferral is handled by kswapd, because when a filesystem workload is causing memory pressure then most allocations are done in GFP_NOFS conditions. Hence the only memory reclaim that can make progress here is kswapd.

Right now, you aren't deferring any of this memory pressure to some other agent, so it just does not get done. That's a massive problem
- it's a design flaw - and instead I see lots of crazy hacks being added to do stuff that should simply be deferred to kswapd like is done for global memory pressure.

Quoted 5 lines

Hell, kswapd shoul dbe allowed to walk memcg LRU lists and trim them, just like it does for the global lists. We only need a single "deferred work" counter per node for that - just let kswapd proportion the deferred work over the per-node LRU and the memcgs....

Seems I misunderstand :-(

Let me try. You mean we have the only nr_deferred counter per-node, and kswapd scans

Quoted 8 lines

nr_deferred*memcg_kmem_size/total_kmem_size

objects in each memcg, right?

Then if there were a lot of objects deferred on memcg (not global) pressure due to a memcg issuing a lot of GFP_NOFS allocations, kswapd will reclaim objects from all, even unlimited, memcgs. This looks like an isolation issue :-/

Which, when you are running out of memory, is a much less of an issue than not being able to make progress reclaiming memory.

Besides, the "isolation" argument runs both ways. e.g. when there isn't memory available, it's entirely possible it's because there is actually no free memory, not because we've hit a memcg limit. e.g. all the memory has been consumed by an unlimited memcg, and we need to reclaim from it so this memcg can make progress.

In those situations we need to reclaim from everyone, not just the memcg that can't find free memory to allocate....

Quoted 2 lines

Currently we have a per-node nr_deferred counter for each shrinker. If we add per-memcg reclaim, we have to make it per-memcg per-node, don't we?

Think about what you just said for a moment. We have how many memcg shrinkers? And we can support how many nodes? And we can support how many memcgs? And when we multiply that all together, how much memory do we need to track that?

FS-shrinkers, which shrink dcaches and icaches, keep dentries and inodes in list_lru structures in order to evict least recently used objects. With per-memcg kmem shrinking infrastructure introduced, we have to make those LRU lists per-memcg in order to allow shrinking FS caches that belong to different memory cgroups independently.

This patch addresses the issue by introducing struct memcg_list_lru. This struct aggregates list_lru objects for each kmem-active memcg, and keeps it uptodate whenever a memcg is created or destroyed. Its interface is very simple: it only allows to get the pointer to the appropriate list_lru object from a memcg or a kmem ptr, which should be further operated with conventional list_lru methods.

include/linux/list_lru.h +16 -0

+/*
+ * The following structure can be used to reclaim kmem objects accounted to
+ * different memory cgroups independently. It aggregates a set of list_lru
+ * objects, one for each kmem-enabled memcg, and provides the method to get
+ * the lru corresponding to a memcg.
+ */
+struct memcg_list_lru {
+	struct list_lru global_lru;
+
+#ifdef CONFIG_MEMCG_KMEM
+	struct list_lru **memcg_lrus;	/* rcu-protected array of per-memcg
+					   lrus, indexed by memcg_cache_id() */
+
+	struct list_head list;		/* list of all memcg-aware lrus */
+
+	/*

Quoted 6 lines · Vladimir Davydov

FS-shrinkers, which shrink dcaches and icaches, keep dentries and inodes in list_lru structures in order to evict least recently used objects. With per-memcg kmem shrinking infrastructure introduced, we have to make those LRU lists per-memcg in order to allow shrinking FS caches that belong to different memory cgroups independently.

Quoted 6 lines

This patch addresses the issue by introducing struct memcg_list_lru. This struct aggregates list_lru objects for each kmem-active memcg, and keeps it uptodate whenever a memcg is created or destroyed. Its interface is very simple: it only allows to get the pointer to the appropriate list_lru object from a memcg or a kmem ptr, which should be further operated with conventional list_lru methods.

Basically The idea was that the memcg LRUs hide entirely behind the generic list_lru interface so that any cache that used the list_lru insfrastructure got memcg capabilities for free. memcg's to shrink were to be passed through the shrinker control shrinkers to the list LRU code, and it then did all the "which lru are we using" logic internally.

What you've done is driven all the "which LRU are we using" logic into every single caller location. i.e. you've just broken the underlying design principle that Glauber and I had worked towards with this code - that memcg aware LRUs should be completely transparent to list_lru users. Just like NUMA awareness came for free with the list_lru code, so should memcg awareness....

Quoted 23 lines

include/linux/list_lru.h +23 -0

+/*
+ * The following structure can be used to reclaim kmem objects accounted to
+ * different memory cgroups independently. It aggregates a set of list_lru
+ * objects, one for each kmem-enabled memcg, and provides the method to get
+ * the lru corresponding to a memcg.
+ */
+struct memcg_list_lru {
+	struct list_lru global_lru;
+
+#ifdef CONFIG_MEMCG_KMEM
+	struct list_lru **memcg_lrus;	/* rcu-protected array of per-memcg
+					   lrus, indexed by memcg_cache_id() */
+
+	struct list_head list;		/* list of all memcg-aware lrus */
+
+	/*
+	 * The memcg_lrus array is rcu protected, so we can only free it after
+	 * a call to synchronize_rcu(). To avoid multiple calls to
+	 * synchronize_rcu() when many lrus get updated at the same time, which
+	 * is a typical scenario, we will store the pointer to the previous
+	 * version of the array in the old_lrus variable for each lru, and then
+	 * free them all at once after a single call to synchronize_rcu().
+	 */

include/linux/list_lru.h struct list_lru { · line 61

Quoted 3 lines

include/linux/list_lru.h +3 -0

+	void *old_lrus;
+#endif
+};

Really, this should be embedded in the struct list_lru, not wrapping around the outside. I don't see any changelog to tell me why you changed the code from what was last in Glauber's tree, so can you explain why exposing all this memcg stuff to everyone is a good idea?

Quoted 50 lines · Dave Chinner

Quoted 6 lines · Vladimir Davydov

FS-shrinkers, which shrink dcaches and icaches, keep dentries and inodes in list_lru structures in order to evict least recently used objects. With per-memcg kmem shrinking infrastructure introduced, we have to make those LRU lists per-memcg in order to allow shrinking FS caches that belong to different memory cgroups independently.

Quoted 6 lines

This patch addresses the issue by introducing struct memcg_list_lru. This struct aggregates list_lru objects for each kmem-active memcg, and keeps it uptodate whenever a memcg is created or destroyed. Its interface is very simple: it only allows to get the pointer to the appropriate list_lru object from a memcg or a kmem ptr, which should be further operated with conventional list_lru methods.

Basically The idea was that the memcg LRUs hide entirely behind the generic list_lru interface so that any cache that used the list_lru insfrastructure got memcg capabilities for free. memcg's to shrink were to be passed through the shrinker control shrinkers to the list LRU code, and it then did all the "which lru are we using" logic internally.

What you've done is driven all the "which LRU are we using" logic into every single caller location. i.e. you've just broken the underlying design principle that Glauber and I had worked towards with this code - that memcg aware LRUs should be completely transparent to list_lru users. Just like NUMA awareness came for free with the list_lru code, so should memcg awareness....

Quoted 23 lines

include/linux/list_lru.h +23 -0

+/*
+ * The following structure can be used to reclaim kmem objects accounted to
+ * different memory cgroups independently. It aggregates a set of list_lru
+ * objects, one for each kmem-enabled memcg, and provides the method to get
+ * the lru corresponding to a memcg.
+ */
+struct memcg_list_lru {
+	struct list_lru global_lru;
+
+#ifdef CONFIG_MEMCG_KMEM
+	struct list_lru **memcg_lrus;	/* rcu-protected array of per-memcg
+					   lrus, indexed by memcg_cache_id() */
+
+	struct list_head list;		/* list of all memcg-aware lrus */
+
+	/*
+	 * The memcg_lrus array is rcu protected, so we can only free it after
+	 * a call to synchronize_rcu(). To avoid multiple calls to
+	 * synchronize_rcu() when many lrus get updated at the same time, which
+	 * is a typical scenario, we will store the pointer to the previous
+	 * version of the array in the old_lrus variable for each lru, and then
+	 * free them all at once after a single call to synchronize_rcu().
+	 */

Quoted 8 lines

include/linux/list_lru.h struct list_lru { · line 61

Quoted 3 lines

include/linux/list_lru.h +3 -0

+	void *old_lrus;
+#endif
+};

Really, this should be embedded in the struct list_lru, not wrapping around the outside. I don't see any changelog to tell me why you changed the code from what was last in Glauber's tree, so can you explain why exposing all this memcg stuff to everyone is a good idea?

I preferred to move from list_lru to memcg_list_lru, because the connection between list_lru and memcgs' turned memcontrol.c and list_lru.c into a monolithic structure. When I read comments to the last version of this patchset submitted by Glauber (v10), I found that Andrew Morton disliked it, that was why I tried to "fix" it the way you observe in this patch. Besides, I though that the list_lru may be used w/o memcgs.

Using the per-memcg LRU infrastructure introduced by previous patches, this patch makes dcache and icache shrinkers memcg-aware. To achieve that, it converts s_dentry_lru and s_inode_lru from list_lru to memcg_list_lru and restricts the reclaim to per-memcg parts of the lists in case of memcg pressure.

Other FS objects are currently ignored and only reclaimed on global pressure, because their shrinkers are heavily FS-specific and can't be converted to be memcg-aware so easily. However, we can pass on target memcg to the FS layer and let it decide if per-memcg objects should be reclaimed.

Note that with this patch applied we lose global LRU order, but it does

fs/dcache.c +0 -0

@@ -343,18 +343,24 @@ static void dentry_unlink_inode(struct dentry * dentry)
 #define D_FLAG_VERIFY(dentry,x) WARN_ON_ONCE(((dentry)->d_flags & (DCACHE_LRU_LIST | DCACHE_SHRINK_LIST)) != (x))
 static void d_lru_add(struct dentry *dentry)

fs/dcache.c +4 -5

@@ -970,9 +976,9 @@ dentry_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
 }
 
 /**
- * prune_dcache_sb - shrink the dcache
- * @sb: superblock
- * @nr_to_scan : number of entries to try to free
+ * prune_dcache_lru - shrink the dcache
+ * @lru: dentry lru list
+ * @nr_to_scan: number of entries to try to free
  * @nid: which node to scan for freeable entities
  *
  * Attempt to shrink the superblock dcache LRU by @nr_to_scan entries. This is

@@ -982,14 +988,13 @@ dentry_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
  * This function may fail to free any resources if all the dentries are in
  * use.
  */
-long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan,
-		     int nid)
+long prune_dcache_lru(struct list_lru *lru, unsigned long nr_to_scan, int nid)
 {

fs/super.c +3 -1

index cece164..b198da4 100644
@@ -57,6 +57,8 @@ static unsigned long super_cache_scan(struct shrinker *shrink,
 				      struct shrink_control *sc)
 {
 	struct super_block *sb;
+	struct list_lru *inode_lru;
+	struct list_lru *dentry_lru;
 	long	fs_objects = 0;
 	long	total_objects;
 	long	freed = 0;

@@ -75,11 +77,14 @@ static unsigned long super_cache_scan(struct shrinker *shrink,
 	if (!grab_super_passive(sb))
 		return SHRINK_STOP;
 
-	if (sb->s_op->nr_cached_objects)
+	if (sb->s_op->nr_cached_objects && !sc->memcg)

Quoted 11 lines · Vladimir Davydov

Using the per-memcg LRU infrastructure introduced by previous patches, this patch makes dcache and icache shrinkers memcg-aware. To achieve that, it converts s_dentry_lru and s_inode_lru from list_lru to memcg_list_lru and restricts the reclaim to per-memcg parts of the lists in case of memcg pressure.

Other FS objects are currently ignored and only reclaimed on global pressure, because their shrinkers are heavily FS-specific and can't be converted to be memcg-aware so easily. However, we can pass on target memcg to the FS layer and let it decide if per-memcg objects should be reclaimed.

And now you have a big problem, because that means filesystems like XFS won't reclaim inodes during memcg reclaim.

That is, for XFS, prune_icache_lru() does not free any memory. All it does is remove all the VFS references to the struct xfs_inode, which is then reclaimed via the sb->s_op->free_cached_objects() method.

IOWs, what you've done is broken.

Quoted 2 lines

Note that with this patch applied we lose global LRU order, but it does

We don't have global LRU order today for the filesystem caches. We have per superblock, per-node LRU reclaim order.

Quoted 5 lines

fs/dcache.c +0 -0

@@ -343,18 +343,24 @@ static void dentry_unlink_inode(struct dentry * dentry)
 #define D_FLAG_VERIFY(dentry,x) WARN_ON_ONCE(((dentry)->d_flags & (DCACHE_LRU_LIST | DCACHE_SHRINK_LIST)) != (x))
 static void d_lru_add(struct dentry *dentry)

fs/dcache.c static void dentry_unlink_inode(struct dentry * dentry) · line 353

Quoted 10 lines

fs/dcache.c +4 -1

 {
+	struct list_lru *lru =
+		mem_cgroup_kmem_list_lru(&dentry->d_sb->s_dentry_lru, dentry);
+
 	D_FLAG_VERIFY(dentry, 0);
 	dentry->d_flags |= DCACHE_LRU_LIST;
 	this_cpu_inc(nr_dentry_unused);
-	WARN_ON_ONCE(!list_lru_add(&dentry->d_sb->s_dentry_lru, &dentry->d_lru));
+	WARN_ON_ONCE(!list_lru_add(lru, &dentry->d_lru));
 }

This is what I mean about pushing memcg cruft into places where it is not necessary. This can be done entirely behind list_lru_add(), without the caller having to care.

Quoted 21 lines

fs/dcache.c +4 -5

@@ -970,9 +976,9 @@ dentry_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
 }
 
 /**
- * prune_dcache_sb - shrink the dcache
- * @sb: superblock
- * @nr_to_scan : number of entries to try to free
+ * prune_dcache_lru - shrink the dcache
+ * @lru: dentry lru list
+ * @nr_to_scan: number of entries to try to free
  * @nid: which node to scan for freeable entities
  *
  * Attempt to shrink the superblock dcache LRU by @nr_to_scan entries. This is

@@ -982,14 +988,13 @@ dentry_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
  * This function may fail to free any resources if all the dentries are in
  * use.
  */
-long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan,
-		     int nid)
+long prune_dcache_lru(struct list_lru *lru, unsigned long nr_to_scan, int nid)
 {

fs/dcache.c dentry_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg) · line 1000

Quoted 10 lines

fs/dcache.c +2 -2

 	LIST_HEAD(dispose);
 	long freed;
 
-	freed = list_lru_walk_node(&sb->s_dentry_lru, nid, dentry_lru_isolate,
-				       &dispose, &nr_to_scan);
+	freed = list_lru_walk_node(lru, nid, dentry_lru_isolate,
+				   &dispose, &nr_to_scan);
 	shrink_dentry_list(&dispose);
 	return freed;
 }

And here, you pass an LRU when what we really need to pass is the struct shrink_control that contains nr_to_scan, nid, and the memcg that pruning is targetting.

Because of the tight integration of the LRUs and shrinkers, it makes sense to pass the shrink control all the way into the list. i.e:

	freed = list_lru_scan(&sb->s_dentry_lru, sc, dentry_lru_isolate,
			      &dispose);

And again, that hides everything to do with memcg based LRUs and reclaim from the callers. It's clean, simple and hard to get wrong.

fs/dcache.c void shrink_dcache_sb(struct super_block *sb) · line 1039

Quoted 8 lines

fs/dcache.c +1 -1

@@ -1029,7 +1034,7 @@ void shrink_dcache_sb(struct super_block *sb)
 	do {
 		LIST_HEAD(dispose);
 
-		freed = list_lru_walk(&sb->s_dentry_lru,
+		freed = memcg_list_lru_walk_all(&sb->s_dentry_lru,
 			dentry_lru_isolate_shrink, &dispose, UINT_MAX);
 

list_lru_walk() is, by definition, supposed to walk every single object on the LRU. With memcg awareness, it should be walking all the memcg lists, too.

Quoted 19 lines

fs/super.c +3 -1

index cece164..b198da4 100644
@@ -57,6 +57,8 @@ static unsigned long super_cache_scan(struct shrinker *shrink,
 				      struct shrink_control *sc)
 {
 	struct super_block *sb;
+	struct list_lru *inode_lru;
+	struct list_lru *dentry_lru;
 	long	fs_objects = 0;
 	long	total_objects;
 	long	freed = 0;

@@ -75,11 +77,14 @@ static unsigned long super_cache_scan(struct shrinker *shrink,
 	if (!grab_super_passive(sb))
 		return SHRINK_STOP;
 
-	if (sb->s_op->nr_cached_objects)
+	if (sb->s_op->nr_cached_objects && !sc->memcg)

fs/super.c static unsigned long super_cache_scan(struct shrinker *shrink, · line 88

Quoted 10 lines

fs/super.c +5 -2

 		fs_objects = sb->s_op->nr_cached_objects(sb, sc->nid);
 
-	inodes = list_lru_count_node(&sb->s_inode_lru, sc->nid);
-	dentries = list_lru_count_node(&sb->s_dentry_lru, sc->nid);
+	inode_lru = mem_cgroup_list_lru(&sb->s_inode_lru, sc->memcg);
+	dentry_lru = mem_cgroup_list_lru(&sb->s_dentry_lru, sc->memcg);
+
+	inodes = list_lru_count_node(inode_lru, sc->nid);
+	dentries = list_lru_count_node(dentry_lru, sc->nid);
 	total_objects = dentries + inodes + fs_objects + 1;

Again: list_lru_count_sc(&sb->s_inode_lru, sc).

And push the scan control down into ->nr_cached_objects, too.

Review

Quoted 7 lines

(quoted diff) +7 -0

+static unsigned long
+zone_nr_reclaimable_pages(struct scan_control *sc, struct zone *zone)
+{
+	if (global_reclaim(sc))
+		return zone_reclaimable_pages(zone);
+	return memcg_zone_reclaimable_pages(sc->target_mem_cgroup, zone);
+}

So we have zone_reclaimable_pages() and zone_nr_reclaimable_pages() with completely different signatures and usecases. Not good.

The intersection between a zone and a memcg is called an lruvec, please use that. Look up an lruvec as early as possible, then implement lruvec_reclaimable_pages() etc. for use during reclaim.

(quoted diff) +11 -6

-		for_each_node_mask(shrinkctl->nid, shrinkctl->nodes_to_scan) {
-			if (!node_online(shrinkctl->nid))
-				continue;
-
-			if (!(shrinker->flags & SHRINKER_NUMA_AWARE) &&
-			    (shrinkctl->nid != 0))
+			if (!memcg || memcg_kmem_is_active(memcg))
+				freed += shrink_slab_one(shrinkctl, shrinker,
+					 nr_pages_scanned, lru_pages);
+			/*
+			 * For non-memcg aware shrinkers, we will arrive here
+			 * at first pass because we need to scan the root
+			 * memcg.  We need to bail out, since exactly because
+			 * they are not memcg aware, instead of noticing they
+			 * have nothing to shrink, they will just shrink again,
+			 * and deplete too many objects.
+			 */

I actually found the code easier to understand without this comment.

Review

Quoted 4 lines

(quoted diff) +3 -0

+			if (!(shrinker->flags & SHRINKER_MEMCG_AWARE))
  				break;
+			shrinkctl->target_mem_cgroup =
+				mem_cgroup_iter(root, memcg, NULL);

The target memcg is always the same, don't change this. Look at the lru scan code for reference. Iterate zones (nodes in this case) first, then iterate the memcgs in each zone (node), look up the lruvec and then call shrink_slab_lruvec(lruvec, ...).

Hi,

This is the 12th iteration of Glauber Costa's patchset implementing targeted shrinking for memory cgroups when kmem limits are present. So far, we've been accounting kernel objects but failing allocations when short of memory. This is because our only option would be to call the global shrinker, depleting objects from all caches and breaking isolation.

The main idea is to make LRU lists used by FS slab shrinkers per-memcg. When adding or removing an element from from the LRU, we use the page information to figure out which memory cgroup it belongs to and relay it to the appropriate list. This allows scanning kmem objects accounted to different memory cgroups independently.

The patchset is based on top of Linux 3.13-rc2 and organized as follows:

* patches 1-8 are for cleanup/preparation;
* patch 9 introduces infrastructure for memcg-aware shrinkers;
* patches 10 and 11 implement the per-memcg LRU list structure;
* patch 12 uses per-memcg LRU lists to make dcache and icache shrinkers

    memcg-aware;

* patch 13 implements kmem-only shrinking;
* patches 14-18 issue kmem shrinking on limit resize, global pressure.

Known issues:

* Since FS shrinkers can't be executed on __GFP_FS allocations, such allocations will fail if memcg kmem limit is less than the user limit and the memcg kmem usage is close to its limit. Glauber proposed to schedule a worker which would shrink kmem in the background on such allocations. However, this approach does not eliminate failures completely, it just makes them rarer. I'm thinking on implementing soft limits for memcg kmem so that striking the soft limit will trigger the reclaimer, but won't fail the allocation. I would appreciate any other proposals on how this can be fixed.

* Only dcache and icache are reclaimed on memcg pressure. Other FS objects are left for global pressure only. However, it should not be a serious problem to make them reclaimable too by passing on memcg to the FS-layer and letting each FS decide if its internal objects are shrinkable on memcg pressure.

Changelog:

Changes in v12:
* Do not prune all slabs on kmem-only pressure.
* Count all on-LRU pages eligible for reclaim to pass to shrink_slab().
* Fix isolation issue due to using shrinker->nr_deferred on memcg pressure.
* Add comments to memcg_list_lru functions.
* Code cleanup/refactoring.

Changes in v11:
* Rework per-memcg list_lru infrastructure.

Glauber Costa (7):
memcg: make cache index determination more robust
memcg: consolidate callers of memcg_cache_id
memcg: move initialization to memcg creation
memcg: allow kmem limit to be resized down
vmpressure: in-kernel notifications
memcg: reap dead memcgs upon global memory pressure
memcg: flush memcg items upon memcg destruction

Vladimir Davydov (11):
memcg: move several kmemcg functions upper
fs: do not use destroy_super() in alloc_super() fail path
vmscan: rename shrink_slab() args to make it more generic
vmscan: move call to shrink_slab() to shrink_zones()
vmscan: do_try_to_free_pages(): remove shrink_control argument
vmscan: shrink slab on memcg pressure
memcg,list_lru: add per-memcg LRU list infrastructure memcg,list_lru: add function walking over all lists of a per-memcg
LRU
fs: make icache, dcache shrinkers memcg-aware
memcg: per-memcg kmem shrinking
vmscan: take at least one pass with shrinkers

  fs/dcache.c                   |   25 +-
  fs/inode.c                    |   16 +-
  fs/internal.h                 |    9 +-
  fs/super.c                    |   48 ++-
  include/linux/fs.h            |    4 +-
  include/linux/list_lru.h      |   83 +++++
  include/linux/memcontrol.h    |   22 ++
  include/linux/mm.h            |    3 +-
  include/linux/shrinker.h      |   10 +-
  include/linux/swap.h          |    2 +
  include/linux/vmpressure.h    |    5 +
  include/trace/events/vmscan.h |   20 +-
  mm/memcontrol.c               |  728 ++++++++++++++++++++++++++++++++++++-----
  mm/vmpressure.c               |   53 ++-
  mm/vmscan.c                   |  249 +++++++++-----
  15 files changed, 1054 insertions(+), 223 deletions(-)