mm: memcontrol: fix cgroup creation failure after many small jobs

kernel/cgroup.c +7 -0

@@ -6205,6 +6205,24 @@ struct cgroup *cgroup_get_from_path(const char *path)
 }
 EXPORT_SYMBOL_GPL(cgroup_get_from_path);
 
+/**
+ * css_id_free - relinquish an existing CSS's ID
+ * @css: the CSS
+ *
+ * This releases the @css's ID and allows it to be recycled while the
+ * CSS continues to exist. This is useful for controllers with state
+ * that extends past a cgroup's lifetime but doesn't need precious ID

mm/memcontrol.c +0 -0

index 75e74408cc8f..1d8a6dffdc25 100644

mm/memcontrol.c +6 -0

+		css_id_free(&memcg->css);
+		css_put(&memcg->css);
+	}
+}
+
+/**

Quoted 17 lines · Johannes Weiner

kernel/cgroup.c +13 -0

@@ -6205,6 +6205,24 @@ struct cgroup *cgroup_get_from_path(const char *path)
 }
 EXPORT_SYMBOL_GPL(cgroup_get_from_path);
 
+/**
+ * css_id_free - relinquish an existing CSS's ID
+ * @css: the CSS
+ *
+ * This releases the @css's ID and allows it to be recycled while the
+ * CSS continues to exist. This is useful for controllers with state
+ * that extends past a cgroup's lifetime but doesn't need precious ID
+ * address space.
+ *
+ * This invalidates @css->id, and css_from_id() might return NULL or a
+ * new css if the ID has been recycled in the meantime.
+ */
+void css_id_free(struct cgroup_subsys_state *css)

kernel/cgroup.c struct cgroup *cgroup_get_from_path(const char *path) · line 6224

Quoted 4 lines

kernel/cgroup.c +4 -0

+{
+	cgroup_idr_remove(&css->ss->css_idr, css->id);
+	css->id = 0;
+}

I don't quite get why we're trying to free css->id earlier when memcg is gonna be using its private id anyway. From cgroup core side, the id space isn't restricted.

Quoted 4 lines

mm/memcontrol.c +0 -0

index 75e74408cc8f..1d8a6dffdc25 100644

...

mm/memcontrol.c static struct cftype mem_cgroup_legacy_files[] = { · line 4070

Quoted 4 lines

mm/memcontrol.c +4 -0

+static void mem_cgroup_id_put(struct mem_cgroup *memcg)
+{
+	if (atomic_dec_and_test(&memcg->id.ref)) {
+		idr_remove(&mem_cgroup_idr, memcg->id.id);

Maybe this should do "memcg->id.id = 0"?

Quoted 11 lines

mm/memcontrol.c +11 -0

+		css_id_free(&memcg->css);
+		css_put(&memcg->css);
+	}
+}
+
+/**
+ * mem_cgroup_from_id - look up a memcg from a memcg id
+ * @id: the memcg id to look up
+ *
+ * Caller must hold rcu_read_lock().
+ */

mm/memcontrol.c static struct cftype mem_cgroup_legacy_files[] = { · line 4086

Quoted 5 lines

mm/memcontrol.c +5 -0

+struct mem_cgroup *mem_cgroup_from_id(unsigned short id)
+{
+	WARN_ON_ONCE(!rcu_read_lock_held());
+	return id > 0 ? idr_find(&mem_cgroup_idr, id) : NULL;
+}

css_from_id() has it too but I don't think id > 0 test is necessary. We prolly should take it out of css_from_id() too.

It might be useful to add comment explaining why memcg needs private ids.

css_idr allocation starts at 1, so index 0 will never point to an item. css_from_id() currently filters that before asking idr_find(), but idr_find() would also just return NULL, so this is not needed.

Signed-off-by Johannes Weiner <[email protected]>

The memory controller has quite a bit of state that usually outlives the cgroup and pins its CSS until said state disappears. At the same time it imposes a 16-bit limit on the CSS ID space to economically store IDs in the wild. Consequently, when we use cgroups to contain frequent but small and short-lived jobs that leave behind some page cache, we quickly run into the 64k limitations of outstanding CSSs. Creating a new cgroup fails with -ENOSPC while there are only a few, or even no user-visible cgroups in existence.

The memory controller has quite a bit of state that usually outlives the cgroup and pins its CSS until said state disappears. At the same time it imposes a 16-bit limit on the CSS ID space to economically store IDs in the wild. Consequently, when we use cgroups to contain frequent but small and short-lived jobs that leave behind some page cache, we quickly run into the 64k limitations of outstanding CSSs. Creating a new cgroup fails with -ENOSPC while there are only a few, or even no user-visible cgroups in existence.

Although pinning CSSs past cgroup removal is common, there are only two instances that actually need an ID after a cgroup is deleted:
cache shadow entries and swapout records.

Cache shadow entries reference the ID weakly and can deal with the CSS having disappeared when it's looked up later. They pose no hurdle.

Swap-out records do need to pin the css to hierarchically attribute swapins after the cgroup has been deleted; though the only pages that remain swapped out after offlining are tmpfs/shmem pages. And those references are under the user's control, so they are manageable.

This patch introduces a private 16-bit memcg ID and switches swap and cache shadow entries over to using that. This ID can then be recycled after offlining when the CSS remains pinned only by objects that don't specifically need it.

This script demonstrates the problem by faulting one cache page in a new cgroup and deleting it again:

set -e
mkdir -p pages
for x in seq 128000; do
[ $((x % 1000)) -eq 0 ] && echo $x
mkdir /cgroup/foo
echo $$ >/cgroup/foo/cgroup.procs
echo trex >pages/$x
echo $$ >/cgroup/cgroup.procs
rmdir /cgroup/foo
done

When run on an unpatched kernel, we eventually run out of possible IDs even though there are no visible cgroups:

[root@ham ~]# ./cssidstress.sh
[...]
65000
mkdir: cannot create directory '/cgroup/foo': No space left on device

After this patch, the IDs get released upon cgroup destruction and the cache and css objects get released once memory reclaim kicks in.

Signed-off-by Johannes Weiner <[email protected]>

The memory controller has quite a bit of state that usually outlives the cgroup and pins its CSS until said state disappears. At the same time it imposes a 16-bit limit on the CSS ID space to economically store IDs in the wild. Consequently, when we use cgroups to contain frequent but small and short-lived jobs that leave behind some page cache, we quickly run into the 64k limitations of outstanding CSSs. Creating a new cgroup fails with -ENOSPC while there are only a few, or even no user-visible cgroups in existence.

Although pinning CSSs past cgroup removal is common, there are only two instances that actually need an ID after a cgroup is deleted:
cache shadow entries and swapout records.

Cache shadow entries reference the ID weakly and can deal with the CSS having disappeared when it's looked up later. They pose no hurdle.

Swap-out records do need to pin the css to hierarchically attribute swapins after the cgroup has been deleted; though the only pages that remain swapped out after offlining are tmpfs/shmem pages. And those references are under the user's control, so they are manageable.

This patch introduces a private 16-bit memcg ID and switches swap and cache shadow entries over to using that. This ID can then be recycled after offlining when the CSS remains pinned only by objects that don't specifically need it.

This script demonstrates the problem by faulting one cache page in a new cgroup and deleting it again:

set -e
mkdir -p pages
for x in seq 128000; do
[ $((x % 1000)) -eq 0 ] && echo $x
mkdir /cgroup/foo
echo $$ >/cgroup/foo/cgroup.procs
echo trex >pages/$x
echo $$ >/cgroup/cgroup.procs
rmdir /cgroup/foo
done

When run on an unpatched kernel, we eventually run out of possible IDs even though there are no visible cgroups:

[root@ham ~]# ./cssidstress.sh
[...]
65000
mkdir: cannot create directory '/cgroup/foo': No space left on device

After this patch, the IDs get released upon cgroup destruction and the cache and css objects get released once memory reclaim kicks in.

Signed-off-by Johannes Weiner <[email protected]>

mm/memcontrol.c +15 -0

index 75e74408cc8f..dc92b2df2585 100644
@@ -4057,6 +4057,60 @@ static struct cftype mem_cgroup_legacy_files[] = {
 	{ },	/* terminate */
 };
 
+/*
+ * Private memory cgroup IDR
+ *
+ * Swap-out records and page cache shadow entries need to store memcg
+ * references in constrained space, so we maintain an ID space that is
+ * limited to 16 bit (MEM_CGROUP_ID_MAX), limiting the total number of
+ * memory-controlled cgroups to 64k.
+ *
+ * However, there usually are many references to the oflline CSS after
+ * the cgroup has been destroyed, such as page cache or reclaimable
+ * slab objects, that don't need to hang on to the ID. We want to keep
+ * those dead CSS from occupying IDs, or we might quickly exhaust the
+ * relatively small ID space and prevent the creation of new cgroups
+ * even when there are much fewer than 64k cgroups - possibly none.
+ *

Quoted 2 lines · Johannes Weiner

After this patch, the IDs get released upon cgroup destruction and the cache and css objects get released once memory reclaim kicks in.

With 65K cgroups it will take the reclaimer a substantial amount of time to iterate over all of them, which might result in latency spikes. Probably, to avoid that, we could move pages from a dead cgroup's lru to its parent's one on offline while still leaving dead cgroups pinned, like we do in case of list_lru entries.

Quoted 1 line

Signed-off-by Johannes Weiner <[email protected]>

Reviewed-by Vladimir Davydov <[email protected]>

Quoted 1 line

+static struct idr mem_cgroup_idr;

static DEFINE_IDR(mem_cgroup_idr);

The memory controller has quite a bit of state that usually outlives the cgroup and pins its CSS until said state disappears. At the same time it imposes a 16-bit limit on the CSS ID space to economically store IDs in the wild. Consequently, when we use cgroups to contain frequent but small and short-lived jobs that leave behind some page cache, we quickly run into the 64k limitations of outstanding CSSs. Creating a new cgroup fails with -ENOSPC while there are only a few, or even no user-visible cgroups in existence.

Although pinning CSSs past cgroup removal is common, there are only two instances that actually need a CSS ID after a cgroup is deleted:
cache shadow entries and swapout records.

Cache shadow entries reference the ID weakly and can deal with the CSS having disappeared when it's looked up later. They pose no hurdle.

Swap-out records do need to pin the css to hierarchically attribute swapins after the cgroup has been deleted; though the only pages that remain swapped out after a process exits are tmpfs/shmem pages. Those references are under the user's control and thus manageable.

This patch introduces a private 16bit memcg ID and switches swap and cache shadow entries over to using that. It then decouples the CSS lifetime from the CSS ID lifetime, such that a CSS ID can be recycled when the CSS is only pinned by common objects that don't need an ID.

Quoted 21 lines · Johannes Weiner

The memory controller has quite a bit of state that usually outlives the cgroup and pins its CSS until said state disappears. At the same time it imposes a 16-bit limit on the CSS ID space to economically store IDs in the wild. Consequently, when we use cgroups to contain frequent but small and short-lived jobs that leave behind some page cache, we quickly run into the 64k limitations of outstanding CSSs. Creating a new cgroup fails with -ENOSPC while there are only a few, or even no user-visible cgroups in existence.

Although pinning CSSs past cgroup removal is common, there are only two instances that actually need a CSS ID after a cgroup is deleted:
cache shadow entries and swapout records.

Cache shadow entries reference the ID weakly and can deal with the CSS having disappeared when it's looked up later. They pose no hurdle.

Swap-out records do need to pin the css to hierarchically attribute swapins after the cgroup has been deleted; though the only pages that remain swapped out after a process exits are tmpfs/shmem pages. Those references are under the user's control and thus manageable.

Quoted 4 lines

This patch introduces a private 16bit memcg ID and switches swap and cache shadow entries over to using that. It then decouples the CSS lifetime from the CSS ID lifetime, such that a CSS ID can be recycled when the CSS is only pinned by common objects that don't need an ID.

There's already id which is only used for online memory cgroups - it's kmemcg_id. May be, instead of introducing one more idr, we could name it generically and reuse it for shadow entries?

Regarding swap entries, would it really make much difference if we used 4 bytes per swap page instead of 2? For a 100 GB swap it'd increase overhead from 50 MB up to 100 MB, which still doesn't seem too much IMO, so may be just use plain unrestricted css->id for swap entries?

The memory controller has quite a bit of state that usually outlives the cgroup and pins its CSS until said state disappears. At the same time it imposes a 16-bit limit on the CSS ID space to economically store IDs in the wild. Consequently, when we use cgroups to contain frequent but small and short-lived jobs that leave behind some page cache, we quickly run into the 64k limitations of outstanding CSSs. Creating a new cgroup fails with -ENOSPC while there are only a few, or even no user-visible cgroups in existence.

Although pinning CSSs past cgroup removal is common, there are only two instances that actually need a CSS ID after a cgroup is deleted:
cache shadow entries and swapout records.

Cache shadow entries reference the ID weakly and can deal with the CSS having disappeared when it's looked up later. They pose no hurdle.

Swap-out records do need to pin the css to hierarchically attribute swapins after the cgroup has been deleted; though the only pages that remain swapped out after a process exits are tmpfs/shmem pages. Those references are under the user's control and thus manageable.

This patch introduces a private 16bit memcg ID and switches swap and cache shadow entries over to using that. It then decouples the CSS lifetime from the CSS ID lifetime, such that a CSS ID can be recycled when the CSS is only pinned by common objects that don't need an ID.

This script demonstrates the problem by faulting one cache page in a new cgroup and deleting it again:

set -e
mkdir -p pages
for x in seq 128000; do
[ $((x % 1000)) -eq 0 ] && echo $x
mkdir /cgroup/foo
echo $$ >/cgroup/foo/cgroup.procs
echo trex >pages/$x
echo $$ >/cgroup/cgroup.procs
rmdir /cgroup/foo
done

When run on an unpatched kernel, we eventually run out of possible CSS IDs even though there is no visible cgroup existing anymore:

[root@ham ~]# ./cssidstress.sh
[...]
65000
mkdir: cannot create directory '/cgroup/foo': No space left on device

After this patch, the CSS IDs get released upon cgroup destruction and the cache and css objects get released once memory reclaim kicks in.