Add memory.max.effective for application's allocators
Subsystems (4)
Michal Koutný
maintainerv5 cover Add memory.max.effective for application's allocators
Some applications use memory cgroup limits to scale their own memory needs. Reading of the immediate membership cgroup's memory.max is not sufficient because of possible ancestral limits. The application could traverse upwards to figure out the tightest limit but this would not work in cgroup namespace where the view of cgroup hierarchy is incomplete and the limit may apply from outer world. Additionally, applications should respond to limit changes.
(cgroup v1 used memory.stat:hierarchical_memory_limit to report the value but there's no such counterpart in cgroup v2 memory.stat.)
Introduce a new memcg attribute file that contains the effective value of memory limit for given cgroup (following cpuset.cpus.effective pattern) and that sends notifications like memory.events when the effective limit changes.
Reasons for RFC:
1) Should global limit be included? (And respond to memory hotplug?)
2) Is swap.max.effective needed? (in v2 without memsw accounting)
3) Should memory.high be also handled?
4) What would be an alternative?
My answers to RFC:
1) No (there's no memory.max in global root memcg)
2) No (app doesn't have full control of memory that's swapped out)
3) No (scaling the allocator against the "soft" limit could end up in dynamics difficult to reason and admin)
4)
- PSI (too obscure for traditional users but better semantics for limit shrinking)
- memory.stat field (like v1 but separate attribute is better for notifications, cpuset precedent)
Changes from v4 (https://lore.kernel.org/r/[email protected])
- split the patch for swap.max.effetive
- add Documentation/
- reword commit messages
- add notification support
Michal Koutný (3):
memcg: Add memory.max.effective attribute
memcg: Add memory.swap.max.effective like hierarchical_memsw_limit
memcg: Notify on memory.max.effective changes
Documentation/admin-guide/cgroup-v2.rst | 6 ++++ include/linux/memcontrol.h | 2 ++ mm/memcontrol.c | 46 +++++++++++++++++++++++++ 3 files changed, 54 insertions(+)
base-commit: 2df0193e62cf887f373995fb8a91068562784adc
Signature · 1 line
2.45.1
Michal Koutný
maintainerRFC PATCH v5 1/3 memcg: Add memory.max.effective attribute
Some applications use memory cgroup limits to scale their own memory needs. Reading of the immediate membership cgroup's memory.max is not sufficient because of possible ancestral limits. The application could traverse upwards to figure out the tightest limit but this would not work in cgroup namespace where the view of cgroup hierarchy is incomplete and the limit may apply from outer world.
(cgroup v1 used memory.stat:hierarchical_memory_limit to report the value but there's no such counterpart in cgroup v2 memory.stat.)
Introduce a new memcg attribute file that contains the effective value of memory limit for given cgroup (following cpuset.cpus.effective pattern).
--- Documentation/admin-guide/cgroup-v2.rst | 6 ++++++ mm/memcontrol.c | 18 ++++++++++++++++++ 2 files changed, 24 insertions(+)
Documentation/admin-guide/cgroup-v2.rst +6 -0
@@ -1293,6 +1293,12 @@ PAGE_SIZE multiple when read back. Caller could retry them differently, return into userspace as -ENOMEM or silently ignore in cases like disk readahead. + memory.max.effective + A read-only file that provides effective value of cgroup's hard usage + limit. It incorporates limits of all ancestors, even those not visible + in cgroupns. The value change in this file generates a file modified + event. + memory.reclaim A write-only nested-keyed file which exists for all cgroups.
mm/memcontrol.c +18 -0
@@ -7065,6 +7065,19 @@ static ssize_t memory_max_write(struct kernfs_open_file *of,return nbytes; } +static int memory_max_effective_show (struct seq_file *m,void *v) +{ +unsigned long memory; +struct mem_cgroup *mi; + +/* Hierarchical information */ + memory =PAGE_COUNTER_MAX ; +for (mi =mem_cgroup_from_seq (m); mi; mi =parent_mem_cgroup (mi)) + memory =min (memory,READ_ONCE (mi->memory.max)); + +return seq_puts_memcg_tunable (m, memory); +} +/* * Note: don't forget to update the 'samples/cgroup/memcg_event_listener' * if any new events become available.
@@ -7259,6 +7272,11 @@ static struct cftype memory_files[] = { .seq_show = memory_max_show, .write = memory_max_write, }, + { + .name = "max.effective", + .flags = CFTYPE_NOT_ON_ROOT, + .seq_show = memory_max_effective_show, + }, { .name = "events", .flags = CFTYPE_NOT_ON_ROOT,
Signature · 1 line
2.45.1
Michal Koutný
maintainerRFC PATCH v5 2/3 memcg: Add memory.swap.max.effective like hierarchical_memsw_limit
In reply to Michal Koutnýcgroup v1 used memory.stat:hierarchical_memsw_limit to report the value of effecitve memsw limit. cgroup v2 has no combined charing but swap.max limit, add a new memcg attribute file that contains the effective value of memory limit for given cgroup (following cpuset.cpus.effective pattern) for cases when whole hierarchy cannot be traversed up due to cgroupns visibility.
--- mm/memcontrol.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
mm/memcontrol.c +18 -0
@@ -8279,6 +8279,19 @@ static ssize_t swap_max_write(struct kernfs_open_file *of,return nbytes; } +static int swap_max_effective_show (struct seq_file *m,void *v) +{ +unsigned long swap; +struct mem_cgroup *mi; + +/* Hierarchical information */ + swap =PAGE_COUNTER_MAX ; +for (mi =mem_cgroup_from_seq (m); mi; mi =parent_mem_cgroup (mi)) + swap =min (swap,READ_ONCE (mi->swap.max)); + +return seq_puts_memcg_tunable (m, swap); +} +static int swap_events_show (struct seq_file *m,void *v) {struct mem_cgroup *memcg =mem_cgroup_from_seq (m);
@@ -8311,6 +8324,11 @@ static struct cftype swap_files[] = { .seq_show = swap_max_show, .write = swap_max_write, }, + { + .name = "swap.max.effective", + .flags = CFTYPE_NOT_ON_ROOT, + .seq_show = swap_max_effective_show, + }, { .name = "swap.peak", .flags = CFTYPE_NOT_ON_ROOT,
Signature · 1 line
2.45.1
Michal Koutný
maintainerRFC PATCH v5 3/3 memcg: Notify on memory.max.effective changes
In reply to Michal KoutnýWhen users are interested in cgroup's effective limit, they typically respond to the value somehow and therefore they should be notified when the value changes. Use the standard menchanism of triggering a modification of respective cgroup file.
--- include/linux/memcontrol.h | 2 ++ mm/memcontrol.c | 10 ++++++++++ 2 files changed, 12 insertions(+)
include/linux/memcontrol.h +2 -0
@@ -232,6 +232,8 @@ struct mem_cgroup {/* memory.events and memory.events.local */ struct cgroup_file events_file;struct cgroup_file events_local_file; +/* memory.max.effective */ +struct cgroup_file mem_max_file;/* handle for "memory.swap.events" */ struct cgroup_file swap_events_file;
mm/memcontrol.c +10 -0
@@ -7022,6 +7022,7 @@ static ssize_t memory_max_write(struct kernfs_open_file *of, char *buf, size_t nbytes, loff_t off) {struct mem_cgroup *memcg =mem_cgroup_from_css (of_css (of)); +struct mem_cgroup *iter;unsigned int nr_reclaims =MAX_RECLAIM_RETRIES ;bool drained = false;unsigned long max;
@@ -7061,6 +7062,14 @@ static ssize_t memory_max_write(struct kernfs_open_file *of,break ; } +/* +* Notification about limit tightening, not about coming OOMs, so it +* can be after reclaim. +*/ +for_each_mem_cgroup_tree (iter, memcg) { +cgroup_file_notify (&iter->mem_max_file); + } +memcg_wb_domain_size_changed (memcg);return nbytes; }
@@ -7275,6 +7284,7 @@ static struct cftype memory_files[] = { { .name = "max.effective", .flags = CFTYPE_NOT_ON_ROOT, + .file_offset = offsetof(struct mem_cgroup, mem_max_file), .seq_show = memory_max_effective_show, }, {
Signature · 1 line
2.45.1
Roman Gushchin
maintainerQuoted 7 lines · Michal Koutný
Some applications use memory cgroup limits to scale their own memory needs. Reading of the immediate membership cgroup's memory.max is not sufficient because of possible ancestral limits. The application could traverse upwards to figure out the tightest limit but this would not work in cgroup namespace where the view of cgroup hierarchy is incomplete and the limit may apply from outer world. Additionally, applications should respond to limit changes.
If the goal is to detect how much memory would it be possible to allocate, I'm not sure that knowing all memory.max limits upper in the hierarchy really buys anything without knowing actual usages and a potential for memory reclaim across the entire tree.
E.g.:
A (max = 100G)
| \
B C
C's effective max will come out as 100G, but if B.anon_usage = 100G and there is no swap, the actual number is 0.
But if it's more about exploring the "invisible" part of the cgroup tree configuration, it makes sense to me.
Not sure about the naming, maybe something like memory.tree.max or memory.parent.max or even memory.hierarchical.max is a better fit.
Thanks!
Quoted 5 lines · Roman Gushchin
If the goal is to detect how much memory would it be possible to allocate, I'm not sure that knowing all memory.max limits upper in the hierarchy really buys anything without knowing actual usages and a potential for memory reclaim across the entire tree.
Quoted 8 lines
E.g.:
A (max = 100G)
| \
B CC's effective max will come out as 100G, but if B.anon_usage = 100G and there is no swap, the actual number is 0.
Yes, it would be better to subtract the used memory from ancestor (and thus even current) cgroups. The original use case of this feature is for cloud nodes running a single Java JVM where the sibling cgroups are not an issue.
Jan Kratochvil
Michal Koutný
maintainerHello.
Quoted 2 lines · Jan Kratochvil
Yes, it would be better to subtract the used memory from ancestor (and thus even current) cgroups.
Then it becomes a more dynamic characterstics and it leads to calculations of available memory. I share a link [1] for completeness and to prevent repeated discussions (that past one ended up with no memory.stat:avail).
Quoted 2 lines
The original use case of this feature is for cloud nodes running a single Java JVM where the sibling cgroups are not an issue.
IIUC, it's a tree like this:
O
/ | \
A B C // B:memory.max < O:memory.max
|
...
|
W // workloadThis picture made me realize that memory controller may not be even enabled all the way down from B to W, i.e. W would have no memory.max.effective, IOW memory.* attribute would not be the right place for such an value. That would even apply in the apparently purposeful case if there was a cgroup NS boundary between B and W.
(At least in the proposed implementation, memory.* file would have to be decoupled from memory controller, similarly to e.g. cpu.stat:usage_usec.)
Jan, do I get the tree shape right? Are B and W in different cgroup namespaces?
Thanks,
Michal
[1] https://lore.kernel.org/all/alpine.DEB.2.23.453.2007142018150.2667860@chino.kir.corp.google.com/