mm, memcg: cg2 memory{.swap,}.peak write handlers
Subsystems (6)
This is an updated patchset following some excellent comments from Roman and Longman. [1]
As suggested, I've broken this into two commits:
1) with the implementation changes
2) extending the tests tests
I haven't been able to induce anything problematic, but I'm a bit unclear as to whether there's reference counting on cgroups such that we don't need to handle the case where the cgroup is freed before the one of the peak files is closed.
Documentation/admin-guide/cgroup-v2.rst | 26 ++- include/linux/cgroup.h | 8 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 11 +- kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 129 +++++++++++++-- mm/page_counter.c | 36 ++++- tools/testing/selftests/cgroup/cgroup_util.c | 22 +++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + tools/testing/selftests/cgroup/test_memcontrol.c | 227 ++++++++++++++++++++++++++- 11 files changed, 444 insertions(+), 31 deletions(-)
[1]: https://lore.kernel.org/cgroups/[email protected]/T/
Thank you for your efforts and reviews,
David Finkel
Senior Principal Software Engineer
Vimeo Inc.
PATCH 1/2 mm, memcg: cg2 memory{.swap,}.peak write handlers
Other mechanisms for querying the peak memory usage of either a process or v1 memory cgroup allow for resetting the high watermark. Restore parity with those mechanisms, but with a less racy API.
For example:
- Any write to memory.max_usage_in_bytes in a cgroup v1 mount resets the high watermark.
- writing "5" to the clear_refs pseudo-file in a processes's proc directory resets the peak RSS.
This change is an evolution of a previous patch, which mostly copied the cgroup v1 behavior, however, there were concerns about races/ownership issues with a global reset, so instead this change makes the reset filedescriptor-local.
Writing a specific string to the memory.peak and memory.swap.peak pseudo-files reset the high watermark to the current usage for subsequent reads through that same fd.
Notably, following Johannes's suggestion, this implementation moves the O(fds that have written) behavior onto the fd write(2) path. Instead, on the page-allocation path, we simply add one additional watermark to conditionally bump per-hierarchy level in the page-counter.
Additionally, this takes Longman's suggestion of nesting the page-charging-path checks for the two watermarks to reduce the number of common-case comparisons.
This behavior is particularly useful for work scheduling systems that need to track memory usage of worker processes/cgroups per-work-item. Since memory can't be squeezed like CPU can (the OOM-killer has opinions), these systems need to track the peak memory usage to compute system/container fullness when binpacking workitems.
Most notably, Vimeo's use-case involves a system that's doing global binpacking across many Kubernetes pods/containers, and while we can use PSI for some local decisions about overload, we strive to avoid packing workloads too tightly in the first place. To facilitate this, we track the peak memory usage. However, since we run with long-lived workers (to amortize startup costs) we need a way to track the high watermark while a work-item is executing. Polling runs the risk of missing short spikes that last for timescales below the polling interval, and peak memory tracking at the cgroup level is otherwise perfect for this use-case.
As this data is used to ensure that binpacked work ends up with sufficient headroom, this use-case mostly avoids the inaccuracies surrounding reclaimable memory.
--- Documentation/admin-guide/cgroup-v2.rst | 26 ++-- include/linux/cgroup.h | 8 ++ include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 11 +- kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 129 +++++++++++++++++-- mm/page_counter.c | 36 +++++- tools/testing/selftests/cgroup/cgroup_util.c | 22 ++++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + 10 files changed, 225 insertions(+), 23 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +18 -8
@@ -1322,11 +1322,16 @@ PAGE_SIZE multiple when read back. reclaim induced by memory.reclaim. memory.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max memory usage recorded for the cgroup and its descendants since + either the creation of the cgroup or the most recent reset for that FD. - The max memory usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of the string "reset" to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. + Attempts to write any other non-empty string will return EINVAL + (modulo leading and trailing whitespace). memory.oom.group A read-write single value file which exists on non-root
@@ -1652,11 +1657,16 @@ PAGE_SIZE multiple when read back. Healthy workloads are not expected to reach this limit. memory.swap.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max swap usage recorded for the cgroup and its descendants since + the creation of the cgroup or the most recent reset for that FD. - The max swap usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of the string "reset" to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. + Attempts to write any other non-empty string will return EINVAL + (modulo leading and trailing whitespace). memory.swap.max A read-write single value file which exists on non-root
include/linux/cgroup.h +8 -0
@@ -12,6 +12,7 @@#include <linux/sched.h> #include <linux/cpumask.h> #include <linux/nodemask.h> +#include <linux/list.h> #include <linux/rculist.h> #include <linux/cgroupstats.h> #include <linux/fs.h>
@@ -855,4 +856,11 @@ static inline void cgroup_bpf_put(struct cgroup *cgrp) {}struct cgroup *task_get_cgroup1 (struct task_struct *tsk,int hierarchy_id); +struct memcg_peak_mem_ctx { +long local_watermark; +struct list_head peers; +}; + +struct memcg_peak_mem_ctx *memcg_extract_peak_mem_ctx (struct kernfs_open_file *of); +#endif /* _LINUX_CGROUP_H */
include/linux/memcontrol.h +5 -0
@@ -198,6 +198,11 @@ struct mem_cgroup {struct page_counter kmem;/* v1 only */ struct page_counter tcpmem;/* v1 only */ +/* lists of memcg peak watching contexts on swap and memory */ +struct list_head peak_memory_local_watermark_watchers; +struct list_head peak_swap_local_watermark_watchers; +spinlock_t swap_memory_peak_watchers_lock; +/* Range enforcement for interrupt charges */ struct work_struct high_work;
include/linux/page_counter.h +10 -1
@@ -26,6 +26,7 @@ struct page_counter {atomic_long_t children_low_usage;unsigned long watermark; +unsigned long local_watermark;/* track min of fd-local resets */ unsigned long failcnt;/* Keep all the read most fields in a separete cacheline. */
@@ -78,7 +79,15 @@ int page_counter_memparse(const char *buf, const char *max,static inline void page_counter_reset_watermark (struct page_counter *counter) { - counter->watermark =page_counter_read (counter); +unsigned long cur =page_counter_read (counter); + + counter->watermark = cur; + counter->local_watermark = cur; +} + +static inline void page_counter_reset_local_watermark (struct page_counter *counter) +{ + counter->local_watermark =page_counter_read (counter); }#endif /* _LINUX_PAGE_COUNTER_H */
kernel/cgroup/cgroup-internal.h +2 -0
@@ -81,6 +81,8 @@ struct cgroup_file_ctx {struct {struct cgroup_pidlist *pidlist; } procs1; + +struct memcg_peak_mem_ctx peak; };/*
kernel/cgroup/cgroup.c +7 -0
@@ -1964,6 +1964,13 @@ static int cgroup2_parse_param(struct fs_context *fc, struct fs_parameter *paramreturn -EINVAL ; } +struct memcg_peak_mem_ctx *memcg_extract_peak_mem_ctx (struct kernfs_open_file *of) +{ +struct cgroup_file_ctx *ctx = of->priv; + +return &ctx->peak; +} +static void apply_cgroup_root_flags (unsigned int root_flags) {if (current->nsproxy->cgroup_ns == &init_cgroup_ns) {
mm/memcontrol.c +120 -9
176 lines not on this page · raw message
mm/page_counter.c +31 -5
@@ -79,9 +79,22 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages)/* * This is indeed racy, but we can live with some * inaccuracy in the watermark. +* +* Notably, we have two watermarks to allow for both a globally +* visible peak and one that can be reset at a smaller scope. +* +* Since we reset both watermarks when the global reset occurs, +* we can guarantee that watermark >= local_watermark, so we +* don't need to do both comparisons every time. +* +* On systems with branch predictors, the inner condition should +* be almost free. */ -if (new >READ_ONCE (c->watermark)) -WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) { +WRITE_ONCE (c->local_watermark, new); +if (new >READ_ONCE (c->watermark)) +WRITE_ONCE (c->watermark, new); + } } }
@@ -131,10 +144,23 @@ bool page_counter_try_charge(struct page_counter *counter,propagate_protected_usage (c, new);/* * Just like with failcnt, we can live with some -* inaccuracy in the watermark. +* inaccuracy in the watermarks. +* +* Notably, we have two watermarks to allow for both a globally +* visible peak and one that can be reset at a smaller scope. +* +* Since we reset both watermarks when the global reset occurs, +* we can guarantee that watermark >= local_watermark, so we +* don't need to do both comparisons every time. +* +* On systems with branch predictors, the inner condition should +* be almost free. */ -if (new >READ_ONCE (c->watermark)) -WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) { +WRITE_ONCE (c->local_watermark, new); +if (new >READ_ONCE (c->watermark)) +WRITE_ONCE (c->watermark, new); + } }return true;
tools/testing/selftests/cgroup/cgroup_util.c +22 -0
@@ -141,6 +141,16 @@ long cg_read_long(const char *cgroup, const char *control)return atol (buf); } +long cg_read_long_fd (int fd) +{ +char buf[128 ]; + +if (pread (fd, buf,sizeof (buf),0 ) <=0 ) +return -1 ; + +return atol (buf); +} +long cg_read_key_long (const char *cgroup,const char *control,const char *key) {char buf[PAGE_SIZE ];
@@ -183,6 +193,18 @@ int cg_write(const char *cgroup, const char *control, char *buf)return ret == len ?0 : ret; } +/* +* Returns fd on success, or -1 on failure. +* (fd should be closed with close() as usual) +*/ +int cg_open (const char *cgroup,const char *control,int flags) +{ +char path[PATH_MAX ]; + +snprintf (path,sizeof (path),"%s/%s" , cgroup, control); +return open (path, flags); +} +int cg_write_numeric (const char *cgroup,const char *control,long value) {char buf[64 ];
tools/testing/selftests/cgroup/cgroup_util.h +2 -0
@@ -34,9 +34,11 @@ extern int cg_read_strcmp(const char *cgroup, const char *control,extern int cg_read_strstr (const char *cgroup,const char *control,const char *needle);extern long cg_read_long (const char *cgroup,const char *control); +extern long cg_read_long_fd (int fd);long cg_read_key_long (const char *cgroup,const char *control,const char *key);extern long cg_read_lc (const char *cgroup,const char *control);extern int cg_write (const char *cgroup,const char *control,char *buf); +extern int cg_open (const char *cgroup,const char *control,int flags);int cg_write_numeric (const char *cgroup,const char *control,long value); extern int cg_run(const char *cgroup, int (*fn)(const char *cgroup, void *arg),
Signature · 1 line
2.40.1
Johannes Weiner
maintainerHi David,
thanks for pursuing this! A couple of comments below.
Quoted 8 lines · David Finkel
Documentation/admin-guide/cgroup-v2.rst +2
-2@@ -1322,11 +1322,16 @@ PAGE_SIZE multiple when read back. reclaim induced by memory.reclaim. memory.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. +
Documentation/admin-guide/cgroup-v2.rst PAGE_SIZE multiple when read back. · line 1334
Quoted 10 lines
Documentation/admin-guide/cgroup-v2.rst +7
-2+ The max memory usage recorded for the cgroup and its descendants since + either the creation of the cgroup or the most recent reset for that FD. - The max memory usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of the string "reset" to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. + Attempts to write any other non-empty string will return EINVAL + (modulo leading and trailing whitespace).
Why not allow any write to reset? This makes it harder to use, and I'm not sure accidental writes are a likely mistake to make.
Quoted 13 lines
include/linux/cgroup.h +1
-0@@ -12,6 +12,7 @@#include <linux/sched.h> #include <linux/cpumask.h> #include <linux/nodemask.h> +#include <linux/list.h> #include <linux/rculist.h> #include <linux/cgroupstats.h> #include <linux/fs.h>
include/linux/cgroup.h static inline void cgroup_bpf_put(struct cgroup *cgrp) {} · line 862
Quoted 7 lines
include/linux/cgroup.h +4
-0@@ -855,4 +856,11 @@ static inline void cgroup_bpf_put(struct cgroup *cgrp) {}struct cgroup *task_get_cgroup1 (struct task_struct *tsk,int hierarchy_id); +struct memcg_peak_mem_ctx { +long local_watermark; +struct list_head peers; +};
Since this is generic cgroup code, and can be conceivably used by other controllers, let's keep the naming generic as well. How about:
struct cgroup_of_peak {
long value;
struct list_head list;
};cgroup-defs.h would be a better place for it.
include/linux/cgroup.h static inline void cgroup_bpf_put(struct cgroup *cgrp) {} · line 864
Quoted 1 line
include/linux/cgroup.h +1
-0+struct memcg_peak_mem_ctx *memcg_extract_peak_mem_ctx (struct kernfs_open_file *of);
of_peak()
include/linux/memcontrol.h struct mem_cgroup { · line 204
Quoted 12 lines
include/linux/memcontrol.h +4
-0@@ -198,6 +198,11 @@ struct mem_cgroup {struct page_counter kmem;/* v1 only */ struct page_counter tcpmem;/* v1 only */ +/* lists of memcg peak watching contexts on swap and memory */ +struct list_head peak_memory_local_watermark_watchers; +struct list_head peak_swap_local_watermark_watchers; +spinlock_t swap_memory_peak_watchers_lock;
These names are too long. How about:
/* Registered local usage peak watchers */ struct list_head memory_peaks; struct list_head swap_peaks; spinlock_t peaks_lock;
Quoted 13 lines
include/linux/page_counter.h +1
-0@@ -26,6 +26,7 @@ struct page_counter {atomic_long_t children_low_usage;unsigned long watermark; +unsigned long local_watermark;/* track min of fd-local resets */ unsigned long failcnt;/* Keep all the read most fields in a separete cacheline. */
include/linux/page_counter.h int page_counter_memparse(const char *buf, const char *max, · line 82
Quoted 5 lines
include/linux/page_counter.h +1
-1@@ -78,7 +79,15 @@ int page_counter_memparse(const char *buf, const char *max,static inline void page_counter_reset_watermark (struct page_counter *counter) { - counter->watermark =page_counter_read (counter); +unsigned long cur =page_counter_read (counter);
cur -> usage
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6917
Quoted 9 lines
include/linux/page_counter.h +2
-2@@ -6907,12 +6912,109 @@ static u64 memory_current_read(struct cgroup_subsys_state *css,return (u64 )page_counter_read (&memcg->memory) *PAGE_SIZE ; } -static u64 memory_peak_read (struct cgroup_subsys_state *css, -struct cftype *cft) +inline int swap_memory_peak_show ( +struct seq_file *sf,void *v,bool swap_cg) {
Leave inlining to the compiler. Just static int.
The name can be simply peak_show().
Customary coding style is to line wrap at the last parameter that fits. Don't wrap if the line fits within 80 cols.
static int peak_show(struct seq_file *sf, void *v, ...,
...)
{
...
}Quoted 14 lines
include/linux/page_counter.h +11
-1+struct cgroup_subsys_state *css =seq_css (sf);struct mem_cgroup *memcg =mem_cgroup_from_css (css); +struct page_counter *pc; +struct kernfs_open_file *of = sf->private; +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of); +s64 fd_peak = ctx->local_watermark; -return (u64 )memcg->memory.watermark *PAGE_SIZE ; +if (swap_cg) + pc = &memcg->swap; +else + pc = &memcg->memory; + +if (fd_peak ==-1 ) {
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6940
Quoted 10 lines
include/linux/page_counter.h +10
-0+seq_printf (sf,"%llu\n" , (u64 )pc->watermark *PAGE_SIZE ); +return 0 ; + } + +s64 pc_peak = pc->local_watermark; +s64 wm = fd_peak > pc_peak ? fd_peak : pc_peak; + +seq_printf (sf,"%lld\n" , wm *PAGE_SIZE ); +return 0 ; +}
As per Roman's feedback, don't mix decls and code.
You can simplify it by extracting css and memcg in the callers, then pass the right struct page counter *pc directly.
That should eliminate most local variables as well.
static int peak_show(struct seq_file *sf, void *v, struct page_counter *pc)
{
struct cgroup_of_peak *ofp = of_peak(sf->private);
u64 peak;
/* User wants global or local peak? */
if (ofp->value == -1)
peak = pc->watermark;
else
peak = max(ofp->value, pc->local_watermark);
seq_printf(sf, "%lld\n", peak * PAGE_SIZE);
}include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6943
Quoted 3 lines
include/linux/page_counter.h +3
-0+static int memory_peak_show (struct seq_file *sf,void *v) +{ +return swap_memory_peak_show (sf, v, false);
And then do:
struct mem_cgroup *memcg = mem_cgroup_from_css(seq_css(sf)); return peak_show(sf, v, &memcg->memory);
Then do the same with ... &memcg->swap.
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6947
Quoted 4 lines
include/linux/page_counter.h +4
-0+inline ssize_t swap_memory_peak_write ( +struct kernfs_open_file *of, +char *buf,size_t nbytes,loff_t off,bool swap_cg) +{
Same feedback as above. Please don't inline explicitly (unless it really is measurably a performance improvement in a critical path), and stick to surrounding coding style.
Here too, pass page_counter directly and save the branches.
Quoted 16 lines
include/linux/page_counter.h +16
-0+unsigned long cur; +struct memcg_peak_mem_ctx *peer_ctx; +struct mem_cgroup *memcg =mem_cgroup_from_css (of_css (of)); +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of); +struct page_counter *pc; +struct list_head *watchers, *pos; + + buf =strstrip (buf); +/* Only allow "reset" to keep the API clear */ +if (strcmp (buf,"reset" )) +return -EINVAL ; + +if (swap_cg) { + pc = &memcg->swap; + watchers = &memcg->peak_swap_local_watermark_watchers; + }else {
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6973
Quoted 10 lines
include/linux/page_counter.h +10
-0+ pc = &memcg->memory; + watchers = &memcg->peak_memory_local_watermark_watchers; + } + +spin_lock (&memcg->swap_memory_peak_watchers_lock); + +page_counter_reset_local_watermark (pc); + cur = pc->local_watermark; + +list_for_each (pos, watchers) {
list_for_each_entry()
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6977
Quoted 4 lines
include/linux/page_counter.h +4
-0+ peer_ctx =list_entry (pos,typeof (*ctx), peers); +if (cur > peer_ctx->local_watermark) + peer_ctx->local_watermark = cur; + }
I don't think this is quite right. local_peak could be higher than the current usage when a new watcher shows up. The other watchers should retain the higher local_peak, not the current usage.
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6983
Quoted 6 lines
include/linux/page_counter.h +6
-0+ +if (ctx->local_watermark ==-1 ) +/* only append to the list if we're not already there */ +list_add_tail (&ctx->peers, watchers); + + ctx->local_watermark = cur;
This makes me think that page_counter_reset_local_watermark() is not a good helper. It obscures what's going on. Try without it.
AFAICS the list ordering doesn't matter, so keep it simple and use a plain list_add().
/* * A new local peak is being tracked in pc->local_watermark. * Save current local peak in all watchers. */ list_for_each_entry(pos, ...) if (pc->local_watermark > pos->value) pos->value = pc->local_watermark; pc->local_watermark = page_counter_read(pc); /* Initital write, register watcher */ if (ofp->value == -1) list_add() ofp->value = pc->local_watermark;
Quoted 35 lines
mm/page_counter.c +16
-3@@ -79,9 +79,22 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages)/* * This is indeed racy, but we can live with some * inaccuracy in the watermark. +* +* Notably, we have two watermarks to allow for both a globally +* visible peak and one that can be reset at a smaller scope. +* +* Since we reset both watermarks when the global reset occurs, +* we can guarantee that watermark >= local_watermark, so we +* don't need to do both comparisons every time. +* +* On systems with branch predictors, the inner condition should +* be almost free. */ -if (new >READ_ONCE (c->watermark)) -WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) { +WRITE_ONCE (c->local_watermark, new); +if (new >READ_ONCE (c->watermark)) +WRITE_ONCE (c->watermark, new); + } } }@@ -131,10 +144,23 @@ bool page_counter_try_charge(struct page_counter *counter,propagate_protected_usage (c, new);/* * Just like with failcnt, we can live with some -* inaccuracy in the watermark. +* inaccuracy in the watermarks.
mm/page_counter.c bool page_counter_try_charge(struct page_counter *counter, · line 157
Quoted 10 lines
mm/page_counter.c +10
-0+* +* Notably, we have two watermarks to allow for both a globally +* visible peak and one that can be reset at a smaller scope. +* +* Since we reset both watermarks when the global reset occurs, +* we can guarantee that watermark >= local_watermark, so we +* don't need to do both comparisons every time. +* +* On systems with branch predictors, the inner condition should +* be almost free.
/* See comment in page_counter_charge() */
tools/testing/selftests/cgroup/cgroup_util.c long cg_read_long(const char *cgroup, const char *control) · line 142
Quoted 7 lines
tools/testing/selftests/cgroup/cgroup_util.c +0
-0@@ -141,6 +141,16 @@ long cg_read_long(const char *cgroup, const char *control)return atol (buf); }
This should be in patch #2.
Hi Johannes,
Thanks for the thorough review!
I'll send a rebased version addressing your comments in a moment.
Quoted 17 lines · Johannes Weiner
Hi David,
thanks for pursuing this! A couple of comments below.
Quoted 11 lines · David Finkel
Documentation/admin-guide/cgroup-v2.rst +4
-2@@ -1322,11 +1322,16 @@ PAGE_SIZE multiple when read back. reclaim induced by memory.reclaim. memory.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max memory usage recorded for the cgroup and its descendants since + either the creation of the cgroup or the most recent reset for that FD.
Quoted 10 lines
Documentation/admin-guide/cgroup-v2.rst PAGE_SIZE multiple when read back. · line 1334
Quoted 7 lines
Documentation/admin-guide/cgroup-v2.rst +5
-2- The max memory usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of the string "reset" to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. + Attempts to write any other non-empty string will return EINVAL + (modulo leading and trailing whitespace).Why not allow any write to reset? This makes it harder to use, and I'm not sure accidental writes are a likely mistake to make.
Accepting any string is a rather wide interface, and since actually empty writes get dropped before this handler is invoked, I think allowing any write is more confusing.
I figured that a narrower interface is a better starting point, as long as it's correctly
documented.
We can always widen it to all writes later, but we can't go the other way.
Quoted 25 lines
Quoted 14 lines
include/linux/cgroup.h +1
-0@@ -12,6 +12,7 @@#include <linux/sched.h> #include <linux/cpumask.h> #include <linux/nodemask.h> +#include <linux/list.h> #include <linux/rculist.h> #include <linux/cgroupstats.h> #include <linux/fs.h> include/linux/cgroup.h static inline void cgroup_bpf_put(struct cgroup *cgrp) {} · line 862
Quoted 7 lines
include/linux/cgroup.h +4
-0@@ -855,4 +856,11 @@ static inline void cgroup_bpf_put(struct cgroup *cgrp) {}struct cgroup *task_get_cgroup1 (struct task_struct *tsk,int hierarchy_id); +struct memcg_peak_mem_ctx { +long local_watermark; +struct list_head peers; +};Since this is generic cgroup code, and can be conceivably used by other controllers, let's keep the naming generic as well. How about:
Quoted 6 lines
struct cgroup_of_peak { long value; struct list_head list; };
cgroup-defs.hwould be a better place for it.
That is much less jarring.
Quoted 20 lines
include/linux/cgroup.h static inline void cgroup_bpf_put(struct cgroup *cgrp) {} · line 864
Quoted 2 lines
include/linux/cgroup.h +1
-0+struct memcg_peak_mem_ctx *memcg_extract_peak_mem_ctx (struct kernfs_open_file *of);
of_peak()include/linux/memcontrol.h struct mem_cgroup { · line 204
Quoted 12 lines
include/linux/memcontrol.h +4
-0@@ -198,6 +198,11 @@ struct mem_cgroup {struct page_counter kmem;/* v1 only */ struct page_counter tcpmem;/* v1 only */ +/* lists of memcg peak watching contexts on swap and memory */ +struct list_head peak_memory_local_watermark_watchers; +struct list_head peak_swap_local_watermark_watchers; +spinlock_t swap_memory_peak_watchers_lock;These names are too long. How about:
Quoted 5 lines
/* Registered local usage peak watchers */
struct list_head memory_peaks;
struct list_head swap_peaks;
spinlock_t peaks_lock;That's much better.
(sometimes when I'm unsure about a name, I default to a terrible, long name with way too much information)
Quoted 14 lines
Quoted 14 lines
include/linux/page_counter.h +1
-0@@ -26,6 +26,7 @@ struct page_counter {atomic_long_t children_low_usage;unsigned long watermark; +unsigned long local_watermark;/* track min of fd-local resets */ unsigned long failcnt;/* Keep all the read most fields in a separete cacheline. */ @@ -78,7 +79,15 @@ int page_counter_memparse(const char *buf, const char *max,
Quoted 6 lines
include/linux/page_counter.h int page_counter_memparse(const char *buf, const char *max, · line 82
Quoted 4 lines
include/linux/page_counter.h +1
-1static inline void page_counter_reset_watermark (struct page_counter *counter) { - counter->watermark =page_counter_read (counter); +unsigned long cur =page_counter_read (counter);cur -> usage
That's a better name.
Quoted 13 lines
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6917
Quoted 10 lines
include/linux/page_counter.h +2
-2@@ -6907,12 +6912,109 @@ static u64 memory_current_read(struct cgroup_subsys_state *css,return (u64 )page_counter_read (&memcg->memory) *PAGE_SIZE ; } -static u64 memory_peak_read (struct cgroup_subsys_state *css, -struct cftype *cft) +inline int swap_memory_peak_show ( +struct seq_file *sf,void *v,bool swap_cg) {Leave inlining to the compiler. Just static int.
Quoted 1 line
The name can be simply
peak_show().
Oh, right, I forgot that since this is local to the translation-unit it doesn't need to be globally unique.
Quoted 10 lines
Customary coding style is to line wrap at the last parameter that fits. Don't wrap if the line fits within 80 cols.
static int peak_show(struct
seq_file*sf, void *v, ...,...) { ... }
Oh yeah, I forgot to move those back to one line when I got rid of the callback args
while addressing Roman's comments.
Thanks for pointing that out.
Quoted 18 lines
Quoted 18 lines
include/linux/page_counter.h +15
-1+struct cgroup_subsys_state *css =seq_css (sf);struct mem_cgroup *memcg =mem_cgroup_from_css (css); +struct page_counter *pc; +struct kernfs_open_file *of = sf->private; +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of); +s64 fd_peak = ctx->local_watermark; -return (u64 )memcg->memory.watermark *PAGE_SIZE ; +if (swap_cg) + pc = &memcg->swap; +else + pc = &memcg->memory; + +if (fd_peak ==-1 ) { +seq_printf (sf,"%llu\n" , (u64 )pc->watermark *PAGE_SIZE ); +return 0 ; + } +
Quoted 8 lines
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6940
Quoted 6 lines
include/linux/page_counter.h +6
-0+s64 pc_peak = pc->local_watermark; +s64 wm = fd_peak > pc_peak ? fd_peak : pc_peak; + +seq_printf (sf,"%lld\n" , wm *PAGE_SIZE ); +return 0 ; +}As per Roman's feedback, don't mix decls and code.
Sorry, that one slipped through. (I've now fixed my syntax highlighting to recognize s64 as a type)
Quoted 26 lines
You can simplify it by extracting css and memcg in the callers, then pass the right struct page counter *pc directly.
That should eliminate most local variables as well.
static int
peak_show(struct seq_file *sf, void *v, struct page_counter *pc){ struct cgroup_of_peak *ofp = of_peak(sf->private); u64 peak; /* User wants global or local peak? */ if (ofp->value == -1) peak = pc->watermark; else peak = max(ofp->value, pc->local_watermark); seq_printf(sf, "%lld\n", peak * PAGE_SIZE); }include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6943
Quoted 3 lines
include/linux/page_counter.h +3
-0+static int memory_peak_show (struct seq_file *sf,void *v) +{ +return swap_memory_peak_show (sf, v, false);And then do:
Quoted 5 lines
struct mem_cgroup *memcg = mem_cgroup_from_css(seq_css(sf)); return peak_show(sf, v, &memcg->memory);Then do the same with ...
&memcg->swap.
That is much cleaner!
Thanks!
Quoted 32 lines
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6947
Quoted 5 lines
include/linux/page_counter.h +4
-0+inline ssize_t swap_memory_peak_write ( +struct kernfs_open_file *of, +char *buf,size_t nbytes,loff_t off,bool swap_cg) +{Same feedback as above. Please don't inline explicitly (unless it really is measurably a performance improvement in a critical path), and stick to surrounding coding style.
Here too, pass
page_counterdirectly and save the branches.Quoted 20 lines
include/linux/page_counter.h +20
-0+unsigned long cur; +struct memcg_peak_mem_ctx *peer_ctx; +struct mem_cgroup *memcg =mem_cgroup_from_css (of_css (of)); +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of); +struct page_counter *pc; +struct list_head *watchers, *pos; + + buf =strstrip (buf); +/* Only allow "reset" to keep the API clear */ +if (strcmp (buf,"reset" )) +return -EINVAL ; + +if (swap_cg) { + pc = &memcg->swap; + watchers = &memcg->peak_swap_local_watermark_watchers; + }else { + pc = &memcg->memory; + watchers = &memcg->peak_memory_local_watermark_watchers; + } +
Quoted 8 lines
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6973
Quoted 6 lines
include/linux/page_counter.h +6
-0+spin_lock (&memcg->swap_memory_peak_watchers_lock); + +page_counter_reset_local_watermark (pc); + cur = pc->local_watermark; + +list_for_each (pos, watchers) {
list_for_each_entry()
Oh, I missed that one. Thanks!
Quoted 9 lines
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6977
Quoted 5 lines
include/linux/page_counter.h +4
-0+ peer_ctx =list_entry (pos,typeof (*ctx), peers); +if (cur > peer_ctx->local_watermark) + peer_ctx->local_watermark = cur; + }I don't think this is quite right.
local_peakcould be higher than the current usage when a new watcher shows up. The other watchers should retain the higherlocal_peak, not the current usage.
Since we have to iterate over the list entries while holding the spinlock, I think we avoid that kind of mismatch by keeping the assignments inside that critical section as well. (and eliminate a few other races)
As-is, we get a snapshot value of the current usage and use that for the rest of the function while holding the lock, so, those fd-local watermarks will only change by assignments in this function.
On the other hand, pc->local_watermark may keep increasing, but that's fine.
Quoted 69 lines
include/linux/page_counter.h static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6983
Quoted 7 lines
include/linux/page_counter.h +6
-0+ +if (ctx->local_watermark ==-1 ) +/* only append to the list if we're not already there */ +list_add_tail (&ctx->peers, watchers); + + ctx->local_watermark = cur;This makes me think that
page_counter_reset_local_watermark()is not a good helper. It obscures what's going on. Try without it.AFAICS the list ordering doesn't matter, so keep it simple and use a plain
list_add()./* * A new local peak is being tracked in pc->local_watermark. * Save current local peak in all watchers. */ list_for_each_entry(pos, ...) if (pc->local_watermark > pos->value) pos->value = pc->local_watermark; pc->local_watermark = page_counter_read(pc); /* Initital write, register watcher */ if (ofp->value == -1) list_add() ofp->value = pc->local_watermark;Quoted 39 lines
mm/page_counter.c +20
-3@@ -79,9 +79,22 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages)/* * This is indeed racy, but we can live with some * inaccuracy in the watermark. +* +* Notably, we have two watermarks to allow for both a globally +* visible peak and one that can be reset at a smaller scope. +* +* Since we reset both watermarks when the global reset occurs, +* we can guarantee that watermark >= local_watermark, so we +* don't need to do both comparisons every time. +* +* On systems with branch predictors, the inner condition should +* be almost free. */ -if (new >READ_ONCE (c->watermark)) -WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) { +WRITE_ONCE (c->local_watermark, new); +if (new >READ_ONCE (c->watermark)) +WRITE_ONCE (c->watermark, new); + } } }@@ -131,10 +144,23 @@ bool page_counter_try_charge(struct page_counter *counter,propagate_protected_usage (c, new);/* * Just like with failcnt, we can live with some -* inaccuracy in the watermark. +* inaccuracy in the watermarks. +* +* Notably, we have two watermarks to allow for both a globally +* visible peak and one that can be reset at a smaller scope. +*
Quoted 8 lines
mm/page_counter.c void page_counter_charge(struct page_counter *counter, unsigned long nr_pages) · line 91
Quoted 6 lines
mm/page_counter.c +6
-0+* Since we reset both watermarks when the global reset occurs, +* we can guarantee that watermark >= local_watermark, so we +* don't need to do both comparisons every time. +* +* On systems with branch predictors, the inner condition should +* be almost free. /* See comment in page_counter_charge() */
Even better!
Quoted 10 lines
tools/testing/selftests/cgroup/cgroup_util.c long cg_read_long(const char *cgroup, const char *control) · line 142
Quoted 8 lines
tools/testing/selftests/cgroup/cgroup_util.c +0
-0@@ -141,6 +141,16 @@ long cg_read_long(const char *cgroup, const char *control)return atol (buf); }This should be in patch #2.
:facepalm: that's what I get for trying to split this and send it out at the end of the day.
Signature · 2 lines
David Finkel Senior Principal Software Engineer, Core Services
PATCH 2/2 mm, memcg: cg2 memory{.swap,}.peak write tests
In reply to David FinkelExtend two existing tests to cover extracting memory usage through the newly mutable memory.peak and memory.swap.peak handlers.
In particular, make sure to exercise adding and removing watchers with overlapping lifetimes so the less-trivial logic gets tested.
--- .../selftests/cgroup/test_memcontrol.c | 227 +++++++++++++++++- 1 file changed, 219 insertions(+), 8 deletions(-)
tools/testing/selftests/cgroup/test_memcontrol.c +219 -8
@@ -161,12 +161,12 @@ static int alloc_pagecache_50M_check(const char *cgroup, void *arg)/* * This test create a memory cgroup, allocates * some anonymous memory and some pagecache -* and check memory.current and some memory.stat values. +* and checks memory.current, memory.peak, and some memory.stat values. */ -static int test_memcg_current (const char *root) +static int test_memcg_current_peak (const char *root) {int ret =KSFT_FAIL ; -long current; +long current, peak, peak_reset;char *memcg; memcg =cg_name (root,"memcg_test" );
@@ -180,15 +180,109 @@ static int test_memcg_current(const char *root)if (current !=0 ) goto cleanup; + peak =cg_read_long (memcg,"memory.peak" ); +if (peak !=0 ) + goto cleanup; +if (cg_run (memcg, alloc_anon_50M_check,NULL )) goto cleanup; + peak =cg_read_long (memcg,"memory.peak" ); +if (peak <MB (50 )) + goto cleanup; + +/* +* We'll open a few FDs for the same memory.peak file to exercise the free-path +* We need at least three to be closed in a different order than writes occurred to test +* the linked-list handling. +*/ +int peak_fd =cg_open (memcg,"memory.peak" ,O_RDWR |O_APPEND |O_CLOEXEC ); + +if (peak_fd ==-1 ) + goto cleanup; + +bool fd2_closed = false, fd3_closed = false, fd4_closed = false; +int peak_fd2 =cg_open (memcg,"memory.peak" ,O_RDWR |O_APPEND |O_CLOEXEC ); + +if (peak_fd2 ==-1 ) + goto cleanup; + +int peak_fd3 =cg_open (memcg,"memory.peak" ,O_RDWR |O_APPEND |O_CLOEXEC ); + +if (peak_fd3 ==-1 ) + goto cleanup; + +static const char reset_string[] ="reset\n" ; + + peak_reset =write (peak_fd, reset_string,sizeof (reset_string)); +if (peak_reset !=sizeof (reset_string)) + goto cleanup; + + peak_reset =write (peak_fd2, reset_string,sizeof (reset_string)); +if (peak_reset !=sizeof (reset_string)) + goto cleanup; + + peak_reset =write (peak_fd3, reset_string,sizeof (reset_string)); +if (peak_reset !=sizeof (reset_string)) + goto cleanup; + +/* Make sure a completely independent read isn't affected by our FD-local reset above*/ + peak =cg_read_long (memcg,"memory.peak" ); +if (peak <MB (50 )) + goto cleanup; + + fd2_closed = true; +if (close (peak_fd2)) + goto cleanup; + +int peak_fd4 =cg_open (memcg,"memory.peak" ,O_RDWR |O_APPEND |O_CLOEXEC ); + +if (peak_fd4 ==-1 ) + goto cleanup; + + peak_reset =write (peak_fd4, reset_string,sizeof (reset_string)); +if (peak_reset !=sizeof (reset_string)) + goto cleanup; + + peak =cg_read_long_fd (peak_fd); +if (peak >MB (30 ) || peak <0 ) + goto cleanup; +if (cg_run (memcg, alloc_pagecache_50M_check,NULL )) goto cleanup; + peak =cg_read_long (memcg,"memory.peak" ); +if (peak <MB (50 )) + goto cleanup; + +/* Make sure everything is back to normal */ + peak =cg_read_long_fd (peak_fd); +if (peak <MB (50 )) + goto cleanup; + + peak =cg_read_long_fd (peak_fd4); +if (peak <MB (50 )) + goto cleanup; + + fd3_closed = true; +if (close (peak_fd3)) + goto cleanup; + + fd4_closed = true; +if (close (peak_fd4)) + goto cleanup; + + ret =KSFT_PASS ; cleanup: +close (peak_fd); +if (!fd2_closed) +close (peak_fd2); +if (!fd3_closed) +close (peak_fd3); +if (!fd4_closed) +close (peak_fd4);cg_destroy (memcg);free (memcg);
@@ -817,13 +911,16 @@ static int alloc_anon_50M_check_swap(const char *cgroup, void *arg)/* * This test checks that memory.swap.max limits the amount of -* anonymous memory which can be swapped out. +* anonymous memory which can be swapped out. Additionally, it verifies that +* memory.swap.peak reflects the high watermark and can be reset. */ -static int test_memcg_swap_max (const char *root) +static int test_memcg_swap_max_peak (const char *root) {int ret =KSFT_FAIL ;char *memcg; -long max; +long max, peak; + +static const char reset_string[] ="reset\n" ;if (!is_swap_enabled ())return KSFT_SKIP ;
@@ -840,6 +937,45 @@ static int test_memcg_swap_max(const char *root) goto cleanup; } +int swap_peak_fd =cg_open (memcg,"memory.swap.peak" , +O_RDWR |O_APPEND |O_CLOEXEC ); + +if (swap_peak_fd ==-1 ) + goto cleanup; + +int mem_peak_fd =cg_open (memcg,"memory.peak" ,O_RDWR |O_APPEND |O_CLOEXEC ); + +if (mem_peak_fd ==-1 ) + goto cleanup; + +if (cg_read_long (memcg,"memory.swap.peak" )) + goto cleanup; + +if (cg_read_long_fd (swap_peak_fd)) + goto cleanup; + +/* switch the swap and mem fds into local-peak tracking mode*/ +int peak_reset =write (swap_peak_fd, reset_string,sizeof (reset_string)); + +if (peak_reset !=sizeof (reset_string)) + goto cleanup; + +if (cg_read_long_fd (swap_peak_fd)) + goto cleanup; + +if (cg_read_long (memcg,"memory.peak" )) + goto cleanup; + +if (cg_read_long_fd (mem_peak_fd)) + goto cleanup; + + peak_reset =write (mem_peak_fd, reset_string,sizeof (reset_string)); +if (peak_reset !=sizeof (reset_string)) + goto cleanup; + +if (cg_read_long_fd (mem_peak_fd)) + goto cleanup; +if (cg_read_strcmp (memcg,"memory.max" ,"max\n" )) goto cleanup;
@@ -862,6 +998,61 @@ static int test_memcg_swap_max(const char *root)if (cg_read_key_long (memcg,"memory.events" ,"oom_kill " ) !=1 ) goto cleanup; + peak =cg_read_long (memcg,"memory.peak" ); +if (peak <MB (29 )) + goto cleanup; + + peak =cg_read_long (memcg,"memory.swap.peak" ); +if (peak <MB (29 )) + goto cleanup; + + peak =cg_read_long_fd (mem_peak_fd); +if (peak <MB (29 )) + goto cleanup; + + peak =cg_read_long_fd (swap_peak_fd); +if (peak <MB (29 )) + goto cleanup; + +/* +* open, reset and close the peak swap on another FD to make sure +* multiple extant fds don't corrupt the linked-list +*/ + peak_reset =cg_write (memcg,"memory.swap.peak" , (char *)reset_string); +if (peak_reset) + goto cleanup; + + peak_reset =cg_write (memcg,"memory.peak" , (char *)reset_string); +if (peak_reset) + goto cleanup; + +/* actually reset on the fds */ + peak_reset =write (swap_peak_fd, reset_string,sizeof (reset_string)); +if (peak_reset !=sizeof (reset_string)) + goto cleanup; + + peak_reset =write (mem_peak_fd, reset_string,sizeof (reset_string)); +if (peak_reset !=sizeof (reset_string)) + goto cleanup; + + peak =cg_read_long_fd (swap_peak_fd); +if (peak >MB (10 )) + goto cleanup; + +/* +* The cgroup is now empty, but there may be a page or two associated +* with the open FD accounted to it. +*/ + peak =cg_read_long_fd (mem_peak_fd); +if (peak >MB (1 )) + goto cleanup; + +if (cg_read_long (memcg,"memory.peak" ) <MB (29 )) + goto cleanup; + +if (cg_read_long (memcg,"memory.swap.peak" ) <MB (29 )) + goto cleanup; +if (cg_run (memcg, alloc_anon_50M_check_swap, (void *)MB (30 ))) goto cleanup;
@@ -869,9 +1060,29 @@ static int test_memcg_swap_max(const char *root)if (max <=0 ) goto cleanup; + peak =cg_read_long (memcg,"memory.peak" ); +if (peak <MB (29 )) + goto cleanup; + + peak =cg_read_long (memcg,"memory.swap.peak" ); +if (peak <MB (29 )) + goto cleanup; + + peak =cg_read_long_fd (mem_peak_fd); +if (peak <MB (29 )) + goto cleanup; + + peak =cg_read_long_fd (swap_peak_fd); +if (peak <MB (19 )) + goto cleanup; + ret =KSFT_PASS ; cleanup: +if (close (mem_peak_fd)) + ret =KSFT_FAIL ; +if (close (swap_peak_fd)) + ret =KSFT_FAIL ;cg_destroy (memcg);free (memcg);
@@ -1295,7 +1506,7 @@ struct memcg_test {const char *name; } tests[] = {T (test_memcg_subtree_control), -T (test_memcg_current), +T (test_memcg_current_peak),T (test_memcg_min),T (test_memcg_low), T(test_memcg_high),
@@ -1303,7 +1514,7 @@ struct memcg_test {T (test_memcg_max),T (test_memcg_reclaim),T (test_memcg_oom_events), -T (test_memcg_swap_max), +T (test_memcg_swap_max_peak),T (test_memcg_sock),T (test_memcg_oom_group_leaf_events), T(test_memcg_oom_group_parent_events),
Signature · 1 line
2.40.1