mm, memcg: cg2 memory{.swap,}.peak write handlers (fd-local edition)
Subsystems (6)
PATCH 1/1 mm, memcg: cg2 memory{.swap,}.peak write handlers (fd-local edition)
My last patch[1] was met with a general desire for a safer scheme that avoided global resets, which expose unclear ownership.
Fortunately, Johannes[2] suggested a reasonably simple scheme to provide an FD-local reset, which eliminates most of those issues.
The one open question I have is whether the cgroup/memcg itself is kept alive by an open FD, or if we need to update the memcg freeing code to traverse the new list of "watchers" so they don't try to access freed memory.
Thank you,
David Finkel
Senior Principal Software Engineer, Core Services
Vimeo Inc.
[1]: https://lore.kernel.org/cgroups/[email protected]/
[2]: https://lore.kernel.org/cgroups/[email protected]/
PATCH 1/1 mm, memcg: cg2 memory{.swap,}.peak write handlers
Other mechanisms for querying the peak memory usage of either a process or v1 memory cgroup allow for resetting the high watermark. Restore parity with those mechanisms, but with a less racy API.
For example:
- Any write to memory.max_usage_in_bytes in a cgroup v1 mount resets the high watermark.
- writing "5" to the clear_refs pseudo-file in a processes's proc directory resets the peak RSS.
This change is an evolution of a previous patch, which mostly copied the cgroup v1 behavior, however, there were concerns about races/ownership issues with a global reset, so instead this change makes the reset filedescriptor-local.
Writing a specific string to the memory.peak and memory.swap.peak pseudo-files reset the high watermark to the current usage for subsequent reads through that same fd.
Notably, following Johannes's suggestion, this implementation moves the O(fds that have written) behavior onto the fd write(2) path. Instead, on the page-allocation path, we simply add one additional watermark to conditionally bump per-hierarchy level in the page-counter.
This behavior is particularly useful for work scheduling systems that need to track memory usage of worker processes/cgroups per-work-item. Since memory can't be squeezed like CPU can (the OOM-killer has opinions), these systems need to track the peak memory usage to compute system/container fullness when binpacking workitems.
Most notably, Vimeo's use-case involves a system that's doing global binpacking across many Kubernetes pods/containers, and while we can use PSI for some local decisions about overload, we strive to avoid packing workloads too tightly in the first place. To facilitate this, we track the peak memory usage. However, since we run with long-lived workers (to amortize startup costs) we need a way to track the high watermark while a work-item is executing. Polling runs the risk of missing short spikes that last for timescales below the polling interval, and peak memory tracking at the cgroup level is otherwise perfect for this use-case.
As this data is used to ensure that binpacked work ends up with sufficient headroom, this use-case mostly avoids the inaccuracies surrounding reclaimable memory.
--- Documentation/admin-guide/cgroup-v2.rst | 26 +- include/linux/cgroup.h | 7 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 6 + kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 165 ++++++++++++- mm/page_counter.c | 4 + tools/testing/selftests/cgroup/cgroup_util.c | 22 ++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + .../selftests/cgroup/test_memcontrol.c | 227 +++++++++++++++++- 11 files changed, 448 insertions(+), 25 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +18 -8
@@ -1322,11 +1322,16 @@ PAGE_SIZE multiple when read back. reclaim induced by memory.reclaim. memory.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max memory usage recorded for the cgroup and its descendants since + either the creation of the cgroup or the most recent reset for that fd. - The max memory usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of the string "fd_local_reset" to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. + Attempts to write any other string will return EINVAL + (modulo leading and trailing whitespace). memory.oom.group A read-write single value file which exists on non-root
@@ -1652,11 +1657,16 @@ PAGE_SIZE multiple when read back. Healthy workloads are not expected to reach this limit. memory.swap.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max swap usage recorded for the cgroup and its descendants since + the creation of the cgroup or the most recent reset for that fd. - The max swap usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of the string "fd_local_reset" to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. + Attempts to write any other string will return EINVAL + (modulo leading and trailing whitespace). memory.swap.max A read-write single value file which exists on non-root
include/linux/cgroup.h +7 -0
@@ -855,4 +855,11 @@ static inline void cgroup_bpf_put(struct cgroup *cgrp) {}struct cgroup *task_get_cgroup1 (struct task_struct *tsk,int hierarchy_id); +struct memcg_peak_mem_ctx { +long local_watermark; +struct memcg_peak_mem_ctx *next, *prev; +}; + +struct memcg_peak_mem_ctx *memcg_extract_peak_mem_ctx (struct kernfs_open_file *of); +#endif /* _LINUX_CGROUP_H */
include/linux/memcontrol.h +5 -0
@@ -198,6 +198,11 @@ struct mem_cgroup {struct page_counter kmem;/* v1 only */ struct page_counter tcpmem;/* v1 only */ +/* lists of memcg peak watching contexts on swap and memory */ +struct memcg_peak_mem_ctx *peak_memory_local_watermark_watchers; +struct memcg_peak_mem_ctx *peak_swap_local_watermark_watchers; +spinlock_t pagecounter_peak_watchers_lock; +/* Range enforcement for interrupt charges */ struct work_struct high_work;
include/linux/page_counter.h +6 -0
@@ -26,6 +26,7 @@ struct page_counter {atomic_long_t children_low_usage;unsigned long watermark; +unsigned long local_watermark;/* track min of fd-local resets */ unsigned long failcnt;/* Keep all the read most fields in a separete cacheline. */
@@ -81,4 +82,9 @@ static inline void page_counter_reset_watermark(struct page_counter *counter) counter->watermark = page_counter_read(counter); } +static inline void page_counter_reset_local_watermark (struct page_counter *counter) +{ + counter->local_watermark =page_counter_read (counter); +} +#endif /* _LINUX_PAGE_COUNTER_H */
kernel/cgroup/cgroup-internal.h +2 -0
@@ -81,6 +81,8 @@ struct cgroup_file_ctx {struct {struct cgroup_pidlist *pidlist; } procs1; + +struct memcg_peak_mem_ctx peak; };/*
kernel/cgroup/cgroup.c +7 -0
@@ -1964,6 +1964,13 @@ static int cgroup2_parse_param(struct fs_context *fc, struct fs_parameter *paramreturn -EINVAL ; } +struct memcg_peak_mem_ctx *memcg_extract_peak_mem_ctx (struct kernfs_open_file *of) +{ +struct cgroup_file_ctx *ctx = of->priv; + +return &ctx->peak; +} +static void apply_cgroup_root_flags (unsigned int root_flags) {if (current->nsproxy->cgroup_ns == &init_cgroup_ns) {
mm/memcontrol.c +156 -9
208 lines not on this page · raw message
mm/page_counter.c +4 -0
@@ -82,6 +82,8 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages)*/ if (new >READ_ONCE (c->watermark))WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) +WRITE_ONCE (c->local_watermark, new); } }
@@ -135,6 +137,8 @@ bool page_counter_try_charge(struct page_counter *counter,*/ if (new >READ_ONCE (c->watermark))WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) +WRITE_ONCE (c->local_watermark, new); }return true;
tools/testing/selftests/cgroup/cgroup_util.c +22 -0
@@ -141,6 +141,16 @@ long cg_read_long(const char *cgroup, const char *control)return atol (buf); } +long cg_read_long_fd (int fd) +{ +char buf[128 ]; + +if (pread (fd, buf,sizeof (buf),0 ) <=0 ) +return -1 ; + +return atol (buf); +} +long cg_read_key_long (const char *cgroup,const char *control,const char *key) {char buf[PAGE_SIZE ];
@@ -183,6 +193,18 @@ int cg_write(const char *cgroup, const char *control, char *buf)return ret == len ?0 : ret; } +/* +* Returns fd on success, or -1 on failure. +* (fd should be closed with close() as usual) +*/ +int cg_open (const char *cgroup,const char *control,int flags) +{ +char path[PATH_MAX ]; + +snprintf (path,sizeof (path),"%s/%s" , cgroup, control); +return open (path, flags); +} +int cg_write_numeric (const char *cgroup,const char *control,long value) {char buf[64 ];
tools/testing/selftests/cgroup/cgroup_util.h +2 -0
@@ -34,9 +34,11 @@ extern int cg_read_strcmp(const char *cgroup, const char *control,extern int cg_read_strstr (const char *cgroup,const char *control,const char *needle);extern long cg_read_long (const char *cgroup,const char *control); +extern long cg_read_long_fd (int fd);long cg_read_key_long (const char *cgroup,const char *control,const char *key);extern long cg_read_lc (const char *cgroup,const char *control);extern int cg_write (const char *cgroup,const char *control,char *buf); +extern int cg_open (const char *cgroup,const char *control,int flags);int cg_write_numeric (const char *cgroup,const char *control,long value); extern int cg_run(const char *cgroup, int (*fn)(const char *cgroup, void *arg),
tools/testing/selftests/cgroup/test_memcontrol.c +219 -8
294 lines not on this page · raw message
Signature · 1 line
2.40.1
Roman Gushchin
maintainerRe: [PATCH] mm, memcg: cg2 memory{.swap,}.peak write handlers
Quoted 69 lines · David Finkel
Other mechanisms for querying the peak memory usage of either a process or v1 memory cgroup allow for resetting the high watermark. Restore parity with those mechanisms, but with a less racy API.
For example:
- Any write to memory.max_usage_in_bytes in a cgroup v1 mount resets the high watermark.
- writing "5" to theclear_refspseudo-file in a processes's proc directory resets the peak RSS.This change is an evolution of a previous patch, which mostly copied the cgroup v1 behavior, however, there were concerns about races/ownership issues with a global reset, so instead this change makes the reset filedescriptor-local.
Writing a specific string to the memory.peak and memory.swap.peak pseudo-files reset the high watermark to the current usage for subsequent reads through that same fd.
Notably, following Johannes's suggestion, this implementation moves the O(fds that have written) behavior onto the fd write(2) path. Instead, on the page-allocation path, we simply add one additional watermark to conditionally bump per-hierarchy level in the page-counter.
This behavior is particularly useful for work scheduling systems that need to track memory usage of worker processes/cgroups per-work-item. Since memory can't be squeezed like CPU can (the OOM-killer has opinions), these systems need to track the peak memory usage to compute system/container fullness when binpacking workitems.
Most notably, Vimeo's use-case involves a system that's doing global binpacking across many Kubernetes pods/containers, and while we can use PSI for some local decisions about overload, we strive to avoid packing workloads too tightly in the first place. To facilitate this, we track the peak memory usage. However, since we run with long-lived workers (to amortize startup costs) we need a way to track the high watermark while a work-item is executing. Polling runs the risk of missing short spikes that last for timescales below the polling interval, and peak memory tracking at the cgroup level is otherwise perfect for this use-case.
As this data is used to ensure that binpacked work ends up with sufficient headroom, this use-case mostly avoids the inaccuracies surrounding reclaimable memory.
Suggested-by Johannes Weiner <[email protected]>Signed-off-by David Finkel <[email protected]>--- Documentation/admin-guide/cgroup-v2.rst | 26 +- include/linux/cgroup.h | 7 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 6 + kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 165 ++++++++++++- mm/page_counter.c | 4 + tools/testing/selftests/cgroup/cgroup_util.c | 22 ++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + .../selftests/cgroup/test_memcontrol.c | 227 +++++++++++++++++- 11 files changed, 448 insertions(+), 25 deletions(-)Documentation/admin-guide/cgroup-v2.rst +0
-1@@ -1322,11 +1322,16 @@ PAGE_SIZE multiple when read back. reclaim induced by memory.reclaim. memory.peak - A read-only single value file which exists on non-root
Documentation/admin-guide/cgroup-v2.rst PAGE_SIZE multiple when read back. · line 1331
Quoted 10 lines
Documentation/admin-guide/cgroup-v2.rst +6
-3- cgroups. + A read-write single value file which exists on non-root cgroups. + + The max memory usage recorded for the cgroup and its descendants since + either the creation of the cgroup or the most recent reset for that fd. - The max memory usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of the string "fd_local_reset" to this file resets it to the + current memory usage for subsequent reads through the same
Hi David!
Not a very strong preference, but with the current design, do we really expect to have a non-local reset? If not, can we agree on a "reset" string instead for a sake of simplicity?
Quoted 32 lines
Documentation/admin-guide/cgroup-v2.rst +12
-4+ file descriptor. + Attempts to write any other string will return EINVAL + (modulo leading and trailing whitespace). memory.oom.group A read-write single value file which exists on non-root@@ -1652,11 +1657,16 @@ PAGE_SIZE multiple when read back. Healthy workloads are not expected to reach this limit. memory.swap.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max swap usage recorded for the cgroup and its descendants since + the creation of the cgroup or the most recent reset for that fd. - The max swap usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of the string "fd_local_reset" to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. + Attempts to write any other string will return EINVAL + (modulo leading and trailing whitespace). memory.swap.max A read-write single value file which exists on non-rootinclude/linux/cgroup.h +0
-0
include/linux/cgroup.h static inline void cgroup_bpf_put(struct cgroup *cgrp) {} · line 860
Quoted 6 lines
include/linux/cgroup.h +3
-0@@ -855,4 +855,11 @@ static inline void cgroup_bpf_put(struct cgroup *cgrp) {}struct cgroup *task_get_cgroup1 (struct task_struct *tsk,int hierarchy_id); +struct memcg_peak_mem_ctx { +long local_watermark; +struct memcg_peak_mem_ctx *next, *prev;
Please, take a look at include/linux/list.h and use it instead of re-implementing list operations from scratch.
Quoted 127 lines
include/linux/cgroup.h +4
-0+}; + +struct memcg_peak_mem_ctx *memcg_extract_peak_mem_ctx (struct kernfs_open_file *of); +#endif /* _LINUX_CGROUP_H */ include/linux/memcontrol.h +5
-0@@ -198,6 +198,11 @@ struct mem_cgroup {struct page_counter kmem;/* v1 only */ struct page_counter tcpmem;/* v1 only */ +/* lists of memcg peak watching contexts on swap and memory */ +struct memcg_peak_mem_ctx *peak_memory_local_watermark_watchers; +struct memcg_peak_mem_ctx *peak_swap_local_watermark_watchers; +spinlock_t pagecounter_peak_watchers_lock; +/* Range enforcement for interrupt charges */ struct work_struct high_work;include/linux/page_counter.h +6
-0@@ -26,6 +26,7 @@ struct page_counter {atomic_long_t children_low_usage;unsigned long watermark; +unsigned long local_watermark;/* track min of fd-local resets */ unsigned long failcnt;/* Keep all the read most fields in a separete cacheline. */ @@ -81,4 +82,9 @@ static inline void page_counter_reset_watermark(struct page_counter *counter) counter->watermark = page_counter_read(counter); } +static inline void page_counter_reset_local_watermark (struct page_counter *counter) +{ + counter->local_watermark =page_counter_read (counter); +} +#endif /* _LINUX_PAGE_COUNTER_H */ kernel/cgroup/cgroup-internal.h +2
-0@@ -81,6 +81,8 @@ struct cgroup_file_ctx {struct {struct cgroup_pidlist *pidlist; } procs1; + +struct memcg_peak_mem_ctx peak; };/* kernel/cgroup/cgroup.c +7
-0@@ -1964,6 +1964,13 @@ static int cgroup2_parse_param(struct fs_context *fc, struct fs_parameter *paramreturn -EINVAL ; } +struct memcg_peak_mem_ctx *memcg_extract_peak_mem_ctx (struct kernfs_open_file *of) +{ +struct cgroup_file_ctx *ctx = of->priv; + +return &ctx->peak; +} +static void apply_cgroup_root_flags (unsigned int root_flags) {if (current->nsproxy->cgroup_ns == &init_cgroup_ns) {mm/memcontrol.c +27
-2@@ -25,6 +25,7 @@* Copyright (C) 2020 Alibaba, Inc, Alex Shi */ +#include <linux/cgroup-defs.h> #include <linux/page_counter.h> #include <linux/memcontrol.h> #include <linux/cgroup.h> @@ -5745,6 +5746,7 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent)vmpressure_init (&memcg->vmpressure);INIT_LIST_HEAD (&memcg->event_list);spin_lock_init (&memcg->event_list_lock); +spin_lock_init (&memcg->pagecounter_peak_watchers_lock); memcg->socket_pressure = jiffies;#ifdef CONFIG_MEMCG_KMEM memcg->kmemcg_id =-1 ;@@ -6907,12 +6909,130 @@ static u64 memory_current_read(struct cgroup_subsys_state *css,return (u64 )page_counter_read (&memcg->memory) *PAGE_SIZE ; } -static u64 memory_peak_read (struct cgroup_subsys_state *css, -struct cftype *cft) +static struct page_counter *memcg_memory_extract_page_counter (struct mem_cgroup *memcg) { +return &memcg->memory; +} + +static struct memcg_peak_mem_ctx **memcg_memory_extract_peak_watchers (struct mem_cgroup *memcg) +{ +return &memcg->peak_memory_local_watermark_watchers; +} + +inline int swap_memory_peak_show ( +struct seq_file *sf,void *v, +struct page_counter *(*extract_pc)(struct mem_cgroup *memcg)) +{ +struct cgroup_subsys_state *css =seq_css (sf);struct mem_cgroup *memcg =mem_cgroup_from_css (css); +struct page_counter *pc =extract_pc (memcg); + +struct kernfs_open_file *of = sf->private; +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of); +s64 fd_peak = ctx->local_watermark; + +if (fd_peak ==-1 ) { +seq_printf (sf,"%llu\n" , (u64 )pc->watermark *PAGE_SIZE ); +return 0 ; + } +
mm/memcontrol.c static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6948
Quoted 10 lines
mm/memcontrol.c +10
-0+s64 pc_peak = pc->local_watermark; +s64 wm = fd_peak > pc_peak ? fd_peak : pc_peak; + +seq_printf (sf,"%lld\n" , wm *PAGE_SIZE ); +return 0 ; +} + +static int memory_peak_show (struct seq_file *sf,void *v) +{ +return swap_memory_peak_show (sf, v, memcg_memory_extract_page_counter);
I think it's really too complex. Why not pass a single boolean argument which will define to use memory page_counter or swap page_counter? It will eliminate a need to pass pointers to functions and also eliminate a need for introducing these helper functions in general.
Quoted 43 lines
mm/memcontrol.c +41
-1+} + +static int swap_memory_peak_open (struct kernfs_open_file *of) +{ +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of); + + ctx->local_watermark =-1 ; +return 0 ; +} + +inline void swap_memory_peak_release ( +struct kernfs_open_file *of, +struct memcg_peak_mem_ctx **(*extract_watchers)(struct mem_cgroup *memcg)) +{ +struct mem_cgroup *memcg =mem_cgroup_from_css (of_css (of)); +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of); + +if (ctx->local_watermark ==-1 ) { +/* fast path (no writes on this fd)*/ +return ; + } +spin_lock (&memcg->pagecounter_peak_watchers_lock); +if (ctx->next) { + ctx->next->prev = ctx->prev; + } +if (ctx->prev) { + ctx->prev->next = ctx->next; + }else { +struct memcg_peak_mem_ctx **watchers =extract_watchers (memcg); + + *watchers = ctx->next; + } +spin_unlock (&memcg->pagecounter_peak_watchers_lock); +} -return (u64 )memcg->memory.watermark *PAGE_SIZE ; +static void memory_peak_release (struct kernfs_open_file *of) +{ +swap_memory_peak_release (of, memcg_memory_extract_peak_watchers); +} + +inline ssize_t swap_memory_peak_write( + struct kernfs_open_file *of,
mm/memcontrol.c static u64 memory_current_read(struct cgroup_subsys_state *css, · line 7000
Quoted 10 lines
mm/memcontrol.c +10
-0+ char *buf, size_t nbytes, loff_t off, + struct page_counter* (*extract_pc)(struct mem_cgroup *memcg), + struct memcg_peak_mem_ctx **(*extract_watchers)(struct mem_cgroup *memcg)) +{ + buf =strstrip (buf); +/* Only allow "fd_local_reset" to keep the API clear */ +if (strcmp (buf,"fd_local_reset" )) +return -EINVAL ; +struct mem_cgroup *memcg =mem_cgroup_from_css (of_css (of)); +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of);
Please, don't mix variable definitions and code. Also, please, use scripts/checkpatch.pl for checking the code before submission. I guess it will raise several issues in this patch.
Quoted 107 lines
mm/memcontrol.c +68
-6+ +spin_lock (&memcg->pagecounter_peak_watchers_lock); + +struct page_counter *pc =extract_pc (memcg); + +page_counter_reset_local_watermark (pc); +const unsigned long cur = pc->local_watermark; +struct memcg_peak_mem_ctx **watchers =extract_watchers (memcg); +struct memcg_peak_mem_ctx *peer_ctx; + +for (peer_ctx = *watchers; peer_ctx; peer_ctx = peer_ctx->next) { +if (cur > peer_ctx->local_watermark) + peer_ctx->local_watermark = cur; + } +if (ctx->local_watermark ==-1 ) { +/* only append to the list if we're not already there */ +if (peer_ctx) { + ctx->prev = peer_ctx; + peer_ctx->next = ctx; + }else { + *watchers = ctx; + } + } + ctx->local_watermark = cur; +spin_unlock (&memcg->pagecounter_peak_watchers_lock); + +return nbytes; +} + +static ssize_t memory_peak_write (struct kernfs_open_file *of,char *buf, +size_t nbytes,loff_t off) +{ +return swap_memory_peak_write (of, buf, nbytes, off, + memcg_memory_extract_page_counter, + memcg_memory_extract_peak_watchers); }static int memory_min_show (struct seq_file *m,void *v)@@ -7231,7 +7351,10 @@ static struct cftype memory_files[] = { { .name = "peak", .flags = CFTYPE_NOT_ON_ROOT, - .read_u64 = memory_peak_read, + .open = swap_memory_peak_open, + .release = memory_peak_release, + .seq_show = memory_peak_show, + .write = memory_peak_write, }, { .name = "min",@@ -8193,14 +8316,35 @@ static u64 swap_current_read(struct cgroup_subsys_state *css,return (u64 )page_counter_read (&memcg->swap) *PAGE_SIZE ; } -static u64 swap_peak_read (struct cgroup_subsys_state *css, -struct cftype *cft) + +static struct page_counter *memcg_swap_extract_page_counter (struct mem_cgroup *memcg) +{ +return &memcg->swap; +} + +static struct memcg_peak_mem_ctx **memcg_swap_extract_peak_watchers (struct mem_cgroup *memcg) { -struct mem_cgroup *memcg =mem_cgroup_from_css (css); +return &memcg->peak_swap_local_watermark_watchers; +} + +static int swap_peak_show (struct seq_file *sf,void *v) +{ +return swap_memory_peak_show (sf, v, memcg_swap_extract_page_counter); +} -return (u64 )memcg->swap.watermark *PAGE_SIZE ; +static ssize_t swap_peak_write (struct kernfs_open_file *of,char *buf, +size_t nbytes,loff_t off) +{ +return swap_memory_peak_write (of, buf, nbytes, off, + memcg_swap_extract_page_counter, + memcg_swap_extract_peak_watchers); +} +static void swap_peak_release (struct kernfs_open_file *of) +{ +swap_memory_peak_release (of, memcg_swap_extract_peak_watchers); } +static int swap_high_show (struct seq_file *m,void *v) { return seq_puts_memcg_tunable(m,@@ -8282,7 +8426,10 @@ static struct cftype swap_files[] = { { .name = "swap.peak", .flags = CFTYPE_NOT_ON_ROOT, - .read_u64 = swap_peak_read, + .open = swap_memory_peak_open, + .release = swap_peak_release, + .seq_show = swap_peak_show, + .write = swap_peak_write, }, { .name = "swap.events",mm/page_counter.c +0
-0
mm/page_counter.c void page_counter_charge(struct page_counter *counter, unsigned long nr_pages) · line 86
Quoted 5 lines
mm/page_counter.c +2
-0@@ -82,6 +82,8 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages)*/ if (new >READ_ONCE (c->watermark))WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) +WRITE_ONCE (c->local_watermark, new);
Hm, can't we have a single comparison on the hot path?
Also, we read and write c->local_watermark speculatively here, Idk if it's still acceptable with an ability to reset watermarks "locally". Maybe it is, but it definitely deserves at least a comment with an explanation.
And btw thank you for including tests into the commit, it's really great to see. I'd suggest you to extract them into a separate commit and post it as a series.
Thank you!
Re: [PATCH] mm, memcg: cg2 memory{.swap,}.peak write handlers
Roman,
Thanks for the review.
Quoted 84 lines · Roman Gushchin
Quoted 69 lines · David Finkel
Other mechanisms for querying the peak memory usage of either a process or v1 memory cgroup allow for resetting the high watermark. Restore parity with those mechanisms, but with a less racy API.
For example:
- Any write to memory.max_usage_in_bytes in a cgroup v1 mount resets the high watermark.
- writing "5" to theclear_refspseudo-file in a processes's proc directory resets the peak RSS.This change is an evolution of a previous patch, which mostly copied the cgroup v1 behavior, however, there were concerns about races/ownership issues with a global reset, so instead this change makes the reset filedescriptor-local.
Writing a specific string to the memory.peak and memory.swap.peak pseudo-files reset the high watermark to the current usage for subsequent reads through that same fd.
Notably, following Johannes's suggestion, this implementation moves the O(fds that have written) behavior onto the fd write(2) path. Instead, on the page-allocation path, we simply add one additional watermark to conditionally bump per-hierarchy level in the page-counter.
This behavior is particularly useful for work scheduling systems that need to track memory usage of worker processes/cgroups per-work-item. Since memory can't be squeezed like CPU can (the OOM-killer has opinions), these systems need to track the peak memory usage to compute system/container fullness when binpacking workitems.
Most notably, Vimeo's use-case involves a system that's doing global binpacking across many Kubernetes pods/containers, and while we can use PSI for some local decisions about overload, we strive to avoid packing workloads too tightly in the first place. To facilitate this, we track the peak memory usage. However, since we run with long-lived workers (to amortize startup costs) we need a way to track the high watermark while a work-item is executing. Polling runs the risk of missing short spikes that last for timescales below the polling interval, and peak memory tracking at the cgroup level is otherwise perfect for this use-case.
As this data is used to ensure that binpacked work ends up with sufficient headroom, this use-case mostly avoids the inaccuracies surrounding reclaimable memory.
Suggested-by Johannes Weiner <[email protected]>Signed-off-by David Finkel <[email protected]>--- Documentation/admin-guide/cgroup-v2.rst | 26 +- include/linux/cgroup.h | 7 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 6 + kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 165 ++++++++++++- mm/page_counter.c | 4 + tools/testing/selftests/cgroup/cgroup_util.c | 22 ++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + .../selftests/cgroup/test_memcontrol.c | 227 +++++++++++++++++- 11 files changed, 448 insertions(+), 25 deletions(-)Documentation/admin-guide/cgroup-v2.rst +0
-1@@ -1322,11 +1322,16 @@ PAGE_SIZE multiple when read back. reclaim induced by memory.reclaim. memory.peak - A read-only single value file which exists on non-rootDocumentation/admin-guide/cgroup-v2.rst PAGE_SIZE multiple when read back. · line 1331
Quoted 10 lines
Documentation/admin-guide/cgroup-v2.rst +6
-3- cgroups. + A read-write single value file which exists on non-root cgroups. + + The max memory usage recorded for the cgroup and its descendants since + either the creation of the cgroup or the most recent reset for that fd. - The max memory usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of the string "fd_local_reset" to this file resets it to the + current memory usage for subsequent reads through the sameHi David!
Quoted 3 lines
Not a very strong preference, but with the current design, do we really expect to have a non-local reset? If not, can we agree on a "reset" string instead for a sake of simplicity?
I put in "fd_local_reset" as a bit of a straw-man. I'm fine with changing it to "reset"
since the consensus seemed to be that a global reset was problematic.
Quoted 34 lines
Quoted 34 lines
Documentation/admin-guide/cgroup-v2.rst +12
-4+ file descriptor. + Attempts to write any other string will return EINVAL + (modulo leading and trailing whitespace). memory.oom.group A read-write single value file which exists on non-root@@ -1652,11 +1657,16 @@ PAGE_SIZE multiple when read back. Healthy workloads are not expected to reach this limit. memory.swap.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max swap usage recorded for the cgroup and its descendants since + the creation of the cgroup or the most recent reset for that fd. - The max swap usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of the string "fd_local_reset" to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. + Attempts to write any other string will return EINVAL + (modulo leading and trailing whitespace). memory.swap.max A read-write single value file which exists on non-rootinclude/linux/cgroup.h +0
-0@@ -855,4 +855,11 @@ static inline void cgroup_bpf_put(struct cgroup *cgrp) {}
Quoted 8 lines
include/linux/cgroup.h static inline void cgroup_bpf_put(struct cgroup *cgrp) {} · line 860
Quoted 5 lines
include/linux/cgroup.h +3
-0struct cgroup *task_get_cgroup1 (struct task_struct *tsk,int hierarchy_id); +struct memcg_peak_mem_ctx { +long local_watermark; +struct memcg_peak_mem_ctx *next, *prev;Please, take a look at
include/linux/list.hand use it instead of re-implementing list operations from scratch.
I clearly didn't look hard enough for that. Thanks! I'll port this to use those helpers/macros.
Quoted 135 lines
Quoted 135 lines
include/linux/cgroup.h +4
-0+}; + +struct memcg_peak_mem_ctx *memcg_extract_peak_mem_ctx (struct kernfs_open_file *of); +#endif /* _LINUX_CGROUP_H */ include/linux/memcontrol.h +5
-0@@ -198,6 +198,11 @@ struct mem_cgroup {struct page_counter kmem;/* v1 only */ struct page_counter tcpmem;/* v1 only */ +/* lists of memcg peak watching contexts on swap and memory */ +struct memcg_peak_mem_ctx *peak_memory_local_watermark_watchers; +struct memcg_peak_mem_ctx *peak_swap_local_watermark_watchers; +spinlock_t pagecounter_peak_watchers_lock; +/* Range enforcement for interrupt charges */ struct work_struct high_work;include/linux/page_counter.h +6
-0@@ -26,6 +26,7 @@ struct page_counter {atomic_long_t children_low_usage;unsigned long watermark; +unsigned long local_watermark;/* track min of fd-local resets */ unsigned long failcnt;/* Keep all the read most fields in a separete cacheline. */ @@ -81,4 +82,9 @@ static inline void page_counter_reset_watermark(struct page_counter *counter) counter->watermark = page_counter_read(counter); } +static inline void page_counter_reset_local_watermark (struct page_counter *counter) +{ + counter->local_watermark =page_counter_read (counter); +} +#endif /* _LINUX_PAGE_COUNTER_H */ kernel/cgroup/cgroup-internal.h +2
-0@@ -81,6 +81,8 @@ struct cgroup_file_ctx {struct {struct cgroup_pidlist *pidlist; } procs1; + +struct memcg_peak_mem_ctx peak; };/* kernel/cgroup/cgroup.c +7
-0@@ -1964,6 +1964,13 @@ static int cgroup2_parse_param(struct fs_context *fc, struct fs_parameter *paramreturn -EINVAL ; } +struct memcg_peak_mem_ctx *memcg_extract_peak_mem_ctx (struct kernfs_open_file *of) +{ +struct cgroup_file_ctx *ctx = of->priv; + +return &ctx->peak; +} +static void apply_cgroup_root_flags (unsigned int root_flags) {if (current->nsproxy->cgroup_ns == &init_cgroup_ns) {mm/memcontrol.c +34
-2@@ -25,6 +25,7 @@* Copyright (C) 2020 Alibaba, Inc, Alex Shi */ +#include <linux/cgroup-defs.h> #include <linux/page_counter.h> #include <linux/memcontrol.h> #include <linux/cgroup.h> @@ -5745,6 +5746,7 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent)vmpressure_init (&memcg->vmpressure);INIT_LIST_HEAD (&memcg->event_list);spin_lock_init (&memcg->event_list_lock); +spin_lock_init (&memcg->pagecounter_peak_watchers_lock); memcg->socket_pressure = jiffies;#ifdef CONFIG_MEMCG_KMEM memcg->kmemcg_id =-1 ;@@ -6907,12 +6909,130 @@ static u64 memory_current_read(struct cgroup_subsys_state *css,return (u64 )page_counter_read (&memcg->memory) *PAGE_SIZE ; } -static u64 memory_peak_read (struct cgroup_subsys_state *css, -struct cftype *cft) +static struct page_counter *memcg_memory_extract_page_counter (struct mem_cgroup *memcg) { +return &memcg->memory; +} + +static struct memcg_peak_mem_ctx **memcg_memory_extract_peak_watchers (struct mem_cgroup *memcg) +{ +return &memcg->peak_memory_local_watermark_watchers; +} + +inline int swap_memory_peak_show ( +struct seq_file *sf,void *v, +struct page_counter *(*extract_pc)(struct mem_cgroup *memcg)) +{ +struct cgroup_subsys_state *css =seq_css (sf);struct mem_cgroup *memcg =mem_cgroup_from_css (css); +struct page_counter *pc =extract_pc (memcg); + +struct kernfs_open_file *of = sf->private; +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of); +s64 fd_peak = ctx->local_watermark; + +if (fd_peak ==-1 ) { +seq_printf (sf,"%llu\n" , (u64 )pc->watermark *PAGE_SIZE ); +return 0 ; + } + +s64 pc_peak = pc->local_watermark; +s64 wm = fd_peak > pc_peak ? fd_peak : pc_peak; + +seq_printf (sf,"%lld\n" , wm *PAGE_SIZE ); +return 0 ; +} +
Quoted 8 lines
mm/memcontrol.c static u64 memory_current_read(struct cgroup_subsys_state *css, · line 6948
Quoted 3 lines
mm/memcontrol.c +3
-0+static int memory_peak_show (struct seq_file *sf,void *v) +{ +return swap_memory_peak_show (sf, v, memcg_memory_extract_page_counter);I think it's really too complex. Why not pass a single boolean argument which will define to use memory
page_counteror swappage_counter? It will eliminate a need to pass pointers to functions and also eliminate a need for introducing these helper functions in general.
Yeah, that's definitely a cleaner option with only two alternatives. I'll make that change.
Quoted 48 lines
Quoted 48 lines
mm/memcontrol.c +45
-1+} + +static int swap_memory_peak_open (struct kernfs_open_file *of) +{ +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of); + + ctx->local_watermark =-1 ; +return 0 ; +} + +inline void swap_memory_peak_release ( +struct kernfs_open_file *of, +struct memcg_peak_mem_ctx **(*extract_watchers)(struct mem_cgroup *memcg)) +{ +struct mem_cgroup *memcg =mem_cgroup_from_css (of_css (of)); +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of); + +if (ctx->local_watermark ==-1 ) { +/* fast path (no writes on this fd)*/ +return ; + } +spin_lock (&memcg->pagecounter_peak_watchers_lock); +if (ctx->next) { + ctx->next->prev = ctx->prev; + } +if (ctx->prev) { + ctx->prev->next = ctx->next; + }else { +struct memcg_peak_mem_ctx **watchers =extract_watchers (memcg); + + *watchers = ctx->next; + } +spin_unlock (&memcg->pagecounter_peak_watchers_lock); +} -return (u64 )memcg->memory.watermark *PAGE_SIZE ; +static void memory_peak_release (struct kernfs_open_file *of) +{ +swap_memory_peak_release (of, memcg_memory_extract_peak_watchers); +} + +inline ssize_t swap_memory_peak_write ( +struct kernfs_open_file *of, +char *buf,size_t nbytes,loff_t off, +struct page_counter * (*extract_pc)(struct mem_cgroup *memcg), +struct memcg_peak_mem_ctx **(*extract_watchers)(struct mem_cgroup *memcg)) +{
Quoted 8 lines
mm/memcontrol.c static u64 memory_current_read(struct cgroup_subsys_state *css, · line 7000
Quoted 6 lines
mm/memcontrol.c +6
-0+ buf =strstrip (buf); +/* Only allow "fd_local_reset" to keep the API clear */ +if (strcmp (buf,"fd_local_reset" )) +return -EINVAL ; +struct mem_cgroup *memcg =mem_cgroup_from_css (of_css (of)); +struct memcg_peak_mem_ctx *ctx =memcg_extract_peak_mem_ctx (of);Please, don't mix variable definitions and code. Also, please, use
Ok, I'll factor that out
Quoted 2 lines
scripts/checkpatch.plfor checking the code before submission. I guess it will raise several issues in this patch.
Thanks, I did run checkpatch.pl on this patch, but I might have made a few changes
since my last pass. (it didn't complain about mixing cod and definitions)
I'll make another pass before mailing the next version.
Quoted 110 lines
Quoted 110 lines
mm/memcontrol.c +68
-6+ +spin_lock (&memcg->pagecounter_peak_watchers_lock); + +struct page_counter *pc =extract_pc (memcg); + +page_counter_reset_local_watermark (pc); +const unsigned long cur = pc->local_watermark; +struct memcg_peak_mem_ctx **watchers =extract_watchers (memcg); +struct memcg_peak_mem_ctx *peer_ctx; + +for (peer_ctx = *watchers; peer_ctx; peer_ctx = peer_ctx->next) { +if (cur > peer_ctx->local_watermark) + peer_ctx->local_watermark = cur; + } +if (ctx->local_watermark ==-1 ) { +/* only append to the list if we're not already there */ +if (peer_ctx) { + ctx->prev = peer_ctx; + peer_ctx->next = ctx; + }else { + *watchers = ctx; + } + } + ctx->local_watermark = cur; +spin_unlock (&memcg->pagecounter_peak_watchers_lock); + +return nbytes; +} + +static ssize_t memory_peak_write (struct kernfs_open_file *of,char *buf, +size_t nbytes,loff_t off) +{ +return swap_memory_peak_write (of, buf, nbytes, off, + memcg_memory_extract_page_counter, + memcg_memory_extract_peak_watchers); }static int memory_min_show (struct seq_file *m,void *v)@@ -7231,7 +7351,10 @@ static struct cftype memory_files[] = { { .name = "peak", .flags = CFTYPE_NOT_ON_ROOT, - .read_u64 = memory_peak_read, + .open = swap_memory_peak_open, + .release = memory_peak_release, + .seq_show = memory_peak_show, + .write = memory_peak_write, }, { .name = "min",@@ -8193,14 +8316,35 @@ static u64 swap_current_read(struct cgroup_subsys_state *css,return (u64 )page_counter_read (&memcg->swap) *PAGE_SIZE ; } -static u64 swap_peak_read (struct cgroup_subsys_state *css, -struct cftype *cft) + +static struct page_counter *memcg_swap_extract_page_counter (struct mem_cgroup *memcg) +{ +return &memcg->swap; +} + +static struct memcg_peak_mem_ctx **memcg_swap_extract_peak_watchers (struct mem_cgroup *memcg) { -struct mem_cgroup *memcg =mem_cgroup_from_css (css); +return &memcg->peak_swap_local_watermark_watchers; +} + +static int swap_peak_show (struct seq_file *sf,void *v) +{ +return swap_memory_peak_show (sf, v, memcg_swap_extract_page_counter); +} -return (u64 )memcg->swap.watermark *PAGE_SIZE ; +static ssize_t swap_peak_write (struct kernfs_open_file *of,char *buf, +size_t nbytes,loff_t off) +{ +return swap_memory_peak_write (of, buf, nbytes, off, + memcg_swap_extract_page_counter, + memcg_swap_extract_peak_watchers); +} +static void swap_peak_release (struct kernfs_open_file *of) +{ +swap_memory_peak_release (of, memcg_swap_extract_peak_watchers); } +static int swap_high_show (struct seq_file *m,void *v) { return seq_puts_memcg_tunable(m,@@ -8282,7 +8426,10 @@ static struct cftype swap_files[] = { { .name = "swap.peak", .flags = CFTYPE_NOT_ON_ROOT, - .read_u64 = swap_peak_read, + .open = swap_memory_peak_open, + .release = swap_peak_release, + .seq_show = swap_peak_show, + .write = swap_peak_write, }, { .name = "swap.events",mm/page_counter.c +0
-0@@ -82,6 +82,8 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages)*/ if (new >READ_ONCE (c->watermark))
Quoted 8 lines
mm/page_counter.c void page_counter_charge(struct page_counter *counter, unsigned long nr_pages) · line 86
Quoted 3 lines
mm/page_counter.c +2
-0WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) +WRITE_ONCE (c->local_watermark, new);Hm, can't we have a single comparison on the hot path?
Also, we read and writec->local_watermarkspeculatively here, Idk if it's still acceptable with an ability to reset watermarks "locally". Maybe it is, but it definitely deserves at least a comment with an explanation.
Unfortunately, since the two watermarks may be reset at different times I don't think we
can consolidate.
e.g. I think that if the usage peaked, dropped down a bit and then was going back
up again when the "local_watermark" was reset, we'll continue only bumping local_watermark,
but we don't want to touch "watermark" until we hit that watermark again.
Quoted 3 lines
And btw thank you for including tests into the commit, it's really great to see. I'd suggest you to extract them into a separate commit and post it as a series.
Sure thing!
I'll split them off into their own commit. (the tests are about half the line-delta)
Quoted 2 lines
Thank you!
Thanks for the prompt review!
I'll try to get another revision out later today.
Signature · 2 lines
David Finkel Senior Principal Software Engineer, Core Services
Re: [PATCH] mm, memcg: cg2 memory{.swap,}.peak write handlers
Quoted 14 lines · David Finkel
Quoted 6 lines
Quoted 6 lines
mm/page_counter.c +0
-0@@ -82,6 +82,8 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages)*/ Quoted 8 lines
mm/page_counter.c void page_counter_charge(struct page_counter *counter, unsigned long nr_pages) · line 86
Quoted 4 lines
mm/page_counter.c +2
-0if (new >READ_ONCE (c->watermark))WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) +WRITE_ONCE (c->local_watermark, new);Hm, can't we have a single comparison on the hot path?
Also, we read and writec->local_watermarkspeculatively here, Idk if it's still acceptable with an ability to reset watermarks "locally". Maybe it is, but it definitely deserves at least a comment with an explanation.
Quoted 8 lines
Unfortunately, since the two watermarks may be reset at different times I don't think we
can consolidate.
e.g. I think that if the usage peaked, dropped down a bit and then was going back
up again when the "local_watermark" was reset, we'll continue only bumpinglocal_watermark,
but we don't want to touch "watermark" until we hit that watermark again.
If we make page_counter_reset_watermark() reset the local_watermark as well, we can guarantee "local_watermark <= watermark" and wrap one check inside the other.
if (new > READ_ONCE(c->local_watermark)) { WRITE_ONCE(c->local_watermark, new); if (new > READ_ONCE(c->watermark)) WRITE_ONCE(c->watermark, new); }
Cheers,
Longman
Re: [PATCH] mm, memcg: cg2 memory{.swap,}.peak write handlers
Quoted 28 lines · Waiman Long
Quoted 14 lines · David Finkel
Quoted 6 lines
Quoted 6 lines
mm/page_counter.c +0
-0@@ -82,6 +82,8 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages)*/ Quoted 8 lines
mm/page_counter.c void page_counter_charge(struct page_counter *counter, unsigned long nr_pages) · line 86
Quoted 4 lines
mm/page_counter.c +2
-0if (new >READ_ONCE (c->watermark))WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) +WRITE_ONCE (c->local_watermark, new);Hm, can't we have a single comparison on the hot path?
Also, we read and writec->local_watermarkspeculatively here, Idk if it's still acceptable with an ability to reset watermarks "locally". Maybe it is, but it definitely deserves at least a comment with an explanation.Quoted 8 lines
Unfortunately, since the two watermarks may be reset at different times I don't think we
can consolidate.
e.g. I think that if the usage peaked, dropped down a bit and then was going back
up again when the "local_watermark" was reset, we'll continue only bumpinglocal_watermark,
but we don't want to touch "watermark" until we hit that watermark again.If we make
page_counter_reset_watermark()reset thelocal_watermarkas well, we can guarantee "local_watermark<= watermark" and wrap one check inside the other.
Quoted 9 lines
if (new > READ_ONCE(c->local_watermark)) { WRITE_ONCE(c->local_watermark, new); if (new > READ_ONCE(c->watermark)) WRITE_ONCE(c->watermark, new); }Cheers,
Longman
Hmm, yeah, given that we'll only be resetting one of the two, I think I'll use this option.
The branch predictor should make that second check pretty
much a noop in the common-case when we enter the outer if, too.
Thanks!
Signature · 2 lines
David Finkel Senior Principal Software Engineer, Core Services