mm, memcg: cg2 memory{.swap,}.peak write handlers
Subsystems (6)
PATCH v7 1/1 mm, memcg: cg2 memory{.swap,}.peak write handlers
This revision only updates the tests from the previous revision[1], and integrates an Acked-by[2] and a Reviewed-By[3] into the first commit message.
Documentation/admin-guide/cgroup-v2.rst | 22 ++- include/linux/cgroup-defs.h | 5 + include/linux/cgroup.h | 3 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 11 +- kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 116 +++++++++++++-- mm/page_counter.c | 30 +++- tools/testing/selftests/cgroup/cgroup_util.c | 22 +++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + tools/testing/selftests/cgroup/test_memcontrol.c | 264 ++++++++++++++++++++++++++++++++- 12 files changed, 454 insertions(+), 35 deletions(-)
[1]: https://lore.kernel.org/cgroups/[email protected]/T/
[2]: https://lore.kernel.org/cgroups/[email protected]/T/#m807225dd0944b0bf78419639272bf6602fe053fc
[3]: https://lore.kernel.org/cgroups/[email protected]/T/#meac510a72b4a282fe1e5edec3323c2204d46cf11
Thank you all for the support and reviews so far!
David Finkel
Senior Principal Software Engineer
Vimeo Inc.
PATCH v7 1/2 mm, memcg: cg2 memory{.swap,}.peak write handlers
Other mechanisms for querying the peak memory usage of either a process or v1 memory cgroup allow for resetting the high watermark. Restore parity with those mechanisms, but with a less racy API.
For example:
- Any write to memory.max_usage_in_bytes in a cgroup v1 mount resets the high watermark.
- writing "5" to the clear_refs pseudo-file in a processes's proc directory resets the peak RSS.
This change is an evolution of a previous patch, which mostly copied the cgroup v1 behavior, however, there were concerns about races/ownership issues with a global reset, so instead this change makes the reset filedescriptor-local.
Writing any non-empty string to the memory.peak and memory.swap.peak pseudo-files reset the high watermark to the current usage for subsequent reads through that same FD.
Notably, following Johannes's suggestion, this implementation moves the O(FDs that have written) behavior onto the FD write(2) path. Instead, on the page-allocation path, we simply add one additional watermark to conditionally bump per-hierarchy level in the page-counter.
Additionally, this takes Longman's suggestion of nesting the page-charging-path checks for the two watermarks to reduce the number of common-case comparisons.
This behavior is particularly useful for work scheduling systems that need to track memory usage of worker processes/cgroups per-work-item. Since memory can't be squeezed like CPU can (the OOM-killer has opinions), these systems need to track the peak memory usage to compute system/container fullness when binpacking workitems.
Most notably, Vimeo's use-case involves a system that's doing global binpacking across many Kubernetes pods/containers, and while we can use PSI for some local decisions about overload, we strive to avoid packing workloads too tightly in the first place. To facilitate this, we track the peak memory usage. However, since we run with long-lived workers (to amortize startup costs) we need a way to track the high watermark while a work-item is executing. Polling runs the risk of missing short spikes that last for timescales below the polling interval, and peak memory tracking at the cgroup level is otherwise perfect for this use-case.
As this data is used to ensure that binpacked work ends up with sufficient headroom, this use-case mostly avoids the inaccuracies surrounding reclaimable memory.
--- Documentation/admin-guide/cgroup-v2.rst | 22 +++-- include/linux/cgroup-defs.h | 5 + include/linux/cgroup.h | 3 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 11 ++- kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 ++ mm/memcontrol.c | 116 ++++++++++++++++++++++-- mm/page_counter.c | 30 ++++-- 9 files changed, 174 insertions(+), 27 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +14 -8
@@ -1333,11 +1333,14 @@ The following nested keys are defined. all the existing limitations and potential future extensions. memory.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max memory usage recorded for the cgroup and its descendants since + either the creation of the cgroup or the most recent reset for that FD. - The max memory usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of any non-empty string to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. memory.oom.group A read-write single value file which exists on non-root
@@ -1663,11 +1666,14 @@ The following nested keys are defined. Healthy workloads are not expected to reach this limit. memory.swap.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max swap usage recorded for the cgroup and its descendants since + the creation of the cgroup or the most recent reset for that FD. - The max swap usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of any non-empty string to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. memory.swap.max A read-write single value file which exists on non-root
include/linux/cgroup-defs.h +5 -0
@@ -775,6 +775,11 @@ struct cgroup_subsys {extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem; +struct cgroup_of_peak { +unsigned long value; +struct list_head list; +}; +/** * cgroup_threadgroup_change_begin - threadgroup exclusion for cgroups * @tsk: target task
include/linux/cgroup.h +3 -0
@@ -11,6 +11,7 @@#include <linux/sched.h> #include <linux/nodemask.h> +#include <linux/list.h> #include <linux/rculist.h> #include <linux/cgroupstats.h> #include <linux/fs.h>
@@ -854,4 +855,6 @@ static inline void cgroup_bpf_put(struct cgroup *cgrp) {}struct cgroup *task_get_cgroup1 (struct task_struct *tsk,int hierarchy_id); +struct cgroup_of_peak *of_peak (struct kernfs_open_file *of); +#endif /* _LINUX_CGROUP_H */
include/linux/memcontrol.h +5 -0
@@ -193,6 +193,11 @@ struct mem_cgroup {struct page_counter memsw;/* v1 only */ }; +/* registered local peak watchers */ +struct list_head memory_peaks; +struct list_head swap_peaks; +spinlock_t peaks_lock; +/* Range enforcement for interrupt charges */ struct work_struct high_work;
include/linux/page_counter.h +10 -1
@@ -26,6 +26,8 @@ struct page_counter {atomic_long_t children_low_usage;unsigned long watermark; +/* Latest cg2 reset watermark */ +unsigned long local_watermark;unsigned long failcnt;/* Keep all the read most fields in a separete cacheline. */
@@ -78,7 +80,14 @@ int page_counter_memparse(const char *buf, const char *max,static inline void page_counter_reset_watermark (struct page_counter *counter) { - counter->watermark =page_counter_read (counter); +unsigned long usage =page_counter_read (counter); + +/* +* Update local_watermark first, so it's always <= watermark +* (modulo CPU/compiler re-ordering) +*/ + counter->local_watermark = usage; + counter->watermark = usage; } void page_counter_calculate_protection(struct page_counter *root,
kernel/cgroup/cgroup-internal.h +2 -0
@@ -81,6 +81,8 @@ struct cgroup_file_ctx {struct {struct cgroup_pidlist *pidlist; } procs1; + +struct cgroup_of_peak peak; };/*
kernel/cgroup/cgroup.c +7 -0
@@ -1972,6 +1972,13 @@ static int cgroup2_parse_param(struct fs_context *fc, struct fs_parameter *paramreturn -EINVAL ; } +struct cgroup_of_peak *of_peak (struct kernfs_open_file *of) +{ +struct cgroup_file_ctx *ctx = of->priv; + +return &ctx->peak; +} +static void apply_cgroup_root_flags (unsigned int root_flags) {if (current->nsproxy->cgroup_ns == &init_cgroup_ns) {
mm/memcontrol.c +106 -10
164 lines not on this page · raw message
mm/page_counter.c +22 -8
@@ -79,9 +79,22 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages)/* * This is indeed racy, but we can live with some * inaccuracy in the watermark. +* +* Notably, we have two watermarks to allow for both a globally +* visible peak and one that can be reset at a smaller scope. +* +* Since we reset both watermarks when the global reset occurs, +* we can guarantee that watermark >= local_watermark, so we +* don't need to do both comparisons every time. +* +* On systems with branch predictors, the inner condition should +* be almost free. */ -if (new >READ_ONCE (c->watermark)) -WRITE_ONCE (c->watermark, new); +if (new >READ_ONCE (c->local_watermark)) { +WRITE_ONCE (c->local_watermark, new); +if (new >READ_ONCE (c->watermark)) +WRITE_ONCE (c->watermark, new); + } } }
@@ -129,12 +142,13 @@ bool page_counter_try_charge(struct page_counter *counter, goto failed; }propagate_protected_usage (c, new); -/* -* Just like with failcnt, we can live with some -* inaccuracy in the watermark. -*/ -if (new >READ_ONCE (c->watermark)) -WRITE_ONCE (c->watermark, new); + +/* see comment on page_counter_charge */ +if (new >READ_ONCE (c->local_watermark)) { +WRITE_ONCE (c->local_watermark, new); +if (new >READ_ONCE (c->watermark)) +WRITE_ONCE (c->watermark, new); + } }return true;
Signature · 1 line
2.40.1
Quoted 48 lines · David Finkel
Other mechanisms for querying the peak memory usage of either a process or v1 memory cgroup allow for resetting the high watermark. Restore parity with those mechanisms, but with a less racy API.
For example:
- Any write to memory.max_usage_in_bytes in a cgroup v1 mount resets the high watermark.
- writing "5" to theclear_refspseudo-file in a processes's proc directory resets the peak RSS.This change is an evolution of a previous patch, which mostly copied the cgroup v1 behavior, however, there were concerns about races/ownership issues with a global reset, so instead this change makes the reset filedescriptor-local.
Writing any non-empty string to the memory.peak and memory.swap.peak pseudo-files reset the high watermark to the current usage for subsequent reads through that same FD.
Notably, following Johannes's suggestion, this implementation moves the O(FDs that have written) behavior onto the FD write(2) path. Instead, on the page-allocation path, we simply add one additional watermark to conditionally bump per-hierarchy level in the page-counter.
Additionally, this takes Longman's suggestion of nesting the page-charging-path checks for the two watermarks to reduce the number of common-case comparisons.
This behavior is particularly useful for work scheduling systems that need to track memory usage of worker processes/cgroups per-work-item. Since memory can't be squeezed like CPU can (the OOM-killer has opinions), these systems need to track the peak memory usage to compute system/container fullness when binpacking workitems.
Most notably, Vimeo's use-case involves a system that's doing global binpacking across many Kubernetes pods/containers, and while we can use PSI for some local decisions about overload, we strive to avoid packing workloads too tightly in the first place. To facilitate this, we track the peak memory usage. However, since we run with long-lived workers (to amortize startup costs) we need a way to track the high watermark while a work-item is executing. Polling runs the risk of missing short spikes that last for timescales below the polling interval, and peak memory tracking at the cgroup level is otherwise perfect for this use-case.
As this data is used to ensure that binpacked work ends up with sufficient headroom, this use-case mostly avoids the inaccuracies surrounding reclaimable memory.
Quoted 5 lines
Suggested-by Johannes Weiner <[email protected]>Suggested-by Waiman Long <[email protected]>Acked-by Johannes Weiner <[email protected]>Reviewed-by Michal Koutný <[email protected]>Signed-off-by David Finkel <[email protected]>
Thanks!
PATCH v7 2/2 mm, memcg: cg2 memory{.swap,}.peak write tests
In reply to David FinkelExtend two existing tests to cover extracting memory usage through the newly mutable memory.peak and memory.swap.peak handlers.
In particular, make sure to exercise adding and removing watchers with overlapping lifetimes so the less-trivial logic gets tested.
The new/updated tests attempt to detect a lack of the write handler by fstat'ing the memory.peak and memory.swap.peak files and skip the tests if that's the case. Additionally, skip if the file doesn't exist at all.
--- tools/testing/selftests/cgroup/cgroup_util.c | 22 ++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + .../selftests/cgroup/test_memcontrol.c | 264 +++++++++++++++++- 3 files changed, 280 insertions(+), 8 deletions(-)
tools/testing/selftests/cgroup/cgroup_util.c +22 -0
@@ -141,6 +141,16 @@ long cg_read_long(const char *cgroup, const char *control)return atol (buf); } +long cg_read_long_fd (int fd) +{ +char buf[128 ]; + +if (pread (fd, buf,sizeof (buf),0 ) <=0 ) +return -1 ; + +return atol (buf); +} +long cg_read_key_long (const char *cgroup,const char *control,const char *key) {char buf[PAGE_SIZE ];
@@ -183,6 +193,18 @@ int cg_write(const char *cgroup, const char *control, char *buf)return ret == len ?0 : ret; } +/* +* Returns fd on success, or -1 on failure. +* (fd should be closed with close() as usual) +*/ +int cg_open (const char *cgroup,const char *control,int flags) +{ +char path[PATH_MAX ]; + +snprintf (path,sizeof (path),"%s/%s" , cgroup, control); +return open (path, flags); +} +int cg_write_numeric (const char *cgroup,const char *control,long value) {char buf[64 ];
tools/testing/selftests/cgroup/cgroup_util.h +2 -0
@@ -34,9 +34,11 @@ extern int cg_read_strcmp(const char *cgroup, const char *control,extern int cg_read_strstr (const char *cgroup,const char *control,const char *needle);extern long cg_read_long (const char *cgroup,const char *control); +extern long cg_read_long_fd (int fd);long cg_read_key_long (const char *cgroup,const char *control,const char *key);extern long cg_read_lc (const char *cgroup,const char *control);extern int cg_write (const char *cgroup,const char *control,char *buf); +extern int cg_open (const char *cgroup,const char *control,int flags);int cg_write_numeric (const char *cgroup,const char *control,long value); extern int cg_run(const char *cgroup, int (*fn)(const char *cgroup, void *arg),
tools/testing/selftests/cgroup/test_memcontrol.c +256 -8
332 lines not on this page · raw message
Signature · 1 line
2.40.1
Re: [PATCH v7 2/2] mm, memcg: cg2 memory{.swap,}.peak write tests
Quoted 11 lines · David Finkel
Extend two existing tests to cover extracting memory usage through the newly mutable memory.peak and memory.swap.peak handlers.
In particular, make sure to exercise adding and removing watchers with overlapping lifetimes so the less-trivial logic gets tested.
The new/updated tests attempt to detect a lack of the write handler by fstat'ing the memory.peak and memory.swap.peak files and skip the tests if that's the case. Additionally, skip if the file doesn't exist at all.
Signed-off-by David Finkel <[email protected]>
Quoted 25 lines · David Finkel
This revision only updates the tests from the previous revision[1], and integrates an Acked-by[2] and a Reviewed-By[3] into the first commit message.
Documentation/admin-guide/cgroup-v2.rst | 22 ++- include/linux/cgroup-defs.h | 5 + include/linux/cgroup.h | 3 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 11 +- kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 116 +++++++++++++-- mm/page_counter.c | 30 +++- tools/testing/selftests/cgroup/cgroup_util.c | 22 +++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + tools/testing/selftests/cgroup/test_memcontrol.c | 264 ++++++++++++++++++++++++++++++++- 12 files changed, 454 insertions(+), 35 deletions(-)[1]: https://lore.kernel.org/cgroups/[email protected]/T/
[2]: https://lore.kernel.org/cgroups/[email protected]/T/#m807225dd0944b0bf78419639272bf6602fe053fc
[3]: https://lore.kernel.org/cgroups/[email protected]/T/#meac510a72b4a282fe1e5edec3323c2204d46cf11
Quoted 8 lines
Thank you all for the support and reviews so far!
David Finkel
Senior Principal Software Engineer
Vimeo Inc.
Tejun or Andrew,
This change seems to have stalled a bit.
Are there any further changes necessary to get this patch merged into a staging branch so it's ready for 6.12?
Thanks,
Signature · 2 lines
David Finkel Senior Principal Software Engineer, Core Services
Quoted 20 lines · David Finkel
Quoted 19 lines · David Finkel
This revision only updates the tests from the previous revision[1], and integrates an Acked-by[2] and a Reviewed-By[3] into the first commit message.
Documentation/admin-guide/cgroup-v2.rst | 22 ++- include/linux/cgroup-defs.h | 5 + include/linux/cgroup.h | 3 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 11 +- kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 116 +++++++++++++-- mm/page_counter.c | 30 +++- tools/testing/selftests/cgroup/cgroup_util.c | 22 +++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + tools/testing/selftests/cgroup/test_memcontrol.c | 264 ++++++++++++++++++++++++++++++++- 12 files changed, 454 insertions(+), 35 deletions(-)
...
Quoted 5 lines
Tejun or Andrew,
This change seems to have stalled a bit.
Are there any further changes necessary to get this patch merged into a staging branch so it's ready for 6.12?
Oh, it sits between cgroup core and memcg, so I guess it wasn't clear who should take it. Given that the crux of the change is in memcg, I think -mm would be a better fit. Andrew, can you please take these patches? FWIW,
Thanks.
Signature · 1 line
tejun
Quoted 29 lines · Tejun Heo
Quoted 20 lines · David Finkel
Quoted 19 lines · David Finkel
This revision only updates the tests from the previous revision[1], and integrates an Acked-by[2] and a Reviewed-By[3] into the first commit message.
Documentation/admin-guide/cgroup-v2.rst | 22 ++- include/linux/cgroup-defs.h | 5 + include/linux/cgroup.h | 3 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 11 +- kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 116 +++++++++++++-- mm/page_counter.c | 30 +++- tools/testing/selftests/cgroup/cgroup_util.c | 22 +++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + tools/testing/selftests/cgroup/test_memcontrol.c | 264 ++++++++++++++++++++++++++++++++- 12 files changed, 454 insertions(+), 35 deletions(-)...
Quoted 5 lines
Tejun or Andrew,
This change seems to have stalled a bit.
Are there any further changes necessary to get this patch merged into a staging branch so it's ready for 6.12?
Quoted 7 lines
Oh, it sits between cgroup core and memcg, so I guess it wasn't clear who should take it. Given that the crux of the change is in memcg, I think -mm would be a better fit. Andrew, can you please take these patches? FWIW,
Acked-by Tejun Heo <[email protected]>Thanks.
Thanks
Quoted 3 lines
--
tejun
Signature · 2 lines
David Finkel Senior Principal Software Engineer, Core Services
Andrew Morton
maintainerQuoted 24 lines · On Tue, 6 Aug 2024 10:18:47 -1000 Tejun Heo <[email protected]> wrote:
Quoted 20 lines · David Finkel
Quoted 19 lines · David Finkel
This revision only updates the tests from the previous revision[1], and integrates an Acked-by[2] and a Reviewed-By[3] into the first commit message.
Documentation/admin-guide/cgroup-v2.rst | 22 ++- include/linux/cgroup-defs.h | 5 + include/linux/cgroup.h | 3 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 11 +- kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 116 +++++++++++++-- mm/page_counter.c | 30 +++- tools/testing/selftests/cgroup/cgroup_util.c | 22 +++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + tools/testing/selftests/cgroup/test_memcontrol.c | 264 ++++++++++++++++++++++++++++++++- 12 files changed, 454 insertions(+), 35 deletions(-)...
Quoted 2 lines
Tejun or Andrew,
Quoted 7 lines
Quoted 3 lines
This change seems to have stalled a bit.
Are there any further changes necessary to get this patch merged into a staging branch so it's ready for 6.12?Oh, it sits between cgroup core and memcg, so I guess it wasn't clear who should take it. Given that the crux of the change is in memcg, I think -mm would be a better fit. Andrew, can you please take these patches? FWIW,
I took 'em on Aug 4. David, your spam folder beckons?
Quoted 1 line
Acked-by Tejun Heo <[email protected]>
Added, thanks.
Quoted 31 lines · Andrew Morton
Quoted 28 lines · On Tue, 6 Aug 2024 10:18:47 -1000 Tejun Heo <[email protected]> wrote:
Quoted 20 lines · David Finkel
Quoted 19 lines · David Finkel
This revision only updates the tests from the previous revision[1], and integrates an Acked-by[2] and a Reviewed-By[3] into the first commit message.
Documentation/admin-guide/cgroup-v2.rst | 22 ++- include/linux/cgroup-defs.h | 5 + include/linux/cgroup.h | 3 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 11 +- kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 + mm/memcontrol.c | 116 +++++++++++++-- mm/page_counter.c | 30 +++- tools/testing/selftests/cgroup/cgroup_util.c | 22 +++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + tools/testing/selftests/cgroup/test_memcontrol.c | 264 ++++++++++++++++++++++++++++++++- 12 files changed, 454 insertions(+), 35 deletions(-)...
Quoted 5 lines
Tejun or Andrew,
This change seems to have stalled a bit.
Are there any further changes necessary to get this patch merged into a staging branch so it's ready for 6.12?
Quoted 5 lines
Quoted 3 lines
Oh, it sits between cgroup core and memcg, so I guess it wasn't clear who should take it. Given that the crux of the change is in memcg, I think -mm would be a better fit. Andrew, can you please take these patches? FWIW,
I took 'em on Aug 4.
Great!
David, your spam folder beckons?
Oddly, I don't see any email from you on the 4th. I do see my patches in mm-unstable now.
(I didn't see them there when I looked over the weekend)
With that said, I had a message go similarly mysteriously missing yesterday, so it's possible something's wrong with our mail system.
Quoted 4 lines
Quoted 2 lines
Acked-by Tejun Heo <[email protected]>Added, thanks.
Thanks again!
Signature · 2 lines
David Finkel Senior Principal Software Engineer, Core Services