Skip to contentSkip to conversation

memcg_ext: dynamic memory.low protection requests

RFC PATCH New Hui Zhu 1 participant 3 messages Nov 11

Subsystems (11)
Networking Network drivers Memory management Kselftest +7 more subsystems
lore
Share & tools
lore

Hui Zhu

author
#0 lore raw

v2 cover memcg_ext: dynamic memory.low protection requests

From: Hui Zhu <[email protected]>

This is v2 of the RFC "memcg_ext: dynamic memory.low and memory.min protection" [1]. It sits on the memcg_ext base [2], which adds bpf_memcg_ops - a struct_ops type attaching memory controller policy to a cgroup - and lets a BPF policy take part in memory.low protection:
not by deciding whether reclaim skips the cgroup, which is what the RFC did and what the review correctly rejected, but by requesting protection and going through the same arbitration as a write to memory.low.

Two things changed since the RFC, and both are answers to the review.

THE CEILING

As posted, a policy could make reclaim skip a cgroup whose usage was above what its parent affords. That defeats the cap effective_protection() applies at subsequent levels precisely to keep delegation safe: a delegated tenant can attach a policy to its own cgroup, so it could grant itself protection its parent never gave. That was a defect in the interface, not a property to defend.

In v2 the hook no longer decides anything; it requests. The request is collected before the effective value is derived, and what a cgroup claims becomes the larger of its memory.low and the request - capped by the parent's effective protection and shared out with the siblings by the same code, on the same inputs, as a declared value. A request can move protection around inside the parent's budget; it cannot lift the subtree above what the parent affords. Anything a policy can request, a write to memory.low can request too. What is left to the policy is when and on what evidence the request is made: every pass, at the point where the effective value is about to be derived, rather than between passes.

min_policy is dropped rather than converted: this round concentrates on memory.low, and the interface goes from two members to one. A memory.min counterpart can be revisited later.

The rework also removes the memcontrol.h change entirely:
mem_cgroup_below_low() and mem_cgroup_below_min() keep their signatures and the series no longer touches the protection checks.

One cost is stated here rather than left to be found: the callback now runs for every memcg on every pass - with MGLRU that includes lru_gen_age_node() walking every memcg on the node - where the RFC asked only about the cgroups reclaim had found unprotected. With nothing attached, a call is a page counter read, a static key check and a comparison against the stored request. Measured under the reclaim storm of the demonstration below: ~56k hook calls in a 30s window, 52ns average per call, 2.9ms of in-hook CPU in total - 0.01% of one core.

THE MEASUREMENT

The review asked why the same cannot be achieved by changing memory.low from userspace - the same question Michal asked on the v7 thread [3] in May. This time the answer is a measurement rather than an argument.

The claim under test is a window, not a throughput. Direct reclaim runs in the allocating task's own context, and shrink_node_memcgs() examines memcgs one at a time, calling mem_cgroup_calculate_protection() per memcg. An agent cannot do anything for the memcgs a pass has already examined, and for the ones it has not reached, a concurrent write to memory.low is picked up only once the agent has been woken, scheduled, read memory.stat or memory.pressure, decided and written - in a burst that window holds at least one full pass, which is where the working set goes. After the rework the hook sits at the point where the request is collected, so it is asked on that pass.

Method. Three implementations run the same integer policy on the same evidence and differ only in where the decision runs:

U userspace agent: preads the victim's memory.current on a 5ms
timer and pwrites the sample to memory.low. Made as strong as I can: pre-opened fds, no allocation in the loop, and its own cgroup with memory.min so it survives the pressure (disclosed). A PSI-poll trigger variant exists and was measured too.
B1 a bpf_wq callback on a 5ms timer reads the same counter through
bpf_mem_cgroup_usage() and stores the verdict; the low_request hook executes it. The control for "the action runs in the kernel".
B2 the low_request hook decides from ctx->usage - the value the
kernel read for this very pass - on every protection pass. The only arm that claims something userspace cannot do.

The shared evidence is the fall rate of the victim's memory.current:
reclaim uncharges it synchronously in-pass, so a falling usage means "the victim is being stolen from" with no statistics-flush dependency. The textbook channels were tried and measured blind: workingset refaults need the shadow entry to survive eviction (96M stolen from the victim counted as 2 refault pages under the filler's shrinker churn), and pgsteal is only readable from flushed counters (a whole burst window passed with the in-hook arm seeing zero movement while the victim lost 19k pages). The threshold is a rate - pages per second, not a per-sample fall - so the ~10x spread in sampling frequency between the arms does not bias the comparison.

Workload. Under T (memory.max=160M, memory.low=64M, swap off): a victim A (32M file working set re-read on a 500ms pace, hooks attached), a filler B (112M of file cache rewritten continuously - a renewable flow of cold pages that keeps satisfying T's reclaim while A is protected), and a one-shot 64M anonymous burst C that is unreclaimable without swap and raises A's share of the scan. Metric:
A's pgsteal over a 30s window. Protocol: 15 rounds, arms in rotating order, a sanity gate every round that verifies the protection chain before any round counts, cache dropped between runs, every knob identical across arms and rounds; then a sweep of U's trigger interval (1/5/20/100ms timer and PSI poll, five repetitions per point, reported as medians). QEMU VM, 8 GiB, no swap, booted with psi=1.

Results (medians [IQR], 15 rounds per arm):

stolen from A B2 452 [339-755] U 3723 [2825-4730] B1 3960 [3211-4853]

- B2's median is 8.2x below U's and 8.8x below B1's, and the IQRs do not overlap (Mann-Whitney p < 1e-4). In 14 of 15 rounds B2's worst (968 pages) stayed below both other arms' best round (2051 / 2615).
- The tails overlap only where the kernel itself broke protection:
B2's outlier round logged 920 MEMCG_LOW overrides, and B1's own outlier - 9303 pages, the worst single round in the dataset - logged 352. That is memory.low's documented soft ceiling: reclaim legally overrides it when T's max cannot be satisfied from unprotected supply. It applies to every arm and to any policy alike; memory.min is the hard guarantee and is out of scope here.
- Overhead: B2's entire in-hook cost is 2.9ms per 30s window (~56k calls at 52ns average, 0.01% of one core); U's agent costs 140ms for ~6.7k decisions and 187 memory.low writes; B1's hook is 1.7ms plus ~7k wq timer callbacks whose CPU is not in that figure.
- U and B1 are statistically indistinguishable. Moving the action into the kernel buys nothing by itself; what makes the difference is deciding at the collection point.
- The sweep answers "poll faster". U at 1ms leaks a median 1308 pages at 374ms of agent CPU per window; at 5ms, 5330 at 138ms; at 20ms, 7746 at 34ms; at 100ms it engages in only 2 of 5 windows and by then the drain has happened. B2 beats U's best point (452 vs 1308) at 1/129 of that point's CPU.
- The PSI-poll trigger never fired in any window - a publishable negative result, not a bug: the victim stalls 0.3-0.6 ms/s, an order of magnitude below the 10 ms/s trigger, while the filler stalls 39-44 ms/s. The cgroup being stolen from is the last place PSI-on-the-victim can see it.
- Time to first protection is statistically identical across the arms (medians 2.1-2.8s): under this pacing it measures when a drain episode happens, not how fast an arm reacts inside one, so I do not rank arms by it.

In short: the same policy, decided at the collection point instead of out of band, cut the victim's losses ~8x at 2% of the userspace agent's CPU, and beat even the agent's most aggressive polling point while costing 129x less CPU. The scripts and the protocol description are in [4]; the raw CSV rows and per-run logs behind the numbers above are at [5].

THE PATCHES

1/2 memcg_ext: arbitrate low protection requests like memory.low
struct page_counter grows low_req; low_setting() becomes max(low, low_req) everywhere protection is derived; and the bpf_memcg_ops callback moves from mem_cgroup_below_low() to mem_cgroup_calculate_protection(), so the request is collected before elow is computed. The request is recomputed every pass and reported as none whenever no program can run, so detaching the last program drops what it asked for on the next pass - nothing to clean up.

2/2 selftests/bpf: Add bpf_memcg_ops protection test
A three-level tree, a policy on the grandchild requesting its current usage, memory.low on the middle cgroup as the budget, and reclaim from the top. Three budgets, same request: with 0 the effective value stays 0; with 2M an 8M request comes back capped to 2M and reclaim still goes into the victim; with 16M it is granted in full, reclaim skips the victim and memory.events low moves. The ctx contents are checked along the way.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://github.com/teawater/memcg-async-reclaim/tree/memcg_ext_20261109
[5] https://github.com/teawater/memcg-async-reclaim/tree/memcg_ext_20261109/log/run-20261111-071332

Hui Zhu (2):
memcg_ext: arbitrate low protection requests like memory.low selftests/bpf: Add bpf_memcg_ops protection test

 include/linux/bpf_memcontrol.h                |  97 +++++-
 include/linux/page_counter.h                  |   8 +
 mm/bpf_memcontrol.c                           | 155 ++++++---
 mm/memcontrol.c                               |  30 ++
 mm/page_counter.c                             |  51 ++-
 .../bpf/prog_tests/memcg_ops_protection.c     | 297 ++++++++++++++++++
 .../bpf/progs/memcg_ops_protection.c          |  82 +++++
 7 files changed, 678 insertions(+), 42 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_ops_protection.c
 create mode 100644 tools/testing/selftests/bpf/progs/memcg_ops_protection.c
Signature · 1 line
2.43.0
author
#1 lore raw

RFC PATCH v2 1/2 memcg_ext: arbitrate low protection requests like memory.low

From: Hui Zhu <[email protected]>

low_policy and min_policy ran after the effective value had already been computed, and BELOW made reclaim skip the cgroup outright. That bypasses the cap effective_protection() applies at subsequent levels:
a policy attached by a delegated tenant to its own cgroup could grant itself protection its parent never afforded.

Collect the request before the effective value is computed instead, and arbitrate it like a declared memory.low: what a cgroup claims is the larger of memory.low and the request, still capped by the parent's effective protection and shared out with the siblings. A request can move protection around inside the parent's budget but cannot lift the subtree above it. What is left to the policy is when and on what evidence to ask - every pass, where the effective value is derived - not how much it can get.

This round covers memory.low only. The RFC's min_policy is dropped:
hard protection has no fallback once nothing else is reclaimable, so a request recomputed every pass could OOM cgroups that never agreed to the policy. A memory.min counterpart can be revisited once that has an answer.

The callback moves from mem_cgroup_below_low() to mem_cgroup_calculate_protection() and its _path() variant; both helpers keep their signatures and memcontrol.h drops the include. The request is recomputed on every pass and reported as none when no program can run, so detaching the last program drops what it asked for on the next pass - nothing to clean up. The cost is a callback for every cgroup reclaim visits (with MGLRU, every memcg on the node); with nothing attached that is a page counter read, a static key check and one comparison.

Signed-off-by Hui Zhu <[email protected]>
---
 include/linux/bpf_memcontrol.h |  97 ++++++++++++++++++++-
 include/linux/page_counter.h   |   8 ++
 mm/bpf_memcontrol.c            | 155 +++++++++++++++++++++++++--------
 mm/memcontrol.c                |  30 +++++++
 mm/page_counter.c              |  51 ++++++++++-
 5 files changed, 299 insertions(+), 42 deletions(-)
include/linux/bpf_memcontrol.h +94 -3
index 76ea5d1c3d32..23ee0d29bd24 100644
@@ -2,9 +2,9 @@
 /*
  * BPF policy hooks for the memory controller.
  *
- * A bpf_memcg_ops is attached to a cgroup.  A charge runs the policies of
- * that cgroup and of every ancestor, and the kernel combines what they
- * return.  BPF only picks between things the kernel already does.
+ * A bpf_memcg_ops is attached to a cgroup.  A hook runs the policies of that
+ * cgroup and of every ancestor, and the kernel combines what they return.
+ * BPF only picks between things the kernel already does.
  */
 #ifndef _LINUX_BPF_MEMCONTROL_H
 #define _LINUX_BPF_MEMCONTROL_H

@@ -30,6 +30,25 @@ enum bpf_memcg_high_request {
 
 #define BPF_MEMCG_HIGH_VALID_MASK	BPF_MEMCG_HIGH_DEFER_INLINE
 
+/*
+ * What a policy can ask for when reclaim computes memory.low protection.
+ * The kernel takes the requests of the cgroup and of every ancestor and acts
+ * on the largest, so one policy cannot take back what another asked for, nor
+ * what the user declared in memory.low.
+ */
+enum bpf_memcg_prot_request {
+	BPF_MEMCG_PROT_NO_REQUEST	= 0,
+	/*
+	 * Ask that this cgroup's current usage be taken as its declared
+	 * memory.low for this pass.  Arbitrated exactly like a declared
+	 * value: capped by the parent's effective protection and shared
+	 * with the siblings.
+	 */
+	BPF_MEMCG_PROT_REQUEST_USAGE	= 1U << 0,
+};
+
+#define BPF_MEMCG_PROT_VALID_MASK	BPF_MEMCG_PROT_REQUEST_USAGE
+
 /* Read-only snapshot.  Only values the caller already has. */
 struct bpf_memcg_ctx {
 	struct mem_cgroup	*memcg;			/* charged memcg */

@@ -41,6 +60,22 @@ struct bpf_memcg_ctx {
 	u32			gfp_flags;
 };
 
+/*
+ * Read-only snapshot taken while reclaim is about to compute memory.low
+ * protection.  @threshold is the effective memory.low left behind by the
+ * previous calculation, not what this pass will end up with - that also
+ * depends on the request.
+ */
+struct bpf_memcg_prot_ctx {
+	struct mem_cgroup	*memcg;			/* cgroup checked */
+	struct mem_cgroup	*target;		/* NULL for global reclaim */
+	struct task_struct	*task;			/* current */
+	u64			cgroup_id;
+	u64			target_cgroup_id;	/* 0 if none */
+	u64			threshold;		/* previous elow, pages */
+	u64			usage;			/* pages */
+};
+
 struct bpf_memcg_ops {
 	/**
 	 * high_policy - say where memory.high should be enforced

@@ -50,8 +85,31 @@ struct bpf_memcg_ops {
 	 * are dropped.
 	 */
 	u32 (*high_policy)(const struct bpf_memcg_ctx *ctx);
+
+	/**
+	 * low_request - ask for memory.low protection for this pass
+	 * @ctx: snapshot of the cgroup protection is about to be computed for
+	 *
+	 * Runs before the effective memory.low is calculated, for every cgroup
+	 * reclaim visits, whether or not it is already protected.  What a
+	 * policy asks for is arbitrated together with the memory.low declared
+	 * by the cgroup and by its siblings, so a request can move protection
+	 * around inside the parent's budget but can never take the subtree
+	 * above what its parent affords.
+	 *
+	 * Only memory.low is covered for now; a memory.min counterpart is
+	 * left for later.  Hard protection has no fallback once nothing else
+	 * is reclaimable, so a request recomputed every pass needs more
+	 * thought before it is safe.
+	 *
+	 * Return: bits from enum bpf_memcg_prot_request, or 0.  Other bits
+	 * are dropped.
+	 */
+	u32 (*low_request)(const struct bpf_memcg_prot_ctx *ctx);
 };
 
+#if IS_ENABLED(CONFIG_BPF_SYSCALL)
+
 /*
  * Run every high_policy on @memcg's cgroup and its ancestors, and return the
  * combined request for the caller to act on.

@@ -63,4 +121,37 @@ struct bpf_memcg_ops {
 u32 bpf_memcg_high_policy(struct mem_cgroup *memcg,
 			  struct mem_cgroup *over_limit, gfp_t gfp_mask);
 
+/*
+ * Run every low_request on @memcg's cgroup and its ancestors, and return the
+ * protection they ask for, in pages: @usage if any of them requested it, 0 if
+ * none did.  The caller feeds the result to page_counter_set_low_req() before
+ * the effective value is computed.
+ *
+ * @memcg:	the memcg protection is about to be computed for, never the root
+ * @target:	the reclaim root, or NULL for global reclaim
+ * @usage:	the current usage, in pages
+ */
+unsigned long bpf_memcg_low_request(struct mem_cgroup *memcg,
+				    struct mem_cgroup *target,
+				    unsigned long usage);
+
+#else /* CONFIG_BPF_SYSCALL */
+
+/* mm/bpf_memcontrol.c is only built with struct_ops support. */
+static inline u32 bpf_memcg_high_policy(struct mem_cgroup *memcg,
+					struct mem_cgroup *over_limit,
+					gfp_t gfp_mask)
+{
+	return BPF_MEMCG_HIGH_NO_OPINION;
+}
+
+static inline unsigned long bpf_memcg_low_request(struct mem_cgroup *memcg,
+						  struct mem_cgroup *target,
+						  unsigned long usage)
+{
+	return 0;
+}
+
+#endif /* CONFIG_BPF_SYSCALL */
+
 #endif /* _LINUX_BPF_MEMCONTROL_H */
include/linux/page_counter.h +8 -0
index 2baf7a2b29b2..4b03e4cafea9 100644
@@ -38,6 +38,12 @@ struct page_counter {
 	bool track_failcnt;
 	unsigned long min;
 	unsigned long low;
+	/*
+	 * Protection requested for this pass by an attached policy, in pages.
+	 * Arbitrated exactly like @low - see effective_protection() - so a
+	 * request can redistribute the parent's budget but never exceed it.
+	 */
+	unsigned long low_req;
 	unsigned long high;
 	unsigned long max;
 	struct page_counter *parent;

@@ -77,6 +83,8 @@ bool page_counter_try_charge(struct page_counter *counter,
 void page_counter_uncharge(struct page_counter *counter, unsigned long nr_pages);
 void page_counter_set_min(struct page_counter *counter, unsigned long nr_pages);
 void page_counter_set_low(struct page_counter *counter, unsigned long nr_pages);
+void page_counter_set_low_req(struct page_counter *counter,
+			      unsigned long nr_pages);
 
 static inline void page_counter_set_high(struct page_counter *counter,
 					 unsigned long nr_pages)
mm/bpf_memcontrol.c +119 -36

198 lines not on this page · raw message

mm/memcontrol.c +30 -0
index b937835d5cb7..93be8f84c0ed 100644
@@ -4490,6 +4490,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
 
 	page_counter_set_min(&memcg->memory, 0);
 	page_counter_set_low(&memcg->memory, 0);
+	page_counter_set_low_req(&memcg->memory, 0);
 
 	zswap_memcg_offline_cleanup(memcg);
 

@@ -4572,6 +4573,7 @@ static void mem_cgroup_css_reset(struct cgroup_subsys_state *css)
 #endif
 	page_counter_set_min(&memcg->memory, 0);
 	page_counter_set_low(&memcg->memory, 0);
+	page_counter_set_low_req(&memcg->memory, 0);
 	page_counter_set_high(&memcg->memory, PAGE_COUNTER_MAX);
 	page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX);
 	memcg_wb_domain_size_changed(memcg);

@@ -5333,6 +5335,7 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root,
 {
 	bool recursive_protection =
 		cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT;
+	struct mem_cgroup *target = root;
 
 	if (mem_cgroup_disabled())
 		return;

@@ -5340,6 +5343,19 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root,
 	if (!root)
 		root = root_mem_cgroup;
 
+	/*
+	 * Collect the request before elow is computed: it is an input to the
+	 * arbitration.  The reclaim root's own protection is ignored, so
+	 * there is nothing to ask for there.  @target keeps the caller's
+	 * NULL, which is how a policy recognises global reclaim.
+	 */
+	if (root != memcg) {
+		unsigned long usage = page_counter_read(&memcg->memory);
+		unsigned long req = bpf_memcg_low_request(memcg, target, usage);
+
+		page_counter_set_low_req(&memcg->memory, req);
+	}
+
 	page_counter_calculate_protection(&root->memory, &memcg->memory, recursive_protection);
 }
 

@@ -5358,6 +5374,8 @@ void mem_cgroup_calculate_protection_path(struct mem_cgroup *root,
 {
 	bool recursive_protection =
 		cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT;
+	struct mem_cgroup *target = root;
+	unsigned long usage, req;
 	struct cgroup *cg;
 	int root_level, i;
 

@@ -5370,6 +5388,18 @@ void mem_cgroup_calculate_protection_path(struct mem_cgroup *root,
 	if (memcg == root)
 		return;
 
+	/*
+	 * Ask for @memcg only: shrink_one() calls this per memcg, so asking
+	 * per level would multiply the cost by the depth of the tree.  The
+	 * ancestors keep the request the last top-down traversal left them
+	 * with; their elow - the cap @memcg is arbitrated against - is no
+	 * more stale than it already is for a value computed outside the
+	 * reclaim iteration.
+	 */
+	usage = page_counter_read(&memcg->memory);
+	req = bpf_memcg_low_request(memcg, target, usage);
+	page_counter_set_low_req(&memcg->memory, req);
+
 	root_level = root->css.cgroup->level;
 	cg = memcg->css.cgroup;
mm/page_counter.c +48 -3

87 lines not on this page · raw message

Signature · 1 line
2.43.0
author
#2 lore raw

RFC PATCH v2 2/2 selftests/bpf: Add bpf_memcg_ops protection test

In reply to Hui Zhu

From: Hui Zhu <[email protected]>

Attach a low_request policy to a grandchild cgroup, give the middle cgroup a memory.low as the budget, and reclaim from the top of the three-level tree. The policy asks for the victim's current usage on every pass; the test checks how the kernel arbitrates that ask. The outcome is read back through the next pass's ctx threshold (the effective value the previous pass computed) and the victim's memory.events low counter.

Three budgets, same request: with 0 the effective value stays 0; with 2M an 8M request comes back capped to 2M and reclaim still goes into the victim; with 16M the request is granted in full, reclaim skips the victim and its events counter moves. A request thus goes through the same arbitration as a declared memory.low and can never lift the subtree above its parent's budget. The ctx contents (victim id, reclaim target, usage) are checked along the way.

The workload is anonymous-only: without swap each memory.reclaim trigger returns EAGAIN, yet every scan still consults the hook, and it avoids temporary-file support issues on shared mounts. Recorded snapshots are filtered by cgroup id and reclaim target, since the dispatcher runs for every memcg and global reclaim recomputes protection with no target.

Signed-off-by Hui Zhu <[email protected]>
---
 .../bpf/prog_tests/memcg_ops_protection.c     | 297 ++++++++++++++++++
 .../bpf/progs/memcg_ops_protection.c          |  82 +++++
 2 files changed, 379 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_ops_protection.c
 create mode 100644 tools/testing/selftests/bpf/progs/memcg_ops_protection.c
tools/testing/selftests/bpf/prog_tests/memcg_ops_protection.c new file +297 -0
new file mode 100644
index 000000000000..b60c573fc480
@@ -0,0 +1,297 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Drive the memcg_ops_protection sample: check that the kernel consults
+ * low_request during reclaim, that the ctx describes the check faithfully,
+ * and - the point of the test - that what the policy asks for is arbitrated
+ * like a declared memory.low rather than granted outright.
+ *
+ * Layout, under the test workdir:
+ *
+ *   memcg_ops_prot                P   reclaim runs from P/memory.reclaim
+ *   memcg_ops_prot/budget         M   memory.low is the budget C is capped to
+ *   memcg_ops_prot/budget/victim  C   holds the memory, carries the ops
+ *   memcg_ops_prot_ctl            -   where the test process normally sits
+ *
+ * The victim asks for its current usage (8M) on every pass.  The runner
+ * varies M's memory.low - the budget the request is arbitrated against -
+ * and reads the outcome back two ways: @threshold in the ctx of the next
+ * pass carries the effective value the last pass computed, and C's
+ * memory.events "low" counts the passes that skipped C as protected.
+ *
+ *   budget   request   elow(C)         C's memory.events low
+ *   0        off       0               unchanged (nothing asked, nothing got)
+ *   2M       on        2M, capped      unchanged (asked 8M, parent affords 2M)
+ *   16M      on        usage, granted  increased (parent affords all of it)
+ *
+ * The middle row is the ceiling: the request can move protection around
+ * inside the parent's budget but cannot lift the subtree above it.  Break
+ * the cap in effective_protection() and that row turns red - elow(C)
+ * becomes usage, reclaim skips C, and the event counter moves.
+ *
+ * The test process joins C itself and faults in 8M of anonymous memory.
+ * Nothing is actually reclaimed - without swap there is nothing to take -
+ * so each trigger walks the subtree repeatedly and returns EAGAIN, and
+ * every walk consulted the hook and recomputed the effective value.  No
+ * file is involved on purpose: a VM's working directory is often a shared
+ * mount where temporary files are unsupported or unevictable.  On the way
+ * out the process rejoins the ctl cgroup, so the framework's cleanup can
+ * rmdir C, M and P.
+ */
+
+#include <errno.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+#include <bpf/bpf.h>
+#include <bpf/libbpf.h>
+
+#include <test_progs.h>
+#include "cgroup_helpers.h"
+#include "memcg_ops_protection.skel.h"
+
+#define CTL_PATH	"/memcg_ops_prot_ctl"
+#define PARENT_PATH	"/memcg_ops_prot"
+#define BUDGET_PATH	PARENT_PATH "/budget"
+#define VICTIM_PATH	BUDGET_PATH "/victim"
+
+#define FILL_SZ		(8UL << 20)
+#define RECLAIM_SZ	"1M"
+/* Below FILL_SZ, so the request overclaims and the cap has to bite. */
+#define BUDGET_CAPPED	(2UL << 20)
+/* Above FILL_SZ, so the parent affords the request in full. */
+#define BUDGET_GRANTED	(16UL << 20)
+
+static int set_budget(unsigned long sz)
+{
+	char buf[32];
+
+	snprintf(buf, sizeof(buf), "%lu", sz);
+	return write_cgroup_file(BUDGET_PATH, "memory.low", buf);
+}
+
+/* .bss is frozen once the skeleton is loaded, so the phase switch that the
+ * policy reads lives in a map rather than in a global.
+ */
+static int set_request(struct memcg_ops_protection *skel, int on)
+{
+	__u32 key = 0, val = !!on;
+
+	return bpf_map_update_elem(bpf_map__fd(skel->maps.cfg), &key, &val, 0);
+}
+
+static long events_low(const char *path)
+{
+	char buf[512];
+	char *p;
+
+	if (read_cgroup_file(path, "memory.events", buf, sizeof(buf)))
+		return -1;
+	/* "low" is the first key in memory.events. */
+	p = strstr(buf, "low ");
+	if (!p)
+		return -1;
+	return strtol(p + 4, NULL, 10);
+}
+
+/* One triggered reclaim walk.  EAGAIN is as good a scan as success. */
+static int trigger_reclaim(int pfd)
+{
+	ssize_t n;
+	int rfd, err;
+
+	rfd = openat(pfd, "memory.reclaim", O_RDWR);
+	if (rfd < 0) {
+		log_err("open memory.reclaim");
+		return -1;
+	}
+	n = write(rfd, RECLAIM_SZ, sizeof(RECLAIM_SZ) - 1);
+	err = n < 0 ? -errno : 0;
+	close(rfd);
+	if (err && err != -EAGAIN) {
+		log_err("write memory.reclaim");
+		return -1;
+	}
+	return 0;
+}
+
+void test_memcg_ops_protection(void)
+{
+	struct memcg_ops_protection *skel = NULL;
+	struct bpf_link *ops_link = NULL;
+	int ctlfd = -1, pfd = -1, bfd = -1, cfd = -1;
+	__u64 seq_before, threshold, usage;
+	long budget_pages, low_before, low_after;
+	void *mem = MAP_FAILED;
+
+	/* Sets the environment up and parks this process outside of P. */
+	ctlfd = test__join_cgroup(CTL_PATH);
+	if (!ASSERT_GE(ctlfd, 0, "join_cgroup"))
+		return;
+
+	pfd = create_and_get_cgroup(PARENT_PATH);
+	if (!ASSERT_GE(pfd, 0, "create_parent"))
+		goto done;
+
+	bfd = create_and_get_cgroup(BUDGET_PATH);
+	if (!ASSERT_GE(bfd, 0, "create_budget"))
+		goto done;
+
+	cfd = create_and_get_cgroup(VICTIM_PATH);
+	if (!ASSERT_GE(cfd, 0, "create_victim"))
+		goto done;
+
+	/*
+	 * The memory.* files of M and C need the memory controller in the
+	 * subtree_control of P and of M.  A failure here is the usual "v1
+	 * still owns memory", which is a skip rather than a fail.
+	 */
+	if (enable_controllers(PARENT_PATH, "memory") ||
+	    enable_controllers(BUDGET_PATH, "memory")) {
+		fprintf(stderr, "no memory controller under %s\n", PARENT_PATH);
+		test__skip();
+		goto done;
+	}
+
+	skel = memcg_ops_protection__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		goto done;
+
+	skel->bss->target_cgroup_id = get_cgroup_id(VICTIM_PATH);
+	skel->bss->parent_cgroup_id = get_cgroup_id(PARENT_PATH);
+	if (!ASSERT_NEQ(skel->bss->target_cgroup_id, 0ULL, "victim_id"))
+		goto done;
+	if (!ASSERT_NEQ(skel->bss->parent_cgroup_id, 0ULL, "parent_id"))
+		goto done;
+
+	if (!ASSERT_OK(memcg_ops_protection__load(skel), "load"))
+		goto done;
+
+	ops_link = bpf_map__attach_cgroup_opts(skel->maps.prot_ops, cfd, NULL);
+	if (!ASSERT_OK_PTR(ops_link, "attach_ops"))
+		goto done;
+
+	/* From here on the test process lives in C; leave via done. */
+	if (!ASSERT_OK(join_cgroup(VICTIM_PATH), "join_victim"))
+		goto done;
+
+	mem = mmap(NULL, FILL_SZ, PROT_READ | PROT_WRITE,
+		   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (!ASSERT_TRUE(mem != MAP_FAILED, "mmap"))
+		goto done;
+	memset(mem, 0xa5, FILL_SZ);
+
+	budget_pages = BUDGET_CAPPED / sysconf(_SC_PAGESIZE);
+
+	/*
+	 * Phase 0 - nothing declared, nothing requested.  The hook still
+	 * runs: reclaim asks every cgroup it visits, protected or not.
+	 */
+	if (!ASSERT_OK(set_budget(0), "budget_zero"))
+		goto done;
+	if (!ASSERT_OK(set_request(skel, 0), "request_off"))
+		goto done;
+	seq_before = skel->bss->record_seq;
+	low_before = events_low(VICTIM_PATH);
+	if (!ASSERT_GE(low_before, 0, "events_phase0"))
+		goto done;
+	if (!ASSERT_OK(trigger_reclaim(pfd), "trigger_phase0"))
+		goto done;
+
+	if (!ASSERT_GT(skel->bss->record_seq, seq_before, "passes_phase0"))
+		goto done;
+	if (!ASSERT_EQ(skel->bss->saw_from_parent, 1, "target_phase0"))
+		goto done;
+	/* No claim anywhere below P, so C ends up with no protection. */
+	if (!ASSERT_EQ(skel->bss->seen_threshold, 0ULL, "idle_threshold"))
+		goto done;
+	low_after = events_low(VICTIM_PATH);
+	if (!ASSERT_EQ(low_after, low_before, "idle_events"))
+		goto done;
+
+	/*
+	 * Phase 1 - the ceiling.  C asks for its full usage but M only
+	 * affords BUDGET_CAPPED, so the arbitration has to cut the request
+	 * down: elow(C) lands at the budget, below usage, and reclaim does
+	 * not treat C as protected.
+	 */
+	if (!ASSERT_OK(set_budget(BUDGET_CAPPED), "budget_capped"))
+		goto done;
+	if (!ASSERT_OK(set_request(skel, 1), "request_on"))
+		goto done;
+	seq_before = skel->bss->record_seq;
+	low_before = events_low(VICTIM_PATH);
+	if (!ASSERT_GE(low_before, 0, "events_phase1"))
+		goto done;
+	if (!ASSERT_OK(trigger_reclaim(pfd), "trigger_phase1"))
+		goto done;
+
+	if (!ASSERT_GT(skel->bss->record_seq, seq_before, "passes_phase1"))
+		goto done;
+	threshold = skel->bss->seen_threshold;
+	usage = skel->bss->seen_usage;
+	if (!ASSERT_GT(usage, 0ULL, "capped_usage"))
+		goto done;
+	/*
+	 * The request was honoured - it is what lifts elow off zero - and
+	 * capped - it never exceeds what M affords.  Exact equality with
+	 * the budget is the normal outcome; the cap is the invariant.
+	 */
+	if (!ASSERT_GT(threshold, 0ULL, "capped_threshold_nonzero"))
+		goto done;
+	if (!ASSERT_LE(threshold, (__u64)budget_pages, "capped_threshold"))
+		goto done;
+	if (!ASSERT_LT(threshold, usage, "capped_below_usage"))
+		goto done;
+	low_after = events_low(VICTIM_PATH);
+	if (!ASSERT_EQ(low_after, low_before, "capped_events"))
+		goto done;
+
+	/*
+	 * Phase 2 - the same request, but now M affords all of it: elow(C)
+	 * reaches usage, reclaim skips C as protected, and the skip shows
+	 * up in C's memory.events.  Together with phase 1 this pins the
+	 * request to the arbitration: same ask, different budget,
+	 * different outcome.
+	 */
+	if (!ASSERT_OK(set_budget(BUDGET_GRANTED), "budget_granted"))
+		goto done;
+	seq_before = skel->bss->record_seq;
+	low_before = events_low(VICTIM_PATH);
+	if (!ASSERT_GE(low_before, 0, "events_phase2"))
+		goto done;
+	if (!ASSERT_OK(trigger_reclaim(pfd), "trigger_phase2"))
+		goto done;
+
+	if (!ASSERT_GT(skel->bss->record_seq, seq_before, "passes_phase2"))
+		goto done;
+	threshold = skel->bss->seen_threshold;
+	usage = skel->bss->seen_usage;
+	/* The budget covers the ask, so the request is granted in full. */
+	if (!ASSERT_EQ(threshold, usage, "granted_threshold"))
+		goto done;
+	if (!ASSERT_GT(usage, 0ULL, "granted_usage"))
+		goto done;
+	low_after = events_low(VICTIM_PATH);
+	ASSERT_GT(low_after, low_before, "granted_events");
+
+done:
+	/* Detach before anything else: no policy may run during teardown. */
+	bpf_link__destroy(ops_link);
+	memcg_ops_protection__destroy(skel);
+	if (mem != MAP_FAILED)
+		munmap(mem, FILL_SZ);
+	/* A no-op when the process never made it into the victim. */
+	join_cgroup(CTL_PATH);
+	if (cfd >= 0)
+		close(cfd);
+	if (bfd >= 0)
+		close(bfd);
+	if (pfd >= 0)
+		close(pfd);
+	if (ctlfd >= 0)
+		close(ctlfd);
+}
tools/testing/selftests/bpf/progs/memcg_ops_protection.c new file +82 -0

82 lines not on this page · raw message

Signature · 1 line
2.43.0