samples/damon: handle damon_{start,stop}() failures

All DAMON sample modules are not correctly handling failures from damon_start(). Among those, mtier also has an additional problem for handling of damon_stop() failures. wsse and prcl also have a problem in their damon_call() failure handling. As a result, memory leaks, next DAMON operation disruptions, and use-after-free can happen. Fix those.

Note that only the damon_start() failure caused issues can reliably be reproduced. Reproducing those issues require the admin permission, though.

Changes from RFC v4
- RFC v4: https://lore.kernel.org/[email protected]/
- Collect R-b: from Zenghui Yu.
- Rebase to latest mm-new.
- Drop RFC.
Changes from RFC v3
- RFC v3: https://lore.kernel.org/[email protected]
- Add damon_Call() failure handling fixes for wsse and prcl. Changes from RFC v2
- RFC v2: https://lore.kernel.org/[email protected]
- Add damon_start() failure handling fixes for wsse and prcl. Changes from RFC v1
- RFC v1: https://lore.kernel.org/[email protected]
- Add damon_stop() failure handling fix to the series.

SJ Park (6):
samples/damon/wsse: handle damon_start() failure samples/damon/prcl: handle damon_start() failure samples/damon/mtier: handle damon_start() failure samples/damon/mtier: handle damon_stop() failure samples/damon/wsse: stop and free damon ctx when damon_call() fails samples/damon/prcl: stop and free damon ctx when damon_call() fails

 samples/damon/mtier.c | 14 ++++++++++++--
 samples/damon/prcl.c  | 11 +++++++++--
 samples/damon/wsse.c  | 11 +++++++++--
 3 files changed, 30 insertions(+), 6 deletions(-)

base-commit: 77fe35dfe005f7d55c8e729e1543b87cfc805a21

2.47.3

damon_sample_wsse_start() callers assume it will clean up resources when it fails. And the function does the cleanup for context buildup failures. However, it is not doing the cleanup for damon_start() failure. As a result, when damon_start() fails, it leaks the memory for DAMON context. Free the context in case of the failure to fix the issues.

Note that the issue can reliably be reproduced because the module calls damon_start() in the exclusive mode. For example,

    $ sudo damo start
    $ echo $$ | sudo tee /sys/module/damon_sample_wsse/parameters/target_pid
    $ echo Y | sudo tee /sys/module/damon_sample_wsse/parameters/enabled
    $ sudo cat /proc/allocinfo | grep damon_new_ctx

Because the first command is running another DAMON instance, the third command fails the damon_start() call because the new DAMON instance cannot exclusively run. And without this fix, by repeating the third and the fourth commands above, we can show the memory consumption is only increasing due to the leaks. It requires the sudo permission though.

The issue was discovered [1] by Sashiko.

[1] https://lore.kernel.org/[email protected]

---
 samples/damon/wsse.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

index 799ad44439434..bbd9392ab5b36 100644
@@ -87,8 +87,10 @@ static int damon_sample_wsse_start(void)
 	target->pid = target_pidp;
 
 	err = damon_start(&ctx, 1, true);
-	if (err)
+	if (err) {
+		damon_destroy_ctx(ctx);
 		return err;
+	}
 	repeat_call_control.data = ctx;
 	return damon_call(ctx, &repeat_call_control);
 }

2.47.3

damon_sample_prcl_start() callers assume it will clean up resources when it fails. And the function does the cleanup for context buildup failures. However, it is not doing the cleanup for damon_start() failure. As a result, when damon_start() fails, it leaks the memory for DAMON context. Free the context in case of the failure to fix the issues.

Note that the issue can reliably be reproduced because the module calls damon_start() in the exclusive mode. For example,

    $ sudo damo start
    $ echo $$ | sudo tee /sys/module/damon_sample_prcl/parameters/target_pid
    $ echo Y | sudo tee /sys/module/damon_sample_prcl/parameters/enabled
    $ sudo cat /proc/allocinfo | grep damon_new_ctx

Because the first command is running another DAMON instance, the third command fails the damon_start() call because the new DAMON instance cannot exclusively run. And without this fix, by repeating the third and the fourth commands above, we can show the memory consumption is only increasing due to the leaks. It requires the sudo permission though.

The issue was discovered [1] by Sashiko.

[1] https://lore.kernel.org/[email protected]

---
 samples/damon/prcl.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

index b7c50f2656ce7..0db2598946911 100644
@@ -106,8 +106,10 @@ static int damon_sample_prcl_start(void)
 	damon_set_schemes(ctx, &scheme, 1);
 
 	err = damon_start(&ctx, 1, true);
-	if (err)
+	if (err) {
+		damon_destroy_ctx(ctx);
 		return err;
+	}
 
 	repeat_call_control.data = ctx;
 	return damon_call(ctx, &repeat_call_control);

2.47.3

damon_sample_mtier_start() callers assume it will clean up resources when it fails. And the function does the cleanup for context buildup failures. However, it is not doing the cleanup for damon_start() failure.

As a result, when damon_start() fails, it could leak the memory for DAMON context. Also, if damon_start() fails for only the second context, the first context will indefinitely run, and avoid starting other DAMON contexts since it is running in the exclusive mode. Stop possibly started DAMON context and free the contexts in case of the failure to fix the issues.

Note that the issue can reliably be reproduced because the module calls damon_start() in the exclusive mode. For example,

    $ sudo damo start
    $ echo Y | sudo tee /sys/module/damon_sample_mtier/parameters/enabled
    $ sudo cat /proc/allocinfo | grep damon_new_ctx

Because the first command is running another DAMON instance, the second command fails the damon_start() call because the new DAMON instance cannot exclusively run. And without this fix, by repeating the second and the third commands above, we can show the memory consumption is only increasing due to the leaks. It requires the sudo permission though.

The issue was discovered [1] by Sashiko.

[1] https://lore.kernel.org/[email protected]

---
 samples/damon/mtier.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

index eb1143de8df17..66b591f2180fa 100644
@@ -174,6 +174,7 @@ static struct damon_ctx *damon_sample_mtier_build_ctx(bool promote)
 static int damon_sample_mtier_start(void)
 {
 	struct damon_ctx *ctx;
+	int err;
 
 	ctx = damon_sample_mtier_build_ctx(true);
 	if (!ctx)

@@ -185,7 +186,15 @@ static int damon_sample_mtier_start(void)
 		return -ENOMEM;
 	}
 	ctxs[1] = ctx;
-	return damon_start(ctxs, 2, true);
+	err = damon_start(ctxs, 2, true);
+	if (!err)
+		return 0;
+
+	if (damon_is_running(ctxs[0]))
+		damon_stop(ctxs, 1);
+	damon_destroy_ctx(ctxs[0]);
+	damon_destroy_ctx(ctxs[1]);
+	return err;
 }
 
 static void damon_sample_mtier_stop(void)

2.47.3

damon_sample_mtier_stop() assumes its damon_stop() call will always successfully stops the two DAMON contexts. Hence it deallocates the two DAMON contexts after the damon_stop() call. However, if a given context is already stopped, damon_stop() fails and returns an error while letting the DAMON contexts that have not yet stopped keep running. This kind of unexpected early DAMON context stops could happen due to memory allocation failures in kdamond_fn(). Because damon_sample_mtier_stop() just deallocates all DAMON contexts with damon_target and damon_region objects that are linked to the contexts, the execution of the unstopped DAMON context (kdamond) ends up using the memory that freed (use-after-free). Fix the issue by separating the damon_stop() to be invoked per context.

Note that DAMON_SYSFS also allows multiple DAMON contexts execution. But, it calls damon_stop() for each context one by one. Hence this issue is only in mtier.

For the long term, it would be better to refactor damon_stop() to always ensure stopping all contexts regardless of the failures in the middle. Make this fix in the current way, though, to keep it simple and easy to backport. I will do the refactoring later.

The issue was discovered [1] by Sashiko.

[1] https://lore.kernel.org/[email protected]

---
 samples/damon/mtier.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index 66b591f2180fa..faaaaa12e6206 100644
@@ -199,7 +199,8 @@ static int damon_sample_mtier_start(void)
 
 static void damon_sample_mtier_stop(void)
 {
-	damon_stop(ctxs, 2);
+	damon_stop(ctxs, 1);
+	damon_stop(&ctxs[1], 1);
 	damon_destroy_ctx(ctxs[0]);
 	damon_destroy_ctx(ctxs[1]);
 }

2.47.3

Wrong. mtier is static module. Cannot be loaded on runtime. So I'm ignoring this.

Thanks,
SJ

[...]

damon_sample_wsse_start() calls damon_call() right after damon_start() is succeeded. The kdamond that has started by the damon_start() could be terminated by itself before or in the middle of the damon_call() execution. There could be multiple reasons for such a stop including monitoring target process termination and kdamond_fn() internal memory allocation failures. In the case, damon_call() will fail and return an error without cleaning up the DAMON context object. The damon_sample_wsse_start() caller assumes it would clean up the object, though. When the user requests to start DAMON again, damon_sample_wsse_start() is called again, allocates a new DAMON context object and overwrites the pointer for the previous object. As a result, the previous context object is leaked.

Safely stop the kdamond and deallocate the context object when the failure is returned. Note that the kdamond should be stopped first, because damon_call() failure means not complete termination of the kdamond but only the fact that the termination process has started.

The user impact shouldn't be that significant because the race is not easy to happen, and only up to one DAMON context object can be leaked per race.

The issue was discovered [1] by Sashiko.

[1] https://lore.kernel.org/[email protected]

---
 samples/damon/wsse.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

index bbd9392ab5b36..ff5e8a890f448 100644
@@ -92,7 +92,12 @@ static int damon_sample_wsse_start(void)
 		return err;
 	}
 	repeat_call_control.data = ctx;
-	return damon_call(ctx, &repeat_call_control);
+	err = damon_call(ctx, &repeat_call_control);
+	if (err) {
+		damon_stop(&ctx, 1);
+		damon_destroy_ctx(ctx);
+	}
+	return err;
 }
 
 static void damon_sample_wsse_stop(void)

2.47.3

damon_sample_prcl_start() calls damon_call() right after damon_start() is succeeded. The kdamond that has started by the damon_start() could be terminated by itself before or in the middle of the damon_call() execution. There could be multiple reasons for such a stop including monitoring target process termination and kdamond_fn() internal memory allocation failures. In the case, damon_call() will fail and return an error without cleaning up the DAMON context object. The damon_sample_prcl_start() caller assumes it would clean up the object, though. When the user requests to start DAMON again, damon_sample_prcl_start() is called again, allocates a new DAMON context object and overwrites the pointer for the previous object. As a result, the previous context object is leaked.

Safely stop the kdamond and deallocate the context object when the failure is returned. Note that the kdamond should be stopped first, because damon_call() failure means not complete termination of the kdamond but only the fact that the termination process has started.

The user impact shouldn't be that significant because the race is not easy to happen, and only up to one DAMON context object can be leaked per race.

The issue was discovered [1] by Sashiko.

[1] https://lore.kernel.org/[email protected]

---
 samples/damon/prcl.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

index 0db2598946911..edeae145c4a8a 100644
@@ -112,7 +112,12 @@ static int damon_sample_prcl_start(void)
 	}
 
 	repeat_call_control.data = ctx;
-	return damon_call(ctx, &repeat_call_control);
+	err = damon_call(ctx, &repeat_call_control);
+	if (err) {
+		damon_stop(&ctx, 1);
+		damon_destroy_ctx(ctx);
+	}
+	return err;
 }
 
 static void damon_sample_prcl_stop(void)

2.47.3

So it doesn't seem that we need to fast-track all this into 7.2-rcX?

Yes, I think it is fine as long as they can be merged into necessary stable kernels in reasonable time. And 7.3 also sounds reasonable to me.

Thanks,
SJ