Skip to contentSkip to conversation

selftests/bpf: fix potential premature unload in bpf_testmod

PATCH Superseded Artem Savkov 4 participants 7 messages Jan 9, 2024 – Jan 16, 2024

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 selftests/bpf: fix potential premature unload in bpf_testmod

It is possible for bpf_kfunc_call_test_release() to be called from bpf_map_free_deferred() when bpf_testmod is already unloaded and perf_test_stuct.cnt which it tries to decrease is no longer in memory. This patch tries to fix the issue by waiting for all references to be dropped in bpf_testmod_exit().

The issue can be triggered by running 'test_progs -t map_kptr' in 6.5, but is obscured in 6.6 by d119357d07435 ("rcu-tasks: Treat only synchronous grace periods urgently").

Fixes 65eb006d85a2a ("bpf: Move kernel test kfuncs to bpf_testmod")
---
 tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 4 ++++
 1 file changed, 4 insertions(+)
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +4 -0
index 91907b321f913..63f0dbd016703 100644
@@ -2,6 +2,7 @@
 /* Copyright (c) 2020 Facebook */
 #include <linux/btf.h>
 #include <linux/btf_ids.h>
+#include <linux/delay.h>
 #include <linux/error-injection.h>
 #include <linux/init.h>
 #include <linux/module.h>
@@ -544,6 +545,9 @@ static int bpf_testmod_init(void)
 
 static void bpf_testmod_exit(void)
 {
+	while (refcount_read(&prog_test_struct.cnt) > 1)
+		msleep(20);
+
 	return sysfs_remove_bin_file(kernel_kobj, &bin_attr_bpf_testmod_file);
 }
 
Signature · 1 line
2.43.0
reviewer Acked-by on v1 1/1
#1 lore raw
Quoted 11 lines · Artem Savkov

It is possible for bpf_kfunc_call_test_release() to be called from bpf_map_free_deferred() when bpf_testmod is already unloaded and perf_test_stuct.cnt which it tries to decrease is no longer in memory. This patch tries to fix the issue by waiting for all references to be dropped in bpf_testmod_exit().

The issue can be triggered by running 'test_progs -t map_kptr' in 6.5, but is obscured in 6.6 by d119357d07435 ("rcu-tasks: Treat only synchronous grace periods urgently").

Fixes 65eb006d85a2a ("bpf: Move kernel test kfuncs to bpf_testmod")

Please add your Signed-off-by tag.

I think the root cause is that bpf_kfunc_call_test_acquire() kfunc is defined in bpf_testmod and the kfunc returns some data in bpf_testmod. But the release function bpf_kfunc_call_test_release() is in the kernel. The release func tries to access some data in bpf_testmod which might have been unloaded. The prog_test_ref_kfunc is defined in the kernel, so no bpf_testmod btf reference is hold so bpf_testmod can be unloaded before bpf_kfunc_call_test_release().
As you mentioned, we won't have this issue if bpf_kfunc_call_test_acquire() is also in the kernel.

I think putting bpf_kfunc_call_test_acquire() in bpf_testmod and bpf_kfunc_call_test_release() in kernel is not a good idea and confusing. But since this is only for tests, I guess we can live with that. With that,

Quoted 26 lines
---
  tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 4 ++++
  1 file changed, 4 insertions(+)
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +4 -0
index 91907b321f913..63f0dbd016703 100644
@@ -2,6 +2,7 @@
  /* Copyright (c) 2020 Facebook */
  #include <linux/btf.h>
  #include <linux/btf_ids.h>
+#include <linux/delay.h>
  #include <linux/error-injection.h>
  #include <linux/init.h>
  #include <linux/module.h>
@@ -544,6 +545,9 @@ static int bpf_testmod_init(void)
 
  static void bpf_testmod_exit(void)
  {
+	while (refcount_read(&prog_test_struct.cnt) > 1)
+		msleep(20);
+
  	return sysfs_remove_bin_file(kernel_kobj, &bin_attr_bpf_testmod_file);
  }
 
#2 lore raw
Quoted 8 lines · Yonghong Song
Quoted 6 lines · Artem Savkov

It is possible for bpf_kfunc_call_test_release() to be called from bpf_map_free_deferred() when bpf_testmod is already unloaded and perf_test_stuct.cnt which it tries to decrease is no longer in memory. This patch tries to fix the issue by waiting for all references to be dropped in bpf_testmod_exit().

Quoted 7 lines
Quoted 5 lines

The issue can be triggered by running 'test_progs -t map_kptr' in 6.5, but is obscured in 6.6 by d119357d07435 ("rcu-tasks: Treat only synchronous grace periods urgently").

Fixes 65eb006d85a2a ("bpf: Move kernel test kfuncs to bpf_testmod")

Please add your Signed-off-by tag.

Thanks for noticing. Will resend with signed-off-by and your ack.

Quoted 10 lines

I think the root cause is that bpf_kfunc_call_test_acquire() kfunc is defined in bpf_testmod and the kfunc returns some data in bpf_testmod. But the release function bpf_kfunc_call_test_release() is in the kernel. The release func tries to access some data in bpf_testmod which might have been unloaded. The prog_test_ref_kfunc is defined in the kernel, so no bpf_testmod btf reference is hold so bpf_testmod can be unloaded before bpf_kfunc_call_test_release().
As you mentioned, we won't have this issue if bpf_kfunc_call_test_acquire() is also in the kernel.

Quoted 3 lines

I think putting bpf_kfunc_call_test_acquire() in bpf_testmod and bpf_kfunc_call_test_release() in kernel is not a good idea and confusing. But since this is only for tests, I guess we can live with that. With that,

Correct. 65eb006d85a2a ("bpf: Move kernel test kfuncs to bpf_testmod") also mentions why bpf_kfunc_call_test_release() is not in the module and states that this is temporary. I'll add a comment in v2 so the wait can be removed once the functions are re-united.

Quoted 27 lines
Quoted 24 lines
---
  tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 4 ++++
  1 file changed, 4 insertions(+)
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +4 -0
index 91907b321f913..63f0dbd016703 100644
@@ -2,6 +2,7 @@
  /* Copyright (c) 2020 Facebook */
  #include <linux/btf.h>
  #include <linux/btf_ids.h>
+#include <linux/delay.h>
  #include <linux/error-injection.h>
  #include <linux/init.h>
  #include <linux/module.h>
@@ -544,6 +545,9 @@ static int bpf_testmod_init(void)
  static void bpf_testmod_exit(void)
  {
+	while (refcount_read(&prog_test_struct.cnt) > 1)
+		msleep(20);
+
  	return sysfs_remove_bin_file(kernel_kobj, &bin_attr_bpf_testmod_file);
  }
Signature · 2 lines
Regards,
  Artem
#3 lore raw
Quoted 29 lines · Artem Savkov
Quoted 8 lines · Yonghong Song
Quoted 6 lines · Artem Savkov

It is possible for bpf_kfunc_call_test_release() to be called from bpf_map_free_deferred() when bpf_testmod is already unloaded and perf_test_stuct.cnt which it tries to decrease is no longer in memory. This patch tries to fix the issue by waiting for all references to be dropped in bpf_testmod_exit().

Quoted 7 lines
Quoted 5 lines

The issue can be triggered by running 'test_progs -t map_kptr' in 6.5, but is obscured in 6.6 by d119357d07435 ("rcu-tasks: Treat only synchronous grace periods urgently").

Fixes 65eb006d85a2a ("bpf: Move kernel test kfuncs to bpf_testmod")

Please add your Signed-off-by tag.

Thanks for noticing. Will resend with signed-off-by and your ack.

Quoted 10 lines

I think the root cause is that bpf_kfunc_call_test_acquire() kfunc is defined in bpf_testmod and the kfunc returns some data in bpf_testmod. But the release function bpf_kfunc_call_test_release() is in the kernel. The release func tries to access some data in bpf_testmod which might have been unloaded. The prog_test_ref_kfunc is defined in the kernel, so no bpf_testmod btf reference is hold so bpf_testmod can be unloaded before bpf_kfunc_call_test_release().
As you mentioned, we won't have this issue if bpf_kfunc_call_test_acquire() is also in the kernel.

Quoted 8 lines
Quoted 3 lines

I think putting bpf_kfunc_call_test_acquire() in bpf_testmod and bpf_kfunc_call_test_release() in kernel is not a good idea and confusing. But since this is only for tests, I guess we can live with that. With that,

Correct. 65eb006d85a2a ("bpf: Move kernel test kfuncs to bpf_testmod") also mentions why bpf_kfunc_call_test_release() is not in the module and states that this is temporary. I'll add a comment in v2 so the wait can be removed once the functions are re-united.

I somehow recall it has to do with the fact you can't have trusted pointer on module's object, so that's why those structs had to stay in kernel.. but I might be wrong

jirka

Quoted 33 lines
Quoted 28 lines
Quoted 24 lines
---
  tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 4 ++++
  1 file changed, 4 insertions(+)
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +4 -0
index 91907b321f913..63f0dbd016703 100644
@@ -2,6 +2,7 @@
  /* Copyright (c) 2020 Facebook */
  #include <linux/btf.h>
  #include <linux/btf_ids.h>
+#include <linux/delay.h>
  #include <linux/error-injection.h>
  #include <linux/init.h>
  #include <linux/module.h>
@@ -544,6 +545,9 @@ static int bpf_testmod_init(void)
  static void bpf_testmod_exit(void)
  {
+	while (refcount_read(&prog_test_struct.cnt) > 1)
+		msleep(20);
+
  	return sysfs_remove_bin_file(kernel_kobj, &bin_attr_bpf_testmod_file);
  }
Signature · 2 lines
Regards,
  Artem
#4 lore raw

PATCH v2 1/1 selftests/bpf: fix potential premature unload in bpf_testmod

In reply to Yonghong Song

It is possible for bpf_kfunc_call_test_release() to be called from bpf_map_free_deferred() when bpf_testmod is already unloaded and perf_test_stuct.cnt which it tries to decrease is no longer in memory. This patch tries to fix the issue by waiting for all references to be dropped in bpf_testmod_exit().

The issue can be triggered by running 'test_progs -t map_kptr' in 6.5, but is obscured in 6.6 by d119357d07435 ("rcu-tasks: Treat only synchronous grace periods urgently").

Fixes 65eb006d85a2a ("bpf: Move kernel test kfuncs to bpf_testmod")
---
 tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 9 +++++++++
 1 file changed, 9 insertions(+)
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +9 -0
index 91907b321f913..e7c9e1c7fde04 100644
@@ -2,6 +2,7 @@
 /* Copyright (c) 2020 Facebook */
 #include <linux/btf.h>
 #include <linux/btf_ids.h>
+#include <linux/delay.h>
 #include <linux/error-injection.h>
 #include <linux/init.h>
 #include <linux/module.h>
@@ -544,6 +545,14 @@ static int bpf_testmod_init(void)
 
 static void bpf_testmod_exit(void)
 {
+        /* Need to wait for all references to be dropped because
+         * bpf_kfunc_call_test_release() which currently resides in kernel can
+         * be called after bpf_testmod is unloaded. Once release function is
+         * moved into the module this wait can be removed.
+         */
+	while (refcount_read(&prog_test_struct.cnt) > 1)
+		msleep(20);
+
 	return sysfs_remove_bin_file(kernel_kobj, &bin_attr_bpf_testmod_file);
 }
 
Signature · 1 line
2.43.0
#5 lore raw
This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Hello:

This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Quoted 11 lines · On Wed, 10 Jan 2024 09:57:37 +0100 you wrote:

It is possible for bpf_kfunc_call_test_release() to be called from bpf_map_free_deferred() when bpf_testmod is already unloaded and perf_test_stuct.cnt which it tries to decrease is no longer in memory. This patch tries to fix the issue by waiting for all references to be dropped in bpf_testmod_exit().

The issue can be triggered by running 'test_progs -t map_kptr' in 6.5, but is obscured in 6.6 by d119357d07435 ("rcu-tasks: Treat only synchronous grace periods urgently").

[...]

Here is the summary with links:
- [bpf-next,v2] selftests/bpf: fix potential premature unload in bpf_testmod

    https://git.kernel.org/bpf/bpf-next/c/6ad61815babf

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
Acked-by on v2 1/1
#6 lore raw
In reply to Artem Savkov
Quoted 6 lines · Artem Savkov

It is possible for bpf_kfunc_call_test_release() to be called from bpf_map_free_deferred() when bpf_testmod is already unloaded and perf_test_stuct.cnt which it tries to decrease is no longer in memory. This patch tries to fix the issue by waiting for all references to be dropped in bpf_testmod_exit().

Quoted 7 lines

The issue can be triggered by running 'test_progs -t map_kptr' in 6.5, but is obscured in 6.6 by d119357d07435 ("rcu-tasks: Treat only synchronous grace periods urgently").

Fixes 65eb006d85a2a ("bpf: Move kernel test kfuncs to bpf_testmod")

jirka

Quoted 34 lines
---
 tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 9 +++++++++
 1 file changed, 9 insertions(+)
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +9 -0
index 91907b321f913..e7c9e1c7fde04 100644
@@ -2,6 +2,7 @@
 /* Copyright (c) 2020 Facebook */
 #include <linux/btf.h>
 #include <linux/btf_ids.h>
+#include <linux/delay.h>
 #include <linux/error-injection.h>
 #include <linux/init.h>
 #include <linux/module.h>
@@ -544,6 +545,14 @@ static int bpf_testmod_init(void)
 
 static void bpf_testmod_exit(void)
 {
+        /* Need to wait for all references to be dropped because
+         * bpf_kfunc_call_test_release() which currently resides in kernel can
+         * be called after bpf_testmod is unloaded. Once release function is
+         * moved into the module this wait can be removed.
+         */
+	while (refcount_read(&prog_test_struct.cnt) > 1)
+		msleep(20);
+
 	return sysfs_remove_bin_file(kernel_kobj, &bin_attr_bpf_testmod_file);
 }
 
Signature · 1 line
2.43.0