Skip to contentSkip to conversation

selftests/bpf: Fix map_kptr test failure

PATCH Superseded Yonghong Song 2 participants 4 messages Jan 15

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

PATCH 1/1 selftests/bpf: Fix map_kptr test failure

On my arm64 machine, I get the following failure:
...
tester_init:PASS:tester_log_buf 0 nsec process_subtest:PASS:obj_open_mem 0 nsec process_subtest:PASS:specs_alloc 0 nsec serial_test_map_kptr:PASS:rcu_tasks_trace_gp__open_and_load 0 nsec
...
test_map_kptr_success:PASS:map_kptr__open_and_load 0 nsec test_map_kptr_success:PASS:test_map_kptr_ref1 refcount 0 nsec test_map_kptr_success:FAIL:test_map_kptr_ref1 retval unexpected error: 2 (errno 2) test_map_kptr_success:PASS:test_map_kptr_ref2 refcount 0 nsec test_map_kptr_success:FAIL:test_map_kptr_ref2 retval unexpected error: 1 (errno 2)
...
#201/21 map_kptr/success-map:FAIL

In serial_test_map_kptr(), before test_map_kptr_success(), one kern_sync_rcu() is used to have some delay for freeing the map. But in my environment, one kern_sync_rcu() seems not enough and caused the test failure.

In bpf_map_free_in_work() in syscall.c, the queue time for
queue_work(system_dfl_wq, &map->work)
may be longer than expected. This may cause the test failure since test_map_kptr_success() expects all previous maps having been freed.

In stead of one kern_sync_rcu() before test_map_kptr_success(), I added two more kern_sync_rcu() to have a longer delay and the test succeeded.

Cc Kumar Kartikeya Dwivedi <[email protected]>
---
 tools/testing/selftests/bpf/prog_tests/map_kptr.c | 4 ++++
 1 file changed, 4 insertions(+)
tools/testing/selftests/bpf/prog_tests/map_kptr.c +4 -0
index 8743df599567..f9cfc4d3153c 100644
@@ -148,11 +148,15 @@ void serial_test_map_kptr(void)
 
 		ASSERT_OK(kern_sync_rcu_tasks_trace(skel), "sync rcu_tasks_trace");
 		ASSERT_OK(kern_sync_rcu(), "sync rcu");
+		ASSERT_OK(kern_sync_rcu(), "sync rcu");
+		ASSERT_OK(kern_sync_rcu(), "sync rcu");
 		/* Observe refcount dropping to 1 on bpf_map_free_deferred */
 		test_map_kptr_success(false);
 
 		ASSERT_OK(kern_sync_rcu_tasks_trace(skel), "sync rcu_tasks_trace");
 		ASSERT_OK(kern_sync_rcu(), "sync rcu");
+		ASSERT_OK(kern_sync_rcu(), "sync rcu");
+		ASSERT_OK(kern_sync_rcu(), "sync rcu");
 		/* Observe refcount dropping to 1 on synchronous delete elem */
 		test_map_kptr_success(true);
 	}
Signature · 1 line
2.47.3
#1 lore raw
Quoted 26 lines · Yonghong Song

On my arm64 machine, I get the following failure:
...
tester_init:PASS:tester_log_buf 0 nsec process_subtest:PASS:obj_open_mem 0 nsec process_subtest:PASS:specs_alloc 0 nsec serial_test_map_kptr:PASS:rcu_tasks_trace_gp__open_and_load 0 nsec
...
test_map_kptr_success:PASS:map_kptr__open_and_load 0 nsec test_map_kptr_success:PASS:test_map_kptr_ref1 refcount 0 nsec test_map_kptr_success:FAIL:test_map_kptr_ref1 retval unexpected error: 2 (errno 2) test_map_kptr_success:PASS:test_map_kptr_ref2 refcount 0 nsec test_map_kptr_success:FAIL:test_map_kptr_ref2 retval unexpected error: 1 (errno 2)
...
#201/21 map_kptr/success-map:FAIL

In serial_test_map_kptr(), before test_map_kptr_success(), one kern_sync_rcu() is used to have some delay for freeing the map. But in my environment, one kern_sync_rcu() seems not enough and caused the test failure.

In bpf_map_free_in_work() in syscall.c, the queue time for
queue_work(system_dfl_wq, &map->work)
may be longer than expected. This may cause the test failure since test_map_kptr_success() expects all previous maps having been freed.

Quoted 7 lines

In stead of one kern_sync_rcu() before test_map_kptr_success(), I added two more kern_sync_rcu() to have a longer delay and the test succeeded.

Cc Kumar Kartikeya Dwivedi <[email protected]>

---

This is still not a proper fix, right? Maybe two works in this case, but it isn't guaranteed to be enough either.
RCU gp wait won't have any synchronization with when wq items are executed. I forgot why I used kern_sync_rcu() originally, but I feel the right way to fix this would be to count when all maps have finished their bpf_map_free through an fexit hook. Thoughts?

Quoted 27 lines
 tools/testing/selftests/bpf/prog_tests/map_kptr.c | 4 ++++
 1 file changed, 4 insertions(+)
tools/testing/selftests/bpf/prog_tests/map_kptr.c +4 -1
index 8743df599567..f9cfc4d3153c 100644
@@ -148,11 +148,15 @@ void serial_test_map_kptr(void)
 
                ASSERT_OK(kern_sync_rcu_tasks_trace(skel), "sync rcu_tasks_trace");
                ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
                /* Observe refcount dropping to 1 on bpf_map_free_deferred */
                test_map_kptr_success(false);
 
                ASSERT_OK(kern_sync_rcu_tasks_trace(skel), "sync rcu_tasks_trace");
                ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
                /* Observe refcount dropping to 1 on synchronous delete elem */
                test_map_kptr_success(true);
        }
--

2.47.3

author reviewer
#2 lore raw
Quoted 33 lines · Kumar Kartikeya Dwivedi
Quoted 25 lines · Yonghong Song

On my arm64 machine, I get the following failure:
...
tester_init:PASS:tester_log_buf 0 nsec process_subtest:PASS:obj_open_mem 0 nsec process_subtest:PASS:specs_alloc 0 nsec serial_test_map_kptr:PASS:rcu_tasks_trace_gp__open_and_load 0 nsec
...
test_map_kptr_success:PASS:map_kptr__open_and_load 0 nsec test_map_kptr_success:PASS:test_map_kptr_ref1 refcount 0 nsec test_map_kptr_success:FAIL:test_map_kptr_ref1 retval unexpected error: 2 (errno 2) test_map_kptr_success:PASS:test_map_kptr_ref2 refcount 0 nsec test_map_kptr_success:FAIL:test_map_kptr_ref2 retval unexpected error: 1 (errno 2)
...
#201/21 map_kptr/success-map:FAIL

In serial_test_map_kptr(), before test_map_kptr_success(), one kern_sync_rcu() is used to have some delay for freeing the map. But in my environment, one kern_sync_rcu() seems not enough and caused the test failure.

In bpf_map_free_in_work() in syscall.c, the queue time for
queue_work(system_dfl_wq, &map->work)
may be longer than expected. This may cause the test failure since test_map_kptr_success() expects all previous maps having been freed.

Quoted 7 lines

In stead of one kern_sync_rcu() before test_map_kptr_success(), I added two more kern_sync_rcu() to have a longer delay and the test succeeded.

Cc Kumar Kartikeya Dwivedi <[email protected]>

---

Quoted 6 lines

This is still not a proper fix, right? Maybe two works in this case, but it isn't guaranteed to be enough either.
RCU gp wait won't have any synchronization with when wq items are executed. I forgot why I used kern_sync_rcu() originally, but I feel the right way to fix this would be to count when all maps have finished their bpf_map_free through an fexit hook. Thoughts?

Agree that this is still not to guarantee it won't break due to queue_work(). One possibility is to count the references in a separate bpf program and until all references are gone then we can do subsequent test_map_kptr_success(). Let me give a try.

Quoted 28 lines
Quoted 28 lines
  tools/testing/selftests/bpf/prog_tests/map_kptr.c | 4 ++++
  1 file changed, 4 insertions(+)
tools/testing/selftests/bpf/prog_tests/map_kptr.c +4 -1
index 8743df599567..f9cfc4d3153c 100644
@@ -148,11 +148,15 @@ void serial_test_map_kptr(void)
 
                 ASSERT_OK(kern_sync_rcu_tasks_trace(skel), "sync rcu_tasks_trace");
                 ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
                 /* Observe refcount dropping to 1 on bpf_map_free_deferred */
                 test_map_kptr_success(false);
 
                 ASSERT_OK(kern_sync_rcu_tasks_trace(skel), "sync rcu_tasks_trace");
                 ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
                 /* Observe refcount dropping to 1 on synchronous delete elem */
                 test_map_kptr_success(true);
         }
--

2.47.3

#3 lore raw
Quoted 44 lines · Yonghong Song
Quoted 33 lines · Kumar Kartikeya Dwivedi
Quoted 25 lines · Yonghong Song

On my arm64 machine, I get the following failure:
...
tester_init:PASS:tester_log_buf 0 nsec process_subtest:PASS:obj_open_mem 0 nsec process_subtest:PASS:specs_alloc 0 nsec serial_test_map_kptr:PASS:rcu_tasks_trace_gp__open_and_load 0 nsec
...
test_map_kptr_success:PASS:map_kptr__open_and_load 0 nsec test_map_kptr_success:PASS:test_map_kptr_ref1 refcount 0 nsec test_map_kptr_success:FAIL:test_map_kptr_ref1 retval unexpected error: 2 (errno 2) test_map_kptr_success:PASS:test_map_kptr_ref2 refcount 0 nsec test_map_kptr_success:FAIL:test_map_kptr_ref2 retval unexpected error: 1 (errno 2)
...
#201/21 map_kptr/success-map:FAIL

In serial_test_map_kptr(), before test_map_kptr_success(), one kern_sync_rcu() is used to have some delay for freeing the map. But in my environment, one kern_sync_rcu() seems not enough and caused the test failure.

In bpf_map_free_in_work() in syscall.c, the queue time for
queue_work(system_dfl_wq, &map->work)
may be longer than expected. This may cause the test failure since test_map_kptr_success() expects all previous maps having been freed.

Quoted 7 lines

In stead of one kern_sync_rcu() before test_map_kptr_success(), I added two more kern_sync_rcu() to have a longer delay and the test succeeded.

Cc Kumar Kartikeya Dwivedi <[email protected]>

---

Quoted 6 lines

This is still not a proper fix, right? Maybe two works in this case, but it isn't guaranteed to be enough either.
RCU gp wait won't have any synchronization with when wq items are executed. I forgot why I used kern_sync_rcu() originally, but I feel the right way to fix this would be to count when all maps have finished their bpf_map_free through an fexit hook. Thoughts?

Quoted 4 lines

Agree that this is still not to guarantee it won't break due to queue_work(). One possibility is to count the references in a separate bpf program and until all references are gone then we can do subsequent test_map_kptr_success(). Let me give a try.

That seems better actually, and probably less code too.

Quoted 30 lines
Quoted 29 lines
Quoted 29 lines

  tools/testing/selftests/bpf/prog_tests/map_kptr.c | 4 ++++
  1 file changed, 4 insertions(+)
tools/testing/selftests/bpf/prog_tests/map_kptr.c +4 -1
index 8743df599567..f9cfc4d3153c 100644
@@ -148,11 +148,15 @@ void serial_test_map_kptr(void)
 
                 ASSERT_OK(kern_sync_rcu_tasks_trace(skel), "sync rcu_tasks_trace");
                 ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
                 /* Observe refcount dropping to 1 on bpf_map_free_deferred */
                 test_map_kptr_success(false);
 
                 ASSERT_OK(kern_sync_rcu_tasks_trace(skel), "sync rcu_tasks_trace");
                 ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
+               ASSERT_OK(kern_sync_rcu(), "sync rcu");
                 /* Observe refcount dropping to 1 on synchronous delete elem */
                 test_map_kptr_success(true);
         }
--

2.47.3