selftests/cgroup: test_zswap: fix single-core flakes
Subsystems (4)
cover selftests/cgroup: test_zswap: fix single-core flakes
When running the cgroup zswap selftests on a single-core VM (-smp 1) with 4GB of RAM, test_zswap_writeback and test_no_kmem_bypass reliably fail on the initial run after boot due to timing and asynchronous tasks not completed.
This 2-patch series fixes both initial-run failures:
* Patch 1 fixes an EBUSY race in test_zswap_writeback() where writing "+memory" to cgroup.subtree_control collides with the kernel's asynchronous RCU removal of dead child tasks from cgroup.procs.
* Patch 2 fixes an implicit unsigned integer promotion bug in test_no_kmem_bypass() where a small negative delta becomes large when compared to an unsigned size_t threshold.
Both tests now reliably pass on the initial run after cold boot on -smp 1 VMs.
Wilson Felipe Pereira (2):
selftests/cgroup: test_zswap: retry subtree_control write on EBUSY in
test_zswap_writeback
selftests/cgroup: test_zswap: fix implicit unsigned promotion bug in
test_no_kmem_bypass
tools/testing/selftests/cgroup/test_zswap.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
--
2.55.0.571.g244d577d93-goog
PATCH 1/2 selftests/cgroup: test_zswap: retry subtree_control write on EBUSY in test_zswap_writeback
When running test_zswap on a single-core VM (-smp 1) with 4GB of RAM, test_zswap_writeback reliably fails on the initial run after boot.
In test_zswap_writeback(), after waitpid() reaps the child process created by test_zswap_writeback_one(), writing "+memory" to cgroup.subtree_control can fail with -EBUSY. Under Cgroup v2, enabling domain subtree controllers is forbidden while any tasks remain in cgroup.procs.
When a child process exits, waitpid() reaps the zombie PID immediately, but the removal of struct task_struct from the cgroup task list is performed asynchronously via an RCU callback (release_task). On single-core systems, this RCU callback is delayed behind CPU softirqs, causing "+memory" to fail if written immediately after waitpid() returns.
Fix this by adding an EBUSY retry loop with usleep(1000) around cg_write(), matching the existing cgroup cleanup pattern in cg_destroy().
--- tools/testing/selftests/cgroup/test_zswap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
tools/testing/selftests/cgroup/test_zswap.c +5 -2
@@ -407,8 +407,11 @@ static int test_zswap_writeback(const char *root, bool wb)* Thus, the parent's setting shall be what's in effect. */ if (cg_write (test_group,"memory.zswap.max" ,"max" )) goto out; -if (cg_write (test_group,"cgroup.subtree_control" ,"+memory" )) - goto out; +while (cg_write (test_group,"cgroup.subtree_control" ,"+memory" )) { +if (errno !=EBUSY ) + goto out; +usleep (1000 ); + } test_group_child =cg_name (test_group,"zswap_writeback_test_child" );if (!test_group_child)
Signature · 1 line
2.55.0.571.g244d577d93-goog
Yosry Ahmed
maintainerRe: [PATCH 1/2] selftests/cgroup: test_zswap: retry subtree_control write on EBUSY in test_zswap_writeback
Quoted 3 lines · Wilson Felipe Pereira
When running
test_zswapon a single-core VM (-smp 1) with 4GB of RAM,test_zswap_writebackreliably fails on the initial run after boot.
I see why this is related to having a single core, but I am not sure why it would only happen for the first run after boot.
Quoted 11 lines
In
test_zswap_writeback(), afterwaitpid()reaps the child process created bytest_zswap_writeback_one(), writing "+memory" to cgroup.subtree_control can fail with-EBUSY. Under Cgroup v2, enabling domain subtree controllers is forbidden while any tasks remain in cgroup.procs.When a child process exits,
waitpid()reaps the zombie PID immediately, but the removal of structtask_structfrom the cgroup task list is performed asynchronously via an RCU callback (release_task). On single-core systems, this RCU callback is delayed behind CPU softirqs, causing "+memory" to fail if written immediately afterwaitpid()returns.
IIUC, the rejection to update subtree_control comes from cgroup_vet_subtree_control_enable() -> cgroup_has_tasks(), which reads nr_populated_csets.
This seems to be updated in this path when a task exits:finish_task_switch()cgroup_task_dead()do_cgroup_task_dead() (in !CONFIG_PREEMPT_RT) css_set_move_task()css_set_update_populated() css_update_populated()
So outside of CONFIG_PREEMPT_RT, I don't think the update is deferred.
Probably what's actually happening is that parent gets woken up through do_exit() -> exit_notify(), then the task is marked as TASK_DEAD in do_task_dead(), and finally finish_task_switch() calls cgroup_task_dead()?
Quoted 14 lines
Fix this by adding an EBUSY retry loop with usleep(1000) around
cg_write(), matching the existing cgroup cleanup pattern incg_destroy().Signed-off-by Wilson Felipe Pereira <[email protected]>--- tools/testing/selftests/cgroup/test_zswap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/testing/selftests/cgroup/test_zswap.c +0
-0
tools/testing/selftests/cgroup/test_zswap.c static int test_zswap_writeback(const char *root, bool wb) · line 414
Quoted 10 lines
tools/testing/selftests/cgroup/test_zswap.c +5
-2@@ -407,8 +407,11 @@ static int test_zswap_writeback(const char *root, bool wb)* Thus, the parent's setting shall be what's in effect. */ if (cg_write (test_group,"memory.zswap.max" ,"max" )) goto out; -if (cg_write (test_group,"cgroup.subtree_control" ,"+memory" )) - goto out; +while (cg_write (test_group,"cgroup.subtree_control" ,"+memory" )) { +if (errno !=EBUSY ) + goto out; +usleep (1000 ); + }
I would honestly rather create a new cgroup here instead of reusing the leaf cgroup as a parent, to avoid any subtleties like this in the future, or even split test_zswap_writeback() into two test cases. I don't feel strongly though if others think the simple wait here is enough.
Quoted 6 lines
tools/testing/selftests/cgroup/test_zswap.c +0
-1test_group_child =cg_name (test_group,"zswap_writeback_test_child" );if (!test_group_child) --2.55.0.571.g244d577d93-goog
Michal Koutný
maintainerRe: [PATCH 1/2] selftests/cgroup: test_zswap: retry subtree_control write on EBUSY in test_zswap_writeback
Quoted 13 lines · Yosry Ahmed
Quoted 13 lines
tools/testing/selftests/cgroup/test_zswap.c +3
-2@@ -407,8 +407,11 @@ static int test_zswap_writeback(const char *root, bool wb)* Thus, the parent's setting shall be what's in effect. */ if (cg_write (test_group,"memory.zswap.max" ,"max" )) goto out; -if (cg_write (test_group,"cgroup.subtree_control" ,"+memory" )) - goto out; +while (cg_write (test_group,"cgroup.subtree_control" ,"+memory" )) { +if (errno !=EBUSY ) + goto out;
Quoted 8 lines
tools/testing/selftests/cgroup/test_zswap.c static int test_zswap_writeback(const char *root, bool wb) · line 414
Quoted 2 lines
tools/testing/selftests/cgroup/test_zswap.c +2
-0+usleep (1000 ); + }I would honestly rather create a new cgroup here instead of reusing the leaf cgroup as a parent, to avoid any subtleties like this in the future, or even split
test_zswap_writeback()into two test cases. I don't feel strongly though if others think the simple wait here is enough.
I somewhat agree, I'm not sure whether the recycling brings any signal to the test.
In the case, the recycling and (re-)enablement is needed, I'd suggest an explicit check for emptiness, the best of utils arsenal seems:
cg_read_strcmp_wait(test_cgroup, "cgroup.events", "populated 0\n")
Thanks,
Michal
PATCH 2/2 selftests/cgroup: test_zswap: fix implicit unsigned promotion bug in test_no_kmem_bypass
In reply to Wilson Felipe PereiraIn test_no_kmem_bypass(), delta (stored_pages * page_size - zswapped) is checked against stored_pages * page_size / 4 to verify that the pages pushed to zswap belong to the test memory cgroup.
Due to slight stat update timing differences, delta can evaluate to a small negative number (e.g. -5MB out of 1GB). Because delta is declared as a signed int and stored_pages is an unsigned size_t, C's usual arithmetic conversions implicitly promote a negative delta to a large unsigned 64-bit integer, causing delta < stored_pages * page_size / 4 to falsely evaluate to 0 and fail the test.
Fix this by checking abs(delta), ensuring the test correctly compares the absolute difference between system zswap and cgroup zswapped bytes.
--- tools/testing/selftests/cgroup/test_zswap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/selftests/cgroup/test_zswap.c +2 -1
@@ -624,7 +624,8 @@ static int test_no_kmem_bypass(const char *root)if (stored_pages > stored_pages_threshold) {int zswapped =cg_read_key_long (test_group,"memory.stat" ,"zswapped " );int delta = stored_pages * page_size - zswapped; -int result_ok = delta < stored_pages * page_size /4 ; +int result_ok =abs (delta) < + stored_pages * page_size /4 ; ret = result_ok ?KSFT_PASS :KSFT_FAIL ;break ;
Signature · 1 line
2.55.0.571.g244d577d93-goog
Michal Koutný
maintainerRe: [PATCH 2/2] selftests/cgroup: test_zswap: fix implicit unsigned promotion bug in test_no_kmem_bypass
Quoted 10 lines · Wilson Felipe Pereira
In
test_no_kmem_bypass(), delta (stored_pages*page_size- zswapped) is checked againststored_pages*page_size/ 4 to verify that the pages pushed to zswap belong to the test memory cgroup.Due to slight stat update timing differences, delta can evaluate to a small negative number (e.g. -5MB out of 1GB). Because delta is declared as a signed int and
stored_pagesis an unsignedsize_t, C's usual arithmetic conversions implicitly promote a negative delta to a large unsigned 64-bit integer, causingdelta < stored_pages * page_size / 4to falsely evaluate to 0 and fail the test.
Thanks for the breakdown.
Quoted 2 lines
Fix this by checking abs(delta), ensuring the test correctly compares the absolute difference between system zswap and cgroup zswapped bytes.
I still think the delta quantity has some meaning here and taking the abs() changes the semantics of the checked inequality.
What about making both zswapped and delta longs? (Under similar reasoning, there's already a possible loss after squashing cg_read_key_long() result into the int.)
In any way, also
Thanks,
Michal