cgroup_regression_test: Use unique names for hierarchies in each test
PATCH 1/1 cgroup_regression_test: Use unique names for hierarchies in each test
Recently we started having failures where mounting cgroup in quick succession after umount of same named hierarchy fails with EBUSY
cgroup_regression_test 1 TPASS: no kernel bug was found
mount: /tmp/LTP_cgroup_regression_test.ZFwS3JkX4c/cgroup: cgroup already mounted or mount point busy.
dmesg(1) may have more information after failed mount system call.cgroup_regression_test 2 TFAIL: Failed to mount cgroup filesystem
Updating test to use unique names for cgroup hierarchies in each subtest.
--- .../controllers/cgroup/cgroup_regression_test.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
testcases/kernel/controllers/cgroup/cgroup_regression_test.sh +5 -5
@@ -93,7 +93,7 @@ test1() cgroup_regression_fork_processes & sleep 1 - mount -t cgroup -o none,name=foo cgroup cgroup/ + mount -t cgroup -o none,name=foo1 cgroup cgroup/ if [ $? -ne 0 ]; then tst_res TFAIL "failed to mount cgroup filesystem" kill -TERM $!
@@ -118,7 +118,7 @@ test2() local val1 local val2 - mount -t cgroup -o none,name=foo cgroup cgroup/ + mount -t cgroup -o none,name=foo2 cgroup cgroup/ if [ $? -ne 0 ]; then tst_res TFAIL "Failed to mount cgroup filesystem" return
@@ -204,7 +204,7 @@ test4() return fi - mount -t cgroup -o none,name=foo cgroup cgroup/ + mount -t cgroup -o none,name=foo4 cgroup cgroup/ mkdir cgroup/0 rmdir cgroup/0 tst_umount $PWD/cgroup
@@ -309,7 +309,7 @@ test_7_2() { local subsys=$1 - mount -t cgroup -o none,name=foo cgroup cgroup/ + mount -t cgroup -o none,name=foo72 cgroup cgroup/ if [ $? -ne 0 ]; then tst_res TFAIL "failed to mount cgroup" return
@@ -370,7 +370,7 @@ test7() #--------------------------------------------------------------------------- test8() { - mount -t cgroup -o none,name=foo cgroup cgroup/ + mount -t cgroup -o none,name=foo8 cgroup cgroup/ if [ $? -ne 0 ]; then tst_res TFAIL "failed to mount cgroup filesystem" return
Signature · 5 lines
2.43.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
I forgot to mention that I also tried to usetst_umount() helper routine, but that did not help as umount does not really report any failures here.
Quoted 67 lines · CEST Avinesh Kumar
Recently we started having failures where mounting cgroup in quick succession after umount of same named hierarchy fails with EBUSY
cgroup_regression_test1 TPASS: no kernel bug was found
mount:/tmp/LTP_cgroup_regression_test.ZFwS3JkX4c/cgroup: cgroup already mounted or mount point busy.
dmesg(1) may have more information after failed mount system call.cgroup_regression_test2 TFAIL: Failed to mount cgroup filesystemUpdating test to use unique names for cgroup hierarchies in each subtest.
Suggested-by Michal Koutný <[email protected]>Signed-off-by Avinesh Kumar <[email protected]>--- .../controllers/cgroup/cgroup_regression_test.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)testcases/kernel/controllers/cgroup/cgroup_regression_test.sh +5
-5@@ -93,7 +93,7 @@ test1() cgroup_regression_fork_processes & sleep 1 - mount -t cgroup -o none,name=foo cgroup cgroup/ + mount -t cgroup -o none,name=foo1 cgroup cgroup/ if [ $? -ne 0 ]; then tst_res TFAIL "failed to mount cgroup filesystem" kill -TERM $!@@ -118,7 +118,7 @@ test2() local val1 local val2 - mount -t cgroup -o none,name=foo cgroup cgroup/ + mount -t cgroup -o none,name=foo2 cgroup cgroup/ if [ $? -ne 0 ]; then tst_res TFAIL "Failed to mount cgroup filesystem" return@@ -204,7 +204,7 @@ test4() return fi - mount -t cgroup -o none,name=foo cgroup cgroup/ + mount -t cgroup -o none,name=foo4 cgroup cgroup/ mkdir cgroup/0 rmdir cgroup/0 tst_umount $PWD/cgroup@@ -309,7 +309,7 @@ test_7_2() { local subsys=$1 - mount -t cgroup -o none,name=foo cgroup cgroup/ + mount -t cgroup -o none,name=foo72 cgroup cgroup/ if [ $? -ne 0 ]; then tst_res TFAIL "failed to mount cgroup" return@@ -370,7 +370,7 @@ test7() #--------------------------------------------------------------------------- test8() { - mount -t cgroup -o none,name=foo cgroup cgroup/ + mount -t cgroup -o none,name=foo8 cgroup cgroup/ if [ $? -ne 0 ]; then tst_res TFAIL "failed to mount cgroup filesystem" return
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 3 lines · Avinesh Kumar
I forgot to mention that I also tried to use
tst_umount()helper routine, but that did not help as umount does not really report any failures here.
The solution doesn't really convince me. If it's true that a util-linux upgrade to v2.41 is causing this issue, we should probably research why this is happening in there and (eventually) fix it. Or, if it's not a library bug, we should fix tst_umount() in order to recognize when a cgroups folder hasn't been fully unmounted.
@Cyril WDYT ?
- Andrea
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 11 lines
Quoted 3 lines
I forgot to mention that I also tried to use
tst_umount()helper routine, but that did not help as umount does not really report any failures here.The solution doesn't really convince me. If it's true that a util-linux upgrade to v2.41 is causing this issue, we should probably research why this is happening in there and (eventually) fix it. Or, if it's not a library bug, we should fix
tst_umount()in order to recognize when a cgroups folder hasn't been fully unmounted.@Cyril WDYT ?
This indeed looks like a workaround for a bug either in the test or in the system.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 10 lines · Cyril Hrubis
Hi!
Quoted 9 lines
Quoted 3 lines
I forgot to mention that I also tried to use
tst_umount()helper routine, but that did not help as umount does not really report any failures here.The solution doesn't really convince me. If it's true that a util-linux upgrade to v2.41 is causing this issue, we should probably research why this is happening in there and (eventually) fix it. Or, if it's not a library bug, we should fix
tst_umount()in order to recognize when a cgroups folder hasn't been fully unmounted.
Quoted 4 lines
Quoted 1 line
@Cyril WDYT ?
This indeed looks like a workaround for a bug either in the test or in the system.
Perhaps, by placing sync before every new test which is going to mount a new cgroup folder we fix the issue as well. I'm wondering if we should dig into it a big deeper...
- Andrea
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
A message that is not in the archive (sent off-list or not yet received): mqkxe24qlrcpb5jxgigbiarxxgemqjw3rmgnsprn55ubghrrda@etaozcz2nemy
Quoted 14 lines · Michal Koutný
Quoted 1 line · Andrea Cervesato
On 7/10/25 11:00 AM, Cyril Hrubis wrote:
...
Quoted 5 lines
Quoted 3 lines
This indeed looks like a workaround for a bug either in the test or in the system.
Perhaps, by placing sync before every new test which is going to mount a new cgroup folder we fix the issue as well.
That may only a workaround too. My bet is you'd rather need some RCU barriers or workqueue flushing, thus the simple separation by name. The cleanup path is just so asynchronous (the code had already attempted some nastiness [1], however, I'm suspecting its efficiency may be reduced under layers of VFS reworks since then).
Quoted 5 lines
Quoted 1 line
I'm wondering if we should dig into it a big deeper...
It may be worth checking that there's a change in util-linux that has (presumably) affected timings.
Note that unmounting a cgroup hierarchy is operation that typically happens once during "lifetime" (if at all) -- so it's your call.
I see, thanks for the suggestion. I will take a look.
Quoted 5 lines
HTH,
Michal
This git commit is part of cgroup v1 code and now in kernel/cgroup/cgroup-v1.c:cgroup1_root_to_use, but I don't see the same mechanism used by cgroup v2, which is our main version. Is that something which might affect our system?
- Andrea
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
A message that is not in the archive (sent off-list or not yet received): svs6pl6jmibp7uylvofkia5smvu2cuezfvnawlgkmjiulbrpfm@w7s7dlxxltap
Quoted 4 lines · Michal Koutný
Quoted 3 lines · Andrea Cervesato
This git commit is part of cgroup v1 code and now in kernel/cgroup/cgroup-v1.c:cgroup1_root_to_use, but I don't see the same mechanism used by cgroup v2, which is our main version.
Quoted 8 lines
The patched tests use cgroup v1.
Quoted 1 line
Is that something which might affect our system?
Since there's only one cgroup v2 (unified) hierarchy, it sort of lives forever (statically alloc'd), so I wouldn't expect such residuals on v2 (with the current implementation).
Michal
I bisected util-linux, but it doesn't seem to be the problem. On my system (Tumbleweed), we are facing the same failure, no matter the commit that is tested in util-linux between 2.39 and 2.41 versions.
I guess this issue might be caused by a specific setup of the Linux kernel and its async nature, mixed with some services which are slowing down the umount process.
I'm wondering how to proceed, since this is actually a soft-failure and any fix to the test might hide future bugs.
- Andrea
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi all¸
I would like to ask if there are any other suggestions on how to address this issue.
I tried a rather simpler approach (as suggested by Martin) of just adding a delay using sleep after the umount call, which also seems to solve this issue of incomplete unmounting.
testcases/kernel/controllers/cgroup/cgroup_regression_test.sh +1 -0
@@ -102,10 +102,11 @@ test1() cat cgroup/tasks > /dev/null kill -TERM $! wait $! 2>/dev/null umount cgroup/ + sleep 2 check_kernel_bug }
I will send this in the patch if we agree to use this approach instead of unique names.
Thanks,
Avinesh
Quoted 28 lines · CEST Andrea Cervesato
Quoted 4 lines · Michal Koutný
Quoted 3 lines · Andrea Cervesato
This git commit is part of cgroup v1 code and now in kernel/cgroup/cgroup-v1.c:cgroup1_root_to_use, but I don't see the same mechanism used by cgroup v2, which is our main version.
Quoted 8 lines
The patched tests use cgroup v1.
Quoted 1 line
Is that something which might affect our system?
Since there's only one cgroup v2 (unified) hierarchy, it sort of lives forever (statically alloc'd), so I wouldn't expect such residuals on v2 (with the current implementation).
Michal
I bisected util-linux, but it doesn't seem to be the problem. On my system (Tumbleweed), we are facing the same failure, no matter the commit that is tested in util-linux between 2.39 and 2.41 versions.
I guess this issue might be caused by a specific setup of the Linux kernel and its async nature, mixed with some services which are slowing down the umount process.
I'm wondering how to proceed, since this is actually a soft-failure and any fix to the test might hide future bugs.
- Andrea
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 20 lines · Avinesh Kumar
Hi all¸
I would like to ask if there are any other suggestions on how to address this issue.
I tried a rather simpler approach (as suggested by Martin) of just adding a delay using sleep after the umount call, which also seems to solve this issue of incomplete unmounting.testcases/kernel/controllers/cgroup/cgroup_regression_test.sh +1
-0@@ -102,10 +102,11 @@ test1() cat cgroup/tasks > /dev/null kill -TERM $! wait $! 2>/dev/null umount cgroup/ + sleep 2 check_kernel_bug }
Quoted 2 lines
I will send this in the patch if we agree to use this approach instead of unique names.
Well, this looks like the unmount happens asynchronously and the command exits before it's actually finished. In that case, sync would be the best solution. But I think we need some feedback from kernel developers whether the asynchronous behavior is intentional or not. Should the umount command wait for the task to finish?
Signature · 10 lines
Martin Doucha [email protected] SW Quality Engineer SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 2 lines · Andrea Cervesato
I'm wondering how to proceed, since this is actually a soft-failure and any fix to the test might hide future bugs.
You can take the (unique names) fix from Avinesh and add a dedicated test for the unmounting (and solve (soft)failure only in that).
0.02€,
Michal
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp