Skip to contentSkip to conversation

memcg/functional: Convert to newlib

PATCH New Joerg Vehlow 3 participants 9 messages Jan 25, 2021 – Feb 15, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 memcg/functional: Convert to newlib

From: Joerg Vehlow <[email protected]>

This is an intermediate step, to allow further improvements to tests in memcg/functional. The next steps are
- fixing synchronization between shell and memcg_process: At the moment signals may get lost (especially in warmup), because the memcg_process is signaled twice without any time in between. If the process is not fast enough, the second signal can get lost. I am already working on something for a better synchronization.
- Some tests fail always (at least for me). This can be due to the fact, that memory.use_hierarchy cannot be disabled easily on systems using systemd, because it enables it and ads lots of subgroups. (Disableing is only possible, if there are no subgroups).
I added TCONF for some tests, that obviously failed because of that, but I am not sure for the remaining errors
The second reason for failed tests is the fact, that the value in memory.limit_in_bytes is not exact. The kernel documentation does not say how this actually works, but I guess it is baed on per_cpu_counters. The checks will propably require an epsilon, but that needs some investigation in the kernel code.

In addtion to converting the tests to the new library:
- Moved non-shared code from memcg_lib to the respective test files
- Removed unused test_hugepage from memcg_lib

But even in this state, I think it should be merged, because it will simplify review for upcomming changes.

BTW: This was the last shell-user of TST_CHECKPOINT. In theory it could be removed from test.sh, if we want to gradually strip down unused code, until the whole file is gone.

J?rg

---
 .../memcg/functional/memcg_failcnt.sh         |  79 +--
 .../memcg/functional/memcg_force_empty.sh     |  64 +-
 .../controllers/memcg/functional/memcg_lib.sh | 559 ++++++------------
 .../memcg/functional/memcg_limit_in_bytes.sh  | 106 ++--
 .../memcg_max_usage_in_bytes_test.sh          | 111 ++--
 .../memcg_memsw_limit_in_bytes_test.sh        |  85 +--
 .../memcg_move_charge_at_immigrate_test.sh    | 101 ++--
 .../memcg/functional/memcg_stat_rss.sh        |  66 +--
 .../memcg/functional/memcg_stat_test.sh       |  93 ++-
 .../memcg/functional/memcg_subgroup_charge.sh |  79 +--
 .../functional/memcg_usage_in_bytes_test.sh   |  66 +--
 .../functional/memcg_use_hierarchy_test.sh    |  59 +-
 12 files changed, 561 insertions(+), 907 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh +30 -49
index d5034a514..774e69d04 100755
@@ -1,58 +1,39 @@
 #!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2009 FUJITSU LIMITED
+# Copyright (c) 2018-2019 ARM Ltd. All Rights Reserved.
+# Copyright (c) 2021 Joerg Vehlow <[email protected]>
+#
+# Author: Li Zefan <[email protected]>
+# Restructure for LTP: Shi Weihua <[email protected]>
+# Added memcg enable/disable functionality: Rishikesh K Rajak <[email protected]>
 
-################################################################################
-##                                                                            ##
-## Copyright (c) 2009 FUJITSU LIMITED                                         ##
-##                                                                            ##
-## This program is free software;  you can redistribute it and#or modify      ##
-## it under the terms of the GNU General Public License as published by       ##
-## the Free Software Foundation; either version 2 of the License, or          ##
-## (at your option) any later version.                                        ##
-##                                                                            ##
-## This program is distributed in the hope that it will be useful, but        ##
-## WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY ##
-## or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License   ##
-## for more details.                                                          ##
-##                                                                            ##
-## You should have received a copy of the GNU General Public License          ##
-## along with this program;  if not, write to the Free Software Foundation,   ##
-## Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA           ##
-##                                                                            ##
-## Author: Li Zefan <[email protected]>                                     ##
-## Restructure for LTP: Shi Weihua <[email protected]>                     ##
-## Added memcg enable/disable functinality: Rishikesh K Rajak		      ##
-##						<[email protected]  ##
-##                                                                            ##
-################################################################################
-
-TCID="memcg_failcnt"
-TST_TOTAL=3
-
+MEMCG_TESTFUNC=test
+MEMCG_SHMMAX=1
+TST_TEST_DATA="--mmap-anon --mmap-file --shm"
 . memcg_lib.sh
 
-# Test memory.failcnt
-testcase_1()
-{
-	echo $PAGESIZE > memory.limit_in_bytes
-	malloc_free_memory "--mmap-anon" $(($PAGESIZE*2))
-	test_failcnt "memory.failcnt"
-}
+MEMORY_LIMIT=$PAGESIZE
+MEMORY_TO_ALLOCATE=$((MEMORY_LIMIT * 2))
 
-testcase_2()
+test()
 {
-	echo $PAGESIZE > memory.limit_in_bytes
-	malloc_free_memory "--mmap-file" $(($PAGESIZE*2))
-	test_failcnt "memory.failcnt"
-}
+	echo $MEMORY_LIMIT > memory.limit_in_bytes
 
-testcase_3()
-{
-	echo $PAGESIZE > memory.limit_in_bytes
-	malloc_free_memory "--shm" $(($PAGESIZE*2))
-	test_failcnt "memory.failcnt"
+	start_memcg_process $2 -s ${MEMORY_TO_ALLOCATE}
+	echo $MEMCG_PROCESS_PID > tasks
+
+	signal_memcg_process ${MEMORY_TO_ALLOCATE}
+	signal_memcg_process ${MEMORY_TO_ALLOCATE}
+
+	stop_memcg_process
+
+	failcnt=$(cat memory.failcnt)
+	if [ $failcnt -gt 0 ]; then
+		tst_res TPASS "memory.failcnt is $failcnt, > 0 as expected"
+	else
+		tst_res TFAIL "memory.failcnt is $failcnt, <= 0 expected"
+	fi
 }
 
-shmmax_setup
-LOCAL_CLEANUP=shmmax_cleanup
-run_tests
-tst_exit
+tst_run
testcases/kernel/controllers/memcg/functional/memcg_force_empty.sh +22 -42

98 lines not on this page · raw message

testcases/kernel/controllers/memcg/functional/memcg_lib.sh +180 -379

708 lines not on this page · raw message

testcases/kernel/controllers/memcg/functional/memcg_limit_in_bytes.sh +47 -59

172 lines not on this page · raw message

testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +54 -57

133 lines not on this page · raw message

testcases/kernel/controllers/memcg/functional/memcg_memsw_limit_in_bytes_test.sh +25 -60

146 lines not on this page · raw message

testcases/kernel/controllers/memcg/functional/memcg_move_charge_at_immigrate_test.sh +60 -41

123 lines not on this page · raw message

testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh +23 -43

112 lines not on this page · raw message

testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh +33 -60

156 lines not on this page · raw message

testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh +45 -34

95 lines not on this page · raw message

testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh +21 -45

79 lines not on this page · raw message

testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh +21 -38

90 lines not on this page · raw message

Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/1 memcg/functional: Convert to newlib

--- a/testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh

testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh line 4

Quoted 6 lines

+++ b/testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh

testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh +3 -0
@@ -1,58 +1,39 @@
  #!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2009 FUJITSU LIMITED
+# Copyright (c) 2018-2019 ARM Ltd. All Rights Reserved.

This is bad copy-and-paste for all files, where I mentioned arm, sorry...

J?rg

#2 lore raw

PATCH 1/1 memcg/functional: Convert to newlib

Hi J?rg,

Quoted 8 lines

--- a/testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh

testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh line 4

Quoted 6 lines

+++ b/testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh

testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh +3 -0
@@ -1,58 +1,39 @@
  #!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2009 FUJITSU LIMITED
+# Copyright (c) 2018-2019 ARM Ltd. All Rights Reserved.

This is bad copy-and-paste for all files, where I mentioned arm, sorry...

That's not a big deal.

On a first look good cleanup, thank you!

I hope these tests aren't planned to be rewritten to C (like NUMA tests). I suppose not, but better to ask earlier.

Kind regards,
Petr

#3 lore raw

PATCH 1/1 memcg/functional: Convert to newlib

Hello Petr,

Quoted 10 lines · Petr Vorel

Hi J?rg,

Quoted 8 lines

--- a/testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh

testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh line 4

Quoted 6 lines

+++ b/testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh

testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh +3 -0
@@ -1,58 +1,39 @@
  #!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2009 FUJITSU LIMITED
+# Copyright (c) 2018-2019 ARM Ltd. All Rights Reserved.

This is bad copy-and-paste for all files, where I mentioned arm, sorry...

Quoted 6 lines

That's not a big deal.

On a first look good cleanup, thank you!

I hope these tests aren't planned to be rewritten to C (like NUMA tests). I suppose not, but better to ask earlier.

I can look into that as they should use the new (WIP) CGroup C API. For now it is probably best to keep them in Shell.

Quoted 3 lines

Kind regards,
Petr

Signature · 2 lines
Thank you,
Richard.
#4 lore raw

PATCH 1/1 memcg/functional: Convert to newlib

In reply to Petr Vorel

Hi Petr,

Quoted 4 lines

On a first look good cleanup, thank you!

I hope these tests aren't planned to be rewritten to C (like NUMA tests). I suppose not, but better to ask earlier.

can this be merged then?
I'd like to submit some more patches on top of this.

J?rg

#5 lore raw

PATCH 1/1 memcg/functional: Convert to newlib

Hi Joerg,

Quoted 8 lines

Hi Petr,

Quoted 4 lines

On a first look good cleanup, thank you!

I hope these tests aren't planned to be rewritten to C (like NUMA tests). I suppose not, but better to ask earlier.

can this be merged then?

Good cleanup, thanks! I fixed copyright and merged.

Quoted 1 line

I'd like to submit some more patches on top of this.

Thanks, these tests certainly deserve more work.

Quoted 1 line

J?rg

Kind regards,
Petr

#6 lore raw

PATCH 1/1 memcg/functional: Convert to newlib

In reply to Joerg Vehlow

Hi Joerg,

Quoted 1 line

I'd like to submit some more patches on top of this.

You probably noticed some failures (on 5.10.14 openSUSE kernel):

memcg_stat_rss, memcg_subgroup_charge, memcg_move_charge_at_immigrate, memcg_stat:
TBROK: timed out on memory.usage_in_bytes
=> maybe just timeout?

memcg_max_usage_in_bytes
memcg_max_usage_in_bytes_test 1 TFAIL: memory.max_usage_in_bytes is 4325376, 4194304 expected memcg_max_usage_in_bytes_test 2 TFAIL: memory.memsw.max_usage_in_bytes is 4325376, 4194304 expected memcg_max_usage_in_bytes_test 3 TFAIL: memory.max_usage_in_bytes is 4325376, 4194304 expected memcg_max_usage_in_bytes_test 3 TFAIL: memory.max_usage_in_bytes is 126976, 0 expected

memcg_usage_in_bytes
memcg_usage_in_bytes_test 1 TFAIL: memory.usage_in_bytes is 4325376, 4194304 expected memcg_usage_in_bytes_test 2 TFAIL: memory.memsw.usage_in_bytes is 4325376, 4194304 expected

Tests which needs further cleanup/rewrite (less important than fixing previous) testcases/kernel/controllers/memcg/regression/memcg_regression_test.sh testcases/kernel/controllers/memcg/regression/memcg_test_4.sh testcases/kernel/controllers/memcg/control/memcg_control_test.sh

Kind regards,
Petr

#8 lore raw

PATCH 1/1 memcg/functional: Convert to newlib

Hi,

Quoted 4 lines · Petr Vorel

Hi,

FYI there is bug related to these tests:
https://github.com/linux-test-project/ltp/issues/780

Additionally there is
https://github.com/linux-test-project/ltp/issues/698
https://github.com/linux-test-project/ltp/issues/326
https://github.com/linux-test-project/ltp/issues/93

I guess the last one is invalid.
326 and 698 are basically duplicates due to inaccurate memory accounting in cgroups. The same as 780.
I did not fix these failures for now. I haven't checked the kernel sources. There is a hint in the documentation for memory.usage_in_bytes "Of course, when necessary, it's synchronized.".
Maybe it is possible to force synchronization somehow.

I will look into this, after this is merged.

Quoted 3 lines

Kind regards,
Petr