controllers/memcg: fixes for CPU nodes on newer kernels
v2 cover controllers/memcg: fixes for CPU nodes on newer kernels
Hi,
Changes since v1
================
1. Address Richard's review by echoing the new node-adjusted limit and counting nodes in easier way.
Description
===========
Rebased on top of latest master - no other dependencies.
The problem is that since v5.9 kernel, not only per-CPU but also per-node kernel memory is accounted to group. Creating a subgroup on v5.11 kernel on a two-node machine eats ~440 kB by itself in kernel memory.
The patchset tries to take it into account and adjust the memory limits. Have in mind however that kernel might change such accounting in the future.
Best regards,
Krzysztof
Krzysztof Kozlowski (4):
controllers/memcg: account per-node kernel memory controllers/memcg: fix memcg_stat_test on two node machines controllers/memcg: fail early to avoid possible false-positives controllers/memcg: check status of commands using interface
.../memcg/functional/memcg_failcnt.sh | 4 +- .../controllers/memcg/functional/memcg_lib.sh | 37 +++++++++++++++++- .../memcg_max_usage_in_bytes_test.sh | 10 ++--- .../memcg_memsw_limit_in_bytes_test.sh | 8 ++-- .../memcg_move_charge_at_immigrate_test.sh | 4 +- .../memcg/functional/memcg_stat_test.sh | 38 ++++++++++--------- .../memcg/functional/memcg_subgroup_charge.sh | 14 +++---- .../functional/memcg_usage_in_bytes_test.sh | 4 +- .../functional/memcg_use_hierarchy_test.sh | 14 ++++--- 9 files changed, 84 insertions(+), 49 deletions(-)
Signature · 1 line
2.30.2
PATCH v2 1/4 controllers/memcg: account per-node kernel memory
Recent Linux kernels charge groups also with kernel memory. This is not limited only to process-allocated memory but also cgroup-handling code memory as well.
For example since kernel v5.9 with commit 3e38e0aaca9e ("mm: memcg:
charge memcg percpu memory to the parent cgroup") creating a subgroup causes several kernel allocations towards this group.
These additional kernel memory allocations are proportional to number of CPUs and number of nodes.
On c4.8xlarge AWS instance with 36 cores in two nodes with v5.11 Linux kernel the memcg_subgroup_charge and memcg_use_hierarchy_test tests were failing:
memcg_use_hierarchy_test 1 TINFO: timeout per run is 0h 5m 0s
memcg_use_hierarchy_test 1 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed memcg_use_hierarchy_test 1 TINFO: test if one of the ancestors goes over its limit, the proces will be killed
mkdir: cannot create directory ?subgroup?: Cannot allocate memory /home/ubuntu/ltp-install/testcases/bin/memcg_use_hierarchy_test.sh: 26: cd: can't cd to subgroup memcg_use_hierarchy_test 1 TINFO: Running memcg_process --mmap-lock1 -s 8192 memcg_use_hierarchy_test 1 TFAIL: process is not killed
rmdir: failed to remove 'subgroup': No such file or directory
The kernel was unable to create the subgroup (mkdir returned -ENOMEM) due to this additional per-node kernel memory allocations.
--- .../controllers/memcg/functional/memcg_lib.sh | 35 +++++++++++++++++++ .../memcg/functional/memcg_subgroup_charge.sh | 8 ++--- .../functional/memcg_use_hierarchy_test.sh | 6 ++-- 3 files changed, 41 insertions(+), 8 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_lib.sh +35 -0
@@ -63,6 +63,41 @@ memcg_require_hierarchy_disabled() fi } +# Kernel memory allocated for the process is also charged. It might depend on +# the number of CPUs and number of nodes. For example on kernel v5.11 +# additionally total_cpus (plus 1 or 2) pages are charged to the group via +# kernel memory. For a two-node machine, additional 108 pages kernel memory +# are charged to the group. +# +# Adjust the limit to account such per-CPU and per-node kernel memory. +# $1 - expected cgroup memory limit value to adjust +memcg_adjust_limit_for_kmem() +{ + [ $# -ne 1 ] && tst_brk TBROK "memcg_adjust_limit_for_kmem expects 1 parameter" + + local limit=$1 + + # Total number of CPUs + local total_cpus=`tst_ncpus` + + # Total number of nodes + if [ ! -d /sys/devices/system/node/node0 ]; then + total_nodes=1 + else + total_nodes=`ls /sys/devices/system/node/ | grep -c "node[0-9][0-9]*"` + fi + + local node_mem=0 + if [ $total_nodes -gt 1 ]; then + node_mem=$((total_nodes - 1)) + node_mem=$((node_mem * PAGESIZE * 128)) + fi + + limit=$((limit + 4 * PAGESIZE + total_cpus * PAGESIZE + node_mem)) + + echo $limit +} + memcg_setup() { if ! is_cgroup_subsystem_available_and_enabled "memory"; then
testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh +2 -6
@@ -24,16 +24,12 @@ test_subgroup() { local limit_parent=$1 local limit_subgroup=$2 - local total_cpus=`tst_ncpus` - # Kernel memory allocated for the process is also charged. - # It might depend on the number of CPUs. For example on kernel v5.11 - # additionally total_cpus plus 1-2 pages are charged to the group. if [ $limit_parent -ne 0 ]; then - limit_parent=$((limit_parent + 4 * PAGESIZE + total_cpus * PAGESIZE)) + limit_parent=$(memcg_adjust_limit_for_kmem $limit_parent) fi if [ $limit_subgroup -ne 0 ]; then - limit_subgroup=$((limit_subgroup + 4 * PAGESIZE + total_cpus * PAGESIZE)) + limit_subgroup=$(memcg_adjust_limit_for_kmem $limit_subgroup) fi mkdir subgroup
testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh +4 -2
@@ -15,12 +15,14 @@ test1() { tst_res TINFO "test if one of the ancestors goes over its limit, the proces will be killed" + local limit=$(memcg_adjust_limit_for_kmem $PAGESIZE) + echo 1 > memory.use_hierarchy - echo $PAGESIZE > memory.limit_in_bytes + echo $limit > memory.limit_in_bytes mkdir subgroup cd subgroup - test_proc_kill $((PAGESIZE * 3)) "--mmap-lock1" $((PAGESIZE * 2)) 0 + test_proc_kill $((limit + PAGESIZE * 3)) "--mmap-lock1" $((limit + PAGESIZE * 2)) 0 cd .. rmdir subgroup
Signature · 1 line
2.30.2
PATCH v2 1/4 controllers/memcg: account per-node kernel memory
Hello Krzysztof,
Thanks, that is much easier to understand!
Signature · 2 lines
Thank you, Richard.
PATCH v2 2/4 controllers/memcg: fix memcg_stat_test on two node machines
In reply to Krzysztof KozlowskiRecent Linux kernels charge groups also with kernel memory. This is not limited only to process-allocated memory but also cgroup-handling code memory as well.
For example since kernel v5.9 with commit 3e38e0aaca9e ("mm: memcg:
charge memcg percpu memory to the parent cgroup") creating a subgroup causes several kernel allocations towards this group.
These additional kernel memory allocations are proportional to number of CPUs and number of nodes.
On c4.8xlarge AWS instance with 36 cores in two nodes with v5.11 Linux kernel the memcg_stat_test was failing because of unable to set memory limit lower than current kernel memory accounted to the group:
memcg_stat_test 5 TINFO: Test hierarchical_memory_limit with enabling hierarchical accounting
shell code from test:
> mkdir subgroup
> cat memory.kmem.usage_in_bytes
> 442368
> echo $PAGESIZE > memory.limit_in_bytes
/home/ubuntu/ltp-install/testcases/bin/memcg_stat_test.sh: 47: echo: echo: I/O error memcg_stat_test 5 TFAIL: hierarchical_memory_limit is 270336, 135168 expected
--- .../controllers/memcg/functional/memcg_stat_test.sh | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh +5 -3
@@ -43,12 +43,14 @@ test5() tst_res TINFO "Test hierarchical_memory_limit with enabling hierarchical accounting" echo 1 > memory.use_hierarchy + local limit=$(memcg_adjust_limit_for_kmem $PAGESIZES) + mkdir subgroup - echo $PAGESIZES > memory.limit_in_bytes - echo $((PAGESIZES * 2)) > subgroup/memory.limit_in_bytes + echo $limit > memory.limit_in_bytes + echo $((limit + PAGESIZES * 2)) > subgroup/memory.limit_in_bytes cd subgroup - check_mem_stat "hierarchical_memory_limit" $PAGESIZES + check_mem_stat "hierarchical_memory_limit" $limit cd .. rmdir subgroup
Signature · 1 line
2.30.2
PATCH v2 2/4 controllers/memcg: fix memcg_stat_test on two node machines
Signature · 2 lines
Thank you, Richard.
PATCH v2 3/4 controllers/memcg: fail early to avoid possible false-positives
In reply to Krzysztof KozlowskiTests require certain prerequisites. If these fail, there is no point to continue with the test. Trying to continue might lead to false positives, as seen on memcg_subgroup_charge which failed to set proper limit and did not catch actual failure:
memcg_subgroup_charge 1 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed memcg_subgroup_charge 1 TINFO: Test that group and subgroup have no relationship /home/ubuntu/ltp-install/testcases/bin/memcg_subgroup_charge.sh: 36: echo: echo: I/O error memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 270336 memcg_subgroup_charge 1 TINFO: Warming up pid: 13496
memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 13496
--- .../controllers/memcg/functional/memcg_failcnt.sh | 4 ++-- .../memcg/functional/memcg_max_usage_in_bytes_test.sh | 2 +- .../functional/memcg_memsw_limit_in_bytes_test.sh | 8 ++++---- .../functional/memcg_move_charge_at_immigrate_test.sh | 4 ++-- .../controllers/memcg/functional/memcg_stat_test.sh | 10 +++++----- .../memcg/functional/memcg_subgroup_charge.sh | 2 +- .../memcg/functional/memcg_use_hierarchy_test.sh | 10 +++++----- 7 files changed, 20 insertions(+), 20 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_failcnt.sh +2 -2
@@ -18,10 +18,10 @@ MEMORY_TO_ALLOCATE=$((MEMORY_LIMIT * 2)) test() { - echo $MEMORY_LIMIT > memory.limit_in_bytes + ROD echo $MEMORY_LIMIT \> memory.limit_in_bytes start_memcg_process $2 -s ${MEMORY_TO_ALLOCATE} - echo $MEMCG_PROCESS_PID > tasks + ROD echo $MEMCG_PROCESS_PID \> tasks signal_memcg_process ${MEMORY_TO_ALLOCATE} signal_memcg_process ${MEMORY_TO_ALLOCATE}
testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +1 -1
@@ -33,7 +33,7 @@ test_max_usage_in_bytes() return fi - echo $MEMCG_PROCESS_PID > tasks + ROD echo $MEMCG_PROCESS_PID \> tasks signal_memcg_process $MEM_TO_ALLOC signal_memcg_process $MEM_TO_ALLOC
testcases/kernel/controllers/memcg/functional/memcg_memsw_limit_in_bytes_test.sh +4 -4
@@ -55,7 +55,7 @@ test9() { memcg_require_memsw - echo 10M > memory.limit_in_bytes + ROD echo 10M \> memory.limit_in_bytes if tst_kvcmp -lt "2.6.31"; then EXPECT_FAIL echo -1 \> memory.memsw.limit_in_bytes
@@ -68,7 +68,7 @@ test10() { memcg_require_memsw - echo 10M > memory.limit_in_bytes + ROD echo 10M \> memory.limit_in_bytes EXPECT_FAIL echo 1.0 \> memory.memsw.limit_in_bytes }
@@ -76,7 +76,7 @@ test11() { memcg_require_memsw - echo 10M > memory.limit_in_bytes + ROD echo 10M \> memory.limit_in_bytes EXPECT_FAIL echo 1xx \> memory.memsw.limit_in_bytes }
@@ -84,7 +84,7 @@ test12() { memcg_require_memsw - echo 10M > memory.limit_in_bytes + ROD echo 10M \> memory.limit_in_bytes EXPECT_FAIL echo xx \> memory.memsw.limit_in_bytes }
testcases/kernel/controllers/memcg/functional/memcg_move_charge_at_immigrate_test.sh +2 -2
@@ -34,10 +34,10 @@ test_move_charge() return fi - echo $MEMCG_PROCESS_PID > subgroup_a/tasks + ROD echo $MEMCG_PROCESS_PID \> subgroup_a/tasks signal_memcg_process $total_size "subgroup_a/" - mkdir subgroup_b + ROD mkdir subgroup_b echo $move_charge_mask > subgroup_b/memory.move_charge_at_immigrate echo $MEMCG_PROCESS_PID > subgroup_b/tasks
testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh +5 -5
@@ -45,7 +45,7 @@ test5() local limit=$(memcg_adjust_limit_for_kmem $PAGESIZES) - mkdir subgroup + ROD mkdir subgroup echo $limit > memory.limit_in_bytes echo $((limit + PAGESIZES * 2)) > subgroup/memory.limit_in_bytes
@@ -61,9 +61,9 @@ test6() tst_res TINFO "Test hierarchical_memory_limit with disabling hierarchical accounting" memcg_require_hierarchy_disabled - echo 0 > memory.use_hierarchy + ROD echo 0 \> memory.use_hierarchy - mkdir subgroup + ROD mkdir subgroup echo $PAGESIZES > memory.limit_in_bytes echo $((PAGESIZES * 2)) > subgroup/memory.limit_in_bytes
@@ -81,7 +81,7 @@ test7() ROD echo 1 \> memory.use_hierarchy - mkdir subgroup + ROD mkdir subgroup echo $PAGESIZES > memory.limit_in_bytes echo $PAGESIZES > memory.memsw.limit_in_bytes echo $((PAGESIZES * 2)) > subgroup/memory.limit_in_bytes
@@ -102,7 +102,7 @@ test8() ROD echo 0 \> memory.use_hierarchy - mkdir subgroup + ROD mkdir subgroup echo $PAGESIZES > memory.limit_in_bytes echo $PAGESIZES > memory.memsw.limit_in_bytes echo $((PAGESIZES * 2)) > subgroup/memory.limit_in_bytes
testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh +1 -1
@@ -32,7 +32,7 @@ test_subgroup() limit_subgroup=$(memcg_adjust_limit_for_kmem $limit_subgroup) fi - mkdir subgroup + ROD mkdir subgroup echo $limit_parent > memory.limit_in_bytes echo $limit_subgroup > subgroup/memory.limit_in_bytes
testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh +5 -5
@@ -17,10 +17,10 @@ test1() local limit=$(memcg_adjust_limit_for_kmem $PAGESIZE) - echo 1 > memory.use_hierarchy - echo $limit > memory.limit_in_bytes + ROD echo 1 \> memory.use_hierarchy + ROD echo $limit \> memory.limit_in_bytes - mkdir subgroup + ROD mkdir subgroup cd subgroup test_proc_kill $((limit + PAGESIZE * 3)) "--mmap-lock1" $((limit + PAGESIZE * 2)) 0
@@ -34,7 +34,7 @@ test2() memcg_require_hierarchy_disabled - mkdir subgroup + ROD mkdir subgroup EXPECT_FAIL echo 1 \> memory.use_hierarchy rmdir subgroup
@@ -46,7 +46,7 @@ test3() memcg_require_hierarchy_disabled - echo 1 > memory.use_hierarchy + ROD echo 1 > memory.use_hierarchy mkdir subgroup EXPECT_FAIL echo 0 \> subgroup/memory.use_hierarchy
Signature · 1 line
2.30.2
PATCH v2 3/4 controllers/memcg: fail early to avoid possible false-positives
Signature · 2 lines
Thank you, Richard.
PATCH v2 4/4 controllers/memcg: check status of commands using interface
In reply to Krzysztof KozlowskiMost of the cgroups interface returns some valid and meaningful error code (e.g. ENOMEM if creating subgroup failed). Check these to be sure that all errors are caught, like in this example:
memcg_subgroup_charge 1 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed memcg_subgroup_charge 1 TINFO: Test that group and subgroup have no relationship /home/ubuntu/ltp-install/testcases/bin/memcg_subgroup_charge.sh: 36: echo: echo: I/O error memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 270336 memcg_subgroup_charge 1 TINFO: Warming up pid: 13496
memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 13496
--- .../controllers/memcg/functional/memcg_lib.sh | 2 +- .../memcg_max_usage_in_bytes_test.sh | 8 +++---- .../memcg/functional/memcg_stat_test.sh | 24 +++++++++---------- .../memcg/functional/memcg_subgroup_charge.sh | 4 ++-- .../functional/memcg_usage_in_bytes_test.sh | 4 ++-- 5 files changed, 21 insertions(+), 21 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_lib.sh +1 -1
@@ -380,7 +380,7 @@ test_limit_in_bytes() local use_memsw=$2 local elimit - echo $limit > memory.limit_in_bytes + EXPECT_PASS echo $limit \> memory.limit_in_bytes if [ $use_memsw -eq 1 ]; then memcg_require_memsw echo $limit > memory.memsw.limit_in_bytes
testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +4 -4
@@ -67,8 +67,8 @@ test2() tst_res TINFO "Test memory.memsw.max_usage_in_bytes" memcg_require_memsw - echo $MEM_LIMIT > memory.limit_in_bytes - echo $MEM_LIMIT > memory.memsw.limit_in_bytes + EXPECT_PASS echo $MEM_LIMIT \> memory.limit_in_bytes + EXPECT_PASS echo $MEM_LIMIT \> memory.memsw.limit_in_bytes test_max_usage_in_bytes 1 0 }
@@ -83,8 +83,8 @@ test4() tst_res TINFO "Test reset memory.memsw.max_usage_in_bytes" memcg_require_memsw - echo $MEM_LIMIT > memory.limit_in_bytes - echo $MEM_LIMIT > memory.memsw.limit_in_bytes + EXPECT_PASS echo $MEM_LIMIT \> memory.limit_in_bytes + EXPECT_PASS echo $MEM_LIMIT \> memory.memsw.limit_in_bytes test_max_usage_in_bytes 1 1 }
testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh +12 -12
@@ -46,8 +46,8 @@ test5() local limit=$(memcg_adjust_limit_for_kmem $PAGESIZES) ROD mkdir subgroup - echo $limit > memory.limit_in_bytes - echo $((limit + PAGESIZES * 2)) > subgroup/memory.limit_in_bytes + EXPECT_PASS echo $limit \> memory.limit_in_bytes + EXPECT_PASS echo $((limit + PAGESIZES * 2)) \> subgroup/memory.limit_in_bytes cd subgroup check_mem_stat "hierarchical_memory_limit" $limit
@@ -64,8 +64,8 @@ test6() ROD echo 0 \> memory.use_hierarchy ROD mkdir subgroup - echo $PAGESIZES > memory.limit_in_bytes - echo $((PAGESIZES * 2)) > subgroup/memory.limit_in_bytes + EXPECT_PASS echo $PAGESIZES \> memory.limit_in_bytes + EXPECT_PASS echo $((PAGESIZES * 2)) \> subgroup/memory.limit_in_bytes cd subgroup check_mem_stat "hierarchical_memory_limit" $((PAGESIZES * 2))
@@ -82,10 +82,10 @@ test7() ROD echo 1 \> memory.use_hierarchy ROD mkdir subgroup - echo $PAGESIZES > memory.limit_in_bytes - echo $PAGESIZES > memory.memsw.limit_in_bytes - echo $((PAGESIZES * 2)) > subgroup/memory.limit_in_bytes - echo $((PAGESIZES * 2)) > subgroup/memory.memsw.limit_in_bytes + EXPECT_PASS echo $PAGESIZES \> memory.limit_in_bytes + EXPECT_PASS echo $PAGESIZES \> memory.memsw.limit_in_bytes + EXPECT_PASS echo $((PAGESIZES * 2)) \> subgroup/memory.limit_in_bytes + EXPECT_PASS echo $((PAGESIZES * 2)) \> subgroup/memory.memsw.limit_in_bytes cd subgroup check_mem_stat "hierarchical_memsw_limit" $PAGESIZES
@@ -103,10 +103,10 @@ test8() ROD echo 0 \> memory.use_hierarchy ROD mkdir subgroup - echo $PAGESIZES > memory.limit_in_bytes - echo $PAGESIZES > memory.memsw.limit_in_bytes - echo $((PAGESIZES * 2)) > subgroup/memory.limit_in_bytes - echo $((PAGESIZES * 2)) > subgroup/memory.memsw.limit_in_bytes + EXPECT_PASS echo $PAGESIZES \> memory.limit_in_bytes + EXPECT_PASS echo $PAGESIZES \> memory.memsw.limit_in_bytes + EXPECT_PASS echo $((PAGESIZES * 2)) \> subgroup/memory.limit_in_bytes + EXPECT_PASS echo $((PAGESIZES * 2)) \> subgroup/memory.memsw.limit_in_bytes cd subgroup check_mem_stat "hierarchical_memsw_limit" $((PAGESIZES * 2))
testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh +2 -2
@@ -33,8 +33,8 @@ test_subgroup() fi ROD mkdir subgroup - echo $limit_parent > memory.limit_in_bytes - echo $limit_subgroup > subgroup/memory.limit_in_bytes + EXPECT_PASS echo $limit_parent \> memory.limit_in_bytes + EXPECT_PASS echo $limit_subgroup \> subgroup/memory.limit_in_bytes start_memcg_process --mmap-anon -s $MEM_TO_ALLOC
testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh +2 -2
@@ -28,8 +28,8 @@ test2() tst_res TINFO "Test memory.memsw.usage_in_bytes" memcg_require_memsw - echo $MEM_LIMIT > memory.limit_in_bytes - echo $MEM_LIMIT > memory.memsw.limit_in_bytes + EXPECT_PASS echo $MEM_LIMIT \> memory.limit_in_bytes + EXPECT_PASS echo $MEM_LIMIT \> memory.memsw.limit_in_bytes test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \ "memory.memsw.usage_in_bytes" $MEM_TO_ALLOC \ $MEM_EXPECTED_UPPER false
Signature · 1 line
2.30.2
PATCH v2 4/4 controllers/memcg: check status of commands using interface
Signature · 2 lines
Thank you, Richard.
v2 cover controllers/memcg: fixes for CPU nodes on newer kernels
In reply to Krzysztof KozlowskiHi!
Patchset applied, thanks.
Signature · 2 lines
Cyril Hrubis [email protected]