controllers/memcg: fixes for newer kernels
v4 cover controllers/memcg: fixes for newer kernels
Hi,
Changes since v3:
1. Patch 4/5: Include per-CPU memory as well in group limits.
Changes since v2:
1. Patch 1/5: Define MEM_USAGE_RANGE and explain it.
2. Patch 3/5: Use PAGESIZES to explain it is the batch accounting.
3. Patch 4/5: Add to this patchset (previously sent separately, no dependencies on others).
4. Patch 5/5: New patch, fixes additional difference, usually of 2-3 pages coming from the kernel.
See also reply from Michal Hocko:
https://lore.kernel.org/linux-mm/[email protected]/T/#m6459b3be3a86f5eaf2cfc48dd586b6faf949e440
v2: https://lists.linux.it/pipermail/ltp/2021-June/023259.html
Best regards,
Krzysztof
Krzysztof Kozlowski (5):
controllers/memcg: accept range of max_usage_in_bytes controllers/memcg: accept range of usage_in_bytes controllers/memcg: accept non-zero max_usage_in_bytes after reset controllers/memcg: increase memory limit in subgroup charge controllers/memcg: offset kernel memory
.../controllers/memcg/functional/memcg_lib.sh | 41 ++++++++++++++++--- .../memcg_max_usage_in_bytes_test.sh | 18 +++++++- .../memcg/functional/memcg_stat_rss.sh | 20 ++++----- .../memcg/functional/memcg_stat_test.sh | 8 ++-- .../memcg/functional/memcg_subgroup_charge.sh | 33 +++++++++++---- .../functional/memcg_usage_in_bytes_test.sh | 7 +++- 6 files changed, 96 insertions(+), 31 deletions(-)
Signature · 1 line
2.27.0
PATCH v4 1/5 controllers/memcg: accept range of max_usage_in_bytes
Several Linux kernel versions report higher max_usage_in_bytes than expected size of 1024 pages. For example v5.4, v5.8, v5.10 and 5.13.0-rc5:
memcg_max_usage_in_bytes_test 1 TINFO: Test memory.max_usage_in_bytes memcg_max_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304 memcg_max_usage_in_bytes_test 1 TINFO: Warming up pid: 1393215 memcg_max_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 1393215 memcg_max_usage_in_bytes_test 1 TFAIL: memory.max_usage_in_bytes is 4325376, 4194304 expected
It seems that recent Linux kernel caches the statistics more aggressively (especially on multi-CPU systems) and the batch updates of 32 pages are visible in usage_in_bytes.
--- .../controllers/memcg/functional/memcg_lib.sh | 20 +++++++++++++++++-- .../memcg_max_usage_in_bytes_test.sh | 3 ++- 2 files changed, 20 insertions(+), 3 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_lib.sh +18 -2
@@ -27,6 +27,15 @@ fi # Post 4.16 kernel updates stat in batch (> 32 pages) every time PAGESIZES=$(($PAGESIZE * 33)) +# On recent Linux kernels (at least v5.4) updating stats happens in batches +# (PAGESIZES) and also might depend on workload and number of CPUs. The kernel +# caches the data and does not prioritize stats precision. This is especially +# visible for max_usage_in_bytes where it usually exceeds +# actual memory allocation. +# When checking for usage_in_bytes and max_usage_in_bytes accept also higher values +# from given range: +MEM_USAGE_RANGE=$((PAGESIZES)) + HUGEPAGESIZE=$(awk '/Hugepagesize/ {print $2}' /proc/meminfo) [ -z $HUGEPAGESIZE ] && HUGEPAGESIZE=0 HUGEPAGESIZE=$(($HUGEPAGESIZE * 1024))
@@ -140,7 +149,8 @@ shmmax_cleanup() # Check size in memcg # $1 - Item name -# $2 - Expected size +# $2 - Expected size lower bound +# $3 - Expected size upper bound (optional) check_mem_stat() { local item_size
@@ -151,7 +161,13 @@ check_mem_stat() item_size=$(grep -w $1 memory.stat | cut -d " " -f 2) fi - if [ "$2" = "$item_size" ]; then + if [ "$3" ]; then + if [ $item_size -ge $2 ] && [ $item_size -le $3 ]; then + tst_res TPASS "$1 is ${2}-${3} as expected" + else + tst_res TFAIL "$1 is $item_size, ${2}-${3} expected" + fi + elif [ "$2" = "$item_size" ]; then tst_res TPASS "$1 is $2 as expected" else tst_res TFAIL "$1 is $item_size, $2 expected"
testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +2 -1
@@ -12,6 +12,7 @@ TST_CNT=4 . memcg_lib.sh MEM_TO_ALLOC=$((PAGESIZE * 1024)) +MEM_EXPECTED_UPPER=$((MEM_TO_ALLOC + MEM_USAGE_RANGE)) MEM_LIMIT=$((MEM_TO_ALLOC * 2)) # Run test cases which checks memory.[memsw.]max_usage_in_bytes after make
@@ -32,7 +33,7 @@ test_max_usage_in_bytes() signal_memcg_process $MEM_TO_ALLOC signal_memcg_process $MEM_TO_ALLOC - check_mem_stat $item $MEM_TO_ALLOC + check_mem_stat $item $MEM_TO_ALLOC $MEM_EXPECTED_UPPER if [ $check_after_reset -eq 1 ]; then echo 0 > $item
Signature · 1 line
2.27.0
PATCH v4 2/5 controllers/memcg: accept range of usage_in_bytes
In reply to Krzysztof KozlowskiSeveral Linux kernel versions report higher usage_in_bytes than expected size of 1024 pages. For example v5.4, v5.8, v5.10 and 5.13.0-rc5:
memcg_usage_in_bytes_test 1 TINFO: Test memory.usage_in_bytes memcg_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304 memcg_usage_in_bytes_test 1 TINFO: Warming up pid: 1160 memcg_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 1160 memcg_usage_in_bytes_test 1 TFAIL: memory.usage_in_bytes is 4325376, 4194304 expected
It seems that recent Linux kernel caches the statistics more aggressively (especially on multi-CPU systems) and the batch updates of 32 pages are visible in usage_in_bytes.
--- .../controllers/memcg/functional/memcg_lib.sh | 11 +++++++--- .../memcg/functional/memcg_stat_rss.sh | 20 +++++++++---------- .../memcg/functional/memcg_stat_test.sh | 8 ++++---- .../functional/memcg_usage_in_bytes_test.sh | 7 +++++-- 4 files changed, 27 insertions(+), 19 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_lib.sh +8 -3
@@ -248,8 +248,9 @@ test_mem_stat() local size=$2 local total_size=$3 local stat_name=$4 - local exp_stat_size=$5 - local check_after_free=$6 + local exp_stat_size_low=$5 + local exp_stat_size_up=$6 + local check_after_free=$7 start_memcg_process $memtypes -s $size
@@ -260,7 +261,11 @@ test_mem_stat() echo $MEMCG_PROCESS_PID > tasks signal_memcg_process $size - check_mem_stat $stat_name $exp_stat_size + if [ "$exp_stat_size_low" = "$exp_stat_size_up" ]; then + check_mem_stat $stat_name $exp_stat_size_low + else + check_mem_stat $stat_name $exp_stat_size_low $exp_stat_size_up + fi signal_memcg_process $size if $check_after_free; then
testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh +10 -10
@@ -18,54 +18,54 @@ TST_CNT=10 # Test the management and counting of memory test1() { - test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES false + test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES false } test2() { - test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 false + test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 0 false } test3() { - test_mem_stat "--shm -k 3" $PAGESIZE $PAGESIZE "rss" 0 false + test_mem_stat "--shm -k 3" $PAGESIZE $PAGESIZE "rss" 0 0 false } test4() { test_mem_stat "--mmap-anon --mmap-file --shm" \ - $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES false + $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES $PAGESIZES false } test5() { - test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES false + test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES false } test6() { - test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES true + test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES true } test7() { - test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 true + test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 0 true } test8() { - test_mem_stat "--shm -k 8" $PAGESIZE $PAGESIZE "rss" 0 true + test_mem_stat "--shm -k 8" $PAGESIZE $PAGESIZE "rss" 0 0 true } test9() { test_mem_stat "--mmap-anon --mmap-file --shm" \ - $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES true + $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES $PAGESIZES true } test10() { - test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES true + test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES true } tst_run
testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh +4 -4
@@ -14,28 +14,28 @@ TST_CNT=8 test1() { tst_res TINFO "Test cache" - test_mem_stat "--shm -k 3" $PAGESIZES $PAGESIZES "cache" $PAGESIZES false + test_mem_stat "--shm -k 3" $PAGESIZES $PAGESIZES "cache" $PAGESIZES $PAGESIZES false } test2() { tst_res TINFO "Test mapped_file" test_mem_stat "--mmap-file" $PAGESIZES $PAGESIZES \ - "mapped_file" $PAGESIZES false + "mapped_file" $PAGESIZES $PAGESIZES false } test3() { tst_res TINFO "Test unevictable with MAP_LOCKED" test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES \ - "unevictable" $PAGESIZES false + "unevictable" $PAGESIZES $PAGESIZES false } test4() { tst_res TINFO "Test unevictable with mlock" test_mem_stat "--mmap-lock2" $PAGESIZES $PAGESIZES \ - "unevictable" $PAGESIZES false + "unevictable" $PAGESIZES $PAGESIZES false } test5()
testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh +5 -2
@@ -12,13 +12,15 @@ TST_CNT=2 . memcg_lib.sh MEM_TO_ALLOC=$((PAGESIZE * 1024)) +MEM_EXPECTED_UPPER=$((MEM_TO_ALLOC + MEM_USAGE_RANGE)) MEM_LIMIT=$((MEM_TO_ALLOC * 2)) test1() { tst_res TINFO "Test memory.usage_in_bytes" test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \ - "memory.usage_in_bytes" $MEM_TO_ALLOC false + "memory.usage_in_bytes" $MEM_TO_ALLOC \ + $MEM_EXPECTED_UPPER false } test2()
@@ -29,7 +31,8 @@ test2() echo $MEM_LIMIT > memory.limit_in_bytes echo $MEM_LIMIT > memory.memsw.limit_in_bytes test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \ - "memory.memsw.usage_in_bytes" $MEM_TO_ALLOC false + "memory.memsw.usage_in_bytes" $MEM_TO_ALLOC \ + $MEM_EXPECTED_UPPER false } tst_run
Signature · 1 line
2.27.0
PATCH v4 3/5 controllers/memcg: accept non-zero max_usage_in_bytes after reset
In reply to Krzysztof KozlowskiSeveral Linux kernel versions report a non-zero max_usage_in_bytes after resetting the counter. For example v5.4, v5.8, v5.10, v5.11, v5.12 and 5.13.0-rc5:
memcg_max_usage_in_bytes_test 4 TINFO: Test reset memory.memsw.max_usage_in_bytes memcg_max_usage_in_bytes_test 4 TINFO: Running memcg_process --mmap-anon -s 4194304 memcg_max_usage_in_bytes_test 4 TINFO: Warming up pid: 1416 memcg_max_usage_in_bytes_test 4 TINFO: Process is still here after warm up: 1416 memcg_max_usage_in_bytes_test 4 TFAIL: memory.memsw.max_usage_in_bytes is 4325376, 4194304 expected memcg_max_usage_in_bytes_test 4 TFAIL: memory.memsw.max_usage_in_bytes is 122880, 0 expected
It seems that recent Linux kernel caches the statistics more aggressively (especially on multi-CPU systems) and the batch updates of 32 pages are visible in usage_in_bytes.
--- .../memcg/functional/memcg_max_usage_in_bytes_test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +1 -1
@@ -37,7 +37,7 @@ test_max_usage_in_bytes() if [ $check_after_reset -eq 1 ]; then echo 0 > $item - check_mem_stat $item 0 + check_mem_stat $item 0 $PAGESIZES fi stop_memcg_process
Signature · 1 line
2.27.0
PATCH v4 4/5 controllers/memcg: increase memory limit in subgroup charge
In reply to Krzysztof KozlowskiThe memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases with:
memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168 memcg_subgroup_charge 1 TINFO: Warming up pid: 19289 memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289 memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected memcg_subgroup_charge 1 TPASS: rss is 0 as expected
In dmesg one could see that OOM killer killed the process even though group memory limit was matching the usage:
memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
...
memory: usage 132kB, limit 132kB, failcnt 9
memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
...
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 19289] 0 19289 669 389 40960 0 0 memcg_process
oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kBIt seems actual kernel memory usage by a given group depends on number of CPUs, where at least one page is allocated per-cpu beside regular (expected) allocation. Fix the test on machines with more CPUs by including this per-CPU memory in group limits, plus some slack of 4 PAGES. Increase also memory allocation from 32 to 64 pages to be more distinctive from kernel per-CPU memory.
--- .../memcg/functional/memcg_subgroup_charge.sh | 33 ++++++++++++++----- 1 file changed, 25 insertions(+), 8 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh +25 -8
@@ -14,16 +14,33 @@ TST_CNT=3 . memcg_lib.sh +# Allocate memory bigger than per-cpu kernel memory +MEM_TO_ALLOC=$((PAGESIZES * 2)) + # Test the memory charge won't move to subgroup # $1 - memory.limit_in_bytes in parent group # $2 - memory.limit_in_bytes in sub group test_subgroup() { + local limit_parent=$1 + local limit_subgroup=$2 + local total_cpus=`tst_ncpus` + + # Kernel memory allocated for the process is also charged. + # It might depend on the number of CPUs. For example on kernel v5.11 + # additionally total_cpus plus 1-2 pages are charged to the group. + if [ $limit_parent -ne 0 ]; then + limit_parent=$((limit_parent + 4 * PAGESIZE + total_cpus * PAGESIZE)) + fi + if [ $limit_subgroup -ne 0 ]; then + limit_subgroup=$((limit_subgroup + 4 * PAGESIZE + total_cpus * PAGESIZE)) + fi + mkdir subgroup - echo $1 > memory.limit_in_bytes - echo $2 > subgroup/memory.limit_in_bytes + echo $limit_parent > memory.limit_in_bytes + echo $limit_subgroup > subgroup/memory.limit_in_bytes - start_memcg_process --mmap-anon -s $PAGESIZES + start_memcg_process --mmap-anon -s $MEM_TO_ALLOC warmup if [ $? -ne 0 ]; then
@@ -31,8 +48,8 @@ test_subgroup() fi echo $MEMCG_PROCESS_PID > tasks - signal_memcg_process $PAGESIZES - check_mem_stat "rss" $PAGESIZES + signal_memcg_process $MEM_TO_ALLOC + check_mem_stat "rss" $MEM_TO_ALLOC cd subgroup echo $MEMCG_PROCESS_PID > tasks
@@ -47,17 +64,17 @@ test_subgroup() test1() { tst_res TINFO "Test that group and subgroup have no relationship" - test_subgroup $PAGESIZES $((2 * PAGESIZES)) + test_subgroup $MEM_TO_ALLOC $((2 * MEM_TO_ALLOC)) } test2() { - test_subgroup $PAGESIZES $PAGESIZES + test_subgroup $MEM_TO_ALLOC $MEM_TO_ALLOC } test3() { - test_subgroup $PAGESIZES 0 + test_subgroup $MEM_TO_ALLOC 0 } tst_run
Signature · 1 line
2.27.0
PATCH v4 4/5 controllers/memcg: increase memory limit in subgroup charge
Hello Krzysztof,
Quoted 27 lines · Krzysztof Kozlowski
The
memcg_subgroup_chargewas failing on kernel v5.8 in around 10% cases with:
memcg_subgroup_charge1 TINFO: Runningmemcg_process--mmap-anon -s 135168memcg_subgroup_charge1 TINFO: Warming up pid: 19289memcg_subgroup_charge1 TINFO: Process is still here after warm up: 19289memcg_subgroup_charge1 TFAIL: rss is 0, 135168 expectedmemcg_subgroup_charge1 TPASS: rss is 0 as expectedIn dmesg one could see that OOM killer killed the process even though group memory limit was matching the usage:
memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0 CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020 ... memory: usage 132kB, limit 132kB, failcnt 9 memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0 kmem: usage 4kB, limit 9007199254740988kB, failcnt 0 ... Tasks state (memory values in pages): [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 19289] 0 19289 669 389 40960 0 0 memcg_process oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0 Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0 oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
Quoted 6 lines
It seems actual kernel memory usage by a given group depends on number of CPUs, where at least one page is allocated per-cpu beside regular (expected) allocation. Fix the test on machines with more CPUs by including this per-CPU memory in group limits, plus some slack of 4 PAGES. Increase also memory allocation from 32 to 64 pages to be more distinctive from kernel per-CPU memory.
Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
Quoted 17 lines
Signed-off-by Krzysztof Kozlowski <[email protected]>--- .../memcg/functional/memcg_subgroup_charge.sh | 33 ++++++++++++++----- 1 file changed, 25 insertions(+), 8 deletions(-)testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh +3
-0@@ -14,16 +14,33 @@ TST_CNT=3 . memcg_lib.sh +# Allocate memory bigger than per-cpu kernel memory +MEM_TO_ALLOC=$((PAGESIZES * 2)) +
Signature · 2 lines
Thank you, Richard.
PATCH v4 4/5 controllers/memcg: increase memory limit in subgroup charge
Quoted 31 lines · Richard Palethorpe
Hello Krzysztof,
Quoted 27 lines · Krzysztof Kozlowski
The
memcg_subgroup_chargewas failing on kernel v5.8 in around 10% cases with:
memcg_subgroup_charge1 TINFO: Runningmemcg_process--mmap-anon -s 135168memcg_subgroup_charge1 TINFO: Warming up pid: 19289memcg_subgroup_charge1 TINFO: Process is still here after warm up: 19289memcg_subgroup_charge1 TFAIL: rss is 0, 135168 expectedmemcg_subgroup_charge1 TPASS: rss is 0 as expectedIn dmesg one could see that OOM killer killed the process even though group memory limit was matching the usage:
memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0 CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020 ... memory: usage 132kB, limit 132kB, failcnt 9 memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0 kmem: usage 4kB, limit 9007199254740988kB, failcnt 0 ... Tasks state (memory values in pages): [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 19289] 0 19289 669 389 40960 0 0 memcg_process oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0 Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0 oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
Quoted 9 lines
Quoted 6 lines
It seems actual kernel memory usage by a given group depends on number of CPUs, where at least one page is allocated per-cpu beside regular (expected) allocation. Fix the test on machines with more CPUs by including this per-CPU memory in group limits, plus some slack of 4 PAGES. Increase also memory allocation from 32 to 64 pages to be more distinctive from kernel per-CPU memory.
Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
Yes, right. Maybe this could be fixed when applying - no need for resend.
Best regards,
Krzysztof
PATCH v4 4/5 controllers/memcg: increase memory limit in subgroup charge
Hello,
Quoted 39 lines · Krzysztof Kozlowski
Quoted 38 lines · Richard Palethorpe
Hello Krzysztof,
Quoted 33 lines · Krzysztof Kozlowski
The
memcg_subgroup_chargewas failing on kernel v5.8 in around 10% cases with:
memcg_subgroup_charge1 TINFO: Runningmemcg_process--mmap-anon -s 135168memcg_subgroup_charge1 TINFO: Warming up pid: 19289memcg_subgroup_charge1 TINFO: Process is still here after warm up: 19289memcg_subgroup_charge1 TFAIL: rss is 0, 135168 expectedmemcg_subgroup_charge1 TPASS: rss is 0 as expectedIn dmesg one could see that OOM killer killed the process even though group memory limit was matching the usage:
memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0 CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020 ... memory: usage 132kB, limit 132kB, failcnt 9 memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0 kmem: usage 4kB, limit 9007199254740988kB, failcnt 0 ... Tasks state (memory values in pages): [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 19289] 0 19289 669 389 40960 0 0 memcg_process oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0 Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0 oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kBIt seems actual kernel memory usage by a given group depends on number of CPUs, where at least one page is allocated per-cpu beside regular (expected) allocation. Fix the test on machines with more CPUs by including this per-CPU memory in group limits, plus some slack of 4 PAGES. Increase also memory allocation from 32 to 64 pages to be more distinctive from kernel per-CPU memory.
Quoted 8 lines
Quoted 2 lines
Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
Yes, right. Maybe this could be fixed when applying - no need for resend.
Best regards,
Krzysztof
Well I am more than happy with the patchset.
Signature · 2 lines
Thank you, Richard.
PATCH v4 4/5 controllers/memcg: increase memory limit in subgroup charge
Hi Krzysztof, Richard,
Quoted 6 lines · Richard Palethorpe
Quoted 5 lines
Quoted 3 lines
Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
Yes, right. Maybe this could be fixed when applying - no need for resend.
I help correct that and applied patchset, thanks for all your work on this.
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20210714/3f229093/attachment.htm>
PATCH v4 5/5 controllers/memcg: offset kernel memory
In reply to Krzysztof KozlowskiThe memory allocated by kernel with __GFP_ACCOUNT is counted as well in memory group usage limits. Add it to fix failures like:
memcg_max_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304
memcg_max_usage_in_bytes_test 1 TINFO: Warming up pid: 925811
memcg_max_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 925811
memcg_max_usage_in_bytes_test 1 TFAIL: memory.max_usage_in_bytes is 4333568, 4194304-4325376 expected
(where memory.kmem.max_usage_in_bytes:8192)--- .../controllers/memcg/functional/memcg_lib.sh | 10 ++++++++++ .../functional/memcg_max_usage_in_bytes_test.sh | 15 ++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-)
testcases/kernel/controllers/memcg/functional/memcg_lib.sh +10 -0
@@ -251,6 +251,7 @@ test_mem_stat() local exp_stat_size_low=$5 local exp_stat_size_up=$6 local check_after_free=$7 + local kmem_stat_name="${stat_name##*.}" start_memcg_process $memtypes -s $size
@@ -261,6 +262,15 @@ test_mem_stat() echo $MEMCG_PROCESS_PID > tasks signal_memcg_process $size + if [ "$kmem_stat_name" = "max_usage_in_bytes" ] || + [ "$kmem_stat_name" = "usage_in_bytes" ]; then + local kmem=$(cat "memory.kmem.${kmem_stat_name}") + if [ $? -eq 0 ]; then + exp_stat_size_low=$((exp_stat_size_low + kmem)) + exp_stat_size_up=$((exp_stat_size_up + kmem)) + fi + fi + if [ "$exp_stat_size_low" = "$exp_stat_size_up" ]; then check_mem_stat $stat_name $exp_stat_size_low else
testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +14 -1
@@ -22,6 +22,10 @@ test_max_usage_in_bytes() local item="memory.max_usage_in_bytes" [ $1 -eq 1 ] && item="memory.memsw.max_usage_in_bytes" local check_after_reset=$2 + local exp_stat_size_low=$MEM_TO_ALLOC + local exp_stat_size_up=$MEM_EXPECTED_UPPER + local kmem_stat_name="${item##*.}" + start_memcg_process --mmap-anon -s $MEM_TO_ALLOC warmup
@@ -33,7 +37,16 @@ test_max_usage_in_bytes() signal_memcg_process $MEM_TO_ALLOC signal_memcg_process $MEM_TO_ALLOC - check_mem_stat $item $MEM_TO_ALLOC $MEM_EXPECTED_UPPER + if [ "$kmem_stat_name" = "max_usage_in_bytes" ] || + [ "$kmem_stat_name" = "usage_in_bytes" ]; then + local kmem=$(cat "memory.kmem.${kmem_stat_name}") + if [ $? -eq 0 ]; then + exp_stat_size_low=$((exp_stat_size_low + kmem)) + exp_stat_size_up=$((exp_stat_size_up + kmem)) + fi + fi + + check_mem_stat $item $exp_stat_size_low $exp_stat_size_up if [ $check_after_reset -eq 1 ]; then echo 0 > $item
Signature · 1 line
2.27.0