Skip to contentSkip to conversation

controllers/memcg: fixes for newer kernels

PATCH New Krzysztof Kozlowski 3 participants 10 messages Jul 13, 2021 – Jul 14, 2021

lore
Share & tools
lore
#0 lore raw

v4 cover controllers/memcg: fixes for newer kernels

Hi,

Changes since v3:
1. Patch 4/5: Include per-CPU memory as well in group limits.

Changes since v2:
1. Patch 1/5: Define MEM_USAGE_RANGE and explain it.
2. Patch 3/5: Use PAGESIZES to explain it is the batch accounting.
3. Patch 4/5: Add to this patchset (previously sent separately, no dependencies on others).
4. Patch 5/5: New patch, fixes additional difference, usually of 2-3 pages coming from the kernel.

See also reply from Michal Hocko:
https://lore.kernel.org/linux-mm/[email protected]/T/#m6459b3be3a86f5eaf2cfc48dd586b6faf949e440

v2: https://lists.linux.it/pipermail/ltp/2021-June/023259.html

Best regards,
Krzysztof

Krzysztof Kozlowski (5):
controllers/memcg: accept range of max_usage_in_bytes controllers/memcg: accept range of usage_in_bytes controllers/memcg: accept non-zero max_usage_in_bytes after reset controllers/memcg: increase memory limit in subgroup charge controllers/memcg: offset kernel memory

 .../controllers/memcg/functional/memcg_lib.sh | 41 ++++++++++++++++---
 .../memcg_max_usage_in_bytes_test.sh          | 18 +++++++-
 .../memcg/functional/memcg_stat_rss.sh        | 20 ++++-----
 .../memcg/functional/memcg_stat_test.sh       |  8 ++--
 .../memcg/functional/memcg_subgroup_charge.sh | 33 +++++++++++----
 .../functional/memcg_usage_in_bytes_test.sh   |  7 +++-
 6 files changed, 96 insertions(+), 31 deletions(-)
Signature · 1 line
2.27.0
#1 lore raw

PATCH v4 1/5 controllers/memcg: accept range of max_usage_in_bytes

Several Linux kernel versions report higher max_usage_in_bytes than expected size of 1024 pages. For example v5.4, v5.8, v5.10 and 5.13.0-rc5:

memcg_max_usage_in_bytes_test 1 TINFO: Test memory.max_usage_in_bytes memcg_max_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304 memcg_max_usage_in_bytes_test 1 TINFO: Warming up pid: 1393215 memcg_max_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 1393215 memcg_max_usage_in_bytes_test 1 TFAIL: memory.max_usage_in_bytes is 4325376, 4194304 expected

It seems that recent Linux kernel caches the statistics more aggressively (especially on multi-CPU systems) and the batch updates of 32 pages are visible in usage_in_bytes.

---
 .../controllers/memcg/functional/memcg_lib.sh | 20 +++++++++++++++++--
 .../memcg_max_usage_in_bytes_test.sh          |  3 ++-
 2 files changed, 20 insertions(+), 3 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_lib.sh +18 -2
index d9bb6d94324d..4c47e63745ff 100755
@@ -27,6 +27,15 @@ fi
 # Post 4.16 kernel updates stat in batch (> 32 pages) every time
 PAGESIZES=$(($PAGESIZE * 33))
 
+# On recent Linux kernels (at least v5.4) updating stats happens in batches
+# (PAGESIZES) and also might depend on workload and number of CPUs.  The kernel
+# caches the data and does not prioritize stats precision.  This is especially
+# visible for max_usage_in_bytes where it usually exceeds
+# actual memory allocation.
+# When checking for usage_in_bytes and max_usage_in_bytes accept also higher values
+# from given range:
+MEM_USAGE_RANGE=$((PAGESIZES))
+
 HUGEPAGESIZE=$(awk '/Hugepagesize/ {print $2}' /proc/meminfo)
 [ -z $HUGEPAGESIZE ] && HUGEPAGESIZE=0
 HUGEPAGESIZE=$(($HUGEPAGESIZE * 1024))
@@ -140,7 +149,8 @@ shmmax_cleanup()
 
 # Check size in memcg
 # $1 - Item name
-# $2 - Expected size
+# $2 - Expected size lower bound
+# $3 - Expected size upper bound (optional)
 check_mem_stat()
 {
 	local item_size
@@ -151,7 +161,13 @@ check_mem_stat()
 		item_size=$(grep -w $1 memory.stat | cut -d " " -f 2)
 	fi
 
-	if [ "$2" = "$item_size" ]; then
+	if [ "$3" ]; then
+		if [ $item_size -ge $2 ] && [ $item_size -le $3 ]; then
+			tst_res TPASS "$1 is ${2}-${3} as expected"
+		else
+			tst_res TFAIL "$1 is $item_size, ${2}-${3} expected"
+		fi
+	elif [ "$2" = "$item_size" ]; then
 		tst_res TPASS "$1 is $2 as expected"
 	else
 		tst_res TFAIL "$1 is $item_size, $2 expected"
testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +2 -1
index 14daa4651798..8831f1937070 100755
@@ -12,6 +12,7 @@ TST_CNT=4
 . memcg_lib.sh
 
 MEM_TO_ALLOC=$((PAGESIZE * 1024))
+MEM_EXPECTED_UPPER=$((MEM_TO_ALLOC + MEM_USAGE_RANGE))
 MEM_LIMIT=$((MEM_TO_ALLOC * 2))
 
 # Run test cases which checks memory.[memsw.]max_usage_in_bytes after make
@@ -32,7 +33,7 @@ test_max_usage_in_bytes()
 	signal_memcg_process $MEM_TO_ALLOC
 	signal_memcg_process $MEM_TO_ALLOC
 
-	check_mem_stat $item $MEM_TO_ALLOC
+	check_mem_stat $item $MEM_TO_ALLOC $MEM_EXPECTED_UPPER
 
 	if [ $check_after_reset -eq 1 ]; then
 		echo 0 > $item
Signature · 1 line
2.27.0
#2 lore raw

PATCH v4 2/5 controllers/memcg: accept range of usage_in_bytes

In reply to Krzysztof Kozlowski

Several Linux kernel versions report higher usage_in_bytes than expected size of 1024 pages. For example v5.4, v5.8, v5.10 and 5.13.0-rc5:

memcg_usage_in_bytes_test 1 TINFO: Test memory.usage_in_bytes memcg_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304 memcg_usage_in_bytes_test 1 TINFO: Warming up pid: 1160 memcg_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 1160 memcg_usage_in_bytes_test 1 TFAIL: memory.usage_in_bytes is 4325376, 4194304 expected

It seems that recent Linux kernel caches the statistics more aggressively (especially on multi-CPU systems) and the batch updates of 32 pages are visible in usage_in_bytes.

---
 .../controllers/memcg/functional/memcg_lib.sh | 11 +++++++---
 .../memcg/functional/memcg_stat_rss.sh        | 20 +++++++++----------
 .../memcg/functional/memcg_stat_test.sh       |  8 ++++----
 .../functional/memcg_usage_in_bytes_test.sh   |  7 +++++--
 4 files changed, 27 insertions(+), 19 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_lib.sh +8 -3
index 4c47e63745ff..9634eb7e9f42 100755
@@ -248,8 +248,9 @@ test_mem_stat()
 	local size=$2
 	local total_size=$3
 	local stat_name=$4
-	local exp_stat_size=$5
-	local check_after_free=$6
+	local exp_stat_size_low=$5
+	local exp_stat_size_up=$6
+	local check_after_free=$7
 
 	start_memcg_process $memtypes -s $size
 
@@ -260,7 +261,11 @@ test_mem_stat()
 	echo $MEMCG_PROCESS_PID > tasks
 	signal_memcg_process $size
 
-	check_mem_stat $stat_name $exp_stat_size
+	if [ "$exp_stat_size_low" = "$exp_stat_size_up" ]; then
+		check_mem_stat $stat_name $exp_stat_size_low
+	else
+		check_mem_stat $stat_name $exp_stat_size_low $exp_stat_size_up
+	fi
 
 	signal_memcg_process $size
 	if $check_after_free; then
testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh +10 -10
index 1a6128a6dba8..d9b4ec287b5f 100755
@@ -18,54 +18,54 @@ TST_CNT=10
 # Test the management and counting of memory
 test1()
 {
-	test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES false
+	test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES false
 }
 
 test2()
 {
-	test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 false
+	test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 0 false
 }
 
 test3()
 {
-	test_mem_stat "--shm -k 3" $PAGESIZE $PAGESIZE "rss" 0 false
+	test_mem_stat "--shm -k 3" $PAGESIZE $PAGESIZE "rss" 0 0 false
 }
 
 test4()
 {
 	test_mem_stat "--mmap-anon --mmap-file --shm" \
-		$PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES false
+		$PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES $PAGESIZES false
 }
 
 test5()
 {
-	test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES false
+	test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES false
 }
 
 test6()
 {
-	test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES true
+	test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES true
 }
 
 test7()
 {
-	test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 true
+	test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 0 true
 }
 
 test8()
 {
-	test_mem_stat "--shm -k 8" $PAGESIZE $PAGESIZE "rss" 0 true
+	test_mem_stat "--shm -k 8" $PAGESIZE $PAGESIZE "rss" 0 0 true
 }
 
 test9()
 {
 	test_mem_stat "--mmap-anon --mmap-file --shm" \
-		$PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES true
+		$PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES $PAGESIZES true
 }
 
 test10()
 {
-	test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES true
+	test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES true
 }
 
 tst_run
testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh +4 -4
index 925c4ecf87bc..e5eb7e5d0001 100755
@@ -14,28 +14,28 @@ TST_CNT=8
 test1()
 {
 	tst_res TINFO "Test cache"
-	test_mem_stat "--shm -k 3" $PAGESIZES $PAGESIZES "cache" $PAGESIZES false
+	test_mem_stat "--shm -k 3" $PAGESIZES $PAGESIZES "cache" $PAGESIZES $PAGESIZES false
 }
 
 test2()
 {
 	tst_res TINFO "Test mapped_file"
 	test_mem_stat "--mmap-file" $PAGESIZES $PAGESIZES \
-		"mapped_file" $PAGESIZES false
+		"mapped_file" $PAGESIZES $PAGESIZES false
 }
 
 test3()
 {
 	tst_res TINFO "Test unevictable with MAP_LOCKED"
 	test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES \
-		"unevictable" $PAGESIZES false
+		"unevictable" $PAGESIZES $PAGESIZES false
 }
 
 test4()
 {
 	tst_res TINFO "Test unevictable with mlock"
 	test_mem_stat "--mmap-lock2" $PAGESIZES $PAGESIZES \
-		"unevictable" $PAGESIZES false
+		"unevictable" $PAGESIZES $PAGESIZES false
 }
 
 test5()
testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh +5 -2
index e77d6bf2ef23..9140fd9d1fd7 100755
@@ -12,13 +12,15 @@ TST_CNT=2
 . memcg_lib.sh
 
 MEM_TO_ALLOC=$((PAGESIZE * 1024))
+MEM_EXPECTED_UPPER=$((MEM_TO_ALLOC + MEM_USAGE_RANGE))
 MEM_LIMIT=$((MEM_TO_ALLOC * 2))
 
 test1()
 {
 	tst_res TINFO "Test memory.usage_in_bytes"
 	test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \
-		"memory.usage_in_bytes" $MEM_TO_ALLOC false
+		"memory.usage_in_bytes" $MEM_TO_ALLOC \
+		$MEM_EXPECTED_UPPER false
 }
 
 test2()
@@ -29,7 +31,8 @@ test2()
 	echo $MEM_LIMIT > memory.limit_in_bytes
 	echo $MEM_LIMIT > memory.memsw.limit_in_bytes
 	test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \
-		"memory.memsw.usage_in_bytes" $MEM_TO_ALLOC false
+		"memory.memsw.usage_in_bytes" $MEM_TO_ALLOC \
+		$MEM_EXPECTED_UPPER false
 }
 
 tst_run
Signature · 1 line
2.27.0
#3 lore raw

PATCH v4 3/5 controllers/memcg: accept non-zero max_usage_in_bytes after reset

In reply to Krzysztof Kozlowski

Several Linux kernel versions report a non-zero max_usage_in_bytes after resetting the counter. For example v5.4, v5.8, v5.10, v5.11, v5.12 and 5.13.0-rc5:

memcg_max_usage_in_bytes_test 4 TINFO: Test reset memory.memsw.max_usage_in_bytes memcg_max_usage_in_bytes_test 4 TINFO: Running memcg_process --mmap-anon -s 4194304 memcg_max_usage_in_bytes_test 4 TINFO: Warming up pid: 1416 memcg_max_usage_in_bytes_test 4 TINFO: Process is still here after warm up: 1416 memcg_max_usage_in_bytes_test 4 TFAIL: memory.memsw.max_usage_in_bytes is 4325376, 4194304 expected memcg_max_usage_in_bytes_test 4 TFAIL: memory.memsw.max_usage_in_bytes is 122880, 0 expected

It seems that recent Linux kernel caches the statistics more aggressively (especially on multi-CPU systems) and the batch updates of 32 pages are visible in usage_in_bytes.

---
 .../memcg/functional/memcg_max_usage_in_bytes_test.sh           | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +1 -1
index 8831f1937070..2d494ac3a78f 100755
@@ -37,7 +37,7 @@ test_max_usage_in_bytes()
 
 	if [ $check_after_reset -eq 1 ]; then
 		echo 0 > $item
-		check_mem_stat $item 0
+		check_mem_stat $item 0 $PAGESIZES
 	fi
 
 	stop_memcg_process
Signature · 1 line
2.27.0
#4 lore raw

PATCH v4 4/5 controllers/memcg: increase memory limit in subgroup charge

In reply to Krzysztof Kozlowski

The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases with:

memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168 memcg_subgroup_charge 1 TINFO: Warming up pid: 19289 memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289 memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected memcg_subgroup_charge 1 TPASS: rss is 0 as expected

In dmesg one could see that OOM killer killed the process even though group memory limit was matching the usage:

    memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
    CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
    ...
    memory: usage 132kB, limit 132kB, failcnt 9
    memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
    kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
    ...
    Tasks state (memory values in pages):
    [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
    [  19289]     0 19289      669      389    40960        0             0 memcg_process
    oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
    Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
    oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB

It seems actual kernel memory usage by a given group depends on number of CPUs, where at least one page is allocated per-cpu beside regular (expected) allocation. Fix the test on machines with more CPUs by including this per-CPU memory in group limits, plus some slack of 4 PAGES. Increase also memory allocation from 32 to 64 pages to be more distinctive from kernel per-CPU memory.

---
 .../memcg/functional/memcg_subgroup_charge.sh | 33 ++++++++++++++-----
 1 file changed, 25 insertions(+), 8 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh +25 -8
index 9b23177a4dc5..0d2b235aff7c 100755
@@ -14,16 +14,33 @@ TST_CNT=3
 
 . memcg_lib.sh
 
+# Allocate memory bigger than per-cpu kernel memory
+MEM_TO_ALLOC=$((PAGESIZES * 2))
+
 # Test the memory charge won't move to subgroup
 # $1 - memory.limit_in_bytes in parent group
 # $2 - memory.limit_in_bytes in sub group
 test_subgroup()
 {
+	local limit_parent=$1
+	local limit_subgroup=$2
+	local total_cpus=`tst_ncpus`
+
+	# Kernel memory allocated for the process is also charged.
+	# It might depend on the number of CPUs. For example on kernel v5.11
+	# additionally total_cpus plus 1-2 pages are charged to the group.
+	if [ $limit_parent -ne 0 ]; then
+		limit_parent=$((limit_parent + 4 * PAGESIZE + total_cpus * PAGESIZE))
+	fi
+	if [ $limit_subgroup -ne 0 ]; then
+		limit_subgroup=$((limit_subgroup + 4 * PAGESIZE + total_cpus * PAGESIZE))
+	fi
+
 	mkdir subgroup
-	echo $1 > memory.limit_in_bytes
-	echo $2 > subgroup/memory.limit_in_bytes
+	echo $limit_parent > memory.limit_in_bytes
+	echo $limit_subgroup > subgroup/memory.limit_in_bytes
 
-	start_memcg_process --mmap-anon -s $PAGESIZES
+	start_memcg_process --mmap-anon -s $MEM_TO_ALLOC
 
 	warmup
 	if [ $? -ne 0 ]; then
@@ -31,8 +48,8 @@ test_subgroup()
 	fi
 
 	echo $MEMCG_PROCESS_PID > tasks
-	signal_memcg_process $PAGESIZES
-	check_mem_stat "rss" $PAGESIZES
+	signal_memcg_process $MEM_TO_ALLOC
+	check_mem_stat "rss" $MEM_TO_ALLOC
 
 	cd subgroup
 	echo $MEMCG_PROCESS_PID > tasks
@@ -47,17 +64,17 @@ test_subgroup()
 test1()
 {
 	tst_res TINFO "Test that group and subgroup have no relationship"
-	test_subgroup $PAGESIZES $((2 * PAGESIZES))
+	test_subgroup $MEM_TO_ALLOC $((2 * MEM_TO_ALLOC))
 }
 
 test2()
 {
-	test_subgroup $PAGESIZES $PAGESIZES
+	test_subgroup $MEM_TO_ALLOC $MEM_TO_ALLOC
 }
 
 test3()
 {
-	test_subgroup $PAGESIZES 0
+	test_subgroup $MEM_TO_ALLOC 0
 }
 
 tst_run
Signature · 1 line
2.27.0
#5 lore raw

PATCH v4 4/5 controllers/memcg: increase memory limit in subgroup charge

Hello Krzysztof,

Quoted 27 lines · Krzysztof Kozlowski

The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases with:

memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168 memcg_subgroup_charge 1 TINFO: Warming up pid: 19289 memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289 memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected memcg_subgroup_charge 1 TPASS: rss is 0 as expected

In dmesg one could see that OOM killer killed the process even though group memory limit was matching the usage:

    memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
    CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
    ...
    memory: usage 132kB, limit 132kB, failcnt 9
    memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
    kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
    ...
    Tasks state (memory values in pages):
    [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
    [  19289]     0 19289      669      389    40960        0             0 memcg_process
    oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
    Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
    oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
Quoted 6 lines

It seems actual kernel memory usage by a given group depends on number of CPUs, where at least one page is allocated per-cpu beside regular (expected) allocation. Fix the test on machines with more CPUs by including this per-CPU memory in group limits, plus some slack of 4 PAGES. Increase also memory allocation from 32 to 64 pages to be more distinctive from kernel per-CPU memory.

Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.

Quoted 17 lines
---
 .../memcg/functional/memcg_subgroup_charge.sh | 33 ++++++++++++++-----
 1 file changed, 25 insertions(+), 8 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh +3 -0
index 9b23177a4dc5..0d2b235aff7c 100755
@@ -14,16 +14,33 @@ TST_CNT=3
 
 . memcg_lib.sh
 
+# Allocate memory bigger than per-cpu kernel memory
+MEM_TO_ALLOC=$((PAGESIZES * 2))
+
Signature · 2 lines
Thank you,
Richard.
#6 lore raw

PATCH v4 4/5 controllers/memcg: increase memory limit in subgroup charge

Quoted 31 lines · Richard Palethorpe

Hello Krzysztof,

Quoted 27 lines · Krzysztof Kozlowski

The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases with:

memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168 memcg_subgroup_charge 1 TINFO: Warming up pid: 19289 memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289 memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected memcg_subgroup_charge 1 TPASS: rss is 0 as expected

In dmesg one could see that OOM killer killed the process even though group memory limit was matching the usage:

    memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
    CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
    ...
    memory: usage 132kB, limit 132kB, failcnt 9
    memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
    kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
    ...
    Tasks state (memory values in pages):
    [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
    [  19289]     0 19289      669      389    40960        0             0 memcg_process
    oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
    Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
    oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
Quoted 9 lines
Quoted 6 lines

It seems actual kernel memory usage by a given group depends on number of CPUs, where at least one page is allocated per-cpu beside regular (expected) allocation. Fix the test on machines with more CPUs by including this per-CPU memory in group limits, plus some slack of 4 PAGES. Increase also memory allocation from 32 to 64 pages to be more distinctive from kernel per-CPU memory.

Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.

Yes, right. Maybe this could be fixed when applying - no need for resend.

Best regards,
Krzysztof

#7 lore raw

PATCH v4 4/5 controllers/memcg: increase memory limit in subgroup charge

Hello,

Quoted 39 lines · Krzysztof Kozlowski
Quoted 38 lines · Richard Palethorpe

Hello Krzysztof,

Quoted 33 lines · Krzysztof Kozlowski

The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases with:

memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168 memcg_subgroup_charge 1 TINFO: Warming up pid: 19289 memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289 memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected memcg_subgroup_charge 1 TPASS: rss is 0 as expected

In dmesg one could see that OOM killer killed the process even though group memory limit was matching the usage:

    memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
    CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
    ...
    memory: usage 132kB, limit 132kB, failcnt 9
    memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
    kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
    ...
    Tasks state (memory values in pages):
    [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
    [  19289]     0 19289      669      389    40960        0             0 memcg_process
    oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
    Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
    oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB

It seems actual kernel memory usage by a given group depends on number of CPUs, where at least one page is allocated per-cpu beside regular (expected) allocation. Fix the test on machines with more CPUs by including this per-CPU memory in group limits, plus some slack of 4 PAGES. Increase also memory allocation from 32 to 64 pages to be more distinctive from kernel per-CPU memory.

Quoted 8 lines
Quoted 2 lines

Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.

Yes, right. Maybe this could be fixed when applying - no need for resend.

Best regards,
Krzysztof

Well I am more than happy with the patchset.

Signature · 2 lines
Thank you,
Richard.
#8 lore raw

PATCH v4 4/5 controllers/memcg: increase memory limit in subgroup charge

Hi Krzysztof, Richard,

Quoted 6 lines · Richard Palethorpe
Quoted 5 lines
Quoted 3 lines

Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.

Yes, right. Maybe this could be fixed when applying - no need for resend.

I help correct that and applied patchset, thanks for all your work on this.

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20210714/3f229093/attachment.htm>
#9 lore raw

PATCH v4 5/5 controllers/memcg: offset kernel memory

In reply to Krzysztof Kozlowski

The memory allocated by kernel with __GFP_ACCOUNT is counted as well in memory group usage limits. Add it to fix failures like:

    memcg_max_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304
    memcg_max_usage_in_bytes_test 1 TINFO: Warming up pid: 925811
    memcg_max_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 925811
    memcg_max_usage_in_bytes_test 1 TFAIL: memory.max_usage_in_bytes is 4333568, 4194304-4325376 expected

    (where memory.kmem.max_usage_in_bytes:8192)
---
 .../controllers/memcg/functional/memcg_lib.sh     | 10 ++++++++++
 .../functional/memcg_max_usage_in_bytes_test.sh   | 15 ++++++++++++++-
 2 files changed, 24 insertions(+), 1 deletion(-)
testcases/kernel/controllers/memcg/functional/memcg_lib.sh +10 -0
index 9634eb7e9f42..dad66c798e19 100755
@@ -251,6 +251,7 @@ test_mem_stat()
 	local exp_stat_size_low=$5
 	local exp_stat_size_up=$6
 	local check_after_free=$7
+	local kmem_stat_name="${stat_name##*.}"
 
 	start_memcg_process $memtypes -s $size
 
@@ -261,6 +262,15 @@ test_mem_stat()
 	echo $MEMCG_PROCESS_PID > tasks
 	signal_memcg_process $size
 
+	if [ "$kmem_stat_name" = "max_usage_in_bytes" ] ||
+	   [ "$kmem_stat_name" = "usage_in_bytes" ]; then
+		local kmem=$(cat "memory.kmem.${kmem_stat_name}")
+		if [ $? -eq 0 ]; then
+			exp_stat_size_low=$((exp_stat_size_low + kmem))
+			exp_stat_size_up=$((exp_stat_size_up + kmem))
+		fi
+	fi
+
 	if [ "$exp_stat_size_low" = "$exp_stat_size_up" ]; then
 		check_mem_stat $stat_name $exp_stat_size_low
 	else
testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +14 -1
index 2d494ac3a78f..24e9d115c19e 100755
@@ -22,6 +22,10 @@ test_max_usage_in_bytes()
 	local item="memory.max_usage_in_bytes"
 	[ $1 -eq 1 ] && item="memory.memsw.max_usage_in_bytes"
 	local check_after_reset=$2
+	local exp_stat_size_low=$MEM_TO_ALLOC
+	local exp_stat_size_up=$MEM_EXPECTED_UPPER
+	local kmem_stat_name="${item##*.}"
+
 	start_memcg_process --mmap-anon -s $MEM_TO_ALLOC
 
 	warmup
@@ -33,7 +37,16 @@ test_max_usage_in_bytes()
 	signal_memcg_process $MEM_TO_ALLOC
 	signal_memcg_process $MEM_TO_ALLOC
 
-	check_mem_stat $item $MEM_TO_ALLOC $MEM_EXPECTED_UPPER
+	if [ "$kmem_stat_name" = "max_usage_in_bytes" ] ||
+	   [ "$kmem_stat_name" = "usage_in_bytes" ]; then
+		local kmem=$(cat "memory.kmem.${kmem_stat_name}")
+		if [ $? -eq 0 ]; then
+			exp_stat_size_low=$((exp_stat_size_low + kmem))
+			exp_stat_size_up=$((exp_stat_size_up + kmem))
+		fi
+	fi
+
+	check_mem_stat $item $exp_stat_size_low $exp_stat_size_up
 
 	if [ $check_after_reset -eq 1 ]; then
 		echo 0 > $item
Signature · 1 line
2.27.0