Skip to contentSkip to conversation

controllers/memcg: fixes for newer kernels

PATCH Superseded Krzysztof Kozlowski 4 participants 11 messages Jun 17, 2021 – Jul 2, 2021

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover controllers/memcg: fixes for newer kernels

Hi,

This is a resend of previous Github pull:
https://github.com/linux-test-project/ltp/pull/830

The patches fix several test failures we are hitting since months, see:
https://bugs.launchpad.net/bugs/1829979
https://bugs.launchpad.net/bugs/1829984

Best regards,
Krzysztof

Krzysztof Kozlowski (3):
controllers/memcg: accept range of max_usage_in_bytes controllers/memcg: accept range of usage_in_bytes controllers/memcg: accept non-zero max_usage_in_bytes after reset

 .../controllers/memcg/functional/memcg_lib.sh | 22 ++++++++++++++-----
 .../memcg_max_usage_in_bytes_test.sh          | 12 ++++++++--
 .../memcg/functional/memcg_stat_rss.sh        | 20 ++++++++---------
 .../memcg/functional/memcg_stat_test.sh       |  8 +++----
 .../functional/memcg_usage_in_bytes_test.sh   | 10 +++++++--
 5 files changed, 49 insertions(+), 23 deletions(-)
Signature · 1 line
2.27.0
#1 lore raw

PATCH v2 1/3 controllers/memcg: accept range of max_usage_in_bytes

Several Linux kernel versions report higher max_usage_in_bytes than expected size of 1024 pages. For example v5.4, v5.8, v5.10 and 5.13.0-rc5:

memcg_max_usage_in_bytes_test 1 TINFO: Test memory.max_usage_in_bytes memcg_max_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304 memcg_max_usage_in_bytes_test 1 TINFO: Warming up pid: 1393215 memcg_max_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 1393215 memcg_max_usage_in_bytes_test 1 TFAIL: memory.max_usage_in_bytes is 4325376, 4194304 expected

It seems that recent Linux kernel reports more memory used than expected.

---
 .../kernel/controllers/memcg/functional/memcg_lib.sh  | 11 +++++++++--
 .../memcg/functional/memcg_max_usage_in_bytes_test.sh |  6 +++++-
 2 files changed, 14 insertions(+), 3 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_lib.sh +9 -2
index d9bb6d94324d..083ef376e120 100755
@@ -140,7 +140,8 @@ shmmax_cleanup()
 
 # Check size in memcg
 # $1 - Item name
-# $2 - Expected size
+# $2 - Expected size lower bound
+# $3 - Expected size upper bound (optional)
 check_mem_stat()
 {
 	local item_size
@@ -151,7 +152,13 @@ check_mem_stat()
 		item_size=$(grep -w $1 memory.stat | cut -d " " -f 2)
 	fi
 
-	if [ "$2" = "$item_size" ]; then
+	if [ "$3" ]; then
+		if [ $item_size -ge $2 ] && [ $item_size -le $3 ]; then
+			tst_res TPASS "$1 is ${2}-${3} as expected"
+		else
+			tst_res TFAIL "$1 is $item_size, ${2}-${3} expected"
+		fi
+	elif [ "$2" = "$item_size" ]; then
 		tst_res TPASS "$1 is $2 as expected"
 	else
 		tst_res TFAIL "$1 is $item_size, $2 expected"
testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +5 -1
index 14daa4651798..a940606cbd34 100755
@@ -12,6 +12,10 @@ TST_CNT=4
 . memcg_lib.sh
 
 MEM_TO_ALLOC=$((PAGESIZE * 1024))
+# Recent Linux kernels (at least v5.4) started reporting memory usage of 32
+# pages higher than requested allocation. Cause is not known, so let's just be
+# flexible.
+MEM_EXPECTED_UPPER=$((MEM_TO_ALLOC + PAGESIZE * 32))
 MEM_LIMIT=$((MEM_TO_ALLOC * 2))
 
 # Run test cases which checks memory.[memsw.]max_usage_in_bytes after make
@@ -32,7 +36,7 @@ test_max_usage_in_bytes()
 	signal_memcg_process $MEM_TO_ALLOC
 	signal_memcg_process $MEM_TO_ALLOC
 
-	check_mem_stat $item $MEM_TO_ALLOC
+	check_mem_stat $item $MEM_TO_ALLOC $MEM_EXPECTED_UPPER
 
 	if [ $check_after_reset -eq 1 ]; then
 		echo 0 > $item
Signature · 1 line
2.27.0
#2 lore raw

PATCH v2 2/3 controllers/memcg: accept range of usage_in_bytes

In reply to Krzysztof Kozlowski

Several Linux kernel versions report higher usage_in_bytes than expected size of 1024 pages. For example v5.4, v5.8, v5.10 and 5.13.0-rc5:

memcg_usage_in_bytes_test 1 TINFO: Test memory.usage_in_bytes memcg_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304 memcg_usage_in_bytes_test 1 TINFO: Warming up pid: 1160 memcg_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 1160 memcg_usage_in_bytes_test 1 TFAIL: memory.usage_in_bytes is 4325376, 4194304 expected

It seems that recent Linux kernel reports more memory used than expected.

---
 .../controllers/memcg/functional/memcg_lib.sh | 11 +++++++---
 .../memcg/functional/memcg_stat_rss.sh        | 20 +++++++++----------
 .../memcg/functional/memcg_stat_test.sh       |  8 ++++----
 .../functional/memcg_usage_in_bytes_test.sh   | 10 ++++++++--
 4 files changed, 30 insertions(+), 19 deletions(-)
testcases/kernel/controllers/memcg/functional/memcg_lib.sh +8 -3
index 083ef376e120..d44bb027076c 100755
@@ -239,8 +239,9 @@ test_mem_stat()
 	local size=$2
 	local total_size=$3
 	local stat_name=$4
-	local exp_stat_size=$5
-	local check_after_free=$6
+	local exp_stat_size_low=$5
+	local exp_stat_size_up=$6
+	local check_after_free=$7
 
 	start_memcg_process $memtypes -s $size
 
@@ -251,7 +252,11 @@ test_mem_stat()
 	echo $MEMCG_PROCESS_PID > tasks
 	signal_memcg_process $size
 
-	check_mem_stat $stat_name $exp_stat_size
+	if [ "$exp_stat_size_low" = "$exp_stat_size_up" ]; then
+		check_mem_stat $stat_name $exp_stat_size_low
+	else
+		check_mem_stat $stat_name $exp_stat_size_low $exp_stat_size_up
+	fi
 
 	signal_memcg_process $size
 	if $check_after_free; then
testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh +10 -10
index 1a6128a6dba8..d9b4ec287b5f 100755
@@ -18,54 +18,54 @@ TST_CNT=10
 # Test the management and counting of memory
 test1()
 {
-	test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES false
+	test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES false
 }
 
 test2()
 {
-	test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 false
+	test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 0 false
 }
 
 test3()
 {
-	test_mem_stat "--shm -k 3" $PAGESIZE $PAGESIZE "rss" 0 false
+	test_mem_stat "--shm -k 3" $PAGESIZE $PAGESIZE "rss" 0 0 false
 }
 
 test4()
 {
 	test_mem_stat "--mmap-anon --mmap-file --shm" \
-		$PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES false
+		$PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES $PAGESIZES false
 }
 
 test5()
 {
-	test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES false
+	test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES false
 }
 
 test6()
 {
-	test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES true
+	test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES true
 }
 
 test7()
 {
-	test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 true
+	test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 0 true
 }
 
 test8()
 {
-	test_mem_stat "--shm -k 8" $PAGESIZE $PAGESIZE "rss" 0 true
+	test_mem_stat "--shm -k 8" $PAGESIZE $PAGESIZE "rss" 0 0 true
 }
 
 test9()
 {
 	test_mem_stat "--mmap-anon --mmap-file --shm" \
-		$PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES true
+		$PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES $PAGESIZES true
 }
 
 test10()
 {
-	test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES true
+	test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES true
 }
 
 tst_run
testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh +4 -4
index 925c4ecf87bc..e5eb7e5d0001 100755
@@ -14,28 +14,28 @@ TST_CNT=8
 test1()
 {
 	tst_res TINFO "Test cache"
-	test_mem_stat "--shm -k 3" $PAGESIZES $PAGESIZES "cache" $PAGESIZES false
+	test_mem_stat "--shm -k 3" $PAGESIZES $PAGESIZES "cache" $PAGESIZES $PAGESIZES false
 }
 
 test2()
 {
 	tst_res TINFO "Test mapped_file"
 	test_mem_stat "--mmap-file" $PAGESIZES $PAGESIZES \
-		"mapped_file" $PAGESIZES false
+		"mapped_file" $PAGESIZES $PAGESIZES false
 }
 
 test3()
 {
 	tst_res TINFO "Test unevictable with MAP_LOCKED"
 	test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES \
-		"unevictable" $PAGESIZES false
+		"unevictable" $PAGESIZES $PAGESIZES false
 }
 
 test4()
 {
 	tst_res TINFO "Test unevictable with mlock"
 	test_mem_stat "--mmap-lock2" $PAGESIZES $PAGESIZES \
-		"unevictable" $PAGESIZES false
+		"unevictable" $PAGESIZES $PAGESIZES false
 }
 
 test5()
testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh +8 -2
index e77d6bf2ef23..b5761a4e4716 100755
@@ -12,13 +12,18 @@ TST_CNT=2
 . memcg_lib.sh
 
 MEM_TO_ALLOC=$((PAGESIZE * 1024))
+# Recent Linux kernels (at least v5.4) started reporting memory usage of 32
+# pages higher than requested allocation. Cause is not known, so let's just be
+# flexible.
+MEM_EXPECTED_UPPER=$((MEM_TO_ALLOC + PAGESIZE * 32))
 MEM_LIMIT=$((MEM_TO_ALLOC * 2))
 
 test1()
 {
 	tst_res TINFO "Test memory.usage_in_bytes"
 	test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \
-		"memory.usage_in_bytes" $MEM_TO_ALLOC false
+		"memory.usage_in_bytes" $MEM_TO_ALLOC \
+		$MEM_EXPECTED_UPPER false
 }
 
 test2()
@@ -29,7 +34,8 @@ test2()
 	echo $MEM_LIMIT > memory.limit_in_bytes
 	echo $MEM_LIMIT > memory.memsw.limit_in_bytes
 	test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \
-		"memory.memsw.usage_in_bytes" $MEM_TO_ALLOC false
+		"memory.memsw.usage_in_bytes" $MEM_TO_ALLOC \
+		$MEM_EXPECTED_UPPER false
 }
 
 tst_run
Signature · 1 line
2.27.0
#3 lore raw

PATCH v2 3/3 controllers/memcg: accept non-zero max_usage_in_bytes after reset

In reply to Krzysztof Kozlowski

Several Linux kernel versions report a non-zero max_usage_in_bytes after resetting the counter. For example v5.4, v5.8, v5.10, v5.11, v5.12 and 5.13.0-rc5:

memcg_max_usage_in_bytes_test 4 TINFO: Test reset memory.memsw.max_usage_in_bytes memcg_max_usage_in_bytes_test 4 TINFO: Running memcg_process --mmap-anon -s 4194304 memcg_max_usage_in_bytes_test 4 TINFO: Warming up pid: 1416 memcg_max_usage_in_bytes_test 4 TINFO: Process is still here after warm up: 1416 memcg_max_usage_in_bytes_test 4 TFAIL: memory.memsw.max_usage_in_bytes is 4325376, 4194304 expected memcg_max_usage_in_bytes_test 4 TFAIL: memory.memsw.max_usage_in_bytes is 122880, 0 expected

It seems that recent Linux kernel still notices some memory allocation by the memcg tool. Accept therefore a range from 0 to 32 pages.

---
 .../memcg/functional/memcg_max_usage_in_bytes_test.sh       | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh +5 -1
index a940606cbd34..8f0fc33996f3 100755
@@ -40,7 +40,11 @@ test_max_usage_in_bytes()
 
 	if [ $check_after_reset -eq 1 ]; then
 		echo 0 > $item
-		check_mem_stat $item 0
+		# Recent Linux kernels (at least v5.4) started reporting
+		# a non-zero max_usage_in_bytes after resetting the counter.
+		# The typical values are 0, 4096, 8096 and up to 122880.
+		# Cause is not known, so let's just be flexible.
+		check_mem_stat $item 0 $((PAGESIZE * 32))
 	fi
 
 	stop_memcg_process
Signature · 1 line
2.27.0
#4 lore raw

v2 cover controllers/memcg: fixes for newer kernels

In reply to Krzysztof Kozlowski

Hello,

Quoted 4 lines · Krzysztof Kozlowski

Hi,

This is a resend of previous Github pull:
https://github.com/linux-test-project/ltp/pull/830

LGTM (Reviewed on Github).

I doubt tests that expect a precise allocation figure are valid.

Quoted 23 lines

The patches fix several test failures we are hitting since months, see:
https://bugs.launchpad.net/bugs/1829979
https://bugs.launchpad.net/bugs/1829984

Best regards,
Krzysztof

Krzysztof Kozlowski (3):
controllers/memcg: accept range of max_usage_in_bytes controllers/memcg: accept range of usage_in_bytes controllers/memcg: accept non-zero max_usage_in_bytes after reset

 .../controllers/memcg/functional/memcg_lib.sh | 22 ++++++++++++++-----
 .../memcg_max_usage_in_bytes_test.sh          | 12 ++++++++--
 .../memcg/functional/memcg_stat_rss.sh        | 20 ++++++++---------
 .../memcg/functional/memcg_stat_test.sh       |  8 +++----
 .../functional/memcg_usage_in_bytes_test.sh   | 10 +++++++--
 5 files changed, 49 insertions(+), 23 deletions(-)
Signature · 1 line
2.27.0
Signature · 2 lines
Thank you,
Richard.
#5 lore raw

v2 cover controllers/memcg: fixes for newer kernels

In reply to Krzysztof Kozlowski
Quoted 9 lines · Krzysztof Kozlowski

Hi,

This is a resend of previous Github pull:
https://github.com/linux-test-project/ltp/pull/830

The patches fix several test failures we are hitting since months, see:
https://bugs.launchpad.net/bugs/1829979
https://bugs.launchpad.net/bugs/1829984

Quoted 8 lines

Best regards,
Krzysztof

Krzysztof Kozlowski (3):
controllers/memcg: accept range of max_usage_in_bytes controllers/memcg: accept range of usage_in_bytes controllers/memcg: accept non-zero max_usage_in_bytes after reset

Hi everyone,

The patchset got positive LGTM on the Github. Any further comments for it or can it be applied?

Best regards,
Krzysztof

#6 lore raw

v2 cover controllers/memcg: fixes for newer kernels

Quoted 20 lines · Krzysztof Kozlowski
Quoted 9 lines · Krzysztof Kozlowski

Hi,

This is a resend of previous Github pull:
https://github.com/linux-test-project/ltp/pull/830

The patches fix several test failures we are hitting since months, see:
https://bugs.launchpad.net/bugs/1829979
https://bugs.launchpad.net/bugs/1829984

Quoted 8 lines

Best regards,
Krzysztof

Krzysztof Kozlowski (3):
controllers/memcg: accept range of max_usage_in_bytes controllers/memcg: accept range of usage_in_bytes controllers/memcg: accept non-zero max_usage_in_bytes after reset

Quoted 5 lines

Hi everyone,

The patchset got positive LGTM on the Github. Any further comments for it or can it be applied?

I slightly agree with Richard that we need an explanation/investigation on where the 32*PAGE_SIZE comes from. Otherwise, we are very possible to mask a counter bug if only to make the test happy.

Forgive me pour cold water on the method though it looks good in coding:).

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20210625/0f253bc8/attachment.htm>
#7 lore raw

v2 cover controllers/memcg: fixes for newer kernels

Quoted 8 lines · Li Wang

On Fri, Jun 25, 2021 at 3:31 AM Krzysztof Kozlowski <[email protected] <mailto:[email protected]>> wrote:

Hi everyone,

Quoted 7 lines

The patchset got positive LGTM on the Github. Any further comments for it or can it be applied?

I slightly agree with Richard that we need an explanation/investigation on where the 32*PAGE_SIZE comes from. Otherwise, we are very possible to mask a counter bug if only to make the test happy.

I don't know where 32*PAGE_SIZE comes from and investigation would require effort/time which I don't have.

I don't think we mask current bug as this appears in multiple kernels - I tested from v5.4 up to 5.13.0-rc5-next-20210608.

It is possible this will mask future bugs but that's life of a project depending on kernel internals, not on API or ABI. The kernel is allowed to change such details any moment because it is neither part of API nor ABI. Therefore you just have to live with inaccurate limits or keep investigating every x-months.

For now the test is simply unreliable.

Best regards,
Krzysztof

#8 lore raw

v2 cover controllers/memcg: fixes for newer kernels

In reply to Li Wang

Hi,

Quoted 7 lines · Li Wang

On Fri, Jun 25, 2021 at 3:31 AM Krzysztof Kozlowski <[email protected] <mailto:[email protected]>> wrote:

On 17/06/2021 09:07, Krzysztof Kozlowski wrote:

Quoted 7 lines

The patchset got positive LGTM on the Github. Any further comments for it or can it be applied?

I slightly agree with Richard that we need an explanation/investigation on where the 32*PAGE_SIZE comes from. Otherwise, we are very possible to mask a counter bug if only to make the test happy.

+1. I think it has something to do with batch processing of counter updates. The memory handling code is heavily batched per cpu and only committed to a global counter after the batch size overflows.
At least for the subgroup_charge test I once found that page table cache stored in kmem contributes to the memory counter used in the test. Maybe something in that region was changed in recent kernel version. Maybe you could ask on the kernel mailing list?

Quoted 2 lines

Forgive me pour cold water on the method though it looks good in coding:).

I would just suggest to make 32 * PAGESIZE a constant calculated in memcg_lib.sh, instead of calculating (and defining) it everywhere.

Quoted 5 lines
Signature · 3 lines
-- 
Regards,
Li Wang

J?rg

#9 lore raw

v2 cover controllers/memcg: fixes for newer kernels

Quoted 3 lines · Joerg Vehlow
Quoted 1 line

Forgive me pour cold water on the method though it looks good in coding:).

I would just suggest to make 32 * PAGESIZE a constant calculated in memcg_lib.sh, instead of calculating (and defining) it everywhere.

I understand that you want to define constant in memcg_lib for a accepted higher bound of some specific memory limit checks? Won't that be extra confusing? Trying to generalize some constant without actually knowing what is it about and to which limits it applies?

Now it's this unknown part is quite local and documented in three places. I can move it to bigger scope and pretend it's generic, if you want.

Best regards,
Krzysztof

#10 lore raw

v2 cover controllers/memcg: fixes for newer kernels

In reply to Li Wang
Quoted 36 lines · Li Wang

On Fri, Jun 25, 2021 at 3:31 AM Krzysztof Kozlowski <[email protected] <mailto:[email protected]>> wrote:

    On 17/06/2021 09:07, Krzysztof Kozlowski wrote:
    > Hi,
    >
    > This is a resend of previous Github pull:
    > https://github.com/linux-test-project/ltp/pull/830
    <https://github.com/linux-test-project/ltp/pull/830>
    >
    > The patches fix several test failures we are hitting since months,
    see:
    > https://bugs.launchpad.net/bugs/1829979
    <https://bugs.launchpad.net/bugs/1829979>
    > https://bugs.launchpad.net/bugs/1829984
    <https://bugs.launchpad.net/bugs/1829984>
    >
    > Best regards,
    > Krzysztof
    >
    >
    > Krzysztof Kozlowski (3):
    >? ?controllers/memcg: accept range of max_usage_in_bytes
    >? ?controllers/memcg: accept range of usage_in_bytes
    >? ?controllers/memcg: accept non-zero max_usage_in_bytes after reset


    Hi everyone,

    The patchset got positive LGTM on the Github. Any further comments for
    it or can it be applied?
Quoted 3 lines

I slightly agree with Richard that we need an explanation/investigation on where the 32*PAGE_SIZE comes from. Otherwise, we are very possible to mask a counter bug if only to make the test happy.

On newer v5.11 the max_usage_in_bytes go even above 32 pages up to 34.

I got some explanation from Michal Hocko [1] from which one can try to conclude:
1. There is significant caching in memory accounting. Not only charging of cgroup is once per MEMCG_CHARGE_BATCH batch (try_charge()), but also statistics are gathered from per-cpu if threshold exceed MEMCG_CHARGE_BATCH (__mod_memcg_state()).

2. Depending on machine (different amount of CPUs), the memory group charging be less or more accurate.

3. The accuracy of group memory accounting is not considered as important thus test relying on it, will be failing from time to time.

I'll send a v3 with significantly increased limits and some explanation.

[1]
https://lore.kernel.org/linux-mm/[email protected]/T/#m6459b3be3a86f5eaf2cfc48dd586b6faf949e440

Best regards,
Krzysztof