mm: rewrite mtest01 with new API
RFC PATCH 1/1 mm: rewrite mtest01 with new API
Test issue:
mtest01 start many children to alloc chunck of memory and do write page(with -w option), but occasionally some children were killed by oom-killer and exit with SIGCHLD signal sending. After the parent reciving this SIGCHLD signal it will report FAIL as a test result.
It seems not a real kernel bug if something just like that, it's trying to use 80% of memory and swap. Once it uses most of memory, system starts swapping, but the test is likely consuming memory at greater rate than kswapd can provide, which eventually triggers OOM.
---- FAIL LOG ----
mtest01 0 TINFO : Total memory already used on system = 1027392 kbytes mtest01 0 TINFO : Total memory used needed to reach maximum = 12715520 kbytes mtest01 0 TINFO : Filling up 80% of ram which is 11688128 kbytes mtest01 1 TFAIL : mtest01.c:314: child process exited unexpectedly
-------------------
Rewrite changes:
To make mtest01 more easier to understand, I just rewrite it into LTP new API and make a little changes in children behavior.
* drop the signal SIGCHLD action becasue new API help to check_child_status
* make child pause itself after finishing their memory allocating/writing
* parent sends SIGCONT to make children continue and exit
* decrease the pressure to 50% total ram+swap for testing
--- runtest/mm | 4 +- testcases/kernel/mem/mtest01/mtest01.c | 430 ++++++++++++------------- 2 files changed, 204 insertions(+), 230 deletions(-)
runtest/mm +2 -2
@@ -9,8 +9,8 @@ mm02 mmap001 # repetitive mmapping test. # Creates a one page map repetitively for one minute. -mtest01 mtest01 -p80 -mtest01w mtest01 -p80 -w +mtest01 mtest01 -p50 +mtest01w mtest01 -p50 -w #test for race conditions mtest05 mmstress
testcases/kernel/mem/mtest01/mtest01.c +202 -228
522 lines not on this page · raw message
Signature · 1 line
2.20.1
RFC PATCH 1/1 mm: rewrite mtest01 with new API
Hi,
----- Original Message -----
Quoted 10 lines
Test issue:
mtest01 start many children to alloc chunck of memory and do write page(with -w option), but occasionally some children were killed by oom-killer and exit with SIGCHLD signal sending. After the parent reciving this SIGCHLD signal it will report FAIL as a test result.It seems not a real kernel bug if something just like that, it's trying to use 80% of memory and swap. Once it uses most of memory, system starts swapping, but the test is likely consuming memory at greater rate than kswapd can provide, which eventually triggers OOM.
This seems to be quite common on ppc systems (64k pages with slow I/O), so I do welcome fix/rewrite.
Quoted 14 lines
---- FAIL LOG ----
mtest01 0 TINFO : Total memory already used on system = 1027392
kbytes
mtest01 0 TINFO : Total memory used needed to reach maximum =
12715520 kbytesmtest01 0 TINFO : Filling up 80% of ram which is 11688128 kbytes mtest01 1 TFAIL : mtest01.c:314: child process exited unexpectedly-------------------
Rewrite changes:
To make mtest01 more easier to understand, I just rewrite it into LTP new API and make a little changes in children behavior.
Quoted 5 lines
* drop the signal SIGCHLD action becasue new API help to
check_child_status
* make child pause itself after finishing their memory allocating/writing
* parent sends SIGCONT to make children continue and exit
* decrease the pressure to 50% total ram+swap for testing
Current behaviour varies a lot depending on system. I'm thinking if we should just set it to 80% of free RAM. We already have number of OOM tests, so maybe we don't need to worry about memory pressure here too.
Quoted 7 lines
Signed-off-by Li Wang <[email protected]>--- runtest/mm | 4 +- testcases/kernel/mem/mtest01/mtest01.c | 430 ++++++++++++------------- 2 files changed, 204 insertions(+), 230 deletions(-)
<snip>
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 204
Quoted 11 lines
testcases/kernel/mem/mtest01/mtest01.c +11
-0+ +static void mem_test (void ) +{ +int i, pid_cntr; +pid_t pid; +struct sigaction act; + + act.sa_handler = handler; + act.sa_flags =0 ; +sigemptyset (&act.sa_mask); +sigaction (SIGRTMIN , &act,0 );
I was thinking if we can't "abuse" tst_futexes a bit. It's a piece of shared memory we already have and could use for an atomic counter.
<snip>
Quoted 20 lines
testcases/kernel/mem/mtest01/mtest01.c +2
-17+if (pid ==0 ) +child_loop_alloc (); -if (dowrite) { -/* Total Free Post-Test RAM */ - post_mem = - (unsigned long long )sstats.mem_unit * - sstats.freeram; - post_mem = - post_mem + - (unsigned long long )sstats.mem_unit * - sstats.freeswap; - -while ((((unsigned long long )pre_mem - post_mem) < - (unsigned long long )original_maxbytes) && - pid_count < pid_cntr && !sigchld_count) { -sleep (1 ); -sysinfo (&sstats); - post_mem = - (unsigned long long)sstats.mem_unit *
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 268
Quoted 10 lines
testcases/kernel/mem/mtest01/mtest01.c +3
-7- sstats.freeram; - post_mem = - post_mem + - (unsigned long long )sstats.mem_unit * - sstats.freeswap; - } - } +/* waits in the loop for all children finish allocating*/ +while (pid_count < pid_cntr) +sleep (1 );
What happens if one child hits OOM?
Quoted 33 lines
testcases/kernel/mem/mtest01/mtest01.c +5
-9-if (sigchld_count) { -tst_resm (TFAIL ,"child process exited unexpectedly" ); - }else if (dowrite) { -tst_resm (TPASS ,"%llu kbytes allocated and used." , - original_maxbytes /1024 ); - }else { -tst_resm (TPASS ,"%llu kbytes allocated only." , - original_maxbytes /1024 ); - } +if (dowrite) { +sysinfo (&sstats); +/* Total Free Post-Test RAM */ + post_mem = (unsigned long long )sstats.mem_unit * sstats.freeram; + post_mem = post_mem + (unsigned long long)sstats.mem_unit *sstats.freeswap;
testcases/kernel/mem/mtest01/mtest01.c +15
-0+if (((pre_mem - post_mem) < original_maxbytes)) +tst_res (TFAIL ,"kbytes allocated and used less than expected %llu" , + original_maxbytes /1024 ); +else +tst_res (TPASS ,"%llu kbytes allocated and used" , + original_maxbytes /1024 ); + }else { +tst_res (TPASS ,"%llu kbytes allocated only" , + original_maxbytes /1024 ); + } + + i =0 ; +while (pid_list[i] >0 ) { +kill (pid_list[i],SIGCONT ); + i++; }
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 299
Quoted 10 lines
testcases/kernel/mem/mtest01/mtest01.c +7
-2-cleanup (); -tst_exit (); } + +static struct tst_test test = { + .forks_child =1 , + .options = mtest_options, + .setup = setup, + .cleanup = cleanup, + .test_all = mem_test,
Is default timeout going to work on large boxes (256GB+ RAM)?
Thinking loud, what if...
- we define at the start of test how much memory we want to allocate (target == 80% of free RAM)
- we allocate a shared memory for counter, that each child increases
as it allocates memory (progress)
(or we abuse tst_futexes)
we could use tst_atomic_add_return() to count allocated chunks globally
- once child finishes allocation it will pause()
- we set timeout to ~3 minutes
- main process runs in loop, sleeps, and periodically checks
- if progress reached target, PASS, break
- if progress hasn't increased in last 15 seconds, FAIL, break
- if we are 15 seconds away from timeout, end test early, PASS, break
(reason is to avoid running too long on big boxes)
- kill all children, exit
Regards,
Jan
RFC PATCH 1/1 mm: rewrite mtest01 with new API
On Fri, Mar 1, 2019 at 6:09 AM Jan Stancek <[email protected]> wrote:
Current behaviour varies a lot depending on system. I'm thinking if we
Quoted 4 lines
should
just set it to 80% of free RAM. We already have number of OOM tests, so maybe we don't need to worry about memory pressure here too.
Yes, I'm ok with that change. Since if we decrease the allocated consumption to 50% mem+swap, that probably only do allocating in the part of free mem too.
Quoted 9 lines
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 204
Quoted 5 lines
testcases/kernel/mem/mtest01/mtest01.c +5
-0+ + act.sa_handler = handler; + act.sa_flags =0 ; +sigemptyset (&act.sa_mask); +sigaction (SIGRTMIN , &act,0 );I was thinking if we can't "abuse"
tst_futexesa bit. It's a piece of shared memory we already have and could use for an atomic counter.
Quoted 8 lines
<snip>
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 268
Quoted 3 lines
testcases/kernel/mem/mtest01/mtest01.c +3
-0+/* waits in the loop for all children finish allocating*/ +while (pid_count < pid_cntr) +sleep (1 );What happens if one child hits OOM?
LTP new API does wait and check child status for the test, if one child_A(allocating finished and status paused) hits OOM, it will just break and report status, but that's ok for this event, because other children which still allocating will keep running after system reclaiming memory from child_A. So parent process will recieve all of children's SIGRTMIN signal and break from the while loop correctly.
Anthoer situation(I haven't hit), is one child_B(still allocating and not finishes) was killed by OOM. that will make parent fall into an infinite loop here. From OOM mechanism, oom-killer likes to choose high score process, so this situation maybe not easy to reproduce. But that not mean it will not, since oom-killer is not perfect.
Anyway, to avoid the second situation occuring, I'd like to take you advice to make parent exiting loop safly with many check actions.
Quoted 40 lines
Review
Quoted 3 lines
(quoted diff) +0
-2- if (sigchld_count) { - tst_resm(TFAIL, "child process exitedunexpectedly");
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[])
Quoted 11 lines
testcases/kernel/mem/mtest01/mtest01.c +4
-7- }else if (dowrite) { -tst_resm (TPASS ,"%llu kbytes allocated and used." , - original_maxbytes /1024 ); - }else { -tst_resm (TPASS ,"%llu kbytes allocated only." , - original_maxbytes /1024 ); - } +if (dowrite) { +sysinfo (&sstats); +/* Total Free Post-Test RAM */ + post_mem = (unsigned long long)sstats.mem_unit *sstats.freeram;
Review
Quoted 5 lines
+
post_mem=post_mem+ (unsigned long long)sstats.mem_unit *
sstats.freeswap;(quoted diff) +2
-0+ if (((pre_mem - post_mem) < original_maxbytes)) + tst_res(TFAIL, "kbytes allocated and used lessthan expected %llu",
Quoted 18 lines
testcases/kernel/mem/mtest01/mtest01.c +14
-2+ original_maxbytes / 1024); + else + tst_res(TPASS, "%llu kbytes allocated and used", + original_maxbytes / 1024); + }else { +tst_res (TPASS ,"%llu kbytes allocated only" , + original_maxbytes /1024 ); + } + + i =0 ; +while (pid_list[i] >0 ) { +kill (pid_list[i],SIGCONT ); + i++; } -cleanup (); -tst_exit (); } +
Quoted 9 lines
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 299
Quoted 6 lines
testcases/kernel/mem/mtest01/mtest01.c +6
-0+static struct tst_test test = { + .forks_child =1 , + .options = mtest_options, + .setup = setup, + .cleanup = cleanup, + .test_all = mem_test,Is default timeout going to work on large boxes (256GB+ RAM)?
No.
I had the same worries before, but in this test, the number of children(max_pids) will be increased dynamically with the system total memory size. And each child allocating won't beyond the 'alloc_bytes' (alloc_bytes = MIN(THREE_GB, alloc_maxbytes)) limitaion, so an extra time consumption part is just by forking, but from my evaluation on a 4T ram system, mtest01 finishes very faster(99% mem+swap, 2m22sec) than I expected. So the default timeout is not trigger at all.
# cat /proc/meminfo | grep Mem MemTotal: 4227087524 kB MemFree: 4223159948 kB MemAvailable: 4213257308 kB
# time ./mtest01 -p99 -w tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s mtest01.c:113: INFO: Total memory already used on system = 3880348 kbytes mtest01.c:120: INFO: Total memory used needed to reach maximum = 4188969005 kbytes mtest01.c:134: INFO: Filling up 99% of ram which is 4185088657 kbytes ... mtest01.c:185: INFO: ... 3221225472 bytes allocated and used in child 41779 mtest01.c:281: PASS: 4185132681 kbytes allocated and used ...
real 2m22.213s
user 79m52.390s
sys 351m56.059s
Quoted 18 lines
Thinking loud, what if...
- we define at the start of test how much memory we want to allocate (target == 80% of free RAM)
- we allocate a shared memory for counter, that each child increases as it allocates memory (progress)
(or we abusetst_futexes)
we could usetst_atomic_add_return()to count allocated chunks globally
- once child finishes allocation it willpause()
- we set timeout to ~3 minutes
- main process runs in loop, sleeps, and periodically checks
- if progress reached target, PASS, break
- if progress hasn't increased in last 15 seconds, FAIL, break
- if we are 15 seconds away from timeout, end test early, PASS, break (reason is to avoid running too long on big boxes)
- kill all children, exit
Real good suggestions, I will try to take some of them in V2.
Quoted 3 lines
Regards,
Jan
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20190301/65522666/attachment.html>
RFC PATCH 1/1 mm: rewrite mtest01 with new API
----- Original Message -----
Quoted 147 lines · Jan Stancek
Current behaviour varies a lot depending on system. I'm thinking if we
Quoted 4 lines
should
just set it to 80% of free RAM. We already have number of OOM tests, so maybe we don't need to worry about memory pressure here too.Yes, I'm ok with that change. Since if we decrease the allocated consumption to 50% mem+swap, that probably only do allocating in the part of free mem too.
Quoted 9 lines
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 204
Quoted 5 lines
testcases/kernel/mem/mtest01/mtest01.c +5
-0+ + act.sa_handler = handler; + act.sa_flags =0 ; +sigemptyset (&act.sa_mask); +sigaction (SIGRTMIN , &act,0 );I was thinking if we can't "abuse"
tst_futexesa bit. It's a piece of shared memory we already have and could use for an atomic counter.Quoted 8 lines
<snip>
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 268
Quoted 3 lines
testcases/kernel/mem/mtest01/mtest01.c +3
-0+/* waits in the loop for all children finish allocating*/ +while (pid_count < pid_cntr) +sleep (1 );What happens if one child hits OOM?
LTP new API does wait and check child status for the test, if one
child_A(allocating finished and status paused)hits OOM, it will just break and report status, but that's ok for this event, because other children which still allocating will keep running after system reclaiming memory fromchild_A. So parent process will recieve all of children's SIGRTMIN signal and break from the while loop correctly.Anthoer situation(I haven't hit), is one
child_B(still allocating and not finishes)was killed by OOM. that will make parent fall into an infinite loop here. From OOM mechanism, oom-killer likes to choose high score process, so this situation maybe not easy to reproduce. But that not mean it will not, since oom-killer is not perfect.Anyway, to avoid the second situation occuring, I'd like to take you advice to make parent exiting loop safly with many check actions.
Quoted 40 lines
Review
Quoted 3 lines
(quoted diff) +0
-2- if (sigchld_count) { - tst_resm(TFAIL, "child process exitedunexpectedly");
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[])
Quoted 11 lines
testcases/kernel/mem/mtest01/mtest01.c +4
-7- }else if (dowrite) { -tst_resm (TPASS ,"%llu kbytes allocated and used." , - original_maxbytes /1024 ); - }else { -tst_resm (TPASS ,"%llu kbytes allocated only." , - original_maxbytes /1024 ); - } +if (dowrite) { +sysinfo (&sstats); +/* Total Free Post-Test RAM */ + post_mem = (unsigned long long)sstats.mem_unit *sstats.freeram;
Review
Quoted 5 lines
+
post_mem=post_mem+ (unsigned long long)sstats.mem_unit *
sstats.freeswap;(quoted diff) +2
-0+ if (((pre_mem - post_mem) < original_maxbytes)) + tst_res(TFAIL, "kbytes allocated and used lessthan expected %llu",
Quoted 18 lines
testcases/kernel/mem/mtest01/mtest01.c +14
-2+ original_maxbytes / 1024); + else + tst_res(TPASS, "%llu kbytes allocated and used", + original_maxbytes / 1024); + }else { +tst_res (TPASS ,"%llu kbytes allocated only" , + original_maxbytes /1024 ); + } + + i =0 ; +while (pid_list[i] >0 ) { +kill (pid_list[i],SIGCONT ); + i++; } -cleanup (); -tst_exit (); } +Quoted 9 lines
testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 299
Quoted 6 lines
testcases/kernel/mem/mtest01/mtest01.c +6
-0+static struct tst_test test = { + .forks_child =1 , + .options = mtest_options, + .setup = setup, + .cleanup = cleanup, + .test_all = mem_test,Is default timeout going to work on large boxes (256GB+ RAM)?
No.
I had the same worries before, but in this test, the number of
children(max_pids)will be increased dynamically with the system total memory size. And each child allocating won't beyond the 'alloc_bytes' (alloc_bytes=MIN(THREE_GB, alloc_maxbytes)) limitaion, so an extra time consumption part is just by forking, but from my evaluation on a 4T ram system, mtest01 finishes veryfaster(99% mem+swap, 2m22sec)than I expected. So the default timeout is not trigger at all.# cat /proc/meminfo | grep Mem MemTotal: 4227087524 kB MemFree: 4223159948 kB MemAvailable: 4213257308 kB# time ./mtest01 -p99 -w tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s mtest01.c:113: INFO: Total memory already used on system = 3880348 kbytes mtest01.c:120: INFO: Total memory used needed to reach maximum = 4188969005 kbytes mtest01.c:134: INFO: Filling up 99% of ram which is 4185088657 kbytes ... mtest01.c:185: INFO: ... 3221225472 bytes allocated and used in child 41779 mtest01.c:281: PASS: 4185132681 kbytes allocated and used ...real 2m22.213s
user 79m52.390s
sys 351m56.059sQuoted 18 lines
Thinking loud, what if...
- we define at the start of test how much memory we want to allocate (target == 80% of free RAM)
- we allocate a shared memory for counter, that each child increases as it allocates memory (progress)
(or we abusetst_futexes)
we could usetst_atomic_add_return()to count allocated chunks globally
- once child finishes allocation it willpause()
- we set timeout to ~3 minutes
- main process runs in loop, sleeps, and periodically checks
- if progress reached target, PASS, break
- if progress hasn't increased in last 15 seconds, FAIL, break
- if we are 15 seconds away from timeout, end test early, PASS, break (reason is to avoid running too long on big boxes)
- kill all children, exit
Quoted 1 line
Real good suggestions, I will try to take some of them in V2.
Maybe give it few days, so other people can respond, if they like/don't like going in this direction.
Quoted 11 lines
Quoted 5 lines
Regards,
Jan--
Regards,
Li Wang
RFC PATCH 1/1 mm: rewrite mtest01 with new API
On Fri, Mar 1, 2019 at 4:03 PM Jan Stancek <[email protected]> wrote:
Maybe give it few days, so other people can respond, if they like/don't like
Quoted 2 lines
going in this direction.
Sure.
Btw, I'm not going to use shared memory here since that probably make things more complicated.
I guess just do remaining time check in the parent while loop is enough for the two OOM situations
which I was mentioned in last mail.
==== mem_test() ======
...
if (pid == 0)child_loop_alloc();
/* waits in the loop for all children finish allocating*/ while(pid_count < pid_cntr) {
if (tst_timeout_remaining() < 15)
break;
sleep(1);
}
if (dowrite) {
pre_free - post_free check...
} else {
...
}
kill children here..
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20190301/6952d779/attachment-0001.html>
RFC PATCH 1/1 mm: rewrite mtest01 with new API
----- Original Message -----
Quoted 25 lines · Jan Stancek
Maybe give it few days, so other people can respond, if they like/don't like
Quoted 2 lines
going in this direction.
Sure.
Btw, I'm not going to use shared memory here since that probably make things more complicated.
I guess just do remaining time check in the parent while loop is enough for the two OOM situations
which I was mentioned in last mail.====
mem_test()======...if (pid == 0)
child_loop_alloc();/* waits in the loop for all children finish allocating*/
while(pid_count < pid_cntr){
if (tst_timeout_remaining()< 15)
break;
Quoted 5 lines
sleep(1);
}if (dowrite) {
pre_free-post_freecheck...
This doesn't look very accurate. Other system processes can influence this.
Do we need to check it at all? If a child didn't complete allocation it's not going to signal parent.
Quoted 10 lines
} else {
...
}kill children here..
--
Regards,
Li Wang
RFC PATCH 1/1 mm: rewrite mtest01 with new API
Quoted 9 lines · Jan Stancek
Quoted 3 lines
if (dowrite) {
pre_free-post_freecheck...This doesn't look very accurate. Other system processes can influence this.
Do we need to check it at all? If a child didn't complete allocation it's not going to signal parent.
Sounds reasonable, if the parent exit from the loop correctly that means children finished their memory allocation.
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20190305/c97248e5/attachment.html>