Skip to contentSkip to conversation

mm: rewrite mtest01 with new API

RFC PATCH New Li Wang 2 participants 7 messages Feb 28, 2019 – Mar 5, 2019

lore
Share & tools
lore

Li Wang

author
#0 lore raw

RFC PATCH 1/1 mm: rewrite mtest01 with new API

Test issue:
mtest01 start many children to alloc chunck of memory and do write page(with -w option), but occasionally some children were killed by oom-killer and exit with SIGCHLD signal sending. After the parent reciving this SIGCHLD signal it will report FAIL as a test result.

It seems not a real kernel bug if something just like that, it's trying to use 80% of memory and swap. Once it uses most of memory, system starts swapping, but the test is likely consuming memory at greater rate than kswapd can provide, which eventually triggers OOM.

---- FAIL LOG ----

   mtest01     0  TINFO  :  Total memory already used on system = 1027392 kbytes
   mtest01     0  TINFO  :  Total memory used needed to reach maximum = 12715520 kbytes
   mtest01     0  TINFO  :  Filling up 80% of ram which is 11688128 kbytes
   mtest01     1  TFAIL  :  mtest01.c:314: child process exited unexpectedly

-------------------

Rewrite changes:
To make mtest01 more easier to understand, I just rewrite it into LTP new API and make a little changes in children behavior.

* drop the signal SIGCHLD action becasue new API help to check_child_status
* make child pause itself after finishing their memory allocating/writing
* parent sends SIGCONT to make children continue and exit
* decrease the pressure to 50% total ram+swap for testing

---
 runtest/mm                             |   4 +-
 testcases/kernel/mem/mtest01/mtest01.c | 430 ++++++++++++-------------
 2 files changed, 204 insertions(+), 230 deletions(-)
runtest/mm +2 -2
index a09f39c1e..93180e834 100644
@@ -9,8 +9,8 @@ mm02 mmap001
 # repetitive mmapping test.
 # Creates a one page map repetitively for one minute.
 
-mtest01 mtest01 -p80
-mtest01w mtest01 -p80 -w
+mtest01 mtest01 -p50
+mtest01w mtest01 -p50 -w
 
 #test for race conditions
 mtest05   mmstress
testcases/kernel/mem/mtest01/mtest01.c +202 -228

522 lines not on this page · raw message

Signature · 1 line
2.20.1
#1 lore raw

RFC PATCH 1/1 mm: rewrite mtest01 with new API

Hi,

----- Original Message -----

Quoted 10 lines

Test issue:
mtest01 start many children to alloc chunck of memory and do write page(with -w option), but occasionally some children were killed by oom-killer and exit with SIGCHLD signal sending. After the parent reciving this SIGCHLD signal it will report FAIL as a test result.

It seems not a real kernel bug if something just like that, it's trying to use 80% of memory and swap. Once it uses most of memory, system starts swapping, but the test is likely consuming memory at greater rate than kswapd can provide, which eventually triggers OOM.

This seems to be quite common on ppc systems (64k pages with slow I/O), so I do welcome fix/rewrite.

Quoted 14 lines

---- FAIL LOG ----
mtest01 0 TINFO : Total memory already used on system = 1027392
kbytes
mtest01 0 TINFO : Total memory used needed to reach maximum =
12715520 kbytes

   mtest01     0  TINFO  :  Filling up 80% of ram which is 11688128 kbytes
   mtest01     1  TFAIL  :  mtest01.c:314: child process exited unexpectedly

-------------------

Rewrite changes:
To make mtest01 more easier to understand, I just rewrite it into LTP new API and make a little changes in children behavior.

Quoted 5 lines

* drop the signal SIGCHLD action becasue new API help to check_child_status
* make child pause itself after finishing their memory allocating/writing
* parent sends SIGCONT to make children continue and exit
* decrease the pressure to 50% total ram+swap for testing

Current behaviour varies a lot depending on system. I'm thinking if we should just set it to 80% of free RAM. We already have number of OOM tests, so maybe we don't need to worry about memory pressure here too.

Quoted 7 lines
---
 runtest/mm                             |   4 +-
 testcases/kernel/mem/mtest01/mtest01.c | 430 ++++++++++++-------------
 2 files changed, 204 insertions(+), 230 deletions(-)

<snip>

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 204

Quoted 11 lines
testcases/kernel/mem/mtest01/mtest01.c +11 -0
+
+static void mem_test(void)
+{
+	int i, pid_cntr;
+	pid_t pid;
+	struct sigaction act;
+
+	act.sa_handler = handler;
+	act.sa_flags = 0;
+	sigemptyset(&act.sa_mask);
+	sigaction(SIGRTMIN, &act, 0);

I was thinking if we can't "abuse" tst_futexes a bit. It's a piece of shared memory we already have and could use for an atomic counter.

<snip>

Quoted 20 lines
testcases/kernel/mem/mtest01/mtest01.c +2 -17
+	if (pid == 0)
+		child_loop_alloc();
 
-		if (dowrite) {
-			/* Total Free Post-Test RAM */
-			post_mem =
-			    (unsigned long long)sstats.mem_unit *
-			    sstats.freeram;
-			post_mem =
-			    post_mem +
-			    (unsigned long long)sstats.mem_unit *
-			    sstats.freeswap;
-
-			while ((((unsigned long long)pre_mem - post_mem) <
-				(unsigned long long)original_maxbytes) &&
-			       pid_count < pid_cntr && !sigchld_count) {
-				sleep(1);
-				sysinfo(&sstats);
-				post_mem =
-				    (unsigned long long)sstats.mem_unit *

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 268

Quoted 10 lines
testcases/kernel/mem/mtest01/mtest01.c +3 -7
-				    sstats.freeram;
-				post_mem =
-				    post_mem +
-				    (unsigned long long)sstats.mem_unit *
-				    sstats.freeswap;
-			}
-		}
+	/* waits in the loop for all children finish allocating*/
+	while(pid_count < pid_cntr)
+		sleep(1);

What happens if one child hits OOM?

Quoted 33 lines
testcases/kernel/mem/mtest01/mtest01.c +5 -9
-		if (sigchld_count) {
-			tst_resm(TFAIL, "child process exited unexpectedly");
-		} else if (dowrite) {
-			tst_resm(TPASS, "%llu kbytes allocated and used.",
-				 original_maxbytes / 1024);
-		} else {
-			tst_resm(TPASS, "%llu kbytes allocated only.",
-				 original_maxbytes / 1024);
-		}
+	if (dowrite) {
+		sysinfo(&sstats);
+		/* Total Free Post-Test RAM */
+		post_mem = (unsigned long long)sstats.mem_unit * sstats.freeram;
+		post_mem = post_mem + (unsigned long long)sstats.mem_unit *

sstats.freeswap;

testcases/kernel/mem/mtest01/mtest01.c +15 -0
+		if (((pre_mem - post_mem) < original_maxbytes))
+			tst_res(TFAIL, "kbytes allocated and used less than expected %llu",
+					original_maxbytes / 1024);
+		else
+			tst_res(TPASS, "%llu kbytes allocated and used",
+					original_maxbytes / 1024);
+	} else {
+		tst_res(TPASS, "%llu kbytes allocated only",
+				original_maxbytes / 1024);
+	}
+
+	i = 0;
+	while (pid_list[i] > 0) {
+		kill(pid_list[i], SIGCONT);
+		i++;
 	}

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 299

Quoted 10 lines
testcases/kernel/mem/mtest01/mtest01.c +7 -2
-	cleanup();
-	tst_exit();
 }
+
+static struct tst_test test = {
+	.forks_child = 1,
+	.options = mtest_options,
+	.setup = setup,
+	.cleanup = cleanup,
+	.test_all = mem_test,

Is default timeout going to work on large boxes (256GB+ RAM)?

Thinking loud, what if...
- we define at the start of test how much memory we want to allocate (target == 80% of free RAM)
- we allocate a shared memory for counter, that each child increases
as it allocates memory (progress)
(or we abuse tst_futexes)
we could use tst_atomic_add_return() to count allocated chunks globally
- once child finishes allocation it will pause()
- we set timeout to ~3 minutes
- main process runs in loop, sleeps, and periodically checks
- if progress reached target, PASS, break
- if progress hasn't increased in last 15 seconds, FAIL, break
- if we are 15 seconds away from timeout, end test early, PASS, break
(reason is to avoid running too long on big boxes)
- kill all children, exit

Regards,
Jan

author
#2 lore raw

RFC PATCH 1/1 mm: rewrite mtest01 with new API

On Fri, Mar 1, 2019 at 6:09 AM Jan Stancek <[email protected]> wrote:

Current behaviour varies a lot depending on system. I'm thinking if we

Quoted 4 lines

should
just set it to 80% of free RAM. We already have number of OOM tests, so maybe we don't need to worry about memory pressure here too.

Yes, I'm ok with that change. Since if we decrease the allocated consumption to 50% mem+swap, that probably only do allocating in the part of free mem too.

Quoted 9 lines

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 204

Quoted 5 lines
testcases/kernel/mem/mtest01/mtest01.c +5 -0
+
+     act.sa_handler = handler;
+     act.sa_flags = 0;
+     sigemptyset(&act.sa_mask);
+     sigaction(SIGRTMIN, &act, 0);

I was thinking if we can't "abuse" tst_futexes a bit. It's a piece of shared memory we already have and could use for an atomic counter.

Quoted 8 lines

<snip>

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 268

Quoted 3 lines
testcases/kernel/mem/mtest01/mtest01.c +3 -0
+     /* waits in the loop for all children finish allocating*/
+     while(pid_count < pid_cntr)
+             sleep(1);

What happens if one child hits OOM?

LTP new API does wait and check child status for the test, if one child_A(allocating finished and status paused) hits OOM, it will just break and report status, but that's ok for this event, because other children which still allocating will keep running after system reclaiming memory from child_A. So parent process will recieve all of children's SIGRTMIN signal and break from the while loop correctly.

Anthoer situation(I haven't hit), is one child_B(still allocating and not finishes) was killed by OOM. that will make parent fall into an infinite loop here. From OOM mechanism, oom-killer likes to choose high score process, so this situation maybe not easy to reproduce. But that not mean it will not, since oom-killer is not perfect.

Anyway, to avoid the second situation occuring, I'd like to take you advice to make parent exiting loop safly with many check actions.

Quoted 40 lines

Review

Quoted 3 lines
(quoted diff) +0 -2
-             if (sigchld_count) {
-                     tst_resm(TFAIL, "child process exited

unexpectedly");

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[])

Quoted 11 lines
testcases/kernel/mem/mtest01/mtest01.c +4 -7
-             } else if (dowrite) {
-                     tst_resm(TPASS, "%llu kbytes allocated and used.",
-                              original_maxbytes / 1024);
-             } else {
-                     tst_resm(TPASS, "%llu kbytes allocated only.",
-                              original_maxbytes / 1024);
-             }
+     if (dowrite) {
+             sysinfo(&sstats);
+             /* Total Free Post-Test RAM */
+             post_mem = (unsigned long long)sstats.mem_unit *

sstats.freeram;

Review

Quoted 5 lines

+ post_mem = post_mem + (unsigned long long)sstats.mem_unit *
sstats.freeswap;

(quoted diff) +2 -0
+             if (((pre_mem - post_mem) < original_maxbytes))
+                     tst_res(TFAIL, "kbytes allocated and used less

than expected %llu",

Quoted 18 lines
testcases/kernel/mem/mtest01/mtest01.c +14 -2
+                                     original_maxbytes / 1024);
+             else
+                     tst_res(TPASS, "%llu kbytes allocated and used",
+                                     original_maxbytes / 1024);
+     } else {
+             tst_res(TPASS, "%llu kbytes allocated only",
+                             original_maxbytes / 1024);
+     }
+
+     i = 0;
+     while (pid_list[i] > 0) {
+             kill(pid_list[i], SIGCONT);
+             i++;
      }
-     cleanup();
-     tst_exit();
 }
+
Quoted 9 lines

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 299

Quoted 6 lines
testcases/kernel/mem/mtest01/mtest01.c +6 -0
+static struct tst_test test = {
+     .forks_child = 1,
+     .options = mtest_options,
+     .setup = setup,
+     .cleanup = cleanup,
+     .test_all = mem_test,

Is default timeout going to work on large boxes (256GB+ RAM)?

No.

I had the same worries before, but in this test, the number of children(max_pids) will be increased dynamically with the system total memory size. And each child allocating won't beyond the 'alloc_bytes' (alloc_bytes = MIN(THREE_GB, alloc_maxbytes)) limitaion, so an extra time consumption part is just by forking, but from my evaluation on a 4T ram system, mtest01 finishes very faster(99% mem+swap, 2m22sec) than I expected. So the default timeout is not trigger at all.

# cat /proc/meminfo  | grep Mem
MemTotal:       4227087524 kB
MemFree:        4223159948 kB
MemAvailable:   4213257308 kB
# time ./mtest01 -p99 -w
tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s
mtest01.c:113: INFO: Total memory already used on system = 3880348 kbytes
mtest01.c:120: INFO: Total memory used needed to reach maximum = 4188969005
kbytes
mtest01.c:134: INFO: Filling up 99% of ram which is 4185088657 kbytes
...
mtest01.c:185: INFO: ... 3221225472 bytes allocated and used in child 41779
mtest01.c:281: PASS: 4185132681 kbytes allocated and used
...

real 2m22.213s
user 79m52.390s
sys 351m56.059s

Quoted 18 lines

Thinking loud, what if...
- we define at the start of test how much memory we want to allocate (target == 80% of free RAM)
- we allocate a shared memory for counter, that each child increases as it allocates memory (progress)
(or we abuse tst_futexes)
we could use tst_atomic_add_return() to count allocated chunks globally
- once child finishes allocation it will pause()
- we set timeout to ~3 minutes
- main process runs in loop, sleeps, and periodically checks
- if progress reached target, PASS, break
- if progress hasn't increased in last 15 seconds, FAIL, break
- if we are 15 seconds away from timeout, end test early, PASS, break (reason is to avoid running too long on big boxes)
- kill all children, exit

Real good suggestions, I will try to take some of them in V2.

Quoted 3 lines

Regards,
Jan

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20190301/65522666/attachment.html>
#3 lore raw

RFC PATCH 1/1 mm: rewrite mtest01 with new API

----- Original Message -----

Quoted 147 lines · Jan Stancek

Current behaviour varies a lot depending on system. I'm thinking if we

Quoted 4 lines

should
just set it to 80% of free RAM. We already have number of OOM tests, so maybe we don't need to worry about memory pressure here too.

Yes, I'm ok with that change. Since if we decrease the allocated consumption to 50% mem+swap, that probably only do allocating in the part of free mem too.

Quoted 9 lines

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 204

Quoted 5 lines
testcases/kernel/mem/mtest01/mtest01.c +5 -0
+
+     act.sa_handler = handler;
+     act.sa_flags = 0;
+     sigemptyset(&act.sa_mask);
+     sigaction(SIGRTMIN, &act, 0);

I was thinking if we can't "abuse" tst_futexes a bit. It's a piece of shared memory we already have and could use for an atomic counter.

Quoted 8 lines

<snip>

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 268

Quoted 3 lines
testcases/kernel/mem/mtest01/mtest01.c +3 -0
+     /* waits in the loop for all children finish allocating*/
+     while(pid_count < pid_cntr)
+             sleep(1);

What happens if one child hits OOM?

LTP new API does wait and check child status for the test, if one child_A(allocating finished and status paused) hits OOM, it will just break and report status, but that's ok for this event, because other children which still allocating will keep running after system reclaiming memory from child_A. So parent process will recieve all of children's SIGRTMIN signal and break from the while loop correctly.

Anthoer situation(I haven't hit), is one child_B(still allocating and not finishes) was killed by OOM. that will make parent fall into an infinite loop here. From OOM mechanism, oom-killer likes to choose high score process, so this situation maybe not easy to reproduce. But that not mean it will not, since oom-killer is not perfect.

Anyway, to avoid the second situation occuring, I'd like to take you advice to make parent exiting loop safly with many check actions.

Quoted 40 lines

Review

Quoted 3 lines
(quoted diff) +0 -2
-             if (sigchld_count) {
-                     tst_resm(TFAIL, "child process exited

unexpectedly");

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[])

Quoted 11 lines
testcases/kernel/mem/mtest01/mtest01.c +4 -7
-             } else if (dowrite) {
-                     tst_resm(TPASS, "%llu kbytes allocated and used.",
-                              original_maxbytes / 1024);
-             } else {
-                     tst_resm(TPASS, "%llu kbytes allocated only.",
-                              original_maxbytes / 1024);
-             }
+     if (dowrite) {
+             sysinfo(&sstats);
+             /* Total Free Post-Test RAM */
+             post_mem = (unsigned long long)sstats.mem_unit *

sstats.freeram;

Review

Quoted 5 lines

+ post_mem = post_mem + (unsigned long long)sstats.mem_unit *
sstats.freeswap;

(quoted diff) +2 -0
+             if (((pre_mem - post_mem) < original_maxbytes))
+                     tst_res(TFAIL, "kbytes allocated and used less

than expected %llu",

Quoted 18 lines
testcases/kernel/mem/mtest01/mtest01.c +14 -2
+                                     original_maxbytes / 1024);
+             else
+                     tst_res(TPASS, "%llu kbytes allocated and used",
+                                     original_maxbytes / 1024);
+     } else {
+             tst_res(TPASS, "%llu kbytes allocated only",
+                             original_maxbytes / 1024);
+     }
+
+     i = 0;
+     while (pid_list[i] > 0) {
+             kill(pid_list[i], SIGCONT);
+             i++;
      }
-     cleanup();
-     tst_exit();
 }
+
Quoted 9 lines

testcases/kernel/mem/mtest01/mtest01.c int main(int argc, char *argv[]) · line 299

Quoted 6 lines
testcases/kernel/mem/mtest01/mtest01.c +6 -0
+static struct tst_test test = {
+     .forks_child = 1,
+     .options = mtest_options,
+     .setup = setup,
+     .cleanup = cleanup,
+     .test_all = mem_test,

Is default timeout going to work on large boxes (256GB+ RAM)?

No.

I had the same worries before, but in this test, the number of children(max_pids) will be increased dynamically with the system total memory size. And each child allocating won't beyond the 'alloc_bytes' (alloc_bytes = MIN(THREE_GB, alloc_maxbytes)) limitaion, so an extra time consumption part is just by forking, but from my evaluation on a 4T ram system, mtest01 finishes very faster(99% mem+swap, 2m22sec) than I expected. So the default timeout is not trigger at all.

# cat /proc/meminfo  | grep Mem
MemTotal:       4227087524 kB
MemFree:        4223159948 kB
MemAvailable:   4213257308 kB
# time ./mtest01 -p99 -w
tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s
mtest01.c:113: INFO: Total memory already used on system = 3880348 kbytes
mtest01.c:120: INFO: Total memory used needed to reach maximum = 4188969005
kbytes
mtest01.c:134: INFO: Filling up 99% of ram which is 4185088657 kbytes
...
mtest01.c:185: INFO: ... 3221225472 bytes allocated and used in child 41779
mtest01.c:281: PASS: 4185132681 kbytes allocated and used
...

real 2m22.213s
user 79m52.390s
sys 351m56.059s

Quoted 18 lines

Thinking loud, what if...
- we define at the start of test how much memory we want to allocate (target == 80% of free RAM)
- we allocate a shared memory for counter, that each child increases as it allocates memory (progress)
(or we abuse tst_futexes)
we could use tst_atomic_add_return() to count allocated chunks globally
- once child finishes allocation it will pause()
- we set timeout to ~3 minutes
- main process runs in loop, sleeps, and periodically checks
- if progress reached target, PASS, break
- if progress hasn't increased in last 15 seconds, FAIL, break
- if we are 15 seconds away from timeout, end test early, PASS, break (reason is to avoid running too long on big boxes)
- kill all children, exit

Quoted 1 line

Real good suggestions, I will try to take some of them in V2.

Maybe give it few days, so other people can respond, if they like/don't like going in this direction.

Quoted 11 lines
Quoted 5 lines

Regards,
Jan

--
Regards,
Li Wang

author
#4 lore raw

RFC PATCH 1/1 mm: rewrite mtest01 with new API

On Fri, Mar 1, 2019 at 4:03 PM Jan Stancek <[email protected]> wrote:

Maybe give it few days, so other people can respond, if they like/don't like

Quoted 2 lines

going in this direction.

Sure.

Btw, I'm not going to use shared memory here since that probably make things more complicated.
I guess just do remaining time check in the parent while loop is enough for the two OOM situations
which I was mentioned in last mail.

==== mem_test() ======

        ...

if (pid == 0)
child_loop_alloc();

/* waits in the loop for all children finish allocating*/ while(pid_count < pid_cntr) {
if (tst_timeout_remaining() < 15)
break;

sleep(1);
}

if (dowrite) {
pre_free - post_free check...
} else {
...
}

kill children here..

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20190301/6952d779/attachment-0001.html>
#5 lore raw

RFC PATCH 1/1 mm: rewrite mtest01 with new API

----- Original Message -----

Quoted 25 lines · Jan Stancek

Maybe give it few days, so other people can respond, if they like/don't like

Quoted 2 lines

going in this direction.

Sure.

Btw, I'm not going to use shared memory here since that probably make things more complicated.
I guess just do remaining time check in the parent while loop is enough for the two OOM situations
which I was mentioned in last mail.

==== mem_test() ======

        ...

if (pid == 0)
child_loop_alloc();

/* waits in the loop for all children finish allocating*/ while(pid_count < pid_cntr) {
if (tst_timeout_remaining() < 15)
break;

Quoted 5 lines

sleep(1);
}

if (dowrite) {
pre_free - post_free check...

This doesn't look very accurate. Other system processes can influence this.

Do we need to check it at all? If a child didn't complete allocation it's not going to signal parent.

Quoted 10 lines

} else {
...
}

kill children here..

--
Regards,
Li Wang

author
#6 lore raw

RFC PATCH 1/1 mm: rewrite mtest01 with new API

Quoted 9 lines · Jan Stancek
Quoted 3 lines

if (dowrite) {
pre_free - post_free check...

This doesn't look very accurate. Other system processes can influence this.

Do we need to check it at all? If a child didn't complete allocation it's not going to signal parent.

Sounds reasonable, if the parent exit from the loop correctly that means children finished their memory allocation.

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20190305/c97248e5/attachment.html>