Skip to contentSkip to conversation

ptest-runner: Add support timeout failure type to XML result file

PATCH In review Aníbal Limón 2 participants 5 messages Jan 3, 2019 – Jan 5, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ptest-runner: Add support timeout failure type to XML result file

The ptest-runner support logging results to stdout and to a XML file in stdout the ptest is mark as:

...
ERROR: Exit status is 1
TIMEOUT: ptest-directory
...

Add the same support in XML file for example,

...
<testcase classname='ptest-directory' name='run-ptest'>
<failure type='exit_code' message='run-ptest exited with code: 1'></failure> <failure type='timeout'/>
</testcase>
...

[YOCTO #13088]

---
 tests/data/reference.xml |  1 +
 tests/utils.c            |  4 ++--
 utils.c                  | 28 ++++++++++++++++------------
 utils.h                  |  2 +-
 4 files changed, 20 insertions(+), 15 deletions(-)
tests/data/reference.xml +1 -0
index 91522c7..17f91c2 100644
@@ -4,5 +4,6 @@
 	</testcase>
 	<testcase classname='test2' name='run-ptest'>
 		<failure type='exit_code' message='run-ptest exited with code: 1'></failure>
+		<failure type='timeout'/>
 	</testcase>
 </testsuite>
tests/utils.c +2 -2
index cf09379..662abe8 100644
@@ -257,8 +257,8 @@ START_TEST(test_xml_pass)
 	FILE *xp;
 	xp = xml_create(2, "./test.xml");
 	ck_assert(xp != NULL);
-	xml_add_case(xp, 0,"test1");
-	xml_add_case(xp, 1,"test2");
+	xml_add_case(xp, 0,"test1", 0);
+	xml_add_case(xp, 1,"test2", 1);
 	xml_finish(xp);
 
 	FILE *fp, *fr;
utils.c +16 -12
index ed2eff7..4a38ea1 100644
@@ -260,14 +260,13 @@ run_child(char *run_ptest, int fd_stdout, int fd_stderr)
 
 static inline int
 wait_child(const char *ptest_dir, const char *run_ptest, pid_t pid,
-		int timeout, int *fds, FILE **fps)
+		int timeout, int *fds, FILE **fps, int *timeouted)
 {
 	struct pollfd pfds[2];
 	struct timespec sentinel;
 	clockid_t clock = CLOCK_MONOTONIC;
 	int r;
 
-	int timeouted = 0;
 	int status;
 	int waitflags;
 
@@ -281,6 +280,8 @@ wait_child(const char *ptest_dir, const char *run_ptest, pid_t pid,
 		clock_gettime(clock, &sentinel);
 	}
 
+	*timeouted = 0;
+
 	while (1) {
 		waitflags = WNOHANG;
 
@@ -305,7 +306,7 @@ wait_child(const char *ptest_dir, const char *run_ptest, pid_t pid,
 
 			clock_gettime(clock, &time);
 			if ((time.tv_sec - sentinel.tv_sec) > timeout) {
-				timeouted = 1;
+				*timeouted = 1;
 				kill(pid, SIGKILL);
 				waitflags = 0;
 			}
@@ -315,11 +316,6 @@ wait_child(const char *ptest_dir, const char *run_ptest, pid_t pid,
 			break;
 	}
 
-	if (status) {
-		fprintf(fps[0], "\nERROR: Exit status is %d\n", status);
-		if (timeouted)
-			fprintf(fps[0], "TIMEOUT: %s\n", ptest_dir);
-	}
 
 	return status;
 }
@@ -337,6 +333,7 @@ run_ptests(struct ptest_list *head, const struct ptest_options opts,
 	pid_t child;
 	int pipefd_stdout[2];
 	int pipefd_stderr[2];
+	int timeouted;
 
 	if (opts.xml_filename) {
 		xh = xml_create(ptest_list_length(head), opts.xml_filename);
@@ -380,12 +377,17 @@ run_ptests(struct ptest_list *head, const struct ptest_options opts,
 				fprintf(fp, "BEGIN: %s\n", ptest_dir);
 
 				status = wait_child(ptest_dir, p->run_ptest, child,
-						opts.timeout, fds, fps);
-				if (status)
+						opts.timeout, fds, fps, &timeouted);
+
+				if (status) {
+					fprintf(fps[0], "\nERROR: Exit status is %d\n", status);
 					rc += 1;
+				}
+				if (timeouted)
+					fprintf(fps[0], "TIMEOUT: %s\n", ptest_dir);
 
 				if (opts.xml_filename)
-					xml_add_case(xh, status, ptest_dir);
+					xml_add_case(xh, status, ptest_dir, timeouted);
 
 				fprintf(fp, "END: %s\n", ptest_dir);
 				fprintf(fp, "%s\n", get_stime(stime, GET_STIME_BUF_SIZE));
@@ -424,7 +426,7 @@ xml_create(int test_count, char *xml_filename)
 }
 
 void
-xml_add_case(FILE *xh, int status, const char *ptest_dir)
+xml_add_case(FILE *xh, int status, const char *ptest_dir, int timeouted)
 {
 	fprintf(xh, "\t<testcase classname='%s' name='run-ptest'>\n", ptest_dir);
 
@@ -433,6 +435,8 @@ xml_add_case(FILE *xh, int status, const char *ptest_dir)
 		fprintf(xh, " message='run-ptest exited with code: %d'>", status);
 		fprintf(xh, "</failure>\n");
 	}
+	if (timeouted)
+		fprintf(xh, "\t\t<failure type='timeout'/>\n");
 
 	fprintf(xh, "\t</testcase>\n");
 }
utils.h +1 -1
index ee85163..880105f 100644
@@ -48,7 +48,7 @@ extern int run_ptests(struct ptest_list *, const struct ptest_options,
 		const char *, FILE *, FILE *);
 
 extern FILE *xml_create(int, char *);
-extern void xml_add_case(FILE *, int, const char *);
+extern void xml_add_case(FILE *, int, const char *, int);
 extern void xml_finish(FILE *);
 
 #endif
Signature · 1 line
2.19.2
#1 lore raw
Quoted 18 lines · Aníbal Limón

The ptest-runner support logging results to stdout and to a XML file in stdout the ptest is mark as:

...
ERROR: Exit status is 1
TIMEOUT: ptest-directory
...

Add the same support in XML file for example,

...
<testcase classname='ptest-directory' name='run-ptest'>
<failure type='exit_code' message='run-ptest exited with code:
1'></failure>
<failure type='timeout'/>
</testcase>
...

Quoted 3 lines

[YOCTO #13088]

Sounds good, thanks!

I have a feeling this will show up a few bugs but we should find and fix the ones which are timing out...

I'd still be interested in the time each directory takes too if we can get that as it would help us decide which tests to run.

Cheers,

Richard

#2 lore raw
Quoted 23 lines · On Thu, 3 Jan 2019 at 15:45, <[email protected]> wrote:
Quoted 18 lines · Aníbal Limón

The ptest-runner support logging results to stdout and to a XML file in stdout the ptest is mark as:

...
ERROR: Exit status is 1
TIMEOUT: ptest-directory
...

Add the same support in XML file for example,

...
<testcase classname='ptest-directory' name='run-ptest'>
<failure type='exit_code' message='run-ptest exited with code:
1'></failure>
<failure type='timeout'/>
</testcase>
...

Quoted 3 lines

[YOCTO #13088]

Quoted 8 lines

Sounds good, thanks!

I have a feeling this will show up a few bugs but we should find and fix the ones which are timing out...

I'd still be interested in the time each directory takes too if we can get that as it would help us decide which tests to run.

The timeout is set to 5 minutes (300 secs) by default and currently there is no way to set a timeout per ptest,
I will create other patch to add the timeout value to stdout and XML file.

Anibal

Quoted 6 lines

Cheers,

Richard

#3 lore raw
Quoted 34 lines · Anibal Limon
Quoted 30 lines · On Thu, 3 Jan 2019 at 15:45, <[email protected]> wrote:
Quoted 1 line · Aníbal Limón

The ptest-runner support logging results to stdout and to a XML

file

Quoted 17 lines

in stdout the ptest is mark as:

...
ERROR: Exit status is 1
TIMEOUT: ptest-directory
...

Add the same support in XML file for example,

...
<testcase classname='ptest-directory' name='run-ptest'>
<failure type='exit_code' message='run-ptest exited with code:
1'></failure>
<failure type='timeout'/>
</testcase>
...

Quoted 3 lines

[YOCTO #13088]

Sounds good, thanks!

I have a feeling this will show up a few bugs but we should find and
fix the ones which are timing out...

Quoted 8 lines
Quoted 3 lines

I'd still be interested in the time each directory takes too if we can
get that as it would help us decide which tests to run.

The timeout is set to 5 minutes (300 secs) by default and currently there is no way to set a timeout per ptest,
I will create other patch to add the timeout value to stdout and XML file.

That is handy but what I mean is the execution time each one takes. If we have test which take 5s its easier to run them regularly than tests which take 500s. Having an idea of their relative execution time would therefore be helpful.

I assume the timeout takes effect after no output for that period?

Cheers,

Richard

#4 lore raw
Quoted 44 lines · On Sat, 5 Jan 2019 at 12:52, <[email protected]> wrote:
Quoted 34 lines · Anibal Limon
Quoted 30 lines · On Thu, 3 Jan 2019 at 15:45, <[email protected]> wrote:
Quoted 1 line · Aníbal Limón

The ptest-runner support logging results to stdout and to a XML

file

Quoted 17 lines

in stdout the ptest is mark as:

...
ERROR: Exit status is 1
TIMEOUT: ptest-directory
...

Add the same support in XML file for example,

...
<testcase classname='ptest-directory' name='run-ptest'>
<failure type='exit_code' message='run-ptest exited with code:
1'></failure>
<failure type='timeout'/>
</testcase>
...

Quoted 3 lines

[YOCTO #13088]

Sounds good, thanks!

I have a feeling this will show up a few bugs but we should find and
fix the ones which are timing out...

Quoted 8 lines
Quoted 3 lines

I'd still be interested in the time each directory takes too if we can
get that as it would help us decide which tests to run.

The timeout is set to 5 minutes (300 secs) by default and currently there is no way to set a timeout per ptest,
I will create other patch to add the timeout value to stdout and XML file.

Quoted 5 lines

That is handy but what I mean is the execution time each one takes. If we have test which take 5s its easier to run them regularly than tests which take 500s. Having an idea of their relative execution time would therefore be helpful.

Got it, I can add a duration for example:.

In stdout will be (after end),

...
BEGIN: TIME
END: ptest-directory
DURATION: Ns
END: TIME
...

In XML:

...
<testcase classname='%s' name='run-ptest' duration='Ns'> </testcase>
...

or

...
<testcase classname='%s' name='run-ptest' >
<duration>Ns</duration>
</testcase>
...

What is the better? or any other?

Quoted 2 lines

I assume the timeout takes effect after no output for that period?

Right, the timeout is based on stdout data.

Anibal

Quoted 6 lines

Cheers,

Richard