Skip to contentSkip to conversation

The Linux Test Project has been Released for June 2007,

LTP Randy Dunlap 2 participants 2 messages Jul 2, 2007 – Sep 18, 2007 1 message not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 11 lines · On Mon, 02 Jul 2007 12:28:50 +0530 Subrata Modak wrote:

:-) *Dear All*,

*Note(s) from the Maintainer: *
* We have seen that the LTP mailing lists has become active for the last couple of months. Thanks to all who has made this happen. We as an integrated team can work hard to make this Project a resounding success. Following are the different milestones/activities that we aim to perform/achieve in the coming months:
1) Investigation into LTP Log(s) Library to make this more meaningful and parsible. Kindly provide your inputs for the output that LTP generates now and whether/how you would like it to be in future

I don't know if this qualifies as an input for the output, but I use a script to parse the results file to compute a gross/overall "SCORE" value. Instead of looking at lines that contain "termination_id" (e.g.:
duration=0 termination_type=exited termination_id=0 corefile=no ), I look at the test results output format messages and count them (ignoring the INFO lines) and then compute a SCORE based on all of those lines. Obviously that's why I've been changing several test cases to use the standard test results output format.

ltprun-summary.sh is attached. It uses prtag2tag.pl, which is also attached. (I also do this summary SCORE for the Open Posix Test suite.)

Example summary + SCORE:

Using logfile=/cruc/runs/1789/test_output/ltp.log

build errors:  0 lines
PASS:          2225
WARNing:       1
BROKen:        1
RETiRed:       0
FAIL:          0
CONFig error:  11
SCORE.ltp:        99.5
Quoted 1 line

*Please also see the ChangeLog Below (June 2007):* *

so "[PATCH] float tests: use standard test results output"
(http://sourceforge.net/mailarchive/forum.php?thread_name=20070622214652.d137d055.rdunlap%40xenotime.net&forum_name=ltp-list)
will be added next month?

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

#2 lore raw

[LTP] Validating exact PASS and FAIL scenarios for LTP

Hi All,

It may be a little high time for us to visit the way LTP defines itś PASS and FAILURES. We know that each of LTP Test cases reports certain keywords. Now, we need to define properly how each of these Keywords needs to be interpreted in context to the addition information these test-cases report, so that, we can arrive at saying that:
1) this information means that TEST has passed (and in such scenario the test-case is bound to exit(0) to pan), and,
2) That set of information means that the TEST has failed (and in such scenario the test-case is bound to exit(non-zero) to pan)

There are several reported incidents where there is ambiguity in judging whether the test is a PASS or FAIL. Now we have come to a point where we need to discuss and debate this out and say that:
1) These are the situations where the test should report exit(0) to pan irrespective of what it has reported to logs through INFO:, and,
2) These are the situations where the test should immediately terminate with exit(non-zero) to pan.

We also need to decide where these keywords like
PASS
WARN
BROK
FAIL
RETR
CONF
INFO
should be placed, and under what context ??

I hope all of you will contribute to this debate, where we finally seal LTP´s fate as a:
1) Broken test-suite, or,
2) a glorious one.

--Regards--
Subrata

Quoted 34 lines · Randy Dunlap
Quoted 4 lines · On Mon, 02 Jul 2007 12:28:50 +0530 Subrata Modak wrote:

:-) *Dear All*,

*Note(s) from the Maintainer: *
generates now and whether/how you would like it to be in future

I don't know if this qualifies as an input for the output, but

ltprun-summary.sh is attached. It uses prtag2tag.pl, which is also attached. (I also do this summary SCORE for the Open Posix Test suite.)

Example summary + SCORE:

Using logfile=/cruc/runs/1789/test_output/ltp.log

build errors:  0 lines
PASS:          2225
WARNing:       1
BROKen:        1
RETiRed:       0
FAIL:          0
CONFig error:  11
SCORE.ltp:        99.5
Quoted 1 line

*Please also see the ChangeLog Below (June 2007):* *

so "[PATCH] float tests: use standard test results output"
(http://sourceforge.net/mailarchive/forum.php?thread_name=20070622214652.d137d055.rdunlap%40xenotime.net&forum_name=ltp-list)
will be added next month?

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***