Fuego Digest, Vol 13, Issue 14
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Tim,
Thanks a lot for the hard work and update about Fuego Release v1.2. I am eagerly waiting for it.
Regards,
Dhinakar
Quoted 334 lines · On Fri, Aug 18, 2017 at 8:19 AM, <[email protected]>…
Send Fuego mailing list submissions to
[email protected]To subscribe or unsubscribe via the World Wide Web, visit
https://lists.linuxfoundation.org/mailman/listinfo/fuego
or, via email, send a message with subject or body 'help' to
[email protected]You can reach the person managing the list at
[email protected]When replying, please edit your Subject line so it is more specific than "Re: Contents of Fuego digest..."
Today's Topics:
1. Re: Status of LTP parser - populating run.json (Daniel Sangorrin)
2. Status update for Fuego (Bird, Timothy)
3. Re: Status update for Fuego (Khiem Nguyen)----------------------------------------------------------------------
Message: 1
Date: Thu, 10 Aug 2017 10:41:52 +0900
From: "Daniel Sangorrin" <[email protected]>
To: "'Bird, Timothy'" <[email protected]>,
<[email protected]>
Subject: Re: [Fuego] Status of LTP parser - populating run.json
Message-ID: <[email protected]>
Content-Type: text/plain; charset="UTF-8"Quoted 13 lines
-----Original Message-----
From: Bird, Timothy [mailto:[email protected]]
Sent: Thursday, August 10, 2017 10:11 AM
To: Daniel Sangorrin; [email protected]
Subject: RE: Status of LTP parser - populating run.jsonQuoted 5 lines
-----Original Message-----
From: Daniel Sangorrin on Wednesday, August 09, 2017 5:49 PMQuoted 3 lines
-----Original Message-----
From: Bird, Timothy [mailto:[email protected]]
I've been doing some testing with LTP and the new parser. I'm onlyseeing
Quoted 2 lines
Quoted 2 lines
one result show up
Quoted 1 line
in the run.json file. This is due to a bug in the
process_dataroutine. When
Quoted 8 lines
Quoted 8 lines
I made it backwards
Quoted 2 lines
compatible with the other functional tests, I broke it for LTP.
Don't worry, it was broken already by my changes and waiting for an upgrade.
Quoted 1 line
I think it would be better to move to the LTP
parser.pyprogramcalling
Quoted 2 lines
Quoted 2 lines
plib.process(), rather thanQuoted 1 line
process_data. Currently, only old-style functional tests call thatroutine,
Quoted 2 lines
Quoted 2 lines
with a single value
Quoted 1 line
provided by
generic-parser.py, and old-style benchmark tests, withvalues
Quoted 15 lines
Quoted 4 lines
that can be made
Quoted 1 line
into measurements.
Yes, definitely.
It's fixed in my 'next' branch - at least LTP and other tests work for me (populating run.json with testcase status).
...
Quoted 5 lines
Thanks for the fixes.
Now that the parser seems to work for many tests, it would be good to upgrade the LTP to support
the new parser natively. The script already uses "test_category" whichwill
Quoted 2 lines
Quoted 2 lines
become "
test_set" and
"test_case" which has a direct equivalent. I hope it will not be toomuch
Quoted 4 lines
Quoted 1 line
work.
I didn't change the variable names in LTP's
parser.py, but I think it does what we want now. Actually, looking at it just now, I'm fineleaving
Quoted 4 lines
it as "
test_category". I'm not sure exactly what the LTP nomenclature for the thing is - I've seen the wording "test scenario" used.LTP is a bit weird, they have multiple test defined in runtests which
can call
Quoted 1 line
lots of sub-tests. I found many different test scenarios (if that's the
right phrase)
Quoted 1 line
that call syscall-related test programs. So these things can be mixed
and matched.
Quoted 2 lines
I thought about adding the sub-test results as "measures" under the test
cases,
Quoted 1 line
but I'm holding off until I make more progress on the report generation.
I think they are fine as
test_cases. Probably you are looking for nested results,
I already implemented a prototype and it was too complicated. The best solution I found
was to collapse that hierarchy in thetest_setname through a convention.For example: results[category1.subcategory2.subsubcategory3] = "PASS" This can be represented with a
test_setcalled "category1.subcategory2" and atest_casecalled subsubcategory3.
Just make sure thattest_setsdo not have 'dots' in their names by convention.Quoted 2 lines
As I'm working through things, there are a few items I might like to
propose
Quoted 2 lines
for the schema, but overall things seem to be working well.
-- TimOh I didn't notice the latest commits. Thanks!
By the way, there is no reference.json or criteria.json files yet. Do you think we should add them?
- The reference.json is useful to know what tests were skipped either because
the result was CONF/BROK or because they were in the skiplist for example.
- The criteria.json would probably need to be customized for each board, but a default one may be useful as well.Other than that, I think there is a way for the schema to allow adding new arbitrary data to an entry. With that, we can also add test-dependent information
such as the LTP errtype value, or the messages generated on test cases that ended up
in INFO/WARN/CONF/BROK for example. In the current schema, we have "attachments"
but this is not enough.Thanks,
Daniel------------------------------
Message: 2
Date: Fri, 18 Aug 2017 00:11:49 +0000
From: "Bird, Timothy" <[email protected]>
To: "[email protected]"
<[email protected]>
Subject: [Fuego] Status update for Fuego
Message-ID:<[email protected]>Content-Type: text/plain; charset="us-ascii"
Hi everyone,
Over the last few days I did some proof-of-concept work on report generation for Fuego using the new unified output format. I also did some additional research on output formats produced by different test frameworks or harnesses.
I think I can safely say that our current format (as documented here
https://bitbucket.org/tbird20d/fuego-core/src/
fac7d69b91008ce0d1c7ccf0dc55dab5b8a1ce64/engine/scripts/ parser/fuego-schema.json?at=next
)contains enough information for us to
be compatible with kernelci. I am confident we can convert our data to
xunit or TAP format as well, which are already supported for visualization
by Jenkins plugins.There are a few little details remaining, but I think we can hash those out fairly soon. I'm trying to figure out what details need to be closed in order to begin the 1.2 release.
In terms of the big 4 goals for this release:
- unified output format - basically complete, although there may be some minor adjustments in the future
- LAVA support - this is getting pushed to a future release
- documentation update for 1.2 - I've already started this. There's a 1.2 version of the Fuego wiki, that I'll swap into place when the release occurs, but it needs a lot more work to document the differences from 1.1 to 1.2. This can be done during the 1.2-rc
series,
so is not technically blocking the start of the release.
- support AGL/JTA reports (HTML output)There are also fixes to individual tests (such as a major fix for LTP and libaio-dev installation).
This support for AGL/JTA reports is something that needs work. Can someone provide me with sample HTML reports from AGL/JTA for the following:
1) Benchmark.dbench
2) Functional.hello_world
3) Functional.LTP?I think with those three report samples, I can add the necessary support to my report generator (in ftc) to finish this feature.
Also, plotting seems to be broken at the moment. When Daniel gets back into town maybe he can take a look at this.
In separate news, I'll be presenting at Linaro Connect (SFO17) on the topic of shared standards between Linux test frameworks, and we'll have a few slots in Embedded Linux Conference Europe on testing and test framework topics.
After the 1.2 release, I plan to ramp up test production and Fuego promotion - possibly planning a mini-summit in Portland at ELC in March 2018.
Sorry this release has taken so long, but I think we're closing in on the 1.2 release (hopefully the first release candidate will be sometime next week.)
-- Tim------------------------------
Message: 3
Date: Fri, 18 Aug 2017 02:49:26 +0000
From: Khiem Nguyen <[email protected]>
To: "Bird, Timothy" <[email protected]>
Cc: Phat Le <[email protected]>, Toru Oishi
<[email protected]>, "[email protected]"
<[email protected]>, Canh Nguyen
<[email protected]>
Subject: Re: [Fuego] Status update for Fuego
Message-ID:<PS1PR06MB16741040892FBB063080E9EDA8800@PS1PR06MB1674.apcprd06.prod.outlook.com>
Content-Type: text/plain; charset="utf-8"
Hi Tim,
Thanks for sending the update.
Quoted 1 line
Over the last few days I did some proof-of-concept work on report
generation
Quoted 1 line
for Fuego using the new unified output format. I also did some
additional
Quoted 1 line
research on output formats produced by different test frameworks or
harnesses.
Quoted 4 lines
I think I can safely say that our current format (as documented here
https://bitbucket.org/tbird20d/fuego-
core/src/fac7d69b91008ce0d1c7ccf0dc55dab5b8a1ce64/engine/scripts/parser/f
Quoted 4 lines
uego-schema.json?at=next
)
contains enough information for us to
be compatible with kernelci. I am confident we can convert our data toxunit or
Quoted 1 line
TAP format as well, which are already supported for visualization by
Jenkins
Quoted 1 line
plugins.
Great.
Quoted 1 line
There are a few little details remaining, but I think we can hash those
out fairly
Quoted 1 line
soon. I'm trying to figure out what details need to be closed in order
to begin the
Quoted 10 lines
1.2 release.
In terms of the big 4 goals for this release:
- unified output format - basically complete, although there may be some minor adjustments in the future
- LAVA support - this is getting pushed to a future release
- documentation update for 1.2 - I've already started this. There's a 1.2 version of the Fuego wiki, that I'll swap into place when the release occurs, but it needs a lot more work to document the differences from 1.1 to 1.2. This can be done during the 1.2-rcseries,
Quoted 2 lines
so is not technically blocking the start of the release.
- support AGL/JTA reports (HTML output)Those changes will reduce the gap between Fuego and AGL/JTA, I guess. :)
Quoted 1 line
In separate news, I'll be presenting at Linaro Connect (SFO17) on the
topic of
Quoted 1 line
shared standards between Linux test frameworks, and we'll have a few
slots in
Quoted 1 line
Embedded Linux Conference Europe on testing and test framework topics.
I personally expect more demonstration with Fuego and board setup.
Quoted 2 lines
After the 1.2 release, I plan to ramp up test production and Fuego
promotion -
Quoted 3 lines
possibly planning a mini-summit in Portland at ELC in March 2018.
Sorry this release has taken so long, but I think we're closing in on
the 1.2 release
Quoted 1 line
(hopefully the first release candidate will be sometime next week.)
Great.
We can try it out and give feedback.Best regards,
KHIEM Nguyen------------------------------
_______________________________________________
Fuego mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/fuegoEnd of Fuego Digest, Vol 13, Issue 14
*************************************