Skip to contentSkip to conversation

Fuego Digest, Vol 13, Issue 14

Fuego dhinakar k 1 participant 1 message Aug 18, 2017 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

dhinakar k

author
#1 lore raw

Hi Tim,

Thanks a lot for the hard work and update about Fuego Release v1.2. I am eagerly waiting for it.

Regards,
Dhinakar

Quoted 334 lines · On Fri, Aug 18, 2017 at 8:19 AM, <[email protected]>…

Send Fuego mailing list submissions to
[email protected]

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.linuxfoundation.org/mailman/listinfo/fuego
or, via email, send a message with subject or body 'help' to
[email protected]

You can reach the person managing the list at
[email protected]

When replying, please edit your Subject line so it is more specific than "Re: Contents of Fuego digest..."

Today's Topics:

1. Re: Status of LTP parser - populating run.json (Daniel Sangorrin)
2. Status update for Fuego (Bird, Timothy)
3. Re: Status update for Fuego (Khiem Nguyen)

----------------------------------------------------------------------

Message: 1
Date: Thu, 10 Aug 2017 10:41:52 +0900
From: "Daniel Sangorrin" <[email protected]>
To: "'Bird, Timothy'" <[email protected]>,
<[email protected]>
Subject: Re: [Fuego] Status of LTP parser - populating run.json
Message-ID: <[email protected]>
Content-Type: text/plain; charset="UTF-8"

Quoted 13 lines

-----Original Message-----
From: Bird, Timothy [mailto:[email protected]]
Sent: Thursday, August 10, 2017 10:11 AM
To: Daniel Sangorrin; [email protected]
Subject: RE: Status of LTP parser - populating run.json

Quoted 5 lines

-----Original Message-----
From: Daniel Sangorrin on Wednesday, August 09, 2017 5:49 PM

Quoted 3 lines

-----Original Message-----
From: Bird, Timothy [mailto:[email protected]]
I've been doing some testing with LTP and the new parser. I'm only

seeing

Quoted 2 lines
Quoted 2 lines

one result show up

Quoted 1 line

in the run.json file. This is due to a bug in the process_data

routine. When

Quoted 8 lines
Quoted 8 lines

I made it backwards

Quoted 2 lines

compatible with the other functional tests, I broke it for LTP.

Don't worry, it was broken already by my changes and waiting for an upgrade.

Quoted 1 line

I think it would be better to move to the LTP parser.py program

calling

Quoted 2 lines
Quoted 2 lines

plib.process(), rather than

Quoted 1 line

process_data. Currently, only old-style functional tests call that

routine,

Quoted 2 lines
Quoted 2 lines

with a single value

Quoted 1 line

provided by generic-parser.py, and old-style benchmark tests, with

values

Quoted 15 lines
Quoted 4 lines

that can be made

Quoted 1 line

into measurements.

Yes, definitely.

It's fixed in my 'next' branch - at least LTP and other tests work for me (populating run.json with testcase status).

...

Quoted 5 lines

Thanks for the fixes.
Now that the parser seems to work for many tests, it would be good to upgrade the LTP to support
the new parser natively. The script already uses "test_category" which

will

Quoted 2 lines
Quoted 2 lines

become "test_set" and
"test_case" which has a direct equivalent. I hope it will not be too

much

Quoted 4 lines
Quoted 1 line

work.

I didn't change the variable names in LTP's parser.py, but I think it does what we want now. Actually, looking at it just now, I'm fine

leaving

Quoted 4 lines

it as "test_category". I'm not sure exactly what the LTP nomenclature for the thing is - I've seen the wording "test scenario" used.

LTP is a bit weird, they have multiple test defined in runtests which

can call

Quoted 1 line

lots of sub-tests. I found many different test scenarios (if that's the

right phrase)

Quoted 1 line

that call syscall-related test programs. So these things can be mixed

and matched.

Quoted 2 lines

I thought about adding the sub-test results as "measures" under the test

cases,

Quoted 1 line

but I'm holding off until I make more progress on the report generation.

I think they are fine as test_cases. Probably you are looking for nested results,
I already implemented a prototype and it was too complicated. The best solution I found
was to collapse that hierarchy in the test_set name through a convention.

For example: results[category1.subcategory2.subsubcategory3] = "PASS" This can be represented with a test_set called "category1.subcategory2" and a test_case called subsubcategory3.
Just make sure that test_sets do not have 'dots' in their names by convention.

Quoted 2 lines

As I'm working through things, there are a few items I might like to

propose

Quoted 2 lines

for the schema, but overall things seem to be working well.
-- Tim

Oh I didn't notice the latest commits. Thanks!

By the way, there is no reference.json or criteria.json files yet. Do you think we should add them?

- The reference.json is useful to know what tests were skipped either because
the result was CONF/BROK or because they were in the skiplist for example.
- The criteria.json would probably need to be customized for each board, but a default one may be useful as well.

Other than that, I think there is a way for the schema to allow adding new arbitrary data to an entry. With that, we can also add test-dependent information
such as the LTP errtype value, or the messages generated on test cases that ended up
in INFO/WARN/CONF/BROK for example. In the current schema, we have "attachments"
but this is not enough.

Thanks,
Daniel

------------------------------

Message: 2
Date: Fri, 18 Aug 2017 00:11:49 +0000
From: "Bird, Timothy" <[email protected]>
To: "[email protected]"
<[email protected]>
Subject: [Fuego] Status update for Fuego
Message-ID:

        <[email protected]>

Content-Type: text/plain; charset="us-ascii"

Hi everyone,

Over the last few days I did some proof-of-concept work on report generation for Fuego using the new unified output format. I also did some additional research on output formats produced by different test frameworks or harnesses.

I think I can safely say that our current format (as documented here
https://bitbucket.org/tbird20d/fuego-core/src/
fac7d69b91008ce0d1c7ccf0dc55dab5b8a1ce64/engine/scripts/ parser/fuego-schema.json?at=next
)

contains enough information for us to
be compatible with kernelci. I am confident we can convert our data to
xunit or TAP format as well, which are already supported for visualization
by Jenkins plugins.

There are a few little details remaining, but I think we can hash those out fairly soon. I'm trying to figure out what details need to be closed in order to begin the 1.2 release.

In terms of the big 4 goals for this release:
- unified output format - basically complete, although there may be some minor adjustments in the future
- LAVA support - this is getting pushed to a future release
- documentation update for 1.2 - I've already started this. There's a 1.2 version of the Fuego wiki, that I'll swap into place when the release occurs, but it needs a lot more work to document the differences from 1.1 to 1.2. This can be done during the 1.2-rc
series,
so is not technically blocking the start of the release.
- support AGL/JTA reports (HTML output)

There are also fixes to individual tests (such as a major fix for LTP and libaio-dev installation).

This support for AGL/JTA reports is something that needs work. Can someone provide me with sample HTML reports from AGL/JTA for the following:
1) Benchmark.dbench
2) Functional.hello_world
3) Functional.LTP?

I think with those three report samples, I can add the necessary support to my report generator (in ftc) to finish this feature.

Also, plotting seems to be broken at the moment. When Daniel gets back into town maybe he can take a look at this.

In separate news, I'll be presenting at Linaro Connect (SFO17) on the topic of shared standards between Linux test frameworks, and we'll have a few slots in Embedded Linux Conference Europe on testing and test framework topics.

After the 1.2 release, I plan to ramp up test production and Fuego promotion - possibly planning a mini-summit in Portland at ELC in March 2018.

Sorry this release has taken so long, but I think we're closing in on the 1.2 release (hopefully the first release candidate will be sometime next week.)
-- Tim

------------------------------

Message: 3
Date: Fri, 18 Aug 2017 02:49:26 +0000
From: Khiem Nguyen <[email protected]>
To: "Bird, Timothy" <[email protected]>
Cc: Phat Le <[email protected]>, Toru Oishi
<[email protected]>, "[email protected]"
<[email protected]>, Canh Nguyen
<[email protected]>
Subject: Re: [Fuego] Status update for Fuego
Message-ID:

        <PS1PR06MB16741040892FBB063080E9EDA8800@PS1PR06MB1674.

apcprd06.prod.outlook.com>

Content-Type: text/plain; charset="utf-8"

Hi Tim,

Thanks for sending the update.

Quoted 1 line

Over the last few days I did some proof-of-concept work on report

generation

Quoted 1 line

for Fuego using the new unified output format. I also did some

additional

Quoted 1 line

research on output formats produced by different test frameworks or

harnesses.

Quoted 4 lines

I think I can safely say that our current format (as documented here
https://bitbucket.org/tbird20d/fuego-
core/src/fac7d69b91008ce0d1c7ccf0dc55dab5b8a1ce64/engine/scripts/

parser/f

Quoted 4 lines

uego-schema.json?at=next
)
contains enough information for us to
be compatible with kernelci. I am confident we can convert our data to

xunit or

Quoted 1 line

TAP format as well, which are already supported for visualization by

Jenkins

Quoted 1 line

plugins.

Great.

Quoted 1 line

There are a few little details remaining, but I think we can hash those

out fairly

Quoted 1 line

soon. I'm trying to figure out what details need to be closed in order

to begin the

Quoted 10 lines

1.2 release.

In terms of the big 4 goals for this release:
- unified output format - basically complete, although there may be some minor adjustments in the future
- LAVA support - this is getting pushed to a future release
- documentation update for 1.2 - I've already started this. There's a 1.2 version of the Fuego wiki, that I'll swap into place when the release occurs, but it needs a lot more work to document the differences from 1.1 to 1.2. This can be done during the 1.2-rc

series,

Quoted 2 lines

so is not technically blocking the start of the release.
- support AGL/JTA reports (HTML output)

Those changes will reduce the gap between Fuego and AGL/JTA, I guess. :)

Quoted 1 line

In separate news, I'll be presenting at Linaro Connect (SFO17) on the

topic of

Quoted 1 line

shared standards between Linux test frameworks, and we'll have a few

slots in

Quoted 1 line

Embedded Linux Conference Europe on testing and test framework topics.

I personally expect more demonstration with Fuego and board setup.

Quoted 2 lines

After the 1.2 release, I plan to ramp up test production and Fuego

promotion -

Quoted 3 lines

possibly planning a mini-summit in Portland at ELC in March 2018.

Sorry this release has taken so long, but I think we're closing in on

the 1.2 release

Quoted 1 line

(hopefully the first release candidate will be sometime next week.)

Great.
We can try it out and give feedback.

Best regards,
KHIEM Nguyen

------------------------------

_______________________________________________
Fuego mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/fuego

End of Fuego Digest, Vol 13, Issue 14
*************************************