status update
Hi all,
I've working on the parsing code. Here is a list of things that I managed to complete and things that need some discussion:
Completed tasks:
- Output a single run.json file for each job's run/build that captures both the metadata and the test results.
- Merge run.json files for a test_suite into a single results.json file that flot can visualize.
+ I have added concurrency locks to protect results.json from concurrent writes.
- Add HTML output support to flot (similar, but not exactly the same yet, to the one in AGL JTA).
- Fixed the test Functional.tiff (AGL test) and confirmed that it works on docker and beaglebone black.
+ There are many more AGL tests to fix.
- Fixed several bug fixes that occurred when a test fails in an unexpected way.
Discarded tasks:
- Output a jUnit XML file so that the plugin "Test Results Analyzer" can display.
+ This is working but it isn't as flexible as I'd like. The new flot's HTML output support that I added should deprecate it.
- Ability to download a PNG/SVG from the flot plugin directly.
+ I managed to get this working by using the canvas2image library or using the canvas.toDataURL interface. Unfortunately, flot doesn't store the axes' information in the canvas so only the plotting space is saved. There is a library to accomplish this task [1] that but there seems to be a version mismatch with the javascript libraries in fuego and it didn't work. I decided to postpone this.
Pending tasks:
- Add report functionality
+ I have removed the generation of plot.png files. In part because I want to do that directly from the flot plugin, and in part because I think it is more useful if we integrate it in the future ftc report command,
- Add more information to the run.json files.
+ I am trying to produce a schema that is very close to the one used in Kernel.CI. Probably I can make it compatible.
+ There is some information that needs to be added to Fuego. Unfortunately, I will probably have to fix a lot of files:
1) Each test_case (remember test_suite > test_set > test_case) should be able to store a list of measurements. Each measurement would consist of a name, value, units, duration and maybe more such as error messages specific to the test_case or expected values (e.g. expected to fail, or expected to be greater than 50).
2) Each test_case should have a status (PASS, FAIL..) that is not necessarily the same as the test_set/suite status.
+ Add vcs_commit information (git or tarball information)
- Handle failed runs better. Sometimes the test fails very early, before even building it.
- I am not sure what to do with the "reference.log" files
+ Currently they are used to store thresholds, but these are probably board dependent.
+ This is probably related to the discussion with Rafael about parameterized builds. We should
be able to define the threshold for a specific test_case's measure.
- Remove testplans?
+ I was thinking that we can substitute testplans by custom scripts that call ftc add-jobs
- Create a staging folder for tests that do not work or files that are not used.
+ Or maybe at least list them up on the wiki.
Thanks,
Daniel
Quoted 5 lines
-----Original Message-----
From: Daniel Sangorrin on Monday, June 26, 2017 11:04 PM
Quoted 7 lines
I've working on the parsing code. Here is a list of things that I managed to complete
and things that need some discussion:Completed tasks:
- Output a single run.json file for each job's run/build that captures both the metadata and the test results.
Sounds good. This is probably the right approach.
Quoted 2 lines
- Merge run.json files for a
test_suiteinto a single results.json file that flot can visualize.
So, just to clarify, the run.json file has the results for a multiple metrics for a single execution (or 'run') of a test, and the results.json file has results for multiple runs of the test? That is, results.json has results for a single Fuego host with results from different runs and on different boards?
Quoted 9 lines
+ I have added concurrency locks to protect results.json from concurrent
writes.
- Add HTML output support to flot (similar, but not exactly the same yet, to the one in AGL JTA).
- Fixed the test Functional.tiff (AGL test) and confirmed that it works on docker and beaglebone black.
+ There are many more AGL tests to fix.
- Fixed several bug fixes that occurred when a test fails in an unexpected way.
Sounds good. Thanks!
Quoted 5 lines
Discarded tasks:
- Output a jUnit XML file so that the plugin "Test Results Analyzer" can display.
+ This is working but it isn't as flexible as I'd like. The new flot's HTML output
support that I added should deprecate it.
OK.
Quoted 7 lines
- Ability to download a PNG/SVG from the flot plugin directly.
+ I managed to get this working by using the canvas2image library or using
the canvas.toDataURL interface. Unfortunately, flot doesn't store the axes' information in the canvas so only the plotting space is saved. There is a library to accomplish this task [1] that but there seems to be a version mismatch with the javascript libraries in fuego and it didn't work. I decided to postpone this.
OK.
Quoted 10 lines
Pending tasks:
- Add report functionality
+ I have removed the generation of plot.png files. In part because I want to
do that directly from the
flot plugin, and in part because I think it is more useful if we integrate it in
the future ftc report command,
- Add more information to the run.json files.
+ I am trying to produce a schema that is very close to the one used in
Kernel.CI. Probably I can make it compatible.
That sounds good. I'm very interested in the schema. I believe that Milo Casagrande mentioned something about groups, that I don't think we have yet. Everything else in your analysis from April
(https://lists.linuxfoundation.org/pipermail/fuego/2017-April/000448.html)
I think shows some analog between Fuego and KernelCI fields.
Quoted 9 lines
+ There is some information that needs to be added to Fuego.
Unfortunately, I will probably have to fix a lot of files:
1) Eachtest_case(remembertest_suite>test_set>test_case) should
be able to store a list of
measurements. Each measurement would consist of a name, value,
units, duration and maybe more
such as error messages specific to thetest_caseor expected values
(e.g. expected to fail, or expected to
be greater than 50).
I think this needs to be somewhere, but possibly not in the results schema. For example, I don't want every listing of dbench results to have to report the units for each benchmark metric. These should be fairly static and we should be able to define them per-test, not per-run. Things like thresholds are a bit different, and we may need to record them per-run, since the success/failure threshold could be different depending on the board, or even changed by the user to fine-tune the regression-checking.
Quoted 2 lines
2) Each
test_caseshould have a status (PASS, FAIL..) that is not
necessarily the same as the test_set/suite status.
Agreed.
Quoted 1 line
+ Add
vcs_commitinformation (git or tarball information)
This is expected to be captured by the test version (in test.yaml), and the test version should be saved in the run.json file.
But I could see saving the test source code version in the run.json file also.
Quoted 8 lines
- Handle failed runs better. Sometimes the test fails very early, before even building it.
- I am not sure what to do with the "reference.log" files
+ Currently they are used to store thresholds, but these are probably board
dependent.
+ This is probably related to the discussion with Rafael about parameterized
builds. We should
be able to define the threshold for a specific test_case's measure.
reference logs should be savable by the user, to compare with future runs. The system we have now uses parsed testlogs, which are generated using log_compare and very simple line-based regular-expresison filters (using 'grep'). It will be much more flexible and robust to compare run.json files instead of a parsed log and a reference log.
The purpose of these is to save the data from a "known good" run, so that regressions can be detected when the data from a current run differs from that. This can include sub-test failures, that we have decided to ignore or postpone resolution of.
I think once we have in place a system to save all the sub-test metric data from the testlogs (using a parser) in json format, then we can eliminate these. We should be able to replace reference.log with reference.json (which is just a saved run.json file). This is a key thing that I would like Fuego to be able to share easily between developers (and sites).
I have already started working on a json difference program (called 'jdiff') to compare 2 json files and report the differences between the two.
On the issue of where to save them, currently they should be saved somehow at the 'board' level. That is, tests will definitely have different results per-board. But there may be such a thing as a reference file that is dependent on the kernel version, or the distribution, or some other parameter. We should discuss the naming and storage of these.
Quoted 3 lines
- Remove testplans?
+ I was thinking that we can substitute testplans by custom scripts that call
ftc add-jobs
Why do we want to remove them? I think they serve a useful function - expressing a set of tests (with their specs) to execute in sequence.
Quoted 2 lines
- Create a staging folder for tests that do not work or files that are not used.
+ Or maybe at least list them up on the wiki.
Currently, if they are not listed in a testplan, the tests are functionally 'dead'. (Although a user can create a job for a single test and try it out). Maybe it would be good to have a 'staging' folder for tests that are under development or conversion (like a lot of the AGL tests). I agree that we should have some notion of the "approved and likely to work" tests, and that should be expressed somehow in the test placement or in documentation.
-- Tim
Hi Tim,
Quoted 10 lines
-----Original Message-----
From: Bird, Timothy [mailto:[email protected]]
Sent: Wednesday, June 28, 2017 9:47 AMQuoted 2 lines
- Merge run.json files for a
test_suiteinto a single results.json file that flot can visualize.So, just to clarify, the run.json file has the results for a multiple metrics for a single execution (or 'run') of a test, and the results.json file has results for multiple runs of the test? That is, results.json has results for a single Fuego host with results from different runs and on different boards?
Yes, that's correct. It's kind of a text "database". The size of this file will increase with the number of runs so it may not scale. Probably we will need to implement a logrotate-like functionality once we can submit all runs to a proper database in a centralized server (maybe one running kernelci or something similar).
Quoted 10 lines
Quoted 9 lines
Pending tasks:
- Add report functionality
+ I have removed the generation of plot.png files. In part because I want to
do that directly from the
flot plugin, and in part because I think it is more useful if we integrate it in
the future ftc report command,
- Add more information to the run.json files.
+ I am trying to produce a schema that is very close to the one used in
Kernel.CI. Probably I can make it compatible.
Quoted 5 lines
That sounds good. I'm very interested in the schema. I believe that Milo Casagrande mentioned something about groups, that I don't think we have yet. Everything else in your analysis from April
(https://lists.linuxfoundation.org/pipermail/fuego/2017-April/000448.html)
I think shows some analog between Fuego and KernelCI fields.
Groups are called "test_sets" in kernel CI and can contain an array of "test_cases". # I'm thinking about using kernel ci's nomenclature for this. What do you think? # I also want to rename platform to toolchain, fwver to kernel_version, spec to # test_spec, testplan to test_plan...
In Fuego, we have a similar concept. If you see bonnie's reference.log [1] you will notice that there are several groups/test_sets and multiple tests/test_cases inside each group:
test_set: Sequential_Outputtest_cases: Block, PerChr, Rewrite
I am making an schema that is compatible with Kernel CI but that it will also allow having test_sets inside test_sets. For example: "LTP > RT tests > func > sched_jitter" contains 2-levels of test_sets (RT tests and func).
[Note] In the past, tests.info used to store this test_set > test_case information. Now this information is actually provided by the test's parser.py and reference.log. The parser.py's information includes information for the test_sets/test_cases that were actually executed (depends on the spec), whereas reference.log contains all possible test_sets/test_cases for that test (including those that were skipped somehow because of the selected test_spec). We need to decouple the reference thresholds from this information though.
Quoted 9 lines
Quoted 9 lines
+ There is some information that needs to be added to Fuego.
Unfortunately, I will probably have to fix a lot of files:
1) Eachtest_case(remembertest_suite>test_set>test_case) should
be able to store a list of
measurements. Each measurement would consist of a name, value,
units, duration and maybe more
such as error messages specific to thetest_caseor expected values
(e.g. expected to fail, or expected to
be greater than 50).
Quoted 7 lines
I think this needs to be somewhere, but possibly not in the results schema. For example, I don't want every listing of dbench results to have to report the units for each benchmark metric. These should be fairly static and we should be able to define them per-test, not per-run. Things like thresholds are a bit different, and we may need to record them per-run, since the success/failure threshold could be different depending on the board, or even changed by the user to fine-tune the regression-checking.
The reasons I wanted to add units to the schema were:
- AGL was using them on their HTML output reports, and it does make reports more readable. If we don't have this information in the results.json file, flot will need to get it from somewhere else (e.g. a json file). The problem is that we will need to update that file everytime we add a new test. I'd rather have that information inside the test directory. What do you think?
- Kernel CI format [2][3] allows including units in their measures (not strictly required) as well.
[2] https://api.kernelci.org/schema-test-case.html
[3] https://api.kernelci.org/json-schema/1.0/measurement.json
Quoted 26 lines
Quoted 8 lines
- Handle failed runs better. Sometimes the test fails very early, before even building it.
- I am not sure what to do with the "reference.log" files
+ Currently they are used to store thresholds, but these are probably board
dependent.
+ This is probably related to the discussion with Rafael about parameterized
builds. We should
be able to define the threshold for a specific test_case's measure.reference logs should be savable by the user, to compare with future runs. The system we have now uses parsed testlogs, which are generated using
log_compareand very simple line-based regular-expresison filters (using 'grep'). It will be much more flexible and robust to compare run.json files instead of a parsed log and a reference log.The purpose of these is to save the data from a "known good" run, so that regressions can be detected when the data from a current run differs from that. This can include sub-test failures, that we have decided to ignore or postpone resolution of.
I think once we have in place a system to save all the sub-test metric data from the testlogs (using a parser) in json format, then we can eliminate these. We should be able to replace reference.log with reference.json (which is just a saved run.json file). This is a key thing that I would like Fuego to be able to share easily between developers (and sites).
Quoted 8 lines
I have already started working on a json difference program (called 'jdiff') to compare 2 json files and report the differences between the two.
On the issue of where to save them, currently they should be saved somehow at the 'board' level. That is, tests will definitely have different results per-board. But there may be such a thing as a reference file that is dependent on the kernel version, or the distribution, or some other parameter. We should discuss the naming and storage of these.
How about saving them at the board's testplan, and also allow users to try different ones through the ftc run-test interface?
Quoted 6 lines
Quoted 3 lines
- Remove testplans?
+ I was thinking that we can substitute testplans by custom scripts that call
ftc add-jobsWhy do we want to remove them? I think they serve a useful function - expressing a set of tests (with their specs) to execute in sequence.
OK. I just wanted to mention that is redundant. But I agree that they are useful.
Quoted 10 lines
Quoted 2 lines
- Create a staging folder for tests that do not work or files that are not used.
+ Or maybe at least list them up on the wiki.Currently, if they are not listed in a testplan, the tests are functionally 'dead'. (Although a user can create a job for a single test and try it out). Maybe it would be good to have a 'staging' folder for tests that are under development or conversion (like a lot of the AGL tests). I agree that we should have some notion of the "approved and likely to work" tests, and that should be expressed somehow in the test placement or in documentation.
Maybe we can write this information on the test.yaml file (you mentioned something about evaluating tests with stars). Actually, I would like to know your plans with the test.yaml files in general. I haven't looked into them deeply yet.
Thanks,
Daniel
Quoted 17 lines
-----Original Message-----
From: Daniel Sangorrin on Tuesday, June 27, 2017 6:58 PMQuoted 4 lines
-----Original Message-----
From: Bird, Timothy [mailto:[email protected]]
Sent: Wednesday, June 28, 2017 9:47 AMQuoted 1 line
- Merge run.json files for a
test_suiteinto a single results.json file thatflot
Quoted 6 lines
Quoted 1 line
can visualize.
So, just to clarify, the run.json file has the results for a multiple metrics for a single execution (or 'run') of a test, and the results.json file has results for multiple runs of the test? That is, results.json has results for a single Fuego host with results from different runs and on different
boards?
Quoted 7 lines
Yes, that's correct. It's kind of a text "database". The size of this file will increase
with the number of runs so it may not scale. Probably we will need to implement
a logrotate-like functionality once we can submit all runs to a proper database
in a centralized server (maybe one running kernelci or something similar).
For a central server, I think we'll definitely have to use a database. If we end up using the same (or a 'similar-enough') schema, we may be able to reuse parts of the kernelci setup for our own Fuego central server. That is, copy their database configuration and maybe reuse their server-side web front-end. Maybe we could end up sharing code and helping each other out. It would be nice not to reinvent that wheel.
However, I don't want to require a database setup for the average developer (with one host and one or a few boards).
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
Pending tasks:
- Add report functionality
+ I have removed the generation of plot.png files. In part because Iwant to
Quoted 2 lines
Quoted 2 lines
do that directly from the
flot plugin, and in part because I think it is more useful if we integrateit in
Quoted 7 lines
Quoted 4 lines
the future ftc report command,
- Add more information to the run.json files.
+ I am trying to produce a schema that is very close to the one used in
Kernel.CI. Probably I can make it compatible.That sounds good. I'm very interested in the schema. I believe that Milo Casagrande mentioned something about groups, that I don't
think
Quoted 8 lines
Quoted 3 lines
we have yet. Everything else in your analysis from April
(https://lists.linuxfoundation.org/pipermail/fuego/2017-April/000448.html)
I think shows some analog between Fuego and KernelCI fields.Groups are called "
test_sets" in kernel CI and can contain an array of "test_cases".
# I'm thinking about using kernel ci's nomenclature for this. What do you think?
Did they ever give us an example of what their LTP results look like? I think Milo described them, but I don't recall actually seeing a json file. LTP is probably the most complicated test we'll need to handle.
I'm not sure I like their nomenclature. They have 3 things:test_suite, test_set and test_case.
I guess these are roughly the same as our:test_plan, test, and (unnamed by us) individual sub-test case. (but it's unclear these are exact analogs).
I find these three levels confusing - particularly because a test_suite in kernelCI can point to both test_sets and test_cases.
Quoted 3 lines
# I also want to rename platform to toolchain, fwver to
kernel_version, spec to
#test_spec, testplan totest_plan...
Agree on rename of platform to toolchain, fwver to kernel_version.
Our 'spec' is essentially the same as their test_set 'parameters' object. Note that their 'test_case' can have a 'parameters' object as well.
I'm thinking of test_case as something like: LTP.syscall.kill10 where, given that they support multiple measurements per test_case, maybe they would classify this as:
- test_suite LTP
- test_set syscall
- test_case: kill10
- measurement: ? (does kill10 do more than one measurement?)
Quoted 9 lines
In Fuego, we have a similar concept. If you see bonnie's reference.log [1] you will
notice that there are several groups/test_sets and multiple tests/test_cases inside
each group:
test_set:Sequential_Outputtest_cases: Block, PerChr, Rewrite
In this case, is Block a set of measurements, or a single measurement?
Quoted 6 lines
I am making an schema that is compatible with Kernel CI but that it will also allow
havingtest_setsinsidetest_sets. For example: "LTP > RT tests > func >sched_jitter"
contains 2-levels oftest_sets(RT tests and func).
2 levels: RT tests and func?
or just 2 test_sets?
I'm not that familiar with LTP, so is 'func' actually nested under 'RT tests'?
Quoted 35 lines
[1] https://bitbucket.org/tbird20d/fuego-
core/src/805adb067afc492382ee23bc9178c059b90c043e/engine/tests/Bench mark.bonnie/reference.log?at=next&fileviewer=file-view-default[Note] In the past, tests.info used to store this
test_set>test_caseinformation. Now this information
is actually provided by the test'sparser.pyand reference.log. The parser.py's information includes
information for the test_sets/test_cases that were actually executed (depends on the spec), whereas reference.log
contains all possible test_sets/test_cases for that test (including those that were skipped somehow because
of the selectedtest_spec). We need to decouple the reference thresholds from this information though.Quoted 3 lines
Quoted 3 lines
+ There is some information that needs to be added to Fuego.
Unfortunately, I will probably have to fix a lot of files:
1) Eachtest_case(remembertest_suite>test_set>test_case)should
Quoted 6 lines
Quoted 6 lines
be able to store a list of
measurements. Each measurement would consist of a name, value,
units, duration and maybe more
such as error messages specific to thetest_caseor expected values
(e.g. expected to fail, or expected to
be greater than 50).Quoted 7 lines
I think this needs to be somewhere, but possibly not in the results schema. For example, I don't want every listing of dbench results to have to report the units for each benchmark metric. These should be fairly static and we should be able to define them per-test, not per-run. Things like thresholds are a bit different, and we may need to record them per-run, since the success/failure threshold could be different depending on the board, or even changed by the user to fine-tune the regression-
checking.
Quoted 10 lines
The reasons I wanted to add units to the schema were:
- AGL was using them on their HTML output reports, and it does make
reports more readable.
If we don't have this information in the results.json file, flot will need to get
it
from somewhere else (e.g. a json file). The problem is that we will need to
update
that file everytime we add a new test. I'd rather have that information
inside the test directory.
What do you think?
OK - that makes sense.
Quoted 48 lines
- Kernel CI format [2][3] allows including units in their measures (not strictly
required) as well.[2] https://api.kernelci.org/schema-test-case.html
[3] https://api.kernelci.org/json-schema/1.0/measurement.jsonQuoted 1 line
Quoted 1 line
- Handle failed runs better. Sometimes the test fails very early, before
even
Quoted 3 lines
Quoted 3 lines
building it.
- I am not sure what to do with the "reference.log" files
+ Currently they are used to store thresholds, but these are probablyboard
Quoted 2 lines
Quoted 2 lines
dependent.
+ This is probably related to the discussion with Rafael aboutparameterized
Quoted 6 lines
Quoted 2 lines
builds. We should
be able to define the threshold for a specific test_case's measure.reference logs should be savable by the user, to compare with future runs. The system we have now uses parsed testlogs, which are generated using
log_compareand very simple line-based regular-expresison filters (using'grep').
Quoted 5 lines
It will be much more flexible and robust to compare run.json files instead of a parsed log and a reference log.
The purpose of these is to save the data from a "known good" run, so that regressions can be detected when the data from a current run differs from
that.
Quoted 1 line
This can include sub-test failures, that we have decided to ignore or
postpone
Quoted 4 lines
resolution of.
I think once we have in place a system to save all the sub-test metric data from the testlogs (using a parser) in json format, then we can eliminate
these.
Quoted 1 line
We should be able to replace reference.log with reference.json (which is
just
Quoted 1 line
a saved run.json file). This is a key thing that I would like Fuego to be able
to
Quoted 6 lines
share easily between developers (and sites).
I have already started working on a json difference program (called 'jdiff') to compare 2 json files and report the differences between the two.
On the issue of where to save them, currently they should be saved
somehow
Quoted 1 line
at the 'board' level. That is, tests will definitely have different results per-
board.
Quoted 8 lines
Quoted 1 line
But there may be such a thing as a reference file that is dependent on the
kernel
Quoted 1 line
version, or the distribution, or some other parameter. We should discuss
the
Quoted 1 line
naming and storage of these.
How about saving them at the board's testplan, and also allow users to try different ones through the ftc run-test interface?
I don't follow this. In /fuego-ro/boards?
They are definitely test-specific, but I'm not sure I want them in /fuego-core/engine/tests/<testname>. That will pollute the fuego-core directory with site-specific data.
I think they need to go into /fuego-rw somewhere, as they can be user-generated (and possibly user-downloaded).
Quoted 26 lines
Quoted 3 lines
Quoted 3 lines
- Remove testplans?
+ I was thinking that we can substitute testplans by custom scripts thatcall
Quoted 3 lines
Quoted 1 line
ftc add-jobs
Why do we want to remove them? I think they serve a useful function -
expressing
Quoted 1 line
a set of tests (with their specs) to execute in sequence.
OK. I just wanted to mention that is redundant. But I agree that they are useful.
Quoted 1 line
Quoted 1 line
- Create a staging folder for tests that do not work or files that are not
used.
Quoted 3 lines
Quoted 1 line
+ Or maybe at least list them up on the wiki.
Currently, if they are not listed in a testplan, the tests are functionally
'dead'.
Quoted 6 lines
(Although a user can create a job for a single test and try it out). Maybe it would be good to have a 'staging' folder for tests that are under development or conversion (like a lot of the AGL tests). I agree that we should have some notion of the "approved and likely to work" tests, and that should be expressed somehow in the test placement or in documentation.
Quoted 3 lines
Maybe we can write this information on the
test.yamlfile (you mentioned something
about evaluating tests with stars).
I hadn't considered putting the rating information into the test.yaml file, but an indicator of test 'readiness' might work there. I have test version, which, if the number is less than 1.0, is a proxy for indicating that the test is not really considered valid yet (ie, it's pre-release quality). Note that the version field in test.yaml is the version of the fuego test, not the version of the test program used in the test.
Quoted 3 lines
Actually, I would like to know your plans
with thetest.yamlfiles in general. I haven't looked into them deeply yet.
They were created as the place to hold data used for packaging a test (for test distribution outside of the fuego-core repository). Currently, they have a manifest of files and some extra information related to packaging a test (author, license, version, etc.). I will formally define these, clean them up, and add them for all the tests in the repository when we roll out "test packages" as an officially supported feature.
Right now, test packages are implemented as just a proof of concept. More work needs to be done server side (with a rating system, security to prevent malware, etc) before this feature is ready. (ie - not this release). You can think of them like rpm .spec files, or debian control files. An example of one is in fuego-core/engine/tests/Functional.bc/test.yaml (Now that I look at it, it's out of date. It lists the base_script name, and that is no longer needed, as the base_script is now always 'fuego_test.sh').
-- Tim
Quoted 13 lines
-----Original Message-----
From: Bird, Timothy on Tuesday, June 27, 2017 9:56 PMQuoted 6 lines
-----Original Message-----
From: Daniel Sangorrin on Tuesday, June 27, 2017 6:58 PMQuoted 3 lines
-----Original Message-----
From: Bird, Timothy [mailto:[email protected]]
Sent: Wednesday, June 28, 2017 9:47 AMIn Fuego, we have a similar concept. If you see bonnie's reference.log [1]
you
Quoted 2 lines
will
notice that there are several groups/test_sets and multiple
Quoted 8 lines
tests/test_cases
Quoted 5 lines
inside
each group:
test_set:Sequential_Outputtest_cases: Block, PerChr, RewriteIn this case, is Block a set of measurements, or a single measurement?
OK - it's a single measurement.
Ugh. I was confused. reference.log is not the file I was thinking of. reference.log is the file that holds thresholds for all of the benchmark test measurements. I was thinking of the 'pn log'.
All the stuff I said about placement of reference.log (well most of it), I meant in regard to pn log files.
OK, getting back to reference.log...
First, it's terribly named. It's not a 'log' by any stretch of the imagination. It's also terribly formatted. It's some weird custom format created just to be easy to read and write with 'awk'. It does need to specify all possible measurement names (that are relevant for determining benchmark pass or fail status) This is usually a small subset of all the measurements made by a benchmark. And, I'm not sure it belongs in the test directory, as it should be writable by the user to reflect their own notion of values that indicate a regression threshold for a measurement.
However, having said that, I'm not sure I want to change it this release.
Please note one other thing, though, related to test schema and test_cases and nomenclature...
I strongly believe that every test case/measurement in the system should have something that I call a TGUID - a test globally unique identifier (something like an URL in World-wide-web parlance). This is a single string that can be used to identify a particular measurement from all other measurements. This should be able to be constructed using a path-like syntax. Personally, I would use dots as delimiters, like bonnie's reference log does (rather than slashes, like URLs or filepaths use).
It is important, IMO, to have a static test identifier, for referring to the test independent of host, board, test_plan or spec:
e.g. "LTP.syscall.kill10"
We should be able to construct these fully-qualified test identifiers using values in the json schemas for a run.
The reason for having TGUIDs is that I would like for people to be able to share meta-data about individual measurements, in a way that is independent of other test infrastructure or layout.
-- Tim
Quoted 28 lines
-----Original Message-----
From: Bird, Timothy [mailto:[email protected]]
Sent: Wednesday, June 28, 2017 2:23 PM
To: Bird, Timothy; Daniel Sangorrin; [email protected]
Subject: RE: [Fuego] status updateQuoted 18 lines
-----Original Message-----
From: Bird, Timothy on Tuesday, June 27, 2017 9:56 PMQuoted 6 lines
-----Original Message-----
From: Daniel Sangorrin on Tuesday, June 27, 2017 6:58 PMQuoted 3 lines
-----Original Message-----
From: Bird, Timothy [mailto:[email protected]]
Sent: Wednesday, June 28, 2017 9:47 AMIn Fuego, we have a similar concept. If you see bonnie's reference.log [1]
you
Quoted 2 lines
will
notice that there are several groups/test_sets and multipletests/test_cases
Quoted 5 lines
inside
each group:
test_set:Sequential_Outputtest_cases: Block, PerChr, Rewrite
Quoted 7 lines
Quoted 1 line
In this case, is Block a set of measurements, or a single measurement?
OK - it's a single measurement.
Ugh. I was confused. reference.log is not the file I was thinking of. reference.log is the file that holds thresholds for all of the benchmark test measurements. I was thinking of the 'pn log'.
Ouch, I had already forgotten about this guys.
Quoted 13 lines
All the stuff I said about placement of reference.log (well most of it), I meant in regard to pn log files.
OK, getting back to reference.log...
First, it's terribly named. It's not a 'log' by any stretch of the imagination. It's also terribly formatted. It's some weird custom format created just to be easy to read and write with 'awk'. It does need to specify all possible measurement names (that are relevant for determining benchmark pass or fail status) This is usually a small subset of all the measurements made by a benchmark. And, I'm not sure it belongs in the test directory, as it should be writable by the user to reflect their own notion of values that indicate a regression threshold for a measurement.
Quoted 1 line
However, having said that, I'm not sure I want to change it this release.
OK, that's what I wanted to know. For now, I will stick to what we have.
Quoted 15 lines
Please note one other thing, though, related to test schema and
test_casesand nomenclature...I strongly believe that every test case/measurement in the system should have something that I call a TGUID - a test globally unique identifier (something like an URL in World-wide-web parlance). This is a single string that can be used to identify a particular measurement from all other measurements. This should be able to be constructed using a path-like syntax. Personally, I would use dots as delimiters, like bonnie's reference log does (rather than slashes, like URLs or filepaths use).
It is important, IMO, to have a static test identifier, for referring to the test independent of host, board,
test_planor spec:
e.g. "LTP.syscall.kill10"
Quoted 6 lines
We should be able to construct these fully-qualified test identifiers using values in the json schemas for a run.
The reason for having TGUIDs is that I would like for people to be able to share meta-data about individual measurements, in a way that is independent of other test infrastructure or layout.
In kernel CI they ask for this ID to the centralized server, before submitting the json files. We can do as them, or create our own identifiers by combining several words.
Thanks,
Daniel
Hi Tim,
# I've added Milo to the Cc.
Quoted 3 lines
-----Original Message-----
From: Bird, Timothy [mailto:[email protected]]
Sent: Wednesday, June 28, 2017 1:56 PM
...
Quoted 19 lines
Quoted 19 lines
Quoted 3 lines
Quoted 3 lines
Pending tasks:
- Add report functionality
+ I have removed the generation of plot.png files. In part because Iwant to
Quoted 2 lines
Quoted 2 lines
do that directly from the
flot plugin, and in part because I think it is more useful if we integrateit in
Quoted 7 lines
Quoted 4 lines
the future ftc report command,
- Add more information to the run.json files.
+ I am trying to produce a schema that is very close to the one used in
Kernel.CI. Probably I can make it compatible.That sounds good. I'm very interested in the schema. I believe that Milo Casagrande mentioned something about groups, that I don't
think
Quoted 3 lines
we have yet. Everything else in your analysis from April
(https://lists.linuxfoundation.org/pipermail/fuego/2017-April/000448.html)
I think shows some analog between Fuego and KernelCI fields.
Quoted 7 lines
Quoted 4 lines
Groups are called "
test_sets" in kernel CI and can contain an array of "test_cases".
# I'm thinking about using kernel ci's nomenclature for this. What do you think?Did they ever give us an example of what their LTP results look like? I think Milo described them, but I don't recall actually seeing a json file. LTP is probably the most complicated test we'll need to handle.
Milo sent 3 hackbench json files but not an LTP one. LTP normally can work with 3 levels (e.g.: LTP > syscalls > kill01) so there is no problem about that. However, LTP now also includes 2 more test suites inside (Posix
open testsuite, and the real-time testsuite) with their own test sets and test cases. For that reason, you could end up with 4 nested levels (unless you create 3 test suites from the same source code).
In any case, I don't see the need to restrict ourselves to only 3 nesting levels.
Quoted 5 lines
I'm not sure I like their nomenclature. They have 3 things:
test_suite,test_setandtest_case.
I guess these are roughly the same as our:test_plan, test, and (unnamed by us) individual sub-test case. (but it's unclear these are exact analogs).
I think this is a better analogy:
test_suite, test_set , test_case == test_name, groupname, test == bonnie, sequential_output, rewrite
The concept of test_plan is not in kernel ci afaik (maybe it is in LAVA with a different name such batch jobs?).
Quoted 2 lines
I find these three levels confusing - particularly because a
test_suitein kernelCI can point to bothtest_setsandtest_cases.
Actually that makes sense. For example, suppose you have a simple test suite (hello world ) with one single test case. Then you don't really need to define a test_set.
Quoted 7 lines
Quoted 3 lines
# I also want to rename platform to toolchain, fwver to
kernel_version, spec to
#test_spec, testplan totest_plan...Agree on rename of platform to toolchain, fwver to
kernel_version.Our 'spec' is essentially the same as their
test_set'parameters' object. Note that their 'test_case' can have a 'parameters' object as well.
It's something like that. But I think we should write the name of the test_spec at the test_suite level in the schema because we do not support per-test_set parameters at the moment.
Quoted 7 lines
I'm thinking of
test_caseas something like: LTP.syscall.kill10 where, given that they support multiple measurements pertest_case, maybe they would classify this as:
-test_suiteLTP
-test_setsyscall
-test_case: kill10
- measurement: ? (does kill10 do more than one measurement?)
Functional test cases, such as LTP test cases, normally just finish with a return value (TPASS, TFAIL, TBROK etc..) so you don't really need measurements for them (unless you want to store each checkpoint/assertion inside the test case).
Benchmark test cases, on the other hand, may have more than one measurement. For example, netperf returns the network throughput but also the CPU utilization. Fuego's Benchmark.netperf currently is doing this:
test_suite: netperf
test_set: MIGRATED_TCP_MAERTS
test_cases: cpu, net
test_set: MIGRATED_TCP_STREAM
test_cases: cpu, net
I think that what we want to achieve is actually:
test_suite: netperf
test_set: TCP
test_case: MIGRATED_TCP_MAERTS
measurements: [cpu, net]
test_case: MIGRATED_TCP_STREAM
measurements: [cpu, net]
test_set: UDP
test_case: whatever..
measurements: [cpu, net]
Quoted 10 lines
Quoted 8 lines
In Fuego, we have a similar concept. If you see bonnie's reference.log [1] you will
notice that there are several groups/test_sets and multiple tests/test_cases inside
each group:
test_set:Sequential_Outputtest_cases: Block, PerChr, RewriteIn this case, is Block a set of measurements, or a single measurement?
A single one, but it could be multiple if we for example did it for different sector sizes.
Quoted 9 lines
Quoted 5 lines
I am making an schema that is compatible with Kernel CI but that it will also allow
havingtest_setsinsidetest_sets. For example: "LTP > RT tests > func >sched_jitter"
contains 2-levels oftest_sets(RT tests and func).2 levels: RT tests and func?
or just 2test_sets?I'm not that familiar with LTP, so is 'func' actually nested under 'RT tests'?
Yes. RT tests have "func", "stress" and "perf" test sets.
Ref: https://github.com/linux-test-project/ltp/tree/master/testcases/realtime
Quoted 96 lines
Quoted 35 lines
[1] https://bitbucket.org/tbird20d/fuego-
core/src/805adb067afc492382ee23bc9178c059b90c043e/engine/tests/Bench mark.bonnie/reference.log?at=next&fileviewer=file-view-default[Note] In the past, tests.info used to store this
test_set>test_caseinformation. Now this information
is actually provided by the test'sparser.pyand reference.log. The parser.py's information includes
information for the test_sets/test_cases that were actually executed (depends on the spec), whereas reference.log
contains all possible test_sets/test_cases for that test (including those that were skipped somehow because
of the selectedtest_spec). We need to decouple the reference thresholds from this information though.Quoted 3 lines
Quoted 3 lines
+ There is some information that needs to be added to Fuego.
Unfortunately, I will probably have to fix a lot of files:
1) Eachtest_case(remembertest_suite>test_set>test_case)should
Quoted 6 lines
Quoted 6 lines
be able to store a list of
measurements. Each measurement would consist of a name, value,
units, duration and maybe more
such as error messages specific to thetest_caseor expected values
(e.g. expected to fail, or expected to
be greater than 50).Quoted 7 lines
I think this needs to be somewhere, but possibly not in the results schema. For example, I don't want every listing of dbench results to have to report the units for each benchmark metric. These should be fairly static and we should be able to define them per-test, not per-run. Things like thresholds are a bit different, and we may need to record them per-run, since the success/failure threshold could be different depending on the board, or even changed by the user to fine-tune the regression-
checking.
Quoted 10 lines
The reasons I wanted to add units to the schema were:
- AGL was using them on their HTML output reports, and it does make
reports more readable.
If we don't have this information in the results.json file, flot will need to get
it
from somewhere else (e.g. a json file). The problem is that we will need to
update
that file everytime we add a new test. I'd rather have that information
inside the test directory.
What do you think?OK - that makes sense.
Quoted 49 lines
- Kernel CI format [2][3] allows including units in their measures (not strictly
required) as well.[2] https://api.kernelci.org/schema-test-case.html
[3] https://api.kernelci.org/json-schema/1.0/measurement.jsonQuoted 1 line
Quoted 1 line
- Handle failed runs better. Sometimes the test fails very early, before
even
Quoted 3 lines
Quoted 3 lines
building it.
- I am not sure what to do with the "reference.log" files
+ Currently they are used to store thresholds, but these are probablyboard
Quoted 2 lines
Quoted 2 lines
dependent.
+ This is probably related to the discussion with Rafael aboutparameterized
Quoted 6 lines
Quoted 2 lines
builds. We should
be able to define the threshold for a specific test_case's measure.reference logs should be savable by the user, to compare with future runs. The system we have now uses parsed testlogs, which are generated using
log_compareand very simple line-based regular-expresison filters (using'grep').
Quoted 5 lines
It will be much more flexible and robust to compare run.json files instead of a parsed log and a reference log.
The purpose of these is to save the data from a "known good" run, so that regressions can be detected when the data from a current run differs from
that.
Quoted 1 line
This can include sub-test failures, that we have decided to ignore or
postpone
Quoted 4 lines
resolution of.
I think once we have in place a system to save all the sub-test metric data from the testlogs (using a parser) in json format, then we can eliminate
these.
Quoted 1 line
We should be able to replace reference.log with reference.json (which is
just
Quoted 1 line
a saved run.json file). This is a key thing that I would like Fuego to be able
to
Quoted 6 lines
share easily between developers (and sites).
I have already started working on a json difference program (called 'jdiff') to compare 2 json files and report the differences between the two.
On the issue of where to save them, currently they should be saved
somehow
Quoted 1 line
at the 'board' level. That is, tests will definitely have different results per-
board.
Quoted 1 line
But there may be such a thing as a reference file that is dependent on the
Quoted 8 lines
Quoted 7 lines
kernel
Quoted 1 line
version, or the distribution, or some other parameter. We should discuss
the
Quoted 1 line
naming and storage of these.
How about saving them at the board's testplan, and also allow users to try different ones through the ftc run-test interface?
I don't follow this. In /fuego-ro/boards?
Sorry, I forgot to say that testplans should also be per-board/user-generated and therefore not in fuego-core.
Quoted 5 lines
They are definitely test-specific, but I'm not sure I want them in /fuego-core/engine/tests/<testname>. That will pollute the fuego-core directory with site-specific data.
I think they need to go into /fuego-rw somewhere, as they can be user-generated (and possibly user-downloaded).
I think that we should add a configuration file for the user to specify a path containing its boards/testplans etc. # I think Rafael mentioned something about this.
Thanks,
Daniel
Hello,
Quoted 3 lines · Daniel Sangorrin
Milo sent 3 hackbench json files but not an LTP one. LTP normally can work with 3 levels (e.g.: LTP > syscalls > kill01) so there is no problem about that.
we don't have any actual LTP runs in our dataset, just some kselftest and hackbench.
Quoted 4 lines
However, LTP now also includes 2 more test suites inside (Posix
open testsuite, and the real-time testsuite) with their own test sets and test cases. For that reason, you could end up with 4 nested levels (unless you create 3 test suites from the same source code).
In this case the kernelci test schemas might pose a problem, I don't think we ever considered this scenario.
Quoted 4 lines
I think this is a better analogy:
test_suite, test_set , test_case == test_name, groupname, test == bonnie, sequential_output, rewriteThe concept of
test_planis not in kernel ci afaik (maybe it is in LAVA with a different name such batch jobs?).
No, kernelci doesn't hold any "test plan", we just keep the results (but it depends also on the definition you are giving to "test plan":
for us is "a document describing the test workflow").
I'm not even sure if those are stored in LAVA though.
Signature · 2 lines
Milo Casagrande Linaro.org <www.linaro.org> │ Open source software for ARM SoCs