tests artifacts retriever
Quoted 57 lines · Alexis Lothoré via lists.openembedded.org
Hello everyone,
I have been briefed about the need to add an infrastructure to be able to retrieve some test files (test output, logs, etc) in Yocto/CI automation to ease debugging of some hard-to-reproduce issues. Before starting to implement a solution, I would like to get some feedback about what I have understood from the original issue and what I propose to implement to improve this. Please find below my reflexions, feel free to comment, correct, enrich and/or approve.# Motivation
Some test failures appearing in automatic testing can be hard to fix, based on too few informations from failing tests (see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes packages/binaries under test are able to generate more detailed output in a dedicated file, but there is actually no standardized way of retrieving such output files. There has been some custom efforts done to solve this issue for specific recipes, for example for lttng-tools tests :
https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408The goal of this RFC is to propose an architecture for a generic system which allows to retrieve an arbitrary list of files from a test run.
# Use case example
lttng-tools recipe provides LTTNG tools tests as ptests (see meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log files (error.log, test-suite.log), it would be convenient to retrieve those files.
# Proposal
- Introduce an "artifacts.conf" file
- this file contains a "files of interest" list to be retrieved onto host after running ptests (exact format to be defined)
- artifacts.conf is specific to a ptest package and is stored next to run-ptest script in package
- artifacts.conf contains, for each ptest package, the following informations:
- name of file to retrieve
- artifacts.conf will be installed in target image (as for package-specific run-ptest script)
- each ptest package MAY have an artifacts.conf file
- When running ptests, for each available ptest, ptest-runner (on the target) will read artifacts.conf if it exists, and retrieve corresponding artifacts file in an artifacts directory, with one subdirectory per ptest.
- Artifacts will be aggregated only if at least one test has a status different from "PASS"
- Once tests are finished, tests manager on host (ptest.py) will fetch artifacts directory. It will put this directory alongside testresults file on host
- This mechanism will work for both real targets and emulated targets (Qemu), and will based on SSH to transfer artifacts back to host# Additional questions
Quoted 10 lines
- once retrieved from target, aside from default test results directory (build/tmp/log/oeqa/<...>), where should we store artifacts ? For automated testing, should we publish them alongside other test and regression reports (in web server directory) ?
- current proposal is focused on ptests: should I broaden the scope and include all kind of runtime tests (as found in meta/lib/oeqa/runtime/cases ?)
- are there examples of tests producing files to retrieve with non-static names (which will then require the artifacts.conf to be smarter) ?
Fixing Alexandre Belloni's address in CC, sorry for the noise
Signature · 3 lines
Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
I think having to write a custom artifacts.conf for each ptest is not a good direction, as it's both too much work, and requires good understanding what is important in the output and what isn't :) When a ptest fails, it would be enough to simply make a tarball of /usr/lib/component/ptest/ with everything in it. Maybe /tmp and /var/log as well.
Alex
Quoted 86 lines · Alexis Lothoré via lists.openembedded.org
Quoted 57 lines · Alexis Lothoré via lists.openembedded.org
Hello everyone,
I have been briefed about the need to add an infrastructure to be able to retrieve some test files (test output, logs, etc) in Yocto/CI automation to ease debugging of some hard-to-reproduce issues. Before starting to implement a solution, I would like to get some feedback about what I have understood from the original issue and what I propose to implement to improve this. Please find below my reflexions, feel free to comment, correct, enrich and/or approve.# Motivation
Some test failures appearing in automatic testing can be hard to fix, based on too few informations from failing tests (see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes packages/binaries under test are able to generate more detailed output in a dedicated file, but there is actually no standardized way of retrieving such output files. There has been some custom efforts done to solve this issue for specific recipes, for example for lttng-tools tests :
https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408The goal of this RFC is to propose an architecture for a generic system which allows to retrieve an arbitrary list of files from a test run.
# Use case example
lttng-tools recipe provides LTTNG tools tests as ptests (see meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log files (error.log, test-suite.log), it would be convenient to retrieve those files.
# Proposal
- Introduce an "artifacts.conf" file
- this file contains a "files of interest" list to be retrieved onto host after running ptests (exact format to be defined)
- artifacts.conf is specific to a ptest package and is stored next to run-ptest script in package
- artifacts.conf contains, for each ptest package, the following informations:
- name of file to retrieve
- artifacts.conf will be installed in target image (as for package-specific run-ptest script)
- each ptest package MAY have an artifacts.conf file
- When running ptests, for each available ptest, ptest-runner (on the target) will read artifacts.conf if it exists, and retrieve corresponding artifacts file in an artifacts directory, with one subdirectory per ptest.
- Artifacts will be aggregated only if at least one test has a status different from "PASS"
- Once tests are finished, tests manager on host (ptest.py) will fetch artifacts directory. It will put this directory alongside testresults file on host
- This mechanism will work for both real targets and emulated targets (Qemu), and will based on SSH to transfer artifacts back to host# Additional questions
Quoted 10 lines
- once retrieved from target, aside from default test results directory (build/tmp/log/oeqa/<...>), where should we store artifacts ? For automated testing, should we publish them alongside other test and regression reports (in web server directory) ?
- current proposal is focused on ptests: should I broaden the scope and include all kind of runtime tests (as found in meta/lib/oeqa/runtime/cases ?)
- are there examples of tests producing files to retrieve with non-static names (which will then require the artifacts.conf to be smarter) ?Fixing Alexandre Belloni's address in CC, sorry for the noise
--
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181597): https://lists.openembedded.org/g/openembedded-core/message/181597
Mute This Topic: https://lists.openembedded.org/mt/99066378/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Hello Alexander, thanks for the feedback
Quoted 6 lines · Alexander Kanavin
I think having to write a custom artifacts.conf for each ptest is not a good direction, as it's both too much work, and requires good understanding what is important in the output and what isn't :) When a ptest fails, it would be enough to simply make a tarball of
/usr/lib/component/ptest/with everything in it. Maybe /tmp and/var/logas well.
Indeed, it can be tedious to write an artifacts conf file for each ptest. If it is tolerable to get an archive of the whole ptest directory (+ possibly /var/log and /tmp), we can go for it and fine-tune later if needed. In this case, I could either keep artifacts.conf to list the directories we want for all ptests, or completely drop it.
I will let this RFC run for more time before starting to implement something, but if no one come back on this, I will apply your suggestion.
Kind regards,
Quoted 92 lines
Alex
Quoted 86 lines · Alexis Lothoré via lists.openembedded.org
Quoted 57 lines · Alexis Lothoré via lists.openembedded.org
Hello everyone,
I have been briefed about the need to add an infrastructure to be able to retrieve some test files (test output, logs, etc) in Yocto/CI automation to ease debugging of some hard-to-reproduce issues. Before starting to implement a solution, I would like to get some feedback about what I have understood from the original issue and what I propose to implement to improve this. Please find below my reflexions, feel free to comment, correct, enrich and/or approve.# Motivation
Some test failures appearing in automatic testing can be hard to fix, based on too few informations from failing tests (see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes packages/binaries under test are able to generate more detailed output in a dedicated file, but there is actually no standardized way of retrieving such output files. There has been some custom efforts done to solve this issue for specific recipes, for example for lttng-tools tests :
https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408The goal of this RFC is to propose an architecture for a generic system which allows to retrieve an arbitrary list of files from a test run.
# Use case example
lttng-tools recipe provides LTTNG tools tests as ptests (see meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log files (error.log, test-suite.log), it would be convenient to retrieve those files.
# Proposal
- Introduce an "artifacts.conf" file
- this file contains a "files of interest" list to be retrieved onto host after running ptests (exact format to be defined)
- artifacts.conf is specific to a ptest package and is stored next to run-ptest script in package
- artifacts.conf contains, for each ptest package, the following informations:
- name of file to retrieve
- artifacts.conf will be installed in target image (as for package-specific run-ptest script)
- each ptest package MAY have an artifacts.conf file
- When running ptests, for each available ptest, ptest-runner (on the target) will read artifacts.conf if it exists, and retrieve corresponding artifacts file in an artifacts directory, with one subdirectory per ptest.
- Artifacts will be aggregated only if at least one test has a status different from "PASS"
- Once tests are finished, tests manager on host (ptest.py) will fetch artifacts directory. It will put this directory alongside testresults file on host
- This mechanism will work for both real targets and emulated targets (Qemu), and will based on SSH to transfer artifacts back to host# Additional questions
Quoted 10 lines
- once retrieved from target, aside from default test results directory (build/tmp/log/oeqa/<...>), where should we store artifacts ? For automated testing, should we publish them alongside other test and regression reports (in web server directory) ?
- current proposal is focused on ptests: should I broaden the scope and include all kind of runtime tests (as found in meta/lib/oeqa/runtime/cases ?)
- are there examples of tests producing files to retrieve with non-static names (which will then require the artifacts.conf to be smarter) ?Fixing Alexandre Belloni's address in CC, sorry for the noise
--
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181597): https://lists.openembedded.org/g/openembedded-core/message/181597
Mute This Topic: https://lists.openembedded.org/mt/99066378/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 10 lines · Alexis Lothor�
Hello Alexander, thanks for the feedback
Quoted 6 lines · Alexander Kanavin
I think having to write a custom artifacts.conf for each ptest is not a good direction, as it's both too much work, and requires good understanding what is important in the output and what isn't :) When a ptest fails, it would be enough to simply make a tarball of
/usr/lib/component/ptest/with everything in it. Maybe /tmp and/var/logas well.
Quoted 9 lines
Indeed, it can be tedious to write an artifacts conf file for each ptest. If it is tolerable to get an archive of the whole ptest directory (+ possibly
/var/logand /tmp), we can go for it and fine-tune later if needed. In this case, I could either keep artifacts.conf to list the directories we want for all ptests, or completely drop it.I will let this RFC run for more time before starting to implement something, but if no one come back on this, I will apply your suggestion.
I think the requirement that I didn't convey properly is that there should be one global conf file for the build.
Quoted 100 lines
Kind regards,
Quoted 92 lines
Alex
Quoted 86 lines · Alexis Lothor� via lists.openembedded.org
Quoted 57 lines · Alexis Lothor� via lists.openembedded.org
Hello everyone,
I have been briefed about the need to add an infrastructure to be able to retrieve some test files (test output, logs, etc) in Yocto/CI automation to ease debugging of some hard-to-reproduce issues. Before starting to implement a solution, I would like to get some feedback about what I have understood from the original issue and what I propose to implement to improve this. Please find below my reflexions, feel free to comment, correct, enrich and/or approve.# Motivation
Some test failures appearing in automatic testing can be hard to fix, based on too few informations from failing tests (see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes packages/binaries under test are able to generate more detailed output in a dedicated file, but there is actually no standardized way of retrieving such output files. There has been some custom efforts done to solve this issue for specific recipes, for example for lttng-tools tests :
https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408The goal of this RFC is to propose an architecture for a generic system which allows to retrieve an arbitrary list of files from a test run.
# Use case example
lttng-tools recipe provides LTTNG tools tests as ptests (see meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log files (error.log, test-suite.log), it would be convenient to retrieve those files.
# Proposal
- Introduce an "artifacts.conf" file
- this file contains a "files of interest" list to be retrieved onto host after running ptests (exact format to be defined)
- artifacts.conf is specific to a ptest package and is stored next to run-ptest script in package
- artifacts.conf contains, for each ptest package, the following informations:
- name of file to retrieve
- artifacts.conf will be installed in target image (as for package-specific run-ptest script)
- each ptest package MAY have an artifacts.conf file
- When running ptests, for each available ptest, ptest-runner (on the target) will read artifacts.conf if it exists, and retrieve corresponding artifacts file in an artifacts directory, with one subdirectory per ptest.
- Artifacts will be aggregated only if at least one test has a status different from "PASS"
- Once tests are finished, tests manager on host (ptest.py) will fetch artifacts directory. It will put this directory alongside testresults file on host
- This mechanism will work for both real targets and emulated targets (Qemu), and will based on SSH to transfer artifacts back to host# Additional questions
Quoted 10 lines
- once retrieved from target, aside from default test results directory (build/tmp/log/oeqa/<...>), where should we store artifacts ? For automated testing, should we publish them alongside other test and regression reports (in web server directory) ?
- current proposal is focused on ptests: should I broaden the scope and include all kind of runtime tests (as found in meta/lib/oeqa/runtime/cases ?)
- are there examples of tests producing files to retrieve with non-static names (which will then require the artifacts.conf to be smarter) ?Fixing Alexandre Belloni's address in CC, sorry for the noise
--
Alexis Lothor�, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181597): https://lists.openembedded.org/g/openembedded-core/message/181597
Mute This Topic: https://lists.openembedded.org/mt/99066378/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-Signature · 3 lines
Alexis Lothor�, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com