Skip to contentSkip to conversation

tests artifacts retriever

RFC Alexis Lothoré 3 participants 4 messages May 22, 2023 – May 23, 2023 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 57 lines · Alexis Lothoré via lists.openembedded.org

Hello everyone,
I have been briefed about the need to add an infrastructure to be able to retrieve some test files (test output, logs, etc) in Yocto/CI automation to ease debugging of some hard-to-reproduce issues. Before starting to implement a solution, I would like to get some feedback about what I have understood from the original issue and what I propose to implement to improve this. Please find below my reflexions, feel free to comment, correct, enrich and/or approve.

# Motivation

Some test failures appearing in automatic testing can be hard to fix, based on too few informations from failing tests (see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes packages/binaries under test are able to generate more detailed output in a dedicated file, but there is actually no standardized way of retrieving such output files. There has been some custom efforts done to solve this issue for specific recipes, for example for lttng-tools tests :
https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408

The goal of this RFC is to propose an architecture for a generic system which allows to retrieve an arbitrary list of files from a test run.

# Use case example

lttng-tools recipe provides LTTNG tools tests as ptests (see meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log files (error.log, test-suite.log), it would be convenient to retrieve those files.

# Proposal

- Introduce an "artifacts.conf" file
- this file contains a "files of interest" list to be retrieved onto host after running ptests (exact format to be defined)
- artifacts.conf is specific to a ptest package and is stored next to run-ptest script in package
- artifacts.conf contains, for each ptest package, the following informations:
- name of file to retrieve
- artifacts.conf will be installed in target image (as for package-specific run-ptest script)
- each ptest package MAY have an artifacts.conf file
- When running ptests, for each available ptest, ptest-runner (on the target) will read artifacts.conf if it exists, and retrieve corresponding artifacts file in an artifacts directory, with one subdirectory per ptest.
- Artifacts will be aggregated only if at least one test has a status different from "PASS"
- Once tests are finished, tests manager on host (ptest.py) will fetch artifacts directory. It will put this directory alongside testresults file on host
- This mechanism will work for both real targets and emulated targets (Qemu), and will based on SSH to transfer artifacts back to host

# Additional questions

Quoted 10 lines

- once retrieved from target, aside from default test results directory (build/tmp/log/oeqa/<...>), where should we store artifacts ? For automated testing, should we publish them alongside other test and regression reports (in web server directory) ?
- current proposal is focused on ptests: should I broaden the scope and include all kind of runtime tests (as found in meta/lib/oeqa/runtime/cases ?)
- are there examples of tests producing files to retrieve with non-static names (which will then require the artifacts.conf to be smarter) ?

Fixing Alexandre Belloni's address in CC, sorry for the noise

Signature · 3 lines
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

I think having to write a custom artifacts.conf for each ptest is not a good direction, as it's both too much work, and requires good understanding what is important in the output and what isn't :) When a ptest fails, it would be enough to simply make a tarball of /usr/lib/component/ptest/ with everything in it. Maybe /tmp and /var/log as well.

Alex

Quoted 86 lines · Alexis Lothoré via lists.openembedded.org
Quoted 57 lines · Alexis Lothoré via lists.openembedded.org

Hello everyone,
I have been briefed about the need to add an infrastructure to be able to retrieve some test files (test output, logs, etc) in Yocto/CI automation to ease debugging of some hard-to-reproduce issues. Before starting to implement a solution, I would like to get some feedback about what I have understood from the original issue and what I propose to implement to improve this. Please find below my reflexions, feel free to comment, correct, enrich and/or approve.

# Motivation

Some test failures appearing in automatic testing can be hard to fix, based on too few informations from failing tests (see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes packages/binaries under test are able to generate more detailed output in a dedicated file, but there is actually no standardized way of retrieving such output files. There has been some custom efforts done to solve this issue for specific recipes, for example for lttng-tools tests :
https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408

The goal of this RFC is to propose an architecture for a generic system which allows to retrieve an arbitrary list of files from a test run.

# Use case example

lttng-tools recipe provides LTTNG tools tests as ptests (see meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log files (error.log, test-suite.log), it would be convenient to retrieve those files.

# Proposal

- Introduce an "artifacts.conf" file
- this file contains a "files of interest" list to be retrieved onto host after running ptests (exact format to be defined)
- artifacts.conf is specific to a ptest package and is stored next to run-ptest script in package
- artifacts.conf contains, for each ptest package, the following informations:
- name of file to retrieve
- artifacts.conf will be installed in target image (as for package-specific run-ptest script)
- each ptest package MAY have an artifacts.conf file
- When running ptests, for each available ptest, ptest-runner (on the target) will read artifacts.conf if it exists, and retrieve corresponding artifacts file in an artifacts directory, with one subdirectory per ptest.
- Artifacts will be aggregated only if at least one test has a status different from "PASS"
- Once tests are finished, tests manager on host (ptest.py) will fetch artifacts directory. It will put this directory alongside testresults file on host
- This mechanism will work for both real targets and emulated targets (Qemu), and will based on SSH to transfer artifacts back to host

# Additional questions

Quoted 10 lines

- once retrieved from target, aside from default test results directory (build/tmp/log/oeqa/<...>), where should we store artifacts ? For automated testing, should we publish them alongside other test and regression reports (in web server directory) ?
- current proposal is focused on ptests: should I broaden the scope and include all kind of runtime tests (as found in meta/lib/oeqa/runtime/cases ?)
- are there examples of tests producing files to retrieve with non-static names (which will then require the artifacts.conf to be smarter) ?

Fixing Alexandre Belloni's address in CC, sorry for the noise

--
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181597): https://lists.openembedded.org/g/openembedded-core/message/181597
Mute This Topic: https://lists.openembedded.org/mt/99066378/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw

Hello Alexander, thanks for the feedback

Quoted 6 lines · Alexander Kanavin

I think having to write a custom artifacts.conf for each ptest is not a good direction, as it's both too much work, and requires good understanding what is important in the output and what isn't :) When a ptest fails, it would be enough to simply make a tarball of /usr/lib/component/ptest/ with everything in it. Maybe /tmp and /var/log as well.

Indeed, it can be tedious to write an artifacts conf file for each ptest. If it is tolerable to get an archive of the whole ptest directory (+ possibly /var/log and /tmp), we can go for it and fine-tune later if needed. In this case, I could either keep artifacts.conf to list the directories we want for all ptests, or completely drop it.

I will let this RFC run for more time before starting to implement something, but if no one come back on this, I will apply your suggestion.

Kind regards,

Quoted 92 lines

Alex

Quoted 86 lines · Alexis Lothoré via lists.openembedded.org
Quoted 57 lines · Alexis Lothoré via lists.openembedded.org

Hello everyone,
I have been briefed about the need to add an infrastructure to be able to retrieve some test files (test output, logs, etc) in Yocto/CI automation to ease debugging of some hard-to-reproduce issues. Before starting to implement a solution, I would like to get some feedback about what I have understood from the original issue and what I propose to implement to improve this. Please find below my reflexions, feel free to comment, correct, enrich and/or approve.

# Motivation

Some test failures appearing in automatic testing can be hard to fix, based on too few informations from failing tests (see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes packages/binaries under test are able to generate more detailed output in a dedicated file, but there is actually no standardized way of retrieving such output files. There has been some custom efforts done to solve this issue for specific recipes, for example for lttng-tools tests :
https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408

The goal of this RFC is to propose an architecture for a generic system which allows to retrieve an arbitrary list of files from a test run.

# Use case example

lttng-tools recipe provides LTTNG tools tests as ptests (see meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log files (error.log, test-suite.log), it would be convenient to retrieve those files.

# Proposal

- Introduce an "artifacts.conf" file
- this file contains a "files of interest" list to be retrieved onto host after running ptests (exact format to be defined)
- artifacts.conf is specific to a ptest package and is stored next to run-ptest script in package
- artifacts.conf contains, for each ptest package, the following informations:
- name of file to retrieve
- artifacts.conf will be installed in target image (as for package-specific run-ptest script)
- each ptest package MAY have an artifacts.conf file
- When running ptests, for each available ptest, ptest-runner (on the target) will read artifacts.conf if it exists, and retrieve corresponding artifacts file in an artifacts directory, with one subdirectory per ptest.
- Artifacts will be aggregated only if at least one test has a status different from "PASS"
- Once tests are finished, tests manager on host (ptest.py) will fetch artifacts directory. It will put this directory alongside testresults file on host
- This mechanism will work for both real targets and emulated targets (Qemu), and will based on SSH to transfer artifacts back to host

# Additional questions

Quoted 10 lines

- once retrieved from target, aside from default test results directory (build/tmp/log/oeqa/<...>), where should we store artifacts ? For automated testing, should we publish them alongside other test and regression reports (in web server directory) ?
- current proposal is focused on ptests: should I broaden the scope and include all kind of runtime tests (as found in meta/lib/oeqa/runtime/cases ?)
- are there examples of tests producing files to retrieve with non-static names (which will then require the artifacts.conf to be smarter) ?

Fixing Alexandre Belloni's address in CC, sorry for the noise

--
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181597): https://lists.openembedded.org/g/openembedded-core/message/181597
Mute This Topic: https://lists.openembedded.org/mt/99066378/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw
Quoted 10 lines · Alexis Lothor�

Hello Alexander, thanks for the feedback

Quoted 6 lines · Alexander Kanavin

I think having to write a custom artifacts.conf for each ptest is not a good direction, as it's both too much work, and requires good understanding what is important in the output and what isn't :) When a ptest fails, it would be enough to simply make a tarball of /usr/lib/component/ptest/ with everything in it. Maybe /tmp and /var/log as well.

Quoted 9 lines

Indeed, it can be tedious to write an artifacts conf file for each ptest. If it is tolerable to get an archive of the whole ptest directory (+ possibly /var/log and /tmp), we can go for it and fine-tune later if needed. In this case, I could either keep artifacts.conf to list the directories we want for all ptests, or completely drop it.

I will let this RFC run for more time before starting to implement something, but if no one come back on this, I will apply your suggestion.

I think the requirement that I didn't convey properly is that there should be one global conf file for the build.

Quoted 100 lines

Kind regards,

Quoted 92 lines

Alex

Quoted 86 lines · Alexis Lothor� via lists.openembedded.org
Quoted 57 lines · Alexis Lothor� via lists.openembedded.org

Hello everyone,
I have been briefed about the need to add an infrastructure to be able to retrieve some test files (test output, logs, etc) in Yocto/CI automation to ease debugging of some hard-to-reproduce issues. Before starting to implement a solution, I would like to get some feedback about what I have understood from the original issue and what I propose to implement to improve this. Please find below my reflexions, feel free to comment, correct, enrich and/or approve.

# Motivation

Some test failures appearing in automatic testing can be hard to fix, based on too few informations from failing tests (see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes packages/binaries under test are able to generate more detailed output in a dedicated file, but there is actually no standardized way of retrieving such output files. There has been some custom efforts done to solve this issue for specific recipes, for example for lttng-tools tests :
https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408

The goal of this RFC is to propose an architecture for a generic system which allows to retrieve an arbitrary list of files from a test run.

# Use case example

lttng-tools recipe provides LTTNG tools tests as ptests (see meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log files (error.log, test-suite.log), it would be convenient to retrieve those files.

# Proposal

- Introduce an "artifacts.conf" file
- this file contains a "files of interest" list to be retrieved onto host after running ptests (exact format to be defined)
- artifacts.conf is specific to a ptest package and is stored next to run-ptest script in package
- artifacts.conf contains, for each ptest package, the following informations:
- name of file to retrieve
- artifacts.conf will be installed in target image (as for package-specific run-ptest script)
- each ptest package MAY have an artifacts.conf file
- When running ptests, for each available ptest, ptest-runner (on the target) will read artifacts.conf if it exists, and retrieve corresponding artifacts file in an artifacts directory, with one subdirectory per ptest.
- Artifacts will be aggregated only if at least one test has a status different from "PASS"
- Once tests are finished, tests manager on host (ptest.py) will fetch artifacts directory. It will put this directory alongside testresults file on host
- This mechanism will work for both real targets and emulated targets (Qemu), and will based on SSH to transfer artifacts back to host

# Additional questions

Quoted 10 lines

- once retrieved from target, aside from default test results directory (build/tmp/log/oeqa/<...>), where should we store artifacts ? For automated testing, should we publish them alongside other test and regression reports (in web server directory) ?
- current proposal is focused on ptests: should I broaden the scope and include all kind of runtime tests (as found in meta/lib/oeqa/runtime/cases ?)
- are there examples of tests producing files to retrieve with non-static names (which will then require the artifacts.conf to be smarter) ?

Fixing Alexandre Belloni's address in CC, sorry for the noise

--
Alexis Lothor�, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181597): https://lists.openembedded.org/g/openembedded-core/message/181597
Mute This Topic: https://lists.openembedded.org/mt/99066378/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexis Lothor�, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com