add failed tests artifacts retriever
v2 cover add failed tests artifacts retriever
This series is a proposal to bring in an "artifact retriever" to ease debugging when some runtime tests fails. This is a follow-up to the initial version ([1]) and its corresponding RFC ([2]), which in turn is a proposal to address general debugging issues like [3]
In the proposed form the retriever is pretty simple/dumb: it waits for all tests listed for a testimage run to be done, and if any of those tests has failed, it tries to read a list of "artifacts of interest to retrieve", and pulls those files onto the host system (next to testresults.json) for further analysis. This is true for ALL runtime tests. So for example, a failing test in a very basic test session running ping, ssh and ptests will trigger artifacts retrieval, the failing test being either a ping test, a ssh test or a ptest.
The artifacts list is provided in a new bitbake TESTIMAGE_FAILED_QA_ARTIFACTS.
There is one single artifacts list to be provided for a whole test run (i.e a run done with bitbake -c testimage). Artifacts list can then be overriden by affecting the variable mentioned above.
Retrieved files are pulled through scp to allow compatibility for both Qemu and SSH targets, and are currently stored "as is" (unarchived/uncompressed) in tmp/log/oeqa/<image>/artifacts.
The series has been tested with the following process:
- add dummy failing ptest in lttng-modules through a custom patch in meta/recipes-kernel/lttng
- build core-image-minimal image with:
- DISTRO_FEATURES:append = " ptest"
- CORE_IMAGE_EXTRA_INSTALL += "dropbear lttng-tools-ptest busybox-ptest"
- TEST_SUITES = "ping ssh ptest"
- IMAGE_CLASSES += "testimage"
- TESTIMAGE_FAILED_QA_ARTIFACTS="${libdir}/*/ptest /etc/version /etc/os-release /var/log"
- run tests: bitbake core-image-minimal -c testimage
- ensure artifacts are properly retrieved and stored
Changes since v1:
- remove legacy scp option
- put back target stop in 'finally' clause
- retrieve artifacts only in nominal target run (ie no exception from ssh/qemu target run)
- list artifacts directly in variable instead of using intermediate file
- use standard variables in artifacts paths
- allow glob patterns in artifacts paths
- expand artifacts path on target before tryiong to retrieve them
[1] https://lore.kernel.org/openembedded-core/[email protected]/
[2] https://lore.kernel.org/openembedded-core/[email protected]/T/#t
[3] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901
Alexis Lothoré (2):
oeqa/target/ssh: update options for SCP
testimage: implement test artifacts retriever for failing tests
meta/classes-recipe/testimage.bbclass | 49 +++++++++++++++++++++++++++ meta/lib/oeqa/core/target/ssh.py | 5 ++- 2 files changed, 53 insertions(+), 1 deletion(-)
Signature · 1 line
2.40.1
PATCH v2 1/2 oeqa/target/ssh: update options for SCP
By default scp expects files. Passing -r option allows to copy directories too
---
Changes since v1:
- drop legacy scp protocol option
--- meta/lib/oeqa/core/target/ssh.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
meta/lib/oeqa/core/target/ssh.py +4 -1
@@ -40,8 +40,11 @@ class OESSHTarget(OETarget): '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR' ] + scp_options = [ + '-r' + ] self.ssh = ['ssh', '-l', self.user ] + ssh_options - self.scp = ['scp'] + ssh_options + self.scp = ['scp'] + ssh_options + scp_options if port: self.ssh = self.ssh + [ '-p', port ] self.scp = self.scp + [ '-P', port ]
Signature · 1 line
2.40.1
PATCH v2 2/2 testimage: implement test artifacts retriever for failing tests
In reply to Alexis LothoréAdd a basic artifacts retrievers in testimage class which:
- triggers when at least one runtime test fails but tests execution encountered no major issue
- reads a list of paths to retrieve from TESTIMAGE_FAILED_QA_ARTIFACTS
- checks for artifacts presence on target
- retrieve those files over scp thanks to existing ssh class
- store those files in an "artifacts" directory in "tmp/log/oeqa/<image>"
This implementation assumes that the SSH or Qemu target has run and finished gracefully. If tests do not finish because of an exception, artifacts will not be retrieved
Bring partial solution to [YOCTO #14901]
---
Changes since v1:
- only gather artifacts in nominal case (ie qemu runs without any raised exception)
- list artifacts directly in variable instead of using external file
- use standard variables in artifacts paths
- allow glob patterns usage in artifacts paths
- expand/filter artifacts list on target before retrieving them
- tune default artifacts list
--- meta/classes-recipe/testimage.bbclass | 49 +++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
meta/classes-recipe/testimage.bbclass +49 -0
@@ -18,6 +18,16 @@ inherit image-artifact-names TESTIMAGE_AUTO ??= "0" +# When any test fails, TESTIMAGE_FAILED_QA ARTIFACTS will be parsed and for +# each entry in it, if artifact pointed by path description exists on target, +# it will be retrieved onto host + +TESTIMAGE_FAILED_QA_ARTIFACTS ??= "\ + ${localstatedir}/log \ + ${sysconfdir}/version \ + ${sysconfdir}/os-release \ + ${libdir}/*/ptest" + # You can set (or append to) TEST_SUITES in local.conf to select the tests # which you want to run for your target. # The test names are the module names in meta/lib/oeqa/runtime/cases.
@@ -192,6 +202,39 @@ def get_testimage_boot_patterns(d): boot_patterns[flag] = flagval.encode().decode('unicode-escape') return boot_patterns +def get_artifacts_list(target, raw_list): + result = [] + # Passed list may contains patterns in paths, expand them directly on target + for raw_path in raw_list.split(): + cmd = f"for p in {raw_path}; do if [ -e $p ]; then echo $p; fi; done" + try: + status, output = target.run(cmd) + if status != 0 or not output: + raise Exception() + result += output.split() + except: + bb.warn(f"No file/directory matching path {raw_path}") + + return result + +def retrieve_test_artifacts(target, artifacts_list, target_dir): + import shutil + + local_artifacts_dir = os.path.join(target_dir, "artifacts") + if os.path.isdir(local_artifacts_dir): + shutil.rmtree(local_artifacts_dir) + + os.makedirs(local_artifacts_dir) + for artifact_path in artifacts_list: + if not os.path.isabs(artifact_path): + bb.warn(f"{artifact_path} is not an absolute path") + continue + try: + dest_dir = os.path.join(local_artifacts_dir, os.path.dirname(artifact_path[1:])) + os.makedirs(dest_dir, exist_ok=True) + target.copyFrom(artifact_path, dest_dir) + except: + bb.warn(f"Can not retrieve {artifact_path} from test target") def testimage_main(d): import os
@@ -383,6 +426,12 @@ def testimage_main(d): pass results = tc.runTests() complete = True + if not results.wasSuccessful(): + artifacts_list = get_artifacts_list(tc.target, d.getVar("TESTIMAGE_FAILED_QA_ARTIFACTS")) + if not artifacts_list: + bb.warn("Could not load artifacts list, skip artifacts retrieval") + else: + retrieve_test_artifacts(tc.target, artifacts_list, get_testimage_json_result_dir(d)) except (KeyboardInterrupt, BlockingIOError) as err: if isinstance(err, KeyboardInterrupt): bb.error('testimage interrupted, shutting down...')
Signature · 1 line
2.40.1
Re: [OE-Core][PATCH v2 2/2] testimage: implement test artifacts retriever for failing tests
Quoted 35 lines · Alexis Lothoré via lists.openembedded.org
Add a basic artifacts retrievers in testimage class which:
- triggers when at least one runtime test fails but tests execution encountered no major issue
- reads a list of paths to retrieve fromTESTIMAGE_FAILED_QA_ARTIFACTS
- checks for artifacts presence on target
- retrieve those files over scp thanks to existing ssh class
- store those files in an "artifacts" directory in "tmp/log/oeqa/<image>"This implementation assumes that the SSH or Qemu target has run and finished gracefully. If tests do not finish because of an exception, artifacts will not be retrieved
Bring partial solution to [YOCTO #14901]
Signed-off-by Alexis Lothoré <[email protected]>---
Changes since v1:
- only gather artifacts in nominal case (ie qemu runs without any raised exception)
- list artifacts directly in variable instead of using external file
- use standard variables in artifacts paths
- allow glob patterns usage in artifacts paths
- expand/filter artifacts list on target before retrieving them
- tune default artifacts list--- meta/classes-recipe/testimage.bbclass | 49 +++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)meta/classes-recipe/testimage.bbclass +0
-0@@ -18,6 +18,16 @@ inherit image-artifact-names TESTIMAGE_AUTO ??= "0"
meta/classes-recipe/testimage.bbclass inherit image-artifact-names · line 29
Quoted 10 lines
meta/classes-recipe/testimage.bbclass +9
-0+# When any test fails, TESTIMAGE_FAILED_QA ARTIFACTS will be parsed and for +# each entry in it, if artifact pointed by path description exists on target, +# it will be retrieved onto host + +TESTIMAGE_FAILED_QA_ARTIFACTS ??= "\ + ${localstatedir}/log \ + ${sysconfdir}/version \ + ${sysconfdir}/os-release \ + ${libdir}/*/ptest"
I know Alex Kanavin asked for this but I'm really not sure ${libdir}/*/ptest is a great idea. This will effectively copy the entire ptest package files for every ptest on the image. Now we split ptests to different images it is less of an issue but it is still potentially rather large. Could we at least limit it to failing ptests?
Cheers,
Richard
Re: [OE-Core][PATCH v2 2/2] testimage: implement test artifacts retriever for failing tests
What might work better without code complications is testimage.bbclass setting only the minimum set (no ptests):
+TESTIMAGE_FAILED_QA_ARTIFACTS ??= "\
+ ${localstatedir}/log \
+ ${sysconfdir}/version \
+ ${sysconfdir}/os-release \"
Then core-image-ptest.bb should append "${libdir}/{MCNAME}/ptest" to that. As that same image recipe installs only a single ptest via IMAGE_INSTALL:append, and the artifacts are retrieved only if that ptest would fail, this would achieve the same outcome. Other images (which can potentially include many ptests) can then set what artifacts to retrieve themselves as they please.
Alex
Quoted 69 lines · Richard Purdie
Quoted 35 lines · Alexis Lothoré via lists.openembedded.org
Add a basic artifacts retrievers in testimage class which:
- triggers when at least one runtime test fails but tests execution encountered no major issue
- reads a list of paths to retrieve fromTESTIMAGE_FAILED_QA_ARTIFACTS
- checks for artifacts presence on target
- retrieve those files over scp thanks to existing ssh class
- store those files in an "artifacts" directory in "tmp/log/oeqa/<image>"This implementation assumes that the SSH or Qemu target has run and finished gracefully. If tests do not finish because of an exception, artifacts will not be retrieved
Bring partial solution to [YOCTO #14901]
Signed-off-by Alexis Lothoré <[email protected]>---
Changes since v1:
- only gather artifacts in nominal case (ie qemu runs without any raised exception)
- list artifacts directly in variable instead of using external file
- use standard variables in artifacts paths
- allow glob patterns usage in artifacts paths
- expand/filter artifacts list on target before retrieving them
- tune default artifacts list--- meta/classes-recipe/testimage.bbclass | 49 +++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)meta/classes-recipe/testimage.bbclass +0
-0@@ -18,6 +18,16 @@ inherit image-artifact-names TESTIMAGE_AUTO ??= "0"meta/classes-recipe/testimage.bbclass inherit image-artifact-names · line 29
Quoted 10 lines
meta/classes-recipe/testimage.bbclass +9
-0+# When any test fails, TESTIMAGE_FAILED_QA ARTIFACTS will be parsed and for +# each entry in it, if artifact pointed by path description exists on target, +# it will be retrieved onto host + +TESTIMAGE_FAILED_QA_ARTIFACTS ??= "\ + ${localstatedir}/log \ + ${sysconfdir}/version \ + ${sysconfdir}/os-release \ + ${libdir}/*/ptest"I know Alex Kanavin asked for this but I'm really not sure ${libdir}/*/ptest is a great idea. This will effectively copy the entire ptest package files for every ptest on the image. Now we split ptests to different images it is less of an issue but it is still potentially rather large. Could we at least limit it to failing ptests?
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#182462): https://lists.openembedded.org/g/openembedded-core/message/182462
Mute This Topic: https://lists.openembedded.org/mt/99380468/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-Core][PATCH v2 2/2] testimage: implement test artifacts retriever for failing tests
Hello Alexander, Richard,
Thanks for the feedback. Indeed my testing setup is quite minimal, so it may not reflect how many files may be pulled in real cases.
Quoted 10 lines · Alexander Kanavin
What might work better without code complications is testimage.bbclass setting only the minimum set (no ptests):
meta/classes-recipe/testimage.bbclass +4
-0+TESTIMAGE_FAILED_QA_ARTIFACTS ??= "\ + ${localstatedir}/log \ + ${sysconfdir}/version \ + ${sysconfdir}/os-release \"
Quoted 6 lines
Then core-image-ptest.bb should append "${libdir}/{MCNAME}/ptest" to that. As that same image recipe installs only a single ptest via IMAGE_INSTALL:append, and the artifacts are retrieved only if that ptest would fail, this would achieve the same outcome. Other images (which can potentially include many ptests) can then set what artifacts to retrieve themselves as they please.
I am not familiar with multiconfig, so I'll have to document myself before trying what you suggest, especially to make sure it will not make the retriever too dependant on a proper multiconfig definition.
Thanks,
Alexis
Quoted 74 lines
Alex
Quoted 69 lines · Richard Purdie
Quoted 35 lines · Alexis Lothoré via lists.openembedded.org
Add a basic artifacts retrievers in testimage class which:
- triggers when at least one runtime test fails but tests execution encountered no major issue
- reads a list of paths to retrieve fromTESTIMAGE_FAILED_QA_ARTIFACTS
- checks for artifacts presence on target
- retrieve those files over scp thanks to existing ssh class
- store those files in an "artifacts" directory in "tmp/log/oeqa/<image>"This implementation assumes that the SSH or Qemu target has run and finished gracefully. If tests do not finish because of an exception, artifacts will not be retrieved
Bring partial solution to [YOCTO #14901]
Signed-off-by Alexis Lothoré <[email protected]>---
Changes since v1:
- only gather artifacts in nominal case (ie qemu runs without any raised exception)
- list artifacts directly in variable instead of using external file
- use standard variables in artifacts paths
- allow glob patterns usage in artifacts paths
- expand/filter artifacts list on target before retrieving them
- tune default artifacts list--- meta/classes-recipe/testimage.bbclass | 49 +++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)meta/classes-recipe/testimage.bbclass +0
-0@@ -18,6 +18,16 @@ inherit image-artifact-names TESTIMAGE_AUTO ??= "0"meta/classes-recipe/testimage.bbclass inherit image-artifact-names · line 29
Quoted 10 lines
meta/classes-recipe/testimage.bbclass +9
-0+# When any test fails, TESTIMAGE_FAILED_QA ARTIFACTS will be parsed and for +# each entry in it, if artifact pointed by path description exists on target, +# it will be retrieved onto host + +TESTIMAGE_FAILED_QA_ARTIFACTS ??= "\ + ${localstatedir}/log \ + ${sysconfdir}/version \ + ${sysconfdir}/os-release \ + ${libdir}/*/ptest"I know Alex Kanavin asked for this but I'm really not sure ${libdir}/*/ptest is a great idea. This will effectively copy the entire ptest package files for every ptest on the image. Now we split ptests to different images it is less of an issue but it is still potentially rather large. Could we at least limit it to failing ptests?
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#182462): https://lists.openembedded.org/g/openembedded-core/message/182462
Mute This Topic: https://lists.openembedded.org/mt/99380468/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [OE-Core][PATCH v2 2/2] testimage: implement test artifacts retriever for failing tests
Quoted 10 lines · Alexis Lothoré
Quoted 6 lines
Then core-image-ptest.bb should append "${libdir}/{MCNAME}/ptest" to that. As that same image recipe installs only a single ptest via IMAGE_INSTALL:append, and the artifacts are retrieved only if that ptest would fail, this would achieve the same outcome. Other images (which can potentially include many ptests) can then set what artifacts to retrieve themselves as they please.
I am not familiar with multiconfig, so I'll have to document myself before trying what you suggest, especially to make sure it will not make the retriever too dependant on a proper multiconfig definition.
core-image-ptest.bb is already set for this, so it just adds another usage of MCNAME variable into it to get the name of the ptest from the mcextend variant and substitute that into the path that gets added to TESTIMAGE_FAILED_QA_ARTIFACTS. The retriever won't know anything about it as it happens at the recipe parsing step.
Alex
Re: [OE-Core][PATCH v2 2/2] testimage: implement test artifacts retriever for failing tests
Quoted 12 lines · Alexander Kanavin
Quoted 10 lines · Alexis Lothoré
Quoted 6 lines
Then core-image-ptest.bb should append "${libdir}/{MCNAME}/ptest" to that. As that same image recipe installs only a single ptest via IMAGE_INSTALL:append, and the artifacts are retrieved only if that ptest would fail, this would achieve the same outcome. Other images (which can potentially include many ptests) can then set what artifacts to retrieve themselves as they please.
I am not familiar with multiconfig, so I'll have to document myself before trying what you suggest, especially to make sure it will not make the retriever too dependant on a proper multiconfig definition.
Quoted 5 lines
core-image-ptest.bb is already set for this, so it just adds another usage of MCNAME variable into it to get the name of the ptest from the mcextend variant and substitute that into the path that gets added to
TESTIMAGE_FAILED_QA_ARTIFACTS. The retriever won't know anything about it as it happens at the recipe parsing step.
It does seem like a neat way to handle things to me FWIW.
Cheers,
Richard