test-manual code block, link, and format update
PATCH 1/1 test-manual code block, link, and format update
--- documentation/conf.py | 4 +- .../test-manual/test-manual-intro.rst | 266 +++++++++++------- .../test-manual-understand-autobuilder.rst | 102 ++++--- 3 files changed, 226 insertions(+), 146 deletions(-)
documentation/conf.py +2 -2
@@ -42,8 +42,8 @@ sys.path.insert(0, os.path.abspath('sphinx')) extensions = [ 'sphinx.ext.autosectionlabel', 'sphinx.ext.extlinks', - 'sphinx.ext.intersphinx', - 'yocto-vars' + 'sphinx.ext.intersphinx' + # 'yocto-vars' ] autosectionlabel_prefix_document = True
documentation/test-manual/test-manual-intro.rst +164 -102
418 lines not on this page · raw message
documentation/test-manual/test-manual-understand-autobuilder.rst +60 -42
166 lines not on this page · raw message
Signature · 1 line
2.17.1
hi Mark,
many thanks for your help!
Quoted 12 lines · Mark Morton
Signed-off-by Mark Morton <[email protected]>--- documentation/conf.py | 4 +- .../test-manual/test-manual-intro.rst | 266 +++++++++++------- .../test-manual-understand-autobuilder.rst | 102 ++++--- 3 files changed, 226 insertions(+), 146 deletions(-)documentation/conf.py +0
-0
documentation/conf.py sys.path.insert(0, os.path.abspath('sphinx')) · line 48
Quoted 9 lines
documentation/conf.py +2
-2@@ -42,8 +42,8 @@ sys.path.insert(0, os.path.abspath('sphinx')) extensions = [ 'sphinx.ext.autosectionlabel', 'sphinx.ext.extlinks', - 'sphinx.ext.intersphinx', - 'yocto-vars' + 'sphinx.ext.intersphinx' + # 'yocto-vars' ]
This extension doesn't work with Python2. To be honest, I don't think i want to bother with Python2 at this point. I am not even sure how well Sphinx is expected to work with Python2, so we should make clear that our doc system requires Python3, and we need to write clear instructions for users who have Python3 on their host as 'default' vs users who still have Python2 (which is the problem you faced)
autosectionlabel_prefix_document = True
Quoted 17 lines
documentation/test-manual/test-manual-intro.rst +1
-3@@ -23,26 +23,25 @@ project core. Currently, the Yocto Project Test Environment Manual has no projected release date. This manual is a work-in-progress and is being initially -loaded with information from the `README <>`__ files and notes from key +loaded with information from the README files and notes from key engineers: -- *``yocto-autobuilder2``:* This - ```README.md`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md
Quoted 5 lines
+- *yocto-autobuilder2:* This + `README.md <http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md
Quoted 1 line
`__
that works. However for links to git trees, we are using the :yocto_git:
extlink , such as (from the BSP guide):
:yocto_git:README.md </cgit.cgi/meta-raspberrypi/tree/README.md> We should standardize on that.
I think i will eventually make sure we don't even need /cgit.git in the link name, but for now it's needed.
is the main README which detials how to set up the Yocto Project
Quoted 19 lines
Autobuilder. The ``yocto-autobuilder2`` repository represents the Yocto Project's console UI plugin to Buildbot and the configuration necessary to configure Buildbot to perform the testing the project requires.
-- *``yocto-autobuilder-helper``:* This
- ```README`
http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README
+- *yocto-autobuilder-helper:* This `README <
http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README/Quoted 1 line
`__
and repository contains Yocto Project Autobuilder Helper scripts and configuration. The ``yocto-autobuilder-helper`` repository contains the "glue" logic that defines which tests to run and how to run them. As a result, it can be used by any Continuous Improvement (CI) system to run builds, support getting the correct code revisions, configure builds and layers, run builds, and collect results. The code is
documentation/test-manual/test-manual-intro.rst +1
-1- independent of any CI system, which means the code can work Buildbot, + independent of any CI system, which means the code can work `Buildbot <
Quoted 2 lines
this is a new link you created which was not in the original Docbook, right? Perhaps best to have that in a separate patch if you think it's needed.
documentation/test-manual/test-manual-intro.rst that the Yocto Project customizes using code from the · line 69
Quoted 10 lines
Jenkins, or others. This repository has a branch per release of the project defining the tests to run on a per release basis.
documentation/test-manual/test-manual-intro.rst +1
-1@@ -66,9 +65,9 @@ that the Yocto Project customizes using code from the resulting UI plug-in allows you to visualize builds in a way suited to the project's needs. -A ``helper`` layer provides configuration and job management through +A helper layer provides configuration and job management through
Why? 'helper' is inside a <filename> tag in docbook to make it verbatim in the output, why did you remove it?
scripts found in the ``yocto-autobuilder-helper`` repository. The
Quoted 6 lines
documentation/test-manual/test-manual-intro.rst +1
-1-``helper`` layer contains the bulk of the build configuration +helper layer contains the bulk of the build configuration information and is release-specific, which makes it highly customizable on a per-project basis. The layer is CI system-agnostic and contains a number of Helper scripts that can generate build configurations from
documentation/test-manual/test-manual-intro.rst thefollowing types of tests: · line 127
Quoted 8 lines
documentation/test-manual/test-manual-intro.rst +1
-3@@ -124,18 +123,21 @@ thefollowing types of tests: ensure we have no single point of failure and can ensure the different distros work effectively. -- *eSDK Testing:* Image tests initiated through the following command: - $ bitbake image -c testsdkext The tests utilize the ``testsdkext`` - class and the ``do_testsdkext`` task. +- *eSDK Testing:* Image tests initiated through the following command::
does that work? I've always done ": ::" to keep the semi column, and I thought the :: needed a space before.
documentation/test-manual/test-manual-intro.rst thefollowing types of tests: · line 131
Quoted 3 lines
documentation/test-manual/test-manual-intro.rst +2
-0+ + $ bitbake image -c testsdkext
Indentation is not typical here. you need 3 space starting from the previous indentation level.
Quoted 4 lines
documentation/test-manual/test-manual-intro.rst +2
-0+ + The tests utilize the ``testsdkext`` class and the ``do_testsdkext``task.
this sentence above is no in the code-block, it should not be at the same indentation level as the code block.
Quoted 13 lines
- *Feature Testing:* Various scenario-based tests are run through the
documentation/test-manual/test-manual-intro.rst +1
-3- `OpenEmbedded - Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__ - (oe-selftest). We test oe-selftest on each of the main distrubutions + :ref:`OpenEmbedded Self test (oe-selftest)<ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions
we support.-- *Image Testing:* Image tests initiated through the following command:
documentation/test-manual/test-manual-intro.rst thefollowing types of tests: · line 140
Quoted 6 lines
documentation/test-manual/test-manual-intro.rst +5
-1- $ bitbake image -c testimage The tests utilize the +- *Image Testing:* Image tests initiated through the following command:: + + $ bitbake image -c testimage + + The tests utilize the
Need to fix indentation here
:ref:testimage* <ref-classes-testimage*>
Quoted 20 lines
classes and the
:ref:ref-tasks-testimagetask.documentation/test-manual/test-manual-intro.rst +2
-2@@ -147,8 +149,8 @@ thefollowing types of tests: - *Package Testing:* A Package Test (ptest) runs tests against packages built by the OpenEmbedded build system on the target machine. See the - "`Testing Packages With - ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section + :ref:`Testing Packages With + ptest <dev-manual/dev-manual-common-tasks:Testing Packages Withptest>` section
in the Yocto Project Development Tasks Manual and the ":yocto_wiki:Ptest </wiki/Ptest>" Wiki page for more information on Ptest.documentation/test-manual/test-manual-intro.rst +0
-2@@ -159,8 +161,8 @@ thefollowing types of tests: the ``do_testsdk`` task. - *Unit Testing:* Unit tests on various components of the system run - through ``oe-selftest`` and - ```bitbake-selftest``
Quoted 6 lines
<&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__.
+ through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing
and Quality Assurance>` and
+ :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality
Assurance>`.
the link is on bitbake-selftest, not oe-selftest (you added it).
Quoted 10 lines
- *Automatic Upgrade Helper:* This target tests whether new versions of software are available and whether we can automatically upgrade to
documentation/test-manual/test-manual-intro.rst +1
-1@@ -174,36 +176,43 @@ How Tests Map to Areas of Code Tests map into the codebase as follows: -- *bitbake-selftest*: +- *bitbake-selftest:*
documentation/test-manual/test-manual-intro.rst How Tests Map to Areas of Code · line 188
Quoted 10 lines
documentation/test-manual/test-manual-intro.rst +3
-2These tests are self-contained and test BitBake as well as its APIs, which include the fetchers. The tests are located in ``bitbake/lib/*/tests``. - From within the BitBake repository, run the following: $ - bitbake-selftest + From within the BitBake repository, run the following:: + + $ bitbake-selftest
indentation and use of :: are not right (please check all other occurrences)
Quoted 420 lines
documentation/test-manual/test-manual-intro.rst +11
-8To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` - variable when running "bitbake-selftest" as follows: $ - BB_SKIP_NETTEST=yes bitbake-selftest + variable when running "bitbake-selftest" as follows:: + + $ BB_SKIP_NETTEST=yes bitbake-selftest The default output is quiet and just prints a summary of what was - run. To see more information, there is a verbose option:$ - bitbake-selftest -v + run. To see more information, there is a verbose option:: + + $ bitbake-selftest -v Use this option when you wish to skip tests that access the network, which are mostly necessary to test the fetcher modules. To specify individual test modules to run, append the test module name to the "bitbake-selftest" command. For example, to specify the tests for the - bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can - also specify individual tests by defining the full name and module - plus the class path of the test, for example: $ bitbake-selftest - bb.tests.data.TestOverrides.test_one_override + bb.data.module, run:: + + $ bitbake-selftest bb.test.data.module + + You can also specify individual tests by defining the full name andmodule
documentation/test-manual/test-manual-intro.rst +50
-24+ plus the class path of the test, for example:: + + $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override The tests are based on `Python unittest <https://docs.python.org/3/library/unittest.html>`__. -- *oe-selftest*: +- *oe-selftest:* - These tests use OE to test the workflows, which include testing specific features, behaviors of tasks, and API unit tests.@@ -219,16 +228,25 @@ Tests map into the codebase as follows: - The code for the tests resides in ``meta/lib/oeqa/selftest/cases/``. - - To run all the tests, enter the following command: $ oe-selftest - -a + - To run all the tests, enter the following command:: + + $ oe-selftest -a - To run a specific test, use the following command form where - testname is the name of the specific test: $ oe-selftest -r - testname For example, the following command would run the tinfoil - getVar API test:$ oe-selftest -r - tinfoil.TinfoilTests.test_getvarIt is also possible to run a set + testname is the name of the specific test:: + + $ oe-selftest -r <testname> + + For example, the following command would run the tinfoil + getVar API test:: + + $ oe-selftest -r tinfoil.TinfoilTests.test_getvar + + It is also possible to run a set of tests. For example the following command will run all of the - tinfoil tests:$ oe-selftest -r tinfoil + tinfoil tests:: + + $ oe-selftest -r tinfoil - *testimage:*@@ -240,10 +258,13 @@ Tests map into the codebase as follows: - You need to set the :term:`IMAGE_CLASSES` - variable as follows: IMAGE_CLASSES += "testimage" + variable as follows:: + + IMAGE_CLASSES += "testimage" - - Run the tests using the following command form: $ bitbake image -c - testimage + - Run the tests using the following command form:: + + $ bitbake image -c testimage - *testsdk:*@@ -252,8 +273,9 @@ Tests map into the codebase as follows: - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``. - - Run the test using the following command form: $ bitbake image -c - testsdk + - Run the test using the following command form:: + + $ bitbake image -c testsdk - *testsdk_ext:*@@ -262,8 +284,9 @@ Tests map into the codebase as follows: - The code for these tests resides in ``meta/lib/oeqa/esdk``. - - To run the tests, use the following command form: $ bitbake image - -c testsdkext + - To run the tests, use the following command form:: + + $ bitbake image -c testsdkext - *oe-build-perf-test:*@@ -272,8 +295,11 @@ Tests map into the codebase as follows: - The code for these tests resides in ``meta/lib/oeqa/buildperf``. - - To run the tests, use the following command form: $ - oe-build-perf-test optionsThe command takes a number of options, + - To run the tests, use the following command form:: + + $ oe-build-perf-test <options> + + The command takes a number of options, such as where to place the test results. The Autobuilder Helper Scripts include the ``build-perf-test-wrapper`` script with examples of how to use the oe-build-perf-test from the command@@ -308,14 +334,20 @@ directory. ``bitbake-selftest`` -------------------- -A simple test example from ``lib/bb/tests/data.py`` is: class -DataExpansions(unittest.TestCase): def setUp(self): self.d = -bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] = -"value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def -test_one_var(self): val = self.d.expand("${foo}") -self.assertEqual(str(val), "value_of_foo") +A simple test example from ``lib/bb/tests/data.py`` is:: -In this example, a ```DataExpansions`` <>`__ class of tests is created, + class DataExpansions(unittest.TestCase): + def setUp(self): + self.d = bb.data.init() + self.d["foo"] = "value_of_foo" + self.d["bar"] = "value_of_bar" + self.d["value_of_foo"] = "value_of_'value_of_foo'" + + def test_one_var(self): + val = self.d.expand("${foo}") + self.assertEqual(str(val), "value_of_foo") + +In this example, a ``DataExpansions`` class of tests is created, derived from standard python unittest. The class has a common ``setUp`` function which is shared by all the tests in the class. A simple test is then added to test that when a variable is expanded, the correct value@@ -334,14 +366,15 @@ These tests are more complex due to the setuprequired behind the scenes
for full builds. Rather than directly using Python's unittest, the code wraps most of the standard objects. The tests can be simple, such as testing a command from within the OE build environment using thedocumentation/test-manual/test-manual-intro.rst +6
-5-following example:class BitbakeLayers(OESelftestTestCase): def -test_bitbakelayers_showcrossdepends(self): result = -runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in -result.output, msg = "No dependencies were shown. bitbake-layers -show-cross-depends output: %s"% result.output) +following example:: + + class BitbakeLayers(OESelftestTestCase): + def test_bitbakelayers_showcrossdepends(self): + result = runCmd('bitbake-layers show-cross-depends') + self.assertTrue('aspell' in result.output, msg = "Nodependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
documentation/test-manual/test-manual-intro.rst +2
-2-creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived +creates a testcase from the ``OESelftestTestCase`` class, derived from ``unittest.TestCase``, which runs the ``bitbake-layers`` command and checks the output to ensure it contains something we know should be here.@@ -367,7 +400,7 @@ so tests within a given test class should always run in the same build, while tests in different classes or modules may be split into different builds. There is no data store available for these tests since the tests launch the ``bitbake`` command and exist outside of its context. As a -result, common bitbake library functions (bb.*) are also unavailable. +result, common bitbake library functions (bb.\*) are also unavailable. .. _testimage-example:@@ -378,14 +411,18 @@ These tests are run once an image is up and running,either on target
hardware or under QEMU. As a result, they are assumed to be running in a target image environment, as opposed to a host build environment. A simple example from ``meta/lib/oeqa/runtime/cases/python.py`` containsdocumentation/test-manual/test-manual-intro.rst +7
-8-the following:class PythonTest(OERuntimeTestCase): -@OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core']) -def test_python3(self): cmd = "python3 -c \\"import codecs; -print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output = -self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output -self.assertEqual(status, 0, msg=msg) - -In this example, the ```OERuntimeTestCase`` <>`__ class wraps +the following:: + + class PythonTest(OERuntimeTestCase): + @OETestDepends(['ssh.SSHTest.test_ssh']) + @OEHasPackage(['python3-core']) + def test_python3(self): + cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb,jbeyq', 'rot13'))\""
documentation/test-manual/test-manual-intro.rst +39
-21+ status, output = self.target.run(cmd) + msg = 'Exit status was not 0. Output: %s' % output + self.assertEqual(status, 0, msg=msg) + +In this example, the ``OERuntimeTestCase`` class wraps ``unittest.TestCase``. Within the test, ``self.target`` represents the target system, where commands can be run on it using the ``run()`` method.@@ -402,19 +439,30 @@ the image. These tests are run against built extensible SDKs (eSDKs). The tests can assume that the eSDK environment has already been setup. An example from -``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class -DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls): -myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst = -os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src, -cls.myapp_dst) subprocess.check_output(['git', 'init', '.'], -cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'], -cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m', -"'test commit'"], cwd=cls.myapp_dst) @classmethod def -tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def -\_test_devtool_build(self, directory): self._run('devtool add myapp %s' -% directory) try: self._run('devtool build myapp') finally: -self._run('devtool reset myapp') def test_devtool_build_make(self): -self._test_devtool_build(self.myapp_dst)In this example, the ``devtool`` +``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: + + class DevtoolTest(OESDKExtTestCase): + @classmethod def setUpClass(cls): + myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") + cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") + shutil.copytree(myapp_src, cls.myapp_dst) + subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) + subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) + subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) + + @classmethod + def tearDownClass(cls): + shutil.rmtree(cls.myapp_dst) + def _test_devtool_build(self, directory): + self._run('devtool add myapp %s' % directory) + try: + self._run('devtool build myapp') + finally: + self._run('devtool reset myapp') + def test_devtool_build_make(self): + self._test_devtool_build(self.myapp_dst) + +In this example, the ``devtool`` command is tested to see whether a sample application can be built with the ``devtool build`` command within the eSDK.@@ -426,14 +474,20 @@ the ``devtool build`` command within the eSDK. These tests are run against built SDKs. The tests can assume that an SDK has already been extracted and its environment file has been sourced. A simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the -following:class Python3Test(OESDKTestCase): def setUp(self): if not -(self.tc.hasHostPackage("nativesdk-python3-core") or -self.tc.hasHostPackage("python3-core-native")): raise -unittest.SkipTest("No python3 package in the SDK") def -test_python3(self): cmd = "python3 -c \\"import codecs; -print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd) -self.assertEqual(output, "Hello, world\n")In this example, if -nativesdk-python3-core has been installed into the SDK, the code runs +following:: + + class Python3Test(OESDKTestCase): + def setUp(self): + if not (self.tc.hasHostPackage("nativesdk-python3-core") or + self.tc.hasHostPackage("python3-core-native")): + raise unittest.SkipTest("No python3 package in the SDK") + + def test_python3(self): + cmd = "python3 -c \\"import codecs;
print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""documentation/test-manual/test-manual-intro.rst +4
-0+ output = self._run(cmd) + self.assertEqual(output, "Hello, world\n") + +In this example, if nativesdk-python3-core has been installed into theSDK, the code runs
the python3 interpreter with a basic command to check it is working correctly. The test would only run if python3 is installed in the SDK.documentation/test-manual/test-manual-intro.rst +0
-0@@ -444,17 +498,25 @@ correctly. The test would only run if python3 isinstalled in the SDK.
The performance tests usually measure how long operations take and the resource utilisation as that happens. An example from
-``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:classdocumentation/test-manual/test-manual-intro.rst +8
-10-Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake --p)""" # Drop all caches and parse self.rm_cache() -oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) -self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no -caches)') # Drop tmp/cache -oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) -self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no -tmp/cache)') # Parse with fully cached data -self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p -(cached)')This example shows how three specific parsing timings are +``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:: + + class Test3(BuildPerfTestCase): + def test3(self): + """Bitbake parsing (bitbake -p)""" + # Drop all caches and parse + self.rm_cache() + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'),True)
documentation/test-manual/test-manual-intro.rst +4
-0+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', + 'bitbake -p (no caches)') + # Drop tmp/cache + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'),True)
documentation/test-manual/test-manual-intro.rst +7
-0+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', + 'bitbake -p (no tmp/cache)') + # Parse with fully cached data + self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', + 'bitbake -p (cached)') + +This example shows how three specific parsing timings are measured, with and without various caches, to show how BitBake’s parsing performance trends over time.diff --git
a/documentation/test-manual/test-manual-understand-autobuilder.rstb/documentation/test-manual/test-manual-understand-autobuilder.rstindex8d3c84415..0d987d245100644documentation/test-manual/test-manual-understand-autobuilder.rst +0
-0@@ -20,32 +20,48 @@ each of those targets on a seperate buildbot worker.To understand the
configuration, you need to look at the entry on ``config.json`` file within the ``yocto-autobuilder-helper`` repository. The targets are defined in the ‘overrides’ section, a quick example could be qemux86-64documentation/test-manual/test-manual-understand-autobuilder.rst +20
-20-which looks like:"qemux86-64" : { "MACHINE" : "qemux86-64", "TEMPLATE" : -"arch-qemu", "step1" : { "extravars" : [ "IMAGE_FSTYPES_append = ' wic -wic.bmap'" ] } },And to expand that, you need the “arch-qemu” entry from -the “templates” section, which looks like:"arch-qemu" : { "BUILDINFO" : -true, "BUILDHISTORY" : true, "step1" : { "BBTARGETS" : "core-image-sato -core-image-sato-dev core-image-sato-sdk core-image-minimal -core-image-minimal-dev core-image-sato:do_populate_sdk", "SANITYTARGETS" -: "core-image-minimal:do_testimage core-image-sato:do_testimage -core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" }, "step2" -: { "SDKMACHINE" : "x86_64", "BBTARGETS" : -"core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext -core-image-sato:do_populate_sdk_ext", "SANITYTARGETS" : -"core-image-sato:do_testsdk core-image-minimal:do_testsdkext -core-image-sato:do_testsdkext" }, "step3" : { "BUILDHISTORY" : false, -"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -${HELPERSTMACHTARGS} -j 15"], "ADDLAYER" : -["${BUILDDIR}/../meta-selftest"] } },Combining these two entries you can -see that “qemux86-64” is a three step build where the -``bitbake BBTARGETS`` would be run, then ``bitbake - SANITYTARGETS`` for each step; all for +which looks like:: + + "qemux86-64" : { + "MACHINE" : "qemux86-64", + "TEMPLATE" : "arch-qemu", + "step1" : { + "extravars" : [ + "IMAGE_FSTYPES_append = ' wic wic.bmap'" + ] + } + }, + +And to expand that, you need the “arch-qemu” entry from +the “templates” section, which looks like:: + + "arch-qemu" : { + "BUILDINFO" : true, + "BUILDHISTORY" : true, + "step1" : { + "BBTARGETS" : "core-image-sato core-image-sato-devcore-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
+ "SANITYTARGETS" : "core-image-minimal:do_testimage
core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"documentation/test-manual/test-manual-understand-autobuilder.rst +4
-0+ }, + "step2" : { + "SDKMACHINE" : "x86_64", + "BBTARGETS" : "core-image-sato:do_populate_sdkcore-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
+ "SANITYTARGETS" : "core-image-sato:do_testsdk
core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"documentation/test-manual/test-manual-understand-autobuilder.rst +4
-0+ }, + "step3" : { + "BUILDHISTORY" : false, + "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1oe-selftest ${HELPERSTMACHTARGS} -j 15"],
documentation/test-manual/test-manual-understand-autobuilder.rst +5
-0+ "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] + } + }, + +Combining these two entries you can see that “qemux86-64” is a three stepbuild where the
+``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step 1 an extra variable is added to the ``auto.conf`` file to enable wic image generation.While not every detail of this is covered here, you can see how the
documentation/test-manual/test-manual-understand-autobuilder.rst each of those targets on a seperate buildbot worker. To understand the · line 64
Quoted 2 lines
documentation/test-manual/test-manual-understand-autobuilder.rst +1
-1-templating mechanism allows quite complex configurations to be built up +template mechanism allows quite complex configurations to be built up
why? let's stick to the original .xml file, we can fix the doc later.
documentation/test-manual/test-manual-understand-autobuilder.rst roughly consist of: · line 90
Quoted 10 lines
yet allows duplication and repetition to be kept to a minimum.
The different build targets are designed to allow for parallelisation,
documentation/test-manual/test-manual-understand-autobuilder.rst +1
-1@@ -70,7 +86,7 @@ roughly consist of: This cleans out any previous build. Old builds are left around to allow easier debugging of failed builds. For additional information, - see ```clobberdir`` <#test-clobberdir>`__. + see :ref:`test-clobberdir`.
It works. However we are now using autosectionlabel which creates sections labels for headings automatically. So I would like to get rid of all custom labels, as much as possible. the following is preferred:
:ref:test-manual/test-manual-understand-autobuilder:clobberdir>
I know we've left some links like that in other manuals. So it's not strictly required to change it now.
documentation/test-manual/test-manual-understand-autobuilder.rst roughly consist of: · line 114
Quoted 9 lines
documentation/test-manual/test-manual-understand-autobuilder.rst +1
-12. *Obtain yocto-autobuilder-helper*@@ -94,7 +110,7 @@ roughly consist of: from the ``layerinfo.json`` file to help understand the configuration. It will also use a local cache of repositories to speed up the clone checkouts. For additional information, see - `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__. + :ref:`test-autobuilder-clone-cache`.
same here (and probably in other places)
Quoted 28 lines
documentation/test-manual/test-manual-understand-autobuilder.rst +0
-0This step has two possible modes of operation. If the build is part of a parent build, its possible that all the repositories needed may@@ -137,8 +153,7 @@ special script that moves files to a special location,rather than
deleting them. Files in this location are deleted by an ``rm`` command, which is run under ``ionice -c 3``. For example, the deletion only happens when there is idle IO capacity on the Worker. The Autobuilderdocumentation/test-manual/test-manual-understand-autobuilder.rst +1
-2-Worker Janitor runs this deletion. See `Autobuilder Worker -Janitor <#test-autobuilder-worker-janitor>`__. +Worker Janitor runs this deletion. See:ref:
test-autobuilder-worker-janitor... _test-autobuilder-clone-cache:
documentation/test-manual/test-manual-understand-autobuilder.rst +0
-0@@ -150,8 +165,8 @@ on the Autobuilder. We therefore have a stash ofcommonly used
repositories pre-cloned on the Workers. Data is fetched from these during clones first, then "topped up" with later revisions from any upstream when necesary. The cache is maintained by the Autobuilderdocumentation/test-manual/test-manual-understand-autobuilder.rst +2
-2-Worker Janitor. See `Autobuilder Worker -Janitor <#test-autobuilder-worker-janitor>`__. +Worker Janitor. See :ref:`Autobuilder Worker +Janitor`. .. _test-autobuilder-worker-janitor:
documentation/test-manual/test-manual-understand-autobuilder.rst Autobuilder Worker Janitor · line 179
Quoted 8 lines
documentation/test-manual/test-manual-understand-autobuilder.rst +2
-2@@ -159,8 +174,8 @@ Autobuilder Worker Janitor -------------------------- This is a process running on each Worker that performs two basic -operations, including background file deletion at IO idle (see `Target -Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and +operations, including background file deletion at IO idle (see Target +Execution: clobberdir :ref:`test-list-tgt-exec-clobberdir`) and
the link and text look wrong here. the link is not on the 'text'
documentation/test-manual/test-manual-understand-autobuilder.rst are general setup steps that are run once and include: · line 234
Quoted 10 lines
documentation/test-manual/test-manual-understand-autobuilder.rst +0
-2maintainenance of a cache of cloned repositories to improve the speed the system can checkout repositories.@@ -216,9 +231,6 @@ are general setup steps that are run once and include: For each step that is configured in ``config.json``, it will perform the following: -## WRITER's question: What does "logging in as stepXa" and others refer -to below? ##
you can keep comment with the ".. xxx" syntax.
Quoted 5 lines
documentation/test-manual/test-manual-understand-autobuilder.rst +0
-1- 1. Add any layers that are specified using the ``bitbake-layers add-layer`` command (logging as stepXa)
documentation/test-manual/test-manual-understand-autobuilder.rst to below? ## · line 252
Quoted 9 lines
documentation/test-manual/test-manual-understand-autobuilder.rst +2
-3@@ -235,9 +247,8 @@ to below? ## 6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the BitBake build environment (logging as stepXd) -7. Remove any layers added in `step - 1 <#test-run-config-add-layers-step>`__ using the - ``bitbake-layers remove-layer`` command (logging as stepXa) +7. Remove any layers added in step + 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
you removed a link here, no?
Quoted 44 lines
documentation/test-manual/test-manual-understand-autobuilder.rst +1
-1Once the execution steps above complete, ``run-config`` executes a set of post-build steps, including:@@ -252,7 +263,7 @@ of post-build steps, including: generated to the remote server. 4. Cleanup the build directory using - ```clobberdir`` <#test-clobberdir>`__ if the build was successful, + :ref:`test-clobberdir` if the build was successful, else rename it to “build-renamed” for potential future debugging. .. _test-deploying-yp-autobuilder:@@ -279,11 +290,18 @@ The standard ``config.json`` minimally attempts toallow substitution of
the paths. The Helper script repository includes a ``local-example.json`` file to show how you could override these from a separate configuration file. Pass the following into the environment ofdocumentation/test-manual/test-manual-understand-autobuilder.rst +10
-4-the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As -another example, you could also pass the following into the -environment:$ ABHELPER_JSON="config.json /some/location/local.json"One -issue users often run into is validation of the ``config.json`` files. A +the Autobuilder:: + + $ ABHELPER_JSON="config.json local-example.json" + +As another example, you could also pass the following into the +environment:: + + $ ABHELPER_JSON="config.json /some/location/local.json" + +One issue users often run into is validation of the ``config.json``files. A
tip for minimizing issues from invalid json files is to use a Git ``pre-commit-hook.sh`` script to verify the JSON file before committingdocumentation/test-manual/test-manual-understand-autobuilder.rst +3
-3-it. Create a symbolic link as follows:$ ln -s -../../scripts/pre-commit-hook.sh .git/hooks/pre-commit +it. Create a symbolic link as follows:: + + $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit --2.17.1
From: Nicolas Dechesne <[email protected]>
Sent: Monday, September 14, 2020 11:08 PM
To: Morton, Mark <[email protected]>
Cc: [email protected]
Subject: Re: [docs] [PATCH] test-manual code block, link, and format update
hi Mark,
many thanks for your help!
On Tue, Sep 15, 2020 at 1:16 AM Mark Morton <[email protected]<mailto:[email protected]>> wrote:
--- documentation/conf.py | 4 +- .../test-manual/test-manual-intro.rst | 266 +++++++++++------- .../test-manual-understand-autobuilder.rst | 102 ++++--- 3 files changed, 226 insertions(+), 146 deletions(-)
documentation/conf.py +2 -2
@@ -42,8 +42,8 @@ sys.path.insert(0, os.path.abspath('sphinx')) extensions = [ 'sphinx.ext.autosectionlabel', 'sphinx.ext.extlinks', - 'sphinx.ext.intersphinx', - 'yocto-vars' + 'sphinx.ext.intersphinx' + # 'yocto-vars' ]
This extension doesn't work with Python2. To be honest, I don't think i want to bother with Python2 at this point. I am not even sure how well Sphinx is expected to work with Python2, so we should make clear that our doc system requires Python3, and we need to write clear instructions for users who have Python3 on their host as 'default' vs users who still have Python2 (which is the problem you faced)
autosectionlabel_prefix_document = True
documentation/test-manual/test-manual-intro.rst +3 -3
@@ -23,26 +23,25 @@ project core. Currently, the Yocto Project Test Environment Manual has no projected release date. This manual is a work-in-progress and is being initially -loaded with information from the `README <>`__ files and notes from key +loaded with information from the README files and notes from key engineers: -- *``yocto-autobuilder2``:* This - ```README.md`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md +- *yocto-autobuilder2:* This + `README.md <http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md>`__
that works. However for links to git trees, we are using the :yocto_git: extlink , such as (from the BSP guide):
:yocto_git:README.md </cgit.cgi/meta-raspberrypi/tree/README.md>
We should standardize on that.
I think i will eventually make sure we don't even need /cgit.git in the link name, but for now it's needed.
I updated both links to use :yocto-git:, however because of the Python2 issue, it does not generate in my output. If you have issues, let me know.
is the main README which detials how to set up the Yocto Project Autobuilder. The ``yocto-autobuilder2`` repository represents the Yocto Project's console UI plugin to Buildbot and the configuration necessary to configure Buildbot to perform the testing the project requires.
-- *``yocto-autobuilder-helper``:* This - ```README`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README +- *yocto-autobuilder-helper:* This `README <http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README/>`__
and repository contains Yocto Project Autobuilder Helper scripts and configuration. The ``yocto-autobuilder-helper`` repository contains the "glue" logic that defines which tests to run and how to run them. As a result, it can be used by any Continuous Improvement (CI) system to run builds, support getting the correct code revisions, configure builds and layers, run builds, and collect results. The code is
- independent of any CI system, which means the code can work Buildbot,
+ independent of any CI system, which means the code can work Buildbot <https://docs.buildbot.net/0.9.15.post1/>__,
this is a new link you created which was not in the original Docbook, right? Perhaps best to have that in a separate patch if you think it's needed.
It was in the original – It actually converted just like that with no additional help from me.
Jenkins, or others. This repository has a branch per release of the project defining the tests to run on a per release basis.
@@ -66,9 +65,9 @@ that the Yocto Project customizes using code from the
resulting UI plug-in allows you to visualize builds in a way suited to the project's needs.
-A ``helper`` layer provides configuration and job management through +A helper layer provides configuration and job management through
Why? 'helper' is inside a <filename> tag in docbook to make it verbatim in the output, why did you remove it?
Not really sure why – I thought it was general editing, but on closer inspection, I should have left it in, as it’s the name of a valid layer.
scripts found in the ``yocto-autobuilder-helper`` repository. The
-``helper`` layer contains the bulk of the build configuration +helper layer contains the bulk of the build configuration
information and is release-specific, which makes it highly customizable on a per-project basis. The layer is CI system-agnostic and contains a number of Helper scripts that can generate build configurations from
@@ -124,18 +123,21 @@ thefollowing types of tests:
ensure we have no single point of failure and can ensure the different distros work effectively.
-- *eSDK Testing:* Image tests initiated through the following command: - $ bitbake image -c testsdkext The tests utilize the ``testsdkext`` - class and the ``do_testsdkext`` task. +- *eSDK Testing:* Image tests initiated through the following command::
does that work? I've always done ": ::" to keep the semi column, and I thought the :: needed a space before.
It’s shorthand, and works so long as the sentence ends with a colon (:)
+
+ $ bitbake image -c testsdkext
Indentation is not typical here. you need 3 space starting from the previous indentation level.
I missed this (and the following one) – good catch.
+
+ The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task.
this sentence above is no in the code-block, it should not be at the same indentation level as the code block.
- *Feature Testing:* Various scenario-based tests are run through the - `OpenEmbedded - Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__ - (oe-selftest). We test oe-selftest on each of the main distrubutions + :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions
we support.
-- *Image Testing:* Image tests initiated through the following command: - $ bitbake image -c testimage The tests utilize the +- *Image Testing:* Image tests initiated through the following command::
+
+ $ bitbake image -c testimage
+
+ The tests utilize the
Need to fix indentation here
Done
:ref:testimage* <ref-classes-testimage*> classes and the :ref:ref-tasks-testimage task.
@@ -147,8 +149,8 @@ thefollowing types of tests:
- *Package Testing:* A Package Test (ptest) runs tests against packages built by the OpenEmbedded build system on the target machine. See the
- "`Testing Packages With - ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section + :ref:`Testing Packages With + ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section
in the Yocto Project Development Tasks Manual and the ":yocto_wiki:Ptest </wiki/Ptest>" Wiki page for more information on Ptest.
@@ -159,8 +161,8 @@ thefollowing types of tests:
the ``do_testsdk`` task.
- *Unit Testing:* Unit tests on various components of the system run - through ``oe-selftest`` and - ```bitbake-selftest`` <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__. + through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and + :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`.
the link is on bitbake-selftest, not oe-selftest (you added it).
They both point to the same location, so to be consistent, I added a link for both, and rearranged them to be in alphabetical order.
- *Automatic Upgrade Helper:* This target tests whether new versions of software are available and whether we can automatically upgrade to
@@ -174,36 +176,43 @@ How Tests Map to Areas of Code
Tests map into the codebase as follows:
-- *bitbake-selftest*:
+- *bitbake-selftest:*
These tests are self-contained and test BitBake as well as its APIs, which include the fetchers. The tests are located in ``bitbake/lib/*/tests``.
- From within the BitBake repository, run the following: $ - bitbake-selftest + From within the BitBake repository, run the following::
+
+ $ bitbake-selftest
indentation and use of :: are not right (please check all other occurrences)
Per our discussion, I checked indentation only. It is three spaces, but I see a slight indentation in the output. This might need a CSS tweak.
To skip tests that access the Internet, use the ``BB_SKIP_NETTEST``
- variable when running "bitbake-selftest" as follows: $ - BB_SKIP_NETTEST=yes bitbake-selftest + variable when running "bitbake-selftest" as follows::
+
+ $ BB_SKIP_NETTEST=yes bitbake-selftest
The default output is quiet and just prints a summary of what was
- run. To see more information, there is a verbose option:$ - bitbake-selftest -v + run. To see more information, there is a verbose option::
+
+ $ bitbake-selftest -v
Use this option when you wish to skip tests that access the network, which are mostly necessary to test the fetcher modules. To specify individual test modules to run, append the test module name to the "bitbake-selftest" command. For example, to specify the tests for the
- bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can - also specify individual tests by defining the full name and module - plus the class path of the test, for example: $ bitbake-selftest - bb.tests.data.TestOverrides.test_one_override + bb.data.module, run::
+
+ $ bitbake-selftest bb.test.data.module
+
+ You can also specify individual tests by defining the full name and module
+ plus the class path of the test, for example::
+
+ $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
The tests are based on `Python
unittest <https://docs.python.org/3/library/unittest.html>`__.
-- *oe-selftest*:
+- *oe-selftest:*
- These tests use OE to test the workflows, which include testing specific features, behaviors of tasks, and API unit tests.
@@ -219,16 +228,25 @@ Tests map into the codebase as follows:
- The code for the tests resides in ``meta/lib/oeqa/selftest/cases/``.
- - To run all the tests, enter the following command: $ oe-selftest - -a + - To run all the tests, enter the following command::
+
+ $ oe-selftest -a
- To run a specific test, use the following command form where - testname is the name of the specific test: $ oe-selftest -r - testname For example, the following command would run the tinfoil - getVar API test:$ oe-selftest -r - tinfoil.TinfoilTests.test_getvarIt is also possible to run a set + testname is the name of the specific test::
+
+ $ oe-selftest -r <testname>
+
+ For example, the following command would run the tinfoil
+ getVar API test::
+
+ $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
+
+ It is also possible to run a set
of tests. For example the following command will run all of the
- tinfoil tests:$ oe-selftest -r tinfoil
+ tinfoil tests::
+
+ $ oe-selftest -r tinfoil
- *testimage:*
@@ -240,10 +258,13 @@ Tests map into the codebase as follows:
- You need to set the
:term:IMAGE_CLASSES
- variable as follows: IMAGE_CLASSES += "testimage"
+ variable as follows::
+
+ IMAGE_CLASSES += "testimage"
- - Run the tests using the following command form: $ bitbake image -c - testimage + - Run the tests using the following command form::
+
+ $ bitbake image -c testimage
- *testsdk:*
@@ -252,8 +273,9 @@ Tests map into the codebase as follows:
- The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
- - Run the test using the following command form: $ bitbake image -c - testsdk + - Run the test using the following command form::
+
+ $ bitbake image -c testsdk
- *testsdk_ext:*
@@ -262,8 +284,9 @@ Tests map into the codebase as follows:
- The code for these tests resides in ``meta/lib/oeqa/esdk``.
- - To run the tests, use the following command form: $ bitbake image - -c testsdkext + - To run the tests, use the following command form::
+
+ $ bitbake image -c testsdkext
- *oe-build-perf-test:*
@@ -272,8 +295,11 @@ Tests map into the codebase as follows:
- The code for these tests resides in ``meta/lib/oeqa/buildperf``.
- - To run the tests, use the following command form: $ - oe-build-perf-test optionsThe command takes a number of options, + - To run the tests, use the following command form::
+
+ $ oe-build-perf-test <options>
+
+ The command takes a number of options,
such as where to place the test results. The Autobuilder Helper Scripts include the ``build-perf-test-wrapper`` script with examples of how to use the oe-build-perf-test from the command
@@ -308,14 +334,20 @@ directory.
``bitbake-selftest``
--------------------
-A simple test example from ``lib/bb/tests/data.py`` is: class -DataExpansions(unittest.TestCase): def setUp(self): self.d = -bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] = -"value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def -test_one_var(self): val = self.d.expand("${foo}") -self.assertEqual(str(val), "value_of_foo")
+A simple test example from ``lib/bb/tests/data.py`` is::
-In this example, a ```DataExpansions`` <>`__ class of tests is created,
+ class DataExpansions(unittest.TestCase):
+ def setUp(self):
+ self.d = bb.data.init() + self.d["foo"] = "value_of_foo" + self.d["bar"] = "value_of_bar" + self.d["value_of_foo"] = "value_of_'value_of_foo'"
+
+ def test_one_var(self):
+ val = self.d.expand("${foo}")
+ self.assertEqual(str(val), "value_of_foo")
+
+In this example, a ``DataExpansions`` class of tests is created,
derived from standard python unittest. The class has a common ``setUp`` function which is shared by all the tests in the class. A simple test is
then added to test that when a variable is expanded, the correct value
@@ -334,14 +366,15 @@ These tests are more complex due to the setup required behind the scenes
for full builds. Rather than directly using Python's unittest, the code
wraps most of the standard objects. The tests can be simple, such as
testing a command from within the OE build environment using the
-following example:class BitbakeLayers(OESelftestTestCase): def -test_bitbakelayers_showcrossdepends(self): result =
-runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in -result.output, msg = "No dependencies were shown. bitbake-layers -show-cross-depends output: %s"% result.output)
+following example::
+
+ class BitbakeLayers(OESelftestTestCase):
+ def test_bitbakelayers_showcrossdepends(self):
+ result = runCmd('bitbake-layers show-cross-depends')
+ self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
-creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived +creates a testcase from the ``OESelftestTestCase`` class, derived
from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
and checks the output to ensure it contains something we know should be here.
@@ -367,7 +400,7 @@ so tests within a given test class should always run in the same build,
while tests in different classes or modules may be split into different builds. There is no data store available for these tests since the tests launch the ``bitbake`` command and exist outside of its context. As a
-result, common bitbake library functions (bb.*) are also unavailable. +result, common bitbake library functions (bb.\*) are also unavailable.
.. _testimage-example:
@@ -378,14 +411,18 @@ These tests are run once an image is up and running, either on target
hardware or under QEMU. As a result, they are assumed to be running in a
target image environment, as opposed to a host build environment. A
simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
-the following:class PythonTest(OERuntimeTestCase):
-@OETestDepends(['ssh.SSHTest.test_ssh'])<mailto:-@OETestDepends(['ssh.SSHTest.test_ssh'])> @OEHasPackage(['python3-core']) -def test_python3(self): cmd = "python3 -c \\"import<file://%22import> codecs; -print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output =
-self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output
-self.assertEqual(status, 0, msg=msg)
-
-In this example, the ```OERuntimeTestCase`` <>`__ class wraps
+the following::
+
+ class PythonTest(OERuntimeTestCase):
+ @OETestDepends(['ssh.SSHTest.test_ssh'])
+ @OEHasPackage(['python3-core'])
+ def test_python3(self):
+ cmd = "python3 -c \\"import<file://%22import> codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
+ status, output = self.target.run(cmd)
+ msg = 'Exit status was not 0. Output: %s' % output
+ self.assertEqual(status, 0, msg=msg)+
+In this example, the ``OERuntimeTestCase`` class wraps
``unittest.TestCase``. Within the test, ``self.target`` represents the target system, where commands can be run on it using the ``run()`` method.
@@ -402,19 +439,30 @@ the image.
These tests are run against built extensible SDKs (eSDKs). The tests can assume that the eSDK environment has already been setup. An example from
-``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class -DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls):
-myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst = -os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src, -cls.myapp_dst) subprocess.check_output(['git', 'init', '.'], -cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'], -cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m', -"'test commit'"], cwd=cls.myapp_dst) @classmethod def -tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def -\_test_devtool_build(self, directory): self._run('devtool add myapp %s' -% directory) try: self._run('devtool build myapp') finally:
-self._run('devtool reset myapp') def test_devtool_build_make(self):
-self._test_devtool_build(self.myapp_dst)In this example, the ``devtool`` +``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
+
+ class DevtoolTest(OESDKExtTestCase):
+ @classmethod def setUpClass(cls):
+ myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") + cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") + shutil.copytree(myapp_src, cls.myapp_dst) + subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) + subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) + subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.myapp_dst)
+ def _test_devtool_build(self, directory):
+ self._run('devtool add myapp %s' % directory)
+ try:
+ self._run('devtool build myapp')
+ finally:
+ self._run('devtool reset myapp')+ def test_devtool_build_make(self):
+ self._test_devtool_build(self.myapp_dst)
+
+In this example, the ``devtool``
command is tested to see whether a sample application can be built with the ``devtool build`` command within the eSDK.
@@ -426,14 +474,20 @@ the ``devtool build`` command within the eSDK.
These tests are run against built SDKs. The tests can assume that an SDK has already been extracted and its environment file has been sourced. A simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
-following:class Python3Test(OESDKTestCase): def setUp(self): if not -(self.tc.hasHostPackage("nativesdk-python3-core") or -self.tc.hasHostPackage("python3-core-native")): raise -unittest.SkipTest("No python3 package in the SDK") def -test_python3(self): cmd = "python3 -c \\"import<file://%22import> codecs; -print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd) -self.assertEqual(output, "Hello, world\n")In this example, if -nativesdk-python3-core has been installed into the SDK, the code runs +following::
+
+ class Python3Test(OESDKTestCase):
+ def setUp(self):
+ if not (self.tc.hasHostPackage("nativesdk-python3-core") or
+ self.tc.hasHostPackage("python3-core-native")):
+ raise unittest.SkipTest("No python3 package in the SDK")
+
+ def test_python3(self):
+ cmd = "python3 -c \\"import<file://%22import> codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
+ output = self._run(cmd)
+ self.assertEqual(output, "Hello, world\n")+
+In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
the python3 interpreter with a basic command to check it is working correctly. The test would only run if python3 is installed in the SDK.
@@ -444,17 +498,25 @@ correctly. The test would only run if python3 is installed in the SDK.
The performance tests usually measure how long operations take and the resource utilisation as that happens. An example from
-``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:class -Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake --p)""" # Drop all caches and parse self.rm_cache() -oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) -self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no -caches)') # Drop tmp/cache -oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) -self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no -tmp/cache)') # Parse with fully cached data -self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p -(cached)')This example shows how three specific parsing timings are +``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
+
+ class Test3(BuildPerfTestCase):
+ def test3(self):
+ """Bitbake parsing (bitbake -p)""" + # Drop all caches and parse + self.rm_cache() + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) + self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', + 'bitbake -p (no caches)') + # Drop tmp/cache + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) + self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
+ 'bitbake -p (no tmp/cache)')
+ # Parse with fully cached data
+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
+ 'bitbake -p (cached)')
+
+This example shows how three specific parsing timings are
measured, with and without various caches, to show how BitBake’s parsing performance trends over time.
documentation/test-manual/test-manual-understand-autobuilder.rst +37 -21
67 lines not on this page · raw message
why? let's stick to the original .xml file, we can fix the doc later.
It’s just more grammatically correct. I feel like I should have caught this the first time around ☺
yet allows duplication and repetition to be kept to a minimum.
The different build targets are designed to allow for parallelisation,
@@ -70,7 +86,7 @@ roughly consist of:
This cleans out any previous build. Old builds are left around to allow easier debugging of failed builds. For additional information,
- see ```clobberdir`` <#test-clobberdir>`__.
+ see :ref:test-clobberdir.
It works. However we are now using autosectionlabel which creates sections labels for headings automatically. So I would like to get rid of all custom labels, as much as possible. the following is preferred:
:ref:test-manual/test-manual-understand-autobuilder:clobberdir>
I’ve updated all these instances
I know we've left some links like that in other manuals. So it's not strictly required to change it now.
2. *Obtain yocto-autobuilder-helper*
@@ -94,7 +110,7 @@ roughly consist of:
from the ``layerinfo.json`` file to help understand the configuration. It will also use a local cache of repositories to speed up the clone checkouts. For additional information, see
- Autobuilder Clone Cache <#test-autobuilder-clone-cache>__.
+ :ref:test-autobuilder-clone-cache.
same here (and probably in other places)
I updated all of them
This step has two possible modes of operation. If the build is part of a parent build, its possible that all the repositories needed may
@@ -137,8 +153,7 @@ special script that moves files to a special location, rather than
deleting them. Files in this location are deleted by an ``rm`` command, which is run under ``ionice -c 3``. For example, the deletion only happens when there is idle IO capacity on the Worker. The Autobuilder
-Worker Janitor runs this deletion. See `Autobuilder Worker
-Janitor <#test-autobuilder-worker-janitor>`__.
+Worker Janitor runs this deletion. See :ref:test-autobuilder-worker-janitor.
.. _test-autobuilder-clone-cache:
@@ -150,8 +165,8 @@ on the Autobuilder. We therefore have a stash of commonly used
repositories pre-cloned on the Workers. Data is fetched from these during clones first, then "topped up" with later revisions from any upstream when necesary. The cache is maintained by the Autobuilder
-Worker Janitor. See `Autobuilder Worker
-Janitor <#test-autobuilder-worker-janitor>`__.
+Worker Janitor. See :ref:`Autobuilder Worker
+Janitor`.
.. _test-autobuilder-worker-janitor:
@@ -159,8 +174,8 @@ Autobuilder Worker Janitor
--------------------------
This is a process running on each Worker that performs two basic
-operations, including background file deletion at IO idle (see Target -Execution: clobberdir <#test-list-tgt-exec-clobberdir>__) and +operations, including background file deletion at IO idle (see Target +Execution: clobberdir :ref:test-list-tgt-exec-clobberdir) and
the link and text look wrong here. the link is not on the 'text'
It was to an old anchor link for the step that did not convert. I updated the link to point to the main section.
maintainenance of a cache of cloned repositories to improve the speed the system can checkout repositories.
@@ -216,9 +231,6 @@ are general setup steps that are run once and include:
For each step that is configured in ``config.json``, it will perform the following:
-## WRITER's question: What does "logging in as stepXa" and others refer -to below? ##
you can keep comment with the ".. xxx" syntax.
It was no longer necessary – should I add it back in?
-
1. Add any layers that are specified using the ``bitbake-layers add-layer`` command (logging as stepXa)
@@ -235,9 +247,8 @@ to below? ##
6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the BitBake build environment (logging as stepXd)
-7. Remove any layers added in `step
- 1 <#test-run-config-add-layers-step>`__ using the
- ``bitbake-layers remove-layer`` command (logging as stepXa)
+7. Remove any layers added in step
+ 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
you removed a link here, no?
Yes – it pointed to a step, so I just used the step number instead, as it’s part of the same procedure.
Once the execution steps above complete, ``run-config`` executes a set of post-build steps, including:
@@ -252,7 +263,7 @@ of post-build steps, including:
generated to the remote server.
4. Cleanup the build directory using
- ```clobberdir`` <#test-clobberdir>`__ if the build was successful,
+ :ref:test-clobberdir if the build was successful,
else rename it to “build-renamed” for potential future debugging.
.. _test-deploying-yp-autobuilder:
@@ -279,11 +290,18 @@ The standard ``config.json`` minimally attempts to allow substitution of
the paths. The Helper script repository includes a
``local-example.json`` file to show how you could override these from a separate configuration file. Pass the following into the environment of
-the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As
-another example, you could also pass the following into the
-environment:$ ABHELPER_JSON="config.json /some/location/local.json"One
-issue users often run into is validation of the ``config.json`` files. A
+the Autobuilder::
+
+ $ ABHELPER_JSON="config.json local-example.json"
+
+As another example, you could also pass the following into the
+environment::
+
+ $ ABHELPER_JSON="config.json /some/location/local.json"
+
+One issue users often run into is validation of the ``config.json`` files. A
tip for minimizing issues from invalid json files is to use a Git ``pre-commit-hook.sh`` script to verify the JSON file before committing
-it. Create a symbolic link as follows:$ ln -s -../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
+it. Create a symbolic link as follows::
+
+ $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
--
2.17.1