Applied Richard's edits to code formatting and logfiles
PATCH 1/1 Applied Richard's edits to code formatting and logfiles
Signed-off-by Mark Morton <[email protected]>
--- .../test-manual/test-manual-intro.xml | 251 +++++++++--------- .../test-manual-understand-autobuilder.xml | 81 +++--- 2 files changed, 164 insertions(+), 168 deletions(-)
documentation/test-manual/test-manual-intro.xml +123 -128
@@ -1,14 +1,10 @@ <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > -<!--SPDX-License-Identifier: CC-BY-2.0-UK--> - -<chapter id='test-manual-intro'> - -<title>The Yocto Project Test Environment Manual</title> - <section id='test-welcome'> +<chapter id="test-manual-intro"> + <title>The Yocto Project Test Environment Manual</title> + <section id="test-welcome"> <title>Welcome</title> - <para> Welcome to the Yocto Project Test Environment Manual! This manual is a work in progress. The manual contains information about the testing environment used by the Yocto Project to make sure each major and minor release works as intended. All the
@@ -18,7 +14,6 @@ organizations can leverage off the process and testing environment used by the Yocto Project to create their own automated, production test environment, building upon the foundations from the project core. </para> - <para> Currently, the Yocto Project Test Environment Manual has no projected release date. This manual is a work-in-progress and is being initially loaded with information from the <ulink url="">README</ulink> files and notes from key engineers: <itemizedlist>
@@ -51,23 +46,19 @@ </itemizedlist> </para> </section> - - <section id='test-yocto-project-autobuilder-overview'> + <section id="test-yocto-project-autobuilder-overview"> <title>Yocto Project Autobuilder Overview</title> - <para>The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and procedures used by the Yocto Project to test released software across supported hardware in an automated and regular fashion. Basically, during the development of a Yocto Project release, the Autobuilder tests if things work. The Autobuilder builds all test targets and runs all the tests. </para> - <para>The Yocto Project uses now uses standard upstream <ulink url="https://docs.buildbot.net/0.9.15.post1/">Buildbot</ulink> (version 9) to drive its integration and testing. Buildbot Nine has a plug-in interface that the Yocto Project customizes using code from the <filename>yocto-autobuilder2</filename> repository, adding its own console UI plugin. The resulting UI plug-in allows you to visualize builds in a way suited to the project's needs.</para> - <para>A <filename>helper</filename> layer provides configuration and job management through scripts found in the <filename>yocto-autobuilder-helper</filename> repository. The <filename>helper</filename> layer contains the bulk of the build configuration
@@ -81,17 +72,14 @@ instead of Buildbot. </para> </note> </para> - <para> The following figure shows the Yocto Project Autobuilder stack with a topology that includes a controller and a cluster of workers: <imagedata fileref="figures/ab-test-cluster.png" width="4.6in" depth="4.35in" align="center" scalefit="1"/> </para> </section> - - <section id='test-project-tests'> + <section id="test-project-tests"> <title>Yocto Project Tests - Types of Testing Overview</title> - <para>The Autobuilder tests different elements of the project by using thefollowing types of tests: <itemizedlist> <listitem>
@@ -206,14 +194,11 @@ </itemizedlist> </para> </section> - - <section id='test-test-mapping'> + <section id="test-test-mapping"> <title>How Tests Map to Areas of Code</title> - - <para> - Tests map into the codebase as follows: - <itemizedlist> - <listitem><para> + <para> Tests map into the codebase as follows: <itemizedlist> + <listitem> + <para> <emphasis>bitbake-selftest</emphasis>: </para> <para>These tests are self-contained and test BitBake as well as its APIs, which include the fetchers. The tests are located in
@@ -248,8 +233,10 @@ </literallayout></para> <para>The tests are based on <ulink url="https://docs.python.org/3/library/unittest.html">Python - unittest</ulink>. </para></listitem> - <listitem><para> + unittest</ulink>. </para> + </listitem> + <listitem> + <para> <emphasis>oe-selftest</emphasis>: <itemizedlist> <listitem> <para>These tests use OE to test the workflows, which include
@@ -296,63 +283,98 @@ </literallayout></para> </listitem> </itemizedlist> - </para></listitem> - <listitem><para> - <emphasis>testimage:</emphasis> - <itemizedlist> - <listitem><para> - These tests build an image, boot it, and run tests - against the image's content. - </para></listitem> - <listitem><para> The code for these tests resides in <filename>meta/lib/oeqa/runtime/cases/</filename>. </para></listitem> - <listitem><para> - You need to set the - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_CLASSES'><filename>IMAGE_CLASSES</filename></ulink> - variable as follows: - <literallayout class='monospaced'> + </para> + </listitem> + <listitem> + <para> + <emphasis>testimage:</emphasis> + <itemizedlist> + <listitem> + <para> These tests build an image, boot it, and run tests against + the image's content. </para> + </listitem> + <listitem> + <para> The code for these tests resides in + <filename>meta/lib/oeqa/runtime/cases/</filename>. </para> + </listitem> + <listitem> + <para> You need to set the <ulink + url="&YOCTO_DOCS_REF_URL;#var-IMAGE_CLASSES" + ><filename>IMAGE_CLASSES</filename></ulink> variable as + follows: + <literallayout class="monospaced"> IMAGE_CLASSES += "testimage" </literallayout> - </para></listitem> - <listitem><para> - Run the tests using the following command form: - <literallayout class='monospaced'> + </para> + </listitem> + <listitem> + <para> Run the tests using the following command form: + <literallayout class="monospaced"> $ bitbake <replaceable>image</replaceable> -c testimage </literallayout> - </para></listitem> - </itemizedlist> - </para></listitem> - <listitem><para> - <emphasis>testsdk:</emphasis> - <itemizedlist> - <listitem><para>These tests build an SDK, install it, and then run tests against that SDK. </para></listitem> - <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/sdk/cases/</filename>. </para></listitem> - <listitem><para>Run the test using the following command form: + </para> + </listitem> + </itemizedlist> + </para> + </listitem> + <listitem> + <para> + <emphasis>testsdk:</emphasis> + <itemizedlist> + <listitem> + <para>These tests build an SDK, install it, and then run tests + against that SDK. </para> + </listitem> + <listitem> + <para>The code for these tests resides in + <filename>meta/lib/oeqa/sdk/cases/</filename>. </para> + </listitem> + <listitem> + <para>Run the test using the following command form: <literallayout class="monospaced"> $ bitbake <replaceable>image</replaceable> -c testsdk </literallayout> - </para></listitem> - </itemizedlist> - </para></listitem> - <listitem><para> - <emphasis>testsdk_ext:</emphasis> - <itemizedlist> - <listitem><para>These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. </para></listitem> - <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/esdk</filename>. </para></listitem> - <listitem><para>To run the tests, use the following command form: + </para> + </listitem> + </itemizedlist> + </para> + </listitem> + <listitem> + <para> + <emphasis>testsdk_ext:</emphasis> + <itemizedlist> + <listitem> + <para>These tests build an extended SDK (eSDK), install that eSDK, + and run tests against the eSDK. </para> + </listitem> + <listitem> + <para>The code for these tests resides in + <filename>meta/lib/oeqa/esdk</filename>. </para> + </listitem> + <listitem> + <para>To run the tests, use the following command form: <literallayout class="monospaced"> $ bitbake <replaceable>image</replaceable> -c testsdkext </literallayout> - </para></listitem> - </itemizedlist> - </para></listitem> - - - <listitem><para> - <emphasis>oe-build-perf-test:</emphasis> - <itemizedlist> - <listitem><para>These tests run through commonly used usage scenarios and measure the performance times. </para></listitem> - <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/buildperf</filename>. </para></listitem> - <listitem><para>To run the tests, use the following command form: + </para> + </listitem> + </itemizedlist> + </para> + </listitem> + <listitem> + <para> + <emphasis>oe-build-perf-test:</emphasis> + <itemizedlist> + <listitem> + <para>These tests run through commonly used usage scenarios and + measure the performance times. </para> + </listitem> + <listitem> + <para>The code for these tests resides in + <filename>meta/lib/oeqa/buildperf</filename>. </para> + </listitem> + <listitem> + <para>To run the tests, use the following command form: <literallayout class="monospaced"> $ oe-build-perf-test <replaceable>options</replaceable> </literallayout>The
@@ -368,34 +390,28 @@ performance data. For examples, see <link linkend="" >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html</link> and <link linkend="" - >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt</link>.</para></listitem> + >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt</link>.</para> + </listitem> <listitem> <para>The tests are contained in <filename>lib/oeqa/buildperf/test_basic.py</filename>.</para> </listitem> - </itemizedlist> - </para></listitem> - - - - + </itemizedlist> + </para> + </listitem> </itemizedlist> </para> </section> - - <section id='test-examples'> + <section id="test-examples"> <title>Test Examples</title> - <para>This section provides example tests for each of the tests listed in the <link linkend="test-test-mapping">How Tests Map to Areas of Code</link> section. </para> <para>For oeqa tests, testcases for each area reside in the main test directory at <filename>meta/lib/oeqa/selftest/cases</filename> directory.</para> <para>For oe-selftest. bitbake testcases reside in the <filename>lib/bb/tests/</filename> directory. </para> - - <section id='bitbake-selftest-example'> + <section id="bitbake-selftest-example"> <title><filename>bitbake-selftest</filename></title> - <para>A simple test example from <filename>lib/bb/tests/data.py</filename> is: <literallayout class="monospaced"> class DataExpansions(unittest.TestCase):
@@ -419,10 +435,8 @@ unittest documentation for additional information on writing these tests at: <link linkend="">https://docs.python.org/3/library/unittest.html</link>.</para> </section> - - <section id='oe-selftest-example'> + <section id="oe-selftest-example"> <title><filename>oe-selftest</filename></title> - <para>These tests are more complex due to the setup required behind the scenes for full builds. Rather than directly using Python's unittest, the code wraps most of the standard objects. The tests can be simple, such as testing a command from within the
@@ -431,9 +445,7 @@ class BitbakeLayers(OESelftestTestCase): def test_bitbakelayers_showcrossdepends(self): result = runCmd('bitbake-layers show-cross-depends') - self.assertTrue('aspell' in result.output, msg = "No dependencies - were shown. bitbake-layers show-cross-depends output: - %s"% result.output) + self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output) </literallayout></para> <para>This example, taken from <filename>meta/lib/oeqa/selftest/cases/bblayers.py</filename>, creates a
@@ -468,10 +480,8 @@ <filename>bitbake</filename> command and exist outside of its context. As a result, common bitbake library functions (bb.*) are also unavailable.</para> </section> - - <section id='testimage-example'> + <section id="testimage-example"> <title><filename>testimage</filename></title> - <para>These tests are run once an image is up and running, either on target hardware or under QEMU. As a result, they are assumed to be running in a target image environment, as opposed to a host build environment. A simple example from
@@ -481,8 +491,7 @@ @OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core']) def test_python3(self): - cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, - jbeyq', 'rot13'))\"" + cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb', 'jbeyq', 'rot13'))\"" status, output = self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output self.assertEqual(status, 0, msg=msg)
@@ -496,10 +505,8 @@ decorators. For example, the test in this example would only make sense if python3-core is installed in the image.</para> </section> - - <section id='testsdk_ext-example'> + <section id="testsdk_ext-example"> <title><filename>testsdk_ext</filename></title> - <para>These tests are run against built extensible SDKs (eSDKs). The tests can assume that the eSDK environment has already been setup. An example from <filename>meta/lib/oeqa/sdk/cases/devtool.py</filename> contains the
@@ -507,33 +514,31 @@ class DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls): - myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") - cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") - shutil.copytree(myapp_src, cls.myapp_dst) - subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) - subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) - subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) + myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") + cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") + shutil.copytree(myapp_src, cls.myapp_dst) + subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) + subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) + subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) @classmethod def tearDownClass(cls): - shutil.rmtree(cls.myapp_dst) + shutil.rmtree(cls.myapp_dst) def _test_devtool_build(self, directory): - self._run('devtool add myapp %s' % directory) - try: - self._run('devtool build myapp') - finally: - self._run('devtool reset myapp') + self._run('devtool add myapp %s' % directory) + try: + self._run('devtool build myapp') + finally: + self._run('devtool reset myapp') def test_devtool_build_make(self): - self._test_devtool_build(self.myapp_dst) + self._test_devtool_build(self.myapp_dst) </literallayout>In this example, the <filename>devtool</filename> command is tested to see whether a sample application can be built with the <filename>devtool build</filename> command within the eSDK.</para> </section> - - <section id='testsdk-example'> + <section id="testsdk-example"> <title><filename>testsdk</filename></title> - <para>These tests are run against built SDKs. The tests can assume that an SDK has already been extracted and its environment file has been sourced. A simple example from <filename>meta/lib/oeqa/sdk/cases/python2.py</filename> contains the
@@ -553,10 +558,8 @@ runs the python3 interpreter with a basic command to check it is working correctly. The test would only run if python3 is installed in the SDK.</para> </section> - - <section id='oe-build-perf-test-example'> + <section id="oe-build-perf-test-example"> <title><filename>oe-build-perf-test</filename></title> - <para>The performance tests usually measure how long operations take and the resource utilisation as that happens. An example from <filename>meta/lib/oeqa/buildperf/test_basic.py</filename> contains the
@@ -582,7 +585,7 @@ various caches, to show how BitBake’s parsing performance trends over time.</para> </section> </section> - <section id='test-writing-considerations'> + <section id="test-writing-considerations"> <title>Considerations When Writing Tests</title> <para>When writing good tests, there are several things to keep in mind. Since things running on the Autobuilder are accessed concurrently by multiple workers, consider the
@@ -610,14 +613,6 @@ metadate from changes, set up temporary copies of that data first.</para> </formalpara> </section> - - - - - - - - </chapter> <!-- vim: expandtab tw=80 ts=4
documentation/test-manual/test-manual-understand-autobuilder.xml +41 -40
183 lines not on this page · raw message
Signature · 1 line
2.17.1