Received: from mail5.wrs.com (mail5.wrs.com [192.103.53.11]) by mx.groups.io with SMTP id smtpd.web11.305.1592602546139963088 for ; Fri, 19 Jun 2020 14:35:46 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: windriver.com, ip: 192.103.53.11, mailfrom: mark.morton@windriver.com) Received: from ALA-HCA.corp.ad.wrs.com (ala-hca.corp.ad.wrs.com [147.11.189.40]) by mail5.wrs.com (8.15.2/8.15.2) with ESMTPS id 05JLZEB0002369 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Fri, 19 Jun 2020 14:35:35 -0700 Received: from san-mmorton-lx1.corp.ad.wrs.com (147.11.4.21) by ALA-HCA.corp.ad.wrs.com (147.11.189.40) with Microsoft SMTP Server id 14.3.487.0; Fri, 19 Jun 2020 14:35:08 -0700 From: "Mark Morton" To: Subject: [PATCH] Applied Richard's edits to code formatting and logfiles Date: Fri, 19 Jun 2020 14:30:35 -0700 Message-ID: <20200619213035.3919-1-mark.morton@windriver.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-MIME-Autoconverted: from 8bit to quoted-printable by mail5.wrs.com id 05JLZEB0002369 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Signed-off-by: Mark Morton --- .../test-manual/test-manual-intro.xml | 251 +++++++++--------- .../test-manual-understand-autobuilder.xml | 81 +++--- 2 files changed, 164 insertions(+), 168 deletions(-) diff --git a/documentation/test-manual/test-manual-intro.xml b/documentat= ion/test-manual/test-manual-intro.xml index 8e2c7cd87..a4a7f9062 100644 --- a/documentation/test-manual/test-manual-intro.xml +++ b/documentation/test-manual/test-manual-intro.xml @@ -1,14 +1,10 @@ %poky; ] > - - - - -The Yocto Project Test Environment Manual -
+ + The Yocto Project Test Environment Manual +
Welcome - Welcome to the Yocto Project Test Environment Manual! Thi= s manual is a work in progress. The manual contains information about the testing = environment used by the Yocto Project to make sure each major and minor release work= s as intended. All the @@ -18,7 +14,6 @@ organizations can leverage off the process and testing envir= onment used by the Yocto Project to create their own automated, production test envir= onment, building upon the foundations from the project core. - Currently, the Yocto Project Test Environment Manual has = no projected release date. This manual is a work-in-progress and is being initially loa= ded with information from the README files and notes from key = engineers: @@ -51,23 +46,19 @@
- -
+
Yocto Project Autobuilder Overview - The Yocto Project Autobuilder collectively refers to the s= oftware, tools, scripts, and procedures used by the Yocto Project to test released softwa= re across supported hardware in an automated and regular fashion. Basically, during the d= evelopment of a Yocto Project release, the Autobuilder tests if things work. The A= utobuilder builds all test targets and runs all the tests. - The Yocto Project uses now uses standard upstream Buildbot= (version 9) to drive its integration and testing. Buildbot Nine has a plug-in int= erface that the Yocto Project customizes using code from the yocto-autob= uilder2 repository, adding its own console UI plugin. The resulting = UI plug-in allows you to visualize builds in a way suited to the project's needs. - A helper layer provides configuration= and job management through scripts found in the yocto-autobuilder-helper repository. The helper layer contains the bulk of t= he build configuration @@ -81,17 +72,14 @@ instead of Buildbot. - The following figure shows the Yocto Project Autobuilder = stack with a topology that includes a controller and a cluster of workers:
- -
+
Yocto Project Tests - Types of Testing Overview - The Autobuilder tests different elements of the project by= using thefollowing types of tests: @@ -206,14 +194,11 @@
- -
+
How Tests Map to Areas of Code - - - Tests map into the codebase as follows: - - + Tests map into the codebase as follows: + + bitbake-selftest: These tests are self-contained and test BitBak= e as well as its APIs, which include the fetchers. The tests are located in @@ -248,8 +233,10 @@ The tests are based on Python - unittest. - + unittest. + + + oe-selftest: These tests use OE to test the wor= kflows, which include @@ -296,63 +283,98 @@ - - - testimage: - - - These tests build an image, boot it, and run= tests - against the image's content. - - The code for these tests reside= s in meta/lib/oeqa/runtime/cases/. - - You need to set the - IMAGE_CLASSES - variable as follows: - + + + + + testimage: + + + These tests build an image, boot = it, and run tests against + the image's content. + + + The code for these tests resides = in + meta/lib/oeqa/runtime/= cases/. + + + You need to set the IMAGE_CLASSES variable as + follows: + IMAGE_CLASSES +=3D "testimage" - - - Run the tests using the following command fo= rm: - + + + + Run the tests using the following= command form: + $ bitbake image -c testimage - - - - - testsdk: - - These tests build an SDK, instal= l it, and then run tests against that SDK. - The code for these tests resides= in meta/lib/oeqa/sdk/cases/. - Run the test using the following= command form: + + + + + + + + testsdk: + + + These tests build an SDK, install = it, and then run tests + against that SDK. + + + The code for these tests resides i= n + meta/lib/oeqa/sdk/case= s/. + + + Run the test using the following c= ommand form: $ bitbake image -c testsdk - - - - - testsdk_ext: - - These tests build an extended SD= K (eSDK), install that eSDK, and run tests against the eSDK. - The code for these tests resides= in meta/lib/oeqa/esdk. - To run the tests, use the follow= ing command form: + + + + + + + + testsdk_ext: + + + These tests build an extended SDK = (eSDK), install that eSDK, + and run tests against the eSDK. + + + The code for these tests resides i= n + meta/lib/oeqa/esdk. + + + To run the tests, use the followin= g command form: $ bitbake image -c testsdkext - - - - - - - oe-build-perf-test: - - These tests run through commonly= used usage scenarios and measure the performance times. - The code for these tests resides= in meta/lib/oeqa/buildperf. - To run the tests, use the follow= ing command form: + + + + + + + + oe-build-perf-test: + + + These tests run through commonly u= sed usage scenarios and + measure the performance times. + + + The code for these tests resides i= n + meta/lib/oeqa/buildper= f. + + + To run the tests, use the followin= g command form: $ oe-build-perf-test options The @@ -368,34 +390,28 @@ performance data. For examples, see = http://downloads.yoctoproject.o= rg/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yo= ctoproject.org_warrior_20190414204758_0e39202.html and http://downloads.yoctoproject.o= rg/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yo= ctoproject.org_warrior_20190414204758_0e39202.txt. + >http://downloads.yoctoproject.o= rg/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yo= ctoproject.org_warrior_20190414204758_0e39202.txt. + The tests are contained in lib/oeqa/buildperf/tes= t_basic.py. - - - - - - + + +
- -
+
Test Examples - This section provides example tests for each of the tests = listed in the How Tests Map to Areas of = Code section. For oeqa tests, testcases for each area reside in the main= test directory at meta/lib/oeqa/selftest/cases direct= ory. For oe-selftest. bitbake testcases reside in the lib/bb/tests/ directory. - -
+
<filename>bitbake-selftest</filename> - A simple test example from lib/bb/tests/data= .py is: class DataExpansions(unittest.TestCase): @@ -419,10 +435,8 @@ unittest documentation for additional information on wri= ting these tests at: https://docs.python.org/3/library/unitt= est.html.
- -
+
<filename>oe-selftest</filename> - These tests are more complex due to the setup required= behind the scenes for full builds. Rather than directly using Python's unittest, th= e code wraps most of the standard objects. The tests can be simple, such as testi= ng a command from within the @@ -431,9 +445,7 @@ class BitbakeLayers(OESelftestTestCase): def test_bitbakelayers_showcrossdepends(self): result =3D runCmd('bitbake-layers show-cross-depends') = =20 - self.assertTrue('aspell' in result.output, msg =3D "No depe= ndencies=20 - were shown. bitbake-layers show-cross-depends output: - %s"% result.output) + self.assertTrue('aspell' in result.output, msg =3D "No depe= ndencies were shown. bitbake-layers show-cross-depends output: %s"% resul= t.output) This example, taken from meta/lib/oeqa/selftest/cases/bblayers.py, creates a @@ -468,10 +480,8 @@ bitbake command and exist outsi= de of its context. As a result, common bitbake library functions (bb.*) are also= unavailable.
- -
+
<filename>testimage</filename> - These tests are run once an image is up and running, e= ither on target hardware or under QEMU. As a result, they are assumed to be running = in a target image environment, as opposed to a host build environment. A s= imple example from @@ -481,8 +491,7 @@ @OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core']) def test_python3(self): - cmd =3D "python3 -c \"import codecs; print(codecs.encode('U= ryyb, - jbeyq', 'rot13'))\"" + cmd =3D "python3 -c \"import codecs; print(codecs.encode('U= ryyb', 'jbeyq', 'rot13'))\"" status, output =3D self.target.run(cmd) msg =3D 'Exit status was not 0. Output: %s' % output self.assertEqual(status, 0, msg=3Dmsg) @@ -496,10 +505,8 @@ decorators. For example, the test in this example would = only make sense if python3-core is installed in the image.
- -
+
<filename>testsdk_ext</filename> - These tests are run against built extensible SDKs (eSD= Ks). The tests can assume that the eSDK environment has already been setup. An exa= mple from meta/lib/oeqa/sdk/cases/devtool.py contains the @@ -507,33 +514,31 @@ class DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls): - myapp_src =3D os.path.join(cls.tc.esdk_files_dir, "myapp") - cls.myapp_dst =3D os.path.join(cls.tc.sdk_dir, "myapp") - shutil.copytree(myapp_src, cls.myapp_dst) - subprocess.check_output(['git', 'init', '.'], cwd=3Dcls.myapp_= dst) - subprocess.check_output(['git', 'add', '.'], cwd=3Dcls.myapp_d= st) - subprocess.check_output(['git', 'commit', '-m', "'test commit'= "], cwd=3Dcls.myapp_dst) + myapp_src =3D os.path.join(cls.tc.esdk_files_dir, "myapp") + cls.myapp_dst =3D os.path.join(cls.tc.sdk_dir, "myapp") + shutil.copytree(myapp_src, cls.myapp_dst) + subprocess.check_output(['git', 'init', '.'], cwd=3Dcls.my= app_dst) + subprocess.check_output(['git', 'add', '.'], cwd=3Dcls.mya= pp_dst) + subprocess.check_output(['git', 'commit', '-m', "'test com= mit'"], cwd=3Dcls.myapp_dst) =20 @classmethod def tearDownClass(cls): - shutil.rmtree(cls.myapp_dst) + shutil.rmtree(cls.myapp_dst) def _test_devtool_build(self, directory): - self._run('devtool add myapp %s' % directory) - try: - self._run('devtool build myapp') - finally: - self._run('devtool reset myapp') + self._run('devtool add myapp %s' % directory) + try: + self._run('devtool build myapp') + finally: + self._run('devtool reset myapp') def test_devtool_build_make(self): - self._test_devtool_build(self.myapp_dst) + self._test_devtool_build(self.myapp_dst) In this example, the devtool command i= s tested to see whether a sample application can be built with the devto= ol build command within the eSDK.
- -
+
<filename>testsdk</filename> - These tests are run against built SDKs. The tests can = assume that an SDK has already been extracted and its environment file has been= sourced. A simple example from meta/lib/oeqa/sdk/cases/python2.py contains the @@ -553,10 +558,8 @@ runs the python3 interpreter with a basic command to che= ck it is working correctly. The test would only run if python3 is installed in the S= DK.
- -
+
<filename>oe-build-perf-test</filename> - The performance tests usually measure how long operati= ons take and the resource utilisation as that happens. An example from meta/lib/oeqa/buildperf/test_basic.py contains the @@ -582,7 +585,7 @@ various caches, to show how BitBake=E2=80=99s parsing pe= rformance trends over time.
-
+
Considerations When Writing Tests When writing good tests, there are several things to keep = in mind. Since things running on the Autobuilder are accessed concurrently by mult= iple workers, consider the @@ -610,14 +613,6 @@ metadate from changes, set up temporary copies of that d= ata first.
- - - - - - - - - - - -Understanding the Yocto Project Autobuilder + + Understanding the Yocto Project Autobuilder
Execution Flow within the Autobuilder The =E2=80=9Ca-full=E2=80=9D and =E2=80=9Ca-quick=E2=80=9D= targets are the usual entry points into the Autobuilder and @@ -69,14 +66,12 @@ repository in the scripts directory. Th= e following section details how this works.
- -
+
Autobuilder Target Execution Overview - For each given target in a build, the Autobuilder executes= several steps. These are configured in yocto-autobuilder2/builders.py and roughly consist of: - + Run clobberdir<= /emphasis> This cleans out any previous build. Old builds= are left around to allow easier debugging of failed builds. For additiona= l information, see
-
+
Autobuilder Technology The Autobuilder has Yocto Project-specific functionality t= o allow builds to operate with increased efficiency and speed. -
+
clobberdir When deleting files, the Autobuilder uses cl= obberdir, which is a special script that moves files to a special locati= on, rather than deleting @@ -147,7 +142,7 @@ runs this deletion. See Autobuilder Worker Janitor.
-
+
Autobuilder Clone Cache Cloning repositories from scratch each time they are r= equired was slow on the Autobuilder. We therefore have a stash of commonly used = repositories pre-cloned on @@ -156,7 +151,7 @@ Autobuilder Worker Janitor. See Autobuilder Worker Janitor.
-
+
Autobuilder Worker Janitor This is a process running on each Worker that performs= two basic operations, including background file deletion at IO idle (see
-
+
Shared DL_DIR The Workers are all connected over NFS which allows DL= _DIR to be shared between them. This reduces network accesses from the system and = allows the build to be sped up. Usage of the directory within the build system is de= signed to be able to be shared over NFS.
-
+
Shared SSTATE_DIR The Workers are all connected over NFS which allows th= e sstate directory to be shared b= etween them. This means once a Worker has built an artefact, all the others can benef= it from it. Usage of the directory within the directory is designed for sharing o= ver NFS.
-
+
Resulttool All of the different tests run as part of the build ge= nerate output into testresults.json files. This al= lows us to determine which @@ -191,11 +186,13 @@ >https://wiki.yoctoproject.org/wiki/Resulttool.
-
+
run-config Target Execution The scripts/run-config execution is w= here most of the work within - the Autobuilder happens. It runs through a number of steps; = the first are general setup - steps that are run once and include: + the Autobuilder happens. It runs through a number of steps, = and logs the details to the + stdio log file in the Web UI by def= ault. Certain steps generate + their own specific log files as detailed in this section. + The first steps perform general setup, are run once, and i= nclude: Set up any buildtools-tarball if configured. @@ -206,13 +203,14 @@ For each step that is configured in config.json<= /filename>, it will perform the following: - ## WRITER's question: What does "logging in as stepX= a" and others refer to - below? ## Add any layers that are specified = using the - bitbake-layers add-layer command (logging as - stepXa) + bitbake-layers add-layer command. + The details are saved in the + stepXa log file, + where X is a number b= etween 1 and 20. This + variable applies to all steps in this section. Call the scripts/setup-c= onfig script to @@ -220,26 +218,37 @@ the build - Run the bitbake BBTARGET= S command (logging - as stepXb) + Run the bitbake BBTARGET= S command. + The output logs to the + stepXb log + file. - Run the bitbake SANITYTA= RGETS command - (logging as stepXc) + Run the bitbake SANITYTA= RGETS + command. + The output logs to the + stepXc log + file. Run the EXTRACMDS command, which are run - within the BitBake build environment (logging as= stepXd) + within the BitBake build environment. + The output logs to the + stepXd log + file. Run the EXTRAPLAINCMDS command(s), which - are run outside the BitBake build environment (l= ogging as stepXd) + are run outside the BitBake build environment. T= he output logs to the + stepXd log + file. Remove any layers added in = step 1 using the - bitbake-layers remove-layer command (logging as - stepXa) + bitbake-layers remove-layer command. The ouput logs + to the stepXa log + file. @@ -265,7 +274,7 @@
-
+
Deploying Yocto Autobuilder The most up to date information about how to setup and dep= loy your own Autbuilder can be found in README.md in the yocto-autobuilder2 repository. @@ -300,14 +309,6 @@ $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
- - - - - - - -