Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pathpartnertech.com; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=vrJ0lCPziccbS+/8mrWIv6vApe/VwdoKQgDwpUPwIL8=; b=mg/WpaGG1gXyZEPDn4obfVh4rJDfc/Aff4+1cAC3bkMDAg2kxY/Ha46hdE6jqfDvPp E+a7MeAbCDJxrw6zqWLb0py4lkezjacvGDQiFgl0lJy8nnuRBgcIFbqvaPSlTA3V2tbu O8mTTU/LVVBXXUkqBorXv48Okwg017uMPAvShXq+NghX8o7V9xSGGFz+6jF8oeza9sqv PGCMoEqoFj0qf6hVbnsJ2xNVMNumJnBXHg+xGnJncBYn73kSTNGNrb+TQH63lkfPCU0Y bxVg0vbcX9ef5kt6KRscWgCajuR3n8o91WS1wc/FQ38BpCTNegtR03ZwlqLsNNyFzWaF oAHA== From: Pooja Date: Fri, 20 Nov 2020 11:01:54 +0530 Message-Id: <1605850314-14796-1-git-send-email-pooja.sm@pathpartnertech.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: [Fuego] [PATCH] Docs:Convert tbwiki pages into .rst files List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: tbird20d@yahoo.com, Tim.Bird@sony.com Cc: fuego@lists.linuxfoundation.org, Pooja More From: Pooja More Following are the pages converted: run.json.rst spec.json.rst test.yaml.rst Signed-off-by: Pooja More --- docs/rst_src/run.json.rst | 323 +++++++++++++++++++++++++++++++++++++++++= ++++ docs/rst_src/spec.json.rst | 183 +++++++++++++++++++++++++ docs/rst_src/test.yaml.rst | 177 +++++++++++++++++++++++++ 3 files changed, 683 insertions(+) create mode 100644 docs/rst_src/run.json.rst create mode 100644 docs/rst_src/spec.json.rst create mode 100644 docs/rst_src/test.yaml.rst diff --git a/docs/rst_src/run.json.rst b/docs/rst_src/run.json.rst new file mode 100644 index 0000000..d128fb3 --- /dev/null +++ b/docs/rst_src/run.json.rst @@ -0,0 +1,323 @@ +########### +run.json +########### + +Summary +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The ``run.json`` file has data about a particular test run. It has +information about the test, including the results for the test. + +The format of portions of this file was inspired by the KernelCI API. +See ``_ + +The results are included in an array of test_set objects, which can +contain arrays of test_case objects, which themselves may contain +measurement objects. + + +Field details +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + + * **duration** - the amount of time, in milliseconds, that the test + took to execute + + * If the test included a build, this time is included in this number + + * **metadata** - various fields that are specific to Fuego + + * **attachments** - a list of the files that are available for this + test - usually logs and such + * **batch_id** - a string indicating the batch of tests this test was r= un + in (if applicable) + * **board** - the board the test was executed on + * **build_number** - the Jenkins build number + * **compiled_on** - indicates the location where the test was compiled + * **fuego_core_version** - version of the fuego core system + * **fuego_version** - version of the fuego container system + * **host_name** - the host. If not configured, it may be 'local_host' + * **job_name** - the Jenkins job name for this test run + * **keep_log** - indicates whether the log is kept (???) + * **kernel_version** - the version of the kernel running on the board + * **reboot** - indicates whether a reboot was requested for this test r= un + * **rebuild** - indicates whether it was requested to rebuild the sourc= e + for this run + * **start_time** - time when this test run was started (in seconds sinc= e + Jan 1, 1970) + * **target_postcleanup** - indicates whether cleanup of test materials = on the + board was requested for after test execution + * **target_precleanup** - indicates whether cleanup of test materials o= n the + board was requested for before test execution + * **test_plan** - test plan being executed for this test run. May be '= None' + if test was not executed in the context of a larger plan + * **test_spec** - test spec used for this run + * **testsuite_version** - version of the source program used for this r= un + + * FIXTHIS - testsuite_version is not calculated properly yet + + * **timestamp** - time when this test run was started (in ISO 8601 form= at) + * **toolchain** - the toolchains (or PLATFORM) used to build the test p= rogram + * **workspace** - a directory on the host where test materials were ext= racted + and built, for this test. + + * This is the parent directory used, and not the specific directory u= sed for + this test. + + * **name** - the name of the test + * **status** - the test result as a string. This can be one of: + + * PASS + * FAIL + * ERROR + * SKIP + + * **test_sets** - list of test_set objects, containing test results + * **test_cases** - list of test_case objects, containing test results + + * Each test_case object has: + + * **name** - the test case name + * **status** - the result for that test case + + * **measurements** - list of measurement objects, containing test results + + * For each measurement, the following attributes may be present: + + * **name** - the measure name + * **status** - the pass/fail result for that test case + * **measure** - the numeric result for that test case + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Examples +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Here are some sample run.json files, from Fuego 1.2 + + +Functional test results +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D + +This was generated using + +:: + + ftc run-test -b docker -t Functional.hello_world + +This example only has a single test_case. + +:: + + { + "duration_ms": 1245, + "metadata": { + "attachments": [ + { + "name": "devlog", + "path": "devlog.txt" + }, + { + "name": "devlog", + "path": "devlog.txt" + }, + { + "name": "syslog.before", + "path": "syslog.before.txt" + }, + { + "name": "syslog.after", + "path": "syslog.after.txt" + }, + { + "name": "testlog", + "path": "testlog.txt" + }, + { + "name": "consolelog", + "path": "consolelog.txt" + }, + { + "name": "test_spec", + "path": "spec.json" + } + ], + "board": "docker", + "build_number": "3", + "compiled_on": "docker", + "fuego_core_version": "v1.1-805adb0", + "fuego_version": "v1.1-5ad677b", + "host_name": "fake_host", + "job_name": "docker.default.Functional.hello_world", + "keep_log": true, + "kernel_version": "3.19.0-47-generic", + "reboot": "false", + "rebuild": "false", + "start_time": "1509662455755", + "target_postcleanup": true, + "target_precleanup": "true", + "test_plan": "None", + "test_spec": "default", + "testsuite_version": "v1.1-805adb0", + "timestamp": "2017-11-02T22:40:55+0000", + "toolchain": "x86_64", + "workspace": "/fuego-rw/buildzone" + }, + "name": "Functional.hello_world", + "schema_version": "1.0", + "status": "PASS", + "test_sets": [ + { + "name": "default", + "status": "PASS", + "test_cases": [ + { + "name": "hello_world", + "status": "PASS" + } + ] + } + ] + } + + + +Benchmark results +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Here is the run.json file for a run of the test ``Benchmark.netperf`` +on the board 'ren1' (which is a Renesas board in my lab). + +:: + + { + "duration_ms": 33915, + "metadata": { + "attachments": [ + { + "name": "devlog", + "path": "devlog.txt" + }, + { + "name": "devlog", + "path": "devlog.txt" + }, + { + "name": "syslog.before", + "path": "syslog.before.txt" + }, + { + "name": "syslog.after", + "path": "syslog.after.txt" + }, + { + "name": "testlog", + "path": "testlog.txt" + }, + { + "name": "consolelog", + "path": "consolelog.txt" + }, + { + "name": "test_spec", + "path": "spec.json" + } + ], + "board": "ren1", + "build_number": "3", + "compiled_on": "docker", + "fuego_core_version": "v1.2.0", + "fuego_version": "v1.2.0", + "host_name": "local_host", + "job_name": "ren1.default.Benchmark.netperf", + "keep_log": true, + "kernel_version": "4.9.0-yocto-standard", + "reboot": "false", + "rebuild": "false", + "start_time": "1509669904085", + "target_postcleanup": true, + "target_precleanup": "true", + "test_plan": "None", + "test_spec": "default", + "testsuite_version": "v1.1-805adb0", + "timestamp": "2017-11-03T00:45:04+0000", + "toolchain": "poky-aarch64", + "workspace": "/fuego-rw/buildzone" + }, + "name": "Benchmark.netperf", + "schema_version": "1.0", + "status": "PASS", + "test_sets": [ + { + "name": "default", + "status": "PASS", + "test_cases": [ + { + "measurements": [ + { + "measure": 928.51, + "name": "net", + "status": "PASS" + }, + { + "measure": 59.43, + "name": "cpu", + "status": "PASS" + } + ], + "name": "MIGRATED_TCP_STREAM", + "status": "PASS" + }, + { + "measurements": [ + { + "measure": 934.1, + "name": "net", + "status": "PASS" + }, + { + "measure": 56.61, + "name": "cpu", + "status": "PASS" + } + ], + "name": "MIGRATED_TCP_MAERTS", + "status": "PASS" + } + ] + } + ] + } + + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Ideas +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Some changes to the fields might be useful: + + * We don't have anything that records the 'cause', from Jenkins + + * This is supposed to indicate what triggered the test + * The Jenkins strings are somewhat indecipherable: + + * Here is a Jenkins cause: 1 + + * It might be worthwhile to add some fields from the board or target: + + * Architecture + * Transport - not sure about this one + * Distrib + * File system + + * If we add monitors or side-processes (stressors), it would be good to a= dd + information about those as well + +Use of flat data +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Parsing the tree-structured data has turned out to be a real pain, and +it might be better to do most of the work in a flat format. The +charting code uses a mixture of both structured (nested objects) and +flat testcase names, and I think there's a lot of duplicate code lying +around that handles the conversion back and forth, that could probably +be coalesced into a single set of library routines. + diff --git a/docs/rst_src/spec.json.rst b/docs/rst_src/spec.json.rst new file mode 100644 index 0000000..6296776 --- /dev/null +++ b/docs/rst_src/spec.json.rst @@ -0,0 +1,183 @@ +############ +spec.json +############ + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Introduction +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The file ``spec.json`` is defined for each test. This file allows for +the same test to be used in multiple different ways. This is often +referred to as a parameterized test. + +The ``spec.json`` file indicates a list of "specs" for the test, and +for each test the values for test variables (parameters) that the test +will use to configure its behavior. + +The variables declared in a spec are made available as shell variables +to the test at test runtime. To avoid naming collisions, the test +variables are prefixed with the name of the test. They are also +converted to all upper-case. + +So, for example, for a test called Functional.mytest, if the spec +declared a variable called 'loops', with a value of "10", the +following test variable would be defined: FUNCTIONAL_MYTEST_LOOPS=3D10 + +The intent is allow for a test author or some other party to declare a +set of parameters to run the test in a different configuration. + +Fuego is often used to wrap existing test programs and benchmarks, +which have command line options for controlling various test execution +parameters. Setting these command line options is one of the primary +purposes of specs, and the spec.json file. + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Schema +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +``spec.json`` holds a single object, with a 'testName' attribute, and an +attribute called 'specs' that is a collection of spec definitions. +Each spec definition has a name and a collection of named attributes. + + * **testName** - this indicates the test that these specs apply to + * **specs** - this indicates the collection of specs + * **fail_case** - this allows a test to provide a list failure expression= s + that will be be checked for in the test or system logs + + * **fail_regexp** - a regular expression that indicates a failure. + This is grep'ed for in the testlog (unless use_syslog is set) + * **fail_message** - a message to output when the regular expression i= s + found + + * **use_syslog** - a flag indicating to scan for the fail_regexp in th= e + system log rather than the test log + +Within each spec, there should be a collection of name/value pairs. +Note that the values in a name/value pair are expanded in test context, +so that the value may reference other test variables (such as from +the board file, or the stored variables file for a board). + +Special variables +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +There are some special variables that can be defined, that are recognized +by the Fuego core system. + +One of these is: + + * **PER_JOB_BUILD** - if this variable is defined, then Fuego will create + a separate build area for each job that this test is included in, even = if + a board or another job uses the same toolchain. This is used when the = test + variables are used in the ''build'' phase, and can affect the binary th= at is + compiled during this phase. + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Examples +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Here is an example, from the test ``Functional.bc``: + +:: + + { + "testName": "Functional.bc", + "fail_case": [ + { + "fail_regexp": "syntax error", + "fail_message": "Text expression has a syntax error" + }, + { + "fail_regexp": "Bug", + "fail_message": "Bug or Oops detected in system log", + "use_syslog": "1" + } + ], + "specs": { + "default": { + "EXPR":"3+3", + "RESULT":"6" + }, + "bc-mult": { + "EXPR":"2*2", + "RESULT": "4" + }, + "bc-add": { + "EXPR":"3+3", + "RESULT":"6" + }, + "bc-by2": { + "PER_JOB_BUILD": "true", + "tarball": "by2.tar.gz", + "EXPR":"3+3", + "RESULT":"12" + }, + "bc-fail": { + "EXPR":"3 3", + "RESULT":"6" + }, + } + } + +In this example, the EXPR variable is used as input to the program +'bc' and the RESULT gives the expected output from bc. + +This particular ``spec.json`` file is this complex for instructional +purposes, and this particular test is somewhat overly parameterized. + + +Here is an example, from the test ``Functional.synctest``: + +:: + + { + "testName": "Functional.synctest", + "specs": { + "sata": { + "MOUNT_BLOCKDEV":"$SATA_DEV", + "MOUNT_POINT":"$SATA_MP", + "LEN":"10", + "LOOP":"10" + }, + "mmc": { + "MOUNT_BLOCKDEV":"$MMC_DEV", + "MOUNT_POINT":"$MMC_MP", + "LEN":"10", + "LOOP":"10" + }, + "usb": { + "MOUNT_BLOCKDEV":"$USB_DEV", + "MOUNT_POINT":"$USB_MP", + "LEN":"10", + "LOOP":"10" + }, + "default": { + "MOUNT_BLOCKDEV":"ROOT", + "MOUNT_POINT":"$BOARD_TESTDIR/work", + "LEN":"30", + "LOOP":"10" + } + } + } + + +Note the use of variables references for ``MOUNT_BLOCKDEV`` and +``MOUNT_POINT``. These use values ($SATA_DEV, $MMC_DEV or $USB_DEV) that +should be defined in a board file for filesystem-related tests. + +When a test defines variables, they should be documented in the test's +``test.yaml`` file. + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Defaults +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +If a test has no ``spec.json``, then default set of values is used, which +is a single spec with the name "default", and no values defined. + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +See also +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + + * See :ref:`Test Spec and Plans` for more information about + Fuego's test spec and testplan system. + diff --git a/docs/rst_src/test.yaml.rst b/docs/rst_src/test.yaml.rst new file mode 100644 index 0000000..1db55f6 --- /dev/null +++ b/docs/rst_src/test.yaml.rst @@ -0,0 +1,177 @@ +############ +test.yaml +############ + +The ``test.yaml`` file is used to hold meta-information about a test. Thi= s is +used by the :ref:`Test package system` for packaging a test and providing +information for viewing and searching for tests in a proposed "test store"= . +The ``test.yaml`` file can also can be used by human maintainers to preser= ve +information (in a structured format) about a test, that is not included in= the +other test materials. + +As an overview, the ``test.yaml`` file indicates where the source for the +test comes from, it's license, the name of the test maintainer, a +description of the test and tags for categorizing the test, and a +formal list of parameters that are used by the test (what they mean +and how to use them). + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +test.yaml fields +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Here are the fields supported in a ``test.yaml`` file: + +``fuego_package_version`` + + Indicates the version of package + (in case of changes to the package schema). For now, this is always 1. + +``name`` + + Has the full Fuego name of the test. Ex: Benchmark.iperf + +``description`` + + Has an English description of the test + +``license`` + + Has an SPDX identifier for the test. This is the main + license of the test project that the Fuego test uses, if the project + has a tarfile or git repo. Otherwise it reflects the license of any + non-Fuego-specific materials in the test directory. In such case, + the test directory should include a LICENSE file. Fuego materials + (``fuego_test.sh``, ``spec.json``, ``chart_config.json``, etc.) are + considered to be under the default Fuego license (which is BSD-3-Clause) + unless otherwise specifically indicated in these files. The license + identifier for this field should be obtained from + ``_ + +``author`` + + The author or authors of the base test + +``maintainer`` + + The maintainer of the Fuego materials for this test + +``version`` + + The version of the base test + +``fuego_release`` + + The version of Fuego materials for this test. This is a monotonically + incrementing integer, starting at 1 for each new version of the base tes= t. + +``type`` + + Either Benchmark or Functional + +``tags`` + + A list of tags used to categorize this test. This is intended to be + used in an eventual online test store. + +``tarball_src`` + + A URL where the tarball was originally obtained from + + +``gitrepo`` + + A git URL where the source may be obtained from + +``host_dependencies`` + + A list of Debian package names that must be installed in the docker + container in order for this test to work properly. This field is + optional, and indicates packages needed that are beyond those included i= n + the standard Fuego host distribution in the Fuego docker container. + +``params`` + + A list of parameters that may be used with this test, including their + descriptions, whether they are optional or required, and an example + value for each one + +``data_files`` + + A list of the files that are included in this test. This is used as the + manifest for packaging the test (``fuego_test.sh``, and ``test.yaml`` ar= e + implicitly included in the packaging manifest). + + +More on params +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The 'params' field in the test.yaml file is a list of dictionaries +with one item per test variable used by the test. + +The name of the parameter is the short name of the parameter, without +the test name prefix (e.g. FUNCTIONAL_LTP). The parameter name is the +key for the dictionary with that parameters attributes. + +Each parameter has a dictionary with attributes describing it. The +dictionary has the following fields (keys): + + - 'description' - text description of the parameter + - 'example' - an example of the parameter + - 'optional' - indicates whether the test requires this parameter + (test variable) to be set or not. The value of the 'optional' + field must be one of 'yes' or 'no'. + +The test variables may be described by the ``test.yaml`` file can be +defined in one of multiple locations in the Fuego test system. Most +commonly the test variables are defined in a spec for the test, but +they can also be defined in the board file, or as a dynamic board +variable. + +=3D=3D=3D=3D=3D=3D=3D=3D=3D +Example +=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Here is an example ``test.yaml`` file, for the package ``Benchmark.iperf3`= `: + +:: + + fuego_package_version: 1 + name: Benchmark.iperf3 + description: | + iPerf3 is a tool for active measurements of the maximum achievable + bandwidth on IP networks. + license: BSD-3-Clause. + author: | + Jon Dugan, Seth Elliott, Bruce A. Mah, Jeff Poskanzer, Kaustubh Prab= hu, + Mark Ashley, Aaron Brown, Aeneas Jai=C3=9Fle, Susant Sahani, Bruce S= impson, + Brian Tierney. + maintainer: Daniel Sangorrin + version: 3.1.3 + fuego_release: 1 + type: Benchmark + tags: ['network', 'performance'] + tarball_src: https://iperf.fr/download/source/iperf-3.1.3-source.tar.gz + gitrepo: https://github.com/esnet/iperf.git + params: + - server_ip: + description: | + IP address of the server machine. If not provided, then SRV_= IP + _must_ be provided on the board file. Otherwise the test wil= l fail. + if the server ip is assigned to the host, the test automatic= ally + starts the iperf3 server daemon. Otherwise, the tester _must= _ make + sure that iperf3 -V -s -D is already running on the server m= achine. + example: 192.168.1.45 + optional: yes + - client_params: + description: extra parameters for the client + example: -p 5223 -u -b 10G + optional: yes + data_files: + - chart_config.json + - fuego_test.sh + - parser.py + - spec.json + - criteria.json + - iperf-3.1.3-source.tar.gz + - reference.json + - test.yaml --=20 2.7.4 --=20 This message contains confidential information and is intended only=20 for the individual(s) named. If you are not the intended recipient, you are=20 notified that disclosing, copying, distributing or taking any action in=20 reliance on the contents of this mail and attached file/s is strictly prohibited. Please notify the sender immediately and delete this e-mail=20 from your system. E-mail transmission cannot be guaranteed to be secured or=20 error-free as information could be intercepted, corrupted, lost, destroyed,=20 arrive late or incomplete, or contain viruses. The sender therefore does=20 not accept liability for any errors or omissions in the contents of this=20 message, which arise as a result of e-mail transmission.