Skip to contentSkip to conversation

guides: add a testing guide for developing tests

PATCH Superseded Aaron Conole 9 participants 20 messages Feb 8, 2021 – Nov 26, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 guides: add a testing guide for developing tests

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

---
v0: Added information for TEST_SKIPPED and details about generating
code coverage to help with ideas for writing unit test cases.

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 231 ++++++++++++++++++++++++++++
 2 files changed, 232 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst
doc/guides/contributing/index.rst +1 -0
index 2fefd91931..41909d949b 100644
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
doc/guides/contributing/testing.rst new file +231 -0

231 lines not on this page · raw message

Signature · 1 line
2.25.4
#1 lore raw
Quoted 48 lines · Aaron Conole

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

---
v0: Added information for TEST_SKIPPED and details about generating
code coverage to help with ideas for writing unit test cases.

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 231 ++++++++++++++++++++++++++++
 2 files changed, 232 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst
doc/guides/contributing/index.rst +1 -0
index 2fefd91931..41909d949b 100644
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
doc/guides/contributing/testing.rst new file +11 -0
new file mode 100644
index 0000000000..3ade654772
@@ -0,0 +1,231 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors
+
+.. _testing_guidelines:
+
+DPDK Testing Guidelines
+=======================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+

doc/guides/contributing/testing.rst line 21

Quoted 10 lines
doc/guides/contributing/testing.rst new file +10 -0
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runniner, the `dpdk-test` app.

runner

doc/guides/contributing/testing.rst line 27

Quoted 6 lines
doc/guides/contributing/testing.rst new file +6 -0
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are two modes of operation.  The first mode is as an interactive
+command shell that allows launching specific test suites.  This is
+the default operating mode of `dpdk-test` and can be done by::

is the double :: intentional?

Quoted 13 lines
doc/guides/contributing/testing.rst new file +13 -0
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>

doc/guides/contributing/testing.rst line 50

Quoted 10 lines
doc/guides/contributing/testing.rst new file +10 -0
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system.  This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --no-huge

better to say --dpdk-options-here instead of --no-huge, as above.

Quoted 30 lines
doc/guides/contributing/testing.rst new file +30 -0
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 20.02.0-rc0'
+  Test OK
+  RTE>>$
+
+The above shows running a specific test case.  On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255').
+
+
+Running all tests
+-----------------
+
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+default test suites.  After building via `ninja`, the ``meson test``
+command will execute the standard tests and report errors.
+
+There are four groups of default test suites.  The first group is

doc/guides/contributing/testing.rst line 90

Quoted 10 lines
doc/guides/contributing/testing.rst new file +10 -0
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second group is the **perf** tests.  These test suites can take
+longer to run and do performance evaluations.  The third group is
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`).  The last group are the
+**debug** tests.  These mostly are used to dump system information.
+
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``

show some console output in this section?

doc/guides/contributing/testing.rst line 98

Quoted 8 lines
doc/guides/contributing/testing.rst new file +8 -0
+
+
+Adding test suites
+------------------
+
+To add a testsuite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite).  There are two useful things:

There are two useful things?

Quoted 6 lines
doc/guides/contributing/testing.rst new file +6 -0
+
+  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+     Registers a test command with the name `command_name` and which
+     runs the function `function_to_execute` when `command_name` is
+     invoked.
+

doc/guides/contributing/testing.rst line 114

Quoted 10 lines
doc/guides/contributing/testing.rst new file +10 -0
+  2. unit_test_suite_runner(struct unit_test_suite \*)
+     Returns a runner for a full test suite object, which contains
+     a test suite name, setup, teardown, and vector of unit test
+     cases.
+
+Each test suite has a setup and teardown function that runs at the
+beginning and end of the test suite execution.  Each unit test has
+a similar function for test case setup and teardown.
+
+Adding test cases is controlled via the `.unit_test_cases` element

Test cases are added to the .unit_test_cases element of the of the unit test suite structure.

Quoted 67 lines
doc/guides/contributing/testing.rst new file +67 -0
+of the unit test suite.  Ex:
+
+.. code-block:: c
+   :linenos:
+
+   #include <time.h>
+
+   #include <rte_common.h>
+   #include <rte_cycles.h>
+   #include <rte_hexdump.h>
+   #include <rte_random.h>
+
+   #include "test.h"
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_testsuite = {
+          .suite_name = "EXAMPLE TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static int example_tests()
+   {
+       return unit_test_suite_runner(&example_testsuite);
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner.  Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code.  This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure.  The basic form is:
+
+.. code-block:: c
+
+   RTE_TEST_ASSERT(cond, msg, ...)

doc/guides/contributing/testing.rst line 191

Quoted 10 lines
doc/guides/contributing/testing.rst new file +10 -0
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here.  The *msg* parameter will be a
+message to display if *cond* evaluates to **false**.  Some specialized
+macros already exist.  See `lib/librte_eal/include/rte_test.h` for
+a list of pre-build test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't availble.  The

available

doc/guides/contributing/testing.rst line 198

Quoted 7 lines
doc/guides/contributing/testing.rst new file +7 -0
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available.  A skipped test isn't the
+same as a failed test.  Failed tests indicate that all required
+functionality was enabled, and the environment was correct but the
+test failed due to some error in logic.  A skipped test indicates
+that the test setup wasn't available to run.

Simpler to provide an example of a skipped test, than a long explanation?

doc/guides/contributing/testing.rst

Quoted 8 lines
doc/guides/contributing/testing.rst new file +8 -0
+
+
+Checking code coverage
+----------------------
+One of the best ways to help design a test case is to add a new test
+which exercises functionality that doesn't currently have a test case.
+The best way to find these is to run the test suite with some
+additional code coverage reporting turned on.  

I would drop the previous sentences.
They would just confuse a reader looking to generate code coverage.

Quoted 27 lines

The meson build system

doc/guides/contributing/testing.rst new file +25 -0
+supports generating a code coverage report via the `-Db_coverage=true`
+option, in conjunction with a package like **lcov**, to generate an
+HTML code coverage report.  Example::
+
+  $ meson covered -Db_coverage=true
+  $ meson test -C covered --suite fast-tests
+  $ ninja coverage-html -C covered
+
+The above will generate an html report in the
+`covered/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
+in test development.
+
+
+Adding a suite to the default
+-----------------------------
+
+Adding to one of the default tests involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class.  Once added, the new test suite will
+be run as part of the appropriate class (fast, perf, driver, etc.).
+
+Some of these default test suites are run during continuous integration
+tests, making regression checking automatic for new patches submitted
+to the project.
#2 lore raw
Quoted 53 lines · Ray
Quoted 52 lines · Aaron Conole

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

---
v0: Added information for TEST_SKIPPED and details about generating
code coverage to help with ideas for writing unit test cases.

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 231 ++++++++++++++++++++++++++++
 2 files changed, 232 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst
doc/guides/contributing/index.rst +1 -0
index 2fefd91931..41909d949b 100644
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
doc/guides/contributing/testing.rst new file +15 -0
new file mode 100644
index 0000000000..3ade654772
@@ -0,0 +1,231 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors
+
+.. _testing_guidelines:
+
+DPDK Testing Guidelines
+=======================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
Quoted 8 lines

doc/guides/contributing/testing.rst line 21

Quoted 6 lines
doc/guides/contributing/testing.rst new file +6 -0
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runniner, the `dpdk-test` app.

runner

D'oh!

Quoted 8 lines

doc/guides/contributing/testing.rst line 27

Quoted 6 lines
doc/guides/contributing/testing.rst new file +6 -0
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are two modes of operation.  The first mode is as an interactive
+command shell that allows launching specific test suites.  This is
+the default operating mode of `dpdk-test` and can be done by::

is the double :: intentional?

Yes - it designates the next block as a code block.

Quoted 18 lines
Quoted 18 lines
doc/guides/contributing/testing.rst new file +5 -0
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+ EAL: Static memory layout is selected, amount of reserved memory

can be adjusted with -m or --socket-mem

doc/guides/contributing/testing.rst new file +12 -0
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
Quoted 8 lines

doc/guides/contributing/testing.rst line 50

Quoted 6 lines
doc/guides/contributing/testing.rst new file +6 -0
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system.  This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --no-huge

better to say --dpdk-options-here instead of --no-huge, as above.

Okay, I can do that.

Quoted 35 lines
Quoted 35 lines
doc/guides/contributing/testing.rst new file +3 -0
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+ EAL: Static memory layout is selected, amount of reserved memory

can be adjusted with -m or --socket-mem

doc/guides/contributing/testing.rst new file +31 -0
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 20.02.0-rc0'
+  Test OK
+  RTE>>$
+
+The above shows running a specific test case.  On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255').
+
+
+Running all tests
+-----------------
+
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+default test suites.  After building via `ninja`, the ``meson test``
+command will execute the standard tests and report errors.
+
+There are four groups of default test suites.  The first group is
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second group is the **perf** tests.  These test suites can take
+longer to run and do performance evaluations.  The third group is
Quoted 8 lines

doc/guides/contributing/testing.rst line 90

Quoted 6 lines
doc/guides/contributing/testing.rst new file +6 -0
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`).  The last group are the
+**debug** tests.  These mostly are used to dump system information.
+
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``

show some console output in this section?

Okay.

Quoted 10 lines

doc/guides/contributing/testing.rst line 98

Quoted 8 lines
doc/guides/contributing/testing.rst new file +8 -0
+
+
+Adding test suites
+------------------
+
+To add a testsuite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite).  There are two useful things:

There are two useful things?

Okay, I will rephrase because it sounds awkward. But I don't think it's wrong ;)

Quoted 11 lines
Quoted 11 lines
doc/guides/contributing/testing.rst new file +11 -0
+
+  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+     Registers a test command with the name `command_name` and which
+     runs the function `function_to_execute` when `command_name` is
+     invoked.
+
+  2. unit_test_suite_runner(struct unit_test_suite \*)
+     Returns a runner for a full test suite object, which contains
+     a test suite name, setup, teardown, and vector of unit test
+     cases.
+
Quoted 8 lines

doc/guides/contributing/testing.rst line 114

Quoted 5 lines
doc/guides/contributing/testing.rst new file +5 -0
+Each test suite has a setup and teardown function that runs at the
+beginning and end of the test suite execution.  Each unit test has
+a similar function for test case setup and teardown.
+
+Adding test cases is controlled via the `.unit_test_cases` element

Test cases are added to the .unit_test_cases element of the of the unit test suite structure.

Will rephrase.

Quoted 71 lines
Quoted 71 lines
doc/guides/contributing/testing.rst new file +71 -0
+of the unit test suite.  Ex:
+
+.. code-block:: c
+   :linenos:
+
+   #include <time.h>
+
+   #include <rte_common.h>
+   #include <rte_cycles.h>
+   #include <rte_hexdump.h>
+   #include <rte_random.h>
+
+   #include "test.h"
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_testsuite = {
+          .suite_name = "EXAMPLE TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static int example_tests()
+   {
+       return unit_test_suite_runner(&example_testsuite);
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner.  Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code.  This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure.  The basic form is:
+
+.. code-block:: c
+
+   RTE_TEST_ASSERT(cond, msg, ...)
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here.  The *msg* parameter will be a
+message to display if *cond* evaluates to **false**.  Some specialized
Quoted 8 lines

doc/guides/contributing/testing.rst line 191

Quoted 6 lines
doc/guides/contributing/testing.rst new file +6 -0
+macros already exist.  See `lib/librte_eal/include/rte_test.h` for
+a list of pre-build test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't availble.  The

available

D'oh!

Quoted 9 lines

doc/guides/contributing/testing.rst line 198

Quoted 7 lines
doc/guides/contributing/testing.rst new file +7 -0
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available.  A skipped test isn't the
+same as a failed test.  Failed tests indicate that all required
+functionality was enabled, and the environment was correct but the
+test failed due to some error in logic.  A skipped test indicates
+that the test setup wasn't available to run.

Simpler to provide an example of a skipped test, than a long explanation?

I can do both.

Quoted 11 lines

doc/guides/contributing/testing.rst

Quoted 8 lines
doc/guides/contributing/testing.rst new file +8 -0
+
+
+Checking code coverage
+----------------------
+One of the best ways to help design a test case is to add a new test
+which exercises functionality that doesn't currently have a test case.
+The best way to find these is to run the test suite with some
+additional code coverage reporting turned on.  

I would drop the previous sentences.
They would just confuse a reader looking to generate code coverage.

Okay. I felt it was a bit strange to introduce code coverage here, and I thought it was a needed bridge. I can drop it.

Quoted 27 lines
Quoted 27 lines

The meson build system

doc/guides/contributing/testing.rst new file +25 -0
+supports generating a code coverage report via the `-Db_coverage=true`
+option, in conjunction with a package like **lcov**, to generate an
+HTML code coverage report.  Example::
+
+  $ meson covered -Db_coverage=true
+  $ meson test -C covered --suite fast-tests
+  $ ninja coverage-html -C covered
+
+The above will generate an html report in the
+`covered/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
+in test development.
+
+
+Adding a suite to the default
+-----------------------------
+
+Adding to one of the default tests involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class.  Once added, the new test suite will
+be run as part of the appropriate class (fast, perf, driver, etc.).
+
+Some of these default test suites are run during continuous integration
+tests, making regression checking automatic for new patches submitted
+to the project.
#3 lore raw

PATCH v2 1/1 guides: add a testing guide for developing tests

In reply to Aaron Conole

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

---
v0->v1: Added information for TEST_SKIPPED and details about generating
code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
Ray.

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 245 ++++++++++++++++++++++++++++
 2 files changed, 246 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst
doc/guides/contributing/index.rst +1 -0
index 2fefd91931..41909d949b 100644
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
doc/guides/contributing/testing.rst new file +245 -0

245 lines not on this page · raw message

Signature · 1 line
2.25.4
#4 lore raw

doc/guides/contributing/testing.rst line 2

Quoted 8 lines · Aaron Conole
doc/guides/contributing/testing.rst new file +2 -0
new file mode 100644
index 0000000000..86ca24ce43
@@ -0,0 +1,245 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors

2021?

doc/guides/contributing/testing.rst line 4

Quoted 2 lines
doc/guides/contributing/testing.rst new file +2 -0
+
+.. _testing_guidelines:

I can't find a call to the testing_guidelines reference, so this can be removed.

[snip]

doc/guides/contributing/testing.rst line 101

Quoted 13 lines
doc/guides/contributing/testing.rst new file +13 -0
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...

Trying this in my build env, I get all tests failing.
This is because I run this as a normal user, but the system has hugepages configured.
I figured this out quickly since I know the test framework (simply added a echo 0; exit at the top of has-hugepages.sh).
But I am not sure a reader of this doc would be able to troubleshoot this.

Not sure if this is worth explaining here, or if we can enhance the hugepage check (permissions maybe?).

[snip]

doc/guides/contributing/testing.rst line 225

Quoted 7 lines
doc/guides/contributing/testing.rst new file +7 -0
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson covered -Db_coverage=true

At first, I read "covered" as a meson command :-).
I prefer an explicit "meson setup covered -Db_coverage=true", but well this is more a matter of taste.

doc/guides/contributing/testing.rst line 233

Quoted 8 lines
doc/guides/contributing/testing.rst new file +8 -0
+  $ meson test -C covered --suite fast-tests
+  $ ninja coverage-html -C covered
+
+The above will generate an html report in the
+`covered/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
+in test development.
+

The rest of the doc lgtm, and this is a good addition, feel free to add my review tag.
Thanks Aaron.

Signature · 1 line
David Marchand
#5 lore raw
Quoted 47 lines · David Marchand

doc/guides/contributing/testing.rst line 2

Quoted 8 lines · Aaron Conole
doc/guides/contributing/testing.rst new file +2 -0
new file mode 100644
index 0000000000..86ca24ce43
@@ -0,0 +1,245 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors

2021?

doc/guides/contributing/testing.rst line 4

Quoted 2 lines
doc/guides/contributing/testing.rst new file +2 -0
+
+.. _testing_guidelines:

I can't find a call to the testing_guidelines reference, so this can be removed.

[snip]

doc/guides/contributing/testing.rst line 101

Quoted 13 lines
doc/guides/contributing/testing.rst new file +13 -0
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...

Trying this in my build env, I get all tests failing.
This is because I run this as a normal user, but the system has hugepages configured.
I figured this out quickly since I know the test framework (simply added a echo 0; exit at the top of has-hugepages.sh).
But I am not sure a reader of this doc would be able to troubleshoot this.

Not sure if this is worth explaining here, or if we can enhance the hugepage check (permissions maybe?).

[snip]

Quoted 13 lines

doc/guides/contributing/testing.rst line 225

Quoted 7 lines
doc/guides/contributing/testing.rst new file +7 -0
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson covered -Db_coverage=true

At first, I read "covered" as a meson command :-).
I prefer an explicit "meson setup covered -Db_coverage=true", but well this is more a matter of taste.

I also tend to prefer the build directory name at the end of the command, so I'd suggest: "meson setup -Db_coverage=true covered". Furthermore, while I can understand the use of "covered" as a build directory name, I think for consistency across all docs, we should just use "build" here as the directory name, which again will reduce confusion. "meson setup -Db_coverage build"

/Bruce

#6 lore raw
Quoted 12 lines · Bruce Richardson
Quoted 11 lines · David Marchand

doc/guides/contributing/testing.rst line 2

Quoted 8 lines · Aaron Conole
doc/guides/contributing/testing.rst new file +2 -0
new file mode 100644
index 0000000000..86ca24ce43
@@ -0,0 +1,245 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors

2021?

Whoops, I forgot to update my time machine.

Quoted 4 lines
Quoted 4 lines

doc/guides/contributing/testing.rst line 4

Quoted 2 lines
doc/guides/contributing/testing.rst new file +2 -0
+
+.. _testing_guidelines:

I can't find a call to the testing_guidelines reference, so this can be removed.

done.

Quoted 24 lines
Quoted 24 lines

[snip]

doc/guides/contributing/testing.rst line 101

Quoted 13 lines
doc/guides/contributing/testing.rst new file +13 -0
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...

Trying this in my build env, I get all tests failing.
This is because I run this as a normal user, but the system has hugepages configured.
I figured this out quickly since I know the test framework (simply added a echo 0; exit at the top of has-hugepages.sh).
But I am not sure a reader of this doc would be able to troubleshoot this.

Quoted 2 lines
Quoted 2 lines

Not sure if this is worth explaining here, or if we can enhance the hugepage check (permissions maybe?).

I prefer to fix the hugepage check to make the tests SKIP when we don't have hugepages accessible (so we need some kind of permission check in there). I will submit it separately.

Quoted 17 lines
Quoted 16 lines

[snip]

doc/guides/contributing/testing.rst line 225

Quoted 7 lines
doc/guides/contributing/testing.rst new file +7 -0
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson covered -Db_coverage=true

At first, I read "covered" as a meson command :-).
I prefer an explicit "meson setup covered -Db_coverage=true", but well this is more a matter of taste.

Quoted 6 lines

I also tend to prefer the build directory name at the end of the command, so I'd suggest: "meson setup -Db_coverage=true covered". Furthermore, while I can understand the use of "covered" as a build directory name, I think for consistency across all docs, we should just use "build" here as the directory name, which again will reduce confusion. "meson setup -Db_coverage build"

Okay - I will paint this bikeshed like:

meson setup build -Db_coverage=true

It's a little inconsistent everywhere - so I guess a good janitor project would be to clean up all the places we have meson commands.

Otherwise, what I've found is that the options generally come after the build directory / command is specified (ex: see octeontx, the arm64 cross build docs, etc.) so I'll keep that for consistency there. Hopefully we will consistently become more consistent :)

Quoted 1 line

/Bruce

#7 lore raw
Quoted 83 lines · Aaron Conole
Quoted 12 lines · Bruce Richardson
Quoted 11 lines · David Marchand

doc/guides/contributing/testing.rst line 2

Quoted 8 lines · Aaron Conole
doc/guides/contributing/testing.rst new file +2 -0
new file mode 100644
index 0000000000..86ca24ce43
@@ -0,0 +1,245 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors

2021?

Whoops, I forgot to update my time machine.

Quoted 4 lines
Quoted 4 lines

doc/guides/contributing/testing.rst line 4

Quoted 2 lines
doc/guides/contributing/testing.rst new file +2 -0
+
+.. _testing_guidelines:

I can't find a call to the testing_guidelines reference, so this can be removed.

done.

Quoted 24 lines
Quoted 24 lines

[snip]

doc/guides/contributing/testing.rst line 101

Quoted 13 lines
doc/guides/contributing/testing.rst new file +13 -0
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...

Trying this in my build env, I get all tests failing.
This is because I run this as a normal user, but the system has hugepages configured.
I figured this out quickly since I know the test framework (simply added a echo 0; exit at the top of has-hugepages.sh).
But I am not sure a reader of this doc would be able to troubleshoot this.

Quoted 2 lines
Quoted 2 lines

Not sure if this is worth explaining here, or if we can enhance the hugepage check (permissions maybe?).

I prefer to fix the hugepage check to make the tests SKIP when we don't have hugepages accessible (so we need some kind of permission check in there). I will submit it separately.

Quoted 17 lines
Quoted 16 lines

[snip]

doc/guides/contributing/testing.rst line 225

Quoted 7 lines
doc/guides/contributing/testing.rst new file +7 -0
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson covered -Db_coverage=true

At first, I read "covered" as a meson command :-).
I prefer an explicit "meson setup covered -Db_coverage=true", but well this is more a matter of taste.

Quoted 6 lines

I also tend to prefer the build directory name at the end of the command, so I'd suggest: "meson setup -Db_coverage=true covered". Furthermore, while I can understand the use of "covered" as a build directory name, I think for consistency across all docs, we should just use "build" here as the directory name, which again will reduce confusion. "meson setup -Db_coverage build"

Okay - I will paint this bikeshed like:

meson setup build -Db_coverage=true

Quoted 8 lines

It's a little inconsistent everywhere - so I guess a good janitor project would be to clean up all the places we have meson commands.

Otherwise, what I've found is that the options generally come after the build directory / command is specified (ex: see octeontx, the arm64 cross build docs, etc.) so I'll keep that for consistency there. Hopefully we will consistently become more consistent :)

That's some fine bikeshed painting! And I'm fine with all your suggestion.

/Bruce

#8 lore raw
In reply to Aaron Conole
Quoted 48 lines · Aaron Conole
Quoted 12 lines · Bruce Richardson
Quoted 11 lines · David Marchand

doc/guides/contributing/testing.rst line 2

Quoted 8 lines · Aaron Conole
doc/guides/contributing/testing.rst new file +2 -0
new file mode 100644
index 0000000000..86ca24ce43
@@ -0,0 +1,245 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors

2021?

Whoops, I forgot to update my time machine.

Quoted 4 lines
Quoted 4 lines

doc/guides/contributing/testing.rst line 4

Quoted 2 lines
doc/guides/contributing/testing.rst new file +2 -0
+
+.. _testing_guidelines:

I can't find a call to the testing_guidelines reference, so this can be removed.

done.

Quoted 24 lines
Quoted 24 lines

[snip]

doc/guides/contributing/testing.rst

Quoted 13 lines
doc/guides/contributing/testing.rst new file +13 -0
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...

Trying this in my build env, I get all tests failing.
This is because I run this as a normal user, but the system has hugepages configured.
I figured this out quickly since I know the test framework (simply added a echo 0; exit at the top of has-hugepages.sh).
But I am not sure a reader of this doc would be able to troubleshoot this.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

Not sure if this is worth explaining here, or if we can enhance the hugepage check (permissions maybe?).

I prefer to fix the hugepage check to make the tests SKIP when we don't have hugepages accessible (so we need some kind of permission check in there). I will submit it separately.

Here is my PoC for this - if you think it's good enough, I'll submit as formal PATCH.

---
index d600fad319..1c3cfb665a 100755

app/test/has-hugepage.sh +11 -1
@@ -3,7 +3,17 @@
 # Copyright 2020 Mellanox Technologies, Ltd
 
 if [ "$(uname)" = "Linux" ] ; then
-       cat /proc/sys/vm/nr_hugepages || echo 0
+       nr_hugepages=$(cat /proc/sys/vm/nr_hugepages)
+       # Need to check if we have permissions to access hugepages
+       perm=""
+       for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do
+               test ! -w $mount/. || perm="$mount"
+       done
+       if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then
+               echo 0
+       else
+               echo $nr_hugepages
+       fi
 elif [ "$(uname)" = "FreeBSD" ] ; then
        echo 1 # assume FreeBSD always has hugepages
 else

---

#9 lore raw
Quoted 47 lines · Aaron Conole
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

Review

Quoted 13 lines
(quoted diff) +13 -0
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...

Trying this in my build env, I get all tests failing.
This is because I run this as a normal user, but the system has hugepages configured.
I figured this out quickly since I know the test framework (simply added a echo 0; exit at the top of has-hugepages.sh).
But I am not sure a reader of this doc would be able to troubleshoot this.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

Not sure if this is worth explaining here, or if we can enhance the hugepage check (permissions maybe?).

I prefer to fix the hugepage check to make the tests SKIP when we don't have hugepages accessible (so we need some kind of permission check in there). I will submit it separately.

Here is my PoC for this - if you think it's good enough, I'll submit as formal PATCH.

---
index d600fad319..1c3cfb665a 100755

app/test/has-hugepage.sh +5 -1
@@ -3,7 +3,17 @@
 # Copyright 2020 Mellanox Technologies, Ltd
 
 if [ "$(uname)" = "Linux" ] ; then
-       cat /proc/sys/vm/nr_hugepages || echo 0
+       nr_hugepages=$(cat /proc/sys/vm/nr_hugepages)
+       # Need to check if we have permissions to access hugepages
+       perm=""
+       for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do
+               test ! -w $mount/. || perm="$mount"

app/test/has-hugepage.sh

Quoted 10 lines
app/test/has-hugepage.sh +6 -1
+       done
+       if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then
+               echo 0
+       else
+               echo $nr_hugepages
+       fi
 elif [ "$(uname)" = "FreeBSD" ] ; then
        echo 1 # assume FreeBSD always has hugepages
 else
---

I need to think more about the multiple mountpoints case (but I may be imagining too much twisted setups..).

At least, this works in my env.
We need tests in travis/GHA, and sending a non-RFC patch is the best way to know :-)

So +1 for a patch.
Thanks Aaron!

Signature · 1 line
David Marchand
#10 lore raw
Quoted 62 lines · David Marchand
Quoted 47 lines · Aaron Conole
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

Review

Quoted 13 lines
(quoted diff) +13 -0
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...

Trying this in my build env, I get all tests failing.
This is because I run this as a normal user, but the system has hugepages configured.
I figured this out quickly since I know the test framework (simply added a echo 0; exit at the top of has-hugepages.sh).
But I am not sure a reader of this doc would be able to troubleshoot this.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

Not sure if this is worth explaining here, or if we can enhance the hugepage check (permissions maybe?).

I prefer to fix the hugepage check to make the tests SKIP when we don't have hugepages accessible (so we need some kind of permission check in there). I will submit it separately.

Here is my PoC for this - if you think it's good enough, I'll submit as formal PATCH.

---
index d600fad319..1c3cfb665a 100755

app/test/has-hugepage.sh +5 -1
@@ -3,7 +3,17 @@
 # Copyright 2020 Mellanox Technologies, Ltd
 
 if [ "$(uname)" = "Linux" ] ; then
-       cat /proc/sys/vm/nr_hugepages || echo 0
+       nr_hugepages=$(cat /proc/sys/vm/nr_hugepages)
+       # Need to check if we have permissions to access hugepages
+       perm=""
+       for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do
+               test ! -w $mount/. || perm="$mount"

app/test/has-hugepage.sh

Quoted 10 lines
app/test/has-hugepage.sh +6 -1
+       done
+       if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then
+               echo 0
+       else
+               echo $nr_hugepages
+       fi
 elif [ "$(uname)" = "FreeBSD" ] ; then
        echo 1 # assume FreeBSD always has hugepages
 else
---

I need to think more about the multiple mountpoints case (but I may be imagining too much twisted setups..).

Quoted 3 lines

At least, this works in my env.
We need tests in travis/GHA, and sending a non-RFC patch is the best way to know :-)

Sent :)

Quoted 2 lines

So +1 for a patch.
Thanks Aaron!

#11 lore raw

PATCH v3 1/1 guides: add a guide for developing unit tests

In reply to Aaron Conole

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

---
v0->v1: Added information for TEST_SKIPPED and details about generating
code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
Ray.
v2->v3: Rewrite the meson build block, updated the copyright section,
and change the title to be a bit nicer

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 243 ++++++++++++++++++++++++++++
 2 files changed, 244 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst
doc/guides/contributing/index.rst +1 -0
index 2fefd91931..41909d949b 100644
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
doc/guides/contributing/testing.rst new file +243 -0

243 lines not on this page · raw message

Signature · 1 line
2.25.4
#12 lore raw

Re: [dpdk-dev] [PATCH v3] guides: add a guide for developing unit tests

Quoted 8 lines · Aaron Conole

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

+1 to adding this long missing documentation, thanks.

Quoted 34 lines

---
v0->v1: Added information for TEST_SKIPPED and details about generating
code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
Ray.
v2->v3: Rewrite the meson build block, updated the copyright section,
and change the title to be a bit nicer

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 243 ++++++++++++++++++++++++++++
 2 files changed, 244 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst
doc/guides/contributing/index.rst +1 -0
index 2fefd91931..41909d949b 100644
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
doc/guides/contributing/testing.rst new file +0 -0
new file mode 100644
index 0000000000..0757d71ad0

doc/guides/contributing/testing.rst line 9

Quoted 9 lines
doc/guides/contributing/testing.rst new file +9 -0
@@ -0,0 +1,243 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2021 The DPDK contributors
+
+DPDK Testing Guidelines
+=======================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+

I think both the section name (DPDK Testing Guidelines) and the file name (testing.rst) is too broad comparing to what is described here.

What about using "DPDK Unit Test Guidelines", and 'unit_test.rst'?

Quoted 44 lines
doc/guides/contributing/testing.rst new file +44 -0
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runner, the `dpdk-test` app.
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are two modes of operation.  The first mode is as an interactive
+command shell that allows launching specific test suites.  This is
+the default operating mode of `dpdk-test` and can be done by::
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system.  This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'

doc/guides/contributing/testing.rst line 63

Quoted 10 lines
doc/guides/contributing/testing.rst new file +10 -0
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 20.02.0-rc0'
+  Test OK
+  RTE>>$
+

According code, it is also possible to run the unit test by providing its name as argument to 'dpdk-test', like bellow. And benefit of this is, it is possible to provide multiple tests at once and run them all, which can't be done via 'DPDK_TEST" env var. Should we document this too?

./build/app/test/dpdk-test version_autotest
EAL: Detected 96 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 1
EAL: No available 2048 kB hugepages reported
APP: HPET is not enabled, using TSC as default timer RTE>>version_autotest
Version string: 'DPDK 21.05.0'
Test OK

Another env variable it "DPDK_TEST_PARAMS" to provide more parameters, since meson has args support for test exec, we are not using it in meson, but maybe good to document it for the individual exec usage.

Also there is a 'RTE_TEST_RECURSIVE' env var used, I am not clear why it is for, anyone knows the history on it?

doc/guides/contributing/testing.rst line 67

Quoted 4 lines
doc/guides/contributing/testing.rst new file +4 -0
+The above shows running a specific test case.  On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255').
+

Also negative value

doc/guides/contributing/testing.rst line 71

Quoted 4 lines
doc/guides/contributing/testing.rst new file +4 -0
+
+Running all tests
+-----------------
+

What about "Running multiple test cases", instead of "all tests"?

Btw, it is possible to run single unit test with meson, perhaps this can be documented in previous section:
"meson test -C build version_autotest"

Also another useful command is to list all test cases, may worth documenting that too:
"meson test -C build --list"

We have a legacy runner as well, 'autotest.py', most probably it is outdated now, if we don't document it we may consider removing it completely (./app/test/autotest*.py):
"python ./app/test/autotest.py ./build/app/test/dpdk-test build version"

Quoted 23 lines
doc/guides/contributing/testing.rst new file +23 -0
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+default test suites.  After building via `ninja`, the ``meson test``
+command will execute the standard tests and report errors.
+
+There are four groups of default test suites.  The first group is
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second group is the **perf** tests.  These test suites can take
+longer to run and do performance evaluations.  The third group is
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`).  The last group are the
+**debug** tests.  These mostly are used to dump system information.
+
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s

doc/guides/contributing/testing.rst line 104

Quoted 10 lines
doc/guides/contributing/testing.rst new file +10 -0
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...
+
+
+Adding test suites
+------------------
+

What about renaming 'test suites' to 'test cases', since 'test suites' is just a group of test case and we can add single test case too.

Quoted 5 lines
doc/guides/contributing/testing.rst new file +5 -0
+To add a test suite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite).  There are two important functions
+for interacting with the test harness:
+

doc/guides/contributing/testing.rst line 119

Quoted 10 lines
doc/guides/contributing/testing.rst new file +10 -0
+  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+     Registers a test command with the name `command_name` and which
+     runs the function `function_to_execute` when `command_name` is
+     invoked.
+
+  2. unit_test_suite_runner(struct unit_test_suite \*)
+     Returns a runner for a full test suite object, which contains
+     a test suite name, setup, tear down, and vector of unit test
+     cases.
+

There are two different usage as "test suite" which is confusing I think,

1) First usage within dpdk-test app

a) A test command can be running single test_case, like 'version_autotest' -> 'test_version'

b) Or it can be a group of test cases, like:
'link_bonding_autotest' -> 'test_create_bonded_device',

			'test_create_bonded_device_with_invalid_params',
			'test_add_slave_to_bonded_device',
			'...'

The second one is called as unit_test_suite within the scope of the 'dpdk-test' application, but at the end of the day it is a single command for test app.

First one has simple execution, but second one has setup/teardown as you already described below.

2) Second usage with meson,

There is a next level of grouping in meson, as described in previous paragraph, like "fast-test suite", "perf test suite" etc.., this is grouping of various test commands.

I wonder if this two different "test suite" can be clarified more. Can we say "meson test suite" to the meson one?

Quoted 106 lines
doc/guides/contributing/testing.rst new file +106 -0
+Each test suite has a setup and tear down function that runs at the
+beginning and end of the test suite execution.  Each unit test has
+a similar function for test case setup and tear down.
+
+Test cases are added to the `.unit_test_cases` element of the unit
+test suite structure.  Ex:
+
+.. code-block:: c
+   :linenos:
+
+   #include <time.h>
+
+   #include <rte_common.h>
+   #include <rte_cycles.h>
+   #include <rte_hexdump.h>
+   #include <rte_random.h>
+
+   #include "test.h"
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_testsuite = {
+          .suite_name = "EXAMPLE TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static int example_tests()
+   {
+       return unit_test_suite_runner(&example_testsuite);
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner.  Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code.  This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure.  The basic form is:
+
+.. code-block:: c
+
+   RTE_TEST_ASSERT(cond, msg, ...)
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here.  The *msg* parameter will be a
+message to display if *cond* evaluates to **false**.  Some specialized
+macros already exist.  See `lib/librte_eal/include/rte_test.h` for
+a list of defined test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't available.  The
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available.  Ex::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  ...
+  46/60 DPDK:fast-tests / ipsec_autotest       SKIP     0.22 s
+  ...
+
+
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson setup build -Db_coverage=true
+  $ meson test -C build --suite fast-tests
+  $ ninja coverage-html -C build

doc/guides/contributing/testing.rst line 235

Quoted 10 lines
doc/guides/contributing/testing.rst new file +10 -0
+
+The above will generate an html report in the
+`build/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
+in test development.
+
+
+Adding a suite to the default
+-----------------------------
+

What is "the default"?

doc/guides/contributing/testing.rst line 240

Quoted 5 lines
doc/guides/contributing/testing.rst new file +5 -0
+Adding to one of the default tests involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class.  Once added, the new test suite will
+be run as part of the appropriate class (fast, perf, driver, etc.).
+

Here "meson test suite" referenced as 'class', lets figure out the naming for it and use it consistently.

doc/guides/contributing/testing.rst line 241

Quoted 1 line
doc/guides/contributing/testing.rst new file +1 -0
+Some of these default test suites are run during continuous integration

Same comment for "default test suites" usage.

doc/guides/contributing/testing.rst line 243

Quoted 3 lines
doc/guides/contributing/testing.rst new file +2 -0
+tests, making regression checking automatic for new patches submitted
+to the project.

Is there a rule to document how to decide to add (or not) a testcase to a meson test suit?

#13 lore raw

Re: [dpdk-dev] [PATCH v3] guides: add a guide for developing unit tests

Quoted 50 lines · Ferruh Yigit
Quoted 8 lines · Aaron Conole

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

+1 to adding this long missing documentation, thanks.

Quoted 38 lines

---
v0->v1: Added information for TEST_SKIPPED and details about generating
code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
Ray.
v2->v3: Rewrite the meson build block, updated the copyright section,
and change the title to be a bit nicer

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 243 ++++++++++++++++++++++++++++
 2 files changed, 244 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst
doc/guides/contributing/index.rst +1 -0
index 2fefd91931..41909d949b 100644
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
doc/guides/contributing/testing.rst new file +4 -0
new file mode 100644
index 0000000000..0757d71ad0
@@ -0,0 +1,243 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2021 The DPDK contributors
+
+DPDK Testing Guidelines
Quoted 8 lines

doc/guides/contributing/testing.rst line 9

Quoted 5 lines
doc/guides/contributing/testing.rst new file +5 -0
+=======================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+

I think both the section name (DPDK Testing Guidelines) and the file name (testing.rst) is too broad comparing to what is described here.

Makes sense to me.

Quoted 1 line

What about using "DPDK Unit Test Guidelines", and 'unit_test.rst'?

Sure - I will make this change.

Quoted 53 lines
Quoted 53 lines
doc/guides/contributing/testing.rst new file +21 -0
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runner, the `dpdk-test` app.
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are two modes of operation.  The first mode is as an interactive
+command shell that allows launching specific test suites.  This is
+the default operating mode of `dpdk-test` and can be done by::
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+ EAL: Static memory layout is selected, amount of reserved memory

can be adjusted with -m or --socket-mem

doc/guides/contributing/testing.rst new file +21 -0
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system.  This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+ EAL: Static memory layout is selected, amount of reserved memory

can be adjusted with -m or --socket-mem

doc/guides/contributing/testing.rst new file +9 -0
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 20.02.0-rc0'
Quoted 8 lines

doc/guides/contributing/testing.rst line 63

Quoted 3 lines
doc/guides/contributing/testing.rst new file +3 -0
+  Test OK
+  RTE>>$
+

According code, it is also possible to run the unit test by providing its name as argument to 'dpdk-test', like bellow. And benefit of this is, it is possible to provide multiple tests at once and run them all, which can't be done via 'DPDK_TEST" env var. Should we document this too?

Such functionality was added after I posted this series. I think it's a worthwhile addition.

Quoted 15 lines

./build/app/test/dpdk-test version_autotest
EAL: Detected 96 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 1
EAL: No available 2048 kB hugepages reported
APP: HPET is not enabled, using TSC as default timer RTE>>version_autotest
Version string: 'DPDK 21.05.0'
Test OK

Quoted 3 lines

Another env variable it "DPDK_TEST_PARAMS" to provide more parameters, since meson has args support for test exec, we are not using it in meson, but maybe good to document it for the individual exec usage.

Makes sense.

Quoted 19 lines

Also there is a 'RTE_TEST_RECURSIVE' env var used, I am not clear why it is for, anyone knows the history on it?

doc/guides/contributing/testing.rst line 67

Quoted 4 lines
doc/guides/contributing/testing.rst new file +4 -0
+The above shows running a specific test case.  On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255').
+

Also negative value

doc/guides/contributing/testing.rst line 71

Quoted 4 lines
doc/guides/contributing/testing.rst new file +4 -0
+
+Running all tests
+-----------------
+

What about "Running multiple test cases", instead of "all tests"?

Quoted 7 lines

Btw, it is possible to run single unit test with meson, perhaps this can be documented in previous section:
"meson test -C build version_autotest"

Also another useful command is to list all test cases, may worth documenting that too:
"meson test -C build --list"

Agreed - I will add this.

Quoted 5 lines

We have a legacy runner as well, 'autotest.py', most probably it is outdated now, if we don't document it we may consider removing it completely (./app/test/autotest*.py):
"python ./app/test/autotest.py ./build/app/test/dpdk-test build version"

I guess it might be best to remove it.

Quoted 28 lines
Quoted 28 lines
doc/guides/contributing/testing.rst new file +28 -0
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+default test suites.  After building via `ninja`, the ``meson test``
+command will execute the standard tests and report errors.
+
+There are four groups of default test suites.  The first group is
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second group is the **perf** tests.  These test suites can take
+longer to run and do performance evaluations.  The third group is
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`).  The last group are the
+**debug** tests.  These mostly are used to dump system information.
+
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...
Quoted 8 lines

doc/guides/contributing/testing.rst line 104

Quoted 5 lines
doc/guides/contributing/testing.rst new file +5 -0
+
+
+Adding test suites
+------------------
+

What about renaming 'test suites' to 'test cases', since 'test suites' is just a group of test case and we can add single test case too.

Okay.

Quoted 18 lines
Quoted 5 lines
doc/guides/contributing/testing.rst new file +5 -0
+To add a test suite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite).  There are two important functions
+for interacting with the test harness:
+

doc/guides/contributing/testing.rst line 119

Quoted 10 lines
doc/guides/contributing/testing.rst new file +10 -0
+  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+     Registers a test command with the name `command_name` and which
+     runs the function `function_to_execute` when `command_name` is
+     invoked.
+
+  2. unit_test_suite_runner(struct unit_test_suite \*)
+     Returns a runner for a full test suite object, which contains
+     a test suite name, setup, tear down, and vector of unit test
+     cases.
+

There are two different usage as "test suite" which is confusing I think,

Quoted 6 lines

1) First usage within dpdk-test app

a) A test command can be running single test_case, like 'version_autotest' -> 'test_version'

b) Or it can be a group of test cases, like:

Well 'link_bonding_autotest' is a "test-suite"

Quoted 7 lines

'link_bonding_autotest' -> 'test_create_bonded_device',

			'test_create_bonded_device_with_invalid_params',
			'test_add_slave_to_bonded_device',
			'...'

The second one is called as unit_test_suite within the scope of the 'dpdk-test' application, but at the end of the day it is a single command for test app.

Sure, the distinction is more important for development rather than executing cases, so maybe I can make it more clear here.

Quoted 6 lines

First one has simple execution, but second one has setup/teardown as you already described below.

2) Second usage with meson,

Quoted 7 lines

There is a next level of grouping in meson, as described in previous paragraph, like "fast-test suite", "perf test suite" etc.., this is grouping of various test commands.

I wonder if this two different "test suite" can be clarified more. Can we say "meson test suite" to the meson one?

Okay, that makes sense to me.

Quoted 110 lines
Quoted 110 lines
doc/guides/contributing/testing.rst new file +110 -0
+Each test suite has a setup and tear down function that runs at the
+beginning and end of the test suite execution.  Each unit test has
+a similar function for test case setup and tear down.
+
+Test cases are added to the `.unit_test_cases` element of the unit
+test suite structure.  Ex:
+
+.. code-block:: c
+   :linenos:
+
+   #include <time.h>
+
+   #include <rte_common.h>
+   #include <rte_cycles.h>
+   #include <rte_hexdump.h>
+   #include <rte_random.h>
+
+   #include "test.h"
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_testsuite = {
+          .suite_name = "EXAMPLE TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static int example_tests()
+   {
+       return unit_test_suite_runner(&example_testsuite);
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner.  Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code.  This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure.  The basic form is:
+
+.. code-block:: c
+
+   RTE_TEST_ASSERT(cond, msg, ...)
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here.  The *msg* parameter will be a
+message to display if *cond* evaluates to **false**.  Some specialized
+macros already exist.  See `lib/librte_eal/include/rte_test.h` for
+a list of defined test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't available.  The
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available.  Ex::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  ...
+  46/60 DPDK:fast-tests / ipsec_autotest       SKIP     0.22 s
+  ...
+
+
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson setup build -Db_coverage=true
+  $ meson test -C build --suite fast-tests
+  $ ninja coverage-html -C build
+
+The above will generate an html report in the
+`build/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
Quoted 8 lines

doc/guides/contributing/testing.rst line 235

Quoted 6 lines
doc/guides/contributing/testing.rst new file +6 -0
+in test development.
+
+
+Adding a suite to the default
+-----------------------------
+

What is "the default"?

I meant the meson test suite here.

Quoted 8 lines

doc/guides/contributing/testing.rst line 240

Quoted 5 lines
doc/guides/contributing/testing.rst new file +5 -0
+Adding to one of the default tests involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class.  Once added, the new test suite will
+be run as part of the appropriate class (fast, perf, driver, etc.).
+

Here "meson test suite" referenced as 'class', lets figure out the naming for it and use it consistently.

Yes, will do.

Quoted 3 lines
Quoted 1 line

+Some of these default test suites are run during continuous integration

Same comment for "default test suites" usage.

ACK

Quoted 6 lines

doc/guides/contributing/testing.rst line 243

Quoted 3 lines
doc/guides/contributing/testing.rst new file +2 -0
+tests, making regression checking automatic for new patches submitted
+to the project.

Is there a rule to document how to decide to add (or not) a testcase to a meson test suit?

No - it's mostly up to developers and maintainers for now, although I would think the normal thing would be to add the case to the meson test suite. I will try to clarify it a bit more.

#14 lore raw

PATCH v4 1/1 guides: add a guide for developing unit tests

In reply to Aaron Conole

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

---
v0->v1: Added information for TEST_SKIPPED and details about generating
code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
Ray.
v2->v3: Rewrite the meson build block, updated the copyright section,
and change the title to be a bit nicer.
v3->v4: Dropped 'reviewed-by' tag from David Marchand.
Incorporated suggestions from Ferruh.
Clarified 'meson test suite' vs. 'dpdk test suite'. Added a bit on when to add tests to meson.
Renamed to 'unit test' to be more specific.
Added section about passing test names as command arguments.

 doc/guides/contributing/index.rst     |   1 +
 doc/guides/contributing/unit_test.rst | 327 ++++++++++++++++++++++++++
 2 files changed, 328 insertions(+)
 create mode 100644 doc/guides/contributing/unit_test.rst
doc/guides/contributing/index.rst +1 -0
index 2fefd91931..493732a518 100644
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    unit_test
     vulnerability
     stable
     cheatsheet
doc/guides/contributing/unit_test.rst new file +327 -0

327 lines not on this page · raw message

Signature · 1 line
2.31.1
Acked-by on v4 1/1
#15 lore raw

Re: [dpdk-dev] [PATCH v4] guides: add a guide for developing unit tests

Hi Aaron,

Quoted 12 lines

From: Aaron Conole <[email protected]>
Sent: Wednesday 14 July 2021 17:41
To: [email protected]
Cc: David Marchand <[email protected]>; Richardson, Bruce <[email protected]>; Yigit, Ferruh <[email protected]>; Power, Ciara <[email protected]>
Subject: [PATCH v4] guides: add a guide for developing unit tests

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Quoted 6 lines

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

---

<snip>

This is a useful doc, looks good.

Acked-by on v4 1/1
#16 lore raw

Re: [dpdk-dev] [PATCH v4] guides: add a guide for developing unit tests

In reply to Aaron Conole

Hi Aaron,

Quoted 13 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Aaron Conole
Sent: Wednesday, July 14, 2021 5:41 PM
To: [email protected]
Cc: David Marchand <[email protected]>; Richardson, Bruce <[email protected]>; Yigit, Ferruh <[email protected]>; Power, Ciara <[email protected]>
Subject: [dpdk-dev] [PATCH v4] guides: add a guide for developing unit tests

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Quoted 7 lines

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

---

Acked-by on v4 1/1
#17 lore raw

Re: [dpdk-dev] [PATCH v4] guides: add a guide for developing unit tests

In reply to Aaron Conole

doc/guides/contributing/unit_test.rst

Quoted 13 lines
doc/guides/contributing/unit_test.rst new file +11 -0
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer  RTE>>

Minor suggestion. You could truncate the long "Static memory" line in this and other blocks to prevent it invoking the scroll bars. Something, like this:

Quoted 1 line

+ EAL: Static memory layout is selected ...

Apart from that, great work on trawling through the weeds of the DPDK units test infrastructure and creating this really useful doc.

Thank you.

#18 lore raw

PATCH v5 1/1 guides: add a guide for developing unit tests

In reply to Aaron Conole

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

---
v0->v1: Added information for TEST_SKIPPED and details about generating
code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
Ray.
v2->v3: Rewrite the meson build block, updated the copyright section,
and change the title to be a bit nicer.
v3->v4: Dropped 'reviewed-by' tag from David Marchand.
Incorporated suggestions from Ferruh.
Clarified 'meson test suite' vs. 'dpdk test suite'. Added a bit on when to add tests to meson.
Renamed to 'unit test' to be more specific.
Added section about passing test names as command arguments.
v4->v5: Update by trimming long line as suggested by John McNamara

 doc/guides/contributing/index.rst     |   1 +
 doc/guides/contributing/unit_test.rst | 327 ++++++++++++++++++++++++++
 2 files changed, 328 insertions(+)
 create mode 100644 doc/guides/contributing/unit_test.rst
doc/guides/contributing/index.rst +1 -0
index 2fefd91931..493732a518 100644
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    unit_test
     vulnerability
     stable
     cheatsheet
doc/guides/contributing/unit_test.rst new file +327 -0

327 lines not on this page · raw message

Signature · 1 line
2.31.1
#19 lore raw

Re: [dpdk-dev] [PATCH v5] guides: add a guide for developing unit tests

15/10/2021 19:06, Aaron Conole:

Quoted 9 lines

The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases.

Quoted 4 lines

Applied with some formatting changes, and added recent "extra" suite, thanks.