Skip to contentSkip to conversation

doc: Add basic shell test description

PATCH Superseded Cyril Hrubis 3 participants 8 messages Nov 10, 2025 – Nov 14, 2025

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 doc: Add basic shell test description

---
 doc/developers/api_shell_tests.rst | 91 ++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
doc/developers/api_shell_tests.rst +91 -0
index b6e8560d9..bad9f1f4b 100644
@@ -2,3 +2,94 @@
 
 LTP shell API
 =============
+
+Shell API overview
+------------------
+
+First lines of the shell test should be a shebang, a license, and copyrights.
+
+.. code-block:: shell
+
+   #!/bin/sh
+   # SPDX-License-Identifier: GPL-2.0-or-later
+   # Copyright 2099 Foo Bar <[email protected]>
+
+A documentation comment block formatted in ReStructuredText should follow right
+after these lines. This comment is parsed and exported into the LTP
+documentation at https://linux-test-project.readthedocs.io/en/latest/users/test_catalog.html
+
+.. code-block:: shell
+
+   # ---
+   # doc
+   # Test for a foo bar.
+   #
+   # This test is testing foo by checking that bar is doing xyz.
+   # ---
+
+The shell loader test library uses the C test library internally by parsing a
+special JSON formatted comment and initializing it accordingly. The JSON format
+is nearly 1:1 serialization of the C tst_test structure into a JSON. The
+environment must be always preset even when it's empty.
+
+.. code-block:: shell
+
+   # ---
+   # env
+   # {
+   #  "needs_root": true,
+   #  "needs_tmpdir": true,
+   #  "needs_kconfigs": ["CONFIG_NUMA=y"],
+   #  "tags": {
+   #   ["linux-git", "432fd03240fa"]
+   #  }
+   # }
+
+After the documentation and environment has been laid out we finally import the
+`tst_loader.sh`. This will, among other things, start the `tst_run_shell`
+binary, that will parse the shell test environment comment and initialize the C
+test library accordingly.
+
+.. code-block:: shell
+
+   . tst_loader.sh
+
+At this point everything has been set up and we can finally write the test
+function. The test results are reported by the usual functions `tst_res` and
+`tst_brk`. As in the C API these functions store results into a piece of shared
+memory as soon as they return so there is no need to propagate results event
+from child processes.
+
+.. code-block:: shell
+
+   tst_test()
+   {
+        tst_res TPASS "Bar enabled Foo"
+   }
+
+In order for the test to be actually executed the very last line of the script
+must source the `tst_run.sh` script.
+
+.. code-block:: shell
+
+   . tst_run.sh
+
+In order to run a test from a LTP tree a few directories has to be added to the
+`$PATH`. Note that the number of `../` may depend on the depth of the current
+directory relative to the LTP root.
+
+.. code-block:: shell
+
+   $ PATH=$PATH:$PWD:$PWD/../../lib/ ./foo01.sh
+
+
+Test setup and cleanup
+----------------------
+
+The test setup and cleanup functions are optioinal and passed via variables.
+Similarily to the C API the setup is executed exactly once at the start of the
+test and the test cleanup is executed at the test end or when test was
+interrupted by `tst_brk`.
+
+.. literalinclude:: ../../testcases/lib/tests/shell_loader_setup_cleanup.sh
+   :language: shell
Signature · 5 lines
2.51.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 1/1
#1 lore raw

Hi Cyril,

doc/developers/api_shell_tests.rst line 89

Quoted 4 lines
doc/developers/api_shell_tests.rst +4 -0
+Test setup and cleanup
+----------------------
+
+The test setup and cleanup functions are optioinal and passed via variables.

s/optioinal/optional

Thanks,
Avinesh

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 1/1
#2 lore raw
In reply to Cyril Hrubis

Hi Cyril,

Thanks for this!

Few notes below.

Quoted 13 lines
doc/developers/api_shell_tests.rst +5 -0
index b6e8560d9..bad9f1f4b 100644
@@ -2,3 +2,94 @@
 
 LTP shell API
 =============
+
+Shell API overview
+------------------
+
+First lines of the shell test should be a shebang, a license, and copyrights.

doc/developers/api_shell_tests.rst line 19

Quoted 10 lines
doc/developers/api_shell_tests.rst +10 -0
+
+.. code-block:: shell
+
+   #!/bin/sh
+   # SPDX-License-Identifier: GPL-2.0-or-later
+   # Copyright 2099 Foo Bar <[email protected]>
+
+A documentation comment block formatted in ReStructuredText should follow right
+after these lines. This comment is parsed and exported into the LTP
+documentation at https://linux-test-project.readthedocs.io/en/latest/users/test_catalog.html

+1. Great, I did not realize it's ok to use at least #doc part for normal shell tests (i.e. these without shell loader).

doc/developers/api_shell_tests.rst line 29

Quoted 10 lines
doc/developers/api_shell_tests.rst +10 -0
+
+.. code-block:: shell
+
+   # ---
+   # doc
+   # Test for a foo bar.
+   #
+   # This test is testing foo by checking that bar is doing xyz.
+   # ---
+

I would frankly link the old API doc (and link :master:testcases/lib/tst_test.sh)...
https://github.com/linux-test-project/ltp/blob/master/doc/old/Shell-Test-API.asciidoc

doc/developers/api_shell_tests.rst line 33

Quoted 4 lines
doc/developers/api_shell_tests.rst +4 -0
+The shell loader test library uses the C test library internally by parsing a
+special JSON formatted comment and initializing it accordingly. The JSON format
+is nearly 1:1 serialization of the C tst_test structure into a JSON. The
+environment must be always preset even when it's empty.

And here would have subtitle "shell loader test library". Why? It'd be good to admit we have plain shell (which uses tst_test.sh) and shell loader. Of course, shell loader could be a mentioned first, then old API:

Shell Loader Test Library
-------------------------
The shell loader test library uses the C test library internally by parsing a special JSON formatted comment and initializing it accordingly. The JSON format is nearly 1:1 serialization of the C tst_test structure into a JSON. The environment must be always preset even when it's empty.

Legacy Shell Test Library
-------------------------
See :master:testcases/lib/tst_test.sh) and
https://github.com/linux-test-project/ltp/blob/master/doc/old/Shell-Test-API.asciidoc

.. warning::

:master:testcases/lib/tst_test.sh is deprecated, please use Shell Loader
Test Library.

Quoted 6 lines
doc/developers/api_shell_tests.rst +6 -0
+
+.. code-block:: shell
+
+   # ---
+   # env
+   # {

doc/developers/api_shell_tests.rst line 49

Quoted 10 lines
doc/developers/api_shell_tests.rst +10 -0
+   #  "needs_root": true,
+   #  "needs_tmpdir": true,
+   #  "needs_kconfigs": ["CONFIG_NUMA=y"],
+   #  "tags": {
+   #   ["linux-git", "432fd03240fa"]
+   #  }
+   # }
+
+After the documentation and environment has been laid out we finally import the
+`tst_loader.sh`. This will, among other things, start the `tst_run_shell`

Could we please instead `` use :master:relative path format which convert source to the link:

:master:testcases/lib/tst_loader.sh. This will, among other things, start the :master:testcases/lib/tst_run_shell.c ...

This will help people to look at the sources which is always helpful.

Quoted 47 lines
doc/developers/api_shell_tests.rst +46 -0
+binary, that will parse the shell test environment comment and initialize the C
+test library accordingly.
+
+.. code-block:: shell
+
+   . tst_loader.sh
+
+At this point everything has been set up and we can finally write the test
+function. The test results are reported by the usual functions `tst_res` and
+`tst_brk`. As in the C API these functions store results into a piece of shared
+memory as soon as they return so there is no need to propagate results event
+from child processes.
+
+.. code-block:: shell
+
+   tst_test()
+   {
+        tst_res TPASS "Bar enabled Foo"
+   }
+
+In order for the test to be actually executed the very last line of the script
+must source the `tst_run.sh` script.
+
+.. code-block:: shell
+
+   . tst_run.sh
+
+In order to run a test from a LTP tree a few directories has to be added to the
+`$PATH`. Note that the number of `../` may depend on the depth of the current
+directory relative to the LTP root.
+
+.. code-block:: shell
+
+   $ PATH=$PATH:$PWD:$PWD/../../lib/ ./foo01.sh
 
+
+
+Test setup and cleanup
+----------------------
+
+The test setup and cleanup functions are optioinal and passed via variables.
+Similarily to the C API the setup is executed exactly once at the start of the
+test and the test cleanup is executed at the test end or when test was
+interrupted by `tst_brk`.
+
+.. literalinclude:: ../../testcases/lib/tests/shell_loader_setup_cleanup.sh
+   :language: shell
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw
Quoted 9 lines
Quoted 9 lines
doc/developers/api_shell_tests.rst +9 -0
+Shell API overview
+------------------
+
+First lines of the shell test should be a shebang, a license, and copyrights.
+
+.. code-block:: shell
+
+   #!/bin/sh
+   # SPDX-License-Identifier: GPL-2.0-or-later
Quoted 8 lines

doc/developers/api_shell_tests.rst line 19

Quoted 5 lines
doc/developers/api_shell_tests.rst +5 -0
+   # Copyright 2099 Foo Bar <[email protected]>
+
+A documentation comment block formatted in ReStructuredText should follow right
+after these lines. This comment is parsed and exported into the LTP
+documentation at https://linux-test-project.readthedocs.io/en/latest/users/test_catalog.html

+1. Great, I did not realize it's ok to use at least #doc part for normal shell tests (i.e. these without shell loader).

That should indeed work fine. You can have a doc block in old shell test and it would be picked up by the metadata parser.

Quoted 14 lines

doc/developers/api_shell_tests.rst line 29

Quoted 10 lines
doc/developers/api_shell_tests.rst +10 -0
+
+.. code-block:: shell
+
+   # ---
+   # doc
+   # Test for a foo bar.
+   #
+   # This test is testing foo by checking that bar is doing xyz.
+   # ---
+

I would frankly link the old API doc (and link :master:testcases/lib/tst_test.sh)...
https://github.com/linux-test-project/ltp/blob/master/doc/old/Shell-Test-API.asciidoc

Quoted 8 lines

doc/developers/api_shell_tests.rst line 33

Quoted 4 lines
doc/developers/api_shell_tests.rst +4 -0
+The shell loader test library uses the C test library internally by parsing a
+special JSON formatted comment and initializing it accordingly. The JSON format
+is nearly 1:1 serialization of the C tst_test structure into a JSON. The
+environment must be always preset even when it's empty.

And here would have subtitle "shell loader test library". Why? It'd be good to admit we have plain shell (which uses tst_test.sh) and shell loader. Of course, shell loader could be a mentioned first, then old API:

Generally I would like to avoid mixing documentation for different APIs. I'm afraid that would make things unnecessarily confusing.

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
doc/developers/api_shell_tests.rst +9 -0
+Shell API overview
+------------------
+
+First lines of the shell test should be a shebang, a license, and copyrights.
+
+.. code-block:: shell
+
+   #!/bin/sh
+   # SPDX-License-Identifier: GPL-2.0-or-later
Quoted 8 lines

doc/developers/api_shell_tests.rst line 19

Quoted 5 lines
doc/developers/api_shell_tests.rst +5 -0
+   # Copyright 2099 Foo Bar <[email protected]>
+
+A documentation comment block formatted in ReStructuredText should follow right
+after these lines. This comment is parsed and exported into the LTP
+documentation at https://linux-test-project.readthedocs.io/en/latest/users/test_catalog.html

+1. Great, I did not realize it's ok to use at least #doc part for normal shell tests (i.e. these without shell loader).

Quoted 2 lines

That should indeed work fine. You can have a doc block in old shell test and it would be picked up by the metadata parser.

Yeah, it works (I tested it). I looked into metadata/parse.sh and it's due it searches for both *.c and *.sh without checking what library they use. Which is good (no side effects so far and we can include docs). I'll probably convert some of network tests docs into docparse to update the test catalog.

Quoted 14 lines
Quoted 13 lines

doc/developers/api_shell_tests.rst line 29

Quoted 10 lines
doc/developers/api_shell_tests.rst +10 -0
+
+.. code-block:: shell
+
+   # ---
+   # doc
+   # Test for a foo bar.
+   #
+   # This test is testing foo by checking that bar is doing xyz.
+   # ---
+

I would frankly link the old API doc (and link :master:testcases/lib/tst_test.sh)...
https://github.com/linux-test-project/ltp/blob/master/doc/old/Shell-Test-API.asciidoc

Quoted 1 line
Quoted 1 line
Quoted 1 line

+The shell loader test library uses the C test library internally by parsing a

Could we instead "the C test library" just use link:
:doc:../developers/api_c_tests

NOTE: I would avoid it, but you can set a custom text via <>
https://www.sphinx-doc.org/en/master/usage/referencing.html

Quoted 2 lines
Quoted 2 lines

doc/developers/api_shell_tests.rst line 32

Quoted 2 lines
doc/developers/api_shell_tests.rst +2 -0
+special JSON formatted comment and initializing it accordingly. The JSON format
+is nearly 1:1 serialization of the C tst_test structure into a JSON. The

And "C tst_test structure" as:
:ref:struct tst_test

The shell loader test library uses the :doc:../developers/api_c_tests internally by parsing a special JSON formatted comment and initializing it accordingly. The JSON format is nearly 1:1 serialization of the :ref:struct tst_test into a JSON. The environment must be always preset even when it's empty.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

+environment must be always preset even when it's empty.

And here would have subtitle "shell loader test library". Why? It'd be good to admit we have plain shell (which uses tst_test.sh) and shell loader. Of course, shell loader could be a mentioned first, then old API:

Generally I would like to avoid mixing documentation for different APIs. I'm afraid that would make things unnecessarily confusing.

OK, no problem (I did not get you want to document only Shell Loader). But hiding the fact that vast majority of the shell tests use tst_test.h is not optimal (it might be confusing for new users).

Also, Shell Loader does not have useful functions from tst_test.h, e.g. ROD(), EXPECT_PASS(). But ok, nobody needs them so far (although ROD() could be useful) and I would also ask people to use C API if possible.

Could we decide that tst_test.sh becomes "legacy/old" shell API which could be on a separate page doc/developers/legacy_api_shell_tests.rst which would just mention it's deprecated and linked:
https://github.com/linux-test-project/ltp/blob/master/doc/old/Shell-Test-API.asciidoc
(AI would convert it quickly to RST, but I'd be ok to keep just a link.)

And test.h becomes undocumented "legacy legacy" shell API (test.h is still used by few tests).

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw

Hi!

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

And here would have subtitle "shell loader test library". Why? It'd be good to admit we have plain shell (which uses tst_test.sh) and shell loader. Of course, shell loader could be a mentioned first, then old API:

Generally I would like to avoid mixing documentation for different APIs. I'm afraid that would make things unnecessarily confusing.

OK, no problem (I did not get you want to document only Shell Loader). But hiding the fact that vast majority of the shell tests use tst_test.h is not optimal (it might be confusing for new users).

I'm not trying to hide it. I'm trying to write at least some documentation for the shell loader, that is independed task from converting the documentation we have for the older shell library into rst and putting it in it's own menu item.

Quoted 3 lines

Also, Shell Loader does not have useful functions from tst_test.h, e.g. ROD(), EXPECT_PASS(). But ok, nobody needs them so far (although ROD() could be useful) and I would also ask people to use C API if possible.

Indeed the shell loader is currently under construction, but it allows us to write a simple tests already and should be good enough for simple enough tests.

Quoted 5 lines

Could we decide that tst_test.sh becomes "legacy/old" shell API which could be on a separate page doc/developers/legacy_api_shell_tests.rst which would just mention it's deprecated and linked:
https://github.com/linux-test-project/ltp/blob/master/doc/old/Shell-Test-API.asciidoc
(AI would convert it quickly to RST, but I'd be ok to keep just a link.)

Yes, that is what I think would be ideal solution, much better than mixing the two APIs in a single page.

Quoted 2 lines

And test.h becomes undocumented "legacy legacy" shell API (test.h is still used by few tests).

You mean 'test.sh' I suppose, yes there are a few tests using it. We should clean that up eventually.

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw
Quoted 8 lines

Hi!

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

And here would have subtitle "shell loader test library". Why? It'd be good to admit we have plain shell (which uses tst_test.sh) and shell loader. Of course, shell loader could be a mentioned first, then old API:

Generally I would like to avoid mixing documentation for different APIs. I'm afraid that would make things unnecessarily confusing.

Quoted 8 lines
Quoted 3 lines

OK, no problem (I did not get you want to document only Shell Loader). But hiding the fact that vast majority of the shell tests use tst_test.h is not optimal (it might be confusing for new users).

I'm not trying to hide it. I'm trying to write at least some documentation for the shell loader, that is independed task from converting the documentation we have for the older shell library into rst and putting it in it's own menu item.

Go ahead and merge as is. Actually the only my objections were that previously "Shell API" was tst_test.sh, now it's shell loader. I wanted to have the same name "Shell Loader" we have used in git commit logs to match the docs to not confuse users which API we mean (we now have 3 shell APIs). But I'm ok with the current version.

Quoted 7 lines
Quoted 3 lines

Also, Shell Loader does not have useful functions from tst_test.h, e.g. ROD(), EXPECT_PASS(). But ok, nobody needs them so far (although ROD() could be useful) and I would also ask people to use C API if possible.

Indeed the shell loader is currently under construction, but it allows us to write a simple tests already and should be good enough for simple enough tests.

Sure. I just realized that we might need slightly extend the API during rewrite, but that's of course not a problem.

Quoted 8 lines
Quoted 5 lines

Could we decide that tst_test.sh becomes "legacy/old" shell API which could be on a separate page doc/developers/legacy_api_shell_tests.rst which would just mention it's deprecated and linked:
https://github.com/linux-test-project/ltp/blob/master/doc/old/Shell-Test-API.asciidoc
(AI would convert it quickly to RST, but I'd be ok to keep just a link.)

Yes, that is what I think would be ideal solution, much better than mixing the two APIs in a single page.

OK, I'll try to convert it soon, under doc/developers/legacy_api_shell_tests.rst. There will be a warning - use new "Shell API (which is Shell loader).

Quoted 5 lines
Quoted 2 lines

And test.h becomes undocumented "legacy legacy" shell API (test.h is still used by few tests).

You mean 'test.sh' I suppose, yes there are a few tests using it. We should clean that up eventually.

Yes, I'm sorry, I meant test.sh.

FYI we need to rewrite 52 tests:

$ git grep -l -e ' test\.sh' -e TST_USE_LEGACY_API=1 |wc -l
52

Some of them are very obsolete tests, e.g. power_management, smack, ftrace, ... I guess that is what holds the rewrite.

9 of them are network tests (ftp/http should be rewritten:
https://github.com/linux-test-project/ltp/issues/1207),
the rest would be trivial to convert even to shell loader.

$ git grep -l TST_USE_LEGACY_API=1 
testcases/network/multicast/mc_cmds/mc_cmds.sh
testcases/network/multicast/mc_commo/mc_commo.sh
testcases/network/multicast/mc_member/mc_member.sh
testcases/network/multicast/mc_opts/mc_opts.sh
testcases/network/stress/dns/dns-stress.sh
testcases/network/stress/ftp/ftp-download-stress.sh
testcases/network/stress/ftp/ftp-upload-stress.sh
testcases/network/stress/http/http-stress.sh
testcases/network/tcp_cmds/tcpdump/tcpdump01.sh

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#7 lore raw
In reply to Cyril Hrubis
Quoted 14 lines
---
 doc/developers/api_shell_tests.rst | 91 ++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
doc/developers/api_shell_tests.rst +1 -0
index b6e8560d9..bad9f1f4b 100644
@@ -2,3 +2,94 @@
 
 LTP shell API
 =============
+

doc/developers/api_shell_tests.rst line 15

Quoted 10 lines
doc/developers/api_shell_tests.rst +10 -0
+Shell API overview
+------------------
+
+First lines of the shell test should be a shebang, a license, and copyrights.
+
+.. code-block:: shell
+
+   #!/bin/sh
+   # SPDX-License-Identifier: GPL-2.0-or-later
+   # Copyright 2099 Foo Bar <[email protected]>

very nit: maybe [email protected]?

...

Quoted 5 lines
doc/developers/api_shell_tests.rst +5 -0
+At this point everything has been set up and we can finally write the test
+function. The test results are reported by the usual functions `tst_res` and
+`tst_brk`. As in the C API these functions store results into a piece of shared
+memory as soon as they return so there is no need to propagate results event
+from child processes.

doc/developers/api_shell_tests.rst line 71

Quoted 10 lines
doc/developers/api_shell_tests.rst +10 -0
+
+.. code-block:: shell
+
+   tst_test()
+   {
+        tst_res TPASS "Bar enabled Foo"
+   }
+
+In order for the test to be actually executed the very last line of the script
+must source the `tst_run.sh` script.

Also this could be :master:testcases/lib/tst_run.sh.

Of course, all the changes can be done before merging (no need to post v2).

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp