babeltrace tests (master/2.0)
Ping?
Quoted 75 lines · Nathan Lynch
How is the babeltrace test suite on the master and stable-2.0 branches intended to be used?
Effectively there are four test suites, one for each definition of TESTS in the tests/ directory. If one of them has a failure, 'make check' will not proceed to the next unless the -k/--keep-going flag is used. Is this intended? I unwittingly was running only the tests under tests/cli (some of which always fail) until I discovered this.
Defining multiple test suites this way seems to be unusual; other projects I've checked just have one top-level definition of TESTS.
I'm aware of the rationale for doing it this way:
commit
6ca1931cb32ca2eb33252896d2a42a4c48af436aAuthor: Philippe Proulx <[email protected]> Date: Fri May 5 16:45:11 2017 -0400tests: put TESTS list in each Makefile.am
So that you can do
make checkin any subdirectory to run only those tests.But it still feels non-idiomatic to me, and I'm wondering if there's a better way. There are other ways to limit the set of tests to run, detailed here:
https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html
Here are my current results from running all test suites; let me know if you want more detail and I'll follow up; there are likely several different issues to investigate.
$ find tests/ -name test-suite.log -exec head {} + ==> tests/lib/test-suite.log <== ===================================================== babeltrace 2.0.0-pre1: tests/lib/test-suite.log =====================================================# TOTAL: 1200
# PASS: 1195
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0==> tests/bindings/python/bt2/test-suite.log <==
=====================================================================
babeltrace 2.0.0-pre1: tests/bindings/python/bt2/test-suite.log
=====================================================================# TOTAL: 11152
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 11151
# XPASS: 0==> tests/cli/test-suite.log <==
=====================================================
babeltrace 2.0.0-pre1: tests/cli/test-suite.log
=====================================================# TOTAL: 153
# PASS: 143
# SKIP: 0
# XFAIL: 0
# FAIL: 7
# XPASS: 0==> tests/plugins/test-suite.log <==
=========================================================
babeltrace 2.0.0-pre1: tests/plugins/test-suite.log
=========================================================
Quoted 8 lines
# TOTAL: 2
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0(BTW, since the tests don't pass/skip/xfail 100%, 'make distcheck' is failing.)
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Quoted 3 lines · Nathan Lynch
Ping?
Sorry. We looked into this. There seems to be no easy way with Automake to have individual check targets in subdirectories and make the top-level check target recurse without stopping when a subdirectory fails.
As you wrote, it is possible with --keep-going, but this is not common practice and we don't expect our users to read make(1) to be able to run all the tests unconditionally.
I asked the Automake mailing list about this
<http://lists.gnu.org/archive/html/automake/2017-08/msg00004.html> and we will most probably implement the suggested solution, which is to have individual top-level (in tests/) check-X targets which rerun Make with specific sets of tests.
Thank you for reporting this by the way.
Phil
Quoted 84 lines
Quoted 82 lines · Nathan Lynch
How is the babeltrace test suite on the master and stable-2.0 branches intended to be used?
Effectively there are four test suites, one for each definition of TESTS in the tests/ directory. If one of them has a failure, 'make check' will not proceed to the next unless the -k/--keep-going flag is used. Is this intended? I unwittingly was running only the tests under tests/cli (some of which always fail) until I discovered this.
Defining multiple test suites this way seems to be unusual; other projects I've checked just have one top-level definition of TESTS.
I'm aware of the rationale for doing it this way:
commit
6ca1931cb32ca2eb33252896d2a42a4c48af436aAuthor: Philippe Proulx <[email protected]> Date: Fri May 5 16:45:11 2017 -0400tests: put TESTS list in each Makefile.am
So that you can do
make checkin any subdirectory to run only those tests.But it still feels non-idiomatic to me, and I'm wondering if there's a better way. There are other ways to limit the set of tests to run, detailed here:
https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html
Here are my current results from running all test suites; let me know if you want more detail and I'll follow up; there are likely several different issues to investigate.
$ find tests/ -name test-suite.log -exec head {} + ==> tests/lib/test-suite.log <== ===================================================== babeltrace 2.0.0-pre1: tests/lib/test-suite.log =====================================================# TOTAL: 1200
# PASS: 1195
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0==> tests/bindings/python/bt2/test-suite.log <==
=====================================================================
babeltrace 2.0.0-pre1: tests/bindings/python/bt2/test-suite.log
=====================================================================# TOTAL: 11152
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 11151
# XPASS: 0==> tests/cli/test-suite.log <==
=====================================================
babeltrace 2.0.0-pre1: tests/cli/test-suite.log
=====================================================# TOTAL: 153
# PASS: 143
# SKIP: 0
# XFAIL: 0
# FAIL: 7
# XPASS: 0==> tests/plugins/test-suite.log <==
=========================================================
babeltrace 2.0.0-pre1: tests/plugins/test-suite.log
=========================================================# TOTAL: 2
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
Quoted 5 lines
Quoted 1 line
(BTW, since the tests don't pass/skip/xfail 100%, 'make distcheck' is failing.)
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Quoted 14 lines · Philippe Proulx
Quoted 3 lines · Nathan Lynch
Ping?
Sorry. We looked into this. There seems to be no easy way with Automake to have individual
checktargets in subdirectories and make the top-levelchecktarget recurse without stopping when a subdirectory fails.As you wrote, it is possible with --keep-going, but this is not common practice and we don't expect our users to read make(1) to be able to run all the tests unconditionally.
Quoted 7 lines
I asked the Automake mailing list about this
<http://lists.gnu.org/archive/html/automake/2017-08/msg00004.html> and we will most probably implement the suggested solution, which is to have individual top-level (in tests/)check-Xtargets which rerun Make with specific sets of tests.Thank you for reporting this by the way.
Thanks for following up. A well-functioning test suite is a really useful guide for occasional contributors like me :-)
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Quoted 11 lines · Nathan Lynch
Quoted 7 lines · Nathan Lynch
Quoted 6 lines · Philippe Proulx
I asked the Automake mailing list about this
<http://lists.gnu.org/archive/html/automake/2017-08/msg00004.html> and we will most probably implement the suggested solution, which is to have individual top-level (in tests/)check-Xtargets which rerun Make with specific sets of tests.I should be able to implement this over the next week or so if you don't anticipate conflicts with other ongoing work. What do you say?
It's been merged into master a few minutes ago. Here's the commit:
https://github.com/efficios/babeltrace/commit/cd1c5347f7583a20ece5a1a62fc0e0eddb82322b
Philippe Proulx
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Quoted 7 lines · Nathan Lynch
Quoted 6 lines · Philippe Proulx
I asked the Automake mailing list about this
<http://lists.gnu.org/archive/html/automake/2017-08/msg00004.html> and we will most probably implement the suggested solution, which is to have individual top-level (in tests/)check-Xtargets which rerun Make with specific sets of tests.
I should be able to implement this over the next week or so if you don't anticipate conflicts with other ongoing work. What do you say?
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Quoted 10 lines · Philippe Proulx
Quoted 9 lines · Nathan Lynch
Quoted 7 lines · Nathan Lynch
Quoted 6 lines · Philippe Proulx
I asked the Automake mailing list about this
<http://lists.gnu.org/archive/html/automake/2017-08/msg00004.html> and we will most probably implement the suggested solution, which is to have individual top-level (in tests/)check-Xtargets which rerun Make with specific sets of tests.
Quoted 7 lines
Quoted 2 lines
I should be able to implement this over the next week or so if you don't anticipate conflicts with other ongoing work. What do you say?
It's been merged into master a few minutes ago. Here's the commit:
https://github.com/efficios/babeltrace/commit/cd1c5347f7583a20ece5a1a62fc0e0eddb82322b
Thanks!
Is this the desired result:
PASS: cli/test_trace_read
PASS: cli/test_packet_seq_num
PASS: cli/test_convert_args
PASS: cli/intersection/test_intersection
PASS: cli/test_trace_copy
PASS: cli/test_trimmer
PASS: lib/test_bitfield
PASS: lib/test_ctf_writer_complete
PASS: lib/test_bt_values
PASS: lib/test_ctf_ir_ref
PASS: lib/test_bt_ctf_field_type_validation
PASS: lib/test_ir_visit
PASS: lib/test_bt_notification_heap
PASS: lib/test_graph_topo
PASS: lib/test_cc_prio_map
PASS: lib/test_bt_notification_iterator
PASS: lib/test_plugin_complete
PASS: plugins/test-utils-muxer-complete
PASS: plugins/test_dwarf_complete
PASS: plugins/test_bin_info_complete
============================================================================
Testsuite summary for babeltrace 2.0.0-pre1
============================================================================
# TOTAL: 20
# PASS: 20
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
The granularity of results seems very much reduced; there had been ~1200 individual tests run before.
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev