Skip to contentSkip to conversation

babeltrace tests (master/2.0)

DISCUSSION Nathan Lynch 2 participants 6 messages Aug 10, 2017 – Sep 18, 2017 6 messages not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

Ping?

Quoted 75 lines · Nathan Lynch

How is the babeltrace test suite on the master and stable-2.0 branches intended to be used?

Effectively there are four test suites, one for each definition of TESTS in the tests/ directory. If one of them has a failure, 'make check' will not proceed to the next unless the -k/--keep-going flag is used. Is this intended? I unwittingly was running only the tests under tests/cli (some of which always fail) until I discovered this.

Defining multiple test suites this way seems to be unusual; other projects I've checked just have one top-level definition of TESTS.

I'm aware of the rationale for doing it this way:

commit 6ca1931cb32ca2eb33252896d2a42a4c48af436a

Author: Philippe Proulx <[email protected]>
Date:   Fri May 5 16:45:11 2017 -0400

tests: put TESTS list in each Makefile.am

So that you can do make check in any subdirectory to run only those tests.

But it still feels non-idiomatic to me, and I'm wondering if there's a better way. There are other ways to limit the set of tests to run, detailed here:

https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html

Here are my current results from running all test suites; let me know if you want more detail and I'll follow up; there are likely several different issues to investigate.

$ find tests/ -name test-suite.log -exec head {} +          
==> tests/lib/test-suite.log <==
=====================================================
   babeltrace 2.0.0-pre1: tests/lib/test-suite.log
=====================================================

# TOTAL: 1200
# PASS: 1195
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0

==> tests/bindings/python/bt2/test-suite.log <==
=====================================================================
babeltrace 2.0.0-pre1: tests/bindings/python/bt2/test-suite.log
=====================================================================

# TOTAL: 11152
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 11151
# XPASS: 0

==> tests/cli/test-suite.log <==
=====================================================
babeltrace 2.0.0-pre1: tests/cli/test-suite.log
=====================================================

# TOTAL: 153
# PASS: 143
# SKIP: 0
# XFAIL: 0
# FAIL: 7
# XPASS: 0

==> tests/plugins/test-suite.log <==
=========================================================
babeltrace 2.0.0-pre1: tests/plugins/test-suite.log
=========================================================

Quoted 8 lines

# TOTAL: 2
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0

(BTW, since the tests don't pass/skip/xfail 100%, 'make distcheck' is failing.)

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#3 lore raw
In reply to a message not in the archive
Quoted 3 lines · Nathan Lynch

Ping?

Sorry. We looked into this. There seems to be no easy way with Automake to have individual check targets in subdirectories and make the top-level check target recurse without stopping when a subdirectory fails.

As you wrote, it is possible with --keep-going, but this is not common practice and we don't expect our users to read make(1) to be able to run all the tests unconditionally.

I asked the Automake mailing list about this
<http://lists.gnu.org/archive/html/automake/2017-08/msg00004.html> and we will most probably implement the suggested solution, which is to have individual top-level (in tests/) check-X targets which rerun Make with specific sets of tests.

Thank you for reporting this by the way.

Phil

Quoted 84 lines
Quoted 82 lines · Nathan Lynch

How is the babeltrace test suite on the master and stable-2.0 branches intended to be used?

Effectively there are four test suites, one for each definition of TESTS in the tests/ directory. If one of them has a failure, 'make check' will not proceed to the next unless the -k/--keep-going flag is used. Is this intended? I unwittingly was running only the tests under tests/cli (some of which always fail) until I discovered this.

Defining multiple test suites this way seems to be unusual; other projects I've checked just have one top-level definition of TESTS.

I'm aware of the rationale for doing it this way:

commit 6ca1931cb32ca2eb33252896d2a42a4c48af436a

Author: Philippe Proulx <[email protected]>
Date:   Fri May 5 16:45:11 2017 -0400

tests: put TESTS list in each Makefile.am

So that you can do make check in any subdirectory to run only those tests.

But it still feels non-idiomatic to me, and I'm wondering if there's a better way. There are other ways to limit the set of tests to run, detailed here:

https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html

Here are my current results from running all test suites; let me know if you want more detail and I'll follow up; there are likely several different issues to investigate.

$ find tests/ -name test-suite.log -exec head {} +
==> tests/lib/test-suite.log <==
=====================================================
   babeltrace 2.0.0-pre1: tests/lib/test-suite.log
=====================================================

# TOTAL: 1200
# PASS: 1195
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0

==> tests/bindings/python/bt2/test-suite.log <==
=====================================================================
babeltrace 2.0.0-pre1: tests/bindings/python/bt2/test-suite.log
=====================================================================

# TOTAL: 11152
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 11151
# XPASS: 0

==> tests/cli/test-suite.log <==
=====================================================
babeltrace 2.0.0-pre1: tests/cli/test-suite.log
=====================================================

# TOTAL: 153
# PASS: 143
# SKIP: 0
# XFAIL: 0
# FAIL: 7
# XPASS: 0

==> tests/plugins/test-suite.log <==
=========================================================
babeltrace 2.0.0-pre1: tests/plugins/test-suite.log
=========================================================

# TOTAL: 2
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0

Quoted 5 lines
Quoted 1 line

(BTW, since the tests don't pass/skip/xfail 100%, 'make distcheck' is failing.)

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#5 lore raw
In reply to a message not in the archive
Quoted 14 lines · Philippe Proulx
Quoted 3 lines · Nathan Lynch

Ping?

Sorry. We looked into this. There seems to be no easy way with Automake to have individual check targets in subdirectories and make the top-level check target recurse without stopping when a subdirectory fails.

As you wrote, it is possible with --keep-going, but this is not common practice and we don't expect our users to read make(1) to be able to run all the tests unconditionally.

Quoted 7 lines

I asked the Automake mailing list about this
<http://lists.gnu.org/archive/html/automake/2017-08/msg00004.html> and we will most probably implement the suggested solution, which is to have individual top-level (in tests/) check-X targets which rerun Make with specific sets of tests.

Thank you for reporting this by the way.

Thanks for following up. A well-functioning test suite is a really useful guide for occasional contributors like me :-)
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#8 lore raw
In reply to a message not in the archive
Quoted 11 lines · Nathan Lynch
Quoted 7 lines · Nathan Lynch
Quoted 6 lines · Philippe Proulx

I asked the Automake mailing list about this
<http://lists.gnu.org/archive/html/automake/2017-08/msg00004.html> and we will most probably implement the suggested solution, which is to have individual top-level (in tests/) check-X targets which rerun Make with specific sets of tests.

I should be able to implement this over the next week or so if you don't anticipate conflicts with other ongoing work. What do you say?

It's been merged into master a few minutes ago. Here's the commit:

    https://github.com/efficios/babeltrace/commit/cd1c5347f7583a20ece5a1a62fc0e0eddb82322b

Philippe Proulx
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#11 lore raw
In reply to a message not in the archive
Quoted 7 lines · Nathan Lynch
Quoted 6 lines · Philippe Proulx

I asked the Automake mailing list about this
<http://lists.gnu.org/archive/html/automake/2017-08/msg00004.html> and we will most probably implement the suggested solution, which is to have individual top-level (in tests/) check-X targets which rerun Make with specific sets of tests.

I should be able to implement this over the next week or so if you don't anticipate conflicts with other ongoing work. What do you say?
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#10 lore raw
In reply to a message not in the archive
Quoted 10 lines · Philippe Proulx
Quoted 9 lines · Nathan Lynch
Quoted 7 lines · Nathan Lynch
Quoted 6 lines · Philippe Proulx

I asked the Automake mailing list about this
<http://lists.gnu.org/archive/html/automake/2017-08/msg00004.html> and we will most probably implement the suggested solution, which is to have individual top-level (in tests/) check-X targets which rerun Make with specific sets of tests.

Quoted 7 lines
Quoted 2 lines

I should be able to implement this over the next week or so if you don't anticipate conflicts with other ongoing work. What do you say?

It's been merged into master a few minutes ago. Here's the commit:

    https://github.com/efficios/babeltrace/commit/cd1c5347f7583a20ece5a1a62fc0e0eddb82322b

Thanks!

Is this the desired result:

PASS: cli/test_trace_read
PASS: cli/test_packet_seq_num
PASS: cli/test_convert_args
PASS: cli/intersection/test_intersection
PASS: cli/test_trace_copy
PASS: cli/test_trimmer
PASS: lib/test_bitfield
PASS: lib/test_ctf_writer_complete
PASS: lib/test_bt_values
PASS: lib/test_ctf_ir_ref
PASS: lib/test_bt_ctf_field_type_validation
PASS: lib/test_ir_visit
PASS: lib/test_bt_notification_heap
PASS: lib/test_graph_topo
PASS: lib/test_cc_prio_map
PASS: lib/test_bt_notification_iterator
PASS: lib/test_plugin_complete
PASS: plugins/test-utils-muxer-complete
PASS: plugins/test_dwarf_complete
PASS: plugins/test_bin_info_complete
============================================================================
Testsuite summary for babeltrace 2.0.0-pre1
============================================================================
# TOTAL: 20
# PASS: 20
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0

The granularity of results seems very much reduced; there had been ~1200 individual tests run before.
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev