Test framework improvment - Do not FAIL if t_client.rc is missing
PATCH 1/1 Test framework improvment - Do not FAIL if t_client.rc is missing
This is to avoid 'make check' fail if this extra test case is not configured. The patch gives a PASS, but will notify the user that this test was skipped.
--- t_client.sh.in | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-)
t_client.sh.in +10 -10
@@ -12,6 +12,16 @@ # - for "ping6" checks: fping6 binary in $PATH # +if [ -r ./t_client.rc ] ; then + . ./t_client.rc +elif [ -r "${srcdir}"/t_client.rc ] ; then + . "${srcdir}"/t_client.rc +else + echo "$0: cannot find 't_client.rc' in current directory or" >&2 + echo "$0: source dir ('${srcdir}'). SKIPPING TEST." >&2 + exit 0 +fi + if [ ! -x ./openvpn ] then echo "no (executable) openvpn binary in current directory. FAIL." >&2
@@ -24,16 +34,6 @@ then exit 1 fi -if [ -r ./t_client.rc ] ; then - . ./t_client.rc -elif [ -r "${srcdir}"/t_client.rc ] ; then - . "${srcdir}"/t_client.rc -else - echo "cannot find 't_client.rc' in current directory or" >&2 - echo "source dir ('${srcdir}'). FAIL." >&2 - exit 1 -fi - if [ -z "$CA_CERT" ] ; then echo "CA_CERT not defined in 't_client.rc'. SKIP test." >&2 exit 0
Signature · 1 line
1.7.2.1
Am 18.08.2010 14:27, schrieb David Sommerseth:
Quoted 3 lines
This is to avoid 'make check' fail if this extra test case is not configured. The patch gives a PASS, but will notify the user that this test was skipped.
The right approach would be to "exit 77" instead of 0,
so that "make check" can report the SKIP properly. Assuming that t_client.sh is listed in TESTS in some Makefile.am, which I didn't check.
Quoted 44 lines
Signed-off-by David Sommerseth <[email protected]...>--- t_client.sh.in | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-)t_client.sh.in +10
-10@@ -12,6 +12,16 @@ # - for "ping6" checks: fping6 binary in $PATH # +if [ -r ./t_client.rc ] ; then + . ./t_client.rc +elif [ -r "${srcdir}"/t_client.rc ] ; then + . "${srcdir}"/t_client.rc +else + echo "$0: cannot find 't_client.rc' in current directory or" >&2 + echo "$0: source dir ('${srcdir}'). SKIPPING TEST." >&2 + exit 0 +fi + if [ ! -x ./openvpn ] then echo "no (executable) openvpn binary in current directory. FAIL." >&2@@ -24,16 +34,6 @@ then exit 1 fi -if [ -r ./t_client.rc ] ; then - . ./t_client.rc -elif [ -r "${srcdir}"/t_client.rc ] ; then - . "${srcdir}"/t_client.rc -else - echo "cannot find 't_client.rc' in current directory or" >&2 - echo "source dir ('${srcdir}'). FAIL." >&2 - exit 1 -fi - if [ -z "$CA_CERT" ] ; then echo "CA_CERT not defined in 't_client.rc'. SKIP test." >&2 exit 0
Signature · 1 line
Matthias Andree
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 9 lines · Matthias Andree
Am 18.08.2010 14:27, schrieb David Sommerseth:
Quoted 3 lines
This is to avoid 'make check' fail if this extra test case is not configured. The patch gives a PASS, but will notify the user that this test was skipped.
The right approach would be to "exit 77" instead of 0,
so that "make check" can report the SKIP properly. Assuming thatt_client.shis listed in TESTS in some Makefile.am, which I didn't check.
Ahh! There it was! I thought I had seen this before, but I couldn't find any good references in the automake manual so far. I'll fix this and send a new patch.
Thanks a lot!
kind regards,
David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAkxr2Q4ACgkQDC186MBRfrrqxwCffcH35sVufO3GUaTueTQeuPW8 nXIAnRw9OJcVScGpEdGn2tYyBanpCy5w
=pE1H
-----END PGP SIGNATURE-----
Am 18.08.2010 14:58, schrieb David Sommerseth:
Quoted 6 lines · Matthias Andree
Quoted 4 lines
Am 18.08.2010 14:27, schrieb David Sommerseth:
Quoted 3 lines
This is to avoid 'make check' fail if this extra test case is not configured. The patch gives a PASS, but will notify the user that this test was skipped.
Quoted 8 lines
Quoted 3 lines
The right approach would be to "exit 77" instead of 0,
so that "make check" can report the SKIP properly. Assuming thatt_client.shis listed in TESTS in some Makefile.am, which I didn't check.Ahh! There it was! I thought I had seen this before, but I couldn't find any good references in the automake manual so far. I'll fix this and send a new patch.
In automake 1.10, it's in chapter 15.1 "Simple Tests" in the second paragraph:
| The number of failures will be printed at the end of the run. If a | given test program exits with a status of 77, then its result is ignored | in the final count. This feature allows non-portable tests to be | ignored in environments where they don't make sense.
Which is not exactly stating the more common use case, as the whole chapter is rather compact... The feature is still suitable.
On a related node, I personally prefer printing a message how a test could be enabled, followed by exit 77, over removing unrunnable tests from the TESTS variable. See bogofilter.sourceforge.net (either use SVN, or a recent tarball), particularly src/tests{/Makefile.am|/*} and doc/Makefile.am for an example of a more elaborate test suite.
Not necessarily the best, and there are more powerful frameworks than what we use, but it also shows how far the simple automake tests can be taken already, and perhaps some ideas can be useful.
Signature · 1 line
Matthias Andree
PATCH v2 1/1 Test framework improvment - Do not FAIL if t_client.rc is missing
In reply to Matthias AndreeThis is to avoid 'make check' fail if this extra test case is not configured.
--- t_client.sh.in | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-)
t_client.sh.in +10 -10
@@ -12,6 +12,16 @@ # - for "ping6" checks: fping6 binary in $PATH # +if [ -r ./t_client.rc ] ; then + . ./t_client.rc +elif [ -r "${srcdir}"/t_client.rc ] ; then + . "${srcdir}"/t_client.rc +else + echo "$0: cannot find 't_client.rc' in current directory or" >&2 + echo "$0: source dir ('${srcdir}'). SKIPPING TEST." >&2 + exit 77 +fi + if [ ! -x ./openvpn ] then echo "no (executable) openvpn binary in current directory. FAIL." >&2
@@ -24,16 +34,6 @@ then exit 1 fi -if [ -r ./t_client.rc ] ; then - . ./t_client.rc -elif [ -r "${srcdir}"/t_client.rc ] ; then - . "${srcdir}"/t_client.rc -else - echo "cannot find 't_client.rc' in current directory or" >&2 - echo "source dir ('${srcdir}'). FAIL." >&2 - exit 1 -fi - if [ -z "$CA_CERT" ] ; then echo "CA_CERT not defined in 't_client.rc'. SKIP test." >&2 exit 0
Signature · 1 line
1.7.2.1
Am 18.08.2010 15:04, schrieb David Sommerseth:
Quoted 16 lines
This is to avoid 'make check' fail if this extra test case is not configured.
Signed-off-by David Sommerseth <[email protected]...>--- t_client.sh.in | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-)t_client.sh.in +0
-0@@ -12,6 +12,16 @@ # - for "ping6" checks: fping6 binary in $PATH #
t_client.sh.in line 24
Quoted 10 lines
t_client.sh.in +10
-0+if [ -r ./t_client.rc ] ; then + . ./t_client.rc +elif [ -r "${srcdir}"/t_client.rc ] ; then + . "${srcdir}"/t_client.rc +else + echo "$0: cannot find 't_client.rc' in current directory or" >&2 + echo "$0: source dir ('${srcdir}'). SKIPPING TEST." >&2 + exit 77 +fi +
I have occasionally been bitten by such ambiguities between current working directory and $srcdir. I'd propose to steer clear of such precedence rules.
Quoted 10 lines
t_client.sh.in +0
-3if [ ! -x ./openvpn ] then echo "no (executable) openvpn binary in current directory. FAIL." >&2@@ -24,16 +34,6 @@ then exit 1 fi -if [ -r ./t_client.rc ] ; then - . ./t_client.rc -elif [ -r "${srcdir}"/t_client.rc ] ; then
t_client.sh.in then · line 39
Quoted 10 lines
t_client.sh.in +0
-7- . "${srcdir}"/t_client.rc -else - echo "cannot find 't_client.rc' in current directory or" >&2 - echo "source dir ('${srcdir}'). FAIL." >&2 - exit 1 -fi - if [ -z "$CA_CERT" ] ; then echo "CA_CERT not defined in 't_client.rc'. SKIP test." >&2 exit 0
I think you missed this "exit 0" - should also be 77 instead - and please check if there are more (I've not been looking at the source script for lack of time).
Signature · 1 line
Matthias Andree
Hi,
Quoted 4 lines · Matthias Andree
Quoted 4 lines
t_client.sh.in +4
-0+if [ -r ./t_client.rc ] ; then + . ./t_client.rc +elif [ -r "${srcdir}"/t_client.rc ] ; then + . "${srcdir}"/t_client.rc
Quoted 8 lines
t_client.sh.in line 23
Quoted 5 lines
t_client.sh.in +5
-0+else + echo "$0: cannot find 't_client.rc' in current directory or" >&2 + echo "$0: source dir ('${srcdir}'). SKIPPING TEST." >&2 + exit 77 +fiI have occasionally been bitten by such ambiguities between current working directory and $srcdir. I'd propose to steer clear of such precedence rules.
I'm using this on purpose here - I have a number of machines that build using the same $srcdir (mounted over NFS), and one of them has no working IPv6-over-tun. So this special machine has a t_client.rc in its build directory which sources $srcdir/t_client.rc and subsequently unsets all IPv6-related test targets.
So all machines can use the same generic t_client.rc, and if I need to fiddle with things on individual hosts, I can do so, without having to maintain a "nearly identical" copy on all machines.
Quoted 6 lines
Quoted 3 lines
if [ -z "$CA_CERT" ] ; then
echo "CA_CERTnot defined in 't_client.rc'. SKIP test." >&2 exit 0I think you missed this "exit 0" - should also be 77 instead - and please check if there are more (I've not been looking at the source script for lack of time).
There are some more.
David, could you patch these as well? Otherwise I need to go cherrypicking your changes first (and this might take longer).
gert
Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany [email protected]...
fax: +49-89-35655025 [email protected]...-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 17 lines · Matthias Andree
Am 18.08.2010 15:04, schrieb David Sommerseth:
Quoted 16 lines
This is to avoid 'make check' fail if this extra test case is not configured.
Signed-off-by David Sommerseth <[email protected]...>--- t_client.sh.in | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-)t_client.sh.in +0
-0@@ -12,6 +12,16 @@ # - for "ping6" checks: fping6 binary in $PATH #
Quoted 13 lines
t_client.sh.in line 24
Quoted 10 lines
t_client.sh.in +10
-0+if [ -r ./t_client.rc ] ; then + . ./t_client.rc +elif [ -r "${srcdir}"/t_client.rc ] ; then + . "${srcdir}"/t_client.rc +else + echo "$0: cannot find 't_client.rc' in current directory or" >&2 + echo "$0: source dir ('${srcdir}'). SKIPPING TEST." >&2 + exit 77 +fi +I have occasionally been bitten by such ambiguities between current working directory and $srcdir. I'd propose to steer clear of such precedence rules.
I'll consider a fix here, or maybe Gert got some good ideas as well. Anyway, I think that should be covered in another patch, where this issue is tackled separately.
Quoted 7 lines
Quoted 7 lines
t_client.sh.in +0
-0if [ ! -x ./openvpn ] then echo "no (executable) openvpn binary in current directory. FAIL." >&2@@ -24,16 +34,6 @@ then exit 1 fi
Quoted 16 lines
t_client.sh.in then · line 39
Quoted 13 lines
t_client.sh.in +0
-10-if [ -r ./t_client.rc ] ; then - . ./t_client.rc -elif [ -r "${srcdir}"/t_client.rc ] ; then - . "${srcdir}"/t_client.rc -else - echo "cannot find 't_client.rc' in current directory or" >&2 - echo "source dir ('${srcdir}'). FAIL." >&2 - exit 1 -fi - if [ -z "$CA_CERT" ] ; then echo "CA_CERT not defined in 't_client.rc'. SKIP test." >&2 exit 0I think you missed this "exit 0" - should also be 77 instead - and please check if there are more (I've not been looking at the source script for lack of time).
Yeah, agreed. I'm not doing that change in this patch. But I've submitted a separate patch fixing all the SKIP places.
Thanks a lot for valuable input!
kind regards,
David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAkxr92wACgkQDC186MBRfrrzfgCbBS1r4GuFhNpnqxoxTnwuIctE GVkAoKAloVAh5CMDQynzB39xaDyXIB9v
=f+Qm
-----END PGP SIGNATURE-----
Hi,
Quoted 2 lines · David Sommerseth
This is to avoid 'make check' fail if this extra test case is not configured.
[..]
t_client.sh.in line 24
Quoted 3 lines
t_client.sh.in +3
-0+ exit 77 +fi +
ACK again. Better :)
gert
Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany [email protected]...
fax: +49-89-35655025 [email protected]...Am 18.08.2010 14:27, schrieb David Sommerseth:
Quoted 3 lines
This is to avoid 'make check' fail if this extra test case is not configured. The patch gives a PASS, but will notify the user that this test was skipped.
The other question is how t_client.rc can go missing...
Signature · 1 line
Matthias Andree
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 6 lines · Matthias Andree
Am 18.08.2010 14:27, schrieb David Sommerseth:
Quoted 3 lines
This is to avoid 'make check' fail if this extra test case is not configured. The patch gives a PASS, but will notify the user that this test was skipped.
The other question is how t_client.rc can go missing...
The default is to only ship a t_client.rc.sample file which really needs to be configured before running this test. Anyhow, as this is a new and additional test to what 'make check' already does, we're taking the slow pace, making sure this framework works before taking it one step further.
Kind regards,
David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAkxr2LsACgkQDC186MBRfrouxgCfYX7rucl2aUrngTXJ/MBgjFaB ge8An05NuCLvZ3Uqx7MumNxLTRi7v8hE
=AxFr
-----END PGP SIGNATURE-----
Hi,
Quoted 6 lines · Matthias Andree
Am 18.08.2010 14:27, schrieb David Sommerseth:
Quoted 3 lines
This is to avoid 'make check' fail if this extra test case is not configured. The patch gives a PASS, but will notify the user that this test was skipped.
The other question is how t_client.rc can go missing...
It's not shipped by default. Since you need to setup a reference server, and this cannot be the same machine (since you can't enforce that packets go through the tun/tap then), it's not something that can be successfully tested "just by extracting the tarball and run make ; make check".
We ship a t_client.rc-sample that explains what goes in there and how to enable this.
gert
Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany [email protected]...
fax: +49-89-35655025 [email protected]...Hi,
Quoted 3 lines · David Sommerseth
This is to avoid 'make check' fail if this extra test case is not configured. The patch gives a PASS, but will notify the user that this test was skipped.
Ummm. Dunno what I have been smoking :-) ("if you don't want to test this, this must be a mistake!!!") - but since this is not something everyone will be using, your patch makes more sense.
ACK, and thanks.
gert
Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany [email protected]...
fax: +49-89-35655025 [email protected]...