selftest: Add futex functional tests

Adapt the futextest Makefiles to use lib.mk macros for RUN_TESTS and EMIT_TESTS. For now, we reuse the run.sh mechanism provided by futextest. This doesn't provide the standard selftests: [PASS|FAIL] format, but the tests provide very similar output already.

This results in the run_kselftest.sh script for futexes including a single line: ./run.sh

Cc Shuah Khan <[email protected]>

Cc [email protected]

Cc Ingo Molnar <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Davidlohr Bueso <[email protected]>

Cc KOSAKI Motohiro <[email protected]>

Signed-off-by Darren Hart <dvhart-VuQAYsv1563Yd54FQh9/[email protected]>

---
 tools/testing/selftests/futex/Makefile            | 21 +++++++++++++++++++++
 tools/testing/selftests/futex/functional/Makefile |  4 ++++
 2 files changed, 25 insertions(+)

tools/testing/selftests/futex/Makefile +0 -0

index 8629187..6a17529 100644

Quoted 30 lines · Darren Hart

Adapt the futextest Makefiles to use lib.mk macros for RUN_TESTS and EMIT_TESTS. For now, we reuse the run.sh mechanism provided by futextest. This doesn't provide the standard selftests: [PASS|FAIL] format, but the tests provide very similar output already.

This results in the run_kselftest.sh script for futexes including a single line: ./run.sh

Cc Shuah Khan <[email protected]>

Cc [email protected]

Cc Ingo Molnar <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Davidlohr Bueso <[email protected]>

Cc KOSAKI Motohiro <[email protected]>

Signed-off-by Darren Hart <[email protected]>

---
 tools/testing/selftests/futex/Makefile            | 21
+++++++++++++++++++++
 tools/testing/selftests/futex/functional/Makefile |  4 ++++
 2 files changed, 25 insertions(+)

tools/testing/selftests/futex/Makefile +0 -0

index 8629187..6a17529 100644
@@ -1,8 +1,29 @@
 SUBDIRS := functional
 

tools/testing/selftests/futex/Makefile line 5

Quoted 3 lines

tools/testing/selftests/futex/Makefile +2 -0

+TEST_PROGS := run.sh
+
 .PHONY: all clean

lib.mk defines all & clean as PHONY for you.

Quoted 4 lines

tools/testing/selftests/futex/Makefile +0 -0

 all:
 	for DIR in $(SUBDIRS); do $(MAKE) -C $$DIR $@ ; done
 

tools/testing/selftests/futex/Makefile line 13

Quoted 5 lines

tools/testing/selftests/futex/Makefile +5 -0

+include ../lib.mk
+
+override define RUN_TESTS
+	./run.sh
+endef

Do you need to do this override? The standard logic should work AFAICS, or do
you not want the echo logic?

Quoted 4 lines

tools/testing/selftests/futex/Makefile +3 -0

+override define INSTALL_RULE
+	mkdir -p $(INSTALL_PATH)
+	install -t $(INSTALL_PATH) $(TEST_PROGS) $(TEST_PROGS_EXTENDED)

tools/testing/selftests/futex/Makefile line 26

Quoted 10 lines

tools/testing/selftests/futex/Makefile +9 -0

 $(TEST_FILES)
+
+	@for SUBDIR in $(SUBDIRS); do \
+		$(MAKE) -C $$SUBDIR INSTALL_PATH=$(INSTALL_PATH)/$$SUBDIR install; \
+	done;
+endef
+
+override define EMIT_TESTS
+	echo "./run.sh"
+endef

Ditto.

cheers

Quoted 17 lines · Michael Ellerman

Quoted 16 lines · Darren Hart

Adapt the futextest Makefiles to use lib.mk macros for RUN_TESTS and EMIT_TESTS. For now, we reuse the run.sh mechanism provided by futextest. This doesn't provide the standard selftests: [PASS|FAIL] format, but the tests provide very similar output already.

This results in the run_kselftest.sh script for futexes including a single line: ./run.sh

tools/testing/selftests/futex/Makefile +0 -0

index 8629187..6a17529 100644
@@ -1,8 +1,29 @@
 SUBDIRS := functional
 

Quoted 5 lines

tools/testing/selftests/futex/Makefile line 5

Quoted 3 lines

tools/testing/selftests/futex/Makefile +2 -0

+TEST_PROGS := run.sh
+
 .PHONY: all clean

lib.mk defines all & clean as PHONY for you.

OK, I can remove this (it was just part of the originals). Shall I resent the whole series or just a follow-on patch?

Quoted 12 lines

tools/testing/selftests/futex/Makefile line 13

Quoted 8 lines

tools/testing/selftests/futex/Makefile +5 -0

 all:
 	for DIR in $(SUBDIRS); do $(MAKE) -C $$DIR $@ ; done
 
+include ../lib.mk
+
+override define RUN_TESTS
+	./run.sh
+endef

Do you need to do this override? The standard logic should work AFAICS, or do
you not want the echo logic?

The standard logic wants to run each individual test and uses the echo logic. My tests have their own pass/fail reporting mechanism. As I run many options on the same tests and have pretty-printing logic, I prefer to keep that in a run script rather than in a makefile.

I don't care for the echo logic as a mechanism, I think the reporting should be part of the test. It also doesn't scale past pass/fail - like xpass and xfail, for example.

Define the exit codes with KSFT_PASS and similar so tests can use these directly if they choose. Also enable harnesses and other tooling to use the defines instead of hardcoding the return codes.

tools/testing/selftests/kselftest.h +7 -0

index 572c888..ef1c80d 100644
@@ -13,6 +13,13 @@
 #include <stdlib.h>
 #include <unistd.h>
 
+/* define kselftest exit codes */
+#define KSFT_PASS  0
+#define KSFT_FAIL  1
+#define KSFT_XFAIL 2
+#define KSFT_XPASS 3
+#define KSFT_SKIP  4
+
 /* counters */
 struct ksft_count {
 	unsigned int ksft_pass;

Quoted 3 lines · Darren Hart

Define the exit codes with KSFT_PASS and similar so tests can use these directly if they choose. Also enable harnesses and other tooling to use the defines instead of hardcoding the return codes.

+1

Quoted 20 lines

tools/testing/selftests/kselftest.h +7 -0

index 572c888..ef1c80d 100644
@@ -13,6 +13,13 @@
 #include <stdlib.h>
 #include <unistd.h>
 
+/* define kselftest exit codes */
+#define KSFT_PASS  0
+#define KSFT_FAIL  1
+#define KSFT_XFAIL 2
+#define KSFT_XPASS 3
+#define KSFT_SKIP  4
+
 /* counters */
 struct ksft_count {
 	unsigned int ksft_pass;

@@ -40,23 +47,23 @@ static inline void ksft_print_cnts(void)
 

tools/testing/selftests/kselftest.h static inline void ksft_print_cnts(void) · line 51

Quoted 5 lines

tools/testing/selftests/kselftest.h +1 -1

 static inline int ksft_exit_pass(void)
 {
-	exit(0);
+	exit(KSFT_PASS);
 }

Am I the only person who's bothered by the fact that these don't actually return int?

Quoted 27 lines · Michael Ellerman

Quoted 3 lines · Darren Hart

Define the exit codes with KSFT_PASS and similar so tests can use these directly if they choose. Also enable harnesses and other tooling to use the defines instead of hardcoding the return codes.

+1

Quoted 20 lines

tools/testing/selftests/kselftest.h +7 -0

index 572c888..ef1c80d 100644
@@ -13,6 +13,13 @@
 #include <stdlib.h>
 #include <unistd.h>
 
+/* define kselftest exit codes */
+#define KSFT_PASS  0
+#define KSFT_FAIL  1
+#define KSFT_XFAIL 2
+#define KSFT_XPASS 3
+#define KSFT_SKIP  4
+
 /* counters */
 struct ksft_count {
 	unsigned int ksft_pass;

@@ -40,23 +47,23 @@ static inline void ksft_print_cnts(void)
 

Quoted 8 lines

tools/testing/selftests/kselftest.h static inline void ksft_print_cnts(void) · line 51

Quoted 5 lines

tools/testing/selftests/kselftest.h +1 -1

 static inline int ksft_exit_pass(void)
 {
-	exit(0);
+	exit(KSFT_PASS);
 }

Am I the only person who's bothered by the fact that these don't actually return int?

That bothered me to, but I couldn't be bothered to go read the manuals apparently to come up with a compelling argument :-)

I also think the ksft_exit* routines should go ahead and increment the counters (at least optionally) so we don't have to call two functions.

Quoted 39 lines · Darren Hart

Quoted 2 lines · Michael Ellerman

Quoted 1 line · Darren Hart

Define the exit codes with KSFT_PASS and similar so tests can use

these

Quoted 1 line

Quoted 1 line

directly if they choose. Also enable harnesses and other tooling to

use

Quoted 5 lines

Quoted 1 line

the defines instead of hardcoding the return codes.

+1

Review

Quoted 1 line

(quoted diff) +0 -0

b/tools/testing/selftests/kselftest.h

Quoted 19 lines

Quoted 19 lines

index 572c888..ef1c80d 100644

tools/testing/selftests/kselftest.h +7 -0

@@ -13,6 +13,13 @@
 #include <stdlib.h>
 #include <unistd.h>
 
+/* define kselftest exit codes */
+#define KSFT_PASS  0
+#define KSFT_FAIL  1
+#define KSFT_XFAIL 2
+#define KSFT_XPASS 3
+#define KSFT_SKIP  4
+
 /* counters */
 struct ksft_count {
 	unsigned int ksft_pass;

@@ -40,23 +47,23 @@ static inline void ksft_print_cnts(void)
 

Quoted 7 lines

tools/testing/selftests/kselftest.h static inline void ksft_print_cnts(void) · line 51

Quoted 5 lines

tools/testing/selftests/kselftest.h +1 -1

 static inline int ksft_exit_pass(void)
 {
-	exit(0);
+	exit(KSFT_PASS);
 }

Am I the only person who's bothered by the fact that these don't

Quoted 5 lines

actually

Quoted 1 line

return int?

That bothered me to, but I couldn't be bothered to go read the manuals apparently to come up with a compelling argument :-)

Yeah, obviously the compiler accepts it, but it's still a bit weird.

Quoted 2 lines

I also think the ksft_exit* routines should go ahead and increment the counters (at least optionally) so we don't have to call two functions.

But the ksft_exit_*() routines exit, so there's no point incrementing the counters. Unless they *also* print the counters before exiting?

To be honest I think we need to decide if the selftests are going to speak TAP
or xUnit or whatever, and then switch to that. In their current form these helpers don't help much.

Fair point. This isn't a space I'm well versed in, but some standard means of doing this would be welcome.

The futextest testsuite [1] provides functional, stress, and performance tests for the various futex op codes. Those tests will be of more use to futex developers if they are included with the kernel source.

Copy the core infrastructure and the functional tests into selftests. Remove reference to the performance and stress tests from the contributed sources.

Remove the Free Software Foundation address paragraph from all contributed files to avoid checkpatch complaints.

A future effort will explore moving the performance and stress tests into the kernel.

1. http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git

Cc Shuah Khan <[email protected]>

Cc [email protected]

Cc Ingo Molnar <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Davidlohr Bueso <[email protected]>

Cc KOSAKI Motohiro <[email protected]>

Signed-off-by Darren Hart <dvhart-VuQAYsv1563Yd54FQh9/[email protected]>

Quoted 17 lines · Darren Hart

The futextest testsuite [1] provides functional, stress, and performance tests for the various futex op codes. Those tests will be of more use to futex developers if they are included with the kernel source.

Copy the core infrastructure and the functional tests into selftests. Remove reference to the performance and stress tests from the contributed sources.

Remove the Free Software Foundation address paragraph from all contributed files to avoid checkpatch complaints.

A future effort will explore moving the performance and stress tests into the kernel.

1. http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git

Quoted 8 lines

Cc Shuah Khan <[email protected]>

Cc [email protected]

Cc Ingo Molnar <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Davidlohr Bueso <[email protected]>

Cc KOSAKI Motohiro <[email protected]>

Signed-off-by Darren Hart <dvhart-VuQAYsv1563Yd54FQh9/[email protected]>

Sorry it took so long. I am just about to apply these to linux-kselftest next and ran into checkpatch errors on this patch.

Hi Daren,

Quoted 14 lines · Darren Hart

Hi Shuah,

This series begins the process of migrating my futextest tests into kselftest.
I've started with only the functional tests, as the performance and stress may
not be appropriate for kselftest as they stand.

I cleaned up various complaints from checkpatch, but I ignored others that would
require significant rework of the testcases, such as not using volatile and not
creating new typedefs.

Quoted 3 lines

The patches will follow, but I'm providing a pull request for your convenience
as well.

Thanks for acting on this so quickly after we talked about it at ELC. Just a quick note that I am going to get to this soon once I get the 4.1 content wrapped up. We can plan upon getting these into 4.2.

Quoted 21 lines · Shuah Khan

Hi Daren,

Quoted 17 lines · Darren Hart

Hi Shuah,

This series begins the process of migrating my futextest tests into kselftest.
I've started with only the functional tests, as the performance and stress may
not be appropriate for kselftest as they stand.

I cleaned up various complaints from checkpatch, but I ignored others that would
require significant rework of the testcases, such as not using volatile and not
creating new typedefs.

The patches will follow, but I'm providing a pull request for your convenience
as well.

Quoted 3 lines

Thanks for acting on this so quickly after we talked about it at ELC. Just a quick note that I am going to get to this soon once I get the 4.1 content wrapped up. We can plan upon getting these into 4.2.

OK. Michael E. provided some feedback which I can either incorporate and respin, or I can send as a follow-on to your -next after you merge these. Which do you prefer?

Quoted 31 lines · Darren Hart

Quoted 17 lines · Shuah Khan

Hi Daren,

Quoted 14 lines · Darren Hart

Hi Shuah,

This series begins the process of migrating my futextest tests into kselftest.
I've started with only the functional tests, as the performance and stress may
not be appropriate for kselftest as they stand.

I cleaned up various complaints from checkpatch, but I ignored others that would
require significant rework of the testcases, such as not using volatile and not
creating new typedefs.

Quoted 7 lines

Quoted 3 lines

The patches will follow, but I'm providing a pull request for your convenience
as well.

Thanks for acting on this so quickly after we talked about it at ELC. Just a quick note that I am going to get to this soon once I get the 4.1 content wrapped up. We can plan upon getting these into 4.2.

OK. Michael E. provided some feedback which I can either incorporate and respin, or I can send as a follow-on to your -next after you merge these. Which do you prefer?

You can wait to re-do patches. I am planning to review the patch set later on this week. That way you can avoid re-spin just in case, I have other comments.

Quoted 32 lines · Shuah Khan

Quoted 31 lines · Darren Hart

Quoted 17 lines · Shuah Khan

Hi Daren,

Quoted 14 lines · Darren Hart

Hi Shuah,

This series begins the process of migrating my futextest tests into kselftest.
I've started with only the functional tests, as the performance and stress may
not be appropriate for kselftest as they stand.

I cleaned up various complaints from checkpatch, but I ignored others that would
require significant rework of the testcases, such as not using volatile and not
creating new typedefs.

Quoted 7 lines

Quoted 3 lines

The patches will follow, but I'm providing a pull request for your convenience
as well.

Thanks for acting on this so quickly after we talked about it at ELC. Just a quick note that I am going to get to this soon once I get the 4.1 content wrapped up. We can plan upon getting these into 4.2.

OK. Michael E. provided some feedback which I can either incorporate and respin, or I can send as a follow-on to your -next after you merge these. Which do you prefer?

...

Quoted 3 lines

You can wait to re-do patches. I am planning to review the patch set later on this week. That way you can avoid re-spin just in case, I have other comments.

Hi Shuah,

Did you have any additional comments for the futex tests?

Thanks,

Quoted 46 lines · Darren Hart

Quoted 36 lines · Shuah Khan

Quoted 35 lines · Darren Hart

Quoted 19 lines · Shuah Khan

Hi Daren,

Quoted 16 lines · Darren Hart

Hi Shuah,

This series begins the process of migrating my futextest tests into kselftest.
I've started with only the functional tests, as the performance and stress may
not be appropriate for kselftest as they stand.

I cleaned up various complaints from checkpatch, but I ignored others
that would
require significant rework of the testcases, such as not using volatile
and not
creating new typedefs.

Quoted 7 lines

Quoted 3 lines

The patches will follow, but I'm providing a pull request for your convenience
as well.

Thanks for acting on this so quickly after we talked about it at ELC. Just a quick note that I am going to get to this soon once I get the 4.1 content wrapped up. We can plan upon getting these into 4.2.

OK. Michael E. provided some feedback which I can either incorporate and
respin, or I can send as a follow-on to your -next after you merge these.
Which do you prefer?

...

Quoted 3 lines

You can wait to re-do patches. I am planning to review the patch set later on this week. That way you can avoid re-spin just in case, I have other comments.

Hi Shuah,

Quoted 4 lines

Did you have any additional comments for the futex tests?

Thanks,

Sorry Daren. I got called away on a family emergency. The patch looks good to me and I don't have any additional comments.

Assume these get pulled in for 4.2 release. I will get to them soon once the 4.1-rc1 is out.

Hi Shuah,

This series begins the process of migrating my futextest tests into kselftest. I've started with only the functional tests, as the performance and stress may not be appropriate for kselftest as they stand.

I cleaned up various complaints from checkpatch, but I ignored others that would require significant rework of the testcases, such as not using volatile and not creating new typedefs.

The patches will follow, but I'm providing a pull request for your convenience as well.

The following changes since commit 0b63accf87225b5eb7e52814c374cf02d733d4bb:

tools, update rtctest.c to verify passage of time (2015-03-24 22:02:59 -0600)

are available in the git repository at:

git://git.infradead.org/users/dvhart/linux.git futextest

Darren Hart (5):
selftests: Add futex functional tests selftest/futex: Update Makefile to use lib.mk selftest/futex: Increment ksft pass and fail counters
selftest: Add futex tests to the top-level Makefile
kselftest: Add exit code defines