selftests: Add futex functional tests

Hi Shuah,

This series begins the process of migrating my futextest tests into kselftest. I've started with only the functional tests, as the performance and stress may not be appropriate for kselftest as they stand.

I cleaned up various complaints from checkpatch, but I ignored others that would require significant rework of the testcases, such as not using volatile and not creating new typedefs.

Since v1:
Avoid checkpatch errors on 1/5 by:
- combining a later patch which did substantial cleanup.
- removing file-local typedefs and replacing with structs
- correcting all >80 char lines, except for quoted strings and header boiler plate due to long email addresses

I did *not* make changes for the following:
- Use of new typdefs for types futex_t and atomic_t as they are used throughout the test suite and I consider them to be worth while.
- Use of volatile as the warning is about use of volatile in kernel code. The usage in futextest is correct, as an indicator that other threads may modify the value.
- Adding parentheses around complex defines as it would break one use case and change the behavior of another.

Since v2:
- Remove trailing newline from selftests/futex/functional/run.sh

Since v3:
- Replace -lpthread in LDFLAGS with -pthread in LDFLAGS and CFLAGS
- Add .gitignore for the target files (compiled tests)
- Resolved bad content encoding header in my patch submission process

The patches will follow, but I'm providing a pull request for your convenience as well.

The following changes since commit b787f68c36d49bb1d9236f403813641efa74a031:

Linux 4.1-rc1 (2015-04-26 17:59:10 -0700)

are available in the git repository at:

git://git.infradead.org/users/dvhart/linux.git futextest-v4

Darren Hart (6):
selftests: Add futex functional tests selftests/futex: Update Makefile to use lib.mk selftests/futex: Increment ksft pass and fail counters
selftests: Add futex tests to the top-level Makefile
kselftest: Add exit code defines selftests/futex: Add .gitignore

 tools/testing/selftests/Makefile                   |   1 +
 tools/testing/selftests/futex/Makefile             |  29 ++
 tools/testing/selftests/futex/README               |  62 ++++
 .../testing/selftests/futex/functional/.gitignore  |   7 +
 tools/testing/selftests/futex/functional/Makefile  |  25 ++
 .../selftests/futex/functional/futex_requeue_pi.c  | 409 +++++++++++++++++++++
 .../functional/futex_requeue_pi_mismatched_ops.c   | 135 +++++++
 .../functional/futex_requeue_pi_signal_restart.c   | 223 +++++++++++
 .../functional/futex_wait_private_mapped_file.c    | 125 +++++++
 .../futex/functional/futex_wait_timeout.c          |  86 +++++
 .../functional/futex_wait_uninitialized_heap.c     | 124 +++++++
 .../futex/functional/futex_wait_wouldblock.c       |  79 ++++
 tools/testing/selftests/futex/functional/run.sh    |  79 ++++
 tools/testing/selftests/futex/include/atomic.h     |  83 +++++
 tools/testing/selftests/futex/include/futextest.h  | 266 ++++++++++++++
 tools/testing/selftests/futex/include/logging.h    | 153 ++++++++
 tools/testing/selftests/futex/run.sh               |  33 ++
 tools/testing/selftests/kselftest.h                |  17 +-
 18 files changed, 1931 insertions(+), 5 deletions(-)
 create mode 100644 tools/testing/selftests/futex/Makefile
 create mode 100644 tools/testing/selftests/futex/README
 create mode 100644 tools/testing/selftests/futex/functional/.gitignore
 create mode 100644 tools/testing/selftests/futex/functional/Makefile
 create mode 100644 tools/testing/selftests/futex/functional/futex_requeue_pi.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_requeue_pi_mismatched_ops.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_requeue_pi_signal_restart.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_wait_private_mapped_file.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_wait_timeout.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_wait_uninitialized_heap.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_wait_wouldblock.c
 create mode 100755 tools/testing/selftests/futex/functional/run.sh
 create mode 100644 tools/testing/selftests/futex/include/atomic.h
 create mode 100644 tools/testing/selftests/futex/include/futextest.h
 create mode 100644 tools/testing/selftests/futex/include/logging.h
 create mode 100755 tools/testing/selftests/futex/run.sh

2.1.4

The futextest testsuite [1] provides functional, stress, and performance tests for the various futex op codes. Those tests will be of more use to futex developers if they are included with the kernel source.

Copy the core infrastructure and the functional tests into selftests, but adapt them for inclusion in the kernel:

- Update the Makefile to include the run_tests target, remove reference to the performance and stress tests from the contributed sources.
- Replace my dead IBM email address with my current Intel email address.
- Remove the warrantee and write-to paragraphs from the license blurbs.
- Remove the NAME section as the filename is easily determined. ;-)
- Make the whitespace usage consistent in a couple of places.
- Cleanup various CodingStyle violations.

A future effort will explore moving the performance and stress tests into the kernel.

1. http://git.kernel.org/cgit/linux/kernel/git/dvhart/futextest.git

---
 tools/testing/selftests/futex/Makefile             |  11 +
 tools/testing/selftests/futex/README               |  62 ++++
 tools/testing/selftests/futex/functional/Makefile  |  24 ++
 .../selftests/futex/functional/futex_requeue_pi.c  | 409 +++++++++++++++++++++
 .../functional/futex_requeue_pi_mismatched_ops.c   | 135 +++++++
 .../functional/futex_requeue_pi_signal_restart.c   | 223 +++++++++++
 .../functional/futex_wait_private_mapped_file.c    | 125 +++++++
 .../futex/functional/futex_wait_timeout.c          |  86 +++++
 .../functional/futex_wait_uninitialized_heap.c     | 124 +++++++
 .../futex/functional/futex_wait_wouldblock.c       |  79 ++++
 tools/testing/selftests/futex/functional/run.sh    |  79 ++++
 tools/testing/selftests/futex/include/atomic.h     |  83 +++++
 tools/testing/selftests/futex/include/futextest.h  | 266 ++++++++++++++
 tools/testing/selftests/futex/include/logging.h    | 150 ++++++++
 tools/testing/selftests/futex/run.sh               |  33 ++
 15 files changed, 1889 insertions(+)
 create mode 100644 tools/testing/selftests/futex/Makefile
 create mode 100644 tools/testing/selftests/futex/README
 create mode 100644 tools/testing/selftests/futex/functional/Makefile
 create mode 100644 tools/testing/selftests/futex/functional/futex_requeue_pi.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_requeue_pi_mismatched_ops.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_requeue_pi_signal_restart.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_wait_private_mapped_file.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_wait_timeout.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_wait_uninitialized_heap.c
 create mode 100644 tools/testing/selftests/futex/functional/futex_wait_wouldblock.c
 create mode 100755 tools/testing/selftests/futex/functional/run.sh
 create mode 100644 tools/testing/selftests/futex/include/atomic.h
 create mode 100644 tools/testing/selftests/futex/include/futextest.h
 create mode 100644 tools/testing/selftests/futex/include/logging.h
 create mode 100755 tools/testing/selftests/futex/run.sh

new file mode 100644
index 0000000..2c26d59
@@ -0,0 +1,11 @@
+SUBDIRS := functional
+
+.PHONY: all clean
+all:
+	for DIR in $(SUBDIRS); do $(MAKE) -C $$DIR $@ ; done
+
+run_tests: all
+	./run.sh
+
+clean:
+	for DIR in $(SUBDIRS); do $(MAKE) -C $$DIR $@ ; done

new file mode 100644
index 0000000..720023e
@@ -0,0 +1,24 @@
+INCLUDES := -I../include
+CFLAGS := $(CFLAGS) -g -O2 -Wall -D_GNU_SOURCE -pthread $(INCLUDES)
+LDFLAGS := $(LDFLAGS) -pthread -lrt
+
+HEADERS := ../include/futextest.h
+TARGETS := \
+	futex_wait_timeout \
+	futex_wait_wouldblock \
+	futex_requeue_pi \
+	futex_requeue_pi_signal_restart \
+	futex_requeue_pi_mismatched_ops \
+	futex_wait_uninitialized_heap \
+	futex_wait_private_mapped_file
+
+.PHONY: all clean
+all: $(TARGETS)
+
+$(TARGETS): $(HEADERS)
+
+run_tests: all
+	./run.sh
+
+clean:
+	rm -f $(TARGETS)

new file mode 100755
index 0000000..4126312
@@ -0,0 +1,33 @@
+#!/bin/sh
+
+###############################################################################
+#
+#   Copyright © International Business Machines  Corp., 2009
+#
+#   This program is free software;  you can redistribute it and/or modify
+#   it under the terms of the GNU General Public License as published by
+#   the Free Software Foundation; either version 2 of the License, or
+#   (at your option) any later version.
+#
+# DESCRIPTION
+#      Run all tests under the functional, performance, and stress directories.
+#      Format and summarize the results.
+#
+# AUTHOR
+#      Darren Hart <dvhart-VuQAYsv1563Yd54FQh9/[email protected]>
+#
+# HISTORY
+#      2009-Nov-9: Initial version by Darren Hart <[email protected]>
+#
+###############################################################################
+
+# Test for a color capable shell and pass the result to the subdir scripts
+USE_COLOR=0
+tput setf 7
+if [ $? -eq 0 ]; then
+    USE_COLOR=1
+    tput sgr0
+fi
+export USE_COLOR
+
+(cd functional; ./run.sh)

2.1.4

Adapt the futextest Makefiles to use lib.mk macros for RUN_TESTS and EMIT_TESTS. For now, we reuse the run.sh mechanism provided by futextest. This doesn't provide the standard selftests: [PASS|FAIL] format, but the tests provide very similar output already.

This results in the run_kselftest.sh script for futexes including a single line: ./run.sh

---
 tools/testing/selftests/futex/Makefile            | 20 +++++++++++++++++++-
 tools/testing/selftests/futex/functional/Makefile |  5 +++--
 2 files changed, 22 insertions(+), 3 deletions(-)

index 2c26d59..6a17529 100644
@@ -1,11 +1,29 @@
 SUBDIRS := functional
 
+TEST_PROGS := run.sh
+
 .PHONY: all clean
 all:
 	for DIR in $(SUBDIRS); do $(MAKE) -C $$DIR $@ ; done
 
-run_tests: all
+include ../lib.mk
+
+override define RUN_TESTS
 	./run.sh
+endef
+
+override define INSTALL_RULE
+	mkdir -p $(INSTALL_PATH)
+	install -t $(INSTALL_PATH) $(TEST_PROGS) $(TEST_PROGS_EXTENDED) $(TEST_FILES)
+
+	@for SUBDIR in $(SUBDIRS); do \
+		$(MAKE) -C $$SUBDIR INSTALL_PATH=$(INSTALL_PATH)/$$SUBDIR install; \
+	done;
+endef
+
+override define EMIT_TESTS
+	echo "./run.sh"
+endef
 
 clean:
 	for DIR in $(SUBDIRS); do $(MAKE) -C $$DIR $@ ; done

index 720023e..4b018fe 100644
@@ -12,13 +12,14 @@ TARGETS := \
 	futex_wait_uninitialized_heap \
 	futex_wait_private_mapped_file
 
+TEST_PROGS := $(TARGETS) run.sh
+
 .PHONY: all clean
 all: $(TARGETS)
 
 $(TARGETS): $(HEADERS)
 
-run_tests: all
-	./run.sh
+include ../../lib.mk
 
 clean:
 	rm -f $(TARGETS)

2.1.4

Add kselftest.h to logging.h and increment the pass and fail counters as part of the print_result routine which is called by all futex tests.

---
 tools/testing/selftests/futex/functional/Makefile | 2 +-
 tools/testing/selftests/futex/include/logging.h   | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)

index 4b018fe..9d6b75e 100644
@@ -1,4 +1,4 @@
-INCLUDES := -I../include
+INCLUDES := -I../include -I../../
 CFLAGS := $(CFLAGS) -g -O2 -Wall -D_GNU_SOURCE -pthread $(INCLUDES)
 LDFLAGS := $(LDFLAGS) -pthread -lrt
 

index f6ed5c2..014aa01 100644
@@ -24,6 +24,7 @@
 #include <string.h>
 #include <unistd.h>
 #include <linux/futex.h>
+#include "kselftest.h"
 
 /*
  * Define PASS, ERROR, and FAIL strings with and without color escape

@@ -111,12 +112,14 @@ void print_result(int ret)
 
 	switch (ret) {
 	case RET_PASS:
+		ksft_inc_pass_cnt();
 		result = PASS;
 		break;
 	case RET_ERROR:
 		result = ERROR;
 		break;
 	case RET_FAIL:
+		ksft_inc_fail_cnt();
 		result = FAIL;
 		break;
 	}

2.1.4

Enable futex tests to be built and run with the make kselftest and associated targets.

Most of the tests require escalated privileges. These return ERROR, and run.sh continues.

---
 tools/testing/selftests/Makefile | 1 +
 1 file changed, 1 insertion(+)

index 95abddc..ebac6b8 100644
@@ -4,6 +4,7 @@ TARGETS += efivarfs
 TARGETS += exec
 TARGETS += firmware
 TARGETS += ftrace
+TARGETS += futex
 TARGETS += kcmp
 TARGETS += memfd
 TARGETS += memory-hotplug

2.1.4

Define the exit codes with KSFT_PASS and similar so tests can use these directly if they choose. Also enable harnesses and other tooling to use the defines instead of hardcoding the return codes.

---
 tools/testing/selftests/kselftest.h | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

index 572c888..ef1c80d 100644
@@ -13,6 +13,13 @@
 #include <stdlib.h>
 #include <unistd.h>
 
+/* define kselftest exit codes */
+#define KSFT_PASS  0
+#define KSFT_FAIL  1
+#define KSFT_XFAIL 2
+#define KSFT_XPASS 3
+#define KSFT_SKIP  4
+
 /* counters */
 struct ksft_count {
 	unsigned int ksft_pass;

@@ -40,23 +47,23 @@ static inline void ksft_print_cnts(void)
 
 static inline int ksft_exit_pass(void)
 {
-	exit(0);
+	exit(KSFT_PASS);
 }
 static inline int ksft_exit_fail(void)
 {
-	exit(1);
+	exit(KSFT_FAIL);
 }
 static inline int ksft_exit_xfail(void)
 {
-	exit(2);
+	exit(KSFT_XFAIL);
 }
 static inline int ksft_exit_xpass(void)
 {
-	exit(3);
+	exit(KSFT_XPASS);
 }
 static inline int ksft_exit_skip(void)
 {
-	exit(4);
+	exit(KSFT_SKIP);
 }
 
 #endif /* __KSELFTEST_H */

2.1.4

Add the futex/functional targets to .gitignore.

---
 tools/testing/selftests/futex/functional/.gitignore | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 tools/testing/selftests/futex/functional/.gitignore

new file mode 100644
index 0000000..a09f570
@@ -0,0 +1,7 @@
+futex_requeue_pi
+futex_requeue_pi_mismatched_ops
+futex_requeue_pi_signal_restart
+futex_wait_private_mapped_file
+futex_wait_timeout
+futex_wait_uninitialized_heap
+futex_wait_wouldblock

2.1.4

Daren,

This patch series is good. kselftest run and install are good. I see the following results. Something you would expect?? No Issues and I am going to get them into 4.2 - should show up in linux-kselftest next.

Thanks for adding these.

thanks,
-- Shuah

futex_requeue_pi: Test requeue functionality
Arguments: broadcast=0 locked=0 owner=0 timeout=0ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=1 locked=0 owner=0 timeout=0ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=1 locked=1 owner=0 timeout=0ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=1 locked=0 owner=1 timeout=0ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating third party blocker thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=0 locked=1 owner=0 timeout=0ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=0 locked=0 owner=1 timeout=0ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating third party blocker thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=1 locked=1 owner=0 timeout=5000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=0 locked=1 owner=0 timeout=5000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=1 locked=1 owner=0 timeout=500000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=0 locked=1 owner=0 timeout=500000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=1 locked=0 owner=0 timeout=5000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=0 locked=0 owner=0 timeout=5000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=1 locked=0 owner=0 timeout=500000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=0 locked=0 owner=0 timeout=500000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=1 locked=0 owner=1 timeout=5000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating third party blocker thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=0 locked=1 owner=0 timeout=5000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=1 locked=0 owner=1 timeout=500000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating third party blocker thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=0 locked=1 owner=0 timeout=500000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=1 locked=1 owner=0 timeout=2000000000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR
futex_requeue_pi: Test requeue functionality
Arguments: broadcast=0 locked=1 owner=0 timeout=2000000000ns
ERROR: Operation not permitted: pthread_create
ERROR: Creating waiting thread failed
Result: ERROR

futex_requeue_pi_mismatched_ops: Detect mismatched requeue_pi operations
Result: PASS

futex_requeue_pi_signal_restart: Test signal handling during requeue_pi
Arguments: <none>
ERROR: Operation not permitted: pthread_create
ERROR: Unknown error -1: Creating waiting thread failedResult: ERROR

futex_wait_timeout: Block on a futex and wait for timeout

	Arguments: timeout=100000ns

Result: PASS

futex_wait_wouldblock: Test the unexpected futex value in FUTEX_WAIT
Result: PASS

futex_wait_uninitialized_heap: Test the uninitialized futex value in FUTEX_WAIT
Result: PASS
futex_wait_private_mapped_file: Test the futex value of private file mappings in FUTEX_WAIT
Result: PASS

Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
[email protected] | (970) 217-8978

Yes, that is all expected behavior if you don't run as root and your user doesn't have the necessary capabilities added - similar to other tests with privileged operations.

Great, thanks. Perhaps some explicit capabilities tests and clearer error messages would make sense as a follow-on patch.

Thanks for working with me to get these into the kernel, I think it will improve futex test exposure and hopefully encourage people to write more tests to fill some of the gaps.

--
Darren

Darren Hart
Intel Open Source Technology Center

Applied to linux-kselftest next for 4.2

thanks,
-- Shuah

Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
[email protected] | (970) 217-8978