Restartable Sequences updates for 5.1
Subsystems (6)
Mathieu Desnoyers
maintainercover Restartable Sequences updates for 5.1
Those changes aiming at 5.1 include one comment cleanup, the removal of the rseq_len field from the task struct which serves no purpose considering that the struct size is fixed by the ABI, and a selftest improvement adapting the number of threads to the number of detected CPUs, which is nicer for smaller systems.
Thanks,
Mathieu
Mathieu Desnoyers (3):
rseq: cleanup: Reflect removal of event counter in comments
rseq: cleanup: remove rseq_len from task_struct rseq/selftests: Adapt number of threads to the number of detected cpus
arch/arm/kernel/signal.c | 3 +-- arch/x86/kernel/signal.c | 5 +---- include/linux/sched.h | 4 ---- kernel/rseq.c | 9 +++------ tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 5 files changed, 10 insertions(+), 18 deletions(-)
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerPATCH 1/3 rseq: cleanup: Reflect removal of event counter in comments
The "event counter" was removed from rseq before it was merged upstream. However, a few comments in the source code still refer to it. Adapt the comments to match reality.
--- arch/arm/kernel/signal.c | 3 +-- arch/x86/kernel/signal.c | 5 +---- kernel/rseq.c | 3 +-- 3 files changed, 3 insertions(+), 8 deletions(-)
arch/arm/kernel/signal.c +1 -2
@@ -549,8 +549,7 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs)int ret;/* -* Increment event counter and perform fixup for the pre-signal -* frame. +* Perform fixup for the pre-signal frame. */ rseq_signal_deliver (ksig, regs);
arch/x86/kernel/signal.c +1 -4
@@ -688,10 +688,7 @@ setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs)sigset_t *set =sigmask_to_save ();compat_sigset_t *cset = (compat_sigset_t *) set; -/* -* Increment event counter and perform fixup for the pre-signal -* frame. -*/ +/* Perform fixup for the pre-signal frame. */ rseq_signal_deliver (ksig, regs);/* Set up the stack frame */
kernel/rseq.c +1 -2
@@ -254,8 +254,7 @@ static int rseq_ip_fixup(struct pt_regs *regs)* - signal delivery, * and return to user-space. * -* This is how we can ensure that the entire rseq critical section, -* consisting of both the C part and the assembly instruction sequence, +* This is how we can ensure that the entire rseq critical section * will issue the commit instruction only if executed atomically with * respect to other threads scheduled on the same CPU, and with respect * to signal handlers.
Signature · 1 line
2.11.0
tip-bot for Mathieu Desnoyers
merge bot[tip:core/rseq] rseq: Clean up comments by reflecting removal of event counter
rseq: Clean up comments by reflecting removal of event counter
Commit-ID: bff9504bfc9c5c6610b42d47f689f350fd969eb8 Gitweb: https://git.kernel.org/tip/bff9504bfc9c5c6610b42d47f689f350fd969eb8 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Tue, 5 Mar 2019 14:47:53 -0500 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 19 Apr 2019 12:39:31 +0200
rseq: Clean up comments by reflecting removal of event counter
The "event counter" was removed from rseq before it was merged upstream. However, a few comments in the source code still refer to it. Adapt the comments to match reality.
--- arch/arm/kernel/signal.c | 3 +-- arch/x86/kernel/signal.c | 5 +---- kernel/rseq.c | 3 +-- 3 files changed, 3 insertions(+), 8 deletions(-)
arch/arm/kernel/signal.c +1 -2
@@ -549,8 +549,7 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs)int ret;/* -* Increment event counter and perform fixup for the pre-signal -* frame. +* Perform fixup for the pre-signal frame. */ rseq_signal_deliver (ksig, regs);
arch/x86/kernel/signal.c +1 -4
@@ -688,10 +688,7 @@ setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs)sigset_t *set =sigmask_to_save ();compat_sigset_t *cset = (compat_sigset_t *) set; -/* -* Increment event counter and perform fixup for the pre-signal -* frame. -*/ +/* Perform fixup for the pre-signal frame. */ rseq_signal_deliver (ksig, regs);/* Set up the stack frame */
kernel/rseq.c +1 -2
@@ -254,8 +254,7 @@ static int rseq_ip_fixup(struct pt_regs *regs)* - signal delivery, * and return to user-space. * -* This is how we can ensure that the entire rseq critical section, -* consisting of both the C part and the assembly instruction sequence, +* This is how we can ensure that the entire rseq critical section * will issue the commit instruction only if executed atomically with * respect to other threads scheduled on the same CPU, and with respect * to signal handlers.
Mathieu Desnoyers
maintainerPATCH 2/3 rseq: cleanup: remove rseq_len from task_struct
In reply to Mathieu DesnoyersThe rseq system call, when invoked with flags of "0" or "RSEQ_FLAG_UNREGISTER" values, expects the rseq_len parameter to be equal to sizeof(struct rseq), which is fixed-size and fixed-layout, specified in uapi linux/rseq.h.
Expecting a fixed size for rseq_len is a design choice that ensures multiple libraries and application defining __rseq_abi in the same process agree on its exact size.
Considering that this size is and will always be the same value, there is no point in saving this value within task_struct rseq_len. Remove this field from task_struct.
--- include/linux/sched.h | 4 ---- kernel/rseq.c | 6 ++---- 2 files changed, 2 insertions(+), 8 deletions(-)
include/linux/sched.h +0 -4
@@ -1059,7 +1059,6 @@ struct task_struct {#ifdef CONFIG_RSEQ struct rseq __user *rseq; -u32 rseq_len;u32 rseq_sig;/* * RmW on rseq_event_mask must be performed atomically
@@ -1852,12 +1851,10 @@ static inline void rseq_fork(struct task_struct *t, unsigned long clone_flags) {if (clone_flags &CLONE_THREAD ) { t->rseq =NULL ; - t->rseq_len =0 ; t->rseq_sig =0 ; t->rseq_event_mask =0 ; }else { t->rseq = current->rseq; - t->rseq_len = current->rseq_len; t->rseq_sig = current->rseq_sig; t->rseq_event_mask = current->rseq_event_mask; }
@@ -1866,7 +1863,6 @@ static inline void rseq_fork(struct task_struct *t, unsigned long clone_flags)static inline void rseq_execve (struct task_struct *t) { t->rseq =NULL ; - t->rseq_len =0 ; t->rseq_sig =0 ; t->rseq_event_mask =0 ; }
kernel/rseq.c +2 -4
@@ -313,7 +313,7 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, rseq_len,/* Unregister rseq for current thread. */ if (current->rseq != rseq || !current->rseq)return -EINVAL ; -if (current->rseq_len != rseq_len) +if (rseq_len !=sizeof (*rseq))return -EINVAL ;if (current->rseq_sig != sig)return -EPERM ;
@@ -321,7 +321,6 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, rseq_len,if (ret)return ret; current->rseq =NULL ; - current->rseq_len =0 ; current->rseq_sig =0 ;return 0 ; }
@@ -335,7 +334,7 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, rseq_len,* the provided address differs from the prior * one. */ -if (current->rseq != rseq || current->rseq_len != rseq_len) +if (current->rseq != rseq || rseq_len !=sizeof (*rseq))return -EINVAL ;if (current->rseq_sig != sig)return -EPERM ;
@@ -353,7 +352,6 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, rseq_len,if (!access_ok (rseq, rseq_len))return -EFAULT ; current->rseq = rseq; - current->rseq_len = rseq_len; current->rseq_sig = sig;/* * If rseq was previously inactive, and has just been
Signature · 1 line
2.11.0
tip-bot for Mathieu Desnoyers
merge bot[tip:core/rseq] rseq: Remove superfluous rseq_len from task_struct
rseq: Remove superfluous rseq_len from task_struct
Commit-ID: 83b0b15bcb0f700e7c1d070aae2e7841170a4c33 Gitweb: https://git.kernel.org/tip/83b0b15bcb0f700e7c1d070aae2e7841170a4c33 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Tue, 5 Mar 2019 14:47:54 -0500 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 19 Apr 2019 12:39:32 +0200
rseq: Remove superfluous rseq_len from task_struct
The rseq system call, when invoked with flags of "0" or "RSEQ_FLAG_UNREGISTER" values, expects the rseq_len parameter to be equal to sizeof(struct rseq), which is fixed-size and fixed-layout, specified in uapi linux/rseq.h.
Expecting a fixed size for rseq_len is a design choice that ensures multiple libraries and application defining __rseq_abi in the same process agree on its exact size.
Considering that this size is and will always be the same value, there is no point in saving this value within task_struct rseq_len. Remove this field from task_struct.
No change in functionality intended.
--- include/linux/sched.h | 4 ---- kernel/rseq.c | 6 ++---- 2 files changed, 2 insertions(+), 8 deletions(-)
include/linux/sched.h +0 -4
@@ -1057,7 +1057,6 @@ struct task_struct {#ifdef CONFIG_RSEQ struct rseq __user *rseq; -u32 rseq_len;u32 rseq_sig;/* * RmW on rseq_event_mask must be performed atomically
@@ -1855,12 +1854,10 @@ static inline void rseq_fork(struct task_struct *t, unsigned long clone_flags) {if (clone_flags &CLONE_THREAD ) { t->rseq =NULL ; - t->rseq_len =0 ; t->rseq_sig =0 ; t->rseq_event_mask =0 ; }else { t->rseq = current->rseq; - t->rseq_len = current->rseq_len; t->rseq_sig = current->rseq_sig; t->rseq_event_mask = current->rseq_event_mask; }
@@ -1869,7 +1866,6 @@ static inline void rseq_fork(struct task_struct *t, unsigned long clone_flags)static inline void rseq_execve (struct task_struct *t) { t->rseq =NULL ; - t->rseq_len =0 ; t->rseq_sig =0 ; t->rseq_event_mask =0 ; }
kernel/rseq.c +2 -4
@@ -313,7 +313,7 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, rseq_len,/* Unregister rseq for current thread. */ if (current->rseq != rseq || !current->rseq)return -EINVAL ; -if (current->rseq_len != rseq_len) +if (rseq_len !=sizeof (*rseq))return -EINVAL ;if (current->rseq_sig != sig)return -EPERM ;
@@ -321,7 +321,6 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, rseq_len,if (ret)return ret; current->rseq =NULL ; - current->rseq_len =0 ; current->rseq_sig =0 ;return 0 ; }
@@ -335,7 +334,7 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, rseq_len,* the provided address differs from the prior * one. */ -if (current->rseq != rseq || current->rseq_len != rseq_len) +if (current->rseq != rseq || rseq_len !=sizeof (*rseq))return -EINVAL ;if (current->rseq_sig != sig)return -EPERM ;
@@ -353,7 +352,6 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, rseq_len,if (!access_ok (rseq, rseq_len))return -EFAULT ; current->rseq = rseq; - current->rseq_len = rseq_len; current->rseq_sig = sig;/* * If rseq was previously inactive, and has just been
Mathieu Desnoyers
maintainerPATCH 3/3 rseq/selftests: Adapt number of threads to the number of detected cpus
In reply to Mathieu DesnoyersOn smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online cpus at test runtime, and multiply that by 6 to have 6 rseq threads per cpu preempting each other.
--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
tools/testing/selftests/rseq/run_param_test.sh +5 -2
@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"
@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0 while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 echo "Running compare-twice test ${TEST_NAME[$i]}" - ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 let "i++" done }
Signature · 1 line
2.11.0
Re: [PATCH for 5.1 3/3] rseq/selftests: Adapt number of threads to the number of detected cpus
Quoted 55 lines · Mathieu Desnoyers
On smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online cpus at test runtime, and multiply that by 6 to have 6 rseq threads per cpu preempting each other.
Signed-off-by Mathieu Desnoyers <[email protected]>Cc Shuah Khan <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Joel Fernandes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Dave Watson <[email protected]>Cc Will Deacon <[email protected]>Cc Andi Kleen <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Chris Lameter <[email protected]>Cc Russell King <[email protected]>Cc Michael Kerrisk <[email protected]>Cc "Paul E . McKenney" <[email protected]>Cc Paul Turner <[email protected]>Cc Boqun Feng <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Maurer <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>Cc Linus Torvalds <[email protected]>--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/testing/selftests/rseq/run_param_test.sh +3
-0@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0
tools/testing/selftests/rseq/run_param_test.sh IFS="$OLDIFS" · line 45
Quoted 10 lines
tools/testing/selftests/rseq/run_param_test.sh +2
-2while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 echo "Running compare-twice test ${TEST_NAME[$i]}" - ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 let "i++" done }
BTW., when trying to build the rseq self-tests I get this build failure:
dagon:~/tip/tools/testing/selftests/rseq> make
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ -shared -fPIC rseq.c -lpthread -o /home/mingo/tip/tools/testing/selftests/rseq/librseq.so gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ basic_test.c -lpthread -lrseq -o /home/mingo/tip/tools/testing/selftests/rseq/basic_test gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ basic_percpu_ops_test.c -lpthread -lrseq -o /home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test /usr/bin/ld: /tmp/ccuHTWnZ.o: in function `rseq_cmpeqv_storev':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84: undefined reference to `.L8'
/usr/bin/ld: /home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84: undefined reference to `.L49'
/usr/bin/ld: /tmp/ccuHTWnZ.o: in function `rseq_cmpnev_storeoffp_load':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:141: undefined reference to `.L57'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x8): undefined reference to `.L8'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x14): undefined reference to `.L49'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x20): undefined reference to `.L55'
collect2: error: ld returned 1 exit status
make: *** [Makefile:22: /home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test] Error 1
Is this a known problem, or do I miss something from my build environment perhaps? Vanilla 64-bit Ubuntu 18.10 (Cosmic).
Thanks,
Ingo
Mathieu Desnoyers
maintainerRe: [PATCH for 5.1 3/3] rseq/selftests: Adapt number of threads to the number of detected cpus
Quoted 100 lines · ----- On Apr 19, 2019, at 6:38 AM, Ingo Molnar [email protected] wrote:
Quoted 66 lines · Mathieu Desnoyers
On smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online cpus at test runtime, and multiply that by 6 to have 6 rseq threads per cpu preempting each other.
Signed-off-by Mathieu Desnoyers <[email protected]>Cc Shuah Khan <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Joel Fernandes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Dave Watson <[email protected]>Cc Will Deacon <[email protected]>Cc Andi Kleen <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Chris Lameter <[email protected]>Cc Russell King <[email protected]>Cc Michael Kerrisk <[email protected]>Cc "Paul E . McKenney" <[email protected]>Cc Paul Turner <[email protected]>Cc Boqun Feng <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Maurer <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>Cc Linus Torvalds <[email protected]>--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/testing/selftests/rseq/run_param_test.sh +4
-1@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0 while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS}|| exit 1
tools/testing/selftests/rseq/run_param_test.sh +0
-1echo "Running compare-twice test ${TEST_NAME[$i]}" - ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} ||exit 1
tools/testing/selftests/rseq/run_param_test.sh +1
-0+ ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@}${EXTRA_ARGS} || exit 1
tools/testing/selftests/rseq/run_param_test.sh IFS="$OLDIFS" · line 45
Quoted 3 lines
tools/testing/selftests/rseq/run_param_test.sh +0
-0let "i++" done }BTW., when trying to build the rseq self-tests I get this build failure:
dagon:~/tip/tools/testing/selftests/rseq> make
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ -shared -fPICrseq.c-lpthread -o/home/mingo/tip/tools/testing/selftests/rseq/librseq.so
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_testgcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_percpu_ops_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpeqv_storev':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84: undefined reference to `.L8'
/usr/bin/ld:/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84:
undefined reference to `.L49'
/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpnev_storeoffp_load':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:141: undefined reference to `.L57'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x8): undefined reference to.L8' /usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x14): undefined reference to.L49'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x20): undefined reference to `.L55'
collect2: error: ld returned 1 exit status
make: *** [Makefile:22:
/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test] Error 1
Quoted 2 lines
Is this a known problem, or do I miss something from my build environment perhaps? Vanilla 64-bit Ubuntu 18.10 (Cosmic).
It works fine with gcc-7 (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) but indeed I get the same failure with gcc-8 (gcc version 8.0.1 20180414 (experimental) [trunk revision 259383] (Ubuntu 8-20180414-1ubuntu2)).
Thanks for reporting! I will investigate.
Mathieu
Quoted 4 lines
Thanks,
Ingo
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [PATCH for 5.1 3/3] rseq/selftests: Adapt number of threads to the number of detected cpus
Quoted 102 lines · ----- On Apr 19, 2019, at 8:41 AM, Mathieu Desnoyers…
Quoted 100 lines · ----- On Apr 19, 2019, at 6:38 AM, Ingo Molnar [email protected] wrote:
Quoted 66 lines · Mathieu Desnoyers
On smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online cpus at test runtime, and multiply that by 6 to have 6 rseq threads per cpu preempting each other.
Signed-off-by Mathieu Desnoyers <[email protected]>Cc Shuah Khan <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Joel Fernandes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Dave Watson <[email protected]>Cc Will Deacon <[email protected]>Cc Andi Kleen <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Chris Lameter <[email protected]>Cc Russell King <[email protected]>Cc Michael Kerrisk <[email protected]>Cc "Paul E . McKenney" <[email protected]>Cc Paul Turner <[email protected]>Cc Boqun Feng <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Maurer <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>Cc Linus Torvalds <[email protected]>--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/testing/selftests/rseq/run_param_test.sh +4
-1@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0 while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS}|| exit 1
tools/testing/selftests/rseq/run_param_test.sh +0
-1echo "Running compare-twice test ${TEST_NAME[$i]}" - ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} ||exit 1
tools/testing/selftests/rseq/run_param_test.sh +1
-0+ ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@}${EXTRA_ARGS} || exit 1
tools/testing/selftests/rseq/run_param_test.sh IFS="$OLDIFS" · line 45
Quoted 3 lines
tools/testing/selftests/rseq/run_param_test.sh +0
-0let "i++" done }BTW., when trying to build the rseq self-tests I get this build failure:
dagon:~/tip/tools/testing/selftests/rseq> make
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ -shared -fPICrseq.c-lpthread -o/home/mingo/tip/tools/testing/selftests/rseq/librseq.so
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_testgcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_percpu_ops_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpeqv_storev':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84: undefined reference to `.L8'
/usr/bin/ld:/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84:
undefined reference to `.L49'
/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpnev_storeoffp_load':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:141: undefined reference to `.L57'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x8): undefined reference to.L8' /usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x14): undefined reference to.L49'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x20): undefined reference to `.L55'
collect2: error: ld returned 1 exit status
make: *** [Makefile:22:
/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test] Error 1
Quoted 8 lines
Quoted 2 lines
Is this a known problem, or do I miss something from my build environment perhaps? Vanilla 64-bit Ubuntu 18.10 (Cosmic).
It works fine with gcc-7 (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) but indeed I get the same failure with gcc-8 (gcc version 8.0.1 20180414 (experimental) [trunk revision 259383] (Ubuntu 8-20180414-1ubuntu2)).
Thanks for reporting! I will investigate.
It looks like gcc-8 optimize away the target of asm goto labels when there are more than one of them on x86-64. I'll try to come up with a simpler reproducer.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [PATCH for 5.1 3/3] rseq/selftests: Adapt number of threads to the number of detected cpus
Quoted 112 lines · ----- On Apr 19, 2019, at 8:55 AM, Mathieu Desnoyers…
----- On Apr 19, 2019, at 8:41 AM, Mathieu Desnoyers
Quoted 109 lines · [email protected] wrote:
Quoted 100 lines · ----- On Apr 19, 2019, at 6:38 AM, Ingo Molnar [email protected] wrote:
Quoted 66 lines · Mathieu Desnoyers
On smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online cpus at test runtime, and multiply that by 6 to have 6 rseq threads per cpu preempting each other.
Signed-off-by Mathieu Desnoyers <[email protected]>Cc Shuah Khan <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Joel Fernandes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Dave Watson <[email protected]>Cc Will Deacon <[email protected]>Cc Andi Kleen <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Chris Lameter <[email protected]>Cc Russell King <[email protected]>Cc Michael Kerrisk <[email protected]>Cc "Paul E . McKenney" <[email protected]>Cc Paul Turner <[email protected]>Cc Boqun Feng <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Maurer <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>Cc Linus Torvalds <[email protected]>--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/testing/selftests/rseq/run_param_test.sh +4
-1@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0 while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS}|| exit 1
tools/testing/selftests/rseq/run_param_test.sh +0
-1echo "Running compare-twice test ${TEST_NAME[$i]}" - ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} ||exit 1
tools/testing/selftests/rseq/run_param_test.sh +1
-0+ ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@}${EXTRA_ARGS} || exit 1
tools/testing/selftests/rseq/run_param_test.sh IFS="$OLDIFS" · line 45
Quoted 3 lines
tools/testing/selftests/rseq/run_param_test.sh +0
-0let "i++" done }BTW., when trying to build the rseq self-tests I get this build failure:
dagon:~/tip/tools/testing/selftests/rseq> make
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ -shared -fPICrseq.c-lpthread -o/home/mingo/tip/tools/testing/selftests/rseq/librseq.so
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_testgcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_percpu_ops_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpeqv_storev':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84: undefined reference to `.L8'
/usr/bin/ld:/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84:
undefined reference to `.L49'
/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpnev_storeoffp_load':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:141: undefined reference to `.L57'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x8): undefined reference to.L8' /usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x14): undefined reference to.L49'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x20): undefined reference to `.L55'
collect2: error: ld returned 1 exit status
make: *** [Makefile:22:
/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test] Error 1Quoted 2 lines
Is this a known problem, or do I miss something from my build environment perhaps? Vanilla 64-bit Ubuntu 18.10 (Cosmic).
It works fine with gcc-7 (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) but indeed I get the same failure with gcc-8 (gcc version 8.0.1 20180414 (experimental) [trunk revision 259383] (Ubuntu 8-20180414-1ubuntu2)).
Quoted 5 lines
Quoted 1 line
Thanks for reporting! I will investigate.
It looks like gcc-8 optimize away the target of asm goto labels when there are more than one of them on x86-64. I'll try to come up with a simpler reproducer.
It appears to be related to gcc-8 mishandling combination of asm goto and thread-local storage input operands on x86-64. Here is a simple reproducer:
__thread int var;
static int fct(void)
{
asm goto ( "jmp %l[testlabel]\n\t"
: : [var] "m" (var) : : testlabel);
return 0;
testlabel:
return 1;
}int main()
{
return fct();
}building with gcc-7 -O2 is fine. Building with gcc-8 -O0 is fine too. Building with gcc-8 -O1 and -O2 fails with:
/tmp/ccuXTFfs.o: In function `main':
test-asm-goto.c:(.text.startup+0x1): undefined reference to `.L2'
collect2: error: ld returned 1 exit status
With gcc-7 -O2, the assembly of main has the .L2 label:
main:
.LFB1:
.cfi_startproc
#APP
# 5 "test-asm-goto.c" 1
jmp .L2
# 0 "" 2
#NO_APP
.L4:
.L3:
xorl %eax, %eax
ret
.L2:
movl $1, %eax
ret
.cfi_endproc
However, with gcc-8 -O2, it's missing:
main:
.LFB1:
.cfi_startproc
.L3:
#APP
# 5 "test-asm-goto.c" 1
jmp .L2
# 0 "" 2
#NO_APP
xorl %eax, %eax
ret
.cfi_endproc
It looks like we have a compiler issue. :-/
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [PATCH for 5.1 3/3] rseq/selftests: Adapt number of threads to the number of detected cpus
Quoted 125 lines · ----- On Apr 19, 2019, at 9:42 AM, Mathieu Desnoyers…
----- On Apr 19, 2019, at 8:55 AM, Mathieu Desnoyers
Quoted 112 lines · [email protected] wrote:
----- On Apr 19, 2019, at 8:41 AM, Mathieu Desnoyers
Quoted 109 lines · [email protected] wrote:
Quoted 100 lines · ----- On Apr 19, 2019, at 6:38 AM, Ingo Molnar [email protected] wrote:
Quoted 66 lines · Mathieu Desnoyers
On smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online cpus at test runtime, and multiply that by 6 to have 6 rseq threads per cpu preempting each other.
Signed-off-by Mathieu Desnoyers <[email protected]>Cc Shuah Khan <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Joel Fernandes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Dave Watson <[email protected]>Cc Will Deacon <[email protected]>Cc Andi Kleen <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Chris Lameter <[email protected]>Cc Russell King <[email protected]>Cc Michael Kerrisk <[email protected]>Cc "Paul E . McKenney" <[email protected]>Cc Paul Turner <[email protected]>Cc Boqun Feng <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Maurer <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>Cc Linus Torvalds <[email protected]>--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/testing/selftests/rseq/run_param_test.sh +4
-1@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0 while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS}|| exit 1
tools/testing/selftests/rseq/run_param_test.sh +0
-1echo "Running compare-twice test ${TEST_NAME[$i]}" - ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} ||exit 1
tools/testing/selftests/rseq/run_param_test.sh +1
-0+ ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@}${EXTRA_ARGS} || exit 1
tools/testing/selftests/rseq/run_param_test.sh IFS="$OLDIFS" · line 45
Quoted 3 lines
tools/testing/selftests/rseq/run_param_test.sh +0
-0let "i++" done }BTW., when trying to build the rseq self-tests I get this build failure:
dagon:~/tip/tools/testing/selftests/rseq> make
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ -shared -fPICrseq.c-lpthread -o/home/mingo/tip/tools/testing/selftests/rseq/librseq.so
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_testgcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_percpu_ops_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpeqv_storev':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84: undefined reference to `.L8'
/usr/bin/ld:/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84:
undefined reference to `.L49'
/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpnev_storeoffp_load':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:141: undefined reference to `.L57'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x8): undefined reference to.L8' /usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x14): undefined reference to.L49'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x20): undefined reference to `.L55'
collect2: error: ld returned 1 exit status
make: *** [Makefile:22:
/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test] Error 1Quoted 2 lines
Is this a known problem, or do I miss something from my build environment perhaps? Vanilla 64-bit Ubuntu 18.10 (Cosmic).
It works fine with gcc-7 (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) but indeed I get the same failure with gcc-8 (gcc version 8.0.1 20180414 (experimental) [trunk revision 259383] (Ubuntu 8-20180414-1ubuntu2)).
Quoted 5 lines
Quoted 1 line
Thanks for reporting! I will investigate.
It looks like gcc-8 optimize away the target of asm goto labels when there are more than one of them on x86-64. I'll try to come up with a simpler reproducer.
It appears to be related to gcc-8 mishandling combination of asm goto and thread-local storage input operands on x86-64. Here is a simple reproducer:
Quoted 8 lines
__threadint var;static int fct(void)
{ asm goto ( "jmp %l[testlabel]\n\t" : : [var] "m" (var) : : testlabel); return 0;testlabel:
FWIW, if I add an empty
asm volatile ("");here after the label, gcc-8 -O2 builds "something" which is a bogus assembler (an endless loop) :
main:
.LFB24:
.cfi_startproc
.L2:
subq $8, %rsp
.cfi_def_cfa_offset 16
#APP
# 6 "test-asm-goto.c" 1
jmp .L2
# 0 "" 2
#NO_APP
movl %fs:var@tpoff, %edx
leaq .LC0(%rip), %rsi
movl $1, %edi
xorl %eax, %eax
call __printf_chk@PLT
xorl %eax, %eax
addq $8, %rsp .cfi_def_cfa_offset 8
ret
.cfi_endproc
Thoughts ?
Thanks,
Mathieu
Quoted 61 lines
return 1; }int
main(){ return fct(); }building with gcc-7 -O2 is fine. Building with gcc-8 -O0 is fine too. Building with gcc-8 -O1 and -O2 fails with:
/tmp/ccuXTFfs.o: In function `main':
test-asm-goto.c:(.text.startup+0x1): undefined reference to `.L2'
collect2: error: ld returned 1 exit statusWith gcc-7 -O2, the assembly of main has the .L2 label:
main:
.LFB1:
.cfi_startproc
#APP
# 5 "test-asm-goto.c" 1jmp .L2# 0 "" 2
#NO_APP
.L4:
.L3:
xorl %eax, %eax
ret
.L2:
movl $1, %eax
ret
.cfi_endprocHowever, with gcc-8 -O2, it's missing:
main:
.LFB1:
.cfi_startproc
.L3:
#APP
# 5 "test-asm-goto.c" 1jmp .L2# 0 "" 2
#NO_APP
xorl %eax, %eax
ret
.cfi_endprocIt looks like we have a compiler issue. :-/
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
shuah
maintainerRe: [PATCH for 5.1 3/3] rseq/selftests: Adapt number of threads to the number of detected cpus
Quoted 162 lines · Mathieu Desnoyers
Quoted 125 lines · ----- On Apr 19, 2019, at 9:42 AM, Mathieu Desnoyers…
----- On Apr 19, 2019, at 8:55 AM, Mathieu Desnoyers
Quoted 112 lines · [email protected] wrote:
----- On Apr 19, 2019, at 8:41 AM, Mathieu Desnoyers
Quoted 109 lines · [email protected] wrote:
Quoted 100 lines · ----- On Apr 19, 2019, at 6:38 AM, Ingo Molnar [email protected] wrote:
Quoted 65 lines · Mathieu Desnoyers
On smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online cpus at test runtime, and multiply that by 6 to have 6 rseq threads per cpu preempting each other.
Signed-off-by Mathieu Desnoyers <[email protected]>Cc Shuah Khan <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Joel Fernandes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Dave Watson <[email protected]>Cc Will Deacon <[email protected]>Cc Andi Kleen <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Chris Lameter <[email protected]>Cc Russell King <[email protected]>Cc Michael Kerrisk <[email protected]>Cc "Paul E . McKenney" <[email protected]>Cc Paul Turner <[email protected]>Cc Boqun Feng <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Maurer <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>Cc Linus Torvalds <[email protected]>--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/testing/selftests/rseq/run_param_test.sh +4
-1@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0 while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS}|| exit 1
echo "Running compare-twice test ${TEST_NAME[$i]}"tools/testing/selftests/rseq/run_param_test.sh +0
-1- ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} ||exit 1
tools/testing/selftests/rseq/run_param_test.sh +1
-0+ ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@}Quoted 4 lines
${EXTRA_ARGS} || exit 1
let "i++"
done
}BTW., when trying to build the rseq self-tests I get this build failure:
dagon:~/tip/tools/testing/selftests/rseq> make
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ -shared -fPICrseq.c-lpthread -o/home/mingo/tip/tools/testing/selftests/rseq/librseq.so
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_testgcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_percpu_ops_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpeqv_storev':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84: undefined reference to `.L8'
/usr/bin/ld:/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84:
undefined reference to `.L49'
/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpnev_storeoffp_load':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:141: undefined reference to `.L57'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x8): undefined reference to.L8' /usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x14): undefined reference to.L49'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x20): undefined reference to `.L55'
collect2: error: ld returned 1 exit status
make: *** [Makefile:22:
/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test] Error 1Quoted 2 lines
Is this a known problem, or do I miss something from my build environment perhaps? Vanilla 64-bit Ubuntu 18.10 (Cosmic).
It works fine with gcc-7 (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) but indeed I get the same failure with gcc-8 (gcc version 8.0.1 20180414 (experimental) [trunk revision 259383] (Ubuntu 8-20180414-1ubuntu2)).
Quoted 5 lines
Quoted 1 line
Thanks for reporting! I will investigate.
It looks like gcc-8 optimize away the target of asm goto labels when there are more than one of them on x86-64. I'll try to come up with a simpler reproducer.
It appears to be related to gcc-8 mishandling combination of asm goto and thread-local storage input operands on x86-64. Here is a simple reproducer:
Quoted 8 lines
__threadint var;static int fct(void)
{ asm goto ( "jmp %l[testlabel]\n\t" : : [var] "m" (var) : : testlabel); return 0;testlabel:
FWIW, if I add an empty
asm volatile ("");here after the label, gcc-8 -O2 builds "something" which is a bogus assembler (an endless loop) :
main:
.LFB24:
.cfi_startproc
.L2:
subq $8, %rsp
.cfi_def_cfa_offset 16
#APP
# 6 "test-asm-goto.c" 1jmp .L2# 0 "" 2
#NO_APPmovl %fs:var@tpoff, %edx leaq .LC0(%rip), %rsi movl $1, %edi xorl %eax, %eax call __printf_chk@PLT xorl %eax, %eax addq $8, %rsp
Quoted 6 lines
.cfi_def_cfa_offset 8
ret
.cfi_endprocThoughts ?
Didn't see problems when I tested it before applying it to linux-kselftest next.
I have gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)
thanks,
-- Shuah
Mathieu Desnoyers
maintainerRe: [PATCH for 5.1 3/3] rseq/selftests: Adapt number of threads to the number of detected cpus
Quoted 168 lines · ----- On Apr 19, 2019, at 10:17 AM, shuah [email protected] wrote:
Quoted 167 lines · Mathieu Desnoyers
----- On Apr 19, 2019, at 9:42 AM, Mathieu Desnoyers
Quoted 125 lines · [email protected] wrote:
----- On Apr 19, 2019, at 8:55 AM, Mathieu Desnoyers
Quoted 112 lines · [email protected] wrote:
----- On Apr 19, 2019, at 8:41 AM, Mathieu Desnoyers
Quoted 109 lines · [email protected] wrote:
Quoted 100 lines · ----- On Apr 19, 2019, at 6:38 AM, Ingo Molnar [email protected] wrote:
Quoted 65 lines · Mathieu Desnoyers
On smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online cpus at test runtime, and multiply that by 6 to have 6 rseq threads per cpu preempting each other.
Signed-off-by Mathieu Desnoyers <[email protected]>Cc Shuah Khan <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Joel Fernandes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Dave Watson <[email protected]>Cc Will Deacon <[email protected]>Cc Andi Kleen <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Chris Lameter <[email protected]>Cc Russell King <[email protected]>Cc Michael Kerrisk <[email protected]>Cc "Paul E . McKenney" <[email protected]>Cc Paul Turner <[email protected]>Cc Boqun Feng <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Maurer <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>Cc Linus Torvalds <[email protected]>--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/testing/selftests/rseq/run_param_test.sh +4
-1@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0 while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS}|| exit 1
echo "Running compare-twice test ${TEST_NAME[$i]}"tools/testing/selftests/rseq/run_param_test.sh +0
-1- ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} ||exit 1
tools/testing/selftests/rseq/run_param_test.sh +1
-0+ ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@}Quoted 4 lines
${EXTRA_ARGS} || exit 1
let "i++"
done
}BTW., when trying to build the rseq self-tests I get this build failure:
dagon:~/tip/tools/testing/selftests/rseq> make
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ -shared -fPICrseq.c-lpthread -o/home/mingo/tip/tools/testing/selftests/rseq/librseq.so
gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_testgcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./basic_percpu_ops_test.c-lpthread -lrseq -o/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpeqv_storev':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84: undefined reference to `.L8'
/usr/bin/ld:/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84:
undefined reference to `.L49'
/usr/bin/ld:/tmp/ccuHTWnZ.o: in function `rseq_cmpnev_storeoffp_load':
/home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:141: undefined reference to `.L57'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x8): undefined reference to.L8' /usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x14): undefined reference to.L49'
/usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x20): undefined reference to `.L55'
collect2: error: ld returned 1 exit status
make: *** [Makefile:22:
/home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test] Error 1Quoted 2 lines
Is this a known problem, or do I miss something from my build environment perhaps? Vanilla 64-bit Ubuntu 18.10 (Cosmic).
It works fine with gcc-7 (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) but indeed I get the same failure with gcc-8 (gcc version 8.0.1 20180414 (experimental) [trunk revision 259383] (Ubuntu 8-20180414-1ubuntu2)).
Quoted 5 lines
Quoted 1 line
Thanks for reporting! I will investigate.
It looks like gcc-8 optimize away the target of asm goto labels when there are more than one of them on x86-64. I'll try to come up with a simpler reproducer.
It appears to be related to gcc-8 mishandling combination of asm goto and thread-local storage input operands on x86-64. Here is a simple reproducer:
Quoted 8 lines
__threadint var;static int fct(void)
{ asm goto ( "jmp %l[testlabel]\n\t" : : [var] "m" (var) : : testlabel); return 0;testlabel:
FWIW, if I add an empty
asm volatile ("");here after the label, gcc-8 -O2 builds "something" which is a bogus assembler (an endless loop) :
main:
.LFB24:
.cfi_startproc
.L2:
subq $8, %rsp
.cfi_def_cfa_offset 16
#APP
# 6 "test-asm-goto.c" 1jmp .L2# 0 "" 2
#NO_APPmovl %fs:var@tpoff, %edx leaq .LC0(%rip), %rsi movl $1, %edi xorl %eax, %eax call __printf_chk@PLT xorl %eax, %eax addq $8, %rsp.cfi_def_cfa_offset 8
ret
.cfi_endproc
Quoted 7 lines
Quoted 2 lines
Thoughts ?
Didn't see problems when I tested it before applying it to linux-kselftest next.
I have gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)
It really appears to be an optimization bug in gcc-8. Considering that bogus compilers are released in the wild, we can hardly justify using the compiler feature that triggers the bogus behavior, even if it gets fixed in the future.
I've prepared a patch that changes the way the __rseq_abi fields are passed to the inline asm. I pass the address of the __rseq_abi TLS as a register input operand rather than each individual field as "m" operand.
I will submit it in a separate thread.
By the way, it affects both x86-32 (building with gcc-8 -m32) and x86-64.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
shuah
maintainerRe: [PATCH for 5.1 3/3] rseq/selftests: Adapt number of threads to the number of detected cpus
Quoted 183 lines · Mathieu Desnoyers
Quoted 168 lines · ----- On Apr 19, 2019, at 10:17 AM, shuah [email protected] wrote:
Quoted 167 lines · Mathieu Desnoyers
----- On Apr 19, 2019, at 9:42 AM, Mathieu Desnoyers
Quoted 125 lines · [email protected] wrote:
----- On Apr 19, 2019, at 8:55 AM, Mathieu Desnoyers
Quoted 112 lines · [email protected] wrote:
----- On Apr 19, 2019, at 8:41 AM, Mathieu Desnoyers
Quoted 109 lines · [email protected] wrote:
Quoted 100 lines · ----- On Apr 19, 2019, at 6:38 AM, Ingo Molnar [email protected] wrote:
Quoted 65 lines · Mathieu Desnoyers
On smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online cpus at test runtime, and multiply that by 6 to have 6 rseq threads per cpu preempting each other.
Signed-off-by Mathieu Desnoyers <[email protected]>Cc Shuah Khan <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Joel Fernandes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Dave Watson <[email protected]>Cc Will Deacon <[email protected]>Cc Andi Kleen <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Chris Lameter <[email protected]>Cc Russell King <[email protected]>Cc Michael Kerrisk <[email protected]>Cc "Paul E . McKenney" <[email protected]>Cc Paul Turner <[email protected]>Cc Boqun Feng <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Maurer <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>Cc Linus Torvalds <[email protected]>--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)tools/testing/selftests/rseq/run_param_test.sh +4
-1@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0 while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS}|| exit 1
echo "Running compare-twice test ${TEST_NAME[$i]}"tools/testing/selftests/rseq/run_param_test.sh +0
-1- ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} ||exit 1
tools/testing/selftests/rseq/run_param_test.sh +1
-0+ ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@}Quoted 4 lines
${EXTRA_ARGS} || exit 1
let "i++"
done
}BTW., when trying to build the rseq self-tests I get this build failure:
dagon:~/tip/tools/testing/selftests/rseq> make gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ -shared -fPIC rseq.c -lpthread -o /home/mingo/tip/tools/testing/selftests/rseq/librseq.so gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ basic_test.c -lpthread -lrseq -o /home/mingo/tip/tools/testing/selftests/rseq/basic_test gcc -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ basic_percpu_ops_test.c -lpthread -lrseq -o /home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test /usr/bin/ld: /tmp/ccuHTWnZ.o: in function `rseq_cmpeqv_storev': /home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84: undefined reference to `.L8' /usr/bin/ld: /home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:84: undefined reference to `.L49' /usr/bin/ld: /tmp/ccuHTWnZ.o: in function `rseq_cmpnev_storeoffp_load': /home/mingo/tip/tools/testing/selftests/rseq/./rseq-x86.h:141: undefined reference to `.L57' /usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x8): undefined reference to `.L8' /usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x14): undefined reference to `.L49' /usr/bin/ld: /tmp/ccuHTWnZ.o:(__rseq_failure+0x20): undefined reference to `.L55' collect2: error: ld returned 1 exit status make: *** [Makefile:22: /home/mingo/tip/tools/testing/selftests/rseq/basic_percpu_ops_test] Error 1Quoted 2 lines
Is this a known problem, or do I miss something from my build environment perhaps? Vanilla 64-bit Ubuntu 18.10 (Cosmic).
It works fine with gcc-7 (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) but indeed I get the same failure with gcc-8 (gcc version 8.0.1 20180414 (experimental) [trunk revision 259383] (Ubuntu 8-20180414-1ubuntu2)).
Quoted 5 lines
Quoted 1 line
Thanks for reporting! I will investigate.
It looks like gcc-8 optimize away the target of asm goto labels when there are more than one of them on x86-64. I'll try to come up with a simpler reproducer.
It appears to be related to gcc-8 mishandling combination of asm goto and thread-local storage input operands on x86-64. Here is a simple reproducer:
Quoted 8 lines
__threadint var;static int fct(void)
{ asm goto ( "jmp %l[testlabel]\n\t" : : [var] "m" (var) : : testlabel); return 0;testlabel:
FWIW, if I add an empty
asm volatile ("");here after the label, gcc-8 -O2 builds "something" which is a bogus assembler (an endless loop) :
main:
.LFB24:
.cfi_startproc
.L2:
subq $8, %rsp
.cfi_def_cfa_offset 16
#APP
# 6 "test-asm-goto.c" 1jmp .L2# 0 "" 2
#NO_APPmovl %fs:var@tpoff, %edx leaq .LC0(%rip), %rsi movl $1, %edi xorl %eax, %eax call __printf_chk@PLT xorl %eax, %eax addq $8, %rsp.cfi_def_cfa_offset 8
ret
.cfi_endprocQuoted 7 lines
Quoted 2 lines
Thoughts ?
Didn't see problems when I tested it before applying it to linux-kselftest next.
I have gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)
It really appears to be an optimization bug in gcc-8. Considering that bogus compilers are released in the wild, we can hardly justify using the compiler feature that triggers the bogus behavior, even if it gets fixed in the future.
Quoted 9 lines
I've prepared a patch that changes the way the
__rseq_abifields are passed to the inline asm. I pass the address of the__rseq_abiTLS as a register input operand rather than each individual field as "m" operand.I will submit it in a separate thread.
By the way, it affects both x86-32 (building with gcc-8 -m32) and x86-64.
Should I drop this patch that is currently in linux-kseltest next? Just confirming if your new patch is supposed to be applied on top of this one or not?
thanks,
-- Shuah
Mathieu Desnoyers
maintainerRe: [PATCH for 5.1 3/3] rseq/selftests: Adapt number of threads to the number of detected cpus
----- On Apr 19, 2019, at 2:57 PM, shuah [email protected] wrote:
[...]
Quoted 3 lines
Should I drop this patch that is currently in linux-kseltest next? Just confirming if your new patch is supposed to be applied on top of this one or not?
We should keep this patch in linux-kselftest next. The fix applies on top of it.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
shuah
maintainerRe: [PATCH for 5.1 3/3] rseq/selftests: Adapt number of threads to the number of detected cpus
Quoted 9 lines · Mathieu Desnoyers
----- On Apr 19, 2019, at 2:57 PM, shuah [email protected] wrote:
[...]Quoted 3 lines
Should I drop this patch that is currently in linux-kseltest next? Just confirming if your new patch is supposed to be applied on top of this one or not?
We should keep this patch in linux-kselftest next. The fix applies on top of it.
Will do. Thanks for a quick response.
-- Shuah
tip-bot for Mathieu Desnoyers
merge bot[tip:core/rseq] rseq/selftests: Adapt number of threads to the number of detected CPUs
In reply to Mathieu Desnoyersrseq/selftests: Adapt number of threads to the number of detected CPUs
Commit-ID: 03a33ecc26af28ca9535de2b0612c54deafa9728 Gitweb: https://git.kernel.org/tip/03a33ecc26af28ca9535de2b0612c54deafa9728 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Tue, 5 Mar 2019 14:47:55 -0500 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 19 Apr 2019 12:42:14 +0200
rseq/selftests: Adapt number of threads to the number of detected CPUs
On smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online CPUs at test runtime, and multiply that by 6 to have 6 rseq threads per CPU preempting each other.
--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
tools/testing/selftests/rseq/run_param_test.sh +5 -2
@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"
@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0 while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 echo "Running compare-twice test ${TEST_NAME[$i]}" - ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 let "i++" done }
tip-bot for Mathieu Desnoyers
merge bot[tip:core/rseq] rseq/selftests: Adapt number of threads to the number of detected cpus
In reply to Mathieu Desnoyersrseq/selftests: Adapt number of threads to the number of detected cpus
Commit-ID: 90083998312324599cb2a546020755cdc934de0f Gitweb: https://git.kernel.org/tip/90083998312324599cb2a546020755cdc934de0f Author: Mathieu Desnoyers <[email protected]> AuthorDate: Tue, 5 Mar 2019 14:47:55 -0500 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 19 Apr 2019 12:39:32 +0200
rseq/selftests: Adapt number of threads to the number of detected cpus
On smaller systems, running a test with 200 threads can take a long time on machines with smaller number of CPUs.
Detect the number of online cpus at test runtime, and multiply that by 6 to have 6 rseq threads per cpu preempting each other.
--- tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
tools/testing/selftests/rseq/run_param_test.sh +5 -2
@@ -1,6 +1,8 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ or MIT +NR_CPUS=`grep '^processor' /proc/cpuinfo | wc -l` + EXTRA_ARGS=${@} OLDIFS="$IFS"
@@ -28,15 +30,16 @@ IFS="$OLDIFS" REPS=1000 SLOW_REPS=100 +NR_THREADS=$((6*${NR_CPUS})) function do_tests() { local i=0 while [ "$i" -lt "${#TEST_LIST[@]}" ]; do echo "Running test ${TEST_NAME[$i]}" - ./param_test ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 echo "Running compare-twice test ${TEST_NAME[$i]}" - ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} ${@} ${EXTRA_ARGS} || exit 1 + ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_ARGS} || exit 1 let "i++" done }
Mathieu Desnoyers
maintainerQuoted 5 lines · ----- On Mar 5, 2019, at 2:47 PM, Mathieu Desnoyers…
Those changes aiming at 5.1 include one comment cleanup, the removal of the
rseq_lenfield from the task struct which serves no purpose considering that the struct size is fixed by the ABI, and a selftest improvement adapting the number of threads to the number of detected CPUs, which is nicer for smaller systems.
For those interested, here is a status update on how things are evolving in terms of rseq Linux ecosystem integration:
I've been working with glibc maintainers for the past months to get rseq registration integrated into glibc. The patchset is awaiting feedback from glibc maintainers at this point. An important part of that integration is the user-level ABI defining interaction between the executable and libraries wishing to register rseq within the same process. This is needed because the rseq system call only supports a single rseq registration per thread (this was done on purpose). If all goes well we should see rseq integration in glibc as part of the glibc 2.30 release in August 2019.
For those interested in upcoming rseq kernel patches I have ready, those are available at [1].
The main reason why we're not seeing more users of rseq out there right now is because the user-level ABI to interact between libc, applications, and early adopter libraries needs to be specified before projects start using rseq. An early adopter of rseq should not be incompatible with a glibc which introduces rseq registration support.
Once glibc integration is done, here are a few things I have ready:
* NUMA node ID in TLS
Having the NUMA node ID available in a TLS variable would allow glibc to perform interesting NUMA performance improvements within its locking implementation, so I have a patch adding NUMA node ID support to rseq as a new rseq system call flag.
* Adaptative mutex improvements
I have done a prototype using rseq to implement an adaptative mutex which can detect preemption using a rseq critical section. This ensures the thread doesn't continue to busy-loop after it returns from preemption, and calls sys_futex() instead. This is part of a user-space prototype branch [2], and does not require any kernel change.
* cpu_opv system call
Use-cases requiring access to remote per-cpu data such as memory migration in a memory allocator and access to specific per-cpu buffers from a ring-buffer consumer will end up requiring this additional system call. I'm awaiting a broader adoption of rseq (which depends on glibc integration) for simpler use-cases before pushing the cpu_opv system call again.
Thanks,
Mathieu
[1] https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git/
[2] https://github.com/compudj/rseq-test/blob/adapt-lock/test-rseq-adaptative-lock.c
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Peter Zijlstra
maintainerQuoted 6 lines · Mathieu Desnoyers
* NUMA node ID in TLS
Having the NUMA node ID available in a TLS variable would allow glibc to perform interesting NUMA performance improvements within its locking implementation, so I have a patch adding NUMA node ID support to rseq as a new rseq system call flag.
Details? There's just not much room in the futex word, and futexes themselves are not numa aware.
Before all this spectre nonsense; tglx and me were looking at a futex2 syscall that would, among other things, cure this.
Quoted 7 lines
* Adaptative mutex improvements
I have done a prototype using rseq to implement an adaptative mutex which can detect preemption using a rseq critical section. This ensures the thread doesn't continue to busy-loop after it returns from preemption, and calls
sys_futex()instead. This is part of a user-space prototype branch [2], and does not require any kernel change.
I'm still not convinced that is actually the right way to go about things. The kernel heuristic is spin while the _owner_ runs, and we don't get preempted, obviously.
And the only userspace spinning that makes sense is to cover the cost of the syscall. Now Obviously PTI wrecked everything, but before that syscalls were actually plenty fast and you didn't need many cmpxchg cycles to amortize the syscall itself -- which could then do kernel side adaptive spinning (when required).
Mathieu Desnoyers
maintainerQuoted 10 lines · ----- On Mar 5, 2019, at 4:58 PM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
* NUMA node ID in TLS
Having the NUMA node ID available in a TLS variable would allow glibc to perform interesting NUMA performance improvements within its locking implementation, so I have a patch adding NUMA node ID support to rseq as a new rseq system call flag.
Details? There's just not much room in the futex word, and futexes themselves are not numa aware.
It was discussed in this libc-alpha mailing list thread:
(adding the relevant people in CC)
I'd like to hear in more details on how they intend to design NUMA-aware spinlocks within glibc. All I know is that quick access to the node ID would help for this.
I would suspect we could split a lock into per-numa-node locks. Grabbing the local numa lock would then allow grabbing the global lock. This should help reducing remote NUMA accesses on the global lock in contended cases, but I'm really just guessing here.
Quoted 3 lines
Before all this spectre nonsense; tglx and me were looking at a futex2 syscall that would, among other things, cure this.
The email thread I point to above talks about "spinlocks", so I'm not sure whether their intent is to apply this to mutexes as well.
Quoted 13 lines
Quoted 8 lines
* Adaptative mutex improvements
I have done a prototype using rseq to implement an adaptative mutex which can detect preemption using a rseq critical section. This ensures the thread doesn't continue to busy-loop after it returns from preemption, and calls
sys_futex()instead. This is part of a user-space prototype branch [2], and does not require any kernel change.I'm still not convinced that is actually the right way to go about things. The kernel heuristic is spin while the _owner_ runs, and we don't get preempted, obviously.
Quoted 5 lines
And the only userspace spinning that makes sense is to cover the cost of the syscall. Now Obviously PTI wrecked everything, but before that syscalls were actually plenty fast and you didn't need many cmpxchg cycles to amortize the syscall itself -- which could then do kernel side adaptive spinning (when required).
Indeed with PTI the system calls are back to their slow self. ;)
You point about owner is interesting. Perhaps there is one tweak that I should add in there. We could write the owner thread ID in the lock word.
When trying to grab a lock, one of a few situations can happen:
- It's unlocked, so we grab it by storing our thread ID,
- It's locked, and we can fetch the CPU number of the thread owning it if we can access its (struct rseq *)->cpu_id through a lookup using its thread ID, We can then check whether it's the same CPU we are running on.
- If so, we _know_ we should let the owner run, so we call futex right away, no spinning. We can even boost it for priority inheritance mutexes,
- If it's owned by a thread which was last running on a different CPU, then it may make sense to actively try to grab the lock by spinning up to a certain number of loops (which can be either fixed or adaptative). After that limit, call futex. If preempted while looping, call futex.
Do you see this as an improvement over what exists today, or am I on the wrong track ?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Peter Zijlstra
maintainerQuoted 15 lines · Mathieu Desnoyers
Quoted 10 lines · ----- On Mar 5, 2019, at 4:58 PM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
* NUMA node ID in TLS
Having the NUMA node ID available in a TLS variable would allow glibc to perform interesting NUMA performance improvements within its locking implementation, so I have a patch adding NUMA node ID support to rseq as a new rseq system call flag.
Details? There's just not much room in the futex word, and futexes themselves are not numa aware.
It was discussed in this libc-alpha mailing list thread:
Quoted 7 lines
(adding the relevant people in CC)
I'd like to hear in more details on how they intend to design NUMA-aware spinlocks within glibc. All I know is that quick access to the node ID would help for this.
Userspace spinlocks are a trainwreck anyway. The only case where they can possibly work is when there's only a single thread on every cpu. Pretty much any other scenario is fail; see why we have paravirt spinlocks.
Peter Zijlstra
maintainerQuoted 18 lines · Mathieu Desnoyers
Quoted 12 lines
Quoted 7 lines
* Adaptative mutex improvements
I have done a prototype using rseq to implement an adaptative mutex which can detect preemption using a rseq critical section. This ensures the thread doesn't continue to busy-loop after it returns from preemption, and calls
sys_futex()instead. This is part of a user-space prototype branch [2], and does not require any kernel change.I'm still not convinced that is actually the right way to go about things. The kernel heuristic is spin while the _owner_ runs, and we don't get preempted, obviously.
Quoted 5 lines
And the only userspace spinning that makes sense is to cover the cost of the syscall. Now Obviously PTI wrecked everything, but before that syscalls were actually plenty fast and you didn't need many cmpxchg cycles to amortize the syscall itself -- which could then do kernel side adaptive spinning (when required).
Quoted 4 lines
Indeed with PTI the system calls are back to their slow self. ;)
You point about owner is interesting. Perhaps there is one tweak that I should add in there. We could write the owner thread ID in the lock word.
This is already required for PI (and I think robust) futexes. There have been proposals for FUTEX_LOCK and FUTEX_UNLOCK (!PI) primitives that require the same.
Waiman had some patches; but I think all went under because 'important' stuff happened.
Quoted 6 lines
When trying to grab a lock, one of a few situations can happen:
- It's unlocked, so we grab it by storing our thread ID,
- It's locked, and we can fetch the CPU number of the thread owning it if we can access its (struct rseq *)->cpu_id through a lookup using its thread ID, We can then check whether it's the same CPU we are running on.
That might just work with threads (private futexes; which are the majority these these I think), but will obviously not work with regular (shared) futexes.
Quoted 9 lines
- If so, we _know_ we should let the owner run, so we call futex right away, no spinning. We can even boost it for priority inheritance mutexes,
- If it's owned by a thread which was last running on a different CPU, then it may make sense to actively try to grab the lock by spinning up to a certain number of loops (which can be either fixed or adaptative). After that limit, call futex. If preempted while looping, call futex.Do you see this as an improvement over what exists today, or am I on the wrong track ?
That's probably better than what they have today. Last time I looked at libc pthread I got really sad -- arguably that was a long time ago, and some of that stuff is because POSIX, but still.
Some day we should redesign all that.. futex2 etc.
Mathieu Desnoyers
maintainerQuoted 34 lines · ----- On Mar 6, 2019, at 3:30 AM, Peter Zijlstra [email protected] wrote:
Quoted 18 lines · Mathieu Desnoyers
Quoted 12 lines
Quoted 7 lines
* Adaptative mutex improvements
I have done a prototype using rseq to implement an adaptative mutex which can detect preemption using a rseq critical section. This ensures the thread doesn't continue to busy-loop after it returns from preemption, and calls
sys_futex()instead. This is part of a user-space prototype branch [2], and does not require any kernel change.I'm still not convinced that is actually the right way to go about things. The kernel heuristic is spin while the _owner_ runs, and we don't get preempted, obviously.
Quoted 5 lines
And the only userspace spinning that makes sense is to cover the cost of the syscall. Now Obviously PTI wrecked everything, but before that syscalls were actually plenty fast and you didn't need many cmpxchg cycles to amortize the syscall itself -- which could then do kernel side adaptive spinning (when required).
Quoted 4 lines
Indeed with PTI the system calls are back to their slow self. ;)
You point about owner is interesting. Perhaps there is one tweak that I should add in there. We could write the owner thread ID in the lock word.
This is already required for PI (and I think robust) futexes. There have been proposals for
FUTEX_LOCKandFUTEX_UNLOCK(!PI) primitives that require the same.Waiman had some patches; but I think all went under because 'important' stuff happened.
Quoted 2 lines
When trying to grab a lock, one of a few situations can happen:
Quoted 8 lines
Quoted 4 lines
- It's unlocked, so we grab it by storing our thread ID,
- It's locked, and we can fetch the CPU number of the thread owning it if we can access its (struct rseq *)->cpu_id through a lookup using its thread ID, We can then check whether it's the same CPU we are running on.That might just work with threads (private futexes; which are the majority these these I think), but will obviously not work with regular (shared) futexes.
If we have enough space available either in the lock word or just nearby, we could write the CPU number that was current when the thread owning the lock grabbed it. Considering that it should be infrequent that threads get migrated to other CPUs while holding the lock, it might be a good enough heuristic to figure out whether a thread needs to busy-wait or immediately call futex.
Writing the CPU number would work both for private and shared futexes.
Quoted 8 lines
Quoted 8 lines
- If so, we _know_ we should let the owner run, so we call futex right away, no spinning. We can even boost it for priority inheritance mutexes,
- If it's owned by a thread which was last running on a different CPU, then it may make sense to actively try to grab the lock by spinning up to a certain number of loops (which can be either fixed or adaptative). After that limit, call futex. If preempted while looping, call futex.
Quoted 8 lines
Quoted 2 lines
Do you see this as an improvement over what exists today, or am I on the wrong track ?
That's probably better than what they have today. Last time I looked at libc pthread I got really sad -- arguably that was a long time ago, and some of that stuff is because POSIX, but still.
Some day we should redesign all that.. futex2 etc.
It sounds like an interesting topic to bring up at the next LPC! In the meantime, a good start would be to state the desiderata of what requirements should be covered by this redesign.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 11 lines · Mathieu Desnoyers
Mathieu Desnoyers (3):
rseq: cleanup: Reflect removal of event counter in comments
rseq: cleanup: removerseq_lenfromtask_structrseq/selftests: Adapt number of threads to the number of detected cpusarch/arm/kernel/signal.c | 3 +-- arch/x86/kernel/signal.c | 5 +---- include/linux/sched.h | 4 ---- kernel/rseq.c | 9 +++------ tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 5 files changed, 10 insertions(+), 18 deletions(-)
Quoted 15 lines · Mathieu Desnoyers
Those changes aiming at 5.1 include one comment cleanup, the removal of the
rseq_lenfield from the task struct which serves no purpose considering that the struct size is fixed by the ABI, and a selftest improvement adapting the number of threads to the number of detected CPUs, which is nicer for smaller systems.Thanks,
Mathieu
Mathieu Desnoyers (3):
rseq: cleanup: Reflect removal of event counter in comments
rseq: cleanup: removerseq_lenfromtask_structrseq/selftests: Adapt number of threads to the number of detected cpus
Quoted 6 lines
arch/arm/kernel/signal.c | 3 +-- arch/x86/kernel/signal.c | 5 +---- include/linux/sched.h | 4 ---- kernel/rseq.c | 9 +++------ tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 5 files changed, 10 insertions(+), 18 deletions(-)
Looks good, I've applied these to tip:core/rseq to make sure they don't miss the v5.2 merge window.
(Let me know if you wanted to handle this differently.)
Thanks,
Ingo
Mathieu Desnoyers
maintainerQuoted 24 lines · ----- On Apr 19, 2019, at 6:41 AM, Ingo Molnar [email protected] wrote:
Quoted 15 lines · Mathieu Desnoyers
Those changes aiming at 5.1 include one comment cleanup, the removal of the
rseq_lenfield from the task struct which serves no purpose considering that the struct size is fixed by the ABI, and a selftest improvement adapting the number of threads to the number of detected CPUs, which is nicer for smaller systems.Thanks,
Mathieu
Mathieu Desnoyers (3):
rseq: cleanup: Reflect removal of event counter in comments
rseq: cleanup: removerseq_lenfromtask_structrseq/selftests: Adapt number of threads to the number of detected cpusQuoted 6 lines
arch/arm/kernel/signal.c | 3 +-- arch/x86/kernel/signal.c | 5 +---- include/linux/sched.h | 4 ---- kernel/rseq.c | 9 +++------ tools/testing/selftests/rseq/run_param_test.sh | 7 +++++-- 5 files changed, 10 insertions(+), 18 deletions(-)
Quoted 4 lines
Looks good, I've applied these to tip:core/rseq to make sure they don't miss the v5.2 merge window.
(Let me know if you wanted to handle this differently.)
That's fine by me!
Thanks,
Mathieu
Quoted 4 lines
Thanks,
Ingo
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com