rcu: idle changes v2
Paul,
This is the second version of my rcu/idle-for-v3.7 which rebases your rcu/idle to address reviews.
It has the same modifications than rcu/idle-for-v3.7 plus an update of:
- "rcu: Exit RCU extended QS on user preemption" : update changelog and comments after a discussion with Peterz. And compile conditionally schedule_user().
- "x86: Use the new schedule_user API on userspace preemption" : build time choose between calling schedule and schedule_user.
I'm only reposting these two patches.
The new branch is:
git://github.com/fweisbec/linux-dynticks.git
rcu/idle-v2-for-v3.7
Thanks.
PATCH 1/2 rcu: Exit RCU extended QS on user preemption
When exceptions or irq are about to resume userspace, if the task needs to be rescheduled, the arch low level code calls schedule() directly.
If we call it, it is because we have the TIF_RESCHED flag:
- It can be set after random local calls to set_need_resched() (RCU, drm, ...)
- A wake up happened and the CPU needs preemption. This can happen in several ways:
* Remotely: the remote waking CPU has set TIF_RESCHED and send the wakee an IPI to schedule the new task.
* Remotely enqueued: the remote waking CPU sends an IPI to the target and the wake up is made by the target.
* Locally: waking CPU == wakee CPU and the wakeup is done locally. set_need_resched() is called without IPI.
In the case of local and remotely enqueued wake ups, the tick can be restarted when we enqueue the new task and RCU can exit the extended quiescent state at the same time. Then by the time we reach irq exit path and we call schedule, we are not in RCU user mode.
But if we call schedule() only because something called set_need_resched(), RCU may still be in user mode when we reach schedule.
Also if a wake up is done remotely, the CPU might see the TIF_RESCHED flag and call schedule while the IPI has not yet happen to restart the tick and exit RCU user mode.
We need to manually protect against these corner cases.
Create a new API schedule_user() that calls schedule() inside rcu_user_exit()-rcu_user_enter() in order to protect it. Archs will need to rely on it now to implement user preemption safely.
--- kernel/sched/core.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-)
kernel/sched/core.c +15 -0
@@ -3463,6 +3463,21 @@ asmlinkage void __sched schedule(void) }EXPORT_SYMBOL (schedule); +#ifdef CONFIG_RCU_USER_QS +asmlinkage void __sched schedule_user(void) +{ +/* +* If we come here after a random call to set_need_resched(), +* or we have been woken up remotely but the IPI has not yet arrived, +* we haven't yet exited the RCU idle mode. Do it here manually until +* we find a better solution. +*/ +rcu_user_exit (); +schedule (); +rcu_user_enter (); +} +#endif +/** * schedule_preempt_disabled - called with preemption disabled *
Signature · 1 line
1.7.5.4
PATCH 2/2 x86: Use the new schedule_user API on userspace preemption
In reply to Frederic WeisbeckerThis way we can exit the RCU extended quiescent state before we schedule a new task from irq/exception exit.
--- arch/x86/include/asm/rcu.h | 12 ++++++++++++ arch/x86/kernel/entry_64.S | 9 +++++---- 2 files changed, 17 insertions(+), 4 deletions(-)
arch/x86/include/asm/rcu.h +12 -0
@@ -1,6 +1,8 @@#ifndef _ASM_X86_RCU_H#define _ASM_X86_RCU_H +#ifndef __ASSEMBLY__ +#include <linux/rcupdate.h> #include <asm/ptrace.h>
@@ -17,4 +19,14 @@ static inline void exception_exit(struct pt_regs *regs)#endif } +#else /* __ASSEMBLY__ */ + +#ifdef CONFIG_RCU_USER_QS +# define SCHEDULE_USER call schedule_user +#else +# define SCHEDULE_USER call schedule +#endif + +#endif /* !__ASSEMBLY__ */ +#endif
arch/x86/kernel/entry_64.S +5 -4
@@ -56,6 +56,7 @@ #include <asm/ftrace.h> #include <asm/percpu.h> #include <asm/asm.h> +#include <asm/rcu.h> #include <linux/err.h> /* Avoid __ASSEMBLER__'ifying <linux/audit.h> just for this. */
@@ -565,7 +566,7 @@ sysret_careful: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) pushq_cfi %rdi - call schedule + SCHEDULE_USER popq_cfi %rdi jmp sysret_check
@@ -678,7 +679,7 @@ int_careful: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) pushq_cfi %rdi - call schedule + SCHEDULE_USER popq_cfi %rdi DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF
@@ -974,7 +975,7 @@ retint_careful: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) pushq_cfi %rdi - call schedule + SCHEDULE_USER popq_cfi %rdi GET_THREAD_INFO(%rcx) DISABLE_INTERRUPTS(CLBR_NONE)
@@ -1449,7 +1450,7 @@ paranoid_userspace: paranoid_schedule: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_ANY) - call schedule + SCHEDULE_USER DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF jmp paranoid_userspace
Signature · 1 line
1.7.5.4
Quoted 15 lines · Frederic Weisbecker
Paul,
This is the second version of my rcu/idle-for-v3.7 which rebases your rcu/idle to address reviews.
It has the same modifications than rcu/idle-for-v3.7 plus an update of:
- "rcu: Exit RCU extended QS on user preemption" : update changelog and comments after a discussion with Peterz. And compile conditionally
schedule_user().- "x86: Use the new
schedule_userAPI on userspace preemption" : build time choose between calling schedule andschedule_user.
Quoted 6 lines
I'm only reposting these two patches.
The new branch is:
git://github.com/fweisbec/linux-dynticks.gitrcu/idle-v2-for-v3.7
Pulled, thank you!
I also modified the first patch in the series to remove the exports based on Josh's patch (7ece55a4: Don't declare trace_*_rcuidle functions in modules) in -tip.
Thanx, Paul