entry/rcu: Check TIF_RESCHED _after_ delayed RCU wake-up
Subsystems (2)
Frederic Weisbecker
maintainerPATCH 1/1 entry/rcu: Check TIF_RESCHED _after_ delayed RCU wake-up
RCU sometimes needs to perform a delayed wake up for specific kthreads handling callbacks offloading (RCU_NOCB). This is handled through timers and upon entry to idle (also guest and user on nohz_full).
However the delayed wake-up on kernel exit is actually performed after the thread flags are fetched towards the fast path check for work to do on exit to user. As a result, and if there is no other pending work to do upon that kernel exit, the current task will resume to userspace with TIF_RESCHED set and the pending wake up ignored.
Fix this with fetching the thread flags _after_ the delayed RCU-nocb kthread wake-up.
--- kernel/entry/common.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/entry/common.c +2 -1
@@ -192,13 +192,14 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,static void exit_to_user_mode_prepare (struct pt_regs *regs) { -unsigned long ti_work =read_thread_flags (); +unsigned long ti_work;lockdep_assert_irqs_disabled ();/* Flush pending rcuog wakeup before the last need_resched() check */ tick_nohz_user_enter_prepare (); + ti_work =read_thread_flags ();if (unlikely (ti_work &EXIT_TO_USER_MODE_WORK )) ti_work =exit_to_user_mode_loop (regs, ti_work);
Signature · 1 line
2.25.1
Quoted 10 lines · Frederic Weisbecker
RCU sometimes needs to perform a delayed wake up for specific kthreads handling offloaded callbacks (
RCU_NOCB). These wakeups are performed by timers and upon entry to idle (also to guest and to user onnohz_full).However the delayed wake-up on kernel exit is actually performed after the thread flags are fetched towards the fast path check for work to do on exit to user. As a result, and if there is no other pending work to do upon that kernel exit, the current task will resume to userspace with
TIF_RESCHEDset and the pending wake up ignored.
Quoted 5 lines
Fix this with fetching the thread flags _after_ the delayed RCU-nocb kthread wake-up.
Fixes 47b8ff194c1f ("entry: Explicitly flush pending rcuog wakeup before last rescheduling point")Signed-off-by Frederic Weisbecker <[email protected]>
Queued and pushed, thank you!
Thanx, Paul
Quoted 27 lines
--- kernel/entry/common.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)kernel/entry/common.c +2
-1@@ -192,13 +192,14 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,static void exit_to_user_mode_prepare (struct pt_regs *regs) { -unsigned long ti_work =read_thread_flags (); +unsigned long ti_work;lockdep_assert_irqs_disabled ();/* Flush pending rcuog wakeup before the last need_resched() check */ tick_nohz_user_enter_prepare (); + ti_work =read_thread_flags ();if (unlikely (ti_work &EXIT_TO_USER_MODE_WORK )) ti_work =exit_to_user_mode_loop (regs, ti_work);Signature · 1 line
2.25.1