Skip to contentSkip to conversation

Proxy Execution fixes for v7.1-rc

PATCH Superseded John Stultz 4 participants 20 messages Apr 27 – May 1

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Proxy Execution fixes for v7.1-rc

Hey All,
So in testing with the full Proxy Execution series,
Vineeth Pillai managed to trip some interesting bugs which initially looked to be KVM or RCU related[1], which he later diagnosed as Proxy Execution related and created a useful test driver to reproduce.

I found these same issues could be triggered with the upstream portions of Proxy Execution, so I wanted to send along these fixes for 7.1-rc

Again, a huge thanks to Vineeth for uncovering these issues that have evaded all my stress testing so far!

Thanks
-john

[1]: https://lore.kernel.org/lkml/[email protected]/

Cc Vineeth Pillai <[email protected]>
Cc Sonam Sanju <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Kunwu Chan <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Qais Yousef <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Will Deacon <[email protected]>
Cc Waiman Long <[email protected]>
Cc Boqun Feng <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Metin Kaya <[email protected]>
Cc Xuewen Yan <[email protected]>
Cc K Prateek Nayak <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Daniel Lezcano <[email protected]>
Cc Suleiman Souhlal <[email protected]>
Cc kuyo chang <[email protected]>

John Stultz (2):
sched: proxy-exec: Close race causing workqueue work being delayed
locking: mutex: Fix proxy-exec potentially deactivating tasks marked
TASK_RUNNING

 kernel/locking/mutex.c |  1 +
 kernel/sched/core.c    | 11 +++++++++++
 2 files changed, 12 insertions(+)
Signature · 1 line
2.54.0.545.g6539524ca2-goog
#1 lore raw

PATCH 1/2 sched: proxy-exec: Close race causing workqueue work being delayed

Vineeth reported seeing a KVM related deadlock connected to work queue lockups using the android17-6.18 tree, which has Proxy Execution enabled (using the full patch stack), but I've subsequently reproduced it on v7.1-rc1.

On further debugging he found:
- kvm-irqfd-cleanup workqueue and rcu_gp lands in a per-cpu pwq(work queue pool)
- one of kvm-irqfd-cleanup worker(say A) takes a mutex and then calls synchronize_srcu_expedited()
- one other kvm-irqfd-cleanup worker worker(Say B) tries to acquire the lock and then gets blocked
- On the way to blocking, this cpu gets an IPI and on return from IPI, it calls __schedule() and did not get to complete workqueue accounting(worker->sleeping = 0 and decrementing pool->nr_running). This is done in sched_submit_work() -> wq_worker_sleeping() called from schedule() and we got preempted before that.
- proxy execution doesn't immediately take it off run queue as p->blocked_on is set during __mutex_lock
- Next time when B is picked for running, it notices A(mutex holder) is not on a runqueue and then blocks B. find_proxy_task() -> proxy_deactivate() -> block_task()
- And things are then stuck. A is waiting for the workqueue to be run, but B can't run the workqueue as it is blocked on A.

The trouble is that with Proxy Execution, in __mutex_lock_common() we set the task state to TASK_UNINTERRUPTIBLE, and set blocked_on before calling into schedule(), where sched_submit_work() will be called.

But if an IPI comes in before we call schedule() the interrupt will call __schedule(SM_PREEMPT) directly. This causes the scheduler to see the current task as blocked_on, and deactivate it (because the owner is off the runqueue).

Since its deactivated, it wont' be run, and it won't get to call sched_submit_work().

Without proxy-execution, the SM_PREEMPT case will prevent the task from being dequeued, and it can be reselected again and run, which will allow it to finish calling into schedule() and calling sched_submit_work() before actually blocking.

So we need to make sure on the SM_PREEMPT case, if current is marked as blocked_on, we should clear the blocked_on state and mark the task RUNNABLE so the task can be selected to complete its call to schedule() -> sched_submit_work().

Now because we cleared BLOCKED_ON and set the task RUNNABLE, the task will be able to be selected and run again and loop back in __mutex_lock_common() where it can re-set the blocked_on state and call back into schedule() in order to properly be chosen as a donor.

Many thanks to Vineeth for figuring this very obscure race out and for implementing a test tool to make it easily reproducible!

---
Cc: Vineeth Pillai <[email protected]>
Cc: Sonam Sanju <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Kunwu Chan <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: Qais Yousef <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Waiman Long <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Metin Kaya <[email protected]>
Cc: Xuewen Yan <[email protected]>
Cc: K Prateek Nayak <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Daniel Lezcano <[email protected]>
Cc: Suleiman Souhlal <[email protected]>
Cc: kuyo chang <[email protected]>
Cc: hupu <[email protected]>
Cc: [email protected]

---
 kernel/sched/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
kernel/sched/core.c +11 -0
index da20fb6ea25ae..5f684caefd8b2 100644
@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)
 		try_to_block_task(rq, prev, &prev_state,
 				  !task_is_blocked(prev));
 		switch_count = &prev->nvcsw;
+	} else if (preempt && prev->blocked_on) {
+		/*
+		 * If we are SM_PREEMPT, we may have interrupted
+		 * after blocked_on was set, before schedule()
+		 * was run, preventing workques from running. So
+		 * clear blocked_on and mark task RUNNING so it
+		 * can be reselected to run and complete its
+		 * logic
+		 */
+		WRITE_ONCE(prev->__state, TASK_RUNNING);
+		clear_task_blocked_on(prev, NULL);
 	}
 
 pick_again:
Signature · 1 line
2.54.0.545.g6539524ca2-goog
reviewer Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

Hello John,

Quoted 26 lines · John Stultz

Vineeth reported seeing a KVM related deadlock connected to work queue lockups using the android17-6.18 tree, which has Proxy Execution enabled (using the full patch stack), but I've subsequently reproduced it on v7.1-rc1.

On further debugging he found:
- kvm-irqfd-cleanup workqueue and rcu_gp lands in a per-cpu pwq(work queue pool)
- one of kvm-irqfd-cleanup worker(say A) takes a mutex and then calls synchronize_srcu_expedited()
- one other kvm-irqfd-cleanup worker worker(Say B) tries to acquire the lock and then gets blocked
- On the way to blocking, this cpu gets an IPI and on return from IPI, it calls __schedule() and did not get to complete workqueue accounting(worker->sleeping = 0 and decrementing pool->nr_running). This is done in sched_submit_work() -> wq_worker_sleeping() called from schedule() and we got preempted before that.
- proxy execution doesn't immediately take it off run queue as p->blocked_on is set during __mutex_lock
- Next time when B is picked for running, it notices A(mutex holder) is not on a runqueue and then blocks B. find_proxy_task() -> proxy_deactivate() -> block_task()
- And things are then stuck. A is waiting for the workqueue to be run, but B can't run the workqueue as it is blocked on A.

Quoted 4 lines

The trouble is that with Proxy Execution, in __mutex_lock_common() we set the task state to TASK_UNINTERRUPTIBLE, and set blocked_on before calling into schedule(), where sched_submit_work() will be called.

Geez! That is an interesting race.

Quoted 25 lines

But if an IPI comes in before we call schedule() the interrupt will call __schedule(SM_PREEMPT) directly. This causes the scheduler to see the current task as blocked_on, and deactivate it (because the owner is off the runqueue).

Since its deactivated, it wont' be run, and it won't get to call sched_submit_work().

Without proxy-execution, the SM_PREEMPT case will prevent the task from being dequeued, and it can be reselected again and run, which will allow it to finish calling into schedule() and calling sched_submit_work() before actually blocking.

So we need to make sure on the SM_PREEMPT case, if current is marked as blocked_on, we should clear the blocked_on state and mark the task RUNNABLE so the task can be selected to complete its call to schedule() -> sched_submit_work().

Now because we cleared BLOCKED_ON and set the task RUNNABLE, the task will be able to be selected and run again and loop back in __mutex_lock_common() where it can re-set the blocked_on state and call back into schedule() in order to properly be chosen as a donor.

Quoted 6 lines

Many thanks to Vineeth for figuring this very obscure race out and for implementing a test tool to make it easily reproducible!

I guess it is missing a:

Fixes be41bde4c3a8 ("sched: Add an initial sketch of the find_proxy_task() function")

since that is where we began blocking a task on task_is_blocked(). I really wish there was a better way to have detected this but I cannot think of any better way at the moment so feel free to include:

Quoted 40 lines

---
Cc: Vineeth Pillai <[email protected]>
Cc: Sonam Sanju <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Kunwu Chan <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: Qais Yousef <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Waiman Long <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Metin Kaya <[email protected]>
Cc: Xuewen Yan <[email protected]>
Cc: K Prateek Nayak <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Daniel Lezcano <[email protected]>
Cc: Suleiman Souhlal <[email protected]>
Cc: kuyo chang <[email protected]>
Cc: hupu <[email protected]>
Cc: [email protected]

---
 kernel/sched/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
kernel/sched/core.c +0 -0
index da20fb6ea25ae..5f684caefd8b2 100644
@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)
 		try_to_block_task(rq, prev, &prev_state,
 				  !task_is_blocked(prev));
 		switch_count = &prev->nvcsw;

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7109

Quoted 10 lines
kernel/sched/core.c +10 -0
+	} else if (preempt && prev->blocked_on) {
+		/*
+		 * If we are SM_PREEMPT, we may have interrupted
+		 * after blocked_on was set, before schedule()
+		 * was run, preventing workques from running. So
+		 * clear blocked_on and mark task RUNNING so it
+		 * can be reselected to run and complete its
+		 * logic
+		 */
+		WRITE_ONCE(prev->__state, TASK_RUNNING);

nit.

You probably need to update "prev_state" too for trace_sched_switch() to capture the right state down below.

Since this is on the way to schedule(), I wonder if it possible to just do a "next = prev" and goto picked ... but that adds more latency on PREEMPT_RT so that is a no go I presume.

Quoted 4 lines
kernel/sched/core.c +1 -0
+		clear_task_blocked_on(prev, NULL);
 	}
 
 pick_again:
Signature · 2 lines
Thanks and Regards,
Prateek
maintainer
#3 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

In reply to John Stultz
Quoted 8 lines · John Stultz
 kernel/sched/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
kernel/sched/core.c +0 -0
index da20fb6ea25ae..5f684caefd8b2 100644

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7104

Quoted 8 lines
kernel/sched/core.c +5 -0
@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)
 		try_to_block_task(rq, prev, &prev_state,
 				  !task_is_blocked(prev));
 		switch_count = &prev->nvcsw;
+	} else if (preempt && prev->blocked_on) {
+		/*
+		 * If we are SM_PREEMPT, we may have interrupted
+		 * after blocked_on was set, before schedule()
+		 * was run, preventing workques from running. So

workqueues

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7113

Quoted 9 lines
kernel/sched/core.c +6 -0
+		 * clear blocked_on and mark task RUNNING so it
+		 * can be reselected to run and complete its
+		 * logic
+		 */
+		WRITE_ONCE(prev->__state, TASK_RUNNING);
+		clear_task_blocked_on(prev, NULL);
 	}
 
 pick_again:

*groan*, this feels wrong. Preemption should never touch state. Let me try and wake up and make sense of this.

maintainer
#4 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

Quoted 29 lines · Peter Zijlstra
Quoted 8 lines · John Stultz
 kernel/sched/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
kernel/sched/core.c +0 -0
index da20fb6ea25ae..5f684caefd8b2 100644

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7104

Quoted 8 lines
kernel/sched/core.c +5 -0
@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)
 		try_to_block_task(rq, prev, &prev_state,
 				  !task_is_blocked(prev));
 		switch_count = &prev->nvcsw;
+	} else if (preempt && prev->blocked_on) {
+		/*
+		 * If we are SM_PREEMPT, we may have interrupted
+		 * after blocked_on was set, before schedule()
+		 * was run, preventing workques from running. So

workqueues

Quoted 8 lines
kernel/sched/core.c +6 -0
+		 * clear blocked_on and mark task RUNNING so it
+		 * can be reselected to run and complete its
+		 * logic
+		 */
+		WRITE_ONCE(prev->__state, TASK_RUNNING);
+		clear_task_blocked_on(prev, NULL);
 	}
 
Quoted 4 lines

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7113

Quoted 1 line
kernel/sched/core.c +0 -0
 pick_again:

*groan*, this feels wrong. Preemption should never touch state. Let me try and wake up and make sense of this.

So all non-special block states *SHOULD* be in a loop and handle spurious wakeups -- I fixed a pile of offenders some many years ago, but there really isn't anything in the kernel that validates this.

[ I suppose someone could try and do a cocci test for this? ]

Any wait for non-special states that is not a loop is fundamentally broken, since many of the lock wake-up paths are explicitly racy in that they can cause spurious wakeups (which is the safe side of the race, since insufficient wakeups is bad etc.).

OTOH special states, are special, esp. because they cannot handle spurious wakeups.

Eg, consider something like:

	set_current_state(TASK_FROZEN)
	<PREEMPT>
	  current->__state = TASK_RUNNING
	</PREEMPT/
	schedule();

is all sorts of broken. Now, obiously special states must never have blocked_on set, so this can be fudged about. But still, touching __state from schedule seems wrong.

Anyway, the historical distinction between a blocked task and a preempted task is that the blocked task is not on the runqueue, while the preempted task is kept on the runqueue.

Obviously PE wrecks this, and hence the problem. And yeah, amazing we never hit this before.

Something like so perhaps?

---

include/linux/sched.h +5 -2
index 368c7b4d7cb5..0bd5da8360f3 100644
@@ -846,7 +846,11 @@ struct task_struct {
 	struct alloc_tag		*alloc_tag;
 #endif
 
-	int				on_cpu;
+	u8				on_cpu;
+	u8				on_rq;
+	u8				is_blocked;
+	u8				__pad;
+
 	struct __call_single_node	wake_entry;
 	unsigned int			wakee_flips;
 	unsigned long			wakee_flip_decay_ts;
@@ -861,7 +865,6 @@ struct task_struct {
 	 */
 	int				recent_used_cpu;
 	int				wake_cpu;
-	int				on_rq;
 
 	int				prio;
 	int				static_prio;
kernel/sched/core.c +10 -1
index da20fb6ea25a..06817ae0cbd9 100644
@@ -615,6 +615,13 @@ EXPORT_SYMBOL(__trace_set_current_state);
  *   [ The astute reader will observe that it is possible for two tasks on one
  *     CPU to have ->on_cpu = 1 at the same time. ]
  *
+*  p->is_blocked <- { 0, 1 }:
+*
+*    is set by block_task() and cleared by ttwu_do_activate() and indicates
+*    this task is blocked, as opposed to runnable. Used to distinguish between
+*    preempted and blocked tasks for proxy exec, which keeps everything on the
+*    runqueue.
+ *
  * task_cpu(p): is changed by set_task_cpu(), the rules are:
  *
  *  - Don't call set_task_cpu() on a blocked task:
@@ -2225,6 +2232,7 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)
 
 static void block_task(struct rq *rq, struct task_struct *p, int flags)
 {
+	p->is_blocked = 1;
 	if (dequeue_task(rq, p, DEQUEUE_SLEEP | flags))
 		__block_task(rq, p);
 }
@@ -3722,6 +3730,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
 		atomic_dec(&task_rq(p)->nr_iowait);
 	}
 
+	p->is_blocked = 0;
 	activate_task(rq, p, en_flags);
 	wakeup_preempt(rq, p, wake_flags);
 
@@ -7107,7 +7116,7 @@ static void __sched notrace __schedule(int sched_mode)
 		struct task_struct *prev_donor = rq->donor;
 
 		rq_set_donor(rq, next);
-		if (unlikely(next->blocked_on)) {
+		if (unlikely(next->is_blocked && next->blocked_on)) {
 			next = find_proxy_task(rq, next, &rf);
 			if (!next) {
 				zap_balance_callbacks(rq);
#5 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

Hello Peter,

Quoted 115 lines · Peter Zijlstra
Quoted 29 lines · Peter Zijlstra
Quoted 8 lines · John Stultz
 kernel/sched/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
kernel/sched/core.c +0 -0
index da20fb6ea25ae..5f684caefd8b2 100644

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7104

Quoted 8 lines
kernel/sched/core.c +5 -0
@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)
 		try_to_block_task(rq, prev, &prev_state,
 				  !task_is_blocked(prev));
 		switch_count = &prev->nvcsw;
+	} else if (preempt && prev->blocked_on) {
+		/*
+		 * If we are SM_PREEMPT, we may have interrupted
+		 * after blocked_on was set, before schedule()
+		 * was run, preventing workques from running. So

workqueues

Quoted 8 lines
kernel/sched/core.c +6 -0
+		 * clear blocked_on and mark task RUNNING so it
+		 * can be reselected to run and complete its
+		 * logic
+		 */
+		WRITE_ONCE(prev->__state, TASK_RUNNING);
+		clear_task_blocked_on(prev, NULL);
 	}
 
Quoted 4 lines

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7113

Quoted 1 line
kernel/sched/core.c +0 -0
 pick_again:

*groan*, this feels wrong. Preemption should never touch state. Let me try and wake up and make sense of this.

So all non-special block states *SHOULD* be in a loop and handle spurious wakeups -- I fixed a pile of offenders some many years ago, but there really isn't anything in the kernel that validates this.

[ I suppose someone could try and do a cocci test for this? ]

Any wait for non-special states that is not a loop is fundamentally broken, since many of the lock wake-up paths are explicitly racy in that they can cause spurious wakeups (which is the safe side of the race, since insufficient wakeups is bad etc.).

OTOH special states, are special, esp. because they cannot handle spurious wakeups.

Eg, consider something like:

	set_current_state(TASK_FROZEN)
	<PREEMPT>
	  current->__state = TASK_RUNNING
	</PREEMPT/
	schedule();

is all sorts of broken. Now, obiously special states must never have blocked_on set, so this can be fudged about. But still, touching __state from schedule seems wrong.

Anyway, the historical distinction between a blocked task and a preempted task is that the blocked task is not on the runqueue, while the preempted task is kept on the runqueue.

Obviously PE wrecks this, and hence the problem. And yeah, amazing we never hit this before.

Something like so perhaps?

---

include/linux/sched.h +5 -2
index 368c7b4d7cb5..0bd5da8360f3 100644
@@ -846,7 +846,11 @@ struct task_struct {
 	struct alloc_tag		*alloc_tag;
 #endif
 
-	int				on_cpu;
+	u8				on_cpu;
+	u8				on_rq;
+	u8				is_blocked;
+	u8				__pad;
+
 	struct __call_single_node	wake_entry;
 	unsigned int			wakee_flips;
 	unsigned long			wakee_flip_decay_ts;
@@ -861,7 +865,6 @@ struct task_struct {
 	 */
 	int				recent_used_cpu;
 	int				wake_cpu;
-	int				on_rq;
 
 	int				prio;
 	int				static_prio;
kernel/sched/core.c +7 -0
index da20fb6ea25a..06817ae0cbd9 100644
@@ -615,6 +615,13 @@ EXPORT_SYMBOL(__trace_set_current_state);
  *   [ The astute reader will observe that it is possible for two tasks on one
  *     CPU to have ->on_cpu = 1 at the same time. ]
  *
+*  p->is_blocked <- { 0, 1 }:
+*
+*    is set by block_task() and cleared by ttwu_do_activate() and indicates
+*    this task is blocked, as opposed to runnable. Used to distinguish between
+*    preempted and blocked tasks for proxy exec, which keeps everything on the
+*    runqueue.
+ *
  * task_cpu(p): is changed by set_task_cpu(), the rules are:
  *
  *  - Don't call set_task_cpu() on a blocked task:

kernel/sched/core.c void deactivate_task(struct rq *rq, struct task_struct *p, int flags) · line 2235

Quoted 4 lines
kernel/sched/core.c +1 -0
@@ -2225,6 +2232,7 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)
 
 static void block_task(struct rq *rq, struct task_struct *p, int flags)
 {
+	p->is_blocked = 1;

We never reach here with PROXY_EXEC. Instead we bail out in the caller try_to_block_task() ...

Quoted 12 lines
kernel/sched/core.c +1 -0
 	if (dequeue_task(rq, p, DEQUEUE_SLEEP | flags))
 		__block_task(rq, p);
 }
@@ -3722,6 +3730,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
 		atomic_dec(&task_rq(p)->nr_iowait);
 	}
 
+	p->is_blocked = 0;
 	activate_task(rq, p, en_flags);
 	wakeup_preempt(rq, p, wake_flags);
 

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7119

Quoted 5 lines
kernel/sched/core.c +1 -1
@@ -7107,7 +7116,7 @@ static void __sched notrace __schedule(int sched_mode)
 		struct task_struct *prev_donor = rq->donor;
 
 		rq_set_donor(rq, next);
-		if (unlikely(next->blocked_on)) {
+		if (unlikely(next->is_blocked && next->blocked_on)) {

... so ->is_blocked here is always false for proxy tasks retained on the runqueue.

I was trying something like below but I'm somewhere missing a clear_task_blocked_on() for PROXY_WAKING before going back into mutex_lock_common():

include/linux/sched.h +4 -0
index 8ec3b6d7d718b..6ea74aecc5fbd 100644
@@ -586,6 +586,7 @@ struct sched_entity {
 	unsigned char			sched_delayed;
 	unsigned char			rel_deadline;
 	unsigned char			custom_slice;
+	unsigned char			sched_proxy;
 					/* hole */
 
 	u64				exec_start;
@@ -2222,6 +2223,7 @@ static inline void __clear_task_blocked_on(struct task_struct *p, struct mutex *
 	 * clearing the relationship with a different lock.
 	 */
 	WARN_ON_ONCE(m && p->blocked_on && p->blocked_on != m && p->blocked_on != PROXY_WAKING);
+	WRITE_ONCE(p->se.sched_proxy, 0);
 	p->blocked_on = NULL;
 }
 
@@ -2250,6 +2252,8 @@ static inline void __set_task_blocked_on_waking(struct task_struct *p, struct mu
 	 * the relationship with a different lock.
 	 */
 	WARN_ON_ONCE(m && p->blocked_on != m && p->blocked_on != PROXY_WAKING);
+	/* Force the task down proxy_force_return() path. */
+	WRITE_ONCE(p->se.sched_proxy, 1);
 	p->blocked_on = PROXY_WAKING;
 }
 
init/init_task.c +1 -0
index b5f48ebdc2b6e..8e8fc680fcd21 100644
@@ -118,6 +118,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
 	},
 	.se		= {
 		.group_node 	= LIST_HEAD_INIT(init_task.se.group_node),
+		.sched_proxy 	= 0,
 	},
 	.rt		= {
 		.run_list	= LIST_HEAD_INIT(init_task.rt.run_list),
kernel/sched/core.c +15 -5
index 49cd5d2171613..8142fba59ad94 100644
@@ -4395,6 +4395,7 @@ static void __sched_fork(u64 clone_flags, struct task_struct *p)
 	p->se.nr_migrations		= 0;
 	p->se.vruntime			= 0;
 	p->se.vlag			= 0;
+	p->se.sched_proxy		= 0;
 	INIT_LIST_HEAD(&p->se.group_node);
 
 	/* A delayed task cannot be in clone(). */
@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,
 	 * blocked on a mutex, and we want to keep it on the runqueue
 	 * to be selectable for proxy-execution.
 	 */
-	if (!should_block)
+	if (!should_block) {
+		guard(raw_spinlock)(&p->blocked_lock);
+		/* Stable against race */
+		if (task_is_blocked(p))
+			WRITE_ONCE(p->se.sched_proxy, 1);
 		return false;
+	}
 
 	p->sched_contributes_to_load =
 		(task_state & TASK_UNINTERRUPTIBLE) &&
@@ -6765,11 +6771,15 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 	bool curr_in_chain = false;
 	int this_cpu = cpu_of(rq);
 	struct task_struct *p;
-	struct mutex *mutex;
 	int owner_cpu;
 
 	/* Follow blocked_on chain. */
-	for (p = donor; (mutex = p->blocked_on); p = owner) {
+	for (p = donor; READ_ONCE(p->se.sched_proxy); p = owner) {
+		struct mutex *mutex = p->blocked_on;
+
+		if (!mutex)
+			return NULL;
+
 		/* if its PROXY_WAKING, do return migration or run if current */
 		if (mutex == PROXY_WAKING) {
 			if (task_current(rq, p)) {
@@ -6787,7 +6797,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 		guard(raw_spinlock)(&p->blocked_lock);
 
 		/* Check again that p is blocked with blocked_lock held */
-		if (mutex != __get_task_blocked_on(p)) {
+		if (!p->se.sched_proxy || mutex != __get_task_blocked_on(p)) {
 			/*
 			 * Something changed in the blocked_on chain and
 			 * we don't know if only at this level. So, let's
@@ -7044,7 +7054,7 @@ static void __sched notrace __schedule(int sched_mode)
 		struct task_struct *prev_donor = rq->donor;
 
 		rq_set_donor(rq, next);
-		if (unlikely(next->blocked_on)) {
+		if (unlikely(READ_ONCE(next->se.sched_proxy))) {
 			next = find_proxy_task(rq, next, &rf);
 			if (!next) {
 				zap_balance_callbacks(rq);

---

Quoted 3 lines
kernel/sched/core.c +0 -0
 			next = find_proxy_task(rq, next, &rf);
 			if (!next) {
 				zap_balance_callbacks(rq);
Signature · 2 lines
Thanks and Regards,
Prateek
#6 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

Quoted 3 lines · K Prateek Nayak

I was trying something like below but I'm somewhere missing a clear_task_blocked_on() for PROXY_WAKING before going back into mutex_lock_common():

And I seem to have been missing:

kernel/sched/core.c +3 -0
index 8142fba59ad94..a8679b759398c 100644
@@ -7046,6 +7046,9 @@ static void __sched notrace __schedule(int sched_mode)
 		switch_count = &prev->nvcsw;
 	}
 
+	if (!prev_state && task_is_blocked(prev))
+		clear_task_blocked_on(prev, NULL);
+
 pick_again:
 	assert_balance_callbacks_empty(rq);
 	next = pick_next_task(rq, rq->donor, &rf);

---

With that, it survives test-ww_mutex and a sched-messaging run without any splats.

Quoted 111 lines
include/linux/sched.h +4 -0
index 8ec3b6d7d718b..6ea74aecc5fbd 100644
@@ -586,6 +586,7 @@ struct sched_entity {
 	unsigned char			sched_delayed;
 	unsigned char			rel_deadline;
 	unsigned char			custom_slice;
+	unsigned char			sched_proxy;
 					/* hole */
 
 	u64				exec_start;
@@ -2222,6 +2223,7 @@ static inline void __clear_task_blocked_on(struct task_struct *p, struct mutex *
 	 * clearing the relationship with a different lock.
 	 */
 	WARN_ON_ONCE(m && p->blocked_on && p->blocked_on != m && p->blocked_on != PROXY_WAKING);
+	WRITE_ONCE(p->se.sched_proxy, 0);
 	p->blocked_on = NULL;
 }
 
@@ -2250,6 +2252,8 @@ static inline void __set_task_blocked_on_waking(struct task_struct *p, struct mu
 	 * the relationship with a different lock.
 	 */
 	WARN_ON_ONCE(m && p->blocked_on != m && p->blocked_on != PROXY_WAKING);
+	/* Force the task down proxy_force_return() path. */
+	WRITE_ONCE(p->se.sched_proxy, 1);
 	p->blocked_on = PROXY_WAKING;
 }
 
init/init_task.c +1 -0
index b5f48ebdc2b6e..8e8fc680fcd21 100644
@@ -118,6 +118,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
 	},
 	.se		= {
 		.group_node 	= LIST_HEAD_INIT(init_task.se.group_node),
+		.sched_proxy 	= 0,
 	},
 	.rt		= {
 		.run_list	= LIST_HEAD_INIT(init_task.rt.run_list),
kernel/sched/core.c +15 -6
index 49cd5d2171613..8142fba59ad94 100644
@@ -4395,6 +4395,7 @@ static void __sched_fork(u64 clone_flags, struct task_struct *p)
 	p->se.nr_migrations		= 0;
 	p->se.vruntime			= 0;
 	p->se.vlag			= 0;
+	p->se.sched_proxy		= 0;
 	INIT_LIST_HEAD(&p->se.group_node);
 
 	/* A delayed task cannot be in clone(). */
@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,
 	 * blocked on a mutex, and we want to keep it on the runqueue
 	 * to be selectable for proxy-execution.
 	 */
-	if (!should_block)
+	if (!should_block) {
+		guard(raw_spinlock)(&p->blocked_lock);
+		/* Stable against race */
+		if (task_is_blocked(p))
+			WRITE_ONCE(p->se.sched_proxy, 1);
 		return false;
+	}
 
 	p->sched_contributes_to_load =
 		(task_state & TASK_UNINTERRUPTIBLE) &&
@@ -6765,11 +6771,15 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 	bool curr_in_chain = false;
 	int this_cpu = cpu_of(rq);
 	struct task_struct *p;
-	struct mutex *mutex;
 	int owner_cpu;
 
 	/* Follow blocked_on chain. */
-	for (p = donor; (mutex = p->blocked_on); p = owner) {
+	for (p = donor; READ_ONCE(p->se.sched_proxy); p = owner) {
+		struct mutex *mutex = p->blocked_on;
+
+		if (!mutex)
+			return NULL;
+
 		/* if its PROXY_WAKING, do return migration or run if current */
 		if (mutex == PROXY_WAKING) {
 			if (task_current(rq, p)) {
@@ -6787,7 +6797,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 		guard(raw_spinlock)(&p->blocked_lock);
 
 		/* Check again that p is blocked with blocked_lock held */
-		if (mutex != __get_task_blocked_on(p)) {
+		if (!p->se.sched_proxy || mutex != __get_task_blocked_on(p)) {
 			/*
 			 * Something changed in the blocked_on chain and
 			 * we don't know if only at this level. So, let's
@@ -7044,7 +7054,7 @@ static void __sched notrace __schedule(int sched_mode)
 		struct task_struct *prev_donor = rq->donor;
 
 		rq_set_donor(rq, next);
-		if (unlikely(next->blocked_on)) {
+		if (unlikely(READ_ONCE(next->se.sched_proxy))) {
 			next = find_proxy_task(rq, next, &rf);
 			if (!next) {
 				zap_balance_callbacks(rq);
---
Quoted 4 lines
(quoted diff) +0 -0
 			next = find_proxy_task(rq, next, &rf);
 			if (!next) {
 				zap_balance_callbacks(rq);
Signature · 2 lines
Thanks and Regards,
Prateek
maintainer
#7 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

In reply to K Prateek Nayak
Quoted 69 lines · K Prateek Nayak
Quoted 47 lines

Something like so perhaps?

---

include/linux/sched.h +5 -2
index 368c7b4d7cb5..0bd5da8360f3 100644
@@ -846,7 +846,11 @@ struct task_struct {
 	struct alloc_tag		*alloc_tag;
 #endif
 
-	int				on_cpu;
+	u8				on_cpu;
+	u8				on_rq;
+	u8				is_blocked;
+	u8				__pad;
+
 	struct __call_single_node	wake_entry;
 	unsigned int			wakee_flips;
 	unsigned long			wakee_flip_decay_ts;
@@ -861,7 +865,6 @@ struct task_struct {
 	 */
 	int				recent_used_cpu;
 	int				wake_cpu;
-	int				on_rq;
 
 	int				prio;
 	int				static_prio;
kernel/sched/core.c +7 -0
index da20fb6ea25a..06817ae0cbd9 100644
@@ -615,6 +615,13 @@ EXPORT_SYMBOL(__trace_set_current_state);
  *   [ The astute reader will observe that it is possible for two tasks on one
  *     CPU to have ->on_cpu = 1 at the same time. ]
  *
+*  p->is_blocked <- { 0, 1 }:
+*
+*    is set by block_task() and cleared by ttwu_do_activate() and indicates
+*    this task is blocked, as opposed to runnable. Used to distinguish between
+*    preempted and blocked tasks for proxy exec, which keeps everything on the
+*    runqueue.
+ *
  * task_cpu(p): is changed by set_task_cpu(), the rules are:
  *
  *  - Don't call set_task_cpu() on a blocked task:

kernel/sched/core.c void deactivate_task(struct rq *rq, struct task_struct *p, int flags) · line 2235

Quoted 4 lines
kernel/sched/core.c +1 -0
@@ -2225,6 +2232,7 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)
 
 static void block_task(struct rq *rq, struct task_struct *p, int flags)
 {
+	p->is_blocked = 1;

We never reach here with PROXY_EXEC. Instead we bail out in the caller try_to_block_task() ...

Quoted 14 lines
kernel/sched/core.c +1 -0
 	if (dequeue_task(rq, p, DEQUEUE_SLEEP | flags))
 		__block_task(rq, p);
 }
@@ -3722,6 +3730,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
 		atomic_dec(&task_rq(p)->nr_iowait);
 	}
 
+	p->is_blocked = 0;
 	activate_task(rq, p, en_flags);
 	wakeup_preempt(rq, p, wake_flags);
 
@@ -7107,7 +7116,7 @@ static void __sched notrace __schedule(int sched_mode)
 		struct task_struct *prev_donor = rq->donor;
 
Quoted 6 lines

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode)

Quoted 3 lines
kernel/sched/core.c +1 -1
 		rq_set_donor(rq, next);
-		if (unlikely(next->blocked_on)) {
+		if (unlikely(next->is_blocked && next->blocked_on)) {

... so ->is_blocked here is always false for proxy tasks retained on the runqueue.

Right. Also, egads, we really should fix that block/ttwu part, this is a mess.

Anyway, idea is simple even if execution turns into a bit of a mess now, set when task really is blocked and clear on wakeup.

Quoted 9 lines

I was trying something like below but I'm somewhere missing a clear_task_blocked_on() for PROXY_WAKING before going back into mutex_lock_common():

include/linux/sched.h +0 -0
index 8ec3b6d7d718b..6ea74aecc5fbd 100644

include/linux/sched.h struct sched_entity { · line 590

Quoted 5 lines
include/linux/sched.h +1 -0
@@ -586,6 +586,7 @@ struct sched_entity {
 	unsigned char			sched_delayed;
 	unsigned char			rel_deadline;
 	unsigned char			custom_slice;
+	unsigned char			sched_proxy;
 					/* hole */

Should not live in sched_entity I suppose.

#8 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

In reply to K Prateek Nayak
Quoted 154 lines · K Prateek Nayak
Quoted 115 lines · Peter Zijlstra
Quoted 29 lines · Peter Zijlstra
Quoted 8 lines · John Stultz
 kernel/sched/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
kernel/sched/core.c +0 -0
index da20fb6ea25ae..5f684caefd8b2 100644

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7104

Quoted 8 lines
kernel/sched/core.c +5 -0
@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)
            try_to_block_task(rq, prev, &prev_state,
                              !task_is_blocked(prev));
            switch_count = &prev->nvcsw;
+   } else if (preempt && prev->blocked_on) {
+           /*
+            * If we are SM_PREEMPT, we may have interrupted
+            * after blocked_on was set, before schedule()
+            * was run, preventing workques from running. So

workqueues

Quoted 8 lines
kernel/sched/core.c +6 -0
+            * clear blocked_on and mark task RUNNING so it
+            * can be reselected to run and complete its
+            * logic
+            */
+           WRITE_ONCE(prev->__state, TASK_RUNNING);
+           clear_task_blocked_on(prev, NULL);
    }
 
Quoted 4 lines

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7113

Quoted 1 line
kernel/sched/core.c +0 -0
 pick_again:

*groan*, this feels wrong. Preemption should never touch state. Let me try and wake up and make sense of this.

So all non-special block states *SHOULD* be in a loop and handle spurious wakeups -- I fixed a pile of offenders some many years ago, but there really isn't anything in the kernel that validates this.

[ I suppose someone could try and do a cocci test for this? ]

Any wait for non-special states that is not a loop is fundamentally broken, since many of the lock wake-up paths are explicitly racy in that they can cause spurious wakeups (which is the safe side of the race, since insufficient wakeups is bad etc.).

OTOH special states, are special, esp. because they cannot handle spurious wakeups.

Eg, consider something like:

      set_current_state(TASK_FROZEN)
      <PREEMPT>
        current->__state = TASK_RUNNING
      </PREEMPT/
      schedule();

is all sorts of broken. Now, obiously special states must never have blocked_on set, so this can be fudged about. But still, touching __state from schedule seems wrong.

Anyway, the historical distinction between a blocked task and a preempted task is that the blocked task is not on the runqueue, while the preempted task is kept on the runqueue.

Obviously PE wrecks this, and hence the problem. And yeah, amazing we never hit this before.

Something like so perhaps?

---

include/linux/sched.h +5 -2
index 368c7b4d7cb5..0bd5da8360f3 100644
@@ -846,7 +846,11 @@ struct task_struct {
      struct alloc_tag                *alloc_tag;
 #endif
 
-     int                             on_cpu;
+     u8                              on_cpu;
+     u8                              on_rq;
+     u8                              is_blocked;
+     u8                              __pad;
+
      struct __call_single_node       wake_entry;
      unsigned int                    wakee_flips;
      unsigned long                   wakee_flip_decay_ts;
@@ -861,7 +865,6 @@ struct task_struct {
       */
      int                             recent_used_cpu;
      int                             wake_cpu;
-     int                             on_rq;
 
      int                             prio;
      int                             static_prio;
kernel/sched/core.c +7 -0
index da20fb6ea25a..06817ae0cbd9 100644
@@ -615,6 +615,13 @@ EXPORT_SYMBOL(__trace_set_current_state);
  *   [ The astute reader will observe that it is possible for two tasks on one
  *     CPU to have ->on_cpu = 1 at the same time. ]
  *
+*  p->is_blocked <- { 0, 1 }:
+*
+*    is set by block_task() and cleared by ttwu_do_activate() and indicates
+*    this task is blocked, as opposed to runnable. Used to distinguish between
+*    preempted and blocked tasks for proxy exec, which keeps everything on the
+*    runqueue.
+ *
  * task_cpu(p): is changed by set_task_cpu(), the rules are:
  *
  *  - Don't call set_task_cpu() on a blocked task:

kernel/sched/core.c void deactivate_task(struct rq *rq, struct task_struct *p, int flags) · line 2235

Quoted 4 lines
kernel/sched/core.c +1 -0
@@ -2225,6 +2232,7 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)
 
 static void block_task(struct rq *rq, struct task_struct *p, int flags)
 {
+     p->is_blocked = 1;

We never reach here with PROXY_EXEC. Instead we bail out in the caller try_to_block_task() ...

Quoted 12 lines
kernel/sched/core.c +1 -0
      if (dequeue_task(rq, p, DEQUEUE_SLEEP | flags))
              __block_task(rq, p);
 }
@@ -3722,6 +3730,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
              atomic_dec(&task_rq(p)->nr_iowait);
      }
 
+     p->is_blocked = 0;
      activate_task(rq, p, en_flags);
      wakeup_preempt(rq, p, wake_flags);
 

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7119

Quoted 5 lines
kernel/sched/core.c +1 -1
@@ -7107,7 +7116,7 @@ static void __sched notrace __schedule(int sched_mode)
              struct task_struct *prev_donor = rq->donor;
 
              rq_set_donor(rq, next);
-             if (unlikely(next->blocked_on)) {
+             if (unlikely(next->is_blocked && next->blocked_on)) {

... so ->is_blocked here is always false for proxy tasks retained on the runqueue.

I was trying something like below but I'm somewhere missing a clear_task_blocked_on() for PROXY_WAKING before going back into mutex_lock_common():

include/linux/sched.h +0 -0
index 8ec3b6d7d718b..6ea74aecc5fbd 100644

include/linux/sched.h struct sched_entity { · line 590

Quoted 5 lines
include/linux/sched.h +1 -0
@@ -586,6 +586,7 @@ struct sched_entity {
        unsigned char                   sched_delayed;
        unsigned char                   rel_deadline;
        unsigned char                   custom_slice;
+       unsigned char                   sched_proxy;
                                        /* hole */

I feel like this is so tied to the blocked_on value, I suspect it makes the most sense to have this flag be the low bit of that pointer?

Sort of a blocked_on latch, to signal its really in effect?

Plus it gets cleared automatically on set and clear, so it looks a little cleaner.

include/linux/sched.h static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6545

Quoted 12 lines
include/linux/sched.h +6 -1
@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,
         * blocked on a mutex, and we want to keep it on the runqueue
         * to be selectable for proxy-execution.
         */
-       if (!should_block)
+       if (!should_block) {
+               guard(raw_spinlock)(&p->blocked_lock);
+               /* Stable against race */
+               if (task_is_blocked(p))
+                       WRITE_ONCE(p->se.sched_proxy, 1);
                return false;
+       }

So if we double check and find the task isn't blocked anymore, we probably shouldn't return early here, no?

Let me take a stab at the bit flag approach and see how it goes.

thanks
-john

#9 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

Hello John,

Quoted 14 lines · John Stultz

include/linux/sched.h struct sched_entity { · line 590

Quoted 10 lines
include/linux/sched.h +1 -0
index 8ec3b6d7d718b..6ea74aecc5fbd 100644
@@ -586,6 +586,7 @@ struct sched_entity {
        unsigned char                   sched_delayed;
        unsigned char                   rel_deadline;
        unsigned char                   custom_slice;
+       unsigned char                   sched_proxy;
                                        /* hole */

I feel like this is so tied to the blocked_on value, I suspect it makes the most sense to have this flag be the low bit of that pointer?

Quoted 4 lines

Sort of a blocked_on latch, to signal its really in effect?

Plus it gets cleared automatically on set and clear, so it looks a little cleaner.

I had actually started looking into that last night but sleep eventually got to me! I agree that is much cleaner to do it alongside p->blocked_on.

Quoted 18 lines
Quoted 4 lines
include/linux/sched.h +0 -0
@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,
         * blocked on a mutex, and we want to keep it on the runqueue

include/linux/sched.h static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6545

Quoted 10 lines
include/linux/sched.h +6 -1
         * to be selectable for proxy-execution.
         */
-       if (!should_block)
+       if (!should_block) {
+               guard(raw_spinlock)(&p->blocked_lock);
+               /* Stable against race */
+               if (task_is_blocked(p))
+                       WRITE_ONCE(p->se.sched_proxy, 1);
                return false;
+       }

So if we double check and find the task isn't blocked anymore, we probably shouldn't return early here, no?

Quoted 1 line

Let me take a stab at the bit flag approach and see how it goes.

In case you want to peek at my homework ;-)

(Lightly tested with test-ww_mutex and sched-messaging)

include/linux/sched.h +40 -8
index 8ec3b6d7d718b..e28f5c05a1689 100644
@@ -1237,7 +1237,7 @@ struct task_struct {
 	struct rt_mutex_waiter		*pi_blocked_on;
 #endif
 
-	struct mutex			*blocked_on;	/* lock we're blocked on */
+	unsigned long			blocked_on;	/* lock we're blocked on */
 	raw_spinlock_t			blocked_lock;
 
 #ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
@@ -2188,12 +2188,20 @@ extern int __cond_resched_rwlock_write(rwlock_t *lock) __must_hold(lock);
  * evaluated for return-migration before it is run. So if the task is
  * blocked_on PROXY_WAKING, return migrate it before running it.
  */
-#define PROXY_WAKING ((struct mutex *)(-1L))
+#define PROXY_WAKING ((unsigned long)(-1L))
+
+#define BO_FLAG_PROXY	BIT(0)
+#define BO_FLAGS	GENMASK(0, 0)
+
+static inline struct mutex *__get_blocked_on_mutex(unsigned long m)
+{
+	return (struct mutex *)(m & ~BO_FLAGS);
+}
 
 static inline struct mutex *__get_task_blocked_on(struct task_struct *p)
 {
 	lockdep_assert_held_once(&p->blocked_lock);
-	return p->blocked_on == PROXY_WAKING ? NULL : p->blocked_on;
+	return p->blocked_on == PROXY_WAKING ? NULL : __get_blocked_on_mutex(p->blocked_on);
 }
 
 static inline void __set_task_blocked_on(struct task_struct *p, struct mutex *m)
@@ -2208,8 +2216,11 @@ static inline void __set_task_blocked_on(struct task_struct *p, struct mutex *m)
 	 * with a different mutex. Note, setting it to the same
 	 * lock repeatedly is ok.
 	 */
-	WARN_ON_ONCE(p->blocked_on && p->blocked_on != m);
-	p->blocked_on = m;
+	WARN_ON_ONCE(p->blocked_on &&
+		     __get_blocked_on_mutex(p->blocked_on) != m);
+
+	BUG_ON((unsigned long)m & BO_FLAGS);
+	p->blocked_on = (unsigned long)m & ~BO_FLAGS;
 }
 
 static inline void __clear_task_blocked_on(struct task_struct *p, struct mutex *m)
@@ -2221,8 +2232,10 @@ static inline void __clear_task_blocked_on(struct task_struct *p, struct mutex *
 	 * blocked_on relationships, but make sure we are not
 	 * clearing the relationship with a different lock.
 	 */
-	WARN_ON_ONCE(m && p->blocked_on && p->blocked_on != m && p->blocked_on != PROXY_WAKING);
-	p->blocked_on = NULL;
+	WARN_ON_ONCE(m && p->blocked_on &&
+		     __get_blocked_on_mutex(p->blocked_on) != m &&
+		     p->blocked_on != PROXY_WAKING);
+	p->blocked_on = 0;
 }
 
 static inline void clear_task_blocked_on(struct task_struct *p, struct mutex *m)
@@ -2249,7 +2262,12 @@ static inline void __set_task_blocked_on_waking(struct task_struct *p, struct mu
 	 * already set to waking, but make sure we are not changing
 	 * the relationship with a different lock.
 	 */
-	WARN_ON_ONCE(m && p->blocked_on != m && p->blocked_on != PROXY_WAKING);
+	WARN_ON_ONCE(m &&
+		     __get_blocked_on_mutex(p->blocked_on) != m &&
+		     p->blocked_on != PROXY_WAKING);
+
+	/* Always ensure PROXY_WAKING implies BO_FLAG_PROXY. */
+	BUILD_BUG_ON(!(PROXY_WAKING & BO_FLAG_PROXY));
 	p->blocked_on = PROXY_WAKING;
 }
 
@@ -2259,6 +2277,20 @@ static inline void set_task_blocked_on_waking(struct task_struct *p, struct mute
 	__set_task_blocked_on_waking(p, m);
 }
 
+static inline void set_task_blocked_on_proxy(struct task_struct *p)
+{
+	guard(raw_spinlock_irqsave)(&p->blocked_lock);
+
+	if (!sched_proxy_exec())
+		return;
+
+	/* Don't add BO_FLAG_PROXY if blocked_on was already cleared */
+	if (!p->blocked_on)
+		return;
+
+	p->blocked_on |= BO_FLAG_PROXY;
+}
+
 #else
 static inline void __clear_task_blocked_on(struct task_struct *p, struct rt_mutex *m)
 {
kernel/fork.c +1 -1
index 079802cb61002..936e8d6ee7a0b 100644
@@ -2177,7 +2177,7 @@ __latent_entropy struct task_struct *copy_process(
 
 	lockdep_init_task(p);
 
-	p->blocked_on = NULL; /* not blocked yet */
+	p->blocked_on = 0; /* not blocked yet */
 
 #ifdef CONFIG_BCACHE
 	p->sequential_io	= 0;
kernel/sched/core.c +31 -13

78 lines not on this page · raw message

kernel/sched/sched.h +1 -1
index c95584191d58f..e4a0aeee9bfc0 100644
@@ -2390,7 +2390,7 @@ static inline bool task_is_blocked(struct task_struct *p)
 	if (!sched_proxy_exec())
 		return false;
 
-	return !!p->blocked_on;
+	return !!(p->blocked_on & BO_FLAG_PROXY);
 }
 
 static inline int task_on_cpu(struct rq *rq, struct task_struct *p)

---

There is probably a better way to do it by inspecting the flags only in find_proxy_task() and keep task_is_blocked() as is but I wanted to see if it can be done as a part of task_ib_blocked() helper since that seemed cleaner in my head.

I'll let you fiddle with it.

Signature · 2 lines
Thanks and Regards,
Prateek
#10 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

Quoted 21 lines · K Prateek Nayak
Quoted 20 lines · John Stultz
Quoted 6 lines
include/linux/sched.h +0 -0
index 8ec3b6d7d718b..6ea74aecc5fbd 100644
@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,
         * blocked on a mutex, and we want to keep it on the runqueue

include/linux/sched.h static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6545

Quoted 10 lines
include/linux/sched.h +6 -1
         * to be selectable for proxy-execution.
         */
-       if (!should_block)
+       if (!should_block) {
+               guard(raw_spinlock)(&p->blocked_lock);
+               /* Stable against race */
+               if (task_is_blocked(p))
+                       WRITE_ONCE(p->se.sched_proxy, 1);
                return false;
+       }

So if we double check and find the task isn't blocked anymore, we probably shouldn't return early here, no?

Quoted 4 lines
Quoted 1 line

Let me take a stab at the bit flag approach and see how it goes.

In case you want to peek at my homework ;-)

Well, now it will just seem like I'm cheating! :)

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7063

Quoted 12 lines
kernel/sched/core.c +8 -0
@@ -7043,8 +7053,16 @@ static void __sched notrace __schedule(int sched_mode)
        if (sched_proxy_exec()) {
                struct task_struct *prev_donor = rq->donor;
 
+               /*
+                * A wakeup raced with block_task();
+                * Clear blocked_on before running the task
+                * again.
+                */
+               if (unlikely(!prev_state && prev->blocked_on))
+                       clear_task_blocked_on(prev, NULL);
+

So similar to your previous change, I was hitting this same case, and adding in your change here was bothering me a bit that it felt like clearing things here was papering over an unexpected state change that we didn't have before.

Part of the issue is the blocked_on state machine is already a little complex:
NULL -> ptr -> PROXY_WAKING ->NULL
(with some limited shortcuting from ptr->NULL when we are dealing with current)

So adding this BO_FLAG_PROXY/"latch" bit to the mix complicates things:
NULL -> ptr:unlatched -> ptr:latched -> PROXY_WAKING -> NULL

With extra lines for:
ptr:unlatched -> NULL
ptr:latched -> NULL
ptr:unlatched -> PROXY_WAKING

I kept on seeing these unexpected transitions where before we latched the blocked_on state, we'd get a wakeup on another cpu from the owner unlocking the lock, so we'd go ptr:unlatched -> PROXY_WAKING (along with setting TASK_RUNNING). Then because we weren't latched, we might be running midway through the lock logic. If we go into schedule() we'll just come back out eventually (as we're TASK_RUNNING). Then when then try again in the lock loop to set blocked_on to ptr:unlatched again, we hit warnings because we are setting it to a ptr when it is PROXY_WAKING and not NULL.

I'm sure this was obvious to you when you wrote the snippit above, but I've been running a little slow, so let's not talk about how long it took for me to actaully understand this. :P

And yes, your snippit above avoids this, but it feels a little incidental, cleaning up after the fact. So I think it might be better to simplify the state machine a little to prevent it.

My current thought is to cut out the ptr:unlatched -> PROXY_WAKING transition. And instead in __set_task_blocked_on_waking(), instead of checking blocked_on and doing an early return if it is null, we could instead check if the latch bit was set, and if so, set blocked_on to NULL. This bascially treats ptr:unlatched the same as NULL in most cases for the state machine.

Now, earlier you made some nice explanations earlier about how some of the blocked_on checks done w/ the rq_lock and not the blocked_lock were safe because of the limited situations where we clear blocked_on. And this at first seems to put that assumption risk, but I think it still holds if we always consider ptr:unlatched equivalent to NULL. So once ptr:latched is set, the rule still holds.

That gives us:
NULL -> ptr:unlatched -> ptr:latched -> PROXY_WAKING -> NULL

Where NULL and ptr:unlatched are functionally equivalent except for the ability to transition to ptr:latched.

With:
ptr:latched -> NULL // only done on current
ptr:unlatched -> NULL // only done on current or when trying to set waking

So far my testing is doing ok with this. Let me know if you see any holes though.

I'll try to cleanup my current changes (surely copying some of your excellent work :), and send this out tomorrow (I'm shot for today) for more concrete review.

thanks
-john

#11 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

Quoted 4 lines · John Stultz

My current thought is to cut out the ptr:unlatched -> PROXY_WAKING transition. And instead in __set_task_blocked_on_waking(), instead of checking blocked_on and doing an early return if it is null, we could instead check if the latch bit was set, and if so, set blocked_on to

Ugh! I meant "check if the latch bit was unset" in the above. Apologies for any confusion there.

Quoted 2 lines

NULL. This bascially treats ptr:unlatched the same as NULL in most cases for the state machine.

thanks
-john

#12 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

In reply to John Stultz

Hello John,

Quoted 22 lines · John Stultz
Quoted 21 lines · K Prateek Nayak
Quoted 20 lines · John Stultz
Quoted 6 lines
include/linux/sched.h +0 -0
index 8ec3b6d7d718b..6ea74aecc5fbd 100644
@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,
         * blocked on a mutex, and we want to keep it on the runqueue

include/linux/sched.h static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6545

Quoted 10 lines
include/linux/sched.h +6 -1
         * to be selectable for proxy-execution.
         */
-       if (!should_block)
+       if (!should_block) {
+               guard(raw_spinlock)(&p->blocked_lock);
+               /* Stable against race */
+               if (task_is_blocked(p))
+                       WRITE_ONCE(p->se.sched_proxy, 1);
                return false;
+       }

So if we double check and find the task isn't blocked anymore, we probably shouldn't return early here, no?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Let me take a stab at the bit flag approach and see how it goes.

In case you want to peek at my homework ;-)

Well, now it will just seem like I'm cheating! :)

But I can't prove I didn't copy your idea first ;-)

Quoted 30 lines

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7063

Quoted 12 lines
kernel/sched/core.c +8 -0
@@ -7043,8 +7053,16 @@ static void __sched notrace __schedule(int sched_mode)
        if (sched_proxy_exec()) {
                struct task_struct *prev_donor = rq->donor;
 
+               /*
+                * A wakeup raced with block_task();
+                * Clear blocked_on before running the task
+                * again.
+                */
+               if (unlikely(!prev_state && prev->blocked_on))
+                       clear_task_blocked_on(prev, NULL);
+

So similar to your previous change, I was hitting this same case, and adding in your change here was bothering me a bit that it felt like clearing things here was papering over an unexpected state change that we didn't have before.

Part of the issue is the blocked_on state machine is already a little complex:
NULL -> ptr -> PROXY_WAKING ->NULL
(with some limited shortcuting from ptr->NULL when we are dealing with current)

So adding this BO_FLAG_PROXY/"latch" bit to the mix complicates things:
NULL -> ptr:unlatched -> ptr:latched -> PROXY_WAKING -> NULL

With extra lines for:
ptr:unlatched -> NULL
ptr:latched -> NULL
ptr:unlatched -> PROXY_WAKING

Quoted 9 lines

I kept on seeing these unexpected transitions where before we latched the blocked_on state, we'd get a wakeup on another cpu from the owner unlocking the lock, so we'd go ptr:unlatched -> PROXY_WAKING (along with setting TASK_RUNNING). Then because we weren't latched, we might be running midway through the lock logic. If we go into schedule() we'll just come back out eventually (as we're TASK_RUNNING). Then when then try again in the lock loop to set blocked_on to ptr:unlatched again, we hit warnings because we are setting it to a ptr when it is PROXY_WAKING and not NULL.

Yes, it was precisely that which caused the splat in my case too for the initial suggestion on Peter's diff. I'll try to add a little bit more context in those tricky comments from next time onwards. Sorry about that!

Quoted 9 lines

I'm sure this was obvious to you when you wrote the snippit above, but I've been running a little slow, so let's not talk about how long it took for me to actaully understand this. :P

And yes, your snippit above avoids this, but it feels a little incidental, cleaning up after the fact. So I think it might be better to simplify the state machine a little to prevent it.

Quoted 6 lines

My current thought is to cut out the ptr:unlatched -> PROXY_WAKING transition. And instead in __set_task_blocked_on_waking(), instead of checking blocked_on and doing an early return if it is null, we could instead check if the latch bit was set, and if so, set blocked_on to NULL. This bascially treats ptr:unlatched the same as NULL in most cases for the state machine.

That is a fantastic idea! Since the sched bits have not blocked it yet, there is no need to go through the return migration path.

Clearing it when not latched should do the trick.

Quoted 7 lines

Now, earlier you made some nice explanations earlier about how some of the blocked_on checks done w/ the rq_lock and not the blocked_lock were safe because of the limited situations where we clear blocked_on. And this at first seems to put that assumption risk, but I think it still holds if we always consider ptr:unlatched equivalent to NULL. So once ptr:latched is set, the rule still holds.

Yes, that is indeed true. Transitioning the task to latched state, and unlatching it, will both be done with rq_lock + blocked_on_lock so it should be okay to peek at the latched state when holding the task's rq lock in ttwu_runnable(), and in schedule().

Quoted 7 lines

That gives us:
NULL -> ptr:unlatched -> ptr:latched -> PROXY_WAKING -> NULL

Where NULL and ptr:unlatched are functionally equivalent except for the ability to transition to ptr:latched.

Quoted 6 lines

With:
ptr:latched -> NULL // only done on current
ptr:unlatched -> NULL // only done on current or when trying to set waking

So far my testing is doing ok with this. Let me know if you see any holes though.

This looks good to me! Cannot spot any holes with that.

However, to completely get rid of that hunk in __schedule() we probably need to order setting the blocked_on wrt task's __state based on Peter's analysis in
https://lore.kernel.org/lkml/[email protected]/

As Peter suggested, I think we need:

    __set_task_blocked_on(current, lock);
    /*
     * Pairs with smp_rmb() after ttwu_state_match() in try_to_wake_up().
     * Ensures ttwu always see the correct blocked_on state for wakeups.
     */
    smp_wmb();
    set_current_state(state);

in __mutex_lock_common() for it to be completely safe with the proxy_needs_return() bits and that should be good enough to avoid that race but I'll let Peter comment if he thinks there is still a race window open.

Quoted 4 lines

I'll try to cleanup my current changes (surely copying some of your excellent work :), and send this out tomorrow (I'm shot for today) for more concrete review.

I'll be looking forward to it :-)

Signature · 2 lines
Thanks and Regards,
Prateek
#13 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

Quoted 15 lines · K Prateek Nayak

However, to completely get rid of that hunk in __schedule() we probably need to order setting the blocked_on wrt task's __state based on Peter's analysis in
https://lore.kernel.org/lkml/[email protected]/

As Peter suggested, I think we need:

    __set_task_blocked_on(current, lock);
    /*
     * Pairs with smp_rmb() after ttwu_state_match() in try_to_wake_up().
     * Ensures ttwu always see the correct blocked_on state for wakeups.
     */
    smp_wmb();
    set_current_state(state);
Quoted 5 lines

in __mutex_lock_common() for it to be completely safe with the proxy_needs_return() bits and that should be good enough to avoid that race but I'll let Peter comment if he thinks there is still a race window open.

Ok, it seems like we could add the smp_wmb in the bottom of __set_task_blocked_on() then?

This is probably worth its own patch and (correct me if I'm wrong) feels separate enough that I might try to look at this after the fixes for this current issue are settled?

thanks
-john

#14 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

In reply to Peter Zijlstra
Quoted 58 lines · Peter Zijlstra
Quoted 29 lines · Peter Zijlstra
Quoted 8 lines · John Stultz
 kernel/sched/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
kernel/sched/core.c +0 -0
index da20fb6ea25ae..5f684caefd8b2 100644

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7104

Quoted 8 lines
kernel/sched/core.c +5 -0
@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)
            try_to_block_task(rq, prev, &prev_state,
                              !task_is_blocked(prev));
            switch_count = &prev->nvcsw;
+   } else if (preempt && prev->blocked_on) {
+           /*
+            * If we are SM_PREEMPT, we may have interrupted
+            * after blocked_on was set, before schedule()
+            * was run, preventing workques from running. So

workqueues

Quoted 8 lines
kernel/sched/core.c +6 -0
+            * clear blocked_on and mark task RUNNING so it
+            * can be reselected to run and complete its
+            * logic
+            */
+           WRITE_ONCE(prev->__state, TASK_RUNNING);
+           clear_task_blocked_on(prev, NULL);
    }
 
Quoted 4 lines

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7113

Quoted 1 line
kernel/sched/core.c +0 -0
 pick_again:

*groan*, this feels wrong. Preemption should never touch state. Let me try and wake up and make sense of this.

So all non-special block states *SHOULD* be in a loop and handle spurious wakeups -- I fixed a pile of offenders some many years ago, but there really isn't anything in the kernel that validates this.

[ I suppose someone could try and do a cocci test for this? ]

Any wait for non-special states that is not a loop is fundamentally broken, since many of the lock wake-up paths are explicitly racy in that they can cause spurious wakeups (which is the safe side of the race, since insufficient wakeups is bad etc.).

OTOH special states, are special, esp. because they cannot handle spurious wakeups.

Eg, consider something like:

        set_current_state(TASK_FROZEN)
        <PREEMPT>
          current->__state = TASK_RUNNING
        </PREEMPT/
        schedule();
Quoted 3 lines

is all sorts of broken. Now, obiously special states must never have blocked_on set, so this can be fudged about. But still, touching __state from schedule seems wrong.

Hey Peter,
Thanks again for the background here and the alternative proposal
that K Prateek and I have iterated on.

I did want to clarify one case here just for my understanding. The approach in this first proposal was sort of leaning on the similar pattern in the signal_pending_state() case in try_to_block_task(). Is that just a hard exception to the rule here?

thanks
-john

#15 lore raw

Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

Hello John,

Quoted 20 lines · John Stultz
Quoted 16 lines

Any wait for non-special states that is not a loop is fundamentally broken, since many of the lock wake-up paths are explicitly racy in that they can cause spurious wakeups (which is the safe side of the race, since insufficient wakeups is bad etc.).

OTOH special states, are special, esp. because they cannot handle spurious wakeups.

Eg, consider something like:

        set_current_state(TASK_FROZEN)
        <PREEMPT>
          current->__state = TASK_RUNNING
        </PREEMPT/
        schedule();
Quoted 3 lines

is all sorts of broken. Now, obiously special states must never have blocked_on set, so this can be fudged about. But still, touching __state from schedule seems wrong.

Quoted 8 lines

Hey Peter,
Thanks again for the background here and the alternative proposal
that K Prateek and I have iterated on.

I did want to clarify one case here just for my understanding. The approach in this first proposal was sort of leaning on the similar pattern in the signal_pending_state() case in try_to_block_task(). Is that just a hard exception to the rule here?

signal_pending_state() looks at the task_state only at a SM_NONE / SM_RTLOCK_WAIT preemption point before making a call and only sets to TASK_RUNNING if the state is (TASK_INTERRUPTIBLE | TASK_WAKEKILL) and there is a signal pending. All the wakeups for that are routed through

Everything outside that state and coming from a SM_PREEMPT path will leave the task state untouched in __schedule() until it hits the natural SM_NONE / SM_RTLOCK_WAIT preemption point.

Only looking at ->blocked_on, and not the prev_state to set prev to TASK_RUNNING can be dangerous. That said, it would be stupid for a task to sets a special state and decide to grab a mutex which overrides the state.

With the latch design, we are more or less aligned with something similar to signal_pending_state() that looks at the tasks state from a safe preemption point.

Signature · 2 lines
Thanks and Regards,
Prateek
#16 lore raw

PATCH 2/2 locking: mutex: Fix proxy-exec potentially deactivating tasks marked TASK_RUNNING

In reply to John Stultz

Vineeth found came up with a test driver that could trip up workqueue stalls. After fixing one issue this test found, Vineeth reported the test was still failing.

Greatly simplified, a task that tries to take a mutex already owned by another task that is sleeping, can hit a edge case in the mutex_lock_common() case.

If the task fails to get the lock, calls into schedule, but gets a spurious wakeup, it will find that it is first waiter, and go into the mutex_optimistic_spin() logic. Though before calling mutex_optimistic_spin(), we clear task blocked_on state, since mutex_optimistic_spin() may call schedule() if need_resched() is set.

After mutex_optimistic_spin() fails, we set blocked_on again, restart the main mutex loop, try to take the lock and call into schedule_preempt_disabled().

From there, with proxy-execution, we'll see the task is blocked_on, follow the chain, see the owner is sleeping and dequeue the waiting task from the runqueue.

This all sounds fine and reasonable. But what I had missed is that in mutex_optimistic_spin(), not only do we call schedule() but we set TASK_RUNNABLE right before doing so.

This is ok for that invocation of schedule(). But when we come back we re-set the blocked_on we had just cleared, but we do not re-set the task state to TASK_INTERRUPTIBLE/UNINTERRUPTIBLE.

This means we have a task that is blocked_on & TASK_RUNNABLE, so when the proxy execution code dequeues the task, we are in trouble since future wakeups will be shortcut by the ttwu_state_match() check.

Thus, to avoid this, after mutex_optimistic_spin(), set the task state back when we set blocked_on.

Many many thanks again to Vineeth for his very useful testing driver that uncovered this long hidden bug, that I hadn't tripped in all my testing! Very impressed with the problems he's uncovered!

---
Cc: Vineeth Pillai <[email protected]>
Cc: Sonam Sanju <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Kunwu Chan <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: Qais Yousef <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Waiman Long <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Metin Kaya <[email protected]>
Cc: Xuewen Yan <[email protected]>
Cc: K Prateek Nayak <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Daniel Lezcano <[email protected]>
Cc: Suleiman Souhlal <[email protected]>
Cc: kuyo chang <[email protected]>
Cc: hupu <[email protected]>
Cc: [email protected]

---
 kernel/locking/mutex.c | 1 +
 1 file changed, 1 insertion(+)
kernel/locking/mutex.c +1 -0
index 09534628dc01a..a93d4c6bee1a3 100644
@@ -763,6 +763,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas
 			raw_spin_lock_irqsave(&lock->wait_lock, flags);
 			raw_spin_lock(&current->blocked_lock);
 			__set_task_blocked_on(current, lock);
+			set_current_state(state);
 
 			if (opt_acquired)
 				break;
Signature · 1 line
2.54.0.545.g6539524ca2-goog
reviewer Reviewed-by on v1 2/2
#17 lore raw

Re: [PATCH 2/2] locking: mutex: Fix proxy-exec potentially deactivating tasks marked TASK_RUNNING

Hello John,

Quoted 39 lines · John Stultz

Vineeth found came up with a test driver that could trip up workqueue stalls. After fixing one issue this test found, Vineeth reported the test was still failing.

Greatly simplified, a task that tries to take a mutex already owned by another task that is sleeping, can hit a edge case in the mutex_lock_common() case.

If the task fails to get the lock, calls into schedule, but gets a spurious wakeup, it will find that it is first waiter, and go into the mutex_optimistic_spin() logic. Though before calling mutex_optimistic_spin(), we clear task blocked_on state, since mutex_optimistic_spin() may call schedule() if need_resched() is set.

After mutex_optimistic_spin() fails, we set blocked_on again, restart the main mutex loop, try to take the lock and call into schedule_preempt_disabled().

From there, with proxy-execution, we'll see the task is blocked_on, follow the chain, see the owner is sleeping and dequeue the waiting task from the runqueue.

This all sounds fine and reasonable. But what I had missed is that in mutex_optimistic_spin(), not only do we call schedule() but we set TASK_RUNNABLE right before doing so.

This is ok for that invocation of schedule(). But when we come back we re-set the blocked_on we had just cleared, but we do not re-set the task state to TASK_INTERRUPTIBLE/UNINTERRUPTIBLE.

This means we have a task that is blocked_on & TASK_RUNNABLE, so when the proxy execution code dequeues the task, we are in trouble since future wakeups will be shortcut by the ttwu_state_match() check.

Thus, to avoid this, after mutex_optimistic_spin(), set the task state back when we set blocked_on.

Quoted 8 lines

Many many thanks again to Vineeth for his very useful testing driver that uncovered this long hidden bug, that I hadn't tripped in all my testing! Very impressed with the problems he's uncovered!

I think this too requires a:

Fixes be41bde4c3a8 ("sched: Add an initial sketch of the find_proxy_task() function")

With that, feel free to include:

Quoted 37 lines

---
Cc: Vineeth Pillai <[email protected]>
Cc: Sonam Sanju <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Kunwu Chan <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: Qais Yousef <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Waiman Long <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Metin Kaya <[email protected]>
Cc: Xuewen Yan <[email protected]>
Cc: K Prateek Nayak <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Daniel Lezcano <[email protected]>
Cc: Suleiman Souhlal <[email protected]>
Cc: kuyo chang <[email protected]>
Cc: hupu <[email protected]>
Cc: [email protected]

---
 kernel/locking/mutex.c | 1 +
 1 file changed, 1 insertion(+)
kernel/locking/mutex.c +0 -0
index 09534628dc01a..a93d4c6bee1a3 100644

kernel/locking/mutex.c __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas · line 769

Quoted 7 lines
kernel/locking/mutex.c +1 -0
@@ -763,6 +763,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas
 			raw_spin_lock_irqsave(&lock->wait_lock, flags);
 			raw_spin_lock(&current->blocked_lock);
 			__set_task_blocked_on(current, lock);
+			set_current_state(state);
 
 			if (opt_acquired)
 				break;

nit.

As a micro-optimization, you can probably move the __set_task_blocked_on() and set_current_state() after this break.

Signature · 2 lines
Thanks and Regards,
Prateek
#18 lore raw

Re: [PATCH 2/2] locking: mutex: Fix proxy-exec potentially deactivating tasks marked TASK_RUNNING

Quoted 8 lines · K Prateek Nayak

On 4/28/2026 12:08 AM, John Stultz wrote:
I think this too requires a:

Fixes be41bde4c3a8 ("sched: Add an initial sketch of the find_proxy_task() function")

With that, feel free to include:

Thanks so much for looking this over!

Quoted 10 lines
Quoted 10 lines
kernel/locking/mutex.c +1 -0
index 09534628dc01a..a93d4c6bee1a3 100644
@@ -763,6 +763,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas
                      raw_spin_lock_irqsave(&lock->wait_lock, flags);
                      raw_spin_lock(&current->blocked_lock);
                      __set_task_blocked_on(current, lock);
+                     set_current_state(state);
 
Quoted 7 lines

kernel/locking/mutex.c __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas · line 769

Quoted 2 lines
kernel/locking/mutex.c +0 -0
                      if (opt_acquired)
                              break;

nit.

As a micro-optimization, you can probably move the __set_task_blocked_on() and set_current_state() after this break.

So my main reason for setting it before the break makes it symmetric with the clearing that happens just outside the loop. Feels a little cleaner to me for it to all match.

thanks
-john

#19 lore raw
In reply to John Stultz
Quoted 12 lines · John Stultz

Hey All,
So in testing with the full Proxy Execution series,
Vineeth Pillai managed to trip some interesting bugs which initially looked to be KVM or RCU related[1], which he later diagnosed as Proxy Execution related and created a useful test driver to reproduce.

I found these same issues could be triggered with the upstream portions of Proxy Execution, so I wanted to send along these fixes for 7.1-rc

Quoted 8 lines

Again, a huge thanks to Vineeth for uncovering these issues that have evaded all my stress testing so far!

Thanks
-john

[1]: https://lore.kernel.org/lkml/[email protected]/

Thanks John, Vineeth.

From the SRCU side, this helps explain the earlier KVM/RCU-looking stall pattern as a downstream symptom, with trigger likely in proxy-exec scheduler/mutex interaction.

It would be good to confirm whether the original workload still reproduces (process_srcu pending + irqfd-side stalls) after applying this series, and whether there is a proxy-exec on/off comparison under the same setup.

Thanks for driving this down.

Quoted 40 lines
Cc Vineeth Pillai <[email protected]>
Cc Sonam Sanju <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Kunwu Chan <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Qais Yousef <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Will Deacon <[email protected]>
Cc Waiman Long <[email protected]>
Cc Boqun Feng <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Metin Kaya <[email protected]>
Cc Xuewen Yan <[email protected]>
Cc K Prateek Nayak <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Daniel Lezcano <[email protected]>
Cc Suleiman Souhlal <[email protected]>
Cc kuyo chang <[email protected]>

John Stultz (2):
sched: proxy-exec: Close race causing workqueue work being delayed
locking: mutex: Fix proxy-exec potentially deactivating tasks marked TASK_RUNNING

 kernel/locking/mutex.c | 1 +
 kernel/sched/core.c | 11 +++++++++++
 2 files changed, 12 insertions(+)
Signature · 1 line
2.54.0.545.g6539524ca2-goog