Skip to contentSkip to conversation

sched: Skip redundant operations for proxy tasks needing return migration

RFC In review hupu 2 participants 17 messages Apr 7, 2025 – Jun 14, 2025 Scheduler

lore
Share & tools
lore

hupu

author
#0 lore raw

RFC 1/1 sched: Skip redundant operations for proxy tasks needing return migration

Move the proxy_needs_return() check earlier in ttwu_runnable() to avoid unnecessary scheduling operations when a proxy task requires return migration to its original CPU.

The current implementation performs several operations (rq clock update, enqueue, and wakeup preemption checks) before checking for return migration needs. This is inefficient because:

1. For tasks needing return migration, these operations are redundant since the task will be dequeued from current rq anyway
2. The task may not even be allowed to run on current CPU due to possible affinity changes during blocking
3. The proper CPU selection will be handled by select_task_rq() in the subsequent try_to_wake_up() logic

By moving the proxy_needs_return() check to the beginning, we:
- Avoid unnecessary rq clock updates
- Skip redundant enqueue operations
- Eliminate meaningless wakeup preemption checks
- Let the normal wakeup path handle proper CPU selection

This optimization is particularly valuable in proxy execution scenarios where tasks frequently migrate between CPUs.

---
 kernel/sched/core.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/sched/core.c +4 -4
index ca4ca739eb85..ebb4bc1800e3 100644
@@ -4162,6 +4162,10 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 
 	rq = __task_rq_lock(p, &rf);
 	if (task_on_rq_queued(p)) {
+		if (proxy_needs_return(rq, p)) {
+			_trace_sched_pe_return_migration(p);
+			goto out;
+		}
 		update_rq_clock(rq);
 		if (p->se.sched_delayed) {
 			proxy_remove_from_sleeping_owner(p);
@@ -4174,10 +4178,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 			 */
 			wakeup_preempt(rq, p, wake_flags);
 		}
-		if (proxy_needs_return(rq, p)) {
-			_trace_sched_pe_return_migration(p);
-			goto out;
-		}
 		ttwu_do_wakeup(p);
 		ret = 1;
 	}
Signature · 1 line
2.47.0
author
#1 lore raw

Dear maintainer,
My sincere apologies for the oversight in my previous submission. I realized I failed to provide the repository and branch details for the patch context. Here are the specifics:

The patch is based on the following repository:
https://github.com/johnstultz-work/linux-dev.git

Specifically, the changes are present in these Proxy Execution development branches:
a) proxy-exec-WIP
b) proxy-exec-v15-WIP

The changes are part of the ongoing Proxy Execution feature development in these branches.
Please let me know if you need further clarifications. Thank you for your understanding and patience.

Quoted 59 lines · hupu

Move the proxy_needs_return() check earlier in ttwu_runnable() to avoid unnecessary scheduling operations when a proxy task requires return migration to its original CPU.

The current implementation performs several operations (rq clock update, enqueue, and wakeup preemption checks) before checking for return migration needs. This is inefficient because:

1. For tasks needing return migration, these operations are redundant since the task will be dequeued from current rq anyway
2. The task may not even be allowed to run on current CPU due to possible affinity changes during blocking
3. The proper CPU selection will be handled by select_task_rq() in the subsequent try_to_wake_up() logic

By moving the proxy_needs_return() check to the beginning, we:
- Avoid unnecessary rq clock updates
- Skip redundant enqueue operations
- Eliminate meaningless wakeup preemption checks
- Let the normal wakeup path handle proper CPU selection

This optimization is particularly valuable in proxy execution scenarios where tasks frequently migrate between CPUs.

---
 kernel/sched/core.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/sched/core.c +4 -5
index ca4ca739eb85..ebb4bc1800e3 100644
@@ -4162,6 +4162,10 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 
        rq = __task_rq_lock(p, &rf);
        if (task_on_rq_queued(p)) {
+               if (proxy_needs_return(rq, p)) {
+                       _trace_sched_pe_return_migration(p);
+                       goto out;
+               }
                update_rq_clock(rq);
                if (p->se.sched_delayed) {
                        proxy_remove_from_sleeping_owner(p);
@@ -4174,10 +4178,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
                         */
                        wakeup_preempt(rq, p, wake_flags);
                }
-               if (proxy_needs_return(rq, p)) {
-                       _trace_sched_pe_return_migration(p);
-                       goto out;
-               }
                ttwu_do_wakeup(p);
                ret = 1;
        }
--

2.47.0

#2 lore raw
In reply to hupu
Quoted 16 lines · hupu

Move the proxy_needs_return() check earlier in ttwu_runnable() to avoid unnecessary scheduling operations when a proxy task requires return migration to its original CPU.

The current implementation performs several operations (rq clock update, enqueue, and wakeup preemption checks) before checking for return migration needs. This is inefficient because:

1. For tasks needing return migration, these operations are redundant since the task will be dequeued from current rq anyway
2. The task may not even be allowed to run on current CPU due to possible affinity changes during blocking
3. The proper CPU selection will be handled by select_task_rq() in the subsequent try_to_wake_up() logic

Quoted 8 lines

By moving the proxy_needs_return() check to the beginning, we:
- Avoid unnecessary rq clock updates
- Skip redundant enqueue operations
- Eliminate meaningless wakeup preemption checks
- Let the normal wakeup path handle proper CPU selection

This optimization is particularly valuable in proxy execution scenarios where tasks frequently migrate between CPUs.

Hey hupu! Thanks again for your review of the proxy series and your sharing of this improvement!

The proposal sounds attractive, however...

Quoted 17 lines
kernel/sched/core.c +4 -0
index ca4ca739eb85..ebb4bc1800e3 100644
@@ -4162,6 +4162,10 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 
        rq = __task_rq_lock(p, &rf);
        if (task_on_rq_queued(p)) {
+               if (proxy_needs_return(rq, p)) {
+                       _trace_sched_pe_return_migration(p);
+                       goto out;
+               }
                update_rq_clock(rq);
                if (p->se.sched_delayed) {
                        proxy_remove_from_sleeping_owner(p);
@@ -4174,10 +4178,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
                         */

kernel/sched/core.c static int ttwu_runnable(struct task_struct *p, int wake_flags)

Quoted 10 lines
kernel/sched/core.c +0 -5
                        wakeup_preempt(rq, p, wake_flags);
                }
-               if (proxy_needs_return(rq, p)) {
-                       _trace_sched_pe_return_migration(p);
-                       goto out;
-               }
                ttwu_do_wakeup(p);
                ret = 1;
        }
--

Unfortunately this patch crashes pretty quickly in my testing. The first issue was proxy_needs_return() calls deactivate_task() w/ DEQUEUE_NOCLOCK, which causes warnings when the update_rq_clock() hasn't been called. Preserving the update_rq_clock() line before checking proxy_needs_return() avoided that issue, but then I saw hangs during bootup, which I suspect is due to us shortcutting over the sched_delayed case.

Moving the proxy_needs_return above the if(task_on_cpu()) wakeup_preempt() logic booted ok, but I'm still a little hesitant of what side-effects that might cause.

Part of the approach with proxy_needs_return() from ttwu_runnable() is making sure everything is ready for the queued task to run, but in the case where we need return migration, to dequeue the task and cause ttwu_runnable() to fail, forcing the rest of try_to_wake_up() to complete. I do really appreciate your thoughts here though, as I agree that the potential enqueue/dequeue/enqueue could be wasteful.

I'll take a bit more time to see how we might be able to avoid this extra work without skipping needed steps. If you have further ideas, I'd love to hear them!

thanks again!
-john

author
#3 lore raw

Hi John:
Thank you for your feedback.

Quoted 9 lines · John Stultz

Unfortunately this patch crashes pretty quickly in my testing. The first issue was proxy_needs_return() calls deactivate_task() w/ DEQUEUE_NOCLOCK, which causes warnings when the update_rq_clock() hasn't been called. Preserving the update_rq_clock() line before checking proxy_needs_return() avoided that issue, but then I saw hangs during bootup, which I suspect is due to us shortcutting over the sched_delayed case.

Quoted 3 lines

Moving the proxy_needs_return above the if(task_on_cpu()) wakeup_preempt() logic booted ok, but I'm still a little hesitant of what side-effects that might cause.

I’m sorry for the confusion caused by this patch. Here is the rationale behind my approach:

To ensure that donor tasks can get a suitable CPU and avoid negative impacts from the Proxy-Execution on load balancing, proxy_needs_return() in ttwu_runnable() should return false for all donor tasks. This allows try_to_wake_up() to use set_task_cpu to reselect a CPU for the donor tasks, unless the donor is already running on a CPU.

This patch worked correctly on my QEMU-based test platform, it seems our testing methods might differ. Could you please share the details of your testing environment and methodology? I’ll try to replicate the issue using the same approach.

In the meantime, I will carefully revisit the logic in this patch to ensure its correctness and consistency. Once I’ve completed the review, I look forward to further discussing the details with you.

Thank you again for your valuable feedback!

Best regards,
hupu

#4 lore raw
Quoted 13 lines · hupu

Hi John:
Thank you for your feedback.

Quoted 8 lines · John Stultz

Unfortunately this patch crashes pretty quickly in my testing. The first issue was proxy_needs_return() calls deactivate_task() w/ DEQUEUE_NOCLOCK, which causes warnings when the update_rq_clock() hasn't been called. Preserving the update_rq_clock() line before checking proxy_needs_return() avoided that issue, but then I saw hangs during bootup, which I suspect is due to us shortcutting over the sched_delayed case.

...

Quoted 4 lines

This patch worked correctly on my QEMU-based test platform, it seems our testing methods might differ. Could you please share the details of your testing environment and methodology? I’ll try to replicate the issue using the same approach.

Usually I boot qemu with 64 cores, and have found stress testing running the following separate programs frequently uncovers issues:

#cyclictest from the rt-test suite to add some rt load cyclictest -t -p90

# rerunning the test_ww_mtuex logic (enabled by a patch in my full
proxy-exec series)
while true; do echo 1 > /sys/kernel/test_ww_mutex/run_tests ; sleep 5; done

# Along with my prio-inversion-demo (which ensures we do the proxying paths)
# https://github.com/johnstultz-work/priority-inversion-demo
while true; do ./run.sh; sleep 10 ; done

# Sometimes I also throw in the ltp sched_football test:
# https://github.com/linux-test-project/ltp/tree/master/testcases/realtime/func/sched_football
while true; do ./sched_football -m | grep Result ; sleep 1; done

But I'm always looking for other scheduling loads to trip up bugs, so open to suggestions.

thanks
-john

#5 lore raw
Quoted 15 lines · John Stultz
Quoted 13 lines · hupu

Hi John:
Thank you for your feedback.

Quoted 8 lines · John Stultz

Unfortunately this patch crashes pretty quickly in my testing. The first issue was proxy_needs_return() calls deactivate_task() w/ DEQUEUE_NOCLOCK, which causes warnings when the update_rq_clock() hasn't been called. Preserving the update_rq_clock() line before checking proxy_needs_return() avoided that issue, but then I saw hangs during bootup, which I suspect is due to us shortcutting over the sched_delayed case.

Quoted 8 lines

...

Quoted 4 lines

This patch worked correctly on my QEMU-based test platform, it seems our testing methods might differ. Could you please share the details of your testing environment and methodology? I’ll try to replicate the issue using the same approach.

Usually I boot qemu with 64 cores, and have found stress testing running the following separate programs frequently uncovers issues:

Oh, also I enable locktorture and boot with:
"torture.random_shuffle=1 locktorture.writer_fifo=1 locktorture.torture_type=mutex_lock locktorture.nested_locks=8 locktorture.rt_boost=1 locktorture.rt_boost_factor=50 locktorture.stutter=0 "

thanks
-john

author
#6 lore raw
Quoted 7 lines · John Stultz

Oh, also I enable locktorture and boot with:
"torture.random_shuffle=1 locktorture.writer_fifo=1 locktorture.torture_type=mutex_lock locktorture.nested_locks=8 locktorture.rt_boost=1 locktorture.rt_boost_factor=50 locktorture.stutter=0 "

Hi John,

Thank you so much for your feedback.

I’ll follow the methods you’ve outlined to try and reproduce the same issue, and will get back to you for further discussion.

Thanks again,
hupu

author
#7 lore raw

Hi John,
Sorry for the delayed response.

Quoted 10 lines

Usually I boot qemu with 64 cores, and have found stress testing running the following separate programs frequently uncovers issues:

#cyclictest from the rt-test suite to add some rt load cyclictest -t -p90

# rerunning the test_ww_mtuex logic (enabled by a patch in my full
proxy-exec series)
while true; do echo 1 > /sys/kernel/test_ww_mutex/run_tests ; sleep 5; done
Quoted 8 lines

# Along with my prio-inversion-demo (which ensures we do the proxying paths) # ? https://github.com/johnstultz-work/priority-inversion-demo
while true; do ./run.sh; sleep 10 ; done

# Sometimes I also throw in the ltp sched_football test:
# https://github.com/linux-test-project/ltp/tree/master/testcases/realtime/func/sched_football
while true; do ./sched_football -m | grep Result ; sleep 1; done

I followed your suggested approach and successfully reproduced the issue you mentioned. The system experienced a hang during bootup and triggered a crash due to a watchdog timeout. This issue is related to the logic of sched_delayed. I will provide a detailed description of the debugging steps below. I apologize in advance if the explanation becomes somewhat lengthy.

I added debugging information to capture the task execution logic prior to the crash. The corresponding patch is provided below:

kernel/sched/core.c +0 -0
old mode 100644
new mode 100755
index 06e9924d3f77..c9adc17f53e5
@@ -2098,6 +2098,14 @@ static inline void __activate_task(struct rq

*rq, struct task_struct *p, int fla
WRITE_ONCE(p->on_rq, TASK_ON_RQ_QUEUED); ASSERT_EXCLUSIVE_WRITER(p->on_rq);
+
+ if (p->blocked_on) {
+ trace_printk("DEBUG BYHP[%d]: "
+ "p=%s$%d, on_cpu=%d, on_rq=%d, __state=%d, sched_delayed=%d\n",
+ __LINE__,
+ p->comm, p->pid, p->on_cpu, p->on_rq, p->__state,
+ p->se.sched_delayed);
+ }
}

#ifdef CONFIG_SCHED_PROXY_EXEC
@@ -4160,6 +4168,19 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
rq = __task_rq_lock(p, &rf);
if (task_on_rq_queued(p)) {
update_rq_clock(rq);
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+
+ if (p->blocked_on) {
+ trace_printk("DEBUG BYHP[%d]: "
+ "p=%s$%d, on_cpu=%d, on_rq=%d, __state=%d, sched_delayed=%d\n",
+ __LINE__,
+ p->comm, p->pid, p->on_cpu, p->on_rq, p->__state,
+ p->se.sched_delayed);
+ }
+ goto out;
+ }
+
if (p->se.sched_delayed) { proxy_remove_from_sleeping_owner(p);
enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
@@ -4171,10 +4192,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
*/
wakeup_preempt(rq, p, wake_flags);
}
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
ttwu_do_wakeup(p);
ret = 1;
}
@@ -4377,6 +4394,14 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
struct rq *rq = cpu_rq(cpu);
struct rq_flags rf;

+ if (p->blocked_on) {
+ trace_printk("DEBUG BYHP[%d]: "
+ "p=%s$%d, on_cpu=%d, on_rq=%d, __state=%d, sched_delayed=%d\n",
+ __LINE__,
+ p->comm, p->pid, p->on_cpu, p->on_rq, p->__state,
+ p->se.sched_delayed);
+ }
+
if (ttwu_queue_wakelist(p, cpu, wake_flags))
return;

kernel/sched/fair.c +0 -0
old mode 100644
new mode 100755
index 1f707b614142..a3857b058ed6
@@ -5623,6 +5623,8 @@ static int dequeue_entities(struct rq *rq,

struct sched_entity *se, int flags);
static struct sched_entity *
pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq)
{
+ struct task_struct *p;
+
/*
* Enabling NEXT_BUDDY will affect latency but not fairness.
*/
@@ -5635,7 +5637,25 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq)

struct sched_entity *se = pick_eevdf(cfs_rq);
if (se->sched_delayed) {
+ p = task_of(se);
+ if (p->blocked_on) {
+ trace_printk("DEBUG BYHP[%d]: "
+ "p=%s$%d, on_cpu=%d, on_rq=%d, __state=%d, sched_delayed=%d\n",
+ __LINE__,
+ p->comm, p->pid, p->on_cpu, p->on_rq, p->__state,
+ p->se.sched_delayed);
+ }
+
dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+
+ if (p->blocked_on) {
+ trace_printk("DEBUG BYHP[%d]: "
+ "p=%s$%d, on_cpu=%d, on_rq=%d, __state=%d, sched_delayed=%d\n",
+ __LINE__,
+ p->comm, p->pid, p->on_cpu, p->on_rq, p->__state,
+ p->se.sched_delayed);
+ }
+
/*
* Must not reference @se again, see __block_task(). */

At the time of the crash, the dmesg logs included multiple warnings similar to the following. For analysis, I will use the task with PID 151 as an example.

[  124.912061][   T61] INFO: task kworker/u32:8:151 blocked for more
than 62 seconds.
[  124.912708][   T61]      on_rq: 0 rq: 4 state: 0x2  bo_state: 1
[  124.913133][   T61]       Not tainted 6.12.0-00050-gf6c6e0ea8a1d-dirty #93
[  124.913535][   T61] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  124.991597][   T61] task:kworker/u32:8   state:D stack:0
pid:151   tgid:151   ppid:2      flags:0x00000008
[  124.992829][   T61] Workqueue: test-ww_mutex stress_reorder_work
[  124.993361][   T61] Call trace:
[  124.993677][   T61]  __switch_to+0xe4/0x1a4
[  125.027050][   T61]  __schedule+0x4e0/0x1960
[  125.031483][   T61]  schedule+0x4c/0x130
[  125.031811][   T61]  schedule_preempt_disabled+0x28/0x40
[  125.032150][   T61]  __ww_mutex_lock.constprop.28+0xbdc/0x1960
[  125.032539][   T61]  ww_mutex_lock+0x48/0x98
[  125.032870][   T61]  stress_reorder_work+0x268/0x3c4
[  125.033287][   T61]  process_scheduled_works+0x254/0x694
[  125.033728][   T61]  worker_thread+0x16c/0x310
[  125.034078][   T61]  kthread+0x128/0x134
[  125.034419][   T61]  ret_from_fork+0x10/0x20

Switching the crash environment to the context of task 151, the info stack command reveals that task 151 is blocked in __ww_mutex_lock while waiting for the lock lock=lock@entry=0xffff000006ab5060.

crash_arm64> set 151

    PID: 151
COMMAND: "kworker/u32:8"
   TASK: ffff0000060ec180  [THREAD_INFO: ffff0000060ec180]
    CPU: 4
  STATE: TASK_UNINTERRUPTIBLE

crash_arm64> info stack
#0 __switch_to (prev=prev@entry=0xffff0000060ec180, next=next@entry=0xffff000005444180) at arch/arm64/kernel/process.c:609 #1 0xffff8000813bf6ac in context_switch (rf=0xffff800086123b40, next=0xffff000005444180, prev=0xffff0000060ec180, rq=0xffff00002cbf48c0) at kernel/sched/core.c:5831
#2 __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:7637 #3 0xffff8000813c0b78 in __schedule_loop (sched_mode=<optimised out>) at kernel/sched/core.c:7718
#4 schedule () at kernel/sched/core.c:7733
#5 0xffff8000813c0c84 in schedule_preempt_disabled () at kernel/sched/core.c:7790
#6 0xffff8000813c6054 in __mutex_lock_common (use_ww_ctx=true, ww_ctx=0xffff800086123cd0, ip=18446603338369690404, nest_lock=<optimised out>, subclass=0, state=2203262976, lock=0xffff000006ab5060) at kernel/locking/mutex.c:665
#7 __ww_mutex_lock (lock=lock@entry=0xffff000006ab5060, state=2203262976, state@entry=2, ip=ip@entry=18446603338369690404, ww_ctx=ww_ctx@entry=0xffff800086123cd0, subclass=0) at kernel/locking/mutex.c:762
#8 0xffff8000813c6ea0 in ww_mutex_lock (lock=lock@entry=0xffff000006ab5060, ctx=0xffff800086123cd0, ctx@entry=0xffff800086123d40) at kernel/locking/mutex.c:879
#9 0xffff8000800f6b24 in ww_mutex_lock_slow (ctx=0xffff800086123d40, lock=0xffff000006ab5060) at ./include/linux/ww_mutex.h:299
#10 stress_reorder_work (work=0xffff0000065938f0) at kernel/locking/test-ww_mutex.c:532
#11 0xffff80008007ef84 in process_one_work (work=0xffff0000065938f0, worker=0xffff000005db1300) at kernel/workqueue.c:3229
#12 process_scheduled_works (worker=worker@entry=0xffff000005db1300) at kernel/workqueue.c:3310
#13 0xffff80008007fa3c in worker_thread (__worker=0xffff000005db1300) at kernel/workqueue.c:3391
#14 0xffff80008008b2e8 in kthread (_create=0xffff000006848400) at kernel/kthread.c:389
#15 0xffff8000800162dc in ret_from_fork () at arch/arm64/kernel/entry.S:860 crash_arm64>

Using the following commands, it was determined that the mutex lock is held by the task with PID 165. The field __state = 0 indicates that the task is in the TASK_RUNNING state, while on_rq = 0 indicates that the task has been removed from the runqueue. These two fields are contradictory, preventing the task with PID 165 from ever getting a chance to run, which is the direct cause of the system hang.

crash_arm64> struct -x mutex.owner 0xffff000006ab5060

  owner = {
    counter = 0xffff0000063ba0c1

},
crash_arm64> struct task_struct.comm,pid,__state,thread_info.cpu,se.on_rq,on_rq,on_cpu,blocked_on,blocked_on_state 0xffff0000063ba0c0
comm = "kworker/u32:22\000",
pid = 165,
__state = 0,
thread_info.cpu = 4
se.on_rq = 0 '\000',
on_rq = 0,
on_cpu = 0,
blocked_on = 0xffff000006aa3520,
blocked_on_state = BO_RUNNABLE,
crash_arm64>

The following trace information provides the execution logic of the task with PID 165 before the system hang occurred.

Line 993: kworker/u32:20-163 [004] 60.082229: bprint:

    try_to_wake_up: DEBUG BYHP[4179]: p=kworker/u32:22$165, on_cpu=0,
on_rq=2, __state=2, sched_delayed=1
Line 994:   kworker/u32:20-163   [004]    60.082250: bprint:
    try_to_wake_up: DEBUG BYHP[4402]: p=kworker/u32:22$165, on_cpu=0,
on_rq=2, __state=512, sched_delayed=1
Line 995:   kworker/u32:20-163   [004]    60.082268: bprint:
    activate_task: DEBUG BYHP[2107]: p=kworker/u32:22$165, on_cpu=0,
on_rq=1, __state=512, sched_delayed=1
Line 1002:           <idle>-0     [004]    60.087364: bprint:
     pick_task_fair: DEBUG BYHP[5646]: p=kworker/u32:22$165, on_cpu=0,
on_rq=1, __state=0, sched_delayed=1
Line 1003:           <idle>-0     [004]    60.087375: bprint:
     pick_task_fair: DEBUG BYHP[5656]: p=kworker/u32:22$165, on_cpu=0,

on_rq=0, __state=0, sched_delayed=0

The trace information indicates that the task with PID 165 has p->se.sched_delayed = 1. From the trace logs at lines 4179, 4402, and 2107, it is evident that the task with PID 165 was successfully removed from the runqueue during the try_to_wake_up process, reassigned to a new CPU, and successfully completed the enqueue operation. This behavior seems to align with our expectations.

try_to_wake_up
|-- ttwu_runnable

      |-- proxy_needs_return    // line: 4179
  |-- cpu = select_task_rq()
  |-- ttwu_queue(p, cpu)        // line: 4402
      |-- __activate_task(p)    // line: 2107

However, the logs at lines 5646 and 5656 reveal that tasks with sched_delayed = 1 are removed from the runqueue again during the pick_next_entity operation. The corresponding code logic in pick_next_entity is as follows.

struct sched_entity *se = pick_eevdf(cfs_rq);
if (se->sched_delayed) {
dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED); /*
* Must not reference @se again, see __block_task(). */
return NULL;
}

This means that in pick_task_fair -> pick_next_entity, a task that has already been successfully woken up and enqueued will be removed from the runqueue again if its sched_delayed = 1. As a result, the task will never get a chance to run, ultimately causing the system to experience hangs.

In the original code logic, the sched_delayed operation is placed before proxy_needs_return, allowing the se->sched_delayed variable to be set to 0 through enqueue_task(ENQUEUE_DELAYED) -> enqueue_task_fair -> requeue_delayed_entity. This prevents the already woken-up task from being removed from the runqueue again in the pick_next_entity function.

To verify the correctness of the above analysis, we forcibly cleared p->se.sched_delayed after the execution of the proxy_needs_return function. The corresponding patch is as follows, and with this change, the system can bootup successfully.

kernel/sched/core.c +0 -0
old mode 100644
new mode 100755
index 06e9924d3f77..0da8d308ec4b

#ifdef CONFIG_SCHED_PROXY_EXEC
@@ -4160,6 +4168,15 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
rq = __task_rq_lock(p, &rf);
if (task_on_rq_queued(p)) {
update_rq_clock(rq);
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+ /*
+ * Force clearing to verify the correctness of the analysis logic
+ */
+ p->se.sched_delayed = 0;
+ goto out;
+ }
+
if (p->se.sched_delayed) { proxy_remove_from_sleeping_owner(p);
enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
@@ -4171,10 +4188,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
*/
wakeup_preempt(rq, p, wake_flags);
}
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
ttwu_do_wakeup(p);
ret = 1;
}

The above patch was solely intended to verify the correctness of the analysis, as directly clearing p->se.sched_delayed is unreasonable. Doing so would cause tasks with non-zero lag to be dequeued, thereby undermining the fairness enforced by EEVDF. Moving the proxy_needs_return operation after the handling of sched_delayed appears to be more reasonable, as for a donor that is about to be dequeued, executing the subsequent wakeup_preempt operation is redundant. Therefore, the following patch seems reasonable, and I propose adopting it as version 2.0.

kernel/sched/core.c +0 -0
old mode 100644
new mode 100755
index 06e9924d3f77..f804c681b861

#ifdef CONFIG_SCHED_PROXY_EXEC
@@ -4164,6 +4172,10 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
proxy_remove_from_sleeping_owner(p);
enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
}
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+ goto out;
+ }
if (!task_on_cpu(rq, p)) {
/*
* When on_rq && !on_cpu the task is preempted, see if
@@ -4171,10 +4183,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
*/
wakeup_preempt(rq, p, wake_flags);
}
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
ttwu_do_wakeup(p);
ret = 1;
}

However, there is a potential issue. Whether it is the original code logic or the v2.0 patch mentioned above, directly removing a task from the runqueue in proxy_needs_return undermines the fairness enforced by EEVDF. In fact, clearing the task's p->se.sched_delayed variable first (via enqueue_task(ENQUEUE_DELAYED) -> enqueue_task_fair -> requeue_delayed_entity) and then removing task from the runqueue in proxy_needs_return is essentially no different from directly removing the task while retaining its lag. This is because the non-zero lag will eventually be discarded. Therefore, regardless of the method used, the fairness maintained by EEVDF will be compromised.

I have not yet determined the best approach to address the issue of EEVDF fairness being compromised. Perhaps we could preserve the lag during the dequeue process and handle it during the next enqueue. I look forward to further discussing this issue with you.

Finally, I have decided to adopt the following patch as version 2.0. This patch is specifically designed to avoid unnecessary wakeup_preempt checks. Regarding the fairness issue discussed earlier, I plan to submit a new patch to address it.

kernel/sched/core.c +0 -0
old mode 100644
new mode 100755
index 06e9924d3f77..f804c681b861

#ifdef CONFIG_SCHED_PROXY_EXEC
@@ -4164,6 +4172,10 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
proxy_remove_from_sleeping_owner(p);
enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
}
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+ goto out;
+ }
if (!task_on_cpu(rq, p)) {
/*
* When on_rq && !on_cpu the task is preempted, see if
@@ -4171,10 +4183,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
*/
wakeup_preempt(rq, p, wake_flags);
}
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
ttwu_do_wakeup(p);
ret = 1;
}

Thanks. Looking forward to your response.
hupu

author
#8 lore raw

Hi John,
It seems this email might have been overlooked. I am resending it to bring it back to your attention and continue the discussion.

Looking forward to your response!
hupu

#9 lore raw
In reply to hupu
Quoted 24 lines · hupu

Hi John,
Sorry for the delayed response.

Quoted 10 lines

Usually I boot qemu with 64 cores, and have found stress testing running the following separate programs frequently uncovers issues:

#cyclictest from the rt-test suite to add some rt load cyclictest -t -p90

# rerunning the test_ww_mtuex logic (enabled by a patch in my full
proxy-exec series)
while true; do echo 1 > /sys/kernel/test_ww_mutex/run_tests ; sleep 5; done
Quoted 8 lines

# Along with my prio-inversion-demo (which ensures we do the proxying paths) # ? https://github.com/johnstultz-work/priority-inversion-demo
while true; do ./run.sh; sleep 10 ; done

# Sometimes I also throw in the ltp sched_football test:
# https://github.com/linux-test-project/ltp/tree/master/testcases/realtime/func/sched_football
while true; do ./sched_football -m | grep Result ; sleep 1; done

Quoted 7 lines

I followed your suggested approach and successfully reproduced the issue you mentioned. The system experienced a hang during bootup and triggered a crash due to a watchdog timeout. This issue is related to the logic of sched_delayed. I will provide a detailed description of the debugging steps below. I apologize in advance if the explanation becomes somewhat lengthy.

Yeah. To be honest, this is a little tough to follow, particularly trying to distinguish between the debug changes and the actual change you are proposing. Especially as they are whitespace damaged.

[snip]

Quoted 40 lines

The above patch was solely intended to verify the correctness of the analysis, as directly clearing p->se.sched_delayed is unreasonable. Doing so would cause tasks with non-zero lag to be dequeued, thereby undermining the fairness enforced by EEVDF. Moving the proxy_needs_return operation after the handling of sched_delayed appears to be more reasonable, as for a donor that is about to be dequeued, executing the subsequent wakeup_preempt operation is redundant. Therefore, the following patch seems reasonable, and I propose adopting it as version 2.0.

kernel/sched/core.c +4 -0
old mode 100644
new mode 100755
index 06e9924d3f77..f804c681b861
 #ifdef CONFIG_SCHED_PROXY_EXEC
@@ -4164,6 +4172,10 @@ static int ttwu_runnable(struct task_struct *p,
 int wake_flags)
  proxy_remove_from_sleeping_owner(p);
  enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
  }
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+ goto out;
+ }
  if (!task_on_cpu(rq, p)) {
  /*

* When on_rq && !on_cpu the task is preempted, see if

kernel/sched/core.c +0 -4
@@ -4171,10 +4183,6 @@ static int ttwu_runnable(struct task_struct *p,
 int wake_flags)
  */
  wakeup_preempt(rq, p, wake_flags);
  }
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
  ttwu_do_wakeup(p);
  ret = 1;
Quoted 1 line
  }

Ok, so this matches what I had tested before modifying your earlier patch.

Quoted 10 lines

However, there is a potential issue. Whether it is the original code logic or the v2.0 patch mentioned above, directly removing a task from the runqueue in proxy_needs_return undermines the fairness enforced by EEVDF. In fact, clearing the task's p->se.sched_delayed variable first (via enqueue_task(ENQUEUE_DELAYED) -> enqueue_task_fair -> requeue_delayed_entity) and then removing task from the runqueue in proxy_needs_return is essentially no different from directly removing the task while retaining its lag. This is because the non-zero lag will eventually be discarded. Therefore, regardless of the method used, the fairness maintained by EEVDF will be compromised.

Hrm. Can you walk me through the specific case you're thinking about here?

Is the idea something like: a mutex blocked task (not sched_delayed) gets migrated to a rq, where it acts as a donor so that a lock holder can be run.
If the lock holder sleeps, it might be set as sched_delayed, but the donor will be dequeued from the rq and enqueued onto the sched_delayed sleeping owner.

And the concern is that in doing this, the donor's lag from the rq it was migrated to won't be preserved (since it isn't set as sched_delayed)?

I'll need to think on this a bit, as I don't quite have my head around how mutex blocked tasks might also end up sched_delayed.

Quoted 40 lines

I have not yet determined the best approach to address the issue of EEVDF fairness being compromised. Perhaps we could preserve the lag during the dequeue process and handle it during the next enqueue. I look forward to further discussing this issue with you.

Finally, I have decided to adopt the following patch as version 2.0. This patch is specifically designed to avoid unnecessary wakeup_preempt checks. Regarding the fairness issue discussed earlier, I plan to submit a new patch to address it.

kernel/sched/core.c +4 -0
old mode 100644
new mode 100755
index 06e9924d3f77..f804c681b861
 #ifdef CONFIG_SCHED_PROXY_EXEC
@@ -4164,6 +4172,10 @@ static int ttwu_runnable(struct task_struct *p,
 int wake_flags)
  proxy_remove_from_sleeping_owner(p);
  enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
  }
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+ goto out;
+ }
  if (!task_on_cpu(rq, p)) {
  /*

* When on_rq && !on_cpu the task is preempted, see if

kernel/sched/core.c +0 -4
@@ -4171,10 +4183,6 @@ static int ttwu_runnable(struct task_struct *p,
 int wake_flags)
  */
  wakeup_preempt(rq, p, wake_flags);
  }
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
  ttwu_do_wakeup(p);
  ret = 1;
Quoted 1 line
  }

This looks identical to the version above, or am I missing something?

thanks
-john

author
#10 lore raw

Hi John
Thank you for your response.

Quoted 3 lines

This looks identical to the version above, or am I missing something?

I sincerely apologize for the confusion caused by my unclear explanation. The complete patch for version 2.0 is as follows. In this patch, I relocated the proxy_needs_return() check to execute after sched_delayed processing but before the wakeup_preempt() judgment. This optimization allows skipping redundant wakeup_preempt() operations when a donor task must migrate back to its original CPU, as it becomes unnecessary in such cases.

Subject: [RFC] sched: Skip redundant operations when donor needs return.

Move the proxy_needs_return() check earlier in ttwu_runnable() to minimize unnecessary operations, particularly in cases where a donor task needs to migrate back to its original CPU.

---
 kernel/sched/core.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
 mode change 100644 => 100755 kernel/sched/core.c
kernel/sched/core.c +0 -0
old mode 100644
new mode 100755
index 06e9924d3f77..2c863ad53173
@@ -4164,6 +4164,10 @@ static int ttwu_runnable(struct task_struct *p,

int wake_flags)
proxy_remove_from_sleeping_owner(p);
enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
}
+ if (proxy_needs_return(rq, p)) {
+ _trace_sched_pe_return_migration(p);
+ goto out;
+ }
if (!task_on_cpu(rq, p)) {
/*
* When on_rq && !on_cpu the task is preempted, see if
@@ -4171,10 +4175,6 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
*/
wakeup_preempt(rq, p, wake_flags);
}
- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }
ttwu_do_wakeup(p);
ret = 1;
}
--
2.17.1

Quoted 9 lines

Hrm. Can you walk me through the specific case you're thinking about here?

Is the idea something like: a mutex blocked task (not sched_delayed) gets migrated to a rq, where it acts as a donor so that a lock holder can be run.
If the lock holder sleeps, it might be set as sched_delayed, but the donor will be dequeued from the rq and enqueued onto the sched_delayed sleeping owner.

Quoted 7 lines

And the concern is that in doing this, the donor's lag from the rq it was migrated to won't be preserved (since it isn't set as sched_delayed)?

I'll need to think on this a bit, as I don't quite have my head around how mutex blocked tasks might also end up sched_delayed.

I need to add some debugging logs to further investigate this issue. This may take a bit of time, and I will get back to you shortly.

Thanks.
hupu

author
#11 lore raw

Hi John

Quoted 21 lines
Quoted 11 lines

However, there is a potential issue. Whether it is the original code logic or the v2.0 patch mentioned above, directly removing a task from the runqueue in proxy_needs_return undermines the fairness enforced by EEVDF. In fact, clearing the task's p->se.sched_delayed variable first (via enqueue_task(ENQUEUE_DELAYED) -> enqueue_task_fair -> requeue_delayed_entity) and then removing task from the runqueue in proxy_needs_return is essentially no different from directly removing the task while retaining its lag. This is because the non-zero lag will eventually be discarded. Therefore, regardless of the method used, the fairness maintained by EEVDF will be compromised.

Hrm. Can you walk me through the specific case you're thinking about here?

Is the idea something like: a mutex blocked task (not sched_delayed) gets migrated to a rq, where it acts as a donor so that a lock holder can be run.
If the lock holder sleeps, it might be set as sched_delayed, but the donor will be dequeued from the rq and enqueued onto the sched_delayed sleeping owner.

Quoted 7 lines

And the concern is that in doing this, the donor's lag from the rq it was migrated to won't be preserved (since it isn't set as sched_delayed)?

I'll need to think on this a bit, as I don't quite have my head around how mutex blocked tasks might also end up sched_delayed.

There may have been an issue with my earlier analysis, and I apologize for my oversight.

During previous debugging, I noticed that when handling the sched_delayed logic in the ttwu_runnable function, there were instances where the donor's vlag was reset to zero. This raised concerns for me, as I worried that directly resetting the vlag might compromise the fairness ensured by EEVDF.

However, upon closer inspection of the code, I realized that this is actually a normal adjustment of the vlag. The purpose of this process is to check whether the sched_delayed task has repaid its past "debt" and determine whether its position in the red-black tree needs to be updated. The corresponding call stack is as follows:

ttwu_runnable
|-- if (p->se.sched_delayed) enqueue_task(flags=ENQUEUE_DELAYED)

      |-- enqueue_task_fair(flags=ENQUEUE_DELAYED)
          |-- requeue_delayed_entity(se)
              |-- update_entity_lag()
                  |-- se->vlag = entity_lag(avg_vruntime(cfs_rq), se);

Therefore, the logic in ttwu_runnable does not compromise the fairness of EEVDF.

However, the above call stack gives me another idea: Executing the sched_delayed logic seems redundant if a donor task needs to return back. This is because the two main works in the sched_delayed logic — "updating the vlag" and "adjusting the task's position in the red-black tree" — can be deferred to later stages, as shown in the call stack below:

a) STEP#1: Update the donor's vlag.
b) STEP#2: Update the donor's vruntime and deadline.
c) STEP#3: Update the donor's position in the red-black tree.

try_to_wake_up

  |-- ttwu_runnable
  |   |-- proxy_needs_return
  |       |-- deactivate_task
  |           |-- dequeue_task
  |               |-- dequeue_task_fair
  |                   |-- dequeue_entities
  |                       |-- dequeue_entity
  |                           |-- update_entity_lag         (STEP #1)
  |                               |-- se->vlag = entity_lag()
  |
  |-- new_cpu = select_task_rq  //Re-select a new CPU for the donor task
  |-- ttwu_queue(p, new_cpu)

|-- ttwu_do_activate
|-- activate_task(flags=ENQUEUE_WAKEUP)
|-- __activate_task(flags=ENQUEUE_WAKEUP)
|-- enqueue_task(flags=ENQUEUE_WAKEUP)
|-- enqueue_task_fair(flags=ENQUEUE_WAKEUP)
|-- enqueue_entity(flags=ENQUEUE_WAKEUP)
|-- place_entity (STEP #2)
|-- __enqueue_entity (STEP #3)

In other words, if the donor task needs to be migrated back, the sched_delayed logic can be skipped entirely. Therefore, the final PATCH should be as follows.
In this PATCH, proxy_needs_return is moved ahead of the sched_delayed logic, and the p->se.sched_delayed variable is forcefully set to 0.

kernel/sched/core.c +0 -0
old mode 100644
new mode 100755
index 06e9924d3f77..f585e28b19eb
@@ -4160,6 +4160,18 @@ static int ttwu_runnable(struct task_struct *p,

int wake_flags)

        rq = __task_rq_lock(p, &rf);
        if (task_on_rq_queued(p)) {
                update_rq_clock(rq);

+ if (proxy_needs_return(rq, p)) {
+ /*
+ * Force clear, as the work of updating the donor's vlag
+ * and its position in the red-black tree can
be deferred
+ * to later stages.
+ */
+ p->se.sched_delayed = 0;
+
+ _trace_sched_pe_return_migration(p);
+ goto out;
+ }
+

                if (p->se.sched_delayed) {
                        proxy_remove_from_sleeping_owner(p);
                        enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
@@ -4171,10 +4183,6 @@ static int ttwu_runnable(struct task_struct *p,
int wake_flags)
                         */
                        wakeup_preempt(rq, p, wake_flags);
                }

- if (proxy_needs_return(rq, p)) {
- _trace_sched_pe_return_migration(p);
- goto out;
- }

                ttwu_do_wakeup(p);
                ret = 1;
        }

This PATCH can boot up successfully. Furthermore, I triggered the mutex-related test cases by executing the following command, and it has been running for approximately 20 hours without any issues.

# while true; do echo 1 > /sys/kernel/test_ww_mutex/run_tests; sleep 10; done

Looking forward to your reply. Thanks.
hupu

author
#12 lore raw

Hi John,
It has been a few days without any response to this email. I'm resending it for further discussion.

Best regards,
hupu

#13 lore raw
Quoted 4 lines · hupu

Hi John,
It has been a few days without any response to this email. I'm resending it for further discussion.

Apologies, I'm swamped at the moment with some other items and haven't had time to give your suggestions enough serious thought to reply. :(

I have made some slow progress on a few items. Suggested by Suleiman, I've reworked the migration logic to be considered after the owner on_rq check, so we avoid migrating tasks to only realize they are waiting on a sleeping owner and pulling them off the rq. This should help reduce the number of migrations we make.

Unfortunately I've also been tripping over a crash with the full series (not the portion I'm currently submitting upstream) that often takes 30+ hours to reproduce and extra debugging logic seems to make it even harder to hit. With all the other tasks, keeping enough context in my head to make progress on this has been frustratingly slow.

I'll hopefully have something to share soon and I can

thanks
-john

author
#14 lore raw

Hi John
Thank you for your response.

Quoted 16 lines · John Stultz
Quoted 4 lines · hupu

Hi John,
It has been a few days without any response to this email. I'm resending it for further discussion.

Apologies, I'm swamped at the moment with some other items and haven't had time to give your suggestions enough serious thought to reply. :(

I have made some slow progress on a few items. Suggested by Suleiman, I've reworked the migration logic to be considered after the owner on_rq check, so we avoid migrating tasks to only realize they are waiting on a sleeping owner and pulling them off the rq. This should help reduce the number of migrations we make.

Quoted 7 lines

Unfortunately I've also been tripping over a crash with the full series (not the portion I'm currently submitting upstream) that often takes 30+ hours to reproduce and extra debugging logic seems to make it even harder to hit. With all the other tasks, keeping enough context in my head to make progress on this has been frustratingly slow.

If you'd like, I'd be happy to help investigate this crash issue. :)

To do that, I would need some additional information, such as the vmlinux file, coredump, and the source code at the point where the crash is triggered.

Quoted 2 lines

I'll hopefully have something to share soon and I can

Look forward to hearing from you with good news.

Thanks.
hupu

author
#15 lore raw

Hi John

Quoted 5 lines

Unfortunately I've also been tripping over a crash with the full series (not the portion I'm currently submitting upstream) that often takes 30+ hours to reproduce and extra debugging logic seems to make it even harder to hit.

I encountered a strange crash while running the code from the proxy-exec-v17-WIP branch for over 150+ hours. I'm currently unsure if this issue is the same as the one you mentioned.

Here is the dmesg log at the time of the crash:

[550195.352234][    C2] watchdog: BUG: soft lockup - CPU#2 stuck for
32s! [lock_torture_wr:139]
[550195.374670][    C2] Modules linked in:
[550195.375936][    C2] irq event stamp: 189580674
[550195.376275][    C2] hardirqs last  enabled at (189580673):
[<ffff8000813b7840>] exit_to_kernel_mode.isra.6+0xf8/0x130
[550195.393675][    C2] hardirqs last disabled at (189580674):
[<ffff8000813b78a4>] el1_interrupt+0x2c/0x70
[550195.393779][    C2] softirqs last  enabled at (189580644):
[<ffff80008005f7dc>] handle_softirqs+0x4b4/0x4cc
[550195.396778][    C2] softirqs last disabled at (189580639):
[<ffff800080010654>] __do_softirq+0x18/0x20
[550195.406109][    C2] CPU: 2 UID: 0 PID: 139 Comm: lock_torture_wr
Not tainted 6.12.0-00050-gf6c6e0ea8a1d-dirty #133
[550195.406661][    C2] Hardware name: linux,dummy-virt (DT)
[550195.407172][    C2] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO
-DIT -SSBS BTYPE=--)
[550195.407396][    C2] pc : queued_spin_lock_slowpath+0x270/0x528
[550195.408270][    C2] lr : queued_spin_lock_slowpath+0x3a8/0x528
[550195.408358][    C2] sp : ffff800084d83d20
[550195.408494][    C2] x29: ffff800084d83d20 x28: 28f5c28f5c28f5c3
x27: 0000000000000000
[550195.408985][    C2] x26: ffff800081ef3000 x25: 0000000000000000
x24: 0000000000000002
[550195.409158][    C2] x23: ffff800083532000 x22: ffff800082b55000
x21: ffff00002c286740
[550195.409322][    C2] x20: ffff800081ef3000 x19: ffff80008355b348
x18: ffff800081ce6e48
[550195.409522][    C2] x17: ffff7fffa954e000 x16: 000000006c0014e7
x15: ffff8000835321f0
[550195.409681][    C2] x14: 0001f45e95248cf0 x13: 0000000000000100
x12: 0000000000000000
[550195.409875][    C2] x11: 00000000002dc6c0 x10: 0000000000000000 x9
: ffff7fffa9731000
[550195.410102][    C2] x8 : ffff800081ef3fb8 x7 : 0000000000000000 x6
: 00000000000c0000
[550195.410262][    C2] x5 : ffff8000835365e8 x4 : 0000000000000000 x3
: 0000000000000000
[550195.410451][    C2] x2 : 0000000000000000 x1 : ffff00002c286740 x0
: ffff00002c286748
[550195.410849][    C2] Call trace:
[550195.410995][    C2]  queued_spin_lock_slowpath+0x270/0x528
[550195.411265][    C2]  do_raw_spin_lock+0xac/0x11c
[550195.421183][    C2]  _raw_spin_lock+0x74/0x90
[550195.421264][    C2]  torture_spin_lock_write_lock+0x1c/0x28
[550195.421333][    C2]  lock_torture_writer+0x104/0x2f4
[550195.421397][    C2]  kthread+0x128/0x134
[550195.421600][    C2]  ret_from_fork+0x10/0x20
[550195.422007][    C1] watchdog: BUG: soft lockup - CPU#1 stuck for
32s! [lock_torture_wr:144]
[550195.422780][    C2] Kernel panic - not syncing: softlockup: hung tasks
[550195.423616][    C2] CPU: 2 UID: 0 PID: 139 Comm: lock_torture_wr
Tainted: G             L     6.12.0-00050-gf6c6e0ea8a1d-dirty #133
[550195.423813][    C2] Tainted: [L]=SOFTLOCKUP
[550195.423929][    C2] Hardware name: linux,dummy-virt (DT)
[550195.424123][    C2] Call trace:
[550195.424174][    C2]  dump_backtrace.part.8+0x8c/0xe8
[550195.424232][    C2]  show_stack+0x2c/0x38
[550195.424256][    C2]  dump_stack_lvl+0x6c/0xcc
[550195.424999][    C2]  dump_stack+0x14/0x1c
[550195.425050][    C2]  panic+0x150/0x428
[550195.425121][    C2]  watchdog_timer_fn+0x258/0x2e8
[550195.426192][    C2]  __hrtimer_run_queues+0x3bc/0x4a0
[550195.426651][    C2]  hrtimer_interrupt+0xd8/0x248
[550195.426680][    C2]  arch_timer_handler_virt+0x38/0x58
[550195.427608][    C2]  handle_percpu_devid_irq+0xac/0x2cc
[550195.427676][    C2]  generic_handle_domain_irq+0x40/0x58
[550195.427743][    C2]  gic_handle_irq+0x64/0x124
[550195.427771][    C2]  call_on_irq_stack+0x24/0x34
[550195.427832][    C2]  do_interrupt_handler+0x90/0x94
[550195.427865][    C2]  el1_interrupt+0x3c/0x70
[550195.427899][    C2]  el1h_64_irq_handler+0x14/0x1c
[550195.427960][    C2]  el1h_64_irq+0x64/0x68
[550195.427989][    C2]  queued_spin_lock_slowpath+0x270/0x528
[550195.428021][    C2]  do_raw_spin_lock+0xac/0x11c
[550195.428051][    C2]  _raw_spin_lock+0x74/0x90
[550195.428082][    C2]  torture_spin_lock_write_lock+0x1c/0x28
[550195.428115][    C2]  lock_torture_writer+0x104/0x2f4
[550195.428147][    C2]  kthread+0x128/0x134
[550195.428182][    C2]  ret_from_fork+0x10/0x20
[550195.428564][    C2] SMP: stopping secondary CPUs
[550195.448256][    C2] Kernel Offset: disabled
[550195.450640][    C2] CPU features: 0x00,00000019,40203020,560042af
[550195.451534][    C2] Memory Limit: none
[550195.452853][    C2] ---[ end Kernel panic - not syncing:

softlockup: hung tasks ]---

At the time of the crash, all CPU call stacks were identical. From the call stack at level #15, it is evident that all threads are waiting for the torture_spinlock lock in kernel/locking/locktorture.c.

-000|cpu_park_loop(inline)
-000|local_cpu_stop(cpu = 0)
-001|panic_smp_self_stop()
-002|panic(:fmt = 0xFFFF800081CFC838)
-003|watchdog_timer_fn(:hrtimer = 0xFFFF00002BEAEE20)
-004|__run_hrtimer(inline)
-004|__hrtimer_run_queues(:cpu_base = 0xFFFF00002BEAE0C0, :now =
550186225371571, :flags = 192, :active_mask = 15)
-005|hrtimer_interrupt(:dev = 0xFFFF00002BEC4A80)
-006|timer_handler(inline)
-006|arch_timer_handler_virt(:irq = 11, dev_id = 0xFFFF00002BEC4A80)
-007|handle_percpu_devid_irq(desc = 0xFFFF000004C2C400)
-008|generic_handle_irq_desc(inline)
-008|handle_irq_desc(inline)
-008|generic_handle_domain_irq(domain = 0xFFFF000004E50000, :hwirq = 27)
-009|__gic_handle_irq(inline)
-009|__gic_handle_irq_from_irqson(inline)
-009|gic_handle_irq(regs = ?)
-010|call_on_irq_stack(asm)
-011|do_interrupt_handler(:regs = 0xFFFF800084D4BBD0, :handler =
0xFFFF80008001020C)
-012|__el1_irq(inline)
-012|el1_interrupt(regs = 0xFFFF800084D4BBD0, handler = 0xFFFF80008001020C)
-013|el1h_64_irq_handler(regs = ?)
-014|el1h_64_irq(asm)
-015|trace_contention_begin(inline)
-015|queued_spin_lock_slowpath(:lock = 0xFFFF80008355B348, val = ?)
-016|queued_spin_lock(inline)
-016|do_raw_spin_lock(:lock = 0xFFFF80008355B348)
-017|__raw_spin_lock(inline)
-017|_raw_spin_lock(:lock = 0xFFFF80008355B348)
-018|spin_lock(inline)
-018|torture_spin_lock_write_lock(:tid = 0)
-019|lock_torture_writer(:arg = 0xFFFF0000060A4800)
-020|kthread(_create = 0xFFFF000006074500)
-021|ret_from_fork(asm)
-022|NSX:0x0::0xFFFFFFFFFF5F8000(asm)
 ---|end of frame

crash_arm64> sym torture_spinlock
ffff80008355b348 (d) torture_spinlock
crash_arm64>

However, based on the crash site, it appears that no thread is currently holding the torture_spinlock lock, yet other threads on different CPUs are still waiting for this lock, which is confusing to me.

crash_arm64> struct -x raw_spinlock 0xFFFF80008355B348 struct raw_spinlock {

  raw_lock = {
    {
      val = {
        counter = 0x180000
      },
      {
        locked = 0x0,
        pending = 0x0
      },
      {
        locked_pending = 0x0,
        tail = 0x18
      }
    }
  },
  magic = 0xdead4ead,
  owner_cpu = 0xffffffff,
  owner = 0xffffffffffffffff,
  dep_map = {
    key = 0xffff80008355b360 <torture_spinlock+24>,
    class_cache = {0xffff8000841d9e58 <lock_classes+127248>, 0x0},
    name = 0xffff800081cebe90 "torture_spinlock",
    wait_type_outer = 0x0,
    wait_type_inner = 0x3,
    lock_type = 0x0,
    cpu = 0x5,
    ip = 0xffff8000800f30b8

}
}
crash_arm64>

Although I haven't yet pinpointed the root cause of this issue, I would like to confirm whether the problem I'm encountering is the same as the one you described.

If possible, could you share the dmesg log and call stack information of the crash you encountered?

Thanks.
hupu

#16 lore raw
Quoted 100 lines · hupu

Hi John

Quoted 5 lines

Unfortunately I've also been tripping over a crash with the full series (not the portion I'm currently submitting upstream) that often takes 30+ hours to reproduce and extra debugging logic seems to make it even harder to hit.

I encountered a strange crash while running the code from the proxy-exec-v17-WIP branch for over 150+ hours. I'm currently unsure if this issue is the same as the one you mentioned.

Here is the dmesg log at the time of the crash:

[550195.352234][    C2] watchdog: BUG: soft lockup - CPU#2 stuck for
32s! [lock_torture_wr:139]
[550195.374670][    C2] Modules linked in:
[550195.375936][    C2] irq event stamp: 189580674
[550195.376275][    C2] hardirqs last  enabled at (189580673):
[<ffff8000813b7840>] exit_to_kernel_mode.isra.6+0xf8/0x130
[550195.393675][    C2] hardirqs last disabled at (189580674):
[<ffff8000813b78a4>] el1_interrupt+0x2c/0x70
[550195.393779][    C2] softirqs last  enabled at (189580644):
[<ffff80008005f7dc>] handle_softirqs+0x4b4/0x4cc
[550195.396778][    C2] softirqs last disabled at (189580639):
[<ffff800080010654>] __do_softirq+0x18/0x20
[550195.406109][    C2] CPU: 2 UID: 0 PID: 139 Comm: lock_torture_wr
Not tainted 6.12.0-00050-gf6c6e0ea8a1d-dirty #133
[550195.406661][    C2] Hardware name: linux,dummy-virt (DT)
[550195.407172][    C2] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO
-DIT -SSBS BTYPE=--)
[550195.407396][    C2] pc : queued_spin_lock_slowpath+0x270/0x528
[550195.408270][    C2] lr : queued_spin_lock_slowpath+0x3a8/0x528
[550195.408358][    C2] sp : ffff800084d83d20
[550195.408494][    C2] x29: ffff800084d83d20 x28: 28f5c28f5c28f5c3
x27: 0000000000000000
[550195.408985][    C2] x26: ffff800081ef3000 x25: 0000000000000000
x24: 0000000000000002
[550195.409158][    C2] x23: ffff800083532000 x22: ffff800082b55000
x21: ffff00002c286740
[550195.409322][    C2] x20: ffff800081ef3000 x19: ffff80008355b348
x18: ffff800081ce6e48
[550195.409522][    C2] x17: ffff7fffa954e000 x16: 000000006c0014e7
x15: ffff8000835321f0
[550195.409681][    C2] x14: 0001f45e95248cf0 x13: 0000000000000100
x12: 0000000000000000
[550195.409875][    C2] x11: 00000000002dc6c0 x10: 0000000000000000 x9
: ffff7fffa9731000
[550195.410102][    C2] x8 : ffff800081ef3fb8 x7 : 0000000000000000 x6
: 00000000000c0000
[550195.410262][    C2] x5 : ffff8000835365e8 x4 : 0000000000000000 x3
: 0000000000000000
[550195.410451][    C2] x2 : 0000000000000000 x1 : ffff00002c286740 x0
: ffff00002c286748
[550195.410849][    C2] Call trace:
[550195.410995][    C2]  queued_spin_lock_slowpath+0x270/0x528
[550195.411265][    C2]  do_raw_spin_lock+0xac/0x11c
[550195.421183][    C2]  _raw_spin_lock+0x74/0x90
[550195.421264][    C2]  torture_spin_lock_write_lock+0x1c/0x28
[550195.421333][    C2]  lock_torture_writer+0x104/0x2f4
[550195.421397][    C2]  kthread+0x128/0x134
[550195.421600][    C2]  ret_from_fork+0x10/0x20
[550195.422007][    C1] watchdog: BUG: soft lockup - CPU#1 stuck for
32s! [lock_torture_wr:144]
[550195.422780][    C2] Kernel panic - not syncing: softlockup: hung tasks
[550195.423616][    C2] CPU: 2 UID: 0 PID: 139 Comm: lock_torture_wr
Tainted: G             L     6.12.0-00050-gf6c6e0ea8a1d-dirty #133
[550195.423813][    C2] Tainted: [L]=SOFTLOCKUP
[550195.423929][    C2] Hardware name: linux,dummy-virt (DT)
[550195.424123][    C2] Call trace:
[550195.424174][    C2]  dump_backtrace.part.8+0x8c/0xe8
[550195.424232][    C2]  show_stack+0x2c/0x38
[550195.424256][    C2]  dump_stack_lvl+0x6c/0xcc
[550195.424999][    C2]  dump_stack+0x14/0x1c
[550195.425050][    C2]  panic+0x150/0x428
[550195.425121][    C2]  watchdog_timer_fn+0x258/0x2e8
[550195.426192][    C2]  __hrtimer_run_queues+0x3bc/0x4a0
[550195.426651][    C2]  hrtimer_interrupt+0xd8/0x248
[550195.426680][    C2]  arch_timer_handler_virt+0x38/0x58
[550195.427608][    C2]  handle_percpu_devid_irq+0xac/0x2cc
[550195.427676][    C2]  generic_handle_domain_irq+0x40/0x58
[550195.427743][    C2]  gic_handle_irq+0x64/0x124
[550195.427771][    C2]  call_on_irq_stack+0x24/0x34
[550195.427832][    C2]  do_interrupt_handler+0x90/0x94
[550195.427865][    C2]  el1_interrupt+0x3c/0x70
[550195.427899][    C2]  el1h_64_irq_handler+0x14/0x1c
[550195.427960][    C2]  el1h_64_irq+0x64/0x68
[550195.427989][    C2]  queued_spin_lock_slowpath+0x270/0x528
[550195.428021][    C2]  do_raw_spin_lock+0xac/0x11c
[550195.428051][    C2]  _raw_spin_lock+0x74/0x90
[550195.428082][    C2]  torture_spin_lock_write_lock+0x1c/0x28
[550195.428115][    C2]  lock_torture_writer+0x104/0x2f4
[550195.428147][    C2]  kthread+0x128/0x134
[550195.428182][    C2]  ret_from_fork+0x10/0x20
[550195.428564][    C2] SMP: stopping secondary CPUs
[550195.448256][    C2] Kernel Offset: disabled
[550195.450640][    C2] CPU features: 0x00,00000019,40203020,560042af
[550195.451534][    C2] Memory Limit: none
[550195.452853][    C2] ---[ end Kernel panic - not syncing:
Quoted 2 lines

softlockup: hung tasks ]---

So, while I'm not sure if this is the same issue, I have found fixes for a few issues (but unfortunately not all) I was hitting with v17.

Not exactly what your seeing, but I was seeing some issues where due to a spurious wakeup we would end up with an owner that was woken but was still in the BO_WAKING state and hadn't been return migrated. This could cause us to get stuck looping in and out of find_proxy_task(), which could cause tasks to lock up in a similar fashion. So I've re-added the return migration for that case and it seems to be resolved.

I've not seen a case where a spinlock was unowned but everyone was spinning on it (particularly because my patch series doesn't really mess with spinlocks, assuming the runnable/spinning tasks get scheduled to run). The owner being stuck and not letting the lock go seems more likely, so that's curious.

I've got my current WIP set here:
https://github.com/johnstultz-work/linux-dev/commits/proxy-exec-WIP/
(Still need to write proper commit messages for many of the new changes)

The problem I see right now, which can sometimes take 30+ hours to trip on an 64core qemu system running cyclictest, my prio-inversion-demo test, and running the in-kernel test_ww_mutex tests in loops.

The backtrace will look something like:

[18071.849807] ------------[ cut here ]------------
[18071.849829]
[18071.849830] ======================================================
[18071.849831] WARNING: possible circular locking dependency detected
[18071.849832] 6.12.0-00060-g6b88c4fed2ba #1362 Not tainted
[18071.849834] ------------------------------------------------------
[18071.849835] kworker/u256:8/940 is trying to acquire lock:
[18071.849837] ffffffff82ffaf38 ((console_sem).lock){-...}-{2:2}
[18071.849887] BUG: unable to handle page fault for address: 0000004800153ad9
[18071.878733] #PF: supervisor read access in kernel mode
[18071.881809] #PF: error_code(0x0000) - not-present page
[18071.884840] PGD 0 P4D 0
[18071.886351] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
[18071.889305] CPU: 11 UID: 0 PID: 940 Comm: kworker/u256:8 Not
tainted 6.12.0-00060-g6b88c4fed2ba #1362
[18071.894553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[18071.900009] Workqueue: test-ww_mutex stress_one_work
[18071.902902] RIP: 0010:pick_next_task_fair+0x2c5/0x3f0
[18071.905860] Code: b5 90 00 00 00 45 39 f7 7e 5c 49 8b bc 24 a0 00
00 00 4c 89 e6 e8 4b a3 ff ff 4d 8b a4 24 98 00 00 00 4d 8b bc 24 a0
00 00 00 <49> 8b bd a0 00 00 00 49 39 ff 75 c0 4d 85 ff 74 bb 4c 89 ee
4c 89
[18071.916706] RSP: 0018:ffffc90002877c38 EFLAGS: 00010006
[18071.919729] RAX: ffff888105810000 RBX: ffff88810087a140 RCX: 0000000000000000
[18071.923810] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[18071.927894] RBP: ffff8881b9c6fa00 R08: 0000000000000000 R09: ffff888103fbc280
[18071.931983] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff810ec0c0
[18071.936051] R13: 0000004800153a39 R14: 0000000000000000 R15: 00441f0ffa1e0ff3
[18071.940143] FS:  0000000000000000(0000) GS:ffff8881b90c0000(0000)
knlGS:0000000000000000
[18071.944776] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18071.948085] CR2: 0000004800153ad9 CR3: 0000000112512004 CR4: 0000000000370ef0
[18071.952210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18071.956277] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[18071.960375] Call Trace:
[18071.961826]  <TASK>
[18071.963097]  ? __die+0x56/0x92
[18071.964919]  ? page_fault_oops+0xca/0x260
[18071.967246]  ? down_trylock+0x13/0x40
[18071.969400]  ? _printk+0x6c/0x87
[18071.971246]  ? __down_trylock_console_sem+0x23/0x90
[18071.974005]  ? exc_page_fault+0x525/0x8e0
[18071.976276]  ? asm_exc_page_fault+0x26/0x30
[18071.978614]  ? put_prev_entity+0x70/0xb0
[18071.980861]  ? pick_next_task_fair+0x2c5/0x3f0
[18071.983409]  ? pick_next_task_fair+0x309/0x3f0
[18071.985977]  __schedule+0x22f/0x20c0
[18071.988041]  ? lock_release+0x16d/0x2e0
[18071.990248]  schedule+0x3d/0x150
[18071.992197]  schedule_hrtimeout_range_clock+0x110/0x160
[18071.995155]  ? __pfx_hrtimer_wakeup+0x10/0x10
[18071.997679]  ? usleep_range_state+0x59/0xc0
[18072.000063]  usleep_range_state+0x87/0xc0
[18072.002341]  stress_one_work+0x41/0x90
[18072.004465]  process_one_work+0x204/0x5a0
[18072.006793]  ? lock_is_held_type+0xca/0x130
[18072.009195]  worker_thread+0x18e/0x350
[18072.011328]  ? __pfx_worker_thread+0x10/0x10
[18072.013773]  kthread+0xee/0x120
[18072.015593]  ? __pfx_kthread+0x10/0x10
[18072.017805]  ret_from_fork+0x31/0x50
[18072.019803]  ? __pfx_kthread+0x10/0x10
[18072.021948]  ret_from_fork_asm+0x1a/0x30
[18072.024214]  </TASK>

Which is due to tripping the:
SCHED_WARN_ON(cfs_rq->curr != prev);
check in put_prev_entity().

And then hitting some sort of memory trouble as the lockdep warning is being printed.

I suspect the issue is that the cfs_rq->curr (which should match the rq->donor set from the previous pick_next_task()) ends up as a dangling reference when we migrate or otherwise re-do the pick_again loop.

Unfortunately as I add more trace_printks to debug the issue, it becomes even more rare to trip, or the log somehow ends up incomplete (missing final lines I'd expect given the backtrace in the splat), so it has been slow progress. :(

Anyway, let me know if you run into anything similar or have ideas. I'll be continuing to try to trap the behavior better, running over this weekend.

thanks
-john