Skip to contentSkip to conversation

lockdep-related warning in kernel/sched/deadline.c::find_lock_later_rq()

PATCH Luca Abeni 4 participants 6 messages Oct 21, 2015 – Oct 23, 2015 Scheduler

lore
Share & tools
lore

Luca Abeni

author
#0 lore raw

Hi all,

after fixing task migrations for SCHED_DEADLINE, I started to see some lockdep-related warnings that look like this:

[  794.428081] WARNING: CPU: 1 PID: 0 at /home/luca/Src/GRUB/linux-reclaiming/kernel/locking/lockdep.c:3407 lock_release+0x3f4/0x440()
[  794.428439] releasing a pinned lock
[  794.428439] Modules linked in:
[  794.428439] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.3.0-rc5+ #32
[  794.428439] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Bochs 01/01/2011
[  794.428439]  ffffffff81c946d0 ffff880007b03cc8 ffffffff8135abd2 ffff880007b03d10
[  794.428439]  ffff880007b03d00 ffffffff810546f1 ffff880007b159d8 ffffffff81096663
[  794.428439]  ffff88000754c558 0000000000000001 ffff88000754bd80 ffff880007b03d60
[  794.428439] Call Trace:
[  794.428439]  <IRQ>  [<ffffffff8135abd2>] dump_stack+0x4b/0x69
[  794.428439]  [<ffffffff810546f1>] warn_slowpath_common+0x81/0xc0
[  794.428439]  [<ffffffff81096663>] ? find_lock_later_rq+0x103/0x180
[  794.428439]  [<ffffffff81054777>] warn_slowpath_fmt+0x47/0x50
[  794.428439]  [<ffffffff8109feb4>] lock_release+0x3f4/0x440
[  794.428439]  [<ffffffff8194215a>] _raw_spin_unlock+0x1a/0x30
[  794.428439]  [<ffffffff81096663>] find_lock_later_rq+0x103/0x180
[  794.428439]  [<ffffffff81096741>] push_dl_task.part.33+0x61/0x190
[  794.428439]  [<ffffffff81096a94>] dl_task_timer+0x194/0x2e0
[  794.428439]  [<ffffffff81096900>] ? task_woken_dl+0x60/0x60
[  794.428439]  [<ffffffff810c3a87>] __hrtimer_run_queues+0x107/0x470
[  794.428439]  [<ffffffff810c3f82>] ? hrtimer_interrupt+0x82/0x1b0
[  794.428439]  [<ffffffff810c3fa6>] hrtimer_interrupt+0xa6/0x1b0
[  794.428439]  [<ffffffff81039570>] local_apic_timer_interrupt+0x30/0x60
[  794.428439]  [<ffffffff81039ee8>] smp_apic_timer_interrupt+0x38/0x50
[  794.428439]  [<ffffffff819436c4>] apic_timer_interrupt+0x84/0x90
[  794.428439]  <EOI>  [<ffffffff8100ea3b>] ? default_idle+0x1b/0x140
[  794.428439]  [<ffffffff8100ea39>] ? default_idle+0x19/0x140
[  794.428439]  [<ffffffff8100f25a>] arch_cpu_idle+0xa/0x10
[  794.428439]  [<ffffffff81097e45>] default_idle_call+0x25/0x40
[  794.428439]  [<ffffffff810981d0>] cpu_startup_entry+0x310/0x390
[  794.428439]  [<ffffffff810380c3>] start_secondary+0xf3/0x100
[  794.428439] ---[ end trace 1d35bf076299f814 ]---

(this happended on KVM, but I can trigger similar warnings on real hardware too).

This was not happening before my last patch, because push_dl_task() never migrated any task, and always returned before calling find_lock_later_rq()...

Now, if I understand correctly the issue is that dl_task_timer() does:

	rq = task_rq_lock(p, &flags);
[...]
	if (has_pushable_dl_tasks(rq))
		push_dl_task(rq);

with task_rq_lock() that pins rq->lock and push_tl_task() that invokes find_lock_later_rq() that unlocks rq->lock() while it is pinned.

I am not sure about how to fix this issue: as a first try, I did

kernel/sched/deadline.c +4 -1
index 142df26..5b1ba95 100644
@@ -668,8 +668,11 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
 	 * Queueing this task back might have overloaded rq, check if we need
 	 * to kick someone away.
 	 */
-	if (has_pushable_dl_tasks(rq))
+	if (has_pushable_dl_tasks(rq)) {
+		lockdep_unpin_lock(&rq->lock);
 		push_dl_task(rq);
+		lockdep_pin_lock(&rq->lock);
+	}
 #endif
 
 unlock:

This removes the warning, but I am not sure if it is the correct fix (is it valid to unpin rq->lock, here?).

If someone can confirm that this is the correct approach, I'll test the patch a little bit more and then I'll send a properly signed-off patch; otherwise, if someone can suggest the correct approach I'll try it.

Thanks,
Luca

#1 lore raw
Quoted 70 lines · Luca Abeni

Hi all,

after fixing task migrations for SCHED_DEADLINE, I started to see some lockdep-related warnings that look like this:

[  794.428081] WARNING: CPU: 1 PID: 0 at /home/luca/Src/GRUB/linux-reclaiming/kernel/locking/lockdep.c:3407 lock_release+0x3f4/0x440()
[  794.428439] releasing a pinned lock
[  794.428439] Modules linked in:
[  794.428439] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.3.0-rc5+ #32
[  794.428439] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Bochs 01/01/2011
[  794.428439]  ffffffff81c946d0 ffff880007b03cc8 ffffffff8135abd2 ffff880007b03d10
[  794.428439]  ffff880007b03d00 ffffffff810546f1 ffff880007b159d8 ffffffff81096663
[  794.428439]  ffff88000754c558 0000000000000001 ffff88000754bd80 ffff880007b03d60
[  794.428439] Call Trace:
[  794.428439]  <IRQ>  [<ffffffff8135abd2>] dump_stack+0x4b/0x69
[  794.428439]  [<ffffffff810546f1>] warn_slowpath_common+0x81/0xc0
[  794.428439]  [<ffffffff81096663>] ? find_lock_later_rq+0x103/0x180
[  794.428439]  [<ffffffff81054777>] warn_slowpath_fmt+0x47/0x50
[  794.428439]  [<ffffffff8109feb4>] lock_release+0x3f4/0x440
[  794.428439]  [<ffffffff8194215a>] _raw_spin_unlock+0x1a/0x30
[  794.428439]  [<ffffffff81096663>] find_lock_later_rq+0x103/0x180
[  794.428439]  [<ffffffff81096741>] push_dl_task.part.33+0x61/0x190
[  794.428439]  [<ffffffff81096a94>] dl_task_timer+0x194/0x2e0
[  794.428439]  [<ffffffff81096900>] ? task_woken_dl+0x60/0x60
[  794.428439]  [<ffffffff810c3a87>] __hrtimer_run_queues+0x107/0x470
[  794.428439]  [<ffffffff810c3f82>] ? hrtimer_interrupt+0x82/0x1b0
[  794.428439]  [<ffffffff810c3fa6>] hrtimer_interrupt+0xa6/0x1b0
[  794.428439]  [<ffffffff81039570>] local_apic_timer_interrupt+0x30/0x60
[  794.428439]  [<ffffffff81039ee8>] smp_apic_timer_interrupt+0x38/0x50
[  794.428439]  [<ffffffff819436c4>] apic_timer_interrupt+0x84/0x90
[  794.428439]  <EOI>  [<ffffffff8100ea3b>] ? default_idle+0x1b/0x140
[  794.428439]  [<ffffffff8100ea39>] ? default_idle+0x19/0x140
[  794.428439]  [<ffffffff8100f25a>] arch_cpu_idle+0xa/0x10
[  794.428439]  [<ffffffff81097e45>] default_idle_call+0x25/0x40
[  794.428439]  [<ffffffff810981d0>] cpu_startup_entry+0x310/0x390
[  794.428439]  [<ffffffff810380c3>] start_secondary+0xf3/0x100
[  794.428439] ---[ end trace 1d35bf076299f814 ]---

(this happended on KVM, but I can trigger similar warnings on real hardware too).

This was not happening before my last patch, because push_dl_task() never migrated any task, and always returned before calling find_lock_later_rq()...

Now, if I understand correctly the issue is that dl_task_timer() does:

    rq = task_rq_lock(p, &flags);
[...]
    if (has_pushable_dl_tasks(rq))
        push_dl_task(rq);

with task_rq_lock() that pins rq->lock and push_tl_task() that invokes find_lock_later_rq() that unlocks rq->lock() while it is pinned.

I am not sure about how to fix this issue: as a first try, I did

kernel/sched/deadline.c +0 -0
index 142df26..5b1ba95 100644
@@ -668,8 +668,11 @@ static enum hrtimer_restart dl_task_timer(struct 
 hrtimer *timer)
      * Queueing this task back might have overloaded rq, check if we 

need
* to kick someone away.
*/

kernel/sched/deadline.c +4 -1
-    if (has_pushable_dl_tasks(rq))
+    if (has_pushable_dl_tasks(rq)) {
+        lockdep_unpin_lock(&rq->lock);
         push_dl_task(rq);
+        lockdep_pin_lock(&rq->lock);
+    }
 #endif
 
 unlock:
Quoted 6 lines

This removes the warning, but I am not sure if it is the correct fix (is it valid to unpin rq->lock, here?).

If someone can confirm that this is the correct approach, I'll test the patch a little bit more and then I'll send a properly signed-off patch; otherwise, if someone can suggest the correct approach I'll try it.

wake_up_new_task()
-> __task_rq_lock()
-> task_woken()

    -> push_dl/rt_tasks()
      -> push_dl/rt_task()

I think you also should consider the lockdep pin_lock in this path.

Regards,
Wanpeng Li

author
#2 lore raw

Hi,

On 10/22/2015 07:35 AM, Wanpeng Li wrote:
[...]

Quoted 37 lines
Quoted 27 lines

Now, if I understand correctly the issue is that dl_task_timer() does:

    rq = task_rq_lock(p, &flags);
[...]
    if (has_pushable_dl_tasks(rq))
        push_dl_task(rq);

with task_rq_lock() that pins rq->lock and push_tl_task() that invokes find_lock_later_rq() that unlocks rq->lock() while it is pinned.

I am not sure about how to fix this issue: as a first try, I did

kernel/sched/deadline.c +4 -1
index 142df26..5b1ba95 100644
@@ -668,8 +668,11 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
      * Queueing this task back might have overloaded rq, check if we need
      * to kick someone away.
      */
-    if (has_pushable_dl_tasks(rq))
+    if (has_pushable_dl_tasks(rq)) {
+        lockdep_unpin_lock(&rq->lock);
         push_dl_task(rq);
+        lockdep_pin_lock(&rq->lock);
+    }
 #endif
 
 unlock:
Quoted 6 lines

This removes the warning, but I am not sure if it is the correct fix (is it valid to unpin rq->lock, here?).

If someone can confirm that this is the correct approach, I'll test the patch a little bit more and then I'll send a properly signed-off patch; otherwise, if someone can suggest the correct approach I'll try it.

wake_up_new_task()
-> __task_rq_lock()
-> task_woken()

Quoted 4 lines
     -> push_dl/rt_tasks()
       -> push_dl/rt_task()

I think you also should consider the lockdep pin_lock in this path.

Well, I never triggered this warning for the task_woken() path, but now I see it... Thanks!

If someone can confirm that unpinning before calling push/pull and pinning again after the call is the correct thing to do, I'll send a proper patch taking into account all the paths... But for the moment I am still not sure if unpinning the lock here is ok.

Thanks,
Luca

maintainer
#3 lore raw
Quoted 16 lines · Luca Abeni
Quoted 16 lines
Quoted 16 lines
kernel/sched/deadline.c +4 -1
index 142df26..5b1ba95 100644
@@ -668,8 +668,11 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
     * Queueing this task back might have overloaded rq, check if we need
     * to kick someone away.
     */
-    if (has_pushable_dl_tasks(rq))
+    if (has_pushable_dl_tasks(rq)) {
+        lockdep_unpin_lock(&rq->lock);
        push_dl_task(rq);
+        lockdep_pin_lock(&rq->lock);
+    }
 #endif
 
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

unlock:

This removes the warning, but I am not sure if it is the correct fix (is it valid to unpin rq->lock, here?).

If someone can confirm that this is the correct approach, I'll test the patch a little bit more and then I'll send a properly signed-off patch; otherwise, if someone can suggest the correct approach I'll try it.

Yes, its fine, although I would like a little comment with each unpin to explain _why_ its fine.

So here its fine because nothing relies on rq->lock being held after push_dl_task(), as the next thing we do is drop it anyway.

Quoted 7 lines
Quoted 7 lines

wake_up_new_task()
-> __task_rq_lock()
-> task_woken()

    -> push_dl/rt_tasks()
      -> push_dl/rt_task()

I think you also should consider the lockdep pin_lock in this path.

Durr, clearly I overlooked both these when I did that. Sorry about that.

So how about:

---
Subject: sched: Add missing lockdep_unpin annotations

Luca and Wanpeng reported two missing annotations that led to false lockdep complaints. Add the missing annotations.

Fixes cbce1a686700 ("sched,lockdep: Employ lock pinning")
---
 kernel/sched/core.c     | 9 ++++++++-
 kernel/sched/deadline.c | 9 ++++++++-
 2 files changed, 16 insertions(+), 2 deletions(-)
kernel/sched/core.c +8 -1
index 1764a0f2a75b..81a74b76346c 100644
@@ -2362,8 +2362,15 @@ void wake_up_new_task(struct task_struct *p)
 	trace_sched_wakeup_new(p);
 	check_preempt_curr(rq, p, WF_FORK);
 #ifdef CONFIG_SMP
-	if (p->sched_class->task_woken)
+	if (p->sched_class->task_woken) {
+		/*
+		 * Nothing relies on rq->lock after this, so its fine to
+		 * drop it.
+		 */
+		lockdep_unpin_lock(&rq->lock);
 		p->sched_class->task_woken(rq, p);
+		lockdep_pin_lock(&rq->lock);
+	}
 #endif
 	task_rq_unlock(rq, p, &flags);
 }
kernel/sched/deadline.c +8 -1
index fc8f01083527..cfdff233099b 100644
@@ -668,8 +668,15 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
 	 * Queueing this task back might have overloaded rq, check if we need
 	 * to kick someone away.
 	 */
-	if (has_pushable_dl_tasks(rq))
+	if (has_pushable_dl_tasks(rq)) {
+		/*
+		 * Nothing relies on rq->lock after this, so its safe to drop
+		 * rq->lock.
+		 */
+		lockdep_unpin_lock(&rq->lock);
 		push_dl_task(rq);
+		lockdep_pin_lock(&rq->lock);
+	}
 #endif
 
 unlock:
#4 lore raw

[tip:sched/urgent] sched/core: Add missing lockdep_unpin() annotations

sched/core: Add missing lockdep_unpin() annotations
Commit-ID:  0aaafaabfcba8aa991913cd3280a5dbf7f111a2a
Gitweb:     http://git.kernel.org/tip/0aaafaabfcba8aa991913cd3280a5dbf7f111a2a
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Fri, 23 Oct 2015 11:50:08 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 23 Oct 2015 12:02:10 +0200

sched/core: Add missing lockdep_unpin() annotations

Luca and Wanpeng reported two missing annotations that led to false lockdep complaints. Add the missing annotations.

Cc Juri Lelli <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Fixes cbce1a686700 ("sched,lockdep: Employ lock pinning")
---
 kernel/sched/core.c     | 9 ++++++++-
 kernel/sched/deadline.c | 9 ++++++++-
 2 files changed, 16 insertions(+), 2 deletions(-)
kernel/sched/core.c +8 -1
index 5bd7d60..bcd214e 100644
@@ -2366,8 +2366,15 @@ void wake_up_new_task(struct task_struct *p)
 	trace_sched_wakeup_new(p);
 	check_preempt_curr(rq, p, WF_FORK);
 #ifdef CONFIG_SMP
-	if (p->sched_class->task_woken)
+	if (p->sched_class->task_woken) {
+		/*
+		 * Nothing relies on rq->lock after this, so its fine to
+		 * drop it.
+		 */
+		lockdep_unpin_lock(&rq->lock);
 		p->sched_class->task_woken(rq, p);
+		lockdep_pin_lock(&rq->lock);
+	}
 #endif
 	task_rq_unlock(rq, p, &flags);
 }
kernel/sched/deadline.c +8 -1
index 142df26..8b0a15e 100644
@@ -668,8 +668,15 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
 	 * Queueing this task back might have overloaded rq, check if we need
 	 * to kick someone away.
 	 */
-	if (has_pushable_dl_tasks(rq))
+	if (has_pushable_dl_tasks(rq)) {
+		/*
+		 * Nothing relies on rq->lock after this, so its safe to drop
+		 * rq->lock.
+		 */
+		lockdep_unpin_lock(&rq->lock);
 		push_dl_task(rq);
+		lockdep_pin_lock(&rq->lock);
+	}
 #endif
 
 unlock:
author
#5 lore raw
In reply to Peter Zijlstra

Hi Peter,

On Fri, 23 Oct 2015 11:50:08 +0200
Peter Zijlstra <[email protected]> wrote:

[...]

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

This removes the warning, but I am not sure if it is the correct fix (is it valid to unpin rq->lock, here?).

If someone can confirm that this is the correct approach, I'll test the patch a little bit more and then I'll send a properly signed-off patch; otherwise, if someone can suggest the correct approach I'll try it.

Quoted 5 lines

Yes, its fine, although I would like a little comment with each unpin to explain _why_ its fine.

So here its fine because nothing relies on rq->lock being held after push_dl_task(), as the next thing we do is drop it anyway.

Thanks for explaining (now I see why it makes sense :), and thanks for the patch!
I am using it locally, and everything is working fine.

Thanks,
Luca

Quoted 75 lines
Quoted 8 lines
Quoted 8 lines

wake_up_new_task()
-> __task_rq_lock()
-> task_woken()

    -> push_dl/rt_tasks()
      -> push_dl/rt_task()

I think you also should consider the lockdep pin_lock in this path.

Durr, clearly I overlooked both these when I did that. Sorry about that.

So how about:

---
Subject: sched: Add missing lockdep_unpin annotations

Luca and Wanpeng reported two missing annotations that led to false lockdep complaints. Add the missing annotations.

Fixes cbce1a686700 ("sched,lockdep: Employ lock pinning")
---
 kernel/sched/core.c     | 9 ++++++++-
 kernel/sched/deadline.c | 9 ++++++++-
 2 files changed, 16 insertions(+), 2 deletions(-)
kernel/sched/core.c +3 -1
index 1764a0f2a75b..81a74b76346c 100644
@@ -2362,8 +2362,15 @@ void wake_up_new_task(struct task_struct *p)
 	trace_sched_wakeup_new(p);
 	check_preempt_curr(rq, p, WF_FORK);
 #ifdef CONFIG_SMP
-	if (p->sched_class->task_woken)
+	if (p->sched_class->task_woken) {
+		/*
+		 * Nothing relies on rq->lock after this, so its

fine to

kernel/sched/core.c +5 -0
+		 * drop it.
+		 */
+		lockdep_unpin_lock(&rq->lock);
 		p->sched_class->task_woken(rq, p);
+		lockdep_pin_lock(&rq->lock);
+	}
 #endif
 	task_rq_unlock(rq, p, &flags);
 }
kernel/sched/deadline.c +0 -0
index fc8f01083527..cfdff233099b 100644
@@ -668,8 +668,15 @@ static enum hrtimer_restart dl_task_timer(struct
 hrtimer *timer)
 	 * Queueing this task back might have overloaded rq, check

if we need

kernel/sched/deadline.c +3 -1
 	 * to kick someone away.
 	 */
-	if (has_pushable_dl_tasks(rq))
+	if (has_pushable_dl_tasks(rq)) {
+		/*
+		 * Nothing relies on rq->lock after this, so its

safe to drop

kernel/sched/deadline.c +5 -0
+		 * rq->lock.
+		 */
+		lockdep_unpin_lock(&rq->lock);
 		push_dl_task(rq);
+		lockdep_pin_lock(&rq->lock);
+	}
 #endif
 
 unlock: