Caused by: sched/deadline: Move CPU frequency selection triggering points
Steven Rostedt
reviewerWhen looking to test SCHED_DEADLINE, I triggered a lockup. The lockup appears to be caused by WARN_ON() done inside the scheduling path, and I'm guessing it tried to grab the rq lock and caused a deadlock (all I would get would be the "--- cut here ---" line, and then nothing after that. But a bit of playing with the printks() I figured out it was happening at:
assert_clock_updated() {
SCHED_WARN_ON(rq->clock_update_flags < RQCF_ACT_SKIP);
}I bisected it down to commit e0367b126 ("sched/deadline: Move CPU frequency selection triggering points"). Reverting it indeed makes the deadlock go away.
I commented out the WARN_ON that was being triggered, and that let the system continue to run. I did the following change:
kernel/sched/sched.h +6 -1
@@ -914,7 +914,12 @@ static inline void assert_clock_updated(struct rq *rq)* The only reason for not seeing a clock update since the * last rq_pin_lock() is if we're currently skipping updates. */ -SCHED_WARN_ON (rq->clock_update_flags <RQCF_ACT_SKIP ); +if (rq->clock_update_flags <RQCF_ACT_SKIP ) +trace_printk ("WARN_ON: [%d] rq->clock_update_flags (%d) < %d)\n" , + rq->cpu, rq->clock_update_flags,RQCF_ACT_SKIP ); +else +trace_printk ("GOOD: [%d] rq->clock_update_flags (%d) >= %d)\n" , + rq->cpu, rq->clock_update_flags,RQCF_ACT_SKIP ); }
To see what was happening. I also, added trace_printk()s to all the updates to clock_update_flags, and ran my test again.
Here's what I got:
deadline_test-1393 [002] 162.127132: bprint: push_dl_task.part.40: WARN_ON: [1] rq->clock_update_flags (0) < 2) deadline_test-1393 [002] 162.127133: bprint: update_rq_clock: [1] clock was 0 now 4 deadline_test-1393 [002] 162.127134: bprint: rq_clock: GOOD: [1] rq->clock_update_flags (4) >= 2)
It appears that we hit this in this path:
push_dl_task {
add_running_bw() {
__add_running_bw() {
cpufreq_update_util() {
data->func(data, rq_clock(rq), flags);
rq_clock() {
assert_clock_updated()And here the clock isn't updated and we get the splat.
Reverting the stated patch works because it added the call to cpufreq_update_util() that does the rq_clock() at an inappropriate time, which causes the splat.
I'm not sure what the right answer to this is. Reverting obviously works, but I'm also guessing a proper placement of update_rq_clock() may also work. I just don't know where that placement is, as I don't understand the rq_clock() updates enough.
Help?
-- Steve
Juri Lelli
maintainerHi Steve,
Quoted 58 lines · Steven Rostedt
When looking to test
SCHED_DEADLINE, I triggered a lockup. The lockup appears to be caused byWARN_ON()done inside the scheduling path, and I'm guessing it tried to grab the rq lock and caused a deadlock (all I would get would be the "--- cut here ---" line, and then nothing after that. But a bit of playing with theprintks()I figured out it was happening at:assert_clock_updated() { SCHED_WARN_ON(rq->clock_update_flags < RQCF_ACT_SKIP); }I bisected it down to commit
e0367b126("sched/deadline: Move CPU frequency selection triggering points"). Reverting it indeed makes the deadlock go away.I commented out the
WARN_ONthat was being triggered, and that let the system continue to run. I did the following change:kernel/sched/sched.h +6
-1@@ -914,7 +914,12 @@ static inline void assert_clock_updated(struct rq *rq)* The only reason for not seeing a clock update since the * last rq_pin_lock() is if we're currently skipping updates. */ -SCHED_WARN_ON (rq->clock_update_flags <RQCF_ACT_SKIP ); +if (rq->clock_update_flags <RQCF_ACT_SKIP ) +trace_printk ("WARN_ON: [%d] rq->clock_update_flags (%d) < %d)\n" , + rq->cpu, rq->clock_update_flags,RQCF_ACT_SKIP ); +else +trace_printk ("GOOD: [%d] rq->clock_update_flags (%d) >= %d)\n" , + rq->cpu, rq->clock_update_flags,RQCF_ACT_SKIP ); }To see what was happening. I also, added trace_printk()s to all the updates to
clock_update_flags, and ran my test again.Here's what I got:
deadline_test-1393 [002] 162.127132: bprint: push_dl_task.part.40: WARN_ON: [1] rq->clock_update_flags (0) < 2) deadline_test-1393 [002] 162.127133: bprint: update_rq_clock: [1] clock was 0 now 4 deadline_test-1393 [002] 162.127134: bprint: rq_clock: GOOD: [1] rq->clock_update_flags (4) >= 2)It appears that we hit this in this path:
push_dl_task { add_running_bw() { __add_running_bw() { cpufreq_update_util() { data->func(data, rq_clock(rq), flags); rq_clock() { assert_clock_updated()And here the clock isn't updated and we get the splat.
Reverting the stated patch works because it added the call to
cpufreq_update_util()that does therq_clock()at an inappropriate time, which causes the splat.
Quoted 6 lines
I'm not sure what the right answer to this is. Reverting obviously works, but I'm also guessing a proper placement of
update_rq_clock()may also work. I just don't know where that placement is, as I don't understand therq_clock()updates enough.Help?
This got into tip quite recently
ecda2b66e263 ("sched/deadline: Fix missing clock update")
could you please double check that you have that in your stack?
Thanks,
- Juri
Steven Rostedt
reviewerQuoted 6 lines · On Tue, 3 Jul 2018 17:07:11 +0200 Juri Lelli <[email protected]> wrote:
This got into tip quite recently
ecda2b66e263("sched/deadline: Fix missing clock update")could you please double check that you have that in your stack?
Unfortunately, I just tested it out, and it still locks up with this patch.
Here's more info: I run with my deadline_test with the following options:
./deadline_test -c 1,2-3 -p 40 -t 9
from https://github.com/rostedt/sched_deadline.git
My box is 1 socket 4 cores and 2 threads each (8 logical CPUS)
Attached is my config.
-- Steve
Steven Rostedt
reviewerQuoted 5 lines · On Tue, 3 Jul 2018 11:31:20 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 2 lines · On Tue, 3 Jul 2018 17:07:11 +0200 Juri Lelli <[email protected]> wrote:
This got into tip quite recently
Quoted 8 lines
Quoted 4 lines
ecda2b66e263("sched/deadline: Fix missing clock update")could you please double check that you have that in your stack?
Unfortunately, I just tested it out, and it still locks up with this patch.
It appears that the below patch fixes the issue for me.
Is this a correct usage?
-- Steve
kernel/sched/deadline.c +2 -0
@@ -2085,6 +2085,8 @@ static int push_dl_task(struct rq *rq) goto retry; } +update_rq_clock (later_rq); +deactivate_task (rq, next_task,0 );sub_running_bw (&next_task->dl, &rq->dl);sub_rq_bw (&next_task->dl, &rq->dl);