Skip to contentSkip to conversation

Hang with fair cgroup scheduler (reproducer is attached.)

DISCUSSION Dmitry Adamushko 2 participants 6 messages Dec 16, 2007 – Dec 17, 2007 3 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Ingo,

what about the following patch instead?

maybe task_is_current() would be a better name though.

Steven,

I guess, there is some analogue of UNLOCKED_CTXSW on -rt
(to reduce contention for rq->lock).
So there can be a race schedule() vs. rt_mutex_setprio() or sched_setscheduler() for some paths that might explain crashes you have been observing?

I haven't analyzed this case for -rt, so I'm just throwing in the idea in case it can be useful.

---------------------------------------------------

From: Dmitry Adamushko <[email protected]>

sched: introduce task_current()

Some services (e.g. sched_setscheduler(), rt_mutex_setprio() and sched_move_task()) must handle a given task differently in case it's the 'rq->curr' task on its run-queue. The task_running() interface is not suitable for determining such tasks for platforms with one of the following options:

#define __ARCH_WANT_UNLOCKED_CTXSW
#define __ARCH_WANT_INTERRUPTS_ON_CTXSW

Due to the fact that it makes use of 'p->oncpu == 1' as a criterion but such a task is not necessarily 'rq->curr'.

The detailed explanation is available here:
https://lists.linux-foundation.org/pipermail/containers/2007-December/009262.html

---

kernel/sched.c +11 -6
index dc6fb24..15d088b 100644
@@ -619,10 +619,15 @@ EXPORT_SYMBOL_GPL(cpu_clock);
 # define finish_arch_switch(prev)	do { } while (0)
 #endif
 
+static inline int task_current(struct rq *rq, struct task_struct *p)
+{
+	return rq->curr == p;
+}
+
 #ifndef __ARCH_WANT_UNLOCKED_CTXSW
 static inline int task_running(struct rq *rq, struct task_struct *p)
 {
-	return rq->curr == p;
+	return task_current(rq, p);
 }
 
 static inline void prepare_lock_switch(struct rq *rq, struct task_struct *next)
@@ -651,7 +656,7 @@ static inline int task_running(struct rq *rq, struct task_struct *p)
 #ifdef CONFIG_SMP
 	return p->oncpu;
 #else
-	return rq->curr == p;
+	return task_current(rq, p);
 #endif
 }
 
@@ -3340,7 +3345,7 @@ unsigned long long task_sched_runtime(struct task_struct *p)
 
 	rq = task_rq_lock(p, &flags);
 	ns = p->se.sum_exec_runtime;
-	if (rq->curr == p) {
+	if (task_current(rq, p)) {
 		update_rq_clock(rq);
 		delta_exec = rq->clock - p->se.exec_start;
 		if ((s64)delta_exec > 0)
@@ -4033,7 +4038,7 @@ void rt_mutex_setprio(struct task_struct *p, int prio)
 
 	oldprio = p->prio;
 	on_rq = p->se.on_rq;
-	running = task_running(rq, p);
+	running = task_current(rq, p);
 	if (on_rq) {
 		dequeue_task(rq, p, 0);
 		if (running)
@@ -4334,7 +4339,7 @@ recheck:
 	}
 	update_rq_clock(rq);
 	on_rq = p->se.on_rq;
-	running = task_running(rq, p);
+	running = task_current(rq, p);
 	if (on_rq) {
 		deactivate_task(rq, p, 0);
 		if (running)
@@ -7360,7 +7365,7 @@ void sched_move_task(struct task_struct *tsk)
 
 	update_rq_clock(rq);
 
-	running = task_running(rq, tsk);
+	running = task_current(rq, tsk);
 	on_rq = tsk->se.on_rq;
 
 	if (on_rq) {

---

#1 lore raw
Quoted 8 lines · Dmitry Adamushko

Steven,

I guess, there is some analogue of UNLOCKED_CTXSW on -rt
(to reduce contention for rq->lock).
So there can be a race schedule() vs. rt_mutex_setprio() or sched_setscheduler() for some paths that might explain crashes you have been observing?

I haven't analyzed this case for -rt, so I'm just throwing in the idea in case it can be useful.

Dmitry,

Thanks! I've been watching this thread on the sidelines with keen interest. I ran hackbench with my logging over the weekend, and after some 30 hours of running, it finally crashed and gave me a dump of a lot of interactions in the put_prev_enitity and pick_next_task. I'll analyze this more on Monday.

-- Steve

#2 lore raw
In reply to Dmitry Adamushko
Quoted 8 lines · Dmitry Adamushko

Steven,

I guess, there is some analogue of UNLOCKED_CTXSW on -rt
(to reduce contention for rq->lock).
So there can be a race schedule() vs. rt_mutex_setprio() or sched_setscheduler() for some paths that might explain crashes you have been observing?

I haven't analyzed this case for -rt, so I'm just throwing in the idea in case it can be useful.

I haven't fully analyzed this either, but will look much deeper tomorrow. I wanted to show you this first just to see if you can easily spot what went wrong.

I used my logdev logging device
http://rostedt.homelinux.com/logdev

The patch that I used to add the logging is here:
http://rostedt.homelinux.com/rt-bug/debug-logdev.patch

To understand this. lfcnprint(...) is just like printk, but it will print output the following format:

[<timestamp>] cpu:<cpu#> (<current-comm>:<current-pid>) <function>:<line#> <printk-fmt>

The tags of lmark() is just

[<timestamp>] cpu:<cpu#> (<current-comm>:<current-pid>) <function>:<line#>

On context switches, we get

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

IN LOGDEV SWITCH <<<< cpu: <cpu#>

CPU=<cpu#> [<timestamp>] <prev>:<prev-pid>(<prios>:<task-state>) -->> <next>:<next-pid>(<prios>)

The full dmesg with logdump and error backtrace is here:
http://rostedt.homelinux.com/rt-bug/rt-bug.log

Here's a little snippet of where things went wrong.

[94359.652019] cpu:3 (hackbench:1658) pick_next_task_fair:1036 nr_running=1
[94359.652020] cpu:3 (hackbench:1658) pick_next_entity:625 se=ffff810009020800
[94359.652021] cpu:0 (hackbench:1473) put_prev_entity:631
[94359.652022] cpu:3 (hackbench:1658) pick_next_entity:625 se=ffff81003906b5a8
[94359.652022] cpu:2 (softirq-timer/2:32) put_prev_task_rt:283
[94359.652023] cpu:0 (hackbench:1473) put_prev_entity:631
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

IN LOGDEV SWITCH <<<< cpu:3

CPU=3 [94359.652023] hackbench:1658(120:120:120:D) -->> hackbench:1586(120:120:120)

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

IN LOGDEV SWITCH <<<< cpu:2

CPU=2 [94359.652024] softirq-timer/2:32(49:115:49:D) -->> softirq-rcu/2:39(49:115:49)

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

IN LOGDEV SWITCH <<<< cpu:0

CPU=0 [94359.652025] hackbench:1473(120:120:120:R) -->> hackbench:1591(49:120:120)

[94359.652029] cpu:3 (hackbench:1586) put_prev_entity:631
[94359.652030] cpu:2 (softirq-rcu/2:39) move_tasks:2472
[94359.652030] cpu:3 (hackbench:1586) put_prev_entity:631
[94359.652032] cpu:3 (hackbench:1586) pick_next_task_fair:1036 nr_running=1
[94359.652033] cpu:3 (hackbench:1586) pick_next_entity:625 se=ffff810009020800
[94359.652034] cpu:3 (hackbench:1586) pick_next_entity:625 se=ffff810014c7ab18
[94359.652034] cpu:2 (softirq-rcu/2:39) put_prev_task_rt:283
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

IN LOGDEV SWITCH <<<< cpu:3

CPU=3 [94359.652035] hackbench:1586(120:120:120:T) -->> hackbench:1623(120:120:120)

[94359.652036] cpu:2 (softirq-rcu/2:39) pick_next_task_fair:1036 nr_running=1
[94359.652038] cpu:2 (softirq-rcu/2:39) pick_next_entity:625 se=0000000000000000

I see that softirq-rcu on cpu 2 started doing a move_tasks, when it got to the state where nr_running returned 1 and the se from pick_next_entity was NULL.

This was the run on 2.6.24-rc5-rt1.

I'll look deeper into this on Monday, but if something jumps out at you, please let me know.

Thanks,

-- Steve

#4 lore raw
In reply to a message not in the archive
Quoted 22 lines · Steven Rostedt

Here's a little snippet of where things went wrong.

[94359.652019] cpu:3 (hackbench:1658) pick_next_task_fair:1036 nr_running=1
[94359.652020] cpu:3 (hackbench:1658) pick_next_entity:625 se=ffff810009020800
[94359.652021] cpu:0 (hackbench:1473) put_prev_entity:631
[94359.652022] cpu:3 (hackbench:1658) pick_next_entity:625 se=ffff81003906b5a8
[94359.652022] cpu:2 (softirq-timer/2:32) put_prev_task_rt:283
[94359.652023] cpu:0 (hackbench:1473) put_prev_entity:631
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

IN LOGDEV SWITCH <<<< cpu:3

CPU=3 [94359.652023] hackbench:1658(120:120:120:D) -->> hackbench:1586(120:120:120)

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

IN LOGDEV SWITCH <<<< cpu:2

CPU=2 [94359.652024] softirq-timer/2:32(49:115:49:D) -->> softirq-rcu/2:39(49:115:49)

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

IN LOGDEV SWITCH <<<< cpu:0

CPU=0 [94359.652025] hackbench:1473(120:120:120:R) -->> hackbench:1591(49:120:120)

[94359.652029] cpu:3 (hackbench:1586) put_prev_entity:631
[94359.652030] cpu:2 (softirq-rcu/2:39) move_tasks:2472
[94359.652030] cpu:3 (hackbench:1586) put_prev_entity:631
[94359.652032] cpu:3 (hackbench:1586) pick_next_task_fair:1036 nr_running=1
[94359.652033] cpu:3 (hackbench:1586) pick_next_entity:625 se=ffff810009020800
[94359.652034] cpu:3 (hackbench:1586) pick_next_entity:625 se=ffff810014c7ab18
[94359.652034] cpu:2 (softirq-rcu/2:39) put_prev_task_rt:283
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

IN LOGDEV SWITCH <<<< cpu:3

CPU=3 [94359.652035] hackbench:1586(120:120:120:T) -->> hackbench:1623(120:120:120)

[94359.652036] cpu:2 (softirq-rcu/2:39) pick_next_task_fair:1036 nr_running=1
[94359.652038] cpu:2 (softirq-rcu/2:39) pick_next_entity:625 se=0000000000000000

I see that softirq-rcu on cpu 2 started doing a move_tasks, when it got to the state where nr_running returned 1 and the se from pick_next_entity was NULL.

move_task() is likely to be run from schedule() --> idle_balance() , for our case it means that 'softirq-rcu' (which is a RT task) went to sleep and it was the last task on this CPU (rq->nr_running == 0 --> idle_balance() was triggered).

It may be related, maybe not. One 'abnormal' thing (at least, it occurs only once in this log. Should be checked wheather it happens when the system works fine) is that a few iterations before the oops happens we observe the following pattern:

CPU=2 [94359.651930] hackbench:1932(120:120:120:T) -->> hackbench:1591(120:120:120)

CPU=2 [94359.651980] hackbench:1591(49:120:120:T) -->> swapper:0(140:120:140)

swapper (idle) --> softirq-timer (RT)
softirq-timer (RT) --> softirq-rcu (RT)
softirq-rcu(RT) --> picks up se == 0 for SCHED_NORMAL upon scheduling out ---> OOPS

'hackbench' was of SCHED_NORMAL upon scheduling _in_, and it's of RT type (prio: 49 and schedule() --> put_prev_task_rt()) upon scheduling _out_.

Unless you run some modified version of 'hackbench', it doesn't chenge scheduling classes... so maybe a lifted prio is a consequence of the resource contention with some RT task ?

This 'hackbench' was the last SCHED_NORMAL task to run on this CPU... so however this NORMAL -> RT transition happened, it might leave a sched_fair's runqueue corrupted...

(Will try to look more when time allows).

Quoted 4 lines

Thanks,

-- Steve

Signature · 2 lines
Best regards,
Dmitry Adamushko
#6 lore raw
In reply to a message not in the archive
Quoted 10 lines · Dmitry Adamushko

It may be related, maybe not. One 'abnormal' thing (at least, it occurs only once in this log. Should be checked wheather it happens when the system works fine) is that a few iterations before the oops happens we observe the following pattern:

CPU=2 [94359.651930] hackbench:1932(120:120:120:T) -->> hackbench:1591(120:120:120)

CPU=2 [94359.651980] hackbench:1591(49:120:120:T) -->> swapper:0(140:120:140)

Note: the 'T' should be a 'D' because my logdev didn't add the change that -rt does (adding a 'M' state).

Thanks for noticing. The -rt patch has more priority inheritance situations than vanilla kernel (sleeping spinlocks or semaphors, and even the Preempt RCU Boost logic).

Quoted 6 lines

swapper (idle) --> softirq-timer (RT)
softirq-timer (RT) --> softirq-rcu (RT)
softirq-rcu(RT) --> picks up se == 0 for SCHED_NORMAL upon scheduling out ---> OOPS

Quoted 7 lines

'hackbench' was of SCHED_NORMAL upon scheduling _in_, and it's of RT type (prio: 49 and schedule() --> put_prev_task_rt()) upon scheduling _out_.

Unless you run some modified version of 'hackbench', it doesn't chenge scheduling classes... so maybe a lifted prio is a consequence of the resource contention with some RT task ?

Yes. Which means it could be an spinlock, mutex, semaphore or RCU read lock. But since it is in the TASK_UNINTERRUPTIBLE state, I'm willing to bet this is a mutex (or converted spinlock).

Quoted 4 lines

This 'hackbench' was the last SCHED_NORMAL task to run on this CPU... so however this NORMAL -> RT transition happened, it might leave a sched_fair's runqueue corrupted...

Could very well have. The PI uses task_setprio (aka. rt_mutex_setprio) to raise the priority. I'll start looking there.

Quoted 2 lines

(Will try to look more when time allows).

Thanks, I'll probably spend the rest of the day on this.

-- Steve

#8 lore raw
In reply to a message not in the archive

[ trimmed the cc' list ]

Quoted 9 lines · Steven Rostedt
Quoted 6 lines · Dmitry Adamushko

It may be related, maybe not. One 'abnormal' thing (at least, it occurs only once in this log. Should be checked wheather it happens when the system works fine) is that a few iterations before the oops happens we observe the following pattern:

Quoted 8 lines
Quoted 4 lines

CPU=2 [94359.651930] hackbench:1932(120:120:120:T) -->> hackbench:1591(120:120:120)

CPU=2 [94359.651980] hackbench:1591(49:120:120:T) -->> swapper:0(140:120:140)

Thanks for noticing. The -rt patch has more priority inheritance situations than vanilla kernel (sleeping spinlocks or semaphors, and even the Preempt RCU Boost logic).

One more thing is that we don't actually see a point where that 'hackbench' gets its priority lifted.

It was scheduled in as a NORMAL task and scheduled out as a RT one. i.e. the task got its prio elevated while it was running... a contention with the task on another CPU?

anyway, i.e. this task must have 'p->se.on_rq == 1' and I'd expect to see "switched_to_rt" message somewhere in between... hmm? (check_class_changed() shoud have been called in task_setprio()).

btw., we do see one 'switched_from_rt --> switched_to_fair' case for another 'hackbench' on CPU #0... according to traces, this one might get a prio lifted while sleeping (it got scheduled in as a RT task).

Quoted 3 lines

-- Steve

Signature · 2 lines
Best regards,
Dmitry Adamushko