Skip to contentSkip to conversation

sched: cfs_rq h_load might not update due to irq disable

PATCH In review YT Chang 2 participants 3 messages Nov 21, 2019 – Nov 26, 2019

Subsystems (2)
lore
Share & tools
lore

YT Chang

author
#0 lore raw

PATCH 1/1 sched: cfs_rq h_load might not update due to irq disable

Syndrome:

Two CPUs might do idle balance in the same time.
One CPU does idle balance and pulls some tasks.
However before pick next task, ALL task are pulled back to other CPU. That results in infinite loop in both CPUs.

=========================================
code flow:

in pick_next_task_fair()

again:

if nr_running == 0
goto idle
pick next task
return

idle:
idle_balance
/* pull some tasks from other CPU,
* However other CPU are also do idle balance,
* and pull back these task */

go to again

=========================================
The result to pull ALL tasks back when the task_h_load is incorrect and too low.

static unsigned long task_h_load(struct task_struct *p)

{
        struct cfs_rq *cfs_rq = task_cfs_rq(p);

	update_cfs_rq_h_load(cfs_rq);
	return div64_ul(p->se.avg.load_avg_contrib * cfs_rq->h_load,
			cfs_rq->runnable_load_avg + 1);
}

The cfs_rq->h_load is incorrect and might too small. The original idea of cfs_rq::last_h_load_update will not update cfs_rq::h_load more than once a jiffies. When the Two CPUs pull each other in the pick_next_task_fair, the irq disabled and result in jiffie not update. (Other CPUs wait for runqueue lock locked by the two CPUs. So, ALL CPUs are irq disabled.)

Solution:
cfs_rq h_load might not update due to irq disable use sched_clock instead jiffies

---
 kernel/sched/fair.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/fair.c +3 -1
index 83ab35e..231c53f 100644
@@ -7578,9 +7578,11 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
 {
 	struct rq *rq = rq_of(cfs_rq);
 	struct sched_entity *se = cfs_rq->tg->se[cpu_of(rq)];
-	unsigned long now = jiffies;
+	u64 now = sched_clock_cpu(cpu_of(rq));
 	unsigned long load;
 
+	now = now * HZ >> 30;
+
 	if (cfs_rq->last_h_load_update == now)
 		return;
 
Signature · 1 line
1.9.1
maintainer
#1 lore raw
Quoted 6 lines · YT Chang

Syndrome:

Two CPUs might do idle balance in the same time.
One CPU does idle balance and pulls some tasks.
However before pick next task, ALL task are pulled back to other CPU. That results in infinite loop in both CPUs.

Can you easily reproduce this?

Quoted 18 lines

=========================================
code flow:

in pick_next_task_fair()

again:

if nr_running == 0
goto idle
pick next task
return

idle:
idle_balance
/* pull some tasks from other CPU,
* However other CPU are also do idle balance,
* and pull back these task */

Quoted 5 lines

go to again

=========================================
The result to pull ALL tasks back when the task_h_load is incorrect and too low.

Clearly you're not running a PREEMPT kernel, otherwise the break in detach_tasks() would've saved you, right?

Quoted 9 lines

static unsigned long task_h_load(struct task_struct *p)

{
        struct cfs_rq *cfs_rq = task_cfs_rq(p);

	update_cfs_rq_h_load(cfs_rq);
	return div64_ul(p->se.avg.load_avg_contrib * cfs_rq->h_load,
			cfs_rq->runnable_load_avg + 1);
}
Quoted 7 lines

The cfs_rq->h_load is incorrect and might too small. The original idea of cfs_rq::last_h_load_update will not update cfs_rq::h_load more than once a jiffies. When the Two CPUs pull each other in the pick_next_task_fair, the irq disabled and result in jiffie not update. (Other CPUs wait for runqueue lock locked by the two CPUs. So, ALL CPUs are irq disabled.)

This cannot be true; because the loop drops rq->lock, so other CPUs should have an opportunity to acquire the lock and make progress.

Quoted 16 lines

Solution:
cfs_rq h_load might not update due to irq disable use sched_clock instead jiffies

---
 kernel/sched/fair.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index 83ab35e..231c53f 100644
@@ -7578,9 +7578,11 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
 {
 	struct rq *rq = rq_of(cfs_rq);

kernel/sched/fair.c static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) · line 7588

Quoted 10 lines
kernel/sched/fair.c +3 -1
 	struct sched_entity *se = cfs_rq->tg->se[cpu_of(rq)];
-	unsigned long now = jiffies;
+	u64 now = sched_clock_cpu(cpu_of(rq));
 	unsigned long load;
 
+	now = now * HZ >> 30;
+
 	if (cfs_rq->last_h_load_update == now)
 		return;
 

This is disguisting and wrong. That is not the correct relation between sched_clock() and jiffies.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#2 lore raw
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines · YT Chang

Syndrome:

Two CPUs might do idle balance in the same time.
One CPU does idle balance and pulls some tasks.
However before pick next task, ALL task are pulled back to other CPU. That results in infinite loop in both CPUs.

Can you easily reproduce this?

No, I can't easily reproduce this.

Quoted 45 lines
Quoted 19 lines

=========================================
code flow:

in pick_next_task_fair()

again:

if nr_running == 0
goto idle
pick next task
return

idle:
idle_balance
/* pull some tasks from other CPU,
* However other CPU are also do idle balance,
* and pull back these task */

Quoted 5 lines

go to again

=========================================
The result to pull ALL tasks back when the task_h_load is incorrect and too low.

Clearly you're not running a PREEMPT kernel, otherwise the break in detach_tasks() would've saved you, right?

Quoted 9 lines

static unsigned long task_h_load(struct task_struct *p)

{
        struct cfs_rq *cfs_rq = task_cfs_rq(p);

	update_cfs_rq_h_load(cfs_rq);
	return div64_ul(p->se.avg.load_avg_contrib * cfs_rq->h_load,
			cfs_rq->runnable_load_avg + 1);
}
Quoted 7 lines

The cfs_rq->h_load is incorrect and might too small. The original idea of cfs_rq::last_h_load_update will not update cfs_rq::h_load more than once a jiffies. When the Two CPUs pull each other in the pick_next_task_fair, the irq disabled and result in jiffie not update. (Other CPUs wait for runqueue lock locked by the two CPUs. So, ALL CPUs are irq disabled.)

Quoted 2 lines

This cannot be true; because the loop drops rq->lock, so other CPUs should have an opportunity to acquire the lock and make progress.

I recheck other CPUs situation.
Other CPUs are irq disabled due to wait for lock (Not runqueue lock).

The root cause should be why other CPUs are waiting for the lock rather than replacing jiffies with sched_clock().

Quoted 30 lines
Quoted 17 lines

Solution:
cfs_rq h_load might not update due to irq disable use sched_clock instead jiffies

---
 kernel/sched/fair.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index 83ab35e..231c53f 100644
@@ -7578,9 +7578,11 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
 {
 	struct rq *rq = rq_of(cfs_rq);

kernel/sched/fair.c static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) · line 7588

Quoted 10 lines
kernel/sched/fair.c +3 -1
 	struct sched_entity *se = cfs_rq->tg->se[cpu_of(rq)];
-	unsigned long now = jiffies;
+	u64 now = sched_clock_cpu(cpu_of(rq));
 	unsigned long load;
 
+	now = now * HZ >> 30;
+
 	if (cfs_rq->last_h_load_update == now)
 		return;
 

This is disguisting and wrong. That is not the correct relation between sched_clock() and jiffies.