Skip to contentSkip to conversation

NOHZ: fix nohz on cpu unplug

RFC Superseded Christian Borntraeger 4 participants 12 messages Jan 30, 2009 – Feb 6, 2009

lore
Share & tools
lore
#0 lore raw

RFC 1/1 NOHZ: fix nohz on cpu unplug

Ingo, Thomas,

After using cpu unplug I have seen one cpu with enabled ticks on s390, even on a idle systems. It turned out that nohz.cpu_mask is not updated on cpu unplug.
In select_nohz_load_balancer we check if the system is completely idle to turn of load balancing. We compare the weights of cpu_online_map with nohz.cpu_mask.
Since cpu_online_map is updated on cpu unplug, but nohz.cpu_mask is not, the check fails and the scheduler believes that we need an "idle load balancer" even on a fully idle system. Since the ilb cpu does not deactivate the timer tick this "breaks" NOHZ.

One possible fix is to only add the cpu to the nohz.cpu_mask, if the cpu is active.

The alternative is to add nohz.cpu_mask in the cpu hotplug path.

This patch seems to work on s390 (Shortly tested with kvm guests with cpu hotplug and kvm_stat on the host), but I want to hear your opinion.

---
 kernel/sched.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched.c +2 -1
Index: kvm/kernel/sched.c
===================================================================
@@ -3880,7 +3880,8 @@ int select_nohz_load_balancer(int stop_t
 	int cpu = smp_processor_id();
 
 	if (stop_tick) {
-		cpumask_set_cpu(cpu, nohz.cpu_mask);
+		if (cpu_active(cpu))
+			cpumask_set_cpu(cpu, nohz.cpu_mask);
 		cpu_rq(cpu)->in_nohz_recently = 1;
 
 		/*
#1 lore raw

This is a reminder mail. I still dont know if my fix is correct, although it seems to work..
I also CCed: Siddha, Suresh B <[email protected]>

Quoted 41 lines

Ingo, Thomas,

After using cpu unplug I have seen one cpu with enabled ticks on s390, even on a idle systems. It turned out that nohz.cpu_mask is not updated on cpu unplug.
In select_nohz_load_balancer we check if the system is completely idle to turn of load balancing. We compare the weights of cpu_online_map with nohz.cpu_mask.
Since cpu_online_map is updated on cpu unplug, but nohz.cpu_mask is not, the check fails and the scheduler believes that we need an "idle load balancer" even on a fully idle system. Since the ilb cpu does not deactivate the timer tick this "breaks" NOHZ.

One possible fix is to only add the cpu to the nohz.cpu_mask, if the cpu is active.

The alternative is to add nohz.cpu_mask in the cpu hotplug path.

This patch seems to work on s390 (Shortly tested with kvm guests with cpu hotplug and kvm_stat on the host), but I want to hear your opinion.

---
 kernel/sched.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: kvm/kernel/sched.c
===================================================================

kernel/sched.c +2 -1
@@ -3880,7 +3880,8 @@ int select_nohz_load_balancer(int stop_t
 	int cpu = smp_processor_id();
 
 	if (stop_tick) {
-		cpumask_set_cpu(cpu, nohz.cpu_mask);
+		if (cpu_active(cpu))
+			cpumask_set_cpu(cpu, nohz.cpu_mask);
 		cpu_rq(cpu)->in_nohz_recently = 1;
 
 		/*
#2 lore raw

PATCH v2 1/1 NOHZ: fix nohz on cpu unplug

After some testing, I think this patch is better:
[PATCH] NOHZ: fix nohz on cpu unplug

From: Christian Borntraeger <[email protected]>

After using cpu unplug I have seen one cpu with full ticks, even on a idle systems. It turns out that nohz.cpu_mask is not updated on cpu unplug.
In select_nohz_load_balancer we check if the system is completely idle to turn of load balancing. We compare cpu_online_map with nohz.cpu_mask.
Since cpu_online_map is updated on cpu unplug, but nohz.cpu_mask is not, the check fails and the scheduler believes that we need an "idle load balancer" even on a fully idle system. Since the ilb cpu does not deactivate the timer tick this breaks NOHZ.

This patch clear the nohz bits in the migration_call, a function that is already called by the cpu hotplug notifier.

Opinions?

---
 kernel/sched.c |    1 +
 1 file changed, 1 insertion(+)
kernel/sched.c +1 -0
Index: kvm/kernel/sched.c
===================================================================
@@ -6696,6 +6696,7 @@ migration_call(struct notifier_block *nf
 		rq->idle->sched_class = &idle_sched_class;
 		migrate_dead_tasks(cpu);
 		spin_unlock_irq(&rq->lock);
+		cpumask_clear_cpu(cpu, nohz.cpu_mask);
 		cpuset_unlock();
 		migrate_nr_uninterruptible(rq);
 		BUG_ON(rq->nr_running != 0);
#3 lore raw
Quoted 16 lines · Christian Borntraeger

After some testing, I think this patch is better:
[PATCH] NOHZ: fix nohz on cpu unplug

From: Christian Borntraeger <[email protected]>

After using cpu unplug I have seen one cpu with full ticks, even on a idle systems. It turns out that nohz.cpu_mask is not updated on cpu unplug.
In select_nohz_load_balancer we check if the system is completely idle to turn of load balancing. We compare cpu_online_map with nohz.cpu_mask.
Since cpu_online_map is updated on cpu unplug, but nohz.cpu_mask is not, the check fails and the scheduler believes that we need an "idle load balancer" even on a fully idle system. Since the ilb cpu does not deactivate the timer tick this breaks NOHZ.

Quoted 4 lines

This patch clear the nohz bits in the migration_call, a function that is already called by the cpu hotplug notifier.

Opinions?

Hi,

select_nohz_load_balancer() already takes care of one cpu offline scenario. It missed one more scenario leading to the issue you explained. We should take care of this in select_nohz_load_balancer().

Does this patch fix the issue? Thanks.

---

kernel/sched.c +11 -6
index 96439a4..8a0419b 100644
@@ -4026,19 +4026,24 @@ int select_nohz_load_balancer(int stop_tick)
 	int cpu = smp_processor_id();
 
 	if (stop_tick) {
-		cpumask_set_cpu(cpu, nohz.cpu_mask);
 		cpu_rq(cpu)->in_nohz_recently = 1;
 
-		/*
-		 * If we are going offline and still the leader, give up!
-		 */
-		if (!cpu_active(cpu) &&
-		    atomic_read(&nohz.load_balancer) == cpu) {
+		if (!cpu_active(cpu) {
+			if (atomic_read(&nohz.load_balancer) != cpu)
+				return 0;
+
+			/*
+			 * If we are going offline and still the leader,
+			 * give up!
+			 */
 			if (atomic_cmpxchg(&nohz.load_balancer, cpu, -1) != cpu)
 				BUG();
+
 			return 0;
 		}
 
+		cpumask_set_cpu(cpu, nohz.cpu_mask);
+
 		/* time for ilb owner also to sleep */
 		if (cpumask_weight(nohz.cpu_mask) == num_online_cpus()) {
 			if (atomic_read(&nohz.load_balancer) == cpu)
author Tested-by
#4 lore raw

Am Tuesday 03 February 2009 00:57:03 schrieb Suresh Siddha:

Quoted 1 line

Does this patch fix the issue? Thanks.

I fixed up a compile error. (See below). After that, the patch does indeed fix the problem.

Quoted 7 lines
kernel/sched.c +0 -0
index 96439a4..8a0419b 100644
@@ -4026,19 +4026,24 @@ int select_nohz_load_balancer(int stop_tick)
 	int cpu = smp_processor_id();
 

kernel/sched.c int select_nohz_load_balancer(int stop_tick) · line 4031

Quoted 10 lines
kernel/sched.c +1 -6
 	if (stop_tick) {
-		cpumask_set_cpu(cpu, nohz.cpu_mask);
 		cpu_rq(cpu)->in_nohz_recently = 1;
 
-		/*
-		 * If we are going offline and still the leader, give up!
-		 */
-		if (!cpu_active(cpu) &&
-		    atomic_read(&nohz.load_balancer) == cpu) {
+		if (!cpu_active(cpu) {

That does not compile...I used the following:

		if (!cpu_active(cpu)) {
Quoted 18 lines
kernel/sched.c +10 -0
+			if (atomic_read(&nohz.load_balancer) != cpu)
+				return 0;
+
+			/*
+			 * If we are going offline and still the leader,
+			 * give up!
+			 */
 			if (atomic_cmpxchg(&nohz.load_balancer, cpu, -1) != cpu)
 				BUG();
+
 			return 0;
 		}
 
+		cpumask_set_cpu(cpu, nohz.cpu_mask);
+
 		/* time for ilb owner also to sleep */
 		if (cpumask_weight(nohz.cpu_mask) == num_online_cpus()) {
 			if (atomic_read(&nohz.load_balancer) == cpu)
#5 lore raw
Quoted 28 lines · Christian Borntraeger

Am Tuesday 03 February 2009 00:57:03 schrieb Suresh Siddha:

Quoted 1 line

Does this patch fix the issue? Thanks.

I fixed up a compile error. (See below). After that, the patch does indeed fix the problem.

Quoted 7 lines
kernel/sched.c +0 -0
index 96439a4..8a0419b 100644
@@ -4026,19 +4026,24 @@ int select_nohz_load_balancer(int stop_tick)
 	int cpu = smp_processor_id();
 

kernel/sched.c int select_nohz_load_balancer(int stop_tick) · line 4031

Quoted 10 lines
kernel/sched.c +1 -6
 	if (stop_tick) {
-		cpumask_set_cpu(cpu, nohz.cpu_mask);
 		cpu_rq(cpu)->in_nohz_recently = 1;
 
-		/*
-		 * If we are going offline and still the leader, give up!
-		 */
-		if (!cpu_active(cpu) &&
-		    atomic_read(&nohz.load_balancer) == cpu) {
+		if (!cpu_active(cpu) {

That does not compile...I used the following:

Quoted 1 line
		if (!cpu_active(cpu)) {

Could you please resubmit an updated patch?

Ingo

author Tested-by
#6 lore raw

Am Tuesday 03 February 2009 13:28:47 schrieb Ingo Molnar:

Quoted 6 lines
Quoted 4 lines

kernel/sched.c int select_nohz_load_balancer(int stop_tick) · line 4031

Quoted 1 line
kernel/sched.c +1 -0
+		if (!cpu_active(cpu) {

That does not compile...I used the following:

		if (!cpu_active(cpu)) {

Could you please resubmit an updated patch?

Suresh,

can you send the updated patch? After all, its your work and we need your Signed-off-by anyway.

You can add my

Thanks

Christian

Tested-by Christian Borntraeger
#7 lore raw
Quoted 8 lines · Christian Borntraeger

Am Tuesday 03 February 2009 13:28:47 schrieb Ingo Molnar:

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

+ if (!cpu_active(cpu) {

That does not compile...I used the following:

            if (!cpu_active(cpu)) {

Could you please resubmit an updated patch?

Quoted 8 lines

Suresh,

can you send the updated patch? After all, its your work and we need your Signed-off-by anyway.

You can add my

Thanks Christian for testing the fix. Ingo, appended is the updated patch. Thanks.
---

From: Suresh Siddha <[email protected]>
Subject: sched: fix nohz load balancer on cpu offline

Christian Borntraeger reports:

Quoted 8 lines

After a logical cpu offline, even on a complete idle system, there is one cpu with full ticks. It turns out that nohz.cpu_mask has the the offlined cpu still set.

In select_nohz_load_balancer() we check if the system is completely idle to turn of load balancing. We compare cpu_online_map with nohz.cpu_mask. Since cpu_online_map is updated on cpu unplug, but nohz.cpu_mask is not, the check fails and the scheduler believes

that

Quoted 1 line

we need an "idle load balancer" even on a fully idle system. Since the

ilb

Quoted 1 line

cpu does not deactivate the timer tick this breaks NOHZ.

Fix the select_nohz_load_balancer() to not set the nohz.cpu_mask while a cpu is going offline.

---

kernel/sched.c +11 -6
index 96439a4..8a0419b 100644
@@ -4026,19 +4026,24 @@ int select_nohz_load_balancer(int stop_tick)
 	int cpu = smp_processor_id();
 
 	if (stop_tick) {
-		cpumask_set_cpu(cpu, nohz.cpu_mask);
 		cpu_rq(cpu)->in_nohz_recently = 1;
 
-		/*
-		 * If we are going offline and still the leader, give up!
-		 */
-		if (!cpu_active(cpu) &&
-		    atomic_read(&nohz.load_balancer) == cpu) {
+		if (!cpu_active(cpu)) {
+			if (atomic_read(&nohz.load_balancer) != cpu)
+				return 0;
+
+			/*
+			 * If we are going offline and still the leader,
+			 * give up!
+			 */
 			if (atomic_cmpxchg(&nohz.load_balancer, cpu, -1) != cpu)
 				BUG();
+
 			return 0;
 		}
 
+		cpumask_set_cpu(cpu, nohz.cpu_mask);
+
 		/* time for ilb owner also to sleep */
 		if (cpumask_weight(nohz.cpu_mask) == num_online_cpus()) {
 			if (atomic_read(&nohz.load_balancer) == cpu)
Applied
#8 lore raw
Quoted 14 lines · Suresh Siddha
Quoted 13 lines · Christian Borntraeger

Am Tuesday 03 February 2009 13:28:47 schrieb Ingo Molnar:

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

+ if (!cpu_active(cpu) {

That does not compile...I used the following:

            if (!cpu_active(cpu)) {

Could you please resubmit an updated patch?

Suresh,

can you send the updated patch? After all, its your work and we need your Signed-off-by anyway.

Quoted 6 lines
Quoted 3 lines

You can add my

Thanks Christian for testing the fix. Ingo, appended is the updated patch. Thanks.

applied to tip:sched/urgent, thanks guys!

Ingo

#9 lore raw
Quoted 13 lines · Ingo Molnar
Quoted 11 lines · Suresh Siddha
Quoted 10 lines · Christian Borntraeger

Am Tuesday 03 February 2009 13:28:47 schrieb Ingo Molnar:

Quoted 4 lines
Quoted 3 lines
Quoted 1 line

+ if (!cpu_active(cpu) {

That does not compile...I used the following:

            if (!cpu_active(cpu)) {

Could you please resubmit an updated patch?

Suresh,

can you send the updated patch? After all, its your work and we need your Signed-off-by anyway.

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

You can add my

Thanks Christian for testing the fix. Ingo, appended is the updated patch. Thanks.

applied to tip:sched/urgent, thanks guys!

Ingo

..

How far back in (kernel release) time does this problem exist? Candidate for -stable ?

Cheers

#10 lore raw
Quoted 2 lines · Mark Lord

How far back in (kernel release) time does this problem exist? Candidate for -stable ?

Problem is present for a while now. But I don't think this is a common case scenario (as the issue happens only for the duration when we leave a cpu offline, and it should get fixed the moment that logical cpu is back online).

thanks,
suresh

#11 lore raw
Quoted 8 lines · Suresh Siddha
Quoted 2 lines · Mark Lord

How far back in (kernel release) time does this problem exist? Candidate for -stable ?

Problem is present for a while now. But I don't think this is a common case scenario (as the issue happens only for the duration when we leave a cpu offline, and it should get fixed the moment that logical cpu is back online).

..

There is an existing bug (for some time now) in the kernel shutdown for multi-CPUs. Once in a while, perhaps every 20-30 halts, the kernel fails to power-off the machine.

I've seen this problem here since 2.6.18 or so, on multiple different machines with Core2duo and Core2quad processors.

It comes and goes, depending upon the kernel version and exact .config that is used. Any attempt to instrument it generally changes the race conditions enough that it stops happening.

I'm just wondering if this bug might explain some of that.

Cheers