Skip to contentSkip to conversation

CPU offline/online related hang with latest git

PATCH In review Sachin Sant 3 participants 9 messages Dec 20, 2009

lore
Share & tools
lore
#0 lore raw

With 2.6.33-rc1-git1 (dd59f6c..) i am having trouble with CPU hotplug on multiple architectures. Trying to offline a CPU results in a machine hang.

Sample o/p from a 4 way x86_64 box :

#:/sys/devices/system/cpu/cpu3 # cat online
1
#:/sys/devices/system/cpu/cpu3 # echo 0 > online

Stack:
Call Trace:
<IRQ>
<EOI>
Code: 45 f0 48 89 45 b8 48 8d 45 d0 4c 89 4d f8 c7 45 b0 10 00 00 00 48 89 45 c0 e8 5a ff ff ff c9 c3 89 f0 b9 40 00 00 00 55 99 f7 f9 <48> 89 e5 48 89 f9 48 83 ec 08 31 d2 41 89 c0 eb 12 48 8b 01 48
Stack:
Call Trace:
Code: 20 00 00 00 00 48 89 3e 49 8d 74 24 28 e8 45 75 1d 00 49 c7 44 24 50 00 00 00 00 48 8b 9b 60 01 00 00 48 85 db 0f 85 25 ff ff ff <5b> 41 5c c9 c3 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 81 ec
Stack:
Call Trace:
Code: 48 0f 45 d8 eb 1a 44 0f a3 20 19 c0 48 c7 c3 10 b5 a0 81 85 c0 48 c7 c0 30 b5 a0 81 48 0f 44 d8 31 c0 f3 90 44 8b 25 82 35 96 00 <41> 39 c4 74 46 41 83 fc 02 74 08 41 83 fc 03 75 12 eb 03 fa eb
Stack:
Call Trace:
Code: 48 c7 c0 30 b5 a0 81 48 0f 45 d8 eb 1a 44 0f a3 20 19 c0 48 c7 c3 10 b5 a0 81 85 c0 48 c7 c0 30 b5 a0 81 48 0f 44 d8 31 c0 f3 90 <44> 8b 25 82 35 96 00 41 39 c4 74 46 41 83 fc 02 74 08 41 83 fc

Uhhuh. NMI received for unknown reason 25 on CPU 0. Do you have a strange power saving mode enabled? Dazed and confused, but trying to continue

The above messages are repeated after this. I observed similar hangs on other architectures as well (s390x, PowerPC, x86_32).

2.6.33-rc1 worked fine. I haven't tried the bisect. Will do that first thing tomorrow morning.

Thanks
-Sachin

Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
#1 lore raw
Quoted 3 lines · Sachin Sant

With 2.6.33-rc1-git1 (dd59f6c..) i am having trouble with CPU hotplug on multiple architectures. Trying to offline a CPU results in a machine hang.

damnit, you're right.. it's getting stuck during hotplug on stop machine some place.

didn't notice it because it didn't crash..

sorry for that :-(

#2 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 3 lines · Sachin Sant

With 2.6.33-rc1-git1 (dd59f6c..) i am having trouble with CPU hotplug on multiple architectures. Trying to offline a CPU results in a machine hang.

damnit, you're right.. it's getting stuck during hotplug on stop machine some place.

didn't notice it because it didn't crash..

sorry for that :-(

Perhaps this is also why suspend doesn't work on current -git... -rc1 is fine.

Signature · 1 line
Jens Axboe
#3 lore raw
Quoted 9 lines · Jens Axboe
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Sachin Sant

With 2.6.33-rc1-git1 (dd59f6c..) i am having trouble with CPU hotplug on multiple architectures. Trying to offline a CPU results in a machine hang.

damnit, you're right.. it's getting stuck during hotplug on stop machine some place.

Quoted 6 lines
Quoted 3 lines

didn't notice it because it didn't crash..

sorry for that :-(

Perhaps this is also why suspend doesn't work on current -git... -rc1 is fine.

Yep, suspend relies on hotplug.

#4 lore raw
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Jens Axboe
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Sachin Sant

With 2.6.33-rc1-git1 (dd59f6c..) i am having trouble with CPU hotplug on multiple architectures. Trying to offline a CPU results in a machine hang.

damnit, you're right.. it's getting stuck during hotplug on stop machine some place.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

didn't notice it because it didn't crash..

sorry for that :-(

Perhaps this is also why suspend doesn't work on current -git... -rc1 is fine.

Yep, suspend relies on hotplug.

Exactly, hence the connection. Shall I bisect, or do you already know what the problem is?

Signature · 1 line
Jens Axboe
#5 lore raw
Quoted 2 lines · Jens Axboe

Exactly, hence the connection. Shall I bisect, or do you already know what the problem is?

I have a definite suspect alright, let me prod at this a bit more.

#6 lore raw

PATCH 1/1 sched: Fix hotplug

In reply to Jens Axboe

The hot-unplug kstopmachine usage does a wakeup after deactivating the cpu, hence we cannot use cpu_active() here but must rely on the good olde online.

---
 kernel/sched.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
index 720df10..0ac4fa5 100644
@@ -2348,7 +2348,7 @@ int select_task_rq(struct task_struct *p, int sd_flags, int wake_flags)
 	 *   not worry about this generic constraint ]
 	 */
 	if (unlikely(!cpumask_test_cpu(cpu, &p->cpus_allowed) ||
-		     !cpu_active(cpu)))
+		     !cpu_online(cpu)))
 		cpu = select_fallback_rq(task_cpu(p), p);
 
 	return cpu;
Tested-by on v1 1/1
#7 lore raw

Re: [PATCH] sched: Fix hotplug

Quoted 3 lines · Peter Zijlstra

The hot-unplug kstopmachine usage does a wakeup after deactivating the cpu, hence we cannot use cpu_active() here but must rely on the good olde online.

Yep, this works for me!

Quoted 21 lines
---
 kernel/sched.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
index 720df10..0ac4fa5 100644
@@ -2348,7 +2348,7 @@ int select_task_rq(struct task_struct *p, int sd_flags, int wake_flags)
 	 *   not worry about this generic constraint ]
 	 */
 	if (unlikely(!cpumask_test_cpu(cpu, &p->cpus_allowed) ||
-		     !cpu_active(cpu)))
+		     !cpu_online(cpu)))
 		cpu = select_fallback_rq(task_cpu(p), p);
 
 	return cpu;
Signature · 1 line
Jens Axboe
#8 lore raw

[tip:sched/urgent] sched: Fix hotplug hang

In reply to Peter Zijlstra
Commit-ID:  70f1120527797adb31c68bdc6f1b45e182c342c7
Gitweb:     http://git.kernel.org/tip/70f1120527797adb31c68bdc6f1b45e182c342c7
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Sun, 20 Dec 2009 17:36:27 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 20 Dec 2009 23:31:23 +0100

sched: Fix hotplug hang

The hot-unplug kstopmachine usage does a wakeup after deactivating the cpu, hence we cannot use cpu_active() here but must rely on the good olde online.

LKML-Reference: <1261326987.4314.24.camel@laptop>

---
 kernel/sched.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
index 7ffde2a..87f1f47 100644
@@ -2346,7 +2346,7 @@ int select_task_rq(struct task_struct *p, int sd_flags, int wake_flags)
 	 *   not worry about this generic constraint ]
 	 */
 	if (unlikely(!cpumask_test_cpu(cpu, &p->cpus_allowed) ||
-		     !cpu_active(cpu)))
+		     !cpu_online(cpu)))
 		cpu = select_fallback_rq(task_cpu(p), p);
 
 	return cpu;