Skip to contentSkip to conversation

Hang in sched_balance_self()

DISCUSSION Jack Steiner 2 participants 4 messages Jun 3, 2005 – Jun 7, 2005

lore
Share & tools
lore
#0 lore raw

Nick -

The latest 2.6.12-rc5-mm2 tree fails to boot on some of the 64p SGI systems. The system hangs immediately after printing:

	...
	Inode-cache hash table entries: 8388608 (order: 12, 67108864 bytes)
	Mount-cache hash table entries: 1024
	Boot processor id 0x0/0x0
	Brought up 64 CPUs
	Total of 64 processors activated (118415.36 BogoMIPS).

I have isolated the failure to cpu 0 hanging in sched_balance_self() during a fork (or clone). The "while" loop at the end of function never terminates, ie. sd is never NULL.

Is this a problem that you have seen before. If not, I'll do some more digging & isolate the problem.

Signature · 4 lines
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.
#1 lore raw
Quoted 13 lines · Jack Steiner

Nick -

The latest 2.6.12-rc5-mm2 tree fails to boot on some of the 64p SGI systems. The system hangs immediately after printing:

	...
	Inode-cache hash table entries: 8388608 (order: 12, 67108864 bytes)
	Mount-cache hash table entries: 1024
	Boot processor id 0x0/0x0
	Brought up 64 CPUs
	Total of 64 processors activated (118415.36 BogoMIPS).
Quoted 6 lines

I have isolated the failure to cpu 0 hanging in sched_balance_self() during
a fork (or clone). The "while" loop at the end of function never terminates, ie. sd is never NULL.

Is this a problem that you have seen before. If not, I'll do some more digging & isolate the problem.

Hi Jack,
I have not seen this problem, however I don't think it has had much testing with multilevel NUMA domains.

If you could do some more digging that would be great, however I plan to get time on a 64-way IA64 next week and look at some scheduler issues, so I'll keep this in mind if you haven't made any progress.

Thanks,
Nick

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#2 lore raw
In reply to Jack Steiner
Quoted 13 lines · Jack Steiner

Nick -

The latest 2.6.12-rc5-mm2 tree fails to boot on some of the 64p SGI systems. The system hangs immediately after printing:

	...
	Inode-cache hash table entries: 8388608 (order: 12, 67108864 bytes)
	Mount-cache hash table entries: 1024
	Boot processor id 0x0/0x0
	Brought up 64 CPUs
	Total of 64 processors activated (118415.36 BogoMIPS).
Quoted 6 lines

I have isolated the failure to cpu 0 hanging in sched_balance_self() during
a fork (or clone). The "while" loop at the end of function never terminates, ie. sd is never NULL.

Is this a problem that you have seen before. If not, I'll do some more digging & isolate the problem.

Hi Jack,
I haven't completely got to the bottom of this yet, but I was able to reproduce on a 64-way Altix, and something like the attached patch seems to 'fix' the problem.

I didn't have time to find what's gone wrong tonight, but I'll get to that tomorrow.

Signature · 1 line
SUSE Labs, Novell Inc.
kernel/sched.c +3 -1
Index: linux-2.6/kernel/sched.c
===================================================================
@@ -1113,6 +1113,7 @@ static int sched_balance_self(int cpu, i
 		cpumask_t span;
 		struct sched_group *group;
 		int new_cpu;
+		int weight;
 
 		span = sd->span;
 		group = find_idlest_group(sd, t, cpu);
@@ -1127,8 +1128,9 @@ static int sched_balance_self(int cpu, i
 		cpu = new_cpu;
 nextlevel:
 		sd = NULL;
+		weight = cpus_weight(span);
 		for_each_domain(cpu, tmp) {
-			if (cpus_subset(span, tmp->span))
+			if (weight <= cpus_weight(tmp->span))
 				break;
 			if (tmp->flags & flag)
 				sd = tmp;
#3 lore raw
Quoted 18 lines · Nick Piggin
Quoted 17 lines · Jack Steiner

Nick -

The latest 2.6.12-rc5-mm2 tree fails to boot on some of the 64p SGI systems. The system hangs immediately after printing:

	...
	Inode-cache hash table entries: 8388608 (order: 12, 67108864 bytes)
	Mount-cache hash table entries: 1024
	Boot processor id 0x0/0x0
	Brought up 64 CPUs
	Total of 64 processors activated (118415.36 BogoMIPS).

I have isolated the failure to cpu 0 hanging in sched_balance_self() during a fork (or clone). The "while" loop at the end of function never terminates, ie. sd is never NULL.

Quoted 8 lines
Quoted 3 lines

Is this a problem that you have seen before. If not, I'll do some more digging & isolate the problem.

Hi Jack,
I haven't completely got to the bottom of this yet, but I was able to reproduce on a 64-way Altix, and something like the attached patch seems to 'fix' the problem.

The fix works for me.

I'll send you notes that I collected. They may help explain what is going wrong. (I learned a lot about scheduler domains...:-)

Quoted 31 lines

I didn't have time to find what's gone wrong tonight, but I'll get to that tomorrow.

Signature · 5 lines
SUSE Labs, Novell Inc.


Index: linux-2.6/kernel/sched.c
===================================================================
kernel/sched.c +3 -1
@@ -1113,6 +1113,7 @@ static int sched_balance_self(int cpu, i
 		cpumask_t span;
 		struct sched_group *group;
 		int new_cpu;
+		int weight;
 
 		span = sd->span;
 		group = find_idlest_group(sd, t, cpu);
@@ -1127,8 +1128,9 @@ static int sched_balance_self(int cpu, i
 		cpu = new_cpu;
 nextlevel:
 		sd = NULL;
+		weight = cpus_weight(span);
 		for_each_domain(cpu, tmp) {
-			if (cpus_subset(span, tmp->span))
+			if (weight <= cpus_weight(tmp->span))
 				break;
 			if (tmp->flags & flag)
 				sd = tmp;
Signature · 4 lines
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.