Hang in sched_balance_self()
Nick -
The latest 2.6.12-rc5-mm2 tree fails to boot on some of the 64p SGI systems. The system hangs immediately after printing:
... Inode-cache hash table entries: 8388608 (order: 12, 67108864 bytes) Mount-cache hash table entries: 1024 Boot processor id 0x0/0x0 Brought up 64 CPUs Total of 64 processors activated (118415.36 BogoMIPS).
I have isolated the failure to cpu 0 hanging in sched_balance_self() during a fork (or clone). The "while" loop at the end of function never terminates, ie. sd is never NULL.
Is this a problem that you have seen before. If not, I'll do some more digging & isolate the problem.
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
Quoted 13 lines · Jack Steiner
Nick -
The latest 2.6.12-rc5-mm2 tree fails to boot on some of the 64p SGI systems. The system hangs immediately after printing:
... Inode-cache hash table entries: 8388608 (order: 12, 67108864 bytes) Mount-cache hash table entries: 1024 Boot processor id 0x0/0x0 Brought up 64 CPUs Total of 64 processors activated (118415.36 BogoMIPS).
Quoted 6 lines
I have isolated the failure to cpu 0 hanging in
sched_balance_self()during
a fork (or clone). The "while" loop at the end of function never terminates, ie. sd is never NULL.Is this a problem that you have seen before. If not, I'll do some more digging & isolate the problem.
Hi Jack,
I have not seen this problem, however I don't think it has had much testing with multilevel NUMA domains.
If you could do some more digging that would be great, however I plan to get time on a 64-way IA64 next week and look at some scheduler issues, so I'll keep this in mind if you haven't made any progress.
Thanks,
Nick
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Quoted 13 lines · Jack Steiner
Nick -
The latest 2.6.12-rc5-mm2 tree fails to boot on some of the 64p SGI systems. The system hangs immediately after printing:
... Inode-cache hash table entries: 8388608 (order: 12, 67108864 bytes) Mount-cache hash table entries: 1024 Boot processor id 0x0/0x0 Brought up 64 CPUs Total of 64 processors activated (118415.36 BogoMIPS).
Quoted 6 lines
I have isolated the failure to cpu 0 hanging in
sched_balance_self()during
a fork (or clone). The "while" loop at the end of function never terminates, ie. sd is never NULL.Is this a problem that you have seen before. If not, I'll do some more digging & isolate the problem.
Hi Jack,
I haven't completely got to the bottom of this yet, but I was able to reproduce on a 64-way Altix, and something like the attached patch seems to 'fix' the problem.
I didn't have time to find what's gone wrong tonight, but I'll get to that tomorrow.
Signature · 1 line
SUSE Labs, Novell Inc.
kernel/sched.c +3 -1
@@ -1113,6 +1113,7 @@ static int sched_balance_self(int cpu, icpumask_t span;struct sched_group *group;int new_cpu; +int weight; span = sd->span; group =find_idlest_group (sd, t, cpu);
@@ -1127,8 +1128,9 @@ static int sched_balance_self(int cpu, i cpu = new_cpu; nextlevel: sd =NULL ; + weight =cpus_weight (span);for_each_domain (cpu, tmp) { -if (cpus_subset (span, tmp->span)) +if (weight <=cpus_weight (tmp->span))break ;if (tmp->flags & flag) sd = tmp;
Quoted 18 lines · Nick Piggin
Quoted 17 lines · Jack Steiner
Nick -
The latest 2.6.12-rc5-mm2 tree fails to boot on some of the 64p SGI systems. The system hangs immediately after printing:
... Inode-cache hash table entries: 8388608 (order: 12, 67108864 bytes) Mount-cache hash table entries: 1024 Boot processor id 0x0/0x0 Brought up 64 CPUs Total of 64 processors activated (118415.36 BogoMIPS).I have isolated the failure to cpu 0 hanging in
sched_balance_self()during a fork (or clone). The "while" loop at the end of function never terminates, ie. sd is never NULL.
Quoted 8 lines
Quoted 3 lines
Is this a problem that you have seen before. If not, I'll do some more digging & isolate the problem.
Hi Jack,
I haven't completely got to the bottom of this yet, but I was able to reproduce on a 64-way Altix, and something like the attached patch seems to 'fix' the problem.
The fix works for me.
I'll send you notes that I collected. They may help explain what is going wrong. (I learned a lot about scheduler domains...:-)
Quoted 31 lines
I didn't have time to find what's gone wrong tonight, but I'll get to that tomorrow.
Signature · 5 lines
SUSE Labs, Novell Inc. Index: linux-2.6/kernel/sched.c ===================================================================kernel/sched.c +3
-1@@ -1113,6 +1113,7 @@ static int sched_balance_self(int cpu, icpumask_t span;struct sched_group *group;int new_cpu; +int weight; span = sd->span; group =find_idlest_group (sd, t, cpu);@@ -1127,8 +1128,9 @@ static int sched_balance_self(int cpu, i cpu = new_cpu; nextlevel: sd =NULL ; + weight =cpus_weight (span);for_each_domain (cpu, tmp) { -if (cpus_subset (span, tmp->span)) +if (weight <=cpus_weight (tmp->span))break ;if (tmp->flags & flag) sd = tmp;
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.