CPU hotplug broken in 2.6.8-rc2 ?

DISCUSSION Dipankar Sarma 7 participants 12 messages Aug 2, 2004 – Aug 4, 2004

Share & tools

Dipankar Sarma

author

2004-08-02 09:52 UTC #0 lore raw

Could it be that recent sched domain stuff broke CPU hotplug ? While testing cpu hotplug with some RCU changes, I got the following panic (while onlining).

Thanks
Dipankar

cpu 0x2: Vector: 380 (Data SLB Access) at [c00000000152f4a0]

    pc: c00000000004b1b0: .find_busiest_group+0x274/0x464
    lr: c00000000004b0e4: .find_busiest_group+0x1a8/0x464
    sp: c00000000152f720

msr: 8000000000001032
dar: 10
current = 0xc000000001520040
paca = 0xc000000000535200
pid = 0, comm = swapper
enter ? for help
2:mon>

2:mon> t
[c00000000152f720] c000000000654f30 (unreliable) [c00000000152f830] c00000000004b4cc .rebalance_tick+0x12c/0x2d4 [c00000000152f920] c00000000005b954 .update_process_times+0xc4/0x154 [c00000000152f9c0] c0000000000385e8 .smp_local_timer_interrupt+0x3c/0x58 [c00000000152fa30] c000000000015088 .timer_interrupt+0x11c/0x3fc [c00000000152fb10] c00000000000a2b4 Decrementer_common+0xb4/0x100 --- Exception: 901 (Decrementer) at c000000000013bc0 .default_idle+0x70/0x110 [c00000000152fe90] c0000000000139e4 .cpu_idle+0x38/0x50 [c00000000152ff00] c000000000038e18 .start_secondary+0xfc/0x150 [c00000000152ff90] c00000000000bf20 .enable_64b_mode+0x0/0x28

2:mon> r

R00 = 000000000000002b   R16 = 0000000000000040
R01 = c00000000152f720   R17 = 0000000000000180
R02 = c0000000006d6d40   R18 = 0000000000000040
R03 = 0000000000000020   R19 = c000000000828e08
R04 = 0000000000000020   R20 = 0000000000000002
R05 = 0000000000000002   R21 = 0000000000000000
R06 = c00000000073f9b0   R22 = 0000000000000000
R07 = 000000000000000b   R23 = c00000000152f790
R08 = c0000000006fc728   R24 = c0000000006d5008
R09 = 0000000000000015   R25 = c000000000529c38
R10 = 0000000000000000   R26 = c000000000529c38
R11 = 0000000000000080   R27 = c00000000073f9b0
R12 = 0000000028282482   R28 = 0000000000000001
R13 = c000000000535200   R29 = 0000000000000015
R14 = c00000000073f980   R30 = c0000000005bbe00
R15 = 0000000000000000   R31 = c00000000152f720

pc = c00000000004b1b0 .find_busiest_group+0x274/0x464 lr = c00000000004b0e4 .find_busiest_group+0x1a8/0x464
msr = 8000000000001032 cr = 28282488
ctr = c000000000013b50 xer = 0000000000000000 trap = 380

Dipankar Sarma

author

2004-08-02 10:01 UTC #1 lore raw

Copied to the right email ids to avoid bouncing emails on replies.

Thanks
Dipankar

Quoted 60 lines · Dipankar Sarma

Could it be that recent sched domain stuff broke CPU hotplug ? While testing cpu hotplug with some RCU changes, I got the following panic (while onlining).
Thanks
Dipankar
cpu 0x2: Vector: 380 (Data SLB Access) at [c00000000152f4a0]
    pc: c00000000004b1b0: .find_busiest_group+0x274/0x464
    lr: c00000000004b0e4: .find_busiest_group+0x1a8/0x464
    sp: c00000000152f720
msr: 8000000000001032
dar: 10
current = 0xc000000001520040
paca = 0xc000000000535200
pid = 0, comm = swapper
enter ? for help
2:mon>
2:mon> t
[c00000000152f720] c000000000654f30 (unreliable) [c00000000152f830] c00000000004b4cc .rebalance_tick+0x12c/0x2d4 [c00000000152f920] c00000000005b954 .update_process_times+0xc4/0x154 [c00000000152f9c0] c0000000000385e8 .smp_local_timer_interrupt+0x3c/0x58 [c00000000152fa30] c000000000015088 .timer_interrupt+0x11c/0x3fc [c00000000152fb10] c00000000000a2b4 Decrementer_common+0xb4/0x100 --- Exception: 901 (Decrementer) at c000000000013bc0 .default_idle+0x70/0x110 [c00000000152fe90] c0000000000139e4 .cpu_idle+0x38/0x50 [c00000000152ff00] c000000000038e18 .start_secondary+0xfc/0x150 [c00000000152ff90] c00000000000bf20 .enable_64b_mode+0x0/0x28
2:mon> r
R00 = 000000000000002b   R16 = 0000000000000040
R01 = c00000000152f720   R17 = 0000000000000180
R02 = c0000000006d6d40   R18 = 0000000000000040
R03 = 0000000000000020   R19 = c000000000828e08
R04 = 0000000000000020   R20 = 0000000000000002
R05 = 0000000000000002   R21 = 0000000000000000
R06 = c00000000073f9b0   R22 = 0000000000000000
R07 = 000000000000000b   R23 = c00000000152f790
R08 = c0000000006fc728   R24 = c0000000006d5008
R09 = 0000000000000015   R25 = c000000000529c38
R10 = 0000000000000000   R26 = c000000000529c38
R11 = 0000000000000080   R27 = c00000000073f9b0
R12 = 0000000028282482   R28 = 0000000000000001
R13 = c000000000535200   R29 = 0000000000000015
R14 = c00000000073f980   R30 = c0000000005bbe00
R15 = 0000000000000000   R31 = c00000000152f720
pc = c00000000004b1b0 .find_busiest_group+0x274/0x464 lr = c00000000004b0e4 .find_busiest_group+0x1a8/0x464
msr = 8000000000001032 cr = 28282488
ctr = c000000000013b50 xer = 0000000000000000 trap = 380
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Anton Blanchard

2004-08-02 13:47 UTC #2 lore raw

Quoted 4 lines

Could it be that recent sched domain stuff broke CPU hotplug ? While testing cpu hotplug with some RCU changes, I got the following panic (while onlining).

Yeah, Im seeing the same thing.

Anton

Nathan Lynch

2004-08-02 22:52 UTC #3 lore raw

In reply to Dipankar Sarma

Quoted 9 lines · Dipankar Sarma

Copied to the right email ids to avoid bouncing emails on replies.
Thanks
Dipankar
Quoted 3 lines · Dipankar Sarma
Could it be that recent sched domain stuff broke CPU hotplug ? While testing cpu hotplug with some RCU changes, I got the following panic (while onlining).

Could you try on 2.6.8-rc2-mm2 along with this patch? Vatsa had a patch go in that should prevent the crash you are seeing -- the patch below is needed to prevent the same crash in the offline case. This check used to be in load_balance and some other scheduler functions, iirc; does anyone know why they were removed?

Nathan

---

kernel/sched.c renamed from kernel/sched.c~check-for-cpu-offline-in-load_balance +3 -0

diff -puN kernel/sched.c~check-for-cpu-offline-in-load_balance kernel/sched.c
@@ -1405,6 +1405,9 @@ static int load_balance(int this_cpu, ru
 
 	spin_lock(&this_rq->lock);
 
+	if (unlikely(cpu_is_offline(this_cpu)))
+		goto out_balanced;
+
 	group = find_busiest_group(sd, this_cpu, &imbalance, idle);
 	if (!group)
 		goto out_balanced;

Nathan Lynch

2004-08-02 20:25 UTC #4 lore raw

Quoted 5 lines · Nathan Lynch

Could you try on 2.6.8-rc2-mm2 along with this patch? Vatsa had a patch go in that should prevent the crash you are seeing -- the patch below is needed to prevent the same crash in the offline case. This check used to be in load_balance and some other scheduler functions, iirc; does anyone know why they were removed?

Er, I meant to put the check in rebalance_tick, not load_balance.

However, after a few minutes with this, I hit the BUG_ON in the CPU_DEAD case in migration_call; not sure whether this is a separate issue.

Nathan

---

kernel/sched.c renamed from kernel/sched.c~check-for-cpu-offline-in-rebalance_tick +3 -0

diff -puN kernel/sched.c~check-for-cpu-offline-in-rebalance_tick kernel/sched.c
@@ -1616,6 +1616,9 @@ static void rebalance_tick(int this_cpu,
 	unsigned long j = jiffies + CPU_OFFSET(this_cpu);
 	struct sched_domain *sd;
 
+	if (cpu_is_offline(this_cpu))
+		return;
+
 	/* Update our load */
 	old_load = this_rq->cpu_load;
 	this_load = this_rq->nr_running * SCHED_LOAD_SCALE;

Nathan Lynch

2004-08-04 06:05 UTC #5 lore raw

Quoted 11 lines · Nathan Lynch

Quoted 5 lines · Nathan Lynch
Could you try on 2.6.8-rc2-mm2 along with this patch? Vatsa had a patch go in that should prevent the crash you are seeing -- the patch below is needed to prevent the same crash in the offline case. This check used to be in load_balance and some other scheduler functions, iirc; does anyone know why they were removed?
Er, I meant to put the check in rebalance_tick, not load_balance.
However, after a few minutes with this, I hit the BUG_ON in the CPU_DEAD case in migration_call; not sure whether this is a separate issue.

So, with the cpu_is_offline check in rebalance_tick on top of 2.6.8-rc2-mm2, this is the BUG_ON in migration_call I tend to hit while hotplugging cpus as quickly as possible while running make -j 40:

        case CPU_DEAD:
                migrate_all_tasks(cpu);
                rq = cpu_rq(cpu);
                kthread_stop(rq->migration_thread);
                rq->migration_thread = NULL;
                /* Idle task back to normal (off runqueue, low prio) */
                rq = task_rq_lock(rq->idle, &flags);
                deactivate_task(rq->idle, rq);
                rq->idle->static_prio = MAX_PRIO;
                __setscheduler(rq->idle, SCHED_NORMAL, 0);
                task_rq_unlock(rq, &flags);
                BUG_ON(rq->nr_running != 0);

I can reproduce this on both ppc64 and i386. Does anyone know why this is happening?

If I remove the BUG_ON, things seem to go ok, but I doubt that's the right thing to do.

Nathan

Srivatsa Vaddagiri

2004-08-04 10:04 UTC #6 lore raw

Quoted 4 lines · Nathan Lynch

                BUG_ON(rq->nr_running != 0);
I can reproduce this on both ppc64 and i386. Does anyone know why this is happening?

I guess some task is still stuck with the dead CPU. Can you put a breakpoint on the BUG_ON and see the ps output (in kdb) to see which task is that when you hit the breakpoint?

I will also try debugging the 2.6.8-rc2 CPU Hotplug woes as soon as I can.

Signature · 5 lines

Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Nathan Lynch

2004-08-04 13:17 UTC #7 lore raw

Quoted 8 lines · Srivatsa Vaddagiri

Quoted 4 lines · Nathan Lynch
                BUG_ON(rq->nr_running != 0);
I can reproduce this on both ppc64 and i386. Does anyone know why this is happening?
I guess some task is still stuck with the dead CPU. Can you put a breakpoint on the BUG_ON and see the ps output (in kdb) to see which task is that when you hit the breakpoint?

The task is always something like cc1 or sh from the build which is running.

Quoted 3 lines

I will also try debugging the 2.6.8-rc2 CPU Hotplug woes as soon as I can.

Well, I am seeing this with 2.6.8-rc2-mm2 -- with 2.6.8-rc2-bk13 (plus the same patch) I cannot reproduce it; I have run the test for 12 hours without problem.

Nathan

Zwane Mwaikambo

2004-08-04 14:47 UTC #8 lore raw

In reply to Nathan Lynch

Quoted 9 lines · Nathan Lynch

                __setscheduler(rq->idle, SCHED_NORMAL, 0);
                task_rq_unlock(rq, &flags);
                BUG_ON(rq->nr_running != 0);
I can reproduce this on both ppc64 and i386. Does anyone know why this is happening?
If I remove the BUG_ON, things seem to go ok, but I doubt that's the right thing to do.

It could have something to do with the staircase scheduler, Con, got any wise words?

Con Kolivas

2004-08-04 21:09 UTC #9 lore raw

Quoted 10 lines · Zwane Mwaikambo

Quoted 7 lines · Nathan Lynch
               __setscheduler(rq->idle, SCHED_NORMAL, 0);
               task_rq_unlock(rq, &flags);
               BUG_ON(rq->nr_running != 0);
I can reproduce this on both ppc64 and i386. Does anyone know why this is happening?

Quoted 6 lines

Quoted 2 lines
If I remove the BUG_ON, things seem to go ok, but I doubt that's the right thing to do.
It could have something to do with the staircase scheduler, Con, got any wise words?

Doesn't this bug report say 2.6.8-rc2? It's mm2 that has staircase.

Con

Rusty Russell

2004-08-03 00:16 UTC #10 lore raw

In reply to Nathan Lynch

kernel/sched.c static int load_balance(int this_cpu, ru · line 1410

Quoted 10 lines · Nathan Lynch

diff -puN kernel/sched.c~check-for-cpu-offline-in-load_balance kernel/sched.c
kernel/sched.c renamed from kernel/sched.c~check-for-cpu-offline-in-load_balance +3 -0
@@ -1405,6 +1405,9 @@ static int load_balance(int this_cpu, ru
 
 	spin_lock(&this_rq->lock);
 
+	if (unlikely(cpu_is_offline(this_cpu)))
+		goto out_balanced;
+

cpu_is_offline() is "unlikely" already. Please just use "if (cpu_is_offline(this_cpu))"

Thanks,
Rusty.

Signature · 1 line

Anyone who quotes me in their signature is an idiot -- Rusty Russell

Zwane Mwaikambo

2004-08-02 15:56 UTC #11 lore raw

In reply to Dipankar Sarma

Quoted 3 lines · Dipankar Sarma

Could it be that recent sched domain stuff broke CPU hotplug ? While testing cpu hotplug with some RCU changes, I got the following panic (while onlining).

This may be related, i bumped into similar backtrace on i386 when a timer interrupt snuck in whilst the cpu was offline, so i ended up enabling timer interrupts only after the processor was on the map. This setup managed to survive 12hours with a kernel compile load over the weekend.