Skip to contentSkip to conversation

CPU Hotplug test failures on powerpc

Next In review Sachin Sant 5 participants 24 messages Dec 11, 2009 – Dec 16, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

While executing cpu_hotplug(from autotest) tests against latest next on a power6 box, the machine locks up. A soft reset shows the following trace

cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]
pc: c0000000003433d8: .find_next_bit+0x54/0xc4
lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
sp: c00000000c933650
msr: 8000000000089032
current = 0xc00000000c173840
paca = 0xc000000000bc2600
pid = 2602, comm = hotplug06.top.s
enter ? for help
[link register ] c000000000342f10 .cpumask_next_and+0x4c/0x94
[c00000000c933650] c0000000000e9f34 .cpuset_cpus_allowed_locked+0x38/0x74 (unreliable) [c00000000c9336e0] c000000000090074 .move_task_off_dead_cpu+0xc4/0x1ac [c00000000c9337a0] c0000000005e4e5c .migration_call+0x304/0x830 [c00000000c933880] c0000000005e0880 .notifier_call_chain+0x68/0xe0 [c00000000c933920] c00000000012a92c ._cpu_down+0x210/0x34c [c00000000c933a90] c00000000012aad8 .cpu_down+0x70/0xa8 [c00000000c933b20] c000000000525940 .store_online+0x54/0x894 [c00000000c933bb0] c000000000463430 .sysdev_store+0x3c/0x50 [c00000000c933c20] c0000000001f8320 .sysfs_write_file+0x124/0x18c [c00000000c933ce0] c00000000017edac .vfs_write+0xd4/0x1fc [c00000000c933d80] c00000000017efdc .SyS_write+0x58/0xa0 [c00000000c933e30] c0000000000085b4 syscall_exit+0x0/0x40
--- Exception: c01 (System Call) at 00000fff9fa8a8f8
SP (fffe7aef200) is in userspace
0:mon> e
cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]
pc: c0000000003433d8: .find_next_bit+0x54/0xc4
lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
sp: c00000000c933650
msr: 8000000000089032
current = 0xc00000000c173840
paca = 0xc000000000bc2600
pid = 2602, comm = hotplug06.top.s

Last few messages from the dmesg log shows

0:mon> <4>IRQ 17 affinity broken off cpu 0
<4>IRQ 18 affinity broken off cpu 0
<4>IRQ 19 affinity broken off cpu 0
<4>IRQ 264 affinity broken off cpu 0
<4>cpu 0 (hwid 0) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[0] <4>Processor 0 found.
<4>IRQ 17 affinity broken off cpu 1
<4>IRQ 18 affinity broken off cpu 1
<4>IRQ 19 affinity broken off cpu 1
<4>IRQ 264 affinity broken off cpu 1
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<6>process 2423 (bash) no longer affine to cpu1 <7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<3>INFO: RCU detected CPU 0 stall (t=1000 jiffies) <3>INFO: RCU detected CPU 0 stall (t=4000 jiffies) 0:mon>

After some debugging a possible suspect seems to be commit 6ad4c18.. : sched: Fix balance vs hotplug race

If i revert this patch i am able to execute the tests on this power6 without any issues.

But at the same time the above patch is required to solve the cpu hotplug related race on x86_64(as a side note this same x86_64 issue can be recreated against latest Linus git as well) that i reported here :

http://marc.info/?l=linux-kernel&m=125802682922299&w=2

I will try few more iterations with and without the above patch just to make sure i have the correct results.

If someone has a suggestion let me know.

Thanks
-Sachin

Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
#1 lore raw
Quoted 3 lines · Sachin Sant

While executing cpu_hotplug(from autotest) tests against latest next on a power6 box, the machine locks up. A soft reset shows the following trace

Have you heard anything about that one yet or it's still to be debugged ? It probably hit upstream by now.

Cheers,
Ben.

Quoted 91 lines

cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current = 0xc00000000c173840
  paca    = 0xc000000000bc2600
    pid   = 2602, comm = hotplug06.top.s

enter ? for help
[link register ] c000000000342f10 .cpumask_next_and+0x4c/0x94
[c00000000c933650] c0000000000e9f34 .cpuset_cpus_allowed_locked+0x38/0x74 (unreliable) [c00000000c9336e0] c000000000090074 .move_task_off_dead_cpu+0xc4/0x1ac [c00000000c9337a0] c0000000005e4e5c .migration_call+0x304/0x830 [c00000000c933880] c0000000005e0880 .notifier_call_chain+0x68/0xe0 [c00000000c933920] c00000000012a92c ._cpu_down+0x210/0x34c [c00000000c933a90] c00000000012aad8 .cpu_down+0x70/0xa8 [c00000000c933b20] c000000000525940 .store_online+0x54/0x894 [c00000000c933bb0] c000000000463430 .sysdev_store+0x3c/0x50 [c00000000c933c20] c0000000001f8320 .sysfs_write_file+0x124/0x18c [c00000000c933ce0] c00000000017edac .vfs_write+0xd4/0x1fc [c00000000c933d80] c00000000017efdc .SyS_write+0x58/0xa0 [c00000000c933e30] c0000000000085b4 syscall_exit+0x0/0x40
--- Exception: c01 (System Call) at 00000fff9fa8a8f8
SP (fffe7aef200) is in userspace
0:mon> e
cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current = 0xc00000000c173840
  paca    = 0xc000000000bc2600
    pid   = 2602, comm = hotplug06.top.s

Last few messages from the dmesg log shows

0:mon>
<4>IRQ 17 affinity broken off cpu 0
<4>IRQ 18 affinity broken off cpu 0
<4>IRQ 19 affinity broken off cpu 0
<4>IRQ 264 affinity broken off cpu 0
<4>cpu 0 (hwid 0) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[0] <4>Processor 0 found.
<4>IRQ 17 affinity broken off cpu 1
<4>IRQ 18 affinity broken off cpu 1
<4>IRQ 19 affinity broken off cpu 1
<4>IRQ 264 affinity broken off cpu 1
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<6>process 2423 (bash) no longer affine to cpu1 <7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<3>INFO: RCU detected CPU 0 stall (t=1000 jiffies) <3>INFO: RCU detected CPU 0 stall (t=4000 jiffies) 0:mon>

After some debugging a possible suspect seems to be commit 6ad4c18.. : sched: Fix balance vs hotplug race

If i revert this patch i am able to execute the tests on this power6 without any issues.

But at the same time the above patch is required to solve the cpu hotplug related race on x86_64(as a side note this same x86_64 issue can be recreated against latest Linus git as well) that i reported here :

http://marc.info/?l=linux-kernel&m=125802682922299&w=2

I will try few more iterations with and without the above patch just to make sure i have the correct results.

If someone has a suggestion let me know.

Thanks
-Sachin

#2 lore raw
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 4 lines · Sachin Sant

While executing cpu_hotplug(from autotest) tests against latest next on a power6 box, the machine locks up. A soft reset shows the following trace

Have you heard anything about that one yet or it's still to be debugged ? It probably hit upstream by now.

Haven't received any response yet.

As you mentioned that patch went upstream and so did the problem.

thanks
-Sachin

Quoted 103 lines

Cheers,
Ben.

Quoted 81 lines

cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current = 0xc00000000c173840
  paca    = 0xc000000000bc2600
    pid   = 2602, comm = hotplug06.top.s

enter ? for help
[link register ] c000000000342f10 .cpumask_next_and+0x4c/0x94
[c00000000c933650] c0000000000e9f34 .cpuset_cpus_allowed_locked+0x38/0x74 (unreliable) [c00000000c9336e0] c000000000090074 .move_task_off_dead_cpu+0xc4/0x1ac [c00000000c9337a0] c0000000005e4e5c .migration_call+0x304/0x830 [c00000000c933880] c0000000005e0880 .notifier_call_chain+0x68/0xe0 [c00000000c933920] c00000000012a92c ._cpu_down+0x210/0x34c [c00000000c933a90] c00000000012aad8 .cpu_down+0x70/0xa8 [c00000000c933b20] c000000000525940 .store_online+0x54/0x894 [c00000000c933bb0] c000000000463430 .sysdev_store+0x3c/0x50 [c00000000c933c20] c0000000001f8320 .sysfs_write_file+0x124/0x18c [c00000000c933ce0] c00000000017edac .vfs_write+0xd4/0x1fc [c00000000c933d80] c00000000017efdc .SyS_write+0x58/0xa0 [c00000000c933e30] c0000000000085b4 syscall_exit+0x0/0x40
--- Exception: c01 (System Call) at 00000fff9fa8a8f8
SP (fffe7aef200) is in userspace
0:mon> e
cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current = 0xc00000000c173840
  paca    = 0xc000000000bc2600
    pid   = 2602, comm = hotplug06.top.s

Last few messages from the dmesg log shows

0:mon> <4>IRQ 17 affinity broken off cpu 0
<4>IRQ 18 affinity broken off cpu 0
<4>IRQ 19 affinity broken off cpu 0
<4>IRQ 264 affinity broken off cpu 0
<4>cpu 0 (hwid 0) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[0] <4>Processor 0 found.
<4>IRQ 17 affinity broken off cpu 1
<4>IRQ 18 affinity broken off cpu 1
<4>IRQ 19 affinity broken off cpu 1
<4>IRQ 264 affinity broken off cpu 1
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<6>process 2423 (bash) no longer affine to cpu1 <7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<3>INFO: RCU detected CPU 0 stall (t=1000 jiffies) <3>INFO: RCU detected CPU 0 stall (t=4000 jiffies) 0:mon>

After some debugging a possible suspect seems to be commit 6ad4c18.. : sched: Fix balance vs hotplug race

If i revert this patch i am able to execute the tests on this power6 without any issues.

But at the same time the above patch is required to solve the cpu hotplug related race on x86_64(as a side note this same x86_64 issue can be recreated against latest Linus git as well) that i reported here :

http://marc.info/?l=linux-kernel&m=125802682922299&w=2

Quoted 10 lines

I will try few more iterations with and without the above patch just to make sure i have the correct results.

If someone has a suggestion let me know.

Thanks
-Sachin

--
To unsubscribe from this list: send the line "unsubscribe linux-next" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
maintainer
#3 lore raw
In reply to Sachin Sant
Quoted 12 lines · Sachin Sant

While executing cpu_hotplug(from autotest) tests against latest next on a power6 box, the machine locks up. A soft reset shows the following trace
=20
cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current =3D 0xc00000000c173840
  paca    =3D 0xc000000000bc2600
    pid   =3D 2602, comm =3D hotplug06.top.s
Quoted 3 lines

enter ? for help
[link register ] c000000000342f10 .cpumask_next_and+0x4c/0x94
[c00000000c933650] c0000000000e9f34 .cpuset_cpus_allowed_locked+0x38/0x74=

(unreliable)

Quoted 22 lines

[c00000000c9336e0] c000000000090074 .move_task_off_dead_cpu+0xc4/0x1ac [c00000000c9337a0] c0000000005e4e5c .migration_call+0x304/0x830 [c00000000c933880] c0000000005e0880 .notifier_call_chain+0x68/0xe0 [c00000000c933920] c00000000012a92c ._cpu_down+0x210/0x34c [c00000000c933a90] c00000000012aad8 .cpu_down+0x70/0xa8 [c00000000c933b20] c000000000525940 .store_online+0x54/0x894 [c00000000c933bb0] c000000000463430 .sysdev_store+0x3c/0x50 [c00000000c933c20] c0000000001f8320 .sysfs_write_file+0x124/0x18c [c00000000c933ce0] c00000000017edac .vfs_write+0xd4/0x1fc [c00000000c933d80] c00000000017efdc .SyS_write+0x58/0xa0 [c00000000c933e30] c0000000000085b4 syscall_exit+0x0/0x40 --- Exception: c01 (System Call) at 00000fff9fa8a8f8
SP (fffe7aef200) is in userspace
0:mon> e
cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current =3D 0xc00000000c173840
  paca    =3D 0xc000000000bc2600
    pid   =3D 2602, comm =3D hotplug06.top.s
Quoted 6 lines

=20
Last few messages from the dmesg log shows

After some debugging a possible suspect seems to be commit 6ad4c18.. : sched: Fix balance vs hotplug race

Oh, wonderful :-/

So what is that thing whining about? Not being able to read a cpumask or something?

Does your .config have cpusets enabled (there's a different cpuset_cpus_allowed_locked implementation depending on that)?

I know of at least one remaining race and am working on closing that, but I'm not sure I can explain this crash with that.

#4 lore raw
Quoted 50 lines · Peter Zijlstra
Quoted 38 lines · Sachin Sant

While executing cpu_hotplug(from autotest) tests against latest next on a power6 box, the machine locks up. A soft reset shows the following trace

cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current = 0xc00000000c173840
  paca    = 0xc000000000bc2600
    pid   = 2602, comm = hotplug06.top.s

enter ? for help
[link register ] c000000000342f10 .cpumask_next_and+0x4c/0x94
[c00000000c933650] c0000000000e9f34 .cpuset_cpus_allowed_locked+0x38/0x74 (unreliable) [c00000000c9336e0] c000000000090074 .move_task_off_dead_cpu+0xc4/0x1ac [c00000000c9337a0] c0000000005e4e5c .migration_call+0x304/0x830 [c00000000c933880] c0000000005e0880 .notifier_call_chain+0x68/0xe0 [c00000000c933920] c00000000012a92c ._cpu_down+0x210/0x34c [c00000000c933a90] c00000000012aad8 .cpu_down+0x70/0xa8 [c00000000c933b20] c000000000525940 .store_online+0x54/0x894 [c00000000c933bb0] c000000000463430 .sysdev_store+0x3c/0x50 [c00000000c933c20] c0000000001f8320 .sysfs_write_file+0x124/0x18c [c00000000c933ce0] c00000000017edac .vfs_write+0xd4/0x1fc [c00000000c933d80] c00000000017efdc .SyS_write+0x58/0xa0 [c00000000c933e30] c0000000000085b4 syscall_exit+0x0/0x40
--- Exception: c01 (System Call) at 00000fff9fa8a8f8
SP (fffe7aef200) is in userspace
0:mon> e
cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current = 0xc00000000c173840
  paca    = 0xc000000000bc2600
    pid   = 2602, comm = hotplug06.top.s
Quoted 8 lines

Last few messages from the dmesg log shows

After some debugging a possible suspect seems to be commit 6ad4c18.. : sched: Fix balance vs hotplug race

Quoted 8 lines

Oh, wonderful :-/

So what is that thing whining about? Not being able to read a cpumask or something?

Does your .config have cpusets enabled (there's a different cpuset_cpus_allowed_locked implementation depending on that)?

Yes CPUSETS config is enabled. I have attached the config.

Thanks
-Sachin

Quoted 4 lines

I know of at least one remaining race and am working on closing that, but I'm not sure I can explain this crash with that.

Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://lists.ozlabs.org/listinfo/linuxppc-dev

maintainer
#5 lore raw
Quoted 7 lines · Sachin Sant
Quoted 6 lines · Peter Zijlstra

On Fri, 2009-12-11 at 16:23 +0530, Sachin Sant wrote:
=20

Quoted 4 lines

While executing cpu_hotplug(from autotest) tests against latest next on a power6 box, the machine locks up. A soft reset shows the following trace

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current =3D 0xc00000000c173840
  paca    =3D 0xc000000000bc2600
    pid   =3D 2602, comm =3D hotplug06.top.s

enter ? for help
[link register ] c000000000342f10 .cpumask_next_and+0x4c/0x94
[c00000000c933650] c0000000000e9f34 .cpuset_cpus_allowed_locked+0x38/0=

x74 (unreliable)

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

[c00000000c9336e0] c000000000090074 .move_task_off_dead_cpu+0xc4/0x1ac [c00000000c9337a0] c0000000005e4e5c .migration_call+0x304/0x830 [c00000000c933880] c0000000005e0880 .notifier_call_chain+0x68/0xe0 [c00000000c933920] c00000000012a92c ._cpu_down+0x210/0x34c [c00000000c933a90] c00000000012aad8 .cpu_down+0x70/0xa8 [c00000000c933b20] c000000000525940 .store_online+0x54/0x894 [c00000000c933bb0] c000000000463430 .sysdev_store+0x3c/0x50 [c00000000c933c20] c0000000001f8320 .sysfs_write_file+0x124/0x18c [c00000000c933ce0] c00000000017edac .vfs_write+0xd4/0x1fc [c00000000c933d80] c00000000017efdc .SyS_write+0x58/0xa0 [c00000000c933e30] c0000000000085b4 syscall_exit+0x0/0x40 --- Exception: c01 (System Call) at 00000fff9fa8a8f8
SP (fffe7aef200) is in userspace
0:mon> e
cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current =3D 0xc00000000c173840
  paca    =3D 0xc000000000bc2600
    pid   =3D 2602, comm =3D hotplug06.top.s

OK so how do I read that above thing? What's a System Reset? Is that like the x86 triple fault thing?

Quoted 1 line

From what I can make of it, its in move_task_off_dead_cpu(), right after

having called cpuset_cpus_allowed_locked(), doing that cpumask_any_and() call.

static void move_task_off_dead_cpu(int dead_cpu, struct task_struct *p) {
int dest_cpu;
const struct cpumask *nodemask =3D cpumask_of_node(cpu_to_node(dead=
_cpu));

again:

        /* Look for allowed, online CPU in same node. */
        for_each_cpu_and(dest_cpu, nodemask, cpu_active_mask)
                if (cpumask_test_cpu(dest_cpu, &p->cpus_allowed))
                        goto move;

        /* Any allowed, online CPU? */
        dest_cpu =3D cpumask_any_and(&p->cpus_allowed, cpu_active_mask);
        if (dest_cpu < nr_cpu_ids)
                goto move;

        /* No more Mr. Nice Guy. */
        if (dest_cpu >=3D nr_cpu_ids) {
                cpuset_cpus_allowed_locked(p, &p->cpus_allowed);

=3D=3D=3D=3D> dest_cpu =3D cpumask_any_and(cpu_active_mask, &p->c=
pus_allowed);

/*
* Don't tell them about moving exiting tasks or
* kernel threads (both mm NULL), since they never
* leave kernel.
*/
if (p->mm && printk_ratelimit()) {
pr_info("process %d (%s) no longer affine to cpu%d\=
n",

                                task_pid_nr(p), p->comm, dead_cpu);
                }
        }

move:

        /* It can have affinity changed while we were choosing. */
        if (unlikely(!__migrate_task_irq(p, dead_cpu, dest_cpu)))
                goto again;
}

Both masks, p->cpus_allowed and cpu_active_mask are stable in that p won't go away since we hold the tasklist_lock (in migrate_list_tasks), and cpu_active_mask is static storage, so WTH is it going funny on?

#6 lore raw
Quoted 35 lines · Peter Zijlstra
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current = 0xc00000000c173840
  paca    = 0xc000000000bc2600
    pid   = 2602, comm = hotplug06.top.s

enter ? for help
[link register ] c000000000342f10 .cpumask_next_and+0x4c/0x94
[c00000000c933650] c0000000000e9f34 .cpuset_cpus_allowed_locked+0x38/0x74 (unreliable) [c00000000c9336e0] c000000000090074 .move_task_off_dead_cpu+0xc4/0x1ac [c00000000c9337a0] c0000000005e4e5c .migration_call+0x304/0x830 [c00000000c933880] c0000000005e0880 .notifier_call_chain+0x68/0xe0 [c00000000c933920] c00000000012a92c ._cpu_down+0x210/0x34c [c00000000c933a90] c00000000012aad8 .cpu_down+0x70/0xa8 [c00000000c933b20] c000000000525940 .store_online+0x54/0x894 [c00000000c933bb0] c000000000463430 .sysdev_store+0x3c/0x50 [c00000000c933c20] c0000000001f8320 .sysfs_write_file+0x124/0x18c [c00000000c933ce0] c00000000017edac .vfs_write+0xd4/0x1fc [c00000000c933d80] c00000000017efdc .SyS_write+0x58/0xa0 [c00000000c933e30] c0000000000085b4 syscall_exit+0x0/0x40
--- Exception: c01 (System Call) at 00000fff9fa8a8f8
SP (fffe7aef200) is in userspace
0:mon> e
cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
    pc: c0000000003433d8: .find_next_bit+0x54/0xc4
    lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
    sp: c00000000c933650
   msr: 8000000000089032
  current = 0xc00000000c173840
  paca    = 0xc000000000bc2600
    pid   = 2602, comm = hotplug06.top.s
Quoted 2 lines

OK so how do I read that above thing? What's a System Reset? Is that like the x86 triple fault thing?

Nah, it's an NMI that throws you into xmon. Basically, the machine was hung and Sachin interrupted it with an NMI to see what was going on. The above is the backtrace. It was at the moment of the NMI inside find_next_bit() called from cpumask_next_and() etc...

Quoted 3 lines
Quoted 1 line

From what I can make of it, its in move_task_off_dead_cpu(), right after

having called cpuset_cpus_allowed_locked(), doing that cpumask_any_and() call.

Yes, it looks like it.

Quoted 33 lines

static void move_task_off_dead_cpu(int dead_cpu, struct task_struct *p)

{
        int dest_cpu;
        const struct cpumask *nodemask = cpumask_of_node(cpu_to_node(dead_cpu));

again:

        /* Look for allowed, online CPU in same node. */
        for_each_cpu_and(dest_cpu, nodemask, cpu_active_mask)
                if (cpumask_test_cpu(dest_cpu, &p->cpus_allowed))
                        goto move;

        /* Any allowed, online CPU? */
        dest_cpu = cpumask_any_and(&p->cpus_allowed, cpu_active_mask);
        if (dest_cpu < nr_cpu_ids)
                goto move;

        /* No more Mr. Nice Guy. */
        if (dest_cpu >= nr_cpu_ids) {
                cpuset_cpus_allowed_locked(p, &p->cpus_allowed);

====> dest_cpu = cpumask_any_and(cpu_active_mask, &p->cpus_allowed);

                /*
                 * Don't tell them about moving exiting tasks or
                 * kernel threads (both mm NULL), since they never
                 * leave kernel.
                 */
                if (p->mm && printk_ratelimit()) {
                        pr_info("process %d (%s) no longer affine to cpu%d\n",
                                task_pid_nr(p), p->comm, dead_cpu);
                }
        }

move:

Quoted 8 lines
        /* It can have affinity changed while we were choosing. */
        if (unlikely(!__migrate_task_irq(p, dead_cpu, dest_cpu)))
                goto again;
}

Both masks, p->cpus_allowed and cpu_active_mask are stable in that p won't go away since we hold the tasklist_lock (in migrate_list_tasks), and cpu_active_mask is static storage, so WTH is it going funny on?

Sachin, this is 100% reproduceable right ? You should be able to sprinkle it with some xmon_printf() (rather than printk, just add a prototype extern void xmon_printf(const char *fmt,...); somewhere, this has the advantage of being fully synchronous and will print out even if the printk sem is held.

Cheers,
Ben.

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#7 lore raw
Quoted 38 lines · Benjamin Herrenschmidt
Quoted 38 lines

static void move_task_off_dead_cpu(int dead_cpu, struct task_struct *p)

{
        int dest_cpu;
        const struct cpumask *nodemask = cpumask_of_node(cpu_to_node(dead_cpu));

again:

        /* Look for allowed, online CPU in same node. */
        for_each_cpu_and(dest_cpu, nodemask, cpu_active_mask)
                if (cpumask_test_cpu(dest_cpu, &p->cpus_allowed))
                        goto move;

        /* Any allowed, online CPU? */
        dest_cpu = cpumask_any_and(&p->cpus_allowed, cpu_active_mask);
        if (dest_cpu < nr_cpu_ids)
                goto move;

        /* No more Mr. Nice Guy. */
        if (dest_cpu >= nr_cpu_ids) {
                cpuset_cpus_allowed_locked(p, &p->cpus_allowed);

====> dest_cpu = cpumask_any_and(cpu_active_mask, &p->cpus_allowed);

                /*
                 * Don't tell them about moving exiting tasks or
                 * kernel threads (both mm NULL), since they never
                 * leave kernel.
                 */
                if (p->mm && printk_ratelimit()) {
                        pr_info("process %d (%s) no longer affine to cpu%d\n",
                                task_pid_nr(p), p->comm, dead_cpu);
                }
        }

move:

        /* It can have affinity changed while we were choosing. */
        if (unlikely(!__migrate_task_irq(p, dead_cpu, dest_cpu)))
                goto again;
}
Quoted 4 lines
Quoted 4 lines

Both masks, p->cpus_allowed and cpu_active_mask are stable in that p won't go away since we hold the tasklist_lock (in migrate_list_tasks), and cpu_active_mask is static storage, so WTH is it going funny on?

I added some debug statements within the above code. This is a 2 cpu machine.

XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2 XMON dest_cpu = 1024 XMON dest_cpu = 1024 . dead_cpu = 1 XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2 XMON dest_cpu = 1024 XMON dest_cpu = 1024 . dead_cpu = 1 XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2 XMON dest_cpu = 1024 XMON dest_cpu = 1024 . dead_cpu = 1

Seems to me that the control is stuck in an infinite loop and hence the machine appears to be in hung state. The dest_cpu value is always 1024 and never changes, which result in an infinite loop.

In working scenario the o/p is something on the following lines

XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2
XMON dest_cpu = 0 XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2 XMON dest_cpu = 0 XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2 XMON dest_cpu = 0

Let me know if i should try to record any specific value ?

Thanks
-Sachin

Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
maintainer
#8 lore raw
Quoted 2 lines · Sachin Sant
Quoted 1 line · Benjamin Herrenschmidt
Quoted 1 line

static void move_task_off_dead_cpu(int dead_cpu, struct task_struct *p=

)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

{
int dest_cpu;
const struct cpumask *nodemask =3D cpumask_of_node(cpu_to_node=

(dead_cpu));

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

again:

        /* Look for allowed, online CPU in same node. */
        for_each_cpu_and(dest_cpu, nodemask, cpu_active_mask)
                if (cpumask_test_cpu(dest_cpu, &p->cpus_allowed))
                        goto move;

        /* Any allowed, online CPU? */
        dest_cpu =3D cpumask_any_and(&p->cpus_allowed, cpu_active_mask=

);

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
        if (dest_cpu < nr_cpu_ids)
                goto move;

        /* No more Mr. Nice Guy. */
        if (dest_cpu >=3D nr_cpu_ids) {
                cpuset_cpus_allowed_locked(p, &p->cpus_allowed);

=3D=3D=3D=3D> dest_cpu =3D cpumask_any_and(cpu_active_mask, =

&p->cpus_allowed);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

/*
* Don't tell them about moving exiting tasks or
* kernel threads (both mm NULL), since they never
* leave kernel.
*/
if (p->mm && printk_ratelimit()) {
pr_info("process %d (%s) no longer affine to c=

pu%d\n",

Quoted 33 lines
Quoted 10 lines
Quoted 10 lines
                                task_pid_nr(p), p->comm, dead_cpu);
                }
        }

move:

        /* It can have affinity changed while we were choosing. */
        if (unlikely(!__migrate_task_irq(p, dead_cpu, dest_cpu)))
                goto again;
}
Quoted 4 lines
Quoted 4 lines

Both masks, p->cpus_allowed and cpu_active_mask are stable in that p won't go away since we hold the tasklist_lock (in migrate_list_tasks), and cpu_active_mask is static storage, so WTH is it going funny on?
=20

I added some debug statements within the above code.=20
This is a 2 cpu machine.
=20
XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_ids =3D 2 XMON dest_cpu =3D 1024=20
XMON dest_cpu =3D 1024 . dead_cpu =3D 1
XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_ids =3D 2 XMON dest_cpu =3D 1024=20
XMON dest_cpu =3D 1024 . dead_cpu =3D 1
XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_ids =3D 2 XMON dest_cpu =3D 1024=20
XMON dest_cpu =3D 1024 . dead_cpu =3D 1
=20
Seems to me that the control is stuck in an infinite loop and hence the machine appears to be in hung state. The dest_cpu value is always 1024 and never changes, which result in an infinite loop.
=20
In working scenario the o/p is something on the following lines =20

Quoted 8 lines

XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_ids =3D 2 XMON dest_cpu =3D 0=20
XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_ids =3D 2 XMON dest_cpu =3D 0=20
XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_ids =3D 2 XMON dest_cpu =3D 0=20
=20
Let me know if i should try to record any specific value ?

Could you possibly print the two masks themselves? cpumask_scnprintf() and friend come in handy for this.

The dest_cpu=3D1024 thing seem to suggest the intersection between p->cpus_allowed and cpu_active_mask is empty for some reason, even though we forcefully reset p->cpus_allowed to the full set using cpuset_cpus_allowed_locked().

/me goes re-read the cpu_active_map code, this really shouldn't happen.

#9 lore raw
Quoted 23 lines · Peter Zijlstra
Quoted 13 lines

I added some debug statements within the above code. This is a 2 cpu machine.

XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2 XMON dest_cpu = 1024 XMON dest_cpu = 1024 . dead_cpu = 1 XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2 XMON dest_cpu = 1024 XMON dest_cpu = 1024 . dead_cpu = 1 XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2 XMON dest_cpu = 1024 XMON dest_cpu = 1024 . dead_cpu = 1

Seems to me that the control is stuck in an infinite loop and hence the machine appears to be in hung state. The dest_cpu value is always 1024 and never changes, which result in an infinite loop.

Quoted 9 lines

In working scenario the o/p is something on the following lines

XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2
XMON dest_cpu = 0 XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2 XMON dest_cpu = 0 XMON dest_cpu = 1024 . dead_cpu = 1 . nr_cpu_ids = 2 XMON dest_cpu = 0

Let me know if i should try to record any specific value ?

Quoted 8 lines

Could you possibly print the two masks themselves? cpumask_scnprintf() and friend come in handy for this.

The dest_cpu=1024 thing seem to suggest the intersection between p->cpus_allowed and cpu_active_mask is empty for some reason, even though we forcefully reset p->cpus_allowed to the full set using cpuset_cpus_allowed_locked().

So here is the data related to the two masks.

cpu_active_mask = 00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000
XMON dest_cpu = 1024

while p->cpus_allowed = 00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000001
XMON dest_cpu = 1024

In working scenario the above data looks like

cpu_active_mask = 00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000002
XMON dest_cpu = 1

while p->cpus_allowed = 00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000002
XMON dest_cpu = 1

hope i got the data correct.

Thanks
-Sachin

Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
maintainer
#10 lore raw

Could you try the below?

---
 init/main.c |    7 +------
 1 files changed, 1 insertions(+), 6 deletions(-)
init/main.c +1 -6
index 4051d75..4be7de2 100644
@@ -369,12 +369,6 @@ static void __init smp_init(void)
 {
 	unsigned int cpu;
 
-	/*
-	 * Set up the current CPU as possible to migrate to.
-	 * The other ones will be done by cpu_up/cpu_down()
-	 */
-	set_cpu_active(smp_processor_id(), true);
-
 	/* FIXME: This should be done in userspace --RR */
 	for_each_present_cpu(cpu) {
 		if (num_online_cpus() >= setup_max_cpus)
@@ -486,6 +480,7 @@ static void __init boot_cpu_init(void)
 	int cpu = smp_processor_id();
 	/* Mark the boot cpu "present", "online" etc for SMP and UP case */
 	set_cpu_online(cpu, true);
+	set_cpu_active(cpu, true);
 	set_cpu_present(cpu, true);
 	set_cpu_possible(cpu, true);
 }
#11 lore raw
Quoted 2 lines · Peter Zijlstra

Could you try the below?

No luck. Still the same issue. The mask values don't change.

Thanks
-Sachin

Quoted 32 lines
---
 init/main.c |    7 +------
 1 files changed, 1 insertions(+), 6 deletions(-)
init/main.c +1 -6
index 4051d75..4be7de2 100644
@@ -369,12 +369,6 @@ static void __init smp_init(void)
 {
 	unsigned int cpu;
 
-	/*
-	 * Set up the current CPU as possible to migrate to.
-	 * The other ones will be done by cpu_up/cpu_down()
-	 */
-	set_cpu_active(smp_processor_id(), true);
-
 	/* FIXME: This should be done in userspace --RR */
 	for_each_present_cpu(cpu) {
 		if (num_online_cpus() >= setup_max_cpus)
@@ -486,6 +480,7 @@ static void __init boot_cpu_init(void)
 	int cpu = smp_processor_id();
 	/* Mark the boot cpu "present", "online" etc for SMP and UP case */
 	set_cpu_online(cpu, true);
+	set_cpu_active(cpu, true);
 	set_cpu_present(cpu, true);
 	set_cpu_possible(cpu, true);
 }
Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
maintainer
#12 lore raw
Quoted 4 lines · Sachin Sant
Quoted 2 lines · Peter Zijlstra

Could you try the below?

No luck. Still the same issue. The mask values don't change.

Bugger, that patch did solve a similar problem for a patch I'm working on.

Can you maybe add a print of the cpu_active_mask() in set_cpu_active() using WARN() so we can see where it changes the mask, and why it things its empty?

Quoted 34 lines
Quoted 32 lines
---
 init/main.c |    7 +------
 1 files changed, 1 insertions(+), 6 deletions(-)
init/main.c +1 -6
index 4051d75..4be7de2 100644
@@ -369,12 +369,6 @@ static void __init smp_init(void)
 {
 	unsigned int cpu;
 
-	/*
-	 * Set up the current CPU as possible to migrate to.
-	 * The other ones will be done by cpu_up/cpu_down()
-	 */
-	set_cpu_active(smp_processor_id(), true);
-
 	/* FIXME: This should be done in userspace --RR */
 	for_each_present_cpu(cpu) {
 		if (num_online_cpus() >= setup_max_cpus)
@@ -486,6 +480,7 @@ static void __init boot_cpu_init(void)
 	int cpu = smp_processor_id();
 	/* Mark the boot cpu "present", "online" etc for SMP and UP case */
 	set_cpu_online(cpu, true);
+	set_cpu_active(cpu, true);
 	set_cpu_present(cpu, true);
 	set_cpu_possible(cpu, true);
 }
#13 lore raw
In reply to Sachin Sant
Quoted 12 lines · Sachin Sant
Quoted 11 lines · Peter Zijlstra
Quoted 11 lines

I added some debug statements within the above code. This is a 2 cpu machine.

XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_ids =3D 2 XMON dest_cpu =3D 1024 XMON dest_cpu =3D 1024 . dead_cpu =3D 1 XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_ids =3D 2 XMON dest_cpu =3D 1024 XMON dest_cpu =3D 1024 . dead_cpu =3D 1 XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_ids =3D 2 XMON dest_cpu =3D 1024 XMON dest_cpu =3D 1024 . dead_cpu =3D 1

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Seems to me that the control is stuck in an infinite loop and hence the machine appears to be in hung state. The dest_cpu value is always 1024 and never changes, which result in an infinite loop.

In working scenario the o/p is something on the following lines

XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_ids =3D 2 XMON dest_cpu =3D 0 XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_id=

s =3D 2

Quoted 1 line
Quoted 1 line
Quoted 1 line

XMON dest_cpu =3D 0 XMON dest_cpu =3D 1024 . dead_cpu =3D 1 . nr_cpu_id=

s =3D 2

Quoted 13 lines
Quoted 4 lines
Quoted 3 lines

XMON dest_cpu =3D 0
Let me know if i should try to record any specific value ?

Quoted 8 lines

Could you possibly print the two masks themselves? cpumask_scnprintf() and friend come in handy for this.

The dest_cpu=3D1024 thing seem to suggest the intersection between p->cpus_allowed and cpu_active_mask is empty for some reason, even though we forcefully reset p->cpus_allowed to the full set using cpuset_cpus_allowed_locked().

Quoted 3 lines

So here is the data related to the two masks.

cpu_active_mask =3D 00000000,00000000,00000000,00000000,00000000,00000000=

,

Quoted 6 lines

00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000
XMON dest_cpu =3D 1024

How about cpu_online_mask? commit 6ad4c1 switches from cpu_online_mask to cpu_active_mask.
Is there a mismatch for cpu_online_mask and cpu_active_mask?

Quoted 1 line

while p->cpus_allowed =3D =C2=A000000000,00000000,00000000,00000000,00000=

000,

Quoted 9 lines

00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000001
XMON dest_cpu =3D 1024

In working scenario the above data looks like

cpu_active_mask =3D 00000000,00000000,00000000,00000000,00000000,00000000=

,

Quoted 7 lines

00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000002
XMON dest_cpu =3D 1

while p->cpus_allowed =3D =C2=A000000000,00000000,00000000,00000000,00000=

000,

Quoted 27 lines

00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000002
XMON dest_cpu =3D 1

hope i got the data correct.

Thanks
-Sachin

--

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

--
To unsubscribe from this list: send the line "unsubscribe linux-next" in the body of a message to [email protected]
More majordomo info at =C2=A0http://vger.kernel.org/majordomo-info.html

#14 lore raw
In reply to Sachin Sant
Quoted 30 lines · Sachin Sant

While executing cpu_hotplug(from autotest) tests against latest next on a power6 box, the machine locks up. A soft reset shows the following trace

cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]
  pc: c0000000003433d8: .find_next_bit+0x54/0xc4
  lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
  sp: c00000000c933650
 msr: 8000000000089032
 current = 0xc00000000c173840
 paca    = 0xc000000000bc2600
  pid   = 2602, comm = hotplug06.top.s
enter ? for help
[link register   ] c000000000342f10 .cpumask_next_and+0x4c/0x94 [c00000000c933650] c0000000000e9f34 .cpuset_cpus_allowed_locked+0x38/0x74 (unreliable)
[c00000000c9336e0] c000000000090074 .move_task_off_dead_cpu+0xc4/0x1ac [c00000000c9337a0] c0000000005e4e5c .migration_call+0x304/0x830 [c00000000c933880] c0000000005e0880 .notifier_call_chain+0x68/0xe0 [c00000000c933920] c00000000012a92c ._cpu_down+0x210/0x34c [c00000000c933a90] c00000000012aad8 .cpu_down+0x70/0xa8 [c00000000c933b20] c000000000525940 .store_online+0x54/0x894 [c00000000c933bb0] c000000000463430 .sysdev_store+0x3c/0x50 [c00000000c933c20] c0000000001f8320 .sysfs_write_file+0x124/0x18c [c00000000c933ce0] c00000000017edac .vfs_write+0xd4/0x1fc [c00000000c933d80] c00000000017efdc .SyS_write+0x58/0xa0 [c00000000c933e30] c0000000000085b4 syscall_exit+0x0/0x40
--- Exception: c01 (System Call) at 00000fff9fa8a8f8
SP (fffe7aef200) is in userspace
0:mon> e

Quoted 9 lines

cpu 0x0: Vector: 100 (System Reset) at [c00000000c9333d0]
  pc: c0000000003433d8: .find_next_bit+0x54/0xc4
  lr: c000000000342f10: .cpumask_next_and+0x4c/0x94
  sp: c00000000c933650
 msr: 8000000000089032
 current = 0xc00000000c173840
 paca    = 0xc000000000bc2600
  pid   = 2602, comm = hotplug06.top.s

Does this testcase hotplug cpu 0 off?

Quoted 70 lines

Last few messages from the dmesg log shows

0:mon> <4>IRQ 17 affinity broken off cpu 0
<4>IRQ 18 affinity broken off cpu 0
<4>IRQ 19 affinity broken off cpu 0
<4>IRQ 264 affinity broken off cpu 0
<4>cpu 0 (hwid 0) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[0] <4>Processor 0 found.
<4>IRQ 17 affinity broken off cpu 1
<4>IRQ 18 affinity broken off cpu 1
<4>IRQ 19 affinity broken off cpu 1
<4>IRQ 264 affinity broken off cpu 1
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<6>process 2423 (bash) no longer affine to cpu1 <7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[1] <4>Processor 1 found.
<4>cpu 1 (hwid 1) Ready to die...
<3>INFO: RCU detected CPU 0 stall (t=1000 jiffies) <3>INFO: RCU detected CPU 0 stall (t=4000 jiffies) 0:mon>

After some debugging a possible suspect seems to be commit 6ad4c18.. : sched: Fix balance vs hotplug race

If i revert this patch i am able to execute the tests on this power6 without any issues.
But at the same time the above patch is required to solve the cpu hotplug related race on x86_64(as a side note this same x86_64 issue can be recreated against latest Linus git as well) that i reported here :

http://marc.info/?l=linux-kernel&m=125802682922299&w=2

I will try few more iterations with and without the above patch just to make sure i have the correct results.

If someone has a suggestion let me know.

Thanks
-Sachin

--

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

#15 lore raw
Quoted 2 lines · Xiaotian Feng

Does this testcase hotplug cpu 0 off?

No, i don't think so. It skips cpu0 during online/offline process.

thanks
-Sachin

Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
#16 lore raw
Quoted 7 lines · Sachin Sant
Quoted 3 lines · Xiaotian Feng

Does this testcase hotplug cpu 0 off?

No, i don't think so. It skips cpu0 during online/offline process.

Then how could this happen ? Looks like cpu 0 is offline .... 0:mon> <4>IRQ 17 affinity broken off cpu 0
<4>IRQ 18 affinity broken off cpu 0
<4>IRQ 19 affinity broken off cpu 0
<4>IRQ 264 affinity broken off cpu 0
<4>cpu 0 (hwid 0) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[0]

Quoted 14 lines

thanks
-Sachin

--

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

#17 lore raw
Quoted 11 lines · Xiaotian Feng
Quoted 8 lines · Sachin Sant
Quoted 3 lines · Xiaotian Feng

Does this testcase hotplug cpu 0 off?

No, i don't think so. It skips cpu0 during online/offline process.

Quoted 8 lines

Then how could this happen ? Looks like cpu 0 is offline .... 0:mon> <4>IRQ 17 affinity broken off cpu 0
<4>IRQ 18 affinity broken off cpu 0
<4>IRQ 19 affinity broken off cpu 0
<4>IRQ 264 affinity broken off cpu 0
<4>cpu 0 (hwid 0) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[0]

Sorry i was looking at only one script. Looking more closely at the test there are 6 different sub tests. The rest of the tests do seem to hotplug CPU 0.

Thanks
-Sachin

Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
maintainer
#18 lore raw
Quoted 20 lines · Sachin Sant
Quoted 11 lines · Xiaotian Feng
Quoted 8 lines · Sachin Sant
Quoted 3 lines · Xiaotian Feng

Does this testcase hotplug cpu 0 off?

No, i don't think so. It skips cpu0 during online/offline process.

Quoted 8 lines

Then how could this happen ? Looks like cpu 0 is offline .... 0:mon> <4>IRQ 17 affinity broken off cpu 0
<4>IRQ 18 affinity broken off cpu 0
<4>IRQ 19 affinity broken off cpu 0
<4>IRQ 264 affinity broken off cpu 0
<4>cpu 0 (hwid 0) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[0]

Quoted 3 lines

Sorry i was looking at only one script. Looking more closely at the test there are 6 different sub tests. The rest of the tests do seem to hotplug CPU 0.

Ooh, cute, so you can actually hotplug cpu 0.. no wonder that didn't get exposed on x86.

Still, the only time cpu_active_mask should not be equal to cpu_online_mask is when we're in the middle of a hotplug, we clear active early and set it late, but its all done under the hotplug mutex, so we can at most have 1 cpu differences with online mask.

Unless of course, I messed up, which appears to be rather likely given these problems ;-)

#19 lore raw
Quoted 25 lines · Peter Zijlstra
Quoted 20 lines · Sachin Sant
Quoted 11 lines · Xiaotian Feng
Quoted 8 lines · Sachin Sant
Quoted 3 lines · Xiaotian Feng

Does this testcase hotplug cpu 0 off?

No, i don't think so. It skips cpu0 during online/offline process.

Quoted 8 lines

Then how could this happen ? Looks like cpu 0 is offline .... 0:mon> <4>IRQ 17 affinity broken off cpu 0
<4>IRQ 18 affinity broken off cpu 0
<4>IRQ 19 affinity broken off cpu 0
<4>IRQ 264 affinity broken off cpu 0
<4>cpu 0 (hwid 0) Ready to die...
<7>clockevent: decrementer mult[83126e97] shift[32] cpu[0]

Quoted 3 lines

Sorry i was looking at only one script. Looking more closely at the test there are 6 different sub tests. The rest of the tests do seem to hotplug CPU 0.

Quoted 8 lines

Ooh, cute, so you can actually hotplug cpu 0.. no wonder that didn't get exposed on x86.

Still, the only time cpu_active_mask should not be equal to cpu_online_mask is when we're in the middle of a hotplug, we clear active early and set it late, but its all done under the hotplug mutex, so we can at most have 1 cpu differences with online mask.

Could follow be possible? We know there's cpu 0 and cpu 1,

offline cpu1 > done
offline cpu0 > false

consider this in cpu_down code,

int __ref cpu_down(unsigned int cpu)
{
<snip>

        set_cpu_active(cpu, false); // here, we set cpu 0 to inactive

        synchronize_sched();

        err = _cpu_down(cpu, 0);

out:
<snip>
}

Then in _cpu_down code:

static int __ref _cpu_down(unsigned int cpu, int tasks_frozen) {
<snip>

        if (num_online_cpus() == 1)        // if we're trying to

offline cpu0, num_online_cpus will be 1
return -EBUSY; // after return back
to cpu_down, we didn't change cpu 0 back to active

        if (!cpu_online(cpu))
                return -EINVAL;

        if (!alloc_cpumask_var(&old_allowed, GFP_KERNEL))
                return -ENOMEM;

<snip>
}

Then cpu 0 is not active, but online, then we try to offline cpu1, ....... This can not be exposed because x86 does not have /sys/devices/system/cpu0/online.
I guess following patch fixes this bug.

---

kernel/cpu.c +0 -0
index 291ac58..21ddace 100644
@@ -199,14 +199,18 @@ static int __ref _cpu_down(unsigned int cpu, int

tasks_frozen)
.hcpu = hcpu,
};

- if (num_online_cpus() == 1)
+ if (num_online_cpus() == 1) {

+               set_cpu_active(cpu, true);
                return -EBUSY;

+ }

        if (!cpu_online(cpu))
                return -EINVAL;

- if (!alloc_cpumask_var(&old_allowed, GFP_KERNEL))
+ if (!alloc_cpumask_var(&old_allowed, GFP_KERNEL)) {

+               set_cpu_active(cpu, true);
                return -ENOMEM;

+ }

cpu_hotplug_begin();
err = __raw_notifier_call_chain(&cpu_chain, CPU_DOWN_PREPARE | mod,

Quoted 4 lines

Unless of course, I messed up, which appears to be rather likely given these problems ;-)

#20 lore raw
Quoted 35 lines · Xiaotian Feng

Could follow be possible? We know there's cpu 0 and cpu 1,

offline cpu1 > done
offline cpu0 > false

consider this in cpu_down code,

int __ref cpu_down(unsigned int cpu)
{
<snip>

        set_cpu_active(cpu, false); // here, we set cpu 0 to inactive

        synchronize_sched();

        err = _cpu_down(cpu, 0);

out:
<snip>
}

Then in _cpu_down code:

static int __ref _cpu_down(unsigned int cpu, int tasks_frozen) {
<snip>

        if (num_online_cpus() == 1)        // if we're trying to

offline cpu0, num_online_cpus will be 1
return -EBUSY; // after return back
to cpu_down, we didn't change cpu 0 back to active

        if (!cpu_online(cpu))
                return -EINVAL;

        if (!alloc_cpumask_var(&old_allowed, GFP_KERNEL))
                return -ENOMEM;
Quoted 8 lines

<snip>
}

Then cpu 0 is not active, but online, then we try to offline cpu1, ....... This can not be exposed because x86 does not have /sys/devices/system/cpu0/online.
I guess following patch fixes this bug.

Just tested this one on the POWER box and the test passed. I did not observe the hang.

Thanks
-Sachin

Quoted 37 lines

---

kernel/cpu.c +6 -2
index 291ac58..21ddace 100644
@@ -199,14 +199,18 @@ static int __ref _cpu_down(unsigned int cpu, int
 tasks_frozen)
                .hcpu = hcpu,
        };
 
-       if (num_online_cpus() == 1)
+       if (num_online_cpus() == 1) {
+               set_cpu_active(cpu, true);
                return -EBUSY;
+       }
 
        if (!cpu_online(cpu))
                return -EINVAL;
 
-       if (!alloc_cpumask_var(&old_allowed, GFP_KERNEL))
+       if (!alloc_cpumask_var(&old_allowed, GFP_KERNEL)) {
+               set_cpu_active(cpu, true);
                return -ENOMEM;
+       }
 
        cpu_hotplug_begin();

err = __raw_notifier_call_chain(&cpu_chain, CPU_DOWN_PREPARE | mod,

Quoted 5 lines

Unless of course, I messed up, which appears to be rather likely given these problems ;-)

Signature · 6 lines
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
#21 lore raw
Quoted 40 lines · Sachin Sant
Quoted 39 lines · Xiaotian Feng

Could follow be possible?  We know there's cpu 0 and cpu 1,

offline cpu1 > done
offline cpu0 > false

consider this in cpu_down code,

int __ref cpu_down(unsigned int cpu)
{
<snip>
       set_cpu_active(cpu, false); // here, we set cpu 0 to inactive

       synchronize_sched();

       err = _cpu_down(cpu, 0);
out:
<snip>
}

Then in _cpu_down code:

static int __ref _cpu_down(unsigned int cpu, int tasks_frozen) {
<snip>
       if (num_online_cpus() == 1)        // if we're trying to offline cpu0, num_online_cpus will be 1
               return -EBUSY;                    // after return back to cpu_down, we didn't change cpu 0 back to active

       if (!cpu_online(cpu))
               return -EINVAL;

       if (!alloc_cpumask_var(&old_allowed, GFP_KERNEL)) return -ENOMEM;
<snip>
}

Quoted 8 lines
Quoted 5 lines

Then cpu 0 is not active, but online, then we try to offline cpu1, ....... This can not be exposed because x86 does not have /sys/devices/system/cpu0/online.
I guess following patch fixes this bug.

Just tested this one on the POWER box and the test passed. I did not observe the hang.

Thanks for confirm, I will send formatted patch to upstream then:-)

Quoted 44 lines

Thanks
-Sachin

Quoted 26 lines

---

kernel/cpu.c +6 -2
index 291ac58..21ddace 100644
@@ -199,14 +199,18 @@ static int __ref _cpu_down(unsigned int cpu, int
 tasks_frozen)
                .hcpu = hcpu,
        };
 
-       if (num_online_cpus() == 1)
+       if (num_online_cpus() == 1) {
+               set_cpu_active(cpu, true);
                return -EBUSY;
+       }
 
        if (!cpu_online(cpu))
                return -EINVAL;
 
-       if (!alloc_cpumask_var(&old_allowed, GFP_KERNEL))
+       if (!alloc_cpumask_var(&old_allowed, GFP_KERNEL)) {
+               set_cpu_active(cpu, true);
                return -ENOMEM;
+       }
 
        cpu_hotplug_begin();
Quoted 12 lines

       err = __raw_notifier_call_chain(&cpu_chain, CPU_DOWN_PREPARE | mod,

Quoted 6 lines

Unless of course, I messed up, which appears to be rather likely given these problems ;-)

Quoted 10 lines

--

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://lists.ozlabs.org/listinfo/linuxppc-dev

#22 lore raw

PATCH 1/1 fix cpu hotplug test failures on powerpc

In reply to Sachin Sant

Sachin found cpu hotplug test failures on powerpc, which made kernel hangs on his POWER box. This is addressed in
http://marc.info/?l=linux-kernel&m=126052886204649&w=2

commit 6ad4c18(sched: Fix balance vs hotplug race), switches to cpu_active_mask, but at some specific situation, kernel may cause some cpu inactive but online.

In some powerpc machine, hotplug cpu0 is allowed. If cpu0 is the last alive cpu, when we tried to offline cpu0, we'll inactive cpu0 in cpu_down(), after goes into __cpu_down(), kernel found num_online_cpus is 1, returned -EBUSY but cpu0 is not changed back to active. So cpu0 is inactive but online.

The fix is to set cpu inactive when we're going to bring down the specific cpu in _cpu_down().

Cc Peter Zijlstra <[email protected]>
Cc Rusty Russell <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Heiko Carstens <[email protected]>
---
 kernel/cpu.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)
kernel/cpu.c +1 -8
index 291ac58..a1e7165 100644
@@ -209,6 +209,7 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen)
 		return -ENOMEM;
 
 	cpu_hotplug_begin();
+	set_cpu_active(cpu, false);
 	err = __raw_notifier_call_chain(&cpu_chain, CPU_DOWN_PREPARE | mod,
 					hcpu, -1, &nr_calls);
 	if (err == NOTIFY_BAD) {
@@ -280,8 +281,6 @@ int __ref cpu_down(unsigned int cpu)
 		goto out;
 	}
 
-	set_cpu_active(cpu, false);
-
 	/*
 	 * Make sure the all cpus did the reschedule and are not
 	 * using stale version of the cpu_active_mask.
@@ -387,12 +386,6 @@ int disable_nonboot_cpus(void)
 	 */
 	cpumask_clear(frozen_cpus);
 
-	for_each_online_cpu(cpu) {
-		if (cpu == first_cpu)
-			continue;
-		set_cpu_active(cpu, false);
-	}
-
 	synchronize_sched();
 
 	printk("Disabling non-boot CPUs ...\n");
maintainer
#23 lore raw

Re: [PATCH] fix cpu hotplug test failures on powerpc

Quoted 8 lines · Xiaotian Feng

Sachin found cpu hotplug test failures on powerpc, which made kernel hangs on his POWER box. This is addressed in
http://marc.info/?l=linux-kernel&m=126052886204649&w=2

commit 6ad4c18(sched: Fix balance vs hotplug race), switches to cpu_active_mask, but at some specific situation, kernel may cause some cpu inactive but online.

Quoted 8 lines

In some powerpc machine, hotplug cpu0 is allowed. If cpu0 is the last alive cpu, when we tried to offline cpu0, we'll inactive cpu0 in cpu_down(), after goes into __cpu_down(), kernel found num_online_cpus is 1, returned -EBUSY but cpu0 is not changed back to active. So cpu0 is inactive but online.

The fix is to set cpu inactive when we're going to bring down the specific cpu in _cpu_down().

Good spotting, thanks! Some comments below.

Quoted 25 lines
Cc Peter Zijlstra <[email protected]>
Cc Rusty Russell <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Heiko Carstens <[email protected]>
---
 kernel/cpu.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)
kernel/cpu.c +1 -0
index 291ac58..a1e7165 100644
@@ -209,6 +209,7 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen)
 		return -ENOMEM;
 
 	cpu_hotplug_begin();
+	set_cpu_active(cpu, false);
 	err = __raw_notifier_call_chain(&cpu_chain, CPU_DOWN_PREPARE | mod,
 					hcpu, -1, &nr_calls);
 	if (err == NOTIFY_BAD) {

kernel/cpu.c int __ref cpu_down(unsigned int cpu) · line 286

Quoted 8 lines
kernel/cpu.c +0 -2
@@ -280,8 +281,6 @@ int __ref cpu_down(unsigned int cpu)
 		goto out;
 	}
 
-	set_cpu_active(cpu, false);
-
 	/*
 	 * Make sure the all cpus did the reschedule and are not
 	 * using stale version of the cpu_active_mask.

That renders the synchronize_sched() call down there useless, so might as well remove it then.

kernel/cpu.c int disable_nonboot_cpus(void) · line 389

Quoted 11 lines
kernel/cpu.c +0 -6
@@ -387,12 +386,6 @@ int disable_nonboot_cpus(void)
 	 */
 	cpumask_clear(frozen_cpus);
 
-	for_each_online_cpu(cpu) {
-		if (cpu == first_cpu)
-			continue;
-		set_cpu_active(cpu, false);
-	}
-
 	synchronize_sched();

And here too.

Quoted 1 line
kernel/cpu.c +0 -0
 	printk("Disabling non-boot CPUs ...\n");