Skip to content

RFC arm64: initial CPU_HOTPLUG support

Hanjun Guo linux-arm-kernel 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest0 of 5 patches 0 repliesNew

Patches in v1 No applies recorded yet

None of the 5 patches the cover letter announces are in this archive.

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/5] arm64: initial CPU_HOTPLUG support

Hi Mark,

I tested this patch set on the armv8 foundation model, it's panic.

I seems that we need to do something more, I'll also checkout what's going on here.

Thanks
Hanjun

dump formation:
root at genericarmv8:/sys/devices/system/cpu/cpu3# echo 0 > online
CPU3: Booted secondary processor
CPU3: shutdown

BUG: failure at kernel/time/clockevents.c:284/clockevents_register_device()!
Kernel panic - not syncing: BUG!
CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.10.0+ #2
Call trace:
[<ffffffc0000873e0>] dump_backtrace+0x0/0x12c
[<ffffffc000087520>] show_stack+0x14/0x1c
[<ffffffc0003f2e7c>] dump_stack+0x20/0x28
[<ffffffc0003efd24>] panic+0xe8/0x214
[<ffffffc0000d049c>] clockevents_set_mode+0x0/0x6c
[<ffffffc0000d074c>] clockevents_config_and_register+0x24/0x30
[<ffffffc0003ef840>] arch_timer_setup+0xd8/0x140
[<ffffffc0003ef8f0>] arch_timer_cpu_notify+0x48/0xc8
[<ffffffc0000b83e4>] notifier_call_chain+0x48/0x88
[<ffffffc0000b8500>] __raw_notifier_call_chain+0xc/0x14
[<ffffffc0000973f0>] __cpu_notify+0x30/0x58
[<ffffffc00009742c>] cpu_notify+0x14/0x1c
[<ffffffc0003ed46c>] notify_cpu_starting+0x14/0x1c
[<ffffffc0003ece74>] secondary_start_kernel+0xc0/0xf4
CPU2: stopping
CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.10.0+ #2
Call trace:
[<ffffffc0000873e0>] dump_backtrace+0x0/0x12c
[<ffffffc000087520>] show_stack+0x14/0x1c
[<ffffffc0003f2e7c>] dump_stack+0x20/0x28
[<ffffffc00008da8c>] handle_IPI+0x120/0x130
[<ffffffc0000812e0>] gic_handle_irq+0x7c/0x80
Exception stack(0xffffffc87fc8de30 to 0xffffffc87fc8df50)

de20: 7fc8c000 ffffffc8 005a1c8b ffffffc0
de40: 7fc8df70 ffffffc8 00084540 ffffffc0 00000e90 00000000 00000000 00000000
de60: 3ffbb9ec ffffffc0 00010000 00000000 00000020 00000000 00000000 00000000
de80: 3ffbb7f8 ffffffc0 00000000 00000000 7fc5ddd0 ffffffc8 7fc8dd80 ffffffc8
dea0: 003fb8e4 ffffffc0 03ee8b1d 00000000 003fb8e8 ffffffc0 00000000 00000000
dec0: 1f7458e8 ffffffbc 00001000 00000000 00000001 00000000 00002000 00000000
dee0: 00000000 00000000 7fc8c000 ffffffc8 005a1c8b ffffffc0 00000001 00000000
df00: 004dc3f0 ffffffc0 005adf00 ffffffc0 003faca0 ffffffc0 8007b000 00000000
df20: 8007d000 00000000 000801a0 ffffffc0 80080188 00000000 7fc8df70 ffffffc8
df40: 0008453c ffffffc0 7fc8df70 ffffffc8

[<ffffffc0000835ac>] el1_irq+0x6c/0xc0
[<ffffffc0000ca4f4>] cpu_startup_entry+0xf0/0x138
[<ffffffc0003ece9c>] secondary_start_kernel+0xe8/0xf4
CPU1: stopping
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.0+ #2
Call trace:
[<ffffffc0000873e0>] dump_backtrace+0x0/0x12c
[<ffffffc000087520>] show_stack+0x14/0x1c
[<ffffffc0003f2e7c>] dump_stack+0x20/0x28
[<ffffffc00008da8c>] handle_IPI+0x120/0x130
[<ffffffc0000812e0>] gic_handle_irq+0x7c/0x80
Exception stack(0xffffffc87fc8be30 to 0xffffffc87fc8bf50)

be20: 7fc8a000 ffffffc8 005a1c8b ffffffc0
be40: 7fc8bf70 ffffffc8 00084540 ffffffc0 00002088 00000000 00000000 00000000
be60: 3ffb19ec ffffffc0 00010000 00000000 00000010 00000000 0fcf5932 0000274b
be80: aba36180 0000008d 7f7bd8b0 ffffffc8 7fc5d4d0 ffffffc8 7fc8bd80 ffffffc8
bea0: 00004d30 00000001 00000020 00000000 003fb8e8 ffffffc0 ae892f33 ffffffff
bec0: 46000000 0021a0f7 00000000 003b9aca 0016dae4 ffffffc0 94294800 0000007f
bee0: f0587350 0000007f 7fc8a000 ffffffc8 005a1c8b ffffffc0 00000001 00000000
bf00: 004dc3f0 ffffffc0 005adf00 ffffffc0 003faca0 ffffffc0 8007b000 00000000
bf20: 8007d000 00000000 000801a0 ffffffc0 80080188 00000000 7fc8bf70 ffffffc8
bf40: 0008453c ffffffc0 7fc8bf70 ffffffc8

[<ffffffc0000835ac>] el1_irq+0x6c/0xc0
[<ffffffc0000ca4f4>] cpu_startup_entry+0xf0/0x138
[<ffffffc0003ece9c>] secondary_start_kernel+0xe8/0xf4
CPU: 0 PID: 957 Comm: sh Not tainted 3.10.0+ #2
Call trace:
[<ffffffc0000873e0>] dump_backtrace+0x0/0x12c
[<ffffffc000087520>] show_stack+0x14/0x1c
[<ffffffc0003f2e7c>] dump_stack+0x20/0x28
[<ffffffc00008da8c>] handle_IPI+0x120/0x130
[<ffffffc0000812e0>] gic_handle_irq+0x7c/0x80
Exception stack(0xffffffc87e1e7a70 to 0xffffffc87e1e7b90)

7a60: 00000007 00000000 0057b000 ffffffc0
7a80: 7e1e7bb0 ffffffc8 00096044 ffffffc0 00000000 00000000 3fa41000 00000000
7aa0: 00000000 00000000 3ffa73d8 ffffffc0 00000000 00000000 00000000 00000000
7ac0: 005b1000 ffffffc0 7e1e6000 ffffffc8 00000002 00000000 0057b000 ffffffc0
7ae0: 00000001 00000000 00000030 00000000 000012b0 00000000 00000000 00000000
7b00: 000000a6 00000000 00000006 00000000 005a8000 ffffffc0 b62e09b8 0000007f
7b20: c32ba320 0000007f 00000007 00000000 0057b000 ffffffc0 00000005 00000000
7b40: 005a4758 ffffffc0 00000140 00000000 0000000e 00000000 00000000 00000000
7b60: 00000000 00000000 0000000e 00000000 00000006 00000000 7e1e7bb0 ffffffc8
7b80: 000960f8 ffffffc0 7e1e7bb0 ffffffc8

[<ffffffc0000835ac>] el1_irq+0x6c/0xc0
[<ffffffc0003efec4>] printk+0x74/0x7c
[<ffffffc0003ecf68>] __cpu_die+0x50/0x8c
[<ffffffc0003eacd4>] _cpu_down.constprop.2+0xec/0x280
[<ffffffc0003eae8c>] cpu_down+0x24/0x40
[<ffffffc0003ebca4>] store_online+0x40/0xa4
[<ffffffc0002f1bf4>] dev_attr_store+0x18/0x28
[<ffffffc00018c0ec>] sysfs_write_file+0xdc/0x154
[<ffffffc00012fefc>] vfs_write+0xac/0x1a4
[<ffffffc00013032c>] SyS_write+0x44/0x8c
Quoted 62 lines · Mark Rutland

The following patches add basic CPU_HOTPLUG support to arm64, which combined with appropriate firmware (e.g. [1]) can be used to power CPUs up and down dynamically. From discussions at connect it seemed that several people were interested in working in this area, so I thought I'd make my current implementation public now that I've managed to regain access to my inbox.

I've tested this series with the bootwrapper PSCI implementation I've placed on linux-arm.org [1] and a modified foundation model dts with a psci node and each CPU's enable-method set to "psci", using a shell while repeatedly cycling all cpus off and on:

for C in $(seq 0 3); do

	./cyclichotplug.sh $C >/dev/null 2>&1 &

done

---->8----
#!/bin/sh

# cyclichotplug.sh

CPU=$1;

if [ -z "$CPU" ]; then

	printf "Usage: $0 <cpu id>\n";
	exit 1;

fi

ONLINEFILE=/sys/devices/system/cpu/cpu$CPU/online;

while true; do

	echo 0 > $ONLINEFILE;
	echo 1 > $ONLINEFILE;

done
---->8----

Patches are based on v3.10.

Thanks,
Mark.

[1] http://linux-arm.org/git?p=boot-wrapper-aarch64.git;a=shortlog;h=refs/tags/simple-psci

Mark Rutland (5):
arm64: reorganise smp_enable_ops
arm64: factor out spin-table boot method
arm64: read enable-method for CPU0
arm64: add CPU_HOTPLUG infrastructure
arm64: add PSCI CPU_OFF-based hotplug support

  arch/arm64/Kconfig                 |    7 ++
  arch/arm64/include/asm/irq.h       |    1 +
  arch/arm64/include/asm/smp.h       |   37 +++++--
  arch/arm64/kernel/cputable.c       |    2 +-
  arch/arm64/kernel/head.S           |   12 +-
  arch/arm64/kernel/irq.c            |   61 ++++++++++
  arch/arm64/kernel/process.c        |    7 ++
  arch/arm64/kernel/smp.c            |  215 ++++++++++++++++++++++--------------
  arch/arm64/kernel/smp_psci.c       |   54 +++++++--
  arch/arm64/kernel/smp_spin_table.c |   85 +++++++++++++-
  arch/arm64/kernel/vmlinux.lds.S    |    1 -
  11 files changed, 375 insertions(+), 107 deletions(-)

Read the cover letter in its thread