Fixing CPU Hotplug for RealView Platforms
Hello,
Currently, CPU hotplug is broken for RealView platforms. I posted some patches previously to try and address this, but they didn't solve the problems fully:
http://lists.infradead.org/pipermail/linux-arm-kernel/2010-September/026157.html
I'm now revisiting the code and it looks like the main problem is when we wish to *leave* the lowpower state. The enter/leave routines look like this:
static inline void cpu_enter_lowpower(void)
{
unsigned int v, smp_ctrl = get_smp_ctrl_mask();
flush_cache_all();
dsb();
asm volatile(
/*
* Turn off coherency
*/
" mrc p15, 0, %0, c1, c0, 1\n"
" bic %0, %0, %1\n"
" mcr p15, 0, %0, c1, c0, 1\n"
/* ISB */
" mcr p15, 0, %2, c7, c5, 4\n"
/* Disable D-cache */
" mrc p15, 0, %0, c1, c0, 0\n"
" bic %0, %0, #0x04\n"
" mcr p15, 0, %0, c1, c0, 0\n"
: "=&r" (v)
: "r" (smp_ctrl), "r" (0)
: "memory");
isb();
}static inline void cpu_leave_lowpower(void)
{
unsigned int v, smp_ctrl = get_smp_ctrl_mask();
asm volatile( "mrc p15, 0, %0, c1, c0, 0\n"
" orr %0, %0, #0x04\n"
" mcr p15, 0, %0, c1, c0, 0\n"
" mrc p15, 0, %0, c1, c0, 1\n"
" orr %0, %0, %1\n"
" mcr p15, 0, %0, c1, c0, 1\n"
: "=&r" (v)
: "r" (smp_ctrl)
: "memory");
isb();
}The problem is that by turning off coherency, the contents of the D-cache becomes stale. If data is prefetched into L1 between the flush_cache_all invocation and disabling the D-cache then this data will still be present when we come out of lowpower. Without coherency, we *must not* use this data and so a D-cache invalidation to the PoC is required in cpu_leave_lowpower().
On v6 this is a simple mcr instruction. On v7, we have to perform a set/way operation across all ways of each cache until we reach the PoC (see the scary but well commented v7_flush_dcache_all function). Implementing this means extending the cpu_cache_fns struct and stubbing out the new function for other caches, so I'd like to see if anybody has any better ideas before I go ahead and make these changes.
One possibility is not to turn off coherency, but if platform_do_lowpower is more than a WFI I don't think this would be suitable.
Any thoughts?
Will
Quoted 12 lines · Will Deacon
Hello,
Currently, CPU hotplug is broken for RealView platforms. I posted some patches previously to try and address this, but they didn't solve the problems fully:
http://lists.infradead.org/pipermail/linux-arm-kernel/2010-September/026157.html
I'm now revisiting the code and it looks like the main problem is when we wish to *leave* the lowpower state. The enter/leave routines look like this:
...
Quoted 7 lines
The problem is that by turning off coherency, the contents of the D-cache becomes stale. If data is prefetched into L1 between the
flush_cache_allinvocation and disabling the D-cache then this data will still be present when we come out of lowpower. Without coherency, we *must not* use this data and so a D-cache invalidation to the PoC is required incpu_leave_lowpower().
What if we fixed the cpu_reset functions for v6 and v7, and when a CPU is taken offline, we actually go through a proper shutdown of that CPU and call the reset vector, re-entering the boot loader?
We can only do this for CPUs other than the original boot CPU, because the boot loader should be checking which are the secondary CPUs and putting those into this simple WFI loop with the GIC appropriately programmed.
This means when we re-activate the CPU, we'll be waking it up in exactly the same way as we do when the kernel boots - and we have all that code around just waiting to be used.
Hi Russell,
Quoted 22 lines · Will Deacon
Quoted 12 lines
Hello,
Currently, CPU hotplug is broken for RealView platforms. I posted some patches previously to try and address this, but they didn't solve the problems fully:
http://lists.infradead.org/pipermail/linux-arm-kernel/2010-September/026157.html
I'm now revisiting the code and it looks like the main problem is when we wish to *leave* the lowpower state. The enter/leave routines look like this:
...
Quoted 7 lines
The problem is that by turning off coherency, the contents of the D-cache becomes stale. If data is prefetched into L1 between the
flush_cache_allinvocation and disabling the D-cache then this data will still be present when we come out of lowpower. Without coherency, we *must not* use this data and so a D-cache invalidation to the PoC is required incpu_leave_lowpower().
Quoted 3 lines
What if we fixed the
cpu_resetfunctions for v6 and v7, and when a CPU is taken offline, we actually go through a proper shutdown of that CPU and call the reset vector, re-entering the boot loader?
This will certainly solve our problem, but people might complain that it's too heavyweight :) That said, for v7 this may be the only solution as the platform can require an IMPLEMENTATION DEFINED initialisation routine to be executed before enabling the D-cache out of reset. This should be something that the bootloader does for us.
Quoted 8 lines
We can only do this for CPUs other than the original boot CPU, because the boot loader should be checking which are the secondary CPUs and putting those into this simple WFI loop with the GIC appropriately programmed.
This means when we re-activate the CPU, we'll be waking it up in exactly the same way as we do when the kernel boots - and we have all that code around just waiting to be used.
As long as the bootloader doesn't mind, then this should work.
Will
Quoted 8 lines
-----Original Message-----
From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm- kernel-bounces at lists.infradead.org] On Behalf Of Will Deacon
Sent: Tuesday, December 07, 2010 11:17 PM
To: 'Russell King - ARM Linux'
Cc: linux-arm-kernel at lists.infradead.org
Subject: RE: [RFC] Fixing CPU Hotplug for RealView Platforms
Quoted 6 lines
Hi Russell,
Quoted 4 lines · Will Deacon
Quoted 3 lines
Hello,
Currently, CPU hotplug is broken for RealView platforms. I posted
some
Quoted 1 line
Quoted 1 line
Quoted 1 line
patches previously to try and address this, but they didn't solve
the
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
problems fully:
September/026157.html
Quoted 2 lines
Quoted 2 lines
I'm now revisiting the code and it looks like the main problem is
when
Quoted 13 lines
Quoted 6 lines
Quoted 3 lines
we wish to *leave* the lowpower state. The enter/leave routines look like this:
...
Quoted 2 lines
The problem is that by turning off coherency, the contents of the D-
cache
Quoted 1 line
Quoted 1 line
becomes stale. If data is prefetched into L1 between the
flush_cache_allQuoted 1 line
Quoted 1 line
invocation and disabling the D-cache then this data will still be
present
Quoted 1 line
Quoted 1 line
when we come out of lowpower. Without coherency, we *must not* use
this
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
data and so a D-cache invalidation to the PoC is required in
cpu_leave_lowpower().What if we fixed the
cpu_resetfunctions for v6 and v7, and when a CPU is taken offline, we actually go through a proper shutdown of that CPU and call the reset vector, re-entering the boot loader?This will certainly solve our problem, but people might complain that
it's
Quoted 1 line
too heavyweight :) That said, for v7 this may be the only solution as
the
Quoted 15 lines
platform can require an IMPLEMENTATION DEFINED initialisation routine to be
executed before enabling the D-cache out of reset. This should be something
that the bootloader does for us.Quoted 8 lines
We can only do this for CPUs other than the original boot CPU, because the boot loader should be checking which are the secondary CPUs and putting those into this simple WFI loop with the GIC appropriately programmed.
This means when we re-activate the CPU, we'll be waking it up in exactly the same way as we do when the kernel boots - and we have all that code around just waiting to be used.
One more simpler thing which could work is disable "C' bit before flushing the L1 cache. That way prefetch would be avoided and cache also will be in clean state while restarting the core.
Regards,
Santosh
Hi Santosh,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
We can only do this for CPUs other than the original boot CPU, because the boot loader should be checking which are the secondary CPUs and putting those into this simple WFI loop with the GIC appropriately programmed.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
This means when we re-activate the CPU, we'll be waking it up in exactly the same way as we do when the kernel boots - and we have all that code around just waiting to be used.
One more simpler thing which could work is disable "C' bit before flushing the L1 cache. That way prefetch would be avoided and cache also will be in clean state while restarting the core.
I like this idea because it's easy to implement! It does, however, rely on caches not containing any random dirty lines when leaving the low-power state. This behaviour is IMPLEMENTATION DEFINED out of reset, so its something that platform code will need to handle anyway.
On RealView, we only do a WFI to enter lowpower so your approach sounds feasible. I'll put in a comment describing the potential problems with random D-cache data out of reset so that if other platforms blindly copy the code, at least they've been warned.
Will
Quoted 7 lines · Will Deacon
Hi Russell,
Quoted 3 lines
What if we fixed the
cpu_resetfunctions for v6 and v7, and when a CPU is taken offline, we actually go through a proper shutdown of that CPU and call the reset vector, re-entering the boot loader?This will certainly solve our problem, but people might complain that it's too heavyweight :)
Our current hot plug-in code already re-runs most of the secondary CPU initialization in the kernel - we reset the stack and jump back to secondary_start_kernel().
What we'll be adding to that is the overheads in the boot loader (which would happen at the point we go offline) and getting back to the kernel code. From that point, the additional things are looking up the CPU type, calling its setup function, and enabling the MMU.
Everything from that point on would be identical to what already happens on the hot plug-in path.
Quoted 7 lines · Will Deacon
Hi Russell,
Quoted 3 lines
What if we fixed the
cpu_resetfunctions for v6 and v7, and when a CPU is taken offline, we actually go through a proper shutdown of that CPU and call the reset vector, re-entering the boot loader?This will certainly solve our problem, but people might complain that it's too heavyweight :)
Well, I've taken some measurements from the CPU boot, and there appears to be some interesting behaviour here:
Boot time bringup:
boot CPU CPU1
Booting: 1084 -> 0ns 0ns (about 1us per print)
cross call: 21750 -> 21.75us
Up: 267167 -> 267.167us
CPU1: Booted secondary processorsecondary_init: 297834 -> 297.834us
writing release: 310917 -> 310.917us
release done: 320334 -> 320.334us
released: 327750 -> 327.75us
Boot returned: 342917 -> 342.917us
sync'd: 343167 -> 343.167us
CPU1: Unknown IPI message 0x1
Online: 218416334 -> 218.416334ms
This looks reasonable - 300us taken to get from requesting the CPU to boot in __cpu_up() to the CPU marking itself online.
The 218ms will be down to calibrate_delay().
CPU2 and CPU3 have very similar boot timings, so I'm pretty happy that this timing is reliable.
Hotplug bringup:
Booting: 1000 -> 0ns 0ns (1us per print)
Restarting: 3976375 -> 3.976375ms
cross call: 3976625 -> 3.976625ms
Up: 4003125 -> 4.003125ms
CPU1: Booted secondary processorsecondary_init: 4022583 -> 4.022583ms
writing release: 4040750 -> 4.04075ms
release done: 4051083 -> 4.051083ms
released: 46509000 -> 4.6509ms
Boot returned: 51745708 -> 5.1745708ms
sync'd: 51745875 -> 5.1745875ms
CPU1: Unknown IPI message 0x1
Switched to NOHz mode on CPU #1
Online: 281251041 -> 281.251041ms
So, it appears to take 4ms to get from just before the call to boot_secondary() in __cpu_up() to writing pen_release.
The secondary CPU appears to run from being woken up to writing the pen release in about 40us - and then spends about 1ms spinning on its lock waiting for the requesting CPU to catch up.
This can be repeated every time without exception when you bring a CPU back online.
Looking at that 500us, it seems to be taken up by 'spin_unlock()' in boot_secondary:
00000000 <boot_secondary>:
a0: ebfffffe bl 0 <sched_clock>
a4: e59f3044 ldr r3, [pc, #68] ; f0 <boot_secondary+0xf0>
a8: e893000c ldm r3, {r2, r3}
ac: e0502002 subs r2, r0, r2
b0: e0c13003 sbc r3, r1, r3
b4: e59f004c ldr r0, [pc, #76] ; 108 <boot_secondary+0x108>
b8: ebfffffe bl 0 <printk> ; "released: %llu\n"
--spin_unlock--
bc: f57ff05f dmb sy
c0: e3a02000 mov r2, #0 ; 0x0
c4: e59f3020 ldr r3, [pc, #32] ; ec <boot_secondary+0xec>
c8: e5832000 str r2, [r3]
cc: f57ff04f dsb sy
d0: e320f004 sev
----
d4: e59f3018 ldr r3, [pc, #24] ; f4 <boot_secondary+0xf4>
d8: e5933000 ldr r3, [r3] ; read pen_release
dc: e3730001 cmn r3, #1 ; 0x1 ; == -1?
e0: 13e00025 mvnne r0, #37 ; 0x25 ; != -1 => -ENOSYS
e4: 01a00002 moveq r0, r2 ; == -1 => 0
e8: e99da870 ldmib sp, {r4, r5, r6, fp, sp, pc}
...
000001d8 <__cpu_up>:
2dc: ebfffffe bl 0 <boot_secondary>
2e0: e1a05000 mov r5, r0
2e4: ebfffffe bl 0 <sched_clock>
2e8: e894000c ldm r4, {r2, r3} ; boot_start
2ec: e0502002 subs r2, r0, r2 ; sched_clock() - boot_start
2f0: e0c13003 sbc r3, r1, r3
2f4: e59f0128 ldr r0, [pc, #296] ; 424 <__cpu_up+0x24c>
2f8: ebfffffe bl 0 <printk> ; "Boot returned: %llu\n"So there's not much going on in that path.
The CPU being brought online is doing this:
00000034 <_raw_spin_lock>:
34: e1a0c00d mov ip, sp
38: e92dd800 push {fp, ip, lr, pc}
3c: e24cb004 sub fp, ip, #4 ; 0x4
40: e3a03001 mov r3, #1 ; 0x1
44: e1902f9f ldrex r2, [r0]
48: e3320000 teq r2, #0 ; 0x0
4c: 1320f002 wfene
50: 01802f93 strexeq r2, r3, [r0]
54: 03320000 teqeq r2, #0 ; 0x0
58: 1afffff9 bne 44 <_raw_spin_lock+0x10>
5c: f57ff05f dmb sy
60: e89da800 ldm sp, {fp, sp, pc}as it's waiting for the lock to be released. So... what could be causing the above code in boot_secondary()/__cpu_up() to take 500us when the system's running? The dmb, dsb, or sev? Or the SCU trying to sort out the str to release the lock?
Hi Russell,
Thanks for looking into this.
Quoted 2 lines · Russell King - ARM Linux
Boot time bringup:
[...]
Quoted 3 lines
CPU2 and CPU3 have very similar boot timings, so I'm pretty happy that this timing is reliable.
Looks sane.
Quoted 20 lines
Hotplug bringup:
Booting: 1000 -> 0ns 0ns (1us per print)
Restarting: 3976375 -> 3.976375ms
cross call: 3976625 -> 3.976625ms
Up: 4003125 -> 4.003125ms
CPU1: Booted secondary processorsecondary_init: 4022583 -> 4.022583ms
writing release: 4040750 -> 4.04075ms
release done: 4051083 -> 4.051083ms
released: 46509000 -> 4.6509ms
Boot returned: 51745708 -> 5.1745708ms
sync'd: 51745875 -> 5.1745875ms
CPU1: Unknown IPI message0x1
Switched to NOHz mode on CPU #1
Online: 281251041 -> 281.251041msSo, it appears to take 4ms to get from just before the call to
boot_secondary()in__cpu_up()to writingpen_release.
Quoted 7 lines
The secondary CPU appears to run from being woken up to writing the pen release in about 40us - and then spends about 1ms spinning on its lock waiting for the requesting CPU to catch up.
This can be repeated every time without exception when you bring a CPU back online.
Hmm, this sounds needlessly expensive.
Quoted 4 lines
Looking at that 500us, it seems to be taken up by '
spin_unlock()' inboot_secondary:00000000 <
boot_secondary>:
[...]
Quoted 8 lines
--spin_unlock--
bc: f57ff05f dmb sy c0: e3a02000 mov r2, #0 ; 0x0 c4: e59f3020 ldr r3, [pc, #32] ; ec <boot_secondary+0xec> c8: e5832000 str r2, [r3] cc: f57ff04f dsb sy d0: e320f004 sev----
One thing that might be worth trying is changing spin_unlock to use strex [alongside a dummy ldrex]. There could be some QoS logic at L2 which favours exclusive accesses, meaning that the unlock is starved by the lock. I don't have access to a board at the moment, so this is purely speculation!
Quoted 16 lines
The CPU being brought online is doing this:
00000034 <
_raw_spin_lock>:34: e1a0c00d mov ip, sp 38: e92dd800 push {fp, ip, lr, pc} 3c: e24cb004 sub fp, ip, #4 ; 0x4 40: e3a03001 mov r3, #1 ; 0x1 44: e1902f9f ldrex r2, [r0] 48: e3320000 teq r2, #0 ; 0x0 4c: 1320f002 wfene 50: 01802f93 strexeq r2, r3, [r0] 54: 03320000 teqeq r2, #0 ; 0x0 58: 1afffff9 bne 44 <_raw_spin_lock+0x10> 5c: f57ff05f dmb sy 60: e89da800 ldm sp, {fp, sp, pc}
Quoted 4 lines
as it's waiting for the lock to be released. So... what could be causing the above code in boot_secondary()/__cpu_up() to take 500us when the system's running? The dmb, dsb, or sev? Or the SCU trying to sort out the str to release the lock?
Another experiment would be to remove the wfe/sev instructions to see if they're eating cycles. I think a WFE on the A9 disables a bunch of clocks, so that could be taking time to do.
<shameless plug>
You could try using perf to identify the most expensive instructions in the functions above (assuming interrupts are enabled). </shameless plug>
Cheers,
Will
Quoted 20 lines · Will Deacon
Quoted 20 lines
Hotplug bringup:
Booting: 1000 -> 0ns 0ns (1us per print)
Restarting: 3976375 -> 3.976375ms
cross call: 3976625 -> 3.976625ms
Up: 4003125 -> 4.003125ms
CPU1: Booted secondary processorsecondary_init: 4022583 -> 4.022583ms
writing release: 4040750 -> 4.04075ms
release done: 4051083 -> 4.051083ms
released: 46509000 -> 4.6509ms
Boot returned: 51745708 -> 5.1745708ms
sync'd: 51745875 -> 5.1745875ms
CPU1: Unknown IPI message0x1
Switched to NOHz mode on CPU #1
Online: 281251041 -> 281.251041msSo, it appears to take 4ms to get from just before the call to
boot_secondary()in__cpu_up()to writingpen_release.
Quoted 8 lines
Quoted 7 lines
The secondary CPU appears to run from being woken up to writing the pen release in about 40us - and then spends about 1ms spinning on its lock waiting for the requesting CPU to catch up.
This can be repeated every time without exception when you bring a CPU back online.
Hmm, this sounds needlessly expensive.
Actually, I'm starting to get concerned about doing timing measurements on Versatile Express - I'm seeing some unexplainable issues with the Versatile Express platform.
I occasionally see the kernel get stuck when initializing the CLCD - and I think this is a hardware lockup - pressing the red 'reset/power on' button is ignored, and the only way to recover it is to press the black 'power off' button first.
Also I keep running into some weird stuff which causes the MMC to underflow, serial output to be corrupted, and rootfs not to be mounted which is 100% reliable with some kernels (iow, the built kernel just will not boot no matter how many times you attempt to do so.) I've sent Catalin & Philippe a copy of one such kernel which exhibits this behaviour a few days ago (but I think they're on holiday.)
Anyway, I decided to implement a slightly different method to measuring the time taken, and the apparant long delays have gone - I suspect that was something to do with printk. I'm not logging the times into an array, and later printing out the values.
So, CPU1 boot:
SMP: Start: 0
SMP: Booting: 916
SMP: Cross call: 3083
SMP: Pen released: 278416
SMP: Unlock: 279583
SMP: Boot returned: 280333
SMP: Sec: up: 238666
SMP: Sec: enter: 264333
SMP: Sec: pen write: 267083
SMP: Sec: pen done: 268916
SMP: Sec: exit: 279916
SMP: Sec: calibrate: 328416
SMP: Sec: online: 218380875
CPU1 hotplug:
SMP: Start: 0
SMP: Booting: 833
SMP: Cross call: 4250
SMP: Pen released: 51500
SMP: Unlock: 52667
SMP: Boot returned: 53500
SMP: Sec: restart: 4667
SMP: Sec: up: 7167
SMP: Sec: enter: 31000
SMP: Sec: pen write: 39667
SMP: Sec: pen done: 42167
SMP: Sec: exit: 53000
SMP: Sec: calibrate: 104583
SMP: Sec: online: 221423333
This looks far saner.
Anyway, with the delay loop calibration, we're looking at a boot time of about 110us to the delay loop calibration, and 221ms for a secondary CPU using the existing code. I don't think that will go up significantly if we re-vector offlined CPUs back through the reset vector.