Skip to contentSkip to conversation

Fixing CPU Hotplug for RealView Platforms

RFC Will Deacon 3 participants 9 messages Dec 7, 2010 – Dec 18, 2010

lore
Share & tools
lore
#0 lore raw

Hello,

Currently, CPU hotplug is broken for RealView platforms. I posted some patches previously to try and address this, but they didn't solve the problems fully:

http://lists.infradead.org/pipermail/linux-arm-kernel/2010-September/026157.html

I'm now revisiting the code and it looks like the main problem is when we wish to *leave* the lowpower state. The enter/leave routines look like this:

static inline void cpu_enter_lowpower(void)

{
	unsigned int v, smp_ctrl = get_smp_ctrl_mask();

	flush_cache_all();
	dsb();
	asm volatile(
	/*
	 * Turn off coherency
	 */
	"	mrc	p15, 0, %0, c1, c0, 1\n"
	"	bic	%0, %0, %1\n"
	"	mcr	p15, 0, %0, c1, c0, 1\n"
	/* ISB */
	"	mcr	p15, 0, %2, c7, c5, 4\n"
	/* Disable D-cache */
	"	mrc	p15, 0, %0, c1, c0, 0\n"
	"	bic	%0, %0, #0x04\n"
	"	mcr	p15, 0, %0, c1, c0, 0\n"
	  : "=&r" (v)
	  : "r" (smp_ctrl), "r" (0)
	  : "memory");
	isb();
}

static inline void cpu_leave_lowpower(void)

{
	unsigned int v, smp_ctrl = get_smp_ctrl_mask();

	asm volatile(	"mrc	p15, 0, %0, c1, c0, 0\n"
	"	orr	%0, %0, #0x04\n"
	"	mcr	p15, 0, %0, c1, c0, 0\n"
	"	mrc	p15, 0, %0, c1, c0, 1\n"
	"	orr	%0, %0, %1\n"
	"	mcr	p15, 0, %0, c1, c0, 1\n"
	  : "=&r" (v)
	  : "r" (smp_ctrl)
	  : "memory");
	isb();
}

The problem is that by turning off coherency, the contents of the D-cache becomes stale. If data is prefetched into L1 between the flush_cache_all invocation and disabling the D-cache then this data will still be present when we come out of lowpower. Without coherency, we *must not* use this data and so a D-cache invalidation to the PoC is required in cpu_leave_lowpower().

On v6 this is a simple mcr instruction. On v7, we have to perform a set/way operation across all ways of each cache until we reach the PoC (see the scary but well commented v7_flush_dcache_all function). Implementing this means extending the cpu_cache_fns struct and stubbing out the new function for other caches, so I'd like to see if anybody has any better ideas before I go ahead and make these changes.

One possibility is not to turn off coherency, but if platform_do_lowpower is more than a WFI I don't think this would be suitable.

Any thoughts?

Will

#1 lore raw
Quoted 12 lines · Will Deacon

Hello,

Currently, CPU hotplug is broken for RealView platforms. I posted some patches previously to try and address this, but they didn't solve the problems fully:

http://lists.infradead.org/pipermail/linux-arm-kernel/2010-September/026157.html

I'm now revisiting the code and it looks like the main problem is when we wish to *leave* the lowpower state. The enter/leave routines look like this:

...

Quoted 7 lines

The problem is that by turning off coherency, the contents of the D-cache becomes stale. If data is prefetched into L1 between the flush_cache_all invocation and disabling the D-cache then this data will still be present when we come out of lowpower. Without coherency, we *must not* use this data and so a D-cache invalidation to the PoC is required in cpu_leave_lowpower().

What if we fixed the cpu_reset functions for v6 and v7, and when a CPU is taken offline, we actually go through a proper shutdown of that CPU and call the reset vector, re-entering the boot loader?

We can only do this for CPUs other than the original boot CPU, because the boot loader should be checking which are the secondary CPUs and putting those into this simple WFI loop with the GIC appropriately programmed.

This means when we re-activate the CPU, we'll be waking it up in exactly the same way as we do when the kernel boots - and we have all that code around just waiting to be used.

#2 lore raw

Hi Russell,

Quoted 22 lines · Will Deacon
Quoted 12 lines

Hello,

Currently, CPU hotplug is broken for RealView platforms. I posted some patches previously to try and address this, but they didn't solve the problems fully:

http://lists.infradead.org/pipermail/linux-arm-kernel/2010-September/026157.html

I'm now revisiting the code and it looks like the main problem is when we wish to *leave* the lowpower state. The enter/leave routines look like this:

...

Quoted 7 lines

The problem is that by turning off coherency, the contents of the D-cache becomes stale. If data is prefetched into L1 between the flush_cache_all invocation and disabling the D-cache then this data will still be present when we come out of lowpower. Without coherency, we *must not* use this data and so a D-cache invalidation to the PoC is required in cpu_leave_lowpower().

Quoted 3 lines

What if we fixed the cpu_reset functions for v6 and v7, and when a CPU is taken offline, we actually go through a proper shutdown of that CPU and call the reset vector, re-entering the boot loader?

This will certainly solve our problem, but people might complain that it's too heavyweight :) That said, for v7 this may be the only solution as the platform can require an IMPLEMENTATION DEFINED initialisation routine to be executed before enabling the D-cache out of reset. This should be something that the bootloader does for us.

Quoted 8 lines

We can only do this for CPUs other than the original boot CPU, because the boot loader should be checking which are the secondary CPUs and putting those into this simple WFI loop with the GIC appropriately programmed.

This means when we re-activate the CPU, we'll be waking it up in exactly the same way as we do when the kernel boots - and we have all that code around just waiting to be used.

As long as the bootloader doesn't mind, then this should work.

Will

#3 lore raw
Quoted 8 lines

-----Original Message-----
From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm- kernel-bounces at lists.infradead.org] On Behalf Of Will Deacon
Sent: Tuesday, December 07, 2010 11:17 PM
To: 'Russell King - ARM Linux'
Cc: linux-arm-kernel at lists.infradead.org
Subject: RE: [RFC] Fixing CPU Hotplug for RealView Platforms

Quoted 6 lines

Hi Russell,

Quoted 4 lines · Will Deacon
Quoted 3 lines

Hello,

Currently, CPU hotplug is broken for RealView platforms. I posted

some

Quoted 1 line
Quoted 1 line
Quoted 1 line

patches previously to try and address this, but they didn't solve

the

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

problems fully:

http://lists.infradead.org/pipermail/linux-arm-kernel/2010-

September/026157.html

Quoted 2 lines
Quoted 2 lines

I'm now revisiting the code and it looks like the main problem is

when

Quoted 13 lines
Quoted 6 lines
Quoted 3 lines

we wish to *leave* the lowpower state. The enter/leave routines look like this:

...

Quoted 2 lines

The problem is that by turning off coherency, the contents of the D-

cache

Quoted 1 line
Quoted 1 line

becomes stale. If data is prefetched into L1 between the

flush_cache_all

Quoted 1 line
Quoted 1 line

invocation and disabling the D-cache then this data will still be

present

Quoted 1 line
Quoted 1 line

when we come out of lowpower. Without coherency, we *must not* use

this

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

data and so a D-cache invalidation to the PoC is required in cpu_leave_lowpower().

What if we fixed the cpu_reset functions for v6 and v7, and when a CPU is taken offline, we actually go through a proper shutdown of that CPU and call the reset vector, re-entering the boot loader?

This will certainly solve our problem, but people might complain that

it's

Quoted 1 line

too heavyweight :) That said, for v7 this may be the only solution as

the

Quoted 15 lines

platform can require an IMPLEMENTATION DEFINED initialisation routine to be
executed before enabling the D-cache out of reset. This should be something
that the bootloader does for us.

Quoted 8 lines

We can only do this for CPUs other than the original boot CPU, because the boot loader should be checking which are the secondary CPUs and putting those into this simple WFI loop with the GIC appropriately programmed.

This means when we re-activate the CPU, we'll be waking it up in exactly the same way as we do when the kernel boots - and we have all that code around just waiting to be used.

One more simpler thing which could work is disable "C' bit before flushing the L1 cache. That way prefetch would be avoided and cache also will be in clean state while restarting the core.

Regards,
Santosh

#4 lore raw

Hi Santosh,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

We can only do this for CPUs other than the original boot CPU, because the boot loader should be checking which are the secondary CPUs and putting those into this simple WFI loop with the GIC appropriately programmed.

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

This means when we re-activate the CPU, we'll be waking it up in exactly the same way as we do when the kernel boots - and we have all that code around just waiting to be used.

One more simpler thing which could work is disable "C' bit before flushing the L1 cache. That way prefetch would be avoided and cache also will be in clean state while restarting the core.

I like this idea because it's easy to implement! It does, however, rely on caches not containing any random dirty lines when leaving the low-power state. This behaviour is IMPLEMENTATION DEFINED out of reset, so its something that platform code will need to handle anyway.

On RealView, we only do a WFI to enter lowpower so your approach sounds feasible. I'll put in a comment describing the potential problems with random D-cache data out of reset so that if other platforms blindly copy the code, at least they've been warned.

Will

#5 lore raw
In reply to Will Deacon
Quoted 7 lines · Will Deacon

Hi Russell,

Quoted 3 lines

What if we fixed the cpu_reset functions for v6 and v7, and when a CPU is taken offline, we actually go through a proper shutdown of that CPU and call the reset vector, re-entering the boot loader?

This will certainly solve our problem, but people might complain that it's too heavyweight :)

Our current hot plug-in code already re-runs most of the secondary CPU initialization in the kernel - we reset the stack and jump back to secondary_start_kernel().

What we'll be adding to that is the overheads in the boot loader (which would happen at the point we go offline) and getting back to the kernel code. From that point, the additional things are looking up the CPU type, calling its setup function, and enabling the MMU.

Everything from that point on would be identical to what already happens on the hot plug-in path.

#6 lore raw
In reply to Will Deacon
Quoted 7 lines · Will Deacon

Hi Russell,

Quoted 3 lines

What if we fixed the cpu_reset functions for v6 and v7, and when a CPU is taken offline, we actually go through a proper shutdown of that CPU and call the reset vector, re-entering the boot loader?

This will certainly solve our problem, but people might complain that it's too heavyweight :)

Well, I've taken some measurements from the CPU boot, and there appears to be some interesting behaviour here:

Boot time bringup:

boot CPU CPU1
Booting: 1084 -> 0ns 0ns (about 1us per print)
cross call: 21750 -> 21.75us
Up: 267167 -> 267.167us
CPU1: Booted secondary processor
secondary_init: 297834 -> 297.834us
writing release: 310917 -> 310.917us
release done: 320334 -> 320.334us
released: 327750 -> 327.75us
Boot returned: 342917 -> 342.917us
sync'd: 343167 -> 343.167us
CPU1: Unknown IPI message 0x1
Online: 218416334 -> 218.416334ms

This looks reasonable - 300us taken to get from requesting the CPU to boot in __cpu_up() to the CPU marking itself online.

The 218ms will be down to calibrate_delay().

CPU2 and CPU3 have very similar boot timings, so I'm pretty happy that this timing is reliable.

Hotplug bringup:

Booting: 1000 -> 0ns 0ns (1us per print)
Restarting: 3976375 -> 3.976375ms
cross call: 3976625 -> 3.976625ms
Up: 4003125 -> 4.003125ms
CPU1: Booted secondary processor
secondary_init: 4022583 -> 4.022583ms
writing release: 4040750 -> 4.04075ms
release done: 4051083 -> 4.051083ms
released: 46509000 -> 4.6509ms
Boot returned: 51745708 -> 5.1745708ms
sync'd: 51745875 -> 5.1745875ms
CPU1: Unknown IPI message 0x1
Switched to NOHz mode on CPU #1
Online: 281251041 -> 281.251041ms

So, it appears to take 4ms to get from just before the call to boot_secondary() in __cpu_up() to writing pen_release.

The secondary CPU appears to run from being woken up to writing the pen release in about 40us - and then spends about 1ms spinning on its lock waiting for the requesting CPU to catch up.

This can be repeated every time without exception when you bring a CPU back online.

Looking at that 500us, it seems to be taken up by 'spin_unlock()' in boot_secondary:

00000000 <boot_secondary>:

  a0:   ebfffffe        bl      0 <sched_clock>
  a4:   e59f3044        ldr     r3, [pc, #68]   ; f0 <boot_secondary+0xf0>
  a8:   e893000c        ldm     r3, {r2, r3}
  ac:   e0502002        subs    r2, r0, r2
  b0:   e0c13003        sbc     r3, r1, r3
  b4:   e59f004c        ldr     r0, [pc, #76]   ; 108 <boot_secondary+0x108>
  b8:   ebfffffe        bl      0 <printk>	; "released: %llu\n"
--spin_unlock--
  bc:   f57ff05f        dmb     sy
  c0:   e3a02000        mov     r2, #0  ; 0x0
  c4:   e59f3020        ldr     r3, [pc, #32]   ; ec <boot_secondary+0xec>
  c8:   e5832000        str     r2, [r3]
  cc:   f57ff04f        dsb     sy
  d0:   e320f004        sev
----
  d4:   e59f3018        ldr     r3, [pc, #24]   ; f4 <boot_secondary+0xf4>
  d8:   e5933000        ldr     r3, [r3]	; read pen_release
  dc:   e3730001        cmn     r3, #1  ; 0x1	; == -1?
  e0:   13e00025        mvnne   r0, #37 ; 0x25	; != -1 => -ENOSYS
  e4:   01a00002        moveq   r0, r2		; == -1 => 0
  e8:   e99da870        ldmib   sp, {r4, r5, r6, fp, sp, pc}
...
000001d8 <__cpu_up>:
 2dc:   ebfffffe        bl      0 <boot_secondary>
 2e0:   e1a05000        mov     r5, r0
 2e4:   ebfffffe        bl      0 <sched_clock>
 2e8:   e894000c        ldm     r4, {r2, r3}	; boot_start
 2ec:   e0502002        subs    r2, r0, r2	; sched_clock() - boot_start
 2f0:   e0c13003        sbc     r3, r1, r3
 2f4:   e59f0128        ldr     r0, [pc, #296]  ; 424 <__cpu_up+0x24c>
 2f8:   ebfffffe        bl      0 <printk>	; "Boot returned: %llu\n"

So there's not much going on in that path.

The CPU being brought online is doing this:

00000034 <_raw_spin_lock>:

  34:   e1a0c00d        mov     ip, sp
  38:   e92dd800        push    {fp, ip, lr, pc}
  3c:   e24cb004        sub     fp, ip, #4      ; 0x4
  40:   e3a03001        mov     r3, #1  ; 0x1
  44:   e1902f9f        ldrex   r2, [r0]
  48:   e3320000        teq     r2, #0  ; 0x0
  4c:   1320f002        wfene
  50:   01802f93        strexeq r2, r3, [r0]
  54:   03320000        teqeq   r2, #0  ; 0x0
  58:   1afffff9        bne     44 <_raw_spin_lock+0x10>
  5c:   f57ff05f        dmb     sy
  60:   e89da800        ldm     sp, {fp, sp, pc}

as it's waiting for the lock to be released. So... what could be causing the above code in boot_secondary()/__cpu_up() to take 500us when the system's running? The dmb, dsb, or sev? Or the SCU trying to sort out the str to release the lock?

#7 lore raw

Hi Russell,

Thanks for looking into this.

Quoted 2 lines · Russell King - ARM Linux

Boot time bringup:

[...]

Quoted 3 lines

CPU2 and CPU3 have very similar boot timings, so I'm pretty happy that this timing is reliable.

Looks sane.

Quoted 20 lines

Hotplug bringup:

Booting: 1000 -> 0ns 0ns (1us per print)
Restarting: 3976375 -> 3.976375ms
cross call: 3976625 -> 3.976625ms
Up: 4003125 -> 4.003125ms
CPU1: Booted secondary processor
secondary_init: 4022583 -> 4.022583ms
writing release: 4040750 -> 4.04075ms
release done: 4051083 -> 4.051083ms
released: 46509000 -> 4.6509ms
Boot returned: 51745708 -> 5.1745708ms
sync'd: 51745875 -> 5.1745875ms
CPU1: Unknown IPI message 0x1
Switched to NOHz mode on CPU #1
Online: 281251041 -> 281.251041ms

So, it appears to take 4ms to get from just before the call to boot_secondary() in __cpu_up() to writing pen_release.

Quoted 7 lines

The secondary CPU appears to run from being woken up to writing the pen release in about 40us - and then spends about 1ms spinning on its lock waiting for the requesting CPU to catch up.

This can be repeated every time without exception when you bring a CPU back online.

Hmm, this sounds needlessly expensive.

Quoted 4 lines

Looking at that 500us, it seems to be taken up by 'spin_unlock()' in boot_secondary:

00000000 <boot_secondary>:

[...]

Quoted 8 lines

--spin_unlock--

  bc:   f57ff05f        dmb     sy
  c0:   e3a02000        mov     r2, #0  ; 0x0
  c4:   e59f3020        ldr     r3, [pc, #32]   ; ec <boot_secondary+0xec>
  c8:   e5832000        str     r2, [r3]
  cc:   f57ff04f        dsb     sy
  d0:   e320f004        sev

----

One thing that might be worth trying is changing spin_unlock to use strex [alongside a dummy ldrex]. There could be some QoS logic at L2 which favours exclusive accesses, meaning that the unlock is starved by the lock. I don't have access to a board at the moment, so this is purely speculation!

Quoted 16 lines

The CPU being brought online is doing this:

00000034 <_raw_spin_lock>:

  34:   e1a0c00d        mov     ip, sp
  38:   e92dd800        push    {fp, ip, lr, pc}
  3c:   e24cb004        sub     fp, ip, #4      ; 0x4
  40:   e3a03001        mov     r3, #1  ; 0x1
  44:   e1902f9f        ldrex   r2, [r0]
  48:   e3320000        teq     r2, #0  ; 0x0
  4c:   1320f002        wfene
  50:   01802f93        strexeq r2, r3, [r0]
  54:   03320000        teqeq   r2, #0  ; 0x0
  58:   1afffff9        bne     44 <_raw_spin_lock+0x10>
  5c:   f57ff05f        dmb     sy
  60:   e89da800        ldm     sp, {fp, sp, pc}
Quoted 4 lines

as it's waiting for the lock to be released. So... what could be causing the above code in boot_secondary()/__cpu_up() to take 500us when the system's running? The dmb, dsb, or sev? Or the SCU trying to sort out the str to release the lock?

Another experiment would be to remove the wfe/sev instructions to see if they're eating cycles. I think a WFE on the A9 disables a bunch of clocks, so that could be taking time to do.

<shameless plug>
You could try using perf to identify the most expensive instructions in the functions above (assuming interrupts are enabled). </shameless plug>

Cheers,

Will

#8 lore raw
Quoted 20 lines · Will Deacon
Quoted 20 lines

Hotplug bringup:

Booting: 1000 -> 0ns 0ns (1us per print)
Restarting: 3976375 -> 3.976375ms
cross call: 3976625 -> 3.976625ms
Up: 4003125 -> 4.003125ms
CPU1: Booted secondary processor
secondary_init: 4022583 -> 4.022583ms
writing release: 4040750 -> 4.04075ms
release done: 4051083 -> 4.051083ms
released: 46509000 -> 4.6509ms
Boot returned: 51745708 -> 5.1745708ms
sync'd: 51745875 -> 5.1745875ms
CPU1: Unknown IPI message 0x1
Switched to NOHz mode on CPU #1
Online: 281251041 -> 281.251041ms

So, it appears to take 4ms to get from just before the call to boot_secondary() in __cpu_up() to writing pen_release.

Quoted 8 lines
Quoted 7 lines

The secondary CPU appears to run from being woken up to writing the pen release in about 40us - and then spends about 1ms spinning on its lock waiting for the requesting CPU to catch up.

This can be repeated every time without exception when you bring a CPU back online.

Hmm, this sounds needlessly expensive.

Actually, I'm starting to get concerned about doing timing measurements on Versatile Express - I'm seeing some unexplainable issues with the Versatile Express platform.

I occasionally see the kernel get stuck when initializing the CLCD - and I think this is a hardware lockup - pressing the red 'reset/power on' button is ignored, and the only way to recover it is to press the black 'power off' button first.

Also I keep running into some weird stuff which causes the MMC to underflow, serial output to be corrupted, and rootfs not to be mounted which is 100% reliable with some kernels (iow, the built kernel just will not boot no matter how many times you attempt to do so.) I've sent Catalin & Philippe a copy of one such kernel which exhibits this behaviour a few days ago (but I think they're on holiday.)

Anyway, I decided to implement a slightly different method to measuring the time taken, and the apparant long delays have gone - I suspect that was something to do with printk. I'm not logging the times into an array, and later printing out the values.

So, CPU1 boot:

SMP: Start: 0
SMP: Booting: 916
SMP: Cross call: 3083
SMP: Pen released: 278416
SMP: Unlock: 279583
SMP: Boot returned: 280333

SMP: Sec: up: 238666
SMP: Sec: enter: 264333
SMP: Sec: pen write: 267083
SMP: Sec: pen done: 268916
SMP: Sec: exit: 279916
SMP: Sec: calibrate: 328416
SMP: Sec: online: 218380875

CPU1 hotplug:
SMP: Start: 0
SMP: Booting: 833
SMP: Cross call: 4250
SMP: Pen released: 51500
SMP: Unlock: 52667
SMP: Boot returned: 53500

SMP: Sec: restart: 4667
SMP: Sec: up: 7167
SMP: Sec: enter: 31000
SMP: Sec: pen write: 39667
SMP: Sec: pen done: 42167
SMP: Sec: exit: 53000
SMP: Sec: calibrate: 104583
SMP: Sec: online: 221423333

This looks far saner.

Anyway, with the delay loop calibration, we're looking at a boot time of about 110us to the delay loop calibration, and 221ms for a secondary CPU using the existing code. I don't think that will go up significantly if we re-vector offlined CPUs back through the reset vector.