ARM: Fixing udelay() for SMP and non-SMP systems
Subsystems (3)
v3 cover ARM: Fixing udelay() for SMP and non-SMP systems
These patches fixing the udelay() issue pointed out on arm-lkml[1][2]. A quick recap: some SMP machines can scale their CPU frequencies independent of one another. loops_per_jiffy is calibrated globally and used in __const_udelay(). If one CPU is running faster than what the loops_per_jiffy is calculated (or scaled) for, udelay() will be incorrect and not wait long enough (or too long). A similar problem occurs if the cpu frequency is scaled during a udelay() call.
We could fix this issue a couple ways, wholesale replacement of __udelay() and __const_udelay() (see [2] for that approach), or replacement of __delay() (this series). Option 1 can fail if anybody uses udelay() before memory is mapped and also duplicates most of the code in asm/delay.h. It also needs to hardcode the timer tick frequency, which can sometimes be inaccurate. The benefit is that loops_per_jiffy stays the same and thus BogoMIPS is unchanged. Option 2 can't fail since the __delay() loop is replaced after memory is mapped in, but it suffers from a low BogoMIPS when timers are clocked slowly. It also more accurately calculates the timer tick frequency through the use of calibrate_delay_direct().
Russell, I'm planning on submitting these to the patch queue unless I hear objections.
Changes since v2:
* Additional patch using the timer based delay
Changes since v1:
* likely() in delay.c
* comment fixup for read_current_timer_delay_loop()
* cosmetic improvements to commit text
Stephen Boyd (4):
ARM: Translate delay.S into (mostly) C
ARM: Allow machines to override __delay()
ARM: Implement a timer based __delay() loop
msm: timer: Migrate to timer based __delay()
arch/arm/include/asm/delay.h | 5 ++- arch/arm/kernel/armksyms.c | 4 -- arch/arm/lib/delay.S | 65 ------------------------ arch/arm/lib/delay.c | 86 ++++++++++++++++++++++++++++++++ arch/arm/mach-msm/include/mach/timex.h | 1 + arch/arm/mach-msm/timer.c | 16 ++++++- 6 files changed, 106 insertions(+), 71 deletions(-) delete mode 100644 arch/arm/lib/delay.S create mode 100644 arch/arm/lib/delay.c
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
PATCH v3 1/4 ARM: Translate delay.S into (mostly) C
We want to allow machines to override the __delay() implementation at runtime so they can use a timer based __delay() routine. It's easier to do this using C, so let's write udelay and friends in C.
We lose the #if 0 code, which according to Russell is used "to make the delay loop more stable and predictable on older CPUs" (see http://article.gmane.org/gmane.linux.kernel/888867 for more info). We shouldn't be too worried though, since we'll soon add functionality allowing a machine to set the __delay() loop themselves, thus allowing machines to resurrect the commented out code should they need it.
Nico expressed concern that fixed lpj cmdlines will break due to compiler optimizations. That doesn't seem to be the case since before and after this patch I get the same lpj value when running my CPU at 19.2 MHz. That should be sufficiently slow enough to cover any machine running Linux.
--- arch/arm/include/asm/delay.h | 2 +- arch/arm/kernel/armksyms.c | 4 -- arch/arm/lib/delay.S | 65 ------------------------------------------ arch/arm/lib/delay.c | 56 ++++++++++++++++++++++++++++++++++++ 4 files changed, 57 insertions(+), 70 deletions(-) delete mode 100644 arch/arm/lib/delay.S create mode 100644 arch/arm/lib/delay.c
arch/arm/include/asm/delay.h +1 -1
@@ -8,7 +8,7 @@#include <asm/param.h> /* HZ */ -extern void __delay (int loops); +extern void __delay (unsigned long loops);/* * This function intentionally does not exist; if you see references to
arch/arm/kernel/armksyms.c +0 -4
@@ -52,10 +52,6 @@ extern void fpundefinstr(void);EXPORT_SYMBOL (__backtrace); -/* platform dependent support */ -EXPORT_SYMBOL (__udelay); -EXPORT_SYMBOL (__const_udelay); -/* networking */ EXPORT_SYMBOL (csum_partial);EXPORT_SYMBOL (csum_partial_copy_from_user);
arch/arm/lib/delay.S deleted +0 -65
65 lines not on this page · raw message
arch/arm/lib/delay.c new file +56 -0
56 lines not on this page · raw message
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
PATCH v3 2/4 ARM: Allow machines to override __delay()
In reply to Stephen BoydSome machines want to implement their own __delay() routine based on fixed rate timers. Expose functionality to set the __delay() routine at runtime. This should allow two machines with different __delay() routines to happily co-exist within the same kernel with minimal overhead.
Russell expressed concern that using a timer based __delay() would cause problems when an iomapped device isn't mapped in prior to a delay call being made (see
http://article.gmane.org/gmane.linux.ports.arm.kernel/78543 for more info). We can sidestep that issue with this approach since the __delay() routine _should_ only be pointed to a timer based delay once the timer has been properly mapped. Up until that point __delay() and udelay() will use delay_loop() which is always safe to call.
This patch is inspired by x86's delay.c
--- arch/arm/include/asm/delay.h | 2 ++ arch/arm/lib/delay.c | 19 ++++++++++++++++--- 2 files changed, 18 insertions(+), 3 deletions(-)
arch/arm/include/asm/delay.h +2 -0
@@ -40,5 +40,7 @@ extern void __const_udelay(unsigned long); __const_udelay((n) * ((2199023U*HZ)>>11))) : \ __udelay(n)) +extern void set_delay_fn (void (*fn)(unsigned long )); +#endif /* defined(_ARM_DELAY_H) */
arch/arm/lib/delay.c +16 -3
@@ -11,11 +11,9 @@#include <linux/delay.h> /* -* loops = usecs * HZ * loops_per_jiffy / 1000000 -* * Oh, if only we had a cycle counter... */ -void __delay (unsigned long loops) +void delay_loop (unsigned long loops) { asmvolatile ( "1: subs %0, %0, #1 \n"
@@ -24,6 +22,21 @@ void __delay(unsigned long loops) : "r" (loops) ); } + +static void (*delay_fn)(unsigned long ) = delay_loop; + +void set_delay_fn (void (*fn)(unsigned long )) +{ + delay_fn = fn; +} + +/* +* loops = usecs * HZ * loops_per_jiffy / 1000000 +*/ +void __delay (unsigned long loops) +{ +delay_fn (loops); +}EXPORT_SYMBOL (__delay);/*
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Re: [PATCHv3 2/4] ARM: Allow machines to override __delay()
Quoted 6 lines · Stephen Boyd
arch/arm/lib/delay.c +6
-0+ +static void (*delay_fn)(unsigned long ) = delay_loop; + +void set_delay_fn (void (*fn)(unsigned long )) +{ + delay_fn = fn;
arch/arm/lib/delay.c void __delay(unsigned long loops) · line 40
Quoted 10 lines
arch/arm/lib/delay.c +9
-0+} + +/* +* loops = usecs * HZ * loops_per_jiffy / 1000000 +*/ +void __delay (unsigned long loops) +{ +delay_fn (loops); +}EXPORT_SYMBOL (__delay);
Didn't we already go over this part ? Why are aren't you putting these in a header file ?
Daniel
Signature · 2 lines
Sent by a consultant of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Re: [PATCHv3 2/4] ARM: Allow machines to override __delay()
Quoted 11 lines · Daniel Walker
Quoted 10 lines · Stephen Boyd
arch/arm/lib/delay.c +10
-0+ +static void (*delay_fn)(unsigned long ) = delay_loop; + +void set_delay_fn (void (*fn)(unsigned long )) +{ + delay_fn = fn; +} + +/* +* loops = usecs * HZ * loops_per_jiffy / 1000000
Quoted 8 lines
arch/arm/lib/delay.c void __delay(unsigned long loops) · line 40
Quoted 6 lines
+ */
arch/arm/lib/delay.c +4
-0+void __delay (unsigned long loops) +{ +delay_fn (loops); +}EXPORT_SYMBOL (__delay);Didn't we already go over this part ? Why are aren't you putting these in a header file ?
Last time we saw that inlining set_delay_fn() actually increased the text size of the kernel. I know it sounds wrong, but its probably due to that compiler behavior Russell posted about last month on arm-lkml.
I'd like to see what Russell wants to do since so far he's been silent on this whole series.
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Re: [PATCHv3 2/4] ARM: Allow machines to override __delay()
Quoted 21 lines · Stephen Boyd
Quoted 11 lines · Daniel Walker
Quoted 10 lines · Stephen Boyd
arch/arm/lib/delay.c +10
-0+ +static void (*delay_fn)(unsigned long ) = delay_loop; + +void set_delay_fn (void (*fn)(unsigned long )) +{ + delay_fn = fn; +} + +/* +* loops = usecs * HZ * loops_per_jiffy / 1000000 Quoted 8 lines
arch/arm/lib/delay.c void __delay(unsigned long loops) · line 40
Quoted 6 lines
+ */
arch/arm/lib/delay.c +4
-0+void __delay (unsigned long loops) +{ +delay_fn (loops); +}EXPORT_SYMBOL (__delay);Didn't we already go over this part ? Why are aren't you putting these in a header file ?
Quoted 3 lines
Last time we saw that inlining
set_delay_fn()actually increased the text size of the kernel. I know it sounds wrong, but its probably due to that compiler behavior Russell posted about last month on arm-lkml.
The last time I thought you said there was no size change ? My tests showed a decrease ..
What compiler optimizations are you talking about ?
Daniel
Signature · 2 lines
Sent by a consultant of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Re: [PATCHv3 2/4] ARM: Allow machines to override __delay()
Quoted 3 lines · Daniel Walker
The last time I thought you said there was no size change ? My tests showed a decrease ..
Oh, right. There was no size change when I moved __delay() into the header in addition to the others.
Quoted 2 lines
What compiler optimizations are you talking about
See
Subject: Weirdo gcc 4.3.x behaviour
Message-ID: <[email protected]>
and
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29950
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Re: [PATCHv3 2/4] ARM: Allow machines to override __delay()
Quoted 7 lines · Stephen Boyd
Quoted 3 lines · Daniel Walker
The last time I thought you said there was no size change ? My tests showed a decrease ..
Oh, right. There was no size change when I moved
__delay()into the header in addition to the others.
So is there a problem with doing the inlining then ? If you get the same size kernel, then that seems like a lateral move in your case.
Daniel
Signature · 2 lines
Sent by a consultant of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
PATCH v3 3/4 ARM: Implement a timer based __delay() loop
In reply to Stephen Boydudelay() can be incorrect on SMP machines that scale their CPU frequencies independently of one another (as pointed out here
http://article.gmane.org/gmane.linux.kernel/977567). The delay loop can either be too fast or too slow depending on which CPU the loops_per_jiffy counter is calibrated on and which CPU the delay loop is running on. udelay() can also be incorrect if the CPU frequency switches during the __delay() loop, causing the loop to either terminate too early, or too late.
Forcing udelay() to run on one CPU is unreasonable and taking the penalty of a rather large loops_per_jiffy in udelay() when the CPU is actually running slower is bad for performance. Solve the problem by adding a timer based__delay() loop unaffected by CPU frequency scaling. Machines should set this loop as their __delay() implementation by calling set_timer_fn() during their timer initialization.
The kernel is already prepared for a timer based approach (evident by the read_current_timer() function). If an arch implements read_current_timer(), calibrate_delay() will use calibrate_delay_direct() to calculate loops_per_jiffy (in which case loops_per_jiffy should really be renamed to timer_ticks_per_jiffy). Since the loops_per_jiffy will be based on timer ticks, __delay() should be implemented as a loop around read_current_timer().
Doing this makes the expensive loops_per_jiffy calculation go away (saving ~150ms on boot time on my machine) and fixes udelay() by making it safe in the face of independently scaling CPUs. The only prerequisite is that read_current_timer() is monotonically increasing across calls (and doesn't overflow within ~2000us).
There is a downside to this approach though. BogoMIPS is no longer "accurate" in that it reflects the BogoMIPS of the timer and not the CPU. On most SoC's the timer isn't running anywhere near as fast as the CPU so BogoMIPS will be ridiculously low (my timer runs at 4.8 MHz and thus my BogoMIPS is 9.6 compared to my CPU's 800). This shouldn't be too much of a concern though since BogoMIPS are bogus anyway (hence the name).
This loop is pretty much a copy of AVR's version.
--- arch/arm/include/asm/delay.h | 1 + arch/arm/lib/delay.c | 17 +++++++++++++++++ 2 files changed, 18 insertions(+), 0 deletions(-)
arch/arm/include/asm/delay.h +1 -0
@@ -41,6 +41,7 @@ extern void __const_udelay(unsigned long); __udelay(n))extern void set_delay_fn (void (*fn)(unsigned long )); +extern void read_current_timer_delay_loop (unsigned long loops);#endif /* defined(_ARM_DELAY_H) */
arch/arm/lib/delay.c +17 -0
@@ -9,6 +9,7 @@*/ #include <linux/module.h> #include <linux/delay.h> +#include <linux/timex.h> /* * Oh, if only we had a cycle counter...
@@ -23,6 +24,22 @@ void delay_loop(unsigned long loops) ); } +#ifdef ARCH_HAS_READ_CURRENT_TIMER +/* +* Assumes read_current_timer() is monotonically increasing +* across calls and wraps at most once within MAX_UDELAY_MS. +*/ +void read_current_timer_delay_loop (unsigned long loops) +{ +unsigned long bclock, now; + +read_current_timer (&bclock); +do { +read_current_timer (&now); + }while ((now - bclock) < loops); +} +#endif +static void (*delay_fn)(unsigned long ) = delay_loop;void set_delay_fn (void (*fn)(unsigned long ))
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
PATCH v3 4/4 msm: timer: Migrate to timer based __delay()
In reply to Stephen BoydWe have to provide a dummy set_mode for the DGT since we don't want it to ever turn off.
--- arch/arm/mach-msm/include/mach/timex.h | 1 + arch/arm/mach-msm/timer.c | 16 +++++++++++++++- 2 files changed, 16 insertions(+), 1 deletions(-)
arch/arm/mach-msm/include/mach/timex.h +1 -0
@@ -17,5 +17,6 @@#define __ASM_ARCH_MSM_TIMEX_H#define CLOCK_TICK_RATE 1000000 +#define ARCH_HAS_READ_CURRENT_TIMER #endif
arch/arm/mach-msm/timer.c +15 -1
@@ -84,6 +84,12 @@ static cycle_t msm_dgt_read(struct clocksource *cs)return readl (MSM_DGT_BASE +TIMER_COUNT_VAL ) >>MSM_DGT_SHIFT ; } +int read_current_timer (unsigned long *timer_val) +{ + *timer_val =readl (MSM_DGT_BASE +TIMER_COUNT_VAL ); +return 0 ; +} +static int msm_timer_set_next_event (unsigned long cycles,struct clock_event_device *evt) {
@@ -122,6 +128,12 @@ static void msm_timer_set_mode(enum clock_event_mode mode, } } +static void msm_timer_set_mode_nop (enum clock_event_mode mode, +struct clock_event_device *evt) +{ +/* The timer is always ticking so do nothing */ +} +static struct msm_clock msm_clocks[] = { { .clockevent = {
@@ -157,7 +169,7 @@ static struct msm_clock msm_clocks[] = { .shift = 32 + MSM_DGT_SHIFT, .rating = 300, .set_next_event = msm_timer_set_next_event, - .set_mode = msm_timer_set_mode, + .set_mode = msm_timer_set_mode_nop, }, .clocksource = { .name = "dg_timer",
@@ -218,6 +230,8 @@ static void __init msm_timer_init(void)clockevents_register_device (ce); } +writel (1 ,MSM_DGT_BASE +TIMER_ENABLE ); +set_delay_fn (read_current_timer_delay_loop); }struct sys_timer msm_timer = {
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.