Skip to contentSkip to conversation

64-bit sched_clock

PATCH New Russell King - ARM Linux 3 participants 20 messages Dec 16, 2010 – Dec 19, 2010

Subsystems (9)
lore
Share & tools
lore
#0 lore raw

cover 64-bit sched_clock

This patch series replaces existing sched_clock() implementations with a version which gives a full 64-bit nanosecond time from counters with less than 33 bits.

This series depends on the prevously posted clocksource patch series, and the notrace sched_clock patch, and the ftrace tree previously merged into my devel-stable branch. It also depends on a fix to the clocksource code to achieve the accuracy quoted below - though it'll build and work without.

We achieve the 64-bit nanosecond time by maintaining a pair of epochs - one for the counter and a corresponding one for the ns value. This allows us to calculate how many counter ticks have happened since the ns epoch was last updated, and therefore how much time has passed.

We use a lockless method to read and update the epoch values, eliminating any locking overhead. The epoch update happens at about 90% of the wrap period, resulting in less maintainence overhead.

Lastly, we allow platforms to provide constants for the counter->ns conversion.

The PXA sched_clock implementation using cnt32_to_63() builds to 72 bytes of code and 12 bytes of data, and provides 54 bits of nanoseconds, with an error of 0.00008%.

The PXA sched_clock implementation using this infrastructure builds to 84 bytes of code and 8 bytes of data, and provides 64 bits of nanoseconds, with an error of 0.00000002%.

The epoch updates in action on Versatile Express (24MHz 32-bit counter):

sched_clock: update 68391420 -> 50e0a246 0ns -> 162637292211ns
sched_clock: update 50e0a246 -> 372c24b7 162637292211ns -> 323625136880ns
sched_clock: update 372c24b7 -> 1d35c46a 323625136880ns -> 484433069633ns
sched_clock: update 1d35c46a -> 0367a736 484433069633ns -> 645350944761ns
...
sched_clock: update f6f7fd55 -> dd2d882e 32669955162290ns -> 32830883021959ns
sched_clock: update dd2d882e -> c36312cd 32830883021959ns -> 32991810879212ns
sched_clock: update c36312cd -> a9989d41 32991810879212ns -> 33152738734673ns
sched_clock: update a9989d41 -> 8fce2781 33152738734673ns -> 33313666587967ns

#1 lore raw

PATCH 1/13 ARM: sched_clock: provide common infrastructure for sched_clock()

Provide common sched_clock() infrastructure for platforms to use to create a 64-bit ns based sched_clock() implementation from a counter running at a non-variable clock rate.

This implementation is based upon maintaining an epoch for the counter and an epoch for the nanosecond time. When we desire a sched_clock() time, we calculate the number of counter ticks since the last epoch update, convert this to nanoseconds and add to the epoch nanoseconds.

We regularly refresh these epochs within the counter wrap interval. We perform a similar calculation as above, and store the new epochs.

We read and write the epochs in such a way that sched_clock() can easily (and locklessly) detect when an update is in progress, and repeat the loading of these constants when they're known not to be stable. The one caveat is that sched_clock() is not called in the middle of an update.

Finally, if the clock rate is known at compile time, the counter to ns conversion factors can be specified, allowing sched_clock() to be tightly optimized. We ensure that these factors are correct by providing an initialization function which performs a run-time check.

---
 arch/arm/Kconfig                   |    3 +
 arch/arm/include/asm/sched_clock.h |  116 ++++++++++++++++++++++++++++++++++++
 arch/arm/kernel/Makefile           |    1 +
 arch/arm/kernel/sched_clock.c      |   69 +++++++++++++++++++++
 4 files changed, 189 insertions(+), 0 deletions(-)
 create mode 100644 arch/arm/include/asm/sched_clock.h
 create mode 100644 arch/arm/kernel/sched_clock.c
arch/arm/Kconfig +3 -0
index 49778bb..ed7a0a7 100644
@@ -38,6 +38,9 @@ config HAVE_PWM
 config SYS_SUPPORTS_APM_EMULATION
 	bool
 
+config HAVE_SCHED_CLOCK
+	bool
+
 config GENERIC_GPIO
 	bool
 
arch/arm/include/asm/sched_clock.h new file +116 -0

116 lines not on this page · raw message

arch/arm/kernel/Makefile +1 -0
index 679851a..fd3ec49 100644
@@ -29,6 +29,7 @@ obj-$(CONFIG_MODULES)		+= armksyms.o module.o
 obj-$(CONFIG_ARTHUR)		+= arthur.o
 obj-$(CONFIG_ISA_DMA)		+= dma-isa.o
 obj-$(CONFIG_PCI)		+= bios32.o isa.o
+obj-$(CONFIG_HAVE_SCHED_CLOCK)	+= sched_clock.o
 obj-$(CONFIG_SMP)		+= smp.o
 obj-$(CONFIG_HAVE_ARM_SCU)	+= smp_scu.o
 obj-$(CONFIG_HAVE_ARM_TWD)	+= smp_twd.o
arch/arm/kernel/sched_clock.c new file +69 -0

69 lines not on this page · raw message

Signature · 1 line
1.6.2.5
#2 lore raw

PATCH 1/13 ARM: sched_clock: provide common infrastructure for sched_clock()

Quoted 3 lines · Russell King - ARM Linux

Provide common sched_clock() infrastructure for platforms to use to create a 64-bit ns based sched_clock() implementation from a counter running at a non-variable clock rate.

Some of the comments in the patch were only relevant to a previous version. He's an updated version with proper comments to avoid confusion.

 arch/arm/Kconfig                   |    3 +
 arch/arm/include/asm/sched_clock.h |  115 ++++++++++++++++++++++++++++++++++++
 arch/arm/kernel/Makefile           |    1 +
 arch/arm/kernel/sched_clock.c      |   69 +++++++++++++++++++++
 4 files changed, 188 insertions(+), 0 deletions(-)
arch/arm/Kconfig +3 -0
index 49778bb..ed7a0a7 100644
@@ -38,6 +38,9 @@ config HAVE_PWM
 config SYS_SUPPORTS_APM_EMULATION
 	bool
 
+config HAVE_SCHED_CLOCK
+	bool
+
 config GENERIC_GPIO
 	bool
 
arch/arm/include/asm/sched_clock.h new file +115 -0

115 lines not on this page · raw message

arch/arm/kernel/Makefile +1 -0
index 679851a..fd3ec49 100644
@@ -29,6 +29,7 @@ obj-$(CONFIG_MODULES)		+= armksyms.o module.o
 obj-$(CONFIG_ARTHUR)		+= arthur.o
 obj-$(CONFIG_ISA_DMA)		+= dma-isa.o
 obj-$(CONFIG_PCI)		+= bios32.o isa.o
+obj-$(CONFIG_HAVE_SCHED_CLOCK)	+= sched_clock.o
 obj-$(CONFIG_SMP)		+= smp.o
 obj-$(CONFIG_HAVE_ARM_SCU)	+= smp_scu.o
 obj-$(CONFIG_HAVE_ARM_TWD)	+= smp_twd.o
arch/arm/kernel/sched_clock.c new file +69 -0

69 lines not on this page · raw message

#3 lore raw

PATCH 2/13 ARM: ixp4xx: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert ixp4xx to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig              |    1 +
 arch/arm/mach-ixp4xx/common.c |   31 ++++++++++++++++++++-----------
 2 files changed, 21 insertions(+), 11 deletions(-)
arch/arm/Kconfig +1 -0
index ed7a0a7..6f58bce 100644
@@ -437,6 +437,7 @@ config ARCH_IXP4XX
 	select CPU_XSCALE
 	select GENERIC_GPIO
 	select GENERIC_CLOCKEVENTS
+	select HAVE_SCHED_CLOCK
 	select DMABOUNCE if PCI
 	help
 	  Support for Intel's IXP4XX (XScale) family of processors.
arch/arm/mach-ixp4xx/common.c +20 -11
index e0b91d8..4dbfcbb 100644
@@ -35,6 +35,7 @@
 #include <asm/pgtable.h>
 #include <asm/page.h>
 #include <asm/irq.h>
+#include <asm/sched_clock.h>
 
 #include <asm/mach/map.h>
 #include <asm/mach/irq.h>
@@ -399,6 +400,23 @@ void __init ixp4xx_sys_init(void)
 }
 
 /*
+ * sched_clock()
+ */
+static DEFINE_CLOCK_DATA(cd);
+
+unsigned long long notrace sched_clock(void)
+{
+	u32 cyc = *IXP4XX_OSTS;
+	return cyc_to_sched_clock(&cd, cyc, (u32)~0);
+}
+
+static void notrace ixp4xx_update_sched_clock(void)
+{
+	u32 cyc = *IXP4XX_OSTS;
+	update_sched_clock(&cd, cyc, (u32)~0);
+}
+
+/*
  * clocksource
  */
 static cycle_t ixp4xx_get_cycles(struct clocksource *cs)
@@ -418,18 +436,9 @@ unsigned long ixp4xx_timer_freq = FREQ;
 EXPORT_SYMBOL(ixp4xx_timer_freq);
 static void __init ixp4xx_clocksource_init(void)
 {
-	clocksource_register_hz(&clocksource_ixp4xx, ixp4xx_timer_freq);
-}
-
-/*
- * sched_clock()
- */
-unsigned long long notrace sched_clock(void)
-{
-	cycle_t cyc = ixp4xx_get_cycles(NULL);
-	struct clocksource *cs = &clocksource_ixp4xx;
+	init_sched_clock(&cd, ixp4xx_update_sched_clock, 32, ixp4xx_timer_freq);
 
-	return clocksource_cyc2ns(cyc, cs->mult, cs->shift);
+	clocksource_register_hz(&clocksource_ixp4xx, ixp4xx_timer_freq);
 }
 
 /*
Signature · 1 line
1.6.2.5
#4 lore raw

PATCH 3/13 ARM: mmp: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert mmp to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig         |    1 +
 arch/arm/mach-mmp/time.c |   32 +++++++++++---------------------
 2 files changed, 12 insertions(+), 21 deletions(-)
arch/arm/Kconfig +1 -0
index 6f58bce..0208e1e 100644
@@ -513,6 +513,7 @@ config ARCH_MMP
 	select ARCH_REQUIRE_GPIOLIB
 	select COMMON_CLKDEV
 	select GENERIC_CLOCKEVENTS
+	select HAVE_SCHED_CLOCK
 	select TICK_ONESHOT
 	select PLAT_PXA
 	select SPARSE_IRQ
arch/arm/mach-mmp/time.c +11 -21
index a2ea33d..aeb9ae2 100644
@@ -26,8 +26,8 @@
 #include <linux/io.h>
 #include <linux/irq.h>
 #include <linux/sched.h>
-#include <linux/cnt32_to_63.h>
 
+#include <asm/sched_clock.h>
 #include <mach/addr-map.h>
 #include <mach/regs-timers.h>
 #include <mach/regs-apbc.h>
@@ -42,23 +42,7 @@
 #define MAX_DELTA		(0xfffffffe)
 #define MIN_DELTA		(16)
 
-#define TCR2NS_SCALE_FACTOR	10
-
-static unsigned long tcr2ns_scale;
-
-static void __init set_tcr2ns_scale(unsigned long tcr_rate)
-{
-	unsigned long long v = 1000000000ULL << TCR2NS_SCALE_FACTOR;
-	do_div(v, tcr_rate);
-	tcr2ns_scale = v;
-	/*
-	 * We want an even value to automatically clear the top bit
-	 * returned by cnt32_to_63() without an additional run time
-	 * instruction. So if the LSB is 1 then round it up.
-	 */
-	if (tcr2ns_scale & 1)
-		tcr2ns_scale++;
-}
+static DEFINE_CLOCK_DATA(cd);
 
 /*
  * FIXME: the timer needs some delay to stablize the counter capture
@@ -77,8 +61,14 @@ static inline uint32_t timer_read(void)
 
 unsigned long long notrace sched_clock(void)
 {
-	unsigned long long v = cnt32_to_63(timer_read());
-	return (v * tcr2ns_scale) >> TCR2NS_SCALE_FACTOR;
+	u32 cyc = timer_read();
+	return cyc_to_sched_clock(&cd, cyc, (u32)~0);
+}
+
+static void notrace mmp_update_sched_clock(void)
+{
+	u32 cyc = timer_read();
+	update_sched_clock(&cd, cyc, (u32)~0);
 }
 
 static irqreturn_t timer_interrupt(int irq, void *dev_id)
@@ -185,7 +175,7 @@ void __init timer_init(int irq)
 {
 	timer_config();
 
-	set_tcr2ns_scale(CLOCK_TICK_RATE);
+	init_sched_clock(&cd, mmp_update_sched_clock, 32, CLOCK_TICK_RATE);
 
 	ckevt.mult = div_sc(CLOCK_TICK_RATE, NSEC_PER_SEC, ckevt.shift);
 	ckevt.max_delta_ns = clockevent_delta2ns(MAX_DELTA, &ckevt);
Signature · 1 line
1.6.2.5
#5 lore raw

PATCH 4/13 ARM: pxa: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert pxa to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig         |    1 +
 arch/arm/mach-pxa/time.c |   29 +++++++++--------------------
 2 files changed, 10 insertions(+), 20 deletions(-)
arch/arm/Kconfig +1 -0
index 0208e1e..23035d6 100644
@@ -593,6 +593,7 @@ config ARCH_PXA
 	select COMMON_CLKDEV
 	select ARCH_REQUIRE_GPIOLIB
 	select GENERIC_CLOCKEVENTS
+	select HAVE_SCHED_CLOCK
 	select TICK_ONESHOT
 	select PLAT_PXA
 	select SPARSE_IRQ
arch/arm/mach-pxa/time.c +9 -20
index b8d9dff..e7f64d9 100644
@@ -17,11 +17,11 @@
 #include <linux/interrupt.h>
 #include <linux/clockchips.h>
 #include <linux/sched.h>
-#include <linux/cnt32_to_63.h>
 
 #include <asm/div64.h>
 #include <asm/mach/irq.h>
 #include <asm/mach/time.h>
+#include <asm/sched_clock.h>
 #include <mach/regs-ost.h>
 
 /*
@@ -32,29 +32,18 @@
  * long as there is always less than 582 seconds between successive
  * calls to sched_clock() which should always be the case in practice.
  */
+static DEFINE_CLOCK_DATA(cd);
 
-#define OSCR2NS_SCALE_FACTOR 10
-
-static unsigned long oscr2ns_scale;
-
-static void __init set_oscr2ns_scale(unsigned long oscr_rate)
+unsigned long long notrace sched_clock(void)
 {
-	unsigned long long v = 1000000000ULL << OSCR2NS_SCALE_FACTOR;
-	do_div(v, oscr_rate);
-	oscr2ns_scale = v;
-	/*
-	 * We want an even value to automatically clear the top bit
-	 * returned by cnt32_to_63() without an additional run time
-	 * instruction. So if the LSB is 1 then round it up.
-	 */
-	if (oscr2ns_scale & 1)
-		oscr2ns_scale++;
+	u32 cyc = OSCR;
+	return cyc_to_sched_clock(&cd, cyc, (u32)~0);
 }
 
-unsigned long long notrace sched_clock(void)
+static void notrace pxa_update_sched_clock(void)
 {
-	unsigned long long v = cnt32_to_63(OSCR);
-	return (v * oscr2ns_scale) >> OSCR2NS_SCALE_FACTOR;
+	u32 cyc = OSCR;
+	update_sched_clock(&cd, cyc, (u32)~0);
 }
 
 
@@ -144,7 +133,7 @@ static void __init pxa_timer_init(void)
 	OIER = 0;
 	OSSR = OSSR_M0 | OSSR_M1 | OSSR_M2 | OSSR_M3;
 
-	set_oscr2ns_scale(clock_tick_rate);
+	init_sched_clock(&cd, pxa_update_sched_clock, 32, clock_tick_rate);
 
 	ckevt_pxa_osmr0.mult =
 		div_sc(clock_tick_rate, NSEC_PER_SEC, ckevt_pxa_osmr0.shift);
Signature · 1 line
1.6.2.5
#6 lore raw

PATCH 4/13 ARM: pxa: convert sched_clock() to use new infrastructure

Hello Russell,

a very minor nit ...

arch/arm/mach-pxa/time.c line 40

Quoted 1 line · Russell King - ARM Linux
arch/arm/mach-pxa/time.c +1 -0
+	return cyc_to_sched_clock(&cd, cyc, (u32)~0);

The value of ~0 is compiler implementation defined, so in general ~((u32)0) is saver. (Though I admit that on arm ~0 will probably always be -1, still I think using the save construct doesn't hurt here.)

Best regards
Uwe

Signature · 2 lines
Pengutronix e.K.                           | Uwe Kleine-K?nig            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
#7 lore raw

PATCH 4/13 ARM: pxa: convert sched_clock() to use new infrastructure

Quoted 9 lines · Uwe Kleine-K?nig

Hello Russell,

a very minor nit ...

arch/arm/mach-pxa/time.c line 40

Quoted 1 line · Russell King - ARM Linux
arch/arm/mach-pxa/time.c +1 -0
+	return cyc_to_sched_clock(&cd, cyc, (u32)~0);

The value of ~0 is compiler implementation defined, so in general ~((u32)0) is saver. (Though I admit that on arm ~0 will probably always be -1, still I think using the save construct doesn't hurt here.)

I've been debating about using something like CLOCKSOURCE_MASK() here, but haven't yet decided it to be worth it. I also toyed with the idea of storing the mask in the struct, but that adds extra expense when it's not needed - everyone should know how many bits are in their counter at build time, and if not, they shouldn't penalize those who do.

#8 lore raw

PATCH 5/13 ARM: sa1100: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert sa1100 to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig               |    1 +
 arch/arm/mach-sa1100/generic.c |   23 -----------------------
 arch/arm/mach-sa1100/time.c    |   30 ++++++++++++++++++++++++++++++
 3 files changed, 31 insertions(+), 23 deletions(-)
arch/arm/Kconfig +1 -0
index 23035d6..0e1a966 100644
@@ -642,6 +642,7 @@ config ARCH_SA1100
 	select CPU_FREQ
 	select GENERIC_CLOCKEVENTS
 	select HAVE_CLK
+	select HAVE_SCHED_CLOCK
 	select TICK_ONESHOT
 	select ARCH_REQUIRE_GPIOLIB
 	help
arch/arm/mach-sa1100/generic.c +0 -23
index 33b4969..fbc224d 100644
@@ -16,9 +16,7 @@
 #include <linux/pm.h>
 #include <linux/cpufreq.h>
 #include <linux/ioport.h>
-#include <linux/sched.h>	/* just for sched_clock() - funny that */
 #include <linux/platform_device.h>
-#include <linux/cnt32_to_63.h>
 
 #include <asm/div64.h>
 #include <mach/hardware.h>
@@ -110,27 +108,6 @@ unsigned int sa11x0_getspeed(unsigned int cpu)
 }
 
 /*
- * This is the SA11x0 sched_clock implementation.  This has
- * a resolution of 271ns, and a maximum value of 32025597s (370 days).
- *
- * The return value is guaranteed to be monotonic in that range as
- * long as there is always less than 582 seconds between successive
- * calls to this function.
- *
- *  ( * 1E9 / 3686400 => * 78125 / 288)
- */
-unsigned long long notrace sched_clock(void)
-{
-	unsigned long long v = cnt32_to_63(OSCR);
-
-	/* the <<1 gets rid of the cnt_32_to_63 top bit saving on a bic insn */
-	v *= 78125<<1;
-	do_div(v, 288<<1);
-
-	return v;
-}
-
-/*
  * Default power-off for SA1100
  */
 static void sa1100_power_off(void)
arch/arm/mach-sa1100/time.c +30 -0
index 96154f5..ae4f3d8 100644
@@ -12,12 +12,39 @@
 #include <linux/errno.h>
 #include <linux/interrupt.h>
 #include <linux/irq.h>
+#include <linux/sched.h>	/* just for sched_clock() - funny that */
 #include <linux/timex.h>
 #include <linux/clockchips.h>
 
 #include <asm/mach/time.h>
+#include <asm/sched_clock.h>
 #include <mach/hardware.h>
 
+/*
+ * This is the SA11x0 sched_clock implementation.
+ */
+static DEFINE_CLOCK_DATA(cd);
+
+/*
+ * Constants generated by clocks_calc_mult_shift(m, s, 3.6864MHz,
+ * NSEC_PER_SEC, 60).
+ * This gives a resolution of about 271ns and a wrap period of about 19min.
+ */
+#define SC_MULT		2275555556u
+#define SC_SHIFT	23
+
+unsigned long long notrace sched_clock(void)
+{
+	u32 cyc = OSCR;
+	return cyc_to_fixed_sched_clock(&cd, cyc, (u32)~0, SC_MULT, SC_SHIFT);
+}
+
+static void notrace sa1100_update_sched_clock(void)
+{
+	u32 cyc = OSCR;
+	update_sched_clock(&cd, cyc, (u32)~0);
+}
+
 #define MIN_OSCR_DELTA 2
 
 static irqreturn_t sa1100_ost0_interrupt(int irq, void *dev_id)
@@ -96,6 +123,9 @@ static void __init sa1100_timer_init(void)
 	OIER = 0;		/* disable any timer interrupts */
 	OSSR = 0xf;		/* clear status on all timers */
 
+	init_fixed_sched_clock(&cd, sa1100_update_sched_clock, 32,
+			       3686400, SC_MULT, SC_SHIFT);
+
 	ckevt_sa1100_osmr0.mult =
 		div_sc(3686400, NSEC_PER_SEC, ckevt_sa1100_osmr0.shift);
 	ckevt_sa1100_osmr0.max_delta_ns =
Signature · 1 line
1.6.2.5
#9 lore raw

PATCH 6/13 ARM: tegra: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert tegra to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig            |    1 +
 arch/arm/mach-tegra/timer.c |   22 ++++++++++++++++++++--
 2 files changed, 21 insertions(+), 2 deletions(-)
arch/arm/Kconfig +1 -0
index 0e1a966..ec7b027 100644
@@ -570,6 +570,7 @@ config ARCH_TEGRA
 	select GENERIC_CLOCKEVENTS
 	select GENERIC_GPIO
 	select HAVE_CLK
+	select HAVE_SCHED_CLOCK
 	select COMMON_CLKDEV
 	select ARCH_HAS_BARRIERS if CACHE_L2X0
 	select ARCH_HAS_CPUFREQ
arch/arm/mach-tegra/timer.c +20 -2
index 19a6f7e..cbd1082 100644
@@ -25,7 +25,6 @@
 #include <linux/clocksource.h>
 #include <linux/clk.h>
 #include <linux/io.h>
-#include <linux/cnt32_to_63.h>
 
 #include <asm/mach/time.h>
 #include <asm/localtimer.h>
@@ -110,9 +109,25 @@ static struct clocksource tegra_clocksource = {
 	.flags	= CLOCK_SOURCE_IS_CONTINUOUS,
 };
 
+static DEFINE_CLOCK_DATA(cd);
+
+/*
+ * Constants generated by clocks_calc_mult_shift(m, s, 1MHz, NSEC_PER_SEC, 60).
+ * This gives a resolution of about 1us and a wrap period of about 1h11min.
+ */
+#define SC_MULT		4194304000u
+#define SC_SHIFT	22
+
 unsigned long long notrace sched_clock(void)
 {
-	return cnt32_to_63(timer_readl(TIMERUS_CNTR_1US)) * 1000;
+	u32 cyc = timer_readl(TIMERUS_CNTR_1US);
+	return cyc_to_fixed_sched_clock(&cd, cyc, (u32)~0, SC_MULT, SC_SHIFT);
+}
+
+static void notrace tegra_update_sched_clock(void)
+{
+	u32 cyc = timer_readl(TIMERUS_CNTR_1US);
+	update_sched_clock(&cd, cyc, (u32)~0);
 }
 
 static irqreturn_t tegra_timer_interrupt(int irq, void *dev_id)
@@ -157,6 +172,9 @@ static void __init tegra_init_timer(void)
 		WARN(1, "Unknown clock rate");
 	}
 
+	init_fixed_sched_clock(&cd, tegra_update_sched_clock, 32,
+			       1000000, SC_MULT, SC_SHIFT);
+
 	if (clocksource_register_hz(&tegra_clocksource, 1000000)) {
 		printk(KERN_ERR "Failed to register clocksource\n");
 		BUG();
Signature · 1 line
1.6.2.5
#10 lore raw

PATCH 7/13 ARM: u300: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert u300 to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig           |    1 +
 arch/arm/mach-u300/timer.c |   17 +++++++++++++----
 2 files changed, 14 insertions(+), 4 deletions(-)
arch/arm/Kconfig +1 -0
index ec7b027..d5bd266 100644
@@ -790,6 +790,7 @@ config ARCH_U300
 	bool "ST-Ericsson U300 Series"
 	depends on MMU
 	select CPU_ARM926T
+	select HAVE_SCHED_CLOCK
 	select HAVE_TCM
 	select ARM_AMBA
 	select ARM_VIC
arch/arm/mach-u300/timer.c +13 -4
index 377ff7f..42b9f5c 100644
@@ -21,6 +21,7 @@
 #include <mach/hardware.h>
 
 /* Generic stuff */
+#include <asm/sched_clock.h>
 #include <asm/mach/map.h>
 #include <asm/mach/time.h>
 #include <asm/mach/irq.h>
@@ -352,12 +353,18 @@ static struct clocksource clocksource_u300_1mhz = {
  * this wraps around for now, since it is just a relative time
  * stamp. (Inspired by OMAP implementation.)
  */
+static DEFINE_CLOCK_DATA(cd);
+
 unsigned long long notrace sched_clock(void)
 {
-	return clocksource_cyc2ns(clocksource_u300_1mhz.read(
-				  &clocksource_u300_1mhz),
-				  clocksource_u300_1mhz.mult,
-				  clocksource_u300_1mhz.shift);
+	u32 cyc = readl(U300_TIMER_APP_VBASE + U300_TIMER_APP_GPT2CC);
+	return cyc_to_sched_clock(&cd, cyc, (u32)~0);
+}
+
+static void notrace u300_update_sched_clock(void)
+{
+	u32 cyc = readl(U300_TIMER_APP_VBASE + U300_TIMER_APP_GPT2CC);
+	update_sched_clock(&cd, cyc, (u32)~0);
 }
 
 
@@ -375,6 +382,8 @@ static void __init u300_timer_init(void)
 	clk_enable(clk);
 	rate = clk_get_rate(clk);
 
+	init_sched_clock(&cd, u300_update_sched_clock, 32, rate);
+
 	/*
 	 * Disable the "OS" and "DD" timers - these are designed for Symbian!
 	 * Example usage in cnh1601578 cpu subsystem pd_timer_app.c
Signature · 1 line
1.6.2.5
#11 lore raw

PATCH 8/13 ARM: iop: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert iop platforms to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig         |    1 +
 arch/arm/plat-iop/time.c |   14 +++++++++++---
 2 files changed, 12 insertions(+), 3 deletions(-)
arch/arm/Kconfig +1 -0
index d5bd266..58e2fe3 100644
@@ -992,6 +992,7 @@ config ARCH_ACORN
 config PLAT_IOP
 	bool
 	select GENERIC_CLOCKEVENTS
+	select HAVE_SCHED_CLOCK
 
 config PLAT_ORION
 	bool
arch/arm/plat-iop/time.c +11 -3
index 2bb7db4..a3babae 100644
@@ -49,15 +49,21 @@ static struct clocksource iop_clocksource = {
 	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
 };
 
+static DEFINE_CLOCK_DATA(cd);
+
 /*
  * IOP sched_clock() implementation via its clocksource.
  */
 unsigned long long notrace sched_clock(void)
 {
-	cycle_t cyc = iop_clocksource_read(NULL);
-	struct clocksource *cs = &iop_clocksource;
+	u32 cyc = 0xffffffffu - read_tcr1();
+	return cyc_to_sched_clock(&cd, cyc, (u32)~0);
+}
 
-	return clocksource_cyc2ns(cyc, cs->mult, cs->shift);
+static void notrace iop_update_sched_clock(void)
+{
+	u32 cyc = 0xffffffffu - read_tcr1();
+	update_sched_clock(&cd, cyc, (u32)~0);
 }
 
 /*
@@ -142,6 +148,8 @@ void __init iop_init_time(unsigned long tick_rate)
 {
 	u32 timer_ctl;
 
+	init_sched_clock(&cd, iop_update_sched_clock, 32, tick_rate);
+
 	ticks_per_jiffy = DIV_ROUND_CLOSEST(tick_rate, HZ);
 	iop_tick_rate = tick_rate;
 
Signature · 1 line
1.6.2.5
#12 lore raw

PATCH 9/13 ARM: nomadik: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert nomadik platforms to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/plat-nomadik/Kconfig |    1 +
 arch/arm/plat-nomadik/timer.c |   76 +++++------------------------------------
 2 files changed, 10 insertions(+), 67 deletions(-)
arch/arm/plat-nomadik/Kconfig +1 -0
index 5da3f97..187f4e8 100644
@@ -14,6 +14,7 @@ if PLAT_NOMADIK
 
 config HAS_MTU
 	bool
+	select HAVE_SCHED_CLOCK
 	help
 	  Support for Multi Timer Unit. MTU provides access
 	  to multiple interrupt generating programmable
arch/arm/plat-nomadik/timer.c +9 -67

106 lines not on this page · raw message

Signature · 1 line
1.6.2.5
#13 lore raw

PATCH 10/13 ARM: omap: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert omap to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig                 |    1 +
 arch/arm/plat-omap/counter_32k.c |   24 ++++++++++++++++++++++--
 2 files changed, 23 insertions(+), 2 deletions(-)
arch/arm/Kconfig +1 -0
index 58e2fe3..726279f 100644
@@ -839,6 +839,7 @@ config ARCH_OMAP
 	select ARCH_REQUIRE_GPIOLIB
 	select ARCH_HAS_CPUFREQ
 	select GENERIC_CLOCKEVENTS
+	select HAVE_SCHED_CLOCK
 	select ARCH_HAS_HOLES_MEMORYMODEL
 	help
 	  Support for TI's OMAP platform (OMAP1/2/3/4).
arch/arm/plat-omap/counter_32k.c +22 -2
index 0de0583..65bde61 100644
@@ -17,6 +17,8 @@
 #include <linux/clk.h>
 #include <linux/io.h>
 
+#include <asm/sched_clock.h>
+
 #include <plat/common.h>
 #include <plat/board.h>
 
@@ -109,10 +111,25 @@ static struct clocksource clocksource_32k = {
  * Returns current time from boot in nsecs. It's OK for this to wrap
  * around for now, as it's just a relative time stamp.
  */
+static DEFINE_CLOCK_DATA(cd);
+
+/*
+ * Constants generated by clocks_calc_mult_shift(m, s, 32768, NSEC_PER_SEC, 60).
+ * This gives a resolution of about 30us and a wrap period of about 36hrs.
+ */
+#define SC_MULT		4000000000u
+#define SC_SHIFT	17
+
 unsigned long long notrace sched_clock(void)
 {
-	return clocksource_cyc2ns(clocksource_32k.read(&clocksource_32k),
-				  clocksource_32k.mult, clocksource_32k.shift);
+	u32 cyc = clocksource_32k.read(&clocksource_32k);
+	return cyc_to_fixed_sched_clock(&cd, cyc, (u32)~0, SC_MULT, SC_SHIFT);
+}
+
+static void notrace omap_update_sched_clock(void)
+{
+	u32 cyc = clocksource_32k.read(&clocksource_32k);
+	update_sched_clock(&cd, cyc, (u32)~0);
 }
 
 /**
@@ -170,6 +187,9 @@ static int __init omap_init_clocksource_32k(void)
 
 		if (clocksource_register_hz(&clocksource_32k, 32768))
 			printk(err, clocksource_32k.name);
+
+		init_fixed_sched_clock(&cd, omap_update_sched_clock, 32,
+				       32768, SC_MULT, SC_SHIFT);
 	}
 	return 0;
 }
Signature · 1 line
1.6.2.5
#14 lore raw

PATCH 11/13 ARM: orion: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert orion platforms to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig           |    1 +
 arch/arm/plat-orion/time.c |   44 +++++++++-----------------------------------
 2 files changed, 10 insertions(+), 35 deletions(-)
arch/arm/Kconfig +1 -0
index 726279f..92e8c01 100644
@@ -997,6 +997,7 @@ config PLAT_IOP
 
 config PLAT_ORION
 	bool
+	select HAVE_SCHED_CLOCK
 
 config PLAT_PXA
 	bool
arch/arm/plat-orion/time.c +9 -35

72 lines not on this page · raw message

Signature · 1 line
1.6.2.5
#15 lore raw

PATCH 12/13 ARM: versatile: convert sched_clock() to use new infrastructure

In reply to Russell King - ARM Linux

Convert versatile platforms to use the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig                                   |    2 +
 arch/arm/mach-realview/core.c                      |   10 ++++
 arch/arm/mach-versatile/core.c                     |   10 ++++
 arch/arm/plat-versatile/include/plat/sched_clock.h |    6 +++
 arch/arm/plat-versatile/sched-clock.c              |   48 +++++++++++---------
 5 files changed, 54 insertions(+), 22 deletions(-)
 create mode 100644 arch/arm/plat-versatile/include/plat/sched_clock.h
arch/arm/Kconfig +2 -0
index 92e8c01..34311f4 100644
@@ -236,6 +236,7 @@ config ARCH_REALVIEW
 	bool "ARM Ltd. RealView family"
 	select ARM_AMBA
 	select COMMON_CLKDEV
+	select HAVE_SCHED_CLOCK
 	select ICST
 	select GENERIC_CLOCKEVENTS
 	select ARCH_WANT_OPTIONAL_GPIOLIB
@@ -250,6 +251,7 @@ config ARCH_VERSATILE
 	select ARM_AMBA
 	select ARM_VIC
 	select COMMON_CLKDEV
+	select HAVE_SCHED_CLOCK
 	select ICST
 	select GENERIC_CLOCKEVENTS
 	select ARCH_WANT_OPTIONAL_GPIOLIB
arch/arm/mach-realview/core.c +10 -0
index 07c0815..3d653e0 100644
@@ -52,6 +52,8 @@
 #include <mach/irqs.h>
 #include <plat/timer-sp.h>
 
+#include <plat/sched_clock.h>
+
 #include "core.h"
 
 /* used by entry-macro.S and platsmp.c */
@@ -658,6 +660,12 @@ void realview_leds_event(led_event_t ledevt)
 #endif	/* CONFIG_LEDS */
 
 /*
+ * The sched_clock counter
+ */
+#define REFCOUNTER		(__io_address(REALVIEW_SYS_BASE) + \
+				 REALVIEW_SYS_24MHz_OFFSET)
+
+/*
  * Where is the timer (VA)?
  */
 void __iomem *timer0_va_base;
@@ -672,6 +680,8 @@ void __init realview_timer_init(unsigned int timer_irq)
 {
 	u32 val;
 
+	versatile_sched_clock_init(REFCOUNTER, 24000000);
+
 	/* 
 	 * set clock frequency: 
 	 *	REALVIEW_REFCLK is 32KHz
arch/arm/mach-versatile/core.c +10 -0
index e38acb0..56cdc22 100644
@@ -51,6 +51,8 @@
 #include <mach/platform.h>
 #include <plat/timer-sp.h>
 
+#include <plat/sched_clock.h>
+
 #include "core.h"
 
 /*
@@ -886,6 +888,12 @@ void __init versatile_init(void)
 }
 
 /*
+ * The sched_clock counter
+ */
+#define REFCOUNTER		(__io_address(VERSATILE_SYS_BASE) + \
+				 VERSATILE_SYS_24MHz_OFFSET)
+
+/*
  * Where is the timer (VA)?
  */
 #define TIMER0_VA_BASE		 __io_address(VERSATILE_TIMER0_1_BASE)
@@ -900,6 +908,8 @@ static void __init versatile_timer_init(void)
 {
 	u32 val;
 
+	versatile_sched_clock_init(REFCOUNTER, 24000000);
+
 	/* 
 	 * set clock frequency: 
 	 *	VERSATILE_REFCLK is 32KHz
arch/arm/plat-versatile/include/plat/sched_clock.h new file +6 -0
new file mode 100644
index 0000000..5c3e4fc
@@ -0,0 +1,6 @@
+#ifndef ARM_PLAT_SCHED_CLOCK_H
+#define ARM_PLAT_SCHED_CLOCK_H
+
+void versatile_sched_clock_init(void __iomem *, unsigned long);
+
+#endif
arch/arm/plat-versatile/sched-clock.c +26 -22

63 lines not on this page · raw message

Signature · 1 line
1.6.2.5
#16 lore raw

PATCH 13/13 ARM: vexpress: add sched_clock() for Versatile Express

In reply to Russell King - ARM Linux

Add a sched_clock() implementation to Versatile Express using the new sched_clock() infrastructure for extending 32bit counters to full 64-bit nanoseconds.

---
 arch/arm/Kconfig                 |    1 +
 arch/arm/mach-vexpress/v2m.c     |    4 ++++
 arch/arm/plat-versatile/Makefile |    4 +---
 3 files changed, 6 insertions(+), 3 deletions(-)
arch/arm/Kconfig +1 -0
index 34311f4..17b5ff0 100644
@@ -268,6 +268,7 @@ config ARCH_VEXPRESS
 	select COMMON_CLKDEV
 	select GENERIC_CLOCKEVENTS
 	select HAVE_CLK
+	select HAVE_SCHED_CLOCK
 	select ICST
 	select PLAT_VERSATILE
 	help
arch/arm/mach-vexpress/v2m.c +4 -0
index 7eaa232..8c28310 100644
@@ -22,6 +22,8 @@
 #include <mach/clkdev.h>
 #include <mach/motherboard.h>
 
+#include <plat/sched_clock.h>
+
 #include <plat/timer-sp.h>
 
 #include "core.h"
@@ -50,6 +52,8 @@ void __init v2m_map_io(struct map_desc *tile, size_t num)
 
 static void __init v2m_timer_init(void)
 {
+	versatile_sched_clock_init(MMIO_P2V(V2M_SYS_24MHZ), 24000000);
+
 	writel(0, MMIO_P2V(V2M_TIMER0) + TIMER_CTRL);
 	writel(0, MMIO_P2V(V2M_TIMER1) + TIMER_CTRL);
 
arch/arm/plat-versatile/Makefile +1 -3
index 5cf88e8..230eb37 100644
@@ -1,7 +1,5 @@
-obj-y	:= clock.o
+obj-y	:= clock.o sched-clock.o
 obj-$(CONFIG_ARM_TIMER_SP804) += timer-sp.o
-obj-$(CONFIG_ARCH_REALVIEW) += sched-clock.o
-obj-$(CONFIG_ARCH_VERSATILE) += sched-clock.o
 ifeq ($(CONFIG_LEDS_CLASS),y)
 obj-$(CONFIG_ARCH_REALVIEW) += leds.o
 obj-$(CONFIG_ARCH_VERSATILE) += leds.o
Signature · 1 line
1.6.2.5
#17 lore raw

cover 64-bit sched_clock

In reply to Russell King - ARM Linux

There is one issue which remains with this code - that is the initialization ordering.

init/main.c:

	setup_arch(&command_line);
	...
	sched_init();
	...
	early_irq_init();
	init_IRQ();
	...
	timekeeping_init();
	time_init();

sched_init() calls init_idle() which calls sched_clock() to set the start time for the idle thread. However, our sched_clock() is not initialized (in general) until init/main.c calls time_init().

This results in a big step in the sched_clock() return value during boot, which is probably not desirable. At the moment, I work around this by explicitly setting the epoch_ns to zero after the first update callback - but this means that sched_clock() will only start at this point, which may not be desirable for everyone.

Some sched_clock() implementations rely upon clocksource code (they use the clocksource read function) and this is only initialized when the clocksource is registered. There's also the matter of the clock rate.

It's probably not a good idea to register clocksources before timekeeping_init() has happened either.

I can provide an init_early() method in the machine description which setup_arch() can call as the last thing it does - this will happen after the ->reserve and ->map_io callbacks, but before interrupts and timers have been setup. This is the ideal time to setup the sched_clock(). However, that means digging through lots of arch code to sort out what can be moved where, and that's going to be inherently unreliable.

I think init_early() has merit in other ways - it'll allow us to remove crap from the ->map_io callbacks which is doing stuff like registering clk structures and so forth.

Any other suggestions?

#18 lore raw

cover 64-bit sched_clock

2010/12/16 Russell King - ARM Linux <[email protected]>:

Quoted 11 lines

I can provide an init_early() method in the machine description which setup_arch() can call as the last thing it does - this will happen after the ->reserve and ->map_io callbacks, but before interrupts and timers have been setup. ?This is the ideal time to setup the sched_clock(). ?However, that means digging through lots of arch code to sort out what can be moved where, and that's going to be inherently unreliable.

I think init_early() has merit in other ways - it'll allow us to remove crap from the ->map_io callbacks which is doing stuff like registering clk structures and so forth.

I agree. I was contemplating a clock_init() or similar a way back for setting up clock trees early. We would have to move clock tree registration in front of the init_sched_clock() call in the init_early() handler in many cases since the rate often comes from the clock framework, but this is indeed where it belongs.

So a clear ACK for this approach.

Yours,
Linus Walleij

#19 lore raw

cover 64-bit sched_clock

Quoted 14 lines · Linus Walleij

2010/12/16 Russell King - ARM Linux <[email protected]>:

Quoted 11 lines

I can provide an init_early() method in the machine description which setup_arch() can call as the last thing it does - this will happen after the ->reserve and ->map_io callbacks, but before interrupts and timers have been setup. ?This is the ideal time to setup the sched_clock(). ?However, that means digging through lots of arch code to sort out what can be moved where, and that's going to be inherently unreliable.

I think init_early() has merit in other ways - it'll allow us to remove crap from the ->map_io callbacks which is doing stuff like registering clk structures and so forth.

Quoted 5 lines

I agree. I was contemplating a clock_init() or similar a way back for setting up clock trees early. We would have to move clock tree registration in front of the init_sched_clock() call in the init_early() handler in many cases since the rate often comes from the clock framework, but this is indeed where it belongs.

Yes, that was indeed one of the subsystems that I thought platforms should initialize there - one of my motivations is in connection with 6433/1 and 6434/1.

If we have everyone setting up their clock subsystems in init_early, then we can have both the TWD and SP804 always getting their clock rates from the clk API, rather than having the conditional approach in those two patches.