Skip to contentSkip to conversation

CONFIG_PRINTK_TIME broken on git HEAD ?

DISCUSSION Applied Gabriel C 4 participants 10 messages May 7, 2008 – May 9, 2008

lore
Share & tools
lore

Gabriel C

author
#0 lore raw

Hi all,

I've updated today to 2.6.26-rc1-00166-gc0a1811 , running an 64bit kernel at the moment.

dmesg shows here :

...

[    0.000000] Allocating PCI resources starting at d0000000 (gap: cf600000:2f800000)
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PERCPU: Allocating 42756 bytes of per cpu data
[    0.000000] NR_CPUS: 8, nr_cpu_ids: 4
[4294014.506571] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 1010753
[4294014.506571] Kernel command line: root=/dev/sdb1 ro debug vga=0x317
[4294014.506571] Initializing CPU#0
[4294014.506571] Preemptible RCU implementation.
[4294014.506571] PID hash table entries: 4096 (order: 12, 32768 bytes) 

...

Is this a known issue ?

Regards,

Gabriel

#1 lore raw
Quoted 3 lines · On Thu, 08 May 2008 00:26:04 +0200 Gabriel C <[email protected]> wrote:

Hi all,

I've updated today to 2.6.26-rc1-00166-gc0a1811 , running an 64bit kernel at the moment.

Which kernel were you runing previously?

Quoted 15 lines

dmesg shows here :

...

[    0.000000] Allocating PCI resources starting at d0000000 (gap: cf600000:2f800000)
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PERCPU: Allocating 42756 bytes of per cpu data
[    0.000000] NR_CPUS: 8, nr_cpu_ids: 4
[4294014.506571] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 1010753
[4294014.506571] Kernel command line: root=/dev/sdb1 ro debug vga=0x317
[4294014.506571] Initializing CPU#0
[4294014.506571] Preemptible RCU implementation.
[4294014.506571] PID hash table entries: 4096 (order: 12, 32768 bytes) 
Quoted 5 lines

...

Is this a known issue ?

Not to me.

author
#2 lore raw
Quoted 8 lines · Andrew Morton
Quoted 3 lines · On Thu, 08 May 2008 00:26:04 +0200 Gabriel C <[email protected]> wrote:

Hi all,

I've updated today to 2.6.26-rc1-00166-gc0a1811 , running an 64bit kernel at the moment.

Which kernel were you runing previously?

2.6.25* , tested 2.6.25-git* , 2.6.26-rc1 , and 2.6.26-rc1-00065-g5717922.

Quoted 1 line

From my log 2.6.26-rc1-00065-g5717922 was fine.

...

May  6 00:13:14 thor kernel: [    0.000000] Allocating PCI resources starting at d0000000 (gap: cf600000:2f800000)
May  6 00:13:14 thor kernel: [    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
May  6 00:13:14 thor kernel: [    0.000000] PERCPU: Allocating 41732 bytes of per cpu data
May  6 00:13:14 thor kernel: [    0.000000] Kernel command line: root=/dev/sdb1 ro debug vga=0x317
May  6 00:13:14 thor kernel: [    0.000000] Preemptible RCU implementation.
May  6 00:13:14 thor kernel: [    0.000000] Extended CMOS year: 2000
May  6 00:13:14 thor kernel: [    0.000000] TSC calibrated against PM_TIMER
May  6 00:13:14 thor kernel: [    0.000014] time.c: Detected 2499.952 MHz processor.
May  6 00:13:14 thor kernel: [    0.000032] console [tty0] enabled
May  6 00:13:14 thor kernel: [    0.000999] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
May  6 00:13:14 thor kernel: [    0.000999] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
May  6 00:13:14 thor kernel: [    0.000999] Checking aperture...

...

Quoted 16 lines
Quoted 16 lines

dmesg shows here :

...

[    0.000000] Allocating PCI resources starting at d0000000 (gap: cf600000:2f800000)
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PERCPU: Allocating 42756 bytes of per cpu data
[    0.000000] NR_CPUS: 8, nr_cpu_ids: 4
[4294014.506571] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 1010753
[4294014.506571] Kernel command line: root=/dev/sdb1 ro debug vga=0x317
[4294014.506571] Initializing CPU#0
[4294014.506571] Preemptible RCU implementation.
[4294014.506571] PID hash table entries: 4096 (order: 12, 32768 bytes) 
Quoted 8 lines
Quoted 5 lines

...

Is this a known issue ?

Not to me.

Gabriel

author
#3 lore raw
Quoted 29 lines · Gabriel C
Quoted 7 lines · Andrew Morton
Quoted 3 lines · On Thu, 08 May 2008 00:26:04 +0200 Gabriel C <[email protected]> wrote:

Hi all,

I've updated today to 2.6.26-rc1-00166-gc0a1811 , running an 64bit kernel at the moment.

Which kernel were you runing previously?

2.6.25* , tested 2.6.25-git* , 2.6.26-rc1 , and 2.6.26-rc1-00065-g5717922.

Quoted 1 line

From my log 2.6.26-rc1-00065-g5717922 was fine.

...

May  6 00:13:14 thor kernel: [    0.000000] Allocating PCI resources starting at d0000000 (gap: cf600000:2f800000)
May  6 00:13:14 thor kernel: [    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
May  6 00:13:14 thor kernel: [    0.000000] PERCPU: Allocating 41732 bytes of per cpu data
May  6 00:13:14 thor kernel: [    0.000000] Kernel command line: root=/dev/sdb1 ro debug vga=0x317
May  6 00:13:14 thor kernel: [    0.000000] Preemptible RCU implementation.
May  6 00:13:14 thor kernel: [    0.000000] Extended CMOS year: 2000
May  6 00:13:14 thor kernel: [    0.000000] TSC calibrated against PM_TIMER
May  6 00:13:14 thor kernel: [    0.000014] time.c: Detected 2499.952 MHz processor.
May  6 00:13:14 thor kernel: [    0.000032] console [tty0] enabled
May  6 00:13:14 thor kernel: [    0.000999] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
May  6 00:13:14 thor kernel: [    0.000999] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
May  6 00:13:14 thor kernel: [    0.000999] Checking aperture...
Quoted 25 lines

...

Quoted 22 lines
Quoted 15 lines

dmesg shows here :

...

[    0.000000] Allocating PCI resources starting at d0000000 (gap: cf600000:2f800000)
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PERCPU: Allocating 42756 bytes of per cpu data
[    0.000000] NR_CPUS: 8, nr_cpu_ids: 4
[4294014.506571] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 1010753
[4294014.506571] Kernel command line: root=/dev/sdb1 ro debug vga=0x317
[4294014.506571] Initializing CPU#0
[4294014.506571] Preemptible RCU implementation.
[4294014.506571] PID hash table entries: 4096 (order: 12, 32768 bytes) 
Quoted 5 lines

...

Is this a known issue ?

Not to me.

I've bisected it.

git bisect log
git-bisect start
# good: [2ddcca36c8bcfa251724fe342c8327451988be0d] Linux 2.6.26-rc1
git-bisect good 2ddcca36c8bcfa251724fe342c8327451988be0d
# bad: [3de2403e6659d71b36ec820dc9b942762ddfe6eb] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc-2.6
git-bisect bad 3de2403e6659d71b36ec820dc9b942762ddfe6eb
# bad: [aac6abca858386438d9a7233c3471d2ecfa2f704] sched: default to n for GROUP_SCHED and FAIR_GROUP_SCHED
git-bisect bad aac6abca858386438d9a7233c3471d2ecfa2f704
# good: [45ea2103d8856454503b30464cc1dba378748d00] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86-fixes
git-bisect good 45ea2103d8856454503b30464cc1dba378748d00
# good: [5717922a1b8cc477f45a0f4f11fe619392ba7133] Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/linux-2.6-kgdb
git-bisect good 5717922a1b8cc477f45a0f4f11fe619392ba7133
# good: [b328ca182f01c2a04b85e0ee8a410720b104fbcc] sched: fix hrtick_start_fair and CPU-Hotplug
git-bisect good b328ca182f01c2a04b85e0ee8a410720b104fbcc
# good: [712555ee4f873515612f89554ad1a3fda5fa887e] sched: fix missing locking in sched_domains code
git-bisect good 712555ee4f873515612f89554ad1a3fda5fa887e
# good: [dfbf4a1bc319f0f9a31e39b2da1fa5c55e85af89] sched: fix cpu clock
git-bisect good dfbf4a1bc319f0f9a31e39b2da1fa5c55e85af89
# bad: [3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd] sched: add optional support for CONFIG_HAVE_UNSTABLE_SCHED_CLOCK git-bisect bad 3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd
# good: [a5574cf65b5f03ce9ade3918764fe22e5e2371e3] sched, x86: add HAVE_UNSTABLE_SCHED_CLOCK
git-bisect good a5574cf65b5f03ce9ade3918764fe22e5e2371e3

cat ./.git/refs/bisect/bad
3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd

git show 3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd commit 3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd
Author: Peter Zijlstra <[email protected]>
Date: Sat May 3 18:29:28 2008 +0200

sched: add optional support for CONFIG_HAVE_UNSTABLE_SCHED_CLOCK

this replaces the rq->clock stuff (and possibly cpu_clock()).

- architectures that have an 'imperfect' hardware clock can set CONFIG_HAVE_UNSTABLE_SCHED_CLOCK

- the 'jiffie' window might be superfulous when we update tick_gtod before the __update_sched_clock() call in sched_clock_tick()

- cpu_clock() might be implemented as:

sched_clock_cpu(smp_processor_id())

if the accuracy proves good enough - how far can TSC drift in a single jiffie when considering the filtering and idle hooks?

[ [email protected]: various fixes and cleanups ]

Gabriel

#4 lore raw
Quoted 58 lines · Gabriel C
Quoted 16 lines
Quoted 1 line

From my log 2.6.26-rc1-00065-g5717922 was fine.

May  6 00:13:14 thor kernel: [    0.000000] Allocating PCI resources starting at d0000000 (gap: cf600000:2f800000)
May  6 00:13:14 thor kernel: [    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
May  6 00:13:14 thor kernel: [    0.000000] PERCPU: Allocating 41732 bytes of per cpu data
May  6 00:13:14 thor kernel: [    0.000000] Kernel command line: root=/dev/sdb1 ro debug vga=0x317
May  6 00:13:14 thor kernel: [    0.000000] Preemptible RCU implementation.
May  6 00:13:14 thor kernel: [    0.000000] Extended CMOS year: 2000
May  6 00:13:14 thor kernel: [    0.000000] TSC calibrated against PM_TIMER
May  6 00:13:14 thor kernel: [    0.000014] time.c: Detected 2499.952 MHz processor.
May  6 00:13:14 thor kernel: [    0.000032] console [tty0] enabled
May  6 00:13:14 thor kernel: [    0.000999] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
May  6 00:13:14 thor kernel: [    0.000999] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
May  6 00:13:14 thor kernel: [    0.000999] Checking aperture...
Quoted 16 lines

...

Quoted 14 lines
Quoted 14 lines

dmesg shows here :

...

[    0.000000] Allocating PCI resources starting at d0000000 (gap: cf600000:2f800000)
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PERCPU: Allocating 42756 bytes of per cpu data
[    0.000000] NR_CPUS: 8, nr_cpu_ids: 4
[4294014.506571] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 1010753
[4294014.506571] Kernel command line: root=/dev/sdb1 ro debug vga=0x317
[4294014.506571] Initializing CPU#0
[4294014.506571] Preemptible RCU implementation.
[4294014.506571] PID hash table entries: 4096 (order: 12, 32768 bytes) 

I've bisected it.

cat ./.git/refs/bisect/bad
3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd

git show 3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd commit 3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd
Author: Peter Zijlstra <[email protected]>
Date: Sat May 3 18:29:28 2008 +0200

sched: add optional support for CONFIG_HAVE_UNSTABLE_SCHED_CLOCK

this replaces the rq->clock stuff (and possibly cpu_clock()).

- architectures that have an 'imperfect' hardware clock can set CONFIG_HAVE_UNSTABLE_SCHED_CLOCK

- the 'jiffie' window might be superfulous when we update tick_gtod before the __update_sched_clock() call in sched_clock_tick()

- cpu_clock() might be implemented as:

sched_clock_cpu(smp_processor_id())

Quoted 8 lines

if the accuracy proves good enough - how far can TSC drift in a single jiffie when considering the filtering and idle hooks?

[ [email protected]: various fixes and cleanups ]

Ok, so the issue is that the clock doesn't start at 0, but at:
4294014.506571 ?

I guess that's a side-effect of using ktime to base the clock off. Thomas, do we have a clock that is uptime based?

author
#5 lore raw
Quoted 63 lines · Peter Zijlstra
Quoted 61 lines · Gabriel C
Quoted 16 lines
Quoted 1 line

From my log 2.6.26-rc1-00065-g5717922 was fine.

May  6 00:13:14 thor kernel: [    0.000000] Allocating PCI resources starting at d0000000 (gap: cf600000:2f800000)
May  6 00:13:14 thor kernel: [    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
May  6 00:13:14 thor kernel: [    0.000000] PERCPU: Allocating 41732 bytes of per cpu data
May  6 00:13:14 thor kernel: [    0.000000] Kernel command line: root=/dev/sdb1 ro debug vga=0x317
May  6 00:13:14 thor kernel: [    0.000000] Preemptible RCU implementation.
May  6 00:13:14 thor kernel: [    0.000000] Extended CMOS year: 2000
May  6 00:13:14 thor kernel: [    0.000000] TSC calibrated against PM_TIMER
May  6 00:13:14 thor kernel: [    0.000014] time.c: Detected 2499.952 MHz processor.
May  6 00:13:14 thor kernel: [    0.000032] console [tty0] enabled
May  6 00:13:14 thor kernel: [    0.000999] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
May  6 00:13:14 thor kernel: [    0.000999] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
May  6 00:13:14 thor kernel: [    0.000999] Checking aperture...
Quoted 16 lines

...

Quoted 14 lines
Quoted 14 lines

dmesg shows here :

...

[    0.000000] Allocating PCI resources starting at d0000000 (gap: cf600000:2f800000)
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PERCPU: Allocating 42756 bytes of per cpu data
[    0.000000] NR_CPUS: 8, nr_cpu_ids: 4
[4294014.506571] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 1010753
[4294014.506571] Kernel command line: root=/dev/sdb1 ro debug vga=0x317
[4294014.506571] Initializing CPU#0
[4294014.506571] Preemptible RCU implementation.
[4294014.506571] PID hash table entries: 4096 (order: 12, 32768 bytes) 

I've bisected it.

cat ./.git/refs/bisect/bad
3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd

git show 3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd commit 3e51f33fcc7f55e6df25d15b55ed10c8b4da84cd
Author: Peter Zijlstra <[email protected]>
Date: Sat May 3 18:29:28 2008 +0200

sched: add optional support for CONFIG_HAVE_UNSTABLE_SCHED_CLOCK

this replaces the rq->clock stuff (and possibly cpu_clock()).

- architectures that have an 'imperfect' hardware clock can set CONFIG_HAVE_UNSTABLE_SCHED_CLOCK

- the 'jiffie' window might be superfulous when we update tick_gtod before the __update_sched_clock() call in sched_clock_tick()

- cpu_clock() might be implemented as:

sched_clock_cpu(smp_processor_id())

if the accuracy proves good enough - how far can TSC drift in a single jiffie when considering the filtering and idle hooks?

Quoted 8 lines
Quoted 5 lines

[ [email protected]: various fixes and cleanups ]

Ok, so the issue is that the clock doesn't start at 0, but at:
4294014.506571 ?

At least on my box , yes.

There is a similar report http://lkml.org/lkml/2008/5/8/100 , there it jumps to 42949372.961619

Quoted 3 lines

I guess that's a side-effect of using ktime to base the clock off. Thomas, do we have a clock that is uptime based?

Gabriel

#6 lore raw
Quoted 5 lines · Gabriel C
Quoted 2 lines

Ok, so the issue is that the clock doesn't start at 0, but at:
4294014.506571 ?

At least on my box , yes.

Does this work for you?

---

Subject: sched: fixup sched-clock offset
From: Peter Zijlstra <[email protected]>

It could happen that ktime doesn't start at 0.

---
 kernel/sched_clock.c |   16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
kernel/sched_clock.c +9 -7
Index: linux-2.6-2/kernel/sched_clock.c
===================================================================
@@ -59,21 +59,23 @@ static inline struct sched_clock_data *c
 	return &per_cpu(sched_clock_data, cpu);
 }
 
+static __read_mostly u64 ktime_offset;
+
 void sched_clock_init(void)
 {
-	u64 ktime_now = ktime_to_ns(ktime_get());
-	u64 now = 0;
 	int cpu;
 
+	ktime_offset = ktime_to_ns(ktime_get());
+
 	for_each_possible_cpu(cpu) {
 		struct sched_clock_data *scd = cpu_sdc(cpu);
 
 		scd->lock = (raw_spinlock_t)__RAW_SPIN_LOCK_UNLOCKED;
 		scd->prev_jiffies = jiffies;
-		scd->prev_raw = now;
-		scd->tick_raw = now;
-		scd->tick_gtod = ktime_now;
-		scd->clock = ktime_now;
+		scd->prev_raw = 0;
+		scd->tick_raw = 0;
+		scd->tick_gtod = 0;
+		scd->clock = 0;
 	}
 }
 
@@ -177,7 +179,7 @@ void sched_clock_tick(void)
 	WARN_ON_ONCE(!irqs_disabled());
 
 	now = sched_clock();
-	now_gtod = ktime_to_ns(ktime_get());
+	now_gtod = ktime_to_ns(ktime_get()) - ktime_offset;
 
 	__raw_spin_lock(&scd->lock);
 	__update_sched_clock(scd, now);
Applied
#7 lore raw
Quoted 4 lines · Peter Zijlstra

Subject: sched: fixup sched-clock offset
From: Peter Zijlstra <[email protected]>

It could happen that ktime doesn't start at 0.

applied, thanks Peter. I'm wondering which clocks do this - HPET perhaps?

Ingo

author
#9 lore raw
In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Gabriel C
Quoted 2 lines

Ok, so the issue is that the clock doesn't start at 0, but at:
4294014.506571 ?

At least on my box , yes.

Does this work for you?

No it does not , I still get :

....

[    0.000000] PERCPU: Allocating 42756 bytes of per cpu data
[    0.000000] NR_CPUS: 8, nr_cpu_ids: 4
[4294014.506571] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 1010723                                                               

...

Quoted 59 lines

---

Subject: sched: fixup sched-clock offset
From: Peter Zijlstra <[email protected]>

It could happen that ktime doesn't start at 0.

---
 kernel/sched_clock.c |   16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

Index: linux-2.6-2/kernel/sched_clock.c
===================================================================

kernel/sched_clock.c +9 -7
@@ -59,21 +59,23 @@ static inline struct sched_clock_data *c
 	return &per_cpu(sched_clock_data, cpu);
 }
 
+static __read_mostly u64 ktime_offset;
+
 void sched_clock_init(void)
 {
-	u64 ktime_now = ktime_to_ns(ktime_get());
-	u64 now = 0;
 	int cpu;
 
+	ktime_offset = ktime_to_ns(ktime_get());
+
 	for_each_possible_cpu(cpu) {
 		struct sched_clock_data *scd = cpu_sdc(cpu);
 
 		scd->lock = (raw_spinlock_t)__RAW_SPIN_LOCK_UNLOCKED;
 		scd->prev_jiffies = jiffies;
-		scd->prev_raw = now;
-		scd->tick_raw = now;
-		scd->tick_gtod = ktime_now;
-		scd->clock = ktime_now;
+		scd->prev_raw = 0;
+		scd->tick_raw = 0;
+		scd->tick_gtod = 0;
+		scd->clock = 0;
 	}
 }
 
@@ -177,7 +179,7 @@ void sched_clock_tick(void)
 	WARN_ON_ONCE(!irqs_disabled());
 
 	now = sched_clock();
-	now_gtod = ktime_to_ns(ktime_get());
+	now_gtod = ktime_to_ns(ktime_get()) - ktime_offset;
 
 	__raw_spin_lock(&scd->lock);
 	__update_sched_clock(scd, now);
author
#8 lore raw
In reply to Ingo Molnar
Quoted 9 lines · Ingo Molnar
Quoted 4 lines · Peter Zijlstra

Subject: sched: fixup sched-clock offset
From: Peter Zijlstra <[email protected]>

It could happen that ktime doesn't start at 0.

applied, thanks Peter. I'm wondering which clocks do this - HPET perhaps?

The box is using TSC clocksource right now , for some reason in >=2.6.25* kernels HPET registered to late.

http://lkml.org/lkml/2008/5/8/164

Quoted 3 lines

Ingo

Gabriel