New: jiffies counter leaps on CPU frequency changes
http://bugzilla.kernel.org/show_bug.cgi?id=9475
Summary: jiffies counter leaps on CPU frequency changes
Product: Power Management
Version: 2.5
KernelVersion: 2.6.24-rc3
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: cpufreq
AssignedTo: [email protected]
ReportedBy: [email protected]
Most recent kernel where this bug did not occur: unknown. I'm sorry, I can't test 2.6.23 because of some dm-crypt issue.
Distribution: Gentoo.
Hardware Environment:
st3@morte ~ $ cat /proc/cpuinfo
processor : 0vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : Intel(R) Pentium(R) M processor 1.50GHz
stepping : 6
cpu MHz : 1500.000
cache size : 2048 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr mce cx8 sep mtrr pge mca cmov pat
clflush dts acpi mmx fxsr sse sse2 ss tm pbe bts est tm2
bogomips : 2993.49
clflush size : 64
Software environment: acpi-cpufreq module.
Problem Description:
when I switch the CPU frequency, the jiffies counter gets scaled by the ratio of <old_frequency/new_frequency>, e.g.:
morte ~ # cpuspeedy max
cpuspeedy: CPU speed is now 1.50 GHz (100%) temperature is 55 C morte ~ # tail -n1 /var/log/messages
[10609.463888] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 10 (50/5) morte ~ # cpuspeedy med cpuspeedy: CPU speed is now 1 GHz (66%) temperature is 72 C [15996.701445] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 10 (60/6)
This isn't just an aesthetic issue, as many device drivers rely on the jiffies counter.
Steps to reproduce:
see above.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
http://bugzilla.kernel.org/show_bug.cgi?id=9475
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
------- Comment #1 from [email protected] 2007-12-01 12:35 ------- Please mark it as a regression.
Apart from this, I'm going to add it to the list of recent regressions.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
OtherBugsDependingO| |9243
nThis| |Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
Regression|0 |1
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #2 from [email protected] 2007-12-04 13:14 ------- Ugh, this seems to be a nasty regression.
Dave, is this some GTOD weirdness? I didnt know jiffies are fixed up during cpu-freq change - that would mess up all sorts of things like all timer_list timers.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #3 from [email protected] 2007-12-04 13:28 ------- we've been doing adjust_jiffies() on cpu speed transitions forever, so this isn't a regression in that sense. Why it's causing a problem now is a mystery though.
Even more puzzling, on lkml, Stefano commented out the adjust_jiffies() code, and still saw leaping counters, so I'm really at a loss to explain what's going on.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #4 from [email protected] 2007-12-04 13:47 -------
Quoted 7 lines
we've been doing
adjust_jiffies()on cpu speed transitions forever, so this isn't a regression in that sense. Why it's causing a problem now is a mystery though.Even more puzzling, on lkml, Stefano commented out the
adjust_jiffies()code, and still saw leaping counters, so I'm really at a loss to explain what's going on.
hm, maybe i'm missing something obvious, but adjust_jiffies() seems to fix up loops_per_jiffy, not the jiffies variable. The bugreport included a /var/log/messages snippets that showed printk timestamps - the printk timestamps are directly derived from the 'jiffies' variable. Those timestamps jumped back and forth according to CPU frequency. So _something_ does seem to affect the jiffies variable - and it's not adjust_jiffies(). Or at least so i think.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #5 from [email protected] 2007-12-04 14:04 ------- (In reply to comment #4)
Quoted 7 lines
hm, maybe i'm missing something obvious, but
adjust_jiffies()seems to fix uploops_per_jiffy, not the jiffies variable. The bugreport included a/var/log/messagessnippets that showed printk timestamps - the printk timestamps are directly derived from the 'jiffies' variable. Those timestamps jumped back and forth according to CPU frequency. So _something_ does seem to affect the jiffies variable - and it's notadjust_jiffies(). Or at least so i think.
Yes, I'm wondering about this as well. But so far, I hadn't been able to find out where in the kernel a CPU frequency switch could trigger a sudden change to the jiffies counter. The only related thing I observed is that adjust_jiffies() is called, but that doesn't seem to be the culprit.
BTW, I just sorted out the dm-crypt issue which prevented me to test 2.6.23. Will let you know ASAP about further tests.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #6 from [email protected] 2007-12-04 14:09 -------
Quoted 5 lines
Yes, I'm wondering about this as well. But so far, I hadn't been able to find out where in the kernel a CPU frequency switch could trigger a sudden change to the jiffies counter. The only related thing I observed is that
adjust_jiffies()is called, but that doesn't seem to be the culprit.
one thing that changes jiffies is the NOHZ code. But that should really only be a forwards adjustment. NTP also changes it. Weird.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #7 from [email protected] 2007-12-04 14:25 ------- does booting with nohz=off change anything?
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #8 from [email protected] 2007-12-05 11:28 ------- (In reply to comment #7)
Quoted 1 line
does booting with nohz=off change anything?
That was already off. I tried to turn it on, but nothing changed.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #9 from [email protected] 2007-12-05 11:38 ------- I just managed to boot 2.6.23. Nothing changes. Do you want me to bisect some specific tree?
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #10 from [email protected] 2007-12-05 12:43 -------
Quoted 2 lines
I just managed to boot 2.6.23. Nothing changes. Do you want me to bisect some specific tree?
nothing changes == same problem?
If yes then please try to find the oldest kernel that still works fine. 2.6.20 would be a good pick i think, 2.6.16 after that. Then if you could try to bisect between the youngest "good" and the oldest "bad" tree that would be fantastic ... Expect a compile-fest though.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |CODE_FIX------- Comment #11 from [email protected] 2007-12-06 17:26 ------- Fixed. Please see:
http://lkml.org/lkml/2007/12/6/377
Thank you for your suggestions.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |acpi-
| |[email protected]
| |et------- Comment #12 from [email protected] 2007-12-06 20:32 ------- Some Intel TSC facts, for the record:
if (family == 6 && model < 14) || (family == 15 && model < 3)
TSC rate = Core MHz rate (ie. it changes with core/bus ratio shifts) WRMSR on the TSC writes bottom 32-bits, clears upper 32 bits.
For these families with models later than above
TSC rate = constant (does not change with core/bus ratio shifts) WRMSR TSC writes all 64 bits.
The system here is a family 6, model 13, so it it will indeed see variable speed.
But note that the bigger problem may be that on all Intel processors with C3 that have shipped to date, the TSC COMPLETELY STOPS for the duration of C3.
I believe that tsc_unstable is set upon either of these situations.
I don't follow all the twisty turny paths in the code here -- and it is wonderful that you bisected and found the cause; but it isn't immediately obvious that this is due to variable speed TSC, but it may be due to TSC stoppage.
if you disable cpufreq and it goes away, you know it was variable TSC. if you boot with "idle=poll" and keep cpufreq enabled and it goes away, then you know it was TSC stoppage.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #13 from [email protected] 2007-12-06 21:16 ------- (In reply to comment #12)
Quoted 5 lines
Some Intel TSC facts, for the record:
if (family == 6 && model < 14) || (family == 15 && model < 3)
TSC rate = Core MHz rate (ie. it changes with core/bus ratio shifts) WRMSR on the TSC writes bottom 32-bits, clears upper 32 bits.
Thank you for the info.
Quoted 3 lines
But note that the bigger problem may be that on all Intel processors with C3 that have shipped to date, the TSC COMPLETELY STOPS for the duration of C3.
During C2 as well, maybe?
Quoted 1 line
I believe that
tsc_unstableis set upon either of these situations.
Yes, no worries, I hit both :)
Nov 29 07:55:29 morte [ 6.530752] Marking TSC unstable due to: possible TSC halt in C2. Nov 29 07:55:29 morte [ 6.530761] Time: acpi_pm clocksource has been installed. Nov 29 07:55:29 morte [ 6.620700] Clocksource tsc unstable (delta =
-456217276 ns)
and
Nov 29 20:50:31 morte [ 297.562722] Marking TSC unstable due to: cpufreq
changes.
Anyway, I tried to follow your two suggestions and I must say that variable TSC causes issues, while TSC stoppage doesn't, in my case. In both cases, the timestamp isn't correct, but while TSC stoppage causes, well, a stop, the big trouble is when the timestamp leaps back because of cpufreq changes, and drivers get confused by this behaviour.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|CODE_FIX |Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
------- Comment #14 from [email protected] 2007-12-07 17:40 ------- *** Bug 9524 has been marked as a duplicate of this bug. ***
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution| |CODE_FIX------- Comment #15 from [email protected] 2007-12-18 16:59 -------
Fix: http://lkml.org/lkml/2007/12/7/100
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9475] jiffies counter leaps on CPU frequency changes
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9475
------- Comment #16 from [email protected] 2007-12-19 05:46 -------
Quoted 1 line
note that due to intrusiveness this will likely only show up in v2.6.25.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.