Skip to contentSkip to conversation

jiffies counter leaps in 2.6.24-rc3

BUG Stefano Brivio 5 participants 18 messages Nov 24, 2007 – Nov 29, 2007

lore
Share & tools
lore
#0 lore raw

It looks like the jiffies counter sometimes jumps back and forth of some hundreds seconds in 2.6.24-rc3. I observed that this happens when I use the su(1) command, e.g.:

Nov 24 06:17:17 morte [190769.065301] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 232 (6730/29)
Nov 24 06:17:22 morte su[16826]: Successful su for root by st3
Nov 24 06:17:22 morte su[16826]: + pts/1 st3:root
Nov 24 06:17:22 morte su(pam_unix)[16826]: session opened for user root by (uid=1000)
Nov 24 06:17:38 morte [715682.606983] b43-phy2 ERROR: PHY transmission error
Nov 24 06:18:17 morte [715707.765415] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 194 (970/5)

or

Nov 23 20:55:40 morte [627074.320296] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 10 (550/55)
Nov 23 20:56:01 morte su[5452]: Successful su for root by st3
Nov 23 20:56:01 morte su[5452]: + pts/4 st3:root
Nov 23 20:56:01 morte su(pam_unix)[5452]: session opened for user root by (uid=1000)
Nov 23 20:56:03 morte su(pam_unix)[5452]: session closed for user root
Nov 23 20:56:40 morte [167187.102931] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 10 (40/4)

or

Nov 23 06:31:00 morte [156536.124549] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 240 (6480/27)
Nov 23 06:31:58 morte su[4350]: Successful su for root by st3
Nov 23 06:31:58 morte su[4350]: + pts/0 st3:root
Nov 23 06:31:58 morte su(pam_unix)[4350]: session opened for user root by (uid=1000)
Nov 23 06:32:09 morte [587438.574530] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 240 (4080/17)

(I checked with a clock the timestamp prepended by syslog-ng, and it's correct.)

I'm thinking this could be somehow related to the setpriority() call made by su(1), but I don't know how to debug this further. Any clue?

morte st3 # cat /proc/interrupts
CPU0

  0:  319167512    XT-PIC-XT        timer
  1:     459332    XT-PIC-XT        i8042
  2:          0    XT-PIC-XT        cascade
  3:          1    XT-PIC-XT
  4:          1    XT-PIC-XT
  5:          1    XT-PIC-XT
  7:          9    XT-PIC-XT
  8:          2    XT-PIC-XT        rtc
  9:       9114    XT-PIC-XT        acpi
 10:   46272099    XT-PIC-XT        uhci_hcd:usb1, i915@pci:0000:00:02.0
 11:   23691749    XT-PIC-XT        yenta, Intel 82801DB-ICH4, uhci_hcd:usb2, uhci_hcd:usb3, ehci_hcd:usb4, Intel 82801DB-ICH4 Modem, ohci1394, b43
 12:    3287633    XT-PIC-XT        i8042
 14:    1076167    XT-PIC-XT        ide0
 15:         47    XT-PIC-XT        ide1
NMI:          0   Non-maskable interrupts
LOC:          0   Local timer interrupts
TRM:          0   Thermal event interrupts
SPU:          0   Spurious interrupts
ERR:          8
MIS:          0

morte st3 # cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : Intel(R) Pentium(R) M processor 1.50GHz
stepping : 6
cpu MHz : 1500.000
cache size : 2048 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr mce cx8 sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss tm pbe bts est tm2
bogomips : 2993.47
clflush size : 64

Signature · 2 lines
Ciao
Stefano
#1 lore raw
Quoted 3 lines · Stefano Brivio

It looks like the jiffies counter sometimes jumps back and forth of some hundreds seconds in 2.6.24-rc3. I observed that this happens when I use the su(1) command, e.g.:

Can you please explain what exactly the problem is here?

Are you perhaps referring to the number between square brackets for the su log lines? In that case there is no problem as in that case the number is not jiffies, but the process ID (PID) of the su process...

Cheers,
FJP

#2 lore raw
Quoted 10 lines · On Sat, 24 Nov 2007 18:56:57 +0100 Frans Pop <[email protected]> wrote:
Quoted 3 lines · Stefano Brivio

It looks like the jiffies counter sometimes jumps back and forth of some hundreds seconds in 2.6.24-rc3. I observed that this happens when I use the su(1) command, e.g.:

Can you please explain what exactly the problem is here?

Are you perhaps referring to the number between square brackets for the su log lines? In that case there is no problem as in that case the number is not jiffies, but the process ID (PID) of the su process...

Sorry guy but I'm not _that_ idiot. Please notice jiffies values before and after that.

Signature · 2 lines
Ciao
Stefano
#3 lore raw
In reply to Stefano Brivio
Quoted 28 lines · Stefano Brivio

It looks like the jiffies counter sometimes jumps back and forth of some hundreds seconds in 2.6.24-rc3. I observed that this happens when I use the su(1) command, e.g.:

Nov 24 06:17:17 morte [190769.065301] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 232 (6730/29)
Nov 24 06:17:22 morte su[16826]: Successful su for root by st3
Nov 24 06:17:22 morte su[16826]: + pts/1 st3:root
Nov 24 06:17:22 morte su(pam_unix)[16826]: session opened for user root by (uid=1000)
Nov 24 06:17:38 morte [715682.606983] b43-phy2 ERROR: PHY transmission error
Nov 24 06:18:17 morte [715707.765415] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 194 (970/5)

or

Nov 23 20:55:40 morte [627074.320296] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 10 (550/55)
Nov 23 20:56:01 morte su[5452]: Successful su for root by st3
Nov 23 20:56:01 morte su[5452]: + pts/4 st3:root
Nov 23 20:56:01 morte su(pam_unix)[5452]: session opened for user root by (uid=1000)
Nov 23 20:56:03 morte su(pam_unix)[5452]: session closed for user root
Nov 23 20:56:40 morte [167187.102931] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 10 (40/4)

or

Nov 23 06:31:00 morte [156536.124549] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 240 (6480/27)
Nov 23 06:31:58 morte su[4350]: Successful su for root by st3
Nov 23 06:31:58 morte su[4350]: + pts/0 st3:root
Nov 23 06:31:58 morte su(pam_unix)[4350]: session opened for user root by (uid=1000)
Nov 23 06:32:09 morte [587438.574530] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 240 (4080/17)
Quoted 5 lines

(I checked with a clock the timestamp prepended by syslog-ng, and it's correct.)

I'm thinking this could be somehow related to the setpriority() call made by su(1), but I don't know how to debug this further. Any clue?

NO_HZ? Highres timers?

I understand that the previous kernels behave correctly. All of them?

Rafael

#4 lore raw
Quoted 1 line · On Sat, 24 Nov 2007 19:48:58 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

NO_HZ? Highres timers?

CONFIG_HZ_1000=y
# CONFIG_HIGH_RES_TIMERS is not set

Quoted 1 line

I understand that the previous kernels behave correctly. All of them?

2.6.21 behaved correctly. Sorry but git-bisect would take a lot of time (I can't reliably reproduce the jiffies jump), so I would avoid that if not strictly needed.

Signature · 2 lines
Ciao
Stefano
#5 lore raw
Quoted 5 lines · Stefano Brivio
Quoted 1 line · On Sat, 24 Nov 2007 19:48:58 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

NO_HZ? Highres timers?

Quoted 8 lines

CONFIG_HZ_1000=y
# CONFIG_HIGH_RES_TIMERS is not set

Quoted 1 line

I understand that the previous kernels behave correctly. All of them?

2.6.21 behaved correctly. Sorry but git-bisect would take a lot of time (I can't reliably reproduce the jiffies jump), so I would avoid that if not strictly needed.

Well, it would be good to know if 2.6.23 behaves correctly, at least.

Thanks,
Rafael

#6 lore raw
Quoted 9 lines · On Sat, 24 Nov 2007 20:31:25 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 8 lines · Stefano Brivio
Quoted 1 line · On Sat, 24 Nov 2007 19:48:58 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

NO_HZ? Highres timers?

CONFIG_HZ_1000=y
# CONFIG_HIGH_RES_TIMERS is not set

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I understand that the previous kernels behave correctly. All of them?

2.6.21 behaved correctly. Sorry but git-bisect would take a lot of time (I can't reliably reproduce the jiffies jump), so I would avoid that if not strictly needed.

Well, it would be good to know if 2.6.23 behaves correctly, at least.

Will try that ASAP, sorry for the delay.

Signature · 2 lines
Ciao
Stefano
#7 lore raw
In reply to Rafael J. Wysocki
Quoted 9 lines · On Sat, 24 Nov 2007 20:31:25 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 8 lines · Stefano Brivio
Quoted 1 line · On Sat, 24 Nov 2007 19:48:58 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

NO_HZ? Highres timers?

CONFIG_HZ_1000=y
# CONFIG_HIGH_RES_TIMERS is not set

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I understand that the previous kernels behave correctly. All of them?

2.6.21 behaved correctly. Sorry but git-bisect would take a lot of time (I can't reliably reproduce the jiffies jump), so I would avoid that if not strictly needed.

Well, it would be good to know if 2.6.23 behaves correctly, at least.

Weird, it looks like I can't boot with 2.6.23.9 because of some issues with dm-crypt (my root filesystem is encrypted). I double-checked the configuration (which I just took from my current one), well, no way. Any other test I can do?

In the meanwhile, I noted another thing: sometimes it happens that I become root and the jiffies counter jumps ahead. Then, when I close any root session, the jiffies counter jumps back to the correct value.

Please remember that this isn't just an aesthetic issue, as some drivers (e.g. b43 and b43legacy, but I guess a lot more) rely on jiffies. Do I need to file a bug to bugzilla.kernel.org? Thank you.

--
Ciao
Stefano

#8 lore raw
Quoted 3 lines · Stefano Brivio

Please remember that this isn't just an aesthetic issue, as some drivers (e.g. b43 and b43legacy, but I guess a lot more) rely on jiffies. Do I need to file a bug to bugzilla.kernel.org? Thank you.

yes, please do file a bugzilla report too - there seems to be no progress or quick ideas at the moment.

Ingo

#9 lore raw
In reply to Rafael J. Wysocki

Ok, I found out how to reliably reproduce this bug. The root session issue was a bit weird, but I noticed I usually switch to root only when I need to change the CPU frequency. And here's what happens:

# cpuspeedy min
[after some time...]
Nov 29 09:42:31 morte [ 4408.722597] b43-phy0 ERROR: PHY transmission error
# cpuspeedy max
Nov 29 09:42:41 morte [ 1177.238599] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 540 (2160/4)

It looks like jiffies get scaled exactly by a 3.75 factor, which is <max_frequency>/<min_frequency> here. I use acpi-cpufreq and this is my (only) CPU:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : Intel(R) Pentium(R) M processor 1.50GHz
stepping : 6
cpu MHz : 1500.000
cache size : 2048 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr mce cx8 sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss tm pbe bts est tm2
bogomips : 2993.49
clflush size : 64

Any hint?

--
Ciao
Stefano

#10 lore raw
Quoted 34 lines · Stefano Brivio

Ok, I found out how to reliably reproduce this bug. The root session issue was a bit weird, but I noticed I usually switch to root only when I need to change the CPU frequency. And here's what happens:

# cpuspeedy min
[after some time...]
Nov 29 09:42:31 morte [ 4408.722597] b43-phy0 ERROR: PHY transmission error
# cpuspeedy max
Nov 29 09:42:41 morte [ 1177.238599] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 540 (2160/4)

It looks like jiffies get scaled exactly by a 3.75 factor, which is <max_frequency>/<min_frequency> here. I use acpi-cpufreq and this is my (only) CPU:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : Intel(R) Pentium(R) M processor 1.50GHz
stepping : 6
cpu MHz : 1500.000
cache size : 2048 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr mce cx8 sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss tm pbe bts est tm2
bogomips : 2993.49
clflush size : 64

Quoted 1 line

Any hint?

ah, the cpufreq/acpi/acpi-cpufreq angle is indeed a big clue. I've Cc:-ed various interested parties.

Ingo

#11 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Stefano Brivio

Ok, I found out how to reliably reproduce this bug. The root session issue was a bit weird, but I noticed I usually switch to root only when I need to change the CPU frequency. And here's what happens:

# cpuspeedy min
[after some time...]
Nov 29 09:42:31 morte [ 4408.722597] b43-phy0 ERROR: PHY transmission error
Quoted 8 lines
Quoted 5 lines
# cpuspeedy max
Nov 29 09:42:41 morte [ 1177.238599] wmaster0: STA 00:14:c1:35:8d:eb Average rate: 540 (2160/4)

It looks like jiffies get scaled exactly by a 3.75 factor, which is <max_frequency>/<min_frequency> here. I use acpi-cpufreq and this is my

ah, the cpufreq/acpi/acpi-cpufreq angle is indeed a big clue. I've Cc:-ed various interested parties.

Hmm, what's in /sys/devices/system/clocksource/clocksource0/current_clocksource ?

adjust_jiffies() always gets called on a speed transition. I'm wondering if perhaps we shouldn't do that if we're using something other than the tsc for timekeeping.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#12 lore raw
Quoted 2 lines · On Thu, 29 Nov 2007 12:45:48 -0500 Dave Jones <[email protected]> wrote:

Hmm, what's
in /sys/devices/system/clocksource/clocksource0/current_clocksource ?

morte st3 # cat /sys/devices/system/clocksource/clocksource0/current_clocksource acpi_pm

Quoted 3 lines

adjust_jiffies() always gets called on a speed transition. I'm wondering if perhaps we shouldn't do that if we're using something other than the tsc for timekeeping.

I gave other clocksources a try:

morte st3 # cat /sys/devices/system/clocksource/clocksource0/available_clocksource acpi_pm pit jiffies tsc

Setting tsc as current clocksource:
morte st3 # echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource

[13367.729060] Time: tsc clocksource has been installed.

didn't fix anything. Other weirdness here: it looks like it affects my typematic delay and repeat rate.

Setting jiffies or pit as current clocksource didn't fix anything, but at least it doesn't cause anything bad.

Do you want me to comment out adjust_jiffies() and see what happens?

Signature · 2 lines
Ciao
Stefano
#13 lore raw
Quoted 6 lines · Stefano Brivio
Quoted 3 lines

adjust_jiffies() always gets called on a speed transition. I'm wondering if perhaps we shouldn't do that if we're using something other than the tsc for timekeeping.

I gave other clocksources a try:

Quoted 8 lines

morte st3 # cat /sys/devices/system/clocksource/clocksource0/available_clocksource acpi_pm pit jiffies tsc

Setting tsc as current clocksource:
morte st3 # echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource

[13367.729060] Time: tsc clocksource has been installed.

didn't fix anything. Other weirdness here: it looks like it affects my typematic delay and repeat rate.

*grumble*

Quoted 4 lines

Setting jiffies or pit as current clocksource didn't fix anything, but at least it doesn't cause anything bad.

Do you want me to comment out adjust_jiffies() and see what happens?

Just for laughs, why not.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#14 lore raw
Quoted 5 lines · On Thu, 29 Nov 2007 13:24:02 -0500 Dave Jones <[email protected]> wrote:
Quoted 1 line · Stefano Brivio

Do you want me to comment out adjust_jiffies() and see what happens?

Just for laughs, why not.

I wanted to laugh, but the behaviour didn't change at all. How boring.

Signature · 2 lines
Ciao
Stefano
#15 lore raw
Quoted 10 lines · Stefano Brivio
Quoted 5 lines · On Thu, 29 Nov 2007 13:24:02 -0500 Dave Jones <[email protected]> wrote:
Quoted 1 line · Stefano Brivio

Do you want me to comment out adjust_jiffies() and see what happens?

Just for laughs, why not.

I wanted to laugh, but the behaviour didn't change at all. How boring.

just to make sure there is no tsc impact, does booting with "notsc" change anything?

if not, does booting with idle=halt or idle=poll help? [your power usage will go up so it's not very useful - but it could help exclude a few sources of trouble.]

Ingo

#16 lore raw
Quoted 2 lines · On Thu, 29 Nov 2007 20:36:24 +0100 Ingo Molnar <[email protected]> wrote:

just to make sure there is no tsc impact, does booting with "notsc" change anything?

No, it doesn't.

Quoted 3 lines

if not, does booting with idle=halt or idle=poll help? [your power usage will go up so it's not very useful - but it could help exclude a few sources of trouble.]

With idle=poll nothing changes, except that - as soon as I switch the CPU frequency - I get this in the logs:

[  297.562722] Marking TSC unstable due to: cpufreq changes.
[  297.566677] Time: acpi_pm clocksource has been installed.

With idle=halt, changing frequency just freezes the system. I can't even use the magic Sys Rq keys then.

I observed another thing. When I don't use any of the parameters you suggested for debugging, I get this at boot:

[    6.530752] Marking TSC unstable due to: possible TSC halt in C2.
[    6.530761] Time: acpi_pm clocksource has been installed.
[    6.620700] Clocksource tsc unstable (delta = -456217276 ns)
Signature · 2 lines
Ciao
Stefano
#17 lore raw
In reply to Stefano Brivio
Quoted 10 lines · Stefano Brivio
Quoted 5 lines · On Thu, 29 Nov 2007 13:24:02 -0500 Dave Jones <[email protected]> wrote:
Quoted 1 line · Stefano Brivio

Do you want me to comment out adjust_jiffies() and see what happens?

Just for laughs, why not.

I wanted to laugh, but the behaviour didn't change at all. How boring.

Unless I'm missing something obvious, this would exonorate cpufreq as a cause, which would seem odd based upon your earlier diagnosis.

deeply confused.

Dave

Signature · 1 line
http://www.codemonkey.org.uk