Skip to contentSkip to conversation

Work around for periodic do_gettimeofday hang

PATCH In review James Bottomley 7 participants 11 messages Nov 24, 2004 – Nov 27, 2004

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Work around for periodic do_gettimeofday hang

On the voyager systems particularly (but also on some of my slower parisc boxes) I periodically get hangs where the system just seems to stop (although it does remain able to execute alt-sysrq). The traces always seem to implicate do_gettimeofday in the xtime seqlock.

I've not been able to trace an exact cause for this, but it does seem to be pretty much eliminated by lowering the clock speed to 100HZ. I propose the following patch to do this for the slower intel processors (basically pentiums and below)

James

--

===== include/asm-i386/param.h 1.6 vs edited =====

include/asm-i386/param.h +8 -1
@@ -1,8 +1,15 @@
 #ifndef _ASMi386_PARAM_H
 #define _ASMi386_PARAM_H
 
+#include <linux/config.h>
+
 #ifdef __KERNEL__
-# define HZ		1000		/* Internal kernel timer frequency */
+# if defined(CONFIG_M386) || defined(CONFIG_M486) || defined(CONFIG_M586) \
+     || defined(CONFIG_M586TSC) || defined(CONFIG_M586MMX)
+#  define HZ		100
+# else
+#  define HZ		1000		/* Internal kernel timer frequency */
+# endif
 # define USER_HZ	100		/* .. some user interfaces are in "ticks" */
 # define CLOCKS_PER_SEC		(USER_HZ)	/* like times() */
 #endif
#1 lore raw
Quoted 5 lines

I've not been able to trace an exact cause for this, but it does seem to be pretty much eliminated by lowering the clock speed to 100HZ. I propose the following patch to do this for the slower intel processors (basically pentiums and below)

while I agree with 100Hz for slower cpus, I rather have a config option for it so people (and distros) can select it independent of the exact cpu type they want to compile a kernel for

#2 lore raw
Quoted 2 lines · Arjan van de Ven

while I agree with 100Hz for slower cpus, I rather have a config option for it so people (and distros) can select it independent of the exact cpu type they want to compile a kernel for

How about this then?

James

===== arch/i386/Kconfig 1.134 vs edited =====

arch/i386/Kconfig +8 -0
@@ -330,6 +330,14 @@
 
 endchoice
 
+config X86_HZ
+       int "Clock Tick Rate"
+       default 1000 if !(M386 || M486 || M586 || M586TSC || M586MMX)	
+       default 100 if (M386 || M486 || M586 || M586TSC || M586MMX)	
+       help
+	  Select the kernel clock tick rate in interrupts per second.
+	  Slower processors should choose 100; everything else 1000.
+
 config X86_GENERIC
        bool "Generic x86 support"
        help

===== include/asm-i386/param.h 1.6 vs edited =====

include/asm-i386/param.h +3 -1
@@ -1,8 +1,10 @@
 #ifndef _ASMi386_PARAM_H
 #define _ASMi386_PARAM_H
 
+#include <linux/config.h>
+
 #ifdef __KERNEL__
-# define HZ		1000		/* Internal kernel timer frequency */
+# define HZ		(CONFIG_X86_HZ)
 # define USER_HZ	100		/* .. some user interfaces are in "ticks" */
 # define CLOCKS_PER_SEC		(USER_HZ)	/* like times() */
 #endif
#3 lore raw
Quoted 8 lines · James Bottomley

+config X86_HZ

 +       int "Clock Tick Rate"
 +       default 1000 if !(M386 || M486 || M586 || M586TSC || M586MMX)	
 +       default 100 if (M386 || M486 || M586 || M586TSC || M586MMX)	
 +       help

+ Select the kernel clock tick rate in interrupts per second.
+ Slower processors should choose 100; everything else 1000.

I guess we don't need the help, given that it's not a menuisable option. (There was a make-HZ-selectable patch once, and Linus spat it out).

Silly question: how come do_gettimeofday() is hanging?

#4 lore raw
Quoted 1 line · Andrew Morton

Silly question: how come do_gettimeofday() is hanging?

That's what I don't know. like I said, the xtime seqlock seemed to be implicated, but I've never managed to trace it down.

I also notice that under heavy load the system starts to lose time (at least according to ntp). I suspect there's something wrong with the PIT timer routine (since that's what voyager has to use) for adding lost ticks.

James

#5 lore raw
In reply to Andrew Morton
Quoted 10 lines · Andrew Morton
Quoted 8 lines · James Bottomley

+config X86_HZ

 +       int "Clock Tick Rate"
 +       default 1000 if !(M386 || M486 || M586 || M586TSC || M586MMX)	
 +       default 100 if (M386 || M486 || M586 || M586TSC || M586MMX)	
 +       help

+ Select the kernel clock tick rate in interrupts per second.
+ Slower processors should choose 100; everything else 1000.

Quoted 2 lines

I guess we don't need the help, given that it's not a menuisable option. (There was a make-HZ-selectable patch once, and Linus spat it out).

I'd rather have it as menu-visible option, but I can see how arbitrary HZ gets spat out. How about a radio-button like thing with "100" and "1000" in it.

HZ=1000 costs you 1% HPC performance, and for slower machines probably more. Also some hw (with slow smm) really doesn't likeit.

#6 lore raw
Quoted 1 line · Arjan van de Ven

HZ=1000 costs you 1% HPC performance, and for slower machines probably more. Also some hw (with slow smm) really doesn't likeit.

I actually profiled the timer ISR on a 600Mhz C3. It runs for 22 usec. So with HZ=1000 the residency of the timer interrupt is 2.2%.

Lee

#7 lore raw
In reply to James Bottomley
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines

while I agree with 100Hz for slower cpus, I rather have a config option for it so people (and distros) can select it independent of the exact cpu type they want to compile a kernel for

How about this then?

I'd rather make it a number field rather than "choose-me", so the wise user can choose any Hz he likes.

Jan Engelhardt

Signature · 2 lines
Gesellschaft für Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 Göttingen, www.gwdg.de
#8 lore raw
In reply to James Bottomley

Hi,

arch/i386/Kconfig line 336

Quoted 4 lines · James Bottomley
arch/i386/Kconfig +4 -0
+config X86_HZ
+       int "Clock Tick Rate"
+       default 1000 if !(M386 || M486 || M586 || M586TSC || M586MMX)	
+       default 100 if (M386 || M486 || M586 || M586TSC || M586MMX)	

You don't have to duplicate the conditions, kconfig takes the first match.

bye, Roman

#9 lore raw
In reply to James Bottomley
Quoted 4 lines · James Bottomley

On the voyager systems particularly (but also on some of my slower parisc boxes) I periodically get hangs where the system just seems to stop (although it does remain able to execute alt-sysrq). The traces always seem to implicate do_gettimeofday in the xtime seqlock.

What are the other CPUs doing in this case ?

#10 lore raw
Quoted 1 line · Alan Cox

What are the other CPUs doing in this case ?

Actually, I don't know ... I can't seem to persuade the voyagers to broadcast an NMI to give up that information.

However, unfortunately, I got a similar hang at HZ=100 this time not implicating do_gettimeofday.

I'll continue investigating.

James