Skip to contentSkip to conversation

deinline sleep/delay functions

PATCH In review Denis Vlasenko 6 participants 16 messages Jun 30, 2005 – Jul 1, 2005 High-resolution timers, timer wheel, clockevents

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 deinline sleep/delay functions

Hi Andrew,

Optimizing delay functions for speed is utterly pointless.

This patch turns ssleep(n), mdelay(n), udelay(n) and ndelay(n) into functions, thus they generate the smallest possible code at the callsite. Previously they were more or less inlined.

Run tested. Saved a few kb off vmlinux.

--
vda

include/asm-i386/delay.h +2 -6
diff -urpN linux-2.6.12.src/include/asm-i386/delay.h linux-2.6.12.new/include/asm-i386/delay.h
@@ -15,12 +15,8 @@ extern void __ndelay(unsigned long nsecs
 extern void __const_udelay(unsigned long usecs);
 extern void __delay(unsigned long loops);
 
-#define udelay(n) (__builtin_constant_p(n) ? \
-	((n) > 20000 ? __bad_udelay() : __const_udelay((n) * 0x10c7ul)) : \
-	__udelay(n))
+#define udelay(n) __udelay(n)
 	
-#define ndelay(n) (__builtin_constant_p(n) ? \
-	((n) > 20000 ? __bad_ndelay() : __const_udelay((n) * 5ul)) : \
-	__ndelay(n))
+#define ndelay(n) __ndelay(n)
 
 #endif /* defined(_I386_DELAY_H) */
include/linux/delay.h +3 -29
diff -urpN linux-2.6.12.src/include/linux/delay.h linux-2.6.12.new/include/linux/delay.h
@@ -11,40 +11,14 @@ extern unsigned long loops_per_jiffy;
 
 #include <asm/delay.h>
 
-/*
- * Using udelay() for intervals greater than a few milliseconds can
- * risk overflow for high loops_per_jiffy (high bogomips) machines. The
- * mdelay() provides a wrapper to prevent this.  For delays greater
- * than MAX_UDELAY_MS milliseconds, the wrapper is used.  Architecture
- * specific values can be defined in asm-???/delay.h as an override.
- * The 2nd mdelay() definition ensures GCC will optimize away the 
- * while loop for the common cases where n <= MAX_UDELAY_MS  --  Paul G.
- */
-
-#ifndef MAX_UDELAY_MS
-#define MAX_UDELAY_MS	5
-#endif
-
-#ifdef notdef
-#define mdelay(n) (\
-	{unsigned long __ms=(n); while (__ms--) udelay(1000);})
-#else
-#define mdelay(n) (\
-	(__builtin_constant_p(n) && (n)<=MAX_UDELAY_MS) ? udelay((n)*1000) : \
-	({unsigned long __ms=(n); while (__ms--) udelay(1000);}))
-#endif
-
+void calibrate_delay(void);
+void mdelay(unsigned int msecs);
 #ifndef ndelay
 #define ndelay(x)	udelay(((x)+999)/1000)
 #endif
 
-void calibrate_delay(void);
 void msleep(unsigned int msecs);
+void ssleep(unsigned int secs);
 unsigned long msleep_interruptible(unsigned int msecs);
-
-static inline void ssleep(unsigned int seconds)
-{
-	msleep(seconds * 1000);
-}
 
 #endif /* defined(_LINUX_DELAY_H) */
kernel/timer.c +22 -0
diff -urpN linux-2.6.12.src/kernel/timer.c linux-2.6.12.new/kernel/timer.c
@@ -33,6 +33,7 @@
 #include <linux/posix-timers.h>
 #include <linux/cpu.h>
 #include <linux/syscalls.h>
+#include <linux/delay.h>
 
 #include <asm/uaccess.h>
 #include <asm/unistd.h>
@@ -89,6 +90,20 @@ static inline void set_running_timer(tve
 /* Fake initialization */
 static DEFINE_PER_CPU(tvec_base_t, tvec_bases) = { SPIN_LOCK_UNLOCKED };
 
+/***
+ * Using udelay() for intervals greater than a few milliseconds can
+ * risk overflow for high loops_per_jiffy (high bogomips) machines. The
+ * mdelay() provides a wrapper to prevent this.
+ *
+ * It's not inlined because we do not optimize delays for speed ;)
+ */
+void mdelay(unsigned int msecs)
+{
+	while (msecs--) udelay(1000);
+}
+
+EXPORT_SYMBOL(mdelay);
+
 static void check_timer_failed(struct timer_list *timer)
 {
 	static int whine_count;
@@ -1592,6 +1607,13 @@ void msleep(unsigned int msecs)
 }
 
 EXPORT_SYMBOL(msleep);
+
+void ssleep(unsigned int secs)
+{
+        msleep(secs * 1000);
+}
+
+EXPORT_SYMBOL(ssleep);
 
 /**
  * msleep_interruptible - sleep waiting for waitqueue interruptions
#1 lore raw
Quoted 11 lines · Denis Vlasenko

Hi Andrew,

Optimizing delay functions for speed is utterly pointless.

This patch turns ssleep(n), mdelay(n), udelay(n) and ndelay(n) into functions, thus they generate the smallest possible code at the callsite. Previously they were more or less inlined.

Run tested. Saved a few kb off vmlinux.

Rejected-by: Russell King 8)

The reason is that now we're unable to find out if anyone's doing udelay(100000000000000000) which breaks on most architectures.

There are a number of compile-time checks that your patch has removed which catch such things, and as such your patch is not acceptable. Some architectures have a lower threshold of acceptability for the maximum udelay value, so it's absolutely necessary to keep this.

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core
#2 lore raw
Quoted 16 lines · Russell King
Quoted 11 lines · Denis Vlasenko

Hi Andrew,

Optimizing delay functions for speed is utterly pointless.

This patch turns ssleep(n), mdelay(n), udelay(n) and ndelay(n) into functions, thus they generate the smallest possible code at the callsite. Previously they were more or less inlined.

Run tested. Saved a few kb off vmlinux.

Rejected-by: Russell King 8)

Quoted 7 lines

The reason is that now we're unable to find out if anyone's doing udelay(100000000000000000) which breaks on most architectures.

There are a number of compile-time checks that your patch has removed which catch such things, and as such your patch is not acceptable. Some architectures have a lower threshold of acceptability for the maximum udelay value, so it's absolutely necessary to keep this.

It removes that check from x86 - other architectures retain it.

I don't recall seeing anyone trigger the check, and it hardly seems worth adding a "few kb" to vmlinux for it?

#3 lore raw
Quoted 9 lines
Quoted 5 lines

There are a number of compile-time checks that your patch has removed which catch such things, and as such your patch is not acceptable. Some architectures have a lower threshold of acceptability for the maximum udelay value, so it's absolutely necessary to keep this.

It removes that check from x86 - other architectures retain it.

I don't recall seeing anyone trigger the check,

I do ;) Esp in vendor out of tree crap. It's a good compile time diagnostic so the junk code doesnt' hit mainline but gets fixed first.

Quoted 3 lines

and it hardly seems worth
adding a "few kb" to vmlinux for it?

but it can be fixed to not add that few kb while retaining the checking value. All that needs is for it to be a define that calls the worker function. Eg the check gets optimized out and all that remains is the call to the worker function that does the actual delay.

#4 lore raw
Quoted 6 lines · Arjan van de Ven
Quoted 6 lines
Quoted 5 lines

There are a number of compile-time checks that your patch has removed which catch such things, and as such your patch is not acceptable. Some architectures have a lower threshold of acceptability for the maximum udelay value, so it's absolutely necessary to keep this.

Quoted 6 lines
Quoted 3 lines

It removes that check from x86 - other architectures retain it.

I don't recall seeing anyone trigger the check,

I do ;) Esp in vendor out of tree crap. It's a good compile time diagnostic so the junk code doesnt' hit mainline but gets fixed first.

It seems my patch was incomplete.

Thinking more about it, since it exists in all arches and also since delaying is not performance critical (hey, if we're going to delay/sleep, we surely can burn a few more cycles), this can be done as follows:

linux/timer.c:

void ndelay(unsigned int nsecs)

{
	unsigned int m = nsecs/(1024*1024);
       	while (m--)
		__ndelay(1024);
	__ndelay(nsecs % (1024*1024));
}

void udelay(unsigned int usecs)

{
	unsigned int k = usecs/1024;
       	while (k--)
		__udelay(1024);
	__udelay(usecs % 1024);
}

void mdelay(unsigned int msecs)

{
       	while (msecs--)
		udelay(1000);
}

void ssleep(unsigned int secs)

{
        msleep(secs * 1000);
}

and arches will need to only supply two functions:

__udelay(n) [n is guaranteed <1024]
__ndelay(n) [n is guaranteed <1024*1024]

For users, _any_ value, however large, will work for any delay function.

Comments?
--
vda

#5 lore raw
Quoted 7 lines · Denis Vlasenko
Quoted 6 lines · Arjan van de Ven
Quoted 6 lines
Quoted 5 lines

There are a number of compile-time checks that your patch has removed which catch such things, and as such your patch is not acceptable. Some architectures have a lower threshold of acceptability for the maximum udelay value, so it's absolutely necessary to keep this.

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

It removes that check from x86 - other architectures retain it.

For users, _any_ value, however large, will work for any delay function.

that's not desired though. Desired is to limit udelay() to say 2000 or so. And force anything above that to go via mdelay() (just to make it stand out as broken code ;)

Over time we also want to phase out mdelay of course...

#6 lore raw
Quoted 11 lines · Arjan van de Ven
Quoted 10 lines · Denis Vlasenko
Quoted 9 lines · Arjan van de Ven
Quoted 9 lines
Quoted 5 lines

There are a number of compile-time checks that your patch has removed which catch such things, and as such your patch is not acceptable. Some architectures have a lower threshold of acceptability for the maximum udelay value, so it's absolutely necessary to keep this.

It removes that check from x86 - other architectures retain it.

Quoted 6 lines
Quoted 2 lines

For users, _any_ value, however large, will work for any delay function.

that's not desired though. Desired is to limit udelay() to say 2000 or so. And force anything above that to go via mdelay() (just to make it stand out as broken code ;)

An if(usec > 2000) { printk(..); dump_stack(); } will do.

But do you really want to do this? There might be legitimate reasons to compute udelay's parameter with results which are sometimes large.

If you really want to, let's decide on this limit now, before patch cooking. I err to conservative (large) limit (mostly in order to catch math underflows) or no limit at all. --
vda

#7 lore raw
Quoted 12 lines · Denis Vlasenko
Quoted 11 lines · Arjan van de Ven
Quoted 10 lines · Denis Vlasenko
Quoted 9 lines · Arjan van de Ven
Quoted 9 lines
Quoted 5 lines

There are a number of compile-time checks that your patch has removed which catch such things, and as such your patch is not acceptable. Some architectures have a lower threshold of acceptability for the maximum udelay value, so it's absolutely necessary to keep this.

It removes that check from x86 - other architectures retain it.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

For users, _any_ value, however large, will work for any delay function.

that's not desired though. Desired is to limit udelay() to say 2000 or so. And force anything above that to go via mdelay() (just to make it stand out as broken code ;)

An if(usec > 2000) { printk(..); dump_stack(); } will do.

No it won't - that's a run time test which will only get caught if the code is run. There's no guarantees of that.

Quoted 2 lines

But do you really want to do this? There might be legitimate reasons to compute udelay's parameter with results which are sometimes large.

Yes. udelay() has overflow issues - if you pass too large a number to udelay() you get a randomised delay because you've lost the top bits.

The maximum delay is dependent on the architecture implementation, and it depends on bogomips. There is no one single value for it. Architectures have to decide this from the way that they do the math and the expected range of bogomips.

Please - leave asm-*/delay.h alone.

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core
#8 lore raw
In reply to Denis Vlasenko
Quoted 2 lines

An if(usec > 2000) { printk(..); dump_stack(); } will do.

that's runtime not compile time.
The old situation was a compile time check which is far more powerful.

Quoted 3 lines

But do you really want to do this? There might be legitimate reasons to compute udelay's parameter with results which are sometimes large.

however that's not valid, because a really large udelay parameter will cause the delay loop math to overflow. That was the main reason for the "no more than 2ms or so" restriction.

Quoted 2 lines

If you really want to, let's decide on this limit now,

the existing code already has a limit so why not just use that?

#9 lore raw
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines

An if(usec > 2000) { printk(..); dump_stack(); } will do.

that's runtime not compile time.
The old situation was a compile time check which is far more powerful.

Ok I like compile checks too, it will stay.

But it won't help on accidental mdelay(some math) == mdelay(-1) == mdelay(4 000 000 000), so we _also_ will have an if() inside udelay(), ok?

Quoted 3 lines · Russell King

Yes. udelay() has overflow issues - if you pass too large a number to udelay() you get a randomised delay because you've lost the top bits.

Thus [umn]delay may fail in unpredictable ways with non-const parameter which is too big. And this is good exactly why?

I'm ok with making it fail, but _predictably_. With printk(), trace, whatever.

Quoted 4 lines

The maximum delay is dependent on the architecture implementation, and it depends on bogomips. There is no one single value for it. Architectures have to decide this from the way that they do the math and the expected range of bogomips.

In example I posted these limitations are lifted. Granted these limitations were not critical, but removing them can't do harm, I guess?

Quoted 1 line

Please - leave asm-*/delay.h alone.

Let's see what udelay(const) will compile down to on ppc:

asm-ppc/delay.h
===============
extern unsigned long loops_per_jiffy;
extern void __delay(unsigned int loops);
...

#define __MAX_UDELAY    (226050910UL/HZ)        /* maximum udelay argument */
#define __MAX_NDELAY    (4294967295UL/HZ)       /* maximum ndelay argument */

extern __inline__ void __udelay(unsigned int x)

{
        unsigned int loops;

        __asm__("mulhwu %0,%1,%2" : "=r" (loops) :
                "r" (x), "r" (loops_per_jiffy * 226));
        __delay(loops);
}

extern __inline__ void __ndelay(unsigned int x)

{
        unsigned int loops;

        __asm__("mulhwu %0,%1,%2" : "=r" (loops) :
                "r" (x), "r" (loops_per_jiffy * 5));
        __delay(loops);
}

extern void __bad_udelay(void); /* deliberately undefined */
extern void __bad_ndelay(void); /* deliberately undefined */

#define udelay(n) (__builtin_constant_p(n)? \
        ((n) > __MAX_UDELAY? __bad_udelay(): __udelay((n) * (19 * HZ))) : \
        __udelay((n) * (19 * HZ)))
#define ndelay(n) (__builtin_constant_p(n)? \
        ((n) > __MAX_NDELAY? __bad_ndelay(): __ndelay((n) * HZ)) : \
        __ndelay((n) * HZ))

Thus:

udelay(const) = loops_per_jiffy * 5; mulhwu thing; call to __delay()

While with proposed code:

udelay(const) = call to udelay()

Which is smaller.
--
vda

#10 lore raw
Quoted 2 lines · Denis Vlasenko

I'm ok with making it fail, but _predictably_. With printk(), trace, whatever.

BUG_ON() is probably a good candidate, especially as the implementation
is expected to be compact and can even be removed for those saving every byte by undefining CONFIG_BUG.

Maciej

#11 lore raw
In reply to Denis Vlasenko
Quoted 9 lines · Denis Vlasenko
Quoted 4 lines · Russell King

The maximum delay is dependent on the architecture implementation, and it depends on bogomips. There is no one single value for it. Architectures have to decide this from the way that they do the math and the expected range of bogomips.

In example I posted these limitations are lifted. Granted these limitations were not critical, but removing them can't do harm, I guess?

They're lifted poorly. You include a mandatory division in the path. On systems where division has to be done in code, this is not acceptable, especially when we're trying to get short delays on embedded CPUs running below 100MHz. The time it takes to do the division could swamp the required delay value.

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core
#12 lore raw
Quoted 11 lines · Russell King
Quoted 9 lines · Denis Vlasenko
Quoted 4 lines · Russell King

The maximum delay is dependent on the architecture implementation, and it depends on bogomips. There is no one single value for it. Architectures have to decide this from the way that they do the math and the expected range of bogomips.

In example I posted these limitations are lifted. Granted these limitations were not critical, but removing them can't do harm, I guess?

Quoted 5 lines

They're lifted poorly. You include a mandatory division in the path. On systems where division has to be done in code, this is not acceptable, especially when we're trying to get short delays on embedded CPUs running below 100MHz. The time it takes to do the division could swamp the required delay value.

What divisions? Where?

void udelay(unsigned int usecs)
{
        unsigned int k = usecs/1024;
        while (k--)
                __udelay(1024);
        __udelay(usecs % 1024);
}

I see no divisions. I see shifts and ANDs.
I can code them explicitly:

void udelay(unsigned int usecs)
{
        unsigned int k = usecs >> 10; /* divide by 1024 */ while (k--)
                __udelay(0x400); /* 1024 */ __udelay(usecs & 0x3ff); /* mod 1024 */
}

Should be ok now.
--
vda

#13 lore raw
In reply to Arjan van de Ven
Quoted 11 lines · Arjan van de Ven
Quoted 10 lines · Denis Vlasenko
Quoted 9 lines · Arjan van de Ven
Quoted 9 lines
Quoted 5 lines

There are a number of compile-time checks that your patch has removed which catch such things, and as such your patch is not acceptable. Some architectures have a lower threshold of acceptability for the maximum udelay value, so it's absolutely necessary to keep this.

It removes that check from x86 - other architectures retain it.

Quoted 8 lines
Quoted 2 lines

For users, _any_ value, however large, will work for any delay function.

that's not desired though. Desired is to limit udelay() to say 2000 or so. And force anything above that to go via mdelay() (just to make it stand out as broken code ;)

Over time we also want to phase out mdelay of course...

The joystick drivers will (sadly) need mdelay forever, due to hardware crappines.

Signature · 2 lines
Vojtech Pavlik
SuSE Labs, SuSE CR
#14 lore raw
In reply to Andrew Morton
Quoted 6 lines · Andrew Morton
Quoted 5 lines · Russell King

Rejected-by: Russell King 8)

The reason is that now we're unable to find out if anyone's doing udelay(100000000000000000) which breaks on most architectures.

Quoted 7 lines
Quoted 4 lines

There are a number of compile-time checks that your patch has removed which catch such things, and as such your patch is not acceptable. Some architectures have a lower threshold of acceptability for the maximum udelay value, so it's absolutely necessary to keep this.

I don't recall seeing anyone trigger the check, and it hardly seems worth adding a "few kb" to vmlinux for it?

Maybe we can have both - would the space saving be achieved by just moving mdelay and ssleep out of linux/delay.h and not touching asm-i386/delay.h?

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core
#15 lore raw
In reply to Denis Vlasenko
Quoted 7 lines · Denis Vlasenko

Hi Andrew,

Optimizing delay functions for speed is utterly pointless.

This patch turns ssleep(n), mdelay(n), udelay(n) and ndelay(n) into functions, thus they generate the smallest possible code at the callsite. Previously they were more or less inlined.

Optimizing mdelay() and udelay() for speed is pointless, but optimizing ndelay() makes a lot of sense - if the setup time (call, etc) of the delay is large, the delay time will be off by many percent.

Signature · 2 lines
Vojtech Pavlik
SuSE Labs, SuSE CR