Skip to contentSkip to conversation

2.6.16-rc6-rt7

DISCUSSION Ingo Molnar 5 participants 11 messages Mar 16, 2006 – Mar 18, 2006 High-resolution timers, timer wheel, clockevents

lore
Share & tools
lore
#0 lore raw

i have released the 2.6.16-rc6-rt7 tree, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

the main change in this release is the merge up to John Stultz's GTOD -B20 patchset, and Thomas Gleixner's latest -hrt (high resolution timers) queue. This, amongst many other fixes, resolves a system-time (and uptime) anomaly observable under high load.

Changes since -rt4:

- merge to John Stultz's GTOD -B20 (Thomas Gleixner)

- merge to latest -hrt (Thomas Gleixner)

- zap_pte_range() latency breaker (Hugh Dickins)

- small latency tracer cleanups

to build a 2.6.16-rc6-rt7 tree, the following patches should be applied:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.15.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.16-rc6.bz2
http://redhat.com/~mingo/realtime-preempt/patch-2.6.16-rc6-rt7

Ingo

#1 lore raw
Quoted 20 lines

i have released the 2.6.16-rc6-rt7 tree, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

the main change in this release is the merge up to John Stultz's GTOD -B20 patchset, and Thomas Gleixner's latest -hrt (high resolution timers) queue. This, amongst many other fixes, resolves a system-time (and uptime) anomaly observable under high load.

Changes since -rt4:

- merge to John Stultz's GTOD -B20 (Thomas Gleixner)

- merge to latest -hrt (Thomas Gleixner)

- zap_pte_range() latency breaker (Hugh Dickins)

- small latency tracer cleanups

Quoted 8 lines

to build a 2.6.16-rc6-rt7 tree, the following patches should be applied:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.15.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.16-rc6.bz2
http://redhat.com/~mingo/realtime-preempt/patch-2.6.16-rc6-rt7

Ingo
-

I've been having issues with the realtime patch set and using scp (specifically scp, wget, curl, git, cvs everything else works fine). I was wondering what extra debugging features are helpful to have built into the kernel that could help me nail down why this bug is happening.

Specifically what's happening is scp is freezing my system, there haven't been any kernel warnings or panics upon execution of scp, it just freezes, every other application that uses network seems to work just fine, so far it's just been scp.

- David Brown

#2 lore raw
Quoted 10 lines · David Brown

I've been having issues with the realtime patch set and using scp (specifically scp, wget, curl, git, cvs everything else works fine). I was wondering what extra debugging features are helpful to have built into the kernel that could help me nail down why this bug is happening.

Specifically what's happening is scp is freezing my system, there haven't been any kernel warnings or panics upon execution of scp, it just freezes, every other application that uses network seems to work just fine, so far it's just been scp.

Just found a problem in the highres timer merge. Can you try the patch below?

tglx

include/linux/hrtimer.h +4 -0
Index: linux-2.6.16-rc6/include/linux/hrtimer.h
===================================================================
@@ -114,6 +114,8 @@ extern void hrtimer_clock_notify(void);
 extern void clock_was_set(void);
 extern int hrtimer_interrupt(void);
 
+#define hrtimer_cb_get_time(t)	(t)->base->get_time()
+
 /*
  * The resolution of the clocks. The resolution value is returned in
  * the clock_getres() system call to give application programmers an
@@ -136,6 +138,8 @@ extern int hrtimer_interrupt(void);
 #define clock_was_set()		do { } while (0)
 #define hrtimer_clock_notify()	do { } while (0)
 
+#define hrtimer_cb_get_time(t)	(t)->base->softirq_time
+
 #endif
 
 # if (BITS_PER_LONG == 64) || defined(CONFIG_KTIME_SCALAR)
kernel/hrtimer.c +0 -2
Index: linux-2.6.16-rc6/kernel/hrtimer.c
===================================================================
@@ -970,8 +970,6 @@ static inline void run_hrtimer_hres_queu
 {
 	spin_lock_irq(&base->lock);
 
-	base->softirq_time = base->get_softirq_time();
-
 	while (!list_empty(&base->cb_pending)) {
 		struct hrtimer *timer;
 		int (*fn)(struct hrtimer *);
kernel/itimer.c +1 -1
Index: linux-2.6.16-rc6/kernel/itimer.c
===================================================================
@@ -136,7 +136,7 @@ int it_real_fn(struct hrtimer *timer)
 	send_group_sig_info(SIGALRM, SEND_SIG_PRIV, sig->tsk);
 
 	if (sig->it_real_incr.tv64 != 0) {
-		hrtimer_forward(timer, timer->base->softirq_time,
+		hrtimer_forward(timer, hrtimer_cb_get_time(timer),
 				sig->it_real_incr);
 		return HRTIMER_RESTART;
 	}
kernel/posix-timers.c +2 -1
Index: linux-2.6.16-rc6/kernel/posix-timers.c
===================================================================
@@ -355,9 +355,10 @@ static int posix_timer_fn(struct hrtimer
 		if (timr->it.real.interval.tv64 != 0) {
 			timr->it_overrun +=
 				hrtimer_forward(timer,
-						timer->base->softirq_time,
+						hrtimer_cb_get_time(timer),
 						timr->it.real.interval);
 			ret = HRTIMER_RESTART;
+			++timr->it_requeue_pending;
 		}
 	}
 
#4 lore raw
In reply to Ingo Molnar

Hi,

Quoted 3 lines · Ingo Molnar

i have released the 2.6.16-rc6-rt7 tree, which can be downloaded from the usual place:

My system hangs on that:
http://www.stardust.webpages.pl/files/rt/2.6.16-rc6-rt7/oops-v1.jpg
http://www.stardust.webpages.pl/files/rt/2.6.16-rc6-rt7/oops-v2.jpg
Both looks the same.

Here is config http://www.stardust.webpages.pl/files/rt/2.6.16-rc6-rt7/rt-config

Regards,
Michal

--
Michal K. K. Piotrowski
LTG - Linux Testers Group
(http://www.stardust.webpages.pl/ltg/wiki/)

#3 lore raw
In reply to Thomas Gleixner
Quoted 7 lines · Thomas Gleixner
Quoted 6 lines · David Brown

I've been having issues with the realtime patch set and using scp (specifically scp, wget, curl, git, cvs everything else works fine). I was wondering what extra debugging features are helpful to have built into the kernel that could help me nail down why this bug is happening.

Quoted 7 lines
Quoted 4 lines

Specifically what's happening is scp is freezing my system, there haven't been any kernel warnings or panics upon execution of scp, it just freezes, every other application that uses network seems to work just fine, so far it's just been scp.

Just found a problem in the highres timer merge. Can you try the patch below?

i have released -rt8 with this fix included.

Ingo

#5 lore raw
In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

i have released the 2.6.16-rc6-rt7 tree, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

I was wondering, is it normal for the nanosleep02 and alarm02 LTP tests to fail? For sometime I've seen these tests fail from time to time with the -RT patch but not the regular kernel.

Signature · 2 lines
Tom Rini
http://gate.crashing.org/~trini/
#6 lore raw
Quoted 10 lines · Tom Rini
Quoted 4 lines · Ingo Molnar

i have released the 2.6.16-rc6-rt7 tree, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

I was wondering, is it normal for the nanosleep02 and alarm02 LTP tests to fail? For sometime I've seen these tests fail from time to time with the -RT patch but not the regular kernel.

no, it's not normal. How repeatable is it?

Ingo

#8 lore raw
In reply to Tom Rini
Quoted 10 lines · Tom Rini
Quoted 4 lines · Ingo Molnar

i have released the 2.6.16-rc6-rt7 tree, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

I was wondering, is it normal for the nanosleep02 and alarm02 LTP tests to fail? For sometime I've seen these tests fail from time to time with the -RT patch but not the regular kernel.

The nanosleep02 failure is incorrect due to rounding errors in the test code.

Requested time to sleep is 5.000009999 seconds (5s 9999ns)

Program flow is:

unsigned long req, rem, before, after, elapsed;

gettimeofday(&otime);
nanosleep(&timereq, &timerem); <- Interrupted by a signal gettimeofday(&ntime);

req = timereq.tv_sec * 1000 + timereq.tv_nsec / 1000000; rem = timerem.tv_sec * 1000 + timerem.tv_nsec / 1000000; before = otime.tv_sec * 1000 + otime.tv_usec/1000; after = ntime.tv_sec * 1000 + ntime.tv_usec/1000; elapsed = after - before;

if (rem - (req -elapsed) > 250)

	fail;

The error message is:
nanosleep02 1 FAIL : Remaining sleep time 3999 msec doesn't match with the expected 4000 msec time

rem: 3999 ms
req - elasped: 4000 ms

The unsigned long subtraction results in a value > 250, where the real result is < 0.

Looking at the real values with usec resolution gives:

req: 5000009 usec
rem: 3999740 usec
elapsed: 1000452 usec
req - elapsed: 3999557 usec

rem - (req -elapsed) = 183 usec

Truncating the real values by the division used in the test code results in:

req_ms = 5000010 / 1000 = 5000 ms
rem_ms = 3999470 / 1000 = 3999 ms
elapsed_ms = 1000452 / 1000 = 1000ms
req_ms - elapsed_ms = 4000ms

This never happens on vanilla, as the nanosleep is rounded to the next jiffie. -rt has high resolution timers which are delivered accurate, so the rounding errors of the testcode surface.

tglx

#9 lore raw
Quoted 14 lines · Thomas Gleixner
Quoted 10 lines · Tom Rini
Quoted 4 lines · Ingo Molnar

i have released the 2.6.16-rc6-rt7 tree, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

I was wondering, is it normal for the nanosleep02 and alarm02 LTP tests to fail? For sometime I've seen these tests fail from time to time with the -RT patch but not the regular kernel.

The nanosleep02 failure is incorrect due to rounding errors in the test code.

[snip]

Quoted 3 lines

This never happens on vanilla, as the nanosleep is rounded to the next jiffie. -rt has high resolution timers which are delivered accurate, so the rounding errors of the testcode surface.

Thanks! Any ideas about the alarm02 test?

Signature · 2 lines
Tom Rini
http://gate.crashing.org/~trini/
#7 lore raw
In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Tom Rini
Quoted 3 lines · Ingo Molnar

i have released the 2.6.16-rc6-rt7 tree, which can be downloaded from the usual place:

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

http://redhat.com/~mingo/realtime-preempt/

I was wondering, is it normal for the nanosleep02 and alarm02 LTP tests to fail? For sometime I've seen these tests fail from time to time with the -RT patch but not the regular kernel.

no, it's not normal. How repeatable is it?

With the feb. release of LTP I was 2/2 (./runltp -p -q -l /tmp/run.log -o /tmp/run.out). I've seen it in previous ones as well, and I think I saw it in the one run I did of the March release as well.

Signature · 2 lines
Tom Rini
http://gate.crashing.org/~trini/
#10 lore raw
In reply to Tom Rini
Quoted 13 lines · Tom Rini
Quoted 12 lines · Thomas Gleixner
Quoted 10 lines · Tom Rini
Quoted 4 lines · Ingo Molnar

i have released the 2.6.16-rc6-rt7 tree, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

I was wondering, is it normal for the nanosleep02 and alarm02 LTP tests to fail? For sometime I've seen these tests fail from time to time with the -RT patch but not the regular kernel.

Quoted 8 lines
Quoted 2 lines

The nanosleep02 failure is incorrect due to rounding errors in the test code.

[snip]

Quoted 3 lines

This never happens on vanilla, as the nanosleep is rounded to the next jiffie. -rt has high resolution timers which are delivered accurate, so the rounding errors of the testcode surface.

Thanks! Any ideas about the alarm02 test?

Yes. Its due (unsigned int) -> long conversion and a missing check.

That one affects mainline as well. I'm fixing this one right now. Patch follows.

tglx