Skip to contentSkip to conversation

2.6.16-rt1

DISCUSSION Ingo Molnar 7 participants 16 messages Mar 20, 2006 – Mar 23, 2006

lore
Share & tools
lore
#1 lore raw
Quoted 8 lines · Ingo Molnar

i have released the 2.6.16-rt1 tree, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

this is mostly a merge from -rc6 to 2.6.16-final, plus some smaller fixes.

Quoted 6 lines

to build a 2.6.16-rt1 tree, the following patches should be applied:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.tar.bz2
http://redhat.com/~mingo/realtime-preempt/patch-2.6.16-rt1

Ingo

Ingo,

I haven't had a chance to look into it yet but this combination brings with it a significant latency regression, at least as measured by the rtc histogram stuff.

Signature · 1 line
   kr
#10 lore raw
In reply to Ingo Molnar

lundi 20 Mars 2006 09:51, Ingo Molnar wrote/a écrit :

Quoted 7 lines

i have released the 2.6.16-rt1 tree, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

this is mostly a merge from -rc6 to 2.6.16-final, plus some smaller fixes.

I get the following error :
...
Kernel: arch/i386/boot/bzImage is ready (#2)
Building modules, stage 2.
MODPOST
*** Warning: "mutex_destroy" [fs/xfs/xfs.ko] undefined!
*** Warning: "there_is_no_init_MUTEX_LOCKED_for_RT_semaphores" [fs/xfs/xfs.ko] undefined!
...
Is it a BUG ?

Signature · 1 line
Serge Noiraud
#11 lore raw

IIRC, I've seen this error already in a post for 2.6.16-rc6-rt1

Quoted 1 line

*** Warning: "mutex_destroy" [fs/xfs/xfs.ko] undefined!

mutex_destroy isn't defined for CONFIG_PREEMPT_RT.

In the past mutex_destroy was defined in fs/xfs/linux_26/mutex.h, but this has been moved. Have a look at: include/linux/mutex-debug.h, kernel/mutex-debug.c, include/linux/mutex.h

JAN

#12 lore raw
In reply to Serge Noiraud
Quoted 9 lines · Serge Noiraud

I get the following error :
...
Kernel: arch/i386/boot/bzImage is ready (#2)
Building modules, stage 2.
MODPOST
*** Warning: "mutex_destroy" [fs/xfs/xfs.ko] undefined!
*** Warning: "there_is_no_init_MUTEX_LOCKED_for_RT_semaphores" [fs/xfs/xfs.ko] undefined!
...
Is it a BUG ?

could you check -rt2?

Ingo

#14 lore raw
Quoted 7 lines · Michal Piotrowski

Hi Ingo,

Quoted 3 lines · Ingo Molnar

could you check -rt2?

Quoted 6 lines

Here is first oops http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/b1.jpg
Here is second oops http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/b2.jpg

Here is config http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/rt-config

I can't boot that kernel, 2.6.16-rt1 was fine.

ok, i broke irqs-off latency tracing in -rt2. Could you try -rt3 - it should be fixed there.

Ingo

#2 lore raw
In reply to K.R. Foley
Quoted 3 lines · K.R. Foley

I haven't had a chance to look into it yet but this combination brings with it a significant latency regression, at least as measured by the rtc histogram stuff.

can you also measure it via rtc_wakeup, a'ka:

chrt -f -p 98 pidof 'IRQ 8'
./rtc_wakeup -f 8192 -t 100000

? I have just tried it, and -rt4 looks pretty good on the latency front (with all debugging disabled).

Ingo

#15 lore raw
In reply to Ingo Molnar
Quoted 13 lines · Ingo Molnar
Quoted 10 lines · Michal Piotrowski

Hi Ingo,

Quoted 3 lines · Ingo Molnar

could you check -rt2?

Here is first oops http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/b1.jpg
Here is second oops http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/b2.jpg

Quoted 9 lines
Quoted 3 lines

Here is config http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/rt-config

I can't boot that kernel, 2.6.16-rt1 was fine.

ok, i broke irqs-off latency tracing in -rt2. Could you try -rt3 - it should be fixed there.

Ingo

Thanks!
Problem fixed.

Regards,
Michal

--
Michal K. K. Piotrowski
LTG - Linux Testers Group
(http://www.stardust.webpages.pl/ltg/wiki/)

#3 lore raw
In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar
Quoted 3 lines · K.R. Foley

I haven't had a chance to look into it yet but this combination brings with it a significant latency regression, at least as measured by the rtc histogram stuff.

can you also measure it via rtc_wakeup, a'ka:

Quoted 8 lines

chrt -f -p 98 pidof 'IRQ 8'
./rtc_wakeup -f 8192 -t 100000

? I have just tried it, and -rt4 looks pretty good on the latency front (with all debugging disabled).

Ingo

Sorry I have been onsite and completely buried today. Am running an initial test on both UP and SMP now with 2.6.16-rt1. UP doesn't look bad at all. SMP on the other hand doesn't look so good. I will give -rt4 a spin when these are done.

Signature · 1 line
   kr
#4 lore raw
Quoted 4 lines · K.R. Foley

Sorry I have been onsite and completely buried today. Am running an initial test on both UP and SMP now with 2.6.16-rt1. UP doesn't look bad at all. SMP on the other hand doesn't look so good. I will give -rt4 a spin when these are done.

thanks for the testing - i'll check SMP too.

Ingo

#5 lore raw
Quoted 11 lines · Ingo Molnar
Quoted 4 lines · K.R. Foley

Sorry I have been onsite and completely buried today. Am running an initial test on both UP and SMP now with 2.6.16-rt1. UP doesn't look bad at all. SMP on the other hand doesn't look so good. I will give -rt4 a spin when these are done.

thanks for the testing - i'll check SMP too.

Ingo

OK. On my dual 933 under heavy load I get the following with 2.6.16-rt4 and I get tons of missed interrupts. Running 2.6.15-rc16 I get a max of 88usec with most falling under 30usec. On my UP AthlonXP 1700 I get a max of 19usec with 2.6.16-rt4 under load. What sort of results do you see on SMP?

rtc latency histogram of {rtc_wakeup/7095, 17528666 samples}:
0 85
1 169
2 200
3 348
4 704
5 950
6 661
7 1123
8 1243
9 1217
10 1396
11 1932
12 1640
13 1136
14 839
15 9896
16 128030
17 653034
18 649117
19 980871
20 3100062
21 1568493
22 979036
23 811244
24 959543
25 732209
26 681492
27 856068
28 939870
29 649806
30 449958
31 367307
32 322118
33 238856
34 205575
35 184051
36 148624
37 147437
38 134690
39 126161
40 141023
41 192116
42 227147
43 199154
44 143090
45 101829
46 74295
47 55930
48 44070
49 35757
50 31143
51 28283
52 26081
53 24561
54 21771
55 20275
56 17621
57 14803
58 12567
59 10606
60 9132
61 8132
62 6978
63 5865
64 4751
65 4077
66 3367
67 3008
68 2718
69 2621
70 2504
71 2390
72 2071
73 1699
74 1430
75 1118
76 957
77 774
78 710
79 575
80 560
81 467
82 410
83 372
84 343
85 298
86 263
87 230
88 176
89 188
90 157
91 130
92 107
93 122
94 89
95 64
96 41
97 40
98 51
99 42
100 34
101 33
102 14
103 20
104 17
105 15
106 12
107 6
108 10
109 9
110 8
111 10
112 6
113 7
114 6
115 1
116 7
117 6
118 4
119 4
120 5
121 1
122 1
123 4
124 5
125 1
126 3
127 1
128 3
129 1
130 4
132 6
133 3
134 7
135 2
136 3
137 4
138 5
139 1
140 1
141 2
142 2
143 5
144 1
147 2
148 2
149 1
150 1
151 2
152 1
154 1
155 1
159 1
160 3
163 1
171 1
211 1
212 1
475 1
9187 1
9192 1
9219 1
9231 1
9259 1
9268 1

Signature · 1 line
   kr
#6 lore raw
Quoted 9 lines · K.R. Foley
Quoted 8 lines · Ingo Molnar
Quoted 4 lines · K.R. Foley

Sorry I have been onsite and completely buried today. Am running an initial test on both UP and SMP now with 2.6.16-rt1. UP doesn't look bad at all. SMP on the other hand doesn't look so good. I will give -rt4 a spin when these are done.

thanks for the testing - i'll check SMP too.

Quoted 8 lines
Quoted 2 lines

Ingo

OK. On my dual 933 under heavy load I get the following with 2.6.16-rt4 and I get tons of missed interrupts. Running 2.6.15-rc16 I get a max of 88usec with most falling under 30usec. On my UP AthlonXP 1700 I get a max of 19usec with 2.6.16-rt4 under load. What sort of results do you see on SMP?

Found something interesting. Having Wakeup latency timing turned on makes a HUGE difference. I turned it off and recompiled and now I am seeing numbers back in line with what I expected from 2.6.16-rt4. Sorry, but I had no idea it would make that much difference. I don't have a complete run yet, but I have seen enough to know that I am not seeing tons of missed interrupts and the highest reported latency thus far is 61 usec.

Signature · 1 line
   kr
#7 lore raw
Quoted 7 lines · K.R. Foley

Found something interesting. Having Wakeup latency timing turned on makes a HUGE difference. I turned it off and recompiled and now I am seeing numbers back in line with what I expected from 2.6.16-rt4. Sorry, but I had no idea it would make that much difference. I don't have a complete run yet, but I have seen enough to know that I am not seeing tons of missed interrupts and the highest reported latency thus far is 61 usec.

Just Wakeup latency timing , and not latency tracing ?

Daniel

#8 lore raw
Quoted 10 lines · Daniel Walker
Quoted 7 lines · K.R. Foley

Found something interesting. Having Wakeup latency timing turned on makes a HUGE difference. I turned it off and recompiled and now I am seeing numbers back in line with what I expected from 2.6.16-rt4. Sorry, but I had no idea it would make that much difference. I don't have a complete run yet, but I have seen enough to know that I am not seeing tons of missed interrupts and the highest reported latency thus far is 61 usec.

Quoted 5 lines

Just Wakeup latency timing , and not latency tracing ?

Daniel

Unfortunately I can't say for sure because I don't remember turning it off, but looking at the log it appears as though latency tracing was turned on

Signature · 1 line
   kr
#9 lore raw
In reply to K.R. Foley
Quoted 19 lines · K.R. Foley
Quoted 9 lines · K.R. Foley
Quoted 8 lines · Ingo Molnar
Quoted 4 lines · K.R. Foley

Sorry I have been onsite and completely buried today. Am running an initial test on both UP and SMP now with 2.6.16-rt1. UP doesn't look bad at all. SMP on the other hand doesn't look so good. I will give -rt4 a spin when these are done.

thanks for the testing - i'll check SMP too.

Quoted 8 lines
Quoted 2 lines

Ingo

OK. On my dual 933 under heavy load I get the following with 2.6.16-rt4 and I get tons of missed interrupts. Running 2.6.15-rc16 I get a max of 88usec with most falling under 30usec. On my UP AthlonXP 1700 I get a max of 19usec with 2.6.16-rt4 under load. What sort of results do you see on SMP?

Quoted 7 lines

Found something interesting. Having Wakeup latency timing turned on makes a HUGE difference. I turned it off and recompiled and now I am seeing numbers back in line with what I expected from 2.6.16-rt4. Sorry, but I had no idea it would make that much difference. I don't have a complete run yet, but I have seen enough to know that I am not seeing tons of missed interrupts and the highest reported latency thus far is 61 usec.

Hmm, high wake up latency on SMP and not on UP...

Ingo, could this be due to the migrate task latency? This was where I saw the problem with the 50ms latency running hack bench. I remember there was a bug in the older latency tool that didn't catch this latency before.

I'm just getting back to looking at the latest stuff. I had some customer deliveries lately and haven't had time to look at the new goodies.

-- Steve