2.6.23.1-rt9 (and others)

This is a special announcement for the latest -rt patches. This is actually announcing more than one tree (pay close attention to the differences between -rt7, -rt8 and -rt9).

2.6.23.1-rt6

- Removed BUG_ON in exit (Steven Rostedt and Daniel Walker)

- Turn RCU preempt boost on by default (Steven Rostedt) (for when RCU PREEMPT is enabled)

- Fixes for PowerPC (Paul McKenney)

2.6.23.1-rt7

- Found that there's a flaw in the PowerPC patch so it was pulled from the tree.

2.6.23.1-rt8

- More aggressive RT Balancing (Gregory Haskins)

2.6.23.1-rt9

- RT balancing by CPU priorities (Gregory Haskins)

Now benchmarks against 2.6.23.1-rt7 -rt8 and -rt9 would be greatly appreciated. These three are all present in

http://www.kernel.org/pub/linux/kernel/projects/rt/

Gregory and I have been having disagreements on how to solve RT task balancing among CPUS. Although we shared ideas back and forth, and both our methods have been greatly influenced by each other, the real answer comes from actual numbers. So these three versions are posted for your convenience to see which actually do the best. I would be happy to tell Gregory he's right, if the numbers prove it.

Currently, what we do to test RT latencies is to run Thomas Gleixner's cyclictest
(http://git.kernel.org/?p=linux/kernel/git/tglx/rt-tests.git;a=summary)
as well as hackbench, to see what the maximum latencies we get are.

Other tests are welcomed too.

to build a 2.6.23.1-rt7 tree, the following patches should be applied:

http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.1.tar.bz2
http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.23.1-rt7.bz2

to build a 2.6.23.1-rt8 tree, the following patches should be applied:

http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.1.tar.bz2
http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.23.1-rt8.bz2

to build a 2.6.23.1-rt9 tree, the following patches should be applied:

http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.1.tar.bz2
http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.23.1-rt9.bz2

And like always, my RT version of Matt Mackall's ketchup will get this for you nicely:

http://people.redhat.com/srostedt/rt/tools/ketchup-0.9.8-rt1

The broken out patches are also available.

Thanks!

-- Steve

hallo, i just tried to compile rt9 and rt7 and both times i get this error at about the end of the 'make && make modules' step:

 GEN     .version
 CHK     include/linux/compile.h
 UPD     include/linux/compile.h
 CC      init/version.o
 LD      init/built-in.o
 LD      .tmp_vmlinux1

kernel/built-in.o: In function `sched_init':
(.init.text+0x1af): undefined reference to `cpupri_init'
make: *** [.tmp_vmlinux1] Fehler 1

i am on uniprozessor machine here, IBM-thinkpad-r50e + debian testing.
if you need parts of my config file, just feel free to tell me so and i ll try to provide you the information!

cheers,
doc

Doh! Your error made me realize that I broke uniprocessor in -rt9. Will fix right away.

As far as -rt7 is concerned, that doesn't make a lot of sense since cpupri isnt introduced until -rt9. Perhaps your tree was dirtied from a previous application of -rt9? Let me know if that doesn't appear to be the case.

Regards,
-Greg

sorry, i am not sure if i did not done some missmatch by copying the config file into the tree, but i am always doing 'rm -r', and unpacking the tree before compiling. i tried again the rt9 (100% without missmatching) but it does not work....
i ll give a try with the rt6 now.

very much thanks for the support and cheers,
doc

-rt6 is broken.

I'd recommend doing the following:

wget -O /usr/local/bin/ketchup http://people.redhat.com/srostedt/rt/tools/ketchup-0.9.8-rt1

mkdir tmp
cd tmp
ketchup -r -G 2.6.23.1-rt7

This will get you the 2.6.23.1-rt7 kernel and rename the "tmp" directory to linux-2.6.23.1-rt7

After you compiled and install -rt7 while in the same directory you can do

ketchup -r -G 2.6.23.1-rt10

and it will update that kernel tree to 2.6.23.1-rt10 and again rename that directory. (2.6.23.1-rt10 which BTW has the compile fix).

-- Steve

hallo steve, very much thanks for this nice how-to, but i already finished with rt7,8 and 10.

i ll keep this message and try another time.

cheers,
doc

you were right, it was some missmatching from my side, so rt7 build successfully without adding any additional patch and rt8 bulding just now.

cheers,
doc

you were right, it was some missmatching from my side, so rt7 build successfully without adding any additional patch and rt8 is bulding just now.

cheers,
doc

Primary issue is cpupri_init() is not defined, but also clean up some warnings related to uniproc builds.

---

 kernel/sched.c        |    2 ++
 kernel/sched_cpupri.h |    5 +++++
 kernel/sched_rt.c     |    2 +-
 3 files changed, 8 insertions(+), 1 deletions(-)

index 6f24aa0..365c987 100644
@@ -863,9 +863,11 @@ static int balance_tasks(struct rq *this_rq, int this_cpu, struct rq *busiest,
 		      int *all_pinned, unsigned long *load_moved,
 		      int *this_best_prio, struct rq_iterator *iterator);
 
+#ifdef CONFIG_SMP
 static unsigned long source_load(int cpu, int type);
 static unsigned long target_load(int cpu, int type);
 static unsigned long cpu_avg_load_per_task(int cpu);
+#endif /* CONFIG_SMP */
 
 #include "sched_stats.h"
 #include "sched_rt.c"

index 8cdd15d..2119495 100644
@@ -5,6 +5,11 @@
 
 int  cpupri_find(struct task_struct *p, cpumask_t *lowest_mask);
 void cpupri_set(int cpu, int pri);
+
+#ifdef CONFIG_SMP
 void cpupri_init(void);
+#else
+#define cpupri_init() do { } while(0)
+#endif
 
 #endif /* _LINUX_CPUPRI_H */

index 71ae9e6..0213aa2 100644
@@ -207,9 +207,9 @@ yield_task_rt(struct rq *rq, struct task_struct *p)
 	requeue_task_rt(rq, p);
 }
 
+#ifdef CONFIG_SMP
 static int find_lowest_rq(struct task_struct *task);
 
-#ifdef CONFIG_SMP
 static int select_task_rq_rt(struct task_struct *p, int sync)
 {
 	struct rq *rq = task_rq(p);

[...]

-rt9 has been found to break UP compilation, so -rt10 has been released with the fix.

So please compare -rt7, -rt8 and -rt10.

-- Steve

<[email protected]>, Steven Rostedt

Here are my results from running:

sudo chrt -f 80 ./cyclictest -n -p 90 -t 8 -d 100 -i 100

with

while true; do make mrproper; make allmodconfig; make -j 128 > /dev/null; done
on a plain kernel.org kernel

running in the background on an 8-way (4core x2) C2D Xeon 5335 system

23.1-rt7
---------------------------

144.60 135.30 116.38 30/814 8184

  T: 0 ( 5990) P:90 I:100 C:11560415 Min:2 Act:  5 Avg: 4 Max:  83
  T: 1 ( 5991) P:89 I:200 C:5780208 Min: 2 Act:  5 Avg: 4 Max:  66
  T: 2 ( 5992) P:88 I:300 C:3853472 Min: 2 Act:  5 Avg: 5 Max:  89
  T: 3 ( 5993) P:87 I:400 C:2890104 Min: 2 Act:  6 Avg: 5 Max:  70
  T: 4 ( 5994) P:86 I:500 C:2312083 Min: 2 Act:  4 Avg: 5 Max:  91
  T: 5 ( 5995) P:85 I:600 C:1926736 Min: 2 Act:  5 Avg: 5 Max:  94

T: 6 ( 5996) P:84 I:700 C:1651488 Min: 2 Act: 12 Avg: 5 Max: 115
T: 7 ( 5997) P:83 I:800 C:1445052 Min: 2 Act: 6 Avg: 5 Max: 79

23.1-rt8
---------------------------
119.95 106.56 107.98 37/811 18445

  T: 0 ( 5052) P:90 I:100 C:29592746 Min: 2 Act: 21 Avg: 4 Max:  78
  T: 1 ( 5053) P:89 I:200 C:14796374 Min: 2 Act:  6 Avg: 4 Max:  81
  T: 2 ( 5054) P:88 I:300 C:9864249 Min:  2 Act: 10 Avg: 4 Max:  88
  T: 3 ( 5055) P:87 I:400 C:7398187 Min:  2 Act:  6 Avg: 4 Max:  86
  T: 4 ( 5056) P:86 I:500 C:5918550 Min:  2 Act: 13 Avg: 9 Max:  69
  T: 5 ( 5057) P:85 I:600 C:4932125 Min:  2 Act: 11 Avg: 4 Max:  71
  T: 6 ( 5058) P:84 I:700 C:4227536 Min:  2 Act:  8 Avg: 5 Max:  65

T: 7 ( 5059) P:83 I:800 C:3699094 Min: 2 Act: 4 Avg: 4 Max: 114

23.1-rt10
---------------------------
143.39 123.49 117.92 22/791 5802

  T: 0 ( 5305) P:90 I:100 C:45332860 Min: 2 Act:  4 Avg: 4 Max: 89
  T: 1 ( 5306) P:89 I:200 C:22666431 Min: 2 Act:  3 Avg: 5 Max: 49
  T: 2 ( 5307) P:88 I:300 C:15110954 Min: 2 Act:  6 Avg: 5 Max: 76
  T: 3 ( 5308) P:87 I:400 C:11333216 Min: 2 Act:  7 Avg: 5 Max: 81
  T: 4 ( 5309) P:86 I:500 C:9066572 Min:  2 Act:  8 Avg: 5 Max: 57
  T: 5 ( 5310) P:85 I:600 C:7555477 Min:  2 Act:  8 Avg: 5 Max: 55

T: 6 ( 5311) P:84 I:700 C:6476123 Min: 2 Act: 14 Avg: 6 Max: 73
T: 7 ( 5312) P:83 I:800 C:5666608 Min: 2 Act: 5 Avg: 6 Max: 78

note that the rt10 image was running for a much longer duration than the other two...which generally will push the max higher. I know in general if I let -rt7 go that long it will hit 120-150+. This means these tests were biased towards -rt7 and -rt8 performing better, but they still came in with higher latencies.

I will officially test -rt11 next, tho in my small runs so far it looks great.

HTH

Regards,
-Greg

hallo, can please someone suggest for how long time you are running cyclictest?
if i run it for about 3 min. i get a feeling that the values will not change anymore, so is there any recommended time for how long to run it?

another thing i do not know is, if i run cyclictest for lets say 3 minutes, than stop ('ctrl+c') and than start again, i get often different values. so is it recommended to run it more times and than to pick up the best/worst cases.

i have done some tests already with rt7,8 and 10 but i am not sure if the tests make any sense - thats why i would really appreciate some answers on the both questions.

cheers,
doc

Actually, you want to run it for a long time, and under very heavy load. I'm currently running it while doing a "make -j128" on a kernel tree. This takes several hours to complete the make. So far with -rt11 I have a max latency with one RT thread at the prio of 98 68us.

When you do run cyclic test, make sure you run it at a higher priority than the irq threads. And also use the -n option since the default is to test signals, and that part of the test isn't working so well.

Here's what I'm running right now:

./cyclictest -n -p98 -t1 -i250

While doing that "make -j120"

Actually I didn't want to discourage you. For UP, -rt7,8 and 10 really should be exactly the same (with some noise factures apart). The changes that were made supposable only affect SMP boxes. But, I'm glad you are doing the tests, since we also would like to know if anything creeped into te UP side, and has caused any regressions.

Thanks, for taking the time to test this.

-- Steve

ok, i was trying like './cyclictest -n -t3 -p99' but i ll have a look into -t and -i options more deeper now. but i was running it always for about 3-4 minutes....

i was expecting this part about 'smp boxes' but anyway, as now i know how to run cyclictest, i will be able to do again and again as the development goes on.

i thank you to for this very informative response!

cheers,
doc