Skip to contentSkip to conversation

Feedback on preemptible kernel patch

DISCUSSION Robert Love 5 participants 7 messages Sep 14, 2001 – Sep 20, 2001 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 16 lines · Dieter Nützel
Quoted 3 lines

first, highmem is fixed and the original patch you have from me is good. second, Daniel Phillips gave me some feedback into how to figure out the VM error. I am working on it, although just the VM potential

Good to hear.

Quoted 1 line

-- ReiserFS may be another problem.

Can't wait for that.

Quoted 5 lines

third, you may be experiencing problems with a kernel optimized for Athlon. this may or may not be related to the current issues with an Athlon-optimized kernel. Basically, functions in arch/i386/lib/mmx.c seem to need some locking to prevent preemption. I have a basic patch and we are working on a final one.

Quoted 3 lines

Can you please send this stuff along to me?
You know I own an Athlon (since yester Athlon II 1 GHz :-) and need some input...

Yes, find the Athlon patch below. Please let me know if it works.

Quoted 4 lines

Mobo is MSI MS-6167 Rev 1.0B (AMD Irongate C4, yes the very first one)

Kernel with preempt patch and mmx/3dnow! optimization crash randomly. Never had that (without preempt) during the last two years.

Oh, I did not remember you having problems with Athlon. I try to keep a list of what problems are being had.

Are there any other configurations where you have problems?

Quoted 2 lines

Thanks,
Dieter

You are welcome; thank you for the feedback.

Before applying, note there are new patches at
http://tech9.net/rml/linux - a patch for 2.4.10-pre9 and a _new_ patch for 2.4.9-ac10. These include everything (highmem, etc) except the Athlon patch.

The problem with Athlon compiled kernels is that MMX/3DNow routines used in the kernel are not preempt safe (but SMP safe, so I missed them). This patch should correct it.

arch/i386/kernel/i387.c +3 -0
diff -urN linux-2.4.10-pre8/arch/i386/kernel/i387.c linux/arch/i386/kernel/i387.c
@@ -10,6 +10,7 @@
 
 #include <linux/config.h>
 #include <linux/sched.h>
+#include <linux/spinlock.h>
 #include <asm/processor.h>
 #include <asm/i387.h>
 #include <asm/math_emu.h>
@@ -65,6 +66,8 @@
 {
 	struct task_struct *tsk = current;
 
+	ctx_sw_off();
+	
 	if (tsk->flags & PF_USEDFPU) {
 		__save_init_fpu(tsk);
 		return;
include/asm-i386/i387.h +2 -1
diff -urN linux-2.4.10-pre8/include/asm-i386/i387.h linux/include/asm-i386/i387.h
@@ -12,6 +12,7 @@
 #define __ASM_I386_I387_H
 
 #include <linux/sched.h>
+#include <linux/spinlock.h>
 #include <asm/processor.h>
 #include <asm/sigcontext.h>
 #include <asm/user.h>
@@ -24,7 +25,7 @@
 extern void restore_fpu( struct task_struct *tsk );
 
 extern void kernel_fpu_begin(void);
-#define kernel_fpu_end() stts()
+#define kernel_fpu_end() stts(); ctx_sw_on()
 
 
 #define unlazy_fpu( tsk ) do { \
Signature · 3 lines
Robert M. Love
rml at ufl.edu
rml at tech9.net
#2 lore raw
In reply to a message not in the archive

Am Freitag, 14. September 2001 06:35 schrieb Robert Love:

Quoted 5 lines · Dieter Nützel
Quoted 4 lines
Quoted 2 lines

-- ReiserFS may be another problem.

Can't wait for that.

Most wanted, now.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

third, you may be experiencing problems with a kernel optimized for Athlon. this may or may not be related to the current issues with an Athlon-optimized kernel. Basically, functions in arch/i386/lib/mmx.c seem to need some locking to prevent preemption. I have a basic patch and we are working on a final one.

Quoted 5 lines
Quoted 3 lines

Can you please send this stuff along to me?
You know I own an Athlon (since yester Athlon II 1 GHz :-) and need some input...

Yes, find the Athlon patch below. Please let me know if it works.

Tried it and it works so far.

It seems to be that kswap put some additional "load" on the disk from time to time. Or is it the ReiserFS thing, again?

Quoted 7 lines
Quoted 4 lines

Mobo is MSI MS-6167 Rev 1.0B (AMD Irongate C4, yes the very first one)

Kernel with preempt patch and mmx/3dnow! optimization crash randomly. Never had that (without preempt) during the last two years.

Oh, I did not remember you having problems with Athlon. I try to keep a list of what problems are being had.

Have you a copy of my posted ksymoops file?
But the oopses seems to be cured.

Quoted 1 line

Are there any other configurations where you have problems?

I don't know exactly 'cause kernel hacking is not my main focus.

But have you thought about the MMX/3DNow! stuff in Mesa/OpenGL (XFree86 DRI)? And what do you think about the XFree86 Xv extentions (video) or the whole MPEG2/3/4, Ogg-Vorbis, etc. (multimedia stuff)?

Do all these libraries (progs) need some preempt patches? That's why I cross posted to the DRI-Devel List (sorry).

Quoted 8 lines

Before applying, note there are new patches at
http://tech9.net/rml/linux - a patch for 2.4.10-pre9 and a _new_ patch for 2.4.9-ac10. These include everything (highmem, etc) except the Athlon patch.

The problem with Athlon compiled kernels is that MMX/3DNow routines used in the kernel are not preempt safe (but SMP safe, so I missed them). This patch should correct it.

I understand ;-)
It seems to calm it.

Quoted 34 lines

diff -urN linux-2.4.10-pre8/arch/i386/kernel/i387.c linux/arch/i386/kernel/i387.c ---
linux-2.4.10-pre8/arch/i386/kernel/i387.c Thu Sep 13 19:24:48 2001 +++
linux/arch/i386/kernel/i387.c Thu Sep 13 20:00:57 2001

(quoted diff) +3 -0
@@ -10,6 +10,7 @@
 
 #include <linux/config.h>
 #include <linux/sched.h>
+#include <linux/spinlock.h>
 #include <asm/processor.h>
 #include <asm/i387.h>
 #include <asm/math_emu.h>
@@ -65,6 +66,8 @@
 {
 	struct task_struct *tsk = current;
 
+	ctx_sw_off();
+
 	if (tsk->flags & PF_USEDFPU) {
 		__save_init_fpu(tsk);
 		return;

diff -urN linux-2.4.10-pre8/include/asm-i386/i387.h linux/include/asm-i386/i387.h ---
linux-2.4.10-pre8/include/asm-i386/i387.h Thu Sep 13 19:27:28 2001 +++
linux/include/asm-i386/i387.h Thu Sep 13 20:01:30 2001

(quoted diff) +1 -0
@@ -12,6 +12,7 @@
 #define __ASM_I386_I387_H
 
 #include <linux/sched.h>
+#include <linux/spinlock.h>
 #include <asm/processor.h>
 #include <asm/sigcontext.h>
 #include <asm/user.h>

Review line 31

Quoted 8 lines
(quoted diff) +1 -1
@@ -24,7 +25,7 @@
 extern void restore_fpu( struct task_struct *tsk );
 
 extern void kernel_fpu_begin(void);
-#define kernel_fpu_end() stts()
+#define kernel_fpu_end() stts(); ctx_sw_on()
 
 
 #define unlazy_fpu( tsk ) do { \

Now, here are my results.

Athlon II 1 GHz (0.18 µm)
MSI MS-6167 Rev 1.0B (Irongate C4)
640 MB PC100-2-2-2 SDRAM
IBM DDYS 18 GB U160 (on AHA-2940UW)
ReiserFS 3.6 on all partitions

dbench-1.1 32 clients

2.4.10-pre9
Throughput 22.8881 MB/sec (NB=28.6102 MB/sec 228.881 MBit/sec)
15.000u 52.710s 3:05.59 36.4% 0+0k 0+0io 911pf+0w
load: 3168

2.4.10-pre9 + patch-rml-2.4.10-pre9-preempt-kernel-1 Throughput 22.7157 MB/sec (NB=28.3946 MB/sec 227.157 MBit/sec)
15.070u 52.730s 3:06.97 36.2% 0+0k 0+0io 911pf+0w
load: 2984

bonnie++

2.4.10-pre
Version 1.92a ------Sequential Output------ --Sequential Input-
--Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 

--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
SunWave1 1248M 117 95 14510 16 6206 6 189 98 27205 16 289.8
4
Latency 107ms 2546ms 720ms 99241us 75832us 3449ms
Version 1.92a ------Sequential Create------ --------Random
Create--------
SunWave1 -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP

                 16  4215  38 +++++ +++ 14227  93  8885  79 +++++ +++  4324  

35
Latency 584ms 8221us 14158us 7681us 14274us 794ms
load: 321

2.4.10-pre9 + patch-rml-2.4.10-pre9-preempt-kernel-1
Version 1.92a ------Sequential Output------ --Sequential Input-
--Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 

--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
SunWave1 1248M 98 96 14716 18 6190 6 181 98 27945 17 291.2
5
Latency 127ms 4241ms 726ms 97889us 63321us 3071ms
Version 1.92a ------Sequential Create------ --------Random
Create--------
SunWave1 -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP

                 16  5781  54 +++++ +++ 14733  95  5899  54 +++++ +++ 11785  

96
Latency 510ms 2160us 4501us 8917us 2130us 6299us
load: 302

Deleting with ReiserFS and the preempt kernel is GREAT!
But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2) or plaympeg together with dbench (16, 32). ReiserFS needs some preemption fixes, too?

I've attached two small compressed bonnie++ HTML files.

-Dieter

#3 lore raw
In reply to a message not in the archive

Am Samstag, 15. September 2001 07:14 schrieb Robert Love:

Quoted 11 lines · Dieter Nützel
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

ReiserFS may be another problem.

Can't wait for that.

Most wanted, now.

I am working on it, but I am unfamilar with it all.

Are you seeing any specific problems, now? With the latest preemption patch on 2.4.10-pre9, do you crash? oops?

No, nothing with 2.4.10-pre9 + patch-rml-2.4.10-pre9-preempt-kernel-1 and your MMX/3Dnow! fix.

2.4.10-pre10 + patch-rml-2.4.10-pre10-preempt-kernel-1 seems to be a winner!

See my results below.

Quoted 1 line

The only outstanding issue now is ReiserFS issues.

Yes, but no crash or oops for me.
"Only" some "stalls" during MPEG/Ogg-Vorbis playback (2-5 sec) :-(

Quoted 4 lines
Quoted 2 lines

It seems to be that kswap put some additional "load" on the disk from time to time. Or is it the ReiserFS thing, again?

I don't think its related to ReiserFS.

I think you are right.

Quoted 2 lines

What sort of activity are you seeing? How often? How do you know its kswapd?

I saw it with "top" at the first line (but only some few percent). It was during untarring some mid-sized archives (DRI) which took normally ~10 sec, but with kswap and 2.4.9-pre9+your patches ~30 sec. Even "sync" needed some additional seconds.

Are there some reschedule/context switch (kernel lock release) statements missing in ReiserFS?

Is this possible? Chris?

Quoted 2 lines

I am glad the patch fixed it, the final version of that is going into the next preemption patch. Stay tuned.

I am very happy with patch-rml-2.4.10-pre10-preempt-kernel-1.

Quoted 3 lines

These results are pretty good. Throughput seems down 2-3% in many cases, although latency is greatly improved. Look at those latency changes! From thousands of ms to hundreds of us in bonnie. Wow.

So look at my latest numbers. This time preempt only, sorry. If you need 2.4.10-pre10 only, too please ask.

Quoted 2 lines

Even if you don't care about latency (I'm not an audio person or anything), these changes should be worth it.

I do. Or better, one of my friend's father will do some digital video editing with Linux:-)

Quoted 4 lines
Quoted 1 line

Deleting with ReiserFS and the preempt kernel is GREAT!

Good. I/O latency should be great now, with little change in throughput...

It is.

Quoted 7 lines
Quoted 3 lines

But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2) or plaympeg together with dbench (16, 32). ReiserFS needs some preemption fixes, too?

You may still get some small hiccups ( < 1 second?) even with the preemption patch, as kernel locks prevent preemption (the patch can't guarentee low latency, just preemption outside of the locks).

Sadly 2-5 seconds at the beginning of dbench and during bonnie++ block operations (huge IO pressure, ~20% system, 3-5% user, 116308 kilobytes paged out).

Quoted 2 lines

However, how bad was the hiccups with preemption disabled? I have heard reports where it is 3-5sec at times.

Yes, nearly the same.

Quoted 3 lines

As the kernel becomes more scalable (finer-grain locking), preemption will improve. Past that, perhaps during 2.5, we can work on some other things to improve preemption.

Is this a ReiserFS only problem? Uninteruptable IO?

Quoted 3 lines
Quoted 1 line

I've attached two small compressed bonnie++ HTML files.

These were neat, thanks.

One more.

Quoted 2 lines

Thank you for your feedback and support. Stay current with the kernel and the preemption patches,

I will.

Quoted 1 line

and I will try to figure the ReiserFS crashes out.

No crashes for me only the stalls.

Regards,
Dieter

2.4.10-pre10 + patch-rml-2.4.10-pre10-preempt-kernel-1

dbench-1.1 32
Throughput 26.2881 MB/sec (NB=32.8601 MB/sec 262.881 MBit/sec)
14.320u 54.140s 2:41.70 42.3% 0+0k 0+0io 911pf+0w
load: 2931
Throughput 27.3814 MB/sec (NB=34.2267 MB/sec 273.814 MBit/sec)
14.040u 55.920s 2:35.29 45.0% 0+0k 0+0io 911pf+0w
load: 2955

dbench 16
Throughput 32.9183 MB/sec (NB=41.1479 MB/sec 329.183 MBit/sec)
6.790u 26.580s 1:05.17 51.2% 0+0k 0+0io 511pf+0w

dbench 8
Throughput 34.8936 MB/sec (NB=43.617 MB/sec 348.936 MBit/sec)
3.300u 12.760s 0:31.27 51.3% 0+0k 0+0io 311pf+0w

dbench 4
Throughput 38.3568 MB/sec (NB=47.946 MB/sec 383.568 MBit/sec)
1.770u 6.160s 0:14.77 53.6% 0+0k 0+0io 211pf+0w
SunWave1>sync

dbench 2
Throughput 53.3066 MB/sec (NB=66.6333 MB/sec 533.066 MBit/sec)
0.940u 2.970s 0:05.95 65.7% 0+0k 0+0io 161pf+0w

dbench 1
Throughput 24.0354 MB/sec (NB=30.0442 MB/sec 240.354 MBit/sec)
0.470u 1.460s 0:06.49 29.7% 0+0k 0+0io 136pf+0w
Throughput 39.1102 MB/sec (NB=48.8878 MB/sec 391.102 MBit/sec)
0.400u 1.530s 0:04.38 44.0% 0+0k 0+0io 136pf+0w
Throughput 44.6889 MB/sec (NB=55.8611 MB/sec 446.889 MBit/sec)
0.500u 1.430s 0:03.96 48.7% 0+0k 0+0io 136pf+0w

Version 1.92a ------Sequential Output------ --Sequential Input-
--Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 

--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
SunWave1 1248M 107 96 16513 21 9004 9 163 98 27844 18 256.9
3
Latency 115ms 2872ms 1743ms 121ms 64671us 3273ms
Version 1.92a ------Sequential Create------ --------Random
Create--------
SunWave1 -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP

                 16  6029  69 +++++ +++ 13252  94  6762  82 +++++ +++ 11342  

94
Latency 338ms 16041us 16309us 24248us 367us 17045us
load: 292

#4 lore raw
In reply to a message not in the archive
Quoted 8 lines · Dieter Nützel

Am Samstag, 15. September 2001 07:14 schrieb Robert Love:

Quoted 5 lines · Dieter Nützel
Quoted 2 lines

It seems to be that kswap put some additional "load" on the disk from time to time. Or is it the ReiserFS thing, again?

I don't think its related to ReiserFS.

I think you are right.

They were bounce buffers, meaning they can come from any page cache io, and I saw at least one report with the same symptoms without the preempt patch. I don't think it's your problem.

--
Daniel

#5 lore raw
In reply to a message not in the archive
Quoted 11 lines · Dieter Nützel

Yes, but no crash or oops for me.
"Only" some "stalls" during MPEG/Ogg-Vorbis playback (2-5 sec) :-(

Quoted 7 lines
Quoted 3 lines

But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2) or plaympeg together with dbench (16, 32). ReiserFS needs some preemption fixes, too?

You may still get some small hiccups ( < 1 second?) even with the preemption patch, as kernel locks prevent preemption (the patch can't guarentee low latency, just preemption outside of the locks).

Quoted 9 lines

Sadly 2-5 seconds at the beginning of dbench and during bonnie++ block operations (huge IO pressure, ~20% system, 3-5% user, 116308 kilobytes paged out).

Quoted 2 lines

However, how bad was the hiccups with preemption disabled? I have heard reports where it is 3-5sec at times.

Yes, nearly the same.

Do you run with the playback process reniced -N?
It should really run with a low SCHED_FIFO or SCHED_RT policy. But renicing it with a negative value gives some of the benefits... (but you need to run as root)
In addition to this the program might need to lock its pages down - the only thing I can think of that could cause several seconds delay would be if it has been swapped out...

/RogerL

Signature · 3 lines
Roger Larsson
Skellefteå
Sweden
#6 lore raw
Quoted 7 lines · Roger Larsson

Do you run with the playback process reniced -N?
It should really run with a low SCHED_FIFO or SCHED_RT policy. But renicing it with a negative value gives some of the benefits... (but you need to run as root)
In addition to this the program might need to lock its pages down - the only thing I can think of that could cause several seconds delay would be if it has been swapped out...

Certainly giving it a higher priority should improve results (especially with preemption), but the application should receive a fair amount of process attention as it is, as it is TASK_RUNNABLE at all times and the disk I/O should be routinely preempted. I am interested how much renicing it helps, though.

Now, if it has to swap pages, that is a very good point. I tend to blame this, or perhaps something with a long held lock (the audio driver?) for the blips.

Its so hard to tell swap/VM issues now with all the VM work, sadly...:)

Signature · 3 lines
Robert M. Love
rml at ufl.edu
rml at tech9.net
#7 lore raw
In reply to a message not in the archive

Am Mittwoch, 19. September 2001 01:31 schrieb Robert Love:

Quoted 9 lines · Roger Larsson
Quoted 7 lines

Do you run with the playback process reniced -N?
It should really run with a low SCHED_FIFO or SCHED_RT policy. But renicing it with a negative value gives some of the benefits... (but you need to run as root)
In addition to this the program might need to lock its pages down - the only thing I can think of that could cause several seconds delay would be if it has been swapped out...

Quoted 5 lines

Certainly giving it a higher priority should improve results (especially with preemption), but the application should receive a fair amount of process attention as it is, as it is TASK_RUNNABLE at all times and the disk I/O should be routinely preempted. I am interested how much renicing it helps, though.

Nearly zero :-)

Quoted 3 lines

Now, if it has to swap pages, that is a very good point. I tend to blame this, or perhaps something with a long held lock (the audio driver?) for the blips.

System didn't go into swap during whole test, sorry.

Quoted 1 line

Its so hard to tell swap/VM issues now with all the VM work, sadly...:)

I point to the second;-)

-Dieter