Feedback on preemptible kernel patch
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 16 lines · Dieter Nützel
Quoted 3 lines
first, highmem is fixed and the original patch you have from me is good. second, Daniel Phillips gave me some feedback into how to figure out the VM error. I am working on it, although just the VM potential
Good to hear.
Quoted 1 line
-- ReiserFS may be another problem.
Can't wait for that.
Quoted 5 lines
third, you may be experiencing problems with a kernel optimized for Athlon. this may or may not be related to the current issues with an Athlon-optimized kernel. Basically, functions in
arch/i386/lib/mmx.cseem to need some locking to prevent preemption. I have a basic patch and we are working on a final one.
Quoted 3 lines
Can you please send this stuff along to me?
You know I own an Athlon (since yester Athlon II 1 GHz :-) and need some input...
Yes, find the Athlon patch below. Please let me know if it works.
Quoted 4 lines
Mobo is MSI MS-6167 Rev 1.0B (AMD Irongate C4, yes the very first one)
Kernel with preempt patch and mmx/3dnow! optimization crash randomly. Never had that (without preempt) during the last two years.
Oh, I did not remember you having problems with Athlon. I try to keep a list of what problems are being had.
Are there any other configurations where you have problems?
Quoted 2 lines
Thanks,
Dieter
You are welcome; thank you for the feedback.
Before applying, note there are new patches at
http://tech9.net/rml/linux - a patch for 2.4.10-pre9 and a _new_ patch for 2.4.9-ac10. These include everything (highmem, etc) except the Athlon patch.
The problem with Athlon compiled kernels is that MMX/3DNow routines used in the kernel are not preempt safe (but SMP safe, so I missed them). This patch should correct it.
arch/i386/kernel/i387.c +3 -0
@@ -10,6 +10,7 @@#include <linux/config.h> #include <linux/sched.h> +#include <linux/spinlock.h> #include <asm/processor.h> #include <asm/i387.h> #include <asm/math_emu.h>
@@ -65,6 +66,8 @@ {struct task_struct *tsk = current; +ctx_sw_off (); +if (tsk->flags &PF_USEDFPU ) {__save_init_fpu (tsk);return ;
include/asm-i386/i387.h +2 -1
@@ -12,6 +12,7 @@#define __ASM_I386_I387_H#include <linux/sched.h> +#include <linux/spinlock.h> #include <asm/processor.h> #include <asm/sigcontext.h> #include <asm/user.h>
@@ -24,7 +25,7 @@extern void restore_fpu (struct task_struct *tsk );extern void kernel_fpu_begin (void ); -#define kernel_fpu_end () stts() +#define kernel_fpu_end () stts(); ctx_sw_on()#define unlazy_fpu ( tsk ) do { \
Signature · 3 lines
Robert M. Love rml at ufl.edu rml at tech9.net
Am Freitag, 14. September 2001 06:35 schrieb Robert Love:
Quoted 5 lines · Dieter Nützel
Quoted 4 lines
Quoted 2 lines
-- ReiserFS may be another problem.
Can't wait for that.
Most wanted, now.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
third, you may be experiencing problems with a kernel optimized for Athlon. this may or may not be related to the current issues with an Athlon-optimized kernel. Basically, functions in
arch/i386/lib/mmx.cseem to need some locking to prevent preemption. I have a basic patch and we are working on a final one.
Quoted 5 lines
Quoted 3 lines
Can you please send this stuff along to me?
You know I own an Athlon (since yester Athlon II 1 GHz :-) and need some input...Yes, find the Athlon patch below. Please let me know if it works.
Tried it and it works so far.
It seems to be that kswap put some additional "load" on the disk from time to time. Or is it the ReiserFS thing, again?
Quoted 7 lines
Quoted 4 lines
Mobo is MSI MS-6167 Rev 1.0B (AMD Irongate C4, yes the very first one)
Kernel with preempt patch and mmx/3dnow! optimization crash randomly. Never had that (without preempt) during the last two years.
Oh, I did not remember you having problems with Athlon. I try to keep a list of what problems are being had.
Have you a copy of my posted ksymoops file?
But the oopses seems to be cured.
Quoted 1 line
Are there any other configurations where you have problems?
I don't know exactly 'cause kernel hacking is not my main focus.
But have you thought about the MMX/3DNow! stuff in Mesa/OpenGL (XFree86 DRI)? And what do you think about the XFree86 Xv extentions (video) or the whole MPEG2/3/4, Ogg-Vorbis, etc. (multimedia stuff)?
Do all these libraries (progs) need some preempt patches? That's why I cross posted to the DRI-Devel List (sorry).
Quoted 8 lines
Before applying, note there are new patches at
http://tech9.net/rml/linux - a patch for 2.4.10-pre9 and a _new_ patch for 2.4.9-ac10. These include everything (highmem, etc) except the Athlon patch.The problem with Athlon compiled kernels is that MMX/3DNow routines used in the kernel are not preempt safe (but SMP safe, so I missed them). This patch should correct it.
I understand ;-)
It seems to calm it.
Quoted 34 lines
diff -urN
linux-2.4.10-pre8/arch/i386/kernel/i387.clinux/arch/i386/kernel/i387.c---linux-2.4.10-pre8/arch/i386/kernel/i387.cThu Sep 13 19:24:48 2001 +++linux/arch/i386/kernel/i387.cThu Sep 13 20:00:57 2001(quoted diff) +3
-0@@ -10,6 +10,7 @@ #include <linux/config.h> #include <linux/sched.h> +#include <linux/spinlock.h> #include <asm/processor.h> #include <asm/i387.h> #include <asm/math_emu.h>@@ -65,6 +66,8 @@ { struct task_struct *tsk = current; + ctx_sw_off(); + if (tsk->flags & PF_USEDFPU) { __save_init_fpu(tsk); return;diff -urN
linux-2.4.10-pre8/include/asm-i386/i387.hlinux/include/asm-i386/i387.h---linux-2.4.10-pre8/include/asm-i386/i387.hThu Sep 13 19:27:28 2001 +++linux/include/asm-i386/i387.hThu Sep 13 20:01:30 2001(quoted diff) +1
-0@@ -12,6 +12,7 @@ #define __ASM_I386_I387_H #include <linux/sched.h> +#include <linux/spinlock.h> #include <asm/processor.h> #include <asm/sigcontext.h> #include <asm/user.h>
Review line 31
Quoted 8 lines
(quoted diff) +1
-1@@ -24,7 +25,7 @@ extern void restore_fpu( struct task_struct *tsk ); extern void kernel_fpu_begin(void); -#define kernel_fpu_end() stts() +#define kernel_fpu_end() stts(); ctx_sw_on() #define unlazy_fpu( tsk ) do { \
Now, here are my results.
Athlon II 1 GHz (0.18 µm)
MSI MS-6167 Rev 1.0B (Irongate C4)
640 MB PC100-2-2-2 SDRAM
IBM DDYS 18 GB U160 (on AHA-2940UW)
ReiserFS 3.6 on all partitions
dbench-1.1 32 clients
2.4.10-pre9
Throughput 22.8881 MB/sec (NB=28.6102 MB/sec 228.881 MBit/sec)
15.000u 52.710s 3:05.59 36.4% 0+0k 0+0io 911pf+0w
load: 3168
2.4.10-pre9 + patch-rml-2.4.10-pre9-preempt-kernel-1 Throughput 22.7157 MB/sec (NB=28.3946 MB/sec 227.157 MBit/sec)
15.070u 52.730s 3:06.97 36.2% 0+0k 0+0io 911pf+0w
load: 2984
bonnie++
2.4.10-pre
Version 1.92a ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
SunWave1 1248M 117 95 14510 16 6206 6 189 98 27205 16 289.8
4
Latency 107ms 2546ms 720ms 99241us 75832us 3449ms
Version 1.92a ------Sequential Create------ --------Random
Create--------
SunWave1 -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 4215 38 +++++ +++ 14227 93 8885 79 +++++ +++ 4324
35
Latency 584ms 8221us 14158us 7681us 14274us 794ms
load: 321
2.4.10-pre9 + patch-rml-2.4.10-pre9-preempt-kernel-1
Version 1.92a ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
SunWave1 1248M 98 96 14716 18 6190 6 181 98 27945 17 291.2
5
Latency 127ms 4241ms 726ms 97889us 63321us 3071ms
Version 1.92a ------Sequential Create------ --------Random
Create--------
SunWave1 -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 5781 54 +++++ +++ 14733 95 5899 54 +++++ +++ 11785
96
Latency 510ms 2160us 4501us 8917us 2130us 6299us
load: 302
Deleting with ReiserFS and the preempt kernel is GREAT!
But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2) or plaympeg together with dbench (16, 32). ReiserFS needs some preemption fixes, too?
I've attached two small compressed bonnie++ HTML files.
-Dieter
Am Samstag, 15. September 2001 07:14 schrieb Robert Love:
Quoted 11 lines · Dieter Nützel
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
ReiserFS may be another problem.
Can't wait for that.
Most wanted, now.
I am working on it, but I am unfamilar with it all.
Are you seeing any specific problems, now? With the latest preemption patch on 2.4.10-pre9, do you crash? oops?
No, nothing with 2.4.10-pre9 + patch-rml-2.4.10-pre9-preempt-kernel-1 and your MMX/3Dnow! fix.
2.4.10-pre10 + patch-rml-2.4.10-pre10-preempt-kernel-1 seems to be a winner!
See my results below.
Quoted 1 line
The only outstanding issue now is ReiserFS issues.
Yes, but no crash or oops for me.
"Only" some "stalls" during MPEG/Ogg-Vorbis playback (2-5 sec) :-(
Quoted 4 lines
Quoted 2 lines
It seems to be that kswap put some additional "load" on the disk from time to time. Or is it the ReiserFS thing, again?
I don't think its related to ReiserFS.
I think you are right.
Quoted 2 lines
What sort of activity are you seeing? How often? How do you know its kswapd?
I saw it with "top" at the first line (but only some few percent). It was during untarring some mid-sized archives (DRI) which took normally ~10 sec, but with kswap and 2.4.9-pre9+your patches ~30 sec. Even "sync" needed some additional seconds.
Are there some reschedule/context switch (kernel lock release) statements missing in ReiserFS?
Is this possible? Chris?
Quoted 2 lines
I am glad the patch fixed it, the final version of that is going into the next preemption patch. Stay tuned.
I am very happy with patch-rml-2.4.10-pre10-preempt-kernel-1.
Quoted 3 lines
These results are pretty good. Throughput seems down 2-3% in many cases, although latency is greatly improved. Look at those latency changes! From thousands of ms to hundreds of us in bonnie. Wow.
So look at my latest numbers. This time preempt only, sorry. If you need 2.4.10-pre10 only, too please ask.
Quoted 2 lines
Even if you don't care about latency (I'm not an audio person or anything), these changes should be worth it.
I do. Or better, one of my friend's father will do some digital video editing with Linux:-)
Quoted 4 lines
Quoted 1 line
Deleting with ReiserFS and the preempt kernel is GREAT!
Good. I/O latency should be great now, with little change in throughput...
It is.
Quoted 7 lines
Quoted 3 lines
But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2) or plaympeg together with dbench (16, 32). ReiserFS needs some preemption fixes, too?
You may still get some small hiccups ( < 1 second?) even with the preemption patch, as kernel locks prevent preemption (the patch can't guarentee low latency, just preemption outside of the locks).
Sadly 2-5 seconds at the beginning of dbench and during bonnie++ block operations (huge IO pressure, ~20% system, 3-5% user, 116308 kilobytes paged out).
Quoted 2 lines
However, how bad was the hiccups with preemption disabled? I have heard reports where it is 3-5sec at times.
Yes, nearly the same.
Quoted 3 lines
As the kernel becomes more scalable (finer-grain locking), preemption will improve. Past that, perhaps during 2.5, we can work on some other things to improve preemption.
Is this a ReiserFS only problem? Uninteruptable IO?
Quoted 3 lines
Quoted 1 line
I've attached two small compressed bonnie++ HTML files.
These were neat, thanks.
One more.
Quoted 2 lines
Thank you for your feedback and support. Stay current with the kernel and the preemption patches,
I will.
Quoted 1 line
and I will try to figure the ReiserFS crashes out.
No crashes for me only the stalls.
Regards,
Dieter
2.4.10-pre10 + patch-rml-2.4.10-pre10-preempt-kernel-1
dbench-1.1 32
Throughput 26.2881 MB/sec (NB=32.8601 MB/sec 262.881 MBit/sec)
14.320u 54.140s 2:41.70 42.3% 0+0k 0+0io 911pf+0w
load: 2931
Throughput 27.3814 MB/sec (NB=34.2267 MB/sec 273.814 MBit/sec)
14.040u 55.920s 2:35.29 45.0% 0+0k 0+0io 911pf+0w
load: 2955
dbench 16
Throughput 32.9183 MB/sec (NB=41.1479 MB/sec 329.183 MBit/sec)
6.790u 26.580s 1:05.17 51.2% 0+0k 0+0io 511pf+0w
dbench 8
Throughput 34.8936 MB/sec (NB=43.617 MB/sec 348.936 MBit/sec)
3.300u 12.760s 0:31.27 51.3% 0+0k 0+0io 311pf+0w
dbench 4
Throughput 38.3568 MB/sec (NB=47.946 MB/sec 383.568 MBit/sec)
1.770u 6.160s 0:14.77 53.6% 0+0k 0+0io 211pf+0w
SunWave1>sync
dbench 2
Throughput 53.3066 MB/sec (NB=66.6333 MB/sec 533.066 MBit/sec)
0.940u 2.970s 0:05.95 65.7% 0+0k 0+0io 161pf+0w
dbench 1
Throughput 24.0354 MB/sec (NB=30.0442 MB/sec 240.354 MBit/sec)
0.470u 1.460s 0:06.49 29.7% 0+0k 0+0io 136pf+0w
Throughput 39.1102 MB/sec (NB=48.8878 MB/sec 391.102 MBit/sec)
0.400u 1.530s 0:04.38 44.0% 0+0k 0+0io 136pf+0w
Throughput 44.6889 MB/sec (NB=55.8611 MB/sec 446.889 MBit/sec)
0.500u 1.430s 0:03.96 48.7% 0+0k 0+0io 136pf+0w
Version 1.92a ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
SunWave1 1248M 107 96 16513 21 9004 9 163 98 27844 18 256.9
3
Latency 115ms 2872ms 1743ms 121ms 64671us 3273ms
Version 1.92a ------Sequential Create------ --------Random
Create--------
SunWave1 -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 6029 69 +++++ +++ 13252 94 6762 82 +++++ +++ 11342
94
Latency 338ms 16041us 16309us 24248us 367us 17045us
load: 292
Quoted 8 lines · Dieter Nützel
Am Samstag, 15. September 2001 07:14 schrieb Robert Love:
Quoted 5 lines · Dieter Nützel
Quoted 2 lines
It seems to be that kswap put some additional "load" on the disk from time to time. Or is it the ReiserFS thing, again?
I don't think its related to ReiserFS.
I think you are right.
They were bounce buffers, meaning they can come from any page cache io, and I saw at least one report with the same symptoms without the preempt patch. I don't think it's your problem.
--
Daniel
Quoted 11 lines · Dieter Nützel
Yes, but no crash or oops for me.
"Only" some "stalls" during MPEG/Ogg-Vorbis playback (2-5 sec) :-(Quoted 7 lines
Quoted 3 lines
But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2) or plaympeg together with dbench (16, 32). ReiserFS needs some preemption fixes, too?
You may still get some small hiccups ( < 1 second?) even with the preemption patch, as kernel locks prevent preemption (the patch can't guarentee low latency, just preemption outside of the locks).
Quoted 9 lines
Sadly 2-5 seconds at the beginning of dbench and during bonnie++ block operations (huge IO pressure, ~20% system, 3-5% user, 116308 kilobytes paged out).
Quoted 2 lines
However, how bad was the hiccups with preemption disabled? I have heard reports where it is 3-5sec at times.
Yes, nearly the same.
Do you run with the playback process reniced -N?
It should really run with a low SCHED_FIFO or SCHED_RT policy. But renicing it with a negative value gives some of the benefits... (but you need to run as root)
In addition to this the program might need to lock its pages down - the only thing I can think of that could cause several seconds delay would be if it has been swapped out...
/RogerL
Signature · 3 lines
Roger Larsson Skellefteå Sweden
Quoted 7 lines · Roger Larsson
Do you run with the playback process reniced -N?
It should really run with a lowSCHED_FIFOorSCHED_RTpolicy. But renicing it with a negative value gives some of the benefits... (but you need to run as root)
In addition to this the program might need to lock its pages down - the only thing I can think of that could cause several seconds delay would be if it has been swapped out...
Certainly giving it a higher priority should improve results (especially with preemption), but the application should receive a fair amount of process attention as it is, as it is TASK_RUNNABLE at all times and the disk I/O should be routinely preempted. I am interested how much renicing it helps, though.
Now, if it has to swap pages, that is a very good point. I tend to blame this, or perhaps something with a long held lock (the audio driver?) for the blips.
Its so hard to tell swap/VM issues now with all the VM work, sadly...:)
Signature · 3 lines
Robert M. Love rml at ufl.edu rml at tech9.net
Am Mittwoch, 19. September 2001 01:31 schrieb Robert Love:
Quoted 9 lines · Roger Larsson
Quoted 7 lines
Do you run with the playback process reniced -N?
It should really run with a lowSCHED_FIFOorSCHED_RTpolicy. But renicing it with a negative value gives some of the benefits... (but you need to run as root)
In addition to this the program might need to lock its pages down - the only thing I can think of that could cause several seconds delay would be if it has been swapped out...
Quoted 5 lines
Certainly giving it a higher priority should improve results (especially with preemption), but the application should receive a fair amount of process attention as it is, as it is
TASK_RUNNABLEat all times and the disk I/O should be routinely preempted. I am interested how much renicing it helps, though.
Nearly zero :-)
Quoted 3 lines
Now, if it has to swap pages, that is a very good point. I tend to blame this, or perhaps something with a long held lock (the audio driver?) for the blips.
System didn't go into swap during whole test, sorry.
Quoted 1 line
Its so hard to tell swap/VM issues now with all the VM work, sadly...:)
I point to the second;-)
-Dieter