Linux 2.4.21-rc6
Subsystems (6)
Hi,
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
Please test it.
Summary of changes from v2.4.21-rc5 to v2.4.21-rc6
============================================
<[email protected]>:
o IDE config.in correctness
Andi Kleen <[email protected]>:
o x86-64 fix for the ioport problem
Andrew Morton <[email protected]>:
o Fix IO stalls and deadlocks
Marcelo Tosatti <[email protected]>:
o Add missing via82xxx PCI ID
o Backout erroneous fsync on last opener at close()
o Changed EXTRAVERSION to -rc6
Quoted 2 lines · Marcelo Tosatti
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
Good for you. Well done Marcelo!
Quoted 1 line
Please test it.
Yes everyone who gets these stalls please test it also!
Quoted 2 lines
Andrew Morton <[email protected]>:
o Fix IO stalls and deadlocks
For those interested these are patches 1 and 2 from akpm's proposed fixes in the looong thread discussing this problem.
Con
Quoted 6 lines · Con Kolivas
Quoted 2 lines · Marcelo Tosatti
Andrew Morton <[email protected]>:
o Fix IO stalls and deadlocksFor those interested these are patches 1 and 2 from akpm's proposed fixes in the looong thread discussing this problem.
Are you sure? I'm no C programmer, but it looks to me like all three patches are in 21-rc6.
And I still see the stalls, although it's much reduced. :( I just had mutt freeze cold on me though for ~15 sec when it tried to open my debian-devel mbox (rather lage file) while brag was beating on the drive.
<whimper>
Signature · 2 lines
Marc Wilson | You have had a long-term stimulation relative to [email protected] | business.
Hi Marc.
Quoted 5 lines
I just had mutt > freeze cold on me though for ~15 sec when it tried to open my debian-devel mbox (rather large file) while brag was beating on the drive.
<whimper>
I used to get the same effect when I asked pine to open the Linux-Kernel mailbox on my system. I long since cured that by having procmail split Linux-Kernel mail into multiple mailboxes, one for each calendar week.
The basic problem there is that any mail client needs to know just how many messages are in a particular folder to handle that folder, and the only way to do this is to count them all. That's what takes the time when one opens a large folder.
Best wishes from Riley.
---
* Nothing as pretty as a smile, nothing as ugly as a frown.
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.484 / Virus Database: 282 - Release Date: 27-May-2003
Quoted 4 lines · Riley Williams
The basic problem there is that any mail client needs to know just how many messages are in a particular folder to handle that folder, and the only way to do this is to count them all. That's what takes the time when one opens a large folder.
No, the basic problem there is that the kernel is deadlocking. Read the VERY long thread for the details.
I think I have enough on the ball to be able to tell the difference between mutt opening a folder and counting messages, with a counter and percentage indicator advancing, and mutt sitting there deadlocked with the HD activity light stuck on and all the rest of X stuck tight.
And it just happened again, so -rc6 is no sure fix. What did y'all that reported the problem had gone away do, patch -rc4 with the akpm patches? ^_^
Signature · 2 lines
Marc Wilson | Fortune favors the lucky. [email protected] |
Hi Marc.
Quoted 5 lines
Quoted 4 lines
The basic problem there is that any mail client needs to know just how many messages are in a particular folder to handle that folder, and the only way to do this is to count them all. That's what takes the time when one opens a large folder.
Quoted 8 lines
No, the basic problem there is that the kernel is deadlocking. Read the VERY long thread for the details.
I think I have enough on the ball to be able to tell the difference between mutt opening a folder and counting messages, with a counter and percentage indicator advancing, and mutt sitting there deadlocked with the HD activity light stuck on and all the rest of X stuck tight.
I thought I was on the ball when a similar situation happened to me. What I observed was that the counters and percentage indicators were NOT advancing for about 30 seconds, and both would then jump up by about 70 messages and the relevant percent rather than counting smoothly through. It was only when I noticed those jumps that I went back to basics and analysed the folder rather than the kernel.
However, I apologise profusely for assuming that my experience in what to me appear to be similar circumstances to yours could have any sort of bearing on the problem you are seeing.
Quoted 3 lines
And it just happened again, so -rc6 is no sure fix. What did y'all that reported the problem had gone away do, patch -rc4 with the akpm patches?
In my case, I fixed the problem by splitting the relevant folder up, as stated in my previous message. However, such a solution apparently doesn't work for you, so I'm unable to help any further.
Best wishes from Riley.
---
* Nothing as pretty as a smile, nothing as ugly as a frown.
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.484 / Virus Database: 282 - Release Date: 27-May-2003
Hi !
Quoted 2 lines · Marc Wilson
No, the basic problem there is that the kernel is deadlocking. Read the VERY long thread for the details.
I didn't follow this thread, what's its subject, please ?
Quoted 4 lines
I think I have enough on the ball to be able to tell the difference between mutt opening a folder and counting messages, with a counter and percentage indicator advancing, and mutt sitting there deadlocked with the HD activity light stuck on and all the rest of X stuck tight.
even on -rc3, I don't observe this behaviour. I tried from a cold cache, and mutt took a little less than 3 seconds to open LKML's May folder (35 MB), and progressed very smoothly. Since it's on my Alpha file server, I can't test with X. But the I/O bandwidth and scheduler frequency (1024 HZ) may have an impact.
Cheers,
Willy
Quoted 7 lines · Willy Tarreau
Hi !
Quoted 2 lines · Marc Wilson
No, the basic problem there is that the kernel is deadlocking. Read the VERY long thread for the details.
I didn't follow this thread, what's its subject, please ?
Hmmm never mind, I easily found it (yes, VERY long) !
Cheers,
Willy
Quoted 9 lines · Marc Wilson
Quoted 4 lines · Riley Williams
The basic problem there is that any mail client needs to know just how many messages are in a particular folder to handle that folder, and the only way to do this is to count them all. That's what takes the time when one opens a large folder.
No, the basic problem there is that the kernel is deadlocking. Read the VERY long thread for the details.
Quoted 8 lines
I think I have enough on the ball to be able to tell the difference between mutt opening a folder and counting messages, with a counter and percentage indicator advancing, and mutt sitting there deadlocked with the HD activity light stuck on and all the rest of X stuck tight.
And it just happened again, so -rc6 is no sure fix. What did y'all that reported the problem had gone away do, patch -rc4 with the akpm patches? ^_^
Ok, so you can reproduce the hangs reliably EVEN with -rc6, Marc?
On Tuesday 03 June 2003 18:02, Marcelo Tosatti wrote:
Hi Marcelo,
Quoted 1 line
Ok, so you can reproduce the hangs reliably EVEN with -rc6, Marc?
well, even if you mean Marc Wilson, I also have to say something (as I've written in my previous email some days ago)
The pauses/stops are _a lot_ less than w/o the fix but they are _not_ gone. Tested with 2.4.21-rc6.
ciao, Marc
Hi!
Quoted 6 lines
Quoted 1 line
Ok, so you can reproduce the hangs reliably EVEN with -rc6, Marc?
well, even if you mean Marc Wilson, I also have to say something (as I've written in my previous email some days ago)
The pauses/stops are _a
lot_less than w/o the fix but they are _not_ gone. Tested with 2.4.21-rc6.
If hangs are not worse than 2.4.20, then I'd go ahead with release....
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Quoted 4 lines · Pavel Machek
Hi!
Quoted 2 lines
Quoted 1 line
Ok, so you can reproduce the hangs reliably EVEN with -rc6, Marc?
Quoted 9 lines
Quoted 5 lines
well, even if you mean Marc Wilson, I also have to say something (as I've written in my previous email some days ago)
The pauses/stops are _a
lot_less than w/o the fix but they are _not_ gone. Tested with 2.4.21-rc6.If hangs are not worse than 2.4.20, then I'd go ahead with release....
I have -rc6 running on a P4 for a few days, doing the test script, compiles, Opera and found it to be comparable to 2.4.18.
It also does well on slower machines of about 1/4 the the CPU and disk bandwidth.
IMHO, interactivity is reasonable (again just IMHO), and others may disagree.
Signature · 9 lines
Powered by linux-2.5.70-mm3 My current linux related activities in rough order of priority: - Testing of 2.4/2.5 kernel interactivity - Testing of Swsusp for 2.4 - Testing of Opera 7.11 emphasizing interactivity - Research of NFS i/o errors during transfer 2.4>2.5 - Learning 2.5 series kernel debugging with kgdb - it's in the -mm tree - Studying 2.5 series serial and ide drivers, ACPI, S3 * Input and feedback is always welcome *
Quoted 15 lines · Marcelo Tosatti
Quoted 14 lines · Marc Wilson
Quoted 4 lines · Riley Williams
The basic problem there is that any mail client needs to know just how many messages are in a particular folder to handle that folder, and the only way to do this is to count them all. That's what takes the time when one opens a large folder.
No, the basic problem there is that the kernel is deadlocking. Read the VERY long thread for the details.
I think I have enough on the ball to be able to tell the difference between mutt opening a folder and counting messages, with a counter and percentage indicator advancing, and mutt sitting there deadlocked with the HD activity light stuck on and all the rest of X stuck tight.
Quoted 5 lines
Quoted 3 lines
And it just happened again, so -rc6 is no sure fix. What did y'all that reported the problem had gone away do, patch -rc4 with the akpm patches? ^_^
Ok, so you can reproduce the hangs reliably EVEN with -rc6, Marc?
-rc6 is better - comparable to 2.4.18 in what I have seen with my script.
After the long obscure problems since 2.4.19x, -rc6 could use serious stress-testing.
User level testing is not sufficient here - it's just like playing roulette.
By serious stress-testing I mean:
Everone testing comes up with one dedicated "tough test" which _must_ be reproducible (program, script) along his line of expertise/application.
Two or more of these independent tests are run in combination.
This method should increase the coverage drastically.
Regards
Michael
Quoted 8 lines · Michael Frank
On Wednesday 04 June 2003 00:02, Marcelo Tosatti wrote:
-rc6 is better - comparable to 2.4.18 in what I have seen with my script.After the long obscure problems since 2.4.19x, -rc6 could use serious stress-testing.
User level testing is not sufficient here - it's just like playing roulette.
Quoted 7 lines
By serious stress-testing I mean:
Everone testing comes up with one dedicated "tough test" which _must_ be reproducible (program, script) along his line of expertise/application.
Two or more of these independent tests are run in combination.
Agreed and I'm willing to run test-scripts on my system, that has these hangs (long ones with 2.4.19-pre1 to 2.4.21-rc5 and only short ones with 2.4.21-rc6). But at the moment I have neither time nor enough knowledge to write a test to reproduce it.
So if someone comes up with a suitable test skript, I'm happy to try it and use it on different kernel versions.
Bye,
Matthias
Signature · 3 lines
[email protected] Rechenzentrum Universitaet Karlsruhe Abteilung Netze
On Tuesday 03 June 2003 18:30, Michael Frank wrote:
Hi Michael,
Quoted 10 lines
Quoted 1 line
Ok, so you can reproduce the hangs reliably EVEN with -rc6, Marc?
-rc6 is better - comparable to 2.4.18 in what I have seen with my script. After the long obscure problems since 2.4.19x, -rc6 could use serious stress-testing.
User level testing is not sufficient here - it's just like playing roulette.
By serious stress-testing I mean:
Everone testing comes up with one dedicated "tough test" which _must_ be reproducible (program, script) along his line of expertise/application.
well, very easy one:
dd if=/dev/zero of=/home/largefile bs=16384 count=131072
then use your mouse, your apps, switch between them, use them, _w/o_ pauses, delay, stops or kinda that. If _that_ will work flawlessly for everyone, then it is fixed, if not, it _needs_ to be fixed.
ciao, Marc
On Tuesday 03 June 2003 18:59, Marc-Christian Petersen wrote:
Hi again,
Quoted 5 lines
well, very easy one:
dd if=/dev/zero of=/home/largefile bs=16384 count=131072 then use your mouse, your apps, switch between them, use them, _w/o_ pauses, delay, stops or kinda that. If _that_ will work flawlessly for everyone, then it is fixed, if not, it _needs_ to be fixed.
I forgot to mention. If you have more than 2GB free memory (the above one will create a 2GB file), the test is useless.
Have less memory free, so the machine will swap, doesn't matter if the same disk or another or whatever!
ciao, Marc
Good Evening,
Quoted 9 lines · Marc-Christian Petersen
On Tuesday 03 June 2003 18:59, Marc-Christian Petersen wrote:
Hi again,
Quoted 5 lines
well, very easy one:
dd if=/dev/zero of=/home/largefile bs=16384 count=131072 then use your mouse, your apps, switch between them, use them, _w/o_ pauses, delay, stops or kinda that. If _that_ will work flawlessly for everyone, then it is fixed, if not, it _needs_ to be fixed.
Quoted 5 lines
I forgot to mention. If you have more than 2GB free memory (the above one will create a 2GB file), the test is useless.
Have less memory free, so the machine will swap, doesn't matter if the same disk or another or whatever!
Would it count if I said I run 2.4.21-rc6-ac1 and had 768MB RAM, ended up using about 250MB swap and when I today suspended VMware and closed a few gnome-terminals, Galeon and Evolution, the mouse cursor would not move, then jump half way across the screen after a second, then 'stick' again before doing another jump.
I thought it sounded a little like what you are describing. If more details are required, let me know and I will try and collect what is asked for.
Regards,
Signature · 2 lines
Anders Karlsson <[email protected]> Trudheim Technology Limited
Quoted 15 lines · Anders Karlsson
Good Evening,
Quoted 12 lines · Marc-Christian Petersen
On Tuesday 03 June 2003 18:59, Marc-Christian Petersen wrote:
Hi again,
Quoted 5 lines
well, very easy one:
dd if=/dev/zero of=/home/largefile bs=16384 count=131072 then use your mouse, your apps, switch between them, use them, _w/o_ pauses, delay, stops or kinda that. If _that_ will work flawlessly for everyone, then it is fixed, if not, it _needs_ to be fixed.I forgot to mention. If you have more than 2GB free memory (the above one will create a 2GB file), the test is useless.
Quoted 9 lines
Quoted 2 lines
Have less memory free, so the machine will swap, doesn't matter if the same disk or another or whatever!
Would it count if I said I run 2.4.21-rc6-ac1 and had 768MB RAM, ended up using about 250MB swap and when I today suspended VMware and closed a few gnome-terminals, Galeon and Evolution, the mouse cursor would not move, then jump half way across the screen after a second, then 'stick' again before doing another jump.
One vote in the opposite sense (I know, nobody uses plain rc6 ???) I am using a -jam kernel (-aa with some additional patches), and I did not notice anything. Dual PII box with 900 Mb, as buffers were filling memory, no stalls. Just a very small (less than half a second) jump in the cursor under gnome when the memory got full, and then smooth again. I use pointer-focus and was rapidly moving the pointer from window to window to change focus and response was ok. Launching an aterm was instant.
Signature · 4 lines
J.A. Magallon <[email protected]> \ Software is like sex: werewolf.able.es \ It's better when it's free Mandrake Linux release 9.2 (Cooker) for i586 Linux 2.4.21-rc6-jam1 (gcc 3.2.3 (Mandrake Linux 9.2 3.2.3-1mdk))
On Tuesday 03 June 2003 23:12, J.A. Magallon wrote:
Hi J.A.,
Quoted 7 lines
One vote in the opposite sense (I know, nobody uses plain rc6 ???) I am using a -jam kernel (-aa with some additional patches), and I did not notice anything. Dual PII box with 900 Mb, as buffers were filling memory, no stalls. Just a very small (less than half a second) jump in the cursor under gnome when the memory got full, and then smooth again. I use pointer-focus and was rapidly moving the pointer from window to window to change focus and response was ok. Launching an aterm was instant.
once again for you ;-)
-aa is using low latency elevator! Pauses/Stops are more less with it.
ciao, Marc
Quoted 4 lines · Marc-Christian Petersen
On Tuesday 03 June 2003 18:30, Michael Frank wrote:
well, very easy one:dd if=/dev/zero of=/home/largefile bs=16384 count=131072
Got that already - more flexible:
http://www.ussg.iu.edu/hypermail/linux/kernel/0305.3/1291.html
Breaks anything >= 2.4.19 < rc6 in no time.
We need more - any ideas
Reagards
Michael
On Wed, Jun 04, 2003 at 12:30:27AM +0800, Michael Frank wrote:
...
Quoted 4 lines
Quoted 2 lines
Ok, so you can reproduce the hangs reliably EVEN with -rc6, Marc?
-rc6 is better - comparable to 2.4.18 in what I have seen with my script.
I've run 2.4.20 for a long time, and have been seriously plagued with the I/O stalls.
On a file server (details below) here I upgraded to 2.4.21-rc6 yesterday.
The I/O stalls have *almost* gone away.
Best of all, we still have our data intact ;)
Server data:
~130 GB data on a ~150 GB ext3fs with >1 million files Software RAID-0+1 on four IDE disks
Two promise controllers 1x20262 1x20269
1x Intel eepro100, 1x Intel e1000
dual PIII, half a gig of memory
NFS server (mainly v3, many different clients)
Quoted 3 lines
After the long obscure problems since 2.4.19x, -rc6 could use serious stress-testing.
This server rarely has load below 1, but frequently above 15. It may run some compilers and linkers locally, but most of the load comes from NFS serving.
So far it's been running for 28 hours with that kind of load. Nothing suspicious in the dmesg yet.
I will of course let you all know if it falls on it's knees.
So far it's all thumbs-up from me! There may still be an occational
stall here and there, but compared to 2.4.20 this is heaven (it really was unbelievably annoying having your emacs stall for 10 seconds every 30 seonds when someone was linking on the cluster) :)
A big *thank*you* to Marcelo for deciding to include a fix for the I/O stalls!
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............:
Quoted 1 line · Marcelo Tosatti
Ok, so you can reproduce the hangs reliably EVEN with -rc6, Marc?
Yes, with -rc6, and this:
rei $ dd if=/dev/zero of=/home/mwilson/largefile bs=16384 count=131072
The mouse starts skipping soon after the box starts swapping. It eventually catches up, but then when I start up another application, it starts again.
I have the test running as I type this e-mail in mutt (with vim as the editor), and there are noticeable pauses where I'm typing, but there isn't anything happening on the screen.
It's *much* better than it was with my prior kernel (-rc2), but it's most definately still there.
Anyone got any other test they want me to make on the box?
Signature · 2 lines
Marc Wilson | You're a card which will have to be dealt with. [email protected] |
Quoted 11 lines · Marcelo Tosatti
Hi,
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
Please test it.
Andrew Morton <[email protected]>:
o Fix IO stalls and deadlocks
As this is only patches 1 and 2 from akpm's suggested changes I was wondering if my report got lost in the huge thread so I've included it here:
Ok patch combination final score for me is as follows in the presence of a large continuous write:
1 No change
2 No change
3 improvement++; minor hangs with reads
1+2 improvement+++; minor pauses with switching applications 1+2+3 improvement++++; no pauses
Applications may start up slowly that's fine. The mouse cursor keeps spinning and responding at all times though with 1+2+3 which it hasn't done in 2.4 for a year or so.
Is there a reason the 3rd patch was omitted?
Con
Quoted 2 lines · On Wed, 28 May 2003 21:55:39 -0300 (BRT) Marcelo Tosatti…
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
While others may be dubious about the efficacy of this patch, I've been running -rc6 on my laptop now since sometime last night and have seen nothing odd.
In case anybody cares, I'm using both ide and a ieee1394 (for a large external drive [which implies scsi]) and I do a _lot_ of big work with BK so I was seeing the problem within hours previously.
-g
-rc7 Re: Linux 2.4.21-rc6
Quoted 6 lines · Georg Nikodym
Quoted 2 lines · On Wed, 28 May 2003 21:55:39 -0300 (BRT) Marcelo Tosatti…
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
Quoted 7 lines
While others may be dubious about the efficacy of this patch, I've been running -rc6 on my laptop now since sometime last night and have seen nothing odd.
In case anybody cares, I'm using both ide and a ieee1394 (for a large external drive [which implies scsi]) and I do a _lot_ of big work with BK so I was seeing the problem within hours previously.
Great!
-rc7 will have to be released due to some problems :(
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 11 lines · Marcelo Tosatti
Quoted 10 lines · Georg Nikodym
On Wed, 28 May 2003 21:55:39 -0300 (BRT)
Quoted 2 lines · Marcelo Tosatti
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
While others may be dubious about the efficacy of this patch, I've been running -rc6 on my laptop now since sometime last night and have seen nothing odd.
Quoted 7 lines
Quoted 3 lines
In case anybody cares, I'm using both ide and a ieee1394 (for a large external drive [which implies scsi]) and I do a _lot_ of big work with BK so I was seeing the problem within hours previously.
Great!
-rc7 will have to be released due to some problems :(
hmm, seems to ide modules and others are broken. Im looking for reason why .. here are depmod errors and my .config file:
make[1]: Nie nic do roboty w `modules_install'.
make[1]: Opuszczam katalog `/home/users/dzimi/rpm/BUILD/linux-2.4.20/arch/i386/l
ib'
cd /lib/modules/2.4.21-rc6; \
mkdir -p pcmcia; \
find kernel -path '*/pcmcia/*' -name '*.o' | xargs -i -r ln -sf ../{} pcmcia if [ -r System.map ]; then /sbin/depmod -ae -F System.map 2.4.21-rc6; fi
depmod: *** Unresolved symbols in /lib/modules/2.4.21-rc6/kernel/drivers/ide/ide
-disk.o
depmod: proc_ide_read_geometry depmod: ide_remove_proc_entries depmod: *** Unresolved symbols in /lib/modules/2.4.21-rc6/kernel/drivers/ide/ide -floppy.o depmod: proc_ide_read_geometry depmod: ide_remove_proc_entries depmod: *** Unresolved symbols in /lib/modules/2.4.21-rc6/kernel/drivers/ide/ide -probe.o depmod: do_ide_request depmod: ide_add_generic_settings depmod: create_proc_ide_interfaces depmod: *** Unresolved symbols in
/lib/modules/2.4.21-rc6/kernel/drivers/ide/ide
-tape.o
depmod: ide_remove_proc_entries
depmod: *** Unresolved symbols in/lib/modules/2.4.21-rc6/kernel/drivers/ide/ide
.o
depmod: ide_release_dma depmod: ide_add_proc_entries depmod: cmd640_vlb depmod: ide_probe_for_cmd640x depmod: ide_scan_pcibus depmod: proc_ide_read_capacity depmod: proc_ide_create depmod: ide_remove_proc_entries depmod: destroy_proc_ide_drives depmod: proc_ide_destroy depmod: create_proc_ide_interfaces depmod: *** Unresolved symbols in
/lib/modules/2.4.21-rc6/kernel/drivers/net/wan
/comx.o
depmod: proc_get_inode
depmod: *** Unresolved symbols in/lib/modules/2.4.21-rc6/kernel/net/atm/common.
o
depmod: free_atm_vcc_sk depmod: atm_init_aal34 depmod: alloc_atm_vcc_sk depmod: atm_init_aal0 depmod: atm_devs depmod: *** Unresolved symbols in /lib/modules/2.4.21-rc6/kernel/net/atm/pvc.o depmod: atm_getsockopt depmod: atm_recvmsg depmod: atm_release depmod: atm_ioctl depmod: atm_create depmod: atm_sendmsg depmod: atm_poll depmod: atm_connect depmod: atm_proc_init depmod: atm_setsockopt depmod: *** Unresolved symbols in /lib/modules/2.4.21-rc6/kernel/net/atm/resourc es.o depmod: atm_proc_dev_deregister depmod: atm_proc_dev_register depmod: *** Unresolved symbols in /lib/modules/2.4.21-rc6/kernel/net/atm/signali ng.o depmod: nodev_vccs depmod: atm_devs depmod: *** Unresolved symbols in /lib/modules/2.4.21-rc6/kernel/net/atm/svc.o depmod: atm_getsockopt depmod: atm_recvmsg depmod: free_atm_vcc_sk depmod: atm_ioctl depmod: atm_create depmod: atm_sendmsg depmod: atm_poll depmod: atm_connect depmod: atm_release_vcc_sk depmod: atm_setsockopt
my .config (it's distro config, im PLD kernel packager) is include. Ok im going to fix those trivial (?) problems, when i worked around 2.2.x i made some hacks on ksyms.c, was it corect ?
Signature · 2 lines
Krzysiek Taraszka ([email protected]) http://cyborg.kernel.pl/~dzimi/
#
# Automatically generated by make menuconfig: don't edit #CONFIG_X86=y
# CONFIG_SBUS is not setCONFIG_UID16=y
#
# Code maturity level options
#CONFIG_EXPERIMENTAL=y
#
# Loadable module support
#CONFIG_MODULES=y
# CONFIG_MODVERSIONS is not setCONFIG_KMOD=y
#
# Processor type and features
#
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not setCONFIG_MK7=y
# CONFIG_MK8 is not set
# CONFIG_MELAN is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not setCONFIG_X86_WP_WORKS_OK=yCONFIG_X86_INVLPG=yCONFIG_X86_CMPXCHG=yCONFIG_X86_XADD=yCONFIG_X86_BSWAP=yCONFIG_X86_POPAD_OK=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not setCONFIG_RWSEM_XCHGADD_ALGORITHM=yCONFIG_X86_L1_CACHE_SHIFT=6CONFIG_X86_HAS_TSC=yCONFIG_X86_GOOD_APIC=yCONFIG_X86_USE_3DNOW=yCONFIG_X86_PGE=yCONFIG_X86_USE_PPRO_CHECKSUM=yCONFIG_X86_F00F_WORKS_OK=yCONFIG_X86_MCE=yCONFIG_TOSHIBA=mCONFIG_I8K=mCONFIG_MICROCODE=mCONFIG_X86_MSR=mCONFIG_X86_CPUID=m
# CONFIG_NOHIGHMEM is not setCONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not setCONFIG_HIGHMEM=yCONFIG_HIGHIO=y
# CONFIG_MATH_EMULATION is not setCONFIG_MTRR=y
# CONFIG_SMP is not setCONFIG_X86_UP_APIC=yCONFIG_X86_UP_IOAPIC=yCONFIG_X86_LOCAL_APIC=yCONFIG_X86_IO_APIC=y
# CONFIG_X86_TSC_DISABLE is not setCONFIG_X86_TSC=y
#
# General setup
#CONFIG_NET=yCONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GODIRECT is not setCONFIG_PCI_GOANY=yCONFIG_PCI_BIOS=yCONFIG_PCI_DIRECT=yCONFIG_ISA=yCONFIG_PCI_NAMES=yCONFIG_EISA=yCONFIG_MCA=yCONFIG_HOTPLUG=y
#
# PCMCIA/CardBus support
#CONFIG_PCMCIA=mCONFIG_CARDBUS=yCONFIG_TCIC=yCONFIG_I82092=yCONFIG_I82365=y
#
# PCI Hotplug Support
#CONFIG_HOTPLUG_PCI=mCONFIG_HOTPLUG_PCI_COMPAQ=m
# CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM is not setCONFIG_HOTPLUG_PCI_IBM=mCONFIG_HOTPLUG_PCI_ACPI=mCONFIG_SYSVIPC=yCONFIG_BSD_PROCESS_ACCT=yCONFIG_SYSCTL=yCONFIG_KCORE_ELF=y
# CONFIG_KCORE_AOUT is not setCONFIG_BINFMT_AOUT=mCONFIG_BINFMT_ELF=yCONFIG_BINFMT_MISC=mCONFIG_PM=yCONFIG_ACPI=y
# CONFIG_ACPI_DEBUG is not setCONFIG_ACPI_BUSMGR=mCONFIG_ACPI_SYS=mCONFIG_ACPI_CPU=mCONFIG_ACPI_BUTTON=mCONFIG_ACPI_AC=mCONFIG_ACPI_EC=mCONFIG_ACPI_CMBATT=mCONFIG_ACPI_THERMAL=mCONFIG_APM=m
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
# CONFIG_APM_DO_ENABLE is not set
# CONFIG_APM_CPU_IDLE is not set
# CONFIG_APM_DISPLAY_BLANK is not setCONFIG_APM_RTC_IS_GMT=y
# CONFIG_APM_ALLOW_INTS is not setCONFIG_APM_REAL_MODE_POWER_OFF=y
#
# Memory Technology Devices (MTD)
#CONFIG_MTD=m
# CONFIG_MTD_DEBUG is not setCONFIG_MTD_PARTITIONS=mCONFIG_MTD_CONCAT=mCONFIG_MTD_REDBOOT_PARTS=m
# CONFIG_MTD_CMDLINE_PARTS is not setCONFIG_MTD_CHAR=mCONFIG_MTD_BLOCK=mCONFIG_MTD_BLOCK_RO=mCONFIG_FTL=mCONFIG_NFTL=mCONFIG_NFTL_RW=y
#
# RAM/ROM/Flash chip drivers
#CONFIG_MTD_CFI=mCONFIG_MTD_JEDECPROBE=mCONFIG_MTD_GEN_PROBE=mCONFIG_MTD_CFI_ADV_OPTIONS=yCONFIG_MTD_CFI_NOSWAP=y
# CONFIG_MTD_CFI_BE_BYTE_SWAP is not set
# CONFIG_MTD_CFI_LE_BYTE_SWAP is not set
# CONFIG_MTD_CFI_GEOMETRY is not setCONFIG_MTD_CFI_INTELEXT=mCONFIG_MTD_CFI_AMDSTD=m
# CONFIG_MTD_CFI_STAA is not setCONFIG_MTD_RAM=mCONFIG_MTD_ROM=mCONFIG_MTD_ABSENT=m
# CONFIG_MTD_OBSOLETE_CHIPS is not set
# CONFIG_MTD_AMDSTD is not set
# CONFIG_MTD_SHARP is not set
# CONFIG_MTD_JEDEC is not set
#
# Mapping drivers for chip access
#CONFIG_MTD_PHYSMAP=mCONFIG_MTD_PHYSMAP_START=8000000CONFIG_MTD_PHYSMAP_LEN=4000000CONFIG_MTD_PHYSMAP_BUSWIDTH=2CONFIG_MTD_PNC2000=mCONFIG_MTD_SC520CDP=mCONFIG_MTD_NETSC520=mCONFIG_MTD_SBC_GXX=mCONFIG_MTD_ELAN_104NC=mCONFIG_MTD_DILNETPC=mCONFIG_MTD_DILNETPC_BOOTSIZE=80000
# CONFIG_MTD_MIXMEM is not set
# CONFIG_MTD_OCTAGON is not set
# CONFIG_MTD_VMAX is not set
# CONFIG_MTD_SCx200_DOCFLASH is not setCONFIG_MTD_L440GX=m
# CONFIG_MTD_AMD76XROM is not setCONFIG_MTD_ICH2ROM=m
# CONFIG_MTD_NETtel is not set
# CONFIG_MTD_SCB2_FLASH is not setCONFIG_MTD_PCI=m
# CONFIG_MTD_PCMCIA is not set
#
# Self-contained MTD device drivers
#CONFIG_MTD_PMC551=mCONFIG_MTD_PMC551_BUGFIX=y
# CONFIG_MTD_PMC551_DEBUG is not setCONFIG_MTD_SLRAM=mCONFIG_MTD_MTDRAM=mCONFIG_MTDRAM_TOTAL_SIZE=4096CONFIG_MTDRAM_ERASE_SIZE=128CONFIG_MTD_BLKMTD=mCONFIG_MTD_DOC1000=mCONFIG_MTD_DOC2000=mCONFIG_MTD_DOC2001=mCONFIG_MTD_DOCPROBE=mCONFIG_MTD_DOCPROBE_ADVANCED=yCONFIG_MTD_DOCPROBE_ADDRESS=0000CONFIG_MTD_DOCPROBE_HIGH=yCONFIG_MTD_DOCPROBE_55AA=y
#
# NAND Flash Device Drivers
#CONFIG_MTD_NAND=mCONFIG_MTD_NAND_VERIFY_WRITE=yCONFIG_MTD_NAND_IDS=m
#
# Parallel port support
#CONFIG_PARPORT=mCONFIG_PARPORT_PC=mCONFIG_PARPORT_PC_CML1=mCONFIG_PARPORT_SERIAL=mCONFIG_PARPORT_PC_FIFO=yCONFIG_PARPORT_PC_SUPERIO=yCONFIG_PARPORT_PC_PCMCIA=m
# CONFIG_PARPORT_AMIGA is not set
# CONFIG_PARPORT_MFC3 is not set
# CONFIG_PARPORT_ATARI is not set
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_SUNBPP is not set
# CONFIG_PARPORT_OTHER is not setCONFIG_PARPORT_1284=y
#
# Plug and Play configuration
#CONFIG_PNP=mCONFIG_ISAPNP=m
#
# Block devices
#CONFIG_BLK_DEV_FD=mCONFIG_BLK_DEV_PS2=mCONFIG_BLK_DEV_XD=mCONFIG_PARIDE=mCONFIG_PARIDE_PARPORT=mCONFIG_PARIDE_PD=mCONFIG_PARIDE_PCD=mCONFIG_PARIDE_PF=mCONFIG_PARIDE_PT=mCONFIG_PARIDE_PG=mCONFIG_PARIDE_ATEN=mCONFIG_PARIDE_BPCK=mCONFIG_PARIDE_BPCK6=mCONFIG_PARIDE_COMM=mCONFIG_PARIDE_DSTR=mCONFIG_PARIDE_FIT2=mCONFIG_PARIDE_FIT3=mCONFIG_PARIDE_EPAT=mCONFIG_PARIDE_EPATC8=yCONFIG_PARIDE_EPIA=mCONFIG_PARIDE_FRIQ=mCONFIG_PARIDE_FRPW=mCONFIG_PARIDE_KBIC=mCONFIG_PARIDE_KTTI=mCONFIG_PARIDE_ON20=mCONFIG_PARIDE_ON26=mCONFIG_BLK_CPQ_DA=mCONFIG_BLK_CPQ_CISS_DA=mCONFIG_CISS_SCSI_TAPE=yCONFIG_BLK_DEV_DAC960=mCONFIG_BLK_DEV_UMEM=mCONFIG_BLK_DEV_LOOP=mCONFIG_BLK_DEV_NBD=mCONFIG_BLK_DEV_RAM=yCONFIG_BLK_DEV_RAM_SIZE=4096CONFIG_BLK_DEV_INITRD=yCONFIG_BLK_STATS=y
#
# Multi-device support (RAID and LVM)
#CONFIG_MD=yCONFIG_BLK_DEV_MD=mCONFIG_MD_LINEAR=mCONFIG_MD_RAID0=mCONFIG_MD_RAID1=mCONFIG_MD_RAID5=mCONFIG_MD_MULTIPATH=mCONFIG_BLK_DEV_LVM=m
#
# Networking options
#CONFIG_PACKET=mCONFIG_PACKET_MMAP=yCONFIG_NETLINK_DEV=yCONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not setCONFIG_FILTER=yCONFIG_UNIX=mCONFIG_INET=yCONFIG_IP_MULTICAST=yCONFIG_IP_ADVANCED_ROUTER=yCONFIG_IP_MULTIPLE_TABLES=yCONFIG_IP_ROUTE_FWMARK=yCONFIG_IP_ROUTE_NAT=yCONFIG_IP_ROUTE_MULTIPATH=yCONFIG_IP_ROUTE_TOS=yCONFIG_IP_ROUTE_VERBOSE=yCONFIG_IP_ROUTE_LARGE_TABLES=y
# CONFIG_IP_PNP is not setCONFIG_NET_IPIP=mCONFIG_NET_IPGRE=mCONFIG_NET_IPGRE_BROADCAST=yCONFIG_IP_MROUTE=yCONFIG_IP_PIMSM_V1=yCONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
# CONFIG_INET_ECN is not setCONFIG_SYN_COOKIES=y
#
# IP: Netfilter Configuration
#CONFIG_IP_NF_CONNTRACK=mCONFIG_IP_NF_FTP=m
# CONFIG_IP_NF_AMANDA is not set
# CONFIG_IP_NF_TFTP is not setCONFIG_IP_NF_IRC=mCONFIG_IP_NF_QUEUE=mCONFIG_IP_NF_IPTABLES=mCONFIG_IP_NF_MATCH_LIMIT=mCONFIG_IP_NF_MATCH_MAC=mCONFIG_IP_NF_MATCH_PKTTYPE=mCONFIG_IP_NF_MATCH_MARK=mCONFIG_IP_NF_MATCH_MULTIPORT=mCONFIG_IP_NF_MATCH_TOS=mCONFIG_IP_NF_MATCH_ECN=mCONFIG_IP_NF_MATCH_DSCP=mCONFIG_IP_NF_MATCH_AH_ESP=mCONFIG_IP_NF_MATCH_LENGTH=mCONFIG_IP_NF_MATCH_TTL=mCONFIG_IP_NF_MATCH_TCPMSS=mCONFIG_IP_NF_MATCH_HELPER=mCONFIG_IP_NF_MATCH_STATE=mCONFIG_IP_NF_MATCH_CONNTRACK=mCONFIG_IP_NF_MATCH_UNCLEAN=mCONFIG_IP_NF_MATCH_OWNER=mCONFIG_IP_NF_FILTER=mCONFIG_IP_NF_TARGET_REJECT=mCONFIG_IP_NF_TARGET_MIRROR=mCONFIG_IP_NF_NAT=mCONFIG_IP_NF_NAT_NEEDED=yCONFIG_IP_NF_TARGET_MASQUERADE=mCONFIG_IP_NF_TARGET_REDIRECT=mCONFIG_IP_NF_NAT_LOCAL=yCONFIG_IP_NF_NAT_SNMP_BASIC=mCONFIG_IP_NF_NAT_IRC=mCONFIG_IP_NF_NAT_FTP=mCONFIG_IP_NF_MANGLE=mCONFIG_IP_NF_TARGET_TOS=mCONFIG_IP_NF_TARGET_ECN=mCONFIG_IP_NF_TARGET_DSCP=mCONFIG_IP_NF_TARGET_MARK=mCONFIG_IP_NF_TARGET_LOG=mCONFIG_IP_NF_TARGET_ULOG=mCONFIG_IP_NF_TARGET_TCPMSS=mCONFIG_IP_NF_ARPTABLES=mCONFIG_IP_NF_ARPFILTER=mCONFIG_IP_NF_COMPAT_IPCHAINS=mCONFIG_IP_NF_NAT_NEEDED=yCONFIG_IP_NF_COMPAT_IPFWADM=mCONFIG_IP_NF_NAT_NEEDED=yCONFIG_IPV6=m
#
# IPv6: Netfilter Configuration
#CONFIG_IP6_NF_QUEUE=mCONFIG_IP6_NF_IPTABLES=mCONFIG_IP6_NF_MATCH_LIMIT=mCONFIG_IP6_NF_MATCH_MAC=m
# CONFIG_IP6_NF_MATCH_RT is not set
# CONFIG_IP6_NF_MATCH_OPTS is not set
# CONFIG_IP6_NF_MATCH_FRAG is not set
# CONFIG_IP6_NF_MATCH_HL is not setCONFIG_IP6_NF_MATCH_MULTIPORT=mCONFIG_IP6_NF_MATCH_OWNER=mCONFIG_IP6_NF_MATCH_MARK=m
# CONFIG_IP6_NF_MATCH_IPV6HEADER is not set
# CONFIG_IP6_NF_MATCH_AHESP is not setCONFIG_IP6_NF_MATCH_LENGTH=mCONFIG_IP6_NF_MATCH_EUI64=mCONFIG_IP6_NF_FILTER=mCONFIG_IP6_NF_TARGET_LOG=mCONFIG_IP6_NF_MANGLE=mCONFIG_IP6_NF_TARGET_MARK=mCONFIG_KHTTPD=mCONFIG_ATM=mCONFIG_VLAN_8021Q=mCONFIG_IPX=mCONFIG_IPX_INTERN=yCONFIG_ATALK=m
#
# Appletalk devices
#CONFIG_DEV_APPLETALK=yCONFIG_LTPC=mCONFIG_COPS=mCONFIG_COPS_DAYNA=yCONFIG_COPS_TANGENT=yCONFIG_IPDDP=mCONFIG_IPDDP_ENCAP=yCONFIG_IPDDP_DECAP=yCONFIG_DECNET=mCONFIG_DECNET_SIOCGIFCONF=yCONFIG_DECNET_ROUTER=y
# CONFIG_DECNET_ROUTE_FWMARK is not setCONFIG_BRIDGE=mCONFIG_X25=mCONFIG_LAPB=mCONFIG_LLC=yCONFIG_NET_DIVERT=yCONFIG_ECONET=mCONFIG_ECONET_AUNUDP=yCONFIG_ECONET_NATIVE=yCONFIG_WAN_ROUTER=m
# CONFIG_NET_FASTROUTE is not set
# CONFIG_NET_HW_FLOWCONTROL is not set
#
# QoS and/or fair queueing
#CONFIG_NET_SCHED=yCONFIG_NET_SCH_CBQ=mCONFIG_NET_SCH_HTB=mCONFIG_NET_SCH_CSZ=mCONFIG_NET_SCH_PRIO=mCONFIG_NET_SCH_RED=mCONFIG_NET_SCH_SFQ=mCONFIG_NET_SCH_TEQL=mCONFIG_NET_SCH_TBF=mCONFIG_NET_SCH_GRED=mCONFIG_NET_SCH_DSMARK=mCONFIG_NET_SCH_INGRESS=mCONFIG_NET_QOS=yCONFIG_NET_ESTIMATOR=yCONFIG_NET_CLS=yCONFIG_NET_CLS_TCINDEX=mCONFIG_NET_CLS_ROUTE4=mCONFIG_NET_CLS_ROUTE=yCONFIG_NET_CLS_FW=mCONFIG_NET_CLS_U32=mCONFIG_NET_CLS_RSVP=mCONFIG_NET_CLS_RSVP6=mCONFIG_NET_CLS_POLICE=y
#
# Network testing
#CONFIG_NET_PKTGEN=m
#
# Telephony Support
#CONFIG_PHONE=mCONFIG_PHONE_IXJ=mCONFIG_PHONE_IXJ_PCMCIA=m
#
# ATA/IDE/MFM/RLL support
#CONFIG_IDE=m
#
# IDE, ATA and ATAPI Block devices
#CONFIG_BLK_DEV_IDE=m
# CONFIG_BLK_DEV_HD_IDE is not set
# CONFIG_BLK_DEV_HD is not setCONFIG_BLK_DEV_IDEDISK=m
# CONFIG_IDEDISK_MULTI_MODE is not setCONFIG_IDEDISK_STROKE=yCONFIG_BLK_DEV_IDECS=mCONFIG_BLK_DEV_IDECD=mCONFIG_BLK_DEV_IDETAPE=mCONFIG_BLK_DEV_IDEFLOPPY=mCONFIG_BLK_DEV_IDESCSI=mCONFIG_IDE_TASK_IOCTL=yCONFIG_BLK_DEV_CMD640=y
# CONFIG_BLK_DEV_CMD640_ENHANCED is not set
# CONFIG_BLK_DEV_ISAPNP is not setCONFIG_BLK_DEV_IDEPCI=y
# CONFIG_BLK_DEV_GENERIC is not setCONFIG_IDEPCI_SHARE_IRQ=yCONFIG_BLK_DEV_IDEDMA_PCI=yCONFIG_BLK_DEV_OFFBOARD=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
# CONFIG_IDEDMA_PCI_AUTO is not set
# CONFIG_IDEDMA_ONLYDISK is not setCONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_PCI_WIP is not set
# CONFIG_BLK_DEV_ADMA100 is not setCONFIG_BLK_DEV_AEC62XX=yCONFIG_BLK_DEV_ALI15X3=y
# CONFIG_WDC_ALI15X3 is not setCONFIG_BLK_DEV_AMD74XX=y
# CONFIG_AMD74XX_OVERRIDE is not setCONFIG_BLK_DEV_CMD64X=y
# CONFIG_BLK_DEV_TRIFLEX is not setCONFIG_BLK_DEV_CY82C693=yCONFIG_BLK_DEV_CS5530=yCONFIG_BLK_DEV_HPT34X=y
# CONFIG_HPT34X_AUTODMA is not setCONFIG_BLK_DEV_HPT366=yCONFIG_BLK_DEV_PIIX=yCONFIG_BLK_DEV_NS87415=yCONFIG_BLK_DEV_OPTI621=y
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_PDC202XX_BURST is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not setCONFIG_BLK_DEV_RZ1000=y
# CONFIG_BLK_DEV_SC1200 is not setCONFIG_BLK_DEV_SVWKS=y
# CONFIG_BLK_DEV_SIIMAGE is not setCONFIG_BLK_DEV_SIS5513=yCONFIG_BLK_DEV_SLC90E66=yCONFIG_BLK_DEV_TRM290=yCONFIG_BLK_DEV_VIA82CXXX=yCONFIG_IDE_CHIPSETS=yCONFIG_BLK_DEV_4DRIVES=yCONFIG_BLK_DEV_ALI14XX=mCONFIG_BLK_DEV_DTC2278=mCONFIG_BLK_DEV_HT6560B=m
# CONFIG_BLK_DEV_PDC4030 is not setCONFIG_BLK_DEV_QD65XX=mCONFIG_BLK_DEV_UMC8672=m
# CONFIG_IDEDMA_AUTO is not set
# CONFIG_IDEDMA_IVB is not set
# CONFIG_DMA_NONPCI is not setCONFIG_BLK_DEV_IDE_MODES=yCONFIG_BLK_DEV_ATARAID=mCONFIG_BLK_DEV_ATARAID_PDC=mCONFIG_BLK_DEV_ATARAID_HPT=m
# CONFIG_BLK_DEV_ATARAID_SII is not set
#
# SCSI support
#CONFIG_SCSI=mCONFIG_BLK_DEV_SD=mCONFIG_SD_EXTRA_DEVS=64CONFIG_CHR_DEV_ST=mCONFIG_CHR_DEV_OSST=mCONFIG_BLK_DEV_SR=mCONFIG_BLK_DEV_SR_VENDOR=yCONFIG_SR_EXTRA_DEVS=4CONFIG_CHR_DEV_SG=m
# CONFIG_SCSI_DEBUG_QUEUES is not setCONFIG_SCSI_MULTI_LUN=yCONFIG_SCSI_CONSTANTS=yCONFIG_SCSI_LOGGING=y
#
# SCSI low-level drivers
#CONFIG_BLK_DEV_3W_XXXX_RAID=mCONFIG_SCSI_7000FASST=mCONFIG_SCSI_ACARD=mCONFIG_SCSI_AHA152X=mCONFIG_SCSI_AHA1542=mCONFIG_SCSI_AHA1740=mCONFIG_SCSI_AACRAID=mCONFIG_SCSI_AIC7XXX=mCONFIG_AIC7XXX_CMDS_PER_DEVICE=253CONFIG_AIC7XXX_RESET_DELAY_MS=15000CONFIG_AIC7XXX_PROBE_EISA_VL=y
# CONFIG_AIC7XXX_BUILD_FIRMWARE is not set
# CONFIG_SCSI_AIC79XX is not setCONFIG_SCSI_AIC7XXX_OLD=mCONFIG_AIC7XXX_OLD_TCQ_ON_BY_DEFAULT=yCONFIG_AIC7XXX_OLD_CMDS_PER_DEVICE=128CONFIG_AIC7XXX_OLD_PROC_STATS=yCONFIG_SCSI_DPT_I2O=mCONFIG_SCSI_ADVANSYS=mCONFIG_SCSI_IN2000=mCONFIG_SCSI_AM53C974=mCONFIG_SCSI_MEGARAID=mCONFIG_SCSI_BUSLOGIC=m
# CONFIG_SCSI_OMIT_FLASHPOINT is not setCONFIG_SCSI_CPQFCTS=mCONFIG_SCSI_DMX3191D=mCONFIG_SCSI_DTC3280=mCONFIG_SCSI_EATA=mCONFIG_SCSI_EATA_TAGGED_QUEUE=y
# CONFIG_SCSI_EATA_LINKED_COMMANDS is not setCONFIG_SCSI_EATA_MAX_TAGS=16CONFIG_SCSI_EATA_DMA=mCONFIG_SCSI_EATA_PIO=mCONFIG_SCSI_FUTURE_DOMAIN=mCONFIG_SCSI_FD_MCS=mCONFIG_SCSI_GDTH=mCONFIG_SCSI_GENERIC_NCR5380=m
# CONFIG_SCSI_GENERIC_NCR53C400 is not setCONFIG_SCSI_G_NCR5380_PORT=y
# CONFIG_SCSI_G_NCR5380_MEM is not setCONFIG_SCSI_IBMMCA=mCONFIG_IBMMCA_SCSI_ORDER_STANDARD=y
# CONFIG_IBMMCA_SCSI_DEV_RESET is not setCONFIG_SCSI_IPS=mCONFIG_SCSI_INITIO=mCONFIG_SCSI_INIA100=mCONFIG_SCSI_PPA=mCONFIG_SCSI_IMM=m
# CONFIG_SCSI_IZIP_EPP16 is not set
# CONFIG_SCSI_IZIP_SLOW_CTR is not setCONFIG_SCSI_NCR53C406A=mCONFIG_SCSI_NCR_D700=mCONFIG_53C700_IO_MAPPED=y
CONFIG_SCSI_NCR53C7xx=m
# CONFIG_SCSI_NCR53C7xx_sync is not set
CONFIG_SCSI_NCR53C7xx_FAST=y
CONFIG_SCSI_NCR53C7xx_DISCONNECT=yCONFIG_SCSI_SYM53C8XX_2=mCONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64CONFIG_SCSI_SYM53C8XX_IOMAPPED=yCONFIG_SCSI_NCR53C8XX=mCONFIG_SCSI_SYM53C8XX=mCONFIG_SCSI_NCR53C8XX_DEFAULT_TAGS=8CONFIG_SCSI_NCR53C8XX_MAX_TAGS=32CONFIG_SCSI_NCR53C8XX_SYNC=20
# CONFIG_SCSI_NCR53C8XX_PROFILE is not set
# CONFIG_SCSI_NCR53C8XX_IOMAPPED is not setCONFIG_SCSI_NCR53C8XX_PQS_PDS=y
# CONFIG_SCSI_NCR53C8XX_SYMBIOS_COMPAT is not setCONFIG_SCSI_MCA_53C9X=mCONFIG_SCSI_PAS16=mCONFIG_SCSI_PCI2000=mCONFIG_SCSI_PCI2220I=mCONFIG_SCSI_PSI240I=mCONFIG_SCSI_QLOGIC_FAS=mCONFIG_SCSI_QLOGIC_ISP=mCONFIG_SCSI_QLOGIC_FC=m
# CONFIG_SCSI_QLOGIC_FC_FIRMWARE is not setCONFIG_SCSI_QLOGIC_1280=mCONFIG_SCSI_SEAGATE=mCONFIG_SCSI_SIM710=mCONFIG_SCSI_SYM53C416=mCONFIG_SCSI_DC390T=m
# CONFIG_SCSI_DC390T_NOGENSUPP is not setCONFIG_SCSI_T128=mCONFIG_SCSI_U14_34F=m
# CONFIG_SCSI_U14_34F_LINKED_COMMANDS is not setCONFIG_SCSI_U14_34F_MAX_TAGS=8CONFIG_SCSI_ULTRASTOR=m
# CONFIG_SCSI_NSP32 is not setCONFIG_SCSI_DEBUG=m
#
# PCMCIA SCSI adapter support
#CONFIG_SCSI_PCMCIA=yCONFIG_PCMCIA_AHA152X=mCONFIG_PCMCIA_FDOMAIN=mCONFIG_PCMCIA_NINJA_SCSI=mCONFIG_PCMCIA_QLOGIC=m
#
# Fusion MPT device support
#CONFIG_FUSION=m
# CONFIG_FUSION_BOOT is not setCONFIG_FUSION_MAX_SGE=40CONFIG_FUSION_ISENSE=mCONFIG_FUSION_CTL=mCONFIG_FUSION_LAN=mCONFIG_NET_FC=y
#
# IEEE 1394 (FireWire) support (EXPERIMENTAL)
#CONFIG_IEEE1394=mCONFIG_IEEE1394_PCILYNX=mCONFIG_IEEE1394_OHCI1394=mCONFIG_IEEE1394_VIDEO1394=mCONFIG_IEEE1394_SBP2=mCONFIG_IEEE1394_SBP2_PHYS_DMA=yCONFIG_IEEE1394_ETH1394=mCONFIG_IEEE1394_DV1394=mCONFIG_IEEE1394_RAWIO=mCONFIG_IEEE1394_CMP=mCONFIG_IEEE1394_AMDTP=m
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
#
# I2O device support
#CONFIG_I2O=mCONFIG_I2O_PCI=mCONFIG_I2O_BLOCK=mCONFIG_I2O_LAN=mCONFIG_I2O_SCSI=mCONFIG_I2O_PROC=m
#
# Network device support
#CONFIG_NETDEVICES=y
#
# ARCnet devices
#CONFIG_ARCNET=mCONFIG_ARCNET_1201=mCONFIG_ARCNET_1051=mCONFIG_ARCNET_RAW=m
CONFIG_ARCNET_COM90xx=m
CONFIG_ARCNET_COM90xxIO=mCONFIG_ARCNET_RIM_I=mCONFIG_ARCNET_COM20020=mCONFIG_ARCNET_COM20020_ISA=mCONFIG_ARCNET_COM20020_PCI=mCONFIG_DUMMY=mCONFIG_BONDING=mCONFIG_EQUALIZER=mCONFIG_TUN=mCONFIG_ETHERTAP=mCONFIG_NET_SB1000=m
#
# Ethernet (10 or 100Mbit)
#CONFIG_NET_ETHERNET=y
# CONFIG_SUNLANCE is not setCONFIG_HAPPYMEAL=m
# CONFIG_SUNBMAC is not set
# CONFIG_SUNQE is not setCONFIG_SUNGEM=mCONFIG_NET_VENDOR_3COM=yCONFIG_EL1=mCONFIG_EL2=mCONFIG_ELPLUS=mCONFIG_EL16=mCONFIG_EL3=mCONFIG_3C515=mCONFIG_ELMC=mCONFIG_ELMC_II=mCONFIG_VORTEX=m
# CONFIG_TYPHOON is not setCONFIG_LANCE=mCONFIG_NET_VENDOR_SMC=y
CONFIG_WD80x3=mCONFIG_ULTRAMCA=mCONFIG_ULTRA=mCONFIG_ULTRA32=mCONFIG_SMC9194=mCONFIG_NET_VENDOR_RACAL=yCONFIG_NI5010=mCONFIG_NI52=mCONFIG_NI65=mCONFIG_AT1700=mCONFIG_DEPCA=mCONFIG_HP100=mCONFIG_NET_ISA=yCONFIG_E2100=mCONFIG_EWRK3=mCONFIG_EEXPRESS=mCONFIG_EEXPRESS_PRO=mCONFIG_HPLAN_PLUS=mCONFIG_HPLAN=mCONFIG_LP486E=mCONFIG_ETH16I=mCONFIG_NE2000=mCONFIG_SKMC=mCONFIG_NE2_MCA=mCONFIG_IBMLANA=mCONFIG_NET_PCI=yCONFIG_PCNET32=m
# CONFIG_AMD8111_ETH is not setCONFIG_ADAPTEC_STARFIRE=mCONFIG_AC3200=mCONFIG_APRICOT=m
CONFIG_CS89x0=mCONFIG_TULIP=mCONFIG_TULIP_MWI=yCONFIG_TULIP_MMIO=yCONFIG_DE4X5=mCONFIG_DGRS=mCONFIG_DM9102=mCONFIG_EEPRO100=m
# CONFIG_EEPRO100_PIO is not setCONFIG_E100=mCONFIG_LNE390=mCONFIG_FEALNX=mCONFIG_NATSEMI=mCONFIG_NE2K_PCI=mCONFIG_NE3210=mCONFIG_ES3210=mCONFIG_8139CP=mCONFIG_8139TOO=m
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not setCONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not setCONFIG_SIS900=mCONFIG_EPIC100=mCONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not setCONFIG_TLAN=mCONFIG_TC35815=mCONFIG_VIA_RHINE=m
# CONFIG_VIA_RHINE_MMIO is not setCONFIG_WINBOND_840=mCONFIG_NET_POCKET=yCONFIG_ATP=mCONFIG_DE600=mCONFIG_DE620=m
#
# Ethernet (1000 Mbit)
#CONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not setCONFIG_DL2K=mCONFIG_E1000=m
# CONFIG_MYRI_SBUS is not setCONFIG_NS83820=mCONFIG_HAMACHI=mCONFIG_YELLOWFIN=m
# CONFIG_R8169 is not setCONFIG_SK98LIN=mCONFIG_TIGON3=mCONFIG_FDDI=yCONFIG_DEFXX=mCONFIG_SKFP=mCONFIG_HIPPI=yCONFIG_ROADRUNNER=m
# CONFIG_ROADRUNNER_LARGE_RINGS is not setCONFIG_PLIP=mCONFIG_PPP=mCONFIG_PPP_MULTILINK=yCONFIG_PPP_FILTER=yCONFIG_PPP_ASYNC=mCONFIG_PPP_SYNC_TTY=mCONFIG_PPP_DEFLATE=mCONFIG_PPP_BSDCOMP=mCONFIG_PPPOE=mCONFIG_PPPOATM=mCONFIG_SLIP=mCONFIG_SLIP_COMPRESSED=yCONFIG_SLIP_SMART=yCONFIG_SLIP_MODE_SLIP6=y
#
# Wireless LAN (non-hamradio)
#CONFIG_NET_RADIO=yCONFIG_STRIP=mCONFIG_WAVELAN=mCONFIG_ARLAN=mCONFIG_AIRONET4500=mCONFIG_AIRONET4500_NONCS=mCONFIG_AIRONET4500_PNP=yCONFIG_AIRONET4500_PCI=y
# CONFIG_AIRONET4500_ISA is not set
# CONFIG_AIRONET4500_I365 is not setCONFIG_AIRONET4500_PROC=mCONFIG_AIRO=mCONFIG_HERMES=mCONFIG_PLX_HERMES=mCONFIG_PCI_HERMES=mCONFIG_PCMCIA_HERMES=mCONFIG_AIRO_CS=mCONFIG_NET_WIRELESS=y
#
# Token Ring devices
#CONFIG_TR=yCONFIG_IBMTR=mCONFIG_IBMOL=mCONFIG_IBMLS=mCONFIG_3C359=mCONFIG_TMS380TR=mCONFIG_TMSPCI=mCONFIG_TMSISA=mCONFIG_ABYSS=mCONFIG_MADGEMC=mCONFIG_SMCTR=mCONFIG_NET_FC=yCONFIG_IPHASE5526=mCONFIG_RCPCI=mCONFIG_SHAPER=m
#
# Wan interfaces
#CONFIG_WAN=yCONFIG_HOSTESS_SV11=mCONFIG_COSA=mCONFIG_COMX=mCONFIG_COMX_HW_COMX=mCONFIG_COMX_HW_LOCOMX=mCONFIG_COMX_HW_MIXCOM=mCONFIG_COMX_HW_MUNICH=mCONFIG_COMX_PROTO_PPP=mCONFIG_COMX_PROTO_LAPB=mCONFIG_COMX_PROTO_FR=mCONFIG_DSCC4=mCONFIG_LANMEDIA=mCONFIG_ATI_XX20=mCONFIG_SEALEVEL_4021=mCONFIG_SYNCLINK_SYNCPPP=mCONFIG_HDLC=m
# CONFIG_HDLC_RAW is not set
# CONFIG_HDLC_CISCO is not set
# CONFIG_HDLC_FR is not setCONFIG_HDLC_PPP=yCONFIG_HDLC_X25=yCONFIG_N2=mCONFIG_C101=mCONFIG_FARSYNC=m
# CONFIG_HDLC_DEBUG_PKT is not set
# CONFIG_HDLC_DEBUG_HARD_HEADER is not set
# CONFIG_HDLC_DEBUG_ECN is not set
# CONFIG_HDLC_DEBUG_RINGS is not setCONFIG_DLCI=mCONFIG_DLCI_COUNT=24CONFIG_DLCI_MAX=8CONFIG_SDLA=mCONFIG_WAN_ROUTER_DRIVERS=yCONFIG_VENDOR_SANGOMA=mCONFIG_WANPIPE_CHDLC=yCONFIG_WANPIPE_FR=yCONFIG_WANPIPE_X25=yCONFIG_WANPIPE_PPP=yCONFIG_WANPIPE_MULTPPP=yCONFIG_CYCLADES_SYNC=mCONFIG_CYCLOMX_X25=yCONFIG_LAPBETHER=mCONFIG_X25_ASY=mCONFIG_SBNI=m
# CONFIG_SBNI_MULTILINE is not set
#
# PCMCIA network device support
#CONFIG_NET_PCMCIA=yCONFIG_PCMCIA_3C589=mCONFIG_PCMCIA_3C574=mCONFIG_PCMCIA_FMVJ18X=mCONFIG_PCMCIA_PCNET=mCONFIG_PCMCIA_AXNET=mCONFIG_PCMCIA_NMCLAN=mCONFIG_PCMCIA_SMC91C92=mCONFIG_PCMCIA_XIRC2PS=mCONFIG_ARCNET_COM20020_CS=mCONFIG_PCMCIA_IBMTR=mCONFIG_PCMCIA_XIRCOM=mCONFIG_PCMCIA_XIRTULIP=mCONFIG_NET_PCMCIA_RADIO=yCONFIG_PCMCIA_RAYCS=mCONFIG_PCMCIA_NETWAVE=mCONFIG_PCMCIA_WAVELAN=mCONFIG_AIRONET4500_CS=m
#
# Amateur Radio support
#CONFIG_HAMRADIO=yCONFIG_AX25=mCONFIG_AX25_DAMA_SLAVE=yCONFIG_NETROM=mCONFIG_ROSE=m
#
# AX.25 network device drivers
#CONFIG_MKISS=mCONFIG_6PACK=mCONFIG_BPQETHER=mCONFIG_DMASCC=mCONFIG_SCC=m
# CONFIG_SCC_DELAY is not set
# CONFIG_SCC_TRXECHO is not setCONFIG_BAYCOM_SER_FDX=mCONFIG_BAYCOM_SER_HDX=mCONFIG_BAYCOM_PAR=mCONFIG_BAYCOM_EPP=mCONFIG_SOUNDMODEM=mCONFIG_SOUNDMODEM_SBC=yCONFIG_SOUNDMODEM_WSS=yCONFIG_SOUNDMODEM_AFSK1200=yCONFIG_SOUNDMODEM_AFSK2400_7=yCONFIG_SOUNDMODEM_AFSK2400_8=yCONFIG_SOUNDMODEM_AFSK2666=yCONFIG_SOUNDMODEM_HAPN4800=yCONFIG_SOUNDMODEM_PSK4800=yCONFIG_SOUNDMODEM_FSK9600=yCONFIG_YAM=m
#
# IrDA (infrared) support
#CONFIG_IRDA=mCONFIG_IRLAN=mCONFIG_IRNET=mCONFIG_IRCOMM=mCONFIG_IRDA_ULTRA=yCONFIG_IRDA_CACHE_LAST_LSAP=yCONFIG_IRDA_FAST_RR=yCONFIG_IRDA_DEBUG=y
#
# Infrared-port device drivers
#CONFIG_IRTTY_SIR=mCONFIG_IRPORT_SIR=mCONFIG_DONGLE=yCONFIG_ESI_DONGLE=mCONFIG_ACTISYS_DONGLE=mCONFIG_TEKRAM_DONGLE=mCONFIG_GIRBIL_DONGLE=mCONFIG_LITELINK_DONGLE=mCONFIG_MCP2120_DONGLE=mCONFIG_OLD_BELKIN_DONGLE=mCONFIG_ACT200L_DONGLE=mCONFIG_MA600_DONGLE=mCONFIG_USB_IRDA=mCONFIG_NSC_FIR=mCONFIG_WINBOND_FIR=m
# CONFIG_TOSHIBA_OLD is not setCONFIG_TOSHIBA_FIR=mCONFIG_SMC_IRCC_FIR=mCONFIG_ALI_FIR=mCONFIG_VLSI_FIR=m
#
# ISDN subsystem
#CONFIG_ISDN=mCONFIG_ISDN_BOOL=yCONFIG_ISDN_PPP=yCONFIG_ISDN_PPP_VJ=yCONFIG_ISDN_MPP=yCONFIG_ISDN_PPP_BSDCOMP=mCONFIG_ISDN_AUDIO=yCONFIG_ISDN_TTY_FAX=yCONFIG_ISDN_X25=y
#
# ISDN feature submodules
#CONFIG_ISDN_DRV_LOOP=mCONFIG_ISDN_DIVERSION=m
#
# Passive ISDN cards
#CONFIG_ISDN_DRV_HISAX=mCONFIG_ISDN_HISAX=yCONFIG_HISAX_EURO=yCONFIG_DE_AOC=y
# CONFIG_HISAX_NO_SENDCOMPLETE is not set
# CONFIG_HISAX_NO_LLC is not set
# CONFIG_HISAX_NO_KEYPAD is not setCONFIG_HISAX_1TR6=yCONFIG_HISAX_NI1=yCONFIG_HISAX_MAX_CARDS=8CONFIG_HISAX_16_0=yCONFIG_HISAX_16_3=yCONFIG_HISAX_AVM_A1=yCONFIG_HISAX_IX1MICROR2=yCONFIG_HISAX_ASUSCOM=yCONFIG_HISAX_TELEINT=yCONFIG_HISAX_HFCS=yCONFIG_HISAX_SPORTSTER=yCONFIG_HISAX_MIC=yCONFIG_HISAX_ISURF=yCONFIG_HISAX_HSTSAPHIR=yCONFIG_HISAX_TELESPCI=yCONFIG_HISAX_S0BOX=yCONFIG_HISAX_FRITZPCI=yCONFIG_HISAX_AVM_A1_PCMCIA=yCONFIG_HISAX_ELSA=yCONFIG_HISAX_DIEHLDIVA=yCONFIG_HISAX_SEDLBAUER=yCONFIG_HISAX_NETJET=yCONFIG_HISAX_NETJET_U=yCONFIG_HISAX_NICCY=yCONFIG_HISAX_BKM_A4T=yCONFIG_HISAX_SCT_QUADRO=yCONFIG_HISAX_GAZEL=yCONFIG_HISAX_HFC_PCI=yCONFIG_HISAX_W6692=yCONFIG_HISAX_HFC_SX=y
# CONFIG_HISAX_ENTERNOW_PCI is not set
# CONFIG_HISAX_DEBUG is not setCONFIG_HISAX_SEDLBAUER_CS=mCONFIG_HISAX_ELSA_CS=mCONFIG_HISAX_AVM_A1_CS=mCONFIG_HISAX_ST5481=mCONFIG_HISAX_FRITZ_PCIPNP=m
# CONFIG_USB_AUERISDN is not set
#
# Active ISDN cards
#CONFIG_ISDN_DRV_ICN=mCONFIG_ISDN_DRV_PCBIT=mCONFIG_ISDN_DRV_SC=mCONFIG_ISDN_DRV_ACT2000=mCONFIG_ISDN_DRV_EICON=yCONFIG_ISDN_DRV_EICON_DIVAS=mCONFIG_ISDN_DRV_EICON_OLD=mCONFIG_ISDN_DRV_EICON_PCI=yCONFIG_ISDN_DRV_EICON_ISA=yCONFIG_ISDN_DRV_TPAM=mCONFIG_ISDN_CAPI=mCONFIG_ISDN_DRV_AVMB1_VERBOSE_REASON=yCONFIG_ISDN_CAPI_MIDDLEWARE=yCONFIG_ISDN_CAPI_CAPI20=mCONFIG_ISDN_CAPI_CAPIFS_BOOL=yCONFIG_ISDN_CAPI_CAPIFS=mCONFIG_ISDN_CAPI_CAPIDRV=mCONFIG_ISDN_DRV_AVMB1_B1ISA=mCONFIG_ISDN_DRV_AVMB1_B1PCI=mCONFIG_ISDN_DRV_AVMB1_B1PCIV4=yCONFIG_ISDN_DRV_AVMB1_T1ISA=mCONFIG_ISDN_DRV_AVMB1_B1PCMCIA=mCONFIG_ISDN_DRV_AVMB1_AVM_CS=mCONFIG_ISDN_DRV_AVMB1_T1PCI=mCONFIG_ISDN_DRV_AVMB1_C4=mCONFIG_HYSDN=mCONFIG_HYSDN_CAPI=y
#
# Old CD-ROM drivers (not SCSI, not IDE)
#CONFIG_CD_NO_IDESCSI=yCONFIG_AZTCD=mCONFIG_GSCD=mCONFIG_SBPCD=mCONFIG_MCD=mCONFIG_MCD_IRQ=11CONFIG_MCD_BASE=300CONFIG_MCDX=mCONFIG_OPTCD=mCONFIG_CM206=mCONFIG_SJCD=mCONFIG_ISP16_CDI=mCONFIG_CDU31A=mCONFIG_CDU535=m
#
# Input core support
#CONFIG_INPUT=mCONFIG_INPUT_KEYBDEV=mCONFIG_INPUT_MOUSEDEV=mCONFIG_INPUT_MOUSEDEV_SCREEN_X=1024CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768CONFIG_INPUT_JOYDEV=mCONFIG_INPUT_EVDEV=m
#
# Character devices
#CONFIG_VT=yCONFIG_VT_CONSOLE=yCONFIG_SERIAL=yCONFIG_SERIAL_CONSOLE=y
# CONFIG_SERIAL_EXTENDED is not setCONFIG_SERIAL_NONSTANDARD=yCONFIG_COMPUTONE=mCONFIG_ROCKETPORT=mCONFIG_CYCLADES=mCONFIG_CYZ_INTR=yCONFIG_DIGIEPCA=mCONFIG_ESPSERIAL=mCONFIG_MOXA_INTELLIO=mCONFIG_MOXA_SMARTIO=mCONFIG_ISI=mCONFIG_SYNCLINK=mCONFIG_SYNCLINKMP=mCONFIG_N_HDLC=mCONFIG_RISCOM8=mCONFIG_SPECIALIX=m
# CONFIG_SPECIALIX_RTSCTS is not setCONFIG_SX=mCONFIG_RIO=m
# CONFIG_RIO_OLDPCI is not set
# CONFIG_STALDRV is not setCONFIG_UNIX98_PTYS=yCONFIG_UNIX98_PTY_COUNT=512CONFIG_PRINTER=mCONFIG_LP_CONSOLE=yCONFIG_PPDEV=m
# CONFIG_TIPAR is not set
#
# I2C support
#CONFIG_I2C=mCONFIG_I2C_ALGOBIT=mCONFIG_I2C_PHILIPSPAR=mCONFIG_I2C_ELV=mCONFIG_I2C_VELLEMAN=m
# CONFIG_SCx200_I2C is not set
# CONFIG_SCx200_ACB is not setCONFIG_I2C_ALGOPCF=mCONFIG_I2C_ELEKTOR=mCONFIG_I2C_CHARDEV=mCONFIG_I2C_PROC=m
#
# Mice
#CONFIG_BUSMOUSE=mCONFIG_ATIXL_BUSMOUSE=mCONFIG_LOGIBUSMOUSE=mCONFIG_MS_BUSMOUSE=mCONFIG_MOUSE=mCONFIG_PSMOUSE=yCONFIG_82C710_MOUSE=mCONFIG_PC110_PAD=mCONFIG_MK712_MOUSE=m
#
# Joysticks
#CONFIG_INPUT_GAMEPORT=mCONFIG_INPUT_NS558=mCONFIG_INPUT_LIGHTNING=mCONFIG_INPUT_PCIGAME=mCONFIG_INPUT_CS461X=mCONFIG_INPUT_EMU10K1=mCONFIG_INPUT_SERIO=mCONFIG_INPUT_SERPORT=mCONFIG_INPUT_ANALOG=mCONFIG_INPUT_A3D=mCONFIG_INPUT_ADI=mCONFIG_INPUT_COBRA=mCONFIG_INPUT_GF2K=mCONFIG_INPUT_GRIP=mCONFIG_INPUT_INTERACT=mCONFIG_INPUT_TMDC=mCONFIG_INPUT_SIDEWINDER=mCONFIG_INPUT_IFORCE_USB=mCONFIG_INPUT_IFORCE_232=mCONFIG_INPUT_WARRIOR=mCONFIG_INPUT_MAGELLAN=mCONFIG_INPUT_SPACEORB=mCONFIG_INPUT_SPACEBALL=mCONFIG_INPUT_STINGER=mCONFIG_INPUT_DB9=mCONFIG_INPUT_GAMECON=mCONFIG_INPUT_TURBOGRAFX=mCONFIG_QIC02_TAPE=mCONFIG_QIC02_DYNCONF=y
# CONFIG_IPMI_HANDLER is not set
# CONFIG_IPMI_PANIC_EVENT is not set
# CONFIG_IPMI_DEVICE_INTERFACE is not set
# CONFIG_IPMI_KCS is not set
# CONFIG_IPMI_WATCHDOG is not set
#
# Watchdog Cards
#CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not setCONFIG_ACQUIRE_WDT=mCONFIG_ADVANTECH_WDT=m
# CONFIG_ALIM1535_WDT is not setCONFIG_ALIM7101_WDT=mCONFIG_SC520_WDT=mCONFIG_PCWATCHDOG=mCONFIG_EUROTECH_WDT=mCONFIG_IB700_WDT=mCONFIG_WAFER_WDT=mCONFIG_I810_TCO=mCONFIG_MIXCOMWD=mCONFIG_60XX_WDT=mCONFIG_SC1200_WDT=m
# CONFIG_SCx200_WDT is not setCONFIG_SOFT_WATCHDOG=mCONFIG_W83877F_WDT=mCONFIG_WDT=mCONFIG_WDTPCI=mCONFIG_WDT_501=y
# CONFIG_WDT_501_FAN is not setCONFIG_MACHZ_WDT=mCONFIG_AMD7XX_TCO=m
# CONFIG_SCx200_GPIO is not setCONFIG_AMD_RNG=mCONFIG_INTEL_RNG=mCONFIG_AMD_PM768=mCONFIG_NVRAM=mCONFIG_RTC=mCONFIG_DTLK=mCONFIG_R3964=mCONFIG_APPLICOM=mCONFIG_SONYPI=m
#
# Ftape, the floppy tape device driver
#CONFIG_FTAPE=mCONFIG_ZFTAPE=mCONFIG_ZFT_DFLT_BLK_SZ=10240CONFIG_ZFT_COMPRESSOR=mCONFIG_FT_NR_BUFFERS=3CONFIG_FT_PROC_FS=yCONFIG_FT_NORMAL_DEBUG=y
# CONFIG_FT_FULL_DEBUG is not set
# CONFIG_FT_NO_TRACE is not set
# CONFIG_FT_NO_TRACE_AT_ALL is not setCONFIG_FT_STD_FDC=y
# CONFIG_FT_MACH2 is not set
# CONFIG_FT_PROBE_FC10 is not set
# CONFIG_FT_ALT_FDC is not setCONFIG_FT_FDC_THR=8CONFIG_FT_FDC_MAX_RATE=2000CONFIG_FT_ALPHA_CLOCK=0CONFIG_AGP=mCONFIG_AGP_INTEL=yCONFIG_AGP_I810=yCONFIG_AGP_VIA=yCONFIG_AGP_AMD=yCONFIG_AGP_AMD_8151=yCONFIG_AGP_SIS=yCONFIG_AGP_ALI=yCONFIG_AGP_SWORKS=yCONFIG_DRM=y
# CONFIG_DRM_OLD is not setCONFIG_DRM_NEW=yCONFIG_DRM_TDFX=mCONFIG_DRM_R128=mCONFIG_DRM_RADEON=mCONFIG_DRM_I810=m
# CONFIG_DRM_I810_XFREE_41 is not setCONFIG_DRM_I830=mCONFIG_DRM_MGA=mCONFIG_DRM_SIS=m
#
# PCMCIA character devices
#CONFIG_PCMCIA_SERIAL_CS=mCONFIG_SYNCLINK_CS=mCONFIG_MWAVE=m
#
# Multimedia devices
#CONFIG_VIDEO_DEV=m
#
# Video For Linux
#CONFIG_VIDEO_PROC_FS=yCONFIG_I2C_PARPORT=mCONFIG_VIDEO_BT848=mCONFIG_VIDEO_PMS=mCONFIG_VIDEO_BWQCAM=mCONFIG_VIDEO_CQCAM=mCONFIG_VIDEO_W9966=mCONFIG_VIDEO_CPIA=mCONFIG_VIDEO_CPIA_PP=mCONFIG_VIDEO_CPIA_USB=mCONFIG_VIDEO_SAA5249=mCONFIG_TUNER_3036=mCONFIG_VIDEO_STRADIS=mCONFIG_VIDEO_ZORAN=mCONFIG_VIDEO_ZORAN_BUZ=mCONFIG_VIDEO_ZORAN_DC10=mCONFIG_VIDEO_ZORAN_LML33=mCONFIG_VIDEO_ZR36120=mCONFIG_VIDEO_MEYE=m
#
# Radio Adapters
#CONFIG_RADIO_CADET=mCONFIG_RADIO_RTRACK=mCONFIG_RADIO_RTRACK2=mCONFIG_RADIO_AZTECH=mCONFIG_RADIO_GEMTEK=mCONFIG_RADIO_GEMTEK_PCI=mCONFIG_RADIO_MAXIRADIO=mCONFIG_RADIO_MAESTRO=mCONFIG_RADIO_MIROPCM20=mCONFIG_RADIO_MIROPCM20_RDS=mCONFIG_RADIO_SF16FMI=m
# CONFIG_RADIO_SF16FMR2 is not setCONFIG_RADIO_TERRATEC=mCONFIG_RADIO_TRUST=mCONFIG_RADIO_TYPHOON=mCONFIG_RADIO_TYPHOON_PROC_FS=yCONFIG_RADIO_ZOLTRIX=m
#
# File systems
#CONFIG_QUOTA=yCONFIG_AUTOFS_FS=mCONFIG_AUTOFS4_FS=mCONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not setCONFIG_REISERFS_PROC_INFO=yCONFIG_ADFS_FS=mCONFIG_ADFS_FS_RW=yCONFIG_AFFS_FS=mCONFIG_HFS_FS=mCONFIG_BEFS_FS=m
# CONFIG_BEFS_DEBUG is not setCONFIG_BFS_FS=mCONFIG_EXT3_FS=mCONFIG_JBD=m
# CONFIG_JBD_DEBUG is not setCONFIG_FAT_FS=mCONFIG_MSDOS_FS=mCONFIG_UMSDOS_FS=mCONFIG_VFAT_FS=mCONFIG_EFS_FS=mCONFIG_JFFS_FS=mCONFIG_JFFS_FS_VERBOSE=0CONFIG_JFFS_PROC_FS=yCONFIG_JFFS2_FS=mCONFIG_JFFS2_FS_DEBUG=0CONFIG_CRAMFS=mCONFIG_TMPFS=yCONFIG_RAMFS=yCONFIG_ISO9660_FS=mCONFIG_JOLIET=yCONFIG_ZISOFS=yCONFIG_JFS_FS=m
# CONFIG_JFS_DEBUG is not setCONFIG_JFS_STATISTICS=yCONFIG_MINIX_FS=mCONFIG_VXFS_FS=mCONFIG_NTFS_FS=mCONFIG_NTFS_RW=yCONFIG_HPFS_FS=mCONFIG_PROC_FS=yCONFIG_DEVFS_FS=y
# CONFIG_DEVFS_MOUNT is not set
# CONFIG_DEVFS_DEBUG is not setCONFIG_DEVPTS_FS=yCONFIG_QNX4FS_FS=mCONFIG_QNX4FS_RW=yCONFIG_ROMFS_FS=yCONFIG_EXT2_FS=mCONFIG_SYSV_FS=mCONFIG_UDF_FS=mCONFIG_UDF_RW=yCONFIG_UFS_FS=mCONFIG_UFS_FS_WRITE=y
#
# Network File Systems
#CONFIG_CODA_FS=mCONFIG_INTERMEZZO_FS=mCONFIG_NFS_FS=mCONFIG_NFS_V3=y
# CONFIG_ROOT_NFS is not setCONFIG_NFSD=mCONFIG_NFSD_V3=yCONFIG_NFSD_TCP=yCONFIG_SUNRPC=mCONFIG_LOCKD=mCONFIG_LOCKD_V4=yCONFIG_SMB_FS=mCONFIG_SMB_NLS_DEFAULT=yCONFIG_SMB_NLS_REMOTE="cp437"CONFIG_NCP_FS=mCONFIG_NCPFS_PACKET_SIGNING=yCONFIG_NCPFS_IOCTL_LOCKING=yCONFIG_NCPFS_STRONG=yCONFIG_NCPFS_NFS_NS=yCONFIG_NCPFS_OS2_NS=yCONFIG_NCPFS_SMALLDOS=yCONFIG_NCPFS_NLS=yCONFIG_NCPFS_EXTRAS=yCONFIG_ZISOFS_FS=m
#
# Partition Types
#CONFIG_PARTITION_ADVANCED=yCONFIG_ACORN_PARTITION=yCONFIG_ACORN_PARTITION_ICS=yCONFIG_ACORN_PARTITION_ADFS=y
# CONFIG_ACORN_PARTITION_POWERTEC is not setCONFIG_ACORN_PARTITION_RISCIX=yCONFIG_OSF_PARTITION=yCONFIG_AMIGA_PARTITION=yCONFIG_ATARI_PARTITION=yCONFIG_MAC_PARTITION=yCONFIG_MSDOS_PARTITION=yCONFIG_BSD_DISKLABEL=yCONFIG_MINIX_SUBPARTITION=yCONFIG_SOLARIS_X86_PARTITION=yCONFIG_UNIXWARE_DISKLABEL=yCONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not setCONFIG_SGI_PARTITION=yCONFIG_ULTRIX_PARTITION=yCONFIG_SUN_PARTITION=yCONFIG_EFI_PARTITION=yCONFIG_SMB_NLS=yCONFIG_NLS=y
#
# Native Language Support
#CONFIG_NLS_DEFAULT="iso8859-1"CONFIG_NLS_CODEPAGE_437=mCONFIG_NLS_CODEPAGE_737=mCONFIG_NLS_CODEPAGE_775=mCONFIG_NLS_CODEPAGE_850=mCONFIG_NLS_CODEPAGE_852=mCONFIG_NLS_CODEPAGE_855=mCONFIG_NLS_CODEPAGE_857=mCONFIG_NLS_CODEPAGE_860=mCONFIG_NLS_CODEPAGE_861=mCONFIG_NLS_CODEPAGE_862=mCONFIG_NLS_CODEPAGE_863=mCONFIG_NLS_CODEPAGE_864=mCONFIG_NLS_CODEPAGE_865=mCONFIG_NLS_CODEPAGE_866=mCONFIG_NLS_CODEPAGE_869=mCONFIG_NLS_CODEPAGE_936=mCONFIG_NLS_CODEPAGE_950=mCONFIG_NLS_CODEPAGE_932=mCONFIG_NLS_CODEPAGE_949=mCONFIG_NLS_CODEPAGE_874=mCONFIG_NLS_ISO8859_8=mCONFIG_NLS_CODEPAGE_1250=mCONFIG_NLS_CODEPAGE_1251=mCONFIG_NLS_ISO8859_1=mCONFIG_NLS_ISO8859_2=mCONFIG_NLS_ISO8859_3=mCONFIG_NLS_ISO8859_4=mCONFIG_NLS_ISO8859_5=mCONFIG_NLS_ISO8859_6=mCONFIG_NLS_ISO8859_7=mCONFIG_NLS_ISO8859_9=mCONFIG_NLS_ISO8859_13=mCONFIG_NLS_ISO8859_14=mCONFIG_NLS_ISO8859_15=mCONFIG_NLS_KOI8_R=mCONFIG_NLS_KOI8_U=mCONFIG_NLS_UTF8=m
#
# Console drivers
#CONFIG_VGA_CONSOLE=yCONFIG_VIDEO_SELECT=yCONFIG_MDA_CONSOLE=m
#
# Frame-buffer support
#CONFIG_FB=yCONFIG_DUMMY_CONSOLE=yCONFIG_FB_RIVA=mCONFIG_FB_CLGEN=mCONFIG_FB_PM2=m
# CONFIG_FB_PM2_FIFO_DISCONNECT is not set
# CONFIG_FB_PM2_PCI is not setCONFIG_FB_PM3=mCONFIG_FB_CYBER2000=mCONFIG_FB_VESA=yCONFIG_FB_VGA16=mCONFIG_FB_HGA=mCONFIG_VIDEO_SELECT=yCONFIG_FB_MATROX=mCONFIG_FB_MATROX_MILLENIUM=yCONFIG_FB_MATROX_MYSTIQUE=yCONFIG_FB_MATROX_G450=mCONFIG_FB_MATROX_I2C=mCONFIG_FB_MATROX_MAVEN=m
# CONFIG_FB_MATROX_PROC is not setCONFIG_FB_MATROX_MULTIHEAD=yCONFIG_FB_ATY=mCONFIG_FB_ATY_GX=yCONFIG_FB_ATY_CT=yCONFIG_FB_RADEON=mCONFIG_FB_ATY128=m
# CONFIG_FB_INTEL is not setCONFIG_FB_SIS=mCONFIG_FB_SIS_300=yCONFIG_FB_SIS_315=yCONFIG_FB_NEOMAGIC=mCONFIG_FB_3DFX=mCONFIG_FB_VOODOO1=mCONFIG_FB_TRIDENT=mCONFIG_FB_VIRTUAL=mCONFIG_FBCON_ADVANCED=yCONFIG_FBCON_MFB=mCONFIG_FBCON_CFB2=mCONFIG_FBCON_CFB4=mCONFIG_FBCON_CFB8=yCONFIG_FBCON_CFB16=yCONFIG_FBCON_CFB24=yCONFIG_FBCON_CFB32=m
# CONFIG_FBCON_AFB is not set
# CONFIG_FBCON_ILBM is not set
# CONFIG_FBCON_IPLAN2P2 is not set
# CONFIG_FBCON_IPLAN2P4 is not set
# CONFIG_FBCON_IPLAN2P8 is not set
# CONFIG_FBCON_MAC is not setCONFIG_FBCON_VGA_PLANES=mCONFIG_FBCON_VGA=mCONFIG_FBCON_HGA=m
# CONFIG_FBCON_FONTWIDTH8_ONLY is not setCONFIG_FBCON_FONTS=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_FONT_SUN8x16 is not set
# CONFIG_FONT_SUN12x22 is not set
# CONFIG_FONT_6x11 is not set
# CONFIG_FONT_PEARL_8x8 is not set
# CONFIG_FONT_ACORN_8x8 is not set
#
# Sound
#CONFIG_SOUND=mCONFIG_SOUND_ALI5455=mCONFIG_SOUND_BT878=mCONFIG_SOUND_CMPCI=m CONFIG_SOUND_CMPCI_FM=y CONFIG_SOUND_CMPCI_FMIO=388 CONFIG_SOUND_CMPCI_FMIO=388 CONFIG_SOUND_CMPCI_MIDI=y CONFIG_SOUND_CMPCI_MPUIO=330 CONFIG_SOUND_CMPCI_JOYSTICK=y CONFIG_SOUND_CMPCI_CM8738=y CONFIG_SOUND_CMPCI_SPDIFINVERSE=y
# CONFIG_SOUND_CMPCI_SPDIFLOOP is not set CONFIG_SOUND_CMPCI_SPEAKERS=2 CONFIG_SOUND_EMU10K1=m CONFIG_MIDI_EMU10K1=yCONFIG_SOUND_FUSION=mCONFIG_SOUND_CS4281=mCONFIG_SOUND_ES1370=mCONFIG_SOUND_ES1371=m CONFIG_SOUND_ESSSOLO1=m CONFIG_SOUND_MAESTRO=m CONFIG_SOUND_MAESTRO3=m CONFIG_SOUND_FORTE=mCONFIG_SOUND_ICH=mCONFIG_SOUND_RME96XX=m CONFIG_SOUND_SONICVIBES=m CONFIG_SOUND_TRIDENT=m CONFIG_SOUND_MSNDCLAS=m
# CONFIG_MSNDCLAS_HAVE_BOOT is not set CONFIG_MSNDCLAS_INIT_FILE="/etc/sound/msndinit.bin" CONFIG_MSNDCLAS_PERM_FILE="/etc/sound/msndperm.bin" CONFIG_SOUND_MSNDPIN=m
# CONFIG_MSNDPIN_HAVE_BOOT is not set CONFIG_MSNDPIN_INIT_FILE="/etc/sound/pndspini.bin" CONFIG_MSNDPIN_PERM_FILE="/etc/sound/pndsperm.bin" CONFIG_SOUND_VIA82CXXX=m CONFIG_MIDI_VIA82CXXX=yCONFIG_SOUND_OSS=m
# CONFIG_SOUND_TRACEINIT is not set
# CONFIG_SOUND_DMAP is not set CONFIG_SOUND_AD1816=m
# CONFIG_SOUND_AD1889 is not set CONFIG_SOUND_SGALAXY=mCONFIG_SOUND_ADLIB=m CONFIG_SOUND_ACI_MIXER=m CONFIG_SOUND_CS4232=mCONFIG_SOUND_SSCAPE=mCONFIG_SOUND_GUS=m
# CONFIG_SOUND_GUS16 is not set
# CONFIG_SOUND_GUSMAX is not set CONFIG_SOUND_VMIDI=mCONFIG_SOUND_TRIX=mCONFIG_SOUND_MSS=mCONFIG_SOUND_MPU401=mCONFIG_SOUND_NM256=mCONFIG_SOUND_MAD16=mCONFIG_MAD16_OLDCARD=yCONFIG_SOUND_PAS=m
# CONFIG_PAS_JOYSTICK is not set CONFIG_SOUND_PSS=m
# CONFIG_PSS_MIXER is not set
# CONFIG_PSS_HAVE_BOOT is not set CONFIG_SOUND_SB=m CONFIG_SOUND_AWE32_SYNTH=m
# CONFIG_SOUND_KAHLUA is not set CONFIG_SOUND_WAVEFRONT=m CONFIG_SOUND_MAUI=mCONFIG_SOUND_YM3812=m CONFIG_SOUND_OPL3SA1=m CONFIG_SOUND_OPL3SA2=m CONFIG_SOUND_YMFPCI=m
# CONFIG_SOUND_YMFPCI_LEGACY is not set CONFIG_SOUND_UART6850=m CONFIG_SOUND_AEDSP16=mCONFIG_SC6600=y
# CONFIG_SC6600_JOY is not set CONFIG_SC6600_CDROM=4 CONFIG_SC6600_CDROMBASE=0 CONFIG_AEDSP16_SBPRO=y CONFIG_AEDSP16_MPU401=y CONFIG_SOUND_TVMIXER=m
#
# USB support
#CONFIG_USB=mCONFIG_USB_DEBUG=yCONFIG_USB_DEVICEFS=yCONFIG_USB_BANDWIDTH=yCONFIG_USB_EHCI_HCD=mCONFIG_USB_UHCI=mCONFIG_USB_UHCI_ALT=mCONFIG_USB_OHCI=mCONFIG_USB_AUDIO=mCONFIG_USB_EMI26=mCONFIG_USB_MIDI=mCONFIG_USB_STORAGE=mCONFIG_USB_STORAGE_DEBUG=yCONFIG_USB_STORAGE_DATAFAB=yCONFIG_USB_STORAGE_FREECOM=yCONFIG_USB_STORAGE_ISD200=yCONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_HP8200e=yCONFIG_USB_STORAGE_SDDR09=yCONFIG_USB_STORAGE_SDDR55=yCONFIG_USB_STORAGE_JUMPSHOT=yCONFIG_USB_ACM=mCONFIG_USB_PRINTER=mCONFIG_USB_HID=mCONFIG_USB_HIDINPUT=yCONFIG_USB_HIDDEV=yCONFIG_USB_KBD=mCONFIG_USB_MOUSE=mCONFIG_USB_AIPTEK=mCONFIG_USB_WACOM=m
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not setCONFIG_USB_DC2XX=mCONFIG_USB_MDC800=mCONFIG_USB_SCANNER=mCONFIG_USB_MICROTEK=mCONFIG_USB_HPUSBSCSI=mCONFIG_USB_IBMCAM=m
# CONFIG_USB_KONICAWC is not setCONFIG_USB_OV511=mCONFIG_USB_PWC=mCONFIG_USB_SE401=mCONFIG_USB_STV680=mCONFIG_USB_VICAM=mCONFIG_USB_DSBR=mCONFIG_USB_DABUSB=mCONFIG_USB_PEGASUS=mCONFIG_USB_RTL8150=mCONFIG_USB_KAWETH=mCONFIG_USB_CATC=mCONFIG_USB_CDCETHER=mCONFIG_USB_USBNET=mCONFIG_USB_USS720=m
#
# USB Serial Converter support
#CONFIG_USB_SERIAL=m
# CONFIG_USB_SERIAL_DEBUG is not setCONFIG_USB_SERIAL_GENERIC=yCONFIG_USB_SERIAL_BELKIN=mCONFIG_USB_SERIAL_WHITEHEAT=mCONFIG_USB_SERIAL_DIGI_ACCELEPORT=mCONFIG_USB_SERIAL_EMPEG=mCONFIG_USB_SERIAL_FTDI_SIO=mCONFIG_USB_SERIAL_VISOR=mCONFIG_USB_SERIAL_IPAQ=mCONFIG_USB_SERIAL_IR=mCONFIG_USB_SERIAL_EDGEPORT=mCONFIG_USB_SERIAL_EDGEPORT_TI=mCONFIG_USB_SERIAL_KEYSPAN_PDA=mCONFIG_USB_SERIAL_KEYSPAN=mCONFIG_USB_SERIAL_KEYSPAN_USA28=yCONFIG_USB_SERIAL_KEYSPAN_USA28X=yCONFIG_USB_SERIAL_KEYSPAN_USA28XA=yCONFIG_USB_SERIAL_KEYSPAN_USA28XB=yCONFIG_USB_SERIAL_KEYSPAN_USA19=yCONFIG_USB_SERIAL_KEYSPAN_USA18X=yCONFIG_USB_SERIAL_KEYSPAN_USA19W=yCONFIG_USB_SERIAL_KEYSPAN_USA19QW=yCONFIG_USB_SERIAL_KEYSPAN_USA19QI=y
# CONFIG_USB_SERIAL_KEYSPAN_MPR is not setCONFIG_USB_SERIAL_KEYSPAN_USA49W=y
# CONFIG_USB_SERIAL_KEYSPAN_USA49WLC is not setCONFIG_USB_SERIAL_MCT_U232=mCONFIG_USB_SERIAL_KLSI=m
# CONFIG_USB_SERIAL_KOBIL_SCT is not setCONFIG_USB_SERIAL_PL2303=mCONFIG_USB_SERIAL_CYBERJACK=mCONFIG_USB_SERIAL_XIRCOM=mCONFIG_USB_SERIAL_OMNINET=mCONFIG_USB_RIO500=mCONFIG_USB_AUERSWALD=mCONFIG_USB_TIGL=mCONFIG_USB_BRLVGER=mCONFIG_USB_LCD=m
#
# Bluetooth support
#CONFIG_BLUEZ=mCONFIG_BLUEZ_L2CAP=mCONFIG_BLUEZ_SCO=m
# CONFIG_BLUEZ_RFCOMM is not setCONFIG_BLUEZ_BNEP=m
# CONFIG_BLUEZ_BNEP_MC_FILTER is not set
# CONFIG_BLUEZ_BNEP_PROTO_FILTER is not set
#
# Bluetooth device drivers
#CONFIG_BLUEZ_HCIUSB=m
# CONFIG_BLUEZ_USB_SCO is not setCONFIG_BLUEZ_USB_ZERO_PACKET=yCONFIG_BLUEZ_HCIUART=mCONFIG_BLUEZ_HCIUART_H4=y
# CONFIG_BLUEZ_HCIUART_BCSP is not set
# CONFIG_BLUEZ_HCIUART_BCSP_TXCRC is not setCONFIG_BLUEZ_HCIDTL1=mCONFIG_BLUEZ_HCIBT3C=mCONFIG_BLUEZ_HCIBLUECARD=m
# CONFIG_BLUEZ_HCIBTUART is not setCONFIG_BLUEZ_HCIVHCI=m
#
# Kernel hacking
#CONFIG_DEBUG_KERNEL=yCONFIG_DEBUG_STACKOVERFLOW=y
# CONFIG_DEBUG_HIGHMEM is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_IOVIRT is not setCONFIG_MAGIC_SYSRQ=y
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_FRAME_POINTER is not set
#
# Library routines
#CONFIG_ZLIB_INFLATE=mCONFIG_ZLIB_DEFLATE=m
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 16 lines · Krzysiek Taraszka
Quoted 15 lines · Marcelo Tosatti
Quoted 13 lines · Georg Nikodym
On Wed, 28 May 2003 21:55:39 -0300 (BRT)
Quoted 2 lines · Marcelo Tosatti
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
While others may be dubious about the efficacy of this patch, I've been running -rc6 on my laptop now since sometime last night and have seen nothing odd.
In case anybody cares, I'm using both ide and a ieee1394 (for a large external drive [which implies scsi]) and I do a _lot_ of big work with BK so I was seeing the problem within hours previously.
Quoted 5 lines
Quoted 3 lines
Great!
-rc7 will have to be released due to some problems :(
hmm, seems to ide modules and others are broken. Im looking for reason why
hmm, for IDE subsystem the ide-proc.o was't made for CONFIG_BLK_DEV_IDE=m ... anyone goes to fix it ? or shall I prepare and send here my own patch ?
Signature · 2 lines
Krzysiek Taraszka ([email protected]) http://cyborg.kernel.pl/~dzimi/
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 17 lines · Krzysiek Taraszka
Dnia czw 29. maja 2003 21:56, Krzysiek Taraszka napisa?:
Quoted 16 lines
Dnia czw 29. maja 2003 21:11, Marcelo Tosatti napisa?:
Quoted 15 lines · Georg Nikodym
Quoted 13 lines
On Wed, 28 May 2003 21:55:39 -0300 (BRT)
Quoted 2 lines · Marcelo Tosatti
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
While others may be dubious about the efficacy of this patch, I've been running -rc6 on my laptop now since sometime last night and have seen nothing odd.
In case anybody cares, I'm using both ide and a ieee1394 (for a large external drive [which implies scsi]) and I do a _lot_ of big work with BK so I was seeing the problem within hours previously.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Great!
-rc7 will have to be released due to some problems :(
hmm, seems to ide modules and others are broken. Im looking for reason why
hmm, for IDE subsystem the ide-proc.o was't made for
CONFIG_BLK_DEV_IDE=m... anyone goes to fix it ? or shall I prepare and send here my own patch ?
Feel free to send your own patch, please :)
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 22 lines · Marcelo Tosatti
Quoted 21 lines · Krzysiek Taraszka
Dnia czw 29. maja 2003 21:56, Krzysiek Taraszka napisa?:
Quoted 20 lines
Dnia czw 29. maja 2003 21:11, Marcelo Tosatti napisa?:
Quoted 18 lines · Georg Nikodym
Quoted 6 lines
On Wed, 28 May 2003 21:55:39 -0300 (BRT)
Quoted 2 lines · Marcelo Tosatti
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
Quoted 7 lines
While others may be dubious about the efficacy of this patch, I've been running -rc6 on my laptop now since sometime last night and have seen nothing odd.
In case anybody cares, I'm using both ide and a ieee1394 (for a large external drive [which implies scsi]) and I do a _lot_ of big work with BK so I was seeing the problem within hours previously.
Great!
-rc7 will have to be released due to some problems :(
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
hmm, seems to ide modules and others are broken. Im looking for reason why
hmm, for IDE subsystem the ide-proc.o was't made for
CONFIG_BLK_DEV_IDE=m... anyone goes to fix it ? or shall I prepare and send here my own patch ?Feel free to send your own patch, please :)
Hm, I send it few days ago (replay to Andrzej Krzysztofowicz post (sth with -rc3 in subject :)) with another fixes but without cmd640 fixes. Alan made almoust the same changes but him ac tree still have got broken cmd640 modular driver (cmd640_vlb still is unresolved).
I tried hack it .. but I droped it ... maybe tomorrow i back to that code ... or someone goes to fix it (maybe Alan ?)
Signature · 2 lines
Krzysiek Taraszka ([email protected]) http://cyborg.kernel.pl/~dzimi/
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 5 lines · Krzysiek Taraszka
-rc3 in subject :)) with another fixes but without cmd640 fixes. Alan made almoust the same changes but him ac tree still have got broken cmd640 modular driver (
cmd640_vlbstill is unresolved).
I tried hack it .. but I droped it ... maybe tomorrow i back to that code ... or someone goes to fix it (maybe Alan ?)
cmd640_vlb is gone from the core code in the -ac tree so that suprises me. Adrian Bunk sent me some more patches to look at. I'm not 100% convinced by them but there are a few cases left and some of his stuff certainly fixes real problems
Re: -rc7 Re: Linux 2.4.21-rc6
In reply to Marcelo TosattiQuoted 14 lines · Marcelo Tosatti
Quoted 10 lines · Georg Nikodym
Quoted 2 lines · On Wed, 28 May 2003 21:55:39 -0300 (BRT) Marcelo Tosatti…
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
While others may be dubious about the efficacy of this patch, I've been running -rc6 on my laptop now since sometime last night and have seen nothing odd.
Quoted 5 lines
Quoted 3 lines
In case anybody cares, I'm using both ide and a ieee1394 (for a large external drive [which implies scsi]) and I do a _lot_ of big work with BK so I was seeing the problem within hours previously.
Great!
are you really sure that it is the right fix?
I mean, the batching has a basic problem (I was discussing it with Jens two days ago and he said he's already addressed in 2.5, I wonder if that could also have an influence on the fact 2.5 is so much better in fariness)
the issue with batching in 2.4, is that it is blocking at 0 and waking at batch_requests. But it's not blocking new get_request to eat requests in the way back from 0 to batch_requests. I mean, there are two directions, when we move from batch_requests to 0 get_requests should return requests. in the way back from 0 to batch_requests the get_request should block (and it doesn't in 2.4, that is the problem)
Quoted 7 lines
(quoted diff) +0
-2-rc7 will have to be released due to some problems :( -To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Andrea
Re: -rc7 Re: Linux 2.4.21-rc6
On Wednesday 04 June 2003 12:22, Andrea Arcangeli wrote:
Hi Andrea,
Quoted 11 lines
are you really sure that it is the right fix?
I mean, the batching has a basic problem (I was discussing it with Jens two days ago and he said he's already addressed in 2.5, I wonder if that could also have an influence on the fact 2.5 is so much better in fariness)
the issue with batching in 2.4, is that it is blocking at 0 and waking atbatch_requests. But it's not blocking newget_requestto eat requests in the way back from 0 tobatch_requests. I mean, there are two directions, when we move frombatch_requeststo 0get_requestsshould return requests. in the way back from 0 tobatch_requeststheget_requestshould block (and it doesn't in 2.4, that is the problem)
do you see a chance to fix this up in 2.4?
ciao, Marc
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 15 lines · Marc-Christian Petersen
On Wednesday 04 June 2003 12:22, Andrea Arcangeli wrote:
Hi Andrea,
Quoted 11 lines
are you really sure that it is the right fix?
I mean, the batching has a basic problem (I was discussing it with Jens two days ago and he said he's already addressed in 2.5, I wonder if that could also have an influence on the fact 2.5 is so much better in fariness)
the issue with batching in 2.4, is that it is blocking at 0 and waking atbatch_requests. But it's not blocking newget_requestto eat requests in the way back from 0 tobatch_requests. I mean, there are two directions, when we move frombatch_requeststo 0get_requestsshould return requests. in the way back from 0 tobatch_requeststheget_requestshould block (and it doesn't in 2.4, that is the problem)
Quoted 1 line
do you see a chance to fix this up in 2.4?
Nick posted a patch to do so the other day and asked people to test.
Signature · 1 line
Jens Axboe
Re: -rc7 Re: Linux 2.4.21-rc6
On Wednesday 04 June 2003 12:42, Jens Axboe wrote:
Hi Jens,
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
the issue with batching in 2.4, is that it is blocking at 0 and waking at
batch_requests. But it's not blocking newget_requestto eat requests in the way back from 0 tobatch_requests. I mean, there are two directions, when we move frombatch_requeststo 0get_requestsshould return requests. in the way back from 0 tobatch_requeststheget_requestshould block (and it doesn't in 2.4, that is the problem)do you see a chance to fix this up in 2.4?
Nick posted a patch to do so the other day and asked people to test.
Silly mcp. His mail was CC'ed to me :( ... F*ck huge inbox.
ciao, Marc
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 10 lines · Marc-Christian Petersen
On Wednesday 04 June 2003 12:42, Jens Axboe wrote:
Hi Jens,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
the issue with batching in 2.4, is that it is blocking at 0 and waking at
batch_requests. But it's not blocking newget_requestto eat requests in the way back from 0 tobatch_requests. I mean, there are two directions, when we move frombatch_requeststo 0get_requestsshould return requests. in the way back from 0 tobatch_requeststheget_requestshould block (and it doesn't in 2.4, that is the problem)
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
do you see a chance to fix this up in 2.4?
Nick posted a patch to do so the other day and asked people to test.
Silly mcp. His mail was CC'ed to me :( ... F*ck huge inbox.
I was probably not CC'ed, I'll search for the email (and I was travelling the last few days so I didn't read every single l-k email yet sorry ;)
Andrea
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 16 lines · Andrea Arcangeli
Quoted 14 lines · Marc-Christian Petersen
On Wednesday 04 June 2003 12:42, Jens Axboe wrote:
Hi Jens,
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
the issue with batching in 2.4, is that it is blocking at 0 and waking at
batch_requests. But it's not blocking newget_requestto eat requests in the way back from 0 tobatch_requests. I mean, there are two directions, when we move frombatch_requeststo 0get_requestsshould return requests. in the way back from 0 tobatch_requeststheget_requestshould block (and it doesn't in 2.4, that is the problem)do you see a chance to fix this up in 2.4?
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
Nick posted a patch to do so the other day and asked people to test.
Silly mcp. His mail was CC'ed to me :( ... F*ck huge inbox.
I was probably not CC'ed, I'll search for the email (and I was travelling the last few days so I didn't read every single l-k email yet sorry ;)
The patch I sent is actually against 2.4.20, contrary to my babling. Reports I have had say it helps, but maybe not so much as Andrew'ss fixes. Then Matthias Mueller ported my patch to 2.4.21-rc6 which include Andrew's fixes.
It seems that they might be fixing two different problems. It looks promising though.
My patch would not affect read IO throughput for a smallish number of readers because the queue should never fill up.
Quoted 1 line
1 writer or a lot of readers could see some throughput
drop due to the patch causing the queue to be more FIFO at high loads.
I have attached the patch again. Its against 2.4.20.
Nick
Quoted 3 lines · Matthias Mueller
Currently I'm running 2.4.21-rc6 with your patch and the patch from Andrew and it looks very promising. Both patches seem to address two different problems, combined I can have 2 parallel dds running and play music with xmms and notice no sounddrops (actually i had one, but that was during very high cpu load). Your patch seems to lower IO-throughput, but I haven't tested this, so no real numbers, just my personal feelings and the numbers 'time dd ...' gave me.
include/linux/blkdev.h renamed from include/linux/blkdev.h.orig +34 -2
@@ -118,13 +118,21 @@ struct request_queue/* * Boolean that indicates whether this queue is plugged or not. */ -char plugged; + int plugged:1;/* * Boolean that indicates whether current_request is active or * not. */ -char head_active; + int head_active:1; + +/* +* Booleans that indicate whether the queue's free requests have +* been exhausted and is waiting to drop below the batch_requests +* threshold +*/ + int read_full:1; + int write_full:1;unsigned long bounce_pfn;
@@ -140,6 +148,30 @@ struct request_queuewait_queue_head_t wait_for_requests[2 ]; }; +static inline void set_queue_full (request_queue_t *q,int rw) +{ +if (rw ==READ ) + q->read_full =1 ; +else + q->write_full =1 ; +} + +static inline void clear_queue_full (request_queue_t *q,int rw) +{ +if (rw ==READ ) + q->read_full =0 ; +else + q->write_full =0 ; +} + +static inline int queue_full (request_queue_t *q,int rw) +{ +if (rw ==READ ) +return q->read_full; +else +return q->write_full; +} +extern unsigned long blk_max_low_pfn, blk_max_pfn;#define BLK_BOUNCE_HIGH (blk_max_low_pfn << PAGE_SHIFT)
drivers/block/ll_rw_blk.c renamed from drivers/block/ll_rw_blk.c.orig +13 -5
@@ -513,7 +513,10 @@ static struct request *get_request(requestruct request *rq =NULL ;struct request_list *rl = q->rq + rw; -if (!list_empty (&rl->free)) { +if (list_empty (&rl->free)) +set_queue_full (q, rw); + +if (!queue_full (q, rw)) { rq =blkdev_free_rq (&rl->free);list_del (&rq->queue); rl->count--;
@@ -594,7 +597,7 @@ static struct request *__get_request_waiadd_wait_queue_exclusive (&q->wait_for_requests[rw], &wait); do { set_current_state(TASK_UNINTERRUPTIBLE); - if (q->rq[rw].count == 0) + if (queue_full(q, rw)) schedule(); spin_lock_irq(&io_request_lock); rq = get_request(q, rw);
@@ -829,9 +832,14 @@ void blkdev_release_request(struct reque*/ if (q) {list_add (&req->queue, &q->rq[rw].free); -if (++q->rq[rw].count >= q->batch_requests && -waitqueue_active (&q->wait_for_requests[rw])) -wake_up (&q->wait_for_requests[rw]); + q->rq[rw].count++; +if (q->rq[rw].count >= q->batch_requests) { +if (q->rq[rw].count == q->batch_requests) +clear_queue_full (q, rw); + +if (waitqueue_active (&q->wait_for_requests[rw])) +wake_up (&q->wait_for_requests[rw]); + } } }
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 28 lines · Nick Piggin
Quoted 16 lines · Andrea Arcangeli
Quoted 14 lines · Marc-Christian Petersen
On Wednesday 04 June 2003 12:42, Jens Axboe wrote:
Hi Jens,
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
the issue with batching in 2.4, is that it is blocking at 0 and waking at
batch_requests. But it's not blocking newget_requestto eat requests in the way back from 0 tobatch_requests. I mean, there are two directions, when we move frombatch_requeststo 0get_requestsshould return requests. in the way back from 0 tobatch_requeststheget_requestshould block (and it doesn't in 2.4, that is the problem)do you see a chance to fix this up in 2.4?
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
Nick posted a patch to do so the other day and asked people to test.
Silly mcp. His mail was CC'ed to me :( ... F*ck huge inbox.
I was probably not CC'ed, I'll search for the email (and I was travelling the last few days so I didn't read every single l-k email yet sorry ;)
Quoted 7 lines
The patch I sent is actually against 2.4.20, contrary to my babling. Reports I have had say it helps, but maybe not so much as Andrew'ss fixes. Then Matthias Mueller ported my patch to 2.4.21-rc6 which include Andrew's fixes.
It seems that they might be fixing two different problems. It looks promising though.
It is a different problem I think, yours will fix the starvation of writers (of readers, writers is much much easier to trigger though) where someone will repeatedly get cheaten by the request allocator.
The other problem is still not clear to anyone. I doubt this patch would make any difference (apart from a psychological one) in this case, since you have a single writer and maybe a reader or two. The single writer cannot starve anyone else.
Signature · 1 line
Jens Axboe
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 2 lines · Jens Axboe
since you have a single writer and maybe a reader or two. The single writer cannot starve anyone else.
unless you're changing an atime and you've to mark_buffer_dirty or similar (balance_dirty will write stuff the same way from cp and the reader then).
Maybe we can get some stack trace with kgdb to be sure where the reader is blocking.
Andrea
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 7 lines · Andrea Arcangeli
Quoted 2 lines · Jens Axboe
since you have a single writer and maybe a reader or two. The single writer cannot starve anyone else.
unless you're changing an atime and you've to
mark_buffer_dirtyor similar (balance_dirtywill write stuff the same way from cp and the reader then).
Yes you are right, could be.
But the whole thing still smells fishy. Read starvation causing mouse stalls, hmm.
Signature · 1 line
Jens Axboe
Re: -rc7 Re: Linux 2.4.21-rc6
Quoted 5 lines · Jens Axboe
Quoted 4 lines · Andrea Arcangeli
Quoted 2 lines · Jens Axboe
since you have a single writer and maybe a reader or two. The single writer cannot starve anyone else.
Quoted 8 lines
Quoted 3 lines
unless you're changing an atime and you've to
mark_buffer_dirtyor similar (balance_dirtywill write stuff the same way from cp and the reader then).Yes you are right, could be.
But the whole thing still smells fishy. Read starvation causing mouse stalls, hmm.
If reads from swap get starved, you can have interactive dropouts in just about anything.
My desktop is usually pretty deep into swap. I upgrade to machines with four times as much memory, but that usually means the graphics resolution went up and it just lets me keep more windows open in more desktops. (Currently six.)
My record was driving the system so deep into swapping frenzy it was still swapping when I came back from lunch. Really. This was under 2.4.4, though. On RH 9/2.4.20-? my record is a little under five minutes of "frozen thrashing on swap" before I got control of the system back. That's just a "go for a soda" break. And at least the mouse cursor never froze for more than a couple seconds at a time during that, even if the desktop was ignoring me... :)
Haven't tried 2.5 on anything but servers yet, but it's on my to-do list...
Rob
(I am the VM subsystem's worst nightmare. Bwahaha.)
Script started on Wed 04 Jun 2003 04:25:29 PM EDT ]0;landley@localhost:~[landley@localhost landley]$ cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 261390336 247234560 14155776 0 9351168 80461824 Swap: 542859264 276152320 266706944 MemTotal: 255264 kB MemFree: 13824 kB MemShared: 0 kB Buffers: 9132 kB Cached: 43372 kB SwapCached: 35204 kB Active: 182324 kB ActiveAnon: 131940 kB ActiveCache: 50384 kB Inact_dirty: 19164 kB Inact_laundry: 14400 kB Inact_clean: 3512 kB Inact_target: 43880 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 255264 kB LowFree: 13824 kB SwapTotal: 530136 kB SwapFree: 260456 kB ]0;landley@localhost:~[landley@localhost landley]$ cat /proc/slabinfo slabinfo - version: 1.1 kmem_cache 65 70 108 2 2 1 ip_fib_hash 11 112 32 1 1 1 urb_priv 0 0 64 0 0 1 journal_head 57 770 48 1 10 1 revoke_table 2 250 12 1 1 1 revoke_record 0 112 32 0 1 1 clip_arp_cache 0 0 128 0 0 1 ip_mrt_cache 0 0 128 0 0 1 tcp_tw_bucket 0 90 128 0 3 1 tcp_bind_bucket 4 224 32 1 2 1 tcp_open_request 0 30 128 0 1 1 inet_peer_cache 0 58 64 0 1 1 ip_dst_cache 5 75 256 1 5 1 arp_cache 2 30 128 1 1 1 blkdev_requests 256 270 128 9 9 1 dnotify_cache 0 0 20 0 0 1 file_lock_cache 0 41 92 0 1 1 fasync_cache 2 200 16 1 1 1 uid_cache 2 112 32 1 1 1 skbuff_head_cache 176 2265 256 32 151 1 sock 589 720 1280 220 240 1 sigqueue 0 29 132 0 1 1 kiobuf 0 0 64 0 0 1 cdev_cache 26 232 64 2 4 1 bdev_cache 4 58 64 1 1 1 mnt_cache 13 58 64 1 1 1 inode_cache 2395 3647 512 519 521 1 dentry_cache 2477 4050 128 135 135 1 dquot 0 0 128 0 0 1 filp 2364 2370 128 79 79 1 names_cache 0 14 4096 0 14 1 buffer_head 16649 30360 128 789 1012 1 mm_struct 173 210 256 14 14 1 vm_area_struct 5840 7770 128 238 259 1 fs_cache 78 116 64 2 2 1 files_cache 78 112 512 15 16 1 signal_cache 243 290 64 5 5 1 sighand_cache 235 253 1408 22 23 4 task_struct 0 0 1792 0 0 1 pte_chain 1958 7590 128 83 253 1 size-131072(DMA) 0 0 131072 0 0 32 size-131072 0 0 131072 0 0 32 size-65536(DMA) 0 0 65536 0 0 16 size-65536 0 0 65536 0 0 16 size-32768(DMA) 0 0 32768 0 0 8 size-32768 0 0 32768 0 0 8 size-16384(DMA) 0 0 16384 0 0 4 size-16384 0 16 16384 0 16 4 size-8192(DMA) 0 0 8192 0 0 2 size-8192 4 19 8192 4 19 2 size-4096(DMA) 0 0 4096 0 0 1 size-4096 35 75 4096 35 75 1 size-2048(DMA) 0 0 2048 0 0 1 size-2048 8 86 2048 5 43 1 size-1024(DMA) 0 0 1024 0 0 1 size-1024 59 124 1024 18 31 1 size-512(DMA) 0 0 512 0 0 1 size-512 43 200 512 11 25 1 size-256(DMA) 0 0 256 0 0 1 size-256 43 1200 256 8 80 1 size-128(DMA) 1 30 128 1 1 1 size-128 707 3240 128 33 108 1 size-64(DMA) 0 0 128 0 0 1 size-64 377 1170 128 30 39 1 size-32(DMA) 17 58 64 1 1 1 size-32 397 754 64 10 13 1
]0;landley@localhost:~[landley@localhost landley]$ Script done on Wed 04 Jun 2003 04:25:42 PM EDT
Re: -rc7 Re: Linux 2.4.21-rc6
In reply to Jens AxboeQuoted 46 lines · Jens Axboe
Quoted 35 lines · Nick Piggin
Quoted 23 lines · Andrea Arcangeli
Quoted 20 lines · Marc-Christian Petersen
On Wednesday 04 June 2003 12:42, Jens Axboe wrote:
Hi Jens,
Quoted 14 lines
Quoted 11 lines
Quoted 8 lines
the issue with batching in 2.4, is that it is blocking at 0 and waking at
batch_requests. But it's not blocking newget_requestto eat requests in the way back from 0 tobatch_requests. I mean, there are two directions, when we move frombatch_requeststo 0get_requestsshould return requests. in the way back from 0 tobatch_requeststheget_requestshould block (and it doesn't in 2.4, that is the problem)do you see a chance to fix this up in 2.4?
Nick posted a patch to do so the other day and asked people to test.
Quoted 9 lines
Quoted 3 lines
Silly mcp. His mail was CC'ed to me :( ... F*ck huge inbox.
I was probably not CC'ed, I'll search for the email (and I was travelling the last few days so I didn't read every single l-k email yet sorry ;)
Quoted 8 lines
The patch I sent is actually against 2.4.20, contrary to my babling. Reports I have had say it helps, but maybe not so much as Andrew'ss fixes. Then Matthias Mueller ported my patch to 2.4.21-rc6 which include Andrew's fixes.
It seems that they might be fixing two different problems. It looks promising though.
Quoted 9 lines
It is a different problem I think, yours will fix the starvation of writers (of readers, writers is much much easier to trigger though) where someone will repeatedly get cheaten by the request allocator.
The other problem is still not clear to anyone. I doubt this patch would make any difference (apart from a psychological one) in this case, since you have a single writer and maybe a reader or two. The single writer cannot starve anyone else.
You are right about what the patch does. It wouldn't surprise me if there are still other problems, but it could be that the reader has to write some swap or other dirty buffers when trying to get memory itself.
I have had 3 or so reports all saying similar things, but it could be psychological I guess.
Re: -rc7 Re: Linux 2.4.21-rc6
In reply to Nick PigginIn article <[email protected]>,
include/linux/blkdev.h struct request_queue · line 121
Quoted 2 lines · Nick Piggin
include/linux/blkdev.h renamed from include/linux/blkdev.h.orig +1
-1-char plugged; + int plugged:1;
This is dangerous:
struct foo {
int bla:1;
};int main()
{
struct foo f;
f.bla = 1;
printf("%d\n", f.bla);
}$ ./a.out -1
If you want to put "0" and "1" in a 1-bit field, use "unsigned int bla:1".
Mike.
Signature · 2 lines
.. somehow I have a feeling the hurting hasn't even begun yet -- Bill, "The Terrible Thunderlizards"
PATCH 1/1 io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
In reply to Nick PigginOk, there are lots of different problems here, and I've spent a little while trying to get some numbers with the __get_request_wait stats patch I posted before. This is all on ext2, since I wanted to rule out interactions with the journal flavors.
Basically a dbench 90 run on ext2 rc6 vanilla kernels can generate latencies of over 2700 jiffies in __get_request_wait, with an average latency over 250 jiffies.
No, most desktop workloads aren't dbench 90, but between balance_dirty() and the way we send stuff to disk during memory allocations, just about any process can get stuck submitting dirty buffers even if you've just got one process doing a dd if=/dev/zero of=foo.
So, for the moment I'm going to pretend people seeing stalls in X are stuck in atime updates or memory allocations, or reading proc or some other silly spot.
For the SMP corner cases, I've merged Andrea's fix-pausing patch into rc7, along with an altered form of Nick Piggin's queue_full patch to try and fix the latency problems.
The major difference from Nick's patch is that once the queue is marked full, I don't clear the full flag until the wait queue is empty. This means new io can't steal available requests until every existing waiter has been granted a request.
The latency results are better, with average time spent in __get_request_wait being around 28 jiffies, and a max of 170 jiffies. The cost is throughput, further benchmarking needs to be done but, but I wanted to get this out for review and testing. It should at least help us decide if the request allocation code really is causing our problems.
The patch below also includes the __get_request_wait latency stats. If people try this and still see stalls, please run elvtune /dev/xxxx and send along the resulting console output.
I haven't yet compared this to Andrea's elevator latency code, but the stat patch was originally developed on top of his 2.4.21pre3aa1, where the average wait was 97 jiffies and the max was 318.
Anyway, less talk, more code. Treat this with care, it has only been lightly tested. Thanks to Andrea and Nick whose patches this is largely based on:
drivers/block/blkpg.c +1 -0
@@ -261,6 +261,7 @@return blkpg_ioctl (dev, (struct blkpg_ioctl_arg *) arg);case BLKELVGET : +blk_print_stats (dev);return blkelvget_ioctl (&blk_get_queue (dev)->elevator, (blkelv_ioctl_arg_t *) arg);case BLKELVSET :
drivers/block/ll_rw_blk.c +123 -10
222 lines not on this page · raw message
fs/buffer.c +21 -1
@@ -153,10 +153,23 @@get_bh (bh);add_wait_queue (&bh->b_wait, &wait);do { -run_task_queue (&tq_disk);set_task_state (tsk,TASK_UNINTERRUPTIBLE );if (!buffer_locked (bh))break ; +/* +* We must read tq_disk in TQ_ACTIVE after the +* add_wait_queue effect is visible to other cpus. +* We could unplug some line above it wouldn't matter +* but we can't do that right after add_wait_queue +* without an smp_mb() in between because spin_unlock +* has inclusive semantics. +* Doing it here is the most efficient place so we +* don't do a suprious unplug if we get a racy +* wakeup that make buffer_locked to return 0, and +* doing it here avoids an explicit smp_mb() we +* rely on the implicit one in set_task_state. +*/ +run_task_queue (&tq_disk);schedule (); }while (buffer_locked (bh)); tsk->state =TASK_RUNNING ;
@@ -1507,6 +1520,9 @@/* Done - end_buffer_io_async will unlock */ SetPageUptodate (page); + +wakeup_page_waiters (page); +return 0 ; out:
@@ -1538,6 +1554,7 @@ }while (bh != head);if (need_unlock)UnlockPage (page); +wakeup_page_waiters (page);return err; }
@@ -1765,6 +1782,8 @@else submit_bh (READ , bh); } + +wakeup_page_waiters (page);return 0 ; }
@@ -2378,6 +2397,7 @@submit_bh (rw, bh); bh = next; }while (bh != head); +wakeup_page_waiters (page);return 0 ; }
fs/super.c +1 -0
@@ -726,6 +726,7 @@if (!fs_type->read_super (s, data, flags &MS_VERBOSE ?1 :0 )) goto Einval; s->s_flags |=MS_ACTIVE ; +blk_reset_stats (dev);path_release (&nd);return s;
fs/reiserfs/inode.c +1 -0
@@ -2048,6 +2048,7 @@*/ if (nr) {submit_bh_for_writepage (arr, nr) ; +wakeup_page_waiters (page); }else {UnlockPage (page) ; }
include/linux/blkdev.h +45 -0
71 lines not on this page · raw message
include/linux/pagemap.h +2 -0
@@ -97,6 +97,8 @@___wait_on_page (page); } +extern void FASTCALL (wakeup_page_waiters (struct page * page)); +/* * Returns locked page at given index in given cache, creating it if needed. */
kernel/ksyms.c +1 -0
@@ -295,6 +295,7 @@EXPORT_SYMBOL (filemap_fdatawait);EXPORT_SYMBOL (lock_page);EXPORT_SYMBOL (unlock_page); +EXPORT_SYMBOL (wakeup_page_waiters);/* device registration */ EXPORT_SYMBOL (register_chrdev);
mm/filemap.c +14 -0
@@ -812,6 +812,20 @@return &wait[hash]; } +/* +* This must be called after every submit_bh with end_io +* callbacks that would result into the blkdev layer waking +* up the page after a queue unplug. +*/ +void wakeup_page_waiters (struct page * page) +{ +wait_queue_head_t * head; + + head =page_waitqueue (page); +if (waitqueue_active (head)) +wake_up (head); +} +/* * Wait for a page to get unlocked. *
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
Hi,
Quoted 37 lines · Chris Mason
Ok, there are lots of different problems here, and I've spent a little while trying to get some numbers with the
__get_request_waitstats patch I posted before. This is all on ext2, since I wanted to rule out interactions with the journal flavors.Basically a dbench 90 run on ext2 rc6 vanilla kernels can generate latencies of over 2700 jiffies in
__get_request_wait, with an average latency over 250 jiffies.No, most desktop workloads aren't dbench 90, but between
balance_dirty()and the way we send stuff to disk during memory allocations, just about any process can get stuck submitting dirty buffers even if you've just got one process doing a dd if=/dev/zero of=foo.So, for the moment I'm going to pretend people seeing stalls in X are stuck in atime updates or memory allocations, or reading proc or some other silly spot.
For the SMP corner cases, I've merged Andrea's fix-pausing patch into rc7, along with an altered form of Nick Piggin's
queue_fullpatch to try and fix the latency problems.The major difference from Nick's patch is that once the queue is marked full, I don't clear the full flag until the wait queue is empty. This means new io can't steal available requests until every existing waiter has been granted a request.
The latency results are better, with average time spent in
__get_request_waitbeing around 28 jiffies, and a max of 170 jiffies. The cost is throughput, further benchmarking needs to be done but, but I wanted to get this out for review and testing. It should at least help us decide if the request allocation code really is causing our problems.The patch below also includes the
__get_request_waitlatency stats. If people try this and still see stalls, please run elvtune/dev/xxxxand send along the resulting console output.
Quoted 7 lines
I haven't yet compared this to Andrea's elevator latency code, but the stat patch was originally developed on top of his 2.4.21pre3aa1, where the average wait was 97 jiffies and the max was 318.
Anyway, less talk, more code. Treat this with care, it has only been lightly tested. Thanks to Andrea and Nick whose patches this is largely based on:
I spent last Saturday working on this too. This is the status of my current patches, would be interesting to compare them. they're not very well tested yet though.
They would obsoletes the old fix-pausing and the old elevator-lowlatency (I was going to release a new tree today, but I delayed it so I fixed uml today too first [tested with skas and w/o skas]).
those backout the rc7 interactivity changes (the only one that wasn't in my tree was the add_wait_queue_exclusive, that IMHO would better stay for scalability reasons).
Of course I would be very interested to know if those two patches (or Chris's one, you also retained the exclusive wakeup) are still greatly improved by removing the _exclusive weakups and going wake-all (in theory they shouldn't).
Andrea
drivers/block/ll_rw_blk.c +44 -6
@@ -596,12 +596,20 @@ static struct request *__get_request_wai registerstruct request *rq;DECLARE_WAITQUEUE (wait, current); -add_wait_queue (&q->wait_for_requests[rw], &wait); +add_wait_queue_exclusive (&q->wait_for_requests[rw], &wait); do { set_current_state(TASK_UNINTERRUPTIBLE); - generic_unplug_device(q); - if (q->rq[rw].count == 0) + if (q->rq[rw].count == 0) { + /* + * All we care about is not to stall if any request + * is been released after we set TASK_UNINTERRUPTIBLE. + * This is the most efficient place to unplug the queue + * in case we hit the race and we can get the request + * without waiting. + */ + generic_unplug_device(q); schedule(); + } spin_lock_irq(q->queue_lock); rq = get_request(q, rw); spin_unlock_irq(q->queue_lock);
@@ -611,6 +619,17 @@ static struct request *__get_request_waireturn rq; } +static void get_request_wait_wakeup (request_queue_t *q,int rw) +{ +/* +* avoid losing an unplug if a second __get_request_wait did the +* generic_unplug_device while our __get_request_wait was running +* w/o the queue_lock held and w/ our request out of the queue. +*/ +if (waitqueue_active (&q->wait_for_requests[rw])) +wake_up (&q->wait_for_requests[rw]); +} +/* RO fail safe mechanism */ static long ro_bits[MAX_BLKDEV ][8 ];
@@ -835,8 +854,11 @@ void blkdev_release_request(struct reque*/ if (q) {list_add (&req->queue, &q->rq[rw].free); -if (++q->rq[rw].count >= q->batch_requests) -wake_up (&q->wait_for_requests[rw]); +if (++q->rq[rw].count >= q->batch_requests) { +smp_mb (); +if (waitqueue_active (&q->wait_for_requests[rw])) +wake_up (&q->wait_for_requests[rw]); + } } }
@@ -954,7 +976,6 @@ static int __make_request(request_queue_*/ max_sectors =get_max_sectors (bh->b_rdev); -again: req =NULL ; head = &q->queue_head;/*
@@ -963,6 +984,7 @@ again:*/ spin_lock_irq (q->queue_lock); +again: insert_here = head->prev;if (list_empty (head)) { q->plug_device_fn (q, bh->b_rdev);/* is atomic */
@@ -1048,6 +1070,9 @@ get_rq:if (req ==NULL ) {spin_unlock_irq (q->queue_lock); freereq =__get_request_wait (q, rw); + head = &q->queue_head; +spin_lock_irq (q->queue_lock); +get_request_wait_wakeup (q, rw); goto again; } }
@@ -1202,8 +1227,21 @@ void __submit_bh(int rw, struct buffer_h bh->b_rdev = bh->b_dev; bh->b_rsector = blocknr; +/* +* we play with the bh wait queue below, need to keep a +* reference so the buffer doesn't get freed after the +* end_io handler runs +*/ +get_bh (bh); +generic_make_request (rw, bh); +/* fix race condition with wait_on_buffer() */ +smp_mb ();/* spin_unlock may have inclusive semantics */ +if (waitqueue_active (&bh->b_wait)) +wake_up (&bh->b_wait); +put_bh (bh); +switch (rw) {case WRITE : kstat.pgpgout += count;
fs/buffer.c +24 -1
@@ -158,10 +158,23 @@ void __wait_on_buffer(struct buffer_headget_bh (bh);add_wait_queue (&bh->b_wait, &wait);do { -run_task_queue (&tq_disk);set_task_state (tsk,TASK_UNINTERRUPTIBLE );if (!buffer_locked (bh))break ; +/* +* We must read tq_disk in TQ_ACTIVE after the +* add_wait_queue effect is visible to other cpus. +* We could unplug some line above it wouldn't matter +* but we can't do that right after add_wait_queue +* without an smp_mb() in between because spin_unlock +* has inclusive semantics. +* Doing it here is the most efficient place so we +* don't do a suprious unplug if we get a racy +* wakeup that make buffer_locked to return 0, and +* doing it here avoids an explicit smp_mb() we +* rely on the implicit one in set_task_state. +*/ +run_task_queue (&tq_disk);schedule (); }while (buffer_locked (bh)); tsk->state =TASK_RUNNING ;
@@ -1471,6 +1484,7 @@ static int __block_write_full_page(strucif (!page->buffers)create_empty_buffers (page, inode->i_dev,1 << inode->i_blkbits); +BUG_ON (page_count (page) <3 ); head = page->buffers; block = page->index << (PAGE_CACHE_SHIFT - inode->i_blkbits);
@@ -1517,6 +1531,9 @@ static int __block_write_full_page(struc/* Done - end_buffer_io_async will unlock */ SetPageUptodate (page); + +wakeup_page_waiters (page); +return 0 ; out:
@@ -1548,6 +1565,7 @@ out: }while (bh != head);if (need_unlock)UnlockPage (page); +wakeup_page_waiters (page);return err; }
@@ -1721,6 +1739,7 @@ int block_read_full_page(struct page *pa blocksize =1 << inode->i_blkbits;if (!page->buffers)create_empty_buffers (page, inode->i_dev, blocksize); +BUG_ON (page_count (page) <3 ); head = page->buffers; blocks =PAGE_CACHE_SIZE >> inode->i_blkbits;
@@ -1781,6 +1800,8 @@ int block_read_full_page(struct page *paelse submit_bh (READ , bh); } + +wakeup_page_waiters (page);return 0 ; }
@@ -2400,6 +2421,7 @@ int brw_page(int rw, struct page *page,if (!page->buffers)create_empty_buffers (page, dev, size); +BUG_ON (page_count (page) <3 ); head = bh = page->buffers;/* Stage 1: lock all the buffers */
@@ -2417,6 +2439,7 @@ int brw_page(int rw, struct page *page,submit_bh (rw, bh); bh = next; }while (bh != head); +wakeup_page_waiters (page);return 0 ; }
fs/reiserfs/inode.c +1 -0
@@ -2048,6 +2048,7 @@ static int reiserfs_write_full_page(stru*/ if (nr) {submit_bh_for_writepage (arr, nr) ; +wakeup_page_waiters (page); }else {UnlockPage (page) ; }
include/linux/pagemap.h +2 -0
@@ -98,6 +98,8 @@ static inline void wait_on_page(struct p___wait_on_page (page); } +extern void FASTCALL (wakeup_page_waiters (struct page * page)); +/* * Returns locked page at given index in given cache, creating it if needed. */
kernel/ksyms.c +1 -0
@@ -319,6 +319,7 @@ EXPORT_SYMBOL(filemap_fdatasync);EXPORT_SYMBOL (filemap_fdatawait);EXPORT_SYMBOL (lock_page);EXPORT_SYMBOL (unlock_page); +EXPORT_SYMBOL (wakeup_page_waiters);/* device registration */ EXPORT_SYMBOL (register_chrdev);
mm/filemap.c +14 -0
@@ -779,6 +779,20 @@ inline wait_queue_head_t * page_waitqueureturn wait_table_hashfn (page, &pgdat->wait_table); } +/* +* This must be called after every submit_bh with end_io +* callbacks that would result into the blkdev layer waking +* up the page after a queue unplug. +*/ +void wakeup_page_waiters (struct page * page) +{ +wait_queue_head_t * head; + + head =page_waitqueue (page); +if (waitqueue_active (head)) +wake_up (head); +} +/* * Wait for a page to get unlocked. *
mm/swapfile.c +2 -0
@@ -984,8 +984,10 @@ asmlinkage long sys_swapon(const char * goto bad_swap; } +get_page (virt_to_page (swap_header));lock_page (virt_to_page (swap_header));rw_swap_page_nolock (READ ,SWP_ENTRY (type,0 ), (char *) swap_header); +put_page (virt_to_page (swap_header));if (!memcmp ("SWAP-SPACE" ,swap_header->magic.magic,10 )) swap_header_version =1 ;
Binary files x-ref/ID and x/ID differ
drivers/block/DAC960.c +21 -13
@@ -19,8 +19,8 @@*/ -#define DAC960_DriverVersion "2.4.11" -#define DAC960_DriverDate "11 October 2001" +#define DAC960_DriverVersion "2.4.20aa1" +#define DAC960_DriverDate "4 December 2002"#include <linux/version.h>
@@ -2975,8 +2975,9 @@ static boolean DAC960_ProcessRequest(DAC Command->SegmentCount = Request->nr_segments; Command->BufferHeader = Request->bh; Command->RequestBuffer = Request->buffer; + Command->Request = Request;blkdev_dequeue_request (Request); -blkdev_release_request (Request); +/* blkdev_release_request(Request); */ DAC960_QueueReadWriteCommand (Command);return true; }
@@ -3023,11 +3024,12 @@ static void DAC960_RequestFunction(Requeindividual Buffer. */ -static inline void DAC960_ProcessCompletedBuffer (BufferHeader_T *BufferHeader, +static inline void DAC960_ProcessCompletedBuffer (IO_Request_T *Req,BufferHeader_T *BufferHeader,boolean SuccessfulIO) { -blk_finished_io (BufferHeader->b_size >>9 ); +blk_finished_io (Req, BufferHeader->b_size >>9 ); BufferHeader->b_end_io (BufferHeader, SuccessfulIO); + }
@@ -3116,9 +3118,10 @@ static void DAC960_V1_ProcessCompletedCo {BufferHeader_T *NextBufferHeader = BufferHeader->b_reqnext; BufferHeader->b_reqnext =NULL ; -DAC960_ProcessCompletedBuffer (BufferHeader, true); +DAC960_ProcessCompletedBuffer (Command->Request, BufferHeader, true); BufferHeader = NextBufferHeader; } +blkdev_release_request (Command->Request);if (Command->Completion !=NULL ) {complete (Command->Completion);
@@ -3161,7 +3164,7 @@ static void DAC960_V1_ProcessCompletedCo {BufferHeader_T *NextBufferHeader = BufferHeader->b_reqnext; BufferHeader->b_reqnext =NULL ; -DAC960_ProcessCompletedBuffer (BufferHeader, false); +DAC960_ProcessCompletedBuffer (Command->Request, BufferHeader, false); BufferHeader = NextBufferHeader; }if (Command->Completion !=NULL )
@@ -3169,6 +3172,7 @@ static void DAC960_V1_ProcessCompletedCocomplete (Command->Completion); Command->Completion =NULL ; } +blkdev_release_request (Command->Request); } } else if (CommandType == DAC960_ReadRetryCommand ||
@@ -3180,12 +3184,12 @@ static void DAC960_V1_ProcessCompletedCoPerform completion processing for this single buffer. */ if (CommandStatus == DAC960_V1_NormalCompletion) -DAC960_ProcessCompletedBuffer (BufferHeader, true); +DAC960_ProcessCompletedBuffer (Command->Request, BufferHeader, true);else {if (CommandStatus != DAC960_V1_LogicalDriveNonexistentOrOffline)DAC960_V1_ReadWriteError (Command); -DAC960_ProcessCompletedBuffer (BufferHeader, false); +DAC960_ProcessCompletedBuffer (Command->Request, BufferHeader, false); }if (NextBufferHeader !=NULL ) {
@@ -3203,6 +3207,7 @@ static void DAC960_V1_ProcessCompletedCoDAC960_QueueCommand (Command);return ; } +blkdev_release_request (Command->Request); } else if (CommandType == DAC960_MonitoringCommand || CommandOpcode == DAC960_V1_Enquiry ||
@@ -4222,9 +4227,10 @@ static void DAC960_V2_ProcessCompletedCo {BufferHeader_T *NextBufferHeader = BufferHeader->b_reqnext; BufferHeader->b_reqnext =NULL ; -DAC960_ProcessCompletedBuffer (BufferHeader, true); +DAC960_ProcessCompletedBuffer (Command->Request, BufferHeader, true); BufferHeader = NextBufferHeader; } +blkdev_release_request (Command->Request);if (Command->Completion !=NULL ) {complete (Command->Completion);
@@ -4267,9 +4273,10 @@ static void DAC960_V2_ProcessCompletedCo {BufferHeader_T *NextBufferHeader = BufferHeader->b_reqnext; BufferHeader->b_reqnext =NULL ; -DAC960_ProcessCompletedBuffer (BufferHeader, false); +DAC960_ProcessCompletedBuffer (Command->Request, BufferHeader, false); BufferHeader = NextBufferHeader; } +blkdev_release_request (Command->Request);if (Command->Completion !=NULL ) {complete (Command->Completion);
@@ -4286,12 +4293,12 @@ static void DAC960_V2_ProcessCompletedCoPerform completion processing for this single buffer. */ if (CommandStatus == DAC960_V2_NormalCompletion) -DAC960_ProcessCompletedBuffer (BufferHeader, true); +DAC960_ProcessCompletedBuffer (Command->Request, BufferHeader, true);else {if (Command->V2.RequestSense.SenseKey != DAC960_SenseKey_NotReady)DAC960_V2_ReadWriteError (Command); -DAC960_ProcessCompletedBuffer (BufferHeader, false); +DAC960_ProcessCompletedBuffer (Command->Request, BufferHeader, false); }if (NextBufferHeader !=NULL ) {
@@ -4319,6 +4326,7 @@ static void DAC960_V2_ProcessCompletedCoDAC960_QueueCommand (Command);return ; } +blkdev_release_request (Command->Request); }else if (CommandType == DAC960_MonitoringCommand) {
drivers/block/DAC960.h +1 -8
@@ -2282,6 +2282,7 @@ typedef struct DAC960_Commandunsigned int SegmentCount;BufferHeader_T *BufferHeader;void *RequestBuffer; +IO_Request_T *Request; union { struct { DAC960_V1_CommandMailbox_T CommandMailbox;
@@ -4265,12 +4266,4 @@ static void DAC960_Message(DAC960_Messagstatic void DAC960_CreateProcEntries (void );static void DAC960_DestroyProcEntries (void ); - -/* -Export the Kernel Mode IOCTL interface. -*/ - -EXPORT_SYMBOL (DAC960_KernelIOCTL); - -#endif /* DAC960_DriverVersion */
drivers/block/cciss.c +4 -4
@@ -1990,14 +1990,14 @@ static void start_io( ctlr_info_t *h) } } -static inline void complete_buffers (struct buffer_head *bh,int status) +static inline void complete_buffers (struct request * req,struct buffer_head *bh,int status) {struct buffer_head *xbh;while (bh) { xbh = bh->b_reqnext; bh->b_reqnext =NULL ; -blk_finished_io (bh->b_size >>9 ); +blk_finished_io (req, bh->b_size >>9 ); bh->b_end_io (bh, status); bh = xbh; }
@@ -2140,7 +2140,7 @@ static inline void complete_command( ctlpci_unmap_page (hba[cmd->ctlr]->pdev, temp64.val, cmd->SG[i].Len, ddir); } -complete_buffers (cmd->rq->bh, status); +complete_buffers (cmd->rq, cmd->rq->bh, status);#ifdef CCISS_DEBUG printk ("Done with %p\n" , cmd->rq);#endif /* CCISS_DEBUG */
@@ -2224,7 +2224,7 @@ next:printk (KERN_WARNING "doreq cmd for %d, %x at %p\n" , h->ctlr, creq->rq_dev, creq);blkdev_dequeue_request (creq); -complete_buffers (creq->bh,0 ); +complete_buffers (creq, creq->bh,0 );end_that_request_last (creq); goto startio; }
drivers/block/cpqarray.c +5 -5
@@ -169,7 +169,7 @@ static void start_io(ctlr_info_t *h);static inline void addQ (cmdlist_t **Qptr,cmdlist_t *c);static inline cmdlist_t *removeQ (cmdlist_t **Qptr,cmdlist_t *c); -static inline void complete_buffers (struct buffer_head *bh,int ok); +static inline void complete_buffers (struct request * req,struct buffer_head *bh,int ok);static inline void complete_command (cmdlist_t *cmd,int timeout);static void do_ida_intr (int irq,void *dev_id,struct pt_regs * regs);
@@ -981,7 +981,7 @@ next:printk (KERN_WARNING "doreq cmd for %d, %x at %p\n" , h->ctlr, creq->rq_dev, creq);blkdev_dequeue_request (creq); -complete_buffers (creq->bh,0 ); +complete_buffers (creq, creq->bh,0 );end_that_request_last (creq); goto startio; }
@@ -1082,14 +1082,14 @@ static void start_io(ctlr_info_t *h) } } -static inline void complete_buffers (struct buffer_head *bh,int ok) +static inline void complete_buffers (struct request * req,struct buffer_head *bh,int ok) {struct buffer_head *xbh;while (bh) { xbh = bh->b_reqnext; bh->b_reqnext =NULL ; -blk_finished_io (bh->b_size >>9 ); +blk_finished_io (req, bh->b_size >>9 ); bh->b_end_io (bh, ok); bh = xbh;
@@ -1131,7 +1131,7 @@ static inline void complete_command(cmdl (cmd->req.hdr.cmd == IDA_READ) ? PCI_DMA_FROMDEVICE : PCI_DMA_TODEVICE); } -complete_buffers (cmd->rq->bh, ok); +complete_buffers (cmd->rq, cmd->rq->bh, ok); DBGPX(printk("Done with %p\n", cmd->rq););req_finished_io (cmd->rq);end_that_request_last (cmd->rq);
drivers/block/ll_rw_blk.c +76 -36
241 lines not on this page · raw message
drivers/scsi/scsi_lib.c +1 -1
@@ -384,7 +384,7 @@ static Scsi_Cmnd *__scsi_end_request(Scs do { if ((bh = req->bh) != NULL) { nsect = bh->b_size >> 9; - blk_finished_io(nsect); + blk_finished_io(req, nsect); req->bh = bh->b_reqnext; bh->b_reqnext = NULL; sectors -= nsect;
include/linux/blkdev.h +63 -13
129 lines not on this page · raw message
include/linux/elevator.h +3 -3
@@ -80,7 +80,7 @@ static inline int elevator_request_latenreturn latency; } -#define ELV_LINUS_SEEK_COST 16 +#define ELV_LINUS_SEEK_COST 1#define ELEVATOR_NOOP \ ((elevator_t) { \
@@ -93,8 +93,8 @@ static inline int elevator_request_laten#define ELEVATOR_LINUS \ ((elevator_t) { \ - 2048,/* read passovers */ \ -8192 ,/* write passovers */ \ + 128,/* read passovers */ \ +512 ,/* write passovers */ \ \ elevator_linus_merge,/* elevator_merge_fn */ \ elevator_linus_merge_req, /* elevator_merge_req_fn */ \
include/linux/nbd.h +1 -1
@@ -48,7 +48,7 @@ nbd_end_request(struct request *req)spin_lock_irqsave (&io_request_lock, flags);while ((bh = req->bh) !=NULL ) { nsect = bh->b_size >>9 ; -blk_finished_io (nsect); +blk_finished_io (req, nsect); req->bh = bh->b_reqnext; bh->b_reqnext =NULL ; bh->b_end_io (bh, uptodate);
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
Quoted 8 lines · Andrea Arcangeli
Quoted 3 lines
Anyway, less talk, more code. Treat this with care, it has only been lightly tested. Thanks to Andrea and Nick whose patches this is largely based on:
I spent last Saturday working on this too. This is the status of my current patches, would be interesting to compare them. they're not very well tested yet though.
I'll try to get some numbers in the morning.
Quoted 8 lines
They would obsoletes the old fix-pausing and the old elevator-lowlatency (I was going to release a new tree today, but I delayed it so I fixed uml today too first [tested with skas and w/o skas]).
those backout the rc7 interactivity changes (the only one that wasn't in my tree was the
add_wait_queue_exclusive, that IMHO would better stay for scalability reasons).
I didn't test without _exlusive for the final iteration of my patch, but in all the early ones using _exclusive improve latencies. I think people are reporting otherwise because they have hit the sweet spot for number of procs going after the requests. With _exclusive they have a higher chance of getting starved by a new process coming in, without the _exclusive, each waiter has a fighting chance of getting to the free request on their own. Hopefully we can do better with the _exclusive, it does seem to scale much better.
Aside from the io in flight calculations, the major difference between our patches is in __get_request_wait. Once a process waits once, that call to __get_request_wait ignores q->full in my code.
I found the q->full checks did help, but as you increased the number of concurrent readers/writers things broke down to the old high latencies. By delaying the point where q->full was cleared, I could make the latency benefit last for a higher number of procs. Finally I gave up and left it set until all the waiters were gone, which seems to have the most consistent results.
The interesting part was it didn't seem to change the hit in throughput. The cost was about the same between the original patch and my final one, but I need to test more.
-chris
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
In reply to Andrea ArcangeliQuoted 8 lines · Andrea Arcangeli
I spent last Saturday working on this too. This is the status of my current patches, would be interesting to compare them. they're not very well tested yet though.
They would obsoletes the old fix-pausing and the old elevator-lowlatency (I was going to release a new tree today, but I delayed it so I fixed uml today too first [tested with skas and w/o skas]).
Quoted 8 lines
those backout the rc7 interactivity changes (the only one that wasn't in my tree was the
add_wait_queue_exclusive, that IMHO would better stay for scalability reasons).Of course I would be very interested to know if those two patches (or Chris's one, you also retained the exclusive wakeup) are still greatly improved by removing the
_exclusiveweakups and going wake-all (in theory they shouldn't).
Ok, I merged these into rc7 along with the __get_request_wait stats patch. All numbers below were on ext2...I'm calling your patches -aa, even though it's just a small part of the real -aa ;-) After a dbench 50 run, the -aa __get_request_wait latencies look like this:
device 08:01: num_req 6029, total jiffies waited 213475
844 forced to wait
2 min wait, 806 max wait
252 average wait
357 < 100, 29 < 200, 110 < 300, 111 < 400, 82 < 500 155 waits longer than 500 jiffies
I changed my patch to have q->nr_requests at 1024 like yours, and reran the dbench 50:
device 08:01: num_req 11122, total jiffies waited 121573
8782 forced to wait
1 min wait, 237 max wait
13 average wait
8654 < 100, 126 < 200, 2 < 300, 0 < 400, 0 < 500 0 waits longer than 500 jiffies
So, I had 5000 more requests for the same workload, and 8000 of my requests were forced to wait (compared to 844 of yours). But the total number of jiffies spent waiting on my patch was lower, as were the average and max waits. Increasing the number of requests with my patch make the system feel slower, even though the __get_request_wait latency numbers didn't change.
On this dbench run, you got a throughput of 118mb/s and I got 90mb/s. The __get_request_wait latency numbers were reliable across runs, but I might as well have thrown a dart to pick throughput numbers. So, next tests were done with iozone.
On aa after iozone -s 100M -i 0 -t 20 (20 procs each doing streaming writes to a private 100M file)
device 08:01: num_req 167133, total jiffies waited 872566
6424 forced to wait
4 min wait, 507 max wait
135 average wait
2619 < 100, 2020 < 200, 1433 < 300, 325 < 400, 26 < 500 1 waits longer than 500 jiffies
And the iozone throughput numbers looked like so (again -aa patches)
Children see throughput for 20 initial writers = 13824.22 KB/sec
Parent sees throughput for 20 initial writers = 6811.29 KB/sec
Min throughput per process = 451.99 KB/sec
Max throughput per process = 904.14 KB/sec
Avg throughput per process = 691.21 KB/sec
Min xfer = 51136.00 KBThe avg throughput per process with vanilla rc7 is 3MB/s, the best I've been able to do was with nr_requests at higher levels was 1.3MB/s. With smaller of iozone threads (10 and lower so far) I can match rc7 speeds, but not with 20 procs.
Anyway, my latency numbers for iozone -s 100M -i 0 -t 20:
device 08:01: num_req 146049, total jiffies waited 434025
130670 forced to wait
1 min wait, 65 max wait
3 average wait
130671 < 100, 0 < 200, 0 < 300, 0 < 400, 0 < 500 0 waits longer than 500 jiffies
And the iozone reported throughput:
Children see throughput for 20 initial writers = 19828.92 KB/sec
Parent sees throughput for 20 initial writers = 7003.36 KB/sec
Min throughput per process = 526.61 KB/sec
Max throughput per process = 1353.45 KB/sec
Avg throughput per process = 991.45 KB/sec
Min xfer = 39968.00 KBThe patch I was working on today was almost the same as the one I posted yesterday, the only difference being the hunk below and changes to nr_requests (256 balanced nicely on my box, all numbers above were at 1024).
This hunk against my patch yesterday just avoids an unplug in __get_request_wait if there are still available requests. A process might be waiting in __get_request_wait just because the queue was full, which has little do to with the queue needing an unplug. He'll get woken up later by get_request_wait_wakeup if nobody else manages to wake him (I think).
drivers/block/ll_rw_blk.c +2 -1
@@ -661,7 +661,8 @@set_current_state (TASK_UNINTERRUPTIBLE );spin_lock_irq (&io_request_lock);if ((!waited &&queue_full (q, rw)) || q->rq[rw].count ==0 ) { -__generic_unplug_device (q); +if (q->rq[rw].count ==0 ) +__generic_unplug_device (q);spin_unlock_irq (&io_request_lock);schedule ();spin_lock_irq (&io_request_lock);
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
Quoted 5 lines · Chris Mason
On Mon, 2003-06-09 at 18:19, Andrea Arcangeli wrote:
The avg throughput per process with vanilla rc7 is 3MB/s, the best I've been able to do was withnr_requestsat higher levels was 1.3MB/s. With smaller of iozone threads (10 and lower so far) I can match rc7 speeds, but not with 20 procs.
at least with my patches, I also made this change:
-#define ELV_LINUS_SEEK_COST 16
+#define ELV_LINUS_SEEK_COST 1
#define ELEVATOR_NOOP \
((elevator_t) { \
@@ -93,8 +93,8 @@ static inline int elevator_request_laten
#define ELEVATOR_LINUS \
((elevator_t) { \
- 2048, /* read passovers */ \
- 8192, /* write passovers */ \
+ 128, /* read passovers */ \
+ 512, /* write passovers */ \
\you didn't change the I/O scheduler at all compared to mainline, so there can be quite a lot of difference in the bandwidth average per process between my patches and mainline and your patches (unless you run elvtune or unless you backed out the above).
Anyways the 130671 < 100, 0 < 200, 0 < 300, 0 < 400, 0 < 500 from your patch sounds perfectly fair and that's unrelated to I/O scheduler and size of runqueue. I believe the most interesting difference is the blocking of tasks until the waitqueue is empty (i.e. clearing the waitqueue-full bit only when nobody is waiting). That is the right thing to do of course, that was a bug in my patch I merged by mistake from Nick's original patch, and that I'm going to fix immediatly of course.
Andrea
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
Quoted 15 lines · Andrea Arcangeli
Quoted 5 lines · Chris Mason
On Mon, 2003-06-09 at 18:19, Andrea Arcangeli wrote:
The avg throughput per process with vanilla rc7 is 3MB/s, the best I've been able to do was withnr_requestsat higher levels was 1.3MB/s. With smaller of iozone threads (10 and lower so far) I can match rc7 speeds, but not with 20 procs.at least with my patches, I also made this change:
(quoted diff) +1
-1-#define ELV_LINUS_SEEK_COST 16 +#define ELV_LINUS_SEEK_COST 1 #define ELEVATOR_NOOP \ ((elevator_t) { \
Review static inline int elevator_request_laten · line 99
Quoted 9 lines
(quoted diff) +2
-2@@ -93,8 +93,8 @@ static inline int elevator_request_laten #define ELEVATOR_LINUS \ ((elevator_t) { \ - 2048, /* read passovers */ \ - 8192, /* write passovers */ \ + 128, /* read passovers */ \ + 512, /* write passovers */ \ \
Right, I had forgotten to elvtune these in before my runs. It shouldn't change the __get_request_wait numbers, except for changes in the percentage of merged requests leading to a different number of requests overall (which my numbers did show).
Quoted 5 lines
you didn't change the I/O scheduler at all compared to mainline, so there can be quite a lot of difference in the bandwidth average per process between my patches and mainline and your patches (unless you run elvtune or unless you backed out the above).
Quoted 7 lines
Anyways the 130671 < 100, 0 < 200, 0 < 300, 0 < 400, 0 < 500 from your patch sounds perfectly fair and that's unrelated to I/O scheduler and size of runqueue. I believe the most interesting difference is the blocking of tasks until the waitqueue is empty (i.e. clearing the waitqueue-full bit only when nobody is waiting). That is the right thing to do of course, that was a bug in my patch I merged by mistake from Nick's original patch, and that I'm going to fix immediatly of course.
Ok, Increasing q->nr_requests also changes the throughput in high merge workloads.
Basically if we have 20 procs doing streaming buffered io, the buffers end up mixed together on the dirty list. So assuming we hit the hard dirty limit and all 20 procs are running write_some_buffers() the only way we'll be able to efficiently merge the end result is if we can get in 20 * 32 requests before unplugging.
This is because write_some_buffers grabs 32 buffers at a time, and each caller has to wait fairly in __get_request_wait. With only 128 requests in the run queue, the disk is unplugged before any of the 20 procs has submitted each of their 32 buffers.
It might make sense to change write_some_buffers to work in smaller units, 32 seems like a lot of times to wait in __get_request_wait just for an atime update.
-chris
Re: [PATCH] io stalls
In reply to Chris MasonQuoted 18 lines · Chris Mason
Ok, there are lots of different problems here, and I've spent a little while trying to get some numbers with the
__get_request_waitstats patch I posted before. This is all on ext2, since I wanted to rule out interactions with the journal flavors.Basically a dbench 90 run on ext2 rc6 vanilla kernels can generate latencies of over 2700 jiffies in
__get_request_wait, with an average latency over 250 jiffies.No, most desktop workloads aren't dbench 90, but between
balance_dirty()and the way we send stuff to disk during memory allocations, just about any process can get stuck submitting dirty buffers even if you've just got one process doing a dd if=/dev/zero of=foo.So, for the moment I'm going to pretend people seeing stalls in X are stuck in atime updates or memory allocations, or reading proc or some other silly spot.
Quoted 9 lines
For the SMP corner cases, I've merged Andrea's fix-pausing patch into rc7, along with an altered form of Nick Piggin's
queue_fullpatch to try and fix the latency problems.The major difference from Nick's patch is that once the queue is marked full, I don't clear the full flag until the wait queue is empty. This means new io can't steal available requests until every existing waiter has been granted a request.
Yes, this is probably a good idea.
Quoted 6 lines
The latency results are better, with average time spent in
__get_request_waitbeing around 28 jiffies, and a max of 170 jiffies. The cost is throughput, further benchmarking needs to be done but, but I wanted to get this out for review and testing. It should at least help
us decide if the request allocation code really is causing our problems.
Well the latency numbers are good - is this with dbench 90?
snip
Quoted 19 lines
include/linux/blkdev.h +17
-0+static inline void set_queue_full (request_queue_t *q,int rw) +{ +wmb (); +if (rw ==READ ) + q->read_full =1 ; +else + q->write_full =1 ; +} + +static inline void clear_queue_full (request_queue_t *q,int rw) +{ +wmb (); +if (rw ==READ ) + q->read_full =0 ; +else + q->write_full =0 ; +}
include/linux/blkdev.h line 202
Quoted 10 lines
include/linux/blkdev.h +10
-0+ +static inline int queue_full (request_queue_t *q,int rw) +{ +rmb (); +if (rw ==READ ) +return q->read_full; +else +return q->write_full; +} +
I don't think you need the barriers here, do you?
Re: [PATCH] io stalls
Quoted 10 lines · Nick Piggin
Quoted 7 lines
The latency results are better, with average time spent in
__get_request_waitbeing around 28 jiffies, and a max of 170 jiffies. The cost is throughput, further benchmarking needs to be done but, but I wanted to get this out for review and testing. It should at least help us decide if the request allocation code really is causing our problems.Well the latency numbers are good - is this with dbench 90?
Yes, that number was dbench 90, but dbench 50,90, and 120 gave about the same stats with the final patch.
Quoted 16 lines
snip
include/linux/blkdev.h line 202
Quoted 11 lines
include/linux/blkdev.h +10
-0+ +static inline int queue_full (request_queue_t *q,int rw) +{ +rmb (); +if (rw ==READ ) +return q->read_full; +else +return q->write_full; +} +I don't think you need the barriers here, do you?
I put the barriers in early on when almost all the calls were done outside spin locks, the current flavor of the patch only does one clear_queue_full without the io_request_lock held. It should be enough to toss a barrier in just that one spot. But I wanted to leave them in so I could move things around until the final version (if there ever is one ;-)
-chris
Re: [PATCH] io stalls
Quoted 9 lines · Chris Mason
Quoted 6 lines · Nick Piggin
Quoted 6 lines
The latency results are better, with average time spent in
__get_request_waitbeing around 28 jiffies, and a max of 170 jiffies. The cost is throughput, further benchmarking needs to be done but, but I wanted to get this out for review and testing. It should at least help
us decide if the request allocation code really is causing our problems.
Quoted 7 lines
Quoted 3 lines
Well the latency numbers are good - is this with dbench 90?
Yes, that number was dbench 90, but dbench 50,90, and 120 gave about the same stats with the final patch.
Great.
Quoted 20 lines
Quoted 4 lines
snip
Quoted 15 lines
include/linux/blkdev.h line 202
Quoted 12 lines
include/linux/blkdev.h +10
-0+ +static inline int queue_full (request_queue_t *q,int rw) +{ +rmb (); +if (rw ==READ ) +return q->read_full; +else +return q->write_full; +} +I don't think you need the barriers here, do you?
Quoted 7 lines
I put the barriers in early on when almost all the calls were done outside spin locks, the current flavor of the patch only does one
clear_queue_fullwithout theio_request_lockheld. It should be enough to toss a barrier in just that one spot. But I wanted to leave them in so I could move things around until the final version (if there ever is one ;-)
Yeah I see.
RE: [PATCH] io stalls
In reply to Nick PigginFrom: [email protected] [mailto:[email protected]]On Behalf Of Nick Piggin
Quoted 8 lines
Quoted 4 lines · Chris Mason
The major difference from Nick's patch is that once the queue is marked full, I don't clear the full flag until the wait queue is empty. This means new io can't steal available requests until every existing waiter has been granted a request.
Yes, this is probably a good idea.
Err... wouldn't this subvert the spirit, if not the warrant, of real time scheduling and time-critical applications?
After all we *do* want to all-but-starve lower priority tasks of IO in the presence of higher priority tasks. A select few applications absolutely need to be pampered (think ProTools audio mixing suite on the Mac etc.) and any solution that doesn't take this into account will have to be re-done by the people who want to bring these kinds of tasks to Linux.
I am not most familiar with this body of code, but wouldn't the people trying to do audio sampling and gaming get really frosted if they had to wait for a list of lower priority IO events to completely drain before they could get back to work? It would certainly produce really bad encoding of live data streams (etc).
Quoted 1 line
From a purely queue-theory stand point, I'm not even sure why this queue can
become "full". Shouldn't the bounding case come about primarily by lack of resources (can't allocate a queue entry or a data block) out where the users can see and cope with the problem before all the expensive blocking and waiting.
Still from a pure-theory standpoint, it would be "better" to make the wait queues priority queues and leave their sizes unbounded.
In practice it is expensive to maintain a fully "proper" priority queue for a queue of non-trivial size. Then again, IO isn't cheap over the domain of time anyway.
The solution proposed, by limiting the queue size sort-of turns the scheduler's wakeup behavior into that priority queue sorting mechanism. That in turn would (it seems to me) lead to some degenerate behaviors just outside the zone of midline stability. In short several very-high-priority tasks could completely starve out the system if they can consistently submit enough request to fill the queue.
[That is: consider a bunch of tasks sleeping in the scheduler because they are waiting for the queue to empty. When they are all woken up, they will actually be scheduled in priority order. So higher priority tasks get first crack at the "empty" queue. If there are "enough" such tasks (which are IO bound on this device) they will keep getting serviced, and then keep going back to sleep on the full queue. (And god help you if they are runaways 8-). The high priority tasks constantly butt in line (because the scheduler is now the keeper of the IO queue) and the lower priority tasks could wait forever.]
{please note; I write some fairly massively-threaded applications, it would only take one such application running at a high priority to produce "a substantial number" of high priority processes submitting IO requests, so the scenario, while not common, is potentially real.}
(so just off the top of my head...)
I would think that the best theoretical solution would be a priority heap. (ignoring heap storage requirements for a moment) you keep the highest priority items in the front of the heap and any time a heap reorg passes a node by you jack that nodes priority by one. For an extremely busy queue nothing is starved, but the incline remains high enough to make sure that the truly desperate priorities (of which there should be few in a real world system) will "never" wait behind some dd(1) of vanishingly close to no import.
Clearly doing a full heap with only pointers is ugly almost beyond comprehension, and doing a heap in an array would tend to be impractical for a large list under variable conditions. A red-black tree gets too expensive if you use them that many times throughout a system. (and so on)
While several possible sort-of-heapish or sort-of-priority-queueish data str uctures come to mind, I don't have a replacement concept that I can really promote just now...
I would say that at a MINIMUM there needs to be some threshold of priority for requests that get to go on a "full list" no matter what. There really "ought to be" a way for requests from higher priority tasks to get closer to the front of the list. There "should be" a priority floor where tasks with lower priorities get their requests queued up with the current first-come-first-served mentality (as we don't need to spend a lot of time thinking about things that have been nice(d) into the noise floor). And then there should be a promotion mechanism to prevent complete starvation.
Anything simpler and it is safer from a system stability standpoint to keep with the current high-latency-on-occasion simple queue solution.
Rob.
RE: [PATCH] io stalls
Quoted 10 lines · Robert White
From: [email protected] [mailto:[email protected]]On Behalf Of Nick Piggin
Quoted 6 lines
Quoted 4 lines · Chris Mason
The major difference from Nick's patch is that once the queue is marked full, I don't clear the full flag until the wait queue is empty. This means new io can't steal available requests until every existing waiter has been granted a request.
Quoted 6 lines
Quoted 1 line
Yes, this is probably a good idea.
Err... wouldn't this subvert the spirit, if not the warrant, of real time scheduling and time-critical applications?
[ lots of interesting points ]
Heh, I didn't really make my goals for the patch clear. They go:
1) quantify the stalls people are seeing with real numbers so we can point at a section of code causing bad performance.
2) Provide a somewhat obvious patch that makes the current __get_request_wait call significantly more fair, in hopes of either blaming it for the stalls or removing it from the list of candidates
3) fix the stalls
Most of your suggestions are 2.5 discussion material, where real experimental work is going on. The 2.4 io request wait queue isn't working on priorities, the current one tries to be fair to everyone and provide good throughput to everyone at the same time. It's failing on at least one of those, and until we can fix that I don't even want to think about more complex issues.
Current users of the vanilla 2.4 tree will hopefully benefit from a lower latency io request wait queue. The next best thing to real time is a consistently small wait, which is what my patch is trying for.
-chris
RE: [PATCH] io stalls
From: Chris Mason [mailto:[email protected]]
Sent: Monday, June 09, 2003 7:13 PM
Quoted 3 lines
2) Provide a somewhat obvious patch that makes the current
__get_request_waitcall significantly more fair, in hopes of either blaming it for the stalls or removing it from the list of candidates
Without the a_w_q_exclusive() on add_wait_queue the FIFO effect is lost when all the members of the wait queue compete for their timeslice in the scheduler. For all intents and purposes the fairness goes up some (you stop having the one guy sorted to the un-happy end of the disk) but low priority tasks will still always end up stalled on the dirty end of the stick. Basically each new round at the queue-empty moment is a mob rush for the door.
With the a_w_q_exclusive(), you get past fair and well into anti-optimal. Your FIFO becomes essentially mandatory with no regard for anything but the order things hit the wait queue. (Particularly on an SMP machine, however) "new requestors" may/will jump to the head of the line because they were never *in* the wait queue. So you have only achieved "fairness" with respect to requests that come in to a io queue that was full-at-the-time of the initial entry into the driver. This becomes exactly like the experience of waiting patiently on line to get off the highway and watching all the rude people driving by you only to cut over and nose into the queue just at the exit sign.
So you need the _exclusive if you want any kind of predictable fairness (without getting into anything obscure) but it is still only "fair" for those that were unfortunate enough to end up on the wait queue originally. There is a small window for tasks to butt in freely.
Quoted 1 line
3) fix the stalls
Without the _exclusive() you can't have fixed the stalls, you can only have moved the locus-of-blame to the scheduler which may (or may not) have some way to compensate and "fake fairness" built in by coincidence.
The thing I suggest in my other email, where you use the non-exclusive version of the routine but temporarily bump the process priority each time a request gets foisted off on the wait_queue instead of the IO queue, actually has semantic fairness built in. This basically builds a fairness elevator that functions both over time-in-queue and original process priority (when built into your basic patch).
It's also quite space/time efficient and fairly clear to reader and implementer alike.
Rob.
RE: [PATCH] io stalls
Quoted 15 lines · Robert White
From: Chris Mason [mailto:[email protected]]
Sent: Monday, June 09, 2003 7:13 PMQuoted 3 lines
2) Provide a somewhat obvious patch that makes the current
__get_request_waitcall significantly more fair, in hopes of either blaming it for the stalls or removing it from the list of candidatesWithout the
a_w_q_exclusive()onadd_wait_queuethe FIFO effect is lost when all the members of the wait queue compete for their timeslice in the scheduler. For all intents and purposes the fairness goes up some (you stop having the one guy sorted to the un-happy end of the disk) but low priority tasks will still always end up stalled on the dirty end of the stick. Basically each new round at the queue-empty moment is a mob rush for the door.
Quoted 5 lines
With the
a_w_q_exclusive(), you get past fair and well into anti-optimal. Your FIFO becomes essentially mandatory with no regard for anything but the order things hit the wait queue. (Particularly on an SMP machine, however) "new requestors" may/will jump to the head of the line because they were never *in* the wait queue.
The patches flying around force new io into the wait queue any time someone else is already waiting, nobody is allowed to jump to the head of the line.
The rest of your ideas are interesting, we just can't smush them into 2.4. Please consider doing some experiments on the 2.5 io schedulers and making suggestions, it's a critical area.
-chris
Re: [PATCH] io stalls
In reply to Robert WhiteQuoted 13 lines · Robert White
From: [email protected] [mailto:[email protected]]On Behalf Of Nick Piggin
Quoted 8 lines
Quoted 5 lines · Chris Mason
The major difference from Nick's patch is that once the queue is marked full, I don't clear the full flag until the wait queue is empty. This means new io can't steal available requests until every existing waiter has been granted a request.
Quoted 7 lines
Quoted 2 lines
Yes, this is probably a good idea.
Err... wouldn't this subvert the spirit, if not the warrant, of real time scheduling and time-critical applications?
No, my patch (plus Chris' modification) change request allocation from an overloaded queue from semi random (timing dependant mixture of LIFO and FIFO), to FIFO.
As Chris has shown, can cause a task to be starved for 2.7s (and theoretically infinite) when it should be woken in < 200ms under similar situations with the FIFO scheme.
Quoted 7 lines
After all we *do* want to all-but-starve lower priority tasks of IO in the presence of higher priority tasks. A select few applications absolutely need to be pampered (think ProTools audio mixing suite on the Mac etc.) and any solution that doesn't take this into account will have to be re-done by the people who want to bring these kinds of tasks to Linux.
Quoted 7 lines
I am not most familiar with this body of code, but wouldn't the people trying to do audio sampling and gaming get really frosted if they had to wait for a list of lower priority IO events to completely drain before they could get back to work? It would certainly produce really bad encoding of live data streams (etc).
Actually, there is no priority other than time (ie. FIFO), and seek distance in the IO subsystem. I guess this is why your arguments fall down ;)
Quoted 6 lines
Quoted 1 line
From a purely queue-theory stand point, I'm not even sure why this queue can
become "full". Shouldn't the bounding case come about primarily by lack of resources (can't allocate a queue entry or a data block) out where the users can see and cope with the problem before all the expensive blocking and waiting.
In practice, the problems of having a memory size limited queue outweigh the benefits.
Quoted 8 lines
Still from a pure-theory standpoint, it would be "better" to make the wait queues priority queues and leave their sizes unbounded.
In practice it is expensive to maintain a fully "proper" priority queue for a queue of non-trivial size. Then again, IO isn't cheap over the domain of time anyway.
If IO priorities were implemented, you still have the problem of starvation. It would be better to simply have a per process limit on request allocation, and implement the priority scheduling in the io scheduler.
I think you would find that most processes do just fine with just a couple of requests each, though.
Quoted 9 lines
The solution proposed, by limiting the queue size sort-of turns the scheduler's wakeup behavior into that priority queue sorting mechanism. That in turn would (it seems to me) lead to some degenerate behaviors just outside the zone of midline stability. In short several very-high-priority tasks could completely starve out the system if they can consistently submit enough request to fill the queue.
Quoted 10 lines
[That is: consider a bunch of tasks sleeping in the scheduler because they are waiting for the queue to empty. When they are all woken up, they will actually be scheduled in priority order. So higher priority tasks get first crack at the "empty" queue. If there are "enough" such tasks (which are IO bound on this device) they will keep getting serviced, and then keep going back to sleep on the full queue. (And god help you if they are runaways 8-). The high priority tasks constantly butt in line (because the scheduler is now the keeper of the IO queue) and the lower priority tasks could wait forever.]
No, they will be woken up one at a time as requests become freed, and in FIFO order. It might be possible for a higher (CPU) priority task to be woken up before the previous has a chance to run, but this scheme is no worse than before (the solution here is per process request limits, but this is 2.4).
RE: [PATCH] io stalls
From: Nick Piggin [mailto:[email protected]]
Sent: Monday, June 09, 2003 8:23 PM
Quoted 4 lines
Actually, there is no priority other than time (ie. FIFO), and seek distance in the IO subsystem. I guess this is why your arguments fall down ;)
I'll buy that for the most part, though one of the differences I read elsewhere in the thread was the choice between add_wait_queue() and add_wait_queue_exclusive(). You will, however, note that one of the factors that is playing in this patch is process priority.
(If I understand correctly) The wait queue in question becomes your FIFOing agent, it is kind of a pre-queue on the actual IO queue, once you reach a "full" condition.
In the later case [add_wait_queue_exclusive()] you are strictly FIFO over the set of processes, where the moment-of-order is determined by insertion into the wait queue.
In the former case [add_wait_queue()] when the queue is woken up all the waiters will be marked executable on the scheduler, and the scheduler will then (at least tend to) sort the submissions into task priority order. So the higher priority tasks will get to butt into line. Worse, the FIFO is essentially lost to the vagaries of the scheduler so without the _exclusive you have no FIFO at all.
I think that is the reason that Chris was saying the add_wait_queue_exclusive() mode "does seem to scale much better."
So your "original new" batching agent is really order-of-arrival that becomes anti-sorted by process priority. Which can lead to scheduler induced starvation (and the observed "improvements" by using the strict FIFO created by a_w_q_exclusive). The problem is that you get a little communist about the FIFO-ness when you use a_w_q_exclusive() and that can *SERIOUSLY* harm a task that must approach real-time behavior.
One solution would be to stick with the add_wait_queue() process-priority influenced never-really-FIFO, but every time a process/task wakes up, and it then doesn't get its request onto the queue, add a small fixed increment to its priority before going back into the wait. This gives you both the process-priority mechanism and a fairness metric.
Something like (in pure pseudo-code since I don't have my references here):
int priority_delta = 0
while (try_enqueing_io_request() == queue_full) {
if (current->priority < priority_max) {
current->priority += priority_increment;
priority_delta += priority_increment; }
wait_on_queue()
}current->priority -= priority_delta;
(and still, of course, only wake the wait queue when the "full" queue reaches empty.)
What that gets you is democratic entry into the io request queue when it is non-full. It gets you seniority-based (plutocratic?) access to the io queue as your request "ages" in the full pool. If the pool gets so large that all the requests are making their tasks reach priority_max then you "degrade" to the fairness of the scheduler, which is an arbitrary but workable metric.
You get all that, but you preserve (or invent) a relationship that lets the task priority automagically factor in "for free" so that relative starvation (which is a good thing for deliberately asymmetric task priorities, and matches user expectations) can be achieved without ever having absolute starvation.
Further if priority_max isn't priority_system_max you get the real-time-trumps-all behavior that something like a live audio stream encoder may need (for any priority >= priority_max).
Rob.
Re: [PATCH] io stalls
Quoted 8 lines · Robert White
From: Nick Piggin [mailto:[email protected]]
Sent: Monday, June 09, 2003 8:23 PMQuoted 4 lines
Actually, there is no priority other than time (ie. FIFO), and seek distance in the IO subsystem. I guess this is why your arguments fall down ;)
Quoted 9 lines
I'll buy that for the most part, though one of the differences I read elsewhere in the thread was the choice between
add_wait_queue()andadd_wait_queue_exclusive(). You will, however, note that one of the factors that is playing in this patch is process priority.(If I understand correctly) The wait queue in question becomes your FIFOing agent, it is kind of a pre-queue on the actual IO queue, once you reach a "full" condition.
Right.
Quoted 12 lines
In the later case [
add_wait_queue_exclusive()] you are strictly FIFO over the set of processes, where the moment-of-order is determined by insertion into the wait queue.In the former case [
add_wait_queue()] when the queue is woken up all the waiters will be marked executable on the scheduler, and the scheduler will then (at least tend to) sort the submissions into task priority order. So the higher priority tasks will get to butt into line. Worse, the FIFO is essentially lost to the vagaries of the scheduler so without the_exclusiveyou have no FIFO at all.
Quoted 3 lines
I think that is the reason that Chris was saying the
add_wait_queue_exclusive()mode "does seem to scale much better."
Yep
Quoted 8 lines
So your "original new" batching agent is really order-of-arrival that becomes anti-sorted by process priority. Which can lead to scheduler induced starvation (and the observed "improvements" by using the strict FIFO created by
a_w_q_exclusive). The problem is that you get a little communist about the FIFO-ness when you usea_w_q_exclusive()and that can *SERIOUSLY* harm a task that must approach real-time behavior.
I think it had better be FIFO for now. If its not, you're making the worst case latency worse. It requires a lot of careful testing to get something like that working right.
You have some good ideas, and quite possibly they would be worth implementing, but the behaviour of the code is quite complex, especially when you take into account its affect on the io scheduler.
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
In reply to Chris Masondrivers/block/ll_rw_blk.c line 677
Quoted 2 lines · Chris Mason
drivers/block/ll_rw_blk.c +2
-0+if (!waitqueue_active (&q->wait_for_requests[rw])) +clear_queue_full (q, rw);
you've an smp race above, the smp safe implementation is this:
if (!waitqueue_active(&q->wait_for_requests[rw])) {
clear_queue_full(q, rw);
mb();
if (unlikely(waitqueue_active(&q->wait_for_requests[rw])))
wake_up(&q->wait_for_requests[rw]);
}I'm also unsure what the "waited" logic does, it doesn't seem necessary.
Andrea
Re: [PATCH] io stalls
Quoted 8 lines · Andrea Arcangeli
drivers/block/ll_rw_blk.c line 677
Quoted 3 lines · Chris Mason
drivers/block/ll_rw_blk.c +2
-0+if (!waitqueue_active (&q->wait_for_requests[rw])) +clear_queue_full (q, rw);you've an smp race above, the smp safe implementation is this:
Quoted 9 lines
if (!waitqueue_active(&q->wait_for_requests[rw])) { clear_queue_full(q, rw); mb(); if (unlikely(waitqueue_active(&q->wait_for_requests[rw]))) wake_up(&q->wait_for_requests[rw]); }I'm also unsure what the "waited" logic does, it doesn't seem necessary.
When a task is woken up, it is quite likely that the queue is still marked full.
Re: [PATCH] io stalls
Quoted 19 lines · Nick Piggin
Quoted 15 lines · Andrea Arcangeli
drivers/block/ll_rw_blk.c line 677
Quoted 3 lines · Chris Mason
drivers/block/ll_rw_blk.c +2
-0+if (!waitqueue_active (&q->wait_for_requests[rw])) +clear_queue_full (q, rw);you've an smp race above, the smp safe implementation is this:
if (!waitqueue_active(&q->wait_for_requests[rw])) { clear_queue_full(q, rw); mb(); if (unlikely(waitqueue_active(&q->wait_for_requests[rw]))) wake_up(&q->wait_for_requests[rw]); }
Quoted 5 lines
Quoted 2 lines
I'm also unsure what the "waited" logic does, it doesn't seem necessary.
When a task is woken up, it is quite likely that the queue is still marked full.
but we don't care if it's marked full, see __get_request. If we cared about full it would deadlock anyways (no matter the waited logic)
Andrea
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
In reply to Andrea ArcangeliQuoted 6 lines · Andrea Arcangeli
drivers/block/ll_rw_blk.c line 677
Quoted 2 lines · Chris Mason
drivers/block/ll_rw_blk.c +2
-0+if (!waitqueue_active (&q->wait_for_requests[rw])) +clear_queue_full (q, rw);you've an smp race above, the smp safe implementation is this:
clear_queue_full has a wmb() in my patch, and queue_full has a rmb(), I thought that covered these cases? I'd rather remove those though, since the spot you point out is the only place done outside the io_request_lock.
Quoted 7 lines
if (!waitqueue_active(&q->wait_for_requests[rw])) {
clear_queue_full(q, rw);
mb();
if (unlikely(waitqueue_active(&q->wait_for_requests[rw])))
wake_up(&q->wait_for_requests[rw]);
}I don't think we need the extra wake_up (this is in __get_request_wait, right?), since it gets done by get_request_wait_wakeup()
Quoted 1 line
I'm also unsure what the "waited" logic does, it doesn't seem necessary.
Once a process waits once, they are allowed to ignore the q->full flag. This way existing waiters can make progress even when q->full is set. Without the waited check, q->full will never get cleared because the last writer wouldn't proceed until the last writer was gone. I had to make __get_request for the same reason.
-chris
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
Quoted 20 lines · Chris Mason
Quoted 6 lines · Andrea Arcangeli
drivers/block/ll_rw_blk.c line 677
Quoted 2 lines · Chris Mason
drivers/block/ll_rw_blk.c +2
-0+if (!waitqueue_active (&q->wait_for_requests[rw])) +clear_queue_full (q, rw);you've an smp race above, the smp safe implementation is this:
clear_queue_fullhas awmb()in my patch, andqueue_fullhas armb(), I thought that covered these cases? I'd rather remove those though, since the spot you point out is the only place done outside theio_request_lock.Quoted 7 lines
if (!waitqueue_active(&q->wait_for_requests[rw])) { clear_queue_full(q, rw); mb(); if (unlikely(waitqueue_active(&q->wait_for_requests[rw]))) wake_up(&q->wait_for_requests[rw]); }
Quoted 2 lines
I don't think we need the extra
wake_up(this is in__get_request_wait, right?), since it gets done byget_request_wait_wakeup()
there's no get_request_wait_wakeup in blkdev_release_request. I put the construct in both places though (i've the clear_queue_full explicit as q->full = 0).
And I don't think any of your barriers is needed at all, I mean, we only need to be careful to clear it right, we don't need to be careful to set or read it right when it transits from 0 to 1. And the above seems enough to me to get right the clearing.
Quoted 7 lines
Quoted 1 line
I'm also unsure what the "waited" logic does, it doesn't seem necessary.
Once a process waits once, they are allowed to ignore the
q->fullflag. This way existing waiters can make progress even whenq->fullis set. Without the waited check,q->fullwill never get cleared because the last writer wouldn't proceed until the last writer was gone. I had to make__get_requestfor the same reason.
__get_request makes perfect sense of course and it's needed, this is not the issue, my point about the waited check is that the last writer has to get the wakeup (and the wakeup has nothing to do with the waited check since waited == 0), and after the wakeup it will get the request and it won't re-run the loop, so I don't see why waited is needed. Furthmore even if for whatever reason it doesn't get the request, it will re-set full to 1 and it'll be still the first to get the wakeup, and it will definitely get another wakeup if none request was available.
Andrea
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
Quoted 5 lines · Andrea Arcangeli
And I don't think any of your barriers is needed at all, I mean, we only need to be careful to clear it right, we don't need to be careful to set or read it right when it transits from 0 to 1. And the above seems enough to me to get right the clearing.
The current form of the patch has way too many barriers. When I first added them the patch was really different, I left them in because it seems to be easier to remember to rip them out than add them back ;-)
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
I'm also unsure what the "waited" logic does, it doesn't seem necessary.
Once a process waits once, they are allowed to ignore the
q->fullflag. This way existing waiters can make progress even whenq->fullis set. Without the waited check,q->fullwill never get cleared because the last writer wouldn't proceed until the last writer was gone. I had to make__get_requestfor the same reason.
Quoted 8 lines
__get_requestmakes perfect sense of course and it's needed, this is not the issue, my point about the waited check is that the last writer has to get the wakeup (and the wakeup has nothing to do with the waited check since waited == 0), and after the wakeup it will get the request and it won't re-run the loop, so I don't see why waited is needed. Furthmore even if for whatever reason it doesn't get the request, it will re-set full to 1 and it'll be still the first to get the wakeup, and it will definitely get another wakeup if none request was available.
Ok, I see your point, we don't strictly need the waited check. I had added it as an optimization at first, so that those who waited once were not penalized by further queue_full checks.
-chris
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
Quoted 3 lines · Chris Mason
Ok, I see your point, we don't strictly need the waited check. I had added it as an optimization at first, so that those who waited once were not penalized by further
queue_fullchecks.
I could taste the feeling of not penalizing while reading the code but that's just a feeling, in reality if they blocked it means they set full by themself and there was no request so they want to go to sleep no matter ->full or not ;)
Andrea
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
Quoted 9 lines · Andrea Arcangeli
Quoted 3 lines · Chris Mason
Ok, I see your point, we don't strictly need the waited check. I had added it as an optimization at first, so that those who waited once were not penalized by further
queue_fullchecks.I could taste the feeling of not penalizing while reading the code but that's just a feeling, in reality if they blocked it means they set full by themself and there was no request so they want to go to sleep no matter ->full or not ;)
You're completely right, as the patch changed I didn't realize waited wasn't needed anymore ;-)
Are you adding the hunk from yesterday to avoid unplugs when q->rq.count != 0?
-chris
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
In reply to Andrea ArcangeliOk here's an updated patch, it changes the barriers around, updates comments, and gets rid of the waited check in __get_request_wait. It is still a combined patch with fix_pausing, queue_full and latency stats, mostly because I want to make really sure any testers are using all three.
So, if someone who saw io stalls in 2.4.21-rc could give this a try, I'd be grateful. If you still see stalls with this applied, run elvtune /dev/xxx and send along the resulting console output.
-chris
drivers/block/blkpg.c +1 -0
@@ -261,6 +261,7 @@return blkpg_ioctl (dev, (struct blkpg_ioctl_arg *) arg);case BLKELVGET : +blk_print_stats (dev);return blkelvget_ioctl (&blk_get_queue (dev)->elevator, (blkelv_ioctl_arg_t *) arg);case BLKELVSET :
drivers/block/ll_rw_blk.c +157 -44
308 lines not on this page · raw message
fs/buffer.c +21 -1
@@ -153,10 +153,23 @@get_bh (bh);add_wait_queue (&bh->b_wait, &wait);do { -run_task_queue (&tq_disk);set_task_state (tsk,TASK_UNINTERRUPTIBLE );if (!buffer_locked (bh))break ; +/* +* We must read tq_disk in TQ_ACTIVE after the +* add_wait_queue effect is visible to other cpus. +* We could unplug some line above it wouldn't matter +* but we can't do that right after add_wait_queue +* without an smp_mb() in between because spin_unlock +* has inclusive semantics. +* Doing it here is the most efficient place so we +* don't do a suprious unplug if we get a racy +* wakeup that make buffer_locked to return 0, and +* doing it here avoids an explicit smp_mb() we +* rely on the implicit one in set_task_state. +*/ +run_task_queue (&tq_disk);schedule (); }while (buffer_locked (bh)); tsk->state =TASK_RUNNING ;
@@ -1507,6 +1520,9 @@/* Done - end_buffer_io_async will unlock */ SetPageUptodate (page); + +wakeup_page_waiters (page); +return 0 ; out:
@@ -1538,6 +1554,7 @@ }while (bh != head);if (need_unlock)UnlockPage (page); +wakeup_page_waiters (page);return err; }
@@ -1765,6 +1782,8 @@else submit_bh (READ , bh); } + +wakeup_page_waiters (page);return 0 ; }
@@ -2378,6 +2397,7 @@submit_bh (rw, bh); bh = next; }while (bh != head); +wakeup_page_waiters (page);return 0 ; }
fs/super.c +1 -0
@@ -726,6 +726,7 @@if (!fs_type->read_super (s, data, flags &MS_VERBOSE ?1 :0 )) goto Einval; s->s_flags |=MS_ACTIVE ; +blk_reset_stats (dev);path_release (&nd);return s;
fs/reiserfs/inode.c +1 -0
@@ -2048,6 +2048,7 @@*/ if (nr) {submit_bh_for_writepage (arr, nr) ; +wakeup_page_waiters (page); }else {UnlockPage (page) ; }
include/linux/blkdev.h +42 -0
68 lines not on this page · raw message
include/linux/pagemap.h +2 -0
@@ -97,6 +97,8 @@___wait_on_page (page); } +extern void FASTCALL (wakeup_page_waiters (struct page * page)); +/* * Returns locked page at given index in given cache, creating it if needed. */
kernel/ksyms.c +1 -0
@@ -295,6 +295,7 @@EXPORT_SYMBOL (filemap_fdatawait);EXPORT_SYMBOL (lock_page);EXPORT_SYMBOL (unlock_page); +EXPORT_SYMBOL (wakeup_page_waiters);/* device registration */ EXPORT_SYMBOL (register_chrdev);
mm/filemap.c +14 -0
@@ -812,6 +812,20 @@return &wait[hash]; } +/* +* This must be called after every submit_bh with end_io +* callbacks that would result into the blkdev layer waking +* up the page after a queue unplug. +*/ +void wakeup_page_waiters (struct page * page) +{ +wait_queue_head_t * head; + + head =page_waitqueue (page); +if (waitqueue_active (head)) +wake_up (head); +} +/* * Wait for a page to get unlocked. *
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
drivers/block/ll_rw_blk.c line 932
Quoted 4 lines · Chris Mason
drivers/block/ll_rw_blk.c +4
-0+if (q->rq[rw].count >= q->batch_requests) { +smp_mb (); +if (waitqueue_active (&q->wait_for_requests[rw])) +wake_up (&q->wait_for_requests[rw]);
in my tree I also changed this to:
wake_up_nr(&q->wait_for_requests[rw], q->rq[rw].count);
otherwise only one waiter will eat the requests, while multiple waiters can eat requests in parallel instead because we freed not just 1 request but many of them.
I wonder if my above change is really the right way to implement the removal of the _exclusive line that went in rc6. However with your patch the wake_up_nr (or ~equivalent removal of _exclusive wakeup of rc6) should mostly improve cpu parallelism in smp and while waiting for I/O, the amount of stuff in the I/O queue and the overall fariness shouldn't change very significantly with this new completely fair FIFO request allocator.
Andrea
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
Quoted 8 lines · Andrea Arcangeli
drivers/block/ll_rw_blk.c line 932
Quoted 4 lines · Chris Mason
drivers/block/ll_rw_blk.c +4
-0+if (q->rq[rw].count >= q->batch_requests) { +smp_mb (); +if (waitqueue_active (&q->wait_for_requests[rw])) +wake_up (&q->wait_for_requests[rw]);in my tree I also changed this to:
Quoted 5 lines
wake_up_nr(&q->wait_for_requests[rw], q->rq[rw].count);otherwise only one waiter will eat the requests, while multiple waiters can eat requests in parallel instead because we freed not just 1 request but many of them.
I tried a few variations of this yesterday and they all led to horrible latencies, but I couldn't really explain why. I had a bunch of other stuff in at the time to try and improve throughput though, so I'll try it again.
I think part of the problem is the cascading wakeups from get_request_wait_wakeup(). So if we wakeup 32 procs they in turn wakeup another 32, etc.
-chris
Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)
Quoted 9 lines · Chris Mason
Quoted 8 lines · Andrea Arcangeli
drivers/block/ll_rw_blk.c line 932
Quoted 4 lines · Chris Mason
drivers/block/ll_rw_blk.c +4
-0+if (q->rq[rw].count >= q->batch_requests) { +smp_mb (); +if (waitqueue_active (&q->wait_for_requests[rw])) +wake_up (&q->wait_for_requests[rw]);in my tree I also changed this to:
Quoted 8 lines
Quoted 5 lines
wake_up_nr(&q->wait_for_requests[rw], q->rq[rw].count);otherwise only one waiter will eat the requests, while multiple waiters can eat requests in parallel instead because we freed not just 1 request but many of them.
I tried a few variations of this yesterday and they all led to horrible latencies, but I couldn't really explain why. I had a bunch of other
the I/O latency in theory shouldn't change, we're not reordering the queue at all, they'll go to sleep immediatly again if __get_request returns null.
Quoted 6 lines
stuff in at the time to try and improve throughput though, so I'll try it again.
I think part of the problem is the cascading wakeups from
get_request_wait_wakeup(). So if we wakeup 32 procs they in turn wakeup another 32, etc.
so maybe it's enough to wakeup count / 2 to account for the double wakeup? that will avoid some overscheduling indeed.
Andrea
Re: [PATCH] io stalls
Quoted 19 lines · Andrea Arcangeli
Quoted 17 lines · Chris Mason
Quoted 15 lines · Andrea Arcangeli
drivers/block/ll_rw_blk.c line 932
Quoted 5 lines · Chris Mason
drivers/block/ll_rw_blk.c +4
-0+if (q->rq[rw].count >= q->batch_requests) { +smp_mb (); +if (waitqueue_active (&q->wait_for_requests[rw])) +wake_up (&q->wait_for_requests[rw]);in my tree I also changed this to:
wake_up_nr(&q->wait_for_requests[rw], q->rq[rw].count);otherwise only one waiter will eat the requests, while multiple waiters can eat requests in parallel instead because we freed not just 1 request but many of them.
Quoted 8 lines
Quoted 3 lines
I tried a few variations of this yesterday and they all led to horrible latencies, but I couldn't really explain why. I had a bunch of other
the I/O latency in theory shouldn't change, we're not reordering the queue at all, they'll go to sleep immediatly again if
__get_requestreturns null.
And go to the end of the queue?
Quoted 4 lines
Quoted 4 lines
stuff in at the time to try and improve throughput though, so I'll try it again.
Quoted 9 lines
Quoted 4 lines
I think part of the problem is the cascading wakeups from
get_request_wait_wakeup(). So if we wakeup 32 procs they in turn wakeup another 32, etc.so maybe it's enough to wakeup count / 2 to account for the double wakeup? that will avoid some overscheduling indeed.
Andrea, this isn't needed because when the queue falls below the batch limit, every retiring request will do a wake up and another request will be put on (as long as the waitqueue is active).
Re: [PATCH] io stalls
Quoted 25 lines · Nick Piggin
Quoted 23 lines · Andrea Arcangeli
Quoted 20 lines · Chris Mason
Quoted 9 lines · Andrea Arcangeli
drivers/block/ll_rw_blk.c line 932
Quoted 5 lines · Chris Mason
drivers/block/ll_rw_blk.c +4
-0+if (q->rq[rw].count >= q->batch_requests) { +smp_mb (); +if (waitqueue_active (&q->wait_for_requests[rw])) +wake_up (&q->wait_for_requests[rw]);in my tree I also changed this to:
Quoted 6 lines
wake_up_nr(&q->wait_for_requests[rw], q->rq[rw].count);otherwise only one waiter will eat the requests, while multiple waiters can eat requests in parallel instead because we freed not just 1 request but many of them.
I tried a few variations of this yesterday and they all led to horrible latencies, but I couldn't really explain why. I had a bunch of other
Quoted 7 lines
Quoted 4 lines
the I/O latency in theory shouldn't change, we're not reordering the queue at all, they'll go to sleep immediatly again if
__get_requestreturns null.And go to the end of the queue?
This got dragged into private mail for a few messages, but we figured out the problem turns into scheduling fairness with wake_up_nr()
32 procs might get woken, but when the first of those procs gets a request, he'll wake another, and so on. But there's no promise that getting woken fairly means you'll get scheduled fairly, so you might not get scheduled in for quite a while, perhaps even after new requests have gone onto the wait queue and gotten woken up.
The real problem is get_request_wait_wakeup, andrea is working on a few changes to that.
-chris
Re: [PATCH] io stalls
In reply to Nick PigginQuoted 28 lines · Nick Piggin
Quoted 24 lines · Andrea Arcangeli
Quoted 21 lines · Chris Mason
Quoted 9 lines · Andrea Arcangeli
drivers/block/ll_rw_blk.c line 932
Quoted 5 lines · Chris Mason
drivers/block/ll_rw_blk.c +4
-0+if (q->rq[rw].count >= q->batch_requests) { +smp_mb (); +if (waitqueue_active (&q->wait_for_requests[rw])) +wake_up (&q->wait_for_requests[rw]);in my tree I also changed this to:
Quoted 7 lines
wake_up_nr(&q->wait_for_requests[rw], q->rq[rw].count);otherwise only one waiter will eat the requests, while multiple waiters can eat requests in parallel instead because we freed not just 1 request but many of them.
I tried a few variations of this yesterday and they all led to horrible latencies, but I couldn't really explain why. I had a bunch of other
Quoted 6 lines
Quoted 4 lines
the I/O latency in theory shouldn't change, we're not reordering the queue at all, they'll go to sleep immediatly again if
__get_requestreturns null.And go to the end of the queue?
they stay in queue, so they don't go to the end.
but as Chris found since we've the get_request_wait_wakeup, even waking free-requests/2 isn't enough since that will generate free-requests *1.5 of wakeups where the last free-requests/2 (implicitly generated by the get_request_wait_wakeup) will become runnable and they will race with the other tasks later waken by another request release.
I sort of fixed that by remebering an old suggestion from Andrew:
static void get_request_wait_wakeup(request_queue_t *q, int rw)
{
/*
* avoid losing an unplug if a second __get_request_wait did the
* generic_unplug_device while our __get_request_wait was
* running
* w/o the queue_lock held and w/ our request out of the queue.
*/
if (waitqueue_active(&q->wait_for_requests))
run_task_queue(&tq_disk);
}this will avoid get_request_wait_wakeup to mess the wakeup, so we can wakep_nr(rq.count) safely.
then there's the last issue raised by Chris, that is if we get request released faster than the tasks can run, still we can generate a not perfect fairness. My solution to that is to change wake_up to have a nr_exclusive not obeying to the try_to_wakeup retval. that should guarantee exact FIFO then, but it's a minor issue because the requests shouldn't be released systematically in a flood. So I'm leaving it opened for now, the others already addressed should be the major ones.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
stuff in at the time to try and improve throughput though, so I'll try it again.
I think part of the problem is the cascading wakeups from
get_request_wait_wakeup(). So if we wakeup 32 procs they in turn wakeup another 32, etc.
Quoted 6 lines
Quoted 4 lines
so maybe it's enough to wakeup count / 2 to account for the double wakeup? that will avoid some overscheduling indeed.
Andrea, this isn't needed because when the queue falls below
actually the problem is that it isn't enough, not that it isn't needed. I had to stop get_request_wait_wakeup to mess the wakeups, so now I can return doing /2.
Quoted 3 lines
the batch limit, every retiring request will do a wake up and another request will be put on (as long as the waitqueue is active).
this was my argument for doing /2, but that will lead to count * 1.5 of wakeups, where the last count /2 will race with further wakeups screwing the FIFO ordering. As said that's fixed completely now and the last issue is the one with the flood of request release that can't keep up with the tasks becoming runnable but it's hopefully a minor issue (I'm not going to change how wake_up_nr works right now, maybe later).
Andrea
Re: [PATCH] io stalls
Quoted 10 lines · Andrea Arcangeli
static void
get_request_wait_wakeup(request_queue_t *q, int rw){ /* * avoid losing an unplug if a second __get_request_wait did the * generic_unplug_device while our __get_request_wait was * running * w/o the queue_lock held and w/ our request out of the queue. */ if (waitqueue_active(&q->wait_for_requests)) run_task_queue(&tq_disk);
btw, that was the old version, Chris did it right s/run_task_queue(&tq_disk)/__generic_unplug_device(q)/
Andrea
Re: [PATCH] io stalls
In reply to Andrea ArcangeliQuoted 10 lines · Andrea Arcangeli
this will avoid
get_request_wait_wakeupto mess the wakeup, so we canwakep_nr(rq.count)safely.then there's the last issue raised by Chris, that is if we get request released faster than the tasks can run, still we can generate a not perfect fairness. My solution to that is to change
wake_upto have anr_exclusivenot obeying to thetry_to_wakeupretval. that should guarantee exact FIFO then, but it's a minor issue because the requests shouldn't be released systematically in a flood. So I'm leaving it opened for now, the others already addressed should be the major ones.
I think the only time we really need to wakeup more than one waiter is when we hit the q->batch_request mark. After that, each new request that is freed can be matched with a single waiter, and we know that any previously finished requests have probably already been matched to their own waiter.
-chris
Re: [PATCH] io stalls
Quoted 15 lines · Chris Mason
Quoted 11 lines · Andrea Arcangeli
this will avoid
get_request_wait_wakeupto mess the wakeup, so we canwakep_nr(rq.count)safely.then there's the last issue raised by Chris, that is if we get request released faster than the tasks can run, still we can generate a not perfect fairness. My solution to that is to change
wake_upto have anr_exclusivenot obeying to thetry_to_wakeupretval. that should guarantee exact FIFO then, but it's a minor issue because the requests shouldn't be released systematically in a flood. So I'm leaving it opened for now, the others already addressed should be the major ones.
Quoted 7 lines
I think the only time we really need to wakeup more than one waiter is when we hit the
q->batch_requestmark. After that, each new request that is freed can be matched with a single waiter, and we know that any previously finished requests have probably already been matched to their own waiter.
Nope. Not even then. Each retiring request should submit a wake up, and the process will submit another request. So the number of requests will be held at the batch_request mark until no more waiters.
Now that begs the question, why have batch_requests anymore? It no longer does anything.
Re: [PATCH] io stalls
Quoted 26 lines · Nick Piggin
Quoted 15 lines · Chris Mason
Quoted 11 lines · Andrea Arcangeli
this will avoid
get_request_wait_wakeupto mess the wakeup, so we canwakep_nr(rq.count)safely.then there's the last issue raised by Chris, that is if we get request released faster than the tasks can run, still we can generate a not perfect fairness. My solution to that is to change
wake_upto have anr_exclusivenot obeying to thetry_to_wakeupretval. that should guarantee exact FIFO then, but it's a minor issue because the requests shouldn't be released systematically in a flood. So I'm leaving it opened for now, the others already addressed should be the major ones.Quoted 7 lines
I think the only time we really need to wakeup more than one waiter is when we hit the
q->batch_requestmark. After that, each new request that is freed can be matched with a single waiter, and we know that any previously finished requests have probably already been matched to their own waiter.
Quoted 7 lines
Nope. Not even then. Each retiring request should submit a wake up, and the process will submit another request. So the number of requests will be held at the
batch_requestmark until no more waiters.Now that begs the question, why have
batch_requestsanymore? It no longer does anything.
it does nothing w/ _exclusive and w/o the wake_up_nr, that's why I added the wake_up_nr.
Andrea
Re: [PATCH] io stalls
Quoted 35 lines · Andrea Arcangeli
Quoted 33 lines · Nick Piggin
Quoted 16 lines · Chris Mason
Quoted 12 lines · Andrea Arcangeli
this will avoid
get_request_wait_wakeupto mess the wakeup, so we canwakep_nr(rq.count)safely.then there's the last issue raised by Chris, that is if we get request released faster than the tasks can run, still we can generate a not perfect fairness. My solution to that is to change
wake_upto have anr_exclusivenot obeying to thetry_to_wakeupretval. that should guarantee exact FIFO then, but it's a minor issue because the requests shouldn't be released systematically in a flood. So I'm leaving it opened for now, the others already addressed should be the major ones.Quoted 8 lines
I think the only time we really need to wakeup more than one waiter is when we hit the
q->batch_requestmark. After that, each new request that is freed can be matched with a single waiter, and we know that any previously finished requests have probably already been matched to their own waiter.Nope. Not even then. Each retiring request should submit a wake up, and the process will submit another request. So the number of requests will be held at the
batch_requestmark until no more waiters.
Quoted 8 lines
Quoted 3 lines
Now that begs the question, why have
batch_requestsanymore? It no longer does anything.it does nothing w/
_exclusiveand w/o thewake_up_nr, that's why I added thewake_up_nr.
That is pretty pointless as well. You might as well just start waking up at the queue full limit, and wake one at a time.
The purpose for batch_requests was I think for devices with a very small request size, to reduce context switches.
Quoted 4 lines
Andrea
Re: [PATCH] io stalls
Quoted 47 lines · Nick Piggin
Quoted 35 lines · Andrea Arcangeli
Quoted 33 lines · Nick Piggin
Quoted 16 lines · Chris Mason
Quoted 12 lines · Andrea Arcangeli
this will avoid
get_request_wait_wakeupto mess the wakeup, so we canwakep_nr(rq.count)safely.then there's the last issue raised by Chris, that is if we get request released faster than the tasks can run, still we can generate a not perfect fairness. My solution to that is to change
wake_upto have anr_exclusivenot obeying to thetry_to_wakeupretval. that should guarantee exact FIFO then, but it's a minor issue because the requests shouldn't be released systematically in a flood. So I'm leaving it opened for now, the others already addressed should be the major ones.Quoted 8 lines
I think the only time we really need to wakeup more than one waiter is when we hit the
q->batch_requestmark. After that, each new request that is freed can be matched with a single waiter, and we know that any previously finished requests have probably already been matched to their own waiter.Nope. Not even then. Each retiring request should submit a wake up, and the process will submit another request. So the number of requests will be held at the
batch_requestmark until no more waiters.Quoted 8 lines
Quoted 3 lines
Now that begs the question, why have
batch_requestsanymore? It no longer does anything.it does nothing w/
_exclusiveand w/o thewake_up_nr, that's why I added thewake_up_nr.
Quoted 5 lines
That is pretty pointless as well. You might as well just start waking up at the queue full limit, and wake one at a time.
The purpose for
batch_requestswas I think for devices with a very small request size, to reduce context switches.
I guess you could fix this by having a "last woken" flag, and allow that process to allocate requests without blocking from the batch limit until the queue full limit. That is how batch_requests is supposed to work.
Re: [PATCH] io stalls
Quoted 5 lines · Nick Piggin
I guess you could fix this by having a "last woken" flag, and allow that process to allocate requests without blocking from the batch limit until the queue full limit. That is how
batch_requestsis supposed to work.
s/flag/pid maybe?
Re: [PATCH] io stalls
In reply to Nick PigginQuoted 4 lines · Nick Piggin
I guess you could fix this by having a "last woken" flag, and allow that process to allocate requests without blocking from the batch limit until the queue full limit. That is how
batch_requestsis supposed to work.
I see what you mean, I did care about the case of each request belonging to a different task, but of course this doesn't work if there's just one task. In such case there will be a single wakeup and one for each request, so it won't be able to eat all the requests and it'll keep hanging on the full bitflag. So yes, the ->full bit partly disabled the batch sectors in presence of only 1 task. With multiple tasks and the wake_up_nr batch_sectors will still work. However I don't care about that right now ;), it's a minor issue I guess, single task I/O normally doesn't seek heavily so more likely it will run into the oversized queue before being able to take advantage of the batch sectors.
Andrea
Re: [PATCH] io stalls
In reply to Nick PigginQuoted 47 lines · Nick Piggin
Quoted 35 lines · Andrea Arcangeli
Quoted 33 lines · Nick Piggin
Quoted 16 lines · Chris Mason
Quoted 12 lines · Andrea Arcangeli
this will avoid
get_request_wait_wakeupto mess the wakeup, so we canwakep_nr(rq.count)safely.then there's the last issue raised by Chris, that is if we get request released faster than the tasks can run, still we can generate a not perfect fairness. My solution to that is to change
wake_upto have anr_exclusivenot obeying to thetry_to_wakeupretval. that should guarantee exact FIFO then, but it's a minor issue because the requests shouldn't be released systematically in a flood. So I'm leaving it opened for now, the others already addressed should be the major ones.Quoted 8 lines
I think the only time we really need to wakeup more than one waiter is when we hit the
q->batch_requestmark. After that, each new request that is freed can be matched with a single waiter, and we know that any previously finished requests have probably already been matched to their own waiter.Nope. Not even then. Each retiring request should submit a wake up, and the process will submit another request. So the number of requests will be held at the
batch_requestmark until no more waiters.Quoted 8 lines
Quoted 3 lines
Now that begs the question, why have
batch_requestsanymore? It no longer does anything.it does nothing w/
_exclusiveand w/o thewake_up_nr, that's why I added thewake_up_nr.
Quoted 5 lines
That is pretty pointless as well. You might as well just start waking up at the queue full limit, and wake one at a time.
The purpose for
batch_requestswas I think for devices with a very small request size, to reduce context switches.
batch_requests at least in my tree matters only when each request is 512btyes and you've some thousand of them to compose a 4M queue or so. To maximize cpu cache usage etc.. I try to wakeup a task every 512bytes written, but every 32*512bytes written or so. Of course w/o the wake_up_nr that I added, that wasn't really working w/ the _exlusive wakeup.
if you check my tree you'll see that for sequential I/O with 512k in each request (not 512bytes!) batch_requests is already a noop.
Andrea
Re: [PATCH] io stalls
Quoted 24 lines · Andrea Arcangeli
Quoted 8 lines · Nick Piggin
Quoted 5 lines · Andrea Arcangeli
it does nothing w/
_exclusiveand w/o thewake_up_nr, that's why I added thewake_up_nr.Quoted 6 lines
That is pretty pointless as well. You might as well just start waking up at the queue full limit, and wake one at a time.
The purpose for
batch_requestswas I think for devices with a very small request size, to reduce context switches.
batch_requestsat least in my tree matters only when each request is 512btyes and you've some thousand of them to compose a 4M queue or so. To maximize cpu cache usage etc.. I try to wakeup a task every 512bytes written, but every 32*512bytes written or so. Of course w/o thewake_up_nrthat I added, that wasn't really working w/ the_exlusivewakeup.
Quoted 3 lines
if you check my tree you'll see that for sequential I/O with 512k in each request (not 512bytes!)
batch_requestsis already a noop.
You are waking up multiple tasks which will each submit 1 request. You want to be waking up 1 task which will submit multiple requests - that is how you will save context switches, cpu cache, etc, and that task's requests will have a much better chance of being merged, or at least serviced as a nice batch than unrelated tasks.
Re: [PATCH] io stalls
Quoted 33 lines · Nick Piggin
Quoted 24 lines · Andrea Arcangeli
Quoted 8 lines · Nick Piggin
Quoted 5 lines · Andrea Arcangeli
it does nothing w/
_exclusiveand w/o thewake_up_nr, that's why I added thewake_up_nr.Quoted 6 lines
That is pretty pointless as well. You might as well just start waking up at the queue full limit, and wake one at a time.
The purpose for
batch_requestswas I think for devices with a very small request size, to reduce context switches.
batch_requestsat least in my tree matters only when each request is 512btyes and you've some thousand of them to compose a 4M queue or so. To maximize cpu cache usage etc.. I try to wakeup a task every 512bytes written, but every 32*512bytes written or so. Of course w/o thewake_up_nrthat I added, that wasn't really working w/ the_exlusivewakeup.Quoted 3 lines
if you check my tree you'll see that for sequential I/O with 512k in each request (not 512bytes!)
batch_requestsis already a noop.
Quoted 6 lines
You are waking up multiple tasks which will each submit 1 request. You want to be waking up 1 task which will submit multiple requests - that is how you will save context switches, cpu cache, etc, and that task's requests will have a much better chance of being merged, or at least serviced as a nice batch than unrelated tasks.
for fairness reasons if there are multiple tasks, I want to wake them all and let the others be able to eat requests before the first allocates all the batch_sectors. So the current code is fine and batch_sectors still works fine with multiple tasks queued in the waitqueue, it still makes sense to wake more than one of them at the same time to improve cpu utilization (regardless they're different tasks, for istance we take less frequently the waitqueue spinlocks etc..).
What we disabled is only the batch_sectors in function of the single task, so if for example there's just 1 single task, we could let it go, but it's quite a special case, if for example there would be two tasks, we wouldn't want to let them go ahead (unless we can distributed exactly count/2 requests to each task w/o reentering into __get_request_wait that's unlikely). So the current code looks ok to me with the wake_up_nr to take advantage of the batch_sectors against different tasks, still w/o penalizing fariness.
Andrea
Re: [PATCH] io stalls
Quoted 45 lines · Andrea Arcangeli
Quoted 35 lines · Nick Piggin
Quoted 26 lines · Andrea Arcangeli
Quoted 9 lines · Nick Piggin
Quoted 6 lines · Andrea Arcangeli
it does nothing w/
_exclusiveand w/o thewake_up_nr, that's why I added thewake_up_nr.Quoted 7 lines
That is pretty pointless as well. You might as well just start waking up at the queue full limit, and wake one at a time.
The purpose for
batch_requestswas I think for devices with a very small request size, to reduce context switches.
batch_requestsat least in my tree matters only when each request is 512btyes and you've some thousand of them to compose a 4M queue or so. To maximize cpu cache usage etc.. I try to wakeup a task every 512bytes written, but every 32*512bytes written or so. Of course w/o thewake_up_nrthat I added, that wasn't really working w/ the_exlusivewakeup.Quoted 4 lines
if you check my tree you'll see that for sequential I/O with 512k in each request (not 512bytes!)
batch_requestsis already a noop.Quoted 7 lines
You are waking up multiple tasks which will each submit 1 request. You want to be waking up 1 task which will submit multiple requests - that is how you will save context switches, cpu cache, etc, and that task's requests will have a much better chance of being merged, or at least serviced as a nice batch than unrelated tasks.
Quoted 9 lines
for fairness reasons if there are multiple tasks, I want to wake them all and let the others be able to eat requests before the first allocates all the
batch_sectors. So the current code is fine andbatch_sectorsstill works fine with multiple tasks queued in the waitqueue, it still makes sense to wake more than one of them at the same time to improve cpu utilization (regardless they're different tasks, for istance we take less frequently the waitqueue spinlocks etc..).
Its no less fair this way, tasks will still be woken in fifo order. They will just be given the chance to submit a batch of requests.
I think the cpu utilization gain of waking a number of tasks at once would be outweighed by advantage of waking 1 task and not putting it to sleep again for a number of requests. You obviously are not claiming concurrency improvements, as your method would also increase contention on the io lock (or the queue lock in 2.5).
Then you have the cache gains of running each task for a longer period of time. You also get possible IO scheduling improvements.
Consider 8 requests, batch_requests at 4, 10 tasks writing to different areas of disk.
Your method still only allows each task to have 1 request in the elevator at once. Mine allows each to have a run of 4 requests in the elevator.
Re: [PATCH] io stalls
Quoted 3 lines · Nick Piggin
Its no less fair this way, tasks will still be woken in fifo order. They will just be given the chance to submit a batch of requests.
If you change the behaviour with queued_task_nr > batch_requests it is less fair period. Whatever else thing I don't care about right now because it is a minor cpu improvement anyways.
I'm not talking about performance, I'm talking about latency and fariness only. This is the whole point of the ->full logic.
Quoted 6 lines
I think the cpu utilization gain of waking a number of tasks at once would be outweighed by advantage of waking 1 task and not putting it to sleep again for a number of requests. You obviously are not claiming concurrency improvements, as your method would also increase contention on the io lock (or the queue lock in 2.5).
I'm claiming that with queued_task_nr > batch_requests the batch_requests logic still has a chance to save some cpu, this is the only reason I didn't nuke it completely as you suggested some email ago.
Quoted 10 lines
Then you have the cache gains of running each task for a longer period of time. You also get possible IO scheduling improvements.
Consider 8 requests,
batch_requestsat 4, 10 tasks writing to different areas of disk.Your method still only allows each task to have 1 request in the elevator at once. Mine allows each to have a run of 4 requests in the elevator.
I definitely want 1 request in the elevator at once or we can as well drop your ->full and return to be unfair. The whole point of ->full is to get the total fariness, across the tasks in the queue queue, and for tasks outside the queue calling get_request too. Since not all tasks will fit in the I/O queue, providing a very fair FIFO in the wait_for_request is fundamental to provide any sort of latency guarantee IMHO (the fact an _exclusive wakeup removal that mixes stuff and probably has the side effect of being more fair, made that much difference to mainline users kind of confirms that).
Andrea
Re: [PATCH] io stalls
Quoted 7 lines · Andrea Arcangeli
Quoted 4 lines · Nick Piggin
Its no less fair this way, tasks will still be woken in fifo order. They will just be given the chance to submit a batch of requests.
Quoted 7 lines
If you change the behaviour with
queued_task_nr>batch_requestsit is less fair period. Whatever else thing I don't care about right now because it is a minor cpu improvement anyways.I'm not talking about performance, I'm talking about latency and fariness only. This is the whole point of the ->full logic.
I say each task getting 8 requests at a time is as fair as each getting 1 request at a time. Yes, you may get a worse latency, but _fairness_ is the same.
Quoted 9 lines
Quoted 8 lines
I think the cpu utilization gain of waking a number of tasks at once would be outweighed by advantage of waking 1 task and not putting it to sleep again for a number of requests. You obviously are not claiming concurrency improvements, as your method would also increase contention on the io lock (or the queue lock in 2.5).
Quoted 4 lines
I'm claiming that with
queued_task_nr>batch_requeststhebatch_requestslogic still has a chance to save some cpu, this is the only reason I didn't nuke it completely as you suggested some email ago.
Well I'm not so sure that your method will do a great deal of good. On SMP you would increase contention on the spinlock. IMO it would be better to serialise them on the waitqueue instead of a spinlock seeing as they are already sleeping.
Quoted 13 lines
Quoted 5 lines
Then you have the cache gains of running each task for a longer period of time. You also get possible IO scheduling improvements.
Quoted 7 lines
Consider 8 requests,
batch_requestsat 4, 10 tasks writing to different areas of disk.Your method still only allows each task to have 1 request in the elevator at once. Mine allows each to have a run of 4 requests in the elevator.
Quoted 11 lines
I definitely want 1 request in the elevator at once or we can as well drop your ->full and return to be unfair. The whole point of ->full is to get the total fariness, across the tasks in the queue queue, and for tasks outside the queue calling
get_requesttoo. Since not all tasks will fit in the I/O queue, providing a very fair FIFO in thewait_for_requestis fundamental to provide any sort of latency guarantee IMHO (the fact an_exclusivewakeup removal that mixes stuff and probably has the side effect of being more fair, made that much difference to mainline users kind of confirms that).
I think we'll just have to agree to disagree here. This sort of thing came up in our CFQ discussion as well. Its not that I think your way is without merits, but I think in an overload situtation its a better aim to attempt to keep throughput up rather than attempt to provide the lowest possible latency.
Re: [PATCH] io stalls
Quoted 15 lines · Nick Piggin
Quoted 11 lines · Andrea Arcangeli
Quoted 4 lines · Nick Piggin
Its no less fair this way, tasks will still be woken in fifo order. They will just be given the chance to submit a batch of requests.
If you change the behaviour with
queued_task_nr>batch_requestsit is less fair period. Whatever else thing I don't care about right now because it is a minor cpu improvement anyways.
Quoted 7 lines
Quoted 3 lines
I'm not talking about performance, I'm talking about latency and fariness only. This is the whole point of the ->full logic.
I say each task getting 8 requests at a time is as fair as each getting 1 request at a time. Yes, you may get a worse latency, but _fairness_ is the same.
It is the worse latency that is the problem of course. Fariness in this case isn't affected (assuming you would write the batch_sectors fair), but latency would definitely be affected.
Quoted 4 lines
Well I'm not so sure that your method will do a great deal of good. On SMP you would increase contention on the spinlock. IMO it would be better to serialise them on the waitqueue instead of a spinlock seeing as they are already sleeping.
I think the worse part is the cacheline bouncing, but that might happen anyways under load. at least certainly it makes sense for UP or if you're lucky and the tasks run serially (possible if all cpus are running).
Quoted 6 lines
I think we'll just have to agree to disagree here. This sort of thing came up in our CFQ discussion as well. Its not that I think your way is without merits, but I think in an overload situtation its a better aim to attempt to keep throughput up rather than attempt to provide the lowest possible latency.
Those changes (like the CFQ I/O scheduler in 2.5) are being developed mostly due the latency complains we get as feedback on l-k. That's why I care about latency first here. But we've to care about throughput too of course. This isn't CFQ, it just tries to provide new requests to different tasks with the minimal possible latency which in turn also guarantees fariness. That sounds a good default to me, especially when I see the removal of the _exclusive wakeup in mainline taken as a major fariness/latency improvement.
Andrea
Re: [PATCH] io stalls
Quoted 22 lines · Andrea Arcangeli
Quoted 20 lines · Nick Piggin
Quoted 16 lines · Andrea Arcangeli
Quoted 5 lines · Nick Piggin
Its no less fair this way, tasks will still be woken in fifo order. They will just be given the chance to submit a batch of requests.
If you change the behaviour with
queued_task_nr>batch_requestsit is less fair period. Whatever else thing I don't care about right now because it is a minor cpu improvement anyways.I'm not talking about performance, I'm talking about latency and fariness only. This is the whole point of the ->full logic.
Quoted 9 lines
Quoted 4 lines
I say each task getting 8 requests at a time is as fair as each getting 1 request at a time. Yes, you may get a worse latency, but _fairness_ is the same.
It is the worse latency that is the problem of course. Fariness in this case isn't affected (assuming you would write the
batch_sectorsfair), but latency would definitely be affected.
Yep.
Quoted 21 lines
Quoted 6 lines
Well I'm not so sure that your method will do a great deal of good. On SMP you would increase contention on the spinlock. IMO it would be better to serialise them on the waitqueue instead of a spinlock seeing as they are already sleeping.
I think the worse part is the cacheline bouncing, but that might happen anyways under load. at least certainly it makes sense for UP or if you're lucky and the tasks run serially (possible if all cpus are running).
Quoted 7 lines
I think we'll just have to agree to disagree here. This sort of thing came up in our CFQ discussion as well. Its not that I think your way is without merits, but I think in an overload situtation its a better aim to attempt to keep throughput up rather than attempt to provide the lowest possible latency.
Quoted 9 lines
Those changes (like the CFQ I/O scheduler in 2.5) are being developed mostly due the latency complains we get as feedback on l-k. That's why I care about latency first here. But we've to care about throughput too of course. This isn't CFQ, it just tries to provide new requests to different tasks with the minimal possible latency which in turn also guarantees fariness. That sounds a good default to me, especially when I see the removal of the
_exclusivewakeup in mainline taken as a major fariness/latency improvement.
Throughput vs latency is always difficult I guess. In this case, I think when there are few waiters, then latency should not be much worse. When there are a lot of waiters it is probably not an interactive load to start with and throughput is more important.
Anyway, the ideas you are following are interesting and worthwhile, so we'll each try our own thing :)
Re: [PATCH] io stalls
In reply to Nick PigginQuoted 7 lines · Nick Piggin
I think the cpu utilization gain of waking a number of tasks at once would be outweighed by advantage of waking 1 task and not putting it to sleep again for a number of requests. You obviously are not claiming concurrency improvements, as your method would also increase contention on the io lock (or the queue lock in 2.5).
I've been trying variations on this for a few days, none have been thrilling but the end result is better dbench and iozone throughput overall. For the 20 writer iozone test, rc7 got an average throughput of 3MB/s, and yesterdays latency patch got 500k/s or so. Ouch.
This gets us up to 1.2MB/s. I'm keeping yesterday's get_request_wait_wake, which wakes up a waiter instead of unplugging.
The basic idea here is that after a process is woken up and grabs a request, he becomes the batch owner. Batch owners get to ignore the q->full flag for either 1/5 second or 32 requests, whichever comes first. The timer part is an attempt at preventing memory pressure writers (who go 1 req at a time) from holding onto batch ownership for too long. Latency stats after dbench 50:
device 08:01: num_req 120077, total jiffies waited 663231
65538 forced to wait
1 min wait, 175 max wait
10 average wait
65296 < 100, 242 < 200, 0 < 300, 0 < 400, 0 < 500 0 waits longer than 500 jiffies
Good latency system wide comes from fair waiting, but it also comes from how fast we can run write_some_buffers(), since that is the unit of throttling. Hopefully this patch decreases the time it takes for write_some_buffers over the past latency patches, or gives someone else a better idea ;-)
Attached is an incremental over yesterday's io-stalls-5.diff.
-chris
drivers/block/ll_rw_blk.c +44 -5
@@ -437,6 +437,12 @@ nr_requests =128 ;if (megs <32 ) nr_requests /=2 ; + q->batch_owner[0 ] =NULL ; + q->batch_owner[1 ] =NULL ; + q->batch_remaining[0 ] =0 ; + q->batch_remaining[1 ] =0 ; + q->batch_jiffies[0 ] =0 ; + q->batch_jiffies[1 ] =0 ;blk_grow_request_list (q, nr_requests);init_waitqueue_head (&q->wait_for_requests[0 ]);
@@ -558,6 +564,31 @@blk_queue_bounce_limit (q,BLK_BOUNCE_HIGH ); } +#define BATCH_JIFFIES (HZ/5) +static void check_batch_owner (request_queue_t *q,int rw) +{ +if (q->batch_owner[rw] != current) +return ; +if (--q->batch_remaining[rw] >0 && + jiffies - q->batch_jiffies[rw] <BATCH_JIFFIES ) { +return ; + } + q->batch_owner[rw] =NULL ; +} + +static void set_batch_owner (request_queue_t *q,int rw) +{ +struct task_struct *tsk = current; +if (q->batch_owner[rw] == tsk) +return ; +if (q->batch_owner[rw] && + jiffies - q->batch_jiffies[rw] <BATCH_JIFFIES ) +return ; + q->batch_jiffies[rw] = jiffies; + q->batch_owner[rw] = current; + q->batch_remaining[rw] = q->batch_requests; +} +#define blkdev_free_rq (list) list_entry((list)->next, struct request, queue);/* * Get a free request. io_request_lock must be held and interrupts
@@ -587,9 +618,13 @@*/ static inline struct request *get_request (request_queue_t *q,int rw) { -if (queue_full (q, rw)) +struct request *rq; +if (queue_full (q, rw) && q->batch_owner[rw] != current)return NULL ; -return __get_request (q, rw); + rq =__get_request (q, rw); +if (rq) +check_batch_owner (q, rw); +return rq; }/*
@@ -657,9 +692,9 @@add_wait_queue_exclusive (&q->wait_for_requests[rw], &wait); +spin_lock_irq (&io_request_lock); do { set_current_state(TASK_UNINTERRUPTIBLE); - spin_lock_irq(&io_request_lock); if (queue_full(q, rw) || q->rq[rw].count == 0) { if (q->rq[rw].count == 0) __generic_unplug_device(q);
@@ -668,8 +703,9 @@spin_lock_irq (&io_request_lock); } rq =__get_request (q, rw); -spin_unlock_irq (&io_request_lock); }while (rq ==NULL ); +set_batch_owner (q, rw); +spin_unlock_irq (&io_request_lock);remove_wait_queue (&q->wait_for_requests[rw], &wait); current->state =TASK_RUNNING ;
@@ -1010,6 +1046,7 @@struct list_head *head, *insert_here;int latency;elevator_t *elevator = &q->elevator; +int need_unplug =0 ; count = bh->b_size >>9 ; sector = bh->b_rsector;
@@ -1145,8 +1182,8 @@spin_unlock_irq (&io_request_lock); freereq =__get_request_wait (q, rw); head = &q->queue_head; + need_unplug =1 ;spin_lock_irq (&io_request_lock); -get_request_wait_wakeup (q, rw); goto again; } }
@@ -1174,6 +1211,8 @@ out:if (freereq)blkdev_release_request (freereq); +if (need_unplug) +get_request_wait_wakeup (q, rw);spin_unlock_irq (&io_request_lock);return 0 ; end_io:
include/linux/blkdev.h +4 -0
@@ -92,6 +92,10 @@*/ int batch_requests; +struct task_struct *batch_owner[2 ]; +int batch_remaining[2 ]; +unsigned long batch_jiffies[2 ]; +/* * Together with queue_head for cacheline sharing */
Re: [PATCH] io stalls
Quoted 19 lines · Chris Mason
Quoted 7 lines · Nick Piggin
I think the cpu utilization gain of waking a number of tasks at once would be outweighed by advantage of waking 1 task and not putting it to sleep again for a number of requests. You obviously are not claiming concurrency improvements, as your method would also increase contention on the io lock (or the queue lock in 2.5).
I've been trying variations on this for a few days, none have been thrilling but the end result is better dbench and iozone throughput overall. For the 20 writer iozone test, rc7 got an average throughput of 3MB/s, and yesterdays latency patch got 500k/s or so. Ouch.
This gets us up to 1.2MB/s. I'm keeping yesterday's
get_request_wait_wake, which wakes up a waiter instead of unplugging.
Quoted 7 lines
The basic idea here is that after a process is woken up and grabs a request, he becomes the batch owner. Batch owners get to ignore the
q->fullflag for either 1/5 second or 32 requests, whichever comes first. The timer part is an attempt at preventing memory pressure writers (who go 1 req at a time) from holding onto batch ownership for too long. Latency stats after dbench 50:
Yeah, I get ~50% more throughput and up to 20% better CPU efficiency on tiobench 256 for sequential and random writers by doing something similar.
Re: [PATCH] io stalls
In reply to Andrea ArcangeliHello all,
[ short version, the patch attached should fix io latencies in 2.4.21. Please review and/or give it a try ]
My last set of patches was directed at reducing the latencies in __get_request_wait, which really helped reduce stalls when you had lots of io to one device and balance_dirty() was causing pauses while you tried to do io to other devices.
But, a streaming write could still starve reads to the same device, mostly because the read would have to send down any huge merged writes that were before it in the queue.
Andrea's kernel has a fix for this too, he limits the total number of sectors that can be in the request queue at any given time. But, his patches change blk_finished_io, both in the arguments it takes and the side effects of calling it. I don't think we can merge his current form without breaking external drivers.
So, I added a can_throttle flag to the queue struct, drivers can enable it if they are going to call the new blk_started_sectors and blk_finished_sectors funcs any time they call blk_{started,finished}_io, and these do all the -aa style sector throttling.
There were a few other small changes to Andrea's patch, he wasn't setting q->full when get_request decided there were too many sectors in flight. This resulted in large latencies in __get_request_wait. He was also unconditionally clearing q->full in blkdev_release_request, my code only clears q->full when all the waiters are gone.
I changed generic_unplug_device to zero the elevator_sequence field of the last request on the queue. This means there won't be any merges with requests pending once an unplug is done, and helps limit the number of sectors that need to be sent down during the run_task_queue(&tq_disk) in wait_on_buffer.
I lowered the -aa default limit on sectors in flight from 4MB to 2MB. We probably want an elvtune for it, large arrays with writeback cache should be able to tolerate larger values.
There's still a little work left to do, this patch enables sector throttling for scsi and IDE. cciss, DAC960 and cpqarray need modification too (99% done already in -aa). No sense in doing that until after the bulk of the patch is reviewed though.
As before, most of the code here is from Andrea and Nick, I've just wrapped a lot of duct tape around it and done some tweaking. The primary pieces are:
fix-pausing (andrea, corner cases where wakeups are missed) elevator-low-latency (andrea, limit sectors in flight) queue_full (Nick, fairness in __get_request_wait)
I've removed my latency stats for __get_request_wait in hopes of making it a better merging candidate.
-chris
drivers/block/ll_rw_blk.c +177 -71
@@ -176,11 +176,12 @@ {int count = q->nr_requests; - count -=__blk_cleanup_queue (&q->rq[READ ]); - count -=__blk_cleanup_queue (&q->rq[WRITE ]); + count -=__blk_cleanup_queue (&q->rq);if (count)printk ("blk_cleanup_queue: leaked requests (%d)\n" , count); +if (atomic_read (&q->nr_sectors)) +printk ("blk_cleanup_queue: leaked sectors (%d)\n" ,atomic_read (&q->nr_sectors));memset (q,0 ,sizeof (*q)); }
@@ -215,6 +216,24 @@ }/** +* blk_queue_throttle_sectors - indicates you will call sector throttling funcs +* @q: The queue which this applies to. +* @active: A flag indication if you want sector throttling on +* +* Description: +* The sector throttling code allows us to put a limit on the number of +* sectors pending io to the disk at a given time, sending @active nonzero +* indicates you will call blk_started_sectors and blk_finished_sectors in +* addition to calling blk_started_io and blk_finished_io in order to +* keep track of the number of sectors in flight. +**/ + +void blk_queue_throttle_sectors (request_queue_t * q,int active) +{ + q->can_throttle = active; +} + +/** * blk_queue_make_request - define an alternate make_request function for a device * @q: the request queue for the device to be affected * @mfn: the alternate make_request function
@@ -360,8 +379,20 @@ {if (q->plugged) { q->plugged =0 ; -if (!list_empty (&q->queue_head)) +if (!list_empty (&q->queue_head)) { +struct request *rq; + +/* we don't want merges later on to come in +* and significantly increase the amount of +* work during an unplug, it can lead to high +* latencies while some poor waiter tries to +* run an ever increasing chunk of io. +* This does lower throughput some though. +*/ + rq =blkdev_entry_prev_request (&q->queue_head), + rq->elevator_sequence =0 ; q->request_fn (q); + } } }
@@ -389,7 +420,7 @@* * Returns the (new) number of requests which the queue has available. */ -int blk_grow_request_list (request_queue_t *q,int nr_requests) +int blk_grow_request_list (request_queue_t *q,int nr_requests,int max_queue_sectors) {unsigned long flags;/* Several broken drivers assume that this function doesn't sleep,
@@ -399,21 +430,34 @@spin_lock_irqsave (&io_request_lock, flags);while (q->nr_requests < nr_requests) {struct request *rq; -int rw; rq =kmem_cache_alloc (request_cachep,SLAB_ATOMIC );if (rq ==NULL )break ;memset (rq,0 ,sizeof (*rq)); rq->rq_status =RQ_INACTIVE ; - rw = q->nr_requests &1 ; -list_add (&rq->queue, &q->rq[rw].free); - q->rq[rw].count++; +list_add (&rq->queue, &q->rq.free); + q->rq.count++; + q->nr_requests++; } + +/* +* Wakeup waiters after both one quarter of the +* max-in-fligh queue and one quarter of the requests +* are available again. +*/ + q->batch_requests = q->nr_requests /4 ;if (q->batch_requests >32 ) q->batch_requests =32 ; + q->batch_sectors = max_queue_sectors /4 ; + + q->max_queue_sectors = max_queue_sectors; + +BUG_ON (!q->batch_sectors); +atomic_set (&q->nr_sectors,0 ); +spin_unlock_irqrestore (&io_request_lock, flags);return q->nr_requests; }
@@ -422,23 +466,27 @@ {struct sysinfo si;int megs;/* Total memory, in megabytes */ -int nr_requests; - -INIT_LIST_HEAD (&q->rq[READ ].free); -INIT_LIST_HEAD (&q->rq[WRITE ].free); - q->rq[READ ].count =0 ; - q->rq[WRITE ].count =0 ; +int nr_requests, max_queue_sectors =MAX_QUEUE_SECTORS ; + +INIT_LIST_HEAD (&q->rq.free); + q->rq.count =0 ; q->nr_requests =0 ;si_meminfo (&si); megs = si.totalram >> (20 -PAGE_SHIFT ); - nr_requests =128 ; -if (megs <32 ) - nr_requests /=2 ; -blk_grow_request_list (q, nr_requests); + nr_requests =MAX_NR_REQUESTS ; +if (megs <30 ) { + nr_requests /=2 ; + max_queue_sectors /=2 ; + } +/* notice early if anybody screwed the defaults */ +BUG_ON (!nr_requests); +BUG_ON (!max_queue_sectors); + +blk_grow_request_list (q, nr_requests, max_queue_sectors); + +init_waitqueue_head (&q->wait_for_requests); -init_waitqueue_head (&q->wait_for_requests[0 ]); -init_waitqueue_head (&q->wait_for_requests[1 ]);spin_lock_init (&q->queue_lock); }
@@ -491,6 +539,9 @@ q->plug_tq.routine = &generic_unplug_device; q->plug_tq.data = q; q->plugged =0 ; + q->full =0 ; + q->can_throttle =0 ; +/* * These booleans describe the queue properties. We set the * default (and most common) values here. Other drivers can
@@ -508,12 +559,13 @@* Get a free request. io_request_lock must be held and interrupts * disabled on the way in. Returns NULL if there are no free requests. */ -static struct request *get_request (request_queue_t *q,int rw) +static struct request *__get_request (request_queue_t *q,int rw) {struct request *rq =NULL ; -struct request_list *rl = q->rq + rw; +struct request_list *rl; -if (!list_empty (&rl->free)) { + rl = &q->rq; +if (!list_empty (&rl->free) && !blk_oversized_queue (q)) { rq =blkdev_free_rq (&rl->free);list_del (&rq->queue); rl->count--;
@@ -521,35 +573,47 @@ rq->cmd = rw; rq->special =NULL ; rq->q = q; - } - + }else + q->full =1 ;return rq; }/* -* Here's the request allocation design: +* get a free request, honoring the queue_full condition +*/ +static inline struct request *get_request (request_queue_t *q,int rw) +{ +if (q->full) +return NULL ; +return __get_request (q, rw); +} + +/* +* helper func to do memory barriers and wakeups when we finally decide +* to clear the queue full condition +*/ +static inline void clear_full_and_wake (request_queue_t *q) +{ + q->full =0 ; +mb (); +if (waitqueue_active (&q->wait_for_requests)) +wake_up (&q->wait_for_requests); +} + +/* +* Here's the request allocation design, low latency version: * * 1: Blocking on request exhaustion is a key part of I/O throttling. * * 2: We want to be `fair' to all requesters. We must avoid starvation, and * attempt to ensure that all requesters sleep for a similar duration. Hence * no stealing requests when there are other processes waiting. -* -* 3: We also wish to support `batching' of requests. So when a process is -* woken, we want to allow it to allocate a decent number of requests -* before it blocks again, so they can be nicely merged (this only really -* matters if the process happens to be adding requests near the head of -* the queue). -* -* 4: We want to avoid scheduling storms. This isn't really important, because -* the system will be I/O bound anyway. But it's easy. -* -* There is tension between requirements 2 and 3. Once a task has woken, -* we don't want to allow it to sleep as soon as it takes its second request. -* But we don't want currently-running tasks to steal all the requests -* from the sleepers. We handle this with wakeup hysteresis around -* 0 .. batch_requests and with the assumption that request taking is much, -* much faster than request freeing. +* +* There used to be more here, attempting to allow a process to send in a +* number of requests once it has woken up. But, there's no way to +* tell if a process has just been woken up, or if it is a new process +* coming in to steal requests from the waiters. So, we give up and force +* everyone to wait fairly. * * So here's what we do: *
@@ -561,28 +625,23 @@* * When a process wants a new request: * -* b) If free_requests == 0, the requester sleeps in FIFO manner. -* -* b) If 0 < free_requests < batch_requests and there are waiters, -* we still take a request non-blockingly. This provides batching. -* -* c) If free_requests >= batch_requests, the caller is immediately -* granted a new request. +* b) If free_requests == 0, the requester sleeps in FIFO manner, and +* the queue full condition is set. The full condition is not +* cleared until there are no longer any waiters. Once the full +* condition is set, all new io must wait, hopefully for a very +* short period of time. * * When a request is released: * -* d) If free_requests < batch_requests, do nothing. -* -* f) If free_requests >= batch_requests, wake up a single waiter. +* c) If free_requests < batch_requests, do nothing. * -* The net effect is that when a process is woken at the batch_requests level, -* it will be able to take approximately (batch_requests) requests before -* blocking again (at the tail of the queue). -* -* This all assumes that the rate of taking requests is much, much higher -* than the rate of releasing them. Which is very true. +* d) If free_requests >= batch_requests, wake up a single waiter. * -* -akpm, Feb 2002. +* As each waiter gets a request, he wakes another waiter. We do this +* to prevent a race where an unplug might get run before a request makes +* it's way onto the queue. The result is a cascade of wakeups, so delaying +* the initial wakeup until we've got batch_requests available helps avoid +* wakeups where there aren't any requests available yet. */ static struct request *__get_request_wait (request_queue_t *q,int rw)
@@ -590,21 +649,40 @@ registerstruct request *rq;DECLARE_WAITQUEUE (wait, current); -add_wait_queue (&q->wait_for_requests[rw], &wait); +add_wait_queue_exclusive (&q->wait_for_requests, &wait); +do {set_current_state (TASK_UNINTERRUPTIBLE ); -generic_unplug_device (q); -if (q->rq[rw].count ==0 ) -schedule ();spin_lock_irq (&io_request_lock); - rq =get_request (q, rw); +if (q->full ||blk_oversized_queue (q)) { +__generic_unplug_device (q); +spin_unlock_irq (&io_request_lock); +schedule (); +spin_lock_irq (&io_request_lock); + } + rq =__get_request (q, rw);spin_unlock_irq (&io_request_lock); }while (rq ==NULL ); -remove_wait_queue (&q->wait_for_requests[rw], &wait); +remove_wait_queue (&q->wait_for_requests, &wait); current->state =TASK_RUNNING ; + +if (!waitqueue_active (&q->wait_for_requests)) +clear_full_and_wake (q); +return rq; } +static void get_request_wait_wakeup (request_queue_t *q,int rw) +{ +/* +* avoid losing an unplug if a second __get_request_wait did the +* generic_unplug_device while our __get_request_wait was running +* w/o the queue_lock held and w/ our request out of the queue. +*/ +if (waitqueue_active (&q->wait_for_requests)) +wake_up (&q->wait_for_requests); +} +/* RO fail safe mechanism */ static long ro_bits[MAX_BLKDEV ][8 ];
@@ -818,7 +896,6 @@void blkdev_release_request (struct request *req) {request_queue_t *q = req->q; -int rw = req->cmd; req->rq_status =RQ_INACTIVE ; req->q =NULL ;
@@ -828,9 +905,19 @@* assume it has free buffers and check waiters */ if (q) { -list_add (&req->queue, &q->rq[rw].free); -if (++q->rq[rw].count >= q->batch_requests) -wake_up (&q->wait_for_requests[rw]); +int oversized_batch =0 ; + +if (q->can_throttle) + oversized_batch =blk_oversized_queue_batch (q); + q->rq.count++; +list_add (&req->queue, &q->rq.free); +if (q->rq.count >= q->batch_requests && !oversized_batch) { +smp_mb (); +if (waitqueue_active (&q->wait_for_requests)) +wake_up (&q->wait_for_requests); +else +clear_full_and_wake (q); + } } }
@@ -908,6 +995,7 @@struct list_head *head, *insert_here;int latency;elevator_t *elevator = &q->elevator; +int should_wake =0 ; count = bh->b_size >>9 ; sector = bh->b_rsector;
@@ -948,7 +1036,6 @@*/ max_sectors =get_max_sectors (bh->b_rdev); -again: req =NULL ; head = &q->queue_head;/*
@@ -957,7 +1044,9 @@*/ spin_lock_irq (&io_request_lock); +again: insert_here = head->prev; +if (list_empty (head)) { q->plug_device_fn (q, bh->b_rdev);/* is atomic */ goto get_rq;
@@ -976,6 +1065,7 @@ req->bhtail = bh; req->nr_sectors = req->hard_nr_sectors += count;blk_started_io (count); +blk_started_sectors (req, count);drive_stat_acct (req->rq_dev, req->cmd, count,0 );req_new_io (req,1 , count);attempt_back_merge (q, req, max_sectors, max_segments);
@@ -998,6 +1088,7 @@ req->sector = req->hard_sector = sector; req->nr_sectors = req->hard_nr_sectors += count;blk_started_io (count); +blk_started_sectors (req, count);drive_stat_acct (req->rq_dev, req->cmd, count,0 );req_new_io (req,1 , count);attempt_front_merge (q, head, req, max_sectors, max_segments);
@@ -1030,7 +1121,7 @@* See description above __get_request_wait() */ if (rw_ahead) { -if (q->rq[rw].count < q->batch_requests) { +if (q->rq.count < q->batch_requests ||blk_oversized_queue_batch (q)) {spin_unlock_irq (&io_request_lock); goto end_io; }
@@ -1042,6 +1133,9 @@if (req ==NULL ) {spin_unlock_irq (&io_request_lock); freereq =__get_request_wait (q, rw); + head = &q->queue_head; +spin_lock_irq (&io_request_lock); + should_wake =1 ; goto again; } }
@@ -1064,10 +1158,13 @@ req->start_time = jiffies;req_new_io (req,0 , count);blk_started_io (count); +blk_started_sectors (req, count);add_request (q, req, insert_here); out:if (freereq)blkdev_release_request (freereq); +if (should_wake) +get_request_wait_wakeup (q, rw);spin_unlock_irq (&io_request_lock);return 0 ; end_io:
@@ -1196,8 +1293,15 @@ bh->b_rdev = bh->b_dev; bh->b_rsector = bh->b_blocknr * count; +get_bh (bh);generic_make_request (rw, bh); +/* fix race condition with wait_on_buffer() */ +smp_mb ();/* spin_unlock may have inclusive semantics */ +if (waitqueue_active (&bh->b_wait)) +wake_up (&bh->b_wait); + +put_bh (bh);switch (rw) {case WRITE : kstat.pgpgout += count;
@@ -1350,6 +1454,7 @@if ((bh = req->bh) !=NULL ) { nsect = bh->b_size >>9 ;blk_finished_io (nsect); +blk_finished_sectors (req, nsect); req->bh = bh->b_reqnext; bh->b_reqnext =NULL ; bh->b_end_io (bh, uptodate);
@@ -1509,6 +1614,7 @@EXPORT_SYMBOL (blk_get_queue);EXPORT_SYMBOL (blk_cleanup_queue);EXPORT_SYMBOL (blk_queue_headactive); +EXPORT_SYMBOL (blk_queue_throttle_sectors);EXPORT_SYMBOL (blk_queue_make_request);EXPORT_SYMBOL (generic_make_request);EXPORT_SYMBOL (blkdev_release_request);
drivers/ide/ide-probe.c +1 -0
@@ -971,6 +971,7 @@ q->queuedata =HWGROUP (drive);blk_init_queue (q, do_ide_request); +blk_queue_throttle_sectors (q,1 ); }#undef __IRQ_HELL_SPIN
drivers/scsi/scsi.c +1 -0
@@ -197,6 +197,7 @@blk_init_queue (q, scsi_request_fn);blk_queue_headactive (q,0 ); +blk_queue_throttle_sectors (q,1 ); q->queuedata = (void *) SDpnt; }
drivers/scsi/scsi_lib.c +1 -0
@@ -378,6 +378,7 @@if ((bh = req->bh) !=NULL ) { nsect = bh->b_size >>9 ;blk_finished_io (nsect); +blk_finished_sectors (req, nsect); req->bh = bh->b_reqnext; bh->b_reqnext =NULL ; sectors -= nsect;
fs/buffer.c +21 -1
@@ -153,10 +153,23 @@get_bh (bh);add_wait_queue (&bh->b_wait, &wait);do { -run_task_queue (&tq_disk);set_task_state (tsk,TASK_UNINTERRUPTIBLE );if (!buffer_locked (bh))break ; +/* +* We must read tq_disk in TQ_ACTIVE after the +* add_wait_queue effect is visible to other cpus. +* We could unplug some line above it wouldn't matter +* but we can't do that right after add_wait_queue +* without an smp_mb() in between because spin_unlock +* has inclusive semantics. +* Doing it here is the most efficient place so we +* don't do a suprious unplug if we get a racy +* wakeup that make buffer_locked to return 0, and +* doing it here avoids an explicit smp_mb() we +* rely on the implicit one in set_task_state. +*/ +run_task_queue (&tq_disk);schedule (); }while (buffer_locked (bh)); tsk->state =TASK_RUNNING ;
@@ -1523,6 +1536,9 @@/* Done - end_buffer_io_async will unlock */ SetPageUptodate (page); + +wakeup_page_waiters (page); +return 0 ; out:
@@ -1554,6 +1570,7 @@ }while (bh != head);if (need_unlock)UnlockPage (page); +wakeup_page_waiters (page);return err; }
@@ -1781,6 +1798,8 @@else submit_bh (READ , bh); } + +wakeup_page_waiters (page);return 0 ; }
@@ -2394,6 +2413,7 @@submit_bh (rw, bh); bh = next; }while (bh != head); +wakeup_page_waiters (page);return 0 ; }
fs/reiserfs/inode.c +1 -0
@@ -2209,6 +2209,7 @@*/ if (nr) {submit_bh_for_writepage (arr, nr) ; +wakeup_page_waiters (page); }else {UnlockPage (page) ; }
include/linux/blkdev.h +81 -11
155 lines not on this page · raw message
include/linux/elevator.h +3 -3
@@ -80,7 +80,7 @@return latency; } -#define ELV_LINUS_SEEK_COST 16 +#define ELV_LINUS_SEEK_COST 1#define ELEVATOR_NOOP \ ((elevator_t) { \
@@ -93,8 +93,8 @@#define ELEVATOR_LINUS \ ((elevator_t) { \ - 2048,/* read passovers */ \ -8192 ,/* write passovers */ \ + 128,/* read passovers */ \ +512 ,/* write passovers */ \ \ elevator_linus_merge,/* elevator_merge_fn */ \ elevator_linus_merge_req, /* elevator_merge_req_fn */ \
include/linux/pagemap.h +2 -0
@@ -97,6 +97,8 @@___wait_on_page (page); } +extern void FASTCALL (wakeup_page_waiters (struct page * page)); +/* * Returns locked page at given index in given cache, creating it if needed. */
kernel/ksyms.c +1 -0
@@ -296,6 +296,7 @@EXPORT_SYMBOL (filemap_fdatawait);EXPORT_SYMBOL (lock_page);EXPORT_SYMBOL (unlock_page); +EXPORT_SYMBOL (wakeup_page_waiters);/* device registration */ EXPORT_SYMBOL (register_chrdev);
mm/filemap.c +14 -0
@@ -812,6 +812,20 @@return &wait[hash]; } +/* +* This must be called after every submit_bh with end_io +* callbacks that would result into the blkdev layer waking +* up the page after a queue unplug. +*/ +void wakeup_page_waiters (struct page * page) +{ +wait_queue_head_t * head; + + head =page_waitqueue (page); +if (waitqueue_active (head)) +wake_up (head); +} +/* * Wait for a page to get unlocked. *
Re: [PATCH] io stalls
Quoted 20 lines · Chris Mason
Hello all,
[ short version, the patch attached should fix io latencies in 2.4.21. Please review and/or give it a try ]
My last set of patches was directed at reducing the latencies in
__get_request_wait, which really helped reduce stalls when you had lots of io to one device andbalance_dirty()was causing pauses while you tried to do io to other devices.But, a streaming write could still starve reads to the same device, mostly because the read would have to send down any huge merged writes that were before it in the queue.
Andrea's kernel has a fix for this too, he limits the total number of sectors that can be in the request queue at any given time. But, his patches change
blk_finished_io, both in the arguments it takes and the side effects of calling it. I don't think we can merge his current form without breaking external drivers.
Quoted 7 lines
So, I added a
can_throttleflag to the queue struct, drivers can enable it if they are going to call the newblk_started_sectorsandblk_finished_sectorsfuncs any time they call blk_{started,finished}_io, and these do all the -aa style sector throttling.There were a few other small changes to Andrea's patch, he wasn't setting
q->fullwhenget_requestdecided there were too many sectors in
wasn't is really in the past, because I'm doing it in 2.4.21rc8aa1 and in my latest status.
Quoted 3 lines
flight. This resulted in large latencies in
__get_request_wait. He was also unconditionally clearingq->fullinblkdev_release_request, my code only clearsq->fullwhen all the waiters are gone.
my current code including the older 2.4.21rc8aa1 does that too, merged from your previous patches.
Quoted 5 lines
I changed
generic_unplug_deviceto zero theelevator_sequencefield of the last request on the queue. This means there won't be any merges with requests pending once an unplug is done, and helps limit the number of sectors that need to be sent down during therun_task_queue(&tq_disk)inwait_on_buffer.
this sounds like an hack, forbidding merges is pretty bad for performance in general, of course most of the merging happens in between the unplugs, but during heavy I/O with frequent unplugs from many readers this may hurt performance. And as you said this mostly has the advantage of limiting the size of the queue, like I enforce in my tree with the elevator-lowlatency. I doubt this is right.
Quoted 1 line
I lowered the -aa default limit on sectors in flight from 4MB to 2MB.
I got a few complains for performance slowdown, originally it was 2MB, so I increased it to 4, from 4M it should be enough for most hardware.
Quoted 2 lines
We probably want an elvtune for it, large arrays with writeback cache should be able to tolerate larger values.
Yes, it largely depends on the speed of the device.
Quoted 9 lines
There's still a little work left to do, this patch enables sector throttling for scsi and IDE. cciss, DAC960 and cpqarray need modification too (99% done already in -aa). No sense in doing that until after the bulk of the patch is reviewed though.
As before, most of the code here is from Andrea and Nick, I've just wrapped a lot of duct tape around it and done some tweaking. The primary pieces are:
Quoted 6 lines
fix-pausing (andrea, corner cases where wakeups are missed) elevator-low-latency (andrea, limit sectors in flight)
queue_full(Nick, fairness in__get_request_wait)I've removed my latency stats for
__get_request_waitin hopes of making it a better merging candidate.
this is very similar to my status in -aa, exept for the hack that forbids merging which I think is wrong and the fact you miss the wake_up_nr that I added to give a meaning to the batching again and that you don't avoid the unplugs in get_request_wait_wakeup until the queue is empty. I mean this:
+static void get_request_wait_wakeup(request_queue_t *q, int rw) +{
+ /*
+ * avoid losing an unplug if a second __get_request_wait did the
+ * generic_unplug_device while our __get_request_wait was
running
+ * w/o the queue_lock held and w/ our request out of the queue.
+ */
+ if (q->rq[rw].count == 0 && waitqueue_active(&q->wait_for_requests[rw]))
+ __generic_unplug_device(q);
+}
+
you said last week the above is racy and it even hanged your box, could you elaborate? The above is in 2.4.21rc8aa1 and it works fine so far (though especially the race in wait_for_request is never been known to be reproducible)
thanks,
Andrea
Re: [PATCH] io stalls
Quoted 6 lines · Andrea Arcangeli
Quoted 2 lines
There were a few other small changes to Andrea's patch, he wasn't setting
q->fullwhenget_requestdecided there were too many sectors inwasn't is really in the past, because I'm doing it in 2.4.21rc8aa1 and in my latest status.
Hmm, I thought I grabbed the patch from rc8aa1, clearly not though, sorry about that.
Quoted 6 lines
Quoted 5 lines
I changed
generic_unplug_deviceto zero theelevator_sequencefield of the last request on the queue. This means there won't be any merges with requests pending once an unplug is done, and helps limit the number of sectors that need to be sent down during therun_task_queue(&tq_disk)inwait_on_buffer.
Quoted 7 lines
this sounds like an hack, forbidding merges is pretty bad for performance in general, of course most of the merging happens in between the unplugs, but during heavy I/O with frequent unplugs from many readers this may hurt performance. And as you said this mostly has the advantage of limiting the size of the queue, like I enforce in my tree with the elevator-lowlatency. I doubt this is right.
Well, I would hit sysrq-t when I noticed read stalls, and the reader was frequently in run_task_queue. I kept the hunk because it made a noticeable difference. I agree there's a throughput tradeoff here, my goal for the patch was to find the major places I could improve latency and change them, then go back later and decide if each one was worth it.
Your elevator-lowlatency patch doesn't enforce sector limits for merged requests, so a merger could constantly come in and steal space in the sector limit from other waiters. This lead to high latency in __get_request_wait. That hunk for generic_unplug_device solves both of those problems.
Quoted 5 lines
Quoted 1 line
I lowered the -aa default limit on sectors in flight from 4MB to 2MB.
I got a few complains for performance slowdown, originally it was 2MB, so I increased it to 4, from 4M it should be enough for most hardware.
I've no preference really. I didn't notice a throughput difference but my scsi drives only have 2MB of cache.
Quoted 19 lines
this is very similar to my status in -aa, exept for the hack that forbids merging which I think is wrong and the fact you miss the
wake_up_nrthat I added to give a meaning to the batching again and that you don't avoid the unplugs inget_request_wait_wakeupuntil the queue is empty. I mean this:drivers/block/ll_rw_blk.c +5
-0+static void get_request_wait_wakeup (request_queue_t *q,int rw) +{ +/* +* avoid losing an unplug if a second __get_request_wait did the +* generic_unplug_device while our __get_request_wait was running
drivers/block/ll_rw_blk.c +6
-0+* w/o the queue_lock held and w/ our request out of the queue. +*/ +if (q->rq[rw].count ==0 &&waitqueue_active (&q->wait_for_requests[rw])) +__generic_unplug_device (q); +} +
Quoted 4 lines
you said last week the above is racy and it even hanged your box, could you elaborate? The above is in 2.4.21rc8aa1 and it works fine so far (though especially the race in
wait_for_requestis never been known to be reproducible)
It caused hangs/stalls, but I didn't have the sector throttling code at the time and it really changes the interaction of things. I think the hang went a little like this:
Lets say all the pending io is done, but the wait queue isn't empty yet because all the waiting tasks haven't yet been scheduled in. Also, we have fewer than nr_requests processes waiting to start io, so when they do all get scheduled in they won't generate an unplug.
q->rq.count = q->nr_requests, q->full = 1
new io comes in, sees q->full = 1, unplugs and sleeps. No io is done because the queue is empty.
All the old waiters finally get scheduled in and grab their requests, but get_request_wait_wakeup doesn't unplug because q->rq.count != 0.
If no additional io comes in, the queue never gets unplugged, and our waiter never gets woken.
With the sector throttling on, we've got additional wakeups coming from blk_finished_io (or blk_finished_sectors in my patch). I kept out the wakeup_nr idea because I couldn't figure out how to keep __get_request_wait fair with it in.
-chris
write-caches, I/O stalls: MUST-FIX (was: [PATCH] io stalls)
Quoted 2 lines · Chris Mason
I've no preference really. I didn't notice a throughput difference but my scsi drives only have 2MB of cache.
You shouldn't be using the drive's write cache in the first place!
The write cache, regardless of ATA or SCSI, can, as far as I know, not be used safely with any Linux file systems (and my questions whether 2.6 will finally change that went unanswered so far), because the write reordering the write cache can do can seriously damage file systems, whether journalling or not.
Please conduct all your tests with write caches turned off because that's what matters in REAL systems; in that case, these latencies become a REAL pain in the back because writing is so much slower because of all the seeks.
Optimizing for write cached behaviour can happen not a single second before:
1. the file systems know how to queue "ordered tags" in the right places (write barrier to enforce proper ordering for on-disk consistency, which I assume will make for a lot of ordered tags for writing to the journal itself)
2. the device driver knows how to map "ordered tags" to flush or whatever operations for drives that don't do tagged command queueing (ATA mostly, or SCSI when TCQ is switched off).
All these "0-bytes in file" problems with XFS, ReiserFS, JFS, ext2 and ext3 in data=writeback mode happen because the kernel doesn't care about write ordering, and these broken files are a) occasionally hard to find,
b) another PITA.
I consider proper write ordering and enforcing thereof a MUST-FIX. This is much more important than getting some extra latencies squished. It must do the right thing in the first place, and then it can do the right thing faster.
I am aware that you're not the only person responsible for the state Linux is in, and I'd like to see the write barriers revived ASAP for at least ext2/ext3/reiserfs, sym53c8xx, aic7xxx, tmscsim and IDE. I am sorry not being able to offer any help on that, I'm not acquainted with the kernel stuff and I can't donate money to anyone for me to do it.
SCNR.
Signature · 1 line
Matthias Andree
Re: [PATCH] io stalls
In reply to Chris MasonQuoted 44 lines · Chris Mason
Hello all,
[ short version, the patch attached should fix io latencies in 2.4.21. Please review and/or give it a try ]
My last set of patches was directed at reducing the latencies in
__get_request_wait, which really helped reduce stalls when you had lots of io to one device andbalance_dirty()was causing pauses while you tried to do io to other devices.But, a streaming write could still starve reads to the same device, mostly because the read would have to send down any huge merged writes that were before it in the queue.
Andrea's kernel has a fix for this too, he limits the total number of sectors that can be in the request queue at any given time. But, his patches change
blk_finished_io, both in the arguments it takes and the side effects of calling it. I don't think we can merge his current form without breaking external drivers.So, I added a
can_throttleflag to the queue struct, drivers can enable it if they are going to call the newblk_started_sectorsandblk_finished_sectorsfuncs any time they call blk_{started,finished}_io, and these do all the -aa style sector throttling.There were a few other small changes to Andrea's patch, he wasn't setting
q->fullwhenget_requestdecided there were too many sectors in flight. This resulted in large latencies in__get_request_wait. He was also unconditionally clearingq->fullinblkdev_release_request, my code only clearsq->fullwhen all the waiters are gone.I changed
generic_unplug_deviceto zero theelevator_sequencefield of the last request on the queue. This means there won't be any merges with requests pending once an unplug is done, and helps limit the number of sectors that need to be sent down during therun_task_queue(&tq_disk)inwait_on_buffer.I lowered the -aa default limit on sectors in flight from 4MB to 2MB. We probably want an elvtune for it, large arrays with writeback cache should be able to tolerate larger values.
There's still a little work left to do, this patch enables sector throttling for scsi and IDE. cciss, DAC960 and cpqarray need modification too (99% done already in -aa). No sense in doing that until after the bulk of the patch is reviewed though.
Quoted 8 lines
As before, most of the code here is from Andrea and Nick, I've just wrapped a lot of duct tape around it and done some tweaking. The primary pieces are:
fix-pausing (andrea, corner cases where wakeups are missed) elevator-low-latency (andrea, limit sectors in flight)
queue_full(Nick, fairness in__get_request_wait)
I am hoping to go a slightly different way in 2.5 pending inclusion of process io contexts. If you had time to look over my changes there (in current mm tree) it would be appreciated, but they don't help your problem for 2.4.
I found that my queue full fairness for 2.4 didn't address the batching issue well. It does, guarantee lowest possible maximum latency for singular requests, but due to lowered throughput this can cause worse "high level" latency.
I couldn't find a really good, comprehensive method of allowing processes to batch without resorting to very complex wakeup methods unless process io contexts are used. The other possibility would be to keep a list of "batching" processes which should achieve the same as io contexts.
An easier approach would be to just allow the last woken process to submit a batch of requests. This wouldn't have as good guaranteed fairness, but not to say that it would have starvation issues. I'll help you implement it if you are interested.
Re: [PATCH] io stalls
Quoted 16 lines · Nick Piggin
I am hoping to go a slightly different way in 2.5 pending inclusion of process io contexts. If you had time to look over my changes there (in current mm tree) it would be appreciated, but they don't help your problem for 2.4.
I found that my queue full fairness for 2.4 didn't address the batching issue well. It does, guarantee lowest possible maximum latency for singular requests, but due to lowered throughput this can cause worse "high level" latency.
I couldn't find a really good, comprehensive method of allowing processes to batch without resorting to very complex wakeup methods unless process io contexts are used. The other possibility would be to keep a list of "batching" processes which should achieve the same as io contexts.
Quoted 5 lines
An easier approach would be to just allow the last woken process to submit a batch of requests. This wouldn't have as good guaranteed fairness, but not to say that it would have starvation issues. I'll help you implement it if you are interested.
One of the things I tried in this area was basically queue ownership. When each process woke up, he was given strict ownership of the queue and could submit up to N number of requests. One process waited for ownership in a yield loop for a max limit of a certain number of jiffies, all the others waited on the request queue.
It generally increased the latency in __get_request wait by a multiple of N. I didn't keep it because the current patch is already full of subtle interactions, I didn't want to make things more confusing than they already were ;-)
The real problem with this approach is that we're guessing about the number of requests a given process wants to submit, and we're assuming those requests are going to be highly mergable. If the higher levels pass these hints down to the elevator, we should be able to do a better job of giving both low latency and high throughput.
Between bios and the pdflush daemons, I think 2.5 is in pretty good shape to do what we need. I'm not 100% sure we need batching when the requests being submitted are not highly mergable, but I haven't put lots of thought into that part yet.
Anyway for 2.4 I'm not sure there's much more we can do. I'd like to add tunables to the current patch, so userland can control the max io in flight and a simple toggle between throughput mode and latency mode on a per device basis. It's not perfect but should tide us over until 2.6.
-chris
Re: [PATCH] io stalls
Quoted 26 lines · Chris Mason
Quoted 16 lines · Nick Piggin
I am hoping to go a slightly different way in 2.5 pending inclusion of process io contexts. If you had time to look over my changes there (in current mm tree) it would be appreciated, but they don't help your problem for 2.4.
I found that my queue full fairness for 2.4 didn't address the batching issue well. It does, guarantee lowest possible maximum latency for singular requests, but due to lowered throughput this can cause worse "high level" latency.
I couldn't find a really good, comprehensive method of allowing processes to batch without resorting to very complex wakeup methods unless process io contexts are used. The other possibility would be to keep a list of "batching" processes which should achieve the same as io contexts.
Quoted 6 lines
An easier approach would be to just allow the last woken process to submit a batch of requests. This wouldn't have as good guaranteed fairness, but not to say that it would have starvation issues. I'll help you implement it if you are interested.
Quoted 5 lines
One of the things I tried in this area was basically queue ownership. When each process woke up, he was given strict ownership of the queue and could submit up to N number of requests. One process waited for
ownership in a yield loop for a max limit of a certain number of
jiffies, all the others waited on the request queue.
Not sure exactly what you mean by one process waiting for ownership in a yield loop, but why don't you simply allow the queue "owner" to submit up to a maximum of N requests within a time limit. Once either limit expires (or, rarely, another might become owner -) the process would just be put to sleep by the normal queue_full mechanism.
Quoted 6 lines
It generally increased the latency in
__get_requestwait by a multiple of N. I didn't keep it because the current patch is already full of subtle interactions, I didn't want to make things more confusing than they already were ;-)
Yeah, something like that. I think that in a queue full situation, the processes are wanting to submit more than 1 request though. So the better thoughput you can achieve by batching translates to better effective throughput. Read my recent debate with Andrea about this though - I couldn't convince him!
I have seen much better maximum latencies, 2-3 times the throughput, and an order of magnitude less context switching on many threaded tiobench write loads when using batching.
In short, measuring get_request latency won't give you the full story.
Quoted 7 lines
The real problem with this approach is that we're guessing about the number of requests a given process wants to submit, and we're assuming those requests are going to be highly mergable. If the higher levels pass these hints down to the elevator, we should be able to do a better job of giving both low latency and high throughput.
No, the numbers (batch # requests, batch time) are not highly scientific. Simply when a process wakes up, we'll let them submit a small burst of requests before they go back to sleep. Now in 2.5 (mm) we can cheat and make this more effective, fair, and without possible missed wakes because io contexts means that multiple processes can be batching at the same time, and dynamically allocated requests means it doesn't matter if we go a bit over the queue limit.
I think a decent solution for 2.4 would be to simply have the one queue owner, but he allowed the queue to fall below the batch limit, wake someone else and make them the owner. It can be a bit less fair, and it doesn't work across queues, but they're less important cases.
Quoted 6 lines
Between bios and the pdflush daemons, I think 2.5 is in pretty good shape to do what we need. I'm not 100% sure we need batching when the requests being submitted are not highly mergable, but I haven't put lots of thought into that part yet.
No, there are a couple of problems here.
First, good locality != sequential. I saw tiobench 256 random write throughput _doubled_ because each process is writing within its own file.
Second, mergeable doesn't mean anything if your request size only grows to say 128KB (IDE). I saw tiobench 256 sequential writes on IDE go from ~ 25% peak throughput to ~70% (4.85->14.11 from 20MB/s disk)
Third, context switch rate. In the latest IBM regression tests, tiobench 64 on ext2, 8xSMP (so don't look at throughput!), average cs/s was about 2500 with mainline (FIFO request allocation), and 140 in mm (batching allocation). So nearly 20x better. This might not be due to batching alone, but I didn't see any other obvious change in mm.
Quoted 7 lines
Anyway for 2.4 I'm not sure there's much more we can do. I'd like to add tunables to the current patch, so userland can control the max io in flight and a simple toggle between throughput mode and latency mode on a per device basis. It's not perfect but should tide us over until 2.6.
The changes do seem to be a critical fix due to the starvation issue, but I'm worried that they take a big step back in performance under high disk load. I found my FIFO mechanism to be unacceptably slow for 2.5.
Re: [PATCH] io stalls
Nick Piggin wrote:
snip
Quoted 5 lines
Yeah, something like that. I think that in a queue full situation, the processes are wanting to submit more than 1 request though. So the better thoughput you can achieve by batching translates to better effective throughput. Read my recent debate with Andrea
^^^^^^^^^^
Err, latency
snip
Quoted 4 lines
No, the numbers (batch # requests, batch time) are not highly scientific. Simply when a process wakes up, we'll let them submit a small burst of requests before they go back to sleep.
by this, I mean that its not a big problem that we don't know how many requests a process wants to submit.
snip
Quoted 5 lines
The changes do seem to be a critical fix due to the starvation issue, but I'm worried that they take a big step back in performance under high disk load. I found my FIFO mechanism to be unacceptably slow for 2.5.
BTW. sorry for the lack of better benchmark numbers. I couldn't find good ones lying around. I found uniprocessor tiobench to be quite helpful at queue_nr_requests * 0.5, 2 threads to measure different types of overloadedness.
Also, I didn't see much gain in read performance in my testing - probably due to AS. I expect 2.4 and 2.5 non AS read performance to show bigger improvements from batching (ie. regressions).
Re: [PATCH] io stalls
In reply to Nick PigginQuoted 7 lines · Nick Piggin
Quoted 6 lines
One of the things I tried in this area was basically queue ownership. When each process woke up, he was given strict ownership of the queue and could submit up to N number of requests. One process waited for ownership in a yield loop for a max limit of a certain number of jiffies, all the others waited on the request queue.
Quoted 6 lines
Not sure exactly what you mean by one process waiting for ownership in a yield loop, but why don't you simply allow the queue "owner" to submit up to a maximum of N requests within a time limit. Once either limit expires (or, rarely, another might become owner -) the process would just be put to sleep by the normal
queue_fullmechanism.
You need some way to wakeup the queue after that time limit has expired, in case the owner never submits another request. This can either be a timer or a process in a yield loop. Given that very short expire time I set (10 jiffies), I went for the yield loop.
Quoted 7 lines
Quoted 6 lines
It generally increased the latency in
__get_requestwait by a multiple of N. I didn't keep it because the current patch is already full of subtle interactions, I didn't want to make things more confusing than they already were ;-)
Quoted 6 lines
Yeah, something like that. I think that in a queue full situation, the processes are wanting to submit more than 1 request though. So the better thoughput you can achieve by batching translates to better effective throughput. Read my recent debate with Andrea about this though - I couldn't convince him!
Well, it depends ;-) I think we've got 3 basic kinds of procs during a q->full condition:
1) wants to submit lots of somewhat contiguous io
2) wants to submit a single io
3) wants to submit lots of random io
Quoted 1 line
From a throughput point of view, we only care about giving batch
ownership to #1. giving batch ownership to #3 will help reduce context switches, but if it helps throughput than the io wasn't really random (you've got a good point about locality below, drive write caches make a huge difference there).
The problem I see in 2.4 is the elevator can't tell any of these cases apart, so any attempt at batch ownership is certain to be wrong at least part of the time.
Quoted 7 lines
I have seen much better maximum latencies, 2-3 times the throughput, and an order of magnitude less context switching on many threaded tiobench write loads when using batching.
In short, measuring
get_requestlatency won't give you the full story.
Very true. But get_request latency is the minimum amount of time a single read is going to wait (in 2.4.x anyway), and that is what we need to focus on when we're trying to fix interactive performance.
Quoted 8 lines
Quoted 7 lines
The real problem with this approach is that we're guessing about the number of requests a given process wants to submit, and we're assuming those requests are going to be highly mergable. If the higher levels pass these hints down to the elevator, we should be able to do a better job of giving both low latency and high throughput.
Quoted 8 lines
No, the numbers (batch # requests, batch time) are not highly scientific. Simply when a process wakes up, we'll let them submit a small burst of requests before they go back to sleep. Now in 2.5 (mm) we can cheat and make this more effective, fair, and without possible missed wakes because io contexts means that multiple processes can be batching at the same time, and dynamically allocated requests means it doesn't matter if we go a bit over the queue limit.
I agree 2.5 has a lot more room for the contexts to be effective, and I think they are a really good idea.
Quoted 12 lines
I think a decent solution for 2.4 would be to simply have the one queue owner, but he allowed the queue to fall below the batch limit, wake someone else and make them the owner. It can be a bit less fair, and it doesn't work across queues, but they're less important cases.
Quoted 6 lines
Between bios and the pdflush daemons, I think 2.5 is in pretty good shape to do what we need. I'm not 100% sure we need batching when the requests being submitted are not highly mergable, but I haven't put lots of thought into that part yet.
Quoted 8 lines
No, there are a couple of problems here.
First, good locality != sequential. I saw tiobench 256 random write throughput _doubled_ because each process is writing within its own file.Second, mergeable doesn't mean anything if your request size only grows to say 128KB (IDE). I saw tiobench 256 sequential writes on IDE go from ~ 25% peak throughput to ~70% (4.85->14.11 from 20MB/s disk)
Well, play around with raw io, my box writes at roughly disk speed with 128k synchronous requests (contiguous writes).
Quoted 7 lines
Third, context switch rate. In the latest IBM regression tests, tiobench 64 on ext2, 8xSMP (so don't look at throughput!), average cs/s was about 2500 with mainline (FIFO request allocation), and 140 in mm (batching allocation). So nearly 20x better. This might not be due to batching alone, but I didn't see any other obvious change in mm.
Makes sense.
Quoted 8 lines
Quoted 7 lines
Anyway for 2.4 I'm not sure there's much more we can do. I'd like to add tunables to the current patch, so userland can control the max io in flight and a simple toggle between throughput mode and latency mode on a per device basis. It's not perfect but should tide us over until 2.6.
Quoted 4 lines
The changes do seem to be a critical fix due to the starvation issue, but I'm worried that they take a big step back in performance under high disk load. I found my FIFO mechanism to be unacceptably slow for 2.5.
Me too, but I'm not sure how to fix it other than a userspace knob to turn off the q->full checks for server workloads. Andrea's elevator-lowlatency alone has pretty good throughput numbers, since it still allows request stealing. Its get_request_wait latency numbers aren't horrible either, it only suffers in a few corner cases.
But, if someone wants to play with this more, I've attached a quick remerge of my batch ownership code. I made a read and write owner, so that a reader doing a single request doesn't grab ownership and make all the writes wait.
It does make throughput better overall, and it also makes latencies worse overall. We'll probably get similar results just by disabling q->full in io-stalls-7, but the batch ownership does a better job of limiting get_request latencies at a fixed (although potentially large) number.
lat-stat-5.diff goes on top of io-stalls-7.diff from yesterday batch_owner.diff goes on top of lat-stat-5.diff.
-chris
===== drivers/block/ll_rw_blk.c 1.47 vs edited =====
drivers/block/ll_rw_blk.c +55 -7
@@ -592,6 +592,8 @@ q->full =0 ; q->can_throttle =0 ; +memset (q->batch,0 ,sizeof (struct queue_batch ) *2 ); +reset_stats (q);/*
@@ -606,6 +608,48 @@blk_queue_bounce_limit (q,BLK_BOUNCE_HIGH ); } +#define MSEC (x) ((x) * 1000 / HZ) +#define BATCH_MAX_AGE 100 +int grab_batch_ownership (request_queue_t *q,int rw) +{ +struct task_struct *tsk = current; +unsigned long age; +struct queue_batch *batch = q->batch + rw; + +if (batch->batch_waiter) +return 0 ; +if (!batch->batch_owner) + goto grab; + batch->batch_waiter = tsk; +while (1 ) { + age = jiffies - batch->batch_jiffies; +if (!batch->batch_owner ||MSEC (age) >BATCH_MAX_AGE ) +break ; +set_current_state (TASK_RUNNING ); +spin_unlock_irq (&io_request_lock); +schedule (); +spin_lock_irq (&io_request_lock); + } + batch->batch_waiter =NULL ; +grab: + batch->batch_owner = tsk; + batch->batch_jiffies = jiffies; + batch->batch_remaining = q->batch_requests; +return 1 ; +} + +void decrement_batch_request (request_queue_t *q,int rw) +{ +struct queue_batch *batch = q->batch + rw; +if (batch->batch_owner == current) { + batch->batch_remaining--; +if (!batch->batch_remaining || +MSEC (jiffies - batch->batch_jiffies) >BATCH_MAX_AGE ) { + batch->batch_owner =NULL ; + } + } +} +#define blkdev_free_rq (list) list_entry((list)->next, struct request, queue);/* * Get a free request. io_request_lock must be held and interrupts
@@ -625,6 +669,7 @@ rq->cmd = rw; rq->special =NULL ; rq->q = q; +decrement_batch_request (q, rw); }else q->full =1 ;return rq;
@@ -635,7 +680,7 @@*/ static inline struct request *get_request (request_queue_t *q,int rw) { -if (q->full) +if (q->full && q->batch[rw].batch_owner != current)return NULL ;return __get_request (q, rw); }
@@ -698,25 +743,28 @@static struct request *__get_request_wait (request_queue_t *q,int rw) { - registerstruct request *rq; + registerstruct request *rq =NULL ;unsigned long wait_start = jiffies;unsigned long time_waited;DECLARE_WAITQUEUE (wait, current);add_wait_queue_exclusive (&q->wait_for_requests, &wait); +spin_lock_irq (&io_request_lock);do {set_current_state (TASK_UNINTERRUPTIBLE ); -spin_lock_irq (&io_request_lock);if (q->full ||blk_oversized_queue (q)) { -__generic_unplug_device (q); +if (blk_oversized_queue (q)) +__generic_unplug_device (q);spin_unlock_irq (&io_request_lock);schedule ();spin_lock_irq (&io_request_lock); +if (!grab_batch_ownership (q, rw)) +continue ; } rq =__get_request (q, rw); -spin_unlock_irq (&io_request_lock); }while (rq ==NULL ); +spin_unlock_irq (&io_request_lock);remove_wait_queue (&q->wait_for_requests, &wait); current->state =TASK_RUNNING ;
@@ -978,9 +1026,9 @@list_add (&req->queue, &q->rq.free);if (q->rq.count >= q->batch_requests && !oversized_batch) {smp_mb (); -if (waitqueue_active (&q->wait_for_requests)) +if (waitqueue_active (&q->wait_for_requests)) {wake_up (&q->wait_for_requests); -else + }else clear_full_and_wake (q); } }
===== include/linux/blkdev.h 1.25 vs edited =====
include/linux/blkdev.h +14 -2
@@ -69,6 +69,15 @@struct list_head free; }; +struct queue_batch +{ +struct task_struct *batch_owner; +struct task_struct *batch_waiter; +unsigned long batch_jiffies; +int batch_remaining; + +}; +struct request_queue {/*
@@ -141,7 +150,7 @@* threshold */ int full:1; - +/* * Boolean that indicates you will use blk_started_sectors * and blk_finished_sectors in addition to blk_started_io
@@ -162,6 +171,9 @@* Tasks wait here for free read and write requests */ wait_queue_head_t wait_for_requests; + +struct queue_batch batch[2 ]; +unsigned long max_wait;unsigned long min_wait;unsigned long total_wait;
@@ -278,7 +290,7 @@#define MAX_SEGMENTS 128#define MAX_SECTORS 255 -#define MAX_QUEUE_SECTORS (2 << (20 - 9))/* 4 mbytes when full sized */ +#define MAX_QUEUE_SECTORS (4 << (20 - 9))/* 4 mbytes when full sized */ #define MAX_NR_REQUESTS 1024/* 1024k when in 512 units, normally min is 1M in 1k units */ #define PageAlignSize (size) (((size) + PAGE_SIZE -1) & PAGE_MASK)
reverted:
drivers/block/blkpg.c +1 -0
@@ -261,6 +261,7 @@return blkpg_ioctl (dev, (struct blkpg_ioctl_arg *) arg);case BLKELVGET : +blk_print_stats (dev);return blkelvget_ioctl (&blk_get_queue (dev)->elevator, (blkelv_ioctl_arg_t *) arg);case BLKELVSET :
reverted:
drivers/block/ll_rw_blk.c +66 -0
96 lines not on this page · raw message
reverted:
fs/super.c +1 -0
@@ -726,6 +726,7 @@if (!fs_type->read_super (s, data, flags &MS_VERBOSE ?1 :0 )) goto Einval; s->s_flags |=MS_ACTIVE ; +blk_reset_stats (dev);path_release (&nd);return s;
reverted:
include/linux/blkdev.h +10 -1
@@ -162,8 +162,17 @@* Tasks wait here for free read and write requests */ wait_queue_head_t wait_for_requests; +unsigned long max_wait; +unsigned long min_wait; +unsigned long total_wait; +unsigned long num_req; +unsigned long num_wait; +unsigned long deviation[5 ]; }; +void blk_reset_stats (kdev_t dev); +void blk_print_stats (kdev_t dev); +#define blk_queue_plugged (q) (q)->plugged#define blk_fs_request (rq) ((rq)->cmd == READ || (rq)->cmd == WRITE)#define blk_queue_empty (q) list_empty(&(q)->queue_head)
@@ -269,7 +278,7 @@#define MAX_SEGMENTS 128#define MAX_SECTORS 255 +#define MAX_QUEUE_SECTORS (2 << (20 - 9))/* 4 mbytes when full sized */ -#define MAX_QUEUE_SECTORS (2 << (20 - 9))/* 2 mbytes when full sized */ #define MAX_NR_REQUESTS 1024/* 1024k when in 512 units, normally min is 1M in 1k units */ #define PageAlignSize (size) (((size) + PAGE_SIZE -1) & PAGE_MASK)
Re: [PATCH] io stalls
Quoted 45 lines · Chris Mason
Quoted 6 lines · Nick Piggin
Quoted 6 lines
One of the things I tried in this area was basically queue ownership. When each process woke up, he was given strict ownership of the queue and could submit up to N number of requests. One process waited for
ownership in a yield loop for a max limit of a certain number of
jiffies, all the others waited on the request queue.Quoted 7 lines
Not sure exactly what you mean by one process waiting for ownership in a yield loop, but why don't you simply allow the queue "owner" to submit up to a maximum of N requests within a time limit. Once either limit expires (or, rarely, another might become owner -) the process would just be put to sleep by the normal
queue_fullmechanism.You need some way to wakeup the queue after that time limit has expired, in case the owner never submits another request. This can either be a timer or a process in a yield loop. Given that very short expire time I set (10 jiffies), I went for the yield loop.
Quoted 6 lines
Quoted 6 lines
It generally increased the latency in
__get_requestwait by a multiple of N. I didn't keep it because the current patch is already full of subtle interactions, I didn't want to make things more confusing than they already were ;-)Quoted 7 lines
Yeah, something like that. I think that in a queue full situation, the processes are wanting to submit more than 1 request though. So the better thoughput you can achieve by batching translates to better effective throughput. Read my recent debate with Andrea about this though - I couldn't convince him!
Well, it depends ;-) I think we've got 3 basic kinds of procs during a
q->fullcondition:1) wants to submit lots of somewhat contiguous io
2) wants to submit a single io
3) wants to submit lots of random ioQuoted 1 line
From a throughput point of view, we only care about giving batch
Quoted 9 lines
ownership to #1. giving batch ownership to #3 will help reduce context switches, but if it helps throughput than the io wasn't really random (you've got a good point about locality below, drive write caches make a huge difference there).
The problem I see in 2.4 is the elevator can't tell any of these cases apart, so any attempt at batch ownership is certain to be wrong at least part of the time.
I think though, for fairness, if we allow one to submit a batch of requests, we have to give that opportunity to the others. And yeah, it does reduce context switches, and it does improve throughput for "random" localised IO.
Quoted 6 lines
Quoted 6 lines
I have seen much better maximum latencies, 2-3 times the throughput, and an order of magnitude less context switching on many threaded tiobench write loads when using batching.
Quoted 9 lines
Quoted 4 lines
In short, measuring
get_requestlatency won't give you the full story.Very true. But
get_requestlatency is the minimum amount of time a single read is going to wait (in 2.4.x anyway), and that is what we need to focus on when we're trying to fix interactive performance.
The read situation is different to write. To fill the read queue, you need queue_nr_requests / 2-3 (for readahead) reading processes to fill the queue, more if the reads are random.
If this kernel is being used interactively, its not our fault we might not give quite as good interactive performance. I'm sure the fileserver admin would rather take the tripled bandwidth ;)
That said, I think a lot of interactive programs will want to do more than 1 request at a time anyway.
Quoted 39 lines
Quoted 8 lines
Quoted 8 lines
The real problem with this approach is that we're guessing about the number of requests a given process wants to submit, and we're assuming those requests are going to be highly mergable. If the higher levels pass these hints down to the elevator, we should be able to do a better job of giving both low latency and high throughput.
Quoted 9 lines
No, the numbers (batch # requests, batch time) are not highly scientific. Simply when a process wakes up, we'll let them submit a small burst of requests before they go back to sleep. Now in 2.5 (mm) we can cheat and make this more effective, fair, and without possible missed wakes because io contexts means that multiple processes can be batching at the same time, and dynamically allocated requests means it doesn't matter if we go a bit over the queue limit.
I agree 2.5 has a lot more room for the contexts to be effective, and I think they are a really good idea.
Quoted 17 lines
I think a decent solution for 2.4 would be to simply have the one queue owner, but he allowed the queue to fall below the batch limit, wake someone else and make them the owner. It can be a bit less fair, and it doesn't work across queues, but they're less important cases.
Quoted 6 lines
Between bios and the pdflush daemons, I think 2.5 is in pretty good shape to do what we need. I'm not 100% sure we need batching when the requests being submitted are not highly mergable, but I haven't put lots of thought into that part yet.
No, there are a couple of problems here.
First, good locality != sequential. I saw tiobench 256 random write throughput _doubled_ because each process is writing within its own file.
Quoted 8 lines
Quoted 4 lines
Second, mergeable doesn't mean anything if your request size only grows to say 128KB (IDE). I saw tiobench 256 sequential writes on IDE go from ~ 25% peak throughput to ~70% (4.85->14.11 from 20MB/s disk)
Well, play around with raw io, my box writes at roughly disk speed with 128k synchronous requests (contiguous writes).
Yeah, I'm not talking about request overhead - I think a 128K sized request is just fine. But when there are 256 threads writing, with FIFO method, 128 threads will each have 1 request in the queue. If they are sequential writers, each request will probably be 128K. That isn't enough to get good disk bandwidth. The elevator _has_ to make a suboptimal decision.
With batching, say 8 processes have 16 sequential requests on the queue each. The elevator can make good choices.
Re: [PATCH] io stalls
Quoted 12 lines · Nick Piggin
Quoted 4 lines
Very true. But
get_requestlatency is the minimum amount of time a single read is going to wait (in 2.4.x anyway), and that is what we need to focus on when we're trying to fix interactive performance.The read situation is different to write. To fill the read queue, you need
queue_nr_requests/ 2-3 (for readahead) reading processes to fill the queue, more if the reads are random.
If this kernel is being used interactively, its not our fault we might not give quite as good interactive performance. I'm sure the fileserver admin would rather take the tripled bandwidth ;)
Quoted 3 lines
That said, I think a lot of interactive programs will want to do more than 1 request at a time anyway.
My intuition agrees with yours, but if this is true then andrea's old elevator-lowlatency patch alone is enough, and we don't need q->full at all. Users continued to complain of bad latencies even with his code applied.
Quoted 1 line
From a practical point of view his old code is the same as the batch
wakeup code for get_request latencies and provides good throughput. There are a few cases where batch wakeup has shorter overall latencies, but I don't think people were in those heavy workloads while they were complaining of stalls in -aa.
Quoted 16 lines
Quoted 8 lines
Quoted 4 lines
Second, mergeable doesn't mean anything if your request size only grows to say 128KB (IDE). I saw tiobench 256 sequential writes on IDE go from ~ 25% peak throughput to ~70% (4.85->14.11 from 20MB/s disk)
Well, play around with raw io, my box writes at roughly disk speed with 128k synchronous requests (contiguous writes).
Yeah, I'm not talking about request overhead - I think a 128K sized request is just fine. But when there are 256 threads writing, with FIFO method, 128 threads will each have 1 request in the queue. If they are sequential writers, each request will probably be 128K. That isn't enough to get good disk bandwidth. The elevator _has_ to make a suboptimal decision.
Quoted 2 lines
With batching, say 8 processes have 16 sequential requests on the queue each. The elevator can make good choices.
I agree here too, it just doesn't match the user reports we've been getting in 2.4 ;-) If 2.5 can dynamically allocate requests now and then you can get much better results with io contexts/dynamic wakeups, but I can't see how to make it work in 2.4 without larger backports.
So, the way I see things, we've got a few choices.
1) do nothing. 2.6 isn't that far off.
2) add elevator-lowlatency without q->full. It solves 90% of the problem
3) add q->full as well and make it the default. Great latencies, not so good throughput. Add userland tunables so people can switch.
4) back port some larger chunk of 2.5 and find a better overall solution.
I vote for #3, don't care much if q->full is on or off by default, as long as we make an easy way for people to set it.
-chris
Re: [PATCH] io stalls
Quoted 20 lines · Chris Mason
Quoted 12 lines · Nick Piggin
Quoted 5 lines
Very true. But
get_requestlatency is the minimum amount of time a single read is going to wait (in 2.4.x anyway), and that is what we need to focus on when we're trying to fix interactive performance.The read situation is different to write. To fill the read queue, you need
queue_nr_requests/ 2-3 (for readahead) reading processes to fill the queue, more if the reads are random.
If this kernel is being used interactively, its not our fault we might not give quite as good interactive performance. I'm sure the fileserver admin would rather take the tripled bandwidth ;)Quoted 4 lines
That said, I think a lot of interactive programs will want to do more than 1 request at a time anyway.
Quoted 5 lines
My intuition agrees with yours, but if this is true then andrea's old elevator-lowlatency patch alone is enough, and we don't need
q->fullat all. Users continued to complain of bad latencies even with his code applied.
Didn't that still have the starvation issues in get_request that my patch addressed though? This batching is needed due to the strict FIFO behaviour that my "q->full" thing did.
Quoted 39 lines
Quoted 2 lines
From a practical point of view his old code is the same as the batch
wakeup code for
get_requestlatencies and provides good throughput. There are a few cases where batch wakeup has shorter overall latencies, but I don't think people were in those heavy workloads while they were
complaining of stalls in -aa.Quoted 16 lines
Quoted 9 lines
Quoted 5 lines
Second, mergeable doesn't mean anything if your request size only grows to say 128KB (IDE). I saw tiobench 256 sequential writes on IDE go from ~ 25% peak throughput to ~70% (4.85->14.11 from 20MB/s disk)
Well, play around with raw io, my box writes at roughly disk speed with 128k synchronous requests (contiguous writes).
Yeah, I'm not talking about request overhead - I think a 128K sized request is just fine. But when there are 256 threads writing, with FIFO method, 128 threads will each have 1 request in the queue. If they are sequential writers, each request will probably be 128K. That isn't enough to get good disk bandwidth. The elevator _has_ to make a suboptimal decision.
Quoted 3 lines
With batching, say 8 processes have 16 sequential requests on the queue each. The elevator can make good choices.
I agree here too, it just doesn't match the user reports we've been getting in 2.4 ;-) If 2.5 can dynamically allocate requests now and then you can get much better results with io contexts/dynamic wakeups, but I can't see how to make it work in 2.4 without larger backports.
So, the way I see things, we've got a few choices.
1) do nothing. 2.6 isn't that far off.
2) add elevator-lowlatency without
q->full. It solves 90% of the problem
Quoted 9 lines
3) add
q->fullas well and make it the default. Great latencies, not so good throughput. Add userland tunables so people can switch.4) back port some larger chunk of 2.5 and find a better overall solution.
I vote for #3, don't care much if
q->fullis on or off by default, as long as we make an easy way for people to set it.
5) include the "q->full" starvation fix; add the concept of a queue owner, the batching process.
I'm a bit busy at the moment and so I won't test this, unfortunately. I would prefer that if something like #5 doesn't get in, then nothing be done for .22 unless its backed up by a few decent benchmarks. But its not my call anyway.
Cheers,
Nick
Re: [PATCH] io stalls
Quoted 20 lines · Nick Piggin
Quoted 13 lines · Chris Mason
Quoted 8 lines
Quoted 1 line
The read situation is different to write. To fill the read queue, you need
queue_nr_requests/ 2-3 (for readahead) reading processes to fill the queue, more if the reads are random.
If this kernel is being used interactively, its not our fault we might not give quite as good interactive performance. I'm sure the fileserver admin would rather take the tripled bandwidth ;)Quoted 4 lines
That said, I think a lot of interactive programs will want to do more than 1 request at a time anyway.
Quoted 5 lines
My intuition agrees with yours, but if this is true then andrea's old elevator-lowlatency patch alone is enough, and we don't need
q->fullat all. Users continued to complain of bad latencies even with his code applied.
Quoted 4 lines
Didn't that still have the starvation issues in
get_requestthat my patch addressed though? This batching is needed due to the strict FIFO behaviour that my "q->full" thing did.
Sure, but even though the batch wakeup code didn't have starvation issues, the overall get_request latency was still high. The end result was basically the same, without q->full we've got a higher max wait and a lower average wait. With batch wakeup we've got a higher average (300-400 jiffies) and a lower max (800-900 jiffies).
Especially for things like directory listings, where 2.4 generally does io a few blocks at a time, the get_request latency is a big part of the latency an interactive user sees.
Quoted 13 lines
Quoted 13 lines
So, the way I see things, we've got a few choices.
1) do nothing. 2.6 isn't that far off.
2) add elevator-lowlatency without
q->full. It solves 90% of the problem3) add
q->fullas well and make it the default. Great latencies, not so good throughput. Add userland tunables so people can switch.4) back port some larger chunk of 2.5 and find a better overall solution.
Quoted 7 lines
Quoted 3 lines
I vote for #3, don't care much if
q->fullis on or off by default, as long as we make an easy way for people to set it.5) include the "
q->full" starvation fix; add the concept of a queue owner, the batching process.
I've tried two different approaches to #5, the first is a just a batch_owner where other procs are still allowed to grab requests and the owner was allowed to ignore q->full. The end result was low latencies but not much better throughput. With a small number of procs, you've got a good chance bdflush is going to get ownership and the throughput is pretty good. With more procs the probability of that goes down and the throughput benefit goes away.
My second attempt was the batch wakeup patch from yesterday. Overall I don't feel the latencies are significantly better with that patch than with Andrea's elevator-lowlatency and q->full disabled.
Quoted 5 lines
I'm a bit busy at the moment and so I won't test this, unfortunately. I would prefer that if something like #5 doesn't get in, then nothing be done for .22 unless its backed up by a few decent benchmarks. But its not my call anyway.
Andrea's code without q->full is a good starting point regardless. The throughput is good and the latencies are better overall. q->full is simple enough that making it available via a tunable is pretty easy. I really do wish I could make one patch that works well for both, but I've honestly run out of ideas ;-)
-chris
Re: [PATCH] io stalls
In reply to Nick PigginQuoted 7 lines · Nick Piggin
Quoted 7 lines
I think the only time we really need to wakeup more than one waiter is when we hit the
q->batch_requestmark. After that, each new request that is freed can be matched with a single waiter, and we know that any previously finished requests have probably already been matched to their own waiter.
Quoted 8 lines
Nope. Not even then. Each retiring request should submit a wake up, and the process will submit another request. So the number of requests will be held at the
batch_requestmark until no more waiters.Now that begs the question, why have
batch_requestsanymore? It no longer does anything.
We've got many flavors of the patch discussed in this thread, so this needs a little qualification. When get_request_wait_wakeup wakes one of the waiters (as in the patch I sent yesterday), you want to make sure that after you wake the first waiter there is a request available for the proccess he is going to wake up, and so on for each other waiter.
I did a quick test of this yesterday, and under the 20 proc iozone test, turning off batch_requests more than doubled the number of context switches hit during the run, I'm assuming this was from wakeups that failed to find requests.
I'm doing a few tests with Andrea's new get_request_wait_wakeup ideas and wake_up_nr.
-chris
Re: -rc7 Re: Linux 2.4.21-rc6
In reply to Marc-Christian PetersenQuoted 15 lines · Marc-Christian Petersen
On Wednesday 04 June 2003 12:22, Andrea Arcangeli wrote:
Hi Andrea,
Quoted 11 lines
are you really sure that it is the right fix?
I mean, the batching has a basic problem (I was discussing it with Jens two days ago and he said he's already addressed in 2.5, I wonder if that could also have an influence on the fact 2.5 is so much better in fariness)
the issue with batching in 2.4, is that it is blocking at 0 and waking atbatch_requests. But it's not blocking newget_requestto eat requests in the way back from 0 tobatch_requests. I mean, there are two directions, when we move frombatch_requeststo 0get_requestsshould return requests. in the way back from 0 tobatch_requeststheget_requestshould block (and it doesn't in 2.4, that is the problem)
Quoted 1 line
do you see a chance to fix this up in 2.4?
sure, it's just a matter of adding a bit to the blkdev structure. However I'm not 100% sure that it is the real thing that could make the difference, but overall the exclusive wakeup FIFO in theory should provide even an higher degree of fariness, so at the very least the "fix" 2 from Andrew makes very little sense to me, and it seems just an hack meant to hide a real problem in the algorithm.
I mean, going wakeall (LIFO btw) rather than wake-one FIFO if something should make things worse unless it is hiding some other issue.
As for 1 and 3 they were just included in my tree for ages.
BTW, Chris recently spotted a nearly impossible to trigger SMP-only race in the fix pausing patch [great spotting Chris] (to trigger it would need an intersection of two races at the same time), it'll be fixed in my next tree, however nobody ever reproduced it and you certainly can ignore it in practice so it can't explain any issue.
Andrea
Re: -rc7 Re: Linux 2.4.21-rc6
On Wednesday 04 June 2003 12:43, Andrea Arcangeli wrote:
Hi Andrea,
Quoted 6 lines
sure, it's just a matter of adding a bit to the blkdev structure. However I'm not 100% sure that it is the real thing that could make the difference, but overall the exclusive wakeup FIFO in theory should provide even an higher degree of fariness, so at the very least the "fix" 2 from Andrew makes very little sense to me, and it seems just an hack meant to hide a real problem in the algorithm.
well, at least it reduces pauses/stops ;)
Quoted 1 line
As for 1 and 3 they were just included in my tree for ages.
err, 1 yes, but I don't see that 3 is in your tree. Well, ok, a bit different. But hey, your 1+3 are still having pauses ;)
Quoted 2 lines
BTW, Chris recently spotted a nearly impossible to trigger SMP-only race in the fix pausing patch [great spotting Chris] (to trigger it would
Cool Chris!
Quoted 3 lines
need an intersection of two races at the same time), it'll be fixed in my next tree, however nobody ever reproduced it and you certainly can ignore it in practice so it can't explain any issue.
Good to know. Thanks.
ciao, Marc
Config issue (CONFIG_X86_TSC) Re: Linux 2.4.21-rc6
In reply to Marcelo TosattiQuoted 8 lines · Marcelo Tosatti <[email protected]>, on Wed May 28, 2003 [09:55:39 PM]…
Hi,
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
Please test it.
Hi; It seems if I run 'make menuconfig', and the only change
I make is to change the processor type from its default to 486, "CONFIG_X86_TSC=y", remains in the .config, which results in a kernel that wont boot on a 486.
Running 'make oldconfig' seems to fix it up, though...
Re: Config issue (CONFIG_X86_TSC) Re: Linux 2.4.21-rc6
On Tue, 2003-06-03 15:45:37 -0400, Paul <[email protected]> wrote in message <[email protected]>:
Quoted 6 lines
Quoted 5 lines · Marcelo Tosatti <[email protected]>, on Wed May 28, 2003 [09:55:39 PM]…
Here goes -rc6. I've decided to delay 2.4.21 a bit and try Andrew's fix for the IO stalls/deadlocks.
Please test it.
Quoted 7 lines
Hi; It seems if I run 'make menuconfig', and the only changeI make is to change the processor type from its default to 486, "
CONFIG_X86_TSC=y", remains in the .config, which results in a kernel that wont boot on a 486.
Running 'make oldconfig' seems to fix it up, though...
Yeah, that's a but hitting i80386 also, I had sent a patch for that some time ago to LKML. There's simply some CONFIG_X86_TSC=n missing in the case of i486 and i486.
MfG, JBG
Signature · 4 lines
Jan-Benedict Glaw [email protected] . +49-172-7608481 "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg fuer einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! ret = do_actions((curr | FREE_SPEECH) & ~(IRAQ_WAR_2 | DRM | TCPA));