Skip to contentSkip to conversation

Streaming disk I/O kills file buffering and makes Linux unusable

DISCUSSION Benno Senoner 8 participants 15 messages Aug 23, 1999 – Aug 24, 1999 Sound

lore
Share & tools
lore
#0 lore raw

Hi,
I'm doing experiments of high performance data streaming from disk, in my case streaming 60 audio tracks (44khz mono 16bit) from disk.

This works quite well, but it has one BIG DRAWBACK:
it makes the rest of the system unusable.
If I run this app which reads huge files from disks continuously, all other regular files and executables buffered in the file buffer cache, are evicted from the filebuffer, and every time I want to launch an executable, which was previously launched, the kernel has to load the entire executable from disk, since it's not longer buffered and therefore it degrades the performance of my streaming app too.
I could use a second disk dedicated for streaming, but the first disk (on which the Linux distro and apps do reside), will become very slow, since it will pratically RUN IN AN UNBUFFERED fashion all the time, since the LRU buffering algorithm will remove all buffered regular files and executable because the kernel caches the huge streaming files.

Is there a way to limit filebuffering on a per-process basis ?
For example let's say, I want that my streaming process can't use more than 5MB of the kernel filebuffers, so that regular files/executables can be buffered witout being removed from the buffers due to the huge filestreaming activity. SGI's had the O_DIRECT flag to pass to the open() call to avoid buffering. But linux is missing this flag , and IMHO O_SYNC would not help, since it syncs only on disk writes, disk reads will still be cached.

I found nothing usable in the /proc/sys/* tuneable values, and setlimit() has no tunable buffers in it's structure.
The setbuffer() of stdio seems not to solve the problem since the kernel uses his own buffering.
Is there a solution for this buffer cache problem ?

Will future kernels ave an O_DIRECT like flag to avoid caching ? I think without some buffer-cache usage limiting, or buffering disabling, Linux is actually UNSUITABLE for streaming applications which do run concurrently with other apps.

comments ?

PS: the easiest way to test this bad behavior is to run in background a script which copies 5-10 files of 50-100MB in size simultaneously, let it run a while and then try to open and xterm .. it will take SEVERAL seconds.

regards,
Benno.

#1 lore raw
Quoted 1 line

Will future kernels ave an O_DIRECT like flag to avoid caching ?

Stephen Tweedie has written a patch for unbuffered I/O; the URL I have from some time ago is

ftp://ftp.linux.org.uk/pub/linux/sct/fs/raw-io

though this host doesn't seem to respond at the moment. I believe Stephen has mentioned that this will go into the main tree at some point. (Possibly it's already in 2.3.x---I haven't looked recently.)

Quoted 3 lines

I think without some buffer-cache usage limiting, or buffering disabling, Linux is actually UNSUITABLE for streaming applications which do run concurrently with other apps.

I agree; the raw-io system will make this much more pleasant.

Cheers,
Peter Monta [email protected]
Imedia Corp.

#2 lore raw
In reply to Benno Senoner
Quoted 6 lines · Peter Monta
Quoted 4 lines

I think without some buffer-cache usage limiting, or buffering disabling, Linux is actually UNSUITABLE for streaming applications which do run concurrently with other apps.

I agree; the raw-io system will make this much more pleasant.

Why do you think so?

IO is IO, whether it is raw or not. And it will eat up resources from everybody else trying to do IO.

I think this is a case of believing in magic cures and supernatural beings.

Linus

#3 lore raw
In reply to Benno Senoner
Quoted 6 lines
Quoted 1 line

I agree; the raw-io system will make this much more pleasant.

Why do you think so?

IO is IO, whether it is raw or not. And it will eat up resources from everybody else trying to do IO.

True; raw I/O won't magically increase resources. But it offers a valuable way to tell the kernel something it can't know/infer: a good buffering policy for streaming data. The existing cache mechanism for normal I/O works fine but is inappropriate for this case; indeed perfectly good working sets are messed up by buffering the streaming data, as Benno points out.

Cheers,
Peter

#4 lore raw
In reply to Benno Senoner
Quoted 10 lines · Linus Torvalds
Quoted 6 lines · Peter Monta
Quoted 4 lines

I think without some buffer-cache usage limiting, or buffering disabling, Linux is actually UNSUITABLE for streaming applications which do run concurrently with other apps.

I agree; the raw-io system will make this much more pleasant.

Why do you think so?

Quoted 7 lines

IO is IO, whether it is raw or not. And it will eat up resources from everybody else trying to do IO.

I think this is a case of believing in magic cures and supernatural beings.

Linus

IO is IO, BUT: buffering algorithms do work well almost all the time
under _NORMAL_ conditions , but
streaming hundreds of MBytes from disk (reading) , allocates all buffers for these files (which you read only once, therefore buffering is not required), and all other apps and files ( your browser , windowmanager, X server xterm , bash etc) are evicted from the buffer cache, and every time (with maybe 10-30 secs in between) you launch a new xterm, click to the KDE's application bar, the disk has to reload the page for the executables, which makes you run the disk as it were not buffered.

Is it hard to implement unbuffered disk I/O in Linux (at least for reading) on per file basis so that reading large files doesn't monopolizes the buffer cache decreasing the overall caching effectiveness ?

Another solution could be limiting file buffering on a per-process basis, If the streaming app can tell to the kernel "give me only 4MB of your buffer cache", then the rest of the apps will still get a chunk of the buffer cache so that launching my xterm multiple times will work nicely without reading from disk every time.
Is the second solution maybe easier to implement ?

I'm afraid, if we want that Linux will be a good multimedia OS, this is a _STRONGLY_NEEDED_ feature.

The user wants to playback his video/audio from disk, and still be able to launch his apps, without waiting 10 secs for loading a simple xterm.

The SCT's raw-io patches are nice, but not very suitable in a general purpose multimedia enviroment, since you can't tell to the user to store his videos on a raw partition.

regards,
Benno.

#5 lore raw
In reply to Benno Senoner
Quoted 9 lines

I'm afraid, if we want that Linux will be a good multimedia OS, this is a _STRONGLY_NEEDED_ feature.

The user wants to playback his video/audio from disk, and still be able to launch his apps, without waiting 10 secs for loading a simple xterm.

The SCT's raw-io patches are nice, but not very suitable in a general purpose multimedia enviroment, since you can't tell to the user to store his videos on a raw partition.

Stephens patches are basis of raw I/O on files in a filesystem too.

However what you are saying and raw-io don't neccessarily tally. You are actually saying "there is a bug in the current page cache handling for this kind of operation". Far better therefore to fix the heuristic used.

#6 lore raw
In reply to Benno Senoner
Quoted 11 lines · Peter Monta
Quoted 1 line

Will future kernels ave an O_DIRECT like flag to avoid caching ?

Stephen Tweedie has written a patch for unbuffered I/O; the URL I have from some time ago is

ftp://ftp.linux.org.uk/pub/linux/sct/fs/raw-io

though this host doesn't seem to respond at the moment. I believe Stephen has mentioned that this will go into the main tree at some point. (Possibly it's already in 2.3.x---I haven't looked recently.)

Quoted 5 lines
Quoted 3 lines

I think without some buffer-cache usage limiting, or buffering disabling, Linux is actually UNSUITABLE for streaming applications which do run concurrently with other apps.

I agree; the raw-io system will make this much more pleasant.

seems to me what you really want is an application-specific file system that is designed to manage streamed data on disks. i've been very impressed with the Sony MiniDisc file system -- it's a simplified file system that transparently manages data on MO MiniDiscs.

the name space is flat -- you can have 1 to 255 separate "tracks", and i've seen this system used for monaural or up to 8 concurrent channels of recording, on 128M MO disks. i'm no expert, but i'd bet a faster, larger disk could be used to boost the number of concurrent channels. it's probably just a matter of how many channels can be multiplexed into a track's block stream.

a simple TOC-based block-allocation system is used. the TOC is stored in the disk writer's volatile memory, and written out when the disk is ejected; this eliminates the extra seeks involved with metadata updates. new blocks are allocated sequentially from free space at the end of the disk, or eventually from blocks freed by erasing tracks of previously recorded material. a text area of about 2K per disk is reserved for tagging each track with "title" data.

for managing a large disk, one might extend such a file system to handle more tracks, more blocks, and generate a periodic TOC update that writes the TOC into special areas allocated across the entire disk to minimize seek latency.

this kind of system would eliminate the need for indirect blocks or even extent-based allocation, and keep metadata updates very cheap. it would also make it easy to combine, divide, and otherwise edit the data in the tracks.

another example of an application-specific file system was early Dyaxis gear. the Dyaxis editing suite ran on a Mac II, but used separate disks and some proprietary form of HFS, i think, to record and edit audio data.

but it might be cool to have a special simplified file system that was designed specifically to manage time-critical streamed data like multitrack audio or video.

- Chuck Lever
--
corporate: <[email protected]>
personal: <[email protected]> or <[email protected]>

The Linux Scalability project:
http://www.citi.umich.edu/projects/linux-scalability/

#7 lore raw
In reply to Benno Senoner
Quoted 3 lines · Benno Senoner

No , I'm not saying that there is a bug, I'm saying that the filebuffering works very well in almost all cases , except of reading large files from disk continuously.

Could you check vmstat 1 and verify if you have continous swapin/swapout? (I bet yes) If I am right then you'll probably want to apply this my patch and try again and notice the performance difference with eyes.

	ftp://ftp.suse.com/pub/people/andrea/kernel-patches/my-2.2.12-final3/no-swapout-2.2.10-B

Andrea

#8 lore raw
In reply to Benno Senoner
Quoted 5 lines · Mark Hahn
Quoted 1 line

Is it hard to implement unbuffered disk I/O in Linux (at least for reading) on

this is basically an oxymoron: apps read only from buffers. the issue is getting the kernel to realize that you're never going to read old buffers again. if you want to help, read the code, don't just complain.

Unfortunately I don't (yet) started kernel hacking, therefore the only thing I can to now is testing/benchmarking using userspace apps.

Quoted 7 lines
Quoted 4 lines

Another solution could be limiting file buffering on a per-process basis, If the streaming app can tell to the kernel "give me only 4MB of your buffer cache", then the rest of the apps will still get a chunk of the buffer cache so

yuck! this would be a grotesque tuning nightmare, and would hold the promise of utterly trashing performance in unexpected ways.

This was only an idea I had, it may be dirty but it would work, without trashing.

Quoted 6 lines
Quoted 3 lines

I'm afraid, if we want that Linux will be a good multimedia OS, this is a _STRONGLY_NEEDED_ feature.

debatable. first, you mean "media _producer_ OS", since media clients do not generally mess with this kind of bandwidth.

Ok, playing streaming an AVI video is low-bandwidth , and will not trash performance of the linux file buffering system.
But what about streaming a DVD movie from the harddisk ? A DVD movie is made up of GBs of data, and playing back a DVD is not a media_producer_OS task, it will be very common on every desktop.

Quoted 5 lines

next, you're also
assuming that media production apps should be naive about how they do IO. for instance, have you tried using mmap, with explicit munmap's on data that you're done with? or played with some of the parameters on the current code that attempts to prevent cache flushing?

Ah ! thank you for this suggestion, I will try this mmap() method, but it is not a very portable way to do disk I/O and makes things a bit more complicated.
I'm predicting that I will not get much benefits of mmap() due to Murphy's law. :-)

Which parameters do you mean ? /proc/sys/* ?
I did not found anything useful here.

Quoted 5 lines

in short, cache flushing is well-understood and media production is just one minor case where it comes up. hinting (via O_ flags, madvise, ioctl) are the right way to turn up the agressiveness of the existing free-behind and pre-fetching mechanisms.

I agree, for special situations like this you have to tell the kernel your intentions, since it's not easy to find out for the kernel through heuristics. When SCT will release his per-file raw i/o patches then I will rerun my tests. In the meantime I will play around with mmap().

regards,
Benno.

#9 lore raw
In reply to Benno Senoner
Quoted 6 lines · Andrea Arcangeli
Quoted 3 lines · Benno Senoner

No , I'm not saying that there is a bug, I'm saying that the filebuffering works very well in almost all cases , except of reading large files from disk continuously.

Quoted 8 lines

Could you check vmstat 1 and verify if you have continous swapin/swapout? (I bet yes) If I am right then you'll probably want to apply this my patch and try again and notice the performance difference with eyes.

	ftp://ftp.suse.com/pub/people/andrea/kernel-patches/my-2.2.12-final3/no-swapout-2.2.10-B

Andrea

No , have very little swapin/swapouts (only occasionally) when clicking on the xterm icon.
Since I run on a 256MB RAM box, I have to wait 30-40sec for the xterm executable disappearing from buffers ( since I stream about 5MB/sec). After this time when I must wait up to 5-6sec from clicking to the icon to the actual launch of the xterm.
Of course this is because the disk is under high load , and loads the executable slowly, but if I I click on the icon again after only a few secs (instead of waiting 30-40secs) the 2nd xterm starts up almost immediately since it's still cached, 200MB filebuffer buffer/5MB/sec = takes about 40secs to fill all buffers.

Browsing the web while runnig the streaming app, becomes frustrating because you have no caching at all while working on a very slow disk (due to disk I/O). IMHO with direct I/O the rest of the system will remain quite snappy.

regards,
Benno.

#10 lore raw
In reply to Benno Senoner
Quoted 11 lines · Alan Cox
Quoted 9 lines

I'm afraid, if we want that Linux will be a good multimedia OS, this is a _STRONGLY_NEEDED_ feature.

The user wants to playback his video/audio from disk, and still be able to launch his apps, without waiting 10 secs for loading a simple xterm.

The SCT's raw-io patches are nice, but not very suitable in a general purpose multimedia enviroment, since you can't tell to the user to store his videos on a raw partition.

Stephens patches are basis of raw I/O on files in a filesystem too.

That sounds nice to me !
Any idea when some patches ( for raw file I/O in a filesystem) will be available ?

Quoted 4 lines

However what you are saying and raw-io don't neccessarily tally. You are actually saying "there is a bug in the current page cache handling for this kind of operation". Far better therefore to fix the heuristic used.

No , I'm not saying that there is a bug, I'm saying that the filebuffering works very well in almost all cases , except of reading large files from disk continuously.
This is of course quite logical since the kernel makes assumptions that you will read this large file again, and therefore the kernel has to put it into the buffer cache.

In this case I don't need raw I/O to get fast response or higher throughput, but only to avoid that the streaming apps do not monopolizes the buffer, which is very bad.

will the SCT's patches add an O_DIRECT -like flag ala SGI ?

regards,
Benno.

#11 lore raw
In reply to Benno Senoner
Quoted 2 lines · Benno Senoner

No , have very little swapin/swapouts (only occasionally) when clicking on the xterm icon.

As expected.

Incidentally that's exactly the _thing_ you report to be slow just in the line below. Maybe it's only an unlucky coincidence? ;)

Quoted 4 lines

After this time when I must wait up to 5-6sec from clicking to the icon to the actual launch of the xterm.

Of course this is because the disk is under high load , and loads the

The swapin slowdown things (still more if the disk is under load).

Andrea

#12 lore raw
In reply to Benno Senoner
Quoted 3 lines · Benno Senoner

In this case I don't need raw I/O to get fast response or higher throughput, but only to avoid that the streaming apps do not monopolizes the buffer, which is very bad.

You probably don't want strict raw I/O, as there would be considerable benefit in some sort of read-ahead buffering. What you want is an open/fcntl attribute which would instruct the kernel that you don't intend to do a seek, in which case it can free each buffer as soon as it has been fully read.

Signature · 3 lines
Dave Mielke           | 856 Grenon Avenue | I believe that the Bible is the
Phone: 1-613-726-0014 | Ottawa, Ontario   | Word of God. Please contact me
EMail: [email protected] | Canada  K2B 6G3   | if you're concerned about Hell.
#13 lore raw
In reply to Benno Senoner

If you are in a hurry but are a courageous guy ;-), you can, for now, add some file control (fcntl) option that tells the kernel to discard the pages of data of this file once read by user (or just to set the pages at the end of the LRU in recent kernels), then make appropriate changes in the kernel and in the application. The kernel will still prefetch file data on read, allowing read streaming, without having to multithread IOs at application level.

Sorry if this has been already suggested.

Gérard.

Quoted 41 lines · Benno Senoner
Quoted 11 lines · Alan Cox
Quoted 9 lines

I'm afraid, if we want that Linux will be a good multimedia OS, this is a _STRONGLY_NEEDED_ feature.

The user wants to playback his video/audio from disk, and still be able to launch his apps, without waiting 10 secs for loading a simple xterm.

The SCT's raw-io patches are nice, but not very suitable in a general purpose multimedia enviroment, since you can't tell to the user to store his videos on a raw partition.

Stephens patches are basis of raw I/O on files in a filesystem too.

That sounds nice to me !
Any idea when some patches ( for raw file I/O in a filesystem) will be available ?

Quoted 4 lines

However what you are saying and raw-io don't neccessarily tally. You are actually saying "there is a bug in the current page cache handling for this kind of operation". Far better therefore to fix the heuristic used.

No , I'm not saying that there is a bug, I'm saying that the filebuffering works very well in almost all cases , except of reading large files from disk continuously.
This is of course quite logical since the kernel makes assumptions that you will read this large file again, and therefore the kernel has to put it into the buffer cache.

In this case I don't need raw I/O to get fast response or higher throughput, but only to avoid that the streaming apps do not monopolizes the buffer, which is very bad.

will the SCT's patches add an O_DIRECT -like flag ala SGI ?

regards,
Benno.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#14 lore raw
In reply to Benno Senoner
Quoted 5 lines · Gerard Roudier

If you are in a hurry but are a courageous guy ;-), you can, for now, add some file control (fcntl) option that tells the kernel to discard the pages of data of this file once read by user

(or just to set the pages at the end of the LRU in recent kernels),

This suggestion anticipates too much about page map LRU inclusion in the kernel main stream and just makes the suggestion just broken). Without Andrea's page LRU, it may happen that useful page of data will still be thrown away after a shrink_mmap revolution.
By the way, if 2.4 does not include Andrea's page LRU, I know of people who will be very disappointed.

Quoted 54 lines

then make appropriate changes in
the kernel and in the application. The kernel will still prefetch file data on read, allowing read streaming, without having to multithread IOs at application level.

Sorry if this has been already suggested.

Gérard.

Quoted 41 lines · Benno Senoner
Quoted 11 lines · Alan Cox
Quoted 9 lines

I'm afraid, if we want that Linux will be a good multimedia OS, this is a _STRONGLY_NEEDED_ feature.

The user wants to playback his video/audio from disk, and still be able to launch his apps, without waiting 10 secs for loading a simple xterm.

The SCT's raw-io patches are nice, but not very suitable in a general purpose multimedia enviroment, since you can't tell to the user to store his videos on a raw partition.

Stephens patches are basis of raw I/O on files in a filesystem too.

That sounds nice to me !
Any idea when some patches ( for raw file I/O in a filesystem) will be available ?

Quoted 4 lines

However what you are saying and raw-io don't neccessarily tally. You are actually saying "there is a bug in the current page cache handling for this kind of operation". Far better therefore to fix the heuristic used.

No , I'm not saying that there is a bug, I'm saying that the filebuffering works very well in almost all cases , except of reading large files from disk continuously.
This is of course quite logical since the kernel makes assumptions that you will read this large file again, and therefore the kernel has to put it into the buffer cache.

In this case I don't need raw I/O to get fast response or higher throughput, but only to avoid that the streaming apps do not monopolizes the buffer, which is very bad.

will the SCT's patches add an O_DIRECT -like flag ala SGI ?

regards,
Benno.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/