Skip to contentSkip to conversation

mmap() better than read() fro streaming, Was: Re: Streaming disk I/O kills file buffering and ma

DISCUSSION Alan Cox 4 participants 6 messages Aug 25, 1999 – Aug 26, 1999 Sound

lore
Share & tools
lore

Alan Cox

author
#0 lore raw
Quoted 6 lines

As you suggested I do basically the following:
- ptr=mmap() at current offset with lenQ2k
- memcpy(targetbuffer,ptr,len) (I must use the memcpy since targetbuffer has
to be mlocked() since the audio-playing thread can't tolerate pagefaults
because it runs in a low-latency cycle.
- munmap(ptr,len)

You can do

	ptr=mmap(blah)
	mlock(ptr, ...)
	munmap(ptr, len)
Quoted 2 lines

Does anyone know if there it/will be a way to do unbuffered mmap() ? I think streaming apps would benefit quite a bit from this.

mmap requires buffering - you are sharing the page with the system page cache. If you mean you want an mgoaway() to go with munmap() thats what madvise() provides on some other systems but not Linux yet

#1 lore raw

mmap() better than read() fro streaming, Was: Re: Streaming disk I/O kills file buffering and makes

Quoted 10 lines
Quoted 3 lines

I'm afraid, if we want that Linux will be a good multimedia OS, this is a _STRONGLY_NEEDED_ feature.

debatable. first, you mean "media _producer_ OS", since media clients do not generally mess with this kind of bandwidth. next, you're also assuming that media production apps should be naive about how they do IO. for instance, have you tried using mmap, with explicit munmap's on data that you're done with? or played with some of the parameters on the current code that attempts to prevent cache flushing?

I adapted my file streaming app to use mmap() instead of using read() to read chunks of up to 512k into a buffer.
As you suggested I do basically the following:
- ptr=mmap() at current offset with lenQ2k
- memcpy(targetbuffer,ptr,len) (I must use the memcpy since targetbuffer has
to be mlocked() since the audio-playing thread can't tolerate pagefaults
because it runs in a low-latency cycle.
- munmap(ptr,len)

I must say it works quite well, and firing up apps, or accessing the KDE menus, is faster,
launching my famous "xterm" still produces disk accesses,
(cached files are still trown away after a while, but not so agressively like in the read() case )
but it's now faster and the system seems more usable,
especially painting into a GIMP window, since gimp uses the hd quite a lot. I'm now able to stream about 50 mono audio tracks (44kHz 16bit) from my IBM 16GB EIDE UDMA disk, while browsing the web, very fun !
Of course if I copy a large file in background, sooner of later, I get a ring-buffer overrun , because the streaming-app and the cp must share disk bandwidth. ( I'm missing SGI's guaranteed rate I/O of XFS , sigh .. :-) )

again , thank you for your suggestion !
It was a bit tricky to mess with all this page aligns, and special cases, but it pretty fun to code ! ( got nice segfaults at the beginning :-) )

Quoted 5 lines

in short, cache flushing is well-understood and media production is just one minor case where it comes up. hinting (via O_ flags, madvise, ioctl) are the right way to turn up the agressiveness of the existing free-behind and pre-fetching mechanisms.

Does anyone know if there it/will be a way to do unbuffered mmap() ? I think streaming apps would benefit quite a bit from this.

Quoted 2 lines

regards, mark hahn.

regards,
Benno.

#2 lore raw
In reply to Alan Cox
Quoted 11 lines · Alan Cox
Quoted 6 lines

As you suggested I do basically the following:
- ptr=mmap() at current offset with lenQ2k
- memcpy(targetbuffer,ptr,len) (I must use the memcpy since targetbuffer has
to be mlocked() since the audio-playing thread can't tolerate pagefaults
because it runs in a low-latency cycle.
- munmap(ptr,len)

You can do

	ptr=mmap(blah)
	mlock(ptr, ...)
	munmap(ptr, len)

will mlock() automatically cause all pages to be read from the file ? (I don't think)

But since I already do an mlockall() at the beginnig, there is no need of additional mlock()s

I had an additional idea: mmap()ing the data directly into the circular-buffer and dirty the pages through writing one word for each page, it might be a bit faster, because you avoid a memcpy() , but
the problem is that since the audio thread acesses this buffer, I have to implement some communication to signal the disk thread to unmap the region, or alternatively keep track of previously mapped regions in the disk thread, which makes things more complicated.
I'm dealing with a data rate of about 5MB/sec (from disk), therefore the main bottleneck is the disk not the memcpy() performance.
But I will try to mmap()/munmap() tracking and mmap() directly into the circular-buffer, let's see if I can lower the CPU load of an additional 1-2%. :-)

BTW, does anyone know the best method to measure the real % of CPU idle time, since top is pretty unreliabe.

Quoted 7 lines
Quoted 3 lines

Does anyone know if there it/will be a way to do unbuffered mmap() ? I think streaming apps would benefit quite a bit from this.

mmap requires buffering - you are sharing the page with the system page cache. If you mean you want an mgoaway() to go with munmap() thats what madvise() provides on some other systems but not Linux yet

This will be nice, similar to O_DIRECT performance, I will use whatever comes first.
:-)

regards,
Benno.

#3 lore raw

Re: mmap() better than read() fro streaming, Was: Re: Streaming disk

In reply to Alan Cox
Quoted 1 line · Benno Senoner

will mlock() automatically cause all pages to be read from the file ?

Yes.

Quoted 2 lines

But since I already do an mlockall() at the beginnig, there is no need of additional mlock()s

Yes (if you use MCL_FUTURE).

Andrea

#4 lore raw
In reply to Alan Cox
Quoted 4 lines · Andrea Arcangeli
Quoted 1 line · Benno Senoner

will mlock() automatically cause all pages to be read from the file ?

Quoted 8 lines

Yes.

Quoted 2 lines

But since I already do an mlockall() at the beginnig, there is no need of additional mlock()s

Yes (if you use MCL_FUTURE).

Andrea

Yes, I run with mlockall(MCL_CURRENT|MCL_FUTURE)
I can confirm this now:
I just removed my memcpy()s for testing purposes, and the app loads the data nicely.
Pratically this means that if you enabled MCL_FUTURE,
mmap() will return only after all the mapped segment of the file resides in memory.
Correct me please if I am wrong.

regards,
Benno.

#5 lore raw
In reply to Alan Cox

Hi,

Quoted 4 lines · Benno Senoner

Yes, I run with mlockall(MCL_CURRENT|MCL_FUTURE)
I can confirm this now:
I just removed my memcpy()s for testing purposes, and the app loads the data nicely.

As it should be. mlock/mlockall apply to _all_ pages which appear in your process' address space, regardless of whether they are page cache or local data pages.

Quoted 3 lines

Pratically this means that if you enabled MCL_FUTURE, mmap() will return only after all the mapped segment of the file resides in memory. Correct me please if I am wrong.

Yes. If MCL_FUTURE is used then all future mmap()s are automatically marked as locked, and we do a make_pages_present() before returning from mmap. This is exactly what mlockall is designed for: memory access should simply never page fault if you use MCL_CURRENT | MCL_FUTURE. This is even true for writable MAP_PRIVATE pages: if you have full mlockall enabled, then the make_pages_present() will do a copy-on-write page fault for you.

--Stephen