fadvise syscall?

Has anyone ever done an madvise(2)-type syscall for file descriptors? (or does the capability exist and I'm missing it?)

I was thinking, in playing around with stuff like cp(1) I've found that standard read(2) and write(2) of a 4-8K buffer is the fastest solution overall, in addition to providing the useful side effect of better error reporting, such as ENOSPC report. Better error reporting than the alternative I see anyway, mmap(2).

So... we have madvise, why not fadvise? I would love the capability for applications to provide hints to the OS like madvise, but for file descriptors...

Quoted 3 lines · Jeff Garzik

Has anyone ever done an madvise(2)-type syscall for file descriptors? (or does the capability exist and I'm missing it?)

Well, question is: is madvise() any use? :)

Quoted 6 lines

was thinking, in playing around with stuff like cp(1) I've found that standard read(2) and write(2) of a 4-8K buffer is the fastest solution overall, in addition to providing the useful side effect of better error reporting, such as ENOSPC report. Better error reporting than the alternative I see anyway, mmap(2).

4k to 8k is best on x86 at least. And if you're actually going to *use* each byte in the file, the zero-copy characteristics of mmap aren't worth much at all.

Quoted 4 lines

So... we have madvise, why not fadvise? I would love the capability for applications to provide hints to the OS like madvise, but for file descriptors...

The one hint which I can think of which would be beneficial would be an equivalent to MADV_SEQUENTIAL. Something which says "this is a big streaming read/write - don't go and evict other stuff because of it". O_STREAMING perhaps. Or working dropbehind heuristics, although I suspect that explicit controls will always do better.

For MADV_RANDOM, readahead window scaling should get that right.

What else were you thinking of?

Quoted 11 lines · Andrew Morton

Quoted 3 lines · Jeff Garzik

So... we have madvise, why not fadvise? I would love the capability for applications to provide hints to the OS like madvise, but for file descriptors...

The one hint which I can think of which would be beneficial would be an equivalent to MADV_SEQUENTIAL. Something which says "this is a big streaming read/write - don't go and evict other stuff because of it". O_STREAMING perhaps. Or working dropbehind heuristics, although I suspect that explicit controls will always do better.

Quoted 3 lines

For MADV_RANDOM, readahead window scaling should get that right.

What else were you thinking of?

Hints for,
* sequential read
* sequential write
* sequential write, where the application considers the data it's writing to be unlikely to be read again any time soon (hopefully implying to the page cache that these pages have low value as cacheable objects)
* some sort of streaming hints, implying that the application cares a lot about maintaining some minimum i/o rate. note I said hint, not requirement. -not- guaranteed-rate-IO.

I might even go so far as to advocate identifying common usage patterns, and creating hint constants for them, even if we don't support them in the kernel immediately (if ever). Makes the interface much more future-proof, at the expense of a few integers in a 32-bit numberspace, and a few more bytes in the C compiler's symbol table.

All of what you are asking for exists in Windows and all the semantics are implemented through a very powerful open(2) equivalent. I don't see why we shouldn't do the same. It makes more sense to me than inventing yet another system call...

Quoted 4 lines · Anton Altaparmakov

All of what you are asking for exists in Windows and all the semantics are implemented through a very powerful open(2) equivalent. I don't see why we shouldn't do the same. It makes more sense to me than inventing yet another system call...

It is easier for application writers to code:

[...]

#ifdef HAVE_FADVISE
        (void)fadvise(fd, FADV_STREAMING);

#endif
[...]

Than to have a forest of #ifdefs to determine which O_* flags are supported. After all, we still want our programs to run under Solaris. :-)

It is easier for application writers to code:

[...]

#ifdef HAVE_FADVISE
	(void)fadvise(fd, FADV_STREAMING);

#endif
[...]

Than to have a forest of #ifdefs to determine which O_* flags are supported. After all, we still want our programs to run under Solaris. :-)

Quoted 12 lines · Andrew Morton

Quoted 4 lines · Jeff Garzik

So... we have madvise, why not fadvise? I would love the capability for applications to provide hints to the OS like madvise, but for file descriptors...

The one hint which I can think of which would be beneficial would be an equivalent to MADV_SEQUENTIAL. Something which says "this is a big streaming read/write - don't go and evict other stuff because of it". O_STREAMING perhaps. Or working dropbehind heuristics, although I suspect that explicit controls will always do better.

Quoted 4 lines

For MADV_RANDOM, readahead window scaling should get that right.

What else were you thinking of?

Hints for,
* sequential read
* sequential write
* sequential write, where the application considers the data it's writing to be unlikely to be read again any time soon (hopefully implying to the page cache that these pages have low value as cacheable objects)
* some sort of streaming hints, implying that the application cares a lot about maintaining some minimum i/o rate. note I said hint, not requirement. -not- guaranteed-rate-IO.

I might even go so far as to advocate identifying common usage patterns, and creating hint constants for them, even if we don't support them in the kernel immediately (if ever). Makes the interface much more future-proof, at the expense of a few integers in a 32-bit numberspace, and a few more bytes in the C compiler's symbol table.

There is a posix_fadvise() syscall in the POSIX Advanced Realtime specification
http://www.opengroup.org/onlinepubs/007904975/functions/posix_fadvise.html

I don't know if this has been mentioned on linux-kernel before, but in January, the Open Group, in cooperation with IEEE, added the POSIX functionality to their specification and made it available online for free. It's at
http://www.opengroup.org/onlinepubs/007904975/toc.htm

There are some useful tables at
http://www.unix-systems.org/version3/online.html and they ask that you register there so that they know how many people are using the specification.

They don't have a downloadable version of this specification, but they do for the previous versions:
http://www.opengroup.org/onlinepubs/007908799/download/

Ken Hirsch

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 3 lines · Ken Hirsch

There is a posix_fadvise() syscall in the POSIX Advanced Realtime specification
http://www.opengroup.org/onlinepubs/007904975/functions/posix_fadvise.html

Posix or not I still don't see why one would want that. You know what you are going to be using a file for at open time and you are not going to be changing your mind later. If you can show me a single _real_world_ example where one would genuinely want to change from one access pattern to another without closing/reopening a particular file I would agree that fadvise is a good idea but otherwise I think open(2) is the superior approach.

Posix or not I still don't see why one would want that. You know what you are going to be using a file for at open time and you are not going to be changing your mind later. If you can show me a single _real_world_ example where one would genuinely want to change from one access pattern to

without closing/reopening a particular file I would agree that fadvise is

good idea but otherwise I think open(2) is the superior approach.

Anton Altaparmakov writes

Quoted 4 lines

Posix or not I still don't see why one would want that. You know what you are going to be using a file for at open time and you are not going to be changing your mind later. If you can show me a single _real_world_ example where one would genuinely want to change from one access pattern to

another

Quoted 1 line

without closing/reopening a particular file I would agree that fadvise is

a

Quoted 2 lines

good idea but otherwise I think open(2) is the superior approach.

Sure, a database manager can change the access pattern on every query. If there's an index and not too many records are expected to match, it will use a random pattern, otherwise it will use sequential access.

When you want large data streaming, i.e. you start getting worried about memory pressure, then you want open(2) + O_DIRECT. No caching done. Perfect for large data streams and we have that already. I agree that you may want some form of asynchronous read ahead with passed pages being dropped from the cache but that could be just a open(2) + O_SEQUENTIAL (doesn't exist yet).

Quoted 2 lines · Anton Altaparmakov

When you want large data streaming, i.e. you start getting worried about memory pressure, then you want open(2) + O_DIRECT. No caching done.

Perfect

Quoted 3 lines

for large data streams and we have that already. I agree that you may want some form of asynchronous read ahead with passed pages being dropped from the cache but that could be just a open(2) + O_SEQUENTIAL (doesn't

exist yet).

O_DIRECT isn't the right thing for large streaming. You want
readahead and dropbehind. O_DIRECT takes substantial penalties for its lack of copy/cacheing. This works fine in certain circumstances (applications that keep their own caching), but for something like a video or mp3, you'll win with working dropbehind easily.

Last time I heard serious databases use their own memmory management/caching in combination with O_DIRECT, i.e. they bypass the kernel's buffering system completely. Hence I would deem them irrelevant

the problem at hand...

If a database were not to use O_DIRECT I would think it would be using

so it would have madvise already... but I am not a database expert so take this with a pinch of salt...

Quoted 3 lines · Anton Altaparmakov

Last time I heard serious databases use their own memmory management/caching in combination with O_DIRECT, i.e. they bypass the kernel's buffering system completely. Hence I would deem them irrelevant

to

Quoted 2 lines

the problem at hand...

Quoted 1 line

If a database were not to use O_DIRECT I would think it would be using

mmap

Quoted 2 lines

so it would have madvise already... but I am not a database expert so take this with a pinch of salt...

I don't think that either MySQL or PostgreSQL use O_DIRECT; I just grepped the source and didn't find it. They can't use mmap() because it uses up too much process address space.

It's true that commercial databases mostly do their own scheduling and caching, and if they are the only thing running on your system and you tune them right, that works. But it's not necessarily a good thing. If there are other processes on your system, there would be a benefit if the DBMS could inform the operating system of its intentions.

A posix_fadvise() call would be a start, but you could potentially go beyond that.

We don't need fadvise IMHO. That is what open(2) is for. The streaming request you are asking for is just a normal open(2). It will do read ahead which is perfect for streaming (of data size << RAM size in its current form).

When you want large data streaming, i.e. you start getting worried about memory pressure, then you want open(2) + O_DIRECT. No caching done. Perfect for large data streams and we have that already. I agree that you may want some form of asynchronous read ahead with passed pages being dropped from the cache but that could be just a open(2) + O_SEQUENTIAL (doesn't exist yet).

All of what you are asking for exists in Windows and all the semantics are implemented through a very powerful open(2) equivalent. I don't see why we shouldn't do the same. It makes more sense to me than inventing yet another system call...

Quoted 6 lines

Quoted 4 lines

I disagree, and here's the main reasons:

* fadvise(2) usefulness extends past open(2). It may be useful to call it at various points during runtime.

open(/proc/self/fd/0, O_NEW_FLAGS)?

So to use fadvise(), the system must have /proc mounted?

Hi!

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

I disagree, and here's the main reasons:

Quoted 6 lines

Quoted 4 lines

Quoted 2 lines

* fadvise(2) usefulness extends past open(2). It may be useful to call it at various points during runtime.

open(/proc/self/fd/0, O_NEW_FLAGS)?

So to use fadvise(), the system must have /proc mounted?

I think it is way more feasible than adding new syscall.

* fadvise(2) usefulness extends past open(2). It may be useful to call it at various points during runtime.

* I think putting hints in open(2) is the wrong direction to go. Hints have a potential to be very flexible. open(2) O_xxx bits are not to be squandered lightly, while I see a lot more value in being a little more loose and free with the bit assignment for an "fadvise mask" (just a list of hint bits). IMO it should be easier to introduce and retire hints, far easier than O_xxx flags.

Note that it applies to a file descriptor. If posix_fadvise(FADV_DONTNEED) is called against a file descriptor, and someone else has an fd open against the same file, that other user gets their foot shot off. That's OK.

Quoted 4 lines · Andrew Morton

Note that it applies to a file descriptor. If posix_fadvise(FADV_DONTNEED) is called against a file descriptor, and someone else has an fd open against the same file, that other user gets their foot shot off. That's OK.

Let me verify that I understand what you're saying. Process A and B independently open the file. The file is already in the cache (because other processes regularly read this file). Process A is slowly reading stuff. Process B does FADV_DONTNEED on the whole file. The pages are dropped.

You're saying this is OK? How about this DoS attack:

        int fd = open ("/lib/libc.so", O_RDONLY, 0);
        while (1) {
                posix_fadvise (fd, 0, 0, FADVISE_DONTNEED);
                sleep (1);
        }

Let me see that disc head move! Wheeee!

Quoted 13 lines · Richard Gooch

Quoted 4 lines · Andrew Morton

Note that it applies to a file descriptor. If posix_fadvise(FADV_DONTNEED) is called against a file descriptor, and someone else has an fd open against the same file, that other user gets their foot shot off. That's OK.

Let me verify that I understand what you're saying. Process A and B independently open the file. The file is already in the cache (because other processes regularly read this file). Process A is slowly reading stuff. Process B does FADV_DONTNEED on the whole file. The pages are dropped.

Quoted 9 lines

You're saying this is OK? How about this DoS attack:

        int fd = open ("/lib/libc.so", O_RDONLY, 0);
        while (1) {
                posix_fadvise (fd, 0, 0, FADVISE_DONTNEED);
                sleep (1);
        }

Let me see that disc head move! Wheeee!

POSIX_FADV_DONTNEED could only unmap pages from the caller's VMA's, so the problem would only affect other processes which share the same mm - CLONE_MM threads.

If some other process has a reference on the pages then they wouldn't get unmapped as a result of this. It's the same as madvise(MADV_DONTNEED).

We don't need fadvise IMHO. That is what open(2) is for. The streaming request you are asking for is just a normal open(2). It will do read ahead which is perfect for streaming (of data size << RAM size in its current form).

posix_fadvise() looks to be a fine interface:

We'll need to cheat a bit on the offset/len thing for NORMAL and SEQUENTIAL - just apply it to the whole file - we don't want to have to attach an arbitrary number of silly range objects to each file for this. (We already cheat a bit this way with msync).

Given this, I don't see a persuasive need to implement a non-standard interface. It takes an off_t, so posix_fadvise64() is also needed.

A 2.4 implementation could be done any time. If anyone decides to do this, please let me know...

Other applications can use this sort of stuff. DBM could, for instance. So might GIMP. Etc. Dynamic hints have real world applications.

Quoted 3 lines · Anton Altaparmakov

We don't need fadvise IMHO. That is what open(2) is for. The streaming request you are asking for is just a normal open(2). It will do read ahead which is perfect for streaming (of data size << RAM size in its current form).

A quick real world example of where fadvise can work well.
Imagine a database appliction that doesn't use O_DIRECT (for whatever reason, could even be that they don't trust the linux implementation yet :-).

So, this database gets a query. That query requires a full table
scan, so it calls fadvise(fd, F_SEQUENTIAL). Then another query does row-specific access, and caching helps. So it wants to turn off F_SEQUENTIAL.

...

Quoted 3 lines

Given this, I don't see a persuasive need to implement a non-standard interface. It takes an off_t, so posix_fadvise64() is also needed.

agreed WRT non-standard.

Are we required to have both foo and foo64 variants? If I had my druthers, I would just do the foo64 version.

to be fair, fcntl(2) could be used in conjunction with open(2), to do dynamic hints.

Followup to:  <[email protected]>
By author:    Anton Altaparmakov <[email protected]>
In newsgroup: linux.dev.fs.devel

Quoted 1 line

Quoted 5 lines

Ok, so basically we want both fadvise() and open(2) semantics, with the open(2) being a superset of the fadvise() capabilities (some things no longer make sense to be specified once the file is open). They can of course both be calling the same common helpers inside the kernel...

If they're open() flags, they should probably be controlled with fcntl() rather than with a new system call.

Quoted 4 lines

Followup to:  <[email protected]>
By author:    Anton Altaparmakov <[email protected]>
In newsgroup: linux.dev.fs.devel

Quoted 5 lines

Quoted 2 lines

Ok, so basically we want both fadvise() and open(2) semantics, with the open(2) being a superset of the fadvise() capabilities (some things no longer make sense to be specified once the file is open). They can of course both be calling the same common helpers inside the kernel...

If they're open() flags, they should probably be controlled with fcntl() rather than with a new system call.

Then posix_fadvise interface can be implemented in libc using fcntl.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Andrew" == Andrew Morton <[email protected]> writes:

Quoted 1 line · Jan Hudec

Then posix_fadvise interface can be implemented in libc using fcntl.

It is far better for future-proofing the interface IMO if fadvise is implementing directly. Hints are less important than open O_xxx flags or F_xxx flags, because an implementation can safely ignore 100% of the fadvise hints, if it so chooses. One cannot say the same thing for open/fcntl flags.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

"Andrew" == Andrew Morton <[email protected]> writes:

Andrew> O_DIRECT is broken against RAID0 (at least) in 2.5 at present. Andrew> The RAID driver gets sent BIOs which straddle two or more Andrew> chunks and RAID spits out lots of unpleasant warnings. Neil Andrew> has been informed...

Yep. I've been porting my original kiobuf based request splitter to biobufs. It's almost there, I've just been extremely busy with something else for a while.

It's not only when you straddle chunks. The current code does not handle requests straddling RAID zones either.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Andrew" == Andrew Morton <[email protected]> writes:

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Andrew" == Andrew Morton <[email protected]> writes:

Andrew> If that's really the only way in which we can solve this Andrew> problem, would it not be better to pass information up to the Andrew> higher layer, telling it when the BIO which is currently under Andrew> assembly cannot be grown further? Say,
Andrew> blk_can_i_add_more_stuff_to_this_bio()?

We tried different approaches. One of them was to be able to signal to upper layers that your I/O was too big and please submit smaller chunks. Running with that, however, the I/O size converged against small requests because you'd often start an I/O - say 4K - from a stripe boundary. And that would kill it right off.

So unless the filesystem knows about stripe/device boundaries it's really hard to get the size signalling right. And then what happens when you stack LVM and MD?

In the end, cloning the kiobuf from the above and adjusting offset/length in the children turned out to be the best approach.

Quoted 4 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Eric" == Eric W Biederman <[email protected]> writes:

In the end, cloning the kiobuf from the above and adjusting offset/length in the children turned out to be the best approach.

Quoted 12 lines

We don't need fadvise IMHO. That is what open(2) is for. The streaming request you are asking for is just a normal open(2). It will do read ahead which is perfect for streaming (of data size << RAM size in its current form).

When you want large data streaming, i.e. you start getting worried about memory pressure, then you want open(2) + O_DIRECT. No caching done. Perfect for large data streams and we have that already. I agree that you may want some form of asynchronous read ahead with passed pages being dropped from the cache but that could be just a open(2) + O_SEQUENTIAL (doesn't exist yet).

Quoted 4 lines

All of what you are asking for exists in Windows and all the semantics are implemented through a very powerful open(2) equivalent. I don't see why we shouldn't do the same. It makes more sense to me than inventing yet another system call...

I disagree, and here's the main reasons:

* fadvise(2) usefulness extends past open(2). It may be useful to call it at various points during runtime.

* I think putting hints in open(2) is the wrong direction to go. Hints have a potential to be very flexible. open(2) O_xxx bits are not to be squandered lightly, while I see a lot more value in being a little more loose and free with the bit assignment for an "fadvise mask" (just a list of hint bits). IMO it should be easier to introduce and retire hints, far easier than O_xxx flags.

Quoted 6 lines

I disagree, and here's the main reasons:

* fadvise(2) usefulness extends past open(2). It may be useful to call it at various points during runtime.

open(/proc/self/fd/0, O_NEW_FLAGS)?

Quoted 10 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

I disagree, and here's the main reasons:

* fadvise(2) usefulness extends past open(2). It may be useful to

call

Quoted 5 lines

Quoted 3 lines

Quoted 1 line

it at various points during runtime.

open(/proc/self/fd/0, O_NEW_FLAGS)?

So to use fadvise(), the system must have /proc mounted?

Quoted 1 line

I think it is way more feasible than adding new syscall.

Sorry but it is silly. (-; What's wrong with open("filename", O_FLAGS); followed by fcntl(); if you want to modify them after opening. That is a lot cleaner than going via proc in such a way...

posix_fadvise() can then be implemented in userspace and that can go via fcntl(). That way we have the best of both worlds.