Skip to contentSkip to conversation

POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

DISCUSSION David Rolenc 4 participants 9 messages Aug 26, 2004 – Aug 27, 2004

lore
Share & tools
lore

David Rolenc

author
#0 lore raw

I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is not implemented any differently than POSIX_FADV_WILLNEED. Am I missing something? I want to read data from disk with readahead and drop the data from the page cache as soon as I am done with it. Do I have to call fadvise with POSIX_FADV_DONTNEED after every read?

Thanks,

David Rolenc
RT Logic!
www.rtlogic.com

#1 lore raw
Quoted 4 lines · David Rolenc

I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is not implemented any differently than POSIX_FADV_WILLNEED. Am I missing something?

Yes, they are currently the same. Their difference in theory is subtle:
NOREUSE says "I will use this once in the future" and WILLNEED says "I will use this [some number of times] in the future".

So for both of them, you want to keep the data in the page cache. But for NOREUSE you could do drop-behind of the pages. But Linux does not currently do drop-behind, so we have them both do the same thing.

Quoted 3 lines

I want to read data from disk with readahead and drop the
data from the page cache as soon as I am done with it. Do I have to call fadvise with POSIX_FADV_DONTNEED after every read?

Yes, you do. Or every couple of reads.

The sort of applications that need DONTNEED already are setup in a way to make that easy. You either have a streaming loop, like:

	do {
		read
		process
	} while (repeat);

Or you are readings lots of files (like updatedb).

In the first case, just stick the fadvise call after the processing. In the latter case, call fadvise before closing the file.

You don't _have_ to call fadvise after every reading. Just whenever you want to drop the pages. Whenever it makes sense.

Robert Love

#3 lore raw
In reply to David Rolenc
Quoted 1 line · David Rolenc

I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is not implemented any differently than POSIX_FADV_WILLNEED. Am I missing something? I want to read data from disk with readahead and drop the data from the page cache as soon as I am done with it. Do I have to call fadvise with POSIX_FADV_DONTNEED after every read?

And will this work over nfs?

David

#4 lore raw

På to , 26/08/2004 klokka 14:38, skreiv David Greaves:

Quoted 10 lines
Quoted 6 lines · David Rolenc

I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is not implemented any differently than POSIX_FADV_WILLNEED. Am I missing something? I want to read data from disk with readahead and drop the data from the page cache as soon as I am done with it. Do I have to call fadvise with POSIX_FADV_DONTNEED after every read?

And will this work over nfs?

What do you mean?

The client will of course respect fadvise() if the generic VM code supports it, but there is no NFS protocol support for this, so the client is not able to communicate your fadvise call on to the server.

Cheers,
Trond

author
#8 lore raw
In reply to David Greaves
Quoted 6 lines · David Greaves
Quoted 1 line · David Rolenc

I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is not implemented any differently than POSIX_FADV_WILLNEED. Am I missing something? I want to read data from disk with readahead and drop the data from the page cache as soon as I am done with it. Do I have to call fadvise with POSIX_FADV_DONTNEED after every read?

And will this work over nfs?

The system will not be using nfs at all.

Quoted 8 lines

David
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

author
#2 lore raw
In reply to Robert Love

Is there any technical reason that we couldn't set a flag in the struct file* and put code in the read/write calls to handle shrinking the page cache? This is similar to what was done with O_STREAMING in 2.4. It seems like it would conform more to the behavior expected from POSIX_FADV_NOREUSE, and I wouldn't have to make a bunch of system calls :-)

-Dave

Quoted 50 lines · Robert Love
Quoted 3 lines · David Rolenc

I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is not implemented any differently than POSIX_FADV_WILLNEED. Am I missing something?

Yes, they are currently the same. Their difference in theory is subtle:
NOREUSE says "I will use this once in the future" and WILLNEED says "I will use this [some number of times] in the future".

So for both of them, you want to keep the data in the page cache. But for NOREUSE you could do drop-behind of the pages. But Linux does not currently do drop-behind, so we have them both do the same thing.

Quoted 3 lines

I want to read data from disk with readahead and drop the data from the page cache as soon as I am done with it. Do I have to call fadvise with POSIX_FADV_DONTNEED after every read?

Yes, you do. Or every couple of reads.

The sort of applications that need DONTNEED already are setup in a way to make that easy. You either have a streaming loop, like:

	do {
		read
		process
	} while (repeat);

Or you are readings lots of files (like updatedb).

In the first case, just stick the fadvise call after the processing. In the latter case, call fadvise before closing the file.

You don't _have_ to call fadvise after every reading. Just whenever you want to drop the pages. Whenever it makes sense.

Robert Love

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#5 lore raw
In reply to Trond Myklebust
Quoted 14 lines · Trond Myklebust

På to , 26/08/2004 klokka 14:38, skreiv David Greaves:

Quoted 10 lines
Quoted 3 lines · David Rolenc

I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is not implemented any differently than POSIX_FADV_WILLNEED. Am I missing something? I want to read data from disk with readahead and drop the data from the page cache as soon as I am done with it. Do I have to call fadvise with POSIX_FADV_DONTNEED after every read?

And will this work over nfs?

Quoted 7 lines

What do you mean?

The client will of course respect fadvise() if the generic VM code supports it, but there is no NFS protocol support for this, so the client is not able to communicate your fadvise call on to the server.

Perfect answer
thank you

I want my nfs client to communicate this to my nfs server. Thus avoiding my nfs server having a cache of useless video. I can see this becoming an important benefit for video distribution (an area linux is likely to see more of)

David

#6 lore raw

På fr , 27/08/2004 klokka 04:25, skreiv David Greaves:

Quoted 4 lines

I want my nfs client to communicate this to my nfs server. Thus avoiding my nfs server having a cache of useless video.
I can see this becoming an important benefit for video distribution (an area linux is likely to see more of)

Then you are probably going to have to invent some out-of-band protocol in order to do so. Creating such a protocol shouldn't be a hard task, but getting all those server manufacturers to adopt it afterwards probably will be be. ;-)

Alternatively, if you can make a good case (and write a decent RFC), you might be able to persuade the IETF working group to add this feature into a future minor version of NFSv4.

Cheers,
Trond

#7 lore raw
In reply to David Greaves

På fr , 27/08/2004 klokka 04:25, skreiv David Greaves:

Quoted 4 lines

I want my nfs client to communicate this to my nfs server. Thus avoiding my nfs server having a cache of useless video.
I can see this becoming an important benefit for video distribution (an area linux is likely to see more of)

Then that will have to be done either through some out-of-band protocol, or as part of some future minor revision of NFSv4 (assuming that you are able to make a good case for it in an RFC and present it to the IETF's NFS working group).

Cheers,
Trond