Thread implementations...

: Not for very large files: the forget-behind is absolutely critical in : that case.

SunOS' local file system, UFS, implements the following alg for forget behind on all types of accesses (SunOS has a unified page cache, all accesses are mmap based, read/write are implemented by the kernel doing an mmap and then a bcopy):

	if ((free_memory < we_will_start_paging_soon) &&
	    (offset is clust_size multiple) &&
	    (offset > small_file) &&
	    (access is sequential)) {
	    	free_behind(vp, offset - clust_size, clust_size);
	}

in the ufs_getpage() code.

I'll admit this was a hack, but it had some nice attributes that you might want to consider:

1) it was nice that I/O took care of itself. The pageout daemon is pretty costly (Stephen, we talked about this at Linux Expo - this is why I want a pageout daemon that works on files, not on pages).

2) Small files aren't worth the trouble and aren't the cause of the trouble.

3) Random access frequently wants caching and randoms are expensive to bring in.

4) I/O is freed in large chunks, not a page at a time. It's about as costly to bring in one page as bring in 64-256K these days.

Hi,

Looks entirely reasonable. I've been thinking of something very similar but just a little more complex, so that we can also cleanly handle the case of sequential mmap()ed reads, both of mapped files and potentially of anonymous datasets. The difference there is that if we are dealing with tiled data, then we may need to allow a larger window between the current pagein cursor and the forget-behind cursor. Again, if we just unmap the pages and place them on a high-priority reuse queue, then getting the guess wrong just results in a minor fault unless we do actually reuse the memory before accessing the data again.

Yes, and Ingo and I have been talking about ways of doing it.

Small files benefit from a similar scheme. For small sequentially-accessed files, as they age, we want to remove the entire file from cache at once. Repopulating a sequential file's fragmented cache is expensive anyway, so it may in fact be _cheaper_ to do this than to just throw out one page at a time.

As long as we have the concept of a virtual extent, where we define that extent as the natural readahead pattern for the workload, then we want to uncache the same units we readahead. That's normally sequential clusters, but if we have things like Ingo's random swap stats-based prediction logic, then we can use exactly the same extent concept there too.

--Stephen

Hmm, it appears this was the legendary swap readahead code I was looking for a while ago :)

But, ehhh, just what _is_ this random swap stats-based prediction algorithm, and how far from implementation is it?
(and if it isn't implemented yet, what should I do to make it implemented; swapin readahead is very wanted on my memory-starved box...)

Rik.

+-------------------------------------------------------------------+
| Linux memory management tour guide.        [email protected] |
| Scouting Vries cubscout leader.      http://www.phys.uu.nl/~riel/ |
+-------------------------------------------------------------------+

--
This is a majordomo managed list. To unsubscribe, send a message with the body 'unsubscribe linux-mm me@address' to: [email protected]

Hi,

It's a per-swap-page readahead predictor which observes the access patterns for vmas.

It is implemented. It is not in the main kernels, nor does it take advantage of the potential for swap readahead in the 2.1.86+ kernels.

--Stephen
--
This is a majordomo managed list. To unsubscribe, send a message with the body 'unsubscribe linux-mm me@address' to: [email protected]

Then where is it? It would be great to test and it would make an excellent link with description for the Linux MM homepage...

Besides, I'm currently somewhat memory starved and I would really like to test and possibly improve or integrate this piece of code with the main kernel.

I know it's too late for inclusion now, but I'm willing to keep the patch up-to-date with the kernel up to the date of inclusion.

Rik.

+-------------------------------------------------------------------+
| Linux memory management tour guide.        [email protected] |
| Scouting Vries cubscout leader.      http://www.phys.uu.nl/~riel/ |
+-------------------------------------------------------------------+

--
This is a majordomo managed list. To unsubscribe, send a message with the body 'unsubscribe linux-mm me@address' to: [email protected]

Hi,

You'd have to ask Ingo for it.

--Stephen
--
This is a majordomo managed list. To unsubscribe, send a message with the body 'unsubscribe linux-mm me@address' to: [email protected]