Skip to contentSkip to conversation

userspace pagecache management tool

DISCUSSION Andrew Morton 11 participants 45 messages Mar 3, 2007 – Mar 8, 2007

lore
Share & tools
lore
#0 lore raw

I've uploaded to http://userweb.kernel.org/~akpm/pagecache-management/ a little tool which permits the management of the pagecache usage of arbitrary applications. Effectively it prevents the targetted application from using any pagecache at all.

It is to address the "waah, backups fill my memory with pagecache" and the "waah, updatedb swapped everything out" and the "waah, copying a DVD gobbled all my memory" problems.

Although it is little more than a proof-of-concept it seems to be fairly useful. When running

pagecache-management.sh dd if=100-mb-file of=foo
or
pagecache-management.sh cp -a /usr/src/linux-2.6.20 /usr/src/foo

the amount of pagecache in the machine is pretty much unaltered. Maybe a megabyte of additional cache in the second case, because of ext3 indirect blocks.

The tool uses an LD_PRELOAD hack to intercept glibc's read(), pread(), write(), pwrite(), close() and dup2() functions. pagecache control is done via posix_fadvise() and sync_file_range().

btw, for a while I was using fdatasync() on close(), but it was slow, because fdatasync() has to run an ext3 commit to commit the metadata. sync_file_range() doesn't do that, and the copy-a-kernel-tree testcase sped up by a factor of five. So sync_file_range() rocks, but the powerpc guys haven't wired it up yet.

There is much more which could be done to make this code smarter, but I think the lesson here is that we can produce a far, far better result doing this work in userspace than we could ever hope to do with an in-kernel implementation. There are some enhancement suggestions in the documentation file.

It would be good if someone could turn this into a real product, get it fed into distros. Once the design is settled we should look at moving all the functionality into glibc itself, IMO, and get rid of the LD_PRELOAD trick.

It might help if the kernel offered APIs which permit userspace to query the number of resident pages in a file (well, actually it already does,
kind-of: mincore()) and the ability to query the number of dirty pages in a file, etc. I'd be reluctant to tie the kernel ABI too closely to the current pagecache implementation and data structures, but we can look at these things.

#1 lore raw
Quoted 3 lines · Andrew Morton

It is to address the "waah, backups fill my memory with pagecache" and the "waah, updatedb swapped everything out" and the "waah, copying a DVD gobbled all my memory" problems.

By removing pressure from the page cache, you'll only allow updatedb to grow the inode and dentry caches larger than before.

I am sick and tired of the "this is hard, let userspace do it" attitude.

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#2 lore raw
Quoted 8 lines · On Sat, 03 Mar 2007 15:40:42 -0500 Rik van Riel <[email protected]> wrote:
Quoted 3 lines · Andrew Morton

It is to address the "waah, backups fill my memory with pagecache" and the "waah, updatedb swapped everything out" and the "waah, copying a DVD gobbled all my memory" problems.

By removing pressure from the page cache, you'll only allow updatedb to grow the inode and dentry caches larger than before.

Well duh.

That's a two-order-of-magnitude lesser problem and only affects one of many problematic workloads.

Quoted 1 line

I am sick and tired of the "this is hard, let userspace do it" attitude.

Anything you try to do in-kernel will catastrophically screw up some workloads. You don't have a chance of getting this right.

You are the kernel. The user just read an entire kernel tree. You face a binary decision: do you cache that tree or do you not? Your time starts now. What is your answer?

#3 lore raw
Quoted 6 lines · Andrew Morton
Quoted 1 line · On Sat, 03 Mar 2007 15:40:42 -0500 Rik van Riel <[email protected]> wrote:

I am sick and tired of the "this is hard, let userspace do it" attitude.

Anything you try to do in-kernel will catastrophically screw up some workloads. You don't have a chance of getting this right.

Any time you follow the directions of one userspace program, you can screw up others. I suspect that userspace has far less of a chance of getting it right than the kernel.

ALSA would be a good example of why it is bad to export tuning knobs directly to userspace - many sound cards have non-standard names for the volume controls, making it almost impossible for userspace to present the user with a simple user interface for tweaking the volume.

Quoted 3 lines

You are the kernel. The user just read an entire kernel tree. You face a binary decision: do you cache that tree or do you not? Your time starts now. What is your answer?

Lets turn this around.

The user has been accessing the kernel tree over and over again, for hours on end (compile testing a patch). Along comes a backup program, that tells you to evict the whole thing from the cache.

What do you do?

How can you make global policy decisions based on the intent of one program?

Only the kernel knows the state of the whole system and has observed the behaviour of all the processes. One process has no idea what the other processes in the system are doing.

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#4 lore raw
Quoted 4 lines · Rik van Riel

The user has been accessing the kernel tree over and over again, for hours on end (compile testing a patch). Along comes a backup program, that tells you to evict the whole thing from the cache.

This is arguably due to a linux misimplementation of posix_fadvise. SuS v3 clearly states:

The posix_fadvise() function shall advise the implementation on the expected behavior of the application with respect to the data in the file associated with the open file descriptor

Note how it refers to the *application*. This is reiterated here:

POSIX_FADV_WILLNEED
Specifies that the application expects to access the specified data in the near future.

POSIX_FADV_DONTNEED
Specifies that the application expects that it will not access the specified data in the near future.

POSIX_FADV_NOREUSE
Specifies that the application expects to access the specified data once and then not reuse it thereafter.

Linux however implements posix_fadvise globally:

POSIX_FADV_DONTNEED
attempts to free cached pages associated with the specified region.

POSIX_FADV_WILLNEED and POSIX_FADV_NOREUSE
both initiate a non-blocking read of the specified region into the page cache.

Quoted 2 lines

How can you make global policy decisions based on the intent of one program?

By not doing so. Andrew's program is fine in principle, except that the linux kernel treats the communication of a program's intent as a global instruction.

Also, Andrew's description is a tad misleasing, as the size of the page cache might be altered a lot in case content is accessed that was previously cached!

With my userspace hat on, I'd love to have a proper way to communicate my *program's* expectations to the kernel, without stomping other programs.

Also with the same hat on, I hope to rarely *need* to communicate my expectations because the kernel correctly predicts many cases.

Bert

Signature · 2 lines
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services
#5 lore raw
Quoted 4 lines · On Sat, 3 Mar 2007 22:41:09 +0100 bert hubert <[email protected]> wrote:
Quoted 2 lines

How can you make global policy decisions based on the intent of one program?

By not doing so.

yup.

Quoted 3 lines

Andrew's program is fine in principle, except that the
linux kernel treats the communication of a program's intent as a global instruction.

argh.

That felt good - let's do it again.

argh.

It is *not* a global instruction. It uses setenv, so the user's policy affects only the target process and its forked children.

Quoted 3 lines

Also, Andrew's description is a tad misleasing, as the size of the page cache might be altered a lot in case content is accessed that was previously cached!

That's true.

Although if the user knows that he'll want to use that data again soon, and he elects to purge it all from cache beforehand then we're dealing with a pretty dumb user.

If this user doesn't plan to use the data again, but some other user does, then he loses.

Quoted 2 lines

With my userspace hat on, I'd love to have a proper way to communicate my *program's* expectations to the kernel, without stomping other programs.

That is the aim and effect of this work.

Quoted 2 lines

Also with the same hat on, I hope to rarely *need* to communicate my expectations because the kernel correctly predicts many cases.

yup. It's those odd cases where it goes wrong which I'm addressing here.

And of course there's no way for the kernel to work out that it's about to go wrong - kernel can't read user's mind.

Well. That's not strictly true. One could envisage a database-backed learning program which observes the users work patterns and, based on various pattern-matchings, works out what the best strategy is likely to be when the user starts some operation which he has performed previously.

#6 lore raw
Quoted 14 lines · Andrew Morton
Quoted 3 lines · On Sat, 3 Mar 2007 22:41:09 +0100 bert hubert <[email protected]> wrote:
Quoted 2 lines

How can you make global policy decisions based on the intent of one program?

By not doing so.

yup.

Quoted 3 lines

Andrew's program is fine in principle, except that the
linux kernel treats the communication of a program's intent as a global instruction.

argh.

Quoted 7 lines

That felt good - let's do it again.

argh.

It is *not* a global instruction. It uses setenv, so the user's policy affects only the target process and its forked children.

... and all other processes accessing the same file(s)!

Your library and the system calls may be limited to one process, but the consequences are global.

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#7 lore raw
Quoted 7 lines · On Sat, 03 Mar 2007 17:19:00 -0500 Rik van Riel <[email protected]> wrote:
Quoted 2 lines

It is *not* a global instruction. It uses setenv, so the user's policy affects only the target process and its forked children.

... and all other processes accessing the same file(s)!

Your library and the system calls may be limited to one process, but the consequences are global.

Yes. So what? If the user wants to go and evict libc.so from pagecache then he can do so - the kernel has provided syscalls with which this can be done for at least seven years. Bad user, shouldn't do that.

#8 lore raw
Quoted 4 lines · Andrew Morton
Quoted 2 lines · On Sat, 03 Mar 2007 17:19:00 -0500 Rik van Riel <[email protected]> wrote:
Quoted 2 lines

It is *not* a global instruction. It uses setenv, so the user's policy affects only the target process and its forked children.

Quoted 8 lines
Quoted 4 lines

... and all other processes accessing the same file(s)!

Your library and the system calls may be limited to one process, but the consequences are global.

Yes. So what? If the user wants to go and evict libc.so from pagecache then he can do so - the kernel has provided syscalls with which this can be done for at least seven years. Bad user, shouldn't do that.

Are you saying the user should not use your script with their backup program?

Then what's the point?

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#9 lore raw
Quoted 10 lines · On Sat, 03 Mar 2007 17:28:35 -0500 Rik van Riel <[email protected]> wrote:
Quoted 9 lines · Andrew Morton
Quoted 6 lines · On Sat, 03 Mar 2007 17:19:00 -0500 Rik van Riel <[email protected]> wrote:
Quoted 2 lines

It is *not* a global instruction. It uses setenv, so the user's policy affects only the target process and its forked children.

... and all other processes accessing the same file(s)!

Your library and the system calls may be limited to one process, but the consequences are global.

Quoted 6 lines
Quoted 3 lines

Yes. So what? If the user wants to go and evict libc.so from pagecache then he can do so - the kernel has provided syscalls with which this can be done for at least seven years. Bad user, shouldn't do that.

Are you saying the user should not use your script with their backup program?

No.

This is getting silly.

#10 lore raw
In reply to Andrew Morton
Quoted 11 lines · Andrew Morton
Quoted 7 lines
Quoted 2 lines

It is *not* a global instruction. It uses setenv, so the user's policy affects only the target process and its forked children.

... and all other processes accessing the same file(s)!

Your library and the system calls may be limited to one process, but the consequences are global.

Yes. So what? If the user wants to go and evict libc.so from pagecache then he can do so - the kernel has provided syscalls with which this can be done for at least seven years. Bad user, shouldn't do that.

While I agree with your sentiments that userspace can have a good idea on how to deal with the page cache, your program does more than it claims to do - because of how linux implements posix_fadvise.

I don't think anybody expects or desires your program to actually *evict* the stuff from the cache you are trying access, which happens in case the data was in the cache prior to starting your program.

What people expect is that a solution such as you wrote it simply won't *add* anything to the cache. They don't expect it will actually globally *remove* stuff from the cache.

Making a backup this way would hurt even worse than usual with your pagecache management tool if the file being backupped was still being read.

This is not your fault, but in practice, it makes your program less useful than it could be.

One could conceivably fix that up using mincore and simply not fadvise if a page was in core already.

Bert

Signature · 2 lines
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services
#11 lore raw
Quoted 25 lines · On Sun, 4 Mar 2007 00:01:55 +0100 bert hubert <[email protected]> wrote:
Quoted 11 lines · Andrew Morton
Quoted 7 lines
Quoted 2 lines

It is *not* a global instruction. It uses setenv, so the user's policy affects only the target process and its forked children.

... and all other processes accessing the same file(s)!

Your library and the system calls may be limited to one process, but the consequences are global.

Yes. So what? If the user wants to go and evict libc.so from pagecache then he can do so - the kernel has provided syscalls with which this can be done for at least seven years. Bad user, shouldn't do that.

While I agree with your sentiments that userspace can have a good idea on how to deal with the page cache, your program does more than it claims to do - because of how linux implements posix_fadvise.

I don't think anybody expects or desires your program to actually *evict* the stuff from the cache you are trying access, which happens in case the data was in the cache prior to starting your program.

What people expect is that a solution such as you wrote it simply won't *add* anything to the cache. They don't expect it will actually globally *remove* stuff from the cache.

Quoted 5 lines

Making a backup this way would hurt even worse than usual with your pagecache management tool if the file being backupped was still being read.

This is not your fault, but in practice, it makes your program less useful than it could be.

yup. As I said, it's a proof-of-concept. It's a project. And I have about one free femtosecond per fortnight :(

Quoted 2 lines

One could conceivably fix that up using mincore and simply not fadvise if a page was in core already.

Yes. Let's flesh it out the backup program policy some more:

- Unconditionally invalidate output files

- on entry to read(), probe pagecache, record which pages in the range are present

- on entry to next read(), shoot down those pages from the previous read which weren't in pagecache.

- But we can do better! LRU the page's files up to a certain number of pages.

- Once that point is exceeded, we need to reclaim some pages. Which ones? Well, we've been observing all reads, so we can record which pages were referenced once, and which ones were referenced multiple times so we can do arbitrarily complex page aging in there.

- On close(), nuke all pages which weren't in core during open(), even if this app referenced them multiple times.

- If the backup program decided to read its input files with mmap we're rather screwed. We can't intercept pagefaults so the best we can do is to restore the file's pagecache to its previous state on close().

Or if it's really a problem, get control in there somehow and periodically poll the pagecache occupancy via mincore(), use madvise() then fadvise() to trim it back.

That all sounds reasonably doable. It'd be pretty complex to do it in-kernel but we could do it there too. Problem is if course that the above strategy is explicitly optimised for the backup program and if it's in-kernel it becomes applicable to all other workloads.

#12 lore raw
Quoted 23 lines · Andrew Morton

Yes. Let's flesh it out the backup program policy some more:

- Unconditionally invalidate output files

- on entry to read(), probe pagecache, record which pages in the range are present

- on entry to next read(), shoot down those pages from the previous read which weren't in pagecache.

- But we can do better! LRU the page's files up to a certain number of pages.

- Once that point is exceeded, we need to reclaim some pages. Which ones? Well, we've been observing all reads, so we can record which pages were referenced once, and which ones were referenced multiple times so we can do arbitrarily complex page aging in there.

- On close(), nuke all pages which weren't in core during open(), even if this app referenced them multiple times.

- If the backup program decided to read its input files with mmap we're rather screwed. We can't intercept pagefaults so the best we can do is to restore the file's pagecache to its previous state on close().

Quoted 8 lines

Or if it's really a problem, get control in there somehow and periodically poll the pagecache occupancy via mincore(), use madvise() then fadvise() to trim it back.

That all sounds reasonably doable. It'd be pretty complex to do it in-kernel but we could do it there too. Problem is if course that the above strategy is explicitly optimised for the backup program and if it's in-kernel it becomes applicable to all other workloads.

I can see the above being possible, but I can't see the reason for exposing that complexity to userspace. If I'm the target audience for that API then it's broken as I'd mess it up, or would take too long to get it right.

Can't we just fix the posix_fadvise() implementation to only evict pages paged in by the current process. Perhaps one could possibly just evict pages with _mapcount==0 ?

cheers,
Pádraig.

#13 lore raw
Quoted 28 lines · On Tue, 06 Mar 2007 12:10:49 +0000 P__draig Brady <[email protected]> wrote:
Quoted 27 lines · Andrew Morton

Yes. Let's flesh it out the backup program policy some more:

- Unconditionally invalidate output files

- on entry to read(), probe pagecache, record which pages in the range are present

- on entry to next read(), shoot down those pages from the previous read which weren't in pagecache.

- But we can do better! LRU the page's files up to a certain number of pages.

- Once that point is exceeded, we need to reclaim some pages. Which ones? Well, we've been observing all reads, so we can record which pages were referenced once, and which ones were referenced multiple times so we can do arbitrarily complex page aging in there.

- On close(), nuke all pages which weren't in core during open(), even if this app referenced them multiple times.

- If the backup program decided to read its input files with mmap we're rather screwed. We can't intercept pagefaults so the best we can do is to restore the file's pagecache to its previous state on close().

Or if it's really a problem, get control in there somehow and periodically poll the pagecache occupancy via mincore(), use madvise() then fadvise() to trim it back.

Quoted 7 lines
Quoted 4 lines

That all sounds reasonably doable. It'd be pretty complex to do it in-kernel but we could do it there too. Problem is if course that the above strategy is explicitly optimised for the backup program and if it's in-kernel it becomes applicable to all other workloads.

I can see the above being possible, but I can't see the reason for exposing that complexity to userspace.

That's sophistication, not complexity. It doesn't have to do all that stuff to be effective.

Quoted 6 lines

If I'm the target
audience for that API then it's broken as I'd mess it up, or would take too long to get it right.

Can't we just fix the posix_fadvise() implementation to only evict pages paged in by the current process.

The kernel doesn't have that information.

Quoted 1 line

Perhaps one could possibly just evict pages with _mapcount==0 ?

That is the present fadvise(FADV_DONTNEED) behaviour.

#14 lore raw
Quoted 4 lines · Andrew Morton
Quoted 1 line · On Tue, 06 Mar 2007 12:10:49 +0000 P__draig Brady <[email protected]> wrote:

Andrew Morton wrote:

Quoted 8 lines
Quoted 6 lines

If I'm the target
audience for that API then it's broken as I'd mess it up, or would take too long to get it right.

Can't we just fix the posix_fadvise() implementation to only evict pages paged in by the current process.

The kernel doesn't have that information.

It doesn't _keep_ the information. File readahead is done in the process context, so we had it originally.

I agree though that we probably should not bother trying to keep that kind of information :)

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#15 lore raw
In reply to Andrew Morton
Quoted 5 lines · Andrew Morton
Quoted 1 line · On Tue, 06 Mar 2007 12:10:49 +0000 P__draig Brady <[email protected]> wrote:

Perhaps one could possibly just evict pages with _mapcount==0 ?

That is the present fadvise(FADV_DONTNEED) behaviour.

Ah right. It doesn't invalidate page_mapped() pages. If that means it doesn't invalidate pages previously cached by other processes, then great.

However I think what I meant though was fadvise(FADV_DONTNEED) should only invalidate pages where page_count()<=1

Quoted 1 line

From include/linux/mm.h

" For pages belonging to inodes, the page_count() is the number of
attaches, plus 1 if `private' contains something, plus one for the page cache itself."

cheers,
Pádraig.

#16 lore raw
Quoted 8 lines · On Wed, 07 Mar 2007 11:39:02 +0000 Pádraig Brady <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 1 line · On Tue, 06 Mar 2007 12:10:49 +0000 P__draig Brady <[email protected]> wrote:

Perhaps one could possibly just evict pages with _mapcount==0 ?

That is the present fadvise(FADV_DONTNEED) behaviour.

Ah right. It doesn't invalidate page_mapped() pages.

yup

Quoted 2 lines

If that means it doesn't invalidate pages previously cached by other processes, then great.

It will do that. This is why I point out that this userspace tool could (easily) be enhanced to not invalidate pages which were in pagecache prior to their being read by the managed application.

#17 lore raw
In reply to Andrew Morton
Quoted 59 lines · Andrew Morton
Quoted 22 lines · On Sun, 4 Mar 2007 00:01:55 +0100 bert hubert <[email protected]> wrote:
Quoted 9 lines · Andrew Morton
Quoted 6 lines
Quoted 2 lines

It is *not* a global instruction. It uses setenv, so the user's policy affects only the target process and its forked children.

... and all other processes accessing the same file(s)!

Your library and the system calls may be limited to one process, but the consequences are global.

Yes. So what? If the user wants to go and evict libc.so from pagecache then he can do so - the kernel has provided syscalls with which this can be done for at least seven years. Bad user, shouldn't do that.

While I agree with your sentiments that userspace can have a good idea on how to deal with the page cache, your program does more than it claims to do - because of how linux implements posix_fadvise.

I don't think anybody expects or desires your program to actually *evict* the stuff from the cache you are trying access, which happens in case the data was in the cache prior to starting your program.

What people expect is that a solution such as you wrote it simply won't *add* anything to the cache. They don't expect it will actually globally *remove* stuff from the cache.

Quoted 5 lines

Making a backup this way would hurt even worse than usual with your pagecache management tool if the file being backupped was still being read.

This is not your fault, but in practice, it makes your program less useful than it could be.

yup. As I said, it's a proof-of-concept. It's a project. And I have about one free femtosecond per fortnight :(

Quoted 2 lines

One could conceivably fix that up using mincore and simply not fadvise if a page was in core already.

Yes. Let's flesh it out the backup program policy some more:

- Unconditionally invalidate output files

- on entry to read(), probe pagecache, record which pages in the range are present

- on entry to next read(), shoot down those pages from the previous read which weren't in pagecache.

- But we can do better! LRU the page's files up to a certain number of pages.

- Once that point is exceeded, we need to reclaim some pages. Which ones? Well, we've been observing all reads, so we can record which pages were referenced once, and which ones were referenced multiple times so we can do arbitrarily complex page aging in there.

- On close(), nuke all pages which weren't in core during open(), even if this app referenced them multiple times.

- If the backup program decided to read its input files with mmap we're rather screwed. We can't intercept pagefaults so the best we can do is to restore the file's pagecache to its previous state on close().

Quoted 8 lines

Or if it's really a problem, get control in there somehow and periodically poll the pagecache occupancy via mincore(), use madvise() then fadvise() to trim it back.

That all sounds reasonably doable. It'd be pretty complex to do it in-kernel but we could do it there too. Problem is if course that the above strategy is explicitly optimised for the backup program and if it's in-kernel it becomes applicable to all other workloads.

This strategy looks very good. However we are not considering the performance impact on the 'backup' application as such. By removing pagecache pages brought in by the application without the knowledge of the applications usage and behavior may severely affect its performance.

Certainly we are interested in improving system performance at the cost certain applications, but not to an extend that the backup process will drag on and on to an unreasonable amount of time.

Also backup processes may consist of a group of applications working on the same stream of data. Like compression program, encryption program etc which could be independent applications.

We should consider having a limit on pagecache usage rather than denying any space in the pagecache for these applications.

Can fadvice() be enhanced to have a limit on pagecache usage and reclaim used pages in LRU order? This way data stays for a little while for other applications to pickup from pagecache.

Pages already in memory or brought in by other applications need not be placed in this list and hence we prevent any collateral pageouts.

#18 lore raw
Quoted 11 lines · On Thu, 08 Mar 2007 13:29:02 +0530 Vaidyanathan Srinivasan…
Quoted 4 lines

That all sounds reasonably doable. It'd be pretty complex to do it in-kernel but we could do it there too. Problem is if course that the above strategy is explicitly optimised for the backup program and if it's in-kernel it becomes applicable to all other workloads.

This strategy looks very good. However we are not considering the performance impact on the 'backup' application as such. By removing pagecache pages brought in by the application without the knowledge of the applications usage and behavior may severely affect its performance.

Quoted 7 lines

Certainly we are interested in improving system performance at the cost certain applications, but not to an extend that the backup process will drag on and on to an unreasonable amount of time.

Also backup processes may consist of a group of applications working on the same stream of data. Like compression program, encryption program etc which could be independent applications.

Well yes, if the application is that funky then suitably funky userspace tricks will be needed to avoid hurting it.

Quoted 2 lines

We should consider having a limit on pagecache usage rather than denying any space in the pagecache for these applications.

That's what containerisation is for:

	run-in-container --memory=16M /bin/backup-program

This can be done today with x86_64 fake-numa, controlled by cpusets. One day, when we get our containerisation story sorted out, things will be more convenient...

Quoted 6 lines

Can fadvice() be enhanced to have a limit on pagecache usage and reclaim used pages in LRU order? This way data stays for a little while for other applications to pickup from pagecache.

Pages already in memory or brought in by other applications need not be placed in this list and hence we prevent any collateral pageouts.

We could teach the presently-unimplemented POSIX_FADV_NOREUSE to dump this file's pages at the tail of the inactive list (after cleaning them if needed). That way, they're the first to get reclaimed.

The standard says "Specifies that the application expects to access the specified data once and then not reuse it thereafter." That's a bit
ambiguous: it it before the process accessed the data, or after? Before, I suspect.

#19 lore raw
In reply to Andrew Morton
Quoted 5 lines · Andrew Morton
Quoted 3 lines · On Sat, 03 Mar 2007 17:19:00 -0500 Rik van Riel <[email protected]> wrote:
Quoted 2 lines

It is *not* a global instruction. It uses setenv, so the user's policy affects only the target process and its forked children.

Quoted 8 lines
Quoted 4 lines

... and all other processes accessing the same file(s)!

Your library and the system calls may be limited to one process, but the consequences are global.

Yes. So what? If the user wants to go and evict libc.so from pagecache then he can do so - the kernel has provided syscalls with which this can be done for at least seven years. Bad user, shouldn't do that.

I think what Rik is pointing out is that as currently implemented, posix_fadvise is a much bigger hammer than is generally useful or desirable.

Using posix_fadvise on the other hand says "immediately drop this stuff from the pagecache, consequences be damned". If someone else happens to be using the specified data, well too bad, they suffer collateral damage. Process A can, maliciously or ignorantly, deny service to process B.

On the other hand, your old but super cool O_STREAMING patch took a kinder gentler approach, where applications could tell the kernel "please do not keep this file descriptor's data in cache on my account since I will not reuse it." If someone else however was using the same data, the kernel would keep things cached as usual and thereby avoid doing collateral damage.

-Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

#20 lore raw
In reply to Rik van Riel
Quoted 12 lines · On Sat, 03 Mar 2007 16:30:56 -0500 Rik van Riel <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 1 line · On Sat, 03 Mar 2007 15:40:42 -0500 Rik van Riel <[email protected]> wrote:

I am sick and tired of the "this is hard, let userspace do it" attitude.

Anything you try to do in-kernel will catastrophically screw up some workloads. You don't have a chance of getting this right.

Any time you follow the directions of one userspace program, you can screw up others. I suspect that userspace has far less of a chance of getting it right than the kernel.

Quoted 5 lines

ALSA would be a good example of why it is bad to export tuning knobs directly to userspace - many sound cards have non-standard names for the volume controls, making it almost impossible for userspace to present the user with a simple user interface for tweaking the volume.

What on earth are you talking about? Please, go and look at the thing.

Quoted 4 lines
Quoted 3 lines

You are the kernel. The user just read an entire kernel tree. You face a binary decision: do you cache that tree or do you not? Your time starts now. What is your answer?

Quoted 8 lines

Lets turn this around.

The user has been accessing the kernel tree over and over again, for hours on end (compile testing a patch). Along comes a backup program, that tells you to evict the whole thing from the cache.

What do you do?

Well, backup programs are a unique case. Let's say instead that the user has just generated a 600MB ISO image.

The kernel *just doesn't know* whether the user will next try to read the kernel tree or will next try to read that ISO image.

That, Rik, is my point, and is the entire point of this work.

Quoted 2 lines

How can you make global policy decisions based on the intent of one program?

You can't, that's why I did this work.

Quoted 2 lines

Only the kernel knows the state of the whole system and has observed the behaviour of all the processes.

The kernel knows the past, and tries to predict the future from that past. Sometimes, as you well know, that goes badly wrong. That's why I did this work.

Quoted 2 lines

One process has
no idea what the other processes in the system are doing.

argh. Please, next time click on the link?

http://userweb.kernel.org/~akpm/pagecache-management/

#21 lore raw
Quoted 7 lines · Andrew Morton

Well, backup programs are a unique case. Let's say instead that the user has just generated a 600MB ISO image.

The kernel *just doesn't know* whether the user will next try to read the kernel tree or will next try to read that ISO image.

That, Rik, is my point, and is the entire point of this work.

I still don't understand why "the backup program flushed my data out of the cache with POSIX_FADV_DONTNEED" is an improvement over "the backup program flushed my data out of the cache by reading other files".

Your code may be useful for a few specialized situations, but I don't see it actually fixing most of the examples you gave in your announcement, except for the DVD copying one.

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#22 lore raw
Quoted 5 lines · On Sat, 03 Mar 2007 17:25:30 -0500 Rik van Riel <[email protected]> wrote:
Quoted 3 lines · Andrew Morton

Well, backup programs are a unique case. Let's say instead that the user has just generated a 600MB ISO image.

Quoted 8 lines
Quoted 4 lines

The kernel *just doesn't know* whether the user will next try to read the kernel tree or will next try to read that ISO image.

That, Rik, is my point, and is the entire point of this work.

I still don't understand why "the backup program flushed my data out of the cache with POSIX_FADV_DONTNEED" is an improvement over "the backup program flushed my data out of the cache by reading other files".

Oh. Well, yes, if the user elected to instruct the backup program to invalidate both its input files and its output files and if it's a full dump, you end up with nothing in pagecache. Possibly a more sensible setting would be to invalidate only the output.

But having some batch program come in from the side and perform a bulk read of your present working set isn't very common.

Quoted 1 line

Your code may be useful for a few specialized situations,

That's quite wrong. It is useful for a great number of well-known problem scenarios, all of which are *already "specialized situations". It's *you* who is chasing down the 1% scenarios and portraying them as general problems. Backups only happen once in 24 hours, for example.

Quoted 4 lines

but I don't
see it actually fixing most of the examples you gave in your announcement, except for the DVD copying one.

I don't know how much benefit it will provide for the updatedb problem - I expect it'll help sometimes. otoh maybe it'll worsen the existing slab internal fragmentation problem, dunno.

But the other scenarios it solves completely and optimally.

#23 lore raw
In reply to Rik van Riel
Quoted 1 line · On Sat, 03 Mar 2007 17:25:30 -0500 Rik van Riel <[email protected]> wrote:

backup program

A suitable policy for a backup program would probably be to invalidate any output file(s) and to invalidate those pages of the input files which were not in cache when the backup program first opened those files. That way the backup program will have no effect on the cache state, except for the race situation where someone read an uncached file while the backup program was reading from it too.

This can be added in an hour or two with no kernel changes (use mincore).

#24 lore raw
Quoted 10 lines · Andrew Morton
Quoted 1 line · On Sat, 03 Mar 2007 17:25:30 -0500 Rik van Riel <[email protected]> wrote:

backup program

A suitable policy for a backup program would probably be to invalidate any output file(s) and to invalidate those pages of the input files which were not in cache when the backup program first opened those files. That way the backup program will have no effect on the cache state, except for the race situation where someone read an uncached file while the backup program was reading from it too.

The use-once policy we have in the kernel should work perfectly fine for backups. All we need to do is actually honor the accessed bit on active page cache pages, instead of flushing them onto the inactive list.

What am I overlooking?

Quoted 1 line

This can be added in an hour or two with no kernel changes (use mincore).

mincore only works for mmaped areas, we'd need an fincore to work with file handles.

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#25 lore raw
Quoted 12 lines · On Sat, 03 Mar 2007 19:01:01 -0500 Rik van Riel <[email protected]> wrote:
Quoted 10 lines · Andrew Morton
Quoted 1 line · On Sat, 03 Mar 2007 17:25:30 -0500 Rik van Riel <[email protected]> wrote:

backup program

A suitable policy for a backup program would probably be to invalidate any output file(s) and to invalidate those pages of the input files which were not in cache when the backup program first opened those files. That way the backup program will have no effect on the cache state, except for the race situation where someone read an uncached file while the backup program was reading from it too.

Quoted 7 lines

The use-once policy we have in the kernel should work perfectly fine for backups. All we need to do is actually honor the accessed bit on active page cache pages, instead of flushing them onto the inactive list.

What am I overlooking?

That'll improve backups but will break other things.

To do this effectively we'd need to change the policy so that new pagecache allocations cause no scanning of used-twice pages at all. So that even after many gigs of backing up, the working set is still there.

Problem is, (for example) what about the person who has 80% of memory in used-twice state and who then reads a file or files which are 20% or more of the size of memory, two or more times. It'll be 100% cache misses, every time. This will happen quite a lot. IOW, once those pages are in used-twice state, how does further pagecache activity ever get them _out_ of that state? Only by joining the used-twice page set, and that can't happen if the used-once-so-far pages got reclaimed.

Doing a refault thing would help a bit, but stops working at a certain point.

Quoted 4 lines
Quoted 1 line

This can be added in an hour or two with no kernel changes (use mincore).

mincore only works for mmaped areas, we'd need an fincore to work with file handles.

The LD_PRELOAD code has the fd and can mmap it to perform the pagecache probe.

fincore() would be a bit neater, but given the rarity with which mincore() is used it's perhaps hard to justify adding a slightly more efficient and slightly more convenient subset of mincore().

#26 lore raw
Quoted 24 lines · Andrew Morton
Quoted 7 lines · On Sat, 03 Mar 2007 19:01:01 -0500 Rik van Riel <[email protected]> wrote:

The use-once policy we have in the kernel should work perfectly fine for backups. All we need to do is actually honor the accessed bit on active page cache pages, instead of flushing them onto the inactive list.

What am I overlooking?

That'll improve backups but will break other things.

To do this effectively we'd need to change the policy so that new pagecache allocations cause no scanning of used-twice pages at all. So that even after many gigs of backing up, the working set is still there.

Problem is, (for example) what about the person who has 80% of memory in used-twice state and who then reads a file or files which are 20% or more of the size of memory, two or more times. It'll be 100% cache misses, every time. This will happen quite a lot. IOW, once those pages are in used-twice state, how does further pagecache activity ever get them _out_ of that state? Only by joining the used-twice page set, and that can't happen if the used-once-so-far pages got reclaimed.

Quoted 1 line

Doing a refault thing would help a bit, but stops working at a certain point.

At what point does it stop working?

I am not asking this to be difficult, I just want to get Linux a VM that does not need to be kludged up every time a distro ships it to its customers.

I believe one starting point would be a concept that people cannot shoot holes in any more. That is no guarantee, but as long as the concept has known holes coding it up is likely to be a waste of time since the code will need kludges to deal with the problems later on and we'd be back to square one.

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#27 lore raw
Quoted 25 lines · On Sat, 03 Mar 2007 20:23:07 -0500 Rik van Riel <[email protected]> wrote:
Quoted 24 lines · Andrew Morton
Quoted 7 lines · On Sat, 03 Mar 2007 19:01:01 -0500 Rik van Riel <[email protected]> wrote:

The use-once policy we have in the kernel should work perfectly fine for backups. All we need to do is actually honor the accessed bit on active page cache pages, instead of flushing them onto the inactive list.

What am I overlooking?

That'll improve backups but will break other things.

To do this effectively we'd need to change the policy so that new pagecache allocations cause no scanning of used-twice pages at all. So that even after many gigs of backing up, the working set is still there.

Problem is, (for example) what about the person who has 80% of memory in used-twice state and who then reads a file or files which are 20% or more of the size of memory, two or more times. It'll be 100% cache misses, every time. This will happen quite a lot. IOW, once those pages are in used-twice state, how does further pagecache activity ever get them _out_ of that state? Only by joining the used-twice page set, and that can't happen if the used-once-so-far pages got reclaimed.

Quoted 3 lines
Quoted 1 line

Doing a refault thing would help a bit, but stops working at a certain point.

At what point does it stop working?

We need to store that this-page-got-reclaimed info somewhere. I don't know how space-efficient that is. Did anyone ever do an implementation?

Of course, the pages need to be re-read again so there's a potential 100% hit there, which is in fact not a huge amount in this context. Depends how often it occurs (all the time when refault is being useful?) versus what we gain from it.

Quoted 3 lines

I am not asking this to be difficult, I just want to get Linux a VM that does not need to be kludged up every time a distro ships it to its customers.

We have a communication problem here. Please please please work harder to get these problems communicated to the MM developers. The only vendor MM kludge of which I'm aware is a thing which Andrea is working on to address a large-shm-segment versus bulk-IO problem (yup, database).

If you have enough of an understanding of a problem to be able to develop and productise a fix then share that info madly, asap.

otoh, rhel-on-the-desktop-or-smaller probably isn't a huge priority, which can be taken advantage of.

Quoted 6 lines

I believe one starting point would be a concept that people cannot shoot holes in any more. That is no guarantee, but as long as the concept has known holes coding it up is likely to be a waste of time since the code will need kludges to deal with the problems later on and we'd be back to square one.

You mean design it and review the design before coding it? You'll find few objections there.

#28 lore raw
Quoted 5 lines · Andrew Morton
Quoted 2 lines
Quoted 1 line

Doing a refault thing would help a bit, but stops working at a certain point.

At what point does it stop working?

We need to store that this-page-got-reclaimed info somewhere. I don't know how space-efficient that is. Did anyone ever do an implementation?

One 32 bit word per evicted page that we keep track of.

Quoted 4 lines

Of course, the pages need to be re-read again so there's a potential 100% hit there, which is in fact not a huge amount in this context. Depends how often it occurs (all the time when refault is being useful?) versus what we gain from it.

At this point, when we see that a refaulted page is more active than the coldest page on the active list, we can also immediately shrink the active list. That gives the next inactive page a better chance to get promoted before it gets evicted.

Quoted 11 lines
Quoted 3 lines

I am not asking this to be difficult, I just want to get Linux a VM that does not need to be kludged up every time a distro ships it to its customers.

We have a communication problem here. Please please please work harder to get these problems communicated to the MM developers. The only vendor MM kludge of which I'm aware is a thing which Andrea is working on to address a large-shm-segment versus bulk-IO problem (yup, database).

If you have enough of an understanding of a problem to be able to develop and productise a fix then share that info madly, asap.

The problem is that most of the distro patches are kludges, which we would rather not see again in future kernels. They tend to work around the problem, instead of being a proper fix, since reorganizing the VM in the middle of a release is not an option.

However, incremental small-to-medium changes might be an option for the upstream kernel, if you are interested.

Quoted 9 lines
Quoted 6 lines

I believe one starting point would be a concept that people cannot shoot holes in any more. That is no guarantee, but as long as the concept has known holes coding it up is likely to be a waste of time since the code will need kludges to deal with the problems later on and we'd be back to square one.

You mean design it and review the design before coding it? You'll find few objections there.

Few objections, but sadly also very few people interested in actually reviewing the design :(

If you can find holes in http://linux-mm.org/PageReplacementDesign
please let me know :)

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#29 lore raw
Quoted 9 lines · On Sat, 03 Mar 2007 20:56:27 -0500 Rik van Riel <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 2 lines
Quoted 1 line

Doing a refault thing would help a bit, but stops working at a certain point.

At what point does it stop working?

We need to store that this-page-got-reclaimed info somewhere. I don't know how space-efficient that is. Did anyone ever do an implementation?

One 32 bit word per evicted page that we keep track of.

ok...

I wonder if we really need a new data structure to track that. I mean, once a file-backed (or indeed swapcache) page has been reclaimed, its radix-tree slot is just sitting there with zeroes in it, asking us to reuse that space for something interesting, no?

Of course, if all 64 pages in a radix-tree node get removed, we'll currently free the node itself. We could stop doing that, but the effects of that might be pretty bad sometimes. Instead, it sounds sensible to populate the now-null slot in the parent radix-tree node with an average/max/min/per-child-bitmap/whatever of the metrics for the 64 non-resident pages which that non-leaf slot represents. So as the period since a single page got evicted increases and increases, our information about its state becomes less and less accurate.

If that inaccuracy is a problem then perhaps we could defer the collapsing of a now-empty node into its parent in some manner.

Quoted 8 lines
Quoted 2 lines

You mean design it and review the design before coding it? You'll find few objections there.

Few objections, but sadly also very few people interested in actually reviewing the design :(

If you can find holes in http://linux-mm.org/PageReplacementDesign
please let me know :)

That all looks pretty non-crazy and implementable to me. Alas, getting the stuff written and working is 1% of the effort. The rest is the nasty hunt for new corner-cases and general productisation hassle. But if initial results show benefit, I expect we could manage all that.

#30 lore raw
Quoted 28 lines · Andrew Morton
Quoted 9 lines · On Sat, 03 Mar 2007 20:56:27 -0500 Rik van Riel <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 2 lines
Quoted 1 line

Doing a refault thing would help a bit, but stops working at a certain point.

At what point does it stop working?

We need to store that this-page-got-reclaimed info somewhere. I don't know how space-efficient that is. Did anyone ever do an implementation?

One 32 bit word per evicted page that we keep track of.

ok...

I wonder if we really need a new data structure to track that. I mean, once a file-backed (or indeed swapcache) page has been reclaimed, its radix-tree slot is just sitting there with zeroes in it, asking us to reuse that space for something interesting, no?

Of course, if all 64 pages in a radix-tree node get removed, we'll currently free the node itself. We could stop doing that, but the effects of that might be pretty bad sometimes. Instead, it sounds sensible to populate the now-null slot in the parent radix-tree node with an average/max/min/per-child-bitmap/whatever of the metrics for the 64 non-resident pages which that non-leaf slot represents. So as the period since a single page got evicted increases and increases, our information about its state becomes less and less accurate.

Quoted 2 lines

If that inaccuracy is a problem then perhaps we could defer the collapsing of a now-empty node into its parent in some manner.

Getting the refault distance out of such a radix tree would be tricky. One solution I can think of would entail keeping a global fault count and storing the current fault count in the radix node and on refault subtract from the global count. The downside however is this global thing, perhaps we could do some smart percpu count aggregate to fix it.

The other point you mention is when to we reap these radix tree nodes, normally nonresident information gets dropped once the distance is further than our memory is big, but these nodes don´t have explicit order.

The collapsing idea is interesting, esp. if we could delay the collapse so that the avg refault distance would be in some relation to the error.

#31 lore raw
In reply to Andrew Morton
Quoted 26 lines · Andrew Morton
Quoted 7 lines · On Sat, 03 Mar 2007 20:56:27 -0500 Rik van Riel <[email protected]> wrote:
Quoted 4 lines · Andrew Morton
Quoted 2 lines
Quoted 1 line

Doing a refault thing would help a bit, but stops working at a certain point.

At what point does it stop working?

We need to store that this-page-got-reclaimed info somewhere. I don't know how space-efficient that is. Did anyone ever do an implementation?

One 32 bit word per evicted page that we keep track of.

ok...

I wonder if we really need a new data structure to track that. I mean, once a file-backed (or indeed swapcache) page has been reclaimed, its radix-tree slot is just sitting there with zeroes in it, asking us to reuse that space for something interesting, no?

Of course, if all 64 pages in a radix-tree node get removed, we'll currently free the node itself. We could stop doing that, but the effects of that might be pretty bad sometimes. Instead, it sounds sensible to populate the now-null slot in the parent radix-tree node with an average/max/min/per-child-bitmap/whatever of the metrics for the 64 non-resident pages which that non-leaf slot represents. So as the period since a single page got evicted increases and increases, our information about its state becomes less and less accurate.

Quoted 2 lines

If that inaccuracy is a problem then perhaps we could defer the collapsing of a now-empty node into its parent in some manner.

We know exactly how far to defer that collapsing, too.

We know at what rate we rotate through the active list, and the size of the active list.

We also know the rate at which we reclaim pages, and the size of the inactive list.

Combine the two, and you have an idea roughly how many page faults there are between the accesses to the coldest page on the active list.

We don't have to keep the evicted page history beyond that point, because pages that get refaulted after such a long interval have a longer inter-reference distance and should go onto the inactive list - ie. the default list for unknown pages.

Quoted 7 lines
Quoted 2 lines

If you can find holes in http://linux-mm.org/PageReplacementDesign
please let me know :)

That all looks pretty non-crazy and implementable to me. Alas, getting the stuff written and working is 1% of the effort. The rest is the nasty hunt for new corner-cases and general productisation hassle. But if initial results show benefit, I expect we could manage all that.

True, but I've looked through a few hundred VM bugzillas to validate the design against all the common corner cases, all the way from RHEL3 (which also has split anon/file lists) through today.

I'm trying to keep the known-good bits of our policy as much as possible, introducing big changes only for those corner cases that plagued multiple VMs in the past.

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#32 lore raw
In reply to Andrew Morton
Quoted 3 lines · Andrew Morton

It is to address the "waah, backups fill my memory with pagecache" and the "waah, updatedb swapped everything out" and the "waah, copying a DVD gobbled all my memory" problems.

Is the updatedb problem really due to pagecache?

Quoted 9 lines

When running

pagecache-management.sh dd if=100-mb-file of=foo
or
pagecache-management.sh cp -a /usr/src/linux-2.6.20 /usr/src/foo

the amount of pagecache in the machine is pretty much unaltered. Maybe a megabyte of additional cache in the second case, because of ext3 indirect blocks.

ray@phoenix:~/work/home/pagecache-management$ grep ext3_i
/proc/slabinfo; ./pagecache-management.sh sudo updatedb; grep ext3_i
/proc/slabinfo
ext3_inode_cache   21024  23722   1584    2    1 : tunables   24   12
 0 : slabdata  11861  11861      0
ext3_inode_cache   41332  41332   1584    2    1 : tunables   24   12
 0 : slabdata  20666  20666      0
ray@phoenix:~/work/home/pagecache-management$ echo $(( 1584 * (41332-21024) ))
32167872

Or is there a /proc/sys/vm/* knob that can be tweaked for this before/after the updatedb?

But yeah, I for one would happily submit patches to upstream authors to address this there. There's no reason code should be making the kernel guess its intention on these things.

Ray

#33 lore raw
Quoted 6 lines · On Sat, 3 Mar 2007 14:58:48 -0800 "Ray Lee" <[email protected]> wrote:
Quoted 3 lines · Andrew Morton

It is to address the "waah, backups fill my memory with pagecache" and the "waah, updatedb swapped everything out" and the "waah, copying a DVD gobbled all my memory" problems.

Is the updatedb problem really due to pagecache?

It's a combination of pagecache, slab cache and of course contention for the disk. In my experience the latter preponderates: the disk is sekeing like mad and I can't get its attention. Others report lots of swapout, which will be a combination of slab and pagecache, varying degrees of each.

Quoted 10 lines
Quoted 9 lines

When running

pagecache-management.sh dd if=100-mb-file of=foo
or
pagecache-management.sh cp -a /usr/src/linux-2.6.20 /usr/src/foo

the amount of pagecache in the machine is pretty much unaltered. Maybe a megabyte of additional cache in the second case, because of ext3 indirect blocks.

Quoted 9 lines
ray@phoenix:~/work/home/pagecache-management$ grep ext3_i
/proc/slabinfo; ./pagecache-management.sh sudo updatedb; grep ext3_i
/proc/slabinfo
ext3_inode_cache   21024  23722   1584    2    1 : tunables   24   12
  0 : slabdata  11861  11861      0
ext3_inode_cache   41332  41332   1584    2    1 : tunables   24   12
  0 : slabdata  20666  20666      0
ray@phoenix:~/work/home/pagecache-management$ echo $(( 1584 * (41332-21024) ))
32167872

If 32 MB is the whole lot then by eliminating pagecache, we just solved the problem. But perhaps you instantiated a lot more VFS cache and all you're seeing there is the leftovers.

Quoted 2 lines

Or is there a /proc/sys/vm/* knob that can be tweaked for this before/after the updatedb?

/proc/sys/vm/vfs_cache_pressure should help. I don't recall anyone reporting its effects with updatedb.

Quoted 3 lines

But yeah, I for one would happily submit patches to upstream authors to address this there. There's no reason code should be making the kernel guess its intention on these things.

I think so. We're dealing with super-special cases here and often trying to fix those in-kernel will degrade other, often more common cases.

<wonders about sys_reclaim_dentry(const char *pathname)>

#34 lore raw
Quoted 1 line · Andrew Morton

<wonders about sys_reclaim_dentry(const char *pathname)>

Would there be any other users of it than updatedb? I'm not coming up with much, but given that I'm not always clever, that doesn't mean much.

<thinks out loud...> A hypothetical on-demand file virus scanner is going to hit already cached or about-to-be-cached entries by definition. Perhaps some system audit daemon, such as tripwire. Well, that has the same access patterns as updatedb, doesn't it: a directory at a time. find, cp -a, the same.

So instead of sys_reclaim_dentry, how about extending fadvise to work on the fd returned via opendir? And extending POSIX_FADV_NOREUSE on a file fd to drop the dentry at close?

(Call me chicken; I just don't want to be the guy suggesting a new syscall for a single or few users.)

~ ~

Alternately, there have been requests for a way for userspace to get notification of all file events for indexing of data and metadata (inotify, unfortunately, doesn't scale to a full filesystem). (cf.
http://lkml.org/lkml/2006/9/30/98 .) That'd allow an updatedb daemon to keep the index up to date all the time, amortizing the cost. More usefully, it'd allow a content indexing daemon to stay up to date all the time, though inotify mostly works for those, I suppose.

(Hmm...

	ray@phoenix:~$ find ~ -type d | wc -l
	14067

...right. So it probably works fine for normal people.)

Hey, waitaminute. This should be a solved problem? SELinux must have some sort of requirement for logging file access attempts. Google, at least, implies so. Perhaps whatever it implements could be lifted into the core kernel without dragging the rest behind it.

Dunno. Who do we CC?

Ray

#35 lore raw
Quoted 4 lines · On Sat, 03 Mar 2007 17:02:31 -0800 Ray Lee <[email protected]> wrote:
Quoted 1 line · Andrew Morton

<wonders about sys_reclaim_dentry(const char *pathname)>

Would there be any other users of it than updatedb?

updatedb is the notorious one.

Alas, one can envisage sane workloads which really really really really want to cache millions of dentries and inodes. Workloads which get run more often than once-per-day-at-4AM. So if we "fix" updatedb, those people with kernel-indistinguishable workloads get real unhappy. We have to pass instructions to the kernel to resolve this.

A sys_reclaim_dentry() would also need a sys_is_dentry_there() so updatedb could restore the previous state.

Probably it'd be better to fix the well-known internal fragmentation problem we have with the VFS caches. That's fairly hard.

Quoted 11 lines

I'm not coming up
with much, but given that I'm not always clever, that doesn't mean much.

<thinks out loud...> A hypothetical on-demand file virus scanner is going to hit already cached or about-to-be-cached entries by definition. Perhaps some system audit daemon, such as tripwire. Well, that has the same access patterns as updatedb, doesn't it: a directory at a time. find, cp -a, the same.

So instead of sys_reclaim_dentry, how about extending fadvise to work on the fd returned via opendir?

That'd be pretty simple, but a) would reclaim the pagecache for the directory and not the dentry object itself and b) will only be easy to do for ext2 and minixfs, which maintain a separate pagecache per directory.

Yes we could do a "nuke all the dentries in this directory thing", but that's equivalent to sys_reclaim_dentry() in a loop.

Quoted 6 lines

And extending POSIX_FADV_NOREUSE on a file
fd to drop the dentry at close?

(Call me chicken; I just don't want to be the guy suggesting a new syscall for a single or few users.)

Quoted 6 lines

~ ~

Alternately, there have been requests for a way for userspace to get notification of all file events for indexing of data and metadata (inotify, unfortunately, doesn't scale to a full filesystem). (cf.
http://lkml.org/lkml/2006/9/30/98 .)

yes, that's a disappointment.

Quoted 9 lines

That'd allow an updatedb daemon to
keep the index up to date all the time, amortizing the cost. More usefully, it'd allow a content indexing daemon to stay up to date all the time, though inotify mostly works for those, I suppose.

(Hmm...

	ray@phoenix:~$ find ~ -type d | wc -l
	14067
Quoted 6 lines

...right. So it probably works fine for normal people.)

Hey, waitaminute. This should be a solved problem? SELinux must have some sort of requirement for logging file access attempts. Google, at least, implies so. Perhaps whatever it implements could be lifted into the core kernel without dragging the rest behind it.

Maybe the syscall auditing code can be persuaded to spit out records which can be used for this.

Quoted 1 line

Dunno. Who do we CC?

That's a problem. Nobody and everybody.

#36 lore raw
In reply to Andrew Morton
Quoted 5 lines · Andrew Morton

There is much more which could be done to make this code smarter, but I think the lesson here is that we can produce a far, far better result doing this work in userspace than we could ever hope to do with an in-kernel implementation. There are some enhancement suggestions in the documentation file.

While I think that more user space applications should use fadvise() to avoid polluting the page cache with unneeded data, I still think the kernel should be more fair in regard to page cache management.

Personally, I've experienced some sluggish performance after copying large files around. Even more when using NFS. It's difficult to file a bug report for "interactive feel", I don't know how to measure it. I just feel it's a weak aspect of the OS.

Surely it's possible to make the kernel a little bit better to protect the page cache from abuse, from simple or badly designed applications.

Why fairness is provided by the process scheduler with good results, yet it somewhat easy for a process to cause slowdowns from page cache usage.

My personal opinion is that the VM seem tuned for database types workloads. Of course, making the page cache more fair to prevent one process to use most of it will most likely slowdown database type applications.

Maybe the situation should be reversed, much like the process scheduler. Fairness by default, and the possibility to request for more system resources by asking for them with necessary privileges. Much like SCHED_FIFO policy.

- Eric

#37 lore raw
Quoted 9 lines · On Sat, 03 Mar 2007 19:14:59 -0500 Eric St-Laurent <[email protected]>…
Quoted 5 lines · Andrew Morton

There is much more which could be done to make this code smarter, but I think the lesson here is that we can produce a far, far better result doing this work in userspace than we could ever hope to do with an in-kernel implementation. There are some enhancement suggestions in the documentation file.

Quoted 8 lines

While I think that more user space applications should use fadvise() to avoid polluting the page cache with unneeded data, I still think the kernel should be more fair in regard to page cache management.

Personally, I've experienced some sluggish performance after copying large files around. Even more when using NFS. It's difficult to file a bug report for "interactive feel", I don't know how to measure it. I just feel it's a weak aspect of the OS.

yeah. It'd be worth spending some time, try to come up with some set of commands which produce an effect which you find objectionable.

Quoted 8 lines

Surely it's possible to make the kernel a little bit better to protect the page cache from abuse, from simple or badly designed applications.

Why fairness is provided by the process scheduler with good results, yet it somewhat easy for a process to cause slowdowns from page cache usage.

My personal opinion is that the VM seem tuned for database types workloads.

VM hasn't actually been tuned *for* anything much at all, really. Looking back on it, much of the tweaking in there has been to avoid really bad situations. We put much work into avoiding the 100%, 1000% or 10000% slowdowns, but not a lot of work into providing the 15% speedups.

So it may well be that the result is not particularly great at anything, but it's also not horridly bad at anything, either. Or at least, it's not supposed to be.

Quoted 3 lines

Of course, making the page cache more fair to prevent one process to use most of it will most likely slowdown database type applications.

databases actually like to manage their own cache via various means. There are some situations in which bulk IO activities can trash the databases's cache. That's the sort of thing which this tool is trying to help address.

Quoted 4 lines

Maybe the situation should be reversed, much like the process scheduler. Fairness by default, and the possibility to request for more system resources by asking for them with necessary privileges. Much like SCHED_FIFO policy.

Well. If the CPU scheduler makes a mistake, we see 5% or 15% degredations.
If VM make a mistake (or fails to read the operator's mind), we go to disk
and can suffer 1000% degredations or worse.

#38 lore raw
In reply to Eric St-Laurent
Quoted 8 lines · Eric St-Laurent

While I think that more user space applications should use fadvise() to avoid polluting the page cache with unneeded data, I still think the kernel should be more fair in regard to page cache management.

Personally, I've experienced some sluggish performance after copying large files around. Even more when using NFS. It's difficult to file a bug report for "interactive feel", I don't know how to measure it. I just feel it's a weak aspect of the OS.

Fairness and interactiveness are very hard to quantify and measure, which makes it hard to justify patches that improve this behaviour.

On the other hand, patches that improve benchmark results are easily justifyable, which makes it easy to merge those even if it comes at the expense of fairness.

I think fairness and robustness are important, but I have not figured out a way to justify such changes for upstream inclusion. Well, except perhaps by coming up with artificial test cases, but that feels like cheating :)

Quoted 4 lines

My personal opinion is that the VM seem tuned for database types workloads. Of course, making the page cache more fair to prevent one process to use most of it will most likely slowdown database type applications.

The database people disagree. For one, the accessed bit on active page gets pages gets ignored, so Linux does not properly keep the most actively used page cache pages in memory.

Secondly, the VM can waste quite a lot of time scanning over the anonymous pages that it does not even want to evict from memory. If the VM does not plan on evicting anonymous memory (or shared memory segments), why waste CPU time scanning them and randomizing their LRU order?

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#39 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

The tool uses an LD_PRELOAD hack to intercept glibc's read(), pread(), write(), pwrite(), close() and dup2() functions. pagecache control is done via posix_fadvise() and sync_file_range().

How could this have any effect on the updatedb problem? updatedb does not read() anything, it just open()s and stat()s every file on the disk.

Lee

#40 lore raw
Quoted 10 lines · On Sat, 3 Mar 2007 20:16:09 -0500 "Lee Revell" <[email protected]> wrote:
Quoted 4 lines · Andrew Morton

The tool uses an LD_PRELOAD hack to intercept glibc's read(), pread(), write(), pwrite(), close() and dup2() functions. pagecache control is done via posix_fadvise() and sync_file_range().

How could this have any effect on the updatedb problem? updatedb does not read() anything, it just open()s and stat()s every file on the disk.

err, good point. _one_ of those dang things which goes off when you've stayed up too late does a lot of pagecache IO, not sure which one. Maybe rpmq? But I'd expect that to be doing direct-io.

But yes, updatedb's pagecache usage will be mainly metadata, and this tool doesn't address metadata pagecache, although it could do so.

<does an updatedb on a modest system>

It instantiated 5MB of pagecache and 20MB of slab, took about one minute.

<runs all the other things in /etc/cron.daily>

rpm uses rather a lot of pagecache.

So yes, it looks like updatedb is a slab problem.

#41 lore raw
Quoted 3 lines · Andrew Morton

But yes, updatedb's pagecache usage will be mainly metadata, and this tool doesn't address metadata pagecache, although it could do so.

With no kernel changes? How? I can't find an equivalent API to posix_fadvise() for metadata.

Lee

#42 lore raw
Quoted 8 lines · On Sat, 3 Mar 2007 21:35:59 -0500 "Lee Revell" <[email protected]> wrote:
Quoted 3 lines · Andrew Morton

But yes, updatedb's pagecache usage will be mainly metadata, and this tool doesn't address metadata pagecache, although it could do so.

With no kernel changes? How? I can't find an equivalent API to posix_fadvise() for metadata.

We can use mincore and fadvise against /dev/sda1, too.

mincore's linear search would hurt but you could just run fadvise regularly. A lot of the blockdev pagecache is pretty useless anyway: we've already copied it much of it into dentries and inodes, and some of ext2/3/4's pagecache is already pinned by the fs.

#43 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

I've uploaded to http://userweb.kernel.org/~akpm/pagecache-management/ a little tool which permits the management of the pagecache usage of arbitrary applications. Effectively it prevents the targetted application from using any pagecache at all.

Cool, Kinda like noca?
http://kernel.umbrella.ro/vm/
Though I could easily read your code,
but couldn't immediately figure out what noca was doing.

I used posix_fadvise in an app I did recently:
http://www.pixelbeat.org/programs/dvd-vr/
There is a stream_data() func there that does:

read(src)
write(dst)
posix_fadvise(src)
posix_fadvise(dst)

for performance I found I needed to do it in that order so that any readahead done with the read(src) was not thrown away by the posix_fadvise(src). In addition to the order, one must be careful to throw away only what you've actually written.

I'm not sure your lib gives enough control over this, as you essentially do:

posix_fadvise(src)
read(src)
posix_fadvise(dst)
write(dst)

cheers,
Pádraig.

#44 lore raw
Quoted 8 lines · On Mon, 05 Mar 2007 11:02:43 +0000 Pádraig Brady <[email protected]> wrote:
Quoted 4 lines · Andrew Morton

I've uploaded to http://userweb.kernel.org/~akpm/pagecache-management/ a little tool which permits the management of the pagecache usage of arbitrary applications. Effectively it prevents the targetted application from using any pagecache at all.

Cool, Kinda like noca?
http://kernel.umbrella.ro/vm/

yup, same concept.

Quoted 18 lines

Though I could easily read your code,
but couldn't immediately figure out what noca was doing.

I used posix_fadvise in an app I did recently:
http://www.pixelbeat.org/programs/dvd-vr/
There is a stream_data() func there that does:

read(src)
write(dst)
posix_fadvise(src)
posix_fadvise(dst)

for performance I found I needed to do it in that order so that any readahead done with the read(src) was not thrown away by the posix_fadvise(src). In addition to the order, one must be careful to throw away only what you've actually written.

Quoted 7 lines

I'm not sure your lib gives enough control over this, as you essentially do:

posix_fadvise(src)
read(src)
posix_fadvise(dst)
write(dst)

That could be so - it's just a demo. But readahead should be OK - I only invalidate from start-of-file up to current-offset-minus-pagesize. So the cache at and ahead of the linear reader is undisturbed.