mmap vs fs cache

2 questions:
why is there data in the FS cache that isn't owned by (the mmap of) the
process that caused it to be paged in in the first place?
is there a tunable knob to discourage the page cache from stealing from the
process?

Quoted 5 lines · Howard Chu

2 questions:
why is there data in the FS cache that isn't owned by (the mmap of) the
process that caused it to be paged in in the first place?
is there a tunable knob to discourage the page cache from stealing from the
process?

This Unmapped page cache control http://lwn.net/Articles/436010/ sounds like it might have been helpful here. I.e., having a way to prioritize so that unmapped cache pages get reclaimed in preference to mapped pages could help. Though I still don't understand why these pages in the cache aren't mapped in the first place.

Quoted 7 lines · Howard Chu

Quoted 5 lines · Howard Chu

2 questions:
why is there data in the FS cache that isn't owned by (the mmap of) the
process that caused it to be paged in in the first place?
is there a tunable knob to discourage the page cache from stealing from the
process?

Quoted 6 lines

This Unmapped page cache control http://lwn.net/Articles/436010/ sounds like it might have been helpful here. I.e., having a way to prioritize so that unmapped cache pages get reclaimed in preference to mapped pages could help. Though I still don't understand why these pages in the cache aren't mapped in the first place.

As implied by this post
http://lkml.indiana.edu/hypermail/linux/kernel/0701.3/0354.html setting swappiness to 0 seems to give the desired effect of preventing mapped pages from being reclaimed.

I'm testing our memory-mapped database code on a small VM. The machine has 32GB of RAM and the size of the DB on disk is ~44GB. The database library mmaps the entire file as a single region and starts accessing it as a tree of B+trees. Running on an Ubuntu 3.5.0-23 kernel, XFS on a local disk.

If I start running read-only queries against the DB with a freshly started server, I see that my process (OpenLDAP slapd) quickly grows to an RSS of about 16GB in tandem with the FS cache. (I.e., "top" shows 16GB cached, and slapd is 16GB.)
If I confine my queries to the first 20% of the data then it all fits in RAM and queries are nice and fast.

if I extend the query range to cover more of the data, approaching the size of physical RAM, I see something strange - the FS cache keeps growing, but the slapd process size grows at a slower rate. This is rather puzzling to me since the only thing triggering reads is accesses through the mmap region. Eventually the FS cache grows to basically all of the 32GB of RAM (+/- some text/data space...) but the slapd process only reaches 25GB, at which point it actually starts to shrink - apparently the FS cache is now stealing pages from it. I find that a bit puzzling; if the pages are present in memory, and the only reason they were paged in was to satisfy an mmap reference, why aren't they simply assigned to the slapd process?

The current behavior gets even more aggravating: I can run a test that spans exactly 30GB of the data. One would expect that the slapd process should simply grow to 30GB in size, and then remain static for the remainder of the test. Instead, the server grows to 25GB, the FS cache grows to 32GB, and starts stealing pages from the server, shrinking it back down to 19GB or so.

If I do an "echo 1 > /proc/sys/vm/drop_caches" at the onset of this condition, the FS cache shrinks back to 25GB, matching the slapd process size.
This then frees up enough RAM for slapd to grow further. If I don't do this, the test is constantly paging in data from disk. Even so, the FS cache continues to grow faster than the slapd process size, so the system may run out of free RAM again, and I have to drop caches multiple times before slapd finally grows to the full 30GB. Once it gets to that size the test runs entirely from RAM with zero I/Os, but it doesn't get there without a lot of babysitting.

2 questions:
why is there data in the FS cache that isn't owned by (the mmap
of) the process that caused it to be paged in in the first place?
is there a tunable knob to discourage the page cache from stealing
from the process?

Signature · 9 lines

  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Added mm list to CC.

Quoted 47 lines · Howard Chu

I'm testing our memory-mapped database code on a small VM. The machine has 32GB of RAM and the size of the DB on disk is ~44GB. The database library mmaps the entire file as a single region and starts accessing it as a tree of B+trees. Running on an Ubuntu 3.5.0-23 kernel, XFS on a local disk.

If I start running read-only queries against the DB with a freshly started server, I see that my process (OpenLDAP slapd) quickly grows to an RSS of about 16GB in tandem with the FS cache. (I.e., "top" shows 16GB cached, and slapd is 16GB.)
If I confine my queries to the first 20% of the data then it all fits in RAM and queries are nice and fast.

if I extend the query range to cover more of the data, approaching the size of physical RAM, I see something strange - the FS cache keeps growing, but the slapd process size grows at a slower rate. This is rather puzzling to me since the only thing triggering reads is accesses through the mmap region. Eventually the FS cache grows to basically all of the 32GB of RAM (+/- some text/data space...) but the slapd process only reaches 25GB, at which point it actually starts to shrink - apparently the FS cache is now stealing pages from it. I find that a bit puzzling; if the pages are present in memory, and the only reason they were paged in was to satisfy an mmap reference, why aren't they simply assigned to the slapd process?

The current behavior gets even more aggravating: I can run a test that spans exactly 30GB of the data. One would expect that the slapd process should simply grow to 30GB in size, and then remain static for the remainder of the test. Instead, the server grows to 25GB, the FS cache grows to 32GB, and starts stealing pages from the server, shrinking it back down to 19GB or so.

If I do an "echo 1 > /proc/sys/vm/drop_caches" at the onset of this condition, the FS cache shrinks back to 25GB, matching the slapd process size.
This then frees up enough RAM for slapd to grow further. If I don't do this, the test is constantly paging in data from disk. Even so, the FS cache continues to grow faster than the slapd process size, so the system may run out of free RAM again, and I have to drop caches multiple times before slapd finally grows to the full 30GB. Once it gets to that size the test runs entirely from RAM with zero I/Os, but it doesn't get there without a lot of babysitting.

2 questions:
why is there data in the FS cache that isn't owned by (the mmap
of) the process that caused it to be paged in in the first place?

Quoted 2 lines

is there a tunable knob to discourage the page cache from stealing
from the process?

Quoted 3 lines · Jan Kara

Quoted 3 lines

2 questions:
why is there data in the FS cache that isn't owned by (the mmap
of) the process that caused it to be paged in in the first place?

The filesystem cache is shared among processes because the filesystem is also shared among processes. If another task were to access the same file, we still should only have one copy of that data in memory.

It sounds to me like slapd is itself caching all the data it reads.

If that is true, shouldn't it really be using direct IO to prevent this double buffering of filesystem data in memory?

Quoted 2 lines

Quoted 2 lines

is there a tunable knob to discourage the page cache from stealing
from the process?

Try reducing /proc/sys/vm/swappiness, which ranges from 0-100 and defaults to 60.

You're misreading the information then. slapd is doing no caching of its own, its RSS and SHR memory size are both the same. All it is using is the mmap, nothing else. The RSS == SHR == FS cache, up to 16GB. RSS is always == SHR, but above 16GB they grow more slowly than the FS cache.

Quoted 5 lines · Howard Chu

You're misreading the information then. slapd is doing no caching of its own, its RSS and SHR memory size are both the same. All it is using is the mmap, nothing else. The RSS == SHR == FS cache, up to 16GB. RSS is always == SHR, but above 16GB they grow more slowly than the FS cache.

It only means, that some pages got unmapped from your process. It can happned, for instance, due page migration. There's nothing worry about: it will be mapped back on next page fault to the page and it's only minor page fault since the page is in pagecache anyway.

There is no way that a process that is accessing only 30GB of a mmap should be able to fill up 32GB of RAM. There's nothing else running on the machine, I've killed or suspended everything else in userland besides a couple shells running top and vmstat. When I manually drop_caches repeatedly, then eventually slapd RSS/SHR grows to 30GB and the physical I/O stops.

Quoted 6 lines · Howard Chu

There is no way that a process that is accessing only 30GB of a mmap should be able to fill up 32GB of RAM. There's nothing else running on the machine, I've killed or suspended everything else in userland besides a couple shells running top and vmstat. When I manually drop_caches repeatedly, then eventually slapd RSS/SHR grows to 30GB and the physical I/O stops.

Is it possible that the kernel is doing some sort of automatic readahead, but it ends up reading pages corresponding to data that isn't ever queried and so doesn't get mapped by the application?

First obvious conclusion - kswapd is being too aggressive. When free memory hits the low watermark, the reclaim shrinks slapd down from 25GB to 18-19GB, while the page cache still contains ~7GB of unmapped pages. Ideally I'd like a tuning knob so I can say to keep no more than 2GB of unmapped pages in the cache. (And the desired effect of that would be to allow user processes to grow to 30GB total, in this case.)

I mentioned this "unmapped page cache control" post already
http://lwn.net/Articles/436010/ but it seems that the idea was ultimately rejected. Is there anything else similar in current kernels?

Quoted 9 lines · Chris Friesen

Quoted 6 lines · Howard Chu

There is no way that a process that is accessing only 30GB of a mmap should be able to fill up 32GB of RAM. There's nothing else running on the machine, I've killed or suspended everything else in userland besides a couple shells running top and vmstat. When I manually drop_caches repeatedly, then eventually slapd RSS/SHR grows to 30GB and the physical I/O stops.

Quoted 3 lines

Is it possible that the kernel is doing some sort of automatic readahead, but it ends up reading pages corresponding to data that isn't ever queried and so doesn't get mapped by the application?

Yes, that's what I was thinking. I added a posix_madvise(..POSIX_MADV_RANDOM) but that had no effect on the test.

First obvious conclusion - kswapd is being too aggressive. When free memory hits the low watermark, the reclaim shrinks slapd down from 25GB to 18-19GB, while the page cache still contains ~7GB of unmapped pages. Ideally I'd like a tuning knob so I can say to keep no more than 2GB of unmapped pages in the cache. (And the desired effect of that would be to allow user processes to grow to 30GB total, in this case.)

Quoted 18 lines · Howard Chu

Quoted 9 lines · Chris Friesen

Quoted 6 lines · Howard Chu

There is no way that a process that is accessing only 30GB of a mmap should be able to fill up 32GB of RAM. There's nothing else running on the machine, I've killed or suspended everything else in userland besides a couple shells running top and vmstat. When I manually drop_caches repeatedly, then eventually slapd RSS/SHR grows to 30GB and the physical I/O stops.

Quoted 3 lines

Is it possible that the kernel is doing some sort of automatic readahead, but it ends up reading pages corresponding to data that isn't ever queried and so doesn't get mapped by the application?

Yes, that's what I was thinking. I added a posix_madvise(..POSIX_MADV_RANDOM) but that had no effect on the test.

Quoted 7 lines

First obvious conclusion - kswapd is being too aggressive. When free memory hits the low watermark, the reclaim shrinks slapd down from 25GB to 18-19GB, while the page cache still contains ~7GB of unmapped pages. Ideally I'd like a tuning knob so I can say to keep no more than 2GB of unmapped pages in the cache. (And the desired effect of that would be to allow user processes to grow to 30GB total, in this case.)

We should find out where the unmapped page cache is coming from if you are only accessing mapped file cache and disabled readahead.

How do you arrive at this number of unmapped page cache?

What could happen is that previously used and activated pages do not get evicted anymore since there is a constant supply of younger reclaimable cache that is actually thrashing. Whenever you drop the caches, you get rid of those stale active pages and allow the previously thrashing cache to get activated. However, that would

require that there is already a significant amount of active file pages before your workload starts (check the nr_active_file number in /proc/vmstat before launching slapd, try sync; echo 3 >drop_caches before launching to eliminate this option) OR that the set of pages accessed during your workload changes and the combined set of pages accessed by your workload is bigger than available memory -- which you claimed would not happen because you only access the 30GB file area on that system.

Quoted 27 lines · Johannes Weiner

Quoted 25 lines · Howard Chu

Quoted 9 lines · Chris Friesen

Quoted 6 lines · Howard Chu

There is no way that a process that is accessing only 30GB of a mmap should be able to fill up 32GB of RAM. There's nothing else running on the machine, I've killed or suspended everything else in userland besides a couple shells running top and vmstat. When I manually drop_caches repeatedly, then eventually slapd RSS/SHR grows to 30GB and the physical I/O stops.

Quoted 3 lines

Is it possible that the kernel is doing some sort of automatic readahead, but it ends up reading pages corresponding to data that isn't ever queried and so doesn't get mapped by the application?

Yes, that's what I was thinking. I added a posix_madvise(..POSIX_MADV_RANDOM) but that had no effect on the test.

First obvious conclusion - kswapd is being too aggressive. When free memory hits the low watermark, the reclaim shrinks slapd down from 25GB to 18-19GB, while the page cache still contains ~7GB of unmapped pages. Ideally I'd like a tuning knob so I can say to keep no more than 2GB of unmapped pages in the cache. (And the desired effect of that would be to allow user processes to grow to 30GB total, in this case.)

Quoted 4 lines

We should find out where the unmapped page cache is coming from if you are only accessing mapped file cache and disabled readahead.

How do you arrive at this number of unmapped page cache?

This number is pretty obvious. When slapd has grown to 25GB, the

page cache has grown to 32GB (less about 200MB, the minfree). So:

7GB unmapped in the cache.

Quoted 5 lines

What could happen is that previously used and activated pages do not get evicted anymore since there is a constant supply of younger reclaimable cache that is actually thrashing. Whenever you drop the caches, you get rid of those stale active pages and allow the previously thrashing cache to get activated. However, that would

Quoted 8 lines

require that there is already a significant amount of active file pages before your workload starts (check the nr_active_file number in /proc/vmstat before launching slapd, try sync; echo 3 >drop_caches before launching to eliminate this option) OR that the set of pages accessed during your workload changes and the combined set of pages accessed by your workload is bigger than available memory -- which you claimed would not happen because you only access the 30GB file area on that system.

There are no other active pages before the test begins. There's nothing else running. caches have been dropped completely at the beginning.

The test clearly is accessing only 30GB of data. Once slapd reaches this process size, the test can be stopped and restarted any number of times, run for any number of hours continuously, and memory use on the system is unchanged, and no pageins occur.

Quoted 4 lines · Howard Chu

The test clearly is accessing only 30GB of data. Once slapd reaches this process size, the test can be stopped and restarted any number of times, run for any number of hours continuously, and memory use on the system is unchanged, and no pageins occur.

Interesting. It might be worth trying what happens if you do
madvise(..., MADV_DONTNEED) on the data file instead of dropping caches with /proc/sys/vm/drop_caches. That way we can establish whether the extra cached data is in the data file (things will look the same way as with drop_caches) or somewhere else (there will be still unmapped page cache).

Quoted 16 lines · Howard Chu

Quoted 11 lines · Chris Friesen

Quoted 6 lines · Howard Chu

There is no way that a process that is accessing only 30GB of a mmap should be able to fill up 32GB of RAM. There's nothing else running on the machine, I've killed or suspended everything else in userland besides a couple shells running top and vmstat. When I manually drop_caches repeatedly, then eventually slapd RSS/SHR grows to 30GB and the physical I/O stops.

Is it possible that the kernel is doing some sort of automatic readahead, but it ends up reading pages corresponding to data that isn't ever queried and so doesn't get mapped by the application?

Yes, that's what I was thinking. I added a posix_madvise(..POSIX_MADV_RANDOM) but that had no effect on the test.

Quoted 7 lines

First obvious conclusion - kswapd is being too aggressive. When free memory hits the low watermark, the reclaim shrinks slapd down from 25GB to 18-19GB, while the page cache still contains ~7GB of unmapped pages. Ideally I'd like a tuning knob so I can say to keep no more than 2GB of unmapped pages in the cache. (And the desired effect of that would be to allow user processes to grow to 30GB total, in this case.)

We should find out where the unmapped page cache is coming from if you are only accessing mapped file cache and disabled readahead.

How do you arrive at this number of unmapped page cache?

What could happen is that previously used and activated pages do not get evicted anymore since there is a constant supply of younger

reclaimable cache that is actually thrashing. Whenever you drop the caches, you get rid of those stale active pages and allow the previously thrashing cache to get activated. However, that would require that there is already a significant amount of active file

pages before your workload starts (check the nr_active_file number in /proc/vmstat before launching slapd, try sync; echo 3 >drop_caches before launching to eliminate this option) OR that the set of pages accessed during your workload changes and the combined set of pages accessed by your workload is bigger than available memory -- which you claimed would not happen because you only access the 30GB file area on that system.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes, that's what I was thinking. I added a posix_madvise(..POSIX_MADV_RANDOM) but that had no effect on the test.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 3 lines · Howard Chu

Yes, that's what I was thinking. I added a posix_madvise(..POSIX_MADV_RANDOM) but that had no effect on the test.

Yep, that's because it isn't implemented.

Quoted 1 line

Yep, that's because it isn't implemented.

Why do you think so? AFAICS it is implemented by setting VM_RAND_READ
flag in the VMA and do_async_mmap_readahead() and do_sync_mmap_readahead() check for the flag and don't do anything if it is set...

Quoted 50 lines · Jan Kara

Added mm list to CC.

Quoted 47 lines · Howard Chu

I'm testing our memory-mapped database code on a small VM. The machine has 32GB of RAM and the size of the DB on disk is ~44GB. The database library mmaps the entire file as a single region and starts accessing it as a tree of B+trees. Running on an Ubuntu 3.5.0-23 kernel, XFS on a local disk.

If I start running read-only queries against the DB with a freshly started server, I see that my process (OpenLDAP slapd) quickly grows to an RSS of about 16GB in tandem with the FS cache. (I.e., "top" shows 16GB cached, and slapd is 16GB.)
If I confine my queries to the first 20% of the data then it all fits in RAM and queries are nice and fast.

if I extend the query range to cover more of the data, approaching the size of physical RAM, I see something strange - the FS cache keeps growing, but the slapd process size grows at a slower rate. This is rather puzzling to me since the only thing triggering reads is accesses through the mmap region. Eventually the FS cache grows to basically all of the 32GB of RAM (+/- some text/data space...) but the slapd process only reaches 25GB, at which point it actually starts to shrink - apparently the FS cache is now stealing pages from it. I find that a bit puzzling; if the pages are present in memory, and the only reason they were paged in was to satisfy an mmap reference, why aren't they simply assigned to the slapd process?

The current behavior gets even more aggravating: I can run a test that spans exactly 30GB of the data. One would expect that the slapd process should simply grow to 30GB in size, and then remain static for the remainder of the test. Instead, the server grows to 25GB, the FS cache grows to 32GB, and starts stealing pages from the server, shrinking it back down to 19GB or so.

If I do an "echo 1 > /proc/sys/vm/drop_caches" at the onset of this condition, the FS cache shrinks back to 25GB, matching the slapd process size.
This then frees up enough RAM for slapd to grow further. If I don't do this, the test is constantly paging in data from disk. Even so, the FS cache continues to grow faster than the slapd process size, so the system may run out of free RAM again, and I have to drop caches multiple times before slapd finally grows to the full 30GB. Once it gets to that size the test runs entirely from RAM with zero I/Os, but it doesn't get there without a lot of babysitting.

2 questions:
why is there data in the FS cache that isn't owned by (the mmap
of) the process that caused it to be paged in in the first place?

The filesystem cache is shared among processes because the filesystem is also shared among processes. If another task were to access the same file, we still should only have one copy of that data in memory.

It sounds to me like slapd is itself caching all the data it reads. If that is true, shouldn't it really be using direct IO to prevent this double buffering of filesystem data in memory?

Quoted 3 lines

Quoted 3 lines

is there a tunable knob to discourage the page cache from stealing
from the process?

Try reducing /proc/sys/vm/swappiness, which ranges from 0-100 and defaults to 60.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>