RFC: vmscan: add min_filelist_kbytes sysctl for protecting the working set

On ChromiumOS, we do not use swap.

When memory is low, the only way to
free memory is to reclaim pages from the file list. This results in a lot of thrashing under low memory conditions. We see the system become unresponsive for minutes before it eventually OOMs. We also see very slow browser tab switching under low memory. Instead of an unresponsive system, we'd really like the kernel to OOM as soon as it starts to thrash. If it can't keep the working set in memory, then OOM. Losing one of many tabs is a better behaviour for the user than an unresponsive system.

This patch create a new sysctl, min_filelist_kbytes, which disables reclaim of file-backed pages when when there are less than min_filelist_bytes worth of such pages in the cache. This tunable is handy for low memory systems using solid-state storage where interactive response is more important than not OOMing.

With this patch and min_filelist_kbytes set to 50000, I see very little block layer activity during low memory. The system stays responsive under low memory and browser tab switching is fast. Eventually, a process a gets killed by OOM. Without this patch, the system gets wedged for minutes before it eventually OOMs. Below is the vmstat output from my test runs.

BEFORE (notice the high bi and wa, also how long it takes to OOM):

mm/vmscan.c +6 -0

+/*
+ * Check low watermark used to prevent fscache thrashing during low memory.
+ */
+static int file_is_low(struct zone *zone, struct scan_control *sc)
+{
+	unsigned long pages_min, active, inactive;

Quoted 1 line · On Thu, 28 Oct 2010 12:15:23 -0700 Mandeep Singh Baines <[email protected]>…

On ChromiumOS, we do not use swap.

Well that's bad. Why not?

Quoted 16 lines

When memory is low, the only way to
free memory is to reclaim pages from the file list. This results in a lot of thrashing under low memory conditions. We see the system become unresponsive for minutes before it eventually OOMs. We also see very slow browser tab switching under low memory. Instead of an unresponsive system, we'd really like the kernel to OOM as soon as it starts to thrash. If it can't keep the working set in memory, then OOM. Losing one of many tabs is a better behaviour for the user than an unresponsive system.

This patch create a new sysctl, min_filelist_kbytes, which disables reclaim of file-backed pages when when there are less than min_filelist_bytes worth of such pages in the cache. This tunable is handy for low memory systems using solid-state storage where interactive response is more important than not OOMing.

Quoted 7 lines

With this patch and min_filelist_kbytes set to 50000, I see very little block layer activity during low memory. The system stays responsive under low memory and browser tab switching is fast. Eventually, a process a gets killed by OOM. Without this patch, the system gets wedged for minutes before it eventually OOMs. Below is the vmstat output from my test runs.

BEFORE (notice the high bi and wa, also how long it takes to OOM):

That's an interesting result.

Having the machine "wedged for minutes" thrashing away paging executable text is pretty bad behaviour. I wonder how to fix it. Perhaps simply declaring oom at an earlier stage.

Your patch is certainly simple enough but a bit sad. It says "the VM gets this wrong, so lets just disable it all". And thereby reduces the motivation to fix it for real.

But the patch definitely improves the situation in real-world situations and there's a case to be made that it should be available at least as an interim thing until the VM gets fixed for real. Which means that the /proc tunable might disappear again (or become a no-op) some time in the future.

Quoted 10 lines · Mandeep Singh Baines

(quoted diff) +10 -0

+/*
+ * Check low watermark used to prevent fscache thrashing during low memory.
+ */
+static int file_is_low(struct zone *zone, struct scan_control *sc)
+{
+	unsigned long pages_min, active, inactive;
+
+	if (!scanning_global_lru(sc))
+		return false;
+

Review

Quoted 6 lines

(quoted diff) +6 -0

+	pages_min = min_filelist_kbytes>>  (PAGE_SHIFT - 10);
+	active = zone_page_state(zone, NR_ACTIVE_FILE);
+	inactive = zone_page_state(zone, NR_INACTIVE_FILE);
+
+	return ((active + inactive)<  pages_min);
+}

This is problematic.

It is quite possible for a NUMA system to have one zone legitimately low on page cache (because all the binaries and libraries got paged in on another NUMA node), without the system being anywhere near out of memory.

This patch looks like it could cause a false OOM kill in that scenario.

At the very minimum, you'd have to check that the system is low on page cache globally, not just locally.

You do point out a real problem though, and it would be nice to find a generic solution to it...

Quoted 7 lines · Andrew Morton

Quoted 1 line · On Thu, 28 Oct 2010 12:15:23 -0700 Mandeep Singh Baines <[email protected]>…

On ChromiumOS, we do not use swap.

Well that's bad.  Why not?

We're using SSDs. We're still in the "make it work" phase so wanted avoid swap unless/until we learn how to use it effectively with an SSD.

You'll want to tune swap differently if you're using an SSD. Not sure if swappiness is the answer. Maybe a new tunable to control how aggressive swap is unless such a thing already exits?

Quoted 26 lines

Quoted 16 lines

When memory is low, the only way to
free memory is to reclaim pages from the file list. This results in a lot of thrashing under low memory conditions. We see the system become unresponsive for minutes before it eventually OOMs. We also see very slow browser tab switching under low memory. Instead of an unresponsive system, we'd really like the kernel to OOM as soon as it starts to thrash. If it can't keep the working set in memory, then OOM. Losing one of many tabs is a better behaviour for the user than an unresponsive system.

This patch create a new sysctl, min_filelist_kbytes, which disables reclaim of file-backed pages when when there are less than min_filelist_bytes worth of such pages in the cache. This tunable is handy for low memory systems using solid-state storage where interactive response is more important than not OOMing.

Quoted 7 lines

With this patch and min_filelist_kbytes set to 50000, I see very little block layer activity during low memory. The system stays responsive under low memory and browser tab switching is fast. Eventually, a process a gets killed by OOM. Without this patch, the system gets wedged for minutes before it eventually OOMs. Below is the vmstat output from my test runs.

BEFORE (notice the high bi and wa, also how long it takes to OOM):

That's an interesting result.

Quoted 8 lines

Having the machine "wedged for minutes" thrashing away paging executable text is pretty bad behaviour.  I wonder how to fix it. Perhaps simply declaring oom at an earlier stage.

Your patch is certainly simple enough but a bit sad.  It says "the VM gets this wrong, so lets just disable it all".  And thereby reduces the motivation to fix it for real.

Yeah, I used the RFC label because we're thinking this is just a temporary bandaid until something better comes along.

Couple of other nits I have with our patch:
* Not really sure what to do for the cgroup case. We do something reasonable for now.
* One of my colleagues also brought up the point that we might want to do something different if swap was enabled.

Quoted 5 lines

But the patch definitely improves the situation in real-world situations and there's a case to be made that it should be available at least as an interim thing until the VM gets fixed for real.  Which means that the /proc tunable might disappear again (or become a no-op) some time in the future.

I think this feature that "System response time doesn't allow but OOM allow".

Quoted 52 lines · Mandeep Singh Baines

Quoted 7 lines · Andrew Morton

Quoted 1 line · On Thu, 28 Oct 2010 12:15:23 -0700 Mandeep Singh Baines <[email protected]>…

On ChromiumOS, we do not use swap.

Well that's bad. A Why not?

We're using SSDs. We're still in the "make it work" phase so wanted avoid swap unless/until we learn how to use it effectively with an SSD.

You'll want to tune swap differently if you're using an SSD. Not sure if swappiness is the answer. Maybe a new tunable to control how aggressive swap is unless such a thing already exits?

Quoted 26 lines

Quoted 16 lines

When memory is low, the only way to
free memory is to reclaim pages from the file list. This results in a lot of thrashing under low memory conditions. We see the system become unresponsive for minutes before it eventually OOMs. We also see very slow browser tab switching under low memory. Instead of an unresponsive system, we'd really like the kernel to OOM as soon as it starts to thrash. If it can't keep the working set in memory, then OOM. Losing one of many tabs is a better behaviour for the user than an unresponsive system.

This patch create a new sysctl, min_filelist_kbytes, which disables reclaim of file-backed pages when when there are less than min_filelist_bytes worth of such pages in the cache. This tunable is handy for low memory systems using solid-state storage where interactive response is more important than not OOMing.

Quoted 7 lines

With this patch and min_filelist_kbytes set to 50000, I see very little block layer activity during low memory. The system stays responsive under low memory and browser tab switching is fast. Eventually, a process a gets killed by OOM. Without this patch, the system gets wedged for minutes before it eventually OOMs. Below is the vmstat output from my test runs.

BEFORE (notice the high bi and wa, also how long it takes to OOM):

That's an interesting result.

Quoted 8 lines

Having the machine "wedged for minutes" thrashing away paging executable text is pretty bad behaviour. A I wonder how to fix it. Perhaps simply declaring oom at an earlier stage.

Your patch is certainly simple enough but a bit sad. A It says "the VM gets this wrong, so lets just disable it all". A And thereby reduces the motivation to fix it for real.

Quoted 14 lines

Yeah, I used the RFC label because we're thinking this is just a temporary bandaid until something better comes along.

Couple of other nits I have with our patch:
* Not really sure what to do for the cgroup case. We do something A reasonable for now.
* One of my colleagues also brought up the point that we might want to do A something different if swap was enabled.

Quoted 5 lines

But the patch definitely improves the situation in real-world situations and there's a case to be made that it should be available at least as an interim thing until the VM gets fixed for real. A Which means that the /proc tunable might disappear again (or become a no-op) some time in the future.

I think this feature that "System response time doesn't allow but OOM allow". While we can control process to not killed by OOM using /oom_score_adj, we can't control response time directly.
But in mobile system, we have to control response time. One of cause to avoid swap is due to response time.

How about using memcg?
Isolate processes related to system response(ex, rendering engine, IPC engine and so no) to another group.

Quoted 74 lines · On Fri, 29 Oct 2010 08:28:23 +0900 Minchan Kim <[email protected]> wrote:

Quoted 52 lines · Mandeep Singh Baines

Quoted 7 lines · Andrew Morton

Quoted 1 line · On Thu, 28 Oct 2010 12:15:23 -0700 Mandeep Singh Baines <[email protected]>…

On ChromiumOS, we do not use swap.

Well that's bad.  Why not?

We're using SSDs. We're still in the "make it work" phase so wanted avoid swap unless/until we learn how to use it effectively with an SSD.

You'll want to tune swap differently if you're using an SSD. Not sure if swappiness is the answer. Maybe a new tunable to control how aggressive swap is unless such a thing already exits?

Quoted 26 lines

Quoted 16 lines

When memory is low, the only way to
free memory is to reclaim pages from the file list. This results in a lot of thrashing under low memory conditions. We see the system become unresponsive for minutes before it eventually OOMs. We also see very slow browser tab switching under low memory. Instead of an unresponsive system, we'd really like the kernel to OOM as soon as it starts to thrash. If it can't keep the working set in memory, then OOM. Losing one of many tabs is a better behaviour for the user than an unresponsive system.

This patch create a new sysctl, min_filelist_kbytes, which disables reclaim of file-backed pages when when there are less than min_filelist_bytes worth of such pages in the cache. This tunable is handy for low memory systems using solid-state storage where interactive response is more important than not OOMing.

Quoted 7 lines

With this patch and min_filelist_kbytes set to 50000, I see very little block layer activity during low memory. The system stays responsive under low memory and browser tab switching is fast. Eventually, a process a gets killed by OOM. Without this patch, the system gets wedged for minutes before it eventually OOMs. Below is the vmstat output from my test runs.

BEFORE (notice the high bi and wa, also how long it takes to OOM):

That's an interesting result.

Quoted 8 lines

Having the machine "wedged for minutes" thrashing away paging executable text is pretty bad behaviour.  I wonder how to fix it. Perhaps simply declaring oom at an earlier stage.

Your patch is certainly simple enough but a bit sad.  It says "the VM gets this wrong, so lets just disable it all".  And thereby reduces the motivation to fix it for real.

Quoted 14 lines

Yeah, I used the RFC label because we're thinking this is just a temporary bandaid until something better comes along.

Couple of other nits I have with our patch:
* Not really sure what to do for the cgroup case. We do something reasonable for now.
* One of my colleagues also brought up the point that we might want to do something different if swap was enabled.

Quoted 5 lines

But the patch definitely improves the situation in real-world situations and there's a case to be made that it should be available at least as an interim thing until the VM gets fixed for real.  Which means that the /proc tunable might disappear again (or become a no-op) some time in the future.

I think this feature that "System response time doesn't allow but OOM allow". While we can control process to not killed by OOM using /oom_score_adj, we can't control response time directly.
But in mobile system, we have to control response time. One of cause to avoid swap is due to response time.

Quoted 4 lines

How about using memcg?
Isolate processes related to system response(ex, rendering engine, IPC engine and so no)  to another group.

Yes, this seems interesting topic on memcg.

maybe configure cgroups as..

/system       ....... limit to X % of the system. /application  ....... limit to 100-X % of the system.

and put management software to /system. Then, the system software can check behavior of applicatoin and measure cpu time and I/O performance in /applicaiton. (And yes, it can watch memory usage.)

Here, memory cgroup has oom-notifier, you may able to do something other than oom-killer by the system. If this patch is applied to global VM, I'll check memcg can support it or not.
Hmm....checking anon/file rate in /application may be enough ?

Or, as a google guy proosed, we may have to add "file-cache-only" memcg. For example, configure system as

/system
/application-anon
/application-file-cache

(But balancing file/anon must be done by user....this is difficult.)

BTW, can we know that "recently paged out file cache comes back immediately!" score ?

On ChromiumOS, we do not use swap. When memory is low, the only way to free memory is to reclaim pages from the file list. This results in a lot of thrashing under low memory conditions. We see the system become unresponsive for minutes before it eventually OOMs. We also see very slow browser tab switching under low memory. Instead of an unresponsive system, we'd really like the kernel to OOM as soon as it starts to thrash. If it can't keep the working set in memory, then OOM. Losing one of many tabs is a better behaviour for the user than an unresponsive system.

This patch create a new sysctl, min_filelist_kbytes, which disables reclaim of file-backed pages when when there are less than min_filelist_bytes worth of such pages in the cache. This tunable is handy for low memory systems using solid-state storage where interactive response is more important than not OOMing.

With this patch and min_filelist_kbytes set to 50000, I see very little block layer activity during low memory. The system stays responsive under low memory and browser tab switching is fast. Eventually, a process a gets killed by OOM. Without this patch, the system gets wedged for minutes before it eventually OOMs. Below is the vmstat output from my test runs.

Hi

Quoted 16 lines

On ChromiumOS, we do not use swap. When memory is low, the only way to free memory is to reclaim pages from the file list. This results in a lot of thrashing under low memory conditions. We see the system become unresponsive for minutes before it eventually OOMs. We also see very slow browser tab switching under low memory. Instead of an unresponsive system, we'd really like the kernel to OOM as soon as it starts to thrash. If it can't keep the working set in memory, then OOM. Losing one of many tabs is a better behaviour for the user than an unresponsive system.

This patch create a new sysctl, min_filelist_kbytes, which disables reclaim of file-backed pages when when there are less than min_filelist_bytes worth of such pages in the cache. This tunable is handy for low memory systems using solid-state storage where interactive response is more important than not OOMing.

Quoted 5 lines

With this patch and min_filelist_kbytes set to 50000, I see very little block layer activity during low memory. The system stays responsive under low memory and browser tab switching is fast. Eventually, a process a gets killed by OOM. Without this patch, the system gets wedged for minutes before it eventually OOMs. Below is the vmstat output from my test runs.

I've heared similar requirement sometimes from embedded people. then also don't use swap. then, I don't think this is hopeless idea. but I hope to clarify some thing at first.

Yes, a system often have should-not-be-evicted-file-caches. Typically, they are libc, libX11 and some GUI libraries. Traditionally, we was making tiny application which linked above important lib and call mlockall() at startup. such technique prevent reclaim. So, Q1: Why do you think above traditional way is insufficient?

Q2: In the above you used min_filelist_kbytes=50000. How do you decide such value? Do other users can calculate proper value?

In addition, I have two request. R1: I think chromium specific feature is harder acceptable because it's harder maintable. but we have good chance to solve embedded generic issue. Please discuss Minchan and/or another embedded

developers. R2: If you want to deal OOM combination, please consider to combination of memcg OOM notifier too. It is most flexible and powerful OOM mechanism. Probably desktop and server people never use bare OOM killer intentionally.

Thanks.

Under memory pressure, I see the active list get smaller and smaller. Its getting smaller because we're scanning it faster and faster, causing more and more page faults which slows forward progress resulting in the active list getting smaller still. One way to approach this might to make the scan rate constant and configurable. It doesn't seem right that we scan memory faster and faster under low memory. For us, we'd rather OOM than evict pages that are likely to be accessed again so we'd prefer to make a conservative estimate as to what belongs in the working set. Other folks (long computations) might want to reclaim more aggressively.

Quoted 3 lines

Q2: In the above you used min_filelist_kbytes=50000. How do you decide such value? Do other users can calculate proper value?

50M was small enough that we were comfortable with keeping 50M of file pages in memory and large enough that it is bigger than the working set. I tested by loading up a bunch of popular web sites in chrome and then observing what happend when I ran out of memory. With 50M, I saw almost no thrashing and the system stayed responsive even under low memory. but I wanted to be conservative since I'm really just guessing.

Other users could calculate their value by doing something similar.

Quoted 9 lines · Mandeep Singh Baines

Under memory pressure, I see the active list get smaller and smaller. Its getting smaller because we're scanning it faster and faster, causing more and more page faults which slows forward progress resulting in the active list getting smaller still. One way to approach this might to make the scan rate constant and configurable. It doesn't seem right that we scan memory faster and faster under low memory. For us, we'd rather OOM than evict pages that are likely to be accessed again so we'd prefer to make a conservative estimate as to what belongs in the working set. Other folks (long computations) might want to reclaim more aggressively.

Have you actually read the code?

The active file list is only ever scanned when it is larger than the inactive file list.

Quoted 14 lines

Quoted 3 lines

Q2: In the above you used min_filelist_kbytes=50000. How do you decide such value? Do other users can calculate proper value?

50M was small enough that we were comfortable with keeping 50M of file pages
in memory and large enough that it is bigger than the working set. I tested
by loading up a bunch of popular web sites in chrome and then observing what
happend when I ran out of memory. With 50M, I saw almost no thrashing and the system stayed responsive even under low memory. but I wanted to be conservative since I'm really just guessing.

Quoted 1 line

Other users could calculate their value by doing something similar.

Maybe we can scale this by memory amount?

Say, make sure the total amount of page cache in the system is at least 2* as much as the sum of all the zone->pages_high watermarks, and refuse to evict page cache if we have less than that?

This may need to be tunable for a few special use cases, like HPC and virtual machine hosting nodes, but it may just do the right thing for everybody else.

Another alternative could be to really slow down the reclaiming of page cache once we hit this level, so virt hosts and HPC nodes can still decrease the page cache to something really small ... but only if it is not being used.

Andrew, could a hack like the above be "good enough"?

Anybody - does the above hack inspire you to come up with an even better idea?

Quoted 24 lines · KOSAKI Motohiro

Hi

Quoted 16 lines

On ChromiumOS, we do not use swap. When memory is low, the only way to free memory is to reclaim pages from the file list. This results in a lot of thrashing under low memory conditions. We see the system become unresponsive for minutes before it eventually OOMs. We also see very slow browser tab switching under low memory. Instead of an unresponsive system, we'd really like the kernel to OOM as soon as it starts to thrash. If it can't keep the working set in memory, then OOM. Losing one of many tabs is a better behaviour for the user than an unresponsive system.

This patch create a new sysctl, min_filelist_kbytes, which disables reclaim of file-backed pages when when there are less than min_filelist_bytes worth of such pages in the cache. This tunable is handy for low memory systems using solid-state storage where interactive response is more important than not OOMing.

Quoted 5 lines

With this patch and min_filelist_kbytes set to 50000, I see very little block layer activity during low memory. The system stays responsive under low memory and browser tab switching is fast. Eventually, a process a gets killed by OOM. Without this patch, the system gets wedged for minutes before it eventually OOMs. Below is the vmstat output from my test runs.

Quoted 4 lines

I've heared similar requirement sometimes from embedded people. then also don't use swap. then, I don't think this is hopeless idea. but I hope to clarify some thing at first.

swap would be intersting if we could somehow control swap thrashing. Maybe we could add min_anonlist_kbytes. Just kidding:)

Quoted 6 lines

Yes, a system often have should-not-be-evicted-file-caches. Typically, they are libc, libX11 and some GUI libraries. Traditionally, we was making tiny application which linked above important lib and call mlockall() at startup. such technique prevent reclaim. So, Q1: Why do you think above traditional way is insufficient?

mlock is too coarse grain. It requires locking the whole file in memory. The chrome and X binaries are quite large so locking them would waste a lot of memory. We could lock just the pages that are part of the working set but that is difficult to do in practice. Its unmaintainable if you do it statically. If you do it at runtime by mlocking the working set, you're sort of giving up on mm's active list.

Like akpm, I'm sad that we need this patch. I'd rather the kernel did a better job of identifying the working set. We did look at ways to do a better job of keeping the working set in the active list but these were tricker patches and never quite worked out. This patch is simple and works great.

Under memory pressure, I see the active list get smaller and smaller. Its getting smaller because we're scanning it faster and faster, causing more and more page faults which slows forward progress resulting in the active list getting smaller still. One way to approach this might to make the scan rate constant and configurable. It doesn't seem right that we scan memory faster and faster under low memory. For us, we'd rather OOM than evict pages that are likely to be accessed again so we'd prefer to make a conservative estimate as to what belongs in the working set. Other folks (long computations) might want to reclaim more aggressively.

Quoted 3 lines

Q2: In the above you used min_filelist_kbytes=50000. How do you decide such value? Do other users can calculate proper value?

50M was small enough that we were comfortable with keeping 50M of file pages in memory and large enough that it is bigger than the working set. I tested by loading up a bunch of popular web sites in chrome and then observing what happend when I ran out of memory. With 50M, I saw almost no thrashing and the system stayed responsive even under low memory. but I wanted to be conservative since I'm really just guessing.

Other users could calculate their value by doing something similar. Load up the system (exhaust free memory) with a typical load and then observe file io via vmstat. They can then set min_filelist_kbytes to the value where they see a tolerable amounting of thrashing (page faults, block io).

Quoted 3 lines

In addition, I have two request. R1: I think chromium specific feature is harder acceptable because it's harder maintable. but we have good chance to solve embedded generic issue. Please discuss Minchan and/or another embedded

I think this feature should be useful to a lot of embedded applications where OOM is OK, especially web browsing applications where the user is OK with losing 1 of many tabs they have open. However, I consider this patch a stop-gap. I think the real solution is to do a better job of protecting the active list.

Quoted 4 lines

developers. R2: If you want to deal OOM combination, please consider to combination of memcg OOM notifier too. It is most flexible and powerful OOM mechanism. Probably desktop and server people never use bare OOM killer intentionally.

Yes, will definitely look at OOM notifier. Currently trying to see if we can get by with oomadj. With OOM notifier you'd have to respond earlier so you might OOM more. However, with a notifier you might be able to take action that might prevent OOM altogether.

I see memcg more as an isolation mechanism but I guess you could use it to isolate the working set from anon browser tab data as Kamezawa suggests.

Regards,
Mandeep

Quoted 4 lines

Thanks.

Yes, this prevents you from reclaiming the active list all at once. But if the memory pressure doesn't go away, you'll start to reclaim the active list little by little. First you'll empty the inactive list, and then you'll start scanning
the active list and pulling pages from inactive to active. The problem is that there is no minimum time limit to how long a page will sit in the inactive list before it is reclaimed. Just depends on scan rate which does not depend on time.

In my experiments, I saw the active list get smaller and smaller over time until eventually it was only a few MB at which point the system came grinding to a halt due to thrashing.

Quoted 12 lines · Mandeep Singh Baines

Yes, this prevents you from reclaiming the active list all at once. But if the
memory pressure doesn't go away, you'll start to reclaim the active list little by little. First you'll empty the inactive list, and then you'll start scanning
the active list and pulling pages from inactive to active. The problem is that
there is no minimum time limit to how long a page will sit in the inactive list
before it is reclaimed. Just depends on scan rate which does not depend on time.

Quoted 4 lines

In my experiments, I saw the active list get smaller and smaller over time until eventually it was only a few MB at which point the system came
grinding to a halt due to thrashing.

I believe that changing the active/inactive ratio has other potential thrashing issues.  Specifically, when the inactive list is too small, pages may not stick around long enough to be accessed multiple times and get promoted to the active list, even when they are in active use.

I prefer a more flexible solution, that automatically does the right thing.

The problem you see is that the file list gets reclaimed very quickly, even when it is already very small.

I wonder if a possible solution would be to limit how fast file pages get reclaimed, when the page cache is very small. Say, inactive_file * active_file < 2 * zone->pages_high ?

At that point, maybe we could slow down the reclaiming of page cache pages to be significantly slower than they can be refilled by the disk.  Maybe 100 pages a second - that can be refilled even by an actual spinning metal disk without even the use of readahead.

That can be rounded up to one batch of SWAP_CLUSTER_MAX file pages every 1/4 second, when the number of page cache pages is very low.

This way HPC and virtual machine hosting nodes can still get rid of totally unused page cache, but on any system that actually uses page cache, some minimal amount of cache will be protected under heavy memory pressure.

Does this sound like a reasonable approach?

I realize the threshold may have to be tweaked...

The big question is, how do we integrate this with the OOM killer?  Do we pretend we are out of memory when we've hit our file cache eviction quota and kill something?

Would there be any downsides to this approach?

Are there any volunteers for implementing this idea? (Maybe someone who needs the feature?)

--
All rights reversed

Quoted 3 lines

I wonder if a possible solution would be to limit how fast file pages get reclaimed, when the page cache is very small. Say, inactive_file * active_file< 2 * zone->pages_high ?

Why do you multiply inactive_file and active_file? What's meaning?

I think it's very difficult to fix _a_ threshold.
At least, user have to set it with proper value to use the feature. Anyway, we need default value. It needs some experiments in desktop and embedded.

Quoted 6 lines

At that point, maybe we could slow down the reclaiming of page cache pages to be significantly slower than they can be refilled by the disk. Maybe 100 pages a second - that can be refilled even by an actual spinning metal disk without even the use of readahead.

Quoted 3 lines

That can be rounded up to one batch of SWAP_CLUSTER_MAX file pages every 1/4 second, when the number of page cache pages is very low.

How about reducing scanning window size?
I think it could approximate the idea.

Quoted 1 line

Would there be any downsides to this approach?

At first feeling, I have a concern unbalance aging of anon/file. But I think it's no problem. It a result user want. User want to protect file-backed page(ex, code page) so many anon swapout is natural result to go on the system. If the system has no swap, we have no choice except OOM.

Quoted 6 lines · Minchan Kim

Quoted 3 lines

I wonder if a possible solution would be to limit how fast file pages get reclaimed, when the page cache is very small. Say, inactive_file * active_file<  2 * zone->pages_high ?

Why do you multiply inactive_file and active_file? What's meaning?

That was a stupid typo, it should have been a + :)

Quoted 4 lines

I think it's very difficult to fix _a_ threshold.
At least, user have to set it with proper value to use the feature. Anyway, we need default value. It needs some experiments in desktop and embedded.

Yes, setting a threshold will be difficult.  However, if the behaviour below that threshold is harmless to pretty much any workload, it doesn't matter a whole lot where we set it...

Quoted 11 lines

Quoted 10 lines

At that point, maybe we could slow down the reclaiming of page cache pages to be significantly slower than they can be refilled by the disk.  Maybe 100 pages a second - that can be refilled even by an actual spinning metal disk without even the use of readahead.

That can be rounded up to one batch of SWAP_CLUSTER_MAX file pages every 1/4 second, when the number of page cache pages is very low.

Quoted 2 lines

How about reducing scanning window size?
I think it could approximate the idea.

A good idea in principle, but if it results in the VM simply calling the pageout code more often, I suspect it will not have any effect.

Your patch looks like it would have that effect.

I suspect we will need a time-based approach to really protect the last bits of page cache in a near-OOM situation.

Quoted 7 lines

Quoted 1 line

Would there be any downsides to this approach?

At first feeling, I have a concern unbalance aging of anon/file. But I think it's no problem. It a result user want. User want to protect file-backed page(ex, code page) so many anon swapout is natural result to go on the system. If the system has no swap, we have no choice except OOM.

We already have an unbalance in aging anon and file pages, several of which are introduced on purpose.

In this proposal, there would only be an imbalance if the number of file pages is really low.

--
All rights reversed

It could.
But time based approach would be same, IMHO.
First of all, I don't want long latency of direct reclaim process. It could affect response of foreground process directly.

If VM limits the number of pages reclaimed per second, direct reclaim process's latency will be affected. so we should avoid throttling in direct reclaim path. Agree?

Quoted 5 lines · Minchan Kim

It could.
But time based approach would be same, IMHO.
First of all, I don't want long latency of direct reclaim process. It could affect response of foreground process directly.

Quoted 3 lines

If VM limits the number of pages reclaimed per second, direct reclaim process's latency will be affected. so we should avoid throttling in direct reclaim path. Agree?

The idea would be to not throttle the processes trying to reclaim page cache pages, but to only reclaim anonymous pages when the page cache pages are low (and occasionally a few page cache pages, say 128 a second).

If too many reclaimers come in when the page cache is low and no swap is available, we will OOM kill instead of stalling.

After all, the entire point of this patch would be to avoid minutes-long latencies in triggering the OOM killer.

Signature · 2 lines

-- 
All rights reversed

Quoted 9 lines · Mandeep Singh Baines

Yes, this prevents you from reclaiming the active list all at once. But if the memory pressure doesn't go away, you'll start to reclaim the active list little by little. First you'll empty the inactive list, and then you'll start scanning
the active list and pulling pages from inactive to active. The problem is that there is no minimum time limit to how long a page will sit in the inactive list before it is reclaimed. Just depends on scan rate which does not depend on time.

Quoted 3 lines

In my experiments, I saw the active list get smaller and smaller over time until eventually it was only a few MB at which point the system came grinding to a halt due to thrashing.

I believe that changing the active/inactive ratio has other potential thrashing issues. Specifically, when the inactive list is too small, pages may not stick around long enough to be accessed multiple times and get promoted to the active list, even when they are in active use.

I prefer a more flexible solution, that automatically does the right thing.

The problem you see is that the file list gets reclaimed very quickly, even when it is already very small.

I wonder if a possible solution would be to limit how fast file pages get reclaimed, when the page cache is very small. Say, inactive_file * active_file < 2 * zone->pages_high ?

At that point, maybe we could slow down the reclaiming of page cache pages to be significantly slower than they can be refilled by the disk. Maybe 100 pages a second - that can be refilled even by an actual spinning metal disk without even the use of readahead.

That can be rounded up to one batch of SWAP_CLUSTER_MAX file pages every 1/4 second, when the number of page cache pages is very low.

This way HPC and virtual machine hosting nodes can still get rid of totally unused page cache, but on any system that actually uses page cache, some minimal amount of cache will be protected under heavy memory pressure.

Does this sound like a reasonable approach?

I realize the threshold may have to be tweaked...

The big question is, how do we integrate this with the OOM killer? Do we pretend we are out of memory when we've hit our file cache eviction quota and kill something?

Would there be any downsides to this approach?

Are there any volunteers for implementing this idea? (Maybe someone who needs the feature?)

Quoted 50 lines · Rik van Riel

Quoted 9 lines · Mandeep Singh Baines

Yes, this prevents you from reclaiming the active list all at once. But if the memory pressure doesn't go away, you'll start to reclaim the active list little by little. First you'll empty the inactive list, and then you'll start scanning
the active list and pulling pages from inactive to active. The problem is that there is no minimum time limit to how long a page will sit in the inactive list before it is reclaimed. Just depends on scan rate which does not depend on time.

Quoted 3 lines

In my experiments, I saw the active list get smaller and smaller over time until eventually it was only a few MB at which point the system came grinding to a halt due to thrashing.

I believe that changing the active/inactive ratio has other potential thrashing issues.  Specifically, when the inactive list is too small, pages may not stick around long enough to be accessed multiple times and get promoted to the active list, even when they are in active use.

I prefer a more flexible solution, that automatically does the right thing.

The problem you see is that the file list gets reclaimed very quickly, even when it is already very small.

I wonder if a possible solution would be to limit how fast file pages get reclaimed, when the page cache is very small. Say, inactive_file * active_file < 2 * zone->pages_high ?

At that point, maybe we could slow down the reclaiming of page cache pages to be significantly slower than they can be refilled by the disk.  Maybe 100 pages a second - that can be refilled even by an actual spinning metal disk without even the use of readahead.

That can be rounded up to one batch of SWAP_CLUSTER_MAX file pages every 1/4 second, when the number of page cache pages is very low.

This way HPC and virtual machine hosting nodes can still get rid of totally unused page cache, but on any system that actually uses page cache, some minimal amount of cache will be protected under heavy memory pressure.

Does this sound like a reasonable approach?

I realize the threshold may have to be tweaked...

Quoted 9 lines

The big question is, how do we integrate this with the OOM killer?  Do we pretend we are out of memory when we've hit our file cache eviction quota and kill something?

Would there be any downsides to this approach?

Are there any volunteers for implementing this idea? (Maybe someone who needs the feature?)

I've created a patch which takes a slightly different approach. Instead of limiting how fast pages get reclaimed, the patch limits how fast the active list gets scanned. This should result in the active list being a better measure of the working set. I've seen fairly good results with this patch and a scan inteval of 1 centisecond. I see no thrashing when the scan interval is non-zero.

I've made it a tunable because I don't know what to set the scan interval. The final patch could set the value based on HZ and some other system parameters. Maybe relate it to sched_period?

---

[PATCH] vmscan: add a configurable scan interval

On ChromiumOS, we see a lot of thrashing under low memory. We do not use swap, so the mm system can only free file-backed pages. Eventually, we are left with little file back pages remaining (a few MB) and the system becomes unresponsive due to thrashing.

Our preference is for the system to OOM instead of becoming unresponsive.

This patch create a tunable, vmscan_interval_centisecs, for controlling the minimum interval between active list scans. At 0, I see the same thrashing. At 1, I see no thrashing. The mm system does a good job of protecting the working set. If a page has been referenced in the last vmscan_interval_centisecs it is kept in memory.

Signed-off-by Mandeep Singh Baines <[email protected]>

Quoted 93 lines · Mandeep Singh Baines

Quoted 24 lines · KOSAKI Motohiro

Hi

Quoted 16 lines

On ChromiumOS, we do not use swap. When memory is low, the only way to free memory is to reclaim pages from the file list. This results in a lot of thrashing under low memory conditions. We see the system become unresponsive for minutes before it eventually OOMs. We also see very slow browser tab switching under low memory. Instead of an unresponsive system, we'd really like the kernel to OOM as soon as it starts to thrash. If it can't keep the working set in memory, then OOM. Losing one of many tabs is a better behaviour for the user than an unresponsive system.

This patch create a new sysctl, min_filelist_kbytes, which disables reclaim of file-backed pages when when there are less than min_filelist_bytes worth of such pages in the cache. This tunable is handy for low memory systems using solid-state storage where interactive response is more important than not OOMing.

Quoted 5 lines

With this patch and min_filelist_kbytes set to 50000, I see very little block layer activity during low memory. The system stays responsive under low memory and browser tab switching is fast. Eventually, a process a gets killed by OOM. Without this patch, the system gets wedged for minutes before it eventually OOMs. Below is the vmstat output from my test runs.

Quoted 4 lines

I've heared similar requirement sometimes from embedded people. then also don't use swap. then, I don't think this is hopeless idea. but I hope to clarify some thing at first.

swap would be intersting if we could somehow control swap thrashing. Maybe we could add min_anonlist_kbytes. Just kidding:)

Quoted 6 lines

Yes, a system often have should-not-be-evicted-file-caches. Typically, they are libc, libX11 and some GUI libraries. Traditionally, we was making tiny application which linked above important lib and call mlockall() at startup. such technique prevent reclaim. So, Q1: Why do you think above traditional way is insufficient?

mlock is too coarse grain. It requires locking the whole file in memory. The chrome and X binaries are quite large so locking them would waste a lot of memory. We could lock just the pages that are part of the working set but that is difficult to do in practice. Its unmaintainable if you do it statically. If you do it at runtime by mlocking the working set, you're sort of giving up on mm's active list.

Like akpm, I'm sad that we need this patch. I'd rather the kernel did a better job of identifying the working set. We did look at ways to do a better job of keeping the working set in the active list but these were tricker patches and never quite worked out. This patch is simple and works great.

Under memory pressure, I see the active list get smaller and smaller. Its getting smaller because we're scanning it faster and faster, causing more and more page faults which slows forward progress resulting in the active list getting smaller still. One way to approach this might to make the scan rate constant and configurable. It doesn't seem right that we scan memory faster and faster under low memory. For us, we'd rather OOM than evict pages that are likely to be accessed again so we'd prefer to make a conservative estimate as to what belongs in the working set. Other folks (long computations) might want to reclaim more aggressively.

Quoted 3 lines

Q2: In the above you used min_filelist_kbytes=50000. How do you decide such value? Do other users can calculate proper value?

50M was small enough that we were comfortable with keeping 50M of file pages in memory and large enough that it is bigger than the working set. I tested by loading up a bunch of popular web sites in chrome and then observing what happend when I ran out of memory. With 50M, I saw almost no thrashing and the system stayed responsive even under low memory. but I wanted to be conservative since I'm really just guessing.

Other users could calculate their value by doing something similar. Load up the system (exhaust free memory) with a typical load and then observe file io via vmstat. They can then set min_filelist_kbytes to the value where they see a tolerable amounting of thrashing (page faults, block io).

Quoted 3 lines

In addition, I have two request. R1: I think chromium specific feature is harder acceptable because it's harder maintable. but we have good chance to solve embedded generic issue. Please discuss Minchan and/or another embedded

I think this feature should be useful to a lot of embedded applications where OOM is OK, especially web browsing applications where the user is OK with losing 1 of many tabs they have open. However, I consider this patch a stop-gap. I think the real solution is to do a better job of protecting the active list.

Quoted 4 lines

developers. R2: If you want to deal OOM combination, please consider to combination of memcg OOM notifier too. It is most flexible and powerful OOM mechanism. Probably desktop and server people never use bare OOM killer intentionally.

Quoted 7 lines

Yes, will definitely look at OOM notifier. Currently trying to see if we can get by with oomadj. With OOM notifier you'd have to respond earlier so you might OOM more. However, with a notifier you might be able to take action that might prevent OOM altogether.

I see memcg more as an isolation mechanism but I guess you could use it to isolate the working set from anon browser tab data as Kamezawa suggests.

I don't think current VM behavior has a problem.
Current problem is that you use up many memory than real memory. As system memory without swap is low, VM doesn't have a many choice. It ends up evict your working set to meet for user request. It's very natural result for greedy user.

Rather than OOM notifier, what we need is memory notifier. AFAIR, before some years ago, KOSAKI tried similar thing .
http://lwn.net/Articles/268732/

(I can't remember why KOSAKI quit it exactly, AFAIR, some signal time can't meet yours requirement. I mean when the user receive the memory low signal, it's too late. Maybe there are other causes for KOSAKi to quit it.)
Anyway, If the system memory is low, your intelligent middleware can control it very well than VM.

In this chance, how about improving it?
Mandeep, Could you feel needing this feature?

Quoted 10 lines

Regards,
Mandeep

Quoted 4 lines

Thanks.

Signature · 2 lines

Kind regards,
Minchan Kim

I've created a patch which takes a slightly different approach. Instead of limiting how fast pages get reclaimed, the patch limits how fast the active list gets scanned. This should result in the active list being a better measure of the working set. I've seen fairly good results with this patch and a scan inteval of 1 centisecond. I see no thrashing when the scan interval is non-zero.

I've made it a tunable because I don't know what to set the scan interval. The final patch could set the value based on HZ and some other system parameters. Maybe relate it to sched_period?

Quoted 11 lines · Minchan Kim

Quoted 2 lines · Mandeep Singh Baines

I see memcg more as an isolation mechanism but I guess you could use it to isolate the working set from anon browser tab data as Kamezawa suggests.

I don't think current VM behavior has a problem.
Current problem is that you use up many memory than real memory. As system memory without swap is low, VM doesn't have a many choice. It ends up evict your working set to meet for user request. It's very natural result for greedy user.

Quoted 3 lines

Rather than OOM notifier, what we need is memory notifier. AFAIR, before some years ago, KOSAKI tried similar thing .
http://lwn.net/Articles/268732/

Thanks! This is perfect. I wonder why its not merged. Was a different solution eventually implemented? Is there another way of doing the same thing?

Quoted 7 lines

(I can't remember why KOSAKI quit it exactly, AFAIR, some signal time can't meet yours requirement. I mean when the user receive the memory low signal, it's too late. Maybe there are other causes for KOSAKi to quit it.)
Anyway, If the system memory is low, your intelligent middleware can control it very well than VM.

Agree.

Quoted 3 lines

In this chance, how about improving it?
Mandeep, Could you feel needing this feature?

mem_notify seems perfect.

Quoted 17 lines

Quoted 11 lines

Regards,
Mandeep

Quoted 4 lines

Thanks.

--
Kind regards,
Minchan Kim

Quoted 10 lines · Mandeep Singh Baines

I've created a patch which takes a slightly different approach. Instead of limiting how fast pages get reclaimed, the patch limits how fast the active list gets scanned. This should result in the active list being a better measure of the working set. I've seen fairly good results with this patch and a scan inteval of 1 centisecond. I see no thrashing when the scan interval is non-zero.

I've made it a tunable because I don't know what to set the scan interval. The final patch could set the value based on HZ and some other system parameters. Maybe relate it to sched_period?

I like your approach. For file pages it looks like it could work fine, since new pages always start on the inactive file list.

However, for anonymous pages I could see your patch leading to problems, because all anonymous pages start on the active list. With a scan interval of 1 centiseconds, that means there would be a limit of 3200 pages, or 12MB of anonymous memory that can be moved to the inactive list a second.

I have seen systems with single SATA disks push out several times that to swap per second, which matters when someone starts up a program that is just too big to fit in memory and requires that something is pushed out.

That would reduce the size of the inactive list to zero, reducing our page replacement to a slow FIFO at best, causing false OOM kills at worst.

Staying with a default of 0 would of course not do anything, which would make merging the code not too useful.

I believe we absolutely need to preserve the ability to evict pages quickly, when new pages are brought into memory or allocated quickly.

However, speed limits are probably a very good idea once a cache has been reduced to a smaller size, or when most IO bypasses the reclaim-speed-limited cache.

Signature · 1 line

All rights reversed

Quoted 11 lines

I've created a patch which takes a slightly different approach. Instead of limiting how fast pages get reclaimed, the patch limits how fast the active list gets scanned. This should result in the active list being a better measure of the working set. I've seen fairly good results with this patch and a scan inteval of 1 centisecond. I see no thrashing when the scan interval is non-zero.

I've made it a tunable because I don't know what to set the scan interval. The final patch could set the value based on HZ and some other system parameters. Maybe relate it to sched_period?

I like your approach. For file pages it looks like it could work fine, since new pages always start on the inactive file list.

However, for anonymous pages I could see your patch leading to problems, because all anonymous pages start on the active list. With a scan interval of 1 centiseconds, that means there would be a limit of 3200 pages, or 12MB of anonymous memory that can be moved to the inactive list a second.

I have seen systems with single SATA disks push out several times that to swap per second, which matters when someone starts up a program that is just too big to fit in memory and requires that something is pushed out.

That would reduce the size of the inactive list to zero, reducing our page replacement to a slow FIFO at best, causing false OOM kills at worst.

Staying with a default of 0 would of course not do anything, which would make merging the code not too useful.

I believe we absolutely need to preserve the ability to evict pages quickly, when new pages are brought into memory or allocated quickly.

However, speed limits are probably a very good idea once a cache has been reduced to a smaller size, or when most IO bypasses the reclaim-speed-limited cache.

Quoted 6 lines

I don't think current VM behavior has a problem.
Current problem is that you use up many memory than real memory. As system memory without swap is low, VM doesn't have a many choice. It ends up evict your working set to meet for user request. It's very natural result for greedy user.

Quoted 3 lines

Rather than OOM notifier, what we need is memory notifier. AFAIR, before some years ago, KOSAKI tried similar thing .
http://lwn.net/Articles/268732/

Thanks! This is perfect. I wonder why its not merged. Was a different solution eventually implemented? Is there another way of doing the same thing?