System locks with kswapd0 and kworker taking full IO and mem

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Guys, could you take a look please?

The machine went oom when there's heaps of unused swap and most memory is being used on active_anon and inactive_anon. We should have just swapped that stuff out and kept going.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Isn't there already logic in the kernel that disables OOM if there's swap space available?

I saw it once, what happened to it?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I think we need to re-evaluate the way we balance file and anon scan pressure. It's not just the "not swapping" aspect that bugs me, it's also the fact that the machine has been thrashing page cache at full load for *minutes* before signalling the OOM.

SSDs can flush and reload pages quick enough that on memory pressure there are always reclaimable cache pages and the scanner never goes after anonymous memory. If anonymous memory does not leave enough room for page cache to hold the libraries and executables, userspace goes into a state where it's mostly waiting for cache to become uptodate.

It's a very frustrating problem because it's hard to even detect.

One idea I had to address the LRU balance problem in the past was to always reclaim the pages in the following order: inactive file, active file, anon*. As one set becomes empty, go after the next one. If the workingset code detects cache thrashing, it depends on the refault distances what to do: if they are smaller than the active file size, deactivate; if they are bigger than that, but smaller than active file
+ anon, we need to start swapping to alleviate the cache thrashing.

Now, if the refault distances are bigger than active file + anon, no amount of deactivating and swapping are going to stop the thrashing and we have to think about triggering OOM. But OOM is drastic and the refaults might happen at a very slow pace (or, with sparse files, not require any IO at all) and the system might be completely fine. So in addition this would require a measure of overall time spent on thrashing IO, comparable to what Tejun proposed in "[RFD] memory pressure and sizing problem", where we say if thrashing IO takes up X percent of all execution time spent, we trigger the OOM killer--not to free memory, but to reduce the tasks that contribute to the thrashing and let the remaining tasks make progress, similar to the swap token or a BSD style memory scheduler.

* we can ignore the difference between inactive and active anon here as anon is not aged the same way as the file LRU is aged

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

We already measure how much the system is slowed down by waiting on IO - iowait time. It's not perfect, but it can give us some indication whether or not we are thrashing on page cache access.

The BSD style process swapping only takes mapped memory into account. Page cache thrashing is pretty much ignored.

Maybe we should only count page fault stalls (do we track those already?) in addition to refault distances?

All rights reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[Sorry for replying here but I couldn't find the original Andrew's email in my mailbox]

I would strongly suspect the memory is pinned by somebody which completely ruins all the get_scan_count assumptions. The first referenced OOM report might contain a hint:

[ 2162.123944] Purging GPU memory, 368640 bytes freed, 615292928 bytes still pinned.
[ 2175.996060] Purging GPU memory, 499712 bytes freed, 615251968 bytes still pinned.
[ 2175.998841] bash invoked oom-killer: gfp_mask=0x20858, order=0, oom_score_adj=0
[ 2175.998844] bash cpuset=/ mems_allowed=0
[...]
[ 2175.999016] active_anon:305425 inactive_anon:141206 isolated_anon:0
                active_file:5109 inactive_file:4666 isolated_file:0
                unevictable:4 dirty:2 writeback:0 unstable:0
                free:13218 slab_reclaimable:6552 slab_unreclaimable:11310
                mapped:21203 shmem:155079 pagetables:10921 bounce:0
                free_cma:0
[...]
[ 2175.999074] 169619 total pagecache pages
[ 2175.999076] 4752 pages in swap cache
[ 2175.999078] Swap cache stats: add 468915, delete 464163, find 76521/98873
[ 2175.999080] Free swap  = 1615656kB
[ 2175.999082] Total swap = 2097148kB
[ 2175.999083] 521838 pages RAM
[ 2175.999084] 0 pages HighMem/MovableOnly
[ 2175.999086] 11811 pages reserved
[ 2175.999087] 0 pages hwpoisoned

So there is more than 600MB used by the GPU. Later OOM invocations do not mention GPU OOM shrinker at all. Anon+File+Unevict+Free+Slab+Pagetbl gives us 1.9G so considerable amount of pinned memory has to be sitting on LRU lists. I would bet it is shmem here but there is still more than 1G on the anon LRU lists. Is it possible they are pinned indirectly? I am CCing Daniel for the GPU memory consumption. Maybe there is some additional diagnostic to look at.

Another interesting thing to note is that

[ 2175.999473] Out of memory: Kill process 3566 (java) score 170 or sacrifice child
[ 2175.999477] Killed process 3566 (java) total-vm:3417044kB, anon-rss:703656kB, file-rss:0kB
[...]
[ 2176.000641] bash invoked oom-killer: gfp_mask=0x20858, order=0, oom_score_adj=0
[ 2176.000644] bash cpuset=/ mems_allowed=0
[...]
[ 2176.000798] active_anon:305425 inactive_anon:141206 isolated_anon:0
                active_file:5109 inactive_file:4666 isolated_file:0
                unevictable:4 dirty:2 writeback:0 unstable:0
                free:13187 slab_reclaimable:6552 slab_unreclaimable:11310
                mapped:21203 shmem:155079 pagetables:10921 bounce:0
                free_cma:0

So the anon LRU lists are intact even after java has exited so something is clearly wrong the anon LRU list and it looks like a leak via elevated page ref. counting.

It sounds like this is reproducible for you Gonzalo, could you invoke a crash dump and save the vmcore so that the LRU can be investigated? We would see the state after something went wrong but maybe there will be some pattern to help us.

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

So this isn't fixed and a number of new reporters (cc'ed) are chiming in (let's please keep this going via email, not via the bugzilla UI!).

We have various theories but I don't think we've nailed it down yet.

Are any of the reporters able to come up with a set of instructions which will permit the developers to reproduce this bug locally?

Can we think up a way of adding some form of debug/instrumentation to the kernel which will permit us to diagnose and fix this? It could be something which a tester manually adds or it could be something permanent, perhaps controlled via a procfs knob.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

So, I'm nowhere close to this at the moment. I was aware of at least one swapping-related problem that was introduced between 4.0 and 4.1. The commit that introduced it only affects NUMA so there is no chance they are related. However, I'll still need to chase that down early next week before considering this problem. Someone else may figure it out faster.

As the problem I'm aware of is NUMA only, I took a momentary look at this. The first log shows MCE errors but they may be overheating related so I'm willing to ignore that.

The log clearly states that a lot of memory is pinned by the GPU just before the OOM triggers.

[ 2175.996060] Purging GPU memory, 499712 bytes freed, 615251968 bytes still pinned.

So that in itself is a major problem. Next the memory usage at the time of failure was

[ 2175.999016] active_anon:305425 inactive_anon:141206 isolated_anon:0
                active_file:5109 inactive_file:4666 isolated_file:0
                unevictable:4 dirty:2 writeback:0 unstable:0
                free:13218 slab_reclaimable:6552 slab_unreclaimable:11310
                mapped:21203 shmem:155079 pagetables:10921 bounce:0
                free_cma:0

1.8G of anony memory usage with almost 600M of that being GPU-related. The file usage is negligible so this is looking closer to being a true OOM situation

[ 2175.999080] Free swap  = 1615656kB
[ 2175.999082] Total swap = 2097148kB

Load of swap available. The IO is likely high because files are probably being continually reclaimed and paged back in so it's thrashing. Johannes is likely correct when he says there is a problem with balancing when the storage is fast. That's one aspect of the problem but it does not explain why the problem is recent. The one major candidate I can spot is this

1da58ee2: mm: vmscan: count only dirty pages as congested

That alters how and when processes are put to sleep waiting on congestion to clear. While I can see the logic behind the patch, the impact was no quantified and it can mean that kswapd is no longer throttling when it used to. Try something like this untested

index 2aec4241b42a..50b24a022db0 100644
@@ -953,8 +953,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 		 * end of the LRU a second time.
 		 */
 		mapping = page_mapping(page);
-		if (((dirty || writeback) && mapping &&
-		     inode_write_congested(mapping->host)) ||
+		if ((mapping && inode_write_congested(mapping->host)) ||
 		    (writeback && PageReclaim(page)))
 			nr_congested++;
 

This is not necessary the right fix, it just may narrow down where the problem is.

The problem is compounded probably by scasnning one third of the LRU before any reclaim candidates are found. Is it known if all the people reporting problems are using an i915 GPU? If so, Daniel, are you aware of any commits between 3.18 and 4.1 that would potentially pin GPU memory permanently or alternative would have busted the shrinker?

Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>