Use up free swap space before reaching OOM kill

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled.

Luigi reported there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

try_to_free_pages disable may_writepage if laptop_mode is enabled. shrink_page_list adds lots of anon pages in swap cache by add_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.

In case of that, isolate_lru_pages fails because it try to isolate clean page due to may_writepage == 0.

The may_writepage could be 1 only if total_scanned is higher than writeback_threshold in do_try_to_free_pages but unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.

This patch prevents to add a page to swap cache unnecessary when may_writepage is unset so anoymous lru list isn't full of Dirty/Swapcache page. So VM can isolate pages from anon lru list, which ends up setting may_writepage to 1 and could swap out anon lru pages. When OOM triggers, I confirmed swap space was full.

Reported-by Luigi Semenzato <[email protected]>

Signed-off-by Minchan Kim <[email protected]>

Quoted 28 lines · Minchan Kim

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled.

Luigi reported there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

try_to_free_pages disable may_writepage if laptop_mode is enabled. shrink_page_list adds lots of anon pages in swap cache by add_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.

In case of that, isolate_lru_pages fails because it try to isolate clean page due to may_writepage == 0.

The may_writepage could be 1 only if total_scanned is higher than writeback_threshold in do_try_to_free_pages but unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.

This patch prevents to add a page to swap cache unnecessary when may_writepage is unset so anoymous lru list isn't full of Dirty/Swapcache page. So VM can isolate pages from anon lru list, which ends up setting may_writepage to 1 and could swap out anon lru pages. When OOM triggers, I confirmed swap space was full.

Quoted 2 lines

Reported-by Luigi Semenzato <[email protected]>

Signed-off-by Minchan Kim <[email protected]>

Acked-by Johannes Weiner <[email protected]>

We used to ignore the page's writeback state on isolation in the past, could you include a reference to since when this problem has been in

the tree? Also, would it make sense to tag it for one of the stable trees?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled.

Luigi reported there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

try_to_free_pages disable may_writepage if laptop_mode is enabled. shrink_page_list adds lots of anon pages in swap cache by add_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.

In case of that, isolate_lru_pages fails because it try to isolate clean page due to may_writepage == 0.

The may_writepage could be 1 only if total_scanned is higher than writeback_threshold in do_try_to_free_pages but unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.

This patch prevents to add a page to swap cache unnecessary when may_writepage is unset so anoymous lru list isn't full of Dirty/Swapcache page. So VM can isolate pages from anon lru list, which ends up setting may_writepage to 1 and could swap out anon lru pages. When OOM triggers, I confirmed swap space was full.

...

Quoted 30 lines · On Wed, 9 Jan 2013 15:21:13 +0900 Minchan Kim <[email protected]> wrote:

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled.

Luigi reported there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

try_to_free_pages disable may_writepage if laptop_mode is enabled. shrink_page_list adds lots of anon pages in swap cache by add_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.

In case of that, isolate_lru_pages fails because it try to isolate clean page due to may_writepage == 0.

The may_writepage could be 1 only if total_scanned is higher than writeback_threshold in do_try_to_free_pages but unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.

This patch prevents to add a page to swap cache unnecessary when may_writepage is unset so anoymous lru list isn't full of Dirty/Swapcache page. So VM can isolate pages from anon lru list, which ends up setting may_writepage to 1 and could swap out anon lru pages. When OOM triggers, I confirmed swap space was full.

...

mm/vmscan.c static unsigned long shrink_page_list(struct list_head *page_list, · line 787

Quoted 11 lines

mm/vmscan.c +2 -0

@@ -780,6 +780,8 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 		if (PageAnon(page) && !PageSwapCache(page)) {
 			if (!(sc->gfp_mask & __GFP_IO))
 				goto keep_locked;
+			if (!sc->may_writepage)
+				goto keep_locked;
 			if (!add_to_swap(page))
 				goto activate_locked;
 			may_enter_fs = 1;

I'm not really getting it, and the description is rather hard to follow :(

We should be adding anon pages to swapcache even when laptop_mode is set. And we should be writing them to swap as well, then reclaiming them. The only thing laptop_mode shouild do is make the disk spin up less frequently - that doesn't mean "not at all"!

So something seems screwed up here and the patch looks like a heavy-handed workaround. Why aren't these anon pages getting written out in laptop_mode?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Andrew,

Quoted 31 lines · Andrew Morton

Quoted 28 lines · On Wed, 9 Jan 2013 15:21:13 +0900 Minchan Kim <[email protected]> wrote:

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled.

Luigi reported there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

try_to_free_pages disable may_writepage if laptop_mode is enabled. shrink_page_list adds lots of anon pages in swap cache by add_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.

In case of that, isolate_lru_pages fails because it try to isolate clean page due to may_writepage == 0.

The may_writepage could be 1 only if total_scanned is higher than writeback_threshold in do_try_to_free_pages but unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.

This patch prevents to add a page to swap cache unnecessary when may_writepage is unset so anoymous lru list isn't full of Dirty/Swapcache page. So VM can isolate pages from anon lru list, which ends up setting may_writepage to 1 and could swap out anon lru pages. When OOM triggers, I confirmed swap space was full.

Quoted 5 lines

mm/vmscan.c static unsigned long shrink_page_list(struct

Quoted 5 lines

...

mm/vmscan.c +0 -0

@@ -780,6 +780,8 @@ static unsigned long shrink_page_list(struct

list_head *page_list,

Quoted 10 lines

mm/vmscan.c static unsigned long shrink_page_list(struct list_head *page_list, · line 787

Quoted 8 lines

mm/vmscan.c +2 -0

            if (PageAnon(page) && !PageSwapCache(page)) {
                    if (!(sc->gfp_mask & __GFP_IO))
                            goto keep_locked;
+                   if (!sc->may_writepage)
+                           goto keep_locked;
                    if (!add_to_swap(page))
                            goto activate_locked;
                    may_enter_fs = 1;

I'm not really getting it, and the description is rather hard to follow

:(

It seems I don't have a talent about description. :( I hope it would be better this year. :)

Quoted 5 lines

We should be adding anon pages to swapcache even when laptop_mode is set. And we should be writing them to swap as well, then reclaiming them. The only thing laptop_mode shouild do is make the disk spin up less frequently - that doesn't mean "not at all"!

So it seems your rationale is that let's save power in only system has enough memory so let's remove may_writepage in reclaim path?

If it is, I love it because I didn't see any number about power saving through reclaiming throttling(But surely there was reason to add it) and not sure it works well during long time because we have tweaked reclaim part too many.

Quoted 4 lines

So something seems screwed up here and the patch looks like a heavy-handed workaround. Why aren't these anon pages getting written out in laptop_mode?

Don't know. It was there long time and I don't want to screw it up. If we decide paging out in reclaim path regardless of laptop_mode, it makes the problem easy without ugly workaround.

Remove may_writepage? If it's too agressive, we can remove it in only direct reclaim path.

Quoted 7 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

For what it's worth, I tested this patch on my 3.4 kernel, and it works as advertised. Here's my setup.

- 2 GB RAM
- a 3 GB zram disk for swapping
- start one "hog" process per second (each hog process mallocs and touches 200 MB of memory).
- watch /proc/meminfo

1. I verified that the problem still exists on my current 3.4 kernel. With laptop_mode = 2, hog processes are oom-killed when about 1.8-1.9 (out of 3) GB of swap space are still left

2. I double-checked that the problem does not exist with laptop_mode = 0:
hog processes are oom-killed when swap space is exhausted (with good approximation).

3. I added the two-line patch, put back laptop_mode = 2, and verified that hog processes are oom-killed when swap space is exhausted, same as case 2.

Let me know if I can run any more tests for you, and thanks for all the support so far!

Quoted 100 lines

Hi Andrew,

Quoted 31 lines · Andrew Morton

Quoted 28 lines · On Wed, 9 Jan 2013 15:21:13 +0900 Minchan Kim <[email protected]> wrote:

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled.

Luigi reported there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

try_to_free_pages disable may_writepage if laptop_mode is enabled. shrink_page_list adds lots of anon pages in swap cache by add_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.

In case of that, isolate_lru_pages fails because it try to isolate clean page due to may_writepage == 0.

The may_writepage could be 1 only if total_scanned is higher than writeback_threshold in do_try_to_free_pages but unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.

This patch prevents to add a page to swap cache unnecessary when may_writepage is unset so anoymous lru list isn't full of Dirty/Swapcache page. So VM can isolate pages from anon lru list, which ends up setting may_writepage to 1 and could swap out anon lru pages. When OOM triggers, I confirmed swap space was full.

Quoted 5 lines

mm/vmscan.c static unsigned long shrink_page_list(struct

Quoted 5 lines

...

mm/vmscan.c +0 -0

@@ -780,6 +780,8 @@ static unsigned long shrink_page_list(struct

list_head *page_list,

Quoted 10 lines

mm/vmscan.c static unsigned long shrink_page_list(struct list_head *page_list, · line 787

Quoted 8 lines

mm/vmscan.c +2 -0

            if (PageAnon(page) && !PageSwapCache(page)) {
                    if (!(sc->gfp_mask & __GFP_IO))
                            goto keep_locked;
+                   if (!sc->may_writepage)
+                           goto keep_locked;
                    if (!add_to_swap(page))
                            goto activate_locked;
                    may_enter_fs = 1;

I'm not really getting it, and the description is rather hard to follow

:(

It seems I don't have a talent about description. :( I hope it would be better this year. :)

Quoted 5 lines

We should be adding anon pages to swapcache even when laptop_mode is set. And we should be writing them to swap as well, then reclaiming them. The only thing laptop_mode shouild do is make the disk spin up less frequently - that doesn't mean "not at all"!

So it seems your rationale is that let's save power in only system has enough memory so let's remove may_writepage in reclaim path?

If it is, I love it because I didn't see any number about power saving through reclaiming throttling(But surely there was reason to add it) and not sure it works well during long time because we have tweaked reclaim part too many.

Quoted 4 lines

So something seems screwed up here and the patch looks like a heavy-handed workaround. Why aren't these anon pages getting written out in laptop_mode?

Don't know. It was there long time and I don't want to screw it up. If we decide paging out in reclaim path regardless of laptop_mode, it makes the problem easy without ugly workaround.

Remove may_writepage? If it's too agressive, we can remove it in only direct reclaim path.

Quoted 7 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled.

Luigi reported there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

try_to_free_pages disable may_writepage if laptop_mode is enabled. shrink_page_list adds lots of anon pages in swap cache by add_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.

In case of that, isolate_lru_pages fails because it try to isolate clean page due to may_writepage == 0.

The may_writepage could be 1 only if total_scanned is higher than writeback_threshold in do_try_to_free_pages but unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.

This patch prevents to add a page to swap cache unnecessary when may_writepage is unset so anoymous lru list isn't full of Dirty/Swapcache page. So VM can isolate pages from anon lru list, which ends up setting may_writepage to 1 and could swap out anon lru pages. When OOM triggers, I confirmed swap space was full.

...

Quoted 1 line · On Wed, 9 Jan 2013 15:21:13 +0900 Minchan Kim <[email protected]> wrote:

This changelog is quite hard to understand :(

Quoted 4 lines

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled.

Quoted 8 lines

Luigi reported there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

try_to_free_pages disable may_writepage if laptop_mode is enabled. shrink_page_list adds lots of anon pages in swap cache by add_to_swap, which makes pages Dirty and rotate them to head of inactive LRU without pageout. If it is repeated, inactive anon LRU is full of Dirty and SwapCache pages.

"Dirty and SwapCache" is ambigious. Does it mean "dirty pages and swapcache pages" or does it mean "dirty swapcache pages". The latter, I expect.

Quoted 5 lines

In case of that, isolate_lru_pages fails because it try to isolate clean page due to may_writepage == 0.

Quoted 5 lines

The may_writepage could be 1 only if total_scanned is higher than writeback_threshold in do_try_to_free_pages but unfortunately, VM can't isolate anon pages from inactive anon lru list by above reason and we already reclaimed all file-backed pages. So it ends up OOM killing.

Here, please expand upon "by above reason". Explain here exactly why scanning is unsuccessful.

Quoted 9 lines

This patch prevents to add a page to swap cache unnecessary when may_writepage is unset so anoymous lru list isn't full of Dirty/Swapcache page. So VM can isolate pages from anon lru list, which ends up setting may_writepage to 1 and could swap out anon lru pages. When OOM triggers, I confirmed swap space was full.

...

mm/vmscan.c static unsigned long shrink_page_list(struct list_head *page_list, · line 787

Quoted 11 lines

mm/vmscan.c +2 -0

@@ -780,6 +780,8 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 		if (PageAnon(page) && !PageSwapCache(page)) {
 			if (!(sc->gfp_mask & __GFP_IO))
 				goto keep_locked;
+			if (!sc->may_writepage)
+				goto keep_locked;
 			if (!add_to_swap(page))
 				goto activate_locked;
 			may_enter_fs = 1;

Needs a comment explaining why we bale out in this case, please.

If I'm understanding it correctly, this change causes the kernel to move less anonymous memory onto the inactive anon LRU and thereby

causes the scanner to be more successful in locating clean swapcache pages on that list? But that makes no sense, because from your description it appears the intent of the patch is to use *more* swap.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled. He said there was no problem when he disabled laptop_mode.

The problem when I investigate problem is following as.

Assumption for easy explanation: There are no page cache page in system because they all are already reclaimed.

1. try_to_free_pages disable may_writepage when laptop_mode is enabled.
2. shrink_inactive_list isolates victim pages from inactive anon lru list.
3. shrink_page_list adds them to swapcache via add_to_swap but it doesn't pageout because sc->may_writepage is 0 so the page is rotated back into inactive anon lru list. The add_to_swap made the page Dirty by SetPageDirty
4. 3 couldn't reclaim any pages so do_try_to_free_pages increase priority and retry reclaim with higher priority.
5. shrink_inactlive_list try to isolate victim pages from inactive anon lru list but got failed because it try to isolate pages with ISOLATE_CLEAN mode but inactive anon lru list is full of dirty pages by 3 so it just returns without any reclaim progress.
6. do_try_to_free_pages doesn't set may_write due to zero total_scanned.

Because sc->nr_scanned is increased by shrink_page_list but we don't call shrink_page_list in 5 due to short of isolated pages.

Above loop is continued until OOM happens.
The problem didn't happen before [1] was merged because old logic's isolatation in shrink_inactive_list was successful and tried to call shrink_page_list to pageout them but it still ends up failed to page out by may_writepage. But important point is that sc->nr_scanned was increased althoug we couldn't swap out them so do_try_to_free_pages could set may_writepages.
So this patch need to go stable tree althoug it's a band-aid. Then, for latest linus tree, we should fix laptop_mode's fundamental problem.

[1] f80c067[mm: zone_reclaim: make isolate_lru_page() filter-aware]

Quoted 10 lines · On Thu, 17 Jan 2013 09:53:14 +0900 Minchan Kim <[email protected]> wrote:

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled. He said there was no problem when he disabled laptop_mode.

The problem when I investigate problem is following as.

Assumption for easy explanation: There are no page cache page in system because they all are already reclaimed.

Quoted 12 lines

1. try_to_free_pages disable may_writepage when laptop_mode is enabled.
2. shrink_inactive_list isolates victim pages from inactive anon lru list.
3. shrink_page_list adds them to swapcache via add_to_swap but it doesn't pageout because sc->may_writepage is 0 so the page is rotated back into inactive anon lru list. The add_to_swap made the page Dirty by SetPageDirty
4. 3 couldn't reclaim any pages so do_try_to_free_pages increase priority and retry reclaim with higher priority.
5. shrink_inactlive_list try to isolate victim pages from inactive anon lru list but got failed because it try to isolate pages with ISOLATE_CLEAN mode but inactive anon lru list is full of dirty pages by 3 so it just returns without any reclaim progress.
6. do_try_to_free_pages doesn't set may_write due to zero total_scanned.

s/may_write/may_writepage/

Quoted 3 lines

Because sc->nr_scanned is increased by shrink_page_list but we don't call shrink_page_list in 5 due to short of isolated pages.

This is the bug, is it not?

In laptop mode, we still need to write out dirty swapcache at some point. An appropriate time to do this is when the scanning priority is

getting high. But it seems that this ISOLATE_CLEAN->total_scanned

interaction is preventing that.

(An enhancement to laptop mode would be to opportunistically write out dirty swapcache in or around laptop_mode_timer_fn()).

Quoted 10 lines

Above loop is continued until OOM happens.
The problem didn't happen before [1] was merged because old logic's isolatation in shrink_inactive_list was successful and tried to call shrink_page_list to pageout them but it still ends up failed to page out by may_writepage. But important point is that sc->nr_scanned was increased althoug we couldn't swap out them so do_try_to_free_pages could set may_writepages.
So this patch need to go stable tree althoug it's a band-aid. Then, for latest linus tree, we should fix laptop_mode's fundamental problem.

Well. Perhaps we can do that now.

Quoted 2 lines

[1] f80c067[mm: zone_reclaim: make isolate_lru_page() filter-aware]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 26 lines · Andrew Morton

Quoted 10 lines · On Thu, 17 Jan 2013 09:53:14 +0900 Minchan Kim <[email protected]> wrote:

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled. He said there was no problem when he disabled laptop_mode.

The problem when I investigate problem is following as.

Assumption for easy explanation: There are no page cache page in system because they all are already reclaimed.

Quoted 12 lines

1. try_to_free_pages disable may_writepage when laptop_mode is enabled.
2. shrink_inactive_list isolates victim pages from inactive anon lru list.
3. shrink_page_list adds them to swapcache via add_to_swap but it doesn't pageout because sc->may_writepage is 0 so the page is rotated back into inactive anon lru list. The add_to_swap made the page Dirty by SetPageDirty
4. 3 couldn't reclaim any pages so do_try_to_free_pages increase priority and retry reclaim with higher priority.
5. shrink_inactlive_list try to isolate victim pages from inactive anon lru list but got failed because it try to isolate pages with ISOLATE_CLEAN mode but inactive anon lru list is full of dirty pages by 3 so it just returns without any reclaim progress.
6. do_try_to_free_pages doesn't set may_write due to zero total_scanned.

Quoted 1 line

s/may_write/may_writepage/

Thanks!

Quoted 8 lines

Quoted 3 lines

Because sc->nr_scanned is increased by shrink_page_list but we don't call shrink_page_list in 5 due to short of isolated pages.

This is the bug, is it not?

In laptop mode, we still need to write out dirty swapcache at some point. An appropriate time to do this is when the scanning priority is

Yes and when to some point is really important. Now, the point for that is depends on on the number of scanned pages by shrink_page_list. It means we must isolate victim pages from inactive LRU list and call shrink_page_list to increase sc->nr_scanned but unfortunately, we have various filters to decrease CPU consumption and LRU churning when VM try to isolate victim pages so it could prevent isolating victim pages from LRU list.

Quoted 1 line

getting high. But it seems that this ISOLATE_CLEAN->total_scanned

Yes. I absolutely agree on that some point should depend on priority, NOT the number of scanned pages. And I already said to you about that.
https://lkml.org/lkml/2013/1/10/643

We used to use such heuristic in several places in VM, ie DEF_PRIORITY - 2 But why I hesitate with the patch is that I think this patch should go to stable tree so the patch should be really small and have no side effect so I don't wanted to change laptop_mode behavior heavily caused by changing condition for may_writepage trigger point.

Quoted 4 lines

interaction is preventing that.

(An enhancement to laptop mode would be to opportunistically write out dirty swapcache in or around laptop_mode_timer_fn()).

It could but it should be another patch and VM shouldn't rely on ONLY laptop_mode_timer_fn, IMHO. VM should have own rule to reclaim pages regardless of laptop_mode's help to prevent OOM kill.

Quoted 11 lines

Quoted 10 lines

Above loop is continued until OOM happens.
The problem didn't happen before [1] was merged because old logic's isolatation in shrink_inactive_list was successful and tried to call shrink_page_list to pageout them but it still ends up failed to page out by may_writepage. But important point is that sc->nr_scanned was increased althoug we couldn't swap out them so do_try_to_free_pages could set may_writepages.
So this patch need to go stable tree althoug it's a band-aid. Then, for latest linus tree, we should fix laptop_mode's fundamental problem.

Quoted 1 line

Well. Perhaps we can do that now.

Okay. If you don't object my suggestion, I will send patches next week. Thanks for the review, Andrew!

From 94086dc7152359d052802c55c82ef19509fe8cce Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Mon, 21 Jan 2013 10:43:43 +0900
Subject: [PATCH] mm: Use up free swap space before reaching OOM kill

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled. He said there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

Assumption for easy explanation: There are no page cache page in system because they all are already reclaimed.

1. try_to_free_pages disable may_writepage when laptop_mode is enabled.
2. shrink_inactive_list isolates victim pages from inactive anon lru list.
3. shrink_page_list adds them to swapcache via add_to_swap but it doesn't pageout because sc->may_writepage is 0 so the page is rotated back into inactive anon lru list. The add_to_swap made the page Dirty by SetPageDirty.
4. 3 couldn't reclaim any pages so do_try_to_free_pages increase priority and retry reclaim with higher priority.
5. shrink_inactlive_list try to isolate victim pages from inactive anon lru list but got failed because it try to isolate pages with ISOLATE_CLEAN mode but inactive anon lru list is full of dirty pages by 3 so it just returns without any reclaim progress.
6. do_try_to_free_pages doesn't set may_writepage due to zero total_scanned. Because sc->nr_scanned is increased by shrink_page_list but we don't call shrink_page_list in 5 due to short of isolated pages.

Above loop is continued until OOM happens.
The problem didn't happen before [1] was merged because old logic's isolatation in shrink_inactive_list was successful and tried to call shrink_page_list to pageout them but it still ends up failed to page out by may_writepage. But important point is that sc->nr_scanned was increased although we couldn't swap out them so do_try_to_free_pages could set may_writepages.

Since [1] was introduced, it's not a good idea any more to depends on only the number of scanned pages for setting may_writepage. So this patch adds new trigger point of setting may_writepage as below DEF_PRIOIRTY - 2 which is used to show the significant memory pressure in VM so it's good fit for our purpose which would be better to lose power saving or clickety rather than OOM killing.

[1] f80c067[mm: zone_reclaim: make isolate_lru_page() filter-aware]

Reported-by Luigi Semenzato <[email protected]>

Signed-off-by Minchan Kim <[email protected]>

Quoted 44 lines · Minchan Kim

From 94086dc7152359d052802c55c82ef19509fe8cce Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Mon, 21 Jan 2013 10:43:43 +0900
Subject: [PATCH] mm: Use up free swap space before reaching OOM kill

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled. He said there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

Assumption for easy explanation: There are no page cache page in system because they all are already reclaimed.

1. try_to_free_pages disable may_writepage when laptop_mode is enabled.
2. shrink_inactive_list isolates victim pages from inactive anon lru list.
3. shrink_page_list adds them to swapcache via add_to_swap but it doesn't pageout because sc->may_writepage is 0 so the page is rotated back into inactive anon lru list. The add_to_swap made the page Dirty by SetPageDirty.
4. 3 couldn't reclaim any pages so do_try_to_free_pages increase priority and retry reclaim with higher priority.
5. shrink_inactlive_list try to isolate victim pages from inactive anon lru list but got failed because it try to isolate pages with ISOLATE_CLEAN mode but inactive anon lru list is full of dirty pages by 3 so it just returns without any reclaim progress.
6. do_try_to_free_pages doesn't set may_writepage due to zero total_scanned. Because sc->nr_scanned is increased by shrink_page_list but we don't call shrink_page_list in 5 due to short of isolated pages.

Above loop is continued until OOM happens.
The problem didn't happen before [1] was merged because old logic's isolatation in shrink_inactive_list was successful and tried to call shrink_page_list to pageout them but it still ends up failed to page out by may_writepage. But important point is that sc->nr_scanned was increased although we couldn't swap out them so do_try_to_free_pages could set may_writepages.

Since [1] was introduced, it's not a good idea any more to depends on only the number of scanned pages for setting may_writepage. So this patch adds new trigger point of setting may_writepage as below DEF_PRIOIRTY - 2 which is used to show the significant memory pressure in VM so it's good fit for our purpose which would be better to lose power saving or clickety rather than OOM killing.

Quoted 4 lines

[1] f80c067[mm: zone_reclaim: make isolate_lru_page() filter-aware]

Reported-by Luigi Semenzato <[email protected]>

Signed-off-by Minchan Kim <[email protected]>

Your patch is a nice simplification. I am ok with the change, provided it works for Luigi :)

Acked-by Rik van Riel <[email protected]>

Signature · 7 lines

All rights reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 51 lines · Rik van Riel

Quoted 48 lines · Minchan Kim

From 94086dc7152359d052802c55c82ef19509fe8cce Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Mon, 21 Jan 2013 10:43:43 +0900
Subject: [PATCH] mm: Use up free swap space before reaching OOM kill

Recently, Luigi reported there are lots of free swap space when OOM happens. It's easily reproduced on zram-over-swap, where many instance of memory hogs are running and laptop_mode is enabled. He said there was no problem when he disabled laptop_mode. The problem when I investigate problem is following as.

Assumption for easy explanation: There are no page cache page in system because they all are already reclaimed.

1. try_to_free_pages disable may_writepage when laptop_mode is enabled.
2. shrink_inactive_list isolates victim pages from inactive anon lru list.
3. shrink_page_list adds them to swapcache via add_to_swap but it doesn't pageout because sc->may_writepage is 0 so the page is rotated back into inactive anon lru list. The add_to_swap made the page Dirty by SetPageDirty.
4. 3 couldn't reclaim any pages so do_try_to_free_pages increase priority and retry reclaim with higher priority.
5. shrink_inactlive_list try to isolate victim pages from inactive anon lru list but got failed because it try to isolate pages with ISOLATE_CLEAN mode but inactive anon lru list is full of dirty pages by 3 so it just returns without any reclaim progress.
6. do_try_to_free_pages doesn't set may_writepage due to zero total_scanned. Because sc->nr_scanned is increased by shrink_page_list but we don't call shrink_page_list in 5 due to short of isolated pages.

Above loop is continued until OOM happens.
The problem didn't happen before [1] was merged because old logic's isolatation in shrink_inactive_list was successful and tried to call shrink_page_list to pageout them but it still ends up failed to page out by may_writepage. But important point is that sc->nr_scanned was increased although we couldn't swap out them so do_try_to_free_pages could set may_writepages.

Since [1] was introduced, it's not a good idea any more to depends on only the number of scanned pages for setting may_writepage. So this patch adds new trigger point of setting may_writepage as below DEF_PRIOIRTY - 2 which is used to show the significant memory pressure in VM so it's good fit for our purpose which would be better to lose power saving or clickety rather than OOM killing.

[1] f80c067[mm: zone_reclaim: make isolate_lru_page() filter-aware]

Reported-by Luigi Semenzato <[email protected]>

Signed-off-by Minchan Kim <[email protected]>

Quoted 2 lines

Your patch is a nice simplification. I am ok with the change, provided it works for Luigi :)

Thanks, Rik.

Oops, I missed to Ccing Luigi. Add him again.
Luigi, Could you test this patch?
Thanks for your endless effort.

Quoted 3 lines

Acked-by Rik van Riel <[email protected]>

If laptop_mode is enable, VM try to avoid I/O for saving the power. But if there isn't reclaimable memory without I/O, we should do I/O for preventing unnecessary OOM kill although we sacrifices power.

One of example is that we are out of page cache. Remained one is only anonymous pages, for swapping out, we needs may_writepage = 1.

Reported-by Luigi Semenzato <[email protected]>

Signed-off-by Minchan Kim <[email protected]>

---
 mm/vmscan.c |    6 ++++++
 1 file changed, 6 insertions(+)

mm/vmscan.c +0 -0

index 439cc47..624c816 100644
@@ -1728,6 +1728,12 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc,
 		free = zone_page_state(zone, NR_FREE_PAGES);
 		if (unlikely(file + free <= high_wmark_pages(zone))) {

Quoted 20 lines · On Wed, 9 Jan 2013 15:21:14 +0900 Minchan Kim <[email protected]> wrote:

If laptop_mode is enable, VM try to avoid I/O for saving the power. But if there isn't reclaimable memory without I/O, we should do I/O for preventing unnecessary OOM kill although we sacrifices power.

One of example is that we are out of page cache. Remained one is only anonymous pages, for swapping out, we needs may_writepage = 1.

Reported-by Luigi Semenzato <[email protected]>

Signed-off-by Minchan Kim <[email protected]>

---
 mm/vmscan.c |    6 ++++++
 1 file changed, 6 insertions(+)

mm/vmscan.c +0 -0

index 439cc47..624c816 100644
@@ -1728,6 +1728,12 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc,
 		free = zone_page_state(zone, NR_FREE_PAGES);
 		if (unlikely(file + free <= high_wmark_pages(zone))) {

mm/vmscan.c static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, · line 1739

Quoted 10 lines

mm/vmscan.c +6 -0

 			scan_balance = SCAN_ANON;
+			/*
+			 * From now on, we have to swap out
+			 * for peventing OOM kill although
+			 * we sacrifice power consumption.
+			 */
+			sc->may_writepage = 1;
 			goto out;
 		}
 	}

This is pretty ugly. get_scan_count() is, as its name implies, an idempotent function which inspects the state of things and returns a result. As such, it has no business going in and altering the state of the scan_control.

We have code in both direct reclaim and in kswapd to set may_writepage if vmscan is getting into trouble. I don't see why adding another instance is necessary if the existing instances are working correctly.

(Is it correct that __zone_reclaim() ignores laptop_mode?)

I have a feeling that laptop mode has bitrotted and these patches are kinda hacking around as-yet-not-understood failures...

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 2 lines

I have a feeling that laptop mode has bitrotted and these patches are kinda hacking around as-yet-not-understood failures...

Absolutely, this patch is last guard for unexpectable behavior. As I mentioned in cover-letter, Luigi's problem could be solved either [1/2] or [2/2] but I wanted to add this as last resort in case of unexpected emergency. But you're right. It's not good to hide the problem like this path so let's drop [2/2].

Also, I absolutely agree it has bitrotted so for correcting it, we need a volunteer who have to inverstigate power saveing experiment with long time. So [1/2] would be band-aid until that.

Quoted 9 lines · On Thu, 10 Jan 2013 11:23:06 +0900 Minchan Kim <[email protected]> wrote:

Quoted 2 lines

I have a feeling that laptop mode has bitrotted and these patches are kinda hacking around as-yet-not-understood failures...

Absolutely, this patch is last guard for unexpectable behavior. As I mentioned in cover-letter, Luigi's problem could be solved either [1/2] or [2/2] but I wanted to add this as last resort in case of unexpected emergency. But you're right. It's not good to hide the problem like this path so let's drop [2/2].

Quoted 3 lines

Also, I absolutely agree it has bitrotted so for correcting it, we need a volunteer who have to inverstigate power saveing experiment with long time. So [1/2] would be band-aid until that.

I'm inclined to hold off on 1/2 as well, really.

The point of laptop_mode isn't to save power btw - it is to minimise the frequency with which the disk drive is spun up. By deferring and then batching writeout operations, basically.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Andrew,

Quoted 12 lines · Andrew Morton

Quoted 9 lines · On Thu, 10 Jan 2013 11:23:06 +0900 Minchan Kim <[email protected]> wrote:

Quoted 2 lines

I have a feeling that laptop mode has bitrotted and these patches are kinda hacking around as-yet-not-understood failures...

Absolutely, this patch is last guard for unexpectable behavior. As I mentioned in cover-letter, Luigi's problem could be solved either [1/2] or [2/2] but I wanted to add this as last resort in case of unexpected emergency. But you're right. It's not good to hide the problem like this path so let's drop [2/2].

Quoted 5 lines

Quoted 3 lines

Also, I absolutely agree it has bitrotted so for correcting it, we need a volunteer who have to inverstigate power saveing experiment with long time. So [1/2] would be band-aid until that.

I'm inclined to hold off on 1/2 as well, really.

Then, what's your plan?

Quoted 4 lines

The point of laptop_mode isn't to save power btw - it is to minimise the frequency with which the disk drive is spun up. By deferring and then batching writeout operations, basically.

I don't get it. Why should we minimise such frequency?

It's for saving the power to increase batter life.

As I real all document about laptop_mode, they all said about the power or battery life saving.

1. Documentation/laptops/laptop-mode.txt
2. http://linux.die.net/man/8/laptop_mode
3. http://samwel.tk/laptop_mode/
3. http://www.thinkwiki.org/wiki/Laptop-mode

Quoted 19 lines · On Fri, 11 Jan 2013 13:43:27 +0900 Minchan Kim <[email protected]> wrote:

Hi Andrew,

Quoted 16 lines · Andrew Morton

Quoted 12 lines · On Thu, 10 Jan 2013 11:23:06 +0900 Minchan Kim <[email protected]> wrote:

Quoted 2 lines

I have a feeling that laptop mode has bitrotted and these patches are kinda hacking around as-yet-not-understood failures...

Absolutely, this patch is last guard for unexpectable behavior. As I mentioned in cover-letter, Luigi's problem could be solved either [1/2] or [2/2] but I wanted to add this as last resort in case of unexpected emergency. But you're right. It's not good to hide the problem like this path so let's drop [2/2].

Also, I absolutely agree it has bitrotted so for correcting it, we need a volunteer who have to inverstigate power saveing experiment with long time. So [1/2] would be band-aid until that.

Quoted 3 lines

Quoted 1 line

I'm inclined to hold off on 1/2 as well, really.

Then, what's your plan?

My plan is to sit here until someone gets down and fully tests and fixes laptop-mode. Making it work properly, reliably and as-designed.

Or perhaps someone wants to make the case that we just don't need it any more (SSDs are silent!) and removes it all.

Quoted 1 line

Quoted 1 line

Quoted 5 lines

Quoted 3 lines

The point of laptop_mode isn't to save power btw - it is to minimise the frequency with which the disk drive is spun up. By deferring and then batching writeout operations, basically.

I don't get it. Why should we minimise such frequency?

Because my laptop was going clickety every minute and was keeping me awake.

Quoted 1 line

It's for saving the power to increase batter life.

It might well have that effect, dunno. That wasn't my intent. Testing needed!

Quoted 7 lines

As I real all document about laptop_mode, they all said about the power or battery life saving.

1. Documentation/laptops/laptop-mode.txt
2. http://linux.die.net/man/8/laptop_mode
3. http://samwel.tk/laptop_mode/
3. http://www.thinkwiki.org/wiki/Laptop-mode

Documentation creep ;)

Ten years ago, gad: http://lwn.net/Articles/1652/

Quoted 5 lines

Quoted 1 line

It's for saving the power to increase batter life.

It might well have that effect, dunno. That wasn't my intent. Testing needed!

Power saving is certainly why we had it on originally for ChromeOS, but we turned it off due to misbehavior.

Specifically, we saw a pathological behavior where we'd end up writing to the disk every few seconds when laptop mode was turned on. This turned out to be because laptop-mode sets a timer which is used to check for new dirty data after the initial flush and writes that out before spinning the disk down, and on ChromeOS various chatty daemons on the system were logging and dirtying data more or less constantly so there was almost always something there to be written out. So what ended up happening was that we'd need to do a read, then wake up the disk, and then keep writing every few seconds for a long period of time, which had the opposite effect from what we wanted.

The issues
with zram swap just confirmed that we didn't want laptop mode.

Most of our devices have had SSDs rather than spinning disks, so noise wasn't an issue, although when we finally did support an official device with a spinning disk people certainly complained when the disk started clicking all the time

(due to the underflow in the writeback code).

We do know that current SSDs save a significant amount of power when they go into standby, so minimizing disk writes is still useful on these devices.

A very simple laptop mode which only does a single sync when we spin up the disk, and didn't bother with the timer behavior or muck with swap behavior might be something that is more useful for us, and I suspect it might simplify the writeback code somewhat as well.

Quoted 9 lines · On Tue, 15 Jan 2013 16:32:38 -0800 Sonny Rao <[email protected]> wrote:

Quoted 5 lines

Quoted 1 line

It's for saving the power to increase batter life.

It might well have that effect, dunno. That wasn't my intent. Testing needed!

Power saving is certainly why we had it on originally for ChromeOS, but we turned it off due to misbehavior.

Quoted 10 lines

Specifically, we saw a pathological behavior where we'd end up writing to the disk every few seconds when laptop mode was turned on. This turned out to be because laptop-mode sets a timer which is used to check for new dirty data after the initial flush and writes that out before spinning the disk down, and on ChromeOS various chatty daemons on the system were logging and dirtying data more or less constantly so there was almost always something there to be written out. So what ended up happening was that we'd need to do a read, then wake up the disk, and then keep writing every few seconds for a long period of time, which had the opposite effect from what we wanted.

So after the read, the disk would chatter away doing a dribble of writes? That sounds like plain brokenness (and why did the chrome guys not tell anyone about it?!?!?).

The idea is that when the physical
read occurs, we should opportunistically flush out all pending writes, while the disk is running. Then go back into buffer-writes-for-a-long-time mode.

I forget what we did with fsync() and friends. Quite a lot of pestiferous applications like to do fsync quite frequently. I had a special kernel in which fsync() consisted of "return 0;", but ISTR there being some resistance to productizing that idea.

Quoted 7 lines

The issues
with zram swap just confirmed that we didn't want laptop mode.

Most of our devices have had SSDs rather than spinning disks, so noise wasn't an issue, although when we finally did support an official device with a spinning disk people certainly complained when the disk started clicking all the time

hm, it's interesting that the general idea still has vailidity. It would be a fun project for someone to sniff out all the requirements, fixup/enhance/rewrite the current implementation and generally make it all spiffy and nice.

Quoted 1 line

(due to the underflow in the writeback code).

To what underflow do you refer?

Quoted 8 lines

We do know that current SSDs save a significant amount of power when they go into standby, so minimizing disk writes is still useful on these devices.

A very simple laptop mode which only does a single sync when we spin up the disk, and didn't bother with the timer behavior or muck with swap behavior might be something that is more useful for us, and I suspect it might simplify the writeback code somewhat as well.

I don't think I understand the problem with the timer. My original RFC said

: laptop_writeback_centisecs
: --------------------------
:
: This tunable determines the maximum age of dirty data when the machine : is operating in Laptop mode. The default value is 30000 - five : minutes. This means that if applications are generating a small amount : of write traffic, the disk will spin up once per five minutes. :
: If the disk is spun up for any other reason (such as for a read) then : all dirty data will be flushed anyway, and this timer is reset to zero.

which all sounds very sensible and shouldn't exhibit the behavior you observed.

Quoted 23 lines · Andrew Morton

Quoted 9 lines · On Tue, 15 Jan 2013 16:32:38 -0800 Sonny Rao <[email protected]> wrote:

Quoted 5 lines

Quoted 1 line

It's for saving the power to increase batter life.

It might well have that effect, dunno. That wasn't my intent. Testing needed!

Power saving is certainly why we had it on originally for ChromeOS, but we turned it off due to misbehavior.

Quoted 10 lines

Specifically, we saw a pathological behavior where we'd end up writing to the disk every few seconds when laptop mode was turned on. This turned out to be because laptop-mode sets a timer which is used to check for new dirty data after the initial flush and writes that out before spinning the disk down, and on ChromeOS various chatty daemons on the system were logging and dirtying data more or less constantly so there was almost always something there to be written out. So what ended up happening was that we'd need to do a read, then wake up the disk, and then keep writing every few seconds for a long period of time, which had the opposite effect from what we wanted.

Quoted 3 lines

So after the read, the disk would chatter away doing a dribble of writes? That sounds like plain brokenness (and why did the chrome guys not tell anyone about it?!?!?).

Yes, either read or fsync. I ranted about it a little (here:
http://marc.info/?l=linux-mm&m=135422986220016&w=4), but mostly assumed it was working as expected, and that ChromeOS was just dirtying data at an absurd pace. Might have been a bad assumption and I could have been more explicit about reporting it, sorry about that.

Quoted 5 lines

The idea is that when the physical
read occurs, we should opportunistically flush out all pending writes, while the disk is running. Then go back into buffer-writes-for-a-long-time mode.

See the comment in page-writeback.c above laptop_io_completion():

/*
* We've spun up the disk and we're in laptop mode: schedule writeback
* of all dirty data a few seconds from now. If the flush is already
scheduled
* then push it back - the user is still using the disk.
*/
void laptop_io_completion(struct backing_dev_info *info)

What ends up happening fairly often is that there's always something dirty with that few seconds (or even one second) on our system.

Quoted 5 lines

I forget what we did with fsync() and friends. Quite a lot of pestiferous applications like to do fsync quite frequently. I had a special kernel in which fsync() consisted of "return 0;", but ISTR there being some resistance to productizing that idea.

Yeah, we have this problem and we try to fix up users of fsync() as we find them but it's a bit of a never-ending battle. Such a feature would be useful.

Quoted 8 lines

Quoted 7 lines

The issues
with zram swap just confirmed that we didn't want laptop mode.

Most of our devices have had SSDs rather than spinning disks, so noise wasn't an issue, although when we finally did support an official device with a spinning disk people certainly complained when the disk started clicking all the time

Quoted 9 lines

hm, it's interesting that the general idea still has vailidity. It would be a fun project for someone to sniff out all the requirements, fixup/enhance/rewrite the current implementation and generally make it all spiffy and nice.

Quoted 1 line

(due to the underflow in the writeback code).

To what underflow do you refer?

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=c8b74c2f6604923de91f8aa6539f8bb934736754

That particular bug caused writes to happen almost instantly after the underflow ocurred, and consequently slowed write throughput to a crawl because there was no chance for contiguous writes to gather.

Quoted 23 lines

Quoted 8 lines

We do know that current SSDs save a significant amount of power when they go into standby, so minimizing disk writes is still useful on these devices.

A very simple laptop mode which only does a single sync when we spin up the disk, and didn't bother with the timer behavior or muck with swap behavior might be something that is more useful for us, and I suspect it might simplify the writeback code somewhat as well.

I don't think I understand the problem with the timer. My original RFC said

: laptop_writeback_centisecs
: --------------------------
:
: This tunable determines the maximum age of dirty data when the machine : is operating in Laptop mode. The default value is 30000 - five : minutes. This means that if applications are generating a small amount : of write traffic, the disk will spin up once per five minutes. :
: If the disk is spun up for any other reason (such as for a read) then : all dirty data will be flushed anyway, and this timer is reset to zero.

Quoted 3 lines

which all sounds very sensible and shouldn't exhibit the behavior you observed.

The laptop-mode timer get re-armed after each writeback (see above laptop_io_completion function), even if it was caused by laptop-mode itself. So, if something is continually dirtying a little bit of data, we end up getting a chain of small writes which keeps the disk awake for long periods of time.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 114 lines · Sonny Rao

Quoted 23 lines · Andrew Morton

Quoted 9 lines · On Tue, 15 Jan 2013 16:32:38 -0800 Sonny Rao <[email protected]> wrote:

Quoted 5 lines

Quoted 1 line

It's for saving the power to increase batter life.

It might well have that effect, dunno. That wasn't my intent. Testing needed!

Power saving is certainly why we had it on originally for ChromeOS, but we turned it off due to misbehavior.

Quoted 10 lines

Specifically, we saw a pathological behavior where we'd end up writing to the disk every few seconds when laptop mode was turned on. This turned out to be because laptop-mode sets a timer which is used to check for new dirty data after the initial flush and writes that out before spinning the disk down, and on ChromeOS various chatty daemons on the system were logging and dirtying data more or less constantly so there was almost always something there to be written out. So what ended up happening was that we'd need to do a read, then wake up the disk, and then keep writing every few seconds for a long period of time, which had the opposite effect from what we wanted.

Quoted 3 lines

So after the read, the disk would chatter away doing a dribble of writes? That sounds like plain brokenness (and why did the chrome guys not tell anyone about it?!?!?).

Yes, either read or fsync. I ranted about it a little (here:
http://marc.info/?l=linux-mm&m=135422986220016&w=4), but mostly assumed it was working as expected, and that ChromeOS was just dirtying data at an absurd pace. Might have been a bad assumption and I could have been more explicit about reporting it, sorry about that.

Quoted 5 lines

The idea is that when the physical
read occurs, we should opportunistically flush out all pending writes, while the disk is running. Then go back into buffer-writes-for-a-long-time mode.

See the comment in page-writeback.c above laptop_io_completion():

/*
* We've spun up the disk and we're in laptop mode: schedule writeback
* of all dirty data a few seconds from now. If the flush is already
scheduled
* then push it back - the user is still using the disk.
*/
void laptop_io_completion(struct backing_dev_info *info)

What ends up happening fairly often is that there's always something dirty with that few seconds (or even one second) on our system.

Quoted 5 lines

I forget what we did with fsync() and friends. Quite a lot of pestiferous applications like to do fsync quite frequently. I had a special kernel in which fsync() consisted of "return 0;", but ISTR there being some resistance to productizing that idea.

Yeah, we have this problem and we try to fix up users of fsync() as we find them but it's a bit of a never-ending battle. Such a feature would be useful.

Quoted 8 lines

Quoted 7 lines

The issues
with zram swap just confirmed that we didn't want laptop mode.

Most of our devices have had SSDs rather than spinning disks, so noise wasn't an issue, although when we finally did support an official device with a spinning disk people certainly complained when the disk started clicking all the time

Quoted 9 lines

hm, it's interesting that the general idea still has vailidity. It would be a fun project for someone to sniff out all the requirements, fixup/enhance/rewrite the current implementation and generally make it all spiffy and nice.

Quoted 1 line

(due to the underflow in the writeback code).

To what underflow do you refer?

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=c8b74c2f6604923de91f8aa6539f8bb934736754

That particular bug caused writes to happen almost instantly after the underflow ocurred, and consequently slowed write throughput to a crawl because there was no chance for contiguous writes to gather.

Quoted 26 lines

Quoted 8 lines

We do know that current SSDs save a significant amount of power when they go into standby, so minimizing disk writes is still useful on these devices.

A very simple laptop mode which only does a single sync when we spin up the disk, and didn't bother with the timer behavior or muck with swap behavior might be something that is more useful for us, and I suspect it might simplify the writeback code somewhat as well.

I don't think I understand the problem with the timer. My original RFC said

: laptop_writeback_centisecs
: --------------------------
:
: This tunable determines the maximum age of dirty data when the machine : is operating in Laptop mode. The default value is 30000 - five : minutes. This means that if applications are generating a small amount : of write traffic, the disk will spin up once per five minutes. :
: If the disk is spun up for any other reason (such as for a read) then : all dirty data will be flushed anyway, and this timer is reset to zero.

which all sounds very sensible and shouldn't exhibit the behavior you observed.

Quoted 5 lines

The laptop-mode timer get re-armed after each writeback (see above laptop_io_completion function), even if it was caused by laptop-mode itself. So, if something is continually dirtying a little bit of data, we end up getting a chain of small writes which keeps the disk awake for long periods of time.

Out of curiosity, for saving the power, why don' you increase the value for laptop_mode?

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
Kind regards,
Minchan Kim

Quoted 15 lines · On Fri, 11 Jan 2013 13:43:27 +0900 Minchan Kim <[email protected]> wrote:

Hi Andrew,

Quoted 12 lines · Andrew Morton

Quoted 9 lines · On Thu, 10 Jan 2013 11:23:06 +0900 Minchan Kim <[email protected]> wrote:

Quoted 2 lines

I have a feeling that laptop mode has bitrotted and these patches are kinda hacking around as-yet-not-understood failures...

Absolutely, this patch is last guard for unexpectable behavior. As I mentioned in cover-letter, Luigi's problem could be solved either [1/2] or [2/2] but I wanted to add this as last resort in case of unexpected emergency. But you're right. It's not good to hide the problem like this path so let's drop [2/2].

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

Also, I absolutely agree it has bitrotted so for correcting it, we need a volunteer who have to inverstigate power saveing experiment with long time. So [1/2] would be band-aid until that.

I'm inclined to hold off on 1/2 as well, really.

Then, what's your plan?

My plan is to sit here until someone gets down and fully tests and fixes laptop-mode. Making it work properly, reliably and as-designed.

Or perhaps someone wants to make the case that we just don't need it any more (SSDs are silent!) and removes it all.

Quoted 6 lines

Quoted 4 lines

The point of laptop_mode isn't to save power btw - it is to minimise the frequency with which the disk drive is spun up. By deferring and then batching writeout operations, basically.

I don't get it. Why should we minimise such frequency?

Because my laptop was going clickety every minute and was keeping me awake.

Quoted 1 line

It's for saving the power to increase batter life.

It might well have that effect, dunno. That wasn't my intent. Testing needed!

Quoted 3 lines

As I real all document about laptop_mode, they all said about the power or battery life saving.

Quoted 4 lines

1. Documentation/laptops/laptop-mode.txt
2. http://linux.die.net/man/8/laptop_mode
3. http://samwel.tk/laptop_mode/
3. http://www.thinkwiki.org/wiki/Laptop-mode

Documentation creep ;)

Ten years ago, gad: http://lwn.net/Articles/1652/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>