mm: fix race condition in MADV_FREE

PATCH Superseded Shaohua Li 3 participants 7 messages Sep 22, 2017 – Sep 27, 2017

Subsystems (3)

Stable branch Memory management MM swap

lore

Share & tools

lore

Shaohua Li

author

2017-09-22 18:46 UTC #0 lore raw

v2 cover mm: fix race condition in MADV_FREE

From: Shaohua Li <[email protected]>

Artem Savkov reported a race condition[1] in MADV_FREE. MADV_FREE clear pte dirty bit and then mark the page lazyfree. There is no lock to prevent the page is added to swap cache between these two steps by page reclaim. There are two problems:
- page in swapcache is marked lazyfree (clear SwapBacked). This confuses some code pathes, like page fault handling.
- The page is added into swapcache, and freed but the page isn't swapout because pte isn't dity. This will cause data corruption.

The patches will fix the issues.

Thanks,
Shaohua

V1->V2:
- dirty page in add_to_swap instead of in shrink_page_list as suggested by Minchan

Shaohua Li (2):
mm: avoid marking swap cached page as lazyfree
mm: fix data corruption caused by lazyfree page

 mm/swap.c       | 4 ++--
 mm/swap_state.c | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

Signature · 7 lines

2.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Shaohua Li

author

2017-09-22 18:46 UTC #1 lore raw

PATCH v2 1/2 mm: avoid marking swap cached page as lazyfree

From: Shaohua Li <[email protected]>

Reported-and-tested-by Artem Savkov <[email protected]>

Fix: 802a3a92ad7a(mm: reclaim MADV_FREE pages)

Signed-off-by Shaohua Li <[email protected]>

Cc [email protected]

Cc Johannes Weiner <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Hillf Danton <[email protected]>

Cc Minchan Kim <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Andrew Morton <[email protected]>

Reviewed-by Rik van Riel <[email protected]>

---
 mm/swap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

mm/swap.c +2 -2

index 9295ae9..a77d68f 100644
@@ -575,7 +575,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
 			    void *arg)
 {
 	if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) &&
-	    !PageUnevictable(page)) {
+	    !PageSwapCache(page) && !PageUnevictable(page)) {
 		bool active = PageActive(page);
 
 		del_page_from_lru_list(page, lruvec,

@@ -665,7 +665,7 @@ void deactivate_file_page(struct page *page)
 void mark_page_lazyfree(struct page *page)
 {
 	if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) &&
-	    !PageUnevictable(page)) {
+	    !PageSwapCache(page) && !PageUnevictable(page)) {
 		struct pagevec *pvec = &get_cpu_var(lru_lazyfree_pvecs);
 
 		get_page(page);

Signature · 7 lines

2.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Minchan Kim

2017-09-25 05:07 UTC #2 lore raw

Re: [PATCH V2 1/2] mm: avoid marking swap cached page as lazyfree

Quoted 7 lines · Shaohua Li

From: Shaohua Li <[email protected]>
MADV_FREE clears pte dirty bit and then marks the page lazyfree (clear SwapBacked). There is no lock to prevent the page is added to swap cache between these two steps by page reclaim. If the page is added to swap cache, marking the page lazyfree will confuse page fault if the page is reclaimed and refault.

If page is added to swapcache while it stays lru_lazyfree_pvec, it ends up having !PG_swapbacked, PG_swapcache and !PG_dirty. Most important thing is PG_dirty. Without it, VM will reclaim the page without *writeback* so we lose the data.

Although we prevent the page adding to swapcache, we lose the data unless we apply [2/2] so this patch alone doesn't fix the problem. That's why I said to you we don't need to separate patches.

Quoted 42 lines

Reported-and-tested-by Artem Savkov <[email protected]>
Fix: 802a3a92ad7a(mm: reclaim MADV_FREE pages)
Signed-off-by Shaohua Li <[email protected]>
Cc [email protected]
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hillf Danton <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Andrew Morton <[email protected]>
Reviewed-by Rik van Riel <[email protected]>
---
 mm/swap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/swap.c +2 -2
index 9295ae9..a77d68f 100644
@@ -575,7 +575,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
 			    void *arg)
 {
 	if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) &&
-	    !PageUnevictable(page)) {
+	    !PageSwapCache(page) && !PageUnevictable(page)) {
 		bool active = PageActive(page);
 
 		del_page_from_lru_list(page, lruvec,
@@ -665,7 +665,7 @@ void deactivate_file_page(struct page *page)
 void mark_page_lazyfree(struct page *page)
 {
 	if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) &&
-	    !PageUnevictable(page)) {
+	    !PageSwapCache(page) && !PageUnevictable(page)) {
 		struct pagevec *pvec = &get_cpu_var(lru_lazyfree_pvecs);
 
 		get_page(page);
Signature · 1 line
2.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Michal Hocko

2017-09-26 13:07 UTC #3 lore raw

Re: [PATCH V2 1/2] mm: avoid marking swap cached page as lazyfree

In reply to Shaohua Li

Quoted 7 lines · Shaohua Li

From: Shaohua Li <[email protected]>
MADV_FREE clears pte dirty bit and then marks the page lazyfree (clear SwapBacked). There is no lock to prevent the page is added to swap cache between these two steps by page reclaim. If the page is added to swap cache, marking the page lazyfree will confuse page fault if the page is reclaimed and refault.

Could you be more specific how exactly what kind of the confusion is the result? I suspect you are talking about VM_BUG_ON_PAGE in __add_to_swap_cache right?

I am also not sure how that would actually happen to be honest. If we raced with the reclaim then the page should have been isolated and so PageLRU is no longer true. Or am I missing something?

Quoted 41 lines

Reported-and-tested-by Artem Savkov <[email protected]>
Fix: 802a3a92ad7a(mm: reclaim MADV_FREE pages)
Signed-off-by Shaohua Li <[email protected]>
Cc [email protected]
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hillf Danton <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Andrew Morton <[email protected]>
Reviewed-by Rik van Riel <[email protected]>
---
 mm/swap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/swap.c +2 -2
index 9295ae9..a77d68f 100644
@@ -575,7 +575,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
 			    void *arg)
 {
 	if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) &&
-	    !PageUnevictable(page)) {
+	    !PageSwapCache(page) && !PageUnevictable(page)) {
 		bool active = PageActive(page);
 
 		del_page_from_lru_list(page, lruvec,
@@ -665,7 +665,7 @@ void deactivate_file_page(struct page *page)
 void mark_page_lazyfree(struct page *page)
 {
 	if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) &&
-	    !PageUnevictable(page)) {
+	    !PageSwapCache(page) && !PageUnevictable(page)) {
 		struct pagevec *pvec = &get_cpu_var(lru_lazyfree_pvecs);
 
 		get_page(page);
Signature · 1 line
2.9.5

Signature · 8 lines

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Michal Hocko

2017-09-27 07:16 UTC #4 lore raw

Re: [PATCH V2 1/2] mm: avoid marking swap cached page as lazyfree

[ups this got stuck in the outgoing queue]

Quoted 9 lines · Michal Hocko

Quoted 7 lines · Shaohua Li
From: Shaohua Li <[email protected]>
MADV_FREE clears pte dirty bit and then marks the page lazyfree (clear SwapBacked). There is no lock to prevent the page is added to swap cache between these two steps by page reclaim. If the page is added to swap cache, marking the page lazyfree will confuse page fault if the page is reclaimed and refault.

Quoted 3 lines

Could you be more specific how exactly what kind of the confusion is the result? I suspect you are talking about VM_BUG_ON_PAGE in __add_to_swap_cache right?

I completely mixed reclaim and the #PF path here

Quoted 3 lines

I am also not sure how that would actually happen to be honest. If we raced with the reclaim then the page should have been isolated and so PageLRU is no longer true. Or am I missing something?

And here I've completely missed that the swapcache page will go back to the LRU. Stupid me. Your new changelog [1] explained it all. Thanks and sorry for these stupid questions.

[1] http://lkml.kernel.org/r/6537ef3814398c0073630b03f176263bc81f0902.1506446061.git.shli@fb.com

Signature · 8 lines

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Shaohua Li

author

2017-09-22 18:46 UTC #5 lore raw

PATCH v2 2/2 mm: fix data corruption caused by lazyfree page

In reply to Shaohua Li

From: Shaohua Li <[email protected]>

MADV_FREE clears pte dirty bit and then marks the page lazyfree (clear SwapBacked). There is no lock to prevent the page is added to swap cache between these two steps by page reclaim. If page reclaim finds such page, it will simply add the page to swap cache without pageout the page to swap because the page is marked as clean. Next time, page fault will read data from the swap slot which doesn't have the original data, so we have a data corruption. To fix issue, we mark the page dirty and pageout the page.

However, we shouldn't dirty all pages which is clean and in swap cache. swapin page is swap cache and clean too. So we only dirty page which is added into swap cache in page reclaim, which shouldn't be swapin page. As Minchan suggested, simply dirty the page in add_to_swap can do the job.

Reported-by Artem Savkov <[email protected]>

Fix: 802a3a92ad7a(mm: reclaim MADV_FREE pages)

Signed-off-by Shaohua Li <[email protected]>

Cc [email protected]

Cc Johannes Weiner <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Hillf Danton <[email protected]>

Cc Minchan Kim <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Rik van Riel <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Andrew Morton <[email protected]>

---
 mm/swap_state.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

mm/swap_state.c +2 -1

index 71ce2d1..2d64ec4 100644
@@ -231,7 +231,7 @@ int add_to_swap(struct page *page)
 	 * deadlock in the swap out path.
 	 */
 	/*
-	 * Add it to the swap cache.
+	 * Add it to the swap cache and mark it dirty
 	 */
 	err = add_to_swap_cache(page, entry,
 			__GFP_HIGH|__GFP_NOMEMALLOC|__GFP_NOWARN);

@@ -242,6 +242,7 @@ int add_to_swap(struct page *page)
 		 * clear SWAP_HAS_CACHE flag.
 		 */
 		goto fail;
+	set_page_dirty(page);
 
 	return 1;

Signature · 7 lines

2.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Michal Hocko

Acked-by on v2 2/2

2017-09-26 13:13 UTC #6 lore raw

Re: [PATCH V2 2/2] mm: fix data corruption caused by lazyfree page

Quoted 19 lines · Shaohua Li

From: Shaohua Li <[email protected]>
MADV_FREE clears pte dirty bit and then marks the page lazyfree (clear SwapBacked). There is no lock to prevent the page is added to swap cache between these two steps by page reclaim. If page reclaim finds such page, it will simply add the page to swap cache without pageout the page to swap because the page is marked as clean. Next time, page fault will read data from the swap slot which doesn't have the original data, so we have a data corruption. To fix issue, we mark the page dirty and pageout the page.
However, we shouldn't dirty all pages which is clean and in swap cache. swapin page is swap cache and clean too. So we only dirty page which is added into swap cache in page reclaim, which shouldn't be swapin page. As Minchan suggested, simply dirty the page in add_to_swap can do the job.
Reported-by Artem Savkov <[email protected]>
Fix: 802a3a92ad7a(mm: reclaim MADV_FREE pages)

Quoted 10 lines

Signed-off-by Shaohua Li <[email protected]>
Cc [email protected]
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hillf Danton <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Andrew Morton <[email protected]>

OK the patch makes sense to me

Acked-by Michal Hocko <[email protected]>

Quoted 9 lines

---
 mm/swap_state.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

mm/swap_state.c +0 -0

index 71ce2d1..2d64ec4 100644

mm/swap_state.c int add_to_swap(struct page *page) · line 235

Quoted 6 lines

mm/swap_state.c +1 -1

@@ -231,7 +231,7 @@ int add_to_swap(struct page *page)
 	 * deadlock in the swap out path.
 	 */
 	/*
-	 * Add it to the swap cache.
+	 * Add it to the swap cache and mark it dirty
 	 */

this comment is simply useless. It doesn't explain why we need it and it is trivial to see what we do there. So I would remove it altogether or make it more useful explaning the MADV_FREE race which would be IMHO much better.

Quoted 13 lines

mm/swap_state.c +1 -0

 	err = add_to_swap_cache(page, entry,
 			__GFP_HIGH|__GFP_NOMEMALLOC|__GFP_NOWARN);

@@ -242,6 +242,7 @@ int add_to_swap(struct page *page)
 		 * clear SWAP_HAS_CACHE flag.
 		 */
 		goto fail;
+	set_page_dirty(page);
 
 	return 1;

Signature · 1 line

2.9.5

Signature · 8 lines

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>