Patch "mm, mlock, vmscan: no more skipping pagevecs" has been added to the 4.14-stable tree

This is a note to let you know that I've just added the patch titled

mm, mlock, vmscan: no more skipping pagevecs

to the 4.14-stable tree which can be found at:

The filename of the patch is:
mm-mlock-vmscan-no-more-skipping-pagevecs.patch
and it can be found in the queue-4.14 subdirectory.

If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.

From foo@baz Tue May 1 16:18:19 PDT 2018
From: Shakeel Butt <[email protected]>
Date: Wed, 21 Feb 2018 14:45:28 -0800
Subject: mm, mlock, vmscan: no more skipping pagevecs

From: Shakeel Butt <[email protected]>

[ Upstream commit 9c4e6b1a7027f102990c0395296015a812525f4d ]

When a thread mlocks an address space backed either by file pages which are currently not present in memory or swapped out anon pages (not in swapcache), a new page is allocated and added to the local pagevec (lru_add_pvec), I/O is triggered and the thread then sleeps on the page. On I/O completion, the thread can wake on a different CPU, the mlock syscall will then sets the PageMlocked() bit of the page but will not be able to put that page in unevictable LRU as the page is on the pagevec of a different CPU. Even on drain, that page will go to evictable LRU because the PageMlocked() bit is not checked on pagevec drain.

The page will eventually go to right LRU on reclaim but the LRU stats will remain skewed for a long time.

This patch puts all the pages, even unevictable, to the pagevecs and on the drain, the pages will be added on their LRUs correctly by checking their evictability. This resolves the mlocked pages on pagevec of other CPUs issue because when those pagevecs will be drained, the mlocked file pages will go to unevictable LRU. Also this makes the race with munlock easier to resolve because the pagevec drains happen in LRU lock.

However there is still one place which makes a page evictable and does PageLRU check on that page without LRU lock and needs special attention. TestClearPageMlocked() and isolate_lru_page() in clear_page_mlock().

#0: __pagevec_lru_add_fn #1: clear_page_mlock

SetPageLRU() if (!TestClearPageMlocked())
return
smp_mb() // <--required
// inside does PageLRU
if (!PageMlocked()) if (isolate_lru_page())
move to evictable LRU putback_lru_page()
else
move to unevictable LRU

In '#1', TestClearPageMlocked() provides full memory barrier semantics and thus the PageLRU check (inside isolate_lru_page) can not be reordered before it.

In '#0', without explicit memory barrier, the PageMlocked() check can be reordered before SetPageLRU(). If that happens, '#0' can put a page in unevictable LRU and '#1' might have just cleared the Mlocked bit of that page but fails to isolate as PageLRU fails as '#0' still hasn't set PageLRU bit of that page. That page will be stranded on the unevictable LRU.

There is one (good) side effect though. Without this patch, the pages allocated for System V shared memory segment are added to evictable LRUs even after shmctl(SHM_LOCK) on that segment. This patch will correctly put such pages to unevictable LRU.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Shakeel Butt <[email protected]>

Acked-by Vlastimil Babka <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc Huang Ying <[email protected]>

Cc Tim Chen <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Greg Thelen <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Balbir Singh <[email protected]>

Cc Minchan Kim <[email protected]>

Cc Shaohua Li <[email protected]>

Cc Jan Kara <[email protected]>

Cc Nicholas Piggin <[email protected]>

Cc Dan Williams <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Vlastimil Babka <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

Signed-off-by Linus Torvalds <[email protected]>

Signed-off-by Sasha Levin <[email protected]>

Signed-off-by Greg Kroah-Hartman <[email protected]>

---
  include/linux/swap.h |    2 -
  mm/mlock.c           |    6 +++
  mm/swap.c            |   82

mm/mlock.c +6 -0

@@ -64,6 +64,12 @@ void clear_page_mlock(struct page *page)
         mod_zone_page_state(page_zone(page), NR_MLOCK,
                             -hpage_nr_pages(page));
         count_vm_event(UNEVICTABLE_PGCLEARED);
+       /*
+        * The previous TestClearPageMlocked() corresponds to the smp_mb()
+        * in __pagevec_lru_add_fn().
+        *
+        * See __pagevec_lru_add_fn for more explanation.
+        */
         if (!isolate_lru_page(page)) {
                 putback_lru_page(page);
         } else {

mm/swap.c +0 -24

@@ -446,30 +446,6 @@ void lru_cache_add(struct page *page)
  }
 
  /**
- * add_page_to_unevictable_list - add a page to the unevictable list
- * @page:  the page to be added to the unevictable list
- *
- * Add page directly to its zone's unevictable list.  To avoid races with
- * tasks that might be making the page evictable, through eg. munlock,
- * munmap or exit, while it's not on the lru, we want to add the page
- * while it's locked or otherwise "invisible" to other tasks.  This is
- * difficult to do when using the pagevec cache, so bypass that.
- */
-void add_page_to_unevictable_list(struct page *page)
-{
-       struct pglist_data *pgdat = page_pgdat(page);
-       struct lruvec *lruvec;
-
-       spin_lock_irq(&pgdat->lru_lock);
-       lruvec = mem_cgroup_page_lruvec(page, pgdat);
-       ClearPageActive(page);
-       SetPageUnevictable(page);
-       SetPageLRU(page);
-       add_page_to_lru_list(page, lruvec, LRU_UNEVICTABLE);
-       spin_unlock_irq(&pgdat->lru_lock);
-}
-
-/**
   * lru_cache_add_active_or_unevictable
   * @page:  the page to be added to LRU
   * @vma:   vma in which page is mapped for determining reclaimability

mm/swap.c +0 -2

                                  void *arg)
  {
-       int file = page_is_file_cache(page);
-       int active = PageActive(page);

mm/swap.c +18 -1

+        * in an unevictable LRU.
+        */
+       smp_mb();
+
+       if (page_evictable(page)) {
+               lru = page_lru(page);
+               update_page_reclaim_stat(lruvec, page_is_file_cache(page),
+                                        PageActive(page));
+               if (was_unevictable)
+                       count_vm_event(UNEVICTABLE_PGRESCUED);
+       } else {
+               lru = LRU_UNEVICTABLE;
+               ClearPageActive(page);
+               SetPageUnevictable(page);
+               if (!was_unevictable)
+                       count_vm_event(UNEVICTABLE_PGCULLED);
+       }
+
         add_page_to_lru_list(page, lruvec, lru);
-       update_page_reclaim_stat(lruvec, file, active);
         trace_mm_lru_insertion(page, lru);
  }
 

mm/vmscan.c +0 -23

@@ -790,64 +790,7 @@ int remove_mapping(struct address_space
   */
  void putback_lru_page(struct page *page)
  {
-       bool is_unevictable;
-       int was_unevictable = PageUnevictable(page);
-
-       VM_BUG_ON_PAGE(PageLRU(page), page);
-
-redo:
-       ClearPageUnevictable(page);
-
-       if (page_evictable(page)) {
-               /*
-                * For evictable pages, we can use the cache.
-                * In event of a race, worst case is we end up with an
-                * unevictable page on [in]active list.
-                * We know how to handle that.
-                */
-               is_unevictable = false;
-               lru_cache_add(page);
-       } else {
-               /*
-                * Put unevictable pages directly on zone's unevictable
-                * list.
-                */
-               is_unevictable = true;

mm/vmscan.c +1 -19

-        * page is on unevictable list, it never be freed. To avoid that,
-        * check after we added it to the list, again.
-        */
-       if (is_unevictable && page_evictable(page)) {
-               if (!isolate_lru_page(page)) {
-                       put_page(page);
-                       goto redo;
-               }
-               /* This means someone else dropped this page from LRU
-                * So, it will be freed or putback to LRU again. There is
-                * nothing to do here.
-                */
-       }
-
-       if (was_unevictable && !is_unevictable)
-               count_vm_event(UNEVICTABLE_PGRESCUED);
-       else if (!was_unevictable && is_unevictable)
-               count_vm_event(UNEVICTABLE_PGCULLED);
-
+       lru_cache_add(page);
         put_page(page);         /* drop ref from isolate */
  }
 

Patches currently in stable-queue which might be from [email protected]

queue-4.14/mm-slab-memcg_link-the-slab-s-kmem_cache.patch queue-4.14/mm-mlock-vmscan-no-more-skipping-pagevecs.patch

Quoted 5 lines · On Tue, May 1, 2018 at 4:52 PM <[email protected]> wrote:

This is a note to let you know that I've just added the patch titled

mm, mlock, vmscan: no more skipping pagevecs

to the 4.14-stable tree which can be found at:

http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary

Quoted 3 lines

The filename of the patch is:
mm-mlock-vmscan-no-more-skipping-pagevecs.patch
and it can be found in the queue-4.14 subdirectory.

Quoted 2 lines

If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.

I would advise not to include this patch in the previous stable trees. This patch does not fix any critical bug and for some very specific microbenchmarks on very large machines, it has shown performance regression. So, I don't think it is worth the risk.