Skip to content

per memcg lru lock

Alex Shi cgroups linux-mm lkml 13 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v21 Latest19 patches 39 repliesIn review 15 of 19 applied
  2. v2020 patches 46 repliesSuperseded
  3. v1920 patches 10 repliesSuperseded
Earlier versions · 10
VersionPostedPatchesRepliesStatus
v1832 patches70 repliesSuperseded
v1721 patches61 repliesSuperseded
v1622 patches43 repliesSuperseded
v15 posting 2/220 of 21 patches2 repliesSuperseded
v15 posting 1/20 of 20 patches0 repliesSuperseded
v1420 patches15 repliesSuperseded
v1318 patches2 repliesSuperseded
v1216 patches4 repliesSuperseded
v1116 patches7 repliesSuperseded
v1015 patches0 repliesSuperseded

Patches in v21 15 of 19 applied

#Subject and filesSizeRepliesApplied
1/19mm/thp: move lru_add_page_tail func to huge_memory.cinclude/linux/swap.h mm/huge_memory.c mm/swap.c+30 -350 replies
2/19mm/thp: use head for head page in lru_add_page_tailmm/huge_memory.c+11 -120 repliesApplied
3/19mm/thp: Simplify lru_add_page_tail()mm/huge_memory.c+6 -140 repliesApplied
4/19mm/thp: narrow lru lockingmm/huge_memory.c+13 -120 repliesApplied
5/19mm/vmscan: remove unnecessary lruvec addingmm/vmscan.c+25 -131 replyApplied
6/19mm/rmap: stop store reordering issue on page->mappingmm/rmap.c+6 -13 replies
7/19mm: page_idle_get_page() does not need lru_lockmm/page_idle.c+0 -43 repliesApplied
8/19mm/memcg: add debug checking in lock_page_memcgmm/memcontrol.c+6 -00 repliesApplied
9/19mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fnmm/swap.c+23 -420 repliesApplied
10/19mm/lru: move lock into lru_note_costmm/swap.c mm/vmscan.c mm/workingset.c+4 -50 repliesApplied
11/19mm/vmscan: remove lruvec reget in move_pages_to_lrumm/vmscan.c+6 -10 repliesApplied
12/19mm/mlock: remove lru_lock on TestClearPageMlockedmm/mlock.c+5 -211 replyApplied
13/19mm/mlock: remove __munlock_isolate_lru_pagemm/mlock.c+9 -221 replyApplied
14/19mm/lru: introduce TestClearPageLRUinclude/linux/page-flags.h mm/mlock.c mm/vmscan.c+21 -223 repliesApplied
15/19mm/compaction: do page isolation first in compactioninclude/linux/swap.h mm/compaction.c mm/vmscan.c+56 -314 repliesApplied
16/19mm/swap.c: serialize memcg changes in pagevec_lru_move_fnmm/swap.c+35 -91 replyApplied
17/19mm/lru: replace pgdat lru_lock with lruvec lockinclude/linux/memcontrol.h include/linux/mmzone.h mm/compaction.c +7 more+270 -1267 replies
18/19mm/lru: introduce the relock_page_lruvec functioninclude/linux/memcontrol.h mm/mlock.c mm/swap.c +1 more+62 -463 replies
19/19mm/lru: revise the comments of lru_lockDocumentation/admin-guide/cgroup-v1/memcg_test.rst Documentation/admin-guide/cgroup-v1/memory.rst Documentation/trace/events-kmem.rst +6 more+50 -641 replyApplied

Reviews trailers per patch

All 19 patches have a Reviewed-by or Acked-by.

PatchKirill A. ShutemovHugh DickinsJohannes WeinerMatthew Wilcox (Oracle)Huang, YingVlastimil Babka
1/19R-b A-b A-b
2/19R-b A-b A-b R-b
3/19R-b A-b
4/19R-b A-b
5/19A-b A-b A-b
6/19A-b A-b
7/19A-b A-b A-b
8/19A-b A-b
9/19A-b A-b
10/19A-b A-b
11/19A-b A-b
12/19A-b A-b A-b
13/19A-b A-b A-b
14/19A-b A-b A-b
15/19A-b A-b A-b
16/19A-b A-b A-b
17/19A-b A-b A-b A-b A-b
18/19A-b A-b A-b A-b A-b A-b
19/19A-b A-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 6 reviewers.

Read the v21 thread

Cover letter [PATCH v21 00/19] per memcg lru lock

This version rebase on next/master 20201104, with much of Johannes's Acks and some changes according to Johannes comments. And add a new patch v21-0006-mm-rmap-stop-store-reordering-issue-on-page-mapp.patch to support v21-0007.

This patchset followed 2 memcg VM_WARN_ON_ONCE_PAGE patches which were added to -mm tree yesterday.

Many thanks for line by line review by Hugh Dickins, Alexander Duyck and Johannes Weiner.

So now this patchset includes 3 parts:
1, some code cleanup and minimum optimization as a preparation. 2, use TestCleanPageLRU as page isolation's precondition. 3, replace per node lru_lock with per memcg per node lru_lock.

Current lru_lock is one for each of node, pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per node lru_lock is clearly unscalable, pages on each of memcgs have to compete each others for a whole lru_lock. This patchset try to use per lruvec/memcg lru_lock to repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain.

Currently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for new atomic func TestClearPageLRU. So now isolating a page need both actions: TestClearPageLRU and hold the lru_lock.

The typical usage of this is isolate_migratepages_block() in compaction.c we have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and new lru_lock in it.

The above solution suggested by Johannes Weiner, and based on his new memcg charge path, then have this patchset. (Hugh Dickins tested and contributed much code from compaction fix to general code polish, thanks a lot!).

Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box on v18, which has no much different with this v20.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/

Thanks Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who give comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox, Alexander Duyck etc.

Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case. Thanks!

Alex Shi (16):
mm/thp: move lru_add_page_tail func to huge_memory.c mm/thp: use head for head page in lru_add_page_tail mm/thp: Simplify lru_add_page_tail()
mm/thp: narrow lru locking
mm/vmscan: remove unnecessary lruvec adding mm/rmap: stop store reordering issue on page->mapping mm/memcg: add debug checking in lock_page_memcg mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn mm/lru: move lock into lru_note_cost
mm/vmscan: remove lruvec reget in move_pages_to_lru mm/mlock: remove lru_lock on TestClearPageMlocked mm/mlock: remove __munlock_isolate_lru_page mm/lru: introduce TestClearPageLRU
mm/compaction: do page isolation first in compaction mm/swap.c: serialize memcg changes in pagevec_lru_move_fn mm/lru: replace pgdat lru_lock with lruvec lock

Alexander Duyck (1):
mm/lru: introduce the relock_page_lruvec function

Hugh Dickins (2):
mm: page_idle_get_page() does not need lru_lock
mm/lru: revise the comments of lru_lock

 Documentation/admin-guide/cgroup-v1/memcg_test.rst |  15 +-
 Documentation/admin-guide/cgroup-v1/memory.rst     |  21 +--
 Documentation/trace/events-kmem.rst                |   2 +-
 Documentation/vm/unevictable-lru.rst               |  22 +--
 include/linux/memcontrol.h                         | 110 +++++++++++
 include/linux/mm_types.h                           |   2 +-
 include/linux/mmzone.h                             |   6 +-
 include/linux/page-flags.h                         |   1 +
 include/linux/swap.h                               |   4 +-
 mm/compaction.c                                    |  94 +++++++---
 mm/filemap.c                                       |   4 +-
 mm/huge_memory.c                                   |  45 +++--
 mm/memcontrol.c                                    |  79 +++++++-
 mm/mlock.c                                         |  63 ++-----
 mm/mmzone.c                                        |   1 +
 mm/page_alloc.c                                    |   1 -
 mm/page_idle.c                                     |   4 -
 mm/rmap.c                                          |  11 +-
 mm/swap.c                                          | 208 ++++++++-------------
 mm/vmscan.c                                        | 207 ++++++++++----------
 mm/workingset.c                                    |   2 -
 21 files changed, 530 insertions(+), 372 deletions(-)
Signature · 1 line
1.8.3.1