mm/migrate: split source folio if it is on deferred split list
Subsystems (4)
PATCH v3 1/1 mm/migrate: split source folio if it is on deferred split list
From: Zi Yan <[email protected]>
If the source folio is on deferred split list, it is likely some subpages are not used. Split it before migration to avoid migrating unused subpages.
Commit 616b8371539a6 ("mm: thp: enable thp migration in generic path") did not check if a THP is on deferred split list before migration, thus, the destination THP is never put on deferred split list even if the source THP might be. The opportunity of reclaiming free pages in a partially mapped THP during deferred list scanning is lost, but no other harmful consequence is present[1].
From v2:
1. Split the source folio instead of migrating it (per Matthew Wilcox)[2].
From v1:
1. Used dst to get correct deferred split list after migration (per Ryan Roberts).
[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/
--- mm/huge_memory.c | 22 ------------------ mm/internal.h | 23 +++++++++++++++++++ mm/migrate.c | 60 +++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 72 insertions(+), 33 deletions(-)
mm/huge_memory.c +0 -22
@@ -766,28 +766,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)return pmd; } -#ifdef CONFIG_MEMCG -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct mem_cgroup *memcg =folio_memcg (folio); -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -if (memcg) -return &memcg->deferred_split_queue; -else -return &pgdat->deferred_split_queue; -} -#else -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -return &pgdat->deferred_split_queue; -} -#endif -void folio_prep_large_rmappable (struct folio *folio) {if (!folio || !folio_test_large (folio))
mm/internal.h +23 -0
@@ -1107,6 +1107,29 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_MEMCG +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct mem_cgroup *memcg =folio_memcg (folio); +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +if (memcg) +return &memcg->deferred_split_queue; +else +return &pgdat->deferred_split_queue; +} +#else +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +return &pgdat->deferred_split_queue; +} +#endif + +/* * mm/mmap.c */
mm/migrate.c +49 -11
74 lines not on this page · raw message
base-commit: 384df0d61b071bb7a38ae63bae9b59d088335911
Signature · 1 line
2.43.0
Hello,
Quoted 82 lines · On Tue, 19 Mar 2024 11:47:53 -0400 Zi Yan <[email protected]> wrote:
From: Zi Yan <[email protected]>
If the source folio is on deferred split list, it is likely some subpages are not used. Split it before migration to avoid migrating unused subpages.
Commit
616b8371539a6("mm: thp: enable thp migration in generic path") did not check if a THP is on deferred split list before migration, thus, the destination THP is never put on deferred split list even if the source THP might be. The opportunity of reclaiming free pages in a partially mapped THP during deferred list scanning is lost, but no other harmful consequence is present[1].From v2:
1. Split the source folio instead of migrating it (per Matthew Wilcox)[2].From v1:
1. Used dst to get correct deferred split list after migration (per Ryan Roberts).[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/Fixes 616b8371539a ("mm: thp: enable thp migration in generic path")Signed-off-by Zi Yan <[email protected]>--- mm/huge_memory.c | 22 ------------------ mm/internal.h | 23 +++++++++++++++++++ mm/migrate.c | 60 +++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 72 insertions(+), 33 deletions(-)mm/huge_memory.c +0
-22@@ -766,28 +766,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)return pmd; } -#ifdef CONFIG_MEMCG -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct mem_cgroup *memcg =folio_memcg (folio); -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -if (memcg) -return &memcg->deferred_split_queue; -else -return &pgdat->deferred_split_queue; -} -#else -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -return &pgdat->deferred_split_queue; -} -#endif -void folio_prep_large_rmappable (struct folio *folio) {if (!folio || !folio_test_large (folio))mm/internal.h +11
-0@@ -1107,6 +1107,29 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_MEMCG +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct mem_cgroup *memcg =folio_memcg (folio); +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +if (memcg) +return &memcg->deferred_split_queue; +else +return &pgdat->deferred_split_queue;
mm/internal.h struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, · line 1130
Quoted 10 lines
mm/internal.h +10
-0+} +#else +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +return &pgdat->deferred_split_queue; +} +#endif
I found this breaks the build when CONFIG_TRANSPARENT_HUGEPAGE is not set, with below error:
.../lib/../mm/internal.h: In function 'get_deferred_split_queue':
.../lib/../mm/internal.h:1127:22: error: 'struct pglist_data' has no member named 'deferred_split_queue'
1127 | return &pgdat->deferred_split_queue;
| ^~Since the code was in hugepage.c, maybe the above chunk need to be wrapped by #ifdef CONFIG_TRANSPARENT_HUGEPAGE? I confirmed below change is fixing the build on my setup.
Thanks,
SJ
[...]
-------------------------------------- >8 -------------------------------------
mm/internal.h +3 -1
@@ -1106,6 +1106,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE #ifdef CONFIG_MEMCG static inline struct deferred_split *get_deferred_split_queue (struct folio *folio)
@@ -1126,7 +1127,8 @@ struct deferred_split *get_deferred_split_queue(struct folio *folio)return &pgdat->deferred_split_queue; } -#endif +#endif /* CONFIG_MEMCG */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /*
Zi Yan
reviewerQuoted 100 lines · SeongJae Park
Hello,
Quoted 82 lines · On Tue, 19 Mar 2024 11:47:53 -0400 Zi Yan <[email protected]> wrote:
From: Zi Yan <[email protected]>
If the source folio is on deferred split list, it is likely some subpages are not used. Split it before migration to avoid migrating unused subpages.
Commit
616b8371539a6("mm: thp: enable thp migration in generic path") did not check if a THP is on deferred split list before migration, thus, the destination THP is never put on deferred split list even if the source THP might be. The opportunity of reclaiming free pages in a partially mapped THP during deferred list scanning is lost, but no other harmful consequence is present[1].From v2:
1. Split the source folio instead of migrating it (per Matthew Wilcox)[2].From v1:
1. Used dst to get correct deferred split list after migration (per Ryan Roberts).[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/Fixes 616b8371539a ("mm: thp: enable thp migration in generic path")Signed-off-by Zi Yan <[email protected]>--- mm/huge_memory.c | 22 ------------------ mm/internal.h | 23 +++++++++++++++++++ mm/migrate.c | 60 +++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 72 insertions(+), 33 deletions(-)mm/huge_memory.c +0
-22@@ -766,28 +766,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)return pmd; } -#ifdef CONFIG_MEMCG -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct mem_cgroup *memcg =folio_memcg (folio); -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -if (memcg) -return &memcg->deferred_split_queue; -else -return &pgdat->deferred_split_queue; -} -#else -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -return &pgdat->deferred_split_queue; -} -#endif -void folio_prep_large_rmappable (struct folio *folio) {if (!folio || !folio_test_large (folio))mm/internal.h +11
-0@@ -1107,6 +1107,29 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_MEMCG +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct mem_cgroup *memcg =folio_memcg (folio); +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +if (memcg) +return &memcg->deferred_split_queue; +else +return &pgdat->deferred_split_queue;mm/internal.h struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, · line 1130
Quoted 10 lines
mm/internal.h +10
-0+} +#else +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +return &pgdat->deferred_split_queue; +} +#endif I found this breaks the build when
CONFIG_TRANSPARENT_HUGEPAGEis not set, with below error:
Quoted 8 lines
.../lib/../mm/internal.h: In function 'get_deferred_split_queue': .../lib/../mm/internal.h:1127:22: error: 'struct pglist_data' has no member named 'deferred_split_queue' 1127 | return &pgdat->deferred_split_queue; | ^~Since the code was in
hugepage.c, maybe the above chunk need to be wrapped by #ifdefCONFIG_TRANSPARENT_HUGEPAGE? I confirmed below change is fixing the build on my setup.
Thanks. Will fix it in the next version.
Quoted 17 lines
[...]
-------------------------------------- >8 -------------------------------------
mm/internal.h +1
-0@@ -1106,6 +1106,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE #ifdef CONFIG_MEMCG static inline struct deferred_split *get_deferred_split_queue (struct folio *folio)
mm/internal.h struct deferred_split *get_deferred_split_queue(struct folio *folio) · line 1134
Quoted 9 lines
mm/internal.h +2
-1@@ -1126,7 +1127,8 @@ struct deferred_split *get_deferred_split_queue(struct folio *folio)return &pgdat->deferred_split_queue; } -#endif +#endif /* CONFIG_MEMCG */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /*
--
Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 107 lines · Zi Yan
Quoted 105 lines · SeongJae Park
Hello,
Quoted 82 lines · On Tue, 19 Mar 2024 11:47:53 -0400 Zi Yan <[email protected]> wrote:
From: Zi Yan <[email protected]>
If the source folio is on deferred split list, it is likely some subpages are not used. Split it before migration to avoid migrating unused subpages.
Commit
616b8371539a6("mm: thp: enable thp migration in generic path") did not check if a THP is on deferred split list before migration, thus, the destination THP is never put on deferred split list even if the source THP might be. The opportunity of reclaiming free pages in a partially mapped THP during deferred list scanning is lost, but no other harmful consequence is present[1].From v2:
1. Split the source folio instead of migrating it (per Matthew Wilcox)[2].From v1:
1. Used dst to get correct deferred split list after migration (per Ryan Roberts).[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/Fixes 616b8371539a ("mm: thp: enable thp migration in generic path")Signed-off-by Zi Yan <[email protected]>--- mm/huge_memory.c | 22 ------------------ mm/internal.h | 23 +++++++++++++++++++ mm/migrate.c | 60 +++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 72 insertions(+), 33 deletions(-)mm/huge_memory.c +0
-22@@ -766,28 +766,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)return pmd; } -#ifdef CONFIG_MEMCG -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct mem_cgroup *memcg =folio_memcg (folio); -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -if (memcg) -return &memcg->deferred_split_queue; -else -return &pgdat->deferred_split_queue; -} -#else -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -return &pgdat->deferred_split_queue; -} -#endif -void folio_prep_large_rmappable (struct folio *folio) {if (!folio || !folio_test_large (folio))mm/internal.h +11
-0@@ -1107,6 +1107,29 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_MEMCG +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct mem_cgroup *memcg =folio_memcg (folio); +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +if (memcg) +return &memcg->deferred_split_queue; +else +return &pgdat->deferred_split_queue;mm/internal.h struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, · line 1130
Quoted 10 lines
mm/internal.h +10
-0+} +#else +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +return &pgdat->deferred_split_queue; +} +#endif I found this breaks the build when
CONFIG_TRANSPARENT_HUGEPAGEis not set, with below error:.../lib/../mm/internal.h: In function 'get_deferred_split_queue': .../lib/../mm/internal.h:1127:22: error: 'struct pglist_data' has no member named 'deferred_split_queue' 1127 | return &pgdat->deferred_split_queue; | ^~
Quoted 5 lines
Quoted 3 lines
Since the code was in
hugepage.c, maybe the above chunk need to be wrapped by #ifdefCONFIG_TRANSPARENT_HUGEPAGE? I confirmed below change is fixing the build on my setup.Thanks. Will fix it in the next version.
Actually, since get_deferred_split_queue() is used in mm/migrate.c, that part needs to be guarded by CONFIG_TRANSPARENT_HUGEPAGE as well.
--
Best Regards,
Yan, Zi
Zi Yan
reviewerHi Matthew,
Quoted 71 lines
Quoted 71 lines
Quoted 71 lines
Quoted 52 lines
mm/huge_memory.c +0
-22@@ -766,28 +766,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)return pmd; } -#ifdef CONFIG_MEMCG -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct mem_cgroup *memcg =folio_memcg (folio); -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -if (memcg) -return &memcg->deferred_split_queue; -else -return &pgdat->deferred_split_queue; -} -#else -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -return &pgdat->deferred_split_queue; -} -#endif -void folio_prep_large_rmappable (struct folio *folio) {if (!folio || !folio_test_large (folio))mm/internal.h +11
-0@@ -1107,6 +1107,29 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_MEMCG +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct mem_cgroup *memcg =folio_memcg (folio); +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +if (memcg) +return &memcg->deferred_split_queue; +else +return &pgdat->deferred_split_queue;mm/internal.h struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, · line 1130
Quoted 10 lines
mm/internal.h +10
-0+} +#else +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +return &pgdat->deferred_split_queue; +} +#endif I found this breaks the build when
CONFIG_TRANSPARENT_HUGEPAGEis not set, with below error:.../lib/../mm/internal.h: In function 'get_deferred_split_queue': .../lib/../mm/internal.h:1127:22: error: 'struct pglist_data' has no member named 'deferred_split_queue' 1127 | return &pgdat->deferred_split_queue; | ^~
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Since the code was in
hugepage.c, maybe the above chunk need to be wrapped by #ifdefCONFIG_TRANSPARENT_HUGEPAGE? I confirmed below change is fixing the build on my setup.Thanks. Will fix it in the next version.
Actually, since
get_deferred_split_queue()is used inmm/migrate.c, that part needs to be guarded byCONFIG_TRANSPARENT_HUGEPAGEas well.
_deferred_list is not hidden behind CONFIG_TRANSPARENT_HUGEPAGE but its users are all behind CONFIG_TRANSPARENT_HUGEPAGE. Should it be moved as well? Or the other way around, moving _deferred_list users out?
Large folios should work without THP, since they would be like THP without PMD mappings, namely mTHP. But mTHP is considered a subset of THP. It seems confusing. Maybe hiding _deferred_list behind CONFIG_TRANSPARENT_HUGEPAGE is better, or just do not change.
--
Best Regards,
Yan, Zi
Quoted 111 lines · On Tue, 19 Mar 2024 21:16:37 -0400 Zi Yan <[email protected]> wrote:
[-- Attachment #1: Type: text/plain, Size: 4354 bytes --]
Quoted 107 lines · Zi Yan
Quoted 105 lines · SeongJae Park
Hello,
Quoted 82 lines · On Tue, 19 Mar 2024 11:47:53 -0400 Zi Yan <[email protected]> wrote:
From: Zi Yan <[email protected]>
If the source folio is on deferred split list, it is likely some subpages are not used. Split it before migration to avoid migrating unused subpages.
Commit
616b8371539a6("mm: thp: enable thp migration in generic path") did not check if a THP is on deferred split list before migration, thus, the destination THP is never put on deferred split list even if the source THP might be. The opportunity of reclaiming free pages in a partially mapped THP during deferred list scanning is lost, but no other harmful consequence is present[1].From v2:
1. Split the source folio instead of migrating it (per Matthew Wilcox)[2].From v1:
1. Used dst to get correct deferred split list after migration (per Ryan Roberts).[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/Fixes 616b8371539a ("mm: thp: enable thp migration in generic path")Signed-off-by Zi Yan <[email protected]>--- mm/huge_memory.c | 22 ------------------ mm/internal.h | 23 +++++++++++++++++++ mm/migrate.c | 60 +++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 72 insertions(+), 33 deletions(-)mm/huge_memory.c +0
-22@@ -766,28 +766,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)return pmd; } -#ifdef CONFIG_MEMCG -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct mem_cgroup *memcg =folio_memcg (folio); -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -if (memcg) -return &memcg->deferred_split_queue; -else -return &pgdat->deferred_split_queue; -} -#else -static inline -struct deferred_split *get_deferred_split_queue (struct folio *folio) -{ -struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); - -return &pgdat->deferred_split_queue; -} -#endif -void folio_prep_large_rmappable (struct folio *folio) {if (!folio || !folio_test_large (folio))mm/internal.h +11
-0@@ -1107,6 +1107,29 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_MEMCG +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct mem_cgroup *memcg =folio_memcg (folio); +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +if (memcg) +return &memcg->deferred_split_queue; +else +return &pgdat->deferred_split_queue;mm/internal.h struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, · line 1130
Quoted 10 lines
mm/internal.h +10
-0+} +#else +static inline +struct deferred_split *get_deferred_split_queue (struct folio *folio) +{ +struct pglist_data *pgdat =NODE_DATA (folio_nid (folio)); + +return &pgdat->deferred_split_queue; +} +#endif I found this breaks the build when
CONFIG_TRANSPARENT_HUGEPAGEis not set, with below error:.../lib/../mm/internal.h: In function 'get_deferred_split_queue': .../lib/../mm/internal.h:1127:22: error: 'struct pglist_data' has no member named 'deferred_split_queue' 1127 | return &pgdat->deferred_split_queue; | ^~
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Since the code was in
hugepage.c, maybe the above chunk need to be wrapped by #ifdefCONFIG_TRANSPARENT_HUGEPAGE? I confirmed below change is fixing the build on my setup.Thanks. Will fix it in the next version.
Actually, since
get_deferred_split_queue()is used inmm/migrate.c, that part needs to be guarded byCONFIG_TRANSPARENT_HUGEPAGEas well.
You're right. I also just confirmed that build breaks with below error when CONFIG_TRANSPARENT_HUGEPAGE is not set but CONFIG_MIGRATION is set.
ERROR:root:.../mm/migrate.c: In function ‘migrate_pages_batch’:
.../mm/migrate.c:1682:49: error: implicit declaration of function ‘get_deferred_split_queue’ [-Werror=implicit-function-declaration]
1682 | get_deferred_split_queue(folio);
| ^~~~~~~~~~~~~~~~~~~~~~~~
Thanks,
SJ
Quoted 4 lines
--
Best Regards,
Yan, Zi