mm/migration: rework movable_ops page migration (part 1)
Subsystems (22)
cover mm/migration: rework movable_ops page migration (part 1)
Based on mm/mm-new.
In the future, as we decouple "struct page" from "struct folio", pages that support "non-lru page migration" -- movable_ops page migration such as memory balloons and zsmalloc -- will no longer be folios. They will not have ->mapping, ->lru, and likely no refcount and no page lock. But they will have flags :)
This is the first part (other parts not written yet) of decoupling movable_ops page migration from folio migration.
In this series, we get rid of the ->mapping usage, and start cleaning up the code + separating it from folio migration.
Migration core will have to be further reworked to not treat movable_ops pages like folios. This is the first step into that direction.
RFC because I only did some light testing with virtio-balloon + zsmalloc page migration, and have to polish some patch descriptions.
David Hildenbrand (29):
mm/balloon_compaction: we cannot have isolated pages in the balloon
list
mm/balloon_compaction: convert balloon_page_delete() to
balloon_page_finalize()
mm/zsmalloc: drop PageIsolated() related VM_BUG_ONs mm/page_alloc: allow for making page types sticky until freed mm/balloon_compaction: make PageOffline sticky mm/zsmalloc: make PageZsmalloc() sticky
mm/migrate: rename isolate_movable_page() to
isolate_movable_ops_page()
mm/migrate: rename putback_movable_folio() to
putback_movable_ops_page()
mm/migrate: factor out movable_ops page handling into
migrate_movable_ops_page()
mm/migrate: remove folio_test_movable() and folio_movable_ops() mm/migrate: move movable_ops page handling out of move_to_new_folio() mm/zsmalloc: stop using __ClearPageMovable() mm/balloon_compaction: stop using __ClearPageMovable() mm/migrate: remove __ClearPageMovable()
mm/migration: remove PageMovable()
mm: rename __PageMovable() to page_has_movable_ops() mm/page_isolation: drop __folio_test_movable() check for large folios
mm: remove __folio_test_movable()
mm: stop storing migration_ops in page->mapping
mm: convert "movable" flag in page->mapping to a page flag
mm: rename PG_isolated to PG_movable_ops_isolated mm/page-flags: rename PAGE_MAPPING_MOVABLE to PAGE_MAPPING_ANON_KSM mm/page-alloc: remove PageMappingFlags()
mm/page-flags: remove folio_mapping_flags()
mm: simplify folio_expected_ref_count()
mm: rename PAGE_MAPPING_* to FOLIO_MAPPING_*
docs/mm: convert from "Non-LRU page migration" to "movable_ops page
migration"
mm/balloon_compaction: "movable_ops" doc updates mm/balloon_compaction: provide single balloon_page_insert() and
balloon_mapping_gfp_mask()
Documentation/mm/page_migration.rst | 39 ++-- arch/powerpc/platforms/pseries/cmm.c | 2 +- drivers/misc/vmw_balloon.c | 3 +- drivers/virtio/virtio_balloon.c | 4 +- fs/proc/page.c | 4 +- include/linux/balloon_compaction.h | 90 ++++----- include/linux/fs.h | 2 +- include/linux/migrate.h | 42 +---- include/linux/mm.h | 4 +- include/linux/mm_types.h | 1 - include/linux/page-flags.h | 104 +++++++---- include/linux/pagemap.h | 2 +- include/linux/zsmalloc.h | 2 + mm/balloon_compaction.c | 21 ++- mm/compaction.c | 44 +---- mm/gup.c | 4 +- mm/internal.h | 2 +- mm/ksm.c | 4 +- mm/memory-failure.c | 4 +- mm/memory_hotplug.c | 8 +- mm/migrate.c | 269 +++++++++++++++------------ mm/page_alloc.c | 12 +- mm/page_isolation.c | 12 +- mm/rmap.c | 16 +- mm/util.c | 6 +- mm/vmscan.c | 6 +- mm/zpdesc.h | 15 +- mm/zsmalloc.c | 29 ++- 28 files changed, 361 insertions(+), 390 deletions(-)
base-commit: 21f39cc0346ea2081a83f0469995144d75d23075
Signature · 1 line
2.49.0
RFC PATCH 1/29 mm/balloon_compaction: we cannot have isolated pages in the balloon list
The core will set PG_isolated only after mops->isolate_page() was called. In case of the balloon, that is where we will remove it from the balloon list. So we cannot have isolated pages in the balloon list.
Let's drop this unnecessary check.
--- mm/balloon_compaction.c | 6 ------ 1 file changed, 6 deletions(-)
mm/balloon_compaction.c +0 -6
@@ -94,12 +94,6 @@ size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info,if (!trylock_page (page))continue ; -if (IS_ENABLED (CONFIG_BALLOON_COMPACTION ) && -PageIsolated (page)) { -/* raced with isolation */ -unlock_page (page); -continue ; - }balloon_page_delete (page);__count_vm_event (BALLOON_DEFLATE );list_add (&page->lru, pages);
Signature · 1 line
2.49.0
Re: [PATCH RFC 01/29] mm/balloon_compaction: we cannot have isolated pages in the balloon list
Quoted 11 lines · David Hildenbrand
The core will set
PG_isolatedonly aftermops->isolate_page()was called. In case of the balloon, that is where we will remove it from the balloon list. So we cannot have isolated pages in the balloon list.Let's drop this unnecessary check.
Signed-off-by David Hildenbrand <[email protected]>--- mm/balloon_compaction.c | 6 ------ 1 file changed, 6 deletions(-)
Best Regards,
Yan, Zi
RFC PATCH 2/29 mm/balloon_compaction: convert balloon_page_delete() to balloon_page_finalize()
In reply to David HildenbrandLet's move the removal of the page from the balloon list into the single caller, to remove the dependency on the PG_isolated flag and clarify locking requirements.
We'll shuffle the operations a bit such that they logically make more sense (e.g., remove from the list before clearing flags).
In balloon migration functions we can now move the balloon_page_finalize() out of the balloon lock and perform the finalization just before dropping the balloon reference.
Document that the page lock is currently required when modifying the movability aspects of a page; hopefully we can soon decouple this from the page lock.
--- arch/powerpc/platforms/pseries/cmm.c | 2 +- drivers/misc/vmw_balloon.c | 3 +- drivers/virtio/virtio_balloon.c | 4 +-- include/linux/balloon_compaction.h | 43 +++++++++++----------------- mm/balloon_compaction.c | 3 +- 5 files changed, 21 insertions(+), 34 deletions(-)
arch/powerpc/platforms/pseries/cmm.c +1 -1
@@ -532,7 +532,6 @@ static int cmm_migratepage(struct balloon_dev_info *b_dev_info,spin_lock_irqsave (&b_dev_info->pages_lock, flags);balloon_page_insert (b_dev_info, newpage); -balloon_page_delete (page); b_dev_info->isolated_pages--;spin_unlock_irqrestore (&b_dev_info->pages_lock, flags);
@@ -542,6 +541,7 @@ static int cmm_migratepage(struct balloon_dev_info *b_dev_info,*/ plpar_page_set_active (page); +balloon_page_finalize (page);/* balloon page list reference */ put_page (page);
drivers/misc/vmw_balloon.c +1 -2
@@ -1778,8 +1778,7 @@ static int vmballoon_migratepage(struct balloon_dev_info *b_dev_info,* @pages_lock . We keep holding @comm_lock since we will need it in a * second. */ -balloon_page_delete (page); - +balloon_page_finalize (page);put_page (page);/* Inflate */
drivers/virtio/virtio_balloon.c +1 -3
@@ -866,15 +866,13 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info,tell_host (vb, vb->inflate_vq);/* balloon's page migration 2nd step -- deflate "page" */ -spin_lock_irqsave (&vb_dev_info->pages_lock, flags); -balloon_page_delete (page); -spin_unlock_irqrestore (&vb_dev_info->pages_lock, flags); vb->num_pfns =VIRTIO_BALLOON_PAGES_PER_PAGE ;set_page_pfns (vb, vb->pfns, page);tell_host (vb, vb->deflate_vq);mutex_unlock (&vb->balloon_lock); +balloon_page_finalize (page);put_page (page);/* balloon reference */ return MIGRATEPAGE_SUCCESS ;
include/linux/balloon_compaction.h +16 -27
61 lines not on this page · raw message
mm/balloon_compaction.c +2 -1
@@ -94,7 +94,8 @@ size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info,if (!trylock_page (page))continue ; -balloon_page_delete (page); +list_del (&page->lru); +balloon_page_finalize (page);__count_vm_event (BALLOON_DEFLATE );list_add (&page->lru, pages);unlock_page (page);
Signature · 1 line
2.49.0
RFC PATCH 3/29 mm/zsmalloc: drop PageIsolated() related VM_BUG_ONs
In reply to David HildenbrandLet's drop these checks; these are conditions the core migration code must make sure will hold either way, no need to double check.
--- mm/zpdesc.h | 5 ----- mm/zsmalloc.c | 5 ----- 2 files changed, 10 deletions(-)
mm/zpdesc.h +0 -5
@@ -168,11 +168,6 @@ static inline void __zpdesc_clear_zsmalloc(struct zpdesc *zpdesc)__ClearPageZsmalloc (zpdesc_page (zpdesc)); } -static inline bool zpdesc_is_isolated (struct zpdesc *zpdesc) -{ -return PageIsolated (zpdesc_page (zpdesc)); -} -static inline struct zone *zpdesc_zone (struct zpdesc *zpdesc) {return page_zone (zpdesc_page (zpdesc));
mm/zsmalloc.c +0 -5
@@ -1719,8 +1719,6 @@ static bool zs_page_isolate(struct page *page, isolate_mode_t mode)* Page is locked so zspage couldn't be destroyed. For detail, look at * lock_zspage in free_zspage. */ -VM_BUG_ON_PAGE (PageIsolated (page), page); -return true; }
@@ -1739,8 +1737,6 @@ static int zs_page_migrate(struct page *newpage, struct page *page,unsigned long old_obj, new_obj;unsigned int obj_idx; -VM_BUG_ON_PAGE (!zpdesc_is_isolated (zpdesc),zpdesc_page (zpdesc)); -/* The page is locked, so this pointer must remain valid */ zspage =get_zspage (zpdesc); pool = zspage->pool;
@@ -1811,7 +1807,6 @@ static int zs_page_migrate(struct page *newpage, struct page *page,static void zs_page_putback (struct page *page) { -VM_BUG_ON_PAGE (!PageIsolated (page), page); }static const struct movable_operations zsmalloc_mops = {
Signature · 1 line
2.49.0
Re: [PATCH RFC 03/29] mm/zsmalloc: drop PageIsolated() related VM_BUG_ONs
Quoted 9 lines · David Hildenbrand
Let's drop these checks; these are conditions the core migration code must make sure will hold either way, no need to double check.
Signed-off-by David Hildenbrand <[email protected]>--- mm/zpdesc.h | 5 ----- mm/zsmalloc.c | 5 ----- 2 files changed, 10 deletions(-)
Best Regards,
Yan, Zi
Re: [PATCH RFC 03/29] mm/zsmalloc: drop PageIsolated() related VM_BUG_ONs
In reply to David HildenbrandQuoted 4 lines · David Hildenbrand
Let's drop these checks; these are conditions the core migration code must make sure will hold either way, no need to double check.
Signed-off-by David Hildenbrand <[email protected]>
Re: [PATCH RFC 03/29] mm/zsmalloc: drop PageIsolated() related VM_BUG_ONs
In reply to David HildenbrandQuoted 5 lines · David Hildenbrand
Let's drop these checks; these are conditions the core migration code must make sure will hold either way, no need to double check.
Signed-off-by David Hildenbrand <[email protected]>---
Signature · 2 lines
Cheers, Harry / Hyeonggon
RFC PATCH 4/29 mm/page_alloc: allow for making page types sticky until freed
In reply to David HildenbrandLet's allow for not clearing a page type before freeing a page to the buddy.
We'll focus on having a type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
--- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+)
mm/page_alloc.c +3 -0
@@ -1380,6 +1380,9 @@ __always_inline bool free_pages_prepare(struct page *page,mod_mthp_stat (order,MTHP_STAT_NR_ANON ,-1 ); page->mapping =NULL ; } +if (unlikely (page_has_type (page))) + page->page_type =UINT_MAX ; +if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;
Signature · 1 line
2.49.0
Zi Yan
reviewerRe: [PATCH RFC 04/29] mm/page_alloc: allow for making page types sticky until freed
Quoted 31 lines · David Hildenbrand
Let's allow for not clearing a page type before freeing a page to the buddy.
We'll focus on having a type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Signed-off-by David Hildenbrand <[email protected]>--- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+)mm/page_alloc.c +3
-0@@ -1380,6 +1380,9 @@ __always_inline bool free_pages_prepare(struct page *page,mod_mthp_stat (order,MTHP_STAT_NR_ANON ,-1 ); page->mapping =NULL ; } +if (unlikely (page_has_type (page))) + page->page_type =UINT_MAX ; +if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;Signature · 1 line
2.49.0
How does this preserve page type? Isn’t page->page_type = UINT_MAX clearing page_type?
Best Regards,
Yan, Zi
Matthew Wilcox
maintainerRe: [PATCH RFC 04/29] mm/page_alloc: allow for making page types sticky until freed
Quoted 10 lines · Zi Yan
Quoted 9 lines
Let's allow for not clearing a page type before freeing a page to the buddy.
We'll focus on having a type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Quoted 11 lines
Quoted 8 lines
mm/page_alloc.c +3
-0+if (unlikely (page_has_type (page))) + page->page_type =UINT_MAX ; +if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;Signature · 1 line
2.49.0How does this preserve page type? Isn’t
page->page_type=UINT_MAXclearingpage_type?
The point is that the _caller_ used to have to clear the page type. This patch allows the caller to free the page without clearing the page type first.
Zi Yan
reviewerRe: [PATCH RFC 04/29] mm/page_alloc: allow for making page types sticky until freed
Quoted 23 lines · Matthew Wilcox
Quoted 10 lines · Zi Yan
Quoted 9 lines
Let's allow for not clearing a page type before freeing a page to the buddy.
We'll focus on having a type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Quoted 11 lines
Quoted 8 lines
mm/page_alloc.c +3
-0+if (unlikely (page_has_type (page))) + page->page_type =UINT_MAX ; +if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;Signature · 1 line
2.49.0How does this preserve page type? Isn’t
page->page_type=UINT_MAXclearingpage_type?
Quoted 3 lines
The point is that the _caller_ used to have to clear the page type. This patch allows the caller to free the page without clearing the page type first.
Yep, find that out when I read the next patch.
I think the change is fine, but the commit message is unclear.
Best Regards,
Yan, Zi
Zi Yan
reviewerRe: [PATCH RFC 04/29] mm/page_alloc: allow for making page types sticky until freed
In reply to Zi YanQuoted 28 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Let's allow for not clearing a page type before freeing a page to the buddy.
We'll focus on having a type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Signed-off-by David Hildenbrand <[email protected]>--- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+)mm/page_alloc.c +3
-0@@ -1380,6 +1380,9 @@ __always_inline bool free_pages_prepare(struct page *page,mod_mthp_stat (order,MTHP_STAT_NR_ANON ,-1 ); page->mapping =NULL ; } +if (unlikely (page_has_type (page))) + page->page_type =UINT_MAX ; +
Quoted 8 lines
Quoted 5 lines
mm/page_alloc.c +0
-0if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;Signature · 1 line
2.49.0How does this preserve page type? Isn’t
page->page_type=UINT_MAXclearingpage_type?
OK, next patch explains it. free_pages_prepare() clears page_type, so that caller does not need to.
I think the message is better to be
mm/page_alloc: clear page_type at page free time
page_type is no longer needed to be cleared before a page is freed, as page free code does that.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Best Regards,
Yan, Zi
Re: [PATCH RFC 04/29] mm/page_alloc: allow for making page types sticky until freed
Quoted 44 lines · Zi Yan
Quoted 28 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Let's allow for not clearing a page type before freeing a page to the buddy.
We'll focus on having a type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Signed-off-by David Hildenbrand <[email protected]>--- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+)mm/page_alloc.c +3
-0@@ -1380,6 +1380,9 @@ __always_inline bool free_pages_prepare(struct page *page,mod_mthp_stat (order,MTHP_STAT_NR_ANON ,-1 ); page->mapping =NULL ; } +if (unlikely (page_has_type (page))) + page->page_type =UINT_MAX ; +Quoted 8 lines
Quoted 5 lines
mm/page_alloc.c +0
-0if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;Signature · 1 line
2.49.0How does this preserve page type? Isn’t
page->page_type=UINT_MAXclearingpage_type?OK, next patch explains it.
free_pages_prepare()clearspage_type, so that caller does not need to.I think the message is better to be
Quoted 8 lines
mm/page_alloc: clearpage_typeat page free time
page_typeis no longer needed to be cleared before a page is freed, as page free code does that.With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
I'll change it to
mm/page_alloc: let page freeing clear any set page type
Currently, any user of page types must clear that type before freeing a page back to the buddy, otherwise we'll run into mapcount related sanity checks (because the page type currently overlays the page mapcount).
Let's allow for not clearing the page type by page type users by letting the buddy handle it instead.
We'll focus on having a page type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH RFC 04/29] mm/page_alloc: allow for making page types sticky until freed
Quoted 57 lines · David Hildenbrand
Quoted 44 lines · Zi Yan
Quoted 28 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Let's allow for not clearing a page type before freeing a page to the buddy.
We'll focus on having a type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Signed-off-by David Hildenbrand <[email protected]>--- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+)mm/page_alloc.c +3
-0@@ -1380,6 +1380,9 @@ __always_inline bool free_pages_prepare(struct page *page,mod_mthp_stat (order,MTHP_STAT_NR_ANON ,-1 ); page->mapping =NULL ; } +if (unlikely (page_has_type (page))) + page->page_type =UINT_MAX ; +Quoted 8 lines
Quoted 5 lines
mm/page_alloc.c +0
-0if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;Signature · 1 line
2.49.0How does this preserve page type? Isn’t
page->page_type=UINT_MAXclearingpage_type?OK, next patch explains it.
free_pages_prepare()clearspage_type, so that caller does not need to.I think the message is better to be
Quoted 8 lines
mm/page_alloc: clearpage_typeat page free time
page_typeis no longer needed to be cleared before a page is freed, as page free code does that.With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
I'll change it to
Quoted 12 lines
mm/page_alloc: let page freeing clear any set page type
Currently, any user of page types must clear that type before freeing
a page back to the buddy, otherwise we'll run into mapcount related sanity checks (because the page type currently overlays the page mapcount).
Let's allow for not clearing the page type by page type users by letting
the buddy handle it instead.
We'll focus on having a page type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Thanks.
--
Best Regards,
Yan, Zi
Harry Yoo
Acked-byRe: [PATCH RFC 04/29] mm/page_alloc: allow for making page types sticky until freed
In reply to David HildenbrandQuoted 57 lines · David Hildenbrand
Quoted 44 lines · Zi Yan
Quoted 28 lines · Zi Yan
Quoted 26 lines · David Hildenbrand
Let's allow for not clearing a page type before freeing a page to the buddy.
We'll focus on having a type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Signed-off-by David Hildenbrand <[email protected]>--- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+)mm/page_alloc.c +3
-0@@ -1380,6 +1380,9 @@ __always_inline bool free_pages_prepare(struct page *page,mod_mthp_stat (order,MTHP_STAT_NR_ANON ,-1 ); page->mapping =NULL ; } +if (unlikely (page_has_type (page))) + page->page_type =UINT_MAX ; +Quoted 8 lines
Quoted 5 lines
mm/page_alloc.c +0
-0if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;Signature · 1 line
2.49.0How does this preserve page type? Isn’t
page->page_type=UINT_MAXclearingpage_type?OK, next patch explains it.
free_pages_prepare()clearspage_type, so that caller does not need to.I think the message is better to be
Quoted 8 lines
mm/page_alloc: clearpage_typeat page free time
page_typeis no longer needed to be cleared before a page is freed, as page free code does that.With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
I'll change it to
Quoted 12 lines
mm/page_alloc: let page freeing clear any set page type Currently, any user of page types must clear that type before freeing a page back to the buddy, otherwise we'll run into mapcount related sanity checks (because the page type currently overlays the page mapcount).
Let's allow for not clearing the page type by page type users by letting the buddy handle it instead.
We'll focus on having a page type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Signature · 2 lines
Cheers, Harry / Hyeonggon
Zi Yan
reviewerRe: [PATCH RFC 04/29] mm/page_alloc: allow for making page types sticky until freed
In reply to David HildenbrandQuoted 20 lines · David Hildenbrand
Let's allow for not clearing a page type before freeing a page to the buddy.
We'll focus on having a type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Signed-off-by David Hildenbrand <[email protected]>--- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+)mm/page_alloc.c +0
-0
mm/page_alloc.c __always_inline bool free_pages_prepare(struct page *page, · line 1388
Quoted 9 lines
mm/page_alloc.c +3
-0@@ -1380,6 +1380,9 @@ __always_inline bool free_pages_prepare(struct page *page,mod_mthp_stat (order,MTHP_STAT_NR_ANON ,-1 ); page->mapping =NULL ; } +if (unlikely (page_has_type (page))) + page->page_type =UINT_MAX ; +if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;
Should we be pedantic to only do this for PageOffline and PageZsmalloc and warn for the rest page types?
Something like:
if (unlikely(page_has_type(page))) {
if (PageOffline(page) || PageZsmalloc(page))
page->page_type = UINT_MAX;
else
VM_WARN_ONCE_PAGE(1, page);
}Best Regards,
Yan, Zi
Re: [PATCH RFC 04/29] mm/page_alloc: allow for making page types sticky until freed
Quoted 26 lines · Zi Yan
Quoted 24 lines · David Hildenbrand
Let's allow for not clearing a page type before freeing a page to the buddy.
We'll focus on having a type set on the first page of a larger allocation only.
With this change, we can reliably identify typed folios even though they might be in the process of getting freed, which will come in handy in migration code (at least in the transition phase).
Signed-off-by David Hildenbrand <[email protected]>--- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+)mm/page_alloc.c +1
-0@@ -1380,6 +1380,9 @@ __always_inline bool free_pages_prepare(struct page *page,mod_mthp_stat (order,MTHP_STAT_NR_ANON ,-1 ); page->mapping =NULL ; } +if (unlikely (page_has_type (page)))
Quoted 8 lines
mm/page_alloc.c __always_inline bool free_pages_prepare(struct page *page, · line 1388
Quoted 5 lines
mm/page_alloc.c +2
-0+ page->page_type =UINT_MAX ; +if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;Should we be pedantic to only do this for PageOffline and PageZsmalloc and warn for the rest page types?
I think we should just allow any page types. Limiting it to specific types sounds like some use-after-free check that probably shouldn't be handled that way.
Signature · 3 lines
Cheers, David / dhildenb
RFC PATCH 5/29 mm/balloon_compaction: make PageOffline sticky
In reply to David HildenbrandLet the buddy handle clearing the type.
--- include/linux/balloon_compaction.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/balloon_compaction.h +1 -1
@@ -140,7 +140,7 @@ static inline void balloon_page_finalize(struct page *page)__ClearPageMovable (page);set_page_private (page,0 ); } -__ClearPageOffline (page); +/* PageOffline is sticky until the page is freed to the buddy. */ }/*
Signature · 1 line
2.49.0
Re: [PATCH RFC 05/29] mm/balloon_compaction: make PageOffline sticky
Quoted 1 line · David Hildenbrand
Let the buddy handle clearing the type.
The below might be more precise? Since page type is cleared before the page reaches buddy code.
Let the free page routine handle clearing the type.
Quoted 21 lines
Signed-off-by David Hildenbrand <[email protected]>--- include/linux/balloon_compaction.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/linux/balloon_compaction.h +1
-1@@ -140,7 +140,7 @@ static inline void balloon_page_finalize(struct page *page)__ClearPageMovable (page);set_page_private (page,0 ); } -__ClearPageOffline (page); +/* PageOffline is sticky until the page is freed to the buddy. */ }/* Signature · 1 line
2.49.0
Best Regards,
Yan, Zi
Re: [PATCH RFC 05/29] mm/balloon_compaction: make PageOffline sticky
Quoted 8 lines · Zi Yan
Quoted 1 line · David Hildenbrand
Let the buddy handle clearing the type.
The below might be more precise? Since page type is cleared before the page reaches buddy code.
Let the free page routine handle clearing the type.
"Let the page freeing code handle clearing the page type."
?
Thanks!
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerRe: [PATCH RFC 05/29] mm/balloon_compaction: make PageOffline sticky
Quoted 5 lines · David Hildenbrand
Quoted 4 lines · Zi Yan
Quoted 1 line · David Hildenbrand
Let the buddy handle clearing the type.
Quoted 9 lines
Quoted 4 lines
The below might be more precise? Since page type is cleared before the page reaches buddy code.
Let the free page routine handle clearing the type.
"Let the page freeing code handle clearing the page type."
?
Sounds good to me. :)
--
Best Regards,
Yan, Zi
Re: [PATCH RFC 05/29] mm/balloon_compaction: make PageOffline sticky
In reply to David HildenbrandQuoted 4 lines · David Hildenbrand
Let the buddy handle clearing the type.
Signed-off-by David Hildenbrand <[email protected]>---
with the updated commit message,
Quoted 19 lines
include/linux/balloon_compaction.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/linux/balloon_compaction.h +1
-1@@ -140,7 +140,7 @@ static inline void balloon_page_finalize(struct page *page)__ClearPageMovable (page);set_page_private (page,0 ); } -__ClearPageOffline (page); +/* PageOffline is sticky until the page is freed to the buddy. */ }/* Signature · 1 line
2.49.0
Signature · 2 lines
Cheers, Harry / Hyeonggon
RFC PATCH 6/29 mm/zsmalloc: make PageZsmalloc() sticky
In reply to David HildenbrandLet the buddy handle clearing the type.
--- mm/zpdesc.h | 5 ----- mm/zsmalloc.c | 3 +-- 2 files changed, 1 insertion(+), 7 deletions(-)
mm/zpdesc.h +0 -5
@@ -163,11 +163,6 @@ static inline void __zpdesc_set_zsmalloc(struct zpdesc *zpdesc)__SetPageZsmalloc (zpdesc_page (zpdesc)); } -static inline void __zpdesc_clear_zsmalloc (struct zpdesc *zpdesc) -{ -__ClearPageZsmalloc (zpdesc_page (zpdesc)); -} -static inline struct zone *zpdesc_zone (struct zpdesc *zpdesc) {return page_zone (zpdesc_page (zpdesc));
mm/zsmalloc.c +1 -2
@@ -880,7 +880,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc)ClearPagePrivate (page); zpdesc->zspage =NULL ; zpdesc->next =NULL ; -__ClearPageZsmalloc (page); +/* PageZsmalloc is sticky until the page is freed to the buddy. */ }static int trylock_zspage (struct zspage *zspage)
@@ -1055,7 +1055,6 @@ static struct zspage *alloc_zspage(struct zs_pool *pool,if (!zpdesc) {while (--i >=0 ) {zpdesc_dec_zone_page_state (zpdescs[i]); -__zpdesc_clear_zsmalloc (zpdescs[i]);free_zpdesc (zpdescs[i]); }cache_free_zspage (pool, zspage);
Signature · 1 line
2.49.0
Re: [PATCH RFC 06/29] mm/zsmalloc: make PageZsmalloc() sticky
Quoted 1 line · David Hildenbrand
Let the buddy handle clearing the type.
Same comment as Patch 5.
Quoted 7 lines
Signed-off-by David Hildenbrand <[email protected]>--- mm/zpdesc.h | 5 ----- mm/zsmalloc.c | 3 +-- 2 files changed, 1 insertion(+), 7 deletions(-)
Best Regards,
Yan, Zi
Re: [PATCH RFC 06/29] mm/zsmalloc: make PageZsmalloc() sticky
In reply to David HildenbrandQuoted 3 lines · David Hildenbrand
Let the buddy handle clearing the type.
Signed-off-by David Hildenbrand <[email protected]>
FWIW,
Re: [PATCH RFC 06/29] mm/zsmalloc: make PageZsmalloc() sticky
In reply to David HildenbrandQuoted 4 lines · David Hildenbrand
Let the buddy handle clearing the type.
Signed-off-by David Hildenbrand <[email protected]>---
with the updated commit message,
Signature · 2 lines
Cheers, Harry / Hyeonggon
RFC PATCH 7/29 mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
In reply to David Hildenbrand... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the VM_BUG_ON() into a WARN_ON_ONCE() and handle it gracefully (relevant with further changes), and convert a WARN_ON_ONCE() into a VM_WARN_ON_ONCE_PAGE().
--- include/linux/migrate.h | 4 ++-- mm/compaction.c | 2 +- mm/migrate.c | 39 +++++++++++++++++++++++++++++---------- 3 files changed, 32 insertions(+), 13 deletions(-)
include/linux/migrate.h +2 -2
@@ -69,7 +69,7 @@ int migrate_pages(struct list_head *l, new_folio_t new, free_folio_t free, unsigned long private, enum migrate_mode mode, int reason, unsigned int *ret_succeeded);struct folio *alloc_migration_target (struct folio *src,unsigned long private); -bool isolate_movable_page (struct page *page,isolate_mode_t mode); +bool isolate_movable_ops_page (struct page *page,isolate_mode_t mode);bool isolate_folio_to_list (struct folio *folio,struct list_head *list); int migrate_huge_page_move_mapping(struct address_space *mapping,
@@ -90,7 +90,7 @@ static inline int migrate_pages(struct list_head *l, new_folio_t new,static inline struct folio *alloc_migration_target (struct folio *src,unsigned long private) {return NULL ; } -static inline bool isolate_movable_page (struct page *page,isolate_mode_t mode) +static inline bool isolate_movable_ops_page (struct page *page,isolate_mode_t mode) {return false; }static inline bool isolate_folio_to_list (struct folio *folio,struct list_head *list) {return false; }
mm/compaction.c +1 -1
@@ -1093,7 +1093,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, locked =NULL ; } -if (isolate_movable_page (page, mode)) { +if (isolate_movable_ops_page (page, mode)) { folio =page_folio (page); goto isolate_success; }
mm/migrate.c +29 -10
@@ -51,8 +51,26 @@#include "internal.h" #include "swap.h" -bool isolate_movable_page (struct page *page,isolate_mode_t mode) +/** +* isolate_movable_ops_page - isolate a movable_ops page for migration +* @page: The page. +* @mode: The isolation mode. +* +* Try to isolate a movable_ops page for migration. Will fail if the page is +* not a movable_ops page, if the page is already isolated for migration +* or if the page was just was released by its owner. +* +* Once isolated, the page cannot get freed until it is either putback +* or migrated. +* +* Returns true if isolation succeeded, otherwise false. +*/ +bool isolate_movable_ops_page (struct page *page,isolate_mode_t mode) { +/* +* TODO: these pages will not be folios in the future. All +* folio dependencies will have to be removed. +*/ struct folio *folio =folio_get_nontail_page (page);const struct movable_operations *mops;
@@ -73,7 +91,7 @@ bool isolate_movable_page(struct page *page, isolate_mode_t mode)* we use non-atomic bitops on newly allocated page flags so * unconditionally grabbing the lock ruins page's owner side. */ -if (unlikely (!__folio_test_movable (folio))) +if (unlikely (!__PageMovable (page))) goto out_putfolio;/*
@@ -90,18 +108,19 @@ bool isolate_movable_page(struct page *page, isolate_mode_t mode)if (unlikely (!folio_trylock (folio))) goto out_putfolio; -if (!folio_test_movable (folio) ||folio_test_isolated (folio)) +if (!PageMovable (page) ||PageIsolated (page)) goto out_no_isolated; - mops =folio_movable_ops (folio); -VM_BUG_ON_FOLIO (!mops, folio); + mops =page_movable_ops (page); +if (WARN_ON_ONCE (!mops)) + goto out_no_isolated; -if (!mops->isolate_page (&folio->page, mode)) +if (!mops->isolate_page (page, mode)) goto out_no_isolated;/* Driver shouldn't use the isolated flag */ -WARN_ON_ONCE (folio_test_isolated (folio)); -folio_set_isolated (folio); +VM_WARN_ON_ONCE_PAGE (PageIsolated (page), page); +SetPageIsolated (page);folio_unlock (folio);return true;
@@ -175,8 +194,8 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list)if (lru) isolated =folio_isolate_lru (folio);else - isolated =isolate_movable_page (&folio->page, -ISOLATE_UNEVICTABLE ); + isolated =isolate_movable_ops_page (&folio->page, +ISOLATE_UNEVICTABLE );if (!isolated)return false;
Signature · 1 line
2.49.0
Zi Yan
reviewerRe: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
Quoted 7 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().
The reason is that there is no upstream code, which use movable_ops for folios? Is there any fundamental reason preventing movable_ops from being used on folios?
Best Regards,
Yan, Zi
Matthew Wilcox
maintainerRe: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
Quoted 6 lines · Zi Yan
Quoted 4 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
Quoted 7 lines
Quoted 3 lines
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().The reason is that there is no upstream code, which use
movable_opsfor folios? Is there any fundamental reason preventingmovable_opsfrom being used on folios?
folios either belong to a filesystem or they are anonymous memory, and so either the filesystem knows how to migrate them (through its a_ops) or the migration code knows how to handle anon folios directly.
Zi Yan
reviewerRe: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · Zi Yan
Quoted 7 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().
Quoted 7 lines
Quoted 3 lines
The reason is that there is no upstream code, which use
movable_opsfor folios? Is there any fundamental reason preventingmovable_opsfrom being used on folios?folios either belong to a filesystem or they are anonymous memory, and so either the filesystem knows how to migrate them (through its
a_ops) or the migration code knows how to handle anon folios directly.
for device private pages, to support migrating >0 order anon or fs folios to device, how should we represent them for devices? if you think folio is only for anon and fs.
Best Regards,
Yan, Zi
Re: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
Quoted 13 lines · Zi Yan
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · Zi Yan
Quoted 7 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
The reason is that there is no upstream code, which use
movable_opsfor folios? Is there any fundamental reason preventingmovable_opsfrom being used on folios?folios either belong to a filesystem or they are anonymous memory, and so either the filesystem knows how to migrate them (through its
a_ops) or the migration code knows how to handle anon folios directly.
Right, migration of folios will be handled by migration core.
Quoted 4 lines
for device private pages, to support migrating >0 order anon or fs folios to device, how should we represent them for devices? if you think folio is only for anon and fs.
I assume they are proper folios, so yes. Just like they are handled today (-> folios)
I was asking a related question at LSF/MM in Alistair's session: are we sure these things will be folios even before they are assigned to a filesystem? I recall the answer was "yes".
So we don't (and will not) support movable_ops for folios.
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerRe: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
Quoted 29 lines · David Hildenbrand
Quoted 13 lines · Zi Yan
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · Zi Yan
Quoted 7 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
The reason is that there is no upstream code, which use
movable_opsfor folios? Is there any fundamental reason preventingmovable_opsfrom being used on folios?folios either belong to a filesystem or they are anonymous memory, and so either the filesystem knows how to migrate them (through its
a_ops) or the migration code knows how to handle anon folios directly.Right, migration of folios will be handled by migration core.
Quoted 4 lines
for device private pages, to support migrating >0 order anon or fs folios to device, how should we represent them for devices? if you think folio is only for anon and fs.
Quoted 5 lines
I assume they are proper folios, so yes. Just like they are handled today (-> folios)
I was asking a related question at LSF/MM in Alistair's session: are we sure these things will be folios even before they are assigned to a filesystem? I recall the answer was "yes".
So we don't (and will not) support
movable_opsfor folios.
Got it. (I was abusing it to help develop alloc_contig_range() at pageblock granularity, since it was easy to write a driver to allocate a compound page at a specific PFN and claim the page is movable, then do page online/offline the range containing the PFNs. :) )
For the patch, Reviewed-by: Zi Yan <[email protected]>
--
Best Regards,
Yan, Zi
Re: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
Quoted 33 lines · Zi Yan
Quoted 31 lines · David Hildenbrand
Quoted 13 lines · Zi Yan
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · Zi Yan
Quoted 7 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
The reason is that there is no upstream code, which use
movable_opsfor folios? Is there any fundamental reason preventingmovable_opsfrom being used on folios?folios either belong to a filesystem or they are anonymous memory, and so either the filesystem knows how to migrate them (through its
a_ops) or the migration code knows how to handle anon folios directly.Right, migration of folios will be handled by migration core.
Quoted 4 lines
for device private pages, to support migrating >0 order anon or fs folios to device, how should we represent them for devices? if you think folio is only for anon and fs.
I assume they are proper folios, so yes. Just like they are handled today (-> folios)
Quoted 8 lines
Quoted 3 lines
I was asking a related question at LSF/MM in Alistair's session: are we sure these things will be folios even before they are assigned to a filesystem? I recall the answer was "yes".
So we don't (and will not) support
movable_opsfor folios.Got it. (I was abusing it to help develop
alloc_contig_range()at pageblock granularity, since it was easy to write a driver to allocate a compound page at a specific PFN and claim the page is movable, then do page online/offline the range containing the PFNs. :) )
Oh, interesting :)
Yeah, with a folio that will be a bit harder ...
Quoted 2 lines
For the patch, Reviewed-by: Zi Yan <[email protected]>
Thanks!
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
In reply to Zi YanQuoted 35 lines · Zi Yan
Quoted 33 lines · David Hildenbrand
Quoted 13 lines · Zi Yan
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · Zi Yan
Quoted 7 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
The reason is that there is no upstream code, which use
movable_opsfor folios? Is there any fundamental reason preventingmovable_opsfrom being used on folios?folios either belong to a filesystem or they are anonymous memory, and so either the filesystem knows how to migrate them (through its
a_ops) or the migration code knows how to handle anon folios directly.Right, migration of folios will be handled by migration core.
Quoted 4 lines
for device private pages, to support migrating >0 order anon or fs folios to device, how should we represent them for devices? if you think folio is only for anon and fs.
I assume they are proper folios, so yes. Just like they are handled today (-> folios)
I was asking a related question at LSF/MM in Alistair's session: are we sure these things will be folios even before they are assigned to a filesystem? I recall the answer was "yes".
Quoted 8 lines
Quoted 1 line
So we don't (and will not) support
movable_opsfor folios.Got it. (I was abusing it to help develop
alloc_contig_range()at pageblock granularity, since it was easy to write a driver to allocate a compound page at a specific PFN and claim the page is movable, then do page online/offline the range containing the PFNs. :) )For the patch, Reviewed-by: Zi Yan <[email protected]>
BTW, thinking about it, I think we could handle compound pages quite easily, we'd just have to lookup the head at some point -- but we wouldn't be using folios for that.
BUT, I am note sure how compound pages without a memdesc would look like in a memdesc world, and if it would actually be "compound pages" in the traditional sense.
So, I think there would be ways to handle that, once we get to it.
Signature · 3 lines
Cheers, David / dhildenb
Huang, Ying
reviewerRe: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
In reply to David HildenbrandQuoted 29 lines · David Hildenbrand
Quoted 13 lines · Zi Yan
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · Zi Yan
Quoted 7 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
The reason is that there is no upstream code, which use
movable_opsfor folios? Is there any fundamental reason preventingmovable_opsfrom being used on folios?folios either belong to a filesystem or they are anonymous memory, and so either the filesystem knows how to migrate them (through its
a_ops) or the migration code knows how to handle anon folios directly.Right, migration of folios will be handled by migration core.
Quoted 4 lines
for device private pages, to support migrating >0 order anon or fs folios
to device, how should we represent them for devices? if you think folio is only for anon and fs.
Quoted 8 lines
I assume they are proper folios, so yes. Just like they are handled today (-> folios)
I was asking a related question at LSF/MM in Alistair's session: are we sure these things will be folios even before they are assigned to a filesystem? I recall the answer was "yes".
So we don't (and will not) support
movable_opsfor folios.
Is it possible to use some device specific callbacks (DMA?) to copy from/to the device private folios (or pages) to/from the normal file/anon folios in the future?
---
Best Regards,
Huang, Ying
Alistair Popple
reviewerRe: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
Quoted 26 lines · Ying
Quoted 24 lines · David Hildenbrand
Quoted 13 lines · Zi Yan
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · Zi Yan
Quoted 7 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
The reason is that there is no upstream code, which use
movable_opsfor folios? Is there any fundamental reason preventingmovable_opsfrom being used on folios?folios either belong to a filesystem or they are anonymous memory, and so either the filesystem knows how to migrate them (through its
a_ops) or the migration code knows how to handle anon folios directly.Right, migration of folios will be handled by migration core.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
for device private pages, to support migrating >0 order anon or fs folios
to device, how should we represent them for devices? if you think folio is only for anon and fs.I assume they are proper folios, so yes. Just like they are handled today (-> folios)
Yes, they should be proper folios.
Quoted 9 lines
Quoted 5 lines
I was asking a related question at LSF/MM in Alistair's session: are we sure these things will be folios even before they are assigned to a filesystem? I recall the answer was "yes".
So we don't (and will not) support
movable_opsfor folios.Is it possible to use some device specific callbacks (DMA?) to copy from/to the device private folios (or pages) to/from the normal file/anon folios in the future?
I guess we could put such callbacks on the folio->pgmap, but I'm not sure why we would want to. Currently all migration to/from device private (or coherent) folios is managed by the device, which is one of the features of ZONE_DEVICE. Did you have some particular reason/idea for why we might want to do this?
Quoted 3 lines
---
Best Regards,
Huang, Ying
Huang, Ying
reviewerRe: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
Quoted 32 lines · Alistair Popple
Quoted 31 lines · Ying
Quoted 29 lines · David Hildenbrand
Quoted 13 lines · Zi Yan
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · Zi Yan
Quoted 7 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
The reason is that there is no upstream code, which use
movable_opsfor folios? Is there any fundamental reason preventingmovable_opsfrom being used on folios?folios either belong to a filesystem or they are anonymous memory, and so either the filesystem knows how to migrate them (through its
a_ops) or the migration code knows how to handle anon folios directly.Right, migration of folios will be handled by migration core.
Quoted 4 lines
for device private pages, to support migrating >0 order anon or fs folios
to device, how should we represent them for devices? if you think folio is only for anon and fs.
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
I assume they are proper folios, so yes. Just like they are handled today (-> folios)
Yes, they should be proper folios.
So, folios include file cache, anonymous, and some device private.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
I was asking a related question at LSF/MM in Alistair's session: are we sure these things will be folios even before they are assigned to a filesystem? I recall the answer was "yes".
So we don't (and will not) support
movable_opsfor folios.
Quoted 7 lines
Quoted 3 lines
Is it possible to use some device specific callbacks (DMA?) to copy from/to the device private folios (or pages) to/from the normal file/anon folios in the future?
I guess we could put such callbacks on the
folio->pgmap, but I'm not sure why we would want to. Currently all migration to/from device private (or coherent) folios is managed by the device, which is one of the features ofZONE_DEVICE.
Yes. The is the current behavior per my understanding too.
Quoted 1 line
Did you have some particular reason/idea for why we might want to do this?
No. Just want to check whether there are some requirements for that. I think that it's just another way to organize code.
---
Best Regards,
Huang, Ying
Alistair Popple
reviewerRe: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
Quoted 34 lines · Ying
Quoted 32 lines · Alistair Popple
Quoted 31 lines · Ying
Quoted 29 lines · David Hildenbrand
Quoted 13 lines · Zi Yan
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · Zi Yan
Quoted 7 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
The reason is that there is no upstream code, which use
movable_opsfor folios? Is there any fundamental reason preventingmovable_opsfrom being used on folios?folios either belong to a filesystem or they are anonymous memory, and so either the filesystem knows how to migrate them (through its
a_ops) or the migration code knows how to handle anon folios directly.Right, migration of folios will be handled by migration core.
Quoted 4 lines
for device private pages, to support migrating >0 order anon or fs folios
to device, how should we represent them for devices? if you think folio is only for anon and fs.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
I assume they are proper folios, so yes. Just like they are handled today (-> folios)
Yes, they should be proper folios.
So, folios include file cache, anonymous, and some device private.
Oh maybe I misunderstood what you were asking. We have anon and fs folios, and we currently have device private versions of the former. However ideally I think in a memdesc world we would have anon/fs folios, and the device private bit would be in the memdesc or some such and so at the level of a folio we'd only be dealing with "proper" anon or fs folios (of course in practice we may never permit device private versions of the latter).
In my earlier answer I just wanted to highlight the fact that order >0 device folios now look the same as normal higher order anon or fs folios. Ie. we don't do any of the special pgmap refcounting, etc. that we used to do for higher order device folios.
Quoted 24 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
I was asking a related question at LSF/MM in Alistair's session: are we sure these things will be folios even before they are assigned to a filesystem? I recall the answer was "yes".
So we don't (and will not) support
movable_opsfor folios.Quoted 7 lines
Quoted 3 lines
Is it possible to use some device specific callbacks (DMA?) to copy from/to the device private folios (or pages) to/from the normal file/anon folios in the future?
I guess we could put such callbacks on the
folio->pgmap, but I'm not sure why we would want to. Currently all migration to/from device private (or coherent) folios is managed by the device, which is one of the features ofZONE_DEVICE.Yes. The is the current behavior per my understanding too.
Quoted 1 line
Did you have some particular reason/idea for why we might want to do this?
No. Just want to check whether there are some requirements for that. I think that it's just another way to organize code.
---
Best Regards,
Huang, Ying
Re: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
In reply to David HildenbrandQuoted 10 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().Signed-off-by David Hildenbrand <[email protected]>---
Haha yeah, back to pages after folio conversion :P But makes sense.
Side question: In the future, maybe we will be unable to tell whether a page is compound or not, without first inspecting page->memdesc? (e.g., struct slab could have an order and a pointer to the head page... just imagining).
Signature · 2 lines
Cheers, Harry / Hyeonggon
Re: [PATCH RFC 07/29] mm/migrate: rename isolate_movable_page() to isolate_movable_ops_page()
Quoted 15 lines · Harry Yoo
Quoted 10 lines · David Hildenbrand
... and start moving back to per-page things that will absolutely not be folio things in the future. Add documentation and a comment that the remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the
VM_BUG_ON()into aWARN_ON_ONCE()and handle it gracefully (relevant with further changes), and convert aWARN_ON_ONCE()into aVM_WARN_ON_ONCE_PAGE().Signed-off-by David Hildenbrand <[email protected]>---
Haha yeah, back to pages after folio conversion :P But makes sense.
Quoted 6 lines
Reviewed-by Harry Yoo <[email protected]>Side question: In the future, maybe we will be unable to tell whether a page is compound or not, without first inspecting
page->memdesc? (e.g., struct slab could have an order and a pointer to the head page... just imagining).
Right, it's not really clear what we would do in the future, but for anything that allocates a memdesc, that is possible.
We wouldn't even need the "head" + "tail" bit indication.
For things without an allocated memdesc (e.g., PageOffline), maybe compound pages will simply not apply.
Likely, the concept of compound pages as we knew it will go away.
Signature · 3 lines
Cheers, David / dhildenb
RFC PATCH 8/29 mm/migrate: rename putback_movable_folio() to putback_movable_ops_page()
In reply to David Hildenbrand... and factor the complete handling of movable_ops pages out. Convert it similar to isolate_movable_ops_page().
While at it, convert the VM_BUG_ON_FOLIO() into a VM_WARN_ON_PAGE().
--- mm/migrate.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-)
mm/migrate.c +24 -13
@@ -133,12 +133,30 @@ bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode)return false; } -static void putback_movable_folio (struct folio *folio) +/** +* putback_movable_ops_page - putback an isolated movable_ops page +* @page: The isolated page. +* +* Putback an isolated movable_ops page. +* +* After the page was putback, it might get freed instantly. +*/ +static void putback_movable_ops_page (struct page *page) { -const struct movable_operations *mops =folio_movable_ops (folio); - - mops->putback_page (&folio->page); -folio_clear_isolated (folio); +/* +* TODO: these pages will not be folios in the future. All +* folio dependencies will have to be removed. +*/ +struct folio *folio =page_folio (page); + +VM_WARN_ON_ONCE_PAGE (!PageIsolated (page), page); +folio_lock (folio); +/* If the page was released by it's owner, there is nothing to do. */ +if (PageMovable (page)) +page_movable_ops (page)->putback_page (page); +ClearPageIsolated (page); +folio_unlock (folio); +folio_put (folio); }/*
@@ -166,14 +184,7 @@ void putback_movable_pages(struct list_head *l)* have PAGE_MAPPING_MOVABLE. */ if (unlikely (__folio_test_movable (folio))) { -VM_BUG_ON_FOLIO (!folio_test_isolated (folio), folio); -folio_lock (folio); -if (folio_test_movable (folio)) -putback_movable_folio (folio); -else -folio_clear_isolated (folio); -folio_unlock (folio); -folio_put (folio); +putback_movable_ops_page (&folio->page); }else {node_stat_mod_folio (folio,NR_ISOLATED_ANON +folio_is_file_lru (folio), -folio_nr_pages (folio));
Signature · 1 line
2.49.0
Re: [PATCH RFC 08/29] mm/migrate: rename putback_movable_folio() to putback_movable_ops_page()
Quoted 37 lines · David Hildenbrand
... and factor the complete handling of
movable_opspages out. Convert it similar toisolate_movable_ops_page().While at it, convert the
VM_BUG_ON_FOLIO()into aVM_WARN_ON_PAGE().Signed-off-by David Hildenbrand <[email protected]>--- mm/migrate.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-)mm/migrate.c +13
-5@@ -133,12 +133,30 @@ bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode)return false; } -static void putback_movable_folio (struct folio *folio) +/** +* putback_movable_ops_page - putback an isolated movable_ops page +* @page: The isolated page. +* +* Putback an isolated movable_ops page. +* +* After the page was putback, it might get freed instantly. +*/ +static void putback_movable_ops_page (struct page *page) { -const struct movable_operations *mops =folio_movable_ops (folio); - - mops->putback_page (&folio->page); -folio_clear_isolated (folio); +/* +* TODO: these pages will not be folios in the future. All +* folio dependencies will have to be removed. +*/
mm/migrate.c bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode) · line 159
Quoted 10 lines
mm/migrate.c +10
-0+struct folio *folio =page_folio (page); + +VM_WARN_ON_ONCE_PAGE (!PageIsolated (page), page); +folio_lock (folio); +/* If the page was released by it's owner, there is nothing to do. */ +if (PageMovable (page)) +page_movable_ops (page)->putback_page (page); +ClearPageIsolated (page); +folio_unlock (folio); +folio_put (folio);
Why not use page version of lock, unlock, and put? Especially you are thinking about not using folio for these pages. Just a question, I am OK with current patch.
Quoted 21 lines
mm/migrate.c +1
-8}/* @@ -166,14 +184,7 @@ void putback_movable_pages(struct list_head *l)* have PAGE_MAPPING_MOVABLE. */ if (unlikely (__folio_test_movable (folio))) { -VM_BUG_ON_FOLIO (!folio_test_isolated (folio), folio); -folio_lock (folio); -if (folio_test_movable (folio)) -putback_movable_folio (folio); -else -folio_clear_isolated (folio); -folio_unlock (folio); -folio_put (folio); +putback_movable_ops_page (&folio->page); }else {node_stat_mod_folio (folio,NR_ISOLATED_ANON +folio_is_file_lru (folio), -folio_nr_pages (folio));Signature · 1 line
2.49.0
Best Regards,
Yan, Zi
Matthew Wilcox
maintainerRe: [PATCH RFC 08/29] mm/migrate: rename putback_movable_folio() to putback_movable_ops_page()
Quoted 11 lines · Zi Yan
Quoted 10 lines · David Hildenbrand
mm/migrate.c +10
-0+/* +* TODO: these pages will not be folios in the future. All +* folio dependencies will have to be removed. +*/ +struct folio *folio =page_folio (page); + +VM_WARN_ON_ONCE_PAGE (!PageIsolated (page), page); +folio_lock (folio); +/* If the page was released by it's owner, there is nothing to do. */ +if (PageMovable (page))
Quoted 8 lines
mm/migrate.c bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode) · line 159
Quoted 4 lines
mm/migrate.c +4
-0+page_movable_ops (page)->putback_page (page); +ClearPageIsolated (page); +folio_unlock (folio); +folio_put (folio);Why not use page version of lock, unlock, and put? Especially you are thinking about not using folio for these pages. Just a question, I am OK with current patch.
That would reintroduce unnecessary calls to compound_head().
Zi Yan
reviewerRe: [PATCH RFC 08/29] mm/migrate: rename putback_movable_folio() to putback_movable_ops_page()
Quoted 21 lines · Matthew Wilcox
Quoted 11 lines · Zi Yan
Quoted 10 lines · David Hildenbrand
mm/migrate.c +10
-0+/* +* TODO: these pages will not be folios in the future. All +* folio dependencies will have to be removed. +*/ +struct folio *folio =page_folio (page); + +VM_WARN_ON_ONCE_PAGE (!PageIsolated (page), page); +folio_lock (folio); +/* If the page was released by it's owner, there is nothing to do. */ +if (PageMovable (page))Quoted 8 lines
mm/migrate.c bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode) · line 159
Quoted 4 lines
mm/migrate.c +4
-0+page_movable_ops (page)->putback_page (page); +ClearPageIsolated (page); +folio_unlock (folio); +folio_put (folio);Why not use page version of lock, unlock, and put? Especially you are thinking about not using folio for these pages. Just a question, I am OK with current patch.
Quoted 1 line
That would reintroduce unnecessary calls to
compound_head().
Got it. But here page is not folio, so it cannot be a compound page. Then, we will need page versions without compound_head() for non compound pages. Could that happen in the future when only folio can be compound and page is only order-0?
Best Regards,
Yan, Zi
Matthew Wilcox
maintainerRe: [PATCH RFC 08/29] mm/migrate: rename putback_movable_folio() to putback_movable_ops_page()
Quoted 11 lines · Zi Yan
Quoted 5 lines · Matthew Wilcox
Quoted 3 lines
Why not use page version of lock, unlock, and put? Especially you are thinking about not using folio for these pages. Just a question, I am OK with current patch.
That would reintroduce unnecessary calls to
compound_head().Got it. But here page is not folio, so it cannot be a compound page. Then, we will need page versions without
compound_head()for non compound pages. Could that happen in the future when only folio can be compound and page is only order-0?
I think the assumption that we'll only see compound pages as part of folios is untrue. For example, slabs will still allocate multiple pages (though slabs aren't migratable at this point). The sketch at
https://kernelnewbies.org/MatthewWilcox/Memdescs supports "misc pages" with an order stored in bits 12-17 of the memdesc. I don't know how useful that will turn out to be; maybe we'll never implement that.
Re: [PATCH RFC 08/29] mm/migrate: rename putback_movable_folio() to putback_movable_ops_page()
Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Zi Yan
Quoted 5 lines · Matthew Wilcox
Quoted 3 lines
Why not use page version of lock, unlock, and put? Especially you are thinking about not using folio for these pages. Just a question, I am OK with current patch.
That would reintroduce unnecessary calls to
compound_head().
Right. And I want to make it clear that everything that uses "folio" here must be reworked: not necessarily switching to the "page" variant, but actually by implementing it entirely differently.
(e.g., store them in an array instead of a list, get rid of the lock bit, try getting rid of the refcount as well)
Quoted 6 lines
Quoted 5 lines
Got it. But here page is not folio, so it cannot be a compound page. Then, we will need page versions without
compound_head()for non compound pages. Could that happen in the future when only folio can be compound and page is only order-0?
Quoted 6 lines
I think the assumption that we'll only see compound pages as part of folios is untrue. For example, slabs will still allocate multiple pages (though slabs aren't migratable at this point). The sketch at
https://kernelnewbies.org/MatthewWilcox/Memdescs supports "misc pages" with an order stored in bits 12-17 of the memdesc. I don't know how useful that will turn out to be; maybe we'll never implement that.
Once we have to support that for movable_ops, it will be an interesting question how to handle that. Most certainly, these things will not be folios. :)
Signature · 3 lines
Cheers, David / dhildenb
RFC PATCH 9/29 mm/migrate: factor out movable_ops page handling into migrate_movable_ops_page()
In reply to David HildenbrandLet's factor it out, simplifying the calling code.
The assumption is that flush_dcache_page() is not required for movable_ops pages: as documented for flush_dcache_folio(), it really only applies when the kernel wrote to pagecache pages / pages in highmem. movable_ops callbacks should be handling flushing caches if ever required.
Note that we can now change folio_mapping_flags() to folio_test_anon() to make it clearer, because movable_ops pages will never take that path.
--- mm/migrate.c | 82 ++++++++++++++++++++++++++++------------------------ 1 file changed, 45 insertions(+), 37 deletions(-)
mm/migrate.c +45 -37
@@ -159,6 +159,45 @@ static void putback_movable_ops_page(struct page *page)folio_put (folio); } +/** +* migrate_movable_ops_page - migrate an isolated movable_ops page +* @page: The isolated page. +* +* Migrate an isolated movable_ops page. +* +* If the src page was already released by its owner, the src page is +* un-isolated (putback) and migration succeeds; the migration core will be the +* owner of both pages. +* +* If the src page was not released by its owner and the migration was +* successful, the owner of the src page and the dst page are swapped and +* the src page is un-isolated. +* +* If migration fails, the ownership stays unmodified and the src page +* remains isolated: migration may be retried later or the page can be putback. +* +* TODO: migration core will treat both pages as folios and lock them before +* this call to unlock them after this call. Further, the folio refcounts on +* src and dst are also released by migration core. These pages will not be +* folios in the future, so that must be reworked. +* +* Returns MIGRATEPAGE_SUCCESS on success, otherwise a negative error +* code. +*/ +static int migrate_movable_ops_page (struct page *dst,struct page *src, +enum migrate_mode mode) +{ +int rc =MIGRATEPAGE_SUCCESS ; + +VM_WARN_ON_ONCE_PAGE (!PageIsolated (src), src); +/* If the page was released by it's owner, there is nothing to do. */ +if (PageMovable (src)) + rc =page_movable_ops (src)->migrate_page (dst, src, mode); +if (rc ==MIGRATEPAGE_SUCCESS ) +ClearPageIsolated (src); +return rc; +} +/* * Put previously isolated pages back onto the appropriate lists * from where they were once taken off for compaction/migration.
@@ -1023,51 +1062,20 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, mode); else rc =fallback_migrate_folio (mapping, dst, src, mode); - }else { -const struct movable_operations *mops; -/* -* In case of non-lru page, it could be released after -* isolation step. In that case, we shouldn't try migration. -*/ -VM_BUG_ON_FOLIO (!folio_test_isolated (src), src); -if (!folio_test_movable (src)) { - rc =MIGRATEPAGE_SUCCESS ; -folio_clear_isolated (src); +if (rc !=MIGRATEPAGE_SUCCESS ) goto out; - } - - mops =folio_movable_ops (src); - rc = mops->migrate_page (&dst->page, &src->page, mode); -WARN_ON_ONCE (rc ==MIGRATEPAGE_SUCCESS && - !folio_test_isolated (src)); - } - -/* -* When successful, old pagecache src->mapping must be cleared before -* src is freed; but stats require that PageAnon be left as PageAnon. -*/ -if (rc ==MIGRATEPAGE_SUCCESS ) { -if (__folio_test_movable (src)) { -VM_BUG_ON_FOLIO (!folio_test_isolated (src), src); - -/* -* We clear PG_movable under page_lock so any compactor -* cannot try to migrate this page. -*/ -folio_clear_isolated (src); - } -/* -* Anonymous and movable src->mapping will be cleared by -* free_pages_prepare so don't reset it here for keeping -* the type to work PageAnon, for example. +* For pagecache folios, src->mapping must be cleared before src +* is freed. Anonymous folios must stay anonymous until freed. */ -if (!folio_mapping_flags (src)) +if (!folio_test_anon (src)) src->mapping =NULL ;if (likely (!folio_is_zone_device (dst)))flush_dcache_folio (dst); + }else { + rc =migrate_movable_ops_page (&dst->page, &src->page, mode); } out:return rc;
Signature · 1 line
2.49.0
Zi Yan
reviewerRe: [PATCH RFC 09/29] mm/migrate: factor out movable_ops page handling into migrate_movable_ops_page()
Quoted 8 lines · David Hildenbrand
Let's factor it out, simplifying the calling code.
The assumption is that
flush_dcache_page()is not required formovable_opspages: as documented forflush_dcache_folio(), it really only applies when the kernel wrote to pagecache pages / pages in highmem.movable_opscallbacks should be handling flushing caches if ever required.
Quoted 8 lines
Note that we can now change
folio_mapping_flags()tofolio_test_anon()to make it clearer, becausemovable_opspages will never take that path.Signed-off-by David Hildenbrand <[email protected]>--- mm/migrate.c | 82 ++++++++++++++++++++++++++++------------------------ 1 file changed, 45 insertions(+), 37 deletions(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
--
Best Regards,
Yan, Zi
RFC PATCH 10/29 mm/migrate: remove folio_test_movable() and folio_movable_ops()
In reply to David HildenbrandFolios will have nothing to do with movable_ops page migration. These functions are now unused, so let's remove them.
--- include/linux/migrate.h | 14 -------------- 1 file changed, 14 deletions(-)
include/linux/migrate.h +0 -14
@@ -118,20 +118,6 @@ static inline void __ClearPageMovable(struct page *page) }#endif -static inline bool folio_test_movable (struct folio *folio) -{ -return PageMovable (&folio->page); -} - -static inline -const struct movable_operations *folio_movable_ops (struct folio *folio) -{ -VM_BUG_ON (!__folio_test_movable (folio)); - -return (const struct movable_operations *) - ((unsigned long )folio->mapping -PAGE_MAPPING_MOVABLE ); -} -static inline const struct movable_operations *page_movable_ops (struct page *page) {
Signature · 1 line
2.49.0
Re: [PATCH RFC 10/29] mm/migrate: remove folio_test_movable() and folio_movable_ops()
Quoted 8 lines · David Hildenbrand
Folios will have nothing to do with
movable_opspage migration. These functions are now unused, so let's remove them.Signed-off-by David Hildenbrand <[email protected]>--- include/linux/migrate.h | 14 -------------- 1 file changed, 14 deletions(-)
--
Best Regards,
Yan, Zi
RFC PATCH 11/29 mm/migrate: move movable_ops page handling out of move_to_new_folio()
In reply to David HildenbrandLet's move that handling directly into migrate_folio_move(), so we can simplify move_to_new_folio(). While at it, fixup the documentation a bit.
Note that unmap_and_move_huge_page() does not care, because it only deals with actual folios. (we only support migration of individual movable_ops pages)
--- mm/migrate.c | 61 ++++++++++++++++++++++++---------------------------- 1 file changed, 28 insertions(+), 33 deletions(-)
mm/migrate.c +28 -33
@@ -1024,11 +1024,12 @@ static int fallback_migrate_folio(struct address_space *mapping, }/* -* Move a page to a newly allocated page -* The page is locked and all ptes have been successfully removed. +* Move a src folio to a newly allocated dst folio. * -* The new page will have replaced the old page if this function -* is successful. +* The src and dst folios are locked and the src folios was unmapped from +* the page tables. +* +* On success, the src folio was replaced by the dst folio. * * Return value: * < 0 - error code
@@ -1037,34 +1038,30 @@ static int fallback_migrate_folio(struct address_space *mapping,static int move_to_new_folio (struct folio *dst,struct folio *src,enum migrate_mode mode) { +struct address_space *mapping =folio_mapping (src);int rc = -EAGAIN ; -bool is_lru = !__folio_test_movable (src);VM_BUG_ON_FOLIO (!folio_test_locked (src), src);VM_BUG_ON_FOLIO (!folio_test_locked (dst), dst); -if (likely (is_lru)) { -struct address_space *mapping =folio_mapping (src); - -if (!mapping) - rc =migrate_folio (mapping, dst, src, mode); -else if (mapping_inaccessible (mapping)) - rc = -EOPNOTSUPP ; -else if (mapping->a_ops->migrate_folio) -/* -* Most folios have a mapping and most filesystems -* provide a migrate_folio callback. Anonymous folios -* are part of swap space which also has its own -* migrate_folio callback. This is the most common path -* for page migration. -*/ - rc = mapping->a_ops->migrate_folio (mapping, dst, src, - mode); -else - rc =fallback_migrate_folio (mapping, dst, src, mode); +if (!mapping) + rc =migrate_folio (mapping, dst, src, mode); +else if (mapping_inaccessible (mapping)) + rc = -EOPNOTSUPP ; +else if (mapping->a_ops->migrate_folio) +/* +* Most folios have a mapping and most filesystems +* provide a migrate_folio callback. Anonymous folios +* are part of swap space which also has its own +* migrate_folio callback. This is the most common path +* for page migration. +*/ + rc = mapping->a_ops->migrate_folio (mapping, dst, src, + mode); +else + rc =fallback_migrate_folio (mapping, dst, src, mode); -if (rc !=MIGRATEPAGE_SUCCESS ) - goto out; +if (rc ==MIGRATEPAGE_SUCCESS ) {/* * For pagecache folios, src->mapping must be cleared before src * is freed. Anonymous folios must stay anonymous until freed.
@@ -1074,10 +1071,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,if (likely (!folio_is_zone_device (dst)))flush_dcache_folio (dst); - }else { - rc =migrate_movable_ops_page (&dst->page, &src->page, mode); } -out:return rc; }
@@ -1328,20 +1322,21 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,int rc;int old_page_state =0 ;struct anon_vma *anon_vma =NULL ; -bool is_lru = !__folio_test_movable (src);struct list_head *prev;__migrate_folio_extract (dst, &old_page_state, &anon_vma); prev = dst->lru.prev;list_del (&dst->lru); +if (unlikely (__folio_test_movable (src))) { + rc =migrate_movable_ops_page (&dst->page, &src->page, mode); + goto out_unlock_both; + } + rc =move_to_new_folio (dst, src, mode);if (rc) goto out; -if (unlikely (!is_lru)) - goto out_unlock_both; -/* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will
Signature · 1 line
2.49.0
Re: [PATCH RFC 11/29] mm/migrate: move movable_ops page handling out of move_to_new_folio()
Quoted 4 lines · David Hildenbrand
Let's move that handling directly into
migrate_folio_move(), so we can simplifymove_to_new_folio(). While at it, fixup the documentation a bit.
Quoted 9 lines
Note that
unmap_and_move_huge_page()does not care, because it only deals with actual folios. (we only support migration of individualmovable_opspages)Signed-off-by David Hildenbrand <[email protected]>--- mm/migrate.c | 61 ++++++++++++++++++++++++---------------------------- 1 file changed, 28 insertions(+), 33 deletions(-)
--
Best Regards,
Yan, Zi
Re: [PATCH RFC 11/29] mm/migrate: move movable_ops page handling out of move_to_new_folio()
In reply to David HildenbrandQuoted 104 lines · David Hildenbrand
Let's move that handling directly into
migrate_folio_move(), so we can simplifymove_to_new_folio(). While at it, fixup the documentation a bit.Note that
unmap_and_move_huge_page()does not care, because it only deals with actual folios. (we only support migration of individualmovable_opspages)Signed-off-by David Hildenbrand <[email protected]>--- mm/migrate.c | 61 ++++++++++++++++++++++++---------------------------- 1 file changed, 28 insertions(+), 33 deletions(-)mm/migrate.c +23
-30@@ -1024,11 +1024,12 @@ static int fallback_migrate_folio(struct address_space *mapping, }/* -* Move a page to a newly allocated page -* The page is locked and all ptes have been successfully removed. +* Move a src folio to a newly allocated dst folio. * -* The new page will have replaced the old page if this function -* is successful. +* The src and dst folios are locked and the src folios was unmapped from +* the page tables. +* +* On success, the src folio was replaced by the dst folio. * * Return value: * < 0 - error code @@ -1037,34 +1038,30 @@ static int fallback_migrate_folio(struct address_space *mapping,static int move_to_new_folio (struct folio *dst,struct folio *src,enum migrate_mode mode) { +struct address_space *mapping =folio_mapping (src);int rc = -EAGAIN ; -bool is_lru = !__folio_test_movable (src);VM_BUG_ON_FOLIO (!folio_test_locked (src), src);VM_BUG_ON_FOLIO (!folio_test_locked (dst), dst); -if (likely (is_lru)) { -struct address_space *mapping =folio_mapping (src); - -if (!mapping) - rc =migrate_folio (mapping, dst, src, mode); -else if (mapping_inaccessible (mapping)) - rc = -EOPNOTSUPP ; -else if (mapping->a_ops->migrate_folio) -/* -* Most folios have a mapping and most filesystems -* provide a migrate_folio callback. Anonymous folios -* are part of swap space which also has its own -* migrate_folio callback. This is the most common path -* for page migration. -*/ - rc = mapping->a_ops->migrate_folio (mapping, dst, src, - mode); -else - rc =fallback_migrate_folio (mapping, dst, src, mode); +if (!mapping) + rc =migrate_folio (mapping, dst, src, mode); +else if (mapping_inaccessible (mapping)) + rc = -EOPNOTSUPP ; +else if (mapping->a_ops->migrate_folio) +/* +* Most folios have a mapping and most filesystems +* provide a migrate_folio callback. Anonymous folios +* are part of swap space which also has its own +* migrate_folio callback. This is the most common path +* for page migration. +*/ + rc = mapping->a_ops->migrate_folio (mapping, dst, src, + mode); +else + rc =fallback_migrate_folio (mapping, dst, src, mode); -if (rc !=MIGRATEPAGE_SUCCESS ) - goto out; +if (rc ==MIGRATEPAGE_SUCCESS ) {/* * For pagecache folios, src->mapping must be cleared before src * is freed. Anonymous folios must stay anonymous until freed. @@ -1074,10 +1071,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,if (likely (!folio_is_zone_device (dst)))flush_dcache_folio (dst); - }else { - rc =migrate_movable_ops_page (&dst->page, &src->page, mode); } -out:return rc; }@@ -1328,20 +1322,21 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,int rc;int old_page_state =0 ;struct anon_vma *anon_vma =NULL ; -bool is_lru = !__folio_test_movable (src);struct list_head *prev;
mm/migrate.c static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · line 1335
Quoted 10 lines
mm/migrate.c +5
-0__migrate_folio_extract (dst, &old_page_state, &anon_vma); prev = dst->lru.prev;list_del (&dst->lru); +if (unlikely (__folio_test_movable (src))) { + rc =migrate_movable_ops_page (&dst->page, &src->page, mode); + goto out_unlock_both; + } +
I suspect this was the issue I was chasing:
mm/migrate.c +2 -0
@@ -1330,6 +1330,8 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (unlikely (__folio_test_movable (src))) { rc =migrate_movable_ops_page (&dst->page, &src->page, mode); +if (rc) + goto out; goto out_unlock_both; }
Testing ...
Signature · 3 lines
Cheers, David / dhildenb
RFC PATCH 12/29 mm/zsmalloc: stop using __ClearPageMovable()
In reply to David HildenbrandInstead, let's check in the callbacks if the page was already destroyed, which can be checked by looking at zpdesc->zspage (see reset_zpdesc()).
If we detect that the page was destroyed:
(1) Fail isolation, just like the migration core would
(2) Fake migration success just like the migration core would
In the putback case there is nothing to do, as we don't do anything just like the migration core would do.
In the future, we should look into not letting these pages get destroyed while they are isolated -- and instead delaying that to the putback/migration call. Add a TODO for that.
--- mm/zsmalloc.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
mm/zsmalloc.c +11 -4
@@ -876,7 +876,6 @@ static void reset_zpdesc(struct zpdesc *zpdesc) {struct page *page =zpdesc_page (zpdesc); -__ClearPageMovable (page);ClearPagePrivate (page); zpdesc->zspage =NULL ; zpdesc->next =NULL ;
@@ -1715,10 +1714,11 @@ static void replace_sub_page(struct size_class *class, struct zspage *zspage,static bool zs_page_isolate (struct page *page,isolate_mode_t mode) {/* -* Page is locked so zspage couldn't be destroyed. For detail, look at -* lock_zspage in free_zspage. +* Page is locked so zspage can't be destroyed concurrently +* (see free_zspage()). But if the page was already destroyed +* (see reset_zpdesc()), refuse isolation here. */ -return true; +return page_zpdesc (page)->zspage; } static int zs_page_migrate(struct page *newpage, struct page *page,
@@ -1736,6 +1736,13 @@ static int zs_page_migrate(struct page *newpage, struct page *page,unsigned long old_obj, new_obj;unsigned int obj_idx; +/* +* TODO: nothing prevents a zspage from getting destroyed while +* isolated: we should disallow that and defer it. +*/ +if (!zpdesc->zspage) +return MIGRATEPAGE_SUCCESS ; +/* The page is locked, so this pointer must remain valid */ zspage =get_zspage (zpdesc); pool = zspage->pool;
Signature · 1 line
2.49.0
RFC PATCH 13/29 mm/balloon_compaction: stop using __ClearPageMovable()
In reply to David HildenbrandWe can just look at the balloon device (stored in page->private), to see of the page is still part of the balloon.
As isolated balloon pages cannot get released (they are taken off the balloon list while isolated), we don't have to worry about this case in the putback and migration callback. Add a WARN_ON_ONCE for now.
--- include/linux/balloon_compaction.h | 4 +--- mm/balloon_compaction.c | 11 +++++++++++ 2 files changed, 12 insertions(+), 3 deletions(-)
include/linux/balloon_compaction.h +1 -3
@@ -136,10 +136,8 @@ static inline gfp_t balloon_mapping_gfp_mask(void)*/ static inline void balloon_page_finalize (struct page *page) { -if (IS_ENABLED (CONFIG_BALLOON_COMPACTION )) { -__ClearPageMovable (page); +if (IS_ENABLED (CONFIG_BALLOON_COMPACTION ))set_page_private (page,0 ); - }/* PageOffline is sticky until the page is freed to the buddy. */ }
mm/balloon_compaction.c +11 -0
@@ -206,6 +206,9 @@ static bool balloon_page_isolate(struct page *page, isolate_mode_t mode)struct balloon_dev_info *b_dev_info =balloon_page_device (page);unsigned long flags; +if (!b_dev_info) +return false; +spin_lock_irqsave (&b_dev_info->pages_lock, flags);list_del (&page->lru); b_dev_info->isolated_pages++;
@@ -219,6 +222,10 @@ static void balloon_page_putback(struct page *page)struct balloon_dev_info *b_dev_info =balloon_page_device (page);unsigned long flags; +/* Isolated balloon pages cannot get deflated. */ +if (WARN_ON_ONCE (!b_dev_info)) +return ; +spin_lock_irqsave (&b_dev_info->pages_lock, flags);list_add (&page->lru, &b_dev_info->pages); b_dev_info->isolated_pages--;
@@ -234,6 +241,10 @@ static int balloon_page_migrate(struct page *newpage, struct page *page,VM_BUG_ON_PAGE (!PageLocked (page), page);VM_BUG_ON_PAGE (!PageLocked (newpage), newpage); +/* Isolated balloon pages cannot get deflated. */ +if (WARN_ON_ONCE (!balloon)) +return -EAGAIN ; +return balloon->migratepage (balloon, newpage, page, mode); }
Signature · 1 line
2.49.0
Huang, Ying
reviewerRe: [PATCH RFC 13/29] mm/balloon_compaction: stop using __ClearPageMovable()
Quoted 2 lines · David Hildenbrand
We can just look at the balloon device (stored in
page->private), to see of the page is still part of the balloon.
s/of/if/
?
just a trivial issue if I'm not wrong.
Quoted 5 lines
As isolated balloon pages cannot get released (they are taken off the balloon list while isolated), we don't have to worry about this case in the putback and migration callback. Add a
WARN_ON_ONCEfor now.Signed-off-by David Hildenbrand <[email protected]>
[snip]
---
Best Regards,
Huang, Ying
Re: [PATCH RFC 13/29] mm/balloon_compaction: stop using __ClearPageMovable()
Quoted 8 lines · Ying
Quoted 2 lines · David Hildenbrand
We can just look at the balloon device (stored in
page->private), to see of the page is still part of the balloon.s/of/if/
?
Yes, thanks!
Signature · 3 lines
Cheers, David / dhildenb
RFC PATCH 14/29 mm/migrate: remove __ClearPageMovable()
In reply to David HildenbrandUnused, let's remove it.
The Chinese docs in Documentation/translations/zh_CN/mm/page_migration.rst still mention it, but that whole docs is destined to get outdated and updated by somebody that actually speaks that language.
--- include/linux/migrate.h | 4 ---- mm/compaction.c | 11 ----------- 2 files changed, 15 deletions(-)
include/linux/migrate.h +0 -4
@@ -106,16 +106,12 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#ifdef CONFIG_COMPACTION bool PageMovable (struct page *page);void __SetPageMovable (struct page *page,const struct movable_operations *ops); -void __ClearPageMovable (struct page *page);#else static inline bool PageMovable (struct page *page) {return false; }static inline void __SetPageMovable (struct page *page,const struct movable_operations *ops) { } -static inline void __ClearPageMovable (struct page *page) -{ -}#endif static inline
mm/compaction.c +0 -11
@@ -137,17 +137,6 @@ void __SetPageMovable(struct page *page, const struct movable_operations *mops) }EXPORT_SYMBOL (__SetPageMovable); -void __ClearPageMovable (struct page *page) -{ -VM_BUG_ON_PAGE (!PageMovable (page), page); -/* -* This page still has the type of a movable page, but it's -* actually not movable any more. -*/ - page->mapping = (void *)PAGE_MAPPING_MOVABLE ; -} -EXPORT_SYMBOL (__ClearPageMovable); -/* Do not skip compaction more than 64 times */ #define COMPACT_MAX_DEFER_SHIFT 6
Signature · 1 line
2.49.0
Zi Yan
reviewerRe: [PATCH RFC 14/29] mm/migrate: remove __ClearPageMovable()
Quoted 6 lines · David Hildenbrand
Unused, let's remove it.
The Chinese docs in
Documentation/translations/zh_CN/mm/page_migration.rststill mention it, but that whole docs is destined to get outdated and updated by somebody that actually speaks that language.
Quoted 6 lines
Signed-off-by David Hildenbrand <[email protected]>--- include/linux/migrate.h | 4 ---- mm/compaction.c | 11 ----------- 2 files changed, 15 deletions(-)
The comment for struct movable_operations needs an update too. "
If page migration is successful, the driver should call __ClearPageMovable(@src) and return MIGRATEPAGE_SUCCESS. "
Otherwise, Reviewed-by: Zi Yan <[email protected]>
--
Best Regards,
Yan, Zi
Re: [PATCH RFC 14/29] mm/migrate: remove __ClearPageMovable()
Quoted 15 lines · Zi Yan
Quoted 6 lines · David Hildenbrand
Unused, let's remove it.
The Chinese docs in
Documentation/translations/zh_CN/mm/page_migration.rststill mention it, but that whole docs is destined to get outdated and updated by somebody that actually speaks that language.Quoted 6 lines
Signed-off-by David Hildenbrand <[email protected]>--- include/linux/migrate.h | 4 ---- mm/compaction.c | 11 ----------- 2 files changed, 15 deletions(-)
Quoted 5 lines
The comment for struct
movable_operationsneeds an update too. "
If page migration is successful, the driver should call__ClearPageMovable(@src)and returnMIGRATEPAGE_SUCCESS. "
Ah, thanks, it will simply be "should return MIGRATEPAGE_SUCCESS".
There is more magic to it (the driver must setup the dst page as movable), but probably that might all be reworked soon.
Signature · 3 lines
Cheers, David / dhildenb
RFC PATCH 15/29 mm/migration: remove PageMovable()
In reply to David HildenbrandAs __ClearPageMovable() is gone that would have only made PageMovable()==false but still __PageMovable()==true, now PageMovable() == __PageMovable().
So we can replace PageMovable() checks by __PageMovable(). In fact, __PageMovable() cannot change until a page is freed, so we can turn some PageMovable() into sanity checks for __PageMovable().
--- include/linux/migrate.h | 2 -- mm/compaction.c | 15 --------------- mm/migrate.c | 18 ++++++++++-------- 3 files changed, 10 insertions(+), 25 deletions(-)
include/linux/migrate.h +0 -2
@@ -104,10 +104,8 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#endif /* CONFIG_MIGRATION */ #ifdef CONFIG_COMPACTION -bool PageMovable (struct page *page);void __SetPageMovable (struct page *page,const struct movable_operations *ops);#else -static inline bool PageMovable (struct page *page) {return false; }static inline void __SetPageMovable (struct page *page,const struct movable_operations *ops) {
mm/compaction.c +0 -15
@@ -114,21 +114,6 @@ static unsigned long release_free_list(struct list_head *freepages) }#ifdef CONFIG_COMPACTION -bool PageMovable (struct page *page) -{ -const struct movable_operations *mops; - -VM_BUG_ON_PAGE (!PageLocked (page), page); -if (!__PageMovable (page)) -return false; - - mops =page_movable_ops (page); -if (mops) -return true; - -return false; -} -void __SetPageMovable (struct page *page,const struct movable_operations *mops) {VM_BUG_ON_PAGE (!PageLocked (page), page);
mm/migrate.c +10 -8
@@ -87,9 +87,12 @@ bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode) goto out;/* -* Check movable flag before taking the page lock because +* Check for movable_ops pages before taking the page lock because * we use non-atomic bitops on newly allocated page flags so * unconditionally grabbing the lock ruins page's owner side. +* +* Note that once a page has movable_ops, it will stay that way +* until the page was freed. */ if (unlikely (!__PageMovable (page))) goto out_putfolio;
@@ -108,7 +111,8 @@ bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode)if (unlikely (!folio_trylock (folio))) goto out_putfolio; -if (!PageMovable (page) ||PageIsolated (page)) +VM_WARN_ON_ONCE_PAGE (!__PageMovable (page), page); +if (PageIsolated (page)) goto out_no_isolated; mops =page_movable_ops (page);
@@ -149,11 +153,10 @@ static void putback_movable_ops_page(struct page *page)*/ struct folio *folio =page_folio (page); +VM_WARN_ON_ONCE_PAGE (!__PageMovable (page), page);VM_WARN_ON_ONCE_PAGE (!PageIsolated (page), page);folio_lock (folio); -/* If the page was released by it's owner, there is nothing to do. */ -if (PageMovable (page)) -page_movable_ops (page)->putback_page (page); +page_movable_ops (page)->putback_page (page);ClearPageIsolated (page);folio_unlock (folio);folio_put (folio);
@@ -189,10 +192,9 @@ static int migrate_movable_ops_page(struct page *dst, struct page *src, {int rc =MIGRATEPAGE_SUCCESS ; +VM_WARN_ON_ONCE_PAGE (!__PageMovable (src), src);VM_WARN_ON_ONCE_PAGE (!PageIsolated (src), src); -/* If the page was released by it's owner, there is nothing to do. */ -if (PageMovable (src)) - rc =page_movable_ops (src)->migrate_page (dst, src, mode); + rc =page_movable_ops (src)->migrate_page (dst, src, mode);if (rc ==MIGRATEPAGE_SUCCESS )ClearPageIsolated (src);return rc;
Signature · 1 line
2.49.0
Re: [PATCH RFC 15/29] mm/migration: remove PageMovable()
Quoted 8 lines · David Hildenbrand
As
__ClearPageMovable()is gone that would have only made PageMovable()==false but still __PageMovable()==true, nowPageMovable()==__PageMovable().So we can replace
PageMovable()checks by__PageMovable(). In fact,__PageMovable()cannot change until a page is freed, so we can turn somePageMovable()into sanity checks for__PageMovable().
Quoted 7 lines
Signed-off-by David Hildenbrand <[email protected]>--- include/linux/migrate.h | 2 -- mm/compaction.c | 15 --------------- mm/migrate.c | 18 ++++++++++-------- 3 files changed, 10 insertions(+), 25 deletions(-)
--
Best Regards,
Yan, Zi
RFC PATCH 16/29 mm: rename __PageMovable() to page_has_movable_ops()
In reply to David HildenbrandLet's make it clearer that we are talking about movable_ops pages.
--- include/linux/migrate.h | 2 +- include/linux/page-flags.h | 2 +- mm/compaction.c | 7 ++----- mm/memory-failure.c | 4 ++-- mm/memory_hotplug.c | 8 +++----- mm/migrate.c | 8 ++++---- mm/page_alloc.c | 2 +- mm/page_isolation.c | 10 +++++----- 8 files changed, 19 insertions(+), 24 deletions(-)
include/linux/migrate.h +1 -1
@@ -115,7 +115,7 @@ static inline void __SetPageMovable(struct page *page,static inline const struct movable_operations *page_movable_ops (struct page *page) { -VM_BUG_ON (!__PageMovable (page)); +VM_BUG_ON (!page_has_movable_ops (page));return (const struct movable_operations *) ((unsigned long )page->mapping -PAGE_MAPPING_MOVABLE );
include/linux/page-flags.h +1 -1
@@ -750,7 +750,7 @@ static __always_inline bool __folio_test_movable(const struct folio *folio)PAGE_MAPPING_MOVABLE ; } -static __always_inline bool __PageMovable(const struct page *page) +static __always_inline bool page_has_movable_ops(const struct page *page) {return ((unsigned long )page->mapping &PAGE_MAPPING_FLAGS ) ==PAGE_MAPPING_MOVABLE ;
mm/compaction.c +2 -5
@@ -1056,11 +1056,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,* Skip any other type of page */ if (!PageLRU (page)) { -/* -* __PageMovable can return false positive so we need -* to verify it under page_lock. -*/ -if (unlikely (__PageMovable (page)) && +/* Isolation will grab the page lock. */ +if (unlikely (page_has_movable_ops (page)) && !PageIsolated (page)) {if (locked) {unlock_page_lruvec_irqrestore (locked, flags);
mm/memory-failure.c +2 -2
@@ -1388,8 +1388,8 @@ static inline bool HWPoisonHandlable(struct page *page, unsigned long flags)if (PageSlab (page))return false; -/* Soft offline could migrate non-LRU movable pages */ -if ((flags &MF_SOFT_OFFLINE ) &&__PageMovable (page)) +/* Soft offline could migrate movable_ops pages */ +if ((flags &MF_SOFT_OFFLINE ) &&page_has_movable_ops (page))return true;return PageLRU (page) ||is_free_buddy_page (page);
mm/memory_hotplug.c +3 -5
@@ -1739,8 +1739,8 @@ bool mhp_range_allowed(u64 start, u64 size, bool need_mapping)#ifdef CONFIG_MEMORY_HOTREMOVE /* -* Scan pfn range [start,end) to find movable/migratable pages (LRU pages, -* non-lru movable pages and hugepages). Will skip over most unmovable +* Scan pfn range [start,end) to find movable/migratable pages (LRU and +* hugetlb folio, movable_ops pages). Will skip over most unmovable * pages (esp., pages that can be skipped when offlining), but bail out on * definitely unmovable pages. *
@@ -1759,9 +1759,7 @@ static int scan_movable_pages(unsigned long start, unsigned long end,struct folio *folio; page =pfn_to_page (pfn); -if (PageLRU (page)) - goto found; -if (__PageMovable (page)) +if (PageLRU (page) ||page_has_movable_ops (page)) goto found;/*
mm/migrate.c +4 -4
@@ -94,7 +94,7 @@ bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode)* Note that once a page has movable_ops, it will stay that way * until the page was freed. */ -if (unlikely (!__PageMovable (page))) +if (unlikely (!page_has_movable_ops (page))) goto out_putfolio;/*
@@ -111,7 +111,7 @@ bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode)if (unlikely (!folio_trylock (folio))) goto out_putfolio; -VM_WARN_ON_ONCE_PAGE (!__PageMovable (page), page); +VM_WARN_ON_ONCE_PAGE (!page_has_movable_ops (page), page);if (PageIsolated (page)) goto out_no_isolated;
@@ -153,7 +153,7 @@ static void putback_movable_ops_page(struct page *page)*/ struct folio *folio =page_folio (page); -VM_WARN_ON_ONCE_PAGE (!__PageMovable (page), page); +VM_WARN_ON_ONCE_PAGE (!page_has_movable_ops (page), page);VM_WARN_ON_ONCE_PAGE (!PageIsolated (page), page);folio_lock (folio);page_movable_ops (page)->putback_page (page);
@@ -192,7 +192,7 @@ static int migrate_movable_ops_page(struct page *dst, struct page *src, {int rc =MIGRATEPAGE_SUCCESS ; -VM_WARN_ON_ONCE_PAGE (!__PageMovable (src), src); +VM_WARN_ON_ONCE_PAGE (!page_has_movable_ops (src), src);VM_WARN_ON_ONCE_PAGE (!PageIsolated (src), src); rc =page_movable_ops (src)->migrate_page (dst, src, mode);if (rc ==MIGRATEPAGE_SUCCESS )
mm/page_alloc.c +1 -1
@@ -2005,7 +2005,7 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page,* migration are movable. But we don't actually try * isolating, as that would be expensive. */ -if (PageLRU (page) ||__PageMovable (page)) +if (PageLRU (page) ||page_has_movable_ops (page)) (*num_movable)++; pfn++; }
mm/page_isolation.c +5 -5
@@ -21,9 +21,9 @@* consequently belong to a single zone. * * PageLRU check without isolation or lru_lock could race so that -* MIGRATE_MOVABLE block might include unmovable pages. And __PageMovable -* check without lock_page also may miss some movable non-lru pages at -* race condition. So you can't expect this function should be exact. +* MIGRATE_MOVABLE block might include unmovable pages. Similarly, pages +* with movable_ops can only be identified some time after they were +* allocated. So you can't expect this function should be exact. * * Returns a page without holding a reference. If the caller wants to * dereference that page (e.g., dumping), it has to make sure that it
@@ -133,7 +133,7 @@ static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long eif ((mode ==PB_ISOLATE_MODE_MEM_OFFLINE ) &&PageOffline (page))continue ; -if (__PageMovable (page) ||PageLRU (page)) +if (PageLRU (page) ||page_has_movable_ops (page))continue ;/*
@@ -421,7 +421,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn,* proper free and split handling for them. */ VM_WARN_ON_ONCE_PAGE (PageLRU (page), page); -VM_WARN_ON_ONCE_PAGE (__PageMovable (page), page); +VM_WARN_ON_ONCE_PAGE (page_has_movable_ops (page), page); goto failed; }
Signature · 1 line
2.49.0
Zi Yan
reviewerRe: [PATCH RFC 16/29] mm: rename __PageMovable() to page_has_movable_ops()
Quoted 45 lines · David Hildenbrand
Let's make it clearer that we are talking about
movable_opspages.Signed-off-by David Hildenbrand <[email protected]>--- include/linux/migrate.h | 2 +- include/linux/page-flags.h | 2 +- mm/compaction.c | 7 ++----- mm/memory-failure.c | 4 ++-- mm/memory_hotplug.c | 8 +++----- mm/migrate.c | 8 ++++---- mm/page_alloc.c | 2 +- mm/page_isolation.c | 10 +++++----- 8 files changed, 19 insertions(+), 24 deletions(-)include/linux/migrate.h +1
-1@@ -115,7 +115,7 @@ static inline void __SetPageMovable(struct page *page,static inline const struct movable_operations *page_movable_ops (struct page *page) { -VM_BUG_ON (!__PageMovable (page)); +VM_BUG_ON (!page_has_movable_ops (page));return (const struct movable_operations *) ((unsigned long )page->mapping -PAGE_MAPPING_MOVABLE );include/linux/page-flags.h +1
-1@@ -750,7 +750,7 @@ static __always_inline bool __folio_test_movable(const struct folio *folio)PAGE_MAPPING_MOVABLE ; } -static __always_inline bool __PageMovable(const struct page *page) +static __always_inline bool page_has_movable_ops(const struct page *page) {return ((unsigned long )page->mapping &PAGE_MAPPING_FLAGS ) ==PAGE_MAPPING_MOVABLE ;mm/compaction.c +0
-0
mm/compaction.c isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, · line 1059
Quoted 9 lines
mm/compaction.c +1
-5@@ -1056,11 +1056,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,* Skip any other type of page */ if (!PageLRU (page)) { -/* -* __PageMovable can return false positive so we need -* to verify it under page_lock. -*/ - if (unlikely(__PageMovable(page)) && +/* Isolation will grab the page lock. */
I feel that the removed comment should stay, since the current comment makes no sense when I read it alone.
In addition, why is __PageMovable() is renamed to page_has_movable_ops() but __SetPageMovable() stays the same? page_has_movable_ops() and __SetPageMovable() are functions for checking and setting PAGE_MAPPING_MOVABLE. The naming just does not look symmetric.
--
Best Regards,
Yan, Zi
Re: [PATCH RFC 16/29] mm: rename __PageMovable() to page_has_movable_ops()
Quoted 51 lines · Zi Yan
Quoted 49 lines · David Hildenbrand
Let's make it clearer that we are talking about
movable_opspages.Signed-off-by David Hildenbrand <[email protected]>--- include/linux/migrate.h | 2 +- include/linux/page-flags.h | 2 +- mm/compaction.c | 7 ++----- mm/memory-failure.c | 4 ++-- mm/memory_hotplug.c | 8 +++----- mm/migrate.c | 8 ++++---- mm/page_alloc.c | 2 +- mm/page_isolation.c | 10 +++++----- 8 files changed, 19 insertions(+), 24 deletions(-)include/linux/migrate.h +1
-1@@ -115,7 +115,7 @@ static inline void __SetPageMovable(struct page *page,static inline const struct movable_operations *page_movable_ops (struct page *page) { -VM_BUG_ON (!__PageMovable (page)); +VM_BUG_ON (!page_has_movable_ops (page));return (const struct movable_operations *) ((unsigned long )page->mapping -PAGE_MAPPING_MOVABLE );include/linux/page-flags.h +1
-1@@ -750,7 +750,7 @@ static __always_inline bool __folio_test_movable(const struct folio *folio)PAGE_MAPPING_MOVABLE ; } -static __always_inline bool __PageMovable(const struct page *page) +static __always_inline bool page_has_movable_ops(const struct page *page) {return ((unsigned long )page->mapping &PAGE_MAPPING_FLAGS ) ==PAGE_MAPPING_MOVABLE ;mm/compaction.c +0
-1@@ -1056,11 +1056,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,* Skip any other type of page */ if (!PageLRU (page)) { -/*
Quoted 8 lines
mm/compaction.c isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, · line 1059
Quoted 5 lines
mm/compaction.c +1
-4-* __PageMovable can return false positive so we need -* to verify it under page_lock. -*/ - if (unlikely(__PageMovable(page)) && +/* Isolation will grab the page lock. */ I feel that the removed comment should stay, since the current comment makes no sense when I read it alone.
Well, talking about the page lock is moot either way. The thing is, anything can change while we don't hold a page reference. So should we change the comment to
/* isolation code will deal with any races. */
... or drop it completely?
Quoted 5 lines
In addition, why is
__PageMovable()is renamed topage_has_movable_ops()but__SetPageMovable()stays the same?page_has_movable_ops()and__SetPageMovable()are functions for checking and settingPAGE_MAPPING_MOVABLE. The naming just does not look symmetric.
See follow-up commits where __SetPageMovable() is cleaned up.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH RFC 16/29] mm: rename __PageMovable() to page_has_movable_ops()
Quoted 61 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 49 lines · David Hildenbrand
Let's make it clearer that we are talking about
movable_opspages.Signed-off-by David Hildenbrand <[email protected]>--- include/linux/migrate.h | 2 +- include/linux/page-flags.h | 2 +- mm/compaction.c | 7 ++----- mm/memory-failure.c | 4 ++-- mm/memory_hotplug.c | 8 +++----- mm/migrate.c | 8 ++++---- mm/page_alloc.c | 2 +- mm/page_isolation.c | 10 +++++----- 8 files changed, 19 insertions(+), 24 deletions(-)include/linux/migrate.h +1
-1@@ -115,7 +115,7 @@ static inline void __SetPageMovable(struct page *page,static inline const struct movable_operations *page_movable_ops (struct page *page) { -VM_BUG_ON (!__PageMovable (page)); +VM_BUG_ON (!page_has_movable_ops (page));return (const struct movable_operations *) ((unsigned long )page->mapping -PAGE_MAPPING_MOVABLE );include/linux/page-flags.h +1
-1@@ -750,7 +750,7 @@ static __always_inline bool __folio_test_movable(const struct folio *folio)PAGE_MAPPING_MOVABLE ; } -static __always_inline bool __PageMovable(const struct page *page) +static __always_inline bool page_has_movable_ops(const struct page *page) {return ((unsigned long )page->mapping &PAGE_MAPPING_FLAGS ) ==PAGE_MAPPING_MOVABLE ;mm/compaction.c +0
-1@@ -1056,11 +1056,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,* Skip any other type of page */ if (!PageLRU (page)) { -/* Quoted 8 lines
mm/compaction.c isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, · line 1059
Quoted 5 lines
mm/compaction.c +1
-4-* __PageMovable can return false positive so we need -* to verify it under page_lock. -*/ - if (unlikely(__PageMovable(page)) && +/* Isolation will grab the page lock. */ I feel that the removed comment should stay, since the current comment makes no sense when I read it alone.
Quoted 3 lines
Well, talking about the page lock is moot either way. The thing is, anything can change while we don't hold a page reference. So should we change the comment to
/* isolation code will deal with any races. */
Sounds good to me.
Quoted 10 lines
... or drop it completely?
Quoted 5 lines
In addition, why is
__PageMovable()is renamed topage_has_movable_ops()but__SetPageMovable()stays the same?page_has_movable_ops()and__SetPageMovable()are functions for checking and settingPAGE_MAPPING_MOVABLE. The naming just does not look symmetric.See follow-up commits where
__SetPageMovable()is cleaned up.
Right. It becomes clear at Patch 20.
--
Best Regards,
Yan, Zi
RFC PATCH 17/29 mm/page_isolation: drop __folio_test_movable() check for large folios
In reply to David HildenbrandCurrently, we only support migration of individual non-folio pages, so we can not run into that.
--- mm/page_isolation.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_isolation.c +1 -1
@@ -92,7 +92,7 @@ static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long e h =size_to_hstate (folio_size (folio));if (h && !hugepage_migration_supported (h))return page; - }else if (!folio_test_lru (folio) && !__folio_test_movable (folio)) { + }else if (!folio_test_lru (folio)) {return page; }
Signature · 1 line
2.49.0
Re: [PATCH RFC 17/29] mm/page_isolation: drop __folio_test_movable() check for large folios
Quoted 8 lines · David Hildenbrand
Currently, we only support migration of individual non-folio pages, so we can not run into that.
Signed-off-by David Hildenbrand <[email protected]>--- mm/page_isolation.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
--
Best Regards,
Yan, Zi
RFC PATCH 18/29 mm: remove __folio_test_movable()
In reply to David HildenbrandConvert to page_has_movable_ops(). While at it, cleanup relevant code a bit.
The data_race() in migrate_folio_unmap() is questionable: we already hold a page reference, and concurrent modifications can no longer happen (iow: __ClearPageMovable() no longer exists). Drop it for now, we'll rework page_has_movable_ops() soon either way to no longer rely on page->mapping.
Wherever we cast from folio to page now is a clear sign that this code has to be decoupled.
--- include/linux/page-flags.h | 6 ------ mm/migrate.c | 43 ++++++++++++-------------------------- mm/vmscan.c | 6 ++++-- 3 files changed, 17 insertions(+), 38 deletions(-)
include/linux/page-flags.h +0 -6
@@ -744,12 +744,6 @@ static __always_inline bool PageAnon(const struct page *page)return folio_test_anon (page_folio (page)); } -static __always_inline bool __folio_test_movable(const struct folio *folio) -{ -return ((unsigned long )folio->mapping &PAGE_MAPPING_FLAGS ) == -PAGE_MAPPING_MOVABLE ; -} - static __always_inline bool page_has_movable_ops(const struct page *page) { return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) ==
mm/migrate.c +13 -30
84 lines not on this page · raw message
mm/vmscan.c +4 -2
@@ -1651,9 +1651,11 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,unsigned int noreclaim_flag;list_for_each_entry_safe (folio, next, folio_list, lru) { +/* TODO: these pages should not even appear in this list. */ +if (page_has_movable_ops (&folio->page)) +continue ;if (!folio_test_hugetlb (folio) &&folio_is_file_lru (folio) && - !folio_test_dirty (folio) && !__folio_test_movable (folio) && - !folio_test_unevictable (folio)) { + !folio_test_dirty (folio) && !folio_test_unevictable (folio)) {folio_clear_active (folio);list_move (&folio->lru, &clean_folios); }
Signature · 1 line
2.49.0
Re: [PATCH RFC 18/29] mm: remove __folio_test_movable()
Quoted 12 lines · David Hildenbrand
Convert to
page_has_movable_ops(). While at it, cleanup relevant code a bit.The
data_race()inmigrate_folio_unmap()is questionable: we already hold a page reference, and concurrent modifications can no longer happen (iow:__ClearPageMovable()no longer exists). Drop it for now, we'll reworkpage_has_movable_ops()soon either way to no longer rely onpage->mapping.Wherever we cast from folio to page now is a clear sign that this code has to be decoupled.
Quoted 7 lines
Signed-off-by David Hildenbrand <[email protected]>--- include/linux/page-flags.h | 6 ------ mm/migrate.c | 43 ++++++++++++-------------------------- mm/vmscan.c | 6 ++++-- 3 files changed, 17 insertions(+), 38 deletions(-)
--
Best Regards,
Yan, Zi
Re: [PATCH RFC 18/29] mm: remove __folio_test_movable()
In reply to David HildenbrandQuoted 9 lines
mm/migrate.c +0
-1@@ -1164,7 +1148,6 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,int rc = -EAGAIN ;int old_page_state =0 ;struct anon_vma *anon_vma =NULL ; -bool is_lru =data_race (!__folio_test_movable (src));bool locked = false;bool dst_locked = false;
mm/migrate.c static int migrate_folio_unmap(new_folio_t get_new_folio, · line 1251
Quoted 5 lines
mm/migrate.c +1
-1@@ -1265,7 +1248,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, goto out; dst_locked = true; -if (unlikely (!is_lru)) { +if (unlikely (page_has_movable_ops (&dst->page))) {
While doing more testing, I found a weird bug where we end up having balloon pages isolated forever, and never putback/migrated. So neither the balloon can deflate them nor the migration migrate them.
This line here must be
if (unlikely(page_has_movable_ops(&src->page))) {
But I am not sure yet if that's the issue I am chasing :)
Signature · 3 lines
Cheers, David / dhildenb
RFC PATCH 19/29 mm: stop storing migration_ops in page->mapping
In reply to David Hildenbrand... instead, look them up statically based on the page type. Maybe in the future we want a registration interface? At least for now, it can be easily handled using the two page types that actually support page migration.
The remaining usage of page->mapping is to flag such pages as actually being movable (having movable_ops), which we will change next.
--- include/linux/balloon_compaction.h | 2 +- include/linux/migrate.h | 14 ++------------ include/linux/zsmalloc.h | 2 ++ mm/balloon_compaction.c | 1 - mm/compaction.c | 5 ++--- mm/migrate.c | 23 +++++++++++++++++++++++ mm/zpdesc.h | 5 ++--- mm/zsmalloc.c | 8 +++----- 8 files changed, 35 insertions(+), 25 deletions(-)
include/linux/balloon_compaction.h +1 -1
@@ -92,7 +92,7 @@ static inline void balloon_page_insert(struct balloon_dev_info *balloon, struct page *page) {__SetPageOffline (page); -__SetPageMovable (page, &balloon_mops); +__SetPageMovable (page);set_page_private (page, (unsigned long )balloon);list_add (&page->lru, &balloon->pages); }
include/linux/migrate.h +2 -12
@@ -104,23 +104,13 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#endif /* CONFIG_MIGRATION */ #ifdef CONFIG_COMPACTION -void __SetPageMovable (struct page *page,const struct movable_operations *ops); +void __SetPageMovable (struct page *page);#else -static inline void __SetPageMovable (struct page *page, -const struct movable_operations *ops) +static inline void __SetPageMovable (struct page *page) { }#endif -static inline -const struct movable_operations *page_movable_ops (struct page *page) -{ -VM_BUG_ON (!page_has_movable_ops (page)); - -return (const struct movable_operations *) - ((unsigned long )page->mapping -PAGE_MAPPING_MOVABLE ); -} -#ifdef CONFIG_NUMA_BALANCING int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node);
include/linux/zsmalloc.h +2 -0
@@ -46,4 +46,6 @@ void zs_obj_read_end(struct zs_pool *pool, unsigned long handle,void zs_obj_write (struct zs_pool *pool,unsigned long handle,void *handle_mem,size_t mem_len); +extern const struct movable_operations zsmalloc_mops; +#endif
mm/balloon_compaction.c +0 -1
@@ -253,6 +253,5 @@ const struct movable_operations balloon_mops = { .isolate_page = balloon_page_isolate, .putback_page = balloon_page_putback, }; -EXPORT_SYMBOL_GPL (balloon_mops);#endif /* CONFIG_BALLOON_COMPACTION */
mm/compaction.c +2 -3
@@ -114,11 +114,10 @@ static unsigned long release_free_list(struct list_head *freepages) }#ifdef CONFIG_COMPACTION -void __SetPageMovable (struct page *page,const struct movable_operations *mops) +void __SetPageMovable (struct page *page) {VM_BUG_ON_PAGE (!PageLocked (page), page); -VM_BUG_ON_PAGE ((unsigned long )mops &PAGE_MAPPING_MOVABLE , page); - page->mapping = (void *)((unsigned long )mops |PAGE_MAPPING_MOVABLE ); + page->mapping = (void *)(PAGE_MAPPING_MOVABLE ); }EXPORT_SYMBOL (__SetPageMovable);
mm/migrate.c +23 -0
@@ -43,6 +43,8 @@#include <linux/sched/sysctl.h> #include <linux/memory-tiers.h> #include <linux/pagewalk.h> +#include <linux/balloon_compaction.h> +#include <linux/zsmalloc.h> #include <asm/tlbflush.h>
@@ -51,6 +53,27 @@#include "internal.h" #include "swap.h" +static const struct movable_operations *page_movable_ops (struct page *page) +{ +VM_WARN_ON_ONCE_PAGE (!page_has_movable_ops (page), page); + +/* +* If we enable page migration for a page of a certain type by marking +* it as movable, the page type must be sticky until the page gets freed +* back to the buddy. +*/ +#ifdef CONFIG_BALLOON_COMPACTION +if (PageOffline (page)) +/* Only balloon compaction sets PageOffline pages movable. */ +return &balloon_mops; +#endif /* CONFIG_BALLOON_COMPACTION */ +#ifdef CONFIG_ZSMALLOC +if (PageZsmalloc (page)) +return &zsmalloc_mops; +#endif /* CONFIG_ZSMALLOC */ +return NULL ; +} +/** * isolate_movable_ops_page - isolate a movable_ops page for migration * @page: The page.
mm/zpdesc.h +2 -3
@@ -152,10 +152,9 @@ static inline struct zpdesc *pfn_zpdesc(unsigned long pfn)return page_zpdesc (pfn_to_page (pfn)); } -static inline void __zpdesc_set_movable (struct zpdesc *zpdesc, -const struct movable_operations *mops) +static inline void __zpdesc_set_movable (struct zpdesc *zpdesc) { -__SetPageMovable (zpdesc_page (zpdesc), mops); +__SetPageMovable (zpdesc_page (zpdesc)); }static inline void __zpdesc_set_zsmalloc (struct zpdesc *zpdesc)
mm/zsmalloc.c +3 -5
@@ -1684,8 +1684,6 @@ static void lock_zspage(struct zspage *zspage)#ifdef CONFIG_COMPACTION -static const struct movable_operations zsmalloc_mops; -static void replace_sub_page (struct size_class *class,struct zspage *zspage,struct zpdesc *newzpdesc,struct zpdesc *oldzpdesc) {
@@ -1708,7 +1706,7 @@ static void replace_sub_page(struct size_class *class, struct zspage *zspage,set_first_obj_offset (newzpdesc, first_obj_offset);if (unlikely (ZsHugePage (zspage))) newzpdesc->handle = oldzpdesc->handle; -__zpdesc_set_movable (newzpdesc, &zsmalloc_mops); +__zpdesc_set_movable (newzpdesc); }static bool zs_page_isolate (struct page *page,isolate_mode_t mode)
@@ -1815,7 +1813,7 @@ static void zs_page_putback(struct page *page) { } -static const struct movable_operations zsmalloc_mops = { +const struct movable_operations zsmalloc_mops = { .isolate_page = zs_page_isolate, .migrate_page = zs_page_migrate, .putback_page = zs_page_putback,
@@ -1878,7 +1876,7 @@ static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage)do {WARN_ON (!zpdesc_trylock (zpdesc)); -__zpdesc_set_movable (zpdesc, &zsmalloc_mops); +__zpdesc_set_movable (zpdesc);zpdesc_unlock (zpdesc); }while ((zpdesc =get_next_zpdesc (zpdesc)) !=NULL ); }
Signature · 1 line
2.49.0
Re: [PATCH RFC 19/29] mm: stop storing migration_ops in page->mapping
Quoted 9 lines · David Hildenbrand
... instead, look them up statically based on the page type. Maybe in the future we want a registration interface? At least for now, it can be easily handled using the two page types that actually support page migration.
The remaining usage of
page->mappingis to flag such pages as actually being movable (havingmovable_ops), which we will change next.Signed-off-by David Hildenbrand <[email protected]>
Quoted 11 lines
--- include/linux/balloon_compaction.h | 2 +- include/linux/migrate.h | 14 ++------------ include/linux/zsmalloc.h | 2 ++ mm/balloon_compaction.c | 1 - mm/compaction.c | 5 ++--- mm/migrate.c | 23 +++++++++++++++++++++++ mm/zpdesc.h | 5 ++--- mm/zsmalloc.c | 8 +++----- 8 files changed, 35 insertions(+), 25 deletions(-)
--
Best Regards,
Yan, Zi
RFC PATCH 20/29 mm: convert "movable" flag in page->mapping to a page flag
In reply to David HildenbrandInstead, let's use a page flag. As the page flag can result in false-positives, glue it to the page types for which we support/implement movable_ops page migration.
--- include/linux/balloon_compaction.h | 2 +- include/linux/migrate.h | 8 ----- include/linux/page-flags.h | 52 ++++++++++++++++++++++++------ mm/compaction.c | 6 ---- mm/zpdesc.h | 2 +- 5 files changed, 44 insertions(+), 26 deletions(-)
include/linux/balloon_compaction.h +1 -1
@@ -92,7 +92,7 @@ static inline void balloon_page_insert(struct balloon_dev_info *balloon, struct page *page) {__SetPageOffline (page); -__SetPageMovable (page); +SetPageMovableOps (page);set_page_private (page, (unsigned long )balloon);list_add (&page->lru, &balloon->pages); }
include/linux/migrate.h +0 -8
@@ -103,14 +103,6 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#endif /* CONFIG_MIGRATION */ -#ifdef CONFIG_COMPACTION -void __SetPageMovable (struct page *page); -#else -static inline void __SetPageMovable (struct page *page) -{ -} -#endif -#ifdef CONFIG_NUMA_BALANCING int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node);
include/linux/page-flags.h +42 -10
76 lines not on this page · raw message
mm/compaction.c +0 -6
@@ -114,12 +114,6 @@ static unsigned long release_free_list(struct list_head *freepages) }#ifdef CONFIG_COMPACTION -void __SetPageMovable (struct page *page) -{ -VM_BUG_ON_PAGE (!PageLocked (page), page); - page->mapping = (void *)(PAGE_MAPPING_MOVABLE ); -} -EXPORT_SYMBOL (__SetPageMovable);/* Do not skip compaction more than 64 times */ #define COMPACT_MAX_DEFER_SHIFT 6
mm/zpdesc.h +1 -1
@@ -154,7 +154,7 @@ static inline struct zpdesc *pfn_zpdesc(unsigned long pfn)static inline void __zpdesc_set_movable (struct zpdesc *zpdesc) { -__SetPageMovable (zpdesc_page (zpdesc)); +SetPageMovableOps (zpdesc_page (zpdesc)); }static inline void __zpdesc_set_zsmalloc (struct zpdesc *zpdesc)
Signature · 1 line
2.49.0
Zi Yan
reviewerRe: [PATCH RFC 20/29] mm: convert "movable" flag in page->mapping to a page flag
Quoted 13 lines · David Hildenbrand
Instead, let's use a page flag. As the page flag can result in false-positives, glue it to the page types for which we support/implement
movable_opspage migration.Signed-off-by David Hildenbrand <[email protected]>--- include/linux/balloon_compaction.h | 2 +- include/linux/migrate.h | 8 ----- include/linux/page-flags.h | 52 ++++++++++++++++++++++++------ mm/compaction.c | 6 ---- mm/zpdesc.h | 2 +- 5 files changed, 44 insertions(+), 26 deletions(-)
<snip>
Quoted 6 lines
include/linux/page-flags.h +6
-0+ +/** +* page_has_movable_ops - test for a movable_ops page +* @page The page to test. +* +* Test whether this is a movable_ops page. Such pages will stay that
include/linux/page-flags.h bool is_free_buddy_page(const struct page *page); · line 1167
Quoted 10 lines
include/linux/page-flags.h +10
-0+* way until freed. +* +* Returns true if this is a movable_ops page, otherwise false. +*/ +static inline bool page_has_movable_ops (const struct page *page) +{ +return PageMovableOps (page) && + (PageOffline (page) ||PageZsmalloc (page)); +} +
Should we do the code below in case PageMovableOps is set on pages other than PageOffline and PageZsmalloc?
return PageMovableOps(page) &&
!VM_WARN_ON_ONCE_PAGE(!(PageOffline(page) || PageZsmalloc(page)), page);
Quoted 35 lines
static
__always_inlineintPageAnonExclusive(const struct page *page){include/linux/page-flags.h +0
-0VM_BUG_ON_PGFLAGS (!PageAnon (page), page);mm/compaction.c +0
-6@@ -114,12 +114,6 @@ static unsigned long release_free_list(struct list_head *freepages) }#ifdef CONFIG_COMPACTION -void __SetPageMovable (struct page *page) -{ -VM_BUG_ON_PAGE (!PageLocked (page), page); - page->mapping = (void *)(PAGE_MAPPING_MOVABLE ); -} -EXPORT_SYMBOL (__SetPageMovable);/* Do not skip compaction more than 64 times */ #define COMPACT_MAX_DEFER_SHIFT 6mm/zpdesc.h +1
-1@@ -154,7 +154,7 @@ static inline struct zpdesc *pfn_zpdesc(unsigned long pfn)static inline void __zpdesc_set_movable (struct zpdesc *zpdesc) { -__SetPageMovable (zpdesc_page (zpdesc)); +SetPageMovableOps (zpdesc_page (zpdesc)); }static inline void __zpdesc_set_zsmalloc (struct zpdesc *zpdesc)Signature · 1 line
2.49.0
Otherwise, LGTM. Reviewed-by: Zi Yan <[email protected]>
--
Best Regards,
Yan, Zi
Re: [PATCH RFC 20/29] mm: convert "movable" flag in page->mapping to a page flag
Quoted 38 lines · Zi Yan
Quoted 13 lines · David Hildenbrand
Instead, let's use a page flag. As the page flag can result in false-positives, glue it to the page types for which we support/implement
movable_opspage migration.Signed-off-by David Hildenbrand <[email protected]>--- include/linux/balloon_compaction.h | 2 +- include/linux/migrate.h | 8 ----- include/linux/page-flags.h | 52 ++++++++++++++++++++++++------ mm/compaction.c | 6 ---- mm/zpdesc.h | 2 +- 5 files changed, 44 insertions(+), 26 deletions(-)<snip>
Quoted 6 lines
include/linux/page-flags.h +6
-0+ +/** +* page_has_movable_ops - test for a movable_ops page +* @page The page to test. +* +* Test whether this is a movable_ops page. Such pages will stay that include/linux/page-flags.h bool is_free_buddy_page(const struct page *page); · line 1167
Quoted 10 lines
include/linux/page-flags.h +10
-0+* way until freed. +* +* Returns true if this is a movable_ops page, otherwise false. +*/ +static inline bool page_has_movable_ops (const struct page *page) +{ +return PageMovableOps (page) && + (PageOffline (page) ||PageZsmalloc (page)); +} +Should we do the code below in case PageMovableOps is set on pages other than PageOffline and PageZsmalloc?
Quoted 3 lines
return PageMovableOps(page) &&
!VM_WARN_ON_ONCE_PAGE(!(PageOffline(page) || PageZsmalloc(page)), page);
I think this could trigger in some corner when other pages (esp. folios?) reuse the flag and we run into some weird races while we don't hold any reference on the page.
Signature · 3 lines
Cheers, David / dhildenb
RFC PATCH 21/29 mm: rename PG_isolated to PG_movable_ops_isolated
In reply to David HildenbrandLet's rename the flag to make it clearer where it applies (not folios ...).
While at it, define the flag only with CONFIG_MIGRATION.
--- include/linux/page-flags.h | 16 +++++++++++----- mm/compaction.c | 2 +- mm/migrate.c | 14 +++++++------- 3 files changed, 19 insertions(+), 13 deletions(-)
include/linux/page-flags.h +11 -5
@@ -167,10 +167,9 @@ enum pageflags {/* Remapped by swiotlb-xen. */ PG_xen_remapped = PG_owner_priv_1, -/* non-lru isolated movable page */ - PG_isolated = PG_reclaim, -#ifdef CONFIG_MIGRATION +/* movable_ops page that is isolated for migration */ + PG_movable_ops_isolated = PG_reclaim, /* this is a movable_ops page (for selected typed pages only) */ PG_movable_ops = PG_uptodate,#endif
@@ -1126,8 +1125,6 @@ static inline bool folio_contain_hwpoisoned_page(struct folio *folio)bool is_free_buddy_page (const struct page *page); -PAGEFLAG (Isolated, isolated,PF_ANY ); -#ifdef CONFIG_MIGRATION /* * This page is migratable through movable_ops (for selected typed pages
@@ -1146,8 +1143,17 @@ PAGEFLAG(Isolated, isolated, PF_ANY);* page_has_movable_ops() instead. */ PAGEFLAG (MovableOps, movable_ops,PF_NO_TAIL ); +/* +* A movable_ops page has this flag set while it is isolated for migration. +* This flag primarily protects against concurrent migration attempts. +* +* Once migration ended (success or failure), the flag is cleared. The +* flag is managed by the migration core. +*/ +PAGEFLAG (MovableOpsIsolated, movable_ops_isolated,PF_NO_TAIL );#else PAGEFLAG_FALSE (MovableOps, movable_ops); +PAGEFLAG_FALSE (MovableOpsIsolated, movable_ops_isolated);#endif /**
mm/compaction.c +1 -1
@@ -1051,7 +1051,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,if (!PageLRU (page)) {/* Isolation will grab the page lock. */ if (unlikely (page_has_movable_ops (page)) && - !PageIsolated (page)) { + !PageMovableOpsIsolated (page)) {if (locked) {unlock_page_lruvec_irqrestore (locked, flags); locked =NULL ;
mm/migrate.c +7 -7
@@ -135,7 +135,7 @@ bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode) goto out_putfolio;VM_WARN_ON_ONCE_PAGE (!page_has_movable_ops (page), page); -if (PageIsolated (page)) +if (PageMovableOpsIsolated (page)) goto out_no_isolated; mops =page_movable_ops (page);
@@ -146,8 +146,8 @@ bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode) goto out_no_isolated;/* Driver shouldn't use the isolated flag */ -VM_WARN_ON_ONCE_PAGE (PageIsolated (page), page); -SetPageIsolated (page); +VM_WARN_ON_ONCE_PAGE (PageMovableOpsIsolated (page), page); +SetPageMovableOpsIsolated (page);folio_unlock (folio);return true;
@@ -177,10 +177,10 @@ static void putback_movable_ops_page(struct page *page)struct folio *folio =page_folio (page);VM_WARN_ON_ONCE_PAGE (!page_has_movable_ops (page), page); -VM_WARN_ON_ONCE_PAGE (!PageIsolated (page), page); +VM_WARN_ON_ONCE_PAGE (!PageMovableOpsIsolated (page), page);folio_lock (folio);page_movable_ops (page)->putback_page (page); -ClearPageIsolated (page); +ClearPageMovableOpsIsolated (page);folio_unlock (folio);folio_put (folio); }
@@ -216,10 +216,10 @@ static int migrate_movable_ops_page(struct page *dst, struct page *src,int rc =MIGRATEPAGE_SUCCESS ;VM_WARN_ON_ONCE_PAGE (!page_has_movable_ops (src), src); -VM_WARN_ON_ONCE_PAGE (!PageIsolated (src), src); +VM_WARN_ON_ONCE_PAGE (!PageMovableOpsIsolated (src), src); rc =page_movable_ops (src)->migrate_page (dst, src, mode);if (rc ==MIGRATEPAGE_SUCCESS ) -ClearPageIsolated (src); +ClearPageMovableOpsIsolated (src);return rc; }
Signature · 1 line
2.49.0
Re: [PATCH RFC 21/29] mm: rename PG_isolated to PG_movable_ops_isolated
Quoted 12 lines · David Hildenbrand
Let's rename the flag to make it clearer where it applies (not folios ...).
While at it, define the flag only with
CONFIG_MIGRATION.Signed-off-by David Hildenbrand <[email protected]>--- include/linux/page-flags.h | 16 +++++++++++----- mm/compaction.c | 2 +- mm/migrate.c | 14 +++++++------- 3 files changed, 19 insertions(+), 13 deletions(-)
--
Best Regards,
Yan, Zi
RFC PATCH 22/29 mm/page-flags: rename PAGE_MAPPING_MOVABLE to PAGE_MAPPING_ANON_KSM
In reply to David HildenbrandKSM is the only remaining user, let's rename the flag. While at it, adjust to remaining page -> folio in the doc.
--- include/linux/page-flags.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
include/linux/page-flags.h +6 -6
@@ -697,10 +697,10 @@ PAGEFLAG_FALSE(VmemmapSelfHosted, vmemmap_self_hosted)* folio->mapping points to its anon_vma, not to a struct address_space; * with the PAGE_MAPPING_ANON bit set to distinguish it. See rmap.h. * -* On an anonymous page in a VM_MERGEABLE area, if CONFIG_KSM is enabled, -* the PAGE_MAPPING_MOVABLE bit may be set along with the PAGE_MAPPING_ANON +* On an anonymous folio in a VM_MERGEABLE area, if CONFIG_KSM is enabled, +* the PAGE_MAPPING_ANON_KSM bit may be set along with the PAGE_MAPPING_ANON * bit; and then folio->mapping points, not to an anon_vma, but to a private -* structure which KSM associates with that merged page. See ksm.h. +* structure which KSM associates with that merged folio. See ksm.h. * * Please note that, confusingly, "folio_mapping" refers to the inode * address_space which maps the folio from disk; whereas "folio_mapped"
@@ -714,9 +714,9 @@ PAGEFLAG_FALSE(VmemmapSelfHosted, vmemmap_self_hosted)* See mm/slab.h. */ #define PAGE_MAPPING_ANON 0x1 -#define PAGE_MAPPING_MOVABLE 0x2 -#define PAGE_MAPPING_KSM (PAGE_MAPPING_ANON | PAGE_MAPPING_MOVABLE) -#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_MOVABLE) +#define PAGE_MAPPING_ANON_KSM 0x2 +#define PAGE_MAPPING_KSM (PAGE_MAPPING_ANON | PAGE_MAPPING_ANON_KSM) +#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_ANON_KSM) static __always_inline bool folio_mapping_flags(const struct folio *folio) {
Signature · 1 line
2.49.0
Re: [PATCH RFC 22/29] mm/page-flags: rename PAGE_MAPPING_MOVABLE to PAGE_MAPPING_ANON_KSM
Quoted 8 lines · David Hildenbrand
KSM is the only remaining user, let's rename the flag. While at it, adjust to remaining page -> folio in the doc.
Signed-off-by David Hildenbrand <[email protected]>--- include/linux/page-flags.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
--
Best Regards,
Yan, Zi
RFC PATCH 23/29 mm/page-alloc: remove PageMappingFlags()
In reply to David HildenbrandWe can now simply check for PageAnon() and remove PageMappingFlags().
... and while at it, use the folio instead and operate on folio->mapping.
--- include/linux/page-flags.h | 5 ----- mm/page_alloc.c | 7 +++---- 2 files changed, 3 insertions(+), 9 deletions(-)
include/linux/page-flags.h +0 -5
@@ -723,11 +723,6 @@ static __always_inline bool folio_mapping_flags(const struct folio *folio)return ((unsigned long )folio->mapping &PAGE_MAPPING_FLAGS ) !=0 ; } -static __always_inline bool PageMappingFlags(const struct page *page) -{ -return ((unsigned long )page->mapping &PAGE_MAPPING_FLAGS ) !=0 ; -} - static __always_inline bool folio_test_anon(const struct folio *folio) {return ((unsigned long )folio->mapping &PAGE_MAPPING_ANON ) !=0 ;
mm/page_alloc.c +3 -4
@@ -1375,10 +1375,9 @@ __always_inline bool free_pages_prepare(struct page *page, (page + i)->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ; } } -if (PageMappingFlags (page)) { -if (PageAnon (page)) -mod_mthp_stat (order,MTHP_STAT_NR_ANON ,-1 ); - page->mapping =NULL ; +if (folio_test_anon (folio)) { +mod_mthp_stat (order,MTHP_STAT_NR_ANON ,-1 ); + folio->mapping =NULL ; }if (unlikely (page_has_type (page))) page->page_type =UINT_MAX ;
Signature · 1 line
2.49.0
Re: [PATCH RFC 23/29] mm/page-alloc: remove PageMappingFlags()
Quoted 11 lines · David Hildenbrand
We can now simply check for
PageAnon()and removePageMappingFlags().... and while at it, use the folio instead and operate on
folio->mapping.Signed-off-by David Hildenbrand <[email protected]>--- include/linux/page-flags.h | 5 ----- mm/page_alloc.c | 7 +++---- 2 files changed, 3 insertions(+), 9 deletions(-)
--
Best Regards,
Yan, Zi
RFC PATCH 24/29 mm/page-flags: remove folio_mapping_flags()
In reply to David HildenbrandIt's unused and the page counterpart is gone, so let's remove it.
--- include/linux/page-flags.h | 5 ----- 1 file changed, 5 deletions(-)
include/linux/page-flags.h +0 -5
@@ -718,11 +718,6 @@ PAGEFLAG_FALSE(VmemmapSelfHosted, vmemmap_self_hosted)#define PAGE_MAPPING_KSM (PAGE_MAPPING_ANON | PAGE_MAPPING_ANON_KSM)#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_ANON_KSM) -static __always_inline bool folio_mapping_flags(const struct folio *folio) -{ -return ((unsigned long )folio->mapping &PAGE_MAPPING_FLAGS ) !=0 ; -} - static __always_inline bool folio_test_anon(const struct folio *folio) {return ((unsigned long )folio->mapping &PAGE_MAPPING_ANON ) !=0 ;
Signature · 1 line
2.49.0
Re: [PATCH RFC 24/29] mm/page-flags: remove folio_mapping_flags()
Quoted 7 lines · David Hildenbrand
It's unused and the page counterpart is gone, so let's remove it.
Signed-off-by David Hildenbrand <[email protected]>--- include/linux/page-flags.h | 5 ----- 1 file changed, 5 deletions(-)
--
Best Regards,
Yan, Zi
RFC PATCH 25/29 mm: simplify folio_expected_ref_count()
In reply to David HildenbrandNow that PAGE_MAPPING_MOVABLE is gone, we can simplify and rely on the folio_test_anon() test only.
... but staring at the users, this function should never even have been called on movable_ops pages. E.g.,
* __buffer_migrate_folio() does not make sense for them
* folio_migrate_mapping() does not make sense for them
* migrate_huge_page_move_mapping() does not make sense for them
* __migrate_folio() does not make sense for them
* ... and khugepaged should never stumble over them
Let's simply refuse typed pages (which includes slab) except hugetlb, and WARN.
--- include/linux/mm.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/mm.h +2 -2
@@ -2167,13 +2167,13 @@ static inline int folio_expected_ref_count(const struct folio *folio)const int order =folio_order (folio);int ref_count =0 ; -if (WARN_ON_ONCE (folio_test_slab (folio))) +if (WARN_ON_ONCE (page_has_type (&folio->page) && !folio_test_hugetlb (folio)))return 0 ;if (folio_test_anon (folio)) {/* One reference per page from the swapcache. */ ref_count +=folio_test_swapcache (folio) << order; - }else if (!((unsigned long )folio->mapping &PAGE_MAPPING_FLAGS )) { + }else {/* One reference per page from the pagecache. */ ref_count += !!folio->mapping << order;/* One reference from PG_private. */
Signature · 1 line
2.49.0
Re: [PATCH RFC 25/29] mm: simplify folio_expected_ref_count()
Quoted 11 lines · David Hildenbrand
Now that
PAGE_MAPPING_MOVABLEis gone, we can simplify and rely on thefolio_test_anon()test only.... but staring at the users, this function should never even have been called on
movable_opspages. E.g.,
*__buffer_migrate_folio()does not make sense for them
*folio_migrate_mapping()does not make sense for them
*migrate_huge_page_move_mapping()does not make sense for them
*__migrate_folio()does not make sense for them
* ... and khugepaged should never stumble over them
Quoted 8 lines
Let's simply refuse typed pages (which includes slab) except hugetlb, and WARN.
Signed-off-by David Hildenbrand <[email protected]>--- include/linux/mm.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
--
Best Regards,
Yan, Zi
RFC PATCH 26/29 mm: rename PAGE_MAPPING_* to FOLIO_MAPPING_*
In reply to David HildenbrandNow that the mapping flags are only used for folios, let's rename the defines.
--- fs/proc/page.c | 4 ++-- include/linux/fs.h | 2 +- include/linux/mm_types.h | 1 - include/linux/page-flags.h | 20 ++++++++++---------- include/linux/pagemap.h | 2 +- mm/gup.c | 4 ++-- mm/internal.h | 2 +- mm/ksm.c | 4 ++-- mm/rmap.c | 16 ++++++++-------- mm/util.c | 6 +++--- 10 files changed, 30 insertions(+), 31 deletions(-)
fs/proc/page.c +2 -2
@@ -149,7 +149,7 @@ u64 stable_page_flags(const struct page *page) k = folio->flags; mapping = (unsigned long )folio->mapping; - is_anon = mapping &PAGE_MAPPING_ANON ; + is_anon = mapping &FOLIO_MAPPING_ANON ;/* * pseudo flags for the well known (anonymous) memory mapped pages
@@ -158,7 +158,7 @@ u64 stable_page_flags(const struct page *page) u |=1 <<KPF_MMAP ;if (is_anon) { u |=1 <<KPF_ANON ; -if (mapping &PAGE_MAPPING_KSM ) +if (mapping &FOLIO_MAPPING_KSM ) u |=1 <<KPF_KSM ; }
include/linux/fs.h +1 -1
@@ -524,7 +524,7 @@ struct address_space {/* * On most architectures that alignment is already the case; but * must be enforced here for CRIS, to let the least significant bit -* of struct page's "mapping" pointer be used for PAGE_MAPPING_ANON. +* of struct folio's "mapping" pointer be used for FOLIO_MAPPING_ANON. */ /* XArray tags, for tagging dirty and writeback pages in the pagecache. */
include/linux/mm_types.h +0 -1
@@ -105,7 +105,6 @@ struct page {unsigned int order; }; }; -/* See page-flags.h for PAGE_MAPPING_FLAGS */ struct address_space *mapping;union {pgoff_t __folio_index;/* Our offset within mapping. */
include/linux/page-flags.h +10 -10
@@ -695,10 +695,10 @@ PAGEFLAG_FALSE(VmemmapSelfHosted, vmemmap_self_hosted)/* * On an anonymous folio mapped into a user virtual memory area, * folio->mapping points to its anon_vma, not to a struct address_space; -* with the PAGE_MAPPING_ANON bit set to distinguish it. See rmap.h. +* with the FOLIO_MAPPING_ANON bit set to distinguish it. See rmap.h. * * On an anonymous folio in a VM_MERGEABLE area, if CONFIG_KSM is enabled, -* the PAGE_MAPPING_ANON_KSM bit may be set along with the PAGE_MAPPING_ANON +* the FOLIO_MAPPING_ANON_KSM bit may be set along with the FOLIO_MAPPING_ANON * bit; and then folio->mapping points, not to an anon_vma, but to a private * structure which KSM associates with that merged folio. See ksm.h. *
@@ -713,21 +713,21 @@ PAGEFLAG_FALSE(VmemmapSelfHosted, vmemmap_self_hosted)* false before calling the following functions (e.g., folio_test_anon). * See mm/slab.h. */ -#define PAGE_MAPPING_ANON 0x1 -#define PAGE_MAPPING_ANON_KSM 0x2 -#define PAGE_MAPPING_KSM (PAGE_MAPPING_ANON | PAGE_MAPPING_ANON_KSM) -#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_ANON_KSM) +#define FOLIO_MAPPING_ANON 0x1 +#define FOLIO_MAPPING_ANON_KSM 0x2 +#define FOLIO_MAPPING_KSM (FOLIO_MAPPING_ANON | FOLIO_MAPPING_ANON_KSM) +#define FOLIO_MAPPING_FLAGS (FOLIO_MAPPING_ANON | FOLIO_MAPPING_ANON_KSM) static __always_inline bool folio_test_anon(const struct folio *folio) { -return ((unsigned long )folio->mapping &PAGE_MAPPING_ANON ) !=0 ; +return ((unsigned long )folio->mapping &FOLIO_MAPPING_ANON ) !=0 ; } static __always_inline bool PageAnonNotKsm(const struct page *page) {unsigned long flags = (unsigned long )page_folio (page)->mapping; -return (flags &PAGE_MAPPING_FLAGS ) ==PAGE_MAPPING_ANON ; +return (flags &FOLIO_MAPPING_FLAGS ) ==FOLIO_MAPPING_ANON ; } static __always_inline bool PageAnon(const struct page *page)
@@ -743,8 +743,8 @@ static __always_inline bool PageAnon(const struct page *page)*/ static __always_inline bool folio_test_ksm(const struct folio *folio) { -return ((unsigned long )folio->mapping &PAGE_MAPPING_FLAGS ) == -PAGE_MAPPING_KSM ; +return ((unsigned long )folio->mapping &FOLIO_MAPPING_FLAGS ) == +FOLIO_MAPPING_KSM ; }#else FOLIO_TEST_FLAG_FALSE (ksm)
include/linux/pagemap.h +1 -1
@@ -502,7 +502,7 @@ static inline pgoff_t mapping_align_index(struct address_space *mapping,static inline bool mapping_large_folio_support (struct address_space *mapping) {/* AS_FOLIO_ORDER is only reasonable for pagecache folios */ -VM_WARN_ONCE ((unsigned long )mapping &PAGE_MAPPING_ANON , +VM_WARN_ONCE ((unsigned long )mapping &FOLIO_MAPPING_ANON ,"Anonymous mapping always supports large folio" );return mapping_max_folio_order (mapping) >0 ;
mm/gup.c +2 -2
@@ -2804,9 +2804,9 @@ static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags)return false;/* Anonymous folios pose no problem. */ - mapping_flags = (unsigned long )mapping &PAGE_MAPPING_FLAGS ; + mapping_flags = (unsigned long )mapping &FOLIO_MAPPING_FLAGS ;if (mapping_flags) -return mapping_flags &PAGE_MAPPING_ANON ; +return mapping_flags &FOLIO_MAPPING_ANON ;/* * At this point, we know the mapping is non-null and points to an
mm/internal.h +1 -1
@@ -150,7 +150,7 @@ static inline void *folio_raw_mapping(const struct folio *folio) {unsigned long mapping = (unsigned long )folio->mapping; -return (void *)(mapping & ~PAGE_MAPPING_FLAGS ); +return (void *)(mapping & ~FOLIO_MAPPING_FLAGS ); }/*
mm/ksm.c +2 -2
@@ -893,7 +893,7 @@ static struct folio *ksm_get_folio(struct ksm_stable_node *stable_node,unsigned long kpfn; expected_mapping = (void *)((unsigned long )stable_node | -PAGE_MAPPING_KSM ); +FOLIO_MAPPING_KSM ); again: kpfn =READ_ONCE (stable_node->kpfn);/* Address dependency. */ folio =pfn_folio (kpfn);
@@ -1070,7 +1070,7 @@ static inline void folio_set_stable_node(struct folio *folio, struct ksm_stable_node *stable_node) {VM_WARN_ON_FOLIO (folio_test_anon (folio) &&PageAnonExclusive (&folio->page), folio); - folio->mapping = (void *)((unsigned long )stable_node |PAGE_MAPPING_KSM ); + folio->mapping = (void *)((unsigned long )stable_node |FOLIO_MAPPING_KSM ); }#ifdef CONFIG_SYSFS
mm/rmap.c +8 -8
@@ -503,12 +503,12 @@ struct anon_vma *folio_get_anon_vma(const struct folio *folio)rcu_read_lock (); anon_mapping = (unsigned long )READ_ONCE (folio->mapping); -if ((anon_mapping &PAGE_MAPPING_FLAGS ) !=PAGE_MAPPING_ANON ) +if ((anon_mapping &FOLIO_MAPPING_FLAGS ) !=FOLIO_MAPPING_ANON ) goto out;if (!folio_mapped (folio)) goto out; - anon_vma = (struct anon_vma *) (anon_mapping -PAGE_MAPPING_ANON ); + anon_vma = (struct anon_vma *) (anon_mapping -FOLIO_MAPPING_ANON );if (!atomic_inc_not_zero (&anon_vma->refcount)) { anon_vma =NULL ; goto out;
@@ -550,12 +550,12 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio, retry:rcu_read_lock (); anon_mapping = (unsigned long )READ_ONCE (folio->mapping); -if ((anon_mapping &PAGE_MAPPING_FLAGS ) !=PAGE_MAPPING_ANON ) +if ((anon_mapping &FOLIO_MAPPING_FLAGS ) !=FOLIO_MAPPING_ANON ) goto out;if (!folio_mapped (folio)) goto out; - anon_vma = (struct anon_vma *) (anon_mapping -PAGE_MAPPING_ANON ); + anon_vma = (struct anon_vma *) (anon_mapping -FOLIO_MAPPING_ANON ); root_anon_vma =READ_ONCE (anon_vma->root);if (down_read_trylock (&root_anon_vma->rwsem)) {/*
@@ -1334,9 +1334,9 @@ void folio_move_anon_rmap(struct folio *folio, struct vm_area_struct *vma)VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio);VM_BUG_ON_VMA (!anon_vma, vma); - anon_vma +=PAGE_MAPPING_ANON ; + anon_vma +=FOLIO_MAPPING_ANON ;/* -* Ensure that anon_vma and the PAGE_MAPPING_ANON bit are written +* Ensure that anon_vma and the FOLIO_MAPPING_ANON bit are written * simultaneously, so a concurrent reader (eg folio_referenced()'s * folio_test_anon()) will not see one without the other. */
@@ -1367,10 +1367,10 @@ static void __folio_set_anon(struct folio *folio, struct vm_area_struct *vma,/* * page_idle does a lockless/optimistic rmap scan on folio->mapping. * Make sure the compiler doesn't split the stores of anon_vma and -* the PAGE_MAPPING_ANON type identifier, otherwise the rmap code +* the FOLIO_MAPPING_ANON type identifier, otherwise the rmap code * could mistake the mapping for a struct address_space and crash. */ - anon_vma = (void *) anon_vma +PAGE_MAPPING_ANON ; + anon_vma = (void *) anon_vma +FOLIO_MAPPING_ANON ;WRITE_ONCE (folio->mapping, (struct address_space *) anon_vma); folio->index =linear_page_index (vma, address); }
mm/util.c +3 -3
@@ -670,9 +670,9 @@ struct anon_vma *folio_anon_vma(const struct folio *folio) {unsigned long mapping = (unsigned long )folio->mapping; -if ((mapping &PAGE_MAPPING_FLAGS ) !=PAGE_MAPPING_ANON ) +if ((mapping &FOLIO_MAPPING_FLAGS ) !=FOLIO_MAPPING_ANON )return NULL ; -return (void *)(mapping -PAGE_MAPPING_ANON ); +return (void *)(mapping -FOLIO_MAPPING_ANON ); }/**
@@ -699,7 +699,7 @@ struct address_space *folio_mapping(struct folio *folio)return swap_address_space (folio->swap); mapping = folio->mapping; -if ((unsigned long )mapping &PAGE_MAPPING_FLAGS ) +if ((unsigned long )mapping &FOLIO_MAPPING_FLAGS )return NULL ;return mapping;
Signature · 1 line
2.49.0
Re: [PATCH RFC 26/29] mm: rename PAGE_MAPPING_* to FOLIO_MAPPING_*
Quoted 4 lines · David Hildenbrand
Now that the mapping flags are only used for folios, let's rename the defines.
Signed-off-by David Hildenbrand <[email protected]>
Quoted 13 lines
--- fs/proc/page.c | 4 ++-- include/linux/fs.h | 2 +- include/linux/mm_types.h | 1 - include/linux/page-flags.h | 20 ++++++++++---------- include/linux/pagemap.h | 2 +- mm/gup.c | 4 ++-- mm/internal.h | 2 +- mm/ksm.c | 4 ++-- mm/rmap.c | 16 ++++++++-------- mm/util.c | 6 +++--- 10 files changed, 30 insertions(+), 31 deletions(-)
--
Best Regards,
Yan, Zi
RFC PATCH 27/29 docs/mm: convert from "Non-LRU page migration" to "movable_ops page migration"
In reply to David HildenbrandLet's bring the docs up-to-date.
--- Documentation/mm/page_migration.rst | 39 ++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 12 deletions(-)
Documentation/mm/page_migration.rst +27 -12
@@ -146,18 +146,33 @@ Steps: 18. The new page is moved to the LRU and can be scanned by the swapper, etc. again. -Non-LRU page migration -====================== - -Although migration originally aimed for reducing the latency of memory -accesses for NUMA, compaction also uses migration to create high-order -pages. For compaction purposes, it is also useful to be able to move -non-LRU pages, such as zsmalloc and virtio-balloon pages. - -If a driver wants to make its pages movable, it should define a struct -movable_operations. It then needs to call __SetPageMovable() on each -page that it may be able to move. This uses the ``page->mapping`` field, -so this field is not available for the driver to use for other purposes. +movable_ops page migration +========================== + +Selected typed, non-folio pages (e.g., pages inflated in a memory balloon, +zsmalloc pages) can be migrated using the movable_ops migration framework. + +The "struct movable_operations" provide callbacks specific to a page type +for isolating, migrating and un-isolating (putback) these pages. + +Once a page is indicated as having movable_ops, that condition must not +change until the page was freed back to the buddy. This includes not +changing/clearing the page type and not changing/clearing the +PG_movable_ops page flag. + +Arbitrary drivers cannot currently make use of this framework, as it +requires: + +(a) a page type +(b) indicating them as possibly having movable_ops in page_has_movable_ops() + based on the page type +(c) returning the movable_ops from page_has_movable_ops() based on the page + type +(d) not reusing the PG_movable_ops and PG_movable_ops_isolated page flags + for other purposes + +For example, balloon drivers can make use of this framework through the +balloon-compaction framework residing in the core kernel. Monitoring Migration =====================
Signature · 1 line
2.49.0
Re: [PATCH RFC 27/29] docs/mm: convert from "Non-LRU page migration" to "movable_ops page migration"
Quoted 21 lines · David Hildenbrand
Let's bring the docs up-to-date.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/mm/page_migration.rst | 39 ++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 12 deletions(-)Documentation/mm/page_migration.rst +0
-6@@ -146,18 +146,33 @@ Steps: 18. The new page is moved to the LRU and can be scanned by the swapper, etc. again. -Non-LRU page migration -====================== - -Although migration originally aimed for reducing the latency of memory -accesses for NUMA, compaction also uses migration to create high-order -pages. For compaction purposes, it is also useful to be able to move
Documentation/mm/page_migration.rst Steps: · line 152
Quoted 10 lines
Documentation/mm/page_migration.rst +4
-6-non-LRU pages, such as zsmalloc and virtio-balloon pages. - -If a driver wants to make its pages movable, it should define a struct -movable_operations. It then needs to call __SetPageMovable() on each -page that it may be able to move. This uses the ``page->mapping`` field, -so this field is not available for the driver to use for other purposes. +movable_ops page migration +========================== + +Selected typed, non-folio pages (e.g., pages inflated in a memory balloon,
This is a great clarification. Thanks.
Quoted 16 lines
Documentation/mm/page_migration.rst +16
-0+zsmalloc pages) can be migrated using the movable_ops migration framework. + +The "struct movable_operations" provide callbacks specific to a page type +for isolating, migrating and un-isolating (putback) these pages. + +Once a page is indicated as having movable_ops, that condition must not +change until the page was freed back to the buddy. This includes not +changing/clearing the page type and not changing/clearing the +PG_movable_ops page flag. + +Arbitrary drivers cannot currently make use of this framework, as it +requires: + +(a) a page type +(b) indicating them as possibly having movable_ops in page_has_movable_ops() + based on the page type
Documentation/mm/page_migration.rst Steps: · line 178
Quoted 10 lines
Documentation/mm/page_migration.rst +7
-0+(c) returning the movable_ops from page_has_movable_ops() based on the page + type +(d) not reusing the PG_movable_ops and PG_movable_ops_isolated page flags + for other purposes + +For example, balloon drivers can make use of this framework through the +balloon-compaction framework residing in the core kernel. Monitoring Migration =====================
--
Best Regards,
Yan, Zi
RFC PATCH 28/29 mm/balloon_compaction: "movable_ops" doc updates
In reply to David HildenbrandLet's bring the docs up-to-date. Setting PG_movable_ops + page->private very likely still requires to be performed under documented locks:
it's complicated.
We will rework this in the future, as we will try avoiding using the page lock.
--- include/linux/balloon_compaction.h | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
include/linux/balloon_compaction.h +6 -7
@@ -4,12 +4,13 @@* * Common interface definitions for making balloon pages movable by compaction. * -* Balloon page migration makes use of the general non-lru movable page +* Balloon page migration makes use of the general "movable_ops page migration" * feature. * * page->private is used to reference the responsible balloon device. -* page->mapping is used in context of non-lru page migration to reference -* the address space operations for page isolation/migration/compaction. +* That these pages have movable_ops, and which movable_ops apply, +* is derived from the page type (PageOffline()) combined with the +* PG_movable_ops flag (PageMovableOps()). * * As the page isolation scanning step a compaction thread does is a lockless * procedure (from a page standpoint), it might bring some racy situations while
@@ -17,12 +18,10 @@* and safely perform balloon's page compaction and migration we must, always, * ensure following these simple rules: * -* i. when updating a balloon's page ->mapping element, strictly do it under -* the following lock order, independently of the far superior -* locking scheme (lru_lock, balloon_lock): +* i. Setting the PG_movable_ops flag and page->private with the following +* lock order * +-page_lock(page); * +--spin_lock_irq(&b_dev_info->pages_lock); -* ... page->mapping updates here ... * * ii. isolation or dequeueing procedure must remove the page from balloon * device page list under b_dev_info->pages_lock.
Signature · 1 line
2.49.0
RFC PATCH 29/29 mm/balloon_compaction: provide single balloon_page_insert() and balloon_mapping_gfp_mask()
In reply to David HildenbrandLet's just special-case based on IS_ENABLED(CONFIG_BALLOON_COMPACTION like we did for balloon_page_finalize().
--- include/linux/balloon_compaction.h | 42 +++++++++++------------------- 1 file changed, 15 insertions(+), 27 deletions(-)
include/linux/balloon_compaction.h +15 -27
@@ -77,6 +77,15 @@ static inline void balloon_devinfo_init(struct balloon_dev_info *balloon)#ifdef CONFIG_BALLOON_COMPACTION extern const struct movable_operations balloon_mops; +/* +* balloon_page_device - get the b_dev_info descriptor for the balloon device +* that enqueues the given page. +*/ +static inline struct balloon_dev_info *balloon_page_device (struct page *page) +{ +return (struct balloon_dev_info *)page_private (page); +} +#endif /* CONFIG_BALLOON_COMPACTION */ /* * balloon_page_insert - insert a page into the balloon's page list and make
@@ -91,41 +100,20 @@ static inline void balloon_page_insert(struct balloon_dev_info *balloon, struct page *page) {__SetPageOffline (page); -SetPageMovableOps (page); -set_page_private (page, (unsigned long )balloon); -list_add (&page->lru, &balloon->pages); -} - -/* -* balloon_page_device - get the b_dev_info descriptor for the balloon device -* that enqueues the given page. -*/ -static inline struct balloon_dev_info *balloon_page_device (struct page *page) -{ -return (struct balloon_dev_info *)page_private (page); -} - -static inline gfp_t balloon_mapping_gfp_mask (void ) -{ -return GFP_HIGHUSER_MOVABLE ; -} - -#else /* !CONFIG_BALLOON_COMPACTION */ - -static inline void balloon_page_insert (struct balloon_dev_info *balloon, -struct page *page) -{ -__SetPageOffline (page); +if (IS_ENABLED (CONFIG_BALLOON_COMPACTION )) { +SetPageMovableOps (page); +set_page_private (page, (unsigned long )balloon); + }list_add (&page->lru, &balloon->pages); }static inline gfp_t balloon_mapping_gfp_mask (void ) { +if (IS_ENABLED (CONFIG_BALLOON_COMPACTION )) +return GFP_HIGHUSER_MOVABLE ;return GFP_HIGHUSER ; } -#endif /* CONFIG_BALLOON_COMPACTION */ -/* * balloon_page_finalize - prepare a balloon page that was removed from the * balloon list for release to the page allocator
Signature · 1 line
2.49.0