mm/huge_memory.c: split should clone page flags before unfreezing pageref
Subsystems (3)
PATCH 1/1 mm/huge_memory.c: split should clone page flags before unfreezing pageref
THP split makes non-atomic change of tail page flags. This is almost ok because tail pages are locked and isolated but this breaks recent changes in page locking: non-atomic operation could clear bit PG_waiters.
As a result concurrent sequence get_page_unless_zero() -> lock_page() might block forever. Especially if this page was truncated later.
Fix is trivial: clone flags before unfreezing page reference counter.
This race exists since commit 62906027091f ("mm: add PageWaiters indicating tasks are waiting for a page bit") while unsave unfreeze itself was added in commit 8df651c7059e ("thp: cleanup split_huge_page()").
--- mm/huge_memory.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-)
mm/huge_memory.c +13 -12
@@ -2357,6 +2357,19 @@ static void __split_huge_page_tail(struct page *head, int tail,VM_BUG_ON_PAGE (atomic_read (&page_tail->_mapcount) !=-1 , page_tail);VM_BUG_ON_PAGE (page_ref_count (page_tail) !=0 , page_tail); +/* Clone page flags before unfreezing refcount. */ + page_tail->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ; + page_tail->flags |= (head->flags & + ((1L << PG_referenced) | + (1L << PG_swapbacked) | + (1L << PG_swapcache) | + (1L << PG_mlocked) | + (1L << PG_uptodate) | + (1L << PG_active) | + (1L << PG_locked) | + (1L << PG_unevictable) | + (1L << PG_dirty))); +/* * tail_page->_refcount is zero and not changing from under us. But * get_page_unless_zero() may be running from under us on the
@@ -2375,18 +2388,6 @@ static void __split_huge_page_tail(struct page *head, int tail,page_ref_add (page_tail,2 ); } - page_tail->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ; - page_tail->flags |= (head->flags & - ((1L << PG_referenced) | - (1L << PG_swapbacked) | - (1L << PG_swapcache) | - (1L << PG_mlocked) | - (1L << PG_uptodate) | - (1L << PG_active) | - (1L << PG_locked) | - (1L << PG_unevictable) | - (1L << PG_dirty))); -/* * After clearing PageTail the gup refcount can be released. * Page flags also must be visible before we make the page non-compound.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · Konstantin Khlebnikov
THP split makes non-atomic change of tail page flags. This is almost ok because tail pages are locked and isolated but this breaks recent changes in page locking: non-atomic operation could clear bit
PG_waiters.
Quoted 8 lines
As a result concurrent sequence
get_page_unless_zero()->lock_page()might block forever. Especially if this page was truncated later.Fix is trivial: clone flags before unfreezing page reference counter.
This race exists since commit
62906027091f("mm: add PageWaiters indicating tasks are waiting for a page bit") while unsave unfreeze itself was added in commit8df651c7059e("thp: cleanupsplit_huge_page()").
Hm. Don't we have to have barrier between setting flags and updating the refcounter in this case? Atomics don't generally have this semantics, so you can see new refcount before new flags even after the change.
Signature · 7 lines
Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Kirill A. Shutemov
Quoted 9 lines · Konstantin Khlebnikov
THP split makes non-atomic change of tail page flags. This is almost ok because tail pages are locked and isolated but this breaks recent changes in page locking: non-atomic operation could clear bit
PG_waiters.As a result concurrent sequence
get_page_unless_zero()->lock_page()might block forever. Especially if this page was truncated later.Fix is trivial: clone flags before unfreezing page reference counter.
Quoted 8 lines
Quoted 3 lines
This race exists since commit
62906027091f("mm: add PageWaiters indicating tasks are waiting for a page bit") while unsave unfreeze itself was added in commit8df651c7059e("thp: cleanupsplit_huge_page()").Hm. Don't we have to have barrier between setting flags and updating the refcounter in this case? Atomics don't generally have this semantics, so you can see new refcount before new flags even after the change.
Ok.
I see another problem here - clear_compound_head() is placed after unfreeze.
This opens race window with get/put_page after speculative get page.
I think successful get_page_unless_zero() must stabilize compound_head() for tails as well as for heads.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 1/1 mm/huge_memory.c: reorder operations in __split_huge_page_tail()
THP split makes non-atomic change of tail page flags. This is almost ok because tail pages are locked and isolated but this breaks recent changes in page locking: non-atomic operation could clear bit PG_waiters.
As a result concurrent sequence get_page_unless_zero() -> lock_page() might block forever. Especially if this page was truncated later.
Fix is trivial: clone flags before unfreezing page reference counter.
This race exists since commit 62906027091f ("mm: add PageWaiters indicating tasks are waiting for a page bit") while unsave unfreeze itself was added in commit 8df651c7059e ("thp: cleanup split_huge_page()").
clear_compound_head() also must be called before unfreezing page reference because successful get_page_unless_zero() must stabilize compound_head().
And replace page_ref_inc()/page_ref_add() with page_ref_unfreeze() which is made especially for that and has semantic of smp_store_release().
--- mm/huge_memory.c | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-)
mm/huge_memory.c +13 -21
@@ -2355,26 +2355,11 @@ static void __split_huge_page_tail(struct page *head, int tail,struct page *page_tail = head + tail;VM_BUG_ON_PAGE (atomic_read (&page_tail->_mapcount) !=-1 , page_tail); -VM_BUG_ON_PAGE (page_ref_count (page_tail) !=0 , page_tail);/* -* tail_page->_refcount is zero and not changing from under us. But -* get_page_unless_zero() may be running from under us on the -* tail_page. If we used atomic_set() below instead of atomic_inc() or -* atomic_add(), we would then run atomic_set() concurrently with -* get_page_unless_zero(), and atomic_set() is implemented in C not -* using locked ops. spin_unlock on x86 sometime uses locked ops -* because of PPro errata 66, 92, so unless somebody can guarantee -* atomic_set() here would be safe on all archs (and not only on x86), -* it's safer to use atomic_inc()/atomic_add(). +* Clone page flags before unfreezing refcount. +* lock_page() after speculative get wants to set PG_waiters. */ -if (PageAnon (head) && !PageSwapCache (head)) { -page_ref_inc (page_tail); - }else { -/* Additional pin to radix tree */ -page_ref_add (page_tail,2 ); - } - page_tail->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ; page_tail->flags |= (head->flags & ((1L << PG_referenced) |
@@ -2387,14 +2372,21 @@ static void __split_huge_page_tail(struct page *head, int tail, (1L << PG_unevictable) | (1L << PG_dirty))); -/* -* After clearing PageTail the gup refcount can be released. -* Page flags also must be visible before we make the page non-compound. -*/ +/* Page flags must be visible before we make the page non-compound. */ smp_wmb (); +/* +* Clear PageTail before unfreezing page refcount: +* speculative refcount must stabilize compound_head(). +*/ clear_compound_head (page_tail); +/* +* Finally unfreeze refcount. Additional pin to radix tree. +*/ +page_ref_unfreeze (page_tail,1 + (!PageAnon (head) || +PageSwapCache (head))); +if (page_is_young (head))set_page_young (page_tail);if (page_is_idle (head))
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v2] mm/huge_memory.c: reorder operations in __split_huge_page_tail()
Quoted 2 lines · Konstantin Khlebnikov
And replace page_ref_inc()/page_ref_add() with
page_ref_unfreeze()which is made especially for that and has semantic ofsmp_store_release().
Nak on this part.
page_ref_unfreeze() uses atomic_set() which neglects the situation in the comment you're removing.
You need at least explain why it's safe now.
I would rather leave page_ref_inc()/page_ref_add() + explcit smp_mb__before_atomic().
Signature · 7 lines
Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v2] mm/huge_memory.c: reorder operations in __split_huge_page_tail()
Quoted 8 lines · Kirill A. Shutemov
Quoted 2 lines · Konstantin Khlebnikov
And replace page_ref_inc()/page_ref_add() with
page_ref_unfreeze()which is made especially for that and has semantic ofsmp_store_release().Nak on this part.
page_ref_unfreeze()usesatomic_set()which neglects the situation in the comment you're removing.
Why? look into x86 smp_store_release
for PPro it use same sequence smp_wb + WRITE_ONCE As I see spin_unlock uses exactly this macro.
Anyway if page_ref_unfreeze cannot handle races with get_page_unless_zero() then it completely useless,
Quoted 6 lines
You need at least explain why it's safe now.
I would rather leave page_ref_inc()/page_ref_add() + explcit
smp_mb__before_atomic().
Quoted 8 lines
--
Kirill A. Shutemov--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v2] mm/huge_memory.c: reorder operations in __split_huge_page_tail()
Quoted 11 lines · Konstantin Khlebnikov
Quoted 8 lines · Kirill A. Shutemov
Quoted 2 lines · Konstantin Khlebnikov
And replace page_ref_inc()/page_ref_add() with
page_ref_unfreeze()which is made especially for that and has semantic ofsmp_store_release().Nak on this part.
page_ref_unfreeze()usesatomic_set()which neglects the situation in the comment you're removing.
Quoted 6 lines
Why? look into x86
smp_store_release
for PPro it use same sequencesmp_wb+WRITE_ONCEAs I seespin_unlockuses exactly this macro.Anyway if
page_ref_unfreezecannot handle races withget_page_unless_zero()then it completely useless,
Okay, fair enough.
But please call it out it the commit message.
Signature · 7 lines
Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v2] mm/huge_memory.c: reorder operations in __split_huge_page_tail()
Quoted 16 lines · Kirill A. Shutemov
Quoted 15 lines · Konstantin Khlebnikov
Quoted 8 lines · Kirill A. Shutemov
Quoted 2 lines · Konstantin Khlebnikov
And replace page_ref_inc()/page_ref_add() with
page_ref_unfreeze()which is made especially for that and has semantic ofsmp_store_release().Nak on this part.
page_ref_unfreeze()usesatomic_set()which neglects the situation in the comment you're removing.Why? look into x86
smp_store_release
for PPro it use same sequencesmp_wb+WRITE_ONCEAs I seespin_unlockuses exactly this macro.
Quoted 6 lines
Quoted 2 lines
Anyway if
page_ref_unfreezecannot handle races withget_page_unless_zero()then it completely useless,Okay, fair enough.
But please call it out it the commit message.
OK, I'll review this yet again and resend tomorrow.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v2] mm/huge_memory.c: reorder operations in __split_huge_page_tail()
In reply to Konstantin Khlebnikovmm/huge_memory.c static void __split_huge_page_tail(struct page *head, int tail, · line 2388
Quoted 5 lines · Konstantin Khlebnikov
mm/huge_memory.c +5
-0+/* +* Finally unfreeze refcount. Additional pin to radix tree. +*/ +page_ref_unfreeze (page_tail,1 + (!PageAnon (head) || +PageSwapCache (head)));
Please say "Additional pin from page cache", not "radix tree".
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v3 1/2 mm/page_ref: use atomic_set_release in page_ref_unfreeze
In reply to Konstantin Khlebnikovpage_ref_unfreeze() has exactly that semantic. No functional
changes: just minus one barrier and proper handling of PPro errata.
--- include/linux/page_ref.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
include/linux/page_ref.h +1 -2
@@ -175,8 +175,7 @@ static inline void page_ref_unfreeze(struct page *page, int count)VM_BUG_ON_PAGE (page_count (page) !=0 , page);VM_BUG_ON (count ==0 ); -smp_mb (); -atomic_set (&page->_refcount, count); +atomic_set_release (&page->_refcount, count);if (page_ref_tracepoint_active (__tracepoint_page_ref_unfreeze))__page_ref_unfreeze (page, count); }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v3 1/2] mm/page_ref: use atomic_set_release in page_ref_unfreeze
Quoted 4 lines · Konstantin Khlebnikov
page_ref_unfreeze()has exactly that semantic. No functional
changes: just minus one barrier and proper handling of PPro errata.Signed-off-by Konstantin Khlebnikov <[email protected]>
Signature · 7 lines
Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v3 2/2 mm/huge_memory.c: reorder operations in __split_huge_page_tail()
In reply to Konstantin KhlebnikovTHP split makes non-atomic change of tail page flags. This is almost ok because tail pages are locked and isolated but this breaks recent changes in page locking: non-atomic operation could clear bit PG_waiters.
As a result concurrent sequence get_page_unless_zero() -> lock_page() might block forever. Especially if this page was truncated later.
Fix is trivial: clone flags before unfreezing page reference counter.
This race exists since commit 62906027091f ("mm: add PageWaiters indicating tasks are waiting for a page bit") while unsave unfreeze itself was added in commit 8df651c7059e ("thp: cleanup split_huge_page()").
clear_compound_head() also must be called before unfreezing page reference because after successful get_page_unless_zero() might follow put_page() which needs correct compound_head().
And replace page_ref_inc()/page_ref_add() with page_ref_unfreeze() which is made especially for that and has semantic of smp_store_release().
--- mm/huge_memory.c | 36 +++++++++++++++--------------------- 1 file changed, 15 insertions(+), 21 deletions(-)
mm/huge_memory.c +15 -21
@@ -2355,26 +2355,13 @@ static void __split_huge_page_tail(struct page *head, int tail,struct page *page_tail = head + tail;VM_BUG_ON_PAGE (atomic_read (&page_tail->_mapcount) !=-1 , page_tail); -VM_BUG_ON_PAGE (page_ref_count (page_tail) !=0 , page_tail);/* -* tail_page->_refcount is zero and not changing from under us. But -* get_page_unless_zero() may be running from under us on the -* tail_page. If we used atomic_set() below instead of atomic_inc() or -* atomic_add(), we would then run atomic_set() concurrently with -* get_page_unless_zero(), and atomic_set() is implemented in C not -* using locked ops. spin_unlock on x86 sometime uses locked ops -* because of PPro errata 66, 92, so unless somebody can guarantee -* atomic_set() here would be safe on all archs (and not only on x86), -* it's safer to use atomic_inc()/atomic_add(). +* Clone page flags before unfreezing refcount. +* +* After successful get_page_unless_zero() might follow flags change, +* for exmaple lock_page() which set PG_waiters. */ -if (PageAnon (head) && !PageSwapCache (head)) { -page_ref_inc (page_tail); - }else { -/* Additional pin to radix tree */ -page_ref_add (page_tail,2 ); - } - page_tail->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ; page_tail->flags |= (head->flags & ((1L << PG_referenced) |
@@ -2387,14 +2374,21 @@ static void __split_huge_page_tail(struct page *head, int tail, (1L << PG_unevictable) | (1L << PG_dirty))); -/* -* After clearing PageTail the gup refcount can be released. -* Page flags also must be visible before we make the page non-compound. -*/ +/* Page flags must be visible before we make the page non-compound. */ smp_wmb (); +/* +* Clear PageTail before unfreezing page refcount. +* +* After successful get_page_unless_zero() might follow put_page() +* which needs correct compound_head(). +*/ clear_compound_head (page_tail); +/* Finally unfreeze refcount. Additional reference from page cache. */ +page_ref_unfreeze (page_tail,1 + (!PageAnon (head) || +PageSwapCache (head))); +if (page_is_young (head))set_page_young (page_tail);if (page_is_idle (head))
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v3 2/2] mm/huge_memory.c: reorder operations in __split_huge_page_tail()
Quoted 13 lines · Konstantin Khlebnikov
THP split makes non-atomic change of tail page flags. This is almost ok because tail pages are locked and isolated but this breaks recent changes in page locking: non-atomic operation could clear bit
PG_waiters.As a result concurrent sequence
get_page_unless_zero()->lock_page()might block forever. Especially if this page was truncated later.Fix is trivial: clone flags before unfreezing page reference counter.
This race exists since commit
62906027091f("mm: add PageWaiters indicating tasks are waiting for a page bit") while unsave unfreeze itself was added in commit8df651c7059e("thp: cleanupsplit_huge_page()").
Quoted 8 lines
clear_compound_head()also must be called before unfreezing page reference because after successfulget_page_unless_zero()might followput_page()which needs correctcompound_head().And replace page_ref_inc()/page_ref_add() with
page_ref_unfreeze()which is made especially for that and has semantic ofsmp_store_release().Signed-off-by Konstantin Khlebnikov <[email protected]>
Signature · 7 lines
Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>