mm-hugetlb_vmemmap-fix-memory-loads-ordering.patch removed from -mm tree
The quilt patch titled
Subject: mm/hugetlb_vmemmap: fix memory loads ordering
has been removed from the -mm tree. Its filename was
mm-hugetlb_vmemmap-fix-memory-loads-ordering.patch
This patch was dropped because it was merged into the mm-hotfixes-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
------------------------------------------------------
From: Yu Zhao <[email protected]>
Subject: mm/hugetlb_vmemmap: fix memory loads ordering
Date: Wed, 8 Jan 2025 00:48:21 -0700
Using x86_64 as an example, for a 32KB struct page[] area describing a 2MB hugeTLB, HVO reduces the area to 4KB by the following steps:
1. Split the (r/w vmemmap) PMD mapping the area into 512 (r/w) PTEs;
2. For the 8 PTEs mapping the area, remap PTE 1-7 to the page mapped by PTE 0, and at the same time change the permission from r/w to r/o;
3. Free the pages PTE 1-7 used to map, hence the reduction from 32KB to 4KB.
However, the following race can happen due to improperly memory loads ordering:
CPU 1 (HVO) CPU 2 (speculative PFN walker)
page_ref_freeze()
synchronize_rcu()
rcu_read_lock() page_is_fake_head() is false
vmemmap_remap_pte()
XXX: struct page[] becomes r/o
page_ref_unfreeze()
page_ref_count() is not zero
atomic_add_unless(&page->_refcount)
XXX: try to modify r/o struct page[]Specifically, page_is_fake_head() must be ordered after page_ref_count() on CPU 2 so that it can only return true for this case, to avoid the later attempt to modify r/o struct page[].
This patch adds the missing memory barrier and makes the tests on page_is_fake_head() and page_ref_count() done in the proper order.
---
include/linux/page-flags.h | 37 +++++++++++++++++++++++++++++++++++ include/linux/page_ref.h | 2 - 2 files changed, 38 insertions(+), 1 deletion(-)
include/linux/page-flags.h renamed from include/linux/page-flags.h~mm-hugetlb_vmemmap-fix-memory-loads-ordering +37 -0
@@ -226,11 +226,48 @@ static __always_inline const struct page }return page; } + +static __always_inline bool page_count_writable(const struct page *page, int u) +{ +if (!static_branch_unlikely (&hugetlb_optimize_vmemmap_key)) +return true; + +/* +* The refcount check is ordered before the fake-head check to prevent +* the following race: +* CPU 1 (HVO) CPU 2 (speculative PFN walker) +* +* page_ref_freeze() +* synchronize_rcu() +* rcu_read_lock() +* page_is_fake_head() is false +* vmemmap_remap_pte() +* XXX: struct page[] becomes r/o +* +* page_ref_unfreeze() +* page_ref_count() is not zero +* +* atomic_add_unless(&page->_refcount) +* XXX: try to modify r/o struct page[] +* +* The refcount check also prevents modification attempts to other (r/o) +* tail pages that are not fake heads. +*/ +if (atomic_read_acquire (&page->_refcount) == u) +return false; + +return page_fixed_fake_head (page) == page; +}#else static inline const struct page *page_fixed_fake_head (const struct page *page) {return page; } + +static inline bool page_count_writable (const struct page *page,int u) +{ +return true; +}#endif static __always_inline int page_is_fake_head(const struct page *page)
include/linux/page_ref.h renamed from include/linux/page_ref.h~mm-hugetlb_vmemmap-fix-memory-loads-ordering +1 -1
@@ -234,7 +234,7 @@ static inline bool page_ref_add_unless(srcu_read_lock ();/* avoid writing to the vmemmap area being remapped */ -if (!page_is_fake_head (page) &&page_ref_count (page) != u) +if (page_count_writable (page, u)) ret =atomic_add_unless (&page->_refcount, nr, u);rcu_read_unlock ();
_
Patches currently in -mm which might be from [email protected] are