Skip to contentSkip to conversation

mm: thp: handle page cache THP correctly in PageTransCompoundMap

PATCH Superseded Yang Shi 3 participants 8 messages Oct 23, 2019

Subsystems (3)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Yang Shi

author
#0 lore raw

PATCH v2 1/1 mm: thp: handle page cache THP correctly in PageTransCompoundMap

We have usecase to use tmpfs as QEMU memory backend and we would like to take the advantage of THP as well. But, our test shows the EPT is not PMD mapped even though the underlying THP are PMD mapped on host. The number showed by /sys/kernel/debug/kvm/largepage is much less than the number of PMD mapped shmem pages as the below:

7f2778200000-7f2878200000 rw-s 00000000 00:14 262232 /dev/shm/qemu_back_mem.mem.Hz2hSf (deleted)
Size: 4194304 kB
[snip]

AnonHugePages:         0 kB
ShmemPmdMapped:   579584 kB

[snip]
Locked: 0 kB

cat /sys/kernel/debug/kvm/largepages
12

And some benchmarks do worse than with anonymous THPs.

By digging into the code we figured out that commit 127393fbe597 ("mm:
thp: kvm: fix memory corruption in KVM with THP enabled") checks if there is a single PTE mapping on the page for anonymous THP when setting up EPT map. But, the _mapcount < 0 check doesn't fit to page cache THP since every subpage of page cache THP would get _mapcount inc'ed once it is PMD mapped, so PageTransCompoundMap() always returns false for page cache THP. This would prevent KVM from setting up PMD mapped EPT entry.

So we need handle page cache THP correctly. However, when page cache THP's PMD gets split, kernel just remove the map instead of setting up PTE map like what anonymous THP does. Before KVM calls get_user_pages() the subpages may get PTE mapped even though it is still a THP since the page cache THP may be mapped by other processes at the mean time.

Checking its _mapcount and whether the THP has PTE mapped or not. Although this may report some false negative cases (PTE mapped by other processes), it looks not trivial to make this accurate.

With this fix /sys/kernel/debug/kvm/largepage would show reasonable pages are PMD mapped by EPT as the below:

7fbeaee00000-7fbfaee00000 rw-s 00000000 00:14 275464 /dev/shm/qemu_back_mem.mem.SKUvat (deleted)
Size: 4194304 kB
[snip]

AnonHugePages:         0 kB
ShmemPmdMapped:   557056 kB

[snip]
Locked: 0 kB

cat /sys/kernel/debug/kvm/largepages
271

And the benchmarks are as same as anonymous THPs.

---
v2: Adopted the suggestion from Hugh to use _mapcount and compound_mapcount.
But I just open coding compound_mapcount to avoid duplicating the definition of compound_mapcount_ptr in two different files. Since "compound_mapcount" looks self-explained so I'm supposed the open coding would not affect the readability.

 include/linux/page-flags.h | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
include/linux/page-flags.h +20 -2
index f91cb88..954a877 100644
@@ -622,12 +622,30 @@ static inline int PageTransCompound(struct page *page)
  *
  * Unlike PageTransCompound, this is safe to be called only while
  * split_huge_pmd() cannot run from under us, like if protected by the
- * MMU notifier, otherwise it may result in page->_mapcount < 0 false
+ * MMU notifier, otherwise it may result in page->_mapcount check false
  * positives.
+ *
+ * We have to treat page cache THP differently since every subpage of it
+ * would get _mapcount inc'ed once it is PMD mapped.  But, it may be PTE
+ * mapped in the current process so comparing subpage's _mapcount to
+ * compound_mapcount ot filter out PTE mapped case.
  */
 static inline int PageTransCompoundMap(struct page *page)
 {
-	return PageTransCompound(page) && atomic_read(&page->_mapcount) < 0;
+	struct page *head;
+	int map_count;
+
+	if (!PageTransCompound(page))
+		return 0;
+
+	if (PageAnon(page))
+		return atomic_read(&page->_mapcount) < 0;
+
+	head = compound_head(page);
+	map_count = atomic_read(&page->_mapcount);
+	/* File THP is PMD mapped and not double mapped */
+	return map_count >= 0 &&
+	       map_count == atomic_read(&head[1].compound_mapcount);
 }
 
 /*
Signature · 1 line
1.8.3.1
#1 lore raw

Review

Quoted 3 lines · Yang Shi
(quoted diff) +2 -0
+	return map_count >= 0 &&
+	       map_count == atomic_read(&head[1].compound_mapcount);
 }

I didn't like Hugh's duplicate definition either. May I suggest:

include/linux/mm.h +0 -5
index 2f2199a51941..3d0efd937d2b 100644
@@ -695,11 +695,6 @@ static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)
 
 extern void kvfree(const void *addr);
 
-static inline atomic_t *compound_mapcount_ptr(struct page *page)
-{
-	return &page[1].compound_mapcount;
-}
-
 static inline int compound_mapcount(struct page *page)
 {
 	VM_BUG_ON_PAGE(!PageCompound(page), page);
include/linux/mm_types.h +5 -0
index 2222fa795284..270aa8fd2800 100644
@@ -221,6 +221,11 @@ struct page {
 #endif
 } _struct_page_alignment;
 
+static inline atomic_t *compound_mapcount_ptr(struct page *page)
+{
+	return &page[1].compound_mapcount;
+}
+
 /*
  * Used for sizing the vmemmap region on some architectures
  */
author
#2 lore raw
Quoted 5 lines · Matthew Wilcox

Review

Quoted 3 lines · Yang Shi
(quoted diff) +2 -0
+	return map_count >= 0 &&
+	       map_count == atomic_read(&head[1].compound_mapcount);
  }

I didn't like Hugh's duplicate definition either. May I suggest:

Thanks, Willy. It is fine to me. Will take it in v3.

Quoted 33 lines
include/linux/mm.h +0 -5
index 2f2199a51941..3d0efd937d2b 100644
@@ -695,11 +695,6 @@ static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)
 
  extern void kvfree(const void *addr);
 
-static inline atomic_t *compound_mapcount_ptr(struct page *page)
-{
-	return &page[1].compound_mapcount;
-}
-
  static inline int compound_mapcount(struct page *page)
  {
  	VM_BUG_ON_PAGE(!PageCompound(page), page);
include/linux/mm_types.h +5 -0
index 2222fa795284..270aa8fd2800 100644
@@ -221,6 +221,11 @@ struct page {
  #endif
  } _struct_page_alignment;
 
+static inline atomic_t *compound_mapcount_ptr(struct page *page)
+{
+	return &page[1].compound_mapcount;
+}
+
  /*
   * Used for sizing the vmemmap region on some architectures
   */
#3 lore raw
Quoted 8 lines · Yang Shi
Quoted 5 lines · Matthew Wilcox

Review

Quoted 3 lines · Yang Shi
(quoted diff) +2 -0
+	return map_count >= 0 &&
+	       map_count == atomic_read(&head[1].compound_mapcount);
  }

I didn't like Hugh's duplicate definition either. May I suggest:

Thanks, Willy. It is fine to me. Will take it in v3.

Agreed, that will be better.

Hugh

#4 lore raw
In reply to Yang Shi
Quoted 46 lines · Yang Shi

We have usecase to use tmpfs as QEMU memory backend and we would like to take the advantage of THP as well. But, our test shows the EPT is not PMD mapped even though the underlying THP are PMD mapped on host. The number showed by /sys/kernel/debug/kvm/largepage is much less than the number of PMD mapped shmem pages as the below:

7f2778200000-7f2878200000 rw-s 00000000 00:14 262232 /dev/shm/qemu_back_mem.mem.Hz2hSf (deleted)
Size: 4194304 kB
[snip]

AnonHugePages:         0 kB
ShmemPmdMapped:   579584 kB

[snip]
Locked: 0 kB

cat /sys/kernel/debug/kvm/largepages
12

And some benchmarks do worse than with anonymous THPs.

By digging into the code we figured out that commit 127393fbe597 ("mm:
thp: kvm: fix memory corruption in KVM with THP enabled") checks if there is a single PTE mapping on the page for anonymous THP when setting up EPT map. But, the _mapcount < 0 check doesn't fit to page cache THP since every subpage of page cache THP would get _mapcount inc'ed once it is PMD mapped, so PageTransCompoundMap() always returns false for page cache THP. This would prevent KVM from setting up PMD mapped EPT entry.

So we need handle page cache THP correctly. However, when page cache THP's PMD gets split, kernel just remove the map instead of setting up PTE map like what anonymous THP does. Before KVM calls get_user_pages() the subpages may get PTE mapped even though it is still a THP since the page cache THP may be mapped by other processes at the mean time.

Checking its _mapcount and whether the THP has PTE mapped or not. Although this may report some false negative cases (PTE mapped by other processes), it looks not trivial to make this accurate.

With this fix /sys/kernel/debug/kvm/largepage would show reasonable pages are PMD mapped by EPT as the below:

7fbeaee00000-7fbfaee00000 rw-s 00000000 00:14 275464 /dev/shm/qemu_back_mem.mem.SKUvat (deleted)
Size: 4194304 kB
[snip]

AnonHugePages:         0 kB
ShmemPmdMapped:   557056 kB
Quoted 8 lines

[snip]
Locked: 0 kB

cat /sys/kernel/debug/kvm/largepages
271

And the benchmarks are as same as anonymous THPs.

Fixes dd78fedde4b9 ("rmap: support file thp")
Quoted 7 lines
Quoted 6 lines

---
v2: Adopted the suggestion from Hugh to use _mapcount and compound_mapcount.
But I just open coding compound_mapcount to avoid duplicating the definition of compound_mapcount_ptr in two different files. Since "compound_mapcount" looks self-explained so I'm supposed the open coding would not affect the readability.

No, relying on head[1] is not nice: Matthew's suggestion better.

Quoted 10 lines
 include/linux/page-flags.h | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
include/linux/page-flags.h +0 -0
index f91cb88..954a877 100644
@@ -622,12 +622,30 @@ static inline int PageTransCompound(struct page *page)
  *

include/linux/page-flags.h static inline int PageTransCompound(struct page *page) · line 631

Quoted 10 lines
include/linux/page-flags.h +6 -1
  * Unlike PageTransCompound, this is safe to be called only while
  * split_huge_pmd() cannot run from under us, like if protected by the
- * MMU notifier, otherwise it may result in page->_mapcount < 0 false
+ * MMU notifier, otherwise it may result in page->_mapcount check false
  * positives.
+ *
+ * We have to treat page cache THP differently since every subpage of it
+ * would get _mapcount inc'ed once it is PMD mapped.  But, it may be PTE
+ * mapped in the current process so comparing subpage's _mapcount to
+ * compound_mapcount ot filter out PTE mapped case.

s/ ot / to /

Quoted 6 lines
include/linux/page-flags.h +2 -1
  */
 static inline int PageTransCompoundMap(struct page *page)
 {
-	return PageTransCompound(page) && atomic_read(&page->_mapcount) < 0;
+	struct page *head;
+	int map_count;

include/linux/page-flags.h static inline int PageTransCompound(struct page *page) · line 646

Quoted 10 lines
include/linux/page-flags.h +10 -0
+
+	if (!PageTransCompound(page))
+		return 0;
+
+	if (PageAnon(page))
+		return atomic_read(&page->_mapcount) < 0;
+
+	head = compound_head(page);
+	map_count = atomic_read(&page->_mapcount);
+	/* File THP is PMD mapped and not double mapped */

s/ double / PTE /

include/linux/page-flags.h static inline int PageTransCompound(struct page *page) · line 647

Quoted 1 line
include/linux/page-flags.h +1 -0
+	return map_count >= 0 &&

You have added a map_count >= 0 test there. Okay, not wrong, but not necessary, and not consistent with what's returned in the PageAnon case (if this were called for an unmapped page).

But asking for consistency in this function is asking for too much. It is *very* specialized to the particular places from which it is called (doesn't really belong in page-flags.h at all), and just so long as it gives them the right answer most of the time, and errs on the safe side the rest of the time, it'll do.

(I don't know if it's ever called on a tail page: it would not crash if it were, but different tail pages may give different answers.)

Quoted 8 lines
include/linux/page-flags.h +1 -0
+	       map_count == atomic_read(&head[1].compound_mapcount);
 }
 
 /*
Signature · 1 line
1.8.3.1
author
#5 lore raw
Quoted 51 lines · Hugh Dickins
Quoted 49 lines · Yang Shi

We have usecase to use tmpfs as QEMU memory backend and we would like to take the advantage of THP as well. But, our test shows the EPT is not PMD mapped even though the underlying THP are PMD mapped on host. The number showed by /sys/kernel/debug/kvm/largepage is much less than the number of PMD mapped shmem pages as the below:

7f2778200000-7f2878200000 rw-s 00000000 00:14 262232 /dev/shm/qemu_back_mem.mem.Hz2hSf (deleted)
Size: 4194304 kB
[snip]

AnonHugePages:         0 kB
ShmemPmdMapped:   579584 kB

[snip]
Locked: 0 kB

cat /sys/kernel/debug/kvm/largepages
12

And some benchmarks do worse than with anonymous THPs.

By digging into the code we figured out that commit 127393fbe597 ("mm:
thp: kvm: fix memory corruption in KVM with THP enabled") checks if there is a single PTE mapping on the page for anonymous THP when setting up EPT map. But, the _mapcount < 0 check doesn't fit to page cache THP since every subpage of page cache THP would get _mapcount inc'ed once it is PMD mapped, so PageTransCompoundMap() always returns false for page cache THP. This would prevent KVM from setting up PMD mapped EPT entry.

So we need handle page cache THP correctly. However, when page cache THP's PMD gets split, kernel just remove the map instead of setting up PTE map like what anonymous THP does. Before KVM calls get_user_pages() the subpages may get PTE mapped even though it is still a THP since the page cache THP may be mapped by other processes at the mean time.

Checking its _mapcount and whether the THP has PTE mapped or not. Although this may report some false negative cases (PTE mapped by other processes), it looks not trivial to make this accurate.

With this fix /sys/kernel/debug/kvm/largepage would show reasonable pages are PMD mapped by EPT as the below:

7fbeaee00000-7fbfaee00000 rw-s 00000000 00:14 275464 /dev/shm/qemu_back_mem.mem.SKUvat (deleted)
Size: 4194304 kB
[snip]

AnonHugePages:         0 kB
ShmemPmdMapped:   557056 kB

[snip]
Locked: 0 kB

Quoted 6 lines
Quoted 5 lines

cat /sys/kernel/debug/kvm/largepages
271

And the benchmarks are as same as anonymous THPs.

Fixes dd78fedde4b9 ("rmap: support file thp")

OK, though it might be the best blame target.

Quoted 14 lines
Quoted 8 lines
Quoted 6 lines

---
v2: Adopted the suggestion from Hugh to use _mapcount and compound_mapcount.
But I just open coding compound_mapcount to avoid duplicating the definition of compound_mapcount_ptr in two different files. Since "compound_mapcount" looks self-explained so I'm supposed the open coding would not affect the readability.

Quoted 1 line

No, relying on head[1] is not nice: Matthew's suggestion better.

Done in v3.

Quoted 13 lines
Quoted 13 lines
  include/linux/page-flags.h | 22 ++++++++++++++++++++--
  1 file changed, 20 insertions(+), 2 deletions(-)
include/linux/page-flags.h +0 -1
index f91cb88..954a877 100644
@@ -622,12 +622,30 @@ static inline int PageTransCompound(struct page *page)
   *
   * Unlike PageTransCompound, this is safe to be called only while
   * split_huge_pmd() cannot run from under us, like if protected by the
- * MMU notifier, otherwise it may result in page->_mapcount < 0 false
Quoted 8 lines

include/linux/page-flags.h static inline int PageTransCompound(struct page *page) · line 631

Quoted 7 lines
include/linux/page-flags.h +6 -0
+ * MMU notifier, otherwise it may result in page->_mapcount check false
   * positives.
+ *
+ * We have to treat page cache THP differently since every subpage of it
+ * would get _mapcount inc'ed once it is PMD mapped.  But, it may be PTE
+ * mapped in the current process so comparing subpage's _mapcount to
+ * compound_mapcount ot filter out PTE mapped case.

s/ ot / to /

oops, will fix it.

Quoted 10 lines
Quoted 10 lines
include/linux/page-flags.h +5 -1
   */
  static inline int PageTransCompoundMap(struct page *page)
  {
-	return PageTransCompound(page) && atomic_read(&page->_mapcount) < 0;
+	struct page *head;
+	int map_count;
+
+	if (!PageTransCompound(page))
+		return 0;
Quoted 8 lines

include/linux/page-flags.h static inline int PageTransCompound(struct page *page) · line 646

Quoted 7 lines
include/linux/page-flags.h +7 -0
+
+	if (PageAnon(page))
+		return atomic_read(&page->_mapcount) < 0;
+
+	head = compound_head(page);
+	map_count = atomic_read(&page->_mapcount);
+	/* File THP is PMD mapped and not double mapped */

s/ double / PTE /

Will fix it.

Quoted 5 lines

include/linux/page-flags.h static inline int PageTransCompound(struct page *page) · line 647

Quoted 2 lines
include/linux/page-flags.h +1 -0
+	return map_count >= 0 &&

You have added a map_count >= 0 test there. Okay, not wrong, but not necessary, and not consistent with what's returned in the PageAnon case (if this were called for an unmapped page).

I was thinking about this too. I'm wondering there might be a case that the PMD is split and it was the last PMD map, in this case subpage's _mapcount is also equal to compound_mapcount (both is -1). So, it would return true, then KVM may setup PMD map in EPT, but it might be PTE mapped later on the host. But, I'm not quite sure if this is really possible or if this is really a integrity problem. So, I thought it might be safer to add this check.

Quoted 18 lines

But asking for consistency in this function is asking for too much. It is *very* specialized to the particular places from which it is called (doesn't really belong in page-flags.h at all), and just so long as it gives them the right answer most of the time, and errs on the safe side the rest of the time, it'll do.

(I don't know if it's ever called on a tail page: it would not crash if it were, but different tail pages may give different answers.)

Quoted 8 lines
include/linux/page-flags.h +1 -0
+	       map_count == atomic_read(&head[1].compound_mapcount);
  }
 
  /*
Signature · 1 line
1.8.3.1
#6 lore raw
Quoted 7 lines · Yang Shi
Quoted 5 lines · Hugh Dickins

include/linux/page-flags.h static inline int PageTransCompound(struct page *page) · line 647

Quoted 2 lines
include/linux/page-flags.h +1 -0
+	return map_count >= 0 &&

You have added a map_count >= 0 test there. Okay, not wrong, but not necessary, and not consistent with what's returned in the PageAnon case (if this were called for an unmapped page).

Quoted 6 lines

I was thinking about this too. I'm wondering there might be a case that the PMD is split and it was the last PMD map, in this case subpage's _mapcount is also equal to compound_mapcount (both is -1). So, it would return true, then KVM may setup PMD map in EPT, but it might be PTE mapped later on the host. But, I'm not quite sure if this is really possible or if this is really a integrity problem. So, I thought it might be safer to add this check.

The mmu_notifier_invalidate_range_start.._end() in __split_huge_pmd(), with KVM's locking and sequence counting, is required to protect against such races.

Hugh

author
#7 lore raw
Quoted 12 lines · Hugh Dickins
Quoted 11 lines · Yang Shi
Quoted 4 lines · Hugh Dickins

include/linux/page-flags.h static inline int PageTransCompound(struct page *page) · line 647

Quoted 1 line
include/linux/page-flags.h +1 -0
+	return map_count >= 0 &&

You have added a map_count >= 0 test there. Okay, not wrong, but not necessary, and not consistent with what's returned in the PageAnon case (if this were called for an unmapped page).

I was thinking about this too. I'm wondering there might be a case that the PMD is split and it was the last PMD map, in this case subpage's _mapcount is also equal to compound_mapcount (both is -1). So, it would return true, then KVM may setup PMD map in EPT, but it might be PTE mapped later on the host. But, I'm not quite sure if this is really possible or if this is really a integrity problem. So, I thought it might be safer to add this check.

Quoted 3 lines

The mmu_notifier_invalidate_range_start.._end() in __split_huge_pmd(), with KVM's locking and sequence counting, is required to protect against such races.

OK, it sounds safe. Thanks for confirming. Will post v4 soon.

Quoted 2 lines

Hugh