Skip to contentSkip to conversation

mm: hold PTL from the first PTE while reclaiming a large folio

RFC PATCH Superseded Barry Song 4 participants 28 messages Mar 4, 2024 – Mar 5, 2024

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Barry Song

author
#0 lore raw

RFC PATCH 1/1 mm: hold PTL from the first PTE while reclaiming a large folio

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().
For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

---
 mm/vmscan.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/vmscan.c +11 -0
index 0b888a2afa58..e4722fbbcd0c 100644
@@ -1270,6 +1270,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
 			if (folio_test_pmd_mappable(folio))
 				flags |= TTU_SPLIT_HUGE_PMD;
+			/*
+			 * if page table lock is not held from the first PTE of
+			 * a large folio, some PTEs might be skipped because of
+			 * races with break-before-make, for example, PTEs can
+			 * be pte_none intermediately, thus one or more PTEs
+			 * might be skipped in try_to_unmap_one, we might result
+			 * in a large folio is partially mapped and partially
+			 * unmapped after try_to_unmap
+			 */
+			if (folio_test_large(folio))
+				flags |= TTU_SYNC;
 
 			try_to_unmap(folio, flags);
 			if (folio_mapped(folio)) {
Signature · 1 line
2.34.1
#1 lore raw

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

Quoted 29 lines

For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

Quoted 2 lines

Do we need a Fixes tag?

Quoted 13 lines
---
 mm/vmscan.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/vmscan.c +1 -0
index 0b888a2afa58..e4722fbbcd0c 100644
@@ -1270,6 +1270,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
 			if (folio_test_pmd_mappable(folio))
 				flags |= TTU_SPLIT_HUGE_PMD;
+			/*

mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1283

Quoted 10 lines
mm/vmscan.c +10 -0
+			 * if page table lock is not held from the first PTE of
+			 * a large folio, some PTEs might be skipped because of
+			 * races with break-before-make, for example, PTEs can
+			 * be pte_none intermediately, thus one or more PTEs
+			 * might be skipped in try_to_unmap_one, we might result
+			 * in a large folio is partially mapped and partially
+			 * unmapped after try_to_unmap
+			 */
+			if (folio_test_large(folio))
+				flags |= TTU_SYNC;

This looks sensible to me after thinking about it for a while. But I also have a gut feeling that there might be some more subtleties that are going over my head, since I'm not expert in this area. So will leave others to provide R-b :)

Thanks,
Ryan

Quoted 3 lines
mm/vmscan.c +0 -0
 			try_to_unmap(folio, flags);
 			if (folio_mapped(folio)) {
#2 lore raw
Quoted 19 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

Quoted 2 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

That is my understanding.

Quoted 22 lines
Quoted 22 lines

For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call

Quoted 8 lines
Quoted 8 lines

page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

I'm sure that wall of text can be formatted in a better way :) . Also, I think we can drop some of the details,

If you need some inspiration, I can give it a shot.

Quoted 5 lines
Quoted 2 lines

Do we need a Fixes tag?

What would be the description of the problem we are fixing?

1) failing to unmap?

That can happen with small folios as well IIUC.

2) Putting the large folio on the deferred split queue?

That sounds more reasonable.

Quoted 19 lines
Quoted 19 lines
---
  mm/vmscan.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
mm/vmscan.c +7 -0
index 0b888a2afa58..e4722fbbcd0c 100644
@@ -1270,6 +1270,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
  			if (folio_test_pmd_mappable(folio))
  				flags |= TTU_SPLIT_HUGE_PMD;
+			/*
+			 * if page table lock is not held from the first PTE of
+			 * a large folio, some PTEs might be skipped because of
+			 * races with break-before-make, for example, PTEs can
+			 * be pte_none intermediately, thus one or more PTEs
+			 * might be skipped in try_to_unmap_one, we might result
+			 * in a large folio is partially mapped and partially
Quoted 9 lines

mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1283

Quoted 4 lines
mm/vmscan.c +4 -0
+			 * unmapped after try_to_unmap
+			 */
+			if (folio_test_large(folio))
+				flags |= TTU_SYNC;

This looks sensible to me after thinking about it for a while. But I also have a gut feeling that there might be some more subtleties that are going over my head, since I'm not expert in this area. So will leave others to provide R-b :)

As we are seeing more such problems with lockless PT walks, maybe we really want some other special value (nonswap entry?) to indicate that a PTE this is currently ondergoing protection changes. So we'd avoid the pte_none() temporarily, if possible.

Without that, TTU_SYNC feels like the right thing to do.

Signature · 3 lines
Cheers,

David / dhildenb
#3 lore raw
Quoted 69 lines · David Hildenbrand
Quoted 19 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

Quoted 2 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

That is my understanding.

Quoted 22 lines
Quoted 22 lines

For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call

Quoted 8 lines
Quoted 8 lines

page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

I'm sure that wall of text can be formatted in a better way :) . Also, I think we can drop some of the details,

If you need some inspiration, I can give it a shot.

Quoted 5 lines
Quoted 2 lines

Do we need a Fixes tag?

What would be the description of the problem we are fixing?

Quoted 7 lines

1) failing to unmap?

That can happen with small folios as well IIUC.

2) Putting the large folio on the deferred split queue?

That sounds more reasonable.

Isn't the real problem today that we can end up writng a THP to the swap file (so 2M more IO and space used) but we can't remove it from memory, so no actual reclaim happens? Although I guess your (2) is really just another way of saying that.

Quoted 37 lines
Quoted 20 lines
Quoted 20 lines

---
  mm/vmscan.c | 11 +++++++++++

  1 file changed, 11 insertions(+)
mm/vmscan.c +7 -0
index 0b888a2afa58..e4722fbbcd0c 100644
@@ -1270,6 +1270,17 @@ static unsigned int shrink_folio_list(struct list_head
 *folio_list,
                 if (folio_test_pmd_mappable(folio))
                   flags |= TTU_SPLIT_HUGE_PMD;
+            /*
+             * if page table lock is not held from the first PTE of
+             * a large folio, some PTEs might be skipped because of
+             * races with break-before-make, for example, PTEs can
+             * be pte_none intermediately, thus one or more PTEs
+             * might be skipped in try_to_unmap_one, we might result
+             * in a large folio is partially mapped and partially
Quoted 9 lines

mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1283

Quoted 4 lines
mm/vmscan.c +4 -0
+             * unmapped after try_to_unmap
+             */
+            if (folio_test_large(folio))
+                flags |= TTU_SYNC;

This looks sensible to me after thinking about it for a while. But I also have a gut feeling that there might be some more subtleties that are going over my head, since I'm not expert in this area. So will leave others to provide R-b :)

As we are seeing more such problems with lockless PT walks, maybe we really want some other special value (nonswap entry?) to indicate that a PTE this is currently ondergoing protection changes. So we'd avoid the pte_none() temporarily, if possible.

Without that, TTU_SYNC feels like the right thing to do.

#4 lore raw
Quoted 74 lines · Ryan Roberts
Quoted 73 lines · David Hildenbrand
Quoted 19 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

Quoted 2 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

That is my understanding.

Quoted 22 lines
Quoted 22 lines

For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call

Quoted 8 lines
Quoted 8 lines

page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

I'm sure that wall of text can be formatted in a better way :) . Also, I think we can drop some of the details,

If you need some inspiration, I can give it a shot.

Quoted 5 lines
Quoted 2 lines

Do we need a Fixes tag?

What would be the description of the problem we are fixing?

1) failing to unmap?

That can happen with small folios as well IIUC.

Quoted 8 lines
Quoted 3 lines

2) Putting the large folio on the deferred split queue?

That sounds more reasonable.

Isn't the real problem today that we can end up writng a THP to the swap file (so 2M more IO and space used) but we can't remove it from memory, so no actual reclaim happens? Although I guess your (2) is really just another way of saying that.

The same could happen with small folios I believe? We might end up running into the

folio_mapped()

after the try_to_unmap().

Note that the actual I/O does not happen during add_to_swap(), but during the pageout() call when we find the folio to be dirty.

So there would not actually be more I/O. Only swap space would be reserved, that would be used later when not running into the race.

Signature · 3 lines
Cheers,

David / dhildenb
author
#5 lore raw
Quoted 90 lines · David Hildenbrand
Quoted 74 lines · Ryan Roberts
Quoted 73 lines · David Hildenbrand
Quoted 19 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

Quoted 2 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

That is my understanding.

Quoted 22 lines
Quoted 22 lines

For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call

Quoted 8 lines
Quoted 8 lines

page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

I'm sure that wall of text can be formatted in a better way :) . Also, I think we can drop some of the details,

If you need some inspiration, I can give it a shot.

Quoted 5 lines
Quoted 2 lines

Do we need a Fixes tag?

What would be the description of the problem we are fixing?

1) failing to unmap?

That can happen with small folios as well IIUC.

Quoted 8 lines
Quoted 3 lines

2) Putting the large folio on the deferred split queue?

That sounds more reasonable.

Isn't the real problem today that we can end up writng a THP to the swap file (so 2M more IO and space used) but we can't remove it from memory, so no actual reclaim happens? Although I guess your (2) is really just another way of saying that.

The same could happen with small folios I believe? We might end up running into the

folio_mapped()

Quoted 7 lines

after the try_to_unmap().

Note that the actual I/O does not happen during add_to_swap(), but during the pageout() call when we find the folio to be dirty.

So there would not actually be more I/O. Only swap space would be reserved, that would be used later when not running into the race.

I am not worried about small folios at all as they have only one PTE. so the PTE is either completely unmapped or completely mapped.

In terms of large folios, it is a different story. for example, a large folio with 16 PTEs with CONT-PTE, we will have

1. unfolded CONT-PTE, eg. PTE0 present, PTE1-PTE15 swap entries

2. page faults on PTE1-PTE15 after try_to_unmap if we access them.

This is totally useless PF and can be avoided if we can try_to_unmap properly at the beginning.

3. potential need to split a large folio afterwards. for example, MADV_PAGEOUT, MADV_FREE might split it after finding it is not completely mapped.

For small folios, we don't have any concern on the above issues.

Quoted 6 lines

--
Cheers,

David / dhildenb

Thanks
Barry

#6 lore raw
Quoted 109 lines · Barry Song
Quoted 90 lines · David Hildenbrand
Quoted 74 lines · Ryan Roberts
Quoted 73 lines · David Hildenbrand
Quoted 19 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

Quoted 2 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

That is my understanding.

Quoted 22 lines
Quoted 22 lines

For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call

Quoted 8 lines
Quoted 8 lines

page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

I'm sure that wall of text can be formatted in a better way :) . Also, I think we can drop some of the details,

If you need some inspiration, I can give it a shot.

Quoted 5 lines
Quoted 2 lines

Do we need a Fixes tag?

What would be the description of the problem we are fixing?

1) failing to unmap?

That can happen with small folios as well IIUC.

Quoted 8 lines
Quoted 3 lines

2) Putting the large folio on the deferred split queue?

That sounds more reasonable.

Isn't the real problem today that we can end up writng a THP to the swap file (so 2M more IO and space used) but we can't remove it from memory, so no actual reclaim happens? Although I guess your (2) is really just another way of saying that.

The same could happen with small folios I believe? We might end up running into the

folio_mapped()

Quoted 7 lines

after the try_to_unmap().

Note that the actual I/O does not happen during add_to_swap(), but during the pageout() call when we find the folio to be dirty.

So there would not actually be more I/O. Only swap space would be reserved, that would be used later when not running into the race.

I am not worried about small folios at all as they have only one PTE. so the PTE is either completely unmapped or completely mapped.

In terms of large folios, it is a different story. for example, a large folio with 16 PTEs with CONT-PTE, we will have

1. unfolded CONT-PTE, eg. PTE0 present, PTE1-PTE15 swap entries

2. page faults on PTE1-PTE15 after try_to_unmap if we access them.

Quoted 7 lines

This is totally useless PF and can be avoided if we can try_to_unmap properly at the beginning.

3. potential need to split a large folio afterwards. for example, MADV_PAGEOUT, MADV_FREE might split it after finding it is not completely mapped.

For small folios, we don't have any concern on the above issues.

Right, but when we talk about "Fixes:", what exactly are we consider "really broken" above and what is "undesired"?

(a) is there a correctness issue? I don't think so.

(b) is there a real performance issue? I'd like to understand.

After all, we've been living with that ever since we supported THP_SWAP, correct? "something does not work ideally in some corner cases" might be reasonable to handle here (and I really think we should), but might not be worth a "Fixes:".

So if we could clarify that, it would be great.

Signature · 3 lines
Cheers,

David / dhildenb
author
#7 lore raw
Quoted 124 lines · David Hildenbrand
Quoted 109 lines · Barry Song
Quoted 90 lines · David Hildenbrand
Quoted 74 lines · Ryan Roberts
Quoted 73 lines · David Hildenbrand
Quoted 19 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

Quoted 2 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

That is my understanding.

Quoted 22 lines
Quoted 22 lines

For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call

Quoted 8 lines
Quoted 8 lines

page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

I'm sure that wall of text can be formatted in a better way :) . Also, I think we can drop some of the details,

If you need some inspiration, I can give it a shot.

Quoted 5 lines
Quoted 2 lines

Do we need a Fixes tag?

What would be the description of the problem we are fixing?

1) failing to unmap?

That can happen with small folios as well IIUC.

Quoted 8 lines
Quoted 3 lines

2) Putting the large folio on the deferred split queue?

That sounds more reasonable.

Isn't the real problem today that we can end up writng a THP to the swap file (so 2M more IO and space used) but we can't remove it from memory, so no actual reclaim happens? Although I guess your (2) is really just another way of saying that.

The same could happen with small folios I believe? We might end up running into the

folio_mapped()

Quoted 7 lines

after the try_to_unmap().

Note that the actual I/O does not happen during add_to_swap(), but during the pageout() call when we find the folio to be dirty.

So there would not actually be more I/O. Only swap space would be reserved, that would be used later when not running into the race.

I am not worried about small folios at all as they have only one PTE. so the PTE is either completely unmapped or completely mapped.

In terms of large folios, it is a different story. for example, a large folio with 16 PTEs with CONT-PTE, we will have

1. unfolded CONT-PTE, eg. PTE0 present, PTE1-PTE15 swap entries

2. page faults on PTE1-PTE15 after try_to_unmap if we access them.

Quoted 7 lines

This is totally useless PF and can be avoided if we can try_to_unmap properly at the beginning.

3. potential need to split a large folio afterwards. for example, MADV_PAGEOUT, MADV_FREE might split it after finding it is not completely mapped.

For small folios, we don't have any concern on the above issues.

Right, but when we talk about "Fixes:", what exactly are we consider "really broken" above and what is "undesired"?

(a) is there a correctness issue? I don't think so.

Quoted 8 lines

(b) is there a real performance issue? I'd like to understand.

After all, we've been living with that ever since we supported THP_SWAP, correct? "something does not work ideally in some corner cases" might be reasonable to handle here (and I really think we should), but might not be worth a "Fixes:".

So if we could clarify that, it would be great.

I don't think this needs a fixes tag, and I would think it is an optimization on corner cases. And I don't think we will see noticeable performance improvement as this isn't happening quite often though I believe it does improve the performance of corner cases. but if the corner case only takes 0.1% of all try_to_unmap_one, no noticeable performance improvement will be seen.

I feel it is more of a behavior to kick flies out while flies don't kill people but it can be sometimes quite annoying :-) I ran into another bug in large-folio swapin series due to this problem, 16 PTEs had contiguous swap entries from Ryan's add_to_swap(), but they were splitted in MADV_PAGEOUT because the folio was not completely mapped after try_to_unmap_one.

Then after some time, some of the 16 pages were in swapcache, while others were not in. In this case, I couldn't swap them in together and had to handle PF one by one, but i was incorrectly handling it and trying to swap-in them together by reading swapfile. if they were atomically handled in try_to_unmap_one, we could avoid this.

we may have to cope with this kind of problem from time to time in future work.

Quoted 5 lines

--
Cheers,

David / dhildenb

Thanks
Barry

author
#8 lore raw
In reply to David Hildenbrand
Quoted 57 lines · David Hildenbrand
Quoted 19 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

Quoted 2 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

That is my understanding.

Quoted 22 lines
Quoted 22 lines

For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call

Quoted 8 lines
Quoted 8 lines

page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

Quoted 9 lines

I'm sure that wall of text can be formatted in a better way :) . Also, I think we can drop some of the details,

If you need some inspiration, I can give it a shot.

Quoted 4 lines
Quoted 2 lines

Do we need a Fixes tag?

I am not quite sure which commit should be here for a fixes tag. I think it's more of an optimization.

Quoted 11 lines
Quoted 1 line

What would be the description of the problem we are fixing?

1) failing to unmap?

That can happen with small folios as well IIUC.

2) Putting the large folio on the deferred split queue?

That sounds more reasonable.

I don't feel it is reasonable. Avoiding this kind of accident splitting from the kernel's improper code is a more reasonable approach as there is always a price to pay for splitting and unfolding PTEs etc.

While we can't avoid splitting coming from userspace's MADV_DONTNEED, munmap, mprotect, we have a way to ensure the kernel itself doesn't accidently break up a large folio.

In OPPO's phones, we ran into some weird bugs due to skipped PTEs in try_to_unmap_one. hardly could we fix it from the root cause. with various races, figuring out their timings was really a big pain :-)

But we did "resolve" those bugs by entirely untouching all PTEs if we found some PTEs were skipped in try_to_unmap_one [1].

While we find we only get the PTL from 2nd, 3rd but not 1st PTE, we entirely give up on try_to_unmap_one, and leave all PTEs untouched.

/* we are not starting from head */

if (!IS_ALIGNED((unsigned long)pvmw.pte, CONT_PTES * sizeof(*pvmw.pte))) {
                   ret = false;
                   atomic64_inc(&perf_stat.mapped_walk_start_from_non_head);
                   set_pte_at(mm, address, pvmw.pte, pteval);
                   page_vma_mapped_walk_done(&pvmw);
                   break;
}

This will ensure all PTEs still have a unified state such as CONT-PTE after try_to_unmap fails.
I feel this could have some false postive because when racing with unmap, 1st PTE might really become pte_none. So explicitly holding PTL from 1st PTE seems a better way.

[1] https://github.com/OnePlusOSS/android_kernel_oneplus_sm8550/blob/oneplus/sm8550_u_14.0.0_oneplus11/mm/rmap.c#L1730

Quoted 35 lines
Quoted 20 lines
Quoted 20 lines
---
  mm/vmscan.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
mm/vmscan.c +7 -0
index 0b888a2afa58..e4722fbbcd0c 100644
@@ -1270,6 +1270,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
                     if (folio_test_pmd_mappable(folio))
                             flags |= TTU_SPLIT_HUGE_PMD;
+                    /*
+                     * if page table lock is not held from the first PTE of
+                     * a large folio, some PTEs might be skipped because of
+                     * races with break-before-make, for example, PTEs can
+                     * be pte_none intermediately, thus one or more PTEs
+                     * might be skipped in try_to_unmap_one, we might result
+                     * in a large folio is partially mapped and partially
Quoted 9 lines

mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1283

Quoted 4 lines
mm/vmscan.c +4 -0
+                     * unmapped after try_to_unmap
+                     */
+                    if (folio_test_large(folio))
+                            flags |= TTU_SYNC;

This looks sensible to me after thinking about it for a while. But I also have a gut feeling that there might be some more subtleties that are going over my head, since I'm not expert in this area. So will leave others to provide R-b :)

As we are seeing more such problems with lockless PT walks, maybe we really want some other special value (nonswap entry?) to indicate that a PTE this is currently ondergoing protection changes. So we'd avoid the pte_none() temporarily, if possible.

Quoted 7 lines

Without that, TTU_SYNC feels like the right thing to do.

--
Cheers,

David / dhildenb

Thanks
Barry

#9 lore raw
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

Do we need a Fixes tag?

I am not quite sure which commit should be here for a fixes tag. I think it's more of an optimization.

Good, that helps!

Quoted 18 lines
Quoted 5 lines
Quoted 2 lines

What would be the description of the problem we are fixing?

Quoted 7 lines

1) failing to unmap?

That can happen with small folios as well IIUC.

2) Putting the large folio on the deferred split queue?

That sounds more reasonable.

I don't feel it is reasonable. Avoiding this kind of accident splitting from the kernel's improper code is a more reasonable approach as there is always a price to pay for splitting and unfolding PTEs etc.

Quoted 4 lines

While we can't avoid splitting coming from userspace's MADV_DONTNEED, munmap, mprotect, we have a way to ensure the kernel itself doesn't accidently break up a large folio.

Note that on the next vmscan we would retry, find the remaining present entries and swapout that thing completely :)

Quoted 5 lines

In OPPO's phones, we ran into some weird bugs due to skipped PTEs in try_to_unmap_one. hardly could we fix it from the root cause. with various races, figuring out their timings was really a big pain :-)

I can imagine. I assume, though, that it might be related to the way the cont-pte bit was handled. Ryan's implementation should be able to cope with that.

Quoted 15 lines

But we did "resolve" those bugs by entirely untouching all PTEs if we found some PTEs were skipped in try_to_unmap_one [1].

While we find we only get the PTL from 2nd, 3rd but not 1st PTE, we entirely give up on try_to_unmap_one, and leave all PTEs untouched.

/* we are not starting from head */

if (!IS_ALIGNED((unsigned long)pvmw.pte, CONT_PTES * sizeof(*pvmw.pte))) {
                    ret = false;
                    atomic64_inc(&perf_stat.mapped_walk_start_from_non_head);
                    set_pte_at(mm, address, pvmw.pte, pteval);
                    page_vma_mapped_walk_done(&pvmw);
                    break;
}
Quoted 5 lines

This will ensure all PTEs still have a unified state such as CONT-PTE after try_to_unmap fails.
I feel this could have some false postive because when racing with unmap, 1st PTE might really become pte_none. So explicitly holding PTL from 1st PTE seems a better way.

Can we estimate the "cost" of holding the PTL?

Signature · 3 lines
Cheers,

David / dhildenb
author
#10 lore raw
Quoted 27 lines · David Hildenbrand
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

Do we need a Fixes tag?

I am not quite sure which commit should be here for a fixes tag. I think it's more of an optimization.

Good, that helps!

Quoted 18 lines
Quoted 5 lines
Quoted 2 lines

What would be the description of the problem we are fixing?

Quoted 7 lines

1) failing to unmap?

That can happen with small folios as well IIUC.

2) Putting the large folio on the deferred split queue?

That sounds more reasonable.

I don't feel it is reasonable. Avoiding this kind of accident splitting from the kernel's improper code is a more reasonable approach as there is always a price to pay for splitting and unfolding PTEs etc.

Quoted 7 lines
Quoted 4 lines

While we can't avoid splitting coming from userspace's MADV_DONTNEED, munmap, mprotect, we have a way to ensure the kernel itself doesn't accidently break up a large folio.

Note that on the next vmscan we would retry, find the remaining present entries and swapout that thing completely :)

This is true, but since we can finish the job the first time, it seems second retry is a cost :-)

Quoted 10 lines
Quoted 6 lines

In OPPO's phones, we ran into some weird bugs due to skipped PTEs in try_to_unmap_one. hardly could we fix it from the root cause. with various races, figuring out their timings was really a big pain :-)

I can imagine. I assume, though, that it might be related to the way the cont-pte bit was handled. Ryan's implementation should be able to cope with that.

I guess you are probably right. Ryan's implementation decouples CONT-PTE from mm core. nice to have it.

Quoted 22 lines
Quoted 16 lines

But we did "resolve" those bugs by entirely untouching all PTEs if we found some PTEs were skipped in try_to_unmap_one [1].

While we find we only get the PTL from 2nd, 3rd but not 1st PTE, we entirely give up on try_to_unmap_one, and leave all PTEs untouched.

/* we are not starting from head */

if (!IS_ALIGNED((unsigned long)pvmw.pte, CONT_PTES * sizeof(*pvmw.pte))) {
                    ret = false;
                    atomic64_inc(&perf_stat.mapped_walk_start_from_non_head);
                    set_pte_at(mm, address, pvmw.pte, pteval);
                    page_vma_mapped_walk_done(&pvmw);
                    break;
}
Quoted 5 lines

This will ensure all PTEs still have a unified state such as CONT-PTE after try_to_unmap fails.
I feel this could have some false postive because when racing with unmap, 1st PTE might really become pte_none. So explicitly holding PTL from 1st PTE seems a better way.

Quoted 2 lines

Can we estimate the "cost" of holding the PTL?

This is just moving PTL acquisition one or two PTE earlier in those corner cases. In normal cases, it doesn't affect when PTL is held.

In normal cases, page_vma_mapped_walk will find PTE0 is present, thus hold PTL immediately. in corner cases, page_vma_mapped_walk races with break- before-make, after skipping one or two PTEs whose states are transferring, it will find a present pte then acquire lock.

Quoted 4 lines

--
Cheers,

David / dhildenb

Thanks
Barry

#11 lore raw
Quoted 22 lines · Barry Song
Quoted 21 lines · David Hildenbrand
Quoted 20 lines

But we did "resolve" those bugs by entirely untouching all PTEs if we found some PTEs were skipped in try_to_unmap_one [1].

While we find we only get the PTL from 2nd, 3rd but not 1st PTE, we entirely give up on try_to_unmap_one, and leave all PTEs untouched.

/* we are not starting from head */

if (!IS_ALIGNED((unsigned long)pvmw.pte, CONT_PTES * sizeof(*pvmw.pte))) {
                    ret = false;
                    atomic64_inc(&perf_stat.mapped_walk_start_from_non_head);
                    set_pte_at(mm, address, pvmw.pte, pteval);
                    page_vma_mapped_walk_done(&pvmw);
                    break;
}

This will ensure all PTEs still have a unified state such as CONT-PTE after try_to_unmap fails.
I feel this could have some false postive because when racing with unmap, 1st PTE might really become pte_none. So explicitly holding PTL from 1st PTE seems a better way.

Quoted 5 lines
Quoted 2 lines

Can we estimate the "cost" of holding the PTL?

This is just moving PTL acquisition one or two PTE earlier in those corner cases. In normal cases, it doesn't affect when PTL is held.

The mTHP may be mapped at the end of page table. In that case, the PTL will be held longer. Or am I missing something?

--
Best Regards,
Huang, Ying

Quoted 12 lines

In normal cases, page_vma_mapped_walk will find PTE0 is present, thus hold PTL immediately. in corner cases, page_vma_mapped_walk races with break- before-make, after skipping one or two PTEs whose states are transferring, it will find a present pte then acquire lock.

Quoted 4 lines

--
Cheers,

David / dhildenb

Thanks
Barry

author
#12 lore raw
Quoted 25 lines · Ying
Quoted 22 lines · Barry Song
Quoted 21 lines · David Hildenbrand
Quoted 20 lines

But we did "resolve" those bugs by entirely untouching all PTEs if we found some PTEs were skipped in try_to_unmap_one [1].

While we find we only get the PTL from 2nd, 3rd but not 1st PTE, we entirely give up on try_to_unmap_one, and leave all PTEs untouched.

/* we are not starting from head */

if (!IS_ALIGNED((unsigned long)pvmw.pte, CONT_PTES * sizeof(*pvmw.pte))) {
                    ret = false;
                    atomic64_inc(&perf_stat.mapped_walk_start_from_non_head);
                    set_pte_at(mm, address, pvmw.pte, pteval);
                    page_vma_mapped_walk_done(&pvmw);
                    break;
}

This will ensure all PTEs still have a unified state such as CONT-PTE after try_to_unmap fails.
I feel this could have some false postive because when racing with unmap, 1st PTE might really become pte_none. So explicitly holding PTL from 1st PTE seems a better way.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Can we estimate the "cost" of holding the PTL?

This is just moving PTL acquisition one or two PTE earlier in those corner cases. In normal cases, it doesn't affect when PTL is held.

The mTHP may be mapped at the end of page table. In that case, the PTL will be held longer. Or am I missing something?

no. this patch doesn't change when we release PTL but change when we get PTL.

when the original code iterates nr_pages PTEs in a large folio, it will skip invalid PTEs, when it meets a valid one, it will acquire PTL. so if it gets intermediate PTE values some other threads are modifying, it might skip PTE0, or sometimes PTE0 and PTE1 according to my test. but arriving at PTE2, likely other threads have written a new value, so we will begin to hold PTL and iterate till the end of the large folio.

The proposal is that we directly get PTL from PTE0, thus we don't get intermediate values for the head of nr_pages PTEs. this will ensure a large folio is either completely unmapped or completely mapped. but not partially mapped and partially unmapped.

Quoted 16 lines

--
Best Regards,
Huang, Ying

Quoted 10 lines

In normal cases, page_vma_mapped_walk will find PTE0 is present, thus hold PTL immediately. in corner cases, page_vma_mapped_walk races with break- before-make, after skipping one or two PTEs whose states are transferring, it will find a present pte then acquire lock.

Quoted 4 lines

--
Cheers,

David / dhildenb

Thanks
Barry

#13 lore raw
Quoted 38 lines · Barry Song
Quoted 25 lines · Ying
Quoted 22 lines · Barry Song
Quoted 21 lines · David Hildenbrand
Quoted 20 lines

But we did "resolve" those bugs by entirely untouching all PTEs if we found some PTEs were skipped in try_to_unmap_one [1].

While we find we only get the PTL from 2nd, 3rd but not 1st PTE, we entirely give up on try_to_unmap_one, and leave all PTEs untouched.

/* we are not starting from head */

if (!IS_ALIGNED((unsigned long)pvmw.pte, CONT_PTES * sizeof(*pvmw.pte))) {
                    ret = false;
                    atomic64_inc(&perf_stat.mapped_walk_start_from_non_head);
                    set_pte_at(mm, address, pvmw.pte, pteval);
                    page_vma_mapped_walk_done(&pvmw);
                    break;
}

This will ensure all PTEs still have a unified state such as CONT-PTE after try_to_unmap fails.
I feel this could have some false postive because when racing with unmap, 1st PTE might really become pte_none. So explicitly holding PTL from 1st PTE seems a better way.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Can we estimate the "cost" of holding the PTL?

This is just moving PTL acquisition one or two PTE earlier in those corner cases. In normal cases, it doesn't affect when PTL is held.

The mTHP may be mapped at the end of page table. In that case, the PTL will be held longer. Or am I missing something?

no. this patch doesn't change when we release PTL but change when we get PTL.

Quoted 6 lines

when the original code iterates nr_pages PTEs in a large folio, it will skip invalid PTEs, when it meets a valid one, it will acquire PTL. so if it gets intermediate PTE values some other threads are modifying, it might skip PTE0, or sometimes PTE0 and PTE1 according to my test. but arriving at PTE2, likely other threads have written a new value, so we will begin to hold PTL and iterate till the end of the large folio.

Is there any guarantee that the mTHP will always be mapped at the beginning of the page table (PTE0)? IIUC, mTHP can be mapped at PTE496. If so, with your patch, PTL will be held from PTE0 instead of PTE496 in some cases.

--
Best Regards,
Huang, Ying

Quoted 23 lines

The proposal is that we directly get PTL from PTE0, thus we don't get intermediate values for the head of nr_pages PTEs. this will ensure a large folio is either completely unmapped or completely mapped. but not partially mapped and partially unmapped.

Quoted 16 lines

--
Best Regards,
Huang, Ying

Quoted 10 lines

In normal cases, page_vma_mapped_walk will find PTE0 is present, thus hold PTL immediately. in corner cases, page_vma_mapped_walk races with break- before-make, after skipping one or two PTEs whose states are transferring, it will find a present pte then acquire lock.

Quoted 4 lines

--
Cheers,

David / dhildenb

Thanks
Barry

author
#14 lore raw
Quoted 48 lines · Ying
Quoted 38 lines · Barry Song
Quoted 25 lines · Ying
Quoted 22 lines · Barry Song
Quoted 21 lines · David Hildenbrand
Quoted 20 lines

But we did "resolve" those bugs by entirely untouching all PTEs if we found some PTEs were skipped in try_to_unmap_one [1].

While we find we only get the PTL from 2nd, 3rd but not 1st PTE, we entirely give up on try_to_unmap_one, and leave all PTEs untouched.

/* we are not starting from head */

if (!IS_ALIGNED((unsigned long)pvmw.pte, CONT_PTES * sizeof(*pvmw.pte))) {
                    ret = false;
                    atomic64_inc(&perf_stat.mapped_walk_start_from_non_head);
                    set_pte_at(mm, address, pvmw.pte, pteval);
                    page_vma_mapped_walk_done(&pvmw);
                    break;
}

This will ensure all PTEs still have a unified state such as CONT-PTE after try_to_unmap fails.
I feel this could have some false postive because when racing with unmap, 1st PTE might really become pte_none. So explicitly holding PTL from 1st PTE seems a better way.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Can we estimate the "cost" of holding the PTL?

This is just moving PTL acquisition one or two PTE earlier in those corner cases. In normal cases, it doesn't affect when PTL is held.

The mTHP may be mapped at the end of page table. In that case, the PTL will be held longer. Or am I missing something?

no. this patch doesn't change when we release PTL but change when we get PTL.

Quoted 6 lines

when the original code iterates nr_pages PTEs in a large folio, it will skip invalid PTEs, when it meets a valid one, it will acquire PTL. so if it gets intermediate PTE values some other threads are modifying, it might skip PTE0, or sometimes PTE0 and PTE1 according to my test. but arriving at PTE2, likely other threads have written a new value, so we will begin to hold PTL and iterate till the end of the large folio.

Quoted 4 lines

Is there any guarantee that the mTHP will always be mapped at the beginning of the page table (PTE0)? IIUC, mTHP can be mapped at PTE496. If so, with your patch, PTL will be held from PTE0 instead of PTE496 in some cases.

I agree. but in another discussion[1], the plan is if we find a large folio has been deferred split, we split it before try_to_unmap and pageout. otherwise, we may result in lots of redundant I/O, because PTE0-495 will still be pageout()-ed.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 26 lines

--
Best Regards,
Huang, Ying

Quoted 21 lines

The proposal is that we directly get PTL from PTE0, thus we don't get intermediate values for the head of nr_pages PTEs. this will ensure a large folio is either completely unmapped or completely mapped. but not partially mapped and partially unmapped.

Quoted 16 lines

--
Best Regards,
Huang, Ying

Quoted 10 lines

In normal cases, page_vma_mapped_walk will find PTE0 is present, thus hold PTL immediately. in corner cases, page_vma_mapped_walk races with break- before-make, after skipping one or two PTEs whose states are transferring, it will find a present pte then acquire lock.

Quoted 4 lines

--
Cheers,

David / dhildenb

Thanks
Barry

author
#15 lore raw
Quoted 55 lines · Barry Song
Quoted 48 lines · Ying
Quoted 38 lines · Barry Song
Quoted 25 lines · Ying
Quoted 22 lines · Barry Song
Quoted 21 lines · David Hildenbrand
Quoted 20 lines

But we did "resolve" those bugs by entirely untouching all PTEs if we found some PTEs were skipped in try_to_unmap_one [1].

While we find we only get the PTL from 2nd, 3rd but not 1st PTE, we entirely give up on try_to_unmap_one, and leave all PTEs untouched.

/* we are not starting from head */

if (!IS_ALIGNED((unsigned long)pvmw.pte, CONT_PTES * sizeof(*pvmw.pte))) {
                    ret = false;
                    atomic64_inc(&perf_stat.mapped_walk_start_from_non_head);
                    set_pte_at(mm, address, pvmw.pte, pteval);
                    page_vma_mapped_walk_done(&pvmw);
                    break;
}

This will ensure all PTEs still have a unified state such as CONT-PTE after try_to_unmap fails.
I feel this could have some false postive because when racing with unmap, 1st PTE might really become pte_none. So explicitly holding PTL from 1st PTE seems a better way.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Can we estimate the "cost" of holding the PTL?

This is just moving PTL acquisition one or two PTE earlier in those corner cases. In normal cases, it doesn't affect when PTL is held.

The mTHP may be mapped at the end of page table. In that case, the PTL will be held longer. Or am I missing something?

no. this patch doesn't change when we release PTL but change when we get PTL.

Quoted 6 lines

when the original code iterates nr_pages PTEs in a large folio, it will skip invalid PTEs, when it meets a valid one, it will acquire PTL. so if it gets intermediate PTE values some other threads are modifying, it might skip PTE0, or sometimes PTE0 and PTE1 according to my test. but arriving at PTE2, likely other threads have written a new value, so we will begin to hold PTL and iterate till the end of the large folio.

Quoted 4 lines

Is there any guarantee that the mTHP will always be mapped at the beginning of the page table (PTE0)? IIUC, mTHP can be mapped at PTE496. If so, with your patch, PTL will be held from PTE0 instead of PTE496 in some cases.

Quoted 6 lines

I agree. but in another discussion[1], the plan is if we find a large folio has been deferred split, we split it before try_to_unmap and pageout. otherwise, we may result in lots of redundant I/O, because PTE0-495 will still be pageout()-ed.

[1] https://lore.kernel.org/linux-mm/[email protected]/

I thought about this again, seems we can cope with it even w/o the above plan by:

+ if (folio_test_large(folio) && list_empty(&folio->_deferred_list))
+ flags |= TTU_SYNC;

if a folio has been deferred split, it seems no sense to have the optimization for the corner cases this patch wants to provide. Only while we know this folio is still entirely mapped, we have this optimization. This should have reduced the chance to be quite small though we still have a bit.

Quoted 6 lines
Quoted 6 lines

--
Best Regards,
Huang, Ying

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The proposal is that we directly get PTL from PTE0, thus we don't get intermediate values for the head of nr_pages PTEs. this will ensure a large folio is either completely unmapped or completely mapped. but not partially mapped and partially unmapped.

Quoted 1 line

Thanks
Barry

#16 lore raw
In reply to Barry Song
Quoted 61 lines · Barry Song
Quoted 60 lines · David Hildenbrand
Quoted 19 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

Quoted 2 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

That is my understanding.

Quoted 22 lines
Quoted 22 lines

For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call

Quoted 8 lines
Quoted 8 lines

page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

I'm sure that wall of text can be formatted in a better way :) . Also, I think we can drop some of the details,

Quoted 6 lines
Quoted 6 lines

If you need some inspiration, I can give it a shot.

Quoted 4 lines
Quoted 2 lines

Do we need a Fixes tag?

It seems my original question has snowballed a bit. I was conflating this change with other reports Barry has made where the kernel was panicking (I think?). Given we are not seeing any incorrect functional behaviour that this change fixes, I agree we don't need a Fixes tag here.

#17 lore raw
In reply to David Hildenbrand
Quoted 5 lines · David Hildenbrand

As we are seeing more such problems with lockless PT walks, maybe we really want some other special value (nonswap entry?) to indicate that a PTE this is currently ondergoing protection changes. So we'd avoid the pte_none() temporarily, if possible.

This sounds like a good idea. This can solve other issue caused by temporarily pte_none() issue too, like the following,

https://lore.kernel.org/linux-mm/[email protected]/

--
Best Regards,
Huang, Ying

author
#18 lore raw
In reply to Ryan Roberts
Quoted 13 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

Quoted 6 lines

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

I think this affects all large folios with PTEs entries more than 1. but hugeTLB is handled as a whole in try_to_unmap_one and its rmap is removed all together, i feel hugeTLB doesn't have this problem.

Quoted 3 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

right. as now large folios become a more common case, and it is my case running in millions of phones.

BTW, I feel we can somehow learn from hugeTLB, for example, we can reclaim all PTEs all together rather than iterating PTEs one by one. This will improve performance. for example, a batched
set_ptes_to_swap_entries()
{
}
then we only need to loop once for a large folio, right now we are looping nr_pages times.

Quoted 30 lines
Quoted 30 lines

For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

Quoted 4 lines
Quoted 2 lines

Do we need a Fixes tag?

I don't feel a strong need for this as this doesn't cause a crash, memory leak or whatever serious.

Quoted 20 lines
Quoted 20 lines
---
 mm/vmscan.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/vmscan.c +7 -0
index 0b888a2afa58..e4722fbbcd0c 100644
@@ -1270,6 +1270,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
                      if (folio_test_pmd_mappable(folio))
                              flags |= TTU_SPLIT_HUGE_PMD;
+                     /*
+                      * if page table lock is not held from the first PTE of
+                      * a large folio, some PTEs might be skipped because of
+                      * races with break-before-make, for example, PTEs can
+                      * be pte_none intermediately, thus one or more PTEs
+                      * might be skipped in try_to_unmap_one, we might result
+                      * in a large folio is partially mapped and partially
Quoted 9 lines

mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1283

Quoted 4 lines
mm/vmscan.c +4 -0
+                      * unmapped after try_to_unmap
+                      */
+                     if (folio_test_large(folio))
+                             flags |= TTU_SYNC;

This looks sensible to me after thinking about it for a while. But I also have a gut feeling that there might be some more subtleties that are going over my head, since I'm not expert in this area. So will leave others to provide R-b :)

ok, thanks :-)

Quoted 7 lines

Thanks,
Ryan

Quoted 3 lines
mm/vmscan.c +0 -0
                      try_to_unmap(folio, flags);
                      if (folio_mapped(folio)) {

Thanks
Barry

#19 lore raw
Quoted 32 lines · Barry Song
Quoted 13 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

Quoted 6 lines

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

I think this affects all large folios with PTEs entries more than 1. but hugeTLB is handled as a whole in try_to_unmap_one and its rmap is removed all together, i feel hugeTLB doesn't have this problem.

Quoted 3 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

right. as now large folios become a more common case, and it is my case running in millions of phones.

Quoted 8 lines

BTW, I feel we can somehow learn from hugeTLB, for example, we can reclaim all PTEs all together rather than iterating PTEs one by one. This will improve performance. for example, a batched
set_ptes_to_swap_entries()
{
}
then we only need to loop once for a large folio, right now we are looping nr_pages times.

You still need a pte-pte loop somewhere. In hugetlb's case it's in the arch implementation. HugeTLB ptes are all a fixed size for a given VMA, which makes things a bit easier too, whereas in the regular mm, they are now a variable size.

David and I introduced folio_pte_batch() to help gather batches of ptes, and it uses the contpte bit to avoid iterating over intermediate ptes. And I'm adding swap_pte_batch() which does a similar thing for swap entry batching in v4 of my swap-out series.

For your set_ptes_to_swap_entries() example, I'm not sure what it would do other than loop over the PTEs setting an incremented swap entry to each one? How is that more performant?

Thanks,
Ryan

author
#20 lore raw
Quoted 47 lines · Ryan Roberts
Quoted 32 lines · Barry Song
Quoted 13 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

Quoted 6 lines

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

I think this affects all large folios with PTEs entries more than 1. but hugeTLB is handled as a whole in try_to_unmap_one and its rmap is removed all together, i feel hugeTLB doesn't have this problem.

Quoted 3 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

right. as now large folios become a more common case, and it is my case running in millions of phones.

Quoted 8 lines

BTW, I feel we can somehow learn from hugeTLB, for example, we can reclaim all PTEs all together rather than iterating PTEs one by one. This will improve performance. for example, a batched
set_ptes_to_swap_entries()
{
}
then we only need to loop once for a large folio, right now we are looping nr_pages times.

You still need a pte-pte loop somewhere. In hugetlb's case it's in the arch implementation. HugeTLB ptes are all a fixed size for a given VMA, which makes things a bit easier too, whereas in the regular mm, they are now a variable size.

Quoted 8 lines

David and I introduced folio_pte_batch() to help gather batches of ptes, and it uses the contpte bit to avoid iterating over intermediate ptes. And I'm adding swap_pte_batch() which does a similar thing for swap entry batching in v4 of my swap-out series.

For your set_ptes_to_swap_entries() example, I'm not sure what it would do other than loop over the PTEs setting an incremented swap entry to each one? How is that more performant?

right now, while (page_vma_mapped_walk(&pvmw)) will loop nr_pages for each PTE, if each PTE, we do lots of checks within the loop.

by implementing set_ptes_to_swap_entries(), we can iterate once for page_vma_mapped_walk(), after folio_pte_batch() has confirmed the large folio is completely mapped, we set nr_pages swap entries all together.

we are replacing

for(i=0;i<nr_pages;i++) /* page_vma_mapped_walk */
{
lots of checks;
clear PTEn
set PTEn to swap
}

by

if (large folio && folio_pte_batch() == nr_pages)

    set_ptes_to_swap_entries().
Quoted 1 line

Thanks,
Ryan

#21 lore raw
Quoted 65 lines · Barry Song
Quoted 47 lines · Ryan Roberts
Quoted 32 lines · Barry Song
Quoted 13 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

Quoted 6 lines

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

I think this affects all large folios with PTEs entries more than 1. but hugeTLB is handled as a whole in try_to_unmap_one and its rmap is removed all together, i feel hugeTLB doesn't have this problem.

Quoted 3 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

right. as now large folios become a more common case, and it is my case running in millions of phones.

Quoted 8 lines

BTW, I feel we can somehow learn from hugeTLB, for example, we can reclaim all PTEs all together rather than iterating PTEs one by one. This will improve performance. for example, a batched
set_ptes_to_swap_entries()
{
}
then we only need to loop once for a large folio, right now we are looping nr_pages times.

You still need a pte-pte loop somewhere. In hugetlb's case it's in the arch implementation. HugeTLB ptes are all a fixed size for a given VMA, which makes things a bit easier too, whereas in the regular mm, they are now a variable size.

Quoted 8 lines

David and I introduced folio_pte_batch() to help gather batches of ptes, and it uses the contpte bit to avoid iterating over intermediate ptes. And I'm adding swap_pte_batch() which does a similar thing for swap entry batching in v4 of my swap-out series.

For your set_ptes_to_swap_entries() example, I'm not sure what it would do other than loop over the PTEs setting an incremented swap entry to each one? How is that more performant?

right now, while (page_vma_mapped_walk(&pvmw)) will loop nr_pages for each PTE, if each PTE, we do lots of checks within the loop.

by implementing set_ptes_to_swap_entries(), we can iterate once for page_vma_mapped_walk(), after folio_pte_batch() has confirmed the large folio is completely mapped, we set nr_pages swap entries all together.

Quoted 8 lines

we are replacing

for(i=0;i<nr_pages;i++) /* page_vma_mapped_walk */
{
lots of checks;
clear PTEn
set PTEn to swap
}

OK so you are effectively hoisting "lots of checks" out of the loop?

Quoted 10 lines

by

if (large folio && folio_pte_batch() == nr_pages)

    set_ptes_to_swap_entries().
Quoted 1 line

Thanks,
Ryan

author
#22 lore raw
Quoted 69 lines · Ryan Roberts
Quoted 67 lines · Barry Song
Quoted 47 lines · Ryan Roberts
Quoted 32 lines · Barry Song
Quoted 13 lines · Ryan Roberts

Hi Barry,

Quoted 8 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().

Quoted 6 lines

I just want to check my understanding here - I think the problem occurs for PTE-mapped, PMD-sized folios as well as smaller-than-PMD-size large folios? Now that I've had a look at the code and have a better understanding, I think that must be the case? And therefore this problem exists independently of my work to support swap-out of mTHP? (From your previous report I was under the impression that it only affected mTHP).

I think this affects all large folios with PTEs entries more than 1. but hugeTLB is handled as a whole in try_to_unmap_one and its rmap is removed all together, i feel hugeTLB doesn't have this problem.

Quoted 3 lines

Its just that the problem is becoming more pronounced because with mTHP, PTE-mapped large folios are much more common?

right. as now large folios become a more common case, and it is my case running in millions of phones.

Quoted 8 lines

BTW, I feel we can somehow learn from hugeTLB, for example, we can reclaim all PTEs all together rather than iterating PTEs one by one. This will improve performance. for example, a batched
set_ptes_to_swap_entries()
{
}
then we only need to loop once for a large folio, right now we are looping nr_pages times.

You still need a pte-pte loop somewhere. In hugetlb's case it's in the arch implementation. HugeTLB ptes are all a fixed size for a given VMA, which makes things a bit easier too, whereas in the regular mm, they are now a variable size.

Quoted 8 lines

David and I introduced folio_pte_batch() to help gather batches of ptes, and it uses the contpte bit to avoid iterating over intermediate ptes. And I'm adding swap_pte_batch() which does a similar thing for swap entry batching in v4 of my swap-out series.

For your set_ptes_to_swap_entries() example, I'm not sure what it would do other than loop over the PTEs setting an incremented swap entry to each one? How is that more performant?

right now, while (page_vma_mapped_walk(&pvmw)) will loop nr_pages for each PTE, if each PTE, we do lots of checks within the loop.

by implementing set_ptes_to_swap_entries(), we can iterate once for page_vma_mapped_walk(), after folio_pte_batch() has confirmed the large folio is completely mapped, we set nr_pages swap entries all together.

we are replacing

Quoted 8 lines
Quoted 6 lines

for(i=0;i<nr_pages;i++) /* page_vma_mapped_walk */
{
lots of checks;
clear PTEn
set PTEn to swap
}

OK so you are effectively hoisting "lots of checks" out of the loop?

no. page_vma_mapped_walk returns nr_pages times. We are doing same check each time. Each time, we do tlbi and set one PTE.

Quoted 6 lines
Quoted 6 lines

by

if (large folio && folio_pte_batch() == nr_pages)

    set_ptes_to_swap_entries().

for this, we do check for one time, and we do much less tlbi.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
Ryan

Thanks
Barry

#23 lore raw
In reply to Barry Song
Quoted 4 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs

Sorry, I don't know what is "break-before-make", can you elaborate? IIUC, ptep_modify_prot_start()/ptep_modify_prot_commit() can clear PTE temporarily, which may cause race with page_vma_mapped_walk(). Is that the issue that you try to fix?

--
Best Regards,
Huang, Ying

Quoted 61 lines

of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().
For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

---
 mm/vmscan.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/vmscan.c +11 -0
index 0b888a2afa58..e4722fbbcd0c 100644
@@ -1270,6 +1270,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
 			if (folio_test_pmd_mappable(folio))
 				flags |= TTU_SPLIT_HUGE_PMD;
+			/*
+			 * if page table lock is not held from the first PTE of
+			 * a large folio, some PTEs might be skipped because of
+			 * races with break-before-make, for example, PTEs can
+			 * be pte_none intermediately, thus one or more PTEs
+			 * might be skipped in try_to_unmap_one, we might result
+			 * in a large folio is partially mapped and partially
+			 * unmapped after try_to_unmap
+			 */
+			if (folio_test_large(folio))
+				flags |= TTU_SYNC;
 
 			try_to_unmap(folio, flags);
 			if (folio_mapped(folio)) {
author
#24 lore raw
Quoted 5 lines · Ying
Quoted 2 lines · Barry Song

From: Barry Song <[email protected]>

Quoted 7 lines
Quoted 2 lines

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs

Sorry, I don't know what is "break-before-make", can you elaborate? IIUC, ptep_modify_prot_start()/ptep_modify_prot_commit() can clear PTE temporarily, which may cause race with page_vma_mapped_walk(). Is that the issue that you try to fix?

we are writing pte to zero(break) before writing a new value(make). while this behavior is within PTL in another thread, page_vma_mapped_walk() of try_to_unmap_one thread won't take PTL till it meets a present PTE. for example, if another threads are modifying nr_pages PTEs under PTL, but we don't hold PTL, we might skip one or two PTEs at the beginning of a large folio.
For a large folio, after try_to_unmap_one(), we may result in PTE0 and PTE1 untouched but PTE2~nr_pages-1 are set to swap entries.

by holding PTL from PTE0 for large folios, we won't get these intermediate values. At the moment we get PTL, other threads have done.

Quoted 64 lines

--
Best Regards,
Huang, Ying

Quoted 59 lines

of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().
For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

---
 mm/vmscan.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/vmscan.c +11 -0
index 0b888a2afa58..e4722fbbcd0c 100644
@@ -1270,6 +1270,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
                      if (folio_test_pmd_mappable(folio))
                              flags |= TTU_SPLIT_HUGE_PMD;
+                     /*
+                      * if page table lock is not held from the first PTE of
+                      * a large folio, some PTEs might be skipped because of
+                      * races with break-before-make, for example, PTEs can
+                      * be pte_none intermediately, thus one or more PTEs
+                      * might be skipped in try_to_unmap_one, we might result
+                      * in a large folio is partially mapped and partially
+                      * unmapped after try_to_unmap
+                      */
+                     if (folio_test_large(folio))
+                             flags |= TTU_SYNC;
 
Quoted 2 lines

mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1286

Quoted 2 lines
mm/vmscan.c +0 -0
                      try_to_unmap(folio, flags);
                      if (folio_mapped(folio)) {

Thanks
Barry

#25 lore raw
Quoted 9 lines · Barry Song
Quoted 8 lines · Ying
Quoted 4 lines · Barry Song

From: Barry Song <[email protected]>

page_vma_mapped_walk() within try_to_unmap_one() races with other PTEs modification such as break-before-make, while iterating PTEs

Quoted 6 lines
Quoted 4 lines

Sorry, I don't know what is "break-before-make", can you elaborate? IIUC, ptep_modify_prot_start()/ptep_modify_prot_commit() can clear PTE temporarily, which may cause race with page_vma_mapped_walk(). Is that the issue that you try to fix?

we are writing pte to zero(break) before writing a new value(make).

OK. Is break and make is commonly used terminology in kernel? If not, it's better to explain a little (e.g., ptep_get_and_clear() / modify / set_pte_at()).

Quoted 3 lines

while
this behavior is within PTL in another thread, page_vma_mapped_walk() of try_to_unmap_one thread won't take PTL till it meets a present PTE.

IIUC, !pte_none() should be enough?

Quoted 8 lines

for example, if another threads are modifying nr_pages PTEs under PTL, but we don't hold PTL, we might skip one or two PTEs at the beginning of a large folio.
For a large folio, after try_to_unmap_one(), we may result in PTE0 and PTE1 untouched but PTE2~nr_pages-1 are set to swap entries.

by holding PTL from PTE0 for large folios, we won't get these intermediate values. At the moment we get PTL, other threads have done.

Got it! Thanks!

--
Best Regards,
Huang, Ying

Quoted 69 lines
Quoted 64 lines

--
Best Regards,
Huang, Ying

Quoted 59 lines

of a large folio, it will only begin to acquire PTL after it gets a valid(present) PTE. break-before-make intermediately sets PTEs to pte_none. Thus, a large folio's PTEs might be partially skipped in try_to_unmap_one().
For example, for an anon folio, after try_to_unmap_one(), we may have PTE0 present, while PTE1 ~ PTE(nr_pages - 1) are swap entries. So folio will be still mapped, the folio fails to be reclaimed. What’s even more worrying is, its PTEs are no longer in a unified state. This might lead to accident folio_split() afterwards. And since a part of PTEs are now swap entries, accessing them will incur page fault - do_swap_page.
It creates both anxiety and more expense. While we can't avoid userspace's unmap to break up unified PTEs such as CONT-PTE for a large folio, we can indeed keep away from kernel's breaking up them due to its code design.
This patch is holding PTL from PTE0, thus, the folio will either be entirely reclaimed or entirely kept. On the other hand, this approach doesn't increase PTL contention. Even w/o the patch, page_vma_mapped_walk() will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short, according to test. Of course, even w/o this patch, the vast majority of try_to_unmap_one still can get PTL from PTE0. This patch makes the number 100%.
The other option is that we can give up in try_to_unmap_one once we find PTE0 is not the first entry we get PTL, we call page_vma_mapped_walk_done() to end the iteration at this case. This will keep the unified PTEs while the folio isn't reclaimed. The result is quite similar with small folios with one PTE - either entirely reclaimed or entirely kept.
Reclaiming large folios by holding PTL from PTE0 seems a better option comparing to giving up after detecting PTL begins from non-PTE0.

---
 mm/vmscan.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/vmscan.c +11 -0
index 0b888a2afa58..e4722fbbcd0c 100644
@@ -1270,6 +1270,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
                      if (folio_test_pmd_mappable(folio))
                              flags |= TTU_SPLIT_HUGE_PMD;
+                     /*
+                      * if page table lock is not held from the first PTE of
+                      * a large folio, some PTEs might be skipped because of
+                      * races with break-before-make, for example, PTEs can
+                      * be pte_none intermediately, thus one or more PTEs
+                      * might be skipped in try_to_unmap_one, we might result
+                      * in a large folio is partially mapped and partially
+                      * unmapped after try_to_unmap
+                      */
+                     if (folio_test_large(folio))
+                             flags |= TTU_SYNC;
 
Quoted 2 lines

mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1286

Quoted 2 lines
mm/vmscan.c +0 -0
                      try_to_unmap(folio, flags);
                      if (folio_mapped(folio)) {

Thanks
Barry

#26 lore raw
In reply to Barry Song
Quoted 1 line · Barry Song

are writing pte to zero(break) before writing a new value(make). while

As an aside, "break-before-make" as defined in the Arm architecture would also require a TLBI, which usually isn't done for these write-0-modify-prots-write-back operations. Arm doesn't require "break-before-make" in these situations so its legal (as long as only certain bits are changed). To my understanding purpose of doing this is to avoid races with HW access/dirty flag updates; if the MMU wants to set either flag and finds the PTE is 0 (invalid) it will cause an exception which will be queued waiting for the PTL.

So I don't think you really mean break-before-make here.

Quoted 10 lines

this behavior is within PTL in another thread, page_vma_mapped_walk() of try_to_unmap_one thread won't take PTL till it meets a present PTE. for example, if another threads are modifying nr_pages PTEs under PTL, but we don't hold PTL, we might skip one or two PTEs at the beginning of a large folio.
For a large folio, after try_to_unmap_one(), we may result in PTE0 and PTE1 untouched but PTE2~nr_pages-1 are set to swap entries.

by holding PTL from PTE0 for large folios, we won't get these intermediate values. At the moment we get PTL, other threads have done.

author
#27 lore raw
Quoted 13 lines · Ryan Roberts
Quoted 1 line · Barry Song

are writing pte to zero(break) before writing a new value(make). while

As an aside, "break-before-make" as defined in the Arm architecture would also require a TLBI, which usually isn't done for these write-0-modify-prots-write-back operations. Arm doesn't require "break-before-make" in these situations so its legal (as long as only certain bits are changed). To my understanding purpose of doing this is to avoid races with HW access/dirty flag updates; if the MMU wants to set either flag and finds the PTE is 0 (invalid) it will cause an exception which will be queued waiting for the PTL.

Quoted 1 line

So I don't think you really mean break-before-make here.

I agree I use a stronger term. will change it to something lighter in v2.

Quoted 9 lines
Quoted 9 lines

this behavior is within PTL in another thread, page_vma_mapped_walk() of try_to_unmap_one thread won't take PTL till it meets a present PTE. for example, if another threads are modifying nr_pages PTEs under PTL, but we don't hold PTL, we might skip one or two PTEs at the beginning of a large folio.
For a large folio, after try_to_unmap_one(), we may result in PTE0 and PTE1 untouched but PTE2~nr_pages-1 are set to swap entries.

Quoted 3 lines
Quoted 2 lines

by holding PTL from PTE0 for large folios, we won't get these intermediate values. At the moment we get PTL, other threads have done.

Thanks
Barry