mm/rmap: fix soft-dirty bit loss when remapping zero-filled mTHP subpage to shared zeropage

From: Lance Yang <[email protected]>

When splitting an mTHP and replacing a zero-filled subpage with the shared zeropage, try_to_map_unused_to_zeropage() currently drops the soft-dirty bit.

For userspace tools like CRIU, which rely on the soft-dirty mechanism for incremental snapshots, losing this bit means modified pages are missed, leading to inconsistent memory state after restore.

Preserve the soft-dirty bit from the old PTE when creating the zeropage mapping to ensure modified pages are correctly tracked.

Cc <[email protected]>

Fixes b1f202060afe ("mm: remap unused subpages to shared zeropage when splitting isolated thp")

Signed-off-by Lance Yang <[email protected]>

---
  mm/migrate.c | 4 ++++
  1 file changed, 4 insertions(+)

mm/migrate.c +0 -0

index ce83c2c3c287..bf364ba07a3f 100644

Quoted 25 lines · Lance Yang

From: Lance Yang <[email protected]>

When splitting an mTHP and replacing a zero-filled subpage with the shared zeropage, try_to_map_unused_to_zeropage() currently drops the soft-dirty bit.

For userspace tools like CRIU, which rely on the soft-dirty mechanism for incremental snapshots, losing this bit means modified pages are missed, leading to inconsistent memory state after restore.

Preserve the soft-dirty bit from the old PTE when creating the zeropage mapping to ensure modified pages are correctly tracked.

Cc <[email protected]>

Fixes b1f202060afe ("mm: remap unused subpages to shared zeropage when splitting isolated thp")

Signed-off-by Lance Yang <[email protected]>

---
  mm/migrate.c | 4 ++++

  1 file changed, 4 insertions(+)

mm/migrate.c +0 -0

index ce83c2c3c287..bf364ba07a3f 100644

mm/migrate.c static bool try_to_map_unused_to_zeropage(struct · line 330

Quoted 9 lines

mm/migrate.c +4 -0

@@ -322,6 +322,10 @@ static bool try_to_map_unused_to_zeropage(struct 
 page_vma_mapped_walk *pvmw,
       newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
                       pvmw->vma->vm_page_prot));
+
+    if (pte_swp_soft_dirty(ptep_get(pvmw->pte)))
+        newpte = pte_mksoft_dirty(newpte);
+
       set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte);
       dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));

I think this should work.

You can pass old_pte = ptep_get(pvmw->pte) to this function to avoid calling ptep_get() multiple times.

From: Lance Yang <[email protected]>

When splitting an mTHP and replacing a zero-filled subpage with the shared zeropage, try_to_map_unused_to_zeropage() currently drops the soft-dirty bit.

For userspace tools like CRIU, which rely on the soft-dirty mechanism for incremental snapshots, losing this bit means modified pages are missed, leading to inconsistent memory state after restore.

Preserve the soft-dirty bit from the old PTE when creating the zeropage mapping to ensure modified pages are correctly tracked.

Cc <[email protected]>

Fixes b1f202060afe ("mm: remap unused subpages to shared zeropage when splitting isolated thp")

Signed-off-by Lance Yang <[email protected]>

---
  mm/migrate.c | 4 ++++
  1 file changed, 4 insertions(+)

mm/migrate.c +0 -0

index ce83c2c3c287..bf364ba07a3f 100644

Quoted 25 lines · Lance Yang

From: Lance Yang <[email protected]>

When splitting an mTHP and replacing a zero-filled subpage with the shared zeropage, try_to_map_unused_to_zeropage() currently drops the soft-dirty bit.

For userspace tools like CRIU, which rely on the soft-dirty mechanism for incremental snapshots, losing this bit means modified pages are missed, leading to inconsistent memory state after restore.

Preserve the soft-dirty bit from the old PTE when creating the zeropage mapping to ensure modified pages are correctly tracked.

Cc <[email protected]>

Fixes b1f202060afe ("mm: remap unused subpages to shared zeropage when splitting isolated thp")

Signed-off-by Lance Yang <[email protected]>

---
  mm/migrate.c | 4 ++++

  1 file changed, 4 insertions(+)

mm/migrate.c +0 -0

index ce83c2c3c287..bf364ba07a3f 100644

mm/migrate.c static bool try_to_map_unused_to_zeropage(struct · line 330

Quoted 9 lines

mm/migrate.c +4 -0

@@ -322,6 +322,10 @@ static bool try_to_map_unused_to_zeropage(struct 
 page_vma_mapped_walk *pvmw,
       newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
                       pvmw->vma->vm_page_prot));
+
+    if (pte_swp_soft_dirty(ptep_get(pvmw->pte)))
+        newpte = pte_mksoft_dirty(newpte);
+
       set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte);
       dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));

It's interesting that there isn't a single occurrence of the stof-dirty flag in khugepaged code. I guess it all works because we do the

    _pmd = maybe_pmd_mkwrite(pmd_mkdirty(_pmd), vma);

and the pmd_mkdirty() will imply marking it soft-dirty.

Now to the problem at hand: I don't think this is particularly problematic in the common case: if the page is zero, it likely was never written to (that's what the unerused shrinker is targeted at), so the soft-dirty setting on the PMD is actually just an over-indication for this page.

For example, when we just install the shared zeropage directly in do_anonymous_page(), we obviously also don't set it dirty/soft-dirty.

Now, one could argue that if the content was changed from non-zero to zero, it ould actually be soft-dirty.

Long-story short: I don't think this matters much in practice, but it's an easy fix.

As said by dev, please avoid double ptep_get() if possible.

Acked-by David Hildenbrand <[email protected]>

@Lance, can you double-check that the uffd-wp bit is handled correctly? I strongly assume we lose that as well here.

Quoted 38 lines · David Hildenbrand

Quoted 25 lines · Lance Yang

From: Lance Yang <[email protected]>

When splitting an mTHP and replacing a zero-filled subpage with the shared zeropage, try_to_map_unused_to_zeropage() currently drops the soft-dirty bit.

For userspace tools like CRIU, which rely on the soft-dirty mechanism for incremental snapshots, losing this bit means modified pages are missed, leading to inconsistent memory state after restore.

Preserve the soft-dirty bit from the old PTE when creating the zeropage mapping to ensure modified pages are correctly tracked.

Cc <[email protected]>

Fixes b1f202060afe ("mm: remap unused subpages to shared zeropage when splitting isolated thp")

Signed-off-by Lance Yang <[email protected]>

---
  mm/migrate.c | 4 ++++

  1 file changed, 4 insertions(+)

mm/migrate.c +0 -0

index ce83c2c3c287..bf364ba07a3f 100644

mm/migrate.c static bool try_to_map_unused_to_zeropage(struct · line 330

Quoted 9 lines

mm/migrate.c +4 -0

@@ -322,6 +322,10 @@ static bool try_to_map_unused_to_zeropage(struct 
 page_vma_mapped_walk *pvmw,
       newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
                       pvmw->vma->vm_page_prot));
+
+    if (pte_swp_soft_dirty(ptep_get(pvmw->pte)))
+        newpte = pte_mksoft_dirty(newpte);
+
       set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte);
       dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));

It's interesting that there isn't a single occurrence of the stof- dirty flag in khugepaged code. I guess it all works because we do the

Quoted 5 lines

     _pmd = maybe_pmd_mkwrite(pmd_mkdirty(_pmd), vma);

and the pmd_mkdirty() will imply marking it soft-dirty.

Now to the problem at hand: I don't think this is particularly problematic in the common case: if the page is zero, it likely was never written to (that's what the unerused shrinker is targeted at), so the soft-dirty setting on the PMD is actually just an over- indication for this page.

Cool. Thanks for the insight! Good to know that ;)

Quoted 4 lines

For example, when we just install the shared zeropage directly in do_anonymous_page(), we obviously also don't set it dirty/soft-dirty.

Now, one could argue that if the content was changed from non-zero to zero, it ould actually be soft-dirty.

Exactly. A false negative could be a problem for the userspace tools, IMO.

Quoted 3 lines

Long-story short: I don't think this matters much in practice, but it's an easy fix.

Quoted 1 line

As said by dev, please avoid double ptep_get() if possible.

Sure, will do. I'll refactor it in the next version.

Quoted 2 lines

Acked-by David Hildenbrand <[email protected]>

Thanks!

Quoted 4 lines

@Lance, can you double-check that the uffd-wp bit is handled correctly? I strongly assume we lose that as well here.

Certainly, I'll check the uffd-wp bit as well and get back to you soon.

Cheers,
Lance

Quoted 70 lines · Lance Yang

Quoted 44 lines · David Hildenbrand

Quoted 28 lines · Lance Yang

From: Lance Yang <[email protected]>

When splitting an mTHP and replacing a zero-filled subpage with the shared
zeropage, try_to_map_unused_to_zeropage() currently drops the soft-dirty bit.

For userspace tools like CRIU, which rely on the soft-dirty mechanism for
incremental snapshots, losing this bit means modified pages are missed, leading to inconsistent memory state after restore.

Preserve the soft-dirty bit from the old PTE when creating the zeropage mapping to ensure modified pages are correctly tracked.

Cc <[email protected]>

Fixes b1f202060afe ("mm: remap unused subpages to shared zeropage

when splitting isolated thp")

Signed-off-by Lance Yang <[email protected]>

---
   mm/migrate.c | 4 ++++
   1 file changed, 4 insertions(+)

mm/migrate.c +0 -0

index ce83c2c3c287..bf364ba07a3f 100644

mm/migrate.c static bool try_to_map_unused_to_zeropage(struct · line 330

Quoted 9 lines

mm/migrate.c +4 -0

@@ -322,6 +322,10 @@ static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
       newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
                       pvmw->vma->vm_page_prot));
+
+    if (pte_swp_soft_dirty(ptep_get(pvmw->pte)))
+        newpte = pte_mksoft_dirty(newpte);
+
       set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte);
       dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));

It's interesting that there isn't a single occurrence of the stof- dirty flag in khugepaged code. I guess it all works because we do the

     _pmd = maybe_pmd_mkwrite(pmd_mkdirty(_pmd), vma);

Quoted 7 lines

and the pmd_mkdirty() will imply marking it soft-dirty.

Now to the problem at hand: I don't think this is particularly problematic in the common case: if the page is zero, it likely was never written to (that's what the unerused shrinker is targeted at), so the soft-dirty setting on the PMD is actually just an over- indication for this page.

Cool. Thanks for the insight! Good to know that ;)

Quoted 6 lines

For example, when we just install the shared zeropage directly in do_anonymous_page(), we obviously also don't set it dirty/soft-dirty.

Now, one could argue that if the content was changed from non-zero to zero, it ould actually be soft-dirty.

Exactly. A false negative could be a problem for the userspace tools, IMO.

Quoted 4 lines

Long-story short: I don't think this matters much in practice, but it's an easy fix.

Quoted 13 lines

Quoted 1 line

As said by dev, please avoid double ptep_get() if possible.

Sure, will do. I'll refactor it in the next version.

Quoted 2 lines

Acked-by David Hildenbrand <[email protected]>

Thanks!

Quoted 4 lines

@Lance, can you double-check that the uffd-wp bit is handled correctly? I strongly assume we lose that as well here.

Yes, the uffd-wp bit was indeed being dropped, but ...

The shared zeropage is read-only, which triggers a fault. IIUC, The kernel then falls back to checking the VM_UFFD_WP flag on the VMA and correctly generates a uffd-wp event, masking the fact that the uffd-wp bit on the PTE was lost.

Quoted 87 lines · Lance Yang

Quoted 83 lines · Lance Yang

Quoted 44 lines · David Hildenbrand

Quoted 28 lines · Lance Yang

From: Lance Yang <[email protected]>

When splitting an mTHP and replacing a zero-filled subpage with the shared
zeropage, try_to_map_unused_to_zeropage() currently drops the soft- dirty bit.

For userspace tools like CRIU, which rely on the soft-dirty mechanism for
incremental snapshots, losing this bit means modified pages are missed, leading to inconsistent memory state after restore.

Preserve the soft-dirty bit from the old PTE when creating the zeropage mapping to ensure modified pages are correctly tracked.

Cc <[email protected]>

Fixes b1f202060afe ("mm: remap unused subpages to shared zeropage

when splitting isolated thp")

Signed-off-by Lance Yang <[email protected]>

---
   mm/migrate.c | 4 ++++

   1 file changed, 4 insertions(+)

mm/migrate.c +0 -0

index ce83c2c3c287..bf364ba07a3f 100644

mm/migrate.c static bool try_to_map_unused_to_zeropage(struct · line 330

Quoted 9 lines

mm/migrate.c +4 -0

@@ -322,6 +322,10 @@ static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
        newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
                        pvmw->vma->vm_page_prot));
+
+    if (pte_swp_soft_dirty(ptep_get(pvmw->pte)))
+        newpte = pte_mksoft_dirty(newpte);
+
        set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte);
        dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));

It's interesting that there isn't a single occurrence of the stof- dirty flag in khugepaged code. I guess it all works because we do the

      _pmd = maybe_pmd_mkwrite(pmd_mkdirty(_pmd), vma);

Quoted 7 lines

and the pmd_mkdirty() will imply marking it soft-dirty.

Now to the problem at hand: I don't think this is particularly problematic in the common case: if the page is zero, it likely was never written to (that's what the unerused shrinker is targeted at), so the soft-dirty setting on the PMD is actually just an over- indication for this page.

Cool. Thanks for the insight! Good to know that ;)

Quoted 6 lines

For example, when we just install the shared zeropage directly in do_anonymous_page(), we obviously also don't set it dirty/soft-dirty.

Now, one could argue that if the content was changed from non-zero to zero, it ould actually be soft-dirty.

Exactly. A false negative could be a problem for the userspace tools, IMO.

Quoted 5 lines

Long-story short: I don't think this matters much in practice, but it's an easy fix.

As said by dev, please avoid double ptep_get() if possible.

Sure, will do. I'll refactor it in the next version.

Quoted 2 lines

Acked-by David Hildenbrand <[email protected]>

Thanks!

Quoted 4 lines

@Lance, can you double-check that the uffd-wp bit is handled correctly? I strongly assume we lose that as well here.

Quoted 6 lines

Yes, the uffd-wp bit was indeed being dropped, but ...

The shared zeropage is read-only, which triggers a fault. IIUC, The kernel then falls back to checking the VM_UFFD_WP flag on the VMA and correctly generates a uffd-wp event, masking the fact that the uffd-wp bit on the PTE was lost.

That's not how VM_UFFD_WP works :)

Quoted 92 lines · David Hildenbrand

Quoted 91 lines · Lance Yang

Quoted 87 lines · Lance Yang

Quoted 47 lines · David Hildenbrand

Quoted 31 lines · Lance Yang

From: Lance Yang <[email protected]>

When splitting an mTHP and replacing a zero-filled subpage with the shared
zeropage, try_to_map_unused_to_zeropage() currently drops the soft- dirty
bit.

For userspace tools like CRIU, which rely on the soft-dirty mechanism for
incremental snapshots, losing this bit means modified pages are missed,
leading to inconsistent memory state after restore.

Preserve the soft-dirty bit from the old PTE when creating the zeropage
mapping to ensure modified pages are correctly tracked.

Cc <[email protected]>

Fixes b1f202060afe ("mm: remap unused subpages to shared zeropage

when splitting isolated thp")

Signed-off-by Lance Yang <[email protected]>

---
    mm/migrate.c | 4 ++++
    1 file changed, 4 insertions(+)

mm/migrate.c +0 -0

index ce83c2c3c287..bf364ba07a3f 100644

mm/migrate.c static bool try_to_map_unused_to_zeropage(struct · line 330

Quoted 9 lines

mm/migrate.c +4 -0

@@ -322,6 +322,10 @@ static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
        newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
                        pvmw->vma->vm_page_prot));
+
+    if (pte_swp_soft_dirty(ptep_get(pvmw->pte)))
+        newpte = pte_mksoft_dirty(newpte);
+
        set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte);
        dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));

It's interesting that there isn't a single occurrence of the stof- dirty flag in khugepaged code. I guess it all works because we do the

      _pmd = maybe_pmd_mkwrite(pmd_mkdirty(_pmd), vma);

Quoted 7 lines

and the pmd_mkdirty() will imply marking it soft-dirty.

Now to the problem at hand: I don't think this is particularly problematic in the common case: if the page is zero, it likely was never written to (that's what the unerused shrinker is targeted at), so the soft-dirty setting on the PMD is actually just an over- indication for this page.

Cool. Thanks for the insight! Good to know that ;)

Quoted 6 lines

For example, when we just install the shared zeropage directly in do_anonymous_page(), we obviously also don't set it dirty/soft-dirty.

Now, one could argue that if the content was changed from non-zero to zero, it ould actually be soft-dirty.

Exactly. A false negative could be a problem for the userspace tools, IMO.

Quoted 5 lines

Long-story short: I don't think this matters much in practice, but it's an easy fix.

As said by dev, please avoid double ptep_get() if possible.

Sure, will do. I'll refactor it in the next version.

Quoted 2 lines

Acked-by David Hildenbrand <[email protected]>

Thanks!

Quoted 4 lines

@Lance, can you double-check that the uffd-wp bit is handled correctly? I strongly assume we lose that as well here.

Quoted 8 lines

Quoted 6 lines

Yes, the uffd-wp bit was indeed being dropped, but ...

The shared zeropage is read-only, which triggers a fault. IIUC, The kernel then falls back to checking the VM_UFFD_WP flag on the VMA and correctly generates a uffd-wp event, masking the fact that the uffd-wp bit on the PTE was lost.

That's not how VM_UFFD_WP works :)

My bad! Please accept my apologies for the earlier confusion :(

I messed up my test environment (forgot to enable mTHP), which led me to a completely wrong conclusion...

You're spot on. With mTHP enabled, the WP fault was not caught on the shared zeropage after it replaced a zero-filled subpage during an mTHP split.

This is because do_wp_page() requires userfaultfd_pte_wp() to be true, which in turn needs both userfaultfd_wp(vma) and pte_uffd_wp(pte).

static inline bool userfaultfd_pte_wp(struct vm_area_struct *vma,
pte_t pte)

{
	return userfaultfd_wp(vma) && pte_uffd_wp(pte);
}

userfaultfd_pte_wp() fails as we lose the uffd-wp bit on the PTE ...

Quoted 124 lines · Lance Yang

Quoted 93 lines · David Hildenbrand

Quoted 92 lines · Lance Yang

Quoted 88 lines · Lance Yang

Quoted 48 lines · David Hildenbrand

Quoted 39 lines · Lance Yang

From: Lance Yang <[email protected]>

When splitting an mTHP and replacing a zero-filled subpage with the shared
zeropage, try_to_map_unused_to_zeropage() currently drops the soft- dirty
bit.

For userspace tools like CRIU, which rely on the soft-dirty mechanism for
incremental snapshots, losing this bit means modified pages are missed,
leading to inconsistent memory state after restore.

Preserve the soft-dirty bit from the old PTE when creating the zeropage
mapping to ensure modified pages are correctly tracked.

Cc <[email protected]>

Fixes b1f202060afe ("mm: remap unused subpages to shared zeropage

when splitting isolated thp")

Signed-off-by Lance Yang <[email protected]>

---
    mm/migrate.c | 4 ++++

    1 file changed, 4 insertions(+)

mm/migrate.c +0 -0

index ce83c2c3c287..bf364ba07a3f 100644
@@ -322,6 +322,10 @@ static bool 

try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
        newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address), pvmw->vma->vm_page_prot));

mm/migrate.c +4 -0

+
+    if (pte_swp_soft_dirty(ptep_get(pvmw->pte)))
+        newpte = pte_mksoft_dirty(newpte);
+

Quoted 2 lines

        set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte); dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));

It's interesting that there isn't a single occurrence of the stof- dirty flag in khugepaged code. I guess it all works because we do the

       _pmd = maybe_pmd_mkwrite(pmd_mkdirty(_pmd), vma);

Quoted 7 lines

and the pmd_mkdirty() will imply marking it soft-dirty.

Now to the problem at hand: I don't think this is particularly problematic in the common case: if the page is zero, it likely was never written to (that's what the unerused shrinker is targeted at), so the soft-dirty setting on the PMD is actually just an over- indication for this page.

Cool. Thanks for the insight! Good to know that ;)

Quoted 6 lines

For example, when we just install the shared zeropage directly in do_anonymous_page(), we obviously also don't set it dirty/soft-dirty.

Now, one could argue that if the content was changed from non-zero to zero, it ould actually be soft-dirty.

Exactly. A false negative could be a problem for the userspace tools, IMO.

Quoted 5 lines

Long-story short: I don't think this matters much in practice, but it's an easy fix.

As said by dev, please avoid double ptep_get() if possible.

Sure, will do. I'll refactor it in the next version.

Quoted 2 lines

Acked-by David Hildenbrand <[email protected]>

Thanks!

Quoted 4 lines

@Lance, can you double-check that the uffd-wp bit is handled correctly? I strongly assume we lose that as well here.

Quoted 8 lines

Quoted 6 lines

Yes, the uffd-wp bit was indeed being dropped, but ...

The shared zeropage is read-only, which triggers a fault. IIUC, The kernel then falls back to checking the VM_UFFD_WP flag on the VMA and correctly generates a uffd-wp event, masking the fact that the uffd-wp bit on the PTE was lost.

That's not how VM_UFFD_WP works :)

My bad! Please accept my apologies for the earlier confusion :(

I messed up my test environment (forgot to enable mTHP), which led me to a completely wrong conclusion...

You're spot on. With mTHP enabled, the WP fault was not caught on the shared zeropage after it replaced a zero-filled subpage during an mTHP split.

This is because do_wp_page() requires userfaultfd_pte_wp() to be true, which in turn needs both userfaultfd_wp(vma) and pte_uffd_wp(pte).

static inline bool userfaultfd_pte_wp(struct vm_area_struct *vma, pte_t pte)
{
    return userfaultfd_wp(vma) && pte_uffd_wp(pte);
}

Quoted 1 line

userfaultfd_pte_wp() fails as we lose the uffd-wp bit on the PTE ...

That's my understanding. And FWIW, that's a much more important fix. (in contrast to soft-dirty, uffd-wp actually is precise)

Can you test+send a fix ... please? :)