Skip to contentSkip to conversation

x86/mm: Fix pmd_modify() dropping the dirty bit

PATCH Applied Vernon Yang 8 participants 17 messages Sep 3 – Sep 9

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/mm: Fix pmd_modify() dropping the dirty bit

From: Vernon Yang <[email protected]>

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

---
 arch/x86/include/asm/pgtable.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
index d5f4917c1edc..d551120a7c88 100644
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
 	pmdval_t val = pmd_val(pmd), oldval = val;
 	pmd_t pmd_result;
 
-	val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
+	val &= _HPAGE_CHG_MASK;
 	val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK;
 	val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);
 
Signature · 1 line
2.53.0
maintainer
#1 lore raw
Quoted 30 lines · On Thu, 3 Sep 2026 11:16:08 +0800 Vernon Yang <[email protected]> wrote:

From: Vernon Yang <[email protected]>

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

Quoted 8 lines

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

Yeah, this is exactly what I came up with, using chatgpt.

Quoted 1 line

We definitely want a Reported-by: Orson here. He obviously did a ton of work on this, and it's the least we can do to thank him. I'll add it.

Quoted 3 lines
Fixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")
#2 lore raw
Quoted 41 lines · Andrew Morton
Quoted 30 lines · On Thu, 3 Sep 2026 11:16:08 +0800 Vernon Yang <[email protected]> wrote:

From: Vernon Yang <[email protected]>

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

Quoted 8 lines

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

Quoted 6 lines

Yeah, this is exactly what I came up with, using chatgpt.

Quoted 1 line

We definitely want a Reported-by: Orson here. He obviously did a ton of work on this, and it's the least we can do to thank him. I'll add it.

Yeah, I missed adding Reported-by: Orson, Sorry. Thank you for adding it.

--
Cheers,
Vernon

#3 lore raw
In reply to Andrew Morton

Dear Andrew, Vernon,

Thanks for your quick reply, confirmation and patch.

What I failed to mention in the original report is that this isn't just a hypothetical issue found by analyzing the code, some of our users (of Polars, a data analytics library) have hit this in production. "Live data can change silently" is never a fun issue report to get.

Transparent huge pages + MADV_FREE + NUMA hinting + cgroup limits isn't your typical mom-and-pop setup, but it is not an unreasonable configuration either in high-performance computing. The only missing ingredient when you use Polars on a multi-NUMA-region machine running default Ubuntu configuration is the cgroup limits.

Considering that the impact is so severe if the bug does occur (silent memory corruption), will there be backports of this patch? If I'm not mistaken the offending line that introduced the bug was first released in version 6.6.

Best,

Orson

Quoted 53 lines · Andrew Morton
Quoted 30 lines · On Thu, 3 Sep 2026 11:16:08 +0800 Vernon Yang <[email protected]> wrote:

From: Vernon Yang <[email protected]>

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

Quoted 8 lines

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

Yeah, this is exactly what I came up with, using chatgpt.

Quoted 1 line

We definitely want a Reported-by: Orson here. He obviously did a ton of work on this, and it's the least we can do to thank him. I'll add it.

Quoted 3 lines
Fixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")
#4 lore raw
Quoted 4 lines · Orson Peters

Considering that the impact is so severe if the bug does occur (silent memory corruption), will there be backports of this patch? If I'm not mistaken the offending line that introduced the bug was first released in version 6.6.

Hi Orson,

It's tagged for stable@ so it should hit the stable trees at some point and make it into the 6.6 series.

#5 lore raw
In reply to Vernon Yang
Quoted 33 lines · Vernon Yang

From: Vernon Yang <[email protected]>

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Quoted 8 lines

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

Fixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")

Hm. I don't understand why would this commit explicitly exclude _PAGE_DIRTY from the mask:

- val &= _HPAGE_CHG_MASK;
+ val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);

Rick, could you comment? It doesn't look like a typo.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#6 lore raw
Quoted 9 lines · Kiryl Shutsemau
Quoted 2 lines
Fixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")

Hm. I don't understand why would this commit explicitly exclude _PAGE_DIRTY from the mask:

(quoted diff) +1 -1
-       val &= _HPAGE_CHG_MASK;
+       val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);

Rick, could you comment? It doesn't look like a typo.

My guess is that it's some remnant from an earlier version of the patch. The asymmetry with pte_modify() vs. pmd_modify() just can't be explained any other way. It _might_ have been some attempt to say, "Hey _PAGE_DIRTY is now a part of the pgprot_t since it's part of the 'permissions' of a shadow stack PTE" that got abandoned.

But I don't see anything wrong with the fix at all.

It does give me pause that this has been losing user data for so long, but it must just be a weird combination of features that few folks use together (huge pages + MADV_FREE).

maintainer
#7 lore raw
Quoted 17 lines · On Thu, 3 Sep 2026 07:18:54 -0700 Dave Hansen <[email protected]> wrote:
Quoted 9 lines · Kiryl Shutsemau
Quoted 2 lines
Fixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")

Hm. I don't understand why would this commit explicitly exclude _PAGE_DIRTY from the mask:

(quoted diff) +1 -1
-       val &= _HPAGE_CHG_MASK;
+       val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);

Rick, could you comment? It doesn't look like a typo.

My guess is that it's some remnant from an earlier version of the patch. The asymmetry with pte_modify() vs. pmd_modify() just can't be explained any other way. It _might_ have been some attempt to say, "Hey _PAGE_DIRTY is now a part of the pgprot_t since it's part of the 'permissions' of a shadow stack PTE" that got abandoned.

Quoted 6 lines

But I don't see anything wrong with the fix at all.

It does give me pause that this has been losing user data for so long, but it must just be a weird combination of features that few folks use together (huge pages + MADV_FREE).

And with heavy reclaim pressure.

#8 lore raw
In reply to Dave Hansen
Quoted 17 lines · Dave Hansen
Quoted 9 lines · Kiryl Shutsemau
Quoted 2 lines
Fixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")

Hm. I don't understand why would this commit explicitly exclude _PAGE_DIRTY from the mask:

-       val &= _HPAGE_CHG_MASK;
+       val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);

Rick, could you comment? It doesn't look like a typo.

My guess is that it's some remnant from an earlier version of the patch. The asymmetry with pte_modify() vs. pmd_modify() just can't be explained any other way. It _might_ have been some attempt to say, "Hey _PAGE_DIRTY is now a part of the pgprot_t since it's part of the 'permissions' of a shadow stack PTE" that got abandoned.

Quoted 5 lines

But I don't see anything wrong with the fix at all.

It does give me pause that this has been losing user data for so long, but it must just be a weird combination of features that few folks use together (huge pages + MADV_FREE).

Oof. Looking back through the patch history, the dirty bit used to be handled separately, such that the stripping was needed. Like this:

static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)

 {
	pteval_t _page_chg_mask_no_dirty = _PAGE_CHG_MASK & ~_PAGE_DIRTY;
 	pteval_t val = pte_val(pte), oldval = val;
	pte_t pte_result;

/*
* Chop off the NX bit (if present), and add the NX portion of
* the newprot (if present):
*/

	val &= _page_chg_mask_no_dirty;
	val |= check_pgprot(newprot) & ~_page_chg_mask_no_dirty;

val = flip_protnone_guard(oldval, val, PTE_PFN_MASK);

	pte_result = __pte(val);

	/*
	 * Dirty bit is not preserved above so it can be done
	 * in a special way for the shadow stack case, where it
	 * may need to set _PAGE_COW. __pte_mkdirty() will do this in
	 * the case of shadow stack.
	 */
	if (pte_dirty(pte))
		pte_result = __pte_mkdirty(pte_result, false);

	return pte_result;

}

static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)

 {
	pteval_t _hpage_chg_mask_no_dirty = _HPAGE_CHG_MASK & ~_PAGE_DIRTY;
 	pmdval_t val = pmd_val(pmd), oldval = val;
	pmd_t pmd_result;

	val &= _hpage_chg_mask_no_dirty;
	val |= check_pgprot(newprot) & ~_hpage_chg_mask_no_dirty;

val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);

	pmd_result = __pmd(val);

	/*
	 * Dirty bit is not preserved above so it can be done
	 * in a special way for the shadow stack case, where it
	 * may need to set _PAGE_COW. __pmd_mkdirty() will do this in
	 * the case of shadow stack.
	 */
	if (pmd_dirty(pmd))
		pmd_result = __pmd_mkdirty(pmd_result, false);

	return pmd_result;

}

The dirty bit was removed from the pte, then handled separately to share logic in the mkdirty helpers. During development it was changed to do the necessary adjustments depending on the dirty bit in 'val', but the mask adjustment on the pmd_modify() side didn't get updated. So I don't remember or see any intention for the difference.

I can't find anything back then that would have prevented it. The bug cause and fix looks correct to me.

#9 lore raw
Quoted 2 lines · Rick P

I can't find anything back then that would have prevented it. The bug cause and fix looks correct to me.

Is that a Reviewed-by? ;)

merge bot Applied
#11 lore raw

[tip: x86/urgent] x86/mm: Fix userspace data loss with MADV_FREE and THP

In reply to Vernon Yang
The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     4846278477f09d1f3b5f03091393b35a92db0b52
Gitweb:        https://git.kernel.org/tip/4846278477f09d1f3b5f03091393b35a92db0b52
Author:        Vernon Yang <[email protected]>
AuthorDate:    Thu, 03 Sep 2026 11:16:08 +08:00
Committer:     Dave Hansen <[email protected]>
CommitterDate: Thu, 03 Sep 2026 12:15:11 -07:00

x86/mm: Fix userspace data loss with MADV_FREE and THP

Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure.

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

---
 arch/x86/include/asm/pgtable.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
index d5f4917..d551120 100644
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
 	pmdval_t val = pmd_val(pmd), oldval = val;
 	pmd_t pmd_result;
 
-	val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
+	val &= _HPAGE_CHG_MASK;
 	val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK;
 	val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);
 
Reviewed-by Tested-by on v1 1/1
#12 lore raw
In reply to Vernon Yang
Quoted 39 lines · Vernon Yang

From: Vernon Yang <[email protected]>

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

Quoted 4 lines

FYI we just had a customer case for our downstream kernel for this exact same issue. This fixed it beautifully (I'm wondering if polars started doing something interesting lately, that nobody else does, and that's why...).

Signature · 1 line
Pedro
merge bot Applied
#13 lore raw

[tip: x86/urgent] x86/mm: Fix user-space data loss with MADV_FREE and THP

In reply to Vernon Yang
The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     c70787db5b6a661940f5a7bedcdbc7daefe3e401
Gitweb:        https://git.kernel.org/tip/c70787db5b6a661940f5a7bedcdbc7daefe3e401
Author:        Vernon Yang <[email protected]>
AuthorDate:    Thu, 03 Sep 2026 11:16:08 +08:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 08 Sep 2026 08:53:15 +02:00

x86/mm: Fix user-space data loss with MADV_FREE and THP

Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure.

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

---
 arch/x86/include/asm/pgtable.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
index d5f4917..d551120 100644
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
 	pmdval_t val = pmd_val(pmd), oldval = val;
 	pmd_t pmd_result;
 
-	val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
+	val &= _HPAGE_CHG_MASK;
 	val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK;
 	val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);
 
merge bot Applied
#14 lore raw

[tip: x86/mm] x86/mm: Fix user-space data loss with MADV_FREE and THP

In reply to Vernon Yang
The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     18d6b3df5aa910b8e3bad889f2658e6a4c8f6a0d
Gitweb:        https://git.kernel.org/tip/18d6b3df5aa910b8e3bad889f2658e6a4c8f6a0d
Author:        Vernon Yang <[email protected]>
AuthorDate:    Thu, 03 Sep 2026 11:16:08 +08:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 08 Sep 2026 09:15:12 +02:00

x86/mm: Fix user-space data loss with MADV_FREE and THP

Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure.

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

---
 arch/x86/include/asm/pgtable.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
index d5f4917..d551120 100644
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
 	pmdval_t val = pmd_val(pmd), oldval = val;
 	pmd_t pmd_result;
 
-	val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
+	val &= _HPAGE_CHG_MASK;
 	val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK;
 	val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);
 
merge bot Applied
#15 lore raw

[tip: x86/urgent] x86/mm: Fix user-space data loss with MADV_FREE and THP

In reply to Vernon Yang
The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     308e2bc308bcde06e56da6e9bf316b72217c3c24
Gitweb:        https://git.kernel.org/tip/308e2bc308bcde06e56da6e9bf316b72217c3c24
Author:        Vernon Yang <[email protected]>
AuthorDate:    Thu, 03 Sep 2026 11:16:08 +08:00
Committer:     Dave Hansen <[email protected]>
CommitterDate: Tue, 08 Sep 2026 15:43:07 -07:00

x86/mm: Fix user-space data loss with MADV_FREE and THP

Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure.

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

---
 arch/x86/include/asm/pgtable.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
index d5f4917..d551120 100644
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
 	pmdval_t val = pmd_val(pmd), oldval = val;
 	pmd_t pmd_result;
 
-	val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
+	val &= _HPAGE_CHG_MASK;
 	val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK;
 	val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);
 
merge bot Applied
#16 lore raw

[tip: x86/urgent] x86/mm: Fix user-space data loss with MADV_FREE and THP

In reply to Vernon Yang
The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     f7491d7c81db0e7c304a7bd757a76d2fbeaff80e
Gitweb:        https://git.kernel.org/tip/f7491d7c81db0e7c304a7bd757a76d2fbeaff80e
Author:        Vernon Yang <[email protected]>
AuthorDate:    Thu, 03 Sep 2026 11:16:08 +08:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Wed, 09 Sep 2026 08:39:12 +02:00

x86/mm: Fix user-space data loss with MADV_FREE and THP

Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure.

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.

---
 arch/x86/include/asm/pgtable.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
index d5f4917..d551120 100644
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
 	pmdval_t val = pmd_val(pmd), oldval = val;
 	pmd_t pmd_result;
 
-	val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
+	val &= _HPAGE_CHG_MASK;
 	val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK;
 	val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);