x86/mm: Fix pmd_modify() dropping the dirty bit
Subsystems (3)
PATCH 1/1 x86/mm: Fix pmd_modify() dropping the dirty bit
From: Vernon Yang <[email protected]>
pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.
Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.
One visible consequence is data loss with MADV_FREE on PMD-mapped THP:
memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty
madvise(buf, size, MADV_FREE); // PMD cleaned but left writable,
// folio marked lazyfree
memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again
mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit
mprotect(buf, size, PROT_READ|PROT_WRITE);
// ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.
PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.
--- arch/x86/include/asm/pgtable.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)pmdval_t val =pmd_val (pmd), oldval = val;pmd_t pmd_result; - val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY); + val &= _HPAGE_CHG_MASK; val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK );
Signature · 1 line
2.53.0
Andrew Morton
maintainerQuoted 30 lines · On Thu, 3 Sep 2026 11:16:08 +0800 Vernon Yang <[email protected]> wrote:
From: Vernon Yang <[email protected]>
pmd_modify()masks the old value with (_HPAGE_CHG_MASK& ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequentpmd_mksaveddirty()call is supposed to transfer_PAGE_DIRTYinto_PAGE_SAVED_DIRTYwhen write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.Contrast with
pte_modify(), which keeps_PAGE_DIRTY_BITSin its mask, andpud_modify(), which keeps_HPAGE_CHG_MASKuntouched:pmd_modify()is the odd one out. Anypmd_modify()on a writable, dirty PMD loses the dirty state.One visible consequence is data loss with
MADV_FREEon PMD-mapped THP:memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty madvise(buf, size, MADV_FREE); // PMD cleaned but left writable, // folio marked lazyfree memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit mprotect(buf, size, PROT_READ|PROT_WRITE); // ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in
__discard_anon_folio_pmd_locked(), even though the data was rewritten afterMADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, asdo_huge_pmd_numa_page()restores the PMD throughpmd_modify()as well.
Quoted 8 lines
PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Fix it by keeping
_PAGE_DIRTYin the preserved mask, exactly likepte_modify()andpud_modify()do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.
Yeah, this is exactly what I came up with, using chatgpt.
Quoted 1 line
We definitely want a Reported-by: Orson here. He obviously did a ton of work on this, and it's the least we can do to thank him. I'll add it.
Quoted 3 lines
Fixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")Signed-off-by Vernon Yang <[email protected]>
Quoted 41 lines · Andrew Morton
Quoted 30 lines · On Thu, 3 Sep 2026 11:16:08 +0800 Vernon Yang <[email protected]> wrote:
From: Vernon Yang <[email protected]>
pmd_modify()masks the old value with (_HPAGE_CHG_MASK& ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequentpmd_mksaveddirty()call is supposed to transfer_PAGE_DIRTYinto_PAGE_SAVED_DIRTYwhen write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.Contrast with
pte_modify(), which keeps_PAGE_DIRTY_BITSin its mask, andpud_modify(), which keeps_HPAGE_CHG_MASKuntouched:pmd_modify()is the odd one out. Anypmd_modify()on a writable, dirty PMD loses the dirty state.One visible consequence is data loss with
MADV_FREEon PMD-mapped THP:memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty madvise(buf, size, MADV_FREE); // PMD cleaned but left writable, // folio marked lazyfree memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit mprotect(buf, size, PROT_READ|PROT_WRITE); // ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in
__discard_anon_folio_pmd_locked(), even though the data was rewritten afterMADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, asdo_huge_pmd_numa_page()restores the PMD throughpmd_modify()as well.Quoted 8 lines
PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Fix it by keeping
_PAGE_DIRTYin the preserved mask, exactly likepte_modify()andpud_modify()do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.
Quoted 6 lines
Yeah, this is exactly what I came up with, using chatgpt.
Quoted 1 line
We definitely want a Reported-by: Orson here. He obviously did a ton of work on this, and it's the least we can do to thank him. I'll add it.
Yeah, I missed adding Reported-by: Orson, Sorry. Thank you for adding it.
--
Cheers,
Vernon
Dear Andrew, Vernon,
Thanks for your quick reply, confirmation and patch.
What I failed to mention in the original report is that this isn't just a hypothetical issue found by analyzing the code, some of our users (of Polars, a data analytics library) have hit this in production. "Live data can change silently" is never a fun issue report to get.
Transparent huge pages + MADV_FREE + NUMA hinting + cgroup limits isn't your typical mom-and-pop setup, but it is not an unreasonable configuration either in high-performance computing. The only missing ingredient when you use Polars on a multi-NUMA-region machine running default Ubuntu configuration is the cgroup limits.
Considering that the impact is so severe if the bug does occur (silent memory corruption), will there be backports of this patch? If I'm not mistaken the offending line that introduced the bug was first released in version 6.6.
Best,
Orson
Quoted 53 lines · Andrew Morton
Quoted 30 lines · On Thu, 3 Sep 2026 11:16:08 +0800 Vernon Yang <[email protected]> wrote:
From: Vernon Yang <[email protected]>
pmd_modify()masks the old value with (_HPAGE_CHG_MASK& ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequentpmd_mksaveddirty()call is supposed to transfer_PAGE_DIRTYinto_PAGE_SAVED_DIRTYwhen write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.Contrast with
pte_modify(), which keeps_PAGE_DIRTY_BITSin its mask, andpud_modify(), which keeps_HPAGE_CHG_MASKuntouched:pmd_modify()is the odd one out. Anypmd_modify()on a writable, dirty PMD loses the dirty state.One visible consequence is data loss with
MADV_FREEon PMD-mapped THP:memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty madvise(buf, size, MADV_FREE); // PMD cleaned but left writable, // folio marked lazyfree memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit mprotect(buf, size, PROT_READ|PROT_WRITE); // ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in
__discard_anon_folio_pmd_locked(), even though the data was rewritten afterMADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, asdo_huge_pmd_numa_page()restores the PMD throughpmd_modify()as well.Quoted 8 lines
PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Fix it by keeping
_PAGE_DIRTYin the preserved mask, exactly likepte_modify()andpud_modify()do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.Yeah, this is exactly what I came up with, using chatgpt.
Quoted 1 line
We definitely want a Reported-by: Orson here. He obviously did a ton of work on this, and it's the least we can do to thank him. I'll add it.
Quoted 3 lines
Fixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")Signed-off-by Vernon Yang <[email protected]>
Quoted 4 lines · Orson Peters
Considering that the impact is so severe if the bug does occur (silent memory corruption), will there be backports of this patch? If I'm not mistaken the offending line that introduced the bug was first released in version 6.6.
Hi Orson,
It's tagged for stable@ so it should hit the stable trees at some point and make it into the 6.6 series.
Quoted 33 lines · Vernon Yang
From: Vernon Yang <[email protected]>
pmd_modify()masks the old value with (_HPAGE_CHG_MASK& ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequentpmd_mksaveddirty()call is supposed to transfer_PAGE_DIRTYinto_PAGE_SAVED_DIRTYwhen write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.Contrast with
pte_modify(), which keeps_PAGE_DIRTY_BITSin its mask, andpud_modify(), which keeps_HPAGE_CHG_MASKuntouched:pmd_modify()is the odd one out. Anypmd_modify()on a writable, dirty PMD loses the dirty state.One visible consequence is data loss with
MADV_FREEon PMD-mapped THP:memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty madvise(buf, size, MADV_FREE); // PMD cleaned but left writable, // folio marked lazyfree memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit mprotect(buf, size, PROT_READ|PROT_WRITE); // ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in
__discard_anon_folio_pmd_locked(), even though the data was rewritten afterMADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, asdo_huge_pmd_numa_page()restores the PMD throughpmd_modify()as well.PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Quoted 8 lines
Fix it by keeping
_PAGE_DIRTYin the preserved mask, exactly likepte_modify()andpud_modify()do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.Closes https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/ open hereFixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")
Hm. I don't understand why would this commit explicitly exclude _PAGE_DIRTY from the mask:
- val &= _HPAGE_CHG_MASK;
+ val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
Rick, could you comment? It doesn't look like a typo.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 9 lines · Kiryl Shutsemau
Quoted 2 lines
Closes https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/ open hereFixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")Hm. I don't understand why would this commit explicitly exclude
_PAGE_DIRTYfrom the mask:(quoted diff) +1
-1- val &= _HPAGE_CHG_MASK; + val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);Rick, could you comment? It doesn't look like a typo.
My guess is that it's some remnant from an earlier version of the patch. The asymmetry with pte_modify() vs. pmd_modify() just can't be explained any other way. It _might_ have been some attempt to say, "Hey _PAGE_DIRTY is now a part of the pgprot_t since it's part of the 'permissions' of a shadow stack PTE" that got abandoned.
But I don't see anything wrong with the fix at all.
It does give me pause that this has been losing user data for so long, but it must just be a weird combination of features that few folks use together (huge pages + MADV_FREE).
Andrew Morton
maintainerQuoted 17 lines · On Thu, 3 Sep 2026 07:18:54 -0700 Dave Hansen <[email protected]> wrote:
Quoted 9 lines · Kiryl Shutsemau
Quoted 2 lines
Closes https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/ open hereFixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")Hm. I don't understand why would this commit explicitly exclude
_PAGE_DIRTYfrom the mask:(quoted diff) +1
-1- val &= _HPAGE_CHG_MASK; + val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);Rick, could you comment? It doesn't look like a typo.
My guess is that it's some remnant from an earlier version of the patch. The asymmetry with
pte_modify()vs.pmd_modify()just can't be explained any other way. It _might_ have been some attempt to say, "Hey_PAGE_DIRTYis now a part of thepgprot_tsince it's part of the 'permissions' of a shadow stack PTE" that got abandoned.
Quoted 6 lines
But I don't see anything wrong with the fix at all.
It does give me pause that this has been losing user data for so long, but it must just be a weird combination of features that few folks use together (huge pages +
MADV_FREE).
And with heavy reclaim pressure.
Quoted 17 lines · Dave Hansen
Quoted 9 lines · Kiryl Shutsemau
Quoted 2 lines
Closes https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/ open hereFixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")Hm. I don't understand why would this commit explicitly exclude
_PAGE_DIRTYfrom the mask:- val &=
_HPAGE_CHG_MASK;
+ val &= (_HPAGE_CHG_MASK& ~_PAGE_DIRTY);Rick, could you comment? It doesn't look like a typo.
My guess is that it's some remnant from an earlier version of the patch. The asymmetry with
pte_modify()vs.pmd_modify()just can't be explained any other way. It _might_ have been some attempt to say, "Hey_PAGE_DIRTYis now a part of thepgprot_tsince it's part of the 'permissions' of a shadow stack PTE" that got abandoned.
Quoted 5 lines
But I don't see anything wrong with the fix at all.
It does give me pause that this has been losing user data for so long, but it must just be a weird combination of features that few folks use together (huge pages +
MADV_FREE).
Oof. Looking back through the patch history, the dirty bit used to be handled separately, such that the stripping was needed. Like this:
static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
{
pteval_t _page_chg_mask_no_dirty = _PAGE_CHG_MASK & ~_PAGE_DIRTY;
pteval_t val = pte_val(pte), oldval = val;
pte_t pte_result; /*
* Chop off the NX bit (if present), and add the NX portion of
* the newprot (if present):
*/
val &= _page_chg_mask_no_dirty; val |= check_pgprot(newprot) & ~_page_chg_mask_no_dirty;
val = flip_protnone_guard(oldval, val, PTE_PFN_MASK);
pte_result = __pte(val); /* * Dirty bit is not preserved above so it can be done * in a special way for the shadow stack case, where it * may need to set _PAGE_COW. __pte_mkdirty() will do this in * the case of shadow stack. */ if (pte_dirty(pte)) pte_result = __pte_mkdirty(pte_result, false); return pte_result;
}
static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
{
pteval_t _hpage_chg_mask_no_dirty = _HPAGE_CHG_MASK & ~_PAGE_DIRTY;
pmdval_t val = pmd_val(pmd), oldval = val;
pmd_t pmd_result;
val &= _hpage_chg_mask_no_dirty;
val |= check_pgprot(newprot) & ~_hpage_chg_mask_no_dirty; val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);
pmd_result = __pmd(val); /* * Dirty bit is not preserved above so it can be done * in a special way for the shadow stack case, where it * may need to set _PAGE_COW. __pmd_mkdirty() will do this in * the case of shadow stack. */ if (pmd_dirty(pmd)) pmd_result = __pmd_mkdirty(pmd_result, false); return pmd_result;
}
The dirty bit was removed from the pte, then handled separately to share logic in the mkdirty helpers. During development it was changed to do the necessary adjustments depending on the dirty bit in 'val', but the mask adjustment on the pmd_modify() side didn't get updated. So I don't remember or see any intention for the difference.
I can't find anything back then that would have prevented it. The bug cause and fix looks correct to me.
Quoted 2 lines · Rick P
I can't find anything back then that would have prevented it. The bug cause and fix looks correct to me.
Is that a Reviewed-by? ;)
Edgecombe, Rick P
Reviewed-byQuoted 5 lines · Dave Hansen
Quoted 2 lines · Rick P
I can't find anything back then that would have prevented it. The bug cause and fix looks correct to me.
Is that a Reviewed-by? ;)
[tip: x86/urgent] x86/mm: Fix userspace data loss with MADV_FREE and THP
In reply to Vernon YangThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: 4846278477f09d1f3b5f03091393b35a92db0b52 Gitweb: https://git.kernel.org/tip/4846278477f09d1f3b5f03091393b35a92db0b52 Author: Vernon Yang <[email protected]> AuthorDate: Thu, 03 Sep 2026 11:16:08 +08:00 Committer: Dave Hansen <[email protected]> CommitterDate: Thu, 03 Sep 2026 12:15:11 -07:00
x86/mm: Fix userspace data loss with MADV_FREE and THP
Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure.
pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.
Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.
One visible consequence is data loss with MADV_FREE on PMD-mapped THP:
memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty
madvise(buf, size, MADV_FREE); // PMD cleaned but left writable,
// folio marked lazyfree
memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again
mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit
mprotect(buf, size, PROT_READ|PROT_WRITE);
// ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.
PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.
--- arch/x86/include/asm/pgtable.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)pmdval_t val =pmd_val (pmd), oldval = val;pmd_t pmd_result; - val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY); + val &= _HPAGE_CHG_MASK; val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK );
Quoted 39 lines · Vernon Yang
From: Vernon Yang <[email protected]>
pmd_modify()masks the old value with (_HPAGE_CHG_MASK& ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequentpmd_mksaveddirty()call is supposed to transfer_PAGE_DIRTYinto_PAGE_SAVED_DIRTYwhen write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.Contrast with
pte_modify(), which keeps_PAGE_DIRTY_BITSin its mask, andpud_modify(), which keeps_HPAGE_CHG_MASKuntouched:pmd_modify()is the odd one out. Anypmd_modify()on a writable, dirty PMD loses the dirty state.One visible consequence is data loss with
MADV_FREEon PMD-mapped THP:memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty madvise(buf, size, MADV_FREE); // PMD cleaned but left writable, // folio marked lazyfree memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit mprotect(buf, size, PROT_READ|PROT_WRITE); // ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in
__discard_anon_folio_pmd_locked(), even though the data was rewritten afterMADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, asdo_huge_pmd_numa_page()restores the PMD throughpmd_modify()as well.PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Fix it by keeping
_PAGE_DIRTYin the preserved mask, exactly likepte_modify()andpud_modify()do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.
Quoted 4 lines
Closes https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/ open hereFixes bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")Signed-off-by Vernon Yang <[email protected]>
FYI we just had a customer case for our downstream kernel for this exact same issue. This fixed it beautifully (I'm wondering if polars started doing something interesting lately, that nobody else does, and that's why...).
Signature · 1 line
Pedro
[tip: x86/urgent] x86/mm: Fix user-space data loss with MADV_FREE and THP
In reply to Vernon YangThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: c70787db5b6a661940f5a7bedcdbc7daefe3e401 Gitweb: https://git.kernel.org/tip/c70787db5b6a661940f5a7bedcdbc7daefe3e401 Author: Vernon Yang <[email protected]> AuthorDate: Thu, 03 Sep 2026 11:16:08 +08:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 08 Sep 2026 08:53:15 +02:00
x86/mm: Fix user-space data loss with MADV_FREE and THP
Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure.
pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.
Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.
One visible consequence is data loss with MADV_FREE on PMD-mapped THP:
memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty
madvise(buf, size, MADV_FREE); // PMD cleaned but left writable,
// folio marked lazyfree
memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again
mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit
mprotect(buf, size, PROT_READ|PROT_WRITE);
// ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.
PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.
--- arch/x86/include/asm/pgtable.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)pmdval_t val =pmd_val (pmd), oldval = val;pmd_t pmd_result; - val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY); + val &= _HPAGE_CHG_MASK; val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK );
[tip: x86/mm] x86/mm: Fix user-space data loss with MADV_FREE and THP
In reply to Vernon YangThe following commit has been merged into the x86/mm branch of tip:
The following commit has been merged into the x86/mm branch of tip:
Commit-ID: 18d6b3df5aa910b8e3bad889f2658e6a4c8f6a0d Gitweb: https://git.kernel.org/tip/18d6b3df5aa910b8e3bad889f2658e6a4c8f6a0d Author: Vernon Yang <[email protected]> AuthorDate: Thu, 03 Sep 2026 11:16:08 +08:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 08 Sep 2026 09:15:12 +02:00
x86/mm: Fix user-space data loss with MADV_FREE and THP
Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure.
pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.
Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.
One visible consequence is data loss with MADV_FREE on PMD-mapped THP:
memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty
madvise(buf, size, MADV_FREE); // PMD cleaned but left writable,
// folio marked lazyfree
memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again
mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit
mprotect(buf, size, PROT_READ|PROT_WRITE);
// ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.
PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.
--- arch/x86/include/asm/pgtable.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)pmdval_t val =pmd_val (pmd), oldval = val;pmd_t pmd_result; - val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY); + val &= _HPAGE_CHG_MASK; val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK );
[tip: x86/urgent] x86/mm: Fix user-space data loss with MADV_FREE and THP
In reply to Vernon YangThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: 308e2bc308bcde06e56da6e9bf316b72217c3c24 Gitweb: https://git.kernel.org/tip/308e2bc308bcde06e56da6e9bf316b72217c3c24 Author: Vernon Yang <[email protected]> AuthorDate: Thu, 03 Sep 2026 11:16:08 +08:00 Committer: Dave Hansen <[email protected]> CommitterDate: Tue, 08 Sep 2026 15:43:07 -07:00
x86/mm: Fix user-space data loss with MADV_FREE and THP
Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure.
pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.
Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.
One visible consequence is data loss with MADV_FREE on PMD-mapped THP:
memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty
madvise(buf, size, MADV_FREE); // PMD cleaned but left writable,
// folio marked lazyfree
memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again
mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit
mprotect(buf, size, PROT_READ|PROT_WRITE);
// ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.
PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.
--- arch/x86/include/asm/pgtable.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)pmdval_t val =pmd_val (pmd), oldval = val;pmd_t pmd_result; - val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY); + val &= _HPAGE_CHG_MASK; val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK );
[tip: x86/urgent] x86/mm: Fix user-space data loss with MADV_FREE and THP
In reply to Vernon YangThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: f7491d7c81db0e7c304a7bd757a76d2fbeaff80e Gitweb: https://git.kernel.org/tip/f7491d7c81db0e7c304a7bd757a76d2fbeaff80e Author: Vernon Yang <[email protected]> AuthorDate: Thu, 03 Sep 2026 11:16:08 +08:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 09 Sep 2026 08:39:12 +02:00
x86/mm: Fix user-space data loss with MADV_FREE and THP
Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure.
pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer.
Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state.
One visible consequence is data loss with MADV_FREE on PMD-mapped THP:
memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty
madvise(buf, size, MADV_FREE); // PMD cleaned but left writable,
// folio marked lazyfree
memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again
mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit
mprotect(buf, size, PROT_READ|PROT_WRITE);
// ... memory pressure ...Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well.
PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back.
Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules.
--- arch/x86/include/asm/pgtable.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)pmdval_t val =pmd_val (pmd), oldval = val;pmd_t pmd_result; - val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY); + val &= _HPAGE_CHG_MASK; val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK );