mm: avoid zeroing user movable page twice with init_on_alloc=1
Subsystems (4)
Zi Yan
reviewerPATCH 1/1 mm: avoid zeroing user movable page twice with init_on_alloc=1
Commit 6471384af2a6 ("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed in post_alloc_hook() when init_on_alloc=1.
For order-0 folios, if arch does not define vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing __GFP_ZERO instead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their own vma_alloc_zeroed_movable_folio().
For >0 order folios (mTHP and PMD THP), folio_zero_user() is called to zero the folio again. Fix it by calling folio_zero_user() only if init_on_alloc is set. All arch are impacted.
Added alloc_zeroed() helper to encapsulate the init_on_alloc check.
--- include/linux/highmem.h | 8 +------- mm/huge_memory.c | 3 ++- mm/internal.h | 6 ++++++ mm/memory.c | 3 ++- 4 files changed, 11 insertions(+), 9 deletions(-)
include/linux/highmem.h +1 -7
@@ -224,13 +224,7 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); }#endif
mm/huge_memory.c +2 -1
@@ -1176,7 +1176,8 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, addr); +if (!alloc_zeroed ()) +folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that * folio_zero_user writes become visible before the set_pmd_at()
mm/internal.h +6 -0
@@ -1233,6 +1233,12 @@ void touch_pud(struct vm_area_struct *vma, unsigned long addr,void touch_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd,bool write); +static inline bool alloc_zeroed (void ) +{ +return static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , + &init_on_alloc); +} +enum {/* mark page accessed */ FOLL_TOUCH =1 <<16 ,
mm/memory.c +2 -1
@@ -4719,7 +4719,8 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto next; }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, vmf->address); +if (!alloc_zeroed ()) +folio_zero_user (folio, vmf->address);return folio; } next:
Signature · 1 line
2.45.2
Zi Yan
reviewer+Vlastimil
Quoted 91 lines · Zi Yan
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>--- include/linux/highmem.h | 8 +------- mm/huge_memory.c | 3 ++- mm/internal.h | 6 ++++++ mm/memory.c | 3 ++- 4 files changed, 11 insertions(+), 9 deletions(-)include/linux/highmem.h +1
-7@@ -224,13 +224,7 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); }#endif mm/huge_memory.c +2
-1@@ -1176,7 +1176,8 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, addr); +if (!alloc_zeroed ()) +folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that * folio_zero_user writes become visible before the set_pmd_at() mm/internal.h +6
-0@@ -1233,6 +1233,12 @@ void touch_pud(struct vm_area_struct *vma, unsigned long addr,void touch_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd,bool write); +static inline bool alloc_zeroed (void ) +{ +return static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , + &init_on_alloc); +} +enum {/* mark page accessed */ FOLL_TOUCH =1 <<16 ,mm/memory.c +2
-1@@ -4719,7 +4719,8 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto next; }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, vmf->address); +if (!alloc_zeroed ()) +folio_zero_user (folio, vmf->address);return folio; } next:Signature · 1 line
2.45.2
Best Regards,
Yan, Zi
Quoted 11 lines · Zi Yan
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().
Quoted 3 lines
For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.
^ not set?
Quoted 4 lines
Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>
Quoted 71 lines
--- include/linux/highmem.h | 8 +------- mm/huge_memory.c | 3 ++- mm/internal.h | 6 ++++++ mm/memory.c | 3 ++- 4 files changed, 11 insertions(+), 9 deletions(-)include/linux/highmem.h +1
-7@@ -224,13 +224,7 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); }#endif mm/huge_memory.c +2
-1@@ -1176,7 +1176,8 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, addr); +if (!alloc_zeroed ()) +folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that * folio_zero_user writes become visible before the set_pmd_at() mm/internal.h +6
-0@@ -1233,6 +1233,12 @@ void touch_pud(struct vm_area_struct *vma, unsigned long addr,void touch_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd,bool write); +static inline bool alloc_zeroed (void ) +{ +return static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , + &init_on_alloc); +} +enum {/* mark page accessed */ FOLL_TOUCH =1 <<16 ,mm/memory.c +2
-1@@ -4719,7 +4719,8 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto next; }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, vmf->address); +if (!alloc_zeroed ()) +folio_zero_user (folio, vmf->address);return folio; } next:
Zi Yan
reviewerQuoted 12 lines · Vlastimil Babka
Quoted 11 lines · Zi Yan
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().
Quoted 5 lines
Quoted 3 lines
For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.^ not set?
You are right. The sentence should be:
"Fix it by calling folio_zero_user() only if init_on_alloc is not set."
Hi Andrew,
Do you want me to resend this with fixed commit log?
Quoted 7 lines
Quoted 5 lines
Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>Acked-by Vlastimil Babka <[email protected]>
Thanks.
--
Best Regards,
Yan, Zi
Am 11.10.24 um 17:03 schrieb Zi Yan:
Quoted 49 lines
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>--- include/linux/highmem.h | 8 +------- mm/huge_memory.c | 3 ++- mm/internal.h | 6 ++++++ mm/memory.c | 3 ++- 4 files changed, 11 insertions(+), 9 deletions(-)include/linux/highmem.h +1
-7@@ -224,13 +224,7 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); }#endif mm/huge_memory.c +0
-0
mm/huge_memory.c static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, · line 1180
Quoted 6 lines
mm/huge_memory.c +2
-1@@ -1176,7 +1176,8 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, addr); +if (!alloc_zeroed ()) +folio_zero_user (folio, addr);
It might be reasonable to spell out why we are not using GFP_ZERO somewhere, something like
/*
* We are not using __GFP_ZERO because folio_zero_user() will make sure that the
* page corresponding to the faulting address will be hot in the cache. */
Sth. like that maybe.
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerQuoted 59 lines · David Hildenbrand
Am 11.10.24 um 17:03 schrieb Zi Yan:
Quoted 48 lines
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>--- include/linux/highmem.h | 8 +------- mm/huge_memory.c | 3 ++- mm/internal.h | 6 ++++++ mm/memory.c | 3 ++- 4 files changed, 11 insertions(+), 9 deletions(-)include/linux/highmem.h +1
-7@@ -224,13 +224,7 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); }#endif diff --git a/mm/huge_memory.c b/mm/huge_memory.cindex
82f464865570..5dcbea96edb7100644mm/huge_memory.c static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, · line 1180
Quoted 5 lines
mm/huge_memory.c +2
-0@@ -1176,7 +1176,8 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, addr); +if (!alloc_zeroed ()) +folio_zero_user (folio, addr);It might be reasonable to spell out why we are not using
GFP_ZEROsomewhere, something like
Quoted 6 lines
/*
* We are not using__GFP_ZERObecausefolio_zero_user()will make sure that the
* page corresponding to the faulting address will be hot in the cache. */Sth. like that maybe.
I changed the wording a bit to fit the if statement and put the comment in both call sites. Let me know how it looks. Thanks.
mm/huge_memory.c +5 -0
@@ -1180,6 +1180,11 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); +/* +* When a folio is not zeroed during allocation (__GFP_ZERO not used), +* folio_zero_user() is used to make sure that the page corresponding +* to the faulting address will be hot in the cache after zeroing. +*/ if (!alloc_zeroed ())folio_zero_user (folio, addr);/*
mm/memory.c +7 -0
@@ -4719,6 +4719,13 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto next; }folio_throttle_swaprate (folio, gfp); +/* +* When a folio is not zeroed during allocation +* (__GFP_ZERO not used), folio_zero_user() is used +* to make sure that the page corresponding to the +* faulting address will be hot in the cache after +* zeroing. +*/ if (!alloc_zeroed ())folio_zero_user (folio, vmf->address);return folio;
Quoted 2 lines
Thanks.
--
Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 66 lines · Zi Yan
Quoted 64 lines · David Hildenbrand
Am 11.10.24 um 17:03 schrieb Zi Yan:
Quoted 48 lines
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>--- include/linux/highmem.h | 8 +------- mm/huge_memory.c | 3 ++- mm/internal.h | 6 ++++++ mm/memory.c | 3 ++- 4 files changed, 11 insertions(+), 9 deletions(-)include/linux/highmem.h +1
-7@@ -224,13 +224,7 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); }#endif diff --git a/mm/huge_memory.c b/mm/huge_memory.cindex
82f464865570..5dcbea96edb7100644mm/huge_memory.c static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, · line 1180
Quoted 5 lines
mm/huge_memory.c +2
-0@@ -1176,7 +1176,8 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, addr); +if (!alloc_zeroed ()) +folio_zero_user (folio, addr);It might be reasonable to spell out why we are not using
GFP_ZEROsomewhere, something like/*
* We are not using__GFP_ZERObecausefolio_zero_user()will make sure that the
* page corresponding to the faulting address will be hot in the cache. */
Quoted 4 lines
Quoted 1 line
Sth. like that maybe.
I changed the wording a bit to fit the if statement and put the comment in both call sites. Let me know how it looks. Thanks.
Hi Andrew,
Do you mind folding the changes below into the original patch?
Thanks.
mm/huge_memory.c +5 -0
@@ -1180,6 +1180,11 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); +/* +* When a folio is not zeroed during allocation (__GFP_ZERO not used), +* folio_zero_user() is used to make sure that the page corresponding +* to the faulting address will be hot in the cache after zeroing. +*/ if (!alloc_zeroed ())folio_zero_user (folio, addr);/*
mm/memory.c +7 -0
@@ -4719,6 +4719,13 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto next; }folio_throttle_swaprate (folio, gfp); +/* +* When a folio is not zeroed during allocation +* (__GFP_ZERO not used), folio_zero_user() is used +* to make sure that the page corresponding to the +* faulting address will be hot in the cache after +* zeroing. +*/ if (!alloc_zeroed ())folio_zero_user (folio, vmf->address);return folio;
Quoted 4 lines
Quoted 2 lines
Thanks.
--
Best Regards,
Yan, Zi
Hi Zi,
Quoted 11 lines · Zi Yan
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().
Quoted 7 lines
For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>
Thanks for your patch, which is now commit 5708d96da20b99b4 ("mm:
avoid zeroing user movable page twice with init_on_alloc=1") in v6.13-rc1.
This causing a panic when starting userspace on MIPS64 RBTX4927:
Run /sbin/init as init process
process '/lib/systemd/systemd' started with executable stack Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ---[ end Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b ]---
or
Run /sbin/init as init process
process '/lib/systemd/systemd' started with executable stack do_page_fault(): sending SIGSEGV to init for invalid read access
from 00000000583399f8
epc = 0000000077e2b094 in ld-2.19.so[3094,77e28000+22000]
ra = 0000000077e2afcc in ld-2.19.so[2fcc,77e28000+22000]
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
---[ end Kernel panic - not syncing: Attempted to kill init!exitcode=0x0000000b ]---
or
Run /sbin/init as init process
process '/lib/systemd/systemd' started with executable stack /sbin/inKernel panic - not syncing: Attempted to kill init!
exitcode=0x00007f00
---[ end Kernel panic - not syncing: Attempted to kill init!
exitcode=0x00007f00 ]---
it: error while loading shared libraries: libpthread.so.0: object
file has no dynamic section
Reverting the commit (and fixing the trivial conflict) fixes the issue.
Quoted 53 lines
include/linux/highmem.h +1
-7@@ -224,13 +224,7 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); }#endif mm/huge_memory.c +2
-1@@ -1176,7 +1176,8 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, addr); +if (!alloc_zeroed ()) +folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that * folio_zero_user writes become visible before the set_pmd_at() mm/internal.h +6
-0@@ -1233,6 +1233,12 @@ void touch_pud(struct vm_area_struct *vma, unsigned long addr,void touch_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd,bool write); +static inline bool alloc_zeroed (void ) +{ +return static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , + &init_on_alloc); +} +enum {/* mark page accessed */ FOLL_TOUCH =1 <<16 ,mm/memory.c +0
-0
mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4726
Quoted 9 lines
mm/memory.c +2
-1@@ -4719,7 +4719,8 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto next; }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, vmf->address); +if (!alloc_zeroed ()) +folio_zero_user (folio, vmf->address);return folio; } next:
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Zi Yan
reviewerQuoted 18 lines · Geert Uytterhoeven
Hi Zi,
Quoted 15 lines · Zi Yan
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.
Quoted 7 lines
Quoted 3 lines
Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>Thanks for your patch, which is now commit
5708d96da20b99b4("mm:
avoid zeroing user movable page twice with init_on_alloc=1") in v6.13-rc1.
Thank you for reporting the error.
Quoted 23 lines
This causing a panic when starting userspace on MIPS64 RBTX4927:
Run
/sbin/initas init process
process '/lib/systemd/systemd' started with executable stack Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ---[ end Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b ]---or
Run
/sbin/initas init process
process '/lib/systemd/systemd' started with executable stackdo_page_fault(): sending SIGSEGV to init for invalid read access
from00000000583399f8epc = 0000000077e2b094 in ld-2.19.so[3094,77e28000+22000] ra = 0000000077e2afcc in ld-2.19.so[2fcc,77e28000+22000] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ---[ end Kernel panic - not syncing: Attempted to kill init!exitcode=0x0000000b ]---
or
Quoted 8 lines
Run
/sbin/initas init process
process '/lib/systemd/systemd' started with executable stack/sbin/inKernelpanic - not syncing: Attempted to kill init!
exitcode=0x00007f00---[ end Kernel panic - not syncing: Attempted to kill init!exitcode=0x00007f00 ]---
it: error while loading shared libraries: libpthread.so.0: object
file has no dynamic section
Do you mind providing the full kernel log for this panic? And your kernel config as well. I am trying to figure out why changing page zeroing from twice to once can cause kernel panic.
--
Best Regards,
Yan, Zi
Quoted 25 lines · Zi Yan
Quoted 16 lines · Geert Uytterhoeven
Quoted 15 lines · Zi Yan
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.Quoted 7 lines
Quoted 3 lines
Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>Thanks for your patch, which is now commit
5708d96da20b99b4("mm:
avoid zeroing user movable page twice with init_on_alloc=1") in v6.13-rc1.
Quoted 7 lines
Thank you for reporting the error.
Quoted 1 line
This causing a panic when starting userspace on MIPS64 RBTX4927:
Do you mind providing the full kernel log for this panic? And your kernel config as well. I am trying to figure out why changing page zeroing from twice to once can cause kernel panic.
Both sent by PM.
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Zi Yan
reviewerQuoted 113 lines · Geert Uytterhoeven
Hi Zi,
Quoted 11 lines · Zi Yan
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().Quoted 7 lines
For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>Thanks for your patch, which is now commit
5708d96da20b99b4("mm:
avoid zeroing user movable page twice with init_on_alloc=1") in v6.13-rc1.This causing a panic when starting userspace on MIPS64 RBTX4927:
Run
/sbin/initas init process
process '/lib/systemd/systemd' started with executable stack Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ---[ end Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b ]---or
Run
/sbin/initas init process
process '/lib/systemd/systemd' started with executable stackdo_page_fault(): sending SIGSEGV to init for invalid read access
from00000000583399f8epc = 0000000077e2b094 in ld-2.19.so[3094,77e28000+22000] ra = 0000000077e2afcc in ld-2.19.so[2fcc,77e28000+22000] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ---[ end Kernel panic - not syncing: Attempted to kill init!exitcode=0x0000000b ]---
or
Run
/sbin/initas init process
process '/lib/systemd/systemd' started with executable stack/sbin/inKernelpanic - not syncing: Attempted to kill init!
exitcode=0x00007f00---[ end Kernel panic - not syncing: Attempted to kill init!exitcode=0x00007f00 ]---
it: error while loading shared libraries: libpthread.so.0: object
file has no dynamic sectionReverting the commit (and fixing the trivial conflict) fixes the issue.
Quoted 54 lines
include/linux/highmem.h +1
-7@@ -224,13 +224,7 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); }#endif mm/huge_memory.c +2
-1@@ -1176,7 +1176,8 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, addr); +if (!alloc_zeroed ()) +folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that * folio_zero_user writes become visible before the set_pmd_at() mm/internal.h +6
-0@@ -1233,6 +1233,12 @@ void touch_pud(struct vm_area_struct *vma, unsigned long addr,void touch_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd,bool write); +static inline bool alloc_zeroed (void ) +{ +return static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , + &init_on_alloc); +} +enum {/* mark page accessed */ FOLL_TOUCH =1 <<16 ,mm/memory.c +0
-0@@ -4719,7 +4719,8 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto next;
Quoted 8 lines
mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4726
Quoted 8 lines
mm/memory.c +2
-1}folio_throttle_swaprate (folio, gfp); -folio_zero_user (folio, vmf->address); +if (!alloc_zeroed ()) +folio_zero_user (folio, vmf->address);return folio; } next:
The provided config does not have THP on, so the changes to mm/huge_memory.c and mm/memory.c do not apply.
Can you try the patch below and see if the machine boots? Thanks.
include/linux/highmem.h +7 -1
@@ -224,7 +224,13 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); +struct folio *folio; + + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); +if (folio) +clear_user_highpage (&folio->page, vaddr); + +return folio; }#endif
Best Regards,
Yan, Zi
Quoted 6 lines · Zi Yan
On 4 Dec 2024, at 5:41, Geert Uytterhoeven wrote:
The provided config does not have THP on, so the changes to
mm/huge_memory.candmm/memory.cdo not apply.Can you try the patch below and see if the machine boots? Thanks.
Hmm looks like mips has some involved clear_user_page() in arch/mips/include/asm/page.h
So maybe the clearing done as part of page allocator isn't enough here.
Quoted 24 lines
include/linux/highmem.h +7
-1@@ -224,7 +224,13 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); +struct folio *folio; + + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); +if (folio) +clear_user_highpage (&folio->page, vaddr); + +return folio; }#endif Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 6 lines · Vlastimil Babka
Quoted 5 lines · Zi Yan
On 4 Dec 2024, at 5:41, Geert Uytterhoeven wrote:
The provided config does not have THP on, so the changes to
mm/huge_memory.candmm/memory.cdo not apply.
Quoted 7 lines
Quoted 1 line
Can you try the patch below and see if the machine boots? Thanks.
Hmm looks like mips has some involved
clear_user_page()inarch/mips/include/asm/page.hSo maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.
One way to fix it is to add something like arch_userpage_post_alloc() to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separate kmap_local() calls,
2) how much overheads the additional kmap_local() and kunmap_local() have.
Quoted 21 lines
Quoted 21 lines
include/linux/highmem.h +7
-1@@ -224,7 +224,13 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); +struct folio *folio; + + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); +if (folio) +clear_user_highpage (&folio->page, vaddr); + +return folio; }#endif
Quoted 3 lines
Quoted 3 lines
Best Regards,
Yan, Zi
Best Regards,
Yan, Zi
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 10 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,
No. It just takes the next address in the stack.
Quoted 1 line
2) how much overheads the additional
kmap_local()andkunmap_local()have.
That's going to be a per-arch question ...
Zi Yan
reviewerQuoted 7 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
I mean if kmap address goes into a different cache line than userspace address, a cache flush is needed to make sure data is visible to userspace.
Quoted 7 lines
Quoted 7 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.
Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
So this fix will not work, since it is possible that first kmap and second kmap have different pages_do_alias() return values.
Another way would be to make a special case for mips, like below. But that looks ugly, let me think about it more.
arch/mips/include/asm/page.h +13 -0
@@ -95,6 +95,19 @@ struct vm_area_struct;extern void copy_user_highpage (struct page *to,struct page *from,unsigned long vaddr,struct vm_area_struct *vma); +struct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma, +unsigned long vaddr) + { +struct folio *folio; + + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); +if (folio) +clear_user_highpage (&folio->page, vaddr); + +return folio; + } +#define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio +#define __HAVE_ARCH_COPY_USER_HIGHPAGE/*
mm/internal.h +2 -1
@@ -1287,7 +1287,8 @@ void touch_pmd(struct vm_area_struct *vma, unsigned long addr,static inline bool alloc_zeroed (void ) { -return static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , +return !IS_ENABLED (CONFIG_MIPS ) && +static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , &init_on_alloc); }
Best Regards,
Yan, Zi
Hi Zi,
Quoted 62 lines · Zi Yan
Quoted 7 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
I mean if kmap address goes into a different cache line than userspace address, a cache flush is needed to make sure data is visible to userspace.
Quoted 7 lines
Quoted 7 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
So this fix will not work, since it is possible that first kmap and second kmap have different
pages_do_alias()return values.Another way would be to make a special case for mips, like below. But that looks ugly, let me think about it more.
arch/mips/include/asm/page.h +13
-0@@ -95,6 +95,19 @@ struct vm_area_struct;extern void copy_user_highpage (struct page *to,struct page *from,unsigned long vaddr,struct vm_area_struct *vma); +struct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma, +unsigned long vaddr) + { +struct folio *folio; + + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); +if (folio) +clear_user_highpage (&folio->page, vaddr); + +return folio; + } +#define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio +#define __HAVE_ARCH_COPY_USER_HIGHPAGE/* mm/internal.h +0
-0
mm/internal.h void touch_pmd(struct vm_area_struct *vma, unsigned long addr, · line 1293
Quoted 8 lines
mm/internal.h +2
-1@@ -1287,7 +1287,8 @@ void touch_pmd(struct vm_area_struct *vma, unsigned long addr,static inline bool alloc_zeroed (void ) { -return static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , +return !IS_ENABLED (CONFIG_MIPS ) && +static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , &init_on_alloc); }
After adding a missing static inline, #include <linux/gfp.h>, and still getting compile failures, I gave up...
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Zi Yan
reviewerQuoted 66 lines · Geert Uytterhoeven
Hi Zi,
Quoted 63 lines · Zi Yan
Quoted 7 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
I mean if kmap address goes into a different cache line than userspace address, a cache flush is needed to make sure data is visible to userspace.
Quoted 7 lines
Quoted 7 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
So this fix will not work, since it is possible that first kmap and second kmap have different
pages_do_alias()return values.Another way would be to make a special case for mips, like below. But that looks ugly, let me think about it more.
arch/mips/include/asm/page.h +13
-0@@ -95,6 +95,19 @@ struct vm_area_struct;extern void copy_user_highpage (struct page *to,struct page *from,unsigned long vaddr,struct vm_area_struct *vma); +struct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma, +unsigned long vaddr) + { +struct folio *folio; + + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); +if (folio) +clear_user_highpage (&folio->page, vaddr); + +return folio; + } +#define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio +#define __HAVE_ARCH_COPY_USER_HIGHPAGE/* mm/internal.h +0
-0@@ -1287,7 +1287,8 @@ void touch_pmd(struct vm_area_struct *vma, unsigned long addr,
Quoted 10 lines
mm/internal.h void touch_pmd(struct vm_area_struct *vma, unsigned long addr, · line 1293
Quoted 7 lines
mm/internal.h +2
-1static inline bool alloc_zeroed (void ) { -return static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , +return !IS_ENABLED (CONFIG_MIPS ) && +static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , &init_on_alloc); }After adding a missing static inline, #include <
linux/gfp.h>, and still getting compile failures, I gave up...
Sorry about that.
Can you try the patch below (it compiles locally for mips and x86) to see if your issue is fixed? Can you please make THP always on in your config, since THP is also affected by the same issue? The patch you tested only fixed non THP config.
Thanks. I appreciate your help. :)
include/linux/highmem.h +7 -1
@@ -224,7 +224,13 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); +struct folio *folio; + + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); +if (folio &&alloc_need_zeroing ()) +clear_user_highpage (&folio->page, vaddr); + +return folio; }#endif
include/linux/mm.h +17 -0
@@ -4175,6 +4175,23 @@ static inline int do_mseal(unsigned long start, size_t len_in, unsigned long fla }#endif +/* +* alloc_need_zeroing checks if a user folio from page allocator needs to be +* zeroed or not. +*/ +static inline bool alloc_need_zeroing (void ) +{ +/* +* for user folios, arch with cache aliasing requires cache flush and +* arc sets folio->flags, so always return false to make caller use +* clear_user_page()/clear_user_highpage() +*/ +return (IS_ENABLED (CONFIG_ARCH_HAS_CPU_CACHE_ALIASING ) || +IS_ENABLED (CONFIG_ARC )) || + !static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , + &init_on_alloc); +} +int arch_get_shadow_stack_status (struct task_struct *t,unsigned long __user *status);int arch_set_shadow_stack_status (struct task_struct *t,unsigned long status);int arch_lock_shadow_stack_status (struct task_struct *t,unsigned long status);
mm/huge_memory.c +5 -4
@@ -1176,11 +1176,12 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,folio_throttle_swaprate (folio, gfp);/* -* When a folio is not zeroed during allocation (__GFP_ZERO not used), -* folio_zero_user() is used to make sure that the page corresponding -* to the faulting address will be hot in the cache after zeroing. +* When a folio is not zeroed during allocation (__GFP_ZERO not used) +* or user folios require special handling, folio_zero_user() is used to +* make sure that the page corresponding to the faulting address will be +* hot in the cache after zeroing. */ -if (!alloc_zeroed ()) +if (alloc_need_zeroing ())folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that
mm/internal.h +0 -6
@@ -1285,12 +1285,6 @@ void touch_pud(struct vm_area_struct *vma, unsigned long addr,void touch_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd,bool write); -static inline bool alloc_zeroed (void ) -{ -return static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , - &init_on_alloc); -} -/* * Parses a string with mem suffixes into its order. Useful to parse kernel * parameters.
mm/memory.c +5 -5
@@ -4733,12 +4733,12 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)folio_throttle_swaprate (folio, gfp);/* * When a folio is not zeroed during allocation -* (__GFP_ZERO not used), folio_zero_user() is used -* to make sure that the page corresponding to the -* faulting address will be hot in the cache after -* zeroing. +* (__GFP_ZERO not used) or user folios require special +* handling, folio_zero_user() is used to make sure +* that the page corresponding to the faulting address +* will be hot in the cache after zeroing. */ -if (!alloc_zeroed ()) +if (alloc_need_zeroing ())folio_zero_user (folio, vmf->address);return folio; }
Best Regards,
Yan, Zi
Geert Uytterhoeven
Tested-byHi Zi,
Quoted 4 lines · Zi Yan
Can you try the patch below (it compiles locally for mips and x86) to see if your issue is fixed? Can you please make THP always on in your config, since THP is also affected by the same issue? The patch you tested only fixed non THP config.
Thanks, this works both without THP, and withCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
Quoted 1 line
Thanks. I appreciate your help. :)
Thanks, I appreciate your quick responses and solutions!
Quoted 95 lines
include/linux/highmem.h +7
-1@@ -224,7 +224,13 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); +struct folio *folio; + + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); +if (folio &&alloc_need_zeroing ()) +clear_user_highpage (&folio->page, vaddr); + +return folio; }#endif include/linux/mm.h +17
-0@@ -4175,6 +4175,23 @@ static inline int do_mseal(unsigned long start, size_t len_in, unsigned long fla }#endif +/* +* alloc_need_zeroing checks if a user folio from page allocator needs to be +* zeroed or not. +*/ +static inline bool alloc_need_zeroing (void ) +{ +/* +* for user folios, arch with cache aliasing requires cache flush and +* arc sets folio->flags, so always return false to make caller use +* clear_user_page()/clear_user_highpage() +*/ +return (IS_ENABLED (CONFIG_ARCH_HAS_CPU_CACHE_ALIASING ) || +IS_ENABLED (CONFIG_ARC )) || + !static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , + &init_on_alloc); +} +int arch_get_shadow_stack_status (struct task_struct *t,unsigned long __user *status);int arch_set_shadow_stack_status (struct task_struct *t,unsigned long status);int arch_lock_shadow_stack_status (struct task_struct *t,unsigned long status);mm/huge_memory.c +5
-4@@ -1176,11 +1176,12 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,folio_throttle_swaprate (folio, gfp);/* -* When a folio is not zeroed during allocation (__GFP_ZERO not used), -* folio_zero_user() is used to make sure that the page corresponding -* to the faulting address will be hot in the cache after zeroing. +* When a folio is not zeroed during allocation (__GFP_ZERO not used) +* or user folios require special handling, folio_zero_user() is used to +* make sure that the page corresponding to the faulting address will be +* hot in the cache after zeroing. */ -if (!alloc_zeroed ()) +if (alloc_need_zeroing ())folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that mm/internal.h +0
-6@@ -1285,12 +1285,6 @@ void touch_pud(struct vm_area_struct *vma, unsigned long addr,void touch_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd,bool write); -static inline bool alloc_zeroed (void ) -{ -return static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , - &init_on_alloc); -} -/* * Parses a string with mem suffixes into its order. Useful to parse kernel * parameters. mm/memory.c +0
-4@@ -4733,12 +4733,12 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)folio_throttle_swaprate (folio, gfp);/* * When a folio is not zeroed during allocation -* (__GFP_ZERO not used), folio_zero_user() is used -* to make sure that the page corresponding to the -* faulting address will be hot in the cache after -* zeroing.
mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4744
Quoted 10 lines
mm/memory.c +5
-1+* (__GFP_ZERO not used) or user folios require special +* handling, folio_zero_user() is used to make sure +* that the page corresponding to the faulting address +* will be hot in the cache after zeroing. */ -if (!alloc_zeroed ()) +if (alloc_need_zeroing ())folio_zero_user (folio, vmf->address);return folio; }
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Zi Yan
reviewerQuoted 14 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 6 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.
Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
Hmm, if kmap_local() gives different addresses, wouldn’t init_on_alloc be causing issues before my patch? In the page allocator, the page is zeroed from one kmap address without flush, then clear_user_highpage() clears it again with another kmap address with flush. After returning to userspace, the user application works on the page but when the cache line used by init_on_alloc is written back (with 0s) at eviction, user data is corrupted. Am I missing anything? Or all arch with cache aliasing never enables init_on_alloc?
Best Regards,
Yan, Zi
Quoted 23 lines · Zi Yan
Quoted 14 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 6 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
Quoted 8 lines
Hmm, if
kmap_local()gives different addresses, wouldn’tinit_on_allocbe causing issues before my patch? In the page allocator, the page is zeroed from one kmap address without flush, thenclear_user_highpage()clears it again with another kmap address with flush. After returning to userspace, the user application works on the page but when the cache line used byinit_on_allocis written back (with 0s) at eviction, user data is corrupted. Am I missing anything? Or all arch with cache aliasing never enablesinit_on_alloc?
Maybe the arch also defines some hooks like arch_kmap_local_post_unmap() ?
As for the fix, could it rely on e.g. __HAVE_ARCH_COPY_USER_HIGHPAGE instead of CONFIG_MIPS? That affects more arches, I don't know if we broke only mips or others too.
Quoted 2 lines
Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 33 lines · Vlastimil Babka
Quoted 23 lines · Zi Yan
Quoted 14 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 6 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
Quoted 8 lines
Hmm, if
kmap_local()gives different addresses, wouldn’tinit_on_allocbe causing issues before my patch? In the page allocator, the page is zeroed from one kmap address without flush, thenclear_user_highpage()clears it again with another kmap address with flush. After returning to userspace, the user application works on the page but when the cache line used byinit_on_allocis written back (with 0s) at eviction, user data is corrupted. Am I missing anything? Or all arch with cache aliasing never enablesinit_on_alloc?
Quoted 1 line
Maybe the arch also defines some hooks like
arch_kmap_local_post_unmap()?
But this does not solve the possible init_on_alloc issue, since init_on_alloc is done in mm/page_alloc.c without userspace address and has no knowledge of whether the zeroed page will be used in userspace nor the cache line will be the same as the userspace page cache line. If dcache is flushed unconditionally for kmap_local, that could degrade performance.
Quoted 3 lines
As for the fix, could it rely on e.g.
__HAVE_ARCH_COPY_USER_HIGHPAGEinstead ofCONFIG_MIPS? That affects more arches, I don't know if we broke only mips or others too.
Yes, this is much better, since this issue affects any arch with cache aliasing. Let me update my fix. Thanks.
Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 43 lines · Zi Yan
Quoted 33 lines · Vlastimil Babka
Quoted 23 lines · Zi Yan
Quoted 14 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 6 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
Quoted 8 lines
Hmm, if
kmap_local()gives different addresses, wouldn’tinit_on_allocbe causing issues before my patch? In the page allocator, the page is zeroed from one kmap address without flush, thenclear_user_highpage()clears it again with another kmap address with flush. After returning to userspace, the user application works on the page but when the cache line used byinit_on_allocis written back (with 0s) at eviction, user data is corrupted. Am I missing anything? Or all arch with cache aliasing never enablesinit_on_alloc?Quoted 1 line
Maybe the arch also defines some hooks like
arch_kmap_local_post_unmap()?But this does not solve the possible
init_on_allocissue, sinceinit_on_allocis done inmm/page_alloc.cwithout userspace address and has no knowledge of whether the zeroed page will be used in userspace nor the cache line will be the same as the userspace page cache line. If dcache is flushed unconditionally forkmap_local, that could degrade performance.
Quoted 6 lines
Quoted 3 lines
As for the fix, could it rely on e.g.
__HAVE_ARCH_COPY_USER_HIGHPAGEinstead ofCONFIG_MIPS? That affects more arches, I don't know if we broke only mips or others too.Yes, this is much better, since this issue affects any arch with cache aliasing. Let me update my fix. Thanks.
I notice that arm64 has __HAVE_ARCH_COPY_USER_HIGHPAGE defined, so I will need to look for an alternative.
Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 49 lines · Zi Yan
Quoted 47 lines · Zi Yan
Quoted 33 lines · Vlastimil Babka
Quoted 23 lines · Zi Yan
Quoted 14 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 6 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
Quoted 8 lines
Hmm, if
kmap_local()gives different addresses, wouldn’tinit_on_allocbe causing issues before my patch? In the page allocator, the page is zeroed from one kmap address without flush, thenclear_user_highpage()clears it again with another kmap address with flush. After returning to userspace, the user application works on the page but when the cache line used byinit_on_allocis written back (with 0s) at eviction, user data is corrupted. Am I missing anything? Or all arch with cache aliasing never enablesinit_on_alloc?Quoted 1 line
Maybe the arch also defines some hooks like
arch_kmap_local_post_unmap()?But this does not solve the possible
init_on_allocissue, sinceinit_on_allocis done inmm/page_alloc.cwithout userspace address and has no knowledge of whether the zeroed page will be used in userspace nor the cache line will be the same as the userspace page cache line. If dcache is flushed unconditionally forkmap_local, that could degrade performance.Quoted 3 lines
As for the fix, could it rely on e.g.
__HAVE_ARCH_COPY_USER_HIGHPAGEinstead ofCONFIG_MIPS? That affects more arches, I don't know if we broke only mips or others too.
Quoted 5 lines
Quoted 2 lines
Yes, this is much better, since this issue affects any arch with cache aliasing. Let me update my fix. Thanks.
I notice that arm64 has
__HAVE_ARCH_COPY_USER_HIGHPAGEdefined, so I will need to look for an alternative.
It turns out sh, sparc, arm, xtensa, nios2, m68k, parisc, csky, and powerpc all have cache flush operations in clear_user_page() compared to clear_page() and arc clears PG_dc_clean bit in addition to clear_page().
So __GFP_ZERO cannot simply replace clear_user_page()/clear_user_highpage(). I can add ARCH_REQUIRE_CLEAR_USER_PAGE for these arch and use it to decide whether clear_user_page()/clear_user_highpage() needs to be used regardless of the presence of init_on_alloc.
I also wonder if INIT_ON_ALLOC_DEFAULT_ON works on these arch or not.
Best Regards,
Yan, Zi
Quoted 60 lines · Zi Yan
Quoted 49 lines · Zi Yan
Quoted 47 lines · Zi Yan
Quoted 33 lines · Vlastimil Babka
Quoted 23 lines · Zi Yan
Quoted 14 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 6 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
Quoted 8 lines
Hmm, if
kmap_local()gives different addresses, wouldn’tinit_on_allocbe causing issues before my patch? In the page allocator, the page is zeroed from one kmap address without flush, thenclear_user_highpage()clears it again with another kmap address with flush. After returning to userspace, the user application works on the page but when the cache line used byinit_on_allocis written back (with 0s) at eviction, user data is corrupted. Am I missing anything? Or all arch with cache aliasing never enablesinit_on_alloc?Quoted 1 line
Maybe the arch also defines some hooks like
arch_kmap_local_post_unmap()?But this does not solve the possible
init_on_allocissue, sinceinit_on_allocis done inmm/page_alloc.cwithout userspace address and has no knowledge of whether the zeroed page will be used in userspace nor the cache line will be the same as the userspace page cache line. If dcache is flushed unconditionally forkmap_local, that could degrade performance.Quoted 3 lines
As for the fix, could it rely on e.g.
__HAVE_ARCH_COPY_USER_HIGHPAGEinstead ofCONFIG_MIPS? That affects more arches, I don't know if we broke only mips or others too.Quoted 5 lines
Quoted 2 lines
Yes, this is much better, since this issue affects any arch with cache aliasing. Let me update my fix. Thanks.
I notice that arm64 has
__HAVE_ARCH_COPY_USER_HIGHPAGEdefined, so I will need to look for an alternative.It turns out sh, sparc, arm, xtensa, nios2, m68k, parisc, csky, and powerpc all have cache flush operations in
clear_user_page()compared toclear_page()and arc clearsPG_dc_cleanbit in addition toclear_page().
Quoted 7 lines
So
__GFP_ZEROcannot simply replace clear_user_page()/clear_user_highpage(). I can addARCH_REQUIRE_CLEAR_USER_PAGEfor these arch and use it to decide whether clear_user_page()/clear_user_highpage() needs to be used regardless of the presence ofinit_on_alloc.I also wonder if
INIT_ON_ALLOC_DEFAULT_ONworks on these arch or not.
Well, I've been waiting to point out that if you actually *delete* the entire INIT_ON_ALLOC feature, you'd be my personal hero. Defense in depth is nice, but at some point, it crosses a line into the absurd, and I think we are there. </pause and put on asbestos flame suit> :)
thanks,
Signature · 1 line
John Hubbard
Zi Yan
reviewerQuoted 23 lines · Zi Yan
Quoted 14 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 6 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
Quoted 8 lines
Hmm, if
kmap_local()gives different addresses, wouldn’tinit_on_allocbe causing issues before my patch? In the page allocator, the page is zeroed from one kmap address without flush, thenclear_user_highpage()clears it again with another kmap address with flush. After returning to userspace, the user application works on the page but when the cache line used byinit_on_allocis written back (with 0s) at eviction, user data is corrupted. Am I missing anything? Or all arch with cache aliasing never enablesinit_on_alloc?
Hi Geert,
Regarding the above concern, have you ever had CONFIG_INIT_ON_ALLOC_DEFAULT_ON for your MIPS machine and encountered any issue? Or let me know if my reasoning above is flawed.
To test it, I wonder if you can 1) revert my patch and 2) turn on CONFIG_INIT_ON_ALLOC_DEFAULT_ON for your MIPS machine and run some applications to see if any error happens.
Thanks.
Best Regards,
Yan, Zi
Hi Zi,
Quoted 34 lines · Zi Yan
Quoted 22 lines · Zi Yan
Quoted 14 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 6 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
Quoted 8 lines
Hmm, if
kmap_local()gives different addresses, wouldn’tinit_on_allocbe causing issues before my patch? In the page allocator, the page is zeroed from one kmap address without flush, thenclear_user_highpage()clears it again with another kmap address with flush. After returning to userspace, the user application works on the page but when the cache line used byinit_on_allocis written back (with 0s) at eviction, user data is corrupted. Am I missing anything? Or all arch with cache aliasing never enablesinit_on_alloc?Hi Geert,
Quoted 7 lines
Regarding the above concern, have you ever had
CONFIG_INIT_ON_ALLOC_DEFAULT_ONfor your MIPS machine and encountered any issue? Or let me know if my reasoning above is flawed.To test it, I wonder if you can 1) revert my patch and 2) turn on
CONFIG_INIT_ON_ALLOC_DEFAULT_ONfor your MIPS machine and run some applications to see if any error happens.
That seems to work fine...
Kernel log confirms it's enabled:
-mem auto-init: stack:off, heap alloc:off, heap free:off +mem auto-init: stack:off, heap alloc:on, heap free:off
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Quoted 48 lines · Geert Uytterhoeven
Hi Zi,
Quoted 34 lines · Zi Yan
Quoted 22 lines · Zi Yan
Quoted 14 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 6 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
Quoted 8 lines
Hmm, if
kmap_local()gives different addresses, wouldn’tinit_on_allocbe causing issues before my patch? In the page allocator, the page is zeroed from one kmap address without flush, thenclear_user_highpage()clears it again with another kmap address with flush. After returning to userspace, the user application works on the page but when the cache line used byinit_on_allocis written back (with 0s) at eviction, user data is corrupted. Am I missing anything? Or all arch with cache aliasing never enablesinit_on_alloc?Hi Geert,
Quoted 7 lines
Regarding the above concern, have you ever had
CONFIG_INIT_ON_ALLOC_DEFAULT_ONfor your MIPS machine and encountered any issue? Or let me know if my reasoning above is flawed.To test it, I wonder if you can 1) revert my patch and 2) turn on
CONFIG_INIT_ON_ALLOC_DEFAULT_ONfor your MIPS machine and run some applications to see if any error happens.That seems to work fine...
Kernel log confirms it's enabled:
Review
Quoted 2 lines
(quoted diff) +1
-1-mem auto-init: stack:off, heap alloc:off, heap free:off +mem auto-init: stack:off, heap alloc:on, heap free:off
If I'm not wrong that's expected ... because we'll be double-zeroing that memory, clearing the cache :)
I guess the question is, how *effective* is CONFIG_INIT_ON_ALLOC_DEFAULT_ON on systems to prevent exposing un-zeroed data to userspace, when it doesn't end up doing the flush we really need.
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerQuoted 48 lines · David Hildenbrand
Quoted 47 lines · Geert Uytterhoeven
Hi Zi,
Quoted 34 lines · Zi Yan
Quoted 22 lines · Zi Yan
Quoted 14 lines · Matthew Wilcox
Quoted 3 lines · Zi Yan
Quoted 2 lines
So maybe the clearing done as part of page allocator isn't enough here.
Basically, mips needs to flush data cache if kmap address is aliased to
People use "aliased" in contronym ways. Do you mean "has a non-congruent alias" or "has a congruent alias"?
Quoted 6 lines
userspace address. This means when mips has THP on, the patch below is not enough to fix the issue.
In
post_alloc_hook(), it does not make sense to pass userspace address in to determine whether to flush dcache or not.Quoted 6 lines
Quoted 4 lines
One way to fix it is to add something like
arch_userpage_post_alloc()to flush dcache if kmap address is aliased to userspace address. But my questions are that
1) if kmap address will always be the same for two separatekmap_local()calls,No. It just takes the next address in the stack.
Quoted 8 lines
Hmm, if
kmap_local()gives different addresses, wouldn’tinit_on_allocbe causing issues before my patch? In the page allocator, the page is zeroed from one kmap address without flush, thenclear_user_highpage()clears it again with another kmap address with flush. After returning to userspace, the user application works on the page but when the cache line used byinit_on_allocis written back (with 0s) at eviction, user data is corrupted. Am I missing anything? Or all arch with cache aliasing never enablesinit_on_alloc?Hi Geert,
Quoted 7 lines
Regarding the above concern, have you ever had
CONFIG_INIT_ON_ALLOC_DEFAULT_ONfor your MIPS machine and encountered any issue? Or let me know if my reasoning above is flawed.To test it, I wonder if you can 1) revert my patch and 2) turn on
CONFIG_INIT_ON_ALLOC_DEFAULT_ONfor your MIPS machine and run some applications to see if any error happens.That seems to work fine...
Quoted 7 lines
Review
Quoted 3 lines
Kernel log confirms it's enabled:
(quoted diff) +1
-1-mem auto-init: stack:off, heap alloc:off, heap free:off +mem auto-init: stack:off, heap alloc:on, heap free:offIf I'm not wrong that's expected ... because we'll be double-zeroing that memory, clearing the cache :)
I guess the question is, how *effective* is
CONFIG_INIT_ON_ALLOC_DEFAULT_ONon systems to prevent exposing un-zeroed data to userspace, when it doesn't end up doing the flush we really need.
Hi Geert,
Is it possible to run a 32bit kernel with HIGHMEM and CONFIG_INIT_ON_ALLOC_DEFAULT_ON on the machine (of course with my patch reverted)? Just to check my reasoning below.
Thanks.
Yes, it should work, since I forgot the actual issue is HIGHMEM+cache flush, not just cache flush is needed after clearing user page.
For arch which needs to flush cache after clearing user page, with HIGHMEM, init_on_alloc first clears the page using kmap_addr0 without flushing the cache, then clear_user_page() clears the page using kmap_addr1 with cache flush. After returning to userspace, the cache lines of kmap_addr0 will be evicted and written back to RAM eventually, corrupting user data with 0s, because no one flushes them before returning to userspace.
For a proper fix, I will add ARCH_HAS_OPS_AFTER_CLEAR_USER_PAGE and make mips, sh, sparc, arm, xtensa, nios2, m68k, parisc, csky, arc, and powerpc select it, then make alloc_zeroed() returns false if ARCH_HAS_OPS_AFTER_CLEAR_USER_PAGE is enabled.
If my reasoning above is verified to be true, I will send a separate patch to disable CONFIG_INIT_ON_ALLOC_DEFAULT_ON if HIGHMEM && ARCH_HAS_OPS_AFTER_CLEAR_USER_PAGE.
Best Regards,
Yan, Zi
Quoted 28 lines · Zi Yan
Quoted 8 lines · David Hildenbrand
Review
Quoted 4 lines
Kernel log confirms it's enabled:
(quoted diff) +1
-1-mem auto-init: stack:off, heap alloc:off, heap free:off +mem auto-init: stack:off, heap alloc:on, heap free:offIf I'm not wrong that's expected ... because we'll be double-zeroing that memory, clearing the cache :)
I guess the question is, how *effective* is
CONFIG_INIT_ON_ALLOC_DEFAULT_ONon systems to prevent exposing un-zeroed data to userspace, when it doesn't end up doing the flush we really need.Hi Geert,
Is it possible to run a 32bit kernel with HIGHMEM and
CONFIG_INIT_ON_ALLOC_DEFAULT_ONon the machine (of course with my patch reverted)? Just to check my reasoning below.Thanks.
Yes, it should work, since I forgot the actual issue is HIGHMEM+cache flush, not just cache flush is needed after clearing user page.
For arch which needs to flush cache after clearing user page, with HIGHMEM,
init_on_allocfirst clears the page usingkmap_addr0without flushing the cache, thenclear_user_page()clears the page usingkmap_addr1with cache flush. After returning to userspace, the cache lines ofkmap_addr0will be evicted and written back to RAM eventually, corrupting user data with 0s, because no one flushes them before returning to userspace.
Quoted 8 lines
For a proper fix, I will add
ARCH_HAS_OPS_AFTER_CLEAR_USER_PAGEand make mips, sh, sparc, arm, xtensa, nios2, m68k, parisc, csky, arc, and powerpc select it, then makealloc_zeroed()returns false ifARCH_HAS_OPS_AFTER_CLEAR_USER_PAGEis enabled.If my reasoning above is verified to be true, I will send a separate patch to disable
CONFIG_INIT_ON_ALLOC_DEFAULT_ONif HIGHMEM &&ARCH_HAS_OPS_AFTER_CLEAR_USER_PAGE.
If your reasoning is true, wouldn't any other user of kmap_local_page() of a highpage on such system also leave the cache unflushed in case the page is ever reused as a userspace page?
Quoted 2 lines
Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 34 lines · Vlastimil Babka
Quoted 33 lines · Zi Yan
Quoted 8 lines · David Hildenbrand
Review
Quoted 4 lines
Kernel log confirms it's enabled:
(quoted diff) +1
-1-mem auto-init: stack:off, heap alloc:off, heap free:off +mem auto-init: stack:off, heap alloc:on, heap free:offIf I'm not wrong that's expected ... because we'll be double-zeroing that memory, clearing the cache :)
I guess the question is, how *effective* is
CONFIG_INIT_ON_ALLOC_DEFAULT_ONon systems to prevent exposing un-zeroed data to userspace, when it doesn't end up doing the flush we really need.Hi Geert,
Is it possible to run a 32bit kernel with HIGHMEM and
CONFIG_INIT_ON_ALLOC_DEFAULT_ONon the machine (of course with my patch reverted)? Just to check my reasoning below.Thanks.
Yes, it should work, since I forgot the actual issue is HIGHMEM+cache flush, not just cache flush is needed after clearing user page.
For arch which needs to flush cache after clearing user page, with HIGHMEM,
init_on_allocfirst clears the page usingkmap_addr0without flushing the cache, thenclear_user_page()clears the page usingkmap_addr1with cache flush. After returning to userspace, the cache lines ofkmap_addr0will be evicted and written back to RAM eventually, corrupting user data with 0s, because no one flushes them before returning to userspace.For a proper fix, I will add
ARCH_HAS_OPS_AFTER_CLEAR_USER_PAGEand make mips, sh, sparc, arm, xtensa, nios2, m68k, parisc, csky, arc, and powerpc select it, then makealloc_zeroed()returns false ifARCH_HAS_OPS_AFTER_CLEAR_USER_PAGEis enabled.
Quoted 7 lines
Quoted 3 lines
If my reasoning above is verified to be true, I will send a separate patch to disable
CONFIG_INIT_ON_ALLOC_DEFAULT_ONif HIGHMEM &&ARCH_HAS_OPS_AFTER_CLEAR_USER_PAGE.If your reasoning is true, wouldn't any other user of
kmap_local_page()of a highpage on such system also leave the cache unflushed in case the page is ever reused as a userspace page?
If the page is written and no cache is flushed, yes. But if the page is read and the cache lines are clean, no write back will be done.
I wonder in what scenarios kernel writes to user pages, besides copy_to_user*(), which should handle the cache flush.
Best Regards,
Yan, Zi
Hi Zi,
Quoted 3 lines · Zi Yan
Is it possible to run a 32bit kernel with HIGHMEM and
CONFIG_INIT_ON_ALLOC_DEFAULT_ONon the machine (of course with my patch reverted)? Just to check my reasoning below.
While I can boot 32-bit kernels, CPU_TX49XX and MACH_TX49XX do not select CPU_SUPPORTS_HIGHMEM resp. SYS_SUPPORTS_HIGHMEM, and thus HIGHMEM is not supported.
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Geert Uytterhoeven
Tested-byHi Zi,
Quoted 71 lines · Zi Yan
Quoted 50 lines · Geert Uytterhoeven
Quoted 11 lines · Zi Yan
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be zeroed inpost_alloc_hook()when init_on_alloc=1.For order-0 folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again zeros the page return from the buddy allocator. So the page is zeroed twice. Fix it by passing__GFP_ZEROinstead to avoid double page zeroing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().Quoted 7 lines
For >0 order folios (mTHP and PMD THP),
folio_zero_user()is called to zero the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.Added
alloc_zeroed()helper to encapsulate theinit_on_alloccheck.Signed-off-by Zi Yan <[email protected]>Thanks for your patch, which is now commit
5708d96da20b99b4("mm:
avoid zeroing user movable page twice with init_on_alloc=1") in v6.13-rc1.This causing a panic when starting userspace on MIPS64 RBTX4927:
Run
/sbin/initas init process
process '/lib/systemd/systemd' started with executable stack Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ---[ end Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b ]---or
Run
/sbin/initas init process
process '/lib/systemd/systemd' started with executable stackdo_page_fault(): sending SIGSEGV to init for invalid read access
from00000000583399f8epc = 0000000077e2b094 in ld-2.19.so[3094,77e28000+22000] ra = 0000000077e2afcc in ld-2.19.so[2fcc,77e28000+22000] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ---[ end Kernel panic - not syncing: Attempted to kill init!exitcode=0x0000000b ]---
or
Run
/sbin/initas init process
process '/lib/systemd/systemd' started with executable stack/sbin/inKernelpanic - not syncing: Attempted to kill init!
exitcode=0x00007f00Quoted 6 lines
---[ end Kernel panic - not syncing: Attempted to kill init!exitcode=0x00007f00 ]---
it: error while loading shared libraries: libpthread.so.0: object
file has no dynamic sectionReverting the commit (and fixing the trivial conflict) fixes the issue.
The provided config does not have THP on, so the changes to
mm/huge_memory.candmm/memory.cdo not apply.Can you try the patch below and see if the machine boots? Thanks.
include/linux/highmem.h +0
-0@@ -224,7 +224,13 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) {
include/linux/highmem.h static inline · line 235
Quoted 10 lines
include/linux/highmem.h +7
-1- return vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr); +struct folio *folio; + + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); +if (folio) +clear_user_highpage (&folio->page, vaddr); + + return folio; }#endif
Thanks, that works!
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds