mm: avoid clearing user movable page twice with init_on_alloc=1
Subsystems (3)
Zi Yan
reviewerRFC PATCH 1/1 mm: avoid clearing user movable page twice with init_on_alloc=1
Commit 6471384af2a6 ("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be cleared in post_alloc_hook() when init_on_alloc=1.
For non PMD folios, if arch does not define vma_alloc_zeroed_movable_folio(), the default implementation again clears the page return from the buddy allocator. So the page is cleared twice. Fix it by passing __GFP_ZERO instead to avoid double page clearing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their own vma_alloc_zeroed_movable_folio().
For PMD folios, folio_zero_user() is called to clear the folio again. Fix it by calling folio_zero_user() only if init_on_alloc is set. All arch are impacted.
--- include/linux/highmem.h | 14 ++------------ mm/huge_memory.c | 4 +++- 2 files changed, 5 insertions(+), 13 deletions(-)
include/linux/highmem.h +2 -12
@@ -220,18 +220,8 @@ static inline void clear_user_highpage(struct page *page, unsigned long vaddr)* Return: A folio containing one allocated and zeroed page or NULL if * we are out of memory. */ -static inline -struct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma, -unsigned long vaddr) -{ -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr, false); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; -} +#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)#endif static inline void clear_highpage (struct page *page)
mm/huge_memory.c +3 -1
@@ -1177,7 +1177,9 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, goto release; } -folio_zero_user (folio, vmf->address); +if (!static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , + &init_on_alloc)) +folio_zero_user (folio, vmf->address);/* * The memory barrier inside __folio_mark_uptodate makes sure that * folio_zero_user writes become visible before the set_pmd_at()
Signature · 1 line
2.45.2
Quoted 51 lines · Zi Yan
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be cleared inpost_alloc_hook()when init_on_alloc=1.For non PMD folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again clears the page return from the buddy allocator. So the page is cleared twice. Fix it by passing__GFP_ZEROinstead to avoid double page clearing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().For PMD folios,
folio_zero_user()is called to clear the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.Signed-off-by Zi Yan <[email protected]>--- include/linux/highmem.h | 14 ++------------ mm/huge_memory.c | 4 +++- 2 files changed, 5 insertions(+), 13 deletions(-)include/linux/highmem.h +2
-12@@ -220,18 +220,8 @@ static inline void clear_user_highpage(struct page *page, unsigned long vaddr)* Return: A folio containing one allocated and zeroed page or NULL if * we are out of memory. */ -static inline -struct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma, -unsigned long vaddr) -{ -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr, false); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; -} +#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)#endif static inline void clear_highpage (struct page *page)mm/huge_memory.c +0
-0
mm/huge_memory.c static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, · line 1185
Quoted 10 lines
mm/huge_memory.c +3
-1@@ -1177,7 +1177,9 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, goto release; } -folio_zero_user (folio, vmf->address); +if (!static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , + &init_on_alloc)) +folio_zero_user (folio, vmf->address);/* * The memory barrier inside __folio_mark_uptodate makes sure that * folio_zero_user writes become visible before the set_pmd_at()
I remember we discussed that in the past and that we do *not* want to sprinkle these CONFIG_INIT_ON_ALLOC_DEFAULT_ON checks all over the kernel.
Ideally, we'd use GFP_ZERO and have the buddy just do that for us? There is the slight chance that we zero-out when we're not going to use the allocated folio, but ... that can happen either way even with the current code?
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerQuoted 63 lines · David Hildenbrand
Quoted 50 lines · Zi Yan
Commit
6471384af2a6("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") forces allocated page to be cleared inpost_alloc_hook()when init_on_alloc=1.For non PMD folios, if arch does not define
vma_alloc_zeroed_movable_folio(), the default implementation again clears the page return from the buddy allocator. So the page is cleared twice. Fix it by passing__GFP_ZEROinstead to avoid double page clearing. At the moment, s390,arm64,x86,alpha,m68k are not impacted since they define their ownvma_alloc_zeroed_movable_folio().For PMD folios,
folio_zero_user()is called to clear the folio again. Fix it by callingfolio_zero_user()only ifinit_on_allocis set. All arch are impacted.Signed-off-by Zi Yan <[email protected]>--- include/linux/highmem.h | 14 ++------------ mm/huge_memory.c | 4 +++- 2 files changed, 5 insertions(+), 13 deletions(-)include/linux/highmem.h +2
-12@@ -220,18 +220,8 @@ static inline void clear_user_highpage(struct page *page, unsigned long vaddr)* Return: A folio containing one allocated and zeroed page or NULL if * we are out of memory. */ -static inline -struct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma, -unsigned long vaddr) -{ -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr, false); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; -} +#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)#endif static inline void clear_highpage (struct page *page)mm/huge_memory.c +0
-0mm/huge_memory.c static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, · line 1185
Quoted 9 lines
mm/huge_memory.c +3
-0@@ -1177,7 +1177,9 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, goto release; } -folio_zero_user (folio, vmf->address); +if (!static_branch_maybe (CONFIG_INIT_ON_ALLOC_DEFAULT_ON , + &init_on_alloc)) +folio_zero_user (folio, vmf->address);/* * The memory barrier inside __folio_mark_uptodate makes sure that * folio_zero_user writes become visible before the set_pmd_at() I remember we discussed that in the past and that we do *not* want to sprinkle these
CONFIG_INIT_ON_ALLOC_DEFAULT_ONchecks all over the kernel.
Quoted 1 line
Ideally, we'd use
GFP_ZEROand have the buddy just do that for us? There is the slight chance that we zero-out when we're not going to use the allocated folio, but ... that can happen either way even with the current code?
I agree that putting CONFIG_INIT_ON_ALLOC_DEFAULT_ON here is not ideal, but folio_zero_user() uses vmf->address to improve cache performance by changing subpage clearing order. See commit c79b57e462b5 ("mm: hugetlb: clear target sub-page last when clearing huge page”). If we use GFP_ZERO, we lose this optimization. To keep it, vmf->address will need to be passed to allocation code. Maybe that is acceptable?
Best Regards,
Yan, Zi
Quoted 8 lines · Zi Yan
Quoted 4 lines · David Hildenbrand
I remember we discussed that in the past and that we do *not* want to sprinkle these
CONFIG_INIT_ON_ALLOC_DEFAULT_ONchecks all over the kernel.Ideally, we'd use
GFP_ZEROand have the buddy just do that for us? There is the slight chance that we zero-out when we're not going to use the allocated folio, but ... that can happen either way even with the current code?I agree that putting
CONFIG_INIT_ON_ALLOC_DEFAULT_ONhere is not ideal, but
Create some nice inline wrapper for the test and it will look less ugly? :)
Quoted 5 lines
folio_zero_user()usesvmf->addressto improve cache performance by changing subpage clearing order. See commitc79b57e462b5("mm: hugetlb: clear target sub-page last when clearing huge page”). If we useGFP_ZERO, we lose this optimization. To keep it,vmf->addresswill need to be passed to allocation code. Maybe that is acceptable?
I'd rather not change the page allocation code for this...
Quoted 2 lines
Best Regards,
Yan, Zi
Quoted 12 lines · Vlastimil Babka
Quoted 8 lines · Zi Yan
Quoted 4 lines · David Hildenbrand
I remember we discussed that in the past and that we do *not* want to sprinkle these
CONFIG_INIT_ON_ALLOC_DEFAULT_ONchecks all over the kernel.Ideally, we'd use
GFP_ZEROand have the buddy just do that for us? There is the slight chance that we zero-out when we're not going to use the allocated folio, but ... that can happen either way even with the current code?I agree that putting
CONFIG_INIT_ON_ALLOC_DEFAULT_ONhere is not ideal, butCreate some nice inline wrapper for the test and it will look less ugly? :)
Quoted 7 lines
Quoted 5 lines
folio_zero_user()usesvmf->addressto improve cache performance by changing subpage clearing order. See commitc79b57e462b5("mm: hugetlb: clear target sub-page last when clearing huge page”). If we useGFP_ZERO, we lose this optimization. To keep it,vmf->addresswill need to be passed to allocation code. Maybe that is acceptable?I'd rather not change the page allocation code for this...
Although I'm curious if that optimization from 2017 is still valuable :)
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerQuoted 5 lines · David Hildenbrand
Quoted 4 lines · Vlastimil Babka
Quoted 3 lines · Zi Yan
Quoted 1 line · David Hildenbrand
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I remember we discussed that in the past and that we do *not* want to sprinkle these
CONFIG_INIT_ON_ALLOC_DEFAULT_ONchecks all over the kernel.Ideally, we'd use
GFP_ZEROand have the buddy just do that for us? There is the slight chance that we zero-out when we're not going to use the allocated folio, but ... that can happen either way even with the current code?I agree that putting
CONFIG_INIT_ON_ALLOC_DEFAULT_ONhere is not ideal, butCreate some nice inline wrapper for the test and it will look less ugly? :)
something like?
static inline bool alloc_zeroed()
{
return static_branch_maybe(CONFIG_INIT_ON_ALLOC_DEFAULT_ON,
&init_on_alloc);
}I missed another folio_zero_user() caller in alloc_anon_folio() for mTHP. So both PMD THP and mTHP are zeroed twice for all arch.
Adding Ryan for mTHP.
Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
folio_zero_user()usesvmf->addressto improve cache performance by changing subpage clearing order. See commitc79b57e462b5("mm: hugetlb: clear target sub-page last when clearing huge page”). If we useGFP_ZERO, we lose this optimization. To keep it,vmf->addresswill need to be passed to allocation code. Maybe that is acceptable?I'd rather not change the page allocation code for this...
Although I'm curious if that optimization from 2017 is still valuable :)
Maybe Ying can give some insight on this.
Do we need some general guidance on who is responsible for zeroing allocated folios? Should people use GFP_ZERO instead of zeroing by themselves if possible?
Best Regards,
Yan, Zi
Quoted 36 lines · Zi Yan
Quoted 5 lines · David Hildenbrand
Quoted 4 lines · Vlastimil Babka
Quoted 3 lines · Zi Yan
Quoted 1 line · David Hildenbrand
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I remember we discussed that in the past and that we do *not* want to sprinkle these
CONFIG_INIT_ON_ALLOC_DEFAULT_ONchecks all over the kernel.Ideally, we'd use
GFP_ZEROand have the buddy just do that for us? There is the slight chance that we zero-out when we're not going to use the allocated folio, but ... that can happen either way even with the current code?I agree that putting
CONFIG_INIT_ON_ALLOC_DEFAULT_ONhere is not ideal, butCreate some nice inline wrapper for the test and it will look less ugly? :)
something like?
static inline bool
alloc_zeroed(){ return static_branch_maybe(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, &init_on_alloc); }I missed another
folio_zero_user()caller inalloc_anon_folio()for mTHP. So both PMD THP and mTHP are zeroed twice for all arch.Adding Ryan for mTHP.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
folio_zero_user()usesvmf->addressto improve cache performance by changing subpage clearing order. See commitc79b57e462b5("mm: hugetlb: clear target sub-page last when clearing huge page”). If we useGFP_ZERO, we lose this optimization. To keep it,vmf->addresswill need to be passed to allocation code. Maybe that is acceptable?
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I'd rather not change the page allocation code for this...
Although I'm curious if that optimization from 2017 is still valuable :)
Maybe Ying can give some insight on this.
I guess the optimization still applies now. Although the size of the per-core(thread) last level cache increases, it's still quite common for it to be smaller than the size of THP. And the sizes of L1/L2 are significantly smaller, the likelihood for the accessed cache line to be in L1/L2/LLC increases with the optimization.
--
Best Regards,
Huang, Ying
Zi Yan <[email protected]> writes:
[snip]
Quoted 12 lines
include/linux/highmem.h +0
-4@@ -220,18 +220,8 @@ static inline void clear_user_highpage(struct page *page, unsigned long vaddr)* Return: A folio containing one allocated and zeroed page or NULL if * we are out of memory. */ -static inline -struct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma, -unsigned long vaddr) -{
include/linux/highmem.h static inline void clear_user_highpage(struct page *page, unsigned long vaddr) · line 224
Quoted 10 lines
include/linux/highmem.h +2
-8-struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr, false); -if (folio) -clear_user_highpage (&folio->page, vaddr); - -return folio; -} +#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)
Although just one line, I still prefer to use inline function instead of macro here. Not strong opinion.
include/linux/highmem.h static inline void clear_user_highpage(struct page *page, unsigned long vaddr) · line 225
Quoted 1 line
include/linux/highmem.h +0
-0#endif
[snip]
--
Best Regards,
Huang, Ying