add persistent huge zero folio support
Subsystems (5)
v2 cover add persistent huge zero folio support
From: Pankaj Raghav <[email protected]>
Many places in the kernel need to zero out larger chunks, but the maximum segment we can zero out at a time by ZERO_PAGE is limited by PAGE_SIZE.
This concern was raised during the review of adding Large Block Size support to XFS[2][3].
This is especially annoying in block devices and filesystems where multiple ZERO_PAGEs are attached to the bio in different bvecs. With multipage bvec support in block layer, it is much more efficient to send out larger zero pages as a part of single bvec.
Some examples of places in the kernel where this could be useful:
- blkdev_issue_zero_pages()
- iomap_dio_zero()
- vmalloc.c:zero_iter()
- rxperf_process_call()
- fscrypt_zeroout_range_inline_crypt()
- bch2_checksum_update()
...
Usually huge_zero_folio is allocated on demand, and it will be deallocated by the shrinker if there are no users of it left. At the moment, huge_zero_folio infrastructure refcount is tied to the process lifetime that created it. This might not work for bio layer as the completions can be async and the process that created the huge_zero_folio might no longer be alive. And, one of the main point that came during discussion is to have something bigger than zero page as a drop-in replacement.
Add a config option PERSISTENT_HUGE_ZERO_FOLIO that will always allocate the huge_zero_folio, and disable the shrinker so that huge_zero_folio is never freed.
This makes using the huge_zero_folio without having to pass any mm struct and does not tie the lifetime of the zero folio to anything, making it a drop-in replacement for ZERO_PAGE.
I have converted blkdev_issue_zero_pages() as an example as a part of this series. I also noticed close to 4% performance improvement just by replacing ZERO_PAGE with persistent huge_zero_folio.
I will send patches to individual subsystems using the huge_zero_folio once this gets upstreamed.
Looking forward to some feedback.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-xfs/[email protected]/
[3] https://lore.kernel.org/linux-xfs/[email protected]/
Changes since v1:
- Simplified the code by allocating in thp_shinrker_init() and disable the shrinker when this PERSISTENT_HUGE_ZERO_FOLIO config is enabled.
- Reworked commit messages and config messages based on Dave's feedback
- Added RVB and Acked-by tags.
Changes since RFC v2:
- Convert get_huge_zero_page and put_huge_zero_page to *_folio.
- Convert MMF_HUGE_ZERO_PAGE to MMF_HUGE_ZERO_FOLIO.
- Make the retry for huge_zero_folio from 2 to 1.
- Add an extra sanity check in shrinker scan for static huge_zero_folio case.
Changes since v1:
- Fixed all warnings.
- Added a retry feature after a particular time.
- Added Acked-by and Signed-off-by from David.
Changes since last series[1]:
- Instead of allocating a new page through memblock, use the same infrastructure as huge_zero_folio but raise the reference and never drop it. (David)
- And some minor cleanups based on Lorenzo's feedback.
Pankaj Raghav (5):
mm: rename huge_zero_page to huge_zero_folio
mm: rename MMF_HUGE_ZERO_PAGE to MMF_HUGE_ZERO_FOLIO
mm: add persistent huge zero folio
mm: add largest_zero_folio() routine
block: use largest_zero_folio in __blkdev_issue_zero_pages()
block/blk-lib.c | 15 +++++---- include/linux/huge_mm.h | 38 ++++++++++++++++++++++ include/linux/mm_types.h | 2 +- mm/Kconfig | 16 +++++++++ mm/huge_memory.c | 70 ++++++++++++++++++++++++++-------------- 5 files changed, 108 insertions(+), 33 deletions(-)
base-commit: 53c448023185717d0ed56b5546dc2be405da92ff
Signature · 1 line
2.49.0
PATCH v2 1/5 mm: rename huge_zero_page to huge_zero_folio
From: Pankaj Raghav <[email protected]>
As the transition already happened from exposing huge_zero_page to huge_zero_folio, change the name of the shrinker and the other helper function to reflect that.
No functional changes.
--- mm/huge_memory.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
mm/huge_memory.c +17 -17
@@ -207,7 +207,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,return orders; } -static bool get_huge_zero_page (void ) +static bool get_huge_zero_folio (void ) {struct folio *zero_folio; retry:
@@ -237,7 +237,7 @@ static bool get_huge_zero_page(void)return true; } -static void put_huge_zero_page (void ) +static void put_huge_zero_folio (void ) {/* * Counter should never go to zero here. Only shrinker can put
@@ -251,11 +251,11 @@ struct folio *mm_get_huge_zero_folio(struct mm_struct *mm)if (test_bit (MMF_HUGE_ZERO_PAGE , &mm->flags))return READ_ONCE (huge_zero_folio); -if (!get_huge_zero_page ()) +if (!get_huge_zero_folio ())return NULL ;if (test_and_set_bit (MMF_HUGE_ZERO_PAGE , &mm->flags)) -put_huge_zero_page (); +put_huge_zero_folio ();return READ_ONCE (huge_zero_folio); }
@@ -263,18 +263,18 @@ struct folio *mm_get_huge_zero_folio(struct mm_struct *mm)void mm_put_huge_zero_folio (struct mm_struct *mm) {if (test_bit (MMF_HUGE_ZERO_PAGE , &mm->flags)) -put_huge_zero_page (); +put_huge_zero_folio (); } -static unsigned long shrink_huge_zero_page_count (struct shrinker *shrink, -struct shrink_control *sc) +static unsigned long shrink_huge_zero_folio_count (struct shrinker *shrink, +struct shrink_control *sc) {/* we can free zero page only if last reference remains */ return atomic_read (&huge_zero_refcount) ==1 ?HPAGE_PMD_NR :0 ; } -static unsigned long shrink_huge_zero_page_scan (struct shrinker *shrink, -struct shrink_control *sc) +static unsigned long shrink_huge_zero_folio_scan (struct shrinker *shrink, +struct shrink_control *sc) {if (atomic_cmpxchg (&huge_zero_refcount,1 ,0 ) ==1 ) {struct folio *zero_folio =xchg (&huge_zero_folio,NULL );
@@ -287,7 +287,7 @@ static unsigned long shrink_huge_zero_page_scan(struct shrinker *shrink,return 0 ; } -static struct shrinker *huge_zero_page_shrinker; +static struct shrinker *huge_zero_folio_shrinker;#ifdef CONFIG_SYSFS static ssize_t enabled_show(struct kobject *kobj,
@@ -849,8 +849,8 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj)static int __init thp_shrinker_init (void ) { - huge_zero_page_shrinker =shrinker_alloc (0 ,"thp-zero" ); -if (!huge_zero_page_shrinker) + huge_zero_folio_shrinker =shrinker_alloc (0 ,"thp-zero" ); +if (!huge_zero_folio_shrinker)return -ENOMEM ; deferred_split_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE |
@@ -858,13 +858,13 @@ static int __init thp_shrinker_init(void)SHRINKER_NONSLAB , "thp-deferred_split");if (!deferred_split_shrinker) { -shrinker_free (huge_zero_page_shrinker); +shrinker_free (huge_zero_folio_shrinker);return -ENOMEM ; } - huge_zero_page_shrinker->count_objects = shrink_huge_zero_page_count; - huge_zero_page_shrinker->scan_objects = shrink_huge_zero_page_scan; -shrinker_register (huge_zero_page_shrinker); + huge_zero_folio_shrinker->count_objects = shrink_huge_zero_folio_count; + huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan; +shrinker_register (huge_zero_folio_shrinker); deferred_split_shrinker->count_objects = deferred_split_count; deferred_split_shrinker->scan_objects = deferred_split_scan;
@@ -875,7 +875,7 @@ static int __init thp_shrinker_init(void)static void __init thp_shrinker_exit (void ) { -shrinker_free (huge_zero_page_shrinker); +shrinker_free (huge_zero_folio_shrinker);shrinker_free (deferred_split_shrinker); }
Signature · 1 line
2.49.0
Re: [PATCH v2 1/5] mm: rename huge_zero_page to huge_zero_folio
Quoted 8 lines · Pankaj Raghav (Samsung)
From: Pankaj Raghav <[email protected]>
As the transition already happened from exposing
huge_zero_pagetohuge_zero_folio, change the name of the shrinker and the other helper function to reflect that.No functional changes.
Quoted 9 lines
Reviewed-by Lorenzo Stoakes <[email protected]>Reviewed-by Zi Yan <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Pankaj Raghav <[email protected]>--- mm/huge_memory.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH v2 2/5 mm: rename MMF_HUGE_ZERO_PAGE to MMF_HUGE_ZERO_FOLIO
In reply to Pankaj Raghav (Samsung)From: Pankaj Raghav <[email protected]>
As all the helper functions has been renamed from *_page to *_folio, rename the MM flag from MMF_HUGE_ZERO_PAGE to MMF_HUGE_ZERO_FOLIO.
No functional changes.
--- include/linux/mm_types.h | 2 +- mm/huge_memory.c | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-)
include/linux/mm_types.h +1 -1
@@ -1758,7 +1758,7 @@ enum {#define MMF_RECALC_UPROBES 20/* MMF_HAS_UPROBES can be wrong */ #define MMF_OOM_SKIP 21/* mm is of no interest for the OOM killer */ #define MMF_UNSTABLE 22/* mm is unstable for copy_from_user */ -#define MMF_HUGE_ZERO_PAGE 23/* mm has ever used the global huge zero page */ +#define MMF_HUGE_ZERO_FOLIO 23/* mm has ever used the global huge zero folio */ #define MMF_DISABLE_THP 24/* disable THP for all VMAs */ #define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP)#define MMF_OOM_REAP_QUEUED 25/* mm was queued for oom_reaper */
mm/huge_memory.c +3 -3
@@ -248,13 +248,13 @@ static void put_huge_zero_folio(void)struct folio *mm_get_huge_zero_folio (struct mm_struct *mm) { -if (test_bit (MMF_HUGE_ZERO_PAGE , &mm->flags)) +if (test_bit (MMF_HUGE_ZERO_FOLIO , &mm->flags))return READ_ONCE (huge_zero_folio);if (!get_huge_zero_folio ())return NULL ; -if (test_and_set_bit (MMF_HUGE_ZERO_PAGE , &mm->flags)) +if (test_and_set_bit (MMF_HUGE_ZERO_FOLIO , &mm->flags))put_huge_zero_folio ();return READ_ONCE (huge_zero_folio);
@@ -262,7 +262,7 @@ struct folio *mm_get_huge_zero_folio(struct mm_struct *mm)void mm_put_huge_zero_folio (struct mm_struct *mm) { -if (test_bit (MMF_HUGE_ZERO_PAGE , &mm->flags)) +if (test_bit (MMF_HUGE_ZERO_FOLIO , &mm->flags))put_huge_zero_folio (); }
Signature · 1 line
2.49.0
Re: [PATCH v2 2/5] mm: rename MMF_HUGE_ZERO_PAGE to MMF_HUGE_ZERO_FOLIO
Quoted 12 lines · Pankaj Raghav (Samsung)
From: Pankaj Raghav <[email protected]>
As all the helper functions has been renamed from
*_pageto*_folio, rename the MM flag fromMMF_HUGE_ZERO_PAGEtoMMF_HUGE_ZERO_FOLIO.No functional changes.
Reviewed-by Lorenzo Stoakes <[email protected]>Reviewed-by Zi Yan <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Pankaj Raghav <[email protected]>
Quoted 5 lines
--- include/linux/mm_types.h | 2 +- mm/huge_memory.c | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH v2 3/5 mm: add persistent huge zero folio
In reply to Pankaj Raghav (Samsung)From: Pankaj Raghav <[email protected]>
Many places in the kernel need to zero out larger chunks, but the maximum segment that can be zeroed out at a time by ZERO_PAGE is limited by PAGE_SIZE.
This is especially annoying in block devices and filesystems where multiple ZERO_PAGEs are attached to the bio in different bvecs. With multipage bvec support in block layer, it is much more efficient to send out larger zero pages as a part of single bvec.
This concern was raised during the review of adding Large Block Size support to XFS[1][2].
Usually huge_zero_folio is allocated on demand, and it will be deallocated by the shrinker if there are no users of it left. At moment, huge_zero_folio infrastructure refcount is tied to the process lifetime that created it. This might not work for bio layer as the completions can be async and the process that created the huge_zero_folio might no longer be alive. And, one of the main points that came up during discussion is to have something bigger than zero page as a drop-in replacement.
Add a config option PERSISTENT_HUGE_ZERO_FOLIO that will result in allocating the huge zero folio during early init and never free the memory by disabling the shrinker. This makes using the huge_zero_folio without having to pass any mm struct and does not tie the lifetime of the zero folio to anything, making it a drop-in replacement for ZERO_PAGE.
If PERSISTENT_HUGE_ZERO_FOLIO config option is enabled, then mm_get_huge_zero_folio() will simply return the allocated page instead of dynamically allocating a new PMD page.
Use this option carefully in resource constrained systems as it uses one full PMD sized page for zeroing purposes.
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/[email protected]/
--- include/linux/huge_mm.h | 16 ++++++++++++++++ mm/Kconfig | 16 ++++++++++++++++ mm/huge_memory.c | 40 ++++++++++++++++++++++++++++++---------- 3 files changed, 62 insertions(+), 10 deletions(-)
include/linux/huge_mm.h +16 -0
@@ -495,6 +495,17 @@ static inline bool is_huge_zero_pmd(pmd_t pmd)struct folio *mm_get_huge_zero_folio (struct mm_struct *mm);void mm_put_huge_zero_folio (struct mm_struct *mm); +static inline struct folio *get_persistent_huge_zero_folio (void ) +{ +if (!IS_ENABLED (CONFIG_PERSISTENT_HUGE_ZERO_FOLIO )) +return NULL ; + +if (unlikely (!huge_zero_folio)) +return NULL ; + +return huge_zero_folio; +} +static inline bool thp_migration_supported (void ) {return IS_ENABLED (CONFIG_ARCH_ENABLE_THP_MIGRATION );
@@ -685,6 +696,11 @@ static inline int change_huge_pud(struct mmu_gather *tlb, {return 0 ; } + +static inline struct folio *get_persistent_huge_zero_folio (void ) +{ +return NULL ; +}#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline int split_folio_to_list_to_order(struct folio *folio,
mm/Kconfig +16 -0
@@ -823,6 +823,22 @@ config ARCH_WANT_GENERAL_HUGETLB config ARCH_WANTS_THP_SWAP def_bool n +config PERSISTENT_HUGE_ZERO_FOLIO + bool "Allocate a PMD sized folio for zeroing" + depends on TRANSPARENT_HUGEPAGE + help + Enable this option to reduce the runtime refcounting overhead + of the huge zero folio and expand the places in the kernel + that can use huge zero folios. This can potentially improve + the performance while performing an I/O. + + With this option enabled, the huge zero folio is allocated + once and never freed. One full huge page worth of memory shall + be used. + + Say Y if your system has lots of memory. Say N if you are + memory constrained. + config MM_ID def_bool n
mm/huge_memory.c +30 -10
@@ -248,6 +248,9 @@ static void put_huge_zero_folio(void)struct folio *mm_get_huge_zero_folio (struct mm_struct *mm) { +if (IS_ENABLED (CONFIG_PERSISTENT_HUGE_ZERO_FOLIO )) +return huge_zero_folio; +if (test_bit (MMF_HUGE_ZERO_FOLIO , &mm->flags))return READ_ONCE (huge_zero_folio);
@@ -262,6 +265,9 @@ struct folio *mm_get_huge_zero_folio(struct mm_struct *mm)void mm_put_huge_zero_folio (struct mm_struct *mm) { +if (IS_ENABLED (CONFIG_PERSISTENT_HUGE_ZERO_FOLIO )) +return ; +if (test_bit (MMF_HUGE_ZERO_FOLIO , &mm->flags))put_huge_zero_folio (); }
@@ -849,16 +855,34 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj)static int __init thp_shrinker_init (void ) { - huge_zero_folio_shrinker =shrinker_alloc (0 ,"thp-zero" ); -if (!huge_zero_folio_shrinker) -return -ENOMEM ; - deferred_split_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE |SHRINKER_MEMCG_AWARE |SHRINKER_NONSLAB ,"thp-deferred_split" ); -if (!deferred_split_shrinker) { -shrinker_free (huge_zero_folio_shrinker); +if (!deferred_split_shrinker) +return -ENOMEM ; + + deferred_split_shrinker->count_objects = deferred_split_count; + deferred_split_shrinker->scan_objects = deferred_split_scan; +shrinker_register (deferred_split_shrinker); + +if (IS_ENABLED (CONFIG_PERSISTENT_HUGE_ZERO_FOLIO )) { +/* +* Bump the reference of the huge_zero_folio and do not +* initialize the shrinker. +* +* huge_zero_folio will always be NULL on failure. We assume +* that get_huge_zero_folio() will most likely not fail as +* thp_shrinker_init() is invoked early on during boot. +*/ +if (!get_huge_zero_folio ()) +pr_warn ("Allocating static huge zero folio failed\n" ); +return 0 ; + } + + huge_zero_folio_shrinker =shrinker_alloc (0 ,"thp-zero" ); +if (!huge_zero_folio_shrinker) { +shrinker_free (deferred_split_shrinker);return -ENOMEM ; }
@@ -866,10 +890,6 @@ static int __init thp_shrinker_init(void) huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan;shrinker_register (huge_zero_folio_shrinker); - deferred_split_shrinker->count_objects = deferred_split_count; - deferred_split_shrinker->scan_objects = deferred_split_scan; -shrinker_register (deferred_split_shrinker); -return 0 ; }
Signature · 1 line
2.49.0
Re: [PATCH v2 3/5] mm: add persistent huge zero folio
mm/huge_memory.c static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj) · line 879
Quoted 2 lines
mm/huge_memory.c +2
-0+if (!get_huge_zero_folio ()) +pr_warn ("Allocating static huge zero folio failed\n" );
Oops, forgot to do s/static/persistent/ here.
I can fold this in the next version after receiving the comments.
Re: [PATCH v2 3/5] mm: add persistent huge zero folio
In reply to Pankaj Raghav (Samsung)Quoted 36 lines · Pankaj Raghav (Samsung)
From: Pankaj Raghav <[email protected]>
Many places in the kernel need to zero out larger chunks, but the maximum segment that can be zeroed out at a time by
ZERO_PAGEis limited byPAGE_SIZE.This is especially annoying in block devices and filesystems where multiple
ZERO_PAGEsare attached to the bio in different bvecs. With multipage bvec support in block layer, it is much more efficient to send out larger zero pages as a part of single bvec.This concern was raised during the review of adding Large Block Size support to XFS[1][2].
Usually
huge_zero_foliois allocated on demand, and it will be deallocated by the shrinker if there are no users of it left. At moment,huge_zero_folioinfrastructure refcount is tied to the process lifetime that created it. This might not work for bio layer as the completions can be async and the process that created thehuge_zero_foliomight no longer be alive. And, one of the main points that came up during discussion is to have something bigger than zero page as a drop-in replacement.Add a config option
PERSISTENT_HUGE_ZERO_FOLIOthat will result in allocating the huge zero folio during early init and never free the memory by disabling the shrinker. This makes using thehuge_zero_foliowithout having to pass any mm struct and does not tie the lifetime of the zero folio to anything, making it a drop-in replacement forZERO_PAGE.If
PERSISTENT_HUGE_ZERO_FOLIOconfig option is enabled, thenmm_get_huge_zero_folio()will simply return the allocated page instead of dynamically allocating a new PMD page.Use this option carefully in resource constrained systems as it uses one full PMD sized page for zeroing purposes.
Quoted 6 lines
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/[email protected]/Co-developed-by David Hildenbrand <[email protected]>Signed-off-by David Hildenbrand <[email protected]>Signed-off-by Pankaj Raghav <[email protected]>
This is much nicer and now _super_ simple, I like it.
A few nits below but generally:
Quoted 45 lines
--- include/linux/huge_mm.h | 16 ++++++++++++++++ mm/Kconfig | 16 ++++++++++++++++ mm/huge_memory.c | 40 ++++++++++++++++++++++++++++++---------- 3 files changed, 62 insertions(+), 10 deletions(-)include/linux/huge_mm.h +16
-0@@ -495,6 +495,17 @@ static inline bool is_huge_zero_pmd(pmd_t pmd)struct folio *mm_get_huge_zero_folio (struct mm_struct *mm);void mm_put_huge_zero_folio (struct mm_struct *mm); +static inline struct folio *get_persistent_huge_zero_folio (void ) +{ +if (!IS_ENABLED (CONFIG_PERSISTENT_HUGE_ZERO_FOLIO )) +return NULL ; + +if (unlikely (!huge_zero_folio)) +return NULL ; + +return huge_zero_folio; +} +static inline bool thp_migration_supported (void ) {return IS_ENABLED (CONFIG_ARCH_ENABLE_THP_MIGRATION );@@ -685,6 +696,11 @@ static inline int change_huge_pud(struct mmu_gather *tlb, {return 0 ; } + +static inline struct folio *get_persistent_huge_zero_folio (void ) +{ +return NULL ; +}#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline int split_folio_to_list_to_order(struct folio *folio,mm/Kconfig +0
-0
mm/Kconfig config ARCH_WANT_GENERAL_HUGETLB · line 828
Quoted 6 lines
mm/Kconfig +3
-0@@ -823,6 +823,22 @@ config ARCH_WANT_GENERAL_HUGETLB config ARCH_WANTS_THP_SWAP def_bool n +config PERSISTENT_HUGE_ZERO_FOLIO + bool "Allocate a PMD sized folio for zeroing" + depends on TRANSPARENT_HUGEPAGE
I feel like we really need to sort out what is/isn't predicated on THP... it seems like THP is sort of short hand for 'any large folio stuff' but not always...
But this is a more general point :)
mm/Kconfig config ARCH_WANT_GENERAL_HUGETLB · line 833
Quoted 5 lines
mm/Kconfig +5
-0+ help + Enable this option to reduce the runtime refcounting overhead + of the huge zero folio and expand the places in the kernel + that can use huge zero folios. This can potentially improve + the performance while performing an I/O.
NIT: I think we can drop 'an', and probably refactor this sentence to something like 'For instance, block I/O benefits from access to large folios for zeroing memory'.
mm/Kconfig config ARCH_WANT_GENERAL_HUGETLB · line 837
Quoted 4 lines
mm/Kconfig +4
-0+ + With this option enabled, the huge zero folio is allocated + once and never freed. One full huge page worth of memory shall + be used.
NIT: huge page worth -> huge page's worth
Quoted 86 lines
mm/Kconfig +4
-0+ + Say Y if your system has lots of memory. Say N if you are + memory constrained. + config MM_ID def_bool nmm/huge_memory.c +30
-11@@ -248,6 +248,9 @@ static void put_huge_zero_folio(void)struct folio *mm_get_huge_zero_folio (struct mm_struct *mm) { +if (IS_ENABLED (CONFIG_PERSISTENT_HUGE_ZERO_FOLIO )) +return huge_zero_folio; +if (test_bit (MMF_HUGE_ZERO_FOLIO , &mm->flags))return READ_ONCE (huge_zero_folio);@@ -262,6 +265,9 @@ struct folio *mm_get_huge_zero_folio(struct mm_struct *mm)void mm_put_huge_zero_folio (struct mm_struct *mm) { +if (IS_ENABLED (CONFIG_PERSISTENT_HUGE_ZERO_FOLIO )) +return ; +if (test_bit (MMF_HUGE_ZERO_FOLIO , &mm->flags))put_huge_zero_folio (); }@@ -849,16 +855,34 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj)static int __init thp_shrinker_init (void ) { - huge_zero_folio_shrinker =shrinker_alloc (0 ,"thp-zero" ); -if (!huge_zero_folio_shrinker) -return -ENOMEM ; - deferred_split_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE |SHRINKER_MEMCG_AWARE |SHRINKER_NONSLAB ,"thp-deferred_split" ); -if (!deferred_split_shrinker) { -shrinker_free (huge_zero_folio_shrinker); +if (!deferred_split_shrinker) +return -ENOMEM ; + + deferred_split_shrinker->count_objects = deferred_split_count; + deferred_split_shrinker->scan_objects = deferred_split_scan; +shrinker_register (deferred_split_shrinker); + +if (IS_ENABLED (CONFIG_PERSISTENT_HUGE_ZERO_FOLIO )) { +/* +* Bump the reference of the huge_zero_folio and do not +* initialize the shrinker. +* +* huge_zero_folio will always be NULL on failure. We assume +* that get_huge_zero_folio() will most likely not fail as +* thp_shrinker_init() is invoked early on during boot. +*/ +if (!get_huge_zero_folio ()) +pr_warn ("Allocating static huge zero folio failed\n" ); +return 0 ; + } + + huge_zero_folio_shrinker =shrinker_alloc (0 ,"thp-zero" ); +if (!huge_zero_folio_shrinker) { +shrinker_free (deferred_split_shrinker);return -ENOMEM ; }@@ -866,10 +890,6 @@ static int __init thp_shrinker_init(void) huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan;shrinker_register (huge_zero_folio_shrinker); - deferred_split_shrinker->count_objects = deferred_split_count; - deferred_split_shrinker->scan_objects = deferred_split_scan; -shrinker_register (deferred_split_shrinker); -return 0 ; } --2.49.0
Re: [PATCH v2 3/5] mm: add persistent huge zero folio
Quoted 1 line
This is much nicer and now _super_ simple, I like it.
Thanks to you and David :)
Quoted 4 lines
A few nits below but generally:
Reviewed-by Lorenzo Stoakes <[email protected]>
Thanks.
Quoted 24 lines
Quoted 13 lines
--- include/linux/huge_mm.h | 16 ++++++++++++++++ mm/Kconfig | 16 ++++++++++++++++ mm/huge_memory.c | 40 ++++++++++++++++++++++++++++++---------- 3 files changed, 62 insertions(+), 10 deletions(-)static inline int split_folio_to_list_to_order(struct folio *folio,
mm/Kconfig +0
-0mm/Kconfig config ARCH_WANT_GENERAL_HUGETLB · line 828
Quoted 6 lines
mm/Kconfig +3
-0@@ -823,6 +823,22 @@ config ARCH_WANT_GENERAL_HUGETLB config ARCH_WANTS_THP_SWAP def_bool n +config PERSISTENT_HUGE_ZERO_FOLIO + bool "Allocate a PMD sized folio for zeroing" + depends on TRANSPARENT_HUGEPAGEI feel like we really need to sort out what is/isn't predicated on THP... it seems like THP is sort of short hand for 'any large folio stuff' but not always...
Quoted 1 line
But this is a more general point :)
I already brought this topic once during THP cabal. I am thinking of submitting a talk about this topic for LPC Memory Management MC.
Quoted 11 lines
mm/Kconfig config ARCH_WANT_GENERAL_HUGETLB · line 833
Quoted 6 lines
mm/Kconfig +5
-0+ help + Enable this option to reduce the runtime refcounting overhead + of the huge zero folio and expand the places in the kernel + that can use huge zero folios. This can potentially improve + the performance while performing an I/O.NIT: I think we can drop 'an', and probably refactor this sentence to something like 'For instance, block I/O benefits from access to large folios for zeroing memory'.
Quoted 7 lines
mm/Kconfig config ARCH_WANT_GENERAL_HUGETLB · line 837
Quoted 4 lines
mm/Kconfig +4
-0+ + With this option enabled, the huge zero folio is allocated + once and never freed. One full huge page worth of memory shall + be used.NIT: huge page worth -> huge page's worth
Thanks for the comments. I will make those changes and send a new version.
--
Pankaj
Re: [PATCH v2 3/5] mm: add persistent huge zero folio
In reply to Pankaj Raghav (Samsung)Quoted 39 lines · Pankaj Raghav (Samsung)
From: Pankaj Raghav <[email protected]>
Many places in the kernel need to zero out larger chunks, but the maximum segment that can be zeroed out at a time by
ZERO_PAGEis limited byPAGE_SIZE.This is especially annoying in block devices and filesystems where multiple
ZERO_PAGEsare attached to the bio in different bvecs. With multipage bvec support in block layer, it is much more efficient to send out larger zero pages as a part of single bvec.This concern was raised during the review of adding Large Block Size support to XFS[1][2].
Usually
huge_zero_foliois allocated on demand, and it will be deallocated by the shrinker if there are no users of it left. At moment,huge_zero_folioinfrastructure refcount is tied to the process lifetime that created it. This might not work for bio layer as the completions can be async and the process that created thehuge_zero_foliomight no longer be alive. And, one of the main points that came up during discussion is to have something bigger than zero page as a drop-in replacement.Add a config option
PERSISTENT_HUGE_ZERO_FOLIOthat will result in allocating the huge zero folio during early init and never free the memory by disabling the shrinker. This makes using thehuge_zero_foliowithout having to pass any mm struct and does not tie the lifetime of the zero folio to anything, making it a drop-in replacement forZERO_PAGE.If
PERSISTENT_HUGE_ZERO_FOLIOconfig option is enabled, thenmm_get_huge_zero_folio()will simply return the allocated page instead of dynamically allocating a new PMD page.Use this option carefully in resource constrained systems as it uses one full PMD sized page for zeroing purposes.
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/[email protected]/
Quoted 9 lines
Co-developed-by David Hildenbrand <[email protected]>Signed-off-by David Hildenbrand <[email protected]>Signed-off-by Pankaj Raghav <[email protected]>--- include/linux/huge_mm.h | 16 ++++++++++++++++ mm/Kconfig | 16 ++++++++++++++++ mm/huge_memory.c | 40 ++++++++++++++++++++++++++++++---------- 3 files changed, 62 insertions(+), 10 deletions(-)
As mentioned, I really would like to have a kernel commandline parameter for disabling huge zero folio.
Otherwise:
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH v2 4/5 mm: add largest_zero_folio() routine
In reply to Pankaj Raghav (Samsung)From: Pankaj Raghav <[email protected]>
The callers of mm_get_huge_zero_folio() have access to a mm struct and the lifetime of the huge_zero_folio is tied to the lifetime of the mm struct.
largest_zero_folio() will give access to huge_zero_folio when PERSISTENT_HUGE_ZERO_FOLIO config option is enabled for callers that do not want to tie the lifetime to a mm struct. This is very useful for filesystem and block layers where the request completions can be async and there is no guarantee on the mm struct lifetime.
This function will return a ZERO_PAGE folio if PERSISTENT_HUGE_ZERO_FOLIO is disabled or if we failed to allocate a huge_zero_folio during early init.
--- include/linux/huge_mm.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
include/linux/huge_mm.h +22 -0
@@ -714,4 +714,26 @@ static inline int split_folio_to_order(struct folio *folio, int new_order) return split_folio_to_list_to_order(folio, NULL, new_order); } +/** +* largest_zero_folio - Get the largest zero size folio available +* +* This function shall be used when mm_get_huge_zero_folio() cannot be +* used as there is no appropriate mm lifetime to tie the huge zero folio +* from the caller. +* +* Deduce the size of the folio with folio_size instead of assuming the +* folio size. +* +* Return: pointer to PMD sized zero folio if CONFIG_PERSISTENT_HUGE_ZERO_FOLIO +* is enabled or a single page sized zero folio +*/ +static inline struct folio *largest_zero_folio (void ) +{ +struct folio *folio =get_persistent_huge_zero_folio (); + +if (folio) +return folio; + +return page_folio (ZERO_PAGE (0 )); +}#endif /* _LINUX_HUGE_MM_H */
Signature · 1 line
2.49.0
Re: [PATCH v2 4/5] mm: add largest_zero_folio() routine
Quoted 12 lines · Pankaj Raghav (Samsung)
From: Pankaj Raghav <[email protected]>
The callers of
mm_get_huge_zero_folio()have access to a mm struct and the lifetime of thehuge_zero_foliois tied to the lifetime of the mm struct.
largest_zero_folio()will give access tohuge_zero_foliowhenPERSISTENT_HUGE_ZERO_FOLIOconfig option is enabled for callers that do not want to tie the lifetime to a mm struct. This is very useful for filesystem and block layers where the request completions can be async and there is no guarantee on the mm struct lifetime.
Quoted 7 lines
This function will return a
ZERO_PAGEfolio ifPERSISTENT_HUGE_ZERO_FOLIOis disabled or if we failed to allocate ahuge_zero_folioduring early init.Co-developed-by David Hildenbrand <[email protected]>Signed-off-by David Hildenbrand <[email protected]>Signed-off-by Pankaj Raghav <[email protected]>
Hm thought I R-b this already :P
LGTM, so:
Quoted 38 lines
--- include/linux/huge_mm.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)include/linux/huge_mm.h +22
-1@@ -714,4 +714,26 @@ static inline int split_folio_to_order(struct folio *folio, int new_order) return split_folio_to_list_to_order(folio, NULL, new_order); } +/** +* largest_zero_folio - Get the largest zero size folio available +* +* This function shall be used when mm_get_huge_zero_folio() cannot be +* used as there is no appropriate mm lifetime to tie the huge zero folio +* from the caller. +* +* Deduce the size of the folio with folio_size instead of assuming the +* folio size. +* +* Return: pointer to PMD sized zero folio if CONFIG_PERSISTENT_HUGE_ZERO_FOLIO +* is enabled or a single page sized zero folio +*/ +static inline struct folio *largest_zero_folio (void ) +{ +struct folio *folio =get_persistent_huge_zero_folio (); + +if (folio) +return folio; + +return page_folio (ZERO_PAGE (0 )); +}#endif /* _LINUX_HUGE_MM_H */ --2.49.0
Re: [PATCH v2 4/5] mm: add largest_zero_folio() routine
In reply to Pankaj Raghav (Samsung)Quoted 16 lines · Pankaj Raghav (Samsung)
From: Pankaj Raghav <[email protected]>
The callers of
mm_get_huge_zero_folio()have access to a mm struct and the lifetime of thehuge_zero_foliois tied to the lifetime of the mm struct.
largest_zero_folio()will give access tohuge_zero_foliowhenPERSISTENT_HUGE_ZERO_FOLIOconfig option is enabled for callers that do not want to tie the lifetime to a mm struct. This is very useful for filesystem and block layers where the request completions can be async and there is no guarantee on the mm struct lifetime.This function will return a
ZERO_PAGEfolio ifPERSISTENT_HUGE_ZERO_FOLIOis disabled or if we failed to allocate ahuge_zero_folioduring early init.
Quoted 7 lines
Co-developed-by David Hildenbrand <[email protected]>Signed-off-by David Hildenbrand <[email protected]>Signed-off-by Pankaj Raghav <[email protected]>--- include/linux/huge_mm.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH v2 5/5 block: use largest_zero_folio in __blkdev_issue_zero_pages()
In reply to Pankaj Raghav (Samsung)From: Pankaj Raghav <[email protected]>
Use largest_zero_folio() in __blkdev_issue_zero_pages().
On systems with CONFIG_PERSISTENT_HUGE_ZERO_FOLIO enabled, we will end up sending larger bvecs instead of multiple small ones.
Noticed a 4% increase in performance on a commercial NVMe SSD which does not support OP_WRITE_ZEROES. The device's MDTS was 128K. The performance gains might be bigger if the device supports bigger MDTS.
--- block/blk-lib.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
block/blk-lib.c +8 -7
@@ -196,6 +196,8 @@ static void __blkdev_issue_zero_pages(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned int flags) { + struct folio *zero_folio = largest_zero_folio(); +while (nr_sects) {unsigned int nr_vecs =__blkdev_sectors_to_bio_pages (nr_sects);struct bio *bio;
@@ -208,15 +210,14 @@ static void __blkdev_issue_zero_pages(struct block_device *bdev,break ;do { -unsigned int len, added; +unsigned int len; - len =min_t (sector_t, -PAGE_SIZE , nr_sects <<SECTOR_SHIFT ); - added =bio_add_page (bio,ZERO_PAGE (0 ), len,0 ); -if (added < len) + len =min_t (sector_t,folio_size (zero_folio), + nr_sects <<SECTOR_SHIFT ); +if (!bio_add_folio (bio, zero_folio, len,0 ))break ; - nr_sects -= added >>SECTOR_SHIFT ; - sector += added >>SECTOR_SHIFT ; + nr_sects -= len >>SECTOR_SHIFT ; + sector += len >>SECTOR_SHIFT ; }while (nr_sects); *biop =bio_chain_and_submit (*biop, bio);
Signature · 1 line
2.49.0
Re: [PATCH v2 5/5] block: use largest_zero_folio in __blkdev_issue_zero_pages()
Quoted 10 lines · Pankaj Raghav (Samsung)
From: Pankaj Raghav <[email protected]>
Use
largest_zero_folio()in__blkdev_issue_zero_pages().
On systems withCONFIG_PERSISTENT_HUGE_ZERO_FOLIOenabled, we will end up sending larger bvecs instead of multiple small ones.Noticed a 4% increase in performance on a commercial NVMe SSD which does not support
OP_WRITE_ZEROES. The device's MDTS was 128K. The performance gains might be bigger if the device supports bigger MDTS.
Quoted 6 lines
Acked-by Lorenzo Stoakes <[email protected]>Signed-off-by Pankaj Raghav <[email protected]>--- block/blk-lib.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich