Reclaim zero subpages of thp to avoid memory bloat
Subsystems (8)
cover Reclaim zero subpages of thp to avoid memory bloat
As we know, thp may lead to memory bloat which may cause OOM. Through testing with some apps, we found that the reason of memory bloat is a huge page may contain some zero subpages (may accessed or not). And we found that most zero subpages are centralized in a few huge pages.
Following is a text_classification_rnn case for tensorflow:
zero_subpages huge_pages waste [ 0, 1) 186 0.00% [ 1, 2) 23 0.01% [ 2, 4) 36 0.02% [ 4, 8) 67 0.08% [ 8, 16) 80 0.23% [ 16, 32) 109 0.61% [ 32, 64) 44 0.49% [ 64, 128) 12 0.30% [ 128, 256) 28 1.54%
[ 256, 513) 159 18.03%
In the case, there are 187 huge pages (25% of the total huge pages) which contain more then 128 zero subpages. And these huge pages lead to 19.57% waste of the total rss. It means we can reclaim 19.57% memory by splitting the 187 huge pages and reclaiming the zero subpages.
This patchset introduce a new mechanism to split the huge page which has zero subpages and reclaim these zero subpages.
We add the anonymous huge page to a list to reduce the cost of finding the huge page. When the memory reclaim is triggering, the list will be walked and the huge page contains enough zero subpages may be reclaimed. Meanwhile, replace the zero subpages by ZERO_PAGE(0).
Yu Zhao has done some similar work when the huge page is swap out or migrated to accelerate[1]. While we do this in the normal memory shrink path for the swapoff scene to avoid OOM.
In the future, we will do the proactive reclaim to reclaim the "cold" huge page proactively. This is for keeping the performance of thp as for as possible. In addition to that, some users want the memory usage using thp is equal to the usage using 4K.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Ning Zhang (6):
mm, thp: introduce thp zero subpages reclaim
mm, thp: add a global interface for zero subapges reclaim mm, thp: introduce zero subpages reclaim threshold mm, thp: introduce a controller to trigger zero subpages reclaim mm, thp: add some statistics for zero subpages reclaim mm, thp: add document for zero subpages reclaim
Documentation/admin-guide/mm/transhuge.rst | 75 ++++++ include/linux/huge_mm.h | 13 + include/linux/memcontrol.h | 26 ++ include/linux/mm.h | 1 + include/linux/mm_types.h | 6 + include/linux/mmzone.h | 9 + mm/huge_memory.c | 374 ++++++++++++++++++++++++++++- mm/memcontrol.c | 243 +++++++++++++++++++ mm/vmscan.c | 61 ++++- 9 files changed, 805 insertions(+), 3 deletions(-)
Signature · 1 line
1.8.3.1
RFC 1/6 mm, thp: introduce thp zero subpages reclaim
Transparent huge pages could reduce the number of tlb misses, which could improve performance for the applications. But one concern is that thp may lead to memory bloat which may cause OOM. The reason is a huge page may contain some zero subpages which user didn't really access them.
This patch introduces a mechanism to reclaim these zero subpages, it works when memory pressure is high. We'll estimate whether a huge page contains enough zero subpages at first, then try split it and reclaim the zero subpages.
Through testing with some apps, we found that the zero subpages are tended to be concentrated into a few huge pages. Following is a text_classification_rnn case for tensorflow:
zero_subpages huge_pages waste [ 0, 1) 186 0.00% [ 1, 2) 23 0.01% [ 2, 4) 36 0.02% [ 4, 8) 67 0.08% [ 8, 16) 80 0.23% [ 16, 32) 109 0.61% [ 32, 64) 44 0.49% [ 64, 128) 12 0.30% [ 128, 256) 28 1.54%
[ 256, 513) 159 18.03%
In the case, a lot of zero subpages are concentrated into 187(28+159) huge pages, which lead to 19.57% waste of the total rss. It means we can reclaim 19.57% memory by splitting the 187 huge pages and reclaiming the zero subpages.
We store the huge pages to a new list in order to find them quickly. And add a interface 'thp_reclaim' to control on or off in memory cgroup:
echo 1 > memory.thp_reclaim to enable.
echo 0 > memory.thp_reclaim to disable.
--- include/linux/huge_mm.h | 9 ++ include/linux/memcontrol.h | 15 +++ include/linux/mm.h | 1 + include/linux/mm_types.h | 6 + include/linux/mmzone.h | 6 + mm/huge_memory.c | 296 ++++++++++++++++++++++++++++++++++++++++++++- mm/memcontrol.c | 107 ++++++++++++++++ mm/vmscan.c | 59 ++++++++- 8 files changed, 496 insertions(+), 3 deletions(-)
include/linux/huge_mm.h +9 -0
@@ -185,6 +185,15 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,void free_transhuge_page (struct page *page);bool is_transparent_hugepage (struct page *page); +#ifdef CONFIG_MEMCG +int zsr_get_hpage (struct hpage_reclaim *hr_queue,struct page **reclaim_page); +unsigned long zsr_reclaim_hpage (struct lruvec *lruvec,struct page *page); +static inline struct list_head *hpage_reclaim_list (struct page *page) +{ +return &page[3 ].hpage_reclaim_list; +} +#endif +bool can_split_huge_page (struct page *page,int *pextra_pins);int split_huge_page_to_list (struct page *page,struct list_head *list);static inline int split_huge_page (struct page *page)
include/linux/memcontrol.h +15 -0
@@ -150,6 +150,9 @@ struct mem_cgroup_per_node {unsigned long usage_in_excess;/* Set to the value by which */ /* the soft limit is exceeded*/ bool on_tree; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +struct hpage_reclaim hpage_reclaim_queue; +#endif struct mem_cgroup *memcg;/* Back pointer, we cannot */ /* use container_of */ };
@@ -228,6 +231,13 @@ struct obj_cgroup { }; }; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +enum thp_reclaim_state { +THP_RECLAIM_DISABLE , +THP_RECLAIM_ENABLE , +THP_RECLAIM_MEMCG ,/* For global configure*/ +}; +#endif /* * The memory controller data structure. The memory controller controls both * page cache and RSS per cgroup. We would eventually like to provide
@@ -345,6 +355,7 @@ struct mem_cgroup {#ifdef CONFIG_TRANSPARENT_HUGEPAGE struct deferred_split deferred_split_queue; +int thp_reclaim;#endif struct mem_cgroup_per_node *nodeinfo[];
@@ -1110,6 +1121,10 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,gfp_t gfp_mask, unsigned long *total_scanned); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +void del_hpage_from_queue (struct page *page); +#endif +#else /* CONFIG_MEMCG */ #define MEM_CGROUP_ID_SHIFT 0
include/linux/mm.h +1 -0
@@ -3061,6 +3061,7 @@ int drop_caches_sysctl_handler(struct ctl_table *, int, void *, size_t *,void drop_slab (void );void drop_slab_node (int nid); +unsigned int move_pages_to_lru (struct lruvec *lruvec,struct list_head *list);#ifndef CONFIG_MMU #define randomize_va_space 0
include/linux/mm_types.h +6 -0
@@ -159,6 +159,12 @@ struct page {/* For both global and memcg */ struct list_head deferred_list; }; +struct {/* Third tail page of compound page */ +unsigned long _compound_pad_2; +unsigned long _compound_pad_3; +/* For zero subpages reclaim */ +struct list_head hpage_reclaim_list; + };struct {/* Page table pages */ unsigned long _pt_pad_1;/* compound_head */ pgtable_t pmd_huge_pte;/* protected by page->ptl */
include/linux/mmzone.h +6 -0
@@ -787,6 +787,12 @@ struct deferred_split {struct list_head split_queue; unsigned long split_queue_len; }; + +struct hpage_reclaim { +spinlock_t reclaim_queue_lock; +struct list_head reclaim_queue; +unsigned long reclaim_queue_len; +};#endif /*
mm/huge_memory.c +295 -1
323 lines not on this page · raw message
mm/memcontrol.c +107 -0
173 lines not on this page · raw message
mm/vmscan.c +57 -2
77 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [RFC 1/6] mm, thp: introduce thp zero subpages reclaim
include/linux/huge_mm.h unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, · line 195
Quoted 13 lines · Ning Zhang
+++
b/include/linux/huge_mm.hinclude/linux/huge_mm.h +8
-0@@ -185,6 +185,15 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,void free_transhuge_page (struct page *page);bool is_transparent_hugepage (struct page *page); +#ifdef CONFIG_MEMCG +int zsr_get_hpage (struct hpage_reclaim *hr_queue,struct page **reclaim_page); +unsigned long zsr_reclaim_hpage (struct lruvec *lruvec,struct page *page); +static inline struct list_head *hpage_reclaim_list (struct page *page) +{ +return &page[3 ].hpage_reclaim_list; +} +#endif
I don't think any of this needs to be under an ifdef. That goes for a lot of your other additions to header files.
include/linux/memcontrol.h unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, · line 1126
Quoted 7 lines
include/linux/memcontrol.h +3
-0@@ -1110,6 +1121,10 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,gfp_t gfp_mask, unsigned long *total_scanned); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +void del_hpage_from_queue (struct page *page); +#endif
That name is too generic. Also, to avoid ifdefs in code, it should be:
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
void del_hpage_from_queue(struct page *page);
#else
static inline void del_hpage_from_queue(struct page *page) { } #endif
include/linux/mm_types.h struct page { · line 166
Quoted 9 lines
include/linux/mm_types.h +5
-0@@ -159,6 +159,12 @@ struct page {/* For both global and memcg */ struct list_head deferred_list; }; +struct {/* Third tail page of compound page */ +unsigned long _compound_pad_2; +unsigned long _compound_pad_3; +/* For zero subpages reclaim */ +struct list_head hpage_reclaim_list;
Why do you need _compound_pad_3 here?
include/linux/mmzone.h struct deferred_split { · line 795
Quoted 11 lines
include/linux/mmzone.h +7
-0+++ b/include/linux/mmzone.h@@ -787,6 +787,12 @@ struct deferred_split {struct list_head split_queue;unsigned long split_queue_len; }; + +struct hpage_reclaim { +spinlock_t reclaim_queue_lock; +struct list_head reclaim_queue; +unsigned long reclaim_queue_len; +};
Have you considered using an XArray instead of a linked list?
mm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · line 3234
Quoted 6 lines
mm/huge_memory.c +6
-0+static bool hpage_estimate_zero (struct page *page) +{ +unsigned int i, maybe_zero_pages =0 , offset =0 ; +void *addr; + +#define BYTES_PER_LONG (BITS_PER_LONG / BITS_PER_BYTE)
BYTES_PER_LONG is simply sizeof(long).
Also, I'd check the entire cacheline rather than just one word; it's essentially free.
mm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · line 3368
Quoted 5 lines
mm/huge_memory.c +5
-0+#ifdef CONFIG_MMU +#define ZSR_PG_MLOCK (flag) (1UL << flag) +#else +#define ZSR_PG_MLOCK (flag) 0 +#endif
Or use __PG_MLOCKED ?
mm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · line 3374
Quoted 5 lines
mm/huge_memory.c +5
-0+#ifdef CONFIG_ARCH_USES_PG_UNCACHED +#define ZSR_PG_UNCACHED (flag) (1UL << flag) +#else +#define ZSR_PG_UNCACHED (flag) 0 +#endif
Define __PG_UNCACHED in page-flags.h?
mm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · line 3380
Quoted 5 lines
mm/huge_memory.c +5
-0+#ifdef CONFIG_MEMORY_FAILURE +#define ZSR_PG_HWPOISON (flag) (1UL << flag) +#else +#define ZSR_PG_HWPOISON (flag) 0 +#endif
__PG_HWPOISON
mm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · line 3404
Quoted 3 lines
mm/huge_memory.c +3
-0+#define hr_queue_list_to_page (head) \ + compound_head(list_entry((head)->prev, struct page,\ + hpage_reclaim_list))
I think you're better off subtracting 3*sizeof(struct page) than loading from compound_head.
mm/memcontrol.c static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) · line 2752
Quoted 11 lines
mm/memcontrol.c +11
-0+#ifdef CONFIG_TRANSPARENT_HUGEPAGE +/* Need the page lock if the page is not a newly allocated page. */ +static void add_hpage_to_queue (struct page *page,struct mem_cgroup *memcg) +{ +struct hpage_reclaim *hr_queue; +unsigned long flags; + +if (READ_ONCE (memcg->thp_reclaim) ==THP_RECLAIM_DISABLE ) +return ; + + page =compound_head (page);
Why do you think the caller might be passing in a tail page here?
Re: [RFC 1/6] mm, thp: introduce thp zero subpages reclaim
Quoted 93 lines · Matthew Wilcox
include/linux/huge_mm.h unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, · line 195
Quoted 13 lines · Ning Zhang
+++
b/include/linux/huge_mm.hinclude/linux/huge_mm.h +8
-0@@ -185,6 +185,15 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,void free_transhuge_page (struct page *page);bool is_transparent_hugepage (struct page *page); +#ifdef CONFIG_MEMCG +int zsr_get_hpage (struct hpage_reclaim *hr_queue,struct page **reclaim_page); +unsigned long zsr_reclaim_hpage (struct lruvec *lruvec,struct page *page); +static inline struct list_head *hpage_reclaim_list (struct page *page) +{ +return &page[3 ].hpage_reclaim_list; +} +#endif I don't think any of this needs to be under an ifdef. That goes for a lot of your other additions to header files.
include/linux/memcontrol.h unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, · line 1126
Quoted 7 lines
include/linux/memcontrol.h +3
-0@@ -1110,6 +1121,10 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,gfp_t gfp_mask, unsigned long *total_scanned); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +void del_hpage_from_queue (struct page *page); +#endif That name is too generic. Also, to avoid ifdefs in code, it should be:
#ifdef
CONFIG_TRANSPARENT_HUGEPAGE
voiddel_hpage_from_queue(struct page *page);
#else
static inline voiddel_hpage_from_queue(struct page *page){ } #endifinclude/linux/mm_types.h struct page { · line 166
Quoted 9 lines
include/linux/mm_types.h +5
-0@@ -159,6 +159,12 @@ struct page {/* For both global and memcg */ struct list_head deferred_list; }; +struct {/* Third tail page of compound page */ +unsigned long _compound_pad_2; +unsigned long _compound_pad_3; +/* For zero subpages reclaim */ +struct list_head hpage_reclaim_list;Why do you need
_compound_pad_3here?include/linux/mmzone.h struct deferred_split { · line 795
Quoted 11 lines
include/linux/mmzone.h +7
-0+++ b/include/linux/mmzone.h@@ -787,6 +787,12 @@ struct deferred_split {struct list_head split_queue;unsigned long split_queue_len; }; + +struct hpage_reclaim { +spinlock_t reclaim_queue_lock; +struct list_head reclaim_queue; +unsigned long reclaim_queue_len; +};Have you considered using an XArray instead of a linked list?
mm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · line 3234
Quoted 6 lines
mm/huge_memory.c +6
-0+static bool hpage_estimate_zero (struct page *page) +{ +unsigned int i, maybe_zero_pages =0 , offset =0 ; +void *addr; + +#define BYTES_PER_LONG (BITS_PER_LONG / BITS_PER_BYTE)
BYTES_PER_LONGis simply sizeof(long).
Also, I'd check the entire cacheline rather than just one word; it's essentially free.mm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · line 3368
Quoted 5 lines
mm/huge_memory.c +5
-0+#ifdef CONFIG_MMU +#define ZSR_PG_MLOCK (flag) (1UL << flag) +#else +#define ZSR_PG_MLOCK (flag) 0 +#endif Or use
__PG_MLOCKED?mm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · line 3374
Quoted 5 lines
mm/huge_memory.c +5
-0+#ifdef CONFIG_ARCH_USES_PG_UNCACHED +#define ZSR_PG_UNCACHED (flag) (1UL << flag) +#else +#define ZSR_PG_UNCACHED (flag) 0 +#endif Define
__PG_UNCACHEDinpage-flags.h?mm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · line 3380
Quoted 5 lines
mm/huge_memory.c +5
-0+#ifdef CONFIG_MEMORY_FAILURE +#define ZSR_PG_HWPOISON (flag) (1UL << flag) +#else +#define ZSR_PG_HWPOISON (flag) 0 +#endif
__PG_HWPOISONmm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · line 3404
Quoted 3 lines
mm/huge_memory.c +3
-0+#define hr_queue_list_to_page (head) \ + compound_head(list_entry((head)->prev, struct page,\ + hpage_reclaim_list))I think you're better off subtracting 3*sizeof(struct page) than loading from
compound_head.
Quoted 12 lines
mm/memcontrol.c static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) · line 2752
Quoted 11 lines
mm/memcontrol.c +11
-0+#ifdef CONFIG_TRANSPARENT_HUGEPAGE +/* Need the page lock if the page is not a newly allocated page. */ +static void add_hpage_to_queue (struct page *page,struct mem_cgroup *memcg) +{ +struct hpage_reclaim *hr_queue; +unsigned long flags; + +if (READ_ONCE (memcg->thp_reclaim) ==THP_RECLAIM_DISABLE ) +return ; + + page =compound_head (page);Why do you think the caller might be passing in a tail page here?
Thanks for the comments! I will modify it.
kernel test robot
build botRe: [RFC 1/6] mm, thp: introduce thp zero subpages reclaim
In reply to Ning Zhang[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Ning,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on linus/master]
[also build test ERROR on v5.15-rc7]
[cannot apply to hnaz-mm/master next-20211028]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Ning-Zhang/Reclaim-zero-subpages-of-thp-to-avoid-memory-bloat/20211028-200001 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 1fc596a56b334f4d593a2b49e5ff55af6aaa0816 config: arm-randconfig-c002-20211028 (attached as .config) compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 5db7568a6a1fcb408eb8988abdaff2a225a8eb72)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install arm cross compiling tool for clang build
# apt-get install binutils-arm-linux-gnueabi
# https://github.com/0day-ci/linux/commit/ba9f8c1a43c2d9ab2d2ac5696aaffbeaf043fa02
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Ning-Zhang/Reclaim-zero-subpages-of-thp-to-avoid-memory-bloat/20211028-200001 git checkout ba9f8c1a43c2d9ab2d2ac5696aaffbeaf043fa02
# save the attached .config to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=arm SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
mm/vmscan.c:1340:6: warning: variable 'err' set but not used [-Wunused-but-set-variable]
int err;
^
>> mm/vmscan.c:2803:36: error: no member named 'hpage_reclaim_queue' in 'struct mem_cgroup_per_node'
hr_queue = &memcg->nodeinfo[nid]->hpage_reclaim_queue;
~~~~~~~~~~~~~~~~~~~~ ^
mm/vmscan.c:2804:24: error: no member named 'thp_reclaim' in 'struct mem_cgroup'
if (!READ_ONCE(memcg->thp_reclaim))
~~~~~ ^
include/asm-generic/rwonce.h:49:33: note: expanded from macro 'READ_ONCE'
compiletime_assert_rwonce_type(x); \
^
include/asm-generic/rwonce.h:36:35: note: expanded from macro 'compiletime_assert_rwonce_type'
compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
^
include/linux/compiler_types.h:290:10: note: expanded from macro '__native_word'
(sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || \
^
include/linux/compiler_types.h:322:22: note: expanded from macro 'compiletime_assert'
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
^~~~~~~~~
include/linux/compiler_types.h:310:23: note: expanded from macro '_compiletime_assert'
__compiletime_assert(condition, msg, prefix, suffix)
^~~~~~~~~
include/linux/compiler_types.h:302:9: note: expanded from macro '__compiletime_assert'
if (!(condition)) \
^~~~~~~~~
mm/vmscan.c:2804:24: error: no member named 'thp_reclaim' in 'struct mem_cgroup'
if (!READ_ONCE(memcg->thp_reclaim))
~~~~~ ^
include/asm-generic/rwonce.h:49:33: note: expanded from macro 'READ_ONCE'
compiletime_assert_rwonce_type(x); \
^
include/asm-generic/rwonce.h:36:35: note: expanded from macro 'compiletime_assert_rwonce_type'
compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
^
include/linux/compiler_types.h:290:39: note: expanded from macro '__native_word'
(sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || \
^
include/linux/compiler_types.h:322:22: note: expanded from macro 'compiletime_assert'
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
^~~~~~~~~
include/linux/compiler_types.h:310:23: note: expanded from macro '_compiletime_assert'
__compiletime_assert(condition, msg, prefix, suffix)
^~~~~~~~~
include/linux/compiler_types.h:302:9: note: expanded from macro '__compiletime_assert'
if (!(condition)) \
^~~~~~~~~
mm/vmscan.c:2804:24: error: no member named 'thp_reclaim' in 'struct mem_cgroup'
if (!READ_ONCE(memcg->thp_reclaim))
~~~~~ ^
include/asm-generic/rwonce.h:49:33: note: expanded from macro 'READ_ONCE'
compiletime_assert_rwonce_type(x); \
^
include/asm-generic/rwonce.h:36:35: note: expanded from macro 'compiletime_assert_rwonce_type'
compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
^
include/linux/compiler_types.h:291:10: note: expanded from macro '__native_word'
sizeof(t) == sizeof(int) || sizeof(t) == sizeof(long))
^
include/linux/compiler_types.h:322:22: note: expanded from macro 'compiletime_assert'
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
^~~~~~~~~
include/linux/compiler_types.h:310:23: note: expanded from macro '_compiletime_assert'
__compiletime_assert(condition, msg, prefix, suffix)
^~~~~~~~~
include/linux/compiler_types.h:302:9: note: expanded from macro '__compiletime_assert'
if (!(condition)) \
^~~~~~~~~
mm/vmscan.c:2804:24: error: no member named 'thp_reclaim' in 'struct mem_cgroup'
if (!READ_ONCE(memcg->thp_reclaim))
~~~~~ ^
include/asm-generic/rwonce.h:49:33: note: expanded from macro 'READ_ONCE'
compiletime_assert_rwonce_type(x); \
^
include/asm-generic/rwonce.h:36:35: note: expanded from macro 'compiletime_assert_rwonce_type'
compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
^
include/linux/compiler_types.h:291:38: note: expanded from macro '__native_word'
sizeof(t) == sizeof(int) || sizeof(t) == sizeof(long))
^
include/linux/compiler_types.h:322:22: note: expanded from macro 'compiletime_assert'
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
^~~~~~~~~
include/linux/compiler_types.h:310:23: note: expanded from macro '_compiletime_assert'
__compiletime_assert(condition, msg, prefix, suffix)
^~~~~~~~~
include/linux/compiler_types.h:302:9: note: expanded from macro '__compiletime_assert'
if (!(condition)) \
^~~~~~~~~
mm/vmscan.c:2804:24: error: no member named 'thp_reclaim' in 'struct mem_cgroup'
if (!READ_ONCE(memcg->thp_reclaim))
~~~~~ ^
include/asm-generic/rwonce.h:49:33: note: expanded from macro 'READ_ONCE'
compiletime_assert_rwonce_type(x); \
^
include/asm-generic/rwonce.h:36:48: note: expanded from macro 'compiletime_assert_rwonce_type'
compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
^
include/linux/compiler_types.h:322:22: note: expanded from macro 'compiletime_assert'
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
^~~~~~~~~
include/linux/compiler_types.h:310:23: note: expanded from macro '_compiletime_assert'
__compiletime_assert(condition, msg, prefix, suffix)
^~~~~~~~~
include/linux/compiler_types.h:302:9: note: expanded from macro '__compiletime_assert'
if (!(condition)) \
^~~~~~~~~
mm/vmscan.c:2804:24: error: no member named 'thp_reclaim' in 'struct mem_cgroup'
if (!READ_ONCE(memcg->thp_reclaim))
~~~~~ ^
include/asm-generic/rwonce.h:50:14: note: expanded from macro 'READ_ONCE'
__READ_ONCE(x); \
^
include/asm-generic/rwonce.h:44:65: note: expanded from macro '__READ_ONCE'
#define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))
^
include/linux/compiler_types.h:279:13: note: expanded from macro '__unqual_scalar_typeof'
_Generic((x), \
^
mm/vmscan.c:2804:24: error: no member named 'thp_reclaim' in 'struct mem_cgroup'
if (!READ_ONCE(memcg->thp_reclaim))
~~~~~ ^
include/asm-generic/rwonce.h:50:14: note: expanded from macro 'READ_ONCE'
__READ_ONCE(x); \
^
include/asm-generic/rwonce.h:44:65: note: expanded from macro '__READ_ONCE'
#define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))
^
include/linux/compiler_types.h:286:15: note: expanded from macro '__unqual_scalar_typeof'
default: (x)))
^
mm/vmscan.c:2804:24: error: no member named 'thp_reclaim' in 'struct mem_cgroup'
if (!READ_ONCE(memcg->thp_reclaim))
~~~~~ ^
include/asm-generic/rwonce.h:50:14: note: expanded from macro 'READ_ONCE'
__READ_ONCE(x); \
^
include/asm-generic/rwonce.h:44:72: note: expanded from macro '__READ_ONCE'
#define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))
^
mm/vmscan.c:2804:6: error: invalid argument type 'void' to unary expression
if (!READ_ONCE(memcg->thp_reclaim))
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> mm/vmscan.c:2813:7: error: implicit declaration of function 'zsr_get_hpage' [-Werror,-Wimplicit-function-declaration]
if (zsr_get_hpage(hr_queue, &page))
^
>> mm/vmscan.c:2819:19: error: implicit declaration of function 'zsr_reclaim_hpage' [-Werror,-Wimplicit-function-declaration]
nr_reclaimed += zsr_reclaim_hpage(lruvec, page);
^1 warning and 12 errors generated.
vim +2803 mm/vmscan.c
2784
2785 #ifdef CONFIG_MEMCG
2786 #define MAX_SCAN_HPAGE 32UL
2787 /*
2788 * Try to reclaim the zero subpages for the transparent huge page.
2789 */
2790 static unsigned long reclaim_hpage_zero_subpages(struct lruvec *lruvec,
2791 int priority,
2792 unsigned long nr_to_reclaim)
2793 {
2794 struct mem_cgroup *memcg;
2795 struct hpage_reclaim *hr_queue;
2796 int nid = lruvec->pgdat->node_id;
2797 unsigned long nr_reclaimed = 0, nr_scanned = 0, nr_to_scan;
2798
2799 memcg = lruvec_memcg(lruvec);
2800 if (!memcg)
2801 goto out;
2802
> 2803 hr_queue = &memcg->nodeinfo[nid]->hpage_reclaim_queue;
2804 if (!READ_ONCE(memcg->thp_reclaim))
2805 goto out;
2806
2807 /* The last scan loop will scan all the huge pages.*/
2808 nr_to_scan = priority == 0 ? 0 : MAX_SCAN_HPAGE;
2809
2810 do {
2811 struct page *page = NULL;
2812
> 2813 if (zsr_get_hpage(hr_queue, &page))
2814 break;
2815
2816 if (!page)
2817 continue;
2818
> 2819 nr_reclaimed += zsr_reclaim_hpage(lruvec, page); 2820
2821 cond_resched();
2822
2823 } while ((nr_reclaimed < nr_to_reclaim) && (++nr_scanned != nr_to_scan));
2824 out:
2825 return nr_reclaimed;
2826 }
2827 #else
2828 static unsigned long reclaim_hpage_zero_subpages(struct lruvec *lruvec,
2829 int priority,
2830 unsigned long nr_to_reclaim)
2831 {
2832 return 0;
2833 }
2834 #endif
2835
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
RFC 2/6 mm, thp: add a global interface for zero subapges reclaim
In reply to Ning ZhangAdd a global interface to configure zero subpages reclaim global:
/sys/kernel/mm/transparent_hugepage/reclaim
It has three modes:
memcg, means every memory cgroup will use their own configure. enable, means every mem cgroup will enable reclaim. disable, means every mem cgroup will disable reclaim.
The default mode is memcg.
--- include/linux/huge_mm.h | 1 + include/linux/memcontrol.h | 8 ++++++++ mm/huge_memory.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ mm/memcontrol.c | 2 +- mm/vmscan.c | 2 +- 5 files changed, 55 insertions(+), 2 deletions(-)
include/linux/huge_mm.h +1 -0
@@ -186,6 +186,7 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,bool is_transparent_hugepage (struct page *page);#ifdef CONFIG_MEMCG +extern int global_thp_reclaim;int zsr_get_hpage (struct hpage_reclaim *hr_queue,struct page **reclaim_page);unsigned long zsr_reclaim_hpage (struct lruvec *lruvec,struct page *page);static inline struct list_head *hpage_reclaim_list (struct page *page)
include/linux/memcontrol.h +8 -0
@@ -1123,6 +1123,14 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,#ifdef CONFIG_TRANSPARENT_HUGEPAGE void del_hpage_from_queue (struct page *page); + +static inline int get_thp_reclaim_mode (struct mem_cgroup *memcg) +{ +int reclaim =READ_ONCE (global_thp_reclaim); + +return (reclaim !=THP_RECLAIM_MEMCG ) ? reclaim : +READ_ONCE (memcg->thp_reclaim); +}#endif #else /* CONFIG_MEMCG */
mm/huge_memory.c +44 -0
62 lines not on this page · raw message
mm/memcontrol.c +1 -1
@@ -2746,7 +2746,7 @@ static void add_hpage_to_queue(struct page *page, struct mem_cgroup *memcg)struct hpage_reclaim *hr_queue;unsigned long flags; -if (READ_ONCE (memcg->thp_reclaim) ==THP_RECLAIM_DISABLE ) +if (get_thp_reclaim_mode (memcg) ==THP_RECLAIM_DISABLE )return ; page =compound_head (page);
mm/vmscan.c +1 -1
@@ -2801,7 +2801,7 @@ static unsigned long reclaim_hpage_zero_subpages(struct lruvec *lruvec, goto out; hr_queue = &memcg->nodeinfo[nid]->hpage_reclaim_queue; -if (!READ_ONCE (memcg->thp_reclaim)) +if (get_thp_reclaim_mode (memcg) ==THP_RECLAIM_DISABLE ) goto out;/* The last scan loop will scan all the huge pages.*/
Signature · 1 line
1.8.3.1
kernel test robot
build botRe: [RFC 2/6] mm, thp: add a global interface for zero subapges reclaim
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Ning,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on linus/master]
[also build test ERROR on v5.15-rc7]
[cannot apply to hnaz-mm/master next-20211028]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Ning-Zhang/Reclaim-zero-subpages-of-thp-to-avoid-memory-bloat/20211028-200001 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 1fc596a56b334f4d593a2b49e5ff55af6aaa0816 config: arm-randconfig-c002-20211028 (attached as .config) compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 5db7568a6a1fcb408eb8988abdaff2a225a8eb72)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install arm cross compiling tool for clang build
# apt-get install binutils-arm-linux-gnueabi
# https://github.com/0day-ci/linux/commit/4111289b6a222d9c2aeb7c593f0f52e0c38f3247
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Ning-Zhang/Reclaim-zero-subpages-of-thp-to-avoid-memory-bloat/20211028-200001 git checkout 4111289b6a222d9c2aeb7c593f0f52e0c38f3247
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 ARCH=arm
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
mm/vmscan.c:1340:6: warning: variable 'err' set but not used [-Wunused-but-set-variable]
int err;
^
mm/vmscan.c:2803:36: error: no member named 'hpage_reclaim_queue' in 'struct mem_cgroup_per_node'
hr_queue = &memcg->nodeinfo[nid]->hpage_reclaim_queue;
~~~~~~~~~~~~~~~~~~~~ ^
>> mm/vmscan.c:2804:6: error: implicit declaration of function 'get_thp_reclaim_mode' [-Werror,-Wimplicit-function-declaration]
if (get_thp_reclaim_mode(memcg) == THP_RECLAIM_DISABLE)
^
>> mm/vmscan.c:2804:37: error: use of undeclared identifier 'THP_RECLAIM_DISABLE'; did you mean 'WB_RECLAIMABLE'?
if (get_thp_reclaim_mode(memcg) == THP_RECLAIM_DISABLE)
^~~~~~~~~~~~~~~~~~~
WB_RECLAIMABLE
include/linux/backing-dev-defs.h:37:2: note: 'WB_RECLAIMABLE' declared here
WB_RECLAIMABLE,
^
mm/vmscan.c:2813:7: error: implicit declaration of function 'zsr_get_hpage' [-Werror,-Wimplicit-function-declaration]
if (zsr_get_hpage(hr_queue, &page))
^
mm/vmscan.c:2819:19: error: implicit declaration of function 'zsr_reclaim_hpage' [-Werror,-Wimplicit-function-declaration]
nr_reclaimed += zsr_reclaim_hpage(lruvec, page);
^1 warning and 5 errors generated.
vim +/get_thp_reclaim_mode +2804 mm/vmscan.c
2784
2785 #ifdef CONFIG_MEMCG
2786 #define MAX_SCAN_HPAGE 32UL
2787 /*
2788 * Try to reclaim the zero subpages for the transparent huge page.
2789 */
2790 static unsigned long reclaim_hpage_zero_subpages(struct lruvec *lruvec,
2791 int priority,
2792 unsigned long nr_to_reclaim)
2793 {
2794 struct mem_cgroup *memcg;
2795 struct hpage_reclaim *hr_queue;
2796 int nid = lruvec->pgdat->node_id;
2797 unsigned long nr_reclaimed = 0, nr_scanned = 0, nr_to_scan;
2798
2799 memcg = lruvec_memcg(lruvec);
2800 if (!memcg)
2801 goto out;
2802
2803 hr_queue = &memcg->nodeinfo[nid]->hpage_reclaim_queue;
> 2804 if (get_thp_reclaim_mode(memcg) == THP_RECLAIM_DISABLE)
2805 goto out;
2806
2807 /* The last scan loop will scan all the huge pages.*/
2808 nr_to_scan = priority == 0 ? 0 : MAX_SCAN_HPAGE;
2809
2810 do {
2811 struct page *page = NULL;
2812
2813 if (zsr_get_hpage(hr_queue, &page))
2814 break;
2815
2816 if (!page)
2817 continue;
2818
2819 nr_reclaimed += zsr_reclaim_hpage(lruvec, page);
2820
2821 cond_resched();
2822
2823 } while ((nr_reclaimed < nr_to_reclaim) && (++nr_scanned != nr_to_scan));
2824 out:
2825 return nr_reclaimed;
2826 }
2827 #else
2828 static unsigned long reclaim_hpage_zero_subpages(struct lruvec *lruvec,
2829 int priority,
2830 unsigned long nr_to_reclaim)
2831 {
2832 return 0;
2833 }
2834 #endif
2835
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
RFC 3/6 mm, thp: introduce zero subpages reclaim threshold
In reply to Ning ZhangIn this patch, we add memory.thp_reclaim_ctrl for each memory cgroup to control thp reclaim.
The first controller "threshold" is to set the reclaim threshold. The default value is 16, which means if a huge page contains over 16 zero subpages (estimated), the huge page can be split and the zero subpages can be reclaimed when the zero subpages reclaim is enable.
You can change this value by:
echo "threshold $v" > /sys/fs/cgroup/memory/{memcg}/thp_reclaim_ctrl
--- include/linux/huge_mm.h | 3 ++- include/linux/memcontrol.h | 3 +++ mm/huge_memory.c | 9 ++++--- mm/memcontrol.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++ mm/vmscan.c | 4 ++- 5 files changed, 75 insertions(+), 6 deletions(-)
include/linux/huge_mm.h +2 -1
@@ -187,7 +187,8 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,#ifdef CONFIG_MEMCG extern int global_thp_reclaim; -int zsr_get_hpage (struct hpage_reclaim *hr_queue,struct page **reclaim_page); +int zsr_get_hpage (struct hpage_reclaim *hr_queue,struct page **reclaim_page, +int threshold);unsigned long zsr_reclaim_hpage (struct lruvec *lruvec,struct page *page);static inline struct list_head *hpage_reclaim_list (struct page *page) {
include/linux/memcontrol.h +3 -0
@@ -237,6 +237,8 @@ enum thp_reclaim_state {THP_RECLAIM_ENABLE , THP_RECLAIM_MEMCG, /* For global configure*/ }; + +#define THP_RECLAIM_THRESHOLD_DEFAULT 16#endif /* * The memory controller data structure. The memory controller controls both
@@ -356,6 +358,7 @@ struct mem_cgroup {#ifdef CONFIG_TRANSPARENT_HUGEPAGE struct deferred_split deferred_split_queue;int thp_reclaim; +int thp_reclaim_threshold;#endif struct mem_cgroup_per_node *nodeinfo[];
mm/huge_memory.c +5 -4
@@ -3270,7 +3270,7 @@ static inline bool is_zero_page(struct page *page)* We'll split the huge page iff it contains at least 1/32 zeros, * estimate it by checking some discrete unsigned long values. */ -static bool hpage_estimate_zero (struct page *page) +static bool hpage_estimate_zero (struct page *page,int threshold) {unsigned int i, maybe_zero_pages =0 , offset =0 ;void *addr;
@@ -3281,7 +3281,7 @@ static bool hpage_estimate_zero(struct page *page)if (unlikely ((offset +1 ) *BYTES_PER_LONG >PAGE_SIZE )) offset =0 ;if (*(const unsigned long *)(addr + offset) ==0UL ) { -if (++maybe_zero_pages ==HPAGE_PMD_NR >>5 ) { +if (++maybe_zero_pages == threshold) {kunmap (page);return true; }
@@ -3456,7 +3456,8 @@ static unsigned long reclaim_zero_subpages(struct list_head *list,* be stored in reclaim_page; otherwise, just delete the page from the * queue. */ -int zsr_get_hpage (struct hpage_reclaim *hr_queue,struct page **reclaim_page) +int zsr_get_hpage (struct hpage_reclaim *hr_queue,struct page **reclaim_page, +int threshold) {struct page *page =NULL ;unsigned long flags;
@@ -3482,7 +3483,7 @@ int zsr_get_hpage(struct hpage_reclaim *hr_queue, struct page **reclaim_page)spin_unlock_irqrestore (&hr_queue->reclaim_queue_lock, flags); -if (hpage_can_reclaim (page) &&hpage_estimate_zero (page) && +if (hpage_can_reclaim (page) &&hpage_estimate_zero (page, threshold) && !isolate_lru_page (page)) {__mod_node_page_state (page_pgdat (page),NR_ISOLATED_ANON ,HPAGE_PMD_NR );
mm/memcontrol.c +62 -0
86 lines not on this page · raw message
mm/vmscan.c +3 -1
@@ -2794,6 +2794,7 @@ static unsigned long reclaim_hpage_zero_subpages(struct lruvec *lruvec,struct mem_cgroup *memcg;struct hpage_reclaim *hr_queue;int nid = lruvec->pgdat->node_id; +int threshold;unsigned long nr_reclaimed =0 , nr_scanned =0 , nr_to_scan; memcg =lruvec_memcg (lruvec);
@@ -2806,11 +2807,12 @@ static unsigned long reclaim_hpage_zero_subpages(struct lruvec *lruvec,/* The last scan loop will scan all the huge pages.*/ nr_to_scan = priority ==0 ?0 :MAX_SCAN_HPAGE ; + threshold =READ_ONCE (memcg->thp_reclaim_threshold); do { struct page *page = NULL; - if (zsr_get_hpage(hr_queue, &page)) + if (zsr_get_hpage(hr_queue, &page, threshold)) break; if (!page)
Signature · 1 line
1.8.3.1
RFC 4/6 mm, thp: introduce a controller to trigger zero subpages reclaim
In reply to Ning ZhangAdd a new controller named "reclaim" for memory.thp_reclaim_ctrl to trigger thp reclaim immediately:
echo "reclaim 1" > memory.thp_reclaim_ctrl
echo "reclaim 2" > memory.thp_reclaim_ctrl
"reclaim 1" means triggering reclaim only for current memcg. "reclaim 2" means triggering reclaim for current memcg and it's children memcgs.
--- include/linux/huge_mm.h | 1 + mm/huge_memory.c | 29 +++++++++++++++++++++++++++++ mm/memcontrol.c | 27 +++++++++++++++++++++++++++ 3 files changed, 57 insertions(+)
include/linux/huge_mm.h +1 -0
@@ -190,6 +190,7 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,int zsr_get_hpage (struct hpage_reclaim *hr_queue,struct page **reclaim_page,int threshold);unsigned long zsr_reclaim_hpage (struct lruvec *lruvec,struct page *page); +void zsr_reclaim_memcg (struct mem_cgroup *memcg);static inline struct list_head *hpage_reclaim_list (struct page *page) {return &page[3 ].hpage_reclaim_list;
mm/huge_memory.c +29 -0
@@ -3541,4 +3541,33 @@ unsigned long zsr_reclaim_hpage(struct lruvec *lruvec, struct page *page) return reclaimed; } + +void zsr_reclaim_memcg (struct mem_cgroup *memcg) +{ +struct lruvec *lruvec; +struct hpage_reclaim *hr_queue; +int threshold, nid; + +if (get_thp_reclaim_mode (memcg) ==THP_RECLAIM_DISABLE ) +return ; + + threshold =READ_ONCE (memcg->thp_reclaim_threshold); +for_each_online_node (nid) { + lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid)); + hr_queue = &memcg->nodeinfo[nid]->hpage_reclaim_queue; +for ( ; ; ) { +struct page *page =NULL ; + +if (zsr_get_hpage (hr_queue, &page, threshold)) +break ; + +if (!page) +continue ; + +zsr_reclaim_hpage (lruvec, page); + +cond_resched (); + } + } +}#endif
mm/memcontrol.c +27 -0
@@ -4521,6 +4521,8 @@ static int memcg_thp_reclaim_ctrl_show(struct seq_file *m, void *v)return 0 ; } +#define CTRL_RECLAIM_MEMCG 1/* only relciam current memcg */ +#define CTRL_RECLAIM_ALL 2/* reclaim current memcg and all the children memcgs */ static ssize_t memcg_thp_reclaim_ctrl_write (struct kernfs_open_file *of,char *buf,size_t nbytes,loff_t off)
@@ -4548,6 +4550,31 @@ static ssize_t memcg_thp_reclaim_ctrl_write(struct kernfs_open_file *of,return -EINVAL ;xchg (&memcg->thp_reclaim_threshold, threshold); + }else if (!strcmp (key,"reclaim" )) { +struct mem_cgroup *iter; +int mode; + + value =strsep_s (&buf," \t\n" ); +if (!value) +return -EINVAL ; + + ret =kstrtouint (value,0 , &mode); +if (ret) +return ret; + +switch (mode) { +case CTRL_RECLAIM_MEMCG : +zsr_reclaim_memcg (memcg); +break ; +case CTRL_RECLAIM_ALL : + iter =mem_cgroup_iter (memcg,NULL ,NULL ); +do { +zsr_reclaim_memcg (iter); + }while ((iter =mem_cgroup_iter (memcg, iter,NULL ))); +break ; +default : +return -EINVAL ; + } }else return -EINVAL ;
Signature · 1 line
1.8.3.1
RFC 5/6 mm, thp: add some statistics for zero subpages reclaim
In reply to Ning Zhangqueue_length show the numbers of huge pages in the queue. split_hpage shows the numbers of huge pages split by thp reclaim. split_failed shows the numbers of huge pages split failed reclaim_subpage shows the numbers of zero subpages reclaimed by thp reclaim.
--- include/linux/huge_mm.h | 3 ++- include/linux/mmzone.h | 3 +++ mm/huge_memory.c | 8 ++++++-- mm/memcontrol.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ mm/vmscan.c | 2 +- 5 files changed, 59 insertions(+), 4 deletions(-)
include/linux/huge_mm.h +2 -1
@@ -189,7 +189,8 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,extern int global_thp_reclaim;int zsr_get_hpage (struct hpage_reclaim *hr_queue,struct page **reclaim_page,int threshold); -unsigned long zsr_reclaim_hpage (struct lruvec *lruvec,struct page *page); +unsigned long zsr_reclaim_hpage (struct hpage_reclaim *hr_queue, +struct lruvec *lruvec,struct page *page);void zsr_reclaim_memcg (struct mem_cgroup *memcg);static inline struct list_head *hpage_reclaim_list (struct page *page) {
include/linux/mmzone.h +3 -0
@@ -792,6 +792,9 @@ struct hpage_reclaim {spinlock_t reclaim_queue_lock;struct list_head reclaim_queue;unsigned long reclaim_queue_len; +atomic_long_t split_hpage; +atomic_long_t split_failed; + atomic_long_t reclaim_subpage; };#endif
mm/huge_memory.c +6 -2
@@ -3506,7 +3506,8 @@ int zsr_get_hpage(struct hpage_reclaim *hr_queue, struct page **reclaim_page, } -unsigned long zsr_reclaim_hpage (struct lruvec *lruvec,struct page *page) +unsigned long zsr_reclaim_hpage (struct hpage_reclaim *hr_queue, +struct lruvec *lruvec,struct page *page) {struct pglist_data *pgdat =page_pgdat (page);unsigned long reclaimed;
@@ -3523,12 +3524,15 @@ unsigned long zsr_reclaim_hpage(struct lruvec *lruvec, struct page *page)putback_lru_page (page);mod_node_page_state (pgdat,NR_ISOLATED_ANON , -HPAGE_PMD_NR ); +atomic_long_inc (&hr_queue->split_failed);return 0 ; }unlock_page (page);list_add_tail (&page->lru, &split_list); reclaimed =reclaim_zero_subpages (&split_list, &keep_list); +atomic_long_inc (&hr_queue->split_hpage); +atomic_long_add (reclaimed, &hr_queue->reclaim_subpage);spin_lock_irqsave (&lruvec->lru_lock, flags);move_pages_to_lru (lruvec, &keep_list);
@@ -3564,7 +3568,7 @@ void zsr_reclaim_memcg(struct mem_cgroup *memcg)if (!page)continue ; -zsr_reclaim_hpage (lruvec, page); +zsr_reclaim_hpage (hr_queue, lruvec, page);cond_resched (); }
mm/memcontrol.c +47 -0
59 lines not on this page · raw message
mm/vmscan.c +1 -1
@@ -2818,7 +2818,7 @@ static unsigned long reclaim_hpage_zero_subpages(struct lruvec *lruvec,if (!page)continue ; - nr_reclaimed +=zsr_reclaim_hpage (lruvec, page); + nr_reclaimed +=zsr_reclaim_hpage (hr_queue, lruvec, page);cond_resched ();
Signature · 1 line
1.8.3.1
RFC 6/6 mm, thp: add document for zero subpages reclaim
In reply to Ning ZhangAdd user guide for thp zero subpages reclaim.
--- Documentation/admin-guide/mm/transhuge.rst | 75 ++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)
Documentation/admin-guide/mm/transhuge.rst +75 -0
@@ -421,3 +421,78 @@ support enabled just fine as always. No difference can be noted in hugetlbfs other than there will be less overall fragmentation. All usual features belonging to hugetlbfs are preserved and unaffected. libhugetlbfs will also work fine as usual. + +THP zero subpages reclaim +========================= +THP may lead to memory bloat which may cause OOM. The reason is a huge +page may contain some zero subpages which users didn't really access them. +To avoid this, a mechanism to reclaim these zero subpages is introduced:: + + echo 1 > /sys/fs/cgroup/memory/{memcg}/memory.thp_reclaim + echo 0 > /sys/fs/cgroup/memory/{memcg}/memory.thp_reclaim + +Echo 1 to enable and echo 0 to disable. +The default value is inherited from its parent. The default mode of root +memcg is disable. + +We also add a global interface, if you don't want to configure it by +configuring every memory cgroup, you can use this one:: + + /sys/kernel/mm/transparent_hugepage/reclaim + +memcg + The default mode. It means every mem cgroup will use their own + configure. + +enable + means every mem cgroup will enable reclaim. + +disable + means every mem cgroup will disable reclaim. + +If zero subpages reclaim is enabled, the new huge page will be add to a +reclaim queue in mem_cgroup, and the queue would be scanned when memory +reclaiming. The queue stat can be checked like this:: + + cat /sys/fs/cgroup/memory/{memcg}/memory.thp_reclaim_stat + +queue_length + means the queue length of each node. + +split_hpage + means the numbers of huge pages split by thp reclaim of each node. + +split_failed + means the numbers of huge pages split failed by thp reclaim of + each node. + +reclaim_subpage + means the numbers of zero subpages reclaimed by thp reclaim of + each node. + +We also add a controller interface to set configs for thp reclaim:: + + /sys/fs/cgroup/memory/{memcg}/memory.thp_reclaim_ctrl + +threshold + means the huge page which contains at least threshold zero pages would + be split (estimate it by checking some discrete unsigned long values). + The default value of threshold is 16, and will inherit from it's parent. + The range of this value is (0, HPAGE_PMD_NR], which means the value must + be less than or equal to HPAGE_PMD_NR (512 in x86), and be greater than 0. + We can set reclaim threshold to be 8 by this:: + + echo "threshold 8" > memory.thp_reclaim_ctrl + +reclaim + triggers action immediately for the huge pages in the reclaim queue. + The action deponds on the thp reclaim config (reclaim, swap or disable, + disable means just remove the huge page from the queue). + This contronller has two value, 1 and 2. 1 means just reclaim the current + memcg, and 2 means reclaim the current memcg and all the children memcgs. + Like this:: + + echo "reclaim 1" > memory.thp_reclaim_ctrl + echo "reclaim 2" > memory.thp_reclaim_ctrl + +Only one of the configs mentioned above can be set at a time.
Signature · 1 line
1.8.3.1
Quoted 26 lines · Ning Zhang
As we know, thp may lead to memory bloat which may cause OOM. Through testing with some apps, we found that the reason of memory bloat is a huge page may contain some zero subpages (may accessed or not). And we found that most zero subpages are centralized in a few huge pages.
Following is a
text_classification_rnncase for tensorflow:zero_subpages huge_pages waste [ 0, 1) 186 0.00% [ 1, 2) 23 0.01% [ 2, 4) 36 0.02% [ 4, 8) 67 0.08% [ 8, 16) 80 0.23% [ 16, 32) 109 0.61% [ 32, 64) 44 0.49% [ 64, 128) 12 0.30% [ 128, 256) 28 1.54%[ 256, 513) 159 18.03%
In the case, there are 187 huge pages (25% of the total huge pages) which contain more then 128 zero subpages. And these huge pages lead to 19.57% waste of the total rss. It means we can reclaim 19.57% memory by splitting the 187 huge pages and reclaiming the zero subpages.
Quoted 8 lines
This patchset introduce a new mechanism to split the huge page which has zero subpages and reclaim these zero subpages.
We add the anonymous huge page to a list to reduce the cost of finding the huge page. When the memory reclaim is triggering, the list will be walked and the huge page contains enough zero subpages may be reclaimed. Meanwhile, replace the zero subpages by
ZERO_PAGE(0).
Does it actually help your workload?
I mean this will only be triggered via vmscan that was going to split pages and free anyway.
You prioritize splitting THP and freeing zero subpages over reclaiming other pages. It may or may not be right thing to do, depending on workload.
Maybe it makes more sense to check for all-zero pages just after split_huge_page_to_list() in vmscan and free such pages immediately rather then add all this complexity?
Quoted 8 lines
Yu Zhao has done some similar work when the huge page is swap out or migrated to accelerate[1]. While we do this in the normal memory shrink path for the swapoff scene to avoid OOM.
In the future, we will do the proactive reclaim to reclaim the "cold" huge page proactively. This is for keeping the performance of thp as for as possible. In addition to that, some users want the memory usage using thp is equal to the usage using 4K.
Proactive reclaim can be harmful if your max_ptes_none allows to recreate THP back.
Signature · 1 line
Kirill A. Shutemov
Quoted 40 lines · Kirill A. Shutemov
Quoted 26 lines · Ning Zhang
As we know, thp may lead to memory bloat which may cause OOM. Through testing with some apps, we found that the reason of memory bloat is a huge page may contain some zero subpages (may accessed or not). And we found that most zero subpages are centralized in a few huge pages.
Following is a
text_classification_rnncase for tensorflow:zero_subpages huge_pages waste [ 0, 1) 186 0.00% [ 1, 2) 23 0.01% [ 2, 4) 36 0.02% [ 4, 8) 67 0.08% [ 8, 16) 80 0.23% [ 16, 32) 109 0.61% [ 32, 64) 44 0.49% [ 64, 128) 12 0.30% [ 128, 256) 28 1.54%[ 256, 513) 159 18.03%
In the case, there are 187 huge pages (25% of the total huge pages) which contain more then 128 zero subpages. And these huge pages lead to 19.57% waste of the total rss. It means we can reclaim 19.57% memory by splitting the 187 huge pages and reclaiming the zero subpages.
Quoted 8 lines
This patchset introduce a new mechanism to split the huge page which has zero subpages and reclaim these zero subpages.
We add the anonymous huge page to a list to reduce the cost of finding the huge page. When the memory reclaim is triggering, the list will be walked and the huge page contains enough zero subpages may be reclaimed. Meanwhile, replace the zero subpages by
ZERO_PAGE(0).Does it actually help your workload?
I mean this will only be triggered via vmscan that was going to split pages and free anyway.
Quoted 8 lines
You prioritize splitting THP and freeing zero subpages over reclaiming other pages. It may or may not be right thing to do, depending on workload.
Maybe it makes more sense to check for all-zero pages just after
split_huge_page_to_list()in vmscan and free such pages immediately rather then add all this complexity?
The purpose of zero subpages reclaim(ZSR) is to pick out the huge pages which have waste and reclaim them.
We do this for two reasons:
1. If swap is off, anonymous pages will not be scanned, and we don't have the opportunity to split the huge page. ZSR can be helpful for this.
2. If swap is on, splitting first will not only split the huge page, but also swap out the nonzero subpages, while ZSR will only split the huge page. Splitting first will result to more performance degradation. If ZSR can't reclaim enough pages, swap can still work.
Why use a seperate ZSR list instead of the default LRU list?
Because it may cause high CPU overhead to scan for target huge pages if there both exist a lot of regular and huge pages. And it maybe especially terrible when swap is off, we may scan the whole LRU list many times. A huge page will be deleted from ZSR list when it was scanned, so the page will be scanned only once. It's hard to use LRU list, because it may add new pages into LRU list continuously when scanning.
Also, we can decrease the priority to prioritize reclaiming file-backed page. For example, only triggerring ZSR when the priority is less than 4.
Quoted 10 lines
Quoted 8 lines
Yu Zhao has done some similar work when the huge page is swap out or migrated to accelerate[1]. While we do this in the normal memory shrink path for the swapoff scene to avoid OOM.
In the future, we will do the proactive reclaim to reclaim the "cold" huge page proactively. This is for keeping the performance of thp as for as possible. In addition to that, some users want the memory usage using thp is equal to the usage using 4K.
Proactive reclaim can be harmful if your
max_ptes_noneallows to recreate THP back.
Thanks! We will consider it.
Quoted 1 line
Quoted 79 lines · ning zhang
Quoted 40 lines · Kirill A. Shutemov
Quoted 26 lines · Ning Zhang
As we know, thp may lead to memory bloat which may cause OOM. Through testing with some apps, we found that the reason of memory bloat is a huge page may contain some zero subpages (may accessed or not). And we found that most zero subpages are centralized in a few huge pages.
Following is a
text_classification_rnncase for tensorflow:zero_subpages huge_pages waste [ 0, 1) 186 0.00% [ 1, 2) 23 0.01% [ 2, 4) 36 0.02% [ 4, 8) 67 0.08% [ 8, 16) 80 0.23% [ 16, 32) 109 0.61% [ 32, 64) 44 0.49% [ 64, 128) 12 0.30% [ 128, 256) 28 1.54%[ 256, 513) 159 18.03%
In the case, there are 187 huge pages (25% of the total huge pages) which contain more then 128 zero subpages. And these huge pages lead to 19.57% waste of the total rss. It means we can reclaim 19.57% memory by splitting the 187 huge pages and reclaiming the zero subpages.
Quoted 8 lines
This patchset introduce a new mechanism to split the huge page which has zero subpages and reclaim these zero subpages.
We add the anonymous huge page to a list to reduce the cost of finding the huge page. When the memory reclaim is triggering, the list will be walked and the huge page contains enough zero subpages may be reclaimed. Meanwhile, replace the zero subpages by
ZERO_PAGE(0).Does it actually help your workload?
I mean this will only be triggered via vmscan that was going to split pages and free anyway.
Quoted 8 lines
You prioritize splitting THP and freeing zero subpages over reclaiming other pages. It may or may not be right thing to do, depending on workload.
Maybe it makes more sense to check for all-zero pages just after
split_huge_page_to_list()in vmscan and free such pages immediately rather then add all this complexity?The purpose of zero subpages reclaim(ZSR) is to pick out the huge pages which
have waste and reclaim them.We do this for two reasons:
1. If swap is off, anonymous pages will not be scanned, and we don't have the
opportunity to split the huge page. ZSR can be helpful for this.
2. If swap is on, splitting first will not only split the huge page, but also
swap out the nonzero subpages, while ZSR will only split the huge page. Splitting first will result to more performance degradation. If ZSR
can't
reclaim enough pages, swap can still work.Why use a seperate ZSR list instead of the default LRU list?
Because it may cause high CPU overhead to scan for target huge pages if there
both exist a lot of regular and huge pages. And it maybe especially terrible
when swap is off, we may scan the whole LRU list many times. A huge page will
be deleted from ZSR list when it was scanned, so the page will be scanned only
once. It's hard to use LRU list, because it may add new pages into LRU list continuously when scanning.
Quoted 3 lines
Also, we can decrease the priority to prioritize reclaiming file-backed page.
For example, only triggerring ZSR when the priority is less than 4.
I'm not sure if this will help the workloads in general or not. The problem is it doesn't check if the huge page is "hot" or not. It just picks up the first huge page from the list, which seems like a FIFO list IIUC. But if the huge page is "hot" even though there is some internal access imbalance it may be better to keep the huge page since the performance gain may outperform the memory saving. But if the huge page is not "hot", then I think the question is why it is a THP in the first place.
Let's step back to think about whether allocating THP upon first access for such area or workload is good or not. We should be able to check the access imbalance in allocation stage instead of reclaim stage. Currently anonymous THP just supports 3 modes: always, madvise and none. Both always and madvise tries to allocate THP in page fault path (assuming anonymous THP) upon first access. I'm wondering if we could add a "defer" mode or not. It defers THP allocation/collapse to khugepaged instead of in page fault path. Then all the knobs used by khugepaged could be applied, particularly max_ptes_none in your case. You could set a low max_ptes_none if you prefer memory saving. IMHO, this seems much simpler than scanning list (may be quite long) to find out suitable candidate then split then replace to zero page.
Of course this may have some potential performance impact since the THP install is delayed for some time. This could be optimized by respecting MADV_HUGEPAGE.
Anyway, just some wild idea.
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Yu Zhao has done some similar work when the huge page is swap out or migrated to accelerate[1]. While we do this in the normal memory shrink path for the swapoff scene to avoid OOM.
In the future, we will do the proactive reclaim to reclaim the "cold" huge page proactively. This is for keeping the performance of thp as for as possible. In addition to that, some users want the memory usage using thp is equal to the usage using 4K.
Proactive reclaim can be harmful if your
max_ptes_noneallows to recreate THP back.Thanks! We will consider it.
Quoted 1 line
Quoted 82 lines · Yang Shi
Quoted 78 lines · ning zhang
Quoted 40 lines · Kirill A. Shutemov
Quoted 26 lines · Ning Zhang
As we know, thp may lead to memory bloat which may cause OOM. Through testing with some apps, we found that the reason of memory bloat is a huge page may contain some zero subpages (may accessed or not). And we found that most zero subpages are centralized in a few huge pages.
Following is a
text_classification_rnncase for tensorflow:zero_subpages huge_pages waste [ 0, 1) 186 0.00% [ 1, 2) 23 0.01% [ 2, 4) 36 0.02% [ 4, 8) 67 0.08% [ 8, 16) 80 0.23% [ 16, 32) 109 0.61% [ 32, 64) 44 0.49% [ 64, 128) 12 0.30% [ 128, 256) 28 1.54% [ 256, 513) 159 18.03%In the case, there are 187 huge pages (25% of the total huge pages) which contain more then 128 zero subpages. And these huge pages lead to 19.57% waste of the total rss. It means we can reclaim 19.57% memory by splitting the 187 huge pages and reclaiming the zero subpages.
Quoted 8 lines
This patchset introduce a new mechanism to split the huge page which has zero subpages and reclaim these zero subpages.
We add the anonymous huge page to a list to reduce the cost of finding the huge page. When the memory reclaim is triggering, the list will be walked and the huge page contains enough zero subpages may be reclaimed. Meanwhile, replace the zero subpages by
ZERO_PAGE(0).Does it actually help your workload?
I mean this will only be triggered via vmscan that was going to split pages and free anyway.
Quoted 8 lines
You prioritize splitting THP and freeing zero subpages over reclaiming other pages. It may or may not be right thing to do, depending on workload.
Maybe it makes more sense to check for all-zero pages just after
split_huge_page_to_list()in vmscan and free such pages immediately rather then add all this complexity?The purpose of zero subpages reclaim(ZSR) is to pick out the huge pages which
have waste and reclaim them.We do this for two reasons:
1. If swap is off, anonymous pages will not be scanned, and we don't have the
opportunity to split the huge page. ZSR can be helpful for this.
2. If swap is on, splitting first will not only split the huge page, but also
swap out the nonzero subpages, while ZSR will only split the huge page. Splitting first will result to more performance degradation. If ZSR
can't
reclaim enough pages, swap can still work.Why use a seperate ZSR list instead of the default LRU list?
Because it may cause high CPU overhead to scan for target huge pages if there
both exist a lot of regular and huge pages. And it maybe especially terrible
when swap is off, we may scan the whole LRU list many times. A huge page will
be deleted from ZSR list when it was scanned, so the page will be scanned only
once. It's hard to use LRU list, because it may add new pages into LRU list continuously when scanning.Quoted 3 lines
Also, we can decrease the priority to prioritize reclaiming file-backed page.
For example, only triggerring ZSR when the priority is less than 4.
Quoted 8 lines
I'm not sure if this will help the workloads in general or not. The problem is it doesn't check if the huge page is "hot" or not. It just picks up the first huge page from the list, which seems like a FIFO list IIUC. But if the huge page is "hot" even though there is some internal access imbalance it may be better to keep the huge page since the performance gain may outperform the memory saving. But if the huge page is not "hot", then I think the question is why it is a THP in the first place.
We don't split all the huge pages, and just split the huge page contains enough zero subpages. It's hard to check a anonymous page is hot or cold, and we are working on it.
We only scan 32 huge pages maximum except the last loop when reclaiming. I think we can start ZSR when priority is 1 or 2, or maybe only when priority is 0. In this case, If we don't start ZSR, the process will be killed by OOM.
Quoted 31 lines
Let's step back to think about whether allocating THP upon first access for such area or workload is good or not. We should be able to check the access imbalance in allocation stage instead of reclaim stage. Currently anonymous THP just supports 3 modes: always, madvise and none. Both always and madvise tries to allocate THP in page fault path (assuming anonymous THP) upon first access. I'm wondering if we could add a "defer" mode or not. It defers THP allocation/collapse to khugepaged instead of in page fault path. Then all the knobs used by khugepaged could be applied, particularly
max_ptes_nonein your case. You could set a lowmax_ptes_noneif you prefer memory saving. IMHO, this seems much simpler than scanning list (may be quite long) to find out suitable candidate then split then replace to zero page.Of course this may have some potential performance impact since the THP install is delayed for some time. This could be optimized by respecting
MADV_HUGEPAGE.Anyway, just some wild idea.
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Yu Zhao has done some similar work when the huge page is swap out or migrated to accelerate[1]. While we do this in the normal memory shrink path for the swapoff scene to avoid OOM.
In the future, we will do the proactive reclaim to reclaim the "cold" huge page proactively. This is for keeping the performance of thp as for as possible. In addition to that, some users want the memory usage using thp is equal to the usage using 4K.
Proactive reclaim can be harmful if your
max_ptes_noneallows to recreate THP back.Thanks! We will consider it.
Michal Hocko
reviewerQuoted 18 lines · Ning Zhang
As we know, thp may lead to memory bloat which may cause OOM. Through testing with some apps, we found that the reason of memory bloat is a huge page may contain some zero subpages (may accessed or not). And we found that most zero subpages are centralized in a few huge pages.
Following is a
text_classification_rnncase for tensorflow:zero_subpages huge_pages waste [ 0, 1) 186 0.00% [ 1, 2) 23 0.01% [ 2, 4) 36 0.02% [ 4, 8) 67 0.08% [ 8, 16) 80 0.23% [ 16, 32) 109 0.61% [ 32, 64) 44 0.49% [ 64, 128) 12 0.30% [ 128, 256) 28 1.54%
Quoted 7 lines
[ 256, 513) 159 18.03%
In the case, there are 187 huge pages (25% of the total huge pages) which contain more then 128 zero subpages. And these huge pages lead to 19.57% waste of the total rss. It means we can reclaim 19.57% memory by splitting the 187 huge pages and reclaiming the zero subpages.
What is the THP policy configuration in your testing? I assume you are using defaults right? That would be always for THP and madvise for defrag. Would it make more sense to use madvise mode for THP for your workload? The THP code is rather complex and just by looking at the diffstat this add quite a lot on top. Is this really worth it?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 26 lines · Michal Hocko
Quoted 18 lines · Ning Zhang
As we know, thp may lead to memory bloat which may cause OOM. Through testing with some apps, we found that the reason of memory bloat is a huge page may contain some zero subpages (may accessed or not). And we found that most zero subpages are centralized in a few huge pages.
Following is a
text_classification_rnncase for tensorflow:zero_subpages huge_pages waste [ 0, 1) 186 0.00% [ 1, 2) 23 0.01% [ 2, 4) 36 0.02% [ 4, 8) 67 0.08% [ 8, 16) 80 0.23% [ 16, 32) 109 0.61% [ 32, 64) 44 0.49% [ 64, 128) 12 0.30% [ 128, 256) 28 1.54%Quoted 7 lines
[ 256, 513) 159 18.03%
In the case, there are 187 huge pages (25% of the total huge pages) which contain more then 128 zero subpages. And these huge pages lead to 19.57% waste of the total rss. It means we can reclaim 19.57% memory by splitting the 187 huge pages and reclaiming the zero subpages.
Quoted 5 lines
What is the THP policy configuration in your testing? I assume you are using defaults right? That would be always for THP and madvise for defrag. Would it make more sense to use madvise mode for THP for your workload? The THP code is rather complex and just by looking at the diffstat this add quite a lot on top. Is this really worth it?
The THP configuration is always.
Madvise needs users to set MADV_HUGEPAGE by themselves if they want use huge page, while many users don't do set this, and they can't control this well.
Such as java, users can set heap and metaspace to use huge pages with madvise, but there is also memory bloat. Users still need to test whether their app can accept the waste.
For the case above, if we set THP configuration to be madvise, all the pages it uses will be 4K-page.
Memory bloat is one of the most important reasons that users disable THP. We do this to popularize THP to be default enabled.
Michal Hocko
reviewerQuoted 34 lines · ning zhang
Quoted 26 lines · Michal Hocko
Quoted 18 lines · Ning Zhang
As we know, thp may lead to memory bloat which may cause OOM. Through testing with some apps, we found that the reason of memory bloat is a huge page may contain some zero subpages (may accessed or not). And we found that most zero subpages are centralized in a few huge pages.
Following is a
text_classification_rnncase for tensorflow:zero_subpages huge_pages waste [ 0, 1) 186 0.00% [ 1, 2) 23 0.01% [ 2, 4) 36 0.02% [ 4, 8) 67 0.08% [ 8, 16) 80 0.23% [ 16, 32) 109 0.61% [ 32, 64) 44 0.49% [ 64, 128) 12 0.30% [ 128, 256) 28 1.54%Quoted 7 lines
[ 256, 513) 159 18.03%
In the case, there are 187 huge pages (25% of the total huge pages) which contain more then 128 zero subpages. And these huge pages lead to 19.57% waste of the total rss. It means we can reclaim 19.57% memory by splitting the 187 huge pages and reclaiming the zero subpages.
Quoted 5 lines
What is the THP policy configuration in your testing? I assume you are using defaults right? That would be always for THP and madvise for defrag. Would it make more sense to use madvise mode for THP for your workload? The THP code is rather complex and just by looking at the diffstat this add quite a lot on top. Is this really worth it?
Quoted 4 lines
The THP configuration is always.
Madvise needs users to set
MADV_HUGEPAGEby themselves if they want use huge page, while many users don't do set this, and they can't control this well.
What do you mean tey can't control this well?
Quoted 3 lines
Such as java, users can set heap and metaspace to use huge pages with madvise, but there is also memory bloat. Users still need to test whether their app can accept the waste.
There will always be some internal fragmentation when huge pages are used. The amount will depend on how well the memory is used but huge pages give a performance boost in return.
If the memory bloat is a significant problem then overeager THP usage is certainly not good and I would argue that applying THP always policy is not a proper configuration. No matter how much the MM code can try to fix up the situation it will be always a catch up game.
Quoted 5 lines
For the case above, if we set THP configuration to be madvise, all the pages it uses will be 4K-page.
Memory bloat is one of the most important reasons that users disable THP. We do this to popularize THP to be default enabled.
To my knowledge the most popular reason to disable THP is the runtime overhead. A large part of that overhead has been reduced by not doing heavy compaction during the page fault allocations by default. Memory overhead is certainly an important aspect as well but there is always a possibility to reduce that by reducing it to madvised regions for page fault (i.e. those where author of the code has considered the costs vs. benefits of the huge page) and setting up a conservative khugepaged policy. So there are existing tools available. You are trying to add quite a lot of code so you should have good arguments to add more complexity. I am not sure that popularizing THP is a strong one TBH.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 33 lines · Michal Hocko
Quoted 32 lines · ning zhang
Quoted 31 lines · Michal Hocko
Quoted 20 lines · Ning Zhang
As we know, thp may lead to memory bloat which may cause OOM. Through testing with some apps, we found that the reason of memory bloat is a huge page may contain some zero subpages (may accessed or not). And we found that most zero subpages are centralized in a few huge pages.
Following is a
text_classification_rnncase for tensorflow:zero_subpages huge_pages waste [ 0, 1) 186 0.00% [ 1, 2) 23 0.01% [ 2, 4) 36 0.02% [ 4, 8) 67 0.08% [ 8, 16) 80 0.23% [ 16, 32) 109 0.61% [ 32, 64) 44 0.49% [ 64, 128) 12 0.30% [ 128, 256) 28 1.54% [ 256, 513) 159 18.03%Quoted 5 lines
In the case, there are 187 huge pages (25% of the total huge pages) which contain more then 128 zero subpages. And these huge pages lead to 19.57% waste of the total rss. It means we can reclaim 19.57% memory by splitting the 187 huge pages and reclaiming the zero subpages.
What is the THP policy configuration in your testing? I assume you are using defaults right? That would be always for THP and madvise for defrag. Would it make more sense to use madvise mode for THP for your workload? The THP code is rather complex and just by looking at the diffstat this add quite a lot on top. Is this really worth it?
Quoted 5 lines
Quoted 4 lines
The THP configuration is always.
Madvise needs users to set
MADV_HUGEPAGEby themselves if they want use huge page, while many users don't do set this, and they can't control this well.What do you mean tey can't control this well?
I means they don't know where they should use THP.
And even if they use madvise, memory bloat still exists.
<https://dict.youdao.com/w/still%20exist/#keyfrom=E2Ctranslation>
Quoted 18 lines
Quoted 4 lines
Such as java, users can set heap and metaspace to use huge pages with madvise, but there is also memory bloat. Users still need to test whether their app can accept the waste.
There will always be some internal fragmentation when huge pages are used. The amount will depend on how well the memory is used but huge pages give a performance boost in return.
If the memory bloat is a significant problem then overeager THP usage is certainly not good and I would argue that applying THP always policy is not a proper configuration. No matter how much the MM code can try to fix up the situation it will be always a catch up game.
Quoted 5 lines
For the case above, if we set THP configuration to be madvise, all the pages it uses will be 4K-page.
Memory bloat is one of the most important reasons that users disable THP. We do this to popularize THP to be default enabled.
Quoted 10 lines
To my knowledge the most popular reason to disable THP is the runtime overhead. A large part of that overhead has been reduced by not doing heavy compaction during the page fault allocations by default. Memory overhead is certainly an important aspect as well but there is always a possibility to reduce that by reducing it to madvised regions for page fault (i.e. those where author of the code has considered the costs vs. benefits of the huge page) and setting up a conservative khugepaged policy. So there are existing tools available. You are trying to add quite a lot of code so you should have good arguments to add more complexity. I am not sure that popularizing THP is a strong one TBH.
Sorry for relpying late. For the compaction, we can set defrag of THP to be defer or never, to avoid overhead produced by direct reclaim. However, there are no way to reduce memory bloat.
If the memory usage reach the limit, and we can't reclaim some pages, the OOM will be triggered and the process will be killed. Our patchest is to avoid OOM.
Much code is interface to control ZSR. And we will try to reduce the complexity.