Allow high order pages to be stored on PCP
Subsystems (4)
cover Allow high order pages to be stored on PCP
The per-cpu page allocator (PCP) only handles order-0 pages. With the series "Use local_lock for pcp protection and reduce stat overhead" and "Calculate pcp->high based on zone sizes and active CPUs", it's now feasible to store high-order pages on PCP lists.
This small series allows PCP to store "cheap" orders where cheap is determined by PAGE_ALLOC_COSTLY_ORDER and THP-sized allocations. For convenience, the series with the prerequisites are at
git://git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git mm-pcphighorder-v1r7
The baseline is 5.13-rc2 because that is what I tested with but the patches rebase cleanly to mmots.
include/linux/mmzone.h | 20 ++++- mm/internal.h | 2 +- mm/page_alloc.c | 171 +++++++++++++++++++++++++++++------------ mm/swap.c | 2 +- 4 files changed, 141 insertions(+), 54 deletions(-)
Signature · 1 line
2.26.2
PATCH 1/2 mm/page_alloc: Move free_the_page
In the next page, free_compount_page is going to use the common helper free_the_page. This patch moves the definition to ease review. No functional change.
--- mm/page_alloc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
mm/page_alloc.c +8 -8
@@ -687,6 +687,14 @@ static void bad_page(struct page *page, const char *reason)add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); } +static inline void free_the_page (struct page *page,unsigned int order) +{ +if (order ==0 )/* Via pcp? */ +free_unref_page (page); +else +__free_pages_ok (page, order,FPI_NONE ); +} +/* * Higher-order pages are called "compound pages". They are structured thusly: *
@@ -5345,14 +5353,6 @@ unsigned long get_zeroed_page(gfp_t gfp_mask) }EXPORT_SYMBOL (get_zeroed_page); -static inline void free_the_page (struct page *page,unsigned int order) -{ -if (order ==0 )/* Via pcp? */ -free_unref_page (page); -else -__free_pages_ok (page, order,FPI_NONE ); -} -/** * __free_pages - Free pages allocated with alloc_pages(). * @page: The page pointer returned from alloc_pages().
Signature · 1 line
2.26.2
Re: [PATCH 1/2] mm/page_alloc: Move free_the_page
Quoted 5 lines · Mel Gorman
In the next page,
free_compount_pageis going to use the common helperfree_the_page. This patch moves the definition to ease review. No functional change.Signed-off-by Mel Gorman <[email protected]>
Quoted 39 lines
--- mm/page_alloc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)mm/page_alloc.c +8
-8@@ -687,6 +687,14 @@ static void bad_page(struct page *page, const char *reason)add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); } +static inline void free_the_page (struct page *page,unsigned int order) +{ +if (order ==0 )/* Via pcp? */ +free_unref_page (page); +else +__free_pages_ok (page, order,FPI_NONE ); +} +/* * Higher-order pages are called "compound pages". They are structured thusly: * @@ -5345,14 +5353,6 @@ unsigned long get_zeroed_page(gfp_t gfp_mask) }EXPORT_SYMBOL (get_zeroed_page); -static inline void free_the_page (struct page *page,unsigned int order) -{ -if (order ==0 )/* Via pcp? */ -free_unref_page (page); -else -__free_pages_ok (page, order,FPI_NONE ); -} -/** * __free_pages - Free pages allocated with alloc_pages(). * @page: The page pointer returned from alloc_pages().
PATCH 2/2 mm/page_alloc: Allow high-order pages to be stored on the per-cpu lists
In reply to Mel GormanThe per-cpu page allocator (PCP) only stores order-0 pages. This means that all THP and "cheap" high-order allocations including SLUB contends on the zone->lock. This patch extends the PCP allocator to store THP and "cheap" high-order pages. Note that struct per_cpu_pages increases in size to 256 bytes (4 cache lines) on x86-64.
Note that this is not necessarily a universal performance win because of how it is implemented. High-order pages can cause pcp->high to be exceeded prematurely for lower-orders so for example, a large number of THP pages being freed could release order-0 pages from the PCP lists. Hence, much depends on the allocation/free pattern as observed by a single CPU to determine if caching helps or hurts a particular workload.
That said, basic performance testing passed. The following is a netperf UDP_STREAM test which hits the relevant patches as some of the network allocations are high-order.
netperf-udp
5.13.0-rc2 5.13.0-rc2
mm-pcpburst-v3r4 mm-pcphighorder-v1r7
Hmean send-64 261.46 ( 0.00%) 266.30 * 1.85%*
Hmean send-128 516.35 ( 0.00%) 536.78 * 3.96%*
Hmean send-256 1014.13 ( 0.00%) 1034.63 * 2.02%*
Hmean send-1024 3907.65 ( 0.00%) 4046.11 * 3.54%*
Hmean send-2048 7492.93 ( 0.00%) 7754.85 * 3.50%*
Hmean send-3312 11410.04 ( 0.00%) 11772.32 * 3.18%*
Hmean send-4096 13521.95 ( 0.00%) 13912.34 * 2.89%*
Hmean send-8192 21660.50 ( 0.00%) 22730.72 * 4.94%*
Hmean send-16384 31902.32 ( 0.00%) 32637.50 * 2.30%*From a functional point of view, a patch like this is necessary to make bulk allocation of high-order pages work with similar performance to order-0 bulk allocations. The bulk allocator is not updated in this series as it would have to be determined by bulk allocation users how they want to track the order of pages allocated with the bulk allocator.
--- include/linux/mmzone.h | 20 +++++- mm/internal.h | 2 +- mm/page_alloc.c | 159 +++++++++++++++++++++++++++++------------ mm/swap.c | 2 +- 4 files changed, 135 insertions(+), 48 deletions(-)
include/linux/mmzone.h +19 -1
@@ -333,6 +333,24 @@ enum zone_watermarks { NR_WMARK }; +/* +* One per migratetype for each PAGE_ALLOC_COSTLY_ORDER plus one additional +* for pageblock size for THP if configured. +*/ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define NR_PCP_THP 1 +#else +#define NR_PCP_THP 0 +#endif +#define NR_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1 + NR_PCP_THP)) + +/* +* Shift to encode migratetype in order in the least significant bits and +* migratetype in the higher bits. +*/ +#define NR_PCP_ORDER_SHIFT 8 +#define NR_PCP_ORDER_MASK ((1<<NR_PCP_ORDER_SHIFT) - 1) +#define min_wmark_pages (z) (z->_watermark[WMARK_MIN] + z->watermark_boost)#define low_wmark_pages (z) (z->_watermark[WMARK_LOW] + z->watermark_boost)#define high_wmark_pages (z) (z->_watermark[WMARK_HIGH] + z->watermark_boost)
@@ -349,7 +367,7 @@ struct per_cpu_pages {#endif /* Lists of pages, one per migrate type stored on the pcp-lists */ -struct list_head lists[MIGRATE_PCPTYPES ]; +struct list_head lists[NR_PCP_LISTS ]; };struct per_cpu_zonestat {
mm/internal.h +1 -1
@@ -203,7 +203,7 @@ extern void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags);extern int user_min_free_kbytes; -extern void free_unref_page (struct page *page); +extern void free_unref_page (struct page *page,unsigned int order);extern void free_unref_page_list (struct list_head *list);extern void zone_pcp_update (struct zone *zone,int cpu_online);
mm/page_alloc.c +114 -45
357 lines not on this page · raw message
mm/swap.c +1 -1
@@ -95,7 +95,7 @@ static void __put_single_page(struct page *page) {__page_cache_release (page);mem_cgroup_uncharge (page); -free_unref_page (page); +free_unref_page (page,0 ); }static void __put_compound_page (struct page *page)
Signature · 1 line
2.26.2
Re: [PATCH 2/2] mm/page_alloc: Allow high-order pages to be stored on the per-cpu lists
Quoted 30 lines · Mel Gorman
The per-cpu page allocator (PCP) only stores order-0 pages. This means that all THP and "cheap" high-order allocations including SLUB contends on the
zone->lock. This patch extends the PCP allocator to store THP and "cheap" high-order pages. Note that structper_cpu_pagesincreases in size to 256 bytes (4 cache lines) on x86-64.Note that this is not necessarily a universal performance win because of how it is implemented. High-order pages can cause
pcp->highto be exceeded prematurely for lower-orders so for example, a large number of THP pages being freed could release order-0 pages from the PCP lists. Hence, much depends on the allocation/free pattern as observed by a single CPU to determine if caching helps or hurts a particular workload.That said, basic performance testing passed. The following is a netperf
UDP_STREAMtest which hits the relevant patches as some of the network allocations are high-order.netperf-udp
5.13.0-rc2 5.13.0-rc2 mm-pcpburst-v3r4 mm-pcphighorder-v1r7 Hmean send-64 261.46 ( 0.00%) 266.30 * 1.85%* Hmean send-128 516.35 ( 0.00%) 536.78 * 3.96%* Hmean send-256 1014.13 ( 0.00%) 1034.63 * 2.02%* Hmean send-1024 3907.65 ( 0.00%) 4046.11 * 3.54%* Hmean send-2048 7492.93 ( 0.00%) 7754.85 * 3.50%* Hmean send-3312 11410.04 ( 0.00%) 11772.32 * 3.18%* Hmean send-4096 13521.95 ( 0.00%) 13912.34 * 2.89%* Hmean send-8192 21660.50 ( 0.00%) 22730.72 * 4.94%* Hmean send-16384 31902.32 ( 0.00%) 32637.50 * 2.30%*
Quoted 7 lines
From a functional point of view, a patch like this is necessary to make bulk allocation of high-order pages work with similar performance to order-0 bulk allocations. The bulk allocator is not updated in this series as it would have to be determined by bulk allocation users how they want to track the order of pages allocated with the bulk allocator.
Signed-off-by Mel Gorman <[email protected]>
Some comments below.
Quoted 19 lines
--- include/linux/mmzone.h | 20 +++++- mm/internal.h | 2 +- mm/page_alloc.c | 159 +++++++++++++++++++++++++++++------------ mm/swap.c | 2 +- 4 files changed, 135 insertions(+), 48 deletions(-)include/linux/mmzone.h +4
-0@@ -333,6 +333,24 @@ enum zone_watermarks { NR_WMARK }; +/* +* One per migratetype for each PAGE_ALLOC_COSTLY_ORDER plus one additional +* for pageblock size for THP if configured. +*/
include/linux/mmzone.h enum zone_watermarks { · line 349
Quoted 10 lines
include/linux/mmzone.h +10
-0+#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define NR_PCP_THP 1 +#else +#define NR_PCP_THP 0 +#endif +#define NR_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1 + NR_PCP_THP)) + +/* +* Shift to encode migratetype in order in the least significant bits and +* migratetype in the higher bits.
Hard for me to understand that comment. I would describe what the code does as e,g, "Shift to encode migratetype and order in the same integer, with order in the least significant bit ..." etc.
include/linux/mmzone.h enum zone_watermarks { · line 351
Quoted 2 lines
include/linux/mmzone.h +2
-0+*/ +#define NR_PCP_ORDER_SHIFT 8
Also ORDER_SHIFT is a bit misnomer, it's more precisely an ORDER_WIDTH, and we are shifting migratetype with it, not order. I'm just comparing with how we name nid/zid/etc bits in page flags.
Quoted 44 lines
include/linux/mmzone.h +3
-1+#define NR_PCP_ORDER_MASK ((1<<NR_PCP_ORDER_SHIFT) - 1) +#define min_wmark_pages (z) (z->_watermark[WMARK_MIN] + z->watermark_boost)#define low_wmark_pages (z) (z->_watermark[WMARK_LOW] + z->watermark_boost)#define high_wmark_pages (z) (z->_watermark[WMARK_HIGH] + z->watermark_boost)@@ -349,7 +367,7 @@ struct per_cpu_pages {#endif /* Lists of pages, one per migrate type stored on the pcp-lists */ -struct list_head lists[MIGRATE_PCPTYPES ]; +struct list_head lists[NR_PCP_LISTS ]; };struct per_cpu_zonestat {mm/internal.h +1
-1@@ -203,7 +203,7 @@ extern void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags);extern int user_min_free_kbytes; -extern void free_unref_page (struct page *page); +extern void free_unref_page (struct page *page,unsigned int order);extern void free_unref_page_list (struct list_head *list);extern void zone_pcp_update (struct zone *zone,int cpu_online);mm/page_alloc.c +9
-0@@ -687,10 +687,53 @@ static void bad_page(struct page *page, const char *reason)add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); } +static inline unsigned int order_to_pindex (int migratetype,int order) +{ +int base = order; + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +if (order >PAGE_ALLOC_COSTLY_ORDER ) { +VM_BUG_ON (order != pageblock_order); + base =PAGE_ALLOC_COSTLY_ORDER +1 ; + }
mm/page_alloc.c static void bad_page(struct page *page, const char *reason) · line 708
Quoted 10 lines
mm/page_alloc.c +10
-0+#else +VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); +#endif + +return (MIGRATE_PCPTYPES * base) + migratetype; +} + +static inline int pindex_to_order (unsigned int pindex) +{ +int order = pindex /PAGE_ALLOC_COSTLY_ORDER ;
This seems wrong, shouldn't we divide by MIGRATE_PCPTYPES? It just happens to be the same number, so testing won't flag this.
Re: [PATCH 2/2] mm/page_alloc: Allow high-order pages to be stored on the per-cpu lists
Quoted 10 lines · Vlastimil Babka
Quoted 7 lines
From a functional point of view, a patch like this is necessary to make bulk allocation of high-order pages work with similar performance to order-0 bulk allocations. The bulk allocator is not updated in this series as it would have to be determined by bulk allocation users how they want to track the order of pages allocated with the bulk allocator.
Signed-off-by Mel Gorman <[email protected]>Acked-by Vlastimil Babka <[email protected]>
Thanks!
Quoted 18 lines
Quoted 18 lines
include/linux/mmzone.h +10
-0@@ -333,6 +333,24 @@ enum zone_watermarks { NR_WMARK }; +/* +* One per migratetype for each PAGE_ALLOC_COSTLY_ORDER plus one additional +* for pageblock size for THP if configured. +*/ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define NR_PCP_THP 1 +#else +#define NR_PCP_THP 0 +#endif +#define NR_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1 + NR_PCP_THP))
Quoted 8 lines
include/linux/mmzone.h enum zone_watermarks { · line 349
Quoted 4 lines
include/linux/mmzone.h +4
-0+ +/* +* Shift to encode migratetype in order in the least significant bits and +* migratetype in the higher bits. Hard for me to understand that comment. I would describe what the code does as e,g, "Shift to encode migratetype and order in the same integer, with order in the least significant bit ..." etc.
That comment is a lot better :)
Quoted 8 lines
include/linux/mmzone.h enum zone_watermarks { · line 351
Quoted 3 lines
include/linux/mmzone.h +2
-0+*/ +#define NR_PCP_ORDER_SHIFT 8Also
ORDER_SHIFTis a bit misnomer, it's more precisely anORDER_WIDTH, and we are shifting migratetype with it, not order. I'm just comparing with how we name nid/zid/etc bits in page flags.
Ok, I can change that.
Quoted 18 lines
Quoted 18 lines
include/linux/mmzone.h +14
-0@@ -687,10 +687,53 @@ static void bad_page(struct page *page, const char *reason)add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); } +static inline unsigned int order_to_pindex (int migratetype,int order) +{ +int base = order; + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +if (order >PAGE_ALLOC_COSTLY_ORDER ) { +VM_BUG_ON (order != pageblock_order); + base =PAGE_ALLOC_COSTLY_ORDER +1 ; + } +#else +VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); +#endif + +return (MIGRATE_PCPTYPES * base) + migratetype;
Quoted 9 lines
include/linux/mmzone.h static void bad_page(struct page *page, const char *reason) · line 708
Quoted 5 lines
include/linux/mmzone.h +5
-0+} + +static inline int pindex_to_order (unsigned int pindex) +{ +int order = pindex /PAGE_ALLOC_COSTLY_ORDER ;This seems wrong, shouldn't we divide by
MIGRATE_PCPTYPES? It just happens to be the same number, so testing won't flag this.
Whoops, yes, basic algebra says you're right so well spotted. If PAGE_ALLOC_COSTLY_ORDER ever changed for some reason, this would blow up.
So this?
include/linux/mmzone.h +4 -4
@@ -345,11 +345,11 @@ enum zone_watermarks {#define NR_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1 + NR_PCP_THP))/* -* Shift to encode migratetype in order in the least significant bits and -* migratetype in the higher bits. +* Shift to encode migratetype and order in the same integer, with order +* in the least significant bits. */ -#define NR_PCP_ORDER_SHIFT 8 -#define NR_PCP_ORDER_MASK ((1<<NR_PCP_ORDER_SHIFT) - 1) +#define NR_PCP_ORDER_WIDTH 8 +#define NR_PCP_ORDER_MASK ((1<<NR_PCP_ORDER_WIDTH) - 1)#define min_wmark_pages (z) (z->_watermark[WMARK_MIN] + z->watermark_boost)#define low_wmark_pages (z) (z->_watermark[WMARK_LOW] + z->watermark_boost)
mm/page_alloc.c +4 -4
@@ -694,7 +694,7 @@ static inline unsigned int order_to_pindex(int migratetype, int order)static inline int pindex_to_order (unsigned int pindex) { -int order = pindex /PAGE_ALLOC_COSTLY_ORDER ; +int order = pindex /MIGRATE_PCPTYPES ;#ifdef CONFIG_TRANSPARENT_HUGEPAGE if (order >PAGE_ALLOC_COSTLY_ORDER ) {
@@ -1487,7 +1487,7 @@ static void free_pcppages_bulk(struct zone *zone, int count, batch_free = count; order =pindex_to_order (pindex); -BUILD_BUG_ON (MAX_ORDER >= (1 <<NR_PCP_ORDER_SHIFT )); +BUILD_BUG_ON (MAX_ORDER >= (1 <<NR_PCP_ORDER_WIDTH )); do { page = list_last_entry(list, struct page, lru); /* must delete to avoid corrupting pcp list */
@@ -1499,7 +1499,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,continue ;/* Encode order with the migratetype */ - page->index <<=NR_PCP_ORDER_SHIFT ; + page->index <<=NR_PCP_ORDER_WIDTH ; page->index |= order;list_add_tail (&page->lru, &head);
@@ -1537,7 +1537,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,/* mt has been encoded with the order (see above) */ order = mt &NR_PCP_ORDER_MASK ; - mt >>=NR_PCP_ORDER_SHIFT ; + mt >>=NR_PCP_ORDER_WIDTH ;/* MIGRATE_ISOLATE page should not go to pcplists */ VM_BUG_ON_PAGE (is_migrate_isolate (mt), page);
Signature · 2 lines
Mel Gorman SUSE Labs
Re: [PATCH 2/2] mm/page_alloc: Allow high-order pages to be stored on the per-cpu lists
Quoted 29 lines · Mel Gorman
Quoted 19 lines
Quoted 19 lines
(quoted diff) +14
-0@@ -687,10 +687,53 @@ static void bad_page(struct page *page, const char *reason) add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE); } +static inline unsigned int order_to_pindex(int migratetype, int order) +{ + int base = order; + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (order > PAGE_ALLOC_COSTLY_ORDER) { + VM_BUG_ON(order != pageblock_order); + base = PAGE_ALLOC_COSTLY_ORDER + 1; + } +#else + VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); +#endif + + return (MIGRATE_PCPTYPES * base) + migratetype;Quoted 9 lines
Review static void bad_page(struct page *page, const char *reason) · line 708
Quoted 5 lines
(quoted diff) +5
-0+} + +static inline int pindex_to_order(unsigned int pindex) +{ + int order = pindex / PAGE_ALLOC_COSTLY_ORDER;This seems wrong, shouldn't we divide by
MIGRATE_PCPTYPES? It just happens to be the same number, so testing won't flag this.
Quoted 4 lines
Whoops, yes, basic algebra says you're right so well spotted. If
PAGE_ALLOC_COSTLY_ORDERever changed for some reason, this would blow up.So this?
Yeah, great!
Quoted 63 lines
include/linux/mmzone.h +4
-4@@ -345,11 +345,11 @@ enum zone_watermarks {#define NR_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1 + NR_PCP_THP))/* -* Shift to encode migratetype in order in the least significant bits and -* migratetype in the higher bits. +* Shift to encode migratetype and order in the same integer, with order +* in the least significant bits. */ -#define NR_PCP_ORDER_SHIFT 8 -#define NR_PCP_ORDER_MASK ((1<<NR_PCP_ORDER_SHIFT) - 1) +#define NR_PCP_ORDER_WIDTH 8 +#define NR_PCP_ORDER_MASK ((1<<NR_PCP_ORDER_WIDTH) - 1)#define min_wmark_pages (z) (z->_watermark[WMARK_MIN] + z->watermark_boost)#define low_wmark_pages (z) (z->_watermark[WMARK_LOW] + z->watermark_boost)mm/page_alloc.c +4
-4@@ -694,7 +694,7 @@ static inline unsigned int order_to_pindex(int migratetype, int order)static inline int pindex_to_order (unsigned int pindex) { -int order = pindex /PAGE_ALLOC_COSTLY_ORDER ; +int order = pindex /MIGRATE_PCPTYPES ;#ifdef CONFIG_TRANSPARENT_HUGEPAGE if (order >PAGE_ALLOC_COSTLY_ORDER ) {@@ -1487,7 +1487,7 @@ static void free_pcppages_bulk(struct zone *zone, int count, batch_free = count; order =pindex_to_order (pindex); -BUILD_BUG_ON (MAX_ORDER >= (1 <<NR_PCP_ORDER_SHIFT )); +BUILD_BUG_ON (MAX_ORDER >= (1 <<NR_PCP_ORDER_WIDTH )); do { page = list_last_entry(list, struct page, lru); /* must delete to avoid corrupting pcp list */@@ -1499,7 +1499,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,continue ;/* Encode order with the migratetype */ - page->index <<=NR_PCP_ORDER_SHIFT ; + page->index <<=NR_PCP_ORDER_WIDTH ; page->index |= order;list_add_tail (&page->lru, &head);@@ -1537,7 +1537,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,/* mt has been encoded with the order (see above) */ order = mt &NR_PCP_ORDER_MASK ; - mt >>=NR_PCP_ORDER_SHIFT ; + mt >>=NR_PCP_ORDER_WIDTH ;/* MIGRATE_ISOLATE page should not go to pcplists */ VM_BUG_ON_PAGE (is_migrate_isolate (mt), page);