Per-cpu page allocator replacement prototype
Subsystems (16)
cover Per-cpu page allocator replacement prototype
Two LSF/MM's ago there was discussion on the per-cpu page allocator and whether it could be replaced due to it's complexity, frequent drains/refills and IPI overhead for global drains. The main obstacle to removal is that without those lists the zone->lock becomes very heavily contended and alternatives are inevitably going to share cache lines. I prototyped a potential replacement on the flight home and then left it on a TODO list for another year.
Last LSF/MM this was talked about in the hallway again although the meat of the discussion was different and took into account Andi Kleen's talk about lock batching. On this flight home, I rebased the old prototype, added some additional bits and pieces and tidied it up a bit since. This TODO item is about 3 years old so apparently sometimes the only way to get someone to do something is to lock them in a metal box for a few hours.
This is a prototype replacement starts with some minor optimisations that have nothing to do with anything really other than they were floating around from another flights worth of work. It then replaces the per-cpu page allocator with an IRQ-unsafe (no interrupts, no calls with local_irq_save) magazine that is protected by a spinlock. This effectively has the allocator use two locks. magazine_lock for non-IRQ users (e.g. page faults) and the zone->lock for magazine drains/refills and users who have IRQs disabled (interrupts, slab). It then splits the magazine into two where the preferred magazine depends on the CPU id of the executing thread. Non-preferred magazines may also be used and are searched round-robin as they are only protected by spinlocks. The last part of the series does some mucking around with lock contention and batching multiple frees due to exit or compaction under the lock.
It has not been heavily tested in low memory, heavy interrupt situations or extensively with all the debugging options enabled so it's likely there are bugs hiding in there. However I'm interested in hearing if the per-cpu page allocator is something we really want to replace or if there is a better potential alternative than this prototype. There are some interesting possibilities with this sort of design. For example, it would be possible to allocate magazines to allocator-intensive processes that are chained together for global drains where they are necessary. For these processes there would be much less contention (only on drains/refills) without having to use IPIs to drain their pages.
In the follow tests, no debugging was enabled but profiling was running so the tests are heavily disrupted. Take the results with a grain of salt. Machine was a single socket i7 with 16G RAM.
kernbench
3.9.0 3.9.0
vanilla magazine
User min 883.59 ( 0.00%) 700.16 ( 20.76%)
User mean 891.11 ( 0.00%) 741.02 ( 16.84%)
User stddev 12.14 ( 0.00%) 50.04 (-312.18%)
User max 915.30 ( 0.00%) 817.50 ( 10.69%)
User range 31.71 ( 0.00%) 117.34 (-270.04%)
System min 58.85 ( 0.00%) 43.65 ( 25.83%)
System mean 59.65 ( 0.00%) 47.65 ( 20.12%)
System stddev 1.35 ( 0.00%) 4.79 (-254.39%)
System max 62.35 ( 0.00%) 55.35 ( 11.23%)
System range 3.50 ( 0.00%) 11.70 (-234.29%)
Elapsed min 127.75 ( 0.00%) 99.37 ( 22.22%)
Elapsed mean 129.19 ( 0.00%) 105.73 ( 18.16%)
Elapsed stddev 2.07 ( 0.00%) 7.56 (-265.78%)
Elapsed max 133.25 ( 0.00%) 117.53 ( 11.80%)
Elapsed range 5.50 ( 0.00%) 18.16 (-230.18%)
CPU min 731.00 ( 0.00%) 742.00 ( -1.50%)
CPU mean 735.20 ( 0.00%) 745.60 ( -1.41%)
CPU stddev 2.99 ( 0.00%) 3.38 (-12.99%)
CPU max 739.00 ( 0.00%) 751.00 ( -1.62%)
CPU range 8.00 ( 0.00%) 9.00 (-12.50%)Kernel build benchmark seemed fairly successful, if anything they seem too good and I suspect this might have been a particularly "lucky" run. A non-profiling run might reveal more.
pagealloc
3.9.0 3.9.0
vanilla magazine
order-0 alloc-1 671.11 ( 0.00%) 734.11 ( -9.39%)
order-0 alloc-2 520.33 ( 0.00%) 517.56 ( 0.53%)
order-0 alloc-4 426.56 ( 0.00%) 461.78 ( -8.26%)
order-0 alloc-8 666.33 ( 0.00%) 381.67 ( 42.72%)
order-0 alloc-16 341.78 ( 0.00%) 354.44 ( -3.71%)
order-0 alloc-32 336.89 ( 0.00%) 345.33 ( -2.51%)
order-0 alloc-64 331.33 ( 0.00%) 324.56 ( 2.05%)
order-0 alloc-128 324.56 ( 0.00%) 325.89 ( -0.41%)
order-0 alloc-256 350.44 ( 0.00%) 326.22 ( 6.91%)
order-0 alloc-512 369.33 ( 0.00%) 345.67 ( 6.41%)
order-0 alloc-1024 381.44 ( 0.00%) 352.67 ( 7.54%)
order-0 alloc-2048 387.50 ( 0.00%) 348.00 ( 10.19%)
order-0 alloc-4096 403.00 ( 0.00%) 384.50 ( 4.59%)
order-0 alloc-8192 413.00 ( 0.00%) 383.00 ( 7.26%)
order-0 alloc-16384 411.00 ( 0.00%) 396.80 ( 3.45%)
order-0 free-1 357.22 ( 0.00%) 458.89 (-28.46%)
order-0 free-2 285.11 ( 0.00%) 349.89 (-22.72%)
order-0 free-4 231.33 ( 0.00%) 296.00 (-27.95%)
order-0 free-8 371.56 ( 0.00%) 229.33 ( 38.28%)
order-0 free-16 189.78 ( 0.00%) 212.89 (-12.18%)
order-0 free-32 185.67 ( 0.00%) 206.11 (-11.01%)
order-0 free-64 178.44 ( 0.00%) 197.78 (-10.83%)
order-0 free-128 178.11 ( 0.00%) 197.00 (-10.61%)
order-0 free-256 227.89 ( 0.00%) 196.56 ( 13.75%)
order-0 free-512 280.11 ( 0.00%) 194.67 ( 30.50%)
order-0 free-1024 301.67 ( 0.00%) 227.50 ( 24.59%)
order-0 free-2048 325.50 ( 0.00%) 238.00 ( 26.88%)
order-0 free-4096 328.00 ( 0.00%) 278.25 ( 15.17%)
order-0 free-8192 337.50 ( 0.00%) 277.00 ( 17.93%)
order-0 free-16384 338.00 ( 0.00%) 285.20 ( 15.62%)
order-0 total-1 1031.89 ( 0.00%) 1197.22 (-16.02%)
order-0 total-2 805.44 ( 0.00%) 867.44 ( -7.70%)
order-0 total-4 657.89 ( 0.00%) 768.67 (-16.84%)
order-0 total-8 1039.56 ( 0.00%) 611.00 ( 41.22%)
order-0 total-16 531.56 ( 0.00%) 569.33 ( -7.11%)
order-0 total-32 523.44 ( 0.00%) 551.44 ( -5.35%)
order-0 total-64 509.78 ( 0.00%) 522.89 ( -2.57%)
order-0 total-128 502.67 ( 0.00%) 522.89 ( -4.02%)
order-0 total-256 579.56 ( 0.00%) 522.78 ( 9.80%)
order-0 total-512 649.44 ( 0.00%) 540.33 ( 16.80%)
order-0 total-1024 686.44 ( 0.00%) 580.17 ( 15.48%)
order-0 total-2048 713.00 ( 0.00%) 586.00 ( 17.81%)
order-0 total-4096 731.00 ( 0.00%) 663.75 ( 9.20%)
order-0 total-8192 750.50 ( 0.00%) 660.00 ( 12.06%)
order-0 total-16384 749.00 ( 0.00%) 683.00 ( 8.81%)This is a systemtap-drive page allocator microbenchmark that allocates order-0 pages in increasingly large "batches" and times the length of time it takes to allocate and free. The results are a bit all over the map. Frees are slower because they always wait for the preferred magazine to be free and freeing single pages in a loop is a worst-case scenario. For larger patches it tends to perform reasonably well though.
pft
3.9.0 3.9.0
vanilla magazine
Faults/cpu 1 953441.5530 ( 0.00%) 954011.8576 ( 0.06%)
Faults/cpu 2 923793.1533 ( 0.00%) 889436.7293 ( -3.72%)
Faults/cpu 3 876829.2292 ( 0.00%) 868230.6471 ( -0.98%)
Faults/cpu 4 819914.9333 ( 0.00%) 735264.4793 (-10.32%)
Faults/cpu 5 689049.0107 ( 0.00%) 663481.5446 ( -3.71%)
Faults/cpu 6 579924.4065 ( 0.00%) 571889.3687 ( -1.39%)
Faults/cpu 7 552024.1040 ( 0.00%) 494345.8698 (-10.45%)
Faults/cpu 8 461452.7560 ( 0.00%) 457877.1810 ( -0.77%)
Faults/sec 1 938245.7112 ( 0.00%) 939198.4047 ( 0.10%)
Faults/sec 2 1814498.4087 ( 0.00%) 1748800.3800 ( -3.62%)
Faults/sec 3 2544466.6368 ( 0.00%) 2359068.2163 ( -7.29%)
Faults/sec 4 3032778.8584 ( 0.00%) 2831753.6553 ( -6.63%)
Faults/sec 5 3025180.2736 ( 0.00%) 2952758.4589 ( -2.39%)
Faults/sec 6 3131131.0106 ( 0.00%) 3058954.4941 ( -2.31%)
Faults/sec 7 3286271.0631 ( 0.00%) 3183931.7940 ( -3.11%)
Faults/sec 8 3135331.0027 ( 0.00%) 3106746.5908 ( -0.91%)This is a page faulting microbenchmark that is forced to use base pages only. Here the new design suffers a bit because the allocation path is likely to contend on the magazine lock.
So preliminary testing indicates the results are mixed bag. As long as locks are not contended, it performs fine but parallel fault testing hits into spinlock contention on the magazine locks. A greater problem is that because CPUs share magazines it means that the struct pages are frequently dirtied cache lines. If CPU A frees a page to a magazine and CPU B immediately allocates it then the cache line for the page and the magazine bounces and this costs. It's on the TODO list to research if the available literature has anything useful to say that does not depend on per-cpu lists and the associated problems with them.
Comments?
arch/sparc/mm/init_64.c | 4 +- arch/sparc/mm/tsb.c | 2 +- arch/tile/mm/homecache.c | 2 +- fs/fuse/dev.c | 2 +- include/linux/gfp.h | 12 +- include/linux/mm.h | 3 - include/linux/mmzone.h | 46 +- include/linux/page-isolation.h | 7 +- include/linux/pagemap.h | 2 +- include/linux/swap.h | 2 +- include/trace/events/kmem.h | 22 +- init/main.c | 1 - kernel/power/snapshot.c | 2 - kernel/sysctl.c | 10 - mm/compaction.c | 18 +- mm/memory-failure.c | 2 +- mm/memory_hotplug.c | 13 +- mm/page_alloc.c | 1109 +++++++++++++++++----------------------- mm/page_isolation.c | 30 +- mm/rmap.c | 2 +- mm/swap.c | 6 +- mm/swap_state.c | 2 +- mm/vmscan.c | 6 +- mm/vmstat.c | 54 +- 24 files changed, 571 insertions(+), 788 deletions(-)
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/22 mm: page allocator: Lookup pageblock migratetype with IRQs enabled during free
get_pageblock_migratetype() is called during free with IRQs disabled. This is unnecessary and disables IRQs for longer than necessary.
--- mm/page_alloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/page_alloc.c +2 -1
@@ -730,9 +730,10 @@ static void __free_pages_ok(struct page *page, unsigned int order)if (!free_pages_prepare (page, order))return ; + migratetype =get_pageblock_migratetype (page); +local_irq_save (flags);__count_vm_events (PGFREE ,1 << order); - migratetype =get_pageblock_migratetype (page);set_freepage_migratetype (page, migratetype);free_one_page (page_zone (page), page, order, migratetype);local_irq_restore (flags);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/22 mm: page allocator: Push down where IRQs are disabled during page free
In reply to Mel GormanThis patch pushes IRQ disabling down into free_one_page(). This simplifies the logic in free_hot_cold_page() slightly by making it clear that zone->lock is an IRQ-safe spinlock. The current arrangement has the IRQ disabling happen in one function and the spinlock been taken in another. Functionally, there is no difference.
--- mm/page_alloc.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-)
mm/page_alloc.c +10 -11
@@ -686,14 +686,18 @@ static void free_pcppages_bulk(struct zone *zone, int count,static void free_one_page (struct zone *zone,struct page *page,int order,int migratetype) { -spin_lock (&zone->lock); +unsigned long flags; +set_freepage_migratetype (page, migratetype); + +spin_lock_irqsave (&zone->lock, flags); zone->all_unreclaimable =0 ; zone->pages_scanned =0 ; +__count_vm_events (PGFREE ,1 << order);__free_one_page (page, zone, order, migratetype);if (unlikely (!is_migrate_isolate (migratetype)))__mod_zone_freepage_state (zone,1 << order, migratetype); -spin_unlock (&zone->lock); +spin_unlock_irqrestore (&zone->lock, flags); }static bool free_pages_prepare (struct page *page,unsigned int order)
@@ -724,7 +728,6 @@ static bool free_pages_prepare(struct page *page, unsigned int order)static void __free_pages_ok (struct page *page,unsigned int order) { -unsigned long flags;int migratetype;if (!free_pages_prepare (page, order))
@@ -732,11 +735,7 @@ static void __free_pages_ok(struct page *page, unsigned int order) migratetype =get_pageblock_migratetype (page); -local_irq_save (flags); -__count_vm_events (PGFREE ,1 << order); -set_freepage_migratetype (page, migratetype);free_one_page (page_zone (page), page, order, migratetype); -local_irq_restore (flags); }/*
@@ -1325,8 +1324,6 @@ void free_hot_cold_page(struct page *page, int cold) migratetype =get_pageblock_migratetype (page);set_freepage_migratetype (page, migratetype); -local_irq_save (flags); -__count_vm_event (PGFREE );/* * We only track unmovable, reclaimable and movable on pcp lists.
@@ -1338,11 +1335,14 @@ void free_hot_cold_page(struct page *page, int cold)if (migratetype >=MIGRATE_PCPTYPES ) {if (unlikely (is_migrate_isolate (migratetype))) {free_one_page (zone, page,0 , migratetype); - goto out; +return ; } migratetype =MIGRATE_MOVABLE ; } +local_irq_save (flags); +__count_vm_event (PGFREE ); + pcp = &this_cpu_ptr (zone->pageset)->pcp;if (cold)list_add_tail (&page->lru, &pcp->lists[migratetype]);
@@ -1354,7 +1354,6 @@ void free_hot_cold_page(struct page *page, int cold) pcp->count -= pcp->batch; } -out:local_irq_restore (flags); }
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/22 mm: page allocator: Use unsigned int for order in more places
In reply to Mel GormanX86 prefers the use of unsigned types for iterators and there is a tendency to mix whether a signed or unsigned type if used for page order. This converts a number of sites in mm/page_alloc.c to use unsigned int for order where possible.
--- include/linux/mmzone.h | 8 ++++---- mm/page_alloc.c | 35 +++++++++++++++++++---------------- 2 files changed, 23 insertions(+), 20 deletions(-)
include/linux/mmzone.h +4 -4
@@ -773,10 +773,10 @@ static inline bool pgdat_is_empty(pg_data_t *pgdat)extern struct mutex zonelists_mutex;void build_all_zonelists (pg_data_t *pgdat,struct zone *zone);void wakeup_kswapd (struct zone *zone,int order,enum zone_type classzone_idx); -bool zone_watermark_ok (struct zone *z,int order,unsigned long mark, -int classzone_idx,int alloc_flags); -bool zone_watermark_ok_safe (struct zone *z,int order,unsigned long mark, -int classzone_idx,int alloc_flags); +bool zone_watermark_ok (struct zone *z,unsigned int order, +unsigned long mark,int classzone_idx,int alloc_flags); +bool zone_watermark_ok_safe (struct zone *z,unsigned int order, +unsigned long mark,int classzone_idx,int alloc_flags);enum memmap_context {MEMMAP_EARLY ,MEMMAP_HOTPLUG ,
mm/page_alloc.c +19 -16
@@ -392,7 +392,8 @@ static int destroy_compound_page(struct page *page, unsigned long order)return bad; } -static inline void prep_zero_page (struct page *page,int order,gfp_t gfp_flags) +static inline void prep_zero_page (struct page *page,unsigned int order, +gfp_t gfp_flags) {int i;
@@ -436,7 +437,7 @@ static inline void set_page_guard_flag(struct page *page) { }static inline void clear_page_guard_flag (struct page *page) { }#endif -static inline void set_page_order (struct page *page,int order) +static inline void set_page_order (struct page *page,unsigned int order) {set_page_private (page, order);__SetPageBuddy (page);
@@ -485,7 +486,7 @@ __find_buddy_index(unsigned long page_idx, unsigned int order)* For recording page's order, we use page_private(page). */ static inline int page_is_buddy (struct page *page,struct page *buddy, -int order) +unsigned int order) {if (!pfn_valid_within (page_to_pfn (buddy)))return 0 ;
@@ -683,7 +684,8 @@ static void free_pcppages_bulk(struct zone *zone, int count,spin_unlock (&zone->lock); } -static void free_one_page (struct zone *zone,struct page *page,int order, +static void free_one_page (struct zone *zone,struct page *page, +unsigned int order,int migratetype) {unsigned long flags;
@@ -853,7 +855,7 @@ static inline int check_new_page(struct page *page)return 0 ; } -static int prep_new_page (struct page *page,int order,gfp_t gfp_flags) +static int prep_new_page (struct page *page,unsigned int order,gfp_t gfp_flags) {int i;
@@ -1278,7 +1280,7 @@ void mark_free_pages(struct zone *zone) {unsigned long pfn, max_zone_pfn;unsigned long flags; -int order, t; +unsigned int order, t;struct list_head *curr;if (!zone->spanned_pages)
@@ -1471,8 +1473,8 @@ int split_free_page(struct page *page)*/ static inline struct page *buffered_rmqueue (struct zone *preferred_zone, -struct zone *zone,int order,gfp_t gfp_flags, -int migratetype) +struct zone *zone,unsigned int order, +gfp_t gfp_flags,int migratetype) {unsigned long flags;struct page *page;
@@ -1619,8 +1621,9 @@ static inline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)* Return true if free pages are above 'mark'. This takes into account the order * of the allocation. */ -static bool __zone_watermark_ok (struct zone *z,int order,unsigned long mark, -int classzone_idx,int alloc_flags,long free_pages) +static bool __zone_watermark_ok (struct zone *z,unsigned int order, +unsigned long mark,int classzone_idx,int alloc_flags, +long free_pages) {/* free_pages my go negative - that's OK */ long min = mark;
@@ -1652,15 +1655,15 @@ static bool __zone_watermark_ok(struct zone *z, int order, unsigned long mark,return true; } -bool zone_watermark_ok (struct zone *z,int order,unsigned long mark, +bool zone_watermark_ok (struct zone *z,unsigned int order,unsigned long mark,int classzone_idx,int alloc_flags) {return __zone_watermark_ok (z, order, mark, classzone_idx, alloc_flags,zone_page_state (z,NR_FREE_PAGES )); } -bool zone_watermark_ok_safe (struct zone *z,int order,unsigned long mark, -int classzone_idx,int alloc_flags) +bool zone_watermark_ok_safe (struct zone *z,unsigned int order, +unsigned long mark,int classzone_idx,int alloc_flags) {long free_pages =zone_page_state (z,NR_FREE_PAGES );
@@ -3942,7 +3945,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,static void __meminit zone_init_free_lists (struct zone *zone) { -int order, t; +unsigned int order, t;for_each_migratetype_order (order, t) {INIT_LIST_HEAD (&zone->free_area[order].free_list[t]); zone->free_area[order].nr_free =0 ;
@@ -6038,7 +6041,7 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn) {struct page *page;struct zone *zone; -int order, i; +unsigned int order, i;unsigned long pfn;unsigned long flags;/* find the first valid pfn */
@@ -6090,7 +6093,7 @@ bool is_free_buddy_page(struct page *page)struct zone *zone =page_zone (page);unsigned long pfn =page_to_pfn (page);unsigned long flags; -int order; +unsigned int order;spin_lock_irqsave (&zone->lock, flags);for (order =0 ; order <MAX_ORDER ; order++) {
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/22 mm: page allocator: Only check migratetype of pages being drained while CMA active
In reply to Mel GormanCMA added a is_migrate_isolate_page in the bulk page free path which does a pageblock migratetype lookup for every page being drained. This is only necessary when CMA is active so skip the expensive checks in the normal case.
--- include/linux/mmzone.h | 8 ++++++-- include/linux/page-isolation.h | 7 ++++--- mm/page_alloc.c | 2 +- mm/page_isolation.c | 27 +++++++++++++++++++++++---- 4 files changed, 34 insertions(+), 10 deletions(-)
include/linux/mmzone.h +6 -2
@@ -354,12 +354,16 @@ struct zone {spinlock_t lock;int all_unreclaimable;/* All pages pinned */ #if definedCONFIG_COMPACTION || definedCONFIG_CMA +unsigned long compact_cached_free_pfn; +unsigned long compact_cached_migrate_pfn; +/* Set to true when the PG_migrate_skip bits should be cleared */ bool compact_blockskip_flush;/* pfns where compaction scanners should start */ -unsigned long compact_cached_free_pfn; -unsigned long compact_cached_migrate_pfn; +#endif +#ifdef CONFIG_MEMORY_ISOLATION +bool memory_isolation_active;#endif #ifdef CONFIG_MEMORY_HOTPLUG /* see spanned/present_pages for more description */
include/linux/page-isolation.h +4 -3
@@ -2,16 +2,17 @@#define __LINUX_PAGEISOLATION_H#ifdef CONFIG_MEMORY_ISOLATION -static inline bool is_migrate_isolate_page (struct page *page) +static inline bool is_migrate_isolate_page (struct zone *zone,struct page *page) { -return get_pageblock_migratetype (page) ==MIGRATE_ISOLATE ; +return zone->memory_isolation_active && +get_pageblock_migratetype (page) ==MIGRATE_ISOLATE ; }static inline bool is_migrate_isolate (int migratetype) {return migratetype ==MIGRATE_ISOLATE ; }#else -static inline bool is_migrate_isolate_page (struct page *page) +static inline bool is_migrate_isolate_page (struct zone *zone,struct page *page) {return false; }
mm/page_alloc.c +1 -1
@@ -674,7 +674,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */ __free_one_page (page, zone,0 , mt);trace_mm_page_pcpu_drain (page,0 , mt); -if (likely (!is_migrate_isolate_page (page))) { +if (likely (!is_migrate_isolate_page (zone, page))) {__mod_zone_page_state (zone,NR_FREE_PAGES ,1 );if (is_migrate_cma (mt))__mod_zone_page_state (zone,NR_FREE_CMA_PAGES ,1 );
mm/page_isolation.c +23 -4
@@ -118,6 +118,8 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,unsigned long pfn;unsigned long undo_pfn;struct page *page; +struct zone *zone =NULL ; +unsigned long flags;BUG_ON ((start_pfn) & (pageblock_nr_pages -1 ));BUG_ON ((end_pfn) & (pageblock_nr_pages -1 ));
@@ -126,12 +128,20 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, pfn < end_pfn; pfn += pageblock_nr_pages) { page =__first_valid_page (pfn, pageblock_nr_pages); -if (page && -set_migratetype_isolate (page, skip_hwpoisoned_pages)) { - undo_pfn = pfn; - goto undo; +if (page) { +if (!zone) + zone =page_zone (page); +if (set_migratetype_isolate (page, + skip_hwpoisoned_pages)) { + undo_pfn = pfn; + goto undo; + } } } + +spin_lock_irqsave (&zone->lock, flags); + zone->memory_isolation_active = true; +spin_unlock_irqrestore (&zone->lock, flags);return 0 ; undo: for (pfn = start_pfn;
@@ -150,6 +160,9 @@ int undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, {unsigned long pfn;struct page *page; +struct zone *zone =NULL ; +unsigned long flags; +BUG_ON ((start_pfn) & (pageblock_nr_pages -1 ));BUG_ON ((end_pfn) & (pageblock_nr_pages -1 )); for (pfn = start_pfn;
@@ -159,7 +172,13 @@ int undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,if (!page ||get_pageblock_migratetype (page) !=MIGRATE_ISOLATE )continue ;unset_migratetype_isolate (page, migratetype); +if (!zone) + zone =page_zone (page); } + +spin_lock_irqsave (&zone->lock, flags); + zone->memory_isolation_active = true; +spin_unlock_irqrestore (&zone->lock, flags);return 0 ; }/*
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/22 oom: Use number of online nodes when deciding whether to suppress messages
In reply to Mel GormanCommit 29423e77 (oom: suppress show_mem() for many nodes in irq context on page alloc failure) was meant to suppress printing excessive amounts of information in IRQ context on large machines. However, it uses a kernel config variable which the maximum supported number of nodes, not the number of online nodes to make the decision. Effectively, on some distribution configurations the message will be suppressed even on small machines. This patch uses nr_online_nodes to decide whether to suppress messges.
--- mm/page_alloc.c | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-)
mm/page_alloc.c +9 -16
@@ -1978,20 +1978,6 @@ this_zone_full:return page; } -/* -* Large machines with many possible nodes should not always dump per-node -* meminfo in irq context. -*/ -static inline bool should_suppress_show_mem (void ) -{ -bool ret = false; - -#if NODES_SHIFT >8 - ret =in_interrupt (); -#endif -return ret; -} - static DEFINE_RATELIMIT_STATE(nopage_rs, DEFAULT_RATELIMIT_INTERVAL, DEFAULT_RATELIMIT_BURST);
@@ -2034,8 +2020,15 @@ void warn_alloc_failed(gfp_t gfp_mask, int order, const char *fmt, ...) current->comm, order, gfp_mask);dump_stack (); -if (!should_suppress_show_mem ()) -show_mem (filter); + +/* +* Large machines with many possible nodes should not always dump +* per-node meminfo in irq context. +*/ +if (in_interrupt () && nr_online_nodes > (1 <<8 )) +return ; + +show_mem (filter); } static inline int
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/22 mm: page allocator: Convert hot/cold parameter and immediate callers to bool
In reply to Mel Gormancold is a bool, make it one for consistency later.
--- arch/sparc/mm/init_64.c | 4 ++-- arch/sparc/mm/tsb.c | 2 +- arch/tile/mm/homecache.c | 2 +- fs/fuse/dev.c | 2 +- include/linux/gfp.h | 4 ++-- include/linux/pagemap.h | 2 +- include/linux/swap.h | 2 +- mm/page_alloc.c | 6 +++--- mm/swap.c | 4 ++-- mm/swap_state.c | 2 +- mm/vmscan.c | 6 +++--- 11 files changed, 18 insertions(+), 18 deletions(-)
arch/sparc/mm/init_64.c +2 -2
@@ -2562,7 +2562,7 @@ void pte_free_kernel(struct mm_struct *mm, pte_t *pte) {struct page *page =virt_to_page (pte);if (put_page_testzero (page)) -free_hot_cold_page (page,0 ); +free_hot_cold_page (page, false); }static void __pte_free (pgtable_t pte)
@@ -2570,7 +2570,7 @@ static void __pte_free(pgtable_t pte)struct page *page =virt_to_page (pte);if (put_page_testzero (page)) {pgtable_page_dtor (page); -free_hot_cold_page (page,0 ); +free_hot_cold_page (page, false); } }
arch/sparc/mm/tsb.c +1 -1
@@ -520,7 +520,7 @@ void destroy_context(struct mm_struct *mm) page = mm->context.pgtable_page;if (page &&put_page_testzero (page)) {pgtable_page_dtor (page); -free_hot_cold_page (page,0 ); +free_hot_cold_page (page, false); }spin_lock_irqsave (&ctx_alloc_lock, flags);
arch/tile/mm/homecache.c +1 -1
@@ -438,7 +438,7 @@ void __homecache_free_pages(struct page *page, unsigned int order)if (put_page_testzero (page)) {homecache_change_page_home (page, order,initial_page_home ());if (order ==0 ) { -free_hot_cold_page (page,0 ); +free_hot_cold_page (page, false); }else {init_page_count (page);__free_pages (page, order);
fs/fuse/dev.c +1 -1
@@ -1580,7 +1580,7 @@ out_finish:static void fuse_retrieve_end (struct fuse_conn *fc,struct fuse_req *req) { -release_pages (req->pages, req->num_pages,0 ); +release_pages (req->pages, req->num_pages, false); } static int fuse_retrieve(struct fuse_conn *fc, struct inode *inode,
include/linux/gfp.h +2 -2
@@ -364,8 +364,8 @@ void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask);extern void __free_pages (struct page *page,unsigned int order);extern void free_pages (unsigned long addr,unsigned int order); -extern void free_hot_cold_page (struct page *page,int cold); -extern void free_hot_cold_page_list (struct list_head *list,int cold); +extern void free_hot_cold_page (struct page *page,bool cold); +extern void free_hot_cold_page_list (struct list_head *list,bool cold);extern void __free_memcg_kmem_pages (struct page *page,unsigned int order);extern void free_memcg_kmem_pages (unsigned long addr,unsigned int order);
include/linux/pagemap.h +1 -1
@@ -99,7 +99,7 @@ static inline void mapping_set_gfp_mask(struct address_space *m, gfp_t mask)#define page_cache_get (page) get_page(page)#define page_cache_release (page) put_page(page) -void release_pages (struct page **pages,int nr,int cold); +void release_pages (struct page **pages,int nr,bool cold);/* * speculatively take a reference to a page.
include/linux/swap.h +1 -1
@@ -414,7 +414,7 @@ mem_cgroup_uncharge_swapcache(struct page *page, swp_entry_t ent, bool swapout)#define free_page_and_swap_cache (page) \ page_cache_release(page)#define free_pages_and_swap_cache (pages, nr) \ - release_pages((pages), (nr), 0); + release_pages((pages), (nr), false);static inline void show_swap_cache_info (void ) {
mm/page_alloc.c +3 -3
@@ -1314,7 +1314,7 @@ void mark_free_pages(struct zone *zone)* Free a 0-order page * cold == 1 ? free a cold page : free a hot page */ -void free_hot_cold_page (struct page *page,int cold) +void free_hot_cold_page (struct page *page,bool cold) {struct zone *zone =page_zone (page);struct per_cpu_pages *pcp;
@@ -1362,7 +1362,7 @@ void free_hot_cold_page(struct page *page, int cold)/* * Free a list of 0-order pages */ -void free_hot_cold_page_list (struct list_head *list,int cold) +void free_hot_cold_page_list (struct list_head *list,bool cold) {struct page *page, *next;
@@ -2680,7 +2680,7 @@ void __free_pages(struct page *page, unsigned int order) {if (put_page_testzero (page)) {if (order ==0 ) -free_hot_cold_page (page,0 ); +free_hot_cold_page (page, false);else __free_pages_ok (page, order); }
mm/swap.c +2 -2
@@ -63,7 +63,7 @@ static void __page_cache_release(struct page *page)static void __put_single_page (struct page *page) {__page_cache_release (page); -free_hot_cold_page (page,0 ); +free_hot_cold_page (page, false); }static void __put_compound_page (struct page *page)
@@ -667,7 +667,7 @@ int lru_add_drain_all(void)* grabbed the page via the LRU. If it did, give up: shrink_inactive_list() * will free it. */ -void release_pages (struct page **pages,int nr,int cold) +void release_pages (struct page **pages,int nr,bool cold) {int i;LIST_HEAD (pages_to_free);
mm/swap_state.c +1 -1
@@ -268,7 +268,7 @@ void free_pages_and_swap_cache(struct page **pages, int nr)for (i =0 ; i < todo; i++)free_swap_cache (pagep[i]); -release_pages (pagep, todo,0 ); +release_pages (pagep, todo, false); pagep += todo; nr -= todo; }
mm/vmscan.c +3 -3
@@ -954,7 +954,7 @@ keep:if (nr_dirty && nr_dirty == nr_congested &&global_reclaim (sc))zone_set_flag (zone,ZONE_CONGESTED ); -free_hot_cold_page_list (&free_pages,1 ); +free_hot_cold_page_list (&free_pages, true);list_splice (&ret_pages, page_list);count_vm_events (PGACTIVATE , pgactivate);
@@ -1343,7 +1343,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,spin_unlock_irq (&zone->lru_lock); -free_hot_cold_page_list (&page_list,1 ); +free_hot_cold_page_list (&page_list, true);/* * If reclaim is isolating dirty pages under writeback, it implies
@@ -1534,7 +1534,7 @@ static void shrink_active_list(unsigned long nr_to_scan,__mod_zone_page_state (zone,NR_ISOLATED_ANON + file, -nr_taken);spin_unlock_irq (&zone->lru_lock); -free_hot_cold_page_list (&l_hold,1 ); +free_hot_cold_page_list (&l_hold, true); }#ifdef CONFIG_SWAP
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 7/22 mm: page allocator: Do not lookup the pageblock migratetype during allocation
In reply to Mel GormanThe pageblock migrate type is checked during allocation in case it's CMA but using get_freepage_migratetype() should be sufficient as the magazines would have been drained prior to a CMA allocation attempt.
--- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
@@ -1523,7 +1523,7 @@ again:if (!page) goto failed;__mod_zone_freepage_state (zone, -(1 << order), -get_pageblock_migratetype (page)); +get_freepage_migratetype (page)); }__count_zone_vm_events (PGALLOC , zone,1 << order);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 8/22 mm: page allocator: Remove the per-cpu page allocator
In reply to Mel GormanThis patch removes the per-cpu page allocator in preparation for placing a different order-0 allocator in front of the buddy allocator. This is to simplify review.
--- include/linux/gfp.h | 3 - include/linux/mm.h | 3 - include/linux/mmzone.h | 12 - include/trace/events/kmem.h | 11 - init/main.c | 1 - kernel/power/snapshot.c | 2 - kernel/sysctl.c | 10 - mm/memory-failure.c | 1 - mm/memory_hotplug.c | 11 +- mm/page_alloc.c | 581 ++------------------------------------------ mm/page_isolation.c | 2 - mm/vmstat.c | 39 +-- 12 files changed, 29 insertions(+), 647 deletions(-)
include/linux/gfp.h +0 -3
@@ -374,9 +374,6 @@ extern void free_memcg_kmem_pages(unsigned long addr, unsigned int order);#define free_page (addr) free_pages((addr), 0)void page_alloc_init (void ); -void drain_zone_pages (struct zone *zone,struct per_cpu_pages *pcp); -void drain_all_pages (void ); -void drain_local_pages (void *dummy);/* * gfp_allowed_mask is set to GFP_BOOT_MASK during early boot to restrict what
include/linux/mm.h +0 -3
@@ -1367,9 +1367,6 @@ void warn_alloc_failed(gfp_t gfp_mask, int order, const char *fmt, ...);extern void setup_per_cpu_pageset (void ); -extern void zone_pcp_update (struct zone *zone); -extern void zone_pcp_reset (struct zone *zone); -/* page_alloc.c */ extern int min_free_kbytes;
include/linux/mmzone.h +0 -12
@@ -235,17 +235,7 @@ enum zone_watermarks {#define low_wmark_pages (z) (z->watermark[WMARK_LOW])#define high_wmark_pages (z) (z->watermark[WMARK_HIGH]) -struct per_cpu_pages { -int count;/* number of pages in the list */ -int high;/* high watermark, emptying needed */ -int batch;/* chunk size for buddy add/remove */ - -/* Lists of pages, one per migrate type stored on the pcp-lists */ -struct list_head lists[MIGRATE_PCPTYPES ]; -}; -struct per_cpu_pageset { -struct per_cpu_pages pcp;#ifdef CONFIG_NUMA s8 expire;#endif
@@ -900,8 +890,6 @@ int min_free_kbytes_sysctl_handler(struct ctl_table *, int,extern int sysctl_lowmem_reserve_ratio[MAX_NR_ZONES -1 ];int lowmem_reserve_ratio_sysctl_handler (struct ctl_table *,int ,void __user *,size_t *,loff_t *); -int percpu_pagelist_fraction_sysctl_handler (struct ctl_table *,int , -void __user *,size_t *,loff_t *);int sysctl_min_unmapped_ratio_sysctl_handler (struct ctl_table *,int ,void __user *,size_t *,loff_t *); int sysctl_min_slab_ratio_sysctl_handler(struct ctl_table *, int,
include/trace/events/kmem.h +0 -11
@@ -253,17 +253,6 @@ DEFINE_EVENT(mm_page, mm_page_alloc_zone_locked,TP_ARGS (page, order, migratetype) ); -DEFINE_EVENT_PRINT (mm_page, mm_page_pcpu_drain, - - TP_PROTO(struct page *page, unsigned int order, int migratetype), - -TP_ARGS (page, order, migratetype), - -TP_printk ("page=%p pfn=%lu order=%d migratetype=%d" , - __entry->page,page_to_pfn (__entry->page), - __entry->order, __entry->migratetype) -); - TRACE_EVENT(mm_page_alloc_extfrag, TP_PROTO(struct page *page,
init/main.c +0 -1
@@ -597,7 +597,6 @@ asmlinkage void __init start_kernel(void)page_cgroup_init ();debug_objects_mem_init ();kmemleak_init (); -setup_per_cpu_pageset ();numa_policy_init ();if (late_time_init)late_time_init ();
kernel/power/snapshot.c +0 -2
@@ -1582,7 +1582,6 @@ asmlinkage int swsusp_save(void)printk (KERN_INFO "PM: Creating hibernation image:\n" ); -drain_local_pages (NULL ); nr_pages =count_data_pages (); nr_highmem =count_highmem_pages ();printk (KERN_INFO "PM: Need to copy %u pages\n" , nr_pages + nr_highmem);
@@ -1600,7 +1599,6 @@ asmlinkage int swsusp_save(void)/* During allocating of suspend pagedir, new cold pages may appear. * Kill them. */ -drain_local_pages (NULL );copy_data_pages (©_bm, &orig_bm);/*
kernel/sysctl.c +0 -10
@@ -107,7 +107,6 @@ extern unsigned int core_pipe_limit;extern int pid_max;extern int pid_max_min, pid_max_max;extern int sysctl_drop_caches; -extern int percpu_pagelist_fraction;extern int compat_log;extern int latencytop_enabled;extern int sysctl_nr_open_min, sysctl_nr_open_max;
@@ -140,7 +139,6 @@ static unsigned long dirty_bytes_min = 2 * PAGE_SIZE;/* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */ static int maxolduid =65535 ;static int minolduid; -static int min_percpu_pagelist_fract =8 ;static int ngroups_max =NGROUPS_MAX ;static const int cap_last_cap =CAP_LAST_CAP ;
@@ -1266,14 +1264,6 @@ static struct ctl_table vm_table[] = { .proc_handler = min_free_kbytes_sysctl_handler, .extra1 = &zero, }, - { - .procname = "percpu_pagelist_fraction", - .data = &percpu_pagelist_fraction, - .maxlen = sizeof(percpu_pagelist_fraction), - .mode = 0644, - .proc_handler = percpu_pagelist_fraction_sysctl_handler, - .extra1 = &min_percpu_pagelist_fract, - },#ifdef CONFIG_MMU { .procname = "max_map_count",
mm/memory-failure.c +0 -1
@@ -237,7 +237,6 @@ void shake_page(struct page *p, int access)lru_add_drain_all ();if (PageLRU (p))return ; -drain_all_pages ();if (PageLRU (p) ||is_free_buddy_page (p))return ; }
mm/memory_hotplug.c +1 -10
@@ -970,8 +970,6 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ ret =walk_system_ram_range (pfn, nr_pages, &onlined_pages, online_pages_range);if (ret) { -if (need_zonelists_rebuild) -zone_pcp_reset (zone);mutex_unlock (&zonelists_mutex); printk(KERN_DEBUG "online_pages [mem %#010llx-%#010llx] failed\n", (unsigned long long) pfn << PAGE_SHIFT,
@@ -989,8 +987,6 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typnode_states_set_node (zone_to_nid (zone), &arg);if (need_zonelists_rebuild)build_all_zonelists (NULL ,NULL ); -else -zone_pcp_update (zone); }mutex_unlock (&zonelists_mutex);
@@ -1530,7 +1526,6 @@ repeat:if (drain) {lru_add_drain_all ();cond_resched (); -drain_all_pages (); } pfn =scan_lru_pages (start_pfn, end_pfn);
@@ -1551,8 +1546,6 @@ repeat:/* drain all zone's lru pagevec, this is asynchronous... */ lru_add_drain_all ();yield (); -/* drain pcp pages, this is synchronous. */ -drain_all_pages ();/* check again */ offlined_pages =check_pages_isolated (start_pfn, end_pfn);if (offlined_pages <0 ) {
@@ -1574,12 +1567,10 @@ repeat:init_per_zone_wmark_min ();if (!populated_zone (zone)) { -zone_pcp_reset (zone);mutex_lock (&zonelists_mutex);build_all_zonelists (NULL ,NULL );mutex_unlock (&zonelists_mutex); - }else -zone_pcp_update (zone); + }node_states_clear_node (node, &arg);if (arg.status_change_nid >=0 )
mm/page_alloc.c +26 -555
715 lines not on this page · raw message
mm/page_isolation.c +0 -2
@@ -65,8 +65,6 @@ out: }spin_unlock_irqrestore (&zone->lock, flags); -if (!ret) -drain_all_pages ();return ret; }
mm/vmstat.c +2 -37
@@ -462,32 +462,6 @@ void refresh_cpu_vm_stats(int cpu)#endif }cond_resched (); -#ifdef CONFIG_NUMA -/* -* Deal with draining the remote pageset of this -* processor -* -* Check if there are pages remaining in this pageset -* if not then there is nothing to expire. -*/ -if (!p->expire || !p->pcp.count) -continue ; - -/* -* We never drain zones local to this processor. -*/ -if (zone_to_nid (zone) ==numa_node_id ()) { - p->expire =0 ; -continue ; - } - - p->expire--; -if (p->expire) -continue ; - -if (p->pcp.count) -drain_zone_pages (zone, &p->pcp); -#endif }for (i =0 ; i <NR_VM_ZONE_STAT_ITEMS ; i++)
@@ -1028,24 +1002,15 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,seq_printf (m,")" "\n pagesets" ); +#ifdef CONFIG_SMP for_each_online_cpu (i) {struct per_cpu_pageset *pageset; pageset =per_cpu_ptr (zone->pageset, i); -seq_printf (m, -"\n cpu: %i" -"\n count: %i" -"\n high: %i" -"\n batch: %i" , - i, - pageset->pcp.count, - pageset->pcp.high, - pageset->pcp.batch); -#ifdef CONFIG_SMP seq_printf (m,"\n vm stats threshold: %d" , pageset->stat_threshold); -#endif } +#endif seq_printf (m,"\n all_unreclaimable: %u" "\n start_pfn: %lu"
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 9/22 mm: page allocator: Allocate/free order-0 pages from a per-zone magazine
In reply to Mel GormanThis patch introduces a simple magazine of order-0 pages that sits between the buddy allocator and the caller. Simplistically there is a "struct free_area zone->noirq_magazine" in each zone protected by an IRQ-unsafe spinlock zone->magazine_lock. It replaces the per-cpu allocator that used to exist but has several properties that may be better depending on the workload.
1. IRQs do not have to be disabled to access the lists reducing IRQs disabled times.
2. As the list is protected by a spinlock, it is not necessary to send IPI to drain the list. As the lists are accessible by multiple CPUs, it is easier to tune.
3. The magazine_lock is potentially hot but it can be split to have one lock per CPU socket to reduce contention. Draining the lists in this case would acquire multiple locks be acquired.
--- include/linux/mmzone.h | 7 +++ mm/page_alloc.c | 114 +++++++++++++++++++++++++++++++++++++++++-------- mm/vmstat.c | 14 ++++-- 3 files changed, 114 insertions(+), 21 deletions(-)
include/linux/mmzone.h +7 -0
@@ -361,6 +361,13 @@ struct zone {#endif struct free_area free_area[MAX_ORDER ]; +/* +* Keep some order-0 pages on a separate free list +* protected by an irq-unsafe lock +*/ +spinlock_t magazine_lock; +struct free_area noirq_magazine; +#ifndef CONFIG_SPARSEMEM /* * Flags for a pageblock_nr_pages block. See pageblock-flags.h.
mm/page_alloc.c +97 -17
190 lines not on this page · raw message
mm/vmstat.c +10 -4
@@ -1001,14 +1001,20 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,seq_printf (m,", %lu" , zone->lowmem_reserve[i]);seq_printf (m,")" -"\n pagesets" ); +"\n noirq magazine" ); +seq_printf (m, +"\n cpu: %i" +"\n count: %lu" , + i, + zone->noirq_magazine.nr_free); +#ifdef CONFIG_SMP for_each_online_cpu (i) {struct per_cpu_pageset *pageset; - pageset =per_cpu_ptr (zone->pageset, i); -seq_printf (m,"\n vm stats threshold: %d" , - pageset->stat_threshold); + pageset =per_cpu_ptr (zone->pageset, i); +seq_printf (m,"\n pagesets\n vm stats threshold: %d" , + pageset->stat_threshold); }#endif seq_printf(m,
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine
Quoted 2 lines · Mel Gorman
1. IRQs do not have to be disabled to access the lists reducing IRQs disabled times.
The per cpu structure access also would not need to disable irq if the fast path would be using this_cpu ops.
Quoted 3 lines
2. As the list is protected by a spinlock, it is not necessary to send IPI to drain the list. As the lists are accessible by multiple CPUs, it is easier to tune.
The lists are a problem since traversing list heads creates a lot of pressure on the processor and TLB caches. Could we either move to an array of pointers to page structs (like in SLAB) or to a linked list that is constrained within physical boundaries like within a PMD? (comparable to the SLUB approach)?
Quoted 3 lines
Quoted 1 line
3. The
magazine_lockis potentially hot but it can be split to haveone lock per CPU socket to reduce contention. Draining the lists in this case would acquire multiple locks be acquired.
IMHO the use of per cpu RMV operations would be lower latency than the use of spinlocks. There is no "lock" prefix overhead with those. Page allocation is a frequent operation that I would think needs to be as fast as possible.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine
Quoted 8 lines · Christoph Lameter
Quoted 2 lines · Mel Gorman
1. IRQs do not have to be disabled to access the lists reducing IRQs disabled times.
The per cpu structure access also would not need to disable irq if the fast path would be using
this_cpuops.
How does this_cpu protect against preemption due to interrupt? this_read() itself only disables preemption and it's explicitly documented that interrupt that modifies the per-cpu data will not be reliable so the use of the per-cpu lists is right out. It would require that a race-prone check be used with cmpxchg which in turn would require arrays, not lists.
Quoted 7 lines
Quoted 3 lines
2. As the list is protected by a spinlock, it is not necessary to send IPI to drain the list. As the lists are accessible by multiple CPUs, it is easier to tune.
The lists are a problem since traversing list heads creates a lot of pressure on the processor and TLB caches. Could we either move to an array of pointers to page structs (like in SLAB)
They would have large memory requirements. The magazine data structure in this series fits in a cache line. An array of 128 struct page pointers would require 1K on 64-bit and if that thing is per possible CPU and per zone then it could get really excessive.
Quoted 4 lines
or to a linked list that is
constrained within physical boundaries like within a PMD? (comparable to the SLUB approach)?
I don't see how as the page allocator does not control the physical location of any pages freed to it and it's the struct pages it is linking together. On some systems at least with 1G pages, the struct pages will be backed by memory mapped with 1G entries so the TLB pressure should be reduced but the cache pressure from struct page modifications is certainly a problem.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
3. The
magazine_lockis potentially hot but it can be split to haveone lock per CPU socket to reduce contention. Draining the lists in this case would acquire multiple locks be acquired.
IMHO the use of per cpu RMV operations would be lower latency than the use of spinlocks. There is no "lock" prefix overhead with those. Page allocation is a frequent operation that I would think needs to be as fast as possible.
The memory requirements may be large because those per-cpu areas sized are allocated depending on num_possible_cpus()s. Correct? Regardless of their size, it would still be required to deal with cpu hot-plug to avoid memory leaks and draining them would still require global IPIs so the overall code complexity would be similar to what exists today. Ultimately all that changes is that we use an array+cmpxchg instead of a list which will shave a small amount of latency but it will still be regularly falling back to the buddy lists and contend on the zone->lock due the limited size of the per-cpu magazines and hiding the advantage of using cmpxchg in the noise.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine
Quoted 10 lines · Mel Gorman
Quoted 4 lines
The per cpu structure access also would not need to disable irq if the fast path would be using
this_cpuops.How does
this_cpuprotect against preemption due to interrupt?this_read()itself only disables preemption and it's explicitly documented that interrupt that modifies the per-cpu data will not be reliable so the use of the per-cpu lists is right out. It would require that a race-prone check be used with cmpxchg which in turn would require arrays, not lists.
this_cpu uses single atomic instructions that cannot be interrupted. The relocation occurs through a segment prefix and therefore is not subject to preemption. The interrupts and rescheduling can occur between this_cpu ops and there races would have to be dealt with. True using cmpxchg (w/o lock semantics) is not that easy. But that is the fastest solution that I know of.
Quoted 5 lines
I don't see how as the page allocator does not control the physical location of any pages freed to it and it's the struct pages it is linking together. On some systems at least with 1G pages, the struct pages will be backed by memory mapped with 1G entries so the TLB pressure should be reduced but the cache pressure from struct page modifications is certainly a problem.
I would be useful if the allocator would hand out pages from the same physical area first. This would reduce fragmentation as well and since it is likely that numerous pages are allocated for some purpose (given that that the page sizes of 4k are rather tiny compared to the data needs these day) would reduce TLB pressure.
Quoted 11 lines
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
3. The
magazine_lockis potentially hot but it can be split to haveone lock per CPU socket to reduce contention. Draining the lists in this case would acquire multiple locks be acquired.
IMHO the use of per cpu RMV operations would be lower latency than the use of spinlocks. There is no "lock" prefix overhead with those. Page allocation is a frequent operation that I would think needs to be as fast as possible.
The memory requirements may be large because those per-cpu areas sized are allocated depending on num_possible_cpus()s. Correct? Regardless of their
Yes. But we have lots of memory in machines these days. Why would that be an issue?
Quoted 7 lines
size, it would still be required to deal with cpu hot-plug to avoid memory leaks and draining them would still require global IPIs so the overall code complexity would be similar to what exists today. Ultimately all that changes is that we use an array+cmpxchg instead of a list which will shave a small amount of latency but it will still be regularly falling back to the buddy lists and contend on the
zone->lockdue the limited size of the per-cpu magazines and hiding the advantage of using cmpxchg in the noise.
The latency would be an order of magnitude less than the approach that you propose here. The magazine approach and the lockless approach both will require slowpaths that replenish the set of pages to be served next.
The problem with the page allocator is that it can serve various types of pages. If one wants to setup caches for all of those then these caches are replicated for each processor or whatever higher unit we decide to use. I think one of the first moves need to be to identify which types of pages are actually useful to serve in a fast way. Higher order pages are already out but what about the different zone types, migration types etc?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine
Quoted 13 lines · Christoph Lameter
Quoted 10 lines · Mel Gorman
Quoted 4 lines
The per cpu structure access also would not need to disable irq if the fast path would be using
this_cpuops.How does
this_cpuprotect against preemption due to interrupt?this_read()itself only disables preemption and it's explicitly documented that interrupt that modifies the per-cpu data will not be reliable so the use of the per-cpu lists is right out. It would require that a race-prone check be used with cmpxchg which in turn would require arrays, not lists.
Quoted 7 lines
this_cpuuses single atomic instructions that cannot be interrupted. The relocation occurs through a segment prefix and therefore is not subject to preemption. The interrupts and rescheduling can occur betweenthis_cpuops and there races would have to be dealt with. True using cmpxchg (w/o lock semantics) is not that easy. But that is the fastest solution that I know of.
And it requires moving to an array so there are going to be strong limits on the size of the per-cpu queue.
Quoted 6 lines
Quoted 5 lines
I don't see how as the page allocator does not control the physical location of any pages freed to it and it's the struct pages it is linking together. On some systems at least with 1G pages, the struct pages will be backed by memory mapped with 1G entries so the TLB pressure should be reduced but the cache pressure from struct page modifications is certainly a problem.
Quoted 6 lines
I would be useful if the allocator would hand out pages from the same physical area first. This would reduce fragmentation as well and since it is likely that numerous pages are allocated for some purpose (given that that the page sizes of 4k are rather tiny compared to the data needs these day) would reduce TLB pressure.
It already does this via the buddy allocator and the treatment of migratetypes.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
3. The
magazine_lockis potentially hot but it can be split to haveone lock per CPU socket to reduce contention. Draining the lists in this case would acquire multiple locks be acquired.
IMHO the use of per cpu RMV operations would be lower latency than the use of spinlocks. There is no "lock" prefix overhead with those. Page allocation is a frequent operation that I would think needs to be as fast as possible.
Quoted 6 lines
Quoted 2 lines
The memory requirements may be large because those per-cpu areas sized are allocated depending on num_possible_cpus()s. Correct? Regardless of their
Yes. But we have lots of memory in machines these days. Why would that be an issue?
Because the embedded people will have a fit if the page allocator needs an additional 1K+ of memory just to turn on.
Quoted 8 lines
Quoted 7 lines
size, it would still be required to deal with cpu hot-plug to avoid memory leaks and draining them would still require global IPIs so the overall code complexity would be similar to what exists today. Ultimately all that changes is that we use an array+cmpxchg instead of a list which will shave a small amount of latency but it will still be regularly falling back to the buddy lists and contend on the
zone->lockdue the limited size of the per-cpu magazines and hiding the advantage of using cmpxchg in the noise.
Quoted 4 lines
The latency would be an order of magnitude less than the approach that you propose here. The magazine approach and the lockless approach both will require slowpaths that replenish the set of pages to be served next.
With this approach the lock can be made more fine or coarse based on the number of CPUs, the queues can be made arbitrarily large and if necessary, per-process magazines for heavily contended workloads could be added. A fixed-size array like you propose would be only marginally better than what is implemented today as far as I can see because it still smacks into the irq-safe zone->lock and pages can be pinned in inaccessible per-cpu queues unless a global IPI is sent.
Quoted 7 lines
The problem with the page allocator is that it can serve various types of pages. If one wants to setup caches for all of those then these caches are replicated for each processor or whatever higher unit we decide to use. I think one of the first moves need to be to identify which types of pages are actually useful to serve in a fast way. Higher order pages are already out but what about the different zone types, migration types etc?
What tpyes of pages are useful to serve in a fast way is workload dependenat and besides the per-cpu allocator as it exists today already has separate queues for migration types.
I strongly suspect that your proposal would end up performing roughly the same as what exists today except that it'll be more complex because it'll have to deal with the race-prone array accesses.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine
Quoted 9 lines · Mel Gorman
Quoted 6 lines
I would be useful if the allocator would hand out pages from the same physical area first. This would reduce fragmentation as well and since it is likely that numerous pages are allocated for some purpose (given that that the page sizes of 4k are rather tiny compared to the data needs these day) would reduce TLB pressure.
It already does this via the buddy allocator and the treatment of migratetypes.
Well it only does if it breaks larger sized pages from the buddy allocator. If large page lists aggregate in the per cpu lists then we have a LIFO order.
Quoted 4 lines
Quoted 2 lines
Yes. But we have lots of memory in machines these days. Why would that be an issue?
Because the embedded people will have a fit if the page allocator needs an additional 1K+ of memory just to turn on.
Why enlarge the existing per cpu areas? The size could
be restricted if we reduce the types of pages supported and/or if we do not use double linked lists but single linked lists within say a PMD area.
Quoted 3 lines
With this approach the lock can be made more fine or coarse based on the number of CPUs, the queues can be made arbitrarily large and if necessary, per-process magazines for heavily contended workloads could be added.
Arbitrarily large queues cause references to pointers all over memory. No good.
Quoted 4 lines
A fixed-size array like you propose would be only marginally better than what is implemented today as far as I can see because it still smacks into the irq-safe
zone->lockand pages can be pinned in inaccessible per-cpu queues unless a global IPI is sent.
We do not send global IPIs but IPIs only to processors that have something cached.
The fixed size array or constrained single linked list would be better since it caches better and it is possible to avoid spin lock operations.
Quoted 8 lines
Quoted 7 lines
The problem with the page allocator is that it can serve various types of pages. If one wants to setup caches for all of those then these caches are replicated for each processor or whatever higher unit we decide to use. I think one of the first moves need to be to identify which types of pages are actually useful to serve in a fast way. Higher order pages are already out but what about the different zone types, migration types etc?
Quoted 7 lines
What tpyes of pages are useful to serve in a fast way is workload dependenat and besides the per-cpu allocator as it exists today already has separate queues for migration types.
I strongly suspect that your proposal would end up performing roughly the same as what exists today except that it'll be more complex because it'll have to deal with the race-prone array accesses.
The problems of the current scheme are the proliferation of page types, the serving of pages in a random mix from all over memory, the heavy high latency processing in the "fast" paths (these paths seem to accumulate more and more procesing in each kernel version) and the disabling of interrupts (which may be the least latency causing issue).
A solution without using locks cannot simply be a modification of the existing scheme that you envison. The amount of processing in the fastpaths must be significantly reduced and the data layout needs to be more cache friendly. Only with these changes will make the use of fast cpu local instructions sense.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 10/22 mm: page allocator: Allocate and free pages from magazine in batches
In reply to Mel GormanWhen the magazine is empty or full the zone lock is taken and a single page is operated on. This makes the zone lock hotter than it needs to be so batch allocations and frees from the zone. A larger number of pages are taken when refilling the magazine to reduce the contention on the zone->lock for IRQ-disabled callers. It's more likely that a workload will notice contention on allocations than contentions on free although of course this is workload dependant
--- mm/page_alloc.c | 172 +++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 127 insertions(+), 45 deletions(-)
mm/page_alloc.c +127 -45
@@ -601,8 +601,6 @@ static inline void __free_one_page(struct page *page,list_add (&page->lru, &zone->free_area[order].free_list[migratetype]); out: zone->free_area[order].nr_free++; -if (unlikely (!is_migrate_isolate (migratetype))) -__mod_zone_freepage_state (zone,1 << order, migratetype); }static inline int free_pages_check (struct page *page)
@@ -634,6 +632,8 @@ static void free_one_page(struct zone *zone, struct page *page,__count_vm_events (PGFREE ,1 << order);__free_one_page (page, zone, order, migratetype); +if (unlikely (!is_migrate_isolate (migratetype))) +__mod_zone_freepage_state (zone,1 << order, migratetype);spin_unlock_irqrestore (&zone->lock, flags); }
@@ -1093,6 +1093,87 @@ void mark_free_pages(struct zone *zone)#endif /* CONFIG_PM */ #define MAGAZINE_LIMIT (1024) +#define MAGAZINE_ALLOC_BATCH (384) +#define MAGAZINE_FREE_BATCH (64) + +static +struct page *__rmqueue_magazine (struct zone *zone,int migratetype) +{ +struct page *page; +struct free_area *area = &(zone->noirq_magazine); + +if (list_empty (&area->free_list[migratetype])) +return NULL ; + +/* Page is available in the magazine, allocate it */ + page = list_entry(area->free_list[migratetype].next, struct page, lru); +list_del (&page->lru); + area->nr_free--; +set_page_private (page,0 ); + +return page; +} + +static void magazine_drain (struct zone *zone,int migratetype) +{ +struct free_area *area = &(zone->noirq_magazine); +struct list_head *list; +struct page *page; +unsigned int batch_free =0 ; +unsigned int to_free =MAGAZINE_FREE_BATCH ; +unsigned int nr_freed_cma =0 ; +unsigned long flags; +LIST_HEAD (free_list); + +if (area->nr_free <MAGAZINE_LIMIT ) { +spin_unlock (&zone->magazine_lock); +return ; + } + +/* Free batch number of pages */ +while (to_free) { +/* +* Removes pages from lists in a round-robin fashion. A +* batch_free count is maintained that is incremented when an +* empty list is encountered. This is so more pages are freed +* off fuller lists instead of spinning excessively around empty +* lists +*/ +do { + batch_free++; +if (++migratetype ==MIGRATE_PCPTYPES ) + migratetype =0 ; + list = &area->free_list[migratetype];; + }while (list_empty (list)); + +/* This is the only non-empty list. Free them all. */ +if (batch_free ==MIGRATE_PCPTYPES ) + batch_free = to_free; + +do { + page = list_entry(list->prev, struct page, lru); + area->nr_free--; +set_page_private (page,0 ); +list_move (&page->lru, &free_list); +if (is_migrate_isolate_page (zone, page)) + nr_freed_cma++; + }while (--to_free && --batch_free && !list_empty (list)); + } + +/* Free the list of pages to the buddy allocator */ +spin_unlock (&zone->magazine_lock); +spin_lock_irqsave (&zone->lock, flags); +while (!list_empty (&free_list)) { + page = list_entry(free_list.prev, struct page, lru); +list_del (&page->lru); +__free_one_page (page, zone,0 ,get_freepage_migratetype (page)); + } +__mod_zone_page_state (zone,NR_FREE_PAGES , +MAGAZINE_FREE_BATCH - nr_freed_cma); +if (nr_freed_cma) +__mod_zone_page_state (zone,NR_FREE_CMA_PAGES , nr_freed_cma); +spin_unlock_irqrestore (&zone->lock, flags); +}/* * Free a 0-order page
@@ -1111,8 +1192,10 @@ void free_hot_cold_page(struct page *page, bool cold)set_freepage_migratetype (page, migratetype);/* magazine_lock is not safe against IRQs */ -if (in_interrupt () ||irqs_disabled ()) - goto free_one; +if (in_interrupt () ||irqs_disabled ()) { +free_one_page (zone, page,0 , migratetype); +return ; + }/* Put the free page on the magazine list */ spin_lock (&zone->magazine_lock);
@@ -1121,32 +1204,10 @@ void free_hot_cold_page(struct page *page, bool cold)list_add (&page->lru, &area->free_list[migratetype]); else list_add_tail(&page->lru, &area->free_list[migratetype]); - page =NULL ; - -/* If the magazine is full, remove a cold page for the buddy list */ -if (area->nr_free >MAGAZINE_LIMIT ) { -struct list_head *list = &area->free_list[migratetype]; -int starttype = migratetype; + area->nr_free++; -while (list_empty (list)) { -if (++migratetype ==MIGRATE_PCPTYPES ) - migratetype =0 ; - list = &area->free_list[migratetype];; - -WARN_ON_ONCE (starttype == migratetype); - } - - page = list_entry(list->prev, struct page, lru); -list_del (&page->lru); - }else { - area->nr_free++; - } -spin_unlock (&zone->magazine_lock); - -free_one: -/* Free a page back to the buddy lists if necessary */ -if (page) -free_one_page (zone, page,0 , migratetype); +/* Drain the magazine if necessary, releases the magazine lock */ +magazine_drain (zone, migratetype); }/*
@@ -1261,25 +1322,46 @@ staticstruct page *rmqueue_magazine (struct zone *zone,int migratetype) {struct page *page =NULL ; -struct free_area *area; -/* Check if it is worth acquiring the lock */ -if (!zone->noirq_magazine.nr_free) -return NULL ; - -spin_lock (&zone->magazine_lock); - area = &(zone->noirq_magazine); -if (list_empty (&area->free_list[migratetype])) - goto out; +/* Only acquire the lock if there is a reasonable chance of success */ +if (zone->noirq_magazine.nr_free) { +spin_lock (&zone->magazine_lock); + page =__rmqueue_magazine (zone, migratetype); +spin_unlock (&zone->magazine_lock); + } -/* Page is available in the magazine, allocate it */ - page = list_entry(area->free_list[migratetype].next, struct page, lru); -list_del (&page->lru); - area->nr_free--; -set_page_private (page,0 ); +/* Try refilling the magazine on allocaion failure */ +if (!page) { +LIST_HEAD (alloc_list); +unsigned long flags; +struct free_area *area = &(zone->noirq_magazine); +unsigned int i; +unsigned int nr_alloced =0 ; + +spin_lock_irqsave (&zone->lock, flags); +for (i =0 ; i <MAGAZINE_ALLOC_BATCH ; i++) { + page =__rmqueue (zone,0 , migratetype); +if (!page) +break ; +list_add_tail (&page->lru, &alloc_list); + nr_alloced++; + } +if (!is_migrate_cma (mt)) +__mod_zone_page_state (zone,NR_FREE_PAGES , -nr_alloced); +else +__mod_zone_page_state (zone,NR_FREE_CMA_PAGES , -nr_alloced); +spin_unlock_irqrestore (&zone->lock, flags); + +spin_lock (&zone->magazine_lock); +while (!list_empty (&alloc_list)) { + page = list_entry(alloc_list.next, struct page, lru); +list_move_tail (&page->lru, &area->free_list[migratetype]); + area->nr_free++; + } + page =__rmqueue_magazine (zone, migratetype); +spin_unlock (&zone->magazine_lock); + } -out: -spin_unlock (&zone->magazine_lock);return page; }
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 11/22 mm: page allocator: Shrink the magazine to the migratetypes in use
In reply to Mel GormanA full free_area is larger than required. Shrink it.
--- include/linux/mmzone.h | 9 +++++++-- mm/page_alloc.c | 23 +++++++++++++++++++---- 2 files changed, 26 insertions(+), 6 deletions(-)
include/linux/mmzone.h +7 -2
@@ -85,6 +85,11 @@ struct free_area {unsigned long nr_free; }; +struct free_area_magazine { +struct list_head free_list[MIGRATE_PCPTYPES ]; +unsigned long nr_free; +}; +struct pglist_data ;/*
@@ -365,8 +370,8 @@ struct zone {* Keep some order-0 pages on a separate free list * protected by an irq-unsafe lock */ -spinlock_t magazine_lock; -struct free_area noirq_magazine; +spinlock_t magazine_lock; +struct free_area_magazine noirq_magazine;#ifndef CONFIG_SPARSEMEM /*
mm/page_alloc.c +19 -4
@@ -1100,7 +1100,7 @@ staticstruct page *__rmqueue_magazine (struct zone *zone,int migratetype) {struct page *page; -struct free_area *area = &(zone->noirq_magazine); +struct free_area_magazine *area = &(zone->noirq_magazine);if (list_empty (&area->free_list[migratetype]))return NULL ;
@@ -1116,7 +1116,7 @@ struct page *__rmqueue_magazine(struct zone *zone, int migratetype)static void magazine_drain (struct zone *zone,int migratetype) { -struct free_area *area = &(zone->noirq_magazine); +struct free_area_magazine *area = &(zone->noirq_magazine);struct list_head *list;struct page *page;unsigned int batch_free =0 ;
@@ -1183,12 +1183,27 @@ void free_hot_cold_page(struct page *page, bool cold) {struct zone *zone =page_zone (page);int migratetype; -struct free_area *area; +struct free_area_magazine *area;if (!free_pages_prepare (page,0 ))return ; migratetype =get_pageblock_migratetype (page); + +/* +* We only track unmovable, reclaimable and movable on magazines. +* Free ISOLATE pages back to the allocator because they are being +* offlined but treat RESERVE as movable pages so we can get those +* areas back if necessary. Otherwise, we may have to free +* excessively into the page allocator +*/ +if (migratetype >=MIGRATE_PCPTYPES ) { +if (unlikely (is_migrate_isolate (migratetype))) { +free_one_page (zone, page,0 , migratetype); +return ; + } + migratetype =MIGRATE_MOVABLE ; + }set_freepage_migratetype (page, migratetype);/* magazine_lock is not safe against IRQs */
@@ -1334,7 +1349,7 @@ struct page *rmqueue_magazine(struct zone *zone, int migratetype)if (!page) {LIST_HEAD (alloc_list);unsigned long flags; -struct free_area *area = &(zone->noirq_magazine); +struct free_area_magazine *area = &(zone->noirq_magazine);unsigned int i;unsigned int nr_alloced =0 ;
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 12/22 mm: page allocator: Remove knowledge of hot/cold from page allocator
In reply to Mel GormanThe intention of hot/cold in the page allocator was that known cache hot pages would be placed at the head of the list and allocations for data to be immediately used would use cache hot plages. Conversely, pages that were reclaimed from the LRU would be treated as cold and allocations for data that was not going to be immediately used such as ring buffers or readahead pages would use cold pages.
With the introduction of magazines, the benefit is questionable. Regardless of the cache hotness of the physical page, the struct page is modified whether hot or cold is requested. "Cold" pages that are freed will still have hot struct page cache lines as a result of the free and placing them at the tail of the magazine list is counter-productive.
As it's of dubious merit, this patch removes the free_hot_cold_page and free_hot_cold_page_list interface. The __GFP_COLD annotations are left in place for now in case a magazine design can be devised that can take advantage of the hot/cold information sensibly.
--- arch/sparc/mm/init_64.c | 4 ++-- arch/sparc/mm/tsb.c | 2 +- arch/tile/mm/homecache.c | 2 +- include/linux/gfp.h | 6 +++--- include/trace/events/kmem.h | 11 ++++------- mm/page_alloc.c | 24 ++++++++---------------- mm/rmap.c | 2 +- mm/swap.c | 4 ++-- mm/vmscan.c | 6 +++--- 9 files changed, 25 insertions(+), 36 deletions(-)
arch/sparc/mm/init_64.c +2 -2
@@ -2562,7 +2562,7 @@ void pte_free_kernel(struct mm_struct *mm, pte_t *pte) {struct page *page =virt_to_page (pte);if (put_page_testzero (page)) -free_hot_cold_page (page, false); +free_base_page (page); }static void __pte_free (pgtable_t pte)
@@ -2570,7 +2570,7 @@ static void __pte_free(pgtable_t pte)struct page *page =virt_to_page (pte);if (put_page_testzero (page)) {pgtable_page_dtor (page); -free_hot_cold_page (page, false); +free_base_page (page); } }
arch/sparc/mm/tsb.c +1 -1
@@ -520,7 +520,7 @@ void destroy_context(struct mm_struct *mm) page = mm->context.pgtable_page;if (page &&put_page_testzero (page)) {pgtable_page_dtor (page); -free_hot_cold_page (page, false); +free_base_page (page); }spin_lock_irqsave (&ctx_alloc_lock, flags);
arch/tile/mm/homecache.c +1 -1
@@ -438,7 +438,7 @@ void __homecache_free_pages(struct page *page, unsigned int order)if (put_page_testzero (page)) {homecache_change_page_home (page, order,initial_page_home ());if (order ==0 ) { -free_hot_cold_page (page, false); +free_base_page (page); }else {init_page_count (page);__free_pages (page, order);
include/linux/gfp.h +3 -3
@@ -70,7 +70,7 @@ struct vm_area_struct;#define __GFP_HIGH ((__force gfp_t)___GFP_HIGH)/* Should access emergency pools? */ #define __GFP_IO ((__force gfp_t)___GFP_IO)/* Can start physical IO? */ #define __GFP_FS ((__force gfp_t)___GFP_FS)/* Can call down to low-level FS? */ -#define __GFP_COLD ((__force gfp_t)___GFP_COLD)/* Cache-cold page required */ +#define __GFP_COLD ((__force gfp_t)___GFP_COLD)/* Cache-cold page requested, currently ignored */ #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)/* Suppress page allocation failure warning */ #define __GFP_REPEAT ((__force gfp_t)___GFP_REPEAT)/* See above */ #define __GFP_NOFAIL ((__force gfp_t)___GFP_NOFAIL)/* See above */
@@ -364,8 +364,8 @@ void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask);extern void __free_pages (struct page *page,unsigned int order);extern void free_pages (unsigned long addr,unsigned int order); -extern void free_hot_cold_page (struct page *page,bool cold); -extern void free_hot_cold_page_list (struct list_head *list,bool cold); +extern void free_base_page (struct page *page); +extern void free_base_page_list (struct list_head *list);extern void __free_memcg_kmem_pages (struct page *page,unsigned int order);extern void free_memcg_kmem_pages (unsigned long addr,unsigned int order);
include/trace/events/kmem.h +4 -7
@@ -171,24 +171,21 @@ TRACE_EVENT(mm_page_free,TRACE_EVENT (mm_page_free_batched, - TP_PROTO(struct page *page, int cold), + TP_PROTO(struct page *page), -TP_ARGS (page, cold), +TP_ARGS (page),TP_STRUCT__entry ( __field( struct page *, page ) -__field ( int, cold ) ),TP_fast_assign ( __entry->page = page; - __entry->cold = cold; ), -TP_printk ("page=%p pfn=%lu order=0 cold=%d" , +TP_printk ("page=%p pfn=%lu order=0" , __entry->page, -page_to_pfn (__entry->page), - __entry->cold) +page_to_pfn (__entry->page)) ); TRACE_EVENT(mm_page_alloc,
mm/page_alloc.c +8 -16
@@ -1175,11 +1175,8 @@ static void magazine_drain(struct zone *zone, int migratetype)spin_unlock_irqrestore (&zone->lock, flags); } -/* -* Free a 0-order page -* cold == 1 ? free a cold page : free a hot page -*/ -void free_hot_cold_page (struct page *page,bool cold) +/* Free a 0-order page */ +void free_base_page (struct page *page) {struct zone *zone =page_zone (page);int migratetype;
@@ -1215,26 +1212,21 @@ void free_hot_cold_page(struct page *page, bool cold)/* Put the free page on the magazine list */ spin_lock (&zone->magazine_lock); area = &(zone->noirq_magazine); -if (!cold) -list_add (&page->lru, &area->free_list[migratetype]); -else -list_add_tail (&page->lru, &area->free_list[migratetype]); +list_add (&page->lru, &area->free_list[migratetype]); area->nr_free++;/* Drain the magazine if necessary, releases the magazine lock */ magazine_drain (zone, migratetype); } -/* -* Free a list of 0-order pages -*/ -void free_hot_cold_page_list (struct list_head *list,bool cold) +/* Free a list of 0-order pages */ +void free_base_page_list (struct list_head *list) {struct page *page, *next;list_for_each_entry_safe (page, next, list, lru) { -trace_mm_page_free_batched (page, cold); -free_hot_cold_page (page, cold); +trace_mm_page_free_batched (page); +free_base_page (page); } }
@@ -2564,7 +2556,7 @@ void __free_pages(struct page *page, unsigned int order) {if (put_page_testzero (page)) {if (order ==0 ) -free_hot_cold_page (page, false); +free_base_page (page);else __free_pages_ok (page, order); }
mm/rmap.c +1 -1
@@ -1166,7 +1166,7 @@ void page_remove_rmap(struct page *page)* It would be tidy to reset the PageAnon mapping here, * but that might overwrite a racing page_add_anon_rmap * which increments mapcount after us but sets mapping -* before us: so leave the reset to free_hot_cold_page, +* before us: so leave the reset to free_base_page, * and remember that it's only reliable while mapped. * Leaving it set also helps swapoff to reinstate ptes * faster for those pages still in swapcache.
mm/swap.c +2 -2
@@ -63,7 +63,7 @@ static void __page_cache_release(struct page *page)static void __put_single_page (struct page *page) {__page_cache_release (page); -free_hot_cold_page (page, false); +free_base_page (page); }static void __put_compound_page (struct page *page)
@@ -712,7 +712,7 @@ void release_pages(struct page **pages, int nr, bool cold)if (zone)spin_unlock_irqrestore (&zone->lru_lock, flags); -free_hot_cold_page_list (&pages_to_free, cold); +free_base_page_list (&pages_to_free); }EXPORT_SYMBOL (release_pages);
mm/vmscan.c +3 -3
@@ -954,7 +954,7 @@ keep:if (nr_dirty && nr_dirty == nr_congested &&global_reclaim (sc))zone_set_flag (zone,ZONE_CONGESTED ); -free_hot_cold_page_list (&free_pages, true); +free_base_page_list (&free_pages);list_splice (&ret_pages, page_list);count_vm_events (PGACTIVATE , pgactivate);
@@ -1343,7 +1343,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,spin_unlock_irq (&zone->lru_lock); -free_hot_cold_page_list (&page_list, true); +free_base_page_list (&page_list);/* * If reclaim is isolating dirty pages under writeback, it implies
@@ -1534,7 +1534,7 @@ static void shrink_active_list(unsigned long nr_to_scan,__mod_zone_page_state (zone,NR_ISOLATED_ANON + file, -nr_taken);spin_unlock_irq (&zone->lru_lock); -free_hot_cold_page_list (&l_hold, true); +free_base_page_list (&l_hold); }#ifdef CONFIG_SWAP
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 13/22 mm: page allocator: Use list_splice to refill the magazine
In reply to Mel GormanNo need to operate on one page at a time.
--- mm/page_alloc.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-)
mm/page_alloc.c +7 -8
@@ -1333,6 +1333,7 @@ struct page *rmqueue_magazine(struct zone *zone, int migratetype)/* Only acquire the lock if there is a reasonable chance of success */ if (zone->noirq_magazine.nr_free) {spin_lock (&zone->magazine_lock); +retry: page =__rmqueue_magazine (zone, migratetype);spin_unlock (&zone->magazine_lock); }
@@ -1350,7 +1351,7 @@ struct page *rmqueue_magazine(struct zone *zone, int migratetype) page =__rmqueue (zone,0 , migratetype);if (!page)break ; -list_add_tail (&page->lru, &alloc_list); +list_add (&page->lru, &alloc_list); nr_alloced++; }if (!is_migrate_cma (mt))
@@ -1358,15 +1359,13 @@ struct page *rmqueue_magazine(struct zone *zone, int migratetype)else __mod_zone_page_state (zone,NR_FREE_CMA_PAGES , -nr_alloced);spin_unlock_irqrestore (&zone->lock, flags); +if (!nr_alloced) +return NULL ;spin_lock (&zone->magazine_lock); -while (!list_empty (&alloc_list)) { - page = list_entry(alloc_list.next, struct page, lru); -list_move_tail (&page->lru, &area->free_list[migratetype]); - area->nr_free++; - } - page =__rmqueue_magazine (zone, migratetype); -spin_unlock (&zone->magazine_lock); +list_splice (&alloc_list, &area->free_list[migratetype]); + area->nr_free += nr_alloced; + goto retry; }return page;
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 14/22 mm: page allocator: Do not disable IRQs just to update stats
In reply to Mel GormanThe fast path of the allocator disables/enables interrupts to update statistics but these statistics are only consumed by userspace. When the page allocator always had to disable IRQs it was ok as we already took the penalty but now with the IRQ-unsafe magazine it is overkill to disable IRQs just to have accurate statistics. This patch does not disable IRQs for updating statistics and accepts that the counters might be slightly inaccurate.
--- mm/page_alloc.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
mm/page_alloc.c +10 -7
@@ -1381,7 +1381,6 @@ struct page *rmqueue(struct zone *preferred_zone, struct zone *zone, unsigned int order, gfp_t gfp_flags, int migratetype) { - unsigned long flags; struct page *page = NULL;if (unlikely (gfp_flags & __GFP_NOFAIL)) {
@@ -1406,11 +1405,9 @@ again:if (order ==0 && !in_interrupt () && !irqs_disabled ()) page =rmqueue_magazine (zone, migratetype); -/* IRQ disabled for buddy list access of updating statistics */ -local_irq_save (flags); -if (!page) { -spin_lock (&zone->lock); +unsigned long flags; +spin_lock_irqsave (&zone->lock, flags); page =__rmqueue (zone, order, migratetype);if (!page) {spin_unlock_irqrestore (&zone->lock, flags);
@@ -1418,12 +1415,18 @@ again: }__mod_zone_freepage_state (zone, -(1 << order),get_freepage_migratetype (page)); -spin_unlock (&zone->lock); +spin_unlock_irqrestore (&zone->lock, flags); } +/* +* NOTE: These are using the non-IRQ safe stats updating which +* means that some updates will be lost. However, these stats +* are not used internally by the VM and collisions are +* expected to be very rare. Disabling/enabling interrupts just +* to have accurate rarely-used counters is overkill. +*/ __count_zone_vm_events (PGALLOC , zone,1 << order);zone_statistics (preferred_zone, zone, gfp_flags); -local_irq_restore (flags);VM_BUG_ON (bad_range (zone, page));if (prep_new_page (page, order, gfp_flags))
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 15/22 mm: page allocator: Check if interrupts are enabled only once per allocation attempt
In reply to Mel Gormanin_interrupt() is not that expensive but it's still potentially called a very large number of times during a page allocation. Ensure it's only called once. As the check if now firmly in the allocation path, we can use __GFP_WAIT as an cheaper check for IRQs being disabled.
--- mm/page_alloc.c | 67 +++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 44 insertions(+), 23 deletions(-)
mm/page_alloc.c +44 -23
@@ -1379,7 +1379,8 @@ retry:static inline struct page *rmqueue (struct zone *preferred_zone,struct zone *zone,unsigned int order, -gfp_t gfp_flags,int migratetype) +gfp_t gfp_flags,int migratetype, +bool use_magazine) {struct page *page =NULL ;
@@ -1398,11 +1399,7 @@ struct page *rmqueue(struct zone *preferred_zone, } again: -/* -* For order-0 allocations that are not from irq context, try -* allocate from a separate magazine of free pages -*/ -if (order ==0 && !in_interrupt () && !irqs_disabled ()) +if (use_magazine) page =rmqueue_magazine (zone, migratetype);if (!page) {
@@ -1739,7 +1736,8 @@ static inline void init_zone_allows_reclaim(int nid)static struct page *get_page_from_freelist (gfp_t gfp_mask,nodemask_t *nodemask,unsigned int order,struct zonelist *zonelist,int high_zoneidx,int alloc_flags, -struct zone *preferred_zone,int migratetype) +struct zone *preferred_zone,int migratetype, +bool use_magazine) {struct zoneref *z;struct page *page =NULL ;
@@ -1845,7 +1843,7 @@ zonelist_scan: try_this_zone: page =rmqueue (preferred_zone, zone, order, - gfp_mask, migratetype); + gfp_mask, migratetype, use_magazine);if (page)break ; this_zone_full:
@@ -1971,7 +1969,7 @@ static inline struct page * __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx, nodemask_t *nodemask, struct zone *preferred_zone, - int migratetype) + int migratetype, bool use_magazine) { struct page *page;
@@ -1989,7 +1987,7 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, zonelist, high_zoneidx,ALLOC_WMARK_HIGH |ALLOC_CPUSET , - preferred_zone, migratetype); + preferred_zone, migratetype, use_magazine);if (page) goto out;
@@ -2024,7 +2022,7 @@ static struct page * __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx, nodemask_t *nodemask, int alloc_flags, struct zone *preferred_zone, - int migratetype, bool sync_migration, + int migratetype, bool sync_migration, bool use_magazine, bool *contended_compaction, bool *deferred_compaction, unsigned long *did_some_progress) {
@@ -2048,7 +2046,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, page =get_page_from_freelist (gfp_mask, nodemask, order, zonelist, high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS , - preferred_zone, migratetype); + preferred_zone, migratetype, use_magazine);if (page) { preferred_zone->compact_blockskip_flush = false; preferred_zone->compact_considered =0 ;
@@ -2124,7 +2122,7 @@ static inline struct page * __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx, nodemask_t *nodemask, int alloc_flags, struct zone *preferred_zone, - int migratetype, unsigned long *did_some_progress) + int migratetype, bool use_magazine, unsigned long *did_some_progress) { struct page *page = NULL;
@@ -2140,7 +2138,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, page =get_page_from_freelist (gfp_mask, nodemask, order, zonelist, high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS , - preferred_zone, migratetype); + preferred_zone, migratetype, + use_magazine);return page; }
@@ -2153,14 +2152,14 @@ static inline struct page * __alloc_pages_high_priority(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx, nodemask_t *nodemask, struct zone *preferred_zone, - int migratetype) + int migratetype, bool use_magazine) { struct page *page; do { page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist, high_zoneidx, ALLOC_NO_WATERMARKS, - preferred_zone, migratetype); + preferred_zone, migratetype, use_magazine); if (!page && gfp_mask & __GFP_NOFAIL) wait_iff_congested(preferred_zone, BLK_RW_ASYNC, HZ/50);
@@ -2239,7 +2238,7 @@ static inline struct page * __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx, nodemask_t *nodemask, struct zone *preferred_zone, - int migratetype) + int migratetype, bool use_magazine) { const gfp_t wait = gfp_mask & __GFP_WAIT;struct page *page =NULL ;
@@ -2297,7 +2296,7 @@ rebalance:/* This is the last chance, in general, before the goto nopage. */ page =get_page_from_freelist (gfp_mask, nodemask, order, zonelist, high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS , - preferred_zone, migratetype); + preferred_zone, migratetype, use_magazine);if (page) goto got_pg;
@@ -2312,7 +2311,7 @@ rebalance: page =__alloc_pages_high_priority (gfp_mask, order, zonelist, high_zoneidx, nodemask, - preferred_zone, migratetype); + preferred_zone, migratetype, use_magazine);if (page) { goto got_pg; }
@@ -2339,6 +2338,7 @@ rebalance: nodemask, alloc_flags, preferred_zone, migratetype, sync_migration, + use_magazine, &contended_compaction, &deferred_compaction, &did_some_progress);
@@ -2361,7 +2361,8 @@ rebalance: zonelist, high_zoneidx, nodemask, alloc_flags, preferred_zone, - migratetype, &did_some_progress); + migratetype, use_magazine, + &did_some_progress);if (page) goto got_pg;
@@ -2380,7 +2381,7 @@ rebalance: page =__alloc_pages_may_oom (gfp_mask, order, zonelist, high_zoneidx, nodemask, preferred_zone, - migratetype); + migratetype, use_magazine);if (page) goto got_pg;
@@ -2424,6 +2425,7 @@ rebalance: nodemask, alloc_flags, preferred_zone, migratetype, sync_migration, + use_magazine, &contended_compaction, &deferred_compaction, &did_some_progress);
@@ -2442,6 +2444,24 @@ got_pg: }/* +* For order-0 allocations that are not from irq context, try +* allocate from a separate magazine of free pages. +*/ +static inline bool should_alloc_use_magazine (gfp_t gfp_mask,unsigned int order) +{ +if (order) +return false; + +if (gfp_mask & __GFP_WAIT) +return true; + +if (in_interrupt () ||irqs_disabled ()) +return false; + +return true; +} + +/* * This is the 'heart' of the zoned buddy allocator. */ struct page *
@@ -2455,6 +2475,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,unsigned int cpuset_mems_cookie;int alloc_flags =ALLOC_WMARK_LOW |ALLOC_CPUSET ;struct mem_cgroup *memcg =NULL ; +bool use_magazine =should_alloc_use_magazine (gfp_mask, order); gfp_mask &= gfp_allowed_mask;
@@ -2497,7 +2518,7 @@ retry_cpuset:/* First allocation attempt */ page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, zonelist, high_zoneidx, alloc_flags, - preferred_zone, migratetype); + preferred_zone, migratetype, use_magazine);if (unlikely (!page)) {/* * Runtime PM, block IO and its error handling path
@@ -2507,7 +2528,7 @@ retry_cpuset: gfp_mask =memalloc_noio_flags (gfp_mask); page =__alloc_pages_slowpath (gfp_mask, order, zonelist, high_zoneidx, nodemask, - preferred_zone, migratetype); + preferred_zone, migratetype, use_magazine); }trace_mm_page_alloc (page, order, gfp_mask, migratetype);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 16/22 mm: page allocator: Remove coalescing improvement heuristic during page free
In reply to Mel GormanCommit 6dda9d55 ( page allocator: reduce fragmentation in buddy allocator by adding buddies that are merging to the tail of the free lists) classified pages according to their probability of being part of a high order merge. This made sense when the number of pages being freed was relatively small as part of a per-cpu list drain.
However, with the introduction of magazines, a drain of the magazines frees larger number of pages in batch and the heuristic is less likely to benefit but adds a lot of weight to the free path in the normal case. The free path can be very hot for workloads with short-lived processes, are fault intensive or work with many in-kernel short-lived buffers. As THP is the main benefit of such a heuristic, it's too marginal a gain to impact the free path so heavily, remove it.
--- mm/page_alloc.c | 22 ---------------------- 1 file changed, 22 deletions(-)
mm/page_alloc.c +0 -22
@@ -577,29 +577,7 @@ static inline void __free_one_page(struct page *page, }set_page_order (page, order); -/* -* If this is not the largest possible page, check if the buddy -* of the next-highest order is free. If it is, it's possible -* that pages are being freed that will coalesce soon. In case, -* that is happening, add the free page to the tail of the list -* so it's less likely to be used soon and more likely to be merged -* as a higher order page -*/ -if ((order <MAX_ORDER -2 ) &&pfn_valid_within (page_to_pfn (buddy))) { -struct page *higher_page, *higher_buddy; - combined_idx = buddy_idx & page_idx; - higher_page = page + (combined_idx - page_idx); - buddy_idx =__find_buddy_index (combined_idx, order +1 ); - higher_buddy = higher_page + (buddy_idx - combined_idx); -if (page_is_buddy (higher_page, higher_buddy, order +1 )) { -list_add_tail (&page->lru, - &zone->free_area[order].free_list[migratetype]); - goto out; - } - } -list_add (&page->lru, &zone->free_area[order].free_list[migratetype]); -out: zone->free_area[order].nr_free++; }
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 17/22 mm: page allocator: Move magazine access behind accessors
In reply to Mel GormanIn preparation for splitting the magazines, move them behind accessors.
--- include/linux/mmzone.h | 4 ++-- mm/page_alloc.c | 57 +++++++++++++++++++++++++++++++++----------------- mm/vmstat.c | 5 +---- 3 files changed, 41 insertions(+), 25 deletions(-)
include/linux/mmzone.h +2 -2
@@ -370,8 +370,8 @@ struct zone {* Keep some order-0 pages on a separate free list * protected by an irq-unsafe lock */ -spinlock_t magazine_lock; -struct free_area_magazine noirq_magazine; +spinlock_t _magazine_lock; +struct free_area_magazine _noirq_magazine;#ifndef CONFIG_SPARSEMEM /*
mm/page_alloc.c +38 -19
127 lines not on this page · raw message
mm/vmstat.c +1 -4
@@ -1003,15 +1003,12 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,")" "\n noirq magazine");seq_printf (m, -"\n cpu: %i" "\n count: %lu" , - i, - zone->noirq_magazine.nr_free); + zone->_noirq_magazine.nr_free);#ifdef CONFIG_SMP for_each_online_cpu (i) {struct per_cpu_pageset *pageset; - pageset =per_cpu_ptr (zone->pageset, i);seq_printf (m,"\n pagesets\n vm stats threshold: %d" , pageset->stat_threshold);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 18/22 mm: page allocator: Split magazine lock in two to reduce contention
In reply to Mel GormanThis is the simpliest example of how the lock can be split arbitrarily on a boundary. Ideally it would be based on SMT characteristics but lets just split it in two to start with and prefer a magazine based on the processor ID.
--- include/linux/mmzone.h | 12 ++++-- mm/page_alloc.c | 114 +++++++++++++++++++++++++++++++++++-------------- mm/vmstat.c | 8 ++-- 3 files changed, 95 insertions(+), 39 deletions(-)
include/linux/mmzone.h +9 -3
@@ -90,6 +90,11 @@ struct free_area_magazine {unsigned long nr_free; }; +struct free_magazine { +spinlock_t lock; +struct free_area_magazine area; +}; +struct pglist_data ;/*
@@ -305,6 +310,8 @@ enum zone_type {#ifndef __GENERATING_BOUNDS_H +#define NR_MAGAZINES 2 + struct zone {/* Fields commonly accessed by the page allocator */
@@ -368,10 +375,9 @@ struct zone {/* * Keep some order-0 pages on a separate free list -* protected by an irq-unsafe lock +* protected by an irq-unsafe lock. */ -spinlock_t _magazine_lock; -struct free_area_magazine _noirq_magazine; +struct free_magazine noirq_magazine[NR_MAGAZINES ];#ifndef CONFIG_SPARSEMEM /*
mm/page_alloc.c +81 -33
205 lines not on this page · raw message
mm/vmstat.c +5 -3
@@ -1002,9 +1002,11 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,seq_printf (m,")" "\n noirq magazine" ); -seq_printf (m, -"\n count: %lu" , - zone->_noirq_magazine.nr_free); +for (i =0 ; i <NR_MAGAZINES ; i++) { +seq_printf (m, +"\n count: %lu" , + zone->noirq_magazine[i].area.nr_free); + }#ifdef CONFIG_SMP for_each_online_cpu (i) {
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 18/22] mm: page allocator: Split magazine lock in two to reduce contention
include/linux/mmzone.h struct zone { · line 380
Quoted 10 lines · Mel Gorman
include/linux/mmzone.h +2
-3@@ -368,10 +375,9 @@ struct zone {/* * Keep some order-0 pages on a separate free list -* protected by an irq-unsafe lock +* protected by an irq-unsafe lock. */ -spinlock_t _magazine_lock; -struct free_area_magazine _noirq_magazine; +struct free_magazine noirq_magazine[NR_MAGAZINES ];
Looks like pretty cool stuff!
The old per-cpu-pages stuff was all hung off alloc_percpu(), which surely wasted lots of memory with many NUMA nodes. It's nice to see this decoupled a bit from the online cpu count.
That said, the alloc_percpu() stuff is nice in how much it hides from you when doing cpu hotplug. We'll _probably_ need this to be dynamically-sized at some point, right?
mm/page_alloc.c void mark_free_pages(struct zone *zone) · line 1082
Quoted 10 lines
mm/page_alloc.c +4
-4-static inline struct free_area_magazine *find_lock_magazine (struct zone *zone) +static inline struct free_magazine *lock_magazine (struct zone *zone) { -struct free_area_magazine *area = &zone->_noirq_magazine; -spin_lock (&zone->_magazine_lock); -return area; +int i = (raw_smp_processor_id () >>1 ) & (NR_MAGAZINES -1 ); +spin_lock (&zone->noirq_magazine[i].lock); +return &zone->noirq_magazine[i]; }
I bet this logic will be fun to play with once we have more magazines around. For instance, on my system processors 0/80 are HT twins, so they'd always be going after the same magazine. I guess that's a good thing.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 18/22] mm: page allocator: Split magazine lock in two to reduce contention
In reply to Mel Gormanmm/page_alloc.c void mark_free_pages(struct zone *zone) · line 1091
Quoted 12 lines · Mel Gorman
mm/page_alloc.c +7
-3-static inline struct free_area_magazine *find_lock_filled_magazine (struct zone *zone) +static inline struct free_magazine *find_lock_magazine (struct zone *zone) { -struct free_area_magazine *area = &zone->_noirq_magazine; -if (!area->nr_free) +int i = (raw_smp_processor_id () >>1 ) & (NR_MAGAZINES -1 ); +int start = i; + + do { + if (spin_trylock(&zone->noirq_magazine[i].lock)) + goto out;
I'm not sure doing it this way is great. It optimizes for lock contention vs the initial cost of just fetching the cache line. Doing the try lock already has to fetch the cache line, even if the lock is contended.
Page allocation should be limited more by the cache line bouncing than long contention
So you may be paying the fetch cost multiple times without actually amortizing it.
If you want to do it this way I would read the lock only. That can be much cheaper because it doesn't have to take the cache line exclusive. It may still need to transfer it though (because another CPU just took it exclusive), which may be already somewhat expensive.
So overall I'm not sure it's a good idea.
-Andi
Signature · 7 lines
[email protected] -- Speaking for myself only -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 19/22 mm: page allocator: Watch for magazine and zone lock contention
In reply to Mel GormanWhen refilling or draining magazines it is possible that the locks are contended. This patch will refill/drain a minimum number of pages and attempt to refill/drain a maximum number. Between the min and max ranges it will check contention and release the lock if it is contended.
--- mm/page_alloc.c | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-)
mm/page_alloc.c +30 -8
@@ -1071,8 +1071,10 @@ void mark_free_pages(struct zone *zone)#endif /* CONFIG_PM */ #define MAGAZINE_LIMIT (1024) -#define MAGAZINE_ALLOC_BATCH (384) -#define MAGAZINE_FREE_BATCH (64) +#define MAGAZINE_MIN_ALLOC_BATCH (16) +#define MAGAZINE_MIN_FREE_BATCH (16) +#define MAGAZINE_MAX_ALLOC_BATCH (384) +#define MAGAZINE_MAX_FREE_BATCH (64)static inline struct free_magazine *lock_magazine (struct zone *zone) {
@@ -1138,6 +1140,11 @@ static inline void unlock_magazine(struct free_magazine *mag)spin_unlock (&mag->lock); } +static inline bool magazine_contended (struct free_magazine *mag) +{ +return spin_is_contended (&mag->lock); +} +static struct page *__rmqueue_magazine (struct free_magazine *mag,int migratetype)
@@ -1163,8 +1170,8 @@ static void magazine_drain(struct zone *zone, struct free_magazine *mag,struct list_head *list;struct page *page;unsigned int batch_free =0 ; -unsigned int to_free =MAGAZINE_FREE_BATCH ; -unsigned int nr_freed_cma =0 ; +unsigned int to_free =MAGAZINE_MAX_FREE_BATCH ; +unsigned int nr_freed_cma =0 , nr_freed =0 ;unsigned long flags;struct free_area_magazine *area = &mag->area;LIST_HEAD (free_list);
@@ -1190,9 +1197,13 @@ static void magazine_drain(struct zone *zone, struct free_magazine *mag, list = &area->free_list[migratetype];; }while (list_empty (list)); -/* This is the only non-empty list. Free them all. */ +/* +* This is the only non-empty list. Free up the the min-free +* batch so that the spinlock contention is still checked +*/ if (batch_free ==MIGRATE_PCPTYPES ) - batch_free = to_free; + batch_free = min_t(unsigned int, +MAGAZINE_MIN_FREE_BATCH , to_free); do { page = list_entry(list->prev, struct page, lru);
@@ -1201,7 +1212,13 @@ static void magazine_drain(struct zone *zone, struct free_magazine *mag,list_move (&page->lru, &free_list);if (is_migrate_isolate_page (zone, page)) nr_freed_cma++; + nr_freed++; }while (--to_free && --batch_free && !list_empty (list)); + +/* Watch for parallel contention */ +if (nr_freed >MAGAZINE_MIN_FREE_BATCH && +magazine_contended (mag)) +break ; }/* Free the list of pages to the buddy allocator */
@@ -1213,7 +1230,7 @@ static void magazine_drain(struct zone *zone, struct free_magazine *mag,__free_one_page (page, zone,0 ,get_freepage_migratetype (page)); }__mod_zone_page_state (zone,NR_FREE_PAGES , -MAGAZINE_FREE_BATCH - nr_freed_cma); + nr_freed - nr_freed_cma);if (nr_freed_cma)__mod_zone_page_state (zone,NR_FREE_CMA_PAGES , nr_freed_cma);spin_unlock_irqrestore (&zone->lock, flags);
@@ -1388,12 +1405,17 @@ retry:unsigned int nr_alloced =0 ;spin_lock_irqsave (&zone->lock, flags); -for (i =0 ; i <MAGAZINE_ALLOC_BATCH ; i++) { +for (i =0 ; i <MAGAZINE_MAX_ALLOC_BATCH ; i++) { page =__rmqueue (zone,0 , migratetype);if (!page)break ;list_add (&page->lru, &alloc_list); nr_alloced++; + +/* Watch for parallel contention */ +if (nr_alloced >MAGAZINE_MIN_ALLOC_BATCH && +spin_is_contended (&zone->lock)) +break ; }if (!is_migrate_cma (mt))__mod_zone_page_state (zone,NR_FREE_PAGES , -nr_alloced);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 20/22 mm: page allocator: Hold magazine lock for a batch of pages
In reply to Mel Gormanfree_base_page_list() frees a list of pages. This patch will batch the magazine lock for the list of pages if possible.
--- mm/page_alloc.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 61 insertions(+), 14 deletions(-)
mm/page_alloc.c +61 -14
@@ -1236,15 +1236,13 @@ static void magazine_drain(struct zone *zone, struct free_magazine *mag,spin_unlock_irqrestore (&zone->lock, flags); } -/* Free a 0-order page */ -void free_base_page (struct page *page) +/* Prepare a page for freeing and return its migratetype */ +static inline int free_base_page_prep (struct page *page) { -struct zone *zone =page_zone (page); -struct free_magazine *mag;int migratetype;if (!free_pages_prepare (page,0 )) -return ; +return -1 ; migratetype =get_pageblock_migratetype (page);
@@ -1256,24 +1254,46 @@ void free_base_page(struct page *page)* excessively into the page allocator */ if (migratetype >=MIGRATE_PCPTYPES ) { -if (unlikely (is_migrate_isolate (migratetype))) { -free_one_page (zone, page,0 , migratetype); -return ; - } - migratetype =MIGRATE_MOVABLE ; +if (likely (!is_migrate_isolate (migratetype))) + migratetype =MIGRATE_MOVABLE ; } + +set_freepage_migratetype (page, migratetype); + +return migratetype; +} + +/* Put the free page on the magazine list with magazine lock held */ +static inline void __free_base_page (struct zone *zone, +struct free_area_magazine *area, +struct page *page,int migratetype) +{ +list_add (&page->lru, &area->free_list[migratetype]); + area->nr_free++; +} + +/* Free a 0-order page */ +void free_base_page (struct page *page) +{ +struct zone *zone =page_zone (page); +struct free_magazine *mag; +int migratetype; + + migratetype =free_base_page_prep (page); +if (migratetype ==-1 ) +return ;set_freepage_migratetype (page, migratetype);/* magazine_lock is not safe against IRQs */ -if (in_interrupt () ||irqs_disabled ()) { +if (migratetype >=MIGRATE_PCPTYPES ||in_interrupt () || +irqs_disabled ()) {free_one_page (zone, page,0 , migratetype);return ; }/* Put the free page on the magazine list */ mag =lock_magazine (zone); -list_add (&page->lru, &mag->area.free_list[migratetype]); - mag->area.nr_free++; +__free_base_page (zone, &mag->area, page, migratetype);/* Drain the magazine if necessary, releases the magazine lock */ magazine_drain (zone, mag, migratetype);
@@ -1283,11 +1303,38 @@ void free_base_page(struct page *page)void free_base_page_list (struct list_head *list) {struct page *page, *next; +struct zone *locked_zone =NULL ; +struct free_magazine *mag =NULL ; +bool use_magazine = (!in_interrupt () && !irqs_disabled ()); +int migratetype =MIGRATE_UNMOVABLE ; +/* Similar to free_hot_cold_page except magazine lock is batched */ list_for_each_entry_safe (page, next, list, lru) { +struct zone *zone =page_zone (page); +int migratetype; +trace_mm_page_free_batched (page); -free_base_page (page); + migratetype =free_base_page_prep (page); +if (migratetype ==-1 ) +continue ; + +if (!use_magazine || migratetype >=MIGRATE_PCPTYPES ) { +free_one_page (zone, page,0 , migratetype); +continue ; + } + +if (zone != locked_zone) { +/* Drain unlocks magazine lock */ +if (locked_zone) +magazine_drain (locked_zone, mag, migratetype); + mag =lock_magazine (zone); + locked_zone = zone; + } +__free_base_page (zone, &mag->area, page, migratetype); } + +if (locked_zone) +magazine_drain (locked_zone, mag, migratetype); }/*
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 21/22 mm: compaction: Release free page list under a batched magazine lock
In reply to Mel GormanCompaction can reuse the vast bulk of free_base_page() to batch hold the magazine lock.
--- include/linux/gfp.h | 1 + mm/compaction.c | 18 ++---------------- mm/page_alloc.c | 21 +++++++++++++++++++-- 3 files changed, 22 insertions(+), 18 deletions(-)
include/linux/gfp.h +1 -0
@@ -366,6 +366,7 @@ extern void __free_pages(struct page *page, unsigned int order);extern void free_pages (unsigned long addr,unsigned int order);extern void free_base_page (struct page *page);extern void free_base_page_list (struct list_head *list); +extern unsigned long release_free_page_list (struct list_head *list);extern void __free_memcg_kmem_pages (struct page *page,unsigned int order);extern void free_memcg_kmem_pages (unsigned long addr,unsigned int order);
mm/compaction.c +2 -16
@@ -38,20 +38,6 @@ static inline void count_compact_events(enum vm_event_item item, long delta)#define CREATE_TRACE_POINTS #include <trace/events/compaction.h> -static unsigned long release_freepages (struct list_head *freelist) -{ -struct page *page, *next; -unsigned long count =0 ; - -list_for_each_entry_safe (page, next, freelist, lru) { -list_del (&page->lru); -__free_page (page); - count++; - } - -return count; -} -static void map_pages (struct list_head *list) {struct page *page;
@@ -382,7 +368,7 @@ isolate_freepages_range(struct compact_control *cc,if (pfn < end_pfn) {/* Loop terminated early, cleanup. */ -release_freepages (&freelist); +release_free_page_list (&freelist);return 0 ; }
@@ -1002,7 +988,7 @@ static int compact_zone(struct zone *zone, struct compact_control *cc) out:/* Release free pages and check accounting */ - cc->nr_freepages -=release_freepages (&cc->freepages); + cc->nr_freepages -=release_free_page_list (&cc->freepages);VM_BUG_ON (cc->nr_freepages !=0 );return ret;
mm/page_alloc.c +19 -2
@@ -1300,20 +1300,24 @@ void free_base_page(struct page *page) }/* Free a list of 0-order pages */ -void free_base_page_list (struct list_head *list) +static unsigned long __free_base_page_list (struct list_head *list,bool release) {struct page *page, *next;struct zone *locked_zone =NULL ;struct free_magazine *mag =NULL ;bool use_magazine = (!in_interrupt () && !irqs_disabled ());int migratetype =MIGRATE_UNMOVABLE ; +unsigned long count =0 ; -/* Similar to free_hot_cold_page except magazine lock is batched */ +/* Similar to free_base_page except magazine lock is batched */ list_for_each_entry_safe (page, next, list, lru) {struct zone *zone =page_zone (page);int migratetype;trace_mm_page_free_batched (page); +if (release) +BUG_ON (!put_page_testzero (page)); + migratetype =free_base_page_prep (page);if (migratetype ==-1 )continue ;
@@ -1331,10 +1335,23 @@ void free_base_page_list(struct list_head *list) locked_zone = zone; }__free_base_page (zone, &mag->area, page, migratetype); + count++; }if (locked_zone)magazine_drain (locked_zone, mag, migratetype); + +return count; +} + +void free_base_page_list (struct list_head *list) +{ +__free_base_page_list (list, false); +} + +unsigned long release_free_page_list (struct list_head *list) +{ +return __free_base_page_list (list, true); }/*
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 22/22 mm: page allocator: Drain magazines for direct compact failures
In reply to Mel GormanTHP allocations may fail due to pages pinned in magazines so drain them in the event of a direct compact failure. Similarly drain the magazines during memory hot-remove, memory failure and page isolation as before.
--- include/linux/gfp.h | 2 ++ mm/memory-failure.c | 1 + mm/memory_hotplug.c | 2 ++ mm/page_alloc.c | 63 +++++++++++++++++++++++++++++++++++++++++++++-------- mm/page_isolation.c | 1 + 5 files changed, 60 insertions(+), 9 deletions(-)
include/linux/gfp.h +2 -0
@@ -375,6 +375,8 @@ extern void free_memcg_kmem_pages(unsigned long addr, unsigned int order);#define free_page (addr) free_pages((addr), 0)void page_alloc_init (void ); +void drain_zone_magazine (struct zone *zone); +void drain_all_magazines (void );/* * gfp_allowed_mask is set to GFP_BOOT_MASK during early boot to restrict what
mm/memory-failure.c +1 -0
@@ -237,6 +237,7 @@ void shake_page(struct page *p, int access)lru_add_drain_all ();if (PageLRU (p))return ; +drain_zone_magazine (page_zone (p));if (PageLRU (p) ||is_free_buddy_page (p))return ; }
mm/memory_hotplug.c +2 -0
@@ -1526,6 +1526,7 @@ repeat:if (drain) {lru_add_drain_all ();cond_resched (); +drain_all_magazines (); } pfn =scan_lru_pages (start_pfn, end_pfn);
@@ -1546,6 +1547,7 @@ repeat:/* drain all zone's lru pagevec, this is asynchronous... */ lru_add_drain_all ();yield (); +drain_all_magazines ();/* check again */ offlined_pages =check_pages_isolated (start_pfn, end_pfn);if (offlined_pages <0 ) {
mm/page_alloc.c +54 -9
102 lines not on this page · raw message
mm/page_isolation.c +1 -0
@@ -62,6 +62,7 @@ out: nr_pages =move_freepages_block (zone, page,MIGRATE_ISOLATE );__mod_zone_freepage_state (zone, -nr_pages, migratetype); +drain_zone_magazine (zone); }spin_unlock_irqrestore (&zone->lock, flags);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · Mel Gorman
So preliminary testing indicates the results are mixed bag. As long as locks are not contended, it performs fine but parallel fault testing hits into spinlock contention on the magazine locks. A greater problem is that because CPUs share magazines it means that the struct pages are frequently dirtied cache lines. If CPU A frees a page to a magazine and CPU B immediately allocates it then the cache line for the page and the magazine bounces and this costs. It's on the TODO list to research if the available literature has anything useful to say that does not depend on per-cpu lists and the associated problems with them.
If we don't want to bounce 'struct page' cache lines around, then we _need_ to make sure that things that don't share caches don't use the same magazine. I'm not sure there's any other way. But, that doesn't mean we have to _statically_ assign cores/thread to particular magazines.
Say we had a percpu hint which points us to the last magazine we used. We always go to it first, and fall back to round-robin if our preferred one is contended. That way, if we have a mixture tasks doing heavy and light allocations, the heavy allocators will tend to "own" a magazine, and the lighter ones would gravitate to sharing one.
It might be taking things too far, but we could even raise the number of magazines only when we actually *see* contention on the existing set.
Quoted 1 line
24 files changed, 571 insertions(+), 788 deletions(-)
oooooooooooooooooohhhhhhhhhhhhh.
The only question is how much we'll have to bloat it as we try to optimize things. :)
BTW, I really like the 'magazine' name. It's not frequently used in this kind of context and it conjures up a nice mental image whether it be of stacks of periodicals or firearm ammunition clips.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Dave Hansen
BTW, I really like the 'magazine' name. It's not frequently used in this kind of context and it conjures up a nice mental image whether it be of stacks of periodicals or firearm ammunition clips.
The term "magazine" was prominently used in the Bonwick article that was the base of the creation of the SLAB allocator.
http://static.usenix.org/event/usenix01/full_papers/bonwick/bonwick.pdf
http://static.usenix.org/publications/library/proceedings/bos94/full_papers/bonwick.a
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Dave Hansen
Quoted 9 lines · Mel Gorman
So preliminary testing indicates the results are mixed bag. As long as locks are not contended, it performs fine but parallel fault testing hits into spinlock contention on the magazine locks. A greater problem is that because CPUs share magazines it means that the struct pages are frequently dirtied cache lines. If CPU A frees a page to a magazine and CPU B immediately allocates it then the cache line for the page and the magazine bounces and this costs. It's on the TODO list to research if the available literature has anything useful to say that does not depend on per-cpu lists and the associated problems with them.
Quoted 5 lines
If we don't want to bounce 'struct page' cache lines around, then we _need_ to make sure that things that don't share caches don't use the same magazine. I'm not sure there's any other way. But, that doesn't mean we have to _statically_ assign cores/thread to particular magazines.
We could do something similar to sd_llc_id in kernel/sched/core.c to match CPUs to magazines where the data is likely to be at least in the last level cache.
Quoted 6 lines
Say we had a percpu hint which points us to the last magazine we used. We always go to it first, and fall back to round-robin if our preferred one is contended. That way, if we have a mixture tasks doing heavy and light allocations, the heavy allocators will tend to "own" a magazine, and the lighter ones would gravitate to sharing one.
We might not need the percpu hint if the sd_llc_id style hint was good enough.
Quoted 3 lines
It might be taking things too far, but we could even raise the number of magazines only when we actually *see* contention on the existing set.
I had considered a similar idea. I think it would be relatively easy to grow the number of magazines or even allocate them on a per-process basis but it was less clear how it would be shrunk again.
Quoted 7 lines
Quoted 1 line
24 files changed, 571 insertions(+), 788 deletions(-)oooooooooooooooooohhhhhhhhhhhhh.
The only question is how much we'll have to bloat it as we try to optimize things. :)
Indeed :/
Quoted 3 lines
BTW, I really like the 'magazine' name. It's not frequently used in this kind of context and it conjures up a nice mental image whether it be of stacks of periodicals or firearm ammunition clips.
I remember the term from the papers Christoph cited.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>