Reduce sequential read overhead
Subsystems (7)
cover Reduce sequential read overhead
This was formerly the series "Improve sequential read throughput" which noted some major differences in performance of tiobench since 3.0. While there are a number of factors, two that dominated were the introduction of the fair zone allocation policy and changes to CFQ.
The behaviour of fair zone allocation policy makes more sense than tiobench as a benchmark and CFQ defaults were not changed due to insufficient benchmarking.
This series is what's left. It's one functional fix to the fair zone allocation policy when used on NUMA machines and a reduction of overhead in general. tiobench was used for the comparison despite its flaws as an IO benchmark as in this case we are primarily interested in the overhead of page allocator and page reclaim activity.
On UMA, it makes little difference to overhead
3.16.0-rc3 3.16.0-rc3
vanilla lowercost-v5
User 383.61 386.77
System 403.83 401.74Elapsed 5411.50 5413.11
On a 4-socket NUMA machine it's a bit more noticable
3.16.0-rc3 3.16.0-rc3
vanilla lowercost-v5
User 746.94 802.00
System 65336.22 40852.33
Elapsed 27553.52 27368.46include/linux/mmzone.h | 217 ++++++++++++++++++++++------------------- include/trace/events/pagemap.h | 16 ++- mm/page_alloc.c | 122 ++++++++++++----------- mm/swap.c | 4 +- mm/vmscan.c | 7 +- mm/vmstat.c | 9 +- 6 files changed, 198 insertions(+), 177 deletions(-)
Signature · 7 lines
1.8.4.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/6 mm: pagemap: Avoid unnecessary overhead when tracepoints are deactivated
The LRU insertion and activate tracepoints take PFN as a parameter forcing the overhead to the caller. Move the overhead to the tracepoint fast-assign method to ensure the cost is only incurred when the tracepoint is active.
--- include/trace/events/pagemap.h | 16 +++++++--------- mm/swap.c | 4 ++-- 2 files changed, 9 insertions(+), 11 deletions(-)
include/trace/events/pagemap.h +7 -9
@@ -28,12 +28,10 @@ TRACE_EVENT(mm_lru_insertion,TP_PROTO ( struct page *page, - unsigned long pfn, - int lru, - unsigned long flags + int lru ), -TP_ARGS (page, pfn, lru, flags), +TP_ARGS (page, lru),TP_STRUCT__entry ( __field(struct page *, page )
@@ -44,9 +42,9 @@ TRACE_EVENT(mm_lru_insertion,TP_fast_assign ( __entry->page = page; - __entry->pfn = pfn; + __entry->pfn =page_to_pfn (page); __entry->lru = lru; - __entry->flags = flags; + __entry->flags =trace_pagemap_flags (page); ),/* Flag format is based on page-types.c formatting for pagemap */
@@ -64,9 +62,9 @@ TRACE_EVENT(mm_lru_insertion, TRACE_EVENT(mm_lru_activate, - TP_PROTO(struct page *page, unsigned long pfn), + TP_PROTO(struct page *page), - TP_ARGS(page, pfn), + TP_ARGS(page), TP_STRUCT__entry( __field(struct page *, page )
@@ -75,7 +73,7 @@ TRACE_EVENT(mm_lru_activate,TP_fast_assign ( __entry->page = page; - __entry->pfn = pfn; + __entry->pfn =page_to_pfn (page); ),/* Flag format is based on page-types.c formatting for pagemap */
mm/swap.c +2 -2
@@ -501,7 +501,7 @@ static void __activate_page(struct page *page, struct lruvec *lruvec,SetPageActive (page); lru +=LRU_ACTIVE ;add_page_to_lru_list (page, lruvec, lru); -trace_mm_lru_activate (page,page_to_pfn (page)); +trace_mm_lru_activate (page);__count_vm_event (PGACTIVATE );update_page_reclaim_stat (lruvec, file,1 );
@@ -996,7 +996,7 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec,SetPageLRU (page);add_page_to_lru_list (page, lruvec, lru);update_page_reclaim_stat (lruvec, file, active); -trace_mm_lru_insertion (page,page_to_pfn (page), lru,trace_pagemap_flags (page)); +trace_mm_lru_insertion (page, lru); }/*
Signature · 7 lines
1.8.4.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 1/6] mm: pagemap: Avoid unnecessary overhead when tracepoints are deactivated
Quoted 5 lines · Mel Gorman
The LRU insertion and activate tracepoints take PFN as a parameter forcing the overhead to the caller. Move the overhead to the tracepoint fast-assign method to ensure the cost is only incurred when the tracepoint is active.
Signed-off-by Mel Gorman <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/6 mm: Rearrange zone fields into read-only, page alloc, statistics and page reclaim lines
In reply to Mel GormanThe arrangement of struct zone has changed over time and now it has reached the point where there is some inappropriate sharing going on. On x86-64 for example
o The zone->node field is shared with the zone lock and zone->node is accessed frequently from the page allocator due to the fair zone allocation policy.
o span_seqlock is almost never used by shares a line with free_area
o Some zone statistics share a cache line with the LRU lock so reclaim-intensive and allocator-intensive workloads can bounce the cache line on a stat update
This patch rearranges struct zone to put read-only and read-mostly fields together and then splits the page allocator intensive fields, the zone statistics and the page reclaim intensive fields into their own cache lines. Note that the type of lowmem_reserve changes due to the watermark calculations being signed and avoiding a signed/unsigned conversion there.
On the test configuration I used the overall size of struct zone shrunk by one cache line. On smaller machines, this is not likely to be noticable. However, on a 4-node NUMA machine running tiobench the system CPU overhead is reduced by this patch.
3.16.0-rc3 3.16.0-rc3
vanillarearrange-v5r9
User 746.94 759.78
System 65336.22 58350.98
Elapsed 27553.52 27282.02--- include/linux/mmzone.h | 211 +++++++++++++++++++++++++------------------------ mm/page_alloc.c | 7 +- mm/vmstat.c | 4 +- 3 files changed, 113 insertions(+), 109 deletions(-)
include/linux/mmzone.h +108 -103
@@ -324,19 +324,12 @@ enum zone_type { #ifndef __GENERATING_BOUNDS_H struct zone { - /* Fields commonly accessed by the page allocator */ + /* Read-mostly fields */ /* zone watermarks, access with *_wmark_pages(zone) macros */ unsigned long watermark[NR_WMARK]; /* - * When free pages are below this point, additional steps are taken - * when reading the number of free pages to avoid per-cpu counter - * drift allowing watermarks to be breached - */ - unsigned long percpu_drift_mark; - - /* * We don't know if the memory that we're going to allocate will be freeable * or/and it will be released eventually, so to avoid totally wasting several * GB of ram we must reserve some of the lower zone memory (otherwise we risk
@@ -344,41 +337,26 @@ struct zone {* on the higher zones). This array is recalculated at runtime if the * sysctl_lowmem_reserve_ratio sysctl changes. */ -unsigned long lowmem_reserve[MAX_NR_ZONES ]; - -/* -* This is a per-zone reserve of pages that should not be -* considered dirtyable memory. -*/ -unsigned long dirty_balance_reserve; +long lowmem_reserve[MAX_NR_ZONES ];#ifdef CONFIG_NUMA int node; +#endif +/* -* zone reclaim becomes active if more unmapped pages exist. +* The target ratio of ACTIVE_ANON to INACTIVE_ANON pages on +* this zone's LRU. Maintained by the pageout code. */ -unsigned long min_unmapped_pages; -unsigned long min_slab_pages; -#endif +unsigned int inactive_ratio; + +struct pglist_data *zone_pgdat;struct per_cpu_pageset __percpu *pageset; +/* -* free areas of different sizes +* This is a per-zone reserve of pages that should not be +* considered dirtyable memory. */ -spinlock_t lock; -#if definedCONFIG_COMPACTION || definedCONFIG_CMA -/* Set to true when the PG_migrate_skip bits should be cleared */ -bool compact_blockskip_flush; - -/* pfn where compaction free scanner should start */ -unsigned long compact_cached_free_pfn; -/* pfn where async and sync compaction migration scanner should start */ -unsigned long compact_cached_migrate_pfn[2 ]; -#endif -#ifdef CONFIG_MEMORY_HOTPLUG -/* see spanned/present_pages for more description */ -seqlock_t span_seqlock; -#endif -struct free_area free_area[MAX_ORDER ]; +unsigned long dirty_balance_reserve;#ifndef CONFIG_SPARSEMEM /*
@@ -388,74 +366,14 @@ struct zone {unsigned long *pageblock_flags;#endif /* CONFIG_SPARSEMEM */ -#ifdef CONFIG_COMPACTION -/* -* On compaction failure, 1<<compact_defer_shift compactions -* are skipped before trying again. The number attempted since -* last failure is tracked with compact_considered. -*/ -unsigned int compact_considered; -unsigned int compact_defer_shift; -int compact_order_failed; -#endif - -ZONE_PADDING (_pad1_) - -/* Fields commonly accessed by the page reclaim scanner */ - spinlock_t lru_lock; -struct lruvec lruvec; - -/* Evictions & activations on the inactive file list */ -atomic_long_t inactive_age; - -unsigned long pages_scanned;/* since last reclaim */ -unsigned long flags;/* zone flags, see below */ - -/* Zone statistics */ -atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ]; - -/* -* The target ratio of ACTIVE_ANON to INACTIVE_ANON pages on -* this zone's LRU. Maintained by the pageout code. -*/ -unsigned int inactive_ratio; - - -ZONE_PADDING (_pad2_) -/* Rarely used or read-mostly fields */ - +#ifdef CONFIG_NUMA /* -* wait_table -- the array holding the hash table -* wait_table_hash_nr_entries -- the size of the hash table array -* wait_table_bits -- wait_table_size == (1 << wait_table_bits) -* -* The purpose of all these is to keep track of the people -* waiting for a page to become available and make them -* runnable again when possible. The trouble is that this -* consumes a lot of space, especially when so few things -* wait on pages at a given time. So instead of using -* per-page waitqueues, we use a waitqueue hash table. -* -* The bucket discipline is to sleep on the same queue when -* colliding and wake all in that wait queue when removing. -* When something wakes, it must check to be sure its page is -* truly available, a la thundering herd. The cost of a -* collision is great, but given the expected load of the -* table, they should be so rare as to be outweighed by the -* benefits from the saved space. -* -* __wait_on_page_locked() and unlock_page() in mm/filemap.c, are the -* primary users of these fields, and in mm/page_alloc.c -* free_area_init_core() performs the initialization of them. +* zone reclaim becomes active if more unmapped pages exist. */ - wait_queue_head_t * wait_table; -unsigned long wait_table_hash_nr_entries; -unsigned long wait_table_bits; +unsigned long min_unmapped_pages; +unsigned long min_slab_pages; +#endif /* CONFIG_NUMA */ -/* -* Discontig memory support fields. -*/ -struct pglist_data *zone_pgdat;/* zone_start_pfn == zone_start_paddr >> PAGE_SHIFT */ unsigned long zone_start_pfn;
@@ -500,9 +418,11 @@ struct zone {* adjust_managed_page_count() should be used instead of directly * touching zone->managed_pages and totalram_pages. */ +unsigned long managed_pages;unsigned long spanned_pages;unsigned long present_pages; -unsigned long managed_pages; + +const char *name;/* * Number of MIGRATE_RESEVE page block. To maintain for just
@@ -510,10 +430,95 @@ struct zone {*/ int nr_migrate_reserve_block; +#ifdef CONFIG_MEMORY_HOTPLUG +/* see spanned/present_pages for more description */ +seqlock_t span_seqlock; +#endif +/* -* rarely used fields: +* wait_table -- the array holding the hash table +* wait_table_hash_nr_entries -- the size of the hash table array +* wait_table_bits -- wait_table_size == (1 << wait_table_bits) +* +* The purpose of all these is to keep track of the people +* waiting for a page to become available and make them +* runnable again when possible. The trouble is that this +* consumes a lot of space, especially when so few things +* wait on pages at a given time. So instead of using +* per-page waitqueues, we use a waitqueue hash table. +* +* The bucket discipline is to sleep on the same queue when +* colliding and wake all in that wait queue when removing. +* When something wakes, it must check to be sure its page is +* truly available, a la thundering herd. The cost of a +* collision is great, but given the expected load of the +* table, they should be so rare as to be outweighed by the +* benefits from the saved space. +* +* __wait_on_page_locked() and unlock_page() in mm/filemap.c, are the +* primary users of these fields, and in mm/page_alloc.c +* free_area_init_core() performs the initialization of them. */ -const char *name; +wait_queue_head_t *wait_table; +unsigned long wait_table_hash_nr_entries; +unsigned long wait_table_bits; + +ZONE_PADDING (_pad1_) + +/* Write-intensive fields used from the page allocator */ + spinlock_t lock; + +/* free areas of different sizes */ +struct free_area free_area[MAX_ORDER ]; + +/* zone flags, see below */ +unsigned long flags; + +ZONE_PADDING (_pad2_) + +/* Write-intensive fields used by page reclaim */ + +/* Fields commonly accessed by the page reclaim scanner */ + spinlock_t lru_lock; +unsigned long pages_scanned;/* since last reclaim */ +struct lruvec lruvec; + +/* Evictions & activations on the inactive file list */ +atomic_long_t inactive_age; + +/* +* When free pages are below this point, additional steps are taken +* when reading the number of free pages to avoid per-cpu counter +* drift allowing watermarks to be breached +*/ +unsigned long percpu_drift_mark; + +#if definedCONFIG_COMPACTION || definedCONFIG_CMA +/* pfn where compaction free scanner should start */ +unsigned long compact_cached_free_pfn; +/* pfn where async and sync compaction migration scanner should start */ +unsigned long compact_cached_migrate_pfn[2 ]; +#endif + +#ifdef CONFIG_COMPACTION +/* +* On compaction failure, 1<<compact_defer_shift compactions +* are skipped before trying again. The number attempted since +* last failure is tracked with compact_considered. +*/ +unsigned int compact_considered; +unsigned int compact_defer_shift; +int compact_order_failed; +#endif + +#if definedCONFIG_COMPACTION || definedCONFIG_CMA +/* Set to true when the PG_migrate_skip bits should be cleared */ +bool compact_blockskip_flush; +#endif + +ZONE_PADDING (_pad3_) +/* Zone statistics */ + atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; }____cacheline_internodealigned_in_smp ; typedef enum {
mm/page_alloc.c +3 -4
@@ -1700,7 +1700,6 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order, {/* free_pages my go negative - that's OK */ long min = mark; -long lowmem_reserve = z->lowmem_reserve[classzone_idx];int o;long free_cma =0 ;
@@ -1715,7 +1714,7 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order, free_cma =zone_page_state (z,NR_FREE_CMA_PAGES );#endif -if (free_pages - free_cma <= min + lowmem_reserve) +if (free_pages - free_cma <= min + z->lowmem_reserve[classzone_idx])return false;for (o =0 ; o < order; o++) {/* At the next order, this order's pages become unavailable */
@@ -3246,7 +3245,7 @@ void show_free_areas(unsigned int filter) );printk ("lowmem_reserve[]:" );for (i =0 ; i <MAX_NR_ZONES ; i++) -printk (" %lu" , zone->lowmem_reserve[i]); +printk (" %ld" , zone->lowmem_reserve[i]);printk ("\n" ); }
@@ -5567,7 +5566,7 @@ static void calculate_totalreserve_pages(void)for_each_online_pgdat (pgdat) {for (i =0 ; i <MAX_NR_ZONES ; i++) {struct zone *zone = pgdat->node_zones + i; -unsigned long max =0 ; +long max =0 ;/* Find valid and maximum lowmem_reserve in the zone */ for (j = i; j <MAX_NR_ZONES ; j++) {
mm/vmstat.c +2 -2
@@ -1077,10 +1077,10 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, zone_page_state(zone, i));seq_printf (m, -"\n protection: (%lu" , +"\n protection: (%ld" , zone->lowmem_reserve[0 ]);for (i =1 ; i <ARRAY_SIZE (zone->lowmem_reserve); i++) -seq_printf (m,", %lu" , zone->lowmem_reserve[i]); +seq_printf (m,", %ld" , zone->lowmem_reserve[i]);seq_printf (m,")" "\n pagesets" );
Signature · 7 lines
1.8.4.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/6] mm: Rearrange zone fields into read-only, page alloc, statistics and page reclaim lines
Quoted 20 lines · Mel Gorman
The arrangement of struct zone has changed over time and now it has reached the point where there is some inappropriate sharing going on. On x86-64 for example
o The
zone->nodefield is shared with the zone lock andzone->nodeis accessed frequently from the page allocator due to the fair zone allocation policy.
ospan_seqlockis almost never used by shares a line withfree_area
o Some zone statistics share a cache line with the LRU lock so reclaim-intensive and allocator-intensive workloads can bounce the cache line on a stat updateThis patch rearranges struct zone to put read-only and read-mostly fields together and then splits the page allocator intensive fields, the zone statistics and the page reclaim intensive fields into their own cache lines. Note that the type of
lowmem_reservechanges due to the watermark calculations being signed and avoiding a signed/unsigned conversion there.On the test configuration I used the overall size of struct zone shrunk by one cache line. On smaller machines, this is not likely to be noticable. However, on a 4-node NUMA machine running tiobench the system CPU overhead is reduced by this patch.
Quoted 7 lines
3.16.0-rc3 3.16.0-rc3 vanillarearrange-v5r9 User 746.94 759.78 System 65336.22 58350.98 Elapsed 27553.52 27282.02Signed-off-by Mel Gorman <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/6 mm: Move zone->pages_scanned into a vmstat counter
In reply to Mel Gormanzone->pages_scanned is a write-intensive cache line during page reclaim and it's also updated during page free. Move the counter into vmstat to take advantage of the per-cpu updates and do not update it in the free paths unless necessary.
On a small UMA machine running tiobench the difference is marginal. On a 4-node machine the overhead is more noticable. Note that automatic NUMA balancing was disabled for this test as otherwise the system CPU overhead is unpredictable.
3.16.0-rc3 3.16.0-rc3 3.16.0-rc3
vanillarearrange-v5 vmstat-v5
User 746.94 759.78 774.56
System 65336.22 58350.98 32847.27
Elapsed 27553.52 27282.02 27415.04Note that the overhead reduction will vary depending on where exactly pages are allocated and freed.
--- include/linux/mmzone.h | 2 +- mm/page_alloc.c | 12 +++++++++--- mm/vmscan.c | 7 ++++--- mm/vmstat.c | 3 ++- 4 files changed, 16 insertions(+), 8 deletions(-)
include/linux/mmzone.h +1 -1
@@ -143,6 +143,7 @@ enum zone_stat_item {NR_SHMEM ,/* shmem pages (included tmpfs/GEM pages) */ NR_DIRTIED ,/* page dirtyings since bootup */ NR_WRITTEN ,/* page writings since bootup */ +NR_PAGES_SCANNED ,/* pages scanned since last reclaim */ #ifdef CONFIG_NUMANUMA_HIT ,/* allocated in intended node */ NUMA_MISS, /* allocated in non intended node */
@@ -480,7 +481,6 @@ struct zone {/* Fields commonly accessed by the page reclaim scanner */ spinlock_t lru_lock; -unsigned long pages_scanned;/* since last reclaim */ struct lruvec lruvec;/* Evictions & activations on the inactive file list */
mm/page_alloc.c +9 -3
@@ -680,9 +680,12 @@ static void free_pcppages_bulk(struct zone *zone, int count,int migratetype =0 ;int batch_free =0 ;int to_free = count; +unsigned long nr_scanned;spin_lock (&zone->lock); - zone->pages_scanned =0 ; + nr_scanned =zone_page_state (zone,NR_PAGES_SCANNED ); +if (nr_scanned) +__mod_zone_page_state (zone,NR_PAGES_SCANNED , -nr_scanned);while (to_free) {struct page *page;
@@ -731,8 +734,11 @@ static void free_one_page(struct zone *zone, unsigned int order, int migratetype) { +unsigned long nr_scanned;spin_lock (&zone->lock); - zone->pages_scanned =0 ; + nr_scanned =zone_page_state (zone,NR_PAGES_SCANNED ); +if (nr_scanned) +__mod_zone_page_state (zone,NR_PAGES_SCANNED , -nr_scanned);__free_one_page (page, pfn, zone, order, migratetype);if (unlikely (!is_migrate_isolate (migratetype)))
@@ -3240,7 +3246,7 @@ void show_free_areas(unsigned int filter)K (zone_page_state (zone,NR_BOUNCE )),K (zone_page_state (zone,NR_FREE_CMA_PAGES )),K (zone_page_state (zone,NR_WRITEBACK_TEMP )), - zone->pages_scanned, +K (zone_page_state (zone,NR_PAGES_SCANNED )), (!zone_reclaimable (zone) ?"yes" :"no" ) );printk ("lowmem_reserve[]:" );
mm/vmscan.c +4 -3
@@ -169,7 +169,8 @@ static unsigned long zone_reclaimable_pages(struct zone *zone)bool zone_reclaimable (struct zone *zone) { -return zone->pages_scanned <zone_reclaimable_pages (zone) *6 ; +return zone_page_state (zone,NR_PAGES_SCANNED ) < +zone_reclaimable_pages (zone) *6 ; }static unsigned long get_lru_size (struct lruvec *lruvec,enum lru_list lru)
@@ -1503,7 +1504,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,__mod_zone_page_state (zone,NR_ISOLATED_ANON + file, nr_taken);if (global_reclaim (sc)) { - zone->pages_scanned += nr_scanned; +__mod_zone_page_state (zone,NR_PAGES_SCANNED , nr_scanned);if (current_is_kswapd ())__count_zone_vm_events (PGSCAN_KSWAPD , zone, nr_scanned);else
@@ -1693,7 +1694,7 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_taken =isolate_lru_pages (nr_to_scan, lruvec, &l_hold, &nr_scanned, sc, isolate_mode, lru);if (global_reclaim (sc)) - zone->pages_scanned += nr_scanned; +__mod_zone_page_state (zone,NR_PAGES_SCANNED , nr_scanned); reclaim_stat->recent_scanned[file] += nr_taken;
mm/vmstat.c +2 -1
@@ -763,6 +763,7 @@ const char * const vmstat_text[] = {"nr_shmem" ,"nr_dirtied" ,"nr_written" , +"nr_pages_scanned" , #ifdef CONFIG_NUMA "numa_hit",
@@ -1067,7 +1068,7 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,min_wmark_pages (zone),low_wmark_pages (zone),high_wmark_pages (zone), - zone->pages_scanned, +zone_page_state (zone,NR_PAGES_SCANNED ), zone->spanned_pages, zone->present_pages, zone->managed_pages);
Signature · 7 lines
1.8.4.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 3/6] mm: Move zone->pages_scanned into a vmstat counter
Quoted 16 lines · Mel Gorman
zone->pages_scannedis a write-intensive cache line during page reclaim and it's also updated during page free. Move the counter into vmstat to take advantage of the per-cpu updates and do not update it in the free paths unless necessary.On a small UMA machine running tiobench the difference is marginal. On a 4-node machine the overhead is more noticable. Note that automatic NUMA balancing was disabled for this test as otherwise the system CPU overhead is unpredictable.
3.16.0-rc3 3.16.0-rc3 3.16.0-rc3 vanillarearrange-v5 vmstat-v5 User 746.94 759.78 774.56 System 65336.22 58350.98 32847.27 Elapsed 27553.52 27282.02 27415.04
Quoted 4 lines
Note that the overhead reduction will vary depending on where exactly pages are allocated and freed.
Signed-off-by Mel Gorman <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/6 mm: vmscan: Only update per-cpu thresholds for online CPU
In reply to Mel GormanWhen kswapd is awake reclaiming, the per-cpu stat thresholds are lowered to get more accurate counts to avoid breaching watermarks. This threshold update iterates over all possible CPUs which is unnecessary. Only online CPUs need to be updated. If a new CPU is onlined, refresh_zone_stat_thresholds() will set the thresholds correctly.
--- mm/vmstat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmstat.c +1 -1
@@ -200,7 +200,7 @@ void set_pgdat_percpu_threshold(pg_data_t *pgdat,continue ; threshold = (*calculate_pressure)(zone); -for_each_possible_cpu (cpu) +for_each_online_cpu (cpu)per_cpu_ptr (zone->pageset, cpu)->stat_threshold = threshold; }
Signature · 7 lines
1.8.4.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 4/6] mm: vmscan: Only update per-cpu thresholds for online CPU
Quoted 7 lines · Mel Gorman
When kswapd is awake reclaiming, the per-cpu stat thresholds are lowered to get more accurate counts to avoid breaching watermarks. This threshold update iterates over all possible CPUs which is unnecessary. Only online CPUs need to be updated. If a new CPU is onlined,
refresh_zone_stat_thresholds()will set the thresholds correctly.Signed-off-by Mel Gorman <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/6 mm: page_alloc: Abort fair zone allocation policy when remotes nodes are encountered
In reply to Mel GormanThe purpose of numa_zonelist_order=zone is to preserve lower zones for use with 32-bit devices. If locality is preferred then the numa_zonelist_order=node policy should be used. Unfortunately, the fair zone allocation policy overrides this by skipping zones on remote nodes until the lower one is found. While this makes sense from a page aging and performance perspective, it breaks the expected zonelist policy. This patch restores the expected behaviour for zone-list ordering.
--- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
@@ -1957,7 +1957,7 @@ zonelist_scan:*/ if (alloc_flags &ALLOC_FAIR ) {if (!zone_local (preferred_zone, zone)) -continue ; +break ;if (zone_page_state (zone,NR_ALLOC_BATCH ) <=0 )continue ; }
Signature · 7 lines
1.8.4.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 5/6] mm: page_alloc: Abort fair zone allocation policy when remotes nodes are encountered
Quoted 9 lines · Mel Gorman
The purpose of numa_zonelist_order=zone is to preserve lower zones for use with 32-bit devices. If locality is preferred then the numa_zonelist_order=node policy should be used. Unfortunately, the fair zone allocation policy overrides this by skipping zones on remote nodes until the lower one is found. While this makes sense from a page aging and performance perspective, it breaks the expected zonelist policy. This patch restores the expected behaviour for zone-list ordering.
Signed-off-by Mel Gorman <[email protected]>
32-bit NUMA? :-) Anyway, this change also cuts down the fair pass overhead on bigger NUMA machines, so I'm all for it.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 5/6] mm: page_alloc: Abort fair zone allocation policy when remotes nodes are encountered
Quoted 9 lines · Johannes Weiner
Quoted 8 lines · Mel Gorman
The purpose of numa_zonelist_order=zone is to preserve lower zones for use with 32-bit devices. If locality is preferred then the numa_zonelist_order=node policy should be used. Unfortunately, the fair zone allocation policy overrides this by skipping zones on remote nodes until the lower one is found. While this makes sense from a page aging and performance perspective, it breaks the expected zonelist policy. This patch restores the expected behaviour for zone-list ordering.
Quoted 3 lines
Quoted 1 line
Signed-off-by Mel Gorman <[email protected]>32-bit NUMA? :-)
I'm tempted to just say "it can go on fire" but realistically speaking they should be configured to use node ordering. I was very tempted to always force node ordering but I didn't have good data on how often lowmem allocations are required on NUMA machines.
Quoted 4 lines
Anyway, this change also cuts down the fair pass overhead on bigger NUMA machines, so I'm all for it.
Acked-by Johannes Weiner <[email protected]>
Thanks for the reviews!
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/6 mm: page_alloc: Reduce cost of the fair zone allocation policy
In reply to Mel GormanThe fair zone allocation policy round-robins allocations between zones within a node to avoid age inversion problems during reclaim. If the first allocation fails, the batch counts is reset and a second attempt made before entering the slow path.
One assumption made with this scheme is that batches expire at roughly the same time and the resets each time are justified. This assumption does not hold when zones reach their low watermark as the batches will be consumed at uneven rates. Allocation failure due to watermark depletion result in additional zonelist scans for the reset and another watermark check before hitting the slowpath.
On UMA, the benefit is negligible -- around 0.25%. On 4-socket NUMA machine it's variable due to the variability of measuring overhead with the vmstat changes. The system CPU overhead comparison looks like
3.16.0-rc3 3.16.0-rc3 3.16.0-rc3
vanilla vmstat-v5 lowercost-v5
User 746.94 774.56 802.00
System 65336.22 32847.27 40852.33 Elapsed 27553.52 27415.04 27368.46
However it is worth noting that the overall benchmark still completed faster and intuitively it makes sense to take as few passes as possible through the zonelists.
--- include/linux/mmzone.h | 6 +++ mm/page_alloc.c | 101 ++++++++++++++++++++++++++----------------------- 2 files changed, 59 insertions(+), 48 deletions(-)
include/linux/mmzone.h +6 -0
@@ -534,6 +534,7 @@ typedef enum {ZONE_WRITEBACK ,/* reclaim scanning has recently found * many pages under writeback */ +ZONE_FAIR_DEPLETED ,/* fair zone policy batch depleted */ } zone_flags_t;static inline void zone_set_flag (struct zone *zone,zone_flags_t flag)
@@ -571,6 +572,11 @@ static inline int zone_is_reclaim_locked(const struct zone *zone)return test_bit (ZONE_RECLAIM_LOCKED , &zone->flags); } +static inline int zone_is_fair_depleted (const struct zone *zone) +{ +return test_bit (ZONE_FAIR_DEPLETED , &zone->flags); +} +static inline int zone_is_oom_locked (const struct zone *zone) {return test_bit (ZONE_OOM_LOCKED , &zone->flags);
mm/page_alloc.c +53 -48
159 lines not on this page · raw message
Signature · 7 lines
1.8.4.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 6/6] mm: page_alloc: Reduce cost of the fair zone allocation policy
Quoted 19 lines · Mel Gorman
The fair zone allocation policy round-robins allocations between zones within a node to avoid age inversion problems during reclaim. If the first allocation fails, the batch counts is reset and a second attempt made before entering the slow path.
One assumption made with this scheme is that batches expire at roughly the same time and the resets each time are justified. This assumption does not hold when zones reach their low watermark as the batches will be consumed at uneven rates. Allocation failure due to watermark depletion result in additional zonelist scans for the reset and another watermark check before hitting the slowpath.
On UMA, the benefit is negligible -- around 0.25%. On 4-socket NUMA machine it's variable due to the variability of measuring overhead with the vmstat changes. The system CPU overhead comparison looks like
3.16.0-rc3 3.16.0-rc3 3.16.0-rc3
vanilla vmstat-v5 lowercost-v5
User 746.94 774.56 802.00
Quoted 8 lines
System 65336.22 32847.27 40852.33 Elapsed 27553.52 27415.04 27368.46However it is worth noting that the overall benchmark still completed faster and intuitively it makes sense to take as few passes as possible through the zonelists.
Signed-off-by Mel Gorman <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 6/6] mm: page_alloc: Reduce cost of the fair zone allocation policy
In reply to Mel Gormanmm/page_alloc.c again: · line 1606
Quoted 6 lines · Mel Gorman
mm/page_alloc.c +0
-0@@ -1604,6 +1604,9 @@ again: }__mod_zone_page_state (zone,NR_ALLOC_BATCH , -(1 << order));
This can underflow zero, right?
mm/page_alloc.c again: · line 1607
Quoted 1 line
mm/page_alloc.c +1
-0+ if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&
AFAICS, zone_page_state will correct negative values to zero only for CONFIG_SMP. Won't this check be broken on !CONFIG_SMP?
I just stumbled upon this when trying to optimize the function. I didn't check how rest of the design copes with negative NR_ALLOC_BATCH values.
mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 1920
Quoted 9 lines
mm/page_alloc.c +2
-0+ !zone_is_fair_depleted(zone)) +zone_set_flag (zone,ZONE_FAIR_DEPLETED );__count_zone_vm_events (PGALLOC , zone,1 << order);zone_statistics (preferred_zone, zone, gfp_flags);@@ -1915,6 +1918,18 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)#endif /* CONFIG_NUMA */
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 6/6] mm: page_alloc: Reduce cost of the fair zone allocation policy
Quoted 10 lines · Vlastimil Babka
mm/page_alloc.c again: · line 1606
Quoted 6 lines · Mel Gorman
mm/page_alloc.c +0
-0@@ -1604,6 +1604,9 @@ again: }__mod_zone_page_state (zone,NR_ALLOC_BATCH , -(1 << order));This can underflow zero, right?
Yes, because of per-cpu accounting drift.
Quoted 5 lines
mm/page_alloc.c again: · line 1607
Quoted 1 line
mm/page_alloc.c +1
-0+ if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&AFAICS,
zone_page_statewill correct negative values to zero only forCONFIG_SMP. Won't this check be broken on !CONFIG_SMP?
On !CONFIG_SMP how can there be per-cpu accounting drift that would make that counter negative?
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 6/6] mm: page_alloc: Reduce cost of the fair zone allocation policy
Quoted 7 lines · Mel Gorman
Quoted 6 lines · Vlastimil Babka
Quoted 5 lines · Mel Gorman
mm/page_alloc.c +0
-0@@ -1604,6 +1604,9 @@ again: }
Quoted 6 lines
Quoted 4 lines
mm/page_alloc.c again: · line 1606
Quoted 1 line
mm/page_alloc.c +0
-0__mod_zone_page_state (zone,NR_ALLOC_BATCH , -(1 << order));This can underflow zero, right?
Yes, because of per-cpu accounting drift.
I meant mainly because of order > 0.
Quoted 8 lines
Quoted 5 lines
mm/page_alloc.c again: · line 1607
Quoted 1 line
mm/page_alloc.c +1
-0+ if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&AFAICS,
zone_page_statewill correct negative values to zero only forCONFIG_SMP. Won't this check be broken on !CONFIG_SMP?On !CONFIG_SMP how can there be per-cpu accounting drift that would make that counter negative?
Well original code used "if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0)" elsewhere, that you are replacing with zone_is_fair_depleted check. I assumed it's because it can get negative due to order > 0. I might have not looked thoroughly enough but it seems to me there's nothing that would prevent it, such as skipping a zone because its remaining batch is lower than 1 << order. So I think the check should be "<= 0" to be safe.
Vlastimil
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Johannes Weiner
maintainerRe: [PATCH 6/6] mm: page_alloc: Reduce cost of the fair zone allocation policy
Quoted 26 lines · Vlastimil Babka
Quoted 7 lines · Mel Gorman
Quoted 6 lines · Vlastimil Babka
Quoted 5 lines · Mel Gorman
mm/page_alloc.c +0
-0@@ -1604,6 +1604,9 @@ again: }Quoted 6 lines
Quoted 4 lines
mm/page_alloc.c again: · line 1606
Quoted 1 line
mm/page_alloc.c +0
-0__mod_zone_page_state (zone,NR_ALLOC_BATCH , -(1 << order));This can underflow zero, right?
Yes, because of per-cpu accounting drift.
I meant mainly because of order > 0.
Quoted 8 lines
Quoted 5 lines
mm/page_alloc.c again: · line 1607
Quoted 1 line
mm/page_alloc.c +1
-0+ if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&AFAICS,
zone_page_statewill correct negative values to zero only forCONFIG_SMP. Won't this check be broken on !CONFIG_SMP?On !CONFIG_SMP how can there be per-cpu accounting drift that would make that counter negative?
Quoted 7 lines
Well original code used "if (
zone_page_state(zone, NR_ALLOC_BATCH)<= 0)" elsewhere, that you are replacing withzone_is_fair_depletedcheck. I assumed it's because it can get negative due to order > 0. I might have not looked thoroughly enough but it seems to me there's nothing that would prevent it, such as skipping a zone because its remaining batch is lower than 1 << order.
So I think the check should be "<= 0" to be safe.
Any updates on this?
The counter can definitely underflow on !CONFIG_SMP, and then the flag gets out of sync with the actual batch state. I'd still prefer just removing this flag again; it's extra complexity and error prone (case in point) while the upsides are not even measurable in real life.
---
include/linux/mmzone.h +0 -6
@@ -534,7 +534,6 @@ typedef enum {ZONE_WRITEBACK ,/* reclaim scanning has recently found * many pages under writeback */ -ZONE_FAIR_DEPLETED ,/* fair zone policy batch depleted */ } zone_flags_t;static inline void zone_set_flag (struct zone *zone,zone_flags_t flag)
@@ -572,11 +571,6 @@ static inline int zone_is_reclaim_locked(const struct zone *zone)return test_bit (ZONE_RECLAIM_LOCKED , &zone->flags); } -static inline int zone_is_fair_depleted (const struct zone *zone) -{ -return test_bit (ZONE_FAIR_DEPLETED , &zone->flags); -} -static inline int zone_is_oom_locked (const struct zone *zone) {return test_bit (ZONE_OOM_LOCKED , &zone->flags);
mm/page_alloc.c +1 -5
@@ -1612,9 +1612,6 @@ again: }__mod_zone_page_state (zone,NR_ALLOC_BATCH , -(1 << order)); -if (zone_page_state (zone,NR_ALLOC_BATCH ) ==0 && - !zone_is_fair_depleted (zone)) -zone_set_flag (zone,ZONE_FAIR_DEPLETED );__count_zone_vm_events (PGALLOC , zone,1 << order);zone_statistics (preferred_zone, zone, gfp_flags);
@@ -1934,7 +1931,6 @@ static void reset_alloc_batches(struct zone *preferred_zone)mod_zone_page_state (zone,NR_ALLOC_BATCH ,high_wmark_pages (zone) -low_wmark_pages (zone) -atomic_long_read (&zone->vm_stat[NR_ALLOC_BATCH ])); -zone_clear_flag (zone,ZONE_FAIR_DEPLETED ); }while (zone++ != preferred_zone); }
@@ -1985,7 +1981,7 @@ zonelist_scan:if (alloc_flags &ALLOC_FAIR ) {if (!zone_local (preferred_zone, zone))break ; -if (zone_is_fair_depleted (zone)) { +if (zone_page_state (zone,NR_ALLOC_BATCH ) <=0 ) { nr_fair_skipped++;continue ; }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/1 mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP
Commit 4ffeaf35 (mm: page_alloc: reduce cost of the fair zone allocation policy) broke the fair zone allocation policy on UP with these hunks.
mm/page_alloc.c +6 -1
@@ -1612,6 +1612,9 @@ again: }__mod_zone_page_state (zone,NR_ALLOC_BATCH , -(1 << order)); +if (zone_page_state (zone,NR_ALLOC_BATCH ) ==0 && + !zone_is_fair_depleted (zone)) +zone_set_flag (zone,ZONE_FAIR_DEPLETED );__count_zone_vm_events (PGALLOC , zone,1 << order);zone_statistics (preferred_zone, zone, gfp_flags);
@@ -1966,8 +1985,10 @@ zonelist_scan:if (alloc_flags &ALLOC_FAIR ) {if (!zone_local (preferred_zone, zone))break ; -if (zone_page_state (zone,NR_ALLOC_BATCH ) <=0 ) +if (zone_is_fair_depleted (zone)) { + nr_fair_skipped++;continue ; + } }
The problem is that a <= check was replaced with a ==. On SMP it doesn't matter because negative values are returned as zero due to per-CPU drift which is not possible in the UP case. Vlastimil Babka correctly pointed out that this can be negative due to high-order allocations. This patch fixes the problem.
--- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
@@ -1612,7 +1612,7 @@ again: }__mod_zone_page_state (zone,NR_ALLOC_BATCH , -(1 << order)); -if (zone_page_state (zone,NR_ALLOC_BATCH ) ==0 && +if (zone_page_state (zone,NR_ALLOC_BATCH ) <=0 && !zone_is_fair_depleted (zone))zone_set_flag (zone,ZONE_FAIR_DEPLETED );
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP
Hi Mel,
mm/page_alloc.c again: · line 1615
Quoted 3 lines
__mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));mm/page_alloc.c +1
-1- if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 && + if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0 &&
zone_page_state is declared to return unsigned long value [1], so it should never be below 0.
So interesting question: what zone_page_state will return for negative atomic_long_read(&zone->vm_stat[item]) ?
130 static inline unsigned long zone_page_state(struct zone *zone,
131 enum zone_stat_item item)
132 {
133 long x = atomic_long_read(&zone->vm_stat[item]);
134 #ifdef CONFIG_SMP
135 if (x < 0)
136 x = 0;
137 #endif
138 return x;
139 }
[1] https://git.kernel.org/cgit/linux/kernel/git/mhocko/mm.git/tree/include/linux/vmstat.h#n130
Signature · 8 lines
Leon Romanovsky | Independent Linux Consultant
www.leon.nu | [email protected]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>PATCH 1/1 mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP v2
Commit 4ffeaf35 (mm: page_alloc: reduce cost of the fair zone allocation policy) arguably broke the fair zone allocation policy on UP with these hunks.
a/mm/page_alloc.cb/mm/page_alloc.c
@@ -1612,6 +1612,9 @@ again:
}
__mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));+ if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&
+ !zone_is_fair_depleted(zone))
+ zone_set_flag(zone, ZONE_FAIR_DEPLETED);
__count_zone_vm_events(PGALLOC, zone, 1 << order);
zone_statistics(preferred_zone, zone, gfp_flags);
@@ -1966,8 +1985,10 @@ zonelist_scan:
if (alloc_flags & ALLOC_FAIR) {
if (!zone_local(preferred_zone, zone))
break;
- if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0)
+ if (zone_is_fair_depleted(zone)) {
+ nr_fair_skipped++;
continue;
+ }
}A <= check was replaced with a ==. On SMP it doesn't matter because negative values are returned as zero due to per-CPU drift which is not possible in the UP case. Vlastimil Babka correctly pointed out that this can wrap negative due to high-order allocations.
However, Leon Romanovsky pointed out that a <= check on zone_page_state was never correct as zone_page_state returns unsigned long so the root cause of the breakage was the <= check in the first place.
zone_page_state is an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allow NR_ALLOC_BATCH to go negative -- when the counter is reset the negative value takes recent activity into account. This patch makes zone_page_state behave the same on SMP and UP as saving one branch on UP is not likely to make a measurable performance difference.
--- include/linux/vmstat.h | 2 -- 1 file changed, 2 deletions(-)
include/linux/vmstat.h +0 -2
@@ -131,10 +131,8 @@ static inline unsigned long zone_page_state(struct zone *zone, enum zone_stat_item item) {long x =atomic_long_read (&zone->vm_stat[item]); -#ifdef CONFIG_SMP if (x <0 ) x =0 ; -#endif return x; }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP v2
Hi Mel,
Quoted 56 lines · Mel Gorman
Commit
4ffeaf35(mm:page_alloc: reduce cost of the fair zone allocation policy) arguably broke the fair zone allocation policy on UP with these hunks.
a/mm/page_alloc.cb/mm/page_alloc.c(quoted diff) +6
-1@@ -1612,6 +1612,9 @@ again: } __mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order)); + if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 && + !zone_is_fair_depleted(zone)) + zone_set_flag(zone, ZONE_FAIR_DEPLETED); __count_zone_vm_events(PGALLOC, zone, 1 << order); zone_statistics(preferred_zone, zone, gfp_flags);@@ -1966,8 +1985,10 @@ zonelist_scan: if (alloc_flags & ALLOC_FAIR) { if (!zone_local(preferred_zone, zone)) break; - if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0) + if (zone_is_fair_depleted(zone)) { + nr_fair_skipped++; continue; + } }A <= check was replaced with a ==. On SMP it doesn't matter because negative values are returned as zero due to per-CPU drift which is not possible in the UP case. Vlastimil Babka correctly pointed out that this can wrap negative due to high-order allocations.
However, Leon Romanovsky pointed out that a <= check on
zone_page_statewas never correct aszone_page_statereturns unsigned long so the root cause of the breakage was the <= check in the first place.
zone_page_stateis an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allowNR_ALLOC_BATCHto go negative -- when the counter is reset the negative value takes recent activity into account. This patch makeszone_page_statebehave the same on SMP and UP as saving one branch on UP is not likely to make a measurable performance difference.Reported-by Vlastimil Babka <[email protected]>Reported-by Leon Romanovsky <[email protected]>Signed-off-by Mel Gorman <[email protected]>--- include/linux/vmstat.h | 2 -- 1 file changed, 2 deletions(-)include/linux/vmstat.h +0
-0
include/linux/vmstat.h static inline unsigned long zone_page_state(struct · line 138
Quoted 10 lines
include/linux/vmstat.h +0
-2@@ -131,10 +131,8 @@ static inline unsigned long zone_page_state(structzone *zone, enum zone_stat_item item) { long x = atomic_long_read(&zone->vm_stat[item]); -#ifdef CONFIG_SMP if (x <0 ) x =0 ; -#endif return x; }
Since you are changing vmstat.h, what do you think about change in all similiar places?
include/linux/vmstat.h +0 -0
@@ -120,10 +120,8 @@ static inline void zone_page_state_add(long x, struct
zone *zone,
static inline unsigned long global_page_state(enum zone_stat_item item)
{
long x = atomic_long_read(&vm_stat[item]);
-#ifdef CONFIG_SMP
if (x < 0)
x = 0;
-#endif
return x;}
@@ -131,10 +129,8 @@ static inline unsigned long zone_page_state(struct zone *zone,
enum zone_stat_item item)
{
long x = atomic_long_read(&zone->vm_stat[item]);
-#ifdef CONFIG_SMP
if (x < 0)
x = 0;
-#endif
return x;}
@@ -153,10 +149,9 @@ static inline unsigned long
zone_page_state_snapshot(struct zone *zone,
int cpu;
for_each_online_cpu(cpu)
x += per_cpu_ptr(zone->pageset, cpu)->vm_stat_diff[item];
-
+#endif
if (x < 0)
x = 0;
-#endif
return x;}
Signature · 2 lines
Leon Romanovsky | Independent Linux Consultant
www.leon.nu | [email protected]Andrew Morton
maintainerRe: [PATCH] mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP v2
In reply to Mel GormanQuoted 12 lines · On Mon, 8 Sep 2014 12:57:18 +0100 Mel Gorman <[email protected]> wrote:
zone_page_stateis an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allowNR_ALLOC_BATCHto go negative -- when the counter is reset the negative value takes recent activity into account. This patch makeszone_page_statebehave the same on SMP and UP as saving one branch on UP is not likely to make a measurable performance difference....
include/linux/vmstat.h static inline unsigned long zone_page_state(struct zone *zone, · line 137
Quoted 9 lines
include/linux/vmstat.h +0
-2@@ -131,10 +131,8 @@ static inline unsigned long zone_page_state(struct zone *zone, enum zone_stat_item item) {long x =atomic_long_read (&zone->vm_stat[item]); -#ifdef CONFIG_SMP if (x <0 ) x =0 ; -#endif return x; }
We now have three fixes for the same thing. I'm presently holding on to hannes's mm-page_alloc-fix-zone-allocation-fairness-on-up.patch.
Regularizing zone_page_state() in this fashion seems a good idea and is presumably safe because callers have been tested with SMP. So unless shouted at I think I'll queue this one for 3.18?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP v2
Quoted 11 lines · Andrew Morton
Quoted 9 lines · On Mon, 8 Sep 2014 12:57:18 +0100 Mel Gorman <[email protected]> wrote:
zone_page_stateis an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allowNR_ALLOC_BATCHto go negative -- when the counter is reset the negative value takes recent activity into account. This patch makeszone_page_statebehave the same on SMP and UP as saving one branch on UP is not likely to make a measurable performance difference....
Quoted 14 lines
include/linux/vmstat.h static inline unsigned long zone_page_state(struct zone *zone, · line 137
Quoted 12 lines
include/linux/vmstat.h +0
-2@@ -131,10 +131,8 @@ static inline unsigned long zone_page_state(struct zone *zone, enum zone_stat_item item) {long x =atomic_long_read (&zone->vm_stat[item]); -#ifdef CONFIG_SMP if (x <0 ) x =0 ; -#endif return x; }We now have three fixes for the same thing.
This might be holding a record for most patches for what should have been a trivial issue :P
Quoted 3 lines
I'm presently holding on
to hannes's mm-page_alloc-fix-zone-allocation-fairness-on-up.patch.
This is my preferred fix because it clearly points to where the source of the original problem is. Furthermore, the second hunk really should be reading the unsigned counter value. It's an inconsequential corner-case but it's still more correct although it's a pity that it's also a layering violation. However, adding a new API to return the raw value on UP and SMP is likely to be interpreted as unwelcome indirection.
Quoted 3 lines
Regularizing
zone_page_state()in this fashion seems a good idea and is presumably safe because callers have been tested with SMP. So unless shouted at I think I'll queue this one for 3.18?
Both are ok but if we really want to regularise the API then all readers should be brought in line and declared an API cleanup. That looks like the following;
---8<---
From: Mel Gorman <[email protected]>
Subject: [PATCH] mm: vmstat: regularize UP and SMP behavior
zone_page_state and friends are an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allow NR_ALLOC_BATCH to go negative -- when the counter is reset the negative value takes recent activity into account. NR_ALLOC_BATCH callers that matter access the raw counter but the API hazard is a lesson.
This patch makes zone_page_state, global_page_state and zone_page_state_snapshot return the same values on SMP and UP as saving the branches on UP is unlikely to make a measurable performance difference.
--- include/linux/vmstat.h | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
include/linux/vmstat.h +2 -6
@@ -120,10 +120,8 @@ static inline void zone_page_state_add(long x, struct zone *zone,static inline unsigned long global_page_state (enum zone_stat_item item) {long x =atomic_long_read (&vm_stat[item]); -#ifdef CONFIG_SMP if (x <0 ) x =0 ; -#endif return x; }
@@ -131,10 +129,8 @@ static inline unsigned long zone_page_state(struct zone *zone, enum zone_stat_item item) {long x =atomic_long_read (&zone->vm_stat[item]); -#ifdef CONFIG_SMP if (x <0 ) x =0 ; -#endif return x; }
@@ -153,10 +149,10 @@ static inline unsigned long zone_page_state_snapshot(struct zone *zone,int cpu;for_each_online_cpu (cpu) x +=per_cpu_ptr (zone->pageset, cpu)->vm_stat_diff[item]; - +#endif if (x <0 ) x =0 ; -#endif +return x; }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Johannes Weiner
maintainerRe: [PATCH] mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP v2
Quoted 59 lines · Mel Gorman
Quoted 11 lines · Andrew Morton
Quoted 9 lines · On Mon, 8 Sep 2014 12:57:18 +0100 Mel Gorman <[email protected]> wrote:
zone_page_stateis an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allowNR_ALLOC_BATCHto go negative -- when the counter is reset the negative value takes recent activity into account. This patch makeszone_page_statebehave the same on SMP and UP as saving one branch on UP is not likely to make a measurable performance difference....
Quoted 14 lines
include/linux/vmstat.h static inline unsigned long zone_page_state(struct zone *zone, · line 137
Quoted 12 lines
include/linux/vmstat.h +0
-2@@ -131,10 +131,8 @@ static inline unsigned long zone_page_state(struct zone *zone, enum zone_stat_item item) {long x =atomic_long_read (&zone->vm_stat[item]); -#ifdef CONFIG_SMP if (x <0 ) x =0 ; -#endif return x; }We now have three fixes for the same thing.
This might be holding a record for most patches for what should have been a trivial issue :P
Quoted 3 lines
I'm presently holding on
to hannes's mm-page_alloc-fix-zone-allocation-fairness-on-up.patch.This is my preferred fix because it clearly points to where the source of the original problem is. Furthermore, the second hunk really should be reading the unsigned counter value. It's an inconsequential corner-case but it's still more correct although it's a pity that it's also a layering violation. However, adding a new API to return the raw value on UP and SMP is likely to be interpreted as unwelcome indirection.
Quoted 3 lines
Regularizing
zone_page_state()in this fashion seems a good idea and is presumably safe because callers have been tested with SMP. So unless shouted at I think I'll queue this one for 3.18?Both are ok but if we really want to regularise the API then all readers should be brought in line and declared an API cleanup. That looks like the following;
---8<---
From: Mel Gorman <[email protected]>
Subject: [PATCH] mm: vmstat: regularize UP and SMP behavior
zone_page_stateand friends are an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allowNR_ALLOC_BATCHto go negative -- when the counter is reset the negative value takes recent activity into account.NR_ALLOC_BATCHcallers that matter access the raw counter but the API hazard is a lesson.
Quoted 3 lines
This patch makes
zone_page_state,global_page_stateandzone_page_state_snapshotreturn the same values on SMP and UP as saving the branches on UP is unlikely to make a measurable performance difference.
The API still returns an unsigned long and so can not really support counters that can go logically negative - as opposed to negative reads that are a side-effect of concurrency and can be interpreted as zero.
The problem is that the fairness batches abuse the internals of what is publicly an unsigned counter to implement an unlocked allocator that needs to be able to go negative. We could make that counter API truly signed to support that, but I think this patch makes the code actually more confusing and inconsistent.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>