Skip to contentSkip to conversation

Reduce sequential read overhead

PATCH New Mel Gorman 5 participants 25 messages Jul 9, 2014 – Sep 10, 2014

Subsystems (7)
lore
Share & tools
lore

Mel Gorman

author
#0 lore raw

cover Reduce sequential read overhead

This was formerly the series "Improve sequential read throughput" which noted some major differences in performance of tiobench since 3.0. While there are a number of factors, two that dominated were the introduction of the fair zone allocation policy and changes to CFQ.

The behaviour of fair zone allocation policy makes more sense than tiobench as a benchmark and CFQ defaults were not changed due to insufficient benchmarking.

This series is what's left. It's one functional fix to the fair zone allocation policy when used on NUMA machines and a reduction of overhead in general. tiobench was used for the comparison despite its flaws as an IO benchmark as in this case we are primarily interested in the overhead of page allocator and page reclaim activity.

On UMA, it makes little difference to overhead

          3.16.0-rc3   3.16.0-rc3
             vanilla lowercost-v5
User          383.61      386.77
System        403.83      401.74

Elapsed 5411.50 5413.11

On a 4-socket NUMA machine it's a bit more noticable

          3.16.0-rc3   3.16.0-rc3
             vanilla lowercost-v5
User          746.94      802.00
System      65336.22    40852.33
Elapsed     27553.52    27368.46
 include/linux/mmzone.h         | 217 ++++++++++++++++++++++-------------------
 include/trace/events/pagemap.h |  16 ++-
 mm/page_alloc.c                | 122 ++++++++++++-----------
 mm/swap.c                      |   4 +-
 mm/vmscan.c                    |   7 +-
 mm/vmstat.c                    |   9 +-
 6 files changed, 198 insertions(+), 177 deletions(-)
Signature · 7 lines
1.8.4.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#1 lore raw

PATCH 1/6 mm: pagemap: Avoid unnecessary overhead when tracepoints are deactivated

The LRU insertion and activate tracepoints take PFN as a parameter forcing the overhead to the caller. Move the overhead to the tracepoint fast-assign method to ensure the cost is only incurred when the tracepoint is active.

---
 include/trace/events/pagemap.h | 16 +++++++---------
 mm/swap.c                      |  4 ++--
 2 files changed, 9 insertions(+), 11 deletions(-)
include/trace/events/pagemap.h +7 -9
index 1c9fabd..ce0803b 100644
@@ -28,12 +28,10 @@ TRACE_EVENT(mm_lru_insertion,
 
 	TP_PROTO(
 		struct page *page,
-		unsigned long pfn,
-		int lru,
-		unsigned long flags
+		int lru
 	),
 
-	TP_ARGS(page, pfn, lru, flags),
+	TP_ARGS(page, lru),
 
 	TP_STRUCT__entry(
 		__field(struct page *,	page	)
@@ -44,9 +42,9 @@ TRACE_EVENT(mm_lru_insertion,
 
 	TP_fast_assign(
 		__entry->page	= page;
-		__entry->pfn	= pfn;
+		__entry->pfn	= page_to_pfn(page);
 		__entry->lru	= lru;
-		__entry->flags	= flags;
+		__entry->flags	= trace_pagemap_flags(page);
 	),
 
 	/* Flag format is based on page-types.c formatting for pagemap */
@@ -64,9 +62,9 @@ TRACE_EVENT(mm_lru_insertion,
 
 TRACE_EVENT(mm_lru_activate,
 
-	TP_PROTO(struct page *page, unsigned long pfn),
+	TP_PROTO(struct page *page),
 
-	TP_ARGS(page, pfn),
+	TP_ARGS(page),
 
 	TP_STRUCT__entry(
 		__field(struct page *,	page	)
@@ -75,7 +73,7 @@ TRACE_EVENT(mm_lru_activate,
 
 	TP_fast_assign(
 		__entry->page	= page;
-		__entry->pfn	= pfn;
+		__entry->pfn	= page_to_pfn(page);
 	),
 
 	/* Flag format is based on page-types.c formatting for pagemap */
mm/swap.c +2 -2
index 9e8e347..d10be45 100644
@@ -501,7 +501,7 @@ static void __activate_page(struct page *page, struct lruvec *lruvec,
 		SetPageActive(page);
 		lru += LRU_ACTIVE;
 		add_page_to_lru_list(page, lruvec, lru);
-		trace_mm_lru_activate(page, page_to_pfn(page));
+		trace_mm_lru_activate(page);
 
 		__count_vm_event(PGACTIVATE);
 		update_page_reclaim_stat(lruvec, file, 1);
@@ -996,7 +996,7 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec,
 	SetPageLRU(page);
 	add_page_to_lru_list(page, lruvec, lru);
 	update_page_reclaim_stat(lruvec, file, active);
-	trace_mm_lru_insertion(page, page_to_pfn(page), lru, trace_pagemap_flags(page));
+	trace_mm_lru_insertion(page, lru);
 }
 
 /*
Signature · 7 lines
1.8.4.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 1/6
#2 lore raw

Re: [PATCH 1/6] mm: pagemap: Avoid unnecessary overhead when tracepoints are deactivated

Quoted 5 lines · Mel Gorman

The LRU insertion and activate tracepoints take PFN as a parameter forcing the overhead to the caller. Move the overhead to the tracepoint fast-assign method to ensure the cost is only incurred when the tracepoint is active.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#3 lore raw

PATCH 2/6 mm: Rearrange zone fields into read-only, page alloc, statistics and page reclaim lines

In reply to Mel Gorman

The arrangement of struct zone has changed over time and now it has reached the point where there is some inappropriate sharing going on. On x86-64 for example

o The zone->node field is shared with the zone lock and zone->node is accessed frequently from the page allocator due to the fair zone allocation policy.
o span_seqlock is almost never used by shares a line with free_area
o Some zone statistics share a cache line with the LRU lock so reclaim-intensive and allocator-intensive workloads can bounce the cache line on a stat update

This patch rearranges struct zone to put read-only and read-mostly fields together and then splits the page allocator intensive fields, the zone statistics and the page reclaim intensive fields into their own cache lines. Note that the type of lowmem_reserve changes due to the watermark calculations being signed and avoiding a signed/unsigned conversion there.

On the test configuration I used the overall size of struct zone shrunk by one cache line. On smaller machines, this is not likely to be noticable. However, on a 4-node NUMA machine running tiobench the system CPU overhead is reduced by this patch.

          3.16.0-rc3  3.16.0-rc3
             vanillarearrange-v5r9
User          746.94      759.78
System      65336.22    58350.98
Elapsed     27553.52    27282.02
---
 include/linux/mmzone.h | 211 +++++++++++++++++++++++++------------------------
 mm/page_alloc.c        |   7 +-
 mm/vmstat.c            |   4 +-
 3 files changed, 113 insertions(+), 109 deletions(-)
include/linux/mmzone.h +108 -103
index 6cbd1b6..fbadc45 100644
@@ -324,19 +324,12 @@ enum zone_type {
 #ifndef __GENERATING_BOUNDS_H
 
 struct zone {
-	/* Fields commonly accessed by the page allocator */
+	/* Read-mostly fields */
 
 	/* zone watermarks, access with *_wmark_pages(zone) macros */
 	unsigned long watermark[NR_WMARK];
 
 	/*
-	 * When free pages are below this point, additional steps are taken
-	 * when reading the number of free pages to avoid per-cpu counter
-	 * drift allowing watermarks to be breached
-	 */
-	unsigned long percpu_drift_mark;
-
-	/*
 	 * We don't know if the memory that we're going to allocate will be freeable
 	 * or/and it will be released eventually, so to avoid totally wasting several
 	 * GB of ram we must reserve some of the lower zone memory (otherwise we risk
@@ -344,41 +337,26 @@ struct zone {
 	 * on the higher zones). This array is recalculated at runtime if the
 	 * sysctl_lowmem_reserve_ratio sysctl changes.
 	 */
-	unsigned long		lowmem_reserve[MAX_NR_ZONES];
-
-	/*
-	 * This is a per-zone reserve of pages that should not be
-	 * considered dirtyable memory.
-	 */
-	unsigned long		dirty_balance_reserve;
+	long lowmem_reserve[MAX_NR_ZONES];
 
 #ifdef CONFIG_NUMA
 	int node;
+#endif
+
 	/*
-	 * zone reclaim becomes active if more unmapped pages exist.
+	 * The target ratio of ACTIVE_ANON to INACTIVE_ANON pages on
+	 * this zone's LRU.  Maintained by the pageout code.
 	 */
-	unsigned long		min_unmapped_pages;
-	unsigned long		min_slab_pages;
-#endif
+	unsigned int inactive_ratio;
+
+	struct pglist_data	*zone_pgdat;
 	struct per_cpu_pageset __percpu *pageset;
+
 	/*
-	 * free areas of different sizes
+	 * This is a per-zone reserve of pages that should not be
+	 * considered dirtyable memory.
 	 */
-	spinlock_t		lock;
-#if defined CONFIG_COMPACTION || defined CONFIG_CMA
-	/* Set to true when the PG_migrate_skip bits should be cleared */
-	bool			compact_blockskip_flush;
-
-	/* pfn where compaction free scanner should start */
-	unsigned long		compact_cached_free_pfn;
-	/* pfn where async and sync compaction migration scanner should start */
-	unsigned long		compact_cached_migrate_pfn[2];
-#endif
-#ifdef CONFIG_MEMORY_HOTPLUG
-	/* see spanned/present_pages for more description */
-	seqlock_t		span_seqlock;
-#endif
-	struct free_area	free_area[MAX_ORDER];
+	unsigned long		dirty_balance_reserve;
 
 #ifndef CONFIG_SPARSEMEM
 	/*
@@ -388,74 +366,14 @@ struct zone {
 	unsigned long		*pageblock_flags;
 #endif /* CONFIG_SPARSEMEM */
 
-#ifdef CONFIG_COMPACTION
-	/*
-	 * On compaction failure, 1<<compact_defer_shift compactions
-	 * are skipped before trying again. The number attempted since
-	 * last failure is tracked with compact_considered.
-	 */
-	unsigned int		compact_considered;
-	unsigned int		compact_defer_shift;
-	int			compact_order_failed;
-#endif
-
-	ZONE_PADDING(_pad1_)
-
-	/* Fields commonly accessed by the page reclaim scanner */
-	spinlock_t		lru_lock;
-	struct lruvec		lruvec;
-
-	/* Evictions & activations on the inactive file list */
-	atomic_long_t		inactive_age;
-
-	unsigned long		pages_scanned;	   /* since last reclaim */
-	unsigned long		flags;		   /* zone flags, see below */
-
-	/* Zone statistics */
-	atomic_long_t		vm_stat[NR_VM_ZONE_STAT_ITEMS];
-
-	/*
-	 * The target ratio of ACTIVE_ANON to INACTIVE_ANON pages on
-	 * this zone's LRU.  Maintained by the pageout code.
-	 */
-	unsigned int inactive_ratio;
-
-
-	ZONE_PADDING(_pad2_)
-	/* Rarely used or read-mostly fields */
-
+#ifdef CONFIG_NUMA
 	/*
-	 * wait_table		-- the array holding the hash table
-	 * wait_table_hash_nr_entries	-- the size of the hash table array
-	 * wait_table_bits	-- wait_table_size == (1 << wait_table_bits)
-	 *
-	 * The purpose of all these is to keep track of the people
-	 * waiting for a page to become available and make them
-	 * runnable again when possible. The trouble is that this
-	 * consumes a lot of space, especially when so few things
-	 * wait on pages at a given time. So instead of using
-	 * per-page waitqueues, we use a waitqueue hash table.
-	 *
-	 * The bucket discipline is to sleep on the same queue when
-	 * colliding and wake all in that wait queue when removing.
-	 * When something wakes, it must check to be sure its page is
-	 * truly available, a la thundering herd. The cost of a
-	 * collision is great, but given the expected load of the
-	 * table, they should be so rare as to be outweighed by the
-	 * benefits from the saved space.
-	 *
-	 * __wait_on_page_locked() and unlock_page() in mm/filemap.c, are the
-	 * primary users of these fields, and in mm/page_alloc.c
-	 * free_area_init_core() performs the initialization of them.
+	 * zone reclaim becomes active if more unmapped pages exist.
 	 */
-	wait_queue_head_t	* wait_table;
-	unsigned long		wait_table_hash_nr_entries;
-	unsigned long		wait_table_bits;
+	unsigned long		min_unmapped_pages;
+	unsigned long		min_slab_pages;
+#endif /* CONFIG_NUMA */
 
-	/*
-	 * Discontig memory support fields.
-	 */
-	struct pglist_data	*zone_pgdat;
 	/* zone_start_pfn == zone_start_paddr >> PAGE_SHIFT */
 	unsigned long		zone_start_pfn;
 
@@ -500,9 +418,11 @@ struct zone {
 	 * adjust_managed_page_count() should be used instead of directly
 	 * touching zone->managed_pages and totalram_pages.
 	 */
+	unsigned long		managed_pages;
 	unsigned long		spanned_pages;
 	unsigned long		present_pages;
-	unsigned long		managed_pages;
+
+	const char		*name;
 
 	/*
 	 * Number of MIGRATE_RESEVE page block. To maintain for just
@@ -510,10 +430,95 @@ struct zone {
 	 */
 	int			nr_migrate_reserve_block;
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+	/* see spanned/present_pages for more description */
+	seqlock_t		span_seqlock;
+#endif
+
 	/*
-	 * rarely used fields:
+	 * wait_table		-- the array holding the hash table
+	 * wait_table_hash_nr_entries	-- the size of the hash table array
+	 * wait_table_bits	-- wait_table_size == (1 << wait_table_bits)
+	 *
+	 * The purpose of all these is to keep track of the people
+	 * waiting for a page to become available and make them
+	 * runnable again when possible. The trouble is that this
+	 * consumes a lot of space, especially when so few things
+	 * wait on pages at a given time. So instead of using
+	 * per-page waitqueues, we use a waitqueue hash table.
+	 *
+	 * The bucket discipline is to sleep on the same queue when
+	 * colliding and wake all in that wait queue when removing.
+	 * When something wakes, it must check to be sure its page is
+	 * truly available, a la thundering herd. The cost of a
+	 * collision is great, but given the expected load of the
+	 * table, they should be so rare as to be outweighed by the
+	 * benefits from the saved space.
+	 *
+	 * __wait_on_page_locked() and unlock_page() in mm/filemap.c, are the
+	 * primary users of these fields, and in mm/page_alloc.c
+	 * free_area_init_core() performs the initialization of them.
 	 */
-	const char		*name;
+	wait_queue_head_t	*wait_table;
+	unsigned long		wait_table_hash_nr_entries;
+	unsigned long		wait_table_bits;
+
+	ZONE_PADDING(_pad1_)
+
+	/* Write-intensive fields used from the page allocator */
+	spinlock_t		lock;
+
+	/* free areas of different sizes */
+	struct free_area	free_area[MAX_ORDER];
+
+	/* zone flags, see below */
+	unsigned long		flags;
+
+	ZONE_PADDING(_pad2_)
+
+	/* Write-intensive fields used by page reclaim */
+
+	/* Fields commonly accessed by the page reclaim scanner */
+	spinlock_t		lru_lock;
+	unsigned long		pages_scanned;	   /* since last reclaim */
+	struct lruvec		lruvec;
+
+	/* Evictions & activations on the inactive file list */
+	atomic_long_t		inactive_age;
+
+	/*
+	 * When free pages are below this point, additional steps are taken
+	 * when reading the number of free pages to avoid per-cpu counter
+	 * drift allowing watermarks to be breached
+	 */
+	unsigned long percpu_drift_mark;
+
+#if defined CONFIG_COMPACTION || defined CONFIG_CMA
+	/* pfn where compaction free scanner should start */
+	unsigned long		compact_cached_free_pfn;
+	/* pfn where async and sync compaction migration scanner should start */
+	unsigned long		compact_cached_migrate_pfn[2];
+#endif
+
+#ifdef CONFIG_COMPACTION
+	/*
+	 * On compaction failure, 1<<compact_defer_shift compactions
+	 * are skipped before trying again. The number attempted since
+	 * last failure is tracked with compact_considered.
+	 */
+	unsigned int		compact_considered;
+	unsigned int		compact_defer_shift;
+	int			compact_order_failed;
+#endif
+
+#if defined CONFIG_COMPACTION || defined CONFIG_CMA
+	/* Set to true when the PG_migrate_skip bits should be cleared */
+	bool			compact_blockskip_flush;
+#endif
+
+	ZONE_PADDING(_pad3_)
+	/* Zone statistics */
+	atomic_long_t		vm_stat[NR_VM_ZONE_STAT_ITEMS];
 } ____cacheline_internodealigned_in_smp;
 
 typedef enum {
mm/page_alloc.c +3 -4
index 20d17f8..c607acd 100644
@@ -1700,7 +1700,6 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
 {
 	/* free_pages my go negative - that's OK */
 	long min = mark;
-	long lowmem_reserve = z->lowmem_reserve[classzone_idx];
 	int o;
 	long free_cma = 0;
 
@@ -1715,7 +1714,7 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
 		free_cma = zone_page_state(z, NR_FREE_CMA_PAGES);
 #endif
 
-	if (free_pages - free_cma <= min + lowmem_reserve)
+	if (free_pages - free_cma <= min + z->lowmem_reserve[classzone_idx])
 		return false;
 	for (o = 0; o < order; o++) {
 		/* At the next order, this order's pages become unavailable */
@@ -3246,7 +3245,7 @@ void show_free_areas(unsigned int filter)
 			);
 		printk("lowmem_reserve[]:");
 		for (i = 0; i < MAX_NR_ZONES; i++)
-			printk(" %lu", zone->lowmem_reserve[i]);
+			printk(" %ld", zone->lowmem_reserve[i]);
 		printk("\n");
 	}
 
@@ -5567,7 +5566,7 @@ static void calculate_totalreserve_pages(void)
 	for_each_online_pgdat(pgdat) {
 		for (i = 0; i < MAX_NR_ZONES; i++) {
 			struct zone *zone = pgdat->node_zones + i;
-			unsigned long max = 0;
+			long max = 0;
 
 			/* Find valid and maximum lowmem_reserve in the zone */
 			for (j = i; j < MAX_NR_ZONES; j++) {
mm/vmstat.c +2 -2
index b37bd49..8267f77 100644
@@ -1077,10 +1077,10 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,
 				zone_page_state(zone, i));
 
 	seq_printf(m,
-		   "\n        protection: (%lu",
+		   "\n        protection: (%ld",
 		   zone->lowmem_reserve[0]);
 	for (i = 1; i < ARRAY_SIZE(zone->lowmem_reserve); i++)
-		seq_printf(m, ", %lu", zone->lowmem_reserve[i]);
+		seq_printf(m, ", %ld", zone->lowmem_reserve[i]);
 	seq_printf(m,
 		   ")"
 		   "\n  pagesets");
Signature · 7 lines
1.8.4.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 2/6
#4 lore raw

Re: [PATCH 2/6] mm: Rearrange zone fields into read-only, page alloc, statistics and page reclaim lines

Quoted 20 lines · Mel Gorman

The arrangement of struct zone has changed over time and now it has reached the point where there is some inappropriate sharing going on. On x86-64 for example

o The zone->node field is shared with the zone lock and zone->node is accessed frequently from the page allocator due to the fair zone allocation policy.
o span_seqlock is almost never used by shares a line with free_area
o Some zone statistics share a cache line with the LRU lock so reclaim-intensive and allocator-intensive workloads can bounce the cache line on a stat update

This patch rearranges struct zone to put read-only and read-mostly fields together and then splits the page allocator intensive fields, the zone statistics and the page reclaim intensive fields into their own cache lines. Note that the type of lowmem_reserve changes due to the watermark calculations being signed and avoiding a signed/unsigned conversion there.

On the test configuration I used the overall size of struct zone shrunk by one cache line. On smaller machines, this is not likely to be noticable. However, on a 4-node NUMA machine running tiobench the system CPU overhead is reduced by this patch.

Quoted 7 lines
          3.16.0-rc3  3.16.0-rc3
             vanillarearrange-v5r9
User          746.94      759.78
System      65336.22    58350.98
Elapsed     27553.52    27282.02

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#5 lore raw

PATCH 3/6 mm: Move zone->pages_scanned into a vmstat counter

In reply to Mel Gorman

zone->pages_scanned is a write-intensive cache line during page reclaim and it's also updated during page free. Move the counter into vmstat to take advantage of the per-cpu updates and do not update it in the free paths unless necessary.

On a small UMA machine running tiobench the difference is marginal. On a 4-node machine the overhead is more noticable. Note that automatic NUMA balancing was disabled for this test as otherwise the system CPU overhead is unpredictable.

          3.16.0-rc3  3.16.0-rc3  3.16.0-rc3
             vanillarearrange-v5   vmstat-v5
User          746.94      759.78      774.56
System      65336.22    58350.98    32847.27
Elapsed     27553.52    27282.02    27415.04

Note that the overhead reduction will vary depending on where exactly pages are allocated and freed.

---
 include/linux/mmzone.h |  2 +-
 mm/page_alloc.c        | 12 +++++++++---
 mm/vmscan.c            |  7 ++++---
 mm/vmstat.c            |  3 ++-
 4 files changed, 16 insertions(+), 8 deletions(-)
include/linux/mmzone.h +1 -1
index fbadc45..c0ee2ec 100644
@@ -143,6 +143,7 @@ enum zone_stat_item {
 	NR_SHMEM,		/* shmem pages (included tmpfs/GEM pages) */
 	NR_DIRTIED,		/* page dirtyings since bootup */
 	NR_WRITTEN,		/* page writings since bootup */
+	NR_PAGES_SCANNED,	/* pages scanned since last reclaim */
 #ifdef CONFIG_NUMA
 	NUMA_HIT,		/* allocated in intended node */
 	NUMA_MISS,		/* allocated in non intended node */
@@ -480,7 +481,6 @@ struct zone {
 
 	/* Fields commonly accessed by the page reclaim scanner */
 	spinlock_t		lru_lock;
-	unsigned long		pages_scanned;	   /* since last reclaim */
 	struct lruvec		lruvec;
 
 	/* Evictions & activations on the inactive file list */
mm/page_alloc.c +9 -3
index c607acd..aa46f00 100644
@@ -680,9 +680,12 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	int migratetype = 0;
 	int batch_free = 0;
 	int to_free = count;
+	unsigned long nr_scanned;
 
 	spin_lock(&zone->lock);
-	zone->pages_scanned = 0;
+	nr_scanned = zone_page_state(zone, NR_PAGES_SCANNED);
+	if (nr_scanned)
+		__mod_zone_page_state(zone, NR_PAGES_SCANNED, -nr_scanned);
 
 	while (to_free) {
 		struct page *page;
@@ -731,8 +734,11 @@ static void free_one_page(struct zone *zone,
 				unsigned int order,
 				int migratetype)
 {
+	unsigned long nr_scanned;
 	spin_lock(&zone->lock);
-	zone->pages_scanned = 0;
+	nr_scanned = zone_page_state(zone, NR_PAGES_SCANNED);
+	if (nr_scanned)
+		__mod_zone_page_state(zone, NR_PAGES_SCANNED, -nr_scanned);
 
 	__free_one_page(page, pfn, zone, order, migratetype);
 	if (unlikely(!is_migrate_isolate(migratetype)))
@@ -3240,7 +3246,7 @@ void show_free_areas(unsigned int filter)
 			K(zone_page_state(zone, NR_BOUNCE)),
 			K(zone_page_state(zone, NR_FREE_CMA_PAGES)),
 			K(zone_page_state(zone, NR_WRITEBACK_TEMP)),
-			zone->pages_scanned,
+			K(zone_page_state(zone, NR_PAGES_SCANNED)),
 			(!zone_reclaimable(zone) ? "yes" : "no")
 			);
 		printk("lowmem_reserve[]:");
mm/vmscan.c +4 -3
index 0f16ffe..761c628 100644
@@ -169,7 +169,8 @@ static unsigned long zone_reclaimable_pages(struct zone *zone)
 
 bool zone_reclaimable(struct zone *zone)
 {
-	return zone->pages_scanned < zone_reclaimable_pages(zone) * 6;
+	return zone_page_state(zone, NR_PAGES_SCANNED) <
+		zone_reclaimable_pages(zone) * 6;
 }
 
 static unsigned long get_lru_size(struct lruvec *lruvec, enum lru_list lru)
@@ -1503,7 +1504,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
 	__mod_zone_page_state(zone, NR_ISOLATED_ANON + file, nr_taken);
 
 	if (global_reclaim(sc)) {
-		zone->pages_scanned += nr_scanned;
+		__mod_zone_page_state(zone, NR_PAGES_SCANNED, nr_scanned);
 		if (current_is_kswapd())
 			__count_zone_vm_events(PGSCAN_KSWAPD, zone, nr_scanned);
 		else
@@ -1693,7 +1694,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 	nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &l_hold,
 				     &nr_scanned, sc, isolate_mode, lru);
 	if (global_reclaim(sc))
-		zone->pages_scanned += nr_scanned;
+		__mod_zone_page_state(zone, NR_PAGES_SCANNED, nr_scanned);
 
 	reclaim_stat->recent_scanned[file] += nr_taken;
 
mm/vmstat.c +2 -1
index 8267f77..e574e883 100644
@@ -763,6 +763,7 @@ const char * const vmstat_text[] = {
 	"nr_shmem",
 	"nr_dirtied",
 	"nr_written",
+	"nr_pages_scanned",
 
 #ifdef CONFIG_NUMA
 	"numa_hit",
@@ -1067,7 +1068,7 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,
 		   min_wmark_pages(zone),
 		   low_wmark_pages(zone),
 		   high_wmark_pages(zone),
-		   zone->pages_scanned,
+		   zone_page_state(zone, NR_PAGES_SCANNED),
 		   zone->spanned_pages,
 		   zone->present_pages,
 		   zone->managed_pages);
Signature · 7 lines
1.8.4.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 3/6
#6 lore raw

Re: [PATCH 3/6] mm: Move zone->pages_scanned into a vmstat counter

Quoted 16 lines · Mel Gorman

zone->pages_scanned is a write-intensive cache line during page reclaim and it's also updated during page free. Move the counter into vmstat to take advantage of the per-cpu updates and do not update it in the free paths unless necessary.

On a small UMA machine running tiobench the difference is marginal. On a 4-node machine the overhead is more noticable. Note that automatic NUMA balancing was disabled for this test as otherwise the system CPU overhead is unpredictable.

          3.16.0-rc3  3.16.0-rc3  3.16.0-rc3
             vanillarearrange-v5   vmstat-v5
User          746.94      759.78      774.56
System      65336.22    58350.98    32847.27
Elapsed     27553.52    27282.02    27415.04
Quoted 4 lines

Note that the overhead reduction will vary depending on where exactly pages are allocated and freed.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#7 lore raw

PATCH 4/6 mm: vmscan: Only update per-cpu thresholds for online CPU

In reply to Mel Gorman

When kswapd is awake reclaiming, the per-cpu stat thresholds are lowered to get more accurate counts to avoid breaching watermarks. This threshold update iterates over all possible CPUs which is unnecessary. Only online CPUs need to be updated. If a new CPU is onlined, refresh_zone_stat_thresholds() will set the thresholds correctly.

---
 mm/vmstat.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmstat.c +1 -1
index e574e883..e9ab104 100644
@@ -200,7 +200,7 @@ void set_pgdat_percpu_threshold(pg_data_t *pgdat,
 			continue;
 
 		threshold = (*calculate_pressure)(zone);
-		for_each_possible_cpu(cpu)
+		for_each_online_cpu(cpu)
 			per_cpu_ptr(zone->pageset, cpu)->stat_threshold
 							= threshold;
 	}
Signature · 7 lines
1.8.4.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 4/6
#8 lore raw

Re: [PATCH 4/6] mm: vmscan: Only update per-cpu thresholds for online CPU

Quoted 7 lines · Mel Gorman

When kswapd is awake reclaiming, the per-cpu stat thresholds are lowered to get more accurate counts to avoid breaching watermarks. This threshold update iterates over all possible CPUs which is unnecessary. Only online CPUs need to be updated. If a new CPU is onlined, refresh_zone_stat_thresholds() will set the thresholds correctly.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#9 lore raw

PATCH 5/6 mm: page_alloc: Abort fair zone allocation policy when remotes nodes are encountered

In reply to Mel Gorman

The purpose of numa_zonelist_order=zone is to preserve lower zones for use with 32-bit devices. If locality is preferred then the numa_zonelist_order=node policy should be used. Unfortunately, the fair zone allocation policy overrides this by skipping zones on remote nodes until the lower one is found. While this makes sense from a page aging and performance perspective, it breaks the expected zonelist policy. This patch restores the expected behaviour for zone-list ordering.

---
 mm/page_alloc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
index aa46f00..0bf384a 100644
@@ -1957,7 +1957,7 @@ zonelist_scan:
 		 */
 		if (alloc_flags & ALLOC_FAIR) {
 			if (!zone_local(preferred_zone, zone))
-				continue;
+				break;
 			if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0)
 				continue;
 		}
Signature · 7 lines
1.8.4.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 5/6
#10 lore raw

Re: [PATCH 5/6] mm: page_alloc: Abort fair zone allocation policy when remotes nodes are encountered

Quoted 9 lines · Mel Gorman

The purpose of numa_zonelist_order=zone is to preserve lower zones for use with 32-bit devices. If locality is preferred then the numa_zonelist_order=node policy should be used. Unfortunately, the fair zone allocation policy overrides this by skipping zones on remote nodes until the lower one is found. While this makes sense from a page aging and performance perspective, it breaks the expected zonelist policy. This patch restores the expected behaviour for zone-list ordering.

32-bit NUMA? :-) Anyway, this change also cuts down the fair pass overhead on bigger NUMA machines, so I'm all for it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#11 lore raw

Re: [PATCH 5/6] mm: page_alloc: Abort fair zone allocation policy when remotes nodes are encountered

Quoted 9 lines · Johannes Weiner
Quoted 8 lines · Mel Gorman

The purpose of numa_zonelist_order=zone is to preserve lower zones for use with 32-bit devices. If locality is preferred then the numa_zonelist_order=node policy should be used. Unfortunately, the fair zone allocation policy overrides this by skipping zones on remote nodes until the lower one is found. While this makes sense from a page aging and performance perspective, it breaks the expected zonelist policy. This patch restores the expected behaviour for zone-list ordering.

Quoted 3 lines
Quoted 1 line

32-bit NUMA? :-)

I'm tempted to just say "it can go on fire" but realistically speaking they should be configured to use node ordering. I was very tempted to always force node ordering but I didn't have good data on how often lowmem allocations are required on NUMA machines.

Quoted 4 lines

Anyway, this change also cuts down the fair pass overhead on bigger NUMA machines, so I'm all for it.

Thanks for the reviews!

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#12 lore raw

PATCH 6/6 mm: page_alloc: Reduce cost of the fair zone allocation policy

In reply to Mel Gorman

The fair zone allocation policy round-robins allocations between zones within a node to avoid age inversion problems during reclaim. If the first allocation fails, the batch counts is reset and a second attempt made before entering the slow path.

One assumption made with this scheme is that batches expire at roughly the same time and the resets each time are justified. This assumption does not hold when zones reach their low watermark as the batches will be consumed at uneven rates. Allocation failure due to watermark depletion result in additional zonelist scans for the reset and another watermark check before hitting the slowpath.

On UMA, the benefit is negligible -- around 0.25%. On 4-socket NUMA machine it's variable due to the variability of measuring overhead with the vmstat changes. The system CPU overhead comparison looks like

3.16.0-rc3 3.16.0-rc3 3.16.0-rc3
vanilla vmstat-v5 lowercost-v5
User 746.94 774.56 802.00

System      65336.22    32847.27    40852.33
Elapsed     27553.52    27415.04    27368.46

However it is worth noting that the overall benchmark still completed faster and intuitively it makes sense to take as few passes as possible through the zonelists.

---
 include/linux/mmzone.h |   6 +++
 mm/page_alloc.c        | 101 ++++++++++++++++++++++++++-----------------------
 2 files changed, 59 insertions(+), 48 deletions(-)
include/linux/mmzone.h +6 -0
index c0ee2ec..08a223b 100644
@@ -534,6 +534,7 @@ typedef enum {
 	ZONE_WRITEBACK,			/* reclaim scanning has recently found
 					 * many pages under writeback
 					 */
+	ZONE_FAIR_DEPLETED,		/* fair zone policy batch depleted */
 } zone_flags_t;
 
 static inline void zone_set_flag(struct zone *zone, zone_flags_t flag)
@@ -571,6 +572,11 @@ static inline int zone_is_reclaim_locked(const struct zone *zone)
 	return test_bit(ZONE_RECLAIM_LOCKED, &zone->flags);
 }
 
+static inline int zone_is_fair_depleted(const struct zone *zone)
+{
+	return test_bit(ZONE_FAIR_DEPLETED, &zone->flags);
+}
+
 static inline int zone_is_oom_locked(const struct zone *zone)
 {
 	return test_bit(ZONE_OOM_LOCKED, &zone->flags);
mm/page_alloc.c +53 -48

159 lines not on this page · raw message

Signature · 7 lines
1.8.4.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 6/6
#13 lore raw

Re: [PATCH 6/6] mm: page_alloc: Reduce cost of the fair zone allocation policy

Quoted 19 lines · Mel Gorman

The fair zone allocation policy round-robins allocations between zones within a node to avoid age inversion problems during reclaim. If the first allocation fails, the batch counts is reset and a second attempt made before entering the slow path.

One assumption made with this scheme is that batches expire at roughly the same time and the resets each time are justified. This assumption does not hold when zones reach their low watermark as the batches will be consumed at uneven rates. Allocation failure due to watermark depletion result in additional zonelist scans for the reset and another watermark check before hitting the slowpath.

On UMA, the benefit is negligible -- around 0.25%. On 4-socket NUMA machine it's variable due to the variability of measuring overhead with the vmstat changes. The system CPU overhead comparison looks like

3.16.0-rc3 3.16.0-rc3 3.16.0-rc3
vanilla vmstat-v5 lowercost-v5
User 746.94 774.56 802.00

Quoted 8 lines
System      65336.22    32847.27    40852.33
Elapsed     27553.52    27415.04    27368.46

However it is worth noting that the overall benchmark still completed faster and intuitively it makes sense to take as few passes as possible through the zonelists.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#14 lore raw

Re: [PATCH 6/6] mm: page_alloc: Reduce cost of the fair zone allocation policy

In reply to Mel Gorman

mm/page_alloc.c again: · line 1606

Quoted 6 lines · Mel Gorman
mm/page_alloc.c +0 -0
@@ -1604,6 +1604,9 @@ again:
 	}
 
 	__mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));

This can underflow zero, right?

mm/page_alloc.c again: · line 1607

Quoted 1 line
mm/page_alloc.c +1 -0
+	if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&

AFAICS, zone_page_state will correct negative values to zero only for CONFIG_SMP. Won't this check be broken on !CONFIG_SMP?

I just stumbled upon this when trying to optimize the function. I didn't check how rest of the design copes with negative NR_ALLOC_BATCH values.

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 1920

Quoted 9 lines
mm/page_alloc.c +2 -0
+	    !zone_is_fair_depleted(zone))
+		zone_set_flag(zone, ZONE_FAIR_DEPLETED);
 
 	__count_zone_vm_events(PGALLOC, zone, 1 << order);
 	zone_statistics(preferred_zone, zone, gfp_flags);
@@ -1915,6 +1918,18 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
 
 #endif	/* CONFIG_NUMA */
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#15 lore raw

Re: [PATCH 6/6] mm: page_alloc: Reduce cost of the fair zone allocation policy

Quoted 10 lines · Vlastimil Babka

mm/page_alloc.c again: · line 1606

Quoted 6 lines · Mel Gorman
mm/page_alloc.c +0 -0
@@ -1604,6 +1604,9 @@ again:
 	}
 
 	__mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));

This can underflow zero, right?

Yes, because of per-cpu accounting drift.

Quoted 5 lines

mm/page_alloc.c again: · line 1607

Quoted 1 line
mm/page_alloc.c +1 -0
+	if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&

AFAICS, zone_page_state will correct negative values to zero only for CONFIG_SMP. Won't this check be broken on !CONFIG_SMP?

On !CONFIG_SMP how can there be per-cpu accounting drift that would make that counter negative?

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#16 lore raw

Re: [PATCH 6/6] mm: page_alloc: Reduce cost of the fair zone allocation policy

Quoted 7 lines · Mel Gorman
Quoted 6 lines · Vlastimil Babka
Quoted 5 lines · Mel Gorman
mm/page_alloc.c +0 -0
@@ -1604,6 +1604,9 @@ again:
  	}
 
Quoted 6 lines
Quoted 4 lines

mm/page_alloc.c again: · line 1606

Quoted 1 line
mm/page_alloc.c +0 -0
  	__mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));

This can underflow zero, right?

Yes, because of per-cpu accounting drift.

I meant mainly because of order > 0.

Quoted 8 lines
Quoted 5 lines

mm/page_alloc.c again: · line 1607

Quoted 1 line
mm/page_alloc.c +1 -0
+	if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&

AFAICS, zone_page_state will correct negative values to zero only for CONFIG_SMP. Won't this check be broken on !CONFIG_SMP?

On !CONFIG_SMP how can there be per-cpu accounting drift that would make that counter negative?

Well original code used "if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0)" elsewhere, that you are replacing with zone_is_fair_depleted check. I assumed it's because it can get negative due to order > 0. I might have not looked thoroughly enough but it seems to me there's nothing that would prevent it, such as skipping a zone because its remaining batch is lower than 1 << order. So I think the check should be "<= 0" to be safe.

Vlastimil

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#17 lore raw

Re: [PATCH 6/6] mm: page_alloc: Reduce cost of the fair zone allocation policy

Quoted 26 lines · Vlastimil Babka
Quoted 7 lines · Mel Gorman
Quoted 6 lines · Vlastimil Babka
Quoted 5 lines · Mel Gorman
mm/page_alloc.c +0 -0
@@ -1604,6 +1604,9 @@ again:
 	}
 
Quoted 6 lines
Quoted 4 lines

mm/page_alloc.c again: · line 1606

Quoted 1 line
mm/page_alloc.c +0 -0
 	__mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));

This can underflow zero, right?

Yes, because of per-cpu accounting drift.

I meant mainly because of order > 0.

Quoted 8 lines
Quoted 5 lines

mm/page_alloc.c again: · line 1607

Quoted 1 line
mm/page_alloc.c +1 -0
+	if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&

AFAICS, zone_page_state will correct negative values to zero only for CONFIG_SMP. Won't this check be broken on !CONFIG_SMP?

On !CONFIG_SMP how can there be per-cpu accounting drift that would make that counter negative?

Quoted 7 lines

Well original code used "if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0)" elsewhere, that you are replacing with zone_is_fair_depleted check. I assumed it's because it can get negative due to order > 0. I might have not looked thoroughly enough but it seems to me there's nothing that would prevent it, such as skipping a zone because its remaining batch is lower than 1 << order.
So I think the check should be "<= 0" to be safe.

Any updates on this?

The counter can definitely underflow on !CONFIG_SMP, and then the flag gets out of sync with the actual batch state. I'd still prefer just removing this flag again; it's extra complexity and error prone (case in point) while the upsides are not even measurable in real life.

---

include/linux/mmzone.h +0 -6
index 318df7051850..0bd77f730b38 100644
@@ -534,7 +534,6 @@ typedef enum {
 	ZONE_WRITEBACK,			/* reclaim scanning has recently found
 					 * many pages under writeback
 					 */
-	ZONE_FAIR_DEPLETED,		/* fair zone policy batch depleted */
 } zone_flags_t;
 
 static inline void zone_set_flag(struct zone *zone, zone_flags_t flag)
@@ -572,11 +571,6 @@ static inline int zone_is_reclaim_locked(const struct zone *zone)
 	return test_bit(ZONE_RECLAIM_LOCKED, &zone->flags);
 }
 
-static inline int zone_is_fair_depleted(const struct zone *zone)
-{
-	return test_bit(ZONE_FAIR_DEPLETED, &zone->flags);
-}
-
 static inline int zone_is_oom_locked(const struct zone *zone)
 {
 	return test_bit(ZONE_OOM_LOCKED, &zone->flags);
mm/page_alloc.c +1 -5
index 18cee0d4c8a2..d913809a328f 100644
@@ -1612,9 +1612,6 @@ again:
 	}
 
 	__mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));
-	if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&
-	    !zone_is_fair_depleted(zone))
-		zone_set_flag(zone, ZONE_FAIR_DEPLETED);
 
 	__count_zone_vm_events(PGALLOC, zone, 1 << order);
 	zone_statistics(preferred_zone, zone, gfp_flags);
@@ -1934,7 +1931,6 @@ static void reset_alloc_batches(struct zone *preferred_zone)
 		mod_zone_page_state(zone, NR_ALLOC_BATCH,
 			high_wmark_pages(zone) - low_wmark_pages(zone) -
 			atomic_long_read(&zone->vm_stat[NR_ALLOC_BATCH]));
-		zone_clear_flag(zone, ZONE_FAIR_DEPLETED);
 	} while (zone++ != preferred_zone);
 }
 
@@ -1985,7 +1981,7 @@ zonelist_scan:
 		if (alloc_flags & ALLOC_FAIR) {
 			if (!zone_local(preferred_zone, zone))
 				break;
-			if (zone_is_fair_depleted(zone)) {
+			if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0) {
 				nr_fair_skipped++;
 				continue;
 			}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#18 lore raw

PATCH 1/1 mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP

Commit 4ffeaf35 (mm: page_alloc: reduce cost of the fair zone allocation policy) broke the fair zone allocation policy on UP with these hunks.

mm/page_alloc.c +6 -1
index 6e5e8f7..fb99081 100644
@@ -1612,6 +1612,9 @@ again:
        }
 
        __mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));
+       if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&
+           !zone_is_fair_depleted(zone))
+               zone_set_flag(zone, ZONE_FAIR_DEPLETED);
 
        __count_zone_vm_events(PGALLOC, zone, 1 << order);
        zone_statistics(preferred_zone, zone, gfp_flags);
@@ -1966,8 +1985,10 @@ zonelist_scan:
                if (alloc_flags & ALLOC_FAIR) {
                        if (!zone_local(preferred_zone, zone))
                                break;
-                       if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0)
+                       if (zone_is_fair_depleted(zone)) {
+                               nr_fair_skipped++;
                                continue;
+                       }
                }
 

The problem is that a <= check was replaced with a ==. On SMP it doesn't matter because negative values are returned as zero due to per-CPU drift which is not possible in the UP case. Vlastimil Babka correctly pointed out that this can be negative due to high-order allocations. This patch fixes the problem.

---
 mm/page_alloc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
index 18cee0d..cd4c05c 100644
@@ -1612,7 +1612,7 @@ again:
 	}
 
 	__mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));
-	if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&
+	if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0 &&
 	    !zone_is_fair_depleted(zone))
 		zone_set_flag(zone, ZONE_FAIR_DEPLETED);
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#19 lore raw

Re: [PATCH] mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP

Hi Mel,

mm/page_alloc.c again: · line 1615

Quoted 3 lines
        __mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));
mm/page_alloc.c +1 -1
-       if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&
+       if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0 &&

zone_page_state is declared to return unsigned long value [1], so it should never be below 0.
So interesting question: what zone_page_state will return for negative atomic_long_read(&zone->vm_stat[item]) ?

130 static inline unsigned long zone_page_state(struct zone *zone,
131 enum zone_stat_item item)
132 {
133 long x = atomic_long_read(&zone->vm_stat[item]);
134 #ifdef CONFIG_SMP
135 if (x < 0)
136 x = 0;
137 #endif
138 return x;
139 }

[1] https://git.kernel.org/cgit/linux/kernel/git/mhocko/mm.git/tree/include/linux/vmstat.h#n130

Signature · 8 lines
Leon Romanovsky | Independent Linux Consultant
        www.leon.nu | [email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#20 lore raw

PATCH 1/1 mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP v2

Commit 4ffeaf35 (mm: page_alloc: reduce cost of the fair zone allocation policy) arguably broke the fair zone allocation policy on UP with these hunks.

a/mm/page_alloc.c
b/mm/page_alloc.c
@@ -1612,6 +1612,9 @@ again:

       	}

       	__mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));

+ if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&
+ !zone_is_fair_depleted(zone))
+ zone_set_flag(zone, ZONE_FAIR_DEPLETED);

       	__count_zone_vm_events(PGALLOC, zone, 1 << order);
       	zone_statistics(preferred_zone, zone, gfp_flags);
@@ -1966,8 +1985,10 @@ zonelist_scan:
               	if (alloc_flags & ALLOC_FAIR) {
                       	if (!zone_local(preferred_zone, zone))
                               	break;
-                       if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0)
+                       if (zone_is_fair_depleted(zone)) {
+                               nr_fair_skipped++;
                               	continue;
+                       }
               	}

A <= check was replaced with a ==. On SMP it doesn't matter because negative values are returned as zero due to per-CPU drift which is not possible in the UP case. Vlastimil Babka correctly pointed out that this can wrap negative due to high-order allocations.

However, Leon Romanovsky pointed out that a <= check on zone_page_state was never correct as zone_page_state returns unsigned long so the root cause of the breakage was the <= check in the first place.

zone_page_state is an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allow NR_ALLOC_BATCH to go negative -- when the counter is reset the negative value takes recent activity into account. This patch makes zone_page_state behave the same on SMP and UP as saving one branch on UP is not likely to make a measurable performance difference.

---
 include/linux/vmstat.h | 2 --
 1 file changed, 2 deletions(-)
include/linux/vmstat.h +0 -2
index 82e7db7..cece0f0 100644
@@ -131,10 +131,8 @@ static inline unsigned long zone_page_state(struct zone *zone,
 					enum zone_stat_item item)
 {
 	long x = atomic_long_read(&zone->vm_stat[item]);
-#ifdef CONFIG_SMP
 	if (x < 0)
 		x = 0;
-#endif
 	return x;
 }
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#21 lore raw

Re: [PATCH] mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP v2

Hi Mel,

Quoted 56 lines · Mel Gorman

Commit 4ffeaf35 (mm: page_alloc: reduce cost of the fair zone allocation policy) arguably broke the fair zone allocation policy on UP with these hunks.

a/mm/page_alloc.c
b/mm/page_alloc.c

(quoted diff) +6 -1
@@ -1612,6 +1612,9 @@ again:
        }
 
        __mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));
+       if (zone_page_state(zone, NR_ALLOC_BATCH) == 0 &&
+           !zone_is_fair_depleted(zone))
+               zone_set_flag(zone, ZONE_FAIR_DEPLETED);
 
        __count_zone_vm_events(PGALLOC, zone, 1 << order);
        zone_statistics(preferred_zone, zone, gfp_flags);
@@ -1966,8 +1985,10 @@ zonelist_scan:
                if (alloc_flags & ALLOC_FAIR) {
                        if (!zone_local(preferred_zone, zone))
                                break;
-                       if (zone_page_state(zone, NR_ALLOC_BATCH) <= 0)
+                       if (zone_is_fair_depleted(zone)) {
+                               nr_fair_skipped++;
                                continue;
+                       }
                }
 

A <= check was replaced with a ==. On SMP it doesn't matter because negative values are returned as zero due to per-CPU drift which is not possible in the UP case. Vlastimil Babka correctly pointed out that this can wrap negative due to high-order allocations.

However, Leon Romanovsky pointed out that a <= check on zone_page_state was never correct as zone_page_state returns unsigned long so the root cause of the breakage was the <= check in the first place.

zone_page_state is an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allow NR_ALLOC_BATCH to go negative -- when the counter is reset the negative value takes recent activity into account. This patch makes zone_page_state behave the same on SMP and UP as saving one branch on UP is not likely to make a measurable performance difference.

---
 include/linux/vmstat.h | 2 --
 1 file changed, 2 deletions(-)
include/linux/vmstat.h +0 -0
index 82e7db7..cece0f0 100644

include/linux/vmstat.h static inline unsigned long zone_page_state(struct · line 138

Quoted 10 lines
include/linux/vmstat.h +0 -2
@@ -131,10 +131,8 @@ static inline unsigned long zone_page_state(struct
 zone *zone,
                                        enum zone_stat_item item)
 {
        long x = atomic_long_read(&zone->vm_stat[item]);
-#ifdef CONFIG_SMP
        if (x < 0)
                x = 0;
-#endif
        return x;
 }

Since you are changing vmstat.h, what do you think about change in all similiar places?

include/linux/vmstat.h +0 -0
index 82e7db7..88d3d3e 100644
@@ -120,10 +120,8 @@ static inline void zone_page_state_add(long x, struct

zone *zone,
static inline unsigned long global_page_state(enum zone_stat_item item)

 {
        long x = atomic_long_read(&vm_stat[item]);
-#ifdef CONFIG_SMP
        if (x < 0)
                x = 0;
-#endif
        return x;

}

@@ -131,10 +129,8 @@ static inline unsigned long zone_page_state(struct zone *zone,
enum zone_stat_item item)

 {
        long x = atomic_long_read(&zone->vm_stat[item]);
-#ifdef CONFIG_SMP
        if (x < 0)
                x = 0;
-#endif
        return x;

}

@@ -153,10 +149,9 @@ static inline unsigned long
zone_page_state_snapshot(struct zone *zone,

        int cpu;
        for_each_online_cpu(cpu)
                x += per_cpu_ptr(zone->pageset, cpu)->vm_stat_diff[item];
-
+#endif
        if (x < 0)
                x = 0;
-#endif
        return x;

}

Signature · 2 lines
Leon Romanovsky | Independent Linux Consultant
        www.leon.nu | [email protected]
maintainer
#22 lore raw

Re: [PATCH] mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP v2

In reply to Mel Gorman
Quoted 12 lines · On Mon, 8 Sep 2014 12:57:18 +0100 Mel Gorman <[email protected]> wrote:

zone_page_state is an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allow NR_ALLOC_BATCH to go negative -- when the counter is reset the negative value takes recent activity into account. This patch makes zone_page_state behave the same on SMP and UP as saving one branch on UP is not likely to make a measurable performance difference.

...

include/linux/vmstat.h static inline unsigned long zone_page_state(struct zone *zone, · line 137

Quoted 9 lines
include/linux/vmstat.h +0 -2
@@ -131,10 +131,8 @@ static inline unsigned long zone_page_state(struct zone *zone,
 					enum zone_stat_item item)
 {
 	long x = atomic_long_read(&zone->vm_stat[item]);
-#ifdef CONFIG_SMP
 	if (x < 0)
 		x = 0;
-#endif
 	return x;
 }

We now have three fixes for the same thing. I'm presently holding on to hannes's mm-page_alloc-fix-zone-allocation-fairness-on-up.patch.

Regularizing zone_page_state() in this fashion seems a good idea and is presumably safe because callers have been tested with SMP. So unless shouted at I think I'll queue this one for 3.18?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#23 lore raw

Re: [PATCH] mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP v2

Quoted 11 lines · Andrew Morton
Quoted 9 lines · On Mon, 8 Sep 2014 12:57:18 +0100 Mel Gorman <[email protected]> wrote:

zone_page_state is an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allow NR_ALLOC_BATCH to go negative -- when the counter is reset the negative value takes recent activity into account. This patch makes zone_page_state behave the same on SMP and UP as saving one branch on UP is not likely to make a measurable performance difference.

...

Quoted 14 lines

include/linux/vmstat.h static inline unsigned long zone_page_state(struct zone *zone, · line 137

Quoted 12 lines
include/linux/vmstat.h +0 -2
@@ -131,10 +131,8 @@ static inline unsigned long zone_page_state(struct zone *zone,
 					enum zone_stat_item item)
 {
 	long x = atomic_long_read(&zone->vm_stat[item]);
-#ifdef CONFIG_SMP
 	if (x < 0)
 		x = 0;
-#endif
 	return x;
 }

We now have three fixes for the same thing.

This might be holding a record for most patches for what should have been a trivial issue :P

Quoted 3 lines

I'm presently holding on
to hannes's mm-page_alloc-fix-zone-allocation-fairness-on-up.patch.

This is my preferred fix because it clearly points to where the source of the original problem is. Furthermore, the second hunk really should be reading the unsigned counter value. It's an inconsequential corner-case but it's still more correct although it's a pity that it's also a layering violation. However, adding a new API to return the raw value on UP and SMP is likely to be interpreted as unwelcome indirection.

Quoted 3 lines

Regularizing zone_page_state() in this fashion seems a good idea and is presumably safe because callers have been tested with SMP. So unless shouted at I think I'll queue this one for 3.18?

Both are ok but if we really want to regularise the API then all readers should be brought in line and declared an API cleanup. That looks like the following;

---8<---
From: Mel Gorman <[email protected]>
Subject: [PATCH] mm: vmstat: regularize UP and SMP behavior

zone_page_state and friends are an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allow NR_ALLOC_BATCH to go negative -- when the counter is reset the negative value takes recent activity into account. NR_ALLOC_BATCH callers that matter access the raw counter but the API hazard is a lesson.

This patch makes zone_page_state, global_page_state and zone_page_state_snapshot return the same values on SMP and UP as saving the branches on UP is unlikely to make a measurable performance difference.

---
 include/linux/vmstat.h | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
include/linux/vmstat.h +2 -6
index 82e7db7..873104e 100644
@@ -120,10 +120,8 @@ static inline void zone_page_state_add(long x, struct zone *zone,
 static inline unsigned long global_page_state(enum zone_stat_item item)
 {
 	long x = atomic_long_read(&vm_stat[item]);
-#ifdef CONFIG_SMP
 	if (x < 0)
 		x = 0;
-#endif
 	return x;
 }
 
@@ -131,10 +129,8 @@ static inline unsigned long zone_page_state(struct zone *zone,
 					enum zone_stat_item item)
 {
 	long x = atomic_long_read(&zone->vm_stat[item]);
-#ifdef CONFIG_SMP
 	if (x < 0)
 		x = 0;
-#endif
 	return x;
 }
 
@@ -153,10 +149,10 @@ static inline unsigned long zone_page_state_snapshot(struct zone *zone,
 	int cpu;
 	for_each_online_cpu(cpu)
 		x += per_cpu_ptr(zone->pageset, cpu)->vm_stat_diff[item];
-
+#endif
 	if (x < 0)
 		x = 0;
-#endif
+
 	return x;
 }
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#24 lore raw

Re: [PATCH] mm: page_alloc: Fix setting of ZONE_FAIR_DEPLETED on UP v2

Quoted 59 lines · Mel Gorman
Quoted 11 lines · Andrew Morton
Quoted 9 lines · On Mon, 8 Sep 2014 12:57:18 +0100 Mel Gorman <[email protected]> wrote:

zone_page_state is an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allow NR_ALLOC_BATCH to go negative -- when the counter is reset the negative value takes recent activity into account. This patch makes zone_page_state behave the same on SMP and UP as saving one branch on UP is not likely to make a measurable performance difference.

...

Quoted 14 lines

include/linux/vmstat.h static inline unsigned long zone_page_state(struct zone *zone, · line 137

Quoted 12 lines
include/linux/vmstat.h +0 -2
@@ -131,10 +131,8 @@ static inline unsigned long zone_page_state(struct zone *zone,
 					enum zone_stat_item item)
 {
 	long x = atomic_long_read(&zone->vm_stat[item]);
-#ifdef CONFIG_SMP
 	if (x < 0)
 		x = 0;
-#endif
 	return x;
 }

We now have three fixes for the same thing.

This might be holding a record for most patches for what should have been a trivial issue :P

Quoted 3 lines

I'm presently holding on
to hannes's mm-page_alloc-fix-zone-allocation-fairness-on-up.patch.

This is my preferred fix because it clearly points to where the source of the original problem is. Furthermore, the second hunk really should be reading the unsigned counter value. It's an inconsequential corner-case but it's still more correct although it's a pity that it's also a layering violation. However, adding a new API to return the raw value on UP and SMP is likely to be interpreted as unwelcome indirection.

Quoted 3 lines

Regularizing zone_page_state() in this fashion seems a good idea and is presumably safe because callers have been tested with SMP. So unless shouted at I think I'll queue this one for 3.18?

Both are ok but if we really want to regularise the API then all readers should be brought in line and declared an API cleanup. That looks like the following;

---8<---
From: Mel Gorman <[email protected]>
Subject: [PATCH] mm: vmstat: regularize UP and SMP behavior

zone_page_state and friends are an API hazard because of the difference in behaviour between SMP and UP is very surprising. There is a good reason to allow NR_ALLOC_BATCH to go negative -- when the counter is reset the negative value takes recent activity into account. NR_ALLOC_BATCH callers that matter access the raw counter but the API hazard is a lesson.

Quoted 3 lines

This patch makes zone_page_state, global_page_state and zone_page_state_snapshot return the same values on SMP and UP as saving the branches on UP is unlikely to make a measurable performance difference.

The API still returns an unsigned long and so can not really support counters that can go logically negative - as opposed to negative reads that are a side-effect of concurrency and can be interpreted as zero.

The problem is that the fairness batches abuse the internals of what is publicly an unsigned counter to implement an unlocked allocator that needs to be able to go negative. We could make that counter API truly signed to support that, but I think this patch makes the code actually more confusing and inconsistent.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>