Skip to contentSkip to conversation

Per-cpu page allocator replacement prototype

RFC PATCH In review Mel Gorman 4 participants 33 messages May 8, 2013 – May 15, 2013

Subsystems (16)
lore
Share & tools
lore

Mel Gorman

author
#0 lore raw

cover Per-cpu page allocator replacement prototype

Two LSF/MM's ago there was discussion on the per-cpu page allocator and whether it could be replaced due to it's complexity, frequent drains/refills and IPI overhead for global drains. The main obstacle to removal is that without those lists the zone->lock becomes very heavily contended and alternatives are inevitably going to share cache lines. I prototyped a potential replacement on the flight home and then left it on a TODO list for another year.

Last LSF/MM this was talked about in the hallway again although the meat of the discussion was different and took into account Andi Kleen's talk about lock batching. On this flight home, I rebased the old prototype, added some additional bits and pieces and tidied it up a bit since. This TODO item is about 3 years old so apparently sometimes the only way to get someone to do something is to lock them in a metal box for a few hours.

This is a prototype replacement starts with some minor optimisations that have nothing to do with anything really other than they were floating around from another flights worth of work. It then replaces the per-cpu page allocator with an IRQ-unsafe (no interrupts, no calls with local_irq_save) magazine that is protected by a spinlock. This effectively has the allocator use two locks. magazine_lock for non-IRQ users (e.g. page faults) and the zone->lock for magazine drains/refills and users who have IRQs disabled (interrupts, slab). It then splits the magazine into two where the preferred magazine depends on the CPU id of the executing thread. Non-preferred magazines may also be used and are searched round-robin as they are only protected by spinlocks. The last part of the series does some mucking around with lock contention and batching multiple frees due to exit or compaction under the lock.

It has not been heavily tested in low memory, heavy interrupt situations or extensively with all the debugging options enabled so it's likely there are bugs hiding in there. However I'm interested in hearing if the per-cpu page allocator is something we really want to replace or if there is a better potential alternative than this prototype. There are some interesting possibilities with this sort of design. For example, it would be possible to allocate magazines to allocator-intensive processes that are chained together for global drains where they are necessary. For these processes there would be much less contention (only on drains/refills) without having to use IPIs to drain their pages.

In the follow tests, no debugging was enabled but profiling was running so the tests are heavily disrupted. Take the results with a grain of salt. Machine was a single socket i7 with 16G RAM.

kernbench

                               3.9.0                 3.9.0
                             vanilla    magazine
User    min         883.59 (  0.00%)      700.16 ( 20.76%)
User    mean        891.11 (  0.00%)      741.02 ( 16.84%)
User    stddev       12.14 (  0.00%)       50.04 (-312.18%)
User    max         915.30 (  0.00%)      817.50 ( 10.69%)
User    range        31.71 (  0.00%)      117.34 (-270.04%)
System  min          58.85 (  0.00%)       43.65 ( 25.83%)
System  mean         59.65 (  0.00%)       47.65 ( 20.12%)
System  stddev        1.35 (  0.00%)        4.79 (-254.39%)
System  max          62.35 (  0.00%)       55.35 ( 11.23%)
System  range         3.50 (  0.00%)       11.70 (-234.29%)
Elapsed min         127.75 (  0.00%)       99.37 ( 22.22%)
Elapsed mean        129.19 (  0.00%)      105.73 ( 18.16%)
Elapsed stddev        2.07 (  0.00%)        7.56 (-265.78%)
Elapsed max         133.25 (  0.00%)      117.53 ( 11.80%)
Elapsed range         5.50 (  0.00%)       18.16 (-230.18%)
CPU     min         731.00 (  0.00%)      742.00 ( -1.50%)
CPU     mean        735.20 (  0.00%)      745.60 ( -1.41%)
CPU     stddev        2.99 (  0.00%)        3.38 (-12.99%)
CPU     max         739.00 (  0.00%)      751.00 ( -1.62%)
CPU     range         8.00 (  0.00%)        9.00 (-12.50%)

Kernel build benchmark seemed fairly successful, if anything they seem too good and I suspect this might have been a particularly "lucky" run. A non-profiling run might reveal more.

pagealloc

                                               3.9.0                      3.9.0
                                             vanilla         magazine
order-0 alloc-1                     671.11 (  0.00%)           734.11 ( -9.39%)
order-0 alloc-2                     520.33 (  0.00%)           517.56 (  0.53%)
order-0 alloc-4                     426.56 (  0.00%)           461.78 ( -8.26%)
order-0 alloc-8                     666.33 (  0.00%)           381.67 ( 42.72%)
order-0 alloc-16                    341.78 (  0.00%)           354.44 ( -3.71%)
order-0 alloc-32                    336.89 (  0.00%)           345.33 ( -2.51%)
order-0 alloc-64                    331.33 (  0.00%)           324.56 (  2.05%)
order-0 alloc-128                   324.56 (  0.00%)           325.89 ( -0.41%)
order-0 alloc-256                   350.44 (  0.00%)           326.22 (  6.91%)
order-0 alloc-512                   369.33 (  0.00%)           345.67 (  6.41%)
order-0 alloc-1024                  381.44 (  0.00%)           352.67 (  7.54%)
order-0 alloc-2048                  387.50 (  0.00%)           348.00 ( 10.19%)
order-0 alloc-4096                  403.00 (  0.00%)           384.50 (  4.59%)
order-0 alloc-8192                  413.00 (  0.00%)           383.00 (  7.26%)
order-0 alloc-16384                 411.00 (  0.00%)           396.80 (  3.45%)
order-0 free-1                      357.22 (  0.00%)           458.89 (-28.46%)
order-0 free-2                      285.11 (  0.00%)           349.89 (-22.72%)
order-0 free-4                      231.33 (  0.00%)           296.00 (-27.95%)
order-0 free-8                      371.56 (  0.00%)           229.33 ( 38.28%)
order-0 free-16                     189.78 (  0.00%)           212.89 (-12.18%)
order-0 free-32                     185.67 (  0.00%)           206.11 (-11.01%)
order-0 free-64                     178.44 (  0.00%)           197.78 (-10.83%)
order-0 free-128                    178.11 (  0.00%)           197.00 (-10.61%)
order-0 free-256                    227.89 (  0.00%)           196.56 ( 13.75%)
order-0 free-512                    280.11 (  0.00%)           194.67 ( 30.50%)
order-0 free-1024                   301.67 (  0.00%)           227.50 ( 24.59%)
order-0 free-2048                   325.50 (  0.00%)           238.00 ( 26.88%)
order-0 free-4096                   328.00 (  0.00%)           278.25 ( 15.17%)
order-0 free-8192                   337.50 (  0.00%)           277.00 ( 17.93%)
order-0 free-16384                  338.00 (  0.00%)           285.20 ( 15.62%)
order-0 total-1                    1031.89 (  0.00%)          1197.22 (-16.02%)
order-0 total-2                     805.44 (  0.00%)           867.44 ( -7.70%)
order-0 total-4                     657.89 (  0.00%)           768.67 (-16.84%)
order-0 total-8                    1039.56 (  0.00%)           611.00 ( 41.22%)
order-0 total-16                    531.56 (  0.00%)           569.33 ( -7.11%)
order-0 total-32                    523.44 (  0.00%)           551.44 ( -5.35%)
order-0 total-64                    509.78 (  0.00%)           522.89 ( -2.57%)
order-0 total-128                   502.67 (  0.00%)           522.89 ( -4.02%)
order-0 total-256                   579.56 (  0.00%)           522.78 (  9.80%)
order-0 total-512                   649.44 (  0.00%)           540.33 ( 16.80%)
order-0 total-1024                  686.44 (  0.00%)           580.17 ( 15.48%)
order-0 total-2048                  713.00 (  0.00%)           586.00 ( 17.81%)
order-0 total-4096                  731.00 (  0.00%)           663.75 (  9.20%)
order-0 total-8192                  750.50 (  0.00%)           660.00 ( 12.06%)
order-0 total-16384                 749.00 (  0.00%)           683.00 (  8.81%)

This is a systemtap-drive page allocator microbenchmark that allocates order-0 pages in increasingly large "batches" and times the length of time it takes to allocate and free. The results are a bit all over the map. Frees are slower because they always wait for the preferred magazine to be free and freeing single pages in a loop is a worst-case scenario. For larger patches it tends to perform reasonably well though.

pft

                             3.9.0                 3.9.0
                           vanilla               magazine
Faults/cpu 1  953441.5530 (  0.00%)  954011.8576 (  0.06%)
Faults/cpu 2  923793.1533 (  0.00%)  889436.7293 ( -3.72%)
Faults/cpu 3  876829.2292 (  0.00%)  868230.6471 ( -0.98%)
Faults/cpu 4  819914.9333 (  0.00%)  735264.4793 (-10.32%)
Faults/cpu 5  689049.0107 (  0.00%)  663481.5446 ( -3.71%)
Faults/cpu 6  579924.4065 (  0.00%)  571889.3687 ( -1.39%)
Faults/cpu 7  552024.1040 (  0.00%)  494345.8698 (-10.45%)
Faults/cpu 8  461452.7560 (  0.00%)  457877.1810 ( -0.77%)
Faults/sec 1  938245.7112 (  0.00%)  939198.4047 (  0.10%)
Faults/sec 2 1814498.4087 (  0.00%) 1748800.3800 ( -3.62%)
Faults/sec 3 2544466.6368 (  0.00%) 2359068.2163 ( -7.29%)
Faults/sec 4 3032778.8584 (  0.00%) 2831753.6553 ( -6.63%)
Faults/sec 5 3025180.2736 (  0.00%) 2952758.4589 ( -2.39%)
Faults/sec 6 3131131.0106 (  0.00%) 3058954.4941 ( -2.31%)
Faults/sec 7 3286271.0631 (  0.00%) 3183931.7940 ( -3.11%)
Faults/sec 8 3135331.0027 (  0.00%) 3106746.5908 ( -0.91%)

This is a page faulting microbenchmark that is forced to use base pages only. Here the new design suffers a bit because the allocation path is likely to contend on the magazine lock.

So preliminary testing indicates the results are mixed bag. As long as locks are not contended, it performs fine but parallel fault testing hits into spinlock contention on the magazine locks. A greater problem is that because CPUs share magazines it means that the struct pages are frequently dirtied cache lines. If CPU A frees a page to a magazine and CPU B immediately allocates it then the cache line for the page and the magazine bounces and this costs. It's on the TODO list to research if the available literature has anything useful to say that does not depend on per-cpu lists and the associated problems with them.

Comments?

 arch/sparc/mm/init_64.c        |    4 +-
 arch/sparc/mm/tsb.c            |    2 +-
 arch/tile/mm/homecache.c       |    2 +-
 fs/fuse/dev.c                  |    2 +-
 include/linux/gfp.h            |   12 +-
 include/linux/mm.h             |    3 -
 include/linux/mmzone.h         |   46 +-
 include/linux/page-isolation.h |    7 +-
 include/linux/pagemap.h        |    2 +-
 include/linux/swap.h           |    2 +-
 include/trace/events/kmem.h    |   22 +-
 init/main.c                    |    1 -
 kernel/power/snapshot.c        |    2 -
 kernel/sysctl.c                |   10 -
 mm/compaction.c                |   18 +-
 mm/memory-failure.c            |    2 +-
 mm/memory_hotplug.c            |   13 +-
 mm/page_alloc.c                | 1109 +++++++++++++++++-----------------------
 mm/page_isolation.c            |   30 +-
 mm/rmap.c                      |    2 +-
 mm/swap.c                      |    6 +-
 mm/swap_state.c                |    2 +-
 mm/vmscan.c                    |    6 +-
 mm/vmstat.c                    |   54 +-
 24 files changed, 571 insertions(+), 788 deletions(-)
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#1 lore raw

PATCH 1/22 mm: page allocator: Lookup pageblock migratetype with IRQs enabled during free

get_pageblock_migratetype() is called during free with IRQs disabled. This is unnecessary and disables IRQs for longer than necessary.

---
 mm/page_alloc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/page_alloc.c +2 -1
index 8fcced7..277ecee 100644
@@ -730,9 +730,10 @@ static void __free_pages_ok(struct page *page, unsigned int order)
 	if (!free_pages_prepare(page, order))
 		return;
 
+	migratetype = get_pageblock_migratetype(page);
+
 	local_irq_save(flags);
 	__count_vm_events(PGFREE, 1 << order);
-	migratetype = get_pageblock_migratetype(page);
 	set_freepage_migratetype(page, migratetype);
 	free_one_page(page_zone(page), page, order, migratetype);
 	local_irq_restore(flags);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#2 lore raw

PATCH 2/22 mm: page allocator: Push down where IRQs are disabled during page free

In reply to Mel Gorman

This patch pushes IRQ disabling down into free_one_page(). This simplifies the logic in free_hot_cold_page() slightly by making it clear that zone->lock is an IRQ-safe spinlock. The current arrangement has the IRQ disabling happen in one function and the spinlock been taken in another. Functionally, there is no difference.

---
 mm/page_alloc.c | 21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)
mm/page_alloc.c +10 -11
index 277ecee..50c9315 100644
@@ -686,14 +686,18 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 static void free_one_page(struct zone *zone, struct page *page, int order,
 				int migratetype)
 {
-	spin_lock(&zone->lock);
+	unsigned long flags;
+	set_freepage_migratetype(page, migratetype);
+
+	spin_lock_irqsave(&zone->lock, flags);
 	zone->all_unreclaimable = 0;
 	zone->pages_scanned = 0;
+	__count_vm_events(PGFREE, 1 << order);
 
 	__free_one_page(page, zone, order, migratetype);
 	if (unlikely(!is_migrate_isolate(migratetype)))
 		__mod_zone_freepage_state(zone, 1 << order, migratetype);
-	spin_unlock(&zone->lock);
+	spin_unlock_irqrestore(&zone->lock, flags);
 }
 
 static bool free_pages_prepare(struct page *page, unsigned int order)
@@ -724,7 +728,6 @@ static bool free_pages_prepare(struct page *page, unsigned int order)
 
 static void __free_pages_ok(struct page *page, unsigned int order)
 {
-	unsigned long flags;
 	int migratetype;
 
 	if (!free_pages_prepare(page, order))
@@ -732,11 +735,7 @@ static void __free_pages_ok(struct page *page, unsigned int order)
 
 	migratetype = get_pageblock_migratetype(page);
 
-	local_irq_save(flags);
-	__count_vm_events(PGFREE, 1 << order);
-	set_freepage_migratetype(page, migratetype);
 	free_one_page(page_zone(page), page, order, migratetype);
-	local_irq_restore(flags);
 }
 
 /*
@@ -1325,8 +1324,6 @@ void free_hot_cold_page(struct page *page, int cold)
 
 	migratetype = get_pageblock_migratetype(page);
 	set_freepage_migratetype(page, migratetype);
-	local_irq_save(flags);
-	__count_vm_event(PGFREE);
 
 	/*
 	 * We only track unmovable, reclaimable and movable on pcp lists.
@@ -1338,11 +1335,14 @@ void free_hot_cold_page(struct page *page, int cold)
 	if (migratetype >= MIGRATE_PCPTYPES) {
 		if (unlikely(is_migrate_isolate(migratetype))) {
 			free_one_page(zone, page, 0, migratetype);
-			goto out;
+			return;
 		}
 		migratetype = MIGRATE_MOVABLE;
 	}
 
+	local_irq_save(flags);
+	__count_vm_event(PGFREE);
+
 	pcp = &this_cpu_ptr(zone->pageset)->pcp;
 	if (cold)
 		list_add_tail(&page->lru, &pcp->lists[migratetype]);
@@ -1354,7 +1354,6 @@ void free_hot_cold_page(struct page *page, int cold)
 		pcp->count -= pcp->batch;
 	}
 
-out:
 	local_irq_restore(flags);
 }
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#3 lore raw

PATCH 3/22 mm: page allocator: Use unsigned int for order in more places

In reply to Mel Gorman

X86 prefers the use of unsigned types for iterators and there is a tendency to mix whether a signed or unsigned type if used for page order. This converts a number of sites in mm/page_alloc.c to use unsigned int for order where possible.

---
 include/linux/mmzone.h |  8 ++++----
 mm/page_alloc.c        | 35 +++++++++++++++++++----------------
 2 files changed, 23 insertions(+), 20 deletions(-)
include/linux/mmzone.h +4 -4
index c74092e..e71e3a6 100644
@@ -773,10 +773,10 @@ static inline bool pgdat_is_empty(pg_data_t *pgdat)
 extern struct mutex zonelists_mutex;
 void build_all_zonelists(pg_data_t *pgdat, struct zone *zone);
 void wakeup_kswapd(struct zone *zone, int order, enum zone_type classzone_idx);
-bool zone_watermark_ok(struct zone *z, int order, unsigned long mark,
-		int classzone_idx, int alloc_flags);
-bool zone_watermark_ok_safe(struct zone *z, int order, unsigned long mark,
-		int classzone_idx, int alloc_flags);
+bool zone_watermark_ok(struct zone *z, unsigned int order,
+		unsigned long mark, int classzone_idx, int alloc_flags);
+bool zone_watermark_ok_safe(struct zone *z, unsigned int order,
+		unsigned long mark, int classzone_idx, int alloc_flags);
 enum memmap_context {
 	MEMMAP_EARLY,
 	MEMMAP_HOTPLUG,
mm/page_alloc.c +19 -16
index 50c9315..4a07771 100644
@@ -392,7 +392,8 @@ static int destroy_compound_page(struct page *page, unsigned long order)
 	return bad;
 }
 
-static inline void prep_zero_page(struct page *page, int order, gfp_t gfp_flags)
+static inline void prep_zero_page(struct page *page, unsigned int order,
+							gfp_t gfp_flags)
 {
 	int i;
 
@@ -436,7 +437,7 @@ static inline void set_page_guard_flag(struct page *page) { }
 static inline void clear_page_guard_flag(struct page *page) { }
 #endif
 
-static inline void set_page_order(struct page *page, int order)
+static inline void set_page_order(struct page *page, unsigned int order)
 {
 	set_page_private(page, order);
 	__SetPageBuddy(page);
@@ -485,7 +486,7 @@ __find_buddy_index(unsigned long page_idx, unsigned int order)
  * For recording page's order, we use page_private(page).
  */
 static inline int page_is_buddy(struct page *page, struct page *buddy,
-								int order)
+							unsigned int order)
 {
 	if (!pfn_valid_within(page_to_pfn(buddy)))
 		return 0;
@@ -683,7 +684,8 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 	spin_unlock(&zone->lock);
 }
 
-static void free_one_page(struct zone *zone, struct page *page, int order,
+static void free_one_page(struct zone *zone, struct page *page,
+				unsigned int order,
 				int migratetype)
 {
 	unsigned long flags;
@@ -853,7 +855,7 @@ static inline int check_new_page(struct page *page)
 	return 0;
 }
 
-static int prep_new_page(struct page *page, int order, gfp_t gfp_flags)
+static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags)
 {
 	int i;
 
@@ -1278,7 +1280,7 @@ void mark_free_pages(struct zone *zone)
 {
 	unsigned long pfn, max_zone_pfn;
 	unsigned long flags;
-	int order, t;
+	unsigned int order, t;
 	struct list_head *curr;
 
 	if (!zone->spanned_pages)
@@ -1471,8 +1473,8 @@ int split_free_page(struct page *page)
  */
 static inline
 struct page *buffered_rmqueue(struct zone *preferred_zone,
-			struct zone *zone, int order, gfp_t gfp_flags,
-			int migratetype)
+			struct zone *zone, unsigned int order,
+			gfp_t gfp_flags, int migratetype)
 {
 	unsigned long flags;
 	struct page *page;
@@ -1619,8 +1621,9 @@ static inline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
  * Return true if free pages are above 'mark'. This takes into account the order
  * of the allocation.
  */
-static bool __zone_watermark_ok(struct zone *z, int order, unsigned long mark,
-		      int classzone_idx, int alloc_flags, long free_pages)
+static bool __zone_watermark_ok(struct zone *z, unsigned int order,
+			unsigned long mark, int classzone_idx, int alloc_flags,
+			long free_pages)
 {
 	/* free_pages my go negative - that's OK */
 	long min = mark;
@@ -1652,15 +1655,15 @@ static bool __zone_watermark_ok(struct zone *z, int order, unsigned long mark,
 	return true;
 }
 
-bool zone_watermark_ok(struct zone *z, int order, unsigned long mark,
+bool zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
 		      int classzone_idx, int alloc_flags)
 {
 	return __zone_watermark_ok(z, order, mark, classzone_idx, alloc_flags,
 					zone_page_state(z, NR_FREE_PAGES));
 }
 
-bool zone_watermark_ok_safe(struct zone *z, int order, unsigned long mark,
-		      int classzone_idx, int alloc_flags)
+bool zone_watermark_ok_safe(struct zone *z, unsigned int order,
+			unsigned long mark, int classzone_idx, int alloc_flags)
 {
 	long free_pages = zone_page_state(z, NR_FREE_PAGES);
 
@@ -3942,7 +3945,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
 
 static void __meminit zone_init_free_lists(struct zone *zone)
 {
-	int order, t;
+	unsigned int order, t;
 	for_each_migratetype_order(order, t) {
 		INIT_LIST_HEAD(&zone->free_area[order].free_list[t]);
 		zone->free_area[order].nr_free = 0;
@@ -6038,7 +6041,7 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
 {
 	struct page *page;
 	struct zone *zone;
-	int order, i;
+	unsigned int order, i;
 	unsigned long pfn;
 	unsigned long flags;
 	/* find the first valid pfn */
@@ -6090,7 +6093,7 @@ bool is_free_buddy_page(struct page *page)
 	struct zone *zone = page_zone(page);
 	unsigned long pfn = page_to_pfn(page);
 	unsigned long flags;
-	int order;
+	unsigned int order;
 
 	spin_lock_irqsave(&zone->lock, flags);
 	for (order = 0; order < MAX_ORDER; order++) {
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#4 lore raw

PATCH 4/22 mm: page allocator: Only check migratetype of pages being drained while CMA active

In reply to Mel Gorman

CMA added a is_migrate_isolate_page in the bulk page free path which does a pageblock migratetype lookup for every page being drained. This is only necessary when CMA is active so skip the expensive checks in the normal case.

---
 include/linux/mmzone.h         |  8 ++++++--
 include/linux/page-isolation.h |  7 ++++---
 mm/page_alloc.c                |  2 +-
 mm/page_isolation.c            | 27 +++++++++++++++++++++++----
 4 files changed, 34 insertions(+), 10 deletions(-)
include/linux/mmzone.h +6 -2
index e71e3a6..57f03b3 100644
@@ -354,12 +354,16 @@ struct zone {
 	spinlock_t		lock;
 	int                     all_unreclaimable; /* All pages pinned */
 #if defined CONFIG_COMPACTION || defined CONFIG_CMA
+	unsigned long		compact_cached_free_pfn;
+	unsigned long		compact_cached_migrate_pfn;
+
 	/* Set to true when the PG_migrate_skip bits should be cleared */
 	bool			compact_blockskip_flush;
 
 	/* pfns where compaction scanners should start */
-	unsigned long		compact_cached_free_pfn;
-	unsigned long		compact_cached_migrate_pfn;
+#endif
+#ifdef CONFIG_MEMORY_ISOLATION
+	bool			memory_isolation_active;
 #endif
 #ifdef CONFIG_MEMORY_HOTPLUG
 	/* see spanned/present_pages for more description */
include/linux/page-isolation.h +4 -3
index 3fff8e7..81287bb 100644
@@ -2,16 +2,17 @@
 #define __LINUX_PAGEISOLATION_H
 
 #ifdef CONFIG_MEMORY_ISOLATION
-static inline bool is_migrate_isolate_page(struct page *page)
+static inline bool is_migrate_isolate_page(struct zone *zone, struct page *page)
 {
-	return get_pageblock_migratetype(page) == MIGRATE_ISOLATE;
+	return zone->memory_isolation_active &&
+		get_pageblock_migratetype(page) == MIGRATE_ISOLATE;
 }
 static inline bool is_migrate_isolate(int migratetype)
 {
 	return migratetype == MIGRATE_ISOLATE;
 }
 #else
-static inline bool is_migrate_isolate_page(struct page *page)
+static inline bool is_migrate_isolate_page(struct zone *zone, struct page *page)
 {
 	return false;
 }
mm/page_alloc.c +1 -1
index 4a07771..f170260 100644
@@ -674,7 +674,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 			/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */
 			__free_one_page(page, zone, 0, mt);
 			trace_mm_page_pcpu_drain(page, 0, mt);
-			if (likely(!is_migrate_isolate_page(page))) {
+			if (likely(!is_migrate_isolate_page(zone, page))) {
 				__mod_zone_page_state(zone, NR_FREE_PAGES, 1);
 				if (is_migrate_cma(mt))
 					__mod_zone_page_state(zone, NR_FREE_CMA_PAGES, 1);
mm/page_isolation.c +23 -4
index 383bdbb..9f0c068 100644
@@ -118,6 +118,8 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
 	unsigned long pfn;
 	unsigned long undo_pfn;
 	struct page *page;
+	struct zone *zone = NULL;
+	unsigned long flags;
 
 	BUG_ON((start_pfn) & (pageblock_nr_pages - 1));
 	BUG_ON((end_pfn) & (pageblock_nr_pages - 1));
@@ -126,12 +128,20 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
 	     pfn < end_pfn;
 	     pfn += pageblock_nr_pages) {
 		page = __first_valid_page(pfn, pageblock_nr_pages);
-		if (page &&
-		    set_migratetype_isolate(page, skip_hwpoisoned_pages)) {
-			undo_pfn = pfn;
-			goto undo;
+		if (page) {
+			if (!zone)
+				zone = page_zone(page);
+			if (set_migratetype_isolate(page,
+						    skip_hwpoisoned_pages)) {
+				undo_pfn = pfn;
+				goto undo;
+			}
 		}
 	}
+
+	spin_lock_irqsave(&zone->lock, flags);
+	zone->memory_isolation_active = true;
+	spin_unlock_irqrestore(&zone->lock, flags);
 	return 0;
 undo:
 	for (pfn = start_pfn;
@@ -150,6 +160,9 @@ int undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
 {
 	unsigned long pfn;
 	struct page *page;
+	struct zone *zone = NULL;
+	unsigned long flags;
+
 	BUG_ON((start_pfn) & (pageblock_nr_pages - 1));
 	BUG_ON((end_pfn) & (pageblock_nr_pages - 1));
 	for (pfn = start_pfn;
@@ -159,7 +172,13 @@ int undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
 		if (!page || get_pageblock_migratetype(page) != MIGRATE_ISOLATE)
 			continue;
 		unset_migratetype_isolate(page, migratetype);
+		if (!zone)
+			zone = page_zone(page);
 	}
+
+	spin_lock_irqsave(&zone->lock, flags);
+	zone->memory_isolation_active = true;
+	spin_unlock_irqrestore(&zone->lock, flags);
 	return 0;
 }
 /*
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#5 lore raw

PATCH 5/22 oom: Use number of online nodes when deciding whether to suppress messages

In reply to Mel Gorman

Commit 29423e77 (oom: suppress show_mem() for many nodes in irq context on page alloc failure) was meant to suppress printing excessive amounts of information in IRQ context on large machines. However, it uses a kernel config variable which the maximum supported number of nodes, not the number of online nodes to make the decision. Effectively, on some distribution configurations the message will be suppressed even on small machines. This patch uses nr_online_nodes to decide whether to suppress messges.

---
 mm/page_alloc.c | 25 +++++++++----------------
 1 file changed, 9 insertions(+), 16 deletions(-)
mm/page_alloc.c +9 -16
index f170260..a66a6fa 100644
@@ -1978,20 +1978,6 @@ this_zone_full:
 	return page;
 }
 
-/*
- * Large machines with many possible nodes should not always dump per-node
- * meminfo in irq context.
- */
-static inline bool should_suppress_show_mem(void)
-{
-	bool ret = false;
-
-#if NODES_SHIFT > 8
-	ret = in_interrupt();
-#endif
-	return ret;
-}
-
 static DEFINE_RATELIMIT_STATE(nopage_rs,
 		DEFAULT_RATELIMIT_INTERVAL,
 		DEFAULT_RATELIMIT_BURST);
@@ -2034,8 +2020,15 @@ void warn_alloc_failed(gfp_t gfp_mask, int order, const char *fmt, ...)
 		current->comm, order, gfp_mask);
 
 	dump_stack();
-	if (!should_suppress_show_mem())
-		show_mem(filter);
+
+	/*
+	 * Large machines with many possible nodes should not always dump
+	 * per-node meminfo in irq context.
+	 */
+	if (in_interrupt() && nr_online_nodes > (1 << 8))
+		return;
+
+	show_mem(filter);
 }
 
 static inline int
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#6 lore raw

PATCH 6/22 mm: page allocator: Convert hot/cold parameter and immediate callers to bool

In reply to Mel Gorman

cold is a bool, make it one for consistency later.

---
 arch/sparc/mm/init_64.c  | 4 ++--
 arch/sparc/mm/tsb.c      | 2 +-
 arch/tile/mm/homecache.c | 2 +-
 fs/fuse/dev.c            | 2 +-
 include/linux/gfp.h      | 4 ++--
 include/linux/pagemap.h  | 2 +-
 include/linux/swap.h     | 2 +-
 mm/page_alloc.c          | 6 +++---
 mm/swap.c                | 4 ++--
 mm/swap_state.c          | 2 +-
 mm/vmscan.c              | 6 +++---
 11 files changed, 18 insertions(+), 18 deletions(-)
arch/sparc/mm/init_64.c +2 -2
index 1588d33..d2e50b9 100644
@@ -2562,7 +2562,7 @@ void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
 {
 	struct page *page = virt_to_page(pte);
 	if (put_page_testzero(page))
-		free_hot_cold_page(page, 0);
+		free_hot_cold_page(page, false);
 }
 
 static void __pte_free(pgtable_t pte)
@@ -2570,7 +2570,7 @@ static void __pte_free(pgtable_t pte)
 	struct page *page = virt_to_page(pte);
 	if (put_page_testzero(page)) {
 		pgtable_page_dtor(page);
-		free_hot_cold_page(page, 0);
+		free_hot_cold_page(page, false);
 	}
 }
 
arch/sparc/mm/tsb.c +1 -1
index 2cc3bce..b16adcd 100644
@@ -520,7 +520,7 @@ void destroy_context(struct mm_struct *mm)
 	page = mm->context.pgtable_page;
 	if (page && put_page_testzero(page)) {
 		pgtable_page_dtor(page);
-		free_hot_cold_page(page, 0);
+		free_hot_cold_page(page, false);
 	}
 
 	spin_lock_irqsave(&ctx_alloc_lock, flags);
arch/tile/mm/homecache.c +1 -1
index 1ae9119..eacb91b 100644
@@ -438,7 +438,7 @@ void __homecache_free_pages(struct page *page, unsigned int order)
 	if (put_page_testzero(page)) {
 		homecache_change_page_home(page, order, initial_page_home());
 		if (order == 0) {
-			free_hot_cold_page(page, 0);
+			free_hot_cold_page(page, false);
 		} else {
 			init_page_count(page);
 			__free_pages(page, order);
fs/fuse/dev.c +1 -1
index 11dfa0c..8532ea3 100644
@@ -1580,7 +1580,7 @@ out_finish:
 
 static void fuse_retrieve_end(struct fuse_conn *fc, struct fuse_req *req)
 {
-	release_pages(req->pages, req->num_pages, 0);
+	release_pages(req->pages, req->num_pages, false);
 }
 
 static int fuse_retrieve(struct fuse_conn *fc, struct inode *inode,
include/linux/gfp.h +2 -2
index 0f615eb..66e45e7 100644
@@ -364,8 +364,8 @@ void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask);
 
 extern void __free_pages(struct page *page, unsigned int order);
 extern void free_pages(unsigned long addr, unsigned int order);
-extern void free_hot_cold_page(struct page *page, int cold);
-extern void free_hot_cold_page_list(struct list_head *list, int cold);
+extern void free_hot_cold_page(struct page *page, bool cold);
+extern void free_hot_cold_page_list(struct list_head *list, bool cold);
 
 extern void __free_memcg_kmem_pages(struct page *page, unsigned int order);
 extern void free_memcg_kmem_pages(unsigned long addr, unsigned int order);
include/linux/pagemap.h +1 -1
index 0e38e13..5d7fe39 100644
@@ -99,7 +99,7 @@ static inline void mapping_set_gfp_mask(struct address_space *m, gfp_t mask)
 
 #define page_cache_get(page)		get_page(page)
 #define page_cache_release(page)	put_page(page)
-void release_pages(struct page **pages, int nr, int cold);
+void release_pages(struct page **pages, int nr, bool cold);
 
 /*
  * speculatively take a reference to a page.
include/linux/swap.h +1 -1
index 2818a12..ef644ab 100644
@@ -414,7 +414,7 @@ mem_cgroup_uncharge_swapcache(struct page *page, swp_entry_t ent, bool swapout)
 #define free_page_and_swap_cache(page) \
 	page_cache_release(page)
 #define free_pages_and_swap_cache(pages, nr) \
-	release_pages((pages), (nr), 0);
+	release_pages((pages), (nr), false);
 
 static inline void show_swap_cache_info(void)
 {
mm/page_alloc.c +3 -3
index a66a6fa..f74e16f 100644
@@ -1314,7 +1314,7 @@ void mark_free_pages(struct zone *zone)
  * Free a 0-order page
  * cold == 1 ? free a cold page : free a hot page
  */
-void free_hot_cold_page(struct page *page, int cold)
+void free_hot_cold_page(struct page *page, bool cold)
 {
 	struct zone *zone = page_zone(page);
 	struct per_cpu_pages *pcp;
@@ -1362,7 +1362,7 @@ void free_hot_cold_page(struct page *page, int cold)
 /*
  * Free a list of 0-order pages
  */
-void free_hot_cold_page_list(struct list_head *list, int cold)
+void free_hot_cold_page_list(struct list_head *list, bool cold)
 {
 	struct page *page, *next;
 
@@ -2680,7 +2680,7 @@ void __free_pages(struct page *page, unsigned int order)
 {
 	if (put_page_testzero(page)) {
 		if (order == 0)
-			free_hot_cold_page(page, 0);
+			free_hot_cold_page(page, false);
 		else
 			__free_pages_ok(page, order);
 	}
mm/swap.c +2 -2
index 8a529a0..36c28e5 100644
@@ -63,7 +63,7 @@ static void __page_cache_release(struct page *page)
 static void __put_single_page(struct page *page)
 {
 	__page_cache_release(page);
-	free_hot_cold_page(page, 0);
+	free_hot_cold_page(page, false);
 }
 
 static void __put_compound_page(struct page *page)
@@ -667,7 +667,7 @@ int lru_add_drain_all(void)
  * grabbed the page via the LRU.  If it did, give up: shrink_inactive_list()
  * will free it.
  */
-void release_pages(struct page **pages, int nr, int cold)
+void release_pages(struct page **pages, int nr, bool cold)
 {
 	int i;
 	LIST_HEAD(pages_to_free);
mm/swap_state.c +1 -1
index 7efcf15..bf0da4d 100644
@@ -268,7 +268,7 @@ void free_pages_and_swap_cache(struct page **pages, int nr)
 
 		for (i = 0; i < todo; i++)
 			free_swap_cache(pagep[i]);
-		release_pages(pagep, todo, 0);
+		release_pages(pagep, todo, false);
 		pagep += todo;
 		nr -= todo;
 	}
mm/vmscan.c +3 -3
index 669fba3..6a56766 100644
@@ -954,7 +954,7 @@ keep:
 	if (nr_dirty && nr_dirty == nr_congested && global_reclaim(sc))
 		zone_set_flag(zone, ZONE_CONGESTED);
 
-	free_hot_cold_page_list(&free_pages, 1);
+	free_hot_cold_page_list(&free_pages, true);
 
 	list_splice(&ret_pages, page_list);
 	count_vm_events(PGACTIVATE, pgactivate);
@@ -1343,7 +1343,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
 
 	spin_unlock_irq(&zone->lru_lock);
 
-	free_hot_cold_page_list(&page_list, 1);
+	free_hot_cold_page_list(&page_list, true);
 
 	/*
 	 * If reclaim is isolating dirty pages under writeback, it implies
@@ -1534,7 +1534,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 	__mod_zone_page_state(zone, NR_ISOLATED_ANON + file, -nr_taken);
 	spin_unlock_irq(&zone->lru_lock);
 
-	free_hot_cold_page_list(&l_hold, 1);
+	free_hot_cold_page_list(&l_hold, true);
 }
 
 #ifdef CONFIG_SWAP
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#7 lore raw

PATCH 7/22 mm: page allocator: Do not lookup the pageblock migratetype during allocation

In reply to Mel Gorman

The pageblock migrate type is checked during allocation in case it's CMA but using get_freepage_migratetype() should be sufficient as the magazines would have been drained prior to a CMA allocation attempt.

---
 mm/page_alloc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
index f74e16f..8867937 100644
@@ -1523,7 +1523,7 @@ again:
 		if (!page)
 			goto failed;
 		__mod_zone_freepage_state(zone, -(1 << order),
-					  get_pageblock_migratetype(page));
+					  get_freepage_migratetype(page));
 	}
 
 	__count_zone_vm_events(PGALLOC, zone, 1 << order);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#8 lore raw

PATCH 8/22 mm: page allocator: Remove the per-cpu page allocator

In reply to Mel Gorman

This patch removes the per-cpu page allocator in preparation for placing a different order-0 allocator in front of the buddy allocator. This is to simplify review.

---
 include/linux/gfp.h         |   3 -
 include/linux/mm.h          |   3 -
 include/linux/mmzone.h      |  12 -
 include/trace/events/kmem.h |  11 -
 init/main.c                 |   1 -
 kernel/power/snapshot.c     |   2 -
 kernel/sysctl.c             |  10 -
 mm/memory-failure.c         |   1 -
 mm/memory_hotplug.c         |  11 +-
 mm/page_alloc.c             | 581 ++------------------------------------------
 mm/page_isolation.c         |   2 -
 mm/vmstat.c                 |  39 +--
 12 files changed, 29 insertions(+), 647 deletions(-)
include/linux/gfp.h +0 -3
index 66e45e7..edf3184 100644
@@ -374,9 +374,6 @@ extern void free_memcg_kmem_pages(unsigned long addr, unsigned int order);
 #define free_page(addr) free_pages((addr), 0)
 
 void page_alloc_init(void);
-void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp);
-void drain_all_pages(void);
-void drain_local_pages(void *dummy);
 
 /*
  * gfp_allowed_mask is set to GFP_BOOT_MASK during early boot to restrict what
include/linux/mm.h +0 -3
index e2091b8..04cb6b4 100644
@@ -1367,9 +1367,6 @@ void warn_alloc_failed(gfp_t gfp_mask, int order, const char *fmt, ...);
 
 extern void setup_per_cpu_pageset(void);
 
-extern void zone_pcp_update(struct zone *zone);
-extern void zone_pcp_reset(struct zone *zone);
-
 /* page_alloc.c */
 extern int min_free_kbytes;
 
include/linux/mmzone.h +0 -12
index 57f03b3..3ee9b27 100644
@@ -235,17 +235,7 @@ enum zone_watermarks {
 #define low_wmark_pages(z) (z->watermark[WMARK_LOW])
 #define high_wmark_pages(z) (z->watermark[WMARK_HIGH])
 
-struct per_cpu_pages {
-	int count;		/* number of pages in the list */
-	int high;		/* high watermark, emptying needed */
-	int batch;		/* chunk size for buddy add/remove */
-
-	/* Lists of pages, one per migrate type stored on the pcp-lists */
-	struct list_head lists[MIGRATE_PCPTYPES];
-};
-
 struct per_cpu_pageset {
-	struct per_cpu_pages pcp;
 #ifdef CONFIG_NUMA
 	s8 expire;
 #endif
@@ -900,8 +890,6 @@ int min_free_kbytes_sysctl_handler(struct ctl_table *, int,
 extern int sysctl_lowmem_reserve_ratio[MAX_NR_ZONES-1];
 int lowmem_reserve_ratio_sysctl_handler(struct ctl_table *, int,
 					void __user *, size_t *, loff_t *);
-int percpu_pagelist_fraction_sysctl_handler(struct ctl_table *, int,
-					void __user *, size_t *, loff_t *);
 int sysctl_min_unmapped_ratio_sysctl_handler(struct ctl_table *, int,
 			void __user *, size_t *, loff_t *);
 int sysctl_min_slab_ratio_sysctl_handler(struct ctl_table *, int,
include/trace/events/kmem.h +0 -11
index 6bc943e..0a5501a 100644
@@ -253,17 +253,6 @@ DEFINE_EVENT(mm_page, mm_page_alloc_zone_locked,
 	TP_ARGS(page, order, migratetype)
 );
 
-DEFINE_EVENT_PRINT(mm_page, mm_page_pcpu_drain,
-
-	TP_PROTO(struct page *page, unsigned int order, int migratetype),
-
-	TP_ARGS(page, order, migratetype),
-
-	TP_printk("page=%p pfn=%lu order=%d migratetype=%d",
-		__entry->page, page_to_pfn(__entry->page),
-		__entry->order, __entry->migratetype)
-);
-
 TRACE_EVENT(mm_page_alloc_extfrag,
 
 	TP_PROTO(struct page *page,
init/main.c +0 -1
index 63534a1..8d0bbce 100644
@@ -597,7 +597,6 @@ asmlinkage void __init start_kernel(void)
 	page_cgroup_init();
 	debug_objects_mem_init();
 	kmemleak_init();
-	setup_per_cpu_pageset();
 	numa_policy_init();
 	if (late_time_init)
 		late_time_init();
kernel/power/snapshot.c +0 -2
index 0de2857..08b2766 100644
@@ -1582,7 +1582,6 @@ asmlinkage int swsusp_save(void)
 
 	printk(KERN_INFO "PM: Creating hibernation image:\n");
 
-	drain_local_pages(NULL);
 	nr_pages = count_data_pages();
 	nr_highmem = count_highmem_pages();
 	printk(KERN_INFO "PM: Need to copy %u pages\n", nr_pages + nr_highmem);
@@ -1600,7 +1599,6 @@ asmlinkage int swsusp_save(void)
 	/* During allocating of suspend pagedir, new cold pages may appear.
 	 * Kill them.
 	 */
-	drain_local_pages(NULL);
 	copy_data_pages(&copy_bm, &orig_bm);
 
 	/*
kernel/sysctl.c +0 -10
index afc1dc6..ce38025 100644
@@ -107,7 +107,6 @@ extern unsigned int core_pipe_limit;
 extern int pid_max;
 extern int pid_max_min, pid_max_max;
 extern int sysctl_drop_caches;
-extern int percpu_pagelist_fraction;
 extern int compat_log;
 extern int latencytop_enabled;
 extern int sysctl_nr_open_min, sysctl_nr_open_max;
@@ -140,7 +139,6 @@ static unsigned long dirty_bytes_min = 2 * PAGE_SIZE;
 /* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */
 static int maxolduid = 65535;
 static int minolduid;
-static int min_percpu_pagelist_fract = 8;
 
 static int ngroups_max = NGROUPS_MAX;
 static const int cap_last_cap = CAP_LAST_CAP;
@@ -1266,14 +1264,6 @@ static struct ctl_table vm_table[] = {
 		.proc_handler	= min_free_kbytes_sysctl_handler,
 		.extra1		= &zero,
 	},
-	{
-		.procname	= "percpu_pagelist_fraction",
-		.data		= &percpu_pagelist_fraction,
-		.maxlen		= sizeof(percpu_pagelist_fraction),
-		.mode		= 0644,
-		.proc_handler	= percpu_pagelist_fraction_sysctl_handler,
-		.extra1		= &min_percpu_pagelist_fract,
-	},
 #ifdef CONFIG_MMU
 	{
 		.procname	= "max_map_count",
mm/memory-failure.c +0 -1
index df0694c..3175ffd 100644
@@ -237,7 +237,6 @@ void shake_page(struct page *p, int access)
 		lru_add_drain_all();
 		if (PageLRU(p))
 			return;
-		drain_all_pages();
 		if (PageLRU(p) || is_free_buddy_page(p))
 			return;
 	}
mm/memory_hotplug.c +1 -10
index ee37657..63f473c 100644
@@ -970,8 +970,6 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
 	ret = walk_system_ram_range(pfn, nr_pages, &onlined_pages,
 		online_pages_range);
 	if (ret) {
-		if (need_zonelists_rebuild)
-			zone_pcp_reset(zone);
 		mutex_unlock(&zonelists_mutex);
 		printk(KERN_DEBUG "online_pages [mem %#010llx-%#010llx] failed\n",
 		       (unsigned long long) pfn << PAGE_SHIFT,
@@ -989,8 +987,6 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
 		node_states_set_node(zone_to_nid(zone), &arg);
 		if (need_zonelists_rebuild)
 			build_all_zonelists(NULL, NULL);
-		else
-			zone_pcp_update(zone);
 	}
 
 	mutex_unlock(&zonelists_mutex);
@@ -1530,7 +1526,6 @@ repeat:
 	if (drain) {
 		lru_add_drain_all();
 		cond_resched();
-		drain_all_pages();
 	}
 
 	pfn = scan_lru_pages(start_pfn, end_pfn);
@@ -1551,8 +1546,6 @@ repeat:
 	/* drain all zone's lru pagevec, this is asynchronous... */
 	lru_add_drain_all();
 	yield();
-	/* drain pcp pages, this is synchronous. */
-	drain_all_pages();
 	/* check again */
 	offlined_pages = check_pages_isolated(start_pfn, end_pfn);
 	if (offlined_pages < 0) {
@@ -1574,12 +1567,10 @@ repeat:
 	init_per_zone_wmark_min();
 
 	if (!populated_zone(zone)) {
-		zone_pcp_reset(zone);
 		mutex_lock(&zonelists_mutex);
 		build_all_zonelists(NULL, NULL);
 		mutex_unlock(&zonelists_mutex);
-	} else
-		zone_pcp_update(zone);
+	}
 
 	node_states_clear_node(node, &arg);
 	if (arg.status_change_nid >= 0)
mm/page_alloc.c +26 -555

715 lines not on this page · raw message

mm/page_isolation.c +0 -2
index 9f0c068..af79199 100644
@@ -65,8 +65,6 @@ out:
 	}
 
 	spin_unlock_irqrestore(&zone->lock, flags);
-	if (!ret)
-		drain_all_pages();
 	return ret;
 }
 
mm/vmstat.c +2 -37
index e1d8ed1..45e699c 100644
@@ -462,32 +462,6 @@ void refresh_cpu_vm_stats(int cpu)
 #endif
 			}
 		cond_resched();
-#ifdef CONFIG_NUMA
-		/*
-		 * Deal with draining the remote pageset of this
-		 * processor
-		 *
-		 * Check if there are pages remaining in this pageset
-		 * if not then there is nothing to expire.
-		 */
-		if (!p->expire || !p->pcp.count)
-			continue;
-
-		/*
-		 * We never drain zones local to this processor.
-		 */
-		if (zone_to_nid(zone) == numa_node_id()) {
-			p->expire = 0;
-			continue;
-		}
-
-		p->expire--;
-		if (p->expire)
-			continue;
-
-		if (p->pcp.count)
-			drain_zone_pages(zone, &p->pcp);
-#endif
 	}
 
 	for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++)
@@ -1028,24 +1002,15 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,
 	seq_printf(m,
 		   ")"
 		   "\n  pagesets");
+#ifdef CONFIG_SMP
 	for_each_online_cpu(i) {
 		struct per_cpu_pageset *pageset;
 
 		pageset = per_cpu_ptr(zone->pageset, i);
-		seq_printf(m,
-			   "\n    cpu: %i"
-			   "\n              count: %i"
-			   "\n              high:  %i"
-			   "\n              batch: %i",
-			   i,
-			   pageset->pcp.count,
-			   pageset->pcp.high,
-			   pageset->pcp.batch);
-#ifdef CONFIG_SMP
 		seq_printf(m, "\n  vm stats threshold: %d",
 				pageset->stat_threshold);
-#endif
 	}
+#endif
 	seq_printf(m,
 		   "\n  all_unreclaimable: %u"
 		   "\n  start_pfn:         %lu"
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#9 lore raw

PATCH 9/22 mm: page allocator: Allocate/free order-0 pages from a per-zone magazine

In reply to Mel Gorman

This patch introduces a simple magazine of order-0 pages that sits between the buddy allocator and the caller. Simplistically there is a "struct free_area zone->noirq_magazine" in each zone protected by an IRQ-unsafe spinlock zone->magazine_lock. It replaces the per-cpu allocator that used to exist but has several properties that may be better depending on the workload.

1. IRQs do not have to be disabled to access the lists reducing IRQs disabled times.

2. As the list is protected by a spinlock, it is not necessary to send IPI to drain the list. As the lists are accessible by multiple CPUs, it is easier to tune.

3. The magazine_lock is potentially hot but it can be split to have one lock per CPU socket to reduce contention. Draining the lists in this case would acquire multiple locks be acquired.

---
 include/linux/mmzone.h |   7 +++
 mm/page_alloc.c        | 114 +++++++++++++++++++++++++++++++++++++++++--------
 mm/vmstat.c            |  14 ++++--
 3 files changed, 114 insertions(+), 21 deletions(-)
include/linux/mmzone.h +7 -0
index 3ee9b27..a6f84f1 100644
@@ -361,6 +361,13 @@ struct zone {
 #endif
 	struct free_area	free_area[MAX_ORDER];
 
+	/*
+	 * Keep some order-0 pages on a separate free list
+	 * protected by an irq-unsafe lock
+	 */
+	spinlock_t		magazine_lock;
+	struct free_area	noirq_magazine;
+
 #ifndef CONFIG_SPARSEMEM
 	/*
 	 * Flags for a pageblock_nr_pages block. See pageblock-flags.h.
mm/page_alloc.c +97 -17

190 lines not on this page · raw message

mm/vmstat.c +10 -4
index 45e699c..7274ca5 100644
@@ -1001,14 +1001,20 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,
 		seq_printf(m, ", %lu", zone->lowmem_reserve[i]);
 	seq_printf(m,
 		   ")"
-		   "\n  pagesets");
+		   "\n  noirq magazine");
+	seq_printf(m,
+		"\n    cpu: %i"
+		"\n              count: %lu",
+		i,
+		zone->noirq_magazine.nr_free);
+
 #ifdef CONFIG_SMP
 	for_each_online_cpu(i) {
 		struct per_cpu_pageset *pageset;
 
-		pageset = per_cpu_ptr(zone->pageset, i);
-		seq_printf(m, "\n  vm stats threshold: %d",
-				pageset->stat_threshold);
+ 		pageset = per_cpu_ptr(zone->pageset, i);
+		seq_printf(m, "\n  pagesets\n  vm stats threshold: %d",
+ 				pageset->stat_threshold);
 	}
 #endif
 	seq_printf(m,
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#10 lore raw

Re: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine

Quoted 2 lines · Mel Gorman

1. IRQs do not have to be disabled to access the lists reducing IRQs disabled times.

The per cpu structure access also would not need to disable irq if the fast path would be using this_cpu ops.

Quoted 3 lines

2. As the list is protected by a spinlock, it is not necessary to send IPI to drain the list. As the lists are accessible by multiple CPUs, it is easier to tune.

The lists are a problem since traversing list heads creates a lot of pressure on the processor and TLB caches. Could we either move to an array of pointers to page structs (like in SLAB) or to a linked list that is constrained within physical boundaries like within a PMD? (comparable to the SLUB approach)?

Quoted 3 lines
Quoted 1 line

3. The magazine_lock is potentially hot but it can be split to have

one lock per CPU socket to reduce contention. Draining the lists in this case would acquire multiple locks be acquired.

IMHO the use of per cpu RMV operations would be lower latency than the use of spinlocks. There is no "lock" prefix overhead with those. Page allocation is a frequent operation that I would think needs to be as fast as possible.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#11 lore raw

Re: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine

Quoted 8 lines · Christoph Lameter
Quoted 2 lines · Mel Gorman

1. IRQs do not have to be disabled to access the lists reducing IRQs disabled times.

The per cpu structure access also would not need to disable irq if the fast path would be using this_cpu ops.

How does this_cpu protect against preemption due to interrupt? this_read() itself only disables preemption and it's explicitly documented that interrupt that modifies the per-cpu data will not be reliable so the use of the per-cpu lists is right out. It would require that a race-prone check be used with cmpxchg which in turn would require arrays, not lists.

Quoted 7 lines
Quoted 3 lines

2. As the list is protected by a spinlock, it is not necessary to send IPI to drain the list. As the lists are accessible by multiple CPUs, it is easier to tune.

The lists are a problem since traversing list heads creates a lot of pressure on the processor and TLB caches. Could we either move to an array of pointers to page structs (like in SLAB)

They would have large memory requirements. The magazine data structure in this series fits in a cache line. An array of 128 struct page pointers would require 1K on 64-bit and if that thing is per possible CPU and per zone then it could get really excessive.

Quoted 4 lines

or to a linked list that is
constrained within physical boundaries like within a PMD? (comparable to the SLUB approach)?

I don't see how as the page allocator does not control the physical location of any pages freed to it and it's the struct pages it is linking together. On some systems at least with 1G pages, the struct pages will be backed by memory mapped with 1G entries so the TLB pressure should be reduced but the cache pressure from struct page modifications is certainly a problem.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

3. The magazine_lock is potentially hot but it can be split to have

one lock per CPU socket to reduce contention. Draining the lists in this case would acquire multiple locks be acquired.

IMHO the use of per cpu RMV operations would be lower latency than the use of spinlocks. There is no "lock" prefix overhead with those. Page allocation is a frequent operation that I would think needs to be as fast as possible.

The memory requirements may be large because those per-cpu areas sized are allocated depending on num_possible_cpus()s. Correct? Regardless of their size, it would still be required to deal with cpu hot-plug to avoid memory leaks and draining them would still require global IPIs so the overall code complexity would be similar to what exists today. Ultimately all that changes is that we use an array+cmpxchg instead of a list which will shave a small amount of latency but it will still be regularly falling back to the buddy lists and contend on the zone->lock due the limited size of the per-cpu magazines and hiding the advantage of using cmpxchg in the noise.

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#12 lore raw

Re: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine

Quoted 10 lines · Mel Gorman
Quoted 4 lines

The per cpu structure access also would not need to disable irq if the fast path would be using this_cpu ops.

How does this_cpu protect against preemption due to interrupt? this_read() itself only disables preemption and it's explicitly documented that interrupt that modifies the per-cpu data will not be reliable so the use of the per-cpu lists is right out. It would require that a race-prone check be used with cmpxchg which in turn would require arrays, not lists.

this_cpu uses single atomic instructions that cannot be interrupted. The relocation occurs through a segment prefix and therefore is not subject to preemption. The interrupts and rescheduling can occur between this_cpu ops and there races would have to be dealt with. True using cmpxchg (w/o lock semantics) is not that easy. But that is the fastest solution that I know of.

Quoted 5 lines

I don't see how as the page allocator does not control the physical location of any pages freed to it and it's the struct pages it is linking together. On some systems at least with 1G pages, the struct pages will be backed by memory mapped with 1G entries so the TLB pressure should be reduced but the cache pressure from struct page modifications is certainly a problem.

I would be useful if the allocator would hand out pages from the same physical area first. This would reduce fragmentation as well and since it is likely that numerous pages are allocated for some purpose (given that that the page sizes of 4k are rather tiny compared to the data needs these day) would reduce TLB pressure.

Quoted 11 lines
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

3. The magazine_lock is potentially hot but it can be split to have

one lock per CPU socket to reduce contention. Draining the lists in this case would acquire multiple locks be acquired.

IMHO the use of per cpu RMV operations would be lower latency than the use of spinlocks. There is no "lock" prefix overhead with those. Page allocation is a frequent operation that I would think needs to be as fast as possible.

The memory requirements may be large because those per-cpu areas sized are allocated depending on num_possible_cpus()s. Correct? Regardless of their

Yes. But we have lots of memory in machines these days. Why would that be an issue?

Quoted 7 lines

size, it would still be required to deal with cpu hot-plug to avoid memory leaks and draining them would still require global IPIs so the overall code complexity would be similar to what exists today. Ultimately all that changes is that we use an array+cmpxchg instead of a list which will shave a small amount of latency but it will still be regularly falling back to the buddy lists and contend on the zone->lock due the limited size of the per-cpu magazines and hiding the advantage of using cmpxchg in the noise.

The latency would be an order of magnitude less than the approach that you propose here. The magazine approach and the lockless approach both will require slowpaths that replenish the set of pages to be served next.

The problem with the page allocator is that it can serve various types of pages. If one wants to setup caches for all of those then these caches are replicated for each processor or whatever higher unit we decide to use. I think one of the first moves need to be to identify which types of pages are actually useful to serve in a fast way. Higher order pages are already out but what about the different zone types, migration types etc?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#13 lore raw

Re: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine

Quoted 13 lines · Christoph Lameter
Quoted 10 lines · Mel Gorman
Quoted 4 lines

The per cpu structure access also would not need to disable irq if the fast path would be using this_cpu ops.

How does this_cpu protect against preemption due to interrupt? this_read() itself only disables preemption and it's explicitly documented that interrupt that modifies the per-cpu data will not be reliable so the use of the per-cpu lists is right out. It would require that a race-prone check be used with cmpxchg which in turn would require arrays, not lists.

Quoted 7 lines

this_cpu uses single atomic instructions that cannot be interrupted. The relocation occurs through a segment prefix and therefore is not subject to preemption. The interrupts and rescheduling can occur between this_cpu ops and there races would have to be dealt with. True using cmpxchg (w/o lock semantics) is not that easy. But that is the fastest solution that I know of.

And it requires moving to an array so there are going to be strong limits on the size of the per-cpu queue.

Quoted 6 lines
Quoted 5 lines

I don't see how as the page allocator does not control the physical location of any pages freed to it and it's the struct pages it is linking together. On some systems at least with 1G pages, the struct pages will be backed by memory mapped with 1G entries so the TLB pressure should be reduced but the cache pressure from struct page modifications is certainly a problem.

Quoted 6 lines

I would be useful if the allocator would hand out pages from the same physical area first. This would reduce fragmentation as well and since it is likely that numerous pages are allocated for some purpose (given that that the page sizes of 4k are rather tiny compared to the data needs these day) would reduce TLB pressure.

It already does this via the buddy allocator and the treatment of migratetypes.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

3. The magazine_lock is potentially hot but it can be split to have

one lock per CPU socket to reduce contention. Draining the lists in this case would acquire multiple locks be acquired.

IMHO the use of per cpu RMV operations would be lower latency than the use of spinlocks. There is no "lock" prefix overhead with those. Page allocation is a frequent operation that I would think needs to be as fast as possible.

Quoted 6 lines
Quoted 2 lines

The memory requirements may be large because those per-cpu areas sized are allocated depending on num_possible_cpus()s. Correct? Regardless of their

Yes. But we have lots of memory in machines these days. Why would that be an issue?

Because the embedded people will have a fit if the page allocator needs an additional 1K+ of memory just to turn on.

Quoted 8 lines
Quoted 7 lines

size, it would still be required to deal with cpu hot-plug to avoid memory leaks and draining them would still require global IPIs so the overall code complexity would be similar to what exists today. Ultimately all that changes is that we use an array+cmpxchg instead of a list which will shave a small amount of latency but it will still be regularly falling back to the buddy lists and contend on the zone->lock due the limited size of the per-cpu magazines and hiding the advantage of using cmpxchg in the noise.

Quoted 4 lines

The latency would be an order of magnitude less than the approach that you propose here. The magazine approach and the lockless approach both will require slowpaths that replenish the set of pages to be served next.

With this approach the lock can be made more fine or coarse based on the number of CPUs, the queues can be made arbitrarily large and if necessary, per-process magazines for heavily contended workloads could be added. A fixed-size array like you propose would be only marginally better than what is implemented today as far as I can see because it still smacks into the irq-safe zone->lock and pages can be pinned in inaccessible per-cpu queues unless a global IPI is sent.

Quoted 7 lines

The problem with the page allocator is that it can serve various types of pages. If one wants to setup caches for all of those then these caches are replicated for each processor or whatever higher unit we decide to use. I think one of the first moves need to be to identify which types of pages are actually useful to serve in a fast way. Higher order pages are already out but what about the different zone types, migration types etc?

What tpyes of pages are useful to serve in a fast way is workload dependenat and besides the per-cpu allocator as it exists today already has separate queues for migration types.

I strongly suspect that your proposal would end up performing roughly the same as what exists today except that it'll be more complex because it'll have to deal with the race-prone array accesses.

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#14 lore raw

Re: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine

Quoted 9 lines · Mel Gorman
Quoted 6 lines

I would be useful if the allocator would hand out pages from the same physical area first. This would reduce fragmentation as well and since it is likely that numerous pages are allocated for some purpose (given that that the page sizes of 4k are rather tiny compared to the data needs these day) would reduce TLB pressure.

It already does this via the buddy allocator and the treatment of migratetypes.

Well it only does if it breaks larger sized pages from the buddy allocator. If large page lists aggregate in the per cpu lists then we have a LIFO order.

Quoted 4 lines
Quoted 2 lines

Yes. But we have lots of memory in machines these days. Why would that be an issue?

Because the embedded people will have a fit if the page allocator needs an additional 1K+ of memory just to turn on.

Why enlarge the existing per cpu areas? The size could
be restricted if we reduce the types of pages supported and/or if we do not use double linked lists but single linked lists within say a PMD area.

Quoted 3 lines

With this approach the lock can be made more fine or coarse based on the number of CPUs, the queues can be made arbitrarily large and if necessary, per-process magazines for heavily contended workloads could be added.

Arbitrarily large queues cause references to pointers all over memory. No good.

Quoted 4 lines

A fixed-size array like you propose would be only marginally better than what is implemented today as far as I can see because it still smacks into the irq-safe zone->lock and pages can be pinned in inaccessible per-cpu queues unless a global IPI is sent.

We do not send global IPIs but IPIs only to processors that have something cached.

The fixed size array or constrained single linked list would be better since it caches better and it is possible to avoid spin lock operations.

Quoted 8 lines
Quoted 7 lines

The problem with the page allocator is that it can serve various types of pages. If one wants to setup caches for all of those then these caches are replicated for each processor or whatever higher unit we decide to use. I think one of the first moves need to be to identify which types of pages are actually useful to serve in a fast way. Higher order pages are already out but what about the different zone types, migration types etc?

Quoted 7 lines

What tpyes of pages are useful to serve in a fast way is workload dependenat and besides the per-cpu allocator as it exists today already has separate queues for migration types.

I strongly suspect that your proposal would end up performing roughly the same as what exists today except that it'll be more complex because it'll have to deal with the race-prone array accesses.

The problems of the current scheme are the proliferation of page types, the serving of pages in a random mix from all over memory, the heavy high latency processing in the "fast" paths (these paths seem to accumulate more and more procesing in each kernel version) and the disabling of interrupts (which may be the least latency causing issue).

A solution without using locks cannot simply be a modification of the existing scheme that you envison. The amount of processing in the fastpaths must be significantly reduced and the data layout needs to be more cache friendly. Only with these changes will make the use of fast cpu local instructions sense.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#15 lore raw

PATCH 10/22 mm: page allocator: Allocate and free pages from magazine in batches

In reply to Mel Gorman

When the magazine is empty or full the zone lock is taken and a single page is operated on. This makes the zone lock hotter than it needs to be so batch allocations and frees from the zone. A larger number of pages are taken when refilling the magazine to reduce the contention on the zone->lock for IRQ-disabled callers. It's more likely that a workload will notice contention on allocations than contentions on free although of course this is workload dependant

---
 mm/page_alloc.c | 172 +++++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 127 insertions(+), 45 deletions(-)
mm/page_alloc.c +127 -45
index 9ed05a5..9426174 100644
@@ -601,8 +601,6 @@ static inline void __free_one_page(struct page *page,
 	list_add(&page->lru, &zone->free_area[order].free_list[migratetype]);
 out:
 	zone->free_area[order].nr_free++;
-	if (unlikely(!is_migrate_isolate(migratetype)))
-		__mod_zone_freepage_state(zone, 1 << order, migratetype);
 }
 
 static inline int free_pages_check(struct page *page)
@@ -634,6 +632,8 @@ static void free_one_page(struct zone *zone, struct page *page,
 	__count_vm_events(PGFREE, 1 << order);
 
 	__free_one_page(page, zone, order, migratetype);
+	if (unlikely(!is_migrate_isolate(migratetype)))
+		__mod_zone_freepage_state(zone, 1 << order, migratetype);
 	spin_unlock_irqrestore(&zone->lock, flags);
 }
 
@@ -1093,6 +1093,87 @@ void mark_free_pages(struct zone *zone)
 #endif /* CONFIG_PM */
 
 #define MAGAZINE_LIMIT (1024)
+#define MAGAZINE_ALLOC_BATCH (384)
+#define MAGAZINE_FREE_BATCH (64)
+
+static
+struct page *__rmqueue_magazine(struct zone *zone, int migratetype)
+{
+	struct page *page;
+	struct free_area *area = &(zone->noirq_magazine);
+
+	if (list_empty(&area->free_list[migratetype]))
+		return NULL;
+
+	/* Page is available in the magazine, allocate it */
+	page = list_entry(area->free_list[migratetype].next, struct page, lru);
+	list_del(&page->lru);
+	area->nr_free--;
+	set_page_private(page, 0);
+
+	return page;
+}
+
+static void magazine_drain(struct zone *zone, int migratetype)
+{
+	struct free_area *area = &(zone->noirq_magazine);
+	struct list_head *list;
+	struct page *page;
+	unsigned int batch_free = 0;
+	unsigned int to_free = MAGAZINE_FREE_BATCH;
+	unsigned int nr_freed_cma = 0;
+	unsigned long flags;
+	LIST_HEAD(free_list);
+
+	if (area->nr_free < MAGAZINE_LIMIT) {
+		spin_unlock(&zone->magazine_lock);
+		return;
+	}
+
+	/* Free batch number of pages */
+	while (to_free) {
+		/*
+		 * Removes pages from lists in a round-robin fashion. A
+		 * batch_free count is maintained that is incremented when an
+		 * empty list is encountered.  This is so more pages are freed
+		 * off fuller lists instead of spinning excessively around empty
+		 * lists
+		 */
+		do {
+			batch_free++;
+			if (++migratetype == MIGRATE_PCPTYPES)
+				migratetype = 0;
+			list = &area->free_list[migratetype];;
+		} while (list_empty(list));
+
+		/* This is the only non-empty list. Free them all. */
+		if (batch_free == MIGRATE_PCPTYPES)
+			batch_free = to_free;
+
+		do {
+			page = list_entry(list->prev, struct page, lru);
+			area->nr_free--;
+			set_page_private(page, 0);
+			list_move(&page->lru, &free_list);
+			if (is_migrate_isolate_page(zone, page))
+				nr_freed_cma++;
+		} while (--to_free && --batch_free && !list_empty(list));
+	}
+
+	/* Free the list of pages to the buddy allocator */
+	spin_unlock(&zone->magazine_lock);
+	spin_lock_irqsave(&zone->lock, flags);
+	while (!list_empty(&free_list)) {
+		page = list_entry(free_list.prev, struct page, lru);
+		list_del(&page->lru);
+		__free_one_page(page, zone, 0, get_freepage_migratetype(page));
+	}
+	__mod_zone_page_state(zone, NR_FREE_PAGES,
+				MAGAZINE_FREE_BATCH - nr_freed_cma);
+	if (nr_freed_cma)
+		__mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_freed_cma);
+	spin_unlock_irqrestore(&zone->lock, flags);
+}
 
 /*
  * Free a 0-order page
@@ -1111,8 +1192,10 @@ void free_hot_cold_page(struct page *page, bool cold)
 	set_freepage_migratetype(page, migratetype);
 
 	/* magazine_lock is not safe against IRQs */
-	if (in_interrupt() || irqs_disabled())
-		goto free_one;
+	if (in_interrupt() || irqs_disabled()) {
+		free_one_page(zone, page, 0, migratetype);
+		return;
+	}
 
 	/* Put the free page on the magazine list */
 	spin_lock(&zone->magazine_lock);
@@ -1121,32 +1204,10 @@ void free_hot_cold_page(struct page *page, bool cold)
 		list_add(&page->lru, &area->free_list[migratetype]);
 	else
 		list_add_tail(&page->lru, &area->free_list[migratetype]);
-	page = NULL;
-
-	/* If the magazine is full, remove a cold page for the buddy list */
-	if (area->nr_free > MAGAZINE_LIMIT) {
-		struct list_head *list = &area->free_list[migratetype];
-		int starttype = migratetype;
+	area->nr_free++;
 
-		while (list_empty(list)) {
-			if (++migratetype == MIGRATE_PCPTYPES)
-				migratetype = 0;
-			list = &area->free_list[migratetype];;
-		
-			WARN_ON_ONCE(starttype == migratetype);
-		}
-			
-		page = list_entry(list->prev, struct page, lru);
-		list_del(&page->lru);
-	} else {
-		area->nr_free++;
-	}
-	spin_unlock(&zone->magazine_lock);
-
-free_one:
-	/* Free a page back to the buddy lists if necessary */
-	if (page)
-		free_one_page(zone, page, 0, migratetype);
+	/* Drain the magazine if necessary, releases the magazine lock */
+	magazine_drain(zone, migratetype);
 }
 
 /*
@@ -1261,25 +1322,46 @@ static
 struct page *rmqueue_magazine(struct zone *zone, int migratetype)
 {
 	struct page *page = NULL;
-	struct free_area *area;
 
-	/* Check if it is worth acquiring the lock */
-	if (!zone->noirq_magazine.nr_free)
-		return NULL;
-		
-	spin_lock(&zone->magazine_lock);
-	area = &(zone->noirq_magazine);
-	if (list_empty(&area->free_list[migratetype]))
-		goto out;
+	/* Only acquire the lock if there is a reasonable chance of success */
+	if (zone->noirq_magazine.nr_free) {
+		spin_lock(&zone->magazine_lock);
+		page = __rmqueue_magazine(zone, migratetype);
+		spin_unlock(&zone->magazine_lock);
+	}
 
-	/* Page is available in the magazine, allocate it */
-	page = list_entry(area->free_list[migratetype].next, struct page, lru);
-	list_del(&page->lru);
-	area->nr_free--;
-	set_page_private(page, 0);
+	/* Try refilling the magazine on allocaion failure */
+	if (!page) {
+		LIST_HEAD(alloc_list);
+		unsigned long flags;
+		struct free_area *area = &(zone->noirq_magazine);
+		unsigned int i;
+		unsigned int nr_alloced = 0;
+
+		spin_lock_irqsave(&zone->lock, flags);
+		for (i = 0; i < MAGAZINE_ALLOC_BATCH; i++) {
+			page = __rmqueue(zone, 0, migratetype);
+			if (!page)
+				break;
+			list_add_tail(&page->lru, &alloc_list);
+			nr_alloced++;
+		}
+		if (!is_migrate_cma(mt))
+			__mod_zone_page_state(zone, NR_FREE_PAGES, -nr_alloced);
+		else
+			__mod_zone_page_state(zone, NR_FREE_CMA_PAGES, -nr_alloced);
+		spin_unlock_irqrestore(&zone->lock, flags);
+
+		spin_lock(&zone->magazine_lock);
+		while (!list_empty(&alloc_list)) {
+			page = list_entry(alloc_list.next, struct page, lru);
+			list_move_tail(&page->lru, &area->free_list[migratetype]);
+			area->nr_free++;
+		}
+		page = __rmqueue_magazine(zone, migratetype);
+		spin_unlock(&zone->magazine_lock);
+	}
 
-out:
-	spin_unlock(&zone->magazine_lock);
 	return page;
 }
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#16 lore raw

PATCH 11/22 mm: page allocator: Shrink the magazine to the migratetypes in use

In reply to Mel Gorman

A full free_area is larger than required. Shrink it.

---
 include/linux/mmzone.h |  9 +++++++--
 mm/page_alloc.c        | 23 +++++++++++++++++++----
 2 files changed, 26 insertions(+), 6 deletions(-)
include/linux/mmzone.h +7 -2
index a6f84f1..ca04853 100644
@@ -85,6 +85,11 @@ struct free_area {
 	unsigned long		nr_free;
 };
 
+struct free_area_magazine {
+	struct list_head	free_list[MIGRATE_PCPTYPES];
+	unsigned long		nr_free;
+};
+
 struct pglist_data;
 
 /*
@@ -365,8 +370,8 @@ struct zone {
 	 * Keep some order-0 pages on a separate free list
 	 * protected by an irq-unsafe lock
 	 */
-	spinlock_t		magazine_lock;
-	struct free_area	noirq_magazine;
+	spinlock_t			magazine_lock;
+	struct free_area_magazine	noirq_magazine;
 
 #ifndef CONFIG_SPARSEMEM
 	/*
mm/page_alloc.c +19 -4
index 9426174..79dfda7 100644
@@ -1100,7 +1100,7 @@ static
 struct page *__rmqueue_magazine(struct zone *zone, int migratetype)
 {
 	struct page *page;
-	struct free_area *area = &(zone->noirq_magazine);
+	struct free_area_magazine *area = &(zone->noirq_magazine);
 
 	if (list_empty(&area->free_list[migratetype]))
 		return NULL;
@@ -1116,7 +1116,7 @@ struct page *__rmqueue_magazine(struct zone *zone, int migratetype)
 
 static void magazine_drain(struct zone *zone, int migratetype)
 {
-	struct free_area *area = &(zone->noirq_magazine);
+	struct free_area_magazine *area = &(zone->noirq_magazine);
 	struct list_head *list;
 	struct page *page;
 	unsigned int batch_free = 0;
@@ -1183,12 +1183,27 @@ void free_hot_cold_page(struct page *page, bool cold)
 {
 	struct zone *zone = page_zone(page);
 	int migratetype;
-	struct free_area *area;
+	struct free_area_magazine *area;
 
 	if (!free_pages_prepare(page, 0))
 		return;
 
 	migratetype = get_pageblock_migratetype(page);
+
+	/*
+	 * We only track unmovable, reclaimable and movable on magazines.
+	 * Free ISOLATE pages back to the allocator because they are being
+	 * offlined but treat RESERVE as movable pages so we can get those
+	 * areas back if necessary. Otherwise, we may have to free
+	 * excessively into the page allocator
+	 */
+	if (migratetype >= MIGRATE_PCPTYPES) {
+		if (unlikely(is_migrate_isolate(migratetype))) {
+			free_one_page(zone, page, 0, migratetype);
+			return;
+		}
+		migratetype = MIGRATE_MOVABLE;
+	}
 	set_freepage_migratetype(page, migratetype);
 
 	/* magazine_lock is not safe against IRQs */
@@ -1334,7 +1349,7 @@ struct page *rmqueue_magazine(struct zone *zone, int migratetype)
 	if (!page) {
 		LIST_HEAD(alloc_list);
 		unsigned long flags;
-		struct free_area *area = &(zone->noirq_magazine);
+		struct free_area_magazine *area = &(zone->noirq_magazine);
 		unsigned int i;
 		unsigned int nr_alloced = 0;
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#17 lore raw

PATCH 12/22 mm: page allocator: Remove knowledge of hot/cold from page allocator

In reply to Mel Gorman

The intention of hot/cold in the page allocator was that known cache hot pages would be placed at the head of the list and allocations for data to be immediately used would use cache hot plages. Conversely, pages that were reclaimed from the LRU would be treated as cold and allocations for data that was not going to be immediately used such as ring buffers or readahead pages would use cold pages.

With the introduction of magazines, the benefit is questionable. Regardless of the cache hotness of the physical page, the struct page is modified whether hot or cold is requested. "Cold" pages that are freed will still have hot struct page cache lines as a result of the free and placing them at the tail of the magazine list is counter-productive.

As it's of dubious merit, this patch removes the free_hot_cold_page and free_hot_cold_page_list interface. The __GFP_COLD annotations are left in place for now in case a magazine design can be devised that can take advantage of the hot/cold information sensibly.

---
 arch/sparc/mm/init_64.c     |  4 ++--
 arch/sparc/mm/tsb.c         |  2 +-
 arch/tile/mm/homecache.c    |  2 +-
 include/linux/gfp.h         |  6 +++---
 include/trace/events/kmem.h | 11 ++++-------
 mm/page_alloc.c             | 24 ++++++++----------------
 mm/rmap.c                   |  2 +-
 mm/swap.c                   |  4 ++--
 mm/vmscan.c                 |  6 +++---
 9 files changed, 25 insertions(+), 36 deletions(-)
arch/sparc/mm/init_64.c +2 -2
index d2e50b9..1fdeecc 100644
@@ -2562,7 +2562,7 @@ void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
 {
 	struct page *page = virt_to_page(pte);
 	if (put_page_testzero(page))
-		free_hot_cold_page(page, false);
+		free_base_page(page);
 }
 
 static void __pte_free(pgtable_t pte)
@@ -2570,7 +2570,7 @@ static void __pte_free(pgtable_t pte)
 	struct page *page = virt_to_page(pte);
 	if (put_page_testzero(page)) {
 		pgtable_page_dtor(page);
-		free_hot_cold_page(page, false);
+		free_base_page(page);
 	}
 }
 
arch/sparc/mm/tsb.c +1 -1
index b16adcd..2fcd9b8 100644
@@ -520,7 +520,7 @@ void destroy_context(struct mm_struct *mm)
 	page = mm->context.pgtable_page;
 	if (page && put_page_testzero(page)) {
 		pgtable_page_dtor(page);
-		free_hot_cold_page(page, false);
+		free_base_page(page);
 	}
 
 	spin_lock_irqsave(&ctx_alloc_lock, flags);
arch/tile/mm/homecache.c +1 -1
index eacb91b..4c748fd 100644
@@ -438,7 +438,7 @@ void __homecache_free_pages(struct page *page, unsigned int order)
 	if (put_page_testzero(page)) {
 		homecache_change_page_home(page, order, initial_page_home());
 		if (order == 0) {
-			free_hot_cold_page(page, false);
+			free_base_page(page);
 		} else {
 			init_page_count(page);
 			__free_pages(page, order);
include/linux/gfp.h +3 -3
index edf3184..45cbc43 100644
@@ -70,7 +70,7 @@ struct vm_area_struct;
 #define __GFP_HIGH	((__force gfp_t)___GFP_HIGH)	/* Should access emergency pools? */
 #define __GFP_IO	((__force gfp_t)___GFP_IO)	/* Can start physical IO? */
 #define __GFP_FS	((__force gfp_t)___GFP_FS)	/* Can call down to low-level FS? */
-#define __GFP_COLD	((__force gfp_t)___GFP_COLD)	/* Cache-cold page required */
+#define __GFP_COLD	((__force gfp_t)___GFP_COLD)	/* Cache-cold page requested, currently ignored */
 #define __GFP_NOWARN	((__force gfp_t)___GFP_NOWARN)	/* Suppress page allocation failure warning */
 #define __GFP_REPEAT	((__force gfp_t)___GFP_REPEAT)	/* See above */
 #define __GFP_NOFAIL	((__force gfp_t)___GFP_NOFAIL)	/* See above */
@@ -364,8 +364,8 @@ void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask);
 
 extern void __free_pages(struct page *page, unsigned int order);
 extern void free_pages(unsigned long addr, unsigned int order);
-extern void free_hot_cold_page(struct page *page, bool cold);
-extern void free_hot_cold_page_list(struct list_head *list, bool cold);
+extern void free_base_page(struct page *page);
+extern void free_base_page_list(struct list_head *list);
 
 extern void __free_memcg_kmem_pages(struct page *page, unsigned int order);
 extern void free_memcg_kmem_pages(unsigned long addr, unsigned int order);
include/trace/events/kmem.h +4 -7
index 0a5501a..f2069e8d 100644
@@ -171,24 +171,21 @@ TRACE_EVENT(mm_page_free,
 
 TRACE_EVENT(mm_page_free_batched,
 
-	TP_PROTO(struct page *page, int cold),
+	TP_PROTO(struct page *page),
 
-	TP_ARGS(page, cold),
+	TP_ARGS(page),
 
 	TP_STRUCT__entry(
 		__field(	struct page *,	page		)
-		__field(	int,		cold		)
 	),
 
 	TP_fast_assign(
 		__entry->page		= page;
-		__entry->cold		= cold;
 	),
 
-	TP_printk("page=%p pfn=%lu order=0 cold=%d",
+	TP_printk("page=%p pfn=%lu order=0",
 			__entry->page,
-			page_to_pfn(__entry->page),
-			__entry->cold)
+			page_to_pfn(__entry->page))
 );
 
 TRACE_EVENT(mm_page_alloc,
mm/page_alloc.c +8 -16
index 79dfda7..bb2f116 100644
@@ -1175,11 +1175,8 @@ static void magazine_drain(struct zone *zone, int migratetype)
 	spin_unlock_irqrestore(&zone->lock, flags);
 }
 
-/*
- * Free a 0-order page
- * cold == 1 ? free a cold page : free a hot page
- */
-void free_hot_cold_page(struct page *page, bool cold)
+/* Free a 0-order page */
+void free_base_page(struct page *page)
 {
 	struct zone *zone = page_zone(page);
 	int migratetype;
@@ -1215,26 +1212,21 @@ void free_hot_cold_page(struct page *page, bool cold)
 	/* Put the free page on the magazine list */
 	spin_lock(&zone->magazine_lock);
 	area = &(zone->noirq_magazine);
-	if (!cold)
-		list_add(&page->lru, &area->free_list[migratetype]);
-	else
-		list_add_tail(&page->lru, &area->free_list[migratetype]);
+	list_add(&page->lru, &area->free_list[migratetype]);
 	area->nr_free++;
 
 	/* Drain the magazine if necessary, releases the magazine lock */
 	magazine_drain(zone, migratetype);
 }
 
-/*
- * Free a list of 0-order pages
- */
-void free_hot_cold_page_list(struct list_head *list, bool cold)
+/* Free a list of 0-order pages */
+void free_base_page_list(struct list_head *list)
 {
 	struct page *page, *next;
 
 	list_for_each_entry_safe(page, next, list, lru) {
-		trace_mm_page_free_batched(page, cold);
-		free_hot_cold_page(page, cold);
+		trace_mm_page_free_batched(page);
+		free_base_page(page);
 	}
 }
 
@@ -2564,7 +2556,7 @@ void __free_pages(struct page *page, unsigned int order)
 {
 	if (put_page_testzero(page)) {
 		if (order == 0)
-			free_hot_cold_page(page, false);
+			free_base_page(page);
 		else
 			__free_pages_ok(page, order);
 	}
mm/rmap.c +1 -1
index 807c96b..f60152b 100644
@@ -1166,7 +1166,7 @@ void page_remove_rmap(struct page *page)
 	 * It would be tidy to reset the PageAnon mapping here,
 	 * but that might overwrite a racing page_add_anon_rmap
 	 * which increments mapcount after us but sets mapping
-	 * before us: so leave the reset to free_hot_cold_page,
+	 * before us: so leave the reset to free_base_page,
 	 * and remember that it's only reliable while mapped.
 	 * Leaving it set also helps swapoff to reinstate ptes
 	 * faster for those pages still in swapcache.
mm/swap.c +2 -2
index 36c28e5..382ca11 100644
@@ -63,7 +63,7 @@ static void __page_cache_release(struct page *page)
 static void __put_single_page(struct page *page)
 {
 	__page_cache_release(page);
-	free_hot_cold_page(page, false);
+	free_base_page(page);
 }
 
 static void __put_compound_page(struct page *page)
@@ -712,7 +712,7 @@ void release_pages(struct page **pages, int nr, bool cold)
 	if (zone)
 		spin_unlock_irqrestore(&zone->lru_lock, flags);
 
-	free_hot_cold_page_list(&pages_to_free, cold);
+	free_base_page_list(&pages_to_free);
 }
 EXPORT_SYMBOL(release_pages);
 
mm/vmscan.c +3 -3
index 6a56766..3ca921a 100644
@@ -954,7 +954,7 @@ keep:
 	if (nr_dirty && nr_dirty == nr_congested && global_reclaim(sc))
 		zone_set_flag(zone, ZONE_CONGESTED);
 
-	free_hot_cold_page_list(&free_pages, true);
+	free_base_page_list(&free_pages);
 
 	list_splice(&ret_pages, page_list);
 	count_vm_events(PGACTIVATE, pgactivate);
@@ -1343,7 +1343,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
 
 	spin_unlock_irq(&zone->lru_lock);
 
-	free_hot_cold_page_list(&page_list, true);
+	free_base_page_list(&page_list);
 
 	/*
 	 * If reclaim is isolating dirty pages under writeback, it implies
@@ -1534,7 +1534,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 	__mod_zone_page_state(zone, NR_ISOLATED_ANON + file, -nr_taken);
 	spin_unlock_irq(&zone->lru_lock);
 
-	free_hot_cold_page_list(&l_hold, true);
+	free_base_page_list(&l_hold);
 }
 
 #ifdef CONFIG_SWAP
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#18 lore raw

PATCH 13/22 mm: page allocator: Use list_splice to refill the magazine

In reply to Mel Gorman

No need to operate on one page at a time.

---
 mm/page_alloc.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
mm/page_alloc.c +7 -8
index bb2f116..c014b7a 100644
@@ -1333,6 +1333,7 @@ struct page *rmqueue_magazine(struct zone *zone, int migratetype)
 	/* Only acquire the lock if there is a reasonable chance of success */
 	if (zone->noirq_magazine.nr_free) {
 		spin_lock(&zone->magazine_lock);
+retry:
 		page = __rmqueue_magazine(zone, migratetype);
 		spin_unlock(&zone->magazine_lock);
 	}
@@ -1350,7 +1351,7 @@ struct page *rmqueue_magazine(struct zone *zone, int migratetype)
 			page = __rmqueue(zone, 0, migratetype);
 			if (!page)
 				break;
-			list_add_tail(&page->lru, &alloc_list);
+			list_add(&page->lru, &alloc_list);
 			nr_alloced++;
 		}
 		if (!is_migrate_cma(mt))
@@ -1358,15 +1359,13 @@ struct page *rmqueue_magazine(struct zone *zone, int migratetype)
 		else
 			__mod_zone_page_state(zone, NR_FREE_CMA_PAGES, -nr_alloced);
 		spin_unlock_irqrestore(&zone->lock, flags);
+		if (!nr_alloced)
+			return NULL;
 
 		spin_lock(&zone->magazine_lock);
-		while (!list_empty(&alloc_list)) {
-			page = list_entry(alloc_list.next, struct page, lru);
-			list_move_tail(&page->lru, &area->free_list[migratetype]);
-			area->nr_free++;
-		}
-		page = __rmqueue_magazine(zone, migratetype);
-		spin_unlock(&zone->magazine_lock);
+		list_splice(&alloc_list, &area->free_list[migratetype]);
+		area->nr_free += nr_alloced;
+		goto retry;
 	}
 
 	return page;
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#19 lore raw

PATCH 14/22 mm: page allocator: Do not disable IRQs just to update stats

In reply to Mel Gorman

The fast path of the allocator disables/enables interrupts to update statistics but these statistics are only consumed by userspace. When the page allocator always had to disable IRQs it was ok as we already took the penalty but now with the IRQ-unsafe magazine it is overkill to disable IRQs just to have accurate statistics. This patch does not disable IRQs for updating statistics and accepts that the counters might be slightly inaccurate.

---
 mm/page_alloc.c | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)
mm/page_alloc.c +10 -7
index c014b7a..3d619e3 100644
@@ -1381,7 +1381,6 @@ struct page *rmqueue(struct zone *preferred_zone,
 			struct zone *zone, unsigned int order,
 			gfp_t gfp_flags, int migratetype)
 {
-	unsigned long flags;
 	struct page *page = NULL;
 
 	if (unlikely(gfp_flags & __GFP_NOFAIL)) {
@@ -1406,11 +1405,9 @@ again:
 	if (order == 0 && !in_interrupt() && !irqs_disabled())
 		page = rmqueue_magazine(zone, migratetype);
 
-	/* IRQ disabled for buddy list access of updating statistics */
-	local_irq_save(flags);
-
 	if (!page) {
-		spin_lock(&zone->lock);
+		unsigned long flags;
+		spin_lock_irqsave(&zone->lock, flags);
 		page = __rmqueue(zone, order, migratetype);
 		if (!page) {
 			spin_unlock_irqrestore(&zone->lock, flags);
@@ -1418,12 +1415,18 @@ again:
 		}
 		__mod_zone_freepage_state(zone, -(1 << order),
 					get_freepage_migratetype(page));
-		spin_unlock(&zone->lock);
+		spin_unlock_irqrestore(&zone->lock, flags);
 	}
 
+	/*
+	 * NOTE: These are using the non-IRQ safe stats updating which
+	 * means that some updates will be lost. However, these stats
+	 * are not used internally by the VM and collisions are
+	 * expected to be very rare. Disabling/enabling interrupts just
+	 * to have accurate rarely-used counters is overkill.
+	 */
 	__count_zone_vm_events(PGALLOC, zone, 1 << order);
 	zone_statistics(preferred_zone, zone, gfp_flags);
-	local_irq_restore(flags);
 
 	VM_BUG_ON(bad_range(zone, page));
 	if (prep_new_page(page, order, gfp_flags))
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#20 lore raw

PATCH 15/22 mm: page allocator: Check if interrupts are enabled only once per allocation attempt

In reply to Mel Gorman

in_interrupt() is not that expensive but it's still potentially called a very large number of times during a page allocation. Ensure it's only called once. As the check if now firmly in the allocation path, we can use __GFP_WAIT as an cheaper check for IRQs being disabled.

---
 mm/page_alloc.c | 67 +++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 44 insertions(+), 23 deletions(-)
mm/page_alloc.c +44 -23
index 3d619e3..b30abe8 100644
@@ -1379,7 +1379,8 @@ retry:
 static inline
 struct page *rmqueue(struct zone *preferred_zone,
 			struct zone *zone, unsigned int order,
-			gfp_t gfp_flags, int migratetype)
+			gfp_t gfp_flags, int migratetype,
+			bool use_magazine)
 {
 	struct page *page = NULL;
 
@@ -1398,11 +1399,7 @@ struct page *rmqueue(struct zone *preferred_zone,
 	}
 
 again:
-	/*
-	 * For order-0 allocations that are not from irq context, try
-	 * allocate from a separate magazine of free pages
-	 */
-	if (order == 0 && !in_interrupt() && !irqs_disabled())
+	if (use_magazine)
 		page = rmqueue_magazine(zone, migratetype);
 
 	if (!page) {
@@ -1739,7 +1736,8 @@ static inline void init_zone_allows_reclaim(int nid)
 static struct page *
 get_page_from_freelist(gfp_t gfp_mask, nodemask_t *nodemask, unsigned int order,
 		struct zonelist *zonelist, int high_zoneidx, int alloc_flags,
-		struct zone *preferred_zone, int migratetype)
+		struct zone *preferred_zone, int migratetype,
+		bool use_magazine)
 {
 	struct zoneref *z;
 	struct page *page = NULL;
@@ -1845,7 +1843,7 @@ zonelist_scan:
 
 try_this_zone:
 		page = rmqueue(preferred_zone, zone, order,
-						gfp_mask, migratetype);
+					gfp_mask, migratetype, use_magazine);
 		if (page)
 			break;
 this_zone_full:
@@ -1971,7 +1969,7 @@ static inline struct page *
 __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,
 	struct zonelist *zonelist, enum zone_type high_zoneidx,
 	nodemask_t *nodemask, struct zone *preferred_zone,
-	int migratetype)
+	int migratetype, bool use_magazine)
 {
 	struct page *page;
 
@@ -1989,7 +1987,7 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,
 	page = get_page_from_freelist(gfp_mask|__GFP_HARDWALL, nodemask,
 		order, zonelist, high_zoneidx,
 		ALLOC_WMARK_HIGH|ALLOC_CPUSET,
-		preferred_zone, migratetype);
+		preferred_zone, migratetype, use_magazine);
 	if (page)
 		goto out;
 
@@ -2024,7 +2022,7 @@ static struct page *
 __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
 	struct zonelist *zonelist, enum zone_type high_zoneidx,
 	nodemask_t *nodemask, int alloc_flags, struct zone *preferred_zone,
-	int migratetype, bool sync_migration,
+	int migratetype, bool sync_migration, bool use_magazine,
 	bool *contended_compaction, bool *deferred_compaction,
 	unsigned long *did_some_progress)
 {
@@ -2048,7 +2046,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
 		page = get_page_from_freelist(gfp_mask, nodemask,
 				order, zonelist, high_zoneidx,
 				alloc_flags & ~ALLOC_NO_WATERMARKS,
-				preferred_zone, migratetype);
+				preferred_zone, migratetype, use_magazine);
 		if (page) {
 			preferred_zone->compact_blockskip_flush = false;
 			preferred_zone->compact_considered = 0;
@@ -2124,7 +2122,7 @@ static inline struct page *
 __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
 	struct zonelist *zonelist, enum zone_type high_zoneidx,
 	nodemask_t *nodemask, int alloc_flags, struct zone *preferred_zone,
-	int migratetype, unsigned long *did_some_progress)
+	int migratetype, bool use_magazine, unsigned long *did_some_progress)
 {
 	struct page *page = NULL;
 
@@ -2140,7 +2138,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
 	page = get_page_from_freelist(gfp_mask, nodemask, order,
 					zonelist, high_zoneidx,
 					alloc_flags & ~ALLOC_NO_WATERMARKS,
-					preferred_zone, migratetype);
+					preferred_zone, migratetype,
+					use_magazine);
 
 	return page;
 }
@@ -2153,14 +2152,14 @@ static inline struct page *
 __alloc_pages_high_priority(gfp_t gfp_mask, unsigned int order,
 	struct zonelist *zonelist, enum zone_type high_zoneidx,
 	nodemask_t *nodemask, struct zone *preferred_zone,
-	int migratetype)
+	int migratetype, bool use_magazine)
 {
 	struct page *page;
 
 	do {
 		page = get_page_from_freelist(gfp_mask, nodemask, order,
 			zonelist, high_zoneidx, ALLOC_NO_WATERMARKS,
-			preferred_zone, migratetype);
+			preferred_zone, migratetype, use_magazine);
 
 		if (!page && gfp_mask & __GFP_NOFAIL)
 			wait_iff_congested(preferred_zone, BLK_RW_ASYNC, HZ/50);
@@ -2239,7 +2238,7 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	struct zonelist *zonelist, enum zone_type high_zoneidx,
 	nodemask_t *nodemask, struct zone *preferred_zone,
-	int migratetype)
+	int migratetype, bool use_magazine)
 {
 	const gfp_t wait = gfp_mask & __GFP_WAIT;
 	struct page *page = NULL;
@@ -2297,7 +2296,7 @@ rebalance:
 	/* This is the last chance, in general, before the goto nopage. */
 	page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
 			high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,
-			preferred_zone, migratetype);
+			preferred_zone, migratetype, use_magazine);
 	if (page)
 		goto got_pg;
 
@@ -2312,7 +2311,7 @@ rebalance:
 
 		page = __alloc_pages_high_priority(gfp_mask, order,
 				zonelist, high_zoneidx, nodemask,
-				preferred_zone, migratetype);
+				preferred_zone, migratetype, use_magazine);
 		if (page) {
 			goto got_pg;
 		}
@@ -2339,6 +2338,7 @@ rebalance:
 					nodemask,
 					alloc_flags, preferred_zone,
 					migratetype, sync_migration,
+					use_magazine,
 					&contended_compaction,
 					&deferred_compaction,
 					&did_some_progress);
@@ -2361,7 +2361,8 @@ rebalance:
 					zonelist, high_zoneidx,
 					nodemask,
 					alloc_flags, preferred_zone,
-					migratetype, &did_some_progress);
+					migratetype, use_magazine,
+					&did_some_progress);
 	if (page)
 		goto got_pg;
 
@@ -2380,7 +2381,7 @@ rebalance:
 			page = __alloc_pages_may_oom(gfp_mask, order,
 					zonelist, high_zoneidx,
 					nodemask, preferred_zone,
-					migratetype);
+					migratetype, use_magazine);
 			if (page)
 				goto got_pg;
 
@@ -2424,6 +2425,7 @@ rebalance:
 					nodemask,
 					alloc_flags, preferred_zone,
 					migratetype, sync_migration,
+					use_magazine,
 					&contended_compaction,
 					&deferred_compaction,
 					&did_some_progress);
@@ -2442,6 +2444,24 @@ got_pg:
 }
 
 /*
+ * For order-0 allocations that are not from irq context, try
+ * allocate from a separate magazine of free pages.
+ */
+static inline bool should_alloc_use_magazine(gfp_t gfp_mask, unsigned int order)
+{
+	if (order)
+		return false;
+
+	if (gfp_mask & __GFP_WAIT)
+		return true;
+
+	if (in_interrupt() || irqs_disabled())
+		return false;
+
+	return true;
+}
+
+/*
  * This is the 'heart' of the zoned buddy allocator.
  */
 struct page *
@@ -2455,6 +2475,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
 	unsigned int cpuset_mems_cookie;
 	int alloc_flags = ALLOC_WMARK_LOW|ALLOC_CPUSET;
 	struct mem_cgroup *memcg = NULL;
+	bool use_magazine = should_alloc_use_magazine(gfp_mask, order);
 
 	gfp_mask &= gfp_allowed_mask;
 
@@ -2497,7 +2518,7 @@ retry_cpuset:
 	/* First allocation attempt */
 	page = get_page_from_freelist(gfp_mask|__GFP_HARDWALL, nodemask, order,
 			zonelist, high_zoneidx, alloc_flags,
-			preferred_zone, migratetype);
+			preferred_zone, migratetype, use_magazine);
 	if (unlikely(!page)) {
 		/*
 		 * Runtime PM, block IO and its error handling path
@@ -2507,7 +2528,7 @@ retry_cpuset:
 		gfp_mask = memalloc_noio_flags(gfp_mask);
 		page = __alloc_pages_slowpath(gfp_mask, order,
 				zonelist, high_zoneidx, nodemask,
-				preferred_zone, migratetype);
+				preferred_zone, migratetype, use_magazine);
 	}
 
 	trace_mm_page_alloc(page, order, gfp_mask, migratetype);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#21 lore raw

PATCH 16/22 mm: page allocator: Remove coalescing improvement heuristic during page free

In reply to Mel Gorman

Commit 6dda9d55 ( page allocator: reduce fragmentation in buddy allocator by adding buddies that are merging to the tail of the free lists) classified pages according to their probability of being part of a high order merge. This made sense when the number of pages being freed was relatively small as part of a per-cpu list drain.

However, with the introduction of magazines, a drain of the magazines frees larger number of pages in batch and the heuristic is less likely to benefit but adds a lot of weight to the free path in the normal case. The free path can be very hot for workloads with short-lived processes, are fault intensive or work with many in-kernel short-lived buffers. As THP is the main benefit of such a heuristic, it's too marginal a gain to impact the free path so heavily, remove it.

---
 mm/page_alloc.c | 22 ----------------------
 1 file changed, 22 deletions(-)
mm/page_alloc.c +0 -22
index b30abe8..6760e00 100644
@@ -577,29 +577,7 @@ static inline void __free_one_page(struct page *page,
 	}
 	set_page_order(page, order);
 
-	/*
-	 * If this is not the largest possible page, check if the buddy
-	 * of the next-highest order is free. If it is, it's possible
-	 * that pages are being freed that will coalesce soon. In case,
-	 * that is happening, add the free page to the tail of the list
-	 * so it's less likely to be used soon and more likely to be merged
-	 * as a higher order page
-	 */
-	if ((order < MAX_ORDER-2) && pfn_valid_within(page_to_pfn(buddy))) {
-		struct page *higher_page, *higher_buddy;
-		combined_idx = buddy_idx & page_idx;
-		higher_page = page + (combined_idx - page_idx);
-		buddy_idx = __find_buddy_index(combined_idx, order + 1);
-		higher_buddy = higher_page + (buddy_idx - combined_idx);
-		if (page_is_buddy(higher_page, higher_buddy, order + 1)) {
-			list_add_tail(&page->lru,
-				&zone->free_area[order].free_list[migratetype]);
-			goto out;
-		}
-	}
-
 	list_add(&page->lru, &zone->free_area[order].free_list[migratetype]);
-out:
 	zone->free_area[order].nr_free++;
 }
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#22 lore raw

PATCH 17/22 mm: page allocator: Move magazine access behind accessors

In reply to Mel Gorman

In preparation for splitting the magazines, move them behind accessors.

---
 include/linux/mmzone.h |  4 ++--
 mm/page_alloc.c        | 57 +++++++++++++++++++++++++++++++++-----------------
 mm/vmstat.c            |  5 +----
 3 files changed, 41 insertions(+), 25 deletions(-)
include/linux/mmzone.h +2 -2
index ca04853..4eb5151 100644
@@ -370,8 +370,8 @@ struct zone {
 	 * Keep some order-0 pages on a separate free list
 	 * protected by an irq-unsafe lock
 	 */
-	spinlock_t			magazine_lock;
-	struct free_area_magazine	noirq_magazine;
+	spinlock_t			_magazine_lock;
+	struct free_area_magazine	_noirq_magazine;
 
 #ifndef CONFIG_SPARSEMEM
 	/*
mm/page_alloc.c +38 -19

127 lines not on this page · raw message

mm/vmstat.c +1 -4
index 7274ca5..3db0d52 100644
@@ -1003,15 +1003,12 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,
 		   ")"
 		   "\n  noirq magazine");
 	seq_printf(m,
-		"\n    cpu: %i"
 		"\n              count: %lu",
-		i,
-		zone->noirq_magazine.nr_free);
+		zone->_noirq_magazine.nr_free);
 
 #ifdef CONFIG_SMP
 	for_each_online_cpu(i) {
 		struct per_cpu_pageset *pageset;
-
  		pageset = per_cpu_ptr(zone->pageset, i);
 		seq_printf(m, "\n  pagesets\n  vm stats threshold: %d",
  				pageset->stat_threshold);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#23 lore raw

PATCH 18/22 mm: page allocator: Split magazine lock in two to reduce contention

In reply to Mel Gorman

This is the simpliest example of how the lock can be split arbitrarily on a boundary. Ideally it would be based on SMT characteristics but lets just split it in two to start with and prefer a magazine based on the processor ID.

---
 include/linux/mmzone.h |  12 ++++--
 mm/page_alloc.c        | 114 +++++++++++++++++++++++++++++++++++--------------
 mm/vmstat.c            |   8 ++--
 3 files changed, 95 insertions(+), 39 deletions(-)
include/linux/mmzone.h +9 -3
index 4eb5151..c0a8958 100644
@@ -90,6 +90,11 @@ struct free_area_magazine {
 	unsigned long		nr_free;
 };
 
+struct free_magazine {
+	spinlock_t			lock;
+	struct free_area_magazine	area;
+};
+
 struct pglist_data;
 
 /*
@@ -305,6 +310,8 @@ enum zone_type {
 
 #ifndef __GENERATING_BOUNDS_H
 
+#define NR_MAGAZINES 2
+
 struct zone {
 	/* Fields commonly accessed by the page allocator */
 
@@ -368,10 +375,9 @@ struct zone {
 
 	/*
 	 * Keep some order-0 pages on a separate free list
-	 * protected by an irq-unsafe lock
+	 * protected by an irq-unsafe lock.
 	 */
-	spinlock_t			_magazine_lock;
-	struct free_area_magazine	_noirq_magazine;
+	struct free_magazine	noirq_magazine[NR_MAGAZINES];
 
 #ifndef CONFIG_SPARSEMEM
 	/*
mm/page_alloc.c +81 -33

205 lines not on this page · raw message

mm/vmstat.c +5 -3
index 3db0d52..1374f92 100644
@@ -1002,9 +1002,11 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,
 	seq_printf(m,
 		   ")"
 		   "\n  noirq magazine");
-	seq_printf(m,
-		"\n              count: %lu",
-		zone->_noirq_magazine.nr_free);
+	for (i = 0; i < NR_MAGAZINES; i++) {
+		seq_printf(m,
+			"\n              count: %lu",
+			zone->noirq_magazine[i].area.nr_free);
+	}
 
 #ifdef CONFIG_SMP
 	for_each_online_cpu(i) {
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#24 lore raw

Re: [PATCH 18/22] mm: page allocator: Split magazine lock in two to reduce contention

include/linux/mmzone.h struct zone { · line 380

Quoted 10 lines · Mel Gorman
include/linux/mmzone.h +2 -3
@@ -368,10 +375,9 @@ struct zone {
 
 	/*
 	 * Keep some order-0 pages on a separate free list
-	 * protected by an irq-unsafe lock
+	 * protected by an irq-unsafe lock.
 	 */
-	spinlock_t			_magazine_lock;
-	struct free_area_magazine	_noirq_magazine;
+	struct free_magazine	noirq_magazine[NR_MAGAZINES];

Looks like pretty cool stuff!

The old per-cpu-pages stuff was all hung off alloc_percpu(), which surely wasted lots of memory with many NUMA nodes. It's nice to see this decoupled a bit from the online cpu count.

That said, the alloc_percpu() stuff is nice in how much it hides from you when doing cpu hotplug. We'll _probably_ need this to be dynamically-sized at some point, right?

mm/page_alloc.c void mark_free_pages(struct zone *zone) · line 1082

Quoted 10 lines
mm/page_alloc.c +4 -4
-static inline struct free_area_magazine *find_lock_magazine(struct zone *zone)
+static inline struct free_magazine *lock_magazine(struct zone *zone)
 {
-	struct free_area_magazine *area = &zone->_noirq_magazine;
-	spin_lock(&zone->_magazine_lock);
-	return area;
+	int i = (raw_smp_processor_id() >> 1) & (NR_MAGAZINES-1);
+	spin_lock(&zone->noirq_magazine[i].lock);
+	return &zone->noirq_magazine[i];
 }

I bet this logic will be fun to play with once we have more magazines around. For instance, on my system processors 0/80 are HT twins, so they'd always be going after the same magazine. I guess that's a good thing.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#25 lore raw

Re: [PATCH 18/22] mm: page allocator: Split magazine lock in two to reduce contention

In reply to Mel Gorman

mm/page_alloc.c void mark_free_pages(struct zone *zone) · line 1091

Quoted 12 lines · Mel Gorman
mm/page_alloc.c +7 -3
-static inline struct free_area_magazine *find_lock_filled_magazine(struct zone *zone)
+static inline struct free_magazine *find_lock_magazine(struct zone *zone)
 {
-	struct free_area_magazine *area = &zone->_noirq_magazine;
-	if (!area->nr_free)
+	int i = (raw_smp_processor_id() >> 1) & (NR_MAGAZINES-1);
+	int start = i;
+
+	do {
+		if (spin_trylock(&zone->noirq_magazine[i].lock))
+			goto out;

I'm not sure doing it this way is great. It optimizes for lock contention vs the initial cost of just fetching the cache line. Doing the try lock already has to fetch the cache line, even if the lock is contended.

Page allocation should be limited more by the cache line bouncing than long contention

So you may be paying the fetch cost multiple times without actually amortizing it.

If you want to do it this way I would read the lock only. That can be much cheaper because it doesn't have to take the cache line exclusive. It may still need to transfer it though (because another CPU just took it exclusive), which may be already somewhat expensive.

So overall I'm not sure it's a good idea.

-Andi

Signature · 7 lines
[email protected] -- Speaking for myself only

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#26 lore raw

PATCH 19/22 mm: page allocator: Watch for magazine and zone lock contention

In reply to Mel Gorman

When refilling or draining magazines it is possible that the locks are contended. This patch will refill/drain a minimum number of pages and attempt to refill/drain a maximum number. Between the min and max ranges it will check contention and release the lock if it is contended.

---
 mm/page_alloc.c | 38 ++++++++++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 8 deletions(-)
mm/page_alloc.c +30 -8
index 63952f6..727c8d3 100644
@@ -1071,8 +1071,10 @@ void mark_free_pages(struct zone *zone)
 #endif /* CONFIG_PM */
 
 #define MAGAZINE_LIMIT (1024)
-#define MAGAZINE_ALLOC_BATCH (384)
-#define MAGAZINE_FREE_BATCH (64)
+#define MAGAZINE_MIN_ALLOC_BATCH (16)
+#define MAGAZINE_MIN_FREE_BATCH (16)
+#define MAGAZINE_MAX_ALLOC_BATCH (384)
+#define MAGAZINE_MAX_FREE_BATCH (64)
 
 static inline struct free_magazine *lock_magazine(struct zone *zone)
 {
@@ -1138,6 +1140,11 @@ static inline void unlock_magazine(struct free_magazine *mag)
 	spin_unlock(&mag->lock);
 }
 
+static inline bool magazine_contended(struct free_magazine *mag)
+{
+	return spin_is_contended(&mag->lock);
+}
+
 static
 struct page *__rmqueue_magazine(struct free_magazine *mag,
 				int migratetype)
@@ -1163,8 +1170,8 @@ static void magazine_drain(struct zone *zone, struct free_magazine *mag,
 	struct list_head *list;
 	struct page *page;
 	unsigned int batch_free = 0;
-	unsigned int to_free = MAGAZINE_FREE_BATCH;
-	unsigned int nr_freed_cma = 0;
+	unsigned int to_free = MAGAZINE_MAX_FREE_BATCH;
+	unsigned int nr_freed_cma = 0, nr_freed = 0;
 	unsigned long flags;
 	struct free_area_magazine *area = &mag->area;
 	LIST_HEAD(free_list);
@@ -1190,9 +1197,13 @@ static void magazine_drain(struct zone *zone, struct free_magazine *mag,
 			list = &area->free_list[migratetype];;
 		} while (list_empty(list));
 
-		/* This is the only non-empty list. Free them all. */
+		/*
+		 * This is the only non-empty list. Free up the the min-free
+		 * batch so that the spinlock contention is still checked
+		 */
 		if (batch_free == MIGRATE_PCPTYPES)
-			batch_free = to_free;
+			batch_free = min_t(unsigned int,
+					   MAGAZINE_MIN_FREE_BATCH, to_free);
 
 		do {
 			page = list_entry(list->prev, struct page, lru);
@@ -1201,7 +1212,13 @@ static void magazine_drain(struct zone *zone, struct free_magazine *mag,
 			list_move(&page->lru, &free_list);
 			if (is_migrate_isolate_page(zone, page))
 				nr_freed_cma++;
+			nr_freed++;
 		} while (--to_free && --batch_free && !list_empty(list));
+
+		/* Watch for parallel contention */
+		if (nr_freed > MAGAZINE_MIN_FREE_BATCH &&
+		    magazine_contended(mag))
+			break;
 	}
 
 	/* Free the list of pages to the buddy allocator */
@@ -1213,7 +1230,7 @@ static void magazine_drain(struct zone *zone, struct free_magazine *mag,
 		__free_one_page(page, zone, 0, get_freepage_migratetype(page));
 	}
 	__mod_zone_page_state(zone, NR_FREE_PAGES,
-				MAGAZINE_FREE_BATCH - nr_freed_cma);
+				nr_freed - nr_freed_cma);
 	if (nr_freed_cma)
 		__mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_freed_cma);
 	spin_unlock_irqrestore(&zone->lock, flags);
@@ -1388,12 +1405,17 @@ retry:
 		unsigned int nr_alloced = 0;
 
 		spin_lock_irqsave(&zone->lock, flags);
-		for (i = 0; i < MAGAZINE_ALLOC_BATCH; i++) {
+		for (i = 0; i < MAGAZINE_MAX_ALLOC_BATCH; i++) {
 			page = __rmqueue(zone, 0, migratetype);
 			if (!page)
 				break;
 			list_add(&page->lru, &alloc_list);
 			nr_alloced++;
+
+			/* Watch for parallel contention */
+			if (nr_alloced > MAGAZINE_MIN_ALLOC_BATCH &&
+			    spin_is_contended(&zone->lock))
+				break;
 		}
 		if (!is_migrate_cma(mt))
 			__mod_zone_page_state(zone, NR_FREE_PAGES, -nr_alloced);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#27 lore raw

PATCH 20/22 mm: page allocator: Hold magazine lock for a batch of pages

In reply to Mel Gorman

free_base_page_list() frees a list of pages. This patch will batch the magazine lock for the list of pages if possible.

---
 mm/page_alloc.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 61 insertions(+), 14 deletions(-)
mm/page_alloc.c +61 -14
index 727c8d3..cf31191 100644
@@ -1236,15 +1236,13 @@ static void magazine_drain(struct zone *zone, struct free_magazine *mag,
 	spin_unlock_irqrestore(&zone->lock, flags);
 }
 
-/* Free a 0-order page */
-void free_base_page(struct page *page)
+/* Prepare a page for freeing and return its migratetype */
+static inline int free_base_page_prep(struct page *page)
 {
-	struct zone *zone = page_zone(page);
-	struct free_magazine *mag;
 	int migratetype;
 
 	if (!free_pages_prepare(page, 0))
-		return;
+		return -1;
 
 	migratetype = get_pageblock_migratetype(page);
 
@@ -1256,24 +1254,46 @@ void free_base_page(struct page *page)
 	 * excessively into the page allocator
 	 */
 	if (migratetype >= MIGRATE_PCPTYPES) {
-		if (unlikely(is_migrate_isolate(migratetype))) {
-			free_one_page(zone, page, 0, migratetype);
-			return;
-		}
-		migratetype = MIGRATE_MOVABLE;
+		if (likely(!is_migrate_isolate(migratetype)))
+			migratetype = MIGRATE_MOVABLE;
 	}
+
+	set_freepage_migratetype(page, migratetype);
+
+	return migratetype;
+}
+
+/* Put the free page on the magazine list with magazine lock held */
+static inline void __free_base_page(struct zone *zone,
+				struct free_area_magazine *area,
+				struct page *page, int migratetype)
+{
+	list_add(&page->lru, &area->free_list[migratetype]);
+	area->nr_free++;
+}
+
+/* Free a 0-order page */
+void free_base_page(struct page *page)
+{
+	struct zone *zone = page_zone(page);
+	struct free_magazine *mag;
+	int migratetype;
+
+	migratetype = free_base_page_prep(page);
+	if (migratetype == -1)
+		return;
 	set_freepage_migratetype(page, migratetype);
 
 	/* magazine_lock is not safe against IRQs */
-	if (in_interrupt() || irqs_disabled()) {
+	if (migratetype >= MIGRATE_PCPTYPES || in_interrupt() ||
+					       irqs_disabled()) {
 		free_one_page(zone, page, 0, migratetype);
 		return;
 	}
 
 	/* Put the free page on the magazine list */
 	mag = lock_magazine(zone);
-	list_add(&page->lru, &mag->area.free_list[migratetype]);
-	mag->area.nr_free++;
+	__free_base_page(zone, &mag->area, page, migratetype);
 
 	/* Drain the magazine if necessary, releases the magazine lock */
 	magazine_drain(zone, mag, migratetype);
@@ -1283,11 +1303,38 @@ void free_base_page(struct page *page)
 void free_base_page_list(struct list_head *list)
 {
 	struct page *page, *next;
+	struct zone *locked_zone = NULL;
+	struct free_magazine *mag = NULL;
+	bool use_magazine = (!in_interrupt() && !irqs_disabled());
+	int migratetype = MIGRATE_UNMOVABLE;
 
+	/* Similar to free_hot_cold_page except magazine lock is batched */
 	list_for_each_entry_safe(page, next, list, lru) {
+		struct zone *zone = page_zone(page);
+		int migratetype;
+
 		trace_mm_page_free_batched(page);
-		free_base_page(page);
+		migratetype = free_base_page_prep(page);
+		if (migratetype == -1)
+			continue;
+
+		if (!use_magazine || migratetype >= MIGRATE_PCPTYPES) {
+			free_one_page(zone, page, 0, migratetype);
+			continue;
+		}
+
+		if (zone != locked_zone) {
+			/* Drain unlocks magazine lock */
+			if (locked_zone)
+				magazine_drain(locked_zone, mag, migratetype);
+			mag = lock_magazine(zone);
+			locked_zone = zone;
+		}
+		__free_base_page(zone, &mag->area, page, migratetype);
 	}
+
+	if (locked_zone)
+		magazine_drain(locked_zone, mag, migratetype);
 }
 
 /*
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#28 lore raw

PATCH 21/22 mm: compaction: Release free page list under a batched magazine lock

In reply to Mel Gorman

Compaction can reuse the vast bulk of free_base_page() to batch hold the magazine lock.

---
 include/linux/gfp.h |  1 +
 mm/compaction.c     | 18 ++----------------
 mm/page_alloc.c     | 21 +++++++++++++++++++--
 3 files changed, 22 insertions(+), 18 deletions(-)
include/linux/gfp.h +1 -0
index 45cbc43..53844b4 100644
@@ -366,6 +366,7 @@ extern void __free_pages(struct page *page, unsigned int order);
 extern void free_pages(unsigned long addr, unsigned int order);
 extern void free_base_page(struct page *page);
 extern void free_base_page_list(struct list_head *list);
+extern unsigned long release_free_page_list(struct list_head *list);
 
 extern void __free_memcg_kmem_pages(struct page *page, unsigned int order);
 extern void free_memcg_kmem_pages(unsigned long addr, unsigned int order);
mm/compaction.c +2 -16
index 05ccb4c..e415d92 100644
@@ -38,20 +38,6 @@ static inline void count_compact_events(enum vm_event_item item, long delta)
 #define CREATE_TRACE_POINTS
 #include <trace/events/compaction.h>
 
-static unsigned long release_freepages(struct list_head *freelist)
-{
-	struct page *page, *next;
-	unsigned long count = 0;
-
-	list_for_each_entry_safe(page, next, freelist, lru) {
-		list_del(&page->lru);
-		__free_page(page);
-		count++;
-	}
-
-	return count;
-}
-
 static void map_pages(struct list_head *list)
 {
 	struct page *page;
@@ -382,7 +368,7 @@ isolate_freepages_range(struct compact_control *cc,
 
 	if (pfn < end_pfn) {
 		/* Loop terminated early, cleanup. */
-		release_freepages(&freelist);
+		release_free_page_list(&freelist);
 		return 0;
 	}
 
@@ -1002,7 +988,7 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
 
 out:
 	/* Release free pages and check accounting */
-	cc->nr_freepages -= release_freepages(&cc->freepages);
+	cc->nr_freepages -= release_free_page_list(&cc->freepages);
 	VM_BUG_ON(cc->nr_freepages != 0);
 
 	return ret;
mm/page_alloc.c +19 -2
index cf31191..374adf8 100644
@@ -1300,20 +1300,24 @@ void free_base_page(struct page *page)
 }
 
 /* Free a list of 0-order pages */
-void free_base_page_list(struct list_head *list)
+static unsigned long __free_base_page_list(struct list_head *list, bool release)
 {
 	struct page *page, *next;
 	struct zone *locked_zone = NULL;
 	struct free_magazine *mag = NULL;
 	bool use_magazine = (!in_interrupt() && !irqs_disabled());
 	int migratetype = MIGRATE_UNMOVABLE;
+	unsigned long count = 0;
 
-	/* Similar to free_hot_cold_page except magazine lock is batched */
+	/* Similar to free_base_page except magazine lock is batched */
 	list_for_each_entry_safe(page, next, list, lru) {
 		struct zone *zone = page_zone(page);
 		int migratetype;
 
 		trace_mm_page_free_batched(page);
+		if (release)
+			BUG_ON(!put_page_testzero(page));
+
 		migratetype = free_base_page_prep(page);
 		if (migratetype == -1)
 			continue;
@@ -1331,10 +1335,23 @@ void free_base_page_list(struct list_head *list)
 			locked_zone = zone;
 		}
 		__free_base_page(zone, &mag->area, page, migratetype);
+		count++;
 	}
 
 	if (locked_zone)
 		magazine_drain(locked_zone, mag, migratetype);
+
+	return count;
+}
+
+void free_base_page_list(struct list_head *list)
+{
+	__free_base_page_list(list, false);
+}
+
+unsigned long release_free_page_list(struct list_head *list)
+{
+	return __free_base_page_list(list, true);
 }
 
 /*
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#29 lore raw

PATCH 22/22 mm: page allocator: Drain magazines for direct compact failures

In reply to Mel Gorman

THP allocations may fail due to pages pinned in magazines so drain them in the event of a direct compact failure. Similarly drain the magazines during memory hot-remove, memory failure and page isolation as before.

---
 include/linux/gfp.h |  2 ++
 mm/memory-failure.c |  1 +
 mm/memory_hotplug.c |  2 ++
 mm/page_alloc.c     | 63 +++++++++++++++++++++++++++++++++++++++++++++--------
 mm/page_isolation.c |  1 +
 5 files changed, 60 insertions(+), 9 deletions(-)
include/linux/gfp.h +2 -0
index 53844b4..fafa28b 100644
@@ -375,6 +375,8 @@ extern void free_memcg_kmem_pages(unsigned long addr, unsigned int order);
 #define free_page(addr) free_pages((addr), 0)
 
 void page_alloc_init(void);
+void drain_zone_magazine(struct zone *zone);
+void drain_all_magazines(void);
 
 /*
  * gfp_allowed_mask is set to GFP_BOOT_MASK during early boot to restrict what
mm/memory-failure.c +1 -0
index 3175ffd..cd201a3 100644
@@ -237,6 +237,7 @@ void shake_page(struct page *p, int access)
 		lru_add_drain_all();
 		if (PageLRU(p))
 			return;
+		drain_zone_magazine(page_zone(p));
 		if (PageLRU(p) || is_free_buddy_page(p))
 			return;
 	}
mm/memory_hotplug.c +2 -0
index 63f473c..b35c6ee 100644
@@ -1526,6 +1526,7 @@ repeat:
 	if (drain) {
 		lru_add_drain_all();
 		cond_resched();
+		drain_all_magazines();
 	}
 
 	pfn = scan_lru_pages(start_pfn, end_pfn);
@@ -1546,6 +1547,7 @@ repeat:
 	/* drain all zone's lru pagevec, this is asynchronous... */
 	lru_add_drain_all();
 	yield();
+	drain_all_magazines();
 	/* check again */
 	offlined_pages = check_pages_isolated(start_pfn, end_pfn);
 	if (offlined_pages < 0) {
mm/page_alloc.c +54 -9

102 lines not on this page · raw message

mm/page_isolation.c +1 -0
index af79199..1279d9d 100644
@@ -62,6 +62,7 @@ out:
 		nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE);
 
 		__mod_zone_freepage_state(zone, -nr_pages, migratetype);
+		drain_zone_magazine(zone);
 	}
 
 	spin_unlock_irqrestore(&zone->lock, flags);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#30 lore raw
In reply to Mel Gorman
Quoted 9 lines · Mel Gorman

So preliminary testing indicates the results are mixed bag. As long as locks are not contended, it performs fine but parallel fault testing hits into spinlock contention on the magazine locks. A greater problem is that because CPUs share magazines it means that the struct pages are frequently dirtied cache lines. If CPU A frees a page to a magazine and CPU B immediately allocates it then the cache line for the page and the magazine bounces and this costs. It's on the TODO list to research if the available literature has anything useful to say that does not depend on per-cpu lists and the associated problems with them.

If we don't want to bounce 'struct page' cache lines around, then we _need_ to make sure that things that don't share caches don't use the same magazine. I'm not sure there's any other way. But, that doesn't mean we have to _statically_ assign cores/thread to particular magazines.

Say we had a percpu hint which points us to the last magazine we used. We always go to it first, and fall back to round-robin if our preferred one is contended. That way, if we have a mixture tasks doing heavy and light allocations, the heavy allocators will tend to "own" a magazine, and the lighter ones would gravitate to sharing one.

It might be taking things too far, but we could even raise the number of magazines only when we actually *see* contention on the existing set.

Quoted 1 line
 24 files changed, 571 insertions(+), 788 deletions(-)

oooooooooooooooooohhhhhhhhhhhhh.

The only question is how much we'll have to bloat it as we try to optimize things. :)

BTW, I really like the 'magazine' name. It's not frequently used in this kind of context and it conjures up a nice mental image whether it be of stacks of periodicals or firearm ammunition clips.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#31 lore raw
Quoted 3 lines · Dave Hansen

BTW, I really like the 'magazine' name. It's not frequently used in this kind of context and it conjures up a nice mental image whether it be of stacks of periodicals or firearm ammunition clips.

The term "magazine" was prominently used in the Bonwick article that was the base of the creation of the SLAB allocator.

http://static.usenix.org/event/usenix01/full_papers/bonwick/bonwick.pdf

http://static.usenix.org/publications/library/proceedings/bos94/full_papers/bonwick.a

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#32 lore raw
In reply to Dave Hansen
Quoted 11 lines · Dave Hansen
Quoted 9 lines · Mel Gorman

So preliminary testing indicates the results are mixed bag. As long as locks are not contended, it performs fine but parallel fault testing hits into spinlock contention on the magazine locks. A greater problem is that because CPUs share magazines it means that the struct pages are frequently dirtied cache lines. If CPU A frees a page to a magazine and CPU B immediately allocates it then the cache line for the page and the magazine bounces and this costs. It's on the TODO list to research if the available literature has anything useful to say that does not depend on per-cpu lists and the associated problems with them.

Quoted 5 lines

If we don't want to bounce 'struct page' cache lines around, then we _need_ to make sure that things that don't share caches don't use the same magazine. I'm not sure there's any other way. But, that doesn't mean we have to _statically_ assign cores/thread to particular magazines.

We could do something similar to sd_llc_id in kernel/sched/core.c to match CPUs to magazines where the data is likely to be at least in the last level cache.

Quoted 6 lines

Say we had a percpu hint which points us to the last magazine we used. We always go to it first, and fall back to round-robin if our preferred one is contended. That way, if we have a mixture tasks doing heavy and light allocations, the heavy allocators will tend to "own" a magazine, and the lighter ones would gravitate to sharing one.

We might not need the percpu hint if the sd_llc_id style hint was good enough.

Quoted 3 lines

It might be taking things too far, but we could even raise the number of magazines only when we actually *see* contention on the existing set.

I had considered a similar idea. I think it would be relatively easy to grow the number of magazines or even allocate them on a per-process basis but it was less clear how it would be shrunk again.

Quoted 7 lines
Quoted 1 line
 24 files changed, 571 insertions(+), 788 deletions(-)

oooooooooooooooooohhhhhhhhhhhhh.

The only question is how much we'll have to bloat it as we try to optimize things. :)

Indeed :/

Quoted 3 lines

BTW, I really like the 'magazine' name. It's not frequently used in this kind of context and it conjures up a nice mental image whether it be of stacks of periodicals or firearm ammunition clips.

I remember the term from the papers Christoph cited.

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>