memory_hotplug: Free pages as higher order

PATCH Superseded Arun KS 3 participants 7 messages Sep 25, 2018 – Sep 26, 2018

Subsystems (4)

Memory management Xen hypervisor interface Hyper-V/Azure core and drivers Memory hot(un)plug

lore

Share & tools

lore

Arun KS

author

2018-09-25 05:42 UTC #0 lore raw

PATCH v2 1/1 memory_hotplug: Free pages as higher order

When free pages are done with higher order, time spend on coalescing pages by buddy allocator can be reduced. With section size of 256MB, hot add latency of a single section shows improvement from 50-60 ms to less than 1 ms, hence improving the hot add latency by 60%.

Modify external providers of online callback to align with the change.

Signed-off-by Arun KS <[email protected]>

---
Changes since v1:
- Removed prefetch()

Changes since RFC:
- Rebase.
- As suggested by Michal Hocko remove pages_per_block.
- Modifed external providers of online_page_callback.

v1: https://lore.kernel.org/patchwork/patch/989445/
RFC: https://lore.kernel.org/patchwork/patch/984754/

---
 drivers/hv/hv_balloon.c        |  6 +++--
 drivers/xen/balloon.c          | 18 ++++++++++++---
 include/linux/memory_hotplug.h |  2 +-
 mm/memory_hotplug.c            | 51 ++++++++++++++++++++++++++++++++----------
 4 files changed, 59 insertions(+), 18 deletions(-)

drivers/hv/hv_balloon.c +4 -2

index b1b7880..c5bc0b5 100644
@@ -771,7 +771,7 @@ static void hv_mem_hot_add(unsigned long start, unsigned long size,
 	}
 }
 
-static void hv_online_page(struct page *pg)
+static int hv_online_page(struct page *pg, unsigned int order)
 {
 	struct hv_hotadd_state *has;
 	unsigned long flags;

@@ -783,10 +783,12 @@ static void hv_online_page(struct page *pg)
 		if ((pfn < has->start_pfn) || (pfn >= has->end_pfn))
 			continue;
 
-		hv_page_online_one(has, pg);
+		hv_bring_pgs_online(has, pfn, (1UL << order));
 		break;
 	}
 	spin_unlock_irqrestore(&dm_device.ha_lock, flags);
+
+	return 0;
 }
 
 static int pfn_covered(unsigned long start_pfn, unsigned long pfn_cnt)

drivers/xen/balloon.c +15 -3

index e12bb25..010cf4d 100644
@@ -390,8 +390,8 @@ static enum bp_state reserve_additional_memory(void)
 
 	/*
 	 * add_memory_resource() will call online_pages() which in its turn
-	 * will call xen_online_page() callback causing deadlock if we don't
-	 * release balloon_mutex here. Unlocking here is safe because the
+	 * will call xen_bring_pgs_online() callback causing deadlock if we
+	 * don't release balloon_mutex here. Unlocking here is safe because the
 	 * callers drop the mutex before trying again.
 	 */
 	mutex_unlock(&balloon_mutex);

@@ -422,6 +422,18 @@ static void xen_online_page(struct page *page)
 	mutex_unlock(&balloon_mutex);
 }
 
+static int xen_bring_pgs_online(struct page *pg, unsigned int order)
+{
+	unsigned long i, size = (1 << order);
+	unsigned long start_pfn = page_to_pfn(pg);
+
+	pr_debug("Online %lu pages starting at pfn 0x%lx\n", size, start_pfn);
+	for (i = 0; i < size; i++)
+		xen_online_page(pfn_to_page(start_pfn + i));
+
+	return 0;
+}
+
 static int xen_memory_notifier(struct notifier_block *nb, unsigned long val, void *v)
 {
 	if (val == MEM_ONLINE)

@@ -744,7 +756,7 @@ static int __init balloon_init(void)
 	balloon_stats.max_retry_count = RETRY_UNLIMITED;
 
 #ifdef CONFIG_XEN_BALLOON_MEMORY_HOTPLUG
-	set_online_page_callback(&xen_online_page);
+	set_online_page_callback(&xen_bring_pgs_online);
 	register_memory_notifier(&xen_memory_nb);
 	register_sysctl_table(xen_root);

include/linux/memory_hotplug.h +1 -1

index 34a2822..7b04c1d 100644
@@ -87,7 +87,7 @@ extern int test_pages_in_a_zone(unsigned long start_pfn, unsigned long end_pfn,
 	unsigned long *valid_start, unsigned long *valid_end);
 extern void __offline_isolated_pages(unsigned long, unsigned long);
 
-typedef void (*online_page_callback_t)(struct page *page);
+typedef int (*online_page_callback_t)(struct page *page, unsigned int order);
 
 extern int set_online_page_callback(online_page_callback_t callback);
 extern int restore_online_page_callback(online_page_callback_t callback);

mm/memory_hotplug.c +39 ~~-12~~

72 lines not on this page · raw message

Signature · 1 line

1.9.1

Vlastimil Babka

2018-09-25 09:59 UTC #1 lore raw

Quoted 10 lines · Arun KS

When free pages are done with higher order, time spend on coalescing pages by buddy allocator can be reduced. With section size of 256MB, hot add latency of a single section shows improvement from 50-60 ms to less than 1 ms, hence improving the hot add latency by 60%.
Modify external providers of online callback to align with the change.
Signed-off-by Arun KS <[email protected]>

Hi,

Quoted 14 lines

mm/memory_hotplug.c +5 -4

@@ -655,26 +655,53 @@ void __online_page_free(struct page *page)
 }
 EXPORT_SYMBOL_GPL(__online_page_free);
 
-static void generic_online_page(struct page *page)
+static int generic_online_page(struct page *page, unsigned int order)
 {
-	__online_page_set_limits(page);
-	__online_page_increment_counters(page);
-	__online_page_free(page);
+	unsigned long nr_pages = 1 << order;
+	struct page *p = page;
+	unsigned int loop;
+

mm/memory_hotplug.c void __online_page_free(struct page *page) · line 673

Quoted 10 lines

mm/memory_hotplug.c +10 -0

+	for (loop = 0 ; loop < nr_pages ; loop++, p++) {
+		__ClearPageReserved(p);
+		set_page_count(p, 0);
+	}
+
+	adjust_managed_page_count(page, nr_pages);
+	set_page_refcounted(page);
+	__free_pages(page, order);
+
+	return 0;

This seems like almost complete copy of __free_pages_boot_core(), could you do some code reuse instead? I think Michal Hocko also suggested that.

Thanks,
Vlastimil

Michal Hocko

2018-09-25 18:18 UTC #2 lore raw

On Tue 25-09-18 11:59:09, Vlastimil Babka wrote:
[...]

Quoted 2 lines

This seems like almost complete copy of __free_pages_boot_core(), could you do some code reuse instead? I think Michal Hocko also suggested that.

Yes, please try to reuse as much code as possible

Signature · 2 lines

Michal Hocko
SUSE Labs

Arun KS

author

2018-09-26 10:57 UTC #3 lore raw

Quoted 6 lines · Michal Hocko

On Tue 25-09-18 11:59:09, Vlastimil Babka wrote:
[...]
Quoted 2 lines
This seems like almost complete copy of __free_pages_boot_core(), could you do some code reuse instead? I think Michal Hocko also suggested that.
Yes, please try to reuse as much code as possible

Sure, Will address in next spin.

Regards,
Arun

Arun KS

author

2018-09-26 10:57 UTC #4 lore raw

In reply to Michal Hocko

Quoted 6 lines · Michal Hocko

On Tue 25-09-18 11:59:09, Vlastimil Babka wrote:
[...]
Quoted 2 lines
This seems like almost complete copy of __free_pages_boot_core(), could you do some code reuse instead? I think Michal Hocko also suggested that.
Yes, please try to reuse as much code as possible

Sure, Will address in next spin.

Regards,
Arun

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Michal Hocko

2018-09-25 18:18 UTC #5 lore raw

In reply to Vlastimil Babka

On Tue 25-09-18 11:59:09, Vlastimil Babka wrote:
[...]

Quoted 2 lines

This seems like almost complete copy of __free_pages_boot_core(), could you do some code reuse instead? I think Michal Hocko also suggested that.

Yes, please try to reuse as much code as possible

Signature · 7 lines

Michal Hocko
SUSE Labs

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Vlastimil Babka

2018-09-25 09:59 UTC #6 lore raw

In reply to Arun KS

Quoted 10 lines · Arun KS

When free pages are done with higher order, time spend on coalescing pages by buddy allocator can be reduced. With section size of 256MB, hot add latency of a single section shows improvement from 50-60 ms to less than 1 ms, hence improving the hot add latency by 60%.
Modify external providers of online callback to align with the change.
Signed-off-by Arun KS <[email protected]>

Hi,

Quoted 14 lines

mm/memory_hotplug.c +5 -4

@@ -655,26 +655,53 @@ void __online_page_free(struct page *page)
 }
 EXPORT_SYMBOL_GPL(__online_page_free);
 
-static void generic_online_page(struct page *page)
+static int generic_online_page(struct page *page, unsigned int order)
 {
-	__online_page_set_limits(page);
-	__online_page_increment_counters(page);
-	__online_page_free(page);
+	unsigned long nr_pages = 1 << order;
+	struct page *p = page;
+	unsigned int loop;
+

mm/memory_hotplug.c void __online_page_free(struct page *page) · line 673

Quoted 10 lines

mm/memory_hotplug.c +10 -0

+	for (loop = 0 ; loop < nr_pages ; loop++, p++) {
+		__ClearPageReserved(p);
+		set_page_count(p, 0);
+	}
+
+	adjust_managed_page_count(page, nr_pages);
+	set_page_refcounted(page);
+	__free_pages(page, order);
+
+	return 0;

This seems like almost complete copy of __free_pages_boot_core(), could you do some code reuse instead? I think Michal Hocko also suggested that.

Thanks,
Vlastimil

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel