hv_balloon: fixes for num_pages_onlined accounting and misc improvements
cover hv_balloon: fixes for num_pages_onlined accounting and misc improvements
While doing routing code review I noticed that commit 6df8d9aaf3af ("Drivers: hv: balloon: Correctly update onlined page count") introduced an issue with num_pages_onlined accounting on memory offlining. Deeper look showed that the accounting was always buggy. This is fixed in PATCH3. PATCHes 1 and 2 are preparatory cleanups, PATCH4 adds a tracepoint to post_status so it's now possible to see what's being sent to the host and where the data comes from.
One more thing: currently core MM code forbids offlining blocks with reserved pages in it, I plat to send a patch making it possible.
Tested with WS2016 only.
Vitaly Kuznetsov (4):
hv_balloon: fix printk loglevel
hv_balloon: simplify hv_online_page()/hv_page_online_one()
hv_balloon: fix bugs in num_pages_onlined accounting
hv_balloon: trace post_status
drivers/hv/Makefile | 1 + drivers/hv/hv_balloon.c | 121 +++++++++++++++++++++++++++++------------- drivers/hv/hv_trace_balloon.h | 48 +++++++++++++++++ 3 files changed, 132 insertions(+), 38 deletions(-) create mode 100644 drivers/hv/hv_trace_balloon.h
Signature · 1 line
2.13.6
PATCH 1/4 hv_balloon: fix printk loglevel
We have a mix of different ideas of which loglevel should be used. Unify on the following:
- pr_info() for normal operation
- pr_warn() for 'strange' host behavior
- pr_err() for all errors.
--- drivers/hv/hv_balloon.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/hv/hv_balloon.c +6 -6
@@ -691,7 +691,7 @@ static void hv_mem_hot_add(unsigned long start, unsigned long size, (HA_CHUNK << PAGE_SHIFT));if (ret) { -pr_warn ("hot_add memory failed error is %d\n" , ret); +pr_err ("hot_add memory failed error is %d\n" , ret);if (ret == -EEXIST ) {/* * This error indicates that the error
@@ -1014,7 +1014,7 @@ static void hot_add_req(struct work_struct *dummy) resp.result =0 ;if (!do_hot_add || (resp.page_count ==0 )) -pr_info ("Memory hot add failed\n" ); +pr_err ("Memory hot add failed\n" ); dm->state =DM_INITIALIZED ; resp.hdr.trans_id =atomic_inc_return (&trans_id);
@@ -1041,7 +1041,7 @@ static void process_info(struct hv_dynmem_device *dm, struct dm_info_msg *msg)break ;default : -pr_info ("Received Unknown type: %d\n" , info_hdr->type); +pr_warn ("Received Unknown type: %d\n" , info_hdr->type); } }
@@ -1290,7 +1290,7 @@ static void balloon_up(struct work_struct *dummy)/* * Free up the memory we allocatted. */ -pr_info ("Balloon response failed\n" ); +pr_err ("Balloon response failed\n" );for (i =0 ; i < bl_resp->range_count; i++) free_balloon_pages(&dm_device,
@@ -1421,7 +1421,7 @@ static void cap_resp(struct hv_dynmem_device *dm, struct dm_capabilities_resp_msg *cap_resp) {if (!cap_resp->is_accepted) { -pr_info ("Capabilities not accepted by host\n" ); +pr_err ("Capabilities not accepted by host\n" ); dm->state =DM_INIT_ERROR ; }complete (&dm->host_event);
@@ -1508,7 +1508,7 @@ static void balloon_onchannelcallback(void *context)break ;default : -pr_err ("Unhandled message: type: %d\n" , dm_hdr->type); +pr_warn ("Unhandled message: type: %d\n" , dm_hdr->type); } }
Signature · 1 line
2.13.6
PATCH 2/4 hv_balloon: simplify hv_online_page()/hv_page_online_one()
In reply to Vitaly KuznetsovInstead of doing pfn_to_page() and continuosly casting page to unsigned long just cache the pfn of the page with page_to_pfn().
--- drivers/hv/hv_balloon.c | 27 +++++---------------------- 1 file changed, 5 insertions(+), 22 deletions(-)
drivers/hv/hv_balloon.c +5 -22
@@ -612,28 +612,17 @@ static struct notifier_block hv_memory_nb = {/* Check if the particular page is backed and can be onlined and online it. */ static void hv_page_online_one (struct hv_hotadd_state *has,struct page *pg) { -unsigned long cur_start_pgp; -unsigned long cur_end_pgp;struct hv_hotadd_gap *gap; - - cur_start_pgp = (unsigned long )pfn_to_page (has->covered_start_pfn); - cur_end_pgp = (unsigned long )pfn_to_page (has->covered_end_pfn); +unsigned long pfn =page_to_pfn (pg);/* The page is not backed. */ -if (((unsigned long )pg < cur_start_pgp) || - ((unsigned long )pg >= cur_end_pgp)) +if ((pfn < has->covered_start_pfn) || (pfn >= has->covered_end_pfn))return ;/* Check for gaps. */ list_for_each_entry (gap, &has->gap_list, list) { - cur_start_pgp = (unsigned long ) -pfn_to_page (gap->start_pfn); - cur_end_pgp = (unsigned long ) -pfn_to_page (gap->end_pfn); -if (((unsigned long )pg >= cur_start_pgp) && - ((unsigned long )pg < cur_end_pgp)) { +if ((pfn >= gap->start_pfn) && (pfn < gap->end_pfn))return ; - } }/* This frame is currently backed; online the page. */
@@ -726,19 +715,13 @@ static void hv_mem_hot_add(unsigned long start, unsigned long size,static void hv_online_page (struct page *pg) {struct hv_hotadd_state *has; -unsigned long cur_start_pgp; -unsigned long cur_end_pgp;unsigned long flags; +unsigned long pfn =page_to_pfn (pg);spin_lock_irqsave (&dm_device.ha_lock, flags);list_for_each_entry (has, &dm_device.ha_region_list, list) { - cur_start_pgp = (unsigned long ) -pfn_to_page (has->start_pfn); - cur_end_pgp = (unsigned long )pfn_to_page (has->end_pfn); -/* The page belongs to a different HAS. */ -if (((unsigned long )pg < cur_start_pgp) || - ((unsigned long )pg >= cur_end_pgp)) +if ((pfn < has->start_pfn) || (pfn >= has->end_pfn))continue ;hv_page_online_one (has, pg);
Signature · 1 line
2.13.6
PATCH 3/4 hv_balloon: fix bugs in num_pages_onlined accounting
In reply to Vitaly KuznetsovOur num_pages_onlined accounting is buggy:
1) In case we're offlining a memory block which was present at boot (e.g. when there was no hotplug at all) we subtract 32k from 0 and as num_pages_onlined is unsigned get a very big positive number.
2) Commit 6df8d9aaf3af ("Drivers: hv: balloon: Correctly update onlined page count") made num_pages_onlined counter accurate on onlining but totally incorrect on offlining for partly populated regions: no matter how many pages were onlined and what was actually added to num_pages_onlined counter we always subtract the full region (32k) so again, num_pages_onlined can wrap around zero. By onlining/offlining the same partly populated region multiple times we can make the situation worse.
Solve these issues by doing accurate accounting on offlining: walk HAS list, check for covered range and gaps.
--- drivers/hv/hv_balloon.c | 82 +++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 69 insertions(+), 13 deletions(-)
drivers/hv/hv_balloon.c +69 -13
@@ -576,11 +576,65 @@ static struct hv_dynmem_device dm_device;static void post_status (struct hv_dynmem_device *dm);#ifdef CONFIG_MEMORY_HOTPLUG +static inline bool has_pfn_is_backed (struct hv_hotadd_state *has, +unsigned long pfn) +{ +struct hv_hotadd_gap *gap; + +/* The page is not backed. */ +if ((pfn < has->covered_start_pfn) || (pfn >= has->covered_end_pfn)) +return false; + +/* Check for gaps. */ +list_for_each_entry (gap, &has->gap_list, list) { +if ((pfn >= gap->start_pfn) && (pfn < gap->end_pfn)) +return false; + } + +return true; +} + +static unsigned long hv_page_offline_check (unsigned long start_pfn, +unsigned long nr_pages) +{ +unsigned long pfn = start_pfn, count =0 ; +struct hv_hotadd_state *has; +bool found; + +while (pfn < start_pfn + nr_pages) { +/* +* Search for HAS which covers the pfn and when we find one +* count how many consequitive PFNs are covered. +*/ + found = false; +list_for_each_entry (has, &dm_device.ha_region_list, list) { +while ((pfn >= has->start_pfn) && + (pfn < has->end_pfn) && + (pfn < start_pfn + nr_pages)) { + found = true; +if (has_pfn_is_backed (has, pfn)) + count++; + pfn++; + } + } + +/* +* This PFN is not in any HAS (e.g. we're offlining a region +* which was present at boot), no need to account for it. Go +* to the next one. +*/ +if (!found) + pfn++; + } + +return count; +} +static int hv_memory_notifier (struct notifier_block *nb,unsigned long val,void *v) {struct memory_notify *mem = (struct memory_notify *)v; -unsigned long flags; +unsigned long flags, pfn_count;switch (val) {case MEM_ONLINE :
@@ -593,7 +647,19 @@ static int hv_memory_notifier(struct notifier_block *nb, unsigned long val,case MEM_OFFLINE :spin_lock_irqsave (&dm_device.ha_lock, flags); - dm_device.num_pages_onlined -= mem->nr_pages; + pfn_count =hv_page_offline_check (mem->start_pfn, + mem->nr_pages); +if (pfn_count <= dm_device.num_pages_onlined) { + dm_device.num_pages_onlined -= pfn_count; + }else { +/* +* We're offlining more pages than we managed to online. +* This is unexpected. In any case don't let +* num_pages_onlined wrap around zero. +*/ +WARN_ON_ONCE (1 ); + dm_device.num_pages_onlined =0 ; + }spin_unlock_irqrestore (&dm_device.ha_lock, flags);break ;case MEM_GOING_ONLINE :
@@ -612,19 +678,9 @@ static struct notifier_block hv_memory_nb = {/* Check if the particular page is backed and can be onlined and online it. */ static void hv_page_online_one (struct hv_hotadd_state *has,struct page *pg) { -struct hv_hotadd_gap *gap; -unsigned long pfn =page_to_pfn (pg); - -/* The page is not backed. */ -if ((pfn < has->covered_start_pfn) || (pfn >= has->covered_end_pfn)) +if (!has_pfn_is_backed (has,page_to_pfn (pg)))return ; -/* Check for gaps. */ -list_for_each_entry (gap, &has->gap_list, list) { -if ((pfn >= gap->start_pfn) && (pfn < gap->end_pfn)) -return ; - } -/* This frame is currently backed; online the page. */ __online_page_set_limits (pg);__online_page_increment_counters (pg);
Signature · 1 line
2.13.6
PATCH 4/4 hv_balloon: trace post_status
In reply to Vitaly KuznetsovHyper-V balloon driver makes non-trivial calculations to convert Linux's representation of free/used memory to what Hyper-V host expects to see. Add a tracepoint to see what's being sent and where the data comes from.
--- drivers/hv/Makefile | 1 + drivers/hv/hv_balloon.c | 6 ++++++ drivers/hv/hv_trace_balloon.h | 48 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 55 insertions(+) create mode 100644 drivers/hv/hv_trace_balloon.h
drivers/hv/Makefile +1 -0
@@ -3,6 +3,7 @@ obj-$(CONFIG_HYPERV_UTILS) += hv_utils.o obj-$(CONFIG_HYPERV_BALLOON) += hv_balloon.o CFLAGS_hv_trace.o = -I$(src) +CFLAGS_hv_balloon.o = -I$(src) hv_vmbus-y := vmbus_drv.o \ hv.o connection.o channel.o \
drivers/hv/hv_balloon.c +6 -0
@@ -34,6 +34,9 @@#include <linux/hyperv.h> +#define CREATE_TRACE_POINTS +#include "hv_trace_balloon.h" +/* * We begin with definitions supporting the Dynamic Memory protocol * with the host.
@@ -1159,6 +1162,9 @@ static void post_status(struct hv_dynmem_device *dm) dm->num_pages_added - dm->num_pages_onlined : 0) +compute_balloon_floor (); +trace_balloon_status (status.num_avail, status.num_committed, +vm_memory_committed (), dm->num_pages_ballooned, + dm->num_pages_added, dm->num_pages_onlined);/* * If our transaction ID is no longer current, just don't * send the status. This can happen if we were interrupted
drivers/hv/hv_trace_balloon.h new file +48 -0
48 lines not on this page · raw message
Signature · 1 line
2.13.6
Quoted 8 lines · Vitaly Kuznetsov
While doing routing code review I noticed that commit
6df8d9aaf3af("Drivers: hv: balloon: Correctly update onlined page count") introduced an issue withnum_pages_onlinedaccounting on memory offlining. Deeper look showed that the accounting was always buggy. This is fixed in PATCH3. PATCHes 1 and 2 are preparatory cleanups, PATCH4 adds a tracepoint topost_statusso it's now possible to see what's being sent to the host and where the data comes from.
K. Y., Alex,
did you have a chance to take a look?
Thanks,
Signature · 1 line
Vitaly
Haiyang Zhang
maintainerQuoted 24 lines
-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]
Sent: Tuesday, November 28, 2017 8:30 AM
To: Alex Ng <[email protected]>; KY Srinivasan <[email protected]>
Cc: [email protected]; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; Dexuan Cui <[email protected]>; [email protected]
Subject: Re: [PATCH 0/4]hv_balloon: fixes fornum_pages_onlinedaccounting and misc improvementsQuoted 4 lines · Vitaly Kuznetsov
While doing routing code review I noticed that commit
6df8d9aaf3af("Drivers: hv: balloon: Correctly update onlined page count") introduced an issue withnum_pages_onlinedaccounting on memory offlining. Deeper look showed that the accounting was always buggy. This isfixed in PATCH3.
Quoted 4 lines
PATCHes 1 and 2 are preparatory cleanups, PATCH4 adds a tracepoint to
post_statusso it's now possible to see what's being sent to the host and where the data comes from.
Quoted 8 lines
K. Y., Alex,
did you have a chance to take a look?
Thanks,
--
Vitaly
Dexuan,
Maybe, you can take a look? Thanks.
Dexuan Cui
maintainerQuoted 21 lines
-----Original Message-----
From: Haiyang Zhang
Sent: Wednesday, November 29, 2017 16:55Quoted 18 lines
-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]Quoted 4 lines · Vitaly Kuznetsov
While doing routing code review I noticed that commit
6df8d9aaf3af("Drivers: hv: balloon: Correctly update onlined page count") introduced an issue withnum_pages_onlinedaccounting on memory offlining. Deeper look showed that the accounting was always buggy. This isfixed in PATCH3.
Quoted 4 lines
PATCHes 1 and 2 are preparatory cleanups, PATCH4 adds a tracepoint to
post_statusso it's now possible to see what's being sent to the host and where the data comes from.K. Y., Alex,
did you have a chance to take a look?
Quoted 7 lines
Quoted 3 lines
Thanks,
--
VitalyDexuan,
Maybe, you can take a look? Thanks.
Sure. I'll take a look.
Thanks,
-- Dexuan