Skip to contentSkip to conversation

Drivers: hv: hv_balloon: avoid OOM killer on the ballooning path

PATCH In review Vitaly Kuznetsov 2 participants 6 messages Feb 19, 2015 – Feb 25, 2015 Hyper-V/Azure core and drivers

lore
Share & tools
lore
#0 lore raw

cover Drivers: hv: hv_balloon: avoid OOM killer on the ballooning path

In some cases host asks us to overballoon and this triggers OOM killer which eventually kills everyone. The easiest way to get into such situation is to avoid onlining memory-hotplugged blocks. Address the issue twice:
- Report offline pages as used to the host so it won't ask us to overballoon;
- Avoid ballooning below the 'floor'.

Vitaly Kuznetsov (2):
Drivers: hv: hv_balloon: report offline pages as being used
Drivers: hv: hv_balloon: refuse to balloon below the floor

 drivers/hv/hv_balloon.c | 44 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 9 deletions(-)
Signature · 1 line
1.9.3
#1 lore raw

PATCH 1/2 Drivers: hv: hv_balloon: report offline pages as being used

When hot-added memory pages are not brought online or when some memory blocks are sent offline the subsequent ballooning process kills the guest with OOM killer. This happens as we don't report these pages as neither used nor free and apparently host algorythm considers them as being unused. Keep track of all online/offline operations and report all currently offline pages as being used so host won't try to balloon them out.

---
 drivers/hv/hv_balloon.c | 33 ++++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)
drivers/hv/hv_balloon.c +24 -9
index a095b70..e4b4454 100644
@@ -503,6 +503,8 @@ struct hv_dynmem_device {
 	 * Number of pages we have currently ballooned out.
 	 */
 	unsigned int num_pages_ballooned;
+	unsigned int num_pages_onlined;
+	unsigned int num_pages_added;
 
 	/*
 	 * State to manage the ballooning (up) operation.
@@ -556,12 +558,15 @@ static void post_status(struct hv_dynmem_device *dm);
 static int hv_memory_notifier(struct notifier_block *nb, unsigned long val,
 			      void *v)
 {
+	struct memory_notify *mem = (struct memory_notify *)v;
+
 	switch (val) {
 	case MEM_GOING_ONLINE:
 		mutex_lock(&dm_device.ha_region_mutex);
 		break;
 
 	case MEM_ONLINE:
+		dm_device.num_pages_onlined += mem->nr_pages;
 	case MEM_CANCEL_ONLINE:
 		mutex_unlock(&dm_device.ha_region_mutex);
 		if (dm_device.ha_waiting) {
@@ -570,8 +575,12 @@ static int hv_memory_notifier(struct notifier_block *nb, unsigned long val,
 		}
 		break;
 
-	case MEM_GOING_OFFLINE:
 	case MEM_OFFLINE:
+		mutex_lock(&dm_device.ha_region_mutex);
+		dm_device.num_pages_onlined -= mem->nr_pages;
+		mutex_unlock(&dm_device.ha_region_mutex);
+		break;
+	case MEM_GOING_OFFLINE:
 	case MEM_CANCEL_OFFLINE:
 		break;
 	}
@@ -896,6 +905,8 @@ static void hot_add_req(struct work_struct *dummy)
 	if (do_hot_add)
 		resp.page_count = process_hot_add(pg_start, pfn_cnt,
 						rg_start, rg_sz);
+
+	dm->num_pages_added += resp.page_count;
 	mutex_unlock(&dm_device.ha_region_mutex);
 #endif
 	/*
@@ -1009,17 +1020,21 @@ static void post_status(struct hv_dynmem_device *dm)
 	status.hdr.trans_id = atomic_inc_return(&trans_id);
 
 	/*
-	 * The host expects the guest to report free memory.
-	 * Further, the host expects the pressure information to
-	 * include the ballooned out pages.
-	 * For a given amount of memory that we are managing, we
-	 * need to compute a floor below which we should not balloon.
-	 * Compute this and add it to the pressure report.
+	 * The host expects the guest to report free and committed memory.
+	 * Furthermore, the host expects the pressure information to include
+	 * the ballooned out pages. For a given amount of memory that we are
+	 * managing we need to compute a floor below which we should not
+	 * balloon. Compute this and add it to the pressure report.
+	 * We also need to report all offline pages (num_pages_added -
+	 * num_pages_onlined) as committed to the host, otherwise it can try
+	 * asking us to balloon them out.
 	 */
 	status.num_avail = val.freeram;
 	status.num_committed = vm_memory_committed() +
-				dm->num_pages_ballooned +
-				compute_balloon_floor();
+		dm->num_pages_ballooned +
+		(dm->num_pages_added > dm->num_pages_onlined ?
+		 dm->num_pages_added - dm->num_pages_onlined : 0) +
+		compute_balloon_floor();
 
 	/*
 	 * If our transaction ID is no longer current, just don't
Signature · 1 line
1.9.3
#2 lore raw

RE: [PATCH 1/2] Drivers: hv: hv_balloon: report offline pages as being used

Quoted 42 lines

-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]
Sent: Thursday, February 19, 2015 8:27 AM
To: KY Srinivasan; [email protected]
Cc: Haiyang Zhang; [email protected]; Dexuan Cui
Subject: [PATCH 1/2] Drivers: hv: hv_balloon: report offline pages as being used

When hot-added memory pages are not brought online or when some memory blocks
are sent offline the subsequent ballooning process kills the guest with OOM killer. This happens as we don't report these pages as neither used nor free and apparently host algorythm considers them as being unused. Keep track of
all online/offline operations and report all currently offline pages as being used so host won't try to balloon them out.

---
 drivers/hv/hv_balloon.c | 33 ++++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)
drivers/hv/hv_balloon.c +2 -0
index a095b70..e4b4454 100644
@@ -503,6 +503,8 @@ struct hv_dynmem_device {
 	 * Number of pages we have currently ballooned out.
 	 */
 	unsigned int num_pages_ballooned;
+	unsigned int num_pages_onlined;
+	unsigned int num_pages_added;
 
 	/*
 	 * State to manage the ballooning (up) operation.
@@ -556,12 +558,15 @@ static void post_status(struct hv_dynmem_device
 *dm);
 static int hv_memory_notifier(struct notifier_block *nb, unsigned long val,
 			      void *v)
 {

drivers/hv/hv_balloon.c static void post_status(struct hv_dynmem_device · line 571

Quoted 10 lines
drivers/hv/hv_balloon.c +3 -0
+	struct memory_notify *mem = (struct memory_notify *)v;
+
 	switch (val) {
 	case MEM_GOING_ONLINE:
 		mutex_lock(&dm_device.ha_region_mutex);
 		break;
 
 	case MEM_ONLINE:
+		dm_device.num_pages_onlined += mem->nr_pages;
 	case MEM_CANCEL_ONLINE:

Why are we not adjusting num_pages_onlined when we cancel the online Operation.

K. Y

#3 lore raw

Re: [PATCH 1/2] Drivers: hv: hv_balloon: report offline pages as being used

Quoted 47 lines · KY Srinivasan
Quoted 47 lines

-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]
Sent: Thursday, February 19, 2015 8:27 AM
To: KY Srinivasan; [email protected]
Cc: Haiyang Zhang; [email protected]; Dexuan Cui
Subject: [PATCH 1/2] Drivers: hv: hv_balloon: report offline pages as being used

When hot-added memory pages are not brought online or when some memory blocks
are sent offline the subsequent ballooning process kills the guest with OOM killer. This happens as we don't report these pages as neither used nor free and apparently host algorythm considers them as being unused. Keep track of
all online/offline operations and report all currently offline pages as being used so host won't try to balloon them out.

---
 drivers/hv/hv_balloon.c | 33 ++++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)
drivers/hv/hv_balloon.c +4 -0
index a095b70..e4b4454 100644
@@ -503,6 +503,8 @@ struct hv_dynmem_device {
 	 * Number of pages we have currently ballooned out.
 	 */
 	unsigned int num_pages_ballooned;
+	unsigned int num_pages_onlined;
+	unsigned int num_pages_added;
 
 	/*
 	 * State to manage the ballooning (up) operation.
@@ -556,12 +558,15 @@ static void post_status(struct hv_dynmem_device
 *dm);
 static int hv_memory_notifier(struct notifier_block *nb, unsigned long val,
 			      void *v)
 {
+	struct memory_notify *mem = (struct memory_notify *)v;
+
 	switch (val) {
 	case MEM_GOING_ONLINE:
 		mutex_lock(&dm_device.ha_region_mutex);
 		break;
 
Quoted 6 lines

drivers/hv/hv_balloon.c static void post_status(struct hv_dynmem_device *dm); · line 570

Quoted 3 lines
drivers/hv/hv_balloon.c +1 -0
 	case MEM_ONLINE:
+		dm_device.num_pages_onlined += mem->nr_pages;
 	case MEM_CANCEL_ONLINE:

Why are we not adjusting num_pages_onlined when we cancel the online Operation.

Because we didn't increase the number yet.

To my understanding, events come in the following order:
1) MEM_GOING_ONLINE - we just take the lock
2) MEM_ONLINE - and we increase nr_pages and drop the lock or
MEM_CANCEL_ONLINE - we just drop the lock (mem never was online so
nr_pages wasn't increased)
3) MEM_GOING_OFFLINE - we do nothing
4) MEM_OFFLINE - and we decrease nr_pages
or
MEM_CANCEL_OFFLINE - we do nothing (mem is still online, no need to
adjust nr_pages)

Quoted 2 lines

K. Y

Signature · 1 line
  Vitaly
#4 lore raw

RE: [PATCH 1/2] Drivers: hv: hv_balloon: report offline pages as being used

Quoted 60 lines

-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]
Sent: Wednesday, February 25, 2015 8:56 AM
To: KY Srinivasan
Cc: [email protected]; Haiyang Zhang; linux- [email protected]; Dexuan Cui
Subject: Re: [PATCH 1/2] Drivers: hv: hv_balloon: report offline pages as being used

Quoted 32 lines · KY Srinivasan
Quoted 32 lines

-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]
Sent: Thursday, February 19, 2015 8:27 AM
To: KY Srinivasan; [email protected]
Cc: Haiyang Zhang; [email protected]; Dexuan Cui
Subject: [PATCH 1/2] Drivers: hv: hv_balloon: report offline pages as being used

When hot-added memory pages are not brought online or when some memory blocks are sent offline the subsequent ballooning process kills the guest with OOM killer. This happens as we don't report these pages as neither used nor free and apparently host algorythm considers them as being unused. Keep track of all online/offline operations and report all currently offline pages as being used so host won't try to balloon them out.

---
 drivers/hv/hv_balloon.c | 33 ++++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)
drivers/hv/hv_balloon.c index renamed from drivers/hv/hv_balloon.c +0 -0

a095b70..e4b4454 100644

drivers/hv/hv_balloon.c +2 -0
@@ -503,6 +503,8 @@ struct hv_dynmem_device {
 	 * Number of pages we have currently ballooned out.
 	 */
 	unsigned int num_pages_ballooned;
+	unsigned int num_pages_onlined;
+	unsigned int num_pages_added;
 
Quoted 3 lines

drivers/hv/hv_balloon.c static void post_status(struct

Quoted 3 lines
drivers/hv/hv_balloon.c +0 -0
 	/*
 	 * State to manage the ballooning (up) operation.
@@ -556,12 +558,15 @@ static void post_status(struct

hv_dynmem_device

Quoted 11 lines
Quoted 11 lines
drivers/hv/hv_balloon.c +2 -0
 *dm);  static int hv_memory_notifier(struct notifier_block *nb,
 unsigned long val,
 			      void *v)
 {
+	struct memory_notify *mem = (struct memory_notify *)v;
+
 	switch (val) {
 	case MEM_GOING_ONLINE:
 		mutex_lock(&dm_device.ha_region_mutex);
 		break;
 
Quoted 8 lines
Quoted 6 lines

drivers/hv/hv_balloon.c static void post_status(struct hv_dynmem_device *dm); · line 570

Quoted 3 lines
drivers/hv/hv_balloon.c +1 -0
 	case MEM_ONLINE:
+		dm_device.num_pages_onlined += mem->nr_pages;
 	case MEM_CANCEL_ONLINE:

Why are we not adjusting num_pages_onlined when we cancel the online Operation.

Because we didn't increase the number yet.

Thanks; my mistake.

K. Y

#5 lore raw

PATCH 2/2 Drivers: hv: hv_balloon: refuse to balloon below the floor

In reply to Vitaly Kuznetsov

When host asks us to balloon up we need to be sure we're not committing suicide by overballooning. Use already existent 'floor' metric as our lowest possible value for free ram.

---
 drivers/hv/hv_balloon.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/hv/hv_balloon.c +11 -0
index e4b4454..3292e97 100644
@@ -1138,6 +1138,8 @@ static void balloon_up(struct work_struct *dummy)
 	bool alloc_error;
 	bool done = false;
 	int i;
+	struct sysinfo val;
+	unsigned long floor;
 
 	/* The host balloons pages in 2M granularity. */
 	WARN_ON_ONCE(num_pages % PAGES_IN_2M != 0);
@@ -1148,6 +1150,15 @@ static void balloon_up(struct work_struct *dummy)
 	 */
 	alloc_unit = 512;
 
+	si_meminfo(&val);
+	floor = compute_balloon_floor();
+
+	/* Refuse to balloon below the floor, keep the 2M granularity. */
+	if (val.freeram - num_pages < floor) {
+		num_pages = val.freeram > floor ? (val.freeram - floor) : 0;
+		num_pages -= num_pages % PAGES_IN_2M;
+	}
+
 	while (!done) {
 		bl_resp = (struct dm_balloon_response *)send_buffer;
 		memset(send_buffer, 0, PAGE_SIZE);
Signature · 1 line
1.9.3