Skip to contentSkip to conversation

VMware balloon driver over virtio

PATCH In review Dmitry Torokhov 3 participants 9 messages Mar 19, 2010 – Mar 21, 2010

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover VMware balloon driver over virtio

Hi,

VMware would like to get its Linux balloon driver merged into mainline and is considering doing it within virtio framework.

The following is the initial implementation of our ballooning interface as a virtio device. It allows our hypervisor to work in tandem with virtio_balloon driver and inflate and deflate balloon on demand.

Due to the fact that our monitor works a bit differently from KVM some changes are needed in virtio_balloon itself, namely:

- our hypervisor may refuse locking certain pages. This may happen, for example, when target balloon size is changed since guest polled it last time. We expect guest to try locking different page and release refused pages once target balloon size is reached.

- our hypervisor may, in certain situations, reset the balloon. In this case it expects guest to quickly release all memory allocated by by the balloon but to not bother telling the host about pages being released.

There are more differences (adaptive allocation rates, sleeping allocations) which we intend to address at a later date if it is decided that virtio is the proper subsystem to be used for our driver.

Thanks!

Signature · 1 line
Dmitry
#1 lore raw

PATCH 1/4 virtio-balloon: allow hypervisor refuse locking some pages

Some hypervisors may refuse our request to lock pages in certain circumstances. For example, VMware may decide that the target balloon size decreased since it was last communicated to the guest and thus current page should not be locked.

---

 drivers/virtio/virtio_balloon.c |   68 +++++++++++++++++++++++++++++++++++----
 include/linux/virtio_balloon.h  |    8 +++--
 2 files changed, 67 insertions(+), 9 deletions(-)
drivers/virtio/virtio_balloon.c +61 -7
index 369f2ee..444c929 100644
@@ -25,6 +25,8 @@
 #include <linux/freezer.h>
 #include <linux/delay.h>
 
+#define VIRTIO_BALLOON_CHUNK_SIZE	256
+
 struct virtio_balloon
 {
 	struct virtio_device *vdev;
@@ -46,9 +48,13 @@ struct virtio_balloon
 	unsigned int num_pages;
 	struct list_head pages;
 
+	/* Pages that host refused to lock for one reason or another. */
+	struct list_head refused_pages;
+
 	/* The array of pfns we tell the Host about. */
 	unsigned int num_pfns;
-	u32 pfns[256];
+	u32 pfns[VIRTIO_BALLOON_CHUNK_SIZE];
+	u8 pfns_status[VIRTIO_BALLOON_CHUNK_SIZE];
 
 	/* Memory statistics */
 	int need_stats_update;
@@ -81,14 +87,21 @@ static void balloon_ack(struct virtqueue *vq)
 
 static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
 {
-	struct scatterlist sg;
+	struct scatterlist sg[2];
+	int n_out = 0;
 
-	sg_init_one(&sg, vb->pfns, sizeof(vb->pfns[0]) * vb->num_pfns);
+	sg_init_table(sg, 2);
+	sg_set_buf(&sg[0], vb->pfns, sizeof(vb->pfns[0]) * vb->num_pfns);
+	sg_set_buf(&sg[1], vb->pfns_status,
+			sizeof(vb->pfns_status[0]) * vb->num_pfns);
 
 	init_completion(&vb->acked);
 
+	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_HOST_MAY_REFUSE))
+		n_out = 1;
+
 	/* We should always be able to add one buffer to an empty queue. */
-	if (vq->vq_ops->add_buf(vq, &sg, 1, 0, vb) < 0)
+	if (vq->vq_ops->add_buf(vq, sg, 1, n_out, vb) < 0)
 		BUG();
 	vq->vq_ops->kick(vq);
 
@@ -98,6 +111,8 @@ static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
 
 static void fill_balloon(struct virtio_balloon *vb, size_t num)
 {
+	int i;
+
 	/* We can only do one array worth at a time. */
 	num = min(num, ARRAY_SIZE(vb->pfns));
 
@@ -113,9 +128,7 @@ static void fill_balloon(struct virtio_balloon *vb, size_t num)
 			break;
 		}
 		vb->pfns[vb->num_pfns] = page_to_balloon_pfn(page);
-		totalram_pages--;
-		vb->num_pages++;
-		list_add(&page->lru, &vb->pages);
+		vb->pfns_status[vb->num_pfns] = VIRTIO_BALLOON_PFN_OK;
 	}
 
 	/* Didn't get any?  Oh well. */
@@ -123,6 +136,32 @@ static void fill_balloon(struct virtio_balloon *vb, size_t num)
 		return;
 
 	tell_host(vb, vb->inflate_vq);
+
+	for (i = 0; i < vb->num_pfns; i++) {
+		struct page *page = pfn_to_page(vb->pfns[i]);
+		if (vb->pfns_status[i] == VIRTIO_BALLOON_PFN_OK) {
+			list_add(&page->lru, &vb->pages);
+			vb->num_pages++;
+			totalram_pages--;
+		} else {
+			dev_dbg(&vb->vdev->dev,
+				"adding page %p (pfn %x) to refused pages list\n",
+				page, vb->pfns[i]);
+			list_add(&page->lru, &vb->refused_pages);
+		}
+	}
+}
+
+static void release_refused_pages(struct virtio_balloon *vb)
+{
+	struct page *page, *next;
+
+	list_for_each_entry_safe(page, next, &vb->refused_pages, lru) {
+		dev_dbg(&vb->vdev->dev,
+			"releasing previously refused page %p", page);
+		list_del(&page->lru);
+		__free_page(page);
+	}
 }
 
 static void release_pages_by_pfn(const u32 pfns[], unsigned int num)
@@ -263,13 +302,26 @@ static int balloon(void *_vballoon)
 					 || vb->need_stats_update
 					 || kthread_should_stop()
 					 || freezing(current));
+
 		if (vb->need_stats_update)
 			stats_handle_request(vb);
+
 		if (diff > 0)
 			fill_balloon(vb, diff);
 		else if (diff < 0)
 			leak_balloon(vb, -diff);
+
 		update_balloon_size(vb);
+
+		if (diff > 0 &&
+		    !list_empty(&vb->refused_pages) &&
+		    towards_target(vb) <= 0) {
+			/*
+			 * We reached our goal so we now can release
+			 * pages that host refused to lock for us.
+			 */
+			release_refused_pages(vb);
+		}
 	}
 	return 0;
 }
@@ -289,6 +341,7 @@ static int virtballoon_probe(struct virtio_device *vdev)
 	}
 
 	INIT_LIST_HEAD(&vb->pages);
+	INIT_LIST_HEAD(&vb->refused_pages);
 	vb->num_pages = 0;
 	init_waitqueue_head(&vb->config_change);
 	vb->vdev = vdev;
@@ -357,6 +410,7 @@ static void __devexit virtballoon_remove(struct virtio_device *vdev)
 static unsigned int features[] = {
 	VIRTIO_BALLOON_F_MUST_TELL_HOST,
 	VIRTIO_BALLOON_F_STATS_VQ,
+	VIRTIO_BALLOON_F_HOST_MAY_REFUSE,
 };
 
 static struct virtio_driver virtio_balloon_driver = {
include/linux/virtio_balloon.h +6 -2
index a50ecd1..4baad1c 100644
@@ -6,12 +6,16 @@
 #include <linux/virtio_config.h>
 
 /* The feature bitmap for virtio balloon */
-#define VIRTIO_BALLOON_F_MUST_TELL_HOST	0 /* Tell before reclaiming pages */
-#define VIRTIO_BALLOON_F_STATS_VQ	1 /* Memory Stats virtqueue */
+#define VIRTIO_BALLOON_F_MUST_TELL_HOST		0 /* Tell before reclaiming pages */
+#define VIRTIO_BALLOON_F_STATS_VQ		1 /* Memory Stats virtqueue */
+#define VIRTIO_BALLOON_F_HOST_MAY_REFUSE	2 /* Host may refuse some pages */
 
 /* Size of a PFN in the balloon interface. */
 #define VIRTIO_BALLOON_PFN_SHIFT 12
 
+#define VIRTIO_BALLOON_PFN_OK		0
+#define VIRTIO_BALLOON_PFN_FAIL		1
+
 struct virtio_balloon_config
 {
 	/* Number of pages host wants Guest to give up. */
#3 lore raw

PATCH 2/4 virtio-balloon: implement balloon reset

In reply to Dmitry Torokhov

VMware hypervisor in certain cases may release all locked pages and signal "reset" condition to the guest. In this case guest is expected to free all pages allocated by its balloon but should not notify host that pages have been released.

Note that hypervisors that require guest to tell about pages being released should implement 'reset' by simply setting target ballon size to 0.

---

 drivers/virtio/virtio_balloon.c |   52 +++++++++++++++++++++++++++++++++++++--
 include/linux/virtio_balloon.h  |    5 ++++
 2 files changed, 54 insertions(+), 3 deletions(-)
drivers/virtio/virtio_balloon.c +49 -3
index 444c929..ff6299c 100644
@@ -197,6 +197,20 @@ static void leak_balloon(struct virtio_balloon *vb, size_t num)
 	}
 }
 
+static void pop_balloon(struct virtio_balloon *vb)
+{
+	struct page *page, *next;
+
+	list_for_each_entry_safe(page, next, &vb->pages, lru) {
+		list_del(&page->lru);
+		__free_page(page);
+		vb->num_pages--;
+		totalram_pages++;
+	}
+
+	release_refused_pages(vb);
+}
+
 static inline void update_stat(struct virtio_balloon *vb, int idx,
 			       u16 tag, u64 val)
 {
@@ -270,9 +284,10 @@ static void virtballoon_changed(struct virtio_device *vdev)
 	wake_up(&vb->config_change);
 }
 
-static inline s64 towards_target(struct virtio_balloon *vb)
+static s64 towards_target(struct virtio_balloon *vb)
 {
 	u32 v;
+
 	vb->vdev->config->get(vb->vdev,
 			      offsetof(struct virtio_balloon_config, num_pages),
 			      &v, sizeof(v));
@@ -288,21 +303,51 @@ static void update_balloon_size(struct virtio_balloon *vb)
 			      &actual, sizeof(actual));
 }
 
+static bool check_need_reset(struct virtio_balloon *vb)
+{
+	bool reset_pending;
+
+	if (!virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_CAN_RESET))
+		return false;
+
+	vb->vdev->config->get(vb->vdev,
+			offsetof(struct virtio_balloon_config, reset_pending),
+			&reset_pending, sizeof(reset_pending));
+	return reset_pending;
+}
+
+static void mark_balloon_reset(struct virtio_balloon *vb)
+{
+	bool reset_completed = true;
+
+	vb->vdev->config->set(vb->vdev,
+			offsetof(struct virtio_balloon_config, reset_completed),
+			&reset_completed, sizeof(reset_completed));
+}
+
+
 static int balloon(void *_vballoon)
 {
 	struct virtio_balloon *vb = _vballoon;
 
 	set_freezable();
 	while (!kthread_should_stop()) {
-		s64 diff;
+		s64 diff = 0;
+		bool reset;
 
 		try_to_freeze();
 		wait_event_interruptible(vb->config_change,
-					 (diff = towards_target(vb)) != 0
+					    (reset = check_need_reset(vb))
+					 || (diff = towards_target(vb)) != 0
 					 || vb->need_stats_update
 					 || kthread_should_stop()
 					 || freezing(current));
 
+		if (reset) {
+			pop_balloon(vb);
+			mark_balloon_reset(vb);
+		}
+
 		if (vb->need_stats_update)
 			stats_handle_request(vb);
 
@@ -411,6 +456,7 @@ static unsigned int features[] = {
 	VIRTIO_BALLOON_F_MUST_TELL_HOST,
 	VIRTIO_BALLOON_F_STATS_VQ,
 	VIRTIO_BALLOON_F_HOST_MAY_REFUSE,
+	VIRTIO_BALLOON_F_CAN_RESET,
 };
 
 static struct virtio_driver virtio_balloon_driver = {
include/linux/virtio_balloon.h +5 -0
index 4baad1c..d83c690 100644
@@ -9,6 +9,7 @@
 #define VIRTIO_BALLOON_F_MUST_TELL_HOST		0 /* Tell before reclaiming pages */
 #define VIRTIO_BALLOON_F_STATS_VQ		1 /* Memory Stats virtqueue */
 #define VIRTIO_BALLOON_F_HOST_MAY_REFUSE	2 /* Host may refuse some pages */
+#define VIRTIO_BALLOON_F_CAN_RESET		3 /* Guest can reset balloon */
 
 /* Size of a PFN in the balloon interface. */
 #define VIRTIO_BALLOON_PFN_SHIFT 12
@@ -22,6 +23,10 @@ struct virtio_balloon_config
 	__le32 num_pages;
 	/* Number of pages we've actually got in balloon. */
 	__le32 actual;
+	/* Host wants guest to reset balloon. */
+	bool reset_pending;
+	/* Guest performed reset. */
+	bool reset_completed;
 };
 
 #define VIRTIO_BALLOON_S_SWAP_IN  0   /* Amount of memory swapped in */
#4 lore raw

PATCH 3/4 x86: export vmware_platform() symbol

In reply to Dmitry Torokhov

Export vmware_platform() symbol which is needed by vmware_balloon driver.

---

 arch/x86/kernel/cpu/vmware.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
arch/x86/kernel/cpu/vmware.c +2 -0
index 1cbed97..dfdb4db 100644
@@ -22,6 +22,7 @@
  */
 
 #include <linux/dmi.h>
+#include <linux/module.h>
 #include <asm/div64.h>
 #include <asm/vmware.h>
 #include <asm/x86_init.h>
@@ -101,6 +102,7 @@ int vmware_platform(void)
 
 	return 0;
 }
+EXPORT_SYMBOL(vmware_platform);
 
 /*
  * VMware hypervisor takes care of exporting a reliable TSC to the guest.
#5 lore raw

PATCH 4/4 Add VMware memory balloon driver

In reply to Dmitry Torokhov

This is a virtio-based transport between VMware's hypervisor and virtio_ballon driver that allows host control guest memory ballooning.

---

 drivers/misc/Kconfig          |   13 +
 drivers/misc/Makefile         |    1 
 drivers/misc/vmware_balloon.c |  626 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 640 insertions(+), 0 deletions(-)
 create mode 100644 drivers/misc/vmware_balloon.c
drivers/misc/Kconfig +13 -0
index 2191c8d..de7e97e 100644
@@ -311,6 +311,19 @@ config TI_DAC7512
 	  This driver can also be built as a module. If so, the module
 	  will be calles ti_dac7512.
 
+config VMWARE_BALLOON
+	tristate "VMware Balloon Driver"
+	depends on VIRTIO && X86
+	help
+	  This option enables virtio based transport between VMware hypervisor
+	  and virtio_balloon and allows host control memory ballooning within
+	  the guest.
+
+	  If unsure, say N.
+
+	  To compile this driver as a module, choose M here: the
+	  module will be called vmware_balloon.
+
 source "drivers/misc/c2port/Kconfig"
 source "drivers/misc/eeprom/Kconfig"
 source "drivers/misc/cb710/Kconfig"
drivers/misc/Makefile +1 -0
index 27c4843..7b6f7ee 100644
@@ -29,3 +29,4 @@ obj-$(CONFIG_C2PORT)		+= c2port/
 obj-$(CONFIG_IWMC3200TOP)      += iwmc3200top/
 obj-y				+= eeprom/
 obj-y				+= cb710/
+obj-$(CONFIG_VMWARE_BALLOON)	+= vmware_balloon.o
drivers/misc/vmware_balloon.c new file +626 -0

626 lines not on this page · raw message

#2 lore raw

Re: [PATCH 1/4] virtio-balloon: allow hypervisor refuse locking some pages

In reply to Dmitry Torokhov
Quoted 21 lines · Dmitry Torokhov

Some hypervisors may refuse our request to lock pages in certain circumstances. For example, VMware may decide that the target balloon size decreased since it was last communicated to the guest and thus current page should not be locked.

  static struct virtio_driver virtio_balloon_driver = {
include/linux/virtio_balloon.h +3 -2
index a50ecd1..4baad1c 100644
@@ -6,12 +6,16 @@
  #include<linux/virtio_config.h>
 
  /* The feature bitmap for virtio balloon */
-#define VIRTIO_BALLOON_F_MUST_TELL_HOST	0 /* Tell before reclaiming pages */
-#define VIRTIO_BALLOON_F_STATS_VQ	1 /* Memory Stats virtqueue */
+#define VIRTIO_BALLOON_F_MUST_TELL_HOST		0 /* Tell before reclaiming pages */
+#define VIRTIO_BALLOON_F_STATS_VQ		1 /* Memory Stats virtqueue */
+#define VIRTIO_BALLOON_F_HOST_MAY_REFUSE	2 /* Host may refuse some pages */

include/linux/virtio_balloon.h line 21

Quoted 10 lines
include/linux/virtio_balloon.h +3 -0
 
  /* Size of a PFN in the balloon interface. */
  #define VIRTIO_BALLOON_PFN_SHIFT 12
 
+#define VIRTIO_BALLOON_PFN_OK		0
+#define VIRTIO_BALLOON_PFN_FAIL		1
+
  struct virtio_balloon_config
  {
  	/* Number of pages host wants Guest to give up. */

Please update the virtio pci spec (can't find the URL - Rusty?) to reflect the protocol change.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#6 lore raw

Re: [PATCH 4/4] Add VMware memory balloon driver

In reply to Dmitry Torokhov
Quoted 5 lines · Dmitry Torokhov

This is a virtio-based transport between VMware's hypervisor and virtio_ballon driver that allows host control guest memory ballooning.

This is a little wierd since it's specific to the balloon driver. The virtio design is to make transport and device orthogonal so we could mix'n'match any driver with any transport, but this will only work with the balloon driver. It's also 640 lines while the original balloon driver is ~300. Some of that is because you're emulating a queue that doesn't really exist, IIUC.

So while I'd love to see other hypervisors adopt virtio, I think in this case it makes more sense to have your own balloon driver.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#7 lore raw

Re: [PATCH 4/4] Add VMware memory balloon driver

Hi Avi,

Quoted 7 lines · Avi Kivity
Quoted 5 lines · Dmitry Torokhov

This is a virtio-based transport between VMware's hypervisor and virtio_ballon driver that allows host control guest memory ballooning.

Quoted 6 lines

This is a little wierd since it's specific to the balloon driver. The virtio design is to make transport and device orthogonal so we could mix'n'match any driver with any transport, but this will only work with the balloon driver. It's also 640 lines while the original balloon driver is ~300. Some of that is because you're emulating a queue that doesn't really exist, IIUC.

Yes, this is corect. We have a dedicated balloon interface, so, while trying to work with existing kernel infrastructure, we had to emulate a virtual queue.

Quoted 4 lines

So while I'd love to see other hypervisors adopt virtio, I think in this case it makes more sense to have your own balloon driver.

Fair enough. If this is a common sentiment we'll gladly prepare our standalone driver for submission.

Thanks.

Signature · 1 line
Dmitry
#8 lore raw

Re: [PATCH 4/4] Add VMware memory balloon driver

Quoted 7 lines · Dmitry Torokhov
Quoted 4 lines

So while I'd love to see other hypervisors adopt virtio, I think in this case it makes more sense to have your own balloon driver.

Fair enough. If this is a common sentiment we'll gladly prepare our standalone driver for submission.

It's also possible to split out a "balloon core" that can be shared. In this case it isn't worth it however as there any user visible interfaces and the shared code would amount to couple dozen lines or so.

Signature · 1 line
error compiling committee.c: too many arguments to function