Skip to contentSkip to conversation

virtio: Report new guest memory statistics pertinent to memory ballooning

RFC In review Adam Litke 4 participants 7 messages Nov 5, 2009 – Nov 9, 2009

Subsystems (2)
lore
Share & tools
lore

Adam Litke

author
#0 lore raw

RFC 1/1 virtio: Report new guest memory statistics pertinent to memory ballooning

[RFC] virtio: Report new guest memory statistics pertinent to memory ballooning

When using ballooning to manage overcommitted memory on a host, a system for guests to communicate their memory usage to the host can provide information that will minimize the impact of ballooning on the guests. The current method employs a daemon running in each guest that communicates memory statistics to a host daemon at a specified time interval. The host daemon aggregates this information and inflates and/or deflates balloons according to the level of host memory pressure. This approach is effective but overly complex since a daemon must be installed inside each guest and coordinated to communicate with the host. A simpler approach is to collect memory statistics in the virtio balloon driver and communicate them to the host via the device config space.

This patch implements the qemu side of the communication channel. I will post the kernel driver modifications in-reply to this message.

I'd like to pose a few questions concerning my implementation:

* Is there a better way to use feature codes than using one per variable?

* This patch causes the 'info balloon' monitor command to generate output like the following:

(qemu) info balloon
balloon: actual=1024 MB
balloon: pswapin=0 pages
balloon: pswapout=0 pages
balloon: panon=3928 KB
balloon: pgmajfault=0
balloon: pgminfault=247914
balloon: memfree=987032 KB
balloon: memtot=1020812 KB

Is this agreeable?

Thank you for your comments...

balloon.h +9 -2
index 60b4a5d..3f67021 100644
@@ -14,14 +14,21 @@
 #ifndef _QEMU_BALLOON_H
 #define _QEMU_BALLOON_H
 
+#include "hw/virtio-balloon.h"
 #include "cpu-defs.h"
 
-typedef ram_addr_t (QEMUBalloonEvent)(void *opaque, ram_addr_t target);
+struct QEMUBalloonState {
+    ram_addr_t actual;
+    struct virtio_balloon_stats stats;
+};
+
+typedef void (QEMUBalloonEvent)(void *opaque, ram_addr_t target,
+                                      struct QEMUBalloonState *state);
 
 void qemu_add_balloon_handler(QEMUBalloonEvent *func, void *opaque);
 
 void qemu_balloon(ram_addr_t target);
 
-ram_addr_t qemu_balloon_status(void);
+int qemu_balloon_status(struct QEMUBalloonState *s);
 
 #endif
hw/virtio-balloon.c +56 -8

101 lines not on this page · raw message

hw/virtio-balloon.h +21 -0
index 9a0d119..c93909e 100644
@@ -25,16 +25,37 @@
 
 /* The feature bitmap for virtio balloon */
 #define VIRTIO_BALLOON_F_MUST_TELL_HOST 0 /* Tell before reclaiming pages */
+                                        /* Guest memory statistic reporting */
+#define VIRTIO_BALLOON_F_RPT_SWAP_IN  1   /* Number of pages swapped in */
+#define VIRTIO_BALLOON_F_RPT_SWAP_OUT 2   /* Number of pages swapped out */
+#define VIRTIO_BALLOON_F_RPT_ANON     3   /* Number of anonymous pages in use */
+#define VIRTIO_BALLOON_F_RPT_MAJFLT   4   /* Number of major faults */
+#define VIRTIO_BALLOON_F_RPT_MINFLT   5   /* Number of minor faults */
+#define VIRTIO_BALLOON_F_RPT_MEMFREE  6   /* Total amount of free memory */
+#define VIRTIO_BALLOON_F_RPT_MEMTOT   7   /* Total amount of memory */
 
 /* Size of a PFN in the balloon interface. */
 #define VIRTIO_BALLOON_PFN_SHIFT 12
 
+struct virtio_balloon_stats
+{
+    uint32_t pswapin;     /* pages swapped in */
+    uint32_t pswapout;    /* pages swapped out */
+    uint32_t panon;       /* anonymous pages in use */
+    uint32_t pgmajfault;  /* Major page faults */
+    uint32_t pgminfault;  /* Minor page faults */
+    uint32_t memfree;     /* Total amount of free memory (in kb) */
+    uint32_t memtot;      /* Total amount of memory (in kb) */
+};
+
 struct virtio_balloon_config
 {
     /* Number of pages host wants Guest to give up. */
     uint32_t num_pages;
     /* Number of pages we've actually got in balloon. */
     uint32_t actual;
+    /* Memory statistics */
+    struct virtio_balloon_stats stats;
 };
 
 #endif
monitor.c +22 -5
index ed1ce6e..36cff11 100644
@@ -1581,18 +1581,35 @@ static void do_balloon(Monitor *mon, int value)
     qemu_balloon(target << 20);
 }
 
+static inline void print_stat(Monitor *mon, uint32_t val, const char *label,
+                              const char *units)
+{
+    if (val != -1)
+        monitor_printf(mon, "balloon: %s=%u %s\n", label, val, units);
+}
+
 static void do_info_balloon(Monitor *mon)
 {
-    ram_addr_t actual;
+    struct QEMUBalloonState s;
+    int ret;
 
-    actual = qemu_balloon_status();
+    ret = qemu_balloon_status(&s);
     if (kvm_enabled() && !kvm_has_sync_mmu())
         monitor_printf(mon, "Using KVM without synchronous MMU, "
                        "ballooning disabled\n");
-    else if (actual == 0)
+    else if (ret < 0)
         monitor_printf(mon, "Ballooning not activated in VM\n");
-    else
-        monitor_printf(mon, "balloon: actual=%d\n", (int)(actual >> 20));
+    else {
+        monitor_printf(mon, "balloon: actual=%d MB\n", (int)(s.actual >> 20));
+        print_stat(mon, s.stats.pswapin, "pswapin", "pages");
+        print_stat(mon, s.stats.pswapout, "pswapout", "pages");
+        print_stat(mon, s.stats.panon, "panon", "KB");
+        print_stat(mon, s.stats.pgmajfault, "pgmajfault", "");
+        print_stat(mon, s.stats.pgminfault, "pgminfault", "");
+        print_stat(mon, s.stats.memfree, "memfree", "KB");
+        print_stat(mon, s.stats.memtot, "memtot", "KB");
+    }
+        
 }
 
 static qemu_acl *find_acl(Monitor *mon, const char *name)
vl.c +3 -3
index 710d52e..de3b4ae 100644
@@ -334,13 +334,13 @@ void qemu_add_balloon_handler(QEMUBalloonEvent *func, void *opaque)
 void qemu_balloon(ram_addr_t target)
 {
     if (qemu_balloon_event)
-        qemu_balloon_event(qemu_balloon_event_opaque, target);
+        qemu_balloon_event(qemu_balloon_event_opaque, target, NULL);
 }
 
-ram_addr_t qemu_balloon_status(void)
+int qemu_balloon_status(struct QEMUBalloonState *s)
 {
     if (qemu_balloon_event)
-        return qemu_balloon_event(qemu_balloon_event_opaque, 0);
+        qemu_balloon_event(qemu_balloon_event_opaque, 0, s);
     return 0;
 }
 
Signature · 2 lines
Thanks,
Adam
author
#1 lore raw

[Qemu-devel] virtio: Add memory statistics reporting to the balloon driver

Here are the corresponding changes to the Linux virtio driver...

virtio: Add memory statistics reporting to the balloon driver

When using ballooning to manage overcommitted memory on a host, a system for guests to communicate their memory usage to the host can provide information that will minimize the impact of ballooning on the guests. The current method employs a daemon running in each guest that communicates memory statistics to a host daemon at a specified time interval. The host daemon aggregates this information and inflates and/or deflates balloons according to the level of host memory pressure. This approach is effective but overly complex since a daemon must be installed inside each guest and coordinated to communicate with the host. A simpler approach is to collect memory statistics in the virtio balloon driver and communicate them to the host via the device config space.

This patch enables the guest-side support by adding stats collection and reporting to the virtio balloon driver.

drivers/virtio/virtio.c +1 -0
index 3a43ebf..1029363 100644
@@ -135,6 +135,7 @@ static int virtio_dev_probe(struct device *_d)
 			set_bit(i, dev->features);
 
 	dev->config->finalize_features(dev);
+	printk("virtio_dev_probe: final features = %lx\n", dev->features[0]);
 
 	err = drv->probe(dev);
 	if (err)
drivers/virtio/virtio_balloon.c +51 -3

85 lines not on this page · raw message

include/linux/virtio_balloon.h +24 -0
index 09d7300..0bff4b8 100644
@@ -6,15 +6,39 @@
 
 /* The feature bitmap for virtio balloon */
 #define VIRTIO_BALLOON_F_MUST_TELL_HOST	0 /* Tell before reclaiming pages */
+                                          /* Guest memory statistic reporting */
+#define VIRTIO_BALLOON_F_RPT_SWAP_IN  1   /* Number of pages swapped in */
+#define VIRTIO_BALLOON_F_RPT_SWAP_OUT 2   /* Number of pages swapped out */
+#define VIRTIO_BALLOON_F_RPT_ANON     3   /* Number of anonymous pages in use */
+#define VIRTIO_BALLOON_F_RPT_MAJFLT   4   /* Number of major faults */
+#define VIRTIO_BALLOON_F_RPT_MINFLT   5   /* Number of minor faults */
+#define VIRTIO_BALLOON_F_RPT_MEMFREE  6   /* Total amount of free memory */
+#define VIRTIO_BALLOON_F_RPT_MEMTOT   7   /* Total amount of memory */
 
 /* Size of a PFN in the balloon interface. */
 #define VIRTIO_BALLOON_PFN_SHIFT 12
 
+struct virtio_balloon_stats
+{
+	__le32 pswapin;      /* pages swapped in */
+	__le32 pswapout;     /* pages swapped out */
+	__le32 panon;        /* anonymous pages in use (in kb) */
+	__le32 pgmajfault;   /* Major page faults */
+	__le32 pgminfault;   /* Minor page faults */
+	__le32 memfree;      /* Total amount of free memory (in kb) */
+	__le32 memtot;       /* Total amount of memory (in kb) */
+};
+
 struct virtio_balloon_config
 {
 	/* Number of pages host wants Guest to give up. */
 	__le32 num_pages;
 	/* Number of pages we've actually got in balloon. */
 	__le32 actual;
+	/* Memory statistics */
+	struct virtio_balloon_stats stats;
 };
+
+#define VIRTIO_BALLOON_TIMEOUT (30 * HZ)
+
 #endif /* _LINUX_VIRTIO_BALLOON_H */
Signature · 2 lines
Thanks,
Adam
#2 lore raw

[Qemu-devel] Re: virtio: Add memory statistics reporting to the balloon driver

Quoted 25 lines · [email protected] wrote:

Here are the corresponding changes to the Linux virtio driver...

virtio: Add memory statistics reporting to the balloon driver

When using ballooning to manage overcommitted memory on a host, a system for guests to communicate their memory usage to the host can provide information that will minimize the impact of ballooning on the guests. The current method employs a daemon running in each guest that communicates memory statistics to a host daemon at a specified time interval. The host daemon aggregates this information and inflates and/or deflates balloons according to the level of host memory pressure. This approach is effective but overly complex since a daemon must be installed inside each guest and coordinated to communicate with the host. A simpler approach is to collect memory statistics in the virtio balloon driver and communicate them to the host via the device config space.

This patch enables the guest-side support by adding stats collection and reporting to the virtio balloon driver.

drivers/virtio/virtio.c +0 -0
index 3a43ebf..1029363 100644

drivers/virtio/virtio.c static int virtio_dev_probe(struct device *_d) · line 139

Quoted 5 lines
drivers/virtio/virtio.c +1 -0
@@ -135,6 +135,7 @@ static int virtio_dev_probe(struct device *_d)
 			set_bit(i, dev->features);
 
 	dev->config->finalize_features(dev);
+	printk("virtio_dev_probe: final features = %lx\n", dev->features[0]);
 

Looks like leftover debugging.

Quoted 7 lines
drivers/virtio/virtio.c +0 -0
 	err = drv->probe(dev);
 	if (err)
drivers/virtio/virtio_balloon.c +0 -0
index 200c22f..77cb953 100644

drivers/virtio/virtio_balloon.c static void update_balloon_size(struct virtio_balloon *vb) · line 188

Quoted 9 lines
drivers/virtio/virtio_balloon.c +5 -0
@@ -180,6 +180,45 @@ static void update_balloon_size(struct virtio_balloon *vb)
 			      &actual, sizeof(actual));
 }
 
+static inline void update_stat(struct virtio_device *vdev, int feature,
+				unsigned long value, unsigned offset)
+{
+	if (virtio_has_feature(vdev, feature)) {
+		vdev->config->set(vdev, offset, &value, sizeof(value));
 

I think this bit assumes a little endian guest. We shouldn't make that assumption.

For virtio kernel patches, please CC the virtualization list and Rusty as he's the maintainer. It wouldn't hurt to CC lkml either.

Signature · 3 lines
Regards,

Anthony Liguori
#3 lore raw
In reply to Adam Litke
Quoted 13 lines · [email protected] wrote:

[RFC] virtio: Report new guest memory statistics pertinent to memory ballooning

When using ballooning to manage overcommitted memory on a host, a system for guests to communicate their memory usage to the host can provide information that will minimize the impact of ballooning on the guests. The current method employs a daemon running in each guest that communicates memory statistics to a host daemon at a specified time interval. The host daemon aggregates this information and inflates and/or deflates balloons according to the level of host memory pressure. This approach is effective but overly complex since a daemon must be installed inside each guest and coordinated to communicate with the host. A simpler approach is to collect memory statistics in the virtio balloon driver and communicate them to the host via the device config space.

Quoted 7 lines

This patch implements the qemu side of the communication channel. I will post the kernel driver modifications in-reply to this message.

I'd like to pose a few questions concerning my implementation:

* Is there a better way to use feature codes than using one per variable?

I think it's the right granularity personally.

Quoted 4 lines
     * This patch causes the 'info balloon' monitor command to generate output like
       the following:
    
    	(qemu) info balloon
Quoted 9 lines
    	balloon: actual=1024 MB
    	balloon: pswapin=0 pages
    	balloon: pswapout=0 pages
    	balloon: panon=3928 KB
    	balloon: pgmajfault=0
    	balloon: pgminfault=247914
    	balloon: memfree=987032 KB
    	balloon: memtot=1020812 KB

We could probably use less obscure names for these stats like total_memory. I know we're facing a global shortage of bits but I think this is excessive conservation :-)

We typically print these out as:

balloon: actual=1024,pswapin=0,pswapout=0,panon=3928...

It's a bit less readable admittedly, but it's how things tend to work.

Quoted 7 lines

Is this agreeable?

Thank you for your comments...

The indent is pretty strange here. git-send-email should help.

Quoted 7 lines
balloon.h +0 -0
index 60b4a5d..3f67021 100644
@@ -14,14 +14,21 @@
 #ifndef _QEMU_BALLOON_H
 #define _QEMU_BALLOON_H

balloon.h line 24

Quoted 10 lines
balloon.h +5 -1
 
+#include "hw/virtio-balloon.h"
 #include "cpu-defs.h"
 
-typedef ram_addr_t (QEMUBalloonEvent)(void *opaque, ram_addr_t target);
+struct QEMUBalloonState {
+    ram_addr_t actual;
+    struct virtio_balloon_stats stats;
+};
 

We like to typedef the struct away from structures in qemu. See CodingStyle.

The rest looks good.

Signature · 3 lines
Regards,

Anthony Liguori
#4 lore raw
In reply to Adam Litke
Quoted 33 lines · Adam Litke

[RFC] virtio: Report new guest memory statistics pertinent to memory ballooning

When using ballooning to manage overcommitted memory on a host, a system for guests to communicate their memory usage to the host can provide information that will minimize the impact of ballooning on the guests. The current method employs a daemon running in each guest that communicates memory statistics to a host daemon at a specified time interval. The host daemon aggregates this information and inflates and/or deflates balloons according to the level of host memory pressure. This approach is effective but overly complex since a daemon must be installed inside each guest and coordinated to communicate with the host. A simpler approach is to collect memory statistics in the virtio balloon driver and communicate them to the host via the device config space.

This patch implements the qemu side of the communication channel. I will post the kernel driver modifications in-reply to this message.

I'd like to pose a few questions concerning my implementation:

* Is there a better way to use feature codes than using one per variable?

* This patch causes the 'info balloon' monitor command to generate output like the following:

(qemu) info balloon
balloon: actual=1024 MB
balloon: pswapin=0 pages
balloon: pswapout=0 pages
balloon: panon=3928 KB
balloon: pgmajfault=0
balloon: pgminfault=247914
balloon: memfree=987032 KB
balloon: memtot=1020812 KB

Quoted 2 lines

Is this agreeable?

It's important that these statistics be cross platform. Can you (or someone knowledgeable in Windows) verify that all of these are meaningful there? I'd expect it to be so with the possible exception of panon.

Can you explain how the host would use these? I can see how major faults are interesting, but how is the minor faults statistic useful?

Normally I'd ask that the values be in machine readable form, but with the machine monitor protocol, that's not necessary.

Signature · 1 line
error compiling committee.c: too many arguments to function
#5 lore raw
Quoted 10 lines · Avi Kivity
Quoted 9 lines
    	(qemu) info balloon
    	balloon: actual=1024 MB
    	balloon: pswapin=0 pages
    	balloon: pswapout=0 pages
    	balloon: panon=3928 KB
    	balloon: pgmajfault=0
    	balloon: pgminfault=247914
    	balloon: memfree=987032 KB
    	balloon: memtot=1020812 KB
Quoted 4 lines

It's important that these statistics be cross platform. Can you (or someone knowledgeable in Windows) verify that all of these are meaningful there? I'd expect it to be so with the possible exception of panon.

pswapin, pswapout, pgmajfault and pgminfault are not meaningful even on some Linux targets (those without MMU, and therefore no page faults). Yet even those could, perhaps, use virtio-balloon.

It's probably best to have a way to omit values that are not meaningful to a guest, or otherwise indicate "not defined by the current guest".

-- Jamie

#6 lore raw
In reply to Avi Kivity
Quoted 12 lines · Avi Kivity
Quoted 12 lines

         (qemu) info balloon
         balloon: actual=1024 MB
         balloon: pswapin=0 pages
         balloon: pswapout=0 pages
         balloon: panon=3928 KB
         balloon: pgmajfault=0
         balloon: pgminfault=247914
         balloon: memfree=987032 KB
         balloon: memtot=1020812 KB
Quoted 4 lines
Quoted 2 lines

Is this agreeable?

It's important that these statistics be cross platform. Can you (or someone knowledgeable in Windows) verify that all of these are meaningful there? I'd expect it to be so with the possible exception of panon.

They don't have to all be cross platform. We can expose platform specific metrics via feature bits. We should only disable features that the guest has acknowledged.

But of course, whenever possible, we want metrics to be meaningful cross platform.

Signature · 3 lines
Regards,

Anthony Liguori