Skip to contentSkip to conversation

xen-blkback: use balloon pages for persistent grants

PATCH In review Roger Pau Monne 2 participants 9 messages Feb 14, 2013 – Feb 15, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xen-blkback: use balloon pages for persistent grants

With current persistent grants implementation we are not freeing the persistent grants after we disconnect the device. Since grant map operations change the mfn of the allocated page, and we can no longer pass it to __free_page without setting the mfn to a sane value, use balloon grant pages instead, as the gntdev device does.

---
 drivers/block/xen-blkback/blkback.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)
drivers/block/xen-blkback/blkback.c +4 -2
index c46824f..e6c2f6a 100644
@@ -46,6 +46,7 @@
 #include <xen/xen.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <xen/balloon.h>
 #include "common.h"
 
 /*
@@ -239,6 +240,7 @@ static void free_persistent_gnts(struct rb_root *root, unsigned int num)
 			ret = gnttab_unmap_refs(unmap, NULL, pages,
 				segs_to_unmap);
 			BUG_ON(ret);
+			free_xenballooned_pages(segs_to_unmap, pages);
 			segs_to_unmap = 0;
 		}
 
@@ -527,8 +529,8 @@ static int xen_blkbk_map(struct blkif_request *req,
 				GFP_KERNEL);
 			if (!persistent_gnt)
 				return -ENOMEM;
-			persistent_gnt->page = alloc_page(GFP_KERNEL);
-			if (!persistent_gnt->page) {
+			if (alloc_xenballooned_pages(1, &persistent_gnt->page,
+			    false)) {
 				kfree(persistent_gnt);
 				return -ENOMEM;
 			}
Signature · 1 line
1.7.7.5 (Apple Git-26)
#1 lore raw
Quoted 9 lines · Roger Pau Monne

With current persistent grants implementation we are not freeing the persistent grants after we disconnect the device. Since grant map operations change the mfn of the allocated page, and we can no longer pass it to __free_page without setting the mfn to a sane value, use balloon grant pages instead, as the gntdev device does.

I think this patch is missing the following change in Kconfig, but gntdev doesn't depend on the balloon driver, which it also uses, so I'm not sure.

---

drivers/block/Kconfig +1 -1
index f529407..74e2415 100644
@@ -488,7 +488,7 @@ config XEN_BLKDEV_FRONTEND
 
 config XEN_BLKDEV_BACKEND
 	tristate "Xen block-device backend driver"
-	depends on XEN_BACKEND
+	depends on XEN_BACKEND && XEN_BALLOON
 	help
 	  The block-device backend driver allows the kernel to export its
 	  block devices to other guests via a high-performance shared-memory
#2 lore raw
In reply to Roger Pau Monne
Quoted 9 lines · Roger Pau Monne

With current persistent grants implementation we are not freeing the persistent grants after we disconnect the device. Since grant map operations change the mfn of the allocated page, and we can no longer pass it to __free_page without setting the mfn to a sane value, use balloon grant pages instead, as the gntdev device does.

I think this patch is missing the following change in Kconfig, but gntdev doesn't depend on the balloon driver, which it also uses, so I'm not sure.

---

drivers/block/Kconfig +1 -1
index f529407..74e2415 100644
@@ -488,7 +488,7 @@ config XEN_BLKDEV_FRONTEND
 
 config XEN_BLKDEV_BACKEND
 	tristate "Xen block-device backend driver"
-	depends on XEN_BACKEND
+	depends on XEN_BACKEND && XEN_BALLOON
 	help
 	  The block-device backend driver allows the kernel to export its
 	  block devices to other guests via a high-performance shared-memory

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#3 lore raw
In reply to Roger Pau Monne
Quoted 2 lines · Roger Pau Monne

With current persistent grants implementation we are not freeing the persistent grants after we disconnect the device. Since grant map

Can you explain this in more details please? Isn't gnttab_set_unmap_op in free_persistent_gnts doing the right job of putting in the right mfn in? And then we could free the page?

Quoted 3 lines

operations change the mfn of the allocated page, and we can no longer pass it to __free_page without setting the mfn to a sane value, use balloon grant pages instead, as the gntdev device does.

Wow. I did not realize that we leaving such a huge memory leak behind! But I guess that was never an issue as we would recycle those persistent grants to other domains.

Quoted 42 lines
---
 drivers/block/xen-blkback/blkback.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)
drivers/block/xen-blkback/blkback.c +4 -2
index c46824f..e6c2f6a 100644
@@ -46,6 +46,7 @@
 #include <xen/xen.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <xen/balloon.h>
 #include "common.h"
 
 /*
@@ -239,6 +240,7 @@ static void free_persistent_gnts(struct rb_root *root, unsigned int num)
 			ret = gnttab_unmap_refs(unmap, NULL, pages,
 				segs_to_unmap);
 			BUG_ON(ret);
+			free_xenballooned_pages(segs_to_unmap, pages);
 			segs_to_unmap = 0;
 		}
 
@@ -527,8 +529,8 @@ static int xen_blkbk_map(struct blkif_request *req,
 				GFP_KERNEL);
 			if (!persistent_gnt)
 				return -ENOMEM;
-			persistent_gnt->page = alloc_page(GFP_KERNEL);
-			if (!persistent_gnt->page) {
+			if (alloc_xenballooned_pages(1, &persistent_gnt->page,
+			    false)) {
 				kfree(persistent_gnt);
 				return -ENOMEM;
 			}
Signature · 1 line
1.7.7.5 (Apple Git-26)
#4 lore raw
In reply to Roger Pau Monne
Quoted 2 lines · Roger Pau Monne

With current persistent grants implementation we are not freeing the persistent grants after we disconnect the device. Since grant map

Can you explain this in more details please? Isn't gnttab_set_unmap_op in free_persistent_gnts doing the right job of putting in the right mfn in? And then we could free the page?

Quoted 3 lines

operations change the mfn of the allocated page, and we can no longer pass it to __free_page without setting the mfn to a sane value, use balloon grant pages instead, as the gntdev device does.

Wow. I did not realize that we leaving such a huge memory leak behind! But I guess that was never an issue as we would recycle those persistent grants to other domains.

Quoted 42 lines
---
 drivers/block/xen-blkback/blkback.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)
drivers/block/xen-blkback/blkback.c +4 -2
index c46824f..e6c2f6a 100644
@@ -46,6 +46,7 @@
 #include <xen/xen.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <xen/balloon.h>
 #include "common.h"
 
 /*
@@ -239,6 +240,7 @@ static void free_persistent_gnts(struct rb_root *root, unsigned int num)
 			ret = gnttab_unmap_refs(unmap, NULL, pages,
 				segs_to_unmap);
 			BUG_ON(ret);
+			free_xenballooned_pages(segs_to_unmap, pages);
 			segs_to_unmap = 0;
 		}
 
@@ -527,8 +529,8 @@ static int xen_blkbk_map(struct blkif_request *req,
 				GFP_KERNEL);
 			if (!persistent_gnt)
 				return -ENOMEM;
-			persistent_gnt->page = alloc_page(GFP_KERNEL);
-			if (!persistent_gnt->page) {
+			if (alloc_xenballooned_pages(1, &persistent_gnt->page,
+			    false)) {
 				kfree(persistent_gnt);
 				return -ENOMEM;
 			}
Signature · 1 line
1.7.7.5 (Apple Git-26)
#5 lore raw
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 2 lines · Roger Pau Monne

With current persistent grants implementation we are not freeing the persistent grants after we disconnect the device. Since grant map

Can you explain this in more details please? Isn't gnttab_set_unmap_op in free_persistent_gnts doing the right job of putting in the right mfn in? And then we could free the page?

No, after gnttab_unmap_refs the page still points to the grant frame mfn. All the users of grant frames either have an internal buffer of pages that are reused (like blkback), or use balloon pages (like gntdev).

I could probably modify gnttab_map to store the mfn and set it back at gnttab_unmap, but that will surely require more work than this simple fix.

Quoted 10 lines
Quoted 5 lines

operations change the mfn of the allocated page, and we can no longer pass it to __free_page without setting the mfn to a sane value, use balloon grant pages instead, as the gntdev device does.

Wow. I did not realize that we leaving such a huge memory leak behind! But I guess that was never an issue as we would recycle those persistent grants to other domains.

No, we didn't recycle them AFAIK, we allocated them using alloc_page, passed them to gnttab_map, used them, and when closing the backend we only unmapped them, but they where never freed.

Quoted 42 lines
Quoted 42 lines
---
 drivers/block/xen-blkback/blkback.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)
drivers/block/xen-blkback/blkback.c +4 -2
index c46824f..e6c2f6a 100644
@@ -46,6 +46,7 @@
 #include <xen/xen.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <xen/balloon.h>
 #include "common.h"
 
 /*
@@ -239,6 +240,7 @@ static void free_persistent_gnts(struct rb_root *root, unsigned int num)
 			ret = gnttab_unmap_refs(unmap, NULL, pages,
 				segs_to_unmap);
 			BUG_ON(ret);
+			free_xenballooned_pages(segs_to_unmap, pages);
 			segs_to_unmap = 0;
 		}
 
@@ -527,8 +529,8 @@ static int xen_blkbk_map(struct blkif_request *req,
 				GFP_KERNEL);
 			if (!persistent_gnt)
 				return -ENOMEM;
-			persistent_gnt->page = alloc_page(GFP_KERNEL);
-			if (!persistent_gnt->page) {
+			if (alloc_xenballooned_pages(1, &persistent_gnt->page,
+			    false)) {
 				kfree(persistent_gnt);
 				return -ENOMEM;
 			}
Signature · 1 line
1.7.7.5 (Apple Git-26)
#6 lore raw
Quoted 9 lines · Roger Pau Monné
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 2 lines · Roger Pau Monne

With current persistent grants implementation we are not freeing the persistent grants after we disconnect the device. Since grant map

Can you explain this in more details please? Isn't gnttab_set_unmap_op in free_persistent_gnts doing the right job of putting in the right mfn in? And then we could free the page?

Quoted 6 lines

No, after gnttab_unmap_refs the page still points to the grant frame mfn. All the users of grant frames either have an internal buffer of pages that are reused (like blkback), or use balloon pages (like gntdev).

I could probably modify gnttab_map to store the mfn and set it back at gnttab_unmap, but that will surely require more work than this simple fix.

Simple fix is what we want right now.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

operations change the mfn of the allocated page, and we can no longer pass it to __free_page without setting the mfn to a sane value, use balloon grant pages instead, as the gntdev device does.

Quoted 7 lines
Quoted 3 lines

Wow. I did not realize that we leaving such a huge memory leak behind! But I guess that was never an issue as we would recycle those persistent grants to other domains.

No, we didn't recycle them AFAIK, we allocated them using alloc_page, passed them to gnttab_map, used them, and when closing the backend we only unmapped them, but they where never freed.

Ghastly!

#7 lore raw
In reply to Roger Pau Monné
Quoted 9 lines · Roger Pau Monné
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 2 lines · Roger Pau Monne

With current persistent grants implementation we are not freeing the persistent grants after we disconnect the device. Since grant map

Can you explain this in more details please? Isn't gnttab_set_unmap_op in free_persistent_gnts doing the right job of putting in the right mfn in? And then we could free the page?

Quoted 6 lines

No, after gnttab_unmap_refs the page still points to the grant frame mfn. All the users of grant frames either have an internal buffer of pages that are reused (like blkback), or use balloon pages (like gntdev).

I could probably modify gnttab_map to store the mfn and set it back at gnttab_unmap, but that will surely require more work than this simple fix.

Simple fix is what we want right now.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

operations change the mfn of the allocated page, and we can no longer pass it to __free_page without setting the mfn to a sane value, use balloon grant pages instead, as the gntdev device does.

Quoted 7 lines
Quoted 3 lines

Wow. I did not realize that we leaving such a huge memory leak behind! But I guess that was never an issue as we would recycle those persistent grants to other domains.

No, we didn't recycle them AFAIK, we allocated them using alloc_page, passed them to gnttab_map, used them, and when closing the backend we only unmapped them, but they where never freed.

Ghastly!

#8 lore raw
In reply to Konrad Rzeszutek Wilk
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 2 lines · Roger Pau Monne

With current persistent grants implementation we are not freeing the persistent grants after we disconnect the device. Since grant map

Can you explain this in more details please? Isn't gnttab_set_unmap_op in free_persistent_gnts doing the right job of putting in the right mfn in? And then we could free the page?

No, after gnttab_unmap_refs the page still points to the grant frame mfn. All the users of grant frames either have an internal buffer of pages that are reused (like blkback), or use balloon pages (like gntdev).

I could probably modify gnttab_map to store the mfn and set it back at gnttab_unmap, but that will surely require more work than this simple fix.

Quoted 10 lines
Quoted 5 lines

operations change the mfn of the allocated page, and we can no longer pass it to __free_page without setting the mfn to a sane value, use balloon grant pages instead, as the gntdev device does.

Wow. I did not realize that we leaving such a huge memory leak behind! But I guess that was never an issue as we would recycle those persistent grants to other domains.

No, we didn't recycle them AFAIK, we allocated them using alloc_page, passed them to gnttab_map, used them, and when closing the backend we only unmapped them, but they where never freed.

Quoted 42 lines
Quoted 42 lines
---
 drivers/block/xen-blkback/blkback.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)
drivers/block/xen-blkback/blkback.c +4 -2
index c46824f..e6c2f6a 100644
@@ -46,6 +46,7 @@
 #include <xen/xen.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <xen/balloon.h>
 #include "common.h"
 
 /*
@@ -239,6 +240,7 @@ static void free_persistent_gnts(struct rb_root *root, unsigned int num)
 			ret = gnttab_unmap_refs(unmap, NULL, pages,
 				segs_to_unmap);
 			BUG_ON(ret);
+			free_xenballooned_pages(segs_to_unmap, pages);
 			segs_to_unmap = 0;
 		}
 
@@ -527,8 +529,8 @@ static int xen_blkbk_map(struct blkif_request *req,
 				GFP_KERNEL);
 			if (!persistent_gnt)
 				return -ENOMEM;
-			persistent_gnt->page = alloc_page(GFP_KERNEL);
-			if (!persistent_gnt->page) {
+			if (alloc_xenballooned_pages(1, &persistent_gnt->page,
+			    false)) {
 				kfree(persistent_gnt);
 				return -ENOMEM;
 			}
Signature · 1 line
1.7.7.5 (Apple Git-26)