Drivers: hv: balloon
Subsystems (3)
v2 cover Drivers: hv: balloon
Support 2M page allocations when memory is ballooned out of the guest. Hyper-V Dynamic Memory protocol is optimized around the ability to move memory in 2M chunks.
I have also included a patch to properly notify the host of permanent hot-add failures.
In this version of the patches, I have added some additional comments to the code and the patch descriptions.
K. Y. Srinivasan (3):
mm: Export split_page()
Drivers: hv: balloon: Support 2M page allocations for ballooning
Drivers: hv: Notify the host of permanent hot-add failures
drivers/hv/hv_balloon.c | 51 +++++++++++++++++++++++++++++++++++++++++++--- mm/page_alloc.c | 1 + 2 files changed, 48 insertions(+), 4 deletions(-)
Signature · 7 lines
1.7.4.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 1/3 mm: Export split_page()
This symbol would be used in the Hyper-V balloon driver to support 2M allocations.
In this version of the patch, based on feedback from Michal Hocko <[email protected]>, I have updated the patch description.
--- mm/page_alloc.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
mm/page_alloc.c +1 -0
@@ -1404,6 +1404,7 @@ void split_page(struct page *page, unsigned int order)for (i =1 ; i < (1 << order); i++)set_page_refcounted (page + i); } +EXPORT_SYMBOL_GPL (split_page);static int __isolate_free_page (struct page *page,unsigned int order) {
Signature · 7 lines
1.7.4.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 2/3 Drivers: hv: balloon: Support 2M page allocations for ballooning
On Hyper-V it will be very efficient to use 2M allocations in the guest as this makes the ballooning protocol with the host that much more efficient. Hyper-V uses page ranges (start pfn : number of pages) to specify memory being moved around and with 2M pages this encoding can be very efficient. However, when memory is returned to the guest, the host does not guarantee any granularity. To deal with this issue, split the page soon after a successful 2M allocation so that this memory can potentially be freed as 4K pages.
If 2M allocations fail, we revert to 4K allocations.
In this version of the patch, based on the feedback from Michal Hocko <[email protected]>, I have added some additional commentary to the patch description.
--- drivers/hv/hv_balloon.c | 18 ++++++++++++++++-- 1 files changed, 16 insertions(+), 2 deletions(-)
drivers/hv/hv_balloon.c +16 -2
@@ -997,6 +997,14 @@ static int alloc_balloon_pages(struct hv_dynmem_device *dm, int num_pages, dm->num_pages_ballooned += alloc_unit; +/* +* If we allocatted 2M pages; split them so we +* can free them in any order we get. +*/ + +if (alloc_unit !=1 ) +split_page (pg,get_order (alloc_unit <<PAGE_SHIFT )); + bl_resp->range_count++; bl_resp->range_array[i].finfo.start_page =page_to_pfn (pg);
@@ -1023,9 +1031,10 @@ static void balloon_up(struct work_struct *dummy)/* -* Currently, we only support 4k allocations. +* We will attempt 2M allocations. However, if we fail to +* allocate 2M chunks, we will go back to 4k allocations. */ - alloc_unit =1 ; + alloc_unit =512 ;while (!done) { bl_resp = (struct dm_balloon_response *)send_buffer;
@@ -1041,6 +1050,11 @@ static void balloon_up(struct work_struct *dummy) bl_resp, alloc_unit, &alloc_error); +if ((alloc_error) && (alloc_unit !=1 )) { + alloc_unit =1 ; +continue ; + } +if ((alloc_error) || (num_ballooned == num_pages)) { bl_resp->more_pages =0 ; done = true;
Signature · 7 lines
1.7.4.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH V2 2/3] Drivers: hv: balloon: Support 2M page allocations for ballooning
Quoted 7 lines · K. Y. Srinivasan
On Hyper-V it will be very efficient to use 2M allocations in the guest as this makes the ballooning protocol with the host that much more efficient. Hyper-V uses page ranges (start pfn : number of pages) to specify memory being moved around and with 2M pages this encoding can be very efficient. However, when memory is returned to the guest, the host does not guarantee any granularity. To deal with this issue, split the page soon after a successful 2M allocation so that this memory can potentially be freed as 4K pages.
How many pages are requested usually?
Quoted 7 lines
If 2M allocations fail, we revert to 4K allocations.
In this version of the patch, based on the feedback from Michal Hocko <[email protected]>, I have added some additional commentary to the patch description.
Signed-off-by K. Y. Srinivasan <[email protected]>
I am not going to ack the patch because I am still not entirely convinced that big allocations are worth it. But that is up to you and hyper-V users.
Quoted 13 lines
--- drivers/hv/hv_balloon.c | 18 ++++++++++++++++-- 1 files changed, 16 insertions(+), 2 deletions(-)drivers/hv/hv_balloon.c +1
-0@@ -997,6 +997,14 @@ static int alloc_balloon_pages(struct hv_dynmem_device *dm, int num_pages, dm->num_pages_ballooned += alloc_unit; +/*
drivers/hv/hv_balloon.c static int alloc_balloon_pages(struct hv_dynmem_device *dm, int num_pages, · line 1010
Quoted 10 lines
drivers/hv/hv_balloon.c +7
-0+* If we allocatted 2M pages; split them so we +* can free them in any order we get. +*/ + +if (alloc_unit !=1 ) +split_page (pg,get_order (alloc_unit <<PAGE_SHIFT )); + bl_resp->range_count++; bl_resp->range_array[i].finfo.start_page =page_to_pfn (pg);
I would suggest also using __GFP_NO_KSWAPD (or basically use GFP_TRANSHUGE for alloc_unit>0) for the allocation to be as least disruptive as possible.
Quoted 14 lines
drivers/hv/hv_balloon.c +3
-2@@ -1023,9 +1031,10 @@ static void balloon_up(struct work_struct *dummy)/* -* Currently, we only support 4k allocations. +* We will attempt 2M allocations. However, if we fail to +* allocate 2M chunks, we will go back to 4k allocations. */ - alloc_unit =1 ; + alloc_unit =512 ;while (!done) { bl_resp = (struct dm_balloon_response *)send_buffer;
drivers/hv/hv_balloon.c static void balloon_up(struct work_struct *dummy) · line 1052
Quoted 3 lines
drivers/hv/hv_balloon.c +0
-0@@ -1041,6 +1050,11 @@ static void balloon_up(struct work_struct *dummy) bl_resp, alloc_unit, &alloc_error);
You should handle alloc_balloon_pages returns 0 && !alloc_error which happens when num_pages < alloc_unit.
Quoted 16 lines
drivers/hv/hv_balloon.c +5
-0+if ((alloc_error) && (alloc_unit !=1 )) { + alloc_unit =1 ; +continue ; + } +if ((alloc_error) || (num_ballooned == num_pages)) { bl_resp->more_pages =0 ; done = true;Signature · 7 lines
1.7.4.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RE: [PATCH V2 2/3] Drivers: hv: balloon: Support 2M page allocations for ballooning
Quoted 22 lines
-----Original Message-----
From: Michal Hocko [mailto:[email protected]]
Sent: Tuesday, March 19, 2013 10:46 AM
To: KY Srinivasan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V2 2/3] Drivers: hv: balloon: Support 2M page allocations for ballooningQuoted 7 lines · K. Y. Srinivasan
On Hyper-V it will be very efficient to use 2M allocations in the guest as this makes the ballooning protocol with the host that much more efficient. Hyper-V uses page ranges (start pfn : number of pages) to specify memory being moved around and with 2M pages this encoding can be very efficient. However, when memory is returned to the guest, the host does not guarantee any granularity. To deal with this issue, split the page soon after a successful 2M allocation so that this memory can potentially be freed as 4K pages.
Quoted 1 line
How many pages are requested usually?
This depends entirely on how many pages the guest has, the pressure reported by the guest and the overall memory demand as perceived by the host. On idling guests that have been configured with several Giga bytes of memory, I have seen several Giga bytes being ballooned out of the guest as soon as new VMs are started or pressure goes up in an existing VM. In these cases, if 2M allocations succeed, the ballooning operation can be done very efficiently.
Quoted 7 lines
Quoted 7 lines
If 2M allocations fail, we revert to 4K allocations.
In this version of the patch, based on the feedback from Michal Hocko <[email protected]>, I have added some additional commentary to the patch description.
Quoted 5 lines
Quoted 1 line
Signed-off-by K. Y. Srinivasan <[email protected]>I am not going to ack the patch because I am still not entirely convinced that big allocations are worth it. But that is up to you and hyper-V users.
As I said, Hyper-V has chosen 2M unit as the unit that will be most efficient in moving memory around. All I am doing is making Linux participate in this protocol just as efficiently as Windows guests do.
Regards,
K. Y
Quoted 72 lines
drivers/hv/hv_balloon.c static int alloc_balloon_pages(struct
Quoted 10 lines
--- drivers/hv/hv_balloon.c | 18 ++++++++++++++++-- 1 files changed, 16 insertions(+), 2 deletions(-)drivers/hv/hv_balloon.c +0
-0@@ -997,6 +997,14 @@ static int alloc_balloon_pages(struct
hv_dynmem_device*dm, intnum_pages,Quoted 4 lines
drivers/hv/hv_balloon.c +1
-0dm->num_pages_ballooned += alloc_unit; +/* drivers/hv/hv_balloon.c static int alloc_balloon_pages(struct hv_dynmem_device *dm, int num_pages, · line 1010
Quoted 10 lines
drivers/hv/hv_balloon.c +7
-0+* If we allocatted 2M pages; split them so we +* can free them in any order we get. +*/ + +if (alloc_unit !=1 ) +split_page (pg,get_order (alloc_unit <<PAGE_SHIFT )); + bl_resp->range_count++; bl_resp->range_array[i].finfo.start_page =page_to_pfn (pg);I would suggest also using
__GFP_NO_KSWAPD(or basically useGFP_TRANSHUGEfor alloc_unit>0) for the allocation to be as least disruptive as possible.Quoted 14 lines
drivers/hv/hv_balloon.c +3
-2@@ -1023,9 +1031,10 @@ static void balloon_up(struct work_struct *dummy)/* -* Currently, we only support 4k allocations. +* We will attempt 2M allocations. However, if we fail to +* allocate 2M chunks, we will go back to 4k allocations. */ - alloc_unit =1 ; + alloc_unit =512 ;while (!done) { bl_resp = (struct dm_balloon_response *)send_buffer;drivers/hv/hv_balloon.c static void balloon_up(struct work_struct *dummy) · line 1052
Quoted 3 lines
drivers/hv/hv_balloon.c +0
-0@@ -1041,6 +1050,11 @@ static void balloon_up(struct work_struct *dummy) bl_resp, alloc_unit, &alloc_error);You should handle
alloc_balloon_pagesreturns 0 && !alloc_error which happens whennum_pages<alloc_unit.Quoted 11 lines
drivers/hv/hv_balloon.c +5
-1+if ((alloc_error) && (alloc_unit !=1 )) { + alloc_unit =1 ; +continue ; + } +if ((alloc_error) || (num_ballooned == num_pages)) { bl_resp->more_pages =0 ; done = true; --1.7.4.1
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/--
Michal Hocko
SUSE Labs
PATCH v2 3/3 Drivers: hv: Notify the host of permanent hot-add failures
In reply to K. Y. SrinivasanIf memory hot-add fails with the error -EEXIST, then this is a permanent failure. Notify the host of this information, so the host will not attempt hot-add again. If the failure were a transient failure, host will attempt a hot-add after some delay.
In this version of the patch, I have added some additional comments to clarify how the host treats different failure conditions.
--- drivers/hv/hv_balloon.c | 33 +++++++++++++++++++++++++++++++-- 1 files changed, 31 insertions(+), 2 deletions(-)
drivers/hv/hv_balloon.c +31 -2
@@ -583,6 +583,16 @@ static void hv_mem_hot_add(unsigned long start, unsigned long size,if (ret) {pr_info ("hot_add memory failed error is %d\n" , ret); +if (ret == -EEXIST ) { +/* +* This error indicates that the error +* is not a transient failure. This is the +* case where the guest's physical address map +* precludes hot adding memory. Stop all further +* memory hot-add. +*/ + do_hot_add = false; + } has->ha_end_pfn -=HA_CHUNK ; has->covered_end_pfn -= processed_pfn;break ;
@@ -842,11 +852,30 @@ static void hot_add_req(struct work_struct *dummy) rg_sz = region_size; } - resp.page_count =process_hot_add (pg_start, pfn_cnt, - rg_start, rg_sz); +if (do_hot_add) + resp.page_count =process_hot_add (pg_start, pfn_cnt, + rg_start, rg_sz);#endif +/* +* The result field of the response structure has the +* following semantics: +* +* 1. If all or some pages hot-added: Guest should return success. +* +* 2. If no pages could be hot-added: +* +* If the guest returns success, then the host +* will not attempt any further hot-add operations. This +* signifies a permanent failure. +* +* If the guest returns failure, then this failure will be +* treated as a transient failure and the host may retry the +* hot-add operation after some delay. +*/ if (resp.page_count >0 ) resp.result =1 ; +else if (!do_hot_add) + resp.result =1 ;else resp.result =0 ;
Signature · 7 lines
1.7.4.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH V2 1/3] mm: Export split_page()
In reply to K. Y. SrinivasanQuoted 5 lines · K. Y. Srinivasan
This symbol would be used in the Hyper-V balloon driver to support 2M allocations.
In this version of the patch, based on feedback from Michal Hocko <[email protected]>, I have updated the patch description.
I guess this part is not necessary ;)
Quoted 2 lines
Signed-off-by K. Y. Srinivasan <[email protected]>
Anyway
Quoted 24 lines
--- mm/page_alloc.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)mm/page_alloc.c +1
-0@@ -1404,6 +1404,7 @@ void split_page(struct page *page, unsigned int order)for (i =1 ; i < (1 << order); i++)set_page_refcounted (page + i); } +EXPORT_SYMBOL_GPL (split_page);static int __isolate_free_page (struct page *page,unsigned int order) {Signature · 7 lines
1.7.4.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RE: [PATCH V2 1/3] mm: Export split_page()
Quoted 19 lines
-----Original Message-----
From: Michal Hocko [mailto:[email protected]]
Sent: Tuesday, March 19, 2013 10:13 AM
To: KY Srinivasan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V2 1/3] mm: Exportsplit_page()Quoted 5 lines · K. Y. Srinivasan
This symbol would be used in the Hyper-V balloon driver to support 2M allocations.
In this version of the patch, based on feedback from Michal Hocko <[email protected]>, I have updated the patch description.
Quoted 7 lines
I guess this part is not necessary ;)
Quoted 2 lines
Signed-off-by K. Y. Srinivasan <[email protected]>Anyway
Acked-by Michal Hocko <[email protected]>
Greg,
Would you be taking this patch through your tree?
Regards,
K. Y
Quoted 20 lines
Quoted 20 lines
--- mm/page_alloc.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)mm/page_alloc.c +1
-1@@ -1404,6 +1404,7 @@ void split_page(struct page *page, unsigned int order)for (i =1 ; i < (1 << order); i++)set_page_refcounted (page + i); } +EXPORT_SYMBOL_GPL (split_page);static int __isolate_free_page (struct page *page,unsigned int order) { --1.7.4.1
Quoted 10 lines
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RE: [PATCH V2 1/3] mm: Export split_page()
Quoted 37 lines
-----Original Message-----
From: devel [mailto:[email protected]] On Behalf Of KY Srinivasan
Sent: Tuesday, March 19, 2013 5:40 PM
To: Michal Hocko
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH V2 1/3] mm: Exportsplit_page()Quoted 22 lines
-----Original Message-----
From: Michal Hocko [mailto:[email protected]]
Sent: Tuesday, March 19, 2013 10:13 AM
To: KY Srinivasan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH V2 1/3] mm: Exportsplit_page()Quoted 5 lines · K. Y. Srinivasan
This symbol would be used in the Hyper-V balloon driver to support 2M allocations.
In this version of the patch, based on feedback from Michal Hocko <[email protected]>, I have updated the patch description.
I guess this part is not necessary ;)
Quoted 2 lines
Signed-off-by K. Y. Srinivasan <[email protected]>
Quoted 8 lines
Quoted 2 lines
Anyway
Acked-by Michal Hocko <[email protected]>Greg,
Would you be taking this patch through your tree?
Regards,
Andrew,
Could you take this patch through your tree.
Regards,
K. Y
Quoted 40 lines
K. Y
Quoted 10 lines
mm/page_alloc.c void split_page(struct page *page, unsigned int
Quoted 10 lines
--- mm/page_alloc.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)mm/page_alloc.c +0
-0@@ -1404,6 +1404,7 @@ void split_page(struct page *page, unsigned intorder)
Quoted 16 lines
Quoted 10 lines
mm/page_alloc.c +1
-1for (i =1 ; i < (1 << order); i++)set_page_refcounted (page + i); } +EXPORT_SYMBOL_GPL (split_page);static int __isolate_free_page (struct page *page,unsigned int order) { --1.7.4.1
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/Quoted 4 lines
--
Michal Hocko
SUSE Labs_______________________________________________
devel mailing list
[email protected]
http://driverdev.linuxdriverproject.org/mailman/listinfo/devel
RE: [PATCH V2 1/3] mm: Export split_page()
Quoted 60 lines
-----Original Message-----
From: KY Srinivasan
Sent: Monday, March 25, 2013 9:18 AM
To: KY Srinivasan; Michal Hocko
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH V2 1/3] mm: Exportsplit_page()Quoted 6 lines
-----Original Message-----
From: devel [mailto:[email protected]] On Behalf Of KY Srinivasan
Sent: Tuesday, March 19, 2013 5:40 PM
To: Michal Hocko
Cc: [email protected]; [email protected]; linux-Quoted 15 lines
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH V2 1/3] mm: Exportsplit_page()Quoted 8 lines
-----Original Message-----
From: Michal Hocko [mailto:[email protected]]
Sent: Tuesday, March 19, 2013 10:13 AM
To: KY Srinivasan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];Quoted 14 lines
Quoted 14 lines
Subject: Re: [PATCH V2 1/3] mm: Export
split_page()Quoted 5 lines · K. Y. Srinivasan
This symbol would be used in the Hyper-V balloon driver to support 2M allocations.
In this version of the patch, based on feedback from Michal Hocko <[email protected]>, I have updated the patch description.
I guess this part is not necessary ;)
Quoted 2 lines
Signed-off-by K. Y. Srinivasan <[email protected]>Quoted 8 lines
Quoted 2 lines
Anyway
Acked-by Michal Hocko <[email protected]>Greg,
Would you be taking this patch through your tree?
Regards,
Quoted 7 lines
Andrew,
Could you take this patch through your tree.
Regards,
K. Y
Greg,
Now that Andrew has checked in this patch, do you want me to resend the remaining two patches here.
Regards,
K. Y
Quoted 40 lines
Quoted 40 lines
K. Y
Quoted 10 lines
mm/page_alloc.c void split_page(struct page *page, unsigned int
Quoted 10 lines
--- mm/page_alloc.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)mm/page_alloc.c +0
-0@@ -1404,6 +1404,7 @@ void split_page(struct page *page, unsigned intorder)
Quoted 16 lines
Quoted 10 lines
mm/page_alloc.c +1
-1for (i =1 ; i < (1 << order); i++)set_page_refcounted (page + i); } +EXPORT_SYMBOL_GPL (split_page);static int __isolate_free_page (struct page *page,unsigned int order) { --1.7.4.1
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/Quoted 4 lines
--
Michal Hocko
SUSE Labs_______________________________________________
devel mailing list
[email protected]
http://driverdev.linuxdriverproject.org/mailman/listinfo/devel