virtio_balloon: Add VIRTIO_BALLOON_F_CONT_PAGES to report continuous pages
Subsystems (3)
v4 cover virtio_balloon: Add VIRTIO_BALLOON_F_CONT_PAGES to report continuous pages
The first, second and third version are in [1], [2] and [3].
Code of current version for Linux and qemu is available in [4] and [5]. Update of this version:
1. Report continuous pages will increase the speed. So added deflate continuous pages.
2. According to the comments from David in [6], added 2 new vqs inflate_cont_vq and deflate_cont_vq to report continuous pages with format 32 bits pfn and 32 bits size.
Following is the introduction of the function.
These patches add VIRTIO_BALLOON_F_CONT_PAGES to virtio_balloon. With this flag, balloon tries to use continuous pages to inflate and deflate. Opening this flag can bring two benefits:
1. Report continuous pages will increase memory report size of each time call tell_host. Then it will increase the speed of balloon inflate and deflate.
2. Host THPs will be splitted when qemu release the page of balloon inflate. Inflate balloon with continuous pages will let QEMU release the pages of same THPs. That will help decrease the splitted THPs number in the host.
Following is an example in a VM with 1G memory 1CPU. This test setups an environment that has a lot of fragmentation pages. Then inflate balloon will split the THPs.
// This is the THP number before VM execution in the host.
// None use THP.
cat /proc/meminfo | grep AnonHugePages:
AnonHugePages: 0 kB
// After VM start, use usemem
// (https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git)
// punch-holes function generates 400m fragmentation pages in the guest // kernel.
usemem --punch-holes -s -1 800m &
// This is the THP number after this command in the host.
// Some THP is used by VM because usemem will access 800M memory
// in the guest.
cat /proc/meminfo | grep AnonHugePages:
AnonHugePages: 911360 kB
// Connect to the QEMU monitor, setup balloon, and set it size to 600M. (qemu) device_add virtio-balloon-pci,id=balloon1
(qemu) info balloon
balloon: actual=1024
(qemu) balloon 600
(qemu) info balloon
balloon: actual=600
// This is the THP number after inflate the balloon in the host. cat /proc/meminfo | grep AnonHugePages:
AnonHugePages: 88064 kB
// Set the size back to 1024M in the QEMU monitor.
(qemu) balloon 1024
(qemu) info balloon
balloon: actual=1024
// Use usemem to increase the memory usage of QEMU.
killall usemem
usemem 800m
// This is the THP number after this operation.
cat /proc/meminfo | grep AnonHugePages:
AnonHugePages: 65536 kB
Following example change to use continuous pages balloon. The number of splitted THPs is decreased.
// This is the THP number before VM execution in the host. // None use THP.
cat /proc/meminfo | grep AnonHugePages:
AnonHugePages: 0 kB
// After VM start, use usemem punch-holes function generates 400M // fragmentation pages in the guest kernel.
usemem --punch-holes -s -1 800m &
// This is the THP number after this command in the host.
// Some THP is used by VM because usemem will access 800M memory // in the guest.
cat /proc/meminfo | grep AnonHugePages:
AnonHugePages: 911360 kB
// Connect to the QEMU monitor, setup balloon, and set it size to 600M. (qemu) device_add virtio-balloon-pci,id=balloon1,cont-pages=on (qemu) info balloon
balloon: actual=1024
(qemu) balloon 600
(qemu) info balloon
balloon: actual=600
// This is the THP number after inflate the balloon in the host. cat /proc/meminfo | grep AnonHugePages:
AnonHugePages: 616448 kB
// Set the size back to 1024M in the QEMU monitor.
(qemu) balloon 1024
(qemu) info balloon
balloon: actual=1024
// Use usemem to increase the memory usage of QEMU.
killall usemem
usemem 800m
// This is the THP number after this operation.
cat /proc/meminfo | grep AnonHugePages:
AnonHugePages: 907264 kB
[1] https://lkml.org/lkml/2020/3/12/144
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lkml.org/lkml/2020/5/12/324
[4] https://github.com/teawater/linux/tree/balloon_conts
[5] https://github.com/teawater/qemu/tree/balloon_conts
[6] https://lkml.org/lkml/2020/5/13/1211
Hui Zhu (2):
virtio_balloon: Add VIRTIO_BALLOON_F_CONT_PAGES and inflate_cont_vq
virtio_balloon: Add deflate_cont_vq to deflate continuous pages
drivers/virtio/virtio_balloon.c | 180 +++++++++++++++++++++++++++++++----- include/linux/balloon_compaction.h | 12 ++ include/uapi/linux/virtio_balloon.h | 1 mm/balloon_compaction.c | 117 +++++++++++++++++++++-- 4 files changed, 280 insertions(+), 30 deletions(-)
RFC v4 1/2 virtio_balloon: Add VIRTIO_BALLOON_F_CONT_PAGES and inflate_cont_vq
This commit adds a new flag VIRTIO_BALLOON_F_CONT_PAGES to virtio_balloon. Add it adds a vq inflate_cont_vq to inflate continuous pages. When VIRTIO_BALLOON_F_CONT_PAGES is set, try to allocate continuous pages and report them use inflate_cont_vq.
--- drivers/virtio/virtio_balloon.c | 119 ++++++++++++++++++++++++++++++------ include/linux/balloon_compaction.h | 9 ++- include/uapi/linux/virtio_balloon.h | 1 + mm/balloon_compaction.c | 41 ++++++++++--- 4 files changed, 142 insertions(+), 28 deletions(-)
drivers/virtio/virtio_balloon.c +100 -19
@@ -42,6 +42,9 @@ (1 << (VIRTIO_BALLOON_HINT_BLOCK_ORDER + PAGE_SHIFT))#define VIRTIO_BALLOON_HINT_BLOCK_PAGES (1 << VIRTIO_BALLOON_HINT_BLOCK_ORDER) +#define VIRTIO_BALLOON_INFLATE_MAX_ORDER min((int) (sizeof(__virtio32) * BITS_PER_BYTE - \ + 1 - PAGE_SHIFT), (MAX_ORDER-1)) +#ifdef CONFIG_BALLOON_COMPACTION static struct vfsmount *balloon_mnt;#endif
@@ -52,6 +55,7 @@ enum virtio_balloon_vq {VIRTIO_BALLOON_VQ_STATS ,VIRTIO_BALLOON_VQ_FREE_PAGE ,VIRTIO_BALLOON_VQ_REPORTING , +VIRTIO_BALLOON_VQ_INFLATE_CONT , VIRTIO_BALLOON_VQ_MAX };
@@ -61,7 +65,7 @@ enum virtio_balloon_config_read {struct virtio_balloon {struct virtio_device *vdev; -struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq; +struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq, *inflate_cont_vq;/* Balloon's own wq for cpu-intensive work items */ struct workqueue_struct *balloon_wq;
@@ -126,6 +130,9 @@ struct virtio_balloon {/* Free page reporting device */ struct virtqueue *reporting_vq;struct page_reporting_dev_info pr_dev_info; + +/* Current order of inflate continuous pages - VIRTIO_BALLOON_F_CONT_PAGES */ +__u32 current_pages_order; };static struct virtio_device_id id_table[] = {
@@ -208,19 +215,59 @@ static void set_page_pfns(struct virtio_balloon *vb, page_to_balloon_pfn(page) + i); } +static void set_page_pfns_order (struct virtio_balloon *vb, +__virtio32 pfns[],struct page *page, +unsigned int order) +{ +if (order ==0 ) +return set_page_pfns (vb, pfns, page); + +/* Set the first pfn of the continuous pages. */ + pfns[0 ] =cpu_to_virtio32 (vb->vdev,page_to_balloon_pfn (page)); +/* Set the size of the continuous pages. */ + pfns[1 ] =PAGE_SIZE << order; +} +static unsigned fill_balloon (struct virtio_balloon *vb,size_t num) {unsigned num_allocated_pages; -unsigned num_pfns; +unsigned int num_pfns, pfn_per_alloc;struct page *page;LIST_HEAD (pages); +bool is_cont = vb->current_pages_order !=0 ; -/* We can only do one array worth at a time. */ - num =min (num,ARRAY_SIZE (vb->pfns)); - -for (num_pfns =0 ; num_pfns < num; - num_pfns +=VIRTIO_BALLOON_PAGES_PER_PAGE ) { -struct page *page =balloon_page_alloc (); +if (is_cont) + pfn_per_alloc =2 ; +else + pfn_per_alloc =VIRTIO_BALLOON_PAGES_PER_PAGE ; + +for (num_pfns =0 , num_allocated_pages =0 ; + num_pfns <ARRAY_SIZE (vb->pfns) && num_allocated_pages < num; + num_pfns += pfn_per_alloc, + num_allocated_pages +=VIRTIO_BALLOON_PAGES_PER_PAGE << vb->current_pages_order) { +struct page *page; + +for (; vb->current_pages_order >=0 ; vb->current_pages_order--) { +if (vb->current_pages_order && + num - num_allocated_pages < +VIRTIO_BALLOON_PAGES_PER_PAGE << vb->current_pages_order) +continue ; + page =balloon_pages_alloc (vb->current_pages_order); +if (page) { +/* If the first allocated page is not continuous pages, +* go back to transport page as signle page. +*/ +if (is_cont && num_pfns ==0 && !vb->current_pages_order) { + is_cont = false; + pfn_per_alloc =VIRTIO_BALLOON_PAGES_PER_PAGE ; + } +set_page_private (page, vb->current_pages_order); +balloon_page_push (&pages, page); +break ; + } +if (!vb->current_pages_order) +break ; + }if (!page) { dev_info_ratelimited(&vb->vdev->dev,
@@ -230,8 +277,6 @@ static unsigned fill_balloon(struct virtio_balloon *vb, size_t num)msleep (200 );break ; } - -balloon_page_push (&pages, page); }mutex_lock (&vb->balloon_lock);
@@ -239,20 +284,34 @@ static unsigned fill_balloon(struct virtio_balloon *vb, size_t num) vb->num_pfns =0 ;while ((page =balloon_page_pop (&pages))) { -balloon_page_enqueue (&vb->vb_dev_info, page); +unsigned int order =page_private (page); + +set_page_private (page,0 ); + +/* Split the continuous pages because they will be freed +* by release_pages_balloon respectively. +*/ +if (order) +split_page (page, order); + +balloon_pages_enqueue (&vb->vb_dev_info, page, order); + +set_page_pfns_order (vb, vb->pfns + vb->num_pfns, page, order); -set_page_pfns (vb, vb->pfns + vb->num_pfns, page); - vb->num_pages +=VIRTIO_BALLOON_PAGES_PER_PAGE ;if (!virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_DEFLATE_ON_OOM )) -adjust_managed_page_count (page,-1 ); - vb->num_pfns +=VIRTIO_BALLOON_PAGES_PER_PAGE ; +adjust_managed_page_count (page, -(1 << order)); + vb->num_pfns += pfn_per_alloc; } + vb->num_pages += num_allocated_pages; - num_allocated_pages = vb->num_pfns;/* Did we get any? */ -if (vb->num_pfns !=0 ) -tell_host (vb, vb->inflate_vq); +if (vb->num_pfns !=0 ) { +if (is_cont) +tell_host (vb, vb->inflate_cont_vq); +else +tell_host (vb, vb->inflate_vq); + }mutex_unlock (&vb->balloon_lock);return num_allocated_pages;
@@ -488,7 +547,7 @@ static void update_balloon_size_func(struct work_struct *work) diff =towards_target (vb);if (!diff) -return ; + goto stop_out;if (diff >0 ) diff -=fill_balloon (vb, diff);
@@ -498,6 +557,11 @@ static void update_balloon_size_func(struct work_struct *work)if (diff)queue_work (system_freezable_wq, work); +else { +stop_out: +if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_CONT_PAGES )) + vb->current_pages_order =VIRTIO_BALLOON_INFLATE_MAX_ORDER ; + } }static int init_vqs (struct virtio_balloon *vb)
@@ -521,6 +585,8 @@ static int init_vqs(struct virtio_balloon *vb) callbacks[VIRTIO_BALLOON_VQ_FREE_PAGE ] =NULL ; names[VIRTIO_BALLOON_VQ_FREE_PAGE ] =NULL ; names[VIRTIO_BALLOON_VQ_REPORTING ] =NULL ; + names[VIRTIO_BALLOON_VQ_INFLATE_CONT ] =NULL ; + callbacks[VIRTIO_BALLOON_VQ_INFLATE_CONT ] =NULL ;if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_STATS_VQ )) { names[VIRTIO_BALLOON_VQ_STATS ] ="stats" ;
@@ -537,6 +603,11 @@ static int init_vqs(struct virtio_balloon *vb) callbacks[VIRTIO_BALLOON_VQ_REPORTING ] = balloon_ack; } +if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_CONT_PAGES )) { + names[VIRTIO_BALLOON_VQ_INFLATE_CONT ] ="inflate_cont" ; + callbacks[VIRTIO_BALLOON_VQ_INFLATE_CONT ] = balloon_ack; + } + err = vb->vdev->config->find_vqs (vb->vdev,VIRTIO_BALLOON_VQ_MAX , vqs, callbacks, names,NULL ,NULL );if (err)
@@ -572,6 +643,10 @@ static int init_vqs(struct virtio_balloon *vb)if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_REPORTING )) vb->reporting_vq = vqs[VIRTIO_BALLOON_VQ_REPORTING ]; +if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_CONT_PAGES )) + vb->inflate_cont_vq + = vqs[VIRTIO_BALLOON_VQ_INFLATE_CONT ]; +return 0 ; }
@@ -997,6 +1072,11 @@ static int virtballoon_probe(struct virtio_device *vdev) goto out_unregister_oom; } +if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_CONT_PAGES )) + vb->current_pages_order =VIRTIO_BALLOON_INFLATE_MAX_ORDER ; +else + vb->current_pages_order =0 ; +virtio_device_ready (vdev);if (towards_target (vb))
@@ -1131,6 +1211,7 @@ static unsigned int features[] = {VIRTIO_BALLOON_F_FREE_PAGE_HINT ,VIRTIO_BALLOON_F_PAGE_POISON ,VIRTIO_BALLOON_F_REPORTING , + VIRTIO_BALLOON_F_CONT_PAGES, }; static struct virtio_driver virtio_balloon_driver = {
include/linux/balloon_compaction.h +8 -1
@@ -60,15 +60,22 @@ struct balloon_dev_info {struct inode *inode; }; -extern struct page *balloon_page_alloc (void ); +extern struct page *balloon_pages_alloc (unsigned int order);extern void balloon_page_enqueue (struct balloon_dev_info *b_dev_info,struct page *page); +extern void balloon_pages_enqueue (struct balloon_dev_info *b_dev_info, +struct page *page,unsigned int order);extern struct page *balloon_page_dequeue (struct balloon_dev_info *b_dev_info);extern size_t balloon_page_list_enqueue (struct balloon_dev_info *b_dev_info,struct list_head *pages);extern size_t balloon_page_list_dequeue (struct balloon_dev_info *b_dev_info,struct list_head *pages,size_t n_req_pages); +static inline struct page *balloon_page_alloc (void ) +{ +return balloon_pages_alloc (0 ); +} +static inline void balloon_devinfo_init (struct balloon_dev_info *balloon) { balloon->isolated_pages =0 ;
include/uapi/linux/virtio_balloon.h +1 -0
@@ -37,6 +37,7 @@#define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3/* VQ to report free pages */ #define VIRTIO_BALLOON_F_PAGE_POISON 4/* Guest is using page poisoning */ #define VIRTIO_BALLOON_F_REPORTING 5/* Page reporting virtqueue */ +#define VIRTIO_BALLOON_F_CONT_PAGES 6/* VQ to report continuous pages */ /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12
mm/balloon_compaction.c +33 -8
61 lines not on this page · raw message
Signature · 1 line
2.7.4
Michael S. Tsirkin
maintainerRe: [RFC for Linux v4 1/2] virtio_balloon: Add VIRTIO_BALLOON_F_CONT_PAGES and inflate_cont_vq
include/uapi/linux/virtio_balloon.h line 43
Quoted 12 lines · Hui Zhu
include/uapi/linux/virtio_balloon.h +1
-0@@ -37,6 +37,7 @@#define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3/* VQ to report free pages */ #define VIRTIO_BALLOON_F_PAGE_POISON 4/* Guest is using page poisoning */ #define VIRTIO_BALLOON_F_REPORTING 5/* Page reporting virtqueue */ +#define VIRTIO_BALLOON_F_CONT_PAGES 6/* VQ to report continuous pages */ /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12
So how does the guest/host interface look like?
Could you write up something about it?
Re: [RFC for Linux v4 1/2] virtio_balloon: Add VIRTIO_BALLOON_F_CONT_PAGES and inflate_cont_vq
Quoted 15 lines
2020年7月16日 14:43,Michael S. Tsirkin <[email protected]> 写道:
Quoted 10 lines · Hui Zhu
include/uapi/linux/virtio_balloon.h +1
-0@@ -37,6 +37,7 @@#define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3/* VQ to report free pages */ #define VIRTIO_BALLOON_F_PAGE_POISON 4/* Guest is using page poisoning */ #define VIRTIO_BALLOON_F_REPORTING 5/* Page reporting virtqueue */ +#define VIRTIO_BALLOON_F_CONT_PAGES 6/* VQ to report continuous pages */
Quoted 5 lines
include/uapi/linux/virtio_balloon.h line 43
Quoted 2 lines
include/uapi/linux/virtio_balloon.h +0
-0/* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12So how does the guest/host interface look like?
Could you write up something about it?
Continuous pages are report by num_pfns and pfns in virtio_balloon too.
The function to set pfns is set_page_pfns_size in https://github.com/teawater/linux/blob/balloon_conts/drivers/virtio/virtio_balloon.c#L221
static void set_page_pfns_size(struct virtio_balloon *vb,
__virtio32 pfns[], struct page *page,
size_t size)
{
/* Set the first pfn of the continuous pages. */
pfns[0] = cpu_to_virtio32(vb->vdev, page_to_balloon_pfn(page));
/* Set the size of the continuous pages. */
pfns[1] = (__virtio32) size;
}Each of continuous pages need 2 pfn.
The first pfn of the pages is set to pfns[0]. The size of the pages is set to pfns[1].
The pfn is 32 bits.
So the max order of inflate continuous pages is VIRTIO_BALLOON_INFLATE_MAX_ORDER.
#define VIRTIO_BALLOON_INFLATE_MAX_ORDER min((int) (sizeof(__virtio32) * BITS_PER_BYTE - \
1 - PAGE_SHIFT), (MAX_ORDER-1))The max page number of deflate continuous pages is VIRTIO_BALLOON_DEFLATE_MAX_PAGES_NUM. #define VIRTIO_BALLOON_DEFLATE_MAX_PAGES_NUM (((__virtio32)~0U) >> PAGE_SHIFT)
Best,
Hui
RFC v4 1/2 virtio_balloon: Add cont-pages and icvq
In reply to Hui ZhuThis commit adds cont-pages option to virtio_balloon. virtio_balloon will open flags VIRTIO_BALLOON_F_CONT_PAGES with this option. And it add a vq icvq to inflate continuous pages.
When VIRTIO_BALLOON_F_CONT_PAGES is set, try to get continuous pages from icvq and use madvise MADV_DONTNEED release the pages.
--- hw/virtio/virtio-balloon.c | 80 ++++++++++++++++--------- include/hw/virtio/virtio-balloon.h | 2 +- include/standard-headers/linux/virtio_balloon.h | 1 + 3 files changed, 55 insertions(+), 28 deletions(-)
hw/virtio/virtio-balloon.c +53 -27
@@ -65,23 +65,26 @@ static bool virtio_balloon_pbp_matches(PartiallyBalloonedPage *pbp,static void balloon_inflate_page (VirtIOBalloon *balloon,MemoryRegion *mr,hwaddr mr_offset, +size_t size,PartiallyBalloonedPage *pbp) {void *addr =memory_region_get_ram_ptr (mr) + mr_offset;ram_addr_t rb_offset, rb_aligned_offset, base_gpa;RAMBlock *rb;size_t rb_page_size; -int subpages; +int subpages, pages_num;/* XXX is there a better way to get to the RAMBlock than via a * host address? */ rb =qemu_ram_block_from_host (addr, false, &rb_offset); rb_page_size =qemu_ram_pagesize (rb); + size &= ~(rb_page_size -1 ); +if (rb_page_size ==BALLOON_PAGE_SIZE ) {/* Easy case */ -ram_block_discard_range (rb, rb_offset, rb_page_size); +ram_block_discard_range (rb, rb_offset, size);/* We ignore errors from ram_block_discard_range(), because it * has already reported them, and failing to discard a balloon * page is not fatal */
@@ -99,32 +102,38 @@ static void balloon_inflate_page(VirtIOBalloon *balloon, rb_aligned_offset =QEMU_ALIGN_DOWN (rb_offset, rb_page_size); subpages = rb_page_size /BALLOON_PAGE_SIZE ; - base_gpa =memory_region_get_ram_addr (mr) + mr_offset - - (rb_offset - rb_aligned_offset); -if (pbp->bitmap && !virtio_balloon_pbp_matches (pbp, base_gpa)) { -/* We've partially ballooned part of a host page, but now -* we're trying to balloon part of a different one. Too hard, -* give up on the old partial page */ -virtio_balloon_pbp_free (pbp); - } +for (pages_num = size /BALLOON_PAGE_SIZE ; + pages_num >0 ; pages_num--) { + base_gpa =memory_region_get_ram_addr (mr) + mr_offset - + (rb_offset - rb_aligned_offset); -if (!pbp->bitmap) { -virtio_balloon_pbp_alloc (pbp, base_gpa, subpages); - } +if (pbp->bitmap && !virtio_balloon_pbp_matches (pbp, base_gpa)) { +/* We've partially ballooned part of a host page, but now +* we're trying to balloon part of a different one. Too hard, +* give up on the old partial page */ +virtio_balloon_pbp_free (pbp); + } -set_bit ((rb_offset - rb_aligned_offset) /BALLOON_PAGE_SIZE , - pbp->bitmap); +if (!pbp->bitmap) { +virtio_balloon_pbp_alloc (pbp, base_gpa, subpages); + } -if (bitmap_full (pbp->bitmap, subpages)) { -/* We've accumulated a full host page, we can actually discard -* it now */ +set_bit ((rb_offset - rb_aligned_offset) /BALLOON_PAGE_SIZE , + pbp->bitmap); -ram_block_discard_range (rb, rb_aligned_offset, rb_page_size); -/* We ignore errors from ram_block_discard_range(), because it -* has already reported them, and failing to discard a balloon -* page is not fatal */ -virtio_balloon_pbp_free (pbp); +if (bitmap_full (pbp->bitmap, subpages)) { +/* We've accumulated a full host page, we can actually discard +* it now */ + +ram_block_discard_range (rb, rb_aligned_offset, rb_page_size); +/* We ignore errors from ram_block_discard_range(), because it +* has already reported them, and failing to discard a balloon +* page is not fatal */ +virtio_balloon_pbp_free (pbp); + } + + mr_offset +=BALLOON_PAGE_SIZE ; } }
@@ -340,12 +349,21 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)while (iov_to_buf (elem->out_sg, elem->out_num, offset, &pfn,4 ) ==4 ) {unsigned int p =virtio_ldl_p (vdev, &pfn);hwaddr pa; +unsigned int psize =BALLOON_PAGE_SIZE ; pa = (hwaddr ) p <<VIRTIO_BALLOON_PFN_SHIFT ; offset +=4 ; - section =memory_region_find (get_system_memory (), pa, -BALLOON_PAGE_SIZE ); +if (vq == s->icvq) { +uint32_t psize_ptr; +if (iov_to_buf (elem->out_sg, elem->out_num, offset, &psize_ptr,4 ) !=4 ) { +break ; + } + psize =virtio_ldl_p (vdev, &psize_ptr); + offset +=4 ; + } + + section =memory_region_find (get_system_memory (), pa, psize);if (!section.mr) {trace_virtio_balloon_bad_addr (pa);continue ;
@@ -361,9 +379,10 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)trace_virtio_balloon_handle_output (memory_region_name (section.mr), pa);if (!qemu_balloon_is_inhibited ()) { -if (vq == s->ivq) { +if (vq == s->ivq || vq == s->icvq) {balloon_inflate_page (s, section.mr, - section.offset_within_region, &pbp); + section.offset_within_region, + psize, &pbp); }else if (vq == s->dvq) {balloon_deflate_page (s, section.mr, section.offset_within_region); }else {
@@ -816,6 +835,11 @@ static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)virtio_error (vdev,"iothread is missing" ); } } + +if (virtio_has_feature (s->host_features,VIRTIO_BALLOON_F_CONT_PAGES )) { + s->icvq =virtio_add_queue (vdev,128 , virtio_balloon_handle_output); + } +reset_stats (s); }
@@ -916,6 +940,8 @@ static Property virtio_balloon_properties[] = { VIRTIO_BALLOON_F_DEFLATE_ON_OOM, false),DEFINE_PROP_BIT ("free-page-hint" , VirtIOBalloon, host_features,VIRTIO_BALLOON_F_FREE_PAGE_HINT , false), +DEFINE_PROP_BIT ("cont-pages" , VirtIOBalloon, host_features, + VIRTIO_BALLOON_F_CONT_PAGES, false),/* QEMU 4.0 accidentally changed the config size even when free-page-hint * is disabled, resulting in QEMU 3.1 migration incompatibility. This * property retains this quirk for QEMU 4.1 machine types.
include/hw/virtio/virtio-balloon.h +1 -1
@@ -42,7 +42,7 @@ enum virtio_balloon_free_page_report_status { typedef struct VirtIOBalloon { VirtIODevice parent_obj; - VirtQueue *ivq, *dvq, *svq, *free_page_vq; + VirtQueue *ivq, *dvq, *svq, *free_page_vq, *icvq; uint32_t free_page_report_status; uint32_t num_pages; uint32_t actual;
include/standard-headers/linux/virtio_balloon.h +1 -0
@@ -36,6 +36,7 @@#define VIRTIO_BALLOON_F_DEFLATE_ON_OOM 2/* Deflate balloon on OOM */ #define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3/* VQ to report free pages */ #define VIRTIO_BALLOON_F_PAGE_POISON 4/* Guest is using page poisoning */ +#define VIRTIO_BALLOON_F_CONT_PAGES 6/* VQ to report continuous pages */ /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12
Signature · 1 line
2.7.4
v4 cover virtio-balloon: Add option cont-pages to set VIRTIO_BALLOON_F_CONT_PAGES
In reply to Hui ZhuCode of current version for Linux and qemu is available in [1] and [2]. Update of this version:
1. Report continuous pages will increase the speed. So added deflate continuous pages.
2. According to the comments from David in [3], added 2 new vqs icvq and dcvq to get continuous pages with format 32 bits pfn and 32 bits size.
Following is the introduction of the function.
Set option cont-pages to on will open flags VIRTIO_BALLOON_F_CONT_PAGES. qemu will get continuous pages from icvq and dcvq and do madvise MADV_WILLNEED and MADV_DONTNEED with the pages.
Opening this flag can bring two benefits:
1. Increase the speed of balloon inflate and deflate.
2. Decrease the splitted THPs number in the host.
[1] https://github.com/teawater/linux/tree/balloon_conts
[2] https://github.com/teawater/qemu/tree/balloon_conts
[3] https://lkml.org/lkml/2020/5/13/1211
Hui Zhu (2):
virtio_balloon: Add cont-pages and icvq
virtio_balloon: Add dcvq to deflate continuous pages
hw/virtio/virtio-balloon.c | 92 +++++++++++++++--------- include/hw/virtio/virtio-balloon.h | 2 include/standard-headers/linux/virtio_balloon.h | 1 3 files changed, 63 insertions(+), 32 deletions(-)
RFC v4 2/2 virtio_balloon: Add deflate_cont_vq to deflate continuous pages
In reply to Hui ZhuThis commit adds a vq deflate_cont_vq to deflate continuous pages. When VIRTIO_BALLOON_F_CONT_PAGES is set, call leak_balloon_cont to leak the balloon.leak_balloon_cont will call balloon_page_list_dequeue_cont get continuous pages from balloon and report them use deflate_cont_vq.
--- drivers/virtio/virtio_balloon.c | 73 ++++++++++++++++++++++++++++++++---- include/linux/balloon_compaction.h | 3 ++ mm/balloon_compaction.c | 76 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 144 insertions(+), 8 deletions(-)
drivers/virtio/virtio_balloon.c +65 -8
@@ -44,6 +44,7 @@#define VIRTIO_BALLOON_INFLATE_MAX_ORDER min((int) (sizeof(__virtio32) * BITS_PER_BYTE - \ 1 - PAGE_SHIFT), (MAX_ORDER-1)) +#define VIRTIO_BALLOON_DEFLATE_MAX_PAGES_NUM (((__virtio32)~0U) >> PAGE_SHIFT)#ifdef CONFIG_BALLOON_COMPACTION static struct vfsmount *balloon_mnt;
@@ -56,6 +57,7 @@ enum virtio_balloon_vq {VIRTIO_BALLOON_VQ_FREE_PAGE ,VIRTIO_BALLOON_VQ_REPORTING ,VIRTIO_BALLOON_VQ_INFLATE_CONT , +VIRTIO_BALLOON_VQ_DEFLATE_CONT , VIRTIO_BALLOON_VQ_MAX };
@@ -65,7 +67,8 @@ enum virtio_balloon_config_read {struct virtio_balloon {struct virtio_device *vdev; -struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq, *inflate_cont_vq; +struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq, + *inflate_cont_vq, *deflate_cont_vq;/* Balloon's own wq for cpu-intensive work items */ struct workqueue_struct *balloon_wq;
@@ -215,6 +218,16 @@ static void set_page_pfns(struct virtio_balloon *vb, page_to_balloon_pfn(page) + i); } +static void set_page_pfns_size (struct virtio_balloon *vb, +__virtio32 pfns[],struct page *page, +size_t size) +{ +/* Set the first pfn of the continuous pages. */ + pfns[0 ] =cpu_to_virtio32 (vb->vdev,page_to_balloon_pfn (page)); +/* Set the size of the continuous pages. */ + pfns[1 ] = (__virtio32 ) size; +} +static void set_page_pfns_order (struct virtio_balloon *vb,__virtio32 pfns[],struct page *page,unsigned int order)
@@ -222,10 +235,7 @@ static void set_page_pfns_order(struct virtio_balloon *vb,if (order ==0 )return set_page_pfns (vb, pfns, page); -/* Set the first pfn of the continuous pages. */ - pfns[0 ] =cpu_to_virtio32 (vb->vdev,page_to_balloon_pfn (page)); -/* Set the size of the continuous pages. */ - pfns[1 ] =PAGE_SIZE << order; +set_page_pfns_size (vb, pfns, page,PAGE_SIZE << order); }static unsigned fill_balloon (struct virtio_balloon *vb,size_t num)
@@ -367,6 +377,42 @@ static unsigned leak_balloon(struct virtio_balloon *vb, size_t num)return num_freed_pages; } +static unsigned int leak_balloon_cont (struct virtio_balloon *vb,size_t num) +{ +unsigned int num_freed_pages; +struct balloon_dev_info *vb_dev_info = &vb->vb_dev_info; +LIST_HEAD (pages); +size_t num_pages; + +mutex_lock (&vb->balloon_lock); +for (vb->num_pfns =0 , num_freed_pages =0 ; + vb->num_pfns <ARRAY_SIZE (vb->pfns) && num_freed_pages < num; + vb->num_pfns +=2 , + num_freed_pages += num_pages << (PAGE_SHIFT -VIRTIO_BALLOON_PFN_SHIFT )) { +struct page *page; + + num_pages =balloon_page_list_dequeue_cont (vb_dev_info, &pages, &page, +min_t (size_t, +VIRTIO_BALLOON_DEFLATE_MAX_PAGES_NUM , + num - num_freed_pages)); +if (!num_pages) +break ; +set_page_pfns_size (vb, vb->pfns + vb->num_pfns, page, num_pages <<PAGE_SHIFT ); + } + vb->num_pages -= num_freed_pages; + +/* +* Note that if +* virtio_has_feature(vdev, VIRTIO_BALLOON_F_MUST_TELL_HOST); +* is true, we *have* to do it in this order +*/ +if (vb->num_pfns !=0 ) +tell_host (vb, vb->deflate_cont_vq); +release_pages_balloon (vb, &pages); +mutex_unlock (&vb->balloon_lock); +return num_freed_pages; +} +static inline void update_stat (struct virtio_balloon *vb,int idx,u16 tag,u64 val) {
@@ -551,8 +597,12 @@ static void update_balloon_size_func(struct work_struct *work)if (diff >0 ) diff -=fill_balloon (vb, diff); -else - diff +=leak_balloon (vb, -diff); +else { +if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_CONT_PAGES )) + diff +=leak_balloon_cont (vb, -diff); +else + diff +=leak_balloon (vb, -diff); + }update_balloon_size (vb);if (diff)
@@ -587,6 +637,8 @@ static int init_vqs(struct virtio_balloon *vb) names[VIRTIO_BALLOON_VQ_REPORTING ] =NULL ; names[VIRTIO_BALLOON_VQ_INFLATE_CONT ] =NULL ; callbacks[VIRTIO_BALLOON_VQ_INFLATE_CONT ] =NULL ; + names[VIRTIO_BALLOON_VQ_DEFLATE_CONT ] =NULL ; + callbacks[VIRTIO_BALLOON_VQ_DEFLATE_CONT ] =NULL ;if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_STATS_VQ )) { names[VIRTIO_BALLOON_VQ_STATS ] ="stats" ;
@@ -606,6 +658,8 @@ static int init_vqs(struct virtio_balloon *vb)if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_CONT_PAGES )) { names[VIRTIO_BALLOON_VQ_INFLATE_CONT ] ="inflate_cont" ; callbacks[VIRTIO_BALLOON_VQ_INFLATE_CONT ] = balloon_ack; + names[VIRTIO_BALLOON_VQ_DEFLATE_CONT ] ="deflate_cont" ; + callbacks[VIRTIO_BALLOON_VQ_DEFLATE_CONT ] = balloon_ack; } err = vb->vdev->config->find_vqs(vb->vdev, VIRTIO_BALLOON_VQ_MAX,
@@ -643,9 +697,12 @@ static int init_vqs(struct virtio_balloon *vb)if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_REPORTING )) vb->reporting_vq = vqs[VIRTIO_BALLOON_VQ_REPORTING ]; -if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_CONT_PAGES )) +if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_CONT_PAGES )) { vb->inflate_cont_vq = vqs[VIRTIO_BALLOON_VQ_INFLATE_CONT ]; + vb->deflate_cont_vq + = vqs[VIRTIO_BALLOON_VQ_DEFLATE_CONT ]; + }return 0 ; }
include/linux/balloon_compaction.h +3 -0
@@ -70,6 +70,9 @@ extern size_t balloon_page_list_enqueue(struct balloon_dev_info *b_dev_info, struct list_head *pages);extern size_t balloon_page_list_dequeue (struct balloon_dev_info *b_dev_info,struct list_head *pages,size_t n_req_pages); +extern size_t balloon_page_list_dequeue_cont (struct balloon_dev_info *b_dev_info, +struct list_head *pages,struct page **first_page, +size_t max_req_pages);static inline struct page *balloon_page_alloc (void ) {
mm/balloon_compaction.c +76 -0
82 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC v4 2/2 virtio_balloon: Add dcvq to deflate continuous pages
In reply to Hui ZhuThis commit adds a vq dcvq to deflate continuous pages. When VIRTIO_BALLOON_F_CONT_PAGES is set, try to get continuous pages from icvq and use madvise MADV_WILLNEED with the pages.
--- hw/virtio/virtio-balloon.c | 14 +++++++++----- include/hw/virtio/virtio-balloon.h | 2 +- 2 files changed, 10 insertions(+), 6 deletions(-)
hw/virtio/virtio-balloon.c +9 -5
@@ -138,7 +138,8 @@ static void balloon_inflate_page(VirtIOBalloon *balloon, }static void balloon_deflate_page (VirtIOBalloon *balloon, -MemoryRegion *mr,hwaddr mr_offset) +MemoryRegion *mr,hwaddr mr_offset, +size_t size) {void *addr =memory_region_get_ram_ptr (mr) + mr_offset;ram_addr_t rb_offset;
@@ -153,10 +154,11 @@ static void balloon_deflate_page(VirtIOBalloon *balloon, rb_page_size =qemu_ram_pagesize (rb); host_addr = (void *)((uintptr_t )addr & ~(rb_page_size -1 )); + size &= ~(rb_page_size -1 );/* When a page is deflated, we hint the whole host page it lives * on, since we can't do anything smaller */ - ret =qemu_madvise (host_addr, rb_page_size,QEMU_MADV_WILLNEED ); + ret =qemu_madvise (host_addr, size,QEMU_MADV_WILLNEED );if (ret !=0 ) {warn_report ("Couldn't MADV_WILLNEED on balloon deflate: %s" ,strerror (errno));
@@ -354,7 +356,7 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq) pa = (hwaddr ) p <<VIRTIO_BALLOON_PFN_SHIFT ; offset +=4 ; -if (vq == s->icvq) { +if (vq == s->icvq || vq == s->dcvq) {uint32_t psize_ptr;if (iov_to_buf (elem->out_sg, elem->out_num, offset, &psize_ptr,4 ) !=4 ) {break ;
@@ -383,8 +385,9 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)balloon_inflate_page (s, section.mr, section.offset_within_region, psize, &pbp); - }else if (vq == s->dvq) { -balloon_deflate_page (s, section.mr, section.offset_within_region); + }else if (vq == s->dvq || vq == s->dcvq) { +balloon_deflate_page (s, section.mr, section.offset_within_region, + psize); }else {g_assert_not_reached (); }
@@ -838,6 +841,7 @@ static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)if (virtio_has_feature (s->host_features,VIRTIO_BALLOON_F_CONT_PAGES )) { s->icvq =virtio_add_queue (vdev,128 , virtio_balloon_handle_output); + s->dcvq =virtio_add_queue (vdev,128 , virtio_balloon_handle_output); }reset_stats (s);
include/hw/virtio/virtio-balloon.h +1 -1
@@ -42,7 +42,7 @@ enum virtio_balloon_free_page_report_status { typedef struct VirtIOBalloon { VirtIODevice parent_obj; - VirtQueue *ivq, *dvq, *svq, *free_page_vq, *icvq; + VirtQueue *ivq, *dvq, *svq, *free_page_vq, *icvq, *dcvq; uint32_t free_page_report_status; uint32_t num_pages; uint32_t actual;
Signature · 1 line
2.7.4
Michael S. Tsirkin
maintainerRe: [RFC for qemu v4 2/2] virtio_balloon: Add dcvq to deflate continuous pages
Quoted 5 lines · Hui Zhu
This commit adds a vq dcvq to deflate continuous pages. When
VIRTIO_BALLOON_F_CONT_PAGESis set, try to get continuous pages from icvq and use madviseMADV_WILLNEEDwith the pages.Signed-off-by Hui Zhu <[email protected]>
This is arguably something to benchmark. Does guest benefit from MADV_WILLNEED or loose performance?
Quoted 76 lines
--- hw/virtio/virtio-balloon.c | 14 +++++++++----- include/hw/virtio/virtio-balloon.h | 2 +- 2 files changed, 10 insertions(+), 6 deletions(-)hw/virtio/virtio-balloon.c +9
-5@@ -138,7 +138,8 @@ static void balloon_inflate_page(VirtIOBalloon *balloon, }static void balloon_deflate_page (VirtIOBalloon *balloon, -MemoryRegion *mr,hwaddr mr_offset) +MemoryRegion *mr,hwaddr mr_offset, +size_t size) {void *addr =memory_region_get_ram_ptr (mr) + mr_offset;ram_addr_t rb_offset;@@ -153,10 +154,11 @@ static void balloon_deflate_page(VirtIOBalloon *balloon, rb_page_size =qemu_ram_pagesize (rb); host_addr = (void *)((uintptr_t )addr & ~(rb_page_size -1 )); + size &= ~(rb_page_size -1 );/* When a page is deflated, we hint the whole host page it lives * on, since we can't do anything smaller */ - ret =qemu_madvise (host_addr, rb_page_size,QEMU_MADV_WILLNEED ); + ret =qemu_madvise (host_addr, size,QEMU_MADV_WILLNEED );if (ret !=0 ) {warn_report ("Couldn't MADV_WILLNEED on balloon deflate: %s" ,strerror (errno));@@ -354,7 +356,7 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq) pa = (hwaddr ) p <<VIRTIO_BALLOON_PFN_SHIFT ; offset +=4 ; -if (vq == s->icvq) { +if (vq == s->icvq || vq == s->dcvq) {uint32_t psize_ptr;if (iov_to_buf (elem->out_sg, elem->out_num, offset, &psize_ptr,4 ) !=4 ) {break ;@@ -383,8 +385,9 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)balloon_inflate_page (s, section.mr, section.offset_within_region, psize, &pbp); - }else if (vq == s->dvq) { -balloon_deflate_page (s, section.mr, section.offset_within_region); + }else if (vq == s->dvq || vq == s->dcvq) { +balloon_deflate_page (s, section.mr, section.offset_within_region, + psize); }else {g_assert_not_reached (); }@@ -838,6 +841,7 @@ static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)if (virtio_has_feature (s->host_features,VIRTIO_BALLOON_F_CONT_PAGES )) { s->icvq =virtio_add_queue (vdev,128 , virtio_balloon_handle_output); + s->dcvq =virtio_add_queue (vdev,128 , virtio_balloon_handle_output); }reset_stats (s);include/hw/virtio/virtio-balloon.h +1
-1@@ -42,7 +42,7 @@ enum virtio_balloon_free_page_report_status { typedef struct VirtIOBalloon { VirtIODevice parent_obj; - VirtQueue *ivq, *dvq, *svq, *free_page_vq, *icvq; + VirtQueue *ivq, *dvq, *svq, *free_page_vq, *icvq, *dcvq; uint32_t free_page_report_status; uint32_t num_pages; uint32_t actual;Signature · 1 line
2.7.4
Re: [virtio-dev] [RFC for qemu v4 2/2] virtio_balloon: Add dcvq to deflate continuous pages
Quoted 5 lines
2020年7月16日 14:39,Michael S. Tsirkin <[email protected]> 写道:
On Thu, Jul 16, 2020 at 10:41:55AM +0800, Hui Zhu wrote:
Quoted 8 lines
Quoted 5 lines
This commit adds a vq dcvq to deflate continuous pages. When
VIRTIO_BALLOON_F_CONT_PAGESis set, try to get continuous pages from icvq and use madviseMADV_WILLNEEDwith the pages.Signed-off-by Hui Zhu <[email protected]>This is arguably something to benchmark. Does guest benefit from
MADV_WILLNEEDor loose performance?
MADV_WILLNEED will call madvise_willneed in the host kernel.madvise_willneed will schedule all required I/O operations (swap in or vfs_fadvise POSIX_FADV_WILLNEED) of the address.
But the pages of the balloon are released by MADV_DONTNEED.
So I think MADV_WILLNEED will not affect the performance of the guest in the most of situations.
Best,
Hui
Quoted 82 lines
Quoted 77 lines
---
hw/virtio/virtio-balloon.c| 14 +++++++++-----include/hw/virtio/virtio-balloon.h| 2 +-2 files changed, 10 insertions(+), 6 deletions(-)hw/virtio/virtio-balloon.c +9
-5@@ -138,7 +138,8 @@ static void balloon_inflate_page(VirtIOBalloon *balloon, }static void balloon_deflate_page (VirtIOBalloon *balloon, -MemoryRegion *mr,hwaddr mr_offset) +MemoryRegion *mr,hwaddr mr_offset, +size_t size) {void *addr =memory_region_get_ram_ptr (mr) + mr_offset;ram_addr_t rb_offset;@@ -153,10 +154,11 @@ static void balloon_deflate_page(VirtIOBalloon *balloon, rb_page_size =qemu_ram_pagesize (rb); host_addr = (void *)((uintptr_t )addr & ~(rb_page_size -1 )); + size &= ~(rb_page_size -1 );/* When a page is deflated, we hint the whole host page it lives * on, since we can't do anything smaller */ - ret =qemu_madvise (host_addr, rb_page_size,QEMU_MADV_WILLNEED ); + ret =qemu_madvise (host_addr, size,QEMU_MADV_WILLNEED );if (ret !=0 ) {warn_report ("Couldn't MADV_WILLNEED on balloon deflate: %s" ,strerror (errno));@@ -354,7 +356,7 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq) pa = (hwaddr ) p <<VIRTIO_BALLOON_PFN_SHIFT ; offset +=4 ; -if (vq == s->icvq) { +if (vq == s->icvq || vq == s->dcvq) {uint32_t psize_ptr;if (iov_to_buf (elem->out_sg, elem->out_num, offset, &psize_ptr,4 ) !=4 ) {break ;@@ -383,8 +385,9 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)balloon_inflate_page (s, section.mr, section.offset_within_region, psize, &pbp); - }else if (vq == s->dvq) { -balloon_deflate_page (s, section.mr, section.offset_within_region); + }else if (vq == s->dvq || vq == s->dcvq) { +balloon_deflate_page (s, section.mr, section.offset_within_region, + psize); }else {g_assert_not_reached (); }@@ -838,6 +841,7 @@ static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)if (virtio_has_feature (s->host_features,VIRTIO_BALLOON_F_CONT_PAGES )) { s->icvq =virtio_add_queue (vdev,128 , virtio_balloon_handle_output); + s->dcvq =virtio_add_queue (vdev,128 , virtio_balloon_handle_output); }reset_stats (s);include/hw/virtio/virtio-balloon.h +1
-1@@ -42,7 +42,7 @@ enum virtio_balloon_free_page_report_status { typedef struct VirtIOBalloon { VirtIODevice parent_obj; - VirtQueue *ivq, *dvq, *svq, *free_page_vq, *icvq; + VirtQueue *ivq, *dvq, *svq, *free_page_vq, *icvq, *dcvq; uint32_t free_page_report_status; uint32_t num_pages; uint32_t actual;Signature · 1 line
2.7.4---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 83 lines · Hui Zhu
The first, second and third version are in [1], [2] and [3].
Code of current version for Linux and qemu is available in [4] and [5]. Update of this version:
1. Report continuous pages will increase the speed. So added deflate continuous pages.
2. According to the comments from David in [6], added 2 new vqsinflate_cont_vqanddeflate_cont_vqto report continuous pages with format 32 bits pfn and 32 bits size.
Following is the introduction of the function.
These patches addVIRTIO_BALLOON_F_CONT_PAGEStovirtio_balloon. With this flag, balloon tries to use continuous pages to inflate and deflate. Opening this flag can bring two benefits:
1. Report continuous pages will increase memory report size of each time calltell_host. Then it will increase the speed of balloon inflate and deflate.
2. Host THPs will be splitted when qemu release the page of balloon inflate. Inflate balloon with continuous pages will let QEMU release the pages of same THPs. That will help decrease the splitted THPs number in the host.
Following is an example in a VM with 1G memory 1CPU. This test setups an environment that has a lot of fragmentation pages. Then inflate balloon will split the THPs.
// This is the THP number before VM execution in the host.
// None use THP.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 0 kB
// After VM start, use usemem
// (https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git)
// punch-holes function generates 400m fragmentation pages in the guest // kernel.
usemem --punch-holes -s -1 800m &
// This is the THP number after this command in the host.
// Some THP is used by VM because usemem will access 800M memory
// in the guest.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 911360 kB
// Connect to the QEMU monitor, setup balloon, and set it size to 600M. (qemu)device_addvirtio-balloon-pci,id=balloon1
(qemu) info balloon
balloon: actual=1024
(qemu) balloon 600
(qemu) info balloon
balloon: actual=600
// This is the THP number after inflate the balloon in the host. cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 88064 kB
// Set the size back to 1024M in the QEMU monitor.
(qemu) balloon 1024
(qemu) info balloon
balloon: actual=1024
// Use usemem to increase the memory usage of QEMU.
killall usemem
usemem 800m
// This is the THP number after this operation.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 65536 kBFollowing example change to use continuous pages balloon. The number of splitted THPs is decreased.
// This is the THP number before VM execution in the host. // None use THP.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 0 kB
// After VM start, use usemem punch-holes function generates 400M // fragmentation pages in the guest kernel.
usemem --punch-holes -s -1 800m &
// This is the THP number after this command in the host.
// Some THP is used by VM because usemem will access 800M memory // in the guest.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 911360 kB
// Connect to the QEMU monitor, setup balloon, and set it size to 600M. (qemu)device_addvirtio-balloon-pci,id=balloon1,cont-pages=on (qemu) info balloon
balloon: actual=1024
(qemu) balloon 600
(qemu) info balloon
balloon: actual=600
// This is the THP number after inflate the balloon in the host. cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 616448 kB
// Set the size back to 1024M in the QEMU monitor.
(qemu) balloon 1024
Quoted 8 lines
(qemu) info balloon
balloon: actual=1024
// Use usemem to increase the memory usage of QEMU. killall usemem
usemem 800m
// This is the THP number after this operation. cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 907264 kB
I'm a bit confused about which of the above run within guest, and which run within host. Could you explain pls?
Quoted 16 lines
[1] https://lkml.org/lkml/2020/3/12/144
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lkml.org/lkml/2020/5/12/324
[4] https://github.com/teawater/linux/tree/balloon_conts
[5] https://github.com/teawater/qemu/tree/balloon_conts
[6] https://lkml.org/lkml/2020/5/13/1211Hui Zhu (2):
virtio_balloon: AddVIRTIO_BALLOON_F_CONT_PAGESandinflate_cont_vq
virtio_balloon: Adddeflate_cont_vqto deflate continuous pagesdrivers/virtio/virtio_balloon.c | 180 +++++++++++++++++++++++++++++++----- include/linux/balloon_compaction.h | 12 ++ include/uapi/linux/virtio_balloon.h | 1 mm/balloon_compaction.c | 117 +++++++++++++++++++++-- 4 files changed, 280 insertions(+), 30 deletions(-)
Quoted 33 lines
2020年7月16日 14:38,Michael S. Tsirkin <[email protected]> 写道:
Quoted 28 lines · Hui Zhu
The first, second and third version are in [1], [2] and [3].
Code of current version for Linux and qemu is available in [4] and [5]. Update of this version:
1. Report continuous pages will increase the speed. So added deflate continuous pages.
2. According to the comments from David in [6], added 2 new vqsinflate_cont_vqanddeflate_cont_vqto report continuous pages with format 32 bits pfn and 32 bits size.
Following is the introduction of the function.
These patches addVIRTIO_BALLOON_F_CONT_PAGEStovirtio_balloon. With this flag, balloon tries to use continuous pages to inflate and deflate. Opening this flag can bring two benefits:
1. Report continuous pages will increase memory report size of each time calltell_host. Then it will increase the speed of balloon inflate and deflate.
2. Host THPs will be splitted when qemu release the page of balloon inflate. Inflate balloon with continuous pages will let QEMU release the pages of same THPs. That will help decrease the splitted THPs number in the host.
Following is an example in a VM with 1G memory 1CPU. This test setups an environment that has a lot of fragmentation pages. Then inflate balloon will split the THPs.// This is the THP number before VM execution in the host. // None use THP.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 0 kB
These lines are from host.
Quoted 5 lines
Quoted 5 lines
// After VM start, use usemem
// (https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git)
// punch-holes function generates 400m fragmentation pages in the guest // kernel.
usemem --punch-holes -s -1 800m &
These lines are from guest. They setups the environment that has a lot of fragmentation pages.
Quoted 5 lines
Quoted 5 lines
// This is the THP number after this command in the host. // Some THP is used by VM because usemem will access 800M memory // in the guest.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 911360 kB
These lines are from host.
Quoted 7 lines
Quoted 7 lines
// Connect to the QEMU monitor, setup balloon, and set it size to 600M. (qemu)
device_addvirtio-balloon-pci,id=balloon1
(qemu) info balloon
balloon: actual=1024
(qemu) balloon 600
(qemu) info balloon
balloon: actual=600
These lines are from host.
Quoted 3 lines
Quoted 3 lines
// This is the THP number after inflate the balloon in the host. cat
/proc/meminfo| grep AnonHugePages:
AnonHugePages: 88064 kB
These lines are from host.
Quoted 4 lines
Quoted 4 lines
// Set the size back to 1024M in the QEMU monitor. (qemu) balloon 1024
(qemu) info balloon
balloon: actual=1024
These lines are from host.
Quoted 3 lines
Quoted 3 lines
// Use usemem to increase the memory usage of QEMU. killall usemem
usemem 800m
These lines are from guest.
Quoted 3 lines
Quoted 3 lines
// This is the THP number after this operation.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 65536 kB
These lines are from host.
Quoted 7 lines
Quoted 7 lines
Following example change to use continuous pages balloon. The number of splitted THPs is decreased.
// This is the THP number before VM execution in the host. // None use THP.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 0 kB
These lines are from host.
Quoted 3 lines
Quoted 3 lines
// After VM start, use usemem punch-holes function generates 400M // fragmentation pages in the guest kernel.
usemem --punch-holes -s -1 800m &
These lines are from guest. They setups the environment that has a lot of fragmentation pages.
Quoted 5 lines
Quoted 5 lines
// This is the THP number after this command in the host. // Some THP is used by VM because usemem will access 800M memory // in the guest.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 911360 kB
These lines are from host.
Quoted 7 lines
Quoted 7 lines
// Connect to the QEMU monitor, setup balloon, and set it size to 600M. (qemu)
device_addvirtio-balloon-pci,id=balloon1,cont-pages=on (qemu) info balloon
balloon: actual=1024
(qemu) balloon 600
(qemu) info balloon
balloon: actual=600
These lines are from host.
Quoted 7 lines
Quoted 7 lines
// This is the THP number after inflate the balloon in the host. cat
/proc/meminfo| grep AnonHugePages:
AnonHugePages: 616448 kB
// Set the size back to 1024M in the QEMU monitor. (qemu) balloon 1024
(qemu) info balloon
balloon: actual=1024
These lines are from host.
Quoted 3 lines
Quoted 3 lines
// Use usemem to increase the memory usage of QEMU. killall usemem
usemem 800m
These lines are from guest.
Quoted 3 lines
Quoted 3 lines
// This is the THP number after this operation.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 907264 kB
These lines are from host.
Quoted 5 lines
I'm a bit confused about which of the above run within guest, and which run within host. Could you explain pls?
I added some introduction to show where these lines is get from.
Best,
Hui
Quoted 22 lines
Quoted 12 lines
[1] https://lkml.org/lkml/2020/3/12/144
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lkml.org/lkml/2020/5/12/324
[4] https://github.com/teawater/linux/tree/balloon_conts
[5] https://github.com/teawater/qemu/tree/balloon_conts
[6] https://lkml.org/lkml/2020/5/13/1211Hui Zhu (2):
virtio_balloon: AddVIRTIO_BALLOON_F_CONT_PAGESandinflate_cont_vq
virtio_balloon: Adddeflate_cont_vqto deflate continuous pagesQuoted 5 lines
drivers/virtio/virtio_balloon.c | 180 +++++++++++++++++++++++++++++++----- include/linux/balloon_compaction.h | 12 ++ include/uapi/linux/virtio_balloon.h | 1 mm/balloon_compaction.c | 117 +++++++++++++++++++++-- 4 files changed, 280 insertions(+), 30 deletions(-)---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 136 lines · teawater
Quoted 33 lines
2020年7月16日 14:38,Michael S. Tsirkin <[email protected]> 写道:
Quoted 28 lines · Hui Zhu
The first, second and third version are in [1], [2] and [3].
Code of current version for Linux and qemu is available in [4] and [5]. Update of this version:
1. Report continuous pages will increase the speed. So added deflate continuous pages.
2. According to the comments from David in [6], added 2 new vqsinflate_cont_vqanddeflate_cont_vqto report continuous pages with format 32 bits pfn and 32 bits size.
Following is the introduction of the function.
These patches addVIRTIO_BALLOON_F_CONT_PAGEStovirtio_balloon. With this flag, balloon tries to use continuous pages to inflate and deflate. Opening this flag can bring two benefits:
1. Report continuous pages will increase memory report size of each time calltell_host. Then it will increase the speed of balloon inflate and deflate.
2. Host THPs will be splitted when qemu release the page of balloon inflate. Inflate balloon with continuous pages will let QEMU release the pages of same THPs. That will help decrease the splitted THPs number in the host.
Following is an example in a VM with 1G memory 1CPU. This test setups an environment that has a lot of fragmentation pages. Then inflate balloon will split the THPs.// This is the THP number before VM execution in the host. // None use THP.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 0 kBThese lines are from host.
Quoted 5 lines
Quoted 5 lines
// After VM start, use usemem
// (https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git)
// punch-holes function generates 400m fragmentation pages in the guest // kernel.
usemem --punch-holes -s -1 800m &These lines are from guest. They setups the environment that has a lot of fragmentation pages.
Quoted 5 lines
Quoted 5 lines
// This is the THP number after this command in the host. // Some THP is used by VM because usemem will access 800M memory // in the guest.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 911360 kBThese lines are from host.
Quoted 7 lines
Quoted 7 lines
// Connect to the QEMU monitor, setup balloon, and set it size to 600M. (qemu)
device_addvirtio-balloon-pci,id=balloon1
(qemu) info balloon
balloon: actual=1024
(qemu) balloon 600
(qemu) info balloon
balloon: actual=600These lines are from host.
Quoted 3 lines
Quoted 3 lines
// This is the THP number after inflate the balloon in the host. cat
/proc/meminfo| grep AnonHugePages:
AnonHugePages: 88064 kBThese lines are from host.
Quoted 4 lines
Quoted 4 lines
// Set the size back to 1024M in the QEMU monitor. (qemu) balloon 1024
(qemu) info balloon
balloon: actual=1024These lines are from host.
Quoted 3 lines
Quoted 3 lines
// Use usemem to increase the memory usage of QEMU. killall usemem
usemem 800mThese lines are from guest.
Quoted 3 lines
Quoted 3 lines
// This is the THP number after this operation.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 65536 kBThese lines are from host.
Quoted 7 lines
Quoted 7 lines
Following example change to use continuous pages balloon. The number of splitted THPs is decreased.
// This is the THP number before VM execution in the host. // None use THP.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 0 kBThese lines are from host.
Quoted 3 lines
Quoted 3 lines
// After VM start, use usemem punch-holes function generates 400M // fragmentation pages in the guest kernel.
usemem --punch-holes -s -1 800m &These lines are from guest. They setups the environment that has a lot of fragmentation pages.
Quoted 5 lines
Quoted 5 lines
// This is the THP number after this command in the host. // Some THP is used by VM because usemem will access 800M memory // in the guest.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 911360 kBThese lines are from host.
Quoted 7 lines
Quoted 7 lines
// Connect to the QEMU monitor, setup balloon, and set it size to 600M. (qemu)
device_addvirtio-balloon-pci,id=balloon1,cont-pages=on (qemu) info balloon
balloon: actual=1024
(qemu) balloon 600
(qemu) info balloon
balloon: actual=600These lines are from host.
Quoted 7 lines
Quoted 7 lines
// This is the THP number after inflate the balloon in the host. cat
/proc/meminfo| grep AnonHugePages:
AnonHugePages: 616448 kB
// Set the size back to 1024M in the QEMU monitor. (qemu) balloon 1024
(qemu) info balloon
balloon: actual=1024These lines are from host.
Quoted 3 lines
Quoted 3 lines
// Use usemem to increase the memory usage of QEMU. killall usemem
usemem 800mThese lines are from guest.
Quoted 3 lines
Quoted 3 lines
// This is the THP number after this operation.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 907264 kBThese lines are from host.
Quoted 5 lines
I'm a bit confused about which of the above run within guest, and which run within host. Could you explain pls?
Quoted 4 lines
I added some introduction to show where these lines is get from.
Best,
Hui
OK so we see host has more free THPs. But guest has presumably less now - so the total page table depth is the same. Did we gain anything?
Quoted 23 lines
Quoted 23 lines
Quoted 13 lines
[1] https://lkml.org/lkml/2020/3/12/144
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lkml.org/lkml/2020/5/12/324
[4] https://github.com/teawater/linux/tree/balloon_conts
[5] https://github.com/teawater/qemu/tree/balloon_conts
[6] https://lkml.org/lkml/2020/5/13/1211Hui Zhu (2):
virtio_balloon: AddVIRTIO_BALLOON_F_CONT_PAGESandinflate_cont_vq
virtio_balloon: Adddeflate_cont_vqto deflate continuous pagesQuoted 5 lines
drivers/virtio/virtio_balloon.c | 180 +++++++++++++++++++++++++++++++----- include/linux/balloon_compaction.h | 12 ++ include/uapi/linux/virtio_balloon.h | 1 mm/balloon_compaction.c | 117 +++++++++++++++++++++-- 4 files changed, 280 insertions(+), 30 deletions(-)---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 141 lines
2020年7月16日 18:45,Michael S. Tsirkin <[email protected]> 写道:
Quoted 136 lines · teawater
Quoted 33 lines
2020年7月16日 14:38,Michael S. Tsirkin <[email protected]> 写道:
Quoted 28 lines · Hui Zhu
The first, second and third version are in [1], [2] and [3].
Code of current version for Linux and qemu is available in [4] and [5]. Update of this version:
1. Report continuous pages will increase the speed. So added deflate continuous pages.
2. According to the comments from David in [6], added 2 new vqsinflate_cont_vqanddeflate_cont_vqto report continuous pages with format 32 bits pfn and 32 bits size.
Following is the introduction of the function.
These patches addVIRTIO_BALLOON_F_CONT_PAGEStovirtio_balloon. With this flag, balloon tries to use continuous pages to inflate and deflate. Opening this flag can bring two benefits:
1. Report continuous pages will increase memory report size of each time calltell_host. Then it will increase the speed of balloon inflate and deflate.
2. Host THPs will be splitted when qemu release the page of balloon inflate. Inflate balloon with continuous pages will let QEMU release the pages of same THPs. That will help decrease the splitted THPs number in the host.
Following is an example in a VM with 1G memory 1CPU. This test setups an environment that has a lot of fragmentation pages. Then inflate balloon will split the THPs.// This is the THP number before VM execution in the host. // None use THP.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 0 kBThese lines are from host.
Quoted 5 lines
Quoted 5 lines
// After VM start, use usemem
// (https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git)
// punch-holes function generates 400m fragmentation pages in the guest // kernel.
usemem --punch-holes -s -1 800m &These lines are from guest. They setups the environment that has a lot of fragmentation pages.
Quoted 5 lines
Quoted 5 lines
// This is the THP number after this command in the host. // Some THP is used by VM because usemem will access 800M memory // in the guest.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 911360 kBThese lines are from host.
Quoted 7 lines
Quoted 7 lines
// Connect to the QEMU monitor, setup balloon, and set it size to 600M. (qemu)
device_addvirtio-balloon-pci,id=balloon1
(qemu) info balloon
balloon: actual=1024
(qemu) balloon 600
(qemu) info balloon
balloon: actual=600These lines are from host.
Quoted 3 lines
Quoted 3 lines
// This is the THP number after inflate the balloon in the host. cat
/proc/meminfo| grep AnonHugePages:
AnonHugePages: 88064 kBThese lines are from host.
Quoted 4 lines
Quoted 4 lines
// Set the size back to 1024M in the QEMU monitor. (qemu) balloon 1024
(qemu) info balloon
balloon: actual=1024These lines are from host.
Quoted 3 lines
Quoted 3 lines
// Use usemem to increase the memory usage of QEMU. killall usemem
usemem 800mThese lines are from guest.
Quoted 3 lines
Quoted 3 lines
// This is the THP number after this operation.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 65536 kBThese lines are from host.
Quoted 7 lines
Quoted 7 lines
Following example change to use continuous pages balloon. The number of splitted THPs is decreased.
// This is the THP number before VM execution in the host. // None use THP.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 0 kBThese lines are from host.
Quoted 3 lines
Quoted 3 lines
// After VM start, use usemem punch-holes function generates 400M // fragmentation pages in the guest kernel.
usemem --punch-holes -s -1 800m &These lines are from guest. They setups the environment that has a lot of fragmentation pages.
Quoted 5 lines
Quoted 5 lines
// This is the THP number after this command in the host. // Some THP is used by VM because usemem will access 800M memory // in the guest.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 911360 kBThese lines are from host.
Quoted 7 lines
Quoted 7 lines
// Connect to the QEMU monitor, setup balloon, and set it size to 600M. (qemu)
device_addvirtio-balloon-pci,id=balloon1,cont-pages=on (qemu) info balloon
balloon: actual=1024
(qemu) balloon 600
(qemu) info balloon
balloon: actual=600These lines are from host.
Quoted 7 lines
Quoted 7 lines
// This is the THP number after inflate the balloon in the host. cat
/proc/meminfo| grep AnonHugePages:
AnonHugePages: 616448 kB
// Set the size back to 1024M in the QEMU monitor. (qemu) balloon 1024
(qemu) info balloon
balloon: actual=1024These lines are from host.
Quoted 3 lines
Quoted 3 lines
// Use usemem to increase the memory usage of QEMU. killall usemem
usemem 800mThese lines are from guest.
Quoted 3 lines
Quoted 3 lines
// This is the THP number after this operation.
cat/proc/meminfo| grep AnonHugePages:
AnonHugePages: 907264 kBThese lines are from host.
Quoted 5 lines
I'm a bit confused about which of the above run within guest, and which run within host. Could you explain pls?
Quoted 9 lines
Quoted 4 lines
I added some introduction to show where these lines is get from.
Best,
HuiOK so we see host has more free THPs. But guest has presumably less now - so the total page table depth is the same. Did we gain anything?
cat /proc/meminfo | grep AnonHugePages:
This command will output how many THPs is used by current system. There is no program using THPs except qemu.
So this command will show how many THPs is used by qemu.
The last outout of “cat /proc/meminfo | grep AnonHugePages:” show how many THPs is used by qemu when this 2 qemu’s anon page number is same. Without “cont-pages=on”, qemu keep 65536kb THPs.
Wiht “cont-pages=on”, qemu keep 907264kb THPs.
Keep more THPs will make memory access speed high.
This is a test record use this 1G 1 cpu qemu after the fragmentation balloon test:
Without “cont-pages=on”, qemu keep 81920kB THPs.
/ # usemem 800m
943718400 bytes / 489412 usecs = 1883076 KB/s
18725 usecs to free memory
/ # usemem 800m
943718400 bytes / 487070 usecs = 1892130 KB/s
18913 usecs to free memory
/ # usemem 800m
943718400 bytes / 484234 usecs = 1903212 KB/s
18538 usecs to free memory
/ # usemem 800m
943718400 bytes / 486568 usecs = 1894082 KB/s
18982 usecs to free memory
With “cont-pages=on”, qemu keep 907264kb THPs.
/ # usemem 800m
943718400 bytes / 479098 usecs = 1923614 KB/s
18980 usecs to free memory
/ # usemem 800m
943718400 bytes / 477433 usecs = 1930323 KB/s
18562 usecs to free memory
/ # usemem 800m
943718400 bytes / 479790 usecs = 1920840 KB/s
18663 usecs to free memory
/ # usemem 800m
943718400 bytes / 480253 usecs = 1918988 KB/s
19011 usecs to free memory
Best,
Hui
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 13 lines
[1] https://lkml.org/lkml/2020/3/12/144
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lkml.org/lkml/2020/5/12/324
[4] https://github.com/teawater/linux/tree/balloon_conts
[5] https://github.com/teawater/qemu/tree/balloon_conts
[6] https://lkml.org/lkml/2020/5/13/1211Hui Zhu (2):
virtio_balloon: AddVIRTIO_BALLOON_F_CONT_PAGESandinflate_cont_vqvirtio_balloon: Adddeflate_cont_vqto deflate continuous pagesQuoted 5 lines
drivers/virtio/virtio_balloon.c | 180 +++++++++++++++++++++++++++++++----- include/linux/balloon_compaction.h | 12 ++ include/uapi/linux/virtio_balloon.h | 1 mm/balloon_compaction.c | 117 +++++++++++++++++++++-- 4 files changed, 280 insertions(+), 30 deletions(-)---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]