vmw_balloon: addressing bugs and issues
Subsystems (3)
v3 cover vmw_balloon: addressing bugs and issues
All of these patches are intended for 4.18:
- Patches 1-5 are bug fixes.
- Patch 6 changes the GPL comment. It is actually required just to remove the maintainers list from the source file (since it has changed).
- Patch 7 is an update to the maintainer list.
This patch-set addresses minor issues and bugs in the VMware balloon driver. Patches 1-5 address bugs that were *not* reported by customers, but we would like to get them to 4.18 if possible.
Patches 1-4 are cc'd to stable. Patch 5 might be too big to be considered for stable.
We will send separately later a patch-set with new features to be set on top of this set, which will be intended for 4.19.
v2->v3: * Removing empty lines (Greg)
* Removing the GPL message entirely (Greg)
v1->v2: * Reordering patches to have bug fixes first (Greg)
* Fixing SPDX tag
* Cc'ing stable when appropriate
Nadav Amit (7):
vmw_balloon: fix inflation of 64-bit GFNs
vmw_balloon: do not use 2MB without batching
vmw_balloon: VMCI_DOORBELL_SET does not check status
vmw_balloon: fix VMCI use when balloon built into kernel
vmw_balloon: remove inflation rate limiting
vmw_balloon: update copyright message
vmw_balloon: update maintainers list
MAINTAINERS | 2 +- drivers/misc/vmw_balloon.c | 181 +++++++++++-------------------------- 2 files changed, 55 insertions(+), 128 deletions(-)
Signature · 1 line
2.17.0
PATCH v3 1/7 vmw_balloon: fix inflation of 64-bit GFNs
When balloon batching is not supported by the hypervisor, the guest frame number (GFN) must fit in 32-bit. However, due to a bug, this check was mistakenly ignored. In practice, when total RAM is greater than 16TB, the balloon does not work currently, making this bug unlikely to happen.
--- drivers/misc/vmw_balloon.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
drivers/misc/vmw_balloon.c +7 -6
@@ -450,7 +450,7 @@ static int vmballoon_send_lock_page(struct vmballoon *b, unsigned long pfn, pfn32 = (u32 )pfn;if (pfn32 != pfn) -return -1 ; +return -EINVAL ;STATS_INC (b->stats.lock[false]);
@@ -460,7 +460,7 @@ static int vmballoon_send_lock_page(struct vmballoon *b, unsigned long pfn,pr_debug ("%s - ppn %lx, hv returns %ld\n" , __func__, pfn, status);STATS_INC (b->stats.lock_fail[false]); -return 1 ; +return -EIO ; } static int vmballoon_send_batched_lock(struct vmballoon *b,
@@ -597,11 +597,12 @@ static int vmballoon_lock_page(struct vmballoon *b, unsigned int num_pages, locked =vmballoon_send_lock_page (b,page_to_pfn (page), &hv_status, target); -if (locked >0 ) { +if (locked) {STATS_INC (b->stats.refused_alloc[false]); -if (hv_status ==VMW_BALLOON_ERROR_RESET || - hv_status ==VMW_BALLOON_ERROR_PPN_NOTNEEDED ) { +if (locked == -EIO && + (hv_status ==VMW_BALLOON_ERROR_RESET || + hv_status ==VMW_BALLOON_ERROR_PPN_NOTNEEDED )) {vmballoon_free_page (page, false);return -EIO ; }
@@ -617,7 +618,7 @@ static int vmballoon_lock_page(struct vmballoon *b, unsigned int num_pages, }else {vmballoon_free_page (page, false); } -return -EIO ; +return locked; }/* track allocated page */
Signature · 1 line
2.17.0
PATCH v3 2/7 vmw_balloon: do not use 2MB without batching
In reply to Nadav AmitIf the hypervisor sets 2MB batching is on, while batching is cleared, the balloon code breaks. In this case the legacy mechanism is used with 2MB page. The VM would report a 2MB page is ballooned, and the hypervisor would only take the first 4KB.
While the hypervisor should not report such settings, make the code more robust by not enabling 2MB support without batching.
--- drivers/misc/vmw_balloon.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
drivers/misc/vmw_balloon.c +7 -1
@@ -341,7 +341,13 @@ static bool vmballoon_send_start(struct vmballoon *b, unsigned long req_caps) success = false; } -if (b->capabilities &VMW_BALLOON_BATCHED_2M_CMDS ) +/* +* 2MB pages are only supported with batching. If batching is for some +* reason disabled, do not use 2MB pages, since otherwise the legacy +* mechanism is used with 2MB pages, causing a failure. +*/ +if ((b->capabilities &VMW_BALLOON_BATCHED_2M_CMDS ) && + (b->capabilities &VMW_BALLOON_BATCHED_CMDS )) b->supported_page_sizes =2 ;else b->supported_page_sizes =1 ;
Signature · 1 line
2.17.0
PATCH v3 3/7 vmw_balloon: VMCI_DOORBELL_SET does not check status
In reply to Nadav AmitWhen vmballoon_vmci_init() sets a doorbell using VMCI_DOORBELL_SET, for some reason it does not consider the status and looks at the result. However, the hypervisor does not update the result - it updates the status. This might cause VMCI doorbell not to be enabled, resulting in degraded performance.
--- drivers/misc/vmw_balloon.c | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-)
drivers/misc/vmw_balloon.c +19 -18
@@ -1036,29 +1036,30 @@ static void vmballoon_vmci_cleanup(struct vmballoon *b)*/ static int vmballoon_vmci_init (struct vmballoon *b) { -int error =0 ; +unsigned long error, dummy; -if ((b->capabilities &VMW_BALLOON_SIGNALLED_WAKEUP_CMD ) !=0 ) { - error =vmci_doorbell_create (&b->vmci_doorbell, -VMCI_FLAG_DELAYED_CB , -VMCI_PRIVILEGE_FLAG_RESTRICTED , - vmballoon_doorbell, b); - -if (error ==VMCI_SUCCESS ) { -VMWARE_BALLOON_CMD (VMCI_DOORBELL_SET , - b->vmci_doorbell.context, - b->vmci_doorbell.resource, error); -STATS_INC (b->stats.doorbell_set); - } - } +if ((b->capabilities &VMW_BALLOON_SIGNALLED_WAKEUP_CMD ) ==0 ) +return 0 ; -if (error !=0 ) { -vmballoon_vmci_cleanup (b); + error =vmci_doorbell_create (&b->vmci_doorbell,VMCI_FLAG_DELAYED_CB , +VMCI_PRIVILEGE_FLAG_RESTRICTED , + vmballoon_doorbell, b); -return -EIO ; - } +if (error !=VMCI_SUCCESS ) + goto fail; + + error =VMWARE_BALLOON_CMD (VMCI_DOORBELL_SET , b->vmci_doorbell.context, + b->vmci_doorbell.resource, dummy); + +STATS_INC (b->stats.doorbell_set); + +if (error !=VMW_BALLOON_SUCCESS ) + goto fail;return 0 ; +fail: +vmballoon_vmci_cleanup (b); +return -EIO ; }/*
Signature · 1 line
2.17.0
PATCH v3 4/7 vmw_balloon: fix VMCI use when balloon built into kernel
In reply to Nadav AmitCurrently, when all modules, including VMCI and VMware balloon are built into the kernel, the initialization of the balloon happens before the VMCI is probed. As a result, the balloon fails to initialize the VMCI doorbell, which it uses to get asynchronous requests for balloon size changes.
The problem can be seen in the logs, in the form of the following message:
"vmw_balloon: failed to initialize vmci doorbell"
The driver would work correctly but slightly less efficiently, probing for requests periodically. This patch changes the balloon to be initialized using late_initcall() instead of module_init() to address this issue. It does not address a situation in which VMCI is built as a module and the balloon is built into the kernel.
--- drivers/misc/vmw_balloon.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/misc/vmw_balloon.c +8 -1
@@ -1297,7 +1297,14 @@ static int __init vmballoon_init(void)return 0 ; } -module_init (vmballoon_init); + +/* +* Using late_initcall() instead of module_init() allows the balloon to use the +* VMCI doorbell even when the balloon is built into the kernel. Otherwise the +* VMCI is probed only after the balloon is initialized. If the balloon is used +* as a module, late_initcall() is equivalent to module_init(). +*/ +late_initcall (vmballoon_init); static void __exit vmballoon_exit(void) {
Signature · 1 line
2.17.0
PATCH v3 5/7 vmw_balloon: remove inflation rate limiting
In reply to Nadav AmitSince commit 33d268ed0019 ("VMware balloon: Do not limit the amount of frees and allocations in non-sleep mode."), the allocations are not increased, and therefore balloon inflation rate limiting is in practice broken.
While we can restore rate limiting, in practice we see that it can result in adverse effect, as the hypervisor throttles down the VM if it does not respond well enough, or alternatively causes it to perform very poorly as the host swaps out the VM memory. Throttling the VM down can even have a cascading effect, in which the VM reclaims memory even slower and consequentially throttled down even further.
We therefore remove all the rate limiting mechanisms, including the slow allocation cycles, as they are likely to do more harm than good.
--- drivers/misc/vmw_balloon.c | 92 +++++--------------------------------- 1 file changed, 11 insertions(+), 81 deletions(-)
drivers/misc/vmw_balloon.c +11 -81
@@ -54,25 +54,6 @@ MODULE_ALIAS("dmi:*:svnVMware*:*");MODULE_ALIAS ("vmware_vmmemctl" );MODULE_LICENSE ("GPL" ); -/* -* Various constants controlling rate of inflaint/deflating balloon, -* measured in pages. -*/ - -/* -* Rates of memory allocaton when guest experiences memory pressure -* (driver performs sleeping allocations). -*/ -#define VMW_BALLOON_RATE_ALLOC_MIN 512U -#define VMW_BALLOON_RATE_ALLOC_MAX 2048U -#define VMW_BALLOON_RATE_ALLOC_INC 16U - -/* -* When guest is under memory pressure, use a reduced page allocation -* rate for next several cycles. -*/ -#define VMW_BALLOON_SLOW_CYCLES 4 -/* * Use __GFP_HIGHMEM to allow pages from HIGHMEM zone. We don't * allow wait (__GFP_RECLAIM) for NOSLEEP page allocations. Use
@@ -284,12 +265,6 @@ struct vmballoon {/* reset flag */ bool reset_required; -/* adjustment rates (pages per second) */ -unsigned int rate_alloc; - -/* slowdown page allocations for next few cycles */ -unsigned int slow_allocation_cycles; -unsigned long capabilities;struct vmballoon_batch_page *batch_page;
@@ -797,8 +772,6 @@ static void vmballoon_add_batched_page(struct vmballoon *b, int idx,*/ static void vmballoon_inflate (struct vmballoon *b) { -unsigned rate; -unsigned int allocations =0 ;unsigned int num_pages =0 ;int error =0 ;gfp_t flags =VMW_PAGE_ALLOC_NOSLEEP ;
@@ -825,17 +798,9 @@ static void vmballoon_inflate(struct vmballoon *b)* Start with no sleep allocation rate which may be higher * than sleeping allocation rate. */ -if (b->slow_allocation_cycles) { - rate = b->rate_alloc; - is_2m_pages = false; - }else { - rate =UINT_MAX ; - is_2m_pages = - b->supported_page_sizes ==VMW_BALLOON_NUM_PAGE_SIZES ; - } + is_2m_pages = b->supported_page_sizes ==VMW_BALLOON_NUM_PAGE_SIZES ; -pr_debug ("%s - goal: %d, no-sleep rate: %u, sleep rate: %d\n" , - __func__, b->target - b->size, rate, b->rate_alloc); +pr_debug ("%s - goal: %d" , __func__, b->target - b->size);while (!b->reset_required && b->size + num_pages * vmballoon_page_size(is_2m_pages)
@@ -868,31 +833,24 @@ static void vmballoon_inflate(struct vmballoon *b)if (flags ==VMW_PAGE_ALLOC_CANSLEEP ) {/* * CANSLEEP page allocation failed, so guest -* is under severe memory pressure. Quickly -* decrease allocation rate. +* is under severe memory pressure. We just log +* the event, but do not stop the inflation +* due to its negative impact on performance. */ - b->rate_alloc =max (b->rate_alloc /2 , -VMW_BALLOON_RATE_ALLOC_MIN );STATS_INC (b->stats.sleep_alloc_fail);break ; }/* * NOSLEEP page allocation failed, so the guest is -* under memory pressure. Let us slow down page -* allocations for next few cycles so that the guest -* gets out of memory pressure. Also, if we already -* allocated b->rate_alloc pages, let's pause, -* otherwise switch to sleeping allocations. +* under memory pressure. Slowing down page alloctions +* seems to be reasonable, but doing so might actually +* cause the hypervisor to throttle us down, resulting +* in degraded performance. We will count on the +* scheduler and standard memory management mechanisms +* for now. */ - b->slow_allocation_cycles =VMW_BALLOON_SLOW_CYCLES ; - -if (allocations >= b->rate_alloc) -break ; - flags =VMW_PAGE_ALLOC_CANSLEEP ; -/* Lower rate for sleeping allocations. */ - rate = b->rate_alloc;continue ; }
@@ -906,28 +864,11 @@ static void vmballoon_inflate(struct vmballoon *b) }cond_resched (); - -if (allocations >= rate) { -/* We allocated enough pages, let's take a break. */ -break ; - } }if (num_pages >0 ) b->ops->lock (b, num_pages, is_2m_pages, &b->target); -/* -* We reached our goal without failures so try increasing -* allocation rate. -*/ -if (error ==0 && allocations >= b->rate_alloc) { -unsigned int mult = allocations / b->rate_alloc; - - b->rate_alloc = -min (b->rate_alloc + mult *VMW_BALLOON_RATE_ALLOC_INC , -VMW_BALLOON_RATE_ALLOC_MAX ); - } -vmballoon_release_refused_pages (b, true);vmballoon_release_refused_pages (b, false); }
@@ -1122,9 +1063,6 @@ static void vmballoon_work(struct work_struct *work)if (b->reset_required)vmballoon_reset (b); -if (b->slow_allocation_cycles >0 ) - b->slow_allocation_cycles--; -if (!b->reset_required &&vmballoon_send_get_target (b, &target)) {/* update target, adjust size */ b->target = target;
@@ -1168,11 +1106,6 @@ static int vmballoon_debug_show(struct seq_file *f, void *offset)"current: %8d pages\n" , b->target, b->size); -/* format rate info */ -seq_printf (f, -"rateSleepAlloc: %8d pages/sec\n" , - b->rate_alloc); -seq_printf (f,"\n" "timer: %8u\n"
@@ -1279,9 +1212,6 @@ static int __init vmballoon_init(void)INIT_LIST_HEAD (&balloon.page_sizes[is_2m_pages].refused_pages); } -/* initialize rates */ - balloon.rate_alloc =VMW_BALLOON_RATE_ALLOC_MAX ; -INIT_DELAYED_WORK (&balloon.dwork, vmballoon_work); error =vmballoon_debugfs_init (&balloon);
Signature · 1 line
2.17.0
PATCH v3 6/7 vmw_balloon: update copyright message
In reply to Nadav AmitRemoving the GPL wording and replace it with an SPDX tag. The immediate trigger for doing it now is the need to remove the list of maintainers from the source file, as the maintainer list changed.
--- drivers/misc/vmw_balloon.c | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-)
drivers/misc/vmw_balloon.c +2 -20
@@ -1,27 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * VMware Balloon driver. * -* Copyright (C) 2000-2014, VMware, Inc. All Rights Reserved. +* Copyright (C) 2000-2018, VMware, Inc. All Rights Reserved. * -* This program is free software; you can redistribute it and/or modify it -* under the terms of the GNU General Public License as published by the -* Free Software Foundation; version 2 of the License and no later version. -* -* This program is distributed in the hope that it will be useful, but -* WITHOUT ANY WARRANTY; without even the implied warranty of -* MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or -* NON INFRINGEMENT. See the GNU General Public License for more -* details. -* -* You should have received a copy of the GNU General Public License -* along with this program; if not, write to the Free Software -* Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. -* -* Maintained by: Xavier Deguillard <[email protected]> -* Philip Moltmann <[email protected]> -*/ - -/* * This is VMware physical memory management driver for Linux. The driver * acts like a "balloon" that can be inflated to reclaim physical pages by * reserving them in the guest and invalidating them in the monitor,
Signature · 1 line
2.17.0
PATCH v3 7/7 vmw_balloon: update maintainers list
In reply to Nadav AmitPhilip Moltman is no longer a maintainer of the VMware balloon. Setting Nadav Amit as one instead.
--- MAINTAINERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
MAINTAINERS +1 -1
@@ -15294,7 +15294,7 @@ F: include/linux/vme* VMWARE BALLOON DRIVER M: Xavier Deguillard <[email protected]> -M: Philip Moltmann <[email protected]> +M: Nadav Amit <[email protected]> M: "VMware, Inc." <[email protected]> L: [email protected] S: Maintained
Signature · 1 line
2.17.0
Quoted 35 lines · Nadav Amit
All of these patches are intended for 4.18:
- Patches 1-5 are bug fixes.
- Patch 6 changes the GPL comment. It is actually required just to remove the maintainers list from the source file (since it has changed).
- Patch 7 is an update to the maintainer list.This patch-set addresses minor issues and bugs in the VMware balloon driver. Patches 1-5 address bugs that were *not* reported by customers, but we would like to get them to 4.18 if possible.
Patches 1-4 are cc'd to stable. Patch 5 might be too big to be considered for stable.
We will send separately later a patch-set with new features to be set on top of this set, which will be intended for 4.19.
v2->v3: * Removing empty lines (Greg)
* Removing the GPL message entirely (Greg)
v1->v2: * Reordering patches to have bug fixes first (Greg)
* Fixing SPDX tag
* Cc'ing stable when appropriateNadav Amit (7):
vmw_balloon: fix inflation of 64-bit GFNs
vmw_balloon: do not use 2MB without batching
vmw_balloon:VMCI_DOORBELL_SETdoes not check status
vmw_balloon: fix VMCI use when balloon built into kernel
vmw_balloon: remove inflation rate limiting
vmw_balloon: update copyright message
vmw_balloon: update maintainers list
Quoted 4 lines
MAINTAINERS| 2 +-drivers/misc/vmw_balloon.c| 181 +++++++++++--------------------------2 files changed, 55 insertions(+), 128 deletions(-)
Greg,
Any reservations regarding this version?
Thanks,
Nadav