Enhance the hv_vmbus driver to support hibernation
Subsystems (5)
Dexuan Cui
maintainerv4 cover Enhance the hv_vmbus driver to support hibernation
Hi all,
The patchset is to enhance hv_vmbus to support hibernation when Linux VM runs on Hyper-V. A second patchset to enhance the high-level VSC drivers (hv_netvsc, hv_storvsc, etc.) for hibernation will be posted after this patchset is acceped. If you want to test this hibernation feaure, all the needed patches can be found on my github branch:
https://github.com/dcui/linux/commits/decui/hibernation/2019-0830/v4/v5.3-rc6-plus
This patchset is based on v5.3-rc6, and can also cleanly apply to -rc7.
Please review.
Hi tglx,
I hope all the 12 patchset, including the below 3 patches,
[PATCH v4 01/12] x86/hyper-v: Suspend/resume the hypercall page for hibernation [PATCH v4 02/12] x86/hyper-v: Implement hv_is_hibernation_supported() [PATCH v4 03/12] clocksource/drivers: Suspend/resume Hyper-V
can go through Sasha's hyperv/linux.git tree, since all the changes belong to the hv stuff. However, if you think it's better for these 3 patches to go through your tip.git tree, it also works for me.
Hi Michael,
I added your Reviewed-by's for patch 1, 3~7, 9~10, since you have reviewed these patches. Please review the v4 of the others: 2, 8, 11 and 12.
Thanks,
Dexuan
Changes in v4:
Patch 2: Enhanced the changelog.
Patch 8: Moved find_primary_channel_by_offer() to channel_mgmt.c, where
it's used.
Patch 10: Improved the changelog (typo) and comment.
Patch 11: Removed WARN_ON_ONCE(), added a comment for the case
nr_chan_close_on_suspend == 0.
Patch 12: Improved the comment.
Changes in v3:
Patch 2: Add a new API hv_is_hibernation_supported().
Patch 6: Add a new helper function is_sub_channel().
Patch 8: Find the old channels via Instance GUID rather than the RELID.
Patch 10: Add code to clean up hv_sock channels by force
Patch 11: Add code to wait in the suspend path: all the hv_sock channels
and sub-channels should be cleaned up first before Linux sends the VMBUS UNLOAD message.
Patch 12: Add code to fix up the old primary channels before further
resuming.
Changes in v2:
Patch 3: Improved the changelog and added a comment.
Patch 4: Remove the "hv_stimer_cleanup" in hv_synic_suspend(), because I
suppose https://lkml.org/lkml/2019/7/27/5 will be accepted. Also improved changelog and the comment.
Patch 5: Fixed the third argument of print_hex_dump_debug(). Also improved
the changelog.
Patch 6: Improved the changelog and the comment. Added a check for the
'vmbus_proto_version' in vmbus_bus_resume().
Dexuan Cui (12):
x86/hyper-v: Suspend/resume the hypercall page for hibernation x86/hyper-v: Implement hv_is_hibernation_supported() clocksource/drivers: Suspend/resume Hyper-V clocksource for
hibernation
Drivers: hv: vmbus: Break out synic enable and disable operations
Drivers: hv: vmbus: Suspend/resume the synic for hibernation
Drivers: hv: vmbus: Add a helper function is_sub_channel()
Drivers: hv: vmbus: Implement suspend/resume for VSC drivers for
hibernation
Drivers: hv: vmbus: Ignore the offers when resuming from hibernation
Drivers: hv: vmbus: Suspend/resume the vmbus itself for hibernation
Drivers: hv: vmbus: Clean up hv_sock channels by force upon suspend
Drivers: hv: vmbus: Suspend after cleaning up hv_sock and sub channels
Drivers: hv: vmbus: Resume after fixing up old primary channels
arch/x86/hyperv/hv_init.c | 41 ++++++ drivers/clocksource/hyperv_timer.c | 25 ++++ drivers/hv/channel_mgmt.c | 161 +++++++++++++++++++--- drivers/hv/connection.c | 8 +- drivers/hv/hv.c | 66 +++++---- drivers/hv/hyperv_vmbus.h | 30 +++++ drivers/hv/vmbus_drv.c | 265 +++++++++++++++++++++++++++++++++++++ include/asm-generic/mshyperv.h | 2 + include/linux/hyperv.h | 16 ++- 9 files changed, 565 insertions(+), 49 deletions(-)
Signature · 1 line
1.8.3.1
Dexuan Cui
maintainerPATCH v4 1/12 x86/hyper-v: Suspend/resume the hypercall page for hibernation
This is needed for hibernation, e.g. when we resume the old kernel, we need to disable the "current" kernel's hypercall page and then resume the old kernel's.
--- arch/x86/hyperv/hv_init.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
arch/x86/hyperv/hv_init.c +34 -0
@@ -20,6 +20,7 @@#include <linux/hyperv.h> #include <linux/slab.h> #include <linux/cpuhotplug.h> +#include <linux/syscore_ops.h> #include <clocksource/hyperv_timer.h> void *hv_hypercall_pg;
@@ -223,6 +224,34 @@ static int __init hv_pci_init(void)return 1 ; } +static int hv_suspend (void ) +{ +union hv_x64_msr_hypercall_contents hypercall_msr; + +/* Reset the hypercall page */ +rdmsrl (HV_X64_MSR_HYPERCALL , hypercall_msr.as_uint64); + hypercall_msr.enable =0 ; +wrmsrl (HV_X64_MSR_HYPERCALL , hypercall_msr.as_uint64); + +return 0 ; +} + +static void hv_resume (void ) +{ +union hv_x64_msr_hypercall_contents hypercall_msr; + +/* Re-enable the hypercall page */ +rdmsrl (HV_X64_MSR_HYPERCALL , hypercall_msr.as_uint64); + hypercall_msr.enable =1 ; + hypercall_msr.guest_physical_address =vmalloc_to_pfn (hv_hypercall_pg); +wrmsrl (HV_X64_MSR_HYPERCALL , hypercall_msr.as_uint64); +} + +static struct syscore_ops hv_syscore_ops = { + .suspend = hv_suspend, + .resume = hv_resume, +}; +/* * This function is to be invoked early in the boot sequence after the * hypervisor has been detected.
@@ -303,6 +332,9 @@ void __init hyperv_init(void)/* Register Hyper-V specific clocksource */ hv_init_clocksource (); + +register_syscore_ops (&hv_syscore_ops); +return ; remove_cpuhp_state:
@@ -322,6 +354,8 @@ void hyperv_cleanup(void) {union hv_x64_msr_hypercall_contents hypercall_msr; +unregister_syscore_ops (&hv_syscore_ops); +/* Reset our OS id */ wrmsrl (HV_X64_MSR_GUEST_OS_ID ,0 );
Signature · 1 line
1.8.3.1
Re: [PATCH v4 01/12] x86/hyper-v: Suspend/resume the hypercall page for hibernation
Quoted 6 lines · Dexuan Cui
This is needed for hibernation, e.g. when we resume the old kernel, we need to disable the "current" kernel's hypercall page and then resume the old kernel's.
Signed-off-by Dexuan Cui <[email protected]>Reviewed-by Michael Kelley <[email protected]>
Hi Dexuan,
When sending patches upstream, please make sure you send them to all maintainers and mailing lists that it needs to go to according to MAINTAINERS/get_maintainers.py rather than cherry-picking names off the list.
This is specially important in subsystems like x86 where it's a group maintainers model, and it's very possible that Thomas is sipping margaritas on a beach while one of the other x86 maintainers is covering the tree.
This is quite easy with git-send-email and get_maintainers.py, something like this:
git send-email --cc-cmd="scripts/get_maintainer.pl --separator=, --no-rolestats" your-work.patch
Will do all of that automatically for you.
--
Thanks,
Sasha
Dexuan Cui
maintainerRE: [PATCH v4 01/12] x86/hyper-v: Suspend/resume the hypercall page for hibernation
Quoted 19 lines
From: Sasha Levin <[email protected]>
Sent: Thursday, September 5, 2019 8:44 AMQuoted 3 lines · Dexuan Cui
This is needed for hibernation, e.g. when we resume the old kernel, we need to disable the "current" kernel's hypercall page and then resume the old kernel's.
Hi Dexuan,
When sending patches upstream, please make sure you send them to all maintainers and mailing lists that it needs to go to according to
MAINTAINERS/get_maintainers.pyrather than cherry-picking names off the list.This is specially important in subsystems like x86 where it's a group maintainers model, and it's very possible that Thomas is sipping margaritas on a beach while one of the other x86 maintainers is covering the tree.
Quoted 8 lines
This is quite easy with git-send-email and
get_maintainers.py, something like this:git send-email --cc-cmd="scripts/get_maintainer.pl --separator=,
--no-rolestats" your-work.patchWill do all of that automatically for you.
Sasha
Thanks for the reminder, Sasha!
I didn't know the very useful parameter of git-send-email. :-) I'm going to post v5 with the parameter.
BTW, I'll split v4 into 2 patchsets.
Patchset #1 consists of the first 3 patches of v4, and should go through the tip.git tree (I need to rebase it to the latest timers/core branch due to a conflict).
Patchset #2 consists of the remaining 9 patches and can go through the hyperv tree.
Thanks,
-- Dexuan
Dexuan Cui
maintainerPATCH v4 2/12 x86/hyper-v: Implement hv_is_hibernation_supported()
In reply to Dexuan CuiThe API will be used by the hv_balloon and hv_vmbus drivers.
Balloon up/down and hot-add of memory must not be active if the user wants the Linux VM to support hibernation, because they are incompatible with hibernation according to Hyper-V team, e.g. upon suspend the balloon VSP doesn't save any info about the ballooned-out pages (if any); so, after Linux resumes, Linux balloon VSC expects that the VSP will return the pages if Linux is under memory pressure, but the VSP will never do that, since the VSP thinks it never stole the pages from the VM.
So, if the user wants Linux VM to support hibernation, Linux must forbid balloon up/down and hot-add, and the only functionality of the balloon VSC driver is reporting the VM's memory pressure to the host.
Ideally, when Linux detects that the user wants it to support hibernation, the balloon VSC should tell the VSP that it does not support ballooning and hot-add. However, the current version of the VSP requires the VSC should support these capabilities, otherwise the capability negotiation fails and the VSC can not load at all, so with the later changes to the VSC driver, Linux VM still reports to the VSP that the VSC supports these capabilities, but the VSC ignores the VSP's requests of balloon up/down and hot add, and reports an error to the VSP, when applicable. BTW, in the future the balloon VSP driver will allow the VSC to not support the capabilities of balloon up/down and hot add.
The ACPI S4 state is not a must for hibernation to work, because Linux is able to hibernate as long as the system can shut down. However in practice we decide to artificially use the presence of the virtual ACPI S4 state as an indicator of the user's intent of using hibernation, because Linux VM must find a way to know if the user wants to use the hibernation feature or not.
By default, Hyper-V does not enable the virtual ACPI S4 state; on recent Hyper-V hosts (e.g. RS5, 19H1), the administrator is able to enable the state for a VM by WMI commands.
Once all the vmbus and VSC patches for the hibernation feature are accepted, an extra patch will be submitted to forbid hibernation if the virtual ACPI S4 state is absent, i.e. hv_is_hibernation_supported() is false.
--- arch/x86/hyperv/hv_init.c | 7 +++++++ include/asm-generic/mshyperv.h | 2 ++ 2 files changed, 9 insertions(+)
arch/x86/hyperv/hv_init.c +7 -0
@@ -7,6 +7,7 @@* Author : K. Y. Srinivasan <[email protected]> */ +#include <linux/acpi.h> #include <linux/efi.h> #include <linux/types.h> #include <asm/apic.h>
@@ -453,3 +454,9 @@ bool hv_is_hyperv_initialized(void)return hypercall_msr.enable; }EXPORT_SYMBOL_GPL (hv_is_hyperv_initialized); + +bool hv_is_hibernation_supported (void ) +{ +return acpi_sleep_state_supported (ACPI_STATE_S4 ); +} +EXPORT_SYMBOL_GPL (hv_is_hibernation_supported);
include/asm-generic/mshyperv.h +2 -0
@@ -166,9 +166,11 @@ static inline int cpumask_to_vpset(struct hv_vpset *vpset,void hyperv_report_panic (struct pt_regs *regs,long err);void hyperv_report_panic_msg (phys_addr_t pa,size_t size);bool hv_is_hyperv_initialized (void ); +bool hv_is_hibernation_supported (void );void hyperv_cleanup (void );#else /* CONFIG_HYPERV */ static inline bool hv_is_hyperv_initialized (void ) {return false; } +static inline bool hv_is_hibernation_supported (void ) {return false; }static inline void hyperv_cleanup (void ) {}#endif /* CONFIG_HYPERV */
Signature · 1 line
1.8.3.1
RE: [PATCH v4 02/12] x86/hyper-v: Implement hv_is_hibernation_supported()
From: Dexuan Cui <[email protected]> Sent: Monday, September 2, 2019 5:23 PM
Quoted 42 lines
The API will be used by the
hv_balloonandhv_vmbusdrivers.Balloon up/down and hot-add of memory must not be active if the user wants the Linux VM to support hibernation, because they are incompatible with hibernation according to Hyper-V team, e.g. upon suspend the balloon VSP doesn't save any info about the ballooned-out pages (if any); so, after Linux resumes, Linux balloon VSC expects that the VSP will return the pages if Linux is under memory pressure, but the VSP will never do that, since the VSP thinks it never stole the pages from the VM.
So, if the user wants Linux VM to support hibernation, Linux must forbid balloon up/down and hot-add, and the only functionality of the balloon VSC driver is reporting the VM's memory pressure to the host.
Ideally, when Linux detects that the user wants it to support hibernation, the balloon VSC should tell the VSP that it does not support ballooning and hot-add. However, the current version of the VSP requires the VSC should support these capabilities, otherwise the capability negotiation fails and the VSC can not load at all, so with the later changes to the VSC driver, Linux VM still reports to the VSP that the VSC supports these capabilities, but the VSC ignores the VSP's requests of balloon up/down and hot add, and reports an error to the VSP, when applicable. BTW, in the future the balloon VSP driver will allow the VSC to not support the capabilities of balloon up/down and hot add.
The ACPI S4 state is not a must for hibernation to work, because Linux is able to hibernate as long as the system can shut down. However in practice we decide to artificially use the presence of the virtual ACPI S4 state as an indicator of the user's intent of using hibernation, because Linux VM must find a way to know if the user wants to use the hibernation feature or not.
By default, Hyper-V does not enable the virtual ACPI S4 state; on recent Hyper-V hosts (e.g. RS5, 19H1), the administrator is able to enable the state for a VM by WMI commands.
Once all the vmbus and VSC patches for the hibernation feature are accepted, an extra patch will be submitted to forbid hibernation if the virtual ACPI S4 state is absent, i.e.
hv_is_hibernation_supported()is false.
Quoted 5 lines
Signed-off-by Dexuan Cui <[email protected]>--- arch/x86/hyperv/hv_init.c | 7 +++++++ include/asm-generic/mshyperv.h | 2 ++ 2 files changed, 9 insertions(+)
Dexuan Cui
maintainerPATCH v4 3/12 clocksource/drivers: Suspend/resume Hyper-V clocksource for hibernation
In reply to Dexuan CuiThis is needed for hibernation, e.g. when we resume the old kernel, we need to disable the "current" kernel's TSC page and then resume the old kernel's.
--- drivers/clocksource/hyperv_timer.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
drivers/clocksource/hyperv_timer.c +25 -0
@@ -238,12 +238,37 @@ static u64 read_hv_clock_tsc(struct clocksource *arg)return read_hv_sched_clock_tsc (); } +static void suspend_hv_clock_tsc (struct clocksource *arg) +{ +u64 tsc_msr; + +/* Disable the TSC page */ +hv_get_reference_tsc (tsc_msr); + tsc_msr &= ~BIT_ULL (0 ); +hv_set_reference_tsc (tsc_msr); +} + + +static void resume_hv_clock_tsc (struct clocksource *arg) +{ +phys_addr_t phys_addr =page_to_phys (vmalloc_to_page (tsc_pg)); +u64 tsc_msr; + +/* Re-enable the TSC page */ +hv_get_reference_tsc (tsc_msr); + tsc_msr &=GENMASK_ULL (11 ,0 ); + tsc_msr |=BIT_ULL (0 ) | (u64 )phys_addr; +hv_set_reference_tsc (tsc_msr); +} +static struct clocksource hyperv_cs_tsc = { .name ="hyperv_clocksource_tsc_page" , .rating =400 , .read = read_hv_clock_tsc, .mask =CLOCKSOURCE_MASK (64 ), .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .suspend= suspend_hv_clock_tsc, + .resume = resume_hv_clock_tsc, };#endif
Signature · 1 line
1.8.3.1
Dexuan Cui
maintainerPATCH v4 4/12 Drivers: hv: vmbus: Break out synic enable and disable operations
In reply to Dexuan CuiBreak out synic enable and disable operations into separate hv_synic_disable_regs() and hv_synic_enable_regs() functions for use by a later patch to support hibernation.
There is no functional change except the unnecessary check
"if (sctrl.enable != 1) return -EFAULT;" which is removed, because when we're in hv_synic_cleanup(), we're absolutely sure sctrl.enable must be 1.
--- drivers/hv/hv.c | 66 ++++++++++++++++++++++++++--------------------- drivers/hv/hyperv_vmbus.h | 2 ++ 2 files changed, 39 insertions(+), 29 deletions(-)
drivers/hv/hv.c +37 -29
@@ -154,7 +154,7 @@ void hv_synic_free(void)* retrieve the initialized message and event pages. Otherwise, we create and * initialize the message and event pages. */ -int hv_synic_init (unsigned int cpu) +void hv_synic_enable_regs (unsigned int cpu) {struct hv_per_cpu_context *hv_cpu =per_cpu_ptr (hv_context.cpu_context, cpu);
@@ -196,6 +196,11 @@ int hv_synic_init(unsigned int cpu) sctrl.enable =1 ;hv_set_synic_state (sctrl.as_uint64); +} + +int hv_synic_init (unsigned int cpu) +{ +hv_synic_enable_regs (cpu);hv_stimer_init (cpu);
@@ -205,20 +210,45 @@ int hv_synic_init(unsigned int cpu)/* * hv_synic_cleanup - Cleanup routine for hv_synic_init(). */ -int hv_synic_cleanup (unsigned int cpu) +void hv_synic_disable_regs (unsigned int cpu) {union hv_synic_sint shared_sint;union hv_synic_simp simp;union hv_synic_siefp siefp;union hv_synic_scontrol sctrl; + +hv_get_synint_state (VMBUS_MESSAGE_SINT , shared_sint.as_uint64); + + shared_sint.masked =1 ; + +/* Need to correctly cleanup in the case of SMP!!! */ +/* Disable the interrupt */ +hv_set_synint_state (VMBUS_MESSAGE_SINT , shared_sint.as_uint64); + +hv_get_simp (simp.as_uint64); + simp.simp_enabled =0 ; + simp.base_simp_gpa =0 ; + +hv_set_simp (simp.as_uint64); + +hv_get_siefp (siefp.as_uint64); + siefp.siefp_enabled =0 ; + siefp.base_siefp_gpa =0 ; + +hv_set_siefp (siefp.as_uint64); + +/* Disable the global synic bit */ +hv_get_synic_state (sctrl.as_uint64); + sctrl.enable =0 ; +hv_set_synic_state (sctrl.as_uint64); +} + +int hv_synic_cleanup (unsigned int cpu) +{struct vmbus_channel *channel, *sc;bool channel_found = false;unsigned long flags; -hv_get_synic_state (sctrl.as_uint64); -if (sctrl.enable !=1 ) -return -EFAULT ; -/* * Search for channels which are bound to the CPU we're about to * cleanup. In case we find one and vmbus is still connected we need to
@@ -249,29 +279,7 @@ int hv_synic_cleanup(unsigned int cpu)hv_stimer_cleanup (cpu); -hv_get_synint_state (VMBUS_MESSAGE_SINT , shared_sint.as_uint64); - - shared_sint.masked =1 ; - -/* Need to correctly cleanup in the case of SMP!!! */ -/* Disable the interrupt */ -hv_set_synint_state (VMBUS_MESSAGE_SINT , shared_sint.as_uint64); - -hv_get_simp (simp.as_uint64); - simp.simp_enabled =0 ; - simp.base_simp_gpa =0 ; - -hv_set_simp (simp.as_uint64); - -hv_get_siefp (siefp.as_uint64); - siefp.siefp_enabled =0 ; - siefp.base_siefp_gpa =0 ; - -hv_set_siefp (siefp.as_uint64); - -/* Disable the global synic bit */ - sctrl.enable =0 ; -hv_set_synic_state (sctrl.as_uint64); +hv_synic_disable_regs (cpu);return 0 ; }
drivers/hv/hyperv_vmbus.h +2 -0
@@ -169,8 +169,10 @@ extern int hv_post_message(union hv_connection_id connection_id,extern void hv_synic_free (void ); +extern void hv_synic_enable_regs (unsigned int cpu);extern int hv_synic_init (unsigned int cpu); +extern void hv_synic_disable_regs (unsigned int cpu);extern int hv_synic_cleanup (unsigned int cpu);/* Interface */
Signature · 1 line
1.8.3.1
Dexuan Cui
maintainerPATCH v4 5/12 Drivers: hv: vmbus: Suspend/resume the synic for hibernation
In reply to Dexuan CuiThis is needed when we resume the old kernel from the "current" kernel.
Note: when hv_synic_suspend() and hv_synic_resume() run, all the non-boot CPUs have been offlined, and interrupts are disabled on CPU0.
--- drivers/hv/vmbus_drv.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)
drivers/hv/vmbus_drv.c +46 -0
@@ -30,6 +30,7 @@#include <linux/kdebug.h> #include <linux/efi.h> #include <linux/random.h> +#include <linux/syscore_ops.h> #include <clocksource/hyperv_timer.h> #include "hyperv_vmbus.h"
@@ -2086,6 +2087,47 @@ static void hv_crash_handler(struct pt_regs *regs)hyperv_cleanup (); }; +static int hv_synic_suspend (void ) +{ +/* +* When we reach here, all the non-boot CPUs have been offlined, and +* the stimers on them have been unbound in hv_synic_cleanup() -> +* hv_stimer_cleanup() -> clockevents_unbind_device(). +* +* hv_synic_suspend() only runs on CPU0 with interrupts disabled. Here +* we do not unbind the stimer on CPU0 because: 1) it's unnecessary +* because the interrupts remain disabled between syscore_suspend() +* and syscore_resume(): see create_image() and resume_target_kernel(); +* 2) the stimer on CPU0 is automatically disabled later by +* syscore_suspend() -> timekeeping_suspend() -> tick_suspend() -> ... +* -> clockevents_shutdown() -> ... -> hv_ce_shutdown(); 3) a warning +* would be triggered if we call clockevents_unbind_device(), which +* may sleep, in an interrupts-disabled context. So, we intentionally +* don't call hv_stimer_cleanup(0) here. +*/ + +hv_synic_disable_regs (0 ); + +return 0 ; +} + +static void hv_synic_resume (void ) +{ +hv_synic_enable_regs (0 ); + +/* +* Note: we don't need to call hv_stimer_init(0), because the timer +* on CPU0 is not unbound in hv_synic_suspend(), and the timer is +* automatically re-enabled in timekeeping_resume(). +*/ +} + +/* The callbacks run only on CPU0, with irqs_disabled. */ +static struct syscore_ops hv_synic_syscore_ops = { + .suspend = hv_synic_suspend, + .resume = hv_synic_resume, +}; +static int __init hv_acpi_init (void ) {int ret, t;
@@ -2116,6 +2158,8 @@ static int __init hv_acpi_init(void)hv_setup_kexec_handler (hv_kexec_handler);hv_setup_crash_handler (hv_crash_handler); +register_syscore_ops (&hv_synic_syscore_ops); +return 0 ; cleanup:
@@ -2128,6 +2172,8 @@ static void __exit vmbus_exit(void) {int cpu; +unregister_syscore_ops (&hv_synic_syscore_ops); +hv_remove_kexec_handler ();hv_remove_crash_handler (); vmbus_connection.conn_state =DISCONNECTED ;
Signature · 1 line
1.8.3.1
Dexuan Cui
maintainerPATCH v4 6/12 Drivers: hv: vmbus: Add a helper function is_sub_channel()
In reply to Dexuan CuiThe existing method of telling if a channel is sub-channel in vmbus_process_offer() is cumbersome. This new simple helper function is preferred in future.
--- include/linux/hyperv.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
include/linux/hyperv.h +9 -1
@@ -245,7 +245,10 @@ struct vmbus_channel_offer { } pipe; } u;/* -* The sub_channel_index is defined in win8. +* The sub_channel_index is defined in Win8: a value of zero means a +* primary channel and a value of non-zero means a sub-channel. +* +* Before Win8, the field is reserved, meaning it's always zero. */ u16 sub_channel_index;u16 reserved3;
@@ -934,6 +937,11 @@ static inline bool is_hvsock_channel(const struct vmbus_channel *c) VMBUS_CHANNEL_TLNPI_PROVIDER_OFFER); } +static inline bool is_sub_channel (const struct vmbus_channel *c) +{ +return c->offermsg.offer.sub_channel_index !=0 ; +} +static inline void set_channel_affinity_state (struct vmbus_channel *c,enum hv_numa_policy policy) {
Signature · 1 line
1.8.3.1
Dexuan Cui
maintainerPATCH v4 7/12 Drivers: hv: vmbus: Implement suspend/resume for VSC drivers for hibernation
In reply to Dexuan CuiThe high-level VSC drivers will implement device-specific callbacks.
--- drivers/hv/vmbus_drv.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ include/linux/hyperv.h | 3 +++ 2 files changed, 49 insertions(+)
drivers/hv/vmbus_drv.c +46 -0
@@ -911,6 +911,43 @@ static void vmbus_shutdown(struct device *child_device) drv->shutdown (dev); } +/* +* vmbus_suspend - Suspend a vmbus device +*/ +static int vmbus_suspend (struct device *child_device) +{ +struct hv_driver *drv; +struct hv_device *dev =device_to_hv_device (child_device); + +/* The device may not be attached yet */ +if (!child_device->driver) +return 0 ; + + drv =drv_to_hv_drv (child_device->driver); +if (!drv->suspend) +return -EOPNOTSUPP ; + +return drv->suspend (dev); +} + +/* +* vmbus_resume - Resume a vmbus device +*/ +static int vmbus_resume (struct device *child_device) +{ +struct hv_driver *drv; +struct hv_device *dev =device_to_hv_device (child_device); + +/* The device may not be attached yet */ +if (!child_device->driver) +return 0 ; + + drv =drv_to_hv_drv (child_device->driver); +if (!drv->resume) +return -EOPNOTSUPP ; + +return drv->resume (dev); +}/* * vmbus_device_release - Final callback release of the vmbus child device
@@ -926,6 +963,14 @@ static void vmbus_device_release(struct device *device)kfree (hv_dev); } +/* +* Note: we must use SET_NOIRQ_SYSTEM_SLEEP_PM_OPS rather than +* SET_SYSTEM_SLEEP_PM_OPS: see the comment before vmbus_bus_pm. +*/ +static const struct dev_pm_ops vmbus_pm = { +SET_NOIRQ_SYSTEM_SLEEP_PM_OPS (vmbus_suspend, vmbus_resume) +}; +/* The one and only one */ static struct bus_type hv_bus = { .name ="vmbus" ,
@@ -936,6 +981,7 @@ static void vmbus_device_release(struct device *device) .uevent = vmbus_uevent, .dev_groups = vmbus_dev_groups, .drv_groups = vmbus_drv_groups, + .pm = &vmbus_pm, };struct onmessage_work_context {
include/linux/hyperv.h +3 -0
@@ -1157,6 +1157,9 @@ struct hv_driver {int (*remove)(struct hv_device *);void (*shutdown)(struct hv_device *); +int (*suspend)(struct hv_device *); +int (*resume)(struct hv_device *); + };/* Base device object */
Signature · 1 line
1.8.3.1
Dexuan Cui
maintainerPATCH v4 8/12 Drivers: hv: vmbus: Ignore the offers when resuming from hibernation
In reply to Dexuan CuiWhen the VM resumes, the host re-sends the offers. We should not add the offers to the global vmbus_connection.chn_list again.
This patch assumes the RELIDs of the channels don't change across hibernation. Actually this is not always true, especially in the case of NIC SR-IOV the VF vmbus device's RELID sometimes can change. A later patch will address this issue by mapping the new offers to the old channels and fixing up the old channels, if necessary.
--- drivers/hv/channel_mgmt.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-)
drivers/hv/channel_mgmt.c +57 -1
@@ -848,18 +848,74 @@ void vmbus_initiate_unload(bool crash) }/* +* find_primary_channel_by_offer - Get the channel object given the new offer. +* This is only used in the resume path of hibernation. +*/ +static struct vmbus_channel * +find_primary_channel_by_offer (const struct vmbus_channel_offer_channel *offer) +{ +struct vmbus_channel *channel =NULL , *iter; +const guid_t *inst1, *inst2; + +/* Ignore sub-channel offers. */ +if (offer->offer.sub_channel_index !=0 ) +return NULL ; + +mutex_lock (&vmbus_connection.channel_mutex); + +list_for_each_entry (iter, &vmbus_connection.chn_list, listentry) { + inst1 = &iter->offermsg.offer.if_instance; + inst2 = &offer->offer.if_instance; + +if (guid_equal (inst1, inst2)) { + channel = iter; +break ; + } + } + +mutex_unlock (&vmbus_connection.channel_mutex); + +return channel; +} + +/* * vmbus_onoffer - Handler for channel offers from vmbus in parent partition. * */ static void vmbus_onoffer (struct vmbus_channel_message_header *hdr) {struct vmbus_channel_offer_channel *offer; -struct vmbus_channel *newchannel; +struct vmbus_channel *oldchannel, *newchannel; +size_t offer_sz; offer = (struct vmbus_channel_offer_channel *)hdr;trace_vmbus_onoffer (offer); + oldchannel =find_primary_channel_by_offer (offer); + +if (oldchannel !=NULL ) { +atomic_dec (&vmbus_connection.offer_in_progress); + +/* +* We're resuming from hibernation: we expect the host to send +* exactly the same offers that we had before the hibernation. +*/ + offer_sz =sizeof (*offer); +if (memcmp (offer, &oldchannel->offermsg, offer_sz) ==0 ) +return ; + +pr_debug ("Mismatched offer from the host (relid=%d)\n" , + offer->child_relid); + +print_hex_dump_debug ("Old vmbus offer: " ,DUMP_PREFIX_OFFSET , +16 ,4 , &oldchannel->offermsg, offer_sz, + false); +print_hex_dump_debug ("New vmbus offer: " ,DUMP_PREFIX_OFFSET , +16 ,4 , offer, offer_sz, false); +return ; + } +/* Allocate the channel object and save this offer. */ newchannel =alloc_channel ();if (!newchannel) {
Signature · 1 line
1.8.3.1
RE: [PATCH v4 08/12] Drivers: hv: vmbus: Ignore the offers when resuming from hibernation
From: Dexuan Cui <[email protected]> Sent: Monday, September 2, 2019 5:23 PM
Quoted 10 lines
When the VM resumes, the host re-sends the offers. We should not add the offers to the global vmbus_connection.chn_list again.
This patch assumes the RELIDs of the channels don't change across hibernation. Actually this is not always true, especially in the case of NIC SR-IOV the VF vmbus device's RELID sometimes can change. A later patch will address this issue by mapping the new offers to the old channels and fixing up the old channels, if necessary.
Quoted 6 lines
Signed-off-by Dexuan Cui <[email protected]>--- drivers/hv/channel_mgmt.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-)
Dexuan Cui
maintainerPATCH v4 9/12 Drivers: hv: vmbus: Suspend/resume the vmbus itself for hibernation
In reply to Dexuan CuiBefore Linux enters hibernation, it sends the CHANNELMSG_UNLOAD message to the host so all the offers are gone. After hibernation, Linux needs to re-negotiate with the host using the same vmbus protocol version (which was in use before hibernation), and ask the host to re-offer the vmbus devices.
--- drivers/hv/connection.c | 3 +-- drivers/hv/hyperv_vmbus.h | 2 ++ drivers/hv/vmbus_drv.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 62 insertions(+), 2 deletions(-)
drivers/hv/connection.c +1 -2
@@ -59,8 +59,7 @@ static __u32 vmbus_get_next_version(__u32 current_version) } } -static int vmbus_negotiate_version (struct vmbus_channel_msginfo *msginfo, -__u32 version) +int vmbus_negotiate_version (struct vmbus_channel_msginfo *msginfo,u32 version) {int ret =0 ;unsigned int cur_cpu;
drivers/hv/hyperv_vmbus.h +2 -0
@@ -272,6 +272,8 @@ struct vmbus_msginfo {extern struct vmbus_connection vmbus_connection; +int vmbus_negotiate_version (struct vmbus_channel_msginfo *msginfo,u32 version); +static inline void vmbus_send_interrupt (u32 relid) {sync_set_bit (relid, vmbus_connection.send_int_page);
drivers/hv/vmbus_drv.c +59 -0
77 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Dexuan Cui
maintainerPATCH v4 10/12 Drivers: hv: vmbus: Clean up hv_sock channels by force upon suspend
In reply to Dexuan CuiFake RESCIND_CHANNEL messages to clean up hv_sock channels by force for hibernation. There is no better method to clean up the channels since some of the channels may still be referenced by the userspace apps when hibernation is triggered: in this case, with this patch, the "rescind" fields of the channels are set, and the apps will thoroughly destroy the channels after hibernation.
--- drivers/hv/vmbus_drv.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
drivers/hv/vmbus_drv.c +55 -0
@@ -24,6 +24,7 @@#include <linux/sched/task_stack.h> #include <asm/mshyperv.h> +#include <linux/delay.h> #include <linux/notifier.h> #include <linux/ptrace.h> #include <linux/screen_info.h>
@@ -1069,6 +1070,41 @@ void vmbus_on_msg_dpc(unsigned long data)vmbus_signal_eom (msg, message_type); } +/* +* Fake RESCIND_CHANNEL messages to clean up hv_sock channels by force for +* hibernation, because hv_sock connections can not persist across hibernation. +*/ +static void vmbus_force_channel_rescinded (struct vmbus_channel *channel) +{ +struct onmessage_work_context *ctx; +struct vmbus_channel_rescind_offer *rescind; + +WARN_ON (!is_hvsock_channel (channel)); + +/* +* sizeof(*ctx) is small and the allocation should really not fail, +* otherwise the state of the hv_sock connections ends up in limbo. +*/ + ctx =kzalloc (sizeof (*ctx),GFP_KERNEL | __GFP_NOFAIL); + +/* +* So far, these are not really used by Linux. Just set them to the +* reasonable values conforming to the definitions of the fields. +*/ + ctx->msg.header.message_type =1 ; + ctx->msg.header.payload_size =sizeof (*rescind); + +/* These values are actually used by Linux. */ + rescind = (struct vmbus_channel_rescind_offer *)ctx->msg.u.payload; + rescind->header.msgtype =CHANNELMSG_RESCIND_CHANNELOFFER ; + rescind->child_relid = channel->offermsg.child_relid; + +INIT_WORK (&ctx->work, vmbus_onmessage_work); + +queue_work_on (vmbus_connection.connect_cpu, + vmbus_connection.work_queue, + &ctx->work); +}/* * Direct callback for channels using other deferred processing
@@ -2091,6 +2127,25 @@ static int vmbus_acpi_add(struct acpi_device *device)static int vmbus_bus_suspend (struct device *dev) { +struct vmbus_channel *channel; + +while (atomic_read (&vmbus_connection.offer_in_progress) !=0 ) { +/* +* We wait here until the completion of any channel +* offers that are currently in progress. +*/ +msleep (1 ); + } + +mutex_lock (&vmbus_connection.channel_mutex); +list_for_each_entry (channel, &vmbus_connection.chn_list, listentry) { +if (!is_hvsock_channel (channel)) +continue ; + +vmbus_force_channel_rescinded (channel); + } +mutex_unlock (&vmbus_connection.channel_mutex); +vmbus_initiate_unload (false); vmbus_connection.conn_state =DISCONNECTED ;
Signature · 1 line
1.8.3.1
Dexuan Cui
maintainerPATCH v4 11/12 Drivers: hv: vmbus: Suspend after cleaning up hv_sock and sub channels
In reply to Dexuan CuiBefore suspend, Linux must make sure all the hv_sock channels have been properly cleaned up, because a hv_sock connection can not persist across hibernation, and the user-space app must be properly notified of the state change of the connection.
Before suspend, Linux also must make sure all the sub-channels have been destroyed, i.e. the related channel structs of the sub-channels must be properly removed, otherwise they would cause a conflict when the sub-channels are recreated upon resume.
Add a counter to track such channels, and vmbus_bus_suspend() should wait for the counter to drop to zero.
--- drivers/hv/channel_mgmt.c | 26 ++++++++++++++++++++++++++ drivers/hv/connection.c | 3 +++ drivers/hv/hyperv_vmbus.h | 12 ++++++++++++ drivers/hv/vmbus_drv.c | 44 +++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 84 insertions(+), 1 deletion(-)
drivers/hv/channel_mgmt.c +26 -0
@@ -545,6 +545,10 @@ static void vmbus_process_offer(struct vmbus_channel *newchannel)mutex_lock (&vmbus_connection.channel_mutex); +/* Remember the channels that should be cleaned up upon suspend. */ +if (is_hvsock_channel (newchannel) ||is_sub_channel (newchannel)) +atomic_inc (&vmbus_connection.nr_chan_close_on_suspend); +/* * Now that we have acquired the channel_mutex, * we can release the potentially racing rescind thread.
@@ -944,6 +948,16 @@ static void vmbus_onoffer(struct vmbus_channel_message_header *hdr)vmbus_process_offer (newchannel); } +static void check_ready_for_suspend_event (void ) +{ +/* +* If all the sub-channels or hv_sock channels have been cleaned up, +* then it's safe to suspend. +*/ +if (atomic_dec_and_test (&vmbus_connection.nr_chan_close_on_suspend)) +complete (&vmbus_connection.ready_for_suspend_event); +} +/* * vmbus_onoffer_rescind - Rescind offer handler. *
@@ -954,6 +968,7 @@ static void vmbus_onoffer_rescind(struct vmbus_channel_message_header *hdr)struct vmbus_channel_rescind_offer *rescind;struct vmbus_channel *channel;struct device *dev; +bool clean_up_chan_for_suspend; rescind = (struct vmbus_channel_rescind_offer *)hdr;
@@ -993,6 +1008,8 @@ static void vmbus_onoffer_rescind(struct vmbus_channel_message_header *hdr)return ; } + clean_up_chan_for_suspend =is_hvsock_channel (channel) || +is_sub_channel (channel);/* * Before setting channel->rescind in vmbus_rescind_cleanup(), we * should make sure the channel callback is not running any more.
@@ -1018,6 +1035,10 @@ static void vmbus_onoffer_rescind(struct vmbus_channel_message_header *hdr)if (channel->device_obj) {if (channel->chn_rescind_callback) { channel->chn_rescind_callback (channel); + +if (clean_up_chan_for_suspend) +check_ready_for_suspend_event (); +return ; }/*
@@ -1050,6 +1071,11 @@ static void vmbus_onoffer_rescind(struct vmbus_channel_message_header *hdr) }mutex_unlock (&vmbus_connection.channel_mutex); } + +/* The "channel" may have been freed. Do not access it any longer. */ + +if (clean_up_chan_for_suspend) +check_ready_for_suspend_event (); }void vmbus_hvsock_device_unregister (struct vmbus_channel *channel)
drivers/hv/connection.c +3 -0
@@ -26,6 +26,9 @@struct vmbus_connection vmbus_connection = { .conn_state =DISCONNECTED , .next_gpadl_handle =ATOMIC_INIT (0xE1E10 ), + + .ready_for_suspend_event=COMPLETION_INITIALIZER ( + vmbus_connection.ready_for_suspend_event), };EXPORT_SYMBOL_GPL (vmbus_connection);
drivers/hv/hyperv_vmbus.h +12 -0
@@ -258,6 +258,18 @@ struct vmbus_connection {struct workqueue_struct *work_queue;struct workqueue_struct *handle_primary_chan_wq;struct workqueue_struct *handle_sub_chan_wq; + +/* +* The number of sub-channels and hv_sock channels that should be +* cleaned up upon suspend: sub-channels will be re-created upon +* resume, and hv_sock channels should not survive suspend. +*/ +atomic_t nr_chan_close_on_suspend; +/* +* vmbus_bus_suspend() waits for "nr_chan_close_on_suspend" to +* drop to zero. +*/ +struct completion ready_for_suspend_event; };
drivers/hv/vmbus_drv.c +43 -1
62 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v4 11/12] Drivers: hv: vmbus: Suspend after cleaning up hv_sock and sub channels
From: Dexuan Cui <[email protected]> Sent: Monday, September 2, 2019 5:23 PM
Quoted 14 lines
Before suspend, Linux must make sure all the
hv_sockchannels have been properly cleaned up, because ahv_sockconnection can not persist across hibernation, and the user-space app must be properly notified of the state change of the connection.Before suspend, Linux also must make sure all the sub-channels have been destroyed, i.e. the related channel structs of the sub-channels must be properly removed, otherwise they would cause a conflict when the sub-channels are recreated upon resume.
Add a counter to track such channels, and
vmbus_bus_suspend()should wait for the counter to drop to zero.
Quoted 8 lines
Signed-off-by Dexuan Cui <[email protected]>--- drivers/hv/channel_mgmt.c | 26 ++++++++++++++++++++++++++ drivers/hv/connection.c | 3 +++ drivers/hv/hyperv_vmbus.h | 12 ++++++++++++ drivers/hv/vmbus_drv.c | 44 +++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 84 insertions(+), 1 deletion(-)
Dexuan Cui
maintainerPATCH v4 12/12 Drivers: hv: vmbus: Resume after fixing up old primary channels
In reply to Dexuan CuiWhen the host re-offers the primary channels upon resume, the host only guarantees the Instance GUID doesn't change, so vmbus_bus_suspend() should invalidate channel->offermsg.child_relid and figure out the number of primary channels that need to be fixed up upon resume.
Upon resume, vmbus_onoffer() finds the old channel structs, and maps the new offers to the old channels, and fixes up the old structs, and finally the resume callbacks of the VSC drivers will re-open the channels.
--- drivers/hv/channel_mgmt.c | 85 ++++++++++++++++++++++++++++++++++++----------- drivers/hv/connection.c | 2 ++ drivers/hv/hyperv_vmbus.h | 14 ++++++++ drivers/hv/vmbus_drv.c | 17 ++++++++++ include/linux/hyperv.h | 3 ++ 5 files changed, 101 insertions(+), 20 deletions(-)
drivers/hv/channel_mgmt.c +65 -20
@@ -407,7 +407,15 @@ void hv_process_channel_removal(struct vmbus_channel *channel)cpumask_clear_cpu (channel->target_cpu, &primary_channel->alloced_cpus_in_node); -vmbus_release_relid (channel->offermsg.child_relid); +/* +* Upon suspend, an in-use hv_sock channel is marked as "rescinded" and +* the relid is invalidated; after hibernation, when the user-space app +* destroys the channel, the relid is INVALID_RELID, and in this case +* it's unnecessary and unsafe to release the old relid, since the same +* relid can refer to a completely different channel now. +*/ +if (channel->offermsg.child_relid !=INVALID_RELID ) +vmbus_release_relid (channel->offermsg.child_relid);free_channel (channel); }
@@ -851,6 +859,36 @@ void vmbus_initiate_unload(bool crash)vmbus_wait_for_unload (); } +static void check_ready_for_resume_event (void ) +{ +/* +* If all the old primary channels have been fixed up, then it's safe +* to resume. +*/ +if (atomic_dec_and_test (&vmbus_connection.nr_chan_fixup_on_resume)) +complete (&vmbus_connection.ready_for_resume_event); +} + +static void vmbus_setup_channel_state (struct vmbus_channel *channel, +struct vmbus_channel_offer_channel *offer) +{ +/* +* Setup state for signalling the host. +*/ + channel->sig_event =VMBUS_EVENT_CONNECTION_ID ; + +if (vmbus_proto_version !=VERSION_WS2008 ) { + channel->is_dedicated_interrupt = + (offer->is_dedicated_interrupt !=0 ); + channel->sig_event = offer->connection_id; + } + +memcpy (&channel->offermsg, offer, +sizeof (struct vmbus_channel_offer_channel )); + channel->monitor_grp = (u8 )offer->monitorid /32 ; + channel->monitor_bit = (u8 )offer->monitorid %32 ; +} +/* * find_primary_channel_by_offer - Get the channel object given the new offer. * This is only used in the resume path of hibernation.
@@ -902,14 +940,29 @@ static void vmbus_onoffer(struct vmbus_channel_message_header *hdr)atomic_dec (&vmbus_connection.offer_in_progress);/* -* We're resuming from hibernation: we expect the host to send -* exactly the same offers that we had before the hibernation. +* We're resuming from hibernation: all the sub-channel and +* hv_sock channels we had before the hibernation should have +* been cleaned up, and now we must be seeing a re-offered +* primary channel that we had before the hibernation. */ + +WARN_ON (oldchannel->offermsg.child_relid !=INVALID_RELID ); +/* Fix up the relid. */ + oldchannel->offermsg.child_relid = offer->child_relid; + offer_sz =sizeof (*offer); -if (memcmp (offer, &oldchannel->offermsg, offer_sz) ==0 ) +if (memcmp (offer, &oldchannel->offermsg, offer_sz) ==0 ) { +check_ready_for_resume_event ();return ; + } -pr_debug ("Mismatched offer from the host (relid=%d)\n" , +/* +* This is not an error, since the host can also change the +* other field(s) of the offer, e.g. on WS RS5 (Build 17763), +* the offer->connection_id of the Mellanox VF vmbus device +* can change when the host reoffers the device upon resume. +*/ +pr_debug ("vmbus offer changed: relid=%d\n" , offer->child_relid); print_hex_dump_debug("Old vmbus offer: ", DUMP_PREFIX_OFFSET,
@@ -917,6 +970,12 @@ static void vmbus_onoffer(struct vmbus_channel_message_header *hdr) false);print_hex_dump_debug ("New vmbus offer: " ,DUMP_PREFIX_OFFSET ,16 ,4 , offer, offer_sz, false); + +/* Fix up the old channel. */ +vmbus_setup_channel_state (oldchannel, offer); + +check_ready_for_resume_event (); +return ; }
@@ -929,21 +988,7 @@ static void vmbus_onoffer(struct vmbus_channel_message_header *hdr)return ; } -/* -* Setup state for signalling the host. -*/ - newchannel->sig_event =VMBUS_EVENT_CONNECTION_ID ; - -if (vmbus_proto_version !=VERSION_WS2008 ) { - newchannel->is_dedicated_interrupt = - (offer->is_dedicated_interrupt !=0 ); - newchannel->sig_event = offer->connection_id; - } - -memcpy (&newchannel->offermsg, offer, -sizeof (struct vmbus_channel_offer_channel )); - newchannel->monitor_grp = (u8 )offer->monitorid /32 ; - newchannel->monitor_bit = (u8 )offer->monitorid %32 ; +vmbus_setup_channel_state (newchannel, offer);vmbus_process_offer (newchannel); }
drivers/hv/connection.c +2 -0
@@ -29,6 +29,8 @@ struct vmbus_connection vmbus_connection = { .ready_for_suspend_event= COMPLETION_INITIALIZER( vmbus_connection.ready_for_suspend_event), + .ready_for_resume_event = COMPLETION_INITIALIZER( + vmbus_connection.ready_for_resume_event), };EXPORT_SYMBOL_GPL (vmbus_connection);
drivers/hv/hyperv_vmbus.h +14 -0
@@ -270,6 +270,20 @@ struct vmbus_connection {* drop to zero. */ struct completion ready_for_suspend_event; + +/* +* The number of primary channels that should be "fixed up" +* upon resume: these channels are re-offered upon resume, and some +* fields of the channel offers (i.e. child_relid and connection_id) +* can change, so the old offermsg must be fixed up, before the resume +* callbacks of the VSC drivers start to further touch the channels. +*/ +atomic_t nr_chan_fixup_on_resume; +/* +* vmbus_bus_resume() waits for "nr_chan_fixup_on_resume" to +* drop to zero. +*/ +struct completion ready_for_resume_event; };
drivers/hv/vmbus_drv.c +17 -0
@@ -2164,9 +2164,17 @@ static int vmbus_bus_suspend(struct device *dev)if (atomic_read (&vmbus_connection.nr_chan_close_on_suspend) >0 )wait_for_completion (&vmbus_connection.ready_for_suspend_event); +WARN_ON (atomic_read (&vmbus_connection.nr_chan_fixup_on_resume) !=0 ); +mutex_lock (&vmbus_connection.channel_mutex);list_for_each_entry (channel, &vmbus_connection.chn_list, listentry) { +/* +* Invalidate the field. Upon resume, vmbus_onoffer() will fix +* up the field, and the other fields (if necessary). +*/ + channel->offermsg.child_relid =INVALID_RELID ; +if (is_hvsock_channel (channel)) {if (!channel->rescind) {pr_err ("hv_sock channel not rescinded!\n" );
@@ -2181,6 +2189,8 @@ static int vmbus_bus_suspend(struct device *dev)WARN_ON_ONCE (1 ); }spin_unlock_irqrestore (&channel->lock, flags); + +atomic_inc (&vmbus_connection.nr_chan_fixup_on_resume); }mutex_unlock (&vmbus_connection.channel_mutex);
@@ -2189,6 +2199,9 @@ static int vmbus_bus_suspend(struct device *dev) vmbus_connection.conn_state =DISCONNECTED ; +/* Reset the event for the next resume. */ +reinit_completion (&vmbus_connection.ready_for_resume_event); +return 0 ; }
@@ -2223,8 +2236,12 @@ static int vmbus_bus_resume(struct device *dev)if (ret !=0 )return ret; +WARN_ON (atomic_read (&vmbus_connection.nr_chan_fixup_on_resume) ==0 ); +vmbus_request_offers (); +wait_for_completion (&vmbus_connection.ready_for_resume_event); +/* Reset the event for the next suspend. */ reinit_completion (&vmbus_connection.ready_for_suspend_event);
include/linux/hyperv.h +3 -0
@@ -426,6 +426,9 @@ enum vmbus_channel_message_type {CHANNELMSG_COUNT }; +/* Hyper-V supports about 2048 channels, and the RELIDs start with 1. */ +#define INVALID_RELID U32_MAX +struct vmbus_channel_message_header {enum vmbus_channel_message_type msgtype;u32 padding;
Signature · 1 line
1.8.3.1
RE: [PATCH v4 12/12] Drivers: hv: vmbus: Resume after fixing up old primary channels
From: Dexuan Cui <[email protected]> Sent: Monday, September 2, 2019 5:23 PM
Quoted 11 lines
When the host re-offers the primary channels upon resume, the host only guarantees the Instance GUID doesn't change, so
vmbus_bus_suspend()should invalidatechannel->offermsg.child_relidand figure out the number of primary channels that need to be fixed up upon resume.Upon resume,
vmbus_onoffer()finds the old channel structs, and maps the new offers to the old channels, and fixes up the old structs, and finally the resume callbacks of the VSC drivers will re-open the channels.
Quoted 9 lines
Signed-off-by Dexuan Cui <[email protected]>--- drivers/hv/channel_mgmt.c | 85 ++++++++++++++++++++++++++++++++++++----------- drivers/hv/connection.c | 2 ++ drivers/hv/hyperv_vmbus.h | 14 ++++++++ drivers/hv/vmbus_drv.c | 17 ++++++++++ include/linux/hyperv.h | 3 ++ 5 files changed, 101 insertions(+), 20 deletions(-)