x86/Hyper-V: Panic code path fixes
Subsystems (4)
v4 cover x86/Hyper-V: Panic code path fixes
From: Tianyu Lan <[email protected]>
This patchset fixes some issues in the Hyper-V panic code path. Patch 1 resolves issue that panic system still responses network packets.
Patch 2-3,5-6 resolves crash enlightenment issues. Patch 4 is to set crash_kexec_post_notifiers to true for Hyper-V VM in order to report crash data or kmsg to host before running kdump kernel.
Tianyu Lan (6):
x86/Hyper-V: Unload vmbus channel in hv panic callback x86/Hyper-V: Free hv_panic_page when fail to register kmsg dump x86/Hyper-V: Trigger crash enlightenment only once during
0;136;0c0;136;0c system crash.
x86/Hyper-V: Report crash register data or kmsg before running crash
kernel
x86/Hyper-V: Report crash register data when sysctl_record_panic_msg
is not set
x86/Hyper-V: Report crash data in die() when panic_on_oops is set
arch/x86/hyperv/hv_init.c | 6 +++- arch/x86/kernel/cpu/mshyperv.c | 10 +++++++ drivers/hv/channel_mgmt.c | 3 ++ drivers/hv/vmbus_drv.c | 62 ++++++++++++++++++++++++++++++------------ include/asm-generic/mshyperv.h | 2 +- 5 files changed, 63 insertions(+), 20 deletions(-)
Signature · 1 line
2.14.5
PATCH v4 1/6 x86/Hyper-V: Unload vmbus channel in hv panic callback
From: Tianyu Lan <[email protected]>
When kdump is not configured, a Hyper-V VM might still respond to network traffic after a kernel panic when kernel parameter panic=0. The panic CPU goes into an infinite loop with interrupts enabled, and the VMbus driver interrupt handler still works because the VMbus connection is unloaded only in the kdump path. The network responses make the other end of the connection think the VM is still functional even though it has panic'ed, which could affect any failover actions that should be taken.
Fix this by unloading the VMbus connection during the panic process. vmbus_initiate_unload() could then be called twice (e.g., by hyperv_panic_event() and hv_crash_handler(), so reset the connection state in vmbus_initiate_unload() to ensure the unload is done only once.
---
Change since v1:
- Update change log
- Use xchg() to change vmbus connection status
Change since v2:
- Update comment of registering panic callback.
Change since v3:
- Add fix commit in the change log.
--- drivers/hv/channel_mgmt.c | 3 +++ drivers/hv/vmbus_drv.c | 21 +++++++++++++-------- 2 files changed, 16 insertions(+), 8 deletions(-)
drivers/hv/channel_mgmt.c +3 -0
@@ -839,6 +839,9 @@ void vmbus_initiate_unload(bool crash) {struct vmbus_channel_message_header hdr; +if (xchg (&vmbus_connection.conn_state,DISCONNECTED ) ==DISCONNECTED ) +return ; +/* Pre-Win2012R2 hosts don't support reconnect */ if (vmbus_proto_version <VERSION_WIN8_1 )return ;
drivers/hv/vmbus_drv.c +13 -8
@@ -53,9 +53,12 @@ static int hyperv_panic_event(struct notifier_block *nb, unsigned long val, {struct pt_regs *regs; - regs =current_pt_regs (); +vmbus_initiate_unload (true); -hyperv_report_panic (regs, val); +if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE ) { + regs =current_pt_regs (); +hyperv_report_panic (regs, val); + }return NOTIFY_DONE ; }
@@ -1391,10 +1394,16 @@ static int vmbus_bus_init(void) }register_die_notifier (&hyperv_die_block); -atomic_notifier_chain_register (&panic_notifier_list, - &hyperv_panic_block); } +/* +* Always register the panic notifier because we need to unload +* the VMbus channel connection to prevent any VMbus +* activity after the VM panics. +*/ +atomic_notifier_chain_register (&panic_notifier_list, + &hyperv_panic_block); +vmbus_request_offers ();return 0 ;
@@ -2204,8 +2213,6 @@ static int vmbus_bus_suspend(struct device *dev)vmbus_initiate_unload (false); - vmbus_connection.conn_state =DISCONNECTED ; -/* Reset the event for the next resume. */ reinit_completion (&vmbus_connection.ready_for_resume_event);
@@ -2289,7 +2296,6 @@ static void hv_kexec_handler(void) {hv_stimer_global_cleanup ();vmbus_initiate_unload (false); - vmbus_connection.conn_state =DISCONNECTED ;/* Make sure conn_state is set as hv_synic_cleanup checks for it */ mb ();cpuhp_remove_state (hyperv_cpuhp_online);
@@ -2306,7 +2312,6 @@ static void hv_crash_handler(struct pt_regs *regs)* doing the cleanup for current CPU only. This should be sufficient * for kdump. */ - vmbus_connection.conn_state =DISCONNECTED ; cpu =smp_processor_id ();hv_stimer_cleanup (cpu);hv_synic_disable_regs (cpu);
Signature · 1 line
2.14.5
PATCH v4 2/6 x86/Hyper-V: Free hv_panic_page when fail to register kmsg dump
In reply to [email protected]From: Tianyu Lan <[email protected]>
If kmsg_dump_register() fails, hv_panic_page will not be used anywhere. So free and reset it.
---
Change since v1:
- Update commit log
- Remove hv_free_hyperv_page() in the error path
Change since v3:
- Add fix commit in the change log.
--- drivers/hv/vmbus_drv.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/hv/vmbus_drv.c +5 -2
@@ -1385,9 +1385,13 @@ static int vmbus_bus_init(void) hv_panic_page = (void *)hv_alloc_hyperv_zeroed_page ();if (hv_panic_page) { ret =kmsg_dump_register (&hv_kmsg_dumper); -if (ret) +if (ret) {pr_err ("Hyper-V: kmsg dump register " "error 0x%x\n" , ret); +hv_free_hyperv_page ( + (unsigned long )hv_panic_page); + hv_panic_page =NULL ; + } }else pr_err ("Hyper-V: panic message page memory " "allocation failed" );
@@ -1416,7 +1420,6 @@ static int vmbus_bus_init(void)hv_remove_vmbus_irq ();bus_unregister (&hv_bus); -hv_free_hyperv_page ((unsigned long )hv_panic_page);unregister_sysctl_table (hv_ctl_table_hdr); hv_ctl_table_hdr =NULL ;return ret;
Signature · 1 line
2.14.5
PATCH v4 3/6 x86/Hyper-V: Trigger crash enlightenment only once during 0;136;0c0;136;0c system crash.
In reply to [email protected]From: Tianyu Lan <[email protected]>
When a guest VM panics, Hyper-V should be notified only once via the crash synthetic MSRs. Current Linux code might write these crash MSRs twice during a system panic:
1) hyperv_panic/die_event() calling hyperv_report_panic()
2) hv_kmsg_dump() calling hyperv_report_panic_msg()
Fix this by not calling hyperv_report_panic() if a kmsg dump has been successfully registered. The notification will happen later via hyperv_report_panic_msg().
---
Change since v1:
- Update commit log
Change since v2:
- Update comment
Change since v3:
- Add fix commit in the change log.
--- drivers/hv/vmbus_drv.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
drivers/hv/vmbus_drv.c +14 -2
@@ -55,7 +55,13 @@ static int hyperv_panic_event(struct notifier_block *nb, unsigned long val,vmbus_initiate_unload (true); -if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE ) { +/* +* Hyper-V should be notified only once about a panic. If we will be +* doing hyperv_report_panic_msg() later with kmsg data, don't do +* the notification here. +*/ +if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE + && !hv_panic_page) { regs =current_pt_regs ();hyperv_report_panic (regs, val); }
@@ -68,7 +74,13 @@ static int hyperv_die_event(struct notifier_block *nb, unsigned long val,struct die_args *die = (struct die_args *)args;struct pt_regs *regs = die->regs; -hyperv_report_panic (regs, val); +/* +* Hyper-V should be notified only once about a panic. If we will be +* doing hyperv_report_panic_msg() later with kmsg data, don't do +* the notification here. +*/ +if (!hv_panic_page) +hyperv_report_panic (regs, val);return NOTIFY_DONE ; }
Signature · 1 line
2.14.5
PATCH v4 4/6 x86/Hyper-V: Report crash register data or kmsg before running crash kernel
In reply to [email protected]From: Tianyu Lan <[email protected]>
We want to notify Hyper-V when a Linux guest VM crash occurs, so
there is a record of the crash even when kdump is enabled. Butcrash_kexec_post_notifiers defaults to "false", so the kdump kernel runs before the notifiers and Hyper-V never gets notified. Fix this by always setting crash_kexec_post_notifiers to be true for Hyper-V VMs.
---
Change since v1:
Update commit log
Change since v3:
- Add fix commit in the change log.
--- arch/x86/kernel/cpu/mshyperv.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/kernel/cpu/mshyperv.c +10 -0
@@ -263,6 +263,16 @@ static void __init ms_hyperv_init_platform(void)cpuid_eax (HYPERV_CPUID_NESTED_FEATURES ); } +/* +* Hyper-V expects to get crash register data or kmsg when +* crash enlightment is available and system crashes. Set +* crash_kexec_post_notifiers to be true to make sure that +* calling crash enlightment interface before running kdump +* kernel. +*/ +if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE ) + crash_kexec_post_notifiers = true; +#ifdef CONFIG_X86_LOCAL_APIC if (ms_hyperv.features &HV_X64_ACCESS_FREQUENCY_MSRS && ms_hyperv.misc_features &HV_FEATURE_FREQUENCY_MSRS_AVAILABLE ) {
Signature · 1 line
2.14.5
PATCH v4 5/6 x86/Hyper-V: Report crash register data when sysctl_record_panic_msg is not set
In reply to [email protected]From: Tianyu Lan <[email protected]>
When sysctl_record_panic_msg is not set, the panic will not be reported to Hyper-V via hyperv_report_panic_msg(). So the crash should be reported via hyperv_report_panic().
---
Change since v3:
- Add fix commit in the change log
--- drivers/hv/vmbus_drv.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
drivers/hv/vmbus_drv.c +14 -9
@@ -48,6 +48,18 @@ static int hyperv_cpuhp_online;static void *hv_panic_page; +/* +* Boolean to control whether to report panic messages over Hyper-V. +* +* It can be set via /proc/sys/kernel/hyperv/record_panic_msg +*/ +static int sysctl_record_panic_msg =1 ; + +static int hyperv_report_reg (void ) +{ +return !sysctl_record_panic_msg || !hv_panic_page; +} +static int hyperv_panic_event (struct notifier_block *nb,unsigned long val,void *args) {
@@ -61,7 +73,7 @@ static int hyperv_panic_event(struct notifier_block *nb, unsigned long val,* the notification here. */ if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE - && !hv_panic_page) { + &&hyperv_report_reg ()) { regs =current_pt_regs ();hyperv_report_panic (regs, val); }
@@ -79,7 +91,7 @@ static int hyperv_die_event(struct notifier_block *nb, unsigned long val,* doing hyperv_report_panic_msg() later with kmsg data, don't do * the notification here. */ -if (!hv_panic_page) +if (hyperv_report_reg ())hyperv_report_panic (regs, val);return NOTIFY_DONE ; }
@@ -1267,13 +1279,6 @@ static void vmbus_isr(void)add_interrupt_randomness (HYPERVISOR_CALLBACK_VECTOR ,0 ); } -/* -* Boolean to control whether to report panic messages over Hyper-V. -* -* It can be set via /proc/sys/kernel/hyperv/record_panic_msg -*/ -static int sysctl_record_panic_msg =1 ; -/* * Callback from kmsg_dump. Grab as much as possible from the end of the kmsg * buffer and call into Hyper-V to transfer the data.
Signature · 1 line
2.14.5
PATCH v4 6/6 x86/Hyper-V: Report crash data in die() when panic_on_oops is set
In reply to [email protected]From: Tianyu Lan <[email protected]>
When oops happens with panic_on_oops unset, the oops thread is killed by die() and system continues to run. In such case, guest should not report crash register data to host since system still runs. Check panic_on_oops and return directly in hyperv_report_panic() when the function is called in the die() and panic_on_oops is unset. Fix it.
---
Change since v3:
- Fix compile error.
- Add fix commit in the change log
--- arch/x86/hyperv/hv_init.c | 6 +++++- drivers/hv/vmbus_drv.c | 5 +++-- include/asm-generic/mshyperv.h | 2 +- 3 files changed, 9 insertions(+), 4 deletions(-)
arch/x86/hyperv/hv_init.c +5 -1
@@ -20,6 +20,7 @@#include <linux/mm.h> #include <linux/hyperv.h> #include <linux/slab.h> +#include <linux/kernel.h> #include <linux/cpuhotplug.h> #include <linux/syscore_ops.h> #include <clocksource/hyperv_timer.h>
@@ -419,11 +420,14 @@ void hyperv_cleanup(void) }EXPORT_SYMBOL_GPL (hyperv_cleanup); -void hyperv_report_panic (struct pt_regs *regs,long err) +void hyperv_report_panic (struct pt_regs *regs,long err,bool in_die) {static bool panic_reported;u64 guest_id; +if (in_die && !panic_on_oops) +return ; +/* * We prefer to report panic on 'die' chain as we have proper * registers to report, but if we miss it (e.g. on BUG()) we need
drivers/hv/vmbus_drv.c +3 -2
@@ -31,6 +31,7 @@#include <linux/kdebug.h> #include <linux/efi.h> #include <linux/random.h> +#include <linux/kernel.h> #include <linux/syscore_ops.h> #include <clocksource/hyperv_timer.h> #include "hyperv_vmbus.h"
@@ -75,7 +76,7 @@ static int hyperv_panic_event(struct notifier_block *nb, unsigned long val,if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE &&hyperv_report_reg ()) { regs =current_pt_regs (); -hyperv_report_panic (regs, val); +hyperv_report_panic (regs, val, false); }return NOTIFY_DONE ; }
@@ -92,7 +93,7 @@ static int hyperv_die_event(struct notifier_block *nb, unsigned long val,* the notification here. */ if (hyperv_report_reg ()) -hyperv_report_panic (regs, val); +hyperv_report_panic (regs, val, true);return NOTIFY_DONE ; }
include/asm-generic/mshyperv.h +1 -1
@@ -163,7 +163,7 @@ static inline int cpumask_to_vpset(struct hv_vpset *vpset,return nr_bank; } -void hyperv_report_panic (struct pt_regs *regs,long err); +void hyperv_report_panic (struct pt_regs *regs,long err,bool in_die);void hyperv_report_panic_msg (phys_addr_t pa,size_t size);bool hv_is_hyperv_initialized (void );bool hv_is_hibernation_supported (void );
Signature · 1 line
2.14.5
RE: [PATCH V4 6/6] x86/Hyper-V: Report crash data in die() when panic_on_oops is set
From: Tianyu Lan <[email protected]> Sent: Monday, April 6, 2020 8:54 AM
Quoted 14 lines
When oops happens with
panic_on_oopsunset, the oops thread is killed bydie()and system continues to run. In such case, guest should not report crash register data to host since system still runs. Checkpanic_on_oopsand return directly inhyperv_report_panic()when the function is called in thedie()andpanic_on_oopsis unset. Fix it.Fixes 7ed4325a44ea ("Drivers: hv: vmbus: Make panic reporting to be more useful")Signed-off-by Tianyu Lan <[email protected]>---
Change since v3:
- Fix compile error.
- Add fix commit in the change log
Quoted 5 lines
--- arch/x86/hyperv/hv_init.c | 6 +++++- drivers/hv/vmbus_drv.c | 5 +++-- include/asm-generic/mshyperv.h | 2 +- 3 files changed, 9 insertions(+), 4 deletions(-)
Wei Liu
maintainerRe: [PATCH V4 6/6] x86/Hyper-V: Report crash data in die() when panic_on_oops is set
Quoted 21 lines · Michael Kelley
From: Tianyu Lan <[email protected]> Sent: Monday, April 6, 2020 8:54 AM
Quoted 14 lines
When oops happens with
panic_on_oopsunset, the oops thread is killed bydie()and system continues to run. In such case, guest should not report crash register data to host since system still runs. Checkpanic_on_oopsand return directly inhyperv_report_panic()when the function is called in thedie()andpanic_on_oopsis unset. Fix it.Fixes 7ed4325a44ea ("Drivers: hv: vmbus: Make panic reporting to be more useful")Signed-off-by Tianyu Lan <[email protected]>---
Change since v3:
- Fix compile error.
- Add fix commit in the change logQuoted 5 lines
--- arch/x86/hyperv/hv_init.c | 6 +++++- drivers/hv/vmbus_drv.c | 5 +++-- include/asm-generic/mshyperv.h | 2 +- 3 files changed, 9 insertions(+), 4 deletions(-)
Quoted 1 line
Reviewed-by Michael Kelley <[email protected]>
It seems to me only the last patch is new, others are already in my tree, so I only apply the last one.
Let me know if my understanding is wrong.
Wei.
RE: [PATCH V4 6/6] x86/Hyper-V: Report crash data in die() when panic_on_oops is set
From: Wei Liu <[email protected]> Sent: Thursday, April 9, 2020 9:41 AM
Quoted 23 lines · Michael Kelley
Quoted 21 lines
From: Tianyu Lan <[email protected]> Sent: Monday, April 6, 2020 8:54 AM
Quoted 19 lines
When oops happens with
panic_on_oopsunset, the oops thread is killed bydie()and system continues to run. In such case, guest should not report crash register data to host since system still runs. Checkpanic_on_oopsand return directly inhyperv_report_panic()when the function is called in thedie()andpanic_on_oopsis unset. Fix it.Fixes 7ed4325a44ea ("Drivers: hv: vmbus: Make panic reporting to be more useful")Signed-off-by Tianyu Lan <[email protected]>---
Change since v3:
- Fix compile error.
- Add fix commit in the change log--- arch/x86/hyperv/hv_init.c | 6 +++++- drivers/hv/vmbus_drv.c | 5 +++-- include/asm-generic/mshyperv.h | 2 +- 3 files changed, 9 insertions(+), 4 deletions(-)
Quoted 7 lines
Quoted 1 line
Reviewed-by Michael Kelley <[email protected]>It seems to me only the last patch is new, others are already in my tree, so I only apply the last one.
Let me know if my understanding is wrong.
Tianyu added "Fixes:" tags to some of the other patches in the series. It appears the version you had already queued doesn't have those "Fixes:" tags.
Michael
Wei Liu
maintainerRe: [PATCH V4 6/6] x86/Hyper-V: Report crash data in die() when panic_on_oops is set
Quoted 29 lines · Michael Kelley
From: Wei Liu <[email protected]> Sent: Thursday, April 9, 2020 9:41 AM
Quoted 28 lines · Michael Kelley
Quoted 21 lines
From: Tianyu Lan <[email protected]> Sent: Monday, April 6, 2020 8:54 AM
Quoted 14 lines
When oops happens with
panic_on_oopsunset, the oops thread is killed bydie()and system continues to run. In such case, guest should not report crash register data to host since system still runs. Checkpanic_on_oopsand return directly inhyperv_report_panic()when the function is called in thedie()andpanic_on_oopsis unset. Fix it.Fixes 7ed4325a44ea ("Drivers: hv: vmbus: Make panic reporting to be more useful")Signed-off-by Tianyu Lan <[email protected]>---
Change since v3:
- Fix compile error.
- Add fix commit in the change logQuoted 5 lines
--- arch/x86/hyperv/hv_init.c | 6 +++++- drivers/hv/vmbus_drv.c | 5 +++-- include/asm-generic/mshyperv.h | 2 +- 3 files changed, 9 insertions(+), 4 deletions(-)Quoted 1 line
Reviewed-by Michael Kelley <[email protected]>It seems to me only the last patch is new, others are already in my tree, so I only apply the last one.
Quoted 6 lines
Quoted 2 lines
Let me know if my understanding is wrong.
Tianyu added "Fixes:" tags to some of the other patches in the series. It appears the version you had already queued doesn't have those "Fixes:" tags.
Oh, I only read the subject lines and checked the Reviewed-by tags. I will queue the new series instead.
Wei.
Quoted 2 lines
Michael