Extending NMI watchdog during LPM
Subsystems (3)
v4 cover Extending NMI watchdog during LPM
When a partition is transferred, once it arrives at the destination node, the partition is active but much of its memory must be transferred from the start node.
It depends on the activity in the partition, but the more CPU the partition has, the more memory to be transferred is likely to be. This causes latency when accessing pages that need to be transferred, and often, for large partitions, it triggers the NMI watchdog.
The NMI watchdog causes the CPU stack to dump where it appears to be stuck. In this case, it does not bring much information since it can happen during any memory access of the kernel.
In addition, the NMI interrupt mechanism is not secure and can generate a dump system in the event that the interruption is taken while MSR[RI]=0.
Depending on the LPAR size and load, it may be interesting to extend the NMI watchdog timer during the LPM.
That's configurable through sysctl with the new introduced variable (specific to powerpc) nmi_watchdog_factor. This value represents the percentage added to watchdog_tresh to set the NMI watchdog timeout during a LPM.
Changes in v4 (no functional changes in this version):
- Patch 1/4 :fix typo and add a comment in pseries_migrate_partition()
- Patch 3/4: rename new variables and functions as Nick requested.
- Patch 4/4: rename the called new function
v2:
https://lore.kernel.org/linuxppc-dev/[email protected]/
Laurent Dufour (4):
powerpc/mobility: wait for memory transfer to complete
watchdog: export lockup_detector_reconfigure powerpc/watchdog: introduce a NMI watchdog's factor pseries/mobility: set NMI watchdog factor during LPM
Documentation/admin-guide/sysctl/kernel.rst | 12 +++ arch/powerpc/include/asm/nmi.h | 2 + arch/powerpc/kernel/watchdog.c | 21 ++++- arch/powerpc/platforms/pseries/mobility.c | 91 ++++++++++++++++++++- include/linux/nmi.h | 2 + kernel/watchdog.c | 21 +++-- 6 files changed, 141 insertions(+), 8 deletions(-)
Signature · 1 line
2.37.0
PATCH v4 1/4 powerpc/mobility: wait for memory transfer to complete
In pseries_migration_partition(), loop until the memory transfer is complete. This way the calling drmgr process will not exit earlier, allowing callbacks to be run only once the migration is fully completed.
If reading the VASI state is done after the hypervisor has completed the migration, the HCALL is returning H_PARAMETER. We can safely assume that the memory transfer is achieved if this happens.
This will also allow to manage the NMI watchdog state in the next commits.
--- arch/powerpc/platforms/pseries/mobility.c | 48 ++++++++++++++++++++++- 1 file changed, 46 insertions(+), 2 deletions(-)
arch/powerpc/platforms/pseries/mobility.c +46 -2
@@ -427,6 +427,43 @@ static int wait_for_vasi_session_suspending(u64 handle)return ret; } +static void wait_for_vasi_session_completed (u64 handle) +{ +unsigned long state =0 ; +int ret; + +pr_info ("waiting for memory transfer to complete...\n" ); + +/* +* Wait for transition from H_VASI_RESUMED to H_VASI_COMPLETED. +*/ +while (true) { + ret =poll_vasi_state (handle, &state); + +/* +* If the memory transfer is already complete and the migration +* has been cleaned up by the hypervisor, H_PARAMETER is return, +* which is translate in EINVAL by poll_vasi_state(). +*/ +if (ret == -EINVAL || (!ret && state ==H_VASI_COMPLETED )) { +pr_info ("memory transfer completed.\n" ); +break ; + } + +if (ret) { +pr_err ("H_VASI_STATE return error (%d)\n" , ret); +break ; + } + +if (state !=H_VASI_RESUMED ) { +pr_err ("unexpected H_VASI_STATE result %lu\n" , state); +break ; + } + +msleep (500 ); + } +} +static void prod_single (unsigned int target_cpu) {long hvrc;
@@ -673,9 +710,16 @@ static int pseries_migrate_partition(u64 handle)vas_migration_handler (VAS_SUSPEND ); ret =pseries_suspend (handle); -if (ret ==0 ) +if (ret ==0 ) {post_mobility_fixup (); -else +/* +* Wait until the memory transfer is complete, so that the user +* space process returns from the syscall after the transfer is +* complete. This allows the user hooks to be executed at the +* right time. +*/ +wait_for_vasi_session_completed (handle); + }else pseries_cancel_migration (handle, ret);vas_migration_handler (VAS_RESUME );
Signature · 1 line
2.37.0
PATCH v4 2/4 watchdog: export lockup_detector_reconfigure
In reply to Laurent DufourIn some circumstances it may be interesting to reconfigure the watchdog from inside the kernel.
On PowerPC, this may helpful before and after a LPAR migration (LPM) is initiated, because it implies some latencies, watchdog, and especially NMI watchdog is expected to be triggered during this operation. Reconfiguring the watchdog with a factor, would prevent it to happen too frequently during LPM.
Rename lockup_detector_reconfigure() as __lockup_detector_reconfigure() and create a new function lockup_detector_reconfigure() calling __lockup_detector_reconfigure() under the protection of watchdog_mutex.
--- include/linux/nmi.h | 2 ++ kernel/watchdog.c | 21 ++++++++++++++++----- 2 files changed, 18 insertions(+), 5 deletions(-)
include/linux/nmi.h +2 -0
@@ -122,6 +122,8 @@ int watchdog_nmi_probe(void);int watchdog_nmi_enable (unsigned int cpu);void watchdog_nmi_disable (unsigned int cpu); +void lockup_detector_reconfigure (void ); +/** * touch_nmi_watchdog - restart NMI watchdog timeout. *
kernel/watchdog.c +16 -5
@@ -541,7 +541,7 @@ int lockup_detector_offline_cpu(unsigned int cpu)return 0 ; } -static void lockup_detector_reconfigure (void ) +static void __lockup_detector_reconfigure (void ) {cpus_read_lock ();watchdog_nmi_stop ();
@@ -561,6 +561,13 @@ static void lockup_detector_reconfigure(void)__lockup_detector_cleanup (); } +void lockup_detector_reconfigure (void ) +{ +mutex_lock (&watchdog_mutex); +__lockup_detector_reconfigure (); +mutex_unlock (&watchdog_mutex); +} +/* * Create the watchdog infrastructure and configure the detector(s). */
@@ -577,13 +584,13 @@ static __init void lockup_detector_setup(void)return ;mutex_lock (&watchdog_mutex); -lockup_detector_reconfigure (); +__lockup_detector_reconfigure (); softlockup_initialized = true; mutex_unlock(&watchdog_mutex); }#else /* CONFIG_SOFTLOCKUP_DETECTOR */ -static void lockup_detector_reconfigure (void ) +void __lockup_detector_reconfigure (void ) {cpus_read_lock ();watchdog_nmi_stop ();
@@ -591,9 +598,13 @@ static void lockup_detector_reconfigure(void)watchdog_nmi_start (); cpus_read_unlock(); } +static inline void lockup_detector_reconfigure (void ) +{ +__lockup_detector_reconfigure (); +}static inline void lockup_detector_setup (void ) { -lockup_detector_reconfigure (); +__lockup_detector_reconfigure (); }#endif /* !CONFIG_SOFTLOCKUP_DETECTOR */
@@ -633,7 +644,7 @@ static void proc_watchdog_update(void) {/* Remove impossible cpus to keep sysctl output clean. */ cpumask_and (&watchdog_cpumask, &watchdog_cpumask, cpu_possible_mask); -lockup_detector_reconfigure (); +__lockup_detector_reconfigure (); }/*
Signature · 1 line
2.37.0
PATCH v4 3/4 powerpc/watchdog: introduce a NMI watchdog's factor
In reply to Laurent DufourIntroduce a factor which would apply to the NMI watchdog timeout.
This factor is a percentage added to the watchdog_tresh value. The value is set under the watchdog_mutex protection and lockup_detector_reconfigure() is called to recompute wd_panic_timeout_tb.
Once the factor is set, it remains until it is set back to 0, which means no impact.
--- arch/powerpc/include/asm/nmi.h | 2 ++ arch/powerpc/kernel/watchdog.c | 21 ++++++++++++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-)
arch/powerpc/include/asm/nmi.h +2 -0
@@ -5,8 +5,10 @@#ifdef CONFIG_PPC_WATCHDOG extern void arch_touch_nmi_watchdog (void );long soft_nmi_interrupt (struct pt_regs *regs); +void watchdog_nmi_set_timeout_pct (u64 pct);#else static inline void arch_touch_nmi_watchdog (void ) {} +static inline void watchdog_nmi_set_timeout_pct (u64 pct) {}#endif #ifdef CONFIG_NMI_IPI
arch/powerpc/kernel/watchdog.c +20 -1
@@ -91,6 +91,10 @@ static cpumask_t wd_smp_cpus_pending;static cpumask_t wd_smp_cpus_stuck;static u64 wd_smp_last_reset_tb; +#ifdef CONFIG_PPC_PSERIES +static u64 wd_timeout_pct; +#endif +/* * Try to take the exclusive watchdog action / NMI IPI / printing lock. * wd_smp_lock must be held. If this fails, we should return and wait
@@ -527,7 +531,13 @@ static int stop_watchdog_on_cpu(unsigned int cpu)static void watchdog_calc_timeouts (void ) { - wd_panic_timeout_tb = watchdog_thresh * ppc_tb_freq; +u64 threshold = watchdog_thresh; + +#ifdef CONFIG_PPC_PSERIES + threshold += (READ_ONCE (wd_timeout_pct) * threshold) /100 ; +#endif + + wd_panic_timeout_tb = threshold * ppc_tb_freq;/* Have the SMP detector trigger a bit later */ wd_smp_panic_timeout_tb = wd_panic_timeout_tb *3 /2 ;
@@ -570,3 +580,12 @@ int __init watchdog_nmi_probe(void) }return 0 ; } + +#ifdef CONFIG_PPC_PSERIES +void watchdog_nmi_set_timeout_pct (u64 pct) +{ +pr_info ("Set the NMI watchdog timeout factor to %llu%%\n" , pct); +WRITE_ONCE (wd_timeout_pct, pct); +lockup_detector_reconfigure (); +} +#endif
Signature · 1 line
2.37.0
PATCH v4 4/4 pseries/mobility: set NMI watchdog factor during LPM
In reply to Laurent DufourDuring a LPM, while the memory transfer is in progress on the arrival side, some latencies is generated when accessing not yet transferred pages on the arrival side. Thus, the NMI watchdog may be triggered too frequently, which increases the risk to hit a NMI interrupt in a bad place in the kernel, leading to a kernel panic.
Disabling the Hard Lockup Watchdog until the memory transfer could be a too strong work around, some users would want this timeout to be eventually triggered if the system is hanging even during LPM.
Introduce a new sysctl variable nmi_watchdog_factor. It allows to apply a factor to the NMI watchdog timeout during a LPM. Just before the CPU are stopped for the switchover sequence, the NMI watchdog timer is set to
watchdog_tresh + factor%
A value of 0 has no effect. The default value is 200, meaning that the NMI watchdog is set to 30s during LPM (based on a 10s watchdog_tresh value). Once the memory transfer is achieved, the factor is reset to 0.
Setting this value to a high number is like disabling the NMI watchdog during a LPM.
--- Documentation/admin-guide/sysctl/kernel.rst | 12 ++++++ arch/powerpc/platforms/pseries/mobility.c | 43 +++++++++++++++++++++ 2 files changed, 55 insertions(+)
Documentation/admin-guide/sysctl/kernel.rst +12 -0
@@ -592,6 +592,18 @@ to the guest kernel command line (see Documentation/admin-guide/kernel-parameters.rst). +nmi_watchdog_factor (PPC only) +================================== + +Factor apply to to the NMI watchdog timeout (only when ``nmi_watchdog`` is +set to 1). This factor represents the percentage added to +``watchdog_thresh`` when calculating the NMI watchdog timeout during a +LPM. The soft lockup timeout is not impacted. + +A value of 0 means no change. The default value is 200 meaning the NMI +watchdog is set to 30s (based on ``watchdog_thresh`` equal to 10). + + numa_balancing ==============
arch/powerpc/platforms/pseries/mobility.c +43 -0
68 lines not on this page · raw message
Signature · 1 line
2.37.0
Randy Dunlap
reviewerRe: [PATCH v4 4/4] pseries/mobility: set NMI watchdog factor during LPM
Hi--
Quoted 2 lines · Laurent Dufour
During a LPM, while the memory transfer is in progress on the arrival side, some latencies is generated when accessing not yet transferred pages on the
are
Quoted 2 lines
arrival side. Thus, the NMI watchdog may be triggered too frequently, which increases the risk to hit a NMI interrupt in a bad place in the kernel,
an NMI
Quoted 8 lines
leading to a kernel panic.
Disabling the Hard Lockup Watchdog until the memory transfer could be a too strong work around, some users would want this timeout to be eventually triggered if the system is hanging even during LPM.
Introduce a new sysctl variable
nmi_watchdog_factor. It allows to apply a factor to the NMI watchdog timeout during a LPM. Just before the CPU are
an LPM. the CPU is
Quoted 2 lines
stopped for the switchover sequence, the NMI watchdog timer is set to
watchdog_tresh+ factor%
watchdog_thresh
Quoted 3 lines
A value of 0 has no effect. The default value is 200, meaning that the NMI watchdog is set to 30s during LPM (based on a 10s
watchdog_treshvalue).
watchdog_thresh
Quoted 4 lines
Once the memory transfer is achieved, the factor is reset to 0.
Setting this value to a high number is like disabling the NMI watchdog during a LPM.
an LPM.
Quoted 13 lines
Reviewed-by Nicholas Piggin <[email protected]>Signed-off-by Laurent Dufour <[email protected]>--- Documentation/admin-guide/sysctl/kernel.rst | 12 ++++++ arch/powerpc/platforms/pseries/mobility.c | 43 +++++++++++++++++++++ 2 files changed, 55 insertions(+)Documentation/admin-guide/sysctl/kernel.rst +0
-0
Documentation/admin-guide/sysctl/kernel.rst to the guest kernel command line (see · line 593
Quoted 2 lines
Documentation/admin-guide/sysctl/kernel.rst +0
-0@@ -592,6 +592,18 @@ to the guest kernel command line (see Documentation/admin-guide/kernel-parameters.rst).
This entire block should be in kernel-parameters.txt, not .rst, and it should be formatted like everything else in the .txt file.
Documentation/admin-guide/sysctl/kernel.rst to the guest kernel command line (see · line 598
Quoted 5 lines
Documentation/admin-guide/sysctl/kernel.rst +4
-0+nmi_watchdog_factor (PPC only) +================================== + +Factor apply to to the NMI watchdog timeout (only when ``nmi_watchdog`` is
Factor to apply to the NMI
Documentation/admin-guide/sysctl/kernel.rst to the guest kernel command line (see · line 600
Quoted 2 lines
Documentation/admin-guide/sysctl/kernel.rst +2
-0+set to 1). This factor represents the percentage added to +``watchdog_thresh`` when calculating the NMI watchdog timeout during a
during an
Quoted 9 lines
Documentation/admin-guide/sysctl/kernel.rst +6
-0+LPM. The soft lockup timeout is not impacted. + +A value of 0 means no change. The default value is 200 meaning the NMI +watchdog is set to 30s (based on ``watchdog_thresh`` equal to 10). + + numa_balancing ==============
Signature · 1 line
~Randy
Re: [PATCH v4 4/4] pseries/mobility: set NMI watchdog factor during LPM
Le 12/07/2022 à 18:25, Randy Dunlap a écrit :
Quoted 50 lines
Hi--
Quoted 2 lines · Laurent Dufour
During a LPM, while the memory transfer is in progress on the arrival side, some latencies is generated when accessing not yet transferred pages on the
are
Quoted 2 lines
arrival side. Thus, the NMI watchdog may be triggered too frequently, which increases the risk to hit a NMI interrupt in a bad place in the kernel,
an NMI
Quoted 8 lines
leading to a kernel panic.
Disabling the Hard Lockup Watchdog until the memory transfer could be a too strong work around, some users would want this timeout to be eventually triggered if the system is hanging even during LPM.
Introduce a new sysctl variable
nmi_watchdog_factor. It allows to apply a factor to the NMI watchdog timeout during a LPM. Just before the CPU arean LPM. the CPU is
Quoted 2 lines
stopped for the switchover sequence, the NMI watchdog timer is set to
watchdog_tresh+ factor%
watchdog_threshQuoted 3 lines
A value of 0 has no effect. The default value is 200, meaning that the NMI watchdog is set to 30s during LPM (based on a 10s
watchdog_treshvalue).
watchdog_threshQuoted 4 lines
Once the memory transfer is achieved, the factor is reset to 0.
Setting this value to a high number is like disabling the NMI watchdog during a LPM.
an LPM.
Quoted 8 lines
Reviewed-by Nicholas Piggin <[email protected]>Signed-off-by Laurent Dufour <[email protected]>--- Documentation/admin-guide/sysctl/kernel.rst | 12 ++++++ arch/powerpc/platforms/pseries/mobility.c | 43 +++++++++++++++++++++ 2 files changed, 55 insertions(+)
Quoted 10 lines
Documentation/admin-guide/sysctl/kernel.rst to the guest kernel command line (see · line 593
Quoted 7 lines
Documentation/admin-guide/sysctl/kernel.rst +0
-0@@ -592,6 +592,18 @@ to the guest kernel command line (see Documentation/admin-guide/kernel-parameters.rst).This entire block should be in kernel-parameters.txt, not .rst, and it should be formatted like everything else in the .txt file.
Thanks for reviewing this patch.
I'll apply your requests in the next version.
However, regarding the change in kernel-parameters.txt, I'm confused. The newly introduced parameter is only exposed through sysctl. Not as a kernel boot option. In that case, should it be mentioned in kernel-parameters.txt?
Documentation/process/4.Coding.rst says:
The file :ref:Documentation/admin-guide/kernel-parameters.rst <kernelparameters> describes all of the kernel's boot-time parameters. Any patch which adds new parameters should add the appropriate entries to this file.
And Documentation/process/submit-checklist.rst says:
16) All new kernel boot parameters are documented in ``Documentation/admin-guide/kernel-parameters.rst``.
What are the rules about editing .txt or .rst files?
Quoted 24 lines
Documentation/admin-guide/sysctl/kernel.rst to the guest kernel command line (see · line 598
Quoted 5 lines
Documentation/admin-guide/sysctl/kernel.rst +4
-0+nmi_watchdog_factor (PPC only) +================================== + +Factor apply to to the NMI watchdog timeout (only when ``nmi_watchdog`` isFactor to apply to the NMI
Documentation/admin-guide/sysctl/kernel.rst to the guest kernel command line (see · line 600
Quoted 2 lines
Documentation/admin-guide/sysctl/kernel.rst +2
-0+set to 1). This factor represents the percentage added to +``watchdog_thresh`` when calculating the NMI watchdog timeout during aduring an
Quoted 9 lines
Documentation/admin-guide/sysctl/kernel.rst +6
-0+LPM. The soft lockup timeout is not impacted. + +A value of 0 means no change. The default value is 200 meaning the NMI +watchdog is set to 30s (based on ``watchdog_thresh`` equal to 10). + + numa_balancing ==============
Randy Dunlap
reviewerRe: [PATCH v4 4/4] pseries/mobility: set NMI watchdog factor during LPM
Hi,
Quoted 23 lines · Laurent Dufour
Le 12/07/2022 à 18:25, Randy Dunlap a écrit :
Quoted 22 lines
Hi--
Quoted 13 lines · Laurent Dufour
Reviewed-by Nicholas Piggin <[email protected]>Signed-off-by Laurent Dufour <[email protected]>--- Documentation/admin-guide/sysctl/kernel.rst | 12 ++++++ arch/powerpc/platforms/pseries/mobility.c | 43 +++++++++++++++++++++ 2 files changed, 55 insertions(+)Documentation/admin-guide/sysctl/kernel.rst +0
-0Documentation/admin-guide/sysctl/kernel.rst to the guest kernel command line (see · line 593
Quoted 2 lines
Documentation/admin-guide/sysctl/kernel.rst +0
-0@@ -592,6 +592,18 @@ to the guest kernel command line (see Documentation/admin-guide/kernel-parameters.rst).This entire block should be in kernel-parameters.txt, not .rst, and it should be formatted like everything else in the .txt file.
My apologies. I misread the file name.
I don't see a problem with this part of the patch or its location.
Quoted 14 lines
Thanks for reviewing this patch.
I'll apply your requests in the next version.
However, regarding the change in kernel-parameters.txt, I'm confused. The newly introduced parameter is only exposed through sysctl. Not as a kernel boot option. In that case, should it be mentioned in kernel-parameters.txt?
Documentation/process/4.Coding.rstsays:
The file :ref:Documentation/admin-guide/kernel-parameters.rst <kernelparameters>describes all of the kernel's boot-time parameters. Any patch which adds new parameters should add the appropriate entries to this file.
Quoted 5 lines
And
Documentation/process/submit-checklist.rstsays:
16) All new kernel boot parameters are documented in ``Documentation/admin-guide/kernel-parameters.rst``.What are the rules about editing .txt or .rst files?
Yeah, that's a little confusing.
kernel-parameters.txt in included in kernel-parameters.rst when 'make htmldocs' is run, so the produced output looks like it is from the .rst file.
Kernel boot parameters should be added to the .txt file. The .rst file is just intro material.
Thanks.
Signature · 1 line
~Randy