serial console vs NMI watchdog
Subsystems (2)
PATCH 1/1 serial console vs NMI watchdog
SYSRQ-T on serial console can crash the machine. This is because a large amount of output is sent to a slow device while interrupts are disabled. The NMI watchdog triggers.
The interrupt disabling happens in pc_keyb.c:keyboard_interrupt(). Changing this code to *not* disable interrupts looks complex.
I see two ways of fixing this. One is to do the sysrq stuff outside the spin_lock_irq(), with:
static void keyboard_interrupt(int irq, void *dev_id, struct pt_regs *regs) {
+ extern void (*sysrq_handler)(void);
+ void (*my_sysrq_handler)(void);
spin_lock_irq(&kbd_controller_lock);
handle_kbd_event();
+ my_sysrq_handler = sysrq_handler;
+ sysrq_handler = 0;
spin_unlock_irq(&kbd_controller_lock);+ if (my_sysrq_handler)
+ (*my_sysrq_handler)();
}
But I didn't do that, because I suspect there are other places in the kernel (development and debug stuff) where we want to turn the NMI watchdog handler off for a while.
So this patch creates a new API function
enable_nmi_watchdog(int yes);
and uses it within the sysrq code.
BTW: NMI watchdog is now disabled by default in 2.4.3-pre3. The `nmi_watchdog=1' boot option is needed to enable it.
include/linux/irq.h +14 -0
@@ -56,6 +56,20 @@#include <asm/hw_irq.h> /* the arch dependent stuff */ +/** +* enable_nmi_watchdog - enables/disables NMI watchdog checking. +* @yes: If zero, disable +* +* If the architecture supports the NMI watchdog, enable_nmi_watchdog() may be used +* to temporarily disable it. Calls to enable_nmi_watchdog() may be nested - it is +* implemented as an up/down counter, so the calls must be balanced. +*/ +#ifdef ARCH_HAS_NMI_WATCHDOG +extern void enable_nmi_watchdog (int yes); +#else +#define enable_nmi_watchdog (yes) do{} while(0) +#endif +extern int handle_IRQ_event (unsigned int ,struct pt_regs *,struct irqaction *);extern int setup_irq (unsigned int ,struct irqaction * );
include/asm-i386/irq.h +4 -0
@@ -32,4 +32,8 @@extern void disable_irq_nosync (unsigned int );extern void enable_irq (unsigned int ); +#ifdef CONFIG_X86_LOCAL_APIC +#define ARCH_HAS_NMI_WATCHDOG /* See include/linux/irq.h */ +#endif +#endif /* _ASM_IRQ_H */
drivers/char/sysrq.c +7 -0
@@ -23,6 +23,7 @@#include <linux/quotaops.h> #include <linux/smp_lock.h> #include <linux/module.h> +#include <linux/irq.h> #include <asm/ptrace.h>
@@ -69,6 +70,11 @@if (!key)return ; +/* +* Interrupts are disabled, and serial consoles are slow. So +* Let's suspend the NMI watchdog. +*/ +enable_nmi_watchdog (0 ); console_loglevel =7 ;printk (KERN_INFO "SysRq: " );switch (key) {
@@ -152,6 +158,7 @@/* Don't use 'A' as it's handled specially on the Sparc */ } +enable_nmi_watchdog (1 ); console_loglevel = orig_log_level; }
arch/i386/kernel/nmi.c +10 -1
@@ -226,6 +226,15 @@ }static spinlock_t nmi_print_lock =SPIN_LOCK_UNLOCKED ; +static atomic_t nmi_watchdog_enabled =ATOMIC_INIT (0 );/* 0 == enabled */ + +void enable_nmi_watchdog (int yes) +{ +if (yes) +atomic_inc (&nmi_watchdog_enabled); +else +atomic_dec (&nmi_watchdog_enabled); +}void nmi_watchdog_tick (struct pt_regs * regs) {
@@ -255,7 +264,7 @@ sum = apic_timer_irqs[cpu]; -if (last_irq_sums[cpu] == sum) { +if (last_irq_sums[cpu] == sum &&atomic_read (&nmi_watchdog_enabled) ==0 ) {/* * Ayiee, looks like this CPU is stuck ... * wait a few IRQs (5 seconds) before doing the oops ...
Quoted 3 lines · Andrew Morton
+/**
+ *enable_nmi_watchdog- enables/disables NMI watchdog checking.
+ * @yes: If zero, disable
Ugh. I have a feeling that your chances to get Linus to accept this are extremely slim.
Just have two functions, enable_nmi_watchdog and disable_nmi_watchdog. You can make them inline, or even macros...
Ion
Signature · 2 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.Quoted 13 lines · Andrew Morton
arch/i386/kernel/nmi.c +9
-0+static atomic_t nmi_watchdog_enabled =ATOMIC_INIT (0 );/* 0 == enabled */ + +void enable_nmi_watchdog (int yes) +{ +if (yes) +atomic_inc (&nmi_watchdog_enabled); +else +atomic_dec (&nmi_watchdog_enabled); +}void nmi_watchdog_tick (struct pt_regs * regs) {
arch/i386/kernel/nmi.c line 267
Quoted 5 lines
arch/i386/kernel/nmi.c +1
-1@@ -255,7 +264,7 @@ sum = apic_timer_irqs[cpu]; -if (last_irq_sums[cpu] == sum) { +if (last_irq_sums[cpu] == sum &&atomic_read (&nmi_watchdog_enabled) ==0 ) {
Shouldn't that be atomic_read(&nmi_watchdog_enabled) != 0?
Quoted 7 lines · Ion Badulescu
Quoted 3 lines · Andrew Morton
+/**
+ *enable_nmi_watchdog- enables/disables NMI watchdog checking.
+ * @yes: If zero, disable
Quoted 5 lines
Ugh. I have a feeling that your chances to get Linus to accept this are extremely slim.
Just have two functions,
enable_nmi_watchdoganddisable_nmi_watchdog. You can make them inline, or even macros...
You're right.
include/linux/irq.h +15 -0
@@ -56,6 +56,21 @@#include <asm/hw_irq.h> /* the arch dependent stuff */ +/** +* nmi_watchdog_disable - disable NMI watchdog checking. +* +* If the architecture supports the NMI watchdog, nmi_watchdog_disable() may be used +* to temporarily disable it. Use nmi_watchdog_enable() later on. It is implemented +* via an up/down counter, so you must keep the calls balanced. +*/ +#ifdef ARCH_HAS_NMI_WATCHDOG +extern void nmi_watchdog_disable (void ); +extern void nmi_watchdog_enable (void ); +#else +#define nmi_watchdog_disable () do{} while(0) +#define nmi_watchdog_enable () do{} while(0) +#endif +extern int handle_IRQ_event (unsigned int ,struct pt_regs *,struct irqaction *);extern int setup_irq (unsigned int ,struct irqaction * );
include/asm-i386/irq.h +4 -0
@@ -32,4 +32,8 @@extern void disable_irq_nosync (unsigned int );extern void enable_irq (unsigned int ); +#ifdef CONFIG_X86_LOCAL_APIC +#define ARCH_HAS_NMI_WATCHDOG /* See include/linux/irq.h */ +#endif +#endif /* _ASM_IRQ_H */
drivers/char/sysrq.c +7 -0
@@ -23,6 +23,7 @@#include <linux/quotaops.h> #include <linux/smp_lock.h> #include <linux/module.h> +#include <linux/irq.h> #include <asm/ptrace.h>
@@ -69,6 +70,11 @@if (!key)return ; +/* +* Interrupts are disabled, and serial consoles are slow. So +* Let's suspend the NMI watchdog. +*/ +nmi_watchdog_disable (); console_loglevel =7 ;printk (KERN_INFO "SysRq: " );switch (key) {
@@ -152,6 +158,7 @@/* Don't use 'A' as it's handled specially on the Sparc */ } +nmi_watchdog_enable (); console_loglevel = orig_log_level; }
arch/i386/kernel/nmi.c +12 -1
@@ -226,6 +226,17 @@ }static spinlock_t nmi_print_lock =SPIN_LOCK_UNLOCKED ; +static atomic_t nmi_watchdog_disabled =ATOMIC_INIT (0 ); + +void nmi_watchdog_disable (void ) +{ +atomic_inc (&nmi_watchdog_disabled); +} + +void nmi_watchdog_enable (void ) +{ +atomic_dec (&nmi_watchdog_disabled); +}void nmi_watchdog_tick (struct pt_regs * regs) {
@@ -255,7 +266,7 @@ sum = apic_timer_irqs[cpu]; -if (last_irq_sums[cpu] == sum) { +if (last_irq_sums[cpu] == sum &&atomic_read (&nmi_watchdog_disabled) ==0 ) {/* * Ayiee, looks like this CPU is stuck ... * wait a few IRQs (5 seconds) before doing the oops ...
Andrew,
your patch looks too complex, and doesnt cover the case of the serial driver deadlocking. Why not add a "touch_nmi_watchdog_counter()" function that just changes last_irq_sums instead of adding locking? This way deadlocks will be caught in the serial code too. (because touch_nmi() will only "postpone" the NMI watchdog lockup event, not disable it.) This should be a matter of 5 lines ...
Ingo
Quoted 7 lines · Ingo Molnar
Andrew,
your patch looks too complex, and doesnt cover the case of the serial driver deadlocking. Why not add a "
touch_nmi_watchdog_counter()" function that just changeslast_irq_sumsinstead of adding locking? This way deadlocks will be caught in the serial code too. (becausetouch_nmi()will only "postpone" the NMI watchdog lockup event, not disable it.)
kdb has to completely disable the nmi counter while it is in control. All interrupts are disabled, all but one cpus are spinning, the control cpu does busy wait while it polls the input devices. With that model there is no alternative to a complete disable.
Quoted 4 lines · Keith Owens
kdb has to completely disable the nmi counter while it is in control. All interrupts are disabled, all but one cpus are spinning, the control cpu does busy wait while it polls the input devices. With that model there is no alternative to a complete disable.
it sure has an alternative. The 'cpus spinning' code calls touch_nmi() within the busy loop, the polling code on the control CPU too. This is sure more robust and catches lockup bugs in kdb too ...
Ingo
Quoted 3 lines · Ingo Molnar
it sure has an alternative. The 'cpus spinning' code calls
touch_nmi()within the busy loop, the polling code on the control CPU too. This is sure more robust and catches lockup bugs in kdb too ...
Works for me. It even makes kdb simpler.
PATCH 1/1 nmi-watchdog-2.4.2-A1
Quoted 1 line · Keith Owens
Works for me. It even makes kdb simpler.
agreed. Also, touch_nmi_watchdog() is stateless and is thus much less prone to locking bugs.
i've attached nmi-watchdog-2.4.2-A1 that implements touch_nmi_watchdog(), ontop of 2.4.2-ac18, and changes show_state() to use this interface. (the patch also takes the NMI counters out of the obscure in-function place they used to be.)
the patch compiles & boots just fine on 2.4.2-ac18 in both SMP and APIC-less-UP mode. The NMI watchdog is functional, and SysRq-T does not cause a lockup if used with a slow serial console that takes more than 5 seconds to output the tasklist.
Ingo
include/linux/irq.h renamed from include/linux/irq.h.orig +6 -8
@@ -57,18 +57,16 @@#include <asm/hw_irq.h> /* the arch dependent stuff */ /** -* nmi_watchdog_disable - disable NMI watchdog checking. +* touch_nmi_watchdog - restart NMI watchdog timeout. * -* If the architecture supports the NMI watchdog, nmi_watchdog_disable() may be used -* to temporarily disable it. Use nmi_watchdog_enable() later on. It is implemented -* via an up/down counter, so you must keep the calls balanced. +* If the architecture supports the NMI watchdog, touch_nmi_watchdog() +* may be used to reset the timeout - for code which intentionally +* disables interrupts for a long time. This call is stateless. */ #ifdef ARCH_HAS_NMI_WATCHDOG -extern void nmi_watchdog_disable (void ); -extern void nmi_watchdog_enable (void ); +extern void touch_nmi_watchdog (void );#else -#define nmi_watchdog_disable () do{} while(0) -#define nmi_watchdog_enable () do{} while(0) +# define touch_nmi_watchdog () do { } while(0)#endif extern int handle_IRQ_event (unsigned int ,struct pt_regs *,struct irqaction *);
drivers/char/sysrq.c renamed from drivers/char/sysrq.c.orig +2 -3
@@ -72,9 +72,9 @@/* * Interrupts are disabled, and serial consoles are slow. So -* Let's suspend the NMI watchdog. +* let's re-start the NMI watchdog timeout. */ -nmi_watchdog_disable (); +touch_nmi_watchdog (); console_loglevel =7 ;printk (KERN_INFO "SysRq: " );switch (key) {
@@ -158,7 +158,6 @@/* Don't use 'A' as it's handled specially on the Sparc */ } -nmi_watchdog_enable (); console_loglevel = orig_log_level; }
arch/i386/kernel/nmi.c renamed from arch/i386/kernel/nmi.c.orig +24 -25
68 lines not on this page · raw message
Re: [patch] nmi-watchdog-2.4.2-A1
In reply to Keith OwensQuoted 13 lines · Ingo Molnar
Quoted 1 line · Keith Owens
Works for me. It even makes kdb simpler.
agreed. Also,
touch_nmi_watchdog()is stateless and is thus much less prone to locking bugs.i've attached nmi-watchdog-2.4.2-A1 that implements
touch_nmi_watchdog(), ontop of 2.4.2-ac18, and changesshow_state()to use this interface. (the patch also takes the NMI counters out of the obscure in-function place they used to be.)
Quoted 5 lines
the patch compiles & boots just fine on 2.4.2-ac18 in both SMP and APIC-less-UP mode. The NMI watchdog is functional, and SysRq-T does not cause a lockup if used with a slow serial console that takes more than 5 seconds to output the tasklist.
Sorry, this doesn't look right. Are you sure you booted with `nmi_watchdog=1'? It was turned off by default in -ac18.
Two things:
- CPU A could be doing the SYSRQ printing, while
CPU B is spinning on a lock which CPU A holds. The NMI watchdog will then whack CPU B. So touch_nmi_watchdog() needs to touch *all* CPUs. (kbd_controller_lock, for example).
- We need to touch the NMI more than once during the SYSRQ-T output - five seconds isn't enough.
The correctest way is, I think, to touch_nmi() in rs285_console_write(), lp_console_write() and serial_console_write(). We _could_ just touch it in show_state(), but that means we still get whacked if we do a lot of printk()s with interrupts disabled from some random place in the kernel.
arch/i386/kernel/nmi.c +27 -25
@@ -226,37 +226,39 @@ }static spinlock_t nmi_print_lock =SPIN_LOCK_UNLOCKED ; -static atomic_t nmi_watchdog_disabled =ATOMIC_INIT (0 ); -void nmi_watchdog_disable (void ) -{ -atomic_inc (&nmi_watchdog_disabled); -} +/* +* The best way to detect whether a CPU has a 'hard lockup' problem +* is to check its local APIC timer IRQ counts. If they are not +* changing then that CPU has some problem. +* +* as these watchdog NMI IRQs are generated on every CPU, we only +* have to check the current processor. +* +* since NMIs dont listen to _any_ locks, we have to be extremely +* careful not to rely on unsafe variables. The printk might lock +* up though, so we have to break up any console locks first ... +* [when there will be more tty-related locks, break them up +* here too!] +*/ + +static unsigned int + last_irq_sums [NR_CPUS ], + alert_counter [NR_CPUS ]; -void nmi_watchdog_enable (void ) +void touch_nmi_watchdog (void ) { -atomic_dec (&nmi_watchdog_disabled); +/* +* Just reset the alert counters. +*/ +int cpu; + +for (cpu =0 ; cpu < smp_num_cpus; cpu++) + alert_counter[cpu] =0 ; }void nmi_watchdog_tick (struct pt_regs * regs) { -/* -* the best way to detect wether a CPU has a 'hard lockup' problem -* is to check it's local APIC timer IRQ counts. If they are not -* changing then that CPU has some problem. -* -* as these watchdog NMI IRQs are broadcasted to every CPU, here -* we only have to check the current processor. -* -* since NMIs dont listen to _any_ locks, we have to be extremely -* careful not to rely on unsafe variables. The printk might lock -* up though, so we have to break up any console locks first ... -* [when there will be more tty-related locks, break them up -* here too!] -*/ - -static unsigned int last_irq_sums [NR_CPUS ], - alert_counter [NR_CPUS ];/* * Since current-> is always on the stack, and we always switch
@@ -266,7 +268,7 @@ sum = apic_timer_irqs[cpu]; -if (last_irq_sums[cpu] == sum &&atomic_read (&nmi_watchdog_disabled) ==0 ) { +if (last_irq_sums[cpu] == sum) {/* * Ayiee, looks like this CPU is stuck ... * wait a few IRQs (5 seconds) before doing the oops ...
drivers/char/sysrq.c +0 -6
@@ -70,11 +70,6 @@if (!key)return ; -/* -* Interrupts are disabled, and serial consoles are slow. So -* Let's suspend the NMI watchdog. -*/ -nmi_watchdog_disable (); console_loglevel =7 ;printk (KERN_INFO "SysRq: " );switch (key) {
@@ -158,7 +153,6 @@/* Don't use 'A' as it's handled specially on the Sparc */ } -nmi_watchdog_enable (); console_loglevel = orig_log_level; }
drivers/char/serial.c +2 -1
@@ -5478,10 +5478,11 @@/* * Wait for transmitter & holding register to empty */ -static inline void wait_for_xmitr (struct async_struct *info) +static void wait_for_xmitr (struct async_struct *info) {unsigned int status, tmout =1000000 ; +touch_nmi_watchdog (); do { status = serial_in(info, UART_LSR);
drivers/char/serial_21285.c +1 -0
@@ -385,6 +385,7 @@while (*CSR_UARTFLG &0x20 ); *CSR_UARTDR ='\r' ; } +touch_nmi_watchdog (); }enable_irq (IRQ_CONTX ); }
drivers/char/lp.c +2 -1
@@ -576,8 +576,8 @@ canwrite = lf - s;if (canwrite >0 ) { +touch_nmi_watchdog (); written =parport_write (port, s, canwrite); -if (written <=0 )continue ;
@@ -594,6 +594,7 @@ s++; count--; do { + touch_nmi_watchdog(); written = parport_write (port, crlf, i); if (written > 0) i -= written, crlf += written;
include/linux/irq.h +6 -8
@@ -57,18 +57,16 @@#include <asm/hw_irq.h> /* the arch dependent stuff */ /** -* nmi_watchdog_disable - disable NMI watchdog checking. +* touch_nmi_watchdog - restart NMI watchdog timeout. * -* If the architecture supports the NMI watchdog, nmi_watchdog_disable() may be used -* to temporarily disable it. Use nmi_watchdog_enable() later on. It is implemented -* via an up/down counter, so you must keep the calls balanced. +* If the architecture supports the NMI watchdog, touch_nmi_watchdog() +* may be used to reset the timeout - for code which intentionally +* disables interrupts for a long time. This call is stateless. */ #ifdef ARCH_HAS_NMI_WATCHDOG -extern void nmi_watchdog_disable (void ); -extern void nmi_watchdog_enable (void ); +extern void touch_nmi_watchdog (void );#else -#define nmi_watchdog_disable () do{} while(0) -#define nmi_watchdog_enable () do{} while(0) +# define touch_nmi_watchdog () do { } while(0)#endif extern int handle_IRQ_event (unsigned int ,struct pt_regs *,struct irqaction *);
PATCH 1/1 nmi-watchdog-2.4.2-A2
Quoted 2 lines · Andrew Morton
Sorry, this doesn't look right. Are you sure you booted with `nmi_watchdog=1'? It was turned off by default in -ac18.
of course i did ...
Quoted 6 lines
Two things:
- CPU A could be doing the SYSRQ printing, while
CPU B is spinning on a lock which CPU A holds. The NMI watchdog will then whack CPU B. Sotouch_nmi_watchdog()needs to touch *all* CPUs. (kbd_controller_lock, for example).
yep, agreed.
Quoted 6 lines
- We need to touch the NMI more than once during the SYSRQ-T output - five seconds isn't enough.
The correctest way is, I think, to
touch_nmi()inrs285_console_write(),lp_console_write()andserial_console_write().
nope:
Quoted 3 lines
We _could_ just touch it in
show_state(), but that means we still get whacked if we do a lot of printk()s with interrupts disabled from some random place in the kernel.
exactly, and that is a feature. We want to find all those places, because disabling IRQs for too long can cause problems in unrelated kernel code. SysRq-T is a special case so touch_nmi() is justified in that and only that case. The NMI watchdog is something that gives security, and we want to be very conservative disabling its effect.
[i've attached nmi-watchdog-2.4.2-A2 (against -ac18) which adds your fix to clear all alert counters in touch_nmi_watchdog().]
Ingo
kernel/sched.c renamed from kernel/sched.c.orig +7 -1
@@ -1183,8 +1183,14 @@printk (" task PC stack pid father child younger older\n" );#endif read_lock (&tasklist_lock); -for_each_task (p) +for_each_task (p) { +/* +* reset the NMI-timeout, listing all files on a slow +* console might take alot of time: +*/ +touch_nmi_watchdog ();show_task (p); + }read_unlock (&tasklist_lock); }
include/linux/irq.h renamed from include/linux/irq.h.orig +6 -8
@@ -57,18 +57,16 @@#include <asm/hw_irq.h> /* the arch dependent stuff */ /** -* nmi_watchdog_disable - disable NMI watchdog checking. +* touch_nmi_watchdog - restart NMI watchdog timeout. * -* If the architecture supports the NMI watchdog, nmi_watchdog_disable() may be used -* to temporarily disable it. Use nmi_watchdog_enable() later on. It is implemented -* via an up/down counter, so you must keep the calls balanced. +* If the architecture supports the NMI watchdog, touch_nmi_watchdog() +* may be used to reset the timeout - for code which intentionally +* disables interrupts for a long time. This call is stateless. */ #ifdef ARCH_HAS_NMI_WATCHDOG -extern void nmi_watchdog_disable (void ); -extern void nmi_watchdog_enable (void ); +extern void touch_nmi_watchdog (void );#else -#define nmi_watchdog_disable () do{} while(0) -#define nmi_watchdog_enable () do{} while(0) +# define touch_nmi_watchdog () do { } while(0)#endif extern int handle_IRQ_event (unsigned int ,struct pt_regs *,struct irqaction *);
drivers/char/sysrq.c renamed from drivers/char/sysrq.c.orig +0 -6
@@ -70,11 +70,6 @@if (!key)return ; -/* -* Interrupts are disabled, and serial consoles are slow. So -* Let's suspend the NMI watchdog. -*/ -nmi_watchdog_disable (); console_loglevel =7 ;printk (KERN_INFO "SysRq: " );switch (key) {
@@ -158,7 +153,6 @@/* Don't use 'A' as it's handled specially on the Sparc */ } -nmi_watchdog_enable (); console_loglevel = orig_log_level; }
arch/i386/kernel/nmi.c renamed from arch/i386/kernel/nmi.c.orig +28 -25
72 lines not on this page · raw message
Quoted 8 lines · Keith Owens
Quoted 3 lines · Ingo Molnar
it sure has an alternative. The 'cpus spinning' code calls
touch_nmi()within the busy loop, the polling code on the control CPU too. This is sure more robust and catches lockup bugs in kdb too ...Works for me. It even makes kdb simpler.
humm...
OK, this seems doable in the case of serial console. For CONFIG_LP_CONSOLE (which has the same problem) it looks like we can just call touch_nmi() in lp_console_write().
I'll see what Tim has to say.
-
Quoted 11 lines · Keith Owens
Quoted 7 lines · Ingo Molnar
Andrew,
your patch looks too complex, and doesnt cover the case of the serial driver deadlocking. Why not add a "
touch_nmi_watchdog_counter()" function that just changeslast_irq_sumsinstead of adding locking? This way deadlocks will be caught in the serial code too. (becausetouch_nmi()will only "postpone" the NMI watchdog lockup event, not disable it.)
Quoted 5 lines
kdb has to completely disable the nmi counter while it is in control. All interrupts are disabled, all but one cpus are spinning, the control cpu does busy wait while it polls the input devices. With that model there is no alternative to a complete disable.
Consider this. Why not use the NMI to sync the cpus. Kdb would have a function that is called each NMI. If it is doing nothing, just return false, else, if waiting for this cpu, well here it is, put it in spin AFTER saving where it came from so the operator can figure out what it is doing. In kgdb I just put the interrupt registers in the task_struct where they are put when a context switch is done. Then the debugger can do a trace, etc. on that task. A global var that the debugger can see is also set to the cpus, "current".
If the cpu is already spinning, return to the nmi code with a true flag which will cause it to ignore the nmi. Same thing if it is the cpu that is doing debug i/o.
I went to this for kgdb after the system failed to return from the call to force the other cpus to execute a function (which means they have to be alive). For extra safety I also time the sync. If one or more expected cpus, don't show while looping reading the cycle counter, the code just continues with out the sync.
George
Quoted 2 lines · george anzinger
Consider this. Why not use the NMI to sync the cpus. Kdb would have a function that is called each NMI.
kdb uses NMI IPI to get the other cpu's attention. One cpu is in control and may or may not be accepting NMI, it depends on the event that entered kdb. The other cpus end up in kdb code, spinning waiting for a cpu switch. Initially they are not receiving NMI because they were invoked via NMI which is masked until they exit. However if the user does a cpu switch then single steps the interrupted code, the cpu has to return from the NMI handler to the interrupted code at which time this cpu starts receiving NMI again.
The kdb context can change from ignoring NMI to accepting NMI. It is easier to bring all the cpus into kdb and let the kdb code decide if it ignores any NMI that is being received.
Quoted 6 lines · Keith Owens
Quoted 2 lines · george anzinger
Consider this. Why not use the NMI to sync the cpus. Kdb would have a function that is called each NMI.
Quoted 8 lines
kdb uses NMI IPI to get the other cpu's attention. One cpu is in control and may or may not be accepting NMI, it depends on the event that entered kdb. The other cpus end up in kdb code, spinning waiting for a cpu switch. Initially they are not receiving NMI because they were invoked via NMI which is masked until they exit. However if the user does a cpu switch then single steps the interrupted code, the cpu has to return from the NMI handler to the interrupted code at which time this cpu starts receiving NMI again.
Are you actually twiddling the hardware, or just changing what happens on NMI?
Quoted 4 lines
The kdb context can change from ignoring NMI to accepting NMI. It is easier to bring all the cpus into kdb and let the kdb code decide if it ignores any NMI that is being received.
Yes. Exactly.
George
Quoted 9 lines · george anzinger
Quoted 5 lines · Keith Owens
kdb uses NMI IPI to get the other cpu's attention. One cpu is in control and may or may not be accepting NMI, it depends on the event that entered kdb. The other cpus end up in kdb code, spinning waiting for a cpu switch. Initially they are not receiving NMI because they were invoked via NMI which is masked until they exit.
Are you actually twiddling the hardware, or just changing what happens on NMI?
No hardware twiddling. One cpu gets an event which triggers kdb, that event may or may not be via NMI. kdb on ix86 then uses NMI to get the attention of the other cpus, even if they are in a disabled spinloop. ix86 hardware will not deliver another NMI to a cpu until the cpu issues iret to return from the NMI handler.
Initially all but one cpu is forced into kdb via the NMI handler so no more NMI events will occur on those cpus. The first cpu may or may not be receiving NMI, it depends on how kdb was invoked on the first cpu. To do single stepping of code, kdb allows one cpu out of kdb state so it can execute one instruction at the point it was interrupted. If the cpu was entered via NMI then that means exiting from the NMI handler back to the original code, do single step then back into kdb again.
Having exited the NMI handler, that cpu will start receiving NMI events again, even though it is still under kdb control. So some cpus will get NMI, some will not, depending on user actions. kdb uses a software mechanism to selectively disable the NMI watchdog.