Skip to contentSkip to conversation

Catching NForce2 lockup with NMI watchdog

DISCUSSION Jesse Allen 10 participants 31 messages Dec 5, 2003 – Dec 17, 2003

lore
Share & tools
lore
#0 lore raw

Hi,

I have a NForce2 board and can easily reproduce a lockup with grep on an IDE hard disk at UDMA 100. The lockup occurs when both Local APIC + IO-APIC are enabled. It was suggested to me to use NMI watchdog to catch it. However, the NMI watchdog doesn't seem to work.

When I set the kernel parameter "nmi_watchdog=1" I get this message in /var/log/syslog:

Dec  4 20:10:30 tesore kernel: ..MP-BIOS bug: 8254 timer not connected to 
IO-APIC
Dec  4 20:10:30 tesore kernel: timer doesn't work through the IO-APIC - 

disabling NMI Watchdog!

"nmi_watchdog=2" seems to work at first, In /var/log/messages:

Dec  4 20:13:11 tesore kernel: testing NMI watchdog ... OK.

but it still locks up.

I have the complete logs when running with nmi_watchdog, kernel config, and more here:
http://www.chez.com/alors/nforce-lockup-logs.tar.gz

If you have any ideas please give them =)

Jesse

#1 lore raw
Quoted 9 lines · Jesse Allen

Hi,

I have a NForce2 board and can easily reproduce a lockup with grep on an IDE hard disk at UDMA 100. The lockup occurs when both Local APIC + IO-APIC are enabled. It was suggested to me to use NMI watchdog to catch it. However, the NMI watchdog doesn't seem to work.

When I set the kernel parameter "nmi_watchdog=1" I get this message in /var/log/syslog:

Quoted 8 lines
Dec  4 20:10:30 tesore kernel: ..MP-BIOS bug: 8254 timer not connected to 
IO-APIC
Dec  4 20:10:30 tesore kernel: timer doesn't work through the IO-APIC - 

disabling NMI Watchdog!

"nmi_watchdog=2" seems to work at first, In /var/log/messages:

Dec  4 20:13:11 tesore kernel: testing NMI watchdog ... OK.

but it still locks up.

The NMI watchdog can only handle software lockups, since it relies on the CPU, and for nmi_watchdog=1 the I/O-APIC + bus, still running. Hardware lockups result in, well, hardware lockups :-(

#2 lore raw
Quoted 15 lines · Mikael Pettersson
Quoted 14 lines · Jesse Allen

Hi,

I have a NForce2 board and can easily reproduce a lockup with grep on an IDE hard disk at UDMA 100. The lockup occurs when both Local APIC + IO-APIC are enabled. It was suggested to me to use NMI watchdog to catch it. However, the NMI watchdog doesn't seem to work.

When I set the kernel parameter "nmi_watchdog=1" I get this message in /var/log/syslog:

Dec  4 20:10:30 tesore kernel: ..MP-BIOS bug: 8254 timer not connected to 
IO-APIC
Dec  4 20:10:30 tesore kernel: timer doesn't work through the IO-APIC - 

disabling NMI Watchdog!

Quoted 7 lines
Quoted 3 lines

"nmi_watchdog=2" seems to work at first, In /var/log/messages:

Dec  4 20:13:11 tesore kernel: testing NMI watchdog ... OK.

but it still locks up.

The NMI watchdog can only handle software lockups, since it relies on the CPU, and for nmi_watchdog=1 the I/O-APIC + bus, still running. Hardware lockups result in, well, hardware lockups :-(

So does this confirm that the lockups with nforce2 chipsets and apic is actually a hardware problem after all?

Signature · 6 lines
Josh McKinney		     |	Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
                             | They that can give up essential liberty
Linux, the choice       -o)  | to obtain a little temporary safety deserve 
of the GNU generation    /\  | neither liberty or safety. 
                        _\_v |                          -Benjamin Franklin
#3 lore raw
Quoted 20 lines · Josh McKinney
Quoted 19 lines · Mikael Pettersson
Quoted 17 lines · Jesse Allen

Hi,

I have a NForce2 board and can easily reproduce a lockup with grep on an IDE hard disk at UDMA 100. The lockup occurs when both Local APIC + IO-APIC are enabled. It was suggested to me to use NMI watchdog to catch it. However, the NMI watchdog doesn't seem to work.

When I set the kernel parameter "nmi_watchdog=1" I get this message in /var/log/syslog:

Dec  4 20:10:30 tesore kernel: ..MP-BIOS bug: 8254 timer not connected to 
IO-APIC
Dec  4 20:10:30 tesore kernel: timer doesn't work through the IO-APIC - 

disabling NMI Watchdog!

"nmi_watchdog=2" seems to work at first, In /var/log/messages:

Dec  4 20:13:11 tesore kernel: testing NMI watchdog ... OK.

but it still locks up.

Quoted 6 lines
Quoted 3 lines

The NMI watchdog can only handle software lockups, since it relies on the CPU, and for nmi_watchdog=1 the I/O-APIC + bus, still running. Hardware lockups result in, well, hardware lockups :-(

So does this confirm that the lockups with nforce2 chipsets and apic is actually a hardware problem after all?

Confirm with very high probability. There may be quirks in nVidia's chipset that we (unlike their Windoze drivers) don't know about.

Ask nVidia for detailed chipset documentation. Then maybe we can fix this.

#4 lore raw

I'm getting those in dmesg too...

..MP-BIOS bug: 8254 timer not connected to IO-APIC ...trying to set up timer (IRQ0) through the 8259A ... failed. ...trying to set up timer as Virtual Wire IRQ... failed. ...trying to set up timer as ExtINT IRQ... works.

Do you really think this could be the problem?

If so, any ideas why I am relatively lucky to not have the crashes people are having? 5.5 days, then 5 hours, and now Im up to 17 hours... with a decent amount of use combined with idle time.

Craig

Quoted 38 lines · Mikael Pettersson
Quoted 20 lines · Josh McKinney
Quoted 19 lines · Mikael Pettersson
Quoted 17 lines · Jesse Allen

Hi,

I have a NForce2 board and can easily reproduce a lockup with grep on an IDE hard disk at UDMA 100. The lockup occurs when both Local APIC + IO-APIC are enabled. It was suggested to me to use NMI watchdog to catch it. However, the NMI watchdog doesn't seem to work.

When I set the kernel parameter "nmi_watchdog=1" I get this message in /var/log/syslog:

Dec  4 20:10:30 tesore kernel: ..MP-BIOS bug: 8254 timer not connected to 
IO-APIC
Dec  4 20:10:30 tesore kernel: timer doesn't work through the IO-APIC - 

disabling NMI Watchdog!

"nmi_watchdog=2" seems to work at first, In /var/log/messages:

Dec  4 20:13:11 tesore kernel: testing NMI watchdog ... OK.

but it still locks up.

Quoted 6 lines
Quoted 3 lines

The NMI watchdog can only handle software lockups, since it relies on the CPU, and for nmi_watchdog=1 the I/O-APIC + bus, still running. Hardware lockups result in, well, hardware lockups :-(

So does this confirm that the lockups with nforce2 chipsets and apic is actually a hardware problem after all?

Confirm with very high probability. There may be quirks in nVidia's chipset that we (unlike their Windoze drivers) don't know about.

Ask nVidia for detailed chipset documentation. Then maybe we can fix this. -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#5 lore raw

Having just had another hang.. I tried booting with nmi-watchdog=1 and then with 2.

I am currently running from the boot with 2 selected.

In my current dmesg I have these which dont normally appear and didnt appear in the boot with 1 set.

Any ideas?

hda: IRQ probe failed (0xfffffcfa)
hdb: IRQ probe failed (0xfffffcfa)
hdb: IRQ probe failed (0xfffffcfa)

Craig

Quoted 63 lines · Craig Bradney

I'm getting those in dmesg too...

..MP-BIOS bug: 8254 timer not connected to IO-APIC ...trying to set up timer (IRQ0) through the 8259A ... failed. ...trying to set up timer as Virtual Wire IRQ... failed. ...trying to set up timer as ExtINT IRQ... works.

Do you really think this could be the problem?

If so, any ideas why I am relatively lucky to not have the crashes people are having? 5.5 days, then 5 hours, and now Im up to 17 hours... with a decent amount of use combined with idle time.

Craig

Quoted 31 lines · Mikael Pettersson
Quoted 20 lines · Josh McKinney
Quoted 19 lines · Mikael Pettersson
Quoted 17 lines · Jesse Allen

Hi,

I have a NForce2 board and can easily reproduce a lockup with grep on an IDE hard disk at UDMA 100. The lockup occurs when both Local APIC + IO-APIC are enabled. It was suggested to me to use NMI watchdog to catch it. However, the NMI watchdog doesn't seem to work.

When I set the kernel parameter "nmi_watchdog=1" I get this message in /var/log/syslog:

Dec  4 20:10:30 tesore kernel: ..MP-BIOS bug: 8254 timer not connected to 
IO-APIC
Dec  4 20:10:30 tesore kernel: timer doesn't work through the IO-APIC - 

disabling NMI Watchdog!

"nmi_watchdog=2" seems to work at first, In /var/log/messages:

Dec  4 20:13:11 tesore kernel: testing NMI watchdog ... OK.

but it still locks up.

Quoted 6 lines
Quoted 3 lines

The NMI watchdog can only handle software lockups, since it relies on the CPU, and for nmi_watchdog=1 the I/O-APIC + bus, still running. Hardware lockups result in, well, hardware lockups :-(

So does this confirm that the lockups with nforce2 chipsets and apic is actually a hardware problem after all?

Confirm with very high probability. There may be quirks in nVidia's chipset that we (unlike their Windoze drivers) don't know about.

Quoted 7 lines

Ask nVidia for detailed chipset documentation. Then maybe we can fix this. -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#6 lore raw
In reply to Craig Bradney

Please don't CC me, I am on the list, thanks.

Quoted 8 lines · Craig Bradney

I'm getting those in dmesg too...

..MP-BIOS bug: 8254 timer not connected to IO-APIC ...trying to set up timer (IRQ0) through the 8259A ... failed. ...trying to set up timer as Virtual Wire IRQ... failed. ...trying to set up timer as ExtINT IRQ... works.

Quoted 8 lines

Do you really think this could be the problem?

If so, any ideas why I am relatively lucky to not have the crashes people are having? 5.5 days, then 5 hours, and now Im up to 17 hours... with a decent amount of use combined with idle time.

Craig

At least two of us are lucky. I can't reproduce the crashes "anymore" either. I am up to 2 days now, was up to 3 or 4 before I booted 2.4.23 for a while to see if I could make that kernel crash, which I couldn't. I will see how long I can go, since 5 days or so seems to be the top uptime.

Signature · 6 lines
Josh McKinney		     |	Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
                             | They that can give up essential liberty
Linux, the choice       -o)  | to obtain a little temporary safety deserve 
of the GNU generation    /\  | neither liberty or safety. 
                        _\_v |                          -Benjamin Franklin
#7 lore raw
In reply to Mikael Pettersson
Quoted 15 lines · Mikael Pettersson
Quoted 14 lines · Jesse Allen

Hi,

I have a NForce2 board and can easily reproduce a lockup with grep on an IDE hard disk at UDMA 100. The lockup occurs when both Local APIC + IO-APIC are enabled. It was suggested to me to use NMI watchdog to catch it. However, the NMI watchdog doesn't seem to work.

When I set the kernel parameter "nmi_watchdog=1" I get this message in /var/log/syslog:

Dec  4 20:10:30 tesore kernel: ..MP-BIOS bug: 8254 timer not connected to 
IO-APIC
Dec  4 20:10:30 tesore kernel: timer doesn't work through the IO-APIC - 

disabling NMI Watchdog!

Quoted 7 lines
Quoted 3 lines

"nmi_watchdog=2" seems to work at first, In /var/log/messages:

Dec  4 20:13:11 tesore kernel: testing NMI watchdog ... OK.

but it still locks up.

The NMI watchdog can only handle software lockups, since it relies on the CPU, and for nmi_watchdog=1 the I/O-APIC + bus, still running. Hardware lockups result in, well, hardware lockups :-(

But nmi_watchdog=1 is supposed to work with APIC, or IO-APIC, and it isn't for his motherboard. It doesn't increment NMI in /proc/interrupts. And it gives the above error message. Isn't that a bug?

#8 lore raw
Quoted 20 lines · Mike Fedyk
Quoted 19 lines · Mikael Pettersson
Quoted 17 lines · Jesse Allen

Hi,

I have a NForce2 board and can easily reproduce a lockup with grep on an IDE hard disk at UDMA 100. The lockup occurs when both Local APIC + IO-APIC are enabled. It was suggested to me to use NMI watchdog to catch it. However, the NMI watchdog doesn't seem to work.

When I set the kernel parameter "nmi_watchdog=1" I get this message in /var/log/syslog:

Dec  4 20:10:30 tesore kernel: ..MP-BIOS bug: 8254 timer not connected to 
IO-APIC
Dec  4 20:10:30 tesore kernel: timer doesn't work through the IO-APIC - 

disabling NMI Watchdog!

"nmi_watchdog=2" seems to work at first, In /var/log/messages:

Dec  4 20:13:11 tesore kernel: testing NMI watchdog ... OK.

but it still locks up.

Quoted 7 lines
Quoted 3 lines

The NMI watchdog can only handle software lockups, since it relies on the CPU, and for nmi_watchdog=1 the I/O-APIC + bus, still running. Hardware lockups result in, well, hardware lockups :-(

But nmi_watchdog=1 is supposed to work with APIC, or IO-APIC, and it isn't for his motherboard. It doesn't increment NMI in /proc/interrupts. And it gives the above error message. Isn't that a bug?

nmi_watchdog=1 only falls back to nmi_watchdog=2 if no SMP is detected. If the I/O-APIC is detected but doesn't work, then the fallback does not happen, and you need to set nmi_watchdog=2 explicitly.

#9 lore raw
In reply to Mike Fedyk
Quoted 5 lines · Mike Fedyk

for his motherboard. It doesn't increment NMI in /proc/interrupts. And it gives the above error message. Isn't that a bug?

But nmi_watchdog=1 is supposed to work with APIC, or IO-APIC, and it isn't

Do you mean like this with an MSI K7N2 Delta MCP2-T mboard
and nmi in kernel and this in cat /proc/interrupts, also in /etc/lilo.conf I have append="nmi_watchdog=1" ? Nothing "nmi" or "NMI" is logged.

cat /proc/interrupts
CPU0

 0:  241105839          XT-PIC  timer
 1:      27337    IO-APIC-edge  i8042
 2:          0          XT-PIC  cascade
 8:          1    IO-APIC-edge  rtc
 9:          0   IO-APIC-level  acpi
12:     217952    IO-APIC-edge  i8042
14:         22    IO-APIC-edge  ide0
15:         24    IO-APIC-edge  ide1
16:    4245875   IO-APIC-level  3ware Storage Controller, yenta, yenta
17:    5428737   IO-APIC-level  eth0
21:          0   IO-APIC-level  NVidia nForce2
NMI:          0
LOC:  241091187
ERR:          0
MIS:          6
#10 lore raw
Quoted 4 lines · Bob

I have append="nmi_watchdog=1" ? Nothing "nmi" or "NMI" is logged.

cat /proc/interrupts
CPU0

Quoted 15 lines
  0:  241105839          XT-PIC  timer
  1:      27337    IO-APIC-edge  i8042
  2:          0          XT-PIC  cascade
  8:          1    IO-APIC-edge  rtc
  9:          0   IO-APIC-level  acpi
 12:     217952    IO-APIC-edge  i8042
 14:         22    IO-APIC-edge  ide0
 15:         24    IO-APIC-edge  ide1
 16:    4245875   IO-APIC-level  3ware Storage Controller, yenta, yenta
 17:    5428737   IO-APIC-level  eth0
 21:          0   IO-APIC-level  NVidia nForce2
NMI:          0
LOC:  241091187
ERR:          0
MIS:          6

You don't have the NMI watchdog working, because the timer interrupt is
configured as an 8259A interrupt ("XT-PIC" for IRQ 0 in the output above). This usually means the wiring of a particular system doesn't provide any other alternative or configuration data provided by the BIOS is broken. The timer interrupt has to be configured as an I/O APIC interrupt for the watchdog to work, or you can select "nmi_watchdog=2" for an alternative watchdog internal to processors if they support it.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#11 lore raw
Quoted 11 lines · Maciej W. Rozycki
Quoted 7 lines · Bob

I have append="nmi_watchdog=1" ? Nothing "nmi" or "NMI" is logged.

cat /proc/interrupts
CPU0
0: 241105839 XT-PIC timer...................
NMI: 0...........

Quoted 8 lines

You don't have the NMI watchdog working, because the timer interrupt is
configured as an 8259A interrupt ("XT-PIC" for IRQ 0 in the output above). This usually means the wiring of a particular system doesn't provide any other alternative or configuration data provided by the BIOS is broken.
The timer interrupt has to be configured as an I/O APIC interrupt for the
watchdog to work, or you can select "nmi_watchdog=2" for an alternative watchdog internal to processors if they support it.

Using a patch that fixes a number of people's nforce2 lockups while enabling io-apic edge timer, I can now use nmi_watchdog=2 but not =1

turn on ioapic edge timer--

http://www.kernel.org/pub/linux/kernel/people/bart/2.6.0-test11-bart1/broken-out/nforce2-apic.patch

We're all trying to get acpi, apic, lapic, io-apic working when turned on in cmos/bios and kernel.

The three things that each alone have achieved stability on somebody's system here are 1) bios update 2) cpu disconnect off either in cmos if available or by athcool or kernel patch with same 3) timing delay patch

For CPU disconnect you still need athcool or this one
http://www.kernel.org/pub/linux/kernel/people/bart/2.6.0-test11-bart1/broken-out/nforce2-disconnect-quirk.patch

Both patches are for 2.6.0-test11 kernel.

#12 lore raw
Quoted 11 lines · Bob
Quoted 8 lines

You don't have the NMI watchdog working, because the timer interrupt is configured as an 8259A interrupt ("XT-PIC" for IRQ 0 in the output above). This usually means the wiring of a particular system doesn't provide any other alternative or configuration data provided by the BIOS is broken. The timer interrupt has to be configured as an I/O APIC interrupt for the watchdog to work, or you can select "nmi_watchdog=2" for an alternative watchdog internal to processors if they support it.

Using a patch that fixes a number of people's nforce2 lockups while enabling io-apic edge timer, I can now use nmi_watchdog=2 but not =1

The I/O APIC NMI watchdog utilizes the property of being transparent to a
single IRQ source of a specially reconfigured 8259A PIC (the master one in the IA32 PC architecture). There are more prerequisites that have to be met and all indeed are for a 100% compatible PC as specified by the Intel's Multiprocessor Specification.

1. The INT output of the master 8259A PIC has to be connected to the LINT0 (or LINTIN0; the name varies by implementations) inputs of all local APICs in the system.

2a. The OUT0 output of the 8254 PIT (IOW the timer source) has to be directly connected to the INTIN2 input of the first I/O APIC.

2b. Alternatively the INT output of the master 8259A PIC has to be connected to the INTIN0 input of the first I/O APIC.

3. There must be no glue logic that would change logical properties of the signal between the INT output of the master 8259A PIC and the respective APIC interrupt inputs.

In practice, assuming the MP IRQ routing information provided the BIOS has been correct (which is not always the case), prerequisites #1 and #2 have been met so far, but #3 has proved to be occasionally problematic.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#13 lore raw

In article <[email protected]>, Maciej W. Rozycki <[email protected]> wrote:

|  The I/O APIC NMI watchdog utilizes the property of being transparent to a
| single IRQ source of a specially reconfigured 8259A PIC (the master one in
| the IA32 PC architecture).  There are more prerequisites that have to be
| met and all indeed are for a 100% compatible PC as specified by the
| Intel's Multiprocessor Specification.
| 
| 1. The INT output of the master 8259A PIC has to be connected to the LINT0
| (or LINTIN0; the name varies by implementations) inputs of all local APICs
| in the system.
| 
| 2a. The OUT0 output of the 8254 PIT (IOW the timer source) has to be 
| directly connected to the INTIN2 input of the first I/O APIC.
| 
| 2b. Alternatively the INT output of the master 8259A PIC has to be
| connected to the INTIN0 input of the first I/O APIC.
| 
| 3. There must be no glue logic that would change logical properties of the
| signal between the INT output of the master 8259A PIC and the respective
| APIC interrupt inputs.
| 
| In practice, assuming the MP IRQ routing information provided the BIOS has
| been correct (which is not always the case), prerequisites #1 and #2 have
| been met so far, but #3 has proved to be occasionally problematic.

In practice many system seem to take a good bit of guessing and testing. I have an old P-II which only works with acpi=force and nmi_watchdog=2, for instance.

It would be nice if there were a program which could poke at the hardware and suggest options which might work, as in eliminating the ones which can be determined not to work. Absent that trial and error rule, unfortunately.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#14 lore raw
Quoted 7 lines · bill davidsen
| In practice, assuming the MP IRQ routing information provided the BIOS has
| been correct (which is not always the case), prerequisites #1 and #2 have
| been met so far, but #3 has proved to be occasionally problematic.

In practice many system seem to take a good bit of guessing and testing. I have an old P-II which only works with acpi=force and nmi_watchdog=2, for instance.

Well, the NMI watchdog is a side-effect feature that works by chance
rather than by design. So you can't really complain it doesn't work somewhere, although I wouldn't mind if new hardware was designed such that it works. You shouldn't have to use "acpi=force" for the watchdog to work though and for a PII system if "nmi_watchdog=1" doesn't work, then I suspect a BIOS bug (set APIC_DEBUG to 1 in asm-i386/apic.h and send me the bootstrap log and a dump from `mptable' for a diagnosis, if interested).

Quoted 4 lines

It would be nice if there were a program which could poke at the hardware and suggest options which might work, as in eliminating the ones which can be determined not to work. Absent that trial and error rule, unfortunately.

Linux has all appropriate bits to set up hardware reasonably as long as
BIOS provides accurate information. The only case our code fails is when BIOS tells us lies and the there's little we can do about it. Actually we are doing hardware manufacturers a favor we try to handle some cases at all -- it's the BIOS that should be fixed instead and it is software and it is stored in Flash memories these days, so there's no excuse. So if there's a problem with running Linux because of BIOS bugs, then please bugger the manufacturer in the first place (and avoid the company in the future if they don't support Linux).

Sometimes the NMI watchdog works in principle, but its activation leads
to system instability -- almost always this is a symptom of buggy SMM code executed by the BIOS behind our back (NMIs are disabled by default in the SMM, but careless code may enable them by accident).

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#15 lore raw
Quoted 33 lines · Maciej W. Rozycki
Quoted 7 lines · bill davidsen
| In practice, assuming the MP IRQ routing information provided the BIOS has
| been correct (which is not always the case), prerequisites #1 and #2 have
| been met so far, but #3 has proved to be occasionally problematic.

In practice many system seem to take a good bit of guessing and testing. I have an old P-II which only works with acpi=force and nmi_watchdog=2, for instance.

Well, the NMI watchdog is a side-effect feature that works by chance
rather than by design. So you can't really complain it doesn't work somewhere, although I wouldn't mind if new hardware was designed such that it works. You shouldn't have to use "acpi=force" for the watchdog to work though and for a PII system if "nmi_watchdog=1" doesn't work, then I suspect a BIOS bug (set APIC_DEBUG to 1 in asm-i386/apic.h and send me the bootstrap log and a dump from `mptable' for a diagnosis, if interested).

Quoted 4 lines

It would be nice if there were a program which could poke at the hardware and suggest options which might work, as in eliminating the ones which can be determined not to work. Absent that trial and error rule, unfortunately.

Linux has all appropriate bits to set up hardware reasonably as long as
BIOS provides accurate information. The only case our code fails is when BIOS tells us lies and the there's little we can do about it. Actually we are doing hardware manufacturers a favor we try to handle some cases at all -- it's the BIOS that should be fixed instead and it is software and it is stored in Flash memories these days, so there's no excuse. So if there's a problem with running Linux because of BIOS bugs, then please bugger the manufacturer in the first place (and avoid the company in the future if they don't support Linux).

Quoted 2 lines

Sometimes the NMI watchdog works in principle, but its activation leads
to system instability -- almost always this is a symptom of buggy SMM code

                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 1 line

executed by the BIOS behind our back (NMIs are disabled by default in the

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Quoted 1 line

SMM, but careless code may enable them by accident).

The NMI vector goes to Linux code. In fact all interrupt vectors go to Linux code. There is no way that some BIOS code could possibly be accidentally executed here. Some Linux code would have to call some 16-bit BIOS code somewhere, and it doesn't even know where..........

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.

#16 lore raw
Quoted 4 lines · Richard B. Johnson
Quoted 2 lines

Sometimes the NMI watchdog works in principle, but its activation leads
to system instability -- almost always this is a symptom of buggy SMM code

                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 1 line

executed by the BIOS behind our back (NMIs are disabled by default in the

Quoted 8 lines

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Quoted 1 line

SMM, but careless code may enable them by accident).

The NMI vector goes to Linux code. In fact all interrupt vectors go to Linux code. There is no way that some BIOS code could possibly be accidentally executed here. Some Linux code would have to call some 16-bit BIOS code somewhere, and it doesn't even know where..........

The problem happens when the SMM is active (i.e. the BIOS code is being
executed) after an SMI has been received during Linux operation (SMIs may get triggered due to various reasons -- a parity/ECC error caught by the chipset, an access to an emulated 8042 controller, a power failure in a notebook, etc.) and an NMI arrives. When in the SMM, no interrupt (including the NMI) causes a switch back into the protected mode (and the processor expects real-mode style interrupt vectors), so the Linux's NMI handler is never reached and the SMM's NMI handler (if at all initialized) isn't appropriate for handling the NMI watchdog. Since the SMM cannot know what NMIs are used for in a particular OS, the code should best keep NMIs disabled -- then an arriving NMI event is latched and postponed until after the RSM instruction is executed.

The SMM was invented to be transparent to a running OS, but care has to
be taken for this to be true and firmware bugs sometimes make the SMM activity visible.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#17 lore raw
In reply to Maciej W. Rozycki
Quoted 10 lines · Maciej W. Rozycki
Quoted 7 lines · bill davidsen
| In practice, assuming the MP IRQ routing information provided the BIOS has
| been correct (which is not always the case), prerequisites #1 and #2 have
| been met so far, but #3 has proved to be occasionally problematic.

In practice many system seem to take a good bit of guessing and testing. I have an old P-II which only works with acpi=force and nmi_watchdog=2, for instance.

Quoted 7 lines

Well, the NMI watchdog is a side-effect feature that works by chance
rather than by design. So you can't really complain it doesn't work somewhere, although I wouldn't mind if new hardware was designed such that it works. You shouldn't have to use "acpi=force" for the watchdog to work though and for a PII system if "nmi_watchdog=1" doesn't work, then I suspect a BIOS bug (set APIC_DEBUG to 1 in asm-i386/apic.h and send me the bootstrap log and a dump from `mptable' for a diagnosis, if interested).

Has the check to see if the BIOS is old than very recent been removed? I used to get a message that the BIOS was too old, I believe that's what prompted the acpi to enable the local apic. Sorrt, I've been running that feature since 2.5.3x or so and I just carried it forward.

Quoted 16 lines
Quoted 5 lines

It would be nice if there were a program which could poke at the hardware and suggest options which might work, as in eliminating the ones which can be determined not to work. Absent that trial and error rule, unfortunately.

Linux has all appropriate bits to set up hardware reasonably as long as
BIOS provides accurate information. The only case our code fails is when BIOS tells us lies and the there's little we can do about it. Actually we are doing hardware manufacturers a favor we try to handle some cases at all -- it's the BIOS that should be fixed instead and it is software and it is stored in Flash memories these days, so there's no excuse. So if there's a problem with running Linux because of BIOS bugs, then please bugger the manufacturer in the first place (and avoid the company in the future if they don't support Linux).

Quoted 4 lines

Sometimes the NMI watchdog works in principle, but its activation leads
to system instability -- almost always this is a symptom of buggy SMM code executed by the BIOS behind our back (NMIs are disabled by default in the SMM, but careless code may enable them by accident).

Works fine for me, system stays up for 30-40 days when I let it... I also run softdog to catch hangs in user mode but not in the kernel. That also works.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#18 lore raw
Quoted 8 lines · Bill Davidsen
Quoted 7 lines

Well, the NMI watchdog is a side-effect feature that works by chance
rather than by design. So you can't really complain it doesn't work somewhere, although I wouldn't mind if new hardware was designed such that it works. You shouldn't have to use "acpi=force" for the watchdog to work though and for a PII system if "nmi_watchdog=1" doesn't work, then I suspect a BIOS bug (set APIC_DEBUG to 1 in asm-i386/apic.h and send me the bootstrap log and a dump from `mptable' for a diagnosis, if interested).

Quoted 4 lines

Has the check to see if the BIOS is old than very recent been removed? I used to get a message that the BIOS was too old, I believe that's what prompted the acpi to enable the local apic. Sorrt, I've been running that feature since 2.5.3x or so and I just carried it forward.

I don't know what check you refer to, sorry. I don't think we do any
version checks in the APIC code. Perhaps ACPI does some, but having no use for it anywhere I'm not familiar with that area.

If the "nmi_watchdog=1" option doesn't work for a PII system, then its
most likely a bug in BIOS IRQ routing tables -- either missing or broken entries for the 8254 timer and/or the 8259A ExtINTA source.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#19 lore raw
In reply to bill davidsen

Having had cause to try and figure out all this, I vote for the following being included in the source somewhere...

-g

Quoted 29 lines · bill davidsen

In article <[email protected]>, Maciej W. Rozycki <[email protected]> wrote:

|  The I/O APIC NMI watchdog utilizes the property of being transparent to a
| single IRQ source of a specially reconfigured 8259A PIC (the master one in
| the IA32 PC architecture).  There are more prerequisites that have to be
| met and all indeed are for a 100% compatible PC as specified by the
| Intel's Multiprocessor Specification.
| | 1. The INT output of the master 8259A PIC has to be connected to the LINT0
| (or LINTIN0; the name varies by implementations) inputs of all local APICs
| in the system.
| | 2a. The OUT0 output of the 8254 PIT (IOW the timer source) has to be | directly connected to the INTIN2 input of the first I/O APIC.
| | 2b. Alternatively the INT output of the master 8259A PIC has to be
| connected to the INTIN0 input of the first I/O APIC.
| | 3. There must be no glue logic that would change logical properties of the
| signal between the INT output of the master 8259A PIC and the respective
| APIC interrupt inputs.
| | In practice, assuming the MP IRQ routing information provided the BIOS has
| been correct (which is not always the case), prerequisites #1 and #2 have
| been met so far, but #3 has proved to be occasionally problematic.

In practice many system seem to take a good bit of guessing and testing. I have an old P-II which only works with acpi=force and nmi_watchdog=2, for instance.

It would be nice if there were a program which could poke at the hardware and suggest options which might work, as in eliminating the ones which can be determined not to work. Absent that trial and error rule, unfortunately.

Signature · 3 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
#20 lore raw
Quoted 2 lines · George Anzinger

Having had cause to try and figure out all this, I vote for the following being included in the source somewhere...

Hmm, you could have simply asked... ;-) Anyway, an inclusion is doable,
I guess.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#21 lore raw
Quoted 9 lines · Maciej W. Rozycki
Quoted 1 line · George Anzinger

Having had cause to try and figure out all this, I vote for the following being included in the source somewhere...

Hmm, you could have simply asked... ;-) Anyway, an inclusion is doable,
I guess.

I suspect I did, but most likey the wrong place. In any case, I would like to think that "read the source, Luke" is the right answer.

So, while I am in the asking mode, is there a simple way to turn off the PIT interrupt without changing the PIT program? I would like a way to stop the interrupts AND also stop the NMIs that it generates for the watchdog. I suspect that this is a bit more complex that it would appear, due to how its wired.

Signature · 3 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
#22 lore raw
Quoted 5 lines · George Anzinger
Quoted 2 lines

Hmm, you could have simply asked... ;-) Anyway, an inclusion is doable,
I guess.

I suspect I did, but most likey the wrong place. In any case, I would like to think that "read the source, Luke" is the right answer.

Certainly it is, but not necessarily the only one. ;-)

Quoted 4 lines

So, while I am in the asking mode, is there a simple way to turn off the PIT interrupt without changing the PIT program? I would like a way to stop the interrupts AND also stop the NMIs that it generates for the watchdog. I suspect that this is a bit more complex that it would appear, due to how its wired.

Well, in PC/AT compatible implementations, the counter #0 of the PIT has
its gate hardwired to active, so you cannot mask the PIT output itself. So the only other choices are either reprogramming the counter to a mode that won't cause periodic triggers (which is probably the easiest way, but you don't want to do that for some purpose, right?) or reprogramming interrupt controllers not to accept interrupts arriving from the PIT.

Note that Linux may behave strangely then. ;-)

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#23 lore raw
Quoted 15 lines · Maciej W. Rozycki
Quoted 5 lines · George Anzinger
Quoted 2 lines

Hmm, you could have simply asked... ;-) Anyway, an inclusion is doable,
I guess.

I suspect I did, but most likey the wrong place. In any case, I would like to think that "read the source, Luke" is the right answer.

Certainly it is, but not necessarily the only one. ;-)

Quoted 4 lines

So, while I am in the asking mode, is there a simple way to turn off the PIT interrupt without changing the PIT program? I would like a way to stop the interrupts AND also stop the NMIs that it generates for the watchdog. I suspect that this is a bit more complex that it would appear, due to how its wired.

Quoted 9 lines

Well, in PC/AT compatible implementations, the counter #0 of the PIT has
its gate hardwired to active, so you cannot mask the PIT output itself. So the only other choices are either reprogramming the counter to a mode that won't cause periodic triggers (which is probably the easiest way, but you don't want to do that for some purpose, right?) or reprogramming interrupt controllers not to accept interrupts arriving from the PIT.

Note that Linux may behave strangely then. ;-)

Masking OFF the timer channel 0 in the interrupt controller is probably the easiest thing to do. The port is read-write, and the OCW default to having it accessible.

	movw	$0x21, %dx	# Controller 0, mask register
	inb	%dx, %al	# Get mask
	orb	$1, %al		# Mask off bit 0
	outb	%al, %dx	# Write it back

You can reenable by:

	movw	$0x21, %dx
	inb	%dx, %al
	andb	$~1, %al
	outb	%al, %dx

With port numbers less that 256, you actually don't need the DX register but I forget if the AT&T assembler needs a $ before the port number when doing this.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.

#24 lore raw
Quoted 3 lines · Richard B. Johnson

Masking OFF the timer channel 0 in the interrupt controller is probably the easiest thing to do. The port is read-write, and the OCW default to having it accessible.

Note we are writing about configurations involving an I/O APIC, so things
are not that easy -- the 8254 timer IRQ may be wired in different ways.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#25 lore raw

Review

Quoted 13 lines · Maciej W. Rozycki
Quoted 3 lines · Richard B. Johnson

Masking OFF the timer channel 0 in the interrupt controller is probably the easiest thing to do. The port is read-write, and the OCW default to having it accessible.

Note we are writing about configurations involving an I/O APIC, so things
are not that easy -- the 8254 timer IRQ may be wired in different ways.

--

(quoted diff) +3 -0
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +

Well if I was trying to isolate a problem, I would make it that easy. You boot the machine in its simplist configuration and work "up" from there.

Although I haven't looked at recent source-code, with APIC, the problem is even simpler. If you booted with APIC, just set the global "using_apic_timer" to zero and, voila`, timer-ticks stop.

Any any event, the caller needs to know that if there is any code executing anywhere that does the equivalent of

		for(;;)
                      ;

...the machine will lock-up forever because without that timer, there will be no preemption. Once a CPU-hog gets the CPU, only and interrupt can get it away.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.

#26 lore raw
Quoted 4 lines · Richard B. Johnson

Although I haven't looked at recent source-code, with APIC, the problem is even simpler. If you booted with APIC, just set the global "using_apic_timer" to zero and, voila`, timer-ticks stop.

Except we are writing of the 8254 timer, not the local APIC one...

Quoted 3 lines

...the machine will lock-up forever because without that timer, there will be no preemption. Once a CPU-hog gets the CPU, only and interrupt can get it away.

And the 8254 timer isn't used for preemption when local APICs are used,
so disabling it won't break the whole system, only the timekeeping.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#27 lore raw
In reply to Maciej W. Rozycki
Quoted 15 lines · Maciej W. Rozycki
Quoted 4 lines · George Anzinger
Quoted 2 lines

Hmm, you could have simply asked... ;-) Anyway, an inclusion is doable, I guess.

I suspect I did, but most likey the wrong place. In any case, I would like to think that "read the source, Luke" is the right answer.

Certainly it is, but not necessarily the only one. ;-)

Quoted 1 line

So, while I am in the asking mode, is there a simple way to turn off the PIT interrupt without changing the PIT program? I would like a way to stop the interrupts AND also stop the NMIs that it generates for the watchdog. I suspect that this is a bit more complex that it would appear, due to how its wired.

Quoted 7 lines

Well, in PC/AT compatible implementations, the counter #0 of the PIT has
its gate hardwired to active, so you cannot mask the PIT output itself. So the only other choices are either reprogramming the counter to a mode that won't cause periodic triggers (which is probably the easiest way, but
you don't want to do that for some purpose, right?) or reprogramming
interrupt controllers not to accept interrupts arriving from the PIT.

Note that Linux may behave strangely then. ;-)

This is for the VST code where we want to stop the timer interrupts for a bit IF and only if we are in the idle task AND there are no timers to service, i.e. the interrupt would be useless. We don't want to mess with the PIT program as that would mess up the time when we turn it on again. So we just want to stop a few interrupts from time to time. We catch up after turning the PIT back on by using the TSC or pm_timer or some other source that keeps something close to reasonable time.

Quoted 1 line
Signature · 3 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
#28 lore raw
Quoted 7 lines · George Anzinger

This is for the VST code where we want to stop the timer interrupts for a bit IF and only if we are in the idle task AND there are no timers to service, i.e. the interrupt would be useless. We don't want to mess with the PIT program as that would mess up the time when we turn it on again. So we just want to stop a few interrupts from time to time. We catch up after turning the PIT back on by using the TSC or pm_timer or some other source that keeps something close to reasonable time.

I see. Well, then disable_irq(0) may be the easiest way to do that for
the regular timer interrupt. For the NMI watchdog from the I/O APIC you'd use disable_8259A_irq(0) and for one from the local APIC -- just mask the APIC_LVTPC interrupt (there's no wrapper function, but that's easy).

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#29 lore raw
Quoted 10 lines · Maciej W. Rozycki
Quoted 1 line · George Anzinger

This is for the VST code where we want to stop the timer interrupts for a bit IF and only if we are in the idle task AND there are no timers to service, i.e. the interrupt would be useless. We don't want to mess with the PIT program as that would mess up the time when we turn it on again. So we just want to stop a few interrupts from time to time. We catch up after turning the PIT back on by using the TSC or pm_timer or some other source that keeps something close to reasonable time.

I see. Well, then disable_irq(0) may be the easiest way to do that for
the regular timer interrupt. For the NMI watchdog from the I/O APIC you'd use disable_8259A_irq(0) and for one from the local APIC -- just mask the APIC_LVTPC interrupt (there's no wrapper function, but that's easy).

How confusing :( Could you give me some idea how this works? I have tried disable_irq(0) and, as best as I can tell, it does not do the trick. The confusion I have is understanding where in the chain of hardware each of these thing is taking place.

For example, it would be "nice" if I could just turn off the PIT interrupt line so that both the NMI (PIT generated) and the PIT interrupt would be put on hold.
Your answer seems to indicate that disable_irq() is working down stream from
where the NMI signal is connected to the PIT interrupt line, so we need to turn of the NMI as well. A picture would be nice here :)

Quoted 1 line
Signature · 3 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
#30 lore raw
Quoted 4 lines · George Anzinger

How confusing :( Could you give me some idea how this works? I have tried disable_irq(0) and, as best as I can tell, it does not do the trick. The confusion I have is understanding where in the chain of hardware each of these thing is taking place.

Well, strange -- it should mask the timer interrupt. But I've never
tried that and have proposed based on a source study only -- perhaps it needs to be further investigated.

Quoted 2 lines

For example, it would be "nice" if I could just turn off the PIT interrupt line so that both the NMI (PIT generated) and the PIT interrupt would be put on hold.

The counter gate of the 8254 chip is designed to do just that -- it's a
pity it's hardwired, but I can understand another SSI TTL latch of a dubious utility was just too costly for the original PC in 1981.

Quoted 3 lines

Your answer seems to indicate that disable_irq() is working down stream from
where the NMI signal is connected to the PIT interrupt line, so we need to turn of the NMI as well. A picture would be nice here :)

I'll try my best:

 +------+ OUT0                                INTIN2 +--------+
 | 8254 +--+-----------------------------------------+        |
 +------+  |                                         |  I/O   |
           | IR0 +------+ INT +------+        INTIN0 |  APIC  |
           +-----+ 8259 +-----+ glue +-+-------------+        |
                 +------+     +------+ |             +---++---+
                                       |                 ||
                                       |                 ||
                                       |                 ||
                           +-----------+---------+-...   ||
                           |                     |       ||
        +--------+         |  +--------+         |       ||
        | CPU #0 |         |  | CPU #1 |         |       ||
        +--------+         |  +--------+         |       ||
        |        | LINT0   |  |        | LINT0   | ...   ||
        | local  +---------+  | local  +---------+       ||
        | APIC   |            | APIC   |                 ||
        |        |            |        |                 ||
        +---++---+            +---++---+                 ||
            ||                    ||                     ||
            || inter-APIC bus     ||                     ||
            ++====================++===============...===++

The system is a traditional i82489DX/Pentium/P6-style virtual-wire setup with a serial inter-APIC bus and a full MP-spec feature set. More limited systems may miss the OUT0->INTIN2 line and/or one or more of the INT->INTIN0 or INT->LINT0 -- there needs to be only one. If any INT->sth connections are missing then either the INT->LINT0 one for the bootstrap processor (BSP) or the INT->LINT0 has to exist; other are optional.

For the system above the path for the 8254 timer interrupt is via INTIN2
and the inter-APIC bus as a LoPri APIC interrupt. The path for the NMI watchdog is via the 8259 reconfigured to pass IR0 transparently to INT and then LINT0 inputs of all processors, reconfigured for a NMI APIC interrupt. Some glue at the INT output may prevent the NMI watchdog from working -- the LINT0 inputs may not toggle back and forth.

If the OUT0->INTIN2 line is missing, the path for the 8254 timer
interrupt is via the 8259 reconfigured to pass IR0 transparently to INT, then INTIN0 and the inter-APIC bus as a LoPri APIC interrupt. The path for the NMI watchdog is also via the 8259 and then LINT0 inputs of all processors, reconfigured for a NMI APIC interrupt. Again, some glue at the INT output may prevent this set up from working, but if it does work, then both the timer interrupt and the NMI watchdog do -- I've not heard of a system having different glue logic for INTIN0 and LINT0.

If the above variant does not work, as a last resort, the path for the
8254 timer interrupt is via the 8259 reconfigured back into its usual mode and then LINT0 of the BSP reconfigured for an ExtINTA APIC interrupt. Additionally, since at this point the glue logic has probably already locked up due to the messing done above, a few artiffical sets of double INTA cycles are sent to the system bus using the RTC chip and INTIN8 reconfigured temporarily to send ExtINTA APIC interrupts via the inter-APIC bus.

I do hope a thorough read of the description will make the available
variants clear. The I/O APIC input numbers may differ but so far they are almost always as noted above.

Maciej

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +