Skip to contentSkip to conversation

Catching NForce2 lockup with NMI watchdog

DISCUSSION Ross Dickson 4 participants 13 messages Dec 17, 2003 – Dec 19, 2003

lore
Share & tools
lore
#0 lore raw
Quoted 8 lines · George Anzinger
Quoted 7 lines

How confusing :( Could you give me some idea how this works? I have tried disable_irq(0) and, as best as I can tell, it does not do the trick. The confusion I have is understanding where in the chain of hardware each of these thing is taking place.

Here is where to find Intel's MP arch spec Maceij mentions. I had to find it recently wrt nforce2 issues

http://www.intel.com/design/pentium/datashts/24201606.pdf

Section 3.6.1 Apic Architecture is relevant
particularly
Section 3.6.2.2 Virtual Wire Mode

<snip>
great diagram!

<snip>

Quoted 9 lines

If the above variant does not work, as a last resort, the path for the 8254 timer interrupt is via the 8259 reconfigured back into its usual mode and then LINT0 of the BSP reconfigured for an ExtINTA APIC interrupt. Additionally, since at this point the glue logic has probably already locked up due to the messing done above, a few artiffical sets of double INTA cycles are sent to the system bus using the RTC chip and INTIN8 reconfigured temporarily to send ExtINTA APIC interrupts via the inter-APIC bus.

Quoted 5 lines

I do hope a thorough read of the description will make the available variants clear. The I/O APIC input numbers may differ but so far they are almost always as noted above.

Maciej

All good.

I would like to add a footnote to highlight a potential gotcha as I understand it.

To clarify, the xt pic 8259A does not in itself have a transparent mode as would a logic buffer or inverter. It always needs inta cycles to function. In PIC mode it is wired to processor pins as per old 8086 and original cpu architecture provides the inta cycles to it (bypasses apic, apic seems off).

In virtual wire mode with the 8259A output wired either to a local apic pin on cpu or through the io-apic. In this mode it is the local apic which has to provide the inta cycles on the bus back to the 8259A for it to function correctly.

The delivery mode has to be set to ExtInt for the register associated with the pin that the 8259A output (int on Maceij diagram) is connected to. This is the only way to force the apic to deliver the inta cycles to the 8259A and that is how it appears transparent to the system. Spec says there can only be one source register (local apic) or redirection register (ioapic) of mode ExtInt per system regardless of how many local apic and io-apic pins it (int on Maceij diagram) is connected to.

Gotcha: If none are set to ExtInt then the 8259A will hang for lack of IntA cycles.

Section 7.5.11 covers it
24319202.pdf available here

http://www.intel.com/design/pentiumii/manuals/243192.htm

Why only one Extint source in virtual wire mode?:

The 8259A in X86 architecture systems needs two inta cycles per interrupt event. Do not confuse them with the EOI which is software, the inta is purely hardware. It only works properly with one source causing inta cycles. Docs I have do not say what happens with more than one source.

How 8259A works in a nutshell (it is more complex in cascade mode).

First the 8259A gets a request from H/ware and if unmasked etc generates its int (int on Maceij diagram) out. 8259A then sits there waiting for Inta from cpu (PIC mode) or local apic (Virtual wire mode). When the inta arrives the 8259A latches its internal ISR bit and waits for second inta. When second inta arrives it outputs a vector onto the data bus indicating which ISR bit was set.

If the request from H/ware is still active when the first inta arrives then we get the correct vector number.

If it is NOT still exerted then its tough luck and the vector we get 7 for the first 8259A or 15 for the second 8259A and it is too late to try and find out where the real source was, hence the spurious irq7 messages and corresponding irq 7 count increase.

It is pretty bad when the apic system that is handling the 8259A in virtual wire mode cannot get the inta to the 8259A in time while the int request hardware is still exerting but it happens.

I certainly agree with Marceij's comments that mixed mode of having 8254 PIT routed via the 8259A was never meant to occur alongside ioapic handling of the other interrupts. It is very problematic not to mention confusing.

I do not know how smoothly the apic handles the 8259A if you would be turning that source on and off frequently.

Regards
Ross Dickson

#1 lore raw

I really want to thank you both for all this information. It will make the code and comments much easier to understand.

Thanks again

George

Quoted 98 lines · Ross Dickson
Quoted 7 lines · George Anzinger
Quoted 6 lines

How confusing :( Could you give me some idea how this works? I have tried disable_irq(0) and, as best as I can tell, it does not do the trick. The confusion I have is understanding where in the chain of hardware each of these thing is taking place.

Here is where to find Intel's MP arch spec Maceij mentions. I had to find it recently wrt nforce2 issues

http://www.intel.com/design/pentium/datashts/24201606.pdf

Section 3.6.1 Apic Architecture is relevant
particularly
Section 3.6.2.2 Virtual Wire Mode

<snip>
great diagram!

<snip>

Quoted 9 lines

If the above variant does not work, as a last resort, the path for the 8254 timer interrupt is via the 8259 reconfigured back into its usual mode and then LINT0 of the BSP reconfigured for an ExtINTA APIC interrupt. Additionally, since at this point the glue logic has probably already locked up due to the messing done above, a few artiffical sets of double INTA cycles are sent to the system bus using the RTC chip and INTIN8 reconfigured temporarily to send ExtINTA APIC interrupts via the inter-APIC bus.

I do hope a thorough read of the description will make the available variants clear. The I/O APIC input numbers may differ but so far they are almost always as noted above.

Maciej

All good.

I would like to add a footnote to highlight a potential gotcha as I understand it.

To clarify, the xt pic 8259A does not in itself have a transparent mode as would a logic buffer or inverter. It always needs inta cycles to function. In PIC mode it is wired to processor pins as per old 8086 and original cpu architecture provides the inta cycles to it (bypasses apic, apic seems off).

In virtual wire mode with the 8259A output wired either to a local apic pin on cpu
or through the io-apic. In this mode it is the local apic which has to provide the inta cycles on the bus back to the 8259A for it to function correctly.

The delivery mode has to be set to ExtInt for the register associated with the pin
that the 8259A output (int on Maceij diagram) is connected to. This is the only
way to force the apic to deliver the inta cycles to the 8259A and that is how it
appears transparent to the system. Spec says there can only be one source register (local apic) or redirection register (ioapic) of mode ExtInt per system regardless of how many local apic and io-apic pins it (int on Maceij diagram)
is connected to.

Gotcha: If none are set to ExtInt then the 8259A will hang for lack of IntA cycles.

Section 7.5.11 covers it
24319202.pdf available here

http://www.intel.com/design/pentiumii/manuals/243192.htm

Why only one Extint source in virtual wire mode?:

The 8259A in X86 architecture systems needs two inta cycles per interrupt event. Do not confuse them with the EOI which is software, the inta is purely hardware. It only works properly with one source causing inta cycles. Docs I have do not say what happens with more than one source.

How 8259A works in a nutshell (it is more complex in cascade mode).

First the 8259A gets a request from H/ware and if unmasked etc generates its int (int on Maceij diagram) out. 8259A then sits there waiting for Inta from cpu (PIC mode) or local apic (Virtual wire mode). When the inta arrives the 8259A latches its internal ISR bit and waits for second inta. When second inta arrives
it outputs a vector onto the data bus indicating which ISR bit was set.

If the request from H/ware is still active when the first inta arrives then we get the correct vector number.

If it is NOT still exerted then its tough luck and the vector we get 7 for the first 8259A or 15 for the second 8259A and it is too late to try and find out where the real source was, hence the spurious irq7 messages and corresponding irq 7 count increase.

It is pretty bad when the apic system that is handling the 8259A in virtual wire mode cannot get the inta to the 8259A in time while the int request hardware is still exerting but it happens.

I certainly agree with Marceij's comments that mixed mode of having 8254 PIT routed via the 8259A was never meant to occur alongside ioapic handling of the other interrupts. It is very problematic not to mention confusing.

I do not know how smoothly the apic handles the 8259A if you would be turning that source on and off frequently.

Regards
Ross Dickson

Signature · 3 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
#2 lore raw
In reply to Ross Dickson
Quoted 3 lines · Ross Dickson

Section 7.5.11 covers it
24319202.pdf available here

I wonder if you might know the difference between the 243190/1/2 and the 245470/1/2 manuals. I have hard copies of the ladder.

Signature · 3 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
#3 lore raw
Quoted 8 lines · George Anzinger
Quoted 3 lines · Ross Dickson

Section 7.5.11 covers it
24319202.pdf available here

I wonder if you might know the difference between the 243190/1/2 and the 245470/1/2 manuals. I have hard copies of the ladder.

I grabbed the manuals that google search found. By the look of it what I had covered P3 and earlier. Yours are more up to date and cover P4.
I have since found them on the web

http://www.intel.com/design/pentium4/manuals/245470.htm

Regards
Ross.

Quoted 10 lines
Signature · 4 lines
-- 
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
#4 lore raw
Quoted 2 lines · Ross Dickson

I grabbed the manuals that google search found. By the look of it what I had covered P3 and earlier. Yours are more up to date and cover P4.

Newer manuals sometimes lack details that are present in older ones. If
you want to have a thorough view of the APIC, you certainly want to have all four variations of processor manuals, i.e. the one for P4+, the one for PII+, the one for PPro and the one for Pentium. Plus manuals for the I/O APIC, e.g. the one for the i82093AA and perhaps for ones embedded into various chipsets. All of them are or used to be available online. If you want to go back to the i82489DX, there is a datasheet and a programming manual for the part, which are IMO the most exhaustive descriptions, though the implementation differed a bit from newer ones (the chip was so far the most powerful implementation of the APIC). These were unfortunately never available online.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#5 lore raw
Quoted 5 lines · Maciej W. Rozycki
Quoted 2 lines · Ross Dickson

I grabbed the manuals that google search found. By the look of it what I had covered P3 and earlier. Yours are more up to date and cover P4.

Quoted 12 lines

Newer manuals sometimes lack details that are present in older ones. If
you want to have a thorough view of the APIC, you certainly want to have all four variations of processor manuals, i.e. the one for P4+, the one for PII+, the one for PPro and the one for Pentium. Plus manuals for the I/O APIC, e.g. the one for the i82093AA and perhaps for ones embedded into various chipsets. All of them are or used to be available online. If you want to go back to the i82489DX, there is a datasheet and a programming manual for the part, which are IMO the most exhaustive descriptions, though the implementation differed a bit from newer ones (the chip was so far the most powerful implementation of the APIC). These were unfortunately never available online.

Point taken, I generally play embedded MPU where the codebase matches the specific hardware version and one set of docs suffice, although it is not uncommon to rediscover an unpublished bug.

This one codebase for all hardware certainly is a lot more work!

Do you know if the Athlon apic programming docs are available or under NDA? I do not even want to ask about the nvidia nforce2 chipset.

Regards
Ross.

Quoted 7 lines
(quoted diff) +3 -0
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#6 lore raw
Quoted 1 line · Ross Dickson

Do you know if the Athlon apic programming docs are available or under NDA?

No idea -- I've been only loosely interested in IA-32-based hardware
recently.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#7 lore raw
In reply to Ross Dickson
Quoted 8 lines · Ross Dickson

Here is where to find Intel's MP arch spec Maceij mentions. I had to find it recently wrt nforce2 issues

http://www.intel.com/design/pentium/datashts/24201606.pdf

Section 3.6.1 Apic Architecture is relevant
particularly
Section 3.6.2.2 Virtual Wire Mode

BTW, I have revision 1.1 as well in case anyone was interested in the
differences.

Quoted 8 lines

I would like to add a footnote to highlight a potential gotcha as I understand it.

To clarify, the xt pic 8259A does not in itself have a transparent mode as would a logic buffer or inverter. It always needs inta cycles to function. In PIC mode it is wired to processor pins as per old 8086 and original cpu architecture provides the inta cycles to it (bypasses apic, apic seems off).

It does have such a mode. ;-) You just have not to ack a pending
interrupt -- if a request goes away, the INT output gets deasserted as well. We are super cautious though and we reprogram the 8259A into the AEOI mode to prevent a lockup in case INTA cycles escape to the 8259A (which is theoretically possible for a broken design of an i82489DX-based system). See the 8259A datasheet for details.

Quoted 3 lines

I certainly agree with Marceij's comments that mixed mode of having 8254 PIT routed via the 8259A was never meant to occur alongside ioapic handling of the other interrupts. It is very problematic not to mention confusing.

Well, the true "mixed mode", i.e. where certain interrupts are delivered
as I/O APIC (either LoPri or Fixed) interrupts and others are routed through an 8259A controller and delivered as ExtINTA interrupts was specifically designed to work since the i8248DX APIC. What wasn't designed but works by the properties of the 8259A PIC is the transparent "through-8259A" mode.

Maciej

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#8 lore raw

Just as an FYI, still going strong here with the old api and ioapic patches. 5d 20h now.

When the official 2.6.0 comes to Gentoo Linux I can try that with whatever patches people are finding stable for these nforce fixes.

Anyone had any luck in talking to ASUS re a BIOS update?

Craig

Quoted 42 lines · Maciej W. Rozycki
Quoted 8 lines · Ross Dickson

Here is where to find Intel's MP arch spec Maceij mentions. I had to find it recently wrt nforce2 issues

http://www.intel.com/design/pentium/datashts/24201606.pdf

Section 3.6.1 Apic Architecture is relevant
particularly
Section 3.6.2.2 Virtual Wire Mode

BTW, I have revision 1.1 as well in case anyone was interested in the
differences.

Quoted 8 lines

I would like to add a footnote to highlight a potential gotcha as I understand it.

To clarify, the xt pic 8259A does not in itself have a transparent mode as would a logic buffer or inverter. It always needs inta cycles to function. In PIC mode it is wired to processor pins as per old 8086 and original cpu architecture provides the inta cycles to it (bypasses apic, apic seems off).

It does have such a mode. ;-) You just have not to ack a pending
interrupt -- if a request goes away, the INT output gets deasserted as well. We are super cautious though and we reprogram the 8259A into the AEOI mode to prevent a lockup in case INTA cycles escape to the 8259A (which is theoretically possible for a broken design of an i82489DX-based system). See the 8259A datasheet for details.

Quoted 3 lines

I certainly agree with Marceij's comments that mixed mode of having 8254 PIT routed via the 8259A was never meant to occur alongside ioapic handling of the other interrupts. It is very problematic not to mention confusing.

Well, the true "mixed mode", i.e. where certain interrupts are delivered
as I/O APIC (either LoPri or Fixed) interrupts and others are routed through an 8259A controller and delivered as ExtINTA interrupts was specifically designed to work since the i8248DX APIC. What wasn't designed but works by the properties of the 8259A PIC is the transparent "through-8259A" mode.

Maciej

#9 lore raw
Quoted 10 lines · Craig Bradney

Just as an FYI, still going strong here with the old api and ioapic patches. 5d 20h now.

When the official 2.6.0 comes to Gentoo Linux I can try that with whatever patches people are finding stable for these nforce fixes.

Anyone had any luck in talking to ASUS re a BIOS update?

Craig

I have not talked to ASUS. I note from peoples postings that with the latest award bios we may need no apic patches (C1 disconnect auto), just an ioapic one to work round a buggy bios. I don't think you can run nmi_watchdog=1 with the old io-apic (not of my doing) patch.

I have pheonix bios MOBOS from albatron and epox so award bios doesn't help me. No disconnect options available in setup.
My apic ack delay patch lets the bios have its disconnect on and keep the cpu a few degrees cooler besides whatever else it and the nforce2 chipset might want to control it for.

I have been advised my query wrt my apic ack delay patch is progressing with AMD but I have nothing technical to report on it.

I have made and am trialling, but have not yet posted a kernel arg controlled version combining my v1 and v2 apic ack delay patches. This would be better than what I have released in the past because people can fix bioses as the fixes become available and use timer ack delay in the mean time. Of course there is still athcool and the earlier disconnect patch to force things if desired.

Regards
Ross.

#10 lore raw
Quoted 33 lines · Ross Dickson
Quoted 10 lines · Craig Bradney

Just as an FYI, still going strong here with the old api and ioapic patches. 5d 20h now.

When the official 2.6.0 comes to Gentoo Linux I can try that with whatever patches people are finding stable for these nforce fixes.

Anyone had any luck in talking to ASUS re a BIOS update?

Craig

I have not talked to ASUS. I note from peoples postings that with the latest award bios we may need no apic patches (C1 disconnect auto), just an ioapic one to work round a buggy bios. I don't think you can run nmi_watchdog=1 with the old io-apic (not of my doing) patch.

I have pheonix bios MOBOS from albatron and epox so award bios doesn't help me. No disconnect options available in setup.
My apic ack delay patch lets the bios have its disconnect on and keep the cpu a few degrees cooler besides whatever else it and the nforce2 chipset might want to control it for.

I have been advised my query wrt my apic ack delay patch is progressing with AMD but I have nothing technical to report on it.

I have made and am trialling, but have not yet posted a kernel arg controlled version combining my v1 and v2 apic ack delay patches. This would be better than what I have released in the past because people can fix bioses as the fixes become available and use timer ack delay in the mean time. Of course there is still athcool and the earlier disconnect patch to force things if desired.

Quoted 2 lines

Regards
Ross.

Ok Ross. Well, Gentoo's 2.6 is out now so whenever you want me to test your new patch I can try it. Ive been looking back through the list for the updated patches but things seemed to have changed here and there even for the v2 patches so I think I'll wait for the next round of patchesas things seem a little confusing.

2.6test11 is still running happily.. 6d15h now.

Craig

#11 lore raw
In reply to Maciej W. Rozycki
Quoted 5 lines · Maciej W. Rozycki
Quoted 3 lines · Ross Dickson

Here is where to find Intel's MP arch spec Maceij mentions. I had to find it recently wrt nforce2 issues

Quoted 8 lines
Quoted 5 lines

http://www.intel.com/design/pentium/datashts/24201606.pdf

Section 3.6.1 Apic Architecture is relevant
particularly
Section 3.6.2.2 Virtual Wire Mode

BTW, I have revision 1.1 as well in case anyone was interested in the
differences.

Yes please if it is forwardable or downloadable.

Quoted 10 lines
Quoted 9 lines

I would like to add a footnote to highlight a potential gotcha as I understand it.

To clarify, the xt pic 8259A does not in itself have a transparent mode as would a logic buffer or inverter. It always needs inta cycles to function. In PIC mode it is wired to processor pins as per old 8086 and original cpu architecture provides the inta cycles to it (bypasses apic, apic seems off).

Quoted 7 lines

It does have such a mode. ;-) You just have not to ack a pending
interrupt -- if a request goes away, the INT output gets deasserted as well. We are super cautious though and we reprogram the 8259A into the AEOI mode to prevent a lockup in case INTA cycles escape to the 8259A (which is theoretically possible for a broken design of an i82489DX-based system). See the 8259A datasheet for details.

I believe what you have written because you say it is how the code works.

I take it you mean that the INT is either never latched? or only latched with IS bit after receipt of first INTA?

It is not obvious in 8259A Datasheet published in Intel Microsystem Components Handbook Volume 1 1983 nor in datasheet December 1988.

I note the data sheet is almost silent on the topic of INT behaviour without INTA cycle.

Almost, as the WAVEFORMS diagrams which have INT displayed only show it in conjunction with the INTA and under said diagram I read

NOTES: Interrupt output must remain HIGH at least until leading edge of first INTA.
implying it can go low for some reason?

And the,
1. Cycle 1 in iAPX 86 , ....
Only indicates its trailing edge is synchrouous with the machine cycle.

Figure 10 in the data sheet does not help either as when the IR goes low it has a LATCH* ARMED notation which I took to mean the INT output was then latched.

I now think it was referring to the transparent D type latch "REQUEST LATCH" in the priority cell diagram but I cannot see a footnote to the *.

Could you please point me to the document where it is made clear? It may be in the i82489DX docs as I do not have them or in a later 8259A data sheet revision?

Thanks,
Ross.

Quoted 10 lines
Quoted 3 lines

I certainly agree with Marceij's comments that mixed mode of having 8254 PIT routed via the 8259A was never meant to occur alongside ioapic handling of the other interrupts. It is very problematic not to mention confusing.

Well, the true "mixed mode", i.e. where certain interrupts are delivered
as I/O APIC (either LoPri or Fixed) interrupts and others are routed through an 8259A controller and delivered as ExtINTA interrupts was specifically designed to work since the i8248DX APIC. What wasn't designed but works by the properties of the 8259A PIC is the transparent "through-8259A" mode.

Clarified thanks.

Quoted 10 lines

Maciej

(quoted diff) +3 -0
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#12 lore raw
Quoted 8 lines · Ross Dickson
Quoted 6 lines

It does have such a mode. ;-) You just have not to ack a pending
interrupt -- if a request goes away, the INT output gets deasserted as well. We are super cautious though and we reprogram the 8259A into the AEOI mode to prevent a lockup in case INTA cycles escape to the 8259A (which is theoretically possible for a broken design of an i82489DX-based system). See the 8259A datasheet for details.

I believe what you have written because you say it is how the code works.

Well, since I'm actually the author of the relevant bits (though Ingo did
some clean-ups before applying them), I must have been completely sure the assumptions are valid.

Quoted 2 lines

I take it you mean that the INT is either never latched? or only latched with IS bit after receipt of first INTA?

Yes, one of these conditions is true, although I've never bothered to
investigate exactly which one. ;-)

Quoted 2 lines

It is not obvious in 8259A Datasheet published in Intel Microsystem Components Handbook Volume 1 1983 nor in datasheet December 1988.

Yep, the datasheet is indeed not that clear on the matter. The latest
version (version 3, dated Nov 1988) used to be available at the Intel's FTP site, but I can't find it anymore.

The 8259A core is documented in many other datasheets, perhaps more
clearly -- e.g. I've found at least one Intel datasheet providing an unambiguos explanation of how the SFNM mode works. I knew of the volatile property of the INT output pretty always and it can be quite easily verified with hardware. Given this property some people find the way Intel defines edge-triggered interrupts quite surprising.

Quoted 3 lines

Could you please point me to the document where it is made clear? It may be in the i82489DX docs as I do not have them or in a later 8259A data sheet revision?

Well, there is actually a hint on how this "transparent" property of the
8259A PIC can be used for delivery of EISA chaining interrupts as APIC interrupts in the i82489DX datasheet. The problem with these interrupts appears with the 82357 ISP EISA component that has a pair of 8259A PICs embedded and does not provide the interrupt line externally, only wiring it to IRQ 13 (IR5 of the slave 8259A -- so both 8259A cores have to be treated as "transparent"!) internally. The same problem exists with the 8254 interrupt in this chip, but the datasheet disregards it, assuming the local APIC timer will be used for periodic interrupts exclusively.

Linux would use IRQ 0 in the "transparent" 8259A mode with this chip and
if that failed (which would be quite possible, since an ISP erratum required glue logic in the 8259A path when used with an APIC and the Intel's suggestion wasn't the most fortunate) the mixed mode with ExtINTA interrupts would be configured. Of course the mixed mode would also permit simultanous use of IRQ 0 and IRQ 13 with ISP -- with the "transparent" 8259A mode can support only a single interrupt source.

Note the interesting internal inconsistency of the document --
implementation of the erratum workaround as proposed by Intel would make the suggested "transparent" 8259A mode inoperational. ;-)

Maciej

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +