QUESTION: Network hangs with BP6 and 2.4.x kernels, hardware related?
A message that is not in the archive (sent off-list or not yet received): [email protected]
On Fri, 12 Jan 2001, Frank de Lange wrote:
[I've cut syslog junk away for clarity -- you could just do `dmesg -s 32768'.]
Quoted 2 lines
before network hang
===================
[...]
Quoted 1 line
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
[...]
Quoted 1 line
13 0FF 0F 0 1 0 1 0 1 1 99
[...]
Quoted 1 line
printing local APIC contents on CPU#0/0:
[...]
Quoted 2 lines
... APIC TMR field:
0123456789abcdef0123456789abcdef
[...]
Quoted 1 line
00000000010000000000000001000000
^
[...]
Quoted 1 line
printing local APIC contents on CPU#1/1:
[...]
Quoted 2 lines
... APIC TMR field:
0123456789abcdef0123456789abcdef
[...]
Quoted 1 line
00000000000000000000000001000000
^
[...]
Here everything is fine. Vector 0x99 (the one you are having troubles
with) is set up as level-triggered (Trig is 1) and the respective bits of the Trigger Mode Register (TMR) of the local APIC of both CPUs are set, i.e. the last 0x99 IRQ processed was level-triggered as expected. As a part of the ususal inter-APIC handshake for level-triggered interrupts an EOI message was sent to the originating I/O APIC (IRR is 0) to inform it, it's free to send the IRQ again if still asserted.
Quoted 2 lines
after network hang
==================
[...]
Quoted 1 line
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
[...]
Quoted 1 line
13 0FF 0F 0 1 1 1 0 1 1 99
[...]
Quoted 1 line
printing local APIC contents on CPU#1/1:
[...]
Quoted 2 lines
... APIC TMR field:
0123456789abcdef0123456789abcdef
[...]
Quoted 1 line
00000000000000000000000001000000
^
Quoted 1 line
printing local APIC contents on CPU#0/0:
[...]
Quoted 2 lines
... APIC TMR field:
0123456789abcdef0123456789abcdef
[...]
Quoted 1 line
00000000010000000000000000000000
^ Gotcha!
[...]
Here the last 0x99 IRQ delivered to CPU#1 was fine, just like before.
But the last 0x99 IRQ CPU#0 received was apparently delivered as edge-triggered -- the respective bits of the Trigger Mode Register (TMR) of the local APIC is cleared. Hence the local APIC decided no EOI message is needed for the originating I/O APIC as edge-triggered interrupts are always sent by an I/O APIC whenever arriving (it's not possible for level-triggered ones as an IRQ storm would result). Upon receiving an EOI command from Linux the local APIC decides everything is finished and the I/O APIC is left stuck with the IRR bit set to 1. It's still waiting for an EOI message to arrive for further 0x99 IRQs to send.
How could it happen? Well, I guess a transmission error could have
happened that remained unnoticed by the checksumming hardware. As the checksum algorithm is pretty trivial -- a cumulative sum of 2-bit values -- it might just have happened two bits got toggled. I believe such errors are happening due to marginal hardware -- not every i386 SMP box shows this problem, even if a high volume of level-triggered interrupts is observed.
Thank you very much for the log. I already have an idea how to
automatically recover from such a situation. No driver change is required -- apparently no driver is at fault, it's just a load of the inter-APIC bus, I/O APICs (including system bus accesses) or the whole system in general.
I'm hereby asking everyone not to modify drivers just to circumvent APIC
lock-ups. Especially if such changes would "punish" perfectly good systems. It won't cure anything -- it might only make it happen less frequently due to different conditions. I hope to have changes ready to test by the next week.
Maciej
Signature · 8 lines
+ Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: [email protected], PGP key available + - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/