Fix kernel lockup in RTL-8169 gigabit ethernet driver
Quoted 12 lines · Andrew
The logic is faulty, or at least very odd.
tx_left = tp->cur_tx - dirty_tx; while (tx_left > 0) { int entry = dirty_tx % NUM_TX_DESC; if (!(le32_to_cpu(tp->TxDescArray[entry].status) & OWNbit)) { ... } }
Quoted 5 lines
Why is that `if' test there at all? If it ever returns false, the box locks up. A
BUG_ON(le32_to_cpu(tp->TxDescArray[entry].status) & OWNbit)might make more sense.
tx_left counts packets submitted by hard_xmit_start to the hardware. Initially OWNbit is set, the packet is owned by the nic. The OWNbit is cleared by the hardware after the packet was sent. A packet with OWNbit set means that the nic didn't send it yet to the wire. I think the "else break;" patch is correct, but someone with docs should confirm that.
Adam: did you see deadlocks that disappeared after applying your patch? It shouldn't deadlock - it should loop until the nic sends the packet to the wire. It might take a few msecs, but then it should continue. Perhaps gcc optimized away the reload from memory and loops on a register. Or there is another bug that is hidden by your patch.
--
Manfred
Manfred Spraul <[email protected]> :
[...]
Quoted 5 lines
tx_leftcounts packets submitted byhard_xmit_startto the hardware. Initially OWNbit is set, the packet is owned by the nic. The OWNbit is cleared by the hardware after the packet was sent. A packet with OWNbit set means that the nic didn't send it yet to the wire. I think the "else break;" patch is correct, but someone with docs should confirm that.
Realtek Gigabit Ethernet Media Access Controller with power management R8169 Rev.1.21, p.54
[...]
Ownership: This bit, when set, indicates that the descriptor is owned by the NIC. When cleared, it indicates that the descriptor is owned by the host system. NIC clears this bit when the relative buffer data is already transmitted. In this case, OWN=0.
[...]
Quoted 1 line
Perhaps gcc optimized away the reload from memory and loops on a
Point taken.
--
Ueimor
Quoted 5 lines
Adam: did you see deadlocks that disappeared after applying your patch? It shouldn't deadlock - it should loop until the nic sends the packet to the wire. It might take a few msecs, but then it should continue. Perhaps gcc optimized away the reload from memory and loops on a register. Or there is another bug that is hidden by your patch.
Yes, it used to deadlock within a second after starting a large transfer across the network (e.g. copying a 1GB+ file over NFS) however smaller transfers (e.g. background gkrellmd traffic, web browsing, etc.) was less likely to cause problems (I could go for a few hours before it would deadlock.)
I initially tried to move the counters outside the loop, as I thought it was just the one entry in the array causing the problem, however this slowed network traffic down to approx 5kB/sec. Upon looking at the RTL8139 code it looked like "else break" was the correct action, and this brought network traffic back up to full speed and it's now been 4.5 days since I booted the kernel with the patch and it's all working perfectly.
When it did deadlock it was more or less permanent, as any programs accessing the NIC (or indeed the hard drive) would immediately deadlock
- so no program could send network data, thus it would loop forever waiting for more traffic.
I did add a 'printk' line in to see what the variables were just to make sure this was the location of the bug, and as I expected none of the values changed at all.
Cheers,
Adam.
Malvineous <[email protected]> :
[...]
Quoted 3 lines
I did add a 'printk' line in to see what the variables were just to make sure this was the location of the bug, and as I expected none of the values changed at all.
Can you send the unpatched r8169.o module ?
Thanks in advance.
--
Ueimor
Quoted 1 line
Can you send the unpatched r8169.o module ?
Here is the compiled source file (r8169.c -> r8169.o) from 2.6.5-rc1 (which had the same problem and has an identical .c file) but I'm not sure if it's different to the final module so please let me know if you wanted something else:
http://members.optushome.com.au/a.nielsen/r8169.o.bz2 (66 kB)
Cheers,
Adam.
$ gcc -v Reading specs from /usr/lib/gcc-lib/i486-slackware-linux/3.2.3/specs Configured with: ../gcc-3.2.3/configure --prefix=/usr --enable-shared --enable-threads=posix --enable-__cxa_atexit --disable-checking --with-gnu-ld --verbose --target=i486-slackware-linux --host=i486-slackware-linux Thread model: posix gcc version 3.2.3
Quoted 4 lines · Adam Nielsen
Quoted 3 lines
Can you send the unpatched r8169.o module ?
Quoted 8 lines
Here is the compiled source file (
r8169.c-> r8169.o) from 2.6.5-rc1 (which had the same problem and has an identical .c file) but I'm not sure if it's different to the final module so please let me know if you wanted something else:http://members.optushome.com.au/a.nielsen/r8169.o.bz2 (66 kB)
Thanks. The code reloads the tx ring value from memory, thus I don't understand why it deadlocks.
--
Manfred
Manfred Spraul <[email protected]> :
[...]
Quoted 2 lines
Thanks. The code reloads the tx ring value from memory, thus I don't understand why it deadlocks.
Well...
- rtl8169_interrupt() acks all events before rtl8169_tx_interrupt() is called
- the count of descriptors handled in rtl8169_tx_interrupt() is only limited by the number of packets submitted for TX at the time rtl8169_tx_interrupt() is called
-> if there is a stream of Tx events, it is possible that Tx descriptors are
processed before the relevant event is notified to the host by the network adapter.
--
Ueimor