Skip to contentSkip to conversation

Fix kernel lockup in RTL-8169 gigabit ethernet driver

PATCH Manfred Spraul 4 participants 7 messages Apr 3, 2004 – Apr 6, 2004

lore
Share & tools
lore
#0 lore raw
Quoted 12 lines · Andrew

The logic is faulty, or at least very odd.

	tx_left = tp->cur_tx - dirty_tx;

	while (tx_left > 0) {
		int entry = dirty_tx % NUM_TX_DESC;

		if (!(le32_to_cpu(tp->TxDescArray[entry].status) & OWNbit)) {
			...
		}
	}
Quoted 5 lines

Why is that `if' test there at all? If it ever returns false, the box locks up. A BUG_ON(le32_to_cpu(tp->TxDescArray[entry].status) & OWNbit) might make more sense.

tx_left counts packets submitted by hard_xmit_start to the hardware. Initially OWNbit is set, the packet is owned by the nic. The OWNbit is cleared by the hardware after the packet was sent. A packet with OWNbit set means that the nic didn't send it yet to the wire. I think the "else break;" patch is correct, but someone with docs should confirm that.

Adam: did you see deadlocks that disappeared after applying your patch? It shouldn't deadlock - it should loop until the nic sends the packet to the wire. It might take a few msecs, but then it should continue. Perhaps gcc optimized away the reload from memory and loops on a register. Or there is another bug that is hidden by your patch.

--
Manfred

#1 lore raw

Manfred Spraul <[email protected]> :
[...]

Quoted 5 lines

tx_left counts packets submitted by hard_xmit_start to the hardware. Initially OWNbit is set, the packet is owned by the nic. The OWNbit is cleared by the hardware after the packet was sent. A packet with OWNbit set means that the nic didn't send it yet to the wire. I think the "else break;" patch is correct, but someone with docs should confirm that.

Realtek Gigabit Ethernet Media Access Controller with power management R8169 Rev.1.21, p.54
[...]
Ownership: This bit, when set, indicates that the descriptor is owned by the NIC. When cleared, it indicates that the descriptor is owned by the host system. NIC clears this bit when the relative buffer data is already transmitted. In this case, OWN=0.

[...]

Quoted 1 line

Perhaps gcc optimized away the reload from memory and loops on a

Point taken.

--
Ueimor

#2 lore raw
In reply to Manfred Spraul
Quoted 5 lines

Adam: did you see deadlocks that disappeared after applying your patch? It shouldn't deadlock - it should loop until the nic sends the packet to the wire. It might take a few msecs, but then it should continue. Perhaps gcc optimized away the reload from memory and loops on a register. Or there is another bug that is hidden by your patch.

Yes, it used to deadlock within a second after starting a large transfer across the network (e.g. copying a 1GB+ file over NFS) however smaller transfers (e.g. background gkrellmd traffic, web browsing, etc.) was less likely to cause problems (I could go for a few hours before it would deadlock.)

I initially tried to move the counters outside the loop, as I thought it was just the one entry in the array causing the problem, however this slowed network traffic down to approx 5kB/sec. Upon looking at the RTL8139 code it looked like "else break" was the correct action, and this brought network traffic back up to full speed and it's now been 4.5 days since I booted the kernel with the patch and it's all working perfectly.

When it did deadlock it was more or less permanent, as any programs accessing the NIC (or indeed the hard drive) would immediately deadlock
- so no program could send network data, thus it would loop forever waiting for more traffic.

I did add a 'printk' line in to see what the variables were just to make sure this was the location of the bug, and as I expected none of the values changed at all.

Cheers,
Adam.

#3 lore raw

Malvineous <[email protected]> :
[...]

Quoted 3 lines

I did add a 'printk' line in to see what the variables were just to make sure this was the location of the bug, and as I expected none of the values changed at all.

Can you send the unpatched r8169.o module ?

Thanks in advance.

--
Ueimor

#4 lore raw
Quoted 1 line

Can you send the unpatched r8169.o module ?

Here is the compiled source file (r8169.c -> r8169.o) from 2.6.5-rc1 (which had the same problem and has an identical .c file) but I'm not sure if it's different to the final module so please let me know if you wanted something else:

http://members.optushome.com.au/a.nielsen/r8169.o.bz2 (66 kB)

Cheers,
Adam.

$ gcc -v
Reading specs from /usr/lib/gcc-lib/i486-slackware-linux/3.2.3/specs
Configured with: ../gcc-3.2.3/configure --prefix=/usr --enable-shared
--enable-threads=posix --enable-__cxa_atexit --disable-checking
--with-gnu-ld --verbose --target=i486-slackware-linux
--host=i486-slackware-linux
Thread model: posix
gcc version 3.2.3
#5 lore raw
Quoted 4 lines · Adam Nielsen
Quoted 3 lines

Can you send the unpatched r8169.o module ?

Quoted 8 lines

Here is the compiled source file (r8169.c -> r8169.o) from 2.6.5-rc1 (which had the same problem and has an identical .c file) but I'm not sure if it's different to the final module so please let me know if you wanted something else:

http://members.optushome.com.au/a.nielsen/r8169.o.bz2 (66 kB)

Thanks. The code reloads the tx ring value from memory, thus I don't understand why it deadlocks.

--
Manfred

#6 lore raw

Manfred Spraul <[email protected]> :
[...]

Quoted 2 lines

Thanks. The code reloads the tx ring value from memory, thus I don't understand why it deadlocks.

Well...
- rtl8169_interrupt() acks all events before rtl8169_tx_interrupt() is called
- the count of descriptors handled in rtl8169_tx_interrupt() is only limited by the number of packets submitted for TX at the time rtl8169_tx_interrupt() is called

-> if there is a stream of Tx events, it is possible that Tx descriptors are
processed before the relevant event is notified to the host by the network adapter.

--
Ueimor