e100 driver on ARM

commit 32c1459bb3814274b3c5e0c5ed4efc6c0aa89eb4

Author: Scott Feldman <[email protected]>
Date:   Wed Nov 9 02:18:52 2005 -0500

[netdrvr e100] experiment with doing RX in a similar manner to eepro100

I was going to say that eepro100's speedo_rx_link() does the same DMA abuse as e100, but then I noticed one little detail: eepro100 sets both EL (end of list) and S (suspend) bits in the RFD as it chains it to the RFD list. e100 was only setting the EL bit. Hmmm, that's interesting. That means that if HW reads a RFD with the S-bit set, it'll process that RFD and then suspend the receive unit. The receive unit will resume when SW clears the S-bit. There is no need for SW to restart the receive unit. Which means a lot of the receive unit state tracking code in the driver goes away.

So here's a patch against 2.6.14. (Sorry for inlining it; the mailer I'm using now will mess with the word wrap). I can't test this on XScale (unless someone has an e100 module for Gumstix :) . It should be doing exactly what eepro100 does with RFDs. I don't believe this change will introduce a performance hit because the S-bit and EL-bit go hand-in-hand meaning if we're going to suspend because of the S- bit, we're on the last resource anyway, so we'll have to wait for SW to replenish.

1) Does e100 driver work on ARM?

Quoted 1 line · Jeff Garzik

1) Does e100 driver work on ARM?

FWIW, e100 seems to work okay for me on an intel ixp2400 (xscale based) board, an ixp2850 (xscale based) board and an ixp2350 (xscale3 based) board. ixp2350 works both with hardware coherency turned on (cpu snoops bus) and turned off (manual dma cache clean/invalidate as usual.)

As for the other ARM platforms that I'm interested in / have hardware for / maintain, the at91/ep93xx/pxa270 don't have PCI, and the other two (iop32x/iop33x) I can't test because I don't have such systems with e100 NICs, but I expect those would work, since they're both xscale based like the ixp2400, and the ixp2400 works.

Quoted 9 lines · Lennert Buytenhek

Quoted 1 line · Jeff Garzik

1) Does e100 driver work on ARM?

FWIW, e100 seems to work okay for me on an intel ixp2400 (xscale based) board, an ixp2850 (xscale based) board and an ixp2350 (xscale3 based) board. ixp2350 works both with hardware coherency turned on (cpu snoops bus) and turned off (manual dma cache clean/invalidate as usual.)

Quoted 5 lines

As for the other ARM platforms that I'm interested in / have hardware for / maintain, the at91/ep93xx/pxa270 don't have PCI, and the other two (iop32x/iop33x) I can't test because I don't have such systems with e100 NICs, but I expect those would work, since they're both xscale based like the ixp2400, and the ixp2400 works.

I just got an iop342 board dropped on my lap. Once it's running, I'll make sure to make this the first thing to test.

I have a pxa255 based system with PCI added to it. The e100 would have memory corruption in its receive buffers detected by slab debugging unless I put in the patch to use the S-bit.

Here is a link to the patch posting:
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.20-rc3/2.6.20-rc3-mm1/broken-out/git-netdev-all.patch
Search for e100.c.

http://www-gatago.com/linux/kernel/15457063.html - This discussion seems to hit the issue.

There appears to be a race on the cache line where the EL bit and the next packet info live. In my case the hardware appeared to write to a free packet. The S-bit seems to make the hardware stop and spin on the bit, while the EL bit seems to let the hardware try to use that packet.

This race would occur less often when the receive buffer chain is always refilled before the hardware can use them up. On our 400 Mhz Xscale, we can use up all 256 buffers if the PCI bus has another busy device on it.
In our case it is an 802.11g miniPCI card and our software was routing
all ethernet packets to the wireless interface and vice versa while TCP streams were running accross these connections.

Which PCI host controller are you using with the PXA255? We tried using a PXA255 based system with a PCI controller a couple of years ago and have to change to a different cpu in the end due to the PCI controller simply not being valid PCI. The PXA255 wasn't designed for PCI, and I get the impression that non of the PCI companion chips for it do a good enough job to actually add it correctly.

Quoted 7 lines · Lennart Sorensen

Which PCI host controller are you using with the PXA255? We tried using a PXA255 based system with a PCI controller a couple of years ago and have to change to a different cpu in the end due to the PCI controller simply not being valid PCI. The PXA255 wasn't designed for PCI, and I get the impression that non of the PCI companion chips for it do a good enough job to actually add it correctly.

Sorry for the delay in responding...my wife and I just had twins!

We are using the IT8152G RISC-to-PCI companion chip.

Well the IT8152G+PXA255 combination used on the SBC we tried a couple of years ago did not work. The PCI bus had errors and the SBC maker gave up trying to fix it. We switched to a Geode SC1200 based board instead which works fine PCI wise.

Here is a quote from Russell that describes what I believe is the main problem:
http://www-gatago.com/linux/kernel/15457063.html
"
Has e100 actually been fixed to use the PCI DMA API correctly yet? Looking at it, it doesn't look like it, so until it does, eepro100 is the far better bet for platforms needing working DMA API.

What I'm talking about is e100's apparant belief that it can modify rfd's in the receive ring on a non-cache coherent architecture and expect the data around it to remain unaffected (see e100_rx_alloc_skb):

struct rfd {
u16 status;
u16 command;
u32 link;
u32 rbd;
u16 actual_size;
u16 size;
};

it touches command and link. This means that the whole rfd plus maybe the following or preceding 16 bytes get loaded into a cache line (assuming cache lines of 32 bytes), and that data written out again at sync. However, it does this on what seems to be an active receive chain.

So, both the CPU _and_ the device own the same data. Which is a violation of the DMA API.
"

I think that the S-bit patch fixes it because the hardware spins on the s-bit instead of using the packet. With just the el-bit, the hardware tries to use the same cache line that the software is updating.

Can someone from Intel let us know if I understand the hardware's handling of the S and EL bits? If my interpretation is correct, can the s-bit patch be applied? It seems like the correct way to lock out the hardware while a packet is being updated. I have not seen a reason given not to apply the patch.

This is all a while ago now, but wasn't the e100 S-bit patch originally written by Intel people in response to the very same quote by Russell King that you've quoted above?

Quoted 3 lines · Lennert Buytenhek

This is all a while ago now, but wasn't the e100 S-bit patch originally written by Intel people in response to the very same quote by Russell King that you've quoted above?

Correct.

I just wanted to make sure it didn't kill any boxes.

Quoted 7 lines · Jeff Garzik

Quoted 3 lines · Lennert Buytenhek

This is all a while ago now, but wasn't the e100 S-bit patch originally written by Intel people in response to the very same quote by Russell King that you've quoted above?

Correct.

Quoted 1 line

I just wanted to make sure it didn't kill any boxes.

Neither did we. On top of that we didn't have the equipment to test on ARM on until recently, and the system that I got recently will not even load the kernel with an e100 NIC in the PCI slot (way way way before e100.ko loads, e1000 works just fine). that doesn't help either.

Meanwhile we've not sat still and jesse wrote a patch to have e100 use IO optionally for plagued platforms which seems to fix some of these issues, and Jeff Kirsher has been actively tracking a e100 IPMI issue on a very specific platform.

Jeff, I think I should just push the IO patch and the sbit code to Andrew and have it sit there. That is a vastly larger test resource than we currently can generate for this. If needed we just let is sit there for a whole release cycle before moving it to #upstream.

Quoted 1 line · Auke

Jeff, I think I should just push the IO patch and the sbit code to Andrew and have it sit there. That is a vastly larger test resource than we currently can generate for this. If needed we just let is sit there for a whole release cycle before moving it to #upstream.

The sbit code has been in -mm via netdev-2.6.git#ALL for many months now. I am certainly open to adding any number of "for -mm only" style patches that you wish to pass along, for /any/ driver: e100, e1000, ixgb.

My suspicision (although it is only that) is that the PXA255 trying to access memory may cause interruptions in PCI bus master transfers, which is of course not permitted by the PCI spec (at least the way I read it).

Why wouldn't that be permitted? It, in fact, happens all the time (the host bridge withdraws the GNT# line and raises STOP#, which does a Termination With Data of the bus transfer.) This is a normal event and if you can't handle it you won't work with many host bridges at all.

The eepro100 removal has been blocked for almost a year by a vague suggestion from Russell that e100 doesn't work on ARM. But he doesn't have that machine anymore. So, we're stuck in limbo.

1) Does e100 driver work on ARM?