ehci-hcd on CARDBUS hangs when stopping card service

I've recently got some similar reports, but with a few less facts ... it was clear to me there was a problem somewhere in the CardBus code, since I know that cleanup via rmmod is working fine, and in fact that's the workaround one person had ended up with!

And the OHCI driver hits a related problem too ...

Yes, absolutely. It's turning a "clean shutdown" scenario into a "dirty shutdown" ... a normal "rmmod" works, correctly, and from the perspective of a device driver (if not the CardBus code) those should be exactly the same: two ways to start the same driver shutdown.

That current sequence (powerdown before pci_dev->remove) violates the device tree sequencing requirement ... which I recall was one of the key features of the original 2.4 CardBus support. Did it change rather recently, or has this bug really been lurking for a very long time? I'd expect to have heard about that OHCI problem (seemingly the same root cause) before, since there are folk using Cardbus OHCI (more using EHCI!), but nobody's reported it that I know of.

I'll hope that problem appears only in 2.4.18-6mdk, and isn't found in other kernels. In particular, if it's in 2.5.17 then there's a big hole in the "new driver model" work (struct device etc)!

Irrespectively,

I partially agree. For a clean shutdown, it's guaranteed not to be endless. For a dirty shutdown -- physically ejecting the card, or the hardware having truly nasty failure mode (one I've not seen but which could conceivably happen) -- it's a problem to fix.

Is there a clean way to detect the "card ejected before anything calls pci_dev->remove()" case? I don't really like the idea of wrapping code around every PCI register access to detect such cases.

- Dave

[I should have mentioned I am not on lkml; as it stands now I reply to thread off web archive]

It looks like discussion is focused on a problem how to detect a removed card. While this problem definitely exists - I'd like to stress that the original report was about a normal shutdown case. Just do init 0 with card plugged in and system hangs. IMHO that should be dealt with in the first place.

As for general case, I do not know hardware well enough. If as mentioned some architectures have a general problem accessing removed devices then CardBus driver should not call normal cleanup sequence in this case and just signal abort condition to a low level driver (so that any in-memory structures may be removed without an attempt to actually touch hardware).

Regards

-andrej

P.S. I would appreciate Cc to me. Thank you.

Well, there are two issues. That's a kind of "dirty shutdown" case, for which there's a clear need to update the EHCI driver. I'm working on a fix for it, presumably someone can help test the patch.

... and that's the "clean shutdown" case, where it seems like the CardBus code in the 2.4 kernels is clearly doing the wrong thing:
an operation sequence turns those into a dirty shutdown. (Which then trip over the EHCI "dirty shutdown" problem.)

It'd be nice if someone were to work on fixing that CardBus problem too, since it's likely it'll break other drivers. The fact that nobody has stepped in there is why the discussion went the direction it did ... :)

- Dave

Absolutely the same code is in 2.4.18 and 2.4.19-pre8 so it is not something Mandrake has introduced. The reason it has not been noticed before is probably that not every driver hangs here (in reported case ohci went through even though it could not access controller properly).

-andrej

In article <[email protected]>,

You don't have much choice with CardBus, I'm afraid.

Even if the user were to do the rmmod "before" yanking out the card, assuming that the rmmod took a bit of time and started the "remove()" call at the same time the card was actually removed, you'll end up in the same situation.

It's just a fact of life with any hot-plug thing that can be removed without software first freeing it.

On most (practically all?) machines, a device that no longer exists will return a nice floating 0xff for device reads, so it's usually reasonably simple to detect (0xff is often not a legal status register value for most devices for example).

Also, it's generally a good idea to "just say no" to endless loops in drivers. Hardware bugs _do_ happen, and it's a lot more pleasant to have the driver do a

	printk("Device does not respond\n");

than for the kernel to hang.

Linus

You get an interrupt _before_ the card goes away, because the pins are of different lengths.

As long as your driver API has some kind of abort() call to tell it that the device is no longer present, and you manage to call that within the few milliseconds between the card detect pin contact breaking and the rest of the pins breaking, you should be fine.

If you're sharing interrupts and have high interrupt latency, there may be a problem -- perhaps it would be better if in that case you could ensure that the socket IRQ handler gets run _before_ the device IRQ handler.

Too late. On some hardware, if you try to talk to the device once it's gone, you're already dead. Not all the world is a PeeCee.

--
dwmw2

That happens even on a "PeeCee".

Situation:
- Normal readl() of a ioremap()ed register set of a

     PCI device[1]:

-> machine hangs hard (no magic sysrq possible, no Ooops, no
panic printed)

- Several DWORD reads of the same register set succeded before and the register is wired according to the SPEC.

Just to provide an argument here

PS: I wish I had an PCI Analyzer available...

[1] TI TMS320C6x EVM Board[2], Rev. 3
with PCI-Bridge Chip AMCC S5933[3] hanging the host side on Busmaster DMA.

[2] PCI-ID: 0x104c:0x1002
[3] PCI-ID: 0x10e8:0x4750 or 0x10e8:0x807d or 0x10e8:0x809c

Science is what we can tell a computer. Art is everything else. --- D.E.Knuth

Seems to me it'd be worth mentioning this issue somewhere in the documentation or source. One could get the impression that the main issue for a CardBus-enabled PCI driver is to make sure that the "new style" driver APIs -- with a DEVICE_TABLE etc -- are used. (Maybe just a brief comment in <asm/io.h> ...)

I'm hardly averse to changing that loop (which normally does have an end :) and I expected to need to at some point. It's interesting to me just how long that has been there without causing problems. In this case the root cause is that Cardbus "improper shutdown sequence" problem, so "no end" is just a particularly nasty secondary failure mode.

- Dave

Documentation might be a good thing, indeed. I doubt <asm/io.h> is the right place for it, people tend to look at other drivers etc to pattern after (as clearly showed by how often a bug in one place is replicated in lots of other places ;)

We've actually fixed a number of these things - there are drivers that notice removal on their own silently, and just turn it into a no-op.

Most people don't unplug their devices while they are in use, and on cardbus the most common case ends up (I think) being the fact that the cardbus PCI static interrupt itself is shared with the device interrupt. So what happens is that when you remove the CardBus card, that causes the cardbus controller to send an interrupt (for removal), but since that interrupt is shared with the card driver, the card driver also sees the interrupt even if the card itself is otherwise idle.

This meant that some tulip cards at least were _guaranteed_ to lock up some time ago, simply because their interrupt handler would loop forever on seeing the "more work" bit continually (all bits in the status register were set due to the removal and floating data lines).

Linus