Skip to contentSkip to conversation

BUG when removing USB flash drive

DISCUSSION Alberto Bertogli 5 participants 12 messages May 14, 2004 – May 15, 2004 SCSI

lore
Share & tools
lore
#0 lore raw

Hi!

I've just hit a bug after removing a Kingston USB flash drive.

I have removed it several (probably more than 40) times today without problems until now. I'm removing it while doing streaming write()s on the device (using /dev/sda) because I'm testing some things with the drive. This was just like all the other times, so it looks like a race somewhere.

This is a stock 2.6.6 kernel, on a Pentium 4 with HT (the kernel is compiled with both SMP and preempt).

After the BUG the kernel doesn't detect the device anymore.

Please let me know if you need any more information or if I can give you a hand with testing.

Thanks a lot,
Alberto

kernel BUG at drivers/usb/storage/usb.c:848!
invalid operand: 0000 [#1]
PREEMPT SMP

CPU:    1
EIP:    0060:[<c030843c>]    Not tainted
EFLAGS: 00010202   (2.6.6)

EIP is at usb_stor_release_resources+0xa3/0xcd
eax: 00002832 ebx: c639e200 ecx: c04bd768 edx: debf2888
esi: c04c6780 edi: c308ac00 ebp: c308ac24 esp: dededeb0
ds: 007b es: 007b ss: 0068
Process khubd (pid: 10, threadinfo=dedec000 task=dedef770)

Stack: debf2888 debf2854 c02e9f48 c639e200 debf2854 de40a670 debf2864
c04c67a0
       c0272926 debf2864 debf288c debf2864 c308acd0 c0272a4d debf2864
debf28bc
       debf2864 c308acd0 c0271b06 debf2864 debf2864 c308ac00 c0271b57
debf2864
Call Trace:
 [<c02e9f48>] usb_unbind_interface+0x7a/0x7c
 [<c0272926>] device_release_driver+0x64/0x66
 [<c0272a4d>] bus_remove_device+0x56/0x98
 [<c0271b06>] device_del+0x5f/0x9d
 [<c0271b57>] device_unregister+0x13/0x23
 [<c02eff98>] usb_disable_device+0x71/0xac
 [<c02eaa32>] usb_disconnect+0x9c/0xeb
 [<c02ecb1d>] hub_port_connect_change+0x274/0x279
 [<c02ec509>] hub_port_status+0x45/0xb0
 [<c02ece0b>] hub_events+0x2e9/0x364
 [<c02eceb3>] hub_thread+0x2d/0xe8
 [<c0105f86>] ret_from_fork+0x6/0x14
 [<c011a3a1>] default_wake_function+0x0/0x12
 [<c02ece86>] hub_thread+0x0/0xe8
 [<c0104271>] kernel_thread_helper+0x5/0xb

Code: 0f 0b 50 03 05 a6 43 c0 e9 7c ff ff ff c7 80 e0 01 00 00 00

Five seconds later, this comes out:

<1>Unable to handle kernel paging request at virtual address 31a875c8 printing eip:
c0307b2b
*pde = 00000000

Oops: 0002 [#2]
PREEMPT SMP
CPU:    0
EIP:    0060:[<c0307b2b>]    Not tainted
EFLAGS: 00010002   (2.6.6)

EIP is at usb_stor_control_thread+0x14d/0x2c9
eax: 31a875c8 ebx: c639e200 ecx: c639e200 edx: 00002003
esi: c639e600 edi: d3c84000 ebp: d3c84000 esp: d3c85f9c
ds: 007b es: 007b ss: 0068
Process usb-storage (pid: 10290, threadinfo=d3c84000 task=c76b0130)

Stack: dec00800 c639e200 c76b0698 c639e310 c639e324 c76b0130 c13e5ca0
dededda8
       c0105f86 dedef770 c03079de 00000000 c639e200 00000000 00000000
00000000
       00000000 c03079de 00000000 00000000 00000000 c0104271 c639e200
00000000
Call Trace:
 [<c0105f86>] ret_from_fork+0x6/0x14
 [<c03079de>] usb_stor_control_thread+0x0/0x2c9
 [<c03079de>] usb_stor_control_thread+0x0/0x2c9
 [<c0104271>] kernel_thread_helper+0x5/0xb

Code: f0 fe 08 0f 88 38 0b 00 00 8b 83 b8 00 00 00 81 b8 50 01 00
 <6>note: usb-storage[10290] exited with preempt_count 1
#1 lore raw
Quoted 15 lines · Alberto Bertogli

Hi!

I've just hit a bug after removing a Kingston USB flash drive.

I have removed it several (probably more than 40) times today without problems until now. I'm removing it while doing streaming write()s on the device (using /dev/sda) because I'm testing some things with the drive. This was just like all the other times, so it looks like a race somewhere.

This is a stock 2.6.6 kernel, on a Pentium 4 with HT (the kernel is compiled with both SMP and preempt).

After the BUG the kernel doesn't detect the device anymore.

Quoted 8 lines

Please let me know if you need any more information or if I can give you a hand with testing.

Thanks a lot,
Alberto

kernel BUG at drivers/usb/storage/usb.c:848!

This BUG happened because the SCSI layer was still using the drive after usb-storage had called scsi_remove_host().

This is an aspect of SCSI midlayer <-> low-level driver interaction that I'm not very clear about. How long after the host is removed can the midlayer continue to use it? How does the driver know when the midlayer is finished using the host so the driver can exit (and unload from memory)?

Alan Stern

#2 lore raw
Quoted 2 lines · Alan Stern [[email protected]] wrote:

This BUG happened because the SCSI layer was still using the drive after usb-storage had called scsi_remove_host().

In this case the scsi_remove_host is being called in a unexpected disconnect case. Any IOs in flight will be canceled in the mid-layer (i.e., they will not be chased down with calls to eh_abort_handler).

Quoted 6 lines

This is an aspect of SCSI midlayer <-> low-level driver interaction that I'm not very clear about. How long after the host is removed can the midlayer continue to use it? How does the driver know when the midlayer is finished using the host so the driver can exit (and unload from memory)?

The LLDD queuecommand should not be called after the return of scsi_remove_host (unless we have a bug). The LLDD should not free its resources until the release function is called on the struct device passed in during the scsi_add_host call. This release function could be called post the scsi_host_dev_release function being called as it does a put on this struct device. This chain of release calls is undefined as a hotplug goes out on the remove calls, but if user space never umounts / closes the device things will not fully cleanup.

-andmike
--
Michael Anderson
[email protected]

#3 lore raw
Quoted 4 lines · Mike Anderson

The LLDD queuecommand should not be called after the return of scsi_remove_host (unless we have a bug). The LLDD should not free its resources until the release function is called on the struct device passed in during the scsi_add_host call.

Ah. We have a problem here, because usb-storage doesn't own that struct device. It is owned by the USB core, and usb-storage has no way to know when its release function has been called.

Even worse, the release function may never be called at all. The device itself may still exist and be attached to the system even though the usb-storage driver has been unbound from it.

There must be many other LLDDs with the same problem. Any driver that manages a PCI host controller, for example. The struct device it receives from the PCI layer isn't owned by the LLDD, yet that's what it must pass to scsi_add_host().

How should we handle this?

Alan Stern

#4 lore raw
Quoted 15 lines · Alan Stern [[email protected]] wrote:
Quoted 4 lines · Mike Anderson

The LLDD queuecommand should not be called after the return of scsi_remove_host (unless we have a bug). The LLDD should not free its resources until the release function is called on the struct device passed in during the scsi_add_host call.

Ah. We have a problem here, because usb-storage doesn't own that struct device. It is owned by the USB core, and usb-storage has no way to know when its release function has been called.

Even worse, the release function may never be called at all. The device itself may still exist and be attached to the system even though the usb-storage driver has been unbound from it.

Quoted 7 lines

There must be many other LLDDs with the same problem. Any driver that manages a PCI host controller, for example. The struct device it receives from the PCI layer isn't owned by the LLDD, yet that's what it must pass to scsi_add_host().

How should we handle this?

I was incorrect in my previous statement. Well we may hold a reference count on the struct device passed into the scsi_add_host until all references on the host instance are gone and there maybe a positive ref count on the hostt module until all devices are closed a instance of LLDD should be able to do cleanup post return of scsi_remove_host. This needs to be true or SCSI mid-layer would not honor the requirement for the pci remove call.

I believe in the past you or someone else has mentioned that post the return of scsi_remove_host a Scsi_Host instance still has a hostt and a transportt which may be dangerous if all code paths are not correctly handling a host in the SHOST_DEL state.

I guess we still have this current issue on what is happening in the usb storage that it believes SCSI mid is still using the device. Is something not cleaning up, or is something in SCSI mid not honoring the SHOST_DEL state.

-andmike
--
Michael Anderson
[email protected]

#5 lore raw
Quoted 8 lines · Mike Anderson

I was incorrect in my previous statement. Well we may hold a reference count on the struct device passed into the scsi_add_host until all references on the host instance are gone and there maybe a positive ref count on the hostt module until all devices are closed a instance of LLDD should be able to do cleanup post return of scsi_remove_host. This needs to be true or SCSI mid-layer would not honor the requirement for the pci remove call.

Quoted 4 lines

I believe in the past you or someone else has mentioned that post the return of scsi_remove_host a Scsi_Host instance still has a hostt and a transportt which may be dangerous if all code paths are not correctly handling a host in the SHOST_DEL state.

Yes, that was me.

Quoted 4 lines

I guess we still have this current issue on what is happening in the usb storage that it believes SCSI mid is still using the device. Is something not cleaning up, or is something in SCSI mid not honoring the SHOST_DEL state.

usb-storage checks the state of its kernel thread after calling scsi_remove_host(), while preparing to kill the thread. This can only be done while the thread is idle, not processing any queued commands or error handler requests. The BUG in Alberto's log was triggered because the thread was not idle. Unfortunately the log doesn't say whether it was executing a queued command or an error handler request at the time.

It does appear to have been some sort of race, as Alberto mentioned in his original message:

http://marc.theaimsgroup.com/?l=linux-usb-devel&m=108449593410567&w=2

It might not be easy to find out where the race is occurring.

Alan Stern

#6 lore raw
Quoted 6 lines · Alan Stern

usb-storage checks the state of its kernel thread after calling scsi_remove_host(), while preparing to kill the thread. This can only be done while the thread is idle, not processing any queued commands or error handler requests. The BUG in Alberto's log was triggered because the thread was not idle. Unfortunately the log doesn't say whether it was executing a queued command or an error handler request at the time.

The idea behind scsi_remove_host() was that it was going to be called from the module_exit() routine. This means that the reference count on the LLD has already dropped to zero and it's safe for removal.

If you call scsi_remove_host() outside of the module_exit() code then what it does is remove all hosts from visibility (and makes it impossible to open or do anything with them), but it cannot revoke outstanding references (which may continue to use the device). The mid-layer takes the devices through CANCEL and then DEL. When scsi_remove_host() returns, you are guaranteed (barring any state model cockups) that the devices on the LLD with outstanding references are all in the DEL state. It will also block the device from above (no commands will be sent down). Any currently outstanding commands will be errored from the mid-layer; however, it will do nothing about outstanding commands in the LLD, you have to deal with them yourself.

The way this is supposed to work is that when you get a disconnection, you start refusing commands (and you also discard any active ones). Then you call scsi_remove_host(). We discard all current commands

So, if the usb storage LLD isn't discarding the commands, then they may be active.

James

#7 lore raw
Quoted 16 lines · James Bottomley

The idea behind scsi_remove_host() was that it was going to be called from the module_exit() routine. This means that the reference count on the LLD has already dropped to zero and it's safe for removal.

If you call scsi_remove_host() outside of the module_exit() code then what it does is remove all hosts from visibility (and makes it impossible to open or do anything with them), but it cannot revoke outstanding references (which may continue to use the device). The mid-layer takes the devices through CANCEL and then DEL. When scsi_remove_host() returns, you are guaranteed (barring any state model cockups) that the devices on the LLD with outstanding references are all in the DEL state. It will also block the device from above (no commands will be sent down). Any currently outstanding commands will be errored from the mid-layer; however, it will do nothing about outstanding commands in the LLD, you have to deal with them yourself.

Quoted 6 lines

The way this is supposed to work is that when you get a disconnection, you start refusing commands (and you also discard any active ones). Then you call scsi_remove_host(). We discard all current commands

So, if the usb storage LLD isn't discarding the commands, then they may be active.

Yes, that would explain it. We do arrange to abort the current command, but this is done before calling scsi_remove_host() when it should be done after. And we don't wait for the kernel thread to become idle afterwards.

I'll work on a patch. Testing it won't be easy, though; it's hard to prove that a race _isn't_ happening!

Alan Stern

#8 lore raw
Quoted 2 lines · Alan Stern

I'll work on a patch. Testing it won't be easy, though; it's hard to prove that a race _isn't_ happening!

I'm going to have to repeat this tests a few more times (in fact, I'll boot -mm2 in a couple of minutes to see if I can reproduce the bug there), so if you want me to try some patch let me know. As you say, I can't _prove_ it, but at least I can confirm I can't hit it =)

Thanks,
Alberto

#9 lore raw

I took a look at scsi_host_cancel(), scsi_device_cancel(), and a few other routines. Three things stood out:

When a host is removed, currently executing requests are aborted within the midlevel but the low-level driver is not informed of this. Presumably it shouldn't care since it ought to cancel its own outstanding I/O?

There's a race between scsi_host_cancel()/scsi_device_cancel() and scsi_times_out(). If the timer fires before scsi_device_cancel() can delete it, and scsi_host_cancel() tests the SHOST_RECOVERY bit before the error handler manages to set it, then you will remove a host that's still in error recovery.

There's no protection against accessing the host template via procfs, if a user process pins the file during scsi_remove_host(). You probably are already aware of this.

Alan Stern

#10 lore raw
In reply to Alberto Bertogli
Quoted 3 lines · Alberto Bertogli

This is a stock 2.6.6 kernel, on a Pentium 4 with HT (the kernel is compiled with both SMP and preempt).

Can you test the latest -mm tree and see if it fixes this problem for you?

thanks,

greg k-h

#11 lore raw
Quoted 7 lines · Greg KH
Quoted 3 lines · Alberto Bertogli

This is a stock 2.6.6 kernel, on a Pentium 4 with HT (the kernel is compiled with both SMP and preempt).

Can you test the latest -mm tree and see if it fixes this problem for you?

Sure. I've just tested -mm2; sorry it took so long but I had a couple of other things going on.

Sadly, it locks up hard (no sysrq or anything) after booting, right in the login prompt. Sometimes I got as far as typing the username and pressing enter, others just frozen without even being able to press a key.

So I removed the SMT scheduler because I thought it might be related, and it didn't even got to boot at all, it seem to freeze somewhere after the initial load, but I can't see anything because the screen goes black (even after I disabled framebuffer, SMP and preempt just in case).

I'll see if I can get a working config tomorrow morning so I can get to do this tests (and report that lockup too).

Thanks a lot,
Alberto