USB changes for 2.5.58

Quoted 2 lines · Matthew Dharm

Hrm... doesn't this all fall to pot when the device is unplugged and repluged?

Um, how? This seems to work for me, but I don't have a lot of devices here. And if there is a problem, you might want to tell the scsi people, as they are the ones advocating this call be added.

Well, we only create the host when the device is first attached. After that, if it goes away and comes back, we re-connect it to the old SCSI host.

But, while the device is gone, you've created an association between a SCSI node that exists and a non-existant USB device. Basically, you've got a pointer that is no longer valid. And when the device is re-attached, there isn't code to re-establish the correct SCSI<->USB association.

Something like this would proabably make sense if the hot-unplugging code for SCSI hosts was really stable -- then we could unregister the host when the device went away, and this relation would be disconnected automatically.

Quoted 3 lines · Matthew Dharm

Well, we only create the host when the device is first attached. After that, if it goes away and comes back, we re-connect it to the old SCSI host.

Ick, so when the device is gone, where does the SCSI host go? Is it still represented in sysfs and in the SCSI core properly?

Quoted 4 lines

Something like this would proabably make sense if the hot-unplugging code for SCSI hosts was really stable -- then we could unregister the host when the device went away, and this relation would be disconnected automatically.

Well, push back on the SCSI people to fix this then, hot-unplug should work properly on the SCSI layer too :)

When the USB storage device is unplugged, the device should be unregistered. It should be _gone_. It isn't sleeping, it's dead. It's an ex-device.

In article <[email protected]>, Greg KH <[email protected]>

Quoted 5 lines · wrote:

Quoted 3 lines · Matthew Dharm

Well, we only create the host when the device is first attached. After that, if it goes away and comes back, we re-connect it to the old SCSI host.

Quoted 2 lines

Ick, so when the device is gone, where does the SCSI host go? Is it still represented in sysfs and in the SCSI core properly?

This is pure and utter USB storage stupidity, and nothing else.

When the USB storage device is unplugged, the device should be unregistered. It should be _gone_. It isn't sleeping, it's dead. It's an ex-device.

Quoted 7 lines · Greg KH

Quoted 3 lines · Matthew Dharm

Well, we only create the host when the device is first attached. After that, if it goes away and comes back, we re-connect it to the old SCSI host.

Ick, so when the device is gone, where does the SCSI host go? Is it still represented in sysfs and in the SCSI core properly?

This is pure and utter USB storage stupidity, and nothing else.

Quoted 3 lines · Matthew Dharm

Well, we only create the host when the device is first attached. After that, if it goes away and comes back, we re-connect it to the old SCSI host.

Ick, so when the device is gone, where does the SCSI host go? Is it still represented in sysfs and in the SCSI core properly?

Have any of the SCSI people been looking much at SCSI hotplug on 2.5? I attach "/etc/hotplug/scsi.agent" from one of my desktops; all it does is make sure the right drivers are loaded, it doesn't have a clue yet about whether/how/where to mount disks or do other stuff.

Quoted 4 lines · David Brownell [[email protected]] wrote:

Have any of the SCSI people been looking much at SCSI hotplug on 2.5? I attach "/etc/hotplug/scsi.agent" from one of my desktops; all it does is make sure the right drivers are loaded, it doesn't have a clue yet about whether/how/where to mount disks or do other stuff.

SCSI has added newer interfaces for host drivers to use that allow single adds and removes. The api document is located in /Documentation/scsi/scsi_mid_low_api.txt. I believe it is fairly current.

Am Donnerstag, 16. Januar 2003 21:48 schrieb Mike Anderson:

Quoted 10 lines

Quoted 4 lines · David Brownell [[email protected]] wrote:

Have any of the SCSI people been looking much at SCSI hotplug on 2.5? I attach "/etc/hotplug/scsi.agent" from one of my desktops; all it does is make sure the right drivers are loaded, it doesn't have a clue yet about whether/how/where to mount disks or do other stuff.

SCSI has added newer interfaces for host drivers to use that allow single adds and removes. The api document is located in /Documentation/scsi/scsi_mid_low_api.txt. I believe it is fairly current.

This is good news in principle.
But what use is a function like scsi_remove_host() if it can fail? If a device is gone, it is gone and all the complaining in the world won't alter that.
Could you explain how a LLD can ensure that this function will always succeed?

Quoted 12 lines · Oliver Neukum [[email protected]] wrote:

Am Donnerstag, 16. Januar 2003 21:48 schrieb Mike Anderson:

Quoted 10 lines

Quoted 4 lines · David Brownell [[email protected]] wrote:

Have any of the SCSI people been looking much at SCSI hotplug on 2.5? I attach "/etc/hotplug/scsi.agent" from one of my desktops; all it does is make sure the right drivers are loaded, it doesn't have a clue yet about whether/how/where to mount disks or do other stuff.

SCSI has added newer interfaces for host drivers to use that allow single adds and removes. The api document is located in /Documentation/scsi/scsi_mid_low_api.txt. I believe it is fairly current.

Quoted 6 lines

This is good news in principle.
But what use is a function like scsi_remove_host() if it can fail? If a device is gone, it is gone and all the complaining in the world won't alter that.
Could you explain how a LLD can ensure that this function will always succeed?

The interface was focused on clean removes. There may be layers above the SCSI subsystem using the device that may need user space intervention to release. Currently the request_queue exported to the block layer is tied to the scsi device and cannot dis-associated.

(NOTE: in looking at this in 2.5.58 it appears that there maybe some checks missing in scsi_check_device_busy that would make scsi_remove_host succeed more often than it should).

If the device disappears the host needs to return all IOs in flight with an error, and we need to ensure new ones do not start ( setting the device offline and maybe host_self_blocked). This is a start on getting the scsi_remove_host to succed.

Am Freitag, 17. Januar 2003 09:50 schrieb Mike Anderson:

Quoted 5 lines

If the device disappears the host needs to return all IOs in flight with an error, and we need to ensure new ones do not start ( setting the device offline and maybe host_self_blocked). This is a start on getting the scsi_remove_host to succed.

Could you provide some details?

That is simply wrong. Reporting somebody having pulled a plug must not fail. What are you supposed to do with an error here?

There must be a way for a LLD to report that reliably.
If the answer is, take that lock, call that function, error all pending requests, release that lock and call that function, it's OK.

But it must work in all cases.

Quoted 8 lines · Oliver Neukum

That is simply wrong. Reporting somebody having pulled a plug must not fail. What are you supposed to do with an error here?

There must be a way for a LLD to report that reliably.
If the answer is, take that lock, call that function, error all pending requests, release that lock and call that function, it's OK.

But it must work in all cases.

I absolutely agree. The device is gone. I can't do anything about it. If the SCSI layer decides it can't let go, what am I supposed to do about it?

In a separate discussion with Mike, he mentioned that you can't scsi_remove_device() unless there are no pending commands.

How the hell is an LLD supposed to assure that!?!?

The minute I error a command and call scsi_done(), I can get a new one. Unless I lock out requests with scsi_block_requests(), but that comes with major warnings about needing to get unblocked.

The way this should work is that the LLD calls scsi_remove_device(), and that cuts off the flow of commands. The LLD can promise to error-out any pending commands in the device command queue.

That is, unless scsi_block_requests() and scsi_unblock_requests() are more useful than the documentation suggests... block(), error all commands, unregister()... that would make some sense. We could call scsi_block_request() as soon as we know the unit is gone, and unregister() as soon as the queue is empty.

Quoted 5 lines

The way this should work is that the LLD calls scsi_remove_device(), and that cuts off the flow of commands. The LLD can promise to error-out any pending commands in the device command queue.

Quoted 5 lines

That is, unless scsi_block_requests() and scsi_unblock_requests() are more useful than the documentation suggests... block(), error all commands, unregister()... that would make some sense. We could call scsi_block_request() as soon as we know the unit is gone, and unregister() as soon as the queue is empty.

We should really ensure that we have good separation between stopping device IO, device gone, and release resources.

- SCSI seems to have the flags to stop the IO, but instead of scsi_block_requests we may want to export the setting of device online. This can be done from sysfs now, but

not from the driver ( the driver does have a handle to the device, but it would be better to have an interface in case we need to do something addition operations).

- Possibly add a scsi_remove_device that would always succeed and a version of scsi_remove_host that calls scsi_remove_device for all devices. Though with the recent change to SCSI remove host to allow non sysfs device registration I do not believe we could ensure devices would be cleaned up.

In a separate discussion with Mike, he mentioned that you can't scsi_remove_device() unless there are no pending commands.

How the hell is an LLD supposed to assure that!?!?

The minute I error a command and call scsi_done(), I can get a new one. Unless I lock out requests with scsi_block_requests(), but that comes with major warnings about needing to get unblocked.

The way this should work is that the LLD calls scsi_remove_device(), and that cuts off the flow of commands. The LLD can promise to error-out any pending commands in the device command queue.

That is, unless scsi_block_requests() and scsi_unblock_requests() are more useful than the documentation suggests... block(), error all commands, unregister()... that would make some sense. We could call scsi_block_request() as soon as we know the unit is gone, and unregister() as soon as the queue is empty.

Quoted 6 lines

In a separate discussion with Mike, he mentioned that you can't scsi_remove_device() unless there are no pending commands.

How the hell is an LLD supposed to assure that!?!?

Quoted 3 lines

The minute I error a command and call scsi_done(), I can get a new one. Unless I lock out requests with scsi_block_requests(), but that comes with major warnings about needing to get unblocked.

If I understand the scsi code correctly, doing that will result in a memory leak at least. Perhaps exporting a function to declare a host's devices offline might do the trick. But as yet I havn't found out where the scsi layer actually checks that flag.

In a separate discussion with Mike, he mentioned that you can't scsi_remove_device() unless there are no pending commands.

How the hell is an LLD supposed to assure that!?!?

The way this should work is that the LLD calls scsi_remove_device(), and that cuts off the flow of commands. The LLD can promise to error-out any pending commands in the device command queue.

I take it you mean that the transport will tell the LLDD that the device is gone and it (LLDD) call the one above, SCSI Core to remove the device.

Hmm, more thinking needs to be done here, as shouldn't this be handled by hotplugging? I.e. Targets do not *initiate* events.

The transport can notify that the device is gone, but an ULP entity will call scsi_remove_device() not the other way around.

Quoted 8 lines

I take it you mean that the transport will tell the LLDD that the device is gone and it (LLDD) call the one above, SCSI Core to remove the device.

Hmm, more thinking needs to be done here, as shouldn't this be handled by hotplugging? I.e. Targets do not *initiate* events.

The transport can notify that the device is gone, but an ULP entity will call scsi_remove_device() not the other way around.

NO!

This is an insanely complicated scheme.
We have no notification beforehand. User yanks out cable. That's it. No preperation at all.

We as the writers of device drivers need a way to get rid of the device as we are notified of the physical disconnect.

It is not our job to maintain devices in an undead state.

And a scheme that goes subsystem driver -> hotplugging -> script finding corresponding devices -> script doing proc magic -> scsi layer notifying low level driver is _not_ sensible. It triples the amount of complexity.

We need a simple scheme like
1. block further requests
2. kill old requests
3. remove device

Quoted 3 lines

We as the writers of device drivers need a way to get rid of the device as we are notified of the physical disconnect.

Yes, and as I explained earlier: you *may* get notified by the transport.

But a removal of a device would probably have to start in top-down approach, to free/release/etc resources/etc, rather than a bottom-up approach (I just cannot see how this would work...)

In general, I wasn't really discussing hotplugging, I was basically hinting that SCSI Core could use a few more functionalities, and that some things need to go away.

Quoted 2 lines

The transport can notify that the device is gone, but an ULP entity will call scsi_remove_device() not the other way around.

NO!

This is an insanely complicated scheme.
We have no notification beforehand. User yanks out cable. That's it. No preperation at all.

We as the writers of device drivers need a way to get rid of the device as we are notified of the physical disconnect. It is not our job to maintain devices in an undead state.

Quoted 9 lines

Quoted 2 lines

The transport can notify that the device is gone, but an ULP entity will call scsi_remove_device() not the other way around.

NO!

This is an insanely complicated scheme.
We have no notification beforehand. User yanks out cable. That's it. No preperation at all.

Quoted 3 lines

We as the writers of device drivers need a way to get rid of the device as we are notified of the physical disconnect. It is not our job to maintain devices in an undead state.

If you think about it rationally there isnt a lot that can be done higher up. At the point the hardware vanishes there may be other threads of execution already in your driver, so undead state is a reality you have to live with, at least briefly.

<snip/>

I simply don't trust the SCSI layer. I've had to much trouble with it already.

Quoted 4 lines · Oliver Neukum

<snip/>

I simply don't trust the SCSI layer. I've had to much trouble with it already.

Hopefully we have some better building blocks and clearer code in 2.5 to look at this problem in the SCSI subsystem again.

The first thing a LLDD should do when it _knows_ the device is gone is set scsi_device::online=0 ** which should stop all

new commands being queued. Now if scsi_device::access_count is zero then we have no problems *** and most of the code we need is in place in latter half of scsi_remove_single_device().

The hard case is when scsi_device::access_count>0 which means open()s or mounts are active on that device. So sd, sr, st, osst and/or sg know about a file descriptor (or the block equivalent)

that is associated with that "departed" scsi_device instance. I have code in sg in lk 2.4 to partially handle the case when detach is called on a device for which sg holds an open fd. Sg can handle this because it shadows scsi_device and scsi_cmnd instances. The next time an app tries to access that fd it gets a ENODEV (even sends out a SIGIO/POLL_HUP for advanced apps). I suspect life would not be so simple for sd and sr due to their close binding with the block subsystem. Another approach to this problem is to keep the scsi_device instances for departed devices around until the access_count drops to zero. One silly idea I had was to change the seldom

used channel number to 1024 (or the next free number above that) to maintain the uniqueness of the host/channel/id/lun tuple **** and keep the original tuple available for the re-appearance of the departed device.

Compounding the hard case is when commands are queued. Can these simply be ENODEV-ed back to the apps that own them? If so, that may help the access_count drop to zero facilitating scsi_device removal.

Question: do we need to worry about hot unplugging of hosts?

** Since 'online' already has a usage (i.e. error recovery couldn't resurrect this device) perhaps something stronger like 'departed' is needed as well (or some sysfs mechanism). BTW sg sidesteps 'online' when a file descriptor is opened non-blocking.

So a disconnection would look like this:

scsi_set_offline_host(...);

synchronize_kernel();

error_queied_commands(...);

scsi_remove_host();

Providing you refcount objects and defer freeing of resources its not normally too terrible

Quoted 3 lines

The way this should work is that the LLD calls scsi_remove_device(), and that cuts off the flow of commands. The LLD can promise to error-out any pending commands in the device command queue.

I take it you mean that the transport will tell the LLDD that the device is gone and it (LLDD) call the one above, SCSI Core to remove the device.

Hmm, more thinking needs to be done here, as shouldn't this be handled by hotplugging? I.e. Targets do not *initiate* events.

The transport can notify that the device is gone, but an ULP entity will call scsi_remove_device() not the other way around.

Quoted 9 lines · Luben Tuikov

Quoted 3 lines

The way this should work is that the LLD calls scsi_remove_device(), and that cuts off the flow of commands. The LLD can promise to error-out any pending commands in the device command queue.

I take it you mean that the transport will tell the LLDD that the device is gone and it (LLDD) call the one above, SCSI Core to remove the device.

Hmm, more thinking needs to be done here, as shouldn't this be handled by hotplugging? I.e. Targets do not *initiate* events.

Not exactly, but the bus driver ("transport"?) certainly does initiate reports like "here's a new device on the bus" or "that device is gone". That's when hotplugging kicks in (both in-kernel and in-userland).

And the only way to access a device ("target") on the bus is to give a request to that bus driver. If, when servicing that request, the bus driver notices the device is gone ... that can act a lot like a device initiating a "device gone" event would look.

Quoted 2 lines

The transport can notify that the device is gone, but an ULP entity will call scsi_remove_device() not the other way around.

That's how USB works today: khubd shuts things down. Device drivers get disconnect() callbacks, just as when their modules are removed.

EXCEPT that "khubd" is part of usbcore (roughly analagous to parts of the scsi mid-layer) ... so the drivers acting as host side proxies for the target hardware ("usb device") are purely reactive. Their only roles in hotplug scenarios are to bind to devices (when a new one appears, using probe callbacks) or unbind from them (when one goes away, using disconnect callbacks).

Those disconnect() callbacks have a few key responsibilities, very much including shutting down the entire higher level I/O queue to that device. I think you're saying that SCSI drivers don't have such a responsibility (unlike USB or PCI) ... if so, that would seem to be worth changing.

That model cannot be applied to SCSI as it is much more diverse in the number of bus types it supports.
USB can do it, because it knows about hubs. SCSI cannot, as there are no hubs in SCSI.

Quoted 5 lines

Those disconnect() callbacks have a few key responsibilities, very much including shutting down the entire higher level I/O queue to that device. I think you're saying that SCSI drivers don't have such a responsibility (unlike USB or PCI) ... if so, that would seem to be worth changing.

If the scsi layer cannot on its own detect that a device or a bus is gone, there'll be no sense in having a callback. It's just a complication.

Quoted 4 lines · Oliver Neukum

That model cannot be applied to SCSI as it is much more diverse in the number of bus types it supports.
USB can do it, because it knows about hubs. SCSI cannot, as there are no hubs in SCSI.

Hubs are irrelevant here, the key functionality is noticing hardware addition/disconnect. Parts of it can be done in bus adapter code, parts of it can't. SCSI probes LUNS in much the same way khubd probes hub ports, and as I recall most of that logic isn't any more specific to the adapter than virtual root hub code is for USB.

Quoted 6 lines

Quoted 5 lines

Those disconnect() callbacks have a few key responsibilities, very much including shutting down the entire higher level I/O queue to that device. I think you're saying that SCSI drivers don't have such a responsibility (unlike USB or PCI) ... if so, that would seem to be worth changing.

Quoted 3 lines

If the scsi layer cannot on its own detect that a device or a bus is gone, there'll be no sense in having a callback. It's just a complication.

Erm ... which of the three SCSI layers are you talking about? I was talking about the highest level, which is precisely the layer I think has been identified as already needing to know when to shut down the I/O queues (sd_mod and friends).

I should be more specific.

The SCSI is different for several reasons:
- we are talking about bus as well as device detection
- there's no common way to probe for devices (the probing of LUNs works only on conventional busses)

- many SCSI devices are not really SCSI devices. They just use the command set

SCSI hotplug detection doesn't work for the same reason that USB can handle only detection of devices by itself. Busses on the other hand are not handled by USB itself.

Quoted 2 lines

Erm ... which of the three SCSI layers are you talking about? I was talking about the highest level,..

In SCSI view device and bus disconnection is recognised by the lowest level. As it knows nothing about the high layers, it notifies the midlayer

So the SCSI stack needs to support more than one model for device/bus detection. This can't be news. And some of them have to handle "conventional" busses, like USB and PCI; maybe even handle booting off them...

If you mean that HCs hook up to a different bus (often PCI), with its own hotplug support, that doesn't seem so different from SCSI HBAs hooking up to such busses (often PCI) and cascading the same hotplug support...

Quoted 6 lines

Quoted 2 lines

Erm ... which of the three SCSI layers are you talking about? I was talking about the highest level,..

In SCSI view device and bus disconnection is recognised by the lowest level. As it knows nothing about the high layers, it notifies the midlayer

So you were talking past what I said about notifying that highest level, not disagreeing with it.

Quoted 2 lines

So you were talking past what I said about notifying that highest level, not disagreeing with it.

I was trying to make the point that callbacks have no place in that process.

It must go bottom to top and that's it.

Refusing to take notice of a device removal is just not an option.
This is exactly what the current SCSI idea of an API to do bus removal does.

Quoted 5 lines · Oliver Neukum

Quoted 2 lines

So you were talking past what I said about notifying that highest level, not disagreeing with it.

I was trying to make the point that callbacks have no place in that process.

If so, you didn't persuade me ...

Quoted 1 line

It must go bottom to top and that's it.

... because those disconnect() callbacks are exactly how USB and PCI deliver that notification to the "top" level, and you've already agreed that SCSI needs to accomodate those models. So clearly they have at least that much of a place.

Quoted 3 lines

Refusing to take notice of a device removal is just not an option.
This is exactly what the current SCSI idea of an API to do bus removal does.

I perceive violent agreement that a change is needed in that area.

But the next step there would seem to be a patch to the SCSI APIs, unless I mis-understood what Matt was saying about the issue he ran into when making usb-storage use the enumeration facilities in the current SCSI mid/low layers.

Disconnect is not really a callback. There's a distinct lack of a back movement here. khubd -> usbcore -> disconnect() in driver -> [layer on top]

The proposed API in SCSI looks like:
<bus system> -> LLD -> midlayer -> top layer -> midlayer -> LLD with destroy_slave() and that's not OK.

Quoted 7 lines · Oliver Neukum

Disconnect is not really a callback. There's a distinct lack of a back movement here. khubd -> usbcore -> disconnect() in driver -> [layer on top]

The proposed API in SCSI looks like:
<bus system> -> LLD -> midlayer -> top layer -> midlayer -> LLD with destroy_slave() and that's not OK.

No, not quite.

When the Low Level Device Driver (LLDD), being the transport portal, notices that the device is going away or has gone away from the ``fabric'' (wlg), it will fire a device-gone event with the kernel. *Not* necessarily with SCSI Core, in fact I'd rather it didn't, but with a well defined kernel entry for device-gone events.

At the same time the LLDD will start returning TARGET gone, or whatever is appropriate to newly queued commands, and error out all internally queued commands (if it does it's own queuing). (I've seen this work nicely on mount and read/write(2) and fsck.)

I.e. the ``synchronization'' has started already by the LLDD erroring out commands, new and queued.

All the while the kernel has started higher level cleaning up, decrementing ref counts, etc, stuff which may not be so easy to be cleaned up just by LLDD returning TARGET error. Even though,

good design dictates that complete cleaning up should happen just by the LLDD returning TARGET error (e.g. on mount), we *have* to allow for this immediate high level entry point (as I mentioned above) notification, which will be kind of ``meeting place'' for events like this.

Depending on what needs to be done at those ``higher'' levels, the event will eventually bubble down to the SCSI Core with something like scsi_remove_device() which will do slave_destroy() in the driver.

The point is that at that point in time, it will be *safe* to do scsi_remove_device() as all ULP have alreay been notified, and they've relinquished their use of the LLD (Low Level Device), thus the safety.

But there's no such thing as ``waiting around indefinitely'' or ``blocking wait'' as you've suggested in some of your emails.

Even if this UL entry point doesn't do anything, ref counts should go to zero, after all users error out on this device, at which point the user can remove the device from *the system* by hand/old method through proc or whatever finalizes for 2.6.

Am Dienstag, 21. Januar 2003 19:16 schrieb Luben Tuikov:

Quoted 6 lines

When the Low Level Device Driver (LLDD), being the transport portal, notices that the device is going away or has gone away from the ``fabric'' (wlg), it will fire a device-gone event with the kernel. *Not* necessarily with SCSI Core, in fact I'd rather it didn't, but with a well defined kernel entry for device-gone events.

Well, we are in feature freeze. I see no alternative but to notify the mid layer. Who else but the mid layer knows what a physical device is logically associated with?

Quoted 5 lines

At the same time the LLDD will start returning TARGET gone, or whatever is appropriate to newly queued commands, and error out all internally queued commands (if it does it's own queuing). (I've seen this work nicely on mount and read/write(2) and fsck.)

Right.

Quoted 6 lines

I.e. the ``synchronization'' has started already by the LLDD erroring out commands, new and queued.

All the while the kernel has started higher level cleaning up, decrementing ref counts, etc, stuff which may not be so easy to be cleaned up just by LLDD returning TARGET error. Even though,

You cannot really make anything depend on errors returned, because there simply may not be any commands queued. You can make it a

requirement for an LLDD to return all commands in flight with an error, but you can do little with these errors. Basically you have to treat them

like uncorrectable errors, except maybe for the error code returned to user space. But the processing of the disconnect itself should be triggered by the LLDD's notification, because it's the only indication of an unplug event you are sure to get.

Quoted 4 lines

good design dictates that complete cleaning up should happen just by the LLDD returning TARGET error (e.g. on mount), we *have* to allow for this immediate high level entry point (as I mentioned above) notification, which will be kind of ``meeting place'' for events like this.

That I don't understand. It would seem to me to be cleanest to have just one path to process a disconnect event.

Quoted 7 lines

Depending on what needs to be done at those ``higher'' levels, the event will eventually bubble down to the SCSI Core with something like scsi_remove_device() which will do slave_destroy() in the driver.

The point is that at that point in time, it will be *safe* to do scsi_remove_device() as all ULP have alreay been notified, and they've relinquished their use of the LLD (Low Level Device), thus the safety.

But there can be no users of the LLDD at this point. There can of course be references to devices and hosts, but not really uses.

After we have done a notification of the event the first things to do are to make further opening of the device fail and make sure no more commands are sent to the device. Likewise all queued commands have returned with an error. So at this point it's impossible to use an unplugged device.

Quoted 7 lines

But there's no such thing as ``waiting around indefinitely'' or ``blocking wait'' as you've suggested in some of your emails.

Even if this UL entry point doesn't do anything, ref counts should go to zero, after all users error out on this device, at which point the user can remove the device from *the system* by hand/old method through proc or whatever finalizes for 2.6.

You cannot be sure that reference counts will go to zero ever. You can be sure that they won't increase as you can fail any operation that would cause them to increase, but you cannot force userland to close its fds. And waiting for somebody to remove a device is wrong. It's gone physically. There's no choice but to remove it. The refcounts can tell you when to free data structures associated with devices, but what else do you want them to do?

Here's another question to add to your discussion.

When a device is unplugged, the system's representation of that device can't be removed immediately; there may be open fd's, mounts, pointers, and so on. Until the time comes when all these handles are released, all interaction with the device has to fail, one way or another.

Whose responsibility is it to fail these interactions?

For something simple, like a USB serial port, it might turn out that the low-level device driver gets all these requests and then fails them. That means the driver has to keep track of the fact that the device is no longer connected until some reference count goes to 0.

For SCSI and emulated SCSI devices, it might be the one of the SCSI layers that keeps track of the fact that the device has disconnected. Or it might be somewhere else in the kernel.

It would nice to have some sort of coherent plan for how to handle this. In fact, it ought to be part of the device-driver model that underlies sysfs. But so far as I am aware, there is currently nothing in the sysfs documentation to address the problem.

Quoted 2 lines

Whose responsibility is it to fail these interactions?

The transport.

When a device is plugged to the SAN/fabric (wlg) it may not be the case that all initiators will know about it. For this reason the transport itself, not SCSI Core, not LLDD*, will decide if the CDB is deliverable.

* An LLDD may keep a table of seen devices (depending on the transport it provides a portal to), and thus the decision may be made there, but this doesn't have to be the case.

(Just think of SANs/IP SANs.)

Quoted 4 lines

For something simple, like a USB serial port, it might turn out that the low-level device driver gets all these requests and then fails them. That means the driver has to keep track of the fact that the device is no longer connected until some reference count goes to 0.

A LLDD doesn't have to keep reference counts. In the simple case you mention above, it will check that the device is no longer reachable and will return TARGET error, which will bubble up the layers, or the

Newer drivers will make use of Service Response code, and be able to address only by (target, lun) rather than (bus, target, lun), and target may not be an int anymore. But this is 2.7 stuff, or maybe a separately distributed SCSI Core and LLDDs subsystem...

Not all the world is a SAN. USB has no possibility to even try an interaction after the device is gone. We have to handle this flexibly.

In fact, if a device
can vanish without a LLDD knowing about it, this is purely a problem of the SCSI layer.

Quoted 7 lines

Quoted 3 lines

For something simple, like a USB serial port, it might turn out that the
low-level device driver gets all these requests and then fails them. That means the driver has to keep track of the fact that the device is no longer connected until some reference count goes to 0.

A LLDD doesn't have to keep reference counts. In the simple case you mention above, it will check that the device is no longer reachable and will return TARGET error, which will bubble up the layers, or the

That is something that is impossible to some LLDDs. We have to keep a record about devices and busses we can reach and can delete these records only after we positively know that no more commands will come down to the LLDD.

The alternative would be to check a table of available devices for every command.

That means that we have to have a way to ensure that no more commands will reach the LLDD which can be triggered without any commands to be executed at all. This functionality has to come from the scsi mid layer.

Quoted 4 lines

Newer drivers will make use of Service Response code, and be able to address only by (target, lun) rather than (bus, target, lun), and target may not be an int anymore. But this is 2.7 stuff, or maybe a separately distributed SCSI Core and LLDDs subsystem...

Yes, but we need a solution for 2.6.
And it has to be reasonably simple.

Quoted 2 lines · Oliver Neukum

Not all the world is a SAN. USB has no possibility to even try an interaction after the device is gone. We have to handle this flexibly.

Thus the example in the original post. I.e. for simple transports whose portals get notified when a device is plugged off (USB), the LLDD can notify SCSI Core, by setting a state variable in scsi_device. In which case SCSI Core can answer with the proper TARGET error code. (This was outlined before, scsi_command->online:1 ...)

Quoted 3 lines

In fact, if a device
can vanish without a LLDD knowing about it, this is purely a problem of the SCSI layer.

No, of course not. (Think of IP.) When a device vanishes and LLDD doesn't know about it (more complicated transports), the CDB will return with the proper Service Response, since the transport(s) won't be able to deliver it. This will bubble up through SCSI Core and the error returned will have to be the same as that of the simpler transports, as outlined above.

Quoted 3 lines

That means that we have to have a way to ensure that no more commands will reach the LLDD which can be triggered without any commands to be executed at all. This functionality has to come from the scsi mid layer.

For simple transports yes; for more complicated ones, the CDB will not be able to be delivered, and will return with error.

Very well, so you agree that the SCSI layer should export to the LLDD a function to set devices offline?

Yes, sorry. To be precise, this means that the LLDD has to do nothing special, as it has to implement checking for a failing command anyway. But it's not entirely the same. If a command cannot be delivered it may or may not be appropriate to start error recovery. After the LLDD has told the SCSI layer that it has noticed a device going away, there must be no error recovery.

Good.
So the first thing a LLDD has to do after it has learned about a device being removed is to have the device block.

1. set device offline
But commands may still be in flight.IMHO it is not right to assume that all commands now in flight to a device have failed, as some may have completed successfully in time, or failed for other reasons than unplugging.

So it should be the LLDD's responsibility to finish the outstanding commands.

Furthermore, there's a window for commands already having passed the check for offline but not yet being noticed by the LLDD.

The simplest solution is to
use a waiting primitive from RCU. So we are at:

1. set device offline
2. synchronize the kernel
3. finish all pending commands

So far with me?
The LLDD could now forget about the device and be done with it.

However there's a problem left. The device may come back. What happens if a device with the same ID is reconnected?

Quoted 2 lines · Oliver Neukum

Very well, so you agree that the SCSI layer should export to the LLDD a function to set devices offline?

I've never really disagreed -- simpler transports will make use of such a function. The important point to note is that the error

return value for simpler and more complicated transports has to be the same (i.e. ones which know about the device disconnect and others which send out the CDB and which will return with error).

I forgot to mention this with my previous email: think of a LLDD more as part of the transport than of SCSI Core.

Quoted 2 lines

So the first thing a LLDD has to do after it has learned about a device being removed is to have the device block.

``block'' (verb) is such a strong word.

* Simple transports: call scsi_set_device_offline(dev) or something like this.

* More complicated transports: SCSI Core sees Service Response of Service Delivery Failure and it itself calls scsi_set_device_offline(dev).

scsi_set_device_offline(dev) calls a high-level kernel function to start higher level things (block queue cut off, etc) which *may* need to be done.

Quoted 2 lines

So it should be the LLDD's responsibility to finish the outstanding commands.

LLDD cannot really ``finish'' outstanding commands, it's just a transport portal.

Quoted 2 lines

Furthermore, there's a window for commands already having passed the check for offline but not yet being noticed by the LLDD.

They will return with an appropriate error.

Quoted 6 lines

The simplest solution is to
use a waiting primitive from RCU. So we are at:

1. set device offline
2. synchronize the kernel
3. finish all pending commands

I told you before: 3 starts *before* 2 and 3 is *part* of 2. Furthermore, after 1 has happened in time, all pending commands will error out (wrt a time line).

2 is what I call ``higher-level hook'', but it's not really ``synchronization''. Synchronization will take delta-time, it will not happen instantaneously.

I.e. you have to allow for the possibility of a command being sent to a non-existent device through LLDD, in which case the LLDD/transport will have to error it out.

Am Donnerstag, 23. Januar 2003 20:07 schrieb Luben Tuikov:

Quoted 3 lines

return value for simpler and more complicated transports has to be the same (i.e. ones which know about the device disconnect and others which send out the CDB and which will return with error).

Why? It throws away information needlessly. If the LLDD knows that the reason is unplugging why not report it?

A LLDD that doesn't
know about devices going away on the other hand can just report an error.

Can the higher layers simply assume that the device was unplugged? IMHO they can't and should at least try to recover from the error.

Quoted 2 lines

I forgot to mention this with my previous email: think of a LLDD more as part of the transport than of SCSI Core.

Hard to do. The scsi mid layer does timing out and error handling. There's a relatively tight connection.

Quoted 4 lines

Quoted 2 lines

So the first thing a LLDD has to do after it has learned about a device being removed is to have the device block.

``block'' (verb) is such a strong word.

What do you prefer ? ;-) I'll certainly use another word if you like me to do so.

Quoted 5 lines

* Simple transports: call scsi_set_device_offline(dev) or something like this.

* More complicated transports: SCSI Core sees Service Response of Service Delivery Failure and it itself calls scsi_set_device_offline(dev).

Quoted 2 lines

scsi_set_device_offline(dev) calls a high-level kernel function to start higher level things (block queue cut off, etc) which *may* need to be done.

How do you differentiate between real failure and device removal?

Quoted 5 lines

Quoted 2 lines

So it should be the LLDD's responsibility to finish the outstanding commands.

LLDD cannot really ``finish'' outstanding commands, it's just a transport portal.

Well, report back the results, if you prefer, thus returning ownership to higher layers.

Quoted 2 lines

Quoted 2 lines

Furthermore, there's a window for commands already having passed the check for offline but not yet being noticed by the LLDD.

Quoted 1 line

They will return with an appropriate error.

Not quite so simple. Some LLDDs need to know at some point that no more commands will arrive for sure and none are still in flight.

/proc/scsi/scsi to remove it.

Actually, I would have both complicated and simple transports call scsi_set_device_offline() and for two reasons. 1) you have to provide that function for simple drivers so duplicating other detection code in the scsi completion handler is a waste. 2) pretty much all transports will learn of the device being offline while they are in their interrupt handler and should already be holding the lock for the device, which means

that calling scsi_set_device_offline() won't race with scsi_request_fn() which also needs the device lock (which in reality is the host lock). Saving this race is convenient enough IMHO to warrant saying that's the way things need to be.

Quoted 3 lines

Quoted 3 lines

scsi_set_device_offline(dev) calls a high-level kernel function to start higher level things (block queue cut off, etc) which *may* need to be done.

No, scsi_set_device_offline() schedules the error handler thread for that host to be woken up.

Quoted 1 line

How do you differentiate between real failure and device removal?

We don't, and we shouldn't. Device removal *is* a real failure.

If the LLDD is the type such that it knows the device is gone (aka, in my driver if I get a selection timeout then I know something is fishy and can proceed from there, iSCSI may not be so lucky), then it has one of two choices. 1) it may flush any commands that it can out of the hardware and return them immediately with the same error condition as the one that it is already returning. 2) it can sit and wait for the commands to timeout one by one if that's what it wants. Since the device has already been marked offline by scsi_set_device_offline() and the error handler thread is already scheduled to run for the device, 2 is probably the easiest thing for the driver to do. The error handler will call the abort/reset

Once all the commands are gone and no more are arriving, then if, and only if, someone actually removes the device from the scsi subsystem (maybe hotplug manager or something) then you will get the typical slave_destroy() call to tell you that it is safe to release all resources related to this device. Otherwise, the device will hang around as an offline device until someone does echo "scsi-remove-single-device a b c d"

Quoted 1 line

/proc/scsi/scsi to remove it.

Basically, as I see it, we need a new function scsi_set_device_offline() that marks the device offline, we need an offline check in

As far as plugging back in, the answer is simple. Until the old instance is dead *and removed* a new one can't be added at the same ID, aka you simply ignore the hot plug until the hot remove has completed.

Hi Doug

Quoted 6 lines

Actually, I would have both complicated and simple transports call scsi_set_device_offline() and for two reasons. 1) you have to provide that function for simple drivers so duplicating other detection code in the scsi completion handler is a waste. 2) pretty much all transports will learn of the device being offline while they are in their interrupt handler and should already be holding the lock for the device, which means

This is not the case for USB and IEEE1394. I am not sure about PCMCIA. We are in context of a kernel thread while we learn about device removal.

Quoted 12 lines

that calling scsi_set_device_offline() won't race with scsi_request_fn() which also needs the device lock (which in reality is the host lock). Saving this race is convenient enough IMHO to warrant saying that's the way things need to be.

Quoted 3 lines

Quoted 3 lines

scsi_set_device_offline(dev) calls a high-level kernel function to start higher level things (block queue cut off, etc) which *may* need to be done.

No, scsi_set_device_offline() schedules the error handler thread for that host to be woken up.

Quoted 3 lines

Quoted 1 line

How do you differentiate between real failure and device removal?

We don't, and we shouldn't. Device removal *is* a real failure.

Well shouldn't a device removal remove the device as a logical entity and a failure should not?

Quoted 10 lines

If the LLDD is the type such that it knows the device is gone (aka, in my driver if I get a selection timeout then I know something is fishy and can proceed from there, iSCSI may not be so lucky), then it has one of two choices. 1) it may flush any commands that it can out of the hardware and return them immediately with the same error condition as the one that it is already returning. 2) it can sit and wait for the commands to timeout one by one if that's what it wants. Since the device has already been marked offline by scsi_set_device_offline() and the error handler thread is already scheduled to run for the device, 2 is probably the easiest thing for the driver to do. The error handler will call the abort/reset

Again not for USB and IEEE1394. We'd have to wait for the error handler to finish. Doing it ourselves is easier.

Quoted 6 lines

Once all the commands are gone and no more are arriving, then if, and only if, someone actually removes the device from the scsi subsystem (maybe hotplug manager or something) then you will get the typical slave_destroy() call to tell you that it is safe to release all resources related to this device. Otherwise, the device will hang around as an offline device until someone does echo "scsi-remove-single-device a b c d"

Eek. That part I must strongly object to. The device is physically gone. Ever bothering the LLDD with it is very inconvinient.

Quoted 4 lines

Quoted 1 line

/proc/scsi/scsi to remove it.

Basically, as I see it, we need a new function scsi_set_device_offline() that marks the device offline, we need an offline check in

These functions are needed for a whole bus as well. USB needs it.

Quoted 3 lines

As far as plugging back in, the answer is simple. Until the old instance is dead *and removed* a new one can't be added at the same ID, aka you simply ignore the hot plug until the hot remove has completed.

What do you mean? It is dead because it is removed. How can a device be anything than dead if it has been unplugged? Please elaborate.

And who should ignore a hot addition, the LLDD or SCSI core. If the former, again I must object.

No. You might be in a kernel thread context when you decode an interrupt down to determining that a device was removed, but somewhere along the line you took an interrupt that told you the device was removed (or else the command simply timed out and you are in the error handler for the command already). Are you saying that the USB subsystem queues up those interrupt packets and decodes them later (which is fine, I just want to be clear on the point)?

No. That's what the user space hot plug manager is for. If you want this type of behaviour, you take an interrupt to tell you that the device is gone, you mark it gone, the error handler cleans up any outstanding commands, then once the device no longer has any commands outstanding *then* the hot plug manager can successfully umount/unattach/whatever the device and then tell the kernel to actually remove it. Putting this into the scsi stack when it's already in place elsewhere makes no sense to me.

Quoted 12 lines

Quoted 11 lines

If the LLDD is the type such that it knows the device is gone (aka, in my driver if I get a selection timeout then I know something is fishy and can proceed from there, iSCSI may not be so lucky), then it has one of two choices. 1) it may flush any commands that it can out of the hardware and return them immediately with the same error condition as the one that it is already returning. 2) it can sit and wait for the commands to timeout one by one if that's what it wants. Since the device has already been marked offline by scsi_set_device_offline() and the error handler thread is already scheduled to run for the device, 2 is probably the easiest thing for the driver to do. The error handler will call the abort/reset

Quoted 2 lines

Again not for USB and IEEE1394. We'd have to wait for the error handler to finish. Doing it ourselves is easier.

OK, are you reading my comments or not? I said "since the error handler

thread is already scheduled to run for the device, 2 is probably easiest". In other words, you don't have to wait for anything, it's gonna happen post-haste. So since you should already have proper error handling functions in place (You do have proper error handler functions in place, don't you?), duplicating that code here won't really buy you anything.

OK, let's look at this realistically. I'm saying you get an interrupt telling you that the device is gone and you tell the scsi core the same thing. Immediately after that the scsi core calls your error handler routines to clean up any pending commands on the device. Once all those pending commands are cleaned up, the hot plug manager is free to remove the device from the system. Once the hot plug manager calls for the free to happen, you get a slave_destroy() call and you free the instances. This all happens in a span of a few milliseconds most likely. Is that really so inconvenient for you?

Quoted 8 lines

No. You might be in a kernel thread context when you decode an interrupt down to determining that a device was removed, but somewhere along the line you took an interrupt that told you the device was removed (or else the command simply timed out and you are in the error handler for the command already). Are you saying that the USB subsystem queues up those interrupt packets and decodes them later (which is fine, I just want to be clear on the point)?

Yes, an interrupt occurs, it's queued, a kernel thread woken up, it decodes the interrupts and notifies the device driver.

Quoted 7 lines

No. That's what the user space hot plug manager is for. If you want this type of behaviour, you take an interrupt to tell you that the device is gone, you mark it gone, the error handler cleans up any outstanding commands, then once the device no longer has any commands outstanding *then* the hot plug manager can successfully umount/unattach/whatever the device and then tell the kernel to actually remove it. Putting this into the scsi stack when it's already in place elsewhere makes no sense to me.

Well, it's a SCSI matter.

Oh, I do. Some of them just seem impractical from a USB point of view.

Quoted 5 lines

thread is already scheduled to run for the device, 2 is probably easiest". In other words, you don't have to wait for anything, it's gonna happen post-haste. So since you should already have proper error handling functions in place (You do have proper error handler functions in place, don't you?), duplicating that code here won't really buy you anything.

I have memory to free. I can do that only after the last command is gone. I'd have to yield in a loop and count commands. All quite messy.

What I want:

I want to be able to free resources associated with a device within a finite (and well bounded) amount of time after I am notified that the device is gone.

I want to be able to inform the SCSI mid-layer, which will then inform higher layers, that the device is gone. This is so that all may deal with this however they want. I really don't care who does what, as long as we don't crash. This implies that all block-type drivers will need to become hotplug aware, or the SCSI mid-layer will have to fake command failures.

I want to be able to do as little command-trickery as possible. If I have to do it, then that means the next hotplug-capable LLDD must do it also. Duplication of code is bad -- it should all be handled in the mid-layer.

Matt's current proposed patch has the USB LLDD calling scsi_unregister_host() (because the device is respresented as an emulated host adapter) when it learns that the device is gone. Provided that routine doesn't block for very long, provided it handles all the details of hotplug notifications, and provided it guarantees that after it returns there will be no more calls to the emulated adapter, I don't see any problem. The LLDD can go ahead and remove all records of the former device, secure in the knowledge that all pointers to data structures and entry points have been erased.

Isn't this exactly what everyone has been asking for and debating about?

Quoted 12 lines · Matthew Dharm

What I want:

I want to be able to free resources associated with a device within a finite (and well bounded) amount of time after I am notified that the device is gone.

I want to be able to inform the SCSI mid-layer, which will then inform higher layers, that the device is gone. This is so that all may deal with this however they want. I really don't care who does what, as long as we don't crash. This implies that all block-type drivers will need to become hotplug aware, or the SCSI mid-layer will have to fake command failures.

Quoted 3 lines

I want to be able to do as little command-trickery as possible. If I have to do it, then that means the next hotplug-capable LLDD must do it also. Duplication of code is bad -- it should all be handled in the mid-layer.

Matt's current proposed patch has the USB LLDD calling scsi_unregister_host() (because the device is respresented as an emulated host adapter) when it learns that the device is gone. Provided that routine doesn't block for very long, provided it handles all the details of hotplug notifications, and provided it guarantees that after it returns there will be no more calls to the emulated adapter,

I don't see any
problem. The LLDD can go ahead and remove all records of the former device, secure in the knowledge that all pointers to data structures and entry points have been erased.

Isn't this exactly what everyone has been asking for and debating about?

Alan Stern

Matt's current proposed patch has the USB LLDD calling scsi_unregister_host() (because the device is respresented as an emulated

host adapter) when it learns that the device is gone. Provided that routine doesn't block for very long, provided it handles all the details of hotplug notifications, and provided it guarantees that after it returns there will be no more calls to the emulated adapter, I don't see any problem. The LLDD can go ahead and remove all records of the former device, secure in the knowledge that all pointers to data structures and entry points have been erased.

Quoted 3 lines · Alan Stern

Matt's current proposed patch has the USB LLDD calling scsi_unregister_host() (because the device is respresented as an emulated

A LLDD should and must *not* call scsi_unregister_host(). This brakes all hierarchy.

It would be nice if the SCSI mid-layer kept track of what commands were in what stages in who's queues.

A LLDD should and must *not* call scsi_unregister_host(). This brakes all hierarchy.

Oh, I forgot to mention in my previous by date letter:
The entity which calls scsi_register_host() should by good design call scsi_unregister_host().

Are you aware that you are contradicting yourself?

Are you aware that you are contradicting yourself?

Quoted 2 lines · Luben Tuikov

A LLDD should and must *not* call scsi_unregister_host(). This brakes all hierarchy.

Quoted 3 lines

Quoted 2 lines

A LLDD should and must *not* call scsi_unregister_host(). This brakes all hierarchy.

What I probably meant is the detect()/release() pair; release() itself normally calls scsi_unregister(host); the point is that it got nudged from ``above'', i.e. SCSI Core.

How can a LLDD be certain that it can safely call scsi_unregister_host() whenever it wishes? As Doug pointed out this leads to problems.

Furhtermore, are we talking about scsi_unregister_host() or scsi_unregister(host)? The former does drivers and the latter does hosts. This would mean that my original statement was nevertheless correct, how can a LLDD decide to unload itself safely?

Apparently it can't. I don't mean to say that this was the right thing to do; I just meant that this is what Matt's currently-proposed patch does. Personally, I'm not very familiar with the details of the SCSI subsystem, and I don't know what preconditions are required for calling the various API's.

Quoted 10 lines · Luben Tuikov

Quoted 3 lines

Quoted 2 lines

A LLDD should and must *not* call scsi_unregister_host(). This brakes all hierarchy.

What I probably meant is the detect()/release() pair; release() itself normally calls scsi_unregister(host); the point is that it got nudged from ``above'', i.e. SCSI Core.

How can a LLDD be certain that it can safely call scsi_unregister_host() whenever it wishes? As Doug pointed out this leads to problems.

Apparently it can't. I don't mean to say that this was the right thing to do; I just meant that this is what Matt's currently-proposed patch does. Personally, I'm not very familiar with the details of the SCSI subsystem, and I don't know what preconditions are required for calling the various API's.

Quoted 4 lines

Furhtermore, are we talking about scsi_unregister_host() or scsi_unregister(host)? The former does drivers and the latter does hosts. This would mean that my original statement was nevertheless correct, how can a LLDD decide to unload itself safely?

I did indeed type it wrong. The code first calls scsi_remove_host(host) and then it calls scsi_unregister(host).

Well, I've been watching this go on for days. I hate to weigh in now, but I think someone needs to understand what the guy writing the code that is facing this problem really wants.

First, let me say that a USB storage device shows up as a HBA. That's because some devices are actually USB/SCSI bridges. But, since I work at the 'emulated host' level, that's where I'm focused.

What I want:

I want to be able to free resources associated with a device within a finite (and well bounded) amount of time after I am notified that the device is gone.

I want to be able to inform the SCSI mid-layer, which will then inform higher layers, that the device is gone.

This is so that all may deal with
this however they want. I really don't care who does what, as long as we don't crash. This implies that all block-type drivers will need to become hotplug aware, or the SCSI mid-layer will have to fake command failures.

I want to be able to do as little command-trickery as possible. If I have to do it, then that means the next hotplug-capable LLDD must do it also. Duplication of code is bad -- it should all be handled in the mid-layer.

As yet, the interface I have to the SCSI mid-layer fails on all three points here.

And now, some of my opinions on how this should all work:

It would be nice if the user informed us about removing the device before they did it. But we shouldn't crash if they don't.

I don't want to be hanging around after a device is gone, spinning my wheels because some other part of the kernel can't handle the fact that the device is gone. My driver is a passthru between the a SCSI emualted host and a physical USB device -- if my device is gone, I want to be out of there. (Oddly enough, I'm starting to think there may be a DoS attack here if you force the LLDD to stay -- after all, it consumes memory....)

Remember, the physical plug doesn't ask me if it's okay, and I don't get to ask the SCSI mid-layer if it's okay. Yes, starting with the user clicking to tell us would be nice, but I don't get to see that. All I get to see is an indication that the plug is pulled.

I don't really give a rat's a** about 'how SCSI works' or how it's specified or CAM models or any of that. I try to live in the real world as much as possible. In that world, I'm not asking to remove an HBA -- I'm telling you it's been removed. I can't call it back. I can't even fake a command (other than perhaps INQUIRY) in any meaningful way. THERE IS NOTHING I CAN DO BUT KEEP INSISTING THAT THE DEVICE IS GONE!

It would be nice if the user could inform various parts of the kernel that this device was going away, and then all sorts of cleanup could happen. But I really don't care -- all I'm trying to do is exit without a resource leak under all circumstances.

It would be nice if the SCSI mid-layer kept track of what commands were in what stages in who's queues. After all, if I hot-unplug a PCI SCSI controller, the controller really isn't going to be able to complete those commands for us -- we have to assume that commands queued by a LLDD are really just being sent to the hardware for queuing. If that wasn't the case, then having LLDD queue capability doesn't make sense.

Now, here's the kicker -- this is what I think Linus wants:

Linus said to me, with a degree of annoyance, that he doesn't want usb-storage to keep any associations of departed devices with SCSI emulated hosts. That means that I need to be able to add and remove hosts at the will of the end-user. In the end, what drives the entire process is what Linus' hand does when it's placed on his USB flashcard reader.

So, what you should be doing in
order to be both a nice scsi host that plays well with the generic mechanism we have in place is when you get this removal event, you should be free'ing all the state you needed about the usb bus and such and taking this usb device off line or whatever you do. Then let the scsi mid layer clean up at it's leisure. You don't need to worry about it because the only thing you will have left is to wait for the scsi subsys to call you when it's time to delete things. You don't even have to keep device references around because we pass those in to your deletion routines anyway.

Quoted 2 lines

I want to be able to inform the SCSI mid-layer, which will then inform higher layers, that the device is gone.

scsi_set_device_offline() as we've been discussing.

Doug,
I started writing the interface you put forth in your email. I am currently debugging it in UML so I can generate the error conditions in a control manner.

I still have some stuff to look
at in the error handler with it running in this mode as it previously expected no one else to be possibly doing operations on the host. This could be the case if other LLDD's use this interface and have another device that happens to timeout an IO post a device being set offline.

A clarification question below.

Quoted 14 lines · Doug Ledford [[email protected]] wrote:

So, what you should be doing in
order to be both a nice scsi host that plays well with the generic mechanism we have in place is when you get this removal event, you should be free'ing all the state you needed about the usb bus and such and taking this usb device off line or whatever you do. Then let the scsi mid layer clean up at it's leisure. You don't need to worry about it because the only thing you will have left is to wait for the scsi subsys to call you when it's time to delete things. You don't even have to keep device references around because we pass those in to your deletion routines anyway.

Quoted 2 lines

I want to be able to inform the SCSI mid-layer, which will then inform higher layers, that the device is gone.

Quoted 2 lines

scsi_set_device_offline() as we've been discussing.

I assumed that the hotplug event would only come from this function if no commands where outstanding. If there where commands outstanding the event would not be generated until the error handler gained ownership of all the commands.

Doug,
I started writing the interface you put forth in your email.

Quoted 2 lines · Mike Anderson

Doug,
I started writing the interface you put forth in your email.

Do you mind clarifying? Either it was a private email, or one posted here, in which case there was an interpretation.

Quoted 8 lines · Matthew Dharm

Well, I've been watching this go on for days. I hate to weigh in now, but I think someone needs to understand what the guy writing the code that is facing this problem really wants.

First, let me say that a USB storage device shows up as a HBA. That's because some devices are actually USB/SCSI bridges. But, since I work at the 'emulated host' level, that's where I'm focused.

Quoted 5 lines

What I want:

I want to be able to free resources associated with a device within a finite (and well bounded) amount of time after I am notified that the device is gone.

This is doable. Partly. You need to do it within the framework of what the scsi subsys needs, but it can be done. Basically, the scsi core has an object allocation oriented paridigm where the lldd is expected to act as a driver that handles creation, use, and destruction of objects at the mid layer's direction. Changing that paridigm would be very difficult. However, we don't need the usb subsys to maintain full state on the device once it has gone away, just enough state to be able to talk to the mid layer and tell it where the outstanding commands are. Once the mid layer is finally done with it, the mid layer will tell your subsys to do a final free of the device by calling your release routine (for hosts) or your slave_destroy routine (for devices). So, what you should be doing in order to be both a nice scsi host that plays well with the generic mechanism we have in place is when you get this removal event, you should be free'ing all the state you needed about the usb bus and such and taking this usb device off line or whatever you do. Then let the scsi mid layer clean up at it's leisure. You don't need to worry about it because the only thing you will have left is to wait for the scsi subsys to call you when it's time to delete things. You don't even have to keep device references around because we pass those in to your deletion routines anyway.

Quoted 2 lines

I want to be able to inform the SCSI mid-layer, which will then inform higher layers, that the device is gone.

scsi_set_device_offline() as we've been discussing.

Quoted 4 lines

This is so that all may deal with
this however they want. I really don't care who does what, as long as we don't crash. This implies that all block-type drivers will need to become hotplug aware, or the SCSI mid-layer will have to fake command failures.

As I stated in my proposed design email, this will already be taken care of. As long as you call scsi_set_device_offline() while holding the host lock for this host, then there will be no races between this call and the queuecommand() call which should be *all* that you care about. You will no longer get commands for this device. Now let me inform you of what the scsi subsys cares about. You need to return *all* outstanding commands that this device has in your lldd before you go off and play "my device is gone so I'm free'ing everything". That hasn't been the case from what I've seen, and that's what the scsi subsys needs. If you don't return absolutely *all* outstanding commands before you go around free'ing stuff, then what happens is the scsi subsys needs those commands back in order to free up the scsi structs and it calls into the usb stack to get a reset or abort on the command and you guys have already free'd up your stuff and you don't claim to know what we are talking about. At that point, we have a permanently stuck device. This is the part where I said yesterday that you could handle this at the time you set the device offline by adding the code to return all the commands or you could just let your already present error handler code get called by the error handler thread and return the outstanding commands that way. But, one way or the other, it has to be done.

Quoted 3 lines

I want to be able to do as little command-trickery as possible. If I have to do it, then that means the next hotplug-capable LLDD must do it also. Duplication of code is bad -- it should all be handled in the mid-layer.

No one has to do anything of the sort if you follow what I wrote. Once the device is marked offline, any further commands already present in the request queue but not yet sent to the device plus any commands that come in to the request queue will all be sent back as I/O errors immediately before ever coming to you.

Quoted 2 lines

As yet, the interface I have to the SCSI mid-layer fails on all three points here.

SCSI <-> USB interaction is wrong right now. We know that.

Quoted 4 lines

And now, some of my opinions on how this should all work:

It would be nice if the user informed us about removing the device before they did it. But we shouldn't crash if they don't.

Correct.

Quoted 6 lines

I don't want to be hanging around after a device is gone, spinning my wheels because some other part of the kernel can't handle the fact that the device is gone. My driver is a passthru between the a SCSI emualted host and a physical USB device -- if my device is gone, I want to be out of there. (Oddly enough, I'm starting to think there may be a DoS attack here if you force the LLDD to stay -- after all, it consumes memory....)

The device will go away. But, because we clean up multiple things on a host, like an error handler thread that needs to be woke up and we have to wait for it to run and acknowledge the death, instantaneous removal of all the structs is simply unrealistic. Unplugging your internal structs from the actual bus and then letting the scsi subsys clean them up at it's leisure is possible though, and that's what I'm asking for.

Quoted 4 lines

Remember, the physical plug doesn't ask me if it's okay, and I don't get to ask the SCSI mid-layer if it's okay. Yes, starting with the user clicking to tell us would be nice, but I don't get to see that. All I get to see is an indication that the plug is pulled.

I don't disagree. But because a plug is pulled has nothing to do with objects that might hold a reference to your object. Just because you *want* it that way doesn't mean it's realistic to actually try and make it that way. We can delete an object only after the last reference is released.

Quoted 6 lines

I don't really give a rat's a** about 'how SCSI works' or how it's specified or CAM models or any of that. I try to live in the real world as much as possible. In that world, I'm not asking to remove an HBA -- I'm telling you it's been removed. I can't call it back. I can't even fake a command (other than perhaps INQUIRY) in any meaningful way. THERE IS NOTHING I CAN DO BUT KEEP INSISTING THAT THE DEVICE IS GONE!

I wouldn't suggest otherwise. I'm saying that in the real world of software design, it doesn't matter if your device is gone or not, if there are still references outstanding to the object then a free of that object is a software bug. Come on now, quit being a prick about all this because you know I'm right on that point. I'm not asking you to leave shit around forever, I'm asking you guys to work within the generic framework of object allocation and freeing that already exists in the scsi subsys to get things done. The real problem is that you guys want this clean up process to be syncronous and I'm telling you that it's ever so slightly async because we have a couple schedules that have to take place and possibly a request queue to clean out. No one seems willing to accept that a slightly async process is going to be OK. Whatever.

Quoted 4 lines

It would be nice if the user could inform various parts of the kernel that this device was going away, and then all sorts of cleanup could happen. But I really don't care -- all I'm trying to do is exit without a resource leak under all circumstances.

And so you shall if you follow the guidelines I set forth.

Quoted 6 lines

It would be nice if the SCSI mid-layer kept track of what commands were in what stages in who's queues. After all, if I hot-unplug a PCI SCSI controller, the controller really isn't going to be able to complete those commands for us -- we have to assume that commands queued by a LLDD are really just being sent to the hardware for queuing. If that wasn't the case, then having LLDD queue capability doesn't make sense.

See, now you're just being a prick again. You know the reason that the scsi mid layer *makes* the lldd tell us it is done with the commands is because if we go around saying "oh, this command was outstanding, let's free it" without getting clearance from the lldd then the lldd might still yet have state associated with the command. We can't *ASSUME* jack crap in the mid layer. You have to *TELL* us that you are done with a command. It is the only way to avoid races, resource leaks, and all sorts of other crap. It amazes me that you think you should be excused from the job of cleaning up your queues after something happens like that. Maybe the USB code doesn't allocate any internal command structs to go along with each scsi command, but pretty much every real scsi driver does and it's imperative that the driver go through all those commands and clean up after an event such as a removal, so what's the big deal about calling scsi_done() to *tell* us that you've cleaned up?

Quoted 7 lines

Now, here's the kicker -- this is what I think Linus wants:

Linus said to me, with a degree of annoyance, that he doesn't want usb-storage to keep any associations of departed devices with SCSI emulated hosts. That means that I need to be able to add and remove hosts at the will of the end-user. In the end, what drives the entire process is what Linus' hand does when it's placed on his USB flashcard reader.

Well, the few milliseconds it might take to properly clean this stuff out is well within the specifications of how fast a human can go around plugging and unplugging a flashcard reader, so I'm not concerned that my proposal doesn't meet these requirements.

Signature · 9 lines

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606
  
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

So, if I read this correctly, you're saying that the correct sequence is:

(1) get disconnect notification from USB
(2) Call scsi_set_device_offline() (must hold host lock for this)

(3) call scsi_done() for all command in queue (max: 1)

(4) Call scsi_remove_host(), which should now work because no commands are outstanding

(5) Call scsi_unregister()

And we're done, all structures can be freed. And, as I understand it, the following is true:

(a) once (2) is done, no more commands will be queued

(b) once (3) is done, (4) is guaranteed to work

(c) there is nothing the user can do to make this sequence take a long time

Tho, this does leave me with a couple of questions:

(i) Doesn't scsi_set_device_offline() work on devices, not hosts? How do I map from my host to my device list?

(ii) Do I need to call scsi_set_device_offline() for each device? I presume 'yes'.

(iii) What should I shove into the status field of the scsi command before I scsi_done() it?

Oh, and as for my being a 'prick'.... my big problem is that the documented interface is synchronous. Async is fine with me, but up until this e-mail, all I've seen is people arguing over what the sequence is, and theoretical issues of what users should and should not do. And I also think that a large number of hotplugable hosts are going to replicate a whole bunch of code to do (2)+(3)+(4) in one, synchronous burst.

If someone will step forward with a 'yes' or 'no' on this sequence, then I'll get it done. If the answer is 'no', then what did I miss?

Quoted 6 lines · Matthew Dharm

So, if I read this correctly, you're saying that the correct sequence is:

(1) get disconnect notification from USB
(2) Call scsi_set_device_offline() (must hold host lock for this)

(3) call scsi_done() for all command in queue (max: 1)

Hmmm...only 1? USB limit or driver limit?

Quoted 9 lines

(4) Call scsi_remove_host(), which should now work because no commands are outstanding

(5) Call scsi_unregister()

And we're done, all structures can be freed. And, as I understand it, the following is true:

(b) once (3) is done, (4) is guaranteed to work

No! Remember, command completion is delayed! We have a tasklet that processes your now complete command, and with that processing comes marking the device unbusy, which is also required for 4 to work. That's why I was suggesting waking up the error handler thread and letting it finish this process off. The error handler thread has the luxury of being able to wait for the command completion to happen, and in my opinion it's a slightly better place to do the work of cleaning out the request queue.

Quoted 4 lines

Tho, this does leave me with a couple of questions:

(i) Doesn't scsi_set_device_offline() work on devices, not hosts? How do I map from my host to my device list?

Well, in hosts.c::scsi_remove_host() we do it thusly:

        list_for_each_entry(sdev, &shost->my_devices, siblings)
                if (scsi_check_device_busy(sdev))
                        return 1;

Quoted 2 lines

(iii) What should I shove into the status field of the scsi command before I scsi_done() it?

Well, to force an error I always put DID_ERROR into the driver byte of the result dword, aka:

cmd->result = DID_ERROR << 16;

Quoted 5 lines

Async is fine with me, but up until this e-mail,
all I've seen is people arguing over what the sequence is, and theoretical issues of what users should and should not do. And I also think that a large number of hotplugable hosts are going to replicate a whole bunch of code to do (2)+(3)+(4) in one, synchronous burst.

Which would be wrong BTW. If you can support multiple devices behind a bridge then you can't put (2)+(3)+(4) together in one burst. That's why they aren't that way now.

Ah... the sweet feeling of progress.

Driver limit. I added support for queueing, but the queue is fixed at size
1. It's an improvement for the future.

Quoted 7 lines

No! Remember, command completion is delayed! We have a tasklet that processes your now complete command, and with that processing comes marking the device unbusy, which is also required for 4 to work. That's why I was suggesting waking up the error handler thread and letting it finish this process off. The error handler thread has the luxury of being able to wait for the command completion to happen, and in my opinion it's a slightly better place to do the work of cleaning out the request queue.

Okay... so what do I do if it fails? Sleep for a while and try again later? Wait on a flag somewhere?

Quoted 6 lines

Quoted 5 lines

Async is fine with me, but up until this e-mail,
all I've seen is people arguing over what the sequence is, and theoretical issues of what users should and should not do. And I also think that a large number of hotplugable hosts are going to replicate a whole bunch of code to do (2)+(3)+(4) in one, synchronous burst.

Quoted 3 lines

Which would be wrong BTW. If you can support multiple devices behind a bridge then you can't put (2)+(3)+(4) together in one burst. That's why they aren't that way now.

Hrm... I can see your point if we're talking about hotplugging an individual device, but I don't see how (2)+(3)+(4) isn't what we want for hotplugging an entire host.

Quoted 2 lines · Matthew Dharm

Okay... so what do I do if it fails? Sleep for a while and try again later? Wait on a flag somewhere?

Well, the better option is what I think we are working on. Instead of trying to remove the host completely, just unhook it from the USB stuff, then call the set_scsi_device_offline(), then send back any outstanding commands via scsi_done(), then possibly call the scsi_schedule_host_removal() if Mike adds that function. Then return. Don't to anything else. Take all the remaining code you would normally run at this point and put it into a function in your source called usb_release() and in your Scsi_Host_Template struct that you pass to the scsi layer, init the release pointer with the address of your usb_release() routine. That way, the scsi layer can do what it's best at, taking care of the clean up details we've been talking about, and when it's all done, it will call your usb_release() routine with a single argument of the host struct you are wanting released. At that point, you can do all the freeing you would have done in that khubd loop (at least I think that's the context you are doing the freeing from now) and know for a fact that not only are you freeing everything, but so is the scsi mid layer. I think this will solve all the issues you've had, because this *won't* leak, it won't block your other actions, and it lets the scsi subsystem clean up properly.

So, was any of this ever implemented? As far as I can tell, the required changes were:

(o) addition of scsi_schedule_host_removal() (possibly optional)
(o) implementation of scsi_set_device_offline() (possibly optional)
(o) change the behavior of the 'hotplug initialization model' to call my release function

Matt

Quoted 23 lines · Doug Ledford

Quoted 2 lines · Matthew Dharm

Okay... so what do I do if it fails? Sleep for a while and try again later? Wait on a flag somewhere?

Well, the better option is what I think we are working on. Instead of trying to remove the host completely, just unhook it from the USB stuff, then call the set_scsi_device_offline(), then send back any outstanding commands via scsi_done(), then possibly call the scsi_schedule_host_removal() if Mike adds that function. Then return. Don't to anything else. Take all the remaining code you would normally run at this point and put it into a function in your source called usb_release() and in your Scsi_Host_Template struct that you pass to the scsi layer, init the release pointer with the address of your usb_release() routine. That way, the scsi layer can do what it's best at, taking care of the clean up details we've been talking about, and when it's all done, it will call your usb_release() routine with a single argument of the host struct you are wanting released. At that point, you can do all the freeing you would have done in that khubd loop (at least I think that's the context you are doing the freeing from now) and know for a fact that not only are you freeing everything, but so is the scsi mid layer. I think this will solve all the issues you've had, because this *won't* leak, it won't block your other actions, and it lets the scsi subsystem clean up properly.

Signature · 6 lines

Matthew Dharm                              Home: [email protected] 
Maintainer, Linux USB Mass Storage Driver

We can customize our colonels.
					-- Tux
User Friendly, 12/1/1998

Willem Riede <[email protected]> suggested to me that I simply set sdev->online = 0 for scsi_set_device_offline()

Any reason that isn't good enough?

Quoted 26 lines · Matthew Dharm

So, was any of this ever implemented? As far as I can tell, the required changes were:

(o) addition of scsi_schedule_host_removal() (possibly optional)
(o) implementation of scsi_set_device_offline() (possibly optional)
(o) change the behavior of the 'hotplug initialization model' to call my release function

Matt

Quoted 17 lines

then call the set_scsi_device_offline(), then send back any outstanding commands via scsi_done(), then possibly call the scsi_schedule_host_removal() if Mike adds that function. Then return. Don't to anything else. Take all the remaining code you would normally run at this point and put it into a function in your source called usb_release() and in your Scsi_Host_Template struct that you pass to the scsi layer, init the release pointer with the address of your usb_release() routine. That way, the scsi layer can do what it's best at, taking care of the clean up details we've been talking about, and when it's all done, it will call your usb_release() routine with a single argument of the host struct you are wanting released. At that point, you can do all the freeing you would have done in that khubd loop (at least I think that's the context you are doing the freeing from now) and know for a fact that not only are you freeing everything, but so is the scsi mid layer. I think this will solve all the issues you've had, because this *won't* leak, it won't block your other actions, and it lets the scsi subsystem clean up properly.

Sorry Matthew I got side tracked on some issues for the last week. The scsi_set_device_offline(); function has not been added to any of James linux-scsi trees. You could add a ifndef in your code until we get the interface in the tree.

Would scsi_set_device_offline() do more than sdev->online = 0? It depends on the state of the device at the calling of the function.

I currently have scsi_set_device_offline() trying to do the following:
1.) set device offline and mark host in_recovery. 2.) mark all outstanding commands to be canceled and wake up error handler.
3.) flush the request queue.
4.) Once device is really offline send hotplug event.

(2) needs some changes in the error handler which are needed in other cases. This the large part of the change, but it is not directly part of your request.

(3) need a cleaner way to flush request specials off the queue . It would also be nice if there was a method to stop the incoming side of the request queue.

(4) need export of do_hotplug interface or a method to generate a call to it for an offline event.

I am working on these changes pretty much in order.

Doug suggested a scsi_schedule_host_removal(), but I thought we could just change scsi_remove_host() to handle this task unless there is a side effect that all callers would not want???.

-andmike

Quoted 32 lines · Matthew Dharm [[email protected]] wrote:

Willem Riede <[email protected]> suggested to me that I simply set sdev->online = 0 for scsi_set_device_offline()

Any reason that isn't good enough?

Quoted 26 lines · Matthew Dharm

So, was any of this ever implemented? As far as I can tell, the required changes were:

(o) addition of scsi_schedule_host_removal() (possibly optional)
(o) implementation of scsi_set_device_offline() (possibly optional)
(o) change the behavior of the 'hotplug initialization model' to call my release function

Matt

Quoted 17 lines

then call the set_scsi_device_offline(), then send back any outstanding commands via scsi_done(), then possibly call the scsi_schedule_host_removal() if Mike adds that function. Then return. Don't to anything else. Take all the remaining code you would normally run at this point and put it into a function in your source called usb_release() and in your Scsi_Host_Template struct that you pass to the scsi layer, init the release pointer with the address of your usb_release() routine. That way, the scsi layer can do what it's best at, taking care of the clean up details we've been talking about, and when it's all done, it will call your usb_release() routine with a single argument of the host struct you are wanting released. At that point, you can do all the freeing you would have done in that khubd loop (at least I think that's the context you are doing the freeing from now) and know for a fact that not only are you freeing everything, but so is the scsi mid layer. I think this will solve all the issues you've had, because this *won't* leak, it won't block your other actions, and it lets the scsi subsystem clean up properly.

-andmike
--
Michael Anderson
[email protected]

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Any updates on this? I saw some patches, but they don't seem to be in my tree (the usb tree, which is synced from Linus' tree).

People are starting to reports OOPSes to me because of this being missing....

Matt

The scsi_set_device_offline interface is part of the last patch (scsi error) I sent to linux-scsi. I updated my patch post some comments from the list, but I am working on issue with the patch before I resend.

-andmike
--
Michael Anderson
[email protected]

Quoted 8 lines · Matthew Dharm [[email protected]] wrote:

Any updates on this? I saw some patches, but they don't seem to be in my tree (the usb tree, which is synced from Linus' tree).

People are starting to reports OOPSes to me because of this being missing....

Matt

The scsi_set_device_offline interface is part of the last patch (scsi error) I sent to linux-scsi. I updated my patch post some comments from the list, but I am working on issue with the patch before I resend.

-andmike
--
Michael Anderson
[email protected]

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Okay, I see Linus has now accepted this into his tree. It should propagate to the USB development trees soon.

One question: What else is needed? We set the device offline, error/complete all pending commands, and the need to (somehow) make certain we're in a good state for calling scsi_remove_host(). How do we make that final guarantee?

There was talk that scsi_set_device_offline() would take care of that for us by waking up the error handler... there seems to be code to do that....

There was talk of using the release() function from the SCSI template to actually release resources....

So, what's the plan?

There still are a few things on the to do list, but should not effect the LLDD interface (at least this is the goal).
- scsi_request_fn needs a fix for device offline that will handle all request types.
- scsi_remove_host needs to call template release at the correct time (ref counting ??).
- need fix for offline hotplug event.
- Should do_hotplug be exported or should device states be added / fixed ??

Right... but I removed the release() function because that was marked (in the documentation) as only for the old-style drivers. So I'll need to re-introduce it -- but it looks like all it has to do is free some memory. Does that sound about right?

Quoted 4 lines · Matthew Dharm [[email protected]] wrote:

Right... but I removed the release() function because that was marked (in the documentation) as only for the old-style drivers. So I'll need to re-introduce it -- but it looks like all it has to do is free some memory. Does that sound about right?

Yes it was previously removed, but IIRC this is the direction discussed on this thread. For a idle device the release functionality could be done in the context of the scsi_remove_host call, but for a busy device we need to have this call to clean up later.

-andmike
--
Michael Anderson
[email protected]

-------------------------------------------------------
This SF.net email is sponsored by: Scholarships for Techies! Can't afford IT training? All 2003 ictp students receive scholarships. Get hands-on training in Microsoft, Cisco, Sun, Linux/UNIX, and more. www.ictp.com/training/sourceforge.asp
_______________________________________________
[email protected]
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

This was discussed, but I didn't recall a firm decision.

I'll keep my eyes open for the patch that uses release().

Matt

Quoted 27 lines · Mike Anderson

Quoted 4 lines · Matthew Dharm [[email protected]] wrote:

Right... but I removed the release() function because that was marked (in the documentation) as only for the old-style drivers. So I'll need to re-introduce it -- but it looks like all it has to do is free some memory. Does that sound about right?

Yes it was previously removed, but IIRC this is the direction discussed on this thread. For a idle device the release functionality could be done in the context of the scsi_remove_host call, but for a busy device we need to have this call to clean up later.

-andmike
--
Michael Anderson
[email protected]

-------------------------------------------------------
This SF.net email is sponsored by: Scholarships for Techies! Can't afford IT training? All 2003 ictp students receive scholarships. Get hands-on training in Microsoft, Cisco, Sun, Linux/UNIX, and more. www.ictp.com/training/sourceforge.asp
_______________________________________________
[email protected]
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

Signature · 7 lines

Matthew Dharm                              Home: [email protected] 
Maintainer, Linux USB Mass Storage Driver

C:  They kicked your ass, didn't they?
S:  They were cheating!
					-- The Chief and Stef
User Friendly, 11/19/1997

Quoted 2 lines

(i) Doesn't scsi_set_device_offline() work on devices, not hosts? How do I map from my host to my device list?

Well, in hosts.c::scsi_remove_host() we do it thusly:

        list_for_each_entry(sdev, &shost->my_devices, siblings)
                if (scsi_check_device_busy(sdev))
                        return 1;

Quoted 2 lines

(ii) Do I need to call scsi_set_device_offline() for each device? I presume 'yes'.

Yes. As people pointed out to me the reason a USB device is done as a host is because it very well may *be* a host with several devices behind it, so it must handle the multiple device scenario correctly and set all devices offline and clean up after all of them that might be behind this bridge.

So, if I read this correctly, you're saying that the correct sequence is:

(1) get disconnect notification from USB
(2) Call scsi_set_device_offline() (must hold host lock for this)
(3) call scsi_done() for all command in queue (max: 1)

(4) Call scsi_remove_host(), which should now work because no commands are outstanding
(5) Call scsi_unregister()

And we're done, all structures can be freed.

And, as I understand it, the
following is true:

(a) once (2) is done, no more commands will be queued

(b) once (3) is done, (4) is guaranteed to work

(c) there is nothing the user can do to make this sequence take a long time

Tho, this does leave me with a couple of questions:

(i) Doesn't scsi_set_device_offline() work on devices, not hosts? How do I map from my host to my device list?

(ii) Do I need to call scsi_set_device_offline() for each device? I presume 'yes'.

(iii) What should I shove into the status field of the scsi command before I scsi_done() it?

Oh, and as for my being a 'prick'.... my big problem is that the documented interface is synchronous. Async is fine with me, but up until this e-mail, all I've seen is people arguing over what the sequence is, and theoretical issues of what users should and should not do.

And I also think that a
large number of hotplugable hosts are going to replicate a whole bunch of code to do (2)+(3)+(4) in one, synchronous burst.

On Thu, Jan 23, 2003 at 08:40:41PM +0100, Oliver Neukum wrote:

No, scsi_set_device_offline() schedules the error handler thread for that host to be woken up.

Once all the commands are gone and no more are arriving, then if, and only if, someone actually removes the device from the scsi subsystem (maybe hotplug manager or something) then you will get the typical slave_destroy() call to tell you that it is safe to release all resources related to this device. Otherwise, the device will hang around as an offline device until someone does

echo "scsi-remove-single-device a b c d" >/proc/scsi/scsi
to remove it.

Am Donnerstag, 23. Januar 2003 18:46 schrieb Luben Tuikov:

Quoted 7 lines

Quoted 4 lines · Oliver Neukum

Not all the world is a SAN. USB has no possibility to even try an interaction after the device is gone. We have to handle this flexibly.

Quoted 7 lines

Thus the example in the original post. I.e. for simple transports whose portals get notified when a device is plugged off (USB), the LLDD can notify SCSI Core, by setting a state variable in scsi_device. In which case SCSI Core can answer with the proper TARGET error code. (This was outlined before, scsi_command->online:1 ...)

Very well, so you agree that the SCSI layer should export to the LLDD a function to set devices offline?

Quoted 5 lines

Quoted 5 lines

In fact, if a device
can vanish without a LLDD knowing about it, this is purely a problem of the SCSI layer.

Quoted 8 lines

No, of course not. (Think of IP.) When a device vanishes and LLDD doesn't know about it (more complicated transports), the CDB will return with the proper Service Response, since the transport(s) won't be able to deliver it. This will bubble up through SCSI Core and the error returned will have to be the same as that of the simpler transports, as outlined above.

Yes, sorry. To be precise, this means that the LLDD has to do nothing special, as it has to implement checking for a failing command anyway. But it's not entirely the same. If a command cannot be delivered it may or may not be appropriate to start error recovery. After the LLDD has told the SCSI layer that it has noticed a device going away, there must be no error recovery.

Quoted 9 lines

Quoted 5 lines

That means that we have to have a way to ensure that no more commands will reach the LLDD which can be triggered without any commands to be executed at all. This functionality has to come from the scsi mid layer.

For simple transports yes; for more complicated ones, the CDB will not be able to be delivered, and will return with error.

Good.
So the first thing a LLDD has to do after it has learned about a device being removed is to have the device block.
1. set device offline
But commands may still be in flight.IMHO it is not right to assume that all commands now in flight to a device have failed, as some may have completed successfully in time, or failed for other reasons than unplugging. So it should be the LLDD's responsibility to finish the outstanding commands. Furthermore, there's a window for commands already having passed the check for offline but not yet being noticed by the LLDD. The simplest solution is to use a waiting primitive from RCU. So we are at:

1. set device offline
2. synchronize the kernel
3. finish all pending commands

So far with me?
The LLDD could now forget about the device and be done with it. However there's a problem left. The device may come back. What happens if a device with the same ID is reconnected?

Regards
Oliver

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

The key is that the removal request should come from the top, not the bottom. If someone is stupid enough to surprise remove a device (ie:
unplug their USB SCSI device while the device is in use by the OS), they get what they deserve (I/O errors, dirty OS data, queued up requests which never shut down). If they tell the OS that the device is going to be removed, so it may flush the device and shut down I/O to the device, the request should be granted on all accounts (expected removal).

Quoted 10 lines

The key is that the removal request should come from the top, not the bottom. If someone is stupid enough to surprise remove a device (ie:
unplug their USB SCSI device while the device is in use by the OS), they get what they deserve (I/O errors, dirty OS data, queued up requests which never shut down). If they tell the OS that the device is going to be removed, so it may flush the device and shut down I/O to the device, the request should be granted on all accounts (expected removal).

Hmm? Windows and OS/X cope with this just fine.

I cant speak about OS/X, but I have crashed windows several times (BSOD) while hot removing a USB SCSI CDROM. As you will notice, when you run windows and attach a device, there is a program that is started that allows you to notify the os of the removal so that it may properly remove the device from the OS instead of it being yanked.

Thanks
-steve

Quoted 18 lines · Matthew Jacob

Quoted 10 lines

The key is that the removal request should come from the top, not the bottom. If someone is stupid enough to surprise remove a device (ie:
unplug their USB SCSI device while the device is in use by the OS), they get what they deserve (I/O errors, dirty OS data, queued up requests which never shut down). If they tell the OS that the device is going to be removed, so it may flush the device and shut down I/O to the device, the request should be granted on all accounts (expected removal).

Hmm? Windows and OS/X cope with this just fine.

Oliver and others,

In regards to hotswap, any real operating system should be _told_ that a block device is going to be removed from the top. There are several reasons.

1) File mounts should be removed from the filesystem layer
2) files accessing block devices directly should be terminated
3) raid members using that block device should be hot removed
4) I'm sure you can think of others :)

The key is that the removal request should come from the top, not the bottom. If someone is stupid enough to surprise remove a device (ie:

The device driver should not be responsible for managing hotswap in any regard. Its only purpose should be to tell the block device removal

layer that a surprise extraction was initiated such that the block device removal code can ask the mid layer drivers to shut down error correction routines to the device and dump its pending I/O queue and clean up after the device. The main advantage of this technique is

If you think about what your suggesting, your suggesting that the LLDD tells the scsi layer that the device is gone, that then times out errors and leaves the filesystem and sys_open/close file tables, and RAID layers in a state of disarray. We don't want the LLDD knowing about the RAID system and whether it should tell the RAID layer to hot remove, do we?

I want:
LLDD to SCSI: device is gone
SCSI to LLDD: Ok. I'll handle from here on.
LLDD: OK. I am gone. And won't have any contact until the next device is plugged in.

The process can be somewhat more complicated, under some conditions:
- it never fails
- it is done within a finite, bounded, reasonable time

Quoted 6 lines

I want:
LLDD to SCSI: device is gone
SCSI to LLDD: Ok. I'll handle from here on.
LLDD: OK. I am gone. And won't have any contact until the next device is plugged in.

Quoted 1 line

...

Could this time limit be fixed (or parameterized) known to all LLDDs? This would allow one to try and avoid flooding SCSI with detach/reattach events for the 'same' device.

Quoted 7 lines

Quoted 2 lines

...

Could this time limit be fixed (or parameterized) known to all LLDDs? This would allow one to try and avoid flooding SCSI with detach/reattach events for the 'same' device.

And what exactly is the "same" device? And who's keeping history about devices that have previously been attached? And, says the guy who's full of questions, didn't Linus want to get rid of such history?

I want:
LLDD to SCSI: device is gone
SCSI to LLDD: Ok. I'll handle from here on.
LLDD: OK. I am gone. And won't have any contact until the next device is plugged in.

The process can be somewhat more complicated, under some conditions:
- it never fails
- it is done within a finite, bounded, reasonable time

Regards
Oliver

Quoted 7 lines

I want:
LLDD to SCSI: device is gone
SCSI to LLDD: Ok. I'll handle from here on.
LLDD: OK. I am gone. And won't have any contact until the next device is plugged in.

The downside of this approach is that the LLDD must now be able to detect insertions and removals when it may not be able to do so. If it is able to do so, then fine, it can tell upper layers about it, but the actual control of removal of a device should occur higher up to fix several problems with the approach of having the LLDD manage the hotswap state of the device.

When the Low Level Device Driver (LLDD), being the transport portal, notices that the device is going away or has gone away from the ``fabric'' (wlg), it will fire a device-gone event with the kernel. *Not* necessarily with SCSI Core, in fact I'd rather it didn't, but with a well defined kernel entry for device-gone events.

At the same time the LLDD will start returning TARGET gone, or whatever is appropriate to newly queued commands, and error out all internally queued commands (if it does it's own queuing). (I've seen this work nicely on mount and read/write(2) and fsck.)

I.e. the ``synchronization'' has started already by the LLDD erroring out commands, new and queued.

All the while the kernel has started higher level cleaning up, decrementing ref counts, etc,

But there's no such thing as ``waiting around indefinitely'' or ``blocking wait'' as you've suggested in some of your emails.

Quoted 11 lines · Luben Tuikov

Quoted 3 lines

The way this should work is that the LLD calls scsi_remove_device(), and that cuts off the flow of commands. The LLD can promise to error-out any pending commands in the device command queue.

I take it you mean that the transport will tell the LLDD that the device is gone and it (LLDD) call the one above, SCSI Core to remove the device.

Hmm, more thinking needs to be done here, as shouldn't this be handled by hotplugging? I.e. Targets do not *initiate* events.

Not exactly, but the bus driver ("transport"?) certainly does initiate reports like "here's a new device on the bus" or "that device is gone". That's when hotplugging kicks in (both in-kernel and in-userland).

And the only way to access a device ("target") on the bus is to give a request to that bus driver. If, when servicing that request, the bus driver notices the device is gone ... that can act a lot like a device initiating a "device gone" event would look.

Quoted 2 lines

The transport can notify that the device is gone, but an ULP entity will call scsi_remove_device() not the other way around.

That's how USB works today: khubd shuts things down. Device drivers get disconnect() callbacks, just as when their modules are removed.

EXCEPT that "khubd" is part of usbcore (roughly analagous to parts of the scsi mid-layer) ... so the drivers acting as host side proxies for the target hardware ("usb device") are purely reactive. Their only roles in hotplug scenarios are to bind to devices (when a new one appears, using probe callbacks) or unbind from them (when one goes away, using disconnect callbacks).

Those disconnect() callbacks have a few key responsibilities, very much including shutting down the entire higher level I/O queue to that device. I think you're saying that SCSI drivers don't have such a responsibility (unlike USB or PCI) ... if so, that would seem to be worth changing.

David, when I said ``... the transport will tell the LLDD that the device ...'' this is *exactly* what I meant. You're just repeating it here in a more broken-down way.

By transport I mean USB, FC, SPI, etc; LLDD is the transport portal and the initiator (aka the initiator port). This terminology is not really that new, but still not that old, and described in SAM-3.

We just cannot let a transport event just wipe out a device, without consulting hotplugging first -- think security.

That is, when a transport event takes place, the LLDD doesn't have to ``run to'' SCSI Core right away. Just let the kernel know about this event, and start returning errors, on newly queued commands.

The kernel will decide what to do about this device going away, i.e. hotplugging, sysop notification, etc.

Quoted 4 lines

By transport I mean USB, FC, SPI, etc; LLDD is the transport portal and the initiator (aka the initiator port). This terminology is not really that new, but still not that old, and described in SAM-3.

I was hoping for something described in the 2.5.58 kernel docs, which only talks about LLD (Documentation/scsi) except in one case (looked like a typo) ... I remember SAM-3 as a kind of missile!

Quoted 2 lines

We just cannot let a transport event just wipe out a device, without consulting hotplugging first -- think security.

Certainly "device gone" would be an auditable event, but this is primarily an integrity issue: don't free objects until other components have stopped using them.

If any components attach security policies to that "gone" state transition, that'd be atypical but purely their own business. (Like a transport erasing session master keys ... most transports wouldn't have them, and would likely erase them as soon as the device is known to be gone, no hotplug involved.)

Quoted 5 lines

That is, when a transport event takes place, the LLDD doesn't have to ``run to'' SCSI Core right away. Just let the kernel know about this event, and start returning errors, on newly queued commands.

Quoted 2 lines

The kernel will decide what to do about this device going away, i.e. hotplugging, sysop notification, etc.

Sounds right. Except that it'd normally be the SCSI core that we "let" know about the event. (Not always, I can imagine that some transports might be able to kick in recovery procedures and find some other path for accessing the device. But in such cases, SCSI might never see the device as "gone" ... )