USB autosuspend fixes for 2.6.23-rc6
PATCH 1/1 USB autosuspend fixes for 2.6.23-rc6
It turns out that USB devices suck when it comes to powermanagement issues :(
A number of patches have been submitted near the end of this kernel release cycle that add new device ids to the quirk table in the kernel to disable autosuspend for specific devices. However, a number of developers are very worried that even with the testing that has been done, once 2.6.23 is released, we are going to get a whole raft of angry users when their devices break in nasty ways.
As an example, it seems that almost 2/3 of all USB printers just can not handle autosuspend. And there's a _lot_ of USB printers out there...
When researching how other operating systems handle this, it was found out that they use a whitelist of devices that are able to properly suspend. So, in the future, that is how we are going to handle this.
These two patches address the need today to have users machines still work, even if they might draw more power than they possibly could (which is not any more than they did in 2.6.22.)
These patches do two things:
- disable USB autosuspend on all devices except for USB hubs. This can be easily overridden by userspace to turn on autosuspend for devices that a user wants to. HAL will use a whitelist in the future for these types of devices.
- revert the usb-storage autosuspend patch. This breaks a number of devices out there that can not handle suspend properly, _AND_ the current patch is broken even for devices that do work properly under some situations. Data loss is not a good thing to have happen, so this patch is reverted for now. Oliver has more specifics about the issues involved here if anyone is curious.
The "disable autosuspend" patch has been in the -mm tree for a while, and is being shipped by Ubuntu and Red Hat in their bleeding-edge kernels in order to handle the huge number of broken devices. openSUSE will also get this patch for its next kernel release once the suse developers return from wondering around the woods of the Czech Republic next week.
Please pull from:
master.kernel.org:/pub/scm/linux/kernel/git/gregkh/usb-2.6.git/
The full patches will be sent to the linux-usb-devel mailing list, if anyone wants to see them.
thanks,
greg k-h
drivers/usb/core/quirks.c | 6 ++++++ drivers/usb/storage/scsiglue.c | 13 ++++--------- drivers/usb/storage/usb.c | 27 +++++++-------------------- 3 files changed, 17 insertions(+), 29 deletions(-)
---------------
Alan Stern (1):
USB: disable autosuspend by default for non-hubs
Greg Kroah-Hartman (1):
Revert "usb-storage: implement autosuspend"
Quoted 21 lines · Greg KH
It turns out that USB devices suck when it comes to powermanagement issues :(
A number of patches have been submitted near the end of this kernel release cycle that add new device ids to the quirk table in the kernel to disable autosuspend for specific devices. However, a number of developers are very worried that even with the testing that has been done, once 2.6.23 is released, we are going to get a whole raft of angry users when their devices break in nasty ways.
As an example, it seems that almost 2/3 of all USB printers just can not handle autosuspend. And there's a _lot_ of USB printers out there...
When researching how other operating systems handle this, it was found out that they use a whitelist of devices that are able to properly suspend. So, in the future, that is how we are going to handle this.
These two patches address the need today to have users machines still work, even if they might draw more power than they possibly could (which is not any more than they did in 2.6.22.)
Quoted 6 lines
These patches do two things:
- disable USB autosuspend on all devices except for USB hubs. This can be easily overridden by userspace to turn on autosuspend for devices that a user wants to. HAL will use a whitelist in the future for these types of devices.
...
Not related to the patch for 2.6.23, but I have a gut feeling that something might be done the wrong way later:
If I understand you correctly, you are saying that I will have to install HAL for getting a whitelist for in-kernel functionality?
It is a good thing if userspace can add currently missing devices to whitelists, but the whitelist itself should be in the kernel.
Quoted 4 lines
thanks,
greg k-h
...
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 11 lines · Adrian Bunk
Quoted 10 lines
These two patches address the need today to have users machines still work, even if they might draw more power than they possibly could (which is not any more than they did in 2.6.22.)
These patches do two things:
- disable USB autosuspend on all devices except for USB hubs. This can be easily overridden by userspace to turn on autosuspend for devices that a user wants to. HAL will use a whitelist in the future for these types of devices.
...
Quoted 5 lines
Not related to the patch for 2.6.23, but I have a gut feeling that something might be done the wrong way later:
If I understand you correctly, you are saying that I will have to install HAL for getting a whitelist for in-kernel functionality?
Your meaning isn't entirely clear. Presumably HAL will contain such a whitelist. But there's nothing to stop you from setting up your own whitelist via udev scripts, or even turning autosuspend on or off by hand.
Quoted 2 lines
It is a good thing if userspace can add currently missing devices to whitelists, but the whitelist itself should be in the kernel.
It's not clear that this sort of approach will turn out to be workable. Whitelists/blacklists do okay in the kernel when they refer to a relatively small subset of devices. However in this case I have the impression that we're talking about roughly a 50/50 split. Keeping an in-kernel list with even 10% of all existing USB devices simply isn't feasible.
Besides, is it really that much harder for userspace to modify device settings as the devices are detected than for it to modify an in-kernel whitelist just once? Don't forget about possible races: Devices may already have been detected and configured before userspace was able to modify the whitelist.
Alan Stern
Quoted 27 lines · Alan Stern
Quoted 11 lines · Adrian Bunk
Quoted 10 lines
These two patches address the need today to have users machines still work, even if they might draw more power than they possibly could (which is not any more than they did in 2.6.22.)
These patches do two things:
- disable USB autosuspend on all devices except for USB hubs. This can be easily overridden by userspace to turn on autosuspend for devices that a user wants to. HAL will use a whitelist in the future for these types of devices.
...Quoted 5 lines
Not related to the patch for 2.6.23, but I have a gut feeling that something might be done the wrong way later:
If I understand you correctly, you are saying that I will have to install HAL for getting a whitelist for in-kernel functionality?
Your meaning isn't entirely clear. Presumably HAL will contain such a whitelist. But there's nothing to stop you from setting up your own whitelist via udev scripts, or even turning autosuspend on or off by hand.
Quoted 2 lines
It is a good thing if userspace can add currently missing devices to whitelists, but the whitelist itself should be in the kernel.
Quoted 6 lines
It's not clear that this sort of approach will turn out to be workable. Whitelists/blacklists do okay in the kernel when they refer to a relatively small subset of devices. However in this case I have the impression that we're talking about roughly a 50/50 split. Keeping an in-kernel list with even 10% of all existing USB devices simply isn't feasible.
What about this is not feasible?
The amount of work for maintaining the list is the same:
No matter whether it's in-kernel or in the userspace, you need a list of working devices in some machine readable format.
Whether this gets used by the kernel, by userspace, or both, shouldn't make any difference.
Kernel image size can be a problem in some cases, but an in-kernel list doesn't have to be mandatory but could be made selectable in kconfig.
Quoted 5 lines
Besides, is it really that much harder for userspace to modify device settings as the devices are detected than for it to modify an in-kernel whitelist just once? Don't forget about possible races: Devices may already have been detected and configured before userspace was able to modify the whitelist.
Above you said "there's nothing to stop you from ... even turning autosuspend on or off by hand".
If this will already be supported race-free, which races will still be possible?
Quoted 1 line
Alan Stern
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 14 lines · Adrian Bunk
Quoted 9 lines
Quoted 2 lines
It is a good thing if userspace can add currently missing devices to whitelists, but the whitelist itself should be in the kernel.
It's not clear that this sort of approach will turn out to be workable. Whitelists/blacklists do okay in the kernel when they refer to a relatively small subset of devices. However in this case I have the impression that we're talking about roughly a 50/50 split. Keeping an in-kernel list with even 10% of all existing USB devices simply isn't feasible.
What about this is not feasible?
The amount of work for maintaining the list is the same:
Quoted 8 lines
No matter whether it's in-kernel or in the userspace, you need a list of working devices in some machine readable format.
Whether this gets used by the kernel, by userspace, or both, shouldn't make any difference.
Kernel image size can be a problem in some cases, but an in-kernel list doesn't have to be mandatory but could be made selectable in kconfig.
Well, size is one problem I had in mind. There are a _lot_ of USB devices in existence.
But mainly it's a question of maintenance and modification. Kernel developers don't really enjoy maintaining black- or whitelists of devices, together with all the work involved in sorting through the issues when somebody posts an email saying "My device doesn't work!".
Also, modifying device lists in the kernel tends to be a slow process, involving at least one kernel release cycle. It's much easier for people to maintain userspace databases. Now I realize you proposed there be a userspace interface for modifying the kernel's whitelist -- but if you're going to do that, why not put the entire whitelist in userspace to begin with?
Maybe you're concerned about propagating updates as painlessly as possible -- if the whitelist is in the kernel then every kernel release would include an update. But in userspace it's possible to do updates even more quickly and painlessly. For example, there could be a network server available for both interactive lookups and automatic queries from HAL.
Quoted 11 lines
Quoted 5 lines
Besides, is it really that much harder for userspace to modify device settings as the devices are detected than for it to modify an in-kernel whitelist just once? Don't forget about possible races: Devices may already have been detected and configured before userspace was able to modify the whitelist.
Above you said "there's nothing to stop you from ... even turning autosuspend on or off by hand".
If this will already be supported race-free, which races will still be possible?
You _can't_ change the autosuspend setting for a device that hasn't yet been detected; you can only do it after detection. But you _can_ modify a whitelist either before or after a device is detected; any such modifications won't affect the already-existing devices.
Does that answer your question?
Alan Stern
Quoted 4 lines · Alan Stern
You _can't_ change the autosuspend setting for a device that hasn't yet been detected; you can only do it after detection. But you _can_ modify a whitelist either before or after a device is detected; any such modifications won't affect the already-existing devices.
Exactly. The kernel will by default get the device up and working. Any capabilities to be changed after it is working can be done by userspace with a "whitelist" type program that can properly tell the kernel that this specific device can be safely suspended.
If you decide to do this in a small bash script, or in HAL/udevd/DeviceKit/whatever, it all doesn't matter to the kernel.
thanks,
greg k-h
Quoted 5 lines · Alan Stern
But mainly it's a question of maintenance and modification. Kernel developers don't really enjoy maintaining black- or whitelists of devices, together with all the work involved in sorting through the issues when somebody posts an email saying "My device doesn't work!".
Yeah.
In general, I think the USB blacklist/whitelists are generally a sign of some deeper bug.
We used to have a lot of those things due to simply incorrect SCSI probing, causing devices to lock up because Linux probed them with bad or unexpected modepages etc. I suspect we still have old blacklist entries from those days that just never got cleaned up, because nobody ever dared remove the blacklist entry.
We should strive to make the default behaviour be so safe that we never need a black-list (or a whitelist), and basically consider blacklists to be not a way to "fix up a device", but a way to avoid some really serious AND *RARE* error.
The moment you have lots of devices having the same blacklist entry, that's a sign that the blacklist is wrong, and the subsystem itself is likely doing something bad!
So, I would seriously suggest:
- look at USB quirks that have more than ten entries (and the entries aren't just the exact same device in various guises)
- start considering that feature to be something that is known broken, and shouldn't be done AT ALL by default.
- have some way to enable some extension on a device-by-device basis from the /sysfs interface, and then users can enable those things on their own with a graphical interface or something (or using whitelists in user space saying "ok, this device can actually do this")
- REMOVE THE DAMN QUIRK
It looks like the current situation now is that the latest autosuspend patches did basically everything but the last point.
Btw, this is in no way just an AUTOSUSPEND issue. The USB layer has a *lot* of these quirks. They are often called "UNUSUAL_DEV()", but the thing is, some of those things seem to be so usual that the naming is dubious, and thus calling it a "quirk" or "unusual" is pretty dubious too.
For example, why do we have that US_FL_MAX_SECTORS_64 at all? The fact that some USB device is broken with more than 64 sectors would seem to indicate that Windows *never* does more than 64 sectors, and that in turn means that pretty much *no* devices have ever been tested with anything bigger.
So why not make the 64 sector limit be the default? Get rid of the quirk:
we already allow people to override it in /sys if they really want to, but realistically, it's probably not going to make any difference what-so-ever for *any* normal load. So we seem to have a quirk that really doesn't buy us anything but headache.
Other quirks worth looking at (but likely unfixable) are:
- US_FL_IGNORE_RESIDUE:
Does this really matter? Can we not just always do the US_FL_IGNORE_RESIDUE thing? Windows must not be doing what we're doing.
- US_FL_FIX_CAPACITY:
This is a generic SCSI issue, not a USB one, and maybe there are better solutions to it. Are we perhaps doing something wrong? Is there some patterns we haven't seen? Why do we need this, when presumably Windows does not?
- US_FL_SINGLE_LUN:
At least a few of these seem to indicate that the real problem could be detected dynamically ("device reports Sub=ff") rather than with a quirk. Quirks are unmaintainable (and change), but noticing when devices return impossible values and going into a "safe mode" is just defensive programming.
Quoted 6 lines
Maybe you're concerned about propagating updates as painlessly as possible -- if the whitelist is in the kernel then every kernel release would include an update. But in userspace it's possible to do updates even more quickly and painlessly. For example, there could be a network server available for both interactive lookups and automatic queries from HAL.
For a lot of these things, you probably do not need a whitelist *at*all*!
IOW, just default to something safe (the 64 sector example), and then perhaps allow people to explicitly play with their settings in a hardware manager. People actually tend to *like* being able to tweak meaningless things, and it makes them feel in control. So you'd have the Gentoo people who want to optimize their iPod access times by 0.2% by raising the maximum sector number - good for them! They'll feel empowered, and if it stops working, they know it was because of something *they* did.
So at least in some cases, I think we should "default to stupid, but give users rope".
Linus
Quoted 8 lines · Linus Torvalds
In general, I think the USB blacklist/whitelists are generally a sign of some deeper bug.
We used to have a lot of those things due to simply incorrect SCSI probing, causing devices to lock up because Linux probed them with bad or unexpected modepages etc. I suspect we still have old blacklist entries from those days that just never got cleaned up, because nobody ever dared remove the blacklist entry.
I don't just suspect -- I know for a fact that we do. Partly because of laziness and partly because of not being able to verify that an entry is no longer needed.
Quoted 4 lines
We should strive to make the default behaviour be so safe that we never need a black-list (or a whitelist), and basically consider blacklists to be not a way to "fix up a device", but a way to avoid some really serious AND *RARE* error.
In general I agree. However there are some problems for which nobody has been able to come up with another solution. See below.
Quoted 15 lines
The moment you have lots of devices having the same blacklist entry, that's a sign that the blacklist is wrong, and the subsystem itself is likely doing something bad!
So, I would seriously suggest:
- look at USB quirks that have more than ten entries (and the entries aren't just the exact same device in various guises)
- start considering that feature to be something that is known broken, and shouldn't be done AT ALL by default.
- have some way to enable some extension on a device-by-device basis from the /sysfs interface, and then users can enable those things on their own with a graphical interface or something (or using whitelists in user space saying "ok, this device can actually do this")
- REMOVE THE DAMN QUIRK
Quoted 2 lines
It looks like the current situation now is that the latest autosuspend patches did basically everything but the last point.
Yes, none of those USB_QUIRK_NO_AUTOSUSPEND entries are needed any more. They can all be removed and handed over to the HAL people as a starting point.
Quoted 4 lines
Btw, this is in no way just an AUTOSUSPEND issue. The USB layer has a *lot* of these quirks. They are often called "
UNUSUAL_DEV()", but the thing is, some of those things seem to be so usual that the naming is dubious, and thus calling it a "quirk" or "unusual" is pretty dubious too.
You have concentrated your attention on the list for usb-storage, but the usbhid driver also has an impressively long quirks list.
Quoted 11 lines
For example, why do we have that
US_FL_MAX_SECTORS_64at all? The fact that some USB device is broken with more than 64 sectors would seem to indicate that Windows *never* does more than 64 sectors, and that in turn means that pretty much *no* devices have ever been tested with anything bigger.So why not make the 64 sector limit be the default? Get rid of the quirk:
we already allow people to override it in /sys if they really want to, but realistically, it's probably not going to make any difference what-so-ever for *any* normal load. So we seem to have a quirk that really doesn't buy us anything but headache.
That's true now, but it wasn't always. Until the last year or so, cdrecord wouldn't work properly with USB CD drives having a 64-sector limit unless the user added a particular command-line argument.
In fact, setting max_sectors down to 64 is probably overkill -- 120 ought to be enough. But there may have been one or two oddball devices that really did have a 32-KB limit, and better safe than sorry. At one point an engineer from Genesys said their devices did, although they do seem to work perfectly well with 64-KB transfers (and that's what Windows gives them).
Quoted 5 lines
Other quirks worth looking at (but likely unfixable) are:
-US_FL_IGNORE_RESIDUE:
Does this really matter? Can we not just always do theUS_FL_IGNORE_RESIDUEthing? Windows must not be doing what we're doing.
Windows does indeed ignore the residue field, as far as I can tell.
But this is a rather tricky thing. The USB mass-storage spec specifically says that one way a device can signal a short transfer is to pad the data with 0s to the requested length and then set the residue to indicate how much of the data is valid. If we ignore the residue then we run a risk of misinterpreting the 0s as valid data.
Now in practice this doesn't matter much because short transfers of block data (READ_10) generally involve other errors that would show up anyway, and for non-block data (MODE SENSE) the padding probably wouldn't matter. Still it seems like a dangerous sort of thing to do, which is why I have resisted it.
(And by the way, there _definitely_ are devices which use this signalling method. In fact, Linux contains a driver that does it.)
Quoted 5 lines
-
US_FL_FIX_CAPACITY:
This is a generic SCSI issue, not a USB one, and maybe there are better solutions to it. Are we perhaps doing something wrong? Is there some patterns we haven't seen? Why do we need this, when presumably Windows does not?
This is another hard case. No, we aren't doing anything wrong. If there are any patterns we haven't seen, we aren't aware of them. :-) You might think that if a device claims to have an odd number of sectors then it must be wrong, but this turns out not to be true.
Why doesn't Windows need this? For all we know, it does. Has anybody ever tried forcing Windows to read the sector beyond the end of one of these buggy devices?
For one reason or another, Linux supports filesystems/partitioning schemes which do need to access the last sector (EFI GUID, md, maybe others). Some devices are so buggy that trying to read the nonexistent "last" sector causes them to lock up, requiring a power cycle. Obviously we can't probe for this sort of behavior. (There was one report of a device which _could_ read its last sector correctly, but only if the transfer was exactly 1 sector long! Attempts to read two sectors starting from the second-to-last sector would cause it to crash.)
There's a straightforward solution: Never try to use the last sector -- in effect, assume every device has the FIX_CAPACITY flag set. Doing this universally could cause data loss, however, so again I have been opposed to it.
Quoted 6 lines
-
US_FL_SINGLE_LUN:
At least a few of these seem to indicate that the real problem could be detected dynamically ("device reports Sub=ff") rather than with a quirk. Quirks are unmaintainable (and change), but noticing when devices return impossible values and going into a "safe mode" is just defensive programming.
This is almost certainly a case where lots of the entries are no longer needed. But it isn't easy to tell which ones can safely be removed.
The problem here isn't that the device reports impossible values -- what happens is that it responds to commands for any LUN, causing the kernel to think it is really multiple devices.
Quoted 10 lines
For a lot of these things, you probably do not need a whitelist *at*all*!
IOW, just default to something safe (the 64 sector example), and then perhaps allow people to explicitly play with their settings in a hardware manager. People actually tend to *like* being able to tweak meaningless things, and it makes them feel in control. So you'd have the Gentoo people who want to optimize their iPod access times by 0.2% by raising the maximum sector number - good for them! They'll feel empowered, and if it stops working, they know it was because of something *they* did.
Quoted 2 lines
So at least in some cases, I think we should "default to stupid, but give users rope".
This will work in some cases, but not in others. In particular, it won't work when the values have to known at device-detection time. Once the sysfs files have been set up and the user can put in the correct values, it's already too late.
Still, I agree that we can get rid of MAX_SECTORS_64. Similarly, FIX_INQUIRY shouldn't be needed, since any device which does need it would fail to work under Windows. However the really popular flags are the ones which would be hardest to remove.
Alan Stern
Quoted 11 lines · Alan Stern
Quoted 8 lines · Linus Torvalds
In general, I think the USB blacklist/whitelists are generally a sign of some deeper bug.
We used to have a lot of those things due to simply incorrect SCSI probing, causing devices to lock up because Linux probed them with bad or unexpected modepages etc. I suspect we still have old blacklist entries from those days that just never got cleaned up, because nobody ever dared remove the blacklist entry.
Quoted 3 lines
I don't just suspect -- I know for a fact that we do. Partly because of laziness and partly because of not being able to verify that an entry is no longer needed.
We do have some code that looks for unneeded entries. When found by an end-user (which confirms that the entry can be removed), it asks the user to drop us an e-mail so we can remove it.
Quoted 7 lines
Quoted 4 lines
We should strive to make the default behaviour be so safe that we never need a black-list (or a whitelist), and basically consider blacklists to be not a way to "fix up a device", but a way to avoid some really serious AND *RARE* error.
In general I agree. However there are some problems for which nobody has been able to come up with another solution. See below.
We generally do strive for such a thing. Over the years, we've made several changes to the way the SCSI core works (especially in the probing department) to allow us to remove all sorts of special-case code and quirk entries.
Quoted 16 lines
Quoted 11 lines
For example, why do we have that
US_FL_MAX_SECTORS_64at all? The fact that some USB device is broken with more than 64 sectors would seem to indicate that Windows *never* does more than 64 sectors, and that in turn means that pretty much *no* devices have ever been tested with anything bigger.So why not make the 64 sector limit be the default? Get rid of the quirk:
we already allow people to override it in /sys if they really want to, but realistically, it's probably not going to make any difference what-so-ever for *any* normal load. So we seem to have a quirk that really doesn't buy us anything but headache.That's true now, but it wasn't always. Until the last year or so, cdrecord wouldn't work properly with USB CD drives having a 64-sector limit unless the user added a particular command-line argument.
Quoted 6 lines
In fact, setting
max_sectorsdown to 64 is probably overkill -- 120 ought to be enough. But there may have been one or two oddball devices that really did have a 32-KB limit, and better safe than sorry. At one point an engineer from Genesys said their devices did, although they do seem to work perfectly well with 64-KB transfers (and that's what Windows gives them).
It's worth pointing out that performance drops like a stone as this number goes down.
Quoted 14 lines
Quoted 5 lines
Other quirks worth looking at (but likely unfixable) are:
-US_FL_IGNORE_RESIDUE:
Does this really matter? Can we not just always do theUS_FL_IGNORE_RESIDUEthing? Windows must not be doing what we're doing.Windows does indeed ignore the residue field, as far as I can tell.
But this is a rather tricky thing. The USB mass-storage spec specifically says that one way a device can signal a short transfer is to pad the data with 0s to the requested length and then set the residue to indicate how much of the data is valid. If we ignore the residue then we run a risk of misinterpreting the 0s as valid data.
Quoted 8 lines
Now in practice this doesn't matter much because short transfers of block data (
READ_10) generally involve other errors that would show up anyway, and for non-block data (MODE SENSE) the padding probably wouldn't matter. Still it seems like a dangerous sort of thing to do, which is why I have resisted it.(And by the way, there _definitely_ are devices which use this signalling method. In fact, Linux contains a driver that does it.)
I think this last point is key. I'm unwilling to sacrifice error detection on properly working devices to enable error-prone use on clearly buggy devices.
Quoted 6 lines
Quoted 5 lines
-
US_FL_FIX_CAPACITY:
This is a generic SCSI issue, not a USB one, and maybe there are better solutions to it. Are we perhaps doing something wrong? Is there some patterns we haven't seen? Why do we need this, when presumably Windows does not?
Quoted 8 lines
This is another hard case. No, we aren't doing anything wrong. If there are any patterns we haven't seen, we aren't aware of them. :-) You might think that if a device claims to have an odd number of sectors then it must be wrong, but this turns out not to be true.
Why doesn't Windows need this? For all we know, it does. Has anybody ever tried forcing Windows to read the sector beyond the end of one of these buggy devices?
As far as I know, Windows doesn't need this because of the way FAT and NTFS work. They never use the end of the disk (by more than a few sectors, or so I'm told).
Quoted 4 lines
There's a straightforward solution: Never try to use the last sector -- in effect, assume every device has the
FIX_CAPACITYflag set. Doing this universally could cause data loss, however, so again I have been opposed to it.
I agree here.
Quoted 9 lines
Quoted 6 lines
-
US_FL_SINGLE_LUN:
At least a few of these seem to indicate that the real problem could be detected dynamically ("device reports Sub=ff") rather than with a quirk. Quirks are unmaintainable (and change), but noticing when devices return impossible values and going into a "safe mode" is just defensive programming.This is almost certainly a case where lots of the entries are no longer needed. But it isn't easy to tell which ones can safely be removed.
I've been meaning to start sending e-mails to see if we can get rid of these. Most of the devices which required it were UFI, which reports "LUN not present" in a goofy way. We fixed the code to detect it properly, but there are still quite a few devices out there that don't implement the correct (if goofy) method.
Most of those entries (which are for UFI devices) can go, if we get a volunteer to take the e-mail addresses listed in unusual_devs.h and work the list.
Matt
Signature · 7 lines
Matthew Dharm Home: [email protected] Maintainer, Linux USB Mass Storage Driver Sir, for the hundreth time, we do NOT carry 600-round boxes of belt-fed suction darts! -- Salesperson to Greg User Friendly, 12/30/1997
Quoted 9 lines · Matthew Dharm
Quoted 6 lines
In fact, setting
max_sectorsdown to 64 is probably overkill -- 120 ought to be enough. But there may have been one or two oddball devices that really did have a 32-KB limit, and better safe than sorry. At one point an engineer from Genesys said their devices did, although they do seem to work perfectly well with 64-KB transfers (and that's what Windows gives them).It's worth pointing out that performance drops like a stone as this number goes down.
Does anybody have good performance figures, using a high-quality device? This is the sort of thing where it helps to have some real numbers.
Alan Stern
Quoted 7 lines · Alan Stern
Quoted 5 lines
Btw, this is in no way just an AUTOSUSPEND issue. The USB layer has a *lot* of these quirks. They are often called "
UNUSUAL_DEV()", but the thing is, some of those things seem to be so usual that the naming is dubious, and thus calling it a "quirk" or "unusual" is pretty dubious too.You have concentrated your attention on the list for usb-storage, but the usbhid driver also has an impressively long quirks list.
This is true. First, there are HID_QUIRK_RDESC_* quirks, which we use to fix up badly broken report descriptor of certain devices before it enters the HID parser. There is nothing much we can do about them - these are just workarounds for really broken devices that don't follow the HID specification, but we can easily fix the things on the fly. Usually, these devices are handled in Windows by specialized driver, but if we fix the descriptor, we can use usbhid to handle them.
For the rest of the HID quirks -- most of them are also workarounds for broken classess of devices. Usually, they claim to be HID-compliant device in their file descriptor, but they do not follow the spec (they send inverted axes values, send usage codes that violate the specification, etc), but we can easily work around these bugs by a few lines of code and let the devices to be handled by usbhid flawlessly. I guess this is worth it.
Probably the only quirk entry that might be considered to be removed is HID_QUIRK_NOGET, I'd say.
Thanks,
Signature · 1 line
Jiri Kosina
Quoted 7 lines · Jiri Kosina
For the rest of the HID quirks -- most of them are also workarounds for broken classess of devices. Usually, they claim to be HID-compliant device in their file descriptor, but they do not follow the spec (they send inverted axes values, send usage codes that violate the specification, etc), but we can easily work around these bugs by a few lines of code and let the devices to be handled by usbhid flawlessly. I guess this is worth it.
The reason for this is that the only way to write a userspace "driver" for USB devices on Windows versions prior to Vista was to be a HID device. Your userspace program could then easily grab the device and control it.
This could be fixed in Linux by providing a way to have driver "heirachy" for USB whereby a vendor/product id providing driver would take a higher priority than a class driver.
But we've been talking about that for over 7 years now :)
thanks,
greg k-h
Quoted 5 lines · Linus Torvalds
So why not make the 64 sector limit be the default? Get rid of the quirk:
we already allow people to override it in /sys if they really want to, but realistically, it's probably not going to make any difference what-so-ever for *any* normal load. So we seem to have a quirk that really doesn't buy us anything but headache.
Well, ub does that today. And there is a measurable performance differential with usb-storage when driving rotating discs, or so I heard.
Quoted 5 lines
Other quirks worth looking at (but likely unfixable) are:
-US_FL_IGNORE_RESIDUE:
Does this really matter? Can we not just always do theUS_FL_IGNORE_RESIDUEthing? Windows must not be doing what we're doing.
I'm afraid this is valuable. However, a number of devices only return garbage as residue if the transfer length is greater than 32KB. Limiting that would trim this blacklist, I think. The vast majority of devices work correctly in this regard, and ub checks the residue without any blacklist.
Quoted 5 lines
-
US_FL_FIX_CAPACITY:
This is a generic SCSI issue, not a USB one, and maybe there are better solutions to it. Are we perhaps doing something wrong? Is there some patterns we haven't seen? Why do we need this, when presumably Windows does not?
It has something to do with the way our partition detection works. Linux tends to rely on the reported device size. Windows reads the first block and then goes further based on its contents. If we exterminate partitioning code which uses the reported device size for autodetection, then this problem will fix itself.
Quoted 6 lines
-
US_FL_SINGLE_LUN:
At least a few of these seem to indicate that the real problem could be detected dynamically ("device reports Sub=ff") rather than with a quirk. Quirks are unmaintainable (and change), but noticing when devices return impossible values and going into a "safe mode" is just defensive programming.
This is being worked upon. The recent change for floppies eliminated a big number of those.
-- Pete
Quoted 19 lines · Alan Stern
Quoted 17 lines · Adrian Bunk
Quoted 9 lines
Quoted 2 lines
It is a good thing if userspace can add currently missing devices to whitelists, but the whitelist itself should be in the kernel.
It's not clear that this sort of approach will turn out to be workable. Whitelists/blacklists do okay in the kernel when they refer to a relatively small subset of devices. However in this case I have the impression that we're talking about roughly a 50/50 split. Keeping an in-kernel list with even 10% of all existing USB devices simply isn't feasible.
What about this is not feasible?
The amount of work for maintaining the list is the same:
No matter whether it's in-kernel or in the userspace, you need a list of working devices in some machine readable format.
Quoted 8 lines
Quoted 5 lines
Whether this gets used by the kernel, by userspace, or both, shouldn't make any difference.
Kernel image size can be a problem in some cases, but an in-kernel list doesn't have to be mandatory but could be made selectable in kconfig.
Well, size is one problem I had in mind. There are a _lot_ of USB devices in existence.
Kernel image size matters much for some uses, but not for all.
Quoted 11 lines
But mainly it's a question of maintenance and modification. Kernel developers don't really enjoy maintaining black- or whitelists of devices, together with all the work involved in sorting through the issues when somebody posts an email saying "My device doesn't work!".
Also, modifying device lists in the kernel tends to be a slow process, involving at least one kernel release cycle. It's much easier for people to maintain userspace databases. Now I realize you proposed there be a userspace interface for modifying the kernel's whitelist -- but if you're going to do that, why not put the entire whitelist in userspace to begin with?
No matter whether the list is in userspace or in the kernel, maintaining it is exactly the same job of adding entries into some machine readable list (no matter whether it's a C struct or a CSV list that later gets converted).
This could be the kernel developers or some external sf project that gets synced with the kernel during each merge window.
Quoted 7 lines
Maybe you're concerned about propagating updates as painlessly as possible -- if the whitelist is in the kernel then every kernel release would include an update. But in userspace it's possible to do updates even more quickly and painlessly. For example, there could be a network server available for both interactive lookups and automatic queries from HAL.
...
No, what I'm concerned about is that this would require userspace for something that is completely in-kernel.
Quoted 1 line
Alan Stern
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 10 lines · Adrian Bunk
Quoted 7 lines
Maybe you're concerned about propagating updates as painlessly as possible -- if the whitelist is in the kernel then every kernel release would include an update. But in userspace it's possible to do updates even more quickly and painlessly. For example, there could be a network server available for both interactive lookups and automatic queries from HAL.
...No, what I'm concerned about is that this would require userspace for something that is completely in-kernel.
But it isn't completely in-kernel. It is policy, and policy belongs in userspace.
The kernel provides the mechanism for autosuspending idle devices and setting the idle-delay length. But userspace must be responsible for deciding what devices to autosuspend and how long the idle delays should be.
Alan Stern
Quoted 3 lines · Adrian Bunk
No, what I'm concerned about is that this would require userspace for something that is completely in-kernel.
If done right (and autosuspend now is), there is no "required" userspace.
If you want autosuspend, you just say so. The kernel doesn't do it by default. This is not about "user space required" - it's about "user space can ask for it if it wants to".
Notice? There doesn't even have to be any blacklists/whitelists at all. It really can be just an application that allows the user to check or uncheck the capability (with a warning saying something like: "Some USB devices may disconnect when suspended - if this affects you, uncheck this").
That's why the kernel shouldn't set policy. It's a *good* thing to just expose the capabilities, but not necessarily use them!
Linus
Quoted 13 lines · Linus Torvalds
Quoted 3 lines · Adrian Bunk
No, what I'm concerned about is that this would require userspace for something that is completely in-kernel.
If done right (and autosuspend now is), there is no "required" userspace.
If you want autosuspend, you just say so. The kernel doesn't do it by default. This is not about "user space required" - it's about "user space can ask for it if it wants to".
Quoted 7 lines
Notice? There doesn't even have to be any blacklists/whitelists at all. It really can be just an application that allows the user to check or uncheck the capability (with a warning saying something like: "Some USB devices may disconnect when suspended - if this affects you, uncheck this").
That's why the kernel shouldn't set policy. It's a *good* thing to just expose the capabilities, but not necessarily use them!
What is not policy is the blacklist or whitelist information.
And I'm also a bit concerned why "is policy" is that much a reason against setting *reasonable default policies* without requiring the user to do various things in userspace.
Especially since this creates some nasty interdependencies between the kernel and userspace.
And as an example, couldn't you equally say it's wrong that the kernel enables DMA on disks instead of leaving it to userspace?
We've already seen the udev disaster where upgrading from Debian 3.1 to Debian 4.0 means upgrading from kernel 2.6.8 to 2.6.18 with the udev version in Debian 3.1 not supporting kernel 2.6.18 and the udev version in Debian 4.0 not supporting kernel 2.6.8, and I don't have a good feeling about outsourcing more and more things to userspace tools not distributed with the kernel.
Quoted 1 line
Linus
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 34 lines · Adrian Bunk
Quoted 13 lines · Linus Torvalds
Quoted 3 lines · Adrian Bunk
No, what I'm concerned about is that this would require userspace for something that is completely in-kernel.
If done right (and autosuspend now is), there is no "required" userspace.
If you want autosuspend, you just say so. The kernel doesn't do it by default. This is not about "user space required" - it's about "user space can ask for it if it wants to".
Quoted 7 lines
Notice? There doesn't even have to be any blacklists/whitelists at all. It really can be just an application that allows the user to check or uncheck the capability (with a warning saying something like: "Some USB devices may disconnect when suspended - if this affects you, uncheck this").
That's why the kernel shouldn't set policy. It's a *good* thing to just expose the capabilities, but not necessarily use them!
What is not policy is the blacklist or whitelist information.
And I'm also a bit concerned why "is policy" is that much a reason against setting *reasonable default policies* without requiring the user to do various things in userspace.
Especially since this creates some nasty interdependencies between the kernel and userspace.
And as an example, couldn't you equally say it's wrong that the kernel enables DMA on disks instead of leaving it to userspace?
Quoted 6 lines
We've already seen the udev disaster where upgrading from Debian 3.1 to Debian 4.0 means upgrading from kernel 2.6.8 to 2.6.18 with the udev version in Debian 3.1 not supporting kernel 2.6.18 and the udev version in Debian 4.0 not supporting kernel 2.6.8, and I don't have a good feeling about outsourcing more and more things to userspace tools not distributed with the kernel.
Let me paraphrase the latter:
Given a distribution shipping with kernel 2.6.23 released in 2007.
What is the maximum amount of userspace I might have to upgrade if I'll want to use kernel 2.6.43 released in 2011 (sic) with this distribution?
E.g. when looking at the reverse dependencies of libhal, it would not be funny if kernel 2.6.43 required a more recent version of HAL.
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 3 lines · Adrian Bunk
E.g. when looking at the reverse dependencies of libhal, it would not be funny if kernel 2.6.43 required a more recent version of HAL.
Adrian, what's the point of answering your questions, WHEN YOU DON'T EVEN READ THE ANSWERS!
So stop repeating your inane question, and instead LISTEN to what people are saying.
There is *no* requirement what-so-ever on user space. End of story.
If user-space doesn't tell devices to go to autosuspend, they just won't do so. But this has nothing to do with white-lists, black-lists, 2.6.43, or your HAL version.
That black-listing was a mistake. We got over it. Now PLEASE you get over it too!
Linus
Quoted 6 lines · Linus Torvalds
Quoted 1 line · Adrian Bunk
E.g. when looking at the reverse dependencies of libhal, it would not be funny if kernel 2.6.43 required a more recent version of HAL.
Adrian, what's the point of answering your questions, WHEN YOU DON'T EVEN READ THE ANSWERS!
Quoted 7 lines
So stop repeating your inane question, and instead LISTEN to what people are saying.
There is *no* requirement what-so-ever on user space. End of story.
If user-space doesn't tell devices to go to autosuspend, they just won't do so. But this has nothing to do with white-lists, black-lists, 2.6.43, or your HAL version.
That black-listing was a mistake. We got over it. Now PLEASE you get over it too!
Adrian, please note that this is NOT the normal "suspend to ram/disk" stuff
that is being discussed / affected here. But rather something brand new (to Linux), the automatic powerdown (well, nearly) of idle USB devices during regular running.
Things will work just fine without chewing up gobs of memory in a futile effort to list everything inside the kernel. Devices will plug in and work fine (which they didn't in 2.6.23 until this stuff got reverted) the same as always.
To gain the last xx% in power savings, some userspace support may be needed, that's all. And with the pace of new/updated USB devices coming/going these days, userspace is really the only appropriate place for this kind of stuff.
Cheers
Quoted 15 lines · Mark Lord
Quoted 13 lines · Linus Torvalds
Quoted 2 lines · Adrian Bunk
E.g. when looking at the reverse dependencies of libhal, it would not be funny if kernel 2.6.43 required a more recent version of HAL.
Adrian, what's the point of answering your questions, WHEN YOU DON'T EVEN READ THE ANSWERS!
So stop repeating your inane question, and instead LISTEN to what people are saying.
There is *no* requirement what-so-ever on user space. End of story. If user-space doesn't tell devices to go to autosuspend, they just won't do so. But this has nothing to do with white-lists, black-lists, 2.6.43, or your HAL version.
That black-listing was a mistake. We got over it. Now PLEASE you get over it too!
Quoted 4 lines
Adrian, please note that this is NOT the normal "suspend to ram/disk" stuff that is being discussed / affected here. But rather something brand new (to Linux), the automatic powerdown (well, nearly) of idle USB devices during regular running.
I am quite aware of this fact.
Quoted 5 lines
Things will work just fine without chewing up gobs of memory in a futile effort
to list everything inside the kernel. Devices will plug in and work fine (which they didn't in 2.6.23 until this stuff got reverted) the same as always.
Space saving has it's place, but there's also the point that for the normal desktop usage a few 100 kB of data wouldn't matter.
Quoted 5 lines
To gain the last xx% in power savings, some userspace support may be needed,
that's all. And with the pace of new/updated USB devices coming/going these days,
userspace is really the only appropriate place for this kind of stuff.
It creates one more userspace _ABI_ for something that doesn't necessarily need one.
Perhaps I'm wrong and people have learnt the lesson from the udev disaster that ABIs used by stuff like udev or HAL are _userspace ABIs_ and have to be treated accordingly.
Quoted 1 line
Cheers
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 7 lines · Adrian Bunk
Let me paraphrase the latter:
Given a distribution shipping with kernel 2.6.23 released in 2007.
What is the maximum amount of userspace I might have to upgrade if I'll want to use kernel 2.6.43 released in 2011 (sic) with this distribution?
I can't speak for other parts of the kernel or userspace. Concerning USB autosuspend, you should not have to upgrade anything.
You might end up getting better power savings if you did upgrade, but failing to upgrade wouldn't make your power usage any worse.
Alan Stern
Quoted 1 line · Adrian Bunk
What is not policy is the blacklist or whitelist information.
That's debatable, but never mind.
Quoted 3 lines
And I'm also a bit concerned why "is policy" is that much a reason against setting *reasonable default policies* without requiring the user to do various things in userspace.
"Don't autosuspend" _is_ a reasonable default policy. It's what Linux has been doing all along.
Quoted 2 lines
Especially since this creates some nasty interdependencies between the kernel and userspace.
No it doesn't. If the kernel fails to provide the mechanism there's nothing that userspace can do about it anyway. If the kernel does provide the mechanism and userspace ignores it, then things continue to work the same as they always have.
Quoted 2 lines
And as an example, couldn't you equally say it's wrong that the kernel enables DMA on disks instead of leaving it to userspace?
It's not analogous. Autosuspend involves setting an idle-delay time, which depends on the type of device and how it is being used. DMA doesn't involve anything like that.
But forget about the delay for the moment. If there were a high percentage of disk drives which would fail when DMA was enabled, then yes, the decision should be left to userspace. If the percentage is low enough (and shrinking as vendors become more clueful) then moving the decision into the kernel is acceptable. At the moment USB autosuspend is not in this position, as we have learned painfully.
Quoted 6 lines
We've already seen the udev disaster where upgrading from Debian 3.1 to Debian 4.0 means upgrading from kernel 2.6.8 to 2.6.18 with the udev version in Debian 3.1 not supporting kernel 2.6.18 and the udev version in Debian 4.0 not supporting kernel 2.6.8, and I don't have a good feeling about outsourcing more and more things to userspace tools not distributed with the kernel.
This isn't a question of not being able to boot, like the udev fiasco; this is a question of saving a few watts of power. While it may be important for laptop users, it is not critical.
Alan Stern