Skip to contentSkip to conversation

Top 10 bugs/warnings for the week of March 23rd, 2008

DISCUSSION Arjan van de Ven 12 participants 29 messages May 23, 2008 – May 27, 2008

lore
Share & tools
lore
#0 lore raw

The http://www.kerneloops.org website collects kernel oops and warning reports from various mailing lists and bugzillas as well as with a client users can install to auto-submit oopses. Below is a top 10 list of the oopses collected in the last 7 days. (Reports prior to 2.6.23 have been omitted in collecting the top 10)

This week, a total of 3192 oopses and warnings have been reported, compared to 1503 reports in the previous week.

Per file statistics
[I'd love to borrow Linus' gitstat stuff for this to get a nicer presentation of the per file/directory data] 1267 kernel/sysctl.c
283 fs/sysfs/dir.c
266 fs/buffer.c
111 lib/iomap.c
81 kernel/spinlock.c
51 mm/page_alloc.c
49 net/mac80211/main.c
38 kernel/irq/manage.c
38 security/selinux/hooks.c
34 net/core/sock.c

Rank 1: __register_sysctl_paths
Reported 1260 times (2491 total reports)
[tainted] Duplicate /proc registration. Bug in the madwifi driver (Occasionally seen in the parport driver)
This oops was last seen in version 2.6.25.4, and first seen in 2.6.25-rc3. More info: http://www.kerneloops.org/searchweek.php?search=__register_sysctl_paths

Rank 2: sysfs_add_one
Reported 264 times (528 total reports)
Duplicate sysfs registration... mostly a GregKH bug ...
This oops was last seen in version 2.6.26-rc3, and first seen in 2.6.24-rc6. More info: http://www.kerneloops.org/searchweek.php?search=sysfs_add_one

Rank 3: mark_buffer_dirty
Reported 253 times (547 total reports)
EXT3 bug while hot-removing a USB device
This oops was last seen in version 2.6.25.3, and first seen in 2.6.24-rc6. More info: http://www.kerneloops.org/searchweek.php?search=mark_buffer_dirty

Rank 4: bad_io_access
Reported 111 times (153 total reports)
Bug with pata_isapnp causing another ata driver to go SPLAT Patch available in http://bugzilla.kernel.org/show_bug.cgi?id=10752
This oops was last seen in version 2.6.25.3, and first seen in 2.6.24. More info: http://www.kerneloops.org/searchweek.php?search=bad_io_access

Rank 5: _spin_unlock_irqrestore
Reported 80 times (207 total reports)
Seems to happen mostly around the idle loop.. weak voltage regulators? This oops was last seen in version 2.6.25.3, and first seen in 2.6.22-rc1. More info: http://www.kerneloops.org/searchweek.php?search=_spin_unlock_irqrestore

Rank 6: ieee80211_stop_tx_ba_session
Reported 49 times (104 total reports)
A bug in the iwl4965 driver
This oops was last seen in version 2.6.25.3, and first seen in 2.6.25-rc7-git6. More info: http://www.kerneloops.org/searchweek.php?search=ieee80211_stop_tx_ba_session

Rank 7: __alloc_pages
Reported 49 times (79 total reports)
This oops was last seen in version 2.6.25.3, and first seen in 2.6.18-rc1. Bugs in the sata_nv and velocity drivers.
sata_nv patch available at http://lkml.org/lkml/2008/5/20/604
More info: http://www.kerneloops.org/searchweek.php?search=__alloc_pages

Rank 8: set_irq_wake
Reported 38 times (43 total reports)
[fixed] Bug in serial_core.c where disable_irq_wake/enable_irq_wake were unbalanced Fix available at http://lkml.org/lkml/2008/5/20/218 (and in -mm) This oops was last seen in version 2.6.25.3, and first seen in 2.6.25-rc9-git1. More info: http://www.kerneloops.org/searchweek.php?search=set_irq_wake

Rank 9: task_has_capability
Reported 34 times
[tainted] Bug in the proprietary firegl driver
Oops only shows up in tainted kernels
This oops was last seen in version 2.6.25.3, and first seen in 2.6.25. More info: http://www.kerneloops.org/searchweek.php?search=task_has_capability

Rank 10: sk_free
Reported 29 times (109 total reports)
[tainted] VMWare driver bug
Oops only shows up in tainted kernels
This oops was last seen in version 2.6.25.4, and first seen in 2.6.23.9. More info: http://www.kerneloops.org/searchweek.php?search=sk_free

#1 lore raw

Re: Top 10 bugs/warnings for the week of May 23rd, 2008

Arjan van de Ven wrote:

eh make that May 23rd

#2 lore raw
In reply to Arjan van de Ven
Quoted 4 lines · Arjan van de Ven

Per file statistics
[I'd love to borrow Linus' gitstat stuff for this to get a nicer presentation of the per file/directory data]

The algorithm is very simple. Just sort your filenames alphabetically, and then you can do it with a simple 40-line recursive function and a trivial data structure. See the git sources, diff.c: gather_dirstat().

Or just do "git show 7df7c019c2a46672c12a11a45600cdc698e03029" in git to show the commit that introduces --dirstat.

(In fact, much of the dirstat code is the thing that turns it into percentages, so it has some setup code that first calculates the total number of changes, and the printout code spends effort in generating the percentage (well, permille) and not showing insignificant stuff - whether you'd want/need that for this is debatable)

Quoted 7 lines

Rank 1: __register_sysctl_paths
Reported 1260 times (2491 total reports)
[tainted] Duplicate /proc registration. Bug in the madwifi driver (Occasionally seen in the parport driver)
This oops was last seen in version 2.6.25.4, and first seen in 2.6.25-rc3. More info:
http://www.kerneloops.org/searchweek.php?search=__register_sysctl_paths

Btw, can you try to call these warnings, not oopses? It's not an oops, and it's not even reported as an oops in the overviews on the top-level things on the web-site, so your scripts do know it's not an oops - but then in this summary and in the "detailed information" reports it's called an oops again.

It's a WARN_ON, and yeah, while they can be bad, it's still different from an actual oops.

Linus

#3 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Arjan van de Ven

Per file statistics
[I'd love to borrow Linus' gitstat stuff for this to get a nicer presentation of the per file/directory data]

The algorithm is very simple. Just sort your filenames alphabetically, and then you can do it with a simple 40-line recursive function and a trivial data structure. See the git sources, diff.c: gather_dirstat().

Or just do "git show 7df7c019c2a46672c12a11a45600cdc698e03029" in git to show the commit that introduces --dirstat.

(In fact, much of the dirstat code is the thing that turns it into percentages, so it has some setup code that first calculates the total number of changes, and the printout code spends effort in generating the percentage (well, permille) and not showing insignificant stuff - whether you'd want/need that for this is debatable)

ok thanks I'll take a look at this; I might have to convert it to php (sigh ;)

Quoted 9 lines
Quoted 8 lines

Rank 1: __register_sysctl_paths
Reported 1260 times (2491 total reports)
[tainted] Duplicate /proc registration. Bug in the madwifi driver (Occasionally seen in the parport driver)
This oops was last seen in version 2.6.25.4, and first seen in 2.6.25-rc3. More info:
http://www.kerneloops.org/searchweek.php?search=__register_sysctl_paths

Quoted 4 lines

Btw, can you try to call these warnings, not oopses? It's not an oops, and it's not even reported as an oops in the overviews on the top-level things on the web-site, so your scripts do know it's not an oops - but then in this summary and in the "detailed information" reports it's called an oops again.

It's a WARN_ON, and yeah, while they can be bad, it's still different from an actual oops.

I'll see how to do this; the complex case of "some are oopses some are warn_ons" I probably can just deal with by prioritization.
(but I do track the "class" so the info is there in the database)

#4 lore raw
In reply to Arjan van de Ven
Quoted 3 lines

Rank 4: bad_io_access
Reported 111 times (153 total reports)
Bug with pata_isapnp causing another ata driver to go SPLAT

Bug in libata-sff and one that really proved the value of the oops work you did. Nobody ever reported it in bugzilla in a way that made it apparent what was up (I guess as it was an install crash) and probably 99% of users hitting it wouldn't even have thought their soundcard had an unused, and possible unwired, IDE port.

#5 lore raw
In reply to Arjan van de Ven
Quoted 6 lines · * Arjan van de Ven ([email protected]) wrote:

Rank 9: task_has_capability
Reported 34 times
[tainted] Bug in the proprietary firegl driver
Oops only shows up in tainted kernels
This oops was last seen in version 2.6.25.3, and first seen in 2.6.25. More info: http://www.kerneloops.org/searchweek.php?search=task_has_capability

looking at first one: http://www.kerneloops.org/raw.php?rawid=13598&msgid=

OK, aside of the obvious (their problem):

  Tainted: P
  EIP is at task_has_capability+0x48/0x76
  Code: ... <0f> 0b 
  	    ^^^^^^^
	    BUG()

This should be listed under the BUG/BUG_ON category as opposed to oops, no? Also, I think the raw data is missing some bit. Where is the:

kernel BUG at...

At any rate, they have a bug in their proprietary module (news at 11).

So, I don't think this should make the top ten. Do you have a way to sort tainted vs non-tainted, and only produce the top ten for untainted?

And one last question re: the stats. Is there a way to tell if the 41 times this was reported are from 41 distinct users. Is there any unique cookie you receive with the raw oops report that can help filter out duplicates (by duplicate I mean a user w/ this proprietary driver and rebooting is likely to reproduce the same info on each boot). You don't want to drop dups, but at least let that info the stats or something.

For the record, that bug triggers:

printk(KERN_ERR "SELinux: out of range capability %d\n", cap); BUG();

meaning they are passing in a capability that's > 63 (2.6.25 introduced 64 bit caps).

BTW, EAX: 00000030 (48)...that suggests their capability they passed in was quite large, likely an address or smth.

"<3>SELinux: out of range capability \n" <-- 38 chars that leaves 10 for %d, which is > 999,999,999 ;-)

thanks,
-chris

#6 lore raw
Quoted 4 lines · Chris Wright
Quoted 3 lines · * Arjan van de Ven ([email protected]) wrote:

Rank 9: task_has_capability
Reported 34 times
[tainted] Bug in the proprietary firegl driver

        ^^^^^^^^^
Quoted 1 line
Quoted 1 line

Oops only shows up in tainted kernels

                              ^^^^^^^^
Quoted 7 lines
Quoted 2 lines

This oops was last seen in version 2.6.25.3, and first seen in 2.6.25. More info: http://www.kerneloops.org/searchweek.php?search=task_has_capability

looking at first one: http://www.kerneloops.org/raw.php?rawid=13598&msgid=

OK, aside of the obvious (their problem):

Quoted 7 lines
  Tainted: P
  EIP is at task_has_capability+0x48/0x76
  Code: ... <0f> 0b 
  	    ^^^^^^^
	    BUG()

This should be listed under the BUG/BUG_ON category as opposed to oops, no?

yeah it should; Linus pointed that out and I've since fixed my report generator script

Quoted 4 lines

Also, I think the raw data is missing some bit. Where is the:

kernel BUG at...

hmm it ought to be there.

Quoted 4 lines

At any rate, they have a bug in their proprietary module (news at 11).

So, I don't think this should make the top ten. Do you have a way to sort tainted vs non-tainted, and only produce the top ten for untainted?

yes absolutely; this is a question I'll have for the customers of the data... do people want to see "only-tainted" in these top 10s? Right now I mark them as such but leave them in. It's trivial for me to just leave them out instead (the info is there, just a matter of not counting)

Quoted 7 lines

And one last question re: the stats. Is there a way to tell if the 41 times this was reported are from 41 distinct users. Is there any unique cookie you receive with the raw oops report that can help filter out duplicates (by duplicate I mean a user w/ this proprietary driver and rebooting is likely to reproduce the same info on each boot). You don't want to drop dups, but at least let that info the stats or something.

There is no unique per-system ID yet; I'm working with the SMOLT guys to get this added potentially.

Quoted 11 lines

For the record, that bug triggers:

printk(KERN_ERR "SELinux: out of range capability %d\n", cap); BUG();

meaning they are passing in a capability that's > 63 (2.6.25 introduced 64 bit caps).

BTW, EAX: 00000030 (48)...that suggests their capability they passed in was quite large, likely an address or smth.

yeah iirc the AMD graphics driver gives the user process full root caps for some time... Annoying things these non-root linux users ;)

#7 lore raw
Quoted 7 lines · Arjan van de Ven
Quoted 6 lines

At any rate, they have a bug in their proprietary module (news at 11).

So, I don't think this should make the top ten. Do you have a way to sort tainted vs non-tainted, and only produce the top ten for untainted?

Quoted 5 lines

yes absolutely; this is a question I'll have for the customers of the data... do people want to see "only-tainted" in these top 10s? Right now I mark them as such but leave them in. It's trivial for me to just leave them out instead (the info is there, just a matter of not counting)

i think they should be included, but perhaps abbreviated [into 1-2 lines] so that they do not hold us up.

We must not whitewash our bug statistics by intentionally excluding the harm that bin-only modules do to our users - but we can make them less visually intrusive, so that we can work on fixing the bugs we can fix.

Perhaps make sure it's top 10 of _our_ bugs, with the bin-only data mixed in as well (in a visually unintrusive way) to make the picture complete. I.e. list 20 bugs if 10 of them are bin-only modules.

Ingo

#8 lore raw
In reply to Arjan van de Ven
Quoted 6 lines · Arjan van de Ven

Rank 2: sysfs_add_one
Reported 264 times (528 total reports)
Duplicate sysfs registration... mostly a GregKH bug ... This oops was last seen in version 2.6.26-rc3, and first seen in
2.6.24-rc6.
More info: http://www.kerneloops.org/searchweek.php?search=sysfs_add_one

Note, only the older 2.6.24 and possibly .25 versions of this are my "fault", as those come from the usb core. That's a bug that I thought we fixed, and I can not duplicate it myself at all.

Other instances of this are real problems in other areas of the kernel (like the pci hotplug drivers), and are not my fault :)

If anyone knows of a way to constantly reproduce this issue with a USB device, please contact me and I'll work to resolve this.

Arjan, thanks a lot for these reports, I find them very useful.

thanks,

gerg k-h

#9 lore raw
In reply to Arjan van de Ven

Hello,

Quoted 7 lines

Rank 3: mark_buffer_dirty
Reported 253 times (547 total reports)
EXT3 bug while hot-removing a USB device
This oops was last seen in version 2.6.25.3, and first seen in 2.6.24-rc6.
More info:
http://www.kerneloops.org/searchweek.php?search=mark_buffer_dirty

Is someone looking into this? It could be somehow connected with commit
1be62dc190ebaca331038962c873e7967de6cc4b where we add smp_mb() to mark_buffer_dirty() under some circumstances. But I don't really see how. The WARN_ON() being triggered is !buffer_uptodate(bh) but that seems ridiculous for call paths like ext2_sync_super() -> mark_buffer_dirty() or journal_destroy() -> journal_update_superblock() -> mark_buffer_dirty() which are in oopses. Also this warning started appearing only recently while ext2 and JBD didn't change those areas recently. Also interesting may be that both ext2_sync_super() and journal_update_superblock() call sync_dirty_buffer() just after calling mark_buffer_dirty()...
Arjan, do we have some more info for these oopses (like hw config,
what was the machine doing while the WARN_ON has been triggered etc.)? Thanks.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#10 lore raw
Quoted 9 lines · On Sun, 25 May 2008 00:23:04 +0200 Jan Kara <[email protected]> wrote:

Hello,

Quoted 7 lines

Rank 3: mark_buffer_dirty
Reported 253 times (547 total reports)
EXT3 bug while hot-removing a USB device
This oops was last seen in version 2.6.25.3, and first seen
in 2.6.24-rc6.
More info:
http://www.kerneloops.org/searchweek.php?search=mark_buffer_dirty

Quoted 15 lines

Is someone looking into this? It could be somehow connected with
commit 1be62dc190ebaca331038962c873e7967de6cc4b where we add smp_mb() to mark_buffer_dirty() under some circumstances. But I don't really see how. The WARN_ON() being triggered is !buffer_uptodate(bh) but that seems ridiculous for call paths like ext2_sync_super() -> mark_buffer_dirty() or journal_destroy() -> journal_update_superblock() -> mark_buffer_dirty() which are in oopses. Also this warning started appearing only recently while ext2 and JBD didn't change those areas recently. Also interesting may be that both ext2_sync_super() and journal_update_superblock() call sync_dirty_buffer() just after calling mark_buffer_dirty()...
Arjan, do we have some more info for these oopses (like hw config,
what was the machine doing while the WARN_ON has been triggered etc.)? Thanks.

Ted looked at these during the LF summit, and his conclusion was that they're all media errors (eg USB unplug) that ext3 then did not handle well at all. Maybe Ted has an update on this?

the 2.6.24-rc is a red herring btw.. that's just the first kernel version that actually printed its version as part of WARN_ON().

#11 lore raw
Quoted 5 lines · Arjan van de Ven

Ted looked at these during the LF summit, and his conclusion was that they're all media errors (eg USB unplug) that ext3 then did not handle well at all. Maybe Ted has an update on this?

Not really. It's on my todo list but fixing a bug caused by users doing something stupid (pulling a mounted USB stick) has been lower than a number of other fires burning on my plate. I'll try to get to it but a lot of other things I need to worry about have deadlines associated with them....

- Ted

#12 lore raw
Quoted 11 lines · Theodore Tso
Quoted 4 lines · Arjan van de Ven

Ted looked at these during the LF summit, and his conclusion was that they're all media errors (eg USB unplug) that ext3 then did not handle well at all. Maybe Ted has an update on this?

Not really. It's on my todo list but fixing a bug caused by users doing something stupid (pulling a mounted USB stick) has been lower than a number of other fires burning on my plate. I'll try to get to it but a lot of other things I need to worry about have deadlines associated with them....

There are more reasons for connection loss than "user doing something stupid". Firmware flaws for example. Or SBP-2 re-login failing on a crowded FireWire bus.

Hot-removal capability is a fundamental requirement for a filesystem, just like for the block drivers and transport drivers. (E.g. don't corrupt the kernel in case of unrecoverable IO failures.)

BTW, hfs+ is also buggy WRT connection loss.

Signature · 3 lines
Stefan Richter
-=====-==--- -=-= ==--=
http://arcgraph.de/sr/
#13 lore raw
In reply to Theodore Tso
Quoted 11 lines · Theodore Tso
Quoted 4 lines · Arjan van de Ven

Ted looked at these during the LF summit, and his conclusion was that they're all media errors (eg USB unplug) that ext3 then did not handle well at all. Maybe Ted has an update on this?

Not really. It's on my todo list but fixing a bug caused by users doing something stupid (pulling a mounted USB stick) has been lower than a number of other fires burning on my plate. I'll try to get to it but a lot of other things I need to worry about have deadlines associated with them....

Exactly why is pulling an USB stick considered "stupid"? Last i checked there was no physical lock preventing users from doing that.

Sure, pulling a mounted USB stick is inconvenient ... for _us_ kernel developers. But the user really doesnt care and shouldnt care.

Ingo

#14 lore raw
Quoted 5 lines · Ingo Molnar

Exactly why is pulling an USB stick considered "stupid"? Last i checked there was no physical lock preventing users from doing that.

Sure, pulling a mounted USB stick is inconvenient ... for _us_ kernel developers. But the user really doesnt care and shouldnt care.

Because they could lose data? Because if the kernel wakes up and tries writing to the USB stick right as they pull it out, it could physically damage the flash format? I know, stupid reason... :-)

I know, I know, it's an issue. If someone wants to look at it, great. It is on my todo list, but I've been doing a lot of travelling, and when you have only one laptop, and pulling USB sticks isn't the sort of thing you can do using UML or KVM, it tends to fall to the bottom of your list.

- Ted

#15 lore raw
Quoted 11 lines · Theodore Tso
Quoted 6 lines · Ingo Molnar

Exactly why is pulling an USB stick considered "stupid"? Last i checked there was no physical lock preventing users from doing that.

Sure, pulling a mounted USB stick is inconvenient ... for _us_ kernel developers. But the user really doesnt care and shouldnt care.

Because they could lose data? Because if the kernel wakes up and tries writing to the USB stick right as they pull it out, it could physically damage the flash format? I know, stupid reason... :-)

user can lose data in many other ways, that's not the issue - the issue here is something very crutial: the kernel gets confused about a _very_ common user-triggerable condition.

That confusion must not happen in a modern OS and the kernel should be resilient and cope with such external events. And we must not deprioritize it with an incorrect "user did something stupid" tag... That argument might have been valid 15 years ago when floppies could be locked and you needed a needle to force-eject it but it is rather lame today when unplugging an USB stick is as easy as moving the mouse.

If there's something stupid here it's the kernel not dealing with that condition properly. Yes, the "user action" here looks "trivial" to the user but what happens below is indeed very hard technically, but who said that writing an OS from scratch would be an easy task? ;-)

Ingo

#16 lore raw
Quoted 26 lines · Ingo Molnar
Quoted 11 lines · Theodore Tso
Quoted 6 lines · Ingo Molnar

Exactly why is pulling an USB stick considered "stupid"? Last i checked there was no physical lock preventing users from doing that.

Sure, pulling a mounted USB stick is inconvenient ... for _us_ kernel developers. But the user really doesnt care and shouldnt care.

Because they could lose data? Because if the kernel wakes up and tries writing to the USB stick right as they pull it out, it could physically damage the flash format? I know, stupid reason... :-)

user can lose data in many other ways, that's not the issue - the issue here is something very crutial: the kernel gets confused about a _very_ common user-triggerable condition.

That confusion must not happen in a modern OS and the kernel should be resilient and cope with such external events. And we must not deprioritize it with an incorrect "user did something stupid" tag... That argument might have been valid 15 years ago when floppies could be locked and you needed a needle to force-eject it but it is rather lame today when unplugging an USB stick is as easy as moving the mouse.

Quoted 4 lines

If there's something stupid here it's the kernel not dealing with that condition properly. Yes, the "user action" here looks "trivial" to the user but what happens below is indeed very hard technically, but who said that writing an OS from scratch would be an easy task? ;-)

Well, Ingo, I don't know if you've noticed but the machine continues to run
just fine, as far as I understand. It only spits a dangerously looking warning and that's it. I agree we shouldn't be doing this but I don't really find this a critical problem.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#17 lore raw
Quoted 3 lines · Jan Kara

just fine, as far as I understand. It only spits a dangerously looking warning and that's it. I agree we shouldn't be doing this but I don't really find this a critical problem.

yeah i know it's "just" a warning, i checked the stack dump on kerneloops.org before i wrote the mail.

Pulling removable media out while it might still be mounted is a fact of life and comes not from the stupidity of the user but from the lack of physical barriers on the device side.

What if the USB stick was pulled mistakenly, the user notices her mistake later on and plugs the USB stick back in and expects all the data to not be corrupted?

What if the user puts the stick back in and it wont be mounted or will be critically damaged?

How do we even know whether these cases all work 100% robustly if the attitude is that removing a mounted device is a stupid thing to do and a bug related to it gets deprioritized? [starting with the issue of why the user should even care about such a relatively low-level abstraction as a "mounted filesystem".]

And any such problems do come up in the enterprise space as well, in terms of multipath IO issues - and Linux still does quite poorly in that area.

Ingo

#18 lore raw
Quoted 4 lines · Ingo Molnar

What if the USB stick was pulled mistakenly, the user notices her mistake later on and plugs the USB stick back in and expects all the data to not be corrupted?

If the USB stack folks would like to work on how to recognize that it's the same USB stick that had been previously pulled, so that it gets the same block device, and we can decide for how long we should keep dirty buffers around associated with a pulled USB stick, we can certainly have that conversation. :-)

Quoted 3 lines

And any such problems do come up in the enterprise space as well, in terms of multipath IO issues - and Linux still does quite poorly in that area.

We definitely have problems here, I agree. But at least with a multipath I/O device we have something that sticks around even when the last I/O path is pulled. We could talk about setting up dm-multipath with USB, I suppose --- that would have the benefit of getting the dm-multipath code much more widely exercised without needing exotic (and expensive) hardware kit being required.

- Ted

#19 lore raw

Am Montag 26 Mai 2008 19:01:48 schrieb Theodore Tso:

Quoted 5 lines

If the USB stack folks would like to work on how to recognize that it's the same USB stick that had been previously pulled, so that it gets the same block device, and we can decide for how long we should keep dirty buffers around associated with a pulled USB stick, we can certainly have that conversation.  :-)

Even if we could tell whether the device has remained the same, how would we know the medium wasn't exchanged?

Regards
Oliver

#20 lore raw
Quoted 9 lines · Oliver Neukum

Am Montag 26 Mai 2008 19:01:48 schrieb Theodore Tso:

Quoted 5 lines

If the USB stack folks would like to work on how to recognize that it's the same USB stick that had been previously pulled, so that it gets the same block device, and we can decide for how long we should keep dirty buffers around associated with a pulled USB stick, we can certainly have that conversation. :-)

Even if we could tell whether the device has remained the same, how would we know the medium wasn't exchanged?

Looking at the filesystem UUID could help -- this is an ID that is present as data on the disk, and that is even independent of the bus type. See also /dev/disk/by-uuid.

For the journaling filesystems I am familiar with the default value for the commit parameter is 5 seconds. Would it be a good idea to leave the default to 5s for non-removable devices, and to change this default to 1s for removable devices ?

Bart.

#21 lore raw
Quoted 8 lines · Bart Van Assche
Quoted 7 lines · Oliver Neukum

Am Montag 26 Mai 2008 19:01:48 schrieb Theodore Tso:

Quoted 5 lines

If the USB stack folks would like to work on how to recognize that it's the same USB stick that had been previously pulled, so that it gets the same block device, and we can decide for how long we should keep dirty buffers around associated with a pulled USB stick, we can certainly have that conversation. :-)

Quoted 6 lines
Quoted 2 lines

Even if we could tell whether the device has remained the same, how would we know the medium wasn't exchanged?

Looking at the filesystem UUID could help -- this is an ID that is present as data on the disk, and that is even independent of the bus type. See also /dev/disk/by-uuid.

Yes, but as Oliver wrote if someone modified the filesystem in the mean
time, you won't notice it - UUID doesn't help here.

Quoted 4 lines

For the journaling filesystems I am familiar with the default value for the commit parameter is 5 seconds. Would it be a good idea to leave the default to 5s for non-removable devices, and to change this default to 1s for removable devices ?

I don't think it's a good idea:
1) You'd even more stress wear-leveling of USB flash drives - btw, given their sizes it does not make much sence to use ext3 on your USB stick. I still use VFAT/ext2 there.

2) You'd probably notice performance decrease because of more journaling overhead.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#22 lore raw
Quoted 5 lines · Jan Kara
Quoted 3 lines

Looking at the filesystem UUID could help -- this is an ID that is present as data on the disk, and that is even independent of the bus type. See also /dev/disk/by-uuid.

Yes, but as Oliver wrote if someone modified the filesystem in the mean
time, you won't notice it - UUID doesn't help here.

That part you could figure out in userspace, by looking at the last mount and last modified time in the superblock. But the problem is it's too late. If you had buffers which had been "in flight" at the time when the USB stick was pulled, the kernel isn't going to be able to send them to the new instantiation of the device for the freshly installed USB stick. And I don't think we want to put filesystem-specific UUID and superblock parsing code in the generic USB layer!

I suspect that if we want to do this, the only way would be with massive amounts of userspace help, and with the dm layer interposing between the filesystem and the device. So when the USB stick gets pulled, from the dm-multipath side it looks like the last I/O path has been pulled, and it role-plays accordingly (with some kind of intelligence where it holds dirty buffers for some reasonable amount of time --- where reasonable is not easy to define) and then when someone re-inserts a USB stick, userspace will have to figure out that it was the same filesystem, and that it apparently hasn't been tampered with, cross its fingers, and then associate the (possibly different USB device) with the dm-multipath device.

If we have a super-bright student who needs humbling, it might make for an interesting GSOC project. :-)

- Ted

#23 lore raw
Quoted 6 lines · Theodore Tso
Quoted 5 lines
Quoted 3 lines

Looking at the filesystem UUID could help -- this is an ID that is present as data on the disk, and that is even independent of the bus type. See also /dev/disk/by-uuid.

Yes, but as Oliver wrote if someone modified the filesystem in the mean
time, you won't notice it - UUID doesn't help here.

Quoted 8 lines

That part you could figure out in userspace, by looking at the last mount and last modified time in the superblock. But the problem is it's too late. If you had buffers which had been "in flight" at the time when the USB stick was pulled, the kernel isn't going to be able to send them to the new instantiation of the device for the freshly installed USB stick. And I don't think we want to put filesystem-specific UUID and superblock parsing code in the generic USB layer!

yeah, i agree it's all ugly - but it's really our making not the user's ;-)

i think we could and should go to quite some length to properly support a rather benign-appearing usecase such as the user removing stuff from a modern computer (stuff that is not specifically bolted down that is). Violating a few artificial abstraction layers within the kernel is a lot better than losing user data, IMHO.

Ingo

#24 lore raw
In reply to Bart Van Assche

Am Montag 26 Mai 2008 19:28:23 schrieb Bart Van Assche:

Quoted 6 lines
Quoted 2 lines

Even if we could tell whether the device has remained the same, how would we know the medium wasn't exchanged?

Looking at the filesystem UUID could help -- this is an ID that is present as data on the disk, and that is even independent of the bus type. See also /dev/disk/by-uuid.

The medium may or may not hold a filesystem that has a UUID. In addition you can clone filesystems with dd but reuse them as independent filesystems.

Yes, you could checksum parts of the filesystem and say that at some point the user should suffer the consequences of his stupidity, but such things don't belong into the kernel and neither are they specific to USB.

Regards
Oliver

#25 lore raw
In reply to Oliver Neukum
Quoted 9 lines · Oliver Neukum

Am Montag 26 Mai 2008 19:01:48 schrieb Theodore Tso:

Quoted 5 lines

If the USB stack folks would like to work on how to recognize that it's the same USB stick that had been previously pulled, so that it gets the same block device, and we can decide for how long we should keep dirty buffers around associated with a pulled USB stick, we can certainly have that conversation.  :-)

Even if we could tell whether the device has remained the same, how would we know the medium wasn't exchanged?

We need filesystem support for that, but I guess we should just bite the bullet and do it for the most common filesystems... it is very useful for suspend/resume, too and I have heared about (misdesigned) machines where it will be basically mandatory...
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#26 lore raw
In reply to Theodore Tso
Quoted 11 lines · Theodore Tso
Quoted 4 lines · Ingo Molnar

What if the USB stick was pulled mistakenly, the user notices her mistake later on and plugs the USB stick back in and expects all the data to not be corrupted?

If the USB stack folks would like to work on how to recognize that it's the same USB stick that had been previously pulled, so that it gets the same block device, and we can decide for how long we should keep dirty buffers around associated with a pulled USB stick, we can certainly have that conversation. :-)

We do that already on suspend/resume, so it should not be hard to move that to a disconnect/connect model as well, it's the same code path :)

Bring it up on the linux-usb list if you are interested in persuing this.

In the meanwhile, getting rid of the warning would be a good idea.

thanks,

greg k-h

#27 lore raw
In reply to Ingo Molnar
Quoted 15 lines · Ingo Molnar
Quoted 11 lines · Theodore Tso
Quoted 6 lines · Ingo Molnar

Exactly why is pulling an USB stick considered "stupid"? Last i checked there was no physical lock preventing users from doing that.

Sure, pulling a mounted USB stick is inconvenient ... for _us_ kernel developers. But the user really doesnt care and shouldnt care.

Because they could lose data? Because if the kernel wakes up and tries writing to the USB stick right as they pull it out, it could physically damage the flash format? I know, stupid reason... :-)

Quoted 7 lines

user can lose data in many other ways, that's not the issue - the issue here is something very crutial: the kernel gets confused about a _very_ common user-triggerable condition.

That confusion must not happen in a modern OS and the kernel should be resilient and cope with such external events. And we must not deprioritize it with an incorrect "user did something stupid" tag...

Of course unavoidable "data corruption happens when user pulls the stick without unmounting" has lower priority than, for example "usb stick stops working randomly when used overnight"?

And we have still bugs of the second class, so yes unexpected pulls do have lower priority.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#28 lore raw
In reply to Theodore Tso
Quoted 4 lines · Theodore Tso

It is on my todo list, but I've been doing a lot of travelling, and when you have only one laptop, and pulling USB sticks isn't the sort of thing you can do using UML or KVM, it tends to fall to the bottom of your list.

Maybe add fault injection code which simply lets a SCSI LLD's queuecommand fail everything when a sysfs switch was flipped. There may be more convenient points to fail IO higher up, I don't know.

Signature · 3 lines
Stefan Richter
-=====-==--- -=-= ==-=-
http://arcgraph.de/sr/