Skip to contentSkip to conversation

data corruption: revalidating a (removable) hdd/flash on re-insert

DISCUSSION Michael Tokarev 4 participants 14 messages Oct 31, 2008 – Nov 5, 2008

lore
Share & tools
lore
#0 lore raw

To make a long story short: is there a way to force kernel to re-validate a replaced usb-connected hard drive (or a flash) *automatically*?

Because right now, the kernel does not see that the drive has been replaced, and uses *some* old cached values, which results in random data corruption here and there, and other similar odd things.

For example, I've an USB flash reader (Carry Computer Eng., Co., Ltd 6-in-1 Card Reader, but that's not really relevant). Among other things it has a compact flash slot. And I've 2 differently-size CF cards.

So I turn the machine on, insert one CF card, mount it, do something with its vfat filesystem, umount it and remove it. Next I insert another card, and when trying to mount it, the kernel says:

FAT: invalid media value (0x1e)
VFS: Can't find a valid FAT filesystem on dev sdb1.

When trying to run cfdisk for example, it complains that "partition 0 ends after end of the drive" (or something similar).

Sometimes the mount succeeds, but there are all sorts of errors here in there, the filesystem is messed up, whith parts of the files "from" just-removed card, and parts from the new card.

And sure enough, when I, forgetting the issue, trying to WRITE something to the card, it becomes almost 100% messy...

What helps is to run, for example,

  blockdev --rereadpt /dev/sdb

manually, after replacing the card.

The first thing I was thinking of when saw the whole mess
is: there must be some process like hald/udev/whatever messy subsystem du jur, which holds the device node open and prevents the kernel from re-reading the drive by its own as it correctly did in the past. Nope, I've shut down everything, and the same happens when only shell process running INSTEAD of init (booting with init=/bin/sh option).

So at some point the kernel stopped noticing the drive change in this configuration some time ago. I can't say when exactly, since I didn't use the card reader for over a year, and certainly didn't try it with more than one
card in a row for even longer time. It worked in the
past, that's for sure. And it definitely does not work (resulting in the above mess) with 2.6.25, 2.6.26 and 2.6.27.

Help?

Thanks!

/mjt

#1 lore raw
Quoted 49 lines · Michael Tokarev

To make a long story short: is there a way to force kernel to re-validate a replaced usb-connected hard drive (or a flash) *automatically*?

Because right now, the kernel does not see that the drive has been replaced, and uses *some* old cached values, which results in random data corruption here and there, and other similar odd things.

For example, I've an USB flash reader (Carry Computer Eng., Co., Ltd 6-in-1 Card Reader, but that's not really relevant). Among other things it has a compact flash slot. And I've 2 differently-size CF cards.

So I turn the machine on, insert one CF card, mount it, do something with its vfat filesystem, umount it and remove it. Next I insert another card, and when trying to mount it, the kernel says:

FAT: invalid media value (0x1e)
VFS: Can't find a valid FAT filesystem on dev sdb1.

When trying to run cfdisk for example, it complains that "partition 0 ends after end of the drive" (or something similar).

Sometimes the mount succeeds, but there are all sorts of errors here in there, the filesystem is messed up, whith parts of the files "from" just-removed card, and parts from the new card.

And sure enough, when I, forgetting the issue, trying to WRITE something to the card, it becomes almost 100% messy...

What helps is to run, for example,

  blockdev --rereadpt /dev/sdb

manually, after replacing the card.

The first thing I was thinking of when saw the whole mess
is: there must be some process like hald/udev/whatever messy subsystem du jur, which holds the device node open and prevents the kernel from re-reading the drive by its own as it correctly did in the past. Nope, I've shut down everything, and the same happens when only shell process running INSTEAD of init (booting with init=/bin/sh option).

Quoted 7 lines

So at some point the kernel stopped noticing the drive change in this configuration some time ago. I can't say when exactly, since I didn't use the card reader for over a year, and certainly didn't try it with more than one
card in a row for even longer time. It worked in the
past, that's for sure. And it definitely does not work (resulting in the above mess) with 2.6.25, 2.6.26 and 2.6.27.

I have had this happen with a few usb flash card readers. My solution was to unplug the usb cable then swap the card and connect the usb cable again. In the end I went and bought a different card reader, which does work correctly.

I highly suspect it is a mistake in the hardware causing the problem given the vast majority of readers do work correctly already.

So far I have had no problems with a silverstone, mitsumi, dell (in monitor), sandisk. I have had a problem with a no name cheap "15 in 1" reader which I stopped using because plugging and unplugging got annoying.

So my recommendation is get a non broken device.

Signature · 1 line
Len Sorensen
#2 lore raw
Quoted 10 lines · Lennart Sorensen
Quoted 9 lines · Michael Tokarev

To make a long story short: is there a way to force kernel to re-validate a replaced usb-connected hard drive (or a flash) *automatically*?

Because right now, the kernel does not see that the drive has been replaced, and uses *some* old cached values, which results in random data corruption here and there, and other similar odd things.

Quoted 4 lines
Quoted 4 lines

For example, I've an USB flash reader (Carry Computer Eng., Co., Ltd 6-in-1 Card Reader, but that's not really relevant). Among other things it has a compact flash slot. And I've 2 differently-size CF cards.

[]

Quoted 8 lines
Quoted 7 lines

So at some point the kernel stopped noticing the drive change in this configuration some time ago. I can't say when exactly, since I didn't use the card reader for over a year, and certainly didn't try it with more than one
card in a row for even longer time. It worked in the
past, that's for sure. And it definitely does not work (resulting in the above mess) with 2.6.25, 2.6.26 and 2.6.27.

Quoted 4 lines

I have had this happen with a few usb flash card readers. My solution was to unplug the usb cable then swap the card and connect the usb cable again. In the end I went and bought a different card reader, which does work correctly.

Well, this one is internal reader, which plugs into a 3" slot. AND it also has the regular floppy drive in it, too. It's a combo, a floppy drive AND a USB flash reader. As such, I can't easily re-plug it (which definitely helps, too, but for that to work I have to open the case), and I can't replace it either, because this device is almost unique: I still need a floppy and there's no other such combo drives, at least I wasn't able to find it. It's a great device if you think of it: it connects two epochs together...

Quoted 2 lines

I highly suspect it is a mistake in the hardware causing the problem given the vast majority of readers do work correctly already.

So at least a) I'm not alone, and b) there's SOMETHING that works. Excellent!

Quoted 6 lines

So far I have had no problems with a silverstone, mitsumi, dell (in monitor), sandisk. I have had a problem with a no name cheap "15 in 1" reader which I stopped using because plugging and unplugging got annoying.

So my recommendation is get a non broken device.

The thing is that with some older kernel(s) it defeinitely worked. So I'd say it's the kernel which broke/regressed, not the hardware. Suggesting to fix the hardware because new kernel does not work with it anymore is.. strange at least.

And yes it was definitely a el cheapo no-name thing. But a.. great (epochs!) and hence unique thing, see above.. ;)

I'll try to find out when it broke. My first suspect was the patch introduced not-so-recently (in 2.6.2x series) to support media change notifications done by some hardware (wait for notify instead of constantly polling).

Thanks!

/mjt

#4 lore raw
In reply to Michael Tokarev
Quoted 49 lines · Michael Tokarev

To make a long story short: is there a way to force kernel to re-validate a replaced usb-connected hard drive (or a flash) *automatically*?

Because right now, the kernel does not see that the drive has been replaced, and uses *some* old cached values, which results in random data corruption here and there, and other similar odd things.

For example, I've an USB flash reader (Carry Computer Eng., Co., Ltd 6-in-1 Card Reader, but that's not really relevant). Among other things it has a compact flash slot. And I've 2 differently-size CF cards.

So I turn the machine on, insert one CF card, mount it, do something with its vfat filesystem, umount it and remove it. Next I insert another card, and when trying to mount it, the kernel says:

FAT: invalid media value (0x1e)
VFS: Can't find a valid FAT filesystem on dev sdb1.

When trying to run cfdisk for example, it complains that "partition 0 ends after end of the drive" (or something similar).

Sometimes the mount succeeds, but there are all sorts of errors here in there, the filesystem is messed up, whith parts of the files "from" just-removed card, and parts from the new card.

And sure enough, when I, forgetting the issue, trying to WRITE something to the card, it becomes almost 100% messy...

What helps is to run, for example,

blockdev --rereadpt /dev/sdb

manually, after replacing the card.

The first thing I was thinking of when saw the whole mess
is: there must be some process like hald/udev/whatever messy subsystem du jur, which holds the device node open and prevents the kernel from re-reading the drive by its own as it correctly did in the past. Nope, I've shut down everything, and the same happens when only shell process running INSTEAD of init (booting with init=/bin/sh option).

Quoted 7 lines

So at some point the kernel stopped noticing the drive change in this configuration some time ago. I can't say when exactly, since I didn't use the card reader for over a year, and certainly didn't try it with more than one
card in a row for even longer time. It worked in the
past, that's for sure. And it definitely does not work (resulting in the above mess) with 2.6.25, 2.6.26 and 2.6.27.

Maybe your card reader is broken. I can not reproduce this with any of the many readers I have. Usually a media change results in media revalidation with the next access to the device. You can easily reproduce that:

Insert the media, and force a validation:

  $ touch /dev/sdb

Start logging of the kernel uevents to the console:

  $ udevadm monitor --kernel &

Access the device:

  $ touch /dev/sdb

Nothing should happen, as the reader/kernel knows it is still valid.

Now remove the media and insert it immediately again.

Access the device:

  $ touch /dev/sdb
  UEVENT[1225468868.803950] change
/devices/pci0000:00/0000:00:1d.7/usb5/5-2/5-2:1.0/host8/target8:0:0/8:0:0:0
(scsi)

and you see the reader told to kernel (scsi unit attention) to revalidate the device.

These events happen only when the device is accessed. That's why distros poll removable devices for media changes.

Every access to removable media is guarded by this revalidation check. If you don't see these events, you should not trust this reader, and at least never change the media while it is connected.

Kay

#5 lore raw
Quoted 4 lines · Kay Sievers
Quoted 3 lines · Michael Tokarev

To make a long story short: is there a way to force kernel to re-validate a replaced usb-connected hard drive (or a flash) *automatically*?

[]

Quoted 2 lines

Insert the media, and force a validation:

  $ touch /dev/sdb

With a newly inserted flash (removed some irrelevant stuff):

DEVTYPE=disk SUBSYSTEM=block MINOR=16 ACTION=change MAJOR=8 DEVTYPE=partition SUBSYSTEM=block MINOR=17 ACTION=add MAJOR=8 DEVTYPE=scsi_device SUBSYSTEM=scsi DRIVER=sd SDEV_MEDIA_CHANGE=1 ACTION=change DEVTYPE=disk SUBSYSTEM=block MINOR=16 ACTION=change MAJOR=8

Quoted 4 lines

Access the device:

  $ touch /dev/sdb

Nothing should happen, as the reader/kernel knows it is still valid.

Yes nothing happens.

Quoted 10 lines

Now remove the media and insert it immediately again.

Access the device:

  $ touch /dev/sdb
  UEVENT[1225468868.803950] change
/devices/pci0000:00/0000:00:1d.7/usb5/5-2/5-2:1.0/host8/target8:0:0/8:0:0:0
(scsi)

and you see the reader told to kernel (scsi unit attention) to revalidate the device.

Ok. So in my case, nothing happens here just like
if it were not removed/inserted.

I replaced the card with another one, and nothing
happened as well.

Only when touch'ing after REMOVING the flash, I see:

DEVTYPE=scsi_device SUBSYSTEM=scsi DRIVER=sd SDEV_MEDIA_CHANGE=1 ACTION=change DEVTYPE=partition SUBSYSTEM=block MINOR=17 ACTION=remove MAJOR=8 DEVTYPE=disk SUBSYSTEM=block MINOR=16 ACTION=change MAJOR=8

Quoted 3 lines

Every access to removable media is guarded by this revalidation check. If you don't see these events, you should not trust this reader, and at least never change the media while it is connected.

Ok. So.. 3 questions.

1) how it worked before (i yet to find which kernel worked)? I can only guess that some older kernel never cached the "validity".

2) 'doze notices the insertions/removals just fine. Again I can only guess that it constantly pools for changes.

3), and the most important one. I think there should be a
way to stop "caching" of the media information, i.e. to force revalidation events on EVERY access, for certain hardware at least. Because corruption in such cases is much worse than any positive effects of caching etc... Maybe some unusual_devs.h way or somesuch?..

Now I see the device is somewhat(?) broken. But as I said before in another email, it's a great device (as in, two epochs connected to each other), and it'd be sad to lose it. A nostalgie, sort of.. ;)

Ok, maybe actually polling for devices sometimes makes sense... ;) And there can be a workaround, using a tiny daemon that constantly accesses the device, in order to catch removals... 'hwell.

/mjt

#3 lore raw
In reply to Michael Tokarev
Quoted 9 lines · Michael Tokarev

Well, this one is internal reader, which plugs into a 3" slot. AND it also has the regular floppy drive in it, too. It's a combo, a floppy drive AND a USB flash reader. As such, I can't easily re-plug it (which definitely helps, too, but for that to work I have to open the case), and I can't replace it either, because this device is almost unique: I still need a floppy and there's no other such combo drives, at least I wasn't able to find it. It's a great device if you think of it: it connects two epochs together...

Yes I use exactly that made by mitsumi. Never had a problem with the mitsumi one. I haven't seen anyone with them lately though, so they made have been discontinued.

It is shown here though:
http://www.mitsumi.com/products/FA404.htm

I have the one on the right with the floppy at the bottom.

Quoted 2 lines

So at least a) I'm not alone, and b) there's SOMETHING that works. Excellent!

That's right.

Quoted 4 lines

The thing is that with some older kernel(s) it defeinitely worked. So I'd say it's the kernel which broke/regressed, not the hardware. Suggesting to fix the hardware because new kernel does not work with it anymore is.. strange at least.

Well the mitsumi I have has worked with every kernel I have ever had (2.6.26 at the moment). The no name reader certainly didn't work right with early 2.6 kernels, and I haven't bothered with it since to check.

Quoted 2 lines

And yes it was definitely a el cheapo no-name thing. But a.. great (epochs!) and hence unique thing, see above.. ;)

I think I paid $27(canadian) for my mitsumi. I paid $20 for the no name thing (which has no floppy and is external).

Quoted 4 lines

I'll try to find out when it broke. My first suspect was the patch introduced not-so-recently (in 2.6.2x series) to support media change notifications done by some hardware (wait for notify instead of constantly polling).

Perhaps. At least you have a simple test case, so a bisect shouldn't be too hard to do.

Signature · 1 line
Len Sorensen
#6 lore raw
In reply to Michael Tokarev
Quoted 55 lines · Michael Tokarev
Quoted 9 lines · Kay Sievers
Quoted 4 lines · Michael Tokarev

To make a long story short: is there a way to force kernel to re-validate a replaced usb-connected hard drive (or a flash) *automatically*?

Insert the media, and force a validation:

 $ touch /dev/sdb

With a newly inserted flash (removed some irrelevant stuff):

DEVTYPE=disk SUBSYSTEM=block MINOR=16 ACTION=change MAJOR=8 DEVTYPE=partition SUBSYSTEM=block MINOR=17 ACTION=add MAJOR=8 DEVTYPE=scsi_device SUBSYSTEM=scsi DRIVER=sd SDEV_MEDIA_CHANGE=1 ACTION=change
DEVTYPE=disk SUBSYSTEM=block MINOR=16 ACTION=change MAJOR=8

Quoted 4 lines

Access the device:

 $ touch /dev/sdb

Nothing should happen, as the reader/kernel knows it is still valid.

Yes nothing happens.

Quoted 11 lines

Now remove the media and insert it immediately again.

Access the device:

 $ touch /dev/sdb
 UEVENT[1225468868.803950] change

/devices/pci0000:00/0000:00:1d.7/usb5/5-2/5-2:1.0/host8/target8:0:0/8:0:0:0 (scsi)

and you see the reader told to kernel (scsi unit attention) to revalidate the device.

Ok. So in my case, nothing happens here just like
if it were not removed/inserted.

I replaced the card with another one, and nothing
happened as well.

Only when touch'ing after REMOVING the flash, I see:

DEVTYPE=scsi_device SUBSYSTEM=scsi DRIVER=sd SDEV_MEDIA_CHANGE=1 ACTION=change DEVTYPE=partition SUBSYSTEM=block MINOR=17 ACTION=remove MAJOR=8
DEVTYPE=disk SUBSYSTEM=block MINOR=16 ACTION=change MAJOR=8

Quoted 3 lines

Every access to removable media is guarded by this revalidation check. If you don't see these events, you should not trust this reader, and at least never change the media while it is connected.

Quoted 5 lines

Ok. So.. 3 questions.

1) how it worked before (i yet to find which kernel worked)? I can only guess that some older kernel never cached the "validity".

The kernel does not cache, it's the device itself that reports a media change, and the kernel asks every removable device before it is accessing it.

Quoted 2 lines

2) 'doze notices the insertions/removals just fine. Again I can only guess that it constantly pools for changes.

It polls the device every few secomds, just like HAL does on most Linux desktop installations. But in your case, when the reader does not report the change correctly, even that might go wrong, just like without polling.

Quoted 4 lines

3), and the most important one. I think there should be a
way to stop "caching" of the media information, i.e. to force revalidation events on EVERY access, for certain hardware at least.

That's how it already is. We just rely on the device to tell us. There is no way to revalidate anything otherwise.

Quoted 3 lines

Because corruption in such cases is much worse than
any positive effects of caching etc... Maybe some unusual_devs.h way or somesuch?..

I can't think of a way to make that working, there is no cache in the kernel, only a state in the device. You would need to checksum the device to find out that it isn't the same media, even that might not work, and it's definitely nothing you want to do.

You could trace with usbmon, or something and investigate the scsi packets if the scsi unit attention really does not signify a media change, which is what I expect.

Quoted 3 lines

Now I see the device is somewhat(?) broken. But as I said before in another email, it's a great device (as in, two epochs connected to each other), and it'd be sad to lose it. A nostalgie, sort of.. ;)

If it works otherwise, use it, but I wouldn't change media while it is connected.

Quoted 3 lines

Ok, maybe actually polling for devices sometimes makes sense... ;) And there can be a workaround, using a tiny daemon that constantly accesses the device, in order to catch removals... 'hwell.

That would still not work with your device, if you change the media during the polling interval, which is usually between 2 and 16 seconds. If the device does not report any change, like it seem in your case, you can not do anything. Polling helps only to reflect the current state of a device while it is not accessed, to get the state of the device into the kernel. Like on a desktop, where you want to auto-mount card reader/cdrom media on insertion.

Kay

#7 lore raw
In reply to Kay Sievers
Quoted 37 lines · Kay Sievers
Quoted 8 lines · Michael Tokarev

To make a long story short: is there a way to force kernel to re-validate a replaced usb-connected hard drive (or a flash) *automatically*?

Because right now, the kernel does not see that the drive has been replaced, and uses *some* old cached values, which results in random data corruption here and there, and other similar odd things.

Maybe your card reader is broken. I can not reproduce this with any of the many readers I have. Usually a media change results in media revalidation with the next access to the device. You can easily reproduce that:

Insert the media, and force a validation:

  $ touch /dev/sdb

Start logging of the kernel uevents to the console:

  $ udevadm monitor --kernel &

Access the device:

  $ touch /dev/sdb

Nothing should happen, as the reader/kernel knows it is still valid.

Now remove the media and insert it immediately again.

Access the device:

  $ touch /dev/sdb
  UEVENT[1225468868.803950] change
/devices/pci0000:00/0000:00:1d.7/usb5/5-2/5-2:1.0/host8/target8:0:0/8:0:0:0
(scsi)

and you see the reader told to kernel (scsi unit attention) to revalidate the device.

Quoted 6 lines

These events happen only when the device is accessed. That's why distros poll removable devices for media changes.

Every access to removable media is guarded by this revalidation check. If you don't see these events, you should not trust this reader, and at least never change the media while it is connected.

This is rather nasty data-corrupter. Could we at least blacklist broken device, and force revalidation on each close or something like that?

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#8 lore raw
Quoted 38 lines · Pavel Machek
Quoted 37 lines · Kay Sievers
Quoted 8 lines · Michael Tokarev

To make a long story short: is there a way to force kernel to re-validate a replaced usb-connected hard drive (or a flash) *automatically*?

Because right now, the kernel does not see that the drive has been replaced, and uses *some* old cached values, which results in random data corruption here and there, and other similar odd things.

Maybe your card reader is broken. I can not reproduce this with any of the many readers I have. Usually a media change results in media revalidation with the next access to the device. You can easily reproduce that:

Insert the media, and force a validation:

  $ touch /dev/sdb

Start logging of the kernel uevents to the console:

  $ udevadm monitor --kernel &

Access the device:

  $ touch /dev/sdb

Nothing should happen, as the reader/kernel knows it is still valid.

Now remove the media and insert it immediately again.

Access the device:

  $ touch /dev/sdb
  UEVENT[1225468868.803950] change
/devices/pci0000:00/0000:00:1d.7/usb5/5-2/5-2:1.0/host8/target8:0:0/8:0:0:0
(scsi)

and you see the reader told to kernel (scsi unit attention) to revalidate the device.

Quoted 8 lines
Quoted 6 lines

These events happen only when the device is accessed. That's why distros poll removable devices for media changes.

Every access to removable media is guarded by this revalidation check. If you don't see these events, you should not trust this reader, and at least never change the media while it is connected.

This is rather nasty data-corrupter.

Sure, it is.

Quoted 3 lines

Could we at least blacklist
broken device, and force revalidation on each close or something like that?

What's your idea of revalidation if the hardware does not tell you? Get an md5 of the disk content? :)

Kay

#9 lore raw
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

Every access to removable media is guarded by this revalidation check. If you don't see these events, you should not trust this reader, and at least never change the media while it is connected.

This is rather nasty data-corrupter.

Quoted 8 lines

Sure, it is.

Quoted 3 lines

Could we at least blacklist
broken device, and force revalidation on each close or something like that?

What's your idea of revalidation if the hardware does not tell you? Get an md5 of the disk content? :)

Well... you should not eject media while fs is mounted or blockdev is open, correct?

So can we simply claim 'media changed' on last close/unmount? Sure, sometimes media was not changed, but that only hurts performance, not correctness... ?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#10 lore raw
Quoted 12 lines · Pavel Machek
Quoted 11 lines
Quoted 4 lines
Quoted 3 lines

Every access to removable media is guarded by this revalidation check. If you don't see these events, you should not trust this reader, and at least never change the media while it is connected.

This is rather nasty data-corrupter.

Sure, it is.

Quoted 3 lines

Could we at least blacklist
broken device, and force revalidation on each close or something like that?

What's your idea of revalidation if the hardware does not tell you? Get an md5 of the disk content? :)

Quoted 6 lines

Well... you should not eject media while fs is mounted or blockdev is open, correct?

So can we simply claim 'media changed' on last close/unmount? Sure, sometimes media was not changed, but that only hurts performance, not correctness... ?

Well, that's what my tiny proggy, which I used here to work around the problem, does. It constantly opens/closes the /dev/sdFOO, every 0.5s currently (I don't think I will be able to replace a media faster than half a second :), in order to catch REMOVALs of media -- because when the drive does not see the media anymore, it correctly reports that the media has changed...

I tried to make it to detect CLOSE of the file (either by userspace or by kernel on umount), to not waste time when the drive is open/mounted as it can't be revalidated anyway, but neither dnotify nor inotify is helpful here.

What is needed is to force "invalidation" on last close, so that on next open, kernel thinks it's a shiny new media, never seen before. Ie. to force-flush caches, or something like that. Sure this is not as good as my program, which still leaves caches in case media was NOT removed. But my approach is wasteful. And the data corruption is indeed quite bad (we've lost whole gig of photos this way already).

But yes, looks like this problem becomes less and less of an issue. So for me, it's easy to deal with (not perfect but it works; it'd be even better if i will be able to wait for umount using inotify, to only wake when really needed), and the real solution is to not use cheap broken hardware... (My unit was about $15, real ones costs $25 or so, but that's not the reason I've got it. Real reason was that it was only once than I actually saw such a thing, and it was the last one as well... ;)

Thanks!

/mjt

#11 lore raw
Quoted 20 lines · Michael Tokarev
Quoted 12 lines · Pavel Machek
Quoted 11 lines
Quoted 4 lines
Quoted 3 lines

Every access to removable media is guarded by this revalidation check. If you don't see these events, you should not trust this reader, and at least never change the media while it is connected.

This is rather nasty data-corrupter.

Sure, it is.

Quoted 3 lines

Could we at least blacklist
broken device, and force revalidation on each close or something like that?

What's your idea of revalidation if the hardware does not tell you? Get an md5 of the disk content? :)

Quoted 6 lines

Well... you should not eject media while fs is mounted or blockdev is open, correct?

So can we simply claim 'media changed' on last close/unmount? Sure, sometimes media was not changed, but that only hurts performance, not correctness... ?

Quoted 6 lines

Well, that's what my tiny proggy, which I used here to work around the problem, does. It constantly opens/closes the /dev/sdFOO, every 0.5s currently (I don't think I will be able to replace a media faster than half a second :), in order to catch REMOVALs of media -- because when the drive does not see the media anymore, it correctly reports that the media has changed...

Ok, so we you need to do is to put it into kernel and activate it via blacklist...?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#13 lore raw
In reply to Michael Tokarev
Quoted 3 lines · Michael Tokarev

So for me, it's easy to deal with (not perfect but it works; it'd be even better if i will be able to wait for umount using inotify, to only wake when really needed),

If I understand it right, what you are looking for, you could sleep in poll() of /proc/mounts, and will be woken up with POLLERR if anything in your mount tree changes. Then you check the state of your usb device and possibly invalidate it.

Kay

#12 lore raw
In reply to Pavel Machek
Quoted 2 lines · Pavel Machek
Quoted 1 line · Michael Tokarev

Pavel Machek wrote:

[]

Quoted 12 lines
Quoted 10 lines
Quoted 3 lines

So can we simply claim 'media changed' on last close/unmount? Sure, sometimes media was not changed, but that only hurts performance, not correctness... ?

Well, that's what my tiny proggy, which I used here to work around the problem, does. It constantly opens/closes the /dev/sdFOO, every 0.5s currently (I don't think I will be able to replace a media faster than half a second :), in order to catch REMOVALs of media -- because when the drive does not see the media anymore, it correctly reports that the media has changed...

Quoted 2 lines

Ok, so we you need to do is to put it into kernel and activate it via blacklist...?

I'm fine with my solution.. ;) Especially once Kay suggested to look at /proc/mounts for notifications.

Original problem was that I didn't understand what happens, and blamed kernel for "breaking" the working device (it looks like it never worked in the first place, it was just that we never hit the bug before). Once the problem become clear (thanks Kay!), I wrote the proggy mentioned above - it's obviously a gross hack, but it stops the corruption for me.

Generally the solution can be one of the 3:

a) leave it as it is now, since it had never been bought up before and hence does not affect many people. And because even if it was, it becomes less and less of a problem with bad drives going away slowly...

b) to use a mechanism like blacklist in kernel to force invalidation on CLOSE automatically for such drives (not when it really necessary as my program detects - on REMOVAL). Less efficient than my solution, but much easier to deal with in kernel.

c) I will use my variant for my problem.. while finding a replacement for the bad hardware.

So no, I'm not asking to put that proggy into the kernel.. ;) For kernelspace solution that'd be a much simple way. If at all.

So to summary: if it is EASY (read: trivial) to do such blacklist in kernel space, I'd do it right away, because potentially it is still possible to see similar corruptions elsewhere. If not, just forget the case as "solved for the reporter" ;)

Thanks!

/mjt