Skip to contentSkip to conversation

Sync option destroys flash!

DISCUSSION Michael H. Warfield 17 participants 30 messages May 13, 2005 – May 18, 2005

lore
Share & tools
lore
#0 lore raw

Under the right circumstances, even copying a single file to a flash
drive mounted with the "sync" option can destroy the entire drive!

Now that I have your attention!

I found this out the hard way. (Kissed one brand new $70 USD 1GB flash
drive good-bye.) According to the man pages for mount, FAT and VFAT file systems ignore the "sync" option. It lies. Maybe it use to be true, but it certainly lies now. A simple test can verify this. Mount a flash drive with a FAT/VFAT file system without the sync option and writes to the drive go very fast. Typing "sync" or unmounting the drive afterwards, takes time as the buffered data is written to the flash drive. This is as it should be. Mount it with the sync option and writes are really REALLY slow (worse than they should be just from copying the data through USB) but sync and umount come back immediately and result in no additional writing to the drive. [Do the preceding with only a few files and less than a few meg of data if you value that flash.] So... FAT and VFAT are honoring the sync option. This is very VERY bad. It's bad for floppies, it's bad for hard drives, it's FATAL for flash drives.

Flash drives have a limited number of write cycles. Many many
thousands of write cycles, but limited, none the less. They are also written in blocks which are much larger than the "sector" size report (several K in a physical nand flash block, IRC).

What happens, with the sync option on a VFAT file system, is that the
FAT tables are getting pounded and over-written over and over and over again as each and every block/cluster is allocated while a new file is written out. This constant overwriting eventually wears out the first block or two of the flash drive.

I had lost a couple of flash keys previously, without realizing what
was going on, but what send me off investigating this was when I copied a 700 Meg file to a brand new 1G USB 2.0 flash memory key in a USB 2.0 slot. It took over a half an hour to copy to the drive, which really had me wondering WTF was wrong! Then, when I went to use the key, I found the first couple of blocks were totally destroyed. Read errors immediately upon insertion. Then I started digging and found the hotplug / HAL / fstab-sync stuff on Fedora Core was mounting USB drives with the "sync" option (if less than 2 Gig). I knew from previous experience (CF backup cards in a PDA) that repeated pounding on the FAT tables would destroy a flash card with a FAT file system. So I reported this on the Fedora list. Someone else noticed that the man pages for mount state that FAT and VFAT ignore the sync option. Not so, obviously... Copying that 700 Meg file resulted in thousands upon thousands upon thousands of writes to the FAT table and backup FAT table. It simply blew through all the rewrites for those blocks and burned them up. Bye bye flash key...

On a floppy, this would result in an insane amount of jacking around
back and forth between data sectors and the FAT sectors. In addition to taking forever, that would shorten the life of the diskettes and the drive itself, but who cares about floppies any more. On a real hard drive, this will cause "head resonances" as the heads go through constant high speed seeks between the cylinder with the FAT tables and the data cylinders. That can't be good, on a continuous basis, for drive life. But it's really a disaster for flash memory. It's going to cause premature failure in most flash memory, even if it doesn't kill them right off as it did in my case with a 700 Meg file.

Can we go back to ignoring "sync" on FAT and VFAT? I can't see where
it does much good. You might corrupt a file system if you unplugged it while dirty but it beats the hell out of physically burning it up and destroying the drive!

If it's decided that the FAT and VFAT file systems MUST obey the sync
option then please do something about a special case for the FAT tables! Sync the data if thou must buti... Thou shalt not, must not, whack off on the FAT tables!!!

Another option would be to only sync the FAT and VFAT file systems upon
close of the file being written or upon close of the last file open on the file system (fs not busy) but that might not help in the case of a whole lotta little files...

I'm also going to file a couple of bug reports in bugzilla at RedHat
but this seems to be a more fundamental problem than a RedHat specific problem. But, IMHO, they should never be setting that damn sync flag arbitrarily.

Mike

Signature · 4 lines
 Michael H. Warfield    |  (770) 985-6132   |  [email protected]  
  /\/\|=mhw=|\/\/       |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!
#1 lore raw
Quoted 7 lines · Michael H. Warfield

I found this out the hard way. (Kissed one brand new $70 USD 1GB flash
drive good-bye.) According to the man pages for mount, FAT and VFAT file systems ignore the "sync" option. It lies. Maybe it use to be true, but it certainly lies now. A simple test can verify this. Mount a flash drive with a FAT/VFAT file system without the sync option and writes to the drive go very fast. Typing "sync" or unmounting the drive afterwards, takes time as the buffered data is written to the flash

Quoted 8 lines

drive. This is as it should be. Mount it with the sync option and writes are really REALLY slow (worse than they should be just from copying the data through USB) but sync and umount come back immediately and result in no additional writing to the drive. [Do the preceding with only a few files and less than a few meg of data if you value that flash.] So... FAT and VFAT are honoring the sync option. This is very VERY bad. It's bad for floppies, it's bad for hard drives, it's FATAL for flash drives.

Certainly causes lots of unnecesary writes which flash doesn't like. Given sync doesn't appear to be the default, whyever would you add sync to vfat?

Quoted 10 lines

Flash drives have a limited number of write cycles. Many many
thousands of write cycles, but limited, none the less. They are also written in blocks which are much larger than the "sector" size report (several K in a physical nand flash block, IRC).

What happens, with the sync option on a VFAT file system, is that the
FAT tables are getting pounded and over-written over and over and over again as each and every block/cluster is allocated while a new file is written out. This constant overwriting eventually wears out the first block or two of the flash drive.

All the flash I have used do automatic wear leveling. Maybe I have only used high quality flash media (given I am doing work with embedded industrial grade gear, that is quite plausible). Of course wear leveling doesn't mean you aren't doing way more writes than necesary, but it helps spread the load away from the FAT, which would otherwise quickly die on most flash cards no matter what system you use for writing to it.

Quoted 9 lines

I had lost a couple of flash keys previously, without realizing what
was going on, but what send me off investigating this was when I copied a 700 Meg file to a brand new 1G USB 2.0 flash memory key in a USB 2.0 slot. It took over a half an hour to copy to the drive, which really had me wondering WTF was wrong! Then, when I went to use the key, I found the first couple of blocks were totally destroyed. Read errors immediately upon insertion. Then I started digging and found the hotplug / HAL / fstab-sync stuff on Fedora Core was mounting USB drives with the "sync" option (if less than 2 Gig). I knew from previous

Quoted 8 lines

experience (CF backup cards in a PDA) that repeated pounding on the FAT tables would destroy a flash card with a FAT file system. So I reported this on the Fedora list. Someone else noticed that the man pages for mount state that FAT and VFAT ignore the sync option. Not so, obviously... Copying that 700 Meg file resulted in thousands upon thousands upon thousands of writes to the FAT table and backup FAT table. It simply blew through all the rewrites for those blocks and burned them up. Bye bye flash key...

700M = 1.4M 512byte sectors. I guess if it actually writes one sector at a time and syncs, and it's not a media with good wear leveling, then yes that would destroy the sectors holding the FAT. Ouch. Crappy media, and bad way to treat it. Unfortunately there is no bug, just user error, and potentially badly designed flash media firmware.

Quoted 11 lines

On a floppy, this would result in an insane amount of jacking around
back and forth between data sectors and the FAT sectors. In addition to taking forever, that would shorten the life of the diskettes and the drive itself, but who cares about floppies any more. On a real hard drive, this will cause "head resonances" as the heads go through constant high speed seeks between the cylinder with the FAT tables and the data cylinders. That can't be good, on a continuous basis, for drive life. But it's really a disaster for flash memory. It's going to cause premature failure in most flash memory, even if it doesn't kill them right off as it did in my case with a 700 Meg file.

Quoted 4 lines

Can we go back to ignoring "sync" on FAT and VFAT? I can't see where
it does much good. You might corrupt a file system if you unplugged it while dirty but it beats the hell out of physically burning it up and destroying the drive!

How about you just don't use the sync option with fat when you don't mean to use sync? sync does exactly what it should, which just happens to not be what you want, so don't use it.

Quoted 4 lines

If it's decided that the FAT and VFAT file systems MUST obey the sync
option then please do something about a special case for the FAT tables! Sync the data if thou must buti... Thou shalt not, must not, whack off on the FAT tables!!!

Then the sync option wouldn't be much use anymore.

Quoted 4 lines

Another option would be to only sync the FAT and VFAT file systems upon
close of the file being written or upon close of the last file open on the file system (fs not busy) but that might not help in the case of a whole lotta little files...

Again not very useful then.

Quoted 4 lines

I'm also going to file a couple of bug reports in bugzilla at RedHat
but this seems to be a more fundamental problem than a RedHat specific problem. But, IMHO, they should never be setting that damn sync flag arbitrarily.

No they certainly should not, but it may have something to do with making life easier for kde/gnome desktops and automatic mount/umount of media. Dumb idea still, but that happens sometimes.

Len Sorensen

#2 lore raw
Quoted 3 lines · Lennart Sorensen

Certainly causes lots of unnecesary writes which flash doesn't like. Given sync doesn't appear to be the default, whyever would you add sync to vfat?

RedHat does it when they automount USB drives. It's in their "hal"
package. Law of unintended consequences, I guess.

Quoted 11 lines
Quoted 10 lines

Flash drives have a limited number of write cycles. Many many
thousands of write cycles, but limited, none the less. They are also written in blocks which are much larger than the "sector" size report (several K in a physical nand flash block, IRC).

What happens, with the sync option on a VFAT file system, is that the
FAT tables are getting pounded and over-written over and over and over again as each and every block/cluster is allocated while a new file is written out. This constant overwriting eventually wears out the first block or two of the flash drive.

Quoted 7 lines

All the flash I have used do automatic wear leveling. Maybe I have only used high quality flash media (given I am doing work with embedded industrial grade gear, that is quite plausible). Of course wear leveling doesn't mean you aren't doing way more writes than necesary, but it helps spread the load away from the FAT, which would otherwise quickly die on most flash cards no matter what system you use for writing to it.

Yeah, I've heard that claimed but a real problem is that nobody can
tell which ones do or don't until you've got a crispy chip. I've looked and I haven't been able to find a single reference on any that you might pick up at Best Buy or Fry's. I've never seen one that has it written on it that it has any sort of wear leveling like that. But I guess we don't all buy our flash drives at an industrial supply house (forget Tiger Direct or CDW either - consumer grade). I would bet that most people don't even REALIZE that flash has a limited life and wears out.

Quoted 5 lines

700M = 1.4M 512byte sectors. I guess if it actually writes one sector at a time and syncs, and it's not a media with good wear leveling, then yes that would destroy the sectors holding the FAT. Ouch. Crappy media, and bad way to treat it. Unfortunately there is no bug, just user error, and potentially badly designed flash media firmware.

Unfortunately, it's the default under some RedHat stuff (yes, I've
filed the bug reports) and the documentation on "mount" is incorrect and it's arguable if it's any value, and there's no warning of the risks. "User error" is a bit off the mark when the user has to take action he has no knowledge of to work around a destructive default he doesn't even know exists (here I'm referring specifically to the need to unmount the automounted flash and then remount it somewhere else without the unsafe options).

User error also boils down to removing drives before they are unmounted
or sync'ed. Windows seems to deal with this problem without jacking off on the FAT tables. Why is it that Linux has a procedure that is potentially destructive and degrades performance so badly when MS gets away without it in Windows? It's an MS file system. One of the contributors on the Fedora list remarked that this explained why his USB keys were so slow on Linux compared to Windows.

Quoted 11 lines
Quoted 11 lines

On a floppy, this would result in an insane amount of jacking around
back and forth between data sectors and the FAT sectors. In addition to taking forever, that would shorten the life of the diskettes and the drive itself, but who cares about floppies any more. On a real hard drive, this will cause "head resonances" as the heads go through constant high speed seeks between the cylinder with the FAT tables and the data cylinders. That can't be good, on a continuous basis, for drive life. But it's really a disaster for flash memory. It's going to cause premature failure in most flash memory, even if it doesn't kill them right off as it did in my case with a 700 Meg file.

Quoted 8 lines
Quoted 4 lines

Can we go back to ignoring "sync" on FAT and VFAT? I can't see where
it does much good. You might corrupt a file system if you unplugged it while dirty but it beats the hell out of physically burning it up and destroying the drive!

How about you just don't use the sync option with fat when you don't mean to use sync? sync does exactly what it should, which just happens to not be what you want, so don't use it.

Already filed bug reports with RedHat. On both HAL and on the
documentation for mount.

Quoted 6 lines
Quoted 4 lines

If it's decided that the FAT and VFAT file systems MUST obey the sync
option then please do something about a special case for the FAT tables! Sync the data if thou must buti... Thou shalt not, must not, whack off on the FAT tables!!!

Then the sync option wouldn't be much use anymore.

Oh? It's of some use now? Beyond degrading performance to the point
that it takes many times longer to write a flash system in Linux than in Windows? Yes, I know, don't use it. What's it there for, then? To protect idiots from removing drives before they're unmounted? Those are the same idiots who will not know what happened when it burns up there drive and they are the least likely to use that option overtly. Most people don't realize that it was being used without their knowledge (much less the risks associated with using it in the first place) and just thought it was a Linux problem that it took many times longer to write USB keys in Linux than in Windows.

Quoted 6 lines
Quoted 4 lines

Another option would be to only sync the FAT and VFAT file systems upon
close of the file being written or upon close of the last file open on the file system (fs not busy) but that might not help in the case of a whole lotta little files...

Again not very useful then.

Rule number one should be "do no harm". Maybe it should be under some
"force" flag or something just warning people not to use it unless they really REALLY know what their doing. I, personally, would never have used it under any circumstances. I was shocked to discover that RedHat was doing this by default. A warning to THEM would have been better than a warning to me, since I already knew not to do stupid shit like that. Not sure who is responsible for that HAL package or what other distros may also be vulnerable to this. I mostly blame RedHat for this problem first plus the mount man pages (which I suspect they replied on) second but I question if there shouldn't be a better way to avoid destroying hardware here.

Quoted 10 lines
Quoted 4 lines

I'm also going to file a couple of bug reports in bugzilla at RedHat
but this seems to be a more fundamental problem than a RedHat specific problem. But, IMHO, they should never be setting that damn sync flag arbitrarily.

No they certainly should not, but it may have something to do with making life easier for kde/gnome desktops and automatic mount/umount of media. Dumb idea still, but that happens sometimes.

Len Sorensen

Mike

Signature · 4 lines
 Michael H. Warfield    |  (770) 985-6132   |  [email protected]  
  /\/\|=mhw=|\/\/       |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!
#3 lore raw
Quoted 8 lines · Michael H. Warfield

Yeah, I've heard that claimed but a real problem is that nobody can
tell which ones do or don't until you've got a crispy chip. I've looked and I haven't been able to find a single reference on any that you might pick up at Best Buy or Fry's. I've never seen one that has it written on it that it has any sort of wear leveling like that. But I guess we don't all buy our flash drives at an industrial supply house (forget Tiger Direct or CDW either - consumer grade). I would bet that most people don't even REALIZE that flash has a limited life and wears out.

I believe any flash with a sandisk controller in it will do pretty good wear leveling (I think all the ones labeled sandisk have their controller, although they don't all have sandisk flash chips). The controller is what does the wear leveling. At least for CF and SD this is what they do.

Quoted 8 lines

Unfortunately, it's the default under some RedHat stuff (yes, I've
filed the bug reports) and the documentation on "mount" is incorrect and it's arguable if it's any value, and there's no warning of the risks. "User error" is a bit off the mark when the user has to take action he has no knowledge of to work around a destructive default he doesn't even know exists (here I'm referring specifically to the need to unmount the automounted flash and then remount it somewhere else without the unsafe options).

Well the documentation for mount isn't authorative on what the kernel does. Not sure where the authoritive information on how vfat treats sync is (besides the source code).

Quoted 7 lines

User error also boils down to removing drives before they are unmounted
or sync'ed. Windows seems to deal with this problem without jacking off on the FAT tables. Why is it that Linux has a procedure that is potentially destructive and degrades performance so badly when MS gets away without it in Windows? It's an MS file system. One of the contributors on the Fedora list remarked that this explained why his USB keys were so slow on Linux compared to Windows.

Well perhaps what windows does is write the hole file, then update the fat, then call sync immediately afterwards, or whenever a file is closed, it calls sync on that file's information.

Quoted 2 lines

Already filed bug reports with RedHat. On both HAL and on the
documentation for mount.

Well that is where the problem has to be solved.

Quoted 10 lines

Oh? It's of some use now? Beyond degrading performance to the point
that it takes many times longer to write a flash system in Linux than in Windows? Yes, I know, don't use it. What's it there for, then? To protect idiots from removing drives before they're unmounted? Those are the same idiots who will not know what happened when it burns up there drive and they are the least likely to use that option overtly. Most people don't realize that it was being used without their knowledge (much less the risks associated with using it in the first place) and just thought it was a Linux problem that it took many times longer to write USB keys in Linux than in Windows.

It does help make sure data is written to fat drives right away, although I don't ever think that should be enabled per filesystem myself, some people seem to think it is the way to do things. I think an application should be responsible for calling sync on a file when it considers it important. If a filesystem doesn't honour that it is broken (and yes many filesystems probably are broken that way).

Quoted 11 lines

Rule number one should be "do no harm". Maybe it should be under some
"force" flag or something just warning people not to use it unless they really REALLY know what their doing. I, personally, would never have used it under any circumstances. I was shocked to discover that RedHat was doing this by default. A warning to THEM would have been better than a warning to me, since I already knew not to do stupid shit like that. Not sure who is responsible for that HAL package or what other distros may also be vulnerable to this. I mostly blame RedHat for this problem first plus the mount man pages (which I suspect they replied on) second but I question if there shouldn't be a better way to avoid destroying hardware here.

flash drives have only been around a few years, and I guess in this case redhat made a mistake. I think any filesystem with sync mount option would be very bad for flash drives. This is not a fat problem, it's a problem with how sync works with flash drives in general. What do you think would happen if you formated your flash disk with ext3 and mounted it with sync? It would die real quick too.

Removing proper sync option is not a solution just because you could do harm. Should we remove write access to the disk just because you might let users run rm -rf /? Isn't that allowing harm to be done by the user too?

Len Sorensen

#4 lore raw
Quoted 10 lines · Lennart Sorensen
Quoted 8 lines · Michael H. Warfield

Yeah, I've heard that claimed but a real problem is that nobody can
tell which ones do or don't until you've got a crispy chip. I've looked and I haven't been able to find a single reference on any that you might pick up at Best Buy or Fry's. I've never seen one that has it written on it that it has any sort of wear leveling like that. But I guess we don't all buy our flash drives at an industrial supply house (forget Tiger Direct or CDW either - consumer grade). I would bet that most people don't even REALIZE that flash has a limited life and wears out.

Quoted 5 lines

I believe any flash with a sandisk controller in it will do pretty good wear leveling (I think all the ones labeled sandisk have their controller, although they don't all have sandisk flash chips). The controller is what does the wear leveling. At least for CF and SD this is what they do.

Funny you should mention SanDisk. Maybe the newer ones have gotten
better but the CF cards I used in my PDA were mixes of the SanDisk 128 Meg and SimpleTech 128 Meg. Burned up several before I realized it was the nightly backup program that was eating them and I didn't notice any difference between the brands. I don't know what controller was in the SimpleTech CF cards. Any way to tell short of dismantling them?

	:
Quoted 1 line

Len Sorensen

Mike

Signature · 4 lines
 Michael H. Warfield    |  (770) 985-6132   |  [email protected]  
  /\/\|=mhw=|\/\/       |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!
#5 lore raw
Quoted 6 lines · Michael H. Warfield

Funny you should mention SanDisk. Maybe the newer ones have gotten
better but the CF cards I used in my PDA were mixes of the SanDisk 128 Meg and SimpleTech 128 Meg. Burned up several before I realized it was the nightly backup program that was eating them and I didn't notice any difference between the brands. I don't know what controller was in the SimpleTech CF cards. Any way to tell short of dismantling them?

Not really. I believe sandisk has wear leveling on the 201 series CF cards and on their new generation CF/SD for sure they have it (and unfortunately for us they discontinued industrial temperature in the new line so we have had to look elsewhere for CF cards).

Unfortunately a lot of what is sold to consumers at retail is cheap crap. :)

Len Sorensen

#6 lore raw

Re: Flash device types

Quoted 7 lines · Lennart Sorensen

Not really. I believe sandisk has wear leveling on the 201 series CF cards and on their new generation CF/SD for sure they have it (and unfortunately for us they discontinued industrial temperature in the new line so we have had to look elsewhere for CF cards).

Unfortunately a lot of what is sold to consumers at retail is cheap crap. :)

It does seem to be a problem finding out how things really work in these devices. There seem to be the following types (from worst to best):

1. No wear leveling. Bad blocks are mapped out at manuf. time and that is it.
2. Bad blocks are detected and remapped dynamically. This is sometimes called wear-leveling, but the device life is a function of how many spares there originally were.
3. "Real" wear leveling. This can move data to fully use the life of all sectors.
4. "Real" wear leveling with lots of optimization and write cache - these are large devices usually with the ability to have battery power to ensure write cache can be flushed out.

#4 is easy to determine because of the size and complexity of the things. The others are much harder to distinguish and it really is important to know what you are dealing with. If you can't find out how it works, I would assume #1 or #2 which are both pretty poor.

It really would be nice to easily find out what category of device these things really are.

Signature · 1 line
Mark Rustad, [email protected]
#7 lore raw
In reply to Lennart Sorensen
Quoted 8 lines · Lennart Sorensen
Quoted 6 lines · Michael H. Warfield

Funny you should mention SanDisk. Maybe the newer ones have gotten
better but the CF cards I used in my PDA were mixes of the SanDisk 128 Meg and SimpleTech 128 Meg. Burned up several before I realized it was the nightly backup program that was eating them and I didn't notice any difference between the brands. I don't know what controller was in the SimpleTech CF cards. Any way to tell short of dismantling them?

Quoted 4 lines

Not really. I believe sandisk has wear leveling on the 201 series CF cards and on their new generation CF/SD for sure they have it (and unfortunately for us they discontinued industrial temperature in the new line so we have had to look elsewhere for CF cards).

Par for the course...

Quoted 2 lines

Unfortunately a lot of what is sold to consumers at retail is cheap crap. :)

You won't get any argument from me there!

Quoted 1 line

Len Sorensen

Latest things I've just started playing with are these "Intelligent
Sticks" or "I Sticks". They look like a chip, similar in form factor to the memory sticks, but slide into the open half of a USB jack, even though the contact pads wouldn't look like a good match. No connector shell, so they're really flat. I wonder just how "Intelligent" they are. I'm bet'n that wear leveling ain't part of it. But they are cute chips that fit in a wallet easily and can boot a fully encrypted laptop. Once working, they'll almost never need rewriting to any large extent. The 1G chips can hold a complete OS like Knoppix or Basilisk (the 512Meg chips too, with a lot of squeezing).

Mike

Signature · 4 lines
 Michael H. Warfield    |  (770) 985-6132   |  [email protected]  
  /\/\|=mhw=|\/\/       |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!
#8 lore raw
In reply to Lennart Sorensen
Quoted 9 lines · Lennart Sorensen
Quoted 4 lines · Michael H. Warfield

I'm also going to file a couple of bug reports in bugzilla at RedHat
but this seems to be a more fundamental problem than a RedHat specific problem. But, IMHO, they should never be setting that damn sync flag arbitrarily.

No they certainly should not, but it may have something to do with making life easier for kde/gnome desktops and automatic mount/umount of media. Dumb idea still, but that happens sometimes.

Err, no, the sync flag is a wonderful idea. I use sync on jump drives (although I won't now that I've read about this problem) because I can copy a file to it and as soon as the green light stops flashing I can yank it out without bothering to click or type anything to "eject" it. Requiring "eject" on removable media is a dumb idea.

Perhaps FAT updates could be cached and delayed until some # of ms after the last write.

Turning the sync flag off and adjusting the values
in /proc/sys/vm/dirty_* would have the same effect, but would change it on all devices, not just removables.

Maybe we need a way to set dirty_* per block device?

Signature · 1 line
Zan Lynx <[email protected]>
#9 lore raw
Quoted 6 lines · Zan Lynx

Err, no, the sync flag is a wonderful idea. I use sync on jump drives (although I won't now that I've read about this problem) because I can copy a file to it and as soon as the green light stops flashing I can yank it out without bothering to click or type anything to "eject" it. Requiring "eject" on removable media is a dumb idea.

Quoted 8 lines

Perhaps FAT updates could be cached and delayed until some # of ms after the last write.

Turning the sync flag off and adjusting the values
in /proc/sys/vm/dirty_* would have the same effect, but would change it on all devices, not just removables.

Maybe we need a way to set dirty_* per block device?

Well you could just make a way to start writing data to the device within 1s or so, and then assuming your device has a light indicating activity (it better), then when the light has been off for a couple of seconds, you can remove it.

I wish every removeable media drive had a soft eject button on it and a locking mechanism like zip drive, jax drive, cdrom/dvd, mac floppy drive, etc. They are seriously needed to tell the os what the user is trying to do to the poor media. That way the os can see when the user wants to take the media out, flush it, unlock and eject. Doesn't help if the eject button doesn't signal the OS of course, but at least on some drives it does. Being able to remove media from a system without telling the OS first is just bad design.

Len Sorensen

#10 lore raw
In reply to Michael H. Warfield
Quoted 5 lines

What happens, with the sync option on a VFAT file system, is that the
FAT tables are getting pounded and over-written over and over and over again as each and every block/cluster is allocated while a new file is written out. This constant overwriting eventually wears out the first block or two of the flash drive.

All non-shite quality flash keys have an on media log structured file system and will take 100,000+ writes per sector or so. They decent ones also map out bad blocks and have spares. The "wear out the same sector" stuff is a myth except on ultra-crap devices.

Quoted 4 lines

I'm also going to file a couple of bug reports in bugzilla at RedHat
but this seems to be a more fundamental problem than a RedHat specific problem. But, IMHO, they should never be setting that damn sync flag arbitrarily.

It sounds like your need to find a vendor who makes decent keys. For that matter several vendors now offer life time guarantees with their USB flash media.

Sync gets set by RH because it seemed the right thing to do to handle random user device pulls. Now O_SYNC works so excessively well on fat/vfat that needs looking at - and as you say likewise perhaps the nature of the FAT rewriting.

However its not a media issue, its primarily a performance issue.

Alan

#11 lore raw

Hey Alan,

Quoted 10 lines · Alan Cox
Quoted 5 lines

What happens, with the sync option on a VFAT file system, is that the
FAT tables are getting pounded and over-written over and over and over again as each and every block/cluster is allocated while a new file is written out. This constant overwriting eventually wears out the first block or two of the flash drive.

All non-shite quality flash keys have an on media log structured file system and will take 100,000+ writes per sector or so. They decent ones also map out bad blocks and have spares. The "wear out the same sector" stuff is a myth except on ultra-crap devices.

That's easy enough to say but AFAICT there doesn't seem to be any easy
well to tell the good from the bad from the just plain ugly. I typically don't buy junk (I didn't think), but I've definitely experienced this, first hand, with Sony Memory Vaults, SanDisk CF cards, some SimpleTech CF cards, some SmartMedia cards (what an oximoron that is), and now this 1G USB stick (which was, I admit, an "off brand" I had never heard of before at Frys Electronics). The CF cards were burned up in a PDA, so it's not just this. For a myth, I've definitely seen too much of it.

Strangely (and in response to another comment someone made) some USB
cards which I reformated for ext2 have survived quite well.

Quoted 8 lines
Quoted 4 lines

I'm also going to file a couple of bug reports in bugzilla at RedHat
but this seems to be a more fundamental problem than a RedHat specific problem. But, IMHO, they should never be setting that damn sync flag arbitrarily.

It sounds like your need to find a vendor who makes decent keys. For that matter several vendors now offer life time guarantees with their USB flash media.

Now THAT I gotta check into. I never noticed anything on the packaging
about a guarantee, but I will now. But how do you determine which are "decent" keys? They don't put stickers on them saying "this one is decent" and "this one is junk" and I'm an old cynic who has learned that price is not always a good indicator either. Maybe the guarantee will be a clue. I've just got to shop for it more.

Quoted 6 lines

Sync gets set by RH because it seemed the right thing to do to handle random user device pulls. Now O_SYNC works so excessively well on fat/vfat that needs looking at - and as you say likewise perhaps the nature of the FAT rewriting.

However its not a media issue, its primarily a performance issue.

Yeah, several of us have noticed the performance issue!

Quoted 1 line

Alan

Mike

Signature · 4 lines
 Michael H. Warfield    |  (770) 985-6132   |  [email protected]  
  /\/\|=mhw=|\/\/       |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!
#12 lore raw
Quoted 6 lines · Michael H. Warfield
Quoted 3 lines

It sounds like your need to find a vendor who makes decent keys. For that matter several vendors now offer life time guarantees with their USB flash media.

Now THAT I gotta check into. I never noticed anything on the packaging
about a guarantee, but I will now.

Most of them have guarantees of some form (this URL might be useful since it lists the guarantee times for a lot of the media - EU guarantees anyway, US often seem to be a lot different)

http://www.valuemedia.co.uk/compact_flash.htm

As you'll see Lexar for example offer lifetime guarantees on their units. I believe Kingston also do.

Quoted 5 lines

But how do you determine which are
"decent" keys? They don't put stickers on them saying "this one is decent" and "this one is junk" and I'm an old cynic who has learned that price is not always a good indicator either. Maybe the guarantee will be a clue. I've just got to shop for it more.

Or it may even be cheaper to "burn" a few - buy one of each type from various shops, do 2 million writes to the same sector and take them back the next day if they died [And publish the review data 8))]

Alan

#13 lore raw
Quoted 8 lines · Alan Cox
Quoted 6 lines · Michael H. Warfield
Quoted 3 lines

It sounds like your need to find a vendor who makes decent keys. For that matter several vendors now offer life time guarantees with their USB flash media.

Now THAT I gotta check into. I never noticed anything on the
packaging about a guarantee, but I will now.

Quoted 8 lines

Most of them have guarantees of some form (this URL might be useful since it lists the guarantee times for a lot of the media - EU guarantees anyway, US often seem to be a lot different)

http://www.valuemedia.co.uk/compact_flash.htm

As you'll see Lexar for example offer lifetime guarantees on their units. I believe Kingston also do.

Isn't a lifetime guarantee terminated by definition at the instant the device dies?

Quoted 9 lines
Quoted 5 lines

But how do you determine which are
"decent" keys? They don't put stickers on them saying "this one is decent" and "this one is junk" and I'm an old cynic who has learned that price is not always a good indicator either. Maybe the guarantee will be a clue. I've just got to shop for it more.

Or it may even be cheaper to "burn" a few - buy one of each type from various shops, do 2 million writes to the same sector and take them back the next day if they died [And publish the review data 8))]

It's probably a good idea to get from different shops, or someone might get suspicious when you take them back.

Signature · 2 lines
Måns Rullgård
[email protected]
#14 lore raw
Quoted 10 lines · Måns Rullgård
Quoted 8 lines · Alan Cox
Quoted 5 lines · Michael H. Warfield

But how do you determine which are
"decent" keys? They don't put stickers on them saying "this one is decent" and "this one is junk" and I'm an old cynic who has learned that price is not always a good indicator either. Maybe the guarantee will be a clue. I've just got to shop for it more.

Quoted 6 lines
Quoted 3 lines

Or it may even be cheaper to "burn" a few - buy one of each type from various shops, do 2 million writes to the same sector and take them back the next day if they died [And publish the review data 8))]

It's probably a good idea to get from different shops, or someone might get suspicious when you take them back.

Not really. I doubt the person in the shop will care.

Incidentally, I've discovered that you really don't want to buy flash devices from camera shops. It would seem that, like I guess might be the case with certain "audio CD" blanks you buy in stores, they don't seem to care about selling you a device with one or two known-bad sectors. You won't notice 512 bytes of lost data in many JPEG images (not that I am saying it's a good practice to have) but my Zaurus /did/ notice when I couldn't reflash it from a CF card with a fault somewhere around 8MB. After quite some time of screwing around with the filesystem by hand, I was able to ensure that something else was occupying the sector in question and eventually was able to reflash (all because it was a Sunday afternoon and we have silly Sunday trading laws here).

When I took the CF card back to the camera shop, I took the Zaurus and did a test on potential replacement cards while in the store - explaining that my PDA had a "CompactFlash tester" installed, or something like that. It's amazing what you can get away with - akin to spending an hour in the store when I got my replacement Powerbook after finding a single bad pixel, testing every unit to find one without any, just to be happy :-)

Jon.

#15 lore raw
In reply to Alan Cox
Quoted 32 lines · Alan Cox
Quoted 9 lines · Michael H. Warfield
Quoted 5 lines

It sounds like your need to find a vendor who makes decent keys. For that matter several vendors now offer life time guarantees with their USB flash media.

Now THAT I gotta check into. I never noticed anything on the packaging
about a guarantee, but I will now.

Most of them have guarantees of some form (this URL might be useful since it lists the guarantee times for a lot of the media - EU guarantees anyway, US often seem to be a lot different)

http://www.valuemedia.co.uk/compact_flash.htm

As you'll see Lexar for example offer lifetime guarantees on their units. I believe Kingston also do.

Quoted 7 lines

But how do you determine which are
"decent" keys? They don't put stickers on them saying "this one is decent" and "this one is junk" and I'm an old cynic who has learned that price is not always a good indicator either. Maybe the guarantee will be a clue. I've just got to shop for it more.

Quoted 6 lines

Or it may even be cheaper to "burn" a few - buy one of each type from various shops, do 2 million writes to the same sector and take them back the next day if they died [And publish the review data 8))]

If someone has a contact at the Linux Journal (or other magazine) it might be good to suggest this for an article.

Signature · 1 line
Jeffrey Hundstad
#16 lore raw
Quoted 2 lines · Jeffrey Hundstad

If someone has a contact at the Linux Journal (or other magazine) it might be good to suggest this for an article.

I'll look into it for an upcoming feature in LU&D magazine, if we deem it interesting enough to the readership. Certainly, I'd enjoy trashing a load of CF cards.

Jon.

([email protected])

#17 lore raw
In reply to Alan Cox
Quoted 4 lines · Alan Cox

Or it may even be cheaper to "burn" a few - buy one of each type from various shops, do 2 million writes to the same sector and take them back the next day if they died [And publish the review data 8))]

Or just accept the fact that flashes are a tad different from spinning rust. Expecting a decent wear levelling on the cheap USB sticks and other forms of flash is plain unrealistic.

USB stick are a bit better than old 3.5" floppies were - if both are used with fat, minix, ext, etc. At least they don't die by lying in a dark drawer. But if you want them to last, your best bet is currently to use JFFS2 on them.

Of course, JFFS2 sucks performance-wise, so the end result currently is that USB sticks suck in some way, no matter what you try.

Jörn

Signature · 3 lines
Fools ignore complexity.  Pragmatists suffer it.
Some can avoid it.  Geniuses remove it.
-- Perlis's Programming Proverb #58, SIGPLAN Notices, Sept.  1982
#18 lore raw
In reply to Alan Cox

Hey Alan...

Quoted 10 lines · Alan Cox
Quoted 5 lines

What happens, with the sync option on a VFAT file system, is that the
FAT tables are getting pounded and over-written over and over and over again as each and every block/cluster is allocated while a new file is written out. This constant overwriting eventually wears out the first block or two of the flash drive.

All non-shite quality flash keys have an on media log structured file system and will take 100,000+ writes per sector or so. They decent ones also map out bad blocks and have spares. The "wear out the same sector" stuff is a myth except on ultra-crap devices.

Yah know... I've been thinking about this... In a former life, we use
to do something very similar with a virtual memory system on some real early (80's vintage) networked VM workstations (back when memory was actually valuable and scarce).

So... This would have to work with a list or pool of "spares" that are
not allocated to the "visible" file system. We used a "least used" algorithm for that VM system. This would seem to be a "replace as rewritten" algorithm. Each time you write to the file system, it grabs a block off the head of the spares list, writes your data to it, and then adds the old block to the tail of the list. Pretty basic stuff and it doesn't have to track what kind of high level file system you are using or know anything about its structure. Cool...

That makes sense. But... How big might this "list" be? Maybe an
additional 10% of the entire drive capacity? That's quite a bit... But now you're beating on that FAT table pretty heavy. For each block allocated and written to, we're rewriting the FAT table (actually TWO FAT tables if you count the back up FAT). Ok... One data block, two FAT table rewrites. So a FAT table block gets added back to the list and a block is grabbed off the list. Seems like there would be a pretty high percentage of old FAT table blocks sitting there circulating on the spares list. That would make the probability of grabbing an old FAT table block and rewriting it again pretty high. Then it would get added back to the list again, in turn.

Because of this systematic thumping of the FAT tables, these old FAT
blocks are going to be circulating in that spares list at a pretty high density. The wear leveling is not going to be nearly as effective BECAUSE of the thumping. I'm not certain if that will be better or worse if there are more blocks in the spares list. Seems like you are going to end up with 50% - 60% (WAG) of the blocks in the spares list being old FAT table blocks and end up with a number that just keep recirculating until they burn out. I would think that they'll burn out faster if that spares list is small and they get reused more frequently (note to follow).

The up side is that, once an beat up old FAT table block does get
allocated to a file data block, it gets to retire in comfort and not get rewritten until the file gets rewritten. But... That's reducing the pool of circulating blocks in the allocated file system... So, a file system that's full is going to rotate through it's spare and free blocks faster as well... Some pluses... Some minuses...

It would seem like this would work well for something like a camera (or
a Mars Rover) where you are periodically removing almost everything from the flash memory and all the blocks have a chance to return to the spares list. But I see lots of possibilities for degrading the wear leveling in other cases...

Now... Flaw recovery could be a big help there. Write the block but
notice that the old one is now bad and don't add it back or the new one failed and you grab another. But then your spares list shrinks. Failure occurs on the first failure where the spares list hits zero. Probability (in the FAT thumping case with the sync option) is that it's going to be a FAT block that takes the hit and takes then entire drive out.

Am I seeing this correctly? Seems to me that the wear leveling is not
going to be nearly as effective as it should in the case where we are beating up on the FAT simply because of this systematic bias the sync option introduces into the write patterns on a FAT file system. And that will be aggravated by significant load of static data. If anything, the "sync" option almost appears to be defeating the wear leveling logic on FAT and VFAT file systems.

Quoted 9 lines
Quoted 4 lines

I'm also going to file a couple of bug reports in bugzilla at RedHat
but this seems to be a more fundamental problem than a RedHat specific problem. But, IMHO, they should never be setting that damn sync flag arbitrarily.

It sounds like your need to find a vendor who makes decent keys. For that matter several vendors now offer life time guarantees with their USB flash media.

Quoted 8 lines

Sync gets set by RH because it seemed the right thing to do to handle random user device pulls. Now O_SYNC works so excessively well on fat/vfat that needs looking at - and as you say likewise perhaps the nature of the FAT rewriting.

However its not a media issue, its primarily a performance issue.

Alan

Mike

Signature · 4 lines
 Michael H. Warfield    |  (770) 985-6132   |  [email protected]  
  /\/\|=mhw=|\/\/       |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!
#19 lore raw
Quoted 5 lines · Michael H. Warfield

Yah know... I've been thinking about this... In a former life, we use
to do something very similar with a virtual memory system on some real early (80's vintage) networked VM workstations (back when memory was actually valuable and scarce).

Quoted 8 lines

So... This would have to work with a list or pool of "spares" that are
not allocated to the "visible" file system. We used a "least used" algorithm for that VM system. This would seem to be a "replace as rewritten" algorithm. Each time you write to the file system, it grabs a block off the head of the spares list, writes your data to it, and then adds the old block to the tail of the list. Pretty basic stuff and it doesn't have to track what kind of high level file system you are using or know anything about its structure. Cool...

Really good wearleveling will even move blocks that "never" seem to change to the more used blocks ones in a while to spread out the wear to blocks that have static content in them. After all if 90% of your flash never changes, and you run a log in the last 10%, you will still wear out that 10% first if you don't occationally move some of the static content to the 10% with some wear, and start running your log on the previously unused area.

I was told by someone from SanDisk that this is how _some_ of their flash media work (at least on the new ones).

I was actually surprised since I assumed at the time this was how all of the CF cards worked.

Len Sorensen

#20 lore raw
In reply to Michael H. Warfield
Quoted 8 lines · Michael H. Warfield

Under the right circumstances, even copying a single file to a flash
drive mounted with the "sync" option can destroy the entire drive!

Now that I have your attention!

I found this out the hard way. (Kissed one brand new $70 USD 1GB flash
drive good-bye.) According to the man pages for mount, FAT and VFAT file systems ignore the "sync" option. It lies.

I guess you found out the hard way that the vast majority of Linux docs are 2-3 years out of date...

Quoted 5 lines

On a real hard
drive, this will cause "head resonances" as the heads go through constant high speed seeks between the cylinder with the FAT tables and the data cylinders. That can't be good, on a continuous basis, for drive life. But it's really a disaster for flash memory.

I have seen a clueless sysadmin destroy several 15,000 RPM SCSI drives this way by putting the syslog partition and mail spool at opposite ends of the drive. I think Alan Cox said something like "these days you can no longer assume that buggy software won't destroy your hardware".

Lee

#21 lore raw
Quoted 2 lines · Lee Revell

I guess you found out the hard way that the vast majority of Linux docs are 2-3 years out of date...

The man pages are normally way better than that, and in this case its only weeks out of date, assuming the current pages havent fixed it and he doesnt
have old manual pages..

#22 lore raw
In reply to Michael H. Warfield
Quoted 26 lines · Michael H. Warfield

Under the right circumstances, even copying a single file to a flash
drive mounted with the "sync" option can destroy the entire drive!

Now that I have your attention!

I found this out the hard way. (Kissed one brand new $70 USD 1GB flash
drive good-bye.) According to the man pages for mount, FAT and VFAT file systems ignore the "sync" option. It lies. Maybe it use to be true, but it certainly lies now. A simple test can verify this. Mount a flash drive with a FAT/VFAT file system without the sync option and writes to the drive go very fast. Typing "sync" or unmounting the drive afterwards, takes time as the buffered data is written to the flash drive. This is as it should be. Mount it with the sync option and writes are really REALLY slow (worse than they should be just from copying the data through USB) but sync and umount come back immediately and result in no additional writing to the drive. [Do the preceding with only a few files and less than a few meg of data if you value that flash.] So... FAT and VFAT are honoring the sync option. This is very VERY bad. It's bad for floppies, it's bad for hard drives, it's FATAL for flash drives.

Flash drives have a limited number of write cycles. Many many
thousands of write cycles, but limited, none the less. They are also written in blocks which are much larger than the "sector" size report (several K in a physical nand flash block, IRC).

Quoted 5 lines

What happens, with the sync option on a VFAT file system, is that the
FAT tables are getting pounded and over-written over and over and over again as each and every block/cluster is allocated while a new file is written out. This constant overwriting eventually wears out the first block or two of the flash drive.

What we really need, is a less thorough version of O_SYNC. O_SYNC currently guarantees that when syscall returns, data is on media (or at least in disk drive's internal cache :).

This is exactly what really paranoid people want. Journalling labels, all that good stuff.

But there are many cases where people just want to say 'write out dirty data asap for this device', so that I can copy files to floppy, wait till it stops making noise, and eject a disk. Samr for flash if it has write indicator (mine has a diode).

The difference from O_SYNC is that in this O_LAZY_SYNC mode writes return early, just filling pagecache, but writeout is started immediately and continues until there is no dirty O_LAZY_SYNC data.

This mode won't eat flash as fast as O_SYNC does. --
vda

#23 lore raw

All flashcards (other than dumb "smart media" cards) have integrated NAND controllers which perform automatic page/block remapping and which implement various wear-leveling algorithms. Rewriting "Sector 0" 10000 times probably only writes once to the first sector of a 1GB card. The other writes are spread around the rest of the card, and remapped logically by the integrated controller.

Linux could be more clever about it all, though. Wear-leveling can only be done efficiently on "unused" or "rewritten" blocks/pages on the cards, and not so well with areas that hold large static data files (from the point of view of the flash controller, not the O/S).

If we were really clever about it, then when Linux deletes a file from a flashcard device, it would also issue CFA ERASE commands for the newly freed sectors. This would let the card's controller know that it can remap/reuse that area of the card as it sees fit.

But it's dubious that a *short term* use (minutes/hours) of O_SYNC would have killed a new 1GB card.

Cheers

#24 lore raw
Quoted 6 lines · Mark Lord

All flashcards (other than dumb "smart media" cards) have integrated NAND controllers which perform automatic page/block remapping and which implement various wear-leveling algorithms. Rewriting "Sector 0" 10000 times probably only writes once to the first sector of a 1GB card. The other writes are spread around the rest of the card, and remapped logically by the integrated controller.

Assuming the firmware of the card is written with a modicum of clue, this is true. It's not clear how valid that assumption is, in the general case. There are reports of cards behaving as if they have almost no wear levelling at all.

Quoted 9 lines

Linux could be more clever about it all, though. Wear-leveling can only be done efficiently on "unused" or "rewritten" blocks/pages on the cards, and not so well with areas that hold large static data files (from the point of view of the flash controller, not the O/S).

If we were really clever about it, then when Linux deletes a file from a flashcard device, it would also issue CFA ERASE commands for the newly freed sectors. This would let the card's controller know that it can remap/reuse that area of the card as it sees fit.

This would be extremely useful, yes. I've said in the past that I want this for the benefit of the purely software flash translation layers (FTL, NFTL etc.). I hadn't realised that CF cards expose the same functionality.

Signature · 1 line
dwmw2
#25 lore raw

Hi!

Quoted 11 lines
Quoted 6 lines

All flashcards (other than dumb "smart media" cards) have integrated NAND controllers which perform automatic page/block remapping and which implement various wear-leveling algorithms. Rewriting "Sector 0" 10000 times probably only writes once to the first sector of a 1GB card. The other writes are spread around the rest of the card, and remapped logically by the integrated controller.

Assuming the firmware of the card is written with a modicum of clue, this is true. It's not clear how valid that assumption is, in the general case. There are reports of cards behaving as if they have almost no wear levelling at all.

I have seen card marked "3.3V/5V", but it only really worked on 3.3V. Linux used 5V and quickly killed it.
Pavel

Signature · 1 line
Boycott Kodak -- for their patent abuse against Java.
#26 lore raw
In reply to Mark Lord
Quoted 17 lines · Mark Lord

All flashcards (other than dumb "smart media" cards) have integrated NAND controllers which perform automatic page/block remapping and which implement various wear-leveling algorithms. Rewriting "Sector 0" 10000 times probably only writes once to the first sector of a 1GB card. The other writes are spread around the rest of the card, and remapped logically by the integrated controller.

Linux could be more clever about it all, though. Wear-leveling can only be done efficiently on "unused" or "rewritten" blocks/pages on the cards, and not so well with areas that hold large static data files (from the point of view of the flash controller, not the O/S).

If we were really clever about it, then when Linux deletes a file from a flashcard device, it would also issue CFA ERASE commands for the newly freed sectors. This would let the card's controller know that it can remap/reuse that area of the card as it sees fit.

Quoted 4 lines

But it's dubious that a *short term* use (minutes/hours) of O_SYNC would have killed a new 1GB card.

Cheers

CompactFlash(tm) like Sandisk and PNY do not write directly to the 'flash' part of the device. Instead, the page-oriented structure, copies the contents of the current page into static RAM. It is only when the page gets changed that the contents of static RAM get written before the actual page- change occurs. The write to flash-RAM is preceded by an erase, which sets all bits, because the flash-RAM bits can only be reset by writing.

These flash-RAM devices are designed to emulate IDE/ATA disk drives so no special software is required except that the software must not fail when the flash-RAM device doesn't respond to a "hardware" command. Some versions of Linux issue large amounts of error messages when accessing these devices. Other versions have very long, annoying, time-outs. Nevertheless, all Linux versions I have tried in resent times will work once the initialization process that occurs during boot is over.

We have systems that mount these CompactFlash(tm) devices R/W, without denying ATIME, and have not had any field failures. Recent design reviews have required that we mount these NOATIME, but it was only a guess that there may be problems many years into the future.

Regular NAND flash-RAM, installed on PC/Boards have been written many more times than the 100k guarantee. The first observation of a "wearing" mechanism is that the erase-cycle for programming takes a shorter time. This means that it takes less time to program the flash-RAM than it did when it was new. NAND flash-RAM fails in an interesting way. Some bits that were low, slowly drift high. This means that sometimes it will be read correctly and sometimes not. With failed NAND flash-RAM, the erase-cycle becomes very short. It will still program okay, but in a few days one will have problems consistently reading correct data.

Therefore, I believe that the end-of-life could be determined if the manufacturer just published end-of-life erase times.

The original poster did not tell how the flash-RAM fails in his system. If he did not encounter the wear-out mechanism I described it is likely that there is an electrical problem that is killing his flash-RAM.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.11 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#27 lore raw
In reply to Denis Vlasenko
Quoted 5 lines · Denis Vlasenko

What we really need, is a less thorough version of O_SYNC. O_SYNC currently guarantees that when syscall returns, data is on media (or at least in disk drive's internal cache :).

Quoted 9 lines

This is exactly what really paranoid people want. Journalling labels, all that good stuff.

But there are many cases where people just want to say 'write out dirty data asap for this device', so that I can copy files to floppy, wait till it stops making noise, and eject a disk. Samr for flash if it has write indicator (mine has a diode).

I don't really see the need for a new mode. Mount without o_sync, but use the sync command once to get things written out. Or
use the umount command before ejecting, it syncs the device before returning. I use umount all the time when using compactflash. The wear is minimal.

Helge Hafting

#28 lore raw
Quoted 15 lines · Helge Hafting
Quoted 5 lines · Denis Vlasenko

What we really need, is a less thorough version of O_SYNC. O_SYNC currently guarantees that when syscall returns, data is on media (or at least in disk drive's internal cache :).

Quoted 9 lines

This is exactly what really paranoid people want. Journalling labels, all that good stuff.

But there are many cases where people just want to say 'write out dirty data asap for this device', so that I can copy files to floppy, wait till it stops making noise, and eject a disk. Samr for flash if it has write indicator (mine has a diode).

Quoted 6 lines

I don't really see the need for a new mode. Mount without o_sync, but use the sync command once to get things written out. Or
use the umount command before ejecting, it syncs the device before returning. I use umount all the time when using compactflash. The wear is minimal.

I just want this to happen automatically
(automounter helps with this) and at once
(i.e. without delay prior to start of writeout).

Caching (delaying) writes for hard disks makes tons of sense, but not as much for removable media.
--
vda

#29 lore raw
In reply to Michael H. Warfield

Hi,

Quoted 3 lines

According to the man pages for mount, FAT and VFAT
file systems ignore the "sync" option. It lies. Maybe it use to be true, but it certainly lies now.

Yes, it does lie. I'm the author of the O_SYNC patch for fat and vfat, and I'd like to point out that I did test a few flash drives (and hard drives) extensively (during about a week) with this flag, and they did not die on me.
As other people said, I think the O_SYNC handling does exactly what it says, and if it doesn't do what you want, tell HAL not to set it (There must be a way).

Signature · 1 line
Colin