ext2/3: document conditions when reliable operation is possible

Quoted 11 lines · Ric Wheeler

Quoted 8 lines

Drive in raid 5 failed; hot spare was available (no idea about UPS). System apparently locked up trying to talk to the failed drive, or maybe admin just was not patient enough, so he just powercycled the array. He lost the array.

So while most people will not agressively powercycle the RAID array, drive failure still provokes little tested error paths, and getting unclean shutdown is quite easy in such case.

Then what we need to document is do not power cycle an array during a rebuild, right?

Well, the softwar raid layer could be improved so that it implements scrubbing by default (i.e., have the md package install a cron job to implement a periodict scrub pass automatically). The MD code could also regularly check to make sure the hot spare is OK; the other possibility is that hot spare, which hadn't been used in a long time, had silently failed.

Quoted 6 lines

In the end, there are cascading failures that will defeat any data protection scheme, but that does not mean that the value of that scheme is zero. We need to be get more people to use RAID (including MD5) and try to enhance it as we go. Just using a single disk is not a good thing...

Yep; the solution is to improve the storage devices. It is *not* to encourage people to think RAID is not worth it, or that somehow ext2 is better than ext3 because it runs fsck's all the time at boot up. That's just crazy talk.

- Ted

Quoted 11 lines · Ric Wheeler

Quoted 8 lines

Drive in raid 5 failed; hot spare was available (no idea about UPS). System apparently locked up trying to talk to the failed drive, or maybe admin just was not patient enough, so he just powercycled the array. He lost the array.

So while most people will not agressively powercycle the RAID array, drive failure still provokes little tested error paths, and getting unclean shutdown is quite easy in such case.

Then what we need to document is do not power cycle an array during a rebuild, right?

Well, the softwar raid layer could be improved so that it implements scrubbing by default (i.e., have the md package install a cron job to implement a periodict scrub pass automatically). The MD code could also regularly check to make sure the hot spare is OK; the other possibility is that hot spare, which hadn't been used in a long time, had silently failed.

Quoted 6 lines

In the end, there are cascading failures that will defeat any data protection scheme, but that does not mean that the value of that scheme is zero. We need to be get more people to use RAID (including MD5) and try to enhance it as we go. Just using a single disk is not a good thing...

Yep; the solution is to improve the storage devices. It is *not* to encourage people to think RAID is not worth it, or that somehow ext2 is better than ext3 because it runs fsck's all the time at boot up. That's just crazy talk.

- Ted

Quoted 13 lines · Theodore Tso

Quoted 11 lines · Ric Wheeler

Quoted 8 lines

Drive in raid 5 failed; hot spare was available (no idea about UPS). System apparently locked up trying to talk to the failed drive, or maybe admin just was not patient enough, so he just powercycled the array. He lost the array.

So while most people will not agressively powercycle the RAID array, drive failure still provokes little tested error paths, and getting unclean shutdown is quite easy in such case.

Then what we need to document is do not power cycle an array during a rebuild, right?

Quoted 6 lines

Well, the softwar raid layer could be improved so that it implements scrubbing by default (i.e., have the md package install a cron job to implement a periodict scrub pass automatically). The MD code could also regularly check to make sure the hot spare is OK; the other possibility is that hot spare, which hadn't been used in a long time, had silently failed.

Actually, MD does this scan already (not automatically, but you can set up a simple cron job to kick off a periodic "check"). It is a delicate balance to get the frequency of the scrubbing correct.

Well, the softwar raid layer could be improved so that it implements scrubbing by default (i.e., have the md package install a cron job to implement a periodict scrub pass automatically).

The issues being raised here are not specific to extX, MD RAID, or Linux at all; they are problems with non-"enterprise-class" RAID setups. There's a reason enterprise-class RAID costs a lot more money than the card you can pick up at Fry's.

Quoted 4 lines

The issues being raised here are not specific to extX, MD RAID, or Linux at all; they are problems with non-"enterprise-class" RAID setups. There's a reason enterprise-class RAID costs a lot more money than the card you can pick up at Fry's.

And you will still need backups ;)

A long time ago I worked on a fault tolerant news server with dual alphaserver boxes and a shared disk array. A power system failure took out both the alpha boxes and the disk controllers and all the disks.

The other thing about this thread is that the only RAID implementation that is being discussed here is the MD RAID stack. There are a lot of RAID implementations that have the same issues:

- motherboard (aka "fake") RAID - In Linux this is typically mapped with device mapper via dmraid; AFAIK there is not a tool to scrub (or even monitor the status of and notify on failure) a Linux DM RAID setup.

- hardware RAID cards without battery backup - these have the exact same issues because they cannot guarantee all writes complete, nor can they keep track of incomplete writes across power failures

- hardware RAID cards _with_ battery backup but that don't periodically test the battery and have a way to notify you of battery failure while Linux is running

The issues being raised here are not specific to extX, MD RAID, or Linux at all; they are problems with non-"enterprise-class" RAID setups. There's a reason enterprise-class RAID costs a lot more money than the card you can pick up at Fry's.

There's no reason to document the design issues of general RAID implementations in the Linux kernel.

Quoted 7 lines

Example I seen went like this:

Drive in raid 5 failed; hot spare was available (no idea about UPS). System apparently locked up trying to talk to the failed drive, or maybe admin just was not patient enough, so he just powercycled the array. He lost the array.

Quoted 3 lines

So while most people will not agressively powercycle the RAID array, drive failure still provokes little tested error paths, and getting unclean shutdown is quite easy in such case.

Then what we need to document is do not power cycle an array during a rebuild, right?

If it wasn't the admin that timed out and the box really was hung (no drive activity lights, etc), you will need to power cycle/reboot but then you should not have this active rebuild issuing writes either...

Look, I don't need full drive failure for this to happen. I can just remove one disk from array. I don't need power failure, I can just press the power button. I don't even need to rebuild anything, I can just write to degraded array.

Given that all events are under my control, statistics make little sense here.

Quoted 7 lines · Pavel Machek

Look, I don't need full drive failure for this to happen. I can just remove one disk from array. I don't need power failure, I can just press the power button. I don't even need to rebuild anything, I can just write to degraded array.

Given that all events are under my control, statistics make little sense here.

I recommend a sledgehammer.

If you want to lose your data, you might as well have some fun.

No need to bore yourself to tears by simulating events that are unlikely to happen simultaneously to careful system administrators.

Sledgehammer is hardware problem, and I'm demonstrating software/documentation problem we have here.

Quoted 2 lines · Pavel Machek

Sledgehammer is hardware problem, and I'm demonstrating software/documentation problem we have here.

So your argument is that a sledgehammer is a hardware problem, while a broken hard disk and a power failure are software/documentation issues?

I'd argue that the broken hard disk and power failure are hardware issues, too.

Quoted 8 lines · Rik van Riel

Quoted 2 lines · Pavel Machek

Sledgehammer is hardware problem, and I'm demonstrating software/documentation problem we have here.

So your argument is that a sledgehammer is a hardware problem, while a broken hard disk and a power failure are software/documentation issues?

Quoted 2 lines

I'd argue that the broken hard disk and power failure are hardware issues, too.

Noone told me that degraded md raid5 is dangerous. Thats documentation issue #1. Maybe I just pulled the disk for fun.

Quoted 2 lines · Pavel Machek

Noone told me that degraded md raid5 is dangerous. Thats documentation issue #1. Maybe I just pulled the disk for fun.

You're kidding, right?

It is not only for system administrators; I was trying to find out if kernel is buggy, and that should be in kernel tree.

There are storage devices that high highly undesirable properties when they are disconnected or suffer power failures while writes are in progress; such devices include flash devices and MD RAID 4/5/6 arrays.

These devices have the property of potentially corrupting blocks being written at the time of the power failure,

and worse yet, amplifying the region where blocks are corrupted such that additional sectors are also damaged during the power failure.

Users who use such storage devices are well advised take countermeasures, such as the use of Uninterruptible Power Supplies, and making sure the flash device is not hot-unplugged while the device is being used. Regular backups when using these devices is also a Very Good Idea.

Otherwise, file systems placed on these devices can suffer silent data and file system corruption. An forced use of fsck may detect metadata corruption resulting in file system corruption, but will not suffice to detect data corruption.

Quoted 4 lines · Pavel Machek

There are storage devices that high highly undesirable properties when they are disconnected or suffer power failures while writes are in progress; such devices include flash devices and MD RAID 4/5/6 arrays.

change this to say 'degraded MD RAID 4/5/6 arrays'

also find out if DM RAID 4/5/6 arrays suffer the same problem (I strongly suspect that they do)

then you need to add a note that if the array becomes degraded before a scrub cycle happens previously hidden damage (that would have been repaired by the scrub) can surface.

Quoted 2 lines

THESE devices have the property of potentially corrupting blocks being written at the time of the power failure,

this is true of all devices

Quoted 11 lines · On Tue 2009-08-25 16:56:40, [email protected] wrote:

Quoted 4 lines · Pavel Machek

There are storage devices that high highly undesirable properties when they are disconnected or suffer power failures while writes are in progress; such devices include flash devices and MD RAID 4/5/6 arrays.

change this to say 'degraded MD RAID 4/5/6 arrays'

also find out if DM RAID 4/5/6 arrays suffer the same problem (I strongly suspect that they do)

I changed it to say MD/DM.

Quoted 3 lines

then you need to add a note that if the array becomes degraded before a scrub cycle happens previously hidden damage (that would have been repaired by the scrub) can surface.

I'd prefer not to talk about scrubing and such details here. Better leave warning here and point to MD documentation.

Quoted 4 lines

Quoted 2 lines

THESE devices have the property of potentially corrupting blocks being written at the time of the power failure,

this is true of all devices

Actually I don't think so. I believe SATA disks do not corrupt even the sector they are writing to -- they just have big enough capacitors. And yes I believe ext3 depends on that.

Quoted 19 lines · Pavel Machek

Quoted 11 lines · On Tue 2009-08-25 16:56:40, [email protected] wrote:

Quoted 4 lines · Pavel Machek

There are storage devices that high highly undesirable properties when they are disconnected or suffer power failures while writes are in progress; such devices include flash devices and MD RAID 4/5/6 arrays.

change this to say 'degraded MD RAID 4/5/6 arrays'

also find out if DM RAID 4/5/6 arrays suffer the same problem (I strongly suspect that they do)

I changed it to say MD/DM.

Quoted 3 lines

then you need to add a note that if the array becomes degraded before a scrub cycle happens previously hidden damage (that would have been repaired by the scrub) can surface.

Quoted 2 lines

I'd prefer not to talk about scrubing and such details here. Better leave warning here and point to MD documentation.

I disagree with that, the way you are wording this makes it sound as if raid isn't worth it. if you are going to say that raid is risky you need to properly specify when it is risky

Quoted 5 lines

Quoted 4 lines

Quoted 2 lines

THESE devices have the property of potentially corrupting blocks being written at the time of the power failure,

this is true of all devices

Quoted 3 lines

Actually I don't think so. I believe SATA disks do not corrupt even the sector they are writing to -- they just have big enough capacitors. And yes I believe ext3 depends on that.

you are incorrect on this.

ext3 (like every other filesystem) just accepts the risk (zfs makes some attempt to detect such corruption)

Quoted 24 lines · On Tue 2009-08-25 17:20:13, [email protected] wrote:

Quoted 21 lines · Pavel Machek

Quoted 11 lines · On Tue 2009-08-25 16:56:40, [email protected] wrote:

Quoted 4 lines · Pavel Machek

There are storage devices that high highly undesirable properties when they are disconnected or suffer power failures while writes are in progress; such devices include flash devices and MD RAID 4/5/6 arrays.

change this to say 'degraded MD RAID 4/5/6 arrays'

also find out if DM RAID 4/5/6 arrays suffer the same problem (I strongly suspect that they do)

I changed it to say MD/DM.

Quoted 3 lines

then you need to add a note that if the array becomes degraded before a scrub cycle happens previously hidden damage (that would have been repaired by the scrub) can surface.

I'd prefer not to talk about scrubing and such details here. Better leave warning here and point to MD documentation.

Quoted 3 lines

I disagree with that, the way you are wording this makes it sound as if raid isn't worth it. if you are going to say that raid is risky you need to properly specify when it is risky

Ok, would this help? I don't really want to go to scrubbing details.

(*) Degraded array or single disk failure "near" the powerfail is neccessary for this property of RAID arrays to bite.

Quoted 11 lines · On Tue 2009-08-25 16:56:40, [email protected] wrote:

Quoted 4 lines · Pavel Machek

There are storage devices that high highly undesirable properties when they are disconnected or suffer power failures while writes are in progress; such devices include flash devices and MD RAID 4/5/6 arrays.

change this to say 'degraded MD RAID 4/5/6 arrays'

also find out if DM RAID 4/5/6 arrays suffer the same problem (I strongly suspect that they do)

I changed it to say MD/DM.

Quoted 3 lines

then you need to add a note that if the array becomes degraded before a scrub cycle happens previously hidden damage (that would have been repaired by the scrub) can surface.

I'd prefer not to talk about scrubing and such details here. Better leave warning here and point to MD documentation.

Quoted 5 lines

Quoted 3 lines

THESE devices have the property of potentially corrupting blocks being written at the time of the power failure,

this is true of all devices

Actually I don't think so. I believe SATA disks do not corrupt even the sector they are writing to -- they just have big enough capacitors. And yes I believe ext3 depends on that.
Pavel

Quoted 9 lines

Quoted 5 lines

Quoted 3 lines

THESE devices have the property of potentially corrupting blocks being written at the time of the power failure,

this is true of all devices

Actually I don't think so. I believe SATA disks do not corrupt even the sector they are writing to -- they just have big enough capacitors. And yes I believe ext3 depends on that.

Pavel, no S-ATA drive has capacitors to hold up during a power failure (or even enough power to destage their write cache). I know this from direct, personal knowledge having built RAID boxes at EMC for years. In fact, almost all RAID boxes require that the write cache be hardwired to off when used in their arrays.

Quoted 10 lines

Quoted 9 lines

Quoted 5 lines

Quoted 3 lines

THESE devices have the property of potentially corrupting blocks being written at the time of the power failure,

this is true of all devices

Actually I don't think so. I believe SATA disks do not corrupt even the sector they are writing to -- they just have big enough capacitors. And yes I believe ext3 depends on that.

Quoted 5 lines

Pavel, no S-ATA drive has capacitors to hold up during a power failure (or even enough power to destage their write cache). I know this from direct, personal knowledge having built RAID boxes at EMC for years. In fact, almost all RAID boxes require that the write cache be hardwired to off when used in their arrays.

I never claimed they have enough power to flush entire cache -- read the paragraph again. I do believe the disks have enough capacitors to finish writing single sector, and I do believe ext3 depends on that.

Pavel

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

Quoted 2 lines

THESE devices have the property of potentially corrupting blocks being written at the time of the power failure,

this is true of all devices

Actually I don't think so. I believe SATA disks do not corrupt even the sector they are writing to -- they just have big enough capacitors. And yes I believe ext3 depends on that.

Quoted 5 lines

Pavel, no S-ATA drive has capacitors to hold up during a power failure (or even enough power to destage their write cache). I know this from direct, personal knowledge having built RAID boxes at EMC for years. In fact, almost all RAID boxes require that the write cache be hardwired to off when used in their arrays.

I never claimed they have enough power to flush entire cache -- read the paragraph again. I do believe the disks have enough capacitors to finish writing single sector, and I do believe ext3 depends on that.

Quoted 15 lines · Pavel Machek

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

Quoted 2 lines

THESE devices have the property of potentially corrupting blocks being written at the time of the power failure,

this is true of all devices

Actually I don't think so. I believe SATA disks do not corrupt even the sector they are writing to -- they just have big enough capacitors. And yes I believe ext3 depends on that.

Quoted 5 lines

Pavel, no S-ATA drive has capacitors to hold up during a power failure (or even enough power to destage their write cache). I know this from direct, personal knowledge having built RAID boxes at EMC for years. In fact, almost all RAID boxes require that the write cache be hardwired to off when used in their arrays.

Quoted 3 lines

I never claimed they have enough power to flush entire cache -- read the paragraph again. I do believe the disks have enough capacitors to finish writing single sector, and I do believe ext3 depends on that.

keep in mind that in a powerfail situation the data being sent to the drive may be corrupt (the ram gets flaky while a DMA to the drive copies the bad data to the drive, which writes it before the power loss gets bad enough for the drive to decide there is a problem and shutdown)

you just plain cannot count on writes that are in flight when a powerfail happens to do predictable things, let alone what you consider sane or proper.

From what I see, this kind of failure is rather harder to reproduce

Quoted 3 lines

you just plain cannot count on writes that are in flight when a powerfail happens to do predictable things, let alone what you consider sane or proper.

From what I see, this kind of failure is rather harder to reproduce than the software problems. And at least SGI machines were designed to avoid this...

Anyway, I'd like to hear from ext3 people... what happens on read errors in journal? That's what you'd expect to see in situation above.

Quoted 10 lines · Pavel Machek

Quoted 3 lines

you just plain cannot count on writes that are in flight when a powerfail happens to do predictable things, let alone what you consider sane or proper.

From what I see, this kind of failure is rather harder to reproduce than the software problems. And at least SGI machines were designed to avoid this...

Anyway, I'd like to hear from ext3 people... what happens on read errors in journal? That's what you'd expect to see in situation above.

On a power failure, what normally happens is that the random garbage gets written into the disk drive's last dying gasp, since the memory starts going insane and sends garbage to the disk. So the disk successfully completes the write, but the sector contains garbage. Since HDD's tend to be last thing to die, being less sensitive to voltage drops than the memory or DMA controller, my experience is that you don't get a read error after the system comes up, you just get garbage written into the journal.

The ext3 journalling code waits until all of the journal code is written, and only then writes the commit block. On restart, we look for the last valid commit block. So if the power failure is before we write the commit block, we replay the journal up until the previous commit block. If the power failure is while we are writing the commit block, garbage will be written out instead of the commit block, and so it falls back to the previous case.

We do not allow any updates to the filesystem metadata to take place until the commit block has been written; therefore the filesystem stays consistent.

If there the journal *does* develop read errors, then fsck will require a manual fsck, and so the boot operation will get stopped so a system administrator can provide manual intervention. The best bet for the sysadmin is to replay as much of the journal she can, and then let fsck fix any resulting filesystem inconsistencies. In practice,

Quoted 10 lines · Pavel Machek

Quoted 3 lines

you just plain cannot count on writes that are in flight when a powerfail happens to do predictable things, let alone what you consider sane or proper.

From what I see, this kind of failure is rather harder to reproduce than the software problems. And at least SGI machines were designed to avoid this...

Anyway, I'd like to hear from ext3 people... what happens on read errors in journal? That's what you'd expect to see in situation above.

On a power failure, what normally happens is that the random garbage gets written into the disk drive's last dying gasp, since the memory starts going insane and sends garbage to the disk. So the disk successfully completes the write, but the sector contains garbage. Since HDD's tend to be last thing to die, being less sensitive to voltage drops than the memory or DMA controller, my experience is that you don't get a read error after the system comes up, you just get garbage written into the journal.

The ext3 journalling code waits until all of the journal code is written, and only then writes the commit block. On restart, we look for the last valid commit block. So if the power failure is before we write the commit block, we replay the journal up until the previous commit block. If the power failure is while we are writing the commit block, garbage will be written out instead of the commit block, and so it falls back to the previous case.

We do not allow any updates to the filesystem metadata to take place until the commit block has been written; therefore the filesystem stays consistent.

If there the journal *does* develop read errors, then fsck will require a manual fsck, and so the boot operation will get stopped so a system administrator can provide manual intervention. The best bet for the sysadmin is to replay as much of the journal she can, and then let fsck fix any resulting filesystem inconsistencies. In practice,

Lets say you are writing to the (healthy) RAID5 and have a powerfail.

So now data blocks do not correspond to the parity block. You don't yet have the corruption, but you already have a problem.

If you get a disk failing at this point, you'll get corruption.

And if your
claim is that several hundred lines of fsck output detailing the filesystem's destruction somehow makes things all better, I suspect most users would disagree with you.

In any case, depending on where the flash was writing at the time of the unplug, the data corruption could be silent anyway.

Maybe this came as a surprise to you, but anyone who has used a compact flash in a digital camera knows that you ***have*** to wait until the led has gone out before trying to eject the flash card.

I
remember seeing all sorts of horror stories from professional photographers about how they lost an important wedding's day worth of pictures with the attendant commercial loss, on various digital photography forums. It tends to be the sort of mistake that digital photographers only make once.

(It's worse with people using Digital SLR's shooting in raw mode, since it can take upwards of 30 seconds or more to write out a 12-30MB raw image, and if you eject at the wrong time, you can trash the contents of the entire CF card; in the worst case, the Flash Translation Layer data can get corrupted, and the card is completely ruined; you can't even reformat it at the filesystem level, but have to get a special Windows program from the CF manufacturer to --maybe-- reset the FTL layer.

Early CF cards were especially vulnerable to
this; more recent CF cards are better, but it's a known failure mode of CF cards.)

Quoted 9 lines · Theodore Tso

(It's worse with people using Digital SLR's shooting in raw mode, since it can take upwards of 30 seconds or more to write out a 12-30MB raw image, and if you eject at the wrong time, you can trash the contents of the entire CF card; in the worst case, the Flash Translation Layer data can get corrupted, and the card is completely ruined; you can't even reformat it at the filesystem level, but have to get a special Windows program from the CF manufacturer to --maybe-- reset the FTL layer.

This just goes to show why having this "translation layer" done in firmware on the device itself is a _bad_ idea. We're much better off when we have full access to the underlying flash and the OS can actually see what's going on. That way, we can actually debug, fix and recover from such problems.

Quoted 3 lines

Early CF cards were especially vulnerable to
this; more recent CF cards are better, but it's a known failure mode of CF cards.)

It's a known failure mode of _everything_ that uses flash to pretend to be a block device. As I see it, there are no SSD devices which don't lose data; there are only SSD devices which haven't lost your data _yet_.

There's no fundamental reason why it should be this way; it just is.

(I'm kind of hoping that the shiny new expensive ones that everyone's talking about right now, that I shouldn't really be slagging off, are actually OK. But they're still new, and I'm certainly not trusting them with my own data _quite_ yet.)

Quoted 26 lines · David Woodhouse

Quoted 9 lines · Theodore Tso

(It's worse with people using Digital SLR's shooting in raw mode, since it can take upwards of 30 seconds or more to write out a 12-30MB raw image, and if you eject at the wrong time, you can trash the contents of the entire CF card; in the worst case, the Flash Translation Layer data can get corrupted, and the card is completely ruined; you can't even reformat it at the filesystem level, but have to get a special Windows program from the CF manufacturer to --maybe-- reset the FTL layer.

This just goes to show why having this "translation layer" done in firmware on the device itself is a _bad_ idea. We're much better off when we have full access to the underlying flash and the OS can actually see what's going on. That way, we can actually debug, fix and recover from such problems.

Quoted 3 lines

Early CF cards were especially vulnerable to
this; more recent CF cards are better, but it's a known failure mode of CF cards.)

It's a known failure mode of _everything_ that uses flash to pretend to be a block device. As I see it, there are no SSD devices which don't lose data; there are only SSD devices which haven't lost your data _yet_.

Quoted 6 lines

There's no fundamental reason why it should be this way; it just is.

(I'm kind of hoping that the shiny new expensive ones that everyone's talking about right now, that I shouldn't really be slagging off, are actually OK. But they're still new, and I'm certainly not trusting them with my own data _quite_ yet.)

so what sort of test would be needed to identify if a device has this problem?

people can do ad-hoc tests by pulling the devices in use and then checking the entire device, but something better should be available.

it seems to me that there are two things needed to define the tests.

1. a predictable write load so that it's easy to detect data getting lose

2. some statistical analysis to decide how many device pulls are needed (under the write load defined in #1) to make the odds high that the problem will be revealed.

for USB devices there may be a way to use the power management functions to cut power to the device without requiring it to physically be pulled, if this is the case (even if this only works on some specific chipsets), it would drasticly speed up the testing

Quoted 8 lines · On Fri 2009-08-28 07:46:42, [email protected] wrote:

so what sort of test would be needed to identify if a device has this problem?

people can do ad-hoc tests by pulling the devices in use and then checking the entire device, but something better should be available.

Quoted 7 lines

it seems to me that there are two things needed to define the tests.

1. a predictable write load so that it's easy to detect data getting lose

2. some statistical analysis to decide how many device pulls are needed (under the write load defined in #1) to make the odds high that the problem will be revealed.

Its simpler than that. It usually breaks after third unplug or so.

Quoted 4 lines

for USB devices there may be a way to use the power management functions to cut power to the device without requiring it to physically be pulled, if this is the case (even if this only works on some specific chipsets), it would drasticly speed up the testing

This is really so easy to reproduce, that such speedup is not neccessary. Just try the scripts :-).

Quoted 2 lines

This is really so easy to reproduce, that such speedup is not neccessary. Just try the scripts :-).

so if it doesn't get corrupted after 5 unplugs does that mean that that particular device doesn't have a problem? or does it just mean you got lucky?

would 10 sucessful unplugs mean that it's safe?

what about 20?

I'll note, BTW, that AIX uses a journal to protect against these sorts of problems with software raid; this also means that with AIX, you also don't have to rebuild a RAID 1 device after an unclean shutdown, like you have do with Linux MD. This was on the EVMS's team development list to implement for Linux, but it got canned after LVM won out, lo those many years ago.

Quoted 6 lines · Pavel Machek

Quoted 3 lines

I have to admit that I have not paid enough attention to this specifics of your ext3 + flash card issue - is it the ftl stuff doing out of order IO's?

The problem is that flash cards destroy whole erase block on unplug, and ext3 can't cope with that.

Sure --- but name **any** filesystem that can deal with the fact that 128k or 256k worth of data might disappear when you pull out the flash card while it is writing a single sector?

Quoted 11 lines · On Monday August 24, [email protected] wrote:

Quoted 6 lines · Pavel Machek

Quoted 3 lines

I have to admit that I have not paid enough attention to this specifics of your ext3 + flash card issue - is it the ftl stuff doing out of order IO's?

The problem is that flash cards destroy whole erase block on unplug, and ext3 can't cope with that.

Sure --- but name **any** filesystem that can deal with the fact that 128k or 256k worth of data might disappear when you pull out the flash card while it is writing a single sector?

A Log structured filesystem could certainly be written to deal with such a situation, providing by 'deal with' you mean 'only loses data that has not yet been acknowledged to the application'. Of course the filesystem would need clear visibility into exactly how these blocks are positioned.

I've been playing with just such a filesystem for some time (never really finding enough time) with the goal of making it work over RAID5 with no data risk due to power loss. One day it will be functional enough for others to try....

It is entirely possible that NILFS could be made to meet that requirement, but I haven't made time to explore NILFS so I cannot be sure.

NeilBrown

Quoted 26 lines · Neil Brown

Quoted 7 lines · On Monday August 24, [email protected] wrote:

Quoted 6 lines · Pavel Machek

Quoted 5 lines

I have to admit that I have not paid enough attention to this specifics
of your ext3 + flash card issue - is it the ftl stuff doing out of order
IO's?

Quoted 6 lines

Quoted 2 lines

The problem is that flash cards destroy whole erase block on unplug, and ext3 can't cope with that.

Sure --- but name **any** filesystem that can deal with the fact that 128k or 256k worth of data might disappear when you pull out the flash card while it is writing a single sector?

A Log structured filesystem could certainly be written to deal with such a situation, providing by 'deal with' you mean 'only loses data that has not yet been acknowledged to the application'. Of course the filesystem would need clear visibility into exactly how these blocks are positioned.

I've been playing with just such a filesystem for some time (never really finding enough time) with the goal of making it work over RAID5 with no data risk due to power loss. One day it will be functional enough for others to try....

Quoted 6 lines

It is entirely possible that NILFS could be made to meet that requirement, but I haven't made time to explore NILFS so I cannot be sure.

NeilBrown

I am not sure that log structure will protect you from this scenario since once
you clean the log, the non-logged data is assumed to be correct.

If your cheap flash storage device can nuke random regions of that clean storage, you will lose data....

The papers show failures in "once a year" range. I have "twice a minute" failure scenario with flashdisks.

Not sure how often "degraded raid5 breaks ext3 atomicity" would bite, but I bet it would be on "once a day" scale.

Quoted 5 lines

The papers show failures in "once a year" range. I have "twice a minute" failure scenario with flashdisks.

Not sure how often "degraded raid5 breaks ext3 atomicity" would bite, but I bet it would be on "once a day" scale.

I agree it should be documented, but the ext3 atomicity issue is only an issue on unexpected shutdown while the array is degraded. I surely hope most people running raid5 are not seeing that level of unexpected shutdown, let along in a degraded array,

If they are, the atomicity issue pretty strongly says they should not be using raid5 in that environment. At least not for any filesystem I know. Having writes to LBA n corrupt LBA n+128 as an example is pretty hard to design around from a fs perspective.

Greg

Quoted 11 lines · On Monday 24 August 2009 16:11:56 Greg Freemyer wrote:

Quoted 5 lines

The papers show failures in "once a year" range. I have "twice a minute" failure scenario with flashdisks.

Not sure how often "degraded raid5 breaks ext3 atomicity" would bite, but I bet it would be on "once a day" scale.

I agree it should be documented, but the ext3 atomicity issue is only an issue on unexpected shutdown while the array is degraded. I surely hope most people running raid5 are not seeing that level of unexpected shutdown, let along in a degraded array,

Quoted 4 lines

If they are, the atomicity issue pretty strongly says they should not be using raid5 in that environment. At least not for any filesystem I know. Having writes to LBA n corrupt LBA n+128 as an example is pretty hard to design around from a fs perspective.

Right now, people think that a degraded raid 5 is equivalent to raid 0. As this thread demonstrates, in the power failure case it's _worse_, due to write granularity being larger than the filesystem sector size. (Just like flash.)

Knowing that, some people might choose to suspend writes to their raid until it's finished recovery. Perhaps they'll set up a system where a degraded raid 5 gets remounted read only until recovery completes, and then writes go to a new blank hot spare disk using all that volume snapshoting or unionfs stuff people have been working on. (The big boys already have hot spare disks standing by on a lot of these systems, ready to power up and go without human intervention. Needing two for actual reliability isn't that big a deal.)

Or maybe the raid guys might want to tweak the recovery logic so it's not entirely linear, but instead prioritizes dirty pages over clean ones. So if somebody dirties a page halfway through a degraded raid 5, skip ahead to recover that chunk first to the new disk first (yes leaving holes, it's not that hard to track), and _then_ let the write go through.

But unless people know the issue exists, they won't even start thinking about ways to address it.

Quoted 2 lines · Pavel Machek

Actually, ext2 should be able to survive that, no? Error writing -> remount ro -> fsck on next boot -> drive relocates the sectors.

I think that the example and the response are both off base. If your head ever touches the platter, you won't be reading from a huge part of your drive ever again

The only one that falls into that category is the one about not being able to handle failed writes, and the way most failures take place,

Quoted 2 lines

Isn't this by design? In other words, if the metadata doesn't survive non-atomic writes, wouldn't it be an ext3 bug?

Part of the problem here is that "atomic-writes" is confusing; it doesn't mean what many people think it means. The assumption which many naive filesystem designers make is that writes succeed or they don't. If they don't succeed, they don't change the previously existing data in any way.

Is that a file system "bug"? Well, it's better to call that a mismatch between the assumptions made of physical devices, and of the file system code. On Irix, SGI hardware had a powerfail interrupt, and the power supply and extra-big capacitors, so that when a power fail interrupt came in, the Irix would run around frantically shutting down pending DMA transfers to prevent this failure mode from causing problems. PC class hardware (according to Ted's law), is cr*p, and doesn't have a powerfail interrupt, so it's not something that we have.

Ext3, ext4, and ocfs2 does physical block journalling, so as long as journal truncate hasn't taken place right before the failure, the replay of the physical block journal tends to repair this most (but not necessarily all) cases of "garbage is written right before power failure". People who care about this should really use a UPS, and wire up the USB and/or serial cable from the UPS to the system, so that the OS can do a controlled shutdown if the UPS is close to shutting down due to an extended power failure.

Documentation/filesystems/ext3.txt debugfs: ext2 and ext3 file system debugger. · line 220

Quoted 7 lines

Documentation/filesystems/ext3.txt +7 -0

+* don't damage the old data on a failed write (ATOMIC-WRITES)
+
+	(Thrash may get written into sectors during powerfail.  And
+	ext3 handles this surprisingly well at least in the
+	catastrophic case of garbage getting written into the inode
+	table, since the journal replay often will "repair" the
+	garbage that was written into the filesystem metadata blocks.

Isn't this by design? In other words, if the metadata doesn't survive non-atomic writes, wouldn't it be an ext3 bug?

Quoted 10 lines

Documentation/filesystems/ext3.txt debugfs: ext2 and ext3 file system debugger. · line 220

Quoted 7 lines

Documentation/filesystems/ext3.txt +7 -0

+* don't damage the old data on a failed write (ATOMIC-WRITES)
+
+	(Thrash may get written into sectors during powerfail.  And
+	ext3 handles this surprisingly well at least in the
+	catastrophic case of garbage getting written into the inode
+	table, since the journal replay often will "repair" the
+	garbage that was written into the filesystem metadata blocks.

Isn't this by design? In other words, if the metadata doesn't survive non-atomic writes, wouldn't it be an ext3 bug?

So I got confused when I quoted your note, which I had assumed was exactly what Pavel had written in his documentation. In fact, what he had written was this:

Documentation/filesystems/expectations.txt new file +7 -0

+Don't damage the old data on a failed write (ATOMIC-WRITES)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Either whole sector is correctly written or nothing is written during
+powerfail.
+
+....

So he had explicitly stated that he only cared about the whole sector being written (or not written) in the power fail case, and not any other. I'd suggest changing ATOMIC-WRITES to ATOMIC-WRITE-ON-POWERFAIL, since the one-line summary, "Don't damage the old data on a failed write", is also singularly misleading.

Quoted 10 lines

Documentation/filesystems/ext3.txt debugfs: ext2 and ext3 file system debugger. · line 220

Quoted 7 lines

Documentation/filesystems/ext3.txt +7 -0

+* don't damage the old data on a failed write (ATOMIC-WRITES)
+
+	(Thrash may get written into sectors during powerfail.  And
+	ext3 handles this surprisingly well at least in the
+	catastrophic case of garbage getting written into the inode
+	table, since the journal replay often will "repair" the
+	garbage that was written into the filesystem metadata blocks.

Isn't this by design? In other words, if the metadata doesn't survive non-atomic writes, wouldn't it be an ext3 bug?

So I got confused when I quoted your note, which I had assumed was exactly what Pavel had written in his documentation. In fact, what he had written was this:

Documentation/filesystems/expectations.txt new file +7 -0

+Don't damage the old data on a failed write (ATOMIC-WRITES)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Either whole sector is correctly written or nothing is written during
+powerfail.
+
+....

So he had explicitly stated that he only cared about the whole sector being written (or not written) in the power fail case, and not any other. I'd suggest changing ATOMIC-WRITES to ATOMIC-WRITE-ON-POWERFAIL, since the one-line summary, "Don't damage the old data on a failed write", is also singularly misleading.

Documentation/filesystems/expectations.txt line 4

Quoted 4 lines

Documentation/filesystems/expectations.txt new file +4 -0

+Linux block-backed filesystems can only work correctly when several
+conditions are met in the block layer and below (disks, flash
+cards). Some of them are obvious ("data on media should not change
+randomly"), some are less so.

You should make clear that the file lists per-file-system rules and that some file sytems can recover from some of the error conditions.

Running journaling filesystem such as ext3 over flashdisk or degraded RAID array is a bad idea: journaling guarantees no longer apply and you will get data corruption on powerfail.

We can't solve it easily, but we should certainly warn the users. I actually lost data because I did not understand these limitations...

Signed-off-by Pavel Machek <[email protected]>

Documentation/filesystems/expectations.txt new file +39 -0

new file mode 100644
index 0000000..80fa886
@@ -0,0 +1,52 @@
+Linux block-backed filesystems can only work correctly when several
+conditions are met in the block layer and below (disks, flash
+cards). Some of them are obvious ("data on media should not change
+randomly"), some are less so.
+
+Write errors not allowed (NO-WRITE-ERRORS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes to media never fail. Even if disk returns error condition
+during write, filesystems can't handle that correctly.
+
+       Fortunately writes failing are very uncommon on traditional
+       spinning disks, as they have spare sectors they use when write
+       fails.
+
+Don't cause collateral damage to adjacent sectors on a failed write (NO-COLLATERALS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Unfortunately, cheap USB/SD flash cards I've seen do have this bug,
+and are thus unsuitable for all filesystems I know.
+
+       An inherent problem with using flash as a normal block device
+       is that the flash erase size is bigger than most filesystem
+       sector sizes.  So when you request a write, it may erase and
+       rewrite some 64k, 128k, or even a couple megabytes on the
+       really _big_ ones.
+
+       If you lose power in the middle of that, filesystem won't
+       notice that data in the "sectors" _around_ the one your were
+       trying to write to got trashed.
+
+       RAID-4/5/6 in degraded mode has same problem.
+
+
+Don't damage the old data on a failed write (ATOMIC-WRITES)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Either whole sector is correctly written or nothing is written during
+powerfail.

Documentation/filesystems/expectations.txt line 49

Quoted 13 lines

Documentation/filesystems/expectations.txt new file +12 -0

+Either whole sector is correctly written or nothing is written during
+powerfail.
+
+       Because RAM tends to fail faster than rest of system during
+       powerfail, special hw killing DMA transfers may be necessary;
+       otherwise, disks may write garbage during powerfail.
+       This may be quite common on generic PC machines.
+
+       Note that atomic write is very hard to guarantee for RAID-4/5/6,
+       because it needs to write both changed data, and parity, to
+       different disks. (But it will only really show up in degraded mode).
+       UPS for RAID array should help.

Can someone clarify if this is true in raid-6 with just a single disk failure? I don't see why it would be.

And if not can the above text be changed to reflect raid 4/5 with a single disk failure and raid 6 with a double disk failure are the modes that have atomicity problems.

Quoted 10 lines

Documentation/filesystems/expectations.txt new file +10 -0

+Don't damage the old data on a failed write (ATOMIC-WRITES)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Either whole sector is correctly written or nothing is written during
+powerfail.
+
+       Because RAM tends to fail faster than rest of system during
+       powerfail, special hw killing DMA transfers may be necessary;
+       otherwise, disks may write garbage during powerfail.
+       This may be quite common on generic PC machines.

Quoted 5 lines

+
+       Note that atomic write is very hard to guarantee for RAID-4/5/6,
+       because it needs to write both changed data, and parity, to
+       different disks. (But it will only really show up in degraded mode).
+       UPS for RAID array should help.

Can someone clarify if this is true in raid-6 with just a single disk failure? I don't see why it would be.

Quoted 10 lines · On Monday August 24, [email protected] wrote:

Quoted 10 lines

Documentation/filesystems/expectations.txt new file +10 -0

+Don't damage the old data on a failed write (ATOMIC-WRITES)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Either whole sector is correctly written or nothing is written during
+powerfail.
+
+       Because RAM tends to fail faster than rest of system during
+       powerfail, special hw killing DMA transfers may be necessary;
+       otherwise, disks may write garbage during powerfail.
+       This may be quite common on generic PC machines.

Quoted 8 lines

Quoted 5 lines

+
+       Note that atomic write is very hard to guarantee for RAID-4/5/6,
+       because it needs to write both changed data, and parity, to
+       different disks. (But it will only really show up in degraded mode).
+       UPS for RAID array should help.

Can someone clarify if this is true in raid-6 with just a single disk failure? I don't see why it would be.

It does affect raid6 with a single drive missing.

After an unclean shutdown you cannot trust any Parity block as it is possible that some of the blocks in the stripe have been updated, but others have not. So you must assume that all parity blocks are wrong and update them. If you have a missing disk you cannot do that.

To take a more concrete example, imagine a 5 device RAID6 with 3 data blocks D0 D1 D2 as well a P and Q on some stripe. Suppose that we crashed while updating D0, which would have involved writing out D0, P and Q.
On restart, suppose D2 is missing. It is possible that 0, 1, 2, or 3 of D0, P and Q have been updated and the others not.
We can try to recompute D2 from D0 D1 and P, from
D0 P and Q or from D1, P and Q.

We could conceivably try each of those and if they all produce the same result we might be confident of it.
If two produced the same result and the other was different we could use a voting process to choose the 'best'. And in this particular case I think that would work. If 0 or 3 had been updates, all would be the same. If only 1 was updated, then the combinations that exclude it will match. If 2 were updated, then the combinations that exclude the non-updated block will match.

But if both D0 and D1 were being updated I think there would be too many combinations and it would be very possibly that all three computed values for D2 would be different.

So yes: a singly degraded RAID6 cannot promise no data corruption after an unclean shutdown. That is why "mdadm" will not assemble such an array unless you use "--force" to acknowledge that there has been a problem.

Running journaling filesystem such as ext3 over flashdisk or degraded RAID array is a bad idea: journaling guarantees no longer apply and you will get data corruption on powerfail.

We can't solve it easily, but we should certainly warn the users. I actually lost data because I did not understand these limitations...

Signed-off-by Pavel Machek <[email protected]>

Quoted 8 lines · On Monday 24 August 2009 04:31:43 Pavel Machek wrote:

Running journaling filesystem such as ext3 over flashdisk or degraded RAID array is a bad idea: journaling guarantees no longer apply and you will get data corruption on powerfail.

We can't solve it easily, but we should certainly warn the users. I actually lost data because I did not understand these limitations...

Signed-off-by Pavel Machek <[email protected]>

Acked-by Rob Landley <[email protected]>

With a couple comments:

Documentation/filesystems/ext2.txt enough 4-character names to make up unique directory entries, so they · line 362

Quoted 2 lines

Documentation/filesystems/ext2.txt +2 -0

+* write caching is disabled. ext2 does not know how to issue barriers
+  as of 2.6.28. hdparm -W0 disables it on SATA disks.

It's coming up on 2.6.31, has it learned anything since or should that version number be bumped?

Documentation/filesystems/ext3.txt debugfs: ext2 and ext3 file system debugger. · line 226

Quoted 11 lines

Documentation/filesystems/ext3.txt +11 -0

+	(Thrash may get written into sectors during powerfail.  And
+	ext3 handles this surprisingly well at least in the
+	catastrophic case of garbage getting written into the inode
+	table, since the journal replay often will "repair" the
+	garbage that was written into the filesystem metadata blocks.
+	It won't do a bit of good for the data blocks, of course
+	(unless you are using data=journal mode).  But this means that
+	in fact, ext3 is more resistant to suriving failures to the
+	first problem (powerfail while writing can damage old data on
+	a failed write) but fortunately, hard drives generally don't
+	cause collateral damage on a failed write.

Possible rewording of this paragraph:

Ext3 handles trash getting written into sectors during powerfail surprisingly well. It's not foolproof, but it is resilient. Incomplete journal entries are ignored, and journal replay of complete entries will often "repair" garbage written into the inode table. The data=journal option extends this behavior to file and directory data blocks as well (without which your dentries can still be badly corrupted by a power fail during a write).

(I'm not entirely sure about that last bit, but clarifying it one way or the other would be nice because I can't tell from reading it which it is. My _guess_ is that directories are just treated as files with an attitude and an extra cacheing layer...?)

Quoted 14 lines · Rob Landley

Quoted 8 lines · On Monday 24 August 2009 04:31:43 Pavel Machek wrote:

Running journaling filesystem such as ext3 over flashdisk or degraded RAID array is a bad idea: journaling guarantees no longer apply and you will get data corruption on powerfail.

We can't solve it easily, but we should certainly warn the users. I actually lost data because I did not understand these limitations...

Signed-off-by Pavel Machek <[email protected]>

Acked-by Rob Landley <[email protected]>

With a couple comments:

Quoted 5 lines

Documentation/filesystems/ext2.txt enough 4-character names to make up unique directory entries, so they · line 362

Quoted 2 lines

Documentation/filesystems/ext2.txt +2 -0

+* write caching is disabled. ext2 does not know how to issue barriers
+  as of 2.6.28. hdparm -W0 disables it on SATA disks.

It's coming up on 2.6.31, has it learned anything since or should that version number be bumped?

Jan, did those "barrier for ext2" patches get merged?

Quoted 5 lines · On Thursday 12 March 2009 04:21:14 Pavel Machek wrote:

Not all block devices are suitable for all filesystems. In fact, some block devices are so broken that reliable operation is pretty much impossible. Document stuff ext2/ext3 needs for reliable operation.

Signed-off-by Pavel Machek <[email protected]>

Updated version here.

Quoted 4 lines · Rob Landley

Quoted 3 lines · On Thursday 12 March 2009 04:21:14 Pavel Machek wrote:

Not all block devices are suitable for all filesystems. In fact, some block devices are so broken that reliable operation is pretty much impossible. Document stuff ext2/ext3 needs for reliable operation.

Documentation/filesystems/expectations.txt new file +4 -0

+
+	Unfortunately, none of the cheap USB/SD flash cards I've seen
+	do behave like this, and are thus unsuitable for all Linux
+	filesystems I know.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Not all block devices are suitable for all filesystems. In fact, some block devices are so broken that reliable operation is pretty much impossible. Document stuff ext2/ext3 needs for reliable operation.

Some of what is here are bugs, and some are legitimate long-term interfaces (for example, the question of losing I/O errors when two processes are writing to the same file, or to a directory entry, and errors aren't or in some cases, can't, be reflected back to userspace).

I'm a little concerned that some of this reads a bit too much like a rant (and I know Pavel was very frustrated when he tried to use a flash card with a sucky flash card socket) and it will get used the

wrong way by apoligists, because it mixes areas where "we suck, we should do better", which a re bug reports, and "Posix or the underlying block device layer makes it hard", and simply states them as fundamental design requirements, when that's probably not true.

There's a lot of work that we could do to make I/O errors get better reflected to userspace by fsync(). So state things as bald requirements I think goes a little too far IMHO. We can surely do better.

Review

Quoted 6 lines

(quoted diff) +6 -0

+Write errors not allowed (NO-WRITE-ERRORS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes to media never fail. Even if disk returns error condition
+during write, filesystems can't handle that correctly, because success
+on fsync was already returned when data hit the journal.

The last half of this sentence "because success on fsync was already returned when data hit the journal", obviously doesn't apply to all filesystems, since some filesystems, like ext2, don't journal data. Even for ext3, it only applies in the case of data=journal mode.

There are other issues here, such as fsync() only reports an I/O problem to one caller, and in some cases I/O errors aren't propagated up the storage stack. The latter is clearly just a bug that should be fixed; the former is more of an interface limitation. But you don't talk about in this section, and I think it would be good to have a more extended discussion about I/O errors when writing data blocks, and I/O errors writing metadata blocks, etc.

Review

Quoted 6 lines

(quoted diff) +6 -0

+
+Sector writes are atomic (ATOMIC-SECTORS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Either whole sector is correctly written or nothing is written during
+powerfail.

This requirement is not quite the same as what you discuss below.

So there are actually two desirable properties for a storage system to have; one is "don't damage the old data on a failed write"; and the other is "don't cause collateral damage to adjacent sectors on a failed write".

Review

Quoted 4 lines

(quoted diff) +4 -0

+	Because RAM tends to fail faster than rest of system during 
+	powerfail, special hw killing DMA transfers may be necessary;
+	otherwise, disks may write garbage during powerfail.
+	Not sure how common that problem is on generic PC machines.

This problem is still relatively common, from what I can tell. And ext3 handles this surprisingly well at least in the catastrophic case of garbage getting written into the inode table, since the journal replay often will "repair" the garbage that was written into the

Documentation/filesystems/expectations.txt line 25

Quoted 3 lines · Pavel Machek

Documentation/filesystems/expectations.txt new file +3 -0

+	Unfortunately, none of the cheap USB/SD flash cards I've seen
+	do behave like this, and are thus unsuitable for all Linux
+	filesystems I know.

When you say Linux filesystems do you mean "filesystems originally designed on Linux" or do you mean "filesystems that Linux supports"? Additionally whatever the answer, people are going to need help answering the "which is the least bad?" question and saying what's not good without offering alternatives is only half helpful... People need to put SOMETHING on these cheap (and not quite so cheap) devices... The last recommendation I heard was that until btrfs/logfs/nilfs arrive people are best off sticking with FAT -
http://marc.info/?l=linux-kernel&m=122398315223323&w=2 . Perhaps that should be mentioned?

Documentation/filesystems/ext3.txt mke2fs: create a ext3 partition with the -j flag. · line 223

Quoted 7 lines

+* either write caching is disabled, or hw can do barriers and they are enabled. +

Documentation/filesystems/ext3.txt +5 -0

+	   (Note that barriers are disabled by default, use "barrier=1"
+	   mount option after making sure hw can support them).
+
+	   hdparm -I reports disk features. If you have "Native
+	   Command Queueing" is the feature you are looking for.

The document makes it sound like nearly everything bar battery backed hardware RAIDed SCSI disks (with perfect firmware) is bad - is this the intent?

Flash gets into trouble when it presents the _interface_ of rotational media (a USB block device with normal 512 byte read/write sectors, which never wear out) which doesn't match what the hardware's actually doing (erase block sizes of up to several megabytes at a time, hidden behind a block remapping layer for wear leveling).

For devices that have built in flash that DON'T pretend to be a conventional block device, but instead expose their flash erase granularity and let the OS do the wear levelling itself, we have special flash filesystems that can be reasonably reliable.  It's just that ext3 isn't one of them, jffs2 and ubifs and logfs are.  The problem with these flash filesystems is they ONLY work on flash, if you want to mount them on something other than flash you need something like a loopback interface to make a normal block device pretend to be flash.  (We've got a ramdisk driver called "mtdram" that does this, but nobody's bothered to write a generic wrapper for a normal block device you can wrap over the loopback driver.)

Documentation/filesystems/expectations.txt line 25

Quoted 3 lines · Pavel Machek

Documentation/filesystems/expectations.txt new file +3 -0

+	Unfortunately, none of the cheap USB/SD flash cards I've seen
+	do behave like this, and are thus unsuitable for all Linux
+	filesystems I know.

When you say Linux filesystems do you mean "filesystems originally designed on Linux" or do you mean "filesystems that Linux supports"?

Additionally whatever the answer, people are going to need help answering the "which is the least bad?" question and saying what's not good without offering alternatives is only half helpful... People need to put SOMETHING on these cheap (and not quite so cheap) devices... The

Documentation/filesystems/ext3.txt mke2fs: create a ext3 partition with the -j flag. · line 223

Quoted 2 lines

Documentation/filesystems/ext3.txt +2 -0

+	   hdparm -I reports disk features. If you have "Native
+	   Command Queueing" is the feature you are looking for.

The document makes it sound like nearly everything bar battery backed hardware RAIDed SCSI disks (with perfect firmware) is bad - is this the intent?

Documentation/filesystems/ext2.txt enough 4-character names to make up unique directory entries, so they · line 357

Quoted 2 lines

Documentation/filesystems/ext2.txt +2 -0

+* write caching is disabled. ext2 does not know how to issue barriers
+  as of 2.6.28. hdparm -W0 disables it on SATA disks.

And here we're talking about ext2. Does neither one know about write barriers, or does this just apply to ext2? (What about ext4?)

Also I remember a historical problem that not all disks honor write barriers, because actual data integrity makes for horrible benchmark numbers. Dunno how current that is with SATA, Alan Cox would probably know.

I've heard rumors of disks that claim to support cache flushes but really just ignore them, but have never heard any specifics of model numbers, etc. which are known to do this, so it may just be legend. If we do have such knowledge then we should really be blacklisting those drives and warning the user that we can't ensure data integrity. (Even powering down the system would be unsafe in this case.)

Documentation/filesystems/expectations.txt new file +9 -0

+Sector writes are atomic (ATOMIC-SECTORS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Either whole sector is correctly written or nothing is written during
+powerfail.
+
+       Unfortuantely, none of the cheap USB/SD flash cards I seen do
+       behave like this, and are unsuitable for all linux filesystems
+       I know.

On Thu, Mar 12, 2009 at 5:21 AM, Pavel Machek <[email protected]> wrote:
<snip>

Quoted 9 lines

Documentation/filesystems/expectations.txt new file +9 -0

+Sector writes are atomic (ATOMIC-SECTORS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Either whole sector is correctly written or nothing is written during
+powerfail.
+
+       Unfortuantely, none of the cheap USB/SD flash cards I seen do
+       behave like this, and are unsuitable for all linux filesystems
+       I know.

Documentation/filesystems/expectations.txt line 35

Quoted 10 lines

Documentation/filesystems/expectations.txt new file +10 -0

+
+               An inherent problem with using flash as a normal block
+               device is that the flash erase size is bigger than
+               most filesystem sector sizes.  So when you request a
+               write, it may erase and rewrite the next 64k, 128k, or
+               even a couple megabytes on the really _big_ ones.
+
+               If you lose power in the middle of that, filesystem
+               won't notice that data in the "sectors" _around_ the
+               one your were trying to write to got trashed.

I had *assumed* that SSDs worked like:

1) write request comes in
2) new unused erase block area marked to hold the new data
3) updated data written to the previously unused erase block
4) mapping updated to replace the old erase block with the new one

If it were done that way, a failure in the middle would just leave the SSD with the old data in it.