ext2/3: document conditions when reliable operation is possible

Hi,

2009/3/12 Pavel Machek <[email protected]>:

Quoted 8 lines

Documentation/filesystems/ext2.txt +0 -0

index 4333e83..b09aa4c 100644
@@ -338,27 +339,25 @@ enough 4-character names to make up unique directory entries, so they

 have to be 8 character filenames, even then we are fairly close to running out of unique filenames.

Documentation/filesystems/ext2.txt enough 4-character names to make up unique directory entries, so they · line 345

Quoted 4 lines

Documentation/filesystems/ext2.txt +4 -0

+Requirements
+============
+
+Ext3 expects disk/storage subsystem to behave sanely. On sanely

^^^^
Shouldn't this be "Ext2"?

Not all block devices are suitable for all filesystems. In fact, some block devices are so broken that reliable operation is pretty much impossible. Document stuff ext2/ext3 needs for reliable operation.

Signed-off-by Pavel Machek <[email protected]>

(quoted diff) +0 -0

b/Documentation/filesystems/expectations.txt new file mode 100644 index 0000000..9c3d729

Documentation/filesystems/expectations.txt new file +5 -0

@@ -0,0 +1,47 @@
+Linux block-backed filesystems can only work correctly when several
+conditions are met in the block layer and below (disks, flash
+cards). Some of them are obvious ("data on media should not change
+randomly"), some are less so.
+

Documentation/filesystems/expectations.txt new file +2 -0

+
+

(quoted diff) +0 -0

b/Documentation/filesystems/ext2.txt index 4333e83..b09aa4c 100644

Documentation/filesystems/ext2.txt +0 -0

@@ -338,27 +339,25 @@ enough 4-character names to make up unique directory

entries, so they have to be 8 character filenames, even then we are fairly close to running out of unique filenames.

Quoted 5 lines

Documentation/filesystems/expectations.txt new file +5 -0

+Write errors not allowed (NO-WRITE-ERRORS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes to media never fail. Even if disk returns error condition
+during write, filesystems can't handle that correctly, because success

Documentation/filesystems/expectations.txt line 15

Quoted 5 lines

Documentation/filesystems/expectations.txt new file +5 -0

+on fsync was already returned when data hit the journal.
+
+	Fortunately writes failing are very uncommon on traditional
+	spinning disks, as they have spare sectors they use when write
+	fails.

I vaguely recall that the behavior of when a write error _does_ occur is to remount the filesystem read only? (Is this VFS or per-fs?)

Is there any kind of hotplug event associated with this?

I'm aware write errors shouldn't happen, and by the time they do it's too late to gracefully handle them, and all we can do is fail. So how do we fail?

Documentation/filesystems/expectations.txt line 23

Quoted 1 line

Documentation/filesystems/expectations.txt new file +1 -0

+	Unfortuantely, none of the cheap USB/SD flash cards I seen do

I've seen

Documentation/filesystems/expectations.txt line 24

Quoted 1 line

Documentation/filesystems/expectations.txt new file +1 -0

+	behave like this, and are unsuitable for all linux filesystems

"are thus unsuitable", perhaps? (Too pretentious? :)

Documentation/filesystems/expectations.txt line 31

Quoted 7 lines

Documentation/filesystems/expectations.txt new file +7 -0

+	I know.
+
+		An inherent problem with using flash as a normal block
+		device is that the flash erase size is bigger than
+		most filesystem sector sizes.  So when you request a
+		write, it may erase and rewrite the next 64k, 128k, or
+		even a couple megabytes on the really _big_ ones.

Somebody corrected me, it's not "the next" it's "the surrounding".

(Writes aren't always cleanly at the start of an erase block, so critical data _before_ what you touch is endangered too.)

Documentation/filesystems/expectations.txt line 44

Quoted 6 lines

Documentation/filesystems/expectations.txt new file +6 -0

+	otherwise, disks may write garbage during powerfail.
+	Not sure how common that problem is on generic PC machines.
+
+	Note that atomic write is very hard to guarantee for RAID-4/5/6,
+	because it needs to write both changed data, and parity, to
+	different disks.

These days instead of "atomic" it's better to think in terms of "barriers".

Quoted 5 lines

Documentation/filesystems/ext2.txt +5 -0

+Requirements
+* write errors not allowed
+
+* sector writes are atomic
+

Documentation/filesystems/ext2.txt enough 4-character names to make up unique directory entries, so they · line 357

Quoted 5 lines

Documentation/filesystems/ext2.txt +5 -0

+(see expectations.txt; note that most/all linux block-based
+filesystems have similar expectations)
+
+* write caching is disabled. ext2 does not know how to issue barriers
+  as of 2.6.28. hdparm -W0 disables it on SATA disks.

And here we're talking about ext2. Does neither one know about write barriers, or does this just apply to ext2? (What about ext4?)

Also I remember a historical problem that not all disks honor write barriers, because actual data integrity makes for horrible benchmark numbers. Dunno how current that is with SATA, Alan Cox would probably know.

Hi!

Quoted 6 lines

Documentation/filesystems/expectations.txt line 15

Quoted 3 lines

Documentation/filesystems/expectations.txt new file +3 -0

+	Fortunately writes failing are very uncommon on traditional
+	spinning disks, as they have spare sectors they use when write
+	fails.

I vaguely recall that the behavior of when a write error _does_ occur is to remount the filesystem read only? (Is this VFS or per-fs?)

Per-fs.

Quoted 1 line

Is there any kind of hotplug event associated with this?

I don't think so.

Quoted 3 lines

I'm aware write errors shouldn't happen, and by the time they do it's too late to gracefully handle them, and all we can do is fail. So how do we fail?

Well, even remount-ro may be too late, IIRC.

Quoted 2 lines

(Writes aren't always cleanly at the start of an erase block, so critical data _before_ what you touch is endangered too.)

Well, flashes do remap, so it is actually "random blocks".

Quoted 9 lines

Documentation/filesystems/expectations.txt line 44

Quoted 6 lines

Documentation/filesystems/expectations.txt new file +6 -0

+	otherwise, disks may write garbage during powerfail.
+	Not sure how common that problem is on generic PC machines.
+
+	Note that atomic write is very hard to guarantee for RAID-4/5/6,
+	because it needs to write both changed data, and parity, to
+	different disks.

These days instead of "atomic" it's better to think in terms of "barriers".

This is not about barriers (that should be different topic). Atomic write means that either whole sector is written, or nothing at all is written. Because raid5 needs to update both master data and parity at the same time, I don't think it can guarantee this during powerfail.

Quoted 5 lines

Quoted 5 lines

Documentation/filesystems/ext2.txt +5 -0

+Requirements
+* write errors not allowed
+
+* sector writes are atomic
+

Quoted 8 lines

Documentation/filesystems/ext2.txt enough 4-character names to make up unique directory entries, so they · line 357

Quoted 5 lines

Documentation/filesystems/ext2.txt +5 -0

+(see expectations.txt; note that most/all linux block-based
+filesystems have similar expectations)
+
+* write caching is disabled. ext2 does not know how to issue barriers
+  as of 2.6.28. hdparm -W0 disables it on SATA disks.

And here we're talking about ext2. Does neither one know about write barriers, or does this just apply to ext2? (What about ext4?)

This document is about ext2. Ext3 can support barriers in 2.6.28. Someone else needs to write ext4 docs :-).

Quoted 4 lines

Also I remember a historical problem that not all disks honor write barriers, because actual data integrity makes for horrible benchmark numbers. Dunno how current that is with SATA, Alan Cox would probably know.

Sounds like broken disk, then. We should blacklist those.

Quoted 8 lines · On Monday 16 March 2009 07:28:47 Pavel Machek wrote:

Hi!

Quoted 6 lines

Documentation/filesystems/expectations.txt line 15

Quoted 3 lines

Documentation/filesystems/expectations.txt new file +3 -0

+	Fortunately writes failing are very uncommon on traditional
+	spinning disks, as they have spare sectors they use when write
+	fails.

I vaguely recall that the behavior of when a write error _does_ occur is to remount the filesystem read only? (Is this VFS or per-fs?)

Quoted 1 line

Per-fs.

Might be nice to note that in the doc.

Quoted 5 lines

Quoted 3 lines

I'm aware write errors shouldn't happen, and by the time they do it's too late to gracefully handle them, and all we can do is fail. So how do we fail?

Well, even remount-ro may be too late, IIRC.

Care to elaborate? (When a filesystem is mounted RO, I'm not sure what happens to the pages that have already been dirtied...)

Quoted 4 lines

Quoted 2 lines

(Writes aren't always cleanly at the start of an erase block, so critical data _before_ what you touch is endangered too.)

Well, flashes do remap, so it is actually "random blocks".

Fun.

Quoted 10 lines

Quoted 9 lines

Documentation/filesystems/expectations.txt line 44

Quoted 6 lines

Documentation/filesystems/expectations.txt new file +6 -0

+	otherwise, disks may write garbage during powerfail.
+	Not sure how common that problem is on generic PC machines.
+
+	Note that atomic write is very hard to guarantee for RAID-4/5/6,
+	because it needs to write both changed data, and parity, to
+	different disks.

These days instead of "atomic" it's better to think in terms of "barriers".

Quoted 4 lines

This is not about barriers (that should be different topic). Atomic write means that either whole sector is written, or nothing at all is written. Because raid5 needs to update both master data and parity at the same time, I don't think it can guarantee this during powerfail.

Good point, but I thought that's what journaling was for?

I'm aware that any flash filesystem _must_ be journaled in order to work sanely, and must be able to view the underlying erase granularity down to the bare metal, through any remapping the hardware's doing. Possibly what's really needed is a "flash is weird" section, since flash filesystems can't be mounted on arbitrary block devices.

Although an "-O erase_size=128" option so they _could_ would be nice. There's "mtdram" which seems to be the only remaining use for ram disks, but why there isn't an "mtdwrap" that works with arbitrary underlying block devices, I have no idea. (Layering it on top of a loopback device would be most useful.)

Quoted 10 lines · Rob Landley

Quoted 9 lines · On Monday 16 March 2009 07:28:47 Pavel Machek wrote:

Quoted 9 lines

Documentation/filesystems/expectations.txt line 44

Quoted 6 lines

Documentation/filesystems/expectations.txt new file +6 -0

+	otherwise, disks may write garbage during powerfail.
+	Not sure how common that problem is on generic PC machines.
+
+	Note that atomic write is very hard to guarantee for RAID-4/5/6,
+	because it needs to write both changed data, and parity, to
+	different disks.

These days instead of "atomic" it's better to think in terms of "barriers".

Quoted 4 lines

Quoted 4 lines

This is not about barriers (that should be different topic). Atomic write means that either whole sector is written, or nothing at all is written. Because raid5 needs to update both master data and parity at the same time, I don't think it can guarantee this during powerfail.

Quoted 1 line

Good point, but I thought that's what journaling was for?

I believe journaling operates on assumption that "either whole sector is written, or nothing at all is written".

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

This is not about barriers (that should be different topic). Atomic write means that either whole sector is written, or nothing at all is written. Because raid5 needs to update both master data and parity at the same time, I don't think it can guarantee this during powerfail.

Actualy raid5 should have no problem with a power failure during normal operations of the raid. The parity block should get marked out of sync, then the new data block should be written, then the new parity block and then the parity block should be flaged in sync.

Quoted 4 lines

Quoted 1 line

Good point, but I thought that's what journaling was for?

I believe journaling operates on assumption that "either whole sector is written, or nothing at all is written".

The real problem comes in degraded mode. In that case the data block (if present) and parity block must be written at the same time atomically. If the system crashes after writing one but before writing the other then the data block on the missng drive changes its contents. And for example with a chunk size of 1MB and 16 disks that could be 15MB away from the block you actualy do change. And you can not recover that after a crash as you need both the original and changed contents of the block.

So writing one sector has the risk of corrupting another (for the FS) totally unconnected sector. No amount of journaling will help there. The raid5 would need to do journaling or use battery backed cache.

* Pavel Machek:

Documentation/filesystems/expectations.txt line 4

Quoted 4 lines

Documentation/filesystems/expectations.txt new file +4 -0

+Linux block-backed filesystems can only work correctly when several
+conditions are met in the block layer and below (disks, flash
+cards). Some of them are obvious ("data on media should not change
+randomly"), some are less so.

You should make clear that the file lists per-file-system rules and that some file sytems can recover from some of the error conditions.

Documentation/filesystems/ext3.txt debugfs: ext2 and ext3 file system debugger. · line 220

Quoted 7 lines

Documentation/filesystems/ext3.txt +7 -0

+* don't damage the old data on a failed write (ATOMIC-WRITES)
+
+	(Thrash may get written into sectors during powerfail.  And
+	ext3 handles this surprisingly well at least in the
+	catastrophic case of garbage getting written into the inode
+	table, since the journal replay often will "repair" the
+	garbage that was written into the filesystem metadata blocks.

Isn't this by design? In other words, if the metadata doesn't survive non-atomic writes, wouldn't it be an ext3 bug?

It's for this reason that I've never been completely sure how useful Pavel's proposed treatise about file systems expectations really are --- because all storage subsystems *usually* provide these guarantees, but it is the very rare storage system that *always* provides these guarantees.

Probably, Pavel did too good job in generalizing things, and it could be better to make a doc about HDD vs SSD or HDD vs Flash-based-storage. Not sure. But the idea to document subtle FS assumption is good, IMO.

Quoted 2 lines

Isn't this by design? In other words, if the metadata doesn't survive non-atomic writes, wouldn't it be an ext3 bug?

Part of the problem here is that "atomic-writes" is confusing; it doesn't mean what many people think it means. The assumption which many naive filesystem designers make is that writes succeed or they don't. If they don't succeed, they don't change the previously existing data in any way.

So in the case of journalling, the assumption which gets made is that when the power fails, the disk either writes a particular disk block, or it doesn't. The problem here is as with humans and animals, death is not an event, it is a process. When the power fails, the system just doesn't stop functioning; the power on the +5 and +12 volt rails start dropping to zero, and different components fail at different times. Specifically, DRAM, being the most voltage sensitve, tends to fail before the DMA subsystem, the PCI bus, and the hard drive fails. So as a result, garbage can get written out to disk as part of the failure. That's just the way hardware works.

Is that a file system "bug"? Well, it's better to call that a mismatch between the assumptions made of physical devices, and of the file system code. On Irix, SGI hardware had a powerfail interrupt,

There is another kind of non-atomic write that nearly all file systems are subject to, however, and to give an example of this, consider what happens if you a laptop is subjected to a sudden shock while it is writing a sector, and the hard drive doesn't an accelerometer which

Depending on how severe the shock happens to be, the head could end up impacting the platter, destroying the medium (which used to be iron-oxide; hence the term "spinning rust platters") at that spot. This will obviously cause a write failure, and the previous contents of the sector will be lost. This is also considered a failure of the ATOMIC-WRITE property, and no, ext3 doesn't handle this case gracefully. Very few file systems do. (It is possible for an OS that

It's for this reason that I've never been completely sure how useful Pavel's proposed treatise about file systems expectations really are --- because all storage subsystems *usually* provide these guarantees, but it is the very rare storage system that *always* provides these guarantees.

We could just as easily have several kilobytes of explanation in Documentation/* explaining how we assume that DRAM always returns the same value that was stored in it previously --- and yet most PC class hardware still does not use ECC memory, and cosmic rays are a reality. That means that most Linux systems run on systems that are vulnerable to this kind of failure --- and the world hasn't ended.

As I recall, the main problem which Pavel had was when he was using ext3 on a *really* trashy flash drive, on a *really* trashing laptop where the flash card stuck out slightly, and any jostling of the netbook would cause the flash card to become disconnected from the laptop, and cause write errors, very easily and very frequently. In those circumstnaces, it's highly unlikely that ***any*** file system would have been able to survive such an unreliable storage system.

One of the problems I have with the break down which Pavel has used is that it doesn't break things down according to probability; the chance of a storage subsystem scribbling garbage on its last write during a

But these things are never absolute, mainly because people aren't willing to pay for either the cost of superior hardware (consider the cost of ECC memory, which isn't *that* much more expensive; and yet most PC class systems don't use it) or in terms of software overhead (historically many file system designers have eschewed the use of physical block journalling because it really hurts on meta-data intensive benchmarks), talking about absolute requirements for ATOMIC-WRITE isn't all that useful --- because nearly all hardware doesn't provide these guarantees, and nearly all filesystems require them. So to call out ext2 and ext3 for requiring them, without making

clear that pretty much *all* file systems require them, ends up causing people to switch over to some other file system that ironically enough, might end up being *more* vulernable, but which didn't earn Pavel's displeasure because he didn't try using, say, XFS on his flashcard on his trashy laptop.

Hi!

Quoted 9 lines

Quoted 3 lines

Isn't this by design? In other words, if the metadata doesn't survive non-atomic writes, wouldn't it be an ext3 bug?

Part of the problem here is that "atomic-writes" is confusing; it doesn't mean what many people think it means. The assumption which many naive filesystem designers make is that writes succeed or they don't. If they don't succeed, they don't change the previously existing data in any way.

Quoted 11 lines

So in the case of journalling, the assumption which gets made is that when the power fails, the disk either writes a particular disk block, or it doesn't. The problem here is as with humans and animals, death is not an event, it is a process. When the power fails, the system just doesn't stop functioning; the power on the +5 and +12 volt rails start dropping to zero, and different components fail at different times. Specifically, DRAM, being the most voltage sensitve, tends to fail before the DMA subsystem, the PCI bus, and the hard drive fails. So as a result, garbage can get written out to disk as part of the failure. That's just the way hardware works.

Yep, and at that point you lost data. You had "silent data corruption" from fs point of view, and that's bad.

It will be probably very bad on XFS, probably okay on Ext3, and certainly okay on Ext2: you do filesystem check, and you should be able to repair any damage. So yes, physical journaling is good, but fsck is better.

Quoted 5 lines

Is that a file system "bug"? Well, it's better to call that a mismatch between the assumptions made of physical devices, and of the file system code. On Irix, SGI hardware had a powerfail interrupt,

If those filesystem assumptions were not documented, I'd call it filesystem bug. So better document them ;-).

Quoted 5 lines

There is another kind of non-atomic write that nearly all file systems are subject to, however, and to give an example of this, consider what happens if you a laptop is subjected to a sudden shock while it is writing a sector, and the hard drive doesn't an accelerometer which

...

Quoted 9 lines

Depending on how severe the shock happens to be, the head could end up impacting the platter, destroying the medium (which used to be iron-oxide; hence the term "spinning rust platters") at that spot. This will obviously cause a write failure, and the previous contents of the sector will be lost. This is also considered a failure of the ATOMIC-WRITE property, and no, ext3 doesn't handle this case gracefully. Very few file systems do. (It is possible for an OS that

Actually, ext2 should be able to survive that, no? Error writing -> remount ro -> fsck on next boot -> drive relocates the sectors.

Quoted 7 lines

It's for this reason that I've never been completely sure how useful Pavel's proposed treatise about file systems expectations really are --- because all storage subsystems *usually* provide these guarantees, but it is the very rare storage system that *always* provides these guarantees.

Well... there's very big difference between harddrives and flash memory. Harddrives usually work, and flash memory never does.

Quoted 8 lines

We could just as easily have several kilobytes of explanation in Documentation/* explaining how we assume that DRAM always returns the same value that was stored in it previously --- and yet most PC class hardware still does not use ECC memory, and cosmic rays are a reality. That means that most Linux systems run on systems that are vulnerable to this kind of failure --- and the world hasn't ended.

There's a difference. In case of cosmic rays, hardware is clearly buggy. I have one machine with bad DRAM (about 1 errors in 2 days), and I still use it. I will not complain if ext3 trashes that.

In case of degraded raid-5, even with perfect hardware, and with ext3 on top of that, you'll get silent data corruption. Nice, eh?

Clearly, Linux is buggy there. It could be argued it is raid-5's fault, or maybe it is ext3's fault, but... linux is still buggy.

Quoted 9 lines

As I recall, the main problem which Pavel had was when he was using ext3 on a *really* trashy flash drive, on a *really* trashing laptop where the flash card stuck out slightly, and any jostling of the netbook would cause the flash card to become disconnected from the laptop, and cause write errors, very easily and very frequently. In those circumstnaces, it's highly unlikely that ***any*** file system would have been able to survive such an unreliable storage system.

Well well well. Before I pulled that flash card, I assumed that doing so is safe, because flashcard is presented as block device and ext3 should cope with sudden disk disconnects.

And I was wrong wrong wrong. (Noone told me at the university. I guess I should want my money back).

Plus note that it is not only my trashy laptop and one trashy MMC card; every USB thumb drive I seen is affected. (OTOH USB disks should be safe AFAICT).

Ext3 is unsuitable for flash cards and RAID arrays, plain and simple. It is not documented anywhere :-(. [ext2 should work better -- at least you'll not get silent data corruption.]

Quoted 5 lines

One of the problems I have with the break down which Pavel has used is that it doesn't break things down according to probability; the chance of a storage subsystem scribbling garbage on its last write during a

Can you suggest better patch? I'm not saying we should redesign ext3, but... someone should have told me that ext3+USB thumb drive=problems.

Quoted 12 lines

But these things are never absolute, mainly because people aren't willing to pay for either the cost of superior hardware (consider the cost of ECC memory, which isn't *that* much more expensive; and yet most PC class systems don't use it) or in terms of software overhead (historically many file system designers have eschewed the use of physical block journalling because it really hurts on meta-data intensive benchmarks), talking about absolute requirements for ATOMIC-WRITE isn't all that useful --- because nearly all hardware doesn't provide these guarantees, and nearly all filesystems require them. So to call out ext2 and ext3 for requiring them, without making

ext3+raid5 will fail even if you have perfect hardware.

Quoted 6 lines

clear that pretty much *all* file systems require them, ends up causing people to switch over to some other file system that ironically enough, might end up being *more* vulernable, but which didn't earn Pavel's displeasure because he didn't try using, say, XFS on his flashcard on his trashy laptop.

I hold ext2/ext3 to higher standards than other filesystem in tree. I'd not use XFS/VFAT etc.

I would not want people to migrate towards XFS/VFAT, and yes I believe XFSs/VFATs/... requirements should be documented, too. (But I know too little about those filesystems).

If you can suggest better wording, please help me. But... those requirements are non-trivial, commonly not met and the result is data loss. It has to be documented somehow. Make it as innocent-looking as you can...

Pavel

Quoted 7 lines

Yep, and at that point you lost data. You had "silent data corruption" from fs point of view, and that's bad.

It will be probably very bad on XFS, probably okay on Ext3, and certainly okay on Ext2: you do filesystem check, and you should be able to repair any damage. So yes, physical journaling is good, but fsck is better.

I don't see why you think that. In general, fsck (for any fs) only checks metadata. If you have silent data corruption that corrupts things that are fixable by fsck, you most likely have silent corruption hitting things users care about like their data blocks inside of files. Fsck will not fix (or notice) any of that, that is where things like full data checksums can help.

Quoted 3 lines

If those filesystem assumptions were not documented, I'd call it filesystem bug. So better document them ;-).

I think that we need to help people understand the full spectrum of data concerns, starting with reasonable best practices that will help most people suffer *less* (not no) data loss. And make very sure that they are not falsely assured that by following any specific script that they can skip backups, remote backups, etc :-)

Nothing in our code in any part of the kernel deals well with every disaster or odd event.

Quoted 3 lines

Actually, ext2 should be able to survive that, no? Error writing -> remount ro -> fsck on next boot -> drive relocates the sectors.

I think that the example and the response are both off base. If your head ever touches the platter, you won't be reading from a huge part of your drive ever again (usually, you have 2 heads per platter, 3-4 platters, impact would kill one head and a corresponding percentage of your data).

Quoted 6 lines

Quoted 5 lines

It's for this reason that I've never been completely sure how useful Pavel's proposed treatise about file systems expectations really are --- because all storage subsystems *usually* provide these guarantees, but it is the very rare storage system that *always* provides these guarantees.

Quoted 2 lines

Well... there's very big difference between harddrives and flash memory. Harddrives usually work, and flash memory never does.

It is hard for anyone to see the real data without looking in detail at large numbers of parts. Back at EMC, we looked at failures for lots of parts so we got a clear grasp on trends. I do agree that flash/SSD parts are still very young so we will have interesting and unexpected failure modes to learn to deal with....

Quoted 14 lines

Quoted 6 lines

We could just as easily have several kilobytes of explanation in Documentation/* explaining how we assume that DRAM always returns the same value that was stored in it previously --- and yet most PC class hardware still does not use ECC memory, and cosmic rays are a reality. That means that most Linux systems run on systems that are vulnerable to this kind of failure --- and the world hasn't ended.

There's a difference. In case of cosmic rays, hardware is clearly buggy. I have one machine with bad DRAM (about 1 errors in 2 days), and I still use it. I will not complain if ext3 trashes that.

In case of degraded raid-5, even with perfect hardware, and with ext3 on top of that, you'll get silent data corruption. Nice, eh?

Quoted 2 lines

Clearly, Linux is buggy there. It could be argued it is raid-5's fault, or maybe it is ext3's fault, but... linux is still buggy.

Nothing is perfect. It is still a trade off between storage utilization (how much storage we give users for say 5 2TB drives), performance and costs (throw away any disks over 2 years old?).

Quoted 3 lines

Ext3 is unsuitable for flash cards and RAID arrays, plain and simple. It is not documented anywhere :-(. [ext2 should work better -- at least you'll not get silent data corruption.]

ext3 is used on lots of raid arrays without any issue.

Quoted 11 lines

I hold ext2/ext3 to higher standards than other filesystem in tree. I'd not use XFS/VFAT etc.

I would not want people to migrate towards XFS/VFAT, and yes I believe XFSs/VFATs/... requirements should be documented, too. (But I know too little about those filesystems).

If you can suggest better wording, please help me. But... those requirements are non-trivial, commonly not met and the result is data loss. It has to be documented somehow. Make it as innocent-looking as you can...

I think that you really need to step back and look harder at real failures - not just your personal experience - but a larger set of real world failures. Many papers have been published recently about that (the google paper, the Bianca paper from FAST, Netapp, etc).

Hi!

Quoted 8 lines

Quoted 8 lines

Yep, and at that point you lost data. You had "silent data corruption" from fs point of view, and that's bad.

It will be probably very bad on XFS, probably okay on Ext3, and certainly okay on Ext2: you do filesystem check, and you should be able to repair any damage. So yes, physical journaling is good, but fsck is better.

Quoted 2 lines

I don't see why you think that. In general, fsck (for any fs) only checks metadata. If you have silent data corruption that corrupts things that are fixable by fsck, you most likely have silent corruption hitting things users care about like their data blocks inside of files. Fsck will not fix (or notice) any of that, that is where things like full data checksums can help.

Ok, but in case of data corruption, at least your filesystem does not degrade further.

Quoted 6 lines

Quoted 4 lines

If those filesystem assumptions were not documented, I'd call it filesystem bug. So better document them ;-).

I think that we need to help people understand the full spectrum of data concerns, starting with reasonable best practices that will help most people suffer *less* (not no) data loss. And make very sure that they are not falsely assured that by following any specific script that they can skip backups, remote backups, etc :-)

Quoted 2 lines

Nothing in our code in any part of the kernel deals well with every disaster or odd event.

I can reproduce data loss with ext3 on flashcard in about 40 seconds. I'd not call that "odd event". It would be nice to handle that, but that is hard. So ... can we at least get that documented please?

Quoted 8 lines

Quoted 6 lines

Actually, ext2 should be able to survive that, no? Error writing -> remount ro -> fsck on next boot -> drive relocates the sectors.

I think that the example and the response are both off base. If your head ever touches the platter, you won't be reading from a huge part of your drive ever again (usually, you have 2 heads per platter, 3-4 platters, impact would kill one head and a corresponding percentage of your data).

Ok, that's obviously game over.

Quoted 10 lines

Quoted 10 lines

Quoted 7 lines

It's for this reason that I've never been completely sure how useful Pavel's proposed treatise about file systems expectations really are --- because all storage subsystems *usually* provide these guarantees, but it is the very rare storage system that *always* provides these guarantees.

Well... there's very big difference between harddrives and flash memory. Harddrives usually work, and flash memory never does.

Quoted 2 lines

It is hard for anyone to see the real data without looking in detail at large numbers of parts. Back at EMC, we looked at failures for lots of parts so we got a clear grasp on trends. I do agree that flash/SSD parts are still very young so we will have interesting and unexpected failure modes to learn to deal with....

_Maybe_ SSDs, being HDD replacements are better. I don't know.

_All_ flash cards (MMC, USB, SD) had the problems. You don't need to get clear grasp on trends. Those cards just don't meet ext3 expectations, and if you pull them, you get data loss.

Quoted 19 lines

Quoted 19 lines

Quoted 7 lines

We could just as easily have several kilobytes of explanation in Documentation/* explaining how we assume that DRAM always returns the same value that was stored in it previously --- and yet most PC class hardware still does not use ECC memory, and cosmic rays are a reality. That means that most Linux systems run on systems that are vulnerable to this kind of failure --- and the world hasn't ended.

There's a difference. In case of cosmic rays, hardware is clearly buggy. I have one machine with bad DRAM (about 1 errors in 2 days), and I still use it. I will not complain if ext3 trashes that.

In case of degraded raid-5, even with perfect hardware, and with ext3 on top of that, you'll get silent data corruption. Nice, eh?

Clearly, Linux is buggy there. It could be argued it is raid-5's fault, or maybe it is ext3's fault, but... linux is still buggy.

Quoted 2 lines

Nothing is perfect. It is still a trade off between storage utilization (how much storage we give users for say 5 2TB drives), performance and costs (throw away any disks over 2 years old?).

"Nothing is perfect"?! That's design decision/problem in raid5/ext3. I believe that should be at least documented. (And understand why ZFS is interesting thing).

Quoted 4 lines

Quoted 4 lines

Ext3 is unsuitable for flash cards and RAID arrays, plain and simple. It is not documented anywhere :-(. [ext2 should work better -- at least you'll not get silent data corruption.]

Quoted 2 lines

ext3 is used on lots of raid arrays without any issue.

And I still use my zaurus with crappy DRAM.

I would not trust raid5 array with my data, for multiple reasons. The fact that degraded raid5 breaks ext3 assumptions should really be documented.

Quoted 8 lines

Quoted 8 lines

I hold ext2/ext3 to higher standards than other filesystem in tree. I'd not use XFS/VFAT etc.

I would not want people to migrate towards XFS/VFAT, and yes I believe XFSs/VFATs/... requirements should be documented, too. (But I know too little about those filesystems).

Quoted 9 lines

Quoted 5 lines

If you can suggest better wording, please help me. But... those requirements are non-trivial, commonly not met and the result is data loss. It has to be documented somehow. Make it as innocent-looking as you can...

I think that you really need to step back and look harder at real failures - not just your personal experience - but a larger set of real world failures. Many papers have been published recently about that (the google paper, the Bianca paper from FAST, Netapp, etc).

The papers show failures in "once a year" range. I have "twice a minute" failure scenario with flashdisks.

Not sure how often "degraded raid5 breaks ext3 atomicity" would bite, but I bet it would be on "once a day" scale.

We should document those.
Pavel

Quoted 5 lines

I can reproduce data loss with ext3 on flashcard in about 40 seconds. I'd not call that "odd event". It would be nice to handle that, but that is hard. So ... can we at least get that documented please?

Part of documenting best practices is to put down very specific things that do/don't work. What I worry about is producing too much detail to be of use for real end users.

I have to admit that I have not paid enough attention to this specifics of your ext3 + flash card issue - is it the ftl stuff doing out of order IO's?

Quoted 4 lines

_All_ flash cards (MMC, USB, SD) had the problems. You don't need to get clear grasp on trends. Those cards just don't meet ext3 expectations, and if you pull them, you get data loss.

Pull them even after an unmount, or pull them hot?

Quoted 9 lines

Quoted 4 lines

Nothing is perfect. It is still a trade off between storage utilization (how much storage we give users for say 5 2TB drives), performance and costs (throw away any disks over 2 years old?).

"Nothing is perfect"?! That's design decision/problem in raid5/ext3. I believe that should be at least documented. (And understand why ZFS is interesting thing).

Your statement is overly broad - ext3 on a commercial RAID array that does RAID5 or RAID6, etc has no issues that I know of.

Quoted 5 lines

And I still use my zaurus with crappy DRAM.

I would not trust raid5 array with my data, for multiple reasons. The fact that degraded raid5 breaks ext3 assumptions should really be documented.

Again, you say RAID5 without enough specifics. Are you pointing just at MD RAID5 on S-ATA? Hardware RAID cards? A specific commercial RAID5 vendor?

Quoted 7 lines

The papers show failures in "once a year" range. I have "twice a minute" failure scenario with flashdisks.

Not sure how often "degraded raid5 breaks ext3 atomicity" would bite, but I bet it would be on "once a day" scale.

We should document those.

Documentation is fine with sufficient, hard data....

Hi!

Quoted 6 lines

Quoted 6 lines

I can reproduce data loss with ext3 on flashcard in about 40 seconds. I'd not call that "odd event". It would be nice to handle that, but that is hard. So ... can we at least get that documented please?

Quoted 2 lines

Part of documenting best practices is to put down very specific things that do/don't work. What I worry about is producing too much detail to be of use for real end users.

Well, I was trying to write for kernel audience. Someone can turn that into nice end-user manual.

Quoted 3 lines

I have to admit that I have not paid enough attention to this specifics of your ext3 + flash card issue - is it the ftl stuff doing out of order IO's?

The problem is that flash cards destroy whole erase block on unplug, and ext3 can't cope with that.

Quoted 5 lines

Quoted 5 lines

_All_ flash cards (MMC, USB, SD) had the problems. You don't need to get clear grasp on trends. Those cards just don't meet ext3 expectations, and if you pull them, you get data loss.

Quoted 2 lines

Pull them even after an unmount, or pull them hot?

Pull them hot.

[Some people try -osync to avoid data loss on flash cards... that will not do the trick. Flashcard will still kill the eraseblock.]

Quoted 9 lines

Quoted 9 lines

Quoted 4 lines

Nothing is perfect. It is still a trade off between storage utilization (how much storage we give users for say 5 2TB drives), performance and costs (throw away any disks over 2 years old?).

"Nothing is perfect"?! That's design decision/problem in raid5/ext3. I believe that should be at least documented. (And understand why ZFS is interesting thing).

Quoted 2 lines

Your statement is overly broad - ext3 on a commercial RAID array that does RAID5 or RAID6, etc has no issues that I know of.

If your commercial RAID array is battery backed, maybe. But I was talking Linux MD here.

Quoted 7 lines

Quoted 7 lines

And I still use my zaurus with crappy DRAM.

I would not trust raid5 array with my data, for multiple reasons. The fact that degraded raid5 breaks ext3 assumptions should really be documented.

Quoted 2 lines

Again, you say RAID5 without enough specifics. Are you pointing just at MD RAID5 on S-ATA? Hardware RAID cards? A specific commercial RAID5 vendor?

Degraded MD RAID5 on anything, including SATA, and including hypothetical "perfect disk".

Quoted 11 lines

Quoted 9 lines

The papers show failures in "once a year" range. I have "twice a minute" failure scenario with flashdisks.

Not sure how often "degraded raid5 breaks ext3 atomicity" would bite, but I bet it would be on "once a day" scale.

We should document those.

Documentation is fine with sufficient, hard data....

Degraded MD RAID5 does not work by design; whole stripe will be damaged on powerfail or reset or kernel bug, and ext3 can not cope with that kind of damage. [I don't see why statistics should be neccessary for that; the same way we don't need statistics to see that ext2 needs fsck after powerfail.]
Pavel

Quoted 7 lines · Pavel Machek

Degraded MD RAID5 does not work by design; whole stripe will be damaged on powerfail or reset or kernel bug, and ext3 can not cope with that kind of damage. [I don't see why statistics should be neccessary for that; the same way we don't need statistics to see that ext2 needs fsck after powerfail.]
Pavel

What you are describing is a double failure and RAID5 is not double failure tolerant regardless of the file system type....

Quoted 10 lines · Ric Wheeler

Quoted 7 lines · Pavel Machek

Degraded MD RAID5 does not work by design; whole stripe will be damaged on powerfail or reset or kernel bug, and ext3 can not cope with that kind of damage. [I don't see why statistics should be neccessary for that; the same way we don't need statistics to see that ext2 needs fsck after powerfail.]
Pavel

What you are describing is a double failure and RAID5 is not double failure tolerant regardless of the file system type....

Are you sure he isn't talking about how RAID must write all the data chunks to make a complete stripe and if there is a power-loss, some of the chunks may be written and some may not?

As I read Pavel's point he is saying that the incomplete write can be detected by the incorrect parity chunk, but degraded RAID-5 has no working parity chunk so the incomplete write would go undetected.

I know this is a RAID failure mode. However, I actually thought this was a problem even for a intact RAID-5. AFAIK, RAID-5 does not generally read the complete stripe and perform verification unless that is requested, because doing so would hurt performance and lose the entire point of the RAID-5 rotating parity blocks.

Quoted 12 lines · Zan Lynx

Quoted 10 lines · Ric Wheeler

Quoted 8 lines · Pavel Machek

Degraded MD RAID5 does not work by design; whole stripe will be damaged on powerfail or reset or kernel bug, and ext3 can not cope with that kind of damage. [I don't see why statistics should be neccessary for that; the same way we don't need statistics to see that ext2 needs fsck after powerfail.]
Pavel

Quoted 6 lines

Quoted 2 lines

What you are describing is a double failure and RAID5 is not double failure tolerant regardless of the file system type....

Are you sure he isn't talking about how RAID must write all the data chunks to make a complete stripe and if there is a power-loss, some of the chunks may be written and some may not?

As I read Pavel's point he is saying that the incomplete write can be detected by the incorrect parity chunk, but degraded RAID-5 has no working parity chunk so the incomplete write would go undetected.

Yep.

Quoted 2 lines

I know this is a RAID failure mode. However, I actually thought this was a problem even for a intact RAID-5. AFAIK, RAID-5 does not generally read the complete stripe and perform verification unless that is requested, because doing so would hurt performance and lose the entire point of the RAID-5 rotating parity blocks.

Not sure; is not RAID expected to verify the array after unclean shutdown?

Pavel

Quoted 8 lines · Ric Wheeler

Quoted 7 lines · Pavel Machek

Degraded MD RAID5 does not work by design; whole stripe will be damaged on powerfail or reset or kernel bug, and ext3 can not cope with that kind of damage. [I don't see why statistics should be neccessary for that; the same way we don't need statistics to see that ext2 needs fsck after powerfail.]
Pavel

Quoted 1 line

What you are describing is a double failure and RAID5 is not double failure tolerant regardless of the file system type....

Are you sure he isn't talking about how RAID must write all the data chunks to make a complete stripe and if there is a power-loss, some of the chunks may be written and some may not?

As I read Pavel's point he is saying that the incomplete write can be detected by the incorrect parity chunk, but degraded RAID-5 has no working parity chunk so the incomplete write would go undetected.

I know this is a RAID failure mode. However, I actually thought this was a problem even for a intact RAID-5. AFAIK, RAID-5 does not generally read the complete stripe and perform verification unless that is requested, because doing so would hurt performance and lose the entire point of the RAID-5 rotating parity blocks.

Quoted 6 lines

Quoted 3 lines

I have to admit that I have not paid enough attention to this specifics of your ext3 + flash card issue - is it the ftl stuff doing out of order IO's?

The problem is that flash cards destroy whole erase block on unplug, and ext3 can't cope with that.

Even if you unmount the file system? Why isn't this an issue with ext2?

Quoted 7 lines

Quoted 2 lines

Pull them even after an unmount, or pull them hot?

Pull them hot.

[Some people try -osync to avoid data loss on flash cards... that will not do the trick. Flashcard will still kill the eraseblock.]

Pulling hot any device will cause data loss for recent data loss, even with ext2 you will have data in the page cache, right?

Quoted 7 lines

Quoted 4 lines

Again, you say RAID5 without enough specifics. Are you pointing just at MD RAID5 on S-ATA? Hardware RAID cards? A specific commercial RAID5 vendor?

Degraded MD RAID5 on anything, including SATA, and including hypothetical "perfect disk".

Degraded is one faulted drive while MD is doing a rebuild? And then you hot unplug it or power cycle? I think that would certainly cause failure for ext2 as well (again, you would lose any data in the page cache).

Quoted 5 lines

Degraded MD RAID5 does not work by design; whole stripe will be damaged on powerfail or reset or kernel bug, and ext3 can not cope with that kind of damage. [I don't see why statistics should be neccessary for that; the same way we don't need statistics to see that ext2 needs fsck after powerfail.]

What you are describing is a double failure and RAID5 is not double failure tolerant regardless of the file system type....

Quoted 3 lines

I have to admit that I have not paid enough attention to this specifics of your ext3 + flash card issue - is it the ftl stuff doing out of order IO's?

The problem is that flash cards destroy whole erase block on unplug, and ext3 can't cope with that.

Quoted 2 lines

Your statement is overly broad - ext3 on a commercial RAID array that does RAID5 or RAID6, etc has no issues that I know of.

If your commercial RAID array is battery backed, maybe. But I was talking Linux MD here.

Quoted 6 lines · Pavel Machek

Quoted 3 lines

I have to admit that I have not paid enough attention to this specifics of your ext3 + flash card issue - is it the ftl stuff doing out of order IO's?

The problem is that flash cards destroy whole erase block on unplug, and ext3 can't cope with that.

Sure --- but name **any** filesystem that can deal with the fact that 128k or 256k worth of data might disappear when you pull out the flash card while it is writing a single sector?

Quoted 5 lines

Quoted 2 lines

Your statement is overly broad - ext3 on a commercial RAID array that does RAID5 or RAID6, etc has no issues that I know of.

If your commercial RAID array is battery backed, maybe. But I was talking Linux MD here.

If your concern is that with Linux MD, you could potentially lose an entire stripe in RAID 5 mode, then you should say that explicitly; but again, this isn't a filesystem specific cliam; it's true for all filesystems. I don't know of any file system that can survive having a RAID stripe-shaped-hole blown into the middle of it due to a power failure.

I'll note, BTW, that AIX uses a journal to protect against these sorts of problems with software raid; this also means that with AIX, you also don't have to rebuild a RAID 1 device after an unclean shutdown, like you have do with Linux MD. This was on the EVMS's team development list to implement for Linux, but it got canned after LVM won out, lo those many years ago. Ce la vie; but it's a problem which is solvable at the RAID layer, and which is traditionally and historically solved in competent RAID implementations.

Quoted 6 lines · Pavel Machek

Quoted 3 lines

I have to admit that I have not paid enough attention to this specifics of your ext3 + flash card issue - is it the ftl stuff doing out of order IO's?

The problem is that flash cards destroy whole erase block on unplug, and ext3 can't cope with that.

Sure --- but name **any** filesystem that can deal with the fact that 128k or 256k worth of data might disappear when you pull out the flash card while it is writing a single sector?

Quoted 5 lines

Quoted 2 lines

Your statement is overly broad - ext3 on a commercial RAID array that does RAID5 or RAID6, etc has no issues that I know of.

If your commercial RAID array is battery backed, maybe. But I was talking Linux MD here.

If your concern is that with Linux MD, you could potentially lose an entire stripe in RAID 5 mode, then you should say that explicitly; but again, this isn't a filesystem specific cliam; it's true for all filesystems. I don't know of any file system that can survive having a RAID stripe-shaped-hole blown into the middle of it due to a power failure.

I'll note, BTW, that AIX uses a journal to protect against these sorts of problems with software raid; this also means that with AIX, you also don't have to rebuild a RAID 1 device after an unclean shutdown, like you have do with Linux MD. This was on the EVMS's team development list to implement for Linux, but it got canned after LVM won out, lo those many years ago. Ce la vie; but it's a problem which is solvable at the RAID layer, and which is traditionally and historically solved in competent RAID implementations.

Quoted 11 lines · Theodore Tso

Quoted 6 lines · Pavel Machek

Quoted 3 lines

I have to admit that I have not paid enough attention to this specifics of your ext3 + flash card issue - is it the ftl stuff doing out of order IO's?

The problem is that flash cards destroy whole erase block on unplug, and ext3 can't cope with that.

Sure --- but name **any** filesystem that can deal with the fact that 128k or 256k worth of data might disappear when you pull out the flash card while it is writing a single sector?

First... I consider myself quite competent in the os level, yet I did not realize what flash does and what that means for data integrity. That means we need some documentation, or maybe we should refuse to mount those devices r/w or something.

Then to answer your question... ext2. You expect to run fsck after unclean shutdown, and you expect to have to solve some problems with it. So the way ext2 deals with the flash media actually matches what the user expects. (*)

OTOH in ext3 case you expect consistent filesystem after unplug; and you don't get that.

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

Your statement is overly broad - ext3 on a commercial RAID array that does RAID5 or RAID6, etc has no issues that I know of.

If your commercial RAID array is battery backed, maybe. But I was talking Linux MD here.

...

Quoted 6 lines

If your concern is that with Linux MD, you could potentially lose an entire stripe in RAID 5 mode, then you should say that explicitly; but again, this isn't a filesystem specific cliam; it's true for all filesystems. I don't know of any file system that can survive having a RAID stripe-shaped-hole blown into the middle of it due to a power failure.

Again, ext2 handles that in a way user expects it.

At least I was teached "ext2 needs fsck after powerfail; ext3 can handle powerfails just ok".

Quoted 8 lines

I'll note, BTW, that AIX uses a journal to protect against these sorts of problems with software raid; this also means that with AIX, you also don't have to rebuild a RAID 1 device after an unclean shutdown, like you have do with Linux MD. This was on the EVMS's team development list to implement for Linux, but it got canned after LVM won out, lo those many years ago. Ce la vie; but it's a problem which is solvable at the RAID layer, and which is traditionally and historically solved in competent RAID implementations.

Yep, we should add journal to RAID; or at least write "Linux MD *needs* an UPS" in big and bold letters. I'm trying to do the second part.

(Attached is current version of the patch).

[If you'd prefer patch saying that MMC/USB flash/Linux MD arrays are generaly unsafe to use without UPS/reliable connection/no kernel bugs... then I may try to push that. I was not sure... maybe some filesystem _can_ handle this kind of issues?]

Pavel

(*) Ok, now... user expects to run fsck, but very advanced users may not expect old data to be damaged. Certainly I was not advanced enough user few months ago.

Documentation/filesystems/expectations.txt new file +57 -0

new file mode 100644
index 0000000..d1ef4d0
@@ -0,0 +1,57 @@
+Linux block-backed filesystems can only work correctly when several
+conditions are met in the block layer and below (disks, flash
+cards). Some of them are obvious ("data on media should not change
+randomly"), some are less so. Not all filesystems require all of these
+to be satisfied for safe operation.
+
+Write errors not allowed (NO-WRITE-ERRORS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes to media never fail. Even if disk returns error condition
+during write, filesystems can't handle that correctly.
+
+	Fortunately writes failing are very uncommon on traditional
+	spinning disks, as they have spare sectors they use when write
+	fails.
+
+Don't cause collateral damage on a failed write (NO-COLLATERALS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+On some storage systems, failed write (for example due to power
+failure) kills data in adjacent (or maybe unrelated) sectors.
+
+Unfortunately, cheap USB/SD flash cards I've seen do have this bug,
+and are thus unsuitable for all filesystems I know.
+
+	An inherent problem with using flash as a normal block device
+	is that the flash erase size is bigger than most filesystem
+	sector sizes.  So when you request a write, it may erase and
+	rewrite some 64k, 128k, or even a couple megabytes on the
+	really _big_ ones.
+
+	If you lose power in the middle of that, filesystem won't
+	notice that data in the "sectors" _around_ the one your were
+	trying to write to got trashed.
+
+	MD RAID-4/5/6 in degraded mode has similar problem, stripes
+	behave similary to eraseblocks.
+
+
+Don't damage the old data on a powerfail (ATOMIC-WRITES-ON-POWERFAIL)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Either whole sector is correctly written or nothing is written during
+powerfail.
+
+	Because RAM tends to fail faster than rest of system during
+	powerfail, special hw killing DMA transfers may be necessary;
+	otherwise, disks may write garbage during powerfail.
+	This may be quite common on generic PC machines.
+
+	Note that atomic write is very hard to guarantee for MD RAID-4/5/6,
+	because it needs to write both changed data, and parity, to
+	different disks. (But it will only really show up in degraded mode).
+	UPS for RAID array should help.
+
+
+

Documentation/filesystems/ext2.txt +0 -0

index 67639f9..ef9ff0f 100644
@@ -338,27 +339,30 @@ enough 4-character names to make up unique directory entries, so they

have to be 8 character filenames, even then we are fairly close to running out of unique filenames.

Documentation/filesystems/ext2.txt +22 -0

+Requirements
+============
+
+Ext2 expects disk/storage subsystem to behave sanely. On sanely
+behaving disk subsystem, data that have been successfully synced will
+stay on the disk. Sane means:
+
+* write errors not allowed (NO-WRITE-ERRORS)
+
+* don't damage the old data on a failed write (ATOMIC-WRITES-ON-POWERFAIL)
+
+and obviously:
+
+* don't cause collateral damage to adjacent sectors on a failed write
+  (NO-COLLATERALS)
+
+(see expectations.txt; note that most/all linux block-based
+filesystems have similar expectations)
+
+* write caching is disabled. ext2 does not know how to issue barriers
+  as of 2.6.28. hdparm -W0 disables it on SATA disks.
+

Journaling
-----------

Documentation/filesystems/ext2.txt +1 -19

-
-A journaling extension to the ext2 code has been developed by Stephen
-Tweedie.  It avoids the risks of metadata corruption and the need to
-wait for e2fsck to complete after a crash, without requiring a change
-to the on-disk ext2 layout.  In a nutshell, the journal is a regular
-file which stores whole metadata (and optionally data) blocks that have
-been modified, prior to writing them into the filesystem.  This means
-it is possible to add a journal to an existing ext2 filesystem without
-the need for data conversion.
-
-When changes to the filesystem (e.g. a file is renamed) they are stored in
-a transaction in the journal and can either be complete or incomplete at
-the time of a crash.  If a transaction is complete at the time of a crash
-(or in the normal case where the system does not crash), then any blocks
-in that transaction are guaranteed to represent a valid filesystem state,
-and are copied into the filesystem.  If a transaction is incomplete at
-the time of the crash, then there is no guarantee of consistency for
-the blocks in that transaction so they are discarded (which means any
-filesystem changes they represent are also lost).
+==========

Check Documentation/filesystems/ext3.txt if you want to read more about ext3 and journaling.

Documentation/filesystems/ext3.txt +0 -0

index 570f9bd..752f4b4 100644
@@ -199,6 +202,43 @@ debugfs: 	ext2 and ext3 file system debugger.

ext2online: online (mounted) ext2 and ext3 filesystem resizer

Documentation/filesystems/ext3.txt +37 -0

+Requirements
+============
+
+Ext3 expects disk/storage subsystem to behave sanely. On sanely
+behaving disk subsystem, data that have been successfully synced will
+stay on the disk. Sane means:
+
+* write errors not allowed (NO-WRITE-ERRORS)
+
+* don't damage the old data on a failed write (ATOMIC-WRITES-ON-POWERFAIL)
+
+  Ext3 handles trash getting written into sectors during powerfail
+  surprisingly well.  It's not foolproof, but it is resilient.
+  Incomplete journal entries are ignored, and journal replay of
+  complete entries will often "repair" garbage written into the inode
+  table.  The data=journal option extends this behavior to file and
+  directory data blocks as well.
+
+
+and obviously:
+
+* don't cause collateral damage to adjacent sectors on a failed write
+  (NO-COLLATERALS)
+
+
+(see expectations.txt; note that most/all linux block-based
+filesystems have similar expectations)
+
+* either write caching is disabled, or hw can do barriers and they are enabled.
+
+	   (Note that barriers are disabled by default, use "barrier=1"
+	   mount option after making sure hw can support them).
+
+	   hdparm -I reports disk features. If you have "Native
+	   Command Queueing" is the feature you are looking for.
+
+

References
==========

Quoted 9 lines

Quoted 3 lines

Sure --- but name **any** filesystem that can deal with the fact that 128k or 256k worth of data might disappear when you pull out the flash card while it is writing a single sector?

First... I consider myself quite competent in the os level, yet I did not realize what flash does and what that means for data integrity. That means we need some documentation, or maybe we should refuse to mount those devices r/w or something.

Quoted 4 lines

Then to answer your question... ext2. You expect to run fsck after unclean shutdown, and you expect to have to solve some problems with it. So the way ext2 deals with the flash media actually matches what the user expects. (*)

you loose data in ext2

Quoted 2 lines

OTOH in ext3 case you expect consistent filesystem after unplug; and you don't get that.

the problem is that people have been preaching that journaling filesystems eliminate all data loss for no cost (or at worst for minimal cost).

they don't, they never did.

they address one specific problem (metadata inconsistancy), but they do not address data loss, and never did (and for the most part the filesystem developers never claimed to)

people somehow have the expectation that ext3 does the data equivalent of solving world hunger, it doesn't, it never did, and it never claimed to.

personally I don't consider the two filesystems to be significantly different in terms of the data loss potential. I think people are more aware of the potentials with XFS than with ext3, but I believe that the risk of loss is really about the same (and pretty much for the same reasons)

Quoted 4 lines

Again, ext2 handles that in a way user expects it.

At least I was teached "ext2 needs fsck after powerfail; ext3 can handle powerfails just ok".

you were teached wrong. the people making these claims for ext3 didn't understand what ext3 does and doesn't do.

Quoted 11 lines · Theodore Tso

Quoted 5 lines · Pavel Machek

Quoted 2 lines

I have to admit that I have not paid enough attention to this specifics of your ext3 + flash card issue - is it the ftl stuff doing out of order IO's?

The problem is that flash cards destroy whole erase block on unplug, and ext3 can't cope with that.

Sure --- but name **any** filesystem that can deal with the fact that 128k or 256k worth of data might disappear when you pull out the flash card while it is writing a single sector?

First... I consider myself quite competent in the os level, yet I did not realize what flash does and what that means for data integrity. That means we need some documentation, or maybe we should refuse to mount those devices r/w or something.

Then to answer your question... ext2. You expect to run fsck after unclean shutdown, and you expect to have to solve some problems with it. So the way ext2 deals with the flash media actually matches what the user expects. (*)

OTOH in ext3 case you expect consistent filesystem after unplug; and you don't get that.

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

Your statement is overly broad - ext3 on a commercial RAID array that does RAID5 or RAID6, etc has no issues that I know of.

If your commercial RAID array is battery backed, maybe. But I was talking Linux MD here.

...

Quoted 7 lines

If your concern is that with Linux MD, you could potentially lose an entire stripe in RAID 5 mode, then you should say that explicitly; but again, this isn't a filesystem specific cliam; it's true for all filesystems. I don't know of any file system that can survive having a RAID stripe-shaped-hole blown into the middle of it due to a power failure.

Again, ext2 handles that in a way user expects it.

At least I was teached "ext2 needs fsck after powerfail; ext3 can handle powerfails just ok".

Quoted 9 lines

I'll note, BTW, that AIX uses a journal to protect against these sorts of problems with software raid; this also means that with AIX, you also don't have to rebuild a RAID 1 device after an unclean shutdown, like you have do with Linux MD. This was on the EVMS's team development list to implement for Linux, but it got canned after LVM won out, lo those many years ago. Ce la vie; but it's a problem which is solvable at the RAID layer, and which is traditionally and historically solved in competent RAID implementations.

Yep, we should add journal to RAID; or at least write "Linux MD *needs* an UPS" in big and bold letters. I'm trying to do the second part.

(Attached is current version of the patch).

[If you'd prefer patch saying that MMC/USB flash/Linux MD arrays are generaly unsafe to use without UPS/reliable connection/no kernel bugs... then I may try to push that. I was not sure... maybe some filesystem _can_ handle this kind of issues?]

Pavel

(*) Ok, now... user expects to run fsck, but very advanced users may not expect old data to be damaged. Certainly I was not advanced enough user few months ago.

Documentation/filesystems/expectations.txt new file +57 -0

new file mode 100644
index 0000000..d1ef4d0
@@ -0,0 +1,57 @@
+Linux block-backed filesystems can only work correctly when several
+conditions are met in the block layer and below (disks, flash
+cards). Some of them are obvious ("data on media should not change
+randomly"), some are less so. Not all filesystems require all of these
+to be satisfied for safe operation.
+
+Write errors not allowed (NO-WRITE-ERRORS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes to media never fail. Even if disk returns error condition
+during write, filesystems can't handle that correctly.
+
+	Fortunately writes failing are very uncommon on traditional 
+	spinning disks, as they have spare sectors they use when write
+	fails.
+
+Don't cause collateral damage on a failed write (NO-COLLATERALS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+On some storage systems, failed write (for example due to power
+failure) kills data in adjacent (or maybe unrelated) sectors.
+
+Unfortunately, cheap USB/SD flash cards I've seen do have this bug,
+and are thus unsuitable for all filesystems I know.
+
+	An inherent problem with using flash as a normal block device
+	is that the flash erase size is bigger than most filesystem
+	sector sizes.  So when you request a write, it may erase and
+	rewrite some 64k, 128k, or even a couple megabytes on the
+	really _big_ ones.
+
+	If you lose power in the middle of that, filesystem won't
+	notice that data in the "sectors" _around_ the one your were
+	trying to write to got trashed.
+
+	MD RAID-4/5/6 in degraded mode has similar problem, stripes
+	behave similary to eraseblocks.
+
+
+Don't damage the old data on a powerfail (ATOMIC-WRITES-ON-POWERFAIL)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Either whole sector is correctly written or nothing is written during
+powerfail.
+
+	Because RAM tends to fail faster than rest of system during 
+	powerfail, special hw killing DMA transfers may be necessary;
+	otherwise, disks may write garbage during powerfail.
+	This may be quite common on generic PC machines.
+
+	Note that atomic write is very hard to guarantee for MD RAID-4/5/6,
+	because it needs to write both changed data, and parity, to 
+	different disks. (But it will only really show up in degraded mode).
+	UPS for RAID array should help.
+
+
+

Documentation/filesystems/ext2.txt +23 -20

index 67639f9..ef9ff0f 100644
@@ -338,27 +339,30 @@ enough 4-character names to make up unique directory entries, so they
 have to be 8 character filenames, even then we are fairly close to
 running out of unique filenames.
 
+Requirements
+============
+
+Ext2 expects disk/storage subsystem to behave sanely. On sanely
+behaving disk subsystem, data that have been successfully synced will
+stay on the disk. Sane means:
+
+* write errors not allowed (NO-WRITE-ERRORS)
+
+* don't damage the old data on a failed write (ATOMIC-WRITES-ON-POWERFAIL)
+
+and obviously:
+
+* don't cause collateral damage to adjacent sectors on a failed write
+  (NO-COLLATERALS)
+
+(see expectations.txt; note that most/all linux block-based
+filesystems have similar expectations)
+
+* write caching is disabled. ext2 does not know how to issue barriers
+  as of 2.6.28. hdparm -W0 disables it on SATA disks.
+
 Journaling
-----------
-
-A journaling extension to the ext2 code has been developed by Stephen
-Tweedie.  It avoids the risks of metadata corruption and the need to
-wait for e2fsck to complete after a crash, without requiring a change
-to the on-disk ext2 layout.  In a nutshell, the journal is a regular
-file which stores whole metadata (and optionally data) blocks that have
-been modified, prior to writing them into the filesystem.  This means
-it is possible to add a journal to an existing ext2 filesystem without
-the need for data conversion.
-
-When changes to the filesystem (e.g. a file is renamed) they are stored in
-a transaction in the journal and can either be complete or incomplete at
-the time of a crash.  If a transaction is complete at the time of a crash
-(or in the normal case where the system does not crash), then any blocks
-in that transaction are guaranteed to represent a valid filesystem state,
-and are copied into the filesystem.  If a transaction is incomplete at
-the time of the crash, then there is no guarantee of consistency for
-the blocks in that transaction so they are discarded (which means any
-filesystem changes they represent are also lost).
+==========
 Check Documentation/filesystems/ext3.txt if you want to read more about
 ext3 and journaling.
 

Documentation/filesystems/ext3.txt +37 -0

index 570f9bd..752f4b4 100644
@@ -199,6 +202,43 @@ debugfs: 	ext2 and ext3 file system debugger.
 ext2online:	online (mounted) ext2 and ext3 filesystem resizer
 
 
+Requirements
+============
+
+Ext3 expects disk/storage subsystem to behave sanely. On sanely
+behaving disk subsystem, data that have been successfully synced will
+stay on the disk. Sane means:
+
+* write errors not allowed (NO-WRITE-ERRORS)
+
+* don't damage the old data on a failed write (ATOMIC-WRITES-ON-POWERFAIL)
+
+  Ext3 handles trash getting written into sectors during powerfail
+  surprisingly well.  It's not foolproof, but it is resilient.
+  Incomplete journal entries are ignored, and journal replay of
+  complete entries will often "repair" garbage written into the inode
+  table.  The data=journal option extends this behavior to file and
+  directory data blocks as well.
+
+
+and obviously:
+
+* don't cause collateral damage to adjacent sectors on a failed write
+  (NO-COLLATERALS)
+
+
+(see expectations.txt; note that most/all linux block-based
+filesystems have similar expectations)
+
+* either write caching is disabled, or hw can do barriers and they are enabled.
+
+	   (Note that barriers are disabled by default, use "barrier=1"
+	   mount option after making sure hw can support them). 
+
+	   hdparm -I reports disk features. If you have "Native
+	   Command Queueing" is the feature you are looking for.
+
+
 References
 ==========
 

Quoted 8 lines

Quoted 7 lines

If your concern is that with Linux MD, you could potentially lose an entire stripe in RAID 5 mode, then you should say that explicitly; but again, this isn't a filesystem specific cliam; it's true for all filesystems. I don't know of any file system that can survive having a RAID stripe-shaped-hole blown into the middle of it due to a power failure.

Quoted 4 lines

Again, ext2 handles that in a way user expects it.

At least I was teached "ext2 needs fsck after powerfail; ext3 can handle powerfails just ok".

So, would you be happy if ext3 fsck was always run on reboot (at least for flash devices)?

Hi!

Quoted 10 lines

Quoted 10 lines

Quoted 7 lines

If your concern is that with Linux MD, you could potentially lose an entire stripe in RAID 5 mode, then you should say that explicitly; but again, this isn't a filesystem specific cliam; it's true for all filesystems. I don't know of any file system that can survive having a RAID stripe-shaped-hole blown into the middle of it due to a power failure.

Again, ext2 handles that in a way user expects it.

Quoted 5 lines

Quoted 2 lines

At least I was teached "ext2 needs fsck after powerfail; ext3 can handle powerfails just ok".

So, would you be happy if ext3 fsck was always run on reboot (at least for flash devices)?

For flash devices, MD Raid 5 and anything else that needs it; yes that would make me happy ;-).

Quoted 1 line

So, would you be happy if ext3 fsck was always run on reboot (at least for flash devices)?

For flash devices, MD Raid 5 and anything else that needs it; yes that would make me happy ;-).

Quoted 5 lines · Pavel Machek

Quoted 2 lines

So, would you be happy if ext3 fsck was always run on reboot (at least for flash devices)?

For flash devices, MD Raid 5 and anything else that needs it; yes that would make me happy ;-).

Sorry, but that just shows your naivete.

Metadata takes up such a small part of the disk that fscking it and finding it to be OK is absolutely no guarantee that the data on the filesystem has not been horribly mangled.

Personally, what I care about is my data.

The metadata is just a way to get to my data, while the data is actually important.

Quoted 14 lines · Rik van Riel

Quoted 5 lines · Pavel Machek

Quoted 2 lines

So, would you be happy if ext3 fsck was always run on reboot (at least for flash devices)?

For flash devices, MD Raid 5 and anything else that needs it; yes that would make me happy ;-).

Sorry, but that just shows your naivete.

Metadata takes up such a small part of the disk that fscking it and finding it to be OK is absolutely no guarantee that the data on the filesystem has not been horribly mangled.

Quoted 4 lines

Personally, what I care about is my data.

The metadata is just a way to get to my data, while the data is actually important.

Personally, I care about metadata consistency, and ext3 documentation suggests that journal protects its integrity. Except that it does not on broken storage devices, and you still need to run fsck there.

How do you protect your data is another question, but ext3 documentation does not claim journal to protect them, so that's up to the user I guess.
Pavel

Quoted 7 lines

Quoted 3 lines

The metadata is just a way to get to my data, while the data is actually important.

Personally, I care about metadata consistency, and ext3 documentation suggests that journal protects its integrity. Except that it does not on broken storage devices, and you still need to run fsck there.

as the ext3 authors have stated many times over the years, you still need to run fsck periodicly anyway.

Quoted 9 lines

Quoted 6 lines

Quoted 2 lines

The metadata is just a way to get to my data, while the data is actually important.

Personally, I care about metadata consistency, and ext3 documentation suggests that journal protects its integrity. Except that it does not on broken storage devices, and you still need to run fsck there.

as the ext3 authors have stated many times over the years, you still need to run fsck periodicly anyway.

Where is that documented?

I very much agree with that, but when suse10
switched periodic fsck off, I could not find any docs to show that it is bad idea.
Pavel

Quoted 5 lines

Quoted 3 lines

as the ext3 authors have stated many times over the years, you still need to run fsck periodicly anyway.

Where is that documented?

linux-kernel mailing list archives.

Quoted 4 lines · On Wed, Aug 26, 2009 at 06:43:24AM -0700, [email protected] wrote:

Quoted 3 lines

Quoted 2 lines

as the ext3 authors have stated many times over the years, you still need to run fsck periodicly anyway.

Where is that documented?

linux-kernel mailing list archives.

Probably from some 6-8 years ago, in e-mail postings that I made. My argument has always been that PC-class hardware is crap, and it's a Really Good Idea to periodically check the metadata because corruption there can end up causing massive data loss. The main problem is that doing it at reboot time really hurt system availability, and "after 20 reboots (plus or minus)" resulted in fsck checks at wildly varying intervals depending on how often people reboot.

What I've been recommending for some time is that people use LVM, and run fsck on a snapshot every week or two, at some convenient time when the system load is at a minimum. There is an e2croncheck script in the e2fsprogs sources, in the contrib directory; it's short enough that I'll attach here here.

Is it *necessary*? In a world where hardware is perfect, no. In a world where people don't bother buying ECC memory because it's 10% more expensive, and PC builders use the cheapest possible parts --- I think it's a really good idea.

- Ted

Quoted 7 lines · On Wed, Aug 26, 2009 at 06:43:24AM -0700, [email protected] wrote:

Quoted 5 lines

Quoted 3 lines

as the ext3 authors have stated many times over the years, you still need to run fsck periodicly anyway.

Where is that documented?

linux-kernel mailing list archives.

Probably from some 6-8 years ago, in e-mail postings that I made. My argument has always been that PC-class hardware is crap, and it's a

Quoted 7 lines · On Wed, Aug 26, 2009 at 06:43:24AM -0700, [email protected] wrote:

Quoted 5 lines

Quoted 3 lines

as the ext3 authors have stated many times over the years, you still need to run fsck periodicly anyway.

Where is that documented?

linux-kernel mailing list archives.

Probably from some 6-8 years ago, in e-mail postings that I made. My argument has always been that PC-class hardware is crap, and it's a

Well, in SUSE11-or-so, distro stopped period fscks, silently :-(. I believed that it was really bad idea at that point, but because I could not find piece of documentation recommending them, I lost the argument.

So these days, what I strongly recommend is that people use LVM snapshots, and schedule weekly checks during some low usage period (i.e., 3am on Saturdays), using something like the e2croncheck shell script.

Quoted 7 lines · Pavel Machek

Quoted 7 lines

Quoted 6 lines

Quoted 2 lines

The metadata is just a way to get to my data, while the data is actually important.

Personally, I care about metadata consistency, and ext3 documentation suggests that journal protects its integrity. Except that it does not on broken storage devices, and you still need to run fsck there.

Quoted 4 lines

Quoted 2 lines

as the ext3 authors have stated many times over the years, you still need to run fsck periodicly anyway.

Where is that documented?

linux-kernel mailing list archives.

Quoted 4 lines

Metadata takes up such a small part of the disk that fscking it and finding it to be OK is absolutely no guarantee that the data on the filesystem has not been horribly mangled.

Quoted 4 lines

Personally, what I care about is my data.

The metadata is just a way to get to my data, while the data is actually important.

Personally, I care about metadata consistency, and ext3 documentation suggests that journal protects its integrity. Except that it does not on broken storage devices, and you still need to run fsck there.

Quoted 4 lines · Pavel Machek

Quoted 4 lines

Metadata takes up such a small part of the disk that fscking it and finding it to be OK is absolutely no guarantee that the data on the filesystem has not been horribly mangled.

Quoted 8 lines

Quoted 4 lines

Personally, what I care about is my data.

The metadata is just a way to get to my data, while the data is actually important.

Personally, I care about metadata consistency, and ext3 documentation suggests that journal protects its integrity. Except that it does not on broken storage devices, and you still need to run fsck there.

Caring about metadata consistency and not data is just weird, I'm sorry. I can't imagine anyone who actually *cares* about what they have stored, whether it's digital photographs of child taking a first step, or their thesis research, caring about more about the metadata than the data. Giving advice that pretends that most users have that priority is Just Wrong.

Quoted 14 lines · On Wednesday 26 August 2009 07:28:13 Theodore Tso wrote:

Quoted 4 lines · Pavel Machek

Quoted 4 lines

Metadata takes up such a small part of the disk that fscking it and finding it to be OK is absolutely no guarantee that the data on the filesystem has not been horribly mangled.

Quoted 8 lines

Quoted 4 lines

Personally, what I care about is my data.

The metadata is just a way to get to my data, while the data is actually important.

Personally, I care about metadata consistency, and ext3 documentation suggests that journal protects its integrity. Except that it does not on broken storage devices, and you still need to run fsck there.

Quoted 6 lines

Caring about metadata consistency and not data is just weird, I'm sorry. I can't imagine anyone who actually *cares* about what they have stored, whether it's digital photographs of child taking a first step, or their thesis research, caring about more about the metadata than the data. Giving advice that pretends that most users have that priority is Just Wrong.

I thought the reason for that was that if your metadata is horked, further writes to the disk can trash unrelated existing data because it's lost track of what's allocated and what isn't. So back when the assumption was "what's written stays written", then keeping the metadata sane was still darn important to prevent normal operation from overwriting unrelated existing data.

Then Pavel notified us of a situation where interrupted writes to the disk can trash unrelated existing data _anyway_, because the flash block size on the 16 gig flash key I bought retail at Fry's is 2 megabytes, and the filesystem thinks it's 4k or smaller. It seems like what _broke_ was the assumption that the filesystem block size >= the disk block size, and nobody noticed for a while. (Except the people making jffs2 and friends, anyway.)

Today we have cheap plentiful USB keys that act like hard drives, except that their write block size isn't remotely the same as hard drives', but they pretend it is, and then the block wear levelling algorithms fuzz things further. (Gee, a drive controller lying about drive geometry, the scsi crowd should feel right at home.)

Now Pavel's coming back with a second situation where RAID stripes (under certain circumstances) seem to have similar granularity issues, again breaking what seems to be the same assumption. Big media use big chunks for data, and media is getting bigger. It doesn't seem like this problem is going to diminish in future.

I agree that it seems like a good idea to have BIG RED WARNING SIGNS about those kind of media and how _any_ journaling filesystem doesn't really help here. So specifically documenting "These kinds of media lose unrelated random data if writes to them are interrupted, journaling filesystems can't help with this and may actually hide the problem, and even an fsck will only find corrupted metadata not lost file contents" seems kind of useful.

That said, ext3's assumption that filesystem block size always >= disk update block size _is_ a fundamental part of this problem, and one that isn't shared by things like jffs2, and which things like btrfs might be able to address if they try, by adding awareness of the real media update granularity to their node layout algorithms. (Heck, ext2 has a stripe size parameter already. Does setting that appropriately for your raid make this suck less? I haven't heard anybody comment on that one yet...)

Quoted 37 lines · Rob Landley

Quoted 14 lines · On Wednesday 26 August 2009 07:28:13 Theodore Tso wrote:

Quoted 4 lines · Pavel Machek

Quoted 4 lines

Metadata takes up such a small part of the disk that fscking it and finding it to be OK is absolutely no guarantee that the data on the filesystem has not been horribly mangled.

Quoted 8 lines

Quoted 4 lines

Personally, what I care about is my data.

The metadata is just a way to get to my data, while the data is actually important.

Personally, I care about metadata consistency, and ext3 documentation suggests that journal protects its integrity. Except that it does not on broken storage devices, and you still need to run fsck there.

Quoted 6 lines

Caring about metadata consistency and not data is just weird, I'm sorry. I can't imagine anyone who actually *cares* about what they have stored, whether it's digital photographs of child taking a first step, or their thesis research, caring about more about the metadata than the data. Giving advice that pretends that most users have that priority is Just Wrong.

I thought the reason for that was that if your metadata is horked, further writes to the disk can trash unrelated existing data because it's lost track of what's allocated and what isn't. So back when the assumption was "what's written stays written", then keeping the metadata sane was still darn important to prevent normal operation from overwriting unrelated existing data.

Then Pavel notified us of a situation where interrupted writes to the disk can trash unrelated existing data _anyway_, because the flash block size on the 16 gig flash key I bought retail at Fry's is 2 megabytes, and the filesystem thinks it's 4k or smaller. It seems like what _broke_ was the assumption that the filesystem block size >= the disk block size, and nobody noticed for a while. (Except the people making jffs2 and friends, anyway.)

Quoted 5 lines

Today we have cheap plentiful USB keys that act like hard drives, except that their write block size isn't remotely the same as hard drives', but they pretend it is, and then the block wear levelling algorithms fuzz things further. (Gee, a drive controller lying about drive geometry, the scsi crowd should feel right at home.)

actually, you don't know if your USB key works that way or not.

Pavel has ssome that do, that doesn't mean that all flash drives do

Let's start with hardware wear-leveling. Basically, nearly all practical implementations of it suck. You'd imagine that it would spread out writes over all the blocks in the drive, only rewriting any particular block after every other block has been written. But I've heard from experts several times that hardware wear-leveling can be as dumb as a ring buffer of 12 blocks; each time you write a block, it pulls something out of the queue and sticks the old block in. If you only write one block over and over, this means that writes will be spread out over a staggering 12 blocks! My direct experience working with corrupted flash with built-in wear-leveling

is that corruption was centered around frequently written blocks (with interesting patterns resulting from the interleaving of blocks from different erase blocks). As a file systems person, I know what it takes to do high-quality wear-leveling: it's called a log-structured file system and they are non-trivial pieces of software. Your average consumer SSD is not going to have sufficient hardware to implement even a half-assed log-structured file system, so clearly it's going to be a lot stupider than that.

when you do a write to a flash drive you have to do the following items

1. allocate an empty eraseblock to put the data on

2. read the old eraseblock

3. merge the incoming write to the eraseblock

4. write the updated data to the flash

5. update the flash trnslation layer to point reads at the new location instead of the old location.

now if the flash drive does things in this order you will not loose any previously written data.

if the flash drive does step 5 before it does step 4, then you have a window where a crash can loose data (and no btrfs won't survive any better to have a large chunk of data just disappear)

it's possible that some super-cheap flash drives

but if the device doesn't have a flash translation layer, then repeated writes to any one sector will kill the drive fairly quickly. (updates to the FAT would kill the sectors the FAT, journal, root directory, or superblock lives in due to the fact that every change to the disk requires an update to this file for example)

Quoted 7 lines

That said, ext3's assumption that filesystem block size always >= disk update block size _is_ a fundamental part of this problem, and one that isn't shared by things like jffs2, and which things like btrfs might be able to address if they try, by adding awareness of the real media update granularity to their node layout algorithms. (Heck, ext2 has a stripe size parameter already. Does setting that appropriately for your raid make this suck less? I haven't heard anybody comment on that one yet...)

I thought that that assumption was in the VFS layer, not in any particular filesystem

David Lang

Quoted 9 lines · On Thursday 27 August 2009 01:54:30 [email protected] wrote:

Quoted 6 lines · Rob Landley

Today we have cheap plentiful USB keys that act like hard drives, except that their write block size isn't remotely the same as hard drives', but they pretend it is, and then the block wear levelling algorithms fuzz things further. (Gee, a drive controller lying about drive geometry, the scsi crowd should feel right at home.)

actually, you don't know if your USB key works that way or not.

Um, yes, I think I do.

Quoted 1 line

Pavel has ssome that do, that doesn't mean that all flash drives do

Pretty much all the ones that present a USB disk interface to the outside world and then thus have to do hardware levelling. Here's Valerie Aurora on the topic:

http://valhenson.livejournal.com/25228.html

Quoted 9 lines

Let's start with hardware wear-leveling. Basically, nearly all practical implementations of it suck. You'd imagine that it would spread out writes over all the blocks in the drive, only rewriting any particular block after every other block has been written. But I've heard from experts several times that hardware wear-leveling can be as dumb as a ring buffer of 12 blocks; each time you write a block, it pulls something out of the queue and sticks the old block in. If you only write one block over and over, this means that writes will be spread out over a staggering 12 blocks! My direct experience working with corrupted flash with built-in wear-leveling

Quoted 8 lines

is that corruption was centered around frequently written blocks (with interesting patterns resulting from the interleaving of blocks from different erase blocks). As a file systems person, I know what it takes to do high-quality wear-leveling: it's called a log-structured file system and they are non-trivial pieces of software. Your average consumer SSD is not going to have sufficient hardware to implement even a half-assed log-structured file system, so clearly it's going to be a lot stupider than that.

Back to you:

Quoted 8 lines

when you do a write to a flash drive you have to do the following items

1. allocate an empty eraseblock to put the data on

2. read the old eraseblock

3. merge the incoming write to the eraseblock

Quoted 7 lines

4. write the updated data to the flash

5. update the flash trnslation layer to point reads at the new location instead of the old location.

now if the flash drive does things in this order you will not loose any previously written data.

That's what something like jffs2 will do, sure. (And note that mounting those suckers is slow while it reads the whole disk to figure out what order to put the chunks in.)

However, your average consumer level device A) isn't very smart, B) is judged almost entirely by price/capacity ratio and thus usually won't even hide capacity for bad block remapping. You expect them to have significant hidden capacity to do safer updates with when customers aren't demanding it yet?

Quoted 5 lines

if the flash drive does step 5 before it does step 4, then you have a window where a crash can loose data (and no btrfs won't survive any better to have a large chunk of data just disappear)

it's possible that some super-cheap flash drives

I've never seen one that presented a USB disk interface that _didn't_ do this. (Not that this observation means much.) Neither the windows nor the Macintosh world is calling for this yet. Even the Linux guys barely know about it. And these are the same kinds of manufacturers that NOPed out the flush commands to make their benchmarks look better...

Quoted 5 lines

but if the device doesn't have a flash translation layer, then repeated writes to any one sector will kill the drive fairly quickly. (updates to the FAT would kill the sectors the FAT, journal, root directory, or superblock lives in due to the fact that every change to the disk requires an update to this file for example)

Yup. It's got enough of one to get past the warantee, but beyond that they're intended for archiving and sneakernet, not for running compiles on.

Quoted 8 lines

Quoted 7 lines

That said, ext3's assumption that filesystem block size always >= disk update block size _is_ a fundamental part of this problem, and one that isn't shared by things like jffs2, and which things like btrfs might be able to address if they try, by adding awareness of the real media update granularity to their node layout algorithms. (Heck, ext2 has a stripe size parameter already. Does setting that appropriately for your raid make this suck less? I haven't heard anybody comment on that one yet...)

Quoted 2 lines

I thought that that assumption was in the VFS layer, not in any particular filesystem

The VFS layer cares about how to talk to the backing store? I thought that was the filesystem driver's job...

I wonder how jffs2 gets around it, then? (Or for that matter, squashfs...)

Quoted 14 lines

I thought the reason for that was that if your metadata is horked, further writes to the disk can trash unrelated existing data because it's lost track of what's allocated and what isn't. So back when the assumption was "what's written stays written", then keeping the metadata sane was still darn important to prevent normal operation from overwriting unrelated existing data.

Then Pavel notified us of a situation where interrupted writes to the disk can trash unrelated existing data _anyway_, because the flash block size on the 16 gig flash key I bought retail at Fry's is 2 megabytes, and the filesystem thinks it's 4k or smaller. It seems like what _broke_ was the assumption that the filesystem block size >= the disk block size, and nobody noticed for a while. (Except the people making jffs2 and friends, anyway.)

Quoted 5 lines

Today we have cheap plentiful USB keys that act like hard drives, except that their write block size isn't remotely the same as hard drives', but they pretend it is, and then the block wear levelling algorithms fuzz things further. (Gee, a drive controller lying about drive geometry, the scsi crowd should feel right at home.)

actually, you don't know if your USB key works that way or not. Pavel has ssome that do, that doesn't mean that all flash drives do

when you do a write to a flash drive you have to do the following items

1. allocate an empty eraseblock to put the data on

2. read the old eraseblock

3. merge the incoming write to the eraseblock

4. write the updated data to the flash

5. update the flash trnslation layer to point reads at the new location instead of the old location.

Quoted 31 lines

Quoted 14 lines

I thought the reason for that was that if your metadata is horked, further writes to the disk can trash unrelated existing data because it's lost track of what's allocated and what isn't. So back when the assumption was "what's written stays written", then keeping the metadata sane was still darn important to prevent normal operation from overwriting unrelated existing data.

Then Pavel notified us of a situation where interrupted writes to the disk can trash unrelated existing data _anyway_, because the flash block size on the 16 gig flash key I bought retail at Fry's is 2 megabytes, and the filesystem thinks it's 4k or smaller. It seems like what _broke_ was the assumption that the filesystem block size >= the disk block size, and nobody noticed for a while. (Except the people making jffs2 and friends, anyway.)

Quoted 5 lines

Today we have cheap plentiful USB keys that act like hard drives, except that their write block size isn't remotely the same as hard drives', but they pretend it is, and then the block wear levelling algorithms fuzz things further. (Gee, a drive controller lying about drive geometry, the scsi crowd should feel right at home.)

actually, you don't know if your USB key works that way or not. Pavel has ssome that do, that doesn't mean that all flash drives do

when you do a write to a flash drive you have to do the following items

1. allocate an empty eraseblock to put the data on

2. read the old eraseblock

3. merge the incoming write to the eraseblock

Quoted 4 lines

4. write the updated data to the flash

5. update the flash trnslation layer to point reads at the new location instead of the old location.

That would need two erases per single sector writen, no? Erase is in milisecond range, so the performance would be just way too bad :-(.

Quoted 5 lines · Pavel Machek

Quoted 2 lines

So, would you be happy if ext3 fsck was always run on reboot (at least for flash devices)?

For flash devices, MD Raid 5 and anything else that needs it; yes that would make me happy ;-).

Sorry, but that just shows your naivete.

Metadata takes up such a small part of the disk that fscking it and finding it to be OK is absolutely no guarantee that the data on the filesystem has not been horribly mangled.

Personally, what I care about is my data.

The metadata is just a way to get to my data, while the data is actually important.

Then to answer your question... ext2. You expect to run fsck after unclean shutdown, and you expect to have to solve some problems with it. So the way ext2 deals with the flash media actually matches what the user expects. (*)

OTOH in ext3 case you expect consistent filesystem after unplug; and you don't get that.

Quoted 4 lines · Pavel Machek

Then to answer your question... ext2. You expect to run fsck after unclean shutdown, and you expect to have to solve some problems with it. So the way ext2 deals with the flash media actually matches what the user expects. (*)

But if the 256k hole is in data blocks, fsck won't find a problem, even with ext2.

And if the 256k hole is the inode table, you will *still* suffer massive data loss. Fsck will tell you how badly screwed you are, but it doesn't "fix" the disk; most users don't consider questions of the form "directory entry <precious-thesis-data> points to trashed inode, may I delete directory entry?" as being terribly helpful. :-/

Maybe this came as a surprise to you, but anyone who has used a compact flash in a digital camera knows that you ***have*** to wait until the led has gone out before trying to eject the flash card. I remember seeing all sorts of horror stories from professional photographers about how they lost an important wedding's day worth of pictures with the attendant commercial loss, on various digital photography forums. It tends to be the sort of mistake that digital photographers only make once.

Quoted 4 lines · Pavel Machek

Then to answer your question... ext2. You expect to run fsck after unclean shutdown, and you expect to have to solve some problems with it. So the way ext2 deals with the flash media actually matches what the user expects. (*)

But if the 256k hole is in data blocks, fsck won't find a problem, even with ext2.

And if the 256k hole is the inode table, you will *still* suffer massive data loss. Fsck will tell you how badly screwed you are, but it doesn't "fix" the disk; most users don't consider questions of the form "directory entry <precious-thesis-data> points to trashed inode, may I delete directory entry?" as being terribly helpful. :-/

Maybe this came as a surprise to you, but anyone who has used a compact flash in a digital camera knows that you ***have*** to wait until the led has gone out before trying to eject the flash card. I remember seeing all sorts of horror stories from professional photographers about how they lost an important wedding's day worth of pictures with the attendant commercial loss, on various digital photography forums. It tends to be the sort of mistake that digital photographers only make once.

Quoted 8 lines · Theodore Tso

Quoted 4 lines · Pavel Machek

Then to answer your question... ext2. You expect to run fsck after unclean shutdown, and you expect to have to solve some problems with it. So the way ext2 deals with the flash media actually matches what the user expects. (*)

But if the 256k hole is in data blocks, fsck won't find a problem, even with ext2.

True.

Quoted 5 lines

And if the 256k hole is the inode table, you will *still* suffer massive data loss. Fsck will tell you how badly screwed you are, but it doesn't "fix" the disk; most users don't consider questions of the form "directory entry<precious-thesis-data> points to trashed inode, may I delete directory entry?" as being terribly helpful. :-/

Well it will fix the disk in the end. And no, "directory entry <precious-thesis-data> points to trashed inode, may I delete directory entry?" is not _terribly_ helpful, but it is slightly helpful and people actually expect that from ext2.

Quoted 8 lines

Maybe this came as a surprise to you, but anyone who has used a compact flash in a digital camera knows that you ***have*** to wait until the led has gone out before trying to eject the flash card. I remember seeing all sorts of horror stories from professional photographers about how they lost an important wedding's day worth of pictures with the attendant commercial loss, on various digital photography forums. It tends to be the sort of mistake that digital photographers only make once.

It actually comes as surprise to me. Actually yes and no. I know that digital cameras use VFAT, so pulling CF card out of it may do bad thing, unless I run fsck.vfat afterwards. If digital camera was using ext3, I'd expect it to be safely pullable at any time.

Will IBM microdrive do any difference there?

Anyway, it was not known to me. Rather than claiming "everyone knows" (when clearly very few people really understand all the details), can we simply document that?
Pavel

I really think that the expectation that all OS's (windows, mac, even your ipod) all teach you not to hot unplug a device with any file system. Users have an "eject" or "safe unload" in windows, your iPod tells you not to power off or disconnect, etc.

I don't object to making that general statement - "Don't hot unplug a device with an active file system or actively used raw device" - but would object to the overly general statement about ext3 not working on flash, RAID5 not working, etc...

On Tue, 25 Aug 2009 09:37:12 -0400

Quoted 4 lines · Ric Wheeler

I really think that the expectation that all OS's (windows, mac, even your ipod) all teach you not to hot unplug a device with any file system. Users have an "eject" or "safe unload" in windows, your iPod tells you not to power off or disconnect, etc.

Agreed

Quoted 4 lines

I don't object to making that general statement - "Don't hot unplug a device with an active file system or actively used raw device" - but would object to the overly general statement about ext3 not working on flash, RAID5 not working, etc...

The overall general statement for all media and all OS's should be

"Do you have a backup, have you tested it recently"

Quoted 17 lines

Quoted 9 lines

Maybe this came as a surprise to you, but anyone who has used a compact flash in a digital camera knows that you ***have*** to wait until the led has gone out before trying to eject the flash card. I remember seeing all sorts of horror stories from professional photographers about how they lost an important wedding's day worth of pictures with the attendant commercial loss, on various digital photography forums. It tends to be the sort of mistake that digital photographers only make once.

It actually comes as surprise to me. Actually yes and no. I know that digital cameras use VFAT, so pulling CF card out of it may do bad thing, unless I run fsck.vfat afterwards. If digital camera was using ext3, I'd expect it to be safely pullable at any time.

Will IBM microdrive do any difference there?

Quoted 3 lines

Anyway, it was not known to me. Rather than claiming "everyone knows" (when clearly very few people really understand all the details), can we simply document that?

I really think that the expectation that all OS's (windows, mac, even your ipod) all teach you not to hot unplug a device with any file system. Users have an "eject" or "safe unload" in windows, your iPod tells you not to power off or disconnect, etc.

I don't object to making that general statement - "Don't hot unplug a device with an active file system or actively used raw device" - but would object to the overly general statement about ext3 not working on flash, RAID5 not working, etc...

Quoted 21 lines

Quoted 20 lines

Quoted 9 lines

Maybe this came as a surprise to you, but anyone who has used a compact flash in a digital camera knows that you ***have*** to wait until the led has gone out before trying to eject the flash card. I remember seeing all sorts of horror stories from professional photographers about how they lost an important wedding's day worth of pictures with the attendant commercial loss, on various digital photography forums. It tends to be the sort of mistake that digital photographers only make once.

It actually comes as surprise to me. Actually yes and no. I know that digital cameras use VFAT, so pulling CF card out of it may do bad thing, unless I run fsck.vfat afterwards. If digital camera was using ext3, I'd expect it to be safely pullable at any time.

Will IBM microdrive do any difference there?

Anyway, it was not known to me. Rather than claiming "everyone knows" (when clearly very few people really understand all the details), can we simply document that?

Quoted 4 lines

I really think that the expectation that all OS's (windows, mac, even your ipod) all teach you not to hot unplug a device with any file system. Users have an "eject" or "safe unload" in windows, your iPod tells you not to power off or disconnect, etc.

That was before journaling filesystems...

Quoted 5 lines

I don't object to making that general statement - "Don't hot unplug a device with an active file system or actively used raw device" - but would object to the overly general statement about ext3 not working on flash, RAID5 not working, etc...

You can object any way you want, but running ext3 on flash or MD RAID5 is stupid:

* ext2 would be faster

* ext2 would provide better protection against powerfail.

"ext3 works on flash and MD RAID5, as long as you do not have powerfail" seems to be the accurate statement, and if you don't need to protect against powerfails, you can just use ext2.
Pavel

Quoted 7 lines

Quoted 6 lines

I really think that the expectation that all OS's (windows, mac, even your ipod) all teach you not to hot unplug a device with any file system. Users have an "eject" or "safe unload" in windows, your iPod tells you not to power off or disconnect, etc.

Quoted 1 line

That was before journaling filesystems...

Not true - that is true today with or without journals as we have discussed in great detail. Including specifically ext2.

Basically, any file system (Linux, windows, OSX, etc) that writes into the page cache will lose data when you hot unplug its storage. End of story, don't do it!

Quoted 8 lines

Quoted 4 lines

I don't object to making that general statement - "Don't hot unplug a device with an active file system or actively used raw device" - but would object to the overly general statement about ext3 not working on flash, RAID5 not working, etc...

You can object any way you want, but running ext3 on flash or MD RAID5 is stupid:

Quoted 3 lines

* ext2 would be faster

* ext2 would provide better protection against powerfail.

Not true in the slightest, you continue to ignore the ext2/3/4 developers telling you that it will lose data.

Quoted 3 lines

"ext3 works on flash and MD RAID5, as long as you do not have powerfail" seems to be the accurate statement, and if you don't need to protect against powerfails, you can just use ext2.

Strange how your personal preference is totally out of sync with the entire enterprise class user base.

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

I really think that the expectation that all OS's (windows, mac, even your ipod) all teach you not to hot unplug a device with any file system. Users have an "eject" or "safe unload" in windows, your iPod tells you not to power off or disconnect, etc.

Quoted 8 lines

Quoted 1 line

That was before journaling filesystems...

Not true - that is true today with or without journals as we have discussed in great detail. Including specifically ext2.

Basically, any file system (Linux, windows, OSX, etc) that writes into the page cache will lose data when you hot unplug its storage. End of story, don't do it!

No, not ext3 on SATA disk with barriers on and proper use of fsync(). I actually tested that.

Yes, I should be able to hotunplug SATA drives and expect the data that was fsync-ed to be there.

Quoted 13 lines

Quoted 12 lines

Quoted 5 lines

I don't object to making that general statement - "Don't hot unplug a device with an active file system or actively used raw device" - but would object to the overly general statement about ext3 not working on flash, RAID5 not working, etc...

You can object any way you want, but running ext3 on flash or MD RAID5 is stupid:

* ext2 would be faster

* ext2 would provide better protection against powerfail.

Quoted 2 lines

Not true in the slightest, you continue to ignore the ext2/3/4 developers telling you that it will lose data.

I know I will lose data. Both ext2 and ext3 will lose data on flashdisk. (That's what I'm trying to document). But... what is the benefit of ext3 journaling on MD RAID5? (On flash, ext3 at least protects you against kernel panic. MD RAID5 is in software, so... that additional protection is just not there).

Quoted 7 lines

Quoted 4 lines

"ext3 works on flash and MD RAID5, as long as you do not have powerfail" seems to be the accurate statement, and if you don't need to protect against powerfails, you can just use ext2.

Strange how your personal preference is totally out of sync with the entire enterprise class user base.

Perhaps noone told them MD RAID5 is dangerous? You see, that's exactly what I'm trying to document here.
Pavel

Quoted 8 lines

Quoted 4 lines

Basically, any file system (Linux, windows, OSX, etc) that writes into the page cache will lose data when you hot unplug its storage. End of story, don't do it!

No, not ext3 on SATA disk with barriers on and proper use of fsync(). I actually tested that.

Quoted 2 lines

Yes, I should be able to hotunplug SATA drives and expect the data that was fsync-ed to be there.

You can and will lose data (even after fsync) with any type of storage at some rate. What you are missing here is that data loss needs to be measured in hard numbers - say percentage of installed boxes that have config X that lose data.

I can promise you that hot unplugging and replugging a S-ATA drive will also lose you data if you are actively writing to it (ext2, 3, whatever).

Quoted 15 lines

Quoted 14 lines

Quoted 11 lines

Quoted 4 lines

I don't object to making that general statement - "Don't hot unplug a device with an active file system or actively used raw device" - but would object to the overly general statement about ext3 not working on flash, RAID5 not working, etc...

You can object any way you want, but running ext3 on flash or MD RAID5 is stupid:

* ext2 would be faster

* ext2 would provide better protection against powerfail.

Not true in the slightest, you continue to ignore the ext2/3/4 developers telling you that it will lose data.

Quoted 5 lines

I know I will lose data. Both ext2 and ext3 will lose data on flashdisk. (That's what I'm trying to document). But... what is the benefit of ext3 journaling on MD RAID5? (On flash, ext3 at least protects you against kernel panic. MD RAID5 is in software, so... that additional protection is just not there).

Faster recovery time on any normal kernel crash or power outage. Data loss would be equivalent with or without the journal.

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

Basically, any file system (Linux, windows, OSX, etc) that writes into the page cache will lose data when you hot unplug its storage. End of story, don't do it!

No, not ext3 on SATA disk with barriers on and proper use of fsync(). I actually tested that.

Quoted 7 lines

Quoted 2 lines

Yes, I should be able to hotunplug SATA drives and expect the data that was fsync-ed to be there.

You can and will lose data (even after fsync) with any type of storage at some rate. What you are missing here is that data loss needs to be measured in hard numbers - say percentage of installed boxes that have config X that lose data.

I'm talking "by design" here.

I will lose data even on SATA drive that is properly powered on if I wait 5 years.

Quoted 2 lines

I can promise you that hot unplugging and replugging a S-ATA drive will also lose you data if you are actively writing to it (ext2, 3, whatever).

I can promise you that running S-ATA drive will also lose you data, even if you are not actively writing to it. Just wait 10 years; so what is your point?

But ext3 is _designed_ to preserve fsynced data on SATA drive, while it is _not_ designed to preserve fsynced data on MD RAID5.

Do you really think that's not a difference?

Quoted 22 lines

Quoted 21 lines

Quoted 9 lines

Quoted 9 lines

Quoted 5 lines

I don't object to making that general statement - "Don't hot unplug a device with an active file system or actively used raw device" - but would object to the overly general statement about ext3 not working on flash, RAID5 not working, etc...

You can object any way you want, but running ext3 on flash or MD RAID5 is stupid:

Quoted 6 lines

Quoted 3 lines

* ext2 would be faster

* ext2 would provide better protection against powerfail.

Not true in the slightest, you continue to ignore the ext2/3/4 developers telling you that it will lose data.

I know I will lose data. Both ext2 and ext3 will lose data on flashdisk. (That's what I'm trying to document). But... what is the benefit of ext3 journaling on MD RAID5? (On flash, ext3 at least protects you against kernel panic. MD RAID5 is in software, so... that additional protection is just not there).

Quoted 2 lines

Faster recovery time on any normal kernel crash or power outage. Data loss would be equivalent with or without the journal.

No, because you'll actually repair the ext2 with fsck after the kernel crash or power outage. Data loss will not be equivalent; in particular you'll not lose data writen _after_ power outage to ext2.
Pavel

Quoted 18 lines · Pavel Machek

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

Basically, any file system (Linux, windows, OSX, etc) that writes into the page cache will lose data when you hot unplug its storage. End of story, don't do it!

No, not ext3 on SATA disk with barriers on and proper use of fsync(). I actually tested that.

Quoted 7 lines

Quoted 2 lines

Yes, I should be able to hotunplug SATA drives and expect the data that was fsync-ed to be there.

You can and will lose data (even after fsync) with any type of storage at some rate. What you are missing here is that data loss needs to be measured in hard numbers - say percentage of installed boxes that have config X that lose data.

I'm talking "by design" here.

Quoted 3 lines

I will lose data even on SATA drive that is properly powered on if I wait 5 years.

You are dead wrong.

For RAID5 arrays, you assume that you have a hard failure and a power outage before you can rebuild the RAID (order of hours at full tilt).

Quoted 6 lines

Quoted 2 lines

I can promise you that hot unplugging and replugging a S-ATA drive will also lose you data if you are actively writing to it (ext2, 3, whatever).

I can promise you that running S-ATA drive will also lose you data, even if you are not actively writing to it. Just wait 10 years; so what is your point?

I lost a s-ata drive 24 hours after installing it in a new box. If I had MD5 RAID5, I would not have lost any.

Why don't you hold all of your most precious data on that single S-ATA drive for five year on one box and put a second copy on a small RAID5 with ext3 for the same period?

Repeat experiment until you get up to something like google scale or the other papers on failures in national labs in the US and then we can have an informed discussion.

Quoted 7 lines

Why don't you hold all of your most precious data on that single S-ATA drive for five year on one box and put a second copy on a small RAID5 with ext3 for the same period?

Repeat experiment until you get up to something like google scale or the other papers on failures in national labs in the US and then we can have an informed discussion.

I'm not interested in discussing statistics with you. I'd rather discuss fsync() and storage design issues.

ext3 is designed to work on single SATA disks, and it is not designed to work on flash cards/degraded MD RAID5s, as Ted acknowledged.

Because that fact is non obvious to the users, I'd like to see it documented, and now have nice short writeup from Ted.

If you want to argue that ext3/MD RAID5/no UPS combination is still less likely to fail than single SATA disk given part fail probabilities, go ahead and present nice statistics. Its just that I'm not interested in them.
Pavel

Quoted 8 lines · Pavel Machek

Quoted 7 lines

Why don't you hold all of your most precious data on that single S-ATA drive for five year on one box and put a second copy on a small RAID5 with ext3 for the same period?

Repeat experiment until you get up to something like google scale or the other papers on failures in national labs in the US and then we can have an informed discussion.

Quoted 5 lines

I'm not interested in discussing statistics with you. I'd rather discuss fsync() and storage design issues.

ext3 is designed to work on single SATA disks, and it is not designed to work on flash cards/degraded MD RAID5s, as Ted acknowledged.

You are simply incorrect, Ted did not say that ext3 does not work with MD raid5.

Quoted 12 lines · Ric Wheeler

Quoted 11 lines · Pavel Machek

Quoted 7 lines

Why don't you hold all of your most precious data on that single S-ATA drive for five year on one box and put a second copy on a small RAID5 with ext3 for the same period?

Repeat experiment until you get up to something like google scale or the other papers on failures in national labs in the US and then we can have an informed discussion.

I'm not interested in discussing statistics with you. I'd rather discuss fsync() and storage design issues.

Quoted 5 lines

Quoted 2 lines

ext3 is designed to work on single SATA disks, and it is not designed to work on flash cards/degraded MD RAID5s, as Ted acknowledged.

You are simply incorrect, Ted did not say that ext3 does not work with MD raid5.

http://lkml.org/lkml/2009/8/25/312
Pavel

Quoted 5 lines

Quoted 2 lines

You are simply incorrect, Ted did not say that ext3 does not work with MD raid5.

http://lkml.org/lkml/2009/8/25/312
Pavel

I will let Ted clarify his text on his own, but the quoted text says "... have potential...".

Why not ask Neil if he designed MD to not work properly with ext3?

Quoted 11 lines · Ric Wheeler

Quoted 6 lines

Quoted 3 lines

You are simply incorrect, Ted did not say that ext3 does not work with MD raid5.

http://lkml.org/lkml/2009/8/25/312
Pavel

I will let Ted clarify his text on his own, but the quoted text says "... have potential...".

Why not ask Neil if he designed MD to not work properly with ext3?

So let me clarify by saying the following things.

1) Filesystems are designed to expect that storage devices have certain properties. These include returning the same data that you wrote, and that an error when writing a sector, or a power failure when writing sector, should not be amplified to cause collateral damage with previously succfessfully written sectors.

2) Degraded RAID 5/6 filesystems do not meet these properties. Neither to cheap flash drives. This increases the chances you can lose, bigtime.

3) Does that mean that you shouldn't use ext3 on RAID drives? Of course not! First of all, Ext3 still saves you against kernel panics and hangs caused by device driver bugs or other kernel hangs. You will lose less data, and avoid needing to run a long and painful fsck after a forced reboot, compared to if you used ext2. You are making an assumption that the only time running the journal takes place is after a power failure. But if the system hangs, and you need to hit the Big Red Switch, or if you using the system in a Linux High Availability setup and the ethernet card fails, so the STONITH ("shoot the other node in the head") system forces a hard reset of the system, or you get a kernel panic which forces a reboot, in all of these cases ext3 will save you from a long fsck, and it will do so safely.

Secondly, what's the probability of a failure causes the RAID array to become degraded, followed by a power failure, versus a power failure while the RAID array is not running in degraded mode? Hopefully you are running with the RAID array in full, proper running order a much larger percentage of the time than running with the RAID array in degraded mode. If not, the bug is with the system administrator!

If you are someone who tends to run for long periods of time in degraded mode --- then better get a UPS. And certainly if you want to avoid the chances of failure, periodically scrubbing the disks so you detect hard drive failures early, instead of waiting until a disk fails before letting the rebuild find the dreaded "second failure" which causes data loss, is a d*mned good idea.

Maybe a random OS engineer doesn't know these things --- but trust me when I say a competent system administrator had better be familiar with these concepts. And someone who wants their data to be reliably stored needs to do some basic storage engineering if they want to have long-term data reliability. (That, or maybe they should outsource their long-term reliable storage some service such as Amazon S3 --- see Jeremy Zawodny's analysis about how it can be cheaper, here:
http://jeremy.zawodny.com/blog/archives/007624.html)

But we *do* need to be careful that we don't write documentation which is ends up giving users the wrong impression. The bottom line is that you're better off using ext3 over ext2, even on a RAID array, for the reasons listed above.

Are you better off using ext3 over ext2 on a crappy flash drive? Maybe --- if you are also using crappy proprietary video drivers, such as Ubuntu ships, where every single time you exit a 3d game the system crashes (and Ubuntu users accept this as normal?!?), then ext3 might be a better choice since you'll reduce the chance of data loss when the system locks up or crashes thanks to the aforemention crappy proprietary video drivers from Nvidia. On the other hand, crappy flash drives *do* have really bad write amplification effects, where a 4K write can cause 128k or more worth of flash to be rewritten, such that using ext3 could seriously degrade the lifetime of said crappy flash drive; furthermore, the crappy flash drives have such terribly write performance that using ext3 can be a performance nightmare. This of course, doesn't apply to well-implemented SSD's, such as the Intel's X25-M and X18-M. So here your mileage may vary. Still, if you are using crappy proprietary drivers which cause system hangs and crashes at a far greater rate than power fail-induced unclean shutdowns, ext3 *still* might be the better choice, even with crappy flash drives.

The best thing to do, of course, is to improve your storage stack; use competently implemented SSD's instead of crap flash cards. If your hardware RAID card supports a battery option, *get* the battery. Add a UPS to your system. Provision your RAID array with hot spares, and regularly scrub (read-test) your array so that failed drives can be detected early. Make sure you configure your MD setup so that you get e-mail when a hard drive fails and the array starts running in degraded mode, so you can replace the failed drive ASAP.

At the end of the day, filesystems are not magic. They can't compensate for crap hardware, or incompetently administered machines.

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 11 lines · Ric Wheeler

Quoted 6 lines

Quoted 3 lines

You are simply incorrect, Ted did not say that ext3 does not work with MD raid5.

http://lkml.org/lkml/2009/8/25/312
Pavel

I will let Ted clarify his text on his own, but the quoted text says "... have potential...".

Why not ask Neil if he designed MD to not work properly with ext3?

So let me clarify by saying the following things.

1) Filesystems are designed to expect that storage devices have certain properties. These include returning the same data that you wrote, and that an error when writing a sector, or a power failure when writing sector, should not be amplified to cause collateral damage with previously succfessfully written sectors.

2) Degraded RAID 5/6 filesystems do not meet these properties. Neither to cheap flash drives. This increases the chances you can lose, bigtime.

3) Does that mean that you shouldn't use ext3 on RAID drives? Of course not! First of all, Ext3 still saves you against kernel panics and hangs caused by device driver bugs or other kernel hangs. You will lose less data, and avoid needing to run a long and painful fsck after a forced reboot, compared to if you used ext2. You are making an assumption that the only time running the journal takes place is after a power failure. But if the system hangs, and you need to hit the Big Red Switch, or if you using the system in a Linux High Availability setup and the ethernet card fails, so the STONITH ("shoot the other node in the head") system forces a hard reset of the system, or you get a kernel panic which forces a reboot, in all of these cases ext3 will save you from a long fsck, and it will do so safely.

Secondly, what's the probability of a failure causes the RAID array to become degraded, followed by a power failure, versus a power failure while the RAID array is not running in degraded mode? Hopefully you are running with the RAID array in full, proper running order a much larger percentage of the time than running with the RAID array in degraded mode. If not, the bug is with the system administrator!

If you are someone who tends to run for long periods of time in degraded mode --- then better get a UPS. And certainly if you want to avoid the chances of failure, periodically scrubbing the disks so you detect hard drive failures early, instead of waiting until a disk fails before letting the rebuild find the dreaded "second failure" which causes data loss, is a d*mned good idea.

Maybe a random OS engineer doesn't know these things --- but trust me when I say a competent system administrator had better be familiar with these concepts. And someone who wants their data to be reliably stored needs to do some basic storage engineering if they want to have long-term data reliability. (That, or maybe they should outsource their long-term reliable storage some service such as Amazon S3 --- see Jeremy Zawodny's analysis about how it can be cheaper, here:
http://jeremy.zawodny.com/blog/archives/007624.html)

But we *do* need to be careful that we don't write documentation which is ends up giving users the wrong impression. The bottom line is that you're better off using ext3 over ext2, even on a RAID array, for the reasons listed above.

Are you better off using ext3 over ext2 on a crappy flash drive? Maybe --- if you are also using crappy proprietary video drivers, such as Ubuntu ships, where every single time you exit a 3d game the system crashes (and Ubuntu users accept this as normal?!?), then ext3 might be a better choice since you'll reduce the chance of data loss when the system locks up or crashes thanks to the aforemention crappy proprietary video drivers from Nvidia. On the other hand, crappy flash drives *do* have really bad write amplification effects, where a 4K write can cause 128k or more worth of flash to be rewritten, such that using ext3 could seriously degrade the lifetime of said crappy flash drive; furthermore, the crappy flash drives have such terribly write performance that using ext3 can be a performance nightmare. This of course, doesn't apply to well-implemented SSD's, such as the Intel's X25-M and X18-M. So here your mileage may vary. Still, if you are using crappy proprietary drivers which cause system hangs and crashes at a far greater rate than power fail-induced unclean shutdowns, ext3 *still* might be the better choice, even with crappy flash drives.

The best thing to do, of course, is to improve your storage stack; use competently implemented SSD's instead of crap flash cards. If your hardware RAID card supports a battery option, *get* the battery. Add a UPS to your system. Provision your RAID array with hot spares, and regularly scrub (read-test) your array so that failed drives can be detected early. Make sure you configure your MD setup so that you get e-mail when a hard drive fails and the array starts running in degraded mode, so you can replace the failed drive ASAP.

At the end of the day, filesystems are not magic. They can't compensate for crap hardware, or incompetently administered machines.

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

I agree with the whole write up outside of the above - degraded RAID does meet this requirement unless you have a second (or third, counting the split write) failure during the rebuild.

Quoted 4 lines · Ric Wheeler

I agree with the whole write up outside of the above - degraded RAID does meet this requirement unless you have a second (or third, counting the split write) failure during the rebuild.

The argument is that if the degraded RAID array is running in this state for a long time, and the power fails while the software RAID is in the middle of writing out a stripe, such that the stripe isn't completely written out, we could lose all of the data in that stripe.

In other words, a power failure in the middle of writing out a stripe in a degraded RAID array counts as a second failure.

To me, this isn't a particularly interesting or newsworthy point, since a competent system administrator who cares about his data and/or his hardware will (a) have a UPS, and (b) be running with a hot spare and/or will imediately replace a failed drive in a RAID array.

- Ted

Quoted 4 lines · Ric Wheeler

I agree with the whole write up outside of the above - degraded RAID does meet this requirement unless you have a second (or third, counting the split write) failure during the rebuild.

The argument is that if the degraded RAID array is running in this state for a long time, and the power fails while the software RAID is in the middle of writing out a stripe, such that the stripe isn't completely written out, we could lose all of the data in that stripe.

In other words, a power failure in the middle of writing out a stripe in a degraded RAID array counts as a second failure.

To me, this isn't a particularly interesting or newsworthy point, since a competent system administrator who cares about his data and/or his hardware will (a) have a UPS, and (b) be running with a hot spare and/or will imediately replace a failed drive in a RAID array.

- Ted

Quoted 11 lines · Theodore Tso

Quoted 4 lines · Ric Wheeler

I agree with the whole write up outside of the above - degraded RAID does meet this requirement unless you have a second (or third, counting the split write) failure during the rebuild.

The argument is that if the degraded RAID array is running in this state for a long time, and the power fails while the software RAID is in the middle of writing out a stripe, such that the stripe isn't completely written out, we could lose all of the data in that stripe.

Quoted 6 lines

In other words, a power failure in the middle of writing out a stripe in a degraded RAID array counts as a second failure.
To me, this isn't a particularly interesting or newsworthy point,
since a competent system administrator who cares about his data and/or his hardware will (a) have a UPS, and (b) be running with a hot spare and/or will imediately replace a failed drive in a RAID array.

I agree that this is not an interesting (or likely) scenario, certainly when compared to the much more frequent failures that RAID will protect against which is why I object to the document as Pavel suggested. It will steer people away from using RAID and directly increase their chances of losing their data if they use just a single disk.

Quoted 19 lines · Ric Wheeler

Quoted 11 lines · Theodore Tso

Quoted 4 lines · Ric Wheeler

I agree with the whole write up outside of the above - degraded RAID does meet this requirement unless you have a second (or third, counting the split write) failure during the rebuild.

The argument is that if the degraded RAID array is running in this state for a long time, and the power fails while the software RAID is in the middle of writing out a stripe, such that the stripe isn't completely written out, we could lose all of the data in that stripe.

Quoted 6 lines

In other words, a power failure in the middle of writing out a stripe in a degraded RAID array counts as a second failure.
To me, this isn't a particularly interesting or newsworthy point,
since a competent system administrator who cares about his data and/or his hardware will (a) have a UPS, and (b) be running with a hot spare and/or will imediately replace a failed drive in a RAID array.

Quoted 5 lines

I agree that this is not an interesting (or likely) scenario, certainly when compared to the much more frequent failures that RAID will protect against which is why I object to the document as Pavel suggested. It will steer people away from using RAID and directly increase their chances of losing their data if they use just a single disk.

So instead of fixing or at least documenting known software deficiency in Linux MD stack, you'll try to surpress that information so that people use more of raid5 setups?

Perhaps the better documentation will push them to RAID1, or maybe make them buy an UPS?

Quoted 7 lines

So instead of fixing or at least documenting known software deficiency in Linux MD stack, you'll try to surpress that information so that people use more of raid5 setups?

Perhaps the better documentation will push them to RAID1, or maybe make them buy an UPS?

people aren't objecting to better documentation, they are objecting to misleading documentation.

for flash drives the danger is very straightforward (although even then you have to note that it depends heavily on the firmware of the device, some will loose lots of data, some won't loose any)

you are generalizing that since you have lost data on flash drives, all flash drives are dangerous.

but the super simplified statement you keep trying to make is significantly overstating and oversimplifying the problem.

Quoted 5 lines

Quoted 5 lines

So instead of fixing or at least documenting known software deficiency in Linux MD stack, you'll try to surpress that information so that people use more of raid5 setups?

Quoted 6 lines

Quoted 3 lines

Perhaps the better documentation will push them to RAID1, or maybe make them buy an UPS?

people aren't objecting to better documentation, they are objecting to misleading documentation.

Actually Ric is. He's trying hard to make RAID5 look better than it really is.

Quoted 6 lines

Quoted 4 lines

people aren't objecting to better documentation, they are objecting to misleading documentation.

Actually Ric is. He's trying hard to make RAID5 look better than it really is.

I object to misleading and dangerous documentation that you have proposed. I spend a lot of time working in data integrity, talking and writing about it so I care deeply that we don't misinform people.

In this thread, I put out a draft that is accurate several times and you have failed to respond to it.

The big picture that you don't agree with is:

(1) RAID (specifically MD RAID) will dramatically improve data integrity for real users. This is not a statement of opinion, this is a statement of fact that has been shown to be true in large scale deployments with commodity hardware.

(2) RAID5 protects you against a single failure and your test case purposely injects a double failure.

(4) Data loss occurs in non-journalling file systems and journalling file systems when you suffer double failures or hot unplug storage, especially inexpensive FLASH parts.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

people aren't objecting to better documentation, they are objecting to misleading documentation.

Quoted 6 lines

Quoted 2 lines

Actually Ric is. He's trying hard to make RAID5 look better than it really is.

I object to misleading and dangerous documentation that you have proposed. I spend a lot of time working in data integrity, talking and writing about it so I care deeply that we don't misinform people.

Yes, truth is dangerous. To vendors selling crap products.

Quoted 10 lines · Pavel Machek

Quoted 9 lines

Quoted 5 lines

Quoted 2 lines

people aren't objecting to better documentation, they are objecting to misleading documentation.

Actually Ric is. He's trying hard to make RAID5 look better than it really is.

I object to misleading and dangerous documentation that you have proposed. I spend a lot of time working in data integrity, talking and writing about it so I care deeply that we don't misinform people.

Quoted 1 line

Yes, truth is dangerous. To vendors selling crap products.

Pavel, you have no information and an attitude of not wanting to listen to anyone who has real experience or facts. Not just me, but also Ted and others.

Totally pointless to reply to you further.

USB keys are the modern successor to floppy drives, and even now Documentation/blockdev/floppy.txt is still full of some of the torturous workarounds implemented for that over the past 2 decades. The hardware existed, and instead of turning up their nose at it they made it work as best they could.

Perhaps what's needed for the flash thing is a userspace package, the way mdutils made floppies a lot more usable than the kernel managed at the time. For the flash problem perhaps some FUSE thing a bit like mtdblock might be nice, a translation layer remapping an arbitrary underlying block device into larger granularity chunks and being sure to do the "write the new one before you erase the old one" trick that so many hardware-only flash devices _don't_, and then maybe even use Pavel's crash tool to figure out the write granularity of various sticks and ship it with a whitelist people can email updates to so we don't have to guess large. (Pressure on the USB vendors to give us a "raw view" extension bypassing the "pretend to be a hard drive, with remapping" hardware in future devices would be nice too, but won't help any of the hardware out in the field. I'm not sure that block remapping wouldn't screw up _this_ approach either, but it's an example of something that culd be _tried_.)

However, thinking about how to _fix_ a problem is predicated on acknowledging that there actually _is_ a problem. "The hardware is not physically damaged but your data was lost" sounds to me like a software problem, and thus something software could at least _attempt_ to address. "There's millions of 'em, Linux can't cope" doesn't seem like a useful approach.

Quoted 5 lines

However, thinking about how to _fix_ a problem is predicated on acknowledging that there actually _is_ a problem. "The hardware is not physically damaged but your data was lost" sounds to me like a software problem, and thus something software could at least _attempt_ to address. "There's millions of 'em, Linux can't cope" doesn't seem like a useful approach.

no other OS avoids this problem either.

I actually don't see how you can do this from userspace, because when you write to the device you have _no_ idea where on the device your data will actually land.

Quoted 2 lines · On Wednesday 02 September 2009 15:42:19 Ric Wheeler wrote:

Totally pointless to reply to you further.

For the record, I've been able to follow Pavel's arguments, and I've been able to follow Ted's arguments. But as far as I can tell, you're arguing about a different topic than the rest of us.

However, thinking about how to _fix_ a problem is predicated on acknowledging that there actually _is_ a problem. "The hardware is not physically damaged but your data was lost" sounds to me like a software problem, and thus something software could at least _attempt_ to address. "There's millions of 'em, Linux can't cope" doesn't seem like a useful approach.

I already addressed the software raid thing last post.

Quoted 6 lines · Rob Landley

Quoted 1 line · On Wednesday 02 September 2009 15:42:19 Ric Wheeler wrote:

Totally pointless to reply to you further.

For the record, I've been able to follow Pavel's arguments, and I've been able to follow Ted's arguments. But as far as I can tell, you're arguing about a different topic than the rest of us.

I had no trouble following what Ric was arguing about.

Ric never said "use only the best devices and you won't have problems".

Ric was arguing the exact opposite - ALL devices are crap if you define crap as "can loose data".

What he is saying is you need to UNDERSTAND
your devices and their behavior and you must act accordingly.

PAVEL DID NOT ACT ACCORDING TO HIS DEVICE LIMITATIONS.

We understand he was clueless, but user error is still user error!

And Ric said do not stigmatize whole classes of A) devices, B) raid, and C) filesystems with "Pavel says...".

I think he understands he was clueless too, that's why he investigated the failure and wrote it up for posterity.

Quoted 2 lines

And Ric said do not stigmatize whole classes of A) devices, B) raid, and C) filesystems with "Pavel says...".

I don't care what "Pavel says", so you can leave the ad hominem at the door, thanks.

The kernel presents abstractions, such as block device nodes. Sometimes implementation details bubble through those abstractions. Presumably, we agree on that so far.

Quoted 6 lines · Rob Landley

I think he understands he was clueless too, that's why he investigated the failure and wrote it up for posterity.

Quoted 2 lines

And Ric said do not stigmatize whole classes of A) devices, B) raid, and C) filesystems with "Pavel says...".

Quoted 2 lines

I don't care what "Pavel says", so you can leave the ad hominem at the door, thanks.

See, this is exactly the problem we have with all the proposed documentation. The reader (you) did not get what the writer (me) was trying to say. That does not say either of us was wrong in what we thought was meant, simply that we did not communicate.

Quoted 9 lines · On Thursday 03 September 2009 09:14:43 jim owens wrote:

Quoted 7 lines · Rob Landley

I think he understands he was clueless too, that's why he investigated the failure and wrote it up for posterity.

Quoted 3 lines

And Ric said do not stigmatize whole classes of A) devices, B) raid, and C) filesystems with "Pavel says...".

Quoted 8 lines

Quoted 3 lines

I don't care what "Pavel says", so you can leave the ad hominem at the door, thanks.

See, this is exactly the problem we have with all the proposed documentation. The reader (you) did not get what the writer (me) was trying to say. That does not say either of us was wrong in what we thought was meant, simply that we did not communicate.

That's why I've mostly stopped bothering with this thread. I could respond to Ric Wheeler's latest (what does write barriers have to do with whether or not a multi-sector stripe is guaranteed to be atomically updated during a panic or power failure?) but there's just no point.

The LWN article on the topic is out, and incomplete as it is I expect it's the best documentation anybody will actually _read_.

Rob

Quoted 19 lines · Rob Landley

Quoted 9 lines · On Thursday 03 September 2009 09:14:43 jim owens wrote:

Quoted 7 lines · Rob Landley

I think he understands he was clueless too, that's why he investigated the failure and wrote it up for posterity.

Quoted 3 lines

And Ric said do not stigmatize whole classes of A) devices, B) raid, and C) filesystems with "Pavel says...".

Quoted 8 lines

Quoted 3 lines

I don't care what "Pavel says", so you can leave the ad hominem at the door, thanks.

See, this is exactly the problem we have with all the proposed documentation. The reader (you) did not get what the writer (me) was trying to say. That does not say either of us was wrong in what we thought was meant, simply that we did not communicate.

Quoted 5 lines

That's why I've mostly stopped bothering with this thread. I could respond to Ric Wheeler's latest (what does write barriers have to do with whether or not a multi-sector stripe is guaranteed to be atomically updated during a panic or power failure?) but there's just no point.

The point of that post was that the failure that you and Pavel both attribute to RAID and journalled fs happens whenever the storage cannot promise to do atomic writes of a logical FS block (prevent torn pages/split writes/etc). I gave a specific example of why this happens even with simple, single disk systems.

Quoted 2 lines

The LWN article on the topic is out, and incomplete as it is I expect it's the best documentation anybody will actually _read_.

Quoted 33 lines · Ric Wheeler

Quoted 31 lines · Rob Landley

Quoted 12 lines · On Thursday 03 September 2009 09:14:43 jim owens wrote:

Quoted 9 lines · Rob Landley

I think he understands he was clueless too, that's why he investigated the failure and wrote it up for posterity.

Quoted 4 lines

And Ric said do not stigmatize whole classes of A) devices, B) raid, and C) filesystems with "Pavel says...".

Quoted 10 lines

Quoted 4 lines

I don't care what "Pavel says", so you can leave the ad hominem at the door, thanks.

See, this is exactly the problem we have with all the proposed documentation. The reader (you) did not get what the writer (me) was trying to say. That does not say either of us was wrong in what we thought was meant, simply that we did not communicate.

That's why I've mostly stopped bothering with this thread. I could respond to Ric Wheeler's latest (what does write barriers have to do with whether or not a multi-sector stripe is guaranteed to be atomically updated during a panic or power failure?) but there's just no point.

Quoted 6 lines

The point of that post was that the failure that you and Pavel both attribute to RAID and journalled fs happens whenever the storage cannot promise to do atomic writes of a logical FS block (prevent torn pages/split writes/etc). I gave a specific example of why this happens even with simple, single disk systems.

ext3 does not expect atomic write of 4K block, according to Ted. So no, it is not broken on single disk.

Quoted 5 lines

Quoted 5 lines

The LWN article on the topic is out, and incomplete as it is I expect it's the best documentation anybody will actually _read_.

Would anyone (probably privately?) share the lwn link?
Pavel

Quoted 2 lines

Quoted 2 lines

The LWN article on the topic is out, and incomplete as it is I expect it's the best documentation anybody will actually _read_.

Would anyone (probably privately?) share the lwn link?

Quoted 4 lines · On Sat, 5 Sep 2009 12:28:10 +0200 Pavel Machek <[email protected]> wrote:

Quoted 2 lines

Quoted 2 lines

The LWN article on the topic is out, and incomplete as it is I expect it's the best documentation anybody will actually _read_.

Would anyone (probably privately?) share the lwn link?

	http://lwn.net/SubscriberLink/349970/9875eff987190551/

assuming you've not already gotten one from elsewhere.

Thanks, and thanks for nice article!

Quoted 2 lines

(2) RAID5 protects you against a single failure and your test case purposely injects a double failure.

Most people would be surprised that press of reset button is 'failure' in this context.

From: Rob Landley <[email protected]>

Add more warnings to the "Boot time assembly of degraded/dirty arrays" section, explaining that using a journaling filesystem can't overcome this problem.

Signed-off-by Rob Landley <[email protected]>

From: Rob Landley<[email protected]>

Add more warnings to the "Boot time assembly of degraded/dirty arrays" section, explaining that using a journaling filesystem can't overcome this problem.

Signed-off-by Rob Landley<[email protected]>

---

  Documentation/md.txt |   17 +++++++++++++++++
  1 file changed, 17 insertions(+)

Documentation/md.txt +10 -0

index 4edd39e..52b8450 100644
@@ -75,6 +75,23 @@ So, to boot with a root filesystem of a dirty degraded raid[56], use
 
     md-mod.start_dirty_degraded=1
 
+Note that Journaling filesystems do not effectively protect data in this
+case, because the update granularity of the RAID is larger than the journal
+was designed to expect.  Reconstructing data via partity information involes
+matching together corresponding stripes, and updating only some of these
+stripes renders the corresponding data in all the unmatched stripes
+meaningless.  Thus seemingly unrelated data in other parts of the filesystem
+(stored in the unmatched stripes) can become unreadable after a partial
+update, but the journal is only aware of the parts it modified, not the
+"collateral damage" elsewhere in the filesystem which was affected by those
+changes.

Quoted 31 lines · Rob Landley

From: Rob Landley<[email protected]>

Add more warnings to the "Boot time assembly of degraded/dirty arrays" section, explaining that using a journaling filesystem can't overcome this problem.

Signed-off-by Rob Landley<[email protected]>

---

  Documentation/md.txt |   17 +++++++++++++++++
  1 file changed, 17 insertions(+)

Documentation/md.txt +10 -0

index 4edd39e..52b8450 100644
@@ -75,6 +75,23 @@ So, to boot with a root filesystem of a dirty degraded raid[56], use
 
     md-mod.start_dirty_degraded=1
 
+Note that Journaling filesystems do not effectively protect data in this
+case, because the update granularity of the RAID is larger than the journal
+was designed to expect.  Reconstructing data via partity information involes
+matching together corresponding stripes, and updating only some of these
+stripes renders the corresponding data in all the unmatched stripes
+meaningless.  Thus seemingly unrelated data in other parts of the filesystem
+(stored in the unmatched stripes) can become unreadable after a partial
+update, but the journal is only aware of the parts it modified, not the
+"collateral damage" elsewhere in the filesystem which was affected by those
+changes.

Documentation/md.txt So, to boot with a root filesystem of a dirty degraded raid[56], use · line 97

Quoted 10 lines

Documentation/md.txt +7 -0

+
+Thus successful journal replay proves nothing in this context, and even a
+full fsck only shows whether or not the filesystem's metadata was affected.
+(A proper solution to this problem would involve adding journaling to the RAID
+itself, at least during degraded writes.  In the meantime, try not to allow
+a system to shut down uncleanly with its RAID both dirty and degraded, it
+can handle one but not both.)
 
  Superblock formats
  ------------------

NACK.

Now you have moved the inaccurate documentation about journalling file systems into the MD documentation.

Repeat after me:

(1) partial writes to a RAID stripe (with or without file systems, with or without journals) create an invalid stripe

(2) partial writes can be prevented in most cases by running with write cache disabled or working barriers

Rob, you should really try to take a few disks, build a working MD RAID5 group and test your ideas. Try it with and without the write cache enabled.

Quoted 3 lines

for flash drives the danger is very straightforward (although even then you have to note that it depends heavily on the firmware of the device, some will loose lots of data, some won't loose any)

I have not seen one that works :-(.

Quoted 2 lines

you are generalizing that since you have lost data on flash drives, all flash drives are dangerous.

Do the flash manufacturers claim they do not cause collateral damage during powerfail? If not, they probably are dangerous.

Anyway, you wanted a test, and one is attached. It normally takes like 4 unplugs to uncover problems.

Quoted 5 lines

Quoted 3 lines

for flash drives the danger is very straightforward (although even then you have to note that it depends heavily on the firmware of the device, some will loose lots of data, some won't loose any)

I have not seen one that works :-(.

so let's get broader testing (including testing the SSDs as well as the thumb drives)

Quoted 2 lines

Anyway, you wanted a test, and one is attached. It normally takes like 4 unplugs to uncover problems.

Ok, help me understand this.

I copy these two files to a system, change them to point at the correct device, run them and unplug the drive while it's running.

when I plug the device back in, how do I tell if it lost something unexpected? since you are writing from urandom I have no idea what data _should_ be on the drive, so how can I detect that a data block has been corrupted?

Quoted 22 lines · Ric Wheeler

Quoted 13 lines · Theodore Tso

Quoted 5 lines · Ric Wheeler

I agree with the whole write up outside of the above - degraded RAID does meet this requirement unless you have a second (or third, counting the split write) failure during the rebuild.

The argument is that if the degraded RAID array is running in this state for a long time, and the power fails while the software RAID is in the middle of writing out a stripe, such that the stripe isn't completely written out, we could lose all of the data in that stripe.

Quoted 7 lines

In other words, a power failure in the middle of writing out a stripe in a degraded RAID array counts as a second failure.
To me, this isn't a particularly interesting or newsworthy point,
since a competent system administrator who cares about his data and/or his hardware will (a) have a UPS, and (b) be running with a hot spare and/or will imediately replace a failed drive in a RAID array.

Quoted 6 lines

I agree that this is not an interesting (or likely) scenario, certainly when compared to the much more frequent failures that RAID will protect against which is why I object to the document as Pavel suggested. It will steer people away from using RAID and directly increase their chances of losing their data if they use just a single disk.

So instead of fixing or at least documenting known software deficiency in Linux MD stack, you'll try to surpress that information so that people use more of raid5 setups?

Perhaps the better documentation will push them to RAID1, or maybe make them buy an UPS?
Pavel

Quoted 5 lines

I agree that this is not an interesting (or likely) scenario, certainly when compared to the much more frequent failures that RAID will protect against which is why I object to the document as Pavel suggested. It will steer people away from using RAID and directly increase their chances of losing their data if they use just a single disk.

So instead of fixing or at least documenting known software deficiency in Linux MD stack, you'll try to surpress that information so that people use more of raid5 setups?

Quoted 6 lines · Pavel Machek

Quoted 5 lines

I agree that this is not an interesting (or likely) scenario, certainly when compared to the much more frequent failures that RAID will protect against which is why I object to the document as Pavel suggested. It will steer people away from using RAID and directly increase their chances of losing their data if they use just a single disk.

Quoted 3 lines

So instead of fixing or at least documenting known software deficiency in Linux MD stack, you'll try to surpress that information so that people use more of raid5 setups?

First of all, it's not a "known software deficiency"; you can't do anything about a degraded RAID array, other than to replace the failed disk.

"ext2 and ext3 have this surprising dependency that disks act like disks". (alarmist)

Quoted 6 lines · Pavel Machek

Quoted 5 lines

I agree that this is not an interesting (or likely) scenario, certainly when compared to the much more frequent failures that RAID will protect against which is why I object to the document as Pavel suggested. It will steer people away from using RAID and directly increase their chances of losing their data if they use just a single disk.

Quoted 3 lines

So instead of fixing or at least documenting known software deficiency in Linux MD stack, you'll try to surpress that information so that people use more of raid5 setups?

First of all, it's not a "known software deficiency"; you can't do anything about a degraded RAID array, other than to replace the failed disk.

"ext2 and ext3 have this surprising dependency that disks act like disks". (alarmist)

Quoted 3 lines · Ric Wheeler

I agree with the whole write up outside of the above - degraded RAID does meet this requirement unless you have a second (or third, counting the split write) failure during the rebuild.

The argument is that if the degraded RAID array is running in this state for a long time, and the power fails while the software RAID is in the middle of writing out a stripe, such that the stripe isn't completely written out, we could lose all of the data in that stripe.

In other words, a power failure in the middle of writing out a stripe in a degraded RAID array counts as a second failure.

To me, this isn't a particularly interesting or newsworthy point, since a competent system administrator

who cares about his data and/or
his hardware will (a) have a UPS,

, and (b) be running with a hot spare
and/or will imediately replace a failed drive in a RAID array.

- Ted

Quoted 2 lines

To me, this isn't a particularly interesting or newsworthy point, since a competent system administrator

I'm a bit concerned by the argument that we don't need to document serious pitfalls because every Linux system has a sufficiently competent administrator they already know stuff that didn't even come up until the second or third day it was discussed on lkml.

"You're documenting it wrong" != "you shouldn't document it".

Quoted 2 lines

who cares about his data and/or
his hardware will (a) have a UPS,

I worked at a company that retested their UPSes a year after installing them and found that _none_ of them supplied more than 15 seconds charge, and when they dismantled them the batteries had physically bloated inside their little plastic cases. (Same company as the dead air conditioner, possibly overheating was involved but the little _lights_ said everything was ok.)

That was by no means the first UPS I'd seen die, the suckers have a higher failure rate than hard drives in my experience. This is a device where the batteries get constantly charged and almost never tested because if it _does_ fail you just rebooted your production server, so a lot of smaller companies think they have one but actually don't.

Quoted 2 lines

, and (b) be running with a hot spare
and/or will imediately replace a failed drive in a RAID array.

Here's hoping they shut the system down properly to install the new drive in the raid then, eh? Not accidentally pull the plug before it's finished running the ~7 minutes of shutdown scripts in the last Red Hat Enterprise I messed with...

If "degraded array" just means "don't have a replacement disk yet", then it sounds like what Pavel wants to document is "don't write to a degraded array at all, because power failures can cost you data due to write granularity being larger than filesystem block size". (Which still comes as news to some of us, and you need a way to remount mount the degraded array read only until the sysadmin can fix it.)

Quoted 7 lines · Rob Landley

Quoted 2 lines

To me, this isn't a particularly interesting or newsworthy point, since a competent system administrator

I'm a bit concerned by the argument that we don't need to document serious pitfalls because every Linux system has a sufficiently competent administrator they already know stuff that didn't even come up until the second or third day it was discussed on lkml.

I'm not convinced that information which needs to be known by System Administrators is best documented in the kernel Documentation directory. Should there be a HOWTO document on stuff like that? Sure, if someone wants to put something like that together, having free documentation about ways to set up your storage stack in a sane way is not a bad thing.

It should be noted that these sorts of issues are discussed in various books targetted at System Administrators, and in Usenix's System Administration tutorials. The computer industry is highly specialized, and so just because an OS kernel hacker might not be familiar with these issues, doesn't mean that professionals whose job it is to run data centers don't know about these things! Similarly, you could be a whiz at Linux's networking stack, but you might not know about certain pitfalls in configuring a Cisco router using IOS; does that mean we should have an IOS tutorial in the kernel documentation directory? I'm not so sure about that!

Quoted 1 line

"You're documenting it wrong" != "you shouldn't document it".

Sure, but the fact that we don't currently say much about storage stacks doesn't mean we should accept a patch that might actively
mislead people. I'm NACK'ing the patch on that basis.

Quoted 10 lines

Quoted 2 lines

who cares about his data and/or
his hardware will (a) have a UPS,

I worked at a company that retested their UPSes a year after installing them and found that _none_ of them supplied more than 15 seconds charge, and when they dismantled them the batteries had physically bloated inside their little plastic cases. (Same company as the dead air conditioner, possibly overheating was involved but the little _lights_ said everything was ok.)

Quoted 6 lines

That was by no means the first UPS I'd seen die, the suckers have a higher failure rate than hard drives in my experience. This is a device where the batteries get constantly charged and almost never tested because if it _does_ fail you just rebooted your production server, so a lot of smaller companies think they have one but actually don't.

Sounds like they were using really cheap UPS's; certainly not the kind I would expect to find in a data center. And if company's system administrator is using the cheapest possible consumer-grade UPS's, then yes, they might have a problem. Even an educational institution like MIT, where I was an network administrator some 15 years ago, had proper UPS's, *and* we had a diesel generator which kicked in after 15 seconds --- and we tested the diesel generator every Friday morning, to make sure it worked properly.

Quoted 7 lines

Quoted 2 lines

, and (b) be running with a hot spare
and/or will imediately replace a failed drive in a RAID array.

Here's hoping they shut the system down properly to install the new drive in the raid then, eh? Not accidentally pull the plug before it's finished running the ~7 minutes of shutdown scripts in the last Red Hat Enterprise I messed with...

Even my home RAID array uses hot-plug SATA disks, so I can replace a failed disk without shutting down my system. (And yes, I have a backup battery for the hardware RAID, and the firmware runs periodic tests on it; the hardware RAID card also will send me e-mail if a RAID array drive fails and it needs to use my hot-spare. At that point, I order a new hard drive, secure in the knowledge that the system can still suffer another drive failure before falling into degraded mode. And no, this isn't some expensive enterprise RAID setup; this is just a mid-range Areca RAID card.)

Quoted 7 lines

If "degraded array" just means "don't have a replacement disk yet", then it sounds like what Pavel wants to document is "don't write to a degraded array at all, because power failures can cost you data due to write granularity being larger than filesystem block size". (Which still comes as news to some of us, and you need a way to remount mount the degraded array read only until the sysadmin can fix it.)

If you want to document that as a property of RAID arrays, sure. But it's not something that should live in Documentation/filesystems/ext2.txt and Documentation/filesystems/ext3.txt. The MD RAID howto might be a better place, since it's far more likely more users will read it. How many system administrators read what's in the kernel's Documentation directory, after all, and this is basic information about how RAID works; it's not necessarily something that someone would *expect* to be in kernel documentation, nor would necessarily go looking for it there. And the reality is that it's not like most people go reading Documentation/* for pleasure. :-)

BTW, the RAID write atomicity issue and the possibility of failures cause data loss *is* documented in the Wikipedia article on RAID. It's not as written as direct practical advice to a system administrator (you'd have to go to a book that is really targetted at system administrators to find that sort of thing).

- Ted

Quoted 7 lines · Rob Landley

Quoted 2 lines

To me, this isn't a particularly interesting or newsworthy point, since a competent system administrator

I'm a bit concerned by the argument that we don't need to document serious pitfalls because every Linux system has a sufficiently competent administrator they already know stuff that didn't even come up until the second or third day it was discussed on lkml.

I'm not convinced that information which needs to be known by System Administrators is best documented in the kernel Documentation directory. Should there be a HOWTO document on stuff like that? Sure, if someone wants to put something like that together, having free documentation about ways to set up your storage stack in a sane way is not a bad thing.

It should be noted that these sorts of issues are discussed in various books targetted at System Administrators, and in Usenix's System Administration tutorials. The computer industry is highly specialized, and so just because an OS kernel hacker might not be familiar with these issues, doesn't mean that professionals whose job it is to run data centers don't know about these things! Similarly, you could be a whiz at Linux's networking stack, but you might not know about certain pitfalls in configuring a Cisco router using IOS; does that mean we should have an IOS tutorial in the kernel documentation directory? I'm not so sure about that!

Quoted 1 line

"You're documenting it wrong" != "you shouldn't document it".

Sure, but the fact that we don't currently say much about storage stacks doesn't mean we should accept a patch that might actively
mislead people. I'm NACK'ing the patch on that basis.

Quoted 10 lines

Quoted 2 lines

who cares about his data and/or
his hardware will (a) have a UPS,

I worked at a company that retested their UPSes a year after installing them and found that _none_ of them supplied more than 15 seconds charge, and when they dismantled them the batteries had physically bloated inside their little plastic cases. (Same company as the dead air conditioner, possibly overheating was involved but the little _lights_ said everything was ok.)

Quoted 6 lines

That was by no means the first UPS I'd seen die, the suckers have a higher failure rate than hard drives in my experience. This is a device where the batteries get constantly charged and almost never tested because if it _does_ fail you just rebooted your production server, so a lot of smaller companies think they have one but actually don't.

Sounds like they were using really cheap UPS's; certainly not the kind I would expect to find in a data center. And if company's system administrator is using the cheapest possible consumer-grade UPS's, then yes, they might have a problem. Even an educational institution like MIT, where I was an network administrator some 15 years ago, had proper UPS's, *and* we had a diesel generator which kicked in after 15 seconds --- and we tested the diesel generator every Friday morning, to make sure it worked properly.

Quoted 7 lines

Quoted 2 lines

, and (b) be running with a hot spare
and/or will imediately replace a failed drive in a RAID array.

Here's hoping they shut the system down properly to install the new drive in the raid then, eh? Not accidentally pull the plug before it's finished running the ~7 minutes of shutdown scripts in the last Red Hat Enterprise I messed with...

Even my home RAID array uses hot-plug SATA disks, so I can replace a failed disk without shutting down my system. (And yes, I have a backup battery for the hardware RAID, and the firmware runs periodic tests on it; the hardware RAID card also will send me e-mail if a RAID array drive fails and it needs to use my hot-spare. At that point, I order a new hard drive, secure in the knowledge that the system can still suffer another drive failure before falling into degraded mode. And no, this isn't some expensive enterprise RAID setup; this is just a mid-range Areca RAID card.)

Quoted 7 lines

If "degraded array" just means "don't have a replacement disk yet", then it sounds like what Pavel wants to document is "don't write to a degraded array at all, because power failures can cost you data due to write granularity being larger than filesystem block size". (Which still comes as news to some of us, and you need a way to remount mount the degraded array read only until the sysadmin can fix it.)

If you want to document that as a property of RAID arrays, sure. But it's not something that should live in Documentation/filesystems/ext2.txt and Documentation/filesystems/ext3.txt. The MD RAID howto might be a better place, since it's far more likely more users will read it. How many system administrators read what's in the kernel's Documentation directory, after all, and this is basic information about how RAID works; it's not necessarily something that someone would *expect* to be in kernel documentation, nor would necessarily go looking for it there. And the reality is that it's not like most people go reading Documentation/* for pleasure. :-)

BTW, the RAID write atomicity issue and the possibility of failures cause data loss *is* documented in the Wikipedia article on RAID. It's not as written as direct practical advice to a system administrator (you'd have to go to a book that is really targetted at system administrators to find that sort of thing).

- Ted

One thing that does need fixing for some MD configurations is to stress again that we need to make sure that barrier operations are properly supported or users will need to disable the write cache on devices with volatile write caches.

DM has some barrier code... but the above code was pasted from DM's make_request function, so I am guessing that DM's barrier stuff is incomplete and disabled at present.

Quoted 4 lines · Ric Wheeler

One thing that does need fixing for some MD configurations is to stress again that we need to make sure that barrier operations are properly supported or users will need to disable the write cache on devices with volatile write caches.

Agreed; chime in on Christoph's linux-vfs thread if people have input.

I quickly glanced at MD and DM. Currently, upstream, we see a lot of

        if (unlikely(bio_barrier(bio))) {
                bio_endio(bio, -EOPNOTSUPP);
                return 0;
        }

in DM and MD make_request functions.

Only md/raid1 supports barriers at present, it seems. None of the other MD drivers support barriers.

On Thu 2009-08-27 12:54:05, Jeff Garzik wrote:

Quoted 1 line

Only md/raid1 supports barriers at present, it seems. None of the other MD drivers support barriers.

Not even md/raid0? Ouch :-(.

Quoted 3 lines · Rob Landley

Quoted 2 lines

To me, this isn't a particularly interesting or newsworthy point, since a competent system administrator

Quoted 4 lines

I'm a bit concerned by the argument that we don't need to document serious pitfalls because every Linux system has a sufficiently competent administrator they already know stuff that didn't even come up until the second or third day it was discussed on lkml.

I'm not convinced that information which needs to be known by System Administrators is best documented in the kernel Documentation directory. Should there be a HOWTO document on stuff like that?

Quoted 7 lines

If "degraded array" just means "don't have a replacement disk yet", then it sounds like what Pavel wants to document is "don't write to a degraded array at all, because power failures can cost you data due to write granularity being larger than filesystem block size". (Which still comes as news to some of us, and you need a way to remount mount the degraded array read only until the sysadmin can fix it.)

If you want to document that as a property of RAID arrays, sure. But it's not something that should live in Documentation/filesystems/ext2.txt and Documentation/filesystems/ext3.txt. The MD RAID howto might be a

Quoted 3 lines · Rob Landley

Quoted 2 lines

To me, this isn't a particularly interesting or newsworthy point, since a competent system administrator

Quoted 4 lines

I'm a bit concerned by the argument that we don't need to document serious pitfalls because every Linux system has a sufficiently competent administrator they already know stuff that didn't even come up until the second or third day it was discussed on lkml.

I'm not convinced that information which needs to be known by System Administrators is best documented in the kernel Documentation directory. Should there be a HOWTO document on stuff like that?

Quoted 7 lines

If "degraded array" just means "don't have a replacement disk yet", then it sounds like what Pavel wants to document is "don't write to a degraded array at all, because power failures can cost you data due to write granularity being larger than filesystem block size". (Which still comes as news to some of us, and you need a way to remount mount the degraded array read only until the sysadmin can fix it.)

If you want to document that as a property of RAID arrays, sure. But it's not something that should live in Documentation/filesystems/ext2.txt and Documentation/filesystems/ext3.txt. The MD RAID howto might be a

3) Does that mean that you shouldn't use ext3 on RAID drives? Of course not! First of all, Ext3 still saves you against kernel panics and hangs caused by device driver bugs or other kernel hangs. You will lose less data, and avoid needing to run a long and painful fsck after a forced reboot, compared to if you used ext2. You are making